Files
swaphb 6e7f5b39fb
Kube-bench CIS scan / Scan ephemeral K3s cluster (push) Successful in 59s
make falco alerting opt in
2026-08-16 21:36:09 -04:00
..
2026-08-16 21:36:09 -04:00

Falco alerting and runtime troubleshooting

The default Falco deployment runs Falco and its custom rules without Falcosidekick. Slack forwarding is opt-in because a webhook is a credential and should not be required for local cluster bootstrap.

Optional Slack alerting

Falco alert delivery must use a Kubernetes Secret. Do not commit a Slack Incoming Webhook URL, Slack token, Discord webhook URL, or generated Secret manifest to this repository.

When alerting is enabled, the Ansible role creates this Secret contract. The Secret is never stored in Git:

apiVersion: v1
kind: Secret
metadata:
  name: falco-alerting
  namespace: falco
type: Opaque
stringData:
  webhook-url: https://hooks.slack.com/services/REPLACE/ME

Store the real value in an Ansible Vault file:

falco_alerting_enabled: true
falco_slack_webhook_url: https://hooks.slack.com/services/REPLACE/ME

Run the bootstrap with the encrypted file:

ansible-vault encrypt ansible/vars/falco-alerting.vault.yml
ansible-playbook \
  -i ansible/inventory.flatcar-k3d.yml \
  ansible/bootstrap-flatcar-k3d.yml \
  --ask-vault-pass \
  -e @ansible/vars/falco-alerting.vault.yml

The Ansible role creates the Secret without logging its value and applies the optional falco-alerting ArgoCD Application only when both the toggle and webhook value are supplied. When disabled, it removes the optional Application, Component, output Config, and Secret. The default Falco Application does not include Falcosidekick resources.

The webhook-url key is injected into Falcosidekick as SLACK_WEBHOOKURL. Falcosidekick sends only WARNING and higher priority events to Slack through SLACK_MINIMUMPRIORITY. Lower-priority events remain available in Falco logs.

Verify only the Secret name and key, never the value:

kubectl get secret falco-alerting -n falco \
  -o jsonpath='{.metadata.name}{" keys: "}{range $key, $value := .data}{$key}{" "}{end}{"\n"}'

Avoid printing Secret data in shared terminals or CI logs. To rotate the webhook, update the Vault value and rerun Ansible.

Falco inotify behavior

The Falco configuration disables watch_config_files. Nested k3d nodes share one Flatcar kernel, and Falco 0.44.1 can fail during startup while initializing its inotify watcher on one node. GitOps changes are applied through resource reconciliation and pod rollout instead of Falco hot reload. This is a reliability trade-off for the nested local profile and should be revisited when testing on separate production-style VMs.

The upstream Falco documentation describes configuration file watching and the manual reload behavior when watching is disabled: https://falco.org/docs/setup/packages/

For runtime alert interpretation, see docs/runtime-detection.md. The local k3d nodes share one Flatcar host kernel, so duplicate Falco notifications are possible. Separate VM nodes provide distinct kernels and are the more realistic production-style validation path.