84 lines
3.0 KiB
Markdown
84 lines
3.0 KiB
Markdown
# Falco alerting and runtime troubleshooting
|
|
|
|
The default Falco deployment runs Falco and its custom rules without
|
|
Falcosidekick. Slack forwarding is opt-in because a webhook is a credential
|
|
and should not be required for local cluster bootstrap.
|
|
|
|
## Optional Slack alerting
|
|
|
|
Falco alert delivery must use a Kubernetes Secret. Do not commit a Slack
|
|
Incoming Webhook URL, Slack token, Discord webhook URL, or generated Secret
|
|
manifest to this repository.
|
|
|
|
When alerting is enabled, the Ansible role creates this Secret contract. The
|
|
Secret is never stored in Git:
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: falco-alerting
|
|
namespace: falco
|
|
type: Opaque
|
|
stringData:
|
|
webhook-url: https://hooks.slack.com/services/REPLACE/ME
|
|
```
|
|
|
|
Store the real value in an Ansible Vault file:
|
|
|
|
```yaml
|
|
falco_alerting_enabled: true
|
|
falco_slack_webhook_url: https://hooks.slack.com/services/REPLACE/ME
|
|
```
|
|
|
|
Run the bootstrap with the encrypted file:
|
|
|
|
```bash
|
|
ansible-vault encrypt ansible/vars/falco-alerting.vault.yml
|
|
ansible-playbook \
|
|
-i ansible/inventory.flatcar-k3d.yml \
|
|
ansible/bootstrap-flatcar-k3d.yml \
|
|
--ask-vault-pass \
|
|
-e @ansible/vars/falco-alerting.vault.yml
|
|
```
|
|
|
|
The Ansible role creates the Secret without logging its value and applies the
|
|
optional `falco-alerting` ArgoCD Application only when both the toggle and
|
|
webhook value are supplied. When disabled, it removes the optional Application,
|
|
Component, output Config, and Secret. The default Falco Application does not
|
|
include Falcosidekick resources.
|
|
|
|
The `webhook-url` key is injected into Falcosidekick as
|
|
`SLACK_WEBHOOKURL`. Falcosidekick sends only `WARNING` and higher priority
|
|
events to Slack through `SLACK_MINIMUMPRIORITY`. Lower-priority events remain
|
|
available in Falco logs.
|
|
|
|
Verify only the Secret name and key, never the value:
|
|
|
|
```bash
|
|
kubectl get secret falco-alerting -n falco \
|
|
-o jsonpath='{.metadata.name}{" keys: "}{range $key, $value := .data}{$key}{" "}{end}{"\n"}'
|
|
```
|
|
|
|
Avoid printing Secret data in shared terminals or CI logs. To rotate the
|
|
webhook, update the Vault value and rerun Ansible.
|
|
|
|
## Falco inotify behavior
|
|
|
|
The Falco configuration disables `watch_config_files`. Nested k3d nodes share
|
|
one Flatcar kernel, and Falco 0.44.1 can fail during startup while initializing
|
|
its inotify watcher on one node. GitOps changes are applied through resource
|
|
reconciliation and pod rollout instead of Falco hot reload. This is a
|
|
reliability trade-off for the nested local profile and should be revisited when
|
|
testing on separate production-style VMs.
|
|
|
|
The upstream Falco documentation describes configuration file watching and the
|
|
manual reload behavior when watching is disabled:
|
|
<https://falco.org/docs/setup/packages/>
|
|
|
|
For runtime alert interpretation, see
|
|
[`docs/runtime-detection.md`](../../docs/runtime-detection.md). The local k3d
|
|
nodes share one Flatcar host kernel, so duplicate Falco notifications are
|
|
possible. Separate VM nodes provide distinct kernels and are the more realistic
|
|
production-style validation path.
|