# Falco alerting and runtime troubleshooting

The default Falco deployment runs Falco and its custom rules without
Falcosidekick. Slack forwarding is opt-in because a webhook is a credential
and should not be required for local cluster bootstrap.

## Optional Slack alerting

Falco alert delivery must use a Kubernetes Secret. Do not commit a Slack
Incoming Webhook URL, Slack token, Discord webhook URL, or generated Secret
manifest to this repository.

When alerting is enabled, the Ansible role creates this Secret contract. The
Secret is never stored in Git:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: falco-alerting
  namespace: falco
type: Opaque
stringData:
  webhook-url: https://hooks.slack.com/services/REPLACE/ME
```

Store the real value in an Ansible Vault file:

```yaml
falco_alerting_enabled: true
falco_slack_webhook_url: https://hooks.slack.com/services/REPLACE/ME
```

Run the bootstrap with the encrypted file:

```bash
ansible-vault encrypt ansible/vars/falco-alerting.vault.yml
ansible-playbook \
  -i ansible/inventory.flatcar-k3d.yml \
  ansible/bootstrap-flatcar-k3d.yml \
  --ask-vault-pass \
  -e @ansible/vars/falco-alerting.vault.yml
```

The Ansible role creates the Secret without logging its value and applies the
optional `falco-alerting` ArgoCD Application only when both the toggle and
webhook value are supplied. When disabled, it removes the optional Application,
Component, output Config, and Secret. The default Falco Application does not
include Falcosidekick resources.

The `webhook-url` key is injected into Falcosidekick as
`SLACK_WEBHOOKURL`. Falcosidekick sends only `WARNING` and higher priority
events to Slack through `SLACK_MINIMUMPRIORITY`. Lower-priority events remain
available in Falco logs.

Verify only the Secret name and key, never the value:

```bash
kubectl get secret falco-alerting -n falco \
  -o jsonpath='{.metadata.name}{" keys: "}{range $key, $value := .data}{$key}{" "}{end}{"\n"}'
```

Avoid printing Secret data in shared terminals or CI logs. To rotate the
webhook, update the Vault value and rerun Ansible.

## Falco inotify behavior

The Falco configuration disables `watch_config_files`. Nested k3d nodes share
one Flatcar kernel, and Falco 0.44.1 can fail during startup while initializing
its inotify watcher on one node. GitOps changes are applied through resource
reconciliation and pod rollout instead of Falco hot reload. This is a
reliability trade-off for the nested local profile and should be revisited when
testing on separate production-style VMs.

The upstream Falco documentation describes configuration file watching and the
manual reload behavior when watching is disabled:
<https://falco.org/docs/setup/packages/>

For runtime alert interpretation, see
[`docs/runtime-detection.md`](../../docs/runtime-detection.md). The local k3d
nodes share one Flatcar host kernel, so duplicate Falco notifications are
possible. Separate VM nodes provide distinct kernels and are the more realistic
production-style validation path.