make falco alerting opt in

deployments/argocd/optional-apps/falco-alerting.yaml

@@ -0,0 +1,23 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: falco-alerting
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "2"
+spec:
+  project: default
+  source:
+    repoURL: https://git.swaphb.com/swaphb/kubernetes-security-baseline-lab.git
+    targetRevision: main
+    path: deployments/falco/alerting-resources
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: falco
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+      - ServerSideApply=true

deployments/falco/README.md

@@ -1,10 +1,17 @@
-# Falco alerting secret
+# Falco alerting and runtime troubleshooting
+
+The default Falco deployment runs Falco and its custom rules without
+Falcosidekick. Slack forwarding is opt-in because a webhook is a credential
+and should not be required for local cluster bootstrap.
+
+## Optional Slack alerting
 
 Falco alert delivery must use a Kubernetes Secret. Do not commit a Slack
 Incoming Webhook URL, Slack token, Discord webhook URL, or generated Secret
 manifest to this repository.
 
-The expected Secret contract for the Git-managed Falcosidekick configuration is:
+When alerting is enabled, the Ansible role creates this Secret contract. The
+Secret is never stored in Git:
 
 ```yaml
 apiVersion: v1
@@ -17,23 +24,34 @@ stringData:
   webhook-url: https://hooks.slack.com/services/REPLACE/ME
 ```
 
-The example value is a placeholder. Create the Secret directly on the cluster
-with the real Slack Incoming Webhook URL:
+Store the real value in an Ansible Vault file:
 
-```bash
-kubectl create namespace falco --dry-run=client -o yaml | kubectl apply -f -
-kubectl create secret generic falco-alerting \
-  --namespace falco \
-  --from-literal=webhook-url='https://hooks.slack.com/services/REPLACE/ME' \
-  --dry-run=client -o yaml | kubectl apply -f -
+```yaml
+falco_alerting_enabled: true
+falco_slack_webhook_url: https://hooks.slack.com/services/REPLACE/ME
 ```
 
-The `webhook-url` key is injected into Falcosidekick as `SLACK_WEBHOOKURL`.
-Falcosidekick sends only `WARNING` and higher priority events to Slack through
-`SLACK_MINIMUMPRIORITY`. Lower-priority events remain available in Falco logs.
-The Secret is intentionally not included in the Falco
-Kustomization because ArgoCD must not manage or render the credential from
-Git.
+Run the bootstrap with the encrypted file:
+
+```bash
+ansible-vault encrypt ansible/vars/falco-alerting.vault.yml
+ansible-playbook \
+  -i ansible/inventory.flatcar-k3d.yml \
+  ansible/bootstrap-flatcar-k3d.yml \
+  --ask-vault-pass \
+  -e @ansible/vars/falco-alerting.vault.yml
+```
+
+The Ansible role creates the Secret without logging its value and applies the
+optional `falco-alerting` ArgoCD Application only when both the toggle and
+webhook value are supplied. When disabled, it removes the optional Application,
+Component, output Config, and Secret. The default Falco Application does not
+include Falcosidekick resources.
+
+The `webhook-url` key is injected into Falcosidekick as
+`SLACK_WEBHOOKURL`. Falcosidekick sends only `WARNING` and higher priority
+events to Slack through `SLACK_MINIMUMPRIORITY`. Lower-priority events remain
+available in Falco logs.
 
 Verify only the Secret name and key, never the value:
 
@@ -43,12 +61,20 @@ kubectl get secret falco-alerting -n falco \
 ```
 
 Avoid printing Secret data in shared terminals or CI logs. To rotate the
-webhook, update the Secret in place with the creation command and restart the
-Falco workloads after the alerting integration is configured.
+webhook, update the Vault value and rerun Ansible.
 
-The repository verifies Falco runtime detections locally. Webhook delivery is
-enabled by the Git-managed Falcosidekick resources after the Secret exists, and
-still requires an interactive Slack or Discord credential for end-to-end testing.
+## Falco inotify behavior
+
+The Falco configuration disables `watch_config_files`. Nested k3d nodes share
+one Flatcar kernel, and Falco 0.44.1 can fail during startup while initializing
+its inotify watcher on one node. GitOps changes are applied through resource
+reconciliation and pod rollout instead of Falco hot reload. This is a
+reliability trade-off for the nested local profile and should be revisited when
+testing on separate production-style VMs.
+
+The upstream Falco documentation describes configuration file watching and the
+manual reload behavior when watching is disabled:
+<https://falco.org/docs/setup/packages/>
 
 For runtime alert interpretation, see
 [`docs/runtime-detection.md`](../../docs/runtime-detection.md). The local k3d

deployments/falco/alerting-resources/falcosidekick-component.yaml

@@ -4,7 +4,7 @@ metadata:
   name: falcosidekick
   namespace: falco
   annotations:
-    # Deploy the event forwarder after the Falco instance and its CRDs exist.
+    # Deploy the event forwarder only when the alerting Secret is enabled.
     argocd.argoproj.io/sync-wave: "2"
 spec:
   component:
@@ -20,7 +20,6 @@ spec:
                 secretKeyRef:
                   name: falco-alerting
                   key: webhook-url
-            # Keep low-priority Falco events in cluster logs, but do not send
-            # NOTICE-level activity to the Slack alert channel.
+            # Keep low-priority events in Falco logs instead of Slack.
             - name: SLACK_MINIMUMPRIORITY
               value: warning

deployments/falco/alerting-resources/kustomization.yaml

@@ -0,0 +1,6 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: falco
+resources:
+  - falcosidekick-component.yaml
+  - falcosidekick-config.yaml

deployments/falco/operator-resources/falco-config.yaml

@@ -10,3 +10,6 @@ spec:
   config:
     json_output: true
     json_include_output_property: true
+    # Avoid the Falco inotify watcher in nested k3d nodes. Configuration and
+    # rule changes are applied through GitOps reconciliation and pod rollout.
+    watch_config_files: false

deployments/falco/operator-resources/kustomization.yaml

@@ -7,8 +7,6 @@ resources:
   - falco-config.yaml
   - container-plugin.yaml
   - custom-rulesfile.yaml
-  - falcosidekick-component.yaml
-  - falcosidekick-config.yaml
 
 configMapGenerator:
   - name: falco-custom-rules