diff --git a/README.md b/README.md index a1ed7b3..59a35c8 100644 --- a/README.md +++ b/README.md @@ -75,6 +75,12 @@ application workloads in the `default` namespace are rejected. Falco runtime testing has verified that the suspicious shell workload produces the custom shell detection while the compliant workload does not. +The local k3d cluster runs its node containers on one Flatcar host kernel, so +multiple Falco Pods can observe the same kernel event. This can create duplicate +notifications during local demonstrations. The production-style separate-VM +path gives each node its own kernel. See [`docs/runtime-detection.md`](docs/runtime-detection.md) +for the topology distinction and alerting implications. + ## Build status - [x] Phase 1: local k3d definition and optional Terraform/Ansible path diff --git a/deployments/falco/README.md b/deployments/falco/README.md index dcebe6e..89f7d12 100644 --- a/deployments/falco/README.md +++ b/deployments/falco/README.md @@ -49,3 +49,9 @@ Falco workloads after the alerting integration is configured. The repository verifies Falco runtime detections locally. Webhook delivery is enabled by the Git-managed Falcosidekick resources after the Secret exists, and still requires an interactive Slack or Discord credential for end-to-end testing. + +For runtime alert interpretation, see +[`docs/runtime-detection.md`](../../docs/runtime-detection.md). The local k3d +nodes share one Flatcar host kernel, so duplicate Falco notifications are +possible. Separate VM nodes provide distinct kernels and are the more realistic +production-style validation path. diff --git a/docs/runtime-detection.md b/docs/runtime-detection.md new file mode 100644 index 0000000..22333e7 --- /dev/null +++ b/docs/runtime-detection.md @@ -0,0 +1,58 @@ +# Runtime detection topology + +Falco observes kernel activity. The topology used for local validation affects +how many Falco instances can observe the same event. + +## Local k3d validation + +The local test environment runs three k3d node containers on one Flatcar VM: + +```text +Flatcar host kernel +├── k3d server container with Falco +├── k3d agent container with Falco +└── k3d agent container with Falco +``` + +k3d nodes are containers, not independent virtual machines. They share the +Flatcar host kernel. Because each Falco DaemonSet Pod observes kernel events, +multiple Falco Pods can report the same underlying process or syscall event. +This explains duplicate Slack notifications during local demonstrations. + +This behavior is a limitation of the test topology, not evidence that Falco is +detecting separate workload executions. The local cluster still demonstrates +the admission-to-runtime security flow and verifies that the custom rule fires. + +## Production-style validation + +The production-style path uses separate Flatcar or cloud VMs for cluster +nodes: + +```text +VM 1 kernel -> Falco -> workloads on VM 1 +VM 2 kernel -> Falco -> workloads on VM 2 +VM 3 kernel -> Falco -> workloads on VM 3 +``` + +Each VM has its own kernel. A single process event should normally be observed +by the Falco instance on the node hosting that workload. This removes the +shared-kernel duplication caused by the local k3d layout. + +Production deployments can still produce legitimate repeated alerts when: + +- A workload is replicated across multiple nodes. +- Events are forwarded through more than one collector. +- Multiple Falco event sources describe the same activity. +- The central alerting system does not deduplicate equivalent events. + +For that reason, production alerting should use priority filtering, event +aggregation, and an explicit deduplication strategy. This project currently +filters Slack notifications to `WARNING` and higher while retaining lower +priority events in Falco logs. + +## Portfolio interpretation + +The local k3d environment is optimized for reproducibility and cost. The +separate-VM path is the more realistic validation environment for kernel-level +runtime detection. Results from the two paths should be labeled accordingly +when presenting screenshots, alert counts, or demo recordings.