kubernetes-security-baseline-lab/docs/runtime-detection.md

Runtime detection topology

Falco observes kernel activity. The topology used for local validation affects how many Falco instances can observe the same event.

Local k3d validation

The local test environment runs three k3d node containers on one Flatcar VM:

Flatcar host kernel
├── k3d server container with Falco
├── k3d agent container with Falco
└── k3d agent container with Falco

k3d nodes are containers, not independent virtual machines. They share the Flatcar host kernel. Because each Falco DaemonSet Pod observes kernel events, multiple Falco Pods can report the same underlying process or syscall event. This explains duplicate Slack notifications during local demonstrations.

This behavior is a limitation of the test topology, not evidence that Falco is detecting separate workload executions. The local cluster still demonstrates the admission-to-runtime security flow and verifies that the custom rule fires.

Production-style validation

The production-style path uses separate Flatcar or cloud VMs for cluster nodes:

VM 1 kernel -> Falco -> workloads on VM 1
VM 2 kernel -> Falco -> workloads on VM 2
VM 3 kernel -> Falco -> workloads on VM 3

Each VM has its own kernel. A single process event should normally be observed by the Falco instance on the node hosting that workload. This removes the shared-kernel duplication caused by the local k3d layout.

Production deployments can still produce legitimate repeated alerts when:

• A workload is replicated across multiple nodes.
• Events are forwarded through more than one collector.
• Multiple Falco event sources describe the same activity.
• The central alerting system does not deduplicate equivalent events.

For that reason, production alerting should use priority filtering, event aggregation, and an explicit deduplication strategy. This project currently filters Slack notifications to WARNING and higher while retaining lower priority events in Falco logs.

Portfolio interpretation

The local k3d environment is optimized for reproducibility and cost. The separate-VM path is the more realistic validation environment for kernel-level runtime detection. Results from the two paths should be labeled accordingly when presenting screenshots, alert counts, or demo recordings.