kubernetes-security-baseline-lab/README.md

Kubernetes Security Baseline

An auditable, GitOps-managed Kubernetes security baseline: local k3d for a fast reproducible demo, optional Flatcar plus Terraform and Ansible for a cloud K3s host, Cilium eBPF networking, CIS Benchmark checks in CI, Kyverno admission policies, and Falco runtime detection.

Repository layout

Deployment and GitOps configuration is grouped under deployments/:

deployments/
├── argocd/       # app-of-apps and child Application definitions
├── cilium/       # Cilium Helm values for the optional kube-proxy-free path
└── falco/        # Falco Operator values and custom resource definitions

The platform payloads remain separate from their deployment definitions. Kyverno policies live under policies/, runtime demonstration workloads live under test-workloads/, and cluster provisioning code remains under terraform/, ansible/, and local-quickstart/.

The planned automation work is tracked in TODO.md, including secure Ansible secret input and Flatcar bootstrap automation.

Phase 1: provision a local cluster

The primary quick-start path requires Docker, k3d, and kubectl:

k3d cluster create --config local-quickstart/k3d-cluster-config.yaml
kubectl get nodes -o wide

Expected result: one server and two agent nodes in Ready state. The bundled Traefik is disabled so later ArgoCD-managed components own add-ons explicitly.

Delete the lab with k3d cluster delete security-baseline.

For repeatable CI and runtime testing, the repository also includes a Flatcar Linux Proxmox host definition under scripts/proxmox/. The current test environment runs k3d on a dedicated Flatcar VM with kernel access for Falco.

Production-style option

terraform/cloud-cluster/ and ansible/bootstrap-k3s.yml provide an optional, documented AWS path. It uses Flatcar as the host OS, disables Flannel and kube-proxy, and installs Cilium as the CNI and eBPF service datapath. It requires your own AWS credentials, an existing EC2 key pair, a Flatcar AMI ID for the chosen region, and a tightly scoped admin CIDR. It is not needed for the portfolio demo and is not run in CI.

The Proxmox scripts also create Flatcar-based hosts for a self-hosted Gitea Actions runner and for the dedicated k3d test environment. They use Ignition for first-boot configuration and do not contain account-specific credentials.

GitOps and security validation

ArgoCD is bootstrapped once and then manages the platform applications through the app-of-apps definition in deployments/argocd/. The managed components are Kyverno, Falco, and the security demonstration workloads. Cilium values and an optional Cilium Application are also stored under deployments/ for the kube-proxy-free profile.

The kube-bench workflow is retained in .github/workflows/ for GitHub Actions and uses a Gitea-compatible artifact action when run by Gitea Actions. It creates an ephemeral k3d cluster, waits for the Kubernetes API, runs the K3s benchmark, and uploads the report without failing the workflow solely because the benchmark contains findings that require review.

Kyverno admission tests have verified that privileged containers, root containers, host networking or host PID access, unapproved registries, and application workloads in the default namespace are rejected. Falco runtime testing has verified that the suspicious shell workload produces the custom shell detection while the compliant workload does not.

The local k3d cluster runs its node containers on one Flatcar host kernel, so multiple Falco Pods can observe the same kernel event. This can create duplicate notifications during local demonstrations. The production-style separate-VM path gives each node its own kernel. See docs/runtime-detection.md for the topology distinction and alerting implications.

Build status

• Phase 1: local k3d definition and optional Terraform/Ansible path
• Phase 2: ArgoCD app-of-apps deployment and remote Flatcar test environment
• Phase 3: kube-bench CI scan
• Architecture revision: Flatcar hosts and Falco kernel access
• Phase 4: Kyverno policy set, CIS mapping, and admission tests
• Phase 5: Falco rules and webhook alerting (runtime detection verified; webhook pending)
• Phase 6: test workloads and runtime verification
• Phase 7: architecture and design documentation (repository layout started)
• Phase 8: final portfolio polish

Remaining work

• Configure a Slack or Discord webhook through a Kubernetes Secret outside Git.
• Add the architecture diagram, CIS evidence summary, and demo capture.
• Finish the design trade-off documentation and final repository review.