kubernetes-security-baseline-lab/TODO.md

# Automation backlog

This project should be reproducible with as few manual cluster operations as
possible. Secrets must be supplied securely at runtime and must never be
committed to Git, stored in Terraform state, or printed in CI logs.

## Priority 1: Ansible-driven bootstrap

Implementation is now present in `ansible/bootstrap-flatcar-k3d.yml` with
Flatcar no-Python compatibility. The standard path was run successfully on the
dedicated Flatcar k3d host. The optional Cilium path has syntax validation but
still needs a clean kube-proxy-free cluster validation.

- [ ] Add an Ansible role for post-provision Flatcar bootstrap.
  - [ ] Install or configure the Docker, k3d, kubectl, Helm, and Git tools
    required by the selected host profile.
  - [ ] Create the workspace, directories, permissions, and systemd units.
  - [ ] Create the k3d cluster from the selected cluster configuration.
  - [ ] Export and install the kubeconfig for the operator workstation or
    configured administrative user.
  - [ ] Verify node readiness, the Kubernetes API, and required CPU features
    for Falco and modern eBPF workloads.
  - [ ] Make the role idempotent and safe to rerun after a partial bootstrap.

- [ ] Add an Ansible role for Kubernetes platform bootstrap.
  - [ ] Install ArgoCD from the pinned official manifest or a pinned chart.
  - [ ] Apply the GitOps root Application from `deployments/argocd/`.
  - [ ] Wait for ArgoCD Applications to become synced and healthy.
  - [ ] Remove the current manual root Application migration step.
  - [ ] Add retries for private Gitea repository access and ArgoCD refresh.

- [ ] Add an Ansible role for optional Cilium installation.
  - [ ] Accept the API server address and port as variables.
  - [ ] Validate that kube-proxy is disabled before enabling replacement mode.
  - [ ] Install Cilium with the values under `deployments/cilium/`.
  - [ ] Wait for Cilium agents and the kube-proxy replacement health check.

## Priority 2: Secure optional secret automation

- [ ] Add a boolean variable such as `falco_alerting_enabled: false`.
  - [ ] Keep the default disabled so a normal bootstrap does not require a
    webhook credential.
  - [ ] When disabled, do not create the Secret, Falcosidekick, or Slack
    forwarding configuration.
  - [ ] When enabled, validate that exactly one supported alert destination is
    configured.

- [ ] Add secure runtime input for the Slack webhook value.
  - [ ] Support an Ansible Vault variable such as
    `vault_falco_slack_webhook_url`.
  - [ ] Support an interactive prompt when no Vault value is supplied.
  - [ ] Mark all secret-handling tasks with `no_log: true`.
  - [ ] Create the `falco-alerting` Secret in the `falco` namespace with the
    `webhook-url` key.
  - [ ] Never render the Secret into a repository file or Terraform state.
  - [ ] Never pass the webhook value through a command line argument.
  - [ ] Restart or reconcile Falcosidekick after rotation.
  - [ ] Verify only the Secret name and key, never the Secret value.

- [ ] Add the same toggle and secure input pattern for Discord.
  - [ ] Use a separate destination variable and Secret key if Discord support
    is retained.
  - [ ] Document Slack and Discord as mutually exclusive defaults, with an
    explicit option for both if the implementation supports both safely.

## Priority 3: Remove remaining manual operations

- [ ] Automate installation of the ArgoCD root Application after the repository
  URL and branch are configured.
- [ ] Automate ArgoCD hard refreshes only when required by a repository cache or
  private repository reconciliation delay.
- [ ] Add an Ansible task to verify that the live root Application points to
  `deployments/argocd/apps`.
- [ ] Add an Ansible task to verify that the Falco child Application points to
  `deployments/falco/operator-resources` and the moved values file.
- [ ] Automate creation of the dedicated `security-baseline` namespace labels
  and other prerequisites used by Kyverno policy matching.
- [ ] Automate creation of the test workload namespace and GitOps health checks.
- [ ] Automate the blocked admission tests and runtime detection test as an
  Ansible verification playbook.
- [ ] Add a cleanup playbook for disposable test workloads and ephemeral k3d
  clusters.
- [ ] Replace direct `kubectl` instructions in operational documentation with
  idempotent Ansible tasks where practical.

## Priority 4: Flatcar provisioning automation

- [ ] Refactor the Proxmox Flatcar VM scripts so Ignition configuration is
  generated from explicit variables rather than embedded one-off values.
- [ ] Add an Ansible inventory example for the Flatcar runner and k3d test host.
- [ ] Add a first-boot Ansible handoff that waits for SSH and verifies the
  expected Flatcar user, hostname, CPU type, and storage paths.
- [ ] Automate Docker and container runtime configuration for Flatcar.
- [ ] Automate installation and version pinning for k3d, kubectl, and Helm.
- [ ] Automate the k3d cluster lifecycle with a profile variable for the
  standard and Cilium kube-proxy-free paths.
- [ ] Automate kubeconfig retrieval, context naming, merge, and validation.
- [ ] Add idempotent teardown for a named test VM and its k3d cluster.
- [ ] Document which Proxmox actions remain outside Ansible, such as selecting
  storage, VM ID, bridge, and Flatcar image source.

## Priority 4a: Flatcar lifecycle management guide

- [ ] Write a dedicated Flatcar lifecycle guide under `docs/`.
  - [ ] Explain Flatcar release channels and how to select a channel for test,
    staging, and production hosts.
  - [ ] Document how to pin and record the Flatcar image or release version.
  - [ ] Document Ignition as first-boot configuration and explain which host
    changes belong in Ignition versus Ansible.
  - [ ] Document update-engine behavior, update deferral, maintenance windows,
    and controlled reboots.
  - [ ] Add pre-update health checks for Kubernetes nodes, Falco, Cilium,
    container runtime access, storage, and kubeconfig connectivity.
  - [ ] Add a drain and uncordon workflow for Kubernetes nodes before and after
    a reboot.
  - [ ] Document how to update Proxmox Flatcar images without overwriting the
    VM disk before the replacement image is validated.
  - [ ] Document cloud image or AMI replacement and instance rotation for the
    Terraform path.
  - [ ] Define rollback procedures for failed updates, including the previous
    Flatcar image, VM snapshot policy, and Kubernetes workload recovery.
  - [ ] Define recovery steps for failed Ignition, failed boot, lost SSH, and
    a node that does not return to `Ready`.
  - [ ] Document backup and recovery requirements for kubeconfig, GitOps state,
    Ansible Vault data, and any locally stored cluster credentials.
  - [ ] Add a lifecycle checklist suitable for interviews and portfolio review.

## Priority 5: Terraform and cloud path

- [ ] Pass Terraform outputs into Ansible inventory generation.
- [ ] Keep cloud credentials, AMI IDs, SSH keys, and network ranges outside
  committed files.
- [ ] Add a documented command sequence for Terraform apply followed by the
  Ansible bootstrap playbook.
- [ ] Add a documented destroy path and state storage guidance.
- [ ] Add validation that the cloud host exposes the CPU features required by
  Falco and modern eBPF.

## Priority 6: Documentation and portfolio evidence

- [ ] Update the root README after each automation milestone.
- [ ] Document the secure secret input flow with an example Vault variable and
  an interactive prompt example that contains no real credential.
- [ ] Add an architecture diagram showing Terraform or Proxmox, Ansible,
  Flatcar, k3d or K3s, ArgoCD, Kyverno, Falco, Falcosidekick, and Slack.
- [ ] Capture evidence for blocked admission, compliant execution, Falco
  detection, and Slack delivery.
- [ ] Add CI checks for YAML, Kustomize rendering, Ansible syntax, and secret
  scanning.
- [ ] Add a final runbook that starts from a clean Flatcar VM and ends with a
  verified GitOps-managed security baseline.

## Interactive inputs that should remain explicit

These values require user intent, but their handling should be automated:

- Git repository URL and branch
- Proxmox VM ID, storage, bridge, and resource sizing
- SSH public key
- Cloud credentials and cloud-specific network choices
- Slack or Discord webhook creation
- Secret value supplied through Ansible Vault or a hidden prompt

The automation should validate these inputs, pass them securely, and avoid
persisting sensitive values outside the intended secret store.