kubernetes-security-baseline-lab/TODO.md

# Automation backlog

This project should be reproducible with as few manual cluster operations as
possible. Secrets must be supplied securely at runtime and must never be
committed to Git, stored in Terraform state, or printed in CI logs.

## Priority 1: Ansible-driven bootstrap

Implementation is now present in `ansible/bootstrap-flatcar-k3d.yml` with
Flatcar no-Python compatibility. The standard path was run successfully on the
dedicated Flatcar k3d host. The optional Cilium path has syntax validation but
still needs a clean kube-proxy-free cluster validation.

- [ ] Add an Ansible role for post-provision Flatcar bootstrap.
  - [ ] Install or configure the Docker, k3d, kubectl, Helm, and Git tools
    required by the selected host profile.
  - [ ] Create the workspace, directories, permissions, and systemd units.
  - [ ] Create the k3d cluster from the selected cluster configuration.
  - [ ] Export and install the kubeconfig for the operator workstation or
    configured administrative user.
  - [ ] Verify node readiness, the Kubernetes API, and required CPU features
    for Falco and modern eBPF workloads.
  - [ ] Make the role idempotent and safe to rerun after a partial bootstrap.

- [ ] Add an Ansible role for Kubernetes platform bootstrap.
  - [ ] Install ArgoCD from the pinned official manifest or a pinned chart.
  - [ ] Apply the GitOps root Application from `deployments/argocd/`.
  - [ ] Wait for ArgoCD Applications to become synced and healthy.
  - [ ] Remove the current manual root Application migration step.
  - [ ] Add retries for private Gitea repository access and ArgoCD refresh.

- [ ] Add an Ansible role for optional Cilium installation.
  - [ ] Accept the API server address and port as variables.
  - [ ] Validate that kube-proxy is disabled before enabling replacement mode.
  - [ ] Install Cilium with the values under `deployments/cilium/`.
  - [ ] Wait for Cilium agents and the kube-proxy replacement health check.

## Priority 2: Secure optional secret automation

- [ ] Add a boolean variable such as `falco_alerting_enabled: false`.
  - [ ] Keep the default disabled so a normal bootstrap does not require a
    webhook credential.
  - [ ] When disabled, do not create the Secret, Falcosidekick, or Slack
    forwarding configuration.
  - [ ] When enabled, validate that exactly one supported alert destination is
    configured.

- [ ] Add secure runtime input for the Slack webhook value.
  - [ ] Support an Ansible Vault variable such as
    `vault_falco_slack_webhook_url`.
  - [ ] Support an interactive prompt when no Vault value is supplied.
  - [ ] Mark all secret-handling tasks with `no_log: true`.
  - [ ] Create the `falco-alerting` Secret in the `falco` namespace with the
    `webhook-url` key.
  - [ ] Never render the Secret into a repository file or Terraform state.
  - [ ] Never pass the webhook value through a command line argument.
  - [ ] Restart or reconcile Falcosidekick after rotation.
  - [ ] Verify only the Secret name and key, never the Secret value.

- [ ] Add the same toggle and secure input pattern for Discord.
  - [ ] Use a separate destination variable and Secret key if Discord support
    is retained.
  - [ ] Document Slack and Discord as mutually exclusive defaults, with an
    explicit option for both if the implementation supports both safely.

## Priority 3: Remove remaining manual operations

- [ ] Automate installation of the ArgoCD root Application after the repository
  URL and branch are configured.
- [ ] Automate ArgoCD hard refreshes only when required by a repository cache or
  private repository reconciliation delay.
- [ ] Add an Ansible task to verify that the live root Application points to
  `deployments/argocd/apps`.
- [ ] Add an Ansible task to verify that the Falco child Application points to
  `deployments/falco/operator-resources` and the moved values file.
- [ ] Automate creation of the dedicated `security-baseline` namespace labels
  and other prerequisites used by Kyverno policy matching.
- [ ] Automate creation of the test workload namespace and GitOps health checks.
- [ ] Automate the blocked admission tests and runtime detection test as an
  Ansible verification playbook.
- [ ] Add a cleanup playbook for disposable test workloads and ephemeral k3d
  clusters.
- [ ] Replace direct `kubectl` instructions in operational documentation with
  idempotent Ansible tasks where practical.

## Priority 4: Flatcar provisioning automation

- [ ] Refactor the Proxmox Flatcar VM scripts so Ignition configuration is
  generated from explicit variables rather than embedded one-off values.
- [ ] Add an Ansible inventory example for the Flatcar runner and k3d test host.
- [ ] Add a first-boot Ansible handoff that waits for SSH and verifies the
  expected Flatcar user, hostname, CPU type, and storage paths.
- [ ] Automate Docker and container runtime configuration for Flatcar.
- [ ] Automate installation and version pinning for k3d, kubectl, and Helm.
- [ ] Automate the k3d cluster lifecycle with a profile variable for the
  standard and Cilium kube-proxy-free paths.
- [ ] Automate kubeconfig retrieval, context naming, merge, and validation.
- [ ] Add idempotent teardown for a named test VM and its k3d cluster.
- [ ] Document which Proxmox actions remain outside Ansible, such as selecting
  storage, VM ID, bridge, and Flatcar image source.

## Priority 4a: Flatcar lifecycle management guide

- [ ] Write a dedicated Flatcar lifecycle guide under `docs/`.
  - [ ] Explain Flatcar release channels and how to select a channel for test,
    staging, and production hosts.
  - [ ] Document how to pin and record the Flatcar image or release version.
  - [ ] Document Ignition as first-boot configuration and explain which host
    changes belong in Ignition versus Ansible.
  - [ ] Document update-engine behavior, update deferral, maintenance windows,
    and controlled reboots.
  - [ ] Add pre-update health checks for Kubernetes nodes, Falco, Cilium,
    container runtime access, storage, and kubeconfig connectivity.
  - [ ] Add a drain and uncordon workflow for Kubernetes nodes before and after
    a reboot.
  - [ ] Document how to update Proxmox Flatcar images without overwriting the
    VM disk before the replacement image is validated.
  - [ ] Document cloud image or AMI replacement and instance rotation for the
    Terraform path.
  - [ ] Define rollback procedures for failed updates, including the previous
    Flatcar image, VM snapshot policy, and Kubernetes workload recovery.
  - [ ] Define recovery steps for failed Ignition, failed boot, lost SSH, and
    a node that does not return to `Ready`.
  - [ ] Document backup and recovery requirements for kubeconfig, GitOps state,
    Ansible Vault data, and any locally stored cluster credentials.
  - [ ] Add a lifecycle checklist suitable for interviews and portfolio review.

## Priority 4b: Proxmox Terraform infrastructure path

- [ ] Add `terraform/proxmox/` using the `bpg/proxmox` provider.
  - [ ] Pin a reviewed provider version and document the Proxmox VE version
    compatibility assumptions.
  - [ ] Use `proxmox_virtual_environment_vm`, not the provider's experimental
    `proxmox_vm` resource.
  - [ ] Define provider endpoint, API token, TLS behavior, and SSH agent access
    through variables or environment variables only.
  - [ ] Download or reference the pinned Flatcar Proxmox image with checksum
    verification.
  - [ ] Upload an Ignition user-data snippet to Proxmox storage with the
    `snippets` content type enabled.
  - [ ] Provision the Flatcar k3d test host with configurable VM ID, node,
    storage, bridge, CPU, memory, and disk size.
  - [ ] Set CPU passthrough so Falco and modern eBPF workloads receive the
    required instruction set.
  - [ ] Keep the Gitea runner VM optional and disabled by default.
  - [ ] Keep all runner registration tokens and webhook values outside
    Terraform configuration and state.
  - [ ] Emit Ansible inventory data or outputs for the created host addresses.
  - [ ] Add a documented `terraform apply` followed by the Ansible bootstrap.
  - [ ] Add safe destroy, import, and image rotation guidance.
  - [ ] Document the required Proxmox storage content types, API permissions,
    SSH access, and node prerequisites.

## Priority 4c: Terraform to Ansible handoff

The Proxmox Terraform path should hand off to the existing Ansible bootstrap
automatically after the VM is reachable. Terraform owns infrastructure
creation. Ansible owns host configuration, k3d or K3s bootstrap, ArgoCD
bootstrap, and verification. GitOps remains the owner of ongoing Kubernetes
application configuration.

- [ ] Pin and configure the `ansible/ansible` provider.
  - [ ] Use `ansible_host` with the Terraform-created VM address and the
    `flatcar_k3d` inventory group.
  - [ ] Use `ansible_playbook` to run the existing
    `ansible/bootstrap-flatcar-k3d.yml` playbook after VM creation.
  - [ ] Add an explicit dependency on the Proxmox VM and its initial network
    configuration so the playbook cannot start too early.
  - [ ] Wait for SSH and verify the expected Flatcar user before the main
    bootstrap tasks run.
  - [ ] Pass only nonsecret settings through Terraform `extra_vars`, such as
    the cluster name, repository URL, branch, and feature toggles.
  - [ ] Keep webhook values, runner tokens, private keys, and Vault passwords
    out of Terraform configuration, `extra_vars`, and Terraform state.
  - [ ] Use SSH agent forwarding or a protected key path supplied through the
    runtime environment rather than committing credentials.
  - [ ] Prevent every Terraform plan from rerunning the full bootstrap. Use a
    VM generation, Ignition hash, or explicit bootstrap trigger.
  - [ ] Make a failed bootstrap easy to retry without recreating the VM.
  - [ ] Capture useful outputs such as the VM address, Ansible inventory group,
    kubeconfig artifact path, and verified Kubernetes context.

- [ ] Test the handoff lifecycle.
  - [ ] Apply from an empty Proxmox project and verify the complete handoff.
  - [ ] Reapply without changes and verify no unnecessary VM replacement or
    full Ansible rerun occurs.
  - [ ] Replace the VM and verify the new Ignition and bootstrap sequence.
  - [ ] Interrupt Ansible, retry Terraform, and verify safe convergence.
  - [ ] Destroy the VM and verify that no credentials or stale inventory data
    remain in the repository.

The provider's [`ansible_host` resource](https://registry.terraform.io/providers/ansible/ansible/latest/docs/resources/host)
creates inventory hosts from Terraform values. The
[`ansible_playbook` resource](https://registry.terraform.io/providers/ansible/ansible/latest/docs/resources/playbook)
then provides the Terraform-to-Ansible execution boundary. This integration
will be documented as an optional convenience for the Proxmox path, because it
also means Terraform runs can be longer and Ansible execution metadata may be
stored in state depending on provider configuration.

## Priority 5: Terraform and cloud path

- [ ] Pass Terraform outputs into Ansible inventory generation.
- [ ] Keep cloud credentials, AMI IDs, SSH keys, and network ranges outside
  committed files.
- [ ] Add a documented command sequence for Terraform apply followed by the
  Ansible bootstrap playbook.
- [ ] Add a documented destroy path and state storage guidance.
- [ ] Add validation that the cloud host exposes the CPU features required by
  Falco and modern eBPF.

## Priority 6: Documentation and portfolio evidence

- [ ] Update the root README after each automation milestone.
- [ ] Document the secure secret input flow with an example Vault variable and
  an interactive prompt example that contains no real credential.
- [ ] Add an architecture diagram showing Terraform or Proxmox, Ansible,
  Flatcar, k3d or K3s, ArgoCD, Kyverno, Falco, Falcosidekick, and Slack.
- [ ] Capture evidence for blocked admission, compliant execution, Falco
  detection, and Slack delivery.
- [ ] Add CI checks for YAML, Kustomize rendering, Ansible syntax, and secret
  scanning.
- [ ] Add a final runbook that starts from a clean Flatcar VM and ends with a
  verified GitOps-managed security baseline.

## Interactive inputs that should remain explicit

These values require user intent, but their handling should be automated:

- Git repository URL and branch
- Proxmox VM ID, storage, bridge, and resource sizing
- SSH public key
- Cloud credentials and cloud-specific network choices
- Slack or Discord webhook creation
- Secret value supplied through Ansible Vault or a hidden prompt

The automation should validate these inputs, pass them securely, and avoid
persisting sensitive values outside the intended secret store.