feat: Kubernetes deployment support (Dockerfile, manifests, docs)

[chaosreload]

Summary

Add first-class Kubernetes deployment support for Zeroboot, addressing all items in #9.

EKS validation: All manifests and scripts validated on EKS 1.31 / ap-southeast-1 / c8i.xlarge with nested virtualization.

Note: The serve bind address fix (127.0.0.1 → 0.0.0.0) required for K8s health probes is tracked separately in #14 and should be merged before or alongside this PR.

Changes

Dockerfile

Multi-stage build: Rust 1.86 compiler stage + Ubuntu 22.04 runtime. Firecracker binary bundled at build time. vmlinux and rootfs images are not baked into the image — they are mounted via host storage, keeping the image lean and allowing runtime upgrades without rebuilding.

docker/entrypoint.sh

• Validates /dev/kvm access on startup (fast-fail with clear error message)
• Creates the snapshot template on first boot (~19s), skips if already present
• Supports Python and optional Node.js runtimes via env vars

deploy/k8s/

Reference manifests ready to apply:

• namespace.yaml — dedicated zeroboot namespace
• daemonset.yaml — default deployment (see DaemonSet section below)
• deployment.yaml — alternative for HPA / manual replica control (single-replica, RWO PVC)
• pvc.yaml — 20 Gi gp3 PVC for deployment.yaml
• service.yaml — ClusterIP service (works with both DaemonSet and Deployment)
• hpa.yaml — custom-metric HPA on zeroboot_concurrent_forks (for deployment.yaml)

deploy/eks/

EKS-specific deployment configs:

• eks-cluster-only.yaml — create cluster without node groups (Step 1)
• eks-self-managed-kvm.sh — end-to-end script for self-managed ASG with CpuOptions.NestedVirtualization=enabled; see EKS note below
• eks-with-kvm-nodegroup.yaml — eksctl managed node group config (⚠️ see caveat)
• eks-add-kvm-nodegroup.yaml — add KVM node group to existing cluster

docs/KUBERNETES.md

Comprehensive deployment guide covering:

• EC2 instance selection: c8i/m8i/r8i recommended
• EKS managed vs self-managed node groups — root cause and solution for CpuOptions being silently dropped
• KVM device access (privileged + hostPath, with note on kubevirt alternative)
• DaemonSet vs Deployment trade-offs and when to use each
• Storage: hostPath for DaemonSet, PVC for Deployment
• Autoscaling, Prometheus, configuration reference, known limitations

DaemonSet as Default (updated after reviewer feedback)

Following @minhoryang's suggestion, the default deployment is now DaemonSet instead of Deployment.

Why DaemonSet:

• zeroboot is tightly bound to /dev/kvm and host CPU microarchitecture — the natural unit of scale is a Node, not a Pod replica
• Adding a KVM-capable node should automatically bring up a new zeroboot instance — DaemonSet is the correct primitive for this semantics
• Eliminates the need for podAntiAffinity + replicas: N workarounds
• Integrates cleanly with Karpenter/Cluster Autoscaler for node-level autoscaling

Storage: DaemonSet uses hostPath: /var/lib/zeroboot (DirectoryOrCreate). Firecracker snapshots are bound to the host CPU microarchitecture and KVM state — local storage is the correct choice. Cross-node snapshot migration is not supported and not needed for short-lived sandbox workloads.

Deployment option retained for cases requiring HPA or manual replica control. Documented as an advanced alternative.

⚠️ EKS Node Group: Use Self-Managed

TL;DR: EKS managed node groups silently drop CpuOptions.NestedVirtualization — your nodes start without /dev/kvm even if the eksctl YAML looks correct.

Root cause: When you provide a Launch Template to a managed node group, EKS generates a new internal LT and merges only a subset of fields. CpuOptions is not in that subset — despite not being listed in the official blocked-fields list. This is a documentation gap.

Solution: Use deploy/eks/eks-self-managed-kvm.sh, which creates an ASG + Launch Template directly via AWS CLI, bypassing EKS's internal LT generation entirely.

Architecture note

Kubernetes manages the lifecycle of the zeroboot server process — it does not schedule individual sandboxes. Each /v1/exec request is handled entirely within the Pod via a KVM fork (~0.8 ms). K8s role is capacity management: health checks, rolling updates, and node-level scaling.

Testing

Validated on EKS 1.31 / ap-southeast-1 / c8i.xlarge (self-managed node group with nested virt):

• ✅ DaemonSet: 2 nodes → 2 pods scheduled automatically (one per KVM-capable node)
• ✅ hostPath /var/lib/zeroboot created automatically on each node
• ✅ /dev/kvm present on nodes
• ✅ Template creation completes (~19s)
• ✅ Readiness probe passes after template creation
• ✅ /v1/health → {"status":"ok","templates":{"python":{"ready":true,"numpy":true}}}
• ✅ CODE:print(1+1) → 2 (fork 2.2ms, exec 118ms)
• ✅ CODE:import numpy as np; print(np.array([1,2,3,4,5]).mean()) → 3.0 (exec 259ms)
• ✅ cat /etc/os-release → Ubuntu 22.04 content (exec 28ms)

Depends on: #14

Closes #9

[minhoryang]

Curious — if we need one Pod per KVM-capable Node, would a DaemonSet make sense here?

[chaosreload]

Good call — switched to DaemonSet as the default in the latest commit.

Added deploy/k8s/daemonset.yaml:

• nodeSelector: kvm-capable: "true" — only runs on KVM-capable nodes
• hostPath: /var/lib/zeroboot with DirectoryOrCreate — local storage is the right choice here since Firecracker snapshots are bound to the host CPU microarchitecture and KVM state; cross-node migration is not meaningful anyway
• updateStrategy: RollingUpdate with maxUnavailable: 1

Kept deployment.yaml as an annotated advanced alternative for HPA or manual replica control scenarios.

Updated KUBERNETES.md: DaemonSet + node-level autoscaling (Karpenter) is now the primary scaling model; Deployment + HPA documented as the fallback.

[chaosreload]

Hi @adammiribyan 👋

This PR adds Kubernetes deployment support for Zeroboot (closes #9). It's been validated on EKS 1.31 / ap-southeast-1 / c8i.xlarge.

Following feedback from @minhoryang, I've updated the default deployment to DaemonSet (instead of Deployment), which better fits Zeroboot's node-bound nature (/dev/kvm + host CPU microarchitecture).

Would you be able to take a look and review when you have a chance? Happy to address any feedback. 🙏