A micro-VM actor's guest RAM is backed by a memfd in the cloud-hypervisor process, so it counts against the worker pod's memory cgroup. The guest size comes from the kata-config default_memory, and the pod limit comes from the WorkerPool's pod template resources. Nothing checks that the guest, plus cloud-hypervisor/virtiofsd/ateom overhead, fits inside the pod limit.
The result is that an undersized worker pod is accepted without complaint. The VM may boot normally and report the full guest size to the workload, then get killed by the host later once enough guest RAM has been faulted in.
When it can cause issues
- Pod limit set at or near
default_memory, with no room for VMM overhead.
- The workload actually uses most of the guest RAM, so the memfd RSS grows toward
default_memory. Idle actors never trigger it.
- Or the pod is BestEffort, so under node pressure the worker is evicted first.
Examples
Both tested on a micro-VM pool with a 2Gi guest (default_memory = 2048, default_vcpus = 1). The pool's worker pods had no memory limit to begin with. The actor just reports the guest memory it sees, then writes into /tmp, which is the tmpfs overlay backed by guest RAM.
Worker pod limit 100Mi. The VM never finishes booting. The ateom container is OOMKilled repeatedly (exit 137) with a container lifetime of 8 to 13 seconds, which is the VM boot window, and goes into BackOff. The second worker in the same pool with the same limit stayed healthy, because no actor was ever assigned to it. Idle ateom is fine at 100Mi. Booting the VM is what crosses the cap.
Worker pod limit 500Mi. The VM boots cleanly and the guest reports its full 2Gi (memtotal_kb=2036276). The pod looks healthy and the actor runs normally for six and a half minutes, then the ateom container is OOMKilled (exit 137). Nothing warned at apply time or at boot, the guest was never over its own limit, and no container inside the guest misbehaved. The kill came from the host, because the VMM plus the guest RAM it had faulted in did not fit the pod limit.
Suggested
At ateom startup, assert guest_size + overhead <= pod memory limit and fail with a clear error naming both numbers, instead of a later host OOM that reads as an actor crash. Document the invariant so operators size the worker pod above the guest rather than equal to it. The overhead figure should be measured against the real VMM build rather than guessed.
Related: #793, #212.
A micro-VM actor's guest RAM is backed by a memfd in the cloud-hypervisor process, so it counts against the worker pod's memory cgroup. The guest size comes from the kata-config
default_memory, and the pod limit comes from the WorkerPool's pod template resources. Nothing checks that the guest, plus cloud-hypervisor/virtiofsd/ateom overhead, fits inside the pod limit.The result is that an undersized worker pod is accepted without complaint. The VM may boot normally and report the full guest size to the workload, then get killed by the host later once enough guest RAM has been faulted in.
When it can cause issues
default_memory, with no room for VMM overhead.default_memory. Idle actors never trigger it.Examples
Both tested on a micro-VM pool with a 2Gi guest (
default_memory = 2048,default_vcpus = 1). The pool's worker pods had no memory limit to begin with. The actor just reports the guest memory it sees, then writes into/tmp, which is the tmpfs overlay backed by guest RAM.Worker pod limit 100Mi. The VM never finishes booting. The ateom container is OOMKilled repeatedly (exit 137) with a container lifetime of 8 to 13 seconds, which is the VM boot window, and goes into BackOff. The second worker in the same pool with the same limit stayed healthy, because no actor was ever assigned to it. Idle ateom is fine at 100Mi. Booting the VM is what crosses the cap.
Worker pod limit 500Mi. The VM boots cleanly and the guest reports its full 2Gi (
memtotal_kb=2036276). The pod looks healthy and the actor runs normally for six and a half minutes, then the ateom container is OOMKilled (exit 137). Nothing warned at apply time or at boot, the guest was never over its own limit, and no container inside the guest misbehaved. The kill came from the host, because the VMM plus the guest RAM it had faulted in did not fit the pod limit.Suggested
At ateom startup, assert
guest_size + overhead <= pod memory limitand fail with a clear error naming both numbers, instead of a later host OOM that reads as an actor crash. Document the invariant so operators size the worker pod above the guest rather than equal to it. The overhead figure should be measured against the real VMM build rather than guessed.Related: #793, #212.