microvm: write container rootfs to host disk instead of guest memory

[Lucky Abolorunke (Oneimu)]

Problem

Each container's rootfs in ateom-microvm is an overlay whose writable upper lived on a tmpfs inside the guest: every byte an actor wrote consumed guest RAM 1:1, and writes failed with ENOSPC at the tmpfs cap — 20% of guest RAM, ~392 MiB on the standard 2 GiB guest (measured). Any actor unpacking a checkout, pip-installing, or logging heavily hits a hard, workload-dependent crash, and write-heavy actors degrade worker density for their whole lifetime.

Changes

The overlay upper moves to host disk: a third virtio-fs share (ateUpper — own virtiofsd, write-through, cache=auto, --xattr), serving a per-actor host directory (<actor>/rootfs-upper/), with each container's upper/work as sibling directories under it. Rootfs writes now cost host disk, not guest RAM, and are no longer capacity-capped. There is no configuration surface — this is the only mode, matching the gVisor runtime's no-knob stance.

Snapshot/restore integration:

• FULL checkpoints: Archive the upper as rootfs-upper.tar, taken while the guest is paused (write-through ⇒ coherent) and concurrently with the CH memory snapshot and durable-dir tar — the paused window costs the slowest artifact, not their sum.
• Restores: Re-materialize the upper from the tar in the background, overlapped with lower staging and network/tap setup, joined just before the share's virtiofsd starts.
• Self-describing restore: The tar's presence routes it. Snapshots taken by the previous tmpfs implementation restore unchanged (their upper rides inside the restored guest memory), and their subsequent checkpoints stay correct (rootfs_upper=0, no tar emitted).
• DATA scope unchanged: Rootfs state is discarded, as before.

Enabling fixes (required for any overlay upper on virtio-fs):

• Overlay mount options index=off,metacopy=off,userxattr (guest kernel returns EINVAL otherwise).
• tarutil now round-trips overlay deletion metadata: whiteout device nodes (mknodat, parent-fd contained) and user.overlay.* opaque-directory xattrs (PAX SCHILY.xattr records, restored through the extraction root so a crafted archive cannot write outside it). Without these, deleted files/directories silently reappeared after resume.

How it was tested

Unit tests (all pass; privileged tests executed as root inside a kind node, no skips):

• tarutil round-trips: contents/modes/mtime/symlinks/hardlinks, device nodes (incl. a literal 0:0 whiteout), user.* xattr PAX round-trip (incl. user.overlay.opaque), extraction-escape rejections.
• Overlay layout regression test (upper/work are siblings on one filesystem — kernel requirement), virtiofsd args (--xattr only on the upper share), snapshot config rewriting for all three fs tags.

End-to-end (kind cluster, counter demo actor with a durable-dir volume — so every run also covers durable-dir coexistence):

• Boot + write path: Actor boots with exactly three virtiofsds (RO lower / durable / upper, --xattr on the upper); a file written inside the guest appears in the host rootfs-upper/ directory.
• Guest-created deletion metadata: In-guest rm of an image file produced a genuine overlayfs whiteout on the host (char 0:0, nlink 2 — kernel-created, via mknod through virtiofsd); in-guest rm -rf && mkdir of an image directory produced user.overlay.opaque="y" on the host upper (xattr passthrough).
• FULL suspend/restore: Checkpoint log shows concurrent captures (snapshot ∥ durable_dir ∥ rootfs_upper); after restore, verified inside the guest: written marker present, deleted file still absent, replaced directory still empty; durable counter state intact.
• Multi-cycle: restore → write 64 MiB → second suspend → restore; payload verified byte-identical by md5 across cycles (OnDemand diff-merge coexistence).
• Scope coverage: DATA-scope commit emits no rootfs-upper.tar (rootfs_upper=0); DATA_ON_GOLDEN resume stitches the golden's rootfs-upper.tar + memory with the actor's own durable tar (verified in atelet download logs), preserving actor data isolation.
• Backward compatibility: A legacy (tmpfs-era) snapshot restores on this build (no upper share staged, two virtiofsds), and its re-checkpoint correctly emits no upper tar.

Performance

in progress.

[Lucky Abolorunke (Oneimu)]

/assign Benjamin Elder (@BenTheElder)

[Benjamin Elder (BenTheElder)]

Is it possible to do the read-only image rootfs + overlay upper on the host side behind the virtiofsd mount, and then tar up the upper if a snapshot is requested (and move the virtiofs to auto instead of always), or does that not work? That might be somewhat less complex and less expensive than the additional virtiofsd if it works.

[Benjamin Elder (BenTheElder)]

🤖 nit 🟢 – The legacy branch is the one entry path that leaves the upper directory as it found it. Cold boot calls resetRootfsUpperDir and a disk-upper restore's untarRootfsUpper starts with RemoveAll, but a no-tar restore touches nothing. Teardown's removal is warn-and-continue, and the directory sits on the BasePath hostPath, so it also survives an ateom kill before teardown runs.

When a stale directory does survive into a legacy restore, actorHasDiskUpper reports true for an actor whose upper is really in guest memory. The next FULL checkpoint then tars that directory and emits a rootfs-upper.tar the guest never used, and the restore after it stages a third virtiofsd for a share the snapshot's config.json does not reference. The real upper still comes back from memory, so this is a mislabelled snapshot rather than a broken one — but the mislabel is sticky.

Calling resetRootfsUpperDir on this branch too would make "directory absent" mean the same thing on all three entry paths.

[Benjamin Elder (BenTheElder)]

This package is used for paths that need to be shared between ateom and atelet, or at least multiple ateom implementations, is there a reason to expose this?