feat(driver): a cold suspend produces the portable workspace checkpoint, and a resume can restore from it

[jiashuoz]

Summary

A cold suspend of a microVM session now turns the session's workspace into a
committed, verified portable workspace checkpoint before the VM is
terminated, and a resume whose workspace image is gone rebuilds it from that
checkpoint. That is ADR-0003 §4.4's durability barrier and §2.3's deep-dormant
tier, on top of the format and library from #99.

The decision this implements, and the reason everything else follows from it:
the workspace is an ext4 image the guest mounts, the checkpoint library
takes an fs.FS, and the host must never mount or parse a tenant's ext4
image. So the guest streams a tar of /workspace over the vsock relay
connection it already has, and the host feeds that stream to the checkpoint
writer as a tar-backed fs.FS. Nothing plaintext reaches host disk on the
suspend path: the bytes go from a socket, through a 32 KiB copy buffer, into an
AES-256-GCM frame, into a blob store.

Design note: docs/design/2026-09-23-workspace-checkpoint-wiring.md.

The commits

1. docs(design) — the note: the six options (host loop-mount, host
   userspace ext4 reader, guest-streamed tar, opaque image blob, network
   filesystem, block diff) and why guest-streamed wins; the handshake, its
   frames and its new budgets; the limits the host enforces; what each failure
   leaves behind; the restore path; the self-hosted store and key; what
   rainier-cloud owes.
2. feat(sessiond) — relay.FrameStream (type 5) carrying the suspend
   nonce as its stream id, ControlSender.SendStream sharing the control
   writer, the workspace_end and checkpoint_committed kinds; the
   internal/wstream format and its write side; and the cold handler streaming
   /workspace before it answers suspend_ready.
3. feat(wstream) — the tar-backed fs.FS the checkpoint writer walks: a
   one-way cursor over the stream, every limit, the walk-order checks, a round
   trip through the real library (write → verify → restore → compare), and
   FuzzFS.
4. feat(driver) — the barrier: Suspend(warm=false) sends the notice,
   reads the stream, writes the checkpoint, waits for the manifest commit,
   verifies, tells the guest, and only then terminates the VM.
5. feat(driver) — the deep-dormant resume: restore into a 0700 scratch
   directory, give the tree to the guest's user, mkfs.ext4 -d, rename into
   place, boot.
6. feat(runnerd) — --checkpoint-store-dir and --checkpoint-key-file
   (plus --checkpoint-restore-uid/-gid), a local-directory blob store whose
   put-if-absent is link(2), and a key loader that fails closed.

Plus two fix commits from the review below.

The handshake

runnerd/driver                                   sessiond (guest)
  |  FrameControl suspending{cold,id=N}  ------------>|
  |<---------------  FrameControl suspend_ack{id=N}   |  (2 s budget)
  |                                                   |  flush, kill execs,
  |                                                   |  unmount agent home,
  |                                                   |  forget secrets
  |<---------------  FrameStream[id=N] chunk          |  (30 s to the 1st chunk)
  |<---------------  FrameStream[id=N] chunk          |  (60 s idle budget,
  |       ... checkpoint writer consumes as it goes ..|   30 min total)
  |<--- FrameControl workspace_end{id=N,ok,entries,bytes}
  |<---------------  FrameControl suspend_ready{id=N} |  (30 s budget)
  |  ... Writer.Write commits the manifest; the       |
  |      generation is recorded and persisted HERE    |
  |  ... Reader.Verify reads the committed bytes ...  |  (10 min budget)
  |  FrameControl checkpoint_committed{id=N} -------->|  (best effort)
  |  engine.Stop, release slot, discard rootfs        |

#98's cold suspend was 2 s + 30 s, which bought a flush, a ten-second exec kill
and an unmount; it cannot also buy a 10 GiB copy. The budgets are now a shape
rather than one stopwatch — 30 s to the first chunk, 60 s of progress between
chunks, 30 min total, 30 s for the ready, 10 min for the verify — worst case
≈ 41 minutes and only for a session that is genuinely stuck. The idle budget
covers this host's slowness too: the stream is back-pressured by the store
through an io.Pipe, so a store that stopped taking bytes fails the suspend,
which keeps the VM.

The stream is not only a tar. fs.WalkDir asks for every child of a
directory before it visits the first one, and a child's subtree sits between it
and its next sibling — so a plain tar is not walkable without buffering content,
which is the one thing this design refuses. The stream is therefore a magic
line, an index of (kind, name) in walk order, then the tar. The index is the
tree's shape only; every other attribute comes from the tar header when the
cursor arrives, so each fact has one source.

The limits the host enforces

limit	default
entries	500,000
index bytes (charged what the structure costs, ~96 B/entry + names)	64 MiB
entry name	4096 bytes, 255 per element
absolute / uncleaned / .. / . / NUL in a name	refused
a symlink target absolute or leaving the tree	refused
one entry's bytes	8 GiB
the whole stream	32 GiB
entry kinds	dir, regular file, symlink only
order	strict fs.WalkDir order, no duplicates

All configurable (wstream.Limits), all applied by the guest and by the host,
because the host must not be the hop that trusts the guest — and re-applied
again by checkpoint itself on the way in and on the way out.

What only a real host can verify

Everything here runs against the simulated engine, the pipe-backed fake
transport, fstest.MapFS/temp directories and the in-memory blob store. Four
things are consequently claimed and not evidenced:

• A real guest streaming a large workspace. The KVM harness from test(driver): a Firecracker-backed RunContract harness and boot smoke for a real KVM host #102 gained
  the same handshake, so RAINIER_MICROVM_KVM_TEST=1 on a real KVM host
  exercises a real sessiond streaming a real ext4 workspace into a real
  checkpoint. It has never run here.
• mkfs.ext4 -d on a restore. The simulated formatter writes the tree as a
  tar so a test can read back what the restore produced; only a real host runs
  mke2fs. Two specifics to watch there: the default inode count for a 10 GiB
  image (~655,000, above our entry ceiling — a tree past it fails the mkfs), and
  the ownership step, which needs a runner running as root to give files to uid
  1000.
• Timings against the cold budget. 30 min for the stream is 10 GiB at
  ~6 MiB/s, from arithmetic and not from a measurement of vsock plus a store.
• The key file's ownership check, which cannot be exercised without root to
  create a file owned by somebody else.

What rainier-cloud owes

• A GCS blob store behind checkpoint.BlobStore
  (x-goog-if-generation-match: 0, resumable upload for the content object).
• A KMS wrapper that passes the authenticated context as AAD and returns the
  concrete key version, plus the regional key-readiness rule Preflight is for.
• The workspace identity: this driver has no workspace id in its Spec and
  uses the workspace volume name, so a cell carrying the control plane's id
  should pass that instead.
• Checkpoint-age policy and the freshness view (PRD §10), the orphan
  sweep for objects whose manifest lost a race or failed its verify, and
  deletion (tenancy §14.2).
• A dispatch timeout for a cold stop that covers the budgets above.

The review

An independent adversarial review of the whole diff (a subagent on opus, given
the diff, the design note, ADR §4.4 and the tenancy rows) was asked for five
specific things: plaintext reaching host disk, a barrier that reports success
before the commit, tar-limit bypasses, a VM terminated on a failure path, and
vacuous tests (revert each guard, see what fails). It found eleven real problems
and confirmed the rest. Every finding is fixed, each with the test that would
have caught it (deca963, d0e344c):

High. (1) A committed-but-unverified checkpoint bricked every later cold
suspend of that session: a manifest key has no attempt suffix and
put-if-absent never overwrites, so a spent generation that the record never
advanced past made every retry lose its manifest put to ErrExists. The
generation is now recorded and persisted the moment Write returns, and the
driver probes for a free generation before it streams, so a record behind the
store heals itself. (2) The generation was persisted only by Suspend's own
saveRecord, with two returns in between. (3) mkfs.ext4 -d's output names the
files it was copying and was going into an error that reaches a session's error
column — counted now, not quoted. (4) RemoveAll's error is an *fs.PathError
naming an entry inside the workspace.

Medium. (5) The chown walk's error, same shape. (6) A crash mid-restore left
a tenant's whole workspace in plaintext under the state directory with nothing
to sweep it — the driver now clears <state>/restore at startup. (7) The image
was created at its final path and populated over minutes, while the
restore's trigger is that path's absence: a crash there left a file the next
resume would see, skip the restore for, and hand the guest unformatted. Built at
.partial and renamed now. (8) Accepted, not fixed, and now an open question
in the note: a session whose sandbox is gone can never be cold-parked and pins
a slot. The narrower rule — park without a checkpoint, gate only the
deep-dormant disk deletion on one — is what §4.4 actually asks for and is a
change to what the runner records rather than to this barrier. (9) The 1 MiB
trailer bound turned a host-side-only exclusion into a permanently failing
suspend; the drain runs under the stream's own MaxTotalBytes now. (10) The
index's memory was understated and unbounded across concurrent suspends: it is
now charged what it costs, and the concurrency bound is an open question with
the arithmetic written down. (11) A session id this driver accepts and
checkpoint.Context refuses failed every cold suspend; it fails the create now.

Low. Key-file ownership, a pre-existing store directory's mode, a symlink
with a body, a stale comment, and an unbounded guest-supplied tail.

Vacuous tests. The review reverted six guards. Three had no test behind them:
the ownership step's call site (covered now by a uid the test cannot give a
file to, which must fail closed and leave no image), the driver's own
exclusions (covered by a checkpoint that must not carry them), and the
put-if-absent (covered deterministically by a winner committing during the
loser's write — os.Rename passes the sequential test and fails this one). It
also found that wstream.FS.ReadLink trusted the cursor's backwards guard and
had none of its own, so a caller asking about a passed entry would have been
handed the current entry's target: a symlink checkpointed with somebody
else's destination, verified and wrong. Fixed, with tests for readlink, stat and
open. Each fix was confirmed by reverting the guard and watching the new test
fail.

Review round 2

A second pass over this branch raised four things. What changed, and where it
did not:

1. The deep-dormant restore after a runnerd restart. The finding is right:
   ensureWorkspaceForResume sits behind the !bootLive refusal in Resume,
   and a session dormant long enough for its disk to be deleted has almost
   certainly outlived the runnerd that created it — so in production the
   restore path is behind that refusal. It is not reordered, and the reason
   is not ordering. A recovered record has no guest configuration at all:
   ADR-0003 §2.7 item 1 keeps the session id, the proxy, the boot chain and the
   secret names off a shared host's disk, and this driver cannot invent them.
   Restoring first would spend minutes of copying and leave a tenant's whole
   workspace in plaintext under <state>/restore for a resume that is certain
   to fail a moment later — widening the very window §7 bounds — and then still
   refuse. So the refusal stays first, and what it now does is tell the two
   cases apart at the cost of one Stat and no tenant bytes
   (internal/driver/microvm_checkpoint.go:477 coldResumeNeedsGuestConfig
   and deepDormantNote): whether the workspace image is also gone, and
   which committed generation holds the only copy of the work, or that no
   checkpoint was ever committed and nothing can recover it. "Re-dispatch this
   session through the control plane" and "this person's work exists only as
   checkpoint generation 4" are different operator actions and are now
   different sentences. The precondition — the restore is reachable only from
   a record this process still holds the guest configuration for — is stated
   in the note's §7, and the create-shaped resume that lifts it (the
   control plane re-resolving the configuration) is now listed among what
   rainier-cloud owes, in §9. Tests:
   TestAColdResumeAfterARestartSaysWhichThingItIsWaitingFor and
   TestAColdResumeAfterARestartWithNoCheckpointSaysThatToo
   (internal/driver/microvm_restore_test.go:615 and :666) build a second driver over
   the state directory a first one left, resume the recovered record, and
   assert the refusal names both halves — and that nothing was spent on it:
   no mkfs, no scratch tree, nothing at the workspace image path.
2. Error sanitization. scratchDir's os.RemoveAll fails on an entry
   inside a previous restore of this workspace, so the *fs.PathError it
   returns names a tenant's file, and this error reaches a session's error
   column through restoreErr. It goes through pathFreeCause now, like the
   deferred removal ten lines below it
   (internal/driver/microvm_checkpoint.go:699). Test:
   TestTheScratchDirectorysRemovalErrorNamesNoPath
   (internal/driver/microvm_restore_test.go:761) plants a directory this
   process cannot clear and asserts the error carries neither the file name,
   nor the state directory, nor a separator.
3. The untested guards. reclaimRestoreScratch:
   TestAPreviousRunsRestoreScratchIsRemovedAtStartup
   (internal/driver/microvm_restore_test.go:706) plants
   <state>/restore/i-old/src/f before the driver is constructed and asserts
   it is gone after — and that its bytes are nowhere else under the state
   directory. checkKeyOwner: internal/driver/checkpointkey_unix_test.go
   fakes the fs.FileInfo (a foreign uid in its *syscall.Stat_t), which is
   the only way to exercise a check a test has no privilege to set up; a
   matching uid and a filesystem that reports no ownership are both accepted,
   so the refusal is about ownership and not about every file.
   clampGuestText: internal/runnerd/workspacestream_test.go:355 — the
   bound, the boundary, the truncation notice, and a 1 MiB guest string coming
   back bounded. Each of the three was confirmed by reverting the guard and
   watching the new test fail.
4. Docs. docs/microvm-host-privileges.md carries --checkpoint-store-dir
   and --checkpoint-key-file on the startup-refusal list, with a note that
   these two are checked in cmd/runnerd rather than generated from
   MicrovmHostRequirements()'s table. In the design note: §4.4's quote now
   carries an ellipsis where "to regional GCS" was dropped, and says what
   was elided and why; §6 no longer calls the before-termination order
   "stricter than §4.4 needs" — it is a deliberate inversion of §4.4's
   stated order (checkpoint from the detached disk, after termination), forced
   by §4.5's rule that the host never mounts or parses a tenant's ext4 image,
   with the consequence stated plainly that a host dying mid-checkpoint
   cannot re-checkpoint that workspace afterwards (rainier-cloud is amending
   ADR-0003 §4.1, §4.3 and §4.4 to match); and §9 gains four things this change
   leaves undone — a host-wide bound on concurrent checkpoints (the index
   is O(entries) per in-flight suspend, capped at 64 MiB, times the host's
   slots: ~2 GB for 32), the "verified checkpoint newer than the disk's last
   write" guard before deep-dormant disk deletion, which is owned by nobody
   today and belongs in the cell-worker path that calls RemoveWorkspace,
   the fact that the record's CheckpointAt is a commit time with nothing
   on the record distinguishing committed from verified, and that
   DirBlobStore has no deletion path at all.

Gates

make verify; the gate set (./internal/driver/... ./cmd/sessiond/... ./internal/runnerd/... ./internal/relay/... ./checkpoint/...) under -race -count=1 three times, each with the full top-level count and no panic; go test ./... -race -count=1; go vet ./...; GOOS=linux go build ./... && GOOS=linux go vet ./internal/driver/... ./cmd/...; git diff --check; a 30 s FuzzFS
run — all clean.

One thing worth recording rather than quietly re-running away: on the first
full go test ./... -race after the review fixes,
TestExecResizeAndSignalReachTheProcess in internal/sandboxexec failed on a
5 s timeout under the load of the whole suite on this 4-core box, then passed on
a targeted -count=3 re-run, on a full run of that package, and on a second
full ./... -race run. That package's only contact with this branch is
relay.Execer/relay.ExecAttachment, neither of which this change touches.

🤖 Generated with Claude Code