Writable catalog volumes with a guest filesystem shutdown lifecycle - #72
Merged
Conversation
…onfirms A live VM keeps the hypervisor's image lock until it is gone, so releasing the admission hold or clearing the dirty marker before removal confirms let a failed removal re-dirty a clean image and turned contracted 409s into attach-time failures. Quiesce now records per-mount outcomes, the payload rides pendingRemoval across retries, and readers re-check markers after reservation. Umount failures fall back to one guest-wide sync; the quiesce budget scales with the writable mount count.
A healthy writer's own write-ahead marker made a concurrent read-only claim report needs-recovery instead of busy. The post-reservation check is the single dirty authority now, and it runs before the warm pop so a refused claim never consumes a VM.
This was referenced Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #70: write-enabled catalog volumes with the guest filesystem
shutdown lifecycle. Closes #70.
What ships
mode(ro|rw, default ro):rwrequires operatorwritable: trueon the catalog entry. Read-only requests and responses staybyte-identical to v1 (
modeomitted,"ro"normalized away everywhere).writer refuses every other claim on the name, live readers refuse a writer —
both as fast 409s before any attach cost. Rebuilt from persisted claims at
restart.
<path>.dirtybeside the image): durablycreated before the first rw attach, cleared only after a clean unmount AND
confirmed VM removal. Dirty + ro claim → 409 needs-recovery; dirty + rw
claim → allowed (journal replay is the recovery). Stat failures read as
dirty (fail closed).
rollback, quarantine): reverse-order umount of rw mounts, one guest-wide
sync fallback when an umount fails, budget scaling with the mount count
(capped 10s). Marker clearing and hold release ride
pendingRemovalacrossremoval retries and finish only when the VM is confirmed gone — a surviving
VM still holds the hypervisor lock, so the name stays truthfully busy.
reader never proceeds onto an image a failed writer dirtied in the gap.
Volume.Mode+VolumeInfo.Writable; Python{name, mount?, mode?}mapping +writablepassthrough. Both emitmodeonly for rw. Checkpoint claims keep rejecting volumes.
fleet-uniform metadata; only the image file lives on one node. Routing
reuses
VolumeOwnersunchanged.refused for any volume claim; idle sweep still excludes them.
HTTP, and a writable leg in
volumesmoke+scripts/sandboxd-e2e.sh(
VOLUME_RW_IMAGE=opt-in).Review rounds applied
Two adversarial correctness lenses (concurrency/lifecycle, contract/wire) and
three simplify lenses (reuse, altitude, efficiency) ran against the branch;
all confirmed findings are fixed in
ed6dca5/b2098b2. The main structuralfix: teardown originally cleared markers and released holds before VM removal
confirmed — a failed removal could re-dirty a clean image and turn contracted
409s into attach-time failures. Ordering pins are mutation-tested (moving the
teardown ahead of removal fails both pins).
Deliberate deviations from the issue text
UnmountVolume/SyncGuestprimitives; reverse order,budget, and marker bookkeeping live in pool's
quiesceVolumes(the issuenamed an engine-level
QuiesceVolumes— the split keeps the enginesingle-purpose).
confirmVolumesClean), not aresolve-time pre-filter: admission answers a live writer with 409 busy,
and needs-recovery is reserved for a genuinely crashed writer's marker.
The first hardware round surfaced the earlier asymmetry (a healthy
writer's own marker misreported needs-recovery); fixed in
b061f35,which also moves the check ahead of the warm pop so a refused claim
never consumes a VM.
Accepted residuals (do not fix)
pendingRemovalteardown payload: markersstay until an rw cycle clears them, holds die with the process. Conservative
direction, converges.
removal is still queued: an instant re-claim can hit the hypervisor lock as
an attach error instead of a 409. Transient, retry-convergent.
teardown (mutating the catalog under live claims is documented operator
error).
Hot-path cost
Volume-less and ro claim/release paths: zero new allocations, iterations, or
syscalls (escape-analysis verified; the only addition under
m.muis rangingan empty slice). rw-only costs: one stat per ro entry at admission, N umount
execs + optional sync at release.
Gates
go build+go test -race -count=1: sandboxd (13 packages), sdk/go,mcp, protocol/wire, e2e (GOWORK=off) — all ok
gofmt/gofumpt: cleansandboxd-e2e.sh: 0 issuesHardware verification (bare metal, Cloud Hypervisor v54, cocoon master-07f1d69)
Full round on the testbed at
b2098b2vs main218eee1, everything inside aprivileged container with an isolated cocoon root:
VOLUME RW PASS(rw claim92.9ms), read-only image sha256 unchanged from creation through every leg,
writable image sha256 advanced, no leftover
.dirtymarkers.samples tier-proven warm; pooled p50/p90 main 0.488/0.527ms vs branch
0.435/0.545ms; |A−B| mean 0.006ms against an A/A same-binary drift of
0.029ms. Release path also inside the A/A envelope.
kill -9the writer's VMM →marker persists → release of the dead claim returns in 12ms with exactly the
two tolerated quiesce errors (no panic, no hang) → next ro claim answers
409 needs-recovery → rw claim mounts with journal replay (106ms), pre-crash
stamp read back byte-identical → clean release clears the marker → ro
claims serve again.
(write-ahead marker fsync + rw attach); rw release equals a volume-less
release when the guest wrote nothing (42.7 vs 42.2ms p50); a 16MiB-dirty
guest adds ~8ms of quiesce flush. The ~40ms volume attach+mount base is
pre-existing (sandboxd: add read-only catalog volumes #69), not added by this PR. The ro-volume control arm
reproduced sandboxd: add read-only catalog volumes #69's settled ~38ms claim figure.