Skip to content

Writable catalog volumes with a guest filesystem shutdown lifecycle - #72

Merged
CMGS merged 10 commits into
mainfrom
feat/issue-70-rw-volumes
Aug 12, 2026
Merged

Writable catalog volumes with a guest filesystem shutdown lifecycle#72
CMGS merged 10 commits into
mainfrom
feat/issue-70-rw-volumes

Conversation

@CMGS

@CMGS CMGS commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Implements #70: write-enabled catalog volumes with the guest filesystem
shutdown lifecycle. Closes #70.

What ships

  • Per-volume mode (ro|rw, default ro): rw requires operator
    writable: true on the catalog entry. Read-only requests and responses stay
    byte-identical to v1 (mode omitted, "ro" normalized away everywhere).
  • Admission registry mirroring the hypervisor's per-image lock: a live
    writer refuses every other claim on the name, live readers refuse a writer —
    both as fast 409s before any attach cost. Rebuilt from persisted claims at
    restart.
  • Write-ahead dirty markers (<path>.dirty beside the image): durably
    created before the first rw attach, cleared only after a clean unmount AND
    confirmed VM removal. Dirty + ro claim → 409 needs-recovery; dirty + rw
    claim → allowed (journal replay is the recovery). Stat failures read as
    dirty (fail closed).
  • Quiesce on every live-guest termination path (release, TTL reap,
    rollback, quarantine): reverse-order umount of rw mounts, one guest-wide
    sync fallback when an umount fails, budget scaling with the mount count
    (capped 10s). Marker clearing and hold release ride pendingRemoval across
    removal retries and finish only when the VM is confirmed gone — a surviving
    VM still holds the hypervisor lock, so the name stays truthfully busy.
  • Post-reservation dirty re-check closes the resolve→reserve TOCTOU: a
    reader never proceeds onto an image a failed writer dirtied in the gap.
  • SDK surfaces: Go Volume.Mode + VolumeInfo.Writable; Python
    {name, mount?, mode?} mapping + writable passthrough. Both emit mode
    only for rw. Checkpoint claims keep rejecting volumes.
  • Fleet: zero wire change. The catalog entry (name, ACL, writable) is
    fleet-uniform metadata; only the image file lives on one node. Routing
    reuses VolumeOwners unchanged.
  • Capture refusal unchanged: hibernate/fork/checkpoint/promote stay
    refused for any volume claim; idle sweep still excludes them.
  • Docs (deploy, api, cluster, security, both SDKs), e2e drift pins over raw
    HTTP, and a writable leg in volumesmoke + scripts/sandboxd-e2e.sh
    (VOLUME_RW_IMAGE= opt-in).

Review rounds applied

Two adversarial correctness lenses (concurrency/lifecycle, contract/wire) and
three simplify lenses (reuse, altitude, efficiency) ran against the branch;
all confirmed findings are fixed in ed6dca5/b2098b2. The main structural
fix: teardown originally cleared markers and released holds before VM removal
confirmed — a failed removal could re-dirty a clean image and turn contracted
409s into attach-time failures. Ordering pins are mutation-tested (moving the
teardown ahead of removal fails both pins).

Deliberate deviations from the issue text

  • Engine exposes UnmountVolume/SyncGuest primitives; reverse order,
    budget, and marker bookkeeping live in pool's quiesceVolumes (the issue
    named an engine-level QuiesceVolumes — the split keeps the engine
    single-purpose).
  • The dirty check is post-reservation only (confirmVolumesClean), not a
    resolve-time pre-filter: admission answers a live writer with 409 busy,
    and needs-recovery is reserved for a genuinely crashed writer's marker.
    The first hardware round surfaced the earlier asymmetry (a healthy
    writer's own marker misreported needs-recovery); fixed in b061f35,
    which also moves the check ahead of the warm pop so a refused claim
    never consumes a VM.

Accepted residuals (do not fix)

  • A restart loses the in-memory pendingRemoval teardown payload: markers
    stay until an rw cycle clears them, holds die with the process. Conservative
    direction, converges.
  • Setup-failure paths release holds via the armed defer while the doomed VM's
    removal is still queued: an instant re-claim can hit the hypervisor lock as
    an attach error instead of a 409. Transient, retry-convergent.
  • A catalog entry removed while its claim is live skips the marker clear at
    teardown (mutating the catalog under live claims is documented operator
    error).

Hot-path cost

Volume-less and ro claim/release paths: zero new allocations, iterations, or
syscalls (escape-analysis verified; the only addition under m.mu is ranging
an empty slice). rw-only costs: one stat per ro entry at admission, N umount
execs + optional sync at release.

Gates

  • go build + go test -race -count=1: sandboxd (13 packages), sdk/go,
    mcp, protocol/wire, e2e (GOWORK=off) — all ok
  • golangci-lint dual-GOOS (cache-cleaned): 0 issues; gofmt/gofumpt: clean
  • asl dual-GOOS: zero findings
  • sdk/python: ruff clean, pytest 147 passed
  • shellcheck on sandboxd-e2e.sh: 0 issues

Hardware verification (bare metal, Cloud Hypervisor v54, cocoon master-07f1d69)

Full round on the testbed at b2098b2 vs main 218eee1, everything inside a
privileged container with an isolated cocoon root:

  • Full e2e including the writable leg: PASS. VOLUME RW PASS (rw claim
    92.9ms), read-only image sha256 unchanged from creation through every leg,
    writable image sha256 advanced, no leftover .dirty markers.
  • Volume-less warm claim unchanged (the hot-path requirement): 96/96
    samples tier-proven warm; pooled p50/p90 main 0.488/0.527ms vs branch
    0.435/0.545ms; |A−B| mean 0.006ms against an A/A same-binary drift of
    0.029ms. Release path also inside the A/A envelope.
  • Issue volumes: writable catalog disks need a guest filesystem shutdown lifecycle #70 crash sequence end-to-end: PASS. kill -9 the writer's VMM →
    marker persists → release of the dead claim returns in 12ms with exactly the
    two tolerated quiesce errors (no panic, no hang) → next ro claim answers
    409 needs-recovery → rw claim mounts with journal replay (106ms), pre-crash
    stamp read back byte-identical → clean release clears the marker → ro
    claims serve again.
  • rw costs on a settled node: rw claim ≈ +3ms over an ro-volume claim
    (write-ahead marker fsync + rw attach); rw release equals a volume-less
    release when the guest wrote nothing (42.7 vs 42.2ms p50); a 16MiB-dirty
    guest adds ~8ms of quiesce flush. The ~40ms volume attach+mount base is
    pre-existing (sandboxd: add read-only catalog volumes #69), not added by this PR. The ro-volume control arm
    reproduced sandboxd: add read-only catalog volumes #69's settled ~38ms claim figure.

CMGS added 10 commits August 12, 2026 20:41
…onfirms

A live VM keeps the hypervisor's image lock until it is gone, so releasing
the admission hold or clearing the dirty marker before removal confirms let
a failed removal re-dirty a clean image and turned contracted 409s into
attach-time failures. Quiesce now records per-mount outcomes, the payload
rides pendingRemoval across retries, and readers re-check markers after
reservation. Umount failures fall back to one guest-wide sync; the quiesce
budget scales with the writable mount count.
A healthy writer's own write-ahead marker made a concurrent read-only
claim report needs-recovery instead of busy. The post-reservation check
is the single dirty authority now, and it runs before the warm pop so a
refused claim never consumes a VM.
@CMGS
CMGS merged commit 34bb49d into main Aug 12, 2026
2 checks passed
@CMGS
CMGS deleted the feat/issue-70-rw-volumes branch August 12, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

volumes: writable catalog disks need a guest filesystem shutdown lifecycle

1 participant