sandboxd: workspace filecache — cross-node NAS-backed workspace sync

[doge-rgb]

What

Bind a claimed sandbox to a shared workspace on a NAS mount and keep the two in sync under a multi-writer, close-to-open contract. A sandbox works against a node-local disk — no network client, no FUSE in the guest — while its workspace is durable and visible to other sandboxes on other nodes.

Feature is off unless workspace_root is configured; a claim opts in with an optional workspace name (WithWorkspace), ignored by nodes that don't have it.

Why

Mounting a NAS (EFS over NFSv4.1) into the data path pays a synchronous round trip per metadata op. Measured from a pool node:

op / workload	EFS direct	local NVMe
create (1 thread)	693 /s @ 1.44 ms	70,200 /s @ 14 µs
unlink	534 /s @ 1.87 ms	54,100 /s @ 17 µs
4K rand write	29.5k IOPS	244k IOPS
untar 100k files	628.8 s	2.66 s
yarn install (78k files)	212.6 s	12.3 s

The metadata path is ~100x a local disk (network RTT is only 0.18 ms — the rest is server-side), same-directory writes serialize server-side, and no mount option moves it. Staging the working set on a node-local disk and syncing deltas to the NAS off the hot path keeps guest I/O at local latency.

With the filecache (measured inside a real sandbox)

Same 100k-file / 565 MB workload, run in a claimed sandbox on this branch (guest-local virtio-blk storage, dedicated workspace disk attached):

workload	EFS direct	sandbox + filecache
untar 100k files	628.8 s	5.66 s in-guest (17.7k files/s — 111x)
rm -rf 100k files	198.5 s	1.98 s
file create burst (2k files)	693 /s	≥49.6k /s (shell-loop bound, not disk bound)
claim + hydrate a 565 MB workspace	—	5 s
cross-node visibility (10 pairs syncing under load)	—	p50 9 s / p95 9 s (30 s SLA)

The guest pays local-disk latency for every operation; the NAS sees batched deltas on the sync cadence and a final barrier at release.

Design

• filecache package — the sync engine. During a session the guest workspace is the source of truth; a per-node writer publishes its delta to <root>/<workspace> plus an append-only journal, and pulls other writers' entries. The NAS itself is the rendezvous — pullers poll a seq file and fetch only named deltas, no central coordinator. Concurrent edits resolve last-writer-wins with the loser preserved as <path>.fc-conflict-<ts>; deletes propagate via journal tombstones.
• pool.Manager arms a session on claim (beside egress) and runs a barrier on release/reap that publishes the final delta before teardown, so the next open sees it through the NAS's own close-to-open semantics.
• dedicated-disk mode (opt-in, workspace_disk_mb) puts the workspace on a fresh read-write ext4 virtio-blk disk, built on the writable catalog-volume primitives from Writable catalog volumes with a guest filesystem shutdown lifecycle #72 (DiskAttach with a rw spec, MountVolume rw, UnmountVolume; by-serial discovery from sandboxd: add read-only catalog volumes #69), isolating it from the guest rootfs COW. /workspace shows up as a real /dev/vdX mount. Unlike a catalog volume — one persistent node-local image, single writer by admission — the workspace disk is per-sandbox scratch: the durable artifact is the NAS workspace, which any number of sandboxes on any node share through the sync plane.
• claim carries an optional workspace name; SDK adds WithWorkspace / WithNoRedirect.

Verified on a 2-node cluster (real EFS filesystem)

Rebased onto #72 and re-verified on both nodes: full unit suite (with -race), dual-GOOS lint clean, and the cross-node E2E below re-run green on the rebased binary (propagation 7 s / 13 s / 9 s, barrier and dedicated-disk mount confirmed).

Cross-node E2E (sandboxd-driven, no external agent), FAILS=0:

test	result
hydrate from NAS on claim	✓ both nodes
A→B propagation	5 s
B→A propagation	10 s
delete propagation	5 s
barrier on release (next claim sees final write)	✓
conflict: concurrent write → LWW + .fc-conflict copy on both nodes	✓

Load (10 workspace-pairs = 20 sandboxes with dedicated disks, all syncing concurrently):

• claim + arm + disk-attach for 20 sandboxes: 3 s
• cross-node visibility under load: p50 9 s / p95 9 s / max 9 s
• warm pools stayed 200/200, no refill degradation
• sandboxd RSS +1–2 MB for all sessions combined

Unit tests cover the journal round-trip, tar hydrate/publish, atomic overwrite, and name parsing (sandboxd/filecache/sync_test.go). Full suite green (14 sandboxd packages + SDK).

Deployment note

The workspace NAS mount must use a low attribute cache (actimeo=1). With the common actimeo=3 we measured cross-client readdir latency of ~8 s (negative-lookup ~21 s) on EFS, which pushes visibility to the edge of the SLA; actimeo=1 drops it to ~1 s.

Follow-ups (not in this PR)

• The barrier publish is currently single-threaded (bounded by the NAS write rate, ~170 files/s); a large one-shot publish could parallelize per-directory as cross-directory writes scale ~15x.
• operator (cocoon-sandbox-operator) annotation → workspace passthrough for the K8s Sandbox CRD.