VFS Right-Thing Restructure: five-crate workspace, correctness fixes, semantic unification

[factory-ain3sh]

What this is

The complete "Right-Thing Restructure" of the vfs fork: 97 commits taking the tree from post-perf-campaign debt to the approved target architecture, executed as a validated 14-milestone mission (M1–M7 plus six hardening batches). This PR body doubles as the campaign's living history; the in-repo authority for design detail is docs/SPEC.md, docs/MANUAL.md, docs/KNOBS.md, docs/TESTING.md, and CHANGELOG.md.

North star (strictly ordered): correctness > consistency > completeness > effortless interface. Interface simplicity beats implementation simplicity. The perf campaign's measured wins were treated as correctness constraints, not negotiable tidiness.

Starting state (grounded in a 14-report, file:line-verified recon corpus): one 9,338-line god-file, two vendored crate forks leaking as public API, 96 undocumented-or-drifting knobs, three hand-synced DDL copies, five divergent mount-supervision paths, and a CI gate that masked failures.

Sacred invariants (held throughout, gate-enforced at HEAD)

1. All virtual FS state lives in the single AgentFS SQLite DB; no writes touch the real filesystem except the DB + transient sidecars (metadata-mutation-no-real-write.py, partial-origin-no-real-write.py in the standing gate).
2. Reads are scopeable; sandboxed writes never reach the host FS. Strengthened beyond plan: macOS Seatbelt got default-deny read scoping (230c30a).
3. noopen/noflush FUSE semantics preserved (coherence harnesses green in the gate).
4. Codex-workload perf floors preserved: final attestation shows all 7 benchmark phases within the 5% band vs the fresh M1 baseline (median-of-5, candidate+sentinel pairing; artifacts in .agents/benchmarks/restructure-attestation/).

What was done, milestone by milestone

M1 — Foundations + correctness bugs (fixed before anything moved)

• Root Cargo workspace, single lockfile, pinned toolchain.
• Deleted: Go/Python/TypeScript SDKs + their CI, examples/, Windows stubs + dist target, the experimental ptrace sandbox + --experimental-sandbox.
• Bug fixes (regression-tested red→green):
  • FUSE teardown deadlock in BOTH transports (generic session join hang + uring queue-thread circular wait) — bounded-teardown test on both legs (308bc1f).
  • NFS FILE_SYNC-without-durability + graceless shutdown → fsync-before-ack, cancellation token, finalize() (8337b61).
  • OverlayFS base-dir rename data loss → EXDEV.
  • Schema blanket-.ok() ALTERs → idempotent-or-propagate.
  • clone/init -c orphaned-mount-on-Ctrl-C.
• Honest gate wired into CI: no || true masking in all.sh/gate.sh, SKIP counts as failure, torture both legs.

M2 — Dead surface deletion

• abi-7-* FUSE feature matrix (17 features) collapsed to the one compiled ABI level; --all-features/--no-default-features builds restored to green.
• Dead vendored fuser/nfsserve surface, macOS/BSD branches in the Linux-only fuser, dead session APIs.
• The 1,400-line path-based SDK API: 8 dead methods deleted, 6 live ones delegated to the trait impl (kills the non-transactional autocommit bug class).

M3 — Config + telemetry extraction

• 96 knobs → typed config structs, one parsing grammar, env-parse-at-the-edge (core stops reading CLI env vars).
• docs/KNOBS.md generated-checked from the config types, including a sunset table with per-knob removal criteria.
• Telemetry macro registry replacing ~1,500 lines of six-way counter boilerplate; single report sink.

M4a/M4b — Core decomposition

• Hot path: store.rs extraction (chunk/inline engine), batcher PendingView/Drain trait split (type-enforced "no drain while holding a conn"), versioned cache-fill fix, lifecycle + ReapHook.
• Schema authority module: one DDL source (ddl-census.py enforces it in the gate), user_version migrations, integrity battery in core.
• Overlay directory-ization: one maps lock, whiteout txn helper, partial-origin writes through store.rs, FORGET-driven pruning. Tests moved out of code files.

M5 — Crate split

Five crates in a clean DAG, vendored code adopted first-party behind sealed entry APIs, blanket #![allow]s dropped:

agentfs-cli ──> agentfs-mount ──> agentfs-fuse ──┐
     │               │                           ├──> agentfs-core
     │               └──────────> agentfs-nfs ───┘
     └───────────────────────────────────────────┘

M6 — Semantic unification (the consistency pillar)

• Semantics facade under both adapters: handler code can only check permissions via one access::* implementation; a FUSE/NFS parity conformance suite runs in the gate, making adapter drift structurally impossible.
• Explicit AckDurability on every write ack — the reply's durability claim is derived from what the core actually did, so the pre-M1 "FILE_SYNC lie" cannot be reintroduced.
• HandleTable with LRU write-authority tokens + unlink invalidation; reply-caching DRC replacing the drop-filter tracker.
• run_supervised adopted by all five mount-owning commands (one supervision path instead of five).

M7 — CLI + docs consolidation

• RunOptions struct (kills the 7-file knob fan-out); unified DEFAULT_ALLOWED_DIRS; one migrate landing any supported old schema (v0.0/v0.2/v0.4) at CURRENT, in place or --copy, with --key/--cipher.
• MCP server fixed: tool parity, phantom tools removed, tool_calls audit logging.
• One error reporter; docs/ move with MANUAL/KNOBS generation-parity checks (the command reference is generated from the CLI's own argument definitions and cannot drift from --help).
• Scripts consolidated behind scripts/gate.sh (phase8 as top-level gate, historical validators archived, shared lib/).
• macOS Seatbelt default-deny read scoping for agentfs run (230c30a) — beyond the original plan.
• exec/run spawn failures now follow shell conventions: missing command → 127, non-executable → 126 (ad663b6).
• Final benchmark attestation committed (cf02f65); README rewritten to final state.

misc-a … misc-f — Hardening batches (validated like every other milestone)

Notable: uring mount drain race bounded; darwin SBPL parameterization (no raw string interpolation into Seatbelt profiles) + CI clippy on macos-latest; /System/Volumes/Preboot metadata path; schema-mismatch errors point at agentfs migrate from every reachable open site; migrate error-path sidecar cleanup; run mount-propagation guard (mount --make-rshared / remediation) and refusal to daemonize inside a live tokio runtime; run abort paths reap the child and unmount before exit; all test suites clean their session dirs (${HOME:?}-guarded); supervise double-fault ordering (spawn error stays the 127/126 root even when unmount also fails); hostile --session ids can't degrade the sandbox log tag.

How it was validated

• 176 behavioral contract assertions, 176 passed — every milestone was sealed by both a code-scrutiny review pass and black-box user-testing flows (CLI + real FUSE mounts + MCP stdio JSON-RPC + signal handling).
• Standing gate at HEAD: fmt/clippy -D warnings/workspace tests, shell suites with SKIP=fail, phase8 smoke, noopen/flush coherence, consistency canon (typed errors in core / anyhow at edge, one reporter, env reads only in config modules, no non-test file over the line cap, documented lock order) — PASS=24 SKIP=0 FAIL=0; pjdfstest 311/311; cross-surface parity 30/30.
• Perf: median-of-5 codex benchmark contract, all phases within the 5% band vs the fresh M1 baseline.

Intentionally deferred work (for future droids)

None of these are regressions; each was a deliberate correctness-first deferral with the landing site prepared:

1. NFS UNSTABLE+COMMIT (perf lever for macOS NFS writes). Today every WRITE is FILE_SYNC-honest: ack_durability_for_write_request() in nfs_handlers.rs deliberately maps everything to Committed (a unit test pins the policy), and NFSPROC3_COMMIT returns PROC_UNAVAIL — protocol-legal since we never reply UNSTABLE. The opt-in path is prepared end-to-end: the reply's committed field already derives from actual core durability, Semantics::commit_barrier(Some(ino)) is the documented COMMIT hook, and the write verifier is already per-server-instance. Remaining work: honor stable_how behind a config knob, dispatch COMMIT, harden the verifier (random u64 per boot rather than ms timestamp), and add crash-window/verifier-across-restart tests. Deferred because UNSTABLE with imperfect verifier semantics is exactly the data-loss class M1 exterminated, and the beneficiary (macOS) has no hardware in the validation loop.
2. Knob sunsets: compatibility kill-switches (AGENTFS_OVERLAY_READS, AGENTFS_DRAIN_ON_SETATTR, AGENTFS_DRAIN_ON_RELEASE, AGENTFS_FUSE_SELF_INVAL, AGENTFS_CLONE_TIMINGS) remain with documented removal criteria in docs/KNOBS.md (N=2 consecutive green milestones per knob). Removing them is mechanical once the criteria are met post-merge.
3. macOS runtime validation is a manual release gate: CI builds + clippys on macos-latest, but scripts/validation/macos-nfs-git-validation.sh plus the three Seatbelt spot-checks in docs/TESTING.md must run on real hardware before trusting a macOS release. No macOS hardware was available to this campaign (explicit, accepted gap).
4. io_uring FUSE coverage is local-only: CI kernels lack fuse.enable_uring; the honest uring gate is the local one (annotated in the CI workflow).
5. Known flakes (pre-existing, documented, rerun-before-concluding-regression): concurrency_integrity::active_workers timing assert under full-workspace parallel load; one-off mcp-server "stdio session failed" under back-to-back gate load.
6. FUSE STATX_BLOCKS kernel patch: written, VM-validated, upstream submission package complete under .agents/kernel/ — DCO sign-off and git send-email deliberately left to the author.
7. NFS UNSTABLE-mode durability aside: DATA_SYNC is currently treated identically to FILE_SYNC (stricter than required); fine to leave.

Merge readiness

• Linux (first-tier): ready — honest CI gate green, docs current, agentfs migrate lands any supported old DB at CURRENT.
• macOS (second-tier): compile/clippy-validated in CI; run the manual hardware gate before cutting a release that advertises macOS.
• No services, no migrations to run at merge time; the DB schema authority migrates user databases on demand.