[Danceiny]

Refs #674

#684 made the graceful handoff lose no acknowledged write. An independent re-review of that fix (from the A2 session) found no new blocker, but left one liveness defect and a list of smaller ones, and said which of the original findings were only half fixed. I did not take the list on trust: for each item I wrote a check from its description, watched it fail on main where it could, then fixed it. Everything below reproduced, except where it says it is a decision.

Base is main; this is independent of #687 (the credential broker and governor follow-up). It changes nothing in the kernel (kernel-manifest-gate: zero diff), adds no dependency, and touches no WriteGate or M5 behavior.

What was wrong, and what changed

Finding	Severity	Fix	Pinned by
The drain had no deadline, and since #684 the lease is renewed through it: one call that never returns kept the tenant on its replica for ever, made router.close() wait for ever, and left the tenants queued behind it unflushed. abandon() did not settle the pending handoff either	major (liveness; latent while the A2 surface drains its requests first)	drainTimeoutMs (default 120 s). Past it handoff() rejects with the typed DrainTimeoutError, a drain_timeout event says so, and the tenant serves again. close() hands tenants off several at a time (closeConcurrency, default 8), so a stuck one holds only its own slot and is reported unflushed. An abandon, a forget or a lost lease now settles a waiting handoff	D1, D2, D3 (three ways), D4, D6
forget() during a flushing handoff lost: the handoff released the lease, forgetTenant then hit FENCED, and the final image stayed in the store	minor, but it is the delete-means-delete red line	The handoff re-checks its state after the upload and neither releases the lease nor reports a flush when a forget or abandon took the tenant. Forget wins	F3
forgetTenant raised the floor and then deleted everything under the tenant's key. An owner paused between the two woke after a successor had taken over and published, and wiped the successor's epoch (found by the review of this branch; it was in the code before)	major (acknowledged writes of the successor, beyond the declared RPO)	It deletes only the epochs at or below the floor it raised	F7
forget() stopped lease renewal and then waited without bound for the upload, so an upload longer than the lease fenced it before it could delete	minor	Renewals run until forgetTenant has handed the lease back	F4
While forget() waited, calls through the handle and the ledger threw an untyped TypeError: the database connection is not open	minor	The typed draining error, from the handle and from the view. An abandon during a forget does nothing, and a second forget, or a forget on a handle whose abandon is under way, is refused	F5, F6, D5
router.close() did not wait for a forget under way (the handle had left its table), so a stop mid-forget left it half done and the lease unreleased	minor	close() waits for it, for as long as a drain may take	F8
A failed handoff could leave a second renewal chain when one renewal outlasted the interval	minor	One chain: no new timer while a tick is under way	R1
run() accepted a callback that returns a promise, which escapes the lease bracket and the drain and becomes an unhandled rejection	minor	A TypeError naming runAsync; the promise it leaves behind is taken	T1
Finding 3 of the original review, half fixed: ledger.db.close() closed the router's own connection, and Object.getOwnPropertyDescriptor(ledger, 'db').value was the raw database	minor hardening	close is a no-op in the guarded database, the descriptor trap returns the guarded one, and the view refuses defineProperty, delete, setPrototypeOf and preventExtensions, so nothing a consumer does to it reaches the ledger the router holds	V2, V3
Finding 4, half fixed: what a superseded owner uploaded after a forget was masked by the floor but never deleted (sweepMaskedCopies had no caller)	minor (privacy)	Opening a tenant that was ever forgotten starts the sweep of dead epochs beside the open (a slow store cannot use up the lease before the first renewal; a failed sweep is retried at the next open). For a tenant never opened again the sweep stays an operator step, now stated exactly	S1, S2
Finding 8: forgetTenant deleted the root the caller named, recursively; a durable delete by a prefix that stops inside a name took its neighbours (acme took acme2)	minor hardening	Every epoch-copies call accepts only <state root>/orgs/<org>/tenants/<tenant> for exactly that tenant, and uses the resolved path it checked, not the string it was given (a .. behind a symlink could lead elsewhere); deletePrefix takes whole segments, and the conformance suite requires it of every binding, with a broken store that shows the check can fail	P1, P2, P4, conformance
Finding 9, drill coverage: the writer only used handle.run, SIGTERM landed outside any write, the SIGSTOP refusals and the per-link check were not asserted	test gap	A consumer writes through the ledger it holds, inside a runAsync bracket the drain waits for, while its replica hands off over a slow object store, then is refused with the draining error; the refusals after the freeze and the number of links checked are asserted	drill (1826 assertions)
Finding 10, docs that overstate	docs	Corrected in both languages (RFC §4.2 item 6, the forget bullet and the §4.6 shared-file row, security design S-3/S-6/S-12, code map, architecture D-15, capacity baseline)	P3 pins the §4.6 shared-file row

Two original findings are decisions, not code, and the docs now say so: finding 5 (an abandoned or lost tenure keeps its local ledger only until the next open of that tenant on the same replica, whose restore replaces it with the durable copy; L2 pins it, a quarantine would be a retention feature) and finding 6 (forgetting one subject inside a live tenant is not propagated into snapshots already published; a restore may fall back to an older verified image; A3 needs a retention policy and a rule against falling back across a forget).

Public-contract check

I found nothing that needed the founder before coding, and say so here so you can overrule it:

• booking.surface schema, event kinds and the error-code list are unchanged. The A2 surface already treats every TenantStateError as a tenant fault; DrainTimeoutError comes only out of handoff() and close(), never out of a request path.
• State-layer additions: the DrainTimeoutError class, the drain_timeout router event, the options drainTimeoutMs and closeConcurrency.
• Behavior that a caller can see: run() now throws for a callback that returns a promise (a misuse that used to escape the bracket); deletePrefix rejects a prefix that does not end with / (the S3 binding has to pass the same conformance check); close() reports unflushed tenants in the order they were opened, as before.
• Severity today: the router is not wired into any runtime path on main (A3 composes it).

Evidence

Local Node v26.9.0, macOS arm64. SHA 22855e6e4ff3532b4be739cb9cbd6eb76f2e024a for the local runs below. CI on that head failed on Node 22 and 24 at one check only, P3 (statSync(file).ino was the same before and after the restore: Linux reuses an inode the moment a file is deleted, macOS did not show it); f904528 replaces the inode comparison with a held descriptor and gives D6 and R1 more room. Nothing else of this PR changed in it.

• Failing first: on main D1, D2, D3 (all three), R1, T1, V2, V3, F3, F4, F5, F6, D5, S1, P1, P2 and the new durable conformance check failed, each for the reason in the table. After the review of this branch, F7, F8, D5 (the forget on an abandoning handle), P4 and V3 (prototype, freeze) failed on the branch as it stood before the fix; F7 shows the successor's image gone from the store. D4, D6, S2, L2 and P3 pin behavior that already held. P1, P2 and P4 were run with sandboxed paths only.
• npx tsc --noEmit clean; node scripts/check-docs-i18n.mjs (94 pairs) and check-doc-readability.mjs OK.
• Full regression on this SHA, clean tree: NPM_CONFIG_USERCONFIG=/dev/null NPM_CONFIG_REGISTRY=https://registry.npmjs.org/ npm_config_registry=https://registry.npmjs.org/ bash scripts/run-all-tests.sh -> ALL SUITES GREEN, exit 0, no ENOSPC. CI runs it again on Node 22 and 24.
• New and changed suites: §101 tenant-state-handoff-tests.ts 12 -> 37 checks; §90 copies 27 (K2 now also shows the whole-segment check fails on a store whose delete is a raw prefix); the drill 1962 assertions over 2 SIGKILLs, 1 SIGSTOP and 1 graceful handoff.
• Mutations (cd ts && npx tsx scripts/mutation-check.ts scripts/tenant-state-handoff-mutations.json, with main's hardened runner): 48 of 48 killed, none by a hang or a broken edit: 47 on a3a3e1a, and J41 again on 22855e6 after V3 was strengthened (the only change since is that test). The first run found J41 alive; the cause was a check that did not look at the ledger after a refused freeze, not a redundant guard
• An independent read-only review of the diff before this PR (a fresh agent that had not seen my reasoning, told to run nothing): no blocker, one real defect and nine smaller points, each checked by me before I changed anything. The defect is the forgetTenant wipe above. Taken: the sweep moved out of the open; the checked root is the deleted root and discarding the ledger takes the guard; a forget is refused on a closing handle; close() waits for a forget; the view also refuses setPrototypeOf and preventExtensions; a check that would deadlock instead of fail (F6); guards nothing pinned (the renewal flag, the option validation, the swallowed sweep failure, the conformance check); tighter timing margins (H4, F4, D2, D3, the drill's refusal check); P3's title that claimed more than it shows. Answered, not changed: bounding the flush and the forget (below).

Not in this PR, and limits

• The defaults are unreviewed, like the others: a 120 s drain deadline and eight tenants at a time. Set drainTimeoutMs below the termination grace period of the deployment; the process stays alive while a drain waits, on purpose, because a graceful stop that gave up the moment nothing else was pending would exit without flushing.
• Closing 500 tenants took 13.2 s eight at a time against 11.3 s one at a time earlier (a busier machine; the work is building snapshots in one process): concurrency does not make a full replica drain faster, it limits what a stuck tenant can hold back. The capacity baseline says so.
• Only the drain is bounded. A durable store call that never returns holds the tenant, because the lease is renewed while it waits, and abandon() is a no-op during a forget; the contract that every call settles is written into DurableStore, the RFC and close(), and the S3 binding of A3 has to give every request a deadline. Worst case for close() is the number of stuck tenants divided by eight, rounded up, times the drain deadline.
• The late upload of a superseded owner is deleted at the next open of that tenant; for one that is never opened again, an operator runs sweepMaskedCopies once the previous owner's lease has run out. There is no scheduler for that.
• The Linux path is still reasoned from the runtime source, not run; CI covers Node 22 and 24.
• The ten minor findings of the review of feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) #683 are in fix(aaas): close the review findings on the credential broker, the process governor and the executor #687 and a later PR, not here.

Review asks

1. The 120 s default of drainTimeoutMs, and eight at a time for close().
2. run() throwing on a callback that returns a promise. A2 does not do it, but any other consumer that did would now get a TypeError instead of an escaped write.
3. Finding 6 left to A3 as a documented limit: a restore may fall back across a forget.