Skip to content

fix(aaas): a graceful handoff loses no acknowledged write, and the guards around it hold (A1 follow-up) - #684

Merged
Danceiny merged 16 commits into
mainfrom
fix/aaas-a1-handoff-correctness
Oct 31, 2026
Merged

Danceiny merged 16 commits into
mainfrom
fix/aaas-a1-handoff-correctness

Conversation

@Danceiny

Copy link
Copy Markdown
Owner

Refs #674

A graceful handoff could lose a write it had acknowledged. An independent, read-only review of the A1 state layer (run from the A2 session on the cumulative tip) found it, with six smaller findings next to it. I did not take the report on trust: I wrote a failing check for each finding from its description, saw it fail on the unfixed code, then fixed it. All of them reproduced.

Base is feat/aaas-a2-identity-seam, the cumulative tip, so this lands with the stack from the top like #683. It changes tenant-router.ts and epoch-copies.ts (A1) and nothing in the kernel (kernel-manifest-gate: zero diff), adds no dependency, and touches no WriteGate or M5 behavior.

What was wrong, and what changed

Finding Severity Fix Pinned by
A write made through handle.ledger after the final image was taken was acknowledged, then lost: the drain only blocked run and runAsync, and the view checked the lease alone blocker The view closes the moment the drain is over and the image is about to be taken: every call gets the typed TenantDrainingError until the tenant is closed, or serving again if the flush fails H1, H2
handoff() stopped renewing the lease at its start, so a drain or an upload longer than the lease failed FENCED or flushed into a prefix no successor reads blocker Only the periodic snapshots stop; renewals continue through the drain and the upload H4 (real timers)
A flush that landed after the lease lapsed was reported as flushed (also when a successor took over mid-upload and pinned a base below it) blocker publishSnapshot checks the lease once the upload landed; a late image is a failed snapshot and handoff() rejects instead of claiming flushed H3, H3b
exec(), defaultSafeIntegers() and the other methods that return the database itself handed out the raw one, behind no gate major The guarded database returns itself V1
forget() with an upload in flight left the tenant's image in the store, where nothing would look for it major forget waits for the upload under way, then deletes; a snapshot queued behind it refuses to start F1, F2
A replica kept the full local ledger after a handoff that had flushed everything major A flushed handoff removes the ledger files. An abandoned or lost tenure keeps its file on purpose: it may hold the only copy of unflushed writes L1
An onEvent observer that threw inside openTenant left the lease held and unrenewed for the TTL minor The router's observer calls are contained O1

Two more checks guard the fix itself: a handoff that fails and resumes service does not start a second renewal chain (H5), and a handoff overtaken by an abandon stays abandoned instead of bringing the handle back (H6).

Public-contract check

I found nothing that needed the founder before coding, and say so here so you can overrule it:

  • booking.surface schema, event kinds and the error-code list: unchanged. TenantHandle's shape is unchanged.
  • Behavior that a caller can see: during the final flush a call through handle.ledger gets TenantDrainingError (a tenant fault: the A2 surface already turns that into a cut connection or a typed 503, never an acknowledgement), and handoff() can now reject where it used to resolve, when the lease lapsed during the upload. router.close() already reports a failed flush as unflushed.
  • Severity today: the router is not wired into any runtime path on main, and the A2 tenancy is not mounted (A3 composes it), so landing feat(aaas): epoch-scoped durable copies — publish, lineage restore, verify, forget (A1.2) #679 before this fix exposes nothing in production. The guarantee has to hold before A3 composes it, and the docs claimed it did.

Evidence

Local Node v26.9.0, macOS arm64. SHA 835dc3336880e6df4c20bd45ff98019a0c4ab35f.

  • Failing first: on the unfixed tip H1, H2, H3, H4, V1, F1, O1 and L1 failed, each for the reason in the table (H5 and F2 passed there and stay as guards; H3b and H6 were added with the fix).
  • npx tsc --noEmit clean; kernel-manifest-gate OK (zero diff); node scripts/check-docs-i18n.mjs (94 pairs) and check-doc-readability.mjs OK.
  • Full regression on this SHA, clean tree: NPM_CONFIG_USERCONFIG=/dev/null NPM_CONFIG_REGISTRY=https://registry.npmjs.org/ npm_config_registry=https://registry.npmjs.org/ bash scripts/run-all-tests.sh -> ALL SUITES GREEN, exit 0, no ENOSPC (the public registry is used because the internal mirror is unreachable from this machine). CI runs it again on Node 22 and 24.
  • New: run-all §101 tenant-state-handoff-tests.ts, 12 checks. The existing state-layer suites pass unchanged: §89 28 checks, §90 27, §91 21, the §92 real-process drill (kills, a pause past the lease, a graceful handoff), and the A2 adversarial suite §94 (34).
  • Mutations (cd ts && npx tsx scripts/mutation-check.ts scripts/tenant-state-handoff-mutations.json): 13 killed, 0 survived. They break, one at a time: the view staying open, its closed-state check, renewals stopping at handoff, the raw database escaping, forget not waiting, a snapshot starting on a closed handle, a flushed handoff leaving its file, an abandoned file being deleted, a second renewal chain, a throwing observer, a handoff overtaken by an abandon, a closed handle coming back, and the missing post-upload lease check.

Docs

RFC §4.2 item 6 describes the handoff as it now behaves; the evidence cells of S-6 and S-12 say what is proven and what an operator still has to sweep; a code map row; run-all §101. Both languages.

Not in this PR, and limits

  • Durability is still D-a: a crash loses the writes after the last snapshot (RPO above zero). Only the graceful path claims no loss.
  • A replica that crashed or lost its lease keeps its local ledger file. Sweeping those is an operator step (A3).
  • H4 and H5 use real timers (about 1.5 s each); the rest run on fake clocks.

Review asks

  1. The view is closed to reads as well as writes once the final image is being taken. A writes-only gate is not possible at the database level, because the ledger's methods use their own closed-over handle; the A2 surface reads nothing through the ledger after its decision bracket, and all A2 suites pass.
  2. Deleting the local ledger files after a flushed handoff, while keeping them after an abandon or a lost lease.
  3. forget waiting for an upload under way instead of cancelling it.

huangzhen.0505 and others added 15 commits October 28, 2026 10:57
…er session (A1.1)

A1 must prove a single writer per tenant before any durable copy or router exists, so the lease protocol and its fencing come first, outside the kernel and not yet wired into a runtime path.

Refs #674
…erify, forget (A1.2)

A lease alone cannot stop a paused owner's late upload from being restored, so copies are scoped by epoch and each new owner pins the exact snapshot it started from. Outside the kernel and not yet wired into a runtime path.

Refs #674
…andoff, forget (A1.3)

Every route to a tenant's ledger has to prove the lease, so the router hands out only a guarded handle, and a graceful handoff flushes before it releases the lease. Outside the kernel and not yet wired into a runtime path.

Refs #674
…(A1.4)

A1 exits only with evidence from real processes under kills, a pause past the lease and a graceful handoff, so the drill compares each tenant's final ledger with what the replicas logged, and the RFC maps every #275 bullet to a passing suite. It also pins the one window mode D-a cannot close. Outside the kernel; engineering evidence only.

Refs #674
…er cannot close the router's ledger

While a handoff drains, this replica still holds the lease, so opening the tenant would only report it held by itself; it now says it is draining. The router owns the ledger's lifecycle, so close on the guarded view does nothing.

Refs #674
The RFC said restore reads the prefix of the highest epoch the lease store records; what A1.2 builds is the newest verified snapshot of the newest lineage, pinned by the new owner's base, so the two now say the same thing.

Refs #674
Keys pad the epoch to eight digits but the restore regex matched exactly eight, so a loop that burned a hundred million epochs would have hidden every copy; it now matches eight or more.

Refs #674
…tas (A2.1)

One function derives a ledger tenant from an authenticated caller and returns an opaque value that every quota API requires, so no string reaches a tenant-taking call. The meter counts concurrent turns, turn rate, LLM tokens and cost priced by the frozen ADR-20 table (its loader moves verbatim into src so the nightly evidence and the meter share it), bridge job rate and ledger size, and refuses with a typed error before any lease, file or planner is touched.

Refs #674
…suite (A2.2)

A request is authenticated, placed in its tenant at the seam, admitted against its quotas before the body is read, and served from that tenant's guarded ledger, with planner sessions and decision flights keyed by tenant; the shared single-ledger form is unchanged. The suite drives it over real HTTP with forged tenant fields, colliding ids, path-shaped claims, foreign handles, injected model output, a lost lease and a flood, and a mutation list breaks each guard in a scratch clone.

Refs #674
A job, claim, result, event, pacing entry and client now belong to one organization and job matching requires equality, so one organization's extension can never claim another's job or have its result accepted; a result for a foreign job is answered like an unknown one and writes nothing. A legacy bridge.db migrates in place under the default organization, and the mounted module keeps its single lane until a host supplies the caller organization.

Refs #674
…and code map

Run-all gains the three A2 suites (§93 quotas and metering, §94 seam and surface, §95 bridge organization); the RFC records what was built and where it is proven, and the threat rows S-1 to S-5, S-9 to S-11, S-13 and S-14 move to mitigated only where a passing falsification test exists, with the S-2, S-5, S-10 and S-11 limits stated beside them.

Refs #674
Three checks slept a fixed 50 to 100 ms before asserting that a request had reached the planner or a slot was held; on a slower runner that is a race, so they now poll for the state they need and fail with a named timeout.

Refs #674
…tenants a replica opens (S-16 to S-19) (#683)

* feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19)

The state layer isolated what a tenant stores, but the tenants of one replica still shared what a turn runs on and with: CPU, memory, file descriptors, the planner's LLM key, the provider's prompt cache and the executor's host browser. Each now has a control with a falsification test. A planner process tree runs under ceilings (V8 heap, sampled tree RSS and CPU, a cap on an unterminated line) and a kill ends the turn as a non-retryable PLANNER_FAILED; a replica can cap the tenants it opens; planner children get a short-lived loopback ticket from a credential broker instead of the key, and measured usage is charged to the tenant whose invocation asked; every tenant prompt carries an HMAC cache-partition label ahead of any tenant-derived byte; the executor's host browser serves only the organization whose supplier login it holds. A capacity probe measures what open tenants, takeovers and a booted planner cost. Without the new options the shared form is unchanged.

Refs #674

* docs(aaas): execution isolation — RFC §4.8, threat rows S-16 to S-19 and a measured capacity baseline

The RFC said the blast radius is per tenant without saying that only state is; it now narrows that claim, lists what the tenants of a replica still share and the control and evidence for each, states the limits plainly, and writes the entry criteria for the two things that would need a real sandbox (the hosted conversational agent and a multi-tenant cloud browser). The threat model gains rows S-16 to S-19 and invariants 8 to 11, and the first measured numbers for the fallback trigger (descriptors, memory, takeover time, a booted planner) are recorded with what they do not show, in both languages.

Refs #674

* test(aaas): follow the executor's request shape in its mutation list

Two executor mutations named a line that the single-organization form no longer has once an observer's request keeps its old shape, so the runner stopped; they now break the conditional that builds the request, and the list is back to 11 killed, 0 survived.

Refs #674

* refactor(aaas): keep one guard where the broker keeps errors out of the meter

An upstream error was kept out of the usage meter twice, once by not creating the tap and once by relaying error bodies down a separate bounded branch, so the first guard could be broken without any test noticing; the tap is now always created and the branch is the single guard, which two mutations of the credential list break and the suite catches. The list follows the code (32 killed, 0 survived).

Refs #674

---------

Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>
…ards around it hold (A1 follow-up)

An independent review of the A1 state layer showed that a write made through the ledger after the final image had been taken was acknowledged and then lost, that handoff stopped renewing the lease at its start so a drain or an upload longer than the lease flushed into a prefix no successor reads, and that a flush which landed after the lease lapsed was reported as flushed. The ledger view now closes when the drain is over and the image is about to be taken, the lease is renewed through the drain and the upload, and an upload that landed after the lease lapsed is reported as failed. Also fixed: the guarded database handed out the raw one from exec and its siblings, forget could leave an upload's image in the store, an observer that threw inside openTenant left the lease held, and a replica kept its local ledger after a handoff that flushed everything. Each is pinned by a check that fails without the fix (13 mutations, all killed).

Refs #674
…vidence

The RFC's description of a graceful handoff, the evidence cells of threats S-6 and S-12 and the code map now say what the follow-up made true, including that an abandoned or lost tenure keeps its local file because it may hold the only copy of unflushed writes.

Refs #674
Base automatically changed from feat/aaas-a2-identity-seam to feat/aaas-a1-fault-injection October 31, 2026 08:22
Base automatically changed from feat/aaas-a1-fault-injection to feat/aaas-a1-epoch-copies October 31, 2026 08:23
Base automatically changed from feat/aaas-a1-epoch-copies to main October 31, 2026 08:23
main took the stack as squashes, so the two histories diverged and every file of it conflicted as add/add. The tree of this merge is main's plus this PR's own delta and nothing else: the diff against main has the same files and line counts as the PR's two commits had against the stack they were written on, and the four docs where main's later edits touched the same table rows were merged row by row.
@Danceiny
Danceiny merged commit 8cdc9af into main Oct 31, 2026
8 checks passed
@Danceiny
Danceiny deleted the fix/aaas-a1-handoff-correctness branch October 31, 2026 09:00
Danceiny pushed a commit that referenced this pull request Oct 31, 2026
#684 and #685 landed on main while this was open; they edit the table rows next to the ones this PR edits (the code map), so those two files were merged row by row.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant