Repository navigation
fix(aaas): a graceful handoff loses no acknowledged write, and the guards around it hold (A1 follow-up) - #684
Merged
Merged
Conversation
…er session (A1.1) A1 must prove a single writer per tenant before any durable copy or router exists, so the lease protocol and its fencing come first, outside the kernel and not yet wired into a runtime path. Refs #674
…erify, forget (A1.2) A lease alone cannot stop a paused owner's late upload from being restored, so copies are scoped by epoch and each new owner pins the exact snapshot it started from. Outside the kernel and not yet wired into a runtime path. Refs #674
…andoff, forget (A1.3) Every route to a tenant's ledger has to prove the lease, so the router hands out only a guarded handle, and a graceful handoff flushes before it releases the lease. Outside the kernel and not yet wired into a runtime path. Refs #674
…(A1.4) A1 exits only with evidence from real processes under kills, a pause past the lease and a graceful handoff, so the drill compares each tenant's final ledger with what the replicas logged, and the RFC maps every #275 bullet to a passing suite. It also pins the one window mode D-a cannot close. Outside the kernel; engineering evidence only. Refs #674
…er cannot close the router's ledger While a handoff drains, this replica still holds the lease, so opening the tenant would only report it held by itself; it now says it is draining. The router owns the ledger's lifecycle, so close on the guarded view does nothing. Refs #674
The RFC said restore reads the prefix of the highest epoch the lease store records; what A1.2 builds is the newest verified snapshot of the newest lineage, pinned by the new owner's base, so the two now say the same thing. Refs #674
Keys pad the epoch to eight digits but the restore regex matched exactly eight, so a loop that burned a hundred million epochs would have hidden every copy; it now matches eight or more. Refs #674
…tas (A2.1) One function derives a ledger tenant from an authenticated caller and returns an opaque value that every quota API requires, so no string reaches a tenant-taking call. The meter counts concurrent turns, turn rate, LLM tokens and cost priced by the frozen ADR-20 table (its loader moves verbatim into src so the nightly evidence and the meter share it), bridge job rate and ledger size, and refuses with a typed error before any lease, file or planner is touched. Refs #674
…suite (A2.2) A request is authenticated, placed in its tenant at the seam, admitted against its quotas before the body is read, and served from that tenant's guarded ledger, with planner sessions and decision flights keyed by tenant; the shared single-ledger form is unchanged. The suite drives it over real HTTP with forged tenant fields, colliding ids, path-shaped claims, foreign handles, injected model output, a lost lease and a flood, and a mutation list breaks each guard in a scratch clone. Refs #674
A job, claim, result, event, pacing entry and client now belong to one organization and job matching requires equality, so one organization's extension can never claim another's job or have its result accepted; a result for a foreign job is answered like an unknown one and writes nothing. A legacy bridge.db migrates in place under the default organization, and the mounted module keeps its single lane until a host supplies the caller organization. Refs #674
…and code map Run-all gains the three A2 suites (§93 quotas and metering, §94 seam and surface, §95 bridge organization); the RFC records what was built and where it is proven, and the threat rows S-1 to S-5, S-9 to S-11, S-13 and S-14 move to mitigated only where a passing falsification test exists, with the S-2, S-5, S-10 and S-11 limits stated beside them. Refs #674
Three checks slept a fixed 50 to 100 ms before asserting that a request had reached the planner or a slot was held; on a slower runner that is a race, so they now poll for the state they need and fail with a named timeout. Refs #674
…tenants a replica opens (S-16 to S-19) (#683) * feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) The state layer isolated what a tenant stores, but the tenants of one replica still shared what a turn runs on and with: CPU, memory, file descriptors, the planner's LLM key, the provider's prompt cache and the executor's host browser. Each now has a control with a falsification test. A planner process tree runs under ceilings (V8 heap, sampled tree RSS and CPU, a cap on an unterminated line) and a kill ends the turn as a non-retryable PLANNER_FAILED; a replica can cap the tenants it opens; planner children get a short-lived loopback ticket from a credential broker instead of the key, and measured usage is charged to the tenant whose invocation asked; every tenant prompt carries an HMAC cache-partition label ahead of any tenant-derived byte; the executor's host browser serves only the organization whose supplier login it holds. A capacity probe measures what open tenants, takeovers and a booted planner cost. Without the new options the shared form is unchanged. Refs #674 * docs(aaas): execution isolation — RFC §4.8, threat rows S-16 to S-19 and a measured capacity baseline The RFC said the blast radius is per tenant without saying that only state is; it now narrows that claim, lists what the tenants of a replica still share and the control and evidence for each, states the limits plainly, and writes the entry criteria for the two things that would need a real sandbox (the hosted conversational agent and a multi-tenant cloud browser). The threat model gains rows S-16 to S-19 and invariants 8 to 11, and the first measured numbers for the fallback trigger (descriptors, memory, takeover time, a booted planner) are recorded with what they do not show, in both languages. Refs #674 * test(aaas): follow the executor's request shape in its mutation list Two executor mutations named a line that the single-organization form no longer has once an observer's request keeps its old shape, so the runner stopped; they now break the conditional that builds the request, and the list is back to 11 killed, 0 survived. Refs #674 * refactor(aaas): keep one guard where the broker keeps errors out of the meter An upstream error was kept out of the usage meter twice, once by not creating the tap and once by relaying error bodies down a separate bounded branch, so the first guard could be broken without any test noticing; the tap is now always created and the branch is the single guard, which two mutations of the credential list break and the suite catches. The list follows the code (32 killed, 0 survived). Refs #674 --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>
…ards around it hold (A1 follow-up) An independent review of the A1 state layer showed that a write made through the ledger after the final image had been taken was acknowledged and then lost, that handoff stopped renewing the lease at its start so a drain or an upload longer than the lease flushed into a prefix no successor reads, and that a flush which landed after the lease lapsed was reported as flushed. The ledger view now closes when the drain is over and the image is about to be taken, the lease is renewed through the drain and the upload, and an upload that landed after the lease lapsed is reported as failed. Also fixed: the guarded database handed out the raw one from exec and its siblings, forget could leave an upload's image in the store, an observer that threw inside openTenant left the lease held, and a replica kept its local ledger after a handoff that flushed everything. Each is pinned by a check that fails without the fix (13 mutations, all killed). Refs #674
…vidence The RFC's description of a graceful handoff, the evidence cells of threats S-6 and S-12 and the code map now say what the follow-up made true, including that an abandoned or lost tenure keeps its local file because it may hold the only copy of unflushed writes. Refs #674
Base automatically changed from
feat/aaas-a2-identity-seam
to
feat/aaas-a1-fault-injection
October 31, 2026 08:22
Base automatically changed from
feat/aaas-a1-fault-injection
to
feat/aaas-a1-epoch-copies
October 31, 2026 08:23
main took the stack as squashes, so the two histories diverged and every file of it conflicted as add/add. The tree of this merge is main's plus this PR's own delta and nothing else: the diff against main has the same files and line counts as the PR's two commits had against the stack they were written on, and the four docs where main's later edits touched the same table rows were merged row by row.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #674
A graceful handoff could lose a write it had acknowledged. An independent, read-only review of the A1 state layer (run from the A2 session on the cumulative tip) found it, with six smaller findings next to it. I did not take the report on trust: I wrote a failing check for each finding from its description, saw it fail on the unfixed code, then fixed it. All of them reproduced.
Base is
feat/aaas-a2-identity-seam, the cumulative tip, so this lands with the stack from the top like #683. It changestenant-router.tsandepoch-copies.ts(A1) and nothing in the kernel (kernel-manifest-gate: zero diff), adds no dependency, and touches no WriteGate or M5 behavior.What was wrong, and what changed
handle.ledgerafter the final image was taken was acknowledged, then lost: the drain only blockedrunandrunAsync, and the view checked the lease aloneTenantDrainingErroruntil the tenant is closed, or serving again if the flush failshandoff()stopped renewing the lease at its start, so a drain or an upload longer than the lease failedFENCEDor flushed into a prefix no successor readspublishSnapshotchecks the lease once the upload landed; a late image is a failed snapshot andhandoff()rejects instead of claimingflushedexec(),defaultSafeIntegers()and the other methods that return the database itself handed out the raw one, behind no gateforget()with an upload in flight left the tenant's image in the store, where nothing would look for itforgetwaits for the upload under way, then deletes; a snapshot queued behind it refuses to startonEventobserver that threw insideopenTenantleft the lease held and unrenewed for the TTLTwo more checks guard the fix itself: a handoff that fails and resumes service does not start a second renewal chain (H5), and a handoff overtaken by an abandon stays abandoned instead of bringing the handle back (H6).
Public-contract check
I found nothing that needed the founder before coding, and say so here so you can overrule it:
booking.surfaceschema, event kinds and the error-code list: unchanged.TenantHandle's shape is unchanged.handle.ledgergetsTenantDrainingError(a tenant fault: the A2 surface already turns that into a cut connection or a typed 503, never an acknowledgement), andhandoff()can now reject where it used to resolve, when the lease lapsed during the upload.router.close()already reports a failed flush asunflushed.main, and the A2 tenancy is not mounted (A3 composes it), so landing feat(aaas): epoch-scoped durable copies — publish, lineage restore, verify, forget (A1.2) #679 before this fix exposes nothing in production. The guarantee has to hold before A3 composes it, and the docs claimed it did.Evidence
Local Node v26.9.0, macOS arm64. SHA
835dc3336880e6df4c20bd45ff98019a0c4ab35f.npx tsc --noEmitclean;kernel-manifest-gateOK (zero diff);node scripts/check-docs-i18n.mjs(94 pairs) andcheck-doc-readability.mjsOK.NPM_CONFIG_USERCONFIG=/dev/null NPM_CONFIG_REGISTRY=https://registry.npmjs.org/ npm_config_registry=https://registry.npmjs.org/ bash scripts/run-all-tests.sh->ALL SUITES GREEN, exit 0, no ENOSPC (the public registry is used because the internal mirror is unreachable from this machine). CI runs it again on Node 22 and 24.tenant-state-handoff-tests.ts, 12 checks. The existing state-layer suites pass unchanged: §89 28 checks, §90 27, §91 21, the §92 real-process drill (kills, a pause past the lease, a graceful handoff), and the A2 adversarial suite §94 (34).cd ts && npx tsx scripts/mutation-check.ts scripts/tenant-state-handoff-mutations.json): 13 killed, 0 survived. They break, one at a time: the view staying open, its closed-state check, renewals stopping at handoff, the raw database escaping, forget not waiting, a snapshot starting on a closed handle, a flushed handoff leaving its file, an abandoned file being deleted, a second renewal chain, a throwing observer, a handoff overtaken by an abandon, a closed handle coming back, and the missing post-upload lease check.Docs
RFC §4.2 item 6 describes the handoff as it now behaves; the evidence cells of S-6 and S-12 say what is proven and what an operator still has to sweep; a code map row; run-all §101. Both languages.
Not in this PR, and limits
Review asks
forgetwaiting for an upload under way instead of cancelling it.