Repository navigation
feat(aaas): epoch-scoped durable copies — publish, lineage restore, verify, forget (A1.2) - #679
Merged
Merged
Conversation
added 2 commits
October 28, 2026 10:57
…er session (A1.1) A1 must prove a single writer per tenant before any durable copy or router exists, so the lease protocol and its fencing come first, outside the kernel and not yet wired into a runtime path. Refs #674
…erify, forget (A1.2) A lease alone cannot stop a paused owner's late upload from being restored, so copies are scoped by epoch and each new owner pins the exact snapshot it started from. Outside the kernel and not yet wired into a runtime path. Refs #674
…andoff, forget (A1.3) (#680) Every route to a tenant's ledger has to prove the lease, so the router hands out only a guarded handle, and a graceful handoff flushes before it releases the lease. Outside the kernel and not yet wired into a runtime path. Refs #674 Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>
A1.1 reached main as a squash, so this branch and main both added the same files with different histories. The six files that conflicted (the code map in both languages, run-all, errors.ts, tenant-lease.ts and the A1.1 suite) take this branch's side, which is main's A1.1 content plus A1.2 and A1.3: for each, main's change equals the A1.1 commit's change, so main adds nothing here. The merge changes exactly the files #677 changed (the RFC acceptance and its reconciliations) and nothing else; the RFC merged without conflict. Refs #674
…(A1.4) (#681) * feat(aaas): tenant router and tenant-bound handle — guarded ledger, handoff, forget (A1.3) Every route to a tenant's ledger has to prove the lease, so the router hands out only a guarded handle, and a graceful handoff flushes before it releases the lease. Outside the kernel and not yet wired into a runtime path. Refs #674 * test(aaas): multi-process fault-injection drill and the #275 closure (A1.4) A1 exits only with evidence from real processes under kills, a pause past the lease and a graceful handoff, so the drill compares each tenant's final ledger with what the replicas logged, and the RFC maps every #275 bullet to a passing suite. It also pins the one window mode D-a cannot close. Outside the kernel; engineering evidence only. Refs #674 * fix(aaas): refuse opening a tenant that is being handed off; a consumer cannot close the router's ledger While a handoff drains, this replica still holds the lease, so opening the tenant would only report it held by itself; it now says it is draining. The router owns the ledger's lifecycle, so close on the guarded view does nothing. Refs #674 * docs(aaas): RFC 4.2 item 3 states the restore rule the code implements The RFC said restore reads the prefix of the highest epoch the lease store records; what A1.2 builds is the newest verified snapshot of the newest lineage, pinned by the new owner's base, so the two now say the same thing. Refs #674 * fix(aaas): find durable copies of epochs past eight digits Keys pad the epoch to eight digits but the restore regex matched exactly eight, so a loop that burned a hundred million epochs would have hidden every copy; it now matches eight or more. Refs #674 * feat(aaas): identity at the seam, quotas and metering, and the adversarial suite (A2) (#682) * feat(aaas): seam-derived tenants and per-tenant, per-organization quotas (A2.1) One function derives a ledger tenant from an authenticated caller and returns an opaque value that every quota API requires, so no string reaches a tenant-taking call. The meter counts concurrent turns, turn rate, LLM tokens and cost priced by the frozen ADR-20 table (its loader moves verbatim into src so the nightly evidence and the meter share it), bridge job rate and ledger size, and refuses with a typed error before any lease, file or planner is touched. Refs #674 * feat(aaas): multi-tenant Booking Copilot surface and its adversarial suite (A2.2) A request is authenticated, placed in its tenant at the seam, admitted against its quotas before the body is read, and served from that tenant's guarded ledger, with planner sessions and decision flights keyed by tenant; the shared single-ledger form is unchanged. The suite drives it over real HTTP with forged tenant fields, colliding ids, path-shaped claims, foreign handles, injected model output, a lost lease and a flood, and a mutation list breaks each guard in a scratch clone. Refs #674 * feat(aaas): organization dimension in the session bridge (A2.3) A job, claim, result, event, pacing entry and client now belong to one organization and job matching requires equality, so one organization's extension can never claim another's job or have its result accepted; a result for a foreign job is answered like an unknown one and writes nothing. A legacy bridge.db migrates in place under the default organization, and the mounted module keeps its single lane until a host supplies the caller organization. Refs #674 * docs(aaas): wire A2 into run-all and reconcile the RFC, threat model and code map Run-all gains the three A2 suites (§93 quotas and metering, §94 seam and surface, §95 bridge organization); the RFC records what was built and where it is proven, and the threat rows S-1 to S-5, S-9 to S-11, S-13 and S-14 move to mitigated only where a passing falsification test exists, with the S-2, S-5, S-10 and S-11 limits stated beside them. Refs #674 * test(aaas): wait for state, not for time, in the adversarial suite Three checks slept a fixed 50 to 100 ms before asserting that a request had reached the planner or a slot was held; on a slower runner that is a race, so they now poll for the state they need and fail with a named timeout. Refs #674 * feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) (#683) * feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) The state layer isolated what a tenant stores, but the tenants of one replica still shared what a turn runs on and with: CPU, memory, file descriptors, the planner's LLM key, the provider's prompt cache and the executor's host browser. Each now has a control with a falsification test. A planner process tree runs under ceilings (V8 heap, sampled tree RSS and CPU, a cap on an unterminated line) and a kill ends the turn as a non-retryable PLANNER_FAILED; a replica can cap the tenants it opens; planner children get a short-lived loopback ticket from a credential broker instead of the key, and measured usage is charged to the tenant whose invocation asked; every tenant prompt carries an HMAC cache-partition label ahead of any tenant-derived byte; the executor's host browser serves only the organization whose supplier login it holds. A capacity probe measures what open tenants, takeovers and a booted planner cost. Without the new options the shared form is unchanged. Refs #674 * docs(aaas): execution isolation — RFC §4.8, threat rows S-16 to S-19 and a measured capacity baseline The RFC said the blast radius is per tenant without saying that only state is; it now narrows that claim, lists what the tenants of a replica still share and the control and evidence for each, states the limits plainly, and writes the entry criteria for the two things that would need a real sandbox (the hosted conversational agent and a multi-tenant cloud browser). The threat model gains rows S-16 to S-19 and invariants 8 to 11, and the first measured numbers for the fallback trigger (descriptors, memory, takeover time, a booted planner) are recorded with what they do not show, in both languages. Refs #674 * test(aaas): follow the executor's request shape in its mutation list Two executor mutations named a line that the single-organization form no longer has once an observer's request keeps its old shape, so the runner stopped; they now break the conditional that builds the request, and the list is back to 11 killed, 0 survived. Refs #674 * refactor(aaas): keep one guard where the broker keeps errors out of the meter An upstream error was kept out of the usage meter twice, once by not creating the tap and once by relaying error bodies down a separate bounded branch, so the first guard could be broken without any test noticing; the tap is now always created and the branch is the single guard, which two mutations of the credential list break and the suite catches. The list follows the code (32 killed, 0 survived). Refs #674 --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com> * fix(aaas): a planner error that meets a lost lease ends one request, never the replica; zero-token usage is no measurement An independent review reproduced a replica crash (a planner error that coincides with a lost lease escaped the request and ended the process under the standalone listener), a zero-usage report that switched the LLM budget off, and quota accounts that were created for refused tenants and never swept. The tenant form's types also lost the shared form's compile-time key requirement. * fix(aaas): bridge pacing is per organization, an older binary's index is dropped on every open, a malformed admission cannot crash the process The in-process pacing gate of the search was shared across organizations (a 429 that also told one organization when another last searched), the cdp transport could still drive the host Chrome for any organization, and a throwing-free but malformed admission return escaped as an unhandled rejection. * test(aaas): a mutation is killed only by a behavioural failure, and its target must stay inside the clone A hang, or an edit that does not load, was reported as killed; a mutation file could point outside the scratch clone, and a failed run leaked its clone. * docs(aaas): state what the A2 guards do and do not cover after the independent review S-9 holds by convention, the 429 ordering depends on the dimension, the migration is one way, admission and pacing coverage are named, and the mutation lists are run by hand. --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com> --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #674
A1.2 of the AaaS charter. A lease alone cannot stop a paused owner's late upload from being restored, so durable copies are scoped by epoch and every new owner pins the exact snapshot it started from. It lives outside the kernel and is not wired into a runtime path.
state-ledger.tsand the kernel manifest are untouched, and no dependency is added.Stacked on #678 (the base of this PR is its branch, so the diff shows only A1.2). When #678 merges I rebase onto
mainand retarget; there is nothing to decide in between.What it adds (
ts/src/tenant-state/)DurableStorecontract and a reference directory store. Create-if-absent, read, list and delete by prefix; an object never changes once it exists. Keys cannot leave the store or touch its scratch area. Conformance checks for any binding, which two deliberately broken stores turn red; the S3-compatible binding of Q4 must pass them.-walis needed) only into the prefix of its own epoch, and a manifest written last marks it complete. The lease is checked before the work and again right before the upload.base.jsonbefore it serves a write. A snapshot a superseded owner uploads after the pin sits above it and is never read. A corrupt newest snapshot is passed over for the previous one and the skip is reported; if every copy is corrupt the restore fails loudly and pins nothing. Local files from an earlier tenure are always discarded: only the durable copy is authoritative.Limits, stated rather than hidden
SIGSTOP) can still upload between its successor acquiring the lease and the successor pinning, and that snapshot would be restored. The window lasts about as long as the restore. D-b (the epoch-conditional append the RFC requires before M5) closes it. A1.4 pins it in a test with a note on how to flip it.Evidence on
7eb1dcb6aab3f83bc0f4c36978eceba0b425b92a(local Node v26.9.0)cd ts && npx tsx scripts/tenant-state-copies-tests.ts: exit 0, 25 checks (run-all §90). It covers publish, verification, lineage restore, a paused owner's late upload, corrupt copies, a foreign-tenant image with a valid checksum, a stale local file on the same node, forget and the sweep, a store outage and a crash between an image and its manifest.tsc --noEmit,smoke.ts,g5-guard.ts,kernel-manifest-gate.ts: exit 0 (kernel manifest zero diff). The A1.1 suite still passes with the shared fixtures.scripts/run-all-tests.shon this commit:ALL SUITES GREEN, exit 0. CI runs it again on Node 22 and 24.Not in this PR. The router and handoff are A1.3. Multi-process fault injection and the #275 closure are A1.4. Minimal E2E: N/A, since nothing is wired into a runtime path.
Review ask. Nothing needs a decision. The on-disk layout (
<org>/<tenant>/e<epoch>/base.json,snap-<n>.db,snap-<n>.json) is what an S3 binding would have to reproduce, so it is worth a look.