Repository navigation
feat(aaas): tenant router and tenant-bound handle — guarded ledger, handoff, forget (A1.3) - #680
Merged
Conversation
…andoff, forget (A1.3) Every route to a tenant's ledger has to prove the lease, so the router hands out only a guarded handle, and a graceful handoff flushes before it releases the lease. Outside the kernel and not yet wired into a runtime path. Refs #674
Danceiny
added a commit
that referenced
this pull request
Oct 31, 2026
…erify, forget (A1.2) (#679) * feat(aaas): tenant ownership core — epoch-fenced lease store and holder session (A1.1) A1 must prove a single writer per tenant before any durable copy or router exists, so the lease protocol and its fencing come first, outside the kernel and not yet wired into a runtime path. Refs #674 * feat(aaas): epoch-scoped durable copies — publish, lineage restore, verify, forget (A1.2) A lease alone cannot stop a paused owner's late upload from being restored, so copies are scoped by epoch and each new owner pins the exact snapshot it started from. Outside the kernel and not yet wired into a runtime path. Refs #674 * feat(aaas): tenant router and tenant-bound handle — guarded ledger, handoff, forget (A1.3) (#680) Every route to a tenant's ledger has to prove the lease, so the router hands out only a guarded handle, and a graceful handoff flushes before it releases the lease. Outside the kernel and not yet wired into a runtime path. Refs #674 Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com> * test(aaas): multi-process fault-injection drill and the #275 closure (A1.4) (#681) * feat(aaas): tenant router and tenant-bound handle — guarded ledger, handoff, forget (A1.3) Every route to a tenant's ledger has to prove the lease, so the router hands out only a guarded handle, and a graceful handoff flushes before it releases the lease. Outside the kernel and not yet wired into a runtime path. Refs #674 * test(aaas): multi-process fault-injection drill and the #275 closure (A1.4) A1 exits only with evidence from real processes under kills, a pause past the lease and a graceful handoff, so the drill compares each tenant's final ledger with what the replicas logged, and the RFC maps every #275 bullet to a passing suite. It also pins the one window mode D-a cannot close. Outside the kernel; engineering evidence only. Refs #674 * fix(aaas): refuse opening a tenant that is being handed off; a consumer cannot close the router's ledger While a handoff drains, this replica still holds the lease, so opening the tenant would only report it held by itself; it now says it is draining. The router owns the ledger's lifecycle, so close on the guarded view does nothing. Refs #674 * docs(aaas): RFC 4.2 item 3 states the restore rule the code implements The RFC said restore reads the prefix of the highest epoch the lease store records; what A1.2 builds is the newest verified snapshot of the newest lineage, pinned by the new owner's base, so the two now say the same thing. Refs #674 * fix(aaas): find durable copies of epochs past eight digits Keys pad the epoch to eight digits but the restore regex matched exactly eight, so a loop that burned a hundred million epochs would have hidden every copy; it now matches eight or more. Refs #674 * feat(aaas): identity at the seam, quotas and metering, and the adversarial suite (A2) (#682) * feat(aaas): seam-derived tenants and per-tenant, per-organization quotas (A2.1) One function derives a ledger tenant from an authenticated caller and returns an opaque value that every quota API requires, so no string reaches a tenant-taking call. The meter counts concurrent turns, turn rate, LLM tokens and cost priced by the frozen ADR-20 table (its loader moves verbatim into src so the nightly evidence and the meter share it), bridge job rate and ledger size, and refuses with a typed error before any lease, file or planner is touched. Refs #674 * feat(aaas): multi-tenant Booking Copilot surface and its adversarial suite (A2.2) A request is authenticated, placed in its tenant at the seam, admitted against its quotas before the body is read, and served from that tenant's guarded ledger, with planner sessions and decision flights keyed by tenant; the shared single-ledger form is unchanged. The suite drives it over real HTTP with forged tenant fields, colliding ids, path-shaped claims, foreign handles, injected model output, a lost lease and a flood, and a mutation list breaks each guard in a scratch clone. Refs #674 * feat(aaas): organization dimension in the session bridge (A2.3) A job, claim, result, event, pacing entry and client now belong to one organization and job matching requires equality, so one organization's extension can never claim another's job or have its result accepted; a result for a foreign job is answered like an unknown one and writes nothing. A legacy bridge.db migrates in place under the default organization, and the mounted module keeps its single lane until a host supplies the caller organization. Refs #674 * docs(aaas): wire A2 into run-all and reconcile the RFC, threat model and code map Run-all gains the three A2 suites (§93 quotas and metering, §94 seam and surface, §95 bridge organization); the RFC records what was built and where it is proven, and the threat rows S-1 to S-5, S-9 to S-11, S-13 and S-14 move to mitigated only where a passing falsification test exists, with the S-2, S-5, S-10 and S-11 limits stated beside them. Refs #674 * test(aaas): wait for state, not for time, in the adversarial suite Three checks slept a fixed 50 to 100 ms before asserting that a request had reached the planner or a slot was held; on a slower runner that is a race, so they now poll for the state they need and fail with a named timeout. Refs #674 * feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) (#683) * feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) The state layer isolated what a tenant stores, but the tenants of one replica still shared what a turn runs on and with: CPU, memory, file descriptors, the planner's LLM key, the provider's prompt cache and the executor's host browser. Each now has a control with a falsification test. A planner process tree runs under ceilings (V8 heap, sampled tree RSS and CPU, a cap on an unterminated line) and a kill ends the turn as a non-retryable PLANNER_FAILED; a replica can cap the tenants it opens; planner children get a short-lived loopback ticket from a credential broker instead of the key, and measured usage is charged to the tenant whose invocation asked; every tenant prompt carries an HMAC cache-partition label ahead of any tenant-derived byte; the executor's host browser serves only the organization whose supplier login it holds. A capacity probe measures what open tenants, takeovers and a booted planner cost. Without the new options the shared form is unchanged. Refs #674 * docs(aaas): execution isolation — RFC §4.8, threat rows S-16 to S-19 and a measured capacity baseline The RFC said the blast radius is per tenant without saying that only state is; it now narrows that claim, lists what the tenants of a replica still share and the control and evidence for each, states the limits plainly, and writes the entry criteria for the two things that would need a real sandbox (the hosted conversational agent and a multi-tenant cloud browser). The threat model gains rows S-16 to S-19 and invariants 8 to 11, and the first measured numbers for the fallback trigger (descriptors, memory, takeover time, a booted planner) are recorded with what they do not show, in both languages. Refs #674 * test(aaas): follow the executor's request shape in its mutation list Two executor mutations named a line that the single-organization form no longer has once an observer's request keeps its old shape, so the runner stopped; they now break the conditional that builds the request, and the list is back to 11 killed, 0 survived. Refs #674 * refactor(aaas): keep one guard where the broker keeps errors out of the meter An upstream error was kept out of the usage meter twice, once by not creating the tap and once by relaying error bodies down a separate bounded branch, so the first guard could be broken without any test noticing; the tap is now always created and the branch is the single guard, which two mutations of the credential list break and the suite catches. The list follows the code (32 killed, 0 survived). Refs #674 --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com> * fix(aaas): a planner error that meets a lost lease ends one request, never the replica; zero-token usage is no measurement An independent review reproduced a replica crash (a planner error that coincides with a lost lease escaped the request and ended the process under the standalone listener), a zero-usage report that switched the LLM budget off, and quota accounts that were created for refused tenants and never swept. The tenant form's types also lost the shared form's compile-time key requirement. * fix(aaas): bridge pacing is per organization, an older binary's index is dropped on every open, a malformed admission cannot crash the process The in-process pacing gate of the search was shared across organizations (a 429 that also told one organization when another last searched), the cdp transport could still drive the host Chrome for any organization, and a throwing-free but malformed admission return escaped as an unhandled rejection. * test(aaas): a mutation is killed only by a behavioural failure, and its target must stay inside the clone A hang, or an edit that does not load, was reported as killed; a mutation file could point outside the scratch clone, and a failed run leaked its clone. * docs(aaas): state what the A2 guards do and do not cover after the independent review S-9 holds by convention, the 429 ordering depends on the dimension, the migration is one way, admission and pacing coverage are named, and the mutation lists are run by hand. --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com> --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com> --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #674
A1.3 of the AaaS charter: the one way to a tenant's ledger. A replica asks a router for a tenant, and gets back a handle that is bound to that tenant and that proves the lease on every call. It lives outside the kernel and is not wired into a runtime path.
state-ledger.tsand the kernel manifest are untouched, and no dependency is added.Stacked on #679 (the base of this PR is its branch, so the diff shows only A1.3). When the stack below it merges I rebase onto
mainand retarget; there is nothing to decide in between.What it adds (
ts/src/tenant-state/)TenantRouter.openTenant(organization, ledgerTenant). It acquires the lease, restores the ledger from the newest verified copy, and returns a handle. Concurrent opens share one acquisition. A tenant another replica owns fails withLeaseHeldError, whose record names the owner, so the deployment layer of A3 can forward to it. An id outside the fixed format is refused before any store or file is touched.runandrunAsyncacknowledge a write only if the lease is provably held before the work and after it. The ledger the handle exposes is a guarded view: every call proves the lease first, and so does every statement, transaction and iterator made earlier, so a long-lived consumer such as a booking runtime is stopped the same way. A write applied locally but not acknowledged is harmless, because a later owner restores from the durable copy and local files never survive.A defect the tests found in my first design. I first made the guard a Proxy over the better-sqlite3 objects. A test that built a transaction while the lease held and used it after the fence showed the flaw: better-sqlite3 gives those objects read-only, non-configurable properties, among them a raw
databasehandle. A Proxy has to return those unchanged, so it either throws on access or leaks the unguarded database. The guard now builds purpose-made facades for statements and transactions and does not pass the raw database on.One deviation from the RFC, flagged. RFC §4.6 lists an epoch guard in
kvfor two replicas that share one file by mistake. I did not build it. The lease already ends a second writer, a restore replaces the file by atomic rename so the older writer is left holding an unlinked file, and a guard checked on every write would have to live in the kernel. I changed the RFC row to say what is built. If you want the detection-only version instead (a row written at open and re-read at each renewal), say so.Evidence on
70cadae026dd22a955a1242d63331ea414efd505(local Node v26.9.0)cd ts && npx tsx scripts/tenant-state-router-tests.ts: exit 0, 21 checks (run-all §91). Two of them use real timers: background renewal across more than two lease terms, and periodic snapshots.tsc --noEmit,smoke.ts,g5-guard.ts,kernel-manifest-gate.ts: exit 0, with a zero-diff kernel manifest. The A1.1 and A1.2 suites still pass.scripts/run-all-tests.shon this commit:ALL SUITES GREEN, exit 0. CI runs it again on Node 22 and 24.Not in this PR. The multi-process fault-injection drill and the #275 closure are A1.4. Minimal E2E: N/A, since nothing is wired into a runtime path.
Review ask. The deviation above is the one thing that needs your call. Everything else follows the accepted RFC.