Skip to content

feat(aaas): identity at the seam, quotas and metering, and the adversarial suite (A2) - #682

Merged
Danceiny merged 10 commits into
feat/aaas-a1-fault-injectionfrom
feat/aaas-a2-identity-seam
Oct 31, 2026
Merged

Danceiny merged 10 commits into
feat/aaas-a1-fault-injectionfrom
feat/aaas-a2-identity-seam

Conversation

@Danceiny

@Danceiny Danceiny commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

Refs #674

A2 of the AaaS charter: tenants derived at the trusted seam, per-tenant and per-organization quotas and metering, and the adversarial suite that tries to reach another tenant's state. It lives outside the kernel (kernel-manifest-gate: zero diff, 5/5 kernel files loaded), adds no dependency, and changes no WriteGate or M5 behavior. The shared single-ledger form of the Booking Copilot surface is unchanged, and the local product keeps tenant local.

Stacked on #681 (the base of this PR is its branch). The branch carries A2 (five commits), #683 (squash-merged into it), and five review follow-up commits described under "Independent review" below. The A1 handoff fix of the other session is #684, which targets this branch.

Where the seam sits and how a turn gets its ledger

Written before coding. bookingCopilotTrafficHandler in server.ts is the only door (the standalone listener and the gotry-backend module both mount it). In the new tenant form, after route matching and before the body is read:

  1. tenancy.authenticate(headers) returns an AuthenticatedCaller or null (401). It is the only place a credential is read.
  2. deriveSeamTenant(caller, keyring) is the only place a ledger tenant is derived (deriveLedgerTenantId, as accepted in the RFC). It returns an opaque SeamTenant that a runtime registry vouches for, so a string, a JSON look-alike or a cast is refused by every API that takes a tenant.
  3. quotas.admitTurn(tenant) answers a typed 429 before the body is read, a lease is taken or a planner starts.
  4. TenantRouter.openTenant(organization, tenant) is the only way to a ledger. The seam checks that the opened handle and its ledger are exactly the derived tenant, then builds a BookingCopilotTaskRuntime over the guarded ledger (the runtime holds no state between requests, so it is built per request and runtime.ts needs no change).
  5. Planner sessions and decision flights, the two process-wide maps keyed by task id today, get a tenant prefix. The decision work (the planner await and the ledger write) runs inside handle.runAsync: a handoff drains it, and a lease lost mid-stream cuts the connection instead of acknowledging.

Payloads, headers, tool arguments and model output are never inputs to any of this.

Public-contract check (the stop condition in the brief)

I found nothing that needed the founder before coding, and say so here so you can overrule it:

  • booking.surface schema, event kinds, version and hash headers: unchanged. No field, event or validator rule is added.
  • ADR-23 bindings: shapes unchanged. In the tenant form the principal handed to BookingIngressBinding.bind and stored as principal and scope digests is the per-request authenticated caller instead of the composition constant. ADR-23's accepted expansion already lists "tenant derived in the trusted seam, requestKey and taskHandle tenant-scoped" as the A2 addition. This is the one interpretive point.
  • HTTP: additive and only in the tenant form: 429 quota_exceeded (+ Retry-After), 503 tenant_unavailable | tenant_draining | authentication_unavailable, 403 invalid_principal | organization_not_served. The shared form is byte-identical.
  • BFF to GoTry wire format: not defined here. The authenticator is an in-process interface; A3's boundary ADR supplies the real one. So A2 alone does not let a deployment serve more than one user: this is the RFC §7 "single-organization deployment" stop point.

What it adds (commit by commit)

  • A2.1 ts/src/tenant-seam/, ts/src/llm-price-table.ts. TenantQuotas meters concurrent turns, turn rate, LLM tokens and cost, bridge job rate and ledger size per tenant and per organization, with a global concurrency cap. Cost is priced by the frozen ADR-20 table through the same code the nightly evidence uses (its loader moved verbatim into src, re-exported by nightly-evidence.ts; an unpriced model stops construction). Rolling windows on an injected clock, the narrowest scope refuses first, a refusal reserves nothing, counters are keyed by the HMAC tenant id.
  • A2.2 ts/src/booking-surface/tenancy.ts, server.ts, startup.ts, plus the adversarial suite and its test kit. Startup in the tenant form builds no shared ledger, runtime or planner. Per-tenant planners come from a TenantPlannerProvider rooted in the tenant's own directory (createDshTenantPlannerProvider). TenantRouter.tenantRootOf is a small addition to an A1 file.
  • A2.3 bridge organization dimension (threat S-5): organization column and keys in bridge-ledger.ts, equality in job matching in extension-bridge.ts, a foreign result answered like an unknown job and written nowhere, two optional hooks in session-search.ts (callerOrganization, admission). A legacy bridge.db migrates in place under local. The mounted production module stays single-organization until a host supplies the organization.
  • Docs and run-all: §93 quotas, §94 seam and surface, §95 bridge. RFC §4.4 and the §7 A2 row, threat rows S-1 to S-5, S-9 to S-11, S-13, S-14 (moved to mitigated only where a passing falsification test exists, with limits stated beside S-2, S-5, S-10, S-11), code map, in both languages.
  • ts/scripts/mutation-check.ts runs a mutation list in a scratch clone and fails if a broken guard leaves the suite green.

Independent review, and what it changed (2026-10-31)

Before merging the AaaS stack, four read-only reviewers went through it (A1 state layer #678-#681; A2 core and the shared-form regression; the bridge migration and the executor binding; #683), each finding reproduced by running code, and I re-ran the repros of the serious ones on the cumulative tip. Findings against A1 went to the session that owns it (graceful handoff can lose an acknowledged write: #684). Findings against this PR are fixed here, five commits (the last one repairs the mutation lists, which running them in full showed stale), each fix with a falsifying test and a mutation entry:

  • A planner error that meets an expired lease ended the whole replica. The error was recorded on the fenced ledger, the fault escaped the request, and the standalone listener (void handler()) had no catch. Now the answering path cuts that one connection, and the tenant form's listener contains whatever escapes a request (F11, F12). Reproduced on real processes (lease store unreachable, a model answer the planner rejects by throwing): the replica process ended before the fix, and the same process keeps serving after it.
  • A usage report of zero tokens switched the LLM budget off. It counted as a measurement; it is now unproven and charged the worst case (L2, G1).
  • Quota accounts: a refused fresh tenant no longer leaves an account, refused attempts advance the idle sweep, and a swept tenant takes its ledger size out of the organization total (I2, I4, I6).
  • Bridge pacing was shared across organizations: the in-process pacing gate of the search was keyed by site only, so organization B got 429 cooldown "last call 296ms ago" right after A began a search. That is an availability coupling and a small timing leak; the first version of this description called it "not a leak", which was wrong. The gate is now keyed by organization and site (M7).
  • Smaller: the single-column idempotency index that an older binary recreates is dropped on every open (L6); a malformed admission return is a 503 instead of an unhandled rejection (M8); a module given an organization resolver refuses the cdp transport at construction (M9); the shared form's key is a compile-time requirement again (apiKey?: never on the tenant form) and its check order is main's; the legacy file is opened with a longer busy timeout; mutation-check.ts no longer counts a hang or an edit that does not load as a kill, and refuses a path outside the clone.
  • Docs (both languages): S-9 now says "by convention" (the bridge stores whatever JSON the extension posts), the 429-before-lease wording is per dimension, the one-way migration and its rollback rule are stated, admission and pacing coverage are named, and the mutation lists are said to be run by hand.

Evidence

Local Node v26.9.0. Gates and suites below were run on 63d86c6; after it only bridge-ledger.ts (one place drops the legacy index), the mutation lists and a count in the code map changed, and for those the §95 suite, the type check and the docs checks were re-run. The mutation table is from 1bf4bfe, the head of this branch.

Gates (from ts/, exit 0 unless noted):

  • npx tsc --noEmit; npx tsx scripts/smoke.ts (SMOKE OK); npx tsx scripts/g5-guard.ts (G5 GUARD: OK); npx tsx scripts/kernel-manifest-gate.ts (zero manifest diff, 5/5 kernel files loaded).
  • Full regression on this SHA, clean tree before and after: NPM_CONFIG_USERCONFIG=/dev/null NPM_CONFIG_REGISTRY=https://registry.npmjs.org/ npm_config_registry=https://registry.npmjs.org/ bash scripts/run-all-tests.sh -> ALL SUITES GREEN, exit 0 (the public registry is used because the internal mirror is unreachable from this machine). CI runs it again on Node 22 and 24.
  • The four A1 suites still pass inside it: §89 28 checks, §90 27, §91 21, §92 2511 assertions (real processes).
  • New in it: §93 tenant-seam-quota-tests.ts 27 checks; §94 tenant-seam-adversarial-tests.ts 36 checks; §95 tenant-bridge-org-tests.ts 31 checks.
  • The seven shared-form booking suites of §49 (contract, runtime, server, startup, event-sequence and the two ledger-concurrency proofs) pass unchanged apart from one ?? '' in the startup proof, because apiKey is optional in the tenant form.
  • node scripts/check-docs-i18n.mjs (93 pairs) and node scripts/check-doc-readability.mjs: OK.

Mutation results (cd ts && npx tsx scripts/mutation-check.ts scripts/<list>.json; each edit breaks one guard in a scratch clone and the suite must go red; a clean baseline is required first; run on this SHA):

List Suite Killed Survived
tenant-seam-quota-mutations.json §93 27 0
tenant-seam-mutations.json §94 26 0
tenant-bridge-org-mutations.json §95 34 0

What the mutations break (examples): the issuer left out of the derivation; the brand check on SeamTenant; a header that names the subject; the strict body boundary; task-id-only planner session and flight keys; a handle for another tenant being served; the turn admitted after the tenant is opened; the concurrency slot never released; decision work not bracketed by the handle (a handoff stops draining); a mid-stream fence reported as an ordinary error; a quota refusal that is not a 429; an unproven invocation charged nothing; a usage report charged to whoever is running; organization equality dropped from job matching; the foreign-result check; organization filters on store updates; pacing, client and idempotency keys without the organization.

Two things the mutation runs taught me, now fixed: one assertion checked only the scope of a refusal, so a broken release survived until it also checked the dimension; and a failed check could leave a held request hanging a shutdown, so the test kit now opens every gate on close and the suites fail fast instead of timing out. One candidate mutation (the request key inside a flight key) was dropped because it breaks pre-existing logic, not a tenancy guard.

End to end on real processes (not part of the repository; run against the cumulative tip, harness and README in the shared session notes): a real dsh planner behind the credential broker and a fake OpenAI-compatible gateway; replicas are OS processes over real HTTP, a shared SQLite lease file and a durable directory. E1 a real turn through every layer; E2 forged headers, body, foreign receipt and replay; E3 SIGKILL mid-turn, takeover, replay, no orphaned planner processes; E4 SIGTERM mid-turn drains and flushes; E5 a hot tenant's flood is refused with typed 429s while a neighbour is not slowed; E6 the exec'd planner runtime holds a ticket and never the real key or gateway address; F1-F6 a legacy bridge.db written by the pre-AaaS code migrates in place, isolation across organizations over HTTP, organization quota, SIGKILL and restart, and what an old binary does to a migrated file. All pass. E10 (above) passes after the fix and ends the replica without it. E9 fails and is not this PR's: a graceful handoff over a slow durable store loses an acknowledged turn (control with a fast store passes); that is the A1 finding behind #684.

Not in this PR, and limits

  • No production authenticator, and the gotry-backend module does not take a tenancy yet (A3).
  • Quota state is per replica; an organization quota needs a shared counter once N replicas serve one organization (A3). Planner and quota accounts of a tenant that leaves a replica live until the process stops (quota idle sweep aside).
  • The dsh adapter reports no token usage today, so every planner invocation is charged the configured worst case; wiring dsh usage events needs a live dsh run (A3). Planner working-directory isolation is proven with fakes, not against a live dsh.
  • Bridge: the migration is one-way. An older binary opens a migrated file but fails its client and pacing writes, and cannot open one that two organizations used; roll back only while every row is local, otherwise restore a copy taken before the first start of the new binary. admission covers search and login/open only: /v1/session/status and the extension endpoints are not metered, and every result for an unknown job id files an orphan row, so one organization can grow the shared ledger (A3). Audit and verdict lines carry no organization. The bridge stores whatever JSON the extension posts, so S-9 holds by convention until A3 allowlists result fields.
  • Quota: a ledger-size 429 is decided after the tenant is opened (its lease and restore have happened); the per-tenant planner map is still not evicted when a tenant leaves a replica. The mutation lists are run by hand with mutation-check.ts; run-all does not run them.
  • The findings of the feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) #683 review (governor fail-open when the process table cannot be read, no connection cap on the broker's port, upstream error bodies relayed to the child) and the executor's null body crash are for the session that owns feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) #683.
  • "Not-found" for a foreign handle is the typed TASK_NOT_FOUND for a receipt; a first-seen request key or task id is simply new in the caller's own ledger and cannot be told apart from a first turn (D1, D2 and D4 pin both).

Review asks

  1. The ADR-23 principal point above.
  2. DEFAULT_QUOTA_POLICY numbers are conservative and unreviewed; nothing reads them yet.
  3. package.json files gained ts/src/tenant-seam/, ts/src/tenant-state/ and ts/src/llm-price-table.ts, so the source form of the tarball has no dangling imports (the installed form compiles all of ts/src into dist).

huangzhen.0505 added 5 commits October 28, 2026 18:23
…tas (A2.1)

One function derives a ledger tenant from an authenticated caller and returns an opaque value that every quota API requires, so no string reaches a tenant-taking call. The meter counts concurrent turns, turn rate, LLM tokens and cost priced by the frozen ADR-20 table (its loader moves verbatim into src so the nightly evidence and the meter share it), bridge job rate and ledger size, and refuses with a typed error before any lease, file or planner is touched.

Refs #674
…suite (A2.2)

A request is authenticated, placed in its tenant at the seam, admitted against its quotas before the body is read, and served from that tenant's guarded ledger, with planner sessions and decision flights keyed by tenant; the shared single-ledger form is unchanged. The suite drives it over real HTTP with forged tenant fields, colliding ids, path-shaped claims, foreign handles, injected model output, a lost lease and a flood, and a mutation list breaks each guard in a scratch clone.

Refs #674
A job, claim, result, event, pacing entry and client now belong to one organization and job matching requires equality, so one organization's extension can never claim another's job or have its result accepted; a result for a foreign job is answered like an unknown one and writes nothing. A legacy bridge.db migrates in place under the default organization, and the mounted module keeps its single lane until a host supplies the caller organization.

Refs #674
…and code map

Run-all gains the three A2 suites (§93 quotas and metering, §94 seam and surface, §95 bridge organization); the RFC records what was built and where it is proven, and the threat rows S-1 to S-5, S-9 to S-11, S-13 and S-14 move to mitigated only where a passing falsification test exists, with the S-2, S-5, S-10 and S-11 limits stated beside them.

Refs #674
Three checks slept a fixed 50 to 100 ms before asserting that a request had reached the planner or a slot was held; on a slower runner that is a race, so they now poll for the state they need and fail with a named timeout.

Refs #674
Danceiny and others added 5 commits October 31, 2026 10:54
…tenants a replica opens (S-16 to S-19) (#683)

* feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19)

The state layer isolated what a tenant stores, but the tenants of one replica still shared what a turn runs on and with: CPU, memory, file descriptors, the planner's LLM key, the provider's prompt cache and the executor's host browser. Each now has a control with a falsification test. A planner process tree runs under ceilings (V8 heap, sampled tree RSS and CPU, a cap on an unterminated line) and a kill ends the turn as a non-retryable PLANNER_FAILED; a replica can cap the tenants it opens; planner children get a short-lived loopback ticket from a credential broker instead of the key, and measured usage is charged to the tenant whose invocation asked; every tenant prompt carries an HMAC cache-partition label ahead of any tenant-derived byte; the executor's host browser serves only the organization whose supplier login it holds. A capacity probe measures what open tenants, takeovers and a booted planner cost. Without the new options the shared form is unchanged.

Refs #674

* docs(aaas): execution isolation — RFC §4.8, threat rows S-16 to S-19 and a measured capacity baseline

The RFC said the blast radius is per tenant without saying that only state is; it now narrows that claim, lists what the tenants of a replica still share and the control and evidence for each, states the limits plainly, and writes the entry criteria for the two things that would need a real sandbox (the hosted conversational agent and a multi-tenant cloud browser). The threat model gains rows S-16 to S-19 and invariants 8 to 11, and the first measured numbers for the fallback trigger (descriptors, memory, takeover time, a booted planner) are recorded with what they do not show, in both languages.

Refs #674

* test(aaas): follow the executor's request shape in its mutation list

Two executor mutations named a line that the single-organization form no longer has once an observer's request keeps its old shape, so the runner stopped; they now break the conditional that builds the request, and the list is back to 11 killed, 0 survived.

Refs #674

* refactor(aaas): keep one guard where the broker keeps errors out of the meter

An upstream error was kept out of the usage meter twice, once by not creating the tap and once by relaying error bodies down a separate bounded branch, so the first guard could be broken without any test noticing; the tap is now always created and the branch is the single guard, which two mutations of the credential list break and the suite catches. The list follows the code (32 killed, 0 survived).

Refs #674

---------

Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>
…never the replica; zero-token usage is no measurement

An independent review reproduced a replica crash (a planner error that coincides with a lost lease escaped the request and ended the process under the standalone listener), a zero-usage report that switched the LLM budget off, and quota accounts that were created for refused tenants and never swept. The tenant form's types also lost the shared form's compile-time key requirement.
… is dropped on every open, a malformed admission cannot crash the process

The in-process pacing gate of the search was shared across organizations (a 429 that also told one organization when another last searched), the cdp transport could still drive the host Chrome for any organization, and a throwing-free but malformed admission return escaped as an unhandled rejection.
…ts target must stay inside the clone

A hang, or an edit that does not load, was reported as killed; a mutation file could point outside the scratch clone, and a failed run leaked its clone.
…dependent review

S-9 holds by convention, the 429 ordering depends on the dimension, the migration is one way, admission and pacing coverage are named, and the mutation lists are run by hand.
@Danceiny
Danceiny merged commit e8e11be into feat/aaas-a1-fault-injection Oct 31, 2026
9 checks passed
@Danceiny
Danceiny deleted the feat/aaas-a2-identity-seam branch October 31, 2026 08:22
Danceiny added a commit that referenced this pull request Oct 31, 2026
…(A1.4) (#681)

* feat(aaas): tenant router and tenant-bound handle — guarded ledger, handoff, forget (A1.3)

Every route to a tenant's ledger has to prove the lease, so the router hands out only a guarded handle, and a graceful handoff flushes before it releases the lease. Outside the kernel and not yet wired into a runtime path.

Refs #674

* test(aaas): multi-process fault-injection drill and the #275 closure (A1.4)

A1 exits only with evidence from real processes under kills, a pause past the lease and a graceful handoff, so the drill compares each tenant's final ledger with what the replicas logged, and the RFC maps every #275 bullet to a passing suite. It also pins the one window mode D-a cannot close. Outside the kernel; engineering evidence only.

Refs #674

* fix(aaas): refuse opening a tenant that is being handed off; a consumer cannot close the router's ledger

While a handoff drains, this replica still holds the lease, so opening the tenant would only report it held by itself; it now says it is draining. The router owns the ledger's lifecycle, so close on the guarded view does nothing.

Refs #674

* docs(aaas): RFC 4.2 item 3 states the restore rule the code implements

The RFC said restore reads the prefix of the highest epoch the lease store records; what A1.2 builds is the newest verified snapshot of the newest lineage, pinned by the new owner's base, so the two now say the same thing.

Refs #674

* fix(aaas): find durable copies of epochs past eight digits

Keys pad the epoch to eight digits but the restore regex matched exactly eight, so a loop that burned a hundred million epochs would have hidden every copy; it now matches eight or more.

Refs #674

* feat(aaas): identity at the seam, quotas and metering, and the adversarial suite (A2) (#682)

* feat(aaas): seam-derived tenants and per-tenant, per-organization quotas (A2.1)

One function derives a ledger tenant from an authenticated caller and returns an opaque value that every quota API requires, so no string reaches a tenant-taking call. The meter counts concurrent turns, turn rate, LLM tokens and cost priced by the frozen ADR-20 table (its loader moves verbatim into src so the nightly evidence and the meter share it), bridge job rate and ledger size, and refuses with a typed error before any lease, file or planner is touched.

Refs #674

* feat(aaas): multi-tenant Booking Copilot surface and its adversarial suite (A2.2)

A request is authenticated, placed in its tenant at the seam, admitted against its quotas before the body is read, and served from that tenant's guarded ledger, with planner sessions and decision flights keyed by tenant; the shared single-ledger form is unchanged. The suite drives it over real HTTP with forged tenant fields, colliding ids, path-shaped claims, foreign handles, injected model output, a lost lease and a flood, and a mutation list breaks each guard in a scratch clone.

Refs #674

* feat(aaas): organization dimension in the session bridge (A2.3)

A job, claim, result, event, pacing entry and client now belong to one organization and job matching requires equality, so one organization's extension can never claim another's job or have its result accepted; a result for a foreign job is answered like an unknown one and writes nothing. A legacy bridge.db migrates in place under the default organization, and the mounted module keeps its single lane until a host supplies the caller organization.

Refs #674

* docs(aaas): wire A2 into run-all and reconcile the RFC, threat model and code map

Run-all gains the three A2 suites (§93 quotas and metering, §94 seam and surface, §95 bridge organization); the RFC records what was built and where it is proven, and the threat rows S-1 to S-5, S-9 to S-11, S-13 and S-14 move to mitigated only where a passing falsification test exists, with the S-2, S-5, S-10 and S-11 limits stated beside them.

Refs #674

* test(aaas): wait for state, not for time, in the adversarial suite

Three checks slept a fixed 50 to 100 ms before asserting that a request had reached the planner or a slot was held; on a slower runner that is a race, so they now poll for the state they need and fail with a named timeout.

Refs #674

* feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) (#683)

* feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19)

The state layer isolated what a tenant stores, but the tenants of one replica still shared what a turn runs on and with: CPU, memory, file descriptors, the planner's LLM key, the provider's prompt cache and the executor's host browser. Each now has a control with a falsification test. A planner process tree runs under ceilings (V8 heap, sampled tree RSS and CPU, a cap on an unterminated line) and a kill ends the turn as a non-retryable PLANNER_FAILED; a replica can cap the tenants it opens; planner children get a short-lived loopback ticket from a credential broker instead of the key, and measured usage is charged to the tenant whose invocation asked; every tenant prompt carries an HMAC cache-partition label ahead of any tenant-derived byte; the executor's host browser serves only the organization whose supplier login it holds. A capacity probe measures what open tenants, takeovers and a booted planner cost. Without the new options the shared form is unchanged.

Refs #674

* docs(aaas): execution isolation — RFC §4.8, threat rows S-16 to S-19 and a measured capacity baseline

The RFC said the blast radius is per tenant without saying that only state is; it now narrows that claim, lists what the tenants of a replica still share and the control and evidence for each, states the limits plainly, and writes the entry criteria for the two things that would need a real sandbox (the hosted conversational agent and a multi-tenant cloud browser). The threat model gains rows S-16 to S-19 and invariants 8 to 11, and the first measured numbers for the fallback trigger (descriptors, memory, takeover time, a booted planner) are recorded with what they do not show, in both languages.

Refs #674

* test(aaas): follow the executor's request shape in its mutation list

Two executor mutations named a line that the single-organization form no longer has once an observer's request keeps its old shape, so the runner stopped; they now break the conditional that builds the request, and the list is back to 11 killed, 0 survived.

Refs #674

* refactor(aaas): keep one guard where the broker keeps errors out of the meter

An upstream error was kept out of the usage meter twice, once by not creating the tap and once by relaying error bodies down a separate bounded branch, so the first guard could be broken without any test noticing; the tap is now always created and the branch is the single guard, which two mutations of the credential list break and the suite catches. The list follows the code (32 killed, 0 survived).

Refs #674

---------

Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>

* fix(aaas): a planner error that meets a lost lease ends one request, never the replica; zero-token usage is no measurement

An independent review reproduced a replica crash (a planner error that coincides with a lost lease escaped the request and ended the process under the standalone listener), a zero-usage report that switched the LLM budget off, and quota accounts that were created for refused tenants and never swept. The tenant form's types also lost the shared form's compile-time key requirement.

* fix(aaas): bridge pacing is per organization, an older binary's index is dropped on every open, a malformed admission cannot crash the process

The in-process pacing gate of the search was shared across organizations (a 429 that also told one organization when another last searched), the cdp transport could still drive the host Chrome for any organization, and a throwing-free but malformed admission return escaped as an unhandled rejection.

* test(aaas): a mutation is killed only by a behavioural failure, and its target must stay inside the clone

A hang, or an edit that does not load, was reported as killed; a mutation file could point outside the scratch clone, and a failed run leaked its clone.

* docs(aaas): state what the A2 guards do and do not cover after the independent review

S-9 holds by convention, the 429 ordering depends on the dimension, the migration is one way, admission and pacing coverage are named, and the mutation lists are run by hand.

---------

Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>

---------

Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>
Danceiny added a commit that referenced this pull request Oct 31, 2026
…erify, forget (A1.2) (#679)

* feat(aaas): tenant ownership core — epoch-fenced lease store and holder session (A1.1)

A1 must prove a single writer per tenant before any durable copy or router exists, so the lease protocol and its fencing come first, outside the kernel and not yet wired into a runtime path.

Refs #674

* feat(aaas): epoch-scoped durable copies — publish, lineage restore, verify, forget (A1.2)

A lease alone cannot stop a paused owner's late upload from being restored, so copies are scoped by epoch and each new owner pins the exact snapshot it started from. Outside the kernel and not yet wired into a runtime path.

Refs #674

* feat(aaas): tenant router and tenant-bound handle — guarded ledger, handoff, forget (A1.3) (#680)

Every route to a tenant's ledger has to prove the lease, so the router hands out only a guarded handle, and a graceful handoff flushes before it releases the lease. Outside the kernel and not yet wired into a runtime path.

Refs #674

Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>

* test(aaas): multi-process fault-injection drill and the #275 closure (A1.4) (#681)

* feat(aaas): tenant router and tenant-bound handle — guarded ledger, handoff, forget (A1.3)

Every route to a tenant's ledger has to prove the lease, so the router hands out only a guarded handle, and a graceful handoff flushes before it releases the lease. Outside the kernel and not yet wired into a runtime path.

Refs #674

* test(aaas): multi-process fault-injection drill and the #275 closure (A1.4)

A1 exits only with evidence from real processes under kills, a pause past the lease and a graceful handoff, so the drill compares each tenant's final ledger with what the replicas logged, and the RFC maps every #275 bullet to a passing suite. It also pins the one window mode D-a cannot close. Outside the kernel; engineering evidence only.

Refs #674

* fix(aaas): refuse opening a tenant that is being handed off; a consumer cannot close the router's ledger

While a handoff drains, this replica still holds the lease, so opening the tenant would only report it held by itself; it now says it is draining. The router owns the ledger's lifecycle, so close on the guarded view does nothing.

Refs #674

* docs(aaas): RFC 4.2 item 3 states the restore rule the code implements

The RFC said restore reads the prefix of the highest epoch the lease store records; what A1.2 builds is the newest verified snapshot of the newest lineage, pinned by the new owner's base, so the two now say the same thing.

Refs #674

* fix(aaas): find durable copies of epochs past eight digits

Keys pad the epoch to eight digits but the restore regex matched exactly eight, so a loop that burned a hundred million epochs would have hidden every copy; it now matches eight or more.

Refs #674

* feat(aaas): identity at the seam, quotas and metering, and the adversarial suite (A2) (#682)

* feat(aaas): seam-derived tenants and per-tenant, per-organization quotas (A2.1)

One function derives a ledger tenant from an authenticated caller and returns an opaque value that every quota API requires, so no string reaches a tenant-taking call. The meter counts concurrent turns, turn rate, LLM tokens and cost priced by the frozen ADR-20 table (its loader moves verbatim into src so the nightly evidence and the meter share it), bridge job rate and ledger size, and refuses with a typed error before any lease, file or planner is touched.

Refs #674

* feat(aaas): multi-tenant Booking Copilot surface and its adversarial suite (A2.2)

A request is authenticated, placed in its tenant at the seam, admitted against its quotas before the body is read, and served from that tenant's guarded ledger, with planner sessions and decision flights keyed by tenant; the shared single-ledger form is unchanged. The suite drives it over real HTTP with forged tenant fields, colliding ids, path-shaped claims, foreign handles, injected model output, a lost lease and a flood, and a mutation list breaks each guard in a scratch clone.

Refs #674

* feat(aaas): organization dimension in the session bridge (A2.3)

A job, claim, result, event, pacing entry and client now belong to one organization and job matching requires equality, so one organization's extension can never claim another's job or have its result accepted; a result for a foreign job is answered like an unknown one and writes nothing. A legacy bridge.db migrates in place under the default organization, and the mounted module keeps its single lane until a host supplies the caller organization.

Refs #674

* docs(aaas): wire A2 into run-all and reconcile the RFC, threat model and code map

Run-all gains the three A2 suites (§93 quotas and metering, §94 seam and surface, §95 bridge organization); the RFC records what was built and where it is proven, and the threat rows S-1 to S-5, S-9 to S-11, S-13 and S-14 move to mitigated only where a passing falsification test exists, with the S-2, S-5, S-10 and S-11 limits stated beside them.

Refs #674

* test(aaas): wait for state, not for time, in the adversarial suite

Three checks slept a fixed 50 to 100 ms before asserting that a request had reached the planner or a slot was held; on a slower runner that is a race, so they now poll for the state they need and fail with a named timeout.

Refs #674

* feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) (#683)

* feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19)

The state layer isolated what a tenant stores, but the tenants of one replica still shared what a turn runs on and with: CPU, memory, file descriptors, the planner's LLM key, the provider's prompt cache and the executor's host browser. Each now has a control with a falsification test. A planner process tree runs under ceilings (V8 heap, sampled tree RSS and CPU, a cap on an unterminated line) and a kill ends the turn as a non-retryable PLANNER_FAILED; a replica can cap the tenants it opens; planner children get a short-lived loopback ticket from a credential broker instead of the key, and measured usage is charged to the tenant whose invocation asked; every tenant prompt carries an HMAC cache-partition label ahead of any tenant-derived byte; the executor's host browser serves only the organization whose supplier login it holds. A capacity probe measures what open tenants, takeovers and a booted planner cost. Without the new options the shared form is unchanged.

Refs #674

* docs(aaas): execution isolation — RFC §4.8, threat rows S-16 to S-19 and a measured capacity baseline

The RFC said the blast radius is per tenant without saying that only state is; it now narrows that claim, lists what the tenants of a replica still share and the control and evidence for each, states the limits plainly, and writes the entry criteria for the two things that would need a real sandbox (the hosted conversational agent and a multi-tenant cloud browser). The threat model gains rows S-16 to S-19 and invariants 8 to 11, and the first measured numbers for the fallback trigger (descriptors, memory, takeover time, a booted planner) are recorded with what they do not show, in both languages.

Refs #674

* test(aaas): follow the executor's request shape in its mutation list

Two executor mutations named a line that the single-organization form no longer has once an observer's request keeps its old shape, so the runner stopped; they now break the conditional that builds the request, and the list is back to 11 killed, 0 survived.

Refs #674

* refactor(aaas): keep one guard where the broker keeps errors out of the meter

An upstream error was kept out of the usage meter twice, once by not creating the tap and once by relaying error bodies down a separate bounded branch, so the first guard could be broken without any test noticing; the tap is now always created and the branch is the single guard, which two mutations of the credential list break and the suite catches. The list follows the code (32 killed, 0 survived).

Refs #674

---------

Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>

* fix(aaas): a planner error that meets a lost lease ends one request, never the replica; zero-token usage is no measurement

An independent review reproduced a replica crash (a planner error that coincides with a lost lease escaped the request and ended the process under the standalone listener), a zero-usage report that switched the LLM budget off, and quota accounts that were created for refused tenants and never swept. The tenant form's types also lost the shared form's compile-time key requirement.

* fix(aaas): bridge pacing is per organization, an older binary's index is dropped on every open, a malformed admission cannot crash the process

The in-process pacing gate of the search was shared across organizations (a 429 that also told one organization when another last searched), the cdp transport could still drive the host Chrome for any organization, and a throwing-free but malformed admission return escaped as an unhandled rejection.

* test(aaas): a mutation is killed only by a behavioural failure, and its target must stay inside the clone

A hang, or an edit that does not load, was reported as killed; a mutation file could point outside the scratch clone, and a failed run leaked its clone.

* docs(aaas): state what the A2 guards do and do not cover after the independent review

S-9 holds by convention, the 429 ordering depends on the dimension, the migration is one way, admission and pacing coverage are named, and the mutation lists are run by hand.

---------

Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>

---------

Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>

---------

Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant