Repository navigation
feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) - #683
Merged
Conversation
added 4 commits
October 30, 2026 15:59
…tenants a replica opens (S-16 to S-19) The state layer isolated what a tenant stores, but the tenants of one replica still shared what a turn runs on and with: CPU, memory, file descriptors, the planner's LLM key, the provider's prompt cache and the executor's host browser. Each now has a control with a falsification test. A planner process tree runs under ceilings (V8 heap, sampled tree RSS and CPU, a cap on an unterminated line) and a kill ends the turn as a non-retryable PLANNER_FAILED; a replica can cap the tenants it opens; planner children get a short-lived loopback ticket from a credential broker instead of the key, and measured usage is charged to the tenant whose invocation asked; every tenant prompt carries an HMAC cache-partition label ahead of any tenant-derived byte; the executor's host browser serves only the organization whose supplier login it holds. A capacity probe measures what open tenants, takeovers and a booted planner cost. Without the new options the shared form is unchanged. Refs #674
…and a measured capacity baseline The RFC said the blast radius is per tenant without saying that only state is; it now narrows that claim, lists what the tenants of a replica still share and the control and evidence for each, states the limits plainly, and writes the entry criteria for the two things that would need a real sandbox (the hosted conversational agent and a multi-tenant cloud browser). The threat model gains rows S-16 to S-19 and invariants 8 to 11, and the first measured numbers for the fallback trigger (descriptors, memory, takeover time, a booted planner) are recorded with what they do not show, in both languages. Refs #674
Two executor mutations named a line that the single-organization form no longer has once an observer's request keeps its old shape, so the runner stopped; they now break the conditional that builds the request, and the list is back to 11 killed, 0 survived. Refs #674
…he meter An upstream error was kept out of the usage meter twice, once by not creating the tap and once by relaying error bodies down a separate bounded branch, so the first guard could be broken without any test noticing; the tap is now always created and the branch is the single guard, which two mutations of the credential list break and the suite catches. The list follows the code (32 killed, 0 survived). Refs #674
Danceiny
added a commit
that referenced
this pull request
Oct 31, 2026
…arial suite (A2) (#682) * feat(aaas): seam-derived tenants and per-tenant, per-organization quotas (A2.1) One function derives a ledger tenant from an authenticated caller and returns an opaque value that every quota API requires, so no string reaches a tenant-taking call. The meter counts concurrent turns, turn rate, LLM tokens and cost priced by the frozen ADR-20 table (its loader moves verbatim into src so the nightly evidence and the meter share it), bridge job rate and ledger size, and refuses with a typed error before any lease, file or planner is touched. Refs #674 * feat(aaas): multi-tenant Booking Copilot surface and its adversarial suite (A2.2) A request is authenticated, placed in its tenant at the seam, admitted against its quotas before the body is read, and served from that tenant's guarded ledger, with planner sessions and decision flights keyed by tenant; the shared single-ledger form is unchanged. The suite drives it over real HTTP with forged tenant fields, colliding ids, path-shaped claims, foreign handles, injected model output, a lost lease and a flood, and a mutation list breaks each guard in a scratch clone. Refs #674 * feat(aaas): organization dimension in the session bridge (A2.3) A job, claim, result, event, pacing entry and client now belong to one organization and job matching requires equality, so one organization's extension can never claim another's job or have its result accepted; a result for a foreign job is answered like an unknown one and writes nothing. A legacy bridge.db migrates in place under the default organization, and the mounted module keeps its single lane until a host supplies the caller organization. Refs #674 * docs(aaas): wire A2 into run-all and reconcile the RFC, threat model and code map Run-all gains the three A2 suites (§93 quotas and metering, §94 seam and surface, §95 bridge organization); the RFC records what was built and where it is proven, and the threat rows S-1 to S-5, S-9 to S-11, S-13 and S-14 move to mitigated only where a passing falsification test exists, with the S-2, S-5, S-10 and S-11 limits stated beside them. Refs #674 * test(aaas): wait for state, not for time, in the adversarial suite Three checks slept a fixed 50 to 100 ms before asserting that a request had reached the planner or a slot was held; on a slower runner that is a race, so they now poll for the state they need and fail with a named timeout. Refs #674 * feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) (#683) * feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) The state layer isolated what a tenant stores, but the tenants of one replica still shared what a turn runs on and with: CPU, memory, file descriptors, the planner's LLM key, the provider's prompt cache and the executor's host browser. Each now has a control with a falsification test. A planner process tree runs under ceilings (V8 heap, sampled tree RSS and CPU, a cap on an unterminated line) and a kill ends the turn as a non-retryable PLANNER_FAILED; a replica can cap the tenants it opens; planner children get a short-lived loopback ticket from a credential broker instead of the key, and measured usage is charged to the tenant whose invocation asked; every tenant prompt carries an HMAC cache-partition label ahead of any tenant-derived byte; the executor's host browser serves only the organization whose supplier login it holds. A capacity probe measures what open tenants, takeovers and a booted planner cost. Without the new options the shared form is unchanged. Refs #674 * docs(aaas): execution isolation — RFC §4.8, threat rows S-16 to S-19 and a measured capacity baseline The RFC said the blast radius is per tenant without saying that only state is; it now narrows that claim, lists what the tenants of a replica still share and the control and evidence for each, states the limits plainly, and writes the entry criteria for the two things that would need a real sandbox (the hosted conversational agent and a multi-tenant cloud browser). The threat model gains rows S-16 to S-19 and invariants 8 to 11, and the first measured numbers for the fallback trigger (descriptors, memory, takeover time, a booted planner) are recorded with what they do not show, in both languages. Refs #674 * test(aaas): follow the executor's request shape in its mutation list Two executor mutations named a line that the single-organization form no longer has once an observer's request keeps its old shape, so the runner stopped; they now break the conditional that builds the request, and the list is back to 11 killed, 0 survived. Refs #674 * refactor(aaas): keep one guard where the broker keeps errors out of the meter An upstream error was kept out of the usage meter twice, once by not creating the tap and once by relaying error bodies down a separate bounded branch, so the first guard could be broken without any test noticing; the tap is now always created and the branch is the single guard, which two mutations of the credential list break and the suite catches. The list follows the code (32 killed, 0 survived). Refs #674 --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com> * fix(aaas): a planner error that meets a lost lease ends one request, never the replica; zero-token usage is no measurement An independent review reproduced a replica crash (a planner error that coincides with a lost lease escaped the request and ended the process under the standalone listener), a zero-usage report that switched the LLM budget off, and quota accounts that were created for refused tenants and never swept. The tenant form's types also lost the shared form's compile-time key requirement. * fix(aaas): bridge pacing is per organization, an older binary's index is dropped on every open, a malformed admission cannot crash the process The in-process pacing gate of the search was shared across organizations (a 429 that also told one organization when another last searched), the cdp transport could still drive the host Chrome for any organization, and a throwing-free but malformed admission return escaped as an unhandled rejection. * test(aaas): a mutation is killed only by a behavioural failure, and its target must stay inside the clone A hang, or an edit that does not load, was reported as killed; a mutation file could point outside the scratch clone, and a failed run leaked its clone. * docs(aaas): state what the A2 guards do and do not cover after the independent review S-9 holds by convention, the 429 ordering depends on the dimension, the migration is one way, admission and pacing coverage are named, and the mutation lists are run by hand. --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>
Danceiny
added a commit
that referenced
this pull request
Oct 31, 2026
…(A1.4) (#681) * feat(aaas): tenant router and tenant-bound handle — guarded ledger, handoff, forget (A1.3) Every route to a tenant's ledger has to prove the lease, so the router hands out only a guarded handle, and a graceful handoff flushes before it releases the lease. Outside the kernel and not yet wired into a runtime path. Refs #674 * test(aaas): multi-process fault-injection drill and the #275 closure (A1.4) A1 exits only with evidence from real processes under kills, a pause past the lease and a graceful handoff, so the drill compares each tenant's final ledger with what the replicas logged, and the RFC maps every #275 bullet to a passing suite. It also pins the one window mode D-a cannot close. Outside the kernel; engineering evidence only. Refs #674 * fix(aaas): refuse opening a tenant that is being handed off; a consumer cannot close the router's ledger While a handoff drains, this replica still holds the lease, so opening the tenant would only report it held by itself; it now says it is draining. The router owns the ledger's lifecycle, so close on the guarded view does nothing. Refs #674 * docs(aaas): RFC 4.2 item 3 states the restore rule the code implements The RFC said restore reads the prefix of the highest epoch the lease store records; what A1.2 builds is the newest verified snapshot of the newest lineage, pinned by the new owner's base, so the two now say the same thing. Refs #674 * fix(aaas): find durable copies of epochs past eight digits Keys pad the epoch to eight digits but the restore regex matched exactly eight, so a loop that burned a hundred million epochs would have hidden every copy; it now matches eight or more. Refs #674 * feat(aaas): identity at the seam, quotas and metering, and the adversarial suite (A2) (#682) * feat(aaas): seam-derived tenants and per-tenant, per-organization quotas (A2.1) One function derives a ledger tenant from an authenticated caller and returns an opaque value that every quota API requires, so no string reaches a tenant-taking call. The meter counts concurrent turns, turn rate, LLM tokens and cost priced by the frozen ADR-20 table (its loader moves verbatim into src so the nightly evidence and the meter share it), bridge job rate and ledger size, and refuses with a typed error before any lease, file or planner is touched. Refs #674 * feat(aaas): multi-tenant Booking Copilot surface and its adversarial suite (A2.2) A request is authenticated, placed in its tenant at the seam, admitted against its quotas before the body is read, and served from that tenant's guarded ledger, with planner sessions and decision flights keyed by tenant; the shared single-ledger form is unchanged. The suite drives it over real HTTP with forged tenant fields, colliding ids, path-shaped claims, foreign handles, injected model output, a lost lease and a flood, and a mutation list breaks each guard in a scratch clone. Refs #674 * feat(aaas): organization dimension in the session bridge (A2.3) A job, claim, result, event, pacing entry and client now belong to one organization and job matching requires equality, so one organization's extension can never claim another's job or have its result accepted; a result for a foreign job is answered like an unknown one and writes nothing. A legacy bridge.db migrates in place under the default organization, and the mounted module keeps its single lane until a host supplies the caller organization. Refs #674 * docs(aaas): wire A2 into run-all and reconcile the RFC, threat model and code map Run-all gains the three A2 suites (§93 quotas and metering, §94 seam and surface, §95 bridge organization); the RFC records what was built and where it is proven, and the threat rows S-1 to S-5, S-9 to S-11, S-13 and S-14 move to mitigated only where a passing falsification test exists, with the S-2, S-5, S-10 and S-11 limits stated beside them. Refs #674 * test(aaas): wait for state, not for time, in the adversarial suite Three checks slept a fixed 50 to 100 ms before asserting that a request had reached the planner or a slot was held; on a slower runner that is a race, so they now poll for the state they need and fail with a named timeout. Refs #674 * feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) (#683) * feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) The state layer isolated what a tenant stores, but the tenants of one replica still shared what a turn runs on and with: CPU, memory, file descriptors, the planner's LLM key, the provider's prompt cache and the executor's host browser. Each now has a control with a falsification test. A planner process tree runs under ceilings (V8 heap, sampled tree RSS and CPU, a cap on an unterminated line) and a kill ends the turn as a non-retryable PLANNER_FAILED; a replica can cap the tenants it opens; planner children get a short-lived loopback ticket from a credential broker instead of the key, and measured usage is charged to the tenant whose invocation asked; every tenant prompt carries an HMAC cache-partition label ahead of any tenant-derived byte; the executor's host browser serves only the organization whose supplier login it holds. A capacity probe measures what open tenants, takeovers and a booted planner cost. Without the new options the shared form is unchanged. Refs #674 * docs(aaas): execution isolation — RFC §4.8, threat rows S-16 to S-19 and a measured capacity baseline The RFC said the blast radius is per tenant without saying that only state is; it now narrows that claim, lists what the tenants of a replica still share and the control and evidence for each, states the limits plainly, and writes the entry criteria for the two things that would need a real sandbox (the hosted conversational agent and a multi-tenant cloud browser). The threat model gains rows S-16 to S-19 and invariants 8 to 11, and the first measured numbers for the fallback trigger (descriptors, memory, takeover time, a booted planner) are recorded with what they do not show, in both languages. Refs #674 * test(aaas): follow the executor's request shape in its mutation list Two executor mutations named a line that the single-organization form no longer has once an observer's request keeps its old shape, so the runner stopped; they now break the conditional that builds the request, and the list is back to 11 killed, 0 survived. Refs #674 * refactor(aaas): keep one guard where the broker keeps errors out of the meter An upstream error was kept out of the usage meter twice, once by not creating the tap and once by relaying error bodies down a separate bounded branch, so the first guard could be broken without any test noticing; the tap is now always created and the branch is the single guard, which two mutations of the credential list break and the suite catches. The list follows the code (32 killed, 0 survived). Refs #674 --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com> * fix(aaas): a planner error that meets a lost lease ends one request, never the replica; zero-token usage is no measurement An independent review reproduced a replica crash (a planner error that coincides with a lost lease escaped the request and ended the process under the standalone listener), a zero-usage report that switched the LLM budget off, and quota accounts that were created for refused tenants and never swept. The tenant form's types also lost the shared form's compile-time key requirement. * fix(aaas): bridge pacing is per organization, an older binary's index is dropped on every open, a malformed admission cannot crash the process The in-process pacing gate of the search was shared across organizations (a 429 that also told one organization when another last searched), the cdp transport could still drive the host Chrome for any organization, and a throwing-free but malformed admission return escaped as an unhandled rejection. * test(aaas): a mutation is killed only by a behavioural failure, and its target must stay inside the clone A hang, or an edit that does not load, was reported as killed; a mutation file could point outside the scratch clone, and a failed run leaked its clone. * docs(aaas): state what the A2 guards do and do not cover after the independent review S-9 holds by convention, the 429 ordering depends on the dimension, the migration is one way, admission and pacing coverage are named, and the mutation lists are run by hand. --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com> --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>
Danceiny
added a commit
that referenced
this pull request
Oct 31, 2026
…erify, forget (A1.2) (#679) * feat(aaas): tenant ownership core — epoch-fenced lease store and holder session (A1.1) A1 must prove a single writer per tenant before any durable copy or router exists, so the lease protocol and its fencing come first, outside the kernel and not yet wired into a runtime path. Refs #674 * feat(aaas): epoch-scoped durable copies — publish, lineage restore, verify, forget (A1.2) A lease alone cannot stop a paused owner's late upload from being restored, so copies are scoped by epoch and each new owner pins the exact snapshot it started from. Outside the kernel and not yet wired into a runtime path. Refs #674 * feat(aaas): tenant router and tenant-bound handle — guarded ledger, handoff, forget (A1.3) (#680) Every route to a tenant's ledger has to prove the lease, so the router hands out only a guarded handle, and a graceful handoff flushes before it releases the lease. Outside the kernel and not yet wired into a runtime path. Refs #674 Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com> * test(aaas): multi-process fault-injection drill and the #275 closure (A1.4) (#681) * feat(aaas): tenant router and tenant-bound handle — guarded ledger, handoff, forget (A1.3) Every route to a tenant's ledger has to prove the lease, so the router hands out only a guarded handle, and a graceful handoff flushes before it releases the lease. Outside the kernel and not yet wired into a runtime path. Refs #674 * test(aaas): multi-process fault-injection drill and the #275 closure (A1.4) A1 exits only with evidence from real processes under kills, a pause past the lease and a graceful handoff, so the drill compares each tenant's final ledger with what the replicas logged, and the RFC maps every #275 bullet to a passing suite. It also pins the one window mode D-a cannot close. Outside the kernel; engineering evidence only. Refs #674 * fix(aaas): refuse opening a tenant that is being handed off; a consumer cannot close the router's ledger While a handoff drains, this replica still holds the lease, so opening the tenant would only report it held by itself; it now says it is draining. The router owns the ledger's lifecycle, so close on the guarded view does nothing. Refs #674 * docs(aaas): RFC 4.2 item 3 states the restore rule the code implements The RFC said restore reads the prefix of the highest epoch the lease store records; what A1.2 builds is the newest verified snapshot of the newest lineage, pinned by the new owner's base, so the two now say the same thing. Refs #674 * fix(aaas): find durable copies of epochs past eight digits Keys pad the epoch to eight digits but the restore regex matched exactly eight, so a loop that burned a hundred million epochs would have hidden every copy; it now matches eight or more. Refs #674 * feat(aaas): identity at the seam, quotas and metering, and the adversarial suite (A2) (#682) * feat(aaas): seam-derived tenants and per-tenant, per-organization quotas (A2.1) One function derives a ledger tenant from an authenticated caller and returns an opaque value that every quota API requires, so no string reaches a tenant-taking call. The meter counts concurrent turns, turn rate, LLM tokens and cost priced by the frozen ADR-20 table (its loader moves verbatim into src so the nightly evidence and the meter share it), bridge job rate and ledger size, and refuses with a typed error before any lease, file or planner is touched. Refs #674 * feat(aaas): multi-tenant Booking Copilot surface and its adversarial suite (A2.2) A request is authenticated, placed in its tenant at the seam, admitted against its quotas before the body is read, and served from that tenant's guarded ledger, with planner sessions and decision flights keyed by tenant; the shared single-ledger form is unchanged. The suite drives it over real HTTP with forged tenant fields, colliding ids, path-shaped claims, foreign handles, injected model output, a lost lease and a flood, and a mutation list breaks each guard in a scratch clone. Refs #674 * feat(aaas): organization dimension in the session bridge (A2.3) A job, claim, result, event, pacing entry and client now belong to one organization and job matching requires equality, so one organization's extension can never claim another's job or have its result accepted; a result for a foreign job is answered like an unknown one and writes nothing. A legacy bridge.db migrates in place under the default organization, and the mounted module keeps its single lane until a host supplies the caller organization. Refs #674 * docs(aaas): wire A2 into run-all and reconcile the RFC, threat model and code map Run-all gains the three A2 suites (§93 quotas and metering, §94 seam and surface, §95 bridge organization); the RFC records what was built and where it is proven, and the threat rows S-1 to S-5, S-9 to S-11, S-13 and S-14 move to mitigated only where a passing falsification test exists, with the S-2, S-5, S-10 and S-11 limits stated beside them. Refs #674 * test(aaas): wait for state, not for time, in the adversarial suite Three checks slept a fixed 50 to 100 ms before asserting that a request had reached the planner or a slot was held; on a slower runner that is a race, so they now poll for the state they need and fail with a named timeout. Refs #674 * feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) (#683) * feat(aaas): bound the planner children, their LLM credential and the tenants a replica opens (S-16 to S-19) The state layer isolated what a tenant stores, but the tenants of one replica still shared what a turn runs on and with: CPU, memory, file descriptors, the planner's LLM key, the provider's prompt cache and the executor's host browser. Each now has a control with a falsification test. A planner process tree runs under ceilings (V8 heap, sampled tree RSS and CPU, a cap on an unterminated line) and a kill ends the turn as a non-retryable PLANNER_FAILED; a replica can cap the tenants it opens; planner children get a short-lived loopback ticket from a credential broker instead of the key, and measured usage is charged to the tenant whose invocation asked; every tenant prompt carries an HMAC cache-partition label ahead of any tenant-derived byte; the executor's host browser serves only the organization whose supplier login it holds. A capacity probe measures what open tenants, takeovers and a booted planner cost. Without the new options the shared form is unchanged. Refs #674 * docs(aaas): execution isolation — RFC §4.8, threat rows S-16 to S-19 and a measured capacity baseline The RFC said the blast radius is per tenant without saying that only state is; it now narrows that claim, lists what the tenants of a replica still share and the control and evidence for each, states the limits plainly, and writes the entry criteria for the two things that would need a real sandbox (the hosted conversational agent and a multi-tenant cloud browser). The threat model gains rows S-16 to S-19 and invariants 8 to 11, and the first measured numbers for the fallback trigger (descriptors, memory, takeover time, a booted planner) are recorded with what they do not show, in both languages. Refs #674 * test(aaas): follow the executor's request shape in its mutation list Two executor mutations named a line that the single-organization form no longer has once an observer's request keeps its old shape, so the runner stopped; they now break the conditional that builds the request, and the list is back to 11 killed, 0 survived. Refs #674 * refactor(aaas): keep one guard where the broker keeps errors out of the meter An upstream error was kept out of the usage meter twice, once by not creating the tap and once by relaying error bodies down a separate bounded branch, so the first guard could be broken without any test noticing; the tap is now always created and the branch is the single guard, which two mutations of the credential list break and the suite catches. The list follows the code (32 killed, 0 survived). Refs #674 --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com> * fix(aaas): a planner error that meets a lost lease ends one request, never the replica; zero-token usage is no measurement An independent review reproduced a replica crash (a planner error that coincides with a lost lease escaped the request and ended the process under the standalone listener), a zero-usage report that switched the LLM budget off, and quota accounts that were created for refused tenants and never swept. The tenant form's types also lost the shared form's compile-time key requirement. * fix(aaas): bridge pacing is per organization, an older binary's index is dropped on every open, a malformed admission cannot crash the process The in-process pacing gate of the search was shared across organizations (a 429 that also told one organization when another last searched), the cdp transport could still drive the host Chrome for any organization, and a throwing-free but malformed admission return escaped as an unhandled rejection. * test(aaas): a mutation is killed only by a behavioural failure, and its target must stay inside the clone A hang, or an edit that does not load, was reported as killed; a mutation file could point outside the scratch clone, and a failed run leaked its clone. * docs(aaas): state what the A2 guards do and do not cover after the independent review S-9 holds by convention, the 429 ordering depends on the dimension, the migration is one way, admission and pacing coverage are named, and the mutation lists are run by hand. --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com> --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com> --------- Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>
Danceiny
added a commit
that referenced
this pull request
Oct 31, 2026
…-ups made stale (#685) Running every list in full showed four entries that no longer meant what they said: one find matched twice after the sweep was added, one matched nothing after #683 reworked the planner provider, and two survived because an earlier guard had become redundant behind a new one (the legacy-index drop, the lease-lost check). They now break the guard that actually holds. Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>
Danceiny
added a commit
that referenced
this pull request
Oct 31, 2026
…ocess governor and the executor (#687) An independent read-only review of #683 showed that a misconfigured upstream could hand the broker's key back to the child through an echoed error body, that the governor stopped governing when it could not see a tree, and that the child-facing port had no bounds on connections; each is fixed behind a check that failed first. - llm-broker: an upstream error body is never read or relayed; the child gets the status, a closed code and kind and a numeric retry hint. Revoking a ticket aborts what it has in flight. The child-facing port caps connections and drops one that does not send its headers. - process-governor: a watched tree the sampler cannot see for N ticks in a row (root missing, table failing or hanging past a timeout) is a breach of kind "blind", and the probe is raced the same way. Every real port probes before it starts a child, not only the first of a process. - booking-executor: a body that is JSON but not an object is a 400 instead of an unhandled rejection. - tests: the guards whose mutations were only ever caught by a suite that hung now fail on an assertion with a deadline, because the mutation runner on main does not count a hang as a kill; the mutation lists are re-anchored and extended. Refs #674 Co-authored-by: huangzhen.0505 <huangzhen.0505@bytedance.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #674
Closes the gaps found when the A2 design was set against a cloud-agent isolation design (per-task sandboxes, an egress proxy that injects credentials, snapshot and warm-pool scaling). The state, identity and quota layer was designed and tested. What the tenants of one replica still share (CPU, memory, file descriptors, the LLM credential, the provider's prompt cache, the executor's host browser) and the scale numbers were not. This PR gives each shared thing a control with a falsification test, measures the numbers, and writes down what is not built and when it would have to be. It lives outside the kernel (
kernel-manifest-gate: zero diff), adds no dependency, and changes no WriteGate or M5 behavior.Stacked on #682 (the base of this PR is its branch, so the diff is this change only). When the A1/A2 stack lands I rebase onto
mainand retarget; there is nothing to decide in between.Why a sandbox per task is not in it
The v1 planner can run no code: dsh's shell, file, sandbox, subprocess and pty plugins are disabled,
toolBashis off, the only tools are the thirteenbooking_*capabilities, and assistant text is ignored. A microVM or gVisor boundary per task would isolate nothing the planner can reach, and supplier login state is not on the server at all (S-9). RFC §4.8 says so, lists what the replica's tenants do share, and writes the entry criteria for the two things that would change that (the hosted conversational agent, Q6, and a multi-tenant cloud executor browser): their own ADR and a founder decision, one disposable sandbox per task, deny-by-default egress through a credential-injecting proxy, sandbox-enforced accounting, a re-measured baseline, and, for a browser, a founder decision on how a tenant's login would reach it without breaking the 2026-09-11 red line.What it adds
ProcessTreeGovernorsamples the process table, finds a worker's tree by parent pid (no process-group or cgroup assumption, so it holds on Darwin and on Linux user-systemd scopes), kills the whole tree over a ceiling.ManagedDshRunPorttakes optionallimits: a V8 heap ceiling (worker command line andNODE_OPTIONS), the tree sampler, a cap on a line that never ends. A kill is a typedManagedDshResourceLimitErrorand the planner answers a non-retryablePLANNER_FAILED, never rerun on a fresh portTenantRoutermaxOpenTenants: a tenant over the cap is refused before its lease or any file exists (a typed 503)LlmCredentialBroker: a loopback forward proxy holding the upstream key and base URL; each planner process tree gets a ticket valid only against it, for a bounded time and number of requests, revoked with its port. One allow-listed POST, real key attached, redirects refused, bounded bodies, streaming relayed as it arrives. The real tenant planner cannot be built without a broker. Measuredusageis reported before the response ends and charged to the tenant whose invocation asked (the A2 "worst case when nothing is reported" limitation)pc-…) sits after the instructions every tenant shares and before any tenant-derived byte; a malformed label is refusedhostBrowserOrganization, defaultlocal); the caller's organization comes from the host (callerOrganization), never from the request; a multi-organization deployment must name the ownertenant-capacity-probe.tsanddocs/ops/aaas-capacity-baseline.mdRFC §3.3 item 4 no longer says the blast radius is per tenant without qualification: it is per tenant for state; compute is per replica and §4.8 says what bounds it. New threat rows S-16 to S-19, invariants 8 to 11, two rejected options, code map rows and docs index, in both languages.
Public-contract check (the stop condition in the brief)
I found nothing that needed the founder before coding, and say so here so you can overrule it:
booking.surfaceschema, event kinds, version and hash headers, and the closed error-code list: unchanged. A resource kill is the existingPLANNER_FAILEDwithretryable: false;resource_limitis a new value of the internal planner metric only.503 tenant_unavailable, deliberately withoutRetry-After(only a handoff frees a slot). Nothing else is added.limits(P7), the executor without hooks (E1), a router without a cap (R3). The seven shared-form booking suites run unchanged in the regression.TenantPlannerContextgained a requiredcachePartition, andcreateDshTenantPlannerProvidernow refuses to build the real dsh planner withoutllmTickets. That is the one behavior change relative to feat(aaas): identity at the seam, quotas and metering, and the adversarial suite (A2) #682, and nothing runs the tenant form in production yet.Evidence
Local Node v26.9.0, macOS arm64. SHA
863c88c78d4a0b6e2155131983283b39bd0b4a4e.Gates (from
ts/):npx tsc --noEmit;npx tsx scripts/smoke.ts;npx tsx scripts/g5-guard.ts;npx tsx scripts/kernel-manifest-gate.ts;node scripts/check-docs-i18n.mjs(94 pairs);node scripts/check-doc-readability.mjs. All exit 0:tscclean;SMOKE OK;G5 GUARD: OK;KERNEL MANIFEST GATE: OK(zero manifest diff, 5/5 kernel files loaded).Full regression on this SHA:
NPM_CONFIG_USERCONFIG=/dev/null NPM_CONFIG_REGISTRY=https://registry.npmjs.org/ npm_config_registry=https://registry.npmjs.org/ bash scripts/run-all-tests.sh->ALL SUITES GREEN, exit 0, clean tree before and after, no ENOSPC in the log (the public registry is used because the internal mirror is unreachable from this machine). CI runs it again on Node 22 and 24.New suites (all offline, isolated temporary roots): §96 compute and replica capacity, 33 checks (the runaway modes use real processes); §97 credentials, 29 checks (including a real dsh run through the broker that yields a decision while the environment offers the real key and address, and the real chat-completions adapter against the broker); §98 prompt partition, 16; §99 executor binding, 8; §100 capacity probe smoke.
Mutation results (
cd ts && npx tsx scripts/mutation-check.ts scripts/<list>.json; each edit breaks one guard in a scratch clone and the suite must go red):tenant-compute-isolation-mutations.jsontenant-egress-credentials-mutations.jsontenant-prompt-partition-mutations.jsontenant-executor-binding-mutations.jsonThe first run had three survivors, all in my own tests: an assertion that counted a watch after unwatching it, an upstream-error guard that was redundant with the branch before it, and a close-time unwatch that the worker's exit made unobservable. A fourth test (an expired-ticket sweep that
stats()also performed) I tightened before it could survive. The suites and the mutations were fixed, not the lists loosened.Measured, on one development machine (
docs/ops/aaas-capacity-baseline.md): an idle open tenant costs about 3 file descriptors and 0.27 MiB and opens in about 9 ms, flat from 1 to 500 tenants, so the descriptor limit is the first ceiling (a few hundred tenants at a soft limit of 1024); takeover is linear in the ledger (about 4 ms/MiB to restore, 100 MiB in 0.4 s); a booted idle planner tree (the worker and the runtime below it) is about 200 MiB; closing 500 tenants gracefully took 11 s.An independent review, and what it changed
Before this was committed, a separate agent read the diff cold and was told to be skeptical. What held, and is fixed here:
[DONE]and cancels while the upstream keeps the stream open (reproduced with the realGatewayChatAdapter): the broker reported usage only when the upstream body ended, so the tenant was charged the worst case and a complete response counted as an upstream failure. It now meters at[DONE], still before the child can see the end of the stream.LLM_BASE_URL. The runtime merges the explicit environment onto the parent's, minus names that look secret, so an address passes. The ambientLLM_API_KEYandLLM_BASE_URLare now removed explicitly and checked against a real worker, with a control showing the leak without it; the docs say what the child still inherits.maxOpenTenantsnever frees a slot (no idle eviction exists). The docs now say so, including that one organization with many users can fill a replica, and the refusal no longer carries a misleadingRetry-After. Idle release belongs to A3 with affinity; I did not add eviction to the A1 router in this PR.organization: 'local'; the request keeps its old shape unless a host resolves organizations.PLANNER_FAILEDanswer without theresource_limitmetric; the P6 timing margin was widened.The review could not run anything on Linux either.
Not in this PR, and limits
psonce a second; a runaway can overshoot between samples, and the V8 heap ceiling is the first line for JavaScript allocations. RSS counts shared pages per process, CPU time has one-second resolution on Linux, and I could only run it on macOS: the Linux path (user-systemd scope,pscolumn formats) is reasoned from the runtime's code and the existing cleanup diagnostics, not exercised.DEEPSEEK_PROVIDER=gateway-chat) only and a broker on another route fails at construction.maxOpenTenants, the global turn cap and the ceilings have no production value yet; the A3 drill list gains "runaway child".Review asks
DEFAULT_TENANT_PLANNER_RESOURCE_LIMITS(heap 512 MiB, tree RSS 1.5 GiB, 120 CPU-seconds, 8 MiB line): unreviewed, set against a measured idle tree of about 200 MiB.maxOpenTenantsis unset by default; A3 should set it from the node's descriptor limit (about 3 per tenant).callerOrganization/hostBrowserOrganizationpair.