[AAASM-5664] 🐛 (adapters): Report only successfully patched adapters as active - #354
Merged
Merged
Conversation
buildActiveAdapters unioned the detection list with a set of patch-success flags. Every non-LangChain flag is itself gated on detection, so the union could only ever reproduce the detection list: a framework found on disk was reported in activeAdapters even when its patch had just warned it was inert. The frozen-ESM Vercel shape (AAASM-4842) hit this on every real install — activeAdapters said "vercel-ai-sdk" in the same run stderr said Vercel tool calls would NOT be governed. Derive active from the patch results alone, and split the outcome into three explicit fields rather than one filtered array — "not installed" and "installed but ungoverned" are different facts with different remedies, and collapsing them into an omission from activeAdapters would trade one misleading signal for another. The two LangChain flags are preserved: an explicit config.langchain wraps callbacks/tools whether or not @langchain/core resolves, so langchain-js is the one id that can legitimately be active without being detected. Refs AAASM-5664, blocks AAASM-5529
Pins the AAASM-5664 contract over the artifact a caller actually reads (ctx.activeAdapters / detectedAdapters / unpatchedAdapters) rather than an internal patch flag. Two negative controls, because there are two distinct routes to reporting a detected-but-ungoverned framework as active: a patch that fails (the frozen-ESM Vercel shape) and a patch that is skipped for a missing prerequisite (LangGraph without an agentId). The Vercel case asserts the omission against the un-silenceable stderr warning captured from the SAME run, so the test measures the exact co-occurrence the ticket reported rather than the omission alone. Each is paired with a positive control over the same framework and code path — without them an always-empty activeAdapters would satisfy every negative assertion, which is the vacuous-evidence shape Epic AAASM-5526 exists to eliminate. Refs AAASM-5664
…efect This guard resolved `ai` without mocking it, so the real frozen ES module namespace rejected the governed tool factory and the patch warned-and-failed — yet the case asserted activeAdapters === ["vercel-ai-sdk"]. It was pinning the AAASM-5664 defect as if it were the contract, which is why the bug survived a green suite. Assert the truthful split instead: detected and unpatched, not active. The shutdown-iterates-adapters half of the guard is unchanged, since shutdown still walks every registered adapter regardless of patch outcome. Refs AAASM-5664
The guides printed `ctx.activeAdapters` with `["vercel-ai-sdk"]` as an example output. That is now provably wrong for a real install: `ai` ships as a frozen ES module namespace, so its patch cannot take effect and it is reported ungoverned. Documenting it as active is the same over-claim as the code defect, one output surface over — AAASM-5529 requires that a failed adapter cannot present as protected anywhere. Adds the two new context fields to the published AssemblyContext shape and states the detection/governance distinction where each doc introduces it. Versioned docs snapshots are left alone; they describe shipped releases. Refs AAASM-5664
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Review finding B1. `initAssembly` patches a SHALLOW COPY of the caller's config, so `config.langchain` is shared by reference only when the caller supplied it. `registerLangChainHandler` reached its body on detection alone, and `ensureLangChainCallbacks`'s `??=` then minted a fresh object on the copy — the handler was pushed into an array the caller can never read and was discarded when `initAssembly` returned. `AssemblyCallbackHandler`'s constructor only calls `super()`, so no global registration rescued it. `patches.langChainHandler` was truthy regardless, so `langchain-js` was reported active with zero governance: the ticket's own defect surviving in the one branch that is not detection-gated. Reproduced — with `@langchain/core` installed and no `langchain` config, `active = ["langchain-js"]`, `unpatched = []`, and the caller's `config.langchain` is still `undefined` after init. On a LangGraph app that reads `active = ["langchain-js","langgraph-js"]` with `unpatched = []`, i.e. an affirmative all-clear for a run governing nothing. Gate both LangChain paths on caller reachability instead of presence, and warn on stderr like every sibling auto-detected framework already does. The tools path returned an empty list here anyway, but only after pointlessly importing the optional LangChain adapter; its now-unused `frameworks` parameter is dropped. Refs AAASM-5664
…nce" Review findings B2, B3, M1 and M2. The previous round asserted "Treat presence here, and only here, as the signal that in-process governance is in force for that framework". That is false three ways and was newly introduced: LangGraph and Mastra are NON_ENFORCING_MODULES lineage-only (AAASM-4830, and the SDK's own runtime NOTE says so), the LangChain callback layer is audit-only (AAASM-4799), and every enforcing path degrades to a non-blocking check under the allow-all no-op client. Fixing a false claim by writing a stronger false claim one layer in is the exact shape this Epic exists to remove. ADR 0033 §6 forbids undifferentiated verbs like "protects"/"enforces". State the mechanism instead — the patch was applied and is reachable — and give the per-framework breakdown plus the mode caveat, so a reader can tell Observed from Denied-before-execution rather than being handed one crisp word that is wrong for half the adapters. Also replaces the `detectedAdapters` "always a superset" sentence. It was a tautology that read like a guarantee of `detected ⊇ active`, which does not hold: an explicitly-configured adapter is active without being detected, so with an inert `ai` install plus explicit langchain config the intersection is empty. The invariants that do hold are stated, along with the filter idiom that silently drops the one genuinely-wrapped adapter. Refs AAASM-5664
…able
Review findings B4, B2, M1. The guides said "With @openai/agents, ai,
@langchain/langgraph, or @mastra/core installed, this is all that is required to
activate governance for it" — three lines above the previous round's own comment
saying `ai` is reported ungoverned, inside the same fenced block. The sentence
above it ("detects it and patches its execution surface") had the same problem.
Installing a package is not sufficient, and an applied patch buys different
things per framework, so the Status column alone could not carry the truth. Adds
a "What an applied patch actually does" column separating lineage-tagging-only
(LangGraph, Mastra) from denied-before-execution (Vercel, OpenAI Agents, and
LangChain's wrapped tools but not its audit-only callbacks), with the
check-capable-mode caveat as a footnote.
Core concepts gains the matching mechanism-not-enforcement framing and a caution
admonition for the `active ⊄ detected` case, which the previous round documented
for maintainers in a code comment but not for consumers anywhere.
Frozen `website/versioned_docs/` snapshots are deliberately untouched — see the
M4 decision recorded on the ticket.
Refs AAASM-5664
… paths Review findings B1, B2 and M3. Adds a negative control for the B1 shape: `@langchain/core` detected with no `langchain` config must not be active. It guards its own guard by asserting the caller's `config.langchain` is still `undefined` after init, so the test proves the handler really is unreachable rather than trusting the shallow-copy reasoning to keep holding. Adds the positive control for the decision the whole PR rests on — explicit `config.langchain` with `@langchain/core` absent — which had no test anywhere in the repo. Mutation testing showed deleting the two LangChain flags failed only a test that installs the package, i.e. the load-bearing case was unguarded. This one asserts over the artifact that decides: the wrapped tool's `invoke` rejects under a denying gateway and the original implementation is never called, not merely that a string appears in a list. The LangGraph positive control previously pinned membership alone while the surrounding prose called it governance, locking in the B2 over-claim. It now asserts the mechanism actually changed (the StateGraph prototype was replaced) and documents that this is not an enforcement claim. Refs AAASM-5664
Round-2 review. `langgraph-js`/`mastra` were labelled *Observed*, but §6 requires "a durable event attributed to the action" and these patches emit none: `wrapCompiledGraph` is `runWithAgentId(...)` around the original call and `runWithAgentId` is `AsyncLocalStorage.run`. A discriminating probe counts 0 gateway/record/check references in langgraph.ts and mastra.ts against 13 in ai-sdk.ts and 26 in openai-agents.ts. No §6 term fits what they do, so none is used: they are described plainly as binding the agent id so OTHER layers' evidence is attributed to it, with the in-process call stated as *Unmeasured*. Checking the sibling labels to the same standard turned up a second unearned term the review did not flag. The degradation clause said an enforcing path "degrades to *Evaluated*" under the allow-all no-op client, but §6 *Evaluated* requires a decision record from the control plane and `createNoopGatewayClient` returns a bare literal — no decision is produced, so the call is *Unmeasured*, not evaluated-then-ignored. Same standard applied to the LangChain callback layer: it reaches *Observed* only if the client persists what it records, and `record`/`recordResult` are deliberate no-ops on both the no-op client and the napi-inprocess client (AAASM-4847), so that needs a caller-supplied client and now says so. *Denied before execution* is kept for the three paths that earn it — the refusal provably precedes the effect. This text ships to the public API reference (docs/06-api-reference generates from src/index.ts), so an unearned term would have been published product documentation. Refs AAASM-5664
…tgun Round-2 review nit. Both `ensure*` helpers run against the shallow copy `initAssembly` makes of the caller's config, so their `??=` writes into a discarded object whenever the caller did not supply `langchain`. That is what made `activeAdapters` credit a handler nobody could reach (AAASM-5664). The reachability guards that fix it live in the two callers, and nothing in the type system connects them to the helpers — a new call site, or a well-meaning simplification of a guard, silently re-arms it. Document the trap where someone would actually hit it. Refs AAASM-5664
|
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Target
Task summary:
AssemblyContext.activeAdaptersreported a framework as active in the same runin which that framework's patch was warned to have failed.
buildActiveAdapters(
src/core/init-assembly.ts:540-555onmain) unioned the detection list witha set of patch-success flags; since every non-LangChain flag is itself gated on
detection, the union could only ever reproduce the detection list.
activeAdapterswas therefore a pure detection list wearing a governance name.
This PR derives the active list from the patch results alone and splits the
outcome into three explicit fields, so a detected-but-ungoverned framework can
never present as protected.
Task tickets:
Key point change (optional):
The defect was reproducible on
main, in the repo's own test suite.tests/architecture/coverage-regression.test.ts:39assertedactiveAdapters === ["vercel-ai-sdk"]in a run whose stderr read:A green suite was pinning the defect as the contract, which is why it survived.
Two source comments already claimed the behaviour that was never implemented,
which is the clearest evidence the union was an oversight rather than a design:
patchDetectedVercelAiSdk— "this auto-detected path never surfacesvercelAiSdkPatched/ an active adapter for an ungoverned Vercel install …It warns and is excluded from the active adapters."
tests/conformance/fail-open-conformance.test.ts— "excluded fromactiveAdapters(asserted in the unit / init suites)."Both are true as of this PR, and now actually asserted.
As-is → to-be for a real
aiinstall (the common case, frozen ESM):activeAdapters["vercel-ai-sdk"](false)[]detectedAdapters["vercel-ai-sdk"]unpatchedAdapters["vercel-ai-sdk"]Why three fields rather than filtering one. "The framework is not installed"
and "the framework is installed but this SDK is not governing it" are different
facts with different remedies. Collapsing them into an omission from
activeAdapterswould trade one misleading signal for another.The dead-code claim held only in part — worth a reviewer's attention. Four of
the six patch flags (
vercelAiSdkPatched,openAIAgentsPatched,langGraphPatched,mastraPatched) are strictly gated onframeworks.includes(...)and were genuinely dead. The two LangChain flags are not:
registerLangChainHandlerand
wrapLangChainToolsboth bail only on!frameworks.includes("langchain-js") && !config.langchain,so an explicit
config.langchaingoverns callbacks/tools whether or not@langchain/coreresolves.langchain-jsis the one id that can legitimately beactive without being detected, and that half of the union is preserved deliberately.
Effecting Scope
Action Types:
Scopes:
Additional description:
Behavioural change, called out deliberately.
activeAdaptersnarrows to itsdocumented meaning. A caller who read it as "which frameworks did you detect"
will now see fewer entries — most visibly, a real
aiinstall yields[]. Thatnarrowing is the fix, not a side effect: the old value was not true. Anyone who
genuinely wanted detection should read the new
detectedAdapters, so noinformation is lost, only correctly labelled.
AssemblyContextgains two required readonly fields. Because the SDK is the onlyproducer of that interface and consumers read it rather than construct it, this
is additive for every realistic caller; a caller who hand-rolls an
AssemblyContextfor a test double would need to add them.
No consumer inside
src/readsactiveAdapters— the only references are theproducer (
src/core/init-assembly.ts) and the type declaration(
src/types/assembly-context.ts), so nothing internal changes behaviour as aresult of the new semantics.
Versioned Docusaurus snapshots under
website/versioned_docs/are intentionallyleft untouched — they describe already-shipped releases.
Description
🐛 (adapters)— replacebuildActiveAdapterswithbuildAdapterStates, whichderives
activefrompatchedAdapterIds(patches)and returnsdetected/active/unpatched.initAssemblysurfaces all three on the returned context. The twoLangChain flags are preserved for the explicit-
config.langchainpath; the fourdetection-gated flags no longer contribute.
🐛 (adapters)— document the three states on the publicAssemblyContext,including why an omission from
activeAdaptersalone is not a sufficient signal.✅ (test)— newtests/active-adapters-truth.test.ts: two negative controlscovering the two distinct routes to the defect (patch fails — frozen-ESM Vercel;
patch skipped — LangGraph with no
agentId), each paired with a positive controlover the same framework and code path. The Vercel case asserts the omission against
the un-silenceable stderr warning captured from the same run, so it measures the
exact co-occurrence the ticket reported rather than the omission alone. A guard
assertion fails the test if that warning ever stops being emitted, so the fixture
cannot go vacuous silently.
✅ (test)— retargetcoverage-regression.test.tsoff the defect it was pinning.📝 (docs)— correctdocs/03-core-concepts,docs/04-guides, anddocs/05-configuration. The guides printed["vercel-ai-sdk"]as sampleactiveAdaptersoutput, which is provably wrong for a real install — the sameover-claim as the code defect, one output surface over.
Verification
The test was written failing first, and proven to bite by reverting the fix:
AssertionError: expected [ 'vercel-ai-sdk' ] to not include 'vercel-ai-sdk'— positive control already greenexpected [ 'vercel-ai-sdk' ] to not include 'vercel-ai-sdk',expected [ 'langgraph-js' ] to not include 'langgraph-js',expected [ 'vercel-ai-sdk' ] to deeply equal []. Both positive controls stayed green, so the suite cannot pass by emptying the list.Gate exit codes on the final tree:
pnpm test(645 passed, 2 skipped, 70 files)0pnpm typecheck0pnpm lint0prettier --checkon touched files1— pre-existingprettier --checkflagssrc/core/init-assembly.tsand the threedocs/*files.All four fail identically on pristine
remote/main(measured in a detached worktreeso the repo's own
.prettierrcresolved), so this PR neither introduces nor fixesthem;
pnpm lintis eslint-only and is clean. The two files this PR adds or rewriteswholesale are prettier-clean.
Round 2 — independent review (CHANGES-REQUIRED) addressed
All four blocking findings reproduced independently before fixing. Head at review was
a55d243d1; the round-2 work isa8dba5506,1fc22c707,d8e58358f,b306f1f15.B1 —
langchain-jsreported active with zero governance. Fixed.initAssemblypatches a shallow copy of the caller's config, soconfig.langchainisshared by reference only when the caller supplied it.
registerLangChainHandlerreachedits body on detection alone, and
ensureLangChainCallbacks's??=minted a fresh objecton the copy — the handler went into an array the caller can never read, then was
discarded. Reproduced:
@langchain/coreinstalled, nolangchainconfigactive=["langchain-js"],unpatched=[], caller cfgundefinedactive=[],unpatched=["langchain-js"], + stderr warningBoth LangChain paths are now gated on caller reachability rather than presence, and the
detected-but-unwired case warns on stderr like every sibling framework already does. This
was the ticket's own defect surviving in the one branch that is not detection-gated.
B2 — new false capability claim. Fixed. The round-1 JSDoc "Treat presence here, and
only here, as the signal that in-process governance is in force" was false three ways:
LangGraph/Mastra are
NON_ENFORCING_MODULESlineage-only (AAASM-4830 — the SDK's ownruntime NOTE says so), the LangChain callback layer is audit-only (AAASM-4799), and every
enforcing path degrades to a non-blocking check under the allow-all no-op client. Replaced
with a mechanism statement plus the per-framework breakdown and the mode caveat, per ADR
0033 §6's prohibition on undifferentiated verbs. The LangGraph positive control that
pinned the over-claim now asserts the mechanism changed and states that it is not an
enforcement claim.
B3 / M1 — arrays not mutually consistent. Fixed (documented, not "corrected"). The
active ⊄ detectedcase is real and intended: an explicitly-configured adapter is activewithout being detected. Verified
active ∩ detected = ∅with an inertaiinstall plusexplicit langchain config. The tautological "always a superset" sentence is gone; the real
invariant (
unpatched === detected \ active) is stated, along with thedetected.filter(d => active.includes(d))idiom that silently drops the one genuinelywrapped adapter. A caution admonition now carries this to consumers, not just maintainers.
B4 — contradicting prose left standing. Fixed.
docs/04-guidesno longer saysinstalling a package is "all that is required to activate governance", nor that init
"patches its execution surface". Added a "What an applied patch actually does" column
separating lineage-only from denied-before-execution, with the check-capable-mode footnote.
M2 — addressed with B2. M3 — fixed: the explicit-
config.langchain-without-installcase, the sole justification for keeping the two LangChain flags, had no test anywhere.
It now has one that asserts the wrapped tool's
invokerejects under a denying gatewayand the original implementation is never called — not that a string appears in a list.
M4 — decision recorded, not silence.
website/versionChannels.jsonpinslastVersion: 0.0.1-rc.6, so the default-served page is the rc.6 snapshot with the oldprose. Decision: do not edit the frozen snapshot. It correctly documents rc.6, where
activeAdaptersreally did mean "detected"; editing it would make the archive lie aboutthe code it describes. This ticket's fixVersion is rc.7, and cutting rc.7 snapshots the
corrected tree and moves
lastVersionforward, closing the gap. The exposure is thewindow between this merge and the rc.7 cut; a follow-up ticket tracks adding an
admonition to the rc.6 snapshot if that window turns out to be long.
Round-2 mutation testing (both restored, verified by positive grep):
expected [ 'langchain-js' ] to not include 'langchain-js'Round-2 gate exit codes:
pnpm test0 (647 passed / 2 skipped) ·pnpm typecheck0 ·
pnpm lint0 (it caught a genuinely deadframeworksparameter, now removed)·
prettier --checkon the files this PR authors 0.Prettier baseline re-measured the reviewer's way —
git archiveofremote/maininto atree with its own
.prettierrc, repo-wide, both directions: 95 at base, 95 at HEAD,zero new, zero gone. The round-1 wording "prettier fails on 4 files" understated it:
four of the files this PR touches were already failing; the repo-wide pre-existing count
is 95 and prettier gates nothing here (no lefthook/husky/pre-commit hook, no CI step).
Round 3 — the one open review clause, plus a sibling instance it surfaced
Round-2 review verified all four blockers and three of four majors resolved, leaving one
open clause. Commits
12be7ec6f,12232ad77.Open finding —
*Observed*unearned forlanggraph-js/mastra. Fixed. ADR 0033 §6Observed requires "a durable event attributed to the action". These patches emit none:
wrapCompiledGraphisrunWithAgentId(...)wrapping the original call, andrunWithAgentIdisAsyncLocalStorage.run. Re-derived with a discriminating probe:No §6 term describes what these adapters do, so none is used. They are stated plainly —
they bind the agent id so that other layers' evidence is attributed to it — with the
in-process call itself called §6 Unmeasured, and the ADR's own scoping caveat that
this says nothing about the proxy/eBPF layers.
Sibling instance found while checking the other labels to the same standard. The
degradation clause claimed an enforcing path "degrades to Evaluated" under the
allow-all no-op client. §6 Evaluated requires a decision record from the control
plane;
createNoopGatewayClient.checkreturns a bare literal, so no decision isproduced and the call is Unmeasured, not evaluated-then-ignored. Corrected.
Applying the standard to the remaining labels also tightened the LangChain callback
claim: it reaches Observed only when the client persists what it records, and
record/recordResultare deliberate no-ops on both the no-op client and thenapi-inprocessclient (AAASM-4847) — so it requires a caller-supplied client, whichthe docs now say. Denied before execution is retained for the three paths that earn
it; verified that
createWrappedExecutethrows beforeexecuteOriginal()is invokedand
openai-agentsreturns beforeoriginalInvoke.call.This mattered because
docs/06-api-referencegenerates the public API reference fromsrc/index.tsTSDoc — an unearned term would have shipped as product documentation.Both review nits taken. The
ensureLangChain*??=footgun is now documented wherea future caller would hit it, and the M4 follow-up is tracked as
AAASM-5676.
Round-3 gate exit codes:
pnpm test0 (647 passed / 2 skipped) ·pnpm typecheck0 ·
pnpm lint0 ·prettier --checkon PR-authored files 0; repo-wideset-difference against
remote/mainempty both directions (95 → 95).Out of scope and filed separately: AAASM-5663 (README non-functional at both
entrypoints) and AAASM-5665 (no audit evidence for denied calls, no agent identity
on the check path — touches a wire contract, owner decision).