[AAASM-5681] 🐛 (gateway): Declare and surface that shipped clients discard audit events - #363
Merged
Merged
Conversation
…s with audit events `record` / `recordResult` / `scanPrompts` all return `Promise<void>`, so a client that retains a hook-layer audit event and one that drops it are indistinguishable to the caller. This type gives the difference a name. Two-valued rather than boolean on purpose: the SDK can only speak for the clients it builds, so `"caller-supplied"` is the absence of a claim, not an assurance that events are retained. Refs AAASM-5681
Measured on remote/main: with a stub at the native boundary, `check()` crosses it and `record` / `recordResult` / `scanPrompts` cross nothing, while the only signal was a one-shot note gated on AA_DEBUG=1 — so a caller had to already suspect the drop to discover it. Both shipped clients now declare `auditSink: "discarded"`. The field is optional on the interface so a caller's own client keeps compiling, but a client this package ships must never be silent about it. Refs AAASM-5681
A caller configuring governance should not have to opt in to learn that their audit trail does not exist. `initAssembly` now warns once on stderr when the resolved client is known to discard, and reports `auditSink` on the returned context as the programmatic counterpart — mirroring how `registered` pairs with its init-time warning. Does not fail init: the proxy and eBPF layers are unaffected and a caller may not need SDK-side audit. A caller-supplied client is reported as `"caller-supplied"` and does not warn, because the SDK cannot speak for it. Refs AAASM-5681
Pins three things separately, because any one alone passes while the defect is present: every shipped client declares a disposition; a client declaring "discarded" reaches nothing, measured against a boundary a positive control proves is reachable; and initAssembly surfaces the drop with AA_DEBUG unset, on stderr and as context.auditSink. The caller-supplied case is a negative control, so the warning assertions cannot pass on a build that warns unconditionally. Refs AAASM-5681
…ault The introduction said governance-relevant actions are "recorded in an audit trail", the quick-start said an allowed call's event "is recorded in the audit trail", and the README said the LangChain callback layer "records denials and results for the audit trail". Measured against both shipped clients, none of the three holds on the default path. Each now states that the event is emitted, that retention depends on the gateway client, and how to tell which one you have. Refs AAASM-5681
…ible `AssemblyContext` is exported from this package's public entry (`exports["."].types` → dist/types/index.d.ts) and is a plain interface, so a consumer can construct one in a test fixture or custom integration. A required field would be a compile break for published consumers, introduced by a diagnostic field. The guarantee does not weaken: it rests on every shipped client declaring its disposition, not on contexts carrying it. `initAssembly` still always populates the field. Refs AAASM-5681
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
The new warning fired on every default init saying "Enforcement is unaffected — allow/deny still applies". On auto / sdk-only / grpc-sidecar `createClient` returns the no-op client, whose check() is allow-all, so that is false exactly where the warning is loudest. It also landed immediately after the AAASM-1847 line saying a policy DENY will NOT block a tool — and that line only fires when a framework was patched, so in a bare consumer app mine was the only posture claim printed. The true statement is that the audit drop does not change the enforcement posture, not that enforcement applies. Corrected in the warning, the disposition type, the client comment and the introduction. Refs AAASM-5681
README's "denies are still recorded but not enforced" sits in the same LangChain section as the bullet I corrected, and asserts exactly the retention documented as absent seven lines above it. Missed because I enumerated three claims and stopped, rather than re-reading the section I had just edited. Also narrows the introduction's "who this is for" bullet, which offered an audit trail as a default-path capability. Refs AAASM-5681
The PR body claimed a future third shipped client that discards without declaring would fail the suite. It would not: SHIPPED_CLIENTS was a hand-maintained literal with no exhaustiveness mechanism, and review demonstrated an exported, discarding, undeclared client sailing through at 9 passed (9). The suite now reads the module's own `create*GatewayClient` exports, so a new factory is covered the moment it exists. Reproduced review's exact counter-example: both new tests fail, naming the offending client. Refs AAASM-5681
…i-inprocess
Attempt 1 removed the claim on three modes and created its inverse on the
fourth: the warning interpolated ${mode} into a clause asserting the
allow-all no-op client, but it fires for BOTH shipped clients, so in
napi-inprocess it told operators running the documented in-process
enforcement mode that their client cannot block — while createClient
returns the native client there, which denies against a denying runtime.
The clause now branches on CHECK_CAPABLE_MODE. Also removes the dangling
`Supply ` + fragment left by that attempt.
Two tests pin the text per mode, driving napi-inprocess through the mocked
native binding the way create-client-mode-routing.test.ts does. Both
inversions are now caught: forcing the no-op clause everywhere reddens the
napi test, forcing the check-capable clause everywhere reddens the auto
test. Nothing pinned this before, which is why it shipped wrong twice.
Refs AAASM-5681
… one Carving out only audit implied by contrast that enforcement and redaction ARE default-path properties. Neither is: auto / sdk-only / grpc-sidecar route checks through the allow-all no-op client, and redaction is applied by the runtime/proxy — under enforce this layer treats a redact verdict as allow, per docs/05-configuration. Same shape as the previous round's finding, one bullet over: correcting one item of a list inside a line I was already editing, and leaving its neighbours contradicted by the same page. The list was the unit to check. Refs AAASM-5681
…s build The cross-link added for the redaction bullet omitted `index.md`, so Docusaurus resolved `../05-configuration/` to `/node-sdk/05-configuration/` and failed the build with a broken link. Every other cross-link in docs/, including one on this same page, uses `../<section>/index.md`. Verified against the gate rather than by inspection: the site builds clean locally, and reverting the link reproduces CI's exact error before the build passes again. Refs AAASM-5681
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Description
Both gateway clients this SDK ships discard hook-layer audit events —
record,recordResult,scanPrompts— and the only signal was a one-shot note gated onAA_DEBUG=1. A caller had to already suspect the problem in order to discoverit. Enforcement is unaffected, which is exactly why the drop is easy to miss:
denies still deny, and every audit call resolves successfully.
This PR does not build an audit sink. It makes the absence declared, surfaced
and tested, and corrects the documentation that claimed otherwise.
The measured absence
Run against the shipped clients (through
createClient, the same factoryinitAssemblyuses), with a stub placed at the native boundary — notsubstituted for the code under test:
A payload of
"SECRET-PAYLOAD"reached nothing, and withAA_DEBUGunset allthree audit methods on both clients produced zero signal.
The positive control, which is what makes that zero admissible
check()crosses the same boundary the audit methods do not. Without this,crossings: []would be indistinguishable from a probe that never ran — which ishow this defect survived its original tests, whose assertions were over mocks of
the thing under test.
Structure verified independently of the ticket: two shipped factories, and
init-assembly.ts:217/226are the only non-test construction sites.Claim vocabulary: Planned, not
UnmeasuredADR 0033 §6 reserves
Unmeasuredfor "no control inspected this action orpayload; nothing is known about it." That is not this case. The event is
constructed, passed, and dropped, and the measurement above says so exactly. A
measured zero is not "nothing is known", so
Unmeasuredwould be wrong here —noted explicitly because it is the kind of label that gets "corrected" later.
Planned is the fit: "Decided but not implemented — a ticket reference; no
capability claim." The FFI record path is a separate new capability and
AAASM-5681 is the reference. The type documents it in those terms.
Observed for the hook layer holds only on the caller-supplied branch. That
is why
AuditSinkDispositionis two-valued rather than a boolean:"caller-supplied"is documented as the absence of a claim, not anassurance — this SDK cannot speak for a client it did not build. Same
discipline as
assertedversusboundin the gateway'sagentIdattribution(AAASM-5665 / #2016): the receiving side may record what was claimed, and may
record that it could not corroborate it, but may not round the two together.
Type of Change
Breaking Changes
Does this PR introduce any breaking changes to public APIs or behaviour?
Two additive fields, both optional, and one new behaviour that does not fail
init:
GatewayClient.auditSink?— optional so a caller's own client keeps compiling.AssemblyContext.auditSink?— optional deliberately.AssemblyContextisexported from this package's public entry (
exports["."].types→dist/types/index.d.ts) and is a plain interface with no brand or privatemember, so a consumer can construct one in a test fixture or custom
integration. Requiring the field would be a compile break for published
consumers, introduced by a diagnostic field, in a ticket about truthfulness.
initAssemblyalways populates it.initAssemblynow writes one warning to stderr when the resolved client isknown to discard. It does not fail init: the proxy and eBPF layers are
unaffected and a caller may not need SDK-side audit.
The guarantee does not rest on either optional field. It rests on every
shipped client declaring its disposition, which the test suite enforces.
Related Issues
Testing
pnpm test659 passed / 2 skipped, exit 0 ·pnpm lintexit 0 ·pnpm typecheckexit 0 ·pnpm buildexit 0. pnpm 10.33.2, verified frominside the repo.
tests/audit-sink-disposition.test.tspins three things separately, because anyone of them alone passes while the defect is present:
"discarded"reachesnothing, measured against a boundary a positive control proves is reachable;
initAssemblysurfaces the drop on the default path withAA_DEBUGunset, on stderr and as
context.auditSink.It is structural, not behavioural-only — and that is now true rather than
asserted. The suite reads the module's own
create*GatewayClientexports ratherthan a hand-maintained literal, so a new factory is covered the moment it exists.
Review demonstrated the original claim was false: an exported, discarding,
undeclared
createThirdGatewayClientpassed at9 passed (9). Reproduced hereas N7 — both exhaustiveness tests now fail and name the offending client.
Mutation evidence
Six mutations, run serially, each in one foreground command with a
trapso therestore is guaranteed, each verified by SHA-256 against the pre-mutation
baseline. Every one reddened its test; final residue check
0in both files.1 failed | 1 passed1 failed | 1 passed"discarded"but actually crosses the boundary1 failed2 failed | 9 passed"caller-supplied"2 failed | 9 passed1 failed2 failed | 9 passedcreateGatewayClientFromEnvto dodge the name pattern2 failed | 11 passed1 failed | 12 passed1 failed | 12 passedN1 and N2 are the discriminating pair. The parameterised test has one row per
shipped client, and each mutation reddens only its own row — neither client
passes on the other's coverage.
N3 is the mutation that makes the suite worth having. It pins declaration
against behaviour, so a client cannot claim
"discarded"while leaking, norclaim it records while dropping.
N6 kills the negative control. Without it, the warning assertions would still
pass on a build that warns unconditionally — a different untruth.
Documentation — the product-truth half
Three claims in this repo asserted that audit evidence is produced by default.
Measured against both shipped clients, none holds on the default path. Corrected
separately from the code change:
docs/01-introduction/index.md— "every governance-relevant action is recordedin an audit trail".
docs/02-quick-start/index.md— an allowed call's governance event "isrecorded in the audit trail".
README.md— the LangChain callback layer "records denials and results for theaudit trail".
Each now states that the event is emitted, that retention depends on the
gateway client, and how to tell which one you have.
Design note
The init-time warning plus programmatic flag is not a new pattern — it mirrors
warnAgentUnregistered, which pairs an unconditional stderr write with theregisteredfield for exactly this reason. Written toprocess.stderrratherthan
console/a swappable logger so it cannot be silenced by log configuration,and fired at init rather than per audit call so it cannot become steady-state
noise.
Checklist
pnpm lint,pnpm typecheck)git commit -s) — DCO sign-off is optional/advisory here, not enforcedDeliberately not done
scoping, not a bug fix. It is why the honest term is Planned.
go-sdkandpython-sdkare untouched. Both share the defect per thisticket's own comments, and Python is worse — its hooks do not resolve at all,
so nothing is emitted for allowed calls either. Reported rather than widened;
filed separately.
condition live in
agent-assembly/governance/**, owned by a concurrent change.wrapToolWithAssemblydeny path — is separateand not folded in.
Round 2 — review fixes
1. The warning shipped a false claim on the default path. It said
"Enforcement is unaffected — allow/deny still applies" on every default init.
On
auto/sdk-only/grpc-sidecarcreateClientreturns the no-op client,whose
check()is allow-all — measured:{"denied":false,"pending":false}for atool named
rm-rf-slash. It also landed immediately after the AAASM-1847 linesaying a policy DENY will not block a tool, contradicting it in consecutive
lines; and since that line only fires when a framework was patched, in a bare
consumer app mine was the only posture claim printed. Corrected in the warning,
gateway-governance.ts,client.tsand the introduction to say what is true:the audit drop does not change the enforcement posture, which is not the same as
enforcement applying.
2. A fourth uncorrected retention claim.
README.md— "denies are stillrecorded but not enforced" — in the same LangChain section, seven lines below
the bullet I had just corrected. Missed because I enumerated three claims and
stopped rather than re-reading the section I had edited.
3. The structural claim. Made true rather than softened; see N7 above.
Finding 4 — why
auditSinkis optional when three siblings are requireddetectedAdapters,unpatchedAdaptersandregisteredwere all added toAssemblyContextas required, all diagnostic, same rationale. Theinconsistency is real and worth stating rather than leaving unremarked.
The reason for diverging: review verified against the published npm tarball
that
AssemblyContextis publicly constructible, and ran the counterfactualcompile — exactly one differing line,
TS2741: Property 'auditSink' is missing.So the compile break is demonstrated, not hypothesised. The three siblings were
added before that was established; whether they were breaks was never measured.
Given a proven break against a published package, in a ticket whose subject is
truthfulness, the smaller harm is the inconsistency. The guarantee does not live
on this field in either case — it lives on shipped clients declaring, which N1,
N2, N3 and N7 enforce. If the maintainers prefer consistency with the three
siblings, making it required is a one-line change and I will take that direction.
Finding 5 — which remaining doc claims are hook-layer and which are runtime
Checked, and they split:
docs/01-introduction/index.mdoffered "an audittrail without rewriting your agent code" as a default-path capability thirty
lines below the corrected paragraph. It is the SDK's own claim and it is now
narrowed.
docs/02-quick-start/index.mdanddocs/05-configuration/index.mddescribe observe mode recording shadow auditevents (
aa audit list --dry-run-only). Those are written by the gateway —transform_for_observe_modeproduces aShadowEventthat flows into thegateway's
record_audit— on a different channel from the SDK's hook-layerrecord/recordResult. They areenforcementModesemantics, not an SDKretention claim, so correcting them here would introduce an error rather than
remove one.
Ticket status
ACs 3 and 4 — the documentation estate across all three SDKs, and the
capability-manifest rows carrying the caller-supplied condition — are not met
by this PR and are out of its scope. AAASM-5681 stays open after this merges.
Round 3 — review fixes
1. The corrected warning was false in
napi-inprocess. Attempt 1 removed theclaim on three modes and created its inverse on the fourth: the clause
interpolated
${mode}but asserted the allow-all no-op client, and the warningfires for both shipped clients. So the one mode that genuinely enforces was
told it could not. The clause now branches on
CHECK_CAPABLE_MODE, and thedangling
`Supply ` +fragment is gone.This claim had shipped wrong twice with nothing pinning it, so it now has a
discriminating pair: N9 forces the no-op clause everywhere and reddens the
napi-inprocesstest; N10 forces the check-capable clause everywhere andreddens the
autotest. Neither could pass on the other's coverage. Thenapi-inprocesstest drives the realcreateClientrouting through a mockednative binding, following
create-client-mode-routing.test.tsrather thanasserting on a hand-built client.
2. The "who this is for" list. Carving out only audit implied by contrast that
enforcement and redaction are default-path properties. Neither is — enforcement
only in a check-capable run, and redaction is applied by the runtime/proxy, with
this layer treating a
redactverdict as allow underenforce. All three itemsnow state their real scope. Same shape as the previous round's finding, one
bullet over: when I correct one item of a list, the list is the unit.
3. Detection is now by shape, not by name.
exportedClientFactoryNamesteststhe returned object for
check/record/recordResult/scanPromptsinstead of matching
/^create[A-Za-z]*GatewayClient$/. Reproduced review'sescape as N8:
createGatewayClientFromEnvpassed at11 passed (11)underthe name pattern and now fails, named in the message.
Commit-hygiene disclosure: fix 3 landed inside
0c67f548e, whose messagedescribes only fix 1 — both touch the same test file and I staged them together.
Two logical units in one commit, and the message under-describes it. Flagging
rather than leaving it for a reader to notice.