[AAASM-5665] 🐛 (wrappers): Emit an audit event when a tool call is denied - #355
Merged
Merged
Conversation
withAssembly called gateway.check and never gateway.record, so a denied call produced no audit event at all — the thrown PolicyViolationError was the only trace and it never leaves the process. Record the deny before throwing, on both the policy-deny and the approval-rejected path. GatewayRecordEvent has no tool-name field, so the tool is named inside reason by reusing the error's own message. Rejections are swallowed so a failing sink cannot mask the enforcement decision. Refs AAASM-5665
Add a control over the audit event withAssembly hands the gateway on a deny: the action, the run id correlating it with the decision, and the tool named in the reason. Assert auditResults stays empty so an implementation recording a bogus empty result cannot pass. Refs AAASM-5665
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
recordDeny attached .catch() to record()'s returned promise, so a
caller-supplied record() that throws synchronously produced no promise to
attach to and escaped past the PolicyViolationError — the audit failure
became the caller's error and a `catch (e) { e instanceof
PolicyViolationError }` no longer recognised the deny.
gatewayClient is a documented public injection point, so this is reachable
user code. Wrap the call instead, matching the Python companion's
try/except around the same hook (AAASM-4782). Also correct the docstring:
this is awaited, not the fire-and-forget idiom it claimed to match.
Refs AAASM-5665
The policy-deny and approval-rejected routes each carried their own recordDeny call. Review found the approval-rejected copy could be deleted outright with the whole suite still green — five tests executed the line and none asserted on it, so codecov reported it covered. Converge both routes on a single record-then-throw, mirroring the Python companion's single merged path. One call site cannot drift from itself. Refs AAASM-5665
createPendingThenRejectGatewayClient drives the second refusal route; createFailingRecordGatewayClient fails record() either as a rejected promise or as a synchronous throw, and counts attempts so a control can tell "the sink failed" from "the sink was never called". Refs AAASM-5665
Two call sites went unasserted. The approval-rejected audit action was executed by five tests and checked by none, and no control covered a record() that fails. Also rename the describe: it called the fixture's in-process array "the audit sink", the same over-claim that blocked the go-sdk PR, in the string CI prints. It now names the call, which is what the block reads. Refs AAASM-5665
…5/fix/deny_audit_record
The approval-rejected control always supplied an approver reason, leaving the wrapper's `?? "Rejected"` fallback a partial branch — the one line codecov flagged. A gateway is not obliged to explain a rejection, and attribution has to survive that: the tool name comes from the wrapper's own message, not the gateway's text. Run the control both ways. Refs AAASM-5665
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Target
Task summary:
withAssemblycalledgateway.checkand nevergateway.record, so a denied tool callproduced no audit event at all — the thrown
PolicyViolationErrorwas the only trace andit never leaves the process. In the negative-control fixture
auditEventsandauditResultswere both length
0whiledecisionswas1: a deny existed only in the decision log.This records the deny before throwing, on both the policy-deny and the approval-rejected path.
Task tickets:
go-sdk📸 (docs): Cut docs snapshot for 0.0.1-beta.5 #195 andpython-sdk[AAASM-4899] 📝 (config): Canonical AA_* env vars in TSDoc + fix ESLint docs glob #311 — same ticket, same finding, same three-SDK gap.AAASM-5664) is open on this repo. No overlap: this branch touches onlysrc/wrappers/with-assembly.tsandtests/quickstart-negative-control.test.ts.Key point change:
The audit sink is
Unmeasured. Both shippedGatewayClientimplementations discard theevent:
recordcreateNoopGatewayClient(src/gateway/client.ts:42)async () => undefinedcreateNativeGatewayClient(src/gateway/client.ts:218)AA_DEBUGstderr note, then droppedsrc/core/init-assembly.tswires one of those two; there is no third. Per ADR 0033 §6 thehonest term for a deny on the shipped path is Unmeasured in audit evidence. This PR makes
the call site correct — the part fixable without new capability. AAASM-5681 tracks the
sink and is the blocking dependency. The same gap is now confirmed in
go-sdk(its onlyproduction
GovernanceClientdiscards the record) andpython-sdk(the SDK's interceptorresolves no audit hook at all), so this is a three-SDK issue, not a Node quirk.
Two ticket claims did not survive verification —
mainhas moved since AAASM-5665 waswritten:
describeblock titled "deny is attributable in audit evidence" readsthe decision log. No such block exists. The current title is "what a deny is and is
not attributed to", and its test is already scrupulous — it documents the
agentIdgap,explains that the old assertion was vacuous, and carries a tripwire on the outbound request
shape. AC3 was already satisfied here, so this PR changes none of it.
withAssembly/recordclaim is confirmed: 0 call sites, against positivecontrols of 4 in
assembly-callback-handler.tsand 1 each inhooks/ai-sdk.tsandhooks/openai-agents.ts, plus.check(at:157in the same file.Design notes.
GatewayRecordEventhas no tool-name field and adding one is a wire change(out of scope), so the tool is named inside
reasonby reusing the thrown error's ownmessage — the audit event and the error a caller sees cannot drift.
Round 2 — three findings fixed.
tool_call_approval_rejectedcopy could be deleted outright with the whole suite bit-identical — five tests executed
the line and none asserted on it, so
codecov/patchreported 100%. Execution is notpinning. Both routes now converge on a single record-then-throw (mirroring the Python
companion, which avoided this by never duplicating), and the approval-rejected action
gets its own control.
.catch()attaches to the returnedpromise; a
recordthat throws synchronously produces none, so the audit error replacedthe
PolicyViolationError.gatewayClientis a documented public injection point, so thisis reachable user code. Now a
try/catch, matching the Python companion'stry/except(AAASM-4782).
recordDenyis awaited, not fire-and-forget. The docstring claimed to matchrecordToolResultNonBlocking(ai-sdk.ts:106,openai-agents.ts:173) — which isvoid-returning and deliberately un-awaited. It does not. The comment now states the realtradeoff: awaiting hands the event over before the throw so it cannot be lost, at the cost
of no timeout on
record, unlikewaitForApprovalWithTimeouta few lines below.Out of scope, deliberately.
wrapToolWithAssembly(the LangChain wrapper) has the samehole and is not touched — the ticket's evidence names
withAssembly, and the LangChain pathhas its own audit story via the callback handler's
policy_post_block. Worth its own ticket.No
agentIdon the check path either;WithAssemblyOptions.agentIdstays declared-and-unreadand
docs/02-quick-start/index.md:140is untouched, per the owner decision pending on thewire contract.
Effecting Scope
Action Types:
Scopes:
Additional description:
pnpm test0— 650 passed, 2 skipped (71 files) after merging the newmain; +3 controls from this PRpnpm typecheck0pnpm lint0No pre-existing failures to baseline — every gate is green on untouched
mainand on thisbranch.
Evidence the control bites, same file each run, 10 positive controls alongside:
with-assembly.tsexpected [] to have a length of 11 failed | 10 passed11 passedaction→"tool_call_check"(event still emitted)expected 'tool_call_check' to be 'tool_call_denied'1 failed | 10 passedreason→"Denied"(tool name dropped)expected 'Denied' to contain 'write_file'1 failed | 10 passedThe two mutations fail at different assertions, so the control discriminates rather than
tripping on one coarse check. The suite cannot pass by emitting nothing.
Round-2 mutations, re-run on the merged call site — the first is the exact scenario that
was invisible in round 1:
action→"tool_call_denied"recordDenycall deletedtry/catchreverted to a trailing.catch()Each mutation fails a different, specific control, and the failing-sink control asserts
recordAttempts === 1so it cannot pass on a path that never callsrecordat all.Rebased onto the new
main. [AAASM-5664] 🐛 (adapters): Report only successfully patched adapters as active #354 merged while this was in review, somainmoved toa9bc52b4. The branch was brought up to date by mergingmainin, not rebasing — a rebasewould require a force-push. No overlap: [AAASM-5664] 🐛 (adapters): Report only successfully patched adapters as active #354 touched
src/core/init-assembly.ts,src/types/assembly-context.ts,docs/, and two test files, none of which this PR touches.One further codecov round. After the round-2 fixes
codecov/patchread90.90%— "0Missing and 1 partial". Not a missing line: a partial branch,
?? "Rejected"atwith-assembly.ts:227, because the approval-rejected fixture always supplied a reason. Worthcovering rather than waiving — a gateway is not obliged to explain a rejection, and attribution
has to survive that, since the tool name comes from the wrapper's own message and not the
gateway's text. The control now runs both ways and the branch is taken. Now
100.00%.Method note: the line-level story only resolved after reading codecov's PR comment ("0
Missing and 1 partial") and the
BRDA:records inlcov.info— the summary percentage and aline-hit scan both said "all added lines hit", which was true and still not the whole answer.
What this does and does not deliver
Even fully fixed, this PR delivers a correctly wired call site, honestly labelled
Unmeasured— which is not the same as audited. AC1 is met on its second branch only ("or the acceptance
criteria that depend on audit evidence are restated against the artifact that actually exists").
Nothing here makes a denied call observable in a released binary; AAASM-5681 remains the blocking
dependency, across all three SDKs.
Description
src/wrappers/with-assembly.ts— newrecordDenyhelper; called before the throw on thepolicy-deny and approval-rejected paths.
tests/quickstart-negative-control.test.ts— new control asserting the emitted audit event'saction, run-id correlation, and tool attribution, plus that
auditResultsstays empty.