Skip to content

[AAASM-5665] 🐛 (wrappers): Emit an audit event when a tool call is denied - #355

Merged
Chisanan232 merged 8 commits into
mainfrom
v0.0.1-rc.7/AAASM-5665/fix/deny_audit_record
Aug 7, 2026
Merged

[AAASM-5665] 🐛 (wrappers): Emit an audit event when a tool call is denied#355
Chisanan232 merged 8 commits into
mainfrom
v0.0.1-rc.7/AAASM-5665/fix/deny_audit_record

Conversation

@Chisanan232

@Chisanan232 Chisanan232 commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Target

  • Task summary:

    withAssembly called gateway.check and never gateway.record, so a denied tool call
    produced no audit event at all — the thrown PolicyViolationError was the only trace and
    it never leaves the process. In the negative-control fixture auditEvents and auditResults
    were both length 0 while decisions was 1: a deny existed only in the decision log.

    This records the deny before throwing, on both the policy-deny and the approval-rejected path.

    Scope of the claim — read before approving. This does not make a denied call
    observable in a released binary, and the PR does not say it does. See Key point change.
    This takes AC1's second branch: "or the acceptance criteria that depend on audit evidence
    are restated against the artifact that actually exists."

  • Task tickets:

  • Key point change:

    The audit sink is Unmeasured. Both shipped GatewayClient implementations discard the
    event:

    Implementation record
    createNoopGatewayClient (src/gateway/client.ts:42) async () => undefined
    createNativeGatewayClient (src/gateway/client.ts:218) one-time AA_DEBUG stderr note, then dropped

    src/core/init-assembly.ts wires one of those two; there is no third. Per ADR 0033 §6 the
    honest term for a deny on the shipped path is Unmeasured in audit evidence. This PR makes
    the call site correct — the part fixable without new capability. AAASM-5681 tracks the
    sink and is the blocking dependency. The same gap is now confirmed in go-sdk (its only
    production GovernanceClient discards the record) and python-sdk (the SDK's interceptor
    resolves no audit hook at all), so this is a three-SDK issue, not a Node quirk.

    Two ticket claims did not survive verificationmain has moved since AAASM-5665 was
    written:

    1. The ticket says a describe block titled "deny is attributable in audit evidence" reads
      the decision log. No such block exists. The current title is "what a deny is and is
      not attributed to"
      , and its test is already scrupulous — it documents the agentId gap,
      explains that the old assertion was vacuous, and carries a tripwire on the outbound request
      shape. AC3 was already satisfied here, so this PR changes none of it.
    2. The withAssembly / record claim is confirmed: 0 call sites, against positive
      controls of 4 in assembly-callback-handler.ts and 1 each in hooks/ai-sdk.ts and
      hooks/openai-agents.ts, plus .check( at :157 in the same file.

    Design notes. GatewayRecordEvent has no tool-name field and adding one is a wire change
    (out of scope), so the tool is named inside reason by reusing the thrown error's own
    message — the audit event and the error a caller sees cannot drift.

    Round 2 — three findings fixed.

    1. The two deny routes are now one call site. Round 1 found the tool_call_approval_rejected
      copy could be deleted outright with the whole suite bit-identical — five tests executed
      the line and none asserted on it, so codecov/patch reported 100%. Execution is not
      pinning.
      Both routes now converge on a single record-then-throw (mirroring the Python
      companion, which avoided this by never duplicating), and the approval-rejected action
      gets its own control.
    2. A synchronously-throwing sink no longer escapes. .catch() attaches to the returned
      promise
      ; a record that throws synchronously produces none, so the audit error replaced
      the PolicyViolationError. gatewayClient is a documented public injection point, so this
      is reachable user code. Now a try/catch, matching the Python companion's try/except
      (AAASM-4782).
    3. recordDeny is awaited, not fire-and-forget. The docstring claimed to match
      recordToolResultNonBlocking (ai-sdk.ts:106, openai-agents.ts:173) — which is
      void-returning and deliberately un-awaited. It does not. The comment now states the real
      tradeoff: awaiting hands the event over before the throw so it cannot be lost, at the cost
      of no timeout on record, unlike waitForApprovalWithTimeout a few lines below.

    Out of scope, deliberately. wrapToolWithAssembly (the LangChain wrapper) has the same
    hole and is not touched — the ticket's evidence names withAssembly, and the LangChain path
    has its own audit story via the callback handler's policy_post_block. Worth its own ticket.
    No agentId on the check path either; WithAssemblyOptions.agentId stays declared-and-unread
    and docs/02-quick-start/index.md:140 is untouched, per the owner decision pending on the
    wire contract.

Effecting Scope

  • Action Types:

    • 🔧 Fixing bug
      • 🟢 No breaking change
  • Scopes:

    • 🧩 SDK public API
    • 🧪 Testing
      • 🧪 Unit testing
  • Additional description:

    Gate Exit
    pnpm test 0650 passed, 2 skipped (71 files) after merging the new main; +3 controls from this PR
    pnpm typecheck 0
    pnpm lint 0

    No pre-existing failures to baseline — every gate is green on untouched main and on this
    branch.

    Evidence the control bites, same file each run, 10 positive controls alongside:

    State New control Positive controls Result
    Unfixed with-assembly.ts FAILexpected [] to have a length of 1 PASS 1 failed | 10 passed
    Fixed PASS PASS 11 passed
    Mutation: action"tool_call_check" (event still emitted) FAILexpected 'tool_call_check' to be 'tool_call_denied' PASS 1 failed | 10 passed
    Mutation: reason"Denied" (tool name dropped) FAILexpected 'Denied' to contain 'write_file' PASS 1 failed | 10 passed

    The two mutations fail at different assertions, so the control discriminates rather than
    tripping on one coarse check. The suite cannot pass by emitting nothing.

    Round-2 mutations, re-run on the merged call site — the first is the exact scenario that
    was invisible in round 1:

    Mutation Result
    approval-rejected action"tool_call_denied" 1 failed | 12 passed — only the approval-rejected control
    merged recordDeny call deleted 3 failed | 10 passed — all three deny controls
    try/catch reverted to a trailing .catch() 1 failed | 12 passed — only the failing-sink control

    Each mutation fails a different, specific control, and the failing-sink control asserts
    recordAttempts === 1 so it cannot pass on a path that never calls record at all.

    Rebased onto the new main. [AAASM-5664] 🐛 (adapters): Report only successfully patched adapters as active #354 merged while this was in review, so main moved to
    a9bc52b4. The branch was brought up to date by merging main in, not rebasing — a rebase
    would require a force-push. No overlap: [AAASM-5664] 🐛 (adapters): Report only successfully patched adapters as active #354 touched src/core/init-assembly.ts,
    src/types/assembly-context.ts, docs/, and two test files, none of which this PR touches.

    One further codecov round. After the round-2 fixes codecov/patch read 90.90%"0
    Missing and 1 partial"
    . Not a missing line: a partial branch, ?? "Rejected" at
    with-assembly.ts:227, because the approval-rejected fixture always supplied a reason. Worth
    covering rather than waiving — a gateway is not obliged to explain a rejection, and attribution
    has to survive that, since the tool name comes from the wrapper's own message and not the
    gateway's text. The control now runs both ways and the branch is taken. Now 100.00%.

    Method note: the line-level story only resolved after reading codecov's PR comment ("0
    Missing and 1 partial") and the BRDA: records in lcov.info — the summary percentage and a
    line-hit scan both said "all added lines hit", which was true and still not the whole answer.

What this does and does not deliver

Even fully fixed, this PR delivers a correctly wired call site, honestly labelled Unmeasured
— which is not the same as audited. AC1 is met on its second branch only ("or the acceptance
criteria that depend on audit evidence are restated against the artifact that actually exists"
).
Nothing here makes a denied call observable in a released binary; AAASM-5681 remains the blocking
dependency, across all three SDKs.

Description

  • src/wrappers/with-assembly.ts — new recordDeny helper; called before the throw on the
    policy-deny and approval-rejected paths.
  • tests/quickstart-negative-control.test.ts — new control asserting the emitted audit event's
    action, run-id correlation, and tool attribution, plus that auditResults stays empty.

withAssembly called gateway.check and never gateway.record, so a denied
call produced no audit event at all — the thrown PolicyViolationError was
the only trace and it never leaves the process.

Record the deny before throwing, on both the policy-deny and the
approval-rejected path. GatewayRecordEvent has no tool-name field, so the
tool is named inside reason by reusing the error's own message. Rejections
are swallowed so a failing sink cannot mask the enforcement decision.

Refs AAASM-5665
Add a control over the audit event withAssembly hands the gateway on a
deny: the action, the run id correlating it with the decision, and the
tool named in the reason. Assert auditResults stays empty so an
implementation recording a bogus empty result cannot pass.

Refs AAASM-5665
@codecov

codecov Bot commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

recordDeny attached .catch() to record()'s returned promise, so a
caller-supplied record() that throws synchronously produced no promise to
attach to and escaped past the PolicyViolationError — the audit failure
became the caller's error and a `catch (e) { e instanceof
PolicyViolationError }` no longer recognised the deny.

gatewayClient is a documented public injection point, so this is reachable
user code. Wrap the call instead, matching the Python companion's
try/except around the same hook (AAASM-4782). Also correct the docstring:
this is awaited, not the fire-and-forget idiom it claimed to match.

Refs AAASM-5665
The policy-deny and approval-rejected routes each carried their own
recordDeny call. Review found the approval-rejected copy could be deleted
outright with the whole suite still green — five tests executed the line
and none asserted on it, so codecov reported it covered.

Converge both routes on a single record-then-throw, mirroring the Python
companion's single merged path. One call site cannot drift from itself.

Refs AAASM-5665
createPendingThenRejectGatewayClient drives the second refusal route;
createFailingRecordGatewayClient fails record() either as a rejected
promise or as a synchronous throw, and counts attempts so a control can
tell "the sink failed" from "the sink was never called".

Refs AAASM-5665
Two call sites went unasserted. The approval-rejected audit action was
executed by five tests and checked by none, and no control covered a
record() that fails.

Also rename the describe: it called the fixture's in-process array "the
audit sink", the same over-claim that blocked the go-sdk PR, in the string
CI prints. It now names the call, which is what the block reads.

Refs AAASM-5665
The approval-rejected control always supplied an approver reason, leaving
the wrapper's `?? "Rejected"` fallback a partial branch — the one line
codecov flagged. A gateway is not obliged to explain a rejection, and
attribution has to survive that: the tool name comes from the wrapper's
own message, not the gateway's text.

Run the control both ways.

Refs AAASM-5665
@sonarqubecloud

sonarqubecloud Bot commented Aug 7, 2026

Copy link
Copy Markdown

@Chisanan232
Chisanan232 merged commit 54bb8a6 into main Aug 7, 2026
25 checks passed
@Chisanan232
Chisanan232 deleted the v0.0.1-rc.7/AAASM-5665/fix/deny_audit_record branch August 7, 2026 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant