Skip to content

Record every decision failure and count them at session end - #273

Open
bar-capsule wants to merge 2 commits into
GenAI-Security-Project:integrationfrom
bar-capsule:spec/37-decision-failure-recording
Open

bar-capsule wants to merge 2 commits into
GenAI-Security-Project:integrationfrom
bar-capsule:spec/37-decision-failure-recording

Conversation

@bar-capsule

@bar-capsule bar-capsule commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

What changed

The rest of the #37 decision, the parts that live outside the Trace event #275 added. §6.4 now requires an audit record for every decision failure under either posture: in the deployment's own audit log, and as an acs.decision.failure event on Trace when the deployment claims ACS-Trace. §4.1 surfaces a session started unguarded as the same event with class handshake_failure. sessionEnd.summary gains decision_failures, a count per failure class that the Observed Agent SHOULD report, and a session that ends without it reads as unmeasurable.

It also closes two gaps from the #275 review. A request the Guardian rejects before trusting it can carry a genuine request's pair, so it produces no failure or decision event under that pair; a deployment that records rejected requests uses other attribute names, as decided on #283. And handshake_failure gets a posture: it comes from the startup posture, since on_decision_failure is never negotiated.

Which issue does this implement

Closes #37
Refs #283, which decided how the reference Guardian records requests it can't vouch for

  • This is an editorial correction (typo, grammar, link, formatting) with no change in
    meaning, so it needs no issue

Base branch

  • integration, because this touches the specification, schemas, a reference
    implementation, an adapter, tests, or CI
  • main, because every changed path is on the documentation lane allowlist in
    CONTRIBUTING.md

Type of change

  • Specification change (schema, hooks, events, AgBOM)
  • Reference implementation or adapter
  • Documentation
  • Tooling or CI
  • Governance (licensing, security policy, contributor docs)

Specification changes

  • I opened a Discussion before this PR
  • Schema changes validate against the JSON Schema spec
  • I described the impact on downstream implementers below

No Discussion was opened. The decisions are on #37 and #283.

File Change
docs/spec/instrument/specification.md §6.4: every decision failure is recorded, fail-open or fail-closed, with the audit log as the MUST and Trace as the SHOULD; the sessionEnd count is a SHOULD. §4.1: an unguarded session becomes acs.decision.failure with class handshake_failure and posture fail_open on the session span.
specification/v0.1.0/hooks/session-end.json summary.decision_failures: an object keyed by failure class, each value an integer of 0 or more. The keys are limited to the eight acs.failure.class values.
specification/v0.1.0/trace/otel-mapping.json No failure or decision event under the pair of a request rejected before it is trusted (failed envelope validation, SIGNATURE_INVALID, REPLAY_DETECTED, TIMESTAMP_OUT_OF_WINDOW). handshake_failure covers both startup postures, and its acs.failure.posture comes from the startup posture.
docs/spec/trace/events.md The same rejected-request rule.
docs/spec/conformance.md The decision-honoring line follows §6.4.
docs/spec/instrument/hooks.md The sessionEnd payload line names the real fields (reason, not session_reason, plus the missing abandoned value) and the new count.
tests/test_decision_failure_classes.py Fails when the classes session-end.json counts differ from the acs.failure.class values in otel-mapping.json.

Breaking for implementers? The schema change is additive: decision_failures is optional. Two rules tighten. §6.4 previously required a record only for a step that proceeded without a decision, and now a fail-closed decision failure needs one too. And a Guardian that used to record a rejected replay, or any other untrusted request, under its claimed pair now records it under other names or not at all.

I tested this

  • I synced my branch with the base branch before opening this
  • uv run pytest -v passes on my machine
  • uv run mkdocs build --strict passes on my machine

pytest reports 919 passed and 1 skipped (a case-insensitive filesystem skip). From reference-implementations/agt/, bun run typecheck is clean and bun test reports 1234 passed, 1 skipped, 0 failed. session-end.json was checked with jsonschema (Draft 2020-12): a count for known classes, a zero, and a payload with no summary are valid; an unknown class, a negative count and a non-integer are rejected.

Merging

  • Merge with an explicit squash message that keeps the Signed-off-by line.

Checklist

  • Commits are signed off with git commit -s (required by the DCO)
  • Prose follows STYLE.md
  • No secrets, tokens, or internal URLs in the diff

Security

  • This change has no security impact

It adds detection rather than surface: a blocked handshake now produces an event and a count, and a forged or replayed request can no longer make a genuine decision read as unenforced.

@github-actions

Copy link
Copy Markdown

This pull request's description says it closes #37, which is on the project roadmap. When it merges, GitHub closes that issue and the roadmap counts it as delivered. Please confirm the change delivers the whole issue before merging.

@acs-board-reconciler acs-board-reconciler Bot moved this from Needs triage to In progress in ACS Project Tracker Oct 14, 2026
rocklambros added a commit that referenced this pull request Oct 14, 2026
…n issue #37

signature_invalid keeps a tampered response apart from organic failures.
handshake_failure counts a session that started without a completed
handshake, and its event goes on the acs.session span, not a step span.
PR #273's sessionEnd count keys on this set of eight classes.

Signed-off-by: rocklambros <rock@rockcyber.com>
@rocklambros

Copy link
Copy Markdown
Collaborator

@bar-capsule, both classes from your issue #37 answer are in PR #275 now (9cfa988 and 199af89). signature_invalid sits apart from malformed_response, and handshake_failure goes on the acs.session span, counted under the handshake's own request_id and session_id. I merged this PR on top of PR #275 in a scratch tree, and tests/test_decision_failure_classes.py passes, so this one goes green as soon as PR #275 lands. Sorry it took me a day to pick up your answer.

Section 6.4 now requires an audit record for every decision failure under either posture, kept in the deployment's own audit log and emitted on Trace as acs.decision.failure when the deployment claims ACS-Trace. Section 4.1 surfaces a session started unguarded as the same event with class handshake_failure. sessionEnd.summary gains decision_failures, a per-class count the Observed Agent SHOULD report, and a session that ends without it reads as unmeasurable. conformance.md and the hooks.md sessionEnd summary follow, and a test keeps the counted classes equal to the acs.failure.class values in the OTel mapping.

Signed-off-by: Bar Kaduri <bar@capsule.security>
…ke_failure its posture

A Guardian emits no acs.decision.failure or acs.decision event under the pair of a request it rejects before trusting it: an envelope that fails validation, a failed signature, a replay, or a timestamp outside the window. Such a request can carry a genuine request's pair. handshake_failure now covers the refuse startup posture too, and its acs.failure.posture comes from the startup posture, since on_decision_failure is never negotiated. Section 4.1 records an unguarded session with posture fail_open.

Signed-off-by: Bar Kaduri <bar@capsule.security>
@bar-capsule

Copy link
Copy Markdown
Collaborator Author

@rocklambros @afogel ready for review - the rest of #37 on top of #275, plus the two gaps from the #275 review: requests the Guardian rejects stay off the genuine request's IDs, and handshake_failure takes its posture from the startup posture. Small spec PR, all checks green.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

[Bug] §6.4's mandatory fail-open audit event has no conformant shape, so the decision-failure rate is unmeasurable

2 participants