Explain how the reference Guardian produces a trace, with diagrams

README.md

@@ -289,12 +289,16 @@ harness measures the wire, not the language behind it.
 
 ### Harden it for production
 
-Two gaps are named and open, not vague. The Guardian writes its envelope and audit logs
-as synchronous file appends on the decision path, with no batching. Trace export is
+Two gaps are named and open, not vague. The Guardian writes its envelope and
+session-context logs, and the host adapter writes its audit log, as synchronous file
+appends on the decision path, with no batching. Trace export is
 partial: set `ACS_OTEL_FILE` or `ACS_OTEL_ENDPOINT` and the Guardian exports OTLP spans
 for the two live methods, `steps/toolCallRequest` and `steps/toolCallResult`, as the
 [Guardian README](reference-implementations/agt/packages/guardian/README.md#export-traces)
-describes. A [sample trace](reference-implementations/agt/samples/trace/acs-trace-sample.jsonl)
+describes. Failures only the host sees, such as an unreachable Guardian, stay in the host's
+audit log until a converter turns them into spans of the same format.
+[How the reference Guardian produces a trace](docs/spec/trace/extend_opentelemetry.md#how-the-reference-guardian-produces-a-trace)
+explains how the two streams join. A [sample trace](reference-implementations/agt/samples/trace/acs-trace-sample.jsonl)
 shows that output without running anything, and [Produce a trace](reference-implementations/agt/README.md#produce-a-trace)
 walks you from a fresh clone to a trace file of your own.
 

docs/spec/trace/README.md

@@ -7,6 +7,22 @@ AI agents make autonomous decisions that impact business outcomes. Without obser
 !!! info "ACS Extends Industry Standards"
     We already have great observability standards, so ACS doesn't introduce a new one. Instead, it extends existing industry-proven standards: OpenTelemetry and OCSF to support AI agent-specific components.
 
+## Where Trace Sits
+
+Instrument decides in the moment. Trace records what happened, out of the request path, so someone can rebuild a session afterwards: what the agent tried, what the Guardian decided, and which steps got no usable decision at all.
+
+Both parties write to the trace, because each one sees something the other can't. In the diagram, notice that the Observed Agent and the Guardian each emit their own stream, and the two streams meet only at the reader.
+
+```mermaid
+flowchart LR
+    agent["Observed Agent"] -->|"hook request"| guardian["Guardian"]
+    agent -->|"spans and events"| formats["OpenTelemetry or OCSF"]
+    guardian -->|"spans and events"| formats
+    formats --> reader["Reader joins on<br/>session id + request id"]
+```
+
+The reader matches records from the two parties by each step's session id and request id. [Trace Events](./events.md) defines that join and the rule for counting a decision as enforced. To see it as real files, read [How the reference Guardian produces a trace](./extend_opentelemetry.md#how-the-reference-guardian-produces-a-trace), or open the committed [sample trace](https://github.com/GenAI-Security-Project/agent-control-standard/blob/main/reference-implementations/agt/samples/trace/acs-trace-sample.jsonl).
+
 ## Why Agent Observability Matters
 
 Modern agents orchestrate complex workflows: reasoning chains, tool execution, knowledge retrieval, multi-agent collaboration. When things go wrong, or right, you need to understand exactly what happened.

docs/spec/trace/extend_opentelemetry.md

@@ -162,6 +162,90 @@ The span carries the join pair (`acs.session.id` and `acs.request_id`), the tool
 
 To produce a trace like this yourself, follow [Produce a trace](https://github.com/GenAI-Security-Project/agent-control-standard/blob/main/reference-implementations/agt/README.md#produce-a-trace) in the reference implementation's README.
 
+## How the reference Guardian produces a trace
+
+The span above is real output from the reference Guardian. This section explains the pipeline behind it, so the span hierarchy and the decision events on this page map onto processes and files you can inspect. The [reference Guardian](https://github.com/GenAI-Security-Project/agent-control-standard/blob/main/reference-implementations/agt/packages/guardian/README.md#export-traces) exports spans for the two hook methods it evaluates, `steps/toolCallRequest` and `steps/toolCallResult`. It makes no ACS-Trace conformance claim. The commands to run it live in [Produce a trace](https://github.com/GenAI-Security-Project/agent-control-standard/blob/main/reference-implementations/agt/README.md#produce-a-trace), and this section doesn't repeat them.
+
+### Two streams, joined by a reader
+
+No single process sees every outcome of a step. The Guardian sees each request it receives and the answer it gives. It exports one span per decided step, with the `acs.decision` event on it. It also reports the failures it can see itself: an error it returned instead of a decision, a decision it finished after the negotiated timeout, and a step still open when it shut down.
+
+The host sees what the Guardian can't. An unreachable Guardian has nothing to record. A Guardian whose answer arrives after the host stopped waiting can't see what the host did in the meantime. The host adapter records both cases in its own audit log. A converter, `audit-to-otlp.ts`, turns that log into spans of the same OTLP/JSON shape, each with an `acs.decision.failure` event whose `acs.failure.reported_by` is `observed_agent`.
+
+A reader joins the two streams on `acs.session.id` plus `acs.request_id`, the pair every step span carries. The reference exporter makes each step span a root span, with no `acs.session` or `acs.turn` parent, so the join runs on those attributes, not on `traceId`.
+
+In the diagram, look for the two separate paths out of the host shim. They meet only at the reader.
+
+```mermaid
+flowchart LR
+    shim["Host shim"] -->|"JSON-RPC request"| guardian["Guardian"]
+    guardian --> exporter["Trace exporter"]
+    exporter -->|"ACS_OTEL_FILE"| gfile[".acs/otel-traces.jsonl"]
+    exporter -->|"ACS_OTEL_ENDPOINT"| collector["OpenTelemetry Collector"]
+    collector --> cfile[".acs/otel-collector-traces.jsonl"]
+    shim -->|"audit entries"| audit[".acs/audit.jsonl"]
+    audit --> converter["audit-to-otlp.ts"]
+    converter --> afile[".acs/otel-audit.jsonl"]
+    gfile --> reader["Reader joins on<br/>acs.session.id + acs.request_id"]
+    cfile --> reader
+    afile --> reader
+```
+
+### Follow one tool call
+
+The span's lifetime follows the Guardian's envelope log. It opens when the Guardian records the request and ends when the Guardian records its response, before the response goes out. Export happens later, on a timer, so serializing and writing spans stay off the decision path.
+
+The sequence below follows one `steps/toolCallRequest`. Watch the `alt` block: the top branch is the working path, and the two lower branches show who reports the failure when no usable decision arrives.
+
+```mermaid
+sequenceDiagram
+    participant Shim as Host shim
+    participant G as Guardian
+    participant X as Trace exporter
+    participant B as Batch processor
+    participant Out as File or collector
+    Shim->>G: steps/toolCallRequest
+    G->>X: request recorded, span starts
+    alt decision returned in time
+        G->>X: response recorded
+        X->>X: add acs.decision event, end span
+        G-->>Shim: decision
+    else Guardian returns an error or decides late
+        G->>X: response recorded
+        X->>X: add acs.decision.failure, reported_by guardian, end span
+        G-->>Shim: error or late decision
+    else no decision reaches the host
+        Shim->>Shim: apply posture, write audit.jsonl entry
+        Note over Shim: the converter later emits acs.decision.failure, reported_by observed_agent
+    end
+    X->>B: queue ended span
+    B->>Out: export batch about every 5 s, and at shutdown
+```
+
+A late decision gets both events on one span: the `acs.decision` with the verdict, then an `acs.decision.failure` with class `timeout`. When either party's stream holds a failure event for a pair, a reader doesn't count that pair's `acs.decision` as enforced. [Trace Events](./events.md) states that rule.
+
+The Guardian never copies its error text into the trace. A failure event carries the error code, the timeout, the elapsed time and the request size when they're known, because error text can quote the request. A replay rejection gets no failure event, since its pair belongs to the earlier request.
+
+If you only need to read a trace, you can stop here. The rest of this section is for running the pipeline or building your own.
+
+### What each output is for
+
+`.acs/otel-traces.jsonl` is the Guardian's own file, written when `ACS_OTEL_FILE` is set. It holds one OTLP `ExportTraceServiceRequest` per line, the shape the OpenTelemetry Collector's file exporter writes. It's the file to read offline or hand to a reviewer.
+
+The collector path, set with `ACS_OTEL_ENDPOINT`, sends the same spans over OTLP/HTTP. The committed [`otel-collector.yaml`](https://github.com/GenAI-Security-Project/agent-control-standard/blob/main/reference-implementations/agt/otel-collector.yaml) writes them to `.acs/otel-collector-traces.jsonl`. Use it as an independent check that the output is valid OTLP, and as the starting point for sending spans to a real backend.
+
+`.acs/otel-audit.jsonl` comes from the converter, not the Guardian. It holds only the host-side failures, and it matters most when the Guardian was down. The converter runs after the session ends and refuses to write to a file that already exists. Its spans carry `service.name` `acs-reference-audit-converter`, where the Guardian's carry `acs-reference-guardian`.
+
+The two Guardian outputs fail independently. The first failed write or send disables that output for the rest of the process and prints one line on stderr. The other output keeps running, and the decision goes out unchanged, as the [failure isolation](#failure-isolation) rule below requires. The [Guardian README](https://github.com/GenAI-Security-Project/agent-control-standard/blob/main/reference-implementations/agt/packages/guardian/README.md#troubleshooting) lists the messages you may see.
+
+The committed [sample trace](https://github.com/GenAI-Security-Project/agent-control-standard/blob/main/reference-implementations/agt/samples/trace/acs-trace-sample.jsonl) holds both streams in one file: five spans from the Guardian and one from the converter. Every value in it is synthetic.
+
+### What the labels don't prove
+
+`service.name` tells you which writer claims a span. It isn't authentication. Any process running as the same user, the observed agent included, can write the trace files, the audit log, or the collector's loopback port. In a container, other containers on the same network can reach that port too. A loopback address isn't authentication either.
+
+`acs.failure.reported_by` is the emitter's own claim too. The trace is a record of what each party says happened, not tamper-evident evidence. Treat every value in it as untrusted data, as [Trace Events](./events.md) requires.
+
 ## Provenance attributes
 
 When a hook payload carries Provenance, the resulting span MUST carry `acs.provenance.origin` and SHOULD carry `acs.provenance.source_id` when populated. The set of provenance ids cited by a Guardian decision (the response envelope's `cited_provenance_ids`) MAY be expressed as OTel span links keyed by `provenance_id`, letting tools that index span links reconstruct the lineage graph without parsing payloads.

reference-implementations/agt/README.md

@@ -199,6 +199,21 @@ The shim never runs the command. It only asks the Guardian.
 
 This walks you from a fresh clone to an OpenTelemetry trace file of your own, in about ten minutes. You don't need Claude Code or a collector for it. Every command runs from `reference-implementations/agt` unless the step says otherwise.
 
+The steps below write four files under `.acs/`. The diagram shows which process writes each one and in which step, so you know what to stop before you delete anything.
+
+```mermaid
+flowchart LR
+    shim["Claude Code shim<br/>(steps 3 and 6)"] -->|"hook payloads"| guardian["Guardian<br/>(step 2)"]
+    guardian -->|"ACS_OTEL_FILE"| traces[".acs/otel-traces.jsonl"]
+    shim -->|"steps with no decision"| audit[".acs/audit.jsonl"]
+    audit --> converter["audit-to-otlp.ts<br/>(step 6)"]
+    converter --> auditotlp[".acs/otel-audit.jsonl"]
+    guardian -->|"ACS_OTEL_ENDPOINT"| collector["otelcol-contrib<br/>(step 7, optional)"]
+    collector --> coltraces[".acs/otel-collector-traces.jsonl"]
+```
+
+The Guardian's file and the converter's file share a format and a join key, so you can read them as one trace. The collector's file holds Guardian spans too, received over OTLP/HTTP instead of written directly.
+
 If you only want to see what the output looks like, open [`samples/trace/acs-trace-sample.jsonl`](samples/trace/acs-trace-sample.jsonl). It's a committed run of the same exporter.
 
 **1. Get the code.** You need `git` and [Bun](https://bun.sh) 1.3 (CI runs 1.3.11). If someone gave you a commit to use, check it out, so your results match theirs.

reference-implementations/agt/packages/guardian/README.md

@@ -72,6 +72,25 @@ The file holds one OTLP `ExportTraceServiceRequest` per line, the shape the Coll
 
 Export never touches a decision. Each output fails on its own: the first failed write or send disables that output for the rest of the process and prints one line on stderr, the other output keeps running, and the response goes out unchanged.
 
+The diagram shows the exporter's inside. Look for the split after a span ends: each output gets its own batch processor and its own failure switch, so nothing on one side can stop the other.
+
+```mermaid
+flowchart LR
+    start["Request recorded:<br/>span starts"] --> finish["Response recorded:<br/>event added, span ends"]
+    finish --> bspfile["BatchSpanProcessor<br/>for the file"]
+    finish --> bspend["BatchSpanProcessor<br/>for the endpoint"]
+    bspfile --> outfile["File output"]
+    bspend --> outend["Endpoint output"]
+    outfile --> file["ACS_OTEL_FILE"]
+    outend --> collector["ACS_OTEL_ENDPOINT"]
+    outfile -.->|"first failure"| off["Output disabled,<br/>one stderr line"]
+    outend -.->|"first failure"| off
+    stop["SIGINT or SIGTERM"] -->|"close open spans, flush, 3 s limit"| bspfile
+    stop -->|"close open spans, flush, 3 s limit"| bspend
+```
+
+Each processor sends a batch about every five seconds. At shutdown, the exporter first ends any span still open with an `acs.decision.failure` event, then flushes both processors. An output that hasn't finished within three seconds is reported on stderr and its unexported spans are dropped.
+
 Use synthetic sessions for lab runs. The envelope log holds raw envelopes before any redaction, a span's `acs.reasoning` repeats the Guardian's explanation of them, and `acs.provenance.source_id` goes out as the host sent it: by `provenance.json` it can be a URL or a file path, so a signed URL or a home-directory path would too ([issue #71](https://github.com/GenAI-Security-Project/agent-control-standard/issues/71)).
 
 ### Convert the host audit log
@@ -105,6 +124,17 @@ bun run sample:trace
 
 That command regenerates it. `test/sample-trace.test.ts` checks the sample against `otel-mapping.json` and fails when the exporter's output no longer matches it, so a change to the exporter or the mapping comes with a regenerated sample. The converted span has no `gen_ai.tool.name`, because the audit log does not record the tool (issue #274). Like the exporter, the sample covers two hook methods and is not an ACS-Trace conformance claim.
 
+### Troubleshooting
+
+These are the messages export can print, with what each one means. None of them changes a decision.
+
+- `otel trace export to <path or URL> disabled after failure: <reason>` on the Guardian's stderr. That output failed its first write or send and is off until the Guardian restarts. The other output keeps running. For an endpoint, the usual cause is a collector that wasn't running yet or was restarted: stop the Guardian, start the collector, then start the Guardian again.
+- `otel trace exporter disabled after failure: <reason>`, with no output named. The exporter as a whole stopped, for example because `otel-mapping.json` didn't load. No spans reach either output for the rest of the process.
+- `otel trace exporter shutdown disabled after failure: did not finish within 3000 ms; unexported spans were dropped`. An output stalled at shutdown. The spans still in its queue are lost.
+- `otel trace export to <URL>: an HTTP proxy is set and NO_PROXY/no_proxy did not exempt <host>, so on some Bun versions these spans go through the proxy.` at startup. Your shell has `HTTP_PROXY` set, and the loopback host isn't in the proxy exemption list Bun reads. The Guardian adds the host itself, which takes effect on Bun 1.3.14 and later. On older versions, start the Guardian with the host in both `NO_PROXY` and `no_proxy`, as step 1 of [Produce a trace](../../README.md#produce-a-trace) does.
+- `refusing to write <path>: it already exists, perhaps from an earlier or failed run.` from the converter, which then exits with status 1. It never appends. Delete the old output or name a new file.
+- An empty `.acs/otel-collector-traces.jsonl` in the first five seconds after you send a step. That's normal: spans leave in batches about every five seconds, and stopping the Guardian with Ctrl-C sends the last one. If the file still hasn't appeared after about ten seconds, the collector isn't running [`otel-collector.yaml`](../../otel-collector.yaml), or the Guardian printed the `disabled after failure` line above.
+
 ## Use it in-process
 
 The tests and the conformance harness start a Guardian in the same process.