feat(observe): attribute recorded runs and costs to explicit tenants

Propagate optional tenant labels through runtime, planner, agent and server traces. Add tenant cost aggregation and CLI reports while preserving unlabelled legacy traces, separating recorded prices from estimates, and counting planner spend.

Author: Shashankss1205

README.md

@@ -70,6 +70,8 @@ print(g.compile().invoke({"question": "meaning of life"}))
 
 Typed state, declared writes, a budget — none optional. The rest of the surface (agents, tools, memory, sessions, policy documents, cost attribution, OTel) is in the [deep dive](docs/deep-dive.md) and the [cookbook](docs/cookbook/).
 
+Cost attribution also supports explicit tenants: pass `tenant="team-a"` to a compiled graph or governed loop, then read its bill with `attribute_tenant(trace, "team-a", rates=...)` or `grapharc cost trace.jsonl --tenant team-a --json`. `run`, `plan`, `go` and policy-governed `agent` runs carry an explicit `--tenant`, `GRAPHARC_TENANT` or config value into the trace. Recorded USD and rate-card estimates stay separate; old unlabelled runs remain unassigned. See [the cost guide](docs/cookbook/06-serving-and-ops.md#where-did-the-tokens-go) for the boundaries.
+
 ## The admission gate
 
 You cannot pre-author a graph for "investigate this incident" — the shape is discovered while working. So the graph is proposed at runtime, and a deterministic checker stands between proposing and running: registry, policy, remaining budget, depth, acyclicity, all on every proposal. A rejection is structured feedback the planner replans against; work discovered mid-run re-enters the same gate. Watch it refuse (free, scripted, the first proposal names a policy-denied kind):

ROADMAP.md

@@ -451,7 +451,7 @@ allowances; the agent records document-driven denials and approval decisions.
       importing the module needs no OTel. Previously documented as unverified
       against the real SDK — verified during this pass against
       `opentelemetry-sdk` 1.44.0, with spans reaching an `InMemorySpanExporter`.
-- [~] **10.4 — Cost attribution** per run, thread (session) and node. The
+- [x] **10.4 — Cost attribution** per run, thread (session), node and explicit tenant. The
       price is now recorded, not guessed: every gateway publishes a
       `cost_usd` through the same `llm_output` envelope, the runtime's usage
       callback accumulates it per node and writes it onto the `end` event, and
@@ -461,9 +461,16 @@ allowances; the agent records document-driven denials and approval decisions.
       figure wins outright rather than being averaged with a guess. Tokens are
       counted from the same events `metrics.summarize` uses — node `end` events
       *plus* work outside any node span, which is what a `grapharc agent` run
-      consists of entirely — and the suite asserts the two agree. **One gap
-      left:** there is no tenant on a trace event, so tenant attribution is not
-      offered rather than being approximated.
+      consists of entirely — and the suite asserts the two agree. An optional
+      tenant on `RunContext` labels every native graph, planner and agent event;
+      the loop carries it through round sub-runs. `attribute_tenant` and
+      `grapharc cost --tenant NAME` select whole labelled runs, refuse conflicting
+      labels, and leave old unlabelled runs unassigned. Planning tokens without
+      a recorded price are estimated or marked unpriced, including in tenant
+      totals. **Producer limit:** planner turns and failed node terminal events
+      still omit the provider price ([#138](https://github.com/CodeGraphContext/GraphARC/issues/138));
+      a rate-card estimate is not a recorded bill.
+      Tenant labels provide attribution, not authorization.
 - [ ] **10.2 — Rollback** and versioned graph/prompt configs.
 - [ ] **10.5 — Alerting** on budget, failure, and verifier-drift.
 

docs/cookbook/06-serving-and-ops.md

@@ -1570,7 +1570,9 @@ from pathlib import Path
 
 from langchain_core.messages import HumanMessage
 
-from grapharc.observe import RateCard, TraceRecorder, attribute, summarize, to_mermaid
+from grapharc.observe import (
+    RateCard, TraceRecorder, attribute, attribute_tenant, summarize, to_mermaid,
+)
 from grapharc.runtime.graph import END, START, GraphARC
 from grapharc.runtime.state import GraphARCState
 from grapharc.testing import ScriptedChatModel
@@ -1612,6 +1614,12 @@ metrics = summarize(trace, "r1")
 print("metrics :", metrics.nodes_executed, "nodes,", metrics.tokens, "tokens,", metrics.per_node)
 print()
 print(to_mermaid(trace, "r1"))
+
+# A second run has an explicit tenant; the old unlabelled run stays excluded.
+g.compile().invoke({"question": "budgets?"}, run_id="r2", tenant="team-a")
+bill = attribute_tenant(trace, "team-a", rates=RateCard(default=3.0))
+assert [run.run_id for run in bill.runs] == ["r2"]
+assert bill.tokens == attribute(trace, "r2").tokens
 ```
 
 ```
@@ -1634,24 +1642,25 @@ flowchart TD
 ```
 
 `RunCost.tokens` and `RunMetrics.tokens` agree by construction — both count the
-`end` events of node executions, and the test suite asserts they match, because
+`end` and `error` events plus work outside node spans, and the suite asserts they match, because
 a cost report and an audit trail that disagree are worse than either alone.
 
 **Three honest limits:**
 
-- **Nothing in GraphARC writes `cost_usd` onto a trace event today.** The field
-  exists and `TraceRecorder.event` accepts it, but the kernel does not pass it,
-  so `recorded_cost_usd` is always `None` on a trace from today's runtime.
-  Everything you see above is `estimated_cost_usd` — tokens × your rate card,
-  reported in its own field so nobody mistakes an estimate for an invoice.
+- **Recorded cost depends on the producer.** The kernel writes provider prices
+  onto successful node terminal events, and agents record their per-call prices.
+  Planner turns and failed node terminal events currently omit the provider
+  price. The example above uses a backend that reports no price, so its money
+  is `estimated_cost_usd` — tokens × your rate card. Recorded and estimated
+  figures remain separate; a recorded figure takes precedence over an estimate.
 - **`complete` is what tells you the total is a total.** Tokens with neither a
   recorded cost nor a matching rate are counted in `unpriced_tokens`, and a
   non-zero count means `cost_usd` is a lower bound. A `RateCard` with no
   `default` and no matching model prices nothing.
-- **A node that raised has no token count at node level.** No `end` event was
-  written. Where it was an `AgentNode`, its per-call sub-events still hold the
-  spend, reported as `tokens_before_error` — kept out of the total so the total
-  keeps matching `metrics`.
+- **Older error events may have no token count.** Current node wrappers record
+  tokens on both `end` and `error`. When an older or hand-built trace omits the
+  failed node's count, an agent's per-call sub-events may still hold the spend,
+  reported as `tokens_before_error` and kept out of the total that matches `metrics`.
 
 `to_mermaid` renders the graph's *declared topology* — the `topology` event every
 run now writes — with execution status overlaid per node: `done`, `running`,
@@ -1666,11 +1675,54 @@ renderer that speaks Mermaid.
 `attribute_thread(trace, thread_id)` is the same for a whole session across
 resumes, and `by_node(trace)` ranks every node in a file by cost.
 
+For tenant attribution, supply one label at the run entry point, as the example
+above does with `tenant="team-a"`. `attribute_tenant` selects its second run,
+`r2`, while the first unlabelled run stays excluded.
+
+Every compiled execution entry (`invoke`, `stream`, `ainvoke`, `astream`,
+`astream_events`) accepts `tenant=`, as does `GovernedLoop.run`. The loop carries
+it through planning, admission, approval, round execution and stop events;
+`AgentNode` inherits its supplied `RunContext.tenant`. Custom producers must
+label their own events consistently. Attribution is a caller-supplied label,
+not an authorization check; it does not apply a policy on its own.
+
+`attribute_tenant(trace, name, rates=...)` reads one snapshot and returns
+`TenantCost`: the matching runs, tokens, duration, recorded USD, estimated USD,
+combined known USD, unpriced tokens and `complete`. It prices the same whole
+runs as `attribute`, so model substeps do not add a second copy of node spend.
+An unknown tenant raises `ReplayError`. Any run with inconsistent labels,
+including a mixture of labelled and unlabelled events, makes the reader refuse
+the report rather than guess a bill. Legacy runs with no tenant remain
+unassigned, and writing an unlabelled run adds no tenant key.
+
+The CLI uses the same reader:
+
+```bash
+grapharc cost trace.jsonl --tenant team-a --json
+grapharc cost trace.jsonl --run-id r2
+```
+
+Choose exactly one scope. JSON includes separate `recorded_cost_usd` and
+`estimated_cost_usd`, `unpriced_tokens`, `complete`, and per-run detail for a
+tenant report. The CLI supplies no rate card, so tokens without recorded prices
+remain unpriced; use the Python API for estimates. Missing or unreadable files
+exit 2, and missing selections or inconsistent labels exit 1, including in JSON.
+
+`run`, `plan`, `go` and policy-governed `agent` runs carry a tenant supplied by
+`--tenant`, `GRAPHARC_TENANT`
+or `grapharc.toml` into their traces. The policy layer's implicit `"default"`
+does not label a trace; an explicit `--tenant default` does. Executing a saved
+plan uses the current invocation's explicit tenant, rather than inheriting a
+label from the planning run. The standalone `grapharc agent` command
+currently requires `--policy` when selecting a tenant; its trace uses the explicit
+selection, while the document's implicit default tenant stays unlabelled.
+
+
 ---
 
 ## The CLI tour
 
-Twelve commands. Every one takes `--json`, which prints the same payload as one
+The commands below all take `--json`, which prints the same payload as one
 document on stdout — including failures, which become the document rather than a
 line on stderr.
 
@@ -1687,6 +1739,7 @@ line on stderr.
 | `grapharc diff <trace> <a> <b>` | compare two runs in one trace |
 | `grapharc trace <trace>` | pretty-print a trace file |
 | `grapharc metrics <trace> <run>` | summarize one run |
+| `grapharc cost <trace> --tenant NAME` | attribute whole labelled runs; alternatively select `--run-id ID` |
 | `grapharc viz <trace> <run>` | render the executed path as Mermaid |
 
 `run --check-only` prints a `topology` digest for comparing the same graph