Merge pull request #139 from CodeGraphContext/feat/tenant-cost

feat(observe): attribute runs and costs to explicit tenants

Author: Shashankss1205

README.md

@@ -70,6 +70,8 @@ print(g.compile().invoke({"question": "meaning of life"}))
 
 Typed state, declared writes, a budget — none optional. The rest of the surface (agents, tools, memory, sessions, policy documents, cost attribution, OTel) is in the [deep dive](docs/deep-dive.md) and the [cookbook](docs/cookbook/).
 
+Cost attribution also supports explicit tenants: pass `tenant="team-a"` to a compiled graph or governed loop, then read its bill with `attribute_tenant(trace, "team-a", rates=...)` or `grapharc cost trace.jsonl --tenant team-a --json`. `run`, `plan`, `go` and policy-governed `agent` runs carry an explicit `--tenant`, `GRAPHARC_TENANT` or config value into the trace. Recorded USD and rate-card estimates stay separate; old unlabelled runs remain unassigned. See [the cost guide](docs/cookbook/06-serving-and-ops.md#where-did-the-tokens-go) for the boundaries.
+
 ## The admission gate
 
 You cannot pre-author a graph for "investigate this incident" — the shape is discovered while working. So the graph is proposed at runtime, and a deterministic checker stands between proposing and running: registry, policy, remaining budget, depth, acyclicity, all on every proposal. A rejection is structured feedback the planner replans against; work discovered mid-run re-enters the same gate. Watch it refuse (free, scripted, the first proposal names a policy-denied kind):

ROADMAP.md

@@ -451,7 +451,7 @@ allowances; the agent records document-driven denials and approval decisions.
       importing the module needs no OTel. Previously documented as unverified
       against the real SDK — verified during this pass against
       `opentelemetry-sdk` 1.44.0, with spans reaching an `InMemorySpanExporter`.
-- [~] **10.4 — Cost attribution** per run, thread (session) and node. The
+- [x] **10.4 — Cost attribution** per run, thread (session), node and explicit tenant. The
       price is now recorded, not guessed: every gateway publishes a
       `cost_usd` through the same `llm_output` envelope, the runtime's usage
       callback accumulates it per node and writes it onto the `end` event, and
@@ -461,9 +461,16 @@ allowances; the agent records document-driven denials and approval decisions.
       figure wins outright rather than being averaged with a guess. Tokens are
       counted from the same events `metrics.summarize` uses — node `end` events
       *plus* work outside any node span, which is what a `grapharc agent` run
-      consists of entirely — and the suite asserts the two agree. **One gap
-      left:** there is no tenant on a trace event, so tenant attribution is not
-      offered rather than being approximated.
+      consists of entirely — and the suite asserts the two agree. An optional
+      tenant on `RunContext` labels every native graph, planner and agent event;
+      the loop carries it through round sub-runs. `attribute_tenant` and
+      `grapharc cost --tenant NAME` select whole labelled runs, refuse conflicting
+      labels, and leave old unlabelled runs unassigned. Planning tokens without
+      a recorded price are estimated or marked unpriced, including in tenant
+      totals. **Producer limit:** planner turns and failed node terminal events
+      still omit the provider price ([#138](https://github.com/CodeGraphContext/GraphARC/issues/138));
+      a rate-card estimate is not a recorded bill.
+      Tenant labels provide attribution, not authorization.
 - [ ] **10.2 — Rollback** and versioned graph/prompt configs.
 - [ ] **10.5 — Alerting** on budget, failure, and verifier-drift.
 

docs/cookbook/06-serving-and-ops.md

@@ -1570,7 +1570,9 @@ from pathlib import Path
 
 from langchain_core.messages import HumanMessage
 
-from grapharc.observe import RateCard, TraceRecorder, attribute, summarize, to_mermaid
+from grapharc.observe import (
+    RateCard, TraceRecorder, attribute, attribute_tenant, summarize, to_mermaid,
+)
 from grapharc.runtime.graph import END, START, GraphARC
 from grapharc.runtime.state import GraphARCState
 from grapharc.testing import ScriptedChatModel
@@ -1612,6 +1614,12 @@ metrics = summarize(trace, "r1")
 print("metrics :", metrics.nodes_executed, "nodes,", metrics.tokens, "tokens,", metrics.per_node)
 print()
 print(to_mermaid(trace, "r1"))
+
+# A second run has an explicit tenant; the old unlabelled run stays excluded.
+g.compile().invoke({"question": "budgets?"}, run_id="r2", tenant="team-a")
+bill = attribute_tenant(trace, "team-a", rates=RateCard(default=3.0))
+assert [run.run_id for run in bill.runs] == ["r2"]
+assert bill.tokens == attribute(trace, "r2").tokens
 ```
 
 ```
@@ -1634,24 +1642,25 @@ flowchart TD
 ```
 
 `RunCost.tokens` and `RunMetrics.tokens` agree by construction — both count the
-`end` events of node executions, and the test suite asserts they match, because
+`end` and `error` events plus work outside node spans, and the suite asserts they match, because
 a cost report and an audit trail that disagree are worse than either alone.
 
 **Three honest limits:**
 
-- **Nothing in GraphARC writes `cost_usd` onto a trace event today.** The field
-  exists and `TraceRecorder.event` accepts it, but the kernel does not pass it,
-  so `recorded_cost_usd` is always `None` on a trace from today's runtime.
-  Everything you see above is `estimated_cost_usd` — tokens × your rate card,
-  reported in its own field so nobody mistakes an estimate for an invoice.
+- **Recorded cost depends on the producer.** The kernel writes provider prices
+  onto successful node terminal events, and agents record their per-call prices.
+  Planner turns and failed node terminal events currently omit the provider
+  price. The example above uses a backend that reports no price, so its money
+  is `estimated_cost_usd` — tokens × your rate card. Recorded and estimated
+  figures remain separate; a recorded figure takes precedence over an estimate.
 - **`complete` is what tells you the total is a total.** Tokens with neither a
   recorded cost nor a matching rate are counted in `unpriced_tokens`, and a
   non-zero count means `cost_usd` is a lower bound. A `RateCard` with no
   `default` and no matching model prices nothing.
-- **A node that raised has no token count at node level.** No `end` event was
-  written. Where it was an `AgentNode`, its per-call sub-events still hold the
-  spend, reported as `tokens_before_error` — kept out of the total so the total
-  keeps matching `metrics`.
+- **Older error events may have no token count.** Current node wrappers record
+  tokens on both `end` and `error`. When an older or hand-built trace omits the
+  failed node's count, an agent's per-call sub-events may still hold the spend,
+  reported as `tokens_before_error` and kept out of the total that matches `metrics`.
 
 `to_mermaid` renders the graph's *declared topology* — the `topology` event every
 run now writes — with execution status overlaid per node: `done`, `running`,
@@ -1666,11 +1675,54 @@ renderer that speaks Mermaid.
 `attribute_thread(trace, thread_id)` is the same for a whole session across
 resumes, and `by_node(trace)` ranks every node in a file by cost.
 
+For tenant attribution, supply one label at the run entry point, as the example
+above does with `tenant="team-a"`. `attribute_tenant` selects its second run,
+`r2`, while the first unlabelled run stays excluded.
+
+Every compiled execution entry (`invoke`, `stream`, `ainvoke`, `astream`,
+`astream_events`) accepts `tenant=`, as does `GovernedLoop.run`. The loop carries
+it through planning, admission, approval, round execution and stop events;
+`AgentNode` inherits its supplied `RunContext.tenant`. Custom producers must
+label their own events consistently. Attribution is a caller-supplied label,
+not an authorization check; it does not apply a policy on its own.
+
+`attribute_tenant(trace, name, rates=...)` reads one snapshot and returns
+`TenantCost`: the matching runs, tokens, duration, recorded USD, estimated USD,
+combined known USD, unpriced tokens and `complete`. It prices the same whole
+runs as `attribute`, so model substeps do not add a second copy of node spend.
+An unknown tenant raises `ReplayError`. Any run with inconsistent labels,
+including a mixture of labelled and unlabelled events, makes the reader refuse
+the report rather than guess a bill. Legacy runs with no tenant remain
+unassigned, and writing an unlabelled run adds no tenant key.
+
+The CLI uses the same reader:
+
+```bash
+grapharc cost trace.jsonl --tenant team-a --json
+grapharc cost trace.jsonl --run-id r2
+```
+
+Choose exactly one scope. JSON includes separate `recorded_cost_usd` and
+`estimated_cost_usd`, `unpriced_tokens`, `complete`, and per-run detail for a
+tenant report. The CLI supplies no rate card, so tokens without recorded prices
+remain unpriced; use the Python API for estimates. Missing or unreadable files
+exit 2, and missing selections or inconsistent labels exit 1, including in JSON.
+
+`run`, `plan`, `go` and policy-governed `agent` runs carry a tenant supplied by
+`--tenant`, `GRAPHARC_TENANT`
+or `grapharc.toml` into their traces. The policy layer's implicit `"default"`
+does not label a trace; an explicit `--tenant default` does. Executing a saved
+plan uses the current invocation's explicit tenant, rather than inheriting a
+label from the planning run. The standalone `grapharc agent` command
+currently requires `--policy` when selecting a tenant; its trace uses the explicit
+selection, while the document's implicit default tenant stays unlabelled.
+
+
 ---
 
 ## The CLI tour
 
-Twelve commands. Every one takes `--json`, which prints the same payload as one
+The commands below all take `--json`, which prints the same payload as one
 document on stdout — including failures, which become the document rather than a
 line on stderr.
 
@@ -1687,6 +1739,7 @@ line on stderr.
 | `grapharc diff <trace> <a> <b>` | compare two runs in one trace |
 | `grapharc trace <trace>` | pretty-print a trace file |
 | `grapharc metrics <trace> <run>` | summarize one run |
+| `grapharc cost <trace> --tenant NAME` | attribute whole labelled runs; alternatively select `--run-id ID` |
 | `grapharc viz <trace> <run>` | render the executed path as Mermaid |
 
 `run --check-only` prints a `topology` digest for comparing the same graph