fix(trace): separate Pi cache token accounting by waveywaves · Pull Request #3719 · chainloop-dev/chainloop

waveywaves · 2026-10-24T21:33:32Z

Summary

keep Pi evidence at schema_version: v1 while recording native input, output, cache-read, and cache-write separately
recompute total_tokens from base input plus output instead of trusting Pi's cache-inclusive totalTokens
compare stored evidence independently with raw Pi records in the compatibility suite
record the cache-active Pi 1.3.0 result as supported

This implements the Pi-only #3720, a slice of the broader cross-provider #3706. The focused tracker was created after this PR opened to keep subsequent work issue-first. This PR does not close the broader all-provider tracker.

Trace provider checklist

Evidence keeps input, output, total, cache read, and cache write semantics explicit.
Provider-reported cost remains authoritative.
Invalid counters and overflow paths produce deterministic warnings.
Parsing reads the copied selected-branch session and preserves resumed history.
Unit tests cover real test data, malformed totals, overflow, resumed usage, and source comparison.
The live Pi compatibility check ran with the pinned version and model.
The compatibility matrix and token-accounting documentation are updated from the live result.
No new dependency, generated artifact, hook, or CLI wiring change is required.
Changed source files retain the current license header.

Validation

go test -race -shuffle=on -count=1 ./internal/trace/pi ./internal/trace/e2e
go build ./...
go vet ./...
golangci-lint run --new-from-rev upstream/main
CHAINLOOP_E2E_HARNESS=Pi CHAINLOOP_E2E_MODEL=openrouter/anthropic/claude-haiku-5.5 CHAINLOOP_E2E_FEATURE='Cache token accounting' make -C app/cli test-trace-e2e

The live run had nonzero cache reads and writes and passed with the matrix expecting Yes.

make -C app/cli test still hits two existing local-environment failures that reproduce unchanged on upstream/main: cmd/TestLoadAuthToken when CHAINLOOP_TOKEN is inherited, and Codex TestCopySessionData* on the macOS storage-root path. The changed Pi and E2E packages pass under the race detector.

AI assistance

Vibhav's Pi assisted with implementation, testing, and review. Both commits are signed, DCO-compliant, and carry Assisted-by: Vibhav's Pi.

The repository requires a skip-ai-session label because the available Chainloop login cannot access the chainloop project to upload the session. GitHub denied the label update with HTTP 403 (repository admin rights required), so a maintainer must apply it. This does not bypass code, test, DCO, or signature checks.

Recompute Pi v2 totals from native input and output counters, preserve cache categories, and verify them against cache-active raw session records. Assisted-by: Vibhav's Pi Signed-off-by: Vibhav Bobade <vibhav.bobde@gmail.com>

chainloop-platform · 2026-10-24T21:33:41Z

PR validation — ⚠️ 1 failing

Status	Policy	Material	Messages
⚠️ Failed	`pr-min-approvals`	`pr-info`	PR/MR #3719 has 0 approving reviews, 1 required.
✅ Passed	`pr-description-required`	`pr-info`	-
✅ Passed	`pr-user-story-linked`	`pr-info`	-

_{View attestation ↗}

AI Session Checks — ⚠️ no AI session found

Missing AI Coding Sessions

This organization requires every PR to be backed by a Chainloop Trace AI coding session, and none was found for this one.

Please make sure the AI coding session evidence has been sent by the Chainloop CLI, or add the skip-ai-session label to this PR to bypass this check.

Learn more about Chainloop Trace.

Security Checks — ✅ 6 passing

✅ `secret-scan`

Status	Policy	Messages
✅ Passed	`secrets-detection`	-

✅ `sast-scan`

Status	Policy	Messages
✅ Passed	`owasp-top10-2025`	-
✅ Passed	`sast`	-
✅ Passed	`cwe-top25`	-
✅ Passed	`cwe-top26-40-cusp`	-

✅ `iac-scan`

Status	Policy	Messages
✅ Passed	`iac-misconfiguration`	-

Scans not applied (2)

Scan	Reason
`vulnerability-scan`	no manifest/lockfile changed
`github-actions-scan`	no workflow files changed

_{View attestation ↗}

Security context

✅ Nothing this change touches has a recorded security-fix history.

_{View in Chainloop ↗ · How this works ↗}

Powered by Chainloop and Chainloop Trace

cubic-dev-ai

No issues found across 7 files

_{View guided diff | Re-trigger cubic}

migmartri · 2026-10-26T14:50:15Z

 # Codex token accounting

-Codex and Claude Code evidence use disjoint token categories:
+Codex v2, Pi v2, and Claude Code evidence use disjoint token categories:


migmartri · 2026-10-26T14:51:33Z

 | Codex `v1` | Input includes cache; total includes input and output. | Use `total_tokens` without adding the cache fields. Counts may still contain the old overcount. |
 | Codex `v2` | Input and total exclude cache, matching the existing Claude convention. | Add both cache fields to `total_tokens` exactly once. |
+| Pi `v1` | Input is separate; total was supplied by Pi and may include cache. | Recompute from native input, output, and cache fields; do not assume supplied totals are consistent. |
+| Pi `v2` | Input and total exclude both cache categories. | Add both cache fields to `total_tokens` exactly once. |


I am confused, is there a v2 for pi?

Latest version is 1.4.0 as far as I know

migmartri · 2026-10-26T14:54:51Z

 			return err
 		}
+	case pi.Name:
+		if d.SchemaVersion != "v2" {


is schemaversion the default one? since what version of pi?

We might want to also support this one if it's the default

jiparis · 2026-10-26T18:13:25Z


 	return aicodingsession.NewEvidence(aicodingsession.Data{
-		SchemaVersion: "v1",
+		SchemaVersion: "v2",


There is no change in the schema in this PR, so we don't need this schema version change. We will remove this field probably in the near future, it's causing some confusion and doesn't provide any benefit.

migmartri · 2026-10-26T21:05:54Z

I see, so I was confused thinking that PI itself went to v2, when the change proposed a change in our session spec to v2. As @jiparis said, I'd not do it for now, let's just make the change in the calculation since there are no schema changes.

Also, can you make sure chainloop trace init adds support for pi?

Retain the existing Chainloop evidence version while normalizing Pi cache counters. Clarify that agent releases and evidence versions are distinct. Assisted-by: Vibhav's Pi Signed-off-by: Vibhav Bobade <vibhav.bobde@gmail.com>

Compare Pi usage with the native session records regardless of the Chainloop evidence version; only Codex requires its v2 accounting guard. Assisted-by: Vibhav's Pi Signed-off-by: Vibhav Bobade <vibhav.bobde@gmail.com>

fix(trace): separate Pi cache token accounting

e54b79a

Recompute Pi v2 totals from native input and output counters, preserve cache categories, and verify them against cache-active raw session records. Assisted-by: Vibhav's Pi Signed-off-by: Vibhav Bobade <vibhav.bobde@gmail.com>

cubic-dev-ai Bot reviewed Oct 24, 2026

View reviewed changes

This was referenced Oct 24, 2026

Trace: normalize Pi cache token totals #3720

Open

Trace: verify Pi credentials warning #3723

Open

migmartri requested a review from a team October 26, 2026 14:48

migmartri reviewed Oct 26, 2026

View reviewed changes

jiparis reviewed Oct 26, 2026

View reviewed changes

fix(trace): keep Pi cache evidence on v1

44ad917

Retain the existing Chainloop evidence version while normalizing Pi cache counters. Clarify that agent releases and evidence versions are distinct. Assisted-by: Vibhav's Pi Signed-off-by: Vibhav Bobade <vibhav.bobde@gmail.com>

migmartri previously approved these changes Oct 26, 2026

View reviewed changes

Comment thread app/cli/internal/trace/e2e/checks_tokens.go Outdated

fix(trace): avoid gating Pi cache checks on schema

b86eeb1

Compare Pi usage with the native session records regardless of the Chainloop evidence version; only Codex requires its v2 accounting guard. Assisted-by: Vibhav's Pi Signed-off-by: Vibhav Bobade <vibhav.bobde@gmail.com>

waveywaves dismissed migmartri’s stale review via b86eeb1 October 26, 2026 21:33

waveywaves mentioned this pull request Oct 26, 2026

fix(trace): attribute commits made inside Bash calls #3631

Open

Navigation Menu

Repository navigation

fix(trace): separate Pi cache token accounting - #3719

fix(trace): separate Pi cache token accounting#3719
waveywaves wants to merge 3 commits into
chainloop-dev:mainfrom
waveywaves:fix/pi-cache-accounting

waveywaves commented Oct 24, 2026 •

edited

Loading

Uh oh!

chainloop-platform Bot commented Oct 24, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

migmartri Oct 26, 2026

Uh oh!

migmartri Oct 26, 2026

Uh oh!

jiparis Oct 26, 2026

Uh oh!

migmartri Oct 26, 2026

Uh oh!

jiparis Oct 26, 2026

Uh oh!

migmartri commented Oct 26, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

waveywaves commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Trace provider checklist

Validation

AI assistance

Uh oh!

chainloop-platform Bot commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR validation — ⚠️ 1 failing

AI Session Checks — ⚠️ no AI session found

Missing AI Coding Sessions

Security Checks — ✅ 6 passing

✅ secret-scan

✅ sast-scan

✅ iac-scan

Security context

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

migmartri Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

migmartri Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

jiparis Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

migmartri Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

jiparis Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

migmartri commented Oct 26, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

waveywaves commented Oct 24, 2026 •

edited

Loading

chainloop-platform Bot commented Oct 24, 2026 •

edited

Loading

✅ `secret-scan`

✅ `sast-scan`

✅ `iac-scan`