Skip to content

feat(trace): capture pinned Pi subagent evidence - #3760

Closed
waveywaves wants to merge 2 commits into
chainloop-dev:mainfrom
waveywaves:feat/pi-subagent-evidence
Closed

waveywaves wants to merge 2 commits into
chainloop-dev:mainfrom
waveywaves:feat/pi-subagent-evidence

Conversation

@waveywaves

@waveywaves waveywaves commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #3713.

  • Parse single, parallel, and chain results from the official Pi subagent example pinned at 71023aec, shipped with Pi 1.3.0.
  • Preserve child conversations and tool results in deterministic, distinct raw-session streams.
  • Aggregate native child usage exactly once under the existing v1 contract: input plus output forms total tokens, caches stay separate, and reported cost remains authoritative.
  • Preserve root branch selection and ignore unrelated extension records or malformed optional child details without blocking push.
  • Add parser and material-packaging regressions plus a supported-format live E2E scenario that verifies the shipped example source hashes.

No schema or dependency change. Arbitrary third-party extensions and child inference from parentSession are outside this scope.

Validation

Passed locally:

  • go test -race -shuffle=on -count=1 ./app/cli/internal/trace/pi ./app/cli/internal/trace/e2e ./app/cli/pkg/action
  • CLI build and go vet for the affected packages.
  • Repository-configured golangci-lint v2.14.0, built with Go 1.27.3: zero issues in the changed packages.
  • Production CLI fixture replay through real hooks, commit, and a local bare-remote push. The stored material retained a distinct child stream and its read result. Root input/output of 2/2 plus child 3/5 produced 5/7 and total 12; caches remained 7/11 and reported cost remained 0.02.

The fixture replay above is not a live Pi/model run. Unit regressions cover single/parallel/chain modes, failures, malformed optional details, repeated results, and abandoned branches.

Real Pi follow-up and Trace

A separate manual check ran Pi 1.3.0 with openai-codex/gpt-6.1-sol, the existing repository .pi/extensions/chainloop-trace.ts, and the hash-verified official subagent example. It used a Trace-capable CLI built from this branch.

  • Real parent session: 01a18e94-0f8c-7007-a09c-25559a2d7ea5.
  • One real worker completed with exit code 0. Its read tool returned the first README heading, ## How it works.
  • The native completed result contains the child's conversation, tool result, model, and usage. The child reported input/output of 21272/418, cache read/write of 11776/0, cost of 0.0479016, and 2 turns. These are native observations, not a claim that the pinned matrix passed.
  • The focused subagent tests passed in the real parent session with -race -shuffle=on -count=1 across trace/pi, trace/e2e, and pkg/action.
  • Signed follow-up commit afad7548ff9e35733faf624bb29af818694d3785 documents that manual validation. The original implementation commit predates the recording and is not retrospectively claimed as traced.

The repository's chainloop/chainloop Trace destination is unchanged. The current CLI login is denied access to that project; the control-plane attempt returned PermissionDenied. The genuine native evidence is retained, but no uploaded session URL or attestation is claimed. The PR remains a draft.

Remaining validation

The pinned OpenRouter compatibility-matrix run is still blocked: the repository key returned 401 API key expired. The successful manual OAuth check is not a pass for that matrix.

CHAINLOOP_E2E_HARNESS=Pi \
CHAINLOOP_E2E_FEATURE=Subagents \
make -C app/cli test-trace-e2e

The capability remains Partial because support is limited to the pinned official result format.

Assisted-by: Vibhav's Pi

View guided diff

Decode single, parallel and chain results from the official Pi 1.3.0 example format pinned at 71023aec1546d6d677a65669c5665c0927ce0dd0. Preserve distinct child conversations and tool results and aggregate normalized v1 usage exactly once.

Add parser, action-packaging and supported-format E2E regressions for issue chainloop-dev#3713. Full affected checks and native production-CLI replay pass; live model validation is blocked by missing OpenRouter authentication.

Assisted-by: Vibhav's Pi
Signed-off-by: Vibhav Bobade <vibhav.bobde@gmail.com>
@chainloop-platform

chainloop-platform Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

PR validation — ⚠️ 1 failing

Status Policy Material Messages
⚠️ Failed pr-min-approvals pr-info PR/MR #3760 has 0 approving reviews, 1 required.
✅ Passed pr-description-required pr-info -
✅ Passed pr-user-story-linked pr-info -

View attestation ↗

AI Session Checks — ⚠️ no AI session found

Missing AI Coding Sessions

This organization requires every PR to be backed by a Chainloop Trace AI coding session, and none was found for this one.

Please make sure the AI coding session evidence has been sent by the Chainloop CLI, or add the skip-ai-session label to this PR to bypass this check.

Learn more about Chainloop Trace.

Security Checks — ✅ 5 passing

✅ secret-scan

Status Policy Messages
✅ Passed secrets-detection -

✅ sast-scan

Status Policy Messages
✅ Passed owasp-top10-2025 -
✅ Passed sast -
✅ Passed cwe-top25 -
✅ Passed cwe-top26-40-cusp -
Scans not applied (3)
Scan Reason
vulnerability-scan no manifest/lockfile changed
github-actions-scan no workflow files changed
iac-scan no IaC files changed

View attestation ↗

Security context

✅ Nothing this change touches has a recorded security-fix history.

View in Chainloop ↗ · How this works ↗


Powered by Chainloop and Chainloop Trace

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 13 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="app/cli/internal/trace/README.md">

<violation number="1" location="app/cli/internal/trace/README.md:60">
P2: The Pi Subagents cell is set to **Partial** without any end-to-end or manual result. This file says Pi cells stay **Unknown** until a live result or explicit manual check exists, and the new live scenario did not run. Set this cell back to **Unknown** until the pinned-format scenario passes.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

| Cache token accounting | Cache reads and writes stay in separate evidence fields. Base input and total exclude cache. The suite compares stored token counts with the raw usage records. | Unknown | No | Unknown | Unknown | Yes | Unknown |
| Model detection | The evidence names the primary model and the other models that the session used. | Yes | Partial | Yes | Yes | Yes | Yes |
| Subagents | The evidence holds the conversations of the subagents, and their token usage, as part of the session. | Yes | No | No | No | No | No |
| Subagents | The evidence holds the conversations of the subagents, and their token usage, as part of the session. | Yes | No | No | No | Partial | No |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The Pi Subagents cell is set to Partial without any end-to-end or manual result. This file says Pi cells stay Unknown until a live result or explicit manual check exists, and the new live scenario did not run. Set this cell back to Unknown until the pinned-format scenario passes.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At app/cli/internal/trace/README.md, line 60:

<comment>The Pi Subagents cell is set to **Partial** without any end-to-end or manual result. This file says Pi cells stay **Unknown** until a live result or explicit manual check exists, and the new live scenario did not run. Set this cell back to **Unknown** until the pinned-format scenario passes.</comment>

<file context>
@@ -53,12 +57,12 @@ What the session evidence holds about the conversation.
 | Cache token accounting | Cache reads and writes stay in separate evidence fields. Base input and total exclude cache. The suite compares stored token counts with the raw usage records. | Unknown | No | Unknown | Unknown | Yes | Unknown |
 | Model detection | The evidence names the primary model and the other models that the session used. | Yes | Partial | Yes | Yes | Yes | Yes |
-| Subagents | The evidence holds the conversations of the subagents, and their token usage, as part of the session. | Yes | No | No | No | No | No |
+| Subagents | The evidence holds the conversations of the subagents, and their token usage, as part of the session. | Yes | No | No | No | Partial | No |
 | Missed session start | A session whose session-start hook did not fire is still recorded, from its first tool hook. | Yes | Unknown | Unknown | Yes | Yes | Unknown |
 
</file context>
Suggested change
| Subagents | The evidence holds the conversations of the subagents, and their token usage, as part of the session. | Yes | No | No | No | Partial | No |
| Subagents | The evidence holds the conversations of the subagents, and their token usage, as part of the session. | Yes | No | No | No | Unknown | No |

Assisted-by: Vibhav's Pi
Signed-off-by: Vibhav Bobade <vibhav.bobde@gmail.com>
@waveywaves
waveywaves marked this pull request as ready for review October 30, 2026 21:43
@waveywaves waveywaves closed this Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Trace: capture Pi subagent evidence from a pinned result format

1 participant