Repository navigation
feat(trace): capture pinned Pi subagent evidence - #3760
waveywaves wants to merge 2 commits into
Conversation
Decode single, parallel and chain results from the official Pi 1.3.0 example format pinned at 71023aec1546d6d677a65669c5665c0927ce0dd0. Preserve distinct child conversations and tool results and aggregate normalized v1 usage exactly once. Add parser, action-packaging and supported-format E2E regressions for issue chainloop-dev#3713. Full affected checks and native production-CLI replay pass; live model validation is blocked by missing OpenRouter authentication. Assisted-by: Vibhav's Pi Signed-off-by: Vibhav Bobade <vibhav.bobde@gmail.com>
PR validation —
|
| Status | Policy | Material | Messages |
|---|---|---|---|
pr-min-approvals |
pr-info |
PR/MR #3760 has 0 approving reviews, 1 required. | |
| ✅ Passed | pr-description-required |
pr-info |
- |
| ✅ Passed | pr-user-story-linked |
pr-info |
- |
AI Session Checks — ⚠️ no AI session found
Missing AI Coding Sessions
This organization requires every PR to be backed by a Chainloop Trace AI coding session, and none was found for this one.
Please make sure the AI coding session evidence has been sent by the Chainloop CLI, or add the skip-ai-session label to this PR to bypass this check.
Learn more about Chainloop Trace.
Security Checks — ✅ 5 passing
✅ secret-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | secrets-detection |
- |
✅ sast-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | owasp-top10-2025 |
- |
| ✅ Passed | sast |
- |
| ✅ Passed | cwe-top25 |
- |
| ✅ Passed | cwe-top26-40-cusp |
- |
Scans not applied (3)
| Scan | Reason |
|---|---|
vulnerability-scan |
no manifest/lockfile changed |
github-actions-scan |
no workflow files changed |
iac-scan |
no IaC files changed |
Security context
✅ Nothing this change touches has a recorded security-fix history.
View in Chainloop ↗ · How this works ↗
Powered by Chainloop and Chainloop Trace
There was a problem hiding this comment.
1 issue found across 13 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="app/cli/internal/trace/README.md">
<violation number="1" location="app/cli/internal/trace/README.md:60">
P2: The Pi Subagents cell is set to **Partial** without any end-to-end or manual result. This file says Pi cells stay **Unknown** until a live result or explicit manual check exists, and the new live scenario did not run. Set this cell back to **Unknown** until the pinned-format scenario passes.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
| | Cache token accounting | Cache reads and writes stay in separate evidence fields. Base input and total exclude cache. The suite compares stored token counts with the raw usage records. | Unknown | No | Unknown | Unknown | Yes | Unknown | | ||
| | Model detection | The evidence names the primary model and the other models that the session used. | Yes | Partial | Yes | Yes | Yes | Yes | | ||
| | Subagents | The evidence holds the conversations of the subagents, and their token usage, as part of the session. | Yes | No | No | No | No | No | | ||
| | Subagents | The evidence holds the conversations of the subagents, and their token usage, as part of the session. | Yes | No | No | No | Partial | No | |
There was a problem hiding this comment.
P2: The Pi Subagents cell is set to Partial without any end-to-end or manual result. This file says Pi cells stay Unknown until a live result or explicit manual check exists, and the new live scenario did not run. Set this cell back to Unknown until the pinned-format scenario passes.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At app/cli/internal/trace/README.md, line 60:
<comment>The Pi Subagents cell is set to **Partial** without any end-to-end or manual result. This file says Pi cells stay **Unknown** until a live result or explicit manual check exists, and the new live scenario did not run. Set this cell back to **Unknown** until the pinned-format scenario passes.</comment>
<file context>
@@ -53,12 +57,12 @@ What the session evidence holds about the conversation.
| Cache token accounting | Cache reads and writes stay in separate evidence fields. Base input and total exclude cache. The suite compares stored token counts with the raw usage records. | Unknown | No | Unknown | Unknown | Yes | Unknown |
| Model detection | The evidence names the primary model and the other models that the session used. | Yes | Partial | Yes | Yes | Yes | Yes |
-| Subagents | The evidence holds the conversations of the subagents, and their token usage, as part of the session. | Yes | No | No | No | No | No |
+| Subagents | The evidence holds the conversations of the subagents, and their token usage, as part of the session. | Yes | No | No | No | Partial | No |
| Missed session start | A session whose session-start hook did not fire is still recorded, from its first tool hook. | Yes | Unknown | Unknown | Yes | Yes | Unknown |
</file context>
| | Subagents | The evidence holds the conversations of the subagents, and their token usage, as part of the session. | Yes | No | No | No | Partial | No | | |
| | Subagents | The evidence holds the conversations of the subagents, and their token usage, as part of the session. | Yes | No | No | No | Unknown | No | |
Assisted-by: Vibhav's Pi Signed-off-by: Vibhav Bobade <vibhav.bobde@gmail.com>
Summary
Fixes #3713.
71023aec, shipped with Pi 1.3.0.No schema or dependency change. Arbitrary third-party extensions and child inference from
parentSessionare outside this scope.Validation
Passed locally:
go test -race -shuffle=on -count=1 ./app/cli/internal/trace/pi ./app/cli/internal/trace/e2e ./app/cli/pkg/actiongo vetfor the affected packages.readresult. Root input/output of 2/2 plus child 3/5 produced 5/7 and total 12; caches remained 7/11 and reported cost remained 0.02.The fixture replay above is not a live Pi/model run. Unit regressions cover single/parallel/chain modes, failures, malformed optional details, repeated results, and abandoned branches.
Real Pi follow-up and Trace
A separate manual check ran Pi 1.3.0 with
openai-codex/gpt-6.1-sol, the existing repository.pi/extensions/chainloop-trace.ts, and the hash-verified official subagent example. It used a Trace-capable CLI built from this branch.01a18e94-0f8c-7007-a09c-25559a2d7ea5.readtool returned the first README heading,## How it works.-race -shuffle=on -count=1acrosstrace/pi,trace/e2e, andpkg/action.afad7548ff9e35733faf624bb29af818694d3785documents that manual validation. The original implementation commit predates the recording and is not retrospectively claimed as traced.The repository's
chainloop/chainloopTrace destination is unchanged. The current CLI login is denied access to that project; the control-plane attempt returnedPermissionDenied. The genuine native evidence is retained, but no uploaded session URL or attestation is claimed. The PR remains a draft.Remaining validation
The pinned OpenRouter compatibility-matrix run is still blocked: the repository key returned
401 API key expired. The successful manual OAuth check is not a pass for that matrix.The capability remains
Partialbecause support is limited to the pinned official result format.Assisted-by: Vibhav's Pi