From 955337c6a0b4745637fd5ca80274c280273fd40d Mon Sep 17 00:00:00 2001 From: Vibhav Bobade Date: Fri, 30 Oct 2026 23:52:51 +0530 Subject: [PATCH 1/2] feat(trace): capture pinned Pi subagent evidence Decode single, parallel and chain results from the official Pi 1.3.0 example format pinned at 71023aec1546d6d677a65669c5665c0927ce0dd0. Preserve distinct child conversations and tool results and aggregate normalized v1 usage exactly once. Add parser, action-packaging and supported-format E2E regressions for issue #3713. Full affected checks and native production-CLI replay pass; live model validation is blocked by missing OpenRouter authentication. Assisted-by: Vibhav's Pi Signed-off-by: Vibhav Bobade --- app/cli/internal/trace/README.md | 20 +- app/cli/internal/trace/e2e/checks.go | 3 + app/cli/internal/trace/e2e/checks_pi.go | 191 +++++++++++++++++++ app/cli/internal/trace/e2e/checks_pi_test.go | 111 +++++++++++ app/cli/internal/trace/e2e/checks_tokens.go | 33 +--- app/cli/internal/trace/e2e/harness.go | 2 + app/cli/internal/trace/e2e/pi.go | 11 +- app/cli/internal/trace/e2e/pi_subagent.go | 82 ++++++++ app/cli/internal/trace/e2e/runner.go | 2 +- app/cli/internal/trace/pi/parse.go | 20 +- app/cli/internal/trace/pi/subagent.go | 132 +++++++++++++ app/cli/internal/trace/pi/subagent_test.go | 172 +++++++++++++++++ app/cli/pkg/action/trace_pi_subagent_test.go | 75 ++++++++ 13 files changed, 817 insertions(+), 37 deletions(-) create mode 100644 app/cli/internal/trace/e2e/checks_pi.go create mode 100644 app/cli/internal/trace/e2e/checks_pi_test.go create mode 100644 app/cli/internal/trace/e2e/pi_subagent.go create mode 100644 app/cli/internal/trace/pi/subagent.go create mode 100644 app/cli/internal/trace/pi/subagent_test.go create mode 100644 app/cli/pkg/action/trace_pi_subagent_test.go diff --git a/app/cli/internal/trace/README.md b/app/cli/internal/trace/README.md index f1f648639..878dd22f4 100644 --- a/app/cli/internal/trace/README.md +++ b/app/cli/internal/trace/README.md @@ -14,7 +14,11 @@ Pi supports the JSONL v3 format used by Pi 0.80.10. The compatibility suite pins Pi evidence preserves native `input`, `output`, `cacheRead`, and `cacheWrite` as separate categories. Evidence `total_tokens` is recomputed as base input + output, excluding both cache categories, regardless of supplied `totalTokens`. The 100/20/10/5 fixture therefore records total 120, not 135. Supplied `cost.total` remains authoritative, without repricing. Invalid or overflowing counters are ignored with deterministic warnings; if input + output overflows, the evidence total is omitted with a warning while individual counters remain. Missing usage contributes zero. Only the selected branch is counted, including its history after resume, not abandoned branches or complete session spend. The Chainloop evidence field `schema_version` remains `v1`; it is not a Pi release or JSONL format version. Earlier Chainloop parsers used supplied totals, which may include cache, under the same evidence version; normalize historical evidence from raw records before combining totals. -Pi 1.3.0 passed 26 feature checks in the local end-to-end suite, including a cache-active accounting run. Pi 0.80.10 also completed targeted TUI and JSON-headless smokes that exercised session start, `write`, `edit`, `bash`, commit attribution and session shutdown; the TUI showed the local-mode welcome message. Subagents, skills tracking, spec source pointers and pasted image pointers are not implemented. The headless suite cannot verify the UI-only welcome message, the control-plane credentials warning, the session link after a control-plane push or control-plane attestation. The multi-agent feature has no registered check. +Pi 1.3.0 passed 26 feature checks in the local end-to-end suite, including a cache-active accounting run. Pi 0.80.10 also completed targeted TUI and JSON-headless smokes that exercised session start, `write`, `edit`, `bash`, commit attribution and session shutdown; the TUI showed the local-mode welcome message. The new pinned-format subagent scenario was attempted with Pi 1.3.0, but could not run because the validation environment has no OpenRouter API key. Skills tracking, spec source pointers and pasted image pointers are not implemented. The headless suite cannot verify the UI-only welcome message, the control-plane credentials warning, the session link after a control-plane push or control-plane attestation. The multi-agent feature has no registered check. + +Pi subagent support is limited to the official example extension's completed result format at [commit `71023aec1546d6d677a65669c5665c0927ce0dd0`](https://github.com/earendil-works/pi/tree/71023aec1546d6d677a65669c5665c0927ce0dd0/packages/coding-agent/examples/extensions/subagent), shipped in [Pi 1.3.0](https://github.com/earendil-works/pi/blob/71023aec1546d6d677a65669c5665c0927ce0dd0/packages/coding-agent/package.json). The supporting sources are [`index.ts`](https://github.com/earendil-works/pi/blob/71023aec1546d6d677a65669c5665c0927ce0dd0/packages/coding-agent/examples/extensions/subagent/index.ts), [`agents.ts`](https://github.com/earendil-works/pi/blob/71023aec1546d6d677a65669c5665c0927ce0dd0/packages/coding-agent/examples/extensions/subagent/agents.ts), and its [README](https://github.com/earendil-works/pi/blob/71023aec1546d6d677a65669c5665c0927ce0dd0/packages/coding-agent/examples/extensions/subagent/README.md). This does not claim arbitrary third-party extension support. + +For `single`, `parallel`, and `chain` results on the selected root branch, Trace reads `toolResult.details.results`, assigns IDs `subagent--<1-based-result-index>`, and preserves each child's native messages, including tool results, in its own `raw_session` stream. Child messages are wrapped as Pi entries; missing message timestamps use the containing result's timestamp. Root conversation and tool summaries stay root-only, and the primary model stays the last selected root assistant's model. Child assistant models join `models_used`. Usage adds the completed result's aggregate `input`, `output`, `cacheRead`, `cacheWrite`, and numeric `cost` exactly once, not again from embedded assistant messages. Cache stays separate, `total_tokens` remains input + output, cost is not repriced, and `schema_version` remains `v1`. Failed children are retained; unavailable conversations or malformed optional data produce best-effort evidence and warnings. Abandoned branches and unrelated tools do not contribute children. The example runs children with `--no-session`; `parentSession` is not used to infer children. Codex uses native hooks in `.codex/hooks.json` (Codex CLI 0.163.0). Run `chainloop trace init --codex` or `chainloop trace run --local --codex -- codex exec …`. Review and trust the installed hooks in Codex before starting a session. See the [official hooks documentation](https://learn.chatgpt.com/docs/hooks). Installed hooks have a `Chainloop Trace` status label in Codex. Re-running init adds labels to existing Chainloop hooks. Changing the hooks requires a new trust review. @@ -53,12 +57,12 @@ What the session evidence holds about the conversation. | Token usage and cost | The evidence holds the tokens that the session used and their cost in USD. | Yes | No | Yes | Yes | Yes | Yes | | Cache token accounting | Cache reads and writes stay in separate evidence fields. Base input and total exclude cache. The suite compares stored token counts with the raw usage records. | Unknown | No | Unknown | Unknown | Yes | Unknown | | Model detection | The evidence names the primary model and the other models that the session used. | Yes | Partial | Yes | Yes | Yes | Yes | -| Subagents | The evidence holds the conversations of the subagents, and their token usage, as part of the session. | Yes | No | No | No | No | No | +| Subagents | The evidence holds the conversations of the subagents, and their token usage, as part of the session. | Yes | No | No | No | Partial | No | | Missed session start | A session whose session-start hook did not fire is still recorded, from its first tool hook. | Yes | Unknown | Unknown | Yes | Yes | Unknown | Why not Yes: -- **Subagents, Pi:** the provider records the selected root session only. Child session transcripts are not collected. +- **Subagents, Pi:** only the commit-pinned official example result format is supported, not arbitrary extensions or inferred child sessions. Parser and packaging regressions cover all three modes, usage, and child content; the supported-format live scenario is blocked by missing OpenRouter authentication. - **Missed session start, Codex:** the scenario produces evidence, but its check also needs hook messages that Codex does not report in headless mode. - **Transcript, OpenCode:** OpenCode has no transcript file. The provider runs `opencode export` (1.x) or `opencode session export` (2) and rebuilds the transcript from the export. It keeps the text and the inputs of the tool calls. It drops the tool outputs, the reasoning and the file parts. OpenCode 2 exports also have no agent version and no session slug. - **Token usage and cost, Cursor:** the Cursor transcript does not hold token usage. Usage and cost are zero, and the evidence has a warning that tells so. (For OpenCode, the cost is the value that OpenCode reports. For Claude Code, Chainloop computes it from its price table.) @@ -243,6 +247,16 @@ To run Pi only, set `OPENROUTER_API_KEY` in your environment, then run: CHAINLOOP_E2E_HARNESS=Pi make -C app/cli test-trace-e2e ``` +To check the supported Pi subagent format only, use an npm installation of Pi 1.3.0 with its shipped examples and set `OPENROUTER_API_KEY`, then run: + +```bash +CHAINLOOP_E2E_HARNESS=Pi \ +CHAINLOOP_E2E_FEATURE=Subagents \ +make -C app/cli test-trace-e2e +``` + +This scenario loads the real official example from the installed package and checks the SHA-256 of both `index.ts` and `agents.ts` against the pinned commit before running. It uses an isolated read-only worker, then lets the parent write, commit, and push after the child returns. Its check compares stored child conversations and tool-result content with the native completed result and verifies parent plus child usage, cache, and cost exactly once. This is a live test command, not a recorded pass. + To retain the workspace and agent output for a failure: ```bash diff --git a/app/cli/internal/trace/e2e/checks.go b/app/cli/internal/trace/e2e/checks.go index fd452000b..d2e1530a6 100644 --- a/app/cli/internal/trace/e2e/checks.go +++ b/app/cli/internal/trace/e2e/checks.go @@ -437,6 +437,9 @@ func checkTicketPointer(s *Session) error { } func checkSubagents(s *Session) error { + if evidence(s).Agent.Name == "pi" { + return checkPiSubagents(evidence(s)) + } subs := evidence(s).Subagents if len(subs) == 0 { return errors.New("the evidence has no subagent") diff --git a/app/cli/internal/trace/e2e/checks_pi.go b/app/cli/internal/trace/e2e/checks_pi.go new file mode 100644 index 000000000..2267c1f81 --- /dev/null +++ b/app/cli/internal/trace/e2e/checks_pi.go @@ -0,0 +1,191 @@ +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package e2e + +import ( + "bytes" + "encoding/json" + "errors" + "fmt" + "math" + "math/big" + "reflect" + + "github.com/chainloop-dev/chainloop/pkg/attestation/crafter/materials/aicodingsession" +) + +type piSourceRecord struct { + Type string `json:"type"` + ID string `json:"id"` + Message struct { + Role string `json:"role"` + ToolName string `json:"toolName"` + Usage map[string]json.RawMessage `json:"usage"` + Details struct { + Mode string `json:"mode"` + Results []json.RawMessage `json:"results"` + } `json:"details"` + } `json:"message"` +} + +type piSourceChild struct { + Agent string `json:"agent"` + Task string `json:"task"` + Usage map[string]json.RawMessage `json:"usage"` + Messages []json.RawMessage `json:"messages"` +} + +// Compare stored children with native completed results, independently of the +// provider parser. Nonempty aggregate metadata alone cannot pass this check. +func checkPiSubagents(d *aicodingsession.Data) error { + var expected []aicodingsession.Subagent + hasToolResult := false + for _, raw := range d.RawSession["main"] { + var r piSourceRecord + if json.Unmarshal(raw, &r) != nil || !piSourceSubagent(r) { + continue + } + for index, rawChild := range r.Message.Details.Results { + var child piSourceChild + if err := json.Unmarshal(rawChild, &child); err != nil { + return fmt.Errorf("decode Pi source child: %w", err) + } + id := fmt.Sprintf("subagent-%s-%d", r.ID, index+1) + var usage aicodingsession.Usage + addPiSourceUsage(&usage, child.Usage, child.Usage["cost"]) + expected = append(expected, aicodingsession.Subagent{ + ID: id, Type: child.Agent, Description: child.Task, + Tokens: aicodingsession.SubagentTokens{Input: usage.InputTokens, Output: usage.OutputTokens}, + }) + stream := d.RawSession[id] + if len(stream) == 0 || len(stream) != len(child.Messages) { + return fmt.Errorf("Pi child %s conversation is missing or incomplete", id) + } + for i, source := range child.Messages { + var entry struct { + Message json.RawMessage `json:"message"` + } + if json.Unmarshal(stream[i], &entry) != nil { + return fmt.Errorf("Pi child %s has an invalid stored message", id) + } + // Material assembly may indent or reorder fields. + var stored, original bytes.Buffer + if json.Compact(&stored, entry.Message) != nil || json.Compact(&original, source) != nil { + return fmt.Errorf("Pi child %s has invalid message JSON", id) + } + var got, want map[string]json.RawMessage + if json.Unmarshal(stored.Bytes(), &got) != nil || json.Unmarshal(original.Bytes(), &want) != nil || !reflect.DeepEqual(got, want) { + return fmt.Errorf("Pi child %s message %d differs from source content", id, i+1) + } + var message struct { + Role string `json:"role"` + Content []struct { + Text string `json:"text"` + } `json:"content"` + } + if json.Unmarshal(source, &message) == nil && message.Role == "toolResult" { + for _, block := range message.Content { + hasToolResult = hasToolResult || block.Text != "" + } + } + } + } + } + if len(expected) == 0 || !reflect.DeepEqual(expected, d.Subagents) { + return errors.New("Pi subagents do not match native completed results") + } + if !hasToolResult { + return errors.New("Pi subagent scenario has no readable child tool result") + } + if len(d.RawSession) != len(expected)+1 { + return errors.New("Pi evidence has unrelated child streams") + } + want := piSourceUsage(d.RawSession["main"]) + if d.SchemaVersion != "v1" || d.Usage == nil || *d.Usage != want { + return fmt.Errorf("Pi usage does not count parent and child aggregates exactly once: got %+v, want %+v", d.Usage, want) + } + return nil +} + +func piSourceSubagent(r piSourceRecord) bool { + if r.Type != "message" || r.Message.Role != "toolResult" || r.Message.ToolName != "subagent" { + return false + } + switch r.Message.Details.Mode { + case "single", "parallel", "chain": + return true + default: + return false + } +} + +func piSourceUsage(records []json.RawMessage) aicodingsession.Usage { + var total aicodingsession.Usage + for _, raw := range records { + var r piSourceRecord + if json.Unmarshal(raw, &r) != nil || r.Type != "message" { + continue + } + if r.Message.Role == "assistant" { + var cost struct { + Total json.RawMessage `json:"total"` + } + if json.Unmarshal(r.Message.Usage["cost"], &cost) != nil { + cost.Total = nil + } + addPiSourceUsage(&total, r.Message.Usage, cost.Total) + } + if piSourceSubagent(r) { + for _, rawChild := range r.Message.Details.Results { + var child piSourceChild + if json.Unmarshal(rawChild, &child) == nil && child.Agent != "" { + addPiSourceUsage(&total, child.Usage, child.Usage["cost"]) + } + } + } + } + if total.OutputTokens <= int(^uint(0)>>1)-total.InputTokens { + total.TotalTokens = total.InputTokens + total.OutputTokens + } + if total.EstimatedCostUSD <= math.MaxFloat64/10000 { + total.EstimatedCostUSD = math.Round(total.EstimatedCostUSD*10000) / 10000 + } + return total +} + +func addPiSourceUsage(total *aicodingsession.Usage, usage map[string]json.RawMessage, rawCost json.RawMessage) { + for _, field := range []struct { + name string + total *int + }{ + {"input", &total.InputTokens}, + {"output", &total.OutputTokens}, + {"cacheRead", &total.CacheReadInputTokens}, + {"cacheWrite", &total.CacheCreationInputTokens}, + } { + value, ok := new(big.Rat).SetString(string(usage[field.name])) + if !ok || value.Sign() < 0 || !value.IsInt() || !value.Num().IsInt64() { + continue + } + count := value.Num().Int64() + if count <= int64(int(^uint(0)>>1)-*field.total) { + *field.total += int(count) + } + } + var cost float64 + if json.Unmarshal(rawCost, &cost) == nil && cost >= 0 && !math.IsInf(cost+total.EstimatedCostUSD, 0) { + total.EstimatedCostUSD += cost + } +} diff --git a/app/cli/internal/trace/e2e/checks_pi_test.go b/app/cli/internal/trace/e2e/checks_pi_test.go new file mode 100644 index 000000000..a344cbd25 --- /dev/null +++ b/app/cli/internal/trace/e2e/checks_pi_test.go @@ -0,0 +1,111 @@ +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package e2e + +import ( + "encoding/json" + "os" + "path/filepath" + "testing" + + "github.com/chainloop-dev/chainloop/app/cli/internal/trace/pi" + "github.com/chainloop-dev/chainloop/pkg/attestation/crafter/materials/aicodingsession" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +const piCheckChildID = "subagent-result-1" + +func piSubagentCheckSession() *Session { + const assistant = `{"role":"assistant","content":[{"type":"toolCall","id":"read-1","name":"read","arguments":{"path":"README.md"}}],"usage":{"input":3,"output":5,"cacheRead":11,"cacheWrite":19,"cost":{"total":0.025}}}` + const toolResult = `{"role":"toolResult","toolCallId":"read-1","toolName":"read","content":[{"type":"text","text":"file contents"}]}` + s := cacheSession(pi.Name, + `{"type":"message","id":"root","message":{"role":"assistant","usage":{"input":7,"output":9,"cacheRead":13,"cacheWrite":17,"cost":{"total":0.1}}}}`, + `{"type":"message","id":"result","message":{"role":"toolResult","toolName":"subagent","details":{"mode":"single","results":[{"agent":"worker","task":"read","usage":{"input":3,"output":5,"cacheRead":11,"cacheWrite":19,"cost":0.025},"messages":[`+assistant+`,`+toolResult+`]}]}}}`, + ) + d := evidence(s) + d.Usage = &aicodingsession.Usage{ + InputTokens: 10, OutputTokens: 14, TotalTokens: 24, + CacheReadInputTokens: 24, CacheCreationInputTokens: 36, EstimatedCostUSD: 0.125, + } + d.Subagents = []aicodingsession.Subagent{{ + ID: piCheckChildID, Type: "worker", Description: "read", + Tokens: aicodingsession.SubagentTokens{Input: 3, Output: 5}, + }} + d.RawSession[piCheckChildID] = []json.RawMessage{ + json.RawMessage(`{"type":"message","message":` + assistant + `}`), + json.RawMessage(`{"type":"message","message":` + toolResult + `}`), + } + return s +} + +func TestCheckPiSubagentsWhenStoredEvidenceChanges(t *testing.T) { + for _, tc := range []struct { + name string + change func(*aicodingsession.Data) + }{ + {name: "complete child content and exact usage"}, + {name: "missing child conversation", change: func(d *aicodingsession.Data) { delete(d.RawSession, piCheckChildID) }}, + {name: "lost tool result", change: func(d *aicodingsession.Data) { + d.RawSession[piCheckChildID] = d.RawSession[piCheckChildID][:1] + }}, + {name: "changed tool content", change: func(d *aicodingsession.Data) { + d.RawSession[piCheckChildID][1] = json.RawMessage(`{"message":{"role":"toolResult","content":[{"type":"text","text":"truncated"}]}}`) + }}, + {name: "wrong child identity", change: func(d *aicodingsession.Data) { d.Subagents[0].ID = "wrong" }}, + {name: "lost child tokens", change: func(d *aicodingsession.Data) { d.Subagents[0].Tokens.Input = 0 }}, + {name: "double counted usage", change: func(d *aicodingsession.Data) { d.Usage.InputTokens += 3; d.Usage.TotalTokens += 3 }}, + {name: "lost cache", change: func(d *aicodingsession.Data) { d.Usage.CacheReadInputTokens = 13 }}, + {name: "lost native cost", change: func(d *aicodingsession.Data) { d.Usage.EstimatedCostUSD = 0.1 }}, + {name: "unrelated stream", change: func(d *aicodingsession.Data) { d.RawSession["unrelated"] = nil }}, + {name: "wrong evidence version", change: func(d *aicodingsession.Data) { d.SchemaVersion = "v2" }}, + } { + t.Run(tc.name, func(t *testing.T) { + // Given + session := piSubagentCheckSession() + if tc.change != nil { + tc.change(evidence(session)) + } + // When + err := checkSubagents(session) + // Then + if tc.change == nil { + require.NoError(t, err) + require.NoError(t, checkCacheTokenAccounting(session)) + } else { + require.Error(t, err) + } + }) + } +} + +func TestPiSubagentSourcePinWhenMissingOrModified(t *testing.T) { + for _, modified := range []bool{false, true} { + t.Run(map[bool]string{false: "missing", true: "modified"}[modified], func(t *testing.T) { + // Given + dir := t.TempDir() + if modified { + require.NoError(t, os.WriteFile(filepath.Join(dir, "index.ts"), []byte("different extension"), 0o600)) + } + // When + err := verifyPiSubagentSource(dir) + // Then + require.Error(t, err) + if modified { + assert.ErrorContains(t, err, piSubagentCommit) + } + }) + } +} diff --git a/app/cli/internal/trace/e2e/checks_tokens.go b/app/cli/internal/trace/e2e/checks_tokens.go index bb73c4690..5c0b80023 100644 --- a/app/cli/internal/trace/e2e/checks_tokens.go +++ b/app/cli/internal/trace/e2e/checks_tokens.go @@ -18,7 +18,6 @@ import ( "encoding/json" "errors" "fmt" - "math/big" "github.com/chainloop-dev/chainloop/app/cli/internal/trace/claude" "github.com/chainloop-dev/chainloop/app/cli/internal/trace/codex" @@ -44,35 +43,9 @@ func checkCacheTokenAccounting(s *Session) error { } case pi.Name: // Pi raw_session contains only the selected branch. Read native fields - // independently of the provider parser; source totalTokens may include cache. - for _, raw := range d.RawSession["main"] { - var r struct { - Type string `json:"type"` - Message struct { - Role string `json:"role"` - Usage map[string]json.RawMessage `json:"usage"` - } `json:"message"` - } - if json.Unmarshal(raw, &r) != nil || r.Type != "message" || r.Message.Role != "assistant" { - continue - } - for _, field := range []struct { - name string - total *int - }{ - {"input", &want.InputTokens}, {"output", &want.OutputTokens}, - {"cacheRead", &want.CacheReadInputTokens}, {"cacheWrite", &want.CacheCreationInputTokens}, - } { - value, ok := new(big.Rat).SetString(string(r.Message.Usage[field.name])) - if !ok || value.Sign() < 0 || !value.IsInt() || !value.Num().IsInt64() { - continue - } - count := value.Num().Int64() - if count <= int64(int(^uint(0)>>1)-*field.total) { - *field.total += int(count) - } - } - } + // and child aggregate usage independently of the provider parser. + want = piSourceUsage(d.RawSession["main"]) + want.EstimatedCostUSD = 0 if want.OutputTokens > int(^uint(0)>>1)-want.InputTokens { return errors.New("Pi source input + output token total overflows") } diff --git a/app/cli/internal/trace/e2e/harness.go b/app/cli/internal/trace/e2e/harness.go index c65b9d6da..9cecdb53d 100644 --- a/app/cli/internal/trace/e2e/harness.go +++ b/app/cli/internal/trace/e2e/harness.go @@ -32,6 +32,8 @@ const ( // RunRequest is one headless session of a harness. type RunRequest struct { + // Scenario lets an adapter prepare a harness-specific extension. + Scenario string // Dir is the repository the session runs in. Dir string // Model is the model of the session, as the harness names it. diff --git a/app/cli/internal/trace/e2e/pi.go b/app/cli/internal/trace/e2e/pi.go index 481e08938..b6adcca4c 100644 --- a/app/cli/internal/trace/e2e/pi.go +++ b/app/cli/internal/trace/e2e/pi.go @@ -98,6 +98,15 @@ func (p *Pi) Run(ctx context.Context, req RunRequest) (*RunOutput, error) { flagModel, req.Model, "--session-dir", sessionDir, } + prompt := req.Prompt + if req.Scenario == scenarioSubagent { + extension, err := p.prepareSubagent(ctx, base) + if err != nil { + return nil, err + } + args = append(args, "--extension", extension) + prompt = piSubagentPrompt + } // A pasted image goes in as an attached file, from outside the // repository, so that the agent cannot commit it. @@ -115,7 +124,7 @@ func (p *Pi) Run(ctx context.Context, req RunRequest) (*RunOutput, error) { args = append(args, "@"+path) } } - args = append(args, req.Prompt) + args = append(args, prompt) name := p.bin() if len(req.Wrapper) > 0 { diff --git a/app/cli/internal/trace/e2e/pi_subagent.go b/app/cli/internal/trace/e2e/pi_subagent.go new file mode 100644 index 000000000..33e59e927 --- /dev/null +++ b/app/cli/internal/trace/e2e/pi_subagent.go @@ -0,0 +1,82 @@ +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package e2e + +import ( + "context" + "crypto/sha256" + "fmt" + "os" + "os/exec" + "path/filepath" +) + +const piSubagentCommit = "71023aec1546d6d677a65669c5665c0927ce0dd0" + +// The parent writes and pushes after the child returns so the push snapshot +// contains the completed result, not an in-flight subagent invocation. +const piSubagentPrompt = `This is an automated test. Do exactly these steps in order: +1. Call the subagent tool with agent worker and agentScope user. Tell it to read README.md with its read tool and report the heading. +2. After the child returns, use your write tool to create ` + subagentFile + ` with the single line: from the subagent. +3. In a separate shell command run: git add ` + subagentFile + ` && git commit -m "add subagent file" && git push` + +func (p *Pi) prepareSubagent(ctx context.Context, base string) (string, error) { + version, err := p.Version(ctx) + if err != nil { + return "", err + } + if version != "1.3.0" { + return "", fmt.Errorf("Pi subagent scenario requires 1.3.0, got %s", version) + } + bin, err := exec.LookPath(p.bin()) + if err != nil { + return "", err + } + bin, err = filepath.EvalSymlinks(bin) + if err != nil { + return "", err + } + // npm ships these files beside dist/bundle/cli.js. Keep them there so + // their imports resolve against the pinned package, not the workspace. + dir := filepath.Join(filepath.Dir(bin), "..", "..", "examples", "extensions", "subagent") + if err := verifyPiSubagentSource(dir); err != nil { + return "", err + } + agentDir := filepath.Join(base, "pi-agent", "agents") + if err := os.MkdirAll(agentDir, 0o700); err != nil { + return "", err + } + const worker = "---\nname: worker\ndescription: Read the requested file.\ntools: read\n---\nRead the requested file with the read tool, then report its heading. Do not modify files or start other agents.\n" + if err := os.WriteFile(filepath.Join(agentDir, "worker.md"), []byte(worker), 0o600); err != nil { + return "", err + } + return filepath.Join(dir, "index.ts"), nil +} + +func verifyPiSubagentSource(dir string) error { + for _, file := range []struct{ name, digest string }{ + {"index.ts", "1d11858d163e2a3a258af51c1923fc55ad825813acd6224462fd25ad61f5401b"}, + {"agents.ts", "cc85ecd33b89aa575bea3d2a9a9da7654a6305da4de02e5041aa5342a1de29b4"}, + } { + data, err := os.ReadFile(filepath.Join(dir, file.name)) + if err != nil { + return fmt.Errorf("read pinned Pi subagent source: %w", err) + } + if fmt.Sprintf("%x", sha256.Sum256(data)) != file.digest { + return fmt.Errorf("Pi subagent %s differs from pinned commit %s", file.name, piSubagentCommit) + } + } + return nil +} diff --git a/app/cli/internal/trace/e2e/runner.go b/app/cli/internal/trace/e2e/runner.go index 64d9056d6..be982ee33 100644 --- a/app/cli/internal/trace/e2e/runner.go +++ b/app/cli/internal/trace/e2e/runner.go @@ -222,7 +222,7 @@ func (r *Runner) runScenario(ctx context.Context, h Harness, model string, sc Sc return &Session{Scenario: sc.Name, Model: model, Harness: h, Workspace: ws, Output: &RunOutput{}, Checkout: ws.Dir}, nil } - req := RunRequest{Dir: ws.Dir, Model: model, Prompt: sc.Prompt, Env: ws.Env, Images: sc.Images} + req := RunRequest{Scenario: sc.Name, Dir: ws.Dir, Model: model, Prompt: sc.Prompt, Env: ws.Env, Images: sc.Images} if sc.WrapTraceRun { req.Wrapper = []string{r.ChainloopBin, "trace", "run", "--local", h.TraceFlag(), "--"} } diff --git a/app/cli/internal/trace/pi/parse.go b/app/cli/internal/trace/pi/parse.go index 4954227a9..c62c4a5eb 100644 --- a/app/cli/internal/trace/pi/parse.go +++ b/app/cli/internal/trace/pi/parse.go @@ -250,9 +250,11 @@ func selectLastBranch(entries []*sessionEntry, byID map[string]*sessionEntry) ([ type messageView struct { Role string + ToolName string Provider string Model string Content json.RawMessage + Details json.RawMessage Usage json.RawMessage HasUsage bool } @@ -270,10 +272,15 @@ type evidenceAccumulator struct { userMessages int assistantMsgs int warnings []string + subagents []aicodingsession.Subagent + childStreams map[string][]json.RawMessage } func buildEvidence(parsed *parsedSession, agentVersion string) *aicodingsession.Evidence { - acc := evidenceAccumulator{models: make(map[string]struct{}), tools: make(map[string]int)} + acc := evidenceAccumulator{ + models: make(map[string]struct{}), tools: make(map[string]int), + childStreams: make(map[string][]json.RawMessage), + } raw := make([]json.RawMessage, 0, len(parsed.Selected)) for _, entry := range parsed.Selected { @@ -289,6 +296,10 @@ func buildEvidence(parsed *parsedSession, agentVersion string) *aicodingsession. continue } switch message.Role { + case "toolResult": + if message.ToolName == "subagent" { + acc.addSubagentResults(entry, message.Details) + } case "user": acc.userMessages++ case "assistant": @@ -320,6 +331,7 @@ func buildEvidence(parsed *parsedSession, agentVersion string) *aicodingsession. } } duration := max(0, int(endedAt.Sub(parsed.StartedAt).Seconds())) + acc.childStreams["main"] = raw return aicodingsession.NewEvidence(aicodingsession.Data{ SchemaVersion: "v1", @@ -355,7 +367,8 @@ func buildEvidence(parsed *parsedSession, agentVersion string) *aicodingsession. UserMessages: acc.userMessages, AssistantMessages: acc.assistantMsgs, }, - RawSession: map[string][]json.RawMessage{"main": raw}, + Subagents: acc.subagents, + RawSession: acc.childStreams, Warnings: acc.warnings, }) } @@ -427,13 +440,16 @@ func decodeMessage(raw json.RawMessage) (messageView, bool) { } provider, _ := rawString(fields["provider"]) model, _ := rawString(fields["model"]) + toolName, _ := rawString(fields["toolName"]) usage, hasUsage := fields["usage"] return messageView{ Role: role, + ToolName: toolName, Provider: provider, Model: model, Content: cloneRaw(fields["content"]), + Details: cloneRaw(fields["details"]), Usage: cloneRaw(usage), HasUsage: hasUsage, }, true diff --git a/app/cli/internal/trace/pi/subagent.go b/app/cli/internal/trace/pi/subagent.go new file mode 100644 index 000000000..fab96a9c3 --- /dev/null +++ b/app/cli/internal/trace/pi/subagent.go @@ -0,0 +1,132 @@ +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package pi + +import ( + "encoding/json" + "fmt" + "time" + + "github.com/chainloop-dev/chainloop/pkg/attestation/crafter/materials/aicodingsession" +) + +// The official example at Pi commit 71023aec1546d6d677a65669c5665c0927ce0dd0 +// (v1.3.0), packages/coding-agent/examples/extensions/subagent/index.ts, +// persists completed results in toolResult.details. Children use --no-session; +// neither parentSession nor filesystem discovery identifies these children. +func (a *evidenceAccumulator) addSubagentResults(entry *sessionEntry, raw json.RawMessage) { + if len(raw) == 0 || isNull(raw) { + return + } + var details struct { + Mode string `json:"mode"` + Results []json.RawMessage `json:"results"` + } + if json.Unmarshal(raw, &details) != nil { + a.warnings = append(a.warnings, fmt.Sprintf("Pi subagent result on line %d has invalid details; ignored", entry.Line)) + return + } + switch details.Mode { + case "single", "parallel", "chain": + default: + return + } + for index, rawResult := range details.Results { + var result struct { + Agent string `json:"agent"` + Task string `json:"task"` + Messages json.RawMessage `json:"messages"` + Usage json.RawMessage `json:"usage"` + } + // Decode siblings separately: malformed optional extension data must + // not discard the root or a valid neighboring child. + if json.Unmarshal(rawResult, &result) != nil || result.Agent == "" { + a.warnings = append(a.warnings, fmt.Sprintf("Pi subagent result on line %d has an invalid child at index %d; ignored", entry.Line, index+1)) + continue + } + id := fmt.Sprintf("subagent-%s-%d", entry.ID, index+1) + messages := a.subagentMessages(entry, id, result.Messages) + if len(messages) > 0 { + a.childStreams[id] = messages + } else { + a.warnings = append(a.warnings, fmt.Sprintf("Pi subagent %s has no conversation", id)) + } + + var usage map[string]json.RawMessage + if len(result.Usage) > 0 && !isNull(result.Usage) && (json.Unmarshal(result.Usage, &usage) != nil || usage == nil) { + a.warnings = append(a.warnings, fmt.Sprintf("Pi subagent %s has invalid usage; ignored", id)) + } + // The example already sums assistant usage, including failed turns. + // Never add the embedded messages' usage again. contextTokens is only + // the last context size, not a cumulative evidence total. + input, _ := nonnegativeInt(usage["input"]) + output, _ := nonnegativeInt(usage["output"]) + a.addInt(entry.Line, "subagent.input", usage["input"], &a.inputTokens) + a.addInt(entry.Line, "subagent.output", usage["output"], &a.outputTokens) + a.addInt(entry.Line, "subagent.cacheRead", usage["cacheRead"], &a.cacheRead) + a.addInt(entry.Line, "subagent.cacheWrite", usage["cacheWrite"], &a.cacheWrite) + a.addCost(entry.Line, usage["cost"]) + a.subagents = append(a.subagents, aicodingsession.Subagent{ + ID: id, Type: result.Agent, Description: result.Task, + Tokens: aicodingsession.SubagentTokens{Input: input, Output: output}, + }) + } +} + +// Wrap native child messages like root Pi entries without translating away +// content, tool results, or unknown fields. Source timestamps are milliseconds; +// missing timestamps fall back to the containing result's persisted time. +func (a *evidenceAccumulator) subagentMessages(entry *sessionEntry, id string, raw json.RawMessage) []json.RawMessage { + var children []json.RawMessage + if len(raw) == 0 || json.Unmarshal(raw, &children) != nil { + return nil + } + var messages []json.RawMessage + var lastID *string + for index, child := range children { + message, ok := decodeMessage(child) + if !ok { + a.warnings = append(a.warnings, fmt.Sprintf("Pi subagent %s has an invalid message at index %d; ignored", id, index+1)) + continue + } + if message.Role == "assistant" && message.Model != "" { + a.models[message.Model] = struct{}{} + } + timestamp := entry.Timestamp + var source struct { + Timestamp int64 `json:"timestamp"` + } + if json.Unmarshal(child, &source) == nil && source.Timestamp > 0 { + if at := time.UnixMilli(source.Timestamp).UTC(); at.Year() >= 1 && at.Year() <= 9999 { + timestamp = at.Format(time.RFC3339Nano) + } + } + messageID := fmt.Sprintf("%s-%d", id, index+1) + wrapped, err := json.Marshal(struct { + Type string `json:"type"` + ID string `json:"id"` + ParentID *string `json:"parentId"` + Timestamp string `json:"timestamp"` + Message json.RawMessage `json:"message"` + }{Type: "message", ID: messageID, ParentID: lastID, Timestamp: timestamp, Message: craftableRaw(child)}) + if err != nil { + a.warnings = append(a.warnings, fmt.Sprintf("Pi subagent %s has an unencodable message; ignored", id)) + continue + } + messages = append(messages, wrapped) + lastID = &messageID + } + return messages +} diff --git a/app/cli/internal/trace/pi/subagent_test.go b/app/cli/internal/trace/pi/subagent_test.go new file mode 100644 index 000000000..7fc53c56a --- /dev/null +++ b/app/cli/internal/trace/pi/subagent_test.go @@ -0,0 +1,172 @@ +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package pi + +import ( + "encoding/json" + "fmt" + "testing" + + "github.com/chainloop-dev/chainloop/pkg/attestation/crafter/materials/aicodingsession" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// Shape verified against the official example, not arbitrary Pi extensions: +// https://github.com/earendil-works/pi/blob/71023aec1546d6d677a65669c5665c0927ce0dd0/packages/coding-agent/examples/extensions/subagent/index.ts +const childResult = `{"agent":"worker","agentSource":"project","task":"Read README.md","exitCode":0,` + + `"usage":{"input":3,"output":5,"cacheRead":11,"cacheWrite":19,"cost":0.025,"contextTokens":38,"turns":2},` + + `"messages":[` + + `{"role":"assistant","model":"child-model","provider":"child-provider","timestamp":1767323042000,"content":[{"type":"toolCall","id":"child-call","name":"read","arguments":{"path":"README.md"}}],"usage":{"input":1,"output":2,"cacheRead":4,"cost":{"total":0.01}}},` + + `{"role":"toolResult","toolCallId":"child-call","toolName":"read","content":[{"type":"text","text":"full file content"}]},` + + `{"role":"assistant","content":[{"type":"text","text":"child answer"}],"usage":{"input":2,"output":3,"cacheRead":7,"cacheWrite":19,"cost":{"total":0.015}}}` + + `]}` + +func subagentSession(details string) []byte { + return []byte(`{"type":"session","version":3,"id":"pi-session-1","timestamp":"2026-01-02T03:04:00Z","cwd":"/work/repo","parentSession":"not-a-child"} +{"type":"message","id":"root0001","parentId":null,"timestamp":"2026-01-02T03:04:01Z","message":{"role":"assistant","model":"root-model","provider":"root-provider","content":[{"type":"toolCall","id":"call-1","name":"subagent","arguments":{"agent":"worker"}}],"usage":{"input":7,"output":9,"cacheRead":13,"cacheWrite":17,"cost":{"total":0.1}}}} +{"type":"message","id":"abandoned","parentId":"root0001","timestamp":"2026-01-02T03:04:02Z","message":{"role":"toolResult","toolName":"subagent","details":{"mode":"single","results":[{"agent":"abandoned","usage":{"input":999}}]}}} +{"type":"message","id":"result01","parentId":"root0001","timestamp":"2026-01-02T03:04:03Z","message":{"role":"toolResult","toolCallId":"call-1","toolName":"subagent","details":` + details + `}} +{"type":"message","id":"final001","parentId":"result01","timestamp":"2026-01-02T03:04:04Z","message":{"role":"assistant","model":"root-final","provider":"root-provider","content":[{"type":"text","text":"done"}]}} +`) +} + +func TestPiSubagentEvidenceWhenSupportedResult(t *testing.T) { + for _, mode := range []string{"single", "parallel", "chain"} { + t.Run(mode, func(t *testing.T) { + // Given: aggregate counters duplicate usage in the child messages. + results := childResult + count := 1 + cost := 0.125 + if mode != "single" { + results += "," + childResult + count = 2 + cost = 0.15 + } + session := subagentSession(`{"mode":"` + mode + `","results":[` + results + `]}`) + + // When: parse through the provider's copied-session entrypoint. + got, err := parseBytes(t, session) + + // Then: children are distinct, and each aggregate is counted once. + require.NoError(t, err) + require.Len(t, got.Data.Subagents, count) + assert.Equal(t, &aicodingsession.Usage{ + InputTokens: 7 + 3*count, OutputTokens: 9 + 5*count, TotalTokens: 16 + 8*count, + CacheReadInputTokens: 13 + 11*count, CacheCreationInputTokens: 17 + 19*count, + EstimatedCostUSD: cost, + }, got.Data.Usage) + assert.Equal(t, "v1", got.Data.SchemaVersion) + assert.Equal(t, "root-final", got.Data.Model.Primary) + assert.Equal(t, []string{"child-model", "root-final", "root-model"}, got.Data.Model.ModelsUsed) + assert.Equal(t, 2, got.Data.Conversation.TotalMessages, "root conversation summary remains root-only") + assert.Equal(t, []string{"root0001", "result01", "final001"}, rawEntryIDs(t, got.Data.RawSession["main"])) + for i, sub := range got.Data.Subagents { + assert.Equal(t, aicodingsession.Subagent{ + ID: fmt.Sprintf("subagent-result01-%d", i+1), Type: "worker", Description: "Read README.md", + Tokens: aicodingsession.SubagentTokens{Input: 3, Output: 5}, + }, sub) + stream := got.Data.RawSession[sub.ID] + require.Len(t, stream, 3) + var wrapped struct { + Type string `json:"type"` + ID string `json:"id"` + ParentID string `json:"parentId"` + Timestamp string `json:"timestamp"` + Message json.RawMessage `json:"message"` + } + require.NoError(t, json.Unmarshal(stream[1], &wrapped)) + assert.Equal(t, "message", wrapped.Type) + assert.Equal(t, sub.ID+"-2", wrapped.ID) + assert.Equal(t, sub.ID+"-1", wrapped.ParentID) + assert.Equal(t, "2026-01-02T03:04:03Z", wrapped.Timestamp) + assert.JSONEq(t, `{"role":"toolResult","toolCallId":"child-call","toolName":"read","content":[{"type":"text","text":"full file content"}]}`, string(wrapped.Message)) + assert.Contains(t, string(stream[2]), "child answer") + } + assert.NoError(t, validateEvidenceData(got.Data)) + again, err := parseBytes(t, session) + require.NoError(t, err) + assert.Equal(t, got, again) + }) + } +} + +func TestPiSubagentEvidenceWhenOptionalDetailsMalformed(t *testing.T) { + for _, tc := range []struct { + name string + details string + children int + input int + }{ + {name: "cancellation", details: `{}`}, + {name: "wrong type", details: `"extension-specific"`}, + {name: "unknown mode", details: `{"mode":"other","results":[` + childResult + `]}`}, + {name: "malformed child", details: `{"mode":"single","results":[{"agent":123}]}`}, + {name: "malformed sibling", details: `{"mode":"parallel","results":[{"agent":123},` + childResult + `]}`, children: 1, input: 3}, + {name: "failed child", details: `{"mode":"chain","results":[{"agent":"worker","exitCode":1,"messages":[],"usage":{"input":2,"output":3,"cacheRead":5,"cost":0.01}}]}`, children: 1, input: 2}, + {name: "malformed optional usage", details: `{"mode":"single","results":[{"agent":"worker","messages":[{"role":"assistant","content":[]}],"usage":"bad"}]}`, children: 1}, + {name: "malformed optional messages", details: `{"mode":"single","results":[{"agent":"worker","messages":"bad","usage":{"input":2}}]}`, children: 1, input: 2}, + } { + t.Run(tc.name, func(t *testing.T) { + // Given / When + got, err := parseBytes(t, subagentSession(tc.details)) + // Then: optional extension failures never discard the root evidence. + require.NoError(t, err) + assert.Len(t, got.Data.Subagents, tc.children) + if tc.name == "malformed sibling" { + require.Len(t, got.Data.Subagents, 1) + assert.Equal(t, "subagent-result01-2", got.Data.Subagents[0].ID) + } + assert.Equal(t, 7+tc.input, got.Data.Usage.InputTokens) + assert.NoError(t, validateEvidenceData(got.Data)) + }) + } +} + +func TestPiSubagentEvidenceWhenUnrelatedExtension(t *testing.T) { + // Given: matching details on an unrelated tool and a parent-session header. + session := subagentSession(`{}`) + session = append(session, []byte(`{"type":"message","id":"unrelated","parentId":"final001","timestamp":"2026-01-02T03:04:05Z","message":{"role":"toolResult","toolName":"read","details":{"mode":"single","results":[`+childResult+`]}}} +`)...) + // When + got, err := parseBytes(t, session) + // Then + require.NoError(t, err) + assert.Empty(t, got.Data.Subagents) + assert.Len(t, got.Data.RawSession, 1) + assert.Equal(t, 7, got.Data.Usage.InputTokens) +} + +func TestPiSubagentUsageWhenFailedChildHasInvalidCounters(t *testing.T) { + // Given: a failed child with a preserved response, malformed counters, + // and a contextTokens value that must never become the evidence total. + session := subagentSession(`{"mode":"chain","results":[{"agent":"worker","task":"attempt","exitCode":1,"stopReason":"error",` + + `"usage":{"input":-1,"output":3,"cacheRead":"bad","cacheWrite":5,"contextTokens":999,"cost":1e400},` + + `"messages":[{"role":"assistant","stopReason":"error","errorMessage":"provider failed","content":[{"type":"text","text":"partial answer"}],"usage":{"input":1e400}}]}]}`) + // When + got, err := parseBytes(t, session) + // Then: only valid aggregate counters contribute; evidence is craftable. + require.NoError(t, err) + require.Len(t, got.Data.Subagents, 1) + assert.Equal(t, aicodingsession.SubagentTokens{Output: 3}, got.Data.Subagents[0].Tokens) + assert.Equal(t, &aicodingsession.Usage{ + InputTokens: 7, OutputTokens: 12, TotalTokens: 19, + CacheReadInputTokens: 13, CacheCreationInputTokens: 22, EstimatedCostUSD: 0.1, + }, got.Data.Usage) + require.Len(t, got.Data.RawSession[got.Data.Subagents[0].ID], 1) + assert.Contains(t, string(got.Data.RawSession[got.Data.Subagents[0].ID][0]), "partial answer") + assert.NotEmpty(t, got.Data.Warnings) + assert.NoError(t, validateEvidenceData(got.Data)) +} diff --git a/app/cli/pkg/action/trace_pi_subagent_test.go b/app/cli/pkg/action/trace_pi_subagent_test.go new file mode 100644 index 000000000..8d847fabd --- /dev/null +++ b/app/cli/pkg/action/trace_pi_subagent_test.go @@ -0,0 +1,75 @@ +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package action + +import ( + "encoding/json" + "os" + "path/filepath" + "testing" + + "github.com/chainloop-dev/chainloop/app/cli/internal/trace" + "github.com/chainloop-dev/chainloop/app/cli/internal/trace/pi" + "github.com/chainloop-dev/chainloop/app/cli/internal/trace/state" + "github.com/chainloop-dev/chainloop/pkg/attestation/crafter/materials/aicodingsession" + "github.com/rs/zerolog" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestPiSubagentMaterialWhenCopiedTranscript(t *testing.T) { + // Given: an official-example result with an embedded read exchange. + const sessionID = "3713ab-session" + const transcript = `{"type":"session","version":3,"id":"3713ab-session","cwd":"/work/repo","timestamp":"2026-10-30T10:00:00Z"} +{"type":"message","id":"result01","parentId":null,"timestamp":"2026-10-30T10:00:01Z","message":{"role":"toolResult","toolName":"subagent","details":{"mode":"single","results":[{"agent":"worker","task":"read file","exitCode":0,"usage":{"input":3,"output":5,"cacheRead":7,"cacheWrite":11,"cost":0.02},"messages":[{"role":"assistant","content":[{"type":"toolCall","id":"read-1","name":"read","arguments":{"path":"README.md"}}]},{"role":"toolResult","toolCallId":"read-1","toolName":"read","content":[{"type":"text","text":"file content retained"}]}]}]}}} +` + path := filepath.Join(t.TempDir(), "session.jsonl") + require.NoError(t, os.WriteFile(path, []byte(transcript), 0o600)) + provider := pi.New() + store := state.NewOutOfTreeStore(t.TempDir()) + require.NoError(t, provider.CopySessionData(store, trace.SessionLocation{SessionID: sessionID, TranscriptPath: path})) + parsed, err := provider.ParseSession(t.Context(), &trace.ParseOpts{SessionID: sessionID, SessionDir: store.RawSessionDir()}) + require.NoError(t, err) + adder := &fakeMaterialAdder{} + + // When: assemble the same session material that push attaches. + attested, _ := attachSessionEvidence(t.Context(), adder, store, []sessionEvidence{{ + sessionID: sessionID, provider: provider, evidence: parsed, + }}, zerolog.Nop(), false) + + // Then: the encoded material contains child messages, not just metadata. + require.Equal(t, []string{sessionID}, attested) + var got aicodingsession.Evidence + require.NoError(t, json.Unmarshal([]byte(adder.byName("ai-coding-session-3713ab").content), &got)) + require.Len(t, got.Data.Subagents, 1) + assert.Equal(t, parsed.Data.Subagents, got.Data.Subagents) + assert.Equal(t, parsed.Data.Usage, got.Data.Usage) + stream := got.Data.RawSession[got.Data.Subagents[0].ID] + require.Len(t, stream, 2) + var result struct { + Message struct { + Role string `json:"role"` + ToolCallID string `json:"toolCallId"` + Content []struct { + Text string `json:"text"` + } `json:"content"` + } `json:"message"` + } + require.NoError(t, json.Unmarshal(stream[1], &result)) + assert.Equal(t, "toolResult", result.Message.Role) + assert.Equal(t, "read-1", result.Message.ToolCallID) + require.Len(t, result.Message.Content, 1) + assert.Equal(t, "file content retained", result.Message.Content[0].Text) +} From afad7548ff9e35733faf624bb29af818694d3785 Mon Sep 17 00:00:00 2001 From: Vibhav Bobade Date: Sat, 31 Oct 2026 02:40:17 +0530 Subject: [PATCH 2/2] docs(trace): record live Pi subagent validation Assisted-by: Vibhav's Pi Signed-off-by: Vibhav Bobade --- app/cli/internal/trace/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app/cli/internal/trace/README.md b/app/cli/internal/trace/README.md index 878dd22f4..f0d86c8d0 100644 --- a/app/cli/internal/trace/README.md +++ b/app/cli/internal/trace/README.md @@ -255,7 +255,7 @@ CHAINLOOP_E2E_FEATURE=Subagents \ make -C app/cli test-trace-e2e ``` -This scenario loads the real official example from the installed package and checks the SHA-256 of both `index.ts` and `agents.ts` against the pinned commit before running. It uses an isolated read-only worker, then lets the parent write, commit, and push after the child returns. Its check compares stored child conversations and tool-result content with the native completed result and verifies parent plus child usage, cache, and cost exactly once. This is a live test command, not a recorded pass. +This scenario loads the real official example from the installed package and checks the SHA-256 of both `index.ts` and `agents.ts` against the pinned commit before running. It uses an isolated read-only worker, then lets the parent write, commit, and push after the child returns. Its check compares stored child conversations and tool-result content with the native completed result and verifies parent plus child usage, cache, and cost exactly once. A manual Pi 1.3.0 check with `openai-codex/gpt-6.1-sol` completed one real worker using the hash-verified official example. The pinned OpenRouter matrix remains blocked by an expired API key; the manual check is not a pass for that matrix. To retain the workspace and agent output for a failure: