diff --git a/app/cli/internal/trace/README.md b/app/cli/internal/trace/README.md index f1f648639..53c10bd0f 100644 --- a/app/cli/internal/trace/README.md +++ b/app/cli/internal/trace/README.md @@ -14,7 +14,7 @@ Pi supports the JSONL v3 format used by Pi 0.80.10. The compatibility suite pins Pi evidence preserves native `input`, `output`, `cacheRead`, and `cacheWrite` as separate categories. Evidence `total_tokens` is recomputed as base input + output, excluding both cache categories, regardless of supplied `totalTokens`. The 100/20/10/5 fixture therefore records total 120, not 135. Supplied `cost.total` remains authoritative, without repricing. Invalid or overflowing counters are ignored with deterministic warnings; if input + output overflows, the evidence total is omitted with a warning while individual counters remain. Missing usage contributes zero. Only the selected branch is counted, including its history after resume, not abandoned branches or complete session spend. The Chainloop evidence field `schema_version` remains `v1`; it is not a Pi release or JSONL format version. Earlier Chainloop parsers used supplied totals, which may include cache, under the same evidence version; normalize historical evidence from raw records before combining totals. -Pi 1.3.0 passed 26 feature checks in the local end-to-end suite, including a cache-active accounting run. Pi 0.80.10 also completed targeted TUI and JSON-headless smokes that exercised session start, `write`, `edit`, `bash`, commit attribution and session shutdown; the TUI showed the local-mode welcome message. Subagents, skills tracking, spec source pointers and pasted image pointers are not implemented. The headless suite cannot verify the UI-only welcome message, the control-plane credentials warning, the session link after a control-plane push or control-plane attestation. The multi-agent feature has no registered check. +Pi 1.3.0 passed 26 feature checks in the local end-to-end suite, including a cache-active accounting run. Pi 0.80.10 also completed targeted TUI and JSON-headless smokes that exercised session start, `write`, `edit`, `bash`, commit attribution and session shutdown; the TUI showed the local-mode welcome message. Subagents, spec source pointers and pasted image pointers are not implemented. Skills tracking is partial as described below. The headless suite cannot verify the UI-only welcome message, the control-plane credentials warning, the session link after a control-plane push or control-plane attestation. The multi-agent feature has no registered check. Codex uses native hooks in `.codex/hooks.json` (Codex CLI 0.163.0). Run `chainloop trace init --codex` or `chainloop trace run --local --codex -- codex exec …`. Review and trust the installed hooks in Codex before starting a session. See the [official hooks documentation](https://learn.chatgpt.com/docs/hooks). Installed hooks have a `Chainloop Trace` status label in Codex. Re-running init adds labels to existing Chainloop hooks. Changing the hooks requires a new trust review. @@ -129,13 +129,12 @@ What the session was asked to build, and which skills it used. | Local spec sources read at push | A capture whose source is a local file holds only a header and a placeholder. Each push reads the file and records its current content, so the evidence follows the edits of the file. Only a regular text file of up to 1 MiB is read. Otherwise the push keeps the body that the agent wrote, or drops the capture with a warning when there is no body (spec issue-3561). | Yes | Yes | Yes | Yes | Yes | Yes | | Spec source role, title, and description | Each captured spec source holds the role, the title, and the description that the agent wrote in its header (spec issue-3531). | Yes | Yes | Unknown | Yes | Yes | Yes | | HTML spec sources | A captured text spec source whose body is an HTML page (it starts with a doctype or an `html` element) is stored as an `.html` file that holds only the page, without the header. The annotations and the spec entry keep the values of the header. A body that only contains HTML tags stays as the agent wrote it (issue 3707). | Yes | Yes | Unknown | Yes | Yes | Unknown | -| Skills tracking | The evidence lists the skills that the session used, with a copy of each skill as it ran. | Yes | No | Partial | Partial | No | Partial | +| Skills tracking | The evidence lists the skills that the session used, with a copy of each skill as it ran. | Yes | No | Partial | Partial | Partial | Partial | | Spec source pointers | Exact copies of spec sources in the transcript (full file reads and file writes) are replaced with a pointer to the spec material, so the evidence does not hold the same content two or three times (spec issue-3556). This includes the copies of pasted images and spec sources in the `api-request-blob` records of a Claude Code transcript (issue 3654). | Yes | No | No | No | No | Partial | | Pasted image pointers | Each distinct image that the user pasted into the session, including images in prompts sent while the agent was busy, is stored once as an image spec entry, from the data in the transcript, with the title "Pasted image N" and no role. A repeated paste of the same image adds no entry. The inline image in the transcript is replaced with a pointer to that material. An image whose data cannot be decoded, or that the push could not store, stays inline, and the push continues. When the agent captured the same image, its entry is kept and no second entry is added (spec issue-3569). | Yes | No | No | No | No | Yes | Why not Yes: -- **Skills tracking, Pi:** the provider does not track Pi skills, so the evidence has no skill entries. - **Spec source pointers and pasted image pointers, Pi:** the provider has no finder for Pi transcript copies or pasted-image records. - **Spec capture reminder, Codex:** Codex does not report hook messages in its headless output, so the suite cannot inspect the reminder. - **Spec capture reminder, Cursor:** Cursor has no hook at each user prompt. The agent gets the session-start instruction only. @@ -144,6 +143,7 @@ Why not Yes: - **Skills tracking, Codex:** tracks filesystem skills loaded through explicit Codex skill instructions or successful shell reads of one identified `SKILL.md`. The definition and supporting files use the shared skill snapshot and packaging pipeline. Multiple distinct skill paths in one tool call cannot be attributed reliably and are not tracked. Skills without a filesystem folder and child sessions are not captured. A snapshot first made at push carries a warning. - **Spec source pointers, Codex:** supports exact text results, native shell envelopes, recognized code-mode JSON envelopes, and complete file creation through native patches or literal patches in code mode. A code-mode write must be a single awaited `tools.apply_patch` call, optionally wrapped in `text`, with a literal argument or an immediately preceding `const` literal. Stored source files read with `cat` can also match within combined command output and native file-change records. Mixed or computed patches, other scripts, arbitrary output wrappers, transformed content, and other unsupported formats remain inline, so the table remains Partial. Earlier live sessions demonstrated this limit when the model trimmed the source's trailing newline. - **Pasted image pointers, Codex:** extracts base64 image data URLs from user input, deduplicates their decoded bytes, and replaces stored copies in both viewer messages and retained events. Tool-result images are not pastes. Invalid data, remote URLs, and path-only image records remain inline. +- **Skills tracking, Pi:** completed `read` calls for filesystem `SKILL.md` files and native `/skill:` expansions are tracked on the selected root branch. Skill folders are snapshotted by the first following Trace hook; push-time fallback keeps the shared warning. Child streams and arbitrary shell reads are not tracked. Pi has no managed organization-root metadata, so unclassified external locations stay `unknown`. A manual Pi 1.3.0 check with `openai-codex/gpt-6.1-sol` loaded and reloaded a project skill and retained the original definition and supporting file after the source changed. The automated suite's pinned OpenRouter run remains blocked by an expired API key. - **Skills tracking, Cursor:** the provider does not track skills. The evidence has no skill entries, which means "not recorded", not "no skill used". - **Skills tracking, OpenCode:** only the skills that the model starts with the `skill` tool are counted. Skills that the user starts, and skills used in subagents, are not. A plugin installed by an earlier version does not send the skill folder for OpenCode 2: install the hooks again to update it. - **Spec source pointers, Cursor and OpenCode:** the provider has no finder for the copies in its transcript yet. For OpenCode, the rebuilt transcript holds no tool outputs, so only the write copies would apply. diff --git a/app/cli/internal/trace/pi/skills.go b/app/cli/internal/trace/pi/skills.go new file mode 100644 index 000000000..ed26737d2 --- /dev/null +++ b/app/cli/internal/trace/pi/skills.go @@ -0,0 +1,244 @@ +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package pi + +import ( + "encoding/json" + "errors" + "os" + "path/filepath" + "regexp" + "strings" + + "github.com/chainloop-dev/chainloop/app/cli/internal/trace" + "github.com/chainloop-dev/chainloop/app/cli/internal/trace/skill" + "github.com/chainloop-dev/chainloop/app/cli/internal/trace/state" + "gopkg.in/yaml.v3" +) + +var _ trace.SkillTracker = (*Provider)(nil) + +var userSkillBlock = regexp.MustCompile(`(?s)^\n.*\n(?:\n\n.*)?$`) + +// NewSkillLoads scans the selected live branch for completed loads. The cursor +// advances over entries, not calls, so a result in a later hook can still pair +// with the original call. Generic hooks copy the skill before the next mutation. +func (p *Provider) NewSkillLoads(store *state.Store, input *trace.HookInput) []string { + if store == nil || input == nil || !state.ValidSessionID(input.SessionID) || !filepath.IsAbs(input.TranscriptPath) { + return nil + } + info, err := os.Stat(input.TranscriptPath) + if err != nil || !info.Mode().IsRegular() { + return nil + } + parsed, err := readSkillSession(input.TranscriptPath, input.SessionID) + if err != nil { + return nil + } + offsets, err := store.SkillScanOffsets(input.SessionID) + if err != nil { + offsets = make(map[string]int64) + } + var last int64 + for _, entry := range parsed.Selected { + last = max(last, int64(entry.Line)) + } + after := offsets["main"] + if after > last { + after = 0 + } + uses := selectedSkills(parsed, after) + dirs := make([]string, 0, len(uses)) + for _, use := range uses { + dirs = append(dirs, use.Dir) + } + offsets["main"] = last + if err := store.SetSkillScanOffsets(input.SessionID, offsets); err != nil { + return nil + } + return dirs +} + +// SessionSkills reads only Trace's private transcript copy. +func (p *Provider) SessionSkills(opts *trace.ParseOpts) ([]trace.SkillUse, []string, error) { + if opts == nil || !state.ValidSessionID(opts.SessionID) { + return nil, nil, errors.New("read Pi skills: invalid session") + } + path, err := state.FindRawSessionFile(opts.SessionDir, opts.SessionID) + if err != nil { + return nil, nil, err + } + parsed, err := readSkillSession(path, opts.SessionID) + if err != nil { + return nil, nil, err + } + return selectedSkills(parsed, 0), nil, nil +} + +func readSkillSession(path, sessionID string) (*parsedSession, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, errors.New("read Pi skills: transcript unavailable") + } + return parseJSONL(data, sessionID) +} + +type skillReadCall struct { + Path string `json:"path"` + Offset int `json:"offset"` +} + +func selectedSkills(parsed *parsedSession, after int64) []trace.SkillUse { + uses := make(map[string]*trace.SkillUse) + calls := make(map[string]skillReadCall) + for _, entry := range parsed.Selected { + if entry.Type != "message" { + continue + } + var message struct { + Role string `json:"role"` + Content json.RawMessage `json:"content"` + ToolCallID string `json:"toolCallId"` + ToolName string `json:"toolName"` + IsError bool `json:"isError"` + } + if json.Unmarshal(entry.Message, &message) != nil { + continue + } + record := func(name, path string, byModel bool) { + if int64(entry.Line) <= after || name == "" { + return + } + path = skillFilePath(path, parsed.Header.Cwd) + if path == "" || filepath.Base(path) != skill.DefinitionFile { + return + } + dir := filepath.Dir(path) + if uses[dir] == nil { + uses[dir] = &trace.SkillUse{Name: name, Dir: dir} + } + uses[dir].Record(entry.Timestamp, byModel, false) + } + switch message.Role { + case "user": + if block := userSkillBlock.FindStringSubmatch(skillMessageText(message.Content)); block != nil { + record(block[1], block[2], false) + } + case "assistant": + var blocks []struct { + Type string `json:"type"` + ID string `json:"id"` + Name string `json:"name"` + Arguments json.RawMessage `json:"arguments"` + } + if json.Unmarshal(message.Content, &blocks) != nil { + continue + } + for _, block := range blocks { + if block.Type == "toolCall" && block.ID != "" && block.Name == "read" { + var call skillReadCall + if json.Unmarshal(block.Arguments, &call) == nil { + calls[block.ID] = call + } + } + } + case "toolResult": + call, ok := calls[message.ToolCallID] + delete(calls, message.ToolCallID) + if !ok || message.IsError || message.ToolName != "read" || call.Offset > 1 || call.Offset < 0 { + continue + } + text := strings.ReplaceAll(skillMessageText(message.Content), "\r\n", "\n") + front, ok := strings.CutPrefix(text, "---\n") + if !ok { + continue + } + front, _, ok = strings.Cut(front, "\n---") + var meta struct { + Name string `yaml:"name"` + Description string `yaml:"description"` + } + if !ok || yaml.Unmarshal([]byte(front), &meta) != nil || strings.TrimSpace(meta.Description) == "" { + continue + } + if meta.Name == "" { + meta.Name = filepath.Base(filepath.Dir(call.Path)) + } + record(meta.Name, call.Path, true) + } + } + return trace.SortedSkillUses(uses) +} + +func skillMessageText(content json.RawMessage) string { + var text string + if json.Unmarshal(content, &text) == nil { + return text + } + var blocks []struct { + Type string `json:"type"` + Text string `json:"text"` + } + if json.Unmarshal(content, &blocks) != nil { + return "" + } + var texts []string + for _, block := range blocks { + if block.Type == "text" { + texts = append(texts, block.Text) + } + } + return strings.Join(texts, "\n") +} + +func skillFilePath(path, cwd string) string { + path = strings.TrimPrefix(path, "@") + if strings.HasPrefix(path, "~/") { + home, err := os.UserHomeDir() + if err != nil { + return "" + } + path = filepath.Join(home, path[2:]) + } + if !filepath.IsAbs(path) { + if !filepath.IsAbs(cwd) { + return "" + } + path = filepath.Join(cwd, path) + } + return filepath.Clean(path) +} + +// SkillRoots follows Pi's user/project and managed package locations. Pi does +// not identify an organization scope; arbitrary external paths stay unknown. +func (p *Provider) SkillRoots(repoRoot string) skill.Roots { + home, err := os.UserHomeDir() + if err != nil { + return skill.Roots{Project: []string{repoRoot}} + } + agentDir := os.Getenv("PI_CODING_AGENT_DIR") + if agentDir == "" { + agentDir = filepath.Join(home, ".pi", "agent") + } + agentDir = skillFilePath(agentDir, repoRoot) + return skill.Roots{ + Plugin: []string{ + filepath.Join(agentDir, "git"), filepath.Join(agentDir, "npm", "node_modules"), + filepath.Join(repoRoot, ".pi", "git"), filepath.Join(repoRoot, ".pi", "npm", "node_modules"), + }, + User: []string{filepath.Join(agentDir, "skills"), filepath.Join(home, ".agents", "skills")}, + Project: []string{repoRoot}, + } +} diff --git a/app/cli/internal/trace/pi/skills_test.go b/app/cli/internal/trace/pi/skills_test.go new file mode 100644 index 000000000..01bb1a41d --- /dev/null +++ b/app/cli/internal/trace/pi/skills_test.go @@ -0,0 +1,214 @@ +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package pi + +import ( + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/chainloop-dev/chainloop/app/cli/internal/trace" + "github.com/chainloop-dev/chainloop/app/cli/internal/trace/skill" + "github.com/chainloop-dev/chainloop/app/cli/internal/trace/state" + "github.com/stretchr/testify/require" +) + +const ( + skillTestDefinition = "---\nname: review\ndescription: Review the implementation.\n---\nRead the surrounding code.\n" + skillSnapshotTool = "write" +) + +func piSkillMessages(cwd string, messages []string) []byte { + lines := make([]string, 0, 1+len(messages)) + lines = append(lines, fmt.Sprintf(`{"type":"session","version":3,"id":%q,"cwd":%q,"timestamp":"2026-10-30T10:00:00Z"}`, testSessionID, cwd)) + for i, message := range messages { + parent := "null" + if i > 0 { + parent = fmt.Sprintf("%q", fmt.Sprintf("entry-%d", i-1)) + } + lines = append(lines, fmt.Sprintf(`{"type":"message","id":"entry-%d","parentId":%s,"timestamp":"2026-10-30T10:00:%02dZ","message":%s}`, + i, parent, i+1, message)) + } + return []byte(strings.Join(lines, "\n") + "\n") +} + +func TestPiSessionSkills(t *testing.T) { + // Given + tracker, ok := any(New()).(trace.SkillTracker) + require.True(t, ok, "Pi must implement the shared skill tracker") + read := `{"role":"assistant","content":[{"type":"toolCall","id":"load-1","name":"read","arguments":{"path":"skills/review/SKILL.md"}}]}` + result := fmt.Sprintf(`{"role":"toolResult","toolCallId":"load-1","toolName":"read","isError":false,"content":[{"type":"text","text":%q}]}`, skillTestDefinition) + cases := []struct { + name string + messages []string + model int + user int + }{ + {name: "no loads", messages: []string{`{"role":"user","content":"List available skills."}`}}, + {name: "available catalog is not a use", messages: []string{`{"role":"user","content":"review/skills/review/SKILL.md"}`}}, + {name: "completed read", messages: []string{read, result}, model: 1}, + { + name: "non-read arguments do not discard completed reads", + messages: []string{ + strings.Replace(read, `"content":[`, `"content":[{"type":"toolCall","id":"other","name":"bash","arguments":{"path":42}},`, 1), + result, + }, + model: 1, + }, + { + name: "malformed read arguments do not discard sibling reads", + messages: []string{ + strings.Replace(read, `"content":[`, `"content":[{"type":"toolCall","id":"invalid","name":"read","arguments":{"path":42}},`, 1), + strings.ReplaceAll(result, "load-1", "invalid"), + result, + }, + model: 1, + }, + { + name: "fractional read offset does not discard sibling reads", + messages: []string{ + strings.Replace(read, `"content":[`, `"content":[{"type":"toolCall","id":"invalid","name":"read","arguments":{"path":"skills/review/SKILL.md","offset":1.5}},`, 1), + strings.ReplaceAll(result, "load-1", "invalid"), + result, + }, + model: 1, + }, + {name: "CRLF definition", messages: []string{read, strings.Replace(result, fmt.Sprintf("%q", skillTestDefinition), fmt.Sprintf("%q", strings.ReplaceAll(skillTestDefinition, "\n", "\r\n")), 1)}, model: 1}, + {name: "unfinished read", messages: []string{read}}, + {name: "failed read", messages: []string{read, strings.Replace(result, `"isError":false`, `"isError":true`, 1)}}, + {name: "partial read", messages: []string{strings.Replace(read, `"path":`, `"offset":2,"path":`, 1), result}}, + {name: "failed assistant response", messages: []string{strings.Replace(read, `"role":"assistant"`, `"role":"assistant","stopReason":"error"`, 1)}}, + {name: "result without matching call", messages: []string{result}}, + {name: "non-skill read", messages: []string{strings.Replace(read, "SKILL.md", "README.md", 1), result}}, + {name: "result without a definition", messages: []string{read, strings.Replace(result, fmt.Sprintf("%q", skillTestDefinition), `"unrelated output"`, 1)}}, + {name: "duplicate result counts once", messages: []string{read, result, result}, model: 1}, + {name: "repeated completed loads", messages: []string{read, result, strings.ReplaceAll(read, "load-1", "load-2"), strings.ReplaceAll(result, "load-1", "load-2")}, model: 2}, + {name: "user skill expansion", messages: []string{`{"role":"user","content":"\nReferences are relative to skills/review.\n\nReview the code.\n"}`}, user: 1}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + root := t.TempDir() + rawDir := t.TempDir() + writeRawSession(t, rawDir, piSkillMessages(root, tc.messages)) + + // When + uses, warnings, err := tracker.SessionSkills(&trace.ParseOpts{SessionID: testSessionID, SessionDir: rawDir}) + + // Then + require.NoError(t, err) + require.Empty(t, warnings) + if tc.model+tc.user == 0 { + require.Empty(t, uses) + return + } + require.Len(t, uses, 1) + require.Equal(t, "review", uses[0].Name) + require.Equal(t, filepath.Join(root, "skills", "review"), uses[0].Dir) + require.Equal(t, tc.model, uses[0].ByModel) + require.Equal(t, tc.user, uses[0].ByUser) + require.Zero(t, uses[0].InSubagents) + }) + } +} + +func TestPiSkillRoots(t *testing.T) { + // Given + home := t.TempDir() + repo := t.TempDir() + t.Setenv("HOME", home) + t.Setenv("USERPROFILE", home) + t.Setenv("PI_CODING_AGENT_DIR", filepath.Join(home, "custom-pi")) + roots := New().SkillRoots(repo) + cases := []struct { + path string + source string + }{ + {filepath.Join(repo, ".agents", "skills", "review"), "project"}, + {filepath.Join(repo, ".pi", "skills", "review"), "project"}, + {filepath.Join(home, "custom-pi", "skills", "review"), "user"}, + {filepath.Join(home, ".agents", "skills", "review"), "user"}, + {filepath.Join(home, "custom-pi", "npm", "node_modules", "example", "skills", "review"), "plugin"}, + {filepath.Join(repo, ".pi", "npm", "node_modules", "example", "skills", "review"), "plugin"}, + {filepath.Join(t.TempDir(), "organization", "review"), "unknown"}, + } + for _, tc := range cases { + // When + source := skill.Source("review", tc.path, roots) + + // Then + require.Equal(t, tc.source, source) + } +} + +func TestPiSessionSkillsSelectedBranchAndFirstUseOrder(t *testing.T) { + // Given + tracker, ok := any(New()).(trace.SkillTracker) + require.True(t, ok) + root := t.TempDir() + rawDir := t.TempDir() + messages := []string{ + `{"role":"user","content":"\nReview.\n"}`, + `{"role":"user","content":[{"type":"text","text":"\nBuild.\n"}]}`, + `{"role":"user","content":"\nReview again.\n"}`, + } + transcript := piSkillMessages(root, messages) + writeRawSession(t, rawDir, transcript) + + // When + uses, _, err := tracker.SessionSkills(&trace.ParseOpts{SessionID: testSessionID, SessionDir: rawDir}) + + // Then + require.NoError(t, err) + require.Len(t, uses, 2) + require.Equal(t, "review", uses[0].Name) + require.Equal(t, 2, uses[0].ByUser) + require.Equal(t, "2026-10-30T10:00:01Z", uses[0].FirstUsedAt) + require.Equal(t, "build", uses[1].Name) + + t.Run("abandoned branch is excluded", func(t *testing.T) { + alternate := `{"type":"message","id":"alternate","parentId":null,"timestamp":"2026-10-30T10:01:00Z","message":{"role":"user","content":"No skills on this branch."}}` + "\n" + writeRawSession(t, rawDir, append(transcript, []byte(alternate)...)) + selected, _, err := tracker.SessionSkills(&trace.ParseOpts{SessionID: testSessionID, SessionDir: rawDir}) + require.NoError(t, err) + require.Empty(t, selected) + }) +} + +func TestPiNewSkillLoadsWaitsForCompletedResult(t *testing.T) { + // Given + tracker, ok := any(New()).(trace.SkillTracker) + require.True(t, ok) + root := t.TempDir() + store := state.NewGitStore(filepath.Join(root, ".git")) + require.NoError(t, store.InitTraceDir()) + transcript := filepath.Join(root, "live.jsonl") + input := &trace.HookInput{SessionID: testSessionID, TranscriptPath: transcript, ToolName: skillSnapshotTool} + call := `{"role":"assistant","content":[{"type":"toolCall","id":"load-1","name":"read","arguments":{"path":"skills/review/SKILL.md"}}]}` + result := fmt.Sprintf(`{"role":"toolResult","toolCallId":"load-1","toolName":"read","content":[{"type":"text","text":%q}]}`, skillTestDefinition) + require.NoError(t, os.WriteFile(transcript, piSkillMessages(root, []string{call}), 0o600)) + require.Empty(t, tracker.NewSkillLoads(store, input)) + + // When + require.NoError(t, os.WriteFile(transcript, piSkillMessages(root, []string{call, result}), 0o600)) + loads := tracker.NewSkillLoads(store, input) + + // Then + require.Equal(t, []string{filepath.Join(root, "skills", "review")}, loads) + require.Empty(t, tracker.NewSkillLoads(store, input), "an unchanged transcript is not a new use") + input.SessionID = "another-session" + require.Empty(t, tracker.NewSkillLoads(store, input), "a different session cannot supply skill snapshots") +} diff --git a/app/cli/pkg/action/trace_pi_skills_test.go b/app/cli/pkg/action/trace_pi_skills_test.go new file mode 100644 index 000000000..6658934f8 --- /dev/null +++ b/app/cli/pkg/action/trace_pi_skills_test.go @@ -0,0 +1,117 @@ +// Copyright 2026 The Chainloop Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package action + +import ( + "fmt" + "os" + "path/filepath" + "testing" + + "github.com/chainloop-dev/chainloop/app/cli/internal/trace" + "github.com/chainloop-dev/chainloop/app/cli/internal/trace/pi" + "github.com/chainloop-dev/chainloop/app/cli/internal/trace/skill" + "github.com/chainloop-dev/chainloop/app/cli/internal/trace/state" + "github.com/rs/zerolog" + "github.com/stretchr/testify/require" +) + +func TestPiSkillSnapshotAtUseAndPushFallback(t *testing.T) { + for _, tc := range []struct { + name string + atUse bool + reload bool + }{ + {name: "first use", atUse: true}, + {name: "changed later use", atUse: true, reload: true}, + {name: "push fallback"}, + } { + t.Run(tc.name, func(t *testing.T) { + // Given + repo, gitDir := initGitRepo(t) + store := state.NewGitStore(gitDir) + require.NoError(t, store.InitTraceDir()) + const sessionID = "pi-skill-snapshot" + provider := pi.New() + skillDir := filepath.Join(repo, ".agents", "skills", "review") + require.NoError(t, os.MkdirAll(skillDir, 0o700)) + original := "---\nname: review\ndescription: Review code.\n---\nOriginal instructions.\n" + modified := "---\nname: review\ndescription: Review code.\n---\nChanged after the skill was loaded.\n" + definition := filepath.Join(skillDir, skill.DefinitionFile) + require.NoError(t, os.WriteFile(definition, []byte(original), 0o600)) + script := filepath.Join(skillDir, "check.sh") + require.NoError(t, os.WriteFile(script, []byte("original script\n"), 0o600)) + transcript := filepath.Join(t.TempDir(), "native.jsonl") + body := fmt.Sprintf(`{"type":"session","version":3,"id":%q,"cwd":%q,"timestamp":"2026-10-30T10:00:00Z"} +{"type":"message","id":"call","parentId":null,"timestamp":"2026-10-30T10:00:00Z","message":{"role":"assistant","content":[{"type":"toolCall","id":"load","name":"read","arguments":{"path":%q}}]}} +{"type":"message","id":"result","parentId":"call","timestamp":"2026-10-30T10:00:01Z","message":{"role":"toolResult","toolName":"read","toolCallId":"load","content":[{"type":"text","text":%q}]}} +`, sessionID, repo, definition, original) + require.NoError(t, os.WriteFile(transcript, []byte(body), 0o600)) + + // When + capture := func() { + captureSkillLoads(provider, store, &trace.HookInput{ + SessionID: sessionID, TranscriptPath: transcript, + ToolName: "write", FilePath: filepath.Join(repo, "changed.go"), + }, zerolog.Nop()) + } + if tc.atUse { + capture() + } + require.NoError(t, os.WriteFile(definition, []byte(modified), 0o600)) + require.NoError(t, os.WriteFile(script, []byte("changed script\n"), 0o600)) + if tc.reload { + reloaded := fmt.Sprintf(`{"type":"message","id":"reload","parentId":"result","timestamp":"2026-10-30T10:01:00Z","message":{"role":"assistant","content":[{"type":"toolCall","id":"load-again","name":"read","arguments":{"path":%q}}]}} +{"type":"message","id":"reloaded","parentId":"reload","timestamp":"2026-10-30T10:01:01Z","message":{"role":"toolResult","toolName":"read","toolCallId":"load-again","content":[{"type":"text","text":%q}]}} +`, definition, modified) + require.NoError(t, os.WriteFile(transcript, []byte(body+reloaded), 0o600)) + capture() + } + require.NoError(t, provider.CopySessionData(store, trace.SessionLocation{SessionID: sessionID, TranscriptPath: transcript})) + opts := &trace.ParseOpts{SessionID: sessionID, SessionDir: store.RawSessionDir()} + uses, _, err := provider.SessionSkills(opts) + require.NoError(t, err) + require.Len(t, uses, 1) + definitions, warnings := readSessionSkills(provider, store, opts, repo, zerolog.Nop()) + + // Then + require.Len(t, definitions, 1, "%v", warnings) + snapshot := definitions[0] + require.Equal(t, "project", snapshot.source) + if tc.reload { + require.Equal(t, 2, snapshot.use.ByModel) + } else { + require.Equal(t, 1, snapshot.use.ByModel) + } + copied, err := os.ReadFile(filepath.Join(snapshot.copyDir, "files", skill.DefinitionFile)) + require.NoError(t, err) + copiedScript, err := os.ReadFile(filepath.Join(snapshot.copyDir, "files", "check.sh")) + require.NoError(t, err) + if tc.atUse { + require.Equal(t, original, string(copied)) + require.Equal(t, "original script\n", string(copiedScript)) + if tc.reload { + require.Len(t, warnings, 1) + } else { + require.Empty(t, warnings) + } + } else { + require.Equal(t, modified, string(copied)) + require.Equal(t, "changed script\n", string(copiedScript)) + require.Len(t, warnings, 1) + } + }) + } +}