Skip to content

feat(trace): record Pi skill loads - #3759

Open
waveywaves wants to merge 3 commits into
chainloop-dev:mainfrom
waveywaves:feat/pi-skill-evidence
Open

waveywaves wants to merge 3 commits into
chainloop-dev:mainfrom
waveywaves:feat/pi-skill-evidence

Conversation

@waveywaves

@waveywaves waveywaves commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #3714.

  • Implement Pi's existing SkillTracker interface for completed filesystem read loads and native user /skill: expansions on the selected root branch.
  • Reuse the shared first-use folder snapshot, changed-use warning, push-time fallback, source classification, redaction, and material packaging pipeline.
  • Ignore catalogs, unfinished or failed calls, unmatched results, non-skill files, and abandoned branches. Keep the native transcript unchanged.
  • Add provider and action regressions, including repeated loads, multiple skills, preserved first-use files, and the warned push fallback.

This follows the applicable root-session requirements of the approved skill evidence spec, including R-002 through R-006 and R-008. There is no schema change.

Validation

Passed locally:

  • go test -race -shuffle=on -count=1 ./app/cli/internal/trace/pi ./app/cli/pkg/action
  • CLI build and go vet for the affected packages.
  • Repository-configured golangci-lint v2.14.0, built with Go 1.27.3: zero issues in the changed packages.
  • Production CLI fixture replay: initialize local Pi tracing, invoke the real Pi hooks, commit, and push to a local bare remote. Inspect the stored session, definition, and package materials. The definition retained its original contents after the source was modified; invocation count was 1 and source was project.

The fixture replay above is not a live Pi/model run.

Real Pi follow-up and Trace

A separate manual check ran Pi 1.3.0 with openai-codex/gpt-6.1-sol through the existing repository .pi/extensions/chainloop-trace.ts. A Trace-capable CLI built from this branch was placed first on that process's PATH; the previously selected development CLI did not support trace.

  • Real session: 01a18e83-e68b-70e0-92e2-d47ad981abbb.
  • Pi loaded the temporary project skill and its supporting file, then loaded both again after the source changed. The recorded first-use snapshot retained the original definition and supporting file.
  • The focused skill tests passed in that real session with -race -shuffle=on -count=1.
  • trace run --local --pi successfully assembled this genuine session and its skill definition/package materials. The local statement references follow-up commit 03f1bf5219b615af89345e720b65fdaced419137. Local evidence is unsigned and is not a control-plane attestation.
  • The original implementation commit predates this recording. This follow-up does not claim retrospective capture of that work.

Control-plane upload was attempted at the repository's configured chainloop/chainloop destination and returned PermissionDenied for the project. The destination is unchanged. There is no uploaded session URL yet; the native transcript, first-use snapshot, and assembled local evidence are retained pending project access. The PR remains a draft.

Remaining validation

The pinned OpenRouter compatibility-matrix run is still blocked: the repository key returned 401 API key expired. The successful manual OAuth check is not a pass for that matrix. An earlier E2E attempt was skipped by the old No matrix cell and is not counted as a pass. The row is now Partial.

CHAINLOOP_E2E_HARNESS=Pi \
CHAINLOOP_E2E_FEATURE='Skills tracking' \
make -C app/cli test-trace-e2e

Support is limited to selected-root filesystem skills and native user expansion. Child skill streams and arbitrary shell reads are not added here. Pi exposes no managed organization-root provenance, so unclassified external locations remain unknown under R-008.

Assisted-by: Vibhav's Pi

View guided diff

Track completed filesystem reads and native user skill expansions on the selected Pi branch through the shared skill snapshot and packaging pipeline. Preserve first-use copies, changed-use warnings and push-time fallback.

Implements the Pi scope of issue chainloop-dev#3714 and the skill evidence requirements of Spec 003 (chainloop-dev#3540). Live model validation remains blocked by missing OpenRouter authentication; native CLI replay and affected checks pass.

Assisted-by: Vibhav's Pi
Signed-off-by: Vibhav Bobade <vibhav.bobde@gmail.com>
@chainloop-platform

chainloop-platform Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

PR validation — ⚠️ 1 failing

Status Policy Material Messages
⚠️ Failed pr-min-approvals pr-info PR/MR #3759 has 0 approving reviews, 1 required.
✅ Passed pr-description-required pr-info -
✅ Passed pr-user-story-linked pr-info -

View attestation ↗

AI Session Checks — ⚠️ no AI session found

Missing AI Coding Sessions

This organization requires every PR to be backed by a Chainloop Trace AI coding session, and none was found for this one.

Please make sure the AI coding session evidence has been sent by the Chainloop CLI, or add the skip-ai-session label to this PR to bypass this check.

Learn more about Chainloop Trace.

Security Checks — ✅ 5 passing

✅ secret-scan

Status Policy Messages
✅ Passed secrets-detection -

✅ sast-scan

Status Policy Messages
✅ Passed owasp-top10-2025 -
✅ Passed sast -
✅ Passed cwe-top25 -
✅ Passed cwe-top26-40-cusp -
Scans not applied (3)
Scan Reason
vulnerability-scan no manifest/lockfile changed
github-actions-scan no workflow files changed
iac-scan no IaC files changed

View attestation ↗

Security context

✅ Nothing this change touches has a recorded security-fix history.

View in Chainloop ↗ · How this works ↗


Powered by Chainloop and Chainloop Trace

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread app/cli/internal/trace/README.md Resolved
Comment thread app/cli/internal/trace/pi/skills.go Resolved
Assisted-by: Vibhav's Pi
Signed-off-by: Vibhav Bobade <vibhav.bobde@gmail.com>
@waveywaves
waveywaves marked this pull request as ready for review October 30, 2026 21:42
Decode native read arguments per tool-call block so malformed sibling calls cannot discard completed skill loads. Cover malformed unrelated arguments, invalid read paths and fractional offsets.

Assisted-by: Vibhav's Pi
Signed-off-by: Vibhav Bobade <vibhav.bobde@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Trace: record Pi skill loads in session evidence

1 participant