Repository navigation
Add exact-head project verification for Claude Code and Codex [#90] - #111
Conversation
Keep the live-verification model and independent tooling outside the shipped plugin; require separate evidence for each selected skill. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Use the existing operator probe while requiring installed-tree confirmation before accepting live evidence. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Keep publisher credentials out of candidate subprocesses and verify the Codex installed tree before accepting operator-reviewed surface evidence. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Require native project-skill proof and compensate detected publication races before allowing draft readiness. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Keep independent helper checks and shared harness discovery outside plugin content, and require operator-owned isolated authentication. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Merge Protections🟢 All 2 merge protections satisfied — ready to merge. Show 2 satisfied protections🟢 Queue ready pull requestsAutomatically queue non-draft pull requests targeting main after verify, Unfret, and live-gate pass on the head.
🟢 🚦 Auto-queueWhen all merge protections are satisfied and these conditions match, this pull request will be queued automatically.
|
Compare real filesystem paths so doctor output and isolated workspace aliases agree. Regress with an actual doctor report, including wrong-workspace and blocked-report rejection. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if (login === 'LOGIN') await this.run(harness === 'claude' ? ['claude', 'auth', 'login'] : ['codex', 'login'], { cwd: workspace, env, interactive: true }); | ||
| else if (login !== 'SKIP') throw new Error('Expected LOGIN or SKIP'); | ||
| const raw = join(home, 'surface.raw'); | ||
| console.log('Exercise every requested feature from the native surface, using the maintained feature map. Exit when done.'); | ||
| await this.run(['script', '-q', raw, ...launch(harness, home, workspace)], { cwd: workspace, env, interactive: true }); |
There was a problem hiding this comment.
When the operator logs in, MacDriver.exercise starts the PR’s plugin with the same HOME and provider configuration directory. A malicious PR can change a startup hook to read the new credentials and send them elsewhere. Excluding daily credentials from the environment does not protect this login. Use disposable provider credentials that cannot be reused outside the test.
How this was verified: The login and candidate session use the same isolated home, and the candidate’s plugin runs hooks in that session.
There was a problem hiding this comment.
Disposition for commit 980a303: the interactive login flow has been removed. Per the operator's explicit architecture choice, the trusted parent copies only two explicitly selected caam file-backed credentials into disposable run-owned configuration, records the account identities, and removes copied credentials before publication and on failures/interruption. Candidate sessions necessarily can read their own selected session credentials; this is the explicitly accepted residual exposure, not a claim of credential invisibility. Daily/vault/publisher credentials and Keychain access are outside the Seatbelt boundary. Linux regressions pass; real Mac denial and no-Keychain-dialog proof remain pending.
There was a problem hiding this comment.
Disposition: addressed / accepted residual risk.
The revised flow removes the candidate-readable login step. The trusted parent selects and validates the two explicit caam identities, copies only their file-backed session credentials into each run-owned home, and removes those copies through the original sandbox profile during both normal completion and failure/interruption cleanup. The candidate sandbox denies the daily home, vault, Keychain paths, and publisher state; source and evidence roots are additionally protected during exercise.
That means the candidate can still read (and potentially exfiltrate) the deliberately selected session credentials for its own harness. Given the operator’s explicit architecture choice, that is an accepted residual exposure rather than a defect in this comment’s threat model. I would keep the PR’s remaining gate as operational: require the real Mac run to demonstrate the Seatbelt denials and absence of Keychain dialogs before treating the live evidence as complete.
This comment has been minimized.
This comment has been minimized.
Operator-reported self-test fixPushed Confirmed the Approved sandbox checks rerun: shipped locked install/tests/strict typecheck (158 passing tests, 648 assertions); static-only invariants with helper locked install/tests/strict typecheck (34 passing tests, 357 assertions); strict plugin validation; four manifest parses; shell syntax; diff whitespace; empty protected-path diff under PR remains draft. Per operator report, the Mac doctor passed at the prior head; |
Live evidence: Open PstackPinned PR #111: |
Codex requires CODEX_HOME to exist before marketplace installation. Create private config directories for both harnesses and regression-test the complete preparation path before every isolated command. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Build handoff — failed Mac preparation fixFixed the operator-reported missing Pushed The new regression runs the real Sandbox verification:
Operator handoff: the failed Mac evidence remains valid as a failure for the old head FindingsNone |
Live evidence: Open PstackPinned PR #111:
|
Assert real home, temporary directory, working directory, and private Claude settings alongside config homes at each preparation command so command stubs cannot conceal missing startup paths. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Build handoffAssigned branch: The config-home fix is in checkpoint Validation rerun: static invariants including locked verifier install, 35 helper tests/510 assertions and strict typecheck; shipped locked install, 158 tests/648 assertions and all three strict typechecks; strict plugin validation; four manifest JSON parses; Bash syntax; whitespace checks; empty protected-path diff for Assumption: installation-owned Codex plugin/marketplace descendants are created by native install commands; this sandbox does not establish live CLI behavior. No additional missing input paths were found in preparation. No open implementation questions. Operator: rerun doctor and live proof at FindingsNone |
Isolate harness configuration and temporary files without replacing HOME, avoiding the operator-observed Keychain reset dialog. Preserve credential scrubbing and verify real HOME alongside private run-owned config paths. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Build handoff — preserve real HOME for Mac authenticationAssigned branch:
Tests check HOME equality with the parent process, run-root config directories and temporary paths, and actual preparation filesystem existence/permissions without treating HOME as disposable. Updated SKILL.md and isolation.md explain config-state isolation, the operator-measured config-scoped Claude OAuth behavior on 2.1.283, and the explicit prohibition on keychain manipulation. Run-owned receipt 'home' fields remain state roots, not process HOME. Sandbox validation passed: verifier locked install, 35 tests/527 assertions and strict typecheck; shipped locked install, 158 tests/648 assertions and all three strict typecheck projects; static invariants; strict plugin validation; all four manifest parses; Bash syntax; whitespace checks. Protected-path diff against origin/main for .github, plugins/pstack and .mergify.yml is empty. Assumption: operator-measured real-HOME/config-dir OAuth separation applies to the probed Claude version; sandbox checks do not establish Mac login success or installed-harness live proof. No open implementation questions. Operator handoff: rerun doctor and live proof on FindingsNone |
Live evidence: Open PstackPinned PR #111:
|
The operator measured missing skill discovery with empty setting sources and unknown Keychain account names without USER/LOGNAME. Load only project settings and preserve account identity without relaxing credential or config isolation. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Build handoff — project discovery and Keychain account identityPushed Operator evidence at prior head
Approved sandbox checks passed:
Assumption: the operator's measured project-only discovery behavior applies to the probed Claude version; sandbox tests verify launch/env construction, not native discovery or login. No open implementation questions. Operator handoff: rerun on exact head FindingsNone |
Live evidence: Open PstackPinned PR #111:
|
| const env: Record<string, string> = { PATH: (process.env.PATH ?? '').split(':').filter(isAbsolute).join(':'), | ||
| HOME: realHome, TMPDIR: join(home, 'tmp'), TERM: process.env.TERM ?? 'xterm-256color', | ||
| GIT_CONFIG_GLOBAL: '/dev/null', GIT_CONFIG_SYSTEM: '/dev/null' }; | ||
| for (const name of ['USER', 'LOGNAME']) { | ||
| if (process.env[name] !== undefined) env[name] = process.env[name]; |
There was a problem hiding this comment.
isolatedEnv gives the candidate session the operator’s real HOME. A changed plugin hook can then read existing files such as ~/.ssh or GitHub credentials without a new login. Redirecting Claude and Codex configuration does not protect those other daily secrets. Keep candidate code from reading the operator’s daily home before treating the run as isolated.
How this was verified: The candidate’s plugin runs a session-start command, and that session receives the operator’s real HOME.
There was a problem hiding this comment.
Addressed in combined review-round commit 980a303. Replaced real-HOME execution with run-owned HOME and a deny-default inherited Seatbelt boundary, including daily home, canonical caam vault, SSH/GitHub/provider state and Keychain denial. Candidate descendants use both isolated provider configuration roots. Real-filesystem Mac sentinel tests are committed but skipped on Linux; operator Mac enforcement proof is still required.
![unfret-eal[bot]](https://avatars.githubusercontent.com/u/28211?s=60&v=4){kind=small}
There was a problem hiding this comment.
🛑 14 blocking findings
🔴 Critical · The setup exercise still targets daily user files
🔴 Critical · Stored GitHub credentials remain available to candidates
🔴 Critical · External provider children fall back to daily configuration
🔴 High · Post-exercise tree check fails once the plugin bootstraps its deps
🔴 High · Do not exempt shipped plugin assets from live verification
🔴 High · Record evidence in the PR template before readiness
🔴 High · Helper tests fail on macOS: fixture root is not canonical
🔴 High · Bootstrap changes select the wrong runtime feature
🟡 Medium · Bind project self-tests to the pinned source tree
🟡 Medium · Redaction can corrupt serialized receipt JSON
🟡 Medium · Evidence roots beneath any state directory are rejected
🟡 Medium · Classification is not bound to the pinned SHAs
🟡 Medium · Previously reviewed evidence is not revalidated
🟡 Medium · Evidence comment overflows budget for shared selections; failure cannot publish
Check
| const raw = join(home, 'surface.raw'); | ||
| console.log('Exercise every requested feature from the native surface, using the maintained feature map. Exit when done.'); | ||
| await this.run(['script', '-q', raw, ...launch(harness, home, workspace)], { cwd: workspace, env, interactive: true }); | ||
| if (await treeHash(installation.location) !== installation.treeHash) throw new Error('Installed tree changed during exercise'); |
There was a problem hiding this comment.
🔴 High · Post-exercise tree check fails once the plugin bootstraps its deps
Line 95 re-hashes installation.location after the native session. For Claude this is the checkout's plugins/pstack passed via --plugin-dir; for Codex it is the installed copy. Per the repo .gitignore, poteto-mode's bootstrap.ts installs bun deps into plugins/pstack/skills/poteto-mode/scripts/node_modules on first run. Exercising runner/shipped-tools through poteto-mode creates that tree inside the hashed location; these features are selected by every shared-path change, such as a plugin.json bump. treeHash then either throws on node_modules/.bin symlinks or returns a new digest. exercise() throws and the run publishes failure for a correct candidate, so these PRs can never get live-gate=success. Also reported for this defect: - Normal tool bootstrapping fails the installed-tree check (.claude/skills/verify-open-pstack/scripts/harness.ts:95): Fresh checkouts have no poteto-mode node_modules. Invoking orch or watch-pr runs bootstrap.ts, which installs dependencies inside the plugin tree and creates symlinks such as node_modules/.bin/tsc. The subsequent treeHash rejects those symlinks before evidence review. This was reproduced with orch --help, so legitimate shipped-tools exercises cannot publish success.
There was a problem hiding this comment.
Addressed in combined review-round commit 980a303. Tree/source validation excludes only the explicitly approved generated dependency leaves. Trusted pinned manifests still validate tracked files, executable modes and the canonical skill alias; filesystem-only post-install/exercise checks reject source mutations. Real Git/filesystem bootstrap and mutation regressions pass.
| const expected = await realpath(join(workspace, '.claude/skills/verify-open-pstack')); | ||
| for (const text of texts) { | ||
| try { | ||
| const d = JSON.parse(text); |
There was a problem hiding this comment.
🟡 Medium · Bind project self-tests to the pinned source tree
The self-test accepts a report solely because its skill path resolves to the current workspace directory. During the writable native session, candidate-controlled instructions can edit or replace that directory before running doctor; the expected and reported paths still match, so modified or stale verifier code can satisfy the exact-head gate without executing the pinned source.
There was a problem hiding this comment.
Addressed in combined review-round commit 980a303. Pinned provenance is built from an independently fetched trusted checkout and immutable Git blobs/modes. Candidate filesystem digests are compared without invoking candidate Git. Self-test doctor output must be candidate-mode and canonical to the pinned workspace. Exercise policy freezes source/plugin paths to prevent modify-execute-restore; redirect/forged-Git regressions pass.
| Invoke `/pstack:setup-pstack` in the isolated Claude session, or `$setup-pstack` in the isolated Codex session. The installed instructions are the authority for the setup inputs. | ||
|
|
||
| ## Driving it with verify-open-pstack | ||
|
|
||
| Use only the run-owned fixture workspace and isolated harness home. Exercise each changed setup branch with explicit requested model/provider inputs. Retain the native invocation transcript and generated configuration file (redacted), then invoke a consuming installed skill and observe it use the configured route. Check that all paths written remain in run-owned state. Record requested versus observed provider/model and changed failure branches. Capture separate local CLI output only as supplemental evidence. |
There was a problem hiding this comment.
🔴 Critical · The setup exercise still targets daily user files
The prescribed setup invocation writes literal ~/.claude/pstack-models.md and ~/.claude/CLAUDE.md, or their ~/.codex equivalents, as specified by the shipped setup skill. HOME remains the real operator home, and the CLI configuration variables do not redirect those paths. Completing this exercise can replace the operator's daily model sheet and integration instead of writing run-owned fixture configuration.
There was a problem hiding this comment.
Addressed in combined review-round commit 980a303. Setup now uses fixture-only run-owned HOME/config paths. Daily user files are outside the inherited candidate sandbox, and both configuration variables are set for all descendants. Documentation and real-filesystem daily-path sentinel tests cover this boundary. Mac OS enforcement remains required operator proof.
| "README.md", "AGENTS.md", "CLAUDE.md", "CHANGES.md", "LICENSE", "LICENSE.md", "UPSTREAM.md", | ||
| ".gitignore", ".mergify.yml", ".github/**", "docs/**", "assets/**", "tests/**", | ||
| "scripts/**", | ||
| "plugins/pstack/assets/**", "plugins/pstack/skills/poteto-mode/scripts/bootstrap.test.ts", |
There was a problem hiding this comment.
🔴 High · Do not exempt shipped plugin assets from live verification
This blanket exemption classifies every plugins/pstack/assets/** change as non-runtime. An asset-only change to a manifest-consumed resource such as the Codex plugin logo therefore skips installation and both native harnesses, yet receives live-gate=success and can be marked ready despite changing the installed user surface.
There was a problem hiding this comment.
Addressed in combined review-round commit 980a303. Shipped plugin assets are runtime-owned shared paths rather than non-runtime exemptions. Logo/asset mapping regressions now require affected feature coverage.
| await github.status(receipt.sha, 'success', receipt.commentUrl, receipt.selection.noRuntime ? 'no runtime change' : 'All mapped features passed in both harnesses'); | ||
| receipt.status = 'success'; await persist(receipt); await current(); | ||
| if (initial.draft) { | ||
| await phase('ready'); readyAttempted = true; await github.ready(receipt.pr); receipt.madeReady = true; await persist(receipt); await current(); |
There was a problem hiding this comment.
🔴 High · Record evidence in the PR template before readiness
The success path posts an issue comment and status, then marks the PR ready without reading or updating its body. Following the documented workflow leaves the PR template's live-evidence fields unchecked and empty while making the PR ready, violating the mandatory rule that a PR without installed version, surface, action, and observed result in the template remains draft. Also reported for this defect: - Readiness bypasses the required PR-template evidence (.claude/skills/verify-open-pstack/scripts/verify.ts:44): AGENTS.md requires installed version, surface, action, and observed result in the pull request template before leaving draft. This path publishes only an issue comment and then marks the PR ready; it neither updates nor checks the template's Live evidence section. Running the documented handoff on the supplied pending-evidence PR can therefore clear draft while its template still lacks the mandatory evidence.
Withdrawn in the latest review.There was a problem hiding this comment.
Addressed in combined review-round commit 980a303. Publication updates and reads back the managed Live evidence block with installed version, surface, action and observed result before success status/readiness. Missing or altered body evidence blocks success. Readback, idempotence and already-ready preservation regressions pass.
| const phase = async (next: Receipt['phase']): Promise<void> => { receipt.phase = next; await persist(receipt); await current(); }; | ||
| try { | ||
| await phase('classify'); | ||
| receipt.selection = classify(await github.files(receipt.pr), options.registry); |
There was a problem hiding this comment.
🟡 Medium · Classification is not bound to the pinned SHAs
The files endpoint reads the mutable PR, not receipt.sha/base. If head A changes to B while files are fetched and returns to A before current(), all boundary checks pass. With equal file counts, a docs-only list from B can classify runtime-changing A as no-runtime, skip both installations, and publish success on A. Endpoint checks cannot detect this reachable A→B→A race.
Withdrawn in the latest review.There was a problem hiding this comment.
Addressed in combined review-round commit 980a303. Classification now comes from exact pinned base...head Git objects in a fresh trusted bare repository, not the mutable PR-files endpoint. Immutable-Git classification tests cover renames, malformed diffs and pinned inputs.
| } | ||
| if (harness === 'claude') env.CLAUDE_CONFIG_DIR = join(home, 'config'); |
There was a problem hiding this comment.
🔴 Critical · External provider children fall back to daily configuration
Only the parent's configuration variable is set. When a Claude parent dispatches Codex through pstack-runner, CODEX_HOME is absent; the reverse dispatch lacks CLAUDE_CONFIG_DIR. The existing runner preserves this environment, so external children resolve their configuration and authentication through the operator's real HOME. Required mixed-provider exercises consequently access daily state rather than isolated provider installations.
Withdrawn in the latest review.There was a problem hiding this comment.
Addressed in combined review-round commit 980a303. Both CLAUDE_CONFIG_DIR and CODEX_HOME are always run-owned, alongside HOME/GH/XDG/TMP roots. Both selected account credentials are copied into each candidate state, and descendant execution inherits the OS boundary. No external-provider child can intentionally fall back to daily files; Mac real-process denial proof remains pending.
| if (!receipt.selection.noRuntime || receipt.selfTest) { | ||
| await phase('prepare'); receipt.installations = await driver.prepare(receipt); await current(); await persist(receipt); | ||
| await phase('exercise'); receipt.observations = await driver.exercise(receipt); await current(); await persist(receipt); | ||
| } |
There was a problem hiding this comment.
🟡 Medium · Previously reviewed evidence is not revalidated
If the Codex exercise overwrites a reviewed transcript or artifact filename previously selected for Claude, the Claude observation retains its old hash although the retained file now contains Codex evidence. completeEvidence checks only hash syntax, so publication still succeeds with hashes that do not match the retained files. This was reproduced by reusing reviewed.log and fixture.json across the two exercises. Also reported for this defect: - Rehash retained evidence before publication (.claude/skills/verify-open-pstack/scripts/core.ts:70-72): Completion validates only that previously recorded hashes have the right syntax; it never rereads the retained files. If an operator reuses and overwrites one artifact path for a later feature, or a file changes after selection, the earlier observation keeps a stale digest while the gate still succeeds and publishes evidence that no longer matches the retained content.
Also affects:
Withdrawn in the latest review.There was a problem hiding this comment.
Addressed in combined review-round commit 980a303. Retained transcript/artifact hashes are recomputed before publication and again at publication/readiness boundaries. Real-filesystem overwrite regressions reject reused evidence that changed after operator review.
| for (const record of receipt.observations) lines.push(`- ${record.harness} / ${safe(record.feature)}: ${safe(record.surface)}; action: ${safe(record.action)}; observed: ${safe(record.observed)}; operator reviewed.`, | ||
| ` Transcript \`${safe(record.transcript)}\` SHA-256 \`${record.transcriptHash}\`; artifacts ${record.artifacts.map(a => `\`${safe(a.path)}\` (\`${a.sha256}\`)`).join(', ')}.`); |
There was a problem hiding this comment.
🟡 Medium · Evidence comment overflows budget for shared selections; failure cannot publish
evidence() emits two uncapped lines per observation: about 300 fixed chars (two 64-hex hashes, paths, labels) plus up to 3x2000 chars of operator text. Publisher.comment (github.ts:27) rejects bodies over 60000 chars. A shared-path PR, e.g. a plugin.json version bump, selects 54 skills plus setup/runner/shipped-tools, giving 114 observations and about 35k chars of fixed overhead. Ordinary operator answers or a second artifact push the body past 60000, so comment() throws after all exercises. The catch block then reposts the full evidence(receipt) at line 62, which also exceeds the budget. Neither the evidence nor a failure live-gate is published, and the non-resumable run is lost.
Withdrawn in the latest review.There was a problem hiding this comment.
Addressed in combined review-round commit 980a303. Evidence rendering now bounds and compacts per-observation details below the publisher budget while retaining the full receipt. Large feature/observation regression covers bounded success and failure comments.
| "setup": ["plugins/pstack/skills/setup-pstack/SKILL.md"], | ||
| "runner": [ | ||
| "plugins/pstack/skills/poteto-mode/scripts/runner/**", | ||
| "plugins/pstack/skills/poteto-mode/scripts/bootstrap.ts" |
There was a problem hiding this comment.
🔴 High · Bootstrap changes select the wrong runtime feature
Classifying a bootstrap.ts change returns only runner, but its actual consumers are orch and watch-pr; pstack-runner does not import it. A bootstrap regression that prevents commander installation can therefore pass the selected external-runner exercise while both shipped tools remain broken. This path needs ownership that requires exercising its actual consumers.
There was a problem hiding this comment.
Addressed in combined review-round commit 980a303. bootstrap.ts is mapped to shipped-tools coverage for its orch/watch-pr consumers rather than only runner. The maintained map and consumer-selection regression were updated.
Step-back result: PR #111 after reviewed head 4 (
|
Remove rejected PR mutation, sandbox, and publisher attestation mechanisms while preserving exact-head evidence, disposable credentials with selected-account caam health checks, private raw evidence, and the setup config-home blocker. Include the already-validated upstream tree without rewriting reviewed branch history. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
…ive gate Integrate the merged main gate contract without rewriting factory/issue-90 history. Resolve the verifier guidance alongside the selected-account health correction. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
There was a problem hiding this comment.
🛑 5 blocking findings
🔴 Critical · Transient refreshed credentials escape evidence scanning
🔴 High · Evidence comment drops surface, action and observed result
🔴 High · Mergify enforcement changes bypass behavioral verification
🔴 High · Project proof never exercises candidate verification operations
🟡 Medium · Publisher binding does not identify already-loaded code
Check
| } | ||
| let shown = 0; | ||
| for (const record of receipt.observations) { | ||
| const line = `- Observation (${record.harness} / ${safe(record.feature, 72)}): surface ${safe(record.surface, 48)}; action ${safe(record.action, 48)}; result ${safe(record.observed, 48)}; operator reviewed; transcript SHA-256 \`${record.transcriptHash}\`; artifact-set SHA-256 \`${sha256(JSON.stringify(record.artifacts))}\`.`; |
There was a problem hiding this comment.
🔴 High · Evidence comment drops surface, action and observed result
evidence() renders surface, action and observed through safe(..., 48). Any value longer than 48 characters, containing non-ASCII, or containing a backtick, < or > is replaced whole with '[value omitted]' (see line 7). The operator gets no length guidance; the feature docs ask for input, expected and observed results, and every changed sub-feature. Nothing blocks publication when a value is omitted. For example, the action 'Ran verify.sh doctor --candidate --output
' or the result 'doctor.json result pass, candidate true, canonical skill path' each becomes '[value omitted]'. verify() still posts live-gate=success targeting that comment. Base AGENTS.md requires the live-gate to link an evidence comment naming the surface, action and observed result. On the normal path the gate record therefore does not meet that requirement. Also reported for this defect: - Success comment silently drops mandated surface/action/result (.claude/skills/verify-open-pstack/scripts/verify.ts:27): AGENTS.md requires live-gate to link an evidence comment naming the installed version, surface, action and observed result. evidence() renders surface, action and observed through safe(…, 48). That replaces any answer longer than 48 characters, or containing non-ASCII, a backtick,< or >, with "[value omitted]". The operator prompts in harness.ts:129-131 accept arbitrary text and never warn or re-prompt, and completeEvidence only checks that the fields are non-empty. Scenario: the operator answers "Concrete action exercised?" with "Invoked /verify-open-pstack and ran doctor --candidate" (54 chars). Alternatively, the operator pastes a result containing ✓ from the native transcript. The run still posts live-gate=success, and the linked comment reads "action [value omitted]; result [value omitted]" for each such observation. The mandated record therefore does not name what was tested. The full text exists only in the private receipt, and this happens on the ordinary review path. - Success comments can omit required live evidence (.claude/skills/verify-open-pstack/scripts/verify.ts:27): Operator descriptions longer than 48 characters, or containing backticks or non-ASCII text, are replaced entirely with '[value omitted]'. completeEvidence accepts those descriptions, and verification still publishes success. A reproduced runtime run omitted every surface, action, and result from its public comment. This violates AGENTS.md's requirement that the live-gate target comment name those facts; private hashes do not supply the missing evidence.
Withdrawn in the latest review.
| ], | ||
| "nonRuntime": [ | ||
| "README.md", "CHANGES.md", "LICENSE", "LICENSE.md", "UPSTREAM.md", | ||
| ".gitignore", ".mergify.yml", "docs/**", "assets/**", |
There was a problem hiding this comment.
🔴 High · .mergify.yml changes auto-pass the live gate
Listing .mergify.yml as nonRuntime makes an enforcement-only PR skip installation and every observation while still receiving live-gate=success. A PR removing the live-gate queue condition can therefore pass the very gate it weakens, leaving subsequent PRs able to queue without the mandatory protection. Also reported for this defect: - .mergify.yml exemption auto-passes merge-gate changes (.claude/skills/verify-open-pstack/features/registry.json:49): Carried v1:51192f49, still present. Because .mergify.yml is listed in nonRuntime, a PR that only edits, renames or deletes it is classified noRuntime. verify() then skips prepare and exercise and publishes 'PASSED — no runtime change' with live-gate=success. A PR that removes the live-gate queue condition therefore gets the mandated success with nothing exercised, and later PRs queue without the gate. core.test encodes this behavior. - Mergify enforcement changes auto-pass the live gate (.claude/skills/verify-open-pstack/features/registry.json:49): Listing .mergify.yml as non-runtime makes an enforcement-only PR select no features or harnesses. A PR that removes the live-gate queue condition therefore skips installation and exercise but still receives live-gate=success, allowing the configuration protecting the gate to pass without testing its changed behavior. - .mergify.yml exemption lets merge-gate changes auto-pass live-gate (.claude/skills/verify-open-pstack/features/registry.json:49): Carried v1:51192f49, still present. .mergify.yml is listed in nonRuntime, so classify() marks a PR that only edits, renames or deletes it as noRuntime. verify() then skips prepare and exercise and posts 'PASSED — no runtime change' with live-gate=success. Scenario: a PR removes the live-gate queue condition. It receives the exact status the base AGENTS.md gate requires without any installation or exercise, and later PRs can queue without the gate. core.test.ts encodes this behavior. Also reported for this defect: - Mergify enforcement changes auto-pass the live gate (.claude/skills/verify-open-pstack/features/registry.json:49): Classifying .mergify.yml as nonRuntime makes an enforcement-only PR skip installation and all observations while still receiving live-gate=success. A PR removing the live-gate queue condition can therefore pass the very gate it weakens, leaving later PRs able to queue without the mandatory protection. - .mergify.yml changes auto-pass their own gate (.claude/skills/verify-open-pstack/features/registry.json:49): Listing .mergify.yml as nonRuntime makes an enforcement-only PR skip installation and exercise while still receiving live-gate=success. A PR removing the live-gate queue condition can therefore pass the very gate it weakens, and subsequent PRs can queue without the mandatory protection. - .mergify.yml exemption auto-passes merge-gate changes (.claude/skills/verify-open-pstack/features/registry.json:49): nonRuntime lists .mergify.yml, so a PR that edits, renames or deletes it is classified noRuntime. verify() then skips prepare and exercise and posts 'PASSED — no runtime change' with live-gate=success on that SHA. Scenario: a PR removes the live-gate queue condition. It gets success with zero installs or observations, and once merged, later PRs queue without the base AGENTS.md live gate. core.test.ts enshrines this. Carried v1:51192f49 is unchanged at this head. Also reported for this defect: - .mergify.yml exemption auto-passes merge-gate changes (.claude/skills/verify-open-pstack/features/registry.json:49): registry.json lists .mergify.yml under nonRuntime. As a result, classify() returns noRuntime, and verify() skips prepare and exercise, posting 'PASSED — no runtime change' with live-gate=success. Scenario: a PR removes the live-gate queue condition, or deletes .mergify.yml. It receives the exact status that base AGENTS.md says Mergify requires, with nothing exercised. After merge, later PRs queue without any live gate. Other enforcement paths, such as .github/**, fail closed. This is still-open demand v1:51192f49. core.test's 'Mergify enforcement is not plugin runtime' encodes the bypass. - .mergify.yml classified nonRuntime bypasses live-gate verification (.claude/skills/verify-open-pstack/features/registry.json:49): registry.json lists .mergify.yml under nonRuntime. For a PR touching only .mergify.yml, classify() returns noRuntime=true (confirmed by core.test 'Mergify enforcement is not plugin runtime'), so verify() skips prepare/exercise and publishes a 'PASSED — no runtime change' comment plus live-gate=success on the exact SHA. A PR that removes or weakens the live-gate/queue conditions in .mergify.yml — the configuration that enforces the merge gate itself — therefore receives a green live-gate with zero behavioral verification of the enforcement change. The AGENTS.md live-gate record is produced for a change that was never exercised, creating a self-referential enforcement hole where the config protecting merges can be edited and the edit auto-passes its own gate. This is a diff-visible classification that lets enforcement-config changes bypass verification. - Mergify enforcement changes bypass behavioral verification (.claude/skills/verify-open-pstack/features/registry.json:49): A PR removing a required status or queue condition from .mergify.yml is classified as noRuntime and receives live-gate=success without validating the changed merge-control behavior. This was reproduced with zero installations and observations. The exemption treats the configuration enforcing the mandatory live gate as ordinary documentation, allowing enforcement changes to receive a successful verification result without exercising their behavior. Also reported for this defect: - .mergify.yml exemption lets merge-gate changes auto-pass live-gate (.claude/skills/verify-open-pstack/features/registry.json:49): Line 49 lists .mergify.yml under nonRuntime. As a result, classify() marks any PR that only edits, renames or deletes it as noRuntime. verify() then skips prepare and exercise and publishes "PASSED — no runtime change" with live-gate=success on that exact SHA. Scenario: a PR removes the live-gate condition from the Mergify queue rules, or deletes .mergify.yml. The verifier posts success immediately and Mergify can queue the PR. After merge, later PRs are no longer held by the live gate, and nothing ever exercised the changed merge-control behavior. This is still-open demand v1:0eb897f in another form. The project-skill mapping was dropped, but the replacement restores the exemption that v1:0589a23 rejected. Other enforcement paths such as .github/** fail closed instead. The test 'Mergify enforcement is not plugin runtime' in core.test.ts encodes this behavior. Also reported for this defect: - .mergify.yml changes auto-pass their own enforcement gate (.claude/skills/verify-open-pstack/features/registry.json:49): Classifying .mergify.yml as non-runtime yields no features or harnesses, so verification skips installation and exercise and publishes live-gate=success. A PR that removes the live-gate queue condition or otherwise weakens merge enforcement therefore receives a successful gate without testing the changed enforcement, and subsequent PRs can queue without the mandated protection. - .mergify.yml changes auto-pass the gate (.claude/skills/verify-open-pstack/features/registry.json:49): Listing .mergify.yml as nonRuntime gives an enforcement-only PR an empty feature selection, so verification skips installation and exercise and publishes live-gate=success. A PR removing the live-gate queue condition can therefore auto-pass the very gate it weakens, leaving later PRs able to queue without behavioral verification. - .mergify.yml exemption auto-passes merge-gate changes (.claude/skills/verify-open-pstack/features/registry.json:49): registry.json lists .mergify.yml under nonRuntime. classify() (core.ts) therefore returns noRuntime=true for a PR that only edits/renames/deletes .mergify.yml; core.test 'Mergify enforcement is not plugin runtime' encodes this. verify() then skips prepare/exercise and publishes 'PASSED — no runtime change' with live-gate=success on that SHA (verify.ts:100,110). Scenario: a PR removes the live-gate queue condition from .mergify.yml. It receives the exact success status base AGENTS.md says Mergify requires, with zero installations or observations and no exercise of the changed merge-enforcement behavior. This diff-visible classification lets a change to the configuration that enforces the live gate bypass that very gate.
Now at .claude/skills/verify-open-pstack/features/registry.json:49
| const env = isolatedEnv(home, harness), candidateRun = this.candidate(home); | ||
| const request = { sha: receipt.sha, harness, workspace, features: requiredFeatures(receipt), | ||
| featureMap: join(workspace, '.claude/skills/verify-open-pstack/features'), | ||
| selfTest: requiredFeatures(receipt).includes('project-skill') ? `Invoke the candidate project skill natively; run doctor --candidate --output "${join(home, 'self-test')}". Do not invoke run, vault listing, login, or publication recursively.` : false, |
There was a problem hiding this comment.
🔴 High · Project-skill proof exercises only doctor
The native project request explicitly runs doctor --candidate and forbids run. A verifier change that breaks a branch not used by this successful delivery run, such as no-runtime classification or failure publication, still passes both capability-only doctors and receives success without the changed behavior being exercised from either native surface. Also reported for this defect: - Project-skill native proof only runs candidate doctor (.claude/skills/verify-open-pstack/scripts/harness.ts:115): Carried v1:536a6e5d, still present. For every project-skill selection, the native request limits each harness to doctor --candidate and forbids run. Doctor only probes versions and help, and verifyProjectDoctor accepts any passing canonical-path report. A verifier PR that breaks classification, preparation, cleanup, evidence or publication still produces accepted observations in both harnesses and live-gate=success, without its changed behavior running from the native surface. - Project-skill proof does not exercise changed verifier behavior (.claude/skills/verify-open-pstack/scripts/harness.ts:115): For every verifier change, the native request explicitly runs only doctor --candidate and forbids run. Doctor probes versions and help, so a change that breaks an unexercised branch such as no-runtime classification or evidence publication can still produce accepted observations in both harnesses and receive success without live-testing that behavior. - Project-skill proof only runs candidate doctor (.claude/skills/verify-open-pstack/scripts/harness.ts:115, .claude/skills/verify-open-pstack/scripts/harness.ts:117): Carried v1:536a6e5d, still present. For every project-skill selection, the native request is limited to doctor --candidate and forbids run. Doctor only probes versions and help, and verifyProjectDoctor accepts any passing canonical-path report. A verifier change that breaks classification, preparation, cleanup, evidence or publication behavior still yields accepted project-skill observations in both harnesses and live-gate=success. The outer candidate-publisher run is not a native-surface exercise of the changed behavior. Also reported for this defect: - Project proof exercises only doctor (.claude/skills/verify-open-pstack/scripts/harness.ts:115): The native project-skill request explicitly runs doctor --candidate and forbids run, while verifyProjectDoctor checks only the passing report and canonical path. A verifier change that breaks its native run instructions or harness-specific verification workflow while preserving capability probes can still produce accepted observations in both harnesses and receive success without live-testing the changed behavior. - Project proof does not exercise changed verifier behavior (.claude/skills/verify-open-pstack/scripts/harness.ts:115): The project-skill request explicitly limits each native harness to doctor --candidate and forbids run. For example, a verifier PR can break no-runtime classification or publication while leaving capability probes intact; both harnesses produce accepted doctor artifacts and the broken branch receives success without its changed behavior being exercised from either real user surface. - Project proof does not exercise changed verifier behavior (.claude/skills/verify-open-pstack/scripts/harness.ts:115, .claude/skills/verify-open-pstack/scripts/harness.ts:117): For every project-skill change, the native request is limited to doctor --candidate and explicitly forbids run. Doctor only probes versions/help and canonical discovery, so changes that break classification, preparation, cleanup, evidence validation, or publication can still produce accepted project-skill observations in both harnesses. The resulting success does not live-test the changed behavior from those real user surfaces as required. Also reported for this defect: - Native project proof exercises only doctor (.claude/skills/verify-open-pstack/scripts/harness.ts:115): The project-skill request explicitly runs only doctor --candidate and forbids run. A verifier change that breaks its native run instructions, classification, installation, cleanup, or publication path while leaving capability probes intact still produces passing doctor artifacts in both harnesses and can receive complete project evidence, despite the changed native behavior never being exercised. - Project proof never exercises candidate verification operations (.claude/skills/verify-open-pstack/scripts/harness.ts:117): Changes to the verifier select project-skill, but this request limits the candidate exercise to doctor --candidate. That command probes CLI capabilities rather than executing candidate classification, installation, cleanup, or publication behavior; the publishing run executes trusted-main implementations instead. A pinned fixture whose candidate verify() always throws still produced an accepted passing doctor.json. Consequently, complete project evidence can approve broken candidate verification behavior without satisfying AGENTS.md's changed-behavior gate.
Now at .claude/skills/verify-open-pstack/scripts/harness.ts:115
Also affects:
Replace required account-vault infrastructure with explicit authenticated Claude and Codex configuration directories, native status checks, and credential-file-only candidate copies. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Re-prompt unpublishable observation values and require verifier-changing PRs to run their reviewed candidate publisher before preparing native sessions. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
There was a problem hiding this comment.
🛑 8 blocking findings
🔴 Critical · Retained evidence never screened for copied/refreshed tokens
🔴 Critical · Transient refreshed credentials escape evidence scanning
🔴 High · .mergify.yml exemption auto-passes merge-gate changes
🔴 High · Project proof never exercises candidate verification operations
🔴 High · macOS Claude logins never provide the required credentials file
🟡 Medium · Forked refresh tokens can log out the daily account
🟡 Medium · Publisher binding does not identify already-loaded code
🟡 Medium · Blank evidence answers accepted, then rejected after all sessions
Check
| await chmod(target.directory, 0o700); | ||
| const file = await open(target.path, 'wx', 0o600); | ||
| copied.push(target.path); | ||
| try { await file.writeFile(bytes); } |
There was a problem hiding this comment.
🟡 Medium · Forked refresh tokens can log out the daily account
copyCredentials copies the source refresh token into the candidate home. The native client may refresh during the session, and cleanup deletes the rotated tokens without writeback. Providers that rotate refresh tokens with reuse detection (OpenAI's Codex login reports refresh-token reuse) then reject the source copy. The operator's daily Codex fails its next refresh and must log in again. Conversely, a daily-session refresh mid-run invalidates the candidate copy and fails the exercise. Both outcomes contradict 'daily-active accounts are allowed'.
Withdrawn in the latest review.| const bytes = await readFile(actual); | ||
| return { path: relative(output, actual), sha256: sha256(bytes) }; |
There was a problem hiding this comment.
🔴 Critical · Retained evidence accepts known live credentials
retainedFile reads and hashes evidence without screening its contents. If a candidate prints the symlinked Codex auth.json into a transcript or selected artifact, the file is accepted outside state; cleanup removes only the auth link, leaving an additional live credential copy in evidence retained and archived through merge. Also reported for this defect: - Known provider credentials can be archived as evidence (.claude/skills/verify-open-pstack/scripts/io.ts:105-106): retainedFile reads and hashes arbitrary transcript or artifact bytes without credential screening. If candidate code prints the symlinked daily Codex auth into an operator-retained file, it is accepted and preserved outside state through merge and archival, leaving an additional live credential copy after cleanup. - Retained evidence is never screened for the linked credential (.claude/skills/verify-open-pstack/scripts/io.ts:105-106): Carried v1:6155b97d, still present. retainedFile checks containment and size, then hashes the bytes; io.test asserts that token-bearing artifacts are accepted. Scenario: a native session prints $CODEX_HOME/auth.json, which is the symlinked daily login. The operator retains that transcript or artifact as required. Cleanup removes only the link, so the live token stays in the evidence root, which is kept through merge and archived. - Retained evidence can contain the daily credential (.claude/skills/verify-open-pstack/scripts/io.ts:105, .claude/skills/verify-open-pstack/scripts/io.ts:105-106, .claude/skills/verify-open-pstack/scripts/io.ts:106-107, .claude/skills/verify-open-pstack/scripts/io.ts:111-112): retainedFile reads and hashes evidence bytes without credential screening. A candidate can print the symlinked Codex auth.json into a transcript or artifact that the operator copies outside state; cleanup removes the auth link but the live credential remains in the retained evidence root that is preserved and archived. Also reported for this defect: - Retained evidence is never screened for the linked Codex credential (.claude/skills/verify-open-pstack/scripts/io.ts:105-106): Carried v1:6155b97d, still present. retainedFile checks only location and size, then hashes the bytes; io.test.ts asserts that token-bearing artifacts are accepted. Scenario: a native session prints $CODEX_HOME/auth.json, which is the symlinked daily login, or a skill dumps its auth. The transcript or artifact is accepted, and cleanup removes only the run-owned .codex directory. The live token then stays in the evidence root, which is kept through merge and archived. - Known provider tokens can be archived as evidence (.claude/skills/verify-open-pstack/scripts/io.ts:106-107): retainedFile hashes transcript and artifact bytes without credential screening. If a candidate prints the symlinked Codex auth token into a selected transcript or artifact, the file is accepted outside state; cleanup removes only the auth link, leaving an additional live-token copy in evidence retained and archived through merge. - Retained evidence never screened for copied/refreshed tokens (.claude/skills/verify-open-pstack/scripts/io.ts:111-112, .claude/skills/verify-open-pstack/scripts/io.ts:105-106): retainedFile only checks location and size, then hashes the bytes. Nothing compares transcripts or artifacts with registered credentials; io.test.ts asserts that a registered token is accepted. copyCredentials registers only the source values (isolation.ts:63) and never tokens refreshed mid-session. Scenario: the native session prints a copied or refreshed access/refresh token. The reviewed transcript is accepted, and cleanup unlinks only the two credential files. The live token stays in the evidence root, which is kept through merge and archived. Carried v1:c19f2376 still reproduces, and the earlier known-token rejection (v1:42d56bb6) is gone. Also reported for this defect: - Raw evidence can retain live provider tokens (.claude/skills/verify-open-pstack/scripts/io.ts:105-106): Evidence bytes are read and hashed without any credential screening. A candidate can print the symlinked Codex token, or a transient refreshed token that is later overwritten, and the operator-required transcript/artifact copy is accepted outside state. Cleanup then leaves that live credential in the evidence root that is preserved and archived through merge.
Now at .claude/skills/verify-open-pstack/scripts/io.ts:105-106
Also affects:
| const source = join(canonical[tool], tool === 'claude' ? '.credentials.json' : 'auth.json'); | ||
| const sourceInfo = await lstat(source); | ||
| if (!sourceInfo.isFile() || sourceInfo.isSymbolicLink() || await realpath(source) !== source) throw new Error('credentials must be a contained regular file'); |
There was a problem hiding this comment.
🔴 High · macOS Claude logins never provide the required credentials file
A normal Claude Code login on macOS stores its OAuth credentials in the login Keychain, not in $CLAUDE_CONFIG_DIR/.credentials.json. credentialSources requires that regular file, and doctor/prepare call it first. A source authenticated with the documented CLAUDE_CONFIG_DIR=<dir> claude auth login therefore fails with ENOENT, and the printed remediation writes to the Keychain again. prepare refuses non-Darwin hosts, so every runtime or --self-test run blocks before installation. No live-gate success can be produced, including for PR #111. Also reported for this defect: - Claude source credentials are not a file on the required Mac (.claude/skills/verify-open-pstack/scripts/isolation.ts:20-22): credentialSources() requires /.credentials.json to be a regular file, and doctor() and prepare() refuse non-Darwin hosts. Claude Code on macOS stores normal-login OAuth credentials in the macOS Keychain, per its credential-management docs, not in .credentials.json. Scenario: a normally logged-in --claude-config directory fails with 'claude credential source unavailable: ENOENT'. The suggested remediation, CLAUDE_CONFIG_DIR=
| for (const feature of requiredFeatures(receipt, harness)) { | ||
| const answers = { surface: '', action: '', observed: '' }; | ||
| for (const [field, question] of [['surface', `${harness}/${feature}: native surface/discovery entry point?`], ['action', 'Concrete action exercised?'], ['observed', 'Observed assertion/result (not the model\'s success claim)?']] as const) { | ||
| do { answers[field] = await this.review(`${question} Use printable ASCII, no backtick, < or >, and at most ${EVIDENCE_LIMIT} characters.`); } while (!publishable(answers[field])); |
There was a problem hiding this comment.
🟡 Medium · Blank evidence answers fail only after the live sessions
publishable('') returns true, while the interactive reader trims whitespace. Pressing Enter therefore escapes the re-prompt loop with an empty surface, action, or result; completeEvidence rejects it only after both native sessions and cleanup, publishing failure and forcing the supervised workflow to be repeated. Also reported for this defect: - Blank evidence answers pass the re-prompt guard (.claude/skills/verify-open-pstack/scripts/verify.ts:8): Carried v1:bb076ab1, still present. ask() trims input and publishable('') returns true, so pressing Enter at a surface, action or observed prompt exits the loop at harness.ts:138. completeEvidence rejects the blank only after both native sessions and cleanup have finished. The run then publishes live-gate=failure, and the whole supervised exercise must be redone. - Blank evidence answers fail only after live work (.claude/skills/verify-open-pstack/scripts/verify.ts:8): publishable returns true for an empty string, and the interactive reader trims whitespace. Pressing Enter therefore exits the re-prompt loop with a blank surface, action, or result; completeEvidence rejects it only after all native sessions and cleanup, publishing failure and forcing the supervised workflow to be repeated. - Blank evidence answers fail only after cleanup (.claude/skills/verify-open-pstack/scripts/verify.ts:8, .claude/skills/verify-open-pstack/scripts/harness.ts:131, .claude/skills/verify-open-pstack/scripts/harness.ts:138): publishable('') returns true, and the interactive prompt trims input. Pressing Enter therefore accepts an empty surface, action, or result; completeEvidence rejects it only after all native sessions and cleanup finish, publishing failure and forcing the entire supervised run to be repeated instead of re-prompting immediately. Also reported for this defect: - Blank evidence answers pass the re-prompt guard (.claude/skills/verify-open-pstack/scripts/verify.ts:8): Carried v1:bb076ab1, still present. publishable('') returns true and ask() trims input, so pressing Enter at a surface, action or observed prompt exits the do/while at harness.ts:138. completeEvidence rejects the blank only after both native sessions and cleanup have finished. The run then publishes failure evidence and live-gate=failure and must be redone from scratch. - Blank evidence is accepted until final validation (.claude/skills/verify-open-pstack/scripts/verify.ts:8): publishable returns true for an empty string, and the interactive prompt trims whitespace. Pressing Enter therefore exits the re-prompt loop; completeEvidence rejects the blank only after both native sessions and cleanup, publishing failure and forcing the entire supervised run to be repeated. - Blank evidence answers pass collection validation (.claude/skills/verify-open-pstack/scripts/verify.ts:8, .claude/skills/verify-open-pstack/scripts/harness.ts:131, .claude/skills/verify-open-pstack/scripts/harness.ts:138): publishable('') returns true because the predicate has no nonempty check. Pressing Enter at a surface, action, or observed-result prompt therefore exits the re-prompt loop; completeEvidence rejects the blank only after the native sessions and cleanup finish, publishing failure and forcing the supervised run to be repeated. Also reported for this defect: - Blank evidence answers accepted by re-prompt guard, fail only after both sessions (.claude/skills/verify-open-pstack/scripts/harness.ts:138): In exercise() the surface/action/observed re-prompt loop repeats while (!publishable(answers[field])), but publishable('') is true (length 0 ≤ 160, matches ^[\x20-\x7e]*$, no backtick/angle brackets) and ask() trims input to '' (harness.ts:16). An accidental Enter is thus accepted as a valid answer. completeEvidence() (core.ts) later rejects empty surface/action/observed with 'Incomplete evidence', but that check runs in verify() only after both native harness sessions and cleanup have finished. One empty answer therefore discards the entire supervised run, publishes a failure live-gate on an otherwise-good candidate, and forces a full rerun, whereas every other unpublishable value is correctly re-prompted. - Blank evidence is accepted until final validation (.claude/skills/verify-open-pstack/scripts/verify.ts:8): publishable('') is true, and the production prompt trims whitespace before this predicate is used. Pressing Enter therefore exits the re-prompt loop with an empty surface, action, or result; only completeEvidence rejects it after the native sessions and cleanup have completed. The run publishes failure and all supervised exercises must be repeated instead of correcting the input immediately. - Blank evidence answers accepted, then rejected after all sessions (.claude/skills/verify-open-pstack/scripts/harness.ts:131): ask() trims input, so pressing Enter or typing only whitespace yields ''. The re-prompt loop accepts '' because publishable('') is true. completeEvidence later rejects an empty surface, action or observed with 'Incomplete evidence'. That check runs only after every native session and credential cleanup have finished. One accidental Enter therefore discards the whole supervised run. It also publishes a failure comment and live-gate=failure, and forces a fresh run from scratch. Also reported for this defect: - Blank evidence answers pass the re-prompt guard (.claude/skills/verify-open-pstack/scripts/harness.ts:131): ask() trims input and publishable('') returns true, so pressing Enter at a surface, action or result prompt is accepted. completeEvidence rejects the blank field ('Incomplete evidence') only after both native sessions have finished. The run then publishes a failure live-gate and all exercise work must be redone. Other unpublishable input is re-prompted instead.
Now at .claude/skills/verify-open-pstack/scripts/verify.ts:8
Also affects:
Step-back result 2: PR #111 after reviewed head 6 (
|
| Finding | Class | Basis |
|---|---|---|
1 .mergify.yml nonRuntime |
inert, third raise | Approved classification. It affects no harness and makes no claim about merge policy. |
| 3 publisher revision read after imports | unconfirmed | Needs the operator to move the trusted checkout during startup. The advisor withdrew the private-worktree proposal. |
| 4 candidate-operation proof | inert | verify.ts:95 makes the candidate publisher run the end-to-end flow itself, and its revision is printed in the comment. |
| 5 and 8 private-evidence scanning | inert | Step-back decision 2: raw evidence is private and never published. |
| 7 blank answer | fails-loudly | An honest "Incomplete evidence" failure, not a false success. |
Verification that could disprove the replacement
Do one real Mac run with both dedicated directories. It must cover authenticated native requests, a token refresh, cross-provider children, candidate install and removal, and unchanged daily login and installs. A failed refresh, any interference with daily state, or an unusable uninstall disproves it.
Reopening conditions
- A normal-operation startup race.
- A changed operation that no run exercises.
- A false success from blank evidence.
- A harness that actually consumes
.mergify.yml. - A public leak.
New direction for #111: reuse the operator's existing logins, with no copies and no extra loginsThe maintainer chose this direction on 2026-10-04. It replaces both the credential copying at head Who runs this tool. Only contributors who open PRs against open-pstack, to produce the exact-head Claude Code: the normal login, with the candidate loaded for one session only
Codex: a run-owned
|
AGENTS.md and the PR template drop the draft-until-evidence rule. The live-gate commit status on the exact head, linking its evidence comment, is the record, and Mergify already refuses to queue without it. PRs may be reviewed before their live test, and every new head needs a fresh one. The factory loop in the triage plan follows the same order. Decided in the PR ericlitman#111 step-back (2026-10-03). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove copied credential handling and source flags so verification reuses existing native logins. Preserve setup files byte-for-byte and prohibit publication after restoration failures. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
There was a problem hiding this comment.
🛑 11 blocking findings
🔴 Critical · Codex auth symlink permits daily-login corruption
🔴 Critical · Candidate hooks can reuse the publisher's GitHub login
🔴 Critical · Retained evidence never screened for copied/refreshed tokens
🔴 Critical · Candidate sessions can recover stored GitHub credentials
🔴 Critical · Transient refreshed credentials escape evidence scanning
🔴 High · .mergify.yml exemption auto-passes merge-gate changes
🔴 High · Claude runner exercises have no Codex authentication
🔴 High · Project proof does not exercise changed verifier behavior
🔴 High · Final hashes allow transient source substitution
🟡 Medium · Publisher binding does not identify already-loaded code
🟡 Medium · Blank evidence answers pass collection validation
Check
| if (harness === 'codex') { | ||
| await linkCodexAuth(home); |
There was a problem hiding this comment.
🔴 High · Claude runner exercises have no Codex authentication
Codex auth is linked only for the Codex installation. During a Claude native exercise, CODEX_HOME points to the Claude run home's empty .codex directory; when a selected runner or skill dispatches its Codex lane, that child cannot find auth.json and fails authentication. Runner/shared changes requiring cross-provider behavior therefore cannot complete their required Claude-harness exercise. Also reported for this defect: - Claude-side runner exercises have no Codex login (.claude/skills/verify-open-pstack/scripts/harness.ts:88-90): Authentication is linked only when the parent harness itself is Codex, although every harness receives a run-owned CODEX_HOME. When a runner change requires a Claude parent to dispatch the external Codex lane, pstack-runner inherits the empty Claude-side .codex directory and codex login status fails. Legitimate runner/shared changes affecting that lane therefore cannot complete their required live proof.
Also affects:
Withdrawn in the latest review.| }); | ||
| if (!info?.isFile()) throw new Error('Codex verification requires existing file-based authentication (auth.json); keyring-only authentication is not supported'); | ||
| await mkdir(join(home, '.codex'), { recursive: true, mode: 0o700 }); | ||
| await symlink(source, join(await codexDirectory(home), 'auth.json')); |
There was a problem hiding this comment.
🔴 Critical · Codex auth symlink lets an exercise corrupt the daily login undetected
Carried v1:2d457389, still present. Every run home's $CODEX_HOME/auth.json is a writable symlink to the operator's daily auth.json. Any candidate write to that path follows the link and replaces the daily credential, for example a fixture reset, a login to another account, or junk bytes. assertCodexAuth checks only the link type and target, and removeCodexHome deletes only the link. The run then publishes while the daily Codex login is broken. Also reported for this defect: - Codex auth symlink permits undetected daily-login corruption (.claude/skills/verify-open-pstack/scripts/isolation.ts:24): The run-owned auth.json is a writable symlink to the operator's daily credential file. Writing junk through $CODEX_HOME/auth.json changes the target while leaving the symlink intact, so assertCodexAuth passes and cleanup deletes only the link; the operator's login remains corrupted. This behavior was reproduced on the reviewed head. - Codex auth symlink permits daily-login corruption (.claude/skills/verify-open-pstack/scripts/isolation.ts:24): The run-owned auth.json points directly to the operator's daily credential file. Writing junk through $CODEX_HOME/auth.json follows this symlink while leaving the link intact, so assertCodexAuth passes and cleanup removes only the link; the daily Codex login remains corrupted and may require reauthentication. - Codex auth symlink permits daily-login corruption (.claude/skills/verify-open-pstack/scripts/isolation.ts:24): The run-owned auth.json points directly at the operator's daily credential file. Candidate code writing to $CODEX_HOME/auth.json follows this symlink and overwrites the source while leaving the link itself intact, so assertCodexAuth passes and cleanup removes only the link. A malicious or faulty exercise can therefore corrupt the operator's daily Codex login and force reauthentication. Also reported for this defect: - Codex auth symlink lets exercises corrupt the daily login undetected (.claude/skills/verify-open-pstack/scripts/isolation.ts:24): Carried v1:2d457389, still present. The run-owned $CODEX_HOME/auth.json is a symlink to the operator's daily auth.json. Any exercise write to that path follows the link and replaces the daily credential, for example a fixture reset, a codex login to another account, or junk bytes. assertCodexAuth checks only the link type and target, and removeCodexHome deletes only the link. The run therefore proceeds to publication while the daily Codex login is damaged. - Codex auth symlink allows daily-login corruption (.claude/skills/verify-open-pstack/scripts/isolation.ts:24): The run-owned auth path is a writable symlink to the operator's daily auth.json. Candidate code writing corrupt bytes to $CODEX_HOME/auth.json follows this link; assertCodexAuth still passes because the symlink itself is unchanged, and cleanup removes only the link. The operator's daily Codex login remains corrupted and may require reauthentication. - Codex auth symlink permits daily-login corruption (.claude/skills/verify-open-pstack/scripts/isolation.ts:24): The run-owned auth.json is a writable symlink to the operator's daily credential file. Candidate code can write corrupt bytes through $CODEX_HOME/auth.json while preserving the symlink, so assertCodexAuth still passes and cleanup removes only the link, leaving the daily login corrupted and requiring reauthentication.
Now at .claude/skills/verify-open-pstack/scripts/isolation.ts:24
| const { HOME, USER, LOGNAME } = process.env; | ||
| if (!HOME || !isAbsolute(HOME)) throw new Error('Real HOME must be absolute'); | ||
| if (!USER || !LOGNAME) throw new Error('Real USER and LOGNAME are required'); | ||
| return { PATH: process.env.PATH ?? '', HOME, USER, LOGNAME, |
There was a problem hiding this comment.
🔴 Critical · Candidate hooks can reuse Keychain-backed GitHub auth
Because candidate commands retain the real HOME and macOS user context, a hook can set GH_CONFIG_DIR=$HOME/.config/gh and invoke the same gh binary used by the publisher. For a normal Keychain-backed gh login, gh auth token or gh api then uses the publisher identity, allowing candidate code to forge live-gate statuses or mutate PR state before review. Also reported for this defect: - Candidate can reuse the publisher's Keychain login (.claude/skills/verify-open-pstack/scripts/io.ts:45): On a normal macOS Keychain-backed gh login, a candidate can override GH_CONFIG_DIR to $HOME/.config/gh and run gh auth token. Real HOME and unrestricted Keychain access therefore expose the publisher identity to candidate hooks, allowing unauthorized status or PR mutations before evidence review. - Candidate hooks can recover the publisher's Keychain gh login (.claude/skills/verify-open-pstack/scripts/io.ts:45-48): Carried v1:43b11b64, still present. isolatedEnv keeps the real HOME and only defaults GH_CONFIG_DIR to a run-owned directory, and candidate commands run unsandboxed. A candidate plugin hook runs as soon as the native session starts, for example a SessionStart hook under plugins/pstack/hooks. It can run GH_CONFIG_DIR=$HOME/.config/gh gh auth token to fetch the Keychain-backed token, then POST live-gate=success or mutate the PR before operator review. - Candidate hooks can reuse the publisher's GitHub login (.claude/skills/verify-open-pstack/scripts/io.ts:45): isolatedEnv preserves the operator's real HOME while providing no filesystem or Keychain isolation. With the publisher authenticated through the standard gh configuration, a candidate SessionStart hook can override GH_CONFIG_DIR to $HOME/.config/gh, recover the Keychain-backed token with gh auth token, and write live-gate or mutate PR state before operator review. The run-owned default GH_CONFIG_DIR does not prevent an untrusted child from selecting the real one. Also reported for this defect: - Candidate hooks can reuse the publisher's Keychain gh login (.claude/skills/verify-open-pstack/scripts/io.ts:45-48): Carried v1:43b11b64, still present. isolatedEnv keeps the real HOME, USER and LOGNAME and only defaults GH_CONFIG_DIR to a run-owned directory. Candidate commands run with no OS boundary. A changed shared hook (plugins/pstack/hooks/**) runs as soon as harness.ts:124 launches the session. It can run GH_CONFIG_DIR=$HOME/.config/gh gh auth token to obtain the Keychain-backed publisher token, then POST live-gate=success or mutate the PR before operator review.
Now at .claude/skills/verify-open-pstack/scripts/io.ts:45
Also affects:
| return { PATH: process.env.PATH ?? '', HOME, USER, LOGNAME, | ||
| TMPDIR: join(state, 'tmp'), TERM: process.env.TERM ?? 'xterm-256color', | ||
| XDG_CONFIG_HOME: join(state, '.config'), XDG_CACHE_HOME: join(state, '.cache'), | ||
| GH_CONFIG_DIR: join(state, '.config/gh'), |
There was a problem hiding this comment.
🔴 Critical · Plaintext gh credentials under real HOME are readable by candidates
Carried v1:a3f6ca75, still present. When gh uses its supported plaintext storage, the oauth_token sits in $HOME/.config/gh/hosts.yml. HOME is the operator's and nothing restricts file reads, so candidate code can read that file whatever GH_CONFIG_DIR says. It can then export GH_TOKEN and write live-gate statuses or PR state as the publisher. Also reported for this defect: - Plaintext gh hosts.yml under real HOME is readable by candidates (.claude/skills/verify-open-pstack/scripts/io.ts:45, .claude/skills/verify-open-pstack/scripts/io.ts:45-48): Carried v1:a3f6ca75, still present. If the publisher uses gh's supported plaintext storage, the oauth_token sits in $HOME/.config/gh/hosts.yml. Because HOME is preserved and there is no filesystem boundary, candidate code can read that file directly whatever GH_CONFIG_DIR says. It can then export GH_TOKEN and write PR state or live-gate as the publisher. Blocking Keychain retrieval does not close this route. Also reported for this defect: - Candidate can read plaintext publisher credentials (.claude/skills/verify-open-pstack/scripts/io.ts:45): Candidate commands retain the publisher's real HOME without a filesystem boundary. With gh's supported plaintext storage, a candidate hook can read $HOME/.config/gh/hosts.yml directly, set GH_TOKEN, and issue authenticated PR or live-gate writes as the publisher; the run-owned GH_CONFIG_DIR does not restrict direct file access. - Plaintext GitHub credentials remain readable (.claude/skills/verify-open-pstack/scripts/io.ts:45): With gh's supported plaintext credential storage, a candidate hook can directly read $HOME/.config/gh/hosts.yml despite the run-owned GH_CONFIG_DIR, extract the OAuth token, and perform publisher-authorized status or PR writes. Preserving the real HOME therefore creates a separate authorization-bypass path from Keychain retrieval. - Candidate sessions can recover stored GitHub credentials (.claude/skills/verify-open-pstack/scripts/io.ts:45-48): isolatedEnv preserves the operator's real HOME without a filesystem boundary. If the publisher uses gh's supported plaintext storage at $HOME/.config/gh/hosts.yml, a candidate hook or native tool can read that file directly despite the run-owned GH_CONFIG_DIR, set GH_TOKEN, and perform PR or live-gate writes as the publisher. This reproduces an authorization bypass from candidate code.
Now at .claude/skills/verify-open-pstack/scripts/io.ts:45
Also affects:
| if (await treeHash(installation.location, ['skills/poteto-mode/scripts/node_modules']) !== installation.treeHash) throw new Error('Installed tree changed during exercise'); | ||
| if (await sourceDigest(workspace) !== installation.sourceHash) throw new Error('Pinned project/plugin source changed during exercise'); |
There was a problem hiding this comment.
🔴 High · Endpoint hashes permit transient source substitution
Source integrity is checked only before and after the interactive session. A candidate hook can replace a broken skill or verifier file with known-good bytes for the exercised invocation and restore the candidate bytes before exit; these final hashes match even though the accepted behavior came from substituted code. Also reported for this defect: - Before/after digests allow transient source substitution (.claude/skills/verify-open-pstack/scripts/harness.ts:132-133): Carried v1:b2d697d5, still present. Provenance is sampled before the session (line 111) and after it, while candidate commands can write the workspace and the installed tree. A candidate hook can swap a broken skill for known-good bytes during the session and restore the candidate bytes before exit. Both digests then match, and the broken exact candidate receives success. - Endpoint hashes allow transient source substitution (.claude/skills/verify-open-pstack/scripts/harness.ts:132-133): Source integrity is sampled before and after the native session, not while behavior executes. A candidate hook can temporarily replace a broken skill or tool with known-good bytes, exercise those bytes, and restore the candidate files before these final checks; both hashes match and the broken candidate can receive success. - Before/after hashes cannot show which bytes executed (.claude/skills/verify-open-pstack/scripts/harness.ts:132-133, .claude/skills/verify-open-pstack/scripts/harness.ts:132): Carried v1:b2d697d5, still present. Provenance is sampled before the session (line 111) and after it (lines 132-133), while candidate commands run with no write restriction. A candidate hook can swap a broken skill for known-good bytes during the session and restore the candidate bytes before exit. Both digests then match, and the broken exact candidate receives success. Also reported for this defect: - Final hashes allow transient source substitution (.claude/skills/verify-open-pstack/scripts/harness.ts:132): Candidate code can temporarily replace a broken skill or tool with known-good bytes during the native invocation and restore the candidate bytes before this final tree check. The before/after digests then match even though the observed behavior came from substituted code, allowing a broken exact candidate to receive success. - Final hashes allow transient source substitution (.claude/skills/verify-open-pstack/scripts/harness.ts:132-133): Source integrity is sampled only before and after the native session. A candidate SessionStart hook can temporarily replace a broken skill with known-good bytes for the exercise and restore the candidate bytes before exit; this final hash then matches, allowing behavior from substituted code to be accepted as the exact candidate. - Final hashes allow transient source substitution (.claude/skills/verify-open-pstack/scripts/harness.ts:132-133): The verifier checks the installation only after the native session. A candidate hook can detect the verifier workspace, temporarily replace a broken skill with known-good bytes for the exercise, and restore the candidate bytes before exit; this final hash then matches and the broken exact candidate receives success. Before/after digests cannot establish which bytes actually executed.
Now at .claude/skills/verify-open-pstack/scripts/harness.ts:132-133
Also affects:
Link existing Codex authentication into every candidate home so Claude setup and runner exercises can dispatch Codex children. Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
There was a problem hiding this comment.
🛑 14 blocking findings
🔴 Critical · Candidate hooks can reuse Keychain-backed GitHub auth
🔴 Critical · Plaintext gh hosts.yml under real HOME is readable by candidates
🔴 Critical · Codex setup exercise rewrites daily ~/.codex files with no restore
🔴 Critical · Codex auth symlink permits daily-login corruption
🔴 Critical · Mid-session refreshed tokens escape any evidence check
🔴 Critical · Retained evidence can contain the daily credential
🔴 Critical · Candidate hooks can modify operator files
🔴 Critical · Setup restoration does not restore file permissions
🔴 High · .mergify.yml exemption lets merge-gate changes auto-pass live-gate
🔴 High · Project-skill proof only runs candidate doctor
🔴 High · Before/after hashes cannot show which bytes executed
🟡 Medium · Publisher revision can describe different loaded code
🟡 Medium · Blank evidence answers fail only after cleanup
🟡 Medium · Publisher-revision gate ignores uncommitted working-tree changes
Check
| const directory = process.env.CLAUDE_CONFIG_DIR ?? join(process.env.HOME!, '.claude'); | ||
| const snapshot: SetupSnapshot = []; | ||
| for (const name of ['pstack-models.md', 'CLAUDE.md']) { |
There was a problem hiding this comment.
🔴 Critical · Codex setup exercise rewrites daily ~/.codex files with no restore
Setup is exercised in both harnesses with the real HOME preserved. snapshotSetup only captures ${CLAUDE_CONFIG_DIR:-$HOME/.claude} pstack-models.md and CLAUDE.md, including for the Codex session (harness.ts:121-122). Per the prior panel finding, the shipped setup skill's Codex branch writes its model sheet and instruction integration to literal ~/.codex paths, not $CODEX_HOME. A shared-path PR such as a plugin.json bump therefore runs $setup-pstack and overwrites the operator's daily ~/.codex setup files with fixture routing. Nothing restores them, and publication proceeds. The same gap applies to literal ~/.claude writes when CLAUDE_CONFIG_DIR is set. Also reported for this defect: - Codex setup writes daily files outside the snapshot (.claude/skills/verify-open-pstack/scripts/isolation.ts:51): snapshotSetup records only CLAUDE_CONFIG_DIR or ~/.claude files, but the shipped Codex setup skill writes literal ~/.codex/pstack-models.md and ~/.codex/AGENTS.md. Because candidate HOME is the real operator home, a Codex setup exercise modifies those daily files; restoring this snapshot and deleting the run-owned CODEX_HOME leaves the daily changes in place.
Also affects:
Withdrawn in the latest review.| const { HOME, USER, LOGNAME } = process.env; | ||
| if (!HOME || !isAbsolute(HOME)) throw new Error('Real HOME must be absolute'); | ||
| if (!USER || !LOGNAME) throw new Error('Real USER and LOGNAME are required'); | ||
| return { PATH: process.env.PATH ?? '', HOME, USER, LOGNAME, |
There was a problem hiding this comment.
🔴 Critical · Candidate hooks can modify arbitrary operator files
Candidate code runs as the operator with the real writable HOME. A PR changing the SessionStart hook can overwrite or delete files such as ~/.ssh/config or ~/.claude/CLAUDE.md; non-setup runs restore nothing, and setup cleanup covers only enumerated files, so the damage persists after verification. Also reported for this defect: - Candidate hooks can overwrite arbitrary operator files (.claude/skills/verify-open-pstack/scripts/io.ts:45): Candidate commands run with the operator's real HOME and ordinary filesystem permissions. A SessionStart hook can overwrite or delete files such as ~/.ssh/config or ~/.claude/CLAUDE.md; non-setup runs restore nothing, and setup restoration covers only selected filenames, so the damage persists after cleanup. - Candidate code can persistently modify operator files (.claude/skills/verify-open-pstack/scripts/io.ts:45): Carried v1:3d8607b8, still present. Candidate processes run as the operator with the real HOME and no filesystem boundary. A PR changing a shared hook can overwrite or delete files such as ~/.claude/settings.json or ~/.ssh/config when the session starts. Non-setup runs take no snapshot, and cleanup removes only run-owned .codex directories, so the damage persists and publication still proceeds. - Candidate hooks can modify operator files (.claude/skills/verify-open-pstack/scripts/io.ts:45): Candidate processes inherit the operator's real HOME without a filesystem boundary. A PR changing a SessionStart hook can overwrite or delete files such as ~/.claude/CLAUDE.md or ~/.ssh/config when the candidate plugin launches; non-setup runs take no snapshot, and cleanup leaves the damage in place.
| await setupFile(path); | ||
| if (bytes === undefined) await rm(path, { force: true }); | ||
| else await writeFile(path, bytes, { mode }); | ||
| } |
There was a problem hiding this comment.
🔴 Critical · Setup restoration leaves weakened file permissions
The mode option to writeFile does not change permissions of an existing file, and the verification loop compares only bytes. If setup changes an original 0600 configuration file to 0644, restoration reports success while it remains world-readable; the reviewed implementation reproduced mode 0644 after restoring the original bytes. Also reported for this defect: - Setup restoration leaves weakened file permissions (.claude/skills/verify-open-pstack/scripts/isolation.ts:66): writeFile's mode option does not change the mode of an existing file. If setup changes an original 0600 CLAUDE.md or pstack-models.md to 0644, restoration rewrites the original bytes but leaves mode 0644; the following verification compares only bytes and can publish success with sensitive configuration newly readable. - Setup restoration leaves weakened file permissions (.claude/skills/verify-open-pstack/scripts/isolation.ts:68): Carried v1:81a445a4, still present. writeFile applies its mode option only when it creates a file. Suppose setup replaces (temp file plus rename) or chmods an original 0600 CLAUDE.md or pstack-models.md to 0644. Restoration rewrites the bytes but leaves mode 0644. The verification loop compares bytes only, so restoration reports success and the run can publish with the daily file left readable by others. - Setup restoration does not restore file permissions (.claude/skills/verify-open-pstack/scripts/isolation.ts:66): writeFile's mode option is ignored when the target already exists. If setup replaces or chmods an original 0600 CLAUDE.md or pstack-models.md to 0644, restoration returns the original bytes but leaves mode 0644; the subsequent verification checks only bytes and can publish success with weakened permissions.
Now at .claude/skills/verify-open-pstack/scripts/isolation.ts:68
Also affects:
| root = await freshRoot(options.output, repository); | ||
| if (options.mode === 'doctor') await doctor(root, undefined, undefined, options.candidate); | ||
| else { | ||
| const publisherRevision = (await command(['git', 'rev-parse', 'HEAD'], { cwd: repository })).trim(); |
There was a problem hiding this comment.
🟡 Medium · Publisher revision ignores dirty working-tree code
publisherRevision records only HEAD, while verifier modules and registry.json are loaded from the working tree. With uncommitted edits to verify.ts, core.ts, or registry.json on the candidate checkout, the SHA equality check still passes and the modified, unreviewed behavior can classify and publish live-gate=success as though it came from the exact candidate. Also reported for this defect: - Publisher-revision gate ignores uncommitted working-tree edits (.claude/skills/verify-open-pstack/scripts/cli.ts:47-48): Carried v1:d9de3416, still present. publisherRevision is only HEAD, but registry.json and all verifier code are read from the working tree. An operator on the candidate SHA with uncommitted edits to core.ts, verify.ts or registry.json passes the verify.ts:95 gate. The comment then records publisher revision = candidate SHA while the modified code classifies and publishes live-gate=success. - Publisher revision ignores uncommitted verifier changes (.claude/skills/verify-open-pstack/scripts/cli.ts:47-49): git rev-parse HEAD identifies only the commit, while imported verifier modules and registry.json come from the working tree. With the candidate SHA checked out but uncommitted edits to classification or publication code, the revision equality check passes and the modified, unreviewed implementation can publish success as though it were the exact candidate. - Publisher-revision gate ignores uncommitted working-tree changes (.claude/skills/verify-open-pstack/scripts/cli.ts:47): publisherRevision is only git rev-parse HEAD, but registry.json and all verifier code are read from the working tree. Scenario: an operator checks out the candidate SHA with uncommitted edits to core.ts, verify.ts or registry.json, either carried over by git checkout <sha> or left from debugging. The verify.ts:95 equality check passes, and the evidence comment records publisher revision = candidate SHA. The modified code still classifies, exercises and publishes live-gate=success.
Now at .claude/skills/verify-open-pstack/scripts/cli.ts:47-48
Also affects:
Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-Authored-By: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
There was a problem hiding this comment.
🛑 19 blocking findings
🔴 Critical · Candidate hooks can reuse Keychain-backed GitHub auth
🔴 Critical · Setup restoration leaves weakened file permissions
🔴 Critical · Plaintext gh credentials under real HOME are readable by candidates
🔴 Critical · Candidate hooks can modify arbitrary operator files
🔴 Critical · Copied credentials survive Codex cleanup outside .codex
🔴 Critical · Codex auth symlink lets an exercise corrupt the daily login undetected
🔴 Critical · Transient refreshed credentials can escape into evidence
🔴 Critical · Retained evidence accepts known live credentials
🔴 Critical · Candidate sessions can recover the publisher's gh authentication
🔴 Critical · Restoration stops at first unrestorable path, skipping later files
🔴 Critical · Setup snapshot is memory-only; failed restoration loses originals
🔴 High · Endpoint hashes permit transient source substitution
🔴 High · .mergify.yml changes auto-pass the live gate
🔴 High · Project-skill proof exercises only doctor
🔴 High · Candidate registry can exempt its own changes
🟡 Medium · Blank evidence answers fail only after the live sessions
🟡 Medium · Publisher revision can describe different loaded code
🟡 Medium · Mistyped evidence path aborts the run instead of re-prompting
🟡 Medium · Publisher revision ignores dirty working-tree code
Check
|
|
||
| export async function removeCodexHome(home: string): Promise<void> { | ||
| await assertCodexAuth(home); | ||
| await rm(await codexDirectory(home), { recursive: true }); |
There was a problem hiding this comment.
🔴 Critical · Copied credentials survive Codex cleanup outside .codex
Cleanup recursively removes only the run-owned .codex directory. A candidate can copy the symlinked auth.json to dirname($CODEX_HOME)/copied-token or another retained state path before exit; the assertion still passes, cleanup succeeds, and the live credential remains in the preserved run root. This exact copy-and-cleanup scenario was reproduced. Also reported for this defect: - Cleanup preserves credential copies outside .codex (.claude/skills/verify-open-pstack/scripts/isolation.ts:37): removeCodexHome deletes only the .codex directory. A candidate can copy the symlinked auth.json into its workspace or another file under the retained state directory; the auth link still passes validation, cleanup succeeds, and the live credential copy remains in state preserved through merge and archival.
| for (const path of (await this.review('Reviewed artifact paths inside output, one or more separated by commas?')).split(',')) { | ||
| artifacts.push(await retainedFile(receipt.artifactRoot, path.trim())); |
There was a problem hiding this comment.
🟡 Medium · Mistyped evidence path aborts the run instead of re-prompting
Transcript answers (line 134) and artifact answers go straight to retainedFile with no retry. Three inputs make retainedFile throw ('within output' or ENOENT): a trailing comma ('a.txt,' splits to ['a.txt','']), an empty answer, or a typo. exercise() then rejects and verify() publishes live-gate=failure, discarding every completed native session for a correct candidate. Unlike surface, action and observed, nothing re-prompts these answers.
| return { PATH: process.env.PATH ?? '', HOME, USER, LOGNAME, | ||
| TMPDIR: join(state, 'tmp'), TERM: process.env.TERM ?? 'xterm-256color', | ||
| XDG_CONFIG_HOME: join(state, '.config'), XDG_CACHE_HOME: join(state, '.cache'), | ||
| GH_CONFIG_DIR: join(state, '.config/gh'), |
There was a problem hiding this comment.
🔴 Critical · Candidate sessions can recover the publisher's gh authentication
isolatedEnv preserves the operator's real HOME without a filesystem or Keychain boundary. A candidate SessionStart hook can point GH_CONFIG_DIR back to $HOME/.config/gh to reuse a Keychain-backed login, or read a plaintext hosts.yml, then call gh api to forge live-gate or mutate the PR as the publisher. Also reported for this defect: - Candidate sessions can recover publisher GitHub credentials (.claude/skills/verify-open-pstack/scripts/io.ts:45): isolatedEnv preserves the publisher's real HOME without a filesystem or Keychain boundary. A candidate hook can set GH_CONFIG_DIR=$HOME/.config/gh to reuse a Keychain-backed gh login, or read a plaintext hosts.yml directly, then invoke gh api to forge live-gate statuses or mutate the PR before operator review.
Also affects:
| if (options.mode === 'doctor') await doctor(root, undefined, undefined, options.candidate); | ||
| else { | ||
| const publisherRevision = (await command(['git', 'rev-parse', 'HEAD'], { cwd: repository })).trim(); | ||
| const registry = validateRegistry(JSON.parse(await readFile(join(import.meta.dir, '../features/registry.json'), 'utf8'))); |
There was a problem hiding this comment.
🔴 High · Candidate registry can exempt its own changes
The classifier loads registry.json from the verifier checkout being reviewed. A PR can remove .claude/skills/verify-open-pstack/** from project and add it to nonRuntime; its registry change then classifies itself as no-runtime, bypassing the publisher-revision check, both native exercises, and all observations before publishing success.
| for (const { path, bytes, mode } of snapshot) { | ||
| await setupFile(path); |
There was a problem hiding this comment.
🔴 Critical · Restoration stops at first unrestorable path, skipping later files
restoreSetup walks the 12 snapshot entries in order and throws on the first non-regular or unwritable entry. Every later entry stays candidate-modified even when it is a regular, restorable file, for example CLAUDE.md after an obstructed pstack-models.md, or all ~/.codex and CLAUDE_CONFIG_DIR files. The cleanup retry aborts at the same entry, and the error names only the first path, so the remaining daily files stay silently modified.
| const raw = join(home, 'surface.raw'); | ||
| console.log('Exercise every requested feature from the native surface, using the maintained feature map. Exit when done.'); | ||
| const setup = requiredFeatures(receipt, harness).some(feature => feature === 'setup' || feature === 'skill-invocation:setup-pstack'); | ||
| if (setup) this.setup = await snapshotSetup(); |
There was a problem hiding this comment.
🔴 Critical · Setup snapshot is memory-only; failed restoration loses originals
The only copy of the operator's original setup files is this.setup in process memory. It is never written to the private run root, although setup.md requires retaining snapshot results there. Scenario: a candidate leaves a non-regular or unwritable file at a snapshotted path, for example replacing ~/.claude/CLAUDE.md with a symlink or making pstack-models.md a directory or read-only. restoreSetup throws in exercise's finally and again in verify's single cleanup. The process then exits, and the original daily CLAUDE.md or model-sheet bytes are permanently lost.
Step-back result 3: PR #111 after reviewed head 9 (dc2c0d4)The advisor was GPT-6 Astra, read-only, in a checkout of Recommendation: make three focused corrections, run the required checks and an exact-head live proof, then merge. No waiver. The PR is not being stopped. Why the finding count rose. Thirteen of the nineteen findings are carried with their prior rulings, most of them under the malicious-candidate premise that decision 3 excludes. Two new defects came in with the no-copy redesign ( Final changes
Maintainer decisionThe base ( |
|
Exact-head waiver for Accepted by the maintainer as known limitations. These three are low likelihood, and each fix was estimated at about three review rounds:
Malicious candidate code. Step-back decision 3 sets a collaborator-authored threat model, and
Private evidence. Step-back decision 2 keeps raw evidence private and never publishes it:
Other carried rulings:
Fails loudly with an honest failure:
#111 merges through the manual exact-head live gate, because the base has no verifier registry. |
Live evidence: Open Pstack
|
Operator note for the manual exact-head live gate (head dc2c0d4)The maintainer authorized this gate after step-back result 3. I ran it on the operator's Mac. The verifier itself was run from a checkout of exactly Observation, not repaired, recorded here:
|
Merge Queue Status
This pull request spent 9 seconds in the queue, including 1 second running CI. Required conditions to merge
|
Bring in ericlitman/open-pstack main at 67c08fa: the repository-local verify-open-pstack skill (ericlitman#90), its AGENTS.md guidance, and static checks. No plugin files change, so 1.5.1-local.9 stays as installed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # .gitignore
Closes #90
What changed
Add one repository-local verification skill shared between Claude Code and Codex. It pins a supplied PR to an exact head, maps changed runtime files to maintained features, requires isolated candidate installations and operator-reviewed native-surface evidence, and publishes a SHA-bound evidence comment/live-gate before draft readiness. Unknown paths, missing evidence, unsupported isolation, and head movement fail closed; detected publication races compensate on the original SHA. Docs/CI-only changes have an explicit no-runtime-change lane.
The verifier has its own non-shipped Bun package and tests. CI reaches its locked install, tests, and strict typecheck through
tests/skill-collision-repro.sh, including static-only execution. No.github,plugins/pstack, or.mergify.ymlfiles change againstorigin/main, per the October 2 amended build condition (issue comment 5962381342). The gross diff is approximately 1,150 added lines including the generated lockfile, below the approved 1,500-line estimate.Verification
Actual sandbox results at
20afcc2d0f9401fcebb778e692014cde6368b708:bun install --frozen-lockfile && bun run test && bun run typecheck: 158 tests passed, 0 failed, 648 assertions; all three strict TypeScript projects passed.PSTACK_STATIC_ONLY=1 bash tests/skill-collision-repro.sh: all static invariants passed; independent locked install, 33 helper tests (353 assertions), and strict helper typecheck passed.claude plugin validate --strict plugins/pstack: validation passed; all four manifests parsed.bash -n tests/skill-collision-repro.sh,git diff --check, andgit diff --exit-code origin/main -- .github plugins/pstack .mergify.yml: passed.Live evidence:
Pending operator Mac proof. This PR remains draft. No live-gate was posted by the builder. The previously approved Mac probe is capability evidence only, not successful live installation/exercise.
Operator handoff
Review the verifier before running its publishing command from a trusted checkout of this exact head. In both isolated native harness sessions, invoke the discovered project skill and run only its child doctor; retain the invocation trace and doctor artifact. Do not recursively publish from a harness session.
Retain
$SESSION/probe/doctor.json,$EVIDENCE/doctor.json,$EVIDENCE/receipt.json, and reviewed transcripts/artifacts. The operator run must produce an exact-head evidence-comment URL andlive-gate=successtargeting that comment, followed by draft-to-ready ordering. Every new head requires a fresh run. Mergify queues automatically onceverify,Unfret, andlive-gatepass and the PR is ready; no manual queue submission or hand-posted replacement status. The builder does not publish, mark ready, queue, or merge.Findings
None
Unfret rebuttals