Skip to content

fix(cli): return safe structured scan failure details - #463

Draft
mldangelo-oai wants to merge 7 commits into
mainfrom
mdangelo/codex/structured-scan-failure-output
Draft

fix(cli): return safe structured scan failure details#463
mldangelo-oai wants to merge 7 commits into
mainfrom
mdangelo/codex/structured-scan-failure-output

Conversation

@mldangelo-oai

@mldangelo-oai mldangelo-oai commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Return privacy-safe, machine-readable errors when scans or saved scan reruns fail in JSON or JSONL mode.

Addresses the structured-output request in #375.

Changes

  • Preserve SCAN_FAILED and SCAN_REPLAY_UNAVAILABLE error envelopes on structured stdout, including failures finding the latest completed scan.
  • Render a closed JSON/JSONL failure payload so incidental framework metadata cannot enter the error contract. Preserve the normal and --full-output envelope shapes.
  • Use fixed messages for authentication, connectivity, timeout, cost-limit, protected-directory, local-input, and cybersecurity-policy failures. Keep upstream details out of structured errors.
  • Preserve the local origin and detailed diagnostics of knowledge-base and plugin setup errors, including their cause and cancellation behavior. Keep installer subprocess failures distinct from local selection, ZIP manifest, and marketplace-copy failures.
  • Preserve detailed stderr diagnostics, partial-output guidance, plain-text behavior, and existing exit codes, including interruption codes.
  • Cover these paths with injected CLI dependencies and synthetic credentials and identifiers.

Testing

  • bun test --timeout 30000 --only-failures tests-ts/runtime.test.ts tests-ts/knowledge-base.test.ts tests-ts/cli.test.ts tests-ts/cli-workbench.test.ts tests-ts/cli-publish.test.ts — 330 passed, 7 skipped, 0 failed, 3,691 assertions (Bun 1.3.14).
  • bun test --timeout 30000 --only-failures tests-ts/api.test.ts --test-name-pattern 'validates knowledge-base documents before initializing the runtime|cleans up knowledge-base documents when a scan fails|attempts both preparation cleanups and preserves the preparation and cleanup failures' — 3 passed, 0 failed, 13 assertions.
  • bun test --timeout 30000 --randomize --seed 12345 --only-failures ./tests-ts with JUnit reporting — 1,299 passed, 11 skipped, 0 failed, 9,500 assertions on 301e5415 (Bun 1.3.14). The first attempt failed the existing interactive-login cleanup test; that unchanged test passed in isolation, followed by a clean complete rerun with the same seed and original limits.
  • pnpm run types — passed.
  • pnpm exec prettier --check src/cli.ts src/knowledge-base.ts src/runtime.ts tests-ts/cli.test.ts tests-ts/knowledge-base.test.ts tests-ts/runtime.test.ts — passed.
  • git diff 5d1afcd312933121e36dc892dcb12e8d2e3e1de3 HEAD --check — passed.

Exact-head source rereview, three native code-review passes, and a separate source verifier found no remaining issues at 301e54159886185274b956d18ac84756477100cc. Fresh pushed-head standard SDK, cross-platform, and package CI remain pending.

Risk and rollout

Structured scan failures now return a safe error envelope instead of empty stdout. They remain complete and parseable when token-count or truncation flags are supplied; successful-result and discovery formatting are unchanged. Local knowledge-base and plugin setup errors use the existing ConfigurationError type, with underlying failures available through cause; installer errors are unchanged. Detailed stderr may still contain local diagnostic context. This does not change scan execution, model moderation, refusal policy, account permissions, or partial-result sealing.

Public disclosure review

Newly authored material was reviewed for public disclosure. Existing automated review comments contain access-restricted report references; maintainer cleanup remains outstanding.

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

@github-actions github-actions Bot added the bug Something isn't working label Aug 15, 2026
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 9c71f34d02

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: 9c71f34d02

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current head, 301e54159886185274b956d18ac84756477100cc.

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: 301e541598

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 301e541598

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant