Skip to content

test: randomize SDK tests by default - #476

Open
mldangelo-oai wants to merge 7 commits into
mainfrom
mdangelo/codex/fix-randomized-test-isolation
Open

test: randomize SDK tests by default#476
mldangelo-oai wants to merge 7 commits into
mainfrom
mdangelo/codex/fix-randomized-test-isolation

Conversation

@mldangelo-oai

@mldangelo-oai mldangelo-oai commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

Run SDK tests in a reproducible random order and remove shared-mock and timing assumptions that can make unrelated test groups fail.

Changes

  • Enable Bun's randomized test order in the SDK configuration, shared by the existing package command, direct Bun runs, and Windows CI shards. Replay failures with pnpm run test --seed <reported-seed>.
  • Isolate the file-replacement regression's persistent filesystem mock in the existing subprocess helper and verify that the replacement hook ran.
  • Check macOS sandbox availability independently of generated scan permissions. Keep configuration validation active everywhere and filesystem-enforcement assertions active on capable hosts.
  • Synchronize the parallel ChatGPT scan test with both mocked scans reaching the barrier. Remove its five-second timer and release the barrier if either run fails, while keeping concurrency and configuration-isolation assertions intact.

Testing

Using Bun 1.3.14:

  • pnpm run test at a6b70bd0 (generated seed 3113991204): 1,294 passed, 11 expected skips, 0 failed. The filesystem-enforcement test ran and passed outside the nested macOS sandbox.
  • pnpm run test --seed 12345 at the same head: 1,294 passed, 11 expected skips, 0 failed.
  • Focused concurrency, mock-isolation, configuration, and preflight checks: 49 passed, 1 expected nested-macOS sandbox skip.
  • node scripts/run-windows-ci-tests.mjs 1 on macOS: 5 passed with generated seed 2503246880, including the parallel-scan regression.
  • pnpm run types, pnpm run format, and git diff --check: passed.

Risk and rollout

Test-only changes. Runtime permissions, credential-home checks, and production behavior are unchanged. Tests continue to fail on unexpected sandbox or credential errors. No CI jobs, alternate test scripts, or dependencies are added. Current-head cross-platform CI and repository-supported review remain required before merge.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

Newly authored material was reviewed for public disclosure. Existing automated review comments contain access-restricted report references; maintainer cleanup remains outstanding.

@github-actions github-actions Bot added the skip-release-notes Omit internal changes from generated release notes label Aug 16, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current head, 5e4e01444f1ee5807af03190153942e2a9a71072.

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: 5e4e01444f

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 5e4e01444f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai
mldangelo-oai marked this pull request as ready for review August 16, 2026 04:36

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current head, dbe870e972198be66f816ceb27e7908a58745438.

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: dbe870e972

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: dbe870e972

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai mldangelo-oai changed the title test: stabilize randomized SDK test runs test: randomize SDK tests by default Aug 16, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current head, a96cf76ad78b89e9c97285228fee468b6dcb347c.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: a96cf76ad7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: a96cf76ad7

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current head, d9cdf42e13a49d98fd51c19b71d7dbb5b5627113.

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: d9cdf42e13

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: d9cdf42e13

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ianw-oai
ianw-oai previously approved these changes Aug 16, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current head, a6b70bd0ef9c8c127afce34439e5a1ac0ff8a03f.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: a6b70bd0ef

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: a6b70bd0ef

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-release-notes Omit internal changes from generated release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants