Skip to content

Add Codex backend to skill eval harness - #197

Draft
amd-mkarvir wants to merge 2 commits into
amd:mainfrom
amd-mkarvir:feat/codex-eval-backend
Draft

Add Codex backend to skill eval harness#197
amd-mkarvir wants to merge 2 commits into
amd:mainfrom
amd-mkarvir:feat/codex-eval-backend

Conversation

@amd-mkarvir

@amd-mkarvir amd-mkarvir commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

What changed

  • Add --agent codex to the existing routing and behavior eval runner.
  • Install only the skills under test in a temporary, isolated CODEX_HOME.
  • Parse Codex JSONL for routing and behavior grading using observed CLI event shapes.
  • Add an advisory Codex routing workflow that skips when OPENAI_API_KEY is unavailable.
  • Reuse the routing plugin installation for preflight instead of installing it twice.
  • Treat retryable Codex error events as non-terminal and stop on turn.failed.
  • Add a sanitized live JSONL fixture and regression tests for parsing and process termination.

Compatibility

Claude remains the default backend. The existing required evals gate is unchanged. Codex results remain advisory while a baseline is established.

Validation

  • 91 unit tests passed.
  • 68 eval cases across 8 skills and 5 shared negatives passed structural validation.
  • A live read-only plugin activation was captured with Codex CLI 0.150.0-alpha.12.2 and gpt-5.4.
  • The live run represented activation as item.started / item.completed with a nested command_execution reading the installed SKILL.md; it did not emit skill_invocation.
  • eval/fixtures/codex-exec-skill-read.jsonl preserves the live event and field structure with identifiers, paths, prose, output, and usage values sanitized.
  • The temporary plugin and marketplace were removed after capture.
  • git diff --check and the public-content credential scan passed.

Fork pull-request workflows do not receive repository secrets, so the advisory Codex job skips cleanly when OPENAI_API_KEY is unavailable.

@amd-mkarvir
amd-mkarvir force-pushed the feat/codex-eval-backend branch from 6773d2b to 7703ebf Compare September 1, 2026 06:05
@danielholanda

Copy link
Copy Markdown
Collaborator

We are significantly refactoring our test harness this week. My recommendation is to not do this work here now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants