Skip to content

Complete live Codex capability evidence for baseline 2.3 #7

Closed

Description

@KeyffMS

Completed live qualification and evidence import

Full run #27 completed C01-C16 successfully on the authenticated self-hosted runner: Codex CLI 0.153.4, gpt-5.6-sol, Debian 13. Actual tested source: a9cdcdc1e0cad70e88b60869e75e4166046dd306.

PR #36 merged the exact sanitized packages, original archive and provenance into main at c7c28cd46a07a353fdd88356c3fe6f8ff1fd6f21. The source SHA in evidence remains the tested SHA, not the import commit. The full archive is preserved under qualifications/34140846679/; the original #25 archive remains unchanged. Strict production evidence validation passes for the imported tree. The original issue tracked baseline 2.2; the completed qualification uses the subsequently adopted baseline 2.3.

Required

  • C01 — repository skill discovery
  • C02 — explicit vs implicit skill activation
  • C03 — PlanAnvil flat direct-child contract
  • C05 — explicit immutable reviewer handoff
  • C06 — PreToolUse supported paths and deterministic postconditions
  • C07 — unsafe Git guard decisions and postconditions
  • C08 — actual missing-checkpoint stop and finite repaired completion without timeout
  • C09 — checkpoint → two automatic compactions → recovery and completed turn
  • C10 — startup and SessionStart(source=compact) recovery context; PostCompact advisory
  • C11 — project instruction scope and precedence
  • C12 — project_doc_max_bytes truncation fixture
  • C13 — SubagentStart context/startup semantics in the explicitly allowed project-scoped non-ephemeral fallback
  • C14 — planning isolation in an actual Codex run
  • C15 — fresh blind reviewer with seeded defects
  • C16 — Git capability matrix across supported Codex permission modes

C04 is informational/non-gating in baseline 2.3 and was also REPRODUCED.

Evidence and retained limits

Each capabilities/CXX contains the exact fixture, prompt, configuration, run command, expectations, sanitized actual results, evaluation and SHA-256 hashes from the same complete full run. Archive integrity covers 166 listed files plus the manifest, including three hidden fixtures. No timeout=true is recorded in the current actual evidence. C13's ephemeral parent-thread limitation remains explicitly documented; this closes the baseline semantic capability requirement, not that upstream transport limitation.

No product/runtime/security or release-validator change was made during import. Qualification in this tested configuration does not guarantee all future model behavior or untested versions. Signed production publication remains a separate action; no tag or GitHub Release was created by this closure.

Historical blocker — 2026-08-28

The original qualification environment had no authenticated Codex runtime or codex executable. Deterministic CI was not accepted as live evidence. That blocker was subsequently resolved through the controlled self-hosted runner. Historical run #25 met its narrow C08 assertion but timed out later; its unedited record is retained. Full #27 supplies the required finite C08 completion.

Activity

  1. KeyffMS commented on Aug 28, 2026

    @KeyffMS
    OwnerAuthor

    Qualification baseline 2.2 was merged to main as 0e1091fc4168671215ebcd89c92db702ea33ccdd. Post-merge GitHub Actions run #26 completed successfully on Ubuntu, macOS and Windows. Deterministic qualification is therefore green; the remaining blocker in this issue is exclusively the required authenticated live Codex evidence packages listed above.

  2. KeyffMS commented on Aug 29, 2026

    @KeyffMS
    OwnerAuthor

    Non-Codex preparation is complete on main at 531492a20554d5c6687f57cf302e2c6f64ecda12. Post-merge Actions run #43 passed all 7 jobs. The repository now includes prepared C01-C16 evidence templates, a safe materializer/validator/rehash workflow, distribution lifecycle tooling, release gates, deterministic archive building, and docs/CODEX_SANDBOX_RUNBOOK.md. The remaining work in this issue is only execution of the prepared live capability cases in an authenticated current Codex sandbox and committing sanitized REPRODUCED evidence.

  3. KeyffMS commented on Sep 7, 2026

    @KeyffMS
    OwnerAuthor

    Qualification closure update (2026-09-07):

    • PR Preserve live qualification evidence and close finite C08 stop/repair testing #34 is merged at ba3f56a644a385c8aa2e8e5c9c934999184cc282. The complete, source-bound baseline 2.3 evidence from full run Fix Codex runner variant dispatcher #25 (34060321283) is now committed, including the unchanged original archive, hashes and C08/C13 caveats.
    • The finite C08 missing-checkpoint stop / same-run checkpoint repair / positive completion implementation is on main. PR CI #118 and post-merge CI #119 passed all eight jobs, including the real Codex 0.153.4 loopback tests. These are not new live-model evidence.
    • I dispatched a new full qualification Add precision Codex runner probes #26 (34109662176) on the repaired main. The local self-hosted runner's administrator hook rejected the automated initiating actor with exit 77: PlanAnvil runner policy denied this job: initiating actor is not allowed. No Codex trial executed and no new capability evidence was produced. The allowlist and runner policy were not changed.
    • An account already allowed by that runner must start a NEW PlanAnvil Codex qualification / main / full run through Run workflow, not rerun the denied automated job. The production validator requires one complete successful full-run archive containing finite C08 proof; a partial c08 diagnostic cannot be spliced into the old Fix Codex runner variant dispatcher #25 archive. PR Document the allowed live-test handoff and the complete-archive publication gate #35 clarifies this operational and archive boundary without runtime changes.

    Keep publication closure open until that actual full result is reviewed, imported with its executed source SHA and immutable provenance, and strict release validation passes. No tag or release has been published.

  4. added a commit that references this issue on Sep 8, 2026
    c7c28cd
  5. changed the title [-]Complete live Codex capability evidence for baseline 2.2[/-] [+]Complete live Codex capability evidence for baseline 2.3[/+] on Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions