Repository navigation
feat(workflows): default automatic Sol workflow to Opus 5.5 and GPT-6.1 Sol - #29
Open
tylerwillis wants to merge 10 commits into
Open
tylerwillis wants to merge 10 commits into
tylerwillis wants to merge 10 commits into
Conversation
added 10 commits
October 27, 2026 22:33
Default new 2119-auto-sol tasks to Claude Opus 5.5 and its first final reviewer to GPT-6.1 Sol. Preserve task overrides, the second reviewer, and test-honesty defaults. Update the existing specification and behavioral expectations for task creation and rendered review skills. Pin Codex images to 0.159.1: a same-account model probe is rejected by 0.144.4 but succeeds on 0.159.1. Verify both Linux release asset names, strict parsing of generated config, and real reviewer identity evidence. Keep historical auth-refresh claims explicitly tied to the older CLI. Validation: 209 focused tests; Ruff, formatting, mypy; clean-wheel smoke; fresh Fable review and four killed default/order mutations. Full suite: 3823 passed, 5 skipped, one native-Claude proxy fixture failure reproduced on unmodified HEAD. The formal gate still fails existing coverage findings; this local checkpoint is not release approval. Linux interactive hooks, resume and token refresh were not requalified on the new CLI.
Codex 0.159.1 can answer prompts as a standalone executable, but its default shell tool needs codex-code-mode-host. Installing only the main binary makes a task appear ready and then fail when it attempts real work. Install the complete official Linux package under /opt/codex, preserve its relative resources, and expose the entrypoint through a replaceable symlink. Check that the CLI and shell helper are executable before replacing the link. The layer remains idempotent when both harness and workflow tiers install it. Update the audited layer fixture and distinguish synthetic refresh validation from observations involving a real provider. Validation: 48 harness/layer tests; exact production layer built twice; missing/non-executable CLI and helper rejected; unprivileged ARM64 CLI/helper smoke; x86_64 CLI/helper smoke under emulation. Manual ARM64 qualification exercised shell execution, hooks, MCP handshake, model identity and interactive resume. Native refresh/reload used synthetic credentials and a loopback endpoint; real provider token rotation was not attempted.
The 2119-auto-sol workflow now starts development on Claude and performs its reviews on Codex. Make both credential requirements visible in the workflow picker so selecting it does not hide a second harness prerequisite. Correct the shared workflow default-pair comment: built-in workflows may now supply a tuned pair. This changes descriptive guidance only; task selection, explicit overrides and persisted task choices keep their existing behavior. Validation: Ruff, strict mypy and independent code review.
String assertions alone could accept broken mouse dispatch or a binding that never sent selected bytes to the clipboard command. Exercise actual SGR drag and double-click input through a PTY in both emacs and vi copy modes. Require exact selected bytes in the paste buffer and capture command, and prove drag copy happens on release. Use unique socket directories, temporary HOME/config paths and a clipboard capture command. Verify the production session constructor excludes the operator config with a negative control, repairs existing defaults, and resolves clipboard precedence on Darwin and other supported platforms. Validation: 32 focused tests, 45 repeated gesture/isolation cases, and six narrow in-memory mutations rejected. Fresh independent requirement reviews confirm the observable clipboard and configuration contracts.
Capture authentication logs at DEBUG before accepted and rejected requests so low-level token leakage cannot pass unnoticed. Compare forbidden repo PATCH fields against the complete request schema, reject malformed values and both terminal setup states, and verify persistent repo state after every denial. Exercise encoded REST targets and real MCP resource requests across self, descendant, unrelated and missing tasks. Assert the authenticated subject is fixed by its capability, normalized targets are scoped correctly, and unsupported encoded forms fail closed without changing stored content. Validation: 247 focused tests, five repetitions of the changed cases and four narrow in-memory mutations rejected. Fresh independent Fable review passed all 38 affected credential-scope requirements. No authorization behavior or credential format changed.
A single-dependency test cannot distinguish gating on any incomplete dependency from gating only when all are incomplete. Use two dependencies in a custom workflow and finish them in both possible orders. Both task GET and list responses must stay gated after the first completion and become queued only after the second custom terminal state is reached. Validation: both parameter cases pass; separate mutations changing each service composition path from any to all are rejected. Independent 2119 review passed the dependency-readiness requirements.
…tion Pin the literal interrupted-turn prompt in the expected argv instead of importing the production constant, which allowed an incorrect prompt to change implementation and expectation together. Also assert that user-turn resume omits the prompt. Validation: harness suite passes; changing the prompt literal or appending it on a user turn is rejected by targeted in-memory mutations. Independent 2119 review passed the resume contract.
Native MCP health checks may retry or negotiate protocol versions. Require nonempty target traffic with the exact endpoint and bearer credential while allowing multiple correct requests. Capture every HTTP method at the proxy, keep the positive control that proves proxy use without bypass, and require zero proxy requests once production bypass values are applied. Set explicit proxy variables and disable unrelated optional CLI traffic in the disposable environment. Cover runner bypass emission both with and without a repository environment file so the no-file path is also checked when the real CLI integration test is unavailable. Validation: native CLI tests passed ten repeated cases; all 37 local-runner tests passed; disabling bypass generally or only for the no-file path is rejected by narrow in-memory mutations. No real model inference is used.
Store the passing review records needed to reproduce the requirements gate from a clean checkout. Reuse existing reviews only where their content hashes still match; record fresh independent Fable judgments for repaired or stale evidence. Preserve tracked audit history and omit superseded untracked records. The records explain concrete rejected counterexamples for clipboard behavior, credential boundaries, dependency readiness, session resume and proxy routing. They do not change requirements, remove coverage or waive failing checks. Validation: pinned rfc2119 0.7.0 check, with clean-checkout verification before publication.
The repository-policy test incorrectly required coveredCount to equal the number of all requirements. The gate legitimately counts unannotated MAY and SHOULD statements in the total without treating them as missing mandatory coverage; the current corpus has four such advisory items. Require zero uncovered requirements and a nonempty, bounded covered count. Add an unannotated MAY to the mixed-grammar fixture and pin six discovered items, five covered MUSTs and no uncovered obligations. Canonical namespace, legacy verdict and mandatory coverage checks remain intact; no enforcement setting or requirement is weakened. Validation: the policy module passes six tests with two migration-window skips. Removing a MUST annotation and incorrectly counting the optional item as covered are each rejected by targeted in-memory mutations. Eight fresh independent Fable reviews passed the affected requirements; their verdicts are included so a clean checkout can reproduce the gate.
tylerwillis
marked this pull request as ready for review
October 28, 2026 06:32
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use Claude Opus 5.5 for development and GPT-6.1 Sol for the first final review in
2119-auto-sol. New tasks inherit the workflow's paired defaults; explicit overrides and persisted tasks retain their selections. The second reviewer and test-honesty reviewer remain GPT-5.6 Sol. The workflow picker now names both credential requirements.GPT-6.1 Sol requires the newer pinned Codex runtime. Install the complete official 0.159.1 Linux package: the standalone executable can answer prompts but fails shell execution because it lacks
codex-code-mode-host. Preserve companion resources, validate both executables before linking, and support repeated installation across harness/workflow image layers.The requirements gate also exposed missing regression coverage. Tests now exercise real isolated mouse gestures, DEBUG-level credential leakage, repo-edit and encoded-target scope boundaries, mixed dependency readiness, literal resume prompts, and native MCP proxy routing. Current independent review records are committed so a clean checkout can reproduce the gate. Most changed files are those review records; the application change remains bounded to workflow defaults, runtime installation and guidance.
Validation:
claude-opus-5-5in CLI response metadata. Native refresh and shared-file reload passed with synthetic credentials. Real-provider token rotation and a full x86_64 model workflow were not tested.No release is triggered by this PR's validation procedure. Merge remains subject to the repository's required approving GitHub review.