test(e2e): native Messages on Sonnet passes; drop the OSWE-278 known failure (MASTRA-768)

[ericlitman]

Linear: MASTRA-768

OSWE-278 is fixed by the v8 upgrade, so the live suite's known-failure test now fails by design. This PR turns it into an ordinary passing test and deletes what existed only to carry the known failure.

Evidence: a controlled A/B on studio2 (2026-10-03), same account, same request: native non-streaming POST /v1/messages on claude-sonnet-4-6.

• 7.3.17: 429 {"type":"error","error":{"type":"rate_limit_error","message":"Error"}}.
• v8.0.13-mob.1: 200, a normal message.

The full record is on OSWE-278 (closed) and MASTRA-766.

Changes:

• The test is now live: native Messages on claude-sonnet-4-6 completes. It asserts 200, the trace header, type: message and stop_reason: end_turn.
• Deleted from live(): the known parameter, whose only caller was this test, along with its clone-and-match handling. Deleted from the test: the isOswe278 predicate. Any 429 or 5xx is now plainly infrastructure (exit 3).
• The timeout comment no longer cites OSWE-278. The 60 s bound keeps its other reason: it stays under the runner's 120 s test timeout.

Size: e2e/lib/live.ts +6/−15; e2e/tests/live.e2e.ts +15/−33.

Verification:

• oxfmt --check and oxlint --deny-warnings are clean.
• Hermetic: 15 of 15.
• Live with the URL unset: exit 2.
• Live on studio2 against v8.0.13-mob.1 at 42f41aa: 7 of 7, three runs in a row.

Live report (attached evidence, per review nit 3)

Live report from studio2, run 1 of 3, against v8.0.13-mob.1: commit 42f41aa7, dirty False, exit 0, summary {"discovered": 22, "selected": 7, "executed": 7, "passed": 7, "failed": 0, "flaky": 0, "skipped": 15}.

🤖 Generated with Claude Code

[linear-code]

OSWE-278 Claude OAuth native Messages Sonnet returns 429 while CLI and Responses work

Context

During Studio2 CLIProxyAPI Claude OAuth setup on 2026-09-25, a built-in -claude-login credential loaded successfully on CLIProxyAPI v7.3.17. /v1/models advertised 17 Claude model IDs. The proxy remained running, and its existing Codex Responses route still returned HTTP 200.

A minimal authenticated POST /v1/messages using claude-sonnet-4-6, anthropic-version: 2023-06-01, one user text message, and max_tokens: 64 returned HTTP 429 rate_limit_error: Error. claude-sonnet-5 returned the same result. The proxy log identified these as upstream Claude execution failures; a proxy restart did not change the result.

Control probes succeeded with the same account and proxy: direct Claude Code CLI using exact claude-sonnet-4-6 returned OK; Claude Code CLI routed through CLIProxyAPI returned OK; proxy /v1/responses with claude-sonnet-4-6 and claude-opus-5-5 returned HTTP 200; minimal native /v1/messages with claude-haiku-4-5-20251001 returned HTTP 200. The model list alone therefore overstates what the minimal native Messages shape can prove. No credentials, OAuth codes, or private prompt data are included here.

Sources: CLIProxyAPI built-in Claude OAuth documentation, v7.3.17 release, studio2 service com.router-for-me.cliproxyapi and its sanitized logs. Related: OSWE-11 (live subscription-backed API compatibility suite), OSWE-77 (proxy credential/upgrade runbook).

Acceptance criteria

• Reproduce and classify the Sonnet native Messages 429 against the same model and account through the direct Claude CLI, Claude CLI via proxy, and proxy Responses routes, using only synthetic prompts and redacted evidence.
• Determine whether this is a supported request-shape requirement, a built-in CLIProxyAPI configuration choice, a model-specific upstream limit, or an upstream bug. Record the evidence and exact CLIProxyAPI version.
• If a supported built-in setting resolves it, apply that setting and prove native Messages Sonnet plus existing Claude Code, Responses, Haiku, and Codex routes still work. Do not add proxy code or new capabilities.
• If the behavior is an upstream limitation, document the working endpoint/client shapes and make the model availability claim precise for operators.

MASTRA-768 Verify our CLIProxyAPI use cases with Tester Army e2e in the fork

Context

Eric's decision (2026-10-03): adopt Tester Army's e2e runner (https://github.com/tester-army/e2e, Apache-2.0) in our CLIProxyAPI fork (MASTRA-765). Use it to verify the proxy use cases we depend on. mastra-pilot already runs this tool (tools/tester-e2e/, docs/tester-army-e2e.md, ADR-014).

Research done 2026-10-03. The plan was reviewed by Astra (gpt-6-astra, xhigh), and every accepted finding was verified against source. This body reflects that review.

Use cases (studio2 main.log, 2026-09-03 to 10-03; factory-host API)

Client	Request shape	Status today
Open SWE	POST /v1/responses, non-streaming, gpt-5.6-sol at xhigh; store:false, include:[reasoning.encrypted_content], about 17 function tools, full history, no previous_response_id. About 28,300 requests; p99 110 s, max 632 s. stream:false is seen in the 9 sampled request dumps	Works; idle since 2026-09-26 (Open SWE reviewer disabled)
Open SWE fallback (OSWE-113)	same, gpt-5.6-terra at max	10 requests, all 429
Claude Code CLI	HEAD /api/hello (404), then POST /v1/messages?beta=true, SSE	Works (1 probe)
Native Messages	POST /v1/messages: Sonnet returns 429 (OSWE-278); Haiku returns 200	Sonnet fails
Responses with Claude models	POST /v1/responses, claude-sonnet-4-6 and claude-opus-5-5	Works
/v1/models, keyless	401 without a key	Works
Planned, MASTRA-767: panel codex-cli	Responses SSE, custom provider	23.6% of panel reviewer attempts
Planned, MASTRA-767: panel claude-cli	Messages ?beta=true SSE	4.1% of panel reviewer attempts
Planned: daily findings pass	claude --print; codex exec --output-schema	1 run/day each
Planned: MASTRA-647 custom provider	/chat/completions, json_schema, include_usage	Blocked: needs https or loopback

Never exercised: client-side SSE on the Codex route, Chat Completions, websocket Responses, and Claude selection across more than one credential.

How the suite works

• Runner: e2e 0.16.0, exact pin. There is no command-only target type, so we use a no-op engine (defineEngine({ name, version, spiVersion: 1, platform })), which no documented example shows for HTTP-only use. The first test proves it works. Tests use fetch, expect and toMatchSchema. We write our own SSE and websocket helpers.
• Hermetic target: a wrapper script starts a mock upstream and the candidate binary (built from the checkout) on loopback ports.
  • Credentials are config codex-api-key / claude-api-key entries with base-url pointing at the mock. They get provider codex / claude (internal/watcher/synthesizer/config.go:205-207) and pass through the same selector, affinity, quota-header capture, 429 classification and cooldown as OAuth logins.
  • OAuth files cannot be pointed at a mock.
  • The serving credential is read from the X-CPA-TRACE-ID response header (internal/logging/cpa_trace.go), formatted <time>-<auth_index>-<request_id>.
  • Sessions are pinned with Session_id (Responses) or X-Claude-Code-Session-Id (Messages).
• Live tests: the runner accepts a plain-HTTP url only for loopback hosts (e2e docs reference/config.mdx), and studio2 listens only on its tailnet address. So the live target declares no app.url, and the tests fetch CLIPROXY_LIVE_URL directly. They run on studio2 and read the client key from the local config. There are no management checks: management accepts only loopback clients.

Scope

1. U2, scaffold. e2e/ gets its own package and lockfile: e2e 0.16.0, zod, oxfmt and oxlint, all exact pins, 100 columns, E2E_TELEMETRY_DISABLED=1.
   • A no-op engine.
   • hermetic/start.mjs starts hermetic/mock-upstream.mjs on a loopback port, writes a temp config, and execs the candidate binary on {port}. The config holds the API-key credentials pointing at the mock, the client key and an empty auth dir, and the binary runs with -local-model.
   • lib/sse.ts.
   • The first test, a keyless GET /v1/models returning 401, proves the no-op engine.
   • CI builds ./cmd/server, runs --target hermetic and uploads report.json.
2. U3, hermetic protocol tests. These must pass on stock v8.0.13. The mock always answers Codex with upstream SSE that ends in a terminal response event, because the proxy calls upstream with SSE even for non-streaming clients (codex_executor_execute.go:130-170).
   • Open SWE shape: non-streaming Responses with function tools, store:false and include:[reasoning.encrypted_content]. Assert the upstream request carries them, and assert the client gets a complete JSON response.
   • Streaming Responses (the codex exec shape): assert SSE event order.
   • Claude Code shape: POST /v1/messages?beta=true, streaming, with X-Claude-Code-Session-Id.
   • Responses with a Claude model.
   • Chat Completions with json_schema and include_usage.
   • /v1/models with a key.
   • Failover: 429 usage_limit_reached on A; the same request is served by B, named in X-CPA-TRACE-ID.
   • All credentials quota-cooled: the proxy returns its local 429 model_cooldown (sdk/cliproxy/auth/selector.go:528-541), not 503.
3. U4, hermetic selector tests. These land as the first commit of the MASTRA-765 PR, not in this suite's required baseline.
   • Selector-specific: S1, S2, S3, S5, S10. Fixtures make the expected winner differ from ID order, so that stock fill-first cannot pass by coincidence. Exhaustion is seeded with 200 responses that carry exhausted headers, not with 429s, so the tests exercise the header rule rather than native cooldown. Every request sends an explicit session ID.
   • Regression only, existing upstream behaviour: S7 (stickiness) and S8 (burst at exhaustion).
   • Isolation. Each scenario gets its own model name and its own credentials, through per-key models lists, so scenarios do not share state in one proxy. The first two-scenario test proves this isolation.
   • S10 recipe. Observe every credential. Trigger a save, since saves are throttled to once a minute and shutdown does not flush. Inspect the saved file. Restart with the same auth dir and the same mock URL, because base-url is part of credential identity. Then send a new session with the mock omitting quota headers.
4. U6, live tests. One minimal request per use case that works today: the Open SWE shape (Codex, tiny input), streaming Responses (Codex), Claude Code Messages (Haiku), Responses with a Claude model, /v1/models, and a keyless 401. The OSWE-278 Sonnet native Messages 429 is recorded as a known failure, not asserted green.
5. Exit codes follow mastra-pilot: 1 = product failure, 2 = configuration, 3 = infrastructure.

Acceptance criteria

• The no-op engine test passes, or the runner gap is reported to Tester Army and the fallback is written down.
• The U3 protocol tests pass on stock v8.0.13 in the private repo's CI.
• The U4 selector-specific tests fail on the tests-only commit of the MASTRA-765 PR and pass at its head. The S7 and S8 regression tests pass on both.
• The live suite's report.json (commit, dirty flag, outcomes, attempts) is attached to MASTRA-766, once before the upgrade and once after.

Source refs

• Parent: MASTRA-763. Related: MASTRA-765 (repo and selector), MASTRA-766 (deploy; uses the live suite), MASTRA-767 (factory routes), OSWE-11 (13-item live Responses suite to absorb), OSWE-113, OSWE-278.

Review in Linear