test(e2e): live tests for the deployed proxy use cases (MASTRA-768)

[ericlitman]

Linear: MASTRA-768 (U6)

Adds the live e2e tests: one minimal request per proxy use case that works today, run against the deployed proxy on studio2 with --target live. They need CLIPROXY_LIVE_URL and CLIPROXY_CLIENT_KEY. On studio2, the key is read from the local proxy config into the environment and never printed.

What is here

• e2e/lib/live.ts: the live() helper, which classifies failures:

  • Exit 2: a missing URL or key, or a 401 on a request that sent the key.
  • Exit 3: the proxy is unreachable, or an upstream 429 or 5xx occurs that the test does not expect.
  • Exit 1: anything else.

  Redirects are not followed, so the key never leaves for another host. One 60 s AbortSignal bounds each request: the slowest case is OSWE-278 at about 10.5 s, and the runner's own test timeout is 120 s. expectTrace() asserts X-CPA-TRACE-ID, which v7.3.17 already sets (internal/api/server.go:147, sdk/api/handlers/handlers.go:233).

• e2e/tests/live.e2e.ts (7 tests):

  • keyless 401;
  • the model list with a key;
  • the Open SWE shape (non-streaming Responses on gpt-5.6-sol at low effort, with store:false, include and a function tool);
  • streaming Responses;
  • Claude Code Messages on Haiku;
  • Responses on claude-sonnet-4-6;
  • OSWE-278, which asserts the current 429 on native Messages with Sonnet. The runner has no expected-failure mechanism, so this test fails with exit 1 once OSWE-278 is fixed.

Evidence

• Failure classes, proven without contacting studio2:
  • URL unset: exit 2 (TEST_SETUP_FAILED);
  • http://127.0.0.1:9: exit 3 (ENVIRONMENT_UNAVAILABLE);
  • a loopback stub answering 401: exit 2 (AUTH_CREDENTIAL_INVALID);
  • a stub answering 503: exit 3.
• Baseline against studio2's deployed v7.3.17 (9bdde54), 2026-10-03 17:19Z, run on studio2 from a clone of 874604f: run.vcs.dirty: false, exit 0, 7 of 7 live tests passed, 9 hermetic tests skipped by target. Durations ran from 23 ms (keyless) to 10.75 s (OSWE-278). The report is attached to MASTRA-766.
• The hermetic suite still passes 9 of 9 on this branch. CI runs it.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith with what you need. Autofix is disabled.}

[linear-code]

MASTRA-768 Verify our CLIProxyAPI use cases with Tester Army e2e in the fork

Context

Eric's decision (2026-10-03): adopt Tester Army's e2e runner (https://github.com/tester-army/e2e, Apache-2.0) in our CLIProxyAPI fork (MASTRA-765). Use it to verify the proxy use cases we depend on. mastra-pilot already runs this tool (tools/tester-e2e/, docs/tester-army-e2e.md, ADR-014).

Research done 2026-10-03. The plan was reviewed by Astra (gpt-6-astra, xhigh), and every accepted finding was verified against source. This body reflects that review.

Use cases (studio2 main.log, 2026-09-03 to 10-03; factory-host API)

Client	Request shape	Status today
Open SWE	POST /v1/responses, non-streaming, gpt-5.6-sol at xhigh; store:false, include:[reasoning.encrypted_content], about 17 function tools, full history, no previous_response_id. About 28,300 requests; p99 110 s, max 632 s. stream:false is seen in the 9 sampled request dumps	Works; idle since 2026-09-26 (Open SWE reviewer disabled)
Open SWE fallback (OSWE-113)	same, gpt-5.6-terra at max	10 requests, all 429
Claude Code CLI	HEAD /api/hello (404), then POST /v1/messages?beta=true, SSE	Works (1 probe)
Native Messages	POST /v1/messages: Sonnet returns 429 (OSWE-278); Haiku returns 200	Sonnet fails
Responses with Claude models	POST /v1/responses, claude-sonnet-4-6 and claude-opus-5-5	Works
/v1/models, keyless	401 without a key	Works
Planned, MASTRA-767: panel codex-cli	Responses SSE, custom provider	23.6% of panel reviewer attempts
Planned, MASTRA-767: panel claude-cli	Messages ?beta=true SSE	4.1% of panel reviewer attempts
Planned: daily findings pass	claude --print; codex exec --output-schema	1 run/day each
Planned: MASTRA-647 custom provider	/chat/completions, json_schema, include_usage	Blocked: needs https or loopback

Never exercised: client-side SSE on the Codex route, Chat Completions, websocket Responses, and Claude selection across more than one credential.

How the suite works

• Runner: e2e 0.16.0, exact pin. There is no command-only target type, so we use a no-op engine (defineEngine({ name, version, spiVersion: 1, platform })), which no documented example shows for HTTP-only use. The first test proves it works. Tests use fetch, expect and toMatchSchema. We write our own SSE and websocket helpers.
• Hermetic target: a wrapper script starts a mock upstream and the candidate binary (built from the checkout) on loopback ports.
  • Credentials are config codex-api-key / claude-api-key entries with base-url pointing at the mock. They get provider codex / claude (internal/watcher/synthesizer/config.go:205-207) and pass through the same selector, affinity, quota-header capture, 429 classification and cooldown as OAuth logins.
  • OAuth files cannot be pointed at a mock.
  • The serving credential is read from the X-CPA-TRACE-ID response header (internal/logging/cpa_trace.go), formatted <time>-<auth_index>-<request_id>.
  • Sessions are pinned with Session_id (Responses) or X-Claude-Code-Session-Id (Messages).
• Live tests: the runner accepts a plain-HTTP url only for loopback hosts (e2e docs reference/config.mdx), and studio2 listens only on its tailnet address. So the live target declares no app.url, and the tests fetch CLIPROXY_LIVE_URL directly. They run on studio2 and read the client key from the local config. There are no management checks: management accepts only loopback clients.

Scope

1. U2, scaffold. e2e/ gets its own package and lockfile: e2e 0.16.0, zod, oxfmt and oxlint, all exact pins, 100 columns, E2E_TELEMETRY_DISABLED=1.
   • A no-op engine.
   • hermetic/start.mjs starts hermetic/mock-upstream.mjs on a loopback port, writes a temp config, and execs the candidate binary on {port}. The config holds the API-key credentials pointing at the mock, the client key and an empty auth dir, and the binary runs with -local-model.
   • lib/sse.ts.
   • The first test, a keyless GET /v1/models returning 401, proves the no-op engine.
   • CI builds ./cmd/server, runs --target hermetic and uploads report.json.
2. U3, hermetic protocol tests. These must pass on stock v8.0.13. The mock always answers Codex with upstream SSE that ends in a terminal response event, because the proxy calls upstream with SSE even for non-streaming clients (codex_executor_execute.go:130-170).
   • Open SWE shape: non-streaming Responses with function tools, store:false and include:[reasoning.encrypted_content]. Assert the upstream request carries them, and assert the client gets a complete JSON response.
   • Streaming Responses (the codex exec shape): assert SSE event order.
   • Claude Code shape: POST /v1/messages?beta=true, streaming, with X-Claude-Code-Session-Id.
   • Responses with a Claude model.
   • Chat Completions with json_schema and include_usage.
   • /v1/models with a key.
   • Failover: 429 usage_limit_reached on A; the same request is served by B, named in X-CPA-TRACE-ID.
   • All credentials quota-cooled: the proxy returns its local 429 model_cooldown (sdk/cliproxy/auth/selector.go:528-541), not 503.
3. U4, hermetic selector tests. These land as the first commit of the MASTRA-765 PR, not in this suite's required baseline.
   • Selector-specific: S1, S2, S3, S5, S10. Fixtures make the expected winner differ from ID order, so that stock fill-first cannot pass by coincidence. Exhaustion is seeded with 200 responses that carry exhausted headers, not with 429s, so the tests exercise the header rule rather than native cooldown. Every request sends an explicit session ID.
   • Regression only, existing upstream behaviour: S7 (stickiness) and S8 (burst at exhaustion).
   • Isolation. Each scenario gets its own model name and its own credentials, through per-key models lists, so scenarios do not share state in one proxy. The first two-scenario test proves this isolation.
   • S10 recipe. Observe every credential. Trigger a save, since saves are throttled to once a minute and shutdown does not flush. Inspect the saved file. Restart with the same auth dir and the same mock URL, because base-url is part of credential identity. Then send a new session with the mock omitting quota headers.
4. U6, live tests. One minimal request per use case that works today: the Open SWE shape (Codex, tiny input), streaming Responses (Codex), Claude Code Messages (Haiku), Responses with a Claude model, /v1/models, and a keyless 401. The OSWE-278 Sonnet native Messages 429 is recorded as a known failure, not asserted green.
5. Exit codes follow mastra-pilot: 1 = product failure, 2 = configuration, 3 = infrastructure.

Acceptance criteria

• The no-op engine test passes, or the runner gap is reported to Tester Army and the fallback is written down.
• The U3 protocol tests pass on stock v8.0.13 in the private repo's CI.
• The U4 selector-specific tests fail on the tests-only commit of the MASTRA-765 PR and pass at its head. The S7 and S8 regression tests pass on both.
• The live suite's report.json (commit, dirty flag, outcomes, attempts) is attached to MASTRA-766, once before the upgrade and once after.

Source refs

• Parent: MASTRA-763. Related: MASTRA-765 (repo and selector), MASTRA-766 (deploy; uses the live suite), MASTRA-767 (factory routes), OSWE-11 (13-item live Responses suite to absorb), OSWE-113, OSWE-278.

Review in Linear

[ericlitman]

Review gate: pstack:interrogate (see #3 and MASTRA-765). Fix-round commits:

• daf4ac1c: the OSWE-278 test now accepts only the defect's exact body, {"type":"error","error":{"type":"rate_limit_error","message":"Error"}}, captured from studio2 today. Any other 429 stays an infrastructure failure, exit 3.
• 4261d34d: the live Claude Code test sends the four Claude Code detection signals.

Live run on studio2 against the deployed v7.3.17, at 4261d34: 7 of 7 passed, exit 0.