test(e2e): hermetic Tester Army suite for proxy use cases (MASTRA-768)

[ericlitman]

Linear: MASTRA-768

Adds a Tester Army e2e suite that runs the proxy binary built from this checkout against a local mock upstream and checks the request shapes we depend on. This PR covers units U2 (scaffold) and U3 (protocol tests). The selector tests (U4) land with the selector in the MASTRA-765 PR. The live tests against studio2 (U6) come next.

What is here

• e2e/: its own package and lockfile, with exact pins: e2e 0.16.0, zod 4.6.1 (the version e2e pins), oxfmt 0.71.0, oxlint 1.86.0. oxfmt and oxlint run at 100 columns with --deny-warnings. Telemetry is off.
• e2e.config.ts: two targets on a no-op engine (defineEngine({ name, version, spiVersion: 1, platform })). The hermetic target starts hermetic/start.mjs. The live target has no app.url, because the runner rejects plain HTTP away from loopback; live tests will read CLIPROXY_LIVE_URL themselves.
• hermetic/start.mjs starts the mock in-process, writes a config whose codex-api-key and claude-api-key entries point at the mock, and runs .bin/cli-proxy-api -local-model. HTTPS_PROXY=http://127.0.0.1:9 blocks the binary's own startup fetches, the antigravity manifest and the management panel. Go never proxies loopback, so calls to the mock still go through.
• hermetic/mock-upstream.mjs answers Codex /v1/responses with upstream SSE that ends in response.completed, even for non-streaming clients, because the proxy always calls upstream with SSE. It answers Claude /v1/messages with Anthropic SSE, or JSON when the request does not stream. Both carry rate-limit headers. A credential key ending in -limited gets 429 usage_limit_reached. GET /_mock/requests returns what the mock received. Each test filters by a random marker, so parallel tests do not interfere.
• CI job e2e-hermetic: builds the binary, runs a frozen install, fmt:check, lint, the hermetic run, and uploads report.json plus the proxy log.

Tests (all on the hermetic target, stock v8.0.13)

1. Keyless GET /v1/models returns 401. This test also proves the no-op engine works.
2. GET /v1/models with the client key lists exactly the configured models.
3. Open SWE shape: non-streaming /v1/responses with a function tool, store:false and include:["reasoning.encrypted_content"]. The upstream request keeps all three, and the client gets a complete JSON response.
4. Streaming /v1/responses: event order created, output_item.added, output_text.delta, output_item.done, completed.
5. Claude Code shape: /v1/messages?beta=true, streaming, with X-Claude-Code-Session-Id; full event order.
6. /v1/responses with claude-sonnet-4-6 returns a complete response.
7. /v1/chat/completions with json_schema and include_usage: a content chunk, a usage chunk, then [DONE].
8. Failover: credential A answers 429 usage_limit_reached. The same request is served by B. X-CPA-TRACE-ID names B's auth index, and the mock saw A then B.
9. With every credential for a model cooled down, the second request gets the local 429 model_cooldown and never reaches the mock (sdk/cliproxy/auth/selector.go:531, status at :349).

Acceptance criteria from MASTRA-768

• "The no-op engine test passes, or the runner gap is reported." Passes. Test 1 runs on the no-op engine; the runner reported "command ready" in about 0.9 s.
• "The U3 protocol tests pass on stock v8.0.13 in the private repo's CI." Local: passed. The run on 1aa79f5 reports run.vcs.dirty: false, exit 0, and a summary of 9 discovered, 9 executed, 9 passed, 0 failed, 0 flaky. CI: the e2e-hermetic job on this PR is the check.
• I broke the failover assertion on purpose: it failed with ASSERTION_FAILED and exit 1. Then I restored it.
• The selector and live criteria are out of scope for this PR (see above).

Notes

• pnpm release age. pnpm 11's default one-day release-age check rejected e2e@0.16.0, published 2026-10-02 17:15Z. I did not add an exception. This PR was pushed after the version cleared the cutoff.
• Exit codes. The hermetic tests use plain expect, so any failure reports as exit 1, a product failure. The configuration (2) and infrastructure (3) classes belong to the live helper in U6.
• Not exercised here. OAuth-only behaviour (token refresh, plan model lists) and real upstream header formats. The live suite covers those.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith with what you need. Autofix is disabled.}

[linear-code]

MASTRA-768 Verify our CLIProxyAPI use cases with Tester Army e2e in the fork

Context

Eric's decision (2026-10-03): adopt Tester Army's e2e runner (https://github.com/tester-army/e2e, Apache-2.0) in our CLIProxyAPI fork (MASTRA-765). Use it to verify the proxy use cases we depend on. mastra-pilot already runs this tool (tools/tester-e2e/, docs/tester-army-e2e.md, ADR-014).

Research done 2026-10-03. The plan was reviewed by Astra (gpt-6-astra, xhigh), and every accepted finding was verified against source. This body reflects that review.

Use cases (studio2 main.log, 2026-09-03 to 10-03; factory-host API)

Client	Request shape	Status today
Open SWE	POST /v1/responses, non-streaming, gpt-5.6-sol at xhigh; store:false, include:[reasoning.encrypted_content], about 17 function tools, full history, no previous_response_id. About 28,300 requests; p99 110 s, max 632 s. stream:false is seen in the 9 sampled request dumps	Works; idle since 2026-09-26 (Open SWE reviewer disabled)
Open SWE fallback (OSWE-113)	same, gpt-5.6-terra at max	10 requests, all 429
Claude Code CLI	HEAD /api/hello (404), then POST /v1/messages?beta=true, SSE	Works (1 probe)
Native Messages	POST /v1/messages: Sonnet returns 429 (OSWE-278); Haiku returns 200	Sonnet fails
Responses with Claude models	POST /v1/responses, claude-sonnet-4-6 and claude-opus-5-5	Works
/v1/models, keyless	401 without a key	Works
Planned, MASTRA-767: panel codex-cli	Responses SSE, custom provider	23.6% of panel reviewer attempts
Planned, MASTRA-767: panel claude-cli	Messages ?beta=true SSE	4.1% of panel reviewer attempts
Planned: daily findings pass	claude --print; codex exec --output-schema	1 run/day each
Planned: MASTRA-647 custom provider	/chat/completions, json_schema, include_usage	Blocked: needs https or loopback

Never exercised: client-side SSE on the Codex route, Chat Completions, websocket Responses, and Claude selection across more than one credential.

How the suite works

• Runner: e2e 0.16.0, exact pin. There is no command-only target type, so we use a no-op engine (defineEngine({ name, version, spiVersion: 1, platform })), which no documented example shows for HTTP-only use. The first test proves it works. Tests use fetch, expect and toMatchSchema. We write our own SSE and websocket helpers.
• Hermetic target: a wrapper script starts a mock upstream and the candidate binary (built from the checkout) on loopback ports.
  • Credentials are config codex-api-key / claude-api-key entries with base-url pointing at the mock. They get provider codex / claude (internal/watcher/synthesizer/config.go:205-207) and pass through the same selector, affinity, quota-header capture, 429 classification and cooldown as OAuth logins.
  • OAuth files cannot be pointed at a mock.
  • The serving credential is read from the X-CPA-TRACE-ID response header (internal/logging/cpa_trace.go), formatted <time>-<auth_index>-<request_id>.
  • Sessions are pinned with Session_id (Responses) or X-Claude-Code-Session-Id (Messages).
• Live tests: the runner accepts a plain-HTTP url only for loopback hosts (e2e docs reference/config.mdx), and studio2 listens only on its tailnet address. So the live target declares no app.url, and the tests fetch CLIPROXY_LIVE_URL directly. They run on studio2 and read the client key from the local config. There are no management checks: management accepts only loopback clients.

Scope

1. U2, scaffold. e2e/ gets its own package and lockfile: e2e 0.16.0, zod, oxfmt and oxlint, all exact pins, 100 columns, E2E_TELEMETRY_DISABLED=1.
   • A no-op engine.
   • hermetic/start.mjs starts hermetic/mock-upstream.mjs on a loopback port, writes a temp config, and execs the candidate binary on {port}. The config holds the API-key credentials pointing at the mock, the client key and an empty auth dir, and the binary runs with -local-model.
   • lib/sse.ts.
   • The first test, a keyless GET /v1/models returning 401, proves the no-op engine.
   • CI builds ./cmd/server, runs --target hermetic and uploads report.json.
2. U3, hermetic protocol tests. These must pass on stock v8.0.13. The mock always answers Codex with upstream SSE that ends in a terminal response event, because the proxy calls upstream with SSE even for non-streaming clients (codex_executor_execute.go:130-170).
   • Open SWE shape: non-streaming Responses with function tools, store:false and include:[reasoning.encrypted_content]. Assert the upstream request carries them, and assert the client gets a complete JSON response.
   • Streaming Responses (the codex exec shape): assert SSE event order.
   • Claude Code shape: POST /v1/messages?beta=true, streaming, with X-Claude-Code-Session-Id.
   • Responses with a Claude model.
   • Chat Completions with json_schema and include_usage.
   • /v1/models with a key.
   • Failover: 429 usage_limit_reached on A; the same request is served by B, named in X-CPA-TRACE-ID.
   • All credentials quota-cooled: the proxy returns its local 429 model_cooldown (sdk/cliproxy/auth/selector.go:528-541), not 503.
3. U4, hermetic selector tests. These land as the first commit of the MASTRA-765 PR, not in this suite's required baseline.
   • Selector-specific: S1, S2, S3, S5, S10. Fixtures make the expected winner differ from ID order, so that stock fill-first cannot pass by coincidence. Exhaustion is seeded with 200 responses that carry exhausted headers, not with 429s, so the tests exercise the header rule rather than native cooldown. Every request sends an explicit session ID.
   • Regression only, existing upstream behaviour: S7 (stickiness) and S8 (burst at exhaustion).
   • Isolation. Each scenario gets its own model name and its own credentials, through per-key models lists, so scenarios do not share state in one proxy. The first two-scenario test proves this isolation.
   • S10 recipe. Observe every credential. Trigger a save, since saves are throttled to once a minute and shutdown does not flush. Inspect the saved file. Restart with the same auth dir and the same mock URL, because base-url is part of credential identity. Then send a new session with the mock omitting quota headers.
4. U6, live tests. One minimal request per use case that works today: the Open SWE shape (Codex, tiny input), streaming Responses (Codex), Claude Code Messages (Haiku), Responses with a Claude model, /v1/models, and a keyless 401. The OSWE-278 Sonnet native Messages 429 is recorded as a known failure, not asserted green.
5. Exit codes follow mastra-pilot: 1 = product failure, 2 = configuration, 3 = infrastructure.

Acceptance criteria

• The no-op engine test passes, or the runner gap is reported to Tester Army and the fallback is written down.
• The U3 protocol tests pass on stock v8.0.13 in the private repo's CI.
• The U4 selector-specific tests fail on the tests-only commit of the MASTRA-765 PR and pass at its head. The S7 and S8 regression tests pass on both.
• The live suite's report.json (commit, dirty flag, outcomes, attempts) is attached to MASTRA-766, once before the upgrade and once after.

Source refs

• Parent: MASTRA-763. Related: MASTRA-765 (repo and selector), MASTRA-766 (deploy; uses the live suite), MASTRA-767 (factory routes), OSWE-11 (13-item live Responses suite to absorb), OSWE-113, OSWE-278.

Review in Linear