feat(auth): read Codex and Claude usage on a timer for soonest-reset (MASTRA-787)

[ericlitman]

Linear: MASTRA-787

The soonest-reset selector learns a credential's quota only from the proxy's own traffic. On studio2 that hides the Codex account that resets soonest, litman.org-pro, which Eric uses outside the proxy. It also leaves the Claude utilization stale. This PR reads each Codex and Claude OAuth credential's usage endpoint every 5 minutes and feeds the reading into the observation the selector already ranks on.

Commits

1. 2fb67d24 feat(auth): read Codex and Claude usage on a timer.

   • sdk/cliproxy/service_quota.go: one loop owned by the service. It sweeps once at start, then every 5 minutes, and is cancelled on shutdown.
   • Each read uses GetByID for the current credential and skips anything that is not OAuth with a valid access token. It then calls Manager.HttpRequest, which goes straight to the executor. That path only sets Authorization from the current token: it never refreshes, never calls MarkResult, and never writes the auth file.
   • A non-2xx read logs its status and keeps the previous observation.
   • ParseCodexUsageHeaders and ParseClaudeUsageHeaders (now internal/runtime/executor/helps/quota_usage.go) map the responses onto the existing X-Codex-* and Anthropic-Ratelimit-Unified-* signals. Claude utilization arrives as a percent and is stored as a fraction, and resets_at is stored as Unix seconds. Nothing ranks on the extra limits (additional_rate_limits, iguana_necktie), so they are not mapped.
   • Manager.ObserveQuotaHeaders (quota_signals.go) replaces only the credential-wide Auth.Quota snapshot, under m.mu, then persists the cooldown state (.cds). Counters, availability and cooldown fields are untouched.

2. 51f8a205 docs(agents): adds the 30 s per-read deadline to the AGENTS.md timeout exceptions. The executor's HTTP client has no timeout, and the reads run serially, so one hung read would otherwise stop every later observation. This is the same kind of exception as the management APICall timeout already listed.

3. 93889015 refactor(auth): subtract review findings. Review was /pstack:interrogate (verdict below).

   • The Codex parser no longer drops a window for used_percent above 100 or a missing limit_window_seconds. A read replaces the whole snapshot, so dropping a window erased the exhaustion the selector gates on. New test: 100.4% with no window length still maps.
   • Deleted the scheduler push in ObserveQuotaHeaders. The scheduler reads only cooldown fields from Quota, and soonest-reset ranks from m.auths.
   • Deleted quotaReadsCancel. The loop runs on Run's own context.
   • Moved both parsers into the new file helps/quota_usage.go. codex_quota.go and claude_ratelimit.go are back to upstream byte for byte, which keeps fork code out of upstream files for rebases.
   • The routing test now asserts the exact set of four usage reads, and that the saved .cds records hold codex-c's new reset and codex-a's previous one.

Size against main: 158 non-test Go lines, 1 AGENTS.md line, the rest tests.

Acceptance criteria (MASTRA-787)

1. "Unit tests map recorded responses." quota_usage_test.go checks the recorded wham/usage and oauth/usage responses, with identities removed:
   • Codex gives Used-Percent 9 and Window-Minutes 10080, plus both reset fields and Limit-Reached false. The null secondary window and the additional limits produce nothing.
   • Claude gives 0.14 and 0.31, with resets 1791081600 and 1791500400.
   • A response with no usable window returns nil.
2. "A never-used Codex credential with the soonest reset wins a new session." TestQuotaUsageSweepRoutesNewSessionToUnusedCredential runs the real Run and HTTP handler. It uses three synthetic Codex OAuth credentials and one Claude credential, with soonest-reset, and every upstream call goes to a mock round-tripper. codex-c has never been used, and its usage read shows the soonest reset, so a new session goes to codex-c. With the sweep start removed, the test fails: "new session went to [tok-a], want [tok-c]".
3. "No token refresh, no auth-file write." During the sweep the token store gets zero saves and no credential's refresh fields change. The mock saw no request other than the usage reads, and the .cds store was written.
4. "A rejected read keeps the previous observation and never clears a cooldown." codex-a's read returns 429 and its earlier reset stays. A test in the auth package checks that an observation leaves cooldown fields, availability, status, NextRetryAfter and Generation unchanged.

Checks, which I ran myself on the committed tree 9388901 (the gate also passed earlier on 51f8a20):

• go build ./...: exit 0.
• go vet ./sdk/cliproxy/... ./internal/runtime/executor/...: exit 0.
• go test ./...: exit 0.
• go test -race -count=1 ./sdk/cliproxy/ ./sdk/cliproxy/auth/ ./internal/runtime/executor/helps/, eight logged runs:
  • 3 of 8 exited 1 on this branch. All failures were in TestHandleAuthUpdates_SameRevisionWaitDoesNotWaitForOtherAuthInBatch and TestEndToEndAuthFile{Replacement,Patch}_RestoresModelsInV1ModelsWithoutRestart.
  • 2 of 8 exited 1 on main 3931a30, in the same three tests. These are the upstream tests MASTRA-765 recorded as flaky. No new test failed in any run.
• Mutation check: with s.startQuotaUsageReads(ctx) commented out, TestQuotaUsageSweepRoutesNewSessionToUnusedCredential fails.
• Live read through the production executors. I built a throwaway binary from 51f8a20 and ran it on studio2. It read all 6 real credentials through CodexExecutor.HttpRequest and ClaudeExecutor.HttpRequest, with exactly the headers this PR sends. All 6 returned 200: Codex over HTTP/2, and Claude over HTTP/1.1 through the uTLS transport. All 6 mapped 4 or 5 signals, and no auth file's mtime changed.

Not yet verified, and required before deploy:

• Reading does not start an idle window. Codex has evidence: two reads of litman.org-pro's idle gpt-reserve limit, 9 s apart, moved reset_at by 9 s. Claude does not: every Claude 5-hour window was running at test time.
• The live studio2 ordering check (acceptance criterion 3).

Review

Code review: /pstack:interrogate. Three reviewers: Opus 5.5 at max (8 findings), gpt-5.6-sol at max (4), and Grok 4.7 at xhigh (4).

• Acted on (commit 3):

  • The Codex parser dropped exhausted windows. Raised by Grok and Opus.
  • The scheduler push, the cancel field and the parser placement. Raised by Opus.
  • Test assertions. Raised by Sol and Opus.

• Disproved as a functional failure: "the reads lack the Codex and Claude Code client headers, so the hosts reject them". Raised by Grok and Opus. The live read above returned 200 for all 6 credentials.

• Open for the owner: the reads still present Go's default User-Agent, from the same IP as each account's CLI traffic. An executor-owned read with the inference headers is the alternative.

• Noted, not acted on:

  • A slow read can overwrite a newer request observation (Grok). The worst case is one 429, then the existing failover.
  • A read stores fewer fields than a proxied response, such as plan type, credits and Claude status (Grok, Opus). Ranking doesn't use them, and exhaustion still gates on utilization.
  • Each read rewrites every .cds file, about 6 per sweep (Sol).
  • Disabled and rejected credentials are still read, which only adds log warnings (Opus).
  • The test injects its transport through the context (Sol).
  • Shutdown no longer stops the loop by itself. Run's exit does, with at most one 30 s read in flight (from the fix lane).

• Dismissed: "a replacement credential is corrupted by an in-flight read" (Sol). It needs an auth file swapped to a different account under the same name within 30 s, which normal operation never does.

• Design: Astra (gpt-6-astra, high). The adjudication is on MASTRA-787.

• Build: Opus 5.5 at medium. I read the full diff and re-ran the checks above.

• Note on the lane: while it was checking that the test fails without the sweep, an early version of the service test let the auto-refresh loop send the fake refresh tokens to OpenAI's real token endpoint, which answered 401. The current test uses tokens with no refresh token that expire in 10 days. Any request not matched by the mock is recorded as a test failure.

🤖 Generated with Claude Code

[linear-code]

MASTRA-787 Read Codex and Claude usage on a timer so soonest-reset sees quota used outside the proxy

Outcome

The soonest-reset selector in mobilyze-llc/CLIProxyAPI ranks every Codex and Claude OAuth credential by its latest usage reading. That includes quota consumed outside the proxy, and credentials the proxy has never sent a request. A reading can be up to one read interval old.

Why

Today the selector knows a credential's quota only from the proxy's own request traffic: response headers, and Codex websocket quota events (MASTRA-765). On 2026-10-04, reading the provider usage endpoints on studio2 showed two problems (details on MASTRA-766):

• Codex. litman.org-pro resets soonest (2026-10-09 21:13Z, 9% used). It has never served a proxy request, so it sorts last, and every new Codex session goes to mobilyze.com-pro (resets 2026-10-10 03:08Z). Eric uses litman.org outside the proxy.
• Claude. The proxy's observation of litman.org was stale: 24% weekly against 31% read from usage. The ordering stays right because Claude's weekly reset is on a fixed schedule. The exhaustion gate, however, runs on old utilization.

Eric chose (2026-10-04) to read usage for every credential on a timer, rather than only once for credentials with no observation.

Scope

• On a timer, read each Codex credential's GET https://chatgpt.com/backend-api/wham/usage and each Claude credential's GET https://api.anthropic.com/api/oauth/usage. Use the proxy's own current access token for each credential.
• Turn each read into the same quota observation the selector already ranks and gates on, and persist it with save-cooldown-status like any other observation.
• Never refresh a token on behalf of a read. The proxy's existing refresh path stays the only refresher for each lineage. A read whose token is expired or rejected is skipped, and the selector keeps the last observation.
• Suggestion, not a requirement: map the usage response onto the existing X-Codex-* and Anthropic-Ratelimit-Unified-* signal names, so the ranking and gate code do not change. Upstream already parses the wham/usage array form (internal/runtime/executor/helps/codex_quota.go).

Acceptance criteria

1. Unit tests. Go tests cover the mapping from a recorded wham/usage response and a recorded api/oauth/usage response to the observation. A recorded response is one captured with tokens and identities removed.
2. Service-level test. A Go test runs the real proxy HTTP handler with synthetic OAuth credentials and routes the usage reads to a mock through the existing transport injection. A never-used Codex credential with the soonest reset wins a new session. (Revised from a Tester Army hermetic test: that suite uses API-key credentials, and reaching a mock usage endpoint there would need a production endpoint override.)
3. Live on studio2. After deploy, a new Codex session goes to litman.org-pro, or whichever credential the usage endpoint shows resetting soonest at that time. Check it with ordering_check.py against the usage readings.
4. No window started. Show that a read does not start an idle window, for Codex and for Claude. Codex evidence: two reads of litman.org-pro's idle gpt-reserve limit 9 s apart moved reset_at by 9 s (MASTRA-766). Claude is not yet shown. Read a Claude credential whose 5-hour window has lapsed, twice, before deploy.
5. No token refresh outside the existing path. No read triggers a token-endpoint call or an auth-file write. A test counts refresh calls and token-store saves during a sweep, and both are zero. The observation .cds write is expected.
6. A rejected or failed read keeps the previous observation, and a read never clears an existing cooldown.

Constraints

• One commit per change on top of v8.0.13-mob.1. Release as v8.0.13-mob.2, and deploy by the studio2-ops runbook section "CLIProxyAPI upgrades".
• No cap, retry budget or backoff machinery that no criterion above needs.
• Read interval: 5 minutes, a constant. The observation that settles it: the read log over the first week after deploy shows no 429 from either usage endpoint. Each read logs its non-2xx status.

Source refs

• Parent: MASTRA-763. Related: MASTRA-765 (selector), MASTRA-766 (deploy and measurement, with the usage readings), MASTRA-768 (e2e).
• Probe scripts on studio2: ~/.local/share/cliproxyapi/mastra766/codex_usage_probe.py and claude_usage_probe.py.

Design pass (Astra, gpt-6-astra high, 2026-10-04) and adjudication

Accepted, verified against code:

• One service-owned loop in a new sdk/cliproxy/service_quota.go: an immediate sweep at start, then every interval, cancelled on shutdown.
• Per read: GetByID for the current credential, skip it unless HasValidAccessToken, then Manager.HttpRequest. That path goes straight to the executor, and PrepareRequest only sets Authorization from the current token (codex_executor_request.go:60-91, claude_executor.go:214-254). No refresh, no MarkResult, no auth-file write.
• Map the response onto the existing signal names, then call QuotaState.ObserveResponseHeadersForProvider on Auth.Quota only, under m.mu, and persistCooldownStates. Claude utilization is a percent, so divide it by 100. resets_at is RFC3339, so emit Unix seconds. Codex limit_window_seconds divided by 60 gives Window-Minutes.
• The Codex websocket-event parser does not accept the usage shape (codex_quota.go:28). Add a usage entry point.

Added by me:

• A per-read context deadline of 30 s. NewUtlsHTTPClient(..., 0) sets no client timeout, so one hung read would stall every later read and stop observations. That fails the outcome.

Rejected:

• Mapping Codex additional_rate_limits and Claude iguana_necktie, limits, spend and extra_usage. The selector does not rank them, so nothing consumes them.
• Any retry, backoff, worker pool or feature flag.

Review in Linear