Skip to content

fix(routing): skip known-exhausted accounts at admission#57

Open
iceteaSA wants to merge 4 commits into
cortexkit:mainfrom
iceteaSA:fix/routing-skip-exhausted
Open

fix(routing): skip known-exhausted accounts at admission#57
iceteaSA wants to merge 4 commits into
cortexkit:mainfrom
iceteaSA:fix/routing-skip-exhausted

Conversation

@iceteaSA

@iceteaSA iceteaSA commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Stacked on #54 and #56 — the base merge (145c489) brings isQuotaExhausted (#54) and the getSidebarState(path) overload + snapshot-level quota.checkedAt (#56). Only the top commit (d5f1fdd) is this PR's change; merge those two first and this reduces to one commit.

Why

Rate-limit marks are per-process and in-memory. With several concurrent opencode processes and fallback-first routing pinned at an exhausted account, every process independently pays one doomed admission probe (admission-time usage_limit_reached → mark → retryable error → reroute) before learning what the machine-global sidebar file already knows: the account is at 100% with a reset days away. The operator sees each discovery as a visible retry/error flash.

What

Admission-time candidate selection now consults quota before probing:

  • Dual source with freshness precedence: in-memory QuotaManager peek vs the shared sidebar file row (compared by primary.checkedAt, then snapshot checkedAt, then entry checkedAt). The fresher source is selected — the file wins only when strictly newer, memory wins ties, and an empty in-memory cache (fresh process) defers to a valid file row. isQuotaExhausted (type-safe, fail-open) is applied only to the selected source.
  • Fallback filter: exhausted candidates are dropped after the existing killswitch/rate-limit filters; skipped accounts are never probed, so their backoff/mark state is untouched. Applied to both the proactive (fallback-first) gate and the reactive iterator via a shared memoized selection.
  • Exhausted primary: synthesizes the existing killswitch-style 429 (reason quota-exhausted, Retry-After from the account's own resetsAt) so the reroute happens without the doomed probe — only when a non-exhausted fallback survives.
  • Safety valves: unknown/missing/malformed/past-reset quota is never exhausted (fail-open); if filtering would remove the last admission path, the current wire-probe order is fully restored — a stale or corrupt file can never brick routing, the wire stays the final authority. Sidebar file read at most once per request, tolerant reader, no-throw.
  • Each skip logs on the quota channel: admission skip: exhausted account {accountId, source, resetsAt}.

Verification

  • RED-first: exhausted first fallback (expected client-alt, got work-alt), fresh-process file exhaustion, newer-exhausted-file vs stale-healthy-memory, exhausted primary reroute — all fail pre-fix; 7 fail-open characterizations proven non-vacuous by reverse-applying the src diff (they pass on reverted source, i.e. they pin non-interference).
  • Review (gemini-3.1-pro): APPROVE 0 must / 0 should — adversarial-file surface (block/steer/brick), empty-cache tie-break, filter ordering (no state mutation for skipped accounts), once-per-request memoization, proactive+reactive coverage, and reverse-apply RED all verified.
  • Gates: build ✓ · tsc ✓ · full suite green ✓ · biome clean.

View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.


Summary by cubic

Skip probing accounts that are already quota‑exhausted at admission by using the freshest quota from memory or the shared sidebar, and add a background quota refresh to keep snapshots current and reduce error flashes. Also fixes stale per‑session active account display and prevents fresher on‑disk quota from being clobbered.

  • New Features

    • Admission quota check picks the fresher source (memory vs getSidebarState and respects mainAccountId), drops exhausted fallbacks, and if main is exhausted returns a synthetic 429 (quota-exhausted) with Retry‑After from its resetsAt; proactive and reactive fallback paths share this selection and log skips.
    • Background refresher (BackgroundQuotaRefresh/refreshQuotaInBackground) runs ~every 5m with ±30s jitter, respects backoff, skips snapshots fresher than 4m (using memory or the shared sidebar), keeps one in‑flight run, and writes the sidebar only when quota changed.
    • Sidebar/state: propagate checkedAt for main/fallbacks, add mainAccountId, add isQuotaExhausted, ignore exhausted entries in resolveSessionSidebarRouting, and record routing for a parent session via x-parent-session-id.
  • Bug Fixes

    • Fail‑open on missing/malformed/past‑reset quota; if filtering would remove the last path, preserve the original probe order.
    • Prevent stale overwrites: setSidebarMachineState merges quota by freshness per account and never carries main quota across identities, fixing stale per‑session active account displays.

Written for commit 8dfde55. Summary will update on new commits.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 10 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/opencode/src/index.ts
Comment thread packages/opencode/src/core/refresh-all-quota.ts
Comment thread packages/opencode/src/index.ts
Comment thread packages/opencode/src/tests/integration.test.ts
@iceteaSA
iceteaSA force-pushed the fix/routing-skip-exhausted branch from d5f1fdd to 8dfde55 Compare July 22, 2026 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant