Skip to content

fix(kosong): fail fast on quota-exhausted 429 instead of retrying#1857

Open
vinlee19 wants to merge 3 commits into
MoonshotAI:mainfrom
vinlee19:fix/quota-429-fail-fast
Open

fix(kosong): fail fast on quota-exhausted 429 instead of retrying#1857
vinlee19 wants to merge 3 commits into
MoonshotAI:mainfrom
vinlee19:fix/quota-429-fail-fast

Conversation

@vinlee19

@vinlee19 vinlee19 commented Jul 17, 2026

Copy link
Copy Markdown

Related Issue

Part of #1860 — this PR fixes the engine-side fail-fast half. Companion UI PR: #1861 (live retry progress in the TUI), which closes the issue together with this one.

Problem

When an account's quota/balance is exhausted, Moonshot returns HTTP 429 with a structured body:

HTTP/2 429            (no retry-after header)
{"error":{"message":"Your account org-… <ak-…> is suspended due to insufficient balance,
  please recharge your account or check your plan and billing details",
  "type":"exceeded_current_quota_error"}}

normalizeAPIStatusError classifies every 429 as a retryable APIProviderRateLimitError — the structured error.type/error.code that the OpenAI SDK parses onto its APIError is dropped at the convertOpenAIError seam. A quota-exhausted 429 can never succeed on retry, yet it burns the full retry budget: 10 attempts with 0.5→32s exponential backoff, ~3 minutes per turn, with no feedback in the interactive TUI. Users read it as a frozen app.

Captured session log (kimi-code 0.26.0, provider type = "kimi" against api.moonshot.cn, one turn ≈ 189s):

11:51:30.355 INFO  llm request  turnStep=15.1
11:51:35.358 INFO  llm request  turnStep=15.1 attempt=2/10
...
11:54:34.426 INFO  llm request  turnStep=15.1 attempt=10/10
11:54:39.021 WARN  llm request failed  ... errorName=APIProviderRateLimitError statusCode=429
11:54:39.037 ERROR turn failed  turnId=15

The same account produced two different message wordings for the same error.type ("You exceeded your current token quota: … please check your account balance" earlier, the "suspended due to insufficient balance" variant later) — so message text alone is not a reliable discriminator, but the structured type is.

What changed

  • New APIProviderQuotaExhaustedError (extends APIStatusError; deliberately not a subclass of APIProviderRateLimitError), minted in normalizeAPIStatusError.
  • Classification is structural first: convertOpenAIError forwards the SDK-parsed body error.type/error.code via a new trailing optional options parameter on normalizeAPIStatusError (all other callers unchanged). Recognized values: exceeded_current_quota_error (Moonshot), insufficient_quota (OpenAI). Billing-anchored message patterns (both observed Moonshot wordings, OpenAI wording, arrears) are the fallback for gateways that flatten the body to text — deliberately no bare /quota/ or /balance/ patterns, so transient "token quota per minute" throttles keep retrying.
  • The new class is excluded from isRetryableGenerateError (fails fast even when a retry-after header is present) and from isProviderRateLimitError (no swarm requeue/suspend).
  • toKimiErrorPayload (v1) and translateProviderError (v2) map it to the existing provider.api_error code (retryable: false) instead of provider.rate_limit (retryable: true) — no new protocol code needed; the provider's actionable "please recharge" message is preserved on the payload. classifyApiError reports quota_exhausted in telemetry.
  • agent-core-v2 mirrors the same classification fix (llmProtocol/errors.ts, providers/openai-common.ts, protocol/errors.ts).
  • Docs: the loop_control section (en/zh) now states retries only apply to transient failures.

Behavior notes for reviewers:

  • Quota-failed swarm subagents now fail instead of suspending indefinitely as "Rate limited...".
  • Quota errors cross the wire as provider.api_error (retryable: false) rather than provider.rate_limit.
  • Transient rate-limit 429 behavior is unchanged — verified end-to-end against a mock provider: the captured quota body fails at attempt 1/10 with the provider's "please recharge" message; a rate_limit_reached_error body with retry-after: 1 still walks the full 10-attempt ladder honoring the header. (The OpenAI SDK's own internal maxRetries=2 still applies beneath one logical attempt — pre-existing behavior, untouched.)

Checklist

  • I have read the CONTRIBUTING document.
  • I have linked a related issue, or explained the problem above.
  • I have added tests that prove my feature works.
  • Ran gen-changesets skill, or this PR needs no changeset. (changeset added: patch for @moonshot-ai/kimi-code and @moonshot-ai/kimi-code-sdk)
  • Ran gen-docs skill, or this PR needs no doc update. (en/zh config-files.md updated)

A 429 caused by an exhausted account quota or insufficient balance
(Moonshot error.type "exceeded_current_quota_error", OpenAI
"insufficient_quota") can never succeed on retry, yet it was classified
as APIProviderRateLimitError and silently retried for the whole budget
(10 attempts, ~3 minutes of backoff) with no UI feedback — the session
appeared frozen on every request.

Introduce APIProviderQuotaExhaustedError, minted in
normalizeAPIStatusError from the structured body error.type/error.code
forwarded by convertOpenAIError, with billing-anchored message patterns
as a fallback for gateways that flatten the body to text. The new class
is excluded from isRetryableGenerateError (fail fast, even when a
retry-after header is present) and from isProviderRateLimitError (no
swarm requeue/suspend). toKimiErrorPayload and translateProviderError
map it to provider.api_error (retryable: false) instead of
provider.rate_limit, and classifyApiError reports it as
quota_exhausted in telemetry. agent-core-v2 mirrors the same fix.

Transient rate-limit 429s keep the existing retry, backoff, and
Retry-After behavior (verified end-to-end against a mock provider:
quota body fails after attempt 1/10; rate-limit body still walks the
full 10-attempt ladder).

Behavior changes to note: quota-failed swarm subagents now fail
instead of suspending indefinitely as "Rate limited...", and quota
errors cross the wire as provider.api_error rather than
provider.rate_limit.
@changeset-bot

changeset-bot Bot commented Jul 17, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 7ceabcc

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
@moonshot-ai/kimi-code Patch
@moonshot-ai/kimi-code-sdk Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f585cbb4f7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/agent-core-v2/src/app/llmProtocol/errors.ts Outdated
Comment thread packages/kosong/src/providers/openai-common.ts
vinlee19 added 2 commits July 17, 2026 23:28
Responses response.failed / error SSE events carry no HTTP status and
were minted by errorFromOpenAIResponsesEvent as either a rate-limit
error (rate_limit_exceeded / embedded status_code=429) or a base
ChatProviderError — and the base class falls into the retryable
unclassified-failure fallback, so an insufficient_quota event still
burned the whole retry budget on the openai_responses path. Route the
event code and message through the same quota-exhausted check before
the rate-limit branch, in kosong and the agent-core-v2 mirror. Covers
all three entry paths (error events, response.failed, nested gateway
frames) since they share the single converter.
…rule

agent-core-v2 comments live solely in the top-of-file block, never
beside functions or statements; the kosong twins keep the full
rationale.
@vinlee19

Copy link
Copy Markdown
Author

Pushed two follow-up commits after a local review pass:

  • 5c60c91 — the openai_responses provider path was still burning the retry budget on quota exhaustion: Responses response.failed/error SSE events carry no HTTP status and were minted as a base ChatProviderError (whose unclassified fallback is retryable) or, with an embedded status_code=429, as a retryable rate limit. errorFromOpenAIResponsesEvent now routes the event code/message through the same quota-exhausted check before the rate-limit branch (kosong + agent-core-v2 mirror; all three entry paths share that converter). Tests added for both engines.
  • 7ceabcc — dropped the inline comments my first commit added in agent-core-v2 sources, per its AGENTS.md header-only comment convention; the kosong twins keep the full rationale.

@vinlee19

Copy link
Copy Markdown
Author

Hi @wbxl2000 , PTAL 🙏 — this one bit us for real: our quota ran out and every prompt turned into what looked like a ~3-minute freeze. I attached the Node inspector to the
stuck process and dumped JS stacks — the event loop was idle apart from the render tick, which led me to the silent 429 retry ladder. Analysis + captured 429 body:
#1860

Fixes: #1857 (fail fast on quota-exhausted 429) and #1861 (show retry progress in
the TUI). Feedback welcome!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant