Gmail watcher should honor 429 retry-after and circuit-break per account

[Countermarch]

Environment

• gogcli 0.15.0
• Trigger path: gog gmail watch serve handling Gmail Pub/Sub push delivery
• Originally surfaced through OpenClaw 2026.5.22 (a374c3a) supervising the external gog watcher process

Impact

A parked legacy Gmail watcher repeatedly hit Gmail 429 rateLimitExceeded during push handling. The Gmail retry-after timestamp kept moving forward because the handler continued receiving Pub/Sub retries and making Gmail API calls instead of circuit-breaking the affected account.

Evidence

Runtime logs repeatedly showed push-handler failures like:

watch: handle push failed: googleapi: Error 429: User-rate limit exceeded. Retry after <redacted-timestamp>, rateLimitExceeded

Additional observations:

• The errors were from push handling, not only manual watch start attempts.
• The caller had already paused the legacy account in the supervising app config.
• The legacy Pub/Sub subscription was absent locally.
• gog gmail watch status --account <redacted-legacy-account> still showed a stored server-side Gmail watch expiring May 31, 2026.
• gog gmail watch stop --account <redacted-legacy-account> --force succeeded and cleared the stale local watch state.

Repository boundary

This was first filed as openclaw/openclaw#86610, but review there found OpenClaw only configures, starts, renews, and supervises the external gog gmail watch serve process. The repeated history.list / messages.get fetches are in the gogcli push handler, so this issue belongs here.

Expected behavior

When Gmail returns 429 with a retry-after value, the watcher should pause Gmail API fetches for that account until after the retry-after time. Pub/Sub push handling should not continue making history.list / messages.get calls on every redelivery while the account is already rate-limited.

Suggested fix

• Add a per-account 429 circuit breaker in gog gmail watch serve.
• Persist the retry-after timestamp in watch state.
• While the circuit is open, return a controlled response and avoid Gmail API calls for that account.
• Consider a lower-cost metadata-only mode for low-importance watchers.
• Add an explicit command to pause/stop the server-side watch while keeping local config parked.

Local recovery used

Stopped the stale server-side watch for the parked legacy account and left only the intended workspace watcher active. After recovery, the parked account had no local listener, no legacy Pub/Sub subscription, and no stored gog watch state.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed May 25, 2026, 7:46 PM ET / 23:46 UTC.

Summary
Keep open: current main still lets Gmail watch push handling bubble 429s back as handler failures, and the existing Retry-After logic is only per HTTP request rather than a persisted per-account watch circuit.

Reproducibility: no. live 429 reproduction was run, but source inspection gives a high-signal reproduction path: make history.list or messages.get return Gmail 429 during watch push handling and the current handler returns an error without opening a persisted account circuit.

Ways to help us reproduce this

• Add a screenshot or short recording showing the behavior.
• Add expected vs actual behavior.

Next step
This is a focused reliability repair in existing Gmail watch code with clear source evidence, likely files, and unit-test coverage points.

Review details

Best possible solution:

Add a focused watch-level per-account retry-until circuit that is checked before Gmail API fetches, persisted in watch state, and cleared after successful processing resumes.

Do we have a high-confidence way to reproduce the issue?

No live 429 reproduction was run, but source inspection gives a high-signal reproduction path: make history.list or messages.get return Gmail 429 during watch push handling and the current handler returns an error without opening a persisted account circuit.

Is this the best way to solve the issue?

Yes, a narrow watch-layer circuit is the right repair for the central bug; the optional metadata-only mode and separate pause command should stay out of the first fix unless maintainers explicitly expand scope.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• I did not run a live Gmail 429 reproduction in this read-only review; the source path is clear, but a fix should still include live Google proof or state the exact credential/account blocker.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 9987f8b2f053.

Label changes

Label changes:

• add P2: This is a normal-priority reliability bug in an existing Gmail watcher path with limited blast radius but real push-delivery impact under rate limits.
• add impact:message-loss: When the watcher repeatedly fails push handling under 429, Gmail changes may not be forwarded to the configured hook until manual recovery or a later successful fetch.
• add issue-rating: 🐚 platinum hermit: Current issue advisory state selects this label.
• add clawsweeper:needs-live-repro: Current issue advisory state selects this label.
• add clawsweeper:queueable-fix: Current issue advisory state selects this label.
• add clawsweeper:fix-shape-clear: Current issue advisory state selects this label.

Label justifications:

• P2: This is a normal-priority reliability bug in an existing Gmail watcher path with limited blast radius but real push-delivery impact under rate limits.
• impact:message-loss: When the watcher repeatedly fails push handling under 429, Gmail changes may not be forwarded to the configured hook until manual recovery or a later successful fetch.

Evidence reviewed

Acceptance criteria:

• go test ./internal/cmd -run 'GmailWatch(Server|Serve|State)'
• go test ./internal/googleapi -run 'RetryTransport|CircuitBreaker'
• go test ./internal/cmd ./internal/googleapi

What I checked:

• Issue evidence: The report includes repeated push-handler failures with Gmail 429 rateLimitExceeded and a moving Retry-After timestamp while the watcher continued handling Pub/Sub redeliveries.
• Push handler still calls Gmail before any account circuit check: handlePush computes a start history ID, sleeps, creates a Gmail service, calls Users.History.List, and returns non-stale errors to ServeHTTP; ServeHTTP logs the failure and responds 500. (internal/cmd/gmail_watch_server.go:150, 9987f8b2f053)
• Message fetches also return Gmail errors without watch-level circuit state: fetchMessages loops over message IDs and returns any non-404 Users.Messages.Get error, so a 429 during message fetch also escapes the push handler. (internal/cmd/gmail_watch_server.go:292, 9987f8b2f053)
• Watch state has no persisted retry-until or circuit field: gmailWatchState stores account, topic, labels, history IDs, delivery status, and last push ID, but no Retry-After/circuit-break timestamp for the account. (internal/cmd/gmail_watch_types.go:27, 9987f8b2f053)
• Generic transport retry is not the requested watch circuit: RetryTransport honors Retry-After while retrying a single 429 response, but its circuit breaker is recorded only for 5xx errors and is not persisted in Gmail watch state. (internal/googleapi/transport.go:89, b25a3c029b37)
• No watch-specific rate-limit handling found: A targeted search of Gmail watch source, tests, and docs found no Retry-After, 429, TooManyRequests, rateLimitExceeded, or circuit handling in the watch layer. (9987f8b2f053)

Likely related people:

• Peter Steinberger: Current blame and path history for the Gmail watch handler, watch state, and retry transport all point to the v0.19.0 release commit authored by Peter Steinberger; the shallow/grafted history limits finer attribution. (role: feature-history owner; confidence: medium; commits: b25a3c029b37; files: internal/cmd/gmail_watch_server.go, internal/cmd/gmail_watch_state.go, internal/cmd/gmail_watch_types.go)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.