Skip to content

fix: bound status.json push retries by wall-clock deadline - #79

Open
cgoea wants to merge 1 commit into
developfrom
users/cgoea/deadline_based_status_push_retry
Open

fix: bound status.json push retries by wall-clock deadline#79
cgoea wants to merge 1 commit into
developfrom
users/cgoea/deadline_based_status_push_retry

Conversation

@cgoea

@cgoea cgoea commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Motivation

Replace the fixed-attempt-count loop in update_status_json with a wall-clock deadline (QUARTZ_STATUS_PUSH_MAX_WAIT_SEC, default 15 minutes): each attempt still fetches and hard-resets onto fresh upstream before reapplying and pushing, exactly as before, but the loop now runs until the deadline elapses rather than a capped attempt count.

A large release fan-out has dozens of runs pushing to the same status.json ref within a minute or two. _max_retries capped that at a fixed number of attempts (12, each backing off up to BACKOFF_CAP_SEC), so a run that kept losing the push race for long enough would exhaust its budget, permanently dropping its update.

Technical Details

  • After a backoff sleep, the loop re-checks the deadline and breaks early rather than starting an attempt it doesn't have time budget left to meaningfully retry from.
  • Log messages and the final RuntimeError on deadline exhaustion now report elapsed attempts and wall-clock time (e.g. "Failed to push status.json after {attempt} attempts over {max_wait_seconds:.0f}s (deadline exceeded)") instead of attempt/max_retries.
  • BACKOFF_BASE_SEC / BACKOFF_CAP_SEC / INITIAL_JITTER_SEC tuning values are left untouched -- this PR only changes the retry loop's stopping condition, not its backoff shape.

Test Plan

  • Updated test_max_retries_env_override / test_max_retries_invalid_env_falls_backtest_max_wait_seconds_env_override / test_max_wait_seconds_invalid_env_falls_back to cover _max_wait_seconds()'s env var parsing/validation/fallback behavior.
  • Added test_update_status_json_retries_until_deadline_then_raises: with _max_wait_seconds patched to a tiny deadline and _commit_and_push always returning RETRY, asserts update_status_json keeps retrying until the deadline elapses and then raises RuntimeError matching "deadline exceeded".
  • Added test_update_status_json_succeeds_after_transient_retries: with _commit_and_push returning RETRY, RETRY, DONE, asserts the loop retries past transient race losses and returns the written status path on eventual success.

Test Result

  • Ran the full scripts/receive_therock/tests/ suite to confirm no other tests assumed a fixed attempt count.

…pt count

Replace the fixed-attempt-count loop in update_status_json with a
wall-clock deadline (QUARTZ_STATUS_PUSH_MAX_WAIT_SEC, default 15
minutes): each attempt still fetches and hard-resets onto fresh
upstream before reapplying and pushing, exactly as before, but the loop
now runs until the deadline elapses rather than a capped attempt
count.
@cgoea
cgoea requested a review from a team August 14, 2026 13:42
@cgoea cgoea changed the title fix: bound status.json push retries by wall-clock deadline, not attem… fix: bound status.json push retries by wall-clock deadline Aug 14, 2026
@cgoea
cgoea changed the base branch from main to develop August 14, 2026 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant