fix: bound status.json push retries by wall-clock deadline - #79
Open
cgoea wants to merge 1 commit into
Open
Conversation
…pt count Replace the fixed-attempt-count loop in update_status_json with a wall-clock deadline (QUARTZ_STATUS_PUSH_MAX_WAIT_SEC, default 15 minutes): each attempt still fetches and hard-resets onto fresh upstream before reapplying and pushing, exactly as before, but the loop now runs until the deadline elapses rather than a capped attempt count.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Replace the fixed-attempt-count loop in update_status_json with a wall-clock deadline (QUARTZ_STATUS_PUSH_MAX_WAIT_SEC, default 15 minutes): each attempt still fetches and hard-resets onto fresh upstream before reapplying and pushing, exactly as before, but the loop now runs until the deadline elapses rather than a capped attempt count.
A large release fan-out has dozens of runs pushing to the same
status.jsonref within a minute or two._max_retriescapped that at a fixed number of attempts (12, each backing off up toBACKOFF_CAP_SEC), so a run that kept losing the push race for long enough would exhaust its budget, permanently dropping its update.Technical Details
RuntimeErroron deadline exhaustion now report elapsed attempts and wall-clock time (e.g."Failed to push status.json after {attempt} attempts over {max_wait_seconds:.0f}s (deadline exceeded)") instead ofattempt/max_retries.BACKOFF_BASE_SEC/BACKOFF_CAP_SEC/INITIAL_JITTER_SECtuning values are left untouched -- this PR only changes the retry loop's stopping condition, not its backoff shape.Test Plan
test_max_retries_env_override/test_max_retries_invalid_env_falls_back→test_max_wait_seconds_env_override/test_max_wait_seconds_invalid_env_falls_backto cover_max_wait_seconds()'s env var parsing/validation/fallback behavior.test_update_status_json_retries_until_deadline_then_raises: with_max_wait_secondspatched to a tiny deadline and_commit_and_pushalways returningRETRY, assertsupdate_status_jsonkeeps retrying until the deadline elapses and then raisesRuntimeErrormatching"deadline exceeded".test_update_status_json_succeeds_after_transient_retries: with_commit_and_pushreturningRETRY, RETRY, DONE, asserts the loop retries past transient race losses and returns the written status path on eventual success.Test Result
scripts/receive_therock/tests/suite to confirm no other tests assumed a fixed attempt count.