fix: guard should_replace against losing completed_at on same-run - #81
Merged
HereThereBeDragons merged 3 commits intoAug 17, 2026
Merged
Conversation
…_replace_completed_at_guard
…_replace_completed_at_guard
HereThereBeDragons
approved these changes
Aug 17, 2026
HereThereBeDragons
deleted the
users/cgoea/should_replace_completed_at_guard
branch
August 17, 2026 17:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
A finalized test result could lose its
completed_attimestamp to a later, less-complete snapshot of the same run.The cause is in
should_replace. A leaf/cell counts as terminal (is_terminal()) as soon as its status is terminal, independent ofcompleted_at. But multiple notify events for the same shared-entry run each re-fetch and re-derive the whole job list, and those snapshots can disagree oncompleted_at: a cell's own notify step sees its job still running (nocompleted_atfor itself), while a later sibling cell's snapshot, fetched after this cell actually finished, captures it. When the earlier, incomplete snapshot is processed after the complete one (same run_id/attempt), both are "terminal by status", so the existing do-not-downgrade guards don't fire, and the incomplete update overwrites the goodcompleted_atwithNone.Technical Details
Variant.should_replaceandRunLeaf.should_replaceintherock_status_document.py: within the same run_id/attempt, an update that has nocompleted_atcannot replace one that does.should_replaceis already the sole gate consulted byupsert_leaf.Test Plan
test_same_attempt_terminal_with_completed_at_beats_terminal_without- existing terminal leaf withcompleted_atis not replaced by a same-run/attempt terminal update withcompleted_at=None.test_same_attempt_terminal_without_completed_at_replaced_by_one_with- the reverse still replaces (a snapshot gainscompleted_at).test_variant_same_attempt_terminal_with_completed_at_not_downgraded- same guard on theVariantpath.test_merge_does_not_lose_completed_at_from_later_incomplete_snapshot- end-to-end reproduction of the production race through_merge_variant_leaf.scripts/receive_therock/tests/suite.Test Result
All tests pass.