Skip to content

fix: guard should_replace against losing completed_at on same-run - #81

Merged
HereThereBeDragons merged 3 commits into
developfrom
users/cgoea/should_replace_completed_at_guard
Aug 17, 2026
Merged

fix: guard should_replace against losing completed_at on same-run#81
HereThereBeDragons merged 3 commits into
developfrom
users/cgoea/should_replace_completed_at_guard

Conversation

@cgoea

@cgoea cgoea commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Motivation

A finalized test result could lose its completed_at timestamp to a later, less-complete snapshot of the same run.

The cause is in should_replace. A leaf/cell counts as terminal (is_terminal()) as soon as its status is terminal, independent of completed_at. But multiple notify events for the same shared-entry run each re-fetch and re-derive the whole job list, and those snapshots can disagree on completed_at: a cell's own notify step sees its job still running (no completed_at for itself), while a later sibling cell's snapshot, fetched after this cell actually finished, captures it. When the earlier, incomplete snapshot is processed after the complete one (same run_id/attempt), both are "terminal by status", so the existing do-not-downgrade guards don't fire, and the incomplete update overwrites the good completed_at with None.

Technical Details

  • Added one guard to both Variant.should_replace and RunLeaf.should_replace in therock_status_document.py: within the same run_id/attempt, an update that has no completed_at cannot replace one that does.
  • Scope is narrow: only the same-run/attempt, both-terminal-by-status case. All other precedence (run_id, run_attempt, terminal-vs-nonterminal) is unchanged.
  • No caller changes: should_replace is already the sole gate consulted by upsert_leaf.

Test Plan

  • test_same_attempt_terminal_with_completed_at_beats_terminal_without - existing terminal leaf with completed_at is not replaced by a same-run/attempt terminal update with completed_at=None.
  • test_same_attempt_terminal_without_completed_at_replaced_by_one_with - the reverse still replaces (a snapshot gains completed_at).
  • test_variant_same_attempt_terminal_with_completed_at_not_downgraded - same guard on the Variant path.
  • test_merge_does_not_lose_completed_at_from_later_incomplete_snapshot - end-to-end reproduction of the production race through _merge_variant_leaf.
  • Ran the full scripts/receive_therock/tests/ suite.

Test Result

All tests pass.

@cgoea
cgoea requested review from a team and HereThereBeDragons August 14, 2026 14:13
@HereThereBeDragons
HereThereBeDragons merged commit d394b94 into develop Aug 17, 2026
3 checks passed
@HereThereBeDragons
HereThereBeDragons deleted the users/cgoea/should_replace_completed_at_guard branch August 17, 2026 17:04
quartz-sync-github-app Bot pushed a commit that referenced this pull request Aug 17, 2026
d394b94, fix: guard should_replace against losing completed_at on same-run (#81), Ciprian Goea (ciprian.goea@amd.com), Mon Aug 17 20:04:08 2026 +0300
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants