Give the procedure watcher a separate staleness window for Held - #793
Merged
Conversation
Running and Held shared one hour-long threshold (procedure_watcher_stale_after_seconds), and Running alone got a second chance via read_procedure_activity_recency before flagging. A Held conduct accepts no activity, so it has no such second chance and structurally cannot have one: any deliberate operator pause past an hour (a bakeout, waiting on beam, waiting on a collaborator) wrote a real cross-BC Decision(context=ProcedureProgress, choice=Stall). Add procedure_watcher_held_stale_after_seconds (default 604800.0, a week) and select the threshold per item's status in _watch_tick. This mirrors campaign_watcher_stale_after_seconds, which already documents a week as how long a campaign may sit Held. No new ProcedureStatus member, hold_origin discriminator, or event field is needed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Coverage reportClick to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
ProcedureWatcherflags a stalled conduct by writingDecision(context=ProcedureProgress, choice=Stall). It watchedRunningandHeldagainst one threshold,procedure_watcher_stale_after_seconds, defaulting to an hour.That works for
Running, which also gets a second chance: if the status timestamp looks stale, the watcher re-checks the latest activityrecorded_atand stays quiet if the procedure is still logging steps.Heldgets no such second chance, and structurally cannot, because a held conduct accepts no activity entries. So staleness came purely fromlast_status_changed_at, and any hold past an hour raised a stall alarm. An operator pausing a conduct overnight for a bakeout, or to wait on beam or on a collaborator, produced a false alarm. Because the alarm is a Decision, it is a durable cross-BC record, not a log line.The fix
A separate
procedure_watcher_held_stale_after_seconds, defaulting to a week, selected per item by status.Runningbehaviour is untouched, second chance included.The default follows the sibling precedent rather than inventing one:
campaign_watcher_stale_after_secondsis already a week and is already documented as "how long a campaign may sit Held." The new setting carries its own@field_validatormatching the others in that file.The module docstring now says why the two statuses need different clocks, so the next reader does not have to rediscover it.
Verification
test_tick_does_not_flag_held_overnight_pauseproves a 12 hour hold stays quiettest_tick_flags_stale_held_without_folding_activitystill proves a genuinely stalledHelddoes flag, with an explicit short window so it does not need to simulate a weektests/unit/operation, 37,444tests/architecture, ruff and pyright cleanTradeoff worth stating
A week-long window trades a false positive for a slower true positive: a conduct parked by the machine rather than by an operator now sits unnoticed for longer. Both live in one bucket today, so the trade is forced. Labelling why a hold happened would dissolve it and let each origin carry its own window; that is a separate design question and deliberately not in this PR.
🤖 Generated with Claude Code