feat(seer): Curate autofix issue labels - #124834
Mihir-Mavalankar wants to merge 5 commits into
Conversation
Mihir-Mavalankar
commented
Sep 17, 2026
Mihir-Mavalankar
commented
- Sample low-scoring issues and control cohorts after Night Shift, then use an Opus judge to record blinded, event-matched fixability reviews with a per-organization run cap to limit cost.
Sample low-scoring issues and control cohorts after Night Shift, then use an Opus judge to record blinded, event-matched fixability reviews with a per-organization cost cap.
github-actions
Bot
added
the
Scope: Backend
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![sentry-warden[bot]](https://avatars.githubusercontent.com/in/2755838?s=60&v=4){kind=small}
| namespace=seer_tasks, | ||
| processing_deadline_duration=5 * 60, | ||
| ) | ||
| def schedule_judging_for_org(organization_id: int) -> None: |
There was a problem hiding this comment.
I think this could schedule overlapping jobs that race against each other since jobs get sharded into multiple runs. It might be best to limit the judge to the issues scoped in the delivery result, rather than all unscored issues for the org.
There was a problem hiding this comment.
The bugbot pointed that out here: #124834 (comment)
It's now in _complete_run which should run once per org right?
There was a problem hiding this comment.
Changed the approach a bit to launch this task after all shards are done to address the bugbots comment above too. Lmk what you think
| ) | ||
| ) | ||
| ) | ||
| bottom = list( |
There was a problem hiding this comment.
nit: is splitting it out like this needed?
There was a problem hiding this comment.
Simplified it
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
| return | ||
|
|
||
| run_extras["autofix_issue_data_judging_scheduled"] = True | ||
| locked_run.update(extras=run_extras) |
There was a problem hiding this comment.
Bug: A transient task dispatch failure will permanently prevent judging because the autofix_issue_data_judging_scheduled state flag is not reverted on error.
Severity: MEDIUM
Suggested Fix
To ensure atomicity, move the task dispatch call schedule_judging_for_org.apply_async(...) inside the transaction.atomic block. This will ensure that if the task dispatch fails, the entire database transaction, including the update to the run_extras flag, is rolled back.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: src/sentry/seer/night_shift/delivery.py#L161-L164
Potential issue: In `_schedule_judging_after_delivery`, the state flag
`autofix_issue_data_judging_scheduled` is set to `True` and committed to the database
before the `schedule_judging_for_org` task is dispatched. If the task dispatch fails due
to a transient issue, such as broker unavailability, the exception is caught and logged,
but the state flag is not reverted. A check at the beginning of the function prevents it
from running again if this flag is set, meaning a temporary dispatch failure will
permanently prevent the judging task from being scheduled for that run.
Did we get this right? 👍 / 👎 to inform future reviews.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 982224b. Configure here.
| or run_extras.get("autofix_issue_data_judging_scheduled") | ||
| or completed_deliveries != locked_run.executions.count() | ||
| ): | ||
| return |
There was a problem hiding this comment.
Missing shard delivery blocks judging
Medium Severity
Judging is dispatched only when every execution row has autofix_issue_data_delivery_completed. A shard that never callbacks, or a success path that fails inside _process_verdicts before _schedule_judging_after_delivery, leaves the count short forever, so captured issues from the other shards are never reviewed.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 982224b. Configure here.
There was a problem hiding this comment.
This is acceptable if it happens rarely.
|
Closing this PR for different approach |
2 participants
