Skip to content

fix(worker): Eliminate N+1 queries in ProcessFlakesTask#1033

Open
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/worker-n-plus-1-flakes-task-DhFxfP
Open

fix(worker): Eliminate N+1 queries in ProcessFlakesTask#1033
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/worker-n-plus-1-flakes-task-DhFxfP

Conversation

@sentry
Copy link
Copy Markdown
Contributor

@sentry sentry Bot commented Jun 7, 2026

This PR addresses an N+1 query issue identified in app.tasks.flakes.ProcessFlakesTask.

Problem:
Previously, the ProcessFlakesTask was inefficiently querying the database in two main areas:

  1. Flake fetching: fetch_current_flakes(repo_id) was called inside process_flakes_for_commit for every commit being processed. Since the repo_id is constant for all commits within a single task execution, this resulted in N identical queries to fetch the same set of flakes.
  2. Testrun fetching: get_testruns(upload) was called for each individual upload within process_single_upload, leading to M separate queries for testruns associated with a commit's uploads.

This pattern led to a high number of database queries, impacting performance.

Solution:

  1. Hoist fetch_current_flakes: The call to fetch_current_flakes(repo_id) has been moved from process_flakes_for_commit to process_flakes_for_repo. This ensures that the current flakes for a given repository are fetched only once per ProcessFlakesTask execution, and the resulting curr_flakes dictionary is passed down to process_flakes_for_commit and process_single_upload.
  2. Batch-fetch testruns: A new function, get_testruns_for_uploads, was introduced to fetch all relevant testruns for a list of uploads in a single database query using upload_id__in. The process_single_upload function was updated to accept these pre-fetched testruns, eliminating the per-upload query.
  3. Consolidate bulk updates: The Testrun.objects.bulk_update call, which was previously executed per upload, is now consolidated into a single call at the end of process_flakes_for_commit for all modified testruns related to a commit.

Impact:
These changes significantly reduce the number of database queries executed by ProcessFlakesTask, improving its efficiency and reducing database load. The flake fetching is now O(1) per task, and testrun fetching is O(1) per commit (instead of O(N commits) and O(N commits * M uploads) respectively).

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. In 2022 this entity acquired Codecov and as result Sentry is going to need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

Fixes WORKER-YQ9

@sentry
Copy link
Copy Markdown
Contributor Author

sentry Bot commented Jun 7, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.89%. Comparing base (508927b) to head (a5f3423).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1033      +/-   ##
==========================================
- Coverage   91.90%   91.89%   -0.01%     
==========================================
  Files        1318     1318              
  Lines       50711    50720       +9     
  Branches     1625     1625              
==========================================
+ Hits        46604    46611       +7     
- Misses       3801     3803       +2     
  Partials      306      306              
Flag Coverage Δ
workerintegration 58.53% <11.76%> (-0.03%) ⬇️
workerunit 90.34% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@codecov-notifications
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants