fix(ci): exclude cancelled jobs from pr-test-summary failed count#1558
Open
ramakrishnap-nv wants to merge 2 commits into
Open
fix(ci): exclude cancelled jobs from pr-test-summary failed count#1558ramakrishnap-nv wants to merge 2 commits into
ramakrishnap-nv wants to merge 2 commits into
Conversation
Cancelled jobs (conclusion="cancelled") were counted as failures because the failed list was built as anything not passed or skipped. When one job fails, GitHub cancels downstream queued jobs, inflating the headline count (e.g. "17 failed" when only 1 test actually failed). Fix: match only conclusion=="failure" or "timed_out". Closes #1223 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
📝 WalkthroughWalkthroughThe PR test summary utility now distinguishes failed, skipped, passed, and cancelled test jobs, analyzes logs only for failures, and reports cancelled or incomplete jobs in the sticky PR comment. ChangesPR test status summary
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ci/utils/pr_test_summary.py`:
- Line 180: Update the job classification in _build_body() to track cancelled
and other non-terminal test jobs separately from failed, passed, and skipped;
when such jobs remain, render an excluded or not-completed status instead of the
“All ... test job(s) passed” message, while preserving existing failure
handling.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 83199161-67cd-4b70-aa0e-e5f502e9c7df
📒 Files selected for processing (1)
ci/utils/pr_test_summary.py
tmckayus
approved these changes
Jul 10, 2026
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ci/utils/pr_test_summary.py`:
- Around line 136-140: Update the no-failure cancelled summary in the
summary-generation function to include the skipped-job count alongside passed
and cancelled counts, preserving the existing separate classification and
formatting for all non-passed categories.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 91dc501f-6959-4ab2-a675-cfb9c779ddcd
📒 Files selected for processing (1)
ci/utils/pr_test_summary.py
Comment on lines
136
to
+140
| if not failed: | ||
| lines.append(f"✅ All {len(passed)} test job(s) passed.") | ||
| if cancelled: | ||
| lines.append( | ||
| f"✅ {len(passed)} passed · {len(cancelled)} cancelled / not completed" | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Preserve skipped jobs in the cancelled summary.
When a run has both skipped and cancelled jobs but no failures, this branch omits the skipped count even though those jobs are classified separately. The sticky comment should report all non-passed categories.
Proposed fix
if cancelled:
- lines.append(
- f"✅ {len(passed)} passed · {len(cancelled)} cancelled / not completed"
- )
+ summary = f"✅ {len(passed)} passed"
+ if skipped:
+ summary += f" · {len(skipped)} skipped"
+ summary += f" · {len(cancelled)} cancelled / not completed"
+ lines.append(summary)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if not failed: | |
| lines.append(f"✅ All {len(passed)} test job(s) passed.") | |
| if cancelled: | |
| lines.append( | |
| f"✅ {len(passed)} passed · {len(cancelled)} cancelled / not completed" | |
| ) | |
| if not failed: | |
| if cancelled: | |
| summary = f"✅ {len(passed)} passed" | |
| if skipped: | |
| summary += f" · {len(skipped)} skipped" | |
| summary += f" · {len(cancelled)} cancelled / not completed" | |
| lines.append(summary) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@ci/utils/pr_test_summary.py` around lines 136 - 140, Update the no-failure
cancelled summary in the summary-generation function to include the skipped-job
count alongside passed and cancelled counts, preserving the existing separate
classification and formatting for all non-passed categories.
CI Test Summary✅ All 9 test job(s) passed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
failedwas computed as "anything not passed or skipped", so cancelled jobs (e.g. downstream jobs queued when one job fails) were counted as failures.conclusion in ("failure", "timed_out").Closes #1223