orchestrator: severity-aware dispatch — stop re-dispatching responder for nits

[microsasa]

Problem

Copilot code review drip-feeds comments across multiple rounds (~2 per pass). The orchestrator currently dispatches the responder on every round regardless of comment severity. This burns responder attempts on trivial nits (naming, style, "consider renaming") and often exhausts the 3-attempt limit, marking PRs as aw-pr-stuck:review when the remaining comments are cosmetic.

Proposed Solution

After the responder's first attempt, the orchestrator should classify NEW comments from Copilot's re-review before dispatching again:

• Bugs / security / correctness → dispatch responder
• Style nits / naming / "consider" suggestions → stop dispatching, label PR ready for human merge

This requires the orchestrator to:

1. Read the new unresolved review threads
2. Classify each by severity (LLM judgment call)
3. Only dispatch if high-severity comments exist
4. Otherwise, add a label like aw-review-nits-only and leave a summary comment

Complexity

This is a non-trivial change:

• Touches the orchestrator workflow (most critical pipeline component)
• "Nit vs real issue" classification is subjective — needs careful prompt engineering
• May need a confidence threshold to avoid suppressing real issues
• Should be gated behind the fix-forward responder improvement (see fix-forward: reduce drip-feed review rounds for implementer and responder #520) — if fix-forward reduces drip-feed rounds, severity tracking may be unnecessary

Research Findings (April 2026)

Deep investigation into Copilot code review behavior:

The drip-feed is by design, not a bug

• GitHub's own data: average 5.1 comments per review, 29% of reviews produce zero comments
• Comment count is driven by internal confidence scoring and clustering, not a hard cap
• GitHub's blog explicitly states: "more comments don't necessarily mean a better review"
• The ~2 comments/pass pattern appears to be an internal behavioral limit that varies by PR size and risk assessment

No configuration exists to change it

• No API endpoint, repo setting, org setting, or ruleset option controls comment volume
• copilot-instructions.md controls what gets flagged but cannot override the internal batching algorithm
• GitHub's own blog warns against instructions like "be more thorough" or "identify all issues" — says it "adds noise"
• Instructions have a 4,000 character limit for code review (does not apply to chat/coding agent)
• Model switching is not supported — "purpose-built product with carefully tuned mix of models"

Community reports confirm the problem

• GitHub Community Discussion #189767: "Copilot Code Review generates new comments on every push" — 5 rounds of review on a ~9-file PR, ~21 of 24 total comments were low-value nits
• Discussion #152385: Users report Copilot reviewing 120 files but generating only 1-2 comments, with re-requests surfacing comments on different files
• No official GitHub staff response addressing the drip-feed behavior specifically

Platform improvements (not user-configurable)

Date	Change
May 2025	"80% more comments per PR"
Jul 2025	Lifted file count/complexity limits for large PRs
Oct 2025	Agentic architecture with full project context + tool calling
Oct 2025	CodeQL/ESLint deterministic detections supplement LLM

Workarounds others have tried

• copilot-cli-codereviewer action (github.com/hancengiz/copilot-cli-codereviewer): Bypasses native review entirely — pipes git diff into gh copilot explain, posts one monolithic comment. Nuclear option.
• Angie Jones at Block/Goose: Uses confidence thresholds (">80%") in instructions — reduces noise but does not increase volume
• Path-specific instruction files (*.instructions.md with applyTo: frontmatter): Focus Copilot on what matters per file type
• excludeAgent: "coding-agent" in instruction frontmatter: Create review-only instructions
• Disable "Review new pushes" in rulesets: Stops the re-review-on-push loop entirely (but loses automatic re-review)

Key documentation

• GitHub Blog: Master your instructions files
• GitHub Docs: Copilot code review
• GitHub Docs: Responsible use of code review
• GitHub Blog: 60M Copilot code reviews
• GitHub Changelog: Large PR handling

Instructions file facts

• copilot-instructions.md applies to code review (first 4K chars, from base branch — though we empirically disproved the base-branch claim in PR test: verify Copilot follows guidelines link (clean retest) #479)
• Path-specific files: .github/instructions/NAME.instructions.md with applyTo: glob
• No copilot-review-instructions.md exists as a recognized file
• You cannot: change comment formatting, modify PR overview, block merging, or follow external links

Depends On

• Responder fix-forward improvement (fix-forward: reduce drip-feed review rounds for implementer and responder #520) — try that first, measure impact
• If drip-feed rounds drop significantly, this issue may become low priority

Acceptance Criteria

• Orchestrator classifies new review comments by severity before dispatching
• High-severity (bugs, security, correctness) → dispatch responder
• Low-severity (nits, style, naming) → label PR, leave summary, skip dispatch
• Metric: reduction in aw-pr-stuck:review labels caused by nit exhaustion