feat(intake): count errored missing scores as 0 in evaluator rollups [ASE-616]

[shanaiabuggy]

Closes ASE-616

Why

Evaluator score rollups (the aggregate means in the Experiments table) averaged only over scored runs — the score SQL INNER JOINs evaluator_results, so a run with no score was dropped from the denominator. That inflates score means (errored runs excluded instead of counted as failures) and makes means non-comparable across experiments (the denominator varies per metric/experiment). Our standard — and Harbor's / switchyard-bench's — is a fixed denominator with no-score counted as 0.

Cost/latency are different: a missing value there is unmeasured, not zero, so those keep excluding missing. The rollup already computes scores and cost/latency on separate SQL paths, so this change is localized to the score path.

Change

Backend — services/intake/src/nmp/intake/spans/evaluation_rollup_repository.py

• _score_rollups_sql now uses a fixed denominator, keyed on the session's root_status:
  • errored run, no score → 0 (a real failed attempt)
  • succeeded run, no score → excluded (the evaluator wasn't run on it; counting 0 would wrongly penalize partial-coverage evaluators)
  • an errored run that did record a score keeps it
• Implemented via UNION ALL (scored sessions) + LEFT ANTI JOIN against the scored set (errored-unscored → 0). _scoped_sessions_sql now selects root_status.
• _metric_rollups_sql (cost/latency) is unchanged.

Studio — the existing aggregate-metric tooltip on score cells appends "Errored runs with no score count as 0." No new popovers/indicators (per the ticket).

Validation

Ran the rollup integration tests against real ClickHouse (testcontainers):

• 7/7 pass, including a new case seeding a scored + errored-unscored + succeeded-unscored run → mean = avg(1.0, 0) = 0.5 over 2 test cases (the succeeded-unscored one excluded).
• Existing 6 rollup tests unchanged (no regression to scored-only behavior).
• Unit rollup test passes (row shape unchanged); ruff + ty clean; Studio typecheck + eslint clean.

Scope notes

• Only root_status = 'error' is treated as a failed→0 attempt; cancelled/unknown/success-with-no-score are excluded (conservative — doesn't penalize ambiguous states).
• The non-zero-based rating case (e.g. 1–5, where missing ≠ 0) is out of scope — a gap in Harbor too; a future per-metric "aggregation kind" flag if such an evaluator appears.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Corrected experiment evaluation rollups so errored runs without scores contribute zero to mean calculations.
  • Successful runs without scores remain excluded from the calculation denominator.
  • Improved consistency of score aggregation across session and evaluator results.

• UI Improvements

  • Updated evaluator score tooltips to clarify when errored, unscored runs are counted as zero.

[coderabbitai]

📝 Walkthrough

Walkthrough

Errored sessions without evaluator scores now contribute zeroes to evaluation rollups. Integration tests cover denominator behavior, and evaluator score tooltips explain when missing scores from errored runs count as zero.

Changes

Evaluation score rollups

Layer / File(s)	Summary
Score rollup SQL services/intake/src/nmp/intake/spans/evaluation_rollup_repository.py	Rollup SQL carries session status and adds explicit zero scores for errored sessions missing evaluator scores.
Rollup integration coverage services/intake/tests/integration/spans/test_experiment_rollups.py	Tests validate that errored missing scores count as zero, while successful missing scores are excluded from the reward denominator.
Evaluator tooltip display web/packages/studio/src/components/dataViews/ExperimentGroupDataView/MeanValueTooltipCell.tsx, web/packages/studio/src/components/dataViews/ExperimentGroupDataView/index.tsx	Evaluator score tooltips receive a missing-as-zero flag and explain the treatment of errored runs without scores.

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#753: Updates evaluator-score rollup weighting and aggregation in the same repository.

Suggested labels: fix

Suggested reviewers: ngoncharenko, briannewsom

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: evaluator rollups now count errored missing scores as zero.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch sbuggy/ase-616

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

web/packages/studio/src/components/dataViews/ExperimentGroupDataView/MeanValueTooltipCell.tsx (1)

31-36: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Make the new prop readonly.

Change it to readonly countsMissingAsZero?: boolean.

As per coding guidelines, “Use readonly for immutable properties in TypeScript interfaces and types.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@web/packages/studio/src/components/dataViews/ExperimentGroupDataView/MeanValueTooltipCell.tsx`
around lines 31 - 36, Update the countsMissingAsZero property in the
MeanValueTooltipCellProps interface to be readonly while preserving its optional
boolean type and existing documentation.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@services/intake/tests/integration/spans/test_experiment_rollups.py`:
- Around line 326-330: Add an errored session with a non-null score to the seeds
in the experiment rollup integration test, then extend the expected assertions
to verify that its recorded score is retained rather than replaced or
supplemented with zero. Keep the existing unscored errored and successful
unscored cases unchanged.

---

Nitpick comments:
In
`@web/packages/studio/src/components/dataViews/ExperimentGroupDataView/MeanValueTooltipCell.tsx`:
- Around line 31-36: Update the countsMissingAsZero property in the
MeanValueTooltipCellProps interface to be readonly while preserving its optional
boolean type and existing documentation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 92f2b99f-8664-4f2d-b9f9-8a87a11b1f59

📥 Commits

Reviewing files that changed from the base of the PR and between 425bfdb and 1ad64e2.

📒 Files selected for processing (4)

• services/intake/src/nmp/intake/spans/evaluation_rollup_repository.py
• services/intake/tests/integration/spans/test_experiment_rollups.py
• web/packages/studio/src/components/dataViews/ExperimentGroupDataView/MeanValueTooltipCell.tsx
• web/packages/studio/src/components/dataViews/ExperimentGroupDataView/index.tsx

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Cover errored sessions that already have a score.

This misses the anti-join protection: a regression that unions 0 alongside a real errored-session score still passes. Add an errored, scored seed and assert its recorded value is retained.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@services/intake/tests/integration/spans/test_experiment_rollups.py` around
lines 326 - 330, Add an errored session with a non-null score to the seeds in
the experiment rollup integration test, then extend the expected assertions to
verify that its recorded score is retained rather than replaced or supplemented
with zero. Keep the existing unscored errored and successful unscored cases
unchanged.

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	25701/32944	78.0%	62.7%
Integration Tests	14784/31593	46.8%	19.2%