fix: [no-ticket] harbor - floored rewards name their cause (candidate crash vs infra outage)

[shehabyasser-scale]

Stacked on #36. Motivated by a live wave-1 incident: in the E5 transfer matrix, three of five champions scored 0/72 on claude-opus-4-8 because their optimizers had hardcoded temperature=0 (a variance trick on their home model) and litellm rejects it for opus. Separately, a budget-exhausted key produced dead attempts that nearly got booked as a champion property. In both cases the harness behaved safely but the record could not distinguish the two floored cases, and they demand opposite actions (report the portability failure vs invalidate and re-run).

Changes

1. Collation names the crash (runner.py). The no-verifier-rewards error string now carries the dead attempts' exception types: No verifier rewards for task 't'. (attempts died: UnsupportedParamsError x6). The error string is the one field that flows everywhere (DB, per-sample files, verifier), so the cause survives to every consumer.

2. Floored targets carry their cause (verifier.py). _admin_eval_score now returns (score, failure_cause); when a target is floored after retries, target_errors[key] includes a frequency summary of the dominant per-sample errors (top 3). One identical cause across every sample is the signature of a deterministic candidate crash; a mixed bag points at infra. The cause extraction is fail-safe (any surprise result shape degrades to a fixed string) so diagnostics can never fail finalize.

Tests

• Runner: error string carries UnsupportedParamsError x2 for attempts dead of the same cause.
• Verifier: an all-errored target floors to 0.0 with target_errors naming the dominant cause and its sample count; existing floor/retry/recover tests unchanged.
• 85 passed across the two affected suites.

🤖 Generated with Claude Code

Greptile Summary

This PR improves observability for floored evaluation scores so operators can distinguish a deterministic candidate crash (e.g., hardcoded temperature=0 rejected by an executor) from an infrastructure outage — two outcomes that demand opposite responses but previously produced identical-looking records.

• Runner (runner.py): Dead attempts are now classified with [infra] / [infra:llm-key-budget] suffixes using a conservative allow-list; bracket neutralization prevents candidate-authored class names from spoofing the label. The no-rewards error string embeds a cause summary, a key-budget alarm fires at ERROR level, and a new n_dead_infra metric splits infra-dead zeros from candidate-caused ones. A bounded within-eval infra retry (infra_retry_rounds, off by default) re-measures only samples whose every attempt died of a transient infra cause.
• Verifier (verifier.py): _admin_eval_score now returns (score, cause); when a target floors after retries the dominant per-sample errors are included in target_errors. Baseline evals correctly receive the same executor-model override for like-for-like transfer comparisons.
• Transfer probe: A per-target model override flows end-to-end from config → serve.json → evaluate_admin via a shallow copy of run_constraints (shared instance not mutated).

Confidence Score: 5/5

Safe to merge; all new behaviour is opt-in, the fail-safe (diagnostics degrade rather than raise) is verified by test, and the zero-fill invariant is explicitly protected against infra-label abuse.

The infra retry is off by default and all adversarial-candidate attack vectors are closed and tested. The model_copy path in evaluate_admin is tested to be non-mutating. The _dominant_sample_errors method is wrapped in a blanket except so it can never surface through finalize. No existing test was weakened.

No files require special attention.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/runner.py	Core of the PR: adds dead-attempt classification, key-budget alarm, n_dead_infra metric, bounded infra retry loop, and cause-enriched no-rewards error strings. Bracket neutralization prevents spoofed [infra] suffixes; retry is off-by-default; recovered samples get an audit marker; sibling-dir collation prevents stale attempt pooling.
vero/src/vero/harbor/verifier.py	_admin_eval_score now returns (score, cause); _dominant_sample_errors groups per-sample errors after normalizing task names (re.sub) to prevent singleton buckets across a multi-task slice; baseline eval receives the same model override for like-for-like comparison.
vero/src/vero/evaluation/engine.py	evaluate_admin gains a model parameter; run_constraints is copied via model_copy before injecting harbor_model_override so the shared instance is never mutated.
vero/src/vero/harbor/config.py	Adds infra_retry_rounds (default 0) and infra_retry_delay_s (default 30 s) with post_init validation keeping the opt-in safety contract.
vero/tests/test_harbor_runner.py	New TestInfraResilience class (11 tests) covers config validation, infra labeling, bracket neutralization, key-budget alarm, retry off-by-default, outaged-sample re-run, crash/spent-key not retried, mixed-cause no-retry, persistent outage keeps error, partially-scored sample not retried, and multi-round audit history.
vero/tests/test_harbor_verifier.py	Adds three new tests: transfer model reaches champion+baseline evals, floored target records dominant crash cause, and cause clusters across task names. Existing tests updated to accept the new model= kwarg.

_{Reviews (3): Last reviewed commit: "Merge pull request #38 from scaleapi/har..." | Re-trigger Greptile}