fix(harbor): make baseline scoring at finalize durable and retried

[shehabyasser-scale]

Problem (observed live, then root-caused)

A trial silently skipped score_baseline at finalize: reward.json landed ~2 minutes after the winner's validation eval with no baseline.json and no baseline ledger row, while a byte-identical config produced one in another run. A three-agent investigation traced the mechanism: the nested baseline eval failed transiently within seconds, _maybe_score_baseline's warn-and-continue except swallowed it, and the only record (a logger.exception to sidecar stderr) was destroyed with the container at teardown. The admin volume holding baseline.json does not survive teardown either. Two subsequent identical runs scored the baseline fine, confirming the failure is intermittent and environmental, not config- or state-dependent.

Fix

• Retry: the baseline eval gets baseline_score_attempts total attempts (default 2, wired through ServeConfig), so a single transient nested-run failure no longer drops the regression check.
• Durable outcome: finalize() now returns {"rewards": ..., "baseline": ...} where baseline is {"scores": ...}, {"skipped": reason} or {"error": ..., "error_type": ..., "attempts": N}. The CLI writes only rewards to reward.json (the outer harness contract is unchanged) and echoes the full payload to stdout, which harbor captures into the trial's test-stdout.txt on the host: the one channel that survives teardown. The CLI tolerates the old bare-rewards response shape for a mixed-version sidecar.
• baseline.json is still written to the admin volume (best-effort) for live in-cluster debugging.

Baseline scoring still never fails the trial.

Tests

• transient failure then success: recovers on retry, reports attempts: 2
• persistent failure: retries, reports the error in the response, trial reward unaffected, nothing persisted
• CLI: wrapper response writes only rewards to reward.json and echoes the baseline to stdout; bare-rewards back-compat kept
• app-layer /finalize mock updated to the real wrapper shape
• existing finalize tests updated to the wrapper (serve integration suite passes)

Stacked on #25 (harbor-2-free-baseline-eval). An adversarial review panel (4 lenses + verification) ran pre-open; all confirmed findings addressed.

🤖 Generated with Claude Code

Greptile Summary

This PR fixes a silent baseline-scoring loss at finalize by adding retry logic and surfacing the outcome through a durable channel (stdout rather than the ephemeral admin volume). It also adds an auto_best_baseline_floor guard that prevents shipping a candidate that fails to beat the baseline on the selection split.

• Verifier.finalize() now returns {\"rewards\": ..., \"baseline\": ...}. The CLI writes only rewards to reward.json (harness contract unchanged) and echoes the full response to stdout, which survives container teardown. A plain-rewards response from older sidecars is handled via a back-compat fallback.
• _maybe_score_baseline() retries up to baseline_score_attempts times (default 2) and returns a structured {\"scores\"} / {\"error\"} / {\"skipped\"} dict instead of swallowing failures; baseline.json on the admin volume is demoted to a best-effort debug copy.
• A new auto_best_baseline_floor (default True) admin-scores base_commit on the selection split during _best_from_db and reverts to it if no candidate strictly improves over the baseline, closing the live regression observed with a weak inner model.

Confidence Score: 5/5

Safe to merge — the harness contract (reward.json shape) is unchanged, failures in baseline scoring still never affect the trial reward, and the back-compat path for older sidecars is tested.

The changes are well-scoped: the new wrapper return type is isolated to Verifier/CLI, retry logic is simple and bounded, and the floor feature is gated on a default-on flag that no-ops safely when base_commit is absent. Test coverage is thorough — retry recovery, persistent failure, floor revert, tie-break, and floor-off cases are all exercised.

No files require special attention; verifier.py carries the most logic but is well-covered by the new test suite.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/verifier.py	Core change: finalize() now returns a {"rewards", "baseline"} wrapper; _maybe_score_baseline() retries up to baseline_score_attempts times and returns a structured outcome instead of swallowing failures; new auto_best_baseline_floor reverts to base_commit when no candidate beats the baseline on the selection split.
vero/src/vero/harbor/serve.py	Adds baseline_score_attempts and auto_best_baseline_floor to ServeConfig and wires them through to Verifier; straightforward plumbing with no logic changes.
vero/src/vero/harbor/cli.py	Unwraps rewards from the new response shape for reward.json; echoes the full response (including baseline) to stdout; backward-compatible with bare-rewards responses from older sidecars.
vero/tests/test_harbor_verifier.py	Comprehensive new test coverage for the baseline floor (5 cases), retry behavior (transient failure recovery + full failure reporting), and updates to existing tests for the new wrapper shape.
vero/tests/test_harbor_cli.py	Adds a test verifying reward.json gets only rewards while stdout captures the full baseline outcome; back-compat test retained.
vero/tests/test_harbor_serve.py	Updated to unpack ["rewards"] from the new wrapper; no functional test logic changed.
vero/tests/test_harbor_app.py	Mock updated to the real finalize wrapper shape; assertion updated to check the full payload passthrough.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant CLI as finalize_cmd (cli.py)
    participant App as /finalize (app)
    participant V as Verifier.finalize()
    participant SC as _select_commit()
    participant BS as _maybe_score_baseline()
    participant E as EvaluationEngine

    CLI->>App: POST /finalize (Bearer token)
    App->>V: await finalize()
    V->>SC: await _select_commit()
    alt auto_best mode
        SC->>E: evaluate_admin (rescore shortlist)
        SC->>E: evaluate_admin (base_commit, floor check) [new]
        alt "best_score <= base_score"
            SC-->>V: return base_commit (reverted)
        else
            SC-->>V: return best_commit
        end
    else submit mode
        SC-->>V: return submitted commit
    end
    V->>E: evaluate_admin (winner, target splits)
    V->>BS: await _maybe_score_baseline(rewards)
    loop up to baseline_score_attempts times [new]
        BS->>E: evaluate_admin (base_commit, target splits)
        alt success
            BS-->>V: "{"scores": {...}, "attempts": N}"
        else transient failure
            Note over BS: log warning, retry
        end
    end
    alt all attempts failed
        BS-->>V: "{"error": ..., "error_type": ..., "attempts": N}"
    end
    V-->>App: "{"rewards": {...}, "baseline": {...}} [new shape]"
    App-->>CLI: HTTP 200 full response
    CLI->>CLI: extract resp["rewards"] to reward.json
    CLI->>CLI: echo full resp to stdout (durable channel)

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant CLI as finalize_cmd (cli.py)
    participant App as /finalize (app)
    participant V as Verifier.finalize()
    participant SC as _select_commit()
    participant BS as _maybe_score_baseline()
    participant E as EvaluationEngine

    CLI->>App: POST /finalize (Bearer token)
    App->>V: await finalize()
    V->>SC: await _select_commit()
    alt auto_best mode
        SC->>E: evaluate_admin (rescore shortlist)
        SC->>E: evaluate_admin (base_commit, floor check) [new]
        alt "best_score <= base_score"
            SC-->>V: return base_commit (reverted)
        else
            SC-->>V: return best_commit
        end
    else submit mode
        SC-->>V: return submitted commit
    end
    V->>E: evaluate_admin (winner, target splits)
    V->>BS: await _maybe_score_baseline(rewards)
    loop up to baseline_score_attempts times [new]
        BS->>E: evaluate_admin (base_commit, target splits)
        alt success
            BS-->>V: "{"scores": {...}, "attempts": N}"
        else transient failure
            Note over BS: log warning, retry
        end
    end
    alt all attempts failed
        BS-->>V: "{"error": ..., "error_type": ..., "attempts": N}"
    end
    V-->>App: "{"rewards": {...}, "baseline": {...}} [new shape]"
    App-->>CLI: HTTP 200 full response
    CLI->>CLI: extract resp["rewards"] to reward.json
    CLI->>CLI: echo full resp to stdout (durable channel)

Loading

_{Reviews (2): Last reviewed commit: "Merge pull request #28 from scaleapi/har..." | Re-trigger Greptile}

[greptile-apps]

logger.exception(..., exc_info=last_error) is called outside an except block with an exception instance as exc_info. In Python 3.11 (the minimum required by pyproject.toml), the logging.LogRecord constructor treats any non-tuple exc_info value by falling through to sys.exc_info(), which returns (None, None, None) when there is no active exception context — so the traceback of last_error is silently dropped from the final log message. Exception-instance support for exc_info was only added in Python 3.12. Passing an explicit 3-tuple makes this work on all supported versions.

Suggested change

	logger.exception(
	"baseline scoring failed after %d attempt(s); reward.json is unaffected",
	self._baseline_score_attempts,
	exc_info=last_error,
	)
	logger.error(
	"baseline scoring failed after %d attempt(s); reward.json is unaffected",
	self._baseline_score_attempts,
	exc_info=(type(last_error), last_error, last_error.__traceback__),
	)

Prompt To Fix With AI

This is a comment left during a code review.
Path: vero/src/vero/harbor/verifier.py
Line: 195-199

Comment:
`logger.exception(..., exc_info=last_error)` is called outside an `except` block with an exception instance as `exc_info`. In Python 3.11 (the minimum required by `pyproject.toml`), the `logging.LogRecord` constructor treats any non-tuple `exc_info` value by falling through to `sys.exc_info()`, which returns `(None, None, None)` when there is no active exception context — so the traceback of `last_error` is silently dropped from the final log message. Exception-instance support for `exc_info` was only added in Python 3.12. Passing an explicit 3-tuple makes this work on all supported versions.

```suggestion
        logger.error(
            "baseline scoring failed after %d attempt(s); reward.json is unaffected",
            self._baseline_score_attempts,
            exc_info=(type(last_error), last_error, last_error.__traceback__),
        )
```

How can I resolve this? If you propose a fix, please make it concise.