feat(harbor): the agent's first baseline eval is budget-free

[shehabyasser-scale]

Stacked on #20 (top of the sidecar chain: #11 -> #19 -> #20 -> this).

The problem, observed twice in live trials

The seeded baseline is the reference every candidate is implicitly compared to, yet it can never win selection (auto_best excludes base_commit, correctly, so "do nothing" cannot win). Metering its evaluation therefore forces the optimizer into a bad choice:

1. Pay for it: the incident behind fix(harbor): floor rewards on no-candidate outcome instead of erroring #11/docs(harbor): warn in the task instruction that baseline evals create no candidate #12, an optimizer spent its whole budget measuring the baseline, left the candidate pool empty, and the trial died (now floored by fix(harbor): floor rewards on no-candidate outcome instead of erroring #11, warned against by docs(harbor): warn in the task instruction that baseline evals create no candidate #12).
2. Skip it and fly blind: today's TB2 trial (exp5 v3). The optimizer, warned off baseline evals by docs(harbor): warn in the task instruction that baseline evals create no candidate #12's instruction, never learned the baseline scored 0.375. Its first edit scored 0.375, and with no reference it could not tell a no-op from an improvement; its second edit scored 0.125; it rationally gave up with 3 of its 5 budgeted evals unspent. Final reward: 0.375, exactly baseline.

#12 treated the symptom (don't waste budget); this treats the cause (the reference measurement should not cost budget).

The fix

EvaluationSidecar.evaluate: when the transferred commit resolves to base_commit and the free eval hasn't been used, run it through the engine's unmetered (admin) path while still routing results through the agent's real split tier. Capped at exactly one, so free compute is bounded; subsequent baseline evals debit normally. /status now reports base_commit and free_baseline_available, so agents can discover the mechanism.

Selection integrity is untouched: the baseline experiment is recorded as before and remains excluded from auto_best by the existing base_commit filter.

Tests

Seven new tests: first baseline eval unmetered but tier-routed; second one metered; non-baseline commits always metered; no-base_commit tasks never free; /status surfaces and flips the flag.

Follow-up (separate, compiler layer): update the #12 instruction paragraph to say the first baseline eval is free once this lands.

🤖 Generated with Claude Code

Greptile Summary

This PR addresses the root cause of two observed trial failures: the seeded baseline commit, which can never win selection, was being charged against the agent's eval budget. It adds a budget-free path for the agent's first baseline eval, a auto_best selection floor that reverts to the seed when no candidate beats it, retry logic for score_baseline, and surfaces baseline outcomes durably via the finalize response.

• Free first baseline eval (server.py): when the transferred commit resolves to base_commit and the slot is unused, the eval is routed through the engine's admin (unmetered) path; the flag is consumed only after a successful eval, so transient failures leave the slot available for a free retry.
• auto_best baseline floor (verifier.py): after rescoring, if the best candidate's admin score does not strictly exceed the baseline's admin score on the selection split, finalize reverts to base_commit.
• Structured finalize response + retry (verifier.py, cli.py): finalize now returns {\"rewards\": …, \"baseline\": …}; _maybe_score_baseline is retried and its outcome surfaced durably.

Confidence Score: 5/5

Safe to merge — the free-baseline slot is consumed only after a successful engine call, admin calls leave the slot intact, and the auto_best floor reverts cleanly to the seed when no candidate improves on it.

The two issues flagged in the previous review round were addressed in 0874f11 and are directly pinned by new tests. The baseline floor is consistent with existing error-handling conventions, the CLI backward-compat shim handles mixed-version sidecars, and test coverage is thorough.

No files require special attention.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/server.py	Adds free-first-baseline-eval logic with correct post-success flag assignment, admin-call guard, and status exposure.
vero/src/vero/harbor/verifier.py	Adds auto_best baseline floor and retry+structured-return for _maybe_score_baseline; consistent with existing error-handling conventions.
vero/src/vero/harbor/cli.py	Backward-compatible unwrapping of the new finalize envelope.
vero/tests/test_harbor_server.py	Seven new tests pin all free-baseline invariants.
vero/tests/test_harbor_verifier.py	Adds TestAutoBestBaselineFloor suite and updates existing callers to unwrap the new finalize envelope.

_{Reviews (3): Last reviewed commit: "Merge pull request #27 from scaleapi/har..." | Re-trigger Greptile}

[shehabyasser-scale]

Nit found by the review panel for #26 while verifying this PR's semantics: _free_baseline_used is a plain in-memory bool, while the budget ledger is persisted to <admin_volume>/ledger.json precisely so a sidecar restart cannot refund budget. After a restart the freebie re-arms: a second baseline eval is free and status reports free_baseline_available: true again. Agent-favorable (bounded free compute, never misleads the agent into wasting budget, instruction and status stay mutually consistent), so not a blocker; a follow-up could persist the flag next to ledger.json.

[greptile-apps]

Race window defeats the "capped at one" guarantee

The _free_baseline_used read at line 87 (inside free_baseline = (... and not self._free_baseline_used)) is separated from the write at line 98 by await self.engine.evaluate(...). If two concurrent evaluate coroutines arrive while none have completed yet, both will read False, both will set free_baseline = True, and both will be served as admin calls — violating the explicit "capped at exactly one" design goal.

The comment "no await runs between the check above and this write" is true about the if free_baseline: … assignment site alone, but that is not the relevant check. The check that controls the boolean is not self._free_baseline_used in the free_baseline tuple expression, and that read is separated from the write by a full await. In a single-threaded asyncio event loop two coroutines can both be suspended at await self.engine.evaluate(...) having already recorded free_baseline = True.

Fixing this requires setting the flag before the await (then restoring it on exception), or guarding it with a sentinel that is set synchronously before yielding — the mirror of the reasoning that justified moving the assignment after the await in the previous review.

Prompt To Fix With AI

This is a comment left during a code review.
Path: vero/src/vero/harbor/server.py
Line: 83-98

Comment:
**Race window defeats the "capped at one" guarantee**

The `_free_baseline_used` read at line 87 (inside `free_baseline = (... and not self._free_baseline_used)`) is separated from the write at line 98 by `await self.engine.evaluate(...)`. If two concurrent `evaluate` coroutines arrive while none have completed yet, both will read `False`, both will set `free_baseline = True`, and both will be served as admin calls — violating the explicit "capped at exactly one" design goal.

The comment "no await runs between the check above and this write" is true about the `if free_baseline: …` assignment site alone, but that is not the relevant check. The check that controls the boolean is `not self._free_baseline_used` in the `free_baseline` tuple expression, and that read is separated from the write by a full `await`. In a single-threaded asyncio event loop two coroutines can both be suspended at `await self.engine.evaluate(...)` having already recorded `free_baseline = True`.

Fixing this requires setting the flag **before** the await (then restoring it on exception), or guarding it with a sentinel that is set synchronously before yielding — the mirror of the reasoning that justified moving the assignment *after* the await in the previous review.

How can I resolve this? If you propose a fix, please make it concise.