Skip to content

feat: [no-ticket] harbor - transfer targets (per-target executor-model override at finalize)#38

Merged
varunursekar merged 4 commits into
harbor-10-crash-visibilityfrom
harbor-11-transfer-targets
Jul 17, 2026
Merged

feat: [no-ticket] harbor - transfer targets (per-target executor-model override at finalize)#38
varunursekar merged 4 commits into
harbor-10-crash-visibilityfrom
harbor-11-transfer-targets

Conversation

@shehabyasser-scale

@shehabyasser-scale shehabyasser-scale commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #37. Together they are the response to the wave-1 transfer-matrix finding: three of five champions independently hardcoded temperature=0 (a variance-reduction trick on their home model) and scored 0/72 on claude-opus-4-8, which rejects that parameter, while looking healthy on every eval the optimization loop ever ran. #37 makes such a crash legible after the fact; this PR makes it measurable DURING every run.

What it does

VerificationTarget gains an optional model: a finalize target with an executor override scores the selected commit under a model it was not optimized on, alongside its home-model reward in the same reward.json:

targets:
  - split: test
    reward_key: reward            # home executor
  - split: test
    reward_key: reward_opus       # transfer probe
    model: anthropic/claude-opus-4-8

A champion with a baked-in model coupling then ships reward_opus: 0.0 with the crash cause in target_errors (#37), instead of the coupling surfacing one substrate away, weeks later, in a manual probe.

Design notes

  • The baseline is scored under the same override, so score_baseline comparisons stay like-for-like per target.
  • Plumbing rides task_params["harbor_model_override"] (an existing per-eval passthrough): TargetSpec (build.yaml) → compiler → _TargetCfg (serve.json) → VerificationTargetengine.evaluate_admin(model=...)HarborRunner -m. Mode A ignores the key; the shared run_constraints are copied, never mutated.
  • Selection is untouched: the override applies to finalize targets only, so the optimizer's incentives during the run are unchanged; portability is measured, not optimized against (that would be its own experiment).

Tests

  • Engine: override rides task_params without mutating shared constraints; no-override passes constraints through unchanged.
  • Runner: -m honors the override over the configured model.
  • Verifier: the override reaches BOTH the champion and baseline evals.
  • Build: TargetSpec.model round-trips through the compiler into serve.json and validates.
  • Full affected suites green (153 passed after widening 11 strict-signature test fakes of evaluate_admin).

🤖 Generated with Claude Code

Greptile Summary

This PR adds an optional model override to VerificationTarget (and the full TargetSpec → compiler → serve.json → verifier pipeline) so finalize can score a champion under a model it was not optimized on — a direct response to finding that three champions independently baked in temperature=0 and scored 0/72 on an executor that rejects that parameter. It also adds bounded within-eval infra retry (infra_retry_rounds, off by default), dead-attempt classification with infra labeling, a key-budget exhaustion alarm, and an n_dead_infra metric.

  • Transfer probe plumbing: evaluate_admin(model=...) creates a shallow copy of run_constraints (never mutates shared state) and injects the override into task_params["harbor_model_override"]; the baseline is scored under the same override so comparisons remain like-for-like.
  • Infra retry (opt-in): samples whose every attempt died of a transient infrastructure exception can be retried in a fresh sibling jobs dir with a linear backoff; the qualifying predicate is restricted to transient causes only, and each recovered sample receives an infra_retry audit marker listing the discarded rounds.
  • Dead-attempt classification: a conservative allowlist of infra exception names labels deaths as [infra] or [infra:llm-key-budget]; bracket-injection in candidate-authored exception names is neutralized before classification; n_dead_infra surfaces the count without affecting the score.

Confidence Score: 5/5

Safe to merge. The transfer-probe plumbing is minimal and non-invasive; infra retry is opt-in and off by default; no shared state is mutated.

The model override creates a copy of run_constraints rather than mutating the shared object; the infra retry is gated behind infra_retry_rounds > 0 (default 0); the adversarial re-roll vector is correctly neutralized by requiring ALL attempts to carry a transient-infra label. All new paths are covered by tests. No logic errors or data-integrity issues were found.

No files require special attention. runner.py is the largest change but is thoroughly tested by test_harbor_runner.py.

Important Files Changed

Filename Overview
vero/src/vero/evaluation/engine.py Adds model parameter to evaluate_admin; correctly uses model_copy to create a shallow copy of run_constraints rather than mutating shared state.
vero/src/vero/harbor/build/compiler.py One-line addition to propagate t.model into the serialized target dict for serve.json.
vero/src/vero/harbor/build/config.py Adds `model: str
vero/src/vero/harbor/config.py Adds infra_retry_rounds and infra_retry_delay_s fields to HarborConfig with validation: rejects negative rounds and rejects zero/negative delay when retries are enabled.
vero/src/vero/harbor/runner.py Largest change: adds dead-attempt classification, key-budget alarm, bounded infra retry with sibling-dir isolation and audit stamping, model override in _build_command, and n_dead_infra metric. Well-documented and tested; no shared-state mutation; adversarial-candidate guard is correctly opt-in.
vero/src/vero/harbor/serve.py Adds `model: str
vero/src/vero/harbor/verifier.py Adds model to VerificationTarget, threads it through _admin_eval_score, and correctly applies the same override to both champion and baseline evals to keep comparisons like-for-like.
vero/tests/test_harbor_runner.py Adds TestInfraResilience with 9 tests covering classification, key-budget alarm, retry eligibility, multi-round backoff, audit-marker content, and adversarial no-retry invariants.
vero/tests/test_harbor_verifier.py Updates 11 _admin fakes to accept the new model keyword arg; adds test verifying the override reaches both champion and baseline calls.
vero/tests/test_engine.py Adds two tests confirming the model override injects into task_params without mutating run_constraints, and that a no-override call passes constraints through as the same object.
vero/tests/test_harbor_build.py Adds a round-trip test verifying TargetSpec.model survives compilation into serve.json and is correctly deserialized.

Reviews (3): Last reviewed commit: "Merge pull request #39 from scaleapi/har..." | Re-trigger Greptile

… finalize

Home-model evals cannot see model-specific couplings the optimizer bakes
in. Measured live in the wave-1 transfer matrix: three of five champions
independently hardcoded temperature=0 (a variance trick on their home
model, gpt-4.1-mini) and scored 0/72 on claude-opus-4-8, which rejects
it, while looking healthy on every eval the optimization loop ever ran.
The portability failure was invisible until a separate, manual,
after-the-fact probe.

VerificationTarget gains `model`: a target with an executor override
scores the selected commit under a model it was NOT optimized on, in the
same finalize battery as its home-model reward. The baseline is scored
under the same override so the comparison stays like-for-like. Plumbing:
build.yaml TargetSpec -> compiler -> serve.json _TargetCfg ->
VerificationTarget -> engine.evaluate_admin(model=...) -> task_params
["harbor_model_override"] -> HarborRunner -m flag. The override rides
task_params, so Mode A ignores it and the runner needs no new state; the
shared run_constraints are copied, never mutated.

Test fakes of evaluate_admin widened to accept the new kwarg (a strict
signature turned the new call into a retried TypeError, flooring
rewards, which is itself a nice demonstration of the floor's fail-safe).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@shehabyasser-scale
shehabyasser-scale force-pushed the harbor-11-transfer-targets branch from 256ffbf to de76acf Compare July 9, 2026 12:26
…utage retry, key-budget alarm

Three infra failure modes measured live in the E5 matrix runs, fixed at
the measurement layer:

- Dead attempts are classified infra vs candidate (conservative
  exception-type allowlist + the litellm key-budget message signature).
  Labels flow into dead_exception_types, error strings, and a new
  n_dead_infra metric. Classification never moves a score: every dead
  attempt still zero-fills, or faking infra would excuse failures.
  Exception type names are candidate-authored, so brackets are
  neutralized before labeling (a class named 'XError[infra]' cannot
  walk in pre-suffixed).

- An OPT-IN, bounded, backoff-spaced within-eval retry re-measures
  samples whose every attempt died of a transient infra cause (the
  65-second DNS blip that killed 44/72 attempts of one eval). Off by
  default: against an adversarial optimizer the qualifying predicate is
  a re-roll lever, since a stochastic candidate that raises allowlisted
  exceptions on failing attempts converts all-bad rounds into fresh
  draws. Retry rounds run in fresh sibling jobs dirs (nesting would
  pool dead attempts into later resumed means) and recovered samples
  carry an infra_retry audit marker naming the discarded attempts.

- An ERROR-level alarm names key-budget exhaustion (a spent key fails
  every later call identically; two matrix cells of budget-exceeded
  zeros were nearly booked as a portability finding), with hedged
  wording since the signature reads candidate-process exceptions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
shehabyasser-scale and others added 2 commits July 9, 2026 17:18
…d audit history

- HarborConfig rejects infra_retry_delay_s <= 0 when retries are enabled
  (a zero delay silently nullified the backoff) and negative
  infra_retry_rounds.
- The infra_retry audit marker now lists EVERY discarded round in order
  (discarded_rounds), not just the one immediately before recovery;
  recovered_round names when the sample finally measured.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat: [no-ticket] harbor - infra resilience (dead-attempt classification, opt-in outage retry, key-budget alarm)
@varunursekar
varunursekar merged commit 783db3d into harbor-10-crash-visibility Jul 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants