feat: [no-ticket] harbor - transfer targets (per-target executor-model override at finalize)#38
Merged
varunursekar merged 4 commits intoJul 17, 2026
Conversation
… finalize Home-model evals cannot see model-specific couplings the optimizer bakes in. Measured live in the wave-1 transfer matrix: three of five champions independently hardcoded temperature=0 (a variance trick on their home model, gpt-4.1-mini) and scored 0/72 on claude-opus-4-8, which rejects it, while looking healthy on every eval the optimization loop ever ran. The portability failure was invisible until a separate, manual, after-the-fact probe. VerificationTarget gains `model`: a target with an executor override scores the selected commit under a model it was NOT optimized on, in the same finalize battery as its home-model reward. The baseline is scored under the same override so the comparison stays like-for-like. Plumbing: build.yaml TargetSpec -> compiler -> serve.json _TargetCfg -> VerificationTarget -> engine.evaluate_admin(model=...) -> task_params ["harbor_model_override"] -> HarborRunner -m flag. The override rides task_params, so Mode A ignores it and the runner needs no new state; the shared run_constraints are copied, never mutated. Test fakes of evaluate_admin widened to accept the new kwarg (a strict signature turned the new call into a retried TypeError, flooring rewards, which is itself a nice demonstration of the floor's fail-safe). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
shehabyasser-scale
force-pushed
the
harbor-11-transfer-targets
branch
from
July 9, 2026 12:26
256ffbf to
de76acf
Compare
…utage retry, key-budget alarm Three infra failure modes measured live in the E5 matrix runs, fixed at the measurement layer: - Dead attempts are classified infra vs candidate (conservative exception-type allowlist + the litellm key-budget message signature). Labels flow into dead_exception_types, error strings, and a new n_dead_infra metric. Classification never moves a score: every dead attempt still zero-fills, or faking infra would excuse failures. Exception type names are candidate-authored, so brackets are neutralized before labeling (a class named 'XError[infra]' cannot walk in pre-suffixed). - An OPT-IN, bounded, backoff-spaced within-eval retry re-measures samples whose every attempt died of a transient infra cause (the 65-second DNS blip that killed 44/72 attempts of one eval). Off by default: against an adversarial optimizer the qualifying predicate is a re-roll lever, since a stochastic candidate that raises allowlisted exceptions on failing attempts converts all-bad rounds into fresh draws. Retry rounds run in fresh sibling jobs dirs (nesting would pool dead attempts into later resumed means) and recovered samples carry an infra_retry audit marker naming the discarded attempts. - An ERROR-level alarm names key-budget exhaustion (a spent key fails every later call identically; two matrix cells of budget-exceeded zeros were nearly booked as a portability finding), with hedged wording since the signature reads candidate-process exceptions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d audit history - HarborConfig rejects infra_retry_delay_s <= 0 when retries are enabled (a zero delay silently nullified the backoff) and negative infra_retry_rounds. - The infra_retry audit marker now lists EVERY discarded round in order (discarded_rounds), not just the one immediately before recovery; recovered_round names when the sample finally measured. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat: [no-ticket] harbor - infra resilience (dead-attempt classification, opt-in outage retry, key-budget alarm)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #37. Together they are the response to the wave-1 transfer-matrix finding: three of five champions independently hardcoded
temperature=0(a variance-reduction trick on their home model) and scored 0/72 on claude-opus-4-8, which rejects that parameter, while looking healthy on every eval the optimization loop ever ran. #37 makes such a crash legible after the fact; this PR makes it measurable DURING every run.What it does
VerificationTargetgains an optionalmodel: a finalize target with an executor override scores the selected commit under a model it was not optimized on, alongside its home-model reward in the same reward.json:A champion with a baked-in model coupling then ships
reward_opus: 0.0with the crash cause intarget_errors(#37), instead of the coupling surfacing one substrate away, weeks later, in a manual probe.Design notes
score_baselinecomparisons stay like-for-like per target.task_params["harbor_model_override"](an existing per-eval passthrough):TargetSpec(build.yaml) → compiler →_TargetCfg(serve.json) →VerificationTarget→engine.evaluate_admin(model=...)→HarborRunner-m. Mode A ignores the key; the sharedrun_constraintsare copied, never mutated.Tests
task_paramswithout mutating shared constraints; no-override passes constraints through unchanged.-mhonors the override over the configured model.TargetSpec.modelround-trips through the compiler into serve.json and validates.evaluate_admin).🤖 Generated with Claude Code
Greptile Summary
This PR adds an optional
modeloverride toVerificationTarget(and the fullTargetSpec → compiler → serve.json → verifierpipeline) so finalize can score a champion under a model it was not optimized on — a direct response to finding that three champions independently baked intemperature=0and scored 0/72 on an executor that rejects that parameter. It also adds bounded within-eval infra retry (infra_retry_rounds, off by default), dead-attempt classification with infra labeling, a key-budget exhaustion alarm, and ann_dead_inframetric.evaluate_admin(model=...)creates a shallow copy ofrun_constraints(never mutates shared state) and injects the override intotask_params["harbor_model_override"]; the baseline is scored under the same override so comparisons remain like-for-like.infra_retryaudit marker listing the discarded rounds.[infra]or[infra:llm-key-budget]; bracket-injection in candidate-authored exception names is neutralized before classification;n_dead_infrasurfaces the count without affecting the score.Confidence Score: 5/5
Safe to merge. The transfer-probe plumbing is minimal and non-invasive; infra retry is opt-in and off by default; no shared state is mutated.
The model override creates a copy of run_constraints rather than mutating the shared object; the infra retry is gated behind infra_retry_rounds > 0 (default 0); the adversarial re-roll vector is correctly neutralized by requiring ALL attempts to carry a transient-infra label. All new paths are covered by tests. No logic errors or data-integrity issues were found.
No files require special attention. runner.py is the largest change but is thoroughly tested by test_harbor_runner.py.
Important Files Changed
modelparameter toevaluate_admin; correctly usesmodel_copyto create a shallow copy ofrun_constraintsrather than mutating shared state.t.modelinto the serialized target dict for serve.json.infra_retry_roundsandinfra_retry_delay_sfields toHarborConfigwith validation: rejects negative rounds and rejects zero/negative delay when retries are enabled._build_command, andn_dead_inframetric. Well-documented and tested; no shared-state mutation; adversarial-candidate guard is correctly opt-in.modeltoVerificationTarget, threads it through_admin_eval_score, and correctly applies the same override to both champion and baseline evals to keep comparisons like-for-like.TestInfraResiliencewith 9 tests covering classification, key-budget alarm, retry eligibility, multi-round backoff, audit-marker content, and adversarial no-retry invariants._adminfakes to accept the newmodelkeyword arg; adds test verifying the override reaches both champion and baseline calls.task_paramswithout mutatingrun_constraints, and that a no-override call passes constraints through as the same object.TargetSpec.modelsurvives compilation into serve.json and is correctly deserialized.Reviews (3): Last reviewed commit: "Merge pull request #39 from scaleapi/har..." | Re-trigger Greptile