refactor(harbor): split Mode A / Mode B into discriminated-union configs

[shehabyasser-scale]

Supersedes #20 by construction. Mode A (vero-native per-sample scoring) and Mode B (nested harbor run) previously shared one flat BuildConfig/ServeConfig, so a field that only works in one mode silently no-opped in the other, and we kept papering over that with runtime warnings (#20's sample_timeout warning, and the Mode-A feedback-lever warning added in #30).

This splits both schemas into discriminated unions on mode:

• BuildConfigA (mode A: task, task_project, task_module, dataset, sample_timeout) and BuildConfigB (mode B: harbor, partition, inner_task, and the feat(harbor): feedback levers: failure transcripts, multi-fidelity screening, per-attempt detail #30 feedback levers), over a shared extra="forbid" base. Same for ServeConfigA/ServeConfigB.
• A wrong-mode field is now a load-time ValidationError instead of a silent no-op: sample_timeout in a Mode-B yaml fails to parse, feedback_transcripts in a Mode-A yaml fails to parse.
• from_file classmethods become load_build_config / load_serve_config (TypeAdapter loaders; a union is not a class). Missing mode still defaults to A.
• Deleted as dead code: _warn_mode_b_sample_timeout (all of fix(harbor): warn that sample_timeout is not enforced in Mode B #20) and _warn_mode_a_ignores_feedback_levers, plus their tests. The conditions they warned about are now structurally impossible.
• This also fixes a latent bug in fix(harbor): warn that sample_timeout is not enforced in Mode B #20's approach: the warning gated on model_fields_set, but the compiler bakes sample_timeout unconditionally into the serve JSON, so the "explicitly set" check would have fired on every compiled Mode-B task.

Acceptance tests cover both wrong-mode rejections and valid round-trips through compile_task. The three live Pawn lever configs parse and compile through the new loader (one of them ran a full trial through this code).

Stacked on #30 (harbor-3-feedback-levers).

🤖 Generated with Claude Code

Greptile Summary

This PR replaces the flat BuildConfig/ServeConfig with pydantic discriminated unions (BuildConfigA/BuildConfigB, ServeConfigA/ServeConfigB) so a wrong-mode field is a load-time ValidationError instead of a silent no-op. It also ships several independent behavioral improvements bundled in the same branch.

• Config split: mode=\"A\" and mode=\"B\" are now distinct pydantic types sharing an extra=\"forbid\" base; from_file classmethods are replaced by module-level load_build_config/load_serve_config TypeAdapter loaders; missing mode defaults to \"A\" for backward compat.
• Mean aggregation fix: Dead-before-scoring attempts are now zero-filled (counted as n_dead) instead of excluded, closing a real exploit where a no-retry candidate won selection by having rate-limited attempts dropped from its denominator.
• Other additions: k-anonymity floor on non-viewable subset evals, infra-retry logic (opt-in), _trial_rank monotone in reward, idempotent finalize, ledger fail-closed on corrupt read, commit-tree dedup in shortlisting, verifier-timeout field, and a trusted harbor_requirement overlay.

Confidence Score: 5/5

Safe to merge. The discriminated-union config split is structurally sound and well-tested; behavioral changes close documented live exploits.

The refactor is thoroughly tested with both positive round-trips and negative (wrong-mode-field) rejection cases. The mean-aggregation dead-attempt zero-fill and trial-rank monotonicity fix real, measured-live issues. The two comments are non-blocking suggestions.

vero/src/vero/harbor/server.py — the _eval_seq resume scan in _route_results lacks an OSError guard; vero/src/vero/harbor/runner.py — dead-attempt label for a reward-key-mismatch uses 'no_rewards_recorded' rather than a more specific label.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/build/config.py	Core of the refactor: flat BuildConfig split into BuildConfigA / BuildConfigB via pydantic discriminated union; load_build_config replaces from_file; backward-compat mode default, extra=forbid base. Clean and well-structured.
vero/src/vero/harbor/serve.py	ServeConfig split into ServeConfigA/B with matching TypeAdapter loader; build_components updated to use isinstance checks; ledger fail-closed fix; k_anonymity_floor wired to sidecar. Correct.
vero/src/vero/harbor/server.py	EvaluationSidecar gains k-anonymity floor enforcement, free-baseline flag separation, and per-eval versioned result dirs (_eval_seq). The _eval_seq resume scan on restart uses results_root.iterdir() without an OSError guard.
vero/src/vero/harbor/runner.py	Major behavioral changes: dead-before-scoring attempts zero-filled in mean, _extract_reward returns None for ambiguous keys, _trial_rank monotone in reward, infra-retry loop, dead-attempt labeling and key-budget alarm. All intentional and well-tested.
vero/src/vero/harbor/verifier.py	Finalize made idempotent (asyncio.Lock + cached result), _admin_eval_score extracts retry logic, commit-tree dedup in shortlisting, transfer-model probe on targets and baseline. Comprehensive and correct.
vero/src/vero/evaluation/engine.py	evaluate gains a separate free flag (distinct from admin); evaluate_admin gains model override via task_params copy. Fixes the latent bug where free-baseline could evaluate no_access splits via admin=True.
vero/tests/test_harbor_build.py	Tests updated to use BuildConfigA/B and load_build_config/load_serve_config; new TestModeMismatchRejection class verifies wrong-mode fields fail at load; target model round-trip test added.
vero/tests/test_harbor_runner.py	Extensive new tests for infra classification, infra retry, key-budget alarm, monotone trial rank, empty transcript fallthrough, and dead-attempts zero-fill. Covers adversarial cases explicitly.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["build.yaml / serve.json"] --> B{"mode field?"}
    B -->|"missing"| C["inject mode=A"]
    B -->|"present"| D{"mode value"}
    C --> D
    D -->|"A"| E["BuildConfigA / ServeConfigA"]
    D -->|"B"| F["BuildConfigB / ServeConfigB"]
    E --> G["extra=forbid on _Base\nMode-B keys are ValidationError"]
    F --> H["extra=forbid on _Base\nMode-A keys are ValidationError"]
    G --> I["compile_task / build_components\nisinstance dispatch"]
    H --> I
    I -->|"BuildConfigA"| J["Mode A: Vero inference + scoring"]
    I -->|"BuildConfigB"| K["Mode B: Nested harbor run"]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A["build.yaml / serve.json"] --> B{"mode field?"}
    B -->|"missing"| C["inject mode=A"]
    B -->|"present"| D{"mode value"}
    C --> D
    D -->|"A"| E["BuildConfigA / ServeConfigA"]
    D -->|"B"| F["BuildConfigB / ServeConfigB"]
    E --> G["extra=forbid on _Base\nMode-B keys are ValidationError"]
    F --> H["extra=forbid on _Base\nMode-A keys are ValidationError"]
    G --> I["compile_task / build_components\nisinstance dispatch"]
    H --> I
    I -->|"BuildConfigA"| J["Mode A: Vero inference + scoring"]
    I -->|"BuildConfigB"| K["Mode B: Nested harbor run"]

Loading

_{Reviews (2): Last reviewed commit: "Merge pull request #32 from scaleapi/har..." | Re-trigger Greptile}