feat(harbor): feedback levers: failure transcripts, multi-fidelity screening, per-attempt detail

[shehabyasser-scale]

Adds three opt-in feedback levers to Mode B, answering the question this stack kept circling: the optimizer only gets one score per paid eval, which is very thin signal. Each lever is a build.yaml switch, default off, byte-identical behavior when off.

• feedback_transcripts (+ feedback_max_bytes, default 3000): every FAILED sample of an eval carries the tail of its rollout transcript in the per-sample feedback field. Rides the per-sample result files the sidecar writes ONLY for viewable splits, so it can never surface for non_viewable / no_access tiers.
• instruct_multifidelity: the compiled instruction teaches subset-eval screening (triage rough ideas cheaply, confirm survivors on the full split). Renders only when a viewable split exists (see leak fix below).
• expose_attempt_detail: per-sample attempts list with {reward, exception} per attempt, same viewable-only exposure.

The final commit closes what adversarial review found: a multi-fidelity hidden-split leak (1-sample subset evals could recover per-sample labels on non_viewable splits; the instruction is now gated to viewable splits), subset evals polluting the auto_best shortlist (ranking now prefers full-split evals), feedback_max_bytes <= 0 returning the whole transcript (now means "no feedback"), symlink/path-escape containment on transcript reads, typo-safe config keys (extra="forbid"), and attempt-ordering.

Live experimental evidence (2026-07-06). Three arms on the same substrate (the VeRO paper's minimal Pawn agent on gpt-4.1-mini, baseline 0.20, ceiling ~0.43), same optimizer (opus), same 10-eval budget, only the levers differ:

Arm	Levers	Hidden-split final	Optimizer behavior
A	multifidelity + attempt detail	0.139	prompt rewording only, flat
B	transcripts only	0.194	rewrote agent CODE: file parsing, image support, fixed a binary-read crash it saw in the transcripts
C	all three	0.167	code fixes too; subset screening diluted the budget

Transcripts changed the optimizer from prompt-tweaker (which never moves the score; see the campaign doc's 6-model flatness result) into code-fixer, which is where all measured gains come from. n=1 per arm; a replication and a 25-eval run are in flight.

Stacked on the harbor stack (base: harbor-all-fixes, the integration union of the open PRs; will re-target as the stack merges).

🤖 Generated with Claude Code

Greptile Summary

This PR adds three opt-in feedback levers to Mode B Harbor evaluations and closes several adversarially-identified integrity gaps: the free/admin authority conflation (no_access split bypass on the free baseline), the multi-fidelity hidden-split leak, subset evals polluting the auto_best shortlist, and missing path-containment on transcript reads.

• Feedback levers (feedback_transcripts, expose_attempt_detail, instruct_multifidelity): each is a build.yaml toggle defaulting to off; existing behavior is byte-identical when all three are off, and transcript exposure is gated to viewable splits through the sidecar's existing tier routing.
• Auth and integrity fixes: engine.evaluate() now distinguishes free (budget-only waiver) from admin (full bypass), closing the no_access escape; _best_from_db prefers full-split evals for shortlisting when any exist; auto_best_baseline_floor prevents shipping regressions; BuildConfig/ServeConfig are split into typed A/B variants with extra=\"forbid\" so a mistyped lever fails loudly at load time rather than silently disabling the feature.

Confidence Score: 5/5

Safe to merge. All three levers default to off, so existing deployments are byte-identical. The integrity fixes are defensive and fail-closed.

The feedback levers are cleanly isolated behind instance-variable flags. The security fixes are narrow, well-tested, and the existing tier routing provides the primary exposure gate independently of the new code. Test coverage for the new levers and verifier ranking changes is thorough.

vero/src/vero/harbor/runner.py — the _read_transcript_tail TOCTOU gap and duplicated feedback_max_bytes default are minor polish items.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/runner.py	Added three feedback levers: _failure_feedback (transcript tail for failed samples), _attempt_detail (per-attempt reward/exception), and path-containment logic in _read_transcript_tail. Logic is well-guarded; two previously-flagged bugs are noted as fixed upstream.
vero/src/vero/harbor/verifier.py	Added auto_best_baseline_floor (reverts to seed when no candidate beats baseline) and subset-eval shortlist filter (full-split evals preferred for ranking). Well-tested.
vero/src/vero/evaluation/engine.py	Added free parameter to evaluate() to separate budget-waiver from admin access, and model parameter to evaluate_admin() for transfer-target overrides. Both changes are narrow and well-documented.
vero/src/vero/harbor/build/config.py	Split BuildConfig into typed BuildConfigA/BuildConfigB with extra=forbid, preventing mistyped levers from silently no-oping. Backward-compatible.
vero/src/vero/harbor/build/compiler.py	_serve_config updated for A/B split; multifidelity template context correctly gates on viewable split existence and EvalRequest field introspection.
vero/src/vero/harbor/serve.py	ServeConfig split into ServeConfigA/ServeConfigB. HarborRunner instantiation now passes all three lever flags. Backward-compatible.
vero/src/vero/harbor/server.py	Free-baseline handling now passes free=True instead of admin=True, closing the no_access bypass. Versioned eval dirs prevent result erasure on re-measurement.
vero/src/vero/workspace/git.py	Added tree_hash() for git rev-parse ^{tree}; used by Verifier._tree_of() to pool re-committed identical trees in auto_best shortlisting.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Agent
    participant Sidecar as EvaluationSidecar
    participant Engine as EvaluationEngine
    participant Runner as HarborRunner
    participant Verifier

    Agent->>Sidecar: POST /eval (commit, split)
    Sidecar->>Sidecar: k-anonymity check (non_viewable subsets)
    Sidecar->>Sidecar: _transfer_commit()
    alt first eval of base_commit (free baseline)
        Sidecar->>Engine: "evaluate(req, free=True)"
        note over Engine: tier gate applies, budget skipped
    else normal agent eval
        Sidecar->>Engine: "evaluate(req, free=False)"
        Engine->>Engine: tier gate (no_access reject)
        Engine->>Engine: budget.reserve()
    end
    Engine->>Runner: produce_sample_results()
    Runner->>Runner: harbor run (nested)
    Runner->>Runner: _collate()
    opt "feedback_transcripts=True and score==0.0"
        Runner->>Runner: _failure_feedback() _read_transcript_tail()
        note over Runner: symlink check + path containment before read
    end
    opt "expose_attempt_detail=True"
        Runner->>Runner: _attempt_detail()
    end
    Sidecar->>Sidecar: _route_results() by split tier
    Sidecar-->>Agent: EvalSummary (aggregate only)

    Note over Verifier: At trial end (finalize)
    Verifier->>Verifier: _best_from_db(): prefer full-split evals
    Verifier->>Engine: evaluate_admin() x rescore_top_k
    opt auto_best_baseline_floor
        Verifier->>Engine: evaluate_admin() baseline
    end
    Verifier->>Engine: evaluate_admin() x targets
    Verifier-->>Agent: reward.json

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Agent
    participant Sidecar as EvaluationSidecar
    participant Engine as EvaluationEngine
    participant Runner as HarborRunner
    participant Verifier

    Agent->>Sidecar: POST /eval (commit, split)
    Sidecar->>Sidecar: k-anonymity check (non_viewable subsets)
    Sidecar->>Sidecar: _transfer_commit()
    alt first eval of base_commit (free baseline)
        Sidecar->>Engine: "evaluate(req, free=True)"
        note over Engine: tier gate applies, budget skipped
    else normal agent eval
        Sidecar->>Engine: "evaluate(req, free=False)"
        Engine->>Engine: tier gate (no_access reject)
        Engine->>Engine: budget.reserve()
    end
    Engine->>Runner: produce_sample_results()
    Runner->>Runner: harbor run (nested)
    Runner->>Runner: _collate()
    opt "feedback_transcripts=True and score==0.0"
        Runner->>Runner: _failure_feedback() _read_transcript_tail()
        note over Runner: symlink check + path containment before read
    end
    opt "expose_attempt_detail=True"
        Runner->>Runner: _attempt_detail()
    end
    Sidecar->>Sidecar: _route_results() by split tier
    Sidecar-->>Agent: EvalSummary (aggregate only)

    Note over Verifier: At trial end (finalize)
    Verifier->>Verifier: _best_from_db(): prefer full-split evals
    Verifier->>Engine: evaluate_admin() x rescore_top_k
    opt auto_best_baseline_floor
        Verifier->>Engine: evaluate_admin() baseline
    end
    Verifier->>Engine: evaluate_admin() x targets
    Verifier-->>Agent: reward.json

Loading

_{Reviews (2): Last reviewed commit: "Merge pull request #31 from scaleapi/har..." | Re-trigger Greptile}