Redesign VeRO as a generic program optimization harness

.gitignore

@@ -1,5 +1,6 @@
 # Python-generated files
 __pycache__/
+.pytest_cache/
 *.py[oc]
 build/
 dist/

README.md

@@ -1,122 +1,79 @@
-# VeRO: Versioning Rewards and Observations
+# VeRO: a harness for agents to optimize programs
 
 [![Paper](https://img.shields.io/badge/arXiv-2602.22480-b31b1b.svg)](https://arxiv.org/abs/2602.22480)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
 
-VeRO is an evaluation harness for using coding agents to optimize LLM-based agents and workflows. It treats agent code as a versioned artifact — making changes, evaluating results, and hill-climbing toward better performance using git version control.
+VeRO gives a coding agent a program to edit, an evaluation boundary, and durable
+memory of every candidate it tried. The target can be an agent, a prompt, a
+compiler pass, a CUDA kernel, a matrix multiplication function, or any other
+Git-versioned program.
 
-> **Paper**: [VeRO: An Evaluation Harness for Agents to Optimize Agents](https://arxiv.org/abs/2602.22480)
-
-## Repository Structure
-
-```
-vero/
-├── vero/               # Core library (scale-vero)
-├── vero-agents/        # Agent implementations (benchmarking targets)
-├── vero-benchmarking/  # Benchmarking scripts and analysis
-└── LICENSE
-```
-
-### [vero/](vero/)
-
-The core optimization framework. Provides:
-
-- **Policy** — orchestrates the optimization loop (agent + evaluator + git)
-- **Agents** — VeroAgent (OpenAI Agents SDK) and ClaudeCodeAgent (Claude Agent SDK)
-- **Evaluator** — runs task evaluations in isolated subprocess environments
-- **Tools** — MCP-based tools for agents (bash, file I/O, experiment runner, dataset viewer, etc.)
-- **Traces** — session analysis and LLM-based trace interpretation
-
-```bash
-cd vero && uv sync --extra optimize
+```text
+strategy -> candidate producers -> evaluation backends -> selection
+                  |                       |
+          isolated workspaces       versioned reports
 ```
 
-See [vero/README.md](vero/README.md) for full documentation.
+The target and evaluator do not need to be Python. External evaluators and
+candidate producers connect through command protocols; Python benchmarks can
+use the optional, optimizer-independent `scale-vero-tasks` package.
 
-### [vero-agents/](vero-agents/)
+Targets may live locally or in an isolated sandbox. VeRO keeps optimization
+state and experiment tracking on the host while running Git worktrees, producer
+commands, builds, and evaluation commands in the target sandbox. The core guide
+includes a no-bind-mount `DockerSandbox` example.
 
-Agent implementations used as optimization targets:
+## Repository layout
 
-| Agent | Description |
-|-------|-------------|
-| **generic-agent** | General-purpose agent for MATH, GPQA, GAIA, GSM8K, etc. |
-| **web_search_agent** | Web search agent for SimpleQA, Facts Search |
-| **KIRA** | Terminal task agent for Terminal Bench 2.0 |
-| **tau-bench** | Customer service tool-use agent |
-| **pharma_summarizer** | Document summarization agent |
+| Directory | Purpose |
+| --- | --- |
+| [`vero/`](vero/) | The `scale-vero` optimization kernel, runtime, CLI, and coding-agent adapters |
+| [`vero-tasks/`](vero-tasks/) | Narrow Python task types and schema-v1 evaluation runner |
+| [`program-opt-bench/`](program-opt-bench/) | Harbor-native target programs and end-to-end optimization benchmarks |
 
-See [vero-agents/README.md](vero-agents/README.md) for details.
+Start with the [generic C matrix-multiplication quickstart](vero/examples/c-matmul/),
+try the [26-circle packing benchmark](vero/examples/circle-packing/), or read the
+[core guide](vero/README.md). The C example demonstrates the language-neutral
+command protocol without model credentials. Circle packing is a substantive
+coding-agent benchmark with exact geometry checks and inspectable search
+artifacts.
 
-### [vero-benchmarking/](vero-benchmarking/)
-
-Scripts and infrastructure for running optimization experiments:
+For a full agent-optimization example, the [GAIA benchmark](program-opt-bench/gaia/)
+pairs a tool-using GPT-5.4 mini target with Harbor's canonical GAIA verifier and
+an immutable 20% / 40% / 40% development, validation, and test split.
 
 ```bash
-cd vero-benchmarking && uv sync --all-extras
-
-# Run an optimization experiment
-uv run python scripts/run_benchmark.py --scaffold claude-code-vmf --model sonnet --task math
-
-# Build datasets
-./scripts/build_datasets.sh
+cd vero
+uv sync --all-extras
+uv run vero --help
 ```
 
-See [vero-benchmarking/README.md](vero-benchmarking/README.md) for full documentation.
-
-## Quick Start
-
-### Prerequisites
+## Paper reproduction
 
-- Python 3.11+
-- [uv](https://docs.astral.sh/uv/getting-started/installation/)
-- Git
-- Access to an LLM provider (via LiteLLM, OpenAI, Anthropic, etc.)
+VeRO was introduced in [*VeRO: A Harness for Agents to Optimize
+Agents*](https://arxiv.org/abs/2602.22480), accepted at ICML 2026. The current
+library generalizes that version/evaluate/select loop from agents to programs.
 
-### Install
+The exact paper implementation is frozen separately from the v0.5 redesign:
 
 ```bash
-git clone <repo-url> && cd vero
-
-# Install core library
-cd vero && uv sync --extra optimize
-
-# Install benchmarking tools
-cd ../vero-benchmarking && uv sync --all-extras
+git checkout paper-v1
 ```
 
-### Run Your First Optimization
-
-```python
-from agents import Agent as OAIAgent
-from vero.policy import Policy
-from vero.agents.vero import VeroAgent
-
-policy = Policy(
-    project_path="/path/to/my-agent",
-    dataset="/path/to/my-dataset",
-    agent=VeroAgent(
-        oai_agent=OAIAgent(name="VeroAgent", model="anthropic/claude-sonnet-4-5-20250929"),
-    ),
-    task="main",
-    train_budget=10,
-    max_turns=200,
-)
-
-best = await policy.run()
-print(f"Best commit: {best.commit}, score: {best.score}")
-```
+Use the `paper/v1` branch or `paper-v1` tag for reproduction. Development of
+the generic program optimizer continues on the `v0.5` branch. The frozen ref
+also preserves the paper-era `vero-agents` and `vero-benchmarking` directories;
+their Harbor-native replacement on `v0.5` is `program-opt-bench`.
 
 ## Citation
 
 ```bibtex
 @article{ursekar2026vero,
-  title={VeRO: An Evaluation Harness for Agents to Optimize Agents},
+  title={VeRO: A Harness for Agents to Optimize Agents},
   author={Ursekar, Varun and Shanker, Apaar and Chatrath, Veronica and Xue, Yuan (Emily) and Denton, Sam},
   journal={arXiv preprint arXiv:2602.22480},
   year={2026}
 }
 ```
 
-## License
-
-[MIT](LICENSE)
+VeRO is licensed under the [MIT License](LICENSE).

program-opt-bench/README.md

@@ -0,0 +1,29 @@
+# Program optimization benchmarks
+
+`program-opt-bench` contains end-to-end benchmarks for improving programs with
+VeRO. Each leaf directory pairs one editable target program with one immutable
+Harbor dataset and compiles them into an outer Harbor optimization task.
+
+The benchmark definitions intentionally keep three boundaries visible:
+
+- `target/` is the program the optimization agent may edit.
+- `partitions/` pins the cases and the development/validation/test split.
+- `build.yaml` is trusted configuration: model, evaluator, access policy,
+  budgets, and final scoring.
+
+In each benchmark, the complete development tasks and attachments are mounted
+read-only for the optimization agent. Development evaluations expose per-case
+results and complete Harbor trial records, including exact failures and
+target-agent logs. Validation remains aggregate-only, and
+test is reachable only by the trusted final verifier.
+
+The paper-era benchmark stack remains available on the `paper/v1` branch and
+the `paper-v1` tag. New benchmarks should use this Harbor-native layout.
+
+## Benchmarks
+
+| Benchmark | Editable target | Dataset | Split |
+| --- | --- | --- | --- |
+| [GAIA baseline](gaia/baseline/) | Tool-using Responses API agent | Harbor `gaia/gaia` | 20% / 40% / 40% |
+| [SWE-Atlas-QnA baseline](swe-atlas-qna/baseline/) | Codebase investigation agent | Harbor `scale-ai/swe-atlas-qna` | 20% / 40% / 40% |
+| [tau3 baseline](tau3/baseline/) | MCP customer-service agent | Harbor `sierra-research/tau3-bench` | 20% / 40% / 40% |

program-opt-bench/gaia/README.md

@@ -0,0 +1,39 @@
+# GAIA
+
+The GAIA benchmark uses the canonical Harbor task packages and verifier. It
+does not reproduce the paper's pure-language subset or its original split.
+
+The committed split is deterministic and stratified by GAIA level and whether
+the task has an attached file:
+
+- development: 33 cases (20%)
+- validation: 66 cases (40%)
+- test: 66 cases (40%)
+
+All 165 cases come from the immutable dataset reference recorded in
+[`partitions/manifest.json`](partitions/manifest.json). The development set is
+available to the optimization agent with full result disclosure. Validation is
+aggregate-only and is used to select candidates. Test is held out until Harbor
+grades the completed outer task. The complete development task packages and
+attachments are mounted read-only under `.vero/cases/`; successful and failed
+development evaluations place their complete Harbor trial records—including
+exact failures and target-agent logs—under
+`.vero/evaluations/`. Neither validation nor test resources are mounted.
+
+The pinned GAIA tasks declare a 600-second Harbor agent timeout. The build
+config sets `case_timeout_seconds: 180` and
+`task_agent_timeout_seconds: 600`, so VeRO invokes Harbor with an agent-timeout
+multiplier of `0.3`. The 180-second limit reported to the optimizer is therefore
+the timeout enforced by the inner Harbor trial.
+
+To verify or regenerate the split after downloading the pinned Harbor dataset:
+
+```bash
+uv run --python 3.12 scripts/partition_gaia.py \
+  --tasks-dir /path/to/downloaded/gaia \
+  --check
+```
+
+Refreshing from a different registry revision is an explicit operation: update
+the dataset constant in the script and `build.yaml`, then run with
+`--fetch-registry`. Review the manifest diff before committing it.

program-opt-bench/gaia/baseline/.gitignore

@@ -0,0 +1,4 @@
+compiled/
+target/.venv/
+target/.pytest_cache/
+

program-opt-bench/gaia/baseline/README.md

@@ -0,0 +1,63 @@
+# GAIA tool-using baseline
+
+This leaf benchmark optimizes a small Harbor-native GAIA agent. The editable
+program uses the OpenAI Responses API with three capabilities:
+
+- web search;
+- shell commands inside the GAIA task environment;
+- image inspection and final-answer submission.
+
+The trusted build fixes the evaluated model to
+`gpt-5.4-mini-2026-03-17`. A candidate may change the agent's prompts, control
+flow, tool definitions, file handling, or dependencies, but it cannot change
+the dataset, verifier, split, model, access policy, or final test target.
+
+## Compile the outer Harbor task
+
+From the repository root:
+
+```bash
+cd vero
+VERO_SKIP_SECRET_CHECK=1 uv run vero harbor build \
+  --config ../program-opt-bench/gaia/baseline/build.yaml \
+  --output ../program-opt-bench/gaia/baseline/compiled
+```
+
+Omit `VERO_SKIP_SECRET_CHECK=1` for a real build so VeRO verifies that the
+OpenAI and Modal credentials declared in `build.yaml` are present. Set
+`OPENAI_BASE_URL` to your OpenAI-compatible endpoint; use
+`https://api.openai.com/v1` when calling OpenAI directly. The `compiled/`
+directory is generated and intentionally ignored.
+
+The compiled task does not expose the upstream OpenAI key to either editable
+program. A separate inference-gateway container holds it. The outer coding
+agent uses a producer-scoped token for `gpt-5.4` or `gpt-5.5`; GAIA candidates use an
+evaluation-scoped token restricted to `gpt-5.4-mini-2026-03-17`. Their request
+and token budgets are recorded separately and are visible through
+`vero harbor status`.
+
+Run the outer optimization through VeRO so it can give Harbor's coding-agent
+adapter the scoped credential while forwarding the upstream credential only to
+the gateway:
+
+```bash
+cd vero
+uv run vero harbor run \
+  --config ../program-opt-bench/gaia/baseline/build.yaml \
+  --environment modal \
+  --agent codex \
+  --model openai/gpt-5.5
+```
+
+The coding agent edits only `target/`; inner evaluations run the candidate
+against the pinned GAIA tasks on Modal. Complete development tasks and
+attachments are mounted read-only under `.vero/cases/`. After each development
+evaluation, complete Harbor trial records for every case—including exact
+failures and target-agent logs—are available under
+`.vero/evaluations/`. Validation remains aggregate-only, and test remains
+verifier-only.
+
+Before Modal teardown, the shared verifier exports the complete VeRO session
+and a self-contained `experiment.html` into Harbor's verifier artifacts. Keep
+the session archive alongside the HTML: it is the canonical candidate and
+evaluation history and can be rendered again with `vero report`.

program-opt-bench/gaia/baseline/build.yaml

@@ -0,0 +1,79 @@
+name: vero/optimize-gaia-baseline
+description: >-
+  Improve a tool-using GAIA agent while preserving the Harbor agent interface.
+  The target must answer canonical GAIA tasks by writing /app/answer.txt.
+agent_repo: target
+task_source: gaia/gaia@sha256:bbc356f476e0b70ba77da11a9be7d6345918d1e4a2daade0d6dfb82ee6f7b761
+task_manifest: ../partitions/manifest.json
+agent_import_path: gaia_agent.agent:GaiaAgent
+harbor_requirement: harbor[modal]==0.18.0
+
+partition_files:
+  development: ../partitions/development.json
+  validation: ../partitions/validation.json
+  test: ../partitions/test.json
+
+agent_access:
+  - partition: development
+    disclosure: full
+    expose_case_resources: true
+    total_runs: 12
+    total_cases: 132
+  - partition: validation
+    disclosure: aggregate
+    expose_case_resources: false
+    min_aggregate_cases: 5
+    total_runs: 8
+    total_cases: 264
+
+selection_partition: validation
+targets:
+  - partition: test
+    reward_key: reward
+    failure_value: 0.0
+    max_attempts: 1
+
+evaluation_set_name: gaia
+objective:
+  selector:
+    metric: score
+  direction: maximize
+reward_mode: auto_best
+baseline_floor: true
+score_baseline: true
+rescore_top_k: 3
+rescore_attempts: 1
+
+model: openai/gpt-5.4-mini-2026-03-17
+environment_name: ${inner_env:-modal}
+harbor_python_version: "3.12"
+n_attempts: 1
+max_retries: 1
+infrastructure_max_attempts: 3
+infrastructure_retry_delay_seconds: 5
+aggregate_attempts: best
+feedback_transcripts: true
+feedback_max_bytes: 12000
+expose_attempt_detail: false
+timeout_seconds: 3600
+case_timeout_seconds: 180
+task_agent_timeout_seconds: 600
+max_concurrency: 8
+error_rate_threshold: 0.1
+verifier_timeout_seconds: 7200
+secrets:
+  - MODAL_TOKEN_ID
+  - MODAL_TOKEN_SECRET
+inference_gateway:
+  upstream_api_key_env: OPENAI_API_KEY
+  upstream_base_url_env: OPENAI_BASE_URL
+  producer:
+    allowed_models: ["${optimizer_model:-gpt-5.4}"]
+    max_concurrency: 8
+  evaluation:
+    allowed_models: [gpt-5.4-mini-2026-03-17]
+    max_requests: 15000
+    max_tokens: 100000000
+    max_concurrency: 64
+instruct_multifidelity: true
+instruct_exhaust_budget: true