diff --git a/docs/amps/empiric-from-assets.md b/docs/amps/empiric-from-assets.md index 2c935e186..20b12b2a2 100644 --- a/docs/amps/empiric-from-assets.md +++ b/docs/amps/empiric-from-assets.md @@ -29,7 +29,7 @@ This is a reduction in supplied domain implementation, not reconstruction from r ## Pilot -The configuration is [continual_from_assets_pilot_r1.yaml](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/continual_from_assets_pilot_r1.yaml). +The configuration is [continual_from_assets_pilot_r1.yaml](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/continual_from_assets_pilot_r1.yaml). It runs Opus 5 on seeds 0 and 1 of Fan + ramp, four-span Bridge, Domino, Balloons and Boil. There are ten intended runs total, each with its training and test levels. Fan uses the previously reviewed 3 mm ramp with the 10 cm landing extension and the repaired shared skills, not Fan maze. @@ -49,7 +49,7 @@ Bridge seeds 0-1 are array `23407427`; Fan maze seeds 0-1 are array `23407428`. The user immediately corrected Fan maze to Fan + ramp; both tasks of array `23407428` were cancelled with their logs preserved. The abandoned maze runs are not part of the intended pilot and must not be resumed or counted as task failures. Bridge array `23407427` remains unchanged. -The [expansion-only configuration](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/continual_from_assets_expansion_r1.yaml) submits only the eight new tasks, with Bridge and Fan maze disabled to prevent duplicates. +The [expansion-only configuration](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/continual_from_assets_expansion_r1.yaml) submits only the eight new tasks, with Bridge and Fan maze disabled to prevent duplicates. Its runtime menus are pinned to the frozen implementation. The resolved Fan + ramp flags match the repaired EMPIRIC ramp cohort, apart from making the default fitting flag explicit. The eight replacement/additional tasks were submitted on account d as the following two-seed arrays: diff --git a/docs/amps/fan-development.md b/docs/amps/fan-development.md index 583b7d28a..0b7b0a547 100644 --- a/docs/amps/fan-development.md +++ b/docs/amps/fan-development.md @@ -303,20 +303,20 @@ The backup account `dat` also passed but is not in the active pool. Account `c` has an active limit marker until September 23 at 00:00 UTC and is excluded. Usage percentages were unavailable from the service endpoint; successful probes establish current access, not a guarantee of sufficient remaining quota for full runs. Each task requests 8 CPUs and 16 GB, with requeue enabled for preemption, time limits, and recognized account-limit exits. -The launch configuration is [continual_fan_ramp_skill_repair_r1.yaml](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/continual_fan_ramp_skill_repair_r1.yaml). +The launch configuration is [continual_fan_ramp_skill_repair_r1.yaml](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/continual_fan_ramp_skill_repair_r1.yaml). The shared repair and its measured extra interaction cost are documented in [the switch investigation](fan-switch-seed4-investigation.md). Seed 0's original job stopped after account `a` reported that its organization had disabled subscription access for Claude Code. This was an infrastructure interruption after training succeeded and the test reached 207 steps, not a task failure. With the user's approval, job `23395354` resumes only seed 0 on `dat`, from the same frozen runtime, experiment key, run directory, sandbox, and recorded state. -The resume configuration is [continual_fan_ramp_skill_repair_seed0_resume.yaml](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/continual_fan_ramp_skill_repair_seed0_resume.yaml). +The resume configuration is [continual_fan_ramp_skill_repair_seed0_resume.yaml](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/continual_fan_ramp_skill_repair_seed0_resume.yaml). ### Six matched comparison arms The user approved five seeds each for all six other paper agents on this repaired Fan ramp setup. All 30 tasks were verified running on compute nodes after submission. They use the same frozen runtime `ff11bc76f4652c6964e73beda7e41a6655d670d9`, reviewed geometry, observation noise, step budget, and preflight-off setting as the repaired EMPIRIC cohort. -The [six-arm configuration](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/continual_fan_ramp_skill_repair_baselines_r1.yaml) resolves to exactly six five-seed arrays, with only the intended approach and ablation flag differences. +The [six-arm configuration](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/continual_fan_ramp_skill_repair_baselines_r1.yaml) resolves to exactly six five-seed arrays, with only the intended approach and ablation flag differences. - Oracle dynamics: array `23398858`, seeds 0-4, accounts b/d. - Direct agent: array `23398859`, seeds 0-4, account dat. diff --git a/docs/amps/oracle-fan-ramp-cohort-investigation.md b/docs/amps/oracle-fan-ramp-cohort-investigation.md index 8c91f2d58..bd75ccb0a 100644 --- a/docs/amps/oracle-fan-ramp-cohort-investigation.md +++ b/docs/amps/oracle-fan-ramp-cohort-investigation.md @@ -90,7 +90,7 @@ The driver is [audit_oracle_ramp_20260922.py](/home/ycliang/predicators/logs/aud Seeds 9 and 10 were submitted as array `23463117` and started on compute nodes using accounts b and d. The launch flags and arguments were checked against the previous extra-seed config; only the seed range changes. -The [launch configuration](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/continual_fan_ramp_oracle_extra_r3.yaml) retains the frozen runtime and original cohort identifier. +The [launch configuration](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/continual_fan_ramp_oracle_extra_r3.yaml) retains the frozen runtime and original cohort identifier. The Markdown benchmark tracks all eleven seeds, retaining all failures. Replacement plot monitor `23463368` refreshes the report and figures every 60 seconds when results change. The expanded-cohort report tests passed: 8 tests. diff --git a/docs/comparisons/empiric-validation-r2.md b/docs/comparisons/empiric-validation-r2.md index c2384b783..8187a0386 100644 --- a/docs/comparisons/empiric-validation-r2.md +++ b/docs/comparisons/empiric-validation-r2.md @@ -1,7 +1,7 @@ # EMPIRIC r2: prospective robustness checks The requested cohort is two additional seeds per domain, seeds 3 and 4, across Boil, Domino, Balloons, Bridge, and Fan. -The launcher is [continual_empiric_benchmark_r2.yaml](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/continual_empiric_benchmark_r2.yaml). +The launcher is [continual_empiric_benchmark_r2.yaml](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/continual_empiric_benchmark_r2.yaml). Its experiment key is `-mb_opus_benchmark_r2`, under `agent_continual`. The main figures retain historical EMPIRIC and show this cohort separately as **EMPIRIC r2**. All finished outcomes count, including failures; an unfinished run is not a failed run. diff --git a/docs/comparisons/five-seed-launch-review.md b/docs/comparisons/five-seed-launch-review.md index e7f43ae09..98c3ff218 100644 --- a/docs/comparisons/five-seed-launch-review.md +++ b/docs/comparisons/five-seed-launch-review.md @@ -35,7 +35,7 @@ These checks do not establish future solve rates or resolve the previously docum ## Launch and reporting -The launcher is [continual_benchmark_five_seeds.yaml](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/continual_benchmark_five_seeds.yaml). +The launcher is [continual_benchmark_five_seeds.yaml](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/continual_benchmark_five_seeds.yaml). All 60 destination seed directories were checked absent before submission. Jobs use `mit_preemptable`, checkpoint resume, automatic requeue, and account labels `a,b,c,d`, all confirmed usable by the user during this launch review. The user's account e corresponds to the launcher's `dat` label and is reserved as backup, not included in the normal rotation. diff --git a/docs/comparisons/ten-agent-opus-benchmark.md b/docs/comparisons/ten-agent-opus-benchmark.md index 00d1b45d2..b3c64346e 100644 --- a/docs/comparisons/ten-agent-opus-benchmark.md +++ b/docs/comparisons/ten-agent-opus-benchmark.md @@ -46,20 +46,20 @@ python ../../../scripts/plotting/plot_benchmark_arms.py benchmark-arms-opus ## Settings and cohort selection -The five settings are the menu defaults in [envs/continual.yaml](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/envs/continual.yaml): Boil two-jug test, Domino high-friction turn, Fan ramp test, Bridge four-span test row and Balloons composition test levels. +The five settings are the menu defaults in [envs/continual.yaml](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/envs/continual.yaml): Boil two-jug test, Domino high-friction turn, Fan ramp test, Bridge four-span test row and Balloons composition test levels. EMPIRIC and the direct agent ran before the benchmark rounds under their own round keys; the four-span EMPIRIC runs are the preflight-off ones (r2 seed 0, r3 seeds 1-2). The other arms ran as `_opus_benchmark_` from frozen worktrees: -- Oracle dynamics, zero-shot and no harness fitting: [continual_five_ablations_benchmark_r1.yaml](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/continual_five_ablations_benchmark_r1.yaml) at `2982f5876` (/home/ycliang/predicators-five-arms-frozen-20260918). -- Standalone simulator and no explicit uncertainty: round r2 from [continual_standalone_no_uncertainty_r2.yaml](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/continual_standalone_no_uncertainty_r2.yaml) at `0ddc8f7f4` (/home/ycliang/predicators-standalone-nounc-frozen-20260918), after the September 18 revision of both surfaces. -- Scene only: [continual_scene_only_benchmark_r1.yaml](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/continual_scene_only_benchmark_r1.yaml). -- Agentic real-to-sim: [continual_real_to_sim_benchmark_r1.yaml](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/continual_real_to_sim_benchmark_r1.yaml) at `5ea35c91c` (/home/ycliang/predicators-real-to-sim-frozen-20260918). +- Oracle dynamics, zero-shot and no harness fitting: [continual_five_ablations_benchmark_r1.yaml](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/continual_five_ablations_benchmark_r1.yaml) at `2982f5876` (/home/ycliang/predicators-five-arms-frozen-20260918). +- Standalone simulator and no explicit uncertainty: round r2 from [continual_standalone_no_uncertainty_r2.yaml](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/continual_standalone_no_uncertainty_r2.yaml) at `0ddc8f7f4` (/home/ycliang/predicators-standalone-nounc-frozen-20260918), after the September 18 revision of both surfaces. +- Scene only: [continual_scene_only_benchmark_r1.yaml](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/continual_scene_only_benchmark_r1.yaml). +- Agentic real-to-sim: [continual_real_to_sim_benchmark_r1.yaml](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/continual_real_to_sim_benchmark_r1.yaml) at `5ea35c91c` (/home/ycliang/predicators-real-to-sim-frozen-20260918). Its prompt asks the agent to write its own `simulator.py` scene from the engine, the manifest and the assets and to model the mechanisms, the same workflow as EMPIRIC. -- EMPIRIC + scene package: [continual_empiric_scene_package_benchmark_r1.yaml](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/continual_empiric_scene_package_benchmark_r1.yaml). +- EMPIRIC + scene package: [continual_empiric_scene_package_benchmark_r1.yaml](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/continual_empiric_scene_package_benchmark_r1.yaml). Fan and Balloons ran as r1 at `d575a8447` (/home/ycliang/predicators-empiric-pkg-frozen-20260918). Boil, Bridge and Domino ran as r2 at `8051333e5` (/home/ycliang/predicators-empiric-pkg-split-frozen-20260919), after their environment files were split so the observable sim core can be shared without the hidden mechanisms; their r1 runs were cancelled and are excluded. To resume the paused seeds, relaunch the same launcher and round key from the same worktree so auto-resume picks up the scorecard and checkpoint. -- Direct agent + scene assets: [continual_direct_scene_files_benchmark_r1.yaml](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/continual_direct_scene_files_benchmark_r1.yaml) at `f6f609636` (/home/ycliang/predicators-direct-scene-files-frozen-20260919). +- Direct agent + scene assets: [continual_direct_scene_files_benchmark_r1.yaml](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/continual_direct_scene_files_benchmark_r1.yaml) at `f6f609636` (/home/ycliang/predicators-direct-scene-files-frozen-20260919). It is the direct agent plus the real-to-sim arm's engine wrapper, scene manifest and URDF and mesh files as read-only references; its prompt only asks it to solve the levels, with no simulator, model files, fitting or model gate. The rendered prompt is in [the prompt review](../prompt-review/2026-09-19-direct-scene-files/direct_agent_scene_files.md). @@ -80,7 +80,7 @@ This is not a matched preflight ablation. The archived Fan transfer experiment is a separate pilot with two seeds each for EMPIRIC and the direct agent. The other agents have not been launched on this variant; their empty rows are missing results, not failures. Only finished seeds enter the bars and curves; pending runs are listed below and do not count as zero successes. -See the [illustrated task description](../amps/fan-exposed-transfer.md) and [launch configuration](https://github.com/BasisResearch/predicators/blob/iclr-empiric-submission/scripts/configs/predicatorv3/continual_fan_transfer_pilot_r1.yaml). +See the [illustrated task description](../amps/fan-exposed-transfer.md) and [launch configuration](https://github.com/BasisResearch/predicators/blob/72d7258ed3992728c8501c615ca1261ec3f49feb/scripts/configs/predicatorv3/continual_fan_transfer_pilot_r1.yaml). This superseded pilot is omitted from the figure; its tables and logs remain archived below. It is also excluded from the paper figure and its data selection. diff --git a/docs/envs/bridge/README.md b/docs/envs/bridge/README.md index 9b3850a5a..f3a38608b 100644 --- a/docs/envs/bridge/README.md +++ b/docs/envs/bridge/README.md @@ -15,5 +15,5 @@ Gray pads mark the two leg sites; standing blocks are legs, lying blocks are spa ## Oracle solve trajectories `oracle_solve_.mp4`: `oracle_process_planning` solving one test task end-to-end (seed 0), recorded with `--make_test_videos`. -The launch flags mirror the `bridge` entry of the retired phased menu `scripts/configs/predicatorv3/envs/all.yaml` (tag `iclr-empiric-submission`), plus `--no_repeated_arguments_in_grounding True` (set globally by that menu's `common.yaml` for config-launched runs, required on a bare CLI for the full spec to plan). +The launch flags mirror the `bridge` entry of the retired phased menu `scripts/configs/predicatorv3/envs/all.yaml` (commit `72d7258ed`), plus `--no_repeated_arguments_in_grounding True` (set globally by that menu's `common.yaml` for config-launched runs, required on a bare CLI for the full spec to plan). The full-spec video was recorded with the default `pybullet_birrt_path_subsample_ratio 1`. diff --git a/predicators/envs/pybullet_domino/task_generators/min_block_generation.py b/predicators/envs/pybullet_domino/task_generators/min_block_generation.py index 1ec0cc17e..cbef2e13a 100644 --- a/predicators/envs/pybullet_domino/task_generators/min_block_generation.py +++ b/predicators/envs/pybullet_domino/task_generators/min_block_generation.py @@ -70,8 +70,8 @@ def _domino_code_digest() -> str: # when no explicit differentiating band is configured - the natural-corner # region shared by the plain and heavy turn variants. Explicit differentiating # bands come from CFG.domino_min_block_turn_* (re-probe them when the friction -# pair moves, with scripts/domino_debug/probe_min_block_bands.py from tag -# iclr-empiric-submission). +# pair moves, with scripts/domino_debug/probe_min_block_bands.py from commit +# 72d7258ed). _DEFAULT_TURN_ENTRY_BAND = (0.26, 0.34) _DEFAULT_TURN_EXIT_BAND = (0.18, 0.26) # Heavy turn variant: legs from the 2026-07-25 canonical-anchor design @@ -561,7 +561,7 @@ def _make_turn_task(env: "PyBulletDominoComposedEnv", # Explicitly configured bands - re-probe them when the friction # pair changes (the differentiating cells move with the # frictions), with scripts/domino_debug/probe_min_block_bands.py - # from tag iclr-empiric-submission. The benchmark's bands live in + # from commit 72d7258ed. The benchmark's bands live in # the domino_high_friction_turn entry of # scripts/configs/empiric/envs.yaml. assert CFG.domino_min_block_turn_entry_hi is not None @@ -585,8 +585,8 @@ def _make_turn_task(env: "PyBulletDominoComposedEnv", "domino_min_block_turn_{entry,exit}_{lo,hi} flags (see the " "domino_high_friction_turn entry of " "scripts/configs/empiric/envs.yaml; probe new bands with " - "scripts/domino_debug/probe_min_block_bands.py from tag " - "iclr-empiric-submission).") + "scripts/domino_debug/probe_min_block_bands.py from commit " + "72d7258ed).") else: entry_leg = round(float(rng.uniform(*_DEFAULT_TURN_ENTRY_BAND)), 2) exit_leg = round(float(rng.uniform(*_DEFAULT_TURN_EXIT_BAND)), 2) diff --git a/predicators/settings.py b/predicators/settings.py index 591c4b5ca..1b395062c 100644 --- a/predicators/settings.py +++ b/predicators/settings.py @@ -1100,7 +1100,7 @@ class GlobalSettings: # agent-intractable pair-corner tasks. Probe candidate bands whenever # the friction pair changes (the differentiating cells move with the # frictions), with scripts/domino_debug/probe_min_block_bands.py from - # tag iclr-empiric-submission. + # commit 72d7258ed. domino_min_block_turn_entry_lo: Optional[float] = None domino_min_block_turn_entry_hi: Optional[float] = None domino_min_block_turn_exit_lo: Optional[float] = None diff --git a/scripts/plotting/monitor_benchmark_arms.py b/scripts/plotting/monitor_benchmark_arms.py index b6dbce841..9c02dc9c5 100644 --- a/scripts/plotting/monitor_benchmark_arms.py +++ b/scripts/plotting/monitor_benchmark_arms.py @@ -274,7 +274,8 @@ def report_text(original: str, rows: List[Row], plot: Dict[str, Any], "(../amps/fan-exposed-transfer.md) and " "[launch configuration]" "(https://github.com/BasisResearch/predicators/blob/" - "iclr-empiric-submission/scripts/configs/predicatorv3/" + "72d7258ed3992728c8501c615ca1261ec3f49feb/" + "scripts/configs/predicatorv3/" "continual_fan_transfer_pilot_r1.yaml).\n" "This column is excluded from the paper figure and its " "data selection.\n\n") diff --git a/tests/approaches/test_oracle_process_planning_boil.py b/tests/approaches/test_oracle_process_planning_boil.py index f407e64db..177f72255 100644 --- a/tests/approaches/test_oracle_process_planning_boil.py +++ b/tests/approaches/test_oracle_process_planning_boil.py @@ -1,7 +1,7 @@ """End-to-end test: oracle_process_planning solves a boil task. Mirrors the retired phased config (``predicatorv3/oracle.yaml`` + -``envs/all.yaml`` + ``common.yaml`` at tag iclr-empiric-submission) so that a +``envs/all.yaml`` + ``common.yaml`` at commit 72d7258ed) so that a regression in either the approach (process planning + bilevel refinement) or the boil env's skill execution would surface here. diff --git a/tests/approaches/test_oracle_process_planning_bridge.py b/tests/approaches/test_oracle_process_planning_bridge.py index 1737a2191..89612b200 100644 --- a/tests/approaches/test_oracle_process_planning_bridge.py +++ b/tests/approaches/test_oracle_process_planning_bridge.py @@ -1,8 +1,8 @@ """End-to-end test: oracle_process_planning solves a bridge task. Mirrors the retired phased config (``predicatorv3/oracle.yaml`` + -``envs/all.yaml`` (bridge entry) + ``common.yaml`` at tag -iclr-empiric-submission) so that a regression in the approach (process +``envs/all.yaml`` (bridge entry) + ``common.yaml`` at commit +72d7258ed) so that a regression in the approach (process planning + bilevel refinement), the bridge env's glue/weld machinery, or the skill factories would surface here.