feature/export benchmark aime#65

Merged

benjibc merged 11 commits intomainfrom

feature/export-benchmark-aime

Aug 12, 2025

Commits on Aug 12, 2025

benchmarks: add export_benchmark, direct runner, suites.aime25 and wire AIME as aime25_low; add module runner
benjibc
committed
bench: robust auto-import of suites; explicit fallback import for known names
benjibc
committed
bench: ensure runtime annotations (drop __future__); fix EvaluationRow type check in suite
benjibc
committed
bench: set extra_body.reasoning_effort rather than extra_body.reasoning.effort for Fireworks
benjibc
committed
bench: consolidate AIME export to suites only; set num_runs=8 default
benjibc
committed
bench: add --max-tokens and --max-concurrency overrides; plumb into runner
benjibc
committed
add aime benchmark
benjibc
committed
bench: add GPQA exported benchmark suite; default low effort, num_runs=8
benjibc
committed
bench: export name aime25; default low effort with max_tokens=131000; keep num_runs=8
benjibc
committed
tests: remove duplicate AIME2025 pytest example; use exported suite in eval_protocol/benchmarks/suites/aime25.py as source of truth
benjibc
committed
fix types
benjibc
committed