Skip to content

Judge tuning (4/5): Run multi-fidelity judge searches - #133

Open
ErlisLushtaku wants to merge 37 commits into
erlislushtaku/feat/cache-native-usagefrom
erlislushtaku/feat/tune-judge
Open

ErlisLushtaku wants to merge 37 commits into
erlislushtaku/feat/cache-native-usagefrom
erlislushtaku/feat/tune-judge

Conversation

@ErlisLushtaku

@ErlisLushtaku ErlisLushtaku commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

Runs the multi-fidelity search. Each NePS trial writes a config for the target meta-eval task and runs it in a subprocess, so the GPU is freed between judges and every trial is a normal meta-eval run that shares its judgement cache.

  • Objectives are agreement (NePS minimizes 1 - agreement) and expected judging cost in USD per 1,000 battles, computed from native token counts including cached judgements. NePS's own cost field stays unset because it measures search budget, not judge cost.
  • Failed trials, such as a CUDA out-of-memory, are recorded with infinite objective values and the search continues (ignore_errors defaults to true), since one bad configuration should not stop a long run. Pricing and config errors still abort.
  • NePS returns once the budget is spent, possibly while other workers are still evaluating. The primary process therefore waits for active trials, reports the agreement/cost Pareto front, picks the best agreement configuration per judge model at full fidelity and evaluates those on the test split. Helpers join with --tune_judge.search_only true and the same run_dir.
  • Prices come from tune_judge.price_per_million_tokens, and a cost run stops before evaluating if a searched model has none. Automatic reference prices follow in Judge tuning (5/5): Add reference prices and usage docs #136.

Merge order: #138 → #135 → #140 → #133 → #136.

Judge tuning needs to select configurations on one partition of the
human-labeled arena and report on another. Battles are assigned by a
salted hash of their first user prompt, so repeated prompts never cross
partitions and changing run.seed cannot move battles between them.
A tune_judge section on a meta-eval task expands a grid of judge
overrides and runs each configuration as its own meta-eval subprocess on
the validation split. After each rung, survivors are chosen by the
non-dominated sort on judge cost and human agreement from Salinas et al.
(ICML 2025). The best configuration per judge model is then scored once on
the held-out test split.
tune-judge-* tasks reuse the meta-eval arena definitions and run each trial
as the matching meta-eval task, so both share one judgement cache. The custom
successive halving and Pareto selection give way to neps (priorband by
default, with the base config as prior; hyperband and mo_hyperband), run via
AskAndTell with validation battles per model as the fidelity.
@ErlisLushtaku ErlisLushtaku changed the title WIP: Add a tune-judge task that searches judge configs with neps WIP: Judge tuning (3/4): Run multi-fidelity judge searches Oct 7, 2026
@ErlisLushtaku
ErlisLushtaku changed the base branch from main to erlislushtaku/feat/judge-tuning-02-neps-config October 7, 2026 13:15
@ErlisLushtaku ErlisLushtaku changed the title WIP: Judge tuning (3/4): Run multi-fidelity judge searches WIP: Judge tuning (3/5): Run multi-fidelity judge searches Oct 9, 2026
@ErlisLushtaku ErlisLushtaku changed the title WIP: Judge tuning (3/5): Run multi-fidelity judge searches WIP: Judge tuning (4/6): Run multi-fidelity judge searches Oct 9, 2026
@ErlisLushtaku
ErlisLushtaku changed the base branch from erlislushtaku/feat/judge-tuning-02-neps-config to erlislushtaku/feat/cache-native-usage October 9, 2026 09:15
@ErlisLushtaku ErlisLushtaku changed the title WIP: Judge tuning (4/6): Run multi-fidelity judge searches WIP: Judge tuning (4/5): Run multi-fidelity judge searches Oct 13, 2026
@ErlisLushtaku ErlisLushtaku changed the title WIP: Judge tuning (4/5): Run multi-fidelity judge searches Judge tuning (4/5): Run multi-fidelity judge searches Oct 13, 2026
@ErlisLushtaku
ErlisLushtaku marked this pull request as ready for review October 13, 2026 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant