Repository navigation
Judge tuning (2/5): Add held-out splits and NePS configuration - #135
Open
ErlisLushtaku wants to merge 19 commits into
Open
ErlisLushtaku wants to merge 19 commits into
ErlisLushtaku wants to merge 19 commits into
Conversation
Judge tuning needs to select configurations on one partition of the human-labeled arena and report on another. Battles are assigned by a salted hash of their first user prompt, so repeated prompts never cross partitions and changing run.seed cannot move battles between them.
This was referenced Oct 7, 2026
…u/feat/judge-tuning-01-arena-split
…lislushtaku/feat/judge-tuning-02-neps-config
…u/feat/judge-tuning-01-arena-split
…lislushtaku/feat/judge-tuning-02-neps-config
…lislushtaku/feat/judge-tuning-02-neps-config
ErlisLushtaku
changed the title
ErlisLushtaku
commented
Oct 9, 2026
ErlisLushtaku
changed the title
ErlisLushtaku
changed the title
ErlisLushtaku
changed the base branch from
erlislushtaku/feat/judge-tuning-01-arena-split
to
erlislushtaku/fix/vllm-request-context
October 13, 2026 12:07
ErlisLushtaku
changed the title
ErlisLushtaku
marked this pull request as ready for review
October 13, 2026 12:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the held-out split and the NePS configuration that the tuning runner in #133 builds on.
meta_eval.split(validationortest, defaultall) restricts sampling to one half of the arena prompts,meta_eval.validation_fractionsets the share (default 0.5). The split hashes the prompt text, so duplicate prompts stay on one side and the partition does not depend onrun.seed. Tuning selects on validation and reports on test, so a different seed must not move battles across.tune-judgetask with defaults:meta-eval-lmarena-140k-enas the target, agreement and cost objectives,mo_hyperbandand 10 to 90 battles per model. Trials run under the target meta-eval task, so they reuse its defaults and judgement cache.tune_judge.search_spaceis nested like the run config. A list is categorical and{lower, upper}is a range whose type follows the config field. The reference judge is the prior unlessprioris set.tune_judge.run_dirselects a directory to resume or share between workers, and a config hash rejects resuming with different search settings. NePS state stays underneps/, apart from the inference cache.tuneextra pins NePS 0.17.0. Download coverage skipstune-judge, which has no dataset. CI also runs for PRs based on another stack branch.Merge order: #138 → #135 → #140 → #133 → #136.