Skip to content

fix(evaluator): guard IndexError on truncated judge choice (JudgeEvaluator/RMBEvaluator) - #2522

Open
WatchTree-19 wants to merge 1 commit into
open-compass:mainfrom
WatchTree-19:fix-judge-evaluator-truncated-choice
Open

fix(evaluator): guard IndexError on truncated judge choice (JudgeEvaluator/RMBEvaluator)#2522
WatchTree-19 wants to merge 1 commit into
open-compass:mainfrom
WatchTree-19:fix-judge-evaluator-truncated-choice

Conversation

@WatchTree-19

Copy link
Copy Markdown

problem

JudgeEvaluator.score and RMBEvaluator.score extract the winning model letter with:

choice = prediction.split("\"Choice\": \"Model ")[-1][0] if len(prediction) != 0 else None

the guard checks that prediction is non-empty, but the [0] indexes the split tail, which can be empty even when the prediction isn't - e.g. a judge response truncated right at the marker (... "Choice": "Model with nothing after, common under max_tokens clipping). then split(...)[-1] is "" and ""[0] raises IndexError, crashing scoring for the whole batch.

repro

p = '{"Choice": "Model '            # truncated before the letter
p.split('"Choice": "Model ')[-1][0]  # IndexError: string index out of range

fix

index the split tail only when it's non-empty (missing choice -> None, which simply doesn't match the gold winner). applied to both JudgeEvaluator and RMBEvaluator.

test

added tests/openicl/test_icl_judge_evaluator.py: the truncated-marker case no longer crashes (scores 0.0), and a normal "Choice": "Model A" still scores correctly.

…aluator/RMBEvaluator

prediction.split('"Choice": "Model ')[-1][0] was guarded only by len(prediction)!=0,
but crashes when the split tail is empty (a judge output truncated right at the
marker). Index the tail only when non-empty. +test.

Signed-off-by: WatchTree-19 <119982314+WatchTree-19@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants