[Dataset] Align ARC-AGI-1/2 evaluation with ARC Prize protocol

[Vicentvankor]

Summary

Align the default ARC-AGI-1 and ARC-AGI-2 dataset configs with the ARC Prize benchmarking protocol.

• render the ARC Prize baseline prompt with JSON grids;
• evaluate every test pair instead of only test[0];
• prefer a boxed grid, then backscan for the final complete JSON grid;
• when the primary output is not parseable, reuse the evaluated OpenCompass model for the official JSON-extraction retry while inference is still active;
• score ARC-AGI-2 as two attempts per pair, where either exact match is accepted, before aggregating pair scores within each task;
• keep the archived arc_prize_public_evaluation_gen_fedd04.py config on the legacy OpenCompass behavior.

Motivation

The existing OpenCompass adapter diverges from the benchmark-owner protocol in several score-affecting places:

Stage	Existing default	ARC Prize protocol
Prompt	custom wizard prompt, Python repr, fabricated empty output	owner baseline prompt with JSON examples
Test coverage	first test pair only	every test pair
Parsing	first [[...]] in the response	boxed answer, then final complete JSON value
Parse recovery	none	same-model JSON extraction retry
ARC-AGI-2 attempts	average two replicated runs	either of two attempts may solve each pair

These differences can turn valid final answers into errors and make scores incomparable with ARC Prize results.

Compatibility and scope

Only the two current ARC-AGI configs opt into protocol='owner' and ARCPrizeGenInferencer.

The shared loader and evaluator continue to default to legacy, and the archived fedd04 config is unchanged. No non-ARC dataset config imports or enables the new protocol.

Primary sources

• ARC Prize baseline prompt
• OpenAI adapter and same-model JSON extraction
• Official scoring implementation
• Official response parsing schemas
• ARC-AGI-2 two-attempt task format

All benchmarking-harness links above are pinned to commit 688c9be4fb270ad7eede09fe8bba6f8187be3bed.

Validation

15 passed in 7.49s

Also verified:

• yapf --diff is clean for all changed Python files;
• git diff --check passes;
• the dataset implementation and tests compile with Python 3.12;
• the pushed branch is exactly one commit ahead of upstream main@954e632d;
• the remote diff contains only the ARC dataset README, the two ARC configs, the shared ARC implementation, and its dedicated tests.