fix(benchmark): gym cards name their artifact + grade on done — the 0-resolved root cause - #2309
Merged
Merged
Conversation
…-resolved root cause (#439) Two defects, one seam, both live-glass-boxed 2026-08-15 while the board sat at 0 resolved with '16 runs gone quiet — live but no artifact activity': 1) DISPATCH NEVER NAMED THE ARTIFACT. benchmark/dispatch loaded gym tasks RAW, skipping the canonical require_hands_for_code normalization cognition/eval applies at load — so the card body named NO solution file (gym rows carry none) while dispatch kept its own divergent `{id}.rs` fallback and eval derives `sol_{id}.rs`. Three surfaces, three answers; the citizen got none of them. Live: a frontier-rs claimer wandering `swe/benchmarks/.../lib.rs`, and 6f69388d saying in-room the file she was pointed at 'does not exist in this workspace'. Fix: normalize at dispatch (require_hands_for_code is now pub(crate) — ONE derivation, both readers), title gist keeps the AUTHORED prompt so boards stay scannable and dispatch_card_key still parses. 2) NO GRADER EVER FIRED ON A GYM CARD. benchmark_grade's done-transition arm graded SWE cards only; the 'different grader' its comment promised was never wired (gym_grader had zero done-event callers), so a perfect artifact could NEVER resolve — while the kickoff told her 'your artifact gets graded'. Fix: gym arm on the same done event — resolve the task from the same embedded gym via normalized_gym_task (same rule as the card, agreement by construction, pinned by test), read <home>/citizens/peers/<owner>/workspace/ <solution_file>, grade through the existing test_grade harness, post the verdict into the room as a participant. Missing/empty artifact posts an ABSENCE line, never a silent skip or fake fail. Adapter doctrine holds: no parallel runner, no cognition changes — the room's own work/state event drives the grade and the verdict lands in the room. Regression test gym_grade_reads_the_same_artifact_path_the_card_names pins the card↔grader artifact-path agreement on a real embedded gym task. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The incident (live, 2026-08-15)
Board at 0 resolved, rail banner: "16 runs gone quiet — live but no artifact activity." Citizens holding gym cards (frontier-rs / hard-rs) wander their workspaces inventing paths (
swe/benchmarks/frontier-rs/src/lib.rs), and one said in-room the file she was pointed at "does not exist in this workspace."Two defects, one seam
1. Dispatch never named the artifact.
benchmark/dispatchloaded gym tasks RAW, skipping the canonicalrequire_hands_for_codenormalizationcognition/evalapplies at load. The card body named NO solution file (gym rows carry none), dispatch kept its own divergent{id}.rsfallback for the grading record, and eval derivessol_{id}.rs. Three surfaces, three answers — the citizen was graded against a path she was never told.2. No grader ever fired on a gym card.
benchmark_grade's done-transition arm graded SWE cards only; the "different grader" its own comment promised was never wired (gym_graderhad zero done-event callers). A perfect artifact could NEVER resolve — while the kickoff promised "your artifact gets graded."The fix (adapter doctrine holds — no parallel runner, no cognition changes)
require_hands_for_codeis nowpub(crate): one derivation, both readers. Dispatch normalizes each gym task before composing the card, so the body namessol_<id>.rsand the prompt demands write→compile→verify. Title gist keeps the AUTHORED prompt so boards stay scannable anddispatch_card_keystill parses.benchmark_grade::grade_card, on the SAMEwork/statedone event the SWE arm uses: resolve the task from the same embedded gym (normalized_gym_task, same rule as the card — agreement by construction), read<home>/citizens/peers/<owner>/workspace/<solution_file>, grade through the existingtest_gradeharness, post the verdict into the room as a participant. Missing/empty artifact posts an ABSENCE line — never a silent skip or a fake fail.Validation
cargo check -p continuum-core --features metal,acceleratecleangym_grade_reads_the_same_artifact_path_the_card_names(pins card↔grader path agreement on a real embedded gym task) — passescommands::benchmark::testspass (title/key contract intact)Task #439. Live proof owed after deploy: a citizen resolving a gym card end-to-end (card names file → she writes it → done → ✅ verdict in room).
🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo