Skip to content

fix(benchmark): gym cards name their artifact + grade on done — the 0-resolved root cause - #2309

Merged
joelteply merged 1 commit into
canaryfrom
fix/gym-card-artifact-grade-on-done
Aug 15, 2026
Merged

fix(benchmark): gym cards name their artifact + grade on done — the 0-resolved root cause#2309
joelteply merged 1 commit into
canaryfrom
fix/gym-card-artifact-grade-on-done

Conversation

@joelteply

Copy link
Copy Markdown
Contributor

The incident (live, 2026-08-15)

Board at 0 resolved, rail banner: "16 runs gone quiet — live but no artifact activity." Citizens holding gym cards (frontier-rs / hard-rs) wander their workspaces inventing paths (swe/benchmarks/frontier-rs/src/lib.rs), and one said in-room the file she was pointed at "does not exist in this workspace."

Two defects, one seam

1. Dispatch never named the artifact. benchmark/dispatch loaded gym tasks RAW, skipping the canonical require_hands_for_code normalization cognition/eval applies at load. The card body named NO solution file (gym rows carry none), dispatch kept its own divergent {id}.rs fallback for the grading record, and eval derives sol_{id}.rs. Three surfaces, three answers — the citizen was graded against a path she was never told.

2. No grader ever fired on a gym card. benchmark_grade's done-transition arm graded SWE cards only; the "different grader" its own comment promised was never wired (gym_grader had zero done-event callers). A perfect artifact could NEVER resolve — while the kickoff promised "your artifact gets graded."

The fix (adapter doctrine holds — no parallel runner, no cognition changes)

  • require_hands_for_code is now pub(crate): one derivation, both readers. Dispatch normalizes each gym task before composing the card, so the body names sol_<id>.rs and the prompt demands write→compile→verify. Title gist keeps the AUTHORED prompt so boards stay scannable and dispatch_card_key still parses.
  • New gym arm in benchmark_grade::grade_card, on the SAME work/state done event the SWE arm uses: resolve the task from the same embedded gym (normalized_gym_task, same rule as the card — agreement by construction), read <home>/citizens/peers/<owner>/workspace/<solution_file>, grade through the existing test_grade harness, post the verdict into the room as a participant. Missing/empty artifact posts an ABSENCE line — never a silent skip or a fake fail.

Validation

  • cargo check -p continuum-core --features metal,accelerate clean
  • New regression test gym_grade_reads_the_same_artifact_path_the_card_names (pins card↔grader path agreement on a real embedded gym task) — passes
  • All 9 existing commands::benchmark::tests pass (title/key contract intact)

Task #439. Live proof owed after deploy: a citizen resolving a gym card end-to-end (card names file → she writes it → done → ✅ verdict in room).

🤖 Generated with Claude Code

https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…-resolved root cause (#439)

Two defects, one seam, both live-glass-boxed 2026-08-15 while the board sat at
0 resolved with '16 runs gone quiet — live but no artifact activity':

1) DISPATCH NEVER NAMED THE ARTIFACT. benchmark/dispatch loaded gym tasks RAW,
   skipping the canonical require_hands_for_code normalization cognition/eval
   applies at load — so the card body named NO solution file (gym rows carry
   none) while dispatch kept its own divergent `{id}.rs` fallback and eval
   derives `sol_{id}.rs`. Three surfaces, three answers; the citizen got none
   of them. Live: a frontier-rs claimer wandering `swe/benchmarks/.../lib.rs`,
   and 6f69388d saying in-room the file she was pointed at 'does not exist in
   this workspace'. Fix: normalize at dispatch (require_hands_for_code is now
   pub(crate) — ONE derivation, both readers), title gist keeps the AUTHORED
   prompt so boards stay scannable and dispatch_card_key still parses.

2) NO GRADER EVER FIRED ON A GYM CARD. benchmark_grade's done-transition arm
   graded SWE cards only; the 'different grader' its comment promised was never
   wired (gym_grader had zero done-event callers), so a perfect artifact could
   NEVER resolve — while the kickoff told her 'your artifact gets graded'.
   Fix: gym arm on the same done event — resolve the task from the same
   embedded gym via normalized_gym_task (same rule as the card, agreement by
   construction, pinned by test), read <home>/citizens/peers/<owner>/workspace/
   <solution_file>, grade through the existing test_grade harness, post the
   verdict into the room as a participant. Missing/empty artifact posts an
   ABSENCE line, never a silent skip or fake fail.

Adapter doctrine holds: no parallel runner, no cognition changes — the room's
own work/state event drives the grade and the verdict lands in the room.

Regression test gym_grade_reads_the_same_artifact_path_the_card_names pins the
card↔grader artifact-path agreement on a real embedded gym task.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
@joelteply
joelteply merged commit 20adbaf into canary Aug 15, 2026
5 checks passed
@joelteply
joelteply deleted the fix/gym-card-artifact-grade-on-done branch August 15, 2026 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant