Skip to content

fairtask-eval: lay out an instance by quoting the decisive lines - #12

Merged
mnkprs merged 1 commit into
mainfrom
eval-skill-layout
Aug 29, 2026
Merged

fairtask-eval: lay out an instance by quoting the decisive lines#12
mnkprs merged 1 commit into
mainfrom
eval-skill-layout

Conversation

@mnkprs

@mnkprs mnkprs commented Aug 29, 2026

Copy link
Copy Markdown
Owner

Why

/fairtask-eval lay out <id> ran npm run show (which only writes examples/<id>/… and prints one line) and then, following the skill's general "show the output verbatim" rule, pasted the whole issue and the test patch into the chat. For a long issue and a diff that is a raw dump; the files are already on disk for anyone who wants the full text.

What changed

  • skills/fairtask-eval/SKILL.md: the lay-out row tells the agent to open the three files itself, and step 3 gains an exception for this operation: show the wrote … line, the human labels on one line, then two to four exact quotes with file:line — what the issue asks for, and what the graded tests require that the issue does not say — with a pointer to examples/<id>/ for the full text. Completion: a reader can see the discrepancy (or its absence) without opening the files.
  • README §2b: the /fairtask-eval lay out row describes the new behaviour.

… by pasting the files

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mnkprs
mnkprs merged commit 29edcc9 into main Aug 29, 2026
2 checks passed
@mnkprs
mnkprs deleted the eval-skill-layout branch August 29, 2026 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant