Repository navigation
Reference rules no-eval-call and prefer-use-over-utilize are narrower than their own prompts #591
Description
The prefer-use-over-utilize entry in the reference export (assets/reference.json, published in 0.13.4) contradicts its own prompt.
The prompt says: "It should match case-insensitively and cover the inflected forms."
The rule swaps only:
swap:
utilize: use
utilizes: uses
utilizing: usingIt has no utilized and no utilization. So a rule written correctly from the prompt, with fail cases like We utilized the API, is graded against a reference that doesn't fire on those cases. taskless verify --json with the reference rule staged against such fixtures reports:
fail case did not fire: …/tests/fail/utilization-noun/ (the rule did not match it)
fail case did not fire: …/tests/fail/utilized-past/ (the rule did not match it)
Downstream, taskless/taskless's generation evals run the reference rule against generated fixtures (their "step 5") and count this as our fixtures being wrong (taskless/taskless#406).
Fix: add utilized: used (and utilization, if the noun is in scope) to the swap, plus a reference fail case for each, so the reference's own tests pin the forms its prompt promises. If the narrower rule is what was meant, the prompt should drop "cover the inflected forms" instead.
Activity
[-]Reference rule prefer-use-over-utilize misses the inflected forms its prompt asks for[/-][+]Reference rules no-eval-call and prefer-use-over-utilize are narrower than their own prompts[/+]theCodeDrift commentedon Oct 30, 2026
The same problem affects
no-eval-call.The prompt says: "Create a rule that flags any call to
eval()in TypeScript."The rule is:
pattern: eval($ARG), which matches exactly one argument. It misseseval()andeval(a, b). That's the arity boundarycreate-sg-rulewarns about ("A pattern that starts at two arguments when it was meant to start at one passes a test suite whose fixtures all have two"). The recipe also tells authors to put a fixture on each side of that boundary.A model that follows the recipe gets marked down for it. On
claude-sonnet-5, taskless/taskless's sg eval missed 2 of 3 runs, each time at step 5 (reference rule × generated fixtures):Fix: match any arity, e.g.
pattern: eval($$$ARGS), and add reference fail cases foreval()andeval(a, b).theCodeDrift commentedon Oct 30, 2026
Reopening: #592 fixed the vale half (
prefer-use-over-utilizenow coversutilized/utilization), but the sg half is still open. Theno-eval-callreference rule matcheseval($ARG), so it misseseval()andeval(a, b), though its prompt says "any call toeval()". Our generator writes fail cases for those shapes, as the CLI's own guidance asks, and step 5 of the eval protocol (the reference rule run against our fixtures) fails on them. Onclaude-sonnet-5that was 2 of 3eval:sgruns. A pattern that covers any arity (e.g.eval($$$ARGS)) plus fail cases for zero and two arguments would close it.