feat(plugin): ship coder_eval as a Claude Code plugin + marketplace

[uipreliga]

What

A new way to use Coder Eval: from inside Claude Code, alongside the CLI and the GitHub
Action. This repo becomes a Claude Code plugin marketplace hosting one coder-eval
plugin, so the whole loop — scaffold a suite, author a task, check whether a skill
triggers, read the results, wire it into CI — runs in the agent.

/plugin marketplace add UiPath/coder_eval
/plugin install coder-eval@coder-eval

That adds five slash commands: /coder-eval:init, /coder-eval:skill-check,
/coder-eval:task, /coder-eval:analyze, /coder-eval:ci. They drive the coder-eval
CLI, which stays the prerequisite — the plugin ships prompts and reference material, not a
second implementation.

Standing cost is ~464 tokens always-on for all five (claude plugin details); bodies
load only on invoke.

What a reviewer should check

No runtime code changes. git diff --stat <base>..HEAD -- src/ is empty. No model,
criterion, agent, CLI flag or merge layer moved, so no existing evaluation result can
change. Everything new is a distribution surface, one test-harness generator, one lint
clause, one CI job, and docs.

Three invariants carry the weight:

1. The bundled criteria reference is generated, not written (CE032,
   tests/lint/plugin_reference.py). An installed plugin is copied to
   ~/.claude/plugins/cache/ without its parent directories, so a skill cannot read
   docs/TASK_DEFINITION_GUIDE.md at runtime — the criterion vocabulary has to ship
   inside plugins/coder-eval/, where a hand-maintained copy would drift on the next
   criterion change. make plugin-reference renders it from the SuccessCriterion union;
   CE032 re-renders and diffs. Inherited base fields and the discriminator name are
   computed, never listed — the stop_early refactor is exactly the change a hardcoded
   list would have leaked into all 14 sections.

2. The frontmatter guards are load-bearing, not belt-and-braces. Verified by spike:
   claude plugin validate --strict does not inspect skill frontmatter at all — a
   SKILL.md carrying both an unsupported name: and an invented key passes with exit 0
   and zero warnings. TestPluginArtifacts is the only thing standing between a typo'd
   key and a skill that silently never triggers. It also enforces path containment (no
   skill may name a repo path that won't exist post-install) across every SKILL.md.

3. The CI assert expands, it does not just validate. coder-eval plan exits 0 even
   when dataset.paths names a file that was never copied — verified both ways — so a
   plan-only check would have passed vacuously. plugin-validate copies the activation
   template into $RUNNER_TEMP, outside the source tree, and asserts expand_dataset
   yields 6 row-tasks with both label polarities. No secrets, no agent run, no per-PR cost.

Also in here

• CE026 gains two things: its doc scan now covers plugins/**/*.md (the ci skill
  emits a workflow users copy verbatim, so it is held to the same agent-runtime
  prerequisite standard), and a fourth clause — every with: key on a snippet's
  action step must be a real action.yml input. GitHub ignores unknown inputs rather than
  failing, so a rename would silently degrade every copied workflow; simulating a model
  rename fires on all four surfaces. Consequence to know: renaming an action input, or
  changing the action's runtime prerequisites, now means updating the ci skill too.
  Documented in CLAUDE.md.
• plugin.json carries a version pinned to pyproject.toml's, because
  --strict rejects a manifest without one. release.yml's existing bump step seds it
  alongside action.yml; tests/test_action_version_pin.py — already the owner of that
  invariant — guards both the value and the sed's line shape at rest.

Evidence the prompts were exercised, not just linted

Mechanical guards prove the plugin loads and is self-contained; they cannot prove the five
prompts are any good. So all five were run against a real scratch repo (two planted skills

• a stub CLI) and, for analyze, a real 947-task two-variant experiment run — tools
  allowlisted so no paid coder-eval run could fire.

They behaved: skill-check designed 18 rows (10 positive / 8 distractor, none naming the
skill, distractors genuinely adjacent), validated, and stopped to ask before spending;
init wrote .env.example and never .env; ci chose pull_request over
pull_request_target with explicit reasoning about fork secrets.

Three prompt-quality defects were found and fixed:

• analyze got a 947-task run's headline exactly right (180 rows lost to a 120s timeout)
  but drifted on two secondary figures — it split them 146/34 where the run holds 144/36.
  Its principles now require every count to come out of the extraction command.
• init scaffolded a task whose prompt dictated pypdf while a criterion grepped for it —
  a criterion that cannot fail but reads as coverage.
• task gave weight 1.5 to a criterion matching a literal (--json) the prompt has to
  name.

Out of scope

/coder-eval:compare and /coder-eval:triage (v2, once usage is observed); a second
coder-eval-dev plugin — .claude/commands/ stays repo-local and is byte-for-byte
unchanged; a duplicate example task under tasks/ — the plugin template is the single
canonical copy.

Verification

make format / make check / make lint (207) / make test (3744 passed, 8 skipped,
87.70% coverage) all clean. Both manifests pass claude plugin validate --strict with
zero warnings.

Two things this branch cannot rehearse, both worth watching:

• plugin-validate has never run on a runner. Every step was verified locally,
  including the scaffold assert and its negative, but the job itself is new.
• Merging triggers a minor release, which runs the new plugin.json sed for the first
  time. It fails loudly on a miss (grep -q guard) and the at-rest test catches drift
  afterwards, but the release path can't be exercised beforehand.

---

🤖 Generated with Claude Code

[github-actions]

Claude finished @uipreliga's task in 1m 33s —— View job

---

Code Review in Progress

Todo List:

• Read .github/code_review.md for review standards
• Read CLAUDE.md for project conventions
• Get full PR diff from origin/main
• Review all 27 changed files with full context
• Verify the three claimed invariants
• Check cross-file consistency
• Perform "what's missing" analysis
• Format and post final review

Starting review now...

[uipreliga]

Follow-up: the earlier response triaged 14 of this review's 16 findings. Two were never reached (nits 8 and 9), and one was deferred pending a "say the word". All three are now closed in f8c9691, each verified against the real schema and CLI rather than by reading.

---

1. [Axis: What's Missing] analyze teaches a task.json field vocabulary the schema does not have — ✅ Fixed

The sharpest of the remaining findings, and it reproduced exactly as filed. Six of the thirteen keys in the jq summary template do not exist at the top level of a real task.json:

Documented	Actual
turns	iterations
total_tokens	total_token_usage.input_tokens + .output_tokens
assistant_turn_count	total_assistant_turns
max_turns	task_config.resolved.run_limits.max_turns
total_cost_usd	total_token_usage.total_cost_usd
criteria_count / all_criteria_perfect	derived from success_criteria_results

Two more beyond what was filed: failed_criteria named type rather than criterion_type, and error_excerpt was defined as the first ~200 chars of error / output / Instructions — but a criterion result carries only error and details; there is no output and no Instructions. That one matters in practice, because the common case has error: null and all its signal in details ("Matched 0/1 required commands (filters: …)"), so the excerpt that drives clustering in step 3 came back empty exactly when a criterion simply did not match.

Why it is worse than a typo: jq answers a missing key with null instead of failing, so this shipped as a summary full of nulls that reads like a run with no data — and it did so precisely on the >20-task path, which is the one place the skill explicitly forbids falling back to reading whole files.

Fixed in both copies — the reviewer's "Parallel paths" note was right that .claude/commands/coder-eval-run-analysis.md carries the same block and the same wrong names. The corrected program was run against a real passing task and a real failing one (7 failing criteria) to confirm every path resolves and failed_criteria populates with real diagnostics.

Guarded, since the same bug shipped twice: TestRunRecordFieldVocabulary resolves every field those two surfaces read off a run record against EvaluationResult / CriterionResult, and is mutation-tested against the exact block that shipped. Scoped deliberately — only fenced blocks mentioning success_criteria_results, and only the head of each dotted path, since task_config is a free-form dict and .task_config.resolved.run_limits.max_turns is unverifiable from the schema. That scoping is documented in the test's docstring rather than left implicit.

2. [Axis 8, nit 9] init instructs coder-eval plan <task-directory>, which the CLI always rejects — ✅ Fixed

Reproduced in a scratch repo: coder-eval plan tasks exits 1 with Expected a YAML task file but got a directory. "Iterate until it exits 0" was therefore an unreachable loop condition. Now the glob form, with coder-eval plan (no argument, discovers tasks/ recursively — verified exit 0) named as the alternative. The follow-on sentence carried the same wrong assumption, so the two outcomes are now distinguished: "reports no tasks" means the scaffold is empty, the directory error means the argument shape is wrong.

On the proposed CE035 ("documented coder-eval invocations must be executable as written"): recorded in .claude/harness-candidates.md rather than built, with the reasoning split out. The cheap half — does the subcommand exist in the Typer app — would not have caught this. The half that would is argument shape, which needs either a real invocation or a per-command arity model duplicating the CLI signature. The live-smoke counterpart the review also proposed is recorded alongside it, since that is the only form that proves argument shape rather than command existence.

3. [Nit 2] write()/check()/__main__ duplicate doc_indexes.py — ✅ Fixed (was deferred)

Taking you up on it. Extracted tests/lint/generated.py with write_all(rendered) / diff_all(rendered); both generated-surface checkers now contribute only their render. The two copies had diverged in exactly one respect — only the plugin-reference one created a missing target and its parents — and the shared engine keeps that more general behaviour, so a generated file that was never written reports as drift rather than raising. Four tests pin it, including the create-parents branch and the missing-file diff. make docs-indexes and make plugin-reference both still no-op.

This also closes the separate 🔵 note that plugin_reference.py::write() was untested — though only partly by this change: a test_write_is_idempotent for it had already landed in a later commit than the review.

---

Nit 8 — no change needed

name: in the skill-frontmatter allowlist. Verified at HEAD: this was fixed in a commit later than the review's baseline. The set is unchanged (still excludes name, deliberately), but the framing the finding actually objected to is gone — the message now reads "are outside the set this plugin deliberately restricts itself to … If one is genuinely needed, add it here with a reason", and the comment states outright that this is the plugin's house style and not the specification's limit. That is what the finding asked for.

---

Verification

make format / make check / make typecheck clean (0 errors; the one warning is pre-existing in agents/antigravity_agent.py, untouched here). make lint 312 passed, up from 305 — the seven new tests are the field-vocabulary guard and the generated-surface engine. make test 4048 passed, 4 skipped.

One note on history: this branch was rebased onto main first, which forced one rename reviewers may have referenced. main shipped its own CE032 (ce032_criteria_path_seam, a real BaseRule), colliding with this branch's plugin-reference parity rule of the same id — so ours moved to the next free id, CE033. The rebase also required regenerating reference/criteria.md (main added the in-tree cli_called criterion and glob-pattern path descriptions) and bumping the plugin manifest's derived version pin to 0.9.5.