Add benchmark workflow#7
Open
Joshdb-Amazon wants to merge 3 commits into
Open
Conversation
A CI gate that scores this repo's agent skills by running them through kiro-cli
in Docker and grading against per-scenario rubrics. A PR touching skills/,
agents/, tests/, or the benchmark/ grader is scored against the repo's own
assets. See benchmark/README.md.
Inputs (repo root):
- tests/ + tests/registry.json — 34 active scenarios (32 nki + 1 autoport +
1 contribution-validation); registry.json is the source of truth for active ones
- skills/neuron-dev-contribution-validation/ + its agent spec (needed by the
contribution-validation scenario; existing skills left untouched)
Grader (benchmark/):
- run_skill_tests.py — load/run/score/gate (stdlib only)
- shell/{docker,common}.sh — build image + run kiro-cli in Docker (kiro-only;
no upstream Claude/Node/LangSmith paths)
- environment/Dockerfile — the run image (python:3.11-slim + kiro-cli)
- README.md
Workflow (.github/workflows/benchmark.yml):
- self-contained: checkout -> setup-python -> run benchmark/run_skill_tests.py
--nad-root=. -> upload results. Scores the repo's own assets; no cross-repo
refs or tokens.
- pull_request (paths: skills/agents/tests/benchmark/workflow) + workflow_dispatch.
- Requires a KIRO_API_KEY Actions secret (agent + judge auth).
Scoring: combined = 0.75*pattern + 0.25*judge; passed = combined >= 0.90;
gate fails below --gate (default 90%).
Reliability:
- salvage-on-timeout: if the agent times out but already wrote a non-empty
output.py (it often completes the task then hangs on a trivial final step),
that output is scored instead of discarded.
- NO infra-error retry: a failure (incl. a no-output timeout from the agent
going down a bad path) is reported as-is, so the benchmark measures skill
quality honestly. Deliberate multi-attempt runs use --count (transparent Pass@k).
- Infra errors are logged (type + container tail) for human legibility.
Adds the label-triggered gate and the surrounding finalization on top of the
single-repo benchmark gate.
- .github/workflows/benchmark-external.yml ("NAD Benchmark (label-triggered)"):
the active gate. Every PR is gated on demand -- a maintainer adds the
`run-benchmark` label to run it. Uses pull_request_target so the run can access
KIRO_API_KEY even for fork PRs; job runs only if the label is `run-benchmark`;
a fork contributor cannot self-label; a later push does not re-run (re-label to
re-review). permissions: contents: read.
- .github/workflows/benchmark.yml: automatic pull_request gating DISABLED (trigger
commented out); kept for workflow_dispatch manual runs and as the re-enable
switch.
- benchmark/README.md updated to describe the label-triggered model and the
reliability behavior (no infra retry; salvage-on-timeout).
Setup on the target repo: add a KIRO_API_KEY Actions secret (service-account key)
and create the `run-benchmark` label.
Squash of the workflow-consolidation change and the doc/security cleanup:
- Drop benchmark.yml; move manual workflow_dispatch into the single
benchmark-external.yml (label-triggered + manual runs).
- README: describe the single-workflow, single-repo layout accurately.
- run_skill_tests.py: rewrite multi-repo comments/--nad-root help for the
single-repo layout; default --nad-root to the repo root; remove dangling
internal-doc pointers.
- Correct label permission wording ("no write access" -> Triage role or above)
and note that contents:read does not shield KIRO_API_KEY from PR code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an opt-in CI gate that scores this repo's own skills/agents by running them
through
kiro-cliin Docker and grading the output against per-scenario rubrics, plusa new
neuron-dev-contribution-validationskill/agent (from the internal repo) that checks a contribution against the NAD contribution model. Upstream currently has no.github/,benchmark/, ortests/.Why
Gives maintainers a reproducible, on-demand way to verify that a contributed
skill/agent actually behaves as intended before merging, without running untrusted code
automatically or exposing secrets on every push.
How it works
Grader (
benchmark/)run_skill_tests.pyloads scenarios fromtests/registry.json(the source of truthfor which scenarios are active/gated), runs each in its own Docker container, and
scores
combined = 0.75·pattern + 0.25·judge.patternis proportionalexpectedStrings/forbiddenStringsmatching;judgeis an LLM verdict against thescenario rubric. A scenario passes at
combined ≥ 0.90; the gate fails below--gate(default 90%).
measuring, not masking), plus salvage-on-timeout (score a completed-but-hung run's
output rather than discarding it).
shell/{docker,common}.sh+environment/Dockerfilebuild the run image(
python:3.11-slim+ kiro-cli).Workflow (
.github/workflows/benchmark-external.yml)run-benchmarklabel to opt it in.Applying a label requires the repo's Triage role or above, so fork contributors (no
role on this repo) can't trigger it themselves. A later push does not re-run it — the
label must be re-applied (re-review).
pull_request_targetso the run can readKIRO_API_KEYeven on fork PRs (aplain
pull_requestrun gets no secrets on forks), withpermissions: contents: readto drop the write-capableGITHUB_TOKEN.workflow_dispatchfrom the Actions tab (write accessrequired), which checks out the selected branch directly.
New skill/agent (
neuron-dev-contribution-validation)NEURON_METADATA.mdfields, agent-spec formats, and minimum test counts, reporting PASS/FAIL per check.
Ships with a Claude Code
.mdagent + Kiro.agent-spec.json, and one active gatedtest.
Repo prerequisites
KIRO_API_KEYActions secret — authenticates both the agent and the LLM judge.run-benchmarklabel.Security note for reviewers / labelers
The labeled job checks out and runs the PR's untrusted contributed code in an
environment that holds
KIRO_API_KEY.contents: readrestricts only theGITHUB_TOKEN— it does not shieldKIRO_API_KEYfrom that code. Applying thelabel is vouching for the diff, so only label PRs whose changes you've reviewed.
Testing
python benchmark/run_skill_tests.py --nad-root=. --subset=all --judge --gate=90labeledevent, not on push/opened.Notes
threshold independent of any code change (documented in
benchmark/README.md).