Skip to content

Add benchmark workflow#7

Open
Joshdb-Amazon wants to merge 3 commits into
aws-neuron:mainfrom
Joshdb-Amazon:main
Open

Add benchmark workflow#7
Joshdb-Amazon wants to merge 3 commits into
aws-neuron:mainfrom
Joshdb-Amazon:main

Conversation

@Joshdb-Amazon

Copy link
Copy Markdown

What

Adds an opt-in CI gate that scores this repo's own skills/agents by running them
through kiro-cli in Docker and grading the output against per-scenario rubrics, plus
a new neuron-dev-contribution-validation skill/agent (from the internal repo) that checks a contribution against the NAD contribution model. Upstream currently has no .github/, benchmark/, or tests/.

Why

Gives maintainers a reproducible, on-demand way to verify that a contributed
skill/agent actually behaves as intended before merging, without running untrusted code
automatically or exposing secrets on every push.

How it works

Grader (benchmark/)

  • run_skill_tests.py loads scenarios from tests/registry.json (the source of truth
    for which scenarios are active/gated), runs each in its own Docker container, and
    scores combined = 0.75·pattern + 0.25·judge. pattern is proportional
    expectedStrings/forbiddenStrings matching; judge is an LLM verdict against the
    scenario rubric. A scenario passes at combined ≥ 0.90; the gate fails below --gate
    (default 90%).
  • Reliability: no infra-error retry by design (a hung agent is skill behavior worth
    measuring, not masking), plus salvage-on-timeout (score a completed-but-hung run's
    output rather than discarding it).
  • shell/{docker,common}.sh + environment/Dockerfile build the run image
    (python:3.11-slim + kiro-cli).

Workflow (.github/workflows/benchmark-external.yml)

  • A maintainer reviews a PR and applies the run-benchmark label to opt it in.
    Applying a label requires the repo's Triage role or above, so fork contributors (no
    role on this repo) can't trigger it themselves. A later push does not re-run it — the
    label must be re-applied (re-review).
  • Uses pull_request_target so the run can read KIRO_API_KEY even on fork PRs (a
    plain pull_request run gets no secrets on forks), with permissions: contents: read to drop the write-capable GITHUB_TOKEN.
  • Also supports manual workflow_dispatch from the Actions tab (write access
    required), which checks out the selected branch directly.

New skill/agent (neuron-dev-contribution-validation)

  • Validates a contribution's namespace, directory structure, NEURON_METADATA.md
    fields, agent-spec formats, and minimum test counts, reporting PASS/FAIL per check.
    Ships with a Claude Code .md agent + Kiro .agent-spec.json, and one active gated
    test.

Repo prerequisites

  • A KIRO_API_KEY Actions secret — authenticates both the agent and the LLM judge.
  • A run-benchmark label.

Security note for reviewers / labelers

The labeled job checks out and runs the PR's untrusted contributed code in an
environment that holds KIRO_API_KEY. contents: read restricts only the
GITHUB_TOKEN — it does not shield KIRO_API_KEY from that code. Applying the
label is vouching for the diff, so only label PRs whose changes you've reviewed.

Testing

python benchmark/run_skill_tests.py --nad-root=. --subset=all --judge --gate=90

  • Confirmed the gate runs only on the labeled event, not on push/opened.

Notes

  • The LLM judge is non-deterministic, so the pass rate can vary run-to-run near the
    threshold independent of any code change (documented in benchmark/README.md).

A CI gate that scores this repo's agent skills by running them through kiro-cli
in Docker and grading against per-scenario rubrics. A PR touching skills/,
agents/, tests/, or the benchmark/ grader is scored against the repo's own
assets. See benchmark/README.md.

Inputs (repo root):
- tests/ + tests/registry.json — 34 active scenarios (32 nki + 1 autoport +
  1 contribution-validation); registry.json is the source of truth for active ones
- skills/neuron-dev-contribution-validation/ + its agent spec (needed by the
  contribution-validation scenario; existing skills left untouched)

Grader (benchmark/):
- run_skill_tests.py — load/run/score/gate (stdlib only)
- shell/{docker,common}.sh — build image + run kiro-cli in Docker (kiro-only;
  no upstream Claude/Node/LangSmith paths)
- environment/Dockerfile — the run image (python:3.11-slim + kiro-cli)
- README.md

Workflow (.github/workflows/benchmark.yml):
- self-contained: checkout -> setup-python -> run benchmark/run_skill_tests.py
  --nad-root=. -> upload results. Scores the repo's own assets; no cross-repo
  refs or tokens.
- pull_request (paths: skills/agents/tests/benchmark/workflow) + workflow_dispatch.
- Requires a KIRO_API_KEY Actions secret (agent + judge auth).

Scoring: combined = 0.75*pattern + 0.25*judge; passed = combined >= 0.90;
gate fails below --gate (default 90%).

Reliability:
- salvage-on-timeout: if the agent times out but already wrote a non-empty
  output.py (it often completes the task then hangs on a trivial final step),
  that output is scored instead of discarded.
- NO infra-error retry: a failure (incl. a no-output timeout from the agent
  going down a bad path) is reported as-is, so the benchmark measures skill
  quality honestly. Deliberate multi-attempt runs use --count (transparent Pass@k).
- Infra errors are logged (type + container tail) for human legibility.
Adds the label-triggered gate and the surrounding finalization on top of the
single-repo benchmark gate.

- .github/workflows/benchmark-external.yml ("NAD Benchmark (label-triggered)"):
  the active gate. Every PR is gated on demand -- a maintainer adds the
  `run-benchmark` label to run it. Uses pull_request_target so the run can access
  KIRO_API_KEY even for fork PRs; job runs only if the label is `run-benchmark`;
  a fork contributor cannot self-label; a later push does not re-run (re-label to
  re-review). permissions: contents: read.
- .github/workflows/benchmark.yml: automatic pull_request gating DISABLED (trigger
  commented out); kept for workflow_dispatch manual runs and as the re-enable
  switch.
- benchmark/README.md updated to describe the label-triggered model and the
  reliability behavior (no infra retry; salvage-on-timeout).

Setup on the target repo: add a KIRO_API_KEY Actions secret (service-account key)
and create the `run-benchmark` label.
Squash of the workflow-consolidation change and the doc/security cleanup:

- Drop benchmark.yml; move manual workflow_dispatch into the single
  benchmark-external.yml (label-triggered + manual runs).
- README: describe the single-workflow, single-repo layout accurately.
- run_skill_tests.py: rewrite multi-repo comments/--nad-root help for the
  single-repo layout; default --nad-root to the repo root; remove dangling
  internal-doc pointers.
- Correct label permission wording ("no write access" -> Triage role or above)
  and note that contents:read does not shield KIRO_API_KEY from PR code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant