Add benchmark workflow

.github/workflows/benchmark-external.yml

@@ -0,0 +1,111 @@
+name: NAD Benchmark (label-triggered)
+
+# The active NAD benchmark gate. Policy: EVERY PR (internal or external fork) is
+# gated on demand — a maintainer reviews the PR and adds the `run-benchmark`
+# label to run it. This is the only NAD benchmark workflow.
+#
+# Maintainers can also run it MANUALLY from the Actions tab (Run workflow /
+# workflow_dispatch) against any branch, with optional subset/count/gate inputs.
+# Manual runs are trusted (only maintainers can trigger them), so they check out
+# the selected branch directly instead of an untrusted PR head.
+#
+# It uses pull_request_target so the run has access to KIRO_API_KEY even for fork
+# PRs (a plain pull_request gate can't — fork PRs receive no secrets), and is
+# guarded so it runs ONLY when a maintainer explicitly opts a PR in.
+#
+# How it's controlled:
+#  - Triggers ONLY on the `labeled` event, and the job runs ONLY if the label is
+#    `run-benchmark`. Applying a label requires the repo's Triage role or above;
+#    a fork contributor has no role on this repo, so nothing runs until a
+#    maintainer (or a member explicitly granted Triage+) reviews the PR and
+#    applies the label.
+#  - A later push does NOT re-run it — a maintainer must re-apply the label, i.e.
+#    re-review, before new commits are scored.
+#  - permissions: contents: read (drops the default write-capable GITHUB_TOKEN).
+#
+# The person who applies the label is vouching for the PR: this job checks out and
+# runs the PR's contributed skill in a job that holds KIRO_API_KEY, so only label
+# PRs whose changes you've looked at. NOTE: `contents: read` restricts only the
+# GITHUB_TOKEN — it does NOT shield KIRO_API_KEY, which is fully exposed to the
+# untrusted PR code this job runs. Review the diff before labeling. Use a
+# service-account key (not a personal one) so it can be rotated independently.
+
+on:
+  pull_request_target:
+    # ONLY the label event — never `synchronize`/`opened`, so untrusted code
+    # never auto-runs on push. A maintainer must (re-)apply the label each time.
+    types: [labeled]
+
+  # Manual runs from the Actions tab. Only users with write access can dispatch
+  # this, so the run is trusted — it scores the selected branch's own code.
+  workflow_dispatch:
+    inputs:
+      subset:
+        description: "Subset(s): 'all', a single name, or a comma-separated list"
+        default: 'all'
+      count:
+        description: 'Pass@k attempts per scenario'
+        default: '1'
+      gate:
+        description: 'Required pass rate (0-100); job fails below this'
+        default: '90'
+
+# Least privilege: the base-context GITHUB_TOKEN is write-capable by default under
+# pull_request_target. Restrict it to read so untrusted code can't use it to push,
+# open releases, or modify the repo.
+permissions:
+  contents: read
+
+concurrency:
+  group: nad-benchmark-external-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  gate:
+    # Run on a manual dispatch, OR when a maintainer added the `run-benchmark`
+    # label. For the label path this `if` is the trust gate — without it,
+    # pull_request_target would run untrusted code with secrets on every PR event.
+    if: github.event_name == 'workflow_dispatch' || github.event.label.name == 'run-benchmark'
+    name: NAD external benchmark (labeled)
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+
+    steps:
+      # Label runs: check out the PR's HEAD — i.e. the contributor's UNTRUSTED
+      # code. This is required (we must score their skill) and is the crux of the
+      # risk above. The workflow FILE itself still comes from base
+      # (pull_request_target), so the contributor cannot alter these steps — only
+      # the skill/agent/test/grader content that the steps then run.
+      # Manual runs: head.sha is empty, so checkout falls back to the selected
+      # branch (trusted — only maintainers can dispatch).
+      - name: Checkout PR head (untrusted on label runs)
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+
+      - name: Run NAD benchmark
+        env:
+          KIRO_API_KEY: ${{ secrets.KIRO_API_KEY }}
+        run: |
+          set -euo pipefail
+          # Inputs come from the manual dispatch form; label runs use the defaults.
+          python benchmark/run_skill_tests.py \
+            --nad-root=. \
+            --subset="${{ github.event.inputs.subset || 'all' }}" \
+            --count="${{ github.event.inputs.count || '1' }}" \
+            --workers=4 \
+            --judge \
+            --gate="${{ github.event.inputs.gate || '90' }}"
+
+      - name: Upload results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: nad-benchmark-results
+          path: benchmark/build/nad_benchmark_results.json
+          if-no-files-found: warn

.gitignore

@@ -0,0 +1,5 @@
+
+# NAD benchmark artifacts
+benchmark/build/
+__pycache__/
+*.pyc

agents/neuron-dev-contribution-validation-agent.agent-spec.json

@@ -0,0 +1,20 @@
+{
+  "schemaVersion": "1",
+  "name": "neuron-dev-contribution-validation-agent",
+  "config": {
+    "description": "Validates agentic artifact contributions to the NeuronAgenticDevelopment (NAD) package against the contribution model requirements. Use when reviewing CRs or checking contribution readiness.",
+    "systemPrompt": "{{aim:include:agents/neuron-dev-contribution-validation-agent.md}}",
+    "model": "claude-sonnet-4.6"
+  },
+  "dependencies": {
+    "skills": {
+      "skillNames": ["neuron-dev-contribution-validation"]
+    }
+  },
+  "clientConfig": {
+    "kiroCli": {
+      "tools": ["@builtin"],
+      "allowedTools": ["fs_read", "fs_write", "grep", "glob", "execute_bash", "todo_list"]
+    }
+  }
+}

agents/neuron-dev-contribution-validation-agent.md

@@ -0,0 +1,42 @@
+---
+name: neuron-dev-contribution-validation-agent
+description: |
+  Validates agentic artifact contributions to the NeuronAgenticDevelopment (NAD) package
+  against the contribution model requirements. Use when reviewing CRs, checking contribution
+  readiness, or validating that skills/agents meet NAD standards.
+
+  <example>
+  Context: User wants to validate their NAD contribution before submitting a CR
+  user: "Validate my contribution to NAD"
+  assistant: "I'll use neuron-dev-contribution-validation-agent to check your artifacts against the contribution model."
+  </example>
+
+  <example>
+  Context: User is reviewing a CR for NAD
+  user: "Review this CR for NAD contribution compliance"
+  assistant: "Let me use neuron-dev-contribution-validation-agent to validate the contribution."
+  </example>
+
+model: sonnet
+tools: ["Read", "Grep", "Glob", "Bash", "Task", "TodoWrite", "Skill"]
+skills:
+  - neuron-dev-contribution-validation
+---
+
+You are a NAD contribution validation agent. Your job is to validate agentic artifact
+contributions (skills, agents, hooks, tools) to the NeuronAgenticDevelopment package
+against the contribution model requirements.
+
+When asked to validate a contribution:
+
+1. Use Glob and Read to discover the artifacts being contributed
+2. Check each artifact against the validation checklist from the neuron-dev-contribution-validation skill
+3. Report results in the structured format defined by the skill
+
+Always check:
+- Namespace compliance (valid prefix from the approved list)
+- Required files exist (SKILL.md, NEURON_METADATA.md)
+- NEURON_METADATA.md has all required fields
+- Agent specs exist in both Kiro and Claude Code formats (if contributing an agent)
+- AIM benchmark tests exist and are registered
+- Minimum test counts are met for the release category

benchmark/README.md

@@ -0,0 +1,94 @@
+# NAD Skill Benchmark Gate
+
+A CI gate that evaluates this repo's agent skills by running them through the
+`kiro-cli` agent in Docker and scoring the output against per-scenario rubrics.
+A PR that changes `skills/`, `agents/`, `tests/`, or this `benchmark/` grader is
+scored against the repo's own assets.
+
+## Layout (single-repo)
+
+Everything the gate needs lives in this repo:
+
+- **Inputs** (at the repo root, scored by the gate):
+  - `skills/` — the skills under test
+  - `agents/` — agent specs (`*.agent-spec.json`) listing each agent's `skillNames`
+  - `tests/` — scenario definitions per subset + `tests/registry.json` (the source
+    of truth for which scenarios are *active/gated*)
+- **Grader** (this `benchmark/` directory):
+  - `run_skill_tests.py` — loads scenarios, runs the agent in Docker, scores, gates
+  - `shell/{docker,common}.sh` — build the image + run `kiro-cli` in Docker
+  - `environment/Dockerfile` — the run image (`python:3.11-slim` + kiro-cli)
+
+## How it runs
+
+The gate scores the repo's own `skills/` + `agents/` + `tests/` by running
+`benchmark/run_skill_tests.py --nad-root=.`. No cross-repo references or tokens.
+
+**Every PR is gated on demand via a label** (`.github/workflows/benchmark-external.yml`):
+
+- A maintainer reviews a PR and adds the **`run-benchmark`** label; that triggers
+  the gate on the PR. Applying a label requires the repo's Triage role or above, so
+  a fork contributor (no role on this repo) can't opt themselves in — nothing runs
+  until a maintainer does. Pushing new commits does not re-run it — the maintainer
+  must re-apply the label (re-review).
+- It uses `pull_request_target` so the run can access the `KIRO_API_KEY` secret
+  even for PRs from external forks (a plain `pull_request` run gets no secrets on
+  forks). The person who labels a PR is vouching for it: the job runs the PR's
+  contributed skill code with the key present, so only label PRs you've reviewed.
+  Note that `contents: read` restricts only the `GITHUB_TOKEN` — it does **not**
+  shield `KIRO_API_KEY`, which is exposed to the untrusted PR code, so review the
+  diff before labeling.
+- **Requires** on the repo: a `KIRO_API_KEY` Actions secret (authenticates both the
+  agent and the LLM judge; use a service-account key) and a `run-benchmark` label.
+
+The same workflow also supports **manual** `workflow_dispatch` runs from the Actions
+tab (inputs: `subset`, `count`, `gate`). Manual dispatch is restricted to users with
+write access, so it checks out the selected branch directly (trusted) rather than an
+untrusted PR head.
+
+## Scoring
+
+Per scenario:
+
+- `pattern_score` (75%) — proportional match of `expectedStrings` (present) and
+  `forbiddenStrings` (absent), matched against the output file + the agent's response.
+- `judge_score` (25%) — an LLM judge (`kiro-cli`) verdict against the scenario's
+  rubric: `1.0` pass / `0.0` fail (or `null` if no rubric).
+- `combined_score = 0.75*pattern + 0.25*judge` (pattern-only if the judge is off).
+- `passed = combined_score >= 0.90` (per-scenario override via `metadata.baseline_k`).
+
+The gate fails if the overall pass rate is below `--gate` (default 90%).
+
+> **Note on the judge:** the LLM judge is non-deterministic, so a scenario with a
+> perfect `pattern_score=1.00` can still score `judge=0` on an unlucky verdict,
+> pulling `combined` to 0.75. The overall pass rate can therefore vary run-to-run
+> near the threshold independent of any code change.
+
+## Reliability handling
+
+- **No infra-error retry.** A failure is reported as-is — including a timeout that
+  produced no output, which typically reflects the *agent* going down a bad path
+  (e.g. fetching a dead URL and stalling). This is skill behavior the benchmark
+  should measure, not hide by retrying until it happens to pass. For deliberate
+  multi-attempt runs use `--count` (transparent Pass@k).
+- **Salvage-on-timeout:** if the agent times out but has already written a non-empty
+  `output.py`, that output is scored instead of being discarded (the agent often
+  finishes the task and then hangs on a trivial final step). This scores what the
+  agent actually produced — it does not grant a fresh attempt. A timeout that
+  produced no output is reported as an infrastructure error (`run_status:
+  infra_error`), logged with its exception type + container-output tail.
+
+## Running it
+
+```bash
+# On a PR (primary path): a maintainer adds the `run-benchmark` label to the PR.
+
+# Manually via CLI (workflow_dispatch on benchmark-external.yml):
+gh workflow run benchmark-external.yml -f subset=all -f count=1 -f gate=90
+
+# Locally (needs Docker + KIRO_API_KEY):
+python benchmark/run_skill_tests.py --nad-root=. --subset=all --count=1 --judge --gate=90
+```
+
+`--subset` accepts `all`, a single subset name (e.g. `neuron-nki-agent`), or a
+comma-separated list. Results are written to `benchmark/build/nad_benchmark_results.json`.

benchmark/environment/Dockerfile

@@ -0,0 +1,13 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+# Install system deps for kiro-cli
+RUN apt-get update && apt-get install -y curl unzip && rm -rf /var/lib/apt/lists/*
+
+# Install kiro-cli
+RUN curl -fsSL https://cli.kiro.dev/install | bash
+
+ENV PATH="/root/.local/bin:$PATH"
+
+COPY . .