docs(nasim): inventory evidence-capture requirements

[doublewhy]

Summary

Adds docs/decisions/nasim-evidence-capture-inventory.md, the static inventory and regression-fixture design that the 2026-08-13 hold on #87 permits, plus one Decisions nav line in mkdocs.yml. The page records, at dev 0272949:

• every SDL and task evidence requirement (attacker-action-log, host-compromise-series, the four task metrics, and the observation requirements), with file and line in both the scenario/experiment files and the packaged examples/nasim-tiny copies;
• each required datum's native source at NetworkAttackSimulator 7c732bc and its class (available or withheld), citing the existing mapping/source-ledger.jsonl rows, the loss disclosures, and the authored statements that withhold data;
• three tensions between the authored requirements and that withholding, recorded for the fix(nasim): reconcile SDL evidence requirements, simulator capture, and backend manifest #87 reconciliation and not resolved;
• today's capture chain: two fail-closed gates, the admission gate (exit 3) and the post-run task/run join (exit 6), so a run that got past the first gate would write only the four episode files and no run.json, summary.json, or batch artifact; the single source-ledger:host-compromise-series ref in runtime_plans.py; the manifest declarations; what the evaluator builds and which parts reach those files; and the CLI's existing SHA-256-bound supplemental-member path, which NASim does not use;
• the equivalence data needs, each marked missing, partly present, or not required;
• a spec-only regression-fixture design with missing-datum and false-claim negative cases, split into hermetic and native lanes.

Three findings worth a look:

• Section 2: R1 asks for a portable record of attacker action outcomes, but the per-step success and failure flags (D3) leave env.step only in info and in the observation vector's auxiliary row. Four authored statements withhold info (the participant boundary, the manifest's native_observation constraint, and the evaluator's loss disclosure and capture notes), and ledger row 11 excludes the observation vector, so D3 is classed withheld. Fixtures F1 and F2 state which fix(nasim): reconcile SDL evidence requirements, simulator capture, and backend manifest #87 decision each presupposes.
• Section 3: even past the admission gate, the run cannot be sealed. _complete_native_run calls RAES validate_experiment_run_against_task (cli.py L1633) before it writes run.json. Since fix: remove hardcoded capability and evidence gates #90 the evidence artifact carries no satisfies_refs, so the join rejects attacker-action-log and host-compromise-series, and the command exits 6 with researcher.artifact.failure. This is the second layer that capability-and-evidence-claim-guardrails.md designs.
• Section 4: the packaged red configuration selects only service-exploit and sends no target, so the driver resolves the same flat action on every step (index 4, the exploit on host (1, 0)), while the SDL, ledger rows 16 and 17, and the task describe the attacker as the run_bruteforce_agent baseline, which cycles all 18 flat actions. A probe of that rule with the pinned nasim 0.12.0 wheel on macOS arm64 truncated at step 1000 with cumulative reward −1000 and the goal unreached; every step costs 1 and root is never taken, so the platform does not change that result. Nothing in this PR changes the behavior.

The hold on #87 remains in force. The naming decision, OpenRAE/rae#1023, is still open; the other resume condition, OpenRAE/rae#1112, closed on 2026-09-07.

CI note

SonarCloud fails before any analysis. The scanner's first API call returns HTTP 403 Forbidden and its message says to check SONAR_TOKEN (CI run 38075461320 at ccbf31a, SonarCloud job 114286160625). PR Gate (job 114286216288) then fails only because it requires SonarCloud to succeed on same-repository PRs. The same 403 occurs on dev 0272949 in workflow_dispatch run 37919671796; the last passing SonarCloud job on dev was in run 32215409423 on 2026-08-19. Every other job passes: Fast checks, Policy, Tool tests, Typecheck, Tests, Distributions, Docs, CodeQL, Lint PR title, and GitGuardian. Tracked in #102.

Requirement UIDs

• None. Docs-only inventory under a hold; no requirement is implemented or traced.

Related Issues

Refs #87

ADR Impact

• None. No ADR is added or amended; adr-index.yaml is untouched.

Changes

• docs/decisions/nasim-evidence-capture-inventory.md (new): the inventory and fixture design. Source references are GitHub links pinned to adapters 0272949, NetworkAttackSimulator 7c732bc, and RAES v3.3.0 (fb8a23a), because the Read the Docs site returns 404.
• mkdocs.yml: one nav entry under Decisions, after the NASim conformance-composition guardrails.
• No runtime, package, manifest, contract, schema, CLI, ledger, SDL, task, or test change. The packaged mapping/ ledgers are cited, not edited.

Test Plan

All commands ran in the worktree at head ccbf31a on 2026-10-10, on macOS arm64 with CPython 3.12.13.

• uv tool run --from 'nox[uv]==2026.4.10' nox -f noxfile.py -s docs: the strict MkDocs build passed. The rendered page has 6 tables, 157 GitHub links pinned to a commit, and no unresolved reference-style link.
• nox -s hygiene (same invocation): passed over 685 tracked files.
• nox -s lint: ruff format --check reported 138 files already formatted; ruff check passed.
• nox -s policy -- --skip-requirement, the form CI uses for a branch name without a UID, and again with --base-rev origin/dev: repo policy, ADR immutability, project services, and identity policy OK; requirement governance skipped. The identity policy covers the retired-token and malformed-rename scans of the new page.
• nox -s tool-tests: 53 tests passed.
• A local link check parsed all 151 reference definitions: each one is used, each of the 139 line anchors lies inside its file at the pinned commit (adapters and RAES files read from git, NetworkAttackSimulator files from the GitHub contents API), and each link's visible line numbers equal its anchor. On the previous head's page the same check counts 127 definitions and 131 pinned links, as reported before.
• Post-run join (section 3): a scratch script outside the worktree drove cli.main for run --backend nasim-tiny through the test suite's native-source seams and a FakeNasimDriver, bypassing only _task_capture_admission_gaps, whose real return was attacker-action-log and host-compromise-series. Smoke and study both exited 6 with researcher.artifact.failure: portable evidence could not be sealed. The join raised run evidence_artifacts must satisfy task observation requirements: attacker-action-log, host-compromise-series, and only evidence-records.json, derived-measures.json, diagnostics.json, and participant-provenance.json were left under runs/nasim-research-example-1/. Controls: with the run join made a no-op, smoke exited 0 and wrote every artifact, while study exited 6 at the study join; with the pre-fix: remove hardcoded capability and evidence gates #90 satisfies_refs restored, both exited 0. No written file contained numpy-global-action-success or gym-environment-reset; nasim-gym-reset-seed appeared only in run.json, and diagnostics.json held two nasim.seed.applied entries with no stream id.
• uv build at this head and at origin/dev: the wheels are byte-identical (SHA-256 068871d1866b944cf65a695f5c49c1a6c8705b334aea565a54b19819d341ebc6) and the 193 sdist members are identical in names and bytes, so the new page and mkdocs.yml are in neither distribution.
• Native facts: the SHA-256 that PyPI publishes for nasim-0.12.0-py3-none-any.whl equals the one in qualification.json and uv.lock (4c059e64…). The eight source_files entries under nasim/ matched their digests in the installed wheel, and the eight cited NetworkAttackSimulator files (action.py, environment.py, network.py, host_vector.py, state.py, observation.py, tiny.yaml, bruteforce_agent.py) matched the blobs at 7c732bc. The four cited RAES files at tag v3.3.0 match the installed raes 3.3.0 wheel from PyPI.
• The section 4 probe ran with nasim==0.12.0, gymnasium==0.26.3, and numpy==1.26.4: index 4 resolved to e_ssh on host (1, 0), which grants user access; every step returned reward −1, and the episode truncated at step 1000 with cumulative reward −1000.0 and terminated false. On this host raes-adapters inspect --backend nasim-tiny reports native_available false, because only the NumPy root tree fails its qualification.json digest.
• Not run locally: typecheck, tests, and distributions. The diff touches only the new page and mkdocs.yml, and none of the three reads the new page or mkdocs.yml; CI runs all three.

Ground Control Checks

• Repository policy command passed (see Test Plan).
• No Ground Control pre-push review was run for this docs-only change.

Traceability

• IMPLEMENTS: none (docs-only inventory).
• TESTS: none (no test added; the page specifies future fixtures only).

Checklist

• FM: not applicable, no semantic change. No executable or runtime adoption is claimed.
• Changelog: owned by Release Please; no CHANGELOG.md edit.
• No version, lock, or packaged-resource change.

Documentation

New decision record listed in the MkDocs Decisions nav.