Skip to content

docs(primaite): inventory evidence-capture requirements - #101

Open
doublewhy wants to merge 4 commits into
devfrom
88-primaite-evidence-capture-inventory
Open

doublewhy wants to merge 4 commits into
devfrom
88-primaite-evidence-capture-inventory

Conversation

@doublewhy

@doublewhy doublewhy commented Oct 9, 2026 •

Copy link
Copy Markdown

Summary

Adds docs/decisions/primaite-evidence-capture-inventory.md, the static inventory and regression-fixture design that the 2026-08-13 hold on #88 permits, plus one Decisions nav line in mkdocs.yml. It follows the format set by #98 (NASim). At dev 0272949 the page records:

  • every SDL and task evidence requirement: blue-action-log, data-integrity-series, service-availability-series, the five task metrics, and the observation requirements, each with file and line. PrimAITE has no packaged example, so the scenario and experiment files are the only copies;
  • each required datum's native source at PrimAITE v4.0.0 (9861798) and its class, citing the existing mapping/source-ledger.jsonl rows and loss disclosures;
  • today's capture chain: the manifest declarations, what the default injected driver yields, and the accepted path that no run reaches;
  • the equivalence data needs against PrimaiteGymEnv, marked present, partly present, or missing;
  • a spec-only regression-fixture design with missing-datum and false-claim negative cases, split into hermetic and native lanes.

Findings worth a look, all recorded as facts without changing anything:

  • No path produces PrimAITE evidence on dev. PrimAITE is not in the researcher CLI registry, and the live driver refuses in-process construction. The participant runtime rejects a BLUE action as unrepresentable, before source mutation, when the driver reports representable=False; the default test FakeDriver does that on every step, and the evaluator withholds the reward. A driver that reports a representable step reaches an accepted path that only unit tests exercise: it records an observation envelope per accepted action and attaches evidence.primaite.blue-action when the observation boundary lists that ref. The ref is not the SDL's blue-action-log.
  • The native source has the data the requirements need. Every agent's per-step action, response, and reward is in info["agent_actions"], and the database file's health_status is in the simulation state. So is each service's operating_state, which gates whether the service can perform actions, beside a separate health signal, health_state_actual; the page records both as R3 candidates for fix(primaite): reconcile SDL evidence requirements, simulator capture, and backend manifest #88. The ledger keeps the info dict and native action ids source-private.
  • The two GREEN penalties that T4 uses only recompute on a GREEN request step and otherwise reuse the last value. They are therefore a lossy per-step availability signal, distinct from both service-state fields.
  • RewardFunction.update keeps only the weighted total, not each component's value. The BLUE total can still be recomputed from the file-health state and the two GREEN agents' rewards, which PrimAITE writes, withheld, into each GREEN history item's reward.

The hold on #88 remains in force. The naming decision, OpenRAE/rae#1023, is still open; the other resume condition, OpenRAE/rae#1112, closed on 2026-09-07.

Stacking

All four inventory PRs add one line to the same mkdocs.yml nav list, so they are stacked. This branch contains #98's commit d89aa60 (NASim, Refs #87), #99's commit ac496e5 (CybORG, Refs #85), and #100's commit cf4a7c8 (CyberBattleSim, Refs #86), followed by its own commit 3d3f0c3; review only 3d3f0c3. Merge #98, #99, and #100 first, and this branch will then be rebased onto dev.

CI note

SonarCloud fails before any analysis. The scanner's JRE metadata request to api.sonarcloud.io returns HTTP 403 Forbidden, and the error says to check SONAR_TOKEN (CI run 37949738253, SonarCloud job). PR Gate then fails, because it requires SonarCloud to succeed on same-repository PRs. The same 403 occurs on dev 0272949 in workflow_dispatch run 37919671796; the last passing SonarCloud job on dev was in run 32215409423 on 2026-08-19. Every other job passes at 3d3f0c3: Fast checks, Policy, Tool tests, Typecheck, Tests, Distributions, Docs, CodeQL, Lint PR title, and GitGuardian.

Requirement UIDs

  • None. Docs-only inventory under a hold; no requirement is implemented or traced.

Related Issues

Refs #88

ADR Impact

  • None. No ADR is added or amended; adr-index.yaml is untouched.

Changes

  • docs/decisions/primaite-evidence-capture-inventory.md (new): the inventory and fixture design. Source references are GitHub links pinned to adapters 0272949, PrimAITE 9861798, and RAES v3.3.0 (fb8a23a), because the Read the Docs site returns 404.
  • mkdocs.yml: one nav entry under Decisions, after the PrimAITE conformance-composition guardrails.
  • No runtime, package, manifest, contract, schema, CLI, ledger, SDL, task, or test change. The packaged mapping/ ledgers are cited, not edited.

Test Plan

All commands ran in the worktree at head 3d3f0c3 on 2026-10-09.

  • uv tool run --from 'nox[uv]==2026.4.10' nox -f noxfile.py -s docs: the strict MkDocs build passed. The rendered page has 6 tables and 100 GitHub links (95 pinned to a commit, plus 5 issue and pull-request links), and no reference-style link is left unresolved.
  • nox -s hygiene, nox -s lint, and nox -s tool-tests (same invocation): passed; tool-tests ran 53 tests.
  • nox -s policy -- --skip-requirement --base-rev origin/dev: repo policy, ADR immutability, project services, and identity policy OK; requirement governance skipped, as CI does for a branch name without a UID.
  • A local link check parsed all 94 reference definitions and their 99 uses: every definition is used and every use is defined. Each of the 88 line anchors lies inside its file at the pinned commit; 85 of those links show line numbers equal to their anchor, and the other 3, the loss-disclosure links, start at the heading of the entry they name.
  • uv build at this head and at origin/dev: the wheels are byte-identical (SHA-256 068871d1866b944cf65a695f5c49c1a6c8705b334aea565a54b19819d341ebc6), and the 193 sdist members are identical in names and bytes.
  • Native facts: the cited PrimAITE files were fetched at 9861798 through the GitHub API. The four listed in qualification.json source_files (environment.py, game.py, probabilistic_agent.py, data_manipulation.yaml) matched their recorded SHA-256. The other six (interface.py, rewards.py, file_system_item_abc.py, software.py, service.py, database_service.py) are not in that list and were read at the commit. No native PrimAITE episode was run.
  • Not run locally: typecheck, tests, and distributions. The PR adds four docs/decisions/*-evidence-capture-inventory.md pages and four mkdocs.yml nav lines. No test or type-checked module reads them, and of the gates only the docs, hygiene, policy, and tool-tests runs above read mkdocs.yml. CI runs all three.

Ground Control Checks

  • Repository policy command passed (see Test Plan).
  • No Ground Control pre-push review was run for this docs-only change.

Traceability

  • IMPLEMENTS: none (docs-only inventory).
  • TESTS: none (no test added; the page specifies future fixtures only).

Checklist

  • FM: not applicable, no semantic change. No executable or runtime adoption is claimed.
  • Changelog: owned by Release Please; no CHANGELOG.md edit.
  • No version, lock, or packaged-resource change.

Documentation

New decision record listed in the MkDocs Decisions nav.

@doublewhy
doublewhy force-pushed the 88-primaite-evidence-capture-inventory branch from e70404e to 2be12b6 Compare October 9, 2026 11:01
@doublewhy
doublewhy force-pushed the 88-primaite-evidence-capture-inventory branch from 2be12b6 to fade521 Compare October 9, 2026 14:43
Add a static inventory of the NASim SDL and task evidence requirements,
their native availability at NetworkAttackSimulator 7c732bc, the capture
chain at dev 0272949, the equivalence data needs, and a spec-only
regression-fixture design, plus its Decisions nav entry.

Docs only, as the 2026-08-13 hold on #87 permits: no contract, schema,
manifest, runtime, package, CLI, ledger, or test change.

Refs #87
@doublewhy
doublewhy force-pushed the 88-primaite-evidence-capture-inventory branch from fade521 to b588bdf Compare October 9, 2026 14:51
Add a static inventory of the CAGE-2 SDL and task evidence requirements,
their native availability at cage-challenge-2 26ce1c1, the capture chain
at dev 0272949 (researcher and reproduce paths), the equivalence data
needs, and a spec-only regression-fixture design, plus its Decisions nav
entry.

Docs only, as the 2026-08-13 hold on #85 permits: no contract, schema,
manifest, runtime, package, CLI, ledger, or test change.

Refs #85
@doublewhy
doublewhy force-pushed the 88-primaite-evidence-capture-inventory branch from b588bdf to 2687951 Compare October 9, 2026 14:53
Add a static inventory of the CyberBattleChain SDL and task evidence
requirements, their native availability at CyberBattleSim 854d696, the
capture chain at dev 0272949 (including the baseline reproduction's
mediated lane), the equivalence data needs, and a spec-only
regression-fixture design, plus its Decisions nav entry.

The RNG inventory covers every generator that the
CredentialCacheExploiter run path seeds or draws from. The
gym-environment and gym-action-space seeds that the driver reports as
applied drive no draw on that path. Every explore draw comes from the
action space's union_np_random, which takes OS entropy at construction,
is reached by no seed call, and is not listed by the spec controls,
ledger rows 24-25, or loss-unbound-random-streams. Per-step reward is
classed withheld under the task's source-private boundary.

Docs only, as the 2026-08-13 hold on #86 permits: no contract, schema,
manifest, runtime, package, CLI, ledger, or test change.

Refs #86
Add a static inventory of the PrimAITE data_manipulation SDL and task
evidence requirements, their native availability at PrimAITE v4.0.0
(9861798), the capture chain at dev 0272949, the equivalence data
needs, and a spec-only regression-fixture design, plus its Decisions
nav entry.

Docs only, as the 2026-08-13 hold on #88 permits: no contract, schema,
manifest, runtime, package, CLI, ledger, or test change.

Refs #88
@doublewhy
doublewhy force-pushed the 88-primaite-evidence-capture-inventory branch from 2687951 to 3d3f0c3 Compare October 9, 2026 15:10
@doublewhy
doublewhy marked this pull request as ready for review October 9, 2026 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant