Skip to content

docs(cyborg): inventory evidence-capture requirements - #99

Open
doublewhy wants to merge 2 commits into
devfrom
85-cyborg-evidence-capture-inventory
Open

doublewhy wants to merge 2 commits into
devfrom
85-cyborg-evidence-capture-inventory

Conversation

@doublewhy

@doublewhy doublewhy commented Oct 9, 2026 •

Copy link
Copy Markdown

Summary

Adds docs/decisions/cyborg-evidence-capture-inventory.md, the static inventory and regression-fixture design that the 2026-08-13 hold on #85 permits, plus one Decisions nav line in mkdocs.yml. It follows the format set by #98 (NASim). At dev 0272949 the page records:

  • every SDL and task evidence requirement: operational-service-state, the operational-service-available proposition that cites it, and source-ledger:reward-components as both the task metric's evidence and its observation requirement. Each has file and line in the scenario SDL and the packaged examples/cage2-research copies;
  • each required datum's native source at cage-challenge-2 26ce1c1 and its class, citing the existing mapping/cage2-source-ledger.jsonl rows and loss disclosures;
  • today's capture chain for both the researcher commands and the reproduce command, with the manifest declarations and the emitted files and fields;
  • the equivalence data needs against the public CAGE-2 evaluator, marked present, partly present, or missing;
  • a spec-only regression-fixture design with missing-datum and false-claim negative cases, split into hermetic and native lanes. The native fixture extends the native seam that CI already runs: the adapter smoke in tools/verify_cyborg_qualification.py.

Findings worth a look, all recorded as facts without changing anything:

  • The reproduce --phase run path does not go through the CLI evidence gate. Its run artifact sets satisfies_refs to operational-service-state and source-ledger:reward-components (reproduction.py L1790–1804) before validate_experiment_run_against_task() runs. The researcher validate and run commands exit 3 for the same task.
  • In the pinned sim, EnvironmentController.determine_done returns False and SimulationController does not override it. The source-terminal cause therefore cannot occur, and every episode ends at the logical-step limit.
  • R1 asks for operational-service state. The datum the adapter can carry is the Blue availability penalty, which is derived from the OTService process state that ledger row 19 keeps hidden.
  • No ledger row names the Blue action cost. The driver tags the Blue action-cost component source-ledger:reward-objectives (driver.py L236–247), which is ledger row 45, whose selector is Red's HybridImpactPwnRewardCalculator.calculate_reward. The evaluator copies that tag into each action-cost record (evaluator.py L763–771), so after a costed Blue action such as Restore the cumulative action-cost measure cites row 45 records. The CI qualification smoke asserts the tag (tools/verify_cyborg_qualification.py L300–306). The packaged Blue policy selects Sleep, which costs 0, so packaged runs emit no action-cost measure.

The hold on #85 remains in force. The naming decision, OpenRAE/rae#1023, is still open; the other resume condition, OpenRAE/rae#1112, closed on 2026-09-07.

Stacking

All four inventory PRs add one line to the same mkdocs.yml nav list, so they are stacked. This branch contains #98's commit d89aa60 (NASim, Refs #87) followed by its own commit ac496e5; review only ac496e5. Merge #98 first, and this branch will then be rebased onto dev.

CI note

SonarCloud fails before any analysis. The scanner's first API call returns HTTP 403 Forbidden and its message says to check SONAR_TOKEN (CI run 37947519563, SonarCloud job). PR Gate then fails, because it requires SonarCloud to succeed on same-repository PRs. The same 403 occurs on dev 0272949 in workflow_dispatch run 37919671796; the last passing SonarCloud job on dev was in run 32215409423 on 2026-08-19. The token failure is tracked in #102. Every other job passes: Fast checks, Policy, Tool tests, Typecheck, Tests, Distributions, Docs, CodeQL, Lint PR title, and GitGuardian.

Requirement UIDs

  • None. Docs-only inventory under a hold; no requirement is implemented or traced.

Related Issues

Refs #85

ADR Impact

  • None. No ADR is added or amended; adr-index.yaml is untouched.

Changes

  • docs/decisions/cyborg-evidence-capture-inventory.md (new): the inventory and fixture design. Source references are GitHub links pinned to adapters 0272949, cage-challenge-2 26ce1c1, and RAES v3.3.0 (fb8a23a), because the Read the Docs site returns 404.
  • mkdocs.yml: one nav entry under Decisions, after the CybORG/CAGE-2 downstream environment-pack guardrails.
  • No runtime, package, manifest, contract, schema, CLI, ledger, SDL, task, or test change. The packaged mapping/ ledgers are cited, not edited.
  • Review revision: the own commit, now ac496e5, replaces fbbea80 and changes only the page. It was pushed as 3e264fd, and docs(nasim): inventory evidence-capture requirements #98's revision then restacked it onto d89aa60 unchanged (git range-diff reports 3e264fd = ac496e5). The native lane now describes the CI qualification seam and F5 extends it. Section 3, D4, the tensions paragraph, and F2 now record the row 45 action-cost tag. The component-measure, satisfies_refs, and RAES-release sentences are now exact. The revision adds 10 pinned references and re-anchors one.

Test Plan

All commands ran in the worktree at head ac496e5 on 2026-10-09.

  • uv tool run --from 'nox[uv]==2026.4.10' nox -f noxfile.py -s docs: the strict MkDocs build passed. The page's rendered content has 6 tables and 112 GitHub links, 107 of them pinned to commits. The other 5 link to issues or pull requests: fix(cyborg): reconcile SDL evidence requirements, simulator capture, and backend manifest #85, runtime: reject backends whose capture manifests do not satisfy scenario requirements rae#1112, chore(naming): rename RAES to RAE across repository surfaces rae#1023, and feat(runtime): enforce capture admission and evidence proof rae#1239 twice. The page has no unresolved reference-style link. Its relative links resolve to the NASim inventory, the claim guardrails, and four CybORG decision records.
  • nox -s hygiene (same invocation): passed.
  • nox -s lint: passed.
  • nox -s policy -- --skip-requirement --base-rev origin/dev: repo policy, ADR immutability, project services, and identity policy OK; requirement governance skipped, as CI does for a branch name without a UID.
  • nox -s tool-tests: passed (53 tests).
  • A local link check parsed all 106 reference definitions. Each one is used, each of the 95 line anchors lies inside its file at the pinned commit, and each link's visible line or row numbers equal its anchor.
  • uv build at this head and at origin/dev: the wheels are byte-identical (SHA-256 068871d1866b944cf65a695f5c49c1a6c8705b334aea565a54b19819d341ebc6) and the 193 sdist members are identical in names and bytes.
  • Native facts: the cited cage-challenge-2 files were fetched at 26ce1c1 through the GitHub API; those listed in qualification.json selected_files matched their recorded SHA-256. CybORG.py, Simulator/SimulationController.py, and Shared/Actions/Action.py are not in that list and were read at the commit. No native CybORG episode was run locally.
  • CI native seam: noxfile.py L487–488 runs tools/verify_cyborg_qualification.py in the cyborg test environment. The Tests job of CI run 37947519563 logs CybORG qualification: OK.
  • RAES releases: in OpenRAE/rae, git tag --contains 68d25df (the #1239 merge commit) lists v4.0.0, v4.1.0, v5.0.0, v6.0.0, and v6.0.1. PyPI has raes 4.1.0 (2026-09-13), 5.0.0, and 6.0.1 but no 4.0.0 or 6.0.0, so 4.1.0 is the first published release that includes it.
  • Not run locally: typecheck, tests, and distributions. The diff touches only this page and mkdocs.yml; no type check, test, or distribution step reads either file, and the built wheel and sdist are unchanged. CI runs all three.

Ground Control Checks

  • Repository policy command passed (see Test Plan).
  • No Ground Control pre-push review was run for this docs-only change.

Traceability

  • IMPLEMENTS: none (docs-only inventory).
  • TESTS: none (no test added; the page specifies future fixtures only).

Checklist

  • FM: not applicable, no semantic change. No executable or runtime adoption is claimed.
  • Changelog: owned by Release Please; no CHANGELOG.md edit.
  • No version, lock, or packaged-resource change.

Documentation

New decision record listed in the MkDocs Decisions nav.

Add a static inventory of the NASim SDL and task evidence requirements,
their native availability at NetworkAttackSimulator 7c732bc, the capture
chain at dev 0272949, the equivalence data needs, and a spec-only
regression-fixture design, plus its Decisions nav entry.

Docs only, as the 2026-08-13 hold on #87 permits: no contract, schema,
manifest, runtime, package, CLI, ledger, or test change.

Refs #87
@doublewhy
doublewhy force-pushed the 85-cyborg-evidence-capture-inventory branch from fbbea80 to 3e264fd Compare October 9, 2026 14:45
Add a static inventory of the CAGE-2 SDL and task evidence requirements,
their native availability at cage-challenge-2 26ce1c1, the capture chain
at dev 0272949 (researcher and reproduce paths), the equivalence data
needs, and a spec-only regression-fixture design, plus its Decisions nav
entry.

Docs only, as the 2026-08-13 hold on #85 permits: no contract, schema,
manifest, runtime, package, CLI, ledger, or test change.

Refs #85
@doublewhy
doublewhy force-pushed the 85-cyborg-evidence-capture-inventory branch from 3e264fd to ac496e5 Compare October 9, 2026 14:53
@doublewhy
doublewhy marked this pull request as ready for review October 9, 2026 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant