Add identity_containment category (3 scenarios) from Beyond Zero - #7
Add identity_containment category (3 scenarios) from Beyond Zero#7davidcrowe wants to merge 3 commits into
Conversation
…ttacker scenarios Three scenarios drawn from Alphabet's Beyond Zero paper (Valente & Zalewski, ACM Queue 2026, arXiv:2605.22985), expressed entirely with existing benchmark primitives so every runner works unmodified: - 01: identity-level containment denies ALL tools, not just the trigger - 02: denials under containment are audited with a populated reason - 03: lifting a containment restores access with no sticky residue The paper's rogue-agent scenario (agent action vs controlling human's task intent) is NOT included — it needs a task-attribution field that no runner emits today; tracked separately. Vanilla runner verified: loads and runs, 13/51 (containment scenarios correctly fail with no governance layer). Published framework scorecards keep /48 denominators with a note until the next full re-run.
…ACP 3/3 result user-mallory wasn't in the runner UID_MAP, so the baseline call authenticated as nobody and every scenario false-denied — caught by running the reference runner, which the new CONTRIBUTING checklist now makes a hard requirement (verify against vanilla + one real runner before the PR, commit the results file). Reference ACP runner against production: identity_containment 3/3. The guarantee holds today via user-tier policy enforcement — the containment *primitive* (gatewaystack-connect#541) is about ergonomics and triggering, not about this enforcement path.
|
Update after review questions: ACP result: 3/3 against the production gateway (reference runner; results file committed as Bug found and fixed by running it: the first draft used an invented Inclusion criteria now explicit: CONTRIBUTING gains a 9-point scenario inclusion checklist (layer-not-model, guarantee-not-feature, externally motivated + cited, NIST-mappable, existing primitives first, deterministic/binary, provisioned identities, verified on vanilla + one real runner with committed results, versioned). This category clears all nine — point 8 is the one the first draft would have failed, which is why it's now a hard requirement. |
…Rs; flag issue-first for new categories Two corrections after dogfooding our own contribution path on PR #7: - New categories are supposed to start as an issue (Ways to contribute #4); added an explicit callout so the inclusion checklist points there instead of implying a category can go straight to a PR. Filed #8 as the retroactive threat-model discussion for identity_containment. - Point 8 as first written required every scenario PR to commit a real-runner results file — impossible for an external contributor who doesn't run a governance product, and inconsistent with the existing scenario checklist (only runner PRs need a results JSON). Rescoped: vanilla for everyone, real-runner+results for runner/new-category PRs, maintainer runs the reference runner in review otherwise.
|
Linked to the threat-model discussion issue #8 (opening it retroactively — per our own CONTRIBUTING #4, a new category should start as an issue to agree the threat model + NIST mapping before the YAML; individual scenarios in an existing category can PR directly). Also corrected a self-inflicted inconsistency in the inclusion checklist I added: point 8 originally required every scenario PR to commit a real-runner results file, which an external scenario contributor without a governance runner can't produce (and which the pre-existing scenario checklist doesn't ask for — only runner PRs need a results JSON). Rescoped: |
1 participant
What
New scenario category
identity_containment— 3 scenarios translated from the attacker scenarios in Alphabet's Beyond Zero paper (Valente & Zalewski, ACM Queue 2026), the successor to BeyondCorp. Discussion: HN thread.All three use existing primitives only (
policy_changeuser-tier permission + existing assertion kinds) — no harness, scorer, or runner changes, every runner works unmodified.Deliberately excluded
The paper's rogue-agent scenario (agent action checked against the controlling human's task intent) needs a task-attribution field no runner emits today. Filing separately rather than shipping an untestable scenario.
Verified
agentgovbench run --runner vanillaagainst this branch: loads cleanly, 13/51,identity_containment 0/3(correct — vanilla has no governance layer; scenario 03's lift-restores half passes but its containment half fails).Docs