Implement Reachability against the schema/ design contracts#1
Open
contantlab wants to merge 23 commits into
Open
Implement Reachability against the schema/ design contracts#1contantlab wants to merge 23 commits into
contantlab wants to merge 23 commits into
Conversation
First working vertical slice, infra-free and fully unit-tested: a questionnaire YAML becomes a context graph, and the reachability pass derives the downgrade facts the scoring policy will consume. Proves the graph + reasoning core end to end before any DB or parser exists (per 05-context-ingestion.md build order). Functional now (no longer stubs): - models/conditions.py: parse_condition() for the namespace:value vocabulary. - graphstore/memory.py: InMemoryGraphStore implementing the GraphStore Protocol, with paths_to() mirroring the inbound-reachability CTE (routes_to/connects_to, per-path cycle guard, depth bound). The infra-free default for tests/small envs; PostgresGraphStore remains the production default behind the same Protocol. - ingestion/questionnaire.py: load()/build() -> declared-confidence nodes/edges (assets + synthesized public entrypoints, controls + protected_by, crown-jewel data_sensitivity, zones + connects_to), with intra-document identity merge. - engine/reachability.py: compute() -> reachable_from_internet, controls on all/any inbound paths (the all-paths set is the downgrade gate), zones traversed. Tests (17 new passing; chaining/policy/grype remain skipped for later slices): - two-path graph proves a control on ALL paths is downgradeable while a control on only SOME paths is not (the core soundness property of the downgrade engine); - end-to-end over the shipped example questionnaire: web-frontend is reachable and behind the WAF; the crown-jewel customer-db is internal-only. Contract note: added GraphStore.get_node() to the Protocol — a point accessor the engine needs that schema/02's illustrative Protocol omitted. Non-breaking addition (no existing method changed); implemented in InMemoryGraphStore, stubbed in Postgres. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Second slice: a scanner becomes normalized findings with zero scanner-specific core code, exercising the "swap any scanner" contract end to end on the SCA capability. normalize() is tested against recorded Grype output, no live binary. Functional now (no longer stubs): - adapters/yaml_adapter.py: full YamlAdapter interpreting an adapter.yaml field map. Self-contained expression language (no external jsonpath dep) covering the grammar in 03-adapter-interface.md: dotted paths ($.a.b), array wildcard/index ([*]/[n]) for the iterator, pipe transforms ($.x | lower), concatenation with 'single-quoted' literals ($.a + '@' + $.b), and @const:. Dotted target keys assign nested; list-typed fields (cve, category.cwe, ...) coerce scalars to a 1-element list. Preserves the native item under 'raw'; finding_id = uuid5 over (name, fingerprint) so it is stable and deterministic (project principle). - adapters/registry.py: auto-discovery of adapters/ (adapter.yaml -> YamlAdapter, adapter.py -> its ADAPTER class), records healthcheck liveness without failing discovery, binds capability -> name via resolve_kind (the swap point). - models/finding.py: Finding.from_dict/to_dict (tolerant: adapters fill what they can, engine enriches the rest) and Severity.coerce mapping scanner-native words (Negligible/Unknown/Important/...) onto the info..critical ladder. Tests (9 new passing; 26 total, only chaining + policy_eval remain skipped): - normalize() maps the shipped adapters/grype/adapter.yaml over recorded matches: category default, CVE list-coercion, component concat, "High" | lower -> high, raw preserved, deterministic finding_id; from_dict roundtrips severity/category. - registry discovers grype(sca)/nmap(port)/zap(dast), skips _template, and resolve_kind binds by capability and rejects a kind mismatch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Third slice ties findings + reachability facts into the contextual severity,
delta, and rationale that ARE the product. Deterministic: same inputs -> same
scores (04-scoring-policy.md, non-negotiable).
Functional now (no longer stubs):
- models/policy.py: Policy.load()/from_dict() parse the YAML policy the user owns.
- engine/policy_eval.py:
* Facts dataclass aggregating reachability + chain + asset facts, with
from_reachability() — the seam the chaining slice fills (pass real chains and
in_chain/chain_* follow).
* evaluate(): rules in order, all matches contribute, deltas sum then clamp to
max_downgrade/max_upgrade, set_severity override, stop halts further rules,
severity walks the info..critical ordinal ladder.
* full when-condition vocabulary (category, base_severity_at_least/most,
reachable_from_internet/zone, asset_data_sensitivity/attribute,
control_on_all/any_path with mitigates+mode+min_confidence, in_chain,
chain_reaches_crown_jewel, chain_min_length, exploit, evidence_confidence).
* render_rationale(): fills {finding|control|chain|asset|reachable_from_zone}
templates; unknown placeholders left literal so typos are visible.
* invariants enforced regardless of policy text: type:"unknown" floored at base
severity (never silently buried, only flagged) and not suppressible.
Tests (6 new; 32 passing, only chaining remains skipped):
- summed deltas clamp to max_downgrade; the load-bearing invariant that a chain
hop is NOT downgraded (internal-only guarded by in_chain); unknown floored at
base; KEV stop halts the later chain-to-crown-jewel upgrade; chain->crown-jewel
sets critical with rendered rationale; end-to-end over the ingested example
graph downgrades an xss high->low behind the WAF with an auditable line.
Contract note: added Contextual.tags (effect.tag from 04 needs a home; the
contextual block in 01 omitted it). Non-breaking; threaded through from/to_dict.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Final slice: the attack-path chaining engine and the reasoning pipeline that
ties all four slices together. The product thesis now works end to end — two
individually-minor findings combine into a critical chain reaching a crown jewel,
and an internal-only "low" that is a load-bearing hop is upgraded, not buried.
Functional now (no longer stubs):
- engine/chaining.py: forward, monotonic state-expansion search over the exploit
graph (findings as steps), constrained at every hop by context-graph
reachability. Seed {network:internet_reachable, auth:unauthenticated} @ internet.
Preconditions satisfied by attacker state or graph facts (network:reachable_from
:<zone> holds once pivoted in). Three termination guards: no-reuse, monotonic
growth check, hard MAX_DEPTH=6. Reportable iff multi-step AND reaches
high/crown_jewel data, authenticated_admin, or cloud_creds/secrets. Deterministic
chain ids (uuid5 over step ids), ranked by sensitivity + exploitability
- length + confidence. chains_by_finding() back-references feed in_chain/chain_*.
- engine/pipeline.py: ReasoningPipeline.run() enforces the load-bearing order —
reachability, THEN chaining, THEN policy — so downgrade rules see in_chain and a
critical chain hop is never silently downgraded (04 §Evaluation order).
Tests (8 new; 38 passing, 0 skipped — full suite green):
- the canonical "two mediums -> crown jewel" chain forms; no chain without the
pivot; MAX_DEPTH bounds a long sensitive pipeline at 6 with no finding reused;
chaining is deterministic.
- capstone pipeline test: an internal-only base-low missing_authentication hop is
re-scored to critical with chain back-reference and WITHOUT the not_internet
_reachable downgrade tag — the thesis, proven end to end; pipeline deterministic.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the loop so real scanner findings — which carry only category.type — become
chainable automatically. Faithful transcription of schema/categories.md as data:
adding a category is a new row here, not engine code.
Functional now (no longer a stub):
- adapters/conditions_map.py:
* CATEGORIES table (20 category types incl. aliases idor/lfi/deserialization)
with cwe, default pre/postconditions, mitigated_by control_types, and
default_base_severity, transcribed verbatim from categories.md.
* defaults_for(): returns seeds, or None for unknown/exotic categories (never
auto-enriched — flagged, not guessed).
* enrich(): best-effort fill of a finding's EMPTY pre/postconditions from its
category defaults, never overwriting adapter-supplied ones. Resolves the
<asset_zone> / <internal_zone> placeholders against the context graph
(resides_in / connects_to); unresolvable placeholders are dropped, not emitted.
- engine/pipeline.py: enrichment runs as step 0, before reachability/chaining, so
scanner output with no hand-authored conditions still feeds the chaining engine.
Invariants preserved (categories.md): vulnerable_dependency stays under-specified
(empty pre/post, base severity scanner-driven) and unknown categories get no
defaults — both verified by tests.
Tests (13 new; 49 passing, 0 skipped):
- defaults transcribe categories.md; aliases share defaults; vulnerable_dependency
and unknown are empty/absent; enrich fills/skips/doesn't-overwrite; <asset_zone>
and <internal_zone> resolve from graph topology; unresolvable placeholders drop.
- capstone: a pipeline run over BARE findings (only category.type + asset) enriches
the SSRF pivot and the internal authz hop and forms the crown-jewel chain ->
both critical, with zero hand-authored pre/postconditions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The whole pipeline now runs from the shell, self-hosted, with no database — the
graph persists as a JSON artifact and findings as line-delimited JSON between
stages. Closes the loop from scanner output to a human-readable re-prioritized
report with attack-chain narratives.
Functional now (no longer stubs):
- cli.py: argparse CLI (stdlib — dropped the click dep) with importable,
side-effect-explicit command functions:
ingest questionnaire -> graph.json
scan run a registered adapter -> findings.jsonl (needs the scanner binary)
reason graph + findings -> scored.jsonl + chains.json (reach->chain->score)
report scored (+ chains) -> text or json
- report/render.py: re-prioritized list sorted by contextual severity with the
base->contextual transition, per-finding rationale, an attack-chains section
with hop breakdown, and an upgraded/downgraded/suppressed summary. Suppressed
findings are hidden by default (filter, not deletion) and counted. JSON format too.
- graphstore/memory.py: InMemoryGraphStore.to_dict/from_dict (the file-based graph
artifact, no heavyweight DB to self-host).
- config.py: Config.load() parses environment.yaml; DATABASE_URL env overrides file.
- engine/pipeline.py: exposes the computed chains for reporting.
Windows fix: the report is ASCII-only (was using U+2190/U+0394/U+2014, which crash
or mojibake on cp1252 consoles) — caught by running the real CLI as a subprocess,
not just the in-process command functions.
Tests (5 new; 53 passing, 0 skipped):
- ingest writes a reloadable graph artifact; end-to-end downgrade (xss high->low
behind the WAF) through ingest->reason->report; report renders the crown-jewel
attack chain with both hops critical; json format is valid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the production storage layer behind the existing GraphStore interface, so durable multi-run history is available without changing the engine. psycopg is an OPTIONAL/lazy import — the package, the in-memory path, and the whole existing suite still work with the driver absent; you only need it to talk to Postgres. Functional now (no longer stubs): - graphstore/postgres.py: PostgresGraphStore implementing the GraphStore Protocol — upsert with ON CONFLICT attribute-merge (matching in-memory dedup), get_node, neighbors (src/dst by direction), nodes_by_type, and paths_to via the recursive CTE from schema/02. Pure row mappers (_node_from_row/_edge_from_row) and an injectable connection make it testable without a live DB. init_schema() applies migrations idempotently. - store/findings_store.py: FindingsStore — `finding` holds the normalized finding; `finding_score` is append-only, one row per scoring run, so a re-score under a new policy is distinguishable from the old one in history (04 §Trust). by_scan reconstructs Findings; suppressed are kept queryable (filter, not delete). - graphstore/migrations/: 0002 (edge natural-key unique index for upsert) and 0003 (finding + finding_score tables), plus apply_all() run in filename order. - cli.py: `reachability init-db` applies migrations (docker compose up); Config.load already resolves the DSN with a DATABASE_URL override. Testing strategy (no live Postgres assumed): pure row-mapper unit tests; a fake connection that records SQL + returns canned rows for wiring/mapping; and real Postgres integration tests gated on REACHABILITY_TEST_DSN (skipped here) that roundtrip the graph + run reachability against it and assert score-history across two policy versions. Tests: 12 new (10 run, 2 PG-integration skipped); 63 passing, 2 skipped overall. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The second ground-truth ingestion source (05-context-ingestion.md §2), and the
piece that lets vulnerable_dependency findings resolve to a real place in the
graph — enabling the reachability downgrade that kills most dependency-CVE noise.
Functional now (no longer a stub):
- ingestion/sbom.py: load()/build() parse CycloneDX and SPDX JSON into
ground_truth component{purl, version, ecosystem, direct} nodes + depends_on
edges from the owning asset (derived from the SBOM's primary component/document,
or an explicit --sbom-asset). CycloneDX uses the dependency graph to mark
direct vs transitive; the SPDX described application becomes the asset, not a
component. Component node ids use component:<name>@<version> so a scanner
finding's locator.component resolves with a direct lookup.
- resolve_component_asset(): for an SCA finding carrying only locator.component
(no node_id), find the component node and the asset that depends on it, and set
node_id — so the dependency CVE scores against that asset's reachability.
- engine/pipeline.py: resolution runs as step 0 (before enrich/reachability).
- cli.py: `ingest` now accepts --sbom (repeatable) + --sbom-asset and merges with
the questionnaire by node id (one graph, sources reconciled by id on upsert).
Tests (5 new; 68 passing, 2 PG-integration skipped):
- CycloneDX/SPDX parse to the expected components, purls, ecosystems, direct flags,
and depends_on edges; the SPDX primary app is skipped as a component; --sbom-asset
override; resolve_component_asset links a finding to its asset; and the payoff
end to end — a critical CVE on a component of an internal-only asset is
downgraded to low (dependency_unreachable), resolved purely via the SBOM graph.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gold ingestion source (05-context-ingestion.md §1): Terraform IS the topology,
not a drawing of it. Auto-derives the zones/controls/entrypoints/datastores the
questionnaire currently fills by hand, at ground_truth confidence, and reconciles
all sources into one graph with a conflict/orphan report.
Functional now (no longer stubs):
- ingestion/iac/common.py: shared Emitter/Index/resource-walker for dialect plugins.
- ingestion/iac/terraform_aws.py: maps subnets->zones, aws_lb->asset+control{lb}
+public entrypoint, wafv2 association->protected_by, listener+target-group
->routes_to backends, instances->assets (internet_reachable), RDS/S3->datastores.
Two-pass: primary nodes + id/arn index, then relationship resources. Key result:
a private-subnet instance is correctly internet-reachable THROUGH the public LB
and behind its WAF.
- ingestion/iac/terraform_azure.py: subset for the azure dialect (subnet, lb,
frontdoor->cdn+waf control + entrypoint, vm, mssql/storage), same node/edge set.
- ingestion/iac/__init__.py load(): auto-detects dialect by resource prefix.
- ingestion/reconcile.py: identity merge by id; on attribute conflict the higher
confidence wins but the conflict is RECORDED ("you said X, your IaC says Y");
declared controls protecting an asset with no inbound path flagged as orphans.
- ingestion/report.py + pipeline.py: render the reconciliation report and run
sources -> reconcile -> upsert.
- cli.py: `ingest` gains --terraform (repeatable), --reconcile-out, and prints the
reconciliation report; questionnaire/SBOM/Terraform merge by id into one graph.
Tests (19 new; 81 passing, 2 PG-integration skipped):
- AWS mapping (zones/LB/WAF/routing/RDS) and the private-instance-via-LB-behind-WAF
reachability; azure dialect auto-detected and parsed; reconcile merges by id,
ground_truth wins conflicts (recorded, order-independent), orphans flagged; the
ingestion pipeline reconciles two sources and renders the report.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The last piece of 06-attack-path-chaining.md: turning a finished, deterministic chain into readable prose. Honors the hard boundary — "the LLM explains and proposes; the engine decides" — and the no-phone-home-by-default principle. Functional now (no longer stubs): - llm/provider.py: LlmProvider Protocol with available(); NullProvider (default, never available); ClaudeProvider (lazy anthropic import + ANTHROPIC_API_KEY, with prompt caching on the static system instruction); default_provider() picks Claude only if key + SDK present, else Null. - llm/narrative.py: narrate_chain/narrate_all over a FINISHED chain. A deterministic TEMPLATE narrator is the default (no network, always works, self-hostable); an optional LLM narrator rephrases the SAME decided facts when a provider is available, and falls back to the template on any error — it never changes, adds, or re-ranks a fact. - report/render.py: optional narratives surfaced under each attack chain (text+json). - cli.py: `report --narrate` (LLM if configured, else template prose). The deterministic core is untouched and the model is never in the scoring path: narratives are computed at report time, purely from the already-decided chain. Tests (6 new; 87 passing, 2 PG-integration skipped): - template narrative is deterministic and concrete (categories, pivots, gains, crown-jewel target, hop count); a fake provider's output is used when available; errors and unavailable providers fall back to the template; default_provider is NullProvider without a key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
psycopg3's extended protocol rejects multiple commands in a single execute(), so applying a whole .sql migration file at once would fail at runtime (init_db / docker compose up). split_statements() now splits each migration into individual statements (ignoring -- comment lines; our DDL has no ';' inside strings/bodies) and apply_all() executes them one at a time. Caught while auditing the Postgres path for the live-integration run, since the gated integration tests can't execute here without a database. Verified by two new unit tests over the splitter and the real migration files (89 passing, 2 skipped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ran the gated integration tests against a real Postgres (docker postgres:15) and fixed two issues the live run surfaced: - Stores now connect with autocommit=True. With autocommit off, the store's SELECTs (get_node/neighbors/paths_to) left the connection idle-in-transaction holding locks — which blocked TRUNCATE/DDL and would also hold locks in a long-lived production store connection. Single-statement ops need no explicit transaction; commit() stays a harmless no-op (the fake-conn tests still assert it). - Integration tests get a pg_clean fixture that TRUNCATEs before each test: the DB is persistent and finding_score is append-only, so prior runs would otherwise accumulate; it also sidesteps graph_node's global PK (a node id is scoped to the first env_id that inserts it). Verified end to end against live Postgres: the recursive-CTE reachability, ON CONFLICT attribute-merge, JSONB, all three migrations via `reachability init-db`, and the two-policy score history. Full suite: 91 passed / 0 skipped with the DSN (repeatable across runs), 89 passed / 2 skipped without it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The last ingestion source (05-context-ingestion.md §4), lowest confidence:
parse a Mermaid flowchart into declared-confidence nodes/edges as a bootstrapping
aid for teams without IaC. Always subordinate to machine sources via reconcile.
Functional now (no longer a stub):
- ingestion/diagrams.py: load()/build() parse Mermaid flowcharts. Node shapes map
to types (cylinder [(..)] -> datastore; circle ((..)) -> entrypoint{public} when
the label looks like internet/user/..., else asset; others -> asset). subgraphs
become zones with resides_in membership (Mermaid semantics: the subgraph block a
node first appears inside). Directed edges -> routes_to; subgraph<->subgraph ->
connects_to. Edge labels (|..|) stripped; %% comments ignored.
- cli.py: `ingest --diagram` (repeatable), reconciled with the other sources.
Tests (6 new; 95 passing, 2 PG-integration skipped):
- node shapes -> types/labels; everything declared/source=diagram; directed edges
-> routes_to; subgraphs -> zones + resides_in; subgraph edge -> connects_to; and
end to end, the diagram alone makes the web frontend internet-reachable when no
IaC exists.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The final piece of 06-attack-path-chaining.md §LLM. When a finding's category defaults are too coarse, the LLM PROPOSES refined pre/postconditions for human review; they never silently enter the deterministic pipeline. "The LLM proposes; the engine decides." Functional now (no longer a stub): - llm/enrichment.py: propose_conditions(finding, provider) returns a Proposal (status "proposed") or None when no model is configured (the deterministic conditions_map defaults stand — the model is never required). Two guardrails: the provider must be available, and every proposed token is validated against the controlled namespace:value vocabulary (invalid tokens dropped) so the model can't inject arbitrary state into chaining. Tolerates prose-wrapped JSON. apply_proposal() is the ONLY path a proposal reaches a finding, and the pipeline never calls it — a human confirms first. - cli.py: `reachability propose --findings f.jsonl [--out proposals.json]`, the human-review entry point; degrades gracefully to "no proposals" without a key. Tests (7 new; 102 passing, 2 PG-integration skipped): - no/unavailable provider -> no proposal; valid JSON -> validated Proposal; bogus-namespace tokens dropped; prose-wrapped JSON tolerated; malformed output -> None; apply_proposal is the sole path into the finding (status -> applied). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ran a real Grype 0.114.0 scan through the adapter framework (registry ->
YamlAdapter.run -> normalize) against the CycloneDX SBOM fixture: 7 genuine
CVE/GHSA findings for lodash@4.17.20 and express@4.18.2, normalized into the
finding schema with no scanner-specific core code. Then the full pipeline scored
them against an ingested graph:
* component on an internal-only asset -> all 7 downgraded to INFO with rationale
("not reachable on any internet-facing execution path") — the dependency-noise
reduction, on real scanner output;
* same SBOM + the questionnaire (asset internet-facing) -> the same 7 CVEs KEPT
at HIGH/MEDIUM/LOW. Same scanner, opposite verdicts, driven only by context.
Fix found by the live run: YamlAdapter.healthcheck reported version "{" because
`grype version -o json` prints JSON. _extract_version() now reads a "version"
field from JSON version output (else first line), so grype reports 0.114.0.
Tests: version extraction (JSON + plain); 103 passing, 2 PG-integration skipped.
(The grype binary lives under .tools/, gitignored.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements the code-adapter path (vs grype's declarative YAML): ZapAdapter drives the ZAP daemon REST API (spider + passive/optional active scan) and maps alerts into the finding schema. normalize() is pure and unit-tested against recorded ZAP alerts. CWE -> our category vocabulary; unmapped CWEs -> "unknown" (never guessed). ZAP risk/confidence map to level / DAST evidence confidence. Verified live: ran a real ZAP 2.17.0 daemon (ghcr.io image) against a throwaway target with missing security headers; the adapter normalized 16 genuine alerts (security_misconfiguration x12: missing CSP/anti-clickjacking/X-Content-Type- Options; info_disclosure x4: server version leak) through the registry's code path. Two bugs the live run caught: - alerts endpoint is /JSON/alert/view/alerts/ (singular component), not /alerts/. - CWE-497 (system info exposure) now maps to info_disclosure (was "unknown"). Also: healthcheck uses a short timeout and 127.0.0.1 so adapter discovery stays fast when no ZAP daemon is running, and test_registry discovers once per module (was re-discovering per test -> 16s; now ~2s). Tests (4 new normalize tests; 107 passing, 2 PG-integration skipped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GitHub Actions workflow (.github/workflows/ci.yml) that runs the suite with the
two real backends, so the previously-gated tests execute on every push/PR:
* Postgres 16 as a service container + REACHABILITY_TEST_DSN -> the integration
tests run (recursive-CTE reachability, ON CONFLICT merge, JSONB, migrations,
score history).
* grype installed on PATH -> a new gated live SCA test scans the SBOM fixture
for real and asserts genuine vulnerable_dependency findings.
* ruff lint gate.
Supporting changes:
- pyproject: requires-python >=3.10 (was >=3.11) — the code runs on 3.10 thanks to
`from __future__ import annotations`, and the whole suite passes there; the stale
floor also broke editable install on 3.10. packages.find now includes only
reachability* (was also installing the test tree). ruff target -> py310.
- test_grype_live.py: gated on shutil.which("grype") (skips locally, runs in CI).
- removed an unused import flagged by ruff.
Verified locally by mirroring CI exactly (Postgres container + grype on PATH):
ruff clean, 110 passed / 0 skipped; editable install + console entrypoint validated
in a fresh venv.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expands terraform_azure from a thin subset toward AWS-level depth, with a two-pass
relationship resolver:
* azurerm_subnet -> zone; an associated NSG with an Internet inbound allow rule
marks the subnet internet_facing.
* azurerm_public_ip -> entrypoint{public}; LB / application_gateway public frontends
wire entrypoint -> routes_to asset and set internet_reachable.
* azurerm_network_interface indexed so VMs resides_in their subnet and become
internet_reachable only when the NIC has a public IP.
* azurerm_application_gateway -> asset + control{app_gateway}; firewall_policy_id
-> protected_by the WAF-policy control.
* azurerm_web_application_firewall_policy -> control{waf, block, mitigates}.
* web/function apps -> asset{internet_reachable} + entrypoint; key_vault, cosmosdb,
postgresql/mysql flexible servers, storage accounts -> datastore.
common.Emitter gains has()/set_attr() helpers (no reaching into internals).
Tests (4 new; 111 passing, 3 gated-skipped): NSG-driven internet_facing; app
gateway WAF-policy association + public entrypoint + internet_reachable; VM
resides_in subnet via NIC and stays private without a public IP; web app /
key vault / datastores. The original azure fixture still parses unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DAST findings carry a URL/host but no node_id, so until now they couldn't be
scored against the graph. Adds locator resolution and the hostnames it needs.
- ingestion/resolve.py: resolve_asset_ref() unifies asset-ref resolution — SCA
component -> owning asset (depends_on, as before), and DAST host/url -> the
entrypoint whose hostname matches, resolved to the asset it fronts (else an
asset that directly advertises the host). The pipeline now calls this in step 0.
- Hostnames captured at ingestion: questionnaire internet_facing_services entries
may be {name, host, url}; terraform-aws LB dns_name and terraform-azure public-IP
fqdn / Front Door host_name become the entrypoint's host attribute.
Verified live: re-ran a real ZAP 2.17.0 scan (16 findings, node_id None) against a
target modeled as an internal-only asset by host; all resolved to asset:internal-
portal and downgraded to INFO ("not internet-reachable") — the DAST analog of the
grype dependency-noise downgrade, on real scanner output.
Tests (6 new; 117 passing, 3 gated-skipped): questionnaire host on entrypoint+asset;
host- and url-based resolution; unknown host doesn't resolve; LB dns_name -> host;
end to end, a ZAP-style xss with only a URL resolves to the WAF-protected frontend
and is downgraded high->low.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings Azure reachability/chaining to parity with AWS by deriving the two missing
relationship classes, via a phased parse:
* ingress -> backend routes_to: application_gateway backend-pool IPs matched to
VM NIC private IPs, and LB backends via NIC backend-pool associations
(azurerm_lb_backend_address_pool + ..._backend_address_pool_association). A
private-subnet VM is now internet-reachable THROUGH the public app gateway/LB
that fronts it, exactly like an AWS instance behind an ALB.
* NSG segmentation: an inbound Allow rule sourced from another subnet's CIDR
becomes connects_to zone->zone (the segmentation map chains pivot over), while
an Internet-sourced rule still marks the subnet internet_facing.
terraform_azure rewritten into explicit phases (primary nodes -> VM placement +
private-IP map -> backend routing -> segmentation -> public-IP frontends).
Tests (4 new; 121 passing, 3 gated-skipped): app gateway routes_to backend by
private IP; LB routes_to backend via NIC pool association; inter-subnet NSG rule
-> connects_to web->app; and the parity check — a private VM is reachable from the
internet through the gateway. Existing Azure tests unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Datastores now get resides_in zone edges from Terraform, so a chain can reach a crown-jewel database derived entirely from IaC. - terraform-aws: aws_db_subnet_group (name -> subnet_ids) places aws_db_instance / aws_rds_cluster resides_in those subnets' zones. - terraform-azure: flexible servers via delegated_subnet_id, and any datastore via azurerm_private_endpoint (private_connection_resource_id -> subnet), placed in a new datastore-placement phase. Tests (4 new; 124 passing, 3 gated-skipped): AWS RDS resides_in its DB-subnet-group zone; Azure postgres (delegated subnet) and key vault (private endpoint) placed; and the capstone — a full crown-jewel chain assembled from the raw AWS Terraform fixture (command_injection on the LB-fronted instance -> foothold in the private subnet -> the now-placed RDS), verified through the CLI with a rendered narrative. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the CloudFormation dialect alongside terraform-aws/azure, emitting the same
generic node/edge set. A CFN template references resources by LOGICAL ID via Ref /
Fn::GetAtt, so wiring resolves logical ids to nodes (cleaner than Terraform state's
resolved ARNs/IPs).
- ingestion/iac/cloudformation.py: parses JSON and YAML templates (a CFN-aware
loader handles the !Ref/!GetAtt/... short-form intrinsics). Maps EC2 subnets ->
zones, ELBv2 LB -> asset+control{lb}+public entrypoint, WAFv2 WebACL +
association -> control + protected_by, target group + listener -> routes_to LB
-> backends, EC2 instance -> asset, RDS + DBSubnetGroup -> datastore placed in
its subnet zones, S3 -> datastore.
- iac.load() auto-detects CloudFormation (top-level Resources with Types) before
falling back to the Terraform dialects.
- cli.py: `ingest --cloudformation` (JSON/YAML; auto-detected by iac.load).
Tests (5 new; 129 passing, 3 gated-skipped): dialect auto-detection; Ref-based LB/
WAF/routing resolution; RDS placed in its DB-subnet-group zone; YAML short-form
intrinsics; and the parity capstone — a private instance reachable through the
internet-facing LB and a full crown-jewel chain assembled from the CFN graph.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The last unwired Azure ingress->backend path. Front Door addresses backends by FQDN, so this builds on the host attribute: web apps now publish their default_hostname (also populating host_to_asset), and Front Door backends match it. - terraform-azure: azurerm_linux/windows_web_app capture default_hostname as the asset host. Classic azurerm_frontdoor backend_pool addresses, and Standard/Premium azurerm_cdn_frontdoor_origin host_names (origin -> origin group -> profile entrypoint), resolve to the backend asset by FQDN -> routes_to from the Front Door entrypoint. App Gateway FQDN backends now resolve via the same host_to_asset map. Tests (4 new; 133 passing, 3 gated-skipped): web app hostname captured; classic Front Door routes_to backend by FQDN; Standard/Premium origin routes_to backend via the origin-group->profile chain; and the backend is reachable from the internet through Front Door with the cdn+waf control on the path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements Reachability end to end against the fixed
schema/design contracts. The base branch (main) is the scaffold commit — the design docs plus an empty package tree — so this PR's diff is the entire implementation: 23 commits, built core-first in the order the contracts prescribe.What's built
GraphStoreProtocol (in-memory + Postgres), recursive-CTE reachability (downgrade), exploit-graph chaining (upgrade) with the load-bearing chain-before-downgrade ordering, and a deterministic declarative scoring policy. The LLM is walled off from scoring.YamlAdapter(grype/nmap) and codeZapAdapter(DAST);category → pre/postconditionenrichment fromcategories.md.PostgresGraphStore+FindingsStorewith append-only score history (re-scores distinguishable perpolicy_version); idempotent migrations viareachability init-db.ingest → scan → reason → report(+init-db,propose), runnable self-hosted with no database.Verified against real tools
ON CONFLICTmerge, JSONB, migrations, two-policy score history) pass..github/workflows/ci.yml) runs the suite with a Postgres service + grype installed, so these gated checks run on every push/PR. ruff lint gate.Tests
133 passing, 0 unimplemented contracts. (Locally 3 are gated-skipped — Postgres + live-grype — which run green in CI.) Every contract in
schema/is implemented and tested; the bugs the live runs surfaced (multi-statement migrations, idle-in-transaction locking, ZAP endpoint, adapter version parsing) are fixed.Notes
.tools/(the grype binary) is gitignored — only source is pushed.GraphStore.query()DSL, multi-backend host disambiguation, CloudFormationFn::Sub/ImportValueinterpolation.🤖 Generated with Claude Code