Replace the one-page handbook with a multi-page docs site

.codacy.yml

@@ -4,11 +4,13 @@
 # tests/ are excluded from analysis: pytest asserts (Bandit B101) and
 # moto's documented fake credentials (B105 "testing") are correct usage
 # there, and they accounted for ~290 false positives. Shipped code under
-# cloudg/ stays fully analysed.
+# cloudg/ stays fully analysed, except the unmodified third-party
+# Chart.js and D3 builds the HTML report inlines (templates/vendor/NOTICE).
 exclude_paths:
   - "tests/**"
   - "docs/**"
   - "assets/**"
+  - "cloudg/templates/vendor/**"
 
 engines:
   # dodgy pattern-matches the word "secret" in any assignment, which the

.github/workflows/pages.yml

@@ -1,6 +1,8 @@
 name: Docs
 
-# Deploys docs/ to GitHub Pages on every push to main that touches it.
+# Builds the documentation site (site/build.py) and deploys it to GitHub Pages
+# on every push to main that touches the docs, the content they are generated
+# from (click commands, docstrings, config.yaml, CHANGELOG.md) or the site itself.
 #
 # One-time setup: under Settings -> Pages, set "Build and deployment"
 # source to "GitHub Actions". The GITHUB_TOKEN cannot create the Pages
@@ -12,18 +14,59 @@ on:
     branches: [main]
     paths:
       - "docs/**"
+      - "site/**"
+      - "cloudg/**"
+      - "config.yaml"
+      - "CHANGELOG.md"
+      - "pyproject.toml"
+      - ".github/workflows/pages.yml"
+  pull_request:
+    paths:
+      - "docs/**"
+      - "site/**"
       - ".github/workflows/pages.yml"
   workflow_dispatch:
 
 permissions:
   contents: read
 
 concurrency:
-  group: pages
+  group: pages-${{ github.ref }}
   cancel-in-progress: true
 
 jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          fetch-depth: 0 # tags, so source links point at the release tag
+
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.12"
+
+      # The CLI and API reference pages are generated by importing cloudg,
+      # so the package is installed; its core dependencies already include
+      # everything the site generator needs (markdown-it-py, Pygments, Jinja2).
+      - name: Install cloudg
+        run: pip install .
+
+      - name: Build the site
+        run: python site/build.py --strict
+
+      - uses: actions/upload-pages-artifact@v3
+        if: github.event_name != 'pull_request'
+        with:
+          path: site/dist/
+          # The artifact name carries the run attempt: on a re-run, the previous
+          # attempt's "github-pages" artifact still exists in the run, and
+          # deploy-pages refuses when it finds two artifacts with the same name.
+          name: github-pages-${{ github.run_attempt }}
+
   deploy:
+    if: github.event_name != 'pull_request'
+    needs: build
     runs-on: ubuntu-latest
     permissions:
       pages: write
@@ -32,20 +75,9 @@ jobs:
       name: github-pages
       url: ${{ steps.deployment.outputs.page_url }}
     steps:
-      - uses: actions/checkout@v7
-
-      - uses: actions/configure-pages@v6
-
-      # The artifact name carries the run attempt: on a re-run, the previous
-      # attempt's "github-pages" artifact still exists in the run, and
-      # deploy-pages refuses when it finds two artifacts with the same name
-      # ("Multiple artifacts named github-pages were unexpectedly found").
-      - uses: actions/upload-pages-artifact@v5
-        with:
-          path: docs/
-          name: github-pages-${{ github.run_attempt }}
+      - uses: actions/configure-pages@v5
 
       - id: deployment
-        uses: actions/deploy-pages@v5
+        uses: actions/deploy-pages@v4
         with:
           artifact_name: github-pages-${{ github.run_attempt }}

.stylelintrc.json

@@ -0,0 +1,14 @@
+{
+  "rules": {
+    "at-rule-no-unknown": true,
+    "color-no-invalid-hex": true,
+    "comment-no-empty": true,
+    "declaration-block-no-duplicate-properties": true,
+    "function-calc-no-unspaced-operator": true,
+    "keyframe-declaration-no-important": true,
+    "media-feature-name-no-unknown": true,
+    "no-invalid-double-slash-comments": true,
+    "string-no-newline": true,
+    "unit-no-unknown": true
+  }
+}

CHANGELOG.md

@@ -2,6 +2,56 @@
 
 Notable changes per release. Patch releases are folded into the major entry they belong to.
 
+## 0.6.1 (2026-10-10)
+
+A security fix for report.html, a round of fixes found while documenting every command and API, and the new docs site. Upgrade if you open reports built from scanner output you did not produce yourself. Several commands now behave the way their help text always said, so check the Changed list if you script `cloudg report`, `cloudg scan` or the exit codes.
+
+Security:
+
+- report.html no longer runs markup or script that arrives in scanner output. Finding titles, descriptions, evidence, resource IDs, asset names and compliance framework names are escaped wherever the page writes them, the page header is autoescaped, and data embedded in the page's script can no longer close the script element.
+- A package can no longer take over a built-in scanner (such as `prowler`) by registering a `cloudg.scanners` entry point with the same name. Built-in names always win and the duplicate is ignored with a warning.
+
+Fixed:
+
+- report.html works offline. Chart.js 4.4.0 and D3 7.9.0 ship with cloudg and are embedded in the page when `report.inline_js` is true (the default). With `inline_js: false` the page loads the same pinned builds from cdn.jsdelivr.net with Subresource Integrity. Clicking a row in a filtered findings table now opens that finding.
+- Config keys that were accepted but ignored now take effect: `graph.persist_graphml`, `graph.export_cytoscape`, `graph.max_nodes_warn`, `rag.chunk_strategy`, `rag.max_chunk_tokens`, `report.formats`, `report.inline_js`, `report.output_dir` (as the default of the `CloudGEngine` methods), `rulesets.load_external` and `ontology.include_raw_metadata`. A misspelt key in any section is logged as `Unknown config key <section>.<key> is ignored`, not only under `ratelimit`. `rulesets.rules_dir: null` loads the packaged rulesets instead of failing validation.
+- `cloudg deps --json` writes only JSON to stdout. Whenever `--json` is used, and for `cloudg mcp`, the banner and logs go to stderr.
+- `cloudg report -i` keeps the scan ID, timestamps, compliance results and edges of a findings.json, rebuilds edges for older files that lack them, and accepts raw-findings.json.
+- `scanners.timeout_seconds` limits each scanner process in `cloudg run`, `cloudg scan` and `CloudGEngine.scan()`. Before, the limits were hard-coded at 3600 and 1800 seconds. A scanner that times out is reported as failed and its partial findings are kept.
+- `cloudg scan` reads `-c config.yaml`, no longer scans the current directory unless asked, runs the IAM linter when given `--assets`, and exits 1 when no requested scanner could run. `cloudg collect` uses the credentials and regions in config.yaml. `cloudg map --sweep` follows `inventory.tagging_sweep` unless the flag is given.
+- Prowler and ScoutSuite get the credentials cloudg was given: session token, the assumed `role_arn`, web identity and `aws.profile` from the config, and real regions. `AWS_DEFAULT_REGION=ALL` is no longer set. Region discovery for `--regions all` uses the same credentials, and falls back to the 17 regions enabled by default in every account when discovery fails.
+- `cloudg run` exits 3 when collection failed for every target, including a run with no credentials at all where every service fails on its own, and lists the targets that failed. Partial collection still exits 0, with a warning.
+- `--accounts` without `--role-name` (or `aws.role_name`) is an error instead of mapping the caller's own account once per listed ID. `aws.role_arn` is assumed first and the member role from there, instead of replacing the member role. An `--ou` that matches nothing is an error that lists the organization's OUs.
+- Scanner plugins registered under `cloudg.scanners` run from `scanners.enabled` and `--scanners`, and unknown scanner names produce a warning. The IAM linter follows `scanners.enabled` and `--scanners` too.
+- `CloudGEngine.run_pipeline()` passes `aws.profile` to the scanners, `PipelineResult.errors` lists collection, scanner and analysis failures as well, `on_finding` fires as each scanner finishes and receives copies normalisation does not change, and the `*_sync` wrappers raise a clear error inside a running event loop. `CloudGEngine.scan()` falls back to a Trivy filesystem scan when no images are set, like the CLI.
+- Prowler compliance control IDs name the check (`CIS/iam_root_hardware_mfa_enabled`) instead of including the account ID, so one check across several accounts is one control. When findings from two scanners are merged, both scanners' check-ID mappings are kept.
+- Compliance results are no longer all FAIL. A control fails when a finding maps to it and passes when only Prowler passing checks (ASFF `PASSED`, OCSF `PASS`) map to it; controls nothing assessed are left out. `FindingsNormaliser.normalise()` takes `passed_checks=`, and the Prowler parser returns them on the list it gives back.
+- Prowler OCSF output (`*.ocsf.json`, the Prowler 4 default) is parsed: status, severity, resource, check ID, compliance and muted findings. Before, every record became an "Unknown Prowler Finding", passing checks included. The MCP loaders accept OCSF instead of refusing it. ScoutSuite documentation URLs no longer show up as compliance frameworks.
+- `build_asset_map()` and `build_compliance_map()` match findings to assets the way the ontology and RAG exports do, so a finding no longer attaches to every asset that shares its name. `CloudGEngine.map_inventory(findings=...)` applies the overlay without `output_dir` too (`result.asset_map`, `result.compliance_map`).
+- GraphML export (`cloudg run`, `InventoryResult.export()`, the MCP export) no longer fails on assets without an ARN. `RelationshipLinker.link()` can be called more than once. `inventory-map.json` saves the collection coverage records and `InventoryResult.load()` restores them.
+- Each ontology format gets its own file (`.ttl`, `.jsonld`, `.rdf`, `.nt`). N-Triples no longer overwrite the Turtle file, and spellings such as `jsonld` no longer land in `ontology.ttl`. `AnalysisResult.rag_chunk_count` is filled in.
+- `CloudOntology.query_asset_neighbourhood()` works with `hops` above 1 and with IDs containing `:` or `/`, and `query()` returns None for unbound variables. `NACL_RULE` edges get the same ontology relations as `SECURITY_GROUP_RULE` edges.
+- `ReachabilityAnalyzer.compute_blast_radius()` stops at the internet placeholder instead of following an egress rule out to everything the internet reaches.
+- findings.json metadata timestamps are ISO 8601, and an unset `completed_at` is `null` rather than `"None"`.
+
+Changed:
+
+- `cloudg report` refuses to write findings.json over its own input; pass `-o` to write elsewhere, or `--overwrite`. findings.json now has an `edges` list.
+- `cloudg scan --scanners` defaults to `scanners.enabled` instead of `prowler,checkov`, and `cloudg scan -p` accepts only aws, azure and gcp.
+- The Terraform recreation goes to `<output>/terraform`, following `-o`, unless `terraform.output_dir` is set to another path. The old default `./reports/terraform` counts as unset.
+- `cloudg run` writes `topology-cytoscape.json` only with `graph.export_cytoscape: true`, and skips `topology.graphml` with `graph.persist_graphml: false`. `run_pipeline()` writes the formats in `report.formats` and now includes `topology.svg`.
+- RAG chunks are sized by `rag.max_chunk_tokens` (about 4 characters per token) instead of fixed caps of 30 relations, 10 findings and 50 triples. Long lists end with an "... and N more" line, and community chunks are bounded too.
+- report.html is about 485 KB larger with the libraries embedded.
+- `query_asset_neighbourhood()` rows for more than one hop are `{neighbour, neighbourName, hops}`. D3 graph JSON has `"arn": ""` instead of `null` for assets without an ARN. `cloudg.mcp.state.check_prowler_input` was removed.
+- `ProwlerScanner` takes its newer settings (`aws_session_token`, `aws_regions`, `timeout_seconds`, `env`) and the access keys as keyword arguments; calls that pass them by keyword work as before. `aws_region` is now the fifth positional argument.
+- Parts of three large modules moved into their own: the engine result types and collection checks to `cloudg.api_results`, the rate-limit models to `cloudg.config_ratelimit`, and the RAG chunk text builders to `cloudg.graph.rag_text`. Everything public is still importable from `cloudg.api`, `cloudg.config` and `cloudg.graph.rag_export`.
+
+Documentation:
+
+- The handbook at morpheuslord.github.io/cloudg is now a site of separate pages instead of one long `docs/index.html`. Guides, the CLI, the Python API, the MCP server, the reference documents and this changelog each have their own tab and sidebar. `site/build.py` builds it: the CLI options tables come from the click definitions, the API pages from the docstrings, the config pages from `config.yaml` and the `CloudGConfig` models, and the long `docs/*.md` references are split into one page per section. Writing rules for new pages are in `site/AUTHORING.md`.
+- New guides cover installation, the pipeline phase by phase, authentication for each provider, AWS Organizations, scanners and ingest input formats, compliance mapping, reports, the graph, ontology and RAG exports, rate limits, Docker, CI and Python recipes. Code samples are complete scripts, and processes are drawn as diagrams.
+- Search (Ctrl K or /), a dark theme, linked code tabs and copy buttons work without a server. Links to the old `index.html#anchor` sections redirect to the new pages.
+
 ## 0.6.0 (2026-10-09)
 
 Two additions and a round of graph fixes. cloudg is now a Model Context Protocol server, so AI agents can query an inventory map, its graph, findings, compliance and ontology under an access and privacy policy. Every cloud API call now goes through a shared rate limiter that slows down when a provider throttles, so an organization-wide map (or an agent asking for one too often) no longer fails or hammers the APIs.