Replace the one-page handbook with a multi-page docs site by morpheuslord · Pull Request #29 · morpheuslord/cloudg

morpheuslord · 2026-10-10T06:35:01Z

documentation updates and simple code fixes

site/build.py builds the GitHub Pages site from the Modernist design: tabs for guides, CLI, Python API, MCP server, reference and changelog, each with its own sidebar. CLI options come from the click commands, API pages from docstrings, config pages from config.yaml and the CloudGConfig models, and the long docs/*.md references are split per section. New hand-written guides, CLI and API pages carry runnable examples, editor and terminal styled code blocks with linked tabs, and mermaid diagrams. Search, dark mode and redirects from the old index.html#anchor links work as static files. The Pages workflow now builds the site.

Finding, asset and compliance values from scanner output and cloud metadata were written into the report with innerHTML and a Jinja environment that never autoescaped report.html.j2, so a crafted finding title or resource ID could run script in the page. The template now builds markup only through an escaping html`` tag, autoescape is on, the data embedded in <script> has <, >, & and ' written as \u escapes, and the fallback report escapes every value. Table rows keep the finding's position, so the detail panel shows the right finding after filtering. Chart.js 4.4.0 (MIT) and D3 7.9.0 (ISC) are vendored unmodified under cloudg/templates/vendor with their licences and a NOTICE, and inlined when report.inline_js is true (the default). With inline_js false the page loads the same pinned builds from cdn.jsdelivr.net with Subresource Integrity.

graph.persist_graphml and graph.export_cytoscape decide which graph files cloudg run writes, and graph.max_nodes_warn replaces the fixed 10,000-asset warning. rag.chunk_strategy picks the chunk kinds and rag.max_chunk_tokens is applied as a size budget (about 4 characters per token) instead of the fixed caps of 30 relations, 10 findings and 50 triples; cut lists end with an "... and N more" line. report.formats and report.inline_js apply to cloudg run and to the engine's reports, report.output_dir is the engine's default output directory, and rulesets.load_external false skips the YAML rulesets. Unknown keys are now logged for every config section, not only ratelimit, rulesets.rules_dir: null means the packaged rules, and the concurrency_limit description says what it limits.

… the docs MCP profile tool counts are re-measured (strict lists 61, soc-analyst 62, analyst 60, collector 69, standard with --read-only 65), the reference-not-found error text matches the code, and the MCP CLI notes cover the active dataset and the VS Code settings.json hint. DOCUMENTATION.md and README.md stop claiming the Docker image ships ScoutSuite, name the Terraform script import_commands.sh, say config.yaml is read only with -c, describe the unused [full] extra, list every hook phase, and split the compliance counts between the generated and the hand-written rulesets.

CloudGEngine.scan() now builds one scanner plan shared with the CLI: scanners.timeout_seconds is the subprocess timeout of every wrapper, AWS scanner credentials are resolved once from the config (keys and session token, role_arn, web identity, aws.profile), regions go to Prowler (-f) and ScoutSuite (--regions) instead of AWS_DEFAULT_REGION=ALL, and Trivy falls back to a filesystem scan. The IAM linter only runs when iam is enabled. PluginRegistry now drives scanner selection: plugin scanners run from scanners.enabled, unknown names are warned about, and built-in names always win over third-party entry points. run_pipeline() records collection, scanner and analysis failures in PipelineResult.errors, on_finding fires per scanner with copies, and the sync wrappers raise a clear error inside a running event loop. Region discovery uses the resolved credentials and falls back to the default-enabled regions. role_arn is chained before the member role instead of replacing it, an account without a role name is an error, and an OU filter that matches nothing raises instead of mapping the caller's account. The Terraform recreation follows the output directory unless terraform.output_dir is set.

- deps --json (and any --json) keeps stdout for the JSON document; the banner and logs go to stderr. - report -i restores scan ID, timestamps, compliance and edges, reads raw-findings.json, and refuses to overwrite its input without --overwrite. findings.json now carries the edges. - scan reads config.yaml, has no "." default for --iac-dir, runs the IAM linter with --assets, and exits 1 when no scanner could run. - run and scan use the engine's scanner plan (timeouts, credentials, regions, plugins); run exits 3 when collection failed for every target and prints a summary. - collect uses the auth and regions from config.yaml. - map --sweep follows inventory.tagging_sweep by default; --accounts needs --role-name; --ou help mentions ARNs.

Prowler control ids were built from the account id and the first region token instead of the check name, so the same check in two accounts gave two controls. The check name is now read from both Prowler finding id shapes (with and without the provider token). A cross-scanner merge kept only the surviving finding's source id, so when a Checkov or Trivy finding won the merge the Prowler check-id mapping was lost. The normaliser now remembers the source ids of every merged finding and maps all of them. Every compliance result was FAIL. Prowler passing checks (ASFF PASSED, OCSF PASS) are now collected and returned on a FindingList, which the normaliser reads: a ruleset control a passing check maps to, and no finding fails, gets a PASS result. Controls neither side covers are not assessed and get no result. FindingsNormaliser.normalise() also takes passed_checks directly. Prowler OCSF output (*.ocsf.json, the Prowler 4 default) was parsed as ASFF and produced "Unknown Prowler Finding" placeholders, passes included. OCSF records are now parsed for status, severity, resource, check id, compliance keys and muted state. The MCP loaders no longer refuse OCSF. ScoutSuite references (documentation URLs) are no longer copied into compliance_frameworks; the rule's compliance entries are used instead.

build_asset_map and build_compliance_map matched findings against asset names that several assets share, so a finding for table "orders" also landed on queue "orders". They now use AssetIndex, the matcher of the ontology and RAG exports: id, ARN from resource_id or resource_arn, unique name, unique ARN tail. CloudGEngine.map_inventory(findings=...) ignored the findings without output_dir. The overlay is now stored on the result as asset_map and compliance_map either way. GraphML export failed on assets with arn=None. Node arn is stored as "" and save_graphml writes through graphml_safe(), which drops None values and writes lists and dicts as JSON strings. RelationshipLinker.link() returned [] on a second call and doubled the unresolved list. Each call now starts from the seeded edges only. Coverage records are saved in inventory-map.json and restored by InventoryResult.load(); files without them load with an empty list.

One format table (ONTOLOGY_FORMATS, ontology_extension) now picks the file extension for CloudOntology.save(), CloudGEngine.analyze() and cloudg run, so N-Triples no longer overwrites ontology.ttl and jsonld, ttl, rdfxml or ntriples no longer land in a .ttl file. AnalysisResult.rag_chunk_count is set. query_asset_neighbourhood() binds the asset IRI instead of formatting it into the query, so ids with ":" or "/" work, and walks more than one hop in Python instead of the invalid {1,n} quantifier. query() returns None for unbound variables and accepts initial bindings. NACL_RULE edges get the relations SECURITY_GROUP_RULE edges get (INGRESS_ALLOWED / EGRESS_ALLOWED, INTERNET_REACHABLE, port relations, CIDR_RESTRICTED), never PROTECTED_BY_NACL. compute_blast_radius() stops at the internet placeholders instead of following an egress rule on to everything the internet reaches. findings.json metadata timestamps are ISO 8601 and an unset completed_at is null instead of "None". ontology.include_raw_metadata is read: each asset's metadata is added as a JSON string on cmp:hasRawMetadata.

cloudg run and CloudGEngine.scan() collected scanner results into plain lists, which dropped the passed_checks the Prowler parser records, so live runs still reported every compliance control as FAIL or missing. The engine's scan runner, CloudGEngine.scan() and the CLI scan outcome now return a FindingList that merges each job's passing checks.

codacy-production · 2026-10-10T06:36:39Z

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 1455 complexity · 0 duplication

Metric Results

Complexity 1455

Duplication 0

View in Codacy

AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.

_{TIP This summary will be updated as you push new changes.}

codacy-production

Pull Request Overview

This PR should not merge yet because the client-side search renderer has a high-risk HTML injection/navigation boundary, and the generator imports API paths derived from front matter without an allowlist. Both issues can enable script execution during site use or arbitrary code execution during builds.

Codacy results are not up to standards overall, with 56 new issues and missing coverage requirements. The primary high-risk area is the complex, untested documentation generator. Required acceptance criteria around strict builds, generated references, redirects, deployment, and browser interactions are not covered by tests.

About this PR

The generator, generated references, redirects, deployment workflow, and browser behavior lack automated verification. Add focused build, link-validation, generation, redirect, workflow, and interaction tests before relying on this site as the replacement handbook.

Test suggestions

Run the site generator in strict mode and verify it produces the expected pages and assets.
Verify broken internal page links and anchors cause a strict build failure.
Verify CLI pages reflect Click command usage and options.
Verify API pages load configured objects and render signatures, members, and docstrings.
Verify long reference documents are split into pages with stable anchor mappings.
Verify legacy index.html hash redirects resolve to the intended new page.
Verify GitHub Actions builds on the documented path changes and deploys the generated site artifact.
Verify generated site interactions including search, theme switching, code tabs, copy buttons, and mobile navigation.

Prompt proposal for missing tests

Consider implementing these tests if applicable:
1. Run the site generator in strict mode and verify it produces the expected pages and assets.
2. Verify broken internal page links and anchors cause a strict build failure.
3. Verify CLI pages reflect Click command usage and options.
4. Verify API pages load configured objects and render signatures, members, and docstrings.
5. Verify long reference documents are split into pages with stable anchor mappings.
6. Verify legacy index.html hash redirects resolve to the intended new page.
7. Verify GitHub Actions builds on the documented path changes and deploys the generated site artifact.
8. Verify generated site interactions including search, theme switching, code tabs, copy buttons, and mobile navigation.

_{TIP Improve review quality by adding custom instructions}
_{TIP How was this review? Give us feedback}

codacy-production · 2026-10-10T06:37:10Z

+        module = importlib.import_module(path)
+        obj, module_name, attr = module, path, path.rsplit(".", 1)[-1]
+    except ImportError:
+        module = importlib.import_module(module_name)


_{🔴 HIGH RISK}

Allowlist API module prefixes before importing, reject traversal or unrelated modules, preserve module-plus-attribute support, and fail clearly for invalid paths. Add a test proving disallowed imports are never executed.

See Issue in Codacy
See Issue in Codacy

codacy-production · 2026-10-10T06:37:10Z

+    var out = esc(text);
+    terms.forEach(function (t) {
+      if (t.length < 2) return;
+      var re = new RegExp("(" + t.replace(/[.*+?^${}()|[\]\\]/g, "\\$&") + ")", "ig");


_{🔴 HIGH RISK}

Replace dynamic RegExp construction with bounded, case-insensitive literal matching and DOM text-node highlighting. If regex remains, enforce a strict query length limit.

See Issue in Codacy
See Issue in Codacy

codacy-production · 2026-10-10T06:37:10Z

+    results.innerHTML = html || '<p class="search-empty">No results for <b>' + esc(q) + "</b>" + (filter !== "All" ? " in " + esc(filter) + '. <button type="button" class="kbd-btn" data-all>Search everything</button>' : ".") + "</p>";
+    var all = $("[data-all]", results);
+    if (all) all.addEventListener("click", function () { setFilter("All"); input.focus(); });
+    select(0);
+  }
+  function select(i) {
+    var els = $$(".search-hit", results);
+    if (!els.length) { input.removeAttribute("aria-activedescendant"); return; }
+    selected = (i + els.length) % els.length;
+    els.forEach(function (el, j) { el.classList.toggle("is-selected", j === selected); el.setAttribute("aria-selected", j === selected ? "true" : "false"); });
+    input.setAttribute("aria-activedescendant", els[selected].id);
+    els[selected].scrollIntoView({ block: "nearest" });
+  }
+  input.addEventListener("input", run);
+  results.addEventListener("mousemove", function (e) {
+    var a = e.target.closest(".search-hit");
+    if (a) { var i = parseInt(a.getAttribute("data-i"), 10); if (i !== selected) select(i); }
+  });
+  results.addEventListener("click", function (e) { if (e.target.closest(".search-hit")) closeSearch(); });
+  modal.addEventListener("keydown", function (e) {
+    if (e.key === "ArrowDown") { e.preventDefault(); select(selected + 1); }
+    else if (e.key === "ArrowUp") { e.preventDefault(); select(selected - 1); }
+    else if (e.key === "Enter" && document.activeElement === input) {
+      var el = $$(".search-hit", results)[selected];
+      if (el) { e.preventDefault(); closeSearch(); location.href = el.getAttribute("href"); }


_{🔴 HIGH RISK}

Avoid innerHTML for search results. Construct the UI with DOM APIs and textContent, highlight literal terms without HTML interpolation, and restrict navigation to same-origin documentation URLs. Add regression tests for malicious search terms and javascript: URLs.

See Issue in Codacy
See Issue in Codacy
See Issue in Codacy

codacy-production · 2026-10-10T06:37:10Z

+            from cloudg.mcp import default_registry
+
+            tools = len(default_registry().tools())
+        except Exception:  # noqa: BLE001


_{🟡 MEDIUM RISK}

Catch only expected optional-dependency failures; log and propagate unexpected exceptions, especially in strict mode, so broken registry behavior cannot be reported as zero tools.

See Issue in Codacy

codacy-production · 2026-10-10T06:37:10Z

+    list_items: list[str] = []
+
+    def flush_list() -> None:
+        nonlocal list_items, last_key


_{🟡 MEDIUM RISK}

Suggestion: Remove the unused last_key from the nonlocal declaration.

See Issue in Codacy

markdown-it binds each render rule to its renderer with __get__. On Python 3.12 that re-binds an already bound method, so the custom rules ran with the renderer as self and the Pages build failed with "'RendererHTML' object has no attribute 'heading_base'". Python 3.13, used locally, keeps the original binding. The rules are now plain functions that ignore the renderer argument.

The docs pages, the handbook and the MCP references described several bugs as they were. They now match the code: offline and escaped report.html, config keys that take effect, report -i round trips and the overwrite guard, scanner timeouts and credentials, exit codes, OCSF parsing, PASS compliance results, coverage in inventory maps, ontology formats and queries, blast radius and the Terraform directory. cloudg run also exits 3 when no credentials are found at all. Every service then fails on its own while the target record says PARTIAL, so collection_failed() ignored it and the failure panel stayed empty. Both now treat a target whose services all failed as failed. CHANGELOG.md gets the Unreleased Security, Fixed and Changed entries.

Codacy flags functions over its complexity and length limits. The recent fixes pushed several over: failed_collection_targets, the scan and report commands, the report input loader, the organization account filter, compliance mapping, scanner selection and plugin runs, the OCSF record parser and the scanner planner. Each is now split into small helpers or a dispatch table; behaviour is unchanged and the full suite passes.

Codacy's ESLint security rules flagged the docs script: search results built as HTML strings and written with innerHTML, a RegExp built from the query for match highlighting, bracket lookups with variables, a location.href assignment, and functions over the length and complexity limits. The script is now a set of ES modules: search results are built with DOM calls, matches are found with indexOf and wrapped in <mark> elements, lookups use Maps, Enter clicks the selected link, and mermaid is imported only on pages that have diagrams. Behaviour is unchanged.

Semgrep's credential-disclosure rule matched the word in the message. The message logs only the resolution error, never a secret.

Codacy limits a file to 500 non-comment lines and a function to eight parameters. The fixes pushed cloudg/api.py, cloudg/config.py and cloudg/graph/rag_export.py over the file limit, and ProwlerScanner and scan_plugin over the parameter limit. The engine result types and collection checks move to cloudg/api_results.py, the rate-limit models and unknown-key warnings to cloudg/config_ratelimit.py and cloudg/config_keys.py, and the RAG chunk text builders to cloudg/graph/rag_text.py; all of them stay importable from their old modules. ProwlerScanner takes its newer settings (session token, regions, timeout, env) as keyword settings, so existing calls keep working, and scan_plugin takes the planner's inputs as one object. scanner_available() imports the scanner classes directly instead of by dotted path.

site/build.py was one 800-line file. It is now a thin command line, and the generator lives in site/docsgen/: page.py (the page model), site.py (the Site class), discover.py and split.py (nav walking and splitting the long docs/*.md files), render.py, output.py and links.py. Long functions in gen_api, gen_config, gen_cli, gen_changelog, highlight and md are broken into helpers, and if/elif chains became dispatch tables, so no function goes over complexity 10 or 50 lines. The fence tracking that was copied into four loops is one FenceTracker in fences.py. Other static-analysis fixes: - Source links use GITHUB_SHA when it is set (CI) and main otherwise, instead of running git to look for a release tag. - API objects are resolved with pkgutil.resolve_name, and only dotted paths inside the cloudg package are accepted. - The numbered-heading regex has no nested quantifiers. - The MCP tool and asset type counts log at debug level when the import fails instead of passing silently. - gen_config.parse is a small parser class, without the nonlocal. The wbr filter (and its Markup) is gone. Option values like aws|azure|gcp are split into inline-block spans and "--x / --no-x" flags into nowrap spans, so the tables break in the same places as before without building HTML in Python. The generated site is unchanged apart from those cells and the source-link ref.

Codacy ran stylelint with a default config that names scss rules and reported an unknown-rule error on docs.css. This config uses a few core rules, none from scss and no extends, so it works without shareable configs installed.

Registry.tools is a property, so the docs build called a dict, the error was swallowed and the home page fell back to the hard-coded tool count. cloudg.config declares its re-exported rate-limit models in __all__, and cloudg.graph.rag_export imports only the chunk helpers it uses.

Turns the Unreleased section into 0.6.1 (2026-10-10) and records the ProwlerScanner keyword settings and the modules split out of cloudg.api, cloudg.config and cloudg.graph.rag_export.

morpheuslord added 11 commits October 9, 2026 22:27

Say which scanners the Docker image bundles in the CI guide

12ef61e

codacy-production Bot reviewed Oct 10, 2026

View reviewed changes

morpheuslord added 10 commits October 10, 2026 12:08

Reword the scanner sign-in failure log message

089ee31

Semgrep's credential-disclosure rule matched the word in the message. The message logs only the resolution error, never a secret.

Add a stylelint config with core rules only

9a3dc86

Codacy ran stylelint with a default config that names scss rules and reported an unknown-rule error on docs.css. This config uses a few core rules, none from scss and no extends, so it works without shareable configs installed.

Add the 0.6.1 changelog entry

fc10d11

Turns the Unreleased section into 0.6.1 (2026-10-10) and records the ProwlerScanner keyword settings and the modules split out of cloudg.api, cloudg.config and cloudg.graph.rag_export.

morpheuslord merged commit 92dac90 into main Oct 10, 2026
9 checks passed

morpheuslord deleted the docs/site-redesign branch October 10, 2026 07:33

Navigation Menu

Repository navigation

Replace the one-page handbook with a multi-page docs site - #29

Replace the one-page handbook with a multi-page docs site#29
morpheuslord merged 21 commits into
mainfrom
docs/site-redesign

morpheuslord commented Oct 10, 2026

Uh oh!

codacy-production Bot commented Oct 10, 2026 •

edited

Loading

Uh oh!

codacy-production Bot left a comment

Uh oh!

codacy-production Bot Oct 10, 2026

Uh oh!

codacy-production Bot Oct 10, 2026

Uh oh!

codacy-production Bot Oct 10, 2026

Uh oh!

codacy-production Bot Oct 10, 2026

Uh oh!

codacy-production Bot Oct 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

morpheuslord commented Oct 10, 2026

Uh oh!

codacy-production Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Up to standards ✅

Uh oh!

codacy-production Bot left a comment

Choose a reason for hiding this comment

Pull Request Overview

About this PR

Test suggestions

Uh oh!

codacy-production Bot Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

codacy-production Bot Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

codacy-production Bot Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

codacy-production Bot Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

codacy-production Bot Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codacy-production Bot commented Oct 10, 2026 •

edited

Loading