Skip to content

ci: analyse every CodeQL language on every pull request, group and push - #353

Merged
botandbang merged 1 commit into
mainfrom
bnb/codeql-every-language
Nov 1, 2026
Merged

botandbang merged 1 commit into
mainfrom
bnb/codeql-every-language

Conversation

@botandbang

Copy link
Copy Markdown
Contributor

What and why

#352 ran CodeQL's Rust and TypeScript analyses only on commits that changed that code. The "Protect Main" ruleset's code scanning rule holds a pull request or merge queue group until its commit has a result in every category main has. So #347, a web-only pull request, got no Rust analysis, and adding it to the queue failed with "Code scanning is still expecting 1 result from CodeQL for 3de1fc5 or 1df9ec4". The same blocks every pull request that leaves out Rust or TypeScript (#329, #338, #342, #345, #347, bnb/lighthouse-history).

  • Changed: ci.yml's CodeQL call runs every language (actions, javascript-typescript, rust) on every pull request into main, merge queue group and push to main, whatever changed. Drafts and pull requests into other branches still skip it. Main's pushes analyse every language too, since pull requests' alerts are compared with main's analysis.
  • Removed: the codeql_rust and codeql_javascript filters, the codeql_languages output, and codeql.yml's languages input. Its matrix is a fixed list again.
  • Added: a policy in tools/policy/ruleset.rego. When the ruleset has a code scanning rule, every job in codeql.yml has to list its languages as a fixed matrix, not an expression, with tests.
  • Updated .github/CONTRIBUTING.md and .github/NEW_PROJECT.md.

Rust's analysis is build-mode none, about 15 to 19 minutes on Linux, so this gives back part of #352's CodeQL saving on pull requests that don't touch Rust. A cheaper option for later: upload main's last Rust SARIF for the pull request's commit when no Rust changed. It would be accurate, since the Rust is identical, but it needs a real run to check that the rule accepts it.

After this merges, each blocked pull request needs a run that includes it: merge main into it, or close and reopen it once its merge ref contains this commit.

How it was tested

  • tools/policy.sh: 84/84 Rego tests at full coverage, 51/51 conftest checks. The new rule's test fails when the rule is broken on purpose.
  • actionlint (with shellcheck) and zizmor --offline are clean.
  • The scope script, run locally, sets codeql=true for pull requests into main (Rust-only, Markdown-only, the all-os label, a 3000-file one), merge groups and pushes, and false for a draft and for a pull request into another branch.
  • That the rule is then satisfied only shows on GitHub: this pull request's own run analyses all three languages, as ci: run macOS and Windows in a merge queue instead of on every push #350's group did.

Checklist

  • Tests cover the change (the policy)
  • The READMEs, docs/ and --help still say what the code does

The code scanning rule holds a commit until it has a result in every category main has, so
#347, which changed no Rust and got no Rust analysis, couldn't join the merge queue. A policy now
fails when codeql.yml's matrix isn't a fixed list of languages.
@botandbang
botandbang requested a review from bnb as a code owner November 1, 2026 16:15
@github-actions github-actions Bot added documentation Improvements or additions to documentation github_actions Pull requests that update GitHub Actions code tooling Repo config, lint and build tooling, and GitHub settings labels Nov 1, 2026
@botandbang
botandbang enabled auto-merge November 1, 2026 16:16
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Coverage Report for apps/web

Status Category Percentage Covered / Total
🟢 Lines 95.43% (🎯 93%)
🟰 ±0 pp
7916 / 8295
🟢 Statements 94.09% (🎯 91%)
🟰 ±0 pp
12053 / 12810
🟢 Functions 94.21% (🎯 92%)
🟰 ±0 pp
3126 / 3318
🟢 Branches 84.71% (🎯 80%)
🟰 ±0 pp
5621 / 6635
File CoverageNo changed files found.
Generated in workflow #19 for commit c2d0754 by the Vitest Coverage Report Action

Size

Compared with main. KiB are 1,024 bytes.

This pull request main Change
Client JavaScript 680.7 KiB 680.7 KiB 0 B (0%)
Client JavaScript, brotli 226.8 KiB 226.8 KiB 0 B (0%)
Client CSS 89.6 KiB 89.6 KiB 0 B (0%)
Client CSS, brotli 12.9 KiB 12.9 KiB 0 B (0%)

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Coverage Report for apps/localtoolbox

Status Category Percentage Covered / Total
🟢 Lines 100% (🎯 100%)
🟰 ±0 pp
4303 / 4303
🟢 Statements 99.64% (🎯 95%)
🟰 ±0 pp
5862 / 5883
🟢 Functions 99.65% (🎯 95%)
🟰 ±0 pp
1715 / 1721
🟢 Branches 97.72% (🎯 50%)
🟰 ±0 pp
2791 / 2856
File CoverageNo changed files found.
Generated in workflow #19 for commit c2d0754 by the Vitest Coverage Report Action

Size

Compared with main. KiB are 1,024 bytes.

This pull request main Change
Packaged (localtoolbox.zip) 366.1 KiB 366.1 KiB +20 B (+0%)
Unpacked (dist/) 842.8 KiB 842.8 KiB 0 B (0%)

@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown

Rust coverage

From run 19. Both are held to the thresholds in .cargo/config.toml.

Tests Lines Functions Regions
All 97.39% (84410/86668) 96.53% (13503/13989) 95.67% (185850/194265)
Hegel property tests alone 95.4% (82678/86668) 94.12% (13166/13989) 93.47% (181575/194265)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation github_actions Pull requests that update GitHub Actions code tooling Repo config, lint and build tooling, and GitHub settings

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants