Trim repology index and filter uncomparable advisories

[andrew]

Makes the ingest.yml sweep complete and its output actionable. No brew changes required.

RepologyIndex.select_ecosystems keeps Debian+Ubuntu when present, otherwise one fallback ecosystem by priority. The distro strategy is CVE discovery for cases git/registry missed; once a CVE id is found it is checked against the formula's own identity, so one well-covered distro is sufficient and the rest add duplicate querybatch traffic and @vuln_cache entries. Sampling curl and openssl@3 showed CVEs reachable only via the dropped ecosystems were srcname collisions (Jenkins, wget, python-cryptography, PHP) or unresolved distro-native ids; a random 15-record sample of dropped output were all false positives (bluez→kernel nilfs2, emacs→ImageMagick, icu4c→Chrome V8, and similar).

RepologyIndex also reads data/overrides.yml and drops skip: true formulae from the index. linux-headers@{4.4,5.15,6.8} are skipped: the Repology mapping to distro kernel packages is correct but kernel runtime CVEs do not apply to a headers-only install, and those three formulae accounted for ~48k of ~122k candidate records in earlier runs. In --all mode Match runs with bulk: true and does not fall back to live Repology, so a formula absent from the index with no git/registry identity is skipped.

data/repology.json is retrimmed with the new policy: 54129 → 16000 srcnames, 5460 → 5457 formulae. rake repology:trim reapplies without re-crawling.

AdvisoryIndex.build excludes source: matched records with no range_state from data/advisories.json. Those records stay in advisories/ for review but are not surfaced to brew vulns. On a full local run: 90820 candidates → 20358 actionable (19247 fixed, 1111 affected) across 625 formulae. The excluded set is 100% of distro-strategy and 98.7% of git-strategy hits; registry-strategy records are essentially all comparable. The consequence is that non-registry formulae (openssl@3, ffmpeg, curl, sqlite, go, openjdk, mysql) are absent from the client index until Vulnerability#range_status can evaluate type: GIT commit-SHA ranges; that is the top brew-side follow-up.

spec/advisory_index_spec.rb replaces test/advisory_index_test.rb, which required a nonexistent test_helper.rb and was never run in CI.

ingest.yml: --verbose so a failure shows progress, free -m/df -h before the match, timeout 360 → 180. Local end-to-end run with this config: 126.4 min, peak RSS 1678 MB, exit 0; git add/commit of the 90820 files ~54s, pack for push 43 MB.

[Copilot]

Pull request overview

This PR trims the Repology-derived distro mapping to reduce noisy/duplicative OSV queries, adds per-formula overrides for skipping unsuitable matches, and filters uncomparable matched advisories out of the client-facing data/advisories.json index so ingest output is more actionable.

Changes:

• Add RepologyIndex.select_ecosystems, RepologyIndex.skipped_formulae, and RepologyIndex.trim, plus override-driven formula skipping and a rake repology:trim task.
• Add AdvisoryIndex.actionable? and exclude source: matched advisories that lack range_state from the concatenated index (while tracking skipped count).
• Replace the non-running Minitest file with RSpec coverage and tweak the ingest workflow runtime/verbosity/diagnostics.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
test/advisory_index_test.rb	Removes unused/non-CI Minitest coverage.
spec/advisory_index_spec.rb	Adds RSpec coverage for AdvisoryIndex including the new “actionable” filter and skipped count reporting.
lib/advisory_index.rb	Filters out uncomparable matched records from data/advisories.json and reports skipped count.
lib/repology_index.rb	Adds ecosystem selection policy, override-driven formula skipping, and an in-place trim mode.
spec/repology_index_spec.rb	Adds/updates specs for ecosystem selection, overrides skipping, and trim behavior.
spec/fixtures/repology_page2.json	Updates fixture data to include Ubuntu libzstd so preferred-ecosystem behavior is exercised.
data/overrides.yml	Introduces skip: true overrides (e.g., linux-headers formulae) to suppress unsuitable matching.
Rakefile	Adds repology:trim task for reapplying selection/override policy without re-crawling Repology.
.github/workflows/ingest.yml	Adjusts ingest job timeout, adds runner diagnostics, and runs brew advisory-match with --verbose.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.