Skip to content

fix(sbom): enforce SemVer evidence for risk decisions - #120

Merged
stacknil merged 3 commits into
mainfrom
test/sbom-version-risk-decision-stability
Oct 21, 2026
Merged

stacknil merged 3 commits into
mainfrom
test/sbom-version-risk-decision-stability

Conversation

@stacknil

@stacknil stacknil commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Problem

On main at f7f9cae1e0ff718225677160fd59a7922ececaca, version-risk admission disagrees with the documented strict SemVer heuristic. From 1.2.3:

After evidence Baseline Corrected
2.0.0-01, 2.0.0-alpha..1, non-ASCII core digits major_upgrade version_change_unclassified
2.0.0-rc.1+build.01 version_change_unclassified major_upgrade
A valid 5,000-digit major integer-conversion error classified without integer conversion

PURL-only and explicit evidence reproduce the same issue. Aggregate risk counts can remain unchanged while major findings are attributed to the wrong component.

Invariant

A major-upgrade finding requires two known, unequal canonical versions with eligible syntax and a strictly higher major. Unknown evidence, equal versions, decreases and same-major transitions cannot prove a major upgrade. Risk remains a local change heuristic, distinct from policy enforcement and SARIF projection.

Design decision

Validate full ASCII SemVer 2.0.0 core, prerelease and build grammar. Preserve the established outer-whitespace and lowercase-v wrappers. Compare validated major digit strings by numeric magnitude, avoiding Python's decimal-conversion limit without changing interpreter settings.

The independent test oracle decomposes delimiters and checks characters/leading zeroes from the SemVer 2.0.0 grammar; it never calls the production regex, parser helpers or classifier. Literal eligibility anchors, transition controls and exact component/rule expectations avoid count-only or circular checks. Syntax eligibility does not imply package-manager precedence.

Threat/failure model

Malformed version evidence can create a confident major finding, while valid combined suffixes lose that finding. Existing policy rules can then block the wrong component or return the wrong exit code. Oversized valid major evidence can prevent report completion. This change does not infer vulnerability, advisory, CVE, reputation or actual API compatibility information.

Rejected alternatives

  • Narrow the documentation to bless the permissive suffix matcher: rejected because it preserves false confidence for malformed identifiers.
  • Use a package-manager parser universally: rejected because it would introduce ecosystem-specific precedence into the shared syntax heuristic.
  • Relax Python's integer-conversion limit: rejected because decimal conversion is unnecessary and a global setting change would widen scope.
  • Broaden SARIF mappings or suppress raw risks through policy: rejected because those contracts are independent of this defect.

Compatibility

Only corrected risk admission changes consumer outcomes. Existing major_upgrade and version_change_unclassified rules may exchange matches, block exits and policy_matched confidence as shown above. Warnings and ignored matches retain exit 0. A suppressed major policy match can retain its independent SARIF note, without active policy escalation. Unclassified risks remain intentionally unmapped to SARIF.

Canonicalization, versionless keys, fail-closed conflicts/duplicates, raw component fields/evidence, unknown handling, schema IDs/legacy fields, ordering and offline defaults remain unchanged. No dependencies, examples/goldens, workflows, release notes, tags or release assets change. This candidate has not shipped in v1.1.0.

Validation

  • Unmodified main: 272 tests and example --check pass.
  • Final new tests against unchanged production: 50 failed / 151 passed; candidate: 201 passed.
  • Windows/Python 3.12.7: 473 passed, pip check clean, Ruff correctness and formatting/syntax checks pass.
  • CycloneDX/SPDX CLI controls validate full JSON, summary, policy sidecar, Markdown and SARIF; exact rule/evidence/component attribution; all policy modes; repeat/reordered replay; and network/enrichment rejection.
  • 36 base/candidate CLI counterfactuals: 12 normal-version controls produce identical five-format artifacts; all 8 exit and 16 confidence changes follow corrected matches to existing rules.
  • 6/6 semantic mutants rejected: permissive syntax (48 failures), downgrade admission (26), unknown admission (19), lost PURL evidence (32), suppressed-policy escalation (2), sidecar disagreement (6). The unchanged control passes 201 tests. Mutations ran in disposable copies.
  • Supported example regeneration --check, existing schema/golden/truncation tests and both-format fail-closed conflict checks pass; no reviewed examples were overwritten.
  • Analysis tests block actual network connections, including child Python processes. CI package installation is separate from analysis enrichment.
  • WSL Linux/Python 3.12.3: 473 passed, pip check and example --check pass in a fresh venv with socket guard. Source/temp files are on a Windows-backed mount; pytest used --capture=sys after fd capture failed before collection. This does not establish native POSIX filesystem behavior.
  • Hosted Linux CI: Ubuntu 24.04 / CPython 3.11.17, 473 passed, CLI JSON/Markdown golden comparison passed. All seven active checks passed, including SARIF upload/code scanning and distribution/route validation; four publication-related checks were skipped as configured. No registry or release publication occurred. Other runtime combinations remain unverified.

Changed files:

  • tools/sbom-diff-and-risk/src/sbom_diff_risk/risk.py
  • tools/sbom-diff-and-risk/tests/test_version_decisions.py
  • tools/sbom-diff-and-risk/tests/test_version_decision_reports.py
  • tools/sbom-diff-and-risk/docs/dependency-risk-heuristics.md

Four read-only review roles found no material regressions, security/privacy issues, reliability issues or contract/coverage gaps. Finite adversarial probes do not constitute an exhaustive complexity proof.

Rollback

Revert the three contribution commits together; no migration, data rewrite or configuration rollback is required.

Review hold: Draft; do not merge for at least 24 hours after PR creation. Corrected risk admission can change security-relevant local policy outcomes. Separate merge authorization is required; no auto-merge.

Commits

  • 24c8e462e72ad790bb248977bebb1a945a8a1bd8 — fix(sbom): validate SemVer risk evidence strictly
  • 33744da3e283a77c28e91f6dd8c0824849756fae — test(sbom): compare version risks with an independent oracle
  • 279cb5db07370a2b49c3ff9f8c51374039696b17 — test(sbom): lock policy and report decision projections

Minimum review hold ends 2026-10-21 20:37:04 SGT (UTC+08:00). Keep this PR Draft and unmerged; expiry does not authorize merging.

Signed-off-by: stacknil <219124580+stacknil@users.noreply.github.com>
Signed-off-by: stacknil <219124580+stacknil@users.noreply.github.com>
Signed-off-by: stacknil <219124580+stacknil@users.noreply.github.com>
@stacknil
stacknil marked this pull request as ready for review October 21, 2026 16:33
@stacknil
stacknil merged commit 29203f9 into main Oct 21, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant