Skip to content

test(sbom): qualify evidence-policy decision stability - #121

Draft
stacknil wants to merge 1 commit into
mainfrom
stacknil/sbom-evidence-decision-stability
Draft

stacknil wants to merge 1 commit into
mainfrom
stacknil/sbom-evidence-decision-stability

Conversation

@stacknil

Copy link
Copy Markdown
Owner

Problem

Equivalent dependency representations could change explicit evidence-policy decisions, while malformed policy or provider evidence could cross validation boundaries. This change adds a reproducible, offline decision-stability qualification and the narrowly justified repairs it exposed.

Invariants

Canonical identity, observed metadata, recorded evidence, policy authorization and consumer decisions remain distinct. Representation-only equivalence is STABLE; changes to active rules or relevant evidence can produce JUSTIFIED_REVERSAL. Contradictory identities remain invalid input. Unknown evidence never becomes proof of authenticity or package safety.

Design decisions

  • A standard-library-only expected-value module specifies 72 literal policy microcases independently of the production evaluator. The campaign covers 55 transitions and fourteen reached negative controls, N01–N14.
  • C02 uses PEP 503 package-name equality for PyPI missing-attestation exceptions, preserving name-level scope and the unavailable/unverified boundaries.
  • C03 uses canonical ecosystem/name/version for explicitly enabled PyPI lookup and cache isolation, including valid PURL-only versions. Both CLI input indexes are validated before either provider starts; direct/batch PyPI APIs validate identities before lookup. Observed fields remain unchanged and versionless components never request latest.
  • C04 binds explicit Scorecard repository identity using ASCII-only case equivalence. Policy parsing rejects Boolean integer fields/non-string keys; Scorecard parsing rejects Boolean/nonfinite scores.
  • Tests check JSON, summary, policy sidecar, Markdown, the supported SARIF subset and exit-code consistency, including post-serialization evidence against literal expectations.

Failure model and rejected alternatives

The regressions cover contradictory/duplicate inputs, cache contamination, omitted violations, severity/suppression corruption, unavailable/missing/unverified confusion, malformed provider evidence and misleading report agreement. Universal alias matching, Unicode repository alias folding, latest-release inference and a new unknown-score blocking policy were rejected. A failed cache experiment based on an invalid distinct-release assumption is retained as a rejected oracle hypothesis, not a production defect.

Compatibility and non-claims

Existing policy/report schema identifiers, risk taxonomy, strict SemVer classification and offline defaults remain unchanged. No dependency or workflow changes are included. Invalid Scorecard evidence follows existing error/unknown handling; an available-score-only policy can therefore pass without usable score evidence. This is not a vulnerability scanner or package safety certification. No live PyPI/Scorecard enrichment or production PyPI publishing was exercised. The qualification's network-denial audit recorded zero actual attempts; explicitly enabled production lookup remains the existing opt-in behavior.

Validation

  • Baseline: 473 passed.
  • Final candidate: Windows CPython 3.12, 817 passed × seeds 103/117/120; native Linux/ext4 CPython 3.11.17, 817 passed, plus targeted reruns.
  • N01–N14 detected; golden artifacts passed; network-denial audit: 0 actual attempts.
  • Changed-file Ruff correctness checks passed. Full Ruff output exactly matches the baseline's existing F821 in test_policy_scorecard.py.
  • Original signed/DCO commit: dce5082e3f943c0e796e74607aa65c02cf86f10c; 30-file frozen diff; semantic tree 0760a5543df366f3a8237c60a187cde7f3aae0b0.

The owner supplied the external disposition ACCEPT_FOR_LOCAL_COMMIT, conditional on exact source identity, after targeted independent execution and review for the defined scope. That external review was not an independently executed rerun of the entire 817-test suite. Frozen qualification documents retain their historical audit-pending checkpoint; the later external disposition is preserved separately. Private transcripts, machine receipts and the private audit package are not part of this PR. Hosted CI is a separate integration gate and will be assessed from actual jobs, not inferred from local results.

Rollback and review

Revert this single contribution to remove the qualification and bounded repairs; no schema migration or data conversion is required. This is a draft because the coherent qualification spans 30 files. It does not close or claim completion of #103 or #117. Merge requires a separate decision; no merge or release is authorized by this integration task.

Signed-off-by: stacknil <219124580+stacknil@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant