Repository navigation
Define an explicit CI policy for incomplete discovery evidence #144
Copy link
Copy link
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Motivation
A scan can finish with partial discovery or a base-page fallback while warnings leave the CLI exit code successful. A product team may want incomplete discovery to block a scheduled discovery audit while allowing a deliberately curated page-regression run to succeed. Missing evidence is not the same as poor page quality.
Depends on the structured completeness semantics in #141. Do not implement this by matching warning strings. Follow-up to #120/PR #140; not a requirement to expand the bounded-walk fix.
Design Questions
- Which evidence states can be gated: known truncation, failed acquisition, unknown completeness, unexpected fallback, or a required check skipped for missing evidence?
- How does a user express which assessment needs that evidence? Curated or single-page workflows intentionally bypass sample discovery and must not fail solely for doing so. Coverage can still be partial during a curated run because it performs independent work.
- Compare explicit opt-in policy with any proposed default change; document migration and compatibility before choosing a flag, field, or exit code.
- Define behavior across CLI, programmatic reports, and both Vitest helpers. Keep complete machine-readable output available when the policy causes a nonzero exit.
- Coordinate with existing single-page diagnostics, bot-protection degradation, check failures, and scoring. Do not silently translate unavailable evidence into a failed page check or a new numeric penalty.
Acceptance Criteria
- Policy evaluation consumes Expose structured discovery completeness and fallback evidence in reports #141 metadata, with documented precedence for check failure, incomplete evidence, warnings, and intentional exclusions.
- The same effective policy has equivalent semantics across supported entry points; invalid configuration fails before network work.
- Tests distinguish intentional curated/none selection from unexpected fallback and distinguish page sampling from an independently incomplete coverage walk.
- Tests cover budget stops, source failures, unknown evidence, dependency skips, and reports preserved on CI failure.
- Existing behavior is preserved by default unless an explicit reviewed migration changes it. Any scoring changes follow the repository scoring-version procedure.
- Include examples of a curated regression job and a stricter discovery audit, explaining their different evidence requirements.
References
#141 is the prerequisite. Related: #120, PR #140, #23, #73, #104 and the Documentation at Scale design notes in working-notes/page-discovery-notes.md.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity