Skip to content

Define an explicit CI policy for incomplete discovery evidence #144

Open

Description

@dacharyc

Motivation

A scan can finish with partial discovery or a base-page fallback while warnings leave the CLI exit code successful. A product team may want incomplete discovery to block a scheduled discovery audit while allowing a deliberately curated page-regression run to succeed. Missing evidence is not the same as poor page quality.

Depends on the structured completeness semantics in #141. Do not implement this by matching warning strings. Follow-up to #120/PR #140; not a requirement to expand the bounded-walk fix.

Design Questions

  • Which evidence states can be gated: known truncation, failed acquisition, unknown completeness, unexpected fallback, or a required check skipped for missing evidence?
  • How does a user express which assessment needs that evidence? Curated or single-page workflows intentionally bypass sample discovery and must not fail solely for doing so. Coverage can still be partial during a curated run because it performs independent work.
  • Compare explicit opt-in policy with any proposed default change; document migration and compatibility before choosing a flag, field, or exit code.
  • Define behavior across CLI, programmatic reports, and both Vitest helpers. Keep complete machine-readable output available when the policy causes a nonzero exit.
  • Coordinate with existing single-page diagnostics, bot-protection degradation, check failures, and scoring. Do not silently translate unavailable evidence into a failed page check or a new numeric penalty.

Acceptance Criteria

  • Policy evaluation consumes Expose structured discovery completeness and fallback evidence in reports #141 metadata, with documented precedence for check failure, incomplete evidence, warnings, and intentional exclusions.
  • The same effective policy has equivalent semantics across supported entry points; invalid configuration fails before network work.
  • Tests distinguish intentional curated/none selection from unexpected fallback and distinguish page sampling from an independently incomplete coverage walk.
  • Tests cover budget stops, source failures, unknown evidence, dependency skips, and reports preserved on CI failure.
  • Existing behavior is preserved by default unless an explicit reviewed migration changes it. Any scoring changes follow the repository scoring-version procedure.
  • Include examples of a curated regression job and a stricter discovery audit, explaining their different evidence requirements.

References

#141 is the prerequisite. Related: #120, PR #140, #23, #73, #104 and the Documentation at Scale design notes in working-notes/page-discovery-notes.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions