Define optional enforced request boundaries separately from documentation scope #146

Motivation

The base URL path and locale/version preferences filter discovered page candidates, but they are not a network allowlist. Curated entries are used directly. Index link checks, Markdown candidates, redirects, and coverage can request other URLs.

Product teams scanning shared hosts may need an explicit request boundary in addition to an assessment scope. This issue investigates and implements that contract without implying the current base URL already provides one.

Design Questions

• Specify the guarantee: a URL-level request policy, an HTTP-layer enforcement boundary, or a stronger network security property. Do not market URL filtering as complete SSRF protection or a process sandbox.
• Define matching and precedence for origins, paths, query strings, aliases, curated entries, selected index sources, and exclusions. Reuse matching/validation only where its existing semantics are appropriate.
• Decide how redirects are checked before following them, including cross-host and same-host path escapes. Account for canonical-origin rewriting and intentionally local/preview targets.
• Resolve whether conventional Markdown candidates, root llms.txt/robots/sitemaps, linked resources, and check-specific probes require explicit allowances. Do not silently exempt a check category.
• Distinguish policy-denied work from network errors, absent content, successful checks, and incomplete acquisition. A blocked request must not silently turn missing support into a pass.

Acceptance Criteria

• The opt-in contract and default behavior are documented before choosing configuration syntax. Existing scopes retain their current meaning when no boundary is configured.
• Enforcement is centralized at the layer needed for the chosen guarantee; code paths cannot bypass it accidentally.
• Tests cover curated and discovered inputs, source selection, Markdown candidates, redirects, localhost/canonical rewriting, and conflicting or invalid rules.
• Request assertions demonstrate that denied targets are not fetched. Results and Expose structured discovery completeness and fallback evidence in reports #141 metadata expose intentional restrictions and their impact on evidence.
• Public HTTP clients and custom clients have a documented compatibility/enforcement contract; guarantees do not exceed what can be enforced.
• No extra page classification probes or deeper traversal. Document cases requiring broader allowances without broadening them automatically.

References

#120, PR #140, #141, #142, #143, #40. Keep this separate from #138: page/non-page classification decides candidate eligibility, whereas this issue decides whether a request is permitted. Markdown URL mapping is a third independent concern.