Design scan-wide discovery budgets and reusable sitemap acquisition #145

Motivation

PR #140 bounds each sitemap walk, but page discovery and coverage can acquire the same sitemaps in separate walks, each with its own budget. Parallel product jobs also do not share a rate limiter. Raising per-walk limits would increase work without establishing representative product coverage.

Investigate scan-wide accounting and bounded acquisition reuse before increasing defaults. This is separate from #120/PR #140 and from a hard streaming response-size cap.

Design Scope

• Inventory existing sample and HTTP caches and identify which sitemap acquisition can be reused within one CheckContext. Separate acquired raw data from path-filtered, locale-refined, version-deduplicated samples.
• Define whether budgets count HTTP attempts, retries, redirects, decoded bytes, elapsed time, or combinations, and at which layer they are enforced. Do not equate a sitemap helper call with a wire request.
• Preserve correct behavior across effective/canonical origins, body rewriting, explicit source selection (Support explicit product-specific sitemap selection across discovery and coverage #142), differing path scopes, and raw coverage versus refined discovery.
• Define cache keys, memory/entry limits, partial-read and failure behavior, cancellation, and reporting of reused versus newly fetched evidence. Sharing a cached subset must not mark it complete.
• Clarify that per-run accounting cannot automatically throttle separate processes. Document aggregate-load guidance; distributed coordination is not assumed to be part of the initial implementation.

Acceptance Criteria

• A repeated discovery/coverage operation can reuse compatible acquired sitemap evidence without another network fetch, while differing scopes/refinement preserve correct results.
• Request-ledger attribution and aggregate counts reflect real requests; reuse does not manufacture duplicate successes or hide earlier truncation/failures.
• Budget exhaustion and cancellation stop the intended work and are exposed through Expose structured discovery completeness and fallback evidence in reports #141 metadata. Cached evidence retains its original scope and completeness.
• Tests cover canonical rewriting, raw versus refined consumers, incompatible cache keys, bounded memory, concurrent acquisition, failures, and cancellation.
• No per-discovered-page probes, increased traversal depth, or silent changes to sample composition or coverage denominators. Any intentional compatibility changes are documented.
• Record the chosen accounting units, tradeoffs, and whether budgets are per operation, scan, or process. Do not promise cross-job coordination.

References

#120, PR #140, #141, #142. See working-notes/page-discovery-notes.md, Documentation at scale design considerations. Hard per-response streaming limits require a separate HTTP-layer design.