Skip to content

feat(core): scans run under a wall-clock budget (#56) - #58

Merged
k2kirov merged 2 commits into
mainfrom
fix/scan-budget
Sep 3, 2026
Merged

feat(core): scans run under a wall-clock budget (#56)#58
k2kirov merged 2 commits into
mainfrom
fix/scan-budget

Conversation

@k2kirov

@k2kirov k2kirov commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

What

Closes #56. A scan now runs under a wall-clock budget.

  • runScan reads timeoutMs (default SCAN_TIMEOUT_MS, raised from 60 s to 180 s: it clears the corpus p95 of 109 s, where 60 s would have cut one site in five). 0 disables it; negative or NaN throws RangeError.
  • Every request answers to the caller's signal and to the budget. Only the caller's signal ends the scan; the budget ends the fetching. A 429 retry wait ends with the budget and keeps the observed 429.
  • When the budget runs out, requests in flight abort, no further request is sent, and every audit not yet started, or still running, reports na tagged skipped:scan-budget. A running audit is withheld rather than believed: a refused request reads to it as a broken link or a missing artifact.
  • conditions.budget records limitMs, elapsedMs, exhausted, skippedCount. The budget's cut counts with the gate's against the 0.35 unscored threshold; the reason names the budget.
  • Audit trace outcome budget. formatBudget, budgetReason, skippedMassShare exported from core.
  • CLI --timeout <seconds> (config key timeout, on AgentLighthouseConfig). Bare, negative and non-numeric values from the flag or the file are refused. One terminal line when the budget went. Markdown and HTML reports show the budget beside the other conditions.
  • Docs: docs/cli.md, docs/config.md, docs/evidence/corpus.md, nightly workflow comment.

Measured

twilio.com, the 1054 s outlier, scanned in 117 s with per-audit tracing on 2026-09-03; the slowest audit was machine-discovery/llms-txt-links-valid at 54 s. With --timeout 30 the same scan stopped at 30.1 s, withheld 145 audits (2 of them mid-run) and reported no score with the budget named.

Review

Two reviewer rounds. Blockers found and fixed: a budget expiring during a 429 backoff threw instead of returning a report; audits still running at the abort turned unsent requests into fail (no-broken-links); the 429 path reported a throttling site as unreachable; the retry's abort race was dead on an already-aborted signal. Tests drive the cut from mocked slow audit-phase fetches with an abort-aware mock; the call-count comparison bypasses the origin cache.

Changeset

minor for core, cli, report (4.1.0), per the release instruction. Note: a scan of a slow origin that used to return a score after many minutes now returns overallScore: null at 180 s unless --timeout is raised; the changeset says so.

Gate: build, test (5350), typecheck, lint, check:dossiers, check:requires, check:audit-map, format:check all green.

https://claude.ai/code/session_01RT21vyEPL6uSMWwUTytw6p

`SCAN_TIMEOUT_MS` was declared and never read; the only bound on a scan was
the 10 s per request, so a slow origin that answered every request stretched
a scan without end (twilio.com: 1054 s in the first corpus pass).

`runScan` now creates a budget from `timeoutMs` (default `SCAN_TIMEOUT_MS`,
raised from 60 s to 180 s, which clears the corpus p95 of 109 s). Every
request answers to the caller's signal and to the budget; only the caller's
signal ends the scan. When the budget runs out, requests in flight abort, no
further request is sent, and every audit not yet started, or still running,
reports `na` tagged `skipped:scan-budget`. A running audit is withheld
because a refused request reads to it as a broken link or a missing
artifact: a claim about the clock, not the site. A 429 retry wait ends with
the budget and keeps the observed 429.

`conditions.budget` records limit, elapsed, exhausted and skipped count. The
budget's cut counts with the gate's against the 0.35 unscored threshold. The
CLI gains `--timeout <seconds>` (config key `timeout`), refuses bare,
negative and non-numeric values, and prints one line when the budget went.
Markdown and HTML reports show the budget beside the other conditions.

Closes #56.

Claude-Session: https://claude.ai/code/session_01RT21vyEPL6uSMWwUTytw6p
@k2kirov
k2kirov merged commit c1a9976 into main Sep 3, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scans have no wall-clock budget: twilio.com took 1054 s

1 participant