refactor(fetch): move web into core, one auto-escalating fetch command

[ankitranjan7]

What this does

Moves all web fetching into the core module (src/fetch/), deletes clis/web, and makes webcmd web fetch walk the whole ladder — plain HTTP → impit → real browser — in one command.

webcmd web fetch --url https://www.g2.com/categories/crm
# Cloudflare-protected. One command. Returns the page. Extraction: browser

Before this PR that sequence was: FETCH_BLOCKED → hint says run web fetch-browser → Site "web" is not installed → dead end.

Why the bugs kept coming back

web fetch and web fetch-browser lived in two different places with two different lifecycles:

	Lived in	Shipped?	Discoverable?
web fetch	hardcoded fast path in main.ts	yes	no
web fetch-browser	adapter in clis/web/	no — clis/ is not in package.json files	yes, in the manifest

So the CLI shipped a command it never listed, and listed a command it never shipped. Every bug below is a symptom of that split, not an independent defect.

Issues fixed

Issue	Was	Now
#247	fetch-browser not installed, so the escalation hint was a dead end	escalation happens inside web fetch; nothing to install
#252	invisible to --help, list, completions, -h	registered in the core registry — all four work
#246	raw Node stack trace on error	standard error envelope, correct exit code
#264	Hacker News → FETCH_BLOCKED (CSP named a CDN)	fetches on the first tier
#283	example.com → FETCH_BLOCKED (server: cloudflare on a 200)	fetches on the first tier

#283 is half-fixed here. Its other half — the unhandled EPIPE that killed the CLI process — was already fixed on main by #265 and is only unreleased. Merging #279 ships it.

The classifier bug (#264 / #283)

isChallengeResponse flattened every response header plus the body into one string and grepped it for cloudflare|captcha|.... Two consequences:

• server: cloudflare on a healthy 200 → "challenge"
• a content-security-policy allow-listing cdnjs.cloudflare.com → "challenge"

Both pages had already returned full usable HTML. The classifier then burned both impit retries and failed with a misleading FETCH_BLOCKED.

Now: header evidence comes only from headers that describe this response (server, cf-mitigated, cf-chl-*, x-datadome*, set-cookie). CSP / report-to / link are excluded — they are third-party allow-lists. Markers are split into two tiers:

• decisive (cf-mitigated, cf-chl, __cf_bm, datadome, "just a moment", "verify you are human") — count at any status, so a real managed-challenge 200 is still caught
• corroborating (cloudflare, akamai, captcha) — need a 403/429/503, so a login page with a reCAPTCHA widget no longer trips it

Command surface

web fetch — the one you want. Escalates on its own.

webcmd web fetch --url <url>                # plain → impit → browser
webcmd web fetch --url <url> --browser false  # stop at HTTP, return FETCH_BLOCKED

web fetch-browser — unchanged, for article export to disk:

webcmd web fetch-browser --url <url> --output ./articles --wait-for "#results tr"

Escalation is local-mode only. Hosted mode executes adapters server-side, so this never silently launches a browser on a hosted user's machine; it returns the error with a hint saying so. Cloud-routing the browser tier through Browser Use is a follow-up PR in webcmd-cloud.

Escalation fires only on FETCH_BLOCKED / FETCH_REQUIRES_BROWSER. A timeout, refused connection, or oversized body still fails fast rather than hiding behind a slow browser run.

Surfaces, before → after

Surface	Before	After
webcmd --help	no web	web under Site adapters
webcmd list	absent	both commands, marked [builtin]
webcmd web fetch -h	ArgumentError + stack trace	real help
--get-completions web ''	empty	fetch, fetch-browser
web fetch --url ftp://x	raw Node stack	error envelope, exit 2
published tarball	shipped no clis/	both tiers in dist/

Also fixed while verifying

• Challenge pages navigate mid-evaluate, destroying the execution context — the normal case on the escalation path, since escalation only happens for pages that blocked us. Both browser paths now retry once after the new document settles. Found by running against a real Cloudflare site, not in review.
• web fetch -f md rendered a nine-column table of the result object while the fast path printed a document. Adds a renderMarkdown hook on CliCommand so a command whose payload is prose owns its markdown. Both paths now emit byte-identical output.
• --format=json (equals form) slipped past the fast-path flag guard and was silently ignored.
• The article-download e2e invoked web read, renamed to web fetch-browser back in 0.5.x. It swallowed the CLI failure and passed vacuously — it had been testing nothing since the rename. All six real sites now exercise the pipeline for real.

Verification

• npm test — 4820 passed, 1 skipped (was 4798; +22 new tests)
• npm run test:e2e article pipeline — 6/6 real sites, genuinely running
• npx tsc --noEmit — clean
• check:hosted-contract, check:package-bin, check:plugin-parity — pass
• check:silent-column-drop, check:typed-error-lint — pass, baselines shrunk (stale clis/web entries removed)
• Live: example.com and news.ycombinator.com fetch on the first tier; g2.com escalates and returns content

Cloud counterpart

webcmd-cloud needs a matching change to load these core-owned commands: agentrhq/webcmd-cloud#33 (draft). Merge order: this PR → release/publish → bump the cloud's pin → un-draft #33.

The currently deployed cloud is unaffected until it bumps its pin: loadDefaultHostedCommands starts from source.manifest, which is [] whenever the package ships no clis/ directory — which it does not.

Manifest note

build-manifest emits core-registered commands with no modulePath/sourceFile — there is no adapter file under clis/ to resolve, and claiming one would point consumers at a path absent from the published tarball. They instead declare packageExport: './fetch/command', a subpath export the package already ships, so a consumer that loads adapters by path resolves it through the package's own export map rather than guessing the dist/ layout.

A test asserts every manifest entry is resolvable — a clis/ path, or a package export that exists and points at a real source file — so a rename cannot silently break hosted execution after publish.

Embedding runtimes

A runtime that drives command functions itself (the hosted cloud executor) must set WEBCMD_EMBEDDED_EXECUTOR=1. Escalation then never self-dispatches.

This was a real bug found by probing, not review: in a worker with no local ~/.webcmd config, shouldUseHostedMode() reports local mode, so a blocked fetch escalated, reached for a local Chromium the worker does not have, and hung ~12s before failing with an opaque BrowserCommandError. With the flag it returns a structured FETCH_BLOCKED in 1.6s.

Supersedes

Closes as superseded: #263 (discoverability), #266 (classifier subset), #271 (packaging), #281 (error formatting), #286 and #287 (both duplicate the classifier fix and were written against a main that no longer exists — both show as CONFLICTING).

🤖 Generated with Claude Code

[github-actions]

🟠 Maintainer review suggested — low confidence

Automated semantic review could not be completed.

Limitations

• A maintainer may need to review documentation requirements manually.

This review is advisory and does not block merging.