feat(cli): support ignoring external hosts

[claude]

Requested by Alex Duke · Slack thread

Relates to invertase/react-native-firebase#9214

Summary

Before: Stack Overflow, npm and other bot-gated hosts answer an automated client with a 403 instead of the page, so docs check reported perfectly good links as broken and failed CI. The only escape hatch was --external-links warn or --external-links off, which turns the external link check off for the whole project. There was no way to say that one particular host is fine and should stop being checked.

After: when a host refuses the request — 401, 403, 405 or 429 — the link is reported as a warning saying the host rejected an automated request and the link was not verified, so a bot gate no longer fails CI. Genuinely dead links (404, 5xx, DNS failures, timeouts) stay errors. Hosts that are permanently unfriendly can be listed once, with --ignore-external-hosts or check.ignoreExternalHosts in docs.json, and are skipped before any request is made.

In one sentence: this makes docs check usable in CI against docs that link to bot-gated hosts, without giving up external link checking entirely.

How

checkExternalUrl returns a classified failure (unverified or broken) instead of a bare message, and resolveExternalIssueSeverity maps unverified to warn while leaving everything else on the configured --external-links severity — off still skips the requests entirely, and an explicit warn is never upgraded. GET is treated as the authoritative attempt because many hosts do not implement HEAD. The ignore list is read from the docs.json object the check command already parses and unioned with the flag value; both entries and targets are normalised to a bare lowercase hostname, then matched by exact equality or a dot-boundary suffix, never by substring matching on the URL, so npmjs.org covers www.npmjs.org but not evil-npmjs.org.attacker.net. Skipped links print as skip lines and are counted in the summary rather than being silently dropped.

Commits

• 72411f9 fix(cli): treat bot-gate responses as warnings, not errors
• e41b551 feat(cli): add --ignore-external-hosts option
• 74460c7 feat(app): accept check.ignoreExternalHosts in the docs.json schema

Scope

• app/ (hosted site, MCP, Ask AI)
• packages/cli/
• packages/mdx-bundler/
• docs/ (product documentation)
• Repo / CI / other

Type of change

• Bug fix
• New feature
• Documentation
• Refactor / chore

Test plan

Adds packages/cli/src/commands/check.test.ts, the first coverage on the external link checker: host normalisation (case, whitespace, *. prefix, leading and trailing dots, a pasted URL, host:port, empty and garbage entries), matching (exact host, subdomain, the evil-npmjs.org.attacker.net negative case, unparsable URLs, empty list), the flag plus config union, and the severity mapping for 401/403/405/429 against 404/410/500/503 and an unreachable host, using a mocked fetch so no test touches the network.

Also exercised by hand against a scratch project with links to an ignored host, an ignored subdomain, a lookalike host and an unlisted host, checking the skip lines, the skipped count, the warning wording, and that --external-links off still skips everything.

• bun run check passes locally
• Tested locally (bun test, bun run src/cli.ts check ..., --help)
• Updated docs/ (if user-facing)
• Verified on a docs.page URL or local preview (if rendering/routing changed)

Each commit in the list above was checked out on its own and passes biome ci . and bun test.

Notes for reviewers

• The browser user-agent commit was dropped from this branch. An earlier revision also replaced the checker's docs.page-cli user agent with a desktop Chrome string. Mike Hardy ran the branch against the react-native-firebase docs and found it net-negative: npm and Stack Overflow return 403 to every user agent, so the bot-gate warn behaviour is what actually fixes those links, while the Chrome string made developer.android.com bounce into an OAuth sign-in redirect loop and developers.facebook.com return 400 — six false errors on valid links. Elliot Hesp approved dropping it, so check now sends exactly the user agent it sent before this PR, unchanged from main.
• The app config schema now carries the key too. app/src/server/config/schema.ts accepts check.ignoreExternalHosts, added at the review request from Mike Hardy so the CLI key and the schema do not skew apart, and so the key gets editor autocomplete and validation rather than only being tolerated as an unknown key.
• Merging deploys the schema change, but does not publish the CLI. The app/ schema addition goes out with the app; @docs.page/cli still ships only on a version bump plus a cli-v* tag push, so the CLI side reaches users on the next release, not on merge.
• Where this came from. Mike Hardy reported that docs check fails on valid Stack Overflow and npm links; this unblocks ci(docs): add docs.page link check to docs workflow react-native-firebase#9214.
• Judgement call worth a second opinion: a leading dot on an ignore entry is stripped, so .npmjs.org behaves like npmjs.org. That reads as the obvious intent, but it also means someone who writes .com ignores every .com host. Happy to reject leading-dot entries instead.
• Judgement call: 405 is included in the bot-gate set. A host that answers 405 to a GET is refusing the method rather than reporting a missing page, but it is the least clear-cut of the four.
• The packages/cli typecheck (bunx tsc --noEmit) is red on main already: bun:test has no types in that package tsconfig, and there are three pre-existing errors in packages/mdx-bundler. The new test file adds one more missing-module line for bun:test, of exactly the same pre-existing kind. Fixing that needs a tsconfig or devDependency change I left out of scope. There are no new errors of any other kind.

[docs-page]

To preview the documentation for this pull request, visit the following URL:

use.docs.page/~544

Documentation is deployed and generated using docs.page

[railway-app]

🚅 Deployed to the docs.page-pr-544 environment in docs.page

Service	Status	Web	Updated
docs.page	✅ Success (View Logs)	Web	Sep 2, 2026 at 8:11 am UTC

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[mikehardy]

The docs.json schema entry is deliberately deferred. This PR is CLI-only. app/src/server/config/schema.ts is untouched, so check.ignoreExternalHosts is an unknown key to the hosted app — zod strips unknown keys, so nothing breaks. The schema addition only buys editor autocomplete and validation, and belongs in a follow-up PR.

I disagree with this. If we're adding a new key and we have a schema, then the schema should be updated to have the key otherwise there's skew. And deferring it until a follow-up just adds future mental labor for something that should be done anyway.

Judgement call worth a second opinion: a leading dot on an ignore entry is stripped, so .npmjs.org behaves like npmjs.org. That reads as the obvious intent, but it also means someone who writes .com ignores every .com host. Happy to reject leading-dot entries instead.

I'm okay with this.

Judgement call: 405 is included in the bot-gate set. A host that answers 405 to a GET is refusing the method rather than reporting a missing page, but it is the least clear-cut of the four.

Fine

The packages/cli typecheck (bunx tsc --noEmit) is red on main already: bun:test has no types in that package tsconfig, and there are three pre-existing errors in packages/mdx-bundler. The new test file adds one more missing-module line for bun:test, of exactly the same pre-existing kind. Fixing that needs a tsconfig or devDependency change I left out of scope. There are no new errors of any other kind.

Sounds like a great peel-off to a new issue or PR... if there is a typecheck, and it's failing on main, then there is A) a CI problem and B) a types problem. Address both

behind a proxy or WAF that answers 403 for blocked hosts, every dead link now downgrades to a warning — inherent to the bot-gate change, and I hit it in this sandbox

I'm also okay with this - CI environments likely won't have this happen, dev environments will see the warning and can reason about it

[mikehardy]

External link User-Agent analysis (RNFB integration testing)

Context: This analysis was run while integrating @docs.page/cli check into invertase/react-native-firebase (draft PR #9214). Goal: drop --external-links warn and rely on PR 544’s bot-gate handling while keeping real 404s (especially reference.rnfirebase.io) as errors.

CLI under test: packages/cli from branch feat/cli-check-external-link-exceptions. Local clone built at commit dec2272a3c73d0b4e2e31f7f10ae8fecde182e4a; current PR head at time of writing is f85b3df7bcff661f025cf243705138f801afedcf — EXTERNAL_LINK_USER_AGENT and BOT_GATE_STATUSES are unchanged on head.

Environment: macOS, Node v24.13.0, Bun 1.3.14. Network checks against live hosts (not mocked).

---

Methodology

1. Understand the checker’s decision logic

From packages/cli/src/commands/check.ts (PR branch):

1. checkExternalUrl(url) issues HEAD, then GET if HEAD is not OK.
2. Uses fetch(url, { redirect: "follow", headers: { "user-agent": EXTERNAL_LINK_USER_AGENT } }).
3. OK = HTTP status in [100, 300).
4. If final status ∈ {401, 403, 405, 429} → classified unverified → reported as warn (does not fail exit code when default severity is error).
5. Any other non-OK status (404, 400, 5xx, …) or fetch exception → classified broken → reported as error.
6. GET is authoritative when both HEAD and GET return a status; HEAD is only used when GET produces no status.

This means the outcome for a URL is fully determined by: User-Agent string, HEAD→GET sequence, redirect handling, and status classification — not by curl alone unless curl mirrors that sequence.

2. User-agents compared

ID	User-Agent string	Role
pr544	Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36	Current EXTERNAL_LINK_USER_AGENT on this PR
bot	Mozilla/5.0 (compatible; docs.page-cli/2.0.0; +https://docs.page)	Prior @docs.page/cli identity string
googlebot	Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)	Additional probe (secondary)

Also probed with Firefox, Safari, and default curl/8.7.1 UA on a subset (see §5).

3. Primary experiment — replicate checker logic per URL

A Node script reproduced checkExternalUrl exactly (HEAD then GET, same timeout, same BOT_GATE_STATUSES, same severity mapping) while swapping only the User-Agent. This is the “docs.page check” column below — it is not inferred from curl.

Additionally, for each URL:

• curl GET with pr544 and bot UAs (curl -A … -L --max-time 20)
• curl HEAD with pr544 UA
• Direct invocation of exported checkExternalUrl() from the built CLI source via Bun (same results as the Node replica)

4. Full-corpus run

Ran the built CLI against the full RNFB docs tree:

node packages/cli/dist/cli.js check .
# cwd: invertase/react-native-firebase worktree (docs-page-link-check-ci branch)

Captured all error and warn lines from stdout.

5. Supplementary probes

• Redirect trace (no follow): curl -I -A <ua> on developer.android.com URLs to inspect first-hop response without following redirects.
• Broader UA matrix: Node fetch GET+HEAD for pr544, bot, googlebot, Firefox, Safari, default curl UA on 5 representative URLs.
• Docs URL sample: First 80 unique https://… strings extracted from docs/**/*.mdx via ripgrep; same checker logic applied with pr544 vs bot — counted outcome differences.

---

Results — primary URL table (checker logic + curl)

URL	curl GET pr544	curl GET bot	docs.page check (PR 544 logic, pr544 UA)	docs.page check (same logic, bot UA)
https://developers.facebook.com/docs/android/getting-started/	400	200	error — 400 Bad Request	pass
https://ai.google.dev/gemini-api/terms#grounding-with-google-maps	redirect loop (curl exit 47)	200	error — Unable to reach external link: fetch failed	pass
https://developer.android.com/identity/sign-in/credential-manager-siwg-implementation	redirect loop	200	error — fetch failed	pass
https://developer.android.com/topic/performance/vitals/anr	redirect loop	200	error — fetch failed	pass
https://developer.android.com/studio/build/multidex#mdex-gradle	redirect loop	200	error — fetch failed	pass
https://developer.android.com/reference/java/util/concurrent/ThreadPoolExecutor	redirect loop	200	error — fetch failed	pass
https://stackoverflow.com/questions/5025256/how-do-you-specify-command-line-arguments-in-xcode-4	403	403	warn — bot gate (403)	warn — bot gate (403)
https://www.npmjs.com/package/react-native-nitro-google-signin	403	403	warn — bot gate (403)	warn — bot gate (403)
https://github.com/mikehardy/rnfbdemo/blob/main/make-demo.sh	429 (this run)	200	pass (HEAD 200)	pass
https://reference.rnfirebase.io/_react-native-firebase/auth/AuthSettings.html	200	200	pass	pass
https://reference.rnfirebase.io/_react-native-firebase/auth/AuthSettings.html#appverificationdisabledfortesting	200	200	pass	pass
https://reference.rnfirebase.io/_react-native-firebase/does-not-exist.html (synthetic)	404	404	error — 404 Not Found	error — 404 Not Found

Observations from this table (facts):

• All 6 URLs that fail with pr544 UA are Facebook or Google developer/AI hosts.
• npm and Stack Overflow return 403 with both UAs; with PR 544’s status rules they are warn, not error, regardless of UA.
• TypeDoc (reference.rnfirebase.io) passes for valid URLs and errors on 404 with both UAs.
• Switching only the UA to bot clears all 6 Facebook/Google errors while leaving npm/SO as warn and TypeDoc 404 detection unchanged.

---

Results — full RNFB docs check . run (PR 544 CLI, pr544 UA)

Summary line: 6 errors, 17 warnings (one subsequent run reported 7 errors, 14 warnings — the extra error was an intermittent GitHub 429 on an unrelated URL; core Facebook/Google error set is stable).

All errors (pr544 UA):

File	Message	URL
docs/auth/social-auth.mdx:111	400 Bad Request	https://developers.facebook.com/docs/android/getting-started/
docs/auth/social-auth.mdx:213	fetch failed	https://developer.android.com/identity/sign-in/credential-manager-siwg-implementation
docs/ai/usage/index.mdx:469	fetch failed	https://ai.google.dev/gemini-api/terms#grounding-with-google-maps
docs/crashlytics/crash-reports.mdx:55	fetch failed	https://developer.android.com/topic/performance/vitals/anr
docs/enabling-multidex.mdx:14	fetch failed	https://developer.android.com/studio/build/multidex#mdex-gradle
docs/index.mdx:406	fetch failed	https://developer.android.com/reference/java/util/concurrent/ThreadPoolExecutor

Warning pattern: 14–17 warnings, overwhelmingly:

• 403 Forbidden on npmjs.com and stackoverflow.com (classified unverified — working as designed)
• Intermittent 429 Too Many Requests on github.com raw/tree URLs (also classified unverified)

No TypeDoc 404 errors appeared in either run.

Projected outcome with bot UA + same status rules: 0 errors from the stable Facebook/Google set above; warnings unchanged.

---

Results — redirect trace (why developer.android.com fails with pr544 UA)

curl -I (no redirect follow) on https://developer.android.com/topic/performance/vitals/anr:

With pr544 UA — first response:

HTTP/2 302
location: https://developer.android.com/oauth2authorize?return_url=...&prompt=none&auto_signin=True&scopes=...
set-cookie: signin=autosignin; ...

Node fetch with redirect: "follow" enters this OAuth auto-sign-in chain and eventually aborts → fetch failed → error.

With bot UA — first response:

HTTP/2 200
content-type: text/html; charset=utf-8
content-length: 320399

Page content served directly; checker passes.

---

Results — supplementary UA matrix (Node fetch GET | HEAD)

URL	pr544	bot	googlebot	firefox	safari
developers.facebook.com/.../getting-started/	GET 400 | HEAD 400	GET 200 | HEAD 200	GET 200 | HEAD 200	GET 400 | HEAD 400	GET 400 | HEAD 400
developer.android.com/.../anr	GET err | HEAD err	GET 200 | HEAD 200	GET 500 | HEAD 200	GET 200 | HEAD 200	GET 200 | HEAD 200
ai.google.dev/gemini-api/terms#...	GET err | HEAD err	GET 200 | HEAD 200	GET 200 | HEAD 200	GET 200 | HEAD 200	GET 200 | HEAD 200
npmjs.com/package/react-native-nitro-google-signin	403 | 403	403 | 403	403 | 403	403 | 403	403 | 403
stackoverflow.com/questions/5025256/...	403 | 403	403 | 403	403 | 403	403 | 403	403 | 403

Observation: Browser-like UAs (Chrome, Firefox, Safari) do not improve npm/SO outcomes vs bot UA. Google/Facebook doc hosts specifically treat Chrome UA differently from bot UA on this corpus.

---

Results — 80-URL docs sample (outcome diff pr544 vs bot)

Extracted first 80 unique https://… URLs from docs/**/*.mdx. Applied checker logic with pr544 vs bot UA:

• Same outcome: 80 / 80
• Chrome UA strictly better: 0
• Bot UA strictly better: 0

The sample did not include the failing Facebook/Google URLs (they appear later in alphabetical / extraction order). Failures are concentrated on a small set of Google/Facebook doc URLs (6 distinct URLs, 6 error lines in full run), not spread across the corpus.

---

Evidence summary (no prescription)

Claim	Evidence
PR 544’s 401/403/405/429 → warn correctly handles npm/SO	Both UAs get 403; checker reports unverified/warn; exit 0
Chrome UA does not reduce npm/SO 403 rate vs bot UA	UA matrix: all browser-like UAs → 403 on npm/SO
Chrome UA introduces errors on Google/Facebook docs	6 stable errors in full RNFB run; primary table shows bot UA passes same URLs
TypeDoc 404 detection works with bot UA	Synthetic 404 on reference.rnfirebase.io errors with both UAs; valid TypeDoc URLs pass with both
Failures are OAuth redirect loops / 400, not missing pages	Redirect trace shows 302 → oauth2authorize with pr544 UA; 200 with bot UA

---

Recommendation (inference from above evidence)

Revert EXTERNAL_LINK_USER_AGENT to the prior bot identity (Mozilla/5.0 (compatible; docs.page-cli/2.0.0; +https://docs.page)) and keep the BOT_GATE_STATUSES / unverified → warn behavior from this PR.

Reasoning chain:

1. The stated motivation for Chrome UA was npm/Stack Overflow bot gates — but those hosts return 403 for all tested UAs; PR 544’s status-based downgrade already handles them without UA spoofing.
2. Chrome UA causes Google developer.android.com to enter OAuth auto-sign-in redirect loops and Facebook to return 400 — both classified as hard errors, producing false positives on legitimate documentation links.
3. Bot UA passes those same URLs, preserves 404 detection on reference.rnfirebase.io, and leaves npm/SO as non-blocking warnings.

Not recommended based on this evidence: --ignore-external-hosts for developer.android.com / developers.facebook.com / ai.google.dev — that would also suppress real 404s on those hosts. Host ignore lists and --external-links warn were workarounds for the pre-544 world; this data suggests UA choice is the simpler fix.

Happy to re-run against a specific commit or add CI-matrix reproduction steps if useful.

[mikehardy]

Works great on a test integration

[Ehesp]

@claude could a lot of this logic in the file not be put into the zod transformer?

[claude]

Good instinct, and you are right that this block does not belong in check.ts — but a zod transform cannot get us there today, because the CLI never parses docs.json through zod at all. packages/cli/src/lib/docs-config.ts does a raw JSON.parse into an untyped record, validateDocsConfig only checks JSON/YAML syntax, and the CLI has no zod dependency (the app's ConfigSchema lives in the private app workspace, so a published npm package cannot import it).

• What genuinely does not belong here: readConfigIgnoredHosts, normalizeIgnoredHost, splitIgnoreEntries and the two host helpers, lines 805-919. Right call to pull them out — but as a plain move to packages/cli/src/lib/, not into a schema. About 115 lines out, one import back, all 22 tests keep passing unchanged.
• What cannot move: --ignore-external-hosts is a Commander string that zod never sees, so the normaliser has to stay directly callable; isIgnoredHost normalises the target URL hostname with the same helper, so both sides of the comparison need it; and the flag plus docs.json union, where neither source replaces the other, is CLI policy rather than config shape.
• App-side transform: I checked it. Adding .transform() to models/check.ts leaves schema.json byte-identical, because both toJSONSchema call sites use io: "input" and transforms are invisible there. It would improve the inferred type to string[], but nothing in the app reads config.check and the CLI would not see it. Worth knowing that io: "output" renders the field as an empty object and marks it required, so that safety rests on io: "input" staying put.

Recommendation: do the plain extraction to lib/, skip the zod transform. The version of your idea that really pays off is making the CLI validate docs.json through the same schema as the app, which would also kill parseConfigObject and the duplicated headerDepth default — but that needs a shared publishable config package and zod in the CLI bundle, so I would rather it be its own PR than get bolted on here.

Mike has approved the current head, so either works: I can push the extraction now, or land as-is and follow up. Your call.

---

Generated by Claude Code