[warp-agent-staging]

Problem

Every docs page has a Markdown twin at /<path>.md (homepage /index.md), advertised on every HTML page with <link rel="alternate" type="text/markdown">. On production (checked 2026-11-04):

• curl -sI https://docs.warp.dev/factories.md returns 200 text/markdown with no Link header, so nothing tells a crawler that the twin duplicates https://docs.warp.dev/factories/.
• The twin body starts with a blockquote preamble (> For the complete documentation index, see [llms.txt]...) and has no YAML frontmatter. ora check markdown-frontmatter = 0/1.
• llms-full.txt and llms-small.txt list https://docs.warp.dev/404/ as a documentation page.

Change

1. Canonical Link header on generated twins only (vercel.json is unchanged):
   • scripts/build-markdown-canonical-routes.mjs runs in postbuild, after scripts/build-tracking-middleware.mjs, and writes header routes into .vercel/output/config.json before the filesystem phase. A twin is <path>.md whose HTML page <path>/index.html advertises it with <link rel="alternate" type="text/markdown" href="/<path>.md">; 409 twins today.
   • /index.md → Link: <https://docs.warp.dev/>; rel="canonical". Every other twin → Link: <https://docs.warp.dev/<path>/>; rel="canonical". The twin paths are grouped into 5 exact alternation routes (each src under 3500 characters), all caseSensitive: true and continue: true.
   • Other .md requests (/assets/x.md, /_astro/x.md, /.well-known/x.md, /robots.md, /AGENTS.md, or any missing .md) return 404 with no Link header. HTML pages, /llms.txt, /openapi.*, and sitemaps get no Link header either.
   • Route building and matching live in src/lib/markdown-canonical-routes.js. scripts/markdown-canonical-headers.test.mjs (npm run test:markdown-canonical in CI) tests exact matches, reserved paths, missing files, near misses, case variants, regex characters, and route splitting. It also asserts that vercel.json sets no Link header. The test needs no Vercel packages.
2. YAML frontmatter on every twin (src/integrations/docs-markdown-integration.js): the first bytes are --- with title, description (omitted when the page has none), and canonical (https://docs.warp.dev/<path>/, homepage https://docs.warp.dev/). Values are written as JSON strings, which YAML reads as double-quoted scalars, so colons, quotes, #, and leading - are safe. The body is otherwise unchanged. Unit tests cover quoting, omitted fields, nested paths, and the homepage. scripts/validate-markdown-twins.mjs (post-build, npm run test:markdown-twins in CI) checks every sitemap page that advertises a twin. The page must have <link rel="canonical">, and the advertised href must equal the expected twin path. The twin must exist and start with frontmatter whose canonical equals both the sitemap URL and the HTML canonical. The routes in config.json must give the twin exactly one Link header with that URL, and 14 non-twin paths must get none. /api/ advertises no twin and is skipped.
3. Copy page button: it fetches the .md twin (since [AR-20261028-3] Put the page description directly after the H1 and drop the hidden MDX copy #964). It now strips the frontmatter client-side (stripFrontmatter in src/lib/docs-markdown.js, unit-tested) so copied Markdown is byte-identical to before. Reason: the copied body already opens with the H1 title and description, so frontmatter would only duplicate them in a user's chat or notes. "View as Markdown" still opens the raw twin with frontmatter.
4. llms files: '404' added to the starlight-llms-txt exclude list (with a comment). scripts/validate-llms-txt.mjs (post-build, npm run test:llms-txt in CI) fails if any Canonical page: [..](url) in llms-full.txt, llms-small.txt, or _llms-txt/*.txt is not in sitemap-*.xml.

Hypothesis

Google supports rel="canonical" HTTP headers for non-HTML documents (Consolidate duplicate URLs). With the header and matching frontmatter, ranking and citation credit for a twin consolidates on its HTML page instead of splitting across .md URLs.

Metric and prediction

• ora docs markdown-frontmatter: 0/1 → 1/1 on the first production audit after deploy.
• Peec share of docs.warp.dev citations on .md URLs: at or below its 14-day pre-merge baseline over the 21 days after deploy. Falsified if it rises.

Verification

Preview https://docs-6yjh8b9ae-warpdotdev.vercel.app (head d01c41f), checked 2026-11-04:

• curl -sI https://docs-6yjh8b9ae-warpdotdev.vercel.app/factories.md → 200 text/markdown, link: <https://docs.warp.dev/factories/>; rel="canonical". /index.md → <https://docs.warp.dev/>. /agents.md → <https://docs.warp.dev/agents/>.
• 11/11 non-twin paths have no Link header: /assets/not-a-twin.md, /_astro/not-a-twin.md, /.well-known/not-a-twin.md, /robots.md, /AGENTS.md, /Factories.md, and /not-a-twin.md (all 404), plus /factories/, /llms.txt, /sitemap-0.xml, and /openapi.yaml (200).
• 20 random sitemap pages: 20/20 twins return 200 with exactly the header matching each HTML page's <link rel="canonical">, and frontmatter canonical equal to the same URL.
• llms-full.txt and llms-small.txt: 0 occurrences of docs.warp.dev/404/.
• verify_site.py --compare (production vs preview): 5 flagged items, all new robots restrictions ['noindex'], caused by the X-Robots-Tag: noindex header Vercel adds to every preview deployment. 0 regressions are attributable to this change.
• CI: all checks pass.

Exact-slug preview links: https://docs-6yjh8b9ae-warpdotdev.vercel.app/factories.md and https://docs-6yjh8b9ae-warpdotdev.vercel.app/llms-full.txt

Local checks

npm run typecheck, test:heading-vars, test:docs-markdown (16/16), test:markdown-canonical (5/5), AFDocs audit tests, test:tracking-redirect, test:agent-discovery, npm run build, test:heading-tocs, test:tracking-routes, test:homepage-json-ld, test:page-title-order, test:llms-txt (1244 entries across 14 files, all in sitemap), test:markdown-twins (409 twins and 14 non-twin paths OK), internal link check, redirect invariants: all pass.

Risks

• The header routes are written into .vercel/output/config.json, the file scripts/build-tracking-middleware.mjs already edits, so they rely on Vercel deploying that file as built. The preview confirms it.
• A twin added outside the normal page build would get no header until the routes are rebuilt. The post-build validator fails CI if a sitemap twin has no exact route.
• Content-negotiated Markdown served at the HTML URL (Accept: text/markdown) gets no canonical header; the HTML URL is already canonical, so nothing to consolidate.

Experiment AR-20261104-2 · Conversation · Run

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Nov 4, 2026 4:48pm UTC

[warp-agent-staging]

This PR was generated with Warp.

Comment @warp-staging-factory on this PR to send it follow-up work.

  View origin

[warp-agent-staging]

Autoresearch gates: all passed (AR-20261104-2).

• Metric (G1): ora docs markdown-frontmatter should go 0/1 → 1/1 on the first production audit after deploy. Peec share of docs.warp.dev citations landing on .md URLs should stay at or below its 14-day pre-merge baseline over 21 days. The experiment is falsified if that share rises.
• Fast verifier (G2) on the preview at head d01c41f:
  • 20/20 random sitemap twins return 200 with a Link canonical exactly equal to the HTML page's own canonical, and the same URL in frontmatter.
  • /index.md → https://docs.warp.dev/.
  • 11/11 non-twin paths (404 .md files, uppercase, /assets, /_astro, /.well-known, HTML pages, llms.txt, sitemaps) carry no Link header.
  • 0 /404/ links in llms-full.txt and llms-small.txt.
  • Copy page output is identical to production.
  • ora on the preview vs a main-based control preview: 59 → 59, markdown-frontmatter 0/1 → 1/1, no check regressed.
• CI and review (G3): CI green. The adversarial review's one blocking finding (the catch-all .md rule put a canonical header on 404 non-twin paths) is fixed. Header routes are now generated at build time for the 409 real twins only. Both non-blocking items are done: the validator is stricter, and the transitive-dependency import is removed.
• Second confirmation (G4): production /factories.md returns 200 with no Link header and no frontmatter.
• Merges cleanly together with docs#990.

Preview links: /factories.md (see the frontmatter at the top; curl -sI shows the Link header) · /llms-full.txt

Posted by a Warp agent