seo: fix Googlebot disallows, missing canonicals, and 10 dropped receive-from pages - #2676
seo: fix Googlebot disallows, missing canonicals, and 10 dropped receive-from pages#26760xkkonrad wants to merge 2 commits into
Conversation
…m pages
Three independent SEO hygiene defects.
robots.ts — the Googlebot group declared only `allow: ['/api/og']` with no
disallow key. A crawler obeys the single most specific group that matches it,
so Googlebot was reading that group INSTEAD of `*` and treating every
auth-gated route (/home, /profile, /settings, /kyc, ...) as crawlable. Extract
the `*` disallow list into a shared DISALLOWED_PATHS const and apply it to the
Googlebot group too, mirroring how the AI-crawler group is built. The narrower
/api/og allow still wins over /api/ by longest-match, so OG images keep
working.
careers + lp/card — both call the metadata helper without `canonical`, so they
inherited the root layout's `alternates: { canonical: '/' }` and declared the
homepage as their canonical while sitting in the sitemap. Pass the real path,
matching the existing exchange/layout.tsx workaround.
corridors.ts — loadReceiveSources seeded from CORRIDORS.from and intersected
with the receive-from content tree. receive-from is authored independently of
corridors, so the intersection silently dropped 10 authored countries
(australia, india, kenya, malaysia, netherlands, pakistan, philippines,
saudi-arabia, singapore, united-arab-emirates) whose articles were live but
unreachable. Enumerate published receive-from content directly via the
existing listPublishedSlugs helper. RECEIVE_SOURCES goes 9 -> 19; colombia and
mexico stay out, correctly, as origins with no article.
corridors.test.ts was a checked-in regression guard from the May 2026 no-content
incident asserting RECEIVE_SOURCES subset-of CORRIDORS.from. That subset relation
was an artifact of how the list was built, not what protected us — the real
invariant is "every entry has a published article". Rewritten to assert that in
both directions: no entry without content (no 404s) and no published article
left out (no orphans). Verified the new guard fails against the old loader.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
📝 WalkthroughWalkthroughThe change makes published ChangesSEO and indexing behavior
Estimated code review effort: 3 (Moderate) | ~20 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Code-analysis diffPainscore total: 7157.87 → 7159.31 (+1.44) 🆕 New findings (4)
✅ Resolved (4)
|
🧪 UI test report — ✅ all greenSuites
📊 Coverage (unit)
⏱ 10 slowest test cases
|
…comments
Adversarial-review findings on this PR:
- scripts/verify-content.ts gateReceiveSources() was a stale mirror of the
old corridor-intersection loader — Pass 11 (blocking CI check) silently
stopped covering the 10 recovered receive-from slugs. Now enumerates
published receive-from articles directly, matching RECEIVE_SOURCES.
- robots.ts: DISALLOWED_PATHS comment overclaimed ('every named crawler
group'); reworded to what the file actually does.
- lp/card: canonical comment misattributed the inherited '/' to the root
layout; it comes from lp/layout.tsx's deliberate alias policy.
|
Adversarial review pass (agent) — verdict: fixable, fixed in 622977c. Main catch: Also fixed two misleading comments the review flagged (robots.ts scope overclaim; lp/card canonical misattribution — it inherits Follow-up noted, not in this PR: the 10 recovered pages have zero inbound internal links from published content — tracked in the content workstream. 🤖 Generated with Claude Code |
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
…itterbot exemption Review findings 1+2 from #2676 (abalinda): the AI-crawler group's 6-entry list left claim links, receipts, QR, card-payment, invite and withdraw crawlable by GPTBot/ClaudeBot/PerplexityBot while every other crawler was blocked. They now share DISALLOWED_PATHS - AI engines should read (and cite) content pages, not transactional surface. Twitterbot's empty disallow is deliberate (card unfurls on shared claim/payment links) and now documented as such.
) - loader + verify-content gate skip a future 'index' meta directory (same guard sitemap.ts already applies to other intents) - test parses frontmatter with gray-matter instead of a regex, so YAML edge cases (published: False, trailing comments) agree with the loader - test fails with an actionable message when the src/content submodule is not initialized, instead of a raw ENOENT
2 participants
Three independent SEO hygiene defects, batched because each is a few lines and they share no code.
T1 — Googlebot ignored every disallow
Defect. The
Googlebotgroup insrc/app/robots.tsdeclared onlyallow: ['/api/og']with nodisallowkey. A crawler obeys the single most specific group that matches it, never the union — so Googlebot read that group instead of*and treated every auth-gated route (/home,/profile,/settings,/kyc,/claim, …) as fair game. The one crawler that matters was the one exempted. Liverobots.txtconfirms this shipped.Fix. Extracted the
*group's disallow list into a sharedDISALLOWED_PATHSconst and applied it to the Googlebot group too — mirroring how the AI-crawler group in the same file is built. The narrower/api/ogallow still wins over/api/by longest-match, so OG images keep rendering in link previews.T2 —
/lp/cardand/careersdeclared the homepage as canonicalDefect. Both pages call the metadata helper without
canonical.src/app/layout.tsxsetsalternates: { canonical: '/' }, andsrc/app/metadata.tsonly emitsalternateswhen a canonical is passed — so both pages inherited the root value and told Google their canonical was/while sitting in the sitemap as indexable URLs. Self-canonical was never declared.Fix. Pass the real path in each
generateMetadata()call, matching the existingsrc/app/exchange/layout.tsxworkaround (comment carried over). Neither route has alayout.tsx, so the fix belongs inpage.tsx.T6 — 10 authored countries silently unreachable
Defect.
loadReceiveSources()insrc/data/seo/corridors.tsseeded fromCORRIDORS.map(c => c.from)and then intersected with the receive-from content tree. But receive-from is authored independently of corridors, so the intersection dropped every article whose country was not also a corridor origin — 10 of them: australia, india, kenya, malaysia, netherlands, pakistan, philippines, saudi-arabia, singapore, united-arab-emirates. Written, published, and unreachable.Fix. Enumerate published receive-from content directly via the existing
listPublishedSlugs('receive-from')helper (same semantics the loader already applied:en.mdpresent andpublished !== false).RECEIVE_SOURCESgoes 9 → 19. colombia and mexico stay excluded, correctly — they are corridor origins with no article, the original May-2026 404 case.The test that had to change
src/data/seo/corridors.test.tsis a checked-in regression guard from the May 2026 no-content incident, and two of its three assertions encoded the old invariant (RECEIVE_SOURCES ⊆ CORRIDORS.from). That subset relation was an artifact of how the list was built, not the thing that protected us. The real invariant is "every entry has a published article". Rewritten to assert exactly that, in both directions — nothing rendered without content (no 404s), nothing authored left behind (no orphans) — plus a duplicate check.The new guard re-derives the expected set straight off the filesystem rather than reusing the loader's own helper, so it cannot pass by tautology. I verified it has teeth by temporarily restoring the old loader and confirming it goes red, naming the exact dropped countries:
Verification
T1 — generated
robots.txt(served from this branch)All 24 disallows now present on the Googlebot group,
/api/ogallow retained,*group unchanged.T2 — canonical
<link>grepped out of the rendered HTML bodyChecked the response headers too — the only
Link:header is font preloads, so the canonical genuinely lives in the HTML, not a header. For contrast, production today:T6 — the 10 recovered countries
The user-visible effect is confirmable on production right now.
receive-money-from/[country]callsnotFound()for anything outsideRECEIVE_SOURCES, and this site's custom 404 returns HTTP 200 — so the dropped pages are soft 404s and status codes tell you nothing. Comparing titles instead:singapore is byte-for-byte the same shell as a garbage URL; brazil is a real article. That is the defect, live.
e2e was not run. The Playwright suite needs a live API on
:5000serving/dev/test-sessionfor itsglobalSetup, which isn't available in this environment. No new e2e specs were written.Caveats for the reviewer
main), which also editsrobots.ts— different hunks (sitemap field + an import), but it lands indevvia back-merge on its own schedule. My hunks are deliberately minimal.RECEIVE_SOURCESgrowing 9 → 19 adds 10 URLs to the sitemap. That is the intended effect: the content already exists and is already published, it just had no route. Worth a glance at the sitemap diff on the preview deploy.next buildnever ran to completion in my sandbox, so CI is the first green signal on the production build specifically. Not a code problem, and I checked rather than assumed: this box runsearlyoomwith--prefer (^|/)(node|next-server|esbuild), so under memory pressure from other agents building concurrently it SIGTERMs whichever node process is largest. Its own log names the victim —sending SIGTERM to process ... "next-server": badness 1128, VmRSS 3383 MiB— which is exactly theNext.js build worker exited with code: null and signal: SIGTERMreported across three build attempts. The dev server behind the render evidence above was killed the same way twice before it survived long enough to answer. Everything else (typecheck, lint, jest, prettier, and the live render evidence) is green locally.🤖 Generated with Claude Code
Summary by CodeRabbit
SEO Improvements
Bug Fixes