seo: technical fix bundle — robots, canonicals, receive-from, noindex catch-all, real 404s, redirects, sitemap lastmod - #2685
Merged
Merged
Conversation
…m pages
Three independent SEO hygiene defects.
robots.ts — the Googlebot group declared only `allow: ['/api/og']` with no
disallow key. A crawler obeys the single most specific group that matches it,
so Googlebot was reading that group INSTEAD of `*` and treating every
auth-gated route (/home, /profile, /settings, /kyc, ...) as crawlable. Extract
the `*` disallow list into a shared DISALLOWED_PATHS const and apply it to the
Googlebot group too, mirroring how the AI-crawler group is built. The narrower
/api/og allow still wins over /api/ by longest-match, so OG images keep
working.
careers + lp/card — both call the metadata helper without `canonical`, so they
inherited the root layout's `alternates: { canonical: '/' }` and declared the
homepage as their canonical while sitting in the sitemap. Pass the real path,
matching the existing exchange/layout.tsx workaround.
corridors.ts — loadReceiveSources seeded from CORRIDORS.from and intersected
with the receive-from content tree. receive-from is authored independently of
corridors, so the intersection silently dropped 10 authored countries
(australia, india, kenya, malaysia, netherlands, pakistan, philippines,
saudi-arabia, singapore, united-arab-emirates) whose articles were live but
unreachable. Enumerate published receive-from content directly via the
existing listPublishedSlugs helper. RECEIVE_SOURCES goes 9 -> 19; colombia and
mexico stay out, correctly, as origins with no article.
corridors.test.ts was a checked-in regression guard from the May 2026 no-content
incident asserting RECEIVE_SOURCES subset-of CORRIDORS.from. That subset relation
was an artifact of how the list was built, not what protected us — the real
invariant is "every entry has a published article". Rewritten to assert that in
both directions: no entry without content (no 404s) and no published article
left out (no orphans). Verified the new guard fails against the old loader.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…comments
Adversarial-review findings on this PR:
- scripts/verify-content.ts gateReceiveSources() was a stale mirror of the
old corridor-intersection loader — Pass 11 (blocking CI check) silently
stopped covering the 10 recovered receive-from slugs. Now enumerates
published receive-from articles directly, matching RECEIVE_SOURCES.
- robots.ts: DISALLOWED_PATHS comment overclaimed ('every named crawler
group'); reworded to what the file actually does.
- lp/card: canonical comment misattributed the inherited '/' to the root
layout; it comes from lp/layout.tsx's deliberate alias policy.
The [...recipient] catch-all renders an indexable "X on Peanut" shell for
ANY username-shaped string — existence is only checked client-side, so a
non-existent handle still returns HTTP 200 with full metadata. Google has
been indexing those as thin/soft-404 pages.
Set robots: { index: false, follow: false } on all four generateMetadata
return paths (reserved route, missing recipient, !couldBeRecipient, and the
full metadata object). No carve-outs: real profiles, ETH addresses, .eth ENS
names and request/receipt links are app surface, not search landing pages.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adversarial-review catch: every branch of this route inherits
alternates.canonical '/' from the root layout, so the noindex was
shipping on pages that canonicalize to the homepage - a combination
Google's canonicalization guidance warns can bleed the noindex into
the cluster head (the homepage itself, sitemap priority 1.0).
alternates: { canonical: null } suppresses the inherited value; the
noindex pages now carry no canonical at all.
The [...recipient] route's loading.tsx created a Suspense boundary at the segment. Page() suspends on use(props.params), so React rendered the fallback, flushed the shell with HTTP 200, and only then resolved notFound() — which landed as a NEXT_HTTP_ERROR_FALLBACK;404 marker inside an already-200 response body. Removing the boundary makes the segment's suspension propagate to the shell, so notFound() throws before headers flush and Next sets a real 404 status. The route's render is otherwise synchronous (params guard checks only), and PaymentPage is a client component, so there is no server data fetch left to cover with a fallback.
…e trailing-slash dupes Four routing defects, all in the redirect/reserved-route layer: 1. /pricing, /stories and /content are real [locale]/(marketing) pages, but the bare paths are 7 lowercase letters, so couldBeRecipient() accepted them and the [...recipient] catch-all served a payment-profile shell on HTTP 200. Reserved in DEDICATED_ROUTES + 301'd to /en/.... 2. /help/delete-account (and every other help article at its bare path) soft-404'd the same way: only bare /help was redirected, never /help/:path*. That URL is the account-deletion link Google Play requires. Added /help/:path* and /stories/:path* so the newly-reserved prefixes keep resolving instead of hard-404ing. 3. skipTrailingSlashRedirect is on (it must stay — the PostHog /relay proxy is called with trailing slashes), so /en/help/ and /en/help both returned 200. Added one slash-stripping redirect scoped to the locale-prefixed tree only. 4. docs.peanut.to had no host rule; only docs.peanut.me did. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adversarial-review blocker: :path* matches zero segments, so bare locale homepages (/es-ar/, /es-419/, /pt-br/) compiled a 308 redirect to themselves - an infinite loop, cached permanently by browsers. Measured live on the preview; /en/ survived only because the /en -> / rule sits earlier in the array. :path+ requires a segment, so bare locale + slash falls through to its current 200 (matches production). Verified with next's checkCustomRoutes + compiled-rule replay of the loop paths and the intended redirect paths.
Every sitemap URL carried lastModified: BUILD_DATE, so each deploy told crawlers all 709 pages had just changed. That is noise, and it costs the signal on pages that genuinely did change. Content-backed URLs now report the generated_at of the exact file that serves them. 684 of 709 URLs get a real date across 21 distinct values; the remaining 25 (hand-built pages and index pages with no single backing file) keep BUILD_DATE as the fallback. contentGeneratedAt() coerces the frontmatter value to a Date. Note the type/runtime mismatch it guards: ContentFrontmatter declares generated_at as a string, but gray-matter runs js-yaml, which parses unquoted YAML timestamps into Date objects — so both shapes have to work. The lookups read through the cache the has*Content() guards already populate, so they add no file reads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Aug 12, 2026
Closed
Contributor
Code-analysis diffPainscore total: 7158.83 → 7158.46 (-0.37) 🆕 New findings (17)
✅ Resolved (18)
📈 Painscore deltas (top movers)
|
Contributor
🧪 UI test report — ✅ all greenSuites
📊 Coverage (unit)
⏱ 10 slowest test cases
|
…itterbot exemption Review findings 1+2 from #2676 (abalinda): the AI-crawler group's 6-entry list left claim links, receipts, QR, card-payment, invite and withdraw crawlable by GPTBot/ClaudeBot/PerplexityBot while every other crawler was blocked. They now share DISALLOWED_PATHS - AI engines should read (and cite) content pages, not transactional surface. Twitterbot's empty disallow is deliberate (card unfurls on shared claim/payment links) and now documented as such.
) - loader + verify-content gate skip a future 'index' meta directory (same guard sitemap.ts already applies to other intents) - test parses frontmatter with gray-matter instead of a regex, so YAML edge cases (published: False, trailing comments) agree with the loader - test fails with an actionable message when the src/content submodule is not initialized, instead of a raw ENOENT
Contributor
Author
|
Response to @abalinda's six findings from #2676 (they all transfer here — this PR contains that code): 1. Twitterbot (fixed — 2. AI-crawler group (fixed — 3. Googlebot newly blocks 4. 5. Frontmatter regex vs YAML (fixed — 6. Submodule ENOENT (fixed — Gates on the new head: jest 3/3, 🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidates the five SEO fix PRs (#2676, #2678, #2681, #2682, #2684) into one PR at the CTO's request — one commit per fix, so any single change can be reverted surgically. Every fix was implemented and then adversarially reviewed by an independent agent that re-derived the evidence itself; all review catches are already folded in as their own commits. Full evidence lives on the five closed PRs; the load-bearing parts are summarized below.
What's in here (commit order)
1.
2648b5aaf— robots.txt, canonicals, receive-from loader (from #2676)robots.ts: the Googlebot group had onlyAllow: /api/og, so per the robots spec Googlebot ignored all 24*disallows. Now shares the disallow list./careers+/lp/cardemittedrel=canonical→ homepage (inherited); both now self-canonicalize. Verified by grepping the canonical<link>from rendered preview HTML vs the production defect.corridors.ts:loadReceiveSources()intersected receive-from content with corridor origins, silently dropping 10 fully-authored countries (australia, india, kenya, malaysia, netherlands, pakistan, philippines, saudi-arabia, singapore, UAE). Now enumerates published content directly:RECEIVE_SOURCES9 → 19.corridors.test.tsrewritten to the invariant it actually protects (every slug has published content — the May-2026 incident was the reverse). Review re-derived the counts and confirmed the new guard goes red on the old loader.2.
936b012f9— review catch on the above:scripts/verify-content.tshad a stale second copy of the old loader — Pass 11 (the blocking sitemap-coverage CI check) would have silently stopped covering the 10 recovered slugs. Script and app now both return the same 19; Pass 11 coverage 1115 → 1165 URLs.3.
153c6bae5— noindex the username catch-all (from #2678)Any username-shaped string (
/brazil,/pricing,/zzzfake1) rendered an indexable "X on Peanut" shell — the existence check is client-side, invisible to crawlers, and the root layout injectsindex, follow, so Google indexed thousands. All 4generateMetadatareturn paths now shiprobots: {index:false, follow:false}. Decision on record (12 Aug): no carve-outs, including real profiles and.ethpages.4.
69b3e5dbf— HIGH review catch on the above: the noindex was shipping alongside the inheritedrel=canonical→ homepage — a combination Google's guidance warns can attribute the noindex to the canonical cluster head (the homepage, sitemap priority 1.0).alternates: {canonical: null}on all four branches; noindex pages now carry no canonical.5.
5cb458a46— real HTTP 404s (from #2681)[...recipient]/loading.tsxmade the catch-all a Suspense boundary, so Next streamed an HTTP 200 shell beforenotFound()resolved — the site never returned a real 404 (breaks GSC soft-404 reporting, link tools, crawl budget). One file deleted; the route's render path was confirmed to have no other boundary. Preview-verified: garbage URL → 404, real pages → 200, profiles → 200. Review verdict SHIP after independently re-probing.6.
74ca6e544+ 7.f5034b218— redirects bundle (from #2682)/pricing,/stories,/contentadded toDEDICATED_ROUTES+ bare→/en/…redirects (they were being squatted by the username catch-all)./help/:path*bare→locale redirect —/help/delete-account(the Google Play account-deletion compliance URL) soft-404'd; it now resolves to the real article.docs.peanut.tohost-redirect rule (inert until ops attaches the domain — see asks).skipTrailingSlashRedirectuntouched (protects the PostHog/relay/*proxy — probed: still proxies).:path*matched zero segments, so/es-ar/,/es-419/,/pt-br/308-looped to themselves (measured on the preview; browsers cache 308s permanently). Fixed with:path+; re-probed — bare locales 200, subpaths still redirect,follow /es-ar/= 0 hops.8.
8ce2b769e— real sitemap lastmod (from #2684)All 709 URLs reported the deploy timestamp as
lastmod, so Google discounts the signal. Dates now come from each page'sgenerated_atfrontmatter; static/legal pages keep BUILD_DATE. Review independently cross-checked 686 URLs against the content tree: 0 mismatches; dev deploy serves 1 distinct lastmod, this branch serves 26, identical URL set.9.
d2765ee81— stale comment cleanup flagged by review.Verification status
validate-linksall passes (Pass 11 = 1165 URLs),tsc --noEmitexit 0, eslint 0 errors, all 9 commits SSH-signed.next buildcannot run on the dev box (earlyoom) — CI + the Vercel preview are the build gates; each constituent branch already had them green. A fresh probe pass runs on THIS PR's preview after CI; evidence will be posted as a comment.robots.txtand thenoindexmeta can't be observed on previews (non-production BASE_URL noindexes everything) — those two rest on code + the per-branch evidence.Asks / follow-ups (not in this PR)
storiesandcontentto the server-side reserved-username list (pricingis already reserved; the other two are claimable today, and frontend routing alone doesn't stop a signup claim).docs.peanut.toto this Vercel project + repoint DNS; flip the redirect.pizza rule forpeanut.todeep paths to path-preserving.last_updatedfrontmatter isn't read — small follow-up).robots.ts/redirects.jsonin non-overlapping hunks; feat: host creator contest on peanut.me #2605 addscreator-contesttoDEDICATED_ROUTESin the same region (trivial).Replaces #2676, #2678, #2681, #2682, #2684 (closed with pointers; branches kept until this merges).
🤖 Generated with Claude Code