Skip to content

Use standards-compliant robots directives - #17

Merged
Jason-Doyle merged 1 commit into
mainfrom
fix/valid-robots
Sep 25, 2026
Merged

Jason-Doyle merged 1 commit into
mainfrom
fix/valid-robots

Conversation

@Jason-Doyle

Copy link
Copy Markdown
Owner

Summary

  • remove the nonstandard Content-Signal robots.txt directive rejected by Lighthouse
  • keep unrestricted crawling through User-agent: * and Allow: /
  • retain AI-oriented discovery through llms.txt and llms-full.txt
  • update site and production validation coverage

Validation

  • Astro check and production build
  • site worker tests
  • Chromium and mobile site tests
  • Cloudflare deployment dry run

@Jason-Doyle
Jason-Doyle merged commit c62a603 into main Sep 25, 2026
1 check passed
@Jason-Doyle
Jason-Doyle deleted the fix/valid-robots branch September 25, 2026 05:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant