Skip to content

Sitemap.from_xml_string cannot opt into enqueue host filtering #2118

Description

@janbuchar

Found while triaging GHSA-c533-9w4q-667c (which I'm closing as informative - a raw XML string has no origin, so there's no host policy to bypass). It does point at a real API gap though.

  • Sitemap.from_xml_string takes only content and hardcodes SitemapSource(type='raw', content=content) - no url, no ParseSitemapOptions
  • host filtering is conditional on source.get('url')
  • so if you have raw sitemap XML and you know where it came from, there's no way to say so - filtering is silently skipped and the same-hostname default never applies
  • suggestion: from_xml_string(content, *, sitemap_url: str | None = None, parse_sitemap_options: ParseSitemapOptions | None = None), forwarded as SitemapSource(type='raw', content=content, url=sitemap_url)
  • explicitly not worth doing: rejecting <loc> values when the origin is unknown - that makes the default behaviour return an empty list
  • while in there: _get_origin_url stamps raw sources with raw://<sha256>, so SitemapUrl.origin_sitemap_url is a pseudo-URL that would fail scheme validation if anyone fed it back into filter_url - harmless today, footgun later

Metadata

Metadata

Assignees

No one assigned

    Labels

    t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions