Fix is_external_url misclassifying sibling domains as internal

[jichaowang02-lang]

Summary

is_external_url decides same-site vs external with:

return not url_domain.endswith(base)

str.endswith is a raw string-suffix test with no domain-label boundary, so
any host whose name merely ends with the base-domain string is treated as
internal:

is_external_url("https://notexample.com/landing", "example.com")
#   -> False   ❌ (classified as internal/same-site)

notexample.com, myexample.com, evilexample.com are all different
registrable domains from example.com, but each ends with the string
example.com, so they are wrongly bucketed as internal.

This matters because quick_extract_links() calls get_base_domain(base_url)
then is_external_url(...) to split every <a href> into the internal /
external lists that drive deep-crawl scoping and link reporting — so a
deep crawl can follow a look-alike/sibling domain as if it were the same site,
and internal/external link sets are simply wrong.

Fix

Require a label boundary — a host is same-site only if it equals the base
domain or is a true sub-domain of it:

return not (url_domain == base or url_domain.endswith("." + base))

host (base = example.com)	before	after
example.com	internal	internal ✅
www.example.com	internal	internal ✅
sub.example.com	internal	internal ✅
notexample.com	internal ❌	external ✅
evilexample.com	internal ❌	external ✅
other.com	external	external ✅

Testing

Adds TestIsExternalUrl to tests/regression/test_reg_utils.py:

$ pytest tests/regression/test_reg_utils.py::TestIsExternalUrl -q
9 passed

The sibling/look-alike cases fail on the current code (3 failed) and pass
with this fix (9 passed); same-domain, www, real-subdomain, unrelated,
special-scheme, and relative-URL cases are all preserved.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[chuenchen309]

Traced this against the base string and it's a real fix. The old not url_domain.endswith(base) matches on a raw string suffix with no label boundary, so for base = "example.com", "notexample.com".endswith("example.com") is True → the host was classified as internal. That's not just cosmetic link-bucketing — it lets a look-alike/sibling domain (evilexample.com, myexample.com) slip inside the internal set, which for deep-crawl scoping means the crawler can walk off the intended site.

The fix is the standard domain-suffix check: exact match, or endswith("." + base) so a real label boundary is required. Walked the cases:

• example.com == base → internal ✓
• sub.example.com / deep.sub.example.com end with .example.com → internal ✓
• notexample.com / evilexample.com don't → external ✓ (the regression)
• other.com → external ✓

Test coverage matches the fix well (same-domain, www variant, real subdomain, unrelated, the parametrized sibling look-alikes, mailto, relative). LGTM.

One adjacent note, not from this PR: the .replace("www.", "") on both domains strips www. anywhere in the string rather than only a leading prefix, so a host like myww w.foo style edge cases could normalize oddly — pre-existing and out of scope here, just flagging in case it's worth a separate follow-up.