fix(deep_crawling): stop BFS batch mode from re-crawling a self-linking start URL

[chuenchen309]

Summary

BFSDeepCrawlStrategy._arun_batch() processed each BFS level's URLs without first adding them to the visited set, unlike its sibling _arun_stream(), which does visited.update(urls) right after computing the level's URL list. Because of that asymmetry, in batch mode the start URL never entered visited before link_discovery() scanned the start page's own outgoing links — link_discovery() only adds a URL to visited when it accepts it as a newly discovered link; the start URL is seeded directly into current_level and never goes through that path.

Concrete failure: crawl any site whose homepage links back to itself (a "Home"/logo nav link, a canonical self-link, or a bare href="#", which normalize_url_for_deep_crawl reduces to the base URL after stripping the fragment). With BFSDeepCrawlStrategy(max_depth=2)._arun_batch(...), that self-link is treated as brand-new and unvisited: it gets appended to next_level, and depths[start_url] is overwritten from 0 to 1 — the start page gets crawled a second time at depth 1 with a corrupted depth/parent_url and an inflated _pages_crawled count. _arun_stream() does not have this bug (it already seeds visited correctly), so this is a batch-mode-only regression relative to the documented "one entry per crawled URL" behavior. CrawlerRunConfig.stream defaults to False, so _arun_batch is the default deep-crawl code path, not an edge case.

Fixes #123: n/a — found via code review, no existing issue filed for this specific bug.

List of files changed and why

• crawl4ai/deep_crawling/bfs_strategy.py — add the same visited.update(urls) call to _arun_batch that _arun_stream already has, restoring parity between the two modes.
• tests/deep_crawling/test_bfs_batch_self_link.py (new) — regression tests using this repo's existing MagicMock-based crawler/config test double pattern (see tests/deep_crawling/test_deep_crawl_cancellation.py).

How Has This Been Tested?

• Added 3 tests verifying: the start URL is only crawled once when it self-links, its recorded depth stays 0, and batch/stream modes now agree on the resulting URL set for the same site graph.
• Confirmed red→green: reverting only bfs_strategy.py reproduces all 3 failures with the exact predicted symptoms (start URL crawled twice, depth overwritten, batch/stream URL-set mismatch); reapplying the fix passes.
• Full tests/deep_crawling/ (72 tests) and tests/unit/ + tests/deep_crawling/ combined (120 tests): all pass, no regressions.
• No lint/format tooling is configured in this repo (checked pyproject.toml, no Makefile/.pre-commit-config.yaml); confirmed changed files compile cleanly with python -m py_compile.

Checklist:

• I have performed a self-review of my own code
• I have added/updated unit tests that prove my fix is effective
• New and existing unit tests pass locally with my changes

AI-Generated disclosure

Found via an AI-assisted code review pass (Claude Code) over crawl4ai/deep_crawling/. I personally traced the visited-set asymmetry between _arun_batch/_arun_stream, confirmed the concrete self-link failure scenario, verified the fix, and ran the relevant test suites before submitting.