Fix merge_chunks pre-allocating a fixed chunk count that ignores overlap

[chuenchen309]

Summary

merge_chunks() (crawl4ai/utils.py) pre-computes:

num_chunks = max(1, (total_tokens + target_size - 1) // target_size)
chunks: List[List[str]] = [[] for _ in range(num_chunks)]
...
if curr_size >= target_size and curr_chunk < num_chunks - 1:
    ...

num_chunks is derived only from total_tokens // target_size and never
accounts for the overlap tokens that get re-injected at every chunk
boundary. With this repo's own defaults (chunk_token_threshold=2048,
overlap_rate=0.1 -> overlap=205), each boundary adds ~205 tokens beyond
that naive budget. Once curr_chunk reaches the last pre-allocated slot,
curr_chunk < num_chunks - 1 is permanently False, so the rollover branch
can never fire again — every remaining token gets dumped into that one last
chunk, uncapped.

merge_chunks feeds both LLMContentFilter
(content_filter_strategy.py:913) and LLMExtractionStrategy
(extraction_strategy.py:778), so a long document silently produces one
"chunk" far larger than the configured token budget — risking truncation or
blown context windows in the downstream LLM call, with no error or warning.

Repro (before fix)

target_size, overlap = 2048, 205  # repo defaults
docs = [f"word{i} " * 50 for i in range(n_docs)]
chunks = merge_chunks(docs, target_size=target_size, overlap=overlap)
# last chunk size vs target_size=2048:
#   n_docs=200  (10,000 tokens): last chunk = 2628  (+28%)
#   n_docs=800  (40,000 tokens): last chunk = 4983  (+143%)
#   n_docs=3200 (160,000 tokens): last chunk = 16246 (+693%)

Overshoot is unbounded and grows with document length — not a fixed
off-by-one.

Fix

Stop pre-allocating a fixed chunk count. Grow the chunk list on demand
(append a new chunk whenever curr_size reaches target_size,
unconditionally) instead of capping rollover at a count computed without
knowledge of the overlap that will be injected. After the fix, every chunk
in the repro above stays at or under target_size.

Testing

Added tests/test_merge_chunks.py:

• no-overlap chunks stay within target_size
• overlap chunks stay within target_size at 10k and 160k tokens (regression
  guard for the unbounded-growth behavior)
• overlap tokens are correctly repeated at chunk boundaries
• full token content is preserved when overlap=0
• empty input returns []

Confirmed red→green: reverting only crawl4ai/utils.py reproduces the two
overshoot-size test failures (2628/16246 > 2048); reapplying passes all 6.

Full baseline (tests/unit/, tests/deep_crawling/,
tests/regression/test_reg_extraction.py -m "not network"): 133 passed, 1
skipped (unrelated, needs live network).

python -m py_compile crawl4ai/utils.py: clean (repo has no
lint/format tooling configured — checked pyproject.toml, no
Makefile/.pre-commit-config.yaml).

xref: no open PR touches merge_chunks.

Checklist:

• I have performed a self-review of my own code
• I have added/updated unit tests that prove my fix is effective
• New and existing unit tests pass locally with my changes

AI-Generated disclosure

Found via an AI-assisted code review pass (Claude Code) over crawl4ai/utils.py.
I personally reproduced the unbounded-overshoot behavior with a standalone
script at three document sizes, verified the fix eliminates it, and ran the
relevant test suites before submitting.