Skip to content

fix: await dispatcher stream cleanup#2084

Open
nightcityblade wants to merge 1 commit into
unclecode:developfrom
nightcityblade:fix/issue-2083
Open

fix: await dispatcher stream cleanup#2084
nightcityblade wants to merge 1 commit into
unclecode:developfrom
nightcityblade:fix/issue-2083

Conversation

@nightcityblade

Copy link
Copy Markdown
Contributor

Summary

Fixes #2083

Explicitly closes the inner dispatcher stream when a caller closes the public arun_many stream, ensuring dispatcher tasks and queue state are cleaned up before proxy-session release.

List of files changed and why

  • crawl4ai/async_webcrawler.py - Retain and close the dispatcher stream in the wrapper cleanup path
  • tests/async/test_dispatchers.py - Exercise early stream closure through AsyncWebCrawler.arun_many and assert task cancellation and queue cleanup

How Has This Been Tested?

  • uv run pytest tests/async/test_dispatchers.py::TestDispatchStrategies::test_arun_many_stream_closure_cleans_up_tasks -q (1 passed)
  • Full dispatcher file excluding one unrelated stale CrawlerMonitor constructor test (10 passed, 1 deselected)
  • Black check for the changed test file, compileall, and git diff --check

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (N/A: this is an internal cleanup fix)
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes (the relevant dispatcher tests pass; one unrelated stale monitor test remains)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant