Skip to content

Prevent worker shutdown hangs after lost cancellation on Python 3.11 - #97

Closed
PaulM5406 wants to merge 1 commit into
TkTech:26_uxfrom
PaulM5406:fix/durable-worker-cancellation
Closed

PaulM5406 wants to merge 1 commit into
TkTech:26_uxfrom
PaulM5406:fix/durable-worker-cancellation

Conversation

@PaulM5406

Copy link
Copy Markdown
Contributor

Hey @TkTech,

I used Astra to help me with this one.

Python 3.11's asyncio.wait_for() can swallow cancellation when its inner awaitable finishes concurrently. If this happens inside Psycopg, a worker polling task can keep running after TaskManager.cancel_all() has cancelled it, leaving final teardown waiting indefinitely.

Restore pending cancellation at Chancy's polling boundaries, after database contexts have exited. The shared sleep helper covers worker and plugin polling, including retry delays. Read notifications in one-second windows on the same connection, checking cancellation between windows and before dispatching a notification.

This addresses the shutdown race discussed in #58 and psycopg#1413. It does not change Psycopg or require a Python upgrade. Arbitrary cancellation-resistant user jobs and database operations that never return remain outside this fix.

@codecov

codecov Bot commented Sep 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.95238% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.57%. Comparing base (28867f1) to head (0c6304e).

Files with missing lines Patch % Lines
chancy/worker.py 75.00% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##            26_ux      #97      +/-   ##
==========================================
- Coverage   67.86%   67.57%   -0.30%     
==========================================
  Files          62       62              
  Lines        3772     3781       +9     
==========================================
- Hits         2560     2555       -5     
- Misses       1212     1226      +14     
Flag Coverage Δ
unittests 67.57% <80.95%> (-0.30%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@PaulM5406

Copy link
Copy Markdown
Contributor Author

The CI failure looks unrelated to this change. In test_trigger_different_schema, the logs show the worker picking up the job before the test checks for state = 'pending', so the query returns no rows. The trigger did create the job.

Removing the worker fixture from this test should fix the race: it only checks job creation, which the PostgreSQL trigger handles directly. All nine cancellation tests passed in the failed job.

@TkTech

TkTech commented Sep 19, 2026

Copy link
Copy Markdown
Owner

Thanks @PaulM5406, luckily fixed upstream so we can skip this one (see #58). I'll take a look at that race test...

@TkTech TkTech closed this Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants