Skip to content

Acquire job row locks in a consistent order to prevent deadlocks (#89) - #90

Merged
TkTech merged 2 commits into
TkTech:26_uxfrom
ouihelp:branch-with-upgrades
Sep 16, 2026
Merged

TkTech merged 2 commits into
TkTech:26_uxfrom
ouihelp:branch-with-upgrades

Conversation

@AudeCstg

@AudeCstg AudeCstg commented Apr 17, 2026

Copy link
Copy Markdown

PR to follow changes used by tags

luca-montaigut and others added 2 commits September 10, 2026 15:40
…ech#89)

Pushing jobs whose unique keys already exist takes row locks through the
ON CONFLICT ... DO UPDATE clause, and the worker's batched job updates
take row locks too. When the two transactions overlapped and locked the
same rows in a different order, Postgres killed one of them with
DeadlockDetected, failing the push or the update batch.

Both code paths now sort by the same key before touching rows: pushes
insert in unique_key order (returning references in the caller's order),
and the worker sorts its pending updates by (unique_key, id). A
regression test replays the interleaving deterministically.

Co-authored-by: audecasteigts <aude.casteigts@gmail.com>
@AudeCstg
AudeCstg changed the base branch from main to 26_ux September 11, 2026 08:27
@AudeCstg AudeCstg changed the title Branch with upgrades Acquire job row locks in a consistent order to prevent deadlocks (#89) Sep 11, 2026
@AudeCstg
AudeCstg marked this pull request as ready for review September 11, 2026 08:29
@luca-montaigut

Copy link
Copy Markdown

Fixes #89.

Taking over this PR from @AudeCstg, context in #58 (comment). It is rebased on 26_ux and reduced to what is still needed there: the deadlock retry Aude originally added is covered by the transient-error backoff that _maintain_updates gained in 26_ux (DeadlockDetected is an OperationalError), but the lock ordering is not, and without it a push of existing unique keys overlapping a batch of job updates still deadlocks and fails one side.

What changes

  • push_many_ex / sync_push_many_ex insert jobs sorted by unique_key, and still return the references in the caller's order.
  • Worker._maintain_updates sorts its pending updates by (unique_key, id) before the executemany.
  • Both sort through one helper, chancy.utils.lock_order_key, so the rule cannot drift between the two code paths.
  • tests/regressions/test_89.py replays the interleaving deterministically (one transaction holds the first row in worker order while a concurrent push submits the same keys in reverse order). It fails with DeadlockDetected on 26_ux and passes with this patch. A second test checks the reference order of sync_push_many.
  • Changelog entry under 0.26.0.

Context

We run Chancy in production at Ouihelp (8 queues, ~60 job types, cron on ~50 schedules) and have been running this fix since April with no deadlock since. Happy to adjust anything.

@TkTech
TkTech merged commit 0318939 into TkTech:26_ux Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants