Commit 2503827
committed
feat(stream): per-task reclaim, prefetch backpressure, NOGROUP self-heal
RedisStreamBroker reclaim was driven by a fixed idle_timeout and a
broker-side Redis lock, which both double-ran long tasks and recovered
short crashed tasks slowly. Replace it with per-task-deadline reclaim:
- Resolve reclaim deadline from the message's `timeout` label (via
formatter.loads) plus reclaim_timeout_grace, falling back to
idle_timeout for messages without a timeout label.
- Drop the autoclaim Redis lock; rely on XCLAIM min-idle-time for
server-side atomic dedup (unacknowledged_lock_timeout is deprecated
and ignored).
- Protect only messages held by the current listener instance from
reclaim (tracked via a local delivered set), so a worker sharing a
consumer_name with a dead predecessor still recovers its pending.
- Gate reclaim sweeps with reclaim_interval (default 30s) to avoid
scanning pending on every listen iteration.
- Enforce prefetch backpressure: do not XREADGROUP while xread_count
delivered-but-unacked messages are outstanding.
- Recreate a missing consumer group on NOGROUP during xpending/xreadgroup.
Tests cover timeout-label reclaim, idle_timeout reclaim, shared
consumer_name reclaim, prefetch backpressure, and NOGROUP self-heal.
README documents the new reclaim/backpressure behavior.1 parent 39c6b7c commit 2503827
3 files changed
Lines changed: 707 additions & 70 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
101 | 118 | | |
102 | 119 | | |
103 | 120 | | |
| |||
0 commit comments