Fix large status@broadcast sends timing out - #6
Open
Elimeshi1 wants to merge 1 commit into
Open
Conversation
Sending a status to a large contact list failed with "timed out waiting for message send response". Shrinking the participants batch from 5000 to 500 did not fix it: it multiplied the number of stanzas by ten and traded the timeouts for "server returned error 429". There were two independent causes. The per-batch ack window. whatsmeow defaults to 75s, which a big status batch can exceed even when it is delivered, so the send is reported as a timeout. Batch size was never the problem: what trips WhatsApp's rate limiting is the number of stanzas, not the participants inside one. Set an explicit timeout (default 180s, WAHA_GOWS_STATUS_BATCH_TIMEOUT) and restore the 5000 default. Database lock contention. The sqlite device store was opened in the default rollback-journal mode, where one writer holds an exclusive lock. During a status broadcast the outgoing send and the flood of incoming decryptions (session/identity/sender-key writes) contend for it, exhaust the 30s busy_timeout and surface as "database is locked" - turning a ~4 minute send into a ~30 minute stall ending in a gRPC DEADLINE_EXCEEDED. Open it with WAL + synchronous=NORMAL + txlock=immediate, mirroring the GContainer store. Also detach the status send from the caller's context. A deadline on the caller side was aborting a send already live on WhatsApp, leaving the later batches undelivered while the caller saw a plain failure and re-sent the whole status - posting it two or three times. Fixes devlikeapro/waha#2096
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sending a status to a large contact list fails with
timed out waiting for message send response. Lowering the participants batch from 5000 to 500 did not fix it — it multiplied the stanza count by ten and traded the timeouts forserver returned error 429.The same list, before and after that change:
7 stanzas vs 123. Batch size was never the problem: what trips the rate limiting is the number of stanzas, not the number of participants inside one.
There are two independent causes behind the original timeout.
The per-batch ack window
SendRequestExtra.Timeoutwas never set, so whatsmeow's default of 75s applied. A large batch can exceed that even when it is delivered, and the send is reported as a timeout.This PR sets it explicitly (default 180s,
WAHA_GOWS_STATUS_BATCH_TIMEOUT) and restores the 5000 default now that slow acks are handled where they belong.Database lock contention
The sqlite device store was opened in the default rollback-journal mode, where a single writer holds an exclusive lock that blocks every other reader and writer. During a status broadcast the outgoing send and the flood of incoming decryptions — each a session/identity/sender-key write — contend for that one lock, exhaust the 30s
busy_timeoutand surface asdatabase is locked.On a ~48k contact list this turned a ~4 minute send into a ~30 minute stall ending in a gRPC
DEADLINE_EXCEEDED. The store is now opened with_journal_mode=WAL&_synchronous=NORMAL&_txlock=immediate, mirroring the PRAGMAs already applied to the GContainer store.Caller deadline aborting a live send
A deadline on the caller side aborted a send that was already live on WhatsApp, leaving the later batches undelivered while the caller saw a plain failure and re-sent the whole status — posting it two or three times. The status send now runs on a context detached from the caller's cancellation.
Testing
Built and tested with
make all(go test ./...passes). Verified in production on a ~48k contact list: the send completes in minutes instead of stalling, with no timeout and no 429.Fixes devlikeapro/waha#2096