Skip to content

Fix large status@broadcast sends timing out - #6

Open
Elimeshi1 wants to merge 1 commit into
devlikeapro:devfrom
Elimeshi1:fix/status-large-audience
Open

Fix large status@broadcast sends timing out#6
Elimeshi1 wants to merge 1 commit into
devlikeapro:devfrom
Elimeshi1:fix/status-large-audience

Conversation

@Elimeshi1

Copy link
Copy Markdown

Sending a status to a large contact list fails with timed out waiting for message send response. Lowering the participants batch from 5000 to 500 did not fix it — it multiplied the stanza count by ten and traded the timeouts for server returned error 429.

The same list, before and after that change:

Failed to send message (...) to (batch 1/7):   timed out waiting for message send response
Failed to send message (3EB0D84EFD1BA60704693D) to (batch 3/123): server returned error 429

7 stanzas vs 123. Batch size was never the problem: what trips the rate limiting is the number of stanzas, not the number of participants inside one.

There are two independent causes behind the original timeout.

The per-batch ack window

SendRequestExtra.Timeout was never set, so whatsmeow's default of 75s applied. A large batch can exceed that even when it is delivered, and the send is reported as a timeout.

This PR sets it explicitly (default 180s, WAHA_GOWS_STATUS_BATCH_TIMEOUT) and restores the 5000 default now that slow acks are handled where they belong.

Database lock contention

The sqlite device store was opened in the default rollback-journal mode, where a single writer holds an exclusive lock that blocks every other reader and writer. During a status broadcast the outgoing send and the flood of incoming decryptions — each a session/identity/sender-key write — contend for that one lock, exhaust the 30s busy_timeout and surface as database is locked.

On a ~48k contact list this turned a ~4 minute send into a ~30 minute stall ending in a gRPC DEADLINE_EXCEEDED. The store is now opened with _journal_mode=WAL&_synchronous=NORMAL&_txlock=immediate, mirroring the PRAGMAs already applied to the GContainer store.

Caller deadline aborting a live send

A deadline on the caller side aborted a send that was already live on WhatsApp, leaving the later batches undelivered while the caller saw a plain failure and re-sent the whole status — posting it two or three times. The status send now runs on a context detached from the caller's cancellation.

Testing

Built and tested with make all (go test ./... passes). Verified in production on a ~48k contact list: the send completes in minutes instead of stalling, with no timeout and no 429.

Fixes devlikeapro/waha#2096

Sending a status to a large contact list failed with "timed out waiting
for message send response". Shrinking the participants batch from 5000 to
500 did not fix it: it multiplied the number of stanzas by ten and traded
the timeouts for "server returned error 429".

There were two independent causes.

The per-batch ack window. whatsmeow defaults to 75s, which a big status
batch can exceed even when it is delivered, so the send is reported as a
timeout. Batch size was never the problem: what trips WhatsApp's rate
limiting is the number of stanzas, not the participants inside one. Set an
explicit timeout (default 180s, WAHA_GOWS_STATUS_BATCH_TIMEOUT) and restore
the 5000 default.

Database lock contention. The sqlite device store was opened in the default
rollback-journal mode, where one writer holds an exclusive lock. During a
status broadcast the outgoing send and the flood of incoming decryptions
(session/identity/sender-key writes) contend for it, exhaust the 30s
busy_timeout and surface as "database is locked" - turning a ~4 minute send
into a ~30 minute stall ending in a gRPC DEADLINE_EXCEEDED. Open it with
WAL + synchronous=NORMAL + txlock=immediate, mirroring the GContainer store.

Also detach the status send from the caller's context. A deadline on the
caller side was aborting a send already live on WhatsApp, leaving the later
batches undelivered while the caller saw a plain failure and re-sent the
whole status - posting it two or three times.

Fixes devlikeapro/waha#2096
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant