Skip to content

Retry failed status batches instead of failing the whole send - #7

Open
Elimeshi1 wants to merge 2 commits into
devlikeapro:devfrom
Elimeshi1:fix/status-batch-retry
Open

Retry failed status batches instead of failing the whole send#7
Elimeshi1 wants to merge 2 commits into
devlikeapro:devfrom
Elimeshi1:fix/status-batch-retry

Conversation

@Elimeshi1

@Elimeshi1 Elimeshi1 commented Aug 5, 2026

Copy link
Copy Markdown

Follow-up to #6, and built on that branch, so this PR carries its commit too — GitHub cannot use a fork branch as the base of a cross-fork PR. Only the second commit belongs to this PR (Retry failed status batches instead of failing the whole send); the diff reduces to it once #6 is merged.

Once a large status send works at all, what is left is how it behaves when one batch fails.

An explicit participant list bypassed batching

participantsBatchSize := statusParticipantsBatchSize
if len(extra.Participants) > 0 {
    participantsBatchSize = len(extra.Participants)
}

When the caller supplied the recipients, the batch size became the size of that list — so the whole thing went out as a single batch. That is exactly the large send that needs splitting. The batch size now always comes from the config.

Transient failures were not retried

An ack timeout or a 429 is a temporary condition, but the loop gave up on the batch immediately. Batches are now retried (default 2 attempts, 5s then 15s, WAHA_GOWS_STATUS_BATCH_MAX_RETRIES / WAHA_GOWS_STATUS_BATCH_RETRY_BACKOFF).

Only those two conditions are retried. Any other server error is permanent, and retrying it would burn the budget and delay the batches behind it.

Retrying is safe: extra.ID is generated once for the whole status and reused by every batch and every attempt, and WhatsApp deduplicates by (sender, message ID), so a recipient that already received the batch does not see the status twice.

One failed batch failed the whole send

A single failed batch made the call return an error even when every other batch had been delivered — and a status that reached anyone is already live on WhatsApp. The caller read that error as "reached nobody" and sent the status again, posting it two or three times.

The call now fails only when no batch got through, and logs a report at batch granularity:

Status (3EB0...) delivery report: 6/7 batches delivered (ok=[1 2 3 4 5 7], failed=[6]), 12 ignored

Batch numbers, not phone numbers — the log stays readable on a 48k list.

Batches were sent back to back

A fixed delay between batches (default 1.5s, WAHA_GOWS_STATUS_BATCH_DELAY) keeps the cadence steady rather than bursty. At the 5000 default this is about 15s across a 48k list.

Open question: a partial delivery is not visible to the caller

Returning success on a partial delivery is the lesser of two evils, not a complete answer. Before this change the caller got a wrong answer — an error for a send that had partly succeeded — and re-sent the whole status, posting it twice. Now it gets an incomplete one: the send succeeded, without saying how much of it.

There is nowhere to put that. gRPC has no partial-success status, and MessageResponse is id / timestamp / message, so a caller cannot distinguish 48k delivered from 5k delivered. Only the log line above carries it.

An additive, backwards-compatible option would be:

message MessageResponse {
  string id = 1;
  int64 timestamp = 2;
  Json message = 3;
  // Set only for status@broadcast sends that were split into batches.
  StatusDelivery status_delivery = 4;
}

message StatusDelivery {
  int32 batches_total = 1;
  int32 batches_delivered = 2;
  int32 participants_not_delivered = 3;
}

Deliberately not part of this PR — it touches the proto and the API side, and the shape is your call. Happy to open it separately if you want it.

Testing

Built and tested with make all (go test ./... passes).

The behaviour here has been running in production on a ~48k contact list — batching an explicit list, spacing the batches out, retrying, and returning success on a partial delivery. Two details were tightened for this PR and are not covered by that: the retry budget is 2 attempts rather than 3, and only a 429 is retried where the deployed version retried every server error.

Sending a status to a large contact list failed with "timed out waiting
for message send response". Shrinking the participants batch from 5000 to
500 did not fix it: it multiplied the number of stanzas by ten and traded
the timeouts for "server returned error 429".

There were two independent causes.

The per-batch ack window. whatsmeow defaults to 75s, which a big status
batch can exceed even when it is delivered, so the send is reported as a
timeout. Batch size was never the problem: what trips WhatsApp's rate
limiting is the number of stanzas, not the participants inside one. Set an
explicit timeout (default 180s, WAHA_GOWS_STATUS_BATCH_TIMEOUT) and restore
the 5000 default.

Database lock contention. The sqlite device store was opened in the default
rollback-journal mode, where one writer holds an exclusive lock. During a
status broadcast the outgoing send and the flood of incoming decryptions
(session/identity/sender-key writes) contend for it, exhaust the 30s
busy_timeout and surface as "database is locked" - turning a ~4 minute send
into a ~30 minute stall ending in a gRPC DEADLINE_EXCEEDED. Open it with
WAL + synchronous=NORMAL + txlock=immediate, mirroring the GContainer store.

Also detach the status send from the caller's context. A deadline on the
caller side was aborting a send already live on WhatsApp, leaving the later
batches undelivered while the caller saw a plain failure and re-sent the
whole status - posting it two or three times.

Fixes devlikeapro/waha#2096
Three related changes to how a status broadcast handles a batch that fails.

Retry transient failures. An ack timeout or a 429 is a temporary condition,
but the loop gave up on the batch immediately. Retry it (default 2 attempts,
5s then 15s). Only those two conditions are retried: any other server error
is permanent and would just burn the budget and delay the batches behind it.
Retrying is safe because the message ID is generated once for the whole
status and reused by every batch and attempt, and WhatsApp deduplicates by
(sender, message ID).

Deliver best effort. A single failed batch failed the entire call, even when
the other batches were already delivered - and a status that reached anyone
is live on WhatsApp. The caller read that error as "reached nobody" and sent
the status again, posting it two or three times. Now the call fails only when
no batch got through, and reports which batch numbers were delivered.

Space the batches out. They were sent back to back with no gap; a fixed delay
(default 1.5s) keeps the cadence steady rather than bursty.

All values are configurable: WAHA_GOWS_STATUS_BATCH_DELAY,
WAHA_GOWS_STATUS_BATCH_MAX_RETRIES, WAHA_GOWS_STATUS_BATCH_RETRY_BACKOFF.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant