feat(rwi): dedicated webhook runtime, event metrics, push retry and configurable queue - #267
Open
ftong2010 wants to merge 1 commit into
Open
feat(rwi): dedicated webhook runtime, event metrics, push retry and configurable queue#267ftong2010 wants to merge 1 commit into
ftong2010 wants to merge 1 commit into
Conversation
Contributor
|
Author
ok, actually i rebased with main and removed call_ringing change, too. let me update PR |
ftong2010
force-pushed
the
rwi-dedicated-worker
branch
from
September 7, 2026 10:26
94b94bd to
bc7b9e6
Compare
…ry and queue
Rebased onto current main, which already carries a fixed-count retry
loop, the event-type-aware dedup key and full-body (never-truncated)
delivery logging — this keeps those and layers the feature set on top:
- dedicated tokio runtime for the RWI webhook handler (queue drain no
longer competes with call/media workers), spawned via
utils::rwi_webhook_spawn
- configurable push retry: [proxy.locator_webhook] retries (default 0 =
single attempt, hard cap 5) with exponential backoff (200 ms base);
retryable = transport error / 5xx / 429 — supersedes main's fixed
WEBHOOK_RETRY_COUNT(3)/500 ms outer loop with the richer policy inside
send_payload
- event pipeline metrics: rwi_event_queue_{size,current} gauges,
rwi_events_{pushed,push_failed,retries}_total counters with an
event_type label, and opt-in rwi_event_queue_latency_seconds histogram
(gateway enqueue → handler dequeue, excludes the HTTP push;
[proxy.locator_webhook] track_queue_latency)
- configurable broadcast queue length: [proxy]
rwi_webhook_channel_size (default 512)
- docs: observability.md metric list + rwi_events_reference*.md config
Existing retry tests updated for the config-driven policy (retries:
Some(3) preserves the 3/4-request expectations); all 26 webhook tests
pass.
ftong2010
force-pushed
the
rwi-dedicated-worker
branch
from
September 7, 2026 10:43
bc7b9e6 to
5dba598
Compare
Author
|
@shenjinti updated and rebased with latest main |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The RWI webhook handler runs sequential, blocking-ish HTTP POSTs to the
router on the shared SIP runtime. A slow webhook consumer therefore
competes with SIP signalling, the HTTP route path and the CDR saver — and a
backpressured router can stall unrelated work. There was also no visibility
into the event pipeline (sent vs delivered vs dropped, queue depth), no retry
on transient push failures, a hardcoded queue capacity, and a gap in the
event stream:
call_ringingwas never emitted for calls whose trunkreplied with a 183 (early media), which is the majority of trunk calls.
What this PR adds
own tokio runtime (
rwi-webhookthreads), so webhook egress and anyslow-router backpressure never contend with the SIP runtime.
webhook path end-to-end (see table below).
429) are retried with exponential backoff (200 ms base, doubling, hard
cap 5); permanent 4xx are not retried.
capacity is now a config knob instead of a hardcoded constant.
in the queue (enqueued → handler dequeued). The HTTP push time is
excluded on purpose: a slow router inflates push time, not queue wait.
call_ringingfix — the event is now emitted on either provisional(183 with SDP or 180 without), exactly once per call.
Configuration
[proxy] rwi_webhook_worker_threads[proxy] rwi_webhook_channel_size[rwi_webhook] retries[rwi_webhook] track_queue_latencyrwi_event_queue_latency_seconds[rwi_webhook] timeout_msMetrics
rwi_event_enqueued_totalevent_typerwi_events_pushed_totalevent_typerwi_events_push_failed_totalevent_typerwi_events_push_retries_totalevent_typerwi_events_dropped_totalrwi_event_queue_sizerwi_event_queue_currentrwi_event_queue_latency_secondsevent_typetrack_queue_latencyVerification
SIP workers=8 Media workers=12 RWI webhook workers=2, with 2rwi-webhookthreads visible in theprocess.
rwi_event_enqueued_total209 ==rwi_events_pushed_total209,rwi_event_queue_current0 — no drops,no backlog.
20,065 pushed, queue stayed drained,
rwi_events_dropped_total0.call_ringingnow fires exactly once per call (600 calls → 600call_ringingenqueued/pushed), including 183-early-media calls thatpreviously emitted nothing.
rwi_webhook_channel_size = 1234→rwi_event_queue_size 1234.cargo check --workspace --binsclean; release image builds.Docs
docs/rwi_events_reference.md(zh) anddocs/rwi_events_reference_en.mdupdated with the new config keys and the metrics table;
docs/observability.mdgained an "RWI Events" section in the metricsreference.