Skip to content

Send standby status updates when the applied LSN changes - #13

Open
vnvo wants to merge 2 commits into
mainfrom
change-driven-feedback
Open

vnvo wants to merge 2 commits into
mainfrom
change-driven-feedback

Conversation

@vnvo

@vnvo vnvo commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

Problem

update_applied_lsn() only stored the position in an atomic, so on an idle stream the new position reached the server only at the next idle_wakeup_interval wakeup, server message or keepalive reply request. A consumer that acknowledges after traffic stops (for example after a downstream commit) held WAL for up to that long. Lowering idle_wakeup_interval hides this but wakes every idle connection on a fixed cadence even when nothing changed.

Change

  • A raise wakes the worker through a watch generation. The worker sends the new position once status_interval has passed since its last status update; updates in between coalesce into one status carrying the newest position.
  • The worker marks the watch seen and clears a signalled flag before reading the position, so a raise after that point always wakes it. Every access to the flag and the position on both sides is SeqCst; the lock-free core is ProgressCell.
  • Only the first raise after the worker last looked bumps the watch. Later raises cost one extra atomic load, so update_applied_lsn stays as cheap as before.
  • A position is recorded as sent only after the write succeeds.
  • Keepalive replies, idle wakeup feedback and feedback under backpressure are unchanged. No task is added outside the worker.

Tests

Protocol tests against a scripted server (paused clock):

  • one update;
  • coalesced updates;
  • an update during a send;
  • the newest position at a new session;
  • a failed send;
  • a requested-reply keepalive;
  • a completely idle connection (status at every idle wakeup, not per interval);
  • stop with the timer armed;
  • a multi-thread stress test of raises racing the worker.

New live test postgres_idle_feedback_is_change_driven: on an idle stream, confirmed_flush_lsn reaches an acknowledged commit within 2 s. It took 2.8 ms, and fails on main.

Mutants caught:

  • removing the new wait arms;
  • marking every position pending;
  • never recording the sent position;
  • never clearing the signal.

Loom model (src/client/progress.rs, run in CI with --cfg pgwire_loom): two raises concurrent with two worker cycles of mark, clear, read and send. In every interleaving, the newest raise is sent, pending, or has a wake waiting.

  • It catches clearing the signal after reading the position.
  • It found a real defect in the first commit: the fast-path load of the flag was Relaxed and could see a flag the worker had already cleared, losing the wake until the idle wakeup.
  • Fixed in 8d93267 by making that load SeqCst. update_applied_lsn is still 8.8 ns per call.

Measurements

Real clients over TCP against a scripted idle server; status interval 1 s; 60 s windows.

update_applied_lsn cost, 10M increasing calls on a live client:

Version ns per call
main 8.9
this change 8.8
without the signalled flag (not merged) 164

Load by number of sources. "main 10 s" is the current client with its 10 s idle wakeup; "main 1 s" lowers that wakeup to 1 s.

Sources Idle status updates/s (main 10 s / main 1 s / this) Idle CPU (main 10 s / main 1 s / this) Feedback delay p50 / max, ack every 5 s (main 10 s / main 1 s / this)
1 0.1 / 1.0 / 0.1 0.00% / 0.03% / 0.00% 2.0 s / 7.0 s · 41 / 81 ms · 0.4 / 0.6 ms
25 2.5 / 25.8 / 2.5 0.03% / 0.22% / 0.02% 4.6 s / 10.0 s · 442 / 883 ms · 0.3 / 0.6 ms
100 10 / 103 / 10 0.10% / 0.85% / 0.08% 4.8 s / 10.0 s · 499 / 1001 ms · 0.3 / 0.9 ms
300 30 / 310 / 30 0.18% / 2.03% / 0.22% 4.8 s / 10.0 s · 507 / 1002 ms · 0.3 / 2.7 ms

With main at a 10 s wakeup, acknowledgements were still unsent when the window ended: 1 of 13 at 1 source, rising to 169 of 3,601 at 300.

vnvo added 2 commits October 13, 2026 01:12
update_applied_lsn now wakes the worker, which sends the new position
once status_interval has passed since its last status update, also on an
idle stream. Before, an idle stream reported it only at the next idle
wakeup, server message or keepalive reply request.
The fast-path load of the signalled flag in update_applied_lsn was
Relaxed, so it could still see a flag the worker had already cleared
after reading the LSN before the raise: the raise then skipped the wake
and the worker kept waiting until its idle wakeup. Make it SeqCst, move
the lock-free core into ProgressCell, and add a loom model (run in CI)
proving a raise concurrent with send or clear is sent, pending or wakes
the worker.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant