Skip to content

Observe how long acknowledged positions wait before the server hears of them - #14

Open
vnvo wants to merge 3 commits into
change-driven-feedbackfrom
feedback-observer
Open

vnvo wants to merge 3 commits into
change-driven-feedbackfrom
feedback-observer

Conversation

@vnvo

@vnvo vnvo commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Stacked on #13 (base change-driven-feedback).

Problem

A consumer acknowledges positions with update_applied_lsn. It cannot tell how long an acknowledged position waited before the server heard of it, so it has nothing to alert on when feedback is slow or stuck.

Change

  • ReplicationConfig::with_feedback_observer(FeedbackObserver): a callback called after each successful standby status update that advanced the reported position. It receives FeedbackSent { pending_for }, the time from the oldest raise of the applied position that this update is the first to report until the write succeeded. It is never called for an update that repeats the position (keepalive replies, idle wakeups).
  • Coalesced raises keep the oldest start.
  • A raise that races the report of a write keeps a start of its own: its own raise time, or the write's load time, so its wait is never understated. A second loom model checks this.
  • A failed write keeps the start. ReplicationClient::feedback_pending_since() (readable after the session ended) and carry_feedback_pending(since) let a caller carry it into a replacement session.
  • No clock is read without an observer, and only for the first raise after a report.
  • No positions or labels are exposed.

Tests

  • Paused-clock unit tests:
    • the reported wait (0 when due at once, 900 ms when it waited for the interval);
    • the oldest of coalesced raises;
    • no observation for keepalive replies, idle wakeups or a non-raising update;
    • a failed write carried into the next session (at least 2 s);
    • no timing without an observer;
    • a raise during a write;
    • a repeated position completing while a raise is pending (no early report).
  • Loom: a raise concurrent with a report keeps a pending start.
  • Live integration test: the observer reports the idle acknowledgement's wait (under 2 s).
  • Mutants caught:
    • not re-arming a racing raise (loom and unit);
    • a raise recording no start;
    • reporting writes that repeat the position.

…hears of them

ReplicationConfig::with_feedback_observer takes a callback, called after
each successful standby status update that advanced the reported
position, with how long that position waited since the oldest raise it
is the first to report. Coalesced raises keep the oldest start; a failed
write keeps it, and feedback_pending_since / carry_feedback_pending carry
it into a replacement session. No clock is read without an observer or
for raises that do not advance the position; positions are not exposed.
vnvo added 2 commits October 13, 2026 21:48
cargo-semver-checks against 0.4.1 found that the observer's dyn Fn made
ReplicationConfig lose UnwindSafe and RefUnwindSafe. The callback is only
called by the worker, so implement both for FeedbackObserver, and test the
public types' auto traits.
Use a compare-exchange loop, which also builds at the MSRV, and test that a
carried pending start keeps the oldest.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant