docs: document the listener thread; narrow the no-threads guarantee - #69
Open
benaliabderrahmane wants to merge 9 commits into
Open
benaliabderrahmane wants to merge 9 commits into
benaliabderrahmane wants to merge 9 commits into
Conversation
The receive loop existed in four near-identical copies (drain_subscription, drain_socket in rmw_wait.cpp, and inline in rmw_take_request and rmw_take_response) that had drifted: each decided for itself whether to cap the queue, warn about an overflow, or notify a listener. drain_endpoint() is now the only receive loop. DrainTarget carries the per-endpoint policy (queue and its mutex, depth cap, accepted msg_type, shm reader cache, ignore_local, the TRANSIENT_LOCAL watermark map, the listener callback, the name to log) and the drain_target() overloads build it from each impl struct, so a call site can no longer pick its own policy by accident. Where the copies disagreed the stricter one wins: the rmw_wait drain now fires on_new_message and reports overflow, and the request/response queues are capped at SERVICE_QUEUE_DEPTH on the take path too - recorded in drain.hpp as a deliberate tradeoff. Squashed from: - refactor(drain): collapse the four socket drains into one - docs(drain): record the take-path queue cap as a deliberate tradeoff - fix(drain): restore the <cstring> include, retarget the names this PR renamed
drain_endpoint() resolved the shm descriptor and then asked whether the
datagram should be ignored. For a subscription with
ignore_local_publications set, a large same-context publication was
therefore mapped out of the sender's ring and copied into the payload
buffer before being thrown away - up to SHM ring-record size of pointless
copying per message, on a path whose whole purpose is to avoid the copy.
is_same_context() reads only WireHeader::gid, which is on the wire before
anything is resolved, so the check moves ahead of the resolve. That is
the same reasoning the TRANSIENT_LOCAL dedup above it already gives for
its own position ("Checked before the descriptor resolve so a duplicate
never maps a segment").
Behaviour is identical either way: a resolve failure and an ignore both
end in `continue`. So the new test does not fail beforehand - it covers a
branch nothing reached. The two existing ignore_local tests publish small
inline payloads, which carry no descriptor, so the shm path combined with
ignore_local had no coverage at all.
Full suite green on Jazzy: 157 tests, 0 failures.
The headline behaviour change of the unified drain - rmw_wait now fires on_new_message, where only the rmw_take path did - had no test on this branch. WaitDrainFiresOnNewMessageCallback publishes three messages, waits without ever taking, and expects the callback to have been credited three events. It fails on devel (0 events) and passes here. The total is summed and awaited so the test stays valid once a listener thread delivers it. Also corrects the SERVICE_QUEUE_DEPTH comment: services and clients do carry a QoS depth, it is just not enforced on these queues. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NUNQNo26cKRPrVXcaHZnje
on_new_request_cb and on_new_response_cb were stored by their setters and flushed once against an already-queued backlog, but no drain ever fired them again - a service or client was callback-dead the moment its registration returned. With one drain filling all three queues, the two drain_target() overloads now carry the callback. The notification also moves out of the receive loop: one call per drain that grew the queue, carrying enqueued - dropped. number_of_events is a take credit, so an overflow that pushed 100 and popped 90 reports 10, and a drain whose net gain is zero stays silent. The setters flush a queued backlog only when a callback takes over from none, because rclcpp registers twice in a row on purpose and paid the same backlog out twice. Squashed from: - fix(drain): notify services and clients, and report the batch count - fix(drain): report the queue's growth, and flush a backlog only once
No user code runs under queue_mutex any more: the setters copy the backlog size, release the queue lock and notify holding callback_mutex alone, and drain_endpoint() notifies the same way. That was only half of it. rmw_take drains before it pops, so a take from inside the callback re-enters drain_endpoint() on the thread already inside the callback, and a nested drain that gains a datagram notifies again - locking callback_mutex a second time on the same thread. With a std::mutex that is a self-deadlock, seen only when a message arrives during the callback. callback_mutex is now a std::recursive_mutex; it still keeps the callback pointer and user_data stable for the duration of a call, which is all it was ever for. ACallbackMayTakeWhileItsOwnEndpointGainsMessages has the callback publish before it takes, so the nested drain gains one every time. It hangs on the previous code and passes in ~130 ms after. Squashed from: - fix(callbacks): never run a listener callback under queue_mutex - fix(callbacks): make callback_mutex recursive so a callback can really take - test(callbacks): name the re-entrant fixture PublishThenTakeCallback Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NUNQNo26cKRPrVXcaHZnje
rclcpp's EventsExecutor never calls rmw_wait and only calls rmw_take once a listener callback has told it there is something to take. With delivery happening only inside rmw_wait that is a hang by construction: nothing drains the socket, so the callback never fires. Each context can now run one listener thread that drains a watched socket and fires its callback with no application thread involved. It starts lazily, on the first callback registration, so an executor that waits still starts no background thread (pinned against /proc/self/task). Three things make it safe to share an endpoint with rmw_wait: - listener_mutex is held across the drain and listener_unwatch() takes it, so destroying an endpoint blocks until an in-flight drain and the callback it fires have returned; the destroy paths unwatch first. - A per-endpoint drain_mutex serialises the listener against an rmw_wait/rmw_take draining the same socket, so one publisher's samples cannot be reordered by interleaved recv/push pairs. It covers the receive loop only and is released before the notification, so a callback that takes from its own endpoint re-enters drain_endpoint() without wedging on it; a callback runs holding callback_mutex alone. - Each wait set owns a delivery eventfd. The listener signals every registered one strictly after enqueueing and rmw_wait drains its own strictly before scanning its queues, which closes the lost-wakeup window without one wait set consuming another's credit. The thread wakes on socket readiness, never on a clock; it does not make deadline or liveliness enforceable. Squashed from: - feat(listener): lazily-started thread for callback-driven delivery - fix(listener): give each wait set its own delivery eventfd - fix(listener): close the shutdown/registration race and serialise drains - test(listener): wait for the event total instead of racing the listener - fix(drain): release drain_mutex before notifying, and allow a re-entrant take - docs(drain): drain_mutex covers the receive loop, not the whole drain Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NUNQNo26cKRPrVXcaHZnje
Only rmw_subscription_set_on_new_message_callback handed its socket to the listener, so a service or client under an EventsExecutor still had nobody moving its datagrams: the callback fired for the backlog its setter flushed, then never again. The two setters now watch on registration and unwatch when the callback is cleared, and rmw_destroy_service / rmw_destroy_client unwatch before tearing anything down - listener_unwatch blocks until an in-flight drain has returned, so the delete behind it cannot race one. Squashed from: - feat(listener): watch service and client sockets too - test(listener): the service and client totals race the listener too
Adding the service and client halves would have made a second and third copy of the same protocol - flush the backlog only when a callback takes over from none, call listener_watch outside callback_mutex, roll the callback back if the watch fails - and the copies had already drifted once. listener_set_callback() in listener.cpp owns it; DrainTarget already carries the fd, queue, mutexes and callback slots, so the setters keep their argument checks and hand the rest over in three lines each. The callback runs holding callback_mutex alone, a std::recursive_mutex here as in the layers below. Squashed from: - refactor(listener): one set_on_new_*_callback implementation, not three - fix(callbacks): run a listener callback under callback_mutex alone - fix(listener): follow callback_mutex to std::recursive_mutex in the shared setter Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NUNQNo26cKRPrVXcaHZnje
"No background threads" was asserted across DESIGN.md and README.md and is no longer unconditional: none for an executor that waits, one per-context listener thread for an application that registers a listener callback. Each assertion now says what it actually guarantees. DESIGN.md gains "The listener thread: delivery without a wait": why EventsExecutor deadlocks against wait-only delivery, the lazy start, the reordering and lost-wakeup hazards and what closes them (drain_mutex, the per-wait-set delivery fd), the listener_mutex lifetime rule and the restrictions it implies, that a callback may take from its own endpoint because callback_mutex is recursive, which context-wide locks exist, and what the thread deliberately is not. README, test/README and CHANGELOG follow. Squashed from: - docs: document the listener thread; narrow the no-threads guarantee - docs: fix "edge-driven", describe the per-wait-set delivery fd, drop #64's entry - docs: correct the drain-safety claim, and log the two blocker fixes - docs: sweep the claims this stack invalidated but the prose pass missed - docs: a callback may now take from its own endpoint - docs: callback_mutex is recursive; listener_mutex is not the only context lock Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NUNQNo26cKRPrVXcaHZnje
benaliabderrahmane
force-pushed
the
docs/listener-thread
branch
from
September 18, 2026 12:44
a32bc18 to
7c8005e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
"No background threads" was asserted across
DESIGN.mdandREADME.mdand is nolonger unconditional, so each of those assertions now says what it actually
guarantees: no background thread for an executor that waits, and one per-context
listener thread for an application that registers a listener callback. The
### No background threads, and whyheading keeps its name; the section under itnow states the exception instead of an absolute. The property that matters to a
reader choosing this RMW — that a
SingleThreadedExecutororMultiThreadedExecutorprocess starts nothing — is unchanged, and saying soprecisely beats leaving a claim a reader can disprove with
ps.DESIGN.mdgains "The listener thread: delivery without a wait", written inthe same shape as the doorbell section it borrows its ordering proof from: why
EventsExecutordeadlocks against wait-only delivery, the lazy start, the twohazards that come with a watched endpoint still sitting in a wait set, the
listener_mutexlifetime rule and the restrictions it implies, and what thethread deliberately is not — it wakes on socket readiness, never on a clock, so
it does not make deadline or liveliness enforceable.
Both hazards are documented, where an earlier draft of this section asserted the
first one away. Reordering comes first:
drain_endpointrecvs outsidequeue_mutexand pushes inside it, so the listener and anrmw_wait/rmw_takedraining the same socket can interleave as recv A / recv B / push B / push A, and
a per-endpoint
drain_mutexheld for the whole drain serialises them. Callingthat "fine, since both use non-blocking
recvinto the same mutex-protectedqueue" was wrong — the queue mutex prevents corruption, not reordering — so this
is a correction, not a rewording. The lost wakeup comes second, and keeps the
delivery_fdordering pair and its "there is no order in which it is missed"proof, now with a subsection on why that fd belongs to the wait set rather
than the context: reading an eventfd drains its whole counter, so a single shared
fd is a single credit, and both ways it went missing are written down — a wait set
that gained no work of its own still consumed it and discarded it, and a wait
already blocked when the listener started had armed nothing at all. The ordering
rule that comes with a per-wait-set fd is written down too:
rmw_destroy_wait_setderegisters under
listener_mutexstrictly before it closes the fd, or thelistener writes eight bytes into a recycled fd number.
The lifetime rule is stated as context-wide, not endpoint-local, because there
is one
listener_mutexper context: a callback must not destroy, register acallback on, or clear the callback of any endpoint in the context, and must
not create or destroy a wait set either, since that mutex also guards the
delivery-fd list. That same context-wide mutex is held across the drain and taken
again by
listener_unwatch, which is why an unwatch blocks on whatever drain isin flight rather than specifically on a drain of the endpoint being unwatched. The
second new lock adds a second restriction — a callback must not
rmw_takeits ownendpoint, because it fires inside
drain_endpointwhile that endpoint'sdrain_mutexis held.The unsupported-features list separates two things that were conflated under
"event callbacks": QoS status events, refused for every
rmw_event_type_twith the reasoning for why refusing beats reporting success for an event that
never arrives, and the listener callbacks, which are supported and are what an
EventsExecutoractually needs. The same split lands inREADME.md's limitationsparagraph, which also gains a Callbacks row in the architecture table. The
file table gains
drain.cppandlistener.cpp, andrmw_event.cppis no longerdescribed as "event stubs".
Two statements elsewhere in
DESIGN.mdstill asserted the old world and arecorrected here as well. The resource profile was wrong on three rows, not just the
thread count: a listener adds an epoll instance and a wake
eventfdper context,and every wait set now owns a delivery
eventfd, so the descriptor undercount isits own fix. And the wait set's In-process concurrency section still opened
"All socket I/O happens inside
rmw_wait()", which the listener thread makesfalse; it now says where I/O actually runs, names both new locks, and is explicit
that two callers draining the same endpoint do serialise on its
drain_mutex—that is what keeps a publisher's samples in order — while different entities still
do not contend.
test/README.mdgains a section fortest_rmw_listener_callbacks.cppin theexisting per-file table format, and its "what is NOT tested" table now covers QoS
status events and notes that the listener is not a timer.
CHANGELOG.mdgains anUnreleasedentry in the existing Added/Changed/Fixedstructure. It covers this stack only: #64's event-API changes are not an ancestor
of this branch and carry their own entry there, so merging this does not ship a
changelog for absent code.
Is this user-facing behavior change?
How was this tested?
Documentation only; no source change. Suite re-run on jazzy to confirm
nothing moved: 176 tests, 0 failures.
Did you use Generative AI?
Additional Information
Two pre-existing inaccuracies in
test/README.mdare left alone rather thanfixed here, to keep this PR to the listener change: it still says "All tests run
in a single process" although
test_rmw_cross_process.cppexists, and it has nosection for
test_rmw_event.cpp(added in #61). Happy to fix either in aseparate docs PR.
The PR template's own note says "CI builds and tests run on jazzy, kilted and
rolling" — lyrical was added to the matrix in #17, so that line is now stale too.