Skip to content

docs: document the listener thread; narrow the no-threads guarantee - #69

Open
benaliabderrahmane wants to merge 9 commits into
develfrom
docs/listener-thread
Open

benaliabderrahmane wants to merge 9 commits into
develfrom
docs/listener-thread

Conversation

@benaliabderrahmane

@benaliabderrahmane benaliabderrahmane commented Sep 8, 2026 •

Copy link
Copy Markdown
Owner

Description

"No background threads" was asserted across DESIGN.md and README.md and is no
longer unconditional, so each of those assertions now says what it actually
guarantees: no background thread for an executor that waits, and one per-context
listener thread for an application that registers a listener callback. The
### No background threads, and why heading keeps its name; the section under it
now states the exception instead of an absolute. The property that matters to a
reader choosing this RMW — that a SingleThreadedExecutor or
MultiThreadedExecutor process starts nothing — is unchanged, and saying so
precisely beats leaving a claim a reader can disprove with ps.

DESIGN.md gains "The listener thread: delivery without a wait", written in
the same shape as the doorbell section it borrows its ordering proof from: why
EventsExecutor deadlocks against wait-only delivery, the lazy start, the two
hazards that come with a watched endpoint still sitting in a wait set, the
listener_mutex lifetime rule and the restrictions it implies, and what the
thread deliberately is not — it wakes on socket readiness, never on a clock, so
it does not make deadline or liveliness enforceable.

Both hazards are documented, where an earlier draft of this section asserted the
first one away. Reordering comes first: drain_endpoint recvs outside
queue_mutex and pushes inside it, so the listener and an rmw_wait/rmw_take
draining the same socket can interleave as recv A / recv B / push B / push A, and
a per-endpoint drain_mutex held for the whole drain serialises them. Calling
that "fine, since both use non-blocking recv into the same mutex-protected
queue" was wrong — the queue mutex prevents corruption, not reordering — so this
is a correction, not a rewording. The lost wakeup comes second, and keeps the
delivery_fd ordering pair and its "there is no order in which it is missed"
proof, now with a subsection on why that fd belongs to the wait set rather
than the context: reading an eventfd drains its whole counter, so a single shared
fd is a single credit, and both ways it went missing are written down — a wait set
that gained no work of its own still consumed it and discarded it, and a wait
already blocked when the listener started had armed nothing at all. The ordering
rule that comes with a per-wait-set fd is written down too: rmw_destroy_wait_set
deregisters under listener_mutex strictly before it closes the fd, or the
listener writes eight bytes into a recycled fd number.

The lifetime rule is stated as context-wide, not endpoint-local, because there
is one listener_mutex per context: a callback must not destroy, register a
callback on, or clear the callback of any endpoint in the context, and must
not create or destroy a wait set either, since that mutex also guards the
delivery-fd list. That same context-wide mutex is held across the drain and taken
again by listener_unwatch, which is why an unwatch blocks on whatever drain is
in flight rather than specifically on a drain of the endpoint being unwatched. The
second new lock adds a second restriction — a callback must not rmw_take its own
endpoint, because it fires inside drain_endpoint while that endpoint's
drain_mutex is held.

The unsupported-features list separates two things that were conflated under
"event callbacks": QoS status events, refused for every rmw_event_type_t
with the reasoning for why refusing beats reporting success for an event that
never arrives, and the listener callbacks, which are supported and are what an
EventsExecutor actually needs. The same split lands in README.md's limitations
paragraph, which also gains a Callbacks row in the architecture table. The
file table gains drain.cpp and listener.cpp, and rmw_event.cpp is no longer
described as "event stubs".

Two statements elsewhere in DESIGN.md still asserted the old world and are
corrected here as well. The resource profile was wrong on three rows, not just the
thread count: a listener adds an epoll instance and a wake eventfd per context,
and every wait set now owns a delivery eventfd, so the descriptor undercount is
its own fix. And the wait set's In-process concurrency section still opened
"All socket I/O happens inside rmw_wait()", which the listener thread makes
false; it now says where I/O actually runs, names both new locks, and is explicit
that two callers draining the same endpoint do serialise on its drain_mutex —
that is what keeps a publisher's samples in order — while different entities still
do not contend.

test/README.md gains a section for test_rmw_listener_callbacks.cpp in the
existing per-file table format, and its "what is NOT tested" table now covers QoS
status events and notes that the listener is not a timer.

CHANGELOG.md gains an Unreleased entry in the existing Added/Changed/Fixed
structure. It covers this stack only: #64's event-API changes are not an ancestor
of this branch and carry their own entry there, so merging this does not ship a
changelog for absent code.

Is this user-facing behavior change?

How was this tested?

Documentation only; no source change. Suite re-run on jazzy to confirm
nothing moved: 176 tests, 0 failures.

Did you use Generative AI?

Additional Information

Two pre-existing inaccuracies in test/README.md are left alone rather than
fixed here, to keep this PR to the listener change: it still says "All tests run
in a single process" although test_rmw_cross_process.cpp exists, and it has no
section for test_rmw_event.cpp (added in #61). Happy to fix either in a
separate docs PR.

The PR template's own note says "CI builds and tests run on jazzy, kilted and
rolling" — lyrical was added to the matrix in #17, so that line is now stale too.

Abderahmane BENALI and others added 9 commits September 18, 2026 14:36
The receive loop existed in four near-identical copies (drain_subscription,
drain_socket in rmw_wait.cpp, and inline in rmw_take_request and
rmw_take_response) that had drifted: each decided for itself whether to
cap the queue, warn about an overflow, or notify a listener.

drain_endpoint() is now the only receive loop. DrainTarget carries the
per-endpoint policy (queue and its mutex, depth cap, accepted msg_type,
shm reader cache, ignore_local, the TRANSIENT_LOCAL watermark map, the
listener callback, the name to log) and the drain_target() overloads build
it from each impl struct, so a call site can no longer pick its own policy
by accident. Where the copies disagreed the stricter one wins: the
rmw_wait drain now fires on_new_message and reports overflow, and the
request/response queues are capped at SERVICE_QUEUE_DEPTH on the take path
too - recorded in drain.hpp as a deliberate tradeoff.

Squashed from:
- refactor(drain): collapse the four socket drains into one
- docs(drain): record the take-path queue cap as a deliberate tradeoff
- fix(drain): restore the <cstring> include, retarget the names this PR renamed
drain_endpoint() resolved the shm descriptor and then asked whether the
datagram should be ignored. For a subscription with
ignore_local_publications set, a large same-context publication was
therefore mapped out of the sender's ring and copied into the payload
buffer before being thrown away - up to SHM ring-record size of pointless
copying per message, on a path whose whole purpose is to avoid the copy.

is_same_context() reads only WireHeader::gid, which is on the wire before
anything is resolved, so the check moves ahead of the resolve. That is
the same reasoning the TRANSIENT_LOCAL dedup above it already gives for
its own position ("Checked before the descriptor resolve so a duplicate
never maps a segment").

Behaviour is identical either way: a resolve failure and an ignore both
end in `continue`. So the new test does not fail beforehand - it covers a
branch nothing reached. The two existing ignore_local tests publish small
inline payloads, which carry no descriptor, so the shm path combined with
ignore_local had no coverage at all.

Full suite green on Jazzy: 157 tests, 0 failures.
The headline behaviour change of the unified drain - rmw_wait now fires
on_new_message, where only the rmw_take path did - had no test on this
branch. WaitDrainFiresOnNewMessageCallback publishes three messages, waits
without ever taking, and expects the callback to have been credited three
events. It fails on devel (0 events) and passes here. The total is summed
and awaited so the test stays valid once a listener thread delivers it.

Also corrects the SERVICE_QUEUE_DEPTH comment: services and clients do
carry a QoS depth, it is just not enforced on these queues.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUNQNo26cKRPrVXcaHZnje
on_new_request_cb and on_new_response_cb were stored by their setters and
flushed once against an already-queued backlog, but no drain ever fired
them again - a service or client was callback-dead the moment its
registration returned. With one drain filling all three queues, the two
drain_target() overloads now carry the callback.

The notification also moves out of the receive loop: one call per drain
that grew the queue, carrying enqueued - dropped. number_of_events is a
take credit, so an overflow that pushed 100 and popped 90 reports 10, and
a drain whose net gain is zero stays silent. The setters flush a queued
backlog only when a callback takes over from none, because rclcpp
registers twice in a row on purpose and paid the same backlog out twice.

Squashed from:
- fix(drain): notify services and clients, and report the batch count
- fix(drain): report the queue's growth, and flush a backlog only once
No user code runs under queue_mutex any more: the setters copy the backlog
size, release the queue lock and notify holding callback_mutex alone, and
drain_endpoint() notifies the same way.

That was only half of it. rmw_take drains before it pops, so a take from
inside the callback re-enters drain_endpoint() on the thread already
inside the callback, and a nested drain that gains a datagram notifies
again - locking callback_mutex a second time on the same thread. With a
std::mutex that is a self-deadlock, seen only when a message arrives
during the callback. callback_mutex is now a std::recursive_mutex; it
still keeps the callback pointer and user_data stable for the duration of
a call, which is all it was ever for.

ACallbackMayTakeWhileItsOwnEndpointGainsMessages has the callback publish
before it takes, so the nested drain gains one every time. It hangs on the
previous code and passes in ~130 ms after.

Squashed from:
- fix(callbacks): never run a listener callback under queue_mutex
- fix(callbacks): make callback_mutex recursive so a callback can really take
- test(callbacks): name the re-entrant fixture PublishThenTakeCallback

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUNQNo26cKRPrVXcaHZnje
rclcpp's EventsExecutor never calls rmw_wait and only calls rmw_take once
a listener callback has told it there is something to take. With delivery
happening only inside rmw_wait that is a hang by construction: nothing
drains the socket, so the callback never fires.

Each context can now run one listener thread that drains a watched socket
and fires its callback with no application thread involved. It starts
lazily, on the first callback registration, so an executor that waits
still starts no background thread (pinned against /proc/self/task).

Three things make it safe to share an endpoint with rmw_wait:
- listener_mutex is held across the drain and listener_unwatch() takes it,
  so destroying an endpoint blocks until an in-flight drain and the
  callback it fires have returned; the destroy paths unwatch first.
- A per-endpoint drain_mutex serialises the listener against an
  rmw_wait/rmw_take draining the same socket, so one publisher's samples
  cannot be reordered by interleaved recv/push pairs. It covers the
  receive loop only and is released before the notification, so a
  callback that takes from its own endpoint re-enters drain_endpoint()
  without wedging on it; a callback runs holding callback_mutex alone.
- Each wait set owns a delivery eventfd. The listener signals every
  registered one strictly after enqueueing and rmw_wait drains its own
  strictly before scanning its queues, which closes the lost-wakeup window
  without one wait set consuming another's credit.

The thread wakes on socket readiness, never on a clock; it does not make
deadline or liveliness enforceable.

Squashed from:
- feat(listener): lazily-started thread for callback-driven delivery
- fix(listener): give each wait set its own delivery eventfd
- fix(listener): close the shutdown/registration race and serialise drains
- test(listener): wait for the event total instead of racing the listener
- fix(drain): release drain_mutex before notifying, and allow a re-entrant take
- docs(drain): drain_mutex covers the receive loop, not the whole drain

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUNQNo26cKRPrVXcaHZnje
Only rmw_subscription_set_on_new_message_callback handed its socket to the
listener, so a service or client under an EventsExecutor still had nobody
moving its datagrams: the callback fired for the backlog its setter
flushed, then never again. The two setters now watch on registration and
unwatch when the callback is cleared, and rmw_destroy_service /
rmw_destroy_client unwatch before tearing anything down - listener_unwatch
blocks until an in-flight drain has returned, so the delete behind it
cannot race one.

Squashed from:
- feat(listener): watch service and client sockets too
- test(listener): the service and client totals race the listener too
Adding the service and client halves would have made a second and third
copy of the same protocol - flush the backlog only when a callback takes
over from none, call listener_watch outside callback_mutex, roll the
callback back if the watch fails - and the copies had already drifted
once. listener_set_callback() in listener.cpp owns it; DrainTarget already
carries the fd, queue, mutexes and callback slots, so the setters keep
their argument checks and hand the rest over in three lines each. The
callback runs holding callback_mutex alone, a std::recursive_mutex here as
in the layers below.

Squashed from:
- refactor(listener): one set_on_new_*_callback implementation, not three
- fix(callbacks): run a listener callback under callback_mutex alone
- fix(listener): follow callback_mutex to std::recursive_mutex in the shared setter

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUNQNo26cKRPrVXcaHZnje
"No background threads" was asserted across DESIGN.md and README.md and
is no longer unconditional: none for an executor that waits, one
per-context listener thread for an application that registers a listener
callback. Each assertion now says what it actually guarantees.

DESIGN.md gains "The listener thread: delivery without a wait": why
EventsExecutor deadlocks against wait-only delivery, the lazy start, the
reordering and lost-wakeup hazards and what closes them (drain_mutex, the
per-wait-set delivery fd), the listener_mutex lifetime rule and the
restrictions it implies, that a callback may take from its own endpoint
because callback_mutex is recursive, which context-wide locks exist, and
what the thread deliberately is not. README, test/README and CHANGELOG
follow.

Squashed from:
- docs: document the listener thread; narrow the no-threads guarantee
- docs: fix "edge-driven", describe the per-wait-set delivery fd, drop #64's entry
- docs: correct the drain-safety claim, and log the two blocker fixes
- docs: sweep the claims this stack invalidated but the prose pass missed
- docs: a callback may now take from its own endpoint
- docs: callback_mutex is recursive; listener_mutex is not the only context lock

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUNQNo26cKRPrVXcaHZnje
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant