ci: run the suite under ASan/UBSan and TSan - #71
Open
benaliabderrahmane wants to merge 2 commits into
Open
benaliabderrahmane wants to merge 2 commits into
benaliabderrahmane wants to merge 2 commits into
Conversation
This was referenced Sep 8, 2026
registry_remove and registry_cleanup_stale both CAS a slot's state to
ENTRY_EMPTY and only afterwards call teardown_slot to zero the payload.
try_add_once claims any slot whose state is ENTRY_EMPTY, so a concurrent
add can win the slot in that gap, write its own entry, and have the
remover's memset land on top of it.
The result passes every seqlock check: the sequence counter is even and
the snapshot is self-consistent, but the slot holds a real entity type
with an empty node_name and pid == 0. Two distinct manifestations, one
per entity type of the racing adder:
- A corrupted ENTRY_NODE makes rmw_get_node_names return an empty
name, which rcl rejects outright ("empty node name returned by the
RMW layer, at ./src/rcl/graph.c:360") — every graph query in the
process fails, not just the affected node.
- A corrupted ENTRY_SUBSCRIPTION has its socket_path zeroed, and
rmw_publisher.cpp skips destinations with an empty path, so the
endpoint is dropped from every publisher's send list with no error
and no log line at all.
Neither self-heals while the owning entity lives, and cleanup_stale
refuses to reclaim any slot with pid == 0, so an owner that then exits
ungracefully leaks the slot for the life of the shm segment — the state
that requires deleting /dev/shm/ros2_uds_<domain> to recover.
The interleaving where teardown overlaps write_slot_payload is also
reachable and leaves a genuinely torn payload: the four seq bumps
interleave to an even count, so the seqlock's single-writer assumption
is violated rather than merely bypassed.
Fix: claim the slot into ENTRY_RESERVED instead of ENTRY_EMPTY, erase
the payload, and publish ENTRY_EMPTY only afterwards. ENTRY_RESERVED is
already invisible to readers and unclaimable by try_add_once, so the
payload has exactly one writer at all times. registry_remove also has to
reject ENTRY_RESERVED on entry, otherwise two concurrent removers both
CAS RESERVED->RESERVED and both run the teardown; without that guard the
existing ConcurrentRemoveOfSameSlotIsIdempotent test fails immediately.
The socket unlink and the stale-reclaim log line are moved out of the
claimed window. A process killed between the claim and the publish
strands the slot in ENTRY_RESERVED, which nothing reclaims — the same
hazard try_add_once already carries — so the window is kept to the
payload memset alone rather than spanning a syscall and a log write.
Verified: two new tests fail on the unfixed code (11709 corrupt
observations of 245068; 3724 of 32000 owned slots clobbered) and pass
after. ThreadSanitizer goes from 19 writer<->writer races on the slot
payload (teardown_slot vs write_slot_payload, teardown_slot vs itself)
to zero, leaving only the inherent reader-side seqlock reports.
Two jobs, because ASan and TSan cannot be linked into one binary. TSan runs with halt_on_error=1 and a suppression for the seqlock READER only (race:snapshot_slot), so a writer-vs-writer race on a registry slot stays fully reported - which is how the registry fix below this commit was found. The ASLR step reports the vm.mmap_rnd_bits actually in force and warns when it is one the runtimes cannot map around: the container is unprivileged, so the previous sysctl -w silently did nothing. Squashed from: - ci: run the suite under ASan/UBSan and TSan - ci(sanitizers): suppress the seqlock reader, correct the detect_leaks note - ci(sanitizers): stop naming tests that do not exist on this branch - ci(sanitizers): report the real vm.mmap_rnd_bits instead of pretending to set it Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NUNQNo26cKRPrVXcaHZnje
benaliabderrahmane
force-pushed
the
ci/sanitizers
branch
from
September 18, 2026 12:44
8661dc9 to
a578b27
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Runs the existing suite under sanitizers. Two jobs, because ASan and TSan cannot be
linked into the same binary and so cannot share a build tree:
sanitize (address,undefined)— ASan + UBSansanitize (thread)— TSanBuild-and-test can only fail on behavior a test asserts. The bugs this codebase
actually produces are the other kind: the registry is a lock-free seqlock over
shared memory, delivery is a level-triggered
epollwhose git history is largely aseries of lost-wakeup fixes, and endpoint pointers are handed between a dispatch
loop and whatever thread is destroying an endpoint. A missing fence, a
use-after-free on a destroy path, or an unsigned overflow in a timeout
accumulation are all invisible to a green suite and all visible to a sanitizer.
Is this user-facing behavior change?
No. CI configuration only; no source, build or packaging change. Nothing shipped is
built with sanitizers. The diff is two new files:
.github/workflows/sanitizers.ymland
.github/tsan.supp.How was this tested?
This PR is its own test, and the TSan job is red — on a real, pre-existing
bug in
devel. That is the workflow working, not the workflow being broken.sanitize (address,undefined)— pass.sanitize (thread)— fail, ontest_registry_concurrent:A write-write race on a slot payload:
registry_removeCASesstatetoENTRY_EMPTYbeforeteardown_slotzeroes the payload, so a concurrenttry_add_oncecan claim the slot in that gap and both threads write it at once.The seqlock does not help — it serializes readers against a writer, not two
writers against each other. Reproduced locally on jazzy, the writer side came
in two pairings:
write_slot_payloadagainstteardown_slot, andteardown_slotagainst itself.
This is already diagnosed and fixed on an unmerged branch,
fix/registry-teardown-before-slot-release(a05ba3f, "erase a removed slot'spayload before publishing it EMPTY"). That commit message documents the two
user-visible manifestations — a corrupted
ENTRY_NODEthat makes every graphquery in the process fail, and a corrupted
ENTRY_SUBSCRIPTIONwhose zeroedsocket_pathsilently drops the endpoint from every publisher's send list.That one stack is not the whole picture, though.
halt_on_error=1stops the runat its first report, so the job shows one race and exits; fixing that one would
simply expose the next. Measured on
test_registry_concurrentwith thesuppression removed: 27 reports and 10
snapshot_slotframes — the readerside. TSan has no seqlock model.
snapshot_slotcopies a slot's payload withplain, unsynchronised loads and validates the copy against an atomic sequence
counter, retrying if the writer touched the slot meanwhile; that torn read is the
design, and the comment above the function says so, but TSan reports every
reader/writer pair it sees.
.github/tsan.supp— one entry,race:snapshot_slot— silences the reader side only: 18 reports and 0
snapshot_slotframes withit, and the genuine writer-vs-writer pair (
write_slot_payloadagainstteardown_slot, which a seqlock does not permit) still fully reported. An earlierrevision of this PR said that if reader-side reports ever surfaced, a documented
suppressions file was the right answer rather than loosening the job; they
surfaced, and this is that file. The suppression is what makes the remaining
redness mean something.
sanitize (address,undefined)was additionally verified locally against thisbranch's sources using the exact flag string and option set the workflow exports:
every test binary clean.
Did you use Generative AI?
Additional Information
Four details that are the actual content of this PR, rather than boilerplate:
-fno-sanitize-recover=allis the point of the ASan job. UBSan's default isto print the diagnostic and keep going, so the test still passes and the finding
scrolls past in a log nobody reads. With this flag, undefined behavior aborts and
the job fails.
.github/tsan.suppsuppresses the seqlock reader, and nothing else. Everyentry in that file is a place TSan cannot see the synchronisation, not a place we
have decided to tolerate a race — which is why it has exactly one line of content.
Two threads writing the same slot at once is a genuine bug a seqlock does not
permit, and those reports are deliberately left visible. Keeping the file this
short is the whole judgment: it is the difference between a job that is red for
one stated reason and a job that is red for an unreadable mix of them.
detect_leaksis left on, but it covers less than this PR first claimed.LeakSanitizer tracks heap allocations and nothing else: a program that leaks an
epollfd and aneventfdand detaches a thread that never joins exits 0 underit, while an unreachable heap allocation exits 1 — both checked against controls
on the same toolchain. So it does not prove a background thread is joined or its
descriptors closed. Descriptor and thread lifetime need explicit test coverage,
and the workflow comment now says so rather than naming tests: the ones that
would cover it live in
test_rmw_listener_callbacks.cpp, which thelistener-callback stack adds and which does not exist on
devel, this branch'sbase.
detect_leaksremains the most likely source ofthird-party noise; if a ROS dependency leaks, the fix is a suppressions file
rather than turning the check off — a precedent this PR now sets.
halt_on_error=1,second_deadlock_stack=1andsuppressions=.github/tsan.supp. The first makes a race fail the job instead ofbeing counted and ignored, and — as above — ends the run there. The second only
widens a report rather than enabling anything: libtsan has deadlock detection on
by default, and the flag makes a lock-order inversion name both lock sites instead
of one, verified against a lock-order-inversion control. The lock order it reports
on is real — the callback setters take
callback_mutexthenqueue_mutex, andgetting that pair out of order deadlocks.
Two workarounds, not preferences — one a container setting, one a step that runs
inside it.
--shm-size=1g, because the default Docker/dev/shmis 64MB while onedomain registry mmap is ~37MB that
rmw_shutdownnevershm_unlinks, so teststhat init multiple domains across processes exhaust the tmpfs and die with
SIGBUS. Andsysctl -w vm.mmap_rnd_bits=28, because Ubuntu 24.04 raised ASLRentropy past what the ASan and TSan shadow mappings can accommodate ("unexpected
memory mapping" before
main()runs); it fails soft, because an unprivilegedcontainer cannot write sysctls at all and the runner image may not need it.
One distro, not the four-distro matrix: these findings are properties of our own
sources. Adding a distro axis is a two-line change if it ever earns the cost.
Merge order. This touches only
.github/, so it conflicts withnothing — but it cannot go green until the registry race is fixed, and
a05ba3fdoes not cherry-pick cleanly onto current
devel(registry.cpp,registry.hpp,DESIGN.mdandtest_registry_concurrent.cppall conflict; it predates thedoorbell and TL-pull work). Three
ways forward, in the order I would pick them:
fix/registry-teardown-before-slot-releaseontodeveland land itfirst. The bug is real and independently worth fixing, and it is the one
finding here that names a race the design does not permit. Whether that alone
turns the job green I can't claim: 18 reports survive the reader suppression, and
a05ba3fnames only thewrite_slot_payload/teardown_slotpairing of the twoseen locally — the rest of that count has not been triaged.
sanitize (thread)ondeveluntil the fixlands — honest, but a red default branch trains people to ignore it.
continue-on-error: trueon the TSan job with a TODO referencing theregistry fix. Gets the ASan/UBSan value immediately without a red branch, at
the cost of a check nobody has to respect.
I would not recommend 3 as a permanent state: a non-blocking sanitizer job is a
sanitizer job that stops being read.
One correction this owes to the
EventsExecutorPRs. What I reported there wasnot a clean TSan run: #67 and #68 both say the suite is clean under
-fsanitize=threadexcept that "the only TSan reports in the suite remain thepre-existing registry.cpp seqlock ones, which this stack does not touch". That
race is indeed pre-existing in
develand unrelated to those changes — but thisPR is what measures the thing I waved past. Once the reader side is suppressed,
"the pre-existing registry ones" is 18 reports from a single test binary, only one
pairing of which is attributed to a known fix. Discounting a whole class of report
by hand, on the strength of one run, is weaker evidence than it reads as.