Describe the bug
The scheduled Long Test run 33573669385 on main at cd60587be4032532e7ce70d2546a15f34f83d765 failed its TSAN job for two independent reasons:
reconfiguration_test_suite found a real data race in the JWT refresh metrics. Two CCF task workers concurrently update the same plain counters in NodeEndpoints::handle_event_request_completed().e2e_logging reported the known closedir/epoll_ctl file-descriptor reuse race which is already listed in tsan_env_suppressions. This test opts into DETECT_DEADLOCKS, and that CMake path replaces TSAN_OPTIONS without retaining the suppression file.
The same workflow also failed Long Shuffled/LTS/Snmalloc, but that failure is already tracked by #8248 and should not be duplicated here.
To Reproduce
Run the TSAN configuration of the Long Test workflow at the commit above.
-
Failing TSAN job and complete logs
-
The first failure is in reconfiguration_test_suite. TSAN reports concurrent writes at node_frontend.h:424 and then at line 432. Both stacks are:
NodeEndpoints::handle_event_request_completed
RpcFrontend::process_command
JwtKeyAutoRefresh::send_refresh_jwt_keys
JwtKeyAutoRefresh::handle_jwt_jwks_response
BasicTask::do_task_implementation
Enclave::run_worker
The conflicting workers are T3 and T4.
-
The second failure is in e2e_logging. TSAN reports:
Write: closedir
std::filesystem::directory_iterator::~directory_iterator
asynchost::get_file_name_with_idx
Ledger::get_file_from_cache
Ledger::on_ledger_get_async
Previous read: epoll_ctl
close_ptr<TCPImpl>::~close_ptr
RPCConnectionsImpl::close
TSAN associates both operations with file descriptor 25.
The closedir signature is recurring:
Expected behavior
JWT refresh metric updates and reads should be synchronised and provide a coherent snapshot.
Tests using DETECT_DEADLOCKS should continue to apply unrelated data-race suppressions, so the known closedir/epoll_ctl report does not terminate the test. TSAN should fail only for unsuppressed, actionable findings.
Environment information
- Repository revision:
cd60587be4032532e7ce70d2546a15f34f83d765
- Workflow:
Long Test, scheduled on main, 2026-09-02
- Job: Debug TSAN
- Runner SKU:
Standard_D16ads_v6
- Stacks use libstdc++ 13.2.0 and libuv
Additional context
Root cause 1: unsynchronised JWT metrics
JWTRefreshMetrics jwt_refresh_metrics is shared mutable state. handle_event_request_completed() increments attempts, failures, and successes without a lock or atomics. Curl completion callbacks can run through independent BasicTasks on multiple enclave workers, so the endpoint completion hook is not single-threaded.
There is a second unsynchronised access: GET /jwt_keys/refresh/metrics copies the same struct while updates may be in flight.
A fix must protect both writers and the reader. A mutex around counter updates plus copying a snapshot for the GET endpoint would preserve coherence across the three counters. If atomics are used instead, serialisation must explicitly load them and define whether a cross-counter coherent snapshot is required.
Root cause 2: the known FD report loses its suppression
The repository already documents and suppresses this exact report:
tsan_env_suppressions:
race:closedir
race:epoll_ctl
Normal TSAN tests receive:
TSAN_OPTIONS=suppressions=<repo>/tsan_env_suppressions
However, add_san_test_properties() makes DETECT_DEADLOCKS mutually exclusive with the suppression file:
TSAN_OPTIONS=detect_deadlocks=1:halt_on_error=1:second_deadlock_stack=1
e2e_logging is registered with DETECT_DEADLOCKS, so the known FD-reuse signature is exposed and halt_on_error=1 terminates the node during an asynchronous historical-ledger read.
The split originally avoided deadlock:*/store.h and deadlock:*/untyped_map.h suppressions. Those deadlock suppressions were removed by #8154; the file now contains only the independent closedir and epoll_ctl race suppressions. The configuration can therefore retain suppressions=... while enabling deadlock detection, or use separate race/deadlock suppression files if future entries require different treatment.
Pending concurrency PRs
None of the current concurrency fixes covers these accesses:
The shuffled ledger failure from the original run reproduced independently on #8242 and is tracked in #8248.
Describe the bug
The scheduled Long Test run 33573669385 on
mainatcd60587be4032532e7ce70d2546a15f34f83d765failed its TSAN job for two independent reasons:reconfiguration_test_suitefound a real data race in the JWT refresh metrics. Two CCF task workers concurrently update the same plain counters inNodeEndpoints::handle_event_request_completed().e2e_loggingreported the knownclosedir/epoll_ctlfile-descriptor reuse race which is already listed intsan_env_suppressions. This test opts intoDETECT_DEADLOCKS, and that CMake path replacesTSAN_OPTIONSwithout retaining the suppression file.The same workflow also failed
Long Shuffled/LTS/Snmalloc, but that failure is already tracked by #8248 and should not be duplicated here.To Reproduce
Run the TSAN configuration of the Long Test workflow at the commit above.
Failing TSAN job and complete logs
The first failure is in
reconfiguration_test_suite. TSAN reports concurrent writes atnode_frontend.h:424and then at line 432. Both stacks are:The conflicting workers are T3 and T4.
The second failure is in
e2e_logging. TSAN reports:TSAN associates both operations with file descriptor 25.
The
closedirsignature is recurring:Expected behavior
JWT refresh metric updates and reads should be synchronised and provide a coherent snapshot.
Tests using
DETECT_DEADLOCKSshould continue to apply unrelated data-race suppressions, so the knownclosedir/epoll_ctlreport does not terminate the test. TSAN should fail only for unsuppressed, actionable findings.Environment information
cd60587be4032532e7ce70d2546a15f34f83d765Long Test, scheduled onmain, 2026-09-02Standard_D16ads_v6Additional context
Root cause 1: unsynchronised JWT metrics
JWTRefreshMetrics jwt_refresh_metricsis shared mutable state.handle_event_request_completed()incrementsattempts,failures, andsuccesseswithout a lock or atomics. Curl completion callbacks can run through independentBasicTasks on multiple enclave workers, so the endpoint completion hook is not single-threaded.There is a second unsynchronised access:
GET /jwt_keys/refresh/metricscopies the same struct while updates may be in flight.A fix must protect both writers and the reader. A mutex around counter updates plus copying a snapshot for the GET endpoint would preserve coherence across the three counters. If atomics are used instead, serialisation must explicitly load them and define whether a cross-counter coherent snapshot is required.
Root cause 2: the known FD report loses its suppression
The repository already documents and suppresses this exact report:
tsan_env_suppressions:Normal TSAN tests receive:
However,
add_san_test_properties()makesDETECT_DEADLOCKSmutually exclusive with the suppression file:e2e_loggingis registered withDETECT_DEADLOCKS, so the known FD-reuse signature is exposed andhalt_on_error=1terminates the node during an asynchronous historical-ledger read.The split originally avoided
deadlock:*/store.handdeadlock:*/untyped_map.hsuppressions. Those deadlock suppressions were removed by #8154; the file now contains only the independentclosedirandepoll_ctlrace suppressions. The configuration can therefore retainsuppressions=...while enabling deadlock detection, or use separate race/deadlock suppression files if future entries require different treatment.Pending concurrency PRs
None of the current concurrency fixes covers these accesses:
closedirsignature.jwt_refresh_metrics; it should be considered when landing the metrics fix because it adds more refresh/retry paths.The shuffled ledger failure from the original run reproduced independently on #8242 and is tracked in #8248.