Phase D1: streamrNode* C API — the full network node in the shared library#90
Merged
Conversation
…brary
Adds the handle-based streamrNode* C API (include/streamrnode.h) next to
the existing proxyClient* API, implemented over NetworkNode/NetworkStack
in the same translation unit so the two APIs share the result registry
and library lifecycle:
- streamrNodeNew/Delete with StreamrNodeConfig (layer-0 entry points as
(websocketUrl, ethereumAddress) pairs, websocket server port/host,
acceptProxyConnections), streamrNodeStart/Stop, JoinStreamPart/
LeaveStreamPart (optional per-stream-part entry points), Publish
(optional ECDSA signing, same payload as proxyClientPublish),
Subscribe/Unsubscribe with a message callback, GetNeighborCount,
SetProxies (the proxy mode folded into the node handle).
- Entry-point-less configurations: a node with a websocket server
becomes its own entry point (connectivity self-check against its own
server); a fully isolated node keeps an empty entry point list and
streamrNodeStart joins its own DHT explicitly (the doJoin path would
wait for connectivity that never comes).
- Stream part ids are canonicalized at the API boundary ("#1" -> "#1"):
the delivery layer keys stream parts by the canonical form derived
from the message's integer partition, so a raw non-canonical key put
the subscription and the publishes of one stream part into two
different empty overlays (found via the two-node exchange test: the
A->B message expired unseen in the propagation buffer while B->A
arrived through the cross-part topology that formed meanwhile).
- createStreamMessage generalized to buildStreamMessage, shared by both
publish paths (behavior-preserving).
Tests:
- test/integration/StreamrNodeTest.cpp (in ctest): input validation and
lifecycle errors, a two-node publish/subscribe exchange over real
websockets, and proxy-mode publish into an accepting full node.
- test/ts-interop/StreamrNodeTsInteropTest.cpp (manual-run target like
the C8 interop test): a client-only C API node — the mobile
configuration — joins a pinned-TS entry point (nodeId derived from its
ethereum address via the DhtNode nodeId option) and messages flow both
ways through the public C API only.
NAMING for review: the node API reuses ProxyResult /
proxyClientResultDelete / proxyClientInitLibrary / proxyClientCleanupLibrary
and the Proxy struct (as StreamrEntryPoint typedef); renaming the shared
infrastructure to a streamr* prefix would break existing proxy-API users,
so it is left as a PR decision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ptesavol
added a commit
that referenced
this pull request
Jul 17, 2026
Resolves the naming question deferred from the D1 review (PR #90) as its own phase: the proxy-prefixed shared infrastructure (ProxyResult, Error, proxyClientResultDelete, the library lifecycle functions, struct Proxy) gets neutral streamr* names in the same breaking release that extends the C++/Swift/Kotlin/Python wrappers with the streamrNode* full-node functionality, so the wrappers are written against the final names once. Old proxyClient* names stay as deprecated aliases for one release; the definition of done is amended accordingly. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
ptesavol
added a commit
that referenced
this pull request
Jul 17, 2026
WebrtcFullNodeNetworkTest.HappyPath hit the 1200 s ctest timeout twice on the ubuntu-latest leg of PR #90: after one concurrent start threw "Timed out" (waitForNetworkConnectivity, 10 s), the process went silent for ~20 minutes until ctest killed it. Runtime-evidence diagnosis (logs + a worker-pool-saturation reproduction) confirmed the mechanism: - Every node's start() had already detached a fire-and-forget joinDht into NetworkStack's GuardedAsyncScope. TearDown drives all nine stop() coroutines on ONE blockingWait thread; the first stop()'s joinScope.close() blocked that thread AND waited for its join to complete NATURALLY (nothing cancelled it first, violating the scope's own cancel-before-close contract - the DhtNode abort only happens later in the stop sequence and cannot be reordered past the ContentDeliveryManager teardown-ordering constraints). - A join only completes with shared-worker-pool capacity and network progress: every coroutine resumption, including folly::coro::timeout resumptions, needs a pool slot. The 8-node start storm starved the 4-thread CI pool, so the drain never returned. Locally, saturating SharedExecutors::worker() reproduced the identical silent hang; the queued joinDht started the same millisecond a pool thread freed. The fix removes teardown's dependence on join completion: - NetworkStack: a CancellationSource wraps the detached joinDht; stop() requests cancellation BEFORE draining, and drains via the new suspending GuardedAsyncScope::closeAsync() so concurrent sibling stops sharing the drive thread proceed meanwhile. - DiscoverySession/RingDiscoverySession: MERGE the ambient cancellation token with the node's abort signal instead of replacing it - co_withCancellation replaces the ambient token, which made stop()-time cancellation invisible inside the sessions' RPC awaits. - PeerDiscovery::runSessions: a join cancelled by its owner's stop() no longer schedules a detached rejoin. - CoroutineHelper: export co_current_cancellation_token and cancellation_token_merge shims (CPOs, same pattern as blockingWait). Verified with the instrumented reproduction: post-fix, all nine joinScope drains enter in the same millisecond, cancelled joins collapse in ~1.5 s, and the failed-SetUp teardown takes 2.5 s (was: wedged for the whole pool-starvation window, 19+ minutes on CI). Under a 20 s artificial pool saturation, teardown completes ~5 s after the first pool liveness instead of never. GuardedAsyncScopeTest adds the deadlock-shape regression test; the dht end-to-end join suite (7 tests) passes under heavy load with the cancellation plumbing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ptesavol
added a commit
that referenced
this pull request
Jul 18, 2026
…in (#92) * Fix the full-node teardown hang: cancel and suspend the joinScope drain WebrtcFullNodeNetworkTest.HappyPath hit the 1200 s ctest timeout twice on the ubuntu-latest leg of PR #90: after one concurrent start threw "Timed out" (waitForNetworkConnectivity, 10 s), the process went silent for ~20 minutes until ctest killed it. Runtime-evidence diagnosis (logs + a worker-pool-saturation reproduction) confirmed the mechanism: - Every node's start() had already detached a fire-and-forget joinDht into NetworkStack's GuardedAsyncScope. TearDown drives all nine stop() coroutines on ONE blockingWait thread; the first stop()'s joinScope.close() blocked that thread AND waited for its join to complete NATURALLY (nothing cancelled it first, violating the scope's own cancel-before-close contract - the DhtNode abort only happens later in the stop sequence and cannot be reordered past the ContentDeliveryManager teardown-ordering constraints). - A join only completes with shared-worker-pool capacity and network progress: every coroutine resumption, including folly::coro::timeout resumptions, needs a pool slot. The 8-node start storm starved the 4-thread CI pool, so the drain never returned. Locally, saturating SharedExecutors::worker() reproduced the identical silent hang; the queued joinDht started the same millisecond a pool thread freed. The fix removes teardown's dependence on join completion: - NetworkStack: a CancellationSource wraps the detached joinDht; stop() requests cancellation BEFORE draining, and drains via the new suspending GuardedAsyncScope::closeAsync() so concurrent sibling stops sharing the drive thread proceed meanwhile. - DiscoverySession/RingDiscoverySession: MERGE the ambient cancellation token with the node's abort signal instead of replacing it - co_withCancellation replaces the ambient token, which made stop()-time cancellation invisible inside the sessions' RPC awaits. - PeerDiscovery::runSessions: a join cancelled by its owner's stop() no longer schedules a detached rejoin. - CoroutineHelper: export co_current_cancellation_token and cancellation_token_merge shims (CPOs, same pattern as blockingWait). Verified with the instrumented reproduction: post-fix, all nine joinScope drains enter in the same millisecond, cancelled joins collapse in ~1.5 s, and the failed-SetUp teardown takes 2.5 s (was: wedged for the whole pool-starvation window, 19+ minutes on CI). Under a 20 s artificial pool saturation, teardown completes ~5 s after the first pool liveness instead of never. GuardedAsyncScopeTest adds the deadlock-shape regression test; the dht end-to-end join suite (7 tests) passes under heavy load with the cancellation plumbing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ContentDeliveryManager: cancel and suspend the joinScope drain in destroy() Same mechanism and fix as NetworkStack's joinScope in the parent commit, one layer down: destroy()'s blocking joinScope.close() drained un-cancelled startLayersAndJoinDht/handleEntryPointLeave tasks on the shared blockingWait drive thread — unbounded under a starved worker pool (sampled as ContentDeliveryManager.cppm:209 -> GuardedAsyncScope::close() wedging the WebrtcFullNodeNetworkTest teardown on a CI-sized 4-thread pool). A joinCancellation source now wraps all four detached joinScope tasks; destroy() requests cancellation first and drains via the suspending closeAsync(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * StoreManager: scope and drain the detached onContactAdded replications The fire-and-forget replication coroutines were started with a bare .start(): they pin the StoreManager via sharedFromThis, but their RPC remotes send through the owning DhtNode's communicator, which nothing kept alive — a replication resuming after DhtNode::stop() dereferenced the freed communicator (SIGSEGV in RpcCommunicatorClientApi::notify<ReplicateDataRequest> resume, reproduced 13/16 on a CI-sized 4-thread pool once the teardown drain no longer serialized everything). Both launch sites now go through a cancellable GuardedAsyncScope that destroy() cancels and drains while the communicator and transport are still alive; the awaited final leave-replication is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ConnectionLocker: make lockConnection/unlockConnection coroutines Both wrapped their RPC round-trip in a thread-blocking blockingWait(). On the shared worker pool that is a self-deadlock: once poolsize concurrent joins sit inside lockConnection at the same time, every pool thread waits for a lock-RPC response that only those same threads could process (sampled: all four StreamrWorkers of a CI-sized pool blocked in PeerDiscovery::joinThroughEntryPoint -> lockConnection -> blockingWait, test wedged). The interface and the ConnectionManager implementation now return Task<void>; PeerDiscovery co_awaits them. ProxyClient's synchronous connection routine wraps the calls in blockingWait at its own call sites for now — it already blocks on its other RPCs, and its callers block at the C-API boundary; async-ifying that path is a follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DhtNode: drain the communicator's detached RPC coroutines in stop() The server-dispatch scope only drained in ~RoutingRpcCommunicator during member destruction — after later-declared members were already gone — so a straggler getClosestPeers dispatch resumed into freed memory (crash reports on the CI-sized pool; the teardown-drain-ordering rule). stop() now removes the transport listeners with offAndWait() (a Message handler mid-invocation can otherwise schedule one more dispatch past the drain) and calls rpcCommunicator->drainAsyncTasks() as its last step, after the leave notices and final replications that still send through the communicator. The RPC client/server APIs gate their scope-add paths on the drained flag: folly forbids add-after-join, so a late request now fails cleanly (client) or is dropped like any request to a stopped node (server). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Websocket connectors: close pending connections outside the mutex Both destroy() methods held the connector mutex across the pendingConnection->close(true) call-outs. The Disconnected fan-out re-enters connection and connector code, so this deadlocks ABBA against any thread that holds a connection's mutex and needs the connector's in its handler chain (sampled: main thread in WebsocketClientConnector::destroy -> PendingConnection::close -> Disconnected handler blocked on a connection mutex, an rtc close callback holding it and blocked on the connector mutex, a pool worker in connect() queued behind — the never-hold-locks-across-call-outs rule). Both now snapshot and clear their maps under the lock and make every call-out after releasing it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * clang-format: converge on the checker's clang-format (llvm 22) The previous commits were formatted with the system clang-format; this re-runs run-clang-format.py -i with the same binary lint.sh checks against, mostly reverting cosmetic reflow the wrong version introduced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * CoroutineHelper: comment touch to invalidate the stale macOS CI cache The restored incremental-build cache on the macos-26 leg carried a pre-#92 BMI of this module and under-rebuilt: DiscoverySession and RingDiscoverySession failed to compile against the cancellation shims this branch added, while identical code built and passed on both Linux legs and locally. Touching the module interface forces the BMI (and its dependents) to rebuild — the documented remedy for this cache trap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Milestone D, phase D1 of
trackerless-network-completion-plan.md: the shared library gains a handle-basedstreamrNode*C API (include/streamrnode.h) — a full network node next to the existingproxyClient*API, which stays unchanged for compatibility.API surface
streamrNodeNew/streamrNodeDeletewithStreamrNodeConfig: layer-0 entry points as(websocketUrl, ethereumAddress)pairs (StreamrEntryPoint, same layout asProxy), websocket server port and host (port 0 = client-only mode, the mobile configuration),acceptProxyConnections.streamrNodeStart/streamrNodeStop(stop is terminal, like the underlyingNetworkStack).streamrNodeJoinStreamPart/streamrNodeLeaveStreamPartwith optional per-stream-part entry points.streamrNodePublishwith optional ECDSA signing (same signature payload asproxyClientPublish).streamrNodeSubscribe/streamrNodeUnsubscribewith a message callback (invoked on an internal network thread; documented constraints in the header).streamrNodeGetNeighborCountfor observing topology formation.streamrNodeSetProxies— the proxy mode folded into the node handle (direction enum matches the proto values).The implementation lives in the same translation unit as the proxy API so the two share the result registry, the
ProxyResult**out-parameter convention, and the library lifecycle.createStreamMessagewas generalized tobuildStreamMessageand is shared by both publish paths (behavior-preserving).The node API deliberately reuses the proxy API's shared infrastructure by its existing names:
ProxyResult,proxyClientResultDelete,proxyClientInitLibrary/proxyClientCleanupLibrary, andstruct Proxy(aliased asStreamrEntryPoint). Renaming these to astreamr*prefix would be cleaner for node-API users but breaks every existing proxy-API consumer. Also up for review: whether the node API should live in its ownstreamrnode.h(as here) or be merged intostreamrproxyclient.h.Notable implementation points
"…#01"key would put the subscription and the publishes of one stream part into two different, empty overlays — the published message expired unseen in the propagation buffer. The C API canonicalizes at the boundary; the callback documents that it reports the canonical form.ws://:0and fail — andstreamrNodeStartusesstart(false)plus an explicitjoinDht([own]), the TS-driver pattern.proxyClientInitLibrary()in their constructors: the existing proxy integration suite segfaults when run as a single process (per-testproxyClientCleanupLibrary()nulls the singleton, which only the dylib-load constructor re-creates). This is invisible under ctest, which runs each test in its own process; the pre-existing suite is left untouched.Tests
test/integration/StreamrNodeTest.cpp(registered in ctest, 9 tests): input validation and lifecycle error codes; a two-node publish/subscribe exchange over real websockets on 127.0.0.1 (signed and unsigned messages, both directions); proxy-mode publish from a client-only node into an accepting full node, including the accepting-node-cannot-set-proxies error.test/ts-interop/StreamrNodeTsInteropTest.cpp(manual-run target, matching the C8 interop test's registration decision): a client-only C API node joins a pinned-TypeScript (@streamr/trackerless-network103.8.0-rc.3 from npm) entry point whose node id is derived from its ethereum address via the pinnedDhtNodeOptions.nodeId, and messages flow both ways through the public C API only — no C++ modules in the test.Verification
lint.sh(clangd-tidy + clang-format) green across all packages (pre-push hook run); the library package's lint file sweep now excludesnode_modules(same fix the trackerless-network package needed for its C8 harness).🤖 Generated with Claude Code