Skip to content

Connection lost after 3 seconds if server responds from different interface than initially approached #8

Description

@HXM4Tech

This issue is related to recent transition of WebX protocol from structured TCP stream connections to UDP datagram communication.

Scenario - when the problem occurs

Peer A has multiple network interfaces facilitating contact between itself and Peer B. Peer A runs WebX daemon with wildcard bind_addr in config.toml.

Peer B tries to contact Peer A using IP address assigned to the interface, which does not align with Peer A's default route to Peer B.

What currently happens

Let's denote Peer A's default interface with if1 and the other one with if2.

  1. Peer B sends hello message to Peer A's if2 to initiate connection
  2. Responding to Peer B's hello message (recieved on if2), Peer A sends its own hello message back to Peer B via if1 and registers the connection as active.
  3. Since if1 and if2 have different IP addresses, Peer B fails to identify this response with Peer A and treats recieved datagram as new, unrelated connection request. Peer B sends its hello message again to Peer A's if1 address and also registers the connection.
  4. Peer A ignores Peer B's duplicated hello message as invalid. At this point both peers log successful connection and WebX internal signed IPv6 packets can be transferred between them normally.
  5. 3 seconds later timeout for initial connection attempt (between Peer B and Peer A's if2) expires. Peer B sends disconnect message to Peer A's if2.
  6. Peer A considers recieved disconnect message as valid, unregisters the connection and responds to Peer A with its own disconnect message. The connection is closed at this point until the cycle is repeated by Peer B retrying the connection with Peer A viaif2.

Logs from Peer B:

2026-08-18 23:12:07 UTC+02:00 [ OK ] Wallet has been loaded!
2026-08-18 23:12:07 UTC+02:00 [INFO] Public key: 034d8863a2292b9c29e1e676af885984451b698671e4f355c602ed4ee1907b4af8
2026-08-18 23:12:07 UTC+02:00 [INFO] IPv6: d48e:15d1:15fc:2551:97d6:bdb4:550b:f43
2026-08-18 23:12:07 UTC+02:00 [ OK ] TUN interface has been set up!
2026-08-18 23:12:07 UTC+02:00 [INFO] The interface has name: webx0
2026-08-18 23:12:07 UTC+02:00 [ OK ] Entered asynchronous runtime!
2026-08-18 23:12:07 UTC+02:00 [ OK ] (P2P) Connected to 10.100.2.171:4760 with WebX IPv6 address d4a5:c957:adaa:d1a5:670b:ae6e:acea:1a38
2026-08-18 23:12:10 UTC+02:00 [WARN] (P2P) Failed to connect to 10.100.2.179:4760, error: Timeout connecting to peer
2026-08-18 23:12:10 UTC+02:00 [WARN] (P2P) Connection with 10.100.2.171:4760 closed
2026-08-18 23:12:20 UTC+02:00 [ OK ] (P2P) Connected to 10.100.2.171:4760 with WebX IPv6 address d4a5:c957:adaa:d1a5:670b:ae6e:acea:1a38
2026-08-18 23:12:23 UTC+02:00 [WARN] (P2P) Connection with 10.100.2.171:4760 closed
2026-08-18 23:12:33 UTC+02:00 [ OK ] (P2P) Connected to 10.100.2.171:4760 with WebX IPv6 address d4a5:c957:adaa:d1a5:670b:ae6e:acea:1a38
2026-08-18 23:12:36 UTC+02:00 [WARN] (P2P) Connection with 10.100.2.171:4760 closed

Here if1 has IP address 10.100.2.171, while if2 has 10.100.2.179.

Suggested solutions:

Option A: Force peers to always reply on the same interface they have been contacted.

Variant 1: Resolve wildcard binding in the userspace

This is probably the simplest, but the most unclean solution. It would require detecting whether the bind_addr provided in config.toml is a wildcard address, and if yes - converting it to the list of all IP addresses assigned on all interfaces and opening separate UdpSocket for each of them. Hotplug interfaces would not be supported or WebX daemon would need to continuously check whether new socket should be openned.

Variant 2: Utilize kernel-provided IP packet information with sendmsg and recvmsg

This requires stepping away from tokio::net::UdpSocket and implementing own low-level wrapper around raw libc socket (or finding appropriate crate) to enable IP_PKTINFO/IPV6_RECVPKTINFO, track the destanation addresses of recieved packets and set the same address as source address for outgoing packets within the connection.

This variant would likely introduce loads of unsafe {} code.

Option B: Track connections without relying on neighbor's SocketAddr

The advantage of this option is not only seamless support for multi-interface machines, but also support for sudden network switches - although this requires tracking the last SocketAddr each neighbor contacted the peer from.

Variant 1: Identify connections with neighbors by their WebX IPv6 addresses

This variant extends each datagram sent by a peer to its neighbor by 16 bytes of peer's WebX internal address to identify the source. While this isn't significant overhead itself, this solution introduces a risk of impersonification of neighbors. Knowing two peers are neighbors and knowing real address of at least one of them, attacker can easily spoof disconnect message or any other communication between those peers to disrupt the network operations.

Signing each message (including applying additional temporary signature on transferred WebX IPv6 on each hop) would introduce significant performance overhead along with yet another 65 bytes added to each message (total of 81 bytes), effectively putting this idea out of consideration.

Variant 2: Identify connections by (variable) connection ID [QUIC-like]

In this variant peers generate non-repeating connection IDs (e.g. 4 bytes/64 bits long) for every of their neighbors and share those connection IDs with them. To ensure uniqness this would work both ways, requiring a pair of connection IDs per connection. Each peer would need to include the connection ID assigned to it by its neighbor in any message sent to them, allowing to identify the source while being save from unsophisticated attacks. For even better security the connection IDs can also be changed periodically every few seconds.

The con of this variant however is a significant protocol complexity, requiring different schema of hello message for server peer and client peer (server peer's hello message should already include connection ID assigned by the client), ensuring that the message announcing connection ID change is not lost, etc.


Thoughts, critiques, or alternative solutions? Feel free to jump into the comments below!

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions