refactor(multipath): challenges on a path generation are on path#712
refactor(multipath): challenges on a path generation are on path#712flub wants to merge 1 commit into
Conversation
|
Documentation for this PR has been generated and is available at: https://n0-computer.github.io/noq/pr/712/docs/noq/ Last updated: 2026-06-18T09:42:38Z |
Performance Comparison Report
|
| Scenario | noq | upstream | Delta | CPU (avg/max) |
|---|---|---|---|---|
| large-single | 5751.6 Mbps | 8042.4 Mbps | -28.5% | 92.4% / 97.7% |
| medium-concurrent | 5503.7 Mbps | 7809.1 Mbps | -29.5% | 92.7% / 99.0% |
| medium-single | 3957.9 Mbps | 4749.1 Mbps | -16.7% | 93.7% / 102.0% |
| small-concurrent | 3798.2 Mbps | 5160.1 Mbps | -26.4% | 97.2% / 155.0% |
| small-single | 3543.5 Mbps | 4841.6 Mbps | -26.8% | 89.2% / 97.2% |
Netsim Benchmarks (network simulation)
| Condition | noq | upstream | Delta |
|---|---|---|---|
| ideal | 3057.6 Mbps | 4092.8 Mbps | -25.3% |
| lan | 782.4 Mbps | 810.4 Mbps | -3.5% |
| lossy | 69.8 Mbps | 69.8 Mbps | ~0% |
| wan | 83.8 Mbps | 83.8 Mbps | ~0% |
Summary
noq is 25.5% slower on average
8d9cdcab6fc1c9a5f4a805defbb05333cb52f58b - artifacts
Raw Benchmarks (localhost)
| Scenario | noq | upstream | Delta | CPU (avg/max) |
|---|---|---|---|---|
| large-single | 5328.5 Mbps | 7990.3 Mbps | -33.3% | 98.6% / 149.0% |
| medium-concurrent | 5512.7 Mbps | 7809.9 Mbps | -29.4% | 93.9% / 99.0% |
| medium-single | 4049.0 Mbps | 4749.5 Mbps | -14.7% | 92.4% / 101.0% |
| small-concurrent | 3805.8 Mbps | 5089.2 Mbps | -25.2% | 99.5% / 154.0% |
| small-single | 3439.1 Mbps | 4824.8 Mbps | -28.7% | 98.0% / 153.0% |
Netsim Benchmarks (network simulation)
| Condition | noq | upstream | Delta |
|---|---|---|---|
| ideal | 3045.1 Mbps | 4009.0 Mbps | -24.0% |
| lan | 782.4 Mbps | 810.3 Mbps | -3.5% |
| lossy | 69.9 Mbps | 55.9 Mbps | +25.0% |
| wan | 83.8 Mbps | 83.8 Mbps | ~0% |
Summary
noq is 26.3% slower on average
bc32f38a9c268f2034ef3bc8dcfe2b0f5a21b1ca - artifacts
Raw Benchmarks (localhost)
| Scenario | noq | upstream | Delta | CPU (avg/max) |
|---|---|---|---|---|
| large-single | 5353.3 Mbps | 7832.6 Mbps | -31.7% | 95.2% / 100.0% |
| medium-concurrent | 5414.2 Mbps | 7694.6 Mbps | -29.6% | 96.5% / 150.0% |
| medium-single | 3817.4 Mbps | 4749.1 Mbps | -19.6% | 88.6% / 96.5% |
| small-concurrent | 3702.1 Mbps | 5318.2 Mbps | -30.4% | 94.9% / 103.0% |
| small-single | 3405.9 Mbps | 4796.9 Mbps | -29.0% | 94.5% / 103.0% |
Netsim Benchmarks (network simulation)
| Condition | noq | upstream | Delta |
|---|---|---|---|
| ideal | 3071.0 Mbps | N/A | N/A |
| lan | 796.4 Mbps | N/A | N/A |
| lossy | 69.9 Mbps | N/A | N/A |
| wan | 83.8 Mbps | N/A | N/A |
Summary
noq is 28.6% slower on average
| // An on-path challenge that was sent on a different 4-tuple would be stored | ||
| // in a different path generation. | ||
| debug_assert!( | ||
| info.network_path.is_probably_same_path(&self.network_path), | ||
| "response can only be for same network path" | ||
| ); |
There was a problem hiding this comment.
Sooo, I asked GLM 5.2 to review this assert (yes I'm still testing various open-weight models 🙃), and I think it's correct in saying that it is possible that this assert will fail:
Connection::handle_network_changewill resetPathData::network_path::local_iptoNoneif the path is deemed recoverable.info.network_pathcan havelocal_ipset toSome(X), withself.network_path.remotematching.- Assuming the remote addresses match,
info.network_path.is_probably_same_path(&self.network_path)simplifies toinfo.network_path.local_ip.is_none() || info.network_path.local_ip == self.network_path.local_ipwhich simplifies toinfo.network_path.local_ip == self.network_path.local_ipwhich is false, because the left one isSome(X)and the right one was cleared toNonebyhandle_network_change.
There was a problem hiding this comment.
So this is weird. iroh's use of the handle_network_change API explicitly says a path is recoverable if the local IP still exists. Yet in the noq implementation we clear the local_ip for the recoverable paths.
Should we just not clear the local_ip for recoverable paths?
Should handle_network_change create a new path generation with PathData::from_previous if the path is recoverable?
There was a problem hiding this comment.
IMO setting local_ip = None in handle_network_change makes sense. But the check in iroh doesn't. The idea of handling the network change gracefully is to be able to re-use a path that existed on a 4-tuple with a different local interface. E.g. someone might've switched from WiFi to Cellular and thus lost their Wifi interface, but gained the Cellular one, which became the OS default interface. So when the Wifi path's local IP gets set to None, the OS chooses the cellular interface for the next send and the path migrates. That seems like it's the correct way to handle recoverable paths.
There was a problem hiding this comment.
Given how broken iroh is, this makes me wonder if we can remove all of the network change hint stuff in iroh and everything will keep working 👀
Let me try that :D
There was a problem hiding this comment.
The purpose was to switch between closing the old and new paths if the 4-tuples changed, which is what multipath says you should do, and try and keep the same path in case the 4-tuple hasn't changed. So to me iroh is doing fine, but noq clearing the local IP is not.
I just pushed a commit that removes the clearing of the local IP and patchbay is happy with it. So I'm currently inclined to think that this is reasonable.
There was a problem hiding this comment.
Seems like patchbay doesn't care about calling noq's handle_network_change
There was a problem hiding this comment.
I think there's another case where local_ip might not match. At the end of process_payload the local_ip is updated for any non-probing packet which has a different local_ip. That means the local_ip can be different all the time during process_payload
The token of any PATH_CHALLENGE sent as a on-path challenge is stored on the path generation state (PathData). Thus any response that matches the token is always on path. The PathData::network_path of a path generation can not change, only the local_ip can be filled in if initially missing. This may not have been the case when this was originally introduced.
8d9cdca to
bc32f38
Compare
|
So, I believe this is wrong. Packets are processed before migration and local_ip updates are handled. This means while processing a packet you always have a valid packet from a peer but the 4-tuples may not match. For path validation this indeed makes a difference: you can receive valid responses that do not validate the path if your local IP changed in between. I believe there's still a bug, but it is so different that I want to create a new PR. |
Description
The token of any PATH_CHALLENGE sent as a on-path challenge is stored
on the path generation state (PathData). Thus any response that
matches the token is always on path. The PathData::network_path of a
path generation can not change, only the local_ip can be filled in if
initially missing.
This may not have been the case when this was originally introduced.
Breaking Changes
none
Notes & open questions
This is the more controversial part of #711, so targets that PR. This
state was originally introduced in #443.
Change checklist