feat(transport): capture the raw inbound stream via IBAPI_RAW_CAPTURE_DIR

[wboayue]

Step 3 of plans/tick-by-tick-reconnect-decode-desync.md. #756 (F1/F2) made a framing desync loud and #757 (F3) made it observable; neither made it capturable, which is what confirming the 2026-07-07 root cause needs.

Why the existing recorder can't do this

IBAPI_RECORDING_DIR calls record_response with an already-parsed message and re-synthesises a frame. The 4-byte length prefix it writes is one this crate computed, not one TWS sent — and that prefix is precisely the field a framing desync corrupts. The recorder is blind to the only failure worth recording.

What this adds

RawFrameTap (src/transport/raw_capture.rs), enabled by IBAPI_RAW_CAPTURE_DIR:

• Taps below the framing. Both readers — transport::sync::read_header and the new transport::r#async::io::read_framed_message — record the length prefix before validate_frame_length sees it. A prefix rejected as Error::InvalidFrame never reaches a caller, but it is the evidence, so it reaches the capture.
• One file pair per connection. A reconnect calls start_new_segment, so no .bin splices two TCP streams — that would read back as a phantom desync at the seam.
• .bin + .idx. The .bin is the inbound stream byte for byte. The .idx carries seq,utc_timestamp,offset,declared_length per frame, written as soon as the prefix is read — so a frame whose body never arrived still appears, and a desync can be lined up against a [2119] farm notice in the operator's log. The .bin has no clock; that is what the sidecar is for.
• Replayable. Because the prefixes are the wire's own, a capture feeds straight back through the frame reader. Tests assert this against both readers.
• Never fatal. An unusable directory or a failed write downgrades to a no-op with one warning. A diagnostic aid must not be the reason a connection fails.

examples/replay_raw_capture.rs walks a capture, summarises frames by kind, and names the byte offset of the first frame whose prefix cannot describe a frame (exit 1 if it finds one). It re-implements the 4-byte walk rather than calling the crate's reader — the point is to see what the wire said, including frames the reader would refuse.

What a capture now settles

Three decisive outcomes, documented in the plan:

Result	Meaning
replay_raw_capture reports a DESYNC	F2 confirmed — the offset is where framing slipped, the .idx gives the wall-clock
Walks clean, prices still offset	F2 falsified — a genuine prost mis-decode of well-framed bytes is back in play
Walks clean, prices sane	The #756 fix held

F8 — found while wiring this, not fixed here

TcpSocket sets a 1 s SO_RCVTIMEO. If bytes of a frame have already arrived when it fires, Read::read_exact has consumed them and still returns Err — its contract discards partial progress. TcpMessageBus::dispatch treats a read timeout as benign and loops straight back into read_header at a shifted boundary. Permanent, silent desync.

Verified against a real socket: a peer writing AA BB, stalling 1.5 s, then writing CC DD EE FF makes a 4-byte read_exact with a 1 s timeout return WouldBlock, after which the next read returns CC DD EE FF — AA BB are gone.

This outranks F2 as an explanation of 07-07: F2 needs the gateway to emit a garbage prefix, F8 needs only a >1 s pause between TCP segments, which is what a data-farm transition looks like from the client. Blocking client only — AsyncTcpSocket sets no read timeout, and the dispatcher's select! cancels the read only on the shutdown branch, which breaks the loop. Recorded as F8 in the plan with a fix sketch; it changes the read loop's error handling and wants its own PR.

Tests

15 tests. Format and replay in raw_capture_tests.rs; tap-below-validation at both reader seams (transport/sync.rs, transport/async/io.rs); end-to-end through the handshake listener over a real socket on both clients, asserting the capture opens with the framed handshake response. src/transport/raw_capture.rs is at 93% line coverage.

Note

Captures are unredacted wire bytes — account ids, positions, orders. Called out in CHANGELOG.md, docs/troubleshooting.md, docs/examples.md, and the plan.

Gates

cargo fmt; clippy x3 configs; rustdoc x3 configs; just test (all three legs green); cargo build --examples x2; both integration crates; just rules-check.

Also corrected the recorder's output paths in docs/troubleshooting.md and docs/examples.md — both documented requests.txt / responses_<stamp>.txt, which the recorder has never written.

---

Follow-up commit: /simplify pass

Cleanup only, no behaviour change to the tap itself.

• One meaning for "off". The tap encoded it three ways (sink: None, segment: None, disabled: true). Sink now opens segment 0 eagerly and capturing_to returns a disabled tap if that fails, so the flag is gone and segment: None means exactly one thing.
• framed() was a hand-rolled encode_raw_length — which four sibling test files on this branch already import. Replaced, plus a record_frame helper for the nine open-coded prefix+body pairs.
• A vacuous test. test_disabled_tap_writes_nothing asserted a fresh TempDir was empty, which a disabled tap — never given a directory — cannot affect.
• Test placement. The async seam test moved to src/transport/async/io_tests.rs, beside the function it tests, per docs/rules/testing/sibling-test-files.md.
• replay_raw_capture keys its histogram on a Copy FrameKind instead of a String built per frame (~2M allocations on a 1M-frame capture), derives trailing at print time, and states the real reason it copies the framing constants — they are pub(crate), not "avoiding internals"; the example does link the crate.
• Documented why the capture files are deliberately unbuffered (the page cache survives kill -9; an 8 KiB user-space buffer would lose the tail, which is the evidence), and that the async writes block a runtime worker while the reader mutex is held.

Behaviour change outside the tap

MessageRecorder::from_env unwrapped create_dir_all, so pointing IBAPI_RECORDING_DIR at an unwritable path panicked during Client::connect. It now warns and disables recording, matching the policy this PR writes down for the tap two files away: a diagnostic aid must not be the reason a connection fails. Covered by a new test and a CHANGELOG.md entry.

Considered and not done

The strongest review suggestion was to relocate the tap to a Read/AsyncRead decorator below read_exact, on the grounds that the current placement is blind to F8. That premise is wrong: when read_exact discards partially-consumed bytes, the next prefix the tap records is the shifted one, so replay_raw_capture reports a DESYNC at that offset — F8 is visible. The genuine limitation is only that a .bin is not byte-exact against the wire, which the module's # Limits section already states. A decorator would produce a more faithful artifact and is worth revisiting, but it is a redesign rather than a cleanup, and it would make F8 less conspicuous: the capture would then walk clean while the live client desynced.

Also skipped: sharing the timestamp/instance-prefix logic with MessageRecorder (~6 lines, and the two build different things — a directory vs a filename prefix); dropping the .idx seq column (derivable from line position, but it helps cut/awk pipelines); the index-line allocations (~263 ns against ~3.3 µs of syscall, and only when capturing); and streaming the capture in the example instead of fs::read (real for multi-GB captures, but restructures walk).