fix(transport): validate frame lengths and surface unroutable frames - #756
Merged
Conversation
Nothing bounded the 4-byte big-endian length prefix. Four garbage bytes were read as a body length of up to 4 GiB, which allocated that much and then blocked in read_exact until that many bytes arrived — consuming and destroying every real message in between, then yielding one bogus frame with the stream left mis-framed. The framing is positional, so nothing re-anchors it, and a mis-framed protobuf payload still decodes without error (prost skips unrecognized field numbers). The visible symptom is plausible-looking wrong field values that never recover. parse_raw_message also indexed the first four bytes unguarded, so a body of 0-3 bytes panicked with "index out of bounds" — killing the dispatcher thread on the blocking client and the dispatcher task on the async one. validate_frame_length now gates both frame readers: bodies below the 4-byte message id and lengths past 0x00FFFFFF are rejected, matching the official client's Constants.MaxMsgSize (EReader.readSingleMessage raises BAD_LENGTH). Both raise the new Error::InvalidFrame, which is_connection_lost reports as true so the dispatchers reconnect rather than shut down — the socket is open but only a fresh connection can re-anchor the framing. Audit findings and the remaining follow-ups are in plans/tick-by-tick-reconnect-decode-desync.md.
A frame no channel claimed was dropped without a signal. The blocking transport logged every such frame at info without distinguishing the cases; the async transport's route_to_shared_channel had no else arm at all. So an unrecognized message id -- the fingerprint of a framing desync, since a slipped read makes every subsequent id garbage -- looked exactly like an idle connection. That is why the 2026-07-07 incident surfaced data-farm notices and no decode error. report_unroutable_frame splits the two: an unknown IncomingMessages kind warns and publishes UNKNOWN_MESSAGE_TYPE_CODE (-5) to the notice stream, so consumers can react programmatically; a known kind with no current subscriber stays at info, since that is ordinary steady state and raising the same signal for it would make the signal worthless. The code joins the existing client-side sentinels (-3 handshake unknown frame, -4 handshake decode failure); TWS uses 0 and up. Notice:: is_handshake_synthetic is deliberately untouched -- this is not a handshake notice. Follow-ups in plans/tick-by-tick-reconnect-decode-desync.md.
Cleanup pass over the two framing commits. No behavior change. parse_raw_message uses split_first_chunk, which does the length check, the array conversion, and hands back the remainder in one call -- so both data[MIN_FRAME_LENGTH..] re-slices go away. route_to_shared_channel goes back to if-let/else; the match reindented an untouched loop for one new branch. The two validate_frame_length reject tests collapse into one table, matching the form the sync reader test already used, and that test's hardcoded 3 derives from MIN_FRAME_LENGTH now. Tests build frames through encode_protobuf_message and encode_raw_length instead of hand-rolling the id offset and the length prefix. The desync narrative was written out six times across errors.rs, transport/common.rs, and messages.rs. It lives on Error::InvalidFrame now; the internal sites keep their local claim and point there. Two doc errors fixed. -2 was described as a notice-code sentinel alongside -3/-4, but it is IncomingMessages::Shutdown, a message-type discriminant -- no notice is ever synthesized with it; the neighbouring HANDSHAKE_UNKNOWN_FRAME_CODE doc carried the same conflation plus an "only other sentinel" claim that -5 had invalidated. And UNKNOWN_MESSAGE_TYPE_CODE promised to distinguish a slipped stream from a new IBKR message type, which it cannot do without naming the id. Adds a test pinning NotValid to ByMessageType routing -- the unknown-id alarm depends on it and nothing guarded it. Restructuring findings from the pass are recorded in plans/.
wboayue
added a commit
that referenced
this pull request
Aug 9, 2026
…rdings (#757) IncomingMessages::from is lossy -- every unrecognized value collapses to the single NotValid variant -- and ResponseMessage::from_protobuf kept only that, so a frame's numeric id was destroyed at construction, before routing ever saw it. The UNKNOWN_MESSAGE_TYPE_CODE notice added in #756 therefore could not say which id it meant: it carried a fixed string, and the warn! printed an empty field list because a protobuf frame has no fields[0] to fall back on. During the incident that machinery exists for, an operator would get N byte-identical notices. ResponseMessage carries message_id alongside kind now -- the value kind was resolved from, with the PROTOBUF_MSG_ID offset already removed -- and both constructors funnel through a private new() that computes kind from it, so the invariant kind == IncomingMessages::from(message_id) has one home instead of being asserted by hand in two struct literals. ResponseMessage is pub(crate), so no public API changes. That is what makes the notice diagnostic rather than decorative: scattered ids mean the framing slipped, one repeated id means IBKR added a message type. Two further consumers had the same loss. The wire recorder reconstructed the id with message_type() as i32, which agrees with the arriving id for every recognized message but yields -1 for an unrecognized one -- so IBAPI_RECORDING_DIR fabricated an id in precisely the capture worth replaying. And the handshake catch-all logged the kind alone, rendering the bare NotValid in exactly the reconnect window this investigation is about. Both name the id now. Rejected while here, recorded so it is not re-litigated: folding kind into a computed message_type() would undo a deliberate cache with 51 non-test call sites, and NotValid(i32) is impossible -- IncomingMessages is a pub fieldless enum used as a HashMap key in both dispatchers, with `as i32` casts and const arrays. The root cause of the original 2026-07-07 corruption remains unconfirmed; see plans/tick-by-tick-reconnect-decode-desync.md for the falsification condition and the remaining follow-ups.
wboayue
added a commit
that referenced
this pull request
Aug 9, 2026
…_DIR (#758) * feat(transport): capture the raw inbound stream via IBAPI_RAW_CAPTURE_DIR Step 3 of plans/tick-by-tick-reconnect-decode-desync.md. F1/F2 (#756) made a framing desync loud and F3 (#757) made it observable; neither made it capturable. IBAPI_RECORDING_DIR cannot: record_response is handed an already-parsed message and re-frames it, so the 4-byte length prefix it writes is one this crate computed. That prefix is the field a desync corrupts, which makes the recorder blind to exactly the failure worth recording. RawFrameTap taps the socket below the framing. Both frame readers record the prefix before validate_frame_length can reject it, so a prefix that never reaches a caller still reaches the capture. A reconnect starts a new file, so no .bin splices two TCP streams. A sidecar .idx carries seq,utc_timestamp,offset,declared_length — the .bin has no clock, and lining a desync up against a data-farm notice in an operator's log needs one. Because the prefixes are the wire's own, a .bin replays through the frame reader unchanged; tests assert that against both readers. examples/replay_raw_capture.rs walks a capture and names the first frame whose prefix cannot describe a frame. Also records F8 in the plan, found while wiring this: on the blocking client a 1s SO_RCVTIMEO landing mid-frame makes read_exact discard bytes it already consumed, and the dispatcher treats the timeout as benign and reads on at a shifted boundary. Verified against a live socket. It needs no corrupt bytes at all, only a stall, which makes it a better fit for the 07-07 incident than F2. Not fixed here. * refactor(transport): /simplify pass on the raw-frame tap - collapse the tap's three encodings of "off" into one: `Sink` now opens segment 0 eagerly and returns a disabled tap if that fails, so `State` drops its `disabled` flag and `segment: None` means exactly one thing - tests use `encode_raw_length` instead of a local `framed()` copy, and a `record_frame` helper instead of open-coding prefix+body at nine sites - `test_disabled_tap_writes_nothing` asserted a fresh TempDir was empty, which a disabled tap cannot affect; it now checks what it meant to - move the async tap seam test to `async/io_tests.rs`, beside the function it tests, per the sibling-test-files convention - `replay_raw_capture`: key the histogram on a `Copy` FrameKind rather than a per-frame `String`, derive `trailing` at print, single usage string, and state the real reason the framing constants are copied (they are pub(crate), not "avoiding internals" — the example does link the crate) - document why the capture files are deliberately unbuffered, and that the async writes block a runtime worker Also fixes a pre-existing panic surfaced by the review: `MessageRecorder::from_env` unwrapped `create_dir_all`, so an unwritable `IBAPI_RECORDING_DIR` aborted `Client::connect`. Same policy as the tap now — warn and disable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the first two findings of the tick-by-tick decode-desync investigation
(
plans/tick-by-tick-reconnect-decode-desync.md), which was filed against alive
all_last()subscription that began emitting prices with a constant+1,337,581,200offset after ausfarm.njdata-farm reconnect and neverrecovered — with no decode error anywhere.
The audit found the framing path trusts the wire completely. Two defects, both
absent from the equivalent path in the official C# client.
Trusting the length prefix (
fe8910d)Nothing bounded the 4-byte big-endian length prefix. Four garbage bytes were
read as a body length of up to 4 GiB, which allocated that much and then
blocked in
read_exactuntil that many bytes arrived — consuming anddestroying every real message in between, then yielding one bogus frame with
the stream left mis-framed. The framing is positional, so nothing re-anchors
it.
That is the "never recovered" signature. And because prost skips unrecognized
field numbers, a mis-framed protobuf payload still decodes without error: a
fixed64read at a shifted offset returns a plausible number. Wrongprice,sane
size/time, no failure — exactly what was reported.parse_raw_messagealso indexed the first four bytes unguarded, so a body of0–3 bytes panicked with
index out of bounds, killing the dispatcher thread onthe blocking client and the dispatcher task on the async one. Confirmed by test
before the fix.
validate_frame_lengthnow gates both frame readers. The 16 MiB cap matchesConstants.MaxMsgSize, whichEReader.readSingleMessageenforces withBAD_LENGTH. Both directions raise the newError::InvalidFrame.Error::InvalidFrameis classified asis_connection_lost, so bothdispatchers take their reconnect branch rather than shutting the client down.
The socket is open but only a fresh connection can re-anchor the framing. This
widened that predicate's documented meaning from "the socket is gone" to "the
stream is unusable in place"; its doc is updated to match.
Dropping unclaimed frames (
8bb60dc)A frame no channel claimed vanished. The blocking transport logged every such
frame at
infowithout distinguishing cases; the async transport'sroute_to_shared_channelhad noelsearm at all. So an unrecognized messageid — the desync fingerprint, since a slipped read makes every subsequent id
garbage — looked identical to an idle connection. That is why the incident
surfaced data-farm notices and no decode error.
report_unroutable_framesplits them: an unknownIncomingMessageskind warnsand publishes
UNKNOWN_MESSAGE_TYPE_CODE(-5) toClient::notice_streamso consumers can react programmatically; a known kind with no current
subscriber stays at
info, since that is ordinary steady state and raising thesame signal for it would make the signal worthless.
-5joins the existing client-side sentinels (-3,-4); TWS uses 0 and up.Notice::is_handshake_syntheticis deliberately untouched — this is not ahandshake notice.
Together: the desync either cannot happen, or announces itself and reconnects.
Tests
Regression coverage at all three seams —
validate_frame_lengthboundaries,read_messageover an oversized prefix (this one would have allocated 4 GiBbefore the fix), and
parse_raw_messageon 0–3 byte bodies. Unroutable-framereporting is tested at the classifier and end-to-end on both transports:
push a frame with an unknown id through
MemoryStream, drive the dispatcher,assert the notice lands on the notice stream.
cargo testcargo test --no-default-features --features synccargo test --all-featuresjust rules-checkNot fixed here
The root cause is not confirmed. The oversized-length path is the most
plausible explanation for the 2026-07-07 corruption, but no capture exists — and
the audit found that
MessageRecordercould not have produced one: it runs onthe already-parsed message, so the length prefix is never recorded and unknown
ids collapse to
NotValid = -1. Capturing a real desync needstcpdumpor atee inside the socket read.
The falsification condition is recorded in the plan: corruption recurring
without an accompanying
InvalidFrameorUNKNOWN_MESSAGE_TYPE_CODEnoticewould rule this out and put the prost-mis-decode hypothesis back in play.
Also left open, and tracked in the plan: the raw numeric message id is not
threaded into the notice (it has already collapsed to
NotValidby the timerouting runs — naming it needs a
ResponseMessageshape change), and the 3×ramp in the original report remains unexplained.