Skip to content

fix(transport): validate frame lengths and surface unroutable frames - #756

Merged
wboayue merged 3 commits into
mainfrom
fix/frame-length-validation
Aug 9, 2026
Merged

fix(transport): validate frame lengths and surface unroutable frames#756
wboayue merged 3 commits into
mainfrom
fix/frame-length-validation

Conversation

@wboayue

@wboayue wboayue commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Closes the first two findings of the tick-by-tick decode-desync investigation
(plans/tick-by-tick-reconnect-decode-desync.md), which was filed against a
live all_last() subscription that began emitting prices with a constant
+1,337,581,200 offset after a usfarm.nj data-farm reconnect and never
recovered — with no decode error anywhere.

The audit found the framing path trusts the wire completely. Two defects, both
absent from the equivalent path in the official C# client.

Trusting the length prefix (fe8910d)

Nothing bounded the 4-byte big-endian length prefix. Four garbage bytes were
read as a body length of up to 4 GiB, which allocated that much and then
blocked in read_exact until that many bytes arrived — consuming and
destroying every real message in between, then yielding one bogus frame with
the stream left mis-framed. The framing is positional, so nothing re-anchors
it.

That is the "never recovered" signature. And because prost skips unrecognized
field numbers, a mis-framed protobuf payload still decodes without error: a
fixed64 read at a shifted offset returns a plausible number. Wrong price,
sane size/time, no failure — exactly what was reported.

parse_raw_message also indexed the first four bytes unguarded, so a body of
0–3 bytes panicked with index out of bounds, killing the dispatcher thread on
the blocking client and the dispatcher task on the async one. Confirmed by test
before the fix.

validate_frame_length now gates both frame readers. The 16 MiB cap matches
Constants.MaxMsgSize, which EReader.readSingleMessage enforces with
BAD_LENGTH. Both directions raise the new Error::InvalidFrame.

Error::InvalidFrame is classified as is_connection_lost, so both
dispatchers take their reconnect branch rather than shutting the client down.
The socket is open but only a fresh connection can re-anchor the framing. This
widened that predicate's documented meaning from "the socket is gone" to "the
stream is unusable in place"; its doc is updated to match.

Dropping unclaimed frames (8bb60dc)

A frame no channel claimed vanished. The blocking transport logged every such
frame at info without distinguishing cases; the async transport's
route_to_shared_channel had no else arm at all. So an unrecognized message
id — the desync fingerprint, since a slipped read makes every subsequent id
garbage — looked identical to an idle connection. That is why the incident
surfaced data-farm notices and no decode error.

report_unroutable_frame splits them: an unknown IncomingMessages kind warns
and publishes UNKNOWN_MESSAGE_TYPE_CODE (-5) to Client::notice_stream
so consumers can react programmatically; a known kind with no current
subscriber stays at info, since that is ordinary steady state and raising the
same signal for it would make the signal worthless.

-5 joins the existing client-side sentinels (-3, -4); TWS uses 0 and up.
Notice::is_handshake_synthetic is deliberately untouched — this is not a
handshake notice.

Together: the desync either cannot happen, or announces itself and reconnects.

Tests

Regression coverage at all three seams — validate_frame_length boundaries,
read_message over an oversized prefix (this one would have allocated 4 GiB
before the fix), and parse_raw_message on 0–3 byte bodies. Unroutable-frame
reporting is tested at the classifier and end-to-end on both transports:
push a frame with an unknown id through MemoryStream, drive the dispatcher,
assert the notice lands on the notice stream.

Gate Result
cargo test 1351 + 209
cargo test --no-default-features --features sync 1374 + 216
cargo test --all-features 1713 + 321
clippy ×3 configs clean
rustdoc ×3 configs clean
integration crates (sync + async) build
examples ×2 configs build
just rules-check 33 nodes resolve

Not fixed here

The root cause is not confirmed. The oversized-length path is the most
plausible explanation for the 2026-07-07 corruption, but no capture exists — and
the audit found that MessageRecorder could not have produced one: it runs on
the already-parsed message, so the length prefix is never recorded and unknown
ids collapse to NotValid = -1. Capturing a real desync needs tcpdump or a
tee inside the socket read.

The falsification condition is recorded in the plan: corruption recurring
without an accompanying InvalidFrame or UNKNOWN_MESSAGE_TYPE_CODE notice
would rule this out and put the prost-mis-decode hypothesis back in play.

Also left open, and tracked in the plan: the raw numeric message id is not
threaded into the notice (it has already collapsed to NotValid by the time
routing runs — naming it needs a ResponseMessage shape change), and the 3×
ramp in the original report remains unexplained.

wboayue added 3 commits August 8, 2026 22:08
Nothing bounded the 4-byte big-endian length prefix. Four garbage bytes
were read as a body length of up to 4 GiB, which allocated that much and
then blocked in read_exact until that many bytes arrived — consuming and
destroying every real message in between, then yielding one bogus frame
with the stream left mis-framed. The framing is positional, so nothing
re-anchors it, and a mis-framed protobuf payload still decodes without
error (prost skips unrecognized field numbers). The visible symptom is
plausible-looking wrong field values that never recover.

parse_raw_message also indexed the first four bytes unguarded, so a body
of 0-3 bytes panicked with "index out of bounds" — killing the dispatcher
thread on the blocking client and the dispatcher task on the async one.

validate_frame_length now gates both frame readers: bodies below the
4-byte message id and lengths past 0x00FFFFFF are rejected, matching the
official client's Constants.MaxMsgSize (EReader.readSingleMessage raises
BAD_LENGTH). Both raise the new Error::InvalidFrame, which
is_connection_lost reports as true so the dispatchers reconnect rather
than shut down — the socket is open but only a fresh connection can
re-anchor the framing.

Audit findings and the remaining follow-ups are in
plans/tick-by-tick-reconnect-decode-desync.md.
A frame no channel claimed was dropped without a signal. The blocking
transport logged every such frame at info without distinguishing the
cases; the async transport's route_to_shared_channel had no else arm at
all. So an unrecognized message id -- the fingerprint of a framing
desync, since a slipped read makes every subsequent id garbage -- looked
exactly like an idle connection. That is why the 2026-07-07 incident
surfaced data-farm notices and no decode error.

report_unroutable_frame splits the two: an unknown IncomingMessages kind
warns and publishes UNKNOWN_MESSAGE_TYPE_CODE (-5) to the notice stream,
so consumers can react programmatically; a known kind with no current
subscriber stays at info, since that is ordinary steady state and
raising the same signal for it would make the signal worthless.

The code joins the existing client-side sentinels (-3 handshake unknown
frame, -4 handshake decode failure); TWS uses 0 and up. Notice::
is_handshake_synthetic is deliberately untouched -- this is not a
handshake notice.

Follow-ups in plans/tick-by-tick-reconnect-decode-desync.md.
Cleanup pass over the two framing commits. No behavior change.

parse_raw_message uses split_first_chunk, which does the length check,
the array conversion, and hands back the remainder in one call -- so
both data[MIN_FRAME_LENGTH..] re-slices go away. route_to_shared_channel
goes back to if-let/else; the match reindented an untouched loop for one
new branch. The two validate_frame_length reject tests collapse into one
table, matching the form the sync reader test already used, and that
test's hardcoded 3 derives from MIN_FRAME_LENGTH now.

Tests build frames through encode_protobuf_message and
encode_raw_length instead of hand-rolling the id offset and the length
prefix.

The desync narrative was written out six times across errors.rs,
transport/common.rs, and messages.rs. It lives on Error::InvalidFrame
now; the internal sites keep their local claim and point there.

Two doc errors fixed. -2 was described as a notice-code sentinel
alongside -3/-4, but it is IncomingMessages::Shutdown, a message-type
discriminant -- no notice is ever synthesized with it; the neighbouring
HANDSHAKE_UNKNOWN_FRAME_CODE doc carried the same conflation plus an
"only other sentinel" claim that -5 had invalidated. And
UNKNOWN_MESSAGE_TYPE_CODE promised to distinguish a slipped stream from
a new IBKR message type, which it cannot do without naming the id.

Adds a test pinning NotValid to ByMessageType routing -- the unknown-id
alarm depends on it and nothing guarded it. Restructuring findings from
the pass are recorded in plans/.
@wboayue
wboayue merged commit 1d4fc4d into main Aug 9, 2026
4 checks passed
@wboayue
wboayue deleted the fix/frame-length-validation branch August 9, 2026 06:00
wboayue added a commit that referenced this pull request Aug 9, 2026
…rdings (#757)

IncomingMessages::from is lossy -- every unrecognized value collapses to the
single NotValid variant -- and ResponseMessage::from_protobuf kept only that,
so a frame's numeric id was destroyed at construction, before routing ever saw
it. The UNKNOWN_MESSAGE_TYPE_CODE notice added in #756 therefore could not say
which id it meant: it carried a fixed string, and the warn! printed an empty
field list because a protobuf frame has no fields[0] to fall back on. During
the incident that machinery exists for, an operator would get N byte-identical
notices.

ResponseMessage carries message_id alongside kind now -- the value kind was
resolved from, with the PROTOBUF_MSG_ID offset already removed -- and both
constructors funnel through a private new() that computes kind from it, so the
invariant kind == IncomingMessages::from(message_id) has one home instead of
being asserted by hand in two struct literals. ResponseMessage is pub(crate),
so no public API changes.

That is what makes the notice diagnostic rather than decorative: scattered ids
mean the framing slipped, one repeated id means IBKR added a message type.

Two further consumers had the same loss. The wire recorder reconstructed the id
with message_type() as i32, which agrees with the arriving id for every
recognized message but yields -1 for an unrecognized one -- so
IBAPI_RECORDING_DIR fabricated an id in precisely the capture worth replaying.
And the handshake catch-all logged the kind alone, rendering the bare NotValid
in exactly the reconnect window this investigation is about. Both name the id
now.

Rejected while here, recorded so it is not re-litigated: folding kind into a
computed message_type() would undo a deliberate cache with 51 non-test call
sites, and NotValid(i32) is impossible -- IncomingMessages is a pub fieldless
enum used as a HashMap key in both dispatchers, with `as i32` casts and const
arrays.

The root cause of the original 2026-07-07 corruption remains unconfirmed; see
plans/tick-by-tick-reconnect-decode-desync.md for the falsification condition
and the remaining follow-ups.
wboayue added a commit that referenced this pull request Aug 9, 2026
…_DIR (#758)

* feat(transport): capture the raw inbound stream via IBAPI_RAW_CAPTURE_DIR

Step 3 of plans/tick-by-tick-reconnect-decode-desync.md. F1/F2 (#756) made a
framing desync loud and F3 (#757) made it observable; neither made it
capturable.

IBAPI_RECORDING_DIR cannot: record_response is handed an already-parsed
message and re-frames it, so the 4-byte length prefix it writes is one this
crate computed. That prefix is the field a desync corrupts, which makes the
recorder blind to exactly the failure worth recording.

RawFrameTap taps the socket below the framing. Both frame readers record the
prefix before validate_frame_length can reject it, so a prefix that never
reaches a caller still reaches the capture. A reconnect starts a new file, so
no .bin splices two TCP streams. A sidecar .idx carries
seq,utc_timestamp,offset,declared_length — the .bin has no clock, and lining a
desync up against a data-farm notice in an operator's log needs one.

Because the prefixes are the wire's own, a .bin replays through the frame
reader unchanged; tests assert that against both readers.
examples/replay_raw_capture.rs walks a capture and names the first frame whose
prefix cannot describe a frame.

Also records F8 in the plan, found while wiring this: on the blocking client a
1s SO_RCVTIMEO landing mid-frame makes read_exact discard bytes it already
consumed, and the dispatcher treats the timeout as benign and reads on at a
shifted boundary. Verified against a live socket. It needs no corrupt bytes at
all, only a stall, which makes it a better fit for the 07-07 incident than F2.
Not fixed here.

* refactor(transport): /simplify pass on the raw-frame tap

- collapse the tap's three encodings of "off" into one: `Sink` now opens
  segment 0 eagerly and returns a disabled tap if that fails, so `State` drops
  its `disabled` flag and `segment: None` means exactly one thing
- tests use `encode_raw_length` instead of a local `framed()` copy, and a
  `record_frame` helper instead of open-coding prefix+body at nine sites
- `test_disabled_tap_writes_nothing` asserted a fresh TempDir was empty, which
  a disabled tap cannot affect; it now checks what it meant to
- move the async tap seam test to `async/io_tests.rs`, beside the function it
  tests, per the sibling-test-files convention
- `replay_raw_capture`: key the histogram on a `Copy` FrameKind rather than a
  per-frame `String`, derive `trailing` at print, single usage string, and
  state the real reason the framing constants are copied (they are pub(crate),
  not "avoiding internals" — the example does link the crate)
- document why the capture files are deliberately unbuffered, and that the
  async writes block a runtime worker

Also fixes a pre-existing panic surfaced by the review: `MessageRecorder::from_env`
unwrapped `create_dir_all`, so an unwritable `IBAPI_RECORDING_DIR` aborted
`Client::connect`. Same policy as the tap now — warn and disable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant