fix(transport): name the offending message id in diagnostics and recordings - #757
Merged
Conversation
The unknown-message-id notice could not say which id it meant. IncomingMessages::from is lossy -- every unrecognized value collapses to the single NotValid variant -- and from_protobuf kept only that, so the id was destroyed at construction, before routing ever saw the frame. An operator hitting a desync got N byte-identical, id-less notices, and the warn! printed an empty field list because a protobuf frame has no fields[0] to fall back on. ResponseMessage carries message_id alongside kind now: the numeric value kind was resolved from, with the PROTOBUF_MSG_ID offset already removed so it is the value that was actually looked up. Both the log line and the notice text interpolate it. That is what makes the notice diagnostic rather than decorative -- scattered ids mean the framing slipped, one repeated id means IBKR added a message type. UNKNOWN_MESSAGE_TYPE_CODE's doc was softened in ae722e8 precisely because it promised that discrimination and could not deliver it; it now states it and it is true. Observability policy stays in transport, per the altitude review that raised this: report_unroutable_frame still decides what is worth reporting, the parse layer just stops discarding the id. Tests cover the id surviving both framings -- protobuf, which has no fields[0] fallback, and text, which needs an id at or below PROTOBUF_MSG_ID to reach that branch at all.
…ved to Cleanup pass over 73db598, plus two places the same information loss survived. The recorder reconstructed a frame's id with message_type() as i32. IncomingMessages::from maps a value to the variant with that discriminant, so the two agree for every recognized id -- but an unrecognized one resolves to NotValid, discriminant -1. IBAPI_RECORDING_DIR therefore wrote a fabricated id into exactly the capture worth replaying: an operator recording a desync burst got -1 where the offending id belonged. It reads message_id() now. The handshake catch-all had the same gap one phase earlier -- it logs {kind:?}, which renders the bare NotValid for any unrecognized id, in precisely the reconnect window the investigation is about. It names the id alongside the kind now. ResponseMessage gains a private new() that computes kind from message_id, so the invariant kind == IncomingMessages::from(message_id) has one home instead of being asserted by hand in two struct literals. That also retires the doubled -1 sentinel in from_text_fields. Rejected while here, recorded so it is not re-litigated: folding kind into a computed message_type() would undo a deliberate cache with 51 non-test call sites, and NotValid(i32) is impossible -- IncomingMessages is a pub fieldless enum used as a HashMap key in both dispatchers, with `as i32` casts and const arrays. Tests move the thrice-repeated unroutable-frame fixture into helpers::unknown_message_frame / UNKNOWN_MESSAGE_ID, so the assertions derive the id rather than spelling it. Deferrals in plans/.
wboayue
added a commit
that referenced
this pull request
Aug 9, 2026
…_DIR (#758) * feat(transport): capture the raw inbound stream via IBAPI_RAW_CAPTURE_DIR Step 3 of plans/tick-by-tick-reconnect-decode-desync.md. F1/F2 (#756) made a framing desync loud and F3 (#757) made it observable; neither made it capturable. IBAPI_RECORDING_DIR cannot: record_response is handed an already-parsed message and re-frames it, so the 4-byte length prefix it writes is one this crate computed. That prefix is the field a desync corrupts, which makes the recorder blind to exactly the failure worth recording. RawFrameTap taps the socket below the framing. Both frame readers record the prefix before validate_frame_length can reject it, so a prefix that never reaches a caller still reaches the capture. A reconnect starts a new file, so no .bin splices two TCP streams. A sidecar .idx carries seq,utc_timestamp,offset,declared_length — the .bin has no clock, and lining a desync up against a data-farm notice in an operator's log needs one. Because the prefixes are the wire's own, a .bin replays through the frame reader unchanged; tests assert that against both readers. examples/replay_raw_capture.rs walks a capture and names the first frame whose prefix cannot describe a frame. Also records F8 in the plan, found while wiring this: on the blocking client a 1s SO_RCVTIMEO landing mid-frame makes read_exact discard bytes it already consumed, and the dispatcher treats the timeout as benign and reads on at a shifted boundary. Verified against a live socket. It needs no corrupt bytes at all, only a stall, which makes it a better fit for the 07-07 incident than F2. Not fixed here. * refactor(transport): /simplify pass on the raw-frame tap - collapse the tap's three encodings of "off" into one: `Sink` now opens segment 0 eagerly and returns a disabled tap if that fails, so `State` drops its `disabled` flag and `segment: None` means exactly one thing - tests use `encode_raw_length` instead of a local `framed()` copy, and a `record_frame` helper instead of open-coding prefix+body at nine sites - `test_disabled_tap_writes_nothing` asserted a fresh TempDir was empty, which a disabled tap cannot affect; it now checks what it meant to - move the async tap seam test to `async/io_tests.rs`, beside the function it tests, per the sibling-test-files convention - `replay_raw_capture`: key the histogram on a `Copy` FrameKind rather than a per-frame `String`, derive `trailing` at print, single usage string, and state the real reason the framing constants are copied (they are pub(crate), not "avoiding internals" — the example does link the crate) - document why the capture files are deliberately unbuffered, and that the async writes block a runtime worker Also fixes a pre-existing panic surfaced by the review: `MessageRecorder::from_env` unwrapped `create_dir_all`, so an unwritable `IBAPI_RECORDING_DIR` aborted `Client::connect`. Same policy as the tap now — warn and disable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #756, which added the
UNKNOWN_MESSAGE_TYPE_CODEnotice but couldnot say which id it meant.
The information loss
IncomingMessages::fromis lossy — every unrecognized value collapses to thesingle
NotValidvariant — andResponseMessage::from_protobufkept only that.So the id was destroyed at construction, before routing ever saw the frame. By
the time
report_unroutable_framematchedNotValid, both of its outputs wereid-less: the notice carried a fixed string literal, and the
warn!printed anempty field list, because a protobuf frame has no
fields[0]to fall back on.During exactly the incident this machinery was built for, an operator would get
N byte-identical notices.
The fix
ResponseMessagenow carriesmessage_idalongsidekind: the numeric valuekindwas resolved from, with thePROTOBUF_MSG_IDoffset already removed soit is the value that was actually looked up.
ResponseMessageispub(crate),so this is not a public API change. Both constructors funnel through a private
new()that computeskindfrom it, so the invariantkind == IncomingMessages::from(message_id)has one home rather than beingasserted by hand in two struct literals.
That is what makes the notice diagnostic rather than decorative: scattered ids
mean the framing slipped; one repeated id means IBKR added a message type.
Two more places the same loss survived
Found by the cleanup pass, both beyond the original scope but the same defect:
recorder.rsreconstructed theframe's id with
message_type() as i32. That agrees with the arriving id forevery recognized message —
IncomingMessages::frommaps a value to thevariant with that discriminant — but an unrecognized id resolves to
NotValid, discriminant-1. SoIBAPI_RECORDING_DIRwrote-1intoprecisely the capture worth replaying: an operator recording a desync burst
lost the one field identifying the fault. Now reads
message_id().in
connection/common.rslogs the kind only, which renders the bareNotValidfor any unrecognized id — in exactly the reconnect windowplans/tick-by-tick-reconnect-decode-desync.mdis about. It names the idalongside the kind now.
Considered and rejected
Recorded so they are not re-litigated:
kindinto a computedmessage_type()would undo a deliberatecache with 51 non-test call sites.
NotValid(i32)is impossible:IncomingMessagesis apubfieldlessenum used as a
HashMapkey in both dispatchers, withas i32casts andconstarrays. A payload-carrying variant would silently splitshared-channel lookup, besides being a public breaking change.
Tests
The id surviving both framings — protobuf, which has no
fields[0]fallback, and text, which needs an id at or below
PROTOBUF_MSG_IDto reachthat branch at all — plus a recorder regression test, plus id assertions on the
three existing unroutable-frame tests. The thrice-repeated fixture moved to
helpers::unknown_message_frame/UNKNOWN_MESSAGE_ID, so assertions derivethe id instead of spelling it.
cargo testcargo test --no-default-features --features synccargo test --all-featuresjust rules-checkStill open on this arc
The root cause of the original 2026-07-07 corruption remains unconfirmed —
see
plans/tick-by-tick-reconnect-decode-desync.md. This makes a recurrencesubstantially easier to diagnose but confirms nothing on its own. Remaining
there: the raw-frame tap (F7) and five structural follow-ups.