app, gpu, gpu-deepseek41, gpu-gates, gpu-glm5next, levers, model, placement: node 1 serves its expert tier to node 0 over a control channel, and a lost node ends both sides

[juliankang4]

What and why

This is the tier server row of the P5 plan in #127, docs/contrib/dgx-spark.md 9.5 items 5 and 6. Node 1 serves its share of the routed experts to node 0 over a control channel, the load plans its tier card from the census node 1 sends, and a lost node ends both sides by name. It is built and checked over loopback, two processes on one DGX Spark. The run on two Sparks needs the transport row of #127 and is not part of this PR. Part of #13.

The branch is four commits on main (6f87bfb8..18ae4660). Since the body was last edited it adds a check that both nodes read the same tier bytes (below), it is rebased over #172 and #174 and then onto main 6f87bfb8, and it fixes the review bot's three must-fixes at c891688f: a send checks its deadline after its last write too, the nodes gate joins a child's stderr reader before it reads the loss lines, and the gate's relay takes the launch tag from wire::Tag. #172 moved the host's reads, stores, releases and resets of the tier page's words into Link, so this branch's node side of the page now goes through those methods, and a loopback link with a tier on a node is refused by name ("After the rebase" below). The DGX Spark runs under "What I ran" open with 18ae4660 itself on main 6f87bfb8; the runs of earlier heads follow. The commits:

1. 582cabac, a refactor with no behavior change: TierCard keeps the stage side, and a tier card's own work on its card moves into TierWork.
2. 21c7e7a7, the tier server, the channel, the load's plan from the census, and both ends of a lost node.
3. f652ad2e, the docs.
4. 18ae4660, a rename with no behavior change: Link::stage_end is Link::own_end, the end of the context that made or opened the page, since a tier node binds its page through it too.

How it works:

• Two processes, one tier. Node 0 is a Qwen3.8 load whose plan has an expert tier, with BLOOMERY_TIER_NODES=<ip:port>[,...] naming the node of each tier in the plan's tier order. The lever takes loopback addresses only and refuses any other by name: the page below is this host's memory, and the channel has no authentication. Node 1 runs tier_server --listen <ip:port> [-m <first shard>]. It serves one load and exits once node 0 closes it; a connection that opened no tier (a census read, a refused open) leaves it serving the next. Unset, every tier is a card of the loading process, as before.
• Control channel (crates/gpu/src/host/node.rs, node/wire.rs): TCP from std::net, one connection a load, one reply a request. A frame is a u32 length, a u8 tag and little-endian fields. The requests are open, bind, launch, capture, enqueue, sync, drain, clear fault, fault read and close. Every request gets Ok with its payload or Err with an error kind and the server's text, and a fault on node 1 comes back to node 0 as a fault, with its layer and sites. On node 0 each request and its reply run under one deadline; node 1 takes its reply's deadline once the request's work returns, so an open or a drain that runs long still gets its reply. A send is late when its deadline has passed after any of its writes, the last one included, and a read once a 50 ms read slice that began past it finds no frame. The server keeps watching the tier's progress while a request is still arriving.
• Census before the plan (9.2). On connect node 1 sends its census: each device's census row. A census of another version, with no device, a free count above the total or a device listed twice is refused by name. node_cards reads the nodes' censuses and makes each tier card with spec_of_device from the row its node sent. Place::on in crates/gpu-gates (the placement of the generate and serve bins and of the gates) calls it when the lever is set, so such a load plans plan (b′) with node 1's card as the tier card before it loads anything, and the gate plans through the same function. A load placed another way plans its tier card from its own census, and node 1 then opens it only if its census lists the same device (TierNode::open refuses another by name).
• The open. Node 0 sends the tier's rows of its plan (tensor, card format, expert ids, resident bytes, and the sha256 of the row's bytes in node 0's file), the tier's set and its shape. Node 1 refuses a routed row whose experts are not the tier's set for that layer, then checks each tensor by name against node 0's description and refuses another shard, type, shape or size by name (NodeOpen::rows_on, Format::of_word). It then hashes each row's bytes in its own file and refuses a row whose sha256 is not node 0's, naming the tensor and its experts. Only then does it open its card and load the rows from its own copy of the file with Weights::load_rows.
• Which bytes the open hashes. A row's sha256 covers the file bytes of the experts the tier holds, in slot order: exactly the bytes node 1 uploads. SegmentFile::of (weights.rs) finds a segment's spans in the file and borrows their bytes for the upload and for the digest alike, and model::fileio::sha256_hex_of hashes them one span after another. A copy of the model whose headers match node 0's file but whose tier experts differ is refused by name instead of serving other weights; bytes outside the tier's rows do not reach node 1's computation, so they are not hashed. The table below compares the ways to check that both nodes read the same model.
• Requests on node 1. Each request is checked before it reaches the tier's stream. One that fails after that ends the connection as a lost node 0 does (below). A second bind is refused before the mapping changes, and a fault clear while work is in flight is refused. No request waits on the stream without a bound.
• The tier card, split (commit 1). TierCard keeps the stage side: the books, the go order each graph was captured for, the page words it reads and the stage's stream. Its tier side is TierWork, a card of this process, or TierNode, a client of the channel. TierWork is the tier half that existed before (its Gpu, weights, the family's TierExperts, staging, graphs and its end of the page). The tier server holds one and calls the same methods. The replay or feed decision, the settle, the release and the reset stay on the stage side for both.
• The page across two processes. Over loopback the tier page is one file under /dev/shm, created with mode 0600. Node 0 makes it, maps it and registers it with its card (MappedHost::shared_file: cuMemHostRegister with PORTABLE | DEVICEMAP). Node 1 opens only a tier page's name there, then maps and registers the same file. Node 0 removes the file's name once node 1 has bound it, and a creation that fails removes it too, so a killed process leaves nothing under /dev/shm. Both cards reach one physical page, so the page's protocol runs unchanged across the two processes. The in-process tier page is MappedHost::new (cuMemHostAlloc, not write-combined), so neither page is write-combined and the bits below compare like with like.
• Lost node 1. On node 0 a request that fails, or gets no reply within its deadline (the go deadline for a step's requests), marks the node lost (GpuError::Link). The host tier then names the tier a lost card (CardLost), and the next step and a reset are refused as for a lost card of node 0's. A node that answers but stops signalling is still the go deadline's.
• Lost node 0. Node 1 ends on two conditions: the channel closes with no close request, or work asked of it makes no progress past the go deadline and its grace (node 0 stopped, its channel still open). Either way node 1 writes RELEASE to its own go words, so every wait its stream holds passes, drains its stream under the drain deadline, and returns the loss by name. tier_server prints it once, as its exit error, and exits 1. The loss names how many of the layers asked of it it had served and how long node 0 sent no go. If the drain fails, the work is not dropped, so nothing frees memory the stream may still use.
• A stopped node 0 that runs again. It first takes the reply node 1 sent while it was stopped: a read is late only when a read that began after its deadline finds no frame. Its stage then finds node 1's go words released when it settles the step. The settle refuses the step with the tier a lost card, naming the release, so no token comes from rows node 1 served after it.

How the open checks the model bytes, and what it costs

Way	Same headers, other tier weights refused?	Bytes each node hashes per open, Qwen3.8 UD-Q4_K_XL [derived]	New code
sha256 of every shard (sha256_hex(g.mapping())) on both nodes	yes	111,334,654,784 B, the whole file	a field and a function
refset's header digest (the census of a reference set)	no, header bytes only
headpick's vocabulary digest	no, vocabulary only
a download's digest (the repository's LFS sha256, hf::fetch's check)	yes for the whole file, but no load reads it and a file made locally has none	the whole file, or a manifest both nodes would have to read	a manifest reader
sha256 of the tier rows' bytes (this PR)	yes, exactly the bytes node 1 uploads	the tier rows' file bytes, which the open prints: 14,639,820,800 B for the 144 rows of this plan's tier, 13 % of the file	a NodeRow field, segment_sha256 and sha256_hex_of; SegmentFile is the upload's own lookup moved out

Each node hashes B bytes on one core, so the open takes B / R more on node 0 (before it sends the open) and B / R more on node 1 (inside the open), where R is one core's sha256 rate. That rate has not been measured on the GB10, so no time is given here. The whole-file digest takes 111,334,654,784 / 14,639,820,800 = 7.6 times the bytes [derived], and on a 128 GB machine it also pulls the whole file through the page cache on every open. Node 1 reads its tier rows to upload them anyway, so its digest reads no byte the load would not. The open's deadline on node 0 still holds the digest and the upload together: 120 s plus 14,639,820,800 B at 100 MiB/s is 259.6 s [derived] for this plan.

What a node tier adds to each pass

Derived from the code, not measured. RTT is one request's round trip over the loopback channel: node 0 writes the frame, node 1 reads it, does the call and writes its reply, node 0 reads it (TierNode::call).

• A replayed pass (each decode step after the first feeds): before the host serves the pass's first layer, node 0 sends one launch request to each node tier with a layer in the pass and waits for its reply (TierCard::replay, TierNode::launch). The host's service of that pass starts at most n_node_tiers × RTT later. A tier of node 0's own launches the same graph with a call, so the round trip is the extra.
• A fed pass (a chain's first pass, before its graph is captured): node 0 sends one enqueue request for each tier layer the pass serves, each before the host serves that layer (HostTier::feed_tiers, TierNode::enqueue), and one capture request after the pass settles. That adds at most (L_served + 1) × RTT.
• The gate's history on the real file has 82 replayed and 4 fed passes, with the tier on 48 layers: at most 82 + 4 × 49 = 278 round trips for the whole history [derived].
• The settle reads the tier's words and fault copy from the shared page, with no request.

The decode A/B below ran plan (a) on one GB10, which holds every routed expert on the card and has no tier card. It times the HostTier path the split touches and covers none of these terms. A same-plan A/B of a tier of node 0's own against a node tier needs a second Gpu on one card, which only the gate's harness opens (the decode runner refuses --place bp on one card), so it is not run here. The transport row replaces the launch message: it can ride in the first image of the mirrored pages (9.5 item 5).

What this PR does not do:

• plan_nodes (placement: a model's plan over several unified nodes (dgx-spark.md §9), and nodes_scale on one to four Sparks #95) is not called. It needs read rates, a NIC figure and per-node budgets that nothing measures on a live load yet. The census carries no host room: node 1's MemAvailable reading joins it with the plan term that reads it, once each node has a pool of its own in the plan (9.2).
• A remote tier serves no prompt batch. The batch port refuses a load with a tier on a node at its group start, by name.
• Only Qwen3.8 opens a tier on a node (NODE_FAMILY in arch/qwen3moe/tier38.rs). GLM and V4.1 open their tier cards through TierOpen::open_here, which refuses a tier on a node by name before any card opens.
• No tier node off this host. Besides the loopback-only lever, the channel has no keepalive. Over loopback a dead peer ends the connection at once, and every request has its deadline. The open's deadline is derived, not measured: 120 s plus the tier's resident bytes at 100 MiB/s.
• The RDMA link replaces the shared file with its mirrored pages, and the launch message can then ride in the first image (9.5 item 5). That is the transport row.

After the rebase over #172. #172 gave Link the methods the host uses on the tier page's words (load, store, release, at_rest, reset). This branch now builds on them:

• Link keeps gpu, levers, justfile, tools, docs: the tier page's counters cross a link as generations; a loopback link carries them between two pages on one machine with a host pump #172's fields and methods. Link::across, Link::open_across and the file's removal on drop sit beside them, and a shared page has no mirror. The node side's word reads, stores and releases, which went through this branch's own word() and host_at before, now go through Link::load, store and release.
• TierPage::new is gpu, levers, justfile, tools, docs: the tier page's counters cross a link as generations; a loopback link carries them between two pages on one machine with a host pump #172's, unchanged. A load with a tier on a node builds its page with TierPage::across, and both go through one constructor for the page's words.
• Node 1 still releases every count word, then every go word (release_opened), now with one Link::release call.
• A word past the page's layout was skipped on the node side and now panics, as gpu, levers, justfile, tools, docs: the tier page's counters cross a link as generations; a loopback link carries them between two pages on one machine with a host pump #172's Link does on the stage side. bind_opened checks the page's size against the layout before any word is read, so no request reaches that.
• New: with a tier on a node, BLOOMERY_TIER_LINK=loopback is refused by name (host/mod.rs, attach_tiers), because the loopback link's tier side is a page of node 0's process and node 1 maps the shared page. The refusal comes when node 0 attaches its tiers, after node 1 has opened its rows. No gate runs it yet.

At 18ae4660 these ran on a DGX Spark ("What I ran"): the PTX scans, gate-gpu-lib, gate-gpu-qwen38-nodes on the fixture and on UD-Q4_K_XL (all six clauses, among them the other-bytes open and the two lost node 0 clauses the stderr fix reaches), and the FAIL-first mutants of the send deadline and of the stderr reader. gate-gpu-ds41-tier and weekly-gpu-ds41-lost, which run the merged page code, end with exit 66 as shipped on that machine, at main as at the head, because it has no DSpark fixture draft; the same gate passed its landing and lost clauses there, over the shared page and over loopback, with two local edits.

Where each part lives

• crates/gpu/src/host/tier.rs (commit 1): the TierCard split into the stage side and TierWork. TierCard::nodes_per_layer and the books are unchanged. Commit 2 adds TierSide, the node calls, the release check the settle reads and TierOpen::open_here.
• crates/gpu/src/host/node.rs: NodeCensus, node_cards, NodeOpen and NodeRow (the open's rows, each with its bytes' sha256, segment_sha256), TierNode (node 0's client: the census, the open, the per-request calls and lost), NodeFamily (a family's tensor reader and tier computation for the server) and serve (node 1's loop, the stopped check and the end on a lost node 0).
• crates/gpu/src/host/node/wire.rs: frames, tags (Tag is public, and the nodes gate's relay reads the launch tag from it), the channel with its deadlines, and the readers and writers of the fields. A payload that ends early or runs past its fields is refused by name. The protocol is version 3.
• crates/gpu/src/weights.rs: SegmentFile::of, the lookup upload_segment did inline (the tensor found and checked in the file, its spans and their bytes), now shared with the digest. crates/model/src/fileio.rs: sha256_hex_of, the sha256 of several slices one after another; sha256_hex calls it.
• crates/gpu/src/host/link.rs: Link::across (node 0 makes and maps the file), Link::open_across (node 1 maps it, a tier page's name only), and the file's removal. Commit 4 renames stage_end to own_end.
• crates/gpu/src/graph.rs: MappedHost::shared_file and the unregister in its drop; a shared page is a whole number of 64 KiB.
• crates/gpu/src/host/mod.rs: attach_tiers opens a node's tier through the channel; a failed request marks the tier lost; the settle refuses a released node tier; the batch port's group start refuses a node tier.
• crates/gpu/src/arch/qwen3moe/tier38.rs and swap38.rs: NODE_FAMILY, and stacks_of_set, the stacks of the layers a tier set gives experts (a node holds one tier's set and no plan).
• crates/gpu-gates/src/generate.rs: Place::on takes the tier cards from node_cards when the lever is set, and a load line's cards field names a node tier's device.
• crates/levers: the BLOOMERY_TIER_NODES row (class C, kind Addrs, loopback only) and tier_nodes().
• crates/placement/src/placement.rs: Format::of_word, the format a word names, with a unit test; Format::ALL is held to the enum by an exhaustive match.
• crates/gpu-gates/src/bin/tier_server.rs, gate_qwen38_nodes.rs and shared/q38_history.rs, the recipe gate-gpu-qwen38-nodes, and the recipe added to 46 of the 62 tools/gate-paths.tsv rows that name gate-gpu-qwen38-twocard: the rows of the app, runtime, tokenizer, model, gpu, gguf, placement and similar sources a Qwen3.8 load reads. The other 16 rows (fixtures, reference-set readers, residency files, vision position rows and the like) are unchanged. The two rows that name gate-gpu-qwen38-twocard among their exceptions, the model readers of mimo2 and nemotron_h, name gate-gpu-qwen38-nodes beside it for the same reason.
• tools/unsafe-ratchet.txt: gpu 1749 to 1754 blocks for the shared file's mapping, its registration, its device pointer, the unregister on that pointer's failure, and MappedHost's drop split into one block a release call, each with its SAFETY line.
• The callers of TierOpen::of_machine, which now returns the lever's error, take a ? (app, gpu-deepseek41, gpu-glm5next and five gate bins).
• docs/contrib/dgx-spark.md: 9.5 items 5 and 6 state what is built and what is not, 9.2 and section 10 say the census reaches node 0 and the load plans from it, and section 1's P5 row names the loopback run.

Code shape

• No second copy. What both Qwen3.8 tier gates hold a load to is one file, crates/gpu-gates/src/bin/shared/q38_history.rs, which gate_qwen38_twocard and gate_qwen38_nodes include: the history (Run, step, window, history, same, WINDOWS), the plan's tier layers (tier_layers), the tier's graph and chain checks after the history (tier_graphs_and_chains) and the nodes a tier layer (TIER_LAYER_NODES). The two-card gate keeps only its residency count, its stage graph line, its tier: stats line and its precondition; the nodes gate keeps its precondition on the tier's experts and resident bytes. Run's write and read in the nodes gate use the channel's own field writer and reader (node/wire.rs). A tier node binds its page through Link::own_end, the stage card's method.

• Model code holds only architecture facts. Qwen3.8's code gains NODE_FAMILY (its tensor reader and tier computation, both existing functions) and stacks_of_set. The node path, the channel and the loss handling are common code.

• Dead code is gone.

• No silent failure. These are each refused by name:

  • a census of another protocol version, with no device, a free count above the total, or a device listed twice;
  • a flag byte other than 0 or 1;
  • a lever address off this host, or a lever that names a different number of nodes than the plan has tiers;
  • an open whose routed rows are not the tier's set, a tensor that differs from node 0's description, or a row whose bytes' sha256 differs from node 0's;
  • a second open or bind, a fault clear with work in flight, a close with a payload, a chain word out of range;
  • a payload that ends early or runs past its fields, a frame over the size limit, an unknown tag, a format word no format has, a stream that ends inside a frame;
  • a node tier on a family without a node opener;
  • a released node tier at the settle.

  The other-file and version refusals, the census refusals and the error reply's kinds (a fault with its words, a protocol error, a refusal) have unit tests (node.rs, where the open's rows read back with their sha256), the channel's frames and ends a loopback test (wire.rs), and the refusal of a node tier for GLM and V4.1 a unit test of TierOpen::open_here (tier.rs); the swapped open and the open on a copy with other tier bytes are gate clauses.

• Comments state what is true now.

• No relaxed gate. No band, tolerance or lint moved. The unsafe ratchet goes up for new code only, with a PIN(2026-10-14) line.

• cargo fmt and cargo clippy are clean on what I touched; the lint count did not go up.

Duplication found or removed (path:line), or "none": the Qwen3.8 history, the tier layers, the tier's graph and chain checks and TIER_LAYER_NODES, now in crates/gpu-gates/src/bin/shared/q38_history.rs, are removed from gate_qwen38_twocard.rs and have no copy in gate_qwen38_nodes.rs. gate_qwen38_residency.rs and the GLM and V4.1 twocard gates keep their own histories, which differ in what they step and compare.

What I ran

On a DGX Spark

The runs here are in the dev container (Hardware, below), with --ipc=host so both processes see one /dev/shm, running tools/gate.sh and tools/gpu-gate.sh directly with BLOOMERY_BOX_LOCAL=1 and PTX for sm_86 (the driver JITs it for the GB10); the V4.1 rows at 18ae4660 went through tools/box.sh (below). Each log starts with the tree's commit; a mutant's log adds the mutant's name.

At 18ae4660, against main 6f87bfb8. Both trees are git archive of the two commits, built from scratch:

At 18ae4660, against main 6f87bfb8	Result
nvidia-smi on the machine of the run	NVIDIA GB10, 12.1, 595.84, CUDA Version 13.2 (block below)
ptx-scan --no-jit of gate_kquant, gate_swap, gate_qwen38_twocard, gate_e2e (gpu), generate_ds41 (gpu, deepseek41) and bloomery-serve-ds41 (deepseek41, vision) on both commits, tools/ref/ptx-canon.py --scans	rc 0 on both, no scan failure; md5 identical=N moved=0 for every bin: 238, 235, 234, 234, 301 and 317
gate-gpu-lib	248 passed (1, 243 and 4 in its three runs), 0 failed, among them a_channel_reads_whole_frames_and_its_ends with the review's expired-send assertion, messages_read_back_as_written and a_tier_on_a_node_is_refused_by_name
gate_qwen38_nodes on the Qwen3.8 fixture	PASSED, clauses: 6 ran, no FAIL line, 124 s; no /dev/shm file and no copy left (block below)
gate_qwen38_nodes on UD-Q4_K_XL	PASSED, clauses: 6 ran, no FAIL line, 555 s; no /dev/shm file and no copy left (block below)
FAIL-first: Channel::send checks its deadline only while bytes are left (at < self.outbuf.len() && Instant::now() > deadline), gate-gpu-lib's command on host::node::wire::tests	red: a_channel_reads_whole_frames_and_its_ends is the one failure, called Result::unwrap_err() on an Ok value; bloomery-gpu compiled again
FAIL-first: wait_within returns without joining the stderr reader, and the reader sleeps 1000 ms before each append; the nodes gate on the fixture	red: FAIL lost node 0 (killed) and FAIL lost node 0 (stopped), each with "the loss named false" and its precondition failing at "past 0 of the 0 layers", then FAILED (exit 1)
The same 1000 ms sleep with the join kept, the nodes gate on the fixture	PASSED, no FAIL line; node 1 ended 1.1 s after node 0 was killed and 11.3 s after it was stopped, against 0.3 s and 10.4 s in the run with no sleep
just gate-gpu-ds41-tier and just weekly-gpu-ds41-lost as shipped, BLOOMERY_TIER=fixture, on both commits	exit 66 on both commits: tools/ref-paths.sh wants the one DSpark fixture draft, which this machine does not have
gate_deepseek41_tier on the V4.1 fixture with the two local edits below: the landing clauses (--union --fault --two --batch --batch2 --bfault) and the lost clauses (--bfirst --blost --lost) on both commits, and both again on the head with BLOOMERY_TIER_LINK=loopback	six runs, all PASSED: gate_deepseek41_tier, no FAIL line; the stage graph 288 nodes, the tier's step and pair graphs 64 and 127, as at #172

Where this run differed from the plan for it:

• The stderr mutant's sleep. The two stderr runs first slept 200 ms, and both passed, the one without the join too. A likely reason, not measured: the reader sleeps once a 4096-byte read, and on this machine node 1 exits more than 200 ms after its last write to stderr, so the reader is done before the gate reads the text. At 1000 ms the run without the join is red and the run with it passes, as the rows above show, and the slower exits in the run with the join are the join waiting on the reader. The two 1000 ms runs were built from the same 18ae4660 tree with BLOOMERY_GIT_COMMIT set to it; their logs have no tree line.
• How the V4.1 rows were built. They ran through just and tools/box.sh in local mode, with a cargo oxide built from cuda-oxide 87bc9518, the workspace's pin. Every other row ran cargo oxide build in the container directly, with the image's cargo oxide from cuda-oxide 4da6c138.
• The V4.1 rows' inputs. The six stand-in runs carry the two edits that the refactor's runs below describe, on both commits and not part of this PR: a draft reserve of 0 B, and a tier depth of at most the real file's 23 experts a layer. They also read engram/corpus-prose.ids, which this machine does not have. The first attempt of the six ended with exit 1 on the missing file. A stand-in from a fixed-seed generator (seed 20261013, sha256 27cc28dbcc6521ec...) was then put in place by hand, and the rows above are the second attempt.
• The nodes gate's corpus. qwen4exp/corpus-prose.ids is not on this machine either, so the nodes gate read the same 4096-id stand-in as every nodes gate run in this PR (its first 64 ids are listed below).

nvidia-smi on the machine of the run:

$ nvidia-smi --query-gpu=name,compute_cap,driver_version --format=csv,noheader
NVIDIA GB10, 12.1, 595.84
$ nvidia-smi | grep 'Driver Version'
| NVIDIA-SMI 595.84                 Driver Version: 595.84         CUDA Version: 13.2     |

On the Qwen3.8 fixture (gate_qwen38_nodes with BLOOMERY_TIER=fixture), abridged: the budget lines between 16 and 8 GiB, the expert lists, the host tier's load lines and node 1's census as node 0 prints it are cut, long lines end in ..., and the data directory shows as $BLOOMERY_DATA. Node 1 prints its loss line once in each lost node 0 clause, as tier_server's exit error.

[tree] head commit 18ae4660fe867c2abae7d3260e20a41ab0d8af5f
[node 1] listening on 127.0.0.1:41195
census of node 1: device 0 "NVIDIA GB10", total 130661761024 B, free 124361629696 B, integrated true
census: the plan's tier card "GB10" takes the free bytes node 1 sent (Some(124361629696) = 124361629696 B): PASS
budget 16 GiB: the tier holds no expert; next
...
budget 8 GiB: no layer before the last holds a host expert (host experts a layer [0, 0, 0, 0]); next
plan (b′) under 7 GiB a card: stage 417 experts (a layer (104, 105)), tier 1489 experts (a layer (372, 373)) on 4 layers, host 142 experts
ok open: node 1 answered the swapped segment with Err(Shape { what: "TierNode", detail: "the node at 127.0.0.1:41195 refused the open request: tier_server: blk.0.ffn_down_exps.weight (layer Some(0)): node 0's plan gives the tier the experts ...
open: the tier's 12 rows hold 5107507200 file bytes, which each node hashes
open: blk.0.ffn_down_exps.weight expert 105: byte 1713607040 of shard 0 flipped in a copy of its 8883314816 B under $BLOOMERY_DATA/gate_qwen38_nodes-206
[node 1] listening on 127.0.0.1:38143
ok open: node 1 on the copy answered the plan's tier with Err(Shape { what: "TierNode", detail: "the node at 127.0.0.1:38143 refused the open request: tier_server: blk.0.ffn_down_exps.weight: the experts Some([105, ...]) hold other bytes in this file (sha256 a04a13340082deb8a766dad6ed4fad2d9e10f5a2732f9a2d3b31f89540157541) than in node 0's (c44b46388f4bfb191566d472d7816dcd851a1454b9330c032846143ea8710a0a): the two processes read other files" })
load in 11.1 s: stage 5779783276 B, tier 1489 experts (5107507200 B) on this process
structure: the tier's graphs of the step and the 2, 3, 4-column verifies hold [Some(37), Some(37), Some(37), Some(37)] nodes, 9 a tier layer and the fault copy (Some(37)): PASS
structure: the tier's chains fed 4 and captured 4 times, replayed 82 times; each of the history's 4 chains fed and captured once, its 86 passes' others replayed (4, 4, 82): PASS
precondition: the tier holds 1489 experts (5107507200 B) and every tier layer was sent a routed slot (missing []): PASS
[node 0] load in 23.0 s: stage 5779783276 B, tier 1489 experts (5107507200 B) on the node at 127.0.0.1:41195
[node 0] node 0: the tier is served by the node at Some(127.0.0.1:41195): PASS
[node 0] structure: the tier's graphs of the step and the 2, 3, 4-column verifies hold [Some(37), Some(37), Some(37), Some(37)] nodes, 9 a tier layer and the fault copy (Some(37)): PASS
[node 0] structure: the tier's chains fed 4 and captured 4 times, replayed 82 times; each of the history's 4 chains fed and captured once, its 86 passes' others replayed (4, 4, 82): PASS
[node 0] precondition: the tier holds 1489 experts (5107507200 B) and every tier layer was sent a routed slot (missing []): PASS
[node 0] lost node 1: kill -KILL 207: exit status: 0
[node 0] ok lost node 1: the step fails in 0.1 s naming the lost node: Hybrid::serve: layer 0: the expert tier on GB10 at the node 127.0.0.1:41195: the launch request: Connection reset by peer (os error 104): the card is lost
[node 0] ok lost node 1: the next step is refused: qwen4exp Body38: position 4: the call there failed after its chain was launched, so the recurrent stores hold it already (they stand at 5); reset ...
[node 0] ok lost node 1: a reset is refused by the lost card's poison: Hybrid::settle: the host tier is poisoned (Hybrid::serve lost the expert tier's card GB10 at layer 0: ...
node 0 ended exit status: 0: its clauses PASS
ok node 0 and node 1: 98 tokens, 98 logits rows bit for bit the reference's
structure: node 0's stage graphs hold [129, 135, 135, 135] nodes, the one-process load's [129, 135, 135, 135]: PASS
node 1 ended Some(ExitStatus(unix_wait_status(9))) (killed by node 0's lost node clause)
[node 0] armed: the next step waits for a line on stdin
[node 1] tier_server: node 0 at 127.0.0.1:59410 is lost: the channel ended with no close, the tier past 12 of the 16 layers asked of it; the tier GB10: the tier released its words and its stream drained
precondition: node 1 was past 12 of the 16 layers node 0 asked of it when node 0 was killed: PASS
ok lost node 0 (killed): node 1 ended Some(ExitStatus(unix_wait_status(256))) 0.3 s after, the loss named true, its words released and its stream drained true
[node 0] armed: the next step waits for a line on stdin
[node 1] tier_server: node 0 at 127.0.0.1:38088 is lost: the tier past 13 of the 16 layers asked of it and node 0 sent no go for 10.1s; the tier GB10: the tier released its words and its stream drained
[node 0] held step: Err(Protocol { what: "Hybrid::serve", detail: "layer 3: the expert tier on GB10 at the node 127.0.0.1:33357: the node released the tier's words, having seen no go from this process past its deadline, so the rows it served since are stale: ...
ok lost node 0 (stopped): continued, node 0's held step gave held step: Err(Protocol { ...
precondition: node 1 was past 13 of the 16 layers node 0 asked of it when node 0 was stopped: PASS
ok lost node 0 (stopped): node 1 ended Some(ExitStatus(unix_wait_status(256))) 10.4 s after, the loss named true, its words released and its stream drained true
clauses: 6 ran, 0 left to the real tier (deferred(real) lines above)
PASSED: gate_qwen38_nodes
gpu-gate.sh: gate_qwen38_nodes ran 124 s (exit 0)
shm after: 0 bloomery-tier files

On the real file, Qwen3.8 UD-Q4_K_XL, abridged the same way. The plan puts 4671 experts on node 1's tier and 17943 on the host:

[tree] head commit 18ae4660fe867c2abae7d3260e20a41ab0d8af5f
census: the plan's tier card "GB10" takes the free bytes node 1 sent (Some(124946776064) = 124946776064 B): PASS
plan (b′) under 16 GiB a card: stage 1962 experts (a layer (40, 41)), tier 4671 experts (a layer (97, 98)) on 48 layers, host 17943 experts
ok open: node 1 answered the swapped segment with Err(Shape { what: "TierNode", detail: "the node at 127.0.0.1:40109 refused the open request: tier_server: blk.0.ffn_down_exps.weight (layer Some(0)): node 0's plan gives the tier the experts ...
open: the tier's 144 rows hold 14639820800 file bytes, which each node hashes
open: blk.41.ffn_down_exps.weight expert 41: byte 50392096 of shard 3 flipped in a copy of its 12087983520 B under $BLOOMERY_DATA/gate_qwen38_nodes-149
ok open: node 1 on the copy answered the plan's tier with Err(Shape { what: "TierNode", detail: "the node at 127.0.0.1:36847 refused the open request: tier_server: blk.41.ffn_down_exps.weight: the experts Some([41, ...]) hold other bytes in this file (sha256 082d4ed32fc6b55bc280a794e39a3ec889b711ffcb5c83abc1029f3164267773) than in node 0's (3e7f52f6dd01906e6a38e706c20e557fcca9ed78d7d777a82c842d8c7bbb373c): the two processes read other files" })
load in 47.4 s: stage 14821690252 B, tier 4671 experts (14639820800 B) on this process
structure: the tier's graphs of the step and the 2, 3, 4-column verifies hold [Some(433), Some(433), Some(433), Some(433)] nodes, 9 a tier layer and the fault copy (Some(433)): PASS
structure: the tier's chains fed 4 and captured 4 times, replayed 82 times; each of the history's 4 chains fed and captured once, its 86 passes' others replayed (4, 4, 82): PASS
[node 0] load in 110.8 s: stage 14821690252 B, tier 4671 experts (14639820800 B) on the node at 127.0.0.1:40109
[node 0] node 0: the tier is served by the node at Some(127.0.0.1:40109): PASS
[node 0] precondition: the tier holds 4671 experts (14639820800 B) and every tier layer was sent a routed slot (missing []): PASS
[node 0] ok lost node 1: the step fails in 0.2 s naming the lost node: Hybrid::serve: layer 0: the expert tier on GB10 at the node 127.0.0.1:40109: the launch request: Connection reset by peer (os error 104): the card is lost
[node 0] ok lost node 1: the next step is refused: ...
[node 0] ok lost node 1: a reset is refused by the lost card's poison: ...
ok node 0 and node 1: 98 tokens, 98 logits rows bit for bit the reference's
structure: node 0's stage graphs hold [1343, 1427, 1427, 1427] nodes, the one-process load's [1343, 1427, 1427, 1427]: PASS
[node 1] tier_server: node 0 at 127.0.0.1:35970 is lost: the channel ended with no close, the tier past 145 of the 192 layers asked of it; the tier GB10: the tier released its words and its stream drained
precondition: node 1 was past 145 of the 192 layers node 0 asked of it when node 0 was killed: PASS
ok lost node 0 (killed): node 1 ended Some(ExitStatus(unix_wait_status(256))) 0.8 s after, the loss named true, its words released and its stream drained true
[node 1] tier_server: node 0 at 127.0.0.1:57868 is lost: the tier past 145 of the 192 layers asked of it and node 0 sent no go for 10.1s; the tier GB10: the tier released its words and its stream drained
[node 0] held step: Err(Protocol { what: "Hybrid::serve", detail: "layer 47: the expert tier on GB10 at the node 127.0.0.1:42343: the node released the tier's words, having seen no go from this process past its deadline, so the rows it served since are stale: ...
ok lost node 0 (stopped): continued, node 0's held step gave held step: Err(Protocol { ...
precondition: node 1 was past 145 of the 192 layers node 0 asked of it when node 0 was stopped: PASS
ok lost node 0 (stopped): node 1 ended Some(ExitStatus(unix_wait_status(256))) 10.6 s after, the loss named true, its words released and its stream drained true
clauses: 6 ran, 0 left to the real tier (deferred(real) lines above)
PASSED: gate_qwen38_nodes
gpu-gate.sh: gate_qwen38_nodes ran 555 s (exit 0)
shm after: 0 bloomery-tier files

At 6b16c8d2, the head before the rebase over #172, against main a13125a8:

At 6b16c8d2, against main a13125a8	Result
nvidia-smi on the machine of the run	NVIDIA GB10, 12.1, 595.84, CUDA Version 13.2
ptx-scan --no-jit of the scan set and gate_qwen38_twocard, tools/ref/ptx-canon.py --scans	base and head rc 0, no scan failure; identical: 238, 235, 234 and 317 entries, moved 0
The pure crates' tests, and cargo test -p bloomery-model --lib -- fileio r8file head_list	209 passed, 0 failed; 3 passed, 0 failed
gate-gpu-lib	243 passed (1, 238 and 4 in its three runs), 0 failed, among them messages_read_back_as_written, a_tier_on_a_node_is_refused_by_name and a_channel_reads_whole_frames_and_its_ends
gate_qwen38_nodes on the Qwen3.8 fixture, with the new clause	PASSED; the tier's 12 rows hold 5,107,507,200 file bytes; node 1 on the copy refuses the plan's tier naming blk.0.ffn_down_exps.weight and its experts; no /dev/shm file and no copy left
gate_qwen38_nodes on UD-Q4_K_XL, with the new clause	PASSED; the tier's 144 rows hold 14,639,820,800 file bytes; node 1 on the copy refuses the plan's tier naming blk.41.ffn_down_exps.weight and its experts; no /dev/shm file and no copy left
FAIL-first: node 1 with the sha256 compare removed, the nodes gate on the fixture	red: the copy passes the open and is refused later by the tier's shape, so the clause fails
FAIL-first: the open's writer dropping each row's sha256, host::node::tests	red: messages_read_back_as_written failed

The new clause copies only the shard that holds the row it changes, the smallest such shard, under $BLOOMERY_DATA, with links to the other shards, and removes the copy after. On the real file that is at most the 12,087,983,520 B of its fourth shard. Main's a13125a8 changed crates/gpu/src/graph.rs (a branch forked again in one capture) beside this PR's lines there, so the runs at 6b16c8d2 are the first of this branch on top of it.

At 69a25bf6 on main 54ceee74, the head before the byte check:

At 69a25bf6	Result
The pure crates' tests (cargo test -p bloomery-levers -p bloomery-placement)	208 passed, 0 failed
gate-gpu-lib (the hw_held_ tests on one thread)	242 passed (1, 237 and 4 in its three runs), 0 failed
ptx-scan --no-jit of the scan set (gate_kquant, gate_swap, bloomery-serve-ds41) and gate_qwen38_twocard, against main 54ceee74	identical: 238, 235, 317 and 234 entries, every table row and md5 the same (tools/ref/ptx-canon.py --scans)
gate_qwen38_nodes on the Qwen3.8 fixture	PASSED in 44 s (five clauses, before the other-bytes clause)
gate_qwen38_nodes on Qwen3.8 UD-Q4_K_XL	PASSED in 227 s (five clauses)

The scan set covers every kernel carrier, so the branch as a whole moves no kernel. The refactor's own scan, against its base, is below.

The corpus the nodes gate read. just gate-gpu-qwen38-nodes reads qwen4exp/corpus-prose.ids, which is not on these machines, so every nodes gate run here used a stand-in: 4096 ids in [1000, 101000) from a fixed-seed generator. The gate reads its first 64, listed below so the run can be repeated. Ids that are not prose change which experts route, not what the clauses check.

The 64 ids the gate read

The file is one id a line; its first 64 ids, in order (sha256 of the whole 4096-id file eff777739fb152324bd267fe93c84bf3914591b3bde787441446012d1eb286ac):

62561 80444 81962 29753 73470 8947 100161 30809 61838 59142 57535 14690 50300 12153 11072 9569 96645 32426 74725 92053 47309 33988 49176 69019 77180 70953 60090 95028 37845 96249 17039 33043 76230 55925 24752 32915 61899 74270 89492 31988 48499 14947 55709 38118 52501 95911 25376 4808 75181 65979 9465 63380 61931 50054 7628 57184 22222 11767 41309 69676 66220 11217 20973 5844

FAIL-first of the unit tests. One mutant each, one edit to 69a25bf6, run with tools/gate.sh --oxide -p bloomery-gpu --release --lib -- --include-ignored host::node::wire host::node::tests host::tier::tests::a_tier_on_a_node. Each log shows bloomery-gpu compiled again, and in each the named test is the only one of the four that fails:

Mutant	Result
Channel::poll refuses a frame of exactly MAX_FRAME (len > MAX_FRAME as >=)	red: a_channel_reads_whole_frames_and_its_ends, "a poll: ... a frame of 67108864 bytes"
Channel::poll loses its Ok(0) if self.have == 0 arm, so a stream that ends between frames is an error	red: a_channel_reads_whole_frames_and_its_ends, "a poll: Kind(UnexpectedEof)"
A read overwrites the bytes the frame already has (self.have += n as self.have = n)	red: a_channel_reads_whole_frames_and_its_ends, the frame written in two parts never completes, "a frame: ... a frame's read passed its deadline"
TierOpen::open_here skips its node check	red: a_tier_on_a_node_is_refused_by_name, "a refusal" (the GB10 opened instead)
write_err sends a fault as a refusal (ERR_REFUSED)	red: messages_read_back_as_written, the fault "read back as Shape { ... }"
read_err drops the fault word it read (Fault::from_words(1, s))	red: messages_read_back_as_written, the fault read back as Fault { layer: 0, code: 1, sites: 24, ... }, not layer 5

FAIL-first of the gate's clauses. One mutant per key check, on the fixture, each one edit to 6fceb4de, an earlier head of this branch. The review fixes since then do not touch the lines these mutants edit:

Mutant	Result
Node 1 sends its census with 0 free bytes on its device	red: the plan refuses the load, "the device had 0 B free of its usable 130595946496 B at plan time"
Node 1 skips the experts' launches of a tier opened over the channel, its signals kept	red: FAIL node 0 and node 1: tokens apart at Some(0)
Node 1 does not release its go words when node 0 is lost	red: both lost node 0 clauses, "its stream drained false"; the stopped node 0's step names a stalled tier, not the release
Node 1 never checks for a stopped node 0	red: FAIL lost node 0 (stopped), the held step gave Ok(50324) and node 1 did not end
Node 0 never marks node 1 lost	red: the three lost node 1 clauses (the poison is Failed, want CardLost)
Node 0's settle ignores a node tier's released words	red: FAIL lost node 0 (stopped), the held step gave Ok(12457)
Node 1 checks only that a routed row has a set, not that its experts are the set	red: FAIL open, the swapped open gets past the set check and fails later in Weights::load_placed on a byte count instead of by name

How the lost node 0 clause holds node 1 mid-step. Holding node 0 with a host flag before a step does not reach node 1: each pass first copies its rows from pageable memory, and that copy blocks the host behind the held stream before the launch request goes out. So the gate relays node 0's channel through a TCP relay of its own. The relay holds the step's first launch frame, waits 500 ms, kills or stops node 0, then delivers the frame (and in the kill case closes the relay's side to node 1). Node 1 then waits on go words that do not come. The precondition line checks that node 1 had served fewer layers than were asked of it. The plan's budget search also requires a host expert on a layer before the last, because a stopped process's card keeps running and only a host leg makes it wait. In the stopped case the gate then continues node 0 and reads the held step's result.

The refactor alone (b570f82c against main 33fb317b). It is a split, so it must change no kernel and no result. Its measured commit b570f82c is the patch now at c9af1729 on a13125a8 (git range-diff shows them equal). Main changed none of the files it touches in the rebases since. These ran on b570f82c and its base:

• ptx-scan --no-jit of the scan set (tools/recipes.py scan-set: bloomery-serve-ds41, gate_kquant, gate_swap) and of gate_qwen38_twocard, on both commits: every entry's row and normalized md5 are identical (tools/ref/ptx-canon.py --scans: 238 rows and 239 md5s, and 315 and 315). The bloomery-gpu bundle is 231 bytes shorter; the comparison is per entry.

• Bits: Qwen3.8 UD-Q4_K_XL at plan (b′), its tier a second Gpu on the same GB10, in one process, the gate's history (98 tokens and logits rows). The tokens, every logit's bits, the stage graph nodes [1343, 1427, 1427, 1427], the tier graph nodes (433 for the step and each verify width) and the tier's feeds, captures and replays (4, 4, 82) are byte for byte the same file on both commits. The run used a small harness bin built from one source on both trees, not part of this PR.

• gate_deepseek41_tier on the V4.1 fixture, the landing clauses (--union --fault --two --batch --batch2 --bfault) and the lost clauses (--bfirst --blost --lost), at both commits: all PASSED, the stage graph 288 nodes, the tier's step and pair graphs 64 and 127. Two local edits, not in this PR, were needed on both: this machine had no DSpark draft fixture, so the plan takes a draft reserve of 0 B, and with that reserve the tier depth is capped at the real file's 23 experts a layer. The same clauses also passed at 6fceb4de, the whole branch before the review fixes, with the same two edits.

• gate-gpu-lib at b570f82c: 234 passed (1, 229 and 4 in its three runs), 0 failed.

• Decode timing, measured before the rebase on b87a62dc against fcdfd4a5 and not run again on main. The refactor's patch and the files it touches did not change in the rebase. One same-lease sitting of tools/ref/depth-qwen3moe.sh 6 bin:<fcdfd4a5 build>:6 6@BLOOMERY_AB_LOAD=arm (BLOOMERY_MODEL=qwen4exp, eight rounds, arm order rotated, BLOOMERY_DRAFT=off BLOOMERY_DECODE_N=160 BLOOMERY_GEN_WARM=16) under a noninf card (predict: -0.2..+0.2, margin 1 %). tools/ref/ab-decode.sh times bloomery-decode, the CPU engine, which links nothing this commit touches, so the GPU decode runner is the one that can see it. On one GB10 the serving plan (a) holds every routed expert on the card and no tier card, so this times the step's path through HostTier with no tier, which the split also touches; the in-process card tier needs two Gpus, and the runner refuses --place bp on one card. The card tier path adds one match and one Box deref a tier request against a step of about 32 ms, which is 0 at this ruler (derived, not measured).

  ratio d=6  ours/bin:base  mean 0.9993 ± 0.0026 (n=8)  per round: r1 1.0016 r2 0.9940 r3 1.0032 r4 0.9994 r5 0.9955 r6 0.9994 r7 1.0019 r8 0.9994
  ratio d=6  ours/ours@BLOOMERY_AB_LOAD=arm  mean 1.0004 ± 0.0019 (n=8)
  mean bin:base d=6  31.36 tok/s  31.45 tok/s(p50) (n=8)
  mean ours d=6      31.34 tok/s  31.44 tok/s(p50) (n=8)
  card: mean 0.9993 ± 0.0026 (n=8): effect -0.070 %, lower bound -0.334 %
  card: verdict: pass: the lower bound -0.334 % is above -1 %
  

  tok/s @ n=160, depth 6, GB10. No round had differing tokens, no row was [cpu-busy], and the lease saw no Xid. The bin: rows are all [cold] because that arm's page faults count over its whole process, its load included.

No run here sets the lever for GLM or V4.1; their refusal of a node tier is the open_here unit test above. open_here is the call that opens their tier card, so deleting the refusal at either call site no longer compiles; a call site rewritten to call Gpu::open_card directly would bypass it.

Not run here: gate-gpu-qwen38-twocard, gate-gpu-glm5next-twocard and gate-gpu-ds41-twocard, which need two cards, and stage-gpu-load-v41, which a change to crates/gpu/src/weights.rs runs by name and which stages plan (b) on two cards. The weights.rs change moves upload_segment's lookup of a segment's bytes into SegmentFile::of with the same checks and refusal texts; every load in the nodes gate goes through it (Weights::load_rows on node 1, Weights::load_placed in one process), and the ptx-scan --no-jit at 18ae4660 is above. gate_qwen38_twocard now includes the shared history and structure checks. It compiles, its ptx-scan --no-jit table is identical to the base's (above), and the history and the tier's graph and chain checks it runs are the code gate_qwen38_nodes ran in its one-process reference above. gate-gpu-qwen38-nodes itself has not run on an A6000. Its recipe names that card (BLOOMERY_CARD=a6000, tools/box.sh's lane on the maintainers' box) because both processes share one card and two 16 GiB budgets pass the 3090's 24 GB; in local mode the run takes the machine's own card.

On a Mac

At 18ae4660, with main 6f87bfb8 as the base: just mac-static, just mac-test, python3 tools/recipes.py check and the aarch64 cross check of the workspace (BLOOMERY_CHECK_TARGET=aarch64-unknown-linux-gnu tools/mac-check.sh check, the Spark's triple). The block is abridged: the eleven check-script lines are merged into one, and the timings and long tails are cut.

$ just mac-static 6f87bfb8
mac-static: fmt-check rc 0 (rustfmt diff blocks: 0)
mac-static: lint rc 0 (lint ^warning: 41 = the ratchet 41 in tools/lint-ratchet.txt)
mac-static: combos rc 0 (0 shapes run, 0 red, 49 skipped (scoped) ...; 42 green in the ledger ...)
mac-static: check-arch, check-comments, check-defaults, check-levers, check-loads, check-ptx-names,
            check-records, check-recipes, check-rustflags, check-unsafe, check-waits: rc 0 each
mac-static: ok
$ just mac-test
mac-check: test rc 101; 12 crates (crate=rc): ... bloomery-placement=0 bloomery-refset=101 ...; 1162 passed, 1 failed, 179 ignored in 43 test binaries
$ python3 tools/recipes.py check
check-recipes: tools/gate-paths.tsv: 120 gate rows held to the derivation, 23 rows with exceptions naming 264 gates
(rc 0)
$ BLOOMERY_CHECK_TARGET=aarch64-unknown-linux-gnu tools/mac-check.sh check
mac-check: check rc 0; ... 14 bloomery crates checked or compiled

The combos come from the ledger: they ran at 2cc0eeb9, the same four patches on main 15838a46 (git range-diff shows them equal), 42 of 42 green, and main's one commit since changes three Python tools, which no build shape reads. The one red is bloomery-refset's a_plain_target_with_no_draft_beside_is_the_family_path: the temporary directory comes back once with and once without macOS's /private prefix. This PR does not touch crates/refset, and the same test is red on the same Mac at main cb5688c9 alone.

Hardware (if the change is for a machine)

One DGX Spark a run (GB10, cc 12.1), Linux 6.17.0-1032-nvidia aarch64, host driver 595.84. The census total was 130,661,761,024 B in the runs at 18ae4660, 130,596,118,528 B in the runs at 69a25bf6 and 130,596,122,624 B in the earlier runs. The checks ran in a container built on nvidia/cuda:13.3.1-devel-ubuntu24.04, with rustc 1.100.0-nightly (e457a7b0d 2026-08-27) and just 1.58.0. Both processes ran on the one GB10. Apart from the decode A/B these are pass and fail results, not timings. The logs of the runs at 69a25bf6 did not keep the machine's nvidia-smi line; the runs at 6b16c8d2 and 18ae4660 recorded NVIDIA GB10, 12.1, 595.84.

AI help

AI 코딩 에이전트가 코드를 작성한 후 설명을 썼고, 테스트를 마쳤습니다. diff 리뷰와 PR 초안 리뷰는 제가 직접 했습니다.

[github-actions]

Verdict: MUST_FIX: one proven duplicate method in the new link code; no behavior failure path survived.

Must fix

1. Link::opened_end is a second copy of Link::stage_end

link.rs:132 and link.rs:139 have the same body: LinkEnd { dev: self.page.dev_at(0) }. opened_end has one caller, tier.rs:1185. If one body changes (for example a page offset) and the other does not, a tier node binds to a different address than the stage card.
Cost: now it is one deleted method; later the two drift unnoticed, because nothing compares them.
Fix: delete opened_end (link.rs:130-136) and call link.stage_end() at tier.rs:1185. Optionally rename stage_end to own_end (callers tier.rs:621, 723, 763, 1185) with the doc "The device address of the page's first byte in the context that made or opened it."

Non-blocking

⚠ Session::handle is 124 lines: node.rs:1086. R12 allows two screens (120 lines). The Open and Capture arms are the long ones. Move each to a fn returning Result<(), Failed>; handle then only dispatches. Not a kernel, so the split is allowed.

⚠ The channel framing has no test without a GPU: wire.rs:309. Channel::poll and recv carry the deadline rule. The mutants len > MAX_FRAME to >=, a dropped Ok(0) if self.have == 0 arm, and a frame split across two writes handled wrongly pass every unit test. A loopback TcpListener test in wire.rs needs no card.

⚠ The GLM and V4.1 refusal test does not protect the call sites: tier.rs:2246 calls refuse_node directly. Deleting t.refuse_node(WHAT)? at gpu-glm5next/src/tier.rs:362 or at gpu-deepseek41/src/body.rs:2629 keeps it green, and the family would then open its tier on a card of node 0.

⚠ NodeCensus::host_room and host_read (node.rs:89) have no consumer except a gate println (gate_qwen38_nodes.rs:531). The PR names a next row for them, so I do not block. If that row slips, remove the two fields.

⚠ just gate-gpu-qwen38-nodes names BLOOMERY_CARD=a6000 (justfile:1922) and never ran there, per the PR. Please run it once on that card or name the card it was proved on.

Checked

• Duplicates: about 40 new items checked. dryer link.rs:132~139 is finding 1; wire.rs:80~85 and lib.rs:221/254/262 share only a shape.
• Complexity: Session::handle (⚠ above); NodeOpen::read, NodeCensus::read, Session::run, Channel::poll are flat reads or state loops.
• Silent failure: 9 sites followed, including opened_progress().unwrap_or(0) at node.rs:1118 and :1184. None reaches a consumer as a wrong value.
• Tests, mutation: the tier-name, census and lever tests go red on their planted mutants. Gates: no pin moved. Escape catalog: no match.

Notes

• Evidence is from 6fceb4de, not the head dc22ecab; the PR says so.
• "Node 0 is any placed load" is wider than the code: only gpu-gates Place::on plans from node 1's census.

_{Card check: gpu-evidence. This pull request changes crates/app/src/arch/deepseek41/mod.rs, crates/gpu-deepseek41/src/body.rs, crates/gpu-gates/Cargo.toml … and carries a gate run on the author's card, which counts as verified.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[juliankang4]

All six points are addressed at 11b3445b. The branch is rebased onto fadec9d0 with no conflicts; main's commits since baeedd73 touch none of the tier code. It is now four commits: the TierWork refactor and the docs as before, the tier server with the fixes below folded in, and a rename on its own. The body's evidence is now from this head.

Must fix: Link::opened_end. Deleted. TierWork::bind_opened binds through the stage card's method. I also took the optional rename: stage_end is now own_end, "the end of the context that made or opened the page", since a tier node calls it too. The rename is its own commit (11b3445b) and changes nothing else; its four callers in tier.rs follow.

Session::handle. The open and capture arms are now Session::handle_open and Session::handle_capture, each returning Result<(), Failed>. handle only dispatches and is 87 lines. The code moved unchanged.

The channel framing. wire.rs has a loopback TcpListener test that needs no card, a_channel_reads_whole_frames_and_its_ends. A frame written in two parts reads back whole, and the poll between the parts is waiting. A frame already in is taken past its deadline, and with none the read is late. A stream that ends between frames is Closed, and one that ends inside a frame is an error. A length of MAX_FRAME waits for its bytes, and one past it is refused. FAIL-first on a DGX Spark, each mutant one edit to the head, tools/gate.sh --oxide -p bloomery-gpu --release --lib on the three tests the filters host::node::wire and host::tier::tests::a_tier_on_a_node select:

Mutant	Result
len > MAX_FRAME as >=	red: "a poll: ... a frame of 67108864 bytes"
the Ok(0) if self.have == 0 arm removed	red: "a poll: Kind(UnexpectedEof)" where the stream ended between frames
self.have += n as self.have = n, so a frame written in two parts never completes	red: "a frame: ... a frame's read passed its deadline"

In each the new channel test is the only one of the three that fails, and each log shows bloomery-gpu compiled again.

The GLM and V4.1 refusal. The check now lives in the call that opens the card. TierOpen::refuse_node became TierOpen::open_here(what), which refuses a tier on a node by name and otherwise opens the tier's card with Gpu::open_card. GLM's and V4.1's open_tier take their card from it, so deleting the refusal at either call site no longer compiles. The unit test calls open_here and goes red when its node check is skipped: the test then opened the GB10 and failed on "a refusal". A call site rewritten to call Gpu::open_card directly would still pass. I did not add a clause that sets the lever to the GLM and V4.1 gates for that, since it would be a new clause in each gate to hold one line of code.

host_room and host_read. Removed. The row that would read them (a pool for each node in the plan, dgx-spark.md 9.2) is not in this PR, so the census now carries the devices only. 9.2, 9.5 item 5 and section 10 say the host room joins the census with the plan term that reads it.

BLOOMERY_CARD=a6000. That value is the lane tools/box.sh picks on the maintainers' box, and it takes only 3090, a6000 or both; in local mode the run uses the machine's own card whatever the lane. The recipe needs the A6000 there because both processes share one card and two 16 GiB budgets pass the 3090's 24 GB. So I kept the value and put that reason in the recipe's comment. The gate was proved on a DGX Spark in local mode, which the body names, and has not run on an A6000.

Notes. "Node 0 is any placed load" is narrowed in the body: only a load placed through Place::on in crates/gpu-gates plans its tier card from node 1's census, and a load placed another way gets its tier opened on node 1 only if node 1's census lists the same device.

[github-actions]

Verdict: MUST_FIX: one proven duplicate between the two Qwen3.8 tier gates, which the PR body says it removed; the code of the change itself held up against every failure I built.

Must fix

1. The tier gates keep a second copy of the tier-layer and tier-structure checks

gate_qwen38_nodes.rs:270 is tier_layers, byte for byte gate_qwen38_twocard.rs:422. tier_structure (nodes:334-362) repeats the tier-graph node check and the fed, captured and replayed count of twocard:468-501. Both read WINDOWS, PROMPT and STEPS, which this PR already moved to shared/q38_history.rs. The PR body says "No second copy"; that holds for the history only.
Cost: when the history changes (a new window, a new chain), one copy is updated and the other passes on a wrong count, or fails with a text nobody can match to the clause.
Fix: move both into shared/q38_history.rs, delete the two local copies, and keep each gate's own extras (twocard's stage-node lines and its tier: stats line; nodes' resident_bytes precondition).

/// The tier layers: the plan's layers with a tier expert.
pub fn tier_layers(plan: &model::placement::Plan<'_>) -> Vec<usize> {
    plan.tier_n_l
        .first()
        .map(|t| (0..t.len()).filter(|&l| t[l] > 0).collect())
        .unwrap_or_default()
}

Then tier_graphs_and_chains(tier: &TierCard, layers: usize) -> bool holds the graph-nodes clause and the feed/capture/replay clause once, and both gates call it. Net result: about 45 lines fewer, no new name beyond these two.

Non-blocking

⚠ The error and fault reply has no test: node.rs:491 write_err and node.rs:504 read_err. Mutant: write_err maps GpuError::Fault to ERR_REFUSED, or read_err drops the fault words. No unit test or gate clause turns red. The PR body says "a fault on node 1 comes back to node 0 as a fault", and no run raises one. The Tag::Fault request (node.rs:761) is reached only through tier_fault (batch.rs:1364), after a prompt group that a node tier refuses at begin_group. Add a round-trip unit test for write_err and read_err on a Fault, a Protocol and a Shape error. Or say in the PR body that the fault path over a node is not covered yet.

⚠ The reply deadline is taken before the request runs: node.rs:989. After an open (up to minutes) or a drain (up to DRAIN_DEADLINE), the reply is sent under an expired deadline. It works today because a small frame goes out in the first write, and send checks the deadline only after a partial write. Compute the deadline after self.handle(..) returns.

Also 2 minor: the cuMemHostUnregister result in the error path of graph.rs:614 is dropped, while Drop records it. In messages_read_back_as_written, the mutant free_bytes > total_bytes as >= stays green, since no case has free equal to total.

Checked

• Duplicates: about 30 new items checked. Hits: tier_layers (exact) and the tier-structure block (finding 1). plan_under in the two gates differs in contract, so it is not a copy. Dryer pairs read (3): Out::u32/u64 and the GpuError constructors differ by width and variant, and merging them adds elements. No finding. The count check of nodes against tiers in Place::on_nodes and TierOpen::of_machine runs at two phases (before the plan and at the load), so both stay.
• Complexity: Session::handle 47 (a tag dispatch whose long arms are already handle_open and handle_capture), NodeOpen::read 37 (a straight field reader), Session::run 23. All are new functions that are complex on their own; I found no split worth its names. Gate bodies are not asked to split.
• Silent failure: 9 sites followed (unwrap_or(0) at node.rs:982, 1087 and 1132, the drop(Result) calls at node.rs:578 and 822 and link.rs:421, unwrap_or(u64::MAX) at node.rs:748, unwrap_or_default in fail_tier). Each is a message value or best-effort cleanup, and none reaches a consumer as data.
• Tests: the PR's FAIL-first table covers MAX_FRAME, the Ok(0) arm, have += n and open_here. My mutant on across_named (drop the prefix check) turns a_tier_node_opens_only_a_tier_pages_name red. Census version, flag, uuid-twice and shape mutants turn messages_read_back_as_written red.
• Gates: none moved. The unsafe ratchet rise has a dated PIN, and I count 5 new blocks in graph.rs, as claimed.
• Hot path: the in-process tier gets one match per request and one node() test per settle, which is constant. Node requests per layer exist only with the lever set. No performance finding.
• Escape catalog: no match. The gate signals pids it captured from its own children.

Notes

• The PR body has no literal nvidia-smi line. Its Hardware section names the GB10 and driver 595.84, with both commit hashes and the gate output.
• gate-gpu-qwen38-nodes has not run on the A6000 that its recipe names (stated in the PR body).

_{Card check: gpu-evidence. This pull request changes crates/app/src/arch/deepseek41/mod.rs, crates/gpu-deepseek41/src/body.rs, crates/gpu-gates/Cargo.toml … and carries a gate run on the author's card, which counts as verified.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[juliankang4]

The must-fix and the two non-blocking points are fixed at 69a25bf6; the two minor points stay as they are, for the reasons at the end. The branch is rebased onto main 84f77b01 (head 671bcc97; the runs below are at 69a25bf6 on 54ceee74, and that rebase changed only context lines of docs/contrib/dgx-spark.md and no file under crates/) and is still four commits: the fixes below are folded into the tier server commit, and the refactor and the rename are unchanged. Two files met main's changes. In tools/gate-paths.tsv the nodes recipe is in the same 47 rows as before, and in dgx-spark.md main's P4 row and its sentence on plan_in_pool in section 10 stand beside this PR's lines. The body's evidence is now from this head.

Must fix: tier_layers and the tier's structure checks. Both now live in shared/q38_history.rs, as tier_layers(plan) and tier_graphs_and_chains(tier: &TierCard, layers: usize) -> bool, and the two local copies are gone. TIER_LAYER_NODES moved there too, with its PIN line and its assert, since the shared check reads it and the nodes gate had a second definition of it. Each gate keeps what is its own: gate_qwen38_twocard its stage-node line, its tier: stats line and its precondition, and gate_qwen38_nodes its precondition on the tier's experts and resident_bytes. Both gates print the same two structure lines, one for the graphs and one for the chains, each with its own verdict, so a failure names the check. The chains line takes the two-card gate's wording, which states the counts it expects. The three files are 23 lines shorter (2933 to 2910).

The reply deadline. Node 1 now takes a reply's deadline when self.handle(..) returns, and the close's reply deadline after its drain. Node 0 still waits for each reply under the request's own deadline (for an open, the derived one), and the close's drain keeps the drain deadline. No test: Channel::send checks the deadline only while a frame is partly unwritten, so a red test needs a request whose work runs past REPLY (the go deadline, 10 s) and a reply larger than the socket buffer that node 0 does not read in time. The requests that run that long open or drain a card, so the test would need a card and over 10 s for a two-line ordering.

The error replies. messages_read_back_as_written now writes a Fault (layer 5, two sites), a Protocol and a Shape error with write_err and reads each back with read_err. Each must come back as its own kind, the fault with its layer, code and sites, and with the original text inside the reply. FAIL-first on a DGX Spark, each mutant one edit to the head, tools/gate.sh --oxide -p bloomery-gpu --release --lib on the four tests the filters host::node::wire, host::node::tests and host::tier::tests::a_tier_on_a_node select:

Mutant	Result
write_err sends a Fault as ERR_REFUSED	red: the fault "read back as Shape { ... }"
read_err drops the fault word it read (Fault::from_words(1, s))	red: the fault read back as Fault { layer: 0, code: 1, sites: 24, ... }, not layer 5

In each, messages_read_back_as_written is the only one of the four that fails, and each log shows bloomery-gpu compiled again. The four mutants of the last round (the channel's three and open_here's) were run again on this head with the same results.

Not changed. The cuMemHostUnregister result on shared_file's error path: it runs only after cuMemHostGetDevicePointer_v2 failed, the caller gets that error, and MappedHost's drop treats the same call the same way. A census with free bytes equal to total: that is a card with nothing allocated, a valid reading, and the census refuses only a free count above the total.

[github-actions]

Verdict: MUST_FIX: the node open accepts different model weights with matching tensor metadata.

Must fix

1. Check model bytes before opening the tier

crates/gpu/src/host/node.rs:277 checks shard, type, dimensions and byte count. Start tier_server -m with a model copy whose tier-resident expert weights differ while its headers stay identical: these checks pass. crates/gpu/src/host/node.rs:1243 loads that copy's weights. When those experts are routed, inference combines node 0's model with node 1's changed weights; finite results return without a model-mismatch error. This is a kind A failure path.
Adding identity to the new protocol now avoids diagnosing wrong outputs after deployment.

Reuse model::fileio::sha256_hex. Add this field to NodeOpen and this method to its existing impl:

pub file_hashes: Vec<String>,

pub fn file_hashes(file: &Split) -> Result<Vec<String>, GpuError> {
    (0..file.shard_count())
        .map(|s| {
            let g = file.shard(s).ok_or_else(|| {
                GpuError::shape(WHAT, format!("missing shard {s}"))
            })?;
            Ok(model::fileio::sha256_hex(g.mapping()))
        })
        .collect()
}

In tier38.rs's open initializer, add:

file_hashes: NodeOpen::file_hashes(file)?,

In swapped_open's initializer, add:

file_hashes: NodeOpen::file_hashes(&Split::open(ref_model_path()?)?)?,

In the message unit test's initializer, add file_hashes: vec!["fixture".into()], and assert back.file_hashes == open.file_hashes.

In NodeOpen::write, append this after the block sizes and before Ok(()):

o.len(self.file_hashes.len());
for hash in &self.file_hashes {
    o.str(hash);
}

In NodeOpen::read's initializer, append:

file_hashes: (0..i.len("shard count")?)
    .map(|_| i.str("shard sha256"))
    .collect::<Result<Vec<_>, _>>()?,

In Session::open, before describing tensors or opening a card, add:

if open.file_hashes != NodeOpen::file_hashes(self.file)? {
    return Err(GpuError::shape(
        SERVER,
        "node 0 and node 1 opened different model bytes",
    ));
}

Set the wire VERSION to 3. Extend the open clause with a model copy whose expert data changes while its headers stay identical; the open must fail.

Non-blocking

⚠ Measure the node launch wait: crates/gpu/src/host/node.rs:702 adds one synchronous TCP launch round trip per participating remote tier per replayed pass. A fed pass instead adds one enqueue round trip per served tier layer at node.rs:735. The reported A/B uses no tier and cannot measure this term. Supply a same-plan local-tier versus loopback-tier GPU decode A/B under the timing lease, with its prediction and output.

Checked

Model identity: traced the server's independent file selection, open validation and weight upload. None compares model contents across nodes.
Performance: traced replay/feed dispatch through TierNode::call's synchronous send/receive.
Tests: the open gate swaps expert IDs; it does not mutate weights while preserving headers. The proposed digest fix uses existing public APIs.
Scope: full review; both findings cite added lines. Neither matches a review exclusion or C1–C4; no ledger lessons are in force.

Notes

Static validation only; Rust and GPU gates were not run. The PR gives gate output and commit references but omits the required nvidia-smi line.

_{Card check: needs-gpu. This pull request changes crates/app/src/arch/deepseek41/mod.rs, crates/gpu-deepseek41/src/body.rs, crates/gpu-gates/Cargo.toml … and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[github-actions]

Verdict: MUST_FIX: write deadlines can be bypassed, the loss gate races its stderr reader, and the launch tag has two owners.

Must fix

1. Check the deadline after the final write

crates/gpu/src/host/node/wire.rs:299
A small frame sent with an expired deadline completes in one write. at == outbuf.len() skips the check and returns Ok, violating send's deadline contract. In TierNode::call:679, a pause after choosing the deadline permits a late request; recv also accepts a complete reply past that deadline, so the call can succeed without marking the node lost.
Fix the channel owner before more requests depend on its bound. Replace line 299:

            if Instant::now() > deadline {

Add this assertion inside the existing channel test, using its pair closure. The current code returns Ok, so unwrap_err fails:

        let (_w, mut ch) = pair();
        assert_eq!(
            ch.send(Tag::Close, &[], Instant::now() - Duration::from_secs(1))
                .unwrap_err()
                .kind(),
            ErrorKind::TimedOut
        );

2. Finish collecting stderr before returning the child's exit

crates/gpu-gates/src/bin/gate_qwen38_nodes.rs:421
Node 1 writes its loss line and exits while the detached reader has not appended it. wait_within returns the exit, then lost_node0:970 reads an empty or partial string. Its loss, release and progress checks fail despite correct server behavior. The mutex does not establish reader completion.
Fix the collection owner now to prevent intermittent gate failures.

Add this field to Proc:

        stderr_reader: Option<std::thread::JoinHandle<()>>,

Replace the stderr thread's opening line at 378:

            let stderr_reader = std::thread::spawn(move || {

Add this field to the Proc initializer:

                stderr_reader: Some(stderr_reader),

Replace the successful exit branch:

                if let Some(s) = self.child.try_wait()? {
                    if let Some(reader) = self.stderr_reader.take() {
                        reader.join().map_err(|_| {
                            format!("{}: stderr reader panicked", self.tag)
                        })?;
                    }
                    return Ok(Some(s));
                }

A regression that delays the reader before its append must still collect the complete loss line after the child exits.

3. Use the wire enum for the relay's launch tag

crates/gpu-gates/src/bin/gate_qwen38_nodes.rs:108 duplicates crates/gpu/src/host/node/wire.rs:30. Both declarations own the protocol value for Launch (R14).
Remove the second owner before a protocol change requires coordinated edits. Delete LAUNCH_TAG and its comment. Replace the enum declaration at wire.rs:26:

pub enum Tag {

Replace the relay condition at gate_qwen38_nodes.rs:906:

            if frame.first().copied()
                == Some(bloomery_gpu::host::node::wire::Tag::Launch as u8)
                && armed.swap(false, Ordering::SeqCst)
            {

Non-blocking

⚠ Attach current gate evidence: gate_qwen38_nodes.rs:1001 adds the other-bytes open clause. Head has six clauses; the pasted runs show five. Attach output covering this clause and the nvidia-smi card/driver line required by CONTRIBUTING.md.

Checked

Duplicates: confirmed the two Launch values; the proposed fix reuses the enum and removes the extra constant.
Complexity: no finding rests on a complexity score; tools were not rerun during validation.
Silent failure: followed the expired send through TierNode::call and its reply handling.
Tests: the existing channel test covers late buffered reads, but lacks the expired-send assertion. No tests executed during validation.
Gates: confirmed the stderr race reaches the loss gate's assertions and the added other-bytes clause is called.
Escapes: no catalog pattern applies to the retained findings.

_{Card check: needs-gpu. This pull request changes crates/app/src/arch/deepseek41/mod.rs, crates/gpu-deepseek41/src/body.rs, crates/gpu-gates/Cargo.toml … and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[github-actions]

The final successful write must also fail when its deadline has passed.

Suggested change

	if at < self.outbuf.len() && Instant::now() > deadline {
	if Instant::now() > deadline {

[juliankang4]

The must-fix is fixed with each tier row's bytes rather than each whole shard; the reasons and the cost are below. The fix ran on a DGX Spark at 6b16c8d2 (on main a13125a8), and the branch is now rebased over #172 at c891688f on main ad503b11, still four commits: the fix is folded into the tier server commit and its doc lines into the docs commit. The rebase moved this branch's page word access onto #172's Link methods and adds one refusal; the body's "After the rebase over #172" lists both, and the runs to repeat at the new head. In tools/unsafe-ratchet.txt main's PIN(2026-10-14) line for Branch::fork stands, and this PR's line reads gpu 1749 to 1754 for its five blocks; tools/check-unsafe.sh passes at the head.

Must fix: the model bytes in the open. Each NodeRow now carries sha256, the digest of the file bytes the tier holds for that row in node 0's file: the spans of its experts, in slot order, which are exactly the bytes node 1 uploads. Node 0 computes it in NodeOpen::rows_of, which now takes the Split. Node 1, after rows_on, hashes the same spans in its own file and refuses a row whose digest differs, before it opens its card: <tensor>: the experts [...] hold other bytes in this file (sha256 ...) than in node 0's (...): the two processes read other files. upload_segment's lookup of a segment's bytes (the tensor found and checked in the file, its spans, their slices) moved into SegmentFile::of in weights.rs, so the upload and the digest read the same bytes through one function. model::fileio::sha256_hex_of hashes several slices one after another, and sha256_hex calls it. The wire VERSION is 3.

Why the tier's rows and not the whole file:

Way	Same headers, other tier weights refused?	Bytes each node hashes per open, Qwen3.8 UD-Q4_K_XL [derived]
sha256 of every shard, on both nodes	yes	111,334,654,784 B, the whole file
refset's header digest	no, header bytes only
headpick's vocabulary digest	no, vocabulary only
a download's digest (LFS sha256)	yes, but no load reads it and a file made locally has none	the whole file
sha256 of the tier rows' bytes	yes, exactly the bytes node 1 uploads	the tier rows' file bytes: 14,639,820,800 B for this plan's 144 rows, 13 % of the file, as the open printed at 6b16c8d2

Each node hashes B bytes on one core, so the open takes B / R more on each node, R being one core's sha256 rate, which has not been measured on the GB10. The whole-file digest hashes 7.6 times the bytes [derived] and pulls the whole file through the page cache on every open. Bytes outside the tier's rows never reach node 1's computation, so they need no check. Node 1 reads its rows to upload them anyway, and node 0's open deadline (120 s plus the resident bytes at 100 MiB/s, 259.6 s for this plan [derived]) covers the digest and the upload together.

The gate clause. The open clause of gate_qwen38_nodes now also starts a second node 1 on a copy of the model whose headers are the file's and whose one tier row holds one other byte: the first byte of its first expert, flipped. The copy is only the shard that holds that row, the smallest such shard, under $BLOOMERY_DATA with links to the other shards, and it is removed after; on the real file that is at most the 12,087,983,520 B of the fourth shard. The plan's tier opened on it must be refused naming the row's other bytes. At 6b16c8d2 the clause passed on the fixture (12 rows, 5,107,507,200 B hashed) and on UD-Q4_K_XL (144 rows, 14,639,820,800 B), node 1 on the copy refusing the plan's tier by the row's tensor and experts, with no copy and no /dev/shm file left. FAIL-first: with node 1's compare removed the copy passes the open and is refused only later by the tier's shape, so the clause is red; with each row's sha256 dropped from the open's writer, messages_read_back_as_written is red.

The launch wait (non-blocking). Not measured. The body now gives the cost from the code, RTT being one request's round trip on the loopback channel:

• a replayed pass: one launch request per node tier with a layer in the pass, sent before the host serves the pass's first layer, so its service starts at most n_node_tiers × RTT later;
• a fed pass: one enqueue request per tier layer it serves and one capture request after it settles, at most (L_served + 1) × RTT;
• the gate's real-file history (82 replayed and 4 fed passes, the tier on 48 layers): at most 278 round trips [derived].

The decode A/B in the body ran plan (a) with no tier card, so it covers none of this. The same-plan A/B of a local tier against a node tier needs a second Gpu on one card, which only the gate's harness opens on one GB10; with the transport row the launch message can ride in the first image of the mirrored pages, and that path is the one to time.

Gates the head does not run. stage-gpu-load-v41, which a change to weights.rs runs by name, stages plan (b) on two cards and does not run on one DGX Spark. The weights.rs change is the same lookup moved into SegmentFile::of, with the same checks and refusal texts, and every load of the nodes gate goes through it.

The nvidia-smi line. The run at 6b16c8d2 recorded it first: NVIDIA GB10, 12.1, 595.84, CUDA Version 13.2. The body's hardware section quotes it.

[juliankang4]

All three must-fixes are fixed at 18ae4660 on main 6f87bfb8, folded into the tier server commit, so the branch is still four commits. The fixes and the rebase ran on a Mac and, at the new head, on a DGX Spark; the body's "What I ran" has the runs and both FAIL-first mutants.

1. The deadline after the last write. Channel::send now checks its deadline after every write, the last one included. A frame whose write starts past the deadline returns TimedOut even when one write sends it whole, and TierNode::call then marks the node lost. The bot's assertion is in a_channel_reads_whole_frames_and_its_ends, and the test's doc line says a send past its deadline is late. FAIL-first, with wire.rs built alone by rustc --test on a Mac (it uses only std): with the old condition the test fails with called Result::unwrap_err() on an Ok value, and with the new one both tests of the file pass. On the DGX Spark the same mutant of the head, run with gate-gpu-lib's command on host::node::wire::tests, fails that test alone with the same message. recv keeps the rule its doc states: a reply that is already in when node 0 runs again is taken. Node 1 sent that reply in time, so it is no sign of a lost node, and a request that leaves late no longer gets that far.

2. The stderr reader. Proc keeps the stderr reader's JoinHandle, and wait_within joins it before it returns the child's exit, so stderr() then holds everything the child wrote. FAIL-first on a Mac: Proc copied out of the gate unchanged, with a child that writes the two loss lines lost_node0 reads and exits. With a 300 ms sleep before each append, the old Proc read a stderr without those lines in 20 of 20 runs and the new one in 0 of 20. Without the sleep both read them in all 20 runs, so on this Mac the race shows only with a slow reader. On the DGX Spark the sleep went into the gate itself, on the fixture. With a 1000 ms sleep and no join, both lost node 0 clauses fail ("the loss named false", each precondition at "past 0 of the 0 layers") and the gate ends FAILED. With the same sleep and the join, the gate passes, and node 1's ends read 1.1 s and 11.3 s after node 0 was killed and stopped, against 0.3 s and 10.4 s with no sleep, because the join waits for the reader. A 200 ms sleep was not enough there: both runs passed, the one without the join too. The likely reason, not measured, is that node 1 exits more than 200 ms after its last write to stderr on that machine, so the reader is done first.

3. One owner of the launch tag. LAUNCH_TAG is gone. wire::Tag is public, and the relay compares a frame's first byte with Tag::Launch as u8.

The rebase. Main moved to 6f87bfb8. The one conflict was tools/gate-paths.tsv. Main's rows are kept, and gate-gpu-qwen38-nodes is in the same 46 rows and in the exceptions of the crates/model/src/arch/mimo2/** row, as before. Main's new crates/model/src/arch/nemotron_h/** row needed it too: recipes.py check found that the nodes gate reaches nemotron_h/hparams.rs through the arch dispatch. The gate is now in that row's exceptions next to gate-gpu-qwen38-twocard, for the row's stated reason (it links the arch dispatch and never opens a nemotron_h file).

Checks at 18ae4660. On a Mac: just mac-static (fmt-check, lint with 41 warnings against the ratchet's 41, 42 combos, and the eleven check scripts) passes, and so do tools/recipes.py check and cargo check --target aarch64-unknown-linux-gnu of the workspace. just mac-test has one red, bloomery-refset's temporary-path test, which the same Mac shows at main cb5688c9 alone. On the DGX Spark (NVIDIA GB10, 12.1, 595.84): ptx-scan --no-jit of six bins on both commits, moved=0 for every bin; gate-gpu-lib, 248 passed and 0 failed; gate_qwen38_nodes PASSED on the fixture and on UD-Q4_K_XL.

The gate evidence (non-blocking). The body now carries both gate_qwen38_nodes runs at 18ae4660, abridged, with all six clauses (clauses: 6 ran, the other-bytes open among them) and the nvidia-smi card and driver line. gate-gpu-ds41-tier and weekly-gpu-ds41-lost, which run the merged page code, end with exit 66 as shipped on that machine, on main and on the head alike, because it has no DSpark fixture draft. With the two local edits the body lists, the gate's landing and lost clauses passed on both commits, over the shared page and over loopback. Two things in that run went other than planned, and the body names both: the V4.1 runs needed a stand-in engram/corpus-prose.ids, put in place by hand after the first attempt stopped on the missing file, and they were built through tools/box.sh in local mode with another cargo oxide build than the other rows.

[github-actions]

Verdict: MUST_FIX: the node gate duplicates an existing sibling-binary helper.

Must fix

1. Reuse the sibling-binary helper (kind B)

crates/gpu-gates/src/bin/gate_qwen38_nodes.rs:471 repeats crates/gpu-gates/src/serve_client.rs:300: both resolve an executable with current_exe().with_file_name(). This is the same path rule with two owners (R14).
Consolidating now removes one function before another gate copies it.

Move the existing function into crates/gpu-gates/src/lib.rs, where PathBuf is already imported. This makes it available with gpu alone; serve_client is gated by deepseek41.

/// The binary `name` beside the running one.
pub fn beside(name: &str) -> Result<PathBuf, GateError> {
    Ok(std::env::current_exe()?.with_file_name(name))
}

Replace the function in serve_client.rs with:

pub use crate::beside;

Replace the gate's local function with:

use bloomery_gpu_gates::beside;

The two function bodies become one; seven code lines become five.

Non-blocking

⚠ Measure the node control overhead: crates/gpu/src/host/node.rs:723 adds one blocking launch request/reply per node tier with work in a replayed pass. A fed pass adds one enqueue request/reply per served tier layer and one capture request/reply per fed tier [derived]. The reported plan (a) A/B has no tier; the PR explains that the decode runner cannot run the needed placement. Provide a same-plan local-tier/node-tier A/B on the contributor's card, retaining the bit and graph checks.

Checked

Duplicates: both cited bodies implement the same path rule; the serve-client helper already exists in the base. The proposed move reuses that function in an existing module and removes the new copy.
Control path: traced replay, per-layer feed and capture through TierCard into TierNode::call, which sends a request and waits for its reply.
Gates: the tier node-count pin moved unchanged into the shared history. The unsafe ratchet increase has PIN(2026-10-14) and its reason. The new binaries declare required-features = ["gpu"].
Scope: full review; both findings concern added code. No ledger lessons are in force. Escape catalog C1–C4 does not apply to these findings.

Notes

Static review only; CUDA gates were not rerun here. The PR reports passing GPU runs through tools scripts, but supplies no passing just gate-* invocation with its output. Under the requested evidence rule, gpu.json records evidence=false.

_{Card check: needs-gpu. This pull request changes crates/app/src/arch/deepseek41/mod.rs, crates/gpu-deepseek41/src/body.rs, crates/gpu-gates/Cargo.toml … and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[juliankang4]

Closing this one: it has used the review bot's automatic reviews. The same branch, now at c9d0dfd8 with the fifth review's must-fix, continues in a new pull request, linked below.

[juliankang4]

Continued in #210.