[juliankang4]

Supersedes #210, which was closed while a fix to the lost node path and its run were still outstanding. Both are in this PR.

What and why

This is the tier server row of the P5 plan in #127 (docs/contrib/dgx-spark.md 9.5 items 5 and 6). Node 1 serves its share of the routed experts to node 0 over a control channel. The load plans its tier card from the census node 1 sends, and a lost node ends both sides by name. All of it runs over loopback, as two processes on one DGX Spark. The run on two Sparks needs the transport row of #127 and is not part of this PR. Part of #13.

Release notes: Added: tier_server serves the expert tier of a Qwen3.8 load to another process on the same host, which names it in BLOOMERY_TIER_NODES.

The branch is 13 commits on main 5caf434c, with no merge:

1. A split with no behavior change: TierCard keeps the stage side, and a tier card's own work on its card moves into TierWork.
2. The tier server, the channel, the load's plan from the census, and the ends of a lost node.
3. The docs.
4. A rename with no behavior change: Link::stage_end is Link::own_end, the end of the context that made or opened the page, since a tier node binds its page through it too.
5. The tier server refuses a chain word the tier cannot serve, and the nodes gate's cleanups return their error.
6. The tier page opens with O_NOFOLLOW, and the nodes gate checks that a refused symlink bind leaves its target unchanged.
7. The OpenExperts alias is pub(crate).
8. Removing node 0's saved run accepts a missing file, so a node 0 that failed early leaves the later clauses reachable.
9. When an open fails and removing its copy fails too, the gate reports both errors.
10. The message test reads back both accepted column bounds.
11. The nodes gate recipe runs the binary's unit tests before the card gate.
12. Node 1 refuses an open's layer count, set and column count before it opens a card, and the Qwen3.8 opener refuses a block route that overflows before it allocates it.
13. The gate's node children and both serve spawners set their parent-death signal through one function, die_with_parent, so killing the gate also kills a stopped node child.

How it works:

• Node 0 is a Qwen3.8 load whose plan has an expert tier, with BLOOMERY_TIER_NODES=<ip:port>[,...] naming the node of each tier in the plan's tier order. The lever takes loopback addresses only and refuses any other by name, because the page below is this host's memory and the channel has no authentication. Node 1 runs tier_server --listen <ip:port> [-m <first shard>], which refuses a listen address off loopback for the same reason. It serves one load and exits once node 0 closes it; a connection that opened no tier (a census read, a refused open) leaves it serving the next. With the lever unset, every tier is a card of the loading process, as before.
• The control channel (crates/gpu/src/host/node.rs, node/wire.rs) is TCP from std::net: one connection a load, one reply a request. A frame is a u32 length, a u8 tag and little-endian fields. The requests are open, bind, launch, capture, enqueue, sync, drain, clear fault, fault read and close. Every request gets Ok with its payload or Err with an error kind and the server's text, and a fault on node 1 comes back to node 0 as a fault, with its layer and sites. On node 0 each request and its reply run under one deadline. Node 1 takes its reply's deadline once the request's work returns, so an open or a drain that runs long still gets its reply. A send is late when its deadline has passed after any of its writes, the last one included, and a read once a 50 ms read slice that began past it finds no frame. The server keeps watching the tier's progress while a request is still arriving.
• On connect node 1 sends its census, each device's census row (9.2). A census of another version, with no device, a free count above the total or a device listed twice is refused by name. node_cards reads the nodes' censuses and makes each tier card with spec_of_device from the row its node sent. Place::on in crates/gpu-gates (the placement of the generate and serve bins and of the gates) calls it when the lever is set, so such a load plans (b′) with node 1's card as the tier card before it loads anything, and the gate plans through the same function. A load placed another way plans its tier card from its own census, and node 1 then opens it only if its census lists the same device (TierNode::open refuses another by name).
• For the open, node 0 sends the tier's rows of its plan (tensor, card format, expert ids, resident bytes, and the sha256 of the row's bytes in node 0's file), the tier's set and its shape. Before node 1 opens a card, NodeOpen::rows_on refuses an open whose layer count is not its file's, whose set does not fit the file, or whose column count is outside the step port's 1 to 8. It then refuses a routed row whose experts are not the tier's set for that layer, and a tensor that differs from node 0's description in shard, type, shape or size (Format::of_word reads the card format). Next it hashes each row's bytes in its own file and refuses a row whose sha256 is not node 0's, naming the tensor and its experts. Only then does node 1 open its card and load the rows from its own copy of the file with Weights::load_rows. The Qwen3.8 opener then refuses a block route whose slot count or byte count overflows, before it allocates the route.
• A row's sha256 covers the file bytes of the experts the tier holds, in slot order: exactly the bytes node 1 uploads. SegmentFile::of (weights.rs) finds a segment's spans in the file and borrows their bytes for the upload and for the digest alike, and model::fileio::sha256_hex_of hashes them one span after another. A copy of the model whose headers match node 0's file but whose tier experts differ is refused by name instead of serving other weights. Bytes outside the tier's rows do not reach node 1's computation, so they are not hashed.
• Node 1 checks each request before it reaches the tier's stream. A Capture or Launch frame whose chain word the tier cannot serve is refused as it is read (chain_of goes through chain_slot), and the connection stays open. A second bind is refused before the mapping changes, and a fault clear while work is in flight is refused. A request that fails after it reached the stream ends the connection as a lost node 0 does (below). No request waits on the stream without a bound.
• TierCard keeps the stage side: the books, the go order each graph was captured for, the page words it reads and the stage's stream. Its tier side is TierWork, a card of this process, or TierNode, a client of the channel. TierWork is the tier half that existed before (its Gpu, weights, the family's TierExperts, staging, graphs and its end of the page). The tier server holds one and calls the same methods. The replay or feed decision, the settle, the release and the reset stay on the stage side for both.
• Over loopback the tier page is one file under /dev/shm, opened with mode 0600 and O_NOFOLLOW, so a symlink under the page's name is refused. Node 0 makes it, maps it and registers it with its card (MappedHost::shared_file: cuMemHostRegister with PORTABLE | DEVICEMAP). Node 1 opens only a tier page's name there, then maps and registers the same file. Node 0 removes the file's name once every tier has bound the page (attach_tiers), and a page dropped before that, after a failed creation or bind, removes it too. A process killed after the binds therefore leaves nothing under /dev/shm. Node 0 killed between the file's creation and the last bind leaves the name behind, since node 1 never removes a page it opened. Both cards reach one physical page, so the page's protocol runs unchanged across the two processes. A shared page's link holds the counters its end releases: the stage side's link holds every crossing's, and a tier node's link holds its own tier's. With a tier on a node, BLOOMERY_TIER_LINK=loopback and BLOOMERY_TIER_LINK=rdma are refused by name, because both ends of either link are pages of the stage card's process. The in-process tier page is MappedHost::new (cuMemHostAlloc, not write-combined), so neither page is write-combined and the bits below compare like with like.
• When node 1 is lost, node 0 sees a request that fails, or gets no reply within its deadline (the go deadline for a step's requests). It marks the node lost (GpuError::Link) and shuts the channel down both ways (TierNode::lose_now, Channel::shutdown). The host tier then names the tier a lost card (CardLost), and the next step and a reset are refused as for a lost card of node 0's. A node that answers but stops signalling is still the go deadline's. A node 1 that was stopped past node 0's deadline and runs again reads the channel's end and ends as for a lost node 0 (below), while node 0 still holds the failed load.
• Node 1 ends on two conditions: the channel closes with no close request, or work asked of it makes no progress past the go deadline and its grace (node 0 stopped, its channel still open). Either way node 1 writes RELEASE to its own counters and go words, so every wait its stream holds passes. TierLayout::counters lists each row's count word before its go word, so a release that stops after any store has already marked that row lost before its go word lets the row run. The stage therefore cannot take a stale row's done as a normal one. It then drains its stream under the drain deadline and returns the loss by name. tier_server prints the loss once, as its exit error, and exits 1. The loss names how many of the layers asked of it it had served and how long node 0 sent no go. If the drain fails, the work is not dropped, so nothing frees memory the stream may still use.
• A stopped node 0 that runs again first takes the reply node 1 sent while it was stopped: a read is late only when a read that began after its deadline finds no frame. Its stage then finds node 1's counters released when it settles the step (TierCard::released). The settle refuses the step with the tier a lost card, naming the release, so no token comes from rows node 1 served after it.

What this PR does not do:

• plan_nodes (placement: a model's plan over several unified nodes (dgx-spark.md §9), and nodes_scale on one to four Sparks #95) is not called. It needs read rates, a NIC figure and per-node budgets that nothing measures on a live load yet. The census carries no host room: node 1's MemAvailable reading joins it with the plan term that reads it, once each node has a pool of its own in the plan (9.2).
• A remote tier serves no prompt batch. The batch port refuses a load with a tier on a node at its group start, by name.
• Only Qwen3.8 opens a tier on a node (NODE_FAMILY in arch/qwen3moe/tier38.rs). GLM and V4.1 open their tier cards through TierOpen::open_here, which refuses a tier on a node by name before any card opens.
• No tier node off this host. Besides the loopback-only lever, the channel has no keepalive. Over loopback a dead peer ends the connection at once, and every request has its deadline. The open's deadline is derived, not measured: 120 s plus the tier's resident bytes at 100 MiB/s.
• The RDMA link replaces the shared file with its mirrored pages once each process holds one end, and the launch message can then ride in the first image (9.5 item 5). That is the transport row.

Code shape

• No second copy. What both Qwen3.8 tier gates hold a load to is one file, crates/gpu-gates/src/bin/shared/q38_history.rs, which gate_qwen38_twocard and gate_qwen38_nodes include: the history (Run, step, window, history, same, WINDOWS), the plan's tier layers (tier_layers), the tier's graph and chain checks after the history (tier_graphs_and_chains) and the nodes a tier layer (TIER_LAYER_NODES). The two-card gate keeps only its residency count, its stage graph line, its tier: stats line and its precondition; the nodes gate keeps its precondition on the tier's experts and resident bytes. The parent-death setup has one owner, die_with_parent in crates/gpu-gates/src/lib.rs, called by the nodes gate's Proc::spawn, serve_client and gate_qwen38_serve. A tier node binds its page through Link::own_end, the stage card's method, and the counters a page's end releases come from one enumeration on the tier layout.

• Model code holds only architecture facts. Qwen3.8's code gains NODE_FAMILY (its tensor reader and tier computation, both existing functions, and the block route's overflow check) and stacks_of_set. The node path, the channel and the loss handling are common code.

• Dead code is gone.

• No silent failure. These are each refused by name:

  • a census of another protocol version, with no device, a free count above the total, or a device listed twice;
  • a flag byte other than 0 or 1;
  • a lever address off this host, a tier_server --listen address off loopback, or a lever that names a different number of nodes than the plan has tiers;
  • BLOOMERY_TIER_LINK=loopback or rdma with a tier on a node;
  • an open whose layer count is not the file's, whose set does not fit the file, whose column count is outside 1 to 8, or whose block route overflows;
  • an open whose routed rows are not the tier's set, a tensor that differs from node 0's description, or a row whose bytes' sha256 differs from node 0's;
  • a symlink under the tier page's name;
  • a second open or bind, a fault clear with work in flight, a close with a payload, a chain word the tier cannot serve;
  • a payload that ends early or runs past its fields, a frame over the size limit, an unknown tag, a format word no format has, a stream that ends inside a frame;
  • a node tier on a family without a node opener;
  • a released node tier at the settle;
  • in the fixture tier, a BLOOMERY_CARD_BUDGET that differs from the fixture header's budget (the nodes gate, as the other gates);
  • in the nodes gate, a cleanup that fails; a missing saved run is accepted, since a node 0 that failed early writes none.

  Unit tests hold the release order (a_tier_release_marks_stale_rows_before_the_stage_can_take_them in tier.rs stops the release after each store, the first included, and checks that no stale row is accepted), the census refusals, the error reply's kinds (a fault with its words, a protocol error, a refusal), the open's refused counts, and the chain words with both accepted column bounds and the refused ones (messages_read_back_as_written in node.rs). A loopback test in wire.rs holds the channel's frames and ends, and a unit test of TierOpen::open_here (tier.rs) holds the refusal of a node tier for GLM and V4.1. The nodes gate's own unit tests hold its plan retry, both cleanups and the parent-death setup. The six invalid opens, the open of other experts, the open on a copy with other tier bytes and the symlink bind are gate clauses.

• Comments state what is true now.

• No relaxed gate. No band, tolerance or lint moved. In tools/unsafe-ratchet.txt, gpu goes from 1844 to 1849 blocks for new code only, with a PIN(2026-10-16) line: the shared file's mapping, its registration, its device pointer, the unregister on that pointer's failure, and MappedHost's drop split into one block a release call. gpu-gates goes from 75 to 78 blocks (PIN(2026-10-15)): the one parent-death setup that replaces two copies has three blocks (prctl, getppid and pre_exec), and its before-registration test adds two more (raise and pre_exec) for new code.

• cargo fmt and cargo clippy are clean on what I touched, and the lint count did not go up: just mac-static counts ^warning: 41 = the ratchet 41 on this branch and on main fc47ee62 alike.

Duplication found or removed (path:line), or "none": the Qwen3.8 history, the tier layers, the tier's graph and chain checks and TIER_LAYER_NODES, now in crates/gpu-gates/src/bin/shared/q38_history.rs, are removed from gate_qwen38_twocard.rs and have no copy in gate_qwen38_nodes.rs. gate_qwen38_residency.rs and the GLM and V4.1 twocard gates keep their own histories, which differ in what they step and compare. The two PR_SET_PDEATHSIG setups in serve_client.rs and gate_qwen38_serve.rs are now die_with_parent (crates/gpu-gates/src/lib.rs:87). The nodes gate starts tier_server through beside, which moves from serve_client to the crate root (crates/gpu-gates/src/lib.rs:177); serve_client re-exports it.

What I ran

This PR's code was checked on main fc47ee62 (unit tests, mutants, the nodes gates, contrib-check, mac-static) and on main 19114825 (the move proof and the affected gates). The code is the same on both; upstream changed justfile comments, one recipe for another model and a paragraph of docs/contrib/dgx-spark.md between them, and none of that reaches the tier server, the node wire or the nodes gate. The rebase onto 60065e57 changed only the generated tables, tools/gate-paths.tsv and tools/unsafe-ratchet.txt, so the results below carry across it. Upstream's commits in that range touch none of this PR's files. The same holds for the rebase onto 5caf434c: it changed only rows of tools/gate-paths.tsv, upstream's commits there add the Bolt fixture and the Nemotron logits reader, and contrib-check --tier t0 and mac-static ran again on this head. WideActs now quantizes for the qwen3moe units the nodes gates drive, so both nodes gates ran again on ec7015ee, the branch before its rebase onto 5caf434c: with BLOOMERY_TIER=fixture they print 5 passed; 0 failed, the same 98 tokens and 98 logits rows bit for bit, PASSED: gate_qwen38_nodes and gate_qwen38_nodes ran 166 s (exit 0), and with BLOOMERY_TIER=real the same lines with gate_qwen38_nodes ran 628 s (exit 0).

Cards and drivers: a DGX Spark reports NVIDIA GB10, 595.84, and the RTX 5090 machine reports NVIDIA GeForce RTX 5090, 595.99.02. Both checks ran with no other process on the card. The model for the real tier runs is Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf.

Check	Result
BLOOMERY_BOX_LOCAL=1 just canon-check 19114825 on a DGX Spark: both sides' --no-jit scans of the bins the scan set picks (bloomery-serve-ds41, gate_kquant, gate_swap), compared entry by entry	canon-check: 3 bins, 826 entries: 826 identical, 0 reordered-only, 0 differs, 0 only-on-one-side; rows equal (jit columns not compared: no-jit both). This is the move proof for the TierWork split: no PTX entry changes across the branch.
Every gate just affected selects against 19114825, with BLOOMERY_BOX_LOCAL=1 and BLOOMERY_TIER=fixture, on a DGX Spark, with the base run beside it	gates-affected: 182 gates, ok 34, same as base 147, differs 1, unproven 0. The one differing gate, gate-ds41-load, is red the same way on the base ("What I ran into").
BLOOMERY_BOX_LOCAL=1 BLOOMERY_TIER=fixture BLOOMERY_GATE_BOUND=3600 just gate-gpu-qwen38-nodes on a DGX Spark	the binary's unit tests 5 passed; 0 failed, then ok node 0 and node 1: 98 tokens, 98 logits rows bit for bit the reference's, clauses: 8 ran, 0 left to the real tier, PASSED: gate_qwen38_nodes, gate_qwen38_nodes ran 171 s (exit 0)
The same with BLOOMERY_TIER=real on the Qwen3.8 UD-Q4_K_XL file	the same 98 tokens and 98 logits rows bit for bit, clauses: 8 ran, 0 left to the real tier, PASSED: gate_qwen38_nodes, gate_qwen38_nodes ran 776 s (exit 0)
BLOOMERY_BOX_LOCAL=1 bash tools/gate.sh --oxide -p bloomery-gpu --release --lib -- host::node::tests, then the same for -p bloomery-gpu-gates --features gpu --release --bin gate_qwen38_nodes -- gate::tests, on a DGX Spark	test result: ok. 1 passed; 0 failed and test result: ok. 5 passed; 0 failed
FAIL-first on a DGX Spark: nine mutants, each run against the test or clause that should catch it, then the head restored	each mutant red: without O_NOFOLLOW the gate's bind clause fails (FAIL bind: symlink refused false); with the accepted columns narrowed at either end (3..= or 2..) messages_read_back_as_written fails; with the missing-file case dropped from remove_run, with the open's error dropped from remove_open_copy, or with the DeviceTwice case dropped from the gate's budget retry, the matching gate unit test fails; without the parent-process check in die_with_parent the before-registration parent-death test fails (the server executed or kept stdout open); without the capture column-count check messages_read_back_as_written fails; with Unused removed from Format::ALL the independent format test fails. The restored head passes all three test sets
LC_ALL=C BLOOMERY_ARCH=sm_121 just contrib-check --tier t0 on a DGX Spark	DONE tier=t0 total=84 red=1 wall=322s red: check-recipes. The one red case is --v41-queue runs under BLOOMERY_BOX_READONLY=1, which fails the same way at main fc47ee62 on the same machine (DONE tier=t0 total=84 red=1 wall=318s, "What I ran into")
LC_ALL=C BLOOMERY_ARCH=sm_121 BLOOMERY_BOX_LOCAL=1 bash tools/gate.sh --oxide -p bloomery-gpu --release --lib -- host::tier::tests host::node::tests, then just gate-gpu-lib, on a DGX Spark at ec7015ee	test result: ok. 9 passed; 0 failed; 1 ignored (the ignored test needs a CUDA device and runs in the second command), then test result: ok. 261 passed; 0 failed
FAIL-first for the release order on a DGX Spark: swap [TWord::Cnt(r), TWord::Go(r)] back to [TWord::Go(r), TWord::Cnt(r)] in TierLayout::counters, run a_tier_release_marks_stale_rows_before_the_stage_can_take_them with --exact, restore	the mutant fails with tier 0 row 0: stale rows accepted after release at 0 (FAILFIRST mutant rc=101 restored rc=0), and the restored head passes
LC_ALL=C BLOOMERY_ARCH=sm_120 just contrib-check --tier t1 on an x86_64 Linux machine with an RTX 5090	DONE tier=t1 total=104 red=1 wall=316s red: check-recipes. The two red cases are stall-term and --v41-queue, and both fail the same way at main fc47ee62 on the same machine (DONE tier=t1 total=104 red=1 wall=321s, "What I ran into")
./tools/mac-static.sh fc47ee62 on a Mac	lint passes at ^warning: 41 = the ratchet 41. The one red step is check-recipes (rc 1); main fc47ee62 on the same Mac gives the same red steps: check-recipes (rc 1).

The nodes gate reads qwen4exp/corpus-prose.ids, which is not on this machine, so both nodes gate runs used a stand-in: 4096 ids in [1000, 101000) from a fixed-seed generator. The gate reads its first 64, listed below so the run can be repeated. Ids that are not prose change which experts route, not what the clauses check.

The 64 ids the gate read

The file is one id a line; its first 64 ids, in order (sha256 of the whole 4096-id file eff777739fb152324bd267fe93c84bf3914591b3bde787441446012d1eb286ac):

62561 80444 81962 29753 73470 8947 100161 30809 61838 59142 57535 14690 50300 12153 11072 9569 96645 32426 74725 92053 47309 33988 49176 69019 77180 70953 60090 95028 37845 96249 17039 33043 76230 55925 24752 32915 61899 74270 89492 31988 48499 14947 55709 38118 52501 95911 25376 4808 75181 65979 9465 63380 61931 50054 7628 57184 22222 11767 41309 69676 66220 11217 20973 5844

Decode A/B, same lease

The default path, with no tier on a node, must not get slower. just ab-decode cannot answer that for this model: bloomery-decode takes only deepseek2 and stops on qwen4exp before the first arm. depth-serve.sh's local mode is the runner for it. It starts a fresh bloomery-serve for every row, and the arms take turns as the first of each of 6 rounds.

The default path does not get slower here: on one DGX Spark, head b08d2a08 decodes 1.89% faster than base 60065e57, and the 95% interval (± 4.97%) holds 0. The prompt pass is also within noise (-2.08% ± 9.48%, the interval holds 0). Between b08d2a08 and 95dc2b8f only the node-loss release order (tier.rs) and the rebase onto 5caf434c changed, and a decode never calls the release, so the numbers stand for this head too. The third arm sets BLOOMERY_PIN_MAIN=1 on the head, so its distance to the head is the run's own spread.

Arm (qwen4exp, initial depth 5, 96 tokens, 6 rounds)	decode tok/s	decode sd	prompt tok/s
base 60065e57	28.93	5.39%	31.89
head b08d2a08	29.47	0.90%	31.23
head b08d2a08 with BLOOMERY_PIN_MAIN=1	29.43	0.51%	31.68

The head with BLOOMERY_PIN_MAIN=1 against base reads +1.76% ± 4.93% for decode. The run's witness lines show no NVRM Xid line, a clock event reason of 0 after the last round and cpu-busy 0/6 on every arm. To repeat it on a DGX Spark checkout with both builds of bloomery-serve in place, with the first shard of the Qwen3.8 UD-Q4_K_XL file as the model:

BLOOMERY_BOX_LOCAL=1 BLOOMERY_ARCH=sm_121 BLOOMERY_MODEL=qwen4exp BLOOMERY_AB_ROUNDS=6 \
BLOOMERY_REF_MODEL=<first shard of the UD-Q4_K_XL file> \
BLOOMERY_BOX_ENV="DEPTH_SERVE_N=96 DEPTH_SERVE_IDS=<file of 5 token ids>" \
./tools/box.sh 'bash tools/ref/depth-serve.sh base-60065e57:5 b08d2a08:5 env=BLOOMERY_PIN_MAIN=1:b08d2a08:5'

Tier on a node versus in this process, same plan

A tier on a node costs decode speed against the same tier in the loading process: 12.53% lower decode throughput (95% interval -17.60% to -7.46%) and 14.59% longer steps (+8.37% to +20.82%), over 6 pairs on the same GB10 under the same lease, with the pairs taking turns as first. Every pair has the same plan and the same tokens. The default path keeps its tier in this process and does not pay this cost (the A/B above).

The plan is the same in both arms: 16 GiB, context 3072, ubatch 3072, with 1,962 experts on the stage, 4,671 in the tier and 17,943 on the host. The local arm decodes at 10.35 tok/s and 96.59 ms a step in the preparation pair, the node arm at 8.64 tok/s and 115.72 ms. The runs were on ec7015ee, the branch before its rebase onto 5caf434c.

The existing CLI cannot make this pair. It cannot put a local stage and a local tier on one GB10 (the generate placement place38 asks for two cards), and the nodes gate prints no times. A small harness times it instead. It calls the public functions the gate calls and includes q38_history.rs as it is. The harness is not part of this PR.

Two limits on the number. Both arms use the one GPU, and in the node arm a second process with its own CUDA context shares that GPU. The figure therefore holds the loopback TCP round trips and the cost of two processes sharing one GPU together, and I did not separate them. It is not a figure for two physical nodes. The transport row is where a Launch message moves into the first image (9.5 item 5).

Not run here

No run here sets the lever for GLM or V4.1; their refusal of a node tier is the open_here unit test. open_here is the call that opens their tier card, so deleting the refusal at either call site no longer compiles; a call site rewritten to call Gpu::open_card directly would bypass it.

gate-gpu-qwen38-twocard, gate-gpu-glm5next-twocard and gate-gpu-ds41-twocard need two cards and did not run to a result: in the affected-gates run each exits 64 on this head and on the base alike. gate_qwen38_twocard now includes the shared history and structure checks. It compiles, and the history and the tier's graph and chain checks it runs are the code gate_qwen38_nodes ran in its one-process reference above. gate-gpu-qwen38-nodes itself has not run on an A6000. Its recipe names that card (BLOOMERY_CARD=a6000, tools/box.sh's lane on the maintainers' box) because both processes share one card and two 16 GiB budgets are more than a 3090 holds; in local mode the run takes the machine's own card.

What I ran into

None of this blocks the PR.

contrib-check has a red check-recipes at main on each machine. On the DGX Spark, t0 on this branch and at main fc47ee62 both fail --v41-queue runs under BLOOMERY_BOX_READONLY=1 and syncs nothing: rc 1 (want 0). On the RTX 5090 machine, t1 on this branch and at main fc47ee62 both fail that case and stall-term: FAIL stall-term: got [post 0 rc=124], want [post 0 rc=143]. On the Mac, mac-static has one red step, check-recipes, on this branch and at main fc47ee62 alike. Reproduce with LC_ALL=C ./tools/contrib-check.sh --only '^check-recipes$'.

gate-ds41-load fails its check v on this branch and on the base, so the affected-gates run lists it as the one differing gate. The check compares GB10's free memory before and after two loads and drops, and that count moves by hundreds of MB between reads. Two runs each side, in the same environment, all fail the same way: after the first drop the head leaves -66 and -74 MB and the base -133 and -47 MB, and the second load takes 70 and 72 MB more than the first on the head and 160 and 2 MB more on the base. Check vi passes in all four runs. The first run differed from the base only in the sign of one normalized number, "took -N" against "took N".

The tier page trusts its name under /dev/shm. Node 1 opens a page only by a tier page's name and refuses a symlink there, but it does not check the file's owner. The channel has no authentication either. Both nodes run on one host as one user by design here, and that check belongs with the transport row, which replaces the shared file.

Hardware (if the change is for a machine)

One DGX Spark (GB10, cc 12.1), host driver 595.84, census total 130,596,122,624 B. The checks ran in the dev container (CUDA Version 13.3.1). Both processes ran on the one GB10. contrib-check t1 ran on an x86_64 Linux machine with an RTX 5090 (driver 595.99.02), and just mac-static on a Mac (just 1.58.0). These are pass and fail results, not timings, apart from the two timing sections above.

AI help

AI 코딩 에이전트가 코드를 작성한 후 설명을 썼고, 테스트를 마쳤습니다. diff 리뷰와 PR 초안 리뷰는 제가 직접 했습니다.

[github-actions]

Verdict: MUST_FIX: an invalid capture request terminates an opened load, cleanup errors are hidden, and one public alias is unnecessary.

Must fix

1. Reject unsupported wire chains before calling the tier

crates/gpu/src/host/node.rs:497 accepts word 16 as Cols(0). On an opened session, a Capture frame with that word and an empty service list reaches TierWork::capture. Its first operation, chain_slot, rejects the width before touching the stream. node.rs:1232 converts that error to Fatal; the server releases the tier and exits instead of refusing the request and keeping the session.
Cost: validating the decoder now prevents every request handler from repeating this error classification.
Replace chain_of with:

fn chain_of(w: u32) -> Result<Chain, String> {
    let chain = match w {
        0 => Chain::Step,
        1 => Chain::Pair,
        2 => Chain::Quad,
        COLS_WORD.. => Chain::Cols((w - COLS_WORD) as usize),
        _ => return Err(format!("a chain word {w}")),
    };
    chain_slot(chain).map_err(|e| e.to_string())?;
    Ok(chain)
}

Add these assertions to messages_read_back_as_written; dropping the decoder validation must turn them red:

for c in [Chain::Cols(0), Chain::Cols(1),
          Chain::Cols(model::ops::DEFER_MAX_COLS + 1)] {
    assert!(chain_of(chain_word(c).unwrap()).is_err());
}

2. Report failed cleanup before the gate returns success

crates/gpu-gates/src/bin/gate_qwen38_nodes.rs:716 discards deletion errors for the mutated model copy. crates/gpu-gates/src/bin/gate_qwen38_nodes.rs:1106 does the same for the saved run. If deletion fails after successful checks, the gate reaches PASSED while leaving those files behind.
Cost: handling the errors now prevents repeated runs from retaining model shards without a diagnostic.
Replace the two cleanup statements, respectively:

std::fs::remove_dir_all(&dir)
    .map_err(|e| format!("remove {}: {e}", dir.display()))?;

std::fs::remove_file(&out)
    .map_err(|e| format!("remove {}: {e}", out.display()))?;

3. Inline the single-use opener alias

crates/gpu/src/host/node.rs:868 adds OpenExperts solely for NodeFamily::experts. Repository-wide Grep finds no other use. The field already owns this function signature; the alias adds an unnecessary public name (fewest elements; R13).
Cost: removing it now avoids carrying a single-use public name into later changes.
Replace NodeFamily and the alias below it with:

pub struct NodeFamily {
    pub word: &'static str,
    pub tensors: fn(&Split) -> Result<ModelTensors, GpuError>,
    pub experts:
        fn(&Gpu, &ModelTensors, &Weights, &NodeOpen)
            -> Result<Box<dyn TierExperts>, GpuError>,
}

Non-blocking

⚠ Supply the dispatch and transport proof: crates/gpu/src/host/node.rs:723 adds one blocking control round trip per participating tier per replayed chain (one token for Step), with at least three socket I/O calls on each side. Provide a timed node/in-process A/B on the contributor's card. The TierWork extraction is the move/split class; provide its identical base/head just ptx-scan --no-jit tables too.

Checked

Elements: repository search confirms OpenExperts has one use, in NodeFamily::experts.
Complexity: no complexity-only finding retained.
Silent failure: both discarded cleanup results were followed to the gate's successful verdict.
Tests: the decoder test rejects word 3 but does not check unsupported column widths. The proposed assertions fail on the current decoder and pass with its validation.
Gates: no gate/PIN finding retained.
Escapes: C1–C4 do not apply to the retained findings.

Notes

No build or GPU gate was run in this review. The PR reports GB10 gate runs and FAIL-first for stopped node 1; it supplies no PTX comparison or transport A/B.

_{Card check: gpu-evidence. This pull request changes crates/app/src/arch/deepseek41/mod.rs, crates/gpu-deepseek41/src/body.rs, crates/gpu-gates/Cargo.toml … and carries a gate run on the author's card, which counts as verified.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[github-actions]

Propagate cleanup errors so this gate cannot report success after deletion fails.

Suggested change

	drop(std::fs::remove_dir_all(&dir));
	std::fs::remove_dir_all(&dir)
	.map_err(|e| format!("remove {}: {e}", dir.display()))?;

[github-actions]

Propagate cleanup errors so this gate cannot report success after deletion fails.

Suggested change

	drop(std::fs::remove_file(&out));
	std::fs::remove_file(&out)
	.map_err(|e| format!("remove {}: {e}", out.display()))?;

[github-actions]

Verdict: MUST_FIX: the lost-node release order can let node 0 accept a token computed from stale images.

Must fix

1. Publish the loss marker before releasing go waits

crates/gpu/src/host/tier.rs:539 enumerates Go before Cnt, and Link::release stores them in that order. Suspend node 0 after its captured replay receives the Launch reply, before it serves the host layers. When node 1's progress deadline expires, it writes Go=RELEASE first. If node 1 is preempted before its Cnt store, the tier GPU can finish the remaining layers from stale images and advance Prog. Resumed node 0 consumes ordinary done counts without another channel request. Its settle checks only Cnt for the loss marker (crates/gpu/src/host/tier.rs:1972), so the step can return a corrupted token.

Fixing this order before the transport ships avoids debugging silent token corruption later. Enumerate Cnt first; update the adjacent comment to say each row's go is paired with its done.

fn counters(self, tier: usize) -> impl Iterator<Item = usize> {
    (0..self.rows)
        .flat_map(|r| [TWord::Cnt(r), TWord::Go(r)])
        .map(move |w| self.word_off(tier, w))
}

The stopped-node clause at crates/gpu-gates/src/bin/gate_qwen38_nodes.rs:1157 waits for node 1 to exit before resuming node 0, so it passes on this bad order. Add an interleaving case that pauses the release after its first store and resumes node 0; that step must fail.

Non-blocking

⚠ Control-channel timing: crates/gpu/src/host/node.rs:747 adds one synchronous TCP round trip per remote tier per one-token replay: 9 request bytes and 5 reply bytes, excluding TCP/IP [derived from the frame layout]. The PR leaves this cost unmeasured. Attach the same-plan local-tier versus node-tier timing on the contributor's card, with the evidence required by CONTRIBUTING.md.

Checked

Duplicates: page and node releases reuse TierLayout::counters; the proposed fix changes that existing owner.

Complexity: diagnostics flag request dispatch and gate workflows; complexity alone supplies no finding.

Silent failure: traced release stores, tier progress, stage done waits and the loss-marker check through token retrieval.

Tests: lost_node0 resumes node 0 only after node 1 exits, so it misses the release-order interleaving. No production tests were run here.

Gates: the changed gpu and gpu-gates unsafe ratchets have dated PIN reasons.

Escape catalog: C1–C4 do not apply to either retained finding. No ledger lessons are in force.

Notes

Dryer did not scan crates/gpu or crates/gpu-gates. The PR includes card gate output and identical canonical PTX evidence; its decode timing remains unfinished.

_{Card check: gpu-evidence. This pull request changes crates/app/src/arch/deepseek41/mod.rs, crates/gpu-deepseek41/src/body.rs, crates/gpu-gates/Cargo.toml … and carries a gate run on the author's card, which counts as verified.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[github-actions]

Publish the loss marker before waking the tier GPU, so a paused releaser cannot let node 0 accept stale rows.

Suggested change

	.flat_map(|r| [TWord::Go(r), TWord::Cnt(r)])
	.flat_map(|r| [TWord::Cnt(r), TWord::Go(r)])

[github-actions]

Keep the counter documentation consistent with publishing Cnt before Go.

Suggested change

	/// Byte offset of every counter of tier `tier`, each row's go and then
	/// Byte offset of every counter of tier `tier`, each row's go paired with

[github-actions]

Verdict: CLEAN: No reachable behavior defect or proven quality defect survived review.

Non-blocking

⚠ Include the node/local timing witness: crates/gpu/src/host/node.rs:744 adds one synchronous TCP request/reply for each nonempty remote tier graph replay during decode. The reported node/local A/B uses an unpublished harness and omits its invocation and output. Attach those with the harness so this dispatch cost can be reproduced.

Checked

Duplicates: 250 added named declarations checked, including moved definitions; 218 distinct names searched. No actionable duplicate found. The history, parent-death setup, segment bytes and hashes reuse common owners. Dryer reported zero pairs; its excluded GPU scopes were searched manually.

Complexity: Session::{handle,run,handle_capture}, NodeOpen::read, NodeCensus::read and Channel::poll are new protocol parsing/dispatch functions. The new symlink_bind, gate, other_bytes_open, lost_node0, stopped_node1, tier_server::run and history functions coordinate gate stages. Split boundaries are request decoding/open/capture and census/open/bind/bits/loss. HostTier::attach_tiers adds transport selection; Kind::parse adds one address-list arm.

Silent failure: 36 error/default candidate sites followed. Invalid protocol fields return named errors; failed gate reads or parses fail their checks. Optional values, diagnostics and best-effort teardown do not replace invalid computation with a token.

Tests: Static mutation checks cover all 12 new tests and the changed lever cases. Removing NodeOpen's layer/column checks, In::take's payload bound, Channel::poll's frame bound, open_here's node refusal or across_named's path restriction breaks assertions. Swapping TierLayout::counters' release order breaks its stale-row test. Removing die_with_parent's signal or parent-PID check breaks the corresponding child test. Dropping plan_refusal's DeviceTwice retry, remove_run's missing-file case or remove_open_copy's retained error breaks its cleanup/retry test. Omitting Format::Unused or accepting nonloopback lever addresses breaks the format/registry tests. Token or logit changes fail the shared history comparison.

Gates: No numerical band or case removed or relaxed. The shared history preserves the existing checks and PIN. Both unsafe-count increases carry dated reasons for new code. New node clauses have failure evidence in the PR.

Adversarial paths: stopped node 0 → released counters → settle refuses stale rows; lost node 1 → failed request or progress deadline → CardLost and reset refusal; mismatched model or allocation bounds → named open refusal. Additional clean-result check: a connection ending inside a frame → protocol error and release/drain.

Escape catalog: C1–C4 do not match. Child signals use recorded PIDs.

Notes

Static review only. No build or GPU test was executed here. Card runs and executed mutants are supplied by the PR.

_{Card check: gpu-evidence. This pull request changes crates/app/src/arch/deepseek41/mod.rs, crates/gpu-deepseek41/src/body.rs, crates/gpu-gates/Cargo.toml … and carries a gate run on the author's card, which counts as verified.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[juliankang4]

The review is done and the contribution check passed, so I'm merging this.