Repository navigation
app, gpu, gpu-deepseek41, gpu-gates, gpu-glm5next, levers, model, placement: node 1 serves its expert tier to node 0 over a control channel, and a lost node ends both sides - #210
Conversation
… side that drives it TierCard held both halves of an expert tier: the stage side (the replay or feed decision, the go order each graph was captured for, the books, the settle, the release and the reset) and the tier's work on its card (its Gpu, its stacks, the architecture's computation, the staging, the row windows over the page and the graphs). The work is now TierWork, which TierCard holds and calls; each TierWork method is one step the stage side asks of the tier. TierWork keeps the page's layout with its end of the link, so TierCard::capture and TierCard::enqueue_eager no longer take the page. A split with no behavior change: each step makes the same driver calls on the tier card, in the same order and under the same context binds. Nothing outside crates/gpu/src/host changes.
|
Runs of the earlier heads, moved here from the description for its size limit. At
The new clause copies only the shard that holds the row it changes, the smallest such shard, under At
The scan set covers every kernel carrier, so the branch as a whole moves no kernel. The refactor's own scan, against its base, is below. The corpus the nodes gate read. The 64 ids the gate readThe file is one id a line; its first 64 ids, in order (sha256 of the whole 4096-id file FAIL-first of the unit tests. One mutant each, one edit to
FAIL-first of the gate's clauses. One mutant per key check, on the fixture, each one edit to
How the lost node 0 clause holds node 1 mid-step. Holding node 0 with a host flag before a step does not reach node 1: each pass first copies its rows from pageable memory, and that copy blocks the host behind the held stream before the launch request goes out. So the gate relays node 0's channel through a TCP relay of its own. The relay holds the step's first launch frame, waits 500 ms, kills or stops node 0, then delivers the frame (and in the kill case closes the relay's side to node 1). Node 1 then waits on go words that do not come. The precondition line checks that node 1 had served fewer layers than were asked of it. The plan's budget search also requires a host expert on a layer before the last, because a stopped process's card keeps running and only a host leg makes it wait. In the stopped case the gate then continues node 0 and reads the held step's result. The refactor alone (
No run here sets the lever for GLM or V4.1; their refusal of a node tier is the Not run here: |
github-actions
Bot
added
dgx-spark
…cement, gate-paths: node 1 serves its expert tier to node 0 over a control channel, and a lost node ends both sides Node 1 runs tier_server, which serves one load's expert tier over a TCP control channel (host/node.rs, host/node/wire.rs): one connection a load, one reply a request, each request and its reply under one deadline. On connect the server sends its census, each device's census row. BLOOMERY_TIER_NODES (a parsed lever of kind Addrs, loopback addresses only: the page is this host's memory and the channel has no authentication) names the node of each tier. node_cards reads the nodes' censuses, and Place::on plans with each node's card as a tier card (spec_of_device of the row the node sent); the gate plans through the same function. TierCard keeps the stage side: its books, the page words it reads and the stage's stream. Its tier side is the TierWork of a tier of this process, or a TierNode, a client of the channel. The open carries the tier's rows of node 0's plan. Node 1 refuses rows whose experts are not the tier's set, then loads them from its own copy of the file, each tensor checked against node 0's description (NodeOpen::rows_on, Format::of_word) and each row's bytes against their sha256 in node 0's file (NodeRow::sha256), captures its graphs from the go order it is sent, and launches, enqueues, drains and clears its fault word on request. Each request is checked before it reaches the stream; one that fails after it ends the connection with the tier's words released. A second bind and a fault clear with work in flight are refused by name. Over loopback the page is one /dev/shm file of mode 0600 that each process maps and registers with its card (MappedHost::shared_file, Link::across and Link::open_across); node 1 opens only a tier page's name, and node 0 removes the name once both ends hold it. A lost node ends both sides by name. On node 0 a request that fails or passes its deadline marks the node lost (GpuError::Link), and the host tier names the tier a lost card (CardLost). Node 1 ends when the channel closes without a close, or when work asked of it makes no progress past the go deadline and its grace. It then writes RELEASE to its own go words, drains its stream under the drain deadline and returns the loss by name. A node 0 that was stopped and runs again finds those words released when it settles the step, and fails the step with the tier a lost card rather than return rows served after the release. Only Qwen3.8 opens a tier on a node (NODE_FAMILY, swap38's stacks_of_set). GLM and V4.1 open their tier cards through TierOpen::open_here, which refuses a tier on a node by name before any card opens, and the batch port's group start refuses one too. TierOpen::of_machine returns the lever's error, so its callers take a ?. A row's sha256 covers the file bytes of the experts the tier holds, the bytes node 1 uploads: weights.rs's SegmentFile finds a segment's spans and borrows their bytes for the upload and for the digest alike, and model::fileio::sha256_hex_of hashes the spans one after another. Both nodes hash the tier's rows on each open, not the whole file, so a copy whose headers match node 0's file but whose tier experts differ is refused by name instead of serving other weights. The protocol is version 3. gate_qwen38_nodes (just gate-gpu-qwen38-nodes) holds the census, an open of other experts refused, an open on a copy of the file whose one tier row holds other bytes refused, the bits against the same plan in one process, and both ends of a lost node. The history it holds a load to, its comparison, the plan's tier layers and the tier's graph and chain checks after the history move to shared/q38_history.rs, which gate_qwen38_twocard now includes too. The gate starts tier_server through beside, the binary beside the running one, which moves from serve_client to the crate root so a gpu-only bin reaches it; serve_client re-exports it. A loopback test in wire.rs holds the channel's framing with no card, and messages_read_back_as_written (node.rs) the census, the open and each kind of error reply read back as written. tools/unsafe-ratchet.txt: gpu 1749 -> 1754 blocks for the shared file's mapping, its registration, its device pointer, the unregister on that pointer's failure, and MappedHost's drop split into one block a release call. The gate reads BLOOMERY_CARD_BUDGET as gate_qwen38_twocard does: the fixture tier's budget, which box.sh exports, is checked against the fixture's header, and tier_server starts without the levers the gate reads. tools/spawn-allow.txt lists the gate's threads, and tools/defaults-twins.tsv names the clause that runs the gate's plan with BLOOMERY_TIER_NODES unset.
…t node as built over loopback 9.5 items 5 and 6 describe what is built: tier_server and its control channel, the census the load plans its tier cards from (node_cards, read by Place::on and by the gate), the open refused for rows that are not the tier's set or whose bytes are not node 0's (each row's sha256, which both nodes take of the tier's rows rather than the whole file), the TierCard split, the /dev/shm page over loopback, and the two ends of a lost node, with a stopped node 0 that runs again failing its step on the released words. BLOOMERY_TIER_NODES takes loopback addresses only, since the page is this host's memory and the channel has no authentication. Item 5 lists what is not yet done: the census carries no host room until a plan term reads it, once each node has a pool of its own, a remote tier serves no prompt batch, and only Qwen3.8 opens a tier on a node. 9.2's remote census bullet and section 10's engine-side item say the census now reaches node 0 over the channel and the load plans from it, and section 1's P5 row names the loopback run and the tier node off this host as open.
…opened the page A tier node binds its page through the same method as the stage card, so the name says whose context the end is in: the one that made the page (the stage card) or opened it (a tier node). A rename with no behavior change; its four callers in host/tier.rs follow.
c9d0dfd to
cd99d4a
Compare
|
Closing this for now. A lost-node path still needs a fix and a run before it is ready for review. |
1 participant
Supersedes #164.
What and why
This is the tier server row of the P5 plan in #127,
docs/contrib/dgx-spark.md9.5 items 5 and 6. Node 1 serves its share of the routed experts to node 0 over a control channel, the load plans its tier card from the census node 1 sends, and a lost node ends both sides by name. It is built and checked over loopback, two processes on one DGX Spark. The run on two Sparks needs the transport row of #127 and is not part of this PR. Part of #13.The branch is four commits on main (
e6d14f0a..cd99d4a4). The runs below are ateb98a9c0on main89d5ce01; the rebase ontoe6d14f0ahad one conflict, intools/gate-paths.tsv, where the 47 rows this branch touches keep main's text and add onlygate-gpu-qwen38-nodes(in the gate list or, on two rows, in the exceptions), andgit range-diffshows no other change to the patch. Since the body of #164 was last edited it adds a check that both nodes read the same tier bytes (below), it is rebased over #172 and #174 and then onto main89d5ce01, and it fixes the review bot's three must-fixes atc891688f: a send checks its deadline after its last write too, the nodes gate joins a child's stderr reader before it reads the loss lines, and the gate's relay takes the launch tag fromwire::Tag. It also fixes the fifth review's must-fix:beside, the binary next to the running one, has one owner incrates/gpu-gates/src/lib.rs, which the nodes gate andserve_clientboth use. The nodes gate's recipe on the fixture then failed before any clause, becausetools/box.shexports the fixture's card budget and the gate refused that lever; the gate now reads it asgate_qwen38_twocarddoes ("At05ab0b68" below). #172 moved the host's reads, stores, releases and resets of the tier page's words intoLink, so this branch's node side of the page now goes through those methods, and a loopback link with a tier on a node is refused by name ("After the rebase" below). The DGX Spark runs under "What I ran" open witheb98a9c0on main89d5ce01(the rebase of05ab0b68onto main's 44 newer commits), then05ab0b68, where the nodes gate ran through itsjustrecipe on the fixture and on the real file, then18ae4660on main6f87bfb8; the runs of earlier heads follow. The commits:059f5c92, a refactor with no behavior change:TierCardkeeps the stage side, and a tier card's own work on its card moves intoTierWork.747dae62, the tier server, the channel, the load's plan from the census, and both ends of a lost node.9a608d65, the docs.eb98a9c0, a rename with no behavior change:Link::stage_endisLink::own_end, the end of the context that made or opened the page, since a tier node binds its page through it too.How it works:
BLOOMERY_TIER_NODES=<ip:port>[,...]naming the node of each tier in the plan's tier order. The lever takes loopback addresses only and refuses any other by name: the page below is this host's memory, and the channel has no authentication. Node 1 runstier_server --listen <ip:port> [-m <first shard>]. It serves one load and exits once node 0 closes it; a connection that opened no tier (a census read, a refused open) leaves it serving the next. Unset, every tier is a card of the loading process, as before.crates/gpu/src/host/node.rs,node/wire.rs): TCP fromstd::net, one connection a load, one reply a request. A frame is au32length, au8tag and little-endian fields. The requests are open, bind, launch, capture, enqueue, sync, drain, clear fault, fault read and close. Every request getsOkwith its payload orErrwith an error kind and the server's text, and a fault on node 1 comes back to node 0 as a fault, with its layer and sites. On node 0 each request and its reply run under one deadline; node 1 takes its reply's deadline once the request's work returns, so an open or a drain that runs long still gets its reply. A send is late when its deadline has passed after any of its writes, the last one included, and a read once a 50 ms read slice that began past it finds no frame. The server keeps watching the tier's progress while a request is still arriving.node_cardsreads the nodes' censuses and makes each tier card withspec_of_devicefrom the row its node sent.Place::onincrates/gpu-gates(the placement of the generate and serve bins and of the gates) calls it when the lever is set, so such a load plans plan (b′) with node 1's card as the tier card before it loads anything, and the gate plans through the same function. A load placed another way plans its tier card from its own census, and node 1 then opens it only if its census lists the same device (TierNode::openrefuses another by name).NodeOpen::rows_on,Format::of_word). It then hashes each row's bytes in its own file and refuses a row whose sha256 is not node 0's, naming the tensor and its experts. Only then does it open its card and load the rows from its own copy of the file withWeights::load_rows.SegmentFile::of(weights.rs) finds a segment's spans in the file and borrows their bytes for the upload and for the digest alike, andmodel::fileio::sha256_hex_ofhashes them one span after another. A copy of the model whose headers match node 0's file but whose tier experts differ is refused by name instead of serving other weights; bytes outside the tier's rows do not reach node 1's computation, so they are not hashed. The table below compares the ways to check that both nodes read the same model.TierCardkeeps the stage side: the books, the go order each graph was captured for, the page words it reads and the stage's stream. Its tier side isTierWork, a card of this process, orTierNode, a client of the channel.TierWorkis the tier half that existed before (itsGpu, weights, the family'sTierExperts, staging, graphs and its end of the page). The tier server holds one and calls the same methods. The replay or feed decision, the settle, the release and the reset stay on the stage side for both./dev/shm, created with mode 0600. Node 0 makes it, maps it and registers it with its card (MappedHost::shared_file:cuMemHostRegisterwithPORTABLE | DEVICEMAP). Node 1 opens only a tier page's name there, then maps and registers the same file. Node 0 removes the file's name once node 1 has bound it, and a creation that fails removes it too, so a killed process leaves nothing under/dev/shm. Both cards reach one physical page, so the page's protocol runs unchanged across the two processes. The in-process tier page isMappedHost::new(cuMemHostAlloc, not write-combined), so neither page is write-combined and the bits below compare like with like.GpuError::Link). The host tier then names the tier a lost card (CardLost), and the next step and a reset are refused as for a lost card of node 0's. A node that answers but stops signalling is still the go deadline's.RELEASEto its own go words, so every wait its stream holds passes, drains its stream under the drain deadline, and returns the loss by name.tier_serverprints it once, as its exit error, and exits 1. The loss names how many of the layers asked of it it had served and how long node 0 sent no go. If the drain fails, the work is not dropped, so nothing frees memory the stream may still use.How the open checks the model bytes, and what it costs
UD-Q4_K_XL[derived]sha256_hex(g.mapping())) on both nodeshf::fetch's check)NodeRowfield,segment_sha256andsha256_hex_of;SegmentFileis the upload's own lookup moved outEach node hashes
Bbytes on one core, so the open takesB / Rmore on node 0 (before it sends the open) andB / Rmore on node 1 (inside the open), whereRis one core's sha256 rate. That rate has not been measured on the DGX Spark, so no time is given here. The whole-file digest takes 111,334,654,784 / 14,639,820,800 = 7.6 times the bytes [derived], and on a 128 GB machine it also pulls the whole file through the page cache on every open. Node 1 reads its tier rows to upload them anyway, so its digest reads no byte the load would not. The open's deadline on node 0 still holds the digest and the upload together: 120 s plus 14,639,820,800 B at 100 MiB/s is 259.6 s [derived] for this plan.What a node tier adds to each pass
Derived from the code, not measured.
RTTis one request's round trip over the loopback channel: node 0 writes the frame, node 1 reads it, does the call and writes its reply, node 0 reads it (TierNode::call).TierCard::replay,TierNode::launch). The host's service of that pass starts at mostn_node_tiers × RTTlater. A tier of node 0's own launches the same graph with a call, so the round trip is the extra.HostTier::feed_tiers,TierNode::enqueue), and one capture request after the pass settles. That adds at most(L_served + 1) × RTT.82 + 4 × 49 = 278round trips for the whole history [derived].The decode A/B below ran plan (a) on one DGX Spark, which holds every routed expert on the card and has no tier card. It times the
HostTierpath the split touches and covers none of these terms. A same-plan A/B of a tier of node 0's own against a node tier needs a secondGpuon one card, which only the gate's harness opens (the decode runner refuses--place bpon one card), so it is not run here. The transport row replaces the launch message: it can ride in the first image of the mirrored pages (9.5 item 5).What this PR does not do:
plan_nodes(placement: a model's plan over several unified nodes (dgx-spark.md §9), and nodes_scale on one to four Sparks #95) is not called. It needs read rates, a NIC figure and per-node budgets that nothing measures on a live load yet. The census carries no host room: node 1'sMemAvailablereading joins it with the plan term that reads it, once each node has a pool of its own in the plan (9.2).NODE_FAMILYinarch/qwen3moe/tier38.rs). GLM and V4.1 open their tier cards throughTierOpen::open_here, which refuses a tier on a node by name before any card opens.After the rebase over #172. #172 gave
Linkthe methods the host uses on the tier page's words (load,store,release,at_rest,reset). This branch now builds on them:Linkkeeps gpu, levers, justfile, tools, docs: the tier page's counters cross a link as generations; a loopback link carries them between two pages on one machine with a host pump #172's fields and methods.Link::across,Link::open_acrossand the file's removal on drop sit beside them, and a shared page has no mirror. The node side's word reads, stores and releases, which went through this branch's ownword()andhost_atbefore, now go throughLink::load,storeandrelease.TierPage::newis gpu, levers, justfile, tools, docs: the tier page's counters cross a link as generations; a loopback link carries them between two pages on one machine with a host pump #172's, unchanged. A load with a tier on a node builds its page withTierPage::across, and both go through one constructor for the page's words.release_opened), now with oneLink::releasecall.Linkdoes on the stage side.bind_openedchecks the page's size against the layout before any word is read, so no request reaches that.BLOOMERY_TIER_LINK=loopbackis refused by name (host/mod.rs,attach_tiers), because the loopback link's tier side is a page of node 0's process and node 1 maps the shared page. The refusal comes when node 0 attaches its tiers, after node 1 has opened its rows. No gate runs it yet.At
eb98a9c0,just gate-gpu-qwen38-nodespassed on a DGX Spark on the Qwen3.8 fixture and refused by name a fixture budget that differs from the header's; at05ab0b68, the same patch on main6f87bfb8, it also passed onUD-Q4_K_XL. At18ae4660these ran there ("What I ran"): the PTX scans, gate-gpu-lib,gate-gpu-qwen38-nodeson the fixture and onUD-Q4_K_XL(all six clauses, among them the other-bytes open and the two lost node 0 clauses the stderr fix reaches), and the FAIL-first mutants of the send deadline and of the stderr reader.gate-gpu-ds41-tierandweekly-gpu-ds41-lost, which run the merged page code, end with exit 66 as shipped on that machine, at main as at the head, because it has no DSpark fixture draft; the same gate passed its landing and lost clauses there, over the shared page and over loopback, with two local edits.Where each part lives
crates/gpu/src/host/tier.rs(commit 1): theTierCardsplit into the stage side andTierWork.TierCard::nodes_per_layerand the books are unchanged. Commit 2 addsTierSide, the node calls, the release check the settle reads andTierOpen::open_here.crates/gpu/src/host/node.rs:NodeCensus,node_cards,NodeOpenandNodeRow(the open's rows, each with its bytes' sha256,segment_sha256),TierNode(node 0's client: the census, the open, the per-request calls andlost),NodeFamily(a family's tensor reader and tier computation for the server) andserve(node 1's loop, the stopped check and the end on a lost node 0).crates/gpu/src/host/node/wire.rs: frames, tags (Tagis public, and the nodes gate's relay reads the launch tag from it), the channel with its deadlines, and the readers and writers of the fields. A payload that ends early or runs past its fields is refused by name. The protocol is version 3.crates/gpu/src/weights.rs:SegmentFile::of, the lookupupload_segmentdid inline (the tensor found and checked in the file, its spans and their bytes), now shared with the digest.crates/model/src/fileio.rs:sha256_hex_of, the sha256 of several slices one after another;sha256_hexcalls it.crates/gpu/src/host/link.rs:Link::across(node 0 makes and maps the file),Link::open_across(node 1 maps it, a tier page's name only), and the file's removal. Commit 4 renamesstage_endtoown_end.crates/gpu/src/graph.rs:MappedHost::shared_fileand the unregister in its drop; a shared page is a whole number of 64 KiB.crates/gpu/src/host/mod.rs:attach_tiersopens a node's tier through the channel; a failed request marks the tier lost; the settle refuses a released node tier; the batch port's group start refuses a node tier.crates/gpu/src/arch/qwen3moe/tier38.rsandswap38.rs:NODE_FAMILY, andstacks_of_set, the stacks of the layers a tier set gives experts (a node holds one tier's set and no plan).crates/gpu-gates/src/generate.rs:Place::ontakes the tier cards fromnode_cardswhen the lever is set, and a load line'scardsfield names a node tier's device.crates/levers: theBLOOMERY_TIER_NODESrow (class C, kindAddrs, loopback only) andtier_nodes().crates/placement/src/placement.rs:Format::of_word, the format a word names, with a unit test;Format::ALLis held to the enum by an exhaustive match.crates/gpu-gates/src/lib.rs:beside, the binary next to the running one, moved here fromserve_client.rs, which re-exports it, so a bin built with thegpufeature alone reaches it.gate_qwen38_nodesstartstier_serverthrough it.crates/gpu-gates/src/bin/tier_server.rs,gate_qwen38_nodes.rsandshared/q38_history.rs, the recipegate-gpu-qwen38-nodes, and the recipe added to 47 of the 62tools/gate-paths.tsvrows that namegate-gpu-qwen38-twocard: the rows of the app, runtime, tokenizer, model, gpu, gguf, placement and similar sources a Qwen3.8 load reads, and the fixture generator's row, whose header key the gate's budget check reads. The other 15 rows (the Qwen3.8 fixture module, reference-set readers, residency files, vision position rows and the like) are unchanged. The gate readsBLOOMERY_CARD_BUDGETasgate_qwen38_twocarddoes:tier::plan_leverschecks it against the fixture's header in the fixture tier, and the plans keep the gate's own budgets.tier_serveracts on none of the four levers the gate reads and refuses each one that is set, so the gate starts it without them. The two rows that namegate-gpu-qwen38-twocardamong their exceptions, the model readers ofmimo2andnemotron_h, namegate-gpu-qwen38-nodesbeside it for the same reason.tools/unsafe-ratchet.txt: gpu 1749 to 1754 blocks for the shared file's mapping, its registration, its device pointer, the unregister on that pointer's failure, andMappedHost's drop split into one block a release call, each with its SAFETY line.TierOpen::of_machine, which now returns the lever's error, take a?(app, gpu-deepseek41, gpu-glm5next and five gate bins).docs/contrib/dgx-spark.md: 9.5 items 5 and 6 state what is built and what is not, 9.2 and section 10 say the census reaches node 0 and the load plans from it, and section 1's P5 row names the loopback run.Code shape
No second copy. What both Qwen3.8 tier gates hold a load to is one file,
crates/gpu-gates/src/bin/shared/q38_history.rs, whichgate_qwen38_twocardandgate_qwen38_nodesinclude: the history (Run,step,window,history,same,WINDOWS), the plan's tier layers (tier_layers), the tier's graph and chain checks after the history (tier_graphs_and_chains) and the nodes a tier layer (TIER_LAYER_NODES). The two-card gate keeps only its residency count, its stage graph line, itstier:stats line and its precondition; the nodes gate keeps its precondition on the tier's experts and resident bytes.Run's write and read in the nodes gate use the channel's own field writer and reader (node/wire.rs). A tier node binds its page throughLink::own_end, the stage card's method.Model code holds only architecture facts. Qwen3.8's code gains
NODE_FAMILY(its tensor reader and tier computation, both existing functions) andstacks_of_set. The node path, the channel and the loss handling are common code.Dead code is gone.
No silent failure. These are each refused by name:
BLOOMERY_CARD_BUDGETthat differs from the fixture header's budget (the nodes gate, as the other gates).The other-file and version refusals, the census refusals and the error reply's kinds (a fault with its words, a protocol error, a refusal) have unit tests (
node.rs, where the open's rows read back with their sha256), the channel's frames and ends a loopback test (wire.rs), and the refusal of a node tier for GLM and V4.1 a unit test ofTierOpen::open_here(tier.rs); the swapped open and the open on a copy with other tier bytes are gate clauses.Comments state what is true now.
No relaxed gate. No band, tolerance or lint moved. The unsafe ratchet goes up for new code only, with a
PIN(2026-10-14)line.cargo fmtandcargo clippyare clean on what I touched at05ab0b68(aarch64, below); the lint count did not go up. The later heads change no Rust code beyond the rebases.Duplication found or removed (
path:line), or "none": the Qwen3.8 history, the tier layers, the tier's graph and chain checks andTIER_LAYER_NODES, now incrates/gpu-gates/src/bin/shared/q38_history.rs, are removed fromgate_qwen38_twocard.rsand have no copy ingate_qwen38_nodes.rs.gate_qwen38_residency.rsand the GLM and V4.1 twocard gates keep their own histories, which differ in what they step and compare. The nodes gate's own copy ofbeside(gate_qwen38_nodes.rs:471at18ae4660) is gone; it uses the one incrates/gpu-gates/src/lib.rs.What I ran
On a DGX Spark
The runs here are in the dev container (Hardware, below), with
--ipc=hostso both processes see one/dev/shm, withBLOOMERY_BOX_LOCAL=1and PTX forsm_86(the driver JITs it for the GB10). The nodes gate at05ab0b68and before its budget fix, and the V4.1 rows at18ae4660ran theirjustrecipes, which go throughtools/box.sh; the other rows rantools/gate.shandtools/gpu-gate.shdirectly. Each log starts with the tree's commit; a mutant's log adds the mutant's name.At
eb98a9c0, against main89d5ce01. The branch is05ab0b68rebased onto main's 44 newer commits.git range-diff 6f87bfb8..05ab0b68 89d5ce01..eb98a9c0shows commits 1, 3 and 4 equal and commit 2 changed only intools/gate-paths.tsvandtools/spawn-allow.txt, the two lists main also changed: three of the rows this branch addsgate-gpu-qwen38-nodesto (crates/runtime/src/**,crates/threads/src/lib.rsandcrates/placement/src/**) carry main's edits to their gate lists as well, and the nodes gate'sspawn-allow.txtline sits beside main's changed lines. The recipe still lands in 47 of the 62 rows that namegate-gpu-qwen38-twocard.git diff --numstatof the whole branch is the same before and after the rebase, file by file. Of the files main changed,crates/levers/src/lib.rs,crates/levers/src/tests.rsandjustfileare also edited here, in other hunks (main removed the slot count from the paged rule; this branch addsTIER_NODES). The nodes gate,tier_serverand the gate's recipe are not touched by main. The runs below are the ones those changes can reach, on the same DGX Spark and container as the05ab0b68runs; the real-file run,just fmt-checkandjust lintwere not run again ateb98a9c0, and their results at05ab0b68stand below for that commit.eb98a9c0, against main89d5ce01BLOOMERY_BOX_LOCAL=1 BLOOMERY_TIER=fixture BLOOMERY_GATE_BOUND=3600 just gate-gpu-qwen38-nodesPASSED: gate_qwen38_nodes,clauses: 6 ran, noFAILline, gate 119 s,justexit 0; no/dev/shmfile and no copy leftBLOOMERY_BOX_ENV=BLOOMERY_CARD_BUDGET=9489612799gate_qwen38_nodes: BLOOMERY_CARD_BUDGET=9489612799 and the fixture's bloomery.fixture.card_budget is 9489612800: the fixture tier plans under the header's budget,justexit 1BLOOMERY_HOST_POPULATE=1 target/release/tier_server --listen 127.0.0.1:005ab0b68cargo test -p bloomery-leverstools/check-*.shcheck-recipesexits 1 on five self-test cases (the Mac remote lock's self-test andgpu-gate.sh's read-only queue case); main89d5ce01fails the same five in the same container, and the failing lines are equal after the temporary directory names are removed. Its gate-path check passes: 120 gate rows held to the derivation, 23 rows with exceptions naming 264 gates, against main's 262At
05ab0b68, against main6f87bfb8.git range-diff 6f87bfb8 18ae4660 05ab0b68shows two changes, both in the tier server commit, whose message says both:besidemoves tocrates/gpu-gates/src/lib.rsand the nodes gate drops its copy, and the nodes gate reads the budget lever (below), with the fixture generator'stools/gate-paths.tsvrow naming the gate. No kernel or engine file changed, so the scans, gate-gpu-lib and the V4.1 rows at18ae4660below still describe this head. The tree is a clone at05ab0b68in the same container on another DGX Spark (NVIDIA GB10, 12.1, 595.84), built in a target directory of its own withcargo oxide0.2.1, whose backend is built from cuda-oxide87bc9518, the workspace's pin.BLOOMERY_GATE_BOUND=3600was set as in the earlier runs; each gate run took less than the default 900 s.The fixture tier through the recipe. Before this fix (and, by the code, at
18ae4660, whose gate lists the same levers [derived]),BLOOMERY_TIER=fixture just gate-gpu-qwen38-nodesfailed before any clause.tools/box.shexports the fixture header's card budget asBLOOMERY_CARD_BUDGETin the fixture tier, and the gate, which did not list that lever, refused it. The fixture rows at18ae4660and before startedtools/gpu-gate.shdirectly, so they never saw the export. The gate now lists the lever and checks it against the fixture's header withtier::plan_levers, the checkgate_qwen38_twocardmakes; the plans keep the gate's own budgets.tier_serveracts on none of the four levers the gate reads and refuses each one that is set, so the gate starts it without them.05ab0b68, against main6f87bfb805ab0b68without it), on the same machine:BLOOMERY_BOX_LOCAL=1 BLOOMERY_TIER=fixture BLOOMERY_GATE_BOUND=3600 just gate-gpu-qwen38-nodesgate_qwen38_nodes: BLOOMERY_CARD_BUDGET="9489612800" is refused: this binary (gate_qwen38_nodes) does not act on it; unset it,justexit 1BLOOMERY_BOX_LOCAL=1 BLOOMERY_TIER=fixture BLOOMERY_GATE_BOUND=3600 just gate-gpu-qwen38-nodesPASSED: gate_qwen38_nodes,clauses: 6 ran, noFAILline, 113 s,justexit 0; no/dev/shmfile and no copy left (block below)BLOOMERY_BOX_ENV=BLOOMERY_CARD_BUDGET=9489612799, one byte below the header's budget (tools/box.shexports the entry after the profile)gate_qwen38_nodes: BLOOMERY_CARD_BUDGET=9489612799 and the fixture's bloomery.fixture.card_budget is 9489612800: the fixture tier plans under the header's budget,justexit 1BLOOMERY_HOST_POPULATE=1 target/release/tier_server --listen 127.0.0.1:0, a lever the gate reads, set fortier_servertier_server: BLOOMERY_HOST_POPULATE="1" is refused: this binary (tier_server) does not act on it; unset it, exit 1. This is why the gate starts node 1 without the four levers it readsBLOOMERY_BOX_LOCAL=1 BLOOMERY_TIER=real BLOOMERY_GATE_BOUND=3600 just gate-gpu-qwen38-nodesonUD-Q4_K_XL, the file the profiletools/ref/models/qwen4exp.shnamesPASSED: gate_qwen38_nodes,clauses: 6 ran, noFAILline, 431 s,justexit 0; no/dev/shmfile and no copy left (block below)just fmt-checkandjust lint(clippy with the recipe's features, native aarch64) in the same container, on05ab0b68and on main6f87bfb8^warning:39 on both commits; the warning lines differ only in which of a pair is marked duplicate. The ratchet's 41 is the Mac's x86_64 count, so the aarch64 count is compared with main'stools/check-*.shthatjust mac-staticruns, in the same container on05ab0b68check-recipesexits 1 on five self-test cases that fail the same way on main6f87bfb8there (the Mac remote lock's self-test andgpu-gate.sh's read-only queue case); its gate-path check passes with 264 gates named in exceptions, against main's 262cargo oxide build --arch sm_86 -- -p bloomery-gpu-gates --features deepseek41 --release --bin gate_qwen3_serve, a bin that takesbesidethroughserve_client, before the budget fix (serve_client.rsandlib.rsare the same at05ab0b68)On the Qwen3.8 fixture through the recipe, abridged: the cargo output, the budget lines between 16 and 8 GiB, the expert lists, the host tier's load lines and node 1's listening, census and loss lines are cut, long lines end in
..., and the data directory shows as$BLOOMERY_DATA.justexited 0.On the real file through the recipe, abridged the same way:
At
18ae4660, against main6f87bfb8. Both trees aregit archiveof the two commits, built from scratch:18ae4660, against main6f87bfb8nvidia-smion the machine of the runNVIDIA GB10, 12.1, 595.84, CUDA Version 13.2 (block below)ptx-scan --no-jitofgate_kquant,gate_swap,gate_qwen38_twocard,gate_e2e(gpu),generate_ds41(gpu, deepseek41) andbloomery-serve-ds41(deepseek41, vision) on both commits,tools/ref/ptx-canon.py --scansmd5 identical=N moved=0for every bin: 238, 235, 234, 234, 301 and 317a_channel_reads_whole_frames_and_its_endswith the review's expired-send assertion,messages_read_back_as_writtenanda_tier_on_a_node_is_refused_by_namegate_qwen38_nodeson the Qwen3.8 fixtureclauses: 6 ran, noFAILline, 124 s; no/dev/shmfile and no copy left (block below)gate_qwen38_nodesonUD-Q4_K_XLclauses: 6 ran, noFAILline, 555 s; no/dev/shmfile and no copy left (block below)Channel::sendchecks its deadline only while bytes are left (at < self.outbuf.len() && Instant::now() > deadline), gate-gpu-lib's command onhost::node::wire::testsa_channel_reads_whole_frames_and_its_endsis the one failure,called Result::unwrap_err() on an Ok value;bloomery-gpucompiled againwait_withinreturns without joining the stderr reader, and the reader sleeps 1000 ms before each append; the nodes gate on the fixtureFAIL lost node 0 (killed)andFAIL lost node 0 (stopped), each with "the loss named false" and its precondition failing at "past 0 of the 0 layers", thenFAILED(exit 1)FAILline; node 1 ended 1.1 s after node 0 was killed and 11.3 s after it was stopped, against 0.3 s and 10.4 s in the run with no sleepjust gate-gpu-ds41-tierandjust weekly-gpu-ds41-lostas shipped,BLOOMERY_TIER=fixture, on both commitstools/ref-paths.shwants the one DSpark fixture draft, which this machine does not havegate_deepseek41_tieron the V4.1 fixture with the two local edits below: the landing clauses (--union --fault --two --batch --batch2 --bfault) and the lost clauses (--bfirst --blost --lost) on both commits, and both again on the head withBLOOMERY_TIER_LINK=loopbackPASSED: gate_deepseek41_tier, noFAILline; the stage graph 288 nodes, the tier's step and pair graphs 64 and 127, as at #172Where this run differed from the plan for it:
18ae4660tree withBLOOMERY_GIT_COMMITset to it; their logs have no tree line.justandtools/box.shin local mode, with acargo oxidebuilt from cuda-oxide87bc9518, the workspace's pin. Every other row rancargo oxide buildin the container directly, with the image'scargo oxidefrom cuda-oxide4da6c138.engram/corpus-prose.ids, which this machine does not have. The first attempt of the six ended with exit 1 on the missing file. A stand-in from a fixed-seed generator (seed 20261013, sha25627cc28dbcc6521ec...) was then put in place by hand, and the rows above are the second attempt.qwen4exp/corpus-prose.idsis not on this machine either, so the nodes gate read the same 4096-id stand-in as every nodes gate run in this PR (its first 64 ids are listed below).nvidia-smion the machine of the run:On the Qwen3.8 fixture (
gate_qwen38_nodeswithBLOOMERY_TIER=fixture), abridged: the budget lines between 16 and 8 GiB, the expert lists, the host tier's load lines and node 1's census as node 0 prints it are cut, long lines end in..., and the data directory shows as$BLOOMERY_DATA. Node 1 prints its loss line once in each lost node 0 clause, astier_server's exit error.On the real file, Qwen3.8
UD-Q4_K_XL, abridged the same way. The plan puts 4671 experts on node 1's tier and 17943 on the host:The runs of the earlier heads
6b16c8d2and69a25bf6are in the first comment below, to keep this body under GitHub's size limit.On a Mac
At
18ae4660, with main6f87bfb8as the base:just mac-static,just mac-test,python3 tools/recipes.py checkand the aarch64 cross check of the workspace (BLOOMERY_CHECK_TARGET=aarch64-unknown-linux-gnu tools/mac-check.sh check, the Spark's triple). The block is abridged: the eleven check-script lines are merged into one, and the timings and long tails are cut.The combos come from the ledger: they ran at
2cc0eeb9, the same four patches on main15838a46(git range-diffshows them equal), 42 of 42 green, and main's one commit since changes three Python tools, which no build shape reads. The one red isbloomery-refset'sa_plain_target_with_no_draft_beside_is_the_family_path: the temporary directory comes back once with and once without macOS's/privateprefix. This PR does not touchcrates/refset, and the same test is red on the same Mac at maincb5688c9alone.Hardware (if the change is for a machine)
One DGX Spark a run (GB10, cc 12.1), Linux 6.17.0-1032-nvidia aarch64, host driver 595.84. The census total was 130,596,122,624 B in the runs at
eb98a9c0and05ab0b68, 130,661,761,024 B in the runs at18ae4660, 130,596,118,528 B in the runs at69a25bf6and 130,596,122,624 B in the earlier runs. The checks ran in a container built onnvidia/cuda:13.3.1-devel-ubuntu24.04, withrustc 1.100.0-nightly (e457a7b0d 2026-08-27)and just 1.58.0. Both processes ran on the one GB10. Apart from the decode A/B these are pass and fail results, not timings. The logs of the runs at69a25bf6did not keep the machine'snvidia-smiline; the runs at6b16c8d2,18ae4660,05ab0b68andeb98a9c0recordedNVIDIA GB10, 12.1, 595.84.AI help
AI 코딩 에이전트가 코드를 작성한 후 설명을 썼고, 테스트를 마쳤습니다. diff 리뷰와 PR 초안 리뷰는 제가 직접 했습니다.