[juliankang4]

Refs #127.

What and why

This is the tier-cost harness the review bot asked for on #219 as a non-blocking item: a published call and output for what a node tier costs against a local tier on the same fixed plan. tools/ref/tier-cost.sh runs Qwen3.8's plan (b′) twice per round, once with the tier in the stage process and once with tier_server serving it, alternating the arm order each round under the timing lease. It reports the node/local ratio of decode rate and step time with a paired interval, and checks that both arms used the same placement, warm history and greedy tokens. Today tier_server refuses a listen address that is not loopback, because the tier page is shared through /dev/shm on one host, so this is one machine and one card. The output starts with one machine, loopback: not a two-machine timing.

Release notes: internal: the node gate gains a loopback local-versus-node tier timing, with a harness and a reproduction recipe in docs/contrib/dgx-spark.md 9.5.

What changes:

• gate_qwen38_nodes takes --measure local|node GIB. It reuses the gate's placement, warm history and child lifetime code. Each arm warms the same history, clears, feeds 32 prompt ids as steps, then times 96 greedy steps with full logits retained. After the warm history it checks that one GPU process (local) or two (node) are on the machine.
• crates/gpu-gates/src/record.rs gains the tier_cost record kind: arm, plan, history and timed tokens, prompt and decode counts, tg_tps, step_ms and the GPU process count. A unit test compares the gate's schema output with the parser's fixture tools/ref/data/tier-cost.schema.jsonl.
• tools/ref/tier-cost.sh reuses the timing lease, the timing-card witness and lease_bounded. GIB is 1 to 16 (default 16) and ROUNDS is 2 to 100 (default 6). --dry-run prints the commands without taking the lease or loading a model.
• tools/ref/tier-cost.py reads the records through tools/bloomery/records.py. It refuses a pair or a round whose placement, history tokens or timed tokens differ. It prints the node/local ratio mean for tg_tps and step_ms with a paired Student t 95% interval, marked [derived].
• tools/check-recipes.sh lists the new parser's self-test with the other Python self-tests.
• docs/contrib/dgx-spark.md 9.5 carries the build, self-test, dry-run and reproduction commands and the output fields.

tg_tps is 96 divided by the timed span and step_ms is the same span divided by 96, so the two are reciprocals and not separate clocks.

What this PR does not do

• It does not time two machines. The harness has no two-machine option, since the current transport cannot run one. The doc points at the off-host transport item in section 10.
• It publishes no timing number. The node arm's two CUDA contexts time-share one GPU, so a result from this setup measures the loopback setup and not a network transport cost.
• It changes no production decode or prefill path and adds no kernel.

Code shape

• No second copy. The gate's Common::machine, plan_under, Common::open, q38_history and the node and stage child code serve both arms. lease.sh, timing-card.sh, the depth runners' arm alternation, records.py and the t table are reused. The only new reader is tier-cost.py, and it calls records.py.
• Model code holds only architecture facts. The change adds no per-model branch to a common owner.
• Dead code is gone. The change makes nothing unused.
• No silent failure. A budget outside 1 to 16 is a named error in the gate and in the script, a round count outside 2 to 100 is refused, and a parse failure or an incomplete record fails the parser.
• Comments state what is true now.
• No relaxed gate. No band, tolerance or lint moved, and tools/unsafe-ratchet.txt is unchanged.
• cargo fmt and cargo clippy are clean on what I touched; the lint count did not go up. The changed Rust files were formatted with the pinned nightly rustfmt, and clippy is not run on this head. See the lint row below, where just mac-static is red on this head and on its base alike.

Duplication found or removed (path:line), or "none": none found. The new record kind and the new parser are the only readers of the new fields, and each reuses the existing owner of its job.

What I ran

The head is 2e29750f. The branch merges into the current upstream/main (85511826) without conflicts, as git merge-tree --write-tree shows. The compile and harness checks below ran on one DGX Spark, in the configured Spark build container (CUDA 13.3.1), on a source copy of this change made before the commit. The copy's seven changed files match the committed ones byte for byte by SHA-256.

Check	Result
python3 tools/ref/tier-cost.py --self-test	rc 0, tier-cost self-test: 16 of 16 ok (synthetic, NOT TIMING)
FAIL-first: the same self-test with the token comparison removed in memory	The self-test fails with tokens mismatch accepted, and the FAIL-first command returns rc 0
cargo oxide build --arch sm_121 -- -p bloomery-gpu-gates --features gpu --release --bin gate_qwen38_nodes --bin tier_server --offline	Cargo build succeeded. It prints the existing DynamicSharedArray warning from bloomery-gpu
The built gate's --records-schema compared with tools/ref/data/tier-cost.schema.jsonl	live record schema matches parser fixture
gate_qwen38_nodes --measure-node0 0, --hold0 0, --measure local 0 and --measure node 17	Each exits rc 1 with budget must be in 1..=16 GiB
bash tools/ref/tier-cost.sh --dry-run 16 6, and the bounds 1 2 and 16 100	rc 0, prints one machine, loopback: not a two-machine timing, the two commands and the order
The same dry run with 0 2, 17 2, 1 1 and 1 101	Each exits rc 64
just mac-static on a Mac, compared with a base log for a688accb	head red steps: check-recipes lint, base red steps: check-recipes lint, warning source files head vs base: identical, verdict: same as base. Both sides read ^warning: 59, above the ratchet 41, so this check is not shown to pass
The new Rust unit test tier_cost_parser_fixture_is_current, and the rest of the unit and host tests	not run
cargo clippy on its own	not run
just contrib-check (--tier t0)	still running on the card when this was posted; the result goes into this description when it ends
The gates just affected selects	still running on the card when this was posted; the result goes into this description when it ends
gate_qwen38_nodes --measure and tools/ref/tier-cost.sh with the real model	not run, so this PR has no tier_cost record and no ratio

just mac-static is red here only through steps that are red on the base. Its comparison base is a688accb, an older upstream commit than the one the branch sits on.

What I ran into

None of this blocks the PR.

tier_server accepts only a loopback listen address, because the node tier shares the tier page through a /dev/shm file on one host. A two-machine run needs the off-host transport first, so I did not add a two-machine call that could not run.

The harness has compiled and passed its parser self-test, its FAIL-first and its argument checks, but it has not timed the real model. The first real run will need the qwen4exp model, the prose corpus and a prediction card, as the doc's command shows. I left that run out so that this PR does not carry a number from the loopback setup, where two CUDA contexts share one card.

Hardware (if the change is for a machine)

The compile, self-test, FAIL-first and dry-run checks ran on one DGX Spark in the Spark build container with CUDA 13.3.1, for sm_121. just mac-static ran on a Mac. These are pass and fail results, not timings.

AI help

AI 코딩 에이전트가 코드를 작성한 후 설명을 썼고, 테스트를 마쳤습니다. diff 리뷰와 PR 초안 리뷰는 제가 직접 했습니다.

[github-actions]

Verdict: CLEAN: No must-fix defect survived the code review and boundary checks.

Non-blocking

⚠ Exercise the real-model integration: tools/ref/tier-cost.sh:51. The PR reports no real-model run. The GPU-process precondition and complete local/node record path remain unverified. Attach one two-round run with the card/driver line, commits, model path, commands and output before using its ratios as evidence.

Checked

Duplicates: 9 new functions/constants checked; no duplicated knowledge found. No dryer pairs were available.
Complexity: crapper reports measure at 29, a new plan/load, warm/check and time/record pipeline; those are its natural stage boundaries. run rises 11→22 through mode dispatch. parse_args rises 8→15 through mode and budget validation. No behavior defect found in these branches.
Silent failure: 4 error paths followed: argument parsing, planning, process census and child exit. Errors reach a nonzero exit. No new ignored Result or error default found.
Tests: Python self-test passes 16/16. Six mutants at tier-cost.py:18, :30, :44, :52 and :54 turn assertions red: dropped record-count, finite-value, token and cross-round checks, reversed ratios and a wrong interval denominator. The fixture matches the Rust kind statically; renaming the arm key at record.rs:1116 breaks its equality check. Valid shell boundary cases pass; invalid budgets and rounds exit 64.
Gates: no existing pin, tolerance or gate was relaxed or removed.
Escape catalog: no C1–C4 pattern matched the added code.

Notes

Dryer did not run. Rust unit tests and GPU runs were not executed during this review. The PR carries no qualifying GPU gate evidence.

_{Card check: needs-gpu. This pull request changes crates/gpu-gates/src/bin/gate_qwen38_nodes.rs, crates/gpu-gates/src/record.rs and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[github-actions]

Verdict: CLEAN: No reachable behavior failure or proven quality defect was found in the changes.

Non-blocking

⚠ Validate the complete harness: tools/ref/tier-cost.sh:51. The PR reports no real-model run, and the Rust fixture test:5159 has not run.
Run the documented recipe and the fixture test. Attach both arms' records, the ratio output, and the completed selected gates with the card/driver line, commit hashes, model path and commands.
This checks the process-count and child-lifetime assumptions before others use the harness for timing evidence.

Checked

Duplicates: 9 new functions/constants checked against existing owners; no duplicate found. Records, history, placement, child cleanup, lease and t quantiles reuse existing owners. Dryer supplied no pairs.
Complexity: measure is new at 29, with setup/checks and timing/reporting stages; run 11→22 adds mode dispatch and child checks; parse_args 8→15 adds budget validation and mode arms. No structural finding.
Silent failure: 4 paths followed: budget parsing, process query, child completion and record parsing. Invalid inputs propagate named failures.
Tests: The PR reports Python self-test 16/16. By inspection, dropping the arm or round comparisons at tier-cost.py:44, reversing the ratio at :52, or dropping the finite check at :29 fails a self-test assertion. Prefixed child output plus the forwarded record parses once. Changing TIER_COST.prompt from U64 to I64 breaks the Rust fixture assertion by inspection; that test was not executed.
Gates: no existing pin, tolerance, test or clause removed or relaxed.
Escape catalog: no C1–C4 pattern matched; shell pipelines retain pipefail and command failures propagate.

Notes

Dryer did not run, according to diagnostics. No GPU run or Rust build/test was performed here. The PR contains no completed card gate evidence.

_{Card check: needs-gpu. This pull request changes crates/gpu-gates/src/bin/gate_qwen38_nodes.rs, crates/gpu-gates/src/record.rs and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[juliankang4]

The review is done and the contribution check passed, so I'm merging this.