A static cost estimator for LLM inference. Given a model, a hardware topology, a sharding configuration, and a workload, it answers one question:
How fast does this run, and what does it cost?
The motivating use case is capacity and hardware planning. When a new model lands, you want to compare deployment options quantitatively — TPU v7 pod vs. GB200 NVL72, under which sharding on each — without standing up a real deployment for every candidate.
You give it four things:
- Model — layers with explicit attention (MHA / GQA / MLA) and FFN (dense / MoE) dimensions. No hardcoded architectures.
- Hardware — per-chip FLOP/s, HBM bandwidth and capacity, and the topology's physical axes, each with its geometry (ring, switch, network hop) and link speeds.
- Sharding — sizes for the parallelism roles (DPA, TP, EP, ETP, PP), with an optional role→axis placement. Omit the placement and it searches for the best one.
- Workload — prefill or decode, token count, batch size.
It lowers that into a graph of costed operations (GEMMs, weight and KV loads, collectives), and a cost backend prices the graph to produce a time estimate.
-
Structure is shared; physics is pluggable. The model description, topology, and lowering pipeline are fixed shared machinery — the product. How a costed graph turns into a time estimate is delegated to an interchangeable cost backend. Today that's a simple roofline model; a more detailed scheduling simulator can be added later without touching anything upstream. The pipeline is agnostic to which backend is active.
-
It never makes a decision it can't justify with a price. Wherever there's a choice — which collectives to emit at a sharding boundary, which physical axes a role should occupy — it enumerates the legal options and lets the active backend pick the cheapest. There are no hand-coded "on TPUs, prefer X" policies, so the tool stays correct on hardware nobody had in mind when it was written.
npm install
npm run dev # interactive explorer
npm test # run the suite
The explorer ranks sharding configurations per chip, and drills into any one with a fabric view (the chips on their real interconnect) and an execution trace (the op graph, colored by what each op is bound on).
The same engine as a command line, estimate, for scripting sweeps and pinning
configurations in other repos. Build it once (npm run build:cli, a single
node script at dist/cli/estimate.mjs, no runtime dependencies) and:
estimate models # the presets
estimate chips --model k3 # chips, with the least chips that hold the weights
estimate search k3 gb300-nvl72 32x1 --slo 20 # rank every sharding, streaming as it prices
estimate top k3 b300 8x1 --phase prefill # the same, silent
estimate explain k3 b300 8x1 --sizes TP=8,EP=8 # one sharding, fully worked, with engine flags
estimate sweep --model k3 --chips b300,gb300-nvl72 --machines 8x1,32x1 --slos none,20 --table
--json emits one row per line; sweep writes one file per cell and report
re-renders them. Two knobs correct the simulator where its defaults mislead,
both off unless asked for: --state-slots N (with --spec-slots,
--state-dtype, --mem-fraction) reserves the recurrent-state slots a hybrid
KV manager actually holds per sequence on linear-attention layers, and
--scheduler dag reads overlap off the op graph instead of the two --overlap
fractions. Every number is an estimate.
src/core engine (lowering, placement search, cost backends), model & hardware specs
src/ui the explorer — leaderboard, fabric view, execution trace
src/cli the estimate command line (bundled by scripts/build-cli.mjs)
tests property / fuzz tests