Skip to content

backend: Add MX quant/dequant on-the-fly compute (size-changing datapath) - #167

Merged
DanielKellerM merged 6 commits into
develfrom
otf/size-changing-compute
Aug 11, 2026
Merged

backend: Add MX quant/dequant on-the-fly compute (size-changing datapath)#167
DanielKellerM merged 6 commits into
develfrom
otf/size-changing-compute

Conversation

@DanielKellerM

@DanielKellerM DanielKellerM commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds size-changing on-the-fly compute (write byte count != read byte count) on top of the transpose/otf_compute framework, with MX (OCP microscaling) quantization and dequantization as the first ops. Based directly on devel (an earlier revision stacked on #166; the per-beat retire signal turned out to be unnecessary — see the retire note below — so #166 was withdrawn).

Mechanism. Per-op source:dest byte ratios in idma_pkg (compute_in_bytes/compute_out_bytes) program the legalizer write length, so the write burst's beat count matches the beats the compute engine emits. The compute seam presents per-lane valid (pack-buffer occupancy) with full strobes; the write manager's own beat mask gates acceptance, and the engine retires lane-exactly against the byte lanes the write port actually consumed (buffer_out_ready), so a partial tail beat pops only its own bytes — no drain FSM, no transport-layer state machine, and back-to-back transfers with an identical compute config pipeline with zero inter-transfer bubbles. The legalizer asserts fence the contract: input-granule alignment, beat-aligned src/dst, per-beat-capable destination protocols (no TileLink), the FP16 width cap, and that the requested op is elaborated (EnableCompute / ComputeOps).

Retire note (transpose unchanged). Transpose keeps devel's per-write-transaction retire (w_dp_req_ready) and its existing contract that every transpose write burst is single-beat (guaranteed by the midend strip decomposition, now made explicit by the ComputeTransposeSingleBeat assert — multi-beat transpose bursts mis-stream under any single-signal retire and were already unsupported on devel; a whole-tile "compact" transpose is future work). The MX ops do not use a scalar retire at all: they pop lane-exactly from buffer_out_ready, which the write ports already produce per beat. An earlier revision (#166) re-exposed a per-beat w_beat_done pulse for this; it is not needed and was dropped.

Ops (selected via compute_cfg.compute_op, same programming model as transpose):

Op Direction Ratio Width
COMPUTE_MXQUANT FP32 -> MXFP8 128B -> 33B per 32-elem block up to 1024b (StrbWidth 128)
COMPUTE_MXQUANT_FP16 FP16 -> MXFP8 64B -> 33B up to 512b (StrbWidth 64)
COMPUTE_MXDEQUANT MXFP8 -> FP32 33B -> 128B length must be 33k with k % StrbWidth == 0

MX block layout is inline [1B E8M0 scale][32B E5M2]. The quantizer is RNE with full subnormal, saturation and Inf/NaN handling, bit-exact with the viDMA ALCU reference except two deliberate corrections (mirrored in the DPI golden): the E8M0 scale saturates instead of wrapping for all-tiny blocks, and Inf/NaN lanes are excluded from the shared-scale scan so a block's finite lanes survive. The split normal/subnormal rounding bands are load-bearing; do not merge them.

Validation

Byte-exact against a DPI-C golden (test/idma_mxquant_dpi.c, pure integer pipeline) on Questa 2026.1 at every legal StrbWidth (DataWidth 32/64/128/256/512/1024):

  • tb_idma_mxquant: FP16 and FP32 sources, a genuine partial 33B tail beat, a 4K page-crossing write.
  • tb_idma_mxroundtrip: quant -> dequant back-to-back through one backend, byte-exact at both stages (the exact E5M2 identity); FP16 leg at 512b, FP32 leg at 1024b (256 blocks, 32KB legs).
  • tb_idma_mxrand: constrained-random campaign — 40 serialized transfers per run with a random op each (FP16/FP32 quant, dequant, plain unaligned copy), random block counts, writes biased across 4K boundaries, behind an AXI shim injecting 40% random per-channel stalls; canary bytes fence each destination.
  • tb_idma_mxneg: every compute guard assert proven to fire (11 cases, incl. the transpose single-beat and dequant length-overflow fences). Note this PR also adds +define+INC_ASSERT to the sim flow — without it ASSERT_NEVER compiles to nothing, so the legalizer guard asserts (including pre-existing ones) were inert in simulation.
  • tb_idma_transpose_b2b / tb_idma_transpose_nd unchanged and green (backward compatibility).
  • tb_idma_mxperf: steady-state throughput vs a plain-copy baseline on ideal memory — the bottleneck channel (quant: R, dequant: W) sustains 100% bus utilization at every StrbWidth, both for single transfers and for pipelined back-to-back streams (aggregate window across 8 transfers including the boundaries); the other channel sits at the compression ratio (e.g. 33/128). Three microarchitectural points make this hold: pack-buffer space only gates block-completing input beats, the dequant pack buffer carries one expanded-block window of refill-during-drain headroom, and lane-exact retire keeps tails sound across transfer boundaries. The pipelined quant stream is also checked byte-exact under the random-stall shim in tb_idma_mxrand.

Known scope / disclosures

  • Compute is functionally validated only on idma_backend_rw_axi. Size-changing compute on non-AXI src/dst is fenced with NOT IMPLEMENTED asserts (ComputeMxSrcProtocol / ComputeMxDstProtocol) until validated; OBI is the natural first candidate (its write ready is already per-beat). Transpose keeps its pre-existing protocol scope.
  • Back-to-back compute transfers with an identical config pipeline freely (verified at 100% utilization). A config change (op, FP16 flag, params) is serialized in hardware: the request is held at the legalizer input until the datapath drained, so software needs no ordering rule. Verified with a different-config transfer issued pipelined behind a quant stream (byte-exact); the sub-unit $fatal asserts remain as backstops.
  • Compute requires beat-aligned src/dst (asserted; all guard asserts covered by firing negative tests in tb_idma_mxneg).
  • ComputeOps defaults to '1: existing EnableCompute designs gain the MX engines on regeneration (area); set ComputeOps explicitly to opt out.
  • Synthesized standalone in a 12 nm technology at 1 GHz, timing met: quant 39.5k/71.5k cells (512b/1024b), dequant 37.4k/52.8k. The second commit restructures the engines for synthesis (hoisted completion datapath, generate-gated FP16, circular dequant buffer) — bit-exact and throughput-neutral, re-verified by the full suite. No in-system PnR data yet.
  • Validated at BufferDepth=3, MemSysDepth=0, 1D transfers only.

Copilot AI lite review requested due to automatic review settings August 6, 2026 06:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@DanielKellerM
DanielKellerM force-pushed the otf/size-changing-compute branch 4 times, most recently from fca1db0 to aa0bf44 Compare August 6, 2026 09:43
…ath)

Per-op source:dest byte ratios in idma_pkg program the legalizer write
length; the compute seam presents per-lane pack occupancy with full
strobes and retires lane-exactly against the byte lanes the write port
accepted (buffer_out_ready), so partial tails stay sound and identical-
config transfers pipeline back-to-back with zero inter-transfer bubbles.
Transpose keeps the per-write-transaction retire and its single-beat
burst contract (now asserted). Guard asserts fence every unsupported
configuration; all of them are covered by firing negative tests.
Hoist the block-completion datapath out of the unrolled element loop (64x
replicated at 1024b: 2048 quantizer sites -> 32), generate-gate the FP16
path above 512b, drop output registers that duplicated pack state, reduce
the subnormal shifter to its 3 reachable amounts, and rebuild the dequant
pack as a one-block-per-cycle expansion into a pow2 circular buffer (no
byte shifting). Bit-exact vs the DPI golden and throughput-neutral; the
naive engines did not finish synthesis in 14h, the restructured ones
close 1 GHz timing in a 12 nm technology (quant 39.5k/71.5k cells at
512b/1024b, dequant 37.4k/52.8k).
@DanielKellerM
DanielKellerM force-pushed the otf/size-changing-compute branch from 0b0abd3 to 4b5b54b Compare August 10, 2026 13:46
@DanielKellerM
DanielKellerM changed the base branch from backend/w-beat-done-retire to devel August 10, 2026 13:47
@DanielKellerM DanielKellerM reopened this Aug 10, 2026
Two deliberate corrections vs the viDMA ALCU reference, mirrored in the
DPI golden: the E8M0 scale saturates at [-128,127] instead of wrapping
(all-tiny blocks no longer flush), and Inf/NaN lanes no longer poison the
shared scale of their block's finite lanes. Directed all-tiny and
Inf/NaN-poisoned blocks added to the quant stimulus.
(length/33)*128 must fit tf_len_t; ComputeMxdequantLengthFits rejects the
rest, negative case 11 proves it fires. The overlap negative case is
retired: the request interlock (next commit) makes that stream legal.
A request whose compute config differs from the last accepted one is held
at the legalizer input until the datapath drained, so its reads can never
race a draining engine (gating at the write seam is provably too late:
the next transfer's read data enters the shared buffer and the still-
armed engine consumes it). Same-config streams pipeline unchanged at
100% utilization; the sub-unit asserts remain as backstops. The rand TB
issues a different-config transfer pipelined behind the quant stream and
checks it byte-exact.
@DanielKellerM
DanielKellerM merged commit 8e59f4e into devel Aug 11, 2026
13 checks passed
@DanielKellerM
DanielKellerM deleted the otf/size-changing-compute branch August 11, 2026 08:04
@DanielKellerM

Copy link
Copy Markdown
Collaborator Author

The merge was reverted off devel to fold in the wide-bus test coverage before landing. Superseded by #170, which re-adds this feature plus the audit fixes (block-scale clamp + Inf/NaN, dequant length fence, config-change interlock, wide-bus DW=256/512 coverage).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants