feat(vindex-factory): PREFLIGHT->RELEASE build driver (§7) - #196
Merged
Conversation
chrishayuk
force-pushed
the
feat/dec-funnel-v0-4
branch
from
July 30, 2026 22:50
521d7b5 to
a75d305
Compare
`larql-factory::build::run_build` orchestrates FETCH/EXTRACT/SLICE/ MANIFEST/VERIFY/PUBLISH/RELEASE as subprocess calls into `larql` itself, behind a CommandRunner trait (SubprocessRunner for real builds, MockRunner for tests). PUBLISH always goes private first; RELEASE only flips a repo public once every output has verified (§8's "nothing goes public unverified"). MIRROR and REGISTER stay external per the earlier scope decision — a BuildRecord JSON is the hand-off point, matching dec-bench's --output-file pattern. Extends larql-vindex's publish path with a private/visibility option and a new `larql hf visibility` command to make the two-phase publish possible. Wired in as `larql recipe build <FILE> [--scratch-dir DIR]`. Every new file at or above the 90% coverage floor.
estimate::http::tests::hf_token_reads_the_env_var and hf_token_treats_blank_as_absent both mutate the process-wide HF_TOKEN env var — a real race under parallel test execution (CI's test-windows-latest job just hit it). #[serial_test::serial] them, matching the pattern larql-vindex already uses for its own env-var tests.
Closes the last unpriced bandwidth surface. Real geometry rather than assumed (30 layers, kv_dim 2048 from num_key_value_heads 8 x head_dim 256, sliding_window 1024, explicit layer_types = 5 global / 25 sliding), against the measured 3.63 GB/token budget rather than a summed weight estimate — that sidesteps an unresolved question about whether the tied lm_head is read at f16 or q4, which would move a weight-based denominator by about a gigabyte. f16 KV would buy 3-8% of a token at chat-typical context. More decisively it cannot cause the decay it was proposed for: the sliding window already bounds 25 of 30 layers, so past ctx 1024 the byte slope collapses six-fold to 0.65 us/ctx-token against a recorded decay implying ~12. Bytes are a hard lower bound on time, so C10's original attribution to GQA compute was substantially right. The bytes thesis died for a couple of hours of arithmetic instead of a KV-quantisation build. It keeps value on the DEC-2 ledger as capacity, not speed: f32 costs 480 KB per context-token per client, ~1 GB per client at ctx 2048, which is what caps clients-per-box — so halving it roughly doubles client density. Registry: experiment c0-bandwidth-roofline (programme dec), run RUN-20260729-221048-00532, artifact 998.
The dispatcher's completion barrier was an unbounded `spin_loop()`. `worker_loop` received a spin -> yield -> park backoff when the pool went default-on; this barrier was left as a pure spin, so a dispatcher burns a whole core waiting on `completed` while the worker that owes the last chunk may not be able to get scheduled. With more runnable threads than cores — several concurrent dispatchers, the pool's own persistent workers, and a test harness running other tests alongside — every spinner holds its core for a full quantum and forward progress can stall for seconds. Escalates to `yield_now` and stays there. It must never park: nothing unparks a dispatcher, since workers signal only through the `completed` counter, so a parked dispatcher would sleep to its own timeout with no one to wake it. Yielding is what actually cedes the core to the worker being waited on. Fast path unchanged — decode-shaped dispatches complete in microseconds and never leave the spin phase, so the measured +28% is preserved by construction. SCOPE, stated honestly: this is hardening, NOT a fix for `stress_concurrent_realistic_decode_shape_no_corruption`. That test asserts buffer *corruption*, which means the barrier released before workers finished — a different and more serious failure than starvation, and one this change does not address. What this change does cover is the *hang* mode, which reproduces on pre-existing code (5 in 22 runs at --test-threads=8 on b39061f, before any of this session's spin-pool edits, so the defect is not a regression from the performance-core cap). An earlier hypothesis for the hang — strided chunk ownership defeating the hardware prefetcher — was wrong and killed by measurement: contiguous blocks did not close the gap, participant count did. The corruption mode needs its own investigation with the actual assertion output (chunk index, buffer, expected vs actual). The obvious candidates are all sound: static ownership means no participant can skip or double-run a chunk; num_chunks (32-320) always exceeds participant count so no participant owns zero chunks and the barrier cannot pass without every worker reporting; and the fetch_add(Release)/load(Acquire) pair does make writes visible. Tests: 797 larql-compute / 193 larql-compute-metal / 1309 larql-inference / 766 larql-kv / 1141 larql-vindex / 446 larql-models green; clippy clean.
…der spec GPT-OSS attention uses a learned per-head sink logit that competes in the softmax and is then discarded, so attention weights over real positions deliberately sum to less than one. Until now larql neither extracted nor applied it — along with all four projection biases — so 5 of 11 attention tensors per layer (120 tensors) were silently dropped and the forward pass was systematically wrong. The architecture module's header claimed both while every accessor returned None, and extraction never requested attention biases for ANY architecture, so Qwen/GPT-2/StarCoder2 declared them and lost them too. Measured sink magnitudes (mean +2.45, max +8.19, positive in every layer) make omitting them a systematic error rather than a rounding one. Now extracted (manifest 145 -> 265 entries) and applied on every path: - one shared `softmax_in_place` replacing four byte-identical copies - one shared sink resolver (`attention/sinks.rs`) - three Metal kernels (attn_fused, fused_attention, kv_append_attend_fused) Parity: NumPy reference < 1e-7 on CPU; CPU-reference < 1e-4 for Metal prefill and kv_append_attend_fused; and an exact analytic invariant for attn_fused — a sink rescales each head uniformly, a fingerprint no other bug reproduces. Also lands `docs/k3-funnel.md` v0.1, splitting the model-side K3 work out of DEC-6 into a three-rung adapter ladder: R1 GPT-OSS-20B (expert/format pipeline) -> R2 Kimi Linear 48B-A3B (the KDA stack, against a local reference implementation) -> R3 K3 as a delta, with phases P1-P6, gates G0/GA/GB/GC/GD and registry programme `k3`. The rationale is verification economics: R2 is K3's direct architectural ancestor and fits on the Mac, so Gate B becomes a layer-by-layer f32 diff instead of API-oracle triangulation. Known gap: `larql shannon score` (and therefore `shannon verify`) uses a dense-only FFN backend, so it cannot score any MoE model, GPT-OSS included. The end-to-end bits/char cost of the sink bug is therefore still unmeasured — and the same gap means the C6 wire-fidelity gate in DEC-1A cannot cover the 26B routed path either. Docs also carry the C0 bandwidth-roofline follow-ups: the C-ladder's acceptance criterion should be judged against x86's own attainable ceiling rather than Apple-Silicon per-core (measure it with the membw probe while the DEC-0.5 box is up), and DEC-2 gains client-side KV footprint as a capacity axis — f32 costs ~1 GB of cache per client at ctx 2048, which may bind the N-at-SLO table before tier bandwidth does. Tests: 797 larql-compute / 193 larql-compute-metal / 1309 larql-inference / 766 larql-kv / 1141 larql-vindex / 446 larql-models green; clippy clean.
The k-quant writers pad every row to the 256-element super-block
boundary (pad_rows_to_block), but the dequant caches
(kquant_ffn_layer{,_once}) and dequantize_matrix decoded assuming an
unpadded [rows, cols] layout — on models with hidden % 256 != 0
(GPT-OSS-20B 2880, Gemma 3 1B 1152) every row after the first read
shifted bytes: silent garbage, no diagnostic. One of three readers
(kquant_forward/walk_ffn.rs) had been fixed for down only; the same
class also hit the attention Q/K/V/O dequant in tensors.rs.
- larql-models: k_quant_padded_cols() — one shared stride formula for
the writer and both readers.
- larql-vindex: new ffn_store/kquant_decode.rs owns the row-padded
decode + feature-major down transpose; both caches rewired through
it.
- larql-inference: dequantize_matrix is padding-aware (callers pass
logical dims); three hand-rolled down special-cases deleted.
- Non-256-aligned (hidden=320) regression fixtures at three levels:
pure decode, cache path through load_interleaved_kquant, and
dequantize_matrix. Every prior Q4K fixture was 256-aligned, so the
suite structurally could not catch this class.
Review: docs/audits/vindex-walkffn-review-2026-07-30.md (H1).
- Overlay gate cache: a zero-length gate vector poisoned layer_gate_cache row alignment — nondeterministic slice panic or wrong KNN scores depending on HashMap iteration order (H3). The guard now bails to the safe slow path, and Vindexfile INSERT no longer stores empty gate vectors (slot claiming moved to the meta override). - Loader: out-of-range layer entries in index.json return VindexError::Parse instead of panicking (H4), in both load_vindex_with_range and synthesize_gate_from_q4k. - format/le_floats.rs: shared explicit-LE f32 slice codec replacing three unaligned &[u8]->&[f32] transmutes (UB) in patch/format.rs, quant/convert.rs, config/dtype.rs. Encoders are explicit LE — byte-identical on all supported targets, and the on-disk format is now portable (M4). - try_apply_patch: all-or-nothing patch application. Vector decode failures (including the base64 decoder's previously-silent truncated tail, now an error) no longer half-apply metadata (M5). apply_patch stays as an infallible wrapper that drops corrupt patches wholesale; migrating larql-server/larql-lql callers to try_apply_patch is a tracked follow-up. - Tombstones: Update resurrects a Deleted slot, matching Insert, so feature_meta() and gate_knn() agree in both directions; a pinned None meta is dropped when the Update carries no replacement (M6). - Deletion oversampling: BASE_KNN_OVERSAMPLE_FACTOR (2x) escalates 4x -> all-features when tombstones would under-fill top-k (M11); layers without deletions keep the single cheap query. - Delete the orphaned src/walk/ module — never declared, never compiled, stale WalkFfnConfig duplicate. Review: docs/audits/vindex-walkffn-review-2026-07-30.md.
Fixes (2026-07-30 review):
- The Q4_0 ladder step gated on supports_quant(Q4_K), admitting
CpuBackend (which advertises Q4-family matvec kernels but leaves
q4_matvec_pair_batch at the trait default) into an unwrap-on-None
panic on the first layer (H2). Gate on Q4_0 — the format the data
actually is — and probe the batch API by calling it, with a
C-kernel fallback on decline. First-ever tests for
interleaved_q4.rs, including exact CPU-vs-backendless equality.
- Overridden layers whose sparse walk fails now route directly to
the override-aware weights_fallback (ladder step 10, extracted)
instead of falling through the L1 cache and the override-blind
whole-layer paths (M7). Regression pins that the L1 cache can
never serve an overridden layer.
- activation_floor is live: effective_activation_floor() =
max(user floor, named ACTIVATION_NOISE_FLOOR), applied on all
three sparse accumulate loops; behavioral test (M3).
- Dispatch thresholds named in thresholds.rs: FULL_K_DENSITY 4/5
(helper doc now states the [80%,100%) K band routes dense — M1),
PARALLEL_DOWN_MIN_HITS, GATHER_MIN_FEATURES.
- selector:fallback dispatch-trace entries + selector_fallback_count()
so joint-selector A/B sweeps can no longer silently degrade to
GateOnly under the wrong label (M10).
- NaN contract unified: selection_weight_cmp_desc panics on NaN
(matching top_k_by_abs's pinned contract) across the four former
partial_cmp().unwrap_or(Equal) sites.
- Gather kernel caveat ("not yet correct for production down")
removed as stale: it dates from task #24's transposed-down
striding; task #25's hard feature-major-sidecar requirement
resolved it (validated vs dense, |err|/||ref|| ~ 6e-3). The
correctness condition and history are documented in
sparse_gather.rs.
Split (no behavior change; walk_ffn suite 60 -> 63 tests):
- mod.rs 1054 -> 396: builders.rs (constructors/builders),
timings.rs (PhaseTimingsHandle + env trace), dispatch_tests.rs.
mod.rs keeps the routing table, struct, ladder, trace accessors.
- sparse.rs 890 -> 386: sparse_gemv.rs (full-K path),
sparse_route.rs (hit/route selection), sparse_parallel.rs
(parallel Q4K down), sparse_gather.rs (gather kernel). sparse.rs
keeps entry, preflight, and the serial correctness baseline.
- CellRouter -> vindex/cell_router.rs, re-exported so existing use
paths are unchanged.
Review: docs/audits/vindex-walkffn-review-2026-07-30.md.
Full review record in docs/audits/vindex-walkffn-review-2026-07-30.md (two-reader subsystem review of larql-vindex and the walk-FFN engine, merged with an external strategic review and a kernel deep-dive; the four high-severity findings and two structural claims hand-verified). ROADMAP.md gains the 24-item programme under Codebase hardening. Items 1-12 and 14 landed same-day on this branch. Remaining: item 13 (the HNSW hot-path decision), Tier 2 (forward/forward_observed split, base+delta patched execution, runtime trace emission, execution planner, two-stage selection), and Tier 3 (walk-vs-dense parity suite, KnnStore unification, v1 conformance contract, doc drift, hygiene).
…tch path The kv_dispatch helpers ran attention -> clip -> run_ffn only, dropping the issue-#98 PLE/layer_scalar fix that lived solely in the legacy path NoCacheEngine uses, so the two exact baselines disagreed on Gemma-4-class archs. All four loops (sync/async x prefill/decode) now mirror the legacy order; MM prefill-from-hidden passes token_ids: None (vision rows have no token identities). Bit-exact parity tests added (prefill + 4 decode steps, sync and async). The E2B fixture is now seeded non-zero with layer_scalar 0.75 - the zero-filled fixture is what had masked the bug.
…-loop PLE sweep markov-rs/-codec: walk and executor decode paths read shape()[0] where the 2026-05-19 doubling-capacity migration made hot_len/cold_len canonical - windowed configs attended zero-padded phantom rows at shifted RoPE positions; the codec walk dropped the cold tier from attention permanently after the first overflow (cold_kv cleared, cold_encoded never consulted). Both fixed, plus a structural kv_recomputable_from_residuals() gate at every prefill entry enforcing the exactness preconditions (spec section 4: stateless norms, RoPE-only position, direct K/V projection - no MLA). turbo-quant: executor decode re-encoded the whole cache per step, decaying old-row norms ~x0.96/step until prompt K rows vanished; now append-only via a shared append_row. Codebooks were trained at sigma sqrt(2) too small; one N(0,1) Lloyd-Max table per bit-width scaled by 1/sqrt(d) lifts 4-bit cos 0.9908 -> 0.9954 and removes the d=64/32 fallback cliff. Non-power-of-two block dims are a typed error, not a mid-prefill panic. Contract relabelled bounded_KL -> codec_bounded_state in state-policy.md. PLE: 25 engine-internal per-layer loops skipped the PLE + layer_scalar tail on Gemma-4-class archs; all now apply it via a shared apply_ple_and_layer_scalar helper, with engine-vs-legacy parity tests on the seeded E2B fixture.
Every walk_ffn file now >= 90% line coverage (was: three split files at 29-82%, selector at 35%, and exact/full_mmap/interleaved at 0%). The load-bearing additions are the parity tests the 2026-07-30 review flagged as absent (roadmap item 20): - serial-vs-parallel parity at >= PARALLEL_DOWN_MIN_HITS asserting output via per-feature linearity (sum of two serial half-routes), with the parallel path's all-zero activation pinned so the Tier-2 forward_observed fix flips it loudly. - gather-vs-serial parity on a constructed feature-major down sidecar (first CI-resident correctness check for the gather kernel; tolerance derives from the two independent down quantisations). - walk-vs-dense parity against WeightFfn for the full-K gemv path and the previously-untested dense trio (exact, full_mmap, interleaved), incl. non-gated bias arms and walk-only fallback. - all six FeatureSelector variants through joint_gate_knn with raw-score and criterion-ranking assertions; norm caches bit-exact vs the dequant cache; fallback-counting pins. - interleaved_q4 batch-branch == C-kernel-branch bit-equality via a mock batch backend, incl. the vecmat-decline fallback. - sparse override arms: zero up/down overrides remove exactly the overridden feature's contribution, rest bit-equal. Fixture support: parameterised q4k test weights (Q4K_TEST_INTER_WIDE 768 to reach the parallel branch), model-gate vindex (enables walk-vs-dense parity), down/up sidecar attachment helpers. The new interleaved-vs-dense parity test immediately caught a latent fixture bug — attach_interleaved_f32_to_test_vindex wrote down untransposed — fixed; no production code changed.
…core) The deep-idle park was fake: park_timeout(50us) woke each parked worker 20k times/sec, and the yield bridge ran 128k sched_yield calls per idle transition. Measured 4.27 CPU-s per 5s of idle (85% of a core, ~836k involuntary context switches, 11 workers) on the process-lifetime global() pool. Now: yield bridge cut to 64 iterations, park backstop 50us -> 1s, Drop unparks workers. After: 0.042 CPU-s per 5s idle (0.8%), 51 context switches; wake-after-idle 80us -> 59us; decode at parity on quiet M3 Max (fixed 35.9/36.3/35.9 vs baseline 35.1/34.9/35.8 tok/s, 26B --cpu -n 50, interleaved pairs). Also: gate the two production-scale stress repros behind heavy_tests (default --lib suite 112s -> 3.5s; run via make larql-compute-test-integration) and set LARQL_SPIN_POOL=0 in CI — a 2-core runner has no core to spin on; the both_arms override tests keep both scheduler arms covered regardless.
…nostics In-flight work across two streams, committed to move the branch: - larql-models: tensor_keys module (qk_norm / moe_experts / mxfp4_dequantised / attn_bias), architecture detect updates across deepseek/gemma/gpt2/gpt-oss/granite/olmoe/qwen/starcoder2, gguf orient + safetensors loading, mxfp4 quant - larql-compute: ffn expert_weight tracing (serde_json promoted to a lib dependency), MoE reference parity test + generator script - larql-compute-metal: attention sinks stage - larql-factory: build exec stage, build tests, test_support, card/estimate/recipe/build_id revisions - larql-cli: moe_locality diagnostics - docs: k3-funnel spec, ROADMAP updates
…ended Roadmap item 13 resolved — the audit's claim was inverted. gate_walk was placed FIRST in the walk's selection chain on 2026-04-04 with the comment "falls back to gate_knn (HNSW or brute)", but the GateLookup trait default (None, "override in VectorIndex") was never overridden: through &dyn GateIndex every walk selection fell to gate_knn, so enable_hnsw() silently leaked APPROXIMATE, recall-80-95% selection into walk numerics for four months. All fidelity work (full-K parity KL=0.0000, FeatureSelector experiments) assumes exact top-K; the docs' evidence (no HNSW speedup at 10K dense features; wins only for wide MoE) supports exact-first for the walk. - VectorIndex::gate_walk delegation shim to the exact gemv, with a populated-index delegation test (an empty-index test cannot distinguish delegation from the default — how this survived). - PatchedVindex::gate_walk declines on layers with gate overrides or tombstones so the overlay-aware merge stays authoritative (3 pins). - enable_hnsw() doc maps exactly which paths consult HNSW (gate_knn, gate_knn_expert) and which never will, with the leak window noted. - Pins: gate_walk_ignores_hnsw_toggle; walk_ffn_sparse_hot_path_ignores_enable_hnsw (fails pre-shim). Behavior change only when enable_hnsw() is on AND the gate is dense-resolvable: walk selection becomes exact, as documented. Default runs byte-identical; Q4K-only gates and patched layers keep gate_knn_q4/gate_knn, preserving the wide-MoE HNSW win.
The larql-models and larql-compute coverage gates merge the lib-test and
integration-binary instantiations of each crate, so code exercised only
by in-crate unit tests still reads uncovered from the integration
binary's copy. Three files sat under their floors with green tests:
- models/config.rs + olmoe.rs: add trait-default and OLMoE sweeps both
in-crate (config::tests, olmoe::tests) and through the public API in
tests/test_architectures.rs; gate passes at 90.40% total, all files
at floor.
- compute expert_weight/trace.rs: split open_sink/record_into out of
the OnceLock/global wrappers so both failure arms are unit-testable,
and drive the env-resolved sink end-to-end from two one-test
integration binaries (tests/test_moe_route_trace_{on,off}.rs — one
test per binary because the sink resolves once per process and
set_var needs a quiet process). The residual instantiation-union gap
is debt-baselined at 76 in coverage-policy.json with rationale;
merged lcov of the same profdata reads 85%+.
- serde_json returns to dev-dependencies: trace.rs hand-writes its
JSON lines by design and only the tests parse them back.
- cargo fmt debt across 13 files committed unformatted in a75d305
(walk_ffn, kv engines, gguf orient, vindex patch overlays) — pure
rustfmt of the committed content, verified against the formatted
blobs; no semantic change.
…nce boilerplate §4.7.6 concluded larql's 3.221 bits/token was "the only figure in that table that is even plausible" and therefore that the reference was the suspect. Both halves are withdrawn: the comparison was between two different texts. The first 384 bytes of data/gutenberg/frankenstein.txt are the Project Gutenberg licence header, not the novel. 3.221 is a boilerplate score. At 2048 bytes larql gives 8.338 against HF bf16's 8.028 — the engines agree to 4%, not 2.5x apart. Adds the control §4.7.6 never had: on the same 2048 bytes through the same ExpertWeightFfn tier, Qwen3-30B-A3B scores 1.400 bits/token. A healthy figure from the same code path exonerates the scorer, so OLMoE's gap is a real bug (strengthened, not withdrawn) while GPT-OSS's forward is simply unvalidated. GPT-OSS is demoted to a secondary rung for any forward-pass metric; Qwen3-30B-A3B becomes the working R1-class reference. §4.7.8 records finding 5's third recurrence and why the trait default was the mechanism rather than the symptom. The remaining structural fix — no behavioural default, plus a lint for fixtures omitting fields the policy branches on — is carried explicitly rather than bundled here. Method note, now the third measurement-shaped error in §4.7: a corpus slice is a fixture. Any bits/char figure must state its byte range, and both sides of a comparison must use the same one.
…alsified
Poses the MoE redundancy question over the EXPERT axis: reshape the bank to
[E, d_out*d_in], one flattened expert per row, so rank caps at E and the
statistic can return "independent". Stacking to [E*d_out, d_in] instead caps
rank at the ambient d_in, which boxes the observable ratio into roughly [1,2]
and reports "redundant" by construction — the same failure as reading a
near-zero cosine between two flattened weight matrices as orthogonality.
Measures both compression routes, because they fail independently and testing
only one would overclaim:
cross-expert (shared atoms) 99% energy needs rank 126-127 of 128; at a 5x
budget only 21-27% of energy survives against a
19.5% flat null
per-expert (low rank) 99% energy needs rank 715-723 of 768, against a
breakeven of 559 — negative compression
Qwen3-30B-A3B, 3 projections x 5 layers. Layer 0 is the most structured;
layers 12+ are near-random, so depth offers no escape either. --random-control
measures the flat null rather than trusting the derivation (19.8% vs 19.5%
analytic).
Not closed: this is linear redundancy in raw weight space, so two experts equal
up to a hidden-unit permutation look independent. Canonicalising before
re-measuring is the one follow-up that could overturn it.
Also names the magic values in moe-locality per the standing rule: block sizes,
per-layer block, shuffle seed and the empty-cell placeholder become consts, and
the script samples layers from the model's actual depth instead of a hardcoded
Qwen3-shaped list.
… dense path
Roadmap item 16 (Tier 2). For a layer with patched slots P:
y_patched = y_base + sum_{i in P} [a_i_new(x)*d_i_new - a_i_old(x)*d_i_old]
is exact for gated FFNs, dropping override cost from O(N) forced-sparse
to O(|P|) extra dots on the fast dense base. Inserts are pure
additions, tombstones pure subtractions. Retires the
override-forces-sparse cliff whenever preconditions hold.
Exactness by construction, each condition enforced by declining
(None -> the pre-existing sparse/fallback override routes):
- OLD terms from the SAME quantised bytes the dense base consumed
(unified ffn_row_dot/scaled_add dispatch, never f32 twins), with
storage-coherence checks so the base ladder and row dispatch can
never read different bytes (exactly one of Q4K/f32-interleaved, no
FP4/Q4_0/full-mmap/feature-major diversions).
- Old down rows from the feature-major Q4K sidecar or native f32
interleaved (the transposed interleaved down is not row-gatherable).
- Dense base via forward_unpatched_whole_layer — the same ladder body
(steps 4-9) unpatched routing runs, so base numerics are identical.
- Half-covered slots (gate without up row) and wrong-width overrides
decline rather than subtract a wrong old term.
larql-vindex: OverrideSlot + PatchOverrides::override_slots_at
(exhaustive per-layer slot enumeration incl. tombstone flag; default
None so point-query-only indexes decline the path);
PatchedVindex implementation.
Tests (base_delta_tests.rs): parity vs the sparse override walk for
up/down/gate-override, insert, and tombstone cases; the no-op-edit pin
(override writing back the ORIGINAL row bytes reproduces the unpatched
base output — the same-bytes condition made testable); trace pins for
override:base_delta and every decline route; unpatched-layer
byte-identical pin.
…atch A worker whose block in dispatch N was empty is not needed by N's completion barrier, so N can finish and N+1 re-publish the task slot while that worker sits between its epoch read and its slot read. It then ran N+1's chunks attributed to N and again on re-observing the epoch - over-counting 'completed', releasing N+1's barrier early, and dropping the closure while another participant still executed it (use-after-free; SIGSEGV under coverage instrumentation where the window is widest). The task slot is now seqlock-guarded and generation-stamped: a participant runs only the dispatch whose epoch it observed. Pinned by an alternating 1-wide/7-wide exactly-once stress test.
unlimited-context: window checkpoints read the backend cache at an absolute index (every window after the first checkpointed the wrong row on Metal); dispatch prefill now segments the prompt into windows like the CPU path; the W10-disabled decode-after-close panic re-allocates the window shadow; lost window shadows are typed errors instead of silent context loss; window_size=0 rejected. boundary-kv: prefill emits a frame per interior chunk boundary (spec 6.1) via chunked prefill that keeps final state bit-identical; re-prefill starts a new generation chain. boundary-per-layer: decode failures no longer destroy the store (dense-walk fallback now reachable); dispatch cold-tier recompute uses the dequant scratch (panicked on real Q4K); cold-kv extension is atomic; stale kv_handle cleared on plain prefill. standard: window honored on the quant path (coarse declined when windowed); per-layer quant fallback builds a real WalkFfn instead of running the harness's NullFfn; recorded prefill dispatch mode replaces the handles.len()==1 heuristic (panicked on 1-layer models). apollo: BOS is a parameter with structural fallback (never hardcoded 2); injection-layer misconfiguration is a fail-closed prefill error; deterministic routing tie-break; store/npy integrity validation. gate telemetry: hard-reject path computes boundary_fragile instead of stamping false.
The vindexfile/mod.rs coverage floor (72%) failed CI at 71.50%: every existing build test stopped at an error arm before the base loaded, so the whole directive loop — INSERT/DELETE/LABELS/EXPOSE, the build history, and both bake-down branches — was untested. Save a tiny synthetic base through the real save path and run the loop end-to-end (plus the no-overrides clone branch). File now measures 96.33%.
b813c01 landed it unformatted; the larql-inference fmt job fails the whole workflow in 20 seconds before anything compiles.
…re races `forward_with_activation` returned `(out, activation)` and every backend that had nothing real to report fabricated the second element as a clone of the first. `forward_observed` replaces it with `FfnActivations`, which can say Dense, emit sparse pairs, or say Absent — never invent zeros. The eight stub impls across larql-kv, larql-server and larql-inference drop their overrides entirely: the trait default returns `FfnActivations::unobserved()`, which is what their call sites now assert with `obs.is_absent()`. Also fixes two instances of the same test-infrastructure bug. `ResidualCapture` and `StageCapture` each pointed a process-global env var at a per-call tempdir with no synchronisation, while `cargo test` runs each suite's cases as threads in one process. Two concurrent captures redirected each other mid-run, so a case either found its own directory empty or read dumps a sibling had written *for a different model*. That produced `parity_mistral_7b_prefill` failing deterministically, `parity_llama2_7b_prefill` failing intermittently with cos=0.0056 — a cross-model comparison that looked exactly like a catastrophic kernel regression — and `stage_bisect_mistral_7b` failing while its three siblings passed. Both helpers now share one lock, since two different helpers racing corrupts a capture as thoroughly as two calls to the same one. The existing tests could not catch this: they are single-threaded, and the hazard only exists under concurrency. The new one spawns 8 callers and fails with the lock removed (5 of 8 observed a sibling's directory). The previous code documented the hazard — "racing `cargo test --test-threads=N` would stomp; tests in this suite run with `--test-threads=1` upstream" — but nothing enforced it and the default invocation violates it. Metal parity suite: 4/4, twice. Workspace: 6479 lib tests green, fmt and clippy clean.
…ed NaN `full_pipeline_seq1_produces_nonzero` failed about one run in ten of `test_metal_shaders`, returning an all-NaN hidden state. It passed in isolation and never reproduced under `--test-threads=1`. Building a backend compiles the whole shader library from source and creates several dozen pipeline state objects, all against the shared `Device::system_default()`. That test binary calls `MetalBackend::new()` once per test, so it builds 54 backends concurrently; doing so intermittently yields a backend whose pipelines compute garbage. Serialising construction fixes it: 0 failures in 85 runs, against a ~10% per-run rate before (P(fluke) ~ 1e-4). Two hypotheses were tested and refuted first, so the lock is not a guess at a symptom. Zeroing every scratch buffer on hand-out did not help, ruling out an uninitialised-memory read — `bufs.output()` recycles non-zeroed buffers, which was the obvious suspect. And the failure never appears serially, ruling out anything input- or dimension-dependent. The lock is held across the whole function rather than just the library compile: pipeline creation shares the same device, and narrowing it would be re-guessing at which half is unsafe. Cost is nil where it matters — production builds one backend per process, so it is uncontended outside test binaries. Also splits the assertion that hid this. `v.abs() > 1e-6` is false for NaN, so an all-NaN result failed with "output should be nonzero", which sent the first look hunting for a zeroed buffer when every element was NaN. Finiteness and magnitude are now separate assertions with distinct messages. Adds `concurrently_constructed_backends_all_compute_correctly`: 8 backends built on 8 threads, each running the same deterministic matmul, asserting every result is finite and agrees with its peers. Being a race, it reproduces probabilistically — a net that catches removal of the lock across repeated CI runs, not a single-run proof. Its doc comment says so.
Completes roadmap item 15 (the bulk landed in 4e592c6): base_delta observation suite (post-patch slot activations, tombstone-observes-0, decline branches — base_delta.rs 20%->95.5% line coverage), dispatch and sparse observation pins, WeightFfn dense-observation test, no-shard Absent pins for the sharded backend, roadmap tick. Split summary now fully landed: forward is allocation-free by construction (single forward_routed body, Observe::{Skip,Record}); FfnActivations::{Dense,Sparse,Absent} replaces fabricated zeros — the parallel Q4K path reports real activations (pinned parity test flipped from all-zeros to bit-equality with serial), L1-cache hits recompute under observation instead of serving zeros, and forward_with_activation is deleted. All four crate suites green, workspace clippy clean. Known debt: remote http/sharded transport paths remain under-covered pending a mock-server harness.
Roadmap item 17. take_trace() used to record residuals during the
forward and RE-RUN plain gate_knn afterwards — the recorded trace
could be a different feature set than the walk executed (ignoring
selector/pools/cell-router, and racing index mutations), and carried
gate scores, not contributions. Traces now emit from the executed
kernels at execution time, riding the item-15 Observe seam — no
parallel capture channel.
Schema (walk_ffn/trace.rs, chuk-introspect vocabulary, no dep):
TracedFeature { feature, rank, gate_score?, up_score?, activation,
down_row_norm? } per (position, layer, feature);
LayerTraceRecord { layer, position, path, residual_delta_norm,
features } per (position, layer). Dense whole-layer paths emit
summaries with empty features — declined, never faked. down_row_norm
is peeked from the selector's lazy cache only if already built
(pinned: tracing never triggers a norm pass). Target-logit
contribution needs lm_head access — documented follow-up.
take_trace rebuilds WalkTrace from runtime records (hits = EXECUTED
features, historic last-position semantics); take_runtime_trace gives
full fidelity. WalkHit extended additively. Consumers migrated:
vindex gate_knn dispatch + overlay walk, infer_patched (documented as
post-hoc view), lql infer_trace, cli walk_cmd, python PyWalkHit.
Proof test (trace_tests.rs): a DecoyKnnIndex whose gate_knn returns a
disjoint set while a pool route executes — the trace must equal the
executed pool route with real activations/ranks; the old take_trace
structurally reports the decoy. Suites green across inference (1385),
vindex (1202), compute, server, lql, cli; workspace clippy clean;
trace.rs 100% line coverage.
Also removes an untracked orphan: crates/larql-compute/examples/ bench_mxfp4_dequant.rs was a first draft written before the crate split mattered. larql-compute deliberately cannot see larql-compute-metal (ADR-019: GPU backends are sibling crates), so a bench comparing CPU against Metal has to live in larql-compute-metal, which is where the surviving copy is. Leaving both would have left a stale duplicate that could not measure what its own docs claimed.
…tests Roadmap item 24 — the programme's final item, done in priority order with the file-split remainder documented rather than rushed. - Generic-engine rule: English word lists, pattern classes, and the Wikidata vocabulary are OUT of engine code into data/ (entity_patterns.json, stop_words.json, wikidata_categories.json) behind one resolver (clustering/data_files.rs: LARQL_DATA_DIR -> workspace data/, never cwd; explicit-path loaders; loud minimal fallbacks). The two new vocabulary files are force-added past the data/ gitignore: unlike the script-fetched wikidata file they have no generator — they ARE the extracted data. The identical cwd-probe in pair_matching/database.rs fixed with the same resolver. Thresholds named (MIN_CATEGORY_SIMILARITY, PATTERN_MATCH_FRACTION — doc/code disagreement resolved in favour of code and pinned). - Activation dispatch: 27 live copies (audit counted 8) of the GeluTanh-vs-SiLU match unified on Activation::uses_gelu_tanh_gate_up() in larql-models — exhaustive, wildcard-free, so a new activation variant is a compile error, not a silent SiLU. Two drifted copies (weight.rs, cached.rs matched GeluTanh only, dropping exact Gelu to SiLU) fixed by the dedupe. - 16384 -> 10240: ffn-cache.md and capture.rs corrected against gemma3.rs; ffn-cache's stale top_k*2 gate claim fixed to the FULL_K_DENSITY 4/5 truth. - FFN_GATE/FFN_UP constants added beside FFN_DOWN (vindex pub + compute mirror with compile-time equality pins); all bare 0/1/2 component indices unified. - First-ever tests for the three untested load-bearing files: hnsw.rs 97.3% (recall@10=0.97 vs brute), mutate/mod.rs 96.0%, write_f32.rs 92.6%. NEW FINDING pinned honestly: HNSW level-0 fragments beyond ~64 nodes (recall@10=0.16 at n=200) — naive add_connection eviction orphans nodes; recorded as a standing follow-up since gate-KNN consumers at 10K+ features may be silently degraded. - Also records the down_meta.rs -> down_meta/ deletion the item-22 commit missed. Deferred (documented in the tick): the one-concept file splits for download/mod.rs (1329), overlay.rs (1071), convert.rs (653). Programme status: ALL 24 ITEMS CLOSED. Standing follow-ups listed in the audit header: try_apply_patch migration, remote transport coverage, logit-contribution traces, thresholds -> WalkFfnConfig, HNSW fragmentation, file-split remainder.
`q4k_grouped_experts` declared `Q4KG_ROWS_PER_TG`, a name `q4kf_qkv_proj` already owns. Every shader is concatenated into one Metal program source, so the second declaration is a redefinition and the entire program fails to compile — not just the new kernel. That took 98 of 216 lib tests with it, every one of them reporting the generic "Metal device available on test host" panic rather than the shader error, so the cause sat two layers below the symptom. Renamed the file's constants to `Q4KGE_`, matching the `Q6KG_` prefix the sibling `q6k_grouped_experts` already uses. A scan across all shaders confirms this was the only colliding constant.
CI runs `cargo fmt -p larql-compute-metal -- --check` before clippy and tests, so the format failure aborted the job and hid everything behind it. Eight files were unformatted; clippy then reported five errors under -D warnings: - manual `%` divisibility in grouped_experts.rs (x2) and mxfp4.rs - needless_range_loop zeroing a scale row -> `fill(0)` - redundant closure around `quantize_q4_k` - two unused imports in bench_grouped_experts `mxfp4_matmul_small_block` keeps its eight arguments behind an explicit allow: two weight planes, the activation, three shape scalars and the layout are the kernel's real signature, and bundling them into a struct would hide the shape contract the guards check one by one.
Three tests added in ea18d1e asserted that `data/wikidata_categories.json` and `data/wikidata_triples.json` exist "in a checkout". They do not — .gitignore excludes `data/` — so they passed only on a machine that happened to carry the generated artifacts, and failed in CI. Two different fixes, because the two files are not the same kind of thing: - wikidata_categories.json (16 KB) is a vocabulary, the same class as entity_patterns.json and stop_words.json, which ea18d1e already force-added past the ignore. Tracked now, for the same reason. - the reference databases are 2.2 MB of generated triples and do not belong in the tree. `load_reference_databases` grows a resolver seam mirroring the existing `resolve_data_file_with_env` pattern, so the resolver -> loader -> struct wiring is covered from tempdir fixtures, including that each filename reaches its own loader. Absence is now covered explicitly too, which is the path CI actually takes. The cwd-independence guard moves to data_files.rs and asserts the workspace dir is absolute. That was the real 2026-07-30 finding, and it holds whether or not any data file is present.
Five new pure modules plus the reporting to surface them.
Format floor, closed in two halves:
- `serving_format` settles the FIXED-width question by counting rather
than searching. An MXFP4 group reconstructs at most 15 distinct values
(16 LUT entries less the +0/-0 collapse), so ceil(log2 15) = 4 payload
bits — exactly what MXFP4 already spends. `formats` reports it and
takes neither a checkpoint nor the network, so `run` dispatches it
before any geometry fetch.
- `symbol_census` prices the remaining loophole, a VARIABLE-rate exact
code, against a measured distribution read by range request. It
carries its own trap guard: plug-in entropy over a B-weight tile is
biased low by about (K-1)/(2*B*ln2) however uniform the source is, and
that bias scales as exactly 1/B. Real local concentration would not,
so the diagnostic is the scaling, not the gap.
`scenario` exists because of a specific misquote: the frontier's "8 tok/s
needs ~3.8 payload bits" row was read beside the composed 5.77 tok/s
ceiling as though they answered the same question. They do not — frontier
is a requirement on the dense path at eta 1.00 with routed bytes credited;
ceilings is a ledger at measured per-class eta 0.64-0.89. The banner now
prints the premises, and `exact_floor_tok_s` derives the bar rather than
quoting a literal that could go stale.
Two KDA reads, both blockers rather than curiosities:
- `kda_a_log` — whether A_log parameterises decay per head or per
channel. Two self-consistent readings exist, each breaking exactly one
tensor, and the wrong one corrupts the recurrent state in all 69 KDA
layers while still producing plausible text.
- `kda_graph` — which projections share an input, since a folded
rotation is only cheap if several large matrices read the same tensor.
The premise that all five share one hidden state is wrong by one, in
the expensive direction.
`fetch` gains `packed_expert_tensors`; `classes` gains GROUPED_ROUTED_ETA
at the 0.89 K3a measured.
…outes? The 2026-05-29 scissors measured gate-KNN sparse at ~3x slower than dense because selection pays a full O(N*d) gate projection. Every candidate router — HNSW/ANN, LSH, static pools — is an attempt to shrink that selection term, so testing them one at a time prices the levers before knowing whether any lever can win. R4 sets the bytes each lever targets to zero and asks the only question that gates the thread: with perfect routes supplied for free, is sparse execution cheaper than dense at all? Three arms — `dense`, `exact` (full production gate chain), `oracle` (precomputed top-K, zero timed search). `oracle` vs `dense` is the decision; `exact` vs `oracle` bounds the most any router could ever recover. Provisional battery output included: 34 layers, 10240 features/layer, decode shape, warm = mean of 30 after 5.
The key is `(data.as_ptr(), data.len())`, which is sound only while the slice points into an allocation that outlives the cache and never changes — mmap'd weight data, the case it was built for. Any caller passing a temporary is a silent correctness bug: the allocator may hand the same address to a later, different temporary of equal length, and the cache returns the earlier buffer. Found from the K2 tournament, which built two 64-byte expert-offset tables as temporaries. The second `get_bytes` returned the first table's buffer, so every arm read slots 1..15 at the wrong offsets. Slot 0's offset is 0 under both tables, so a slot-0 oracle passed while the outputs were wrong; the lesson worth keeping is to probe a slot whose offset is non-zero, and to keep a full cross-arm comparison beside any narrow oracle. A content compare on every hit is not affordable in release — it is a full pass over the weights, as expensive as the matvec. So this states the contract in the doc, verifies it under debug_assert where the cost does not matter, and adds `uncached_bytes` for data whose address says nothing about its content.
…not the skeleton
Seven crossed arms at the real K3 expert shape [3584, 3072] x 16 grouped
experts, batched 8/cmdbuf, cold. Crossed so no comparison confounds two
changes: B/C/D/G share a byte-identical artifact, so those deltas are pure
decode; only A changes the bytes, so B-A is the layout effect. A three-arm
tournament without B would have been uninterpretable.
ms eta
A split / 16-LUT, 4.25 bpw 0.3721 0.685
B inter / 16-LUT 0.3782 0.644
C inter / 256-entry pair-LUT 0.4158 0.586
D inter / 8-entry mag+sign 0.3368 0.724
G inter / table-free bit math 0.5374 0.454
E probe / trivial affine 0.2627 0.928
F probe / weights only, no X 0.2575 0.947
D wins. The ordering is monotone in table pressure — 8 beats 16 beats 256
— and constructing the f32 from the e2m1 bit fields with no table at all
is the worst of the lot. Interleaving is a small consistent LOSS despite
reading 4.4% fewer bytes, so at this shape the kernel is not bandwidth-
bound and the byte saving cannot pay for itself.
The two probes are why this can stop rather than continue. E keeps every
byte read and every X read but swaps the non-uniform fp4 grid for an
affine map; F drops the X gather entirely. That decomposes D into
skeleton 76%, fp4 decode 22%, X gather 2%. The skeleton streams weights at
eta 0.947-0.965, so there is no deficiency to transplant from the Q6_K
kernel and the whole distance from 0.89 is the value set.
Do not read eta across containers. Q6_K reads 1.62x more bytes per weight,
so it becomes bandwidth-bound and reports a high eta more easily. The
comparable quantity is bpw/eta: Q6_K 7.374 against MXFP4-D 5.611, so D is
1.31x faster on the routed class *despite* the lower eta.
Probes are excluded from the correctness gate by an explicit `checked`
flag, since they deliberately compute something else. Every scoring arm
reproduces a CPU oracle to 6.2e-8. The packer emits both layouts from one
quantisation so the arms encode identical values; note the superblock
spread clamp must move exponents UP, as clamping down pushes amax/scale
past the fp4 range and silently saturates.
Benchmarked layout is 4.0625 bpw — fixed 1-bit delta, the 97.12% common
path. The whole-checkpoint exact format is 4.06731 bpw; the difference is
the 2-bit delta for the 2.88% wide tail plus a mode bit, not any global
scale table.
The coverage policy gate had never run on this branch — the format check
aborted the metal job before it, so three files drifted below the 90%
floor while the K3a grouped-expert and MXFP4 work landed:
diag/kernel_profile.rs 74.69%
trait_impl/grouped_experts.rs 87.33%
trait_impl/mxfp4.rs 88.26%
TOTAL 95.38% (min 96%)
What was uncovered was not the kernels but everything around them: both
error `Display` impls in full, the `x`-too-short guards on all three
entry points, the entire `q4k_grouped_experts` guard prologue, and two
whole profiler drivers.
- `profile_shape_census` and `profile_grouped_experts` get smoke tests
at minimum params, matching the existing `profile_all` pattern and
skipping on hosts without a Metal device. The grouped-expert driver
is the measurement the eta 0.64 -> 0.89 claim rests on, so it should
not only ever run by hand from the bench example.
- `measure_batched` gets a pure timing test asserting the result is per
LAYER and that warmup passes are excluded. Those are the two ways it
can be wrong, and getting them wrong is what undercounted q6k_matvec
by 4x in 2026-04-28.
- the q4k path is checked against the same four shape faults as q6k,
since it carries its own prologue and a fault caught on the q6k path
proves nothing about it.
The PerSlot guard test pins the failure mode explicitly: an x sized for
Shared is a legal buffer, so without the guard PerSlot reads the next
slot's activation off the end and returns a plausible wrong answer.
…ddress `profile_all` and `diag_shape_census` build their cold-rotate weights as temporaries inside a `map` and hand each to `get_bytes`. That cache keys on `(ptr, len)`, so every iteration's `Vec` — same size, freed before the next allocation — landed on one address and returned the first buffer. Probed directly: 8 handles, 1 distinct address. The consequence is a measurement bias, not a wrong answer: each "cold" call re-read a single buffer instead of cycling 172 MB, so anything small enough to sit partly in L2 reported inflated bandwidth. Re-running the shape census against a genuine rotation moves gemma down q6k 0.85 -> 0.80 and K3 latent-up 0.87 -> 0.82, while the large shapes barely move (K3 KDA attention q6k stays 0.89) — exactly the shape-dependence you would expect if the old numbers were partially cache-resident. Composed, the ceiling moves 3.70 -> 3.68, so the ledger is robust to it even though the harness was not. The page-aligned path made it worse than a stale read: `get_bytes` hands Metal the caller's pointer with no copy, so the buffer outlived the `Vec` it aliased. `uncached_bytes` copies, so the temporary may die and each rotation slot is a distinct allocation. Regression test asserts 8 slots stay 8 addresses. Per-class eta constants in `k3_ledger::classes` are NOT updated here — one re-run is a data point, not a re-bank.
`diag/mxfp4_layout.rs` landed at 41.03% line coverage, below the 90% floor, and dragged TOTAL to 95.54% against a 96% minimum. The helpers around it were tested; `race` — the ~200-line driver that is the whole point of the module — was not run at all. Adds a smoke test at minimum params, matching the `profile_grouped_experts` pattern and skipping without a Metal device. It asserts the arms produce finite positive rates, and that every *checked* arm still agrees with arm A — `first_bad` is None. Ceiling probes are excluded from the agreement check because they deliberately compute something other than the matvec. Shape is chosen so the row-remainder path runs: N=260 is not a multiple of the tile. This is the test that would catch an arm silently drifting out of agreement. Until now nothing exercised the arms against each other except running the bench example by hand.
…-point oracle R7 comes out of the K2 tournament. At K3's expert shape Q6_K reaches eta 0.89 and MXFP4 arm D only 0.724, yet D is 1.31x faster — a denser container is easier to make bandwidth-bound, so it reports a flattering eta for the same wall-clock work. Ceiling probes decomposed arm D as skeleton 76% / fp4 decode 22% / X gather 2%, with the skeleton streaming at 0.947-0.965, so the low eta was the price of the container and there was no kernel defect to chase. Kept separate from R6 deliberately: R6 is logical-vs-physical reuse, R7 is that the denominator's representation changed. Sharing an identifier would let one retire the other. R8 generalises the slot-0 miss. A cache-aliasing bug made every tournament arm read slots 1-15 from the wrong offset table, and the oracle passed because it only checked slot 0 — where every offset convention yields zero. The same shape has now appeared four times: CN-10's rank-1 first token, shared-vs-per-slot input diverging only from slot 1, and a square A_log fixture where per-head and per-channel coincide on the diagonal. The rule is to choose the probe point after enumerating the plausible wrong implementations, not before.
…not support Support and mass are different questions and the ladder has conflated them once already. Support asks which symbols a session touches; mass asks how often. A session can touch the whole bank and still take most of its events from a handful of symbols, and only the second quantity prices a cache. `selection_trace` keeps "symbol" deliberately unnamed: an MoE expert id at DEC-8.4, an FFN feature-row id at DEC-8.1. One instrument therefore grades the resident-expert bet and the compact-subexpert bet without learning which, and without knowing anything about K3. Sessions stay separable rather than pooled at construction, because the adaptive estimators condition on them. `freqmass` scores every arm on the same held-out events, varying only the key used to rank symbols for residency: `key = lambda*pooled_prior(leave-one-out) + (1-lambda)*session_fit`. lambda=1 is the static slice, lambda=0 the causal adaptive cache, the interior a shrinkage cache, plus an unrealisable oracle bound. The null arm — eval half redrawn from the pooled marginal, session identity destroyed — is load-bearing rather than decoration. A short fit window concentrates by counting noise alone, so the locality signal is oracle minus null; oracle alone overstates it. Scoring adaptation without a null is exactly how a finite-sample artifact becomes a 3x lever. Exposed as `larql k3-ledger freq-mass`, dispatched before geometry fetch since it reads a local capture pool and must keep working on non-K3 traces.
`has_metal_kernel` was too coarse and produced a claim that was simply wrong.
MXFP4 *does* have Metal kernels — `mxfp4_matvec` (K1) and the four
`mxfp4_grouped_experts` tournament arms (K2) — but it is absent from
`QuantMatVec`'s format dispatch, so nothing can route a forward pass through
them. "A kernel exists" and "this container can serve" are different facts and
conflating them let the second be asserted from the first.
`KernelMaturity::{None, Standalone, Grouped, Dispatched, Production}` is
ordered so `is_servable()` is `>= Dispatched`, which is what the filter in
"cheapest exact container" actually needs. MXFP4 is Grouped, Q4_K and Q6_K are
Production, Q8_0 Dispatched, Q5_K None. The corrected statement — Q6_K is the
cheapest exact container that can SERVE today — is now a test rather than
prose, and a second test pins that MXFP4 has kernels, is cheaper, and still
cannot serve.
This also sharpens why Q5_K is dominated: not "it needs a kernel too", but it
has no kernel at all while MXFP4 already has two rungs built, so Q5_K costs
strictly more work AND ships more bytes.
The maturity level is what keeps the 4.15 tok/s routed-MXFP4 projection honest
when it travels — it is a kernel measurement, not a serving result.
Re-banks the per-class etas against three genuine cold-rotating repeats and, more importantly, stops banking them as scalars. The cold-rotate bug removed a false precision as well as a bias. Once the rotation genuinely rotated, the small shapes moved 10-12% across repeats while attention moved 2%. The composed total is stable at 3.61-3.68 because attention is both the largest term and the steady one — but every PER-CLASS decision reads the noisy inputs directly. Which class to compress, the frontier's target row, the serving-format choice: a scalar constant lets a 10% measurement swing on a small class silently select a bit-width. Aggregate robustness does not license per-class precision. So `MeasuredEfficiency` carries central, observed low/high, repeat count and regime, and `compose_at` propagates the ends so `ceilings` prints a band beside the point. Classes whose range exceeds 5% print [NOT DECISION-GRADE], which is currently the down class (0.73-0.82) and the ungrouped expert shape (0.63-0.70). Low/high are the observed extremes, deliberately NOT summarised as +/- x. Three repeats cannot distinguish gaussian noise from two modes — page alignment, thermal state, allocator luck — and a symmetric interval would claim they can. If it turns out bimodal the honest bank is the pessimistic mode, not the mean. The correction is not uniformly downward: the ungrouped expert shape came back 0.63-0.70 against a banked 0.64, so the old numbers were not simply flattering. Down 0.87 -> 0.78 and gate/up 0.87 -> 0.83 are the real movers.
…thholds Three presentation-level R0 hazards, all of which survive correct arithmetic. Scenario rows quoted the BASELINE's composition band beside their own central value — a band that did not even contain the number it sat next to. Each scenario now composes its own low/high through `compose_at`. `ceilings` at a bit width other than the one the etas were measured under (Q6_K, 6.5625 bpw) is reusing eta across a container change, which R7 forbids as a performance claim. That row is the biggest on the board and therefore the likeliest to escape into a document unlabelled, so it now prints DENSITY-ONLY UPPER BOUND and names the measured penalty — MXFP4 arm D at 0.724 against Q6_K's 0.89 on the routed class — that has to be applied before quoting it as reachable. `[NOT DECISION-GRADE]` now refuses rather than warns. The R4 lever ordering is what selects the programme, so when a contributing class carries more than 5% observed spread the ranking is withheld entirely and the offending classes are named with the command that would settle them. A warning beside a plausible float gets read past; a missing ranking does not. Precedent is the 8.8 bench exiting rather than feeding the frontier a number it could not stand behind. `Regime::Definitional` exempts the unquantised f32 row: it is a modelling constant with no variance, so it cannot swing and cannot select a bit-width. Without it the refusal fired on a class that has nothing to be uncertain about. `diag_eta_repeats` is the promotion harness — it calls `profile_shape_census` verbatim so samples are drawn from the identical protocol the bank came from, and reports mean, standard error, sorted samples, a histogram and a largest-gap ratio. It deliberately does not conclude modality: three points cannot separate gaussian noise from two modes, and if bimodal the honest bank is the pessimistic mode rather than a mean that describes neither.
Contributor
|
You might want to put concurrency controls on the CI so that it isn't stuck running on the previous commits to the PR. |
…ated campaign Two corrections, one of them to the criterion I added a commit ago. The decision-grade gate used observed spread, `(max-min)/central <= 5%`. That is backwards: `max - min` is non-decreasing in sample count, so the gate rewards collecting less data. Measured directly — attention showed 2% spread over 3 repeats and 5.7% over 7 while its standard error FELL. The gate is now `n >= 5 && relative SE <= 1%`, which tightens with evidence. Observed extremes are still displayed, because they are the honest thing to show; they are just not the thing to decide on. The 16-run promotion campaign then found something worse than noise. Nine runs were unusable: the machine degraded under back-to-back load and the attention control fell 0.89 -> 0.06, a factor of fifteen. Every cell moved together, which is the signature of run-level state rather than per-shape variance. Averaging all sixteen would have banked attention at 0.52 and the down class at 0.53 — wrong by a factor, while presenting as more data. So runs are not exchangeable draws on this hardware, the control gate is the load-bearing part of the harness and the repeat count is not. Banked from the 7 control-passing runs: attention 0.876 +/-0.0075 (0.9% rel) [0.85-0.90] DECISION-GRADE down 0.784 +/-0.0057 (0.7% rel) [0.76-0.80] DECISION-GRADE <- promoted gate/up 0.799 +/-0.0177 (2.2% rel) [0.70-0.84] not decision-grade routed exp 0.614 +/-0.0145 (2.4% rel) [0.54-0.67] not decision-grade So one of the two target classes promoted and one did not, and gate/up demoted on better evidence. The composed exact-Q6_K baseline moves 3.18 -> 3.02 (2.75-3.21). The R4 ordering stays withheld until gate/up and the routed expert shape are settled — on a cooler machine, not a longer campaign.
The previous commit edited this file's header and dropped the `//!` on one line, so the crate's examples stopped compiling. It reached a push because the build check was chained with `;` rather than `&&`, so its failure did not block the commit that followed. A verification step has to gate the commit, not merely precede it.
Records what `larql k3-ledger freq-mass` measured on the banked DEC-0 routed pool (registry: dec8-10-frequency-mass-coverage), and documents the k3-ledger verb, which had no CLI reference at all. Two standing rules, both earned by errors this measurement caught: R9 — a null is not a prediction. The uniform-routing union law was used as a forecast and is wrong in both directions: it overstates per-session support 3.4x (38.1 of 128 measured against 128.0 predicted) and understates top-C coverage ~6.8x. DEC-8.6's "~1 of 16 hit/layer under uniform routing" inherited the second. The sign of that error travels to K3 even though R2 forbids the magnitude. R10 — a caveat has a sign, and the sign is per-statistic. The same 64-prompt domain mixture is anti-conservative for the coverage curve and conservative for the unobserved count, so stating it once per capture applies it backwards to half the outputs. Includes the support-size confound: corr(unobserved, top-8) reads +0.861 but the shape-only forms read +0.543 / -0.501 / +0.659. Findings, all 8-of-128 / 6.25%-activation (R2 blocks transfer to K3): a 6.25% resident set covers 42% of routing events and 90% needs 35% of the bank; session-adaptive population buys 1.03x on the miss stream at that operating point against a 1.41x oracle bound, so the arm is closed pending a domain-clustered K3 trace; 16.1% of layer-expert pairs went unobserved with per-layer spread 3-68 of 128. Static per-layer structure varies enormously while per-layer adaptation does not — different axes, recorded so "no layer class wins" is not quoted against the first. README gains the three programme docs (dec-funnel, k3-funnel, quant-obs), which the documentation index had never listed.
The control-gated campaign measured `[6144, 7168]` directly under Q6_K, but the provenance string still cited `q4k_ffn_gate_up_8sg 10240x2560, Q4K, Gemma shape` and `is_provisional` still flagged it as borrowed. Both described the old figure while the number beside them came from somewhere else — the exact R0 failure the provenance field exists to prevent, and worse than no provenance because it reads as verified. Nothing is provisional now. gate/up carries its own measurement and fails on precision instead (2.2% relative SE), which `is_decision_grade` already reports, so the two epistemic states are no longer conflated: "measured somewhere else" and "measured here but too noisy" are different problems with different fixes. `Regime::Borrowed` goes with it — no class borrows any more, and an enum variant that describes nothing in the bank is a lie waiting to be reused. Also displays the observed spread beside the SE, so the report shows both the quantity that decides (SE) and the quantity that is honest about the tail (extremes), and routes central composition through `Bound::Central` with a debug assertion that it agrees with `compose`.
…f-selection The band was a component-extrema envelope: every class's independent minimum combined into one low bound, every maximum into one high. That describes a run that never happened, and because the campaign's degradation was common-mode it is 1.18x wider than anything observed — 2.75-3.21 against a real 2.79-3.18. `compose_observed` now composes the whole ledger once per accepted run with the pairing preserved, and reports min/max over those seven compositions. Scenario rows hold their `eta_override` across all runs, since a scenario asserts an efficiency rather than having measured one. Attention's 0.876 carries a bias note it did not have. It is also the control the campaign gates on, so its own low tail is truncated and the estimate is biased upward; the other three classes are gated on a variable external to them and carry no such bias. The obvious fix — a sentinel outside the composed ledger — was tested and is worse. `gemma down` (21 MB) stayed at 0.74-0.77 through two runs where attention (72 MB) had already collapsed to 0.33 and 0.26, so it would have admitted them. Degradation here is size-dependent, which means a sentinel must have a working set at least as large as the class it protects. The K2 weights-only probe at 89.5 MB is the candidate; `gemma down` is not. `scripts/precommit_check.sh` makes the verification a gate rather than a step that merely precedes the commit. `set -euo pipefail`, per-package fmt so an unrelated package's in-flight work cannot make the gate unusable, build over --all-targets so examples are covered, tests, and `git diff --check`.
…-01) Records two closed rungs and one corrected measurement, with seven follow-ups. Registry: dec8-11 (K2 layout tournament), dec8-12 (efficiency re-bank). The exact-format search is finished. K3's experts are MXFP4, so a group reconstructs at most 15 distinct values and 4 payload bits is the floor by counting rather than search — MXFP4 already spends exactly it. The doubled alphabet is not an arithmetic progression, so the smallest affine grid holding it needs 25 levels: Q4_K can never be exact, Q5_K can but is dominated, and Q6_K is the cheapest exact container that can actually serve. MXFP4's low kernel efficiency turned out to be the container rather than a defect: ceiling probes put the skeleton at 0.95 of attainable bandwidth, so there is nothing to transplant, and the expert-side kernel line closes at single-token width. The numbers moved down twice, both times because a measurement got honest and never because anything got slower. The table separates what is measured (3.02 controlled healthy-regime) from what is projected (3.65 grouped, 4.15 routed-MXFP4 at maturity Grouped), from what is an upper bound R7 forbids quoting as reachable (5.49 density-only), from what has never been measured at all (sustained laptop throughput). That last row is why the follow-ups lead with the sustained end-to-end run rather than more kernel work. Item 6 files a flaky prefill test the new commit gate surfaced — all-NaN output at roughly 3-5%. It is recorded rather than resolved because the bisect was underpowered and said so: 3 failures in 88 runs, landing on non-adjacent commits with clean 16-run windows between them, where P(0 in 16) at an 8% rate is 0.26. Reporting a commit as clean on that evidence would have been the weaker claim.
…ge loop
clippy's needless_range_loop flags `for i in 0..TARGETS.len() { samples[i]
.push(...) }` under `-D warnings`, which is exactly what larql-compute-metal's
CI clippy gate runs — blocking the workflow on every push regardless of what
else changed. `scripts/precommit_check.sh` checks fmt/build/test but not
clippy, so this slipped past the gate that exists for this file's history of
breaking CI.
… gate Drift alone (mean of the last quarter of repeats against the first) is porous: quartile means can agree while a single repeat spikes 4x — exactly the cell that must not be believed. `sentinel_spread` adds IQR/median dispersion alongside drift; a cell now reads CELL DISTURBED if either drift or spread crosses its limit, not drift alone. Captures two AC=true paired validation runs under bench/aim-validation: the pre-change baseline (no spread column) and the post-change run showing the new gate.
…, not the row population R4 refuted sparsifying the FFN row population: with routing supplied free, the sparse kernel still captured only 23-40% of its row-count reduction. This probes the other axis of the factorisation — the shared `expert_input` vector every routed expert reads. Masking hidden channel `j` removes column `j` from every active expert's gate AND up matrix in one decision, versus one row bought in one expert, which is the leverage difference R4 didn't have. Applied after routing on purpose: `router_in` is computed from the unmasked vector, so the trained router still selects the same experts with the same weights — only what they're fed shrinks. Zeroing (not mean-ablation) is the faithful semantics, since masking a channel is exactly not reading that weight column. Ceiling computed before believing any quality number: input-side masking scales two-thirds of expert bytes (gate+up) and leaves `down` fixed, capping end-to-end speedup at 1.448x even at zero retention; `both_sides` also masks the aggregated expert output so `down` shrinks too, reaching 1.30x at r=0.5. Opt-in via `LARQL_MOE_LATENT_RETENTION`; unset or >=1 is a no-op and output stays byte-identical. Wired into both `cpu_moe_forward` and `ExpertWeightFfn`. Ships channel-block grouping + permutation support (for testing whether an offline weight permutation could make masking realisable in an output-major layout) and a per-layer channel-survival dump feeding an offline co-activation-ordering follow-up. `larql-cli` flushes that dump at exit.
Two diagnoses for the sparse-FFN thesis, and the rule they share. R4 zero-out (dec8-13). Supplying perfect routes for FREE on the production kernel, the sparse path still runs at 0.837x dense at the accuracy-viable K=4096 and wins only at K=2048, which fails top-1. No overlap between the speed-viable and accuracy-viable regions. The binding term is the kernel: with routing free it captures only 23-40% of its row-count reduction, and the capture fraction RISES as fewer rows are dropped (23/34/40% against ceilings 5.00/2.50/1.67x) - fixed per-row overhead, not row choice. Tying dense needs ~18.6% execution improvement with routing already free, so a cheaper approximation of gate-top-K was never the missing piece. Scope kept narrow on purpose: this refutes a COMPUTE-side claim and does not close dec8-1, which is read-side with a different denominator and a milder retention target. What transfers is the mechanism kill. Latent axis (dec8-14). Masking the shared expert input instead of the row population, where one channel decision removes a column from every active expert's gate and up. Signal is real - magnitude vs random separates 9.7% vs 555% bits/token at 50% retention - but the headline was an artifact of larql's known-divergent OLMoE forward, which called r=0.5 free where the HF reference says +9.66%. At the <=0.5% shannon gate the viable retention is r~0.875, ideal ceiling 1.040x. Static channel sets refuted per-layer; blocked latent sparsity refuted per-layer against exact induced loss and a random-permutation control that turned an apparent 40% win into a null. Surviving: dynamic per-channel via an input-major packed kernel. R11: a structural reduction is a claim about a KERNEL, not about a matrix - price it on the kernel that will actually run it. Two instances a day apart (dec8-8's zero weight reuse, R4's 23-40% capture). Corollary: realisability is a property of layout, not of the mask, since a per-channel mask saves nothing in output-major storage but is contiguous in input-major, and channel order is free under an exact permutation. Numbered 11 because R7-R10 landed since this section was last read.
Both decisive latent-axis results came from these two scripts, which lived only in a scratchpad. Without them the numbers in docs/diagnoses/moe-latent-axis-sparsity.md are not reproducible. moe_latent_reference_olmoe.py - repeats the decisive points through transformers' own OLMoE forward. This is the gate that caught the headline artifact: larql's own forward reported 50% retention as free, the reference says +9.66% bits/token. Patch point mirrors the Rust hook exactly - routing on the unmasked input, then mask, then experts - so the trained router still selects the same experts with the same weights. moe_latent_block_partition.py - per-layer channel partitions optimised against the exact induced loss L_t = sum_kept(16-s) + sum_dropped(s), which prices within-block disagreement AND inter-block competition under the fixed budget; pairwise disagreement only seeds it. Calibration and held-out are disjoint contiguous spans, partitions frozen on calibration. Carries the random-permutation control that turned an apparent 40% win into a null, and the held-out L_t diagnostic separating overfitting from objective mismatch. Asserts the block budget per token rather than checking after, and persists masks/partitions/per-layer L_t so further analysis needs no forwards.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
larql-factory::build::run_buildorchestrates FETCH → EXTRACT → SLICE → MANIFEST → VERIFY → PUBLISH → RELEASE as subprocess calls intolarqlitself, behind aCommandRunnertrait (SubprocessRunnerfor real builds,MockRunnerfor tests) — the whole pipeline's stage ordering and failure handling is unit-tested without spawning a process or touching credentials.--private; RELEASE only flips a repo public once every output has verified (§8's "nothing goes public unverified"). Extendslarql-vindex's publish path with aprivate/visibility option (verified against the real HF OpenAPI spec) and a newlarql hf visibilitycommand to make that possible.BuildRecordJSON is the hand-off point, matchingdec-bench's--output-filepattern.larql recipe build <FILE> [--scratch-dir DIR].docs/cli.md,crates/larql-factory/README.mdupdated; two factual corrections landed indocs/vindex-factory.mditself (driver command name, VERIFY scope).Test plan
cargo build -p larql-factory -p larql-clicargo test -p larql-factory --lib(192 tests) andcargo test -p larql-cli recipe_cmdcargo clippy -p larql-factory -p larql-cli --all-targets -- -D warningscargo fmt -p larql-factory -p larql-cli -- --checkcargo llvm-cov -p larql-factory --lib— every newbuild/file at or above the 90% line-coverage floor