From 531d86f9159590804e7da8d69dafb4eb8bfc5271 Mon Sep 17 00:00:00 2001 From: pernyf Date: Tue, 18 Aug 2026 16:21:00 +0200 Subject: [PATCH 01/11] Phase 5 M2 desk-research spike: tokenizer integration architecture Resolves D3's architecture question (FFM-bind a C shim vs. pure Java) in favor of the former, based on primary-source research: - huggingface/tokenizers issue #185 confirms no official C/C++ API was ever planned (closed stale, no maintainer commitment). - mlc-ai/tokenizers-cpp provides a maintained (pushed 2026-05-20), Apache-2.0-licensed, genuinely plain-C shim (tokenizers_c.h) over the same Rust crate -- the same extern "C" shape mlx-c itself presents, covering HF tokenizer.json and byte-level BPE directly without needing its C++/SentencePiece layer for M3's Llama/Qwen targets. - DJL (Deep Java Library) independently chose the same wrap-the-Rust-crate approach over reimplementing in Java, confirming this from a team solving the identical problem for a different host language. - Two real risks are flagged for whatever plan follows: tokenizers-c is a staticlib (needs an extra link step or a forked cdylib crate-type), and its Rust glue calls .unwrap() with no catch_unwind, so a malformed tokenizer.json currently panics across the FFI boundary rather than surfacing as a catchable error. Also fixes this document's own stale M1 status (branch/PR/commit references predate PR #12's merge to main). Still open, deliberately not resolved here: an actual build-and-measure load-time prototype, which needs a Rust toolchain not currently installed on this machine -- confirming with the user before installing one rather than doing so unilaterally. Co-Authored-By: Claude Sonnet 5 --- req/phase5-plan.md | 123 +++++++++++++++++++++++++++++++++++++++------ 1 file changed, 107 insertions(+), 16 deletions(-) diff --git a/req/phase5-plan.md b/req/phase5-plan.md index 75061f1..f9000b5 100644 --- a/req/phase5-plan.md +++ b/req/phase5-plan.md @@ -2,18 +2,18 @@ ## Status — update this section as work lands -**Branch:** `phase5-plan`, off `main` at `f28a327` (PR #11, Phase 4 M4 `QuantizedLinear`, merged). -Phase 4 (`req/phase4-plan.md`) has every milestone M0a–M4 plus §9 documentation **Done** — but not -"fully Done": §10 (CI, self-hosted runner) is still **Not started**, per that -document's own Status table. (M0d is also `Done` only in a scoped-down sense — six generic op-body -helpers deliberately deferred past their original merge point, per that table's own M0d note; this -document's Research findings section below leans on that same precedent for M1's own C-string -helper.) +**Branch:** this phase started on `phase5-plan`, off `main` at `f28a327` (PR #11, Phase 4 M4 +`QuantizedLinear`, merged); M1 has since merged to `main` via PR #12 (`5c85f8c`) and that branch is +deleted. Phase 4 (`req/phase4-plan.md`) has every milestone M0a–M4 plus §9 documentation **Done** — +but not "fully Done": §10 (CI, self-hosted runner) is still **Not started**, per that document's own +Status table. (M0d is also `Done` only in a scoped-down sense — six generic op-body helpers +deliberately deferred past their original merge point, per that table's own M0d note; this document's +Research findings section below leans on that same precedent for M1's own C-string helper.) | Item | Status | Commit | |---|---|---| -| M1 — Checkpoint I/O: `MLXIO`, safetensors + GGUF (§1) | Plan written, implementation not started — see `req/plans/phase5-m1-plan.md` | — | -| M2 — Tokenizer integration (§2) | Not started — needs its own research spike first | — | +| M1 — Checkpoint I/O: `MLXIO`, safetensors + GGUF (§1) | **Done** (`req/plans/phase5-m1-plan.md`'s own amendments record two runtime-discovered fixes beyond the original plan) | `5c85f8c` (PR #12) | +| M2 — Tokenizer integration (§2) | Desk-research spike done (D3 below); build-and-measure prototyping still pending -- needs a decision on installing a Rust toolchain first | — | | M3 — Reference models: `LlamaModel`, `QwenModel` (§3) | Not started | — | ## Context @@ -139,6 +139,89 @@ compatibility, and prototyping load-time cost) before a plan for it can be writt already can be. Do not start M2 implementation from this document — write `req/plans/phase5-m2-plan.md` only after that spike. +**D3 amendment (M2 desk-research spike, first half): there is no official C API, but a viable +third-party one exists, and prior art independently converged on the same architecture jmlx already +uses for MLX itself.** Findings, each confirmed against a primary source rather than assumed: + +- **No official C/C++ binding.** `huggingface/tokenizers` issue #185 (opened 2020, asking exactly + this question) was closed as "not planned" via the stale-bot, with no maintainer commitment ever + made. The crate remains Rust-native with official bindings only for Python (PyO3) and Node.js. +- **A maintained, permissively-licensed, genuinely plain-C shim exists: `mlc-ai/tokenizers-cpp`.** + Apache-2.0 (compatible with this repo's MIT `LICENSE` -- permissive, no copyleft obligation + conflict), 503 stars, actively maintained (pushed 2026-05-20, per `gh api + repos/mlc-ai/tokenizers-cpp` at spike time), built in part for and used by MLC LLM. Its + `include/tokenizers_c.h` is a genuine `extern "C"` API -- opaque `void*` handle, out-param structs, + no C++ name mangling, no JNI ceremony -- the same shape mlx-c itself presents and that jextract/FFM + already binds cleanly in this codebase: + ```c + typedef void* TokenizerHandle; + typedef struct { int* token_ids; size_t len; } TokenizerEncodeResult; + TokenizerHandle tokenizers_new_from_str(const char* json, size_t len); + TokenizerHandle byte_level_bpe_tokenizers_new_from_str(const char* vocab, size_t vocab_len, + const char* merges, size_t merges_len, const char* added_tokens, size_t added_tokens_len); + void tokenizers_encode(TokenizerHandle, const char* data, size_t len, int add_special_token, + TokenizerEncodeResult* result); + void tokenizers_decode(TokenizerHandle, const uint32_t* data, size_t len, int skip_special_token); + void tokenizers_get_decode_str(TokenizerHandle, const char** data, size_t* len); + void tokenizers_free(TokenizerHandle); + ``` + (full list also has `encode_batch`/`free_encode_results`/`get_vocab_size`/`id_to_token`/ + `token_to_id`). This plain-C layer covers HF `tokenizer.json` (`tokenizers_new_from_str`) and raw + byte-level BPE vocab+merges directly; SentencePiece and RWKV-World support exist only at the C++ + layer above it (`include/tokenizers_cpp.h`'s `Tokenizer::FromBlobSentencePiece`/ + `FromBlobRWKVWorld`), which this project would not need to bind at all if only HF-JSON-format + tokenizers are in scope for M3's reference models (Llama/Qwen both ship `tokenizer.json`). +- **Prior art already chose this exact path, independently.** DJL (Deep Java Library, the most + prominent Java ML library with an equivalent problem) does not reimplement HF tokenizers in pure + Java -- `extensions/tokenizers` wraps the same upstream `tokenizers` Rust crate via its own + hand-written native bridge (JNI in DJL's case, since that predates or simply didn't adopt FFM; + jmlx would use FFM instead, which needs no JNI ceremony at all -- one more reason to prefer + `tokenizers-cpp`'s plain-`extern "C"` shape over reusing DJL's crate directly, whose native + functions are JNI-shaped `Java_...` symbols, not callable via FFM). This is independent + confirmation that "FFM/JNI-bind the Rust crate" beats "reimplement in pure Java" for this exact + problem, from a team solving it for a different host language. +- **Build shape: `tokenizers-c` is a Rust `staticlib`, not a `cdylib` -- jmlx cannot load it directly + the way `NativeLoader` loads `libmlxc.dylib`.** `tokenizers-cpp/rust/Cargo.toml` declares `crate-type + = ["staticlib"]`; producing something `System.load()`-able would need either (a) a small additional + link step producing a `.dylib` that statically links `libtokenizers_c.a` (mirroring how + `bootstrap-native.sh` already builds `libmlxc.dylib` from source against a pinned wheel), or (b) a + jmlx-owned fork of `rust/Cargo.toml` + `rust/src/lib.rs` with `crate-type = ["cdylib"]` instead -- + the latter is simpler since it also sidesteps needing the C++/CMake/submodule machinery + (`sentencepiece`, `msgpack`) that only the C++ layer requires, if HF-JSON-only scope (see above) + is accepted for M2. +- **One real risk, not previously visible from the plan text alone: Rust-side panics cross the FFI + boundary as failures with no recoverable status.** `rust/src/lib.rs`'s wrapper calls `.unwrap()` + on `Tokenizer::from_str`/`encode`/`decode` -- a malformed `tokenizer.json` or a decode error panics + inside Rust rather than returning a checkable error code. A Rust panic unwinding across an `extern + "C"` boundary without `catch_unwind` is undefined behavior, not a catchable `MLXException`-style + failure -- structurally worse than mlx-c's own error convention (`printf` + `exit(-1)`, which + `NativeLoader`'s custom handler already replaces) precisely because there is no error-handler hook + to intercept it the way `NativeLoader` intercepts mlx-c's. Whatever plan follows this spike needs + to either wrap every entry point in `catch_unwind` in a jmlx-owned fork of the Rust glue, or + explicitly accept malformed-tokenizer-file input as an unrecoverable-crash case (unlike every other + failure path in this codebase, which surfaces as a catchable `MLXException`). +- **A build-fragility note, not a blocker:** `tokenizers-cpp`'s `onig` Cargo feature (enabled in its + `Cargo.toml`, needed to replicate Python `regex`-module-exact Unicode splitting for GPT-2/GPT-4-style + BPE pretokenizers) pulls in `onig_sys`, which vendors and compiles an old bundled copy of the + Oniguruma C source when no system library is found via `pkg-config` -- known to hit compiler + compatibility issues on newer GCC (unconfirmed either way against Apple's clang on this repo's + actual macOS 26/Apple Silicon target, since that combination has not yet been built here). Whether + M2 needs `onig` at all depends on which reference models' tokenizers M3 actually targets: Llama/Qwen + both use byte-level BPE without the exact GPT-2 regex-split behavior `onig` exists for, so it may be + possible to build with `default-features = false` and skip `onig` entirely, avoiding this risk + rather than resolving it. + +**Still open, deliberately not resolved by this desk-research pass:** actually building a minimal +`cdylib` from a jmlx-owned fork of `tokenizers-cpp/rust` (or from scratch against the plain +`tokenizers` crate) and measuring real load-time cost for a representative `tokenizer.json`, per D3's +original "prototyping load-time cost" requirement. This machine has no Rust toolchain installed +(`cargo`/`rustc` both absent); doing so is an environment change worth confirming with a human before +taking, not something to do unilaterally mid-spike. `req/plans/phase5-m2-plan.md` should not be +written until that prototyping step also lands -- the desk research above resolves the *architecture* +question (FFM-bind a plain-C shim, most likely a jmlx-owned fork of `tokenizers-cpp/rust` scoped to +HF-JSON + byte-level-BPE only, skipping the C++/SentencePiece layer) but not the *cost* question D3 +also asked for. + **D4 — Reference models (M3) are pure composition, deferred until M1 and M2 both land.** `LlamaModel`/`QwenModel` need nothing new at the tensor/module level: `se.alipsa.jmlx.nn` already has `Linear`, `QuantizedLinear`, `RMSNorm`, `MultiHeadAttention`, `KVCache`, and RoPE (via @@ -230,9 +313,12 @@ in M1's own first task, not built ahead of time here. ## Work breakdown -### 1. Checkpoint I/O — `MLXIO`, safetensors + GGUF (M1) — **PLAN WRITTEN, IMPLEMENTATION NOT STARTED** +### 1. Checkpoint I/O — `MLXIO`, safetensors + GGUF (M1) — **DONE** (PR #12, `5c85f8c`) -Full task-by-task plan lives in `req/plans/phase5-m1-plan.md`. Summary: a new +Full task-by-task plan lives in `req/plans/phase5-m1-plan.md`, whose own amendments record two +runtime-discovered fixes beyond what's summarized below (a CPU-stream requirement for +`mlx_load_safetensors`/`mlx_load_gguf`, and a redesign of `loadGguf`'s metadata parameters once +testing showed `mlx_io_gguf_get_keys` cannot enumerate metadata-only keys). Summary: a new `se.alipsa.jmlx.core.MLXIO` facade class (package-private-constructor constraints rule out a new package — see D2) exposing `loadSafetensors`/`saveSafetensors`/`loadGguf`/`saveGguf`, following two distinct precedents for two distinct properties rather than one class as a blanket model: `MLXGrad`'s @@ -248,10 +334,11 @@ parameter (D2a above); `saveGguf` additionally builds and frees its own `mlx_io_ paragraph and architecture-diagram class list to name `MLXIO` as the fifth native-loading-guard class alongside `MLX`/`MLXScope`/`NativeOps`/`MLXGrad`. -### 2. Tokenizer integration (M2) — **NOT STARTED — blocked on a research spike, see D3** +### 2. Tokenizer integration (M2) — **DESK RESEARCH DONE, PROTOTYPE PENDING — see D3's amendment** -Not planned in detail here. First step is the spike named in D3, written up as its own findings -section before `req/plans/phase5-m2-plan.md` exists. +Not planned in detail here. Desk research (license, C-API existence, build shape, prior art, risks) +is written up as D3's amendment above; `req/plans/phase5-m2-plan.md` still should not be written +until the load-time-cost prototype D3 also asked for actually lands. ### 3. Reference models — `LlamaModel`, `QwenModel` (M3) — **NOT STARTED — blocked on M1 and M2** @@ -279,8 +366,12 @@ named scope boundary rather than left implicit. ## Open questions -- M2's core question (FFM-bind HF `tokenizers`'s C API vs. a pure-Java implementation) is - unresolved — see D3. +- M2's architecture question (FFM-bind a plain-C shim over HF `tokenizers` vs. a pure-Java + implementation) is resolved in favor of the former — see D3's amendment. What remains open: + real load-time cost (needs an actual build-and-measure prototype, blocked on a Rust toolchain + decision), whether `onig` is actually needed for M3's target models, and whether to fork + `tokenizers-cpp/rust` or write a from-scratch minimal `cdylib` crate against the plain + `tokenizers` crate directly. No open question remains on the checkpoint-I/O (M1) side: `mlx_vector_string_get`'s ownership, the last unresolved item blocking `loadGguf`'s design, is settled — see Research findings above. From 0631e2381b1630a62b316d2e35dfe70c990d55e9 Mon Sep 17 00:00:00 2001 From: pernyf Date: Tue, 18 Aug 2026 16:30:23 +0200 Subject: [PATCH 02/11] Phase 5 M2 spike: confirm Rust is unavoidable via tokenizers-cpp tokenizers-cpp's C/C++ layers are thin wrappers around the actual tokenizer logic, which is Rust (the upstream tokenizers crate itself). Confirmed via CMakeLists.txt (explicitly targets aarch64-apple-darwin, shells out to cargo build) and its GitHub releases (both tags publish zero binary assets -- no prebuilt library to download the way bootstrap-native.sh already does for MLX itself). Rust becomes a new wherever-this-builds toolchain dependency this project doesn't otherwise have, unlike mlx-c's C/C++ requirement, which Xcode Command Line Tools already satisfies. Co-Authored-By: Claude Sonnet 5 --- req/phase5-plan.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/req/phase5-plan.md b/req/phase5-plan.md index f9000b5..f28f7b6 100644 --- a/req/phase5-plan.md +++ b/req/phase5-plan.md @@ -189,6 +189,23 @@ uses for MLX itself.** Findings, each confirmed against a primary source rather the latter is simpler since it also sidesteps needing the C++/CMake/submodule machinery (`sentencepiece`, `msgpack`) that only the C++ layer requires, if HF-JSON-only scope (see above) is accepted for M2. +- **A Rust toolchain is unavoidable for this whole path -- there is no way to use `tokenizers-cpp` + (or any fork of it) without one.** The actual tokenizer logic is the upstream `tokenizers` Rust + crate itself; `tokenizers-cpp`'s C and C++ layers are thin wrapper headers that call into a Rust + `staticlib` compiled by `cargo`/`rustc` -- they do not replace or reimplement it. Confirmed two + ways: `tokenizers-cpp/CMakeLists.txt` explicitly branches on `CMAKE_SYSTEM_NAME STREQUAL "Darwin"` + + `CMAKE_SYSTEM_PROCESSOR STREQUAL "arm64"` to set `TOKENIZERS_CPP_CARGO_TARGET + aarch64-apple-darwin` (this repo's own target triple) and shells out to `cargo build` via + `CARGO_EXTRA_ENVS`; and neither of its two GitHub releases (`v0.1.0`, `v0.1.1`) publishes any + binary assets at all (source tags only), so there is no prebuilt `.a`/`.dylib` to download the way + `bootstrap-native.sh` already downloads a prebuilt `mlx-metal` wheel for MLX itself. Practically: + `aarch64-apple-darwin` being an explicit, named target is a genuinely good sign for this repo's + specific platform (unlike the ambiguity noted below about `onig`'s clang compatibility), but Rust + becomes a wherever-this-builds toolchain dependency (dev machines and CI both) that this project + does not otherwise have, unlike mlx-c's C/C++ toolchain requirement, which Xcode Command Line + Tools already satisfies on any macOS machine capable of running this project at all. The only way + to avoid a Rust toolchain entirely is the pure-Java alternative D3 already named -- not something + this bullet resolves, only sharpens the actual trade-off being decided. - **One real risk, not previously visible from the plan text alone: Rust-side panics cross the FFI boundary as failures with no recoverable status.** `rust/src/lib.rs`'s wrapper calls `.unwrap()` on `Tokenizer::from_str`/`encode`/`decode` -- a malformed `tokenizer.json` or a decode error panics From f570d94c6b84efb21d4c422d1b4b9097ec9f3d7a Mon Sep 17 00:00:00 2001 From: pernyf Date: Tue, 18 Aug 2026 16:35:41 +0200 Subject: [PATCH 03/11] Phase 5 M2 spike: mlx-c won't add tokenizer support, and DJL vs. HF's own Swift port un-settles the architecture choice Directly answers whether deferring M2 until mlx-c adds tokenizer support is viable: no. mlx-c's own README scopes it as a low-level array/tensor C API mirroring MLX's own scope, and MLX's two other official language bindings (mlx-lm, mlx-swift-lm) both already solved tokenization by depending on an external tokenizer package rather than asking MLX/mlx-c for it -- no roadmap signal anywhere suggests that will change. While checking mlx-swift-lm's own tokenizer dependency (huggingface/swift-transformers), found it's a from-scratch pure-Swift reimplementation (BPETokenizer.swift, UnigramTokenizer.swift, Normalizer.swift, etc.), not a Rust FFI wrapper -- confirmed via its Sources/Tokenizers/ file list and its own TokenizersTests suite, actively maintained by Hugging Face themselves. This corrects an overstated conclusion from the prior commit: DJL's Rust+JNI choice was framed as "independent confirmation" that FFM-binding beats pure Java. It doesn't settle that -- Hugging Face chose the opposite, for the same problem, in a comparably-shaped language. Both directions now have a credible, maintained precedent. What actually changes: "pure Java" is no longer the vaguely-scoped option D3 originally framed it as -- there's a concrete reference implementation, from the organization that owns the tokenizer.json format, to port from instead of reverse-engineering from spec alone. Co-Authored-By: Claude Sonnet 5 --- req/phase5-plan.md | 81 ++++++++++++++++++++++++++++++++++------------ 1 file changed, 61 insertions(+), 20 deletions(-) diff --git a/req/phase5-plan.md b/req/phase5-plan.md index f28f7b6..470924c 100644 --- a/req/phase5-plan.md +++ b/req/phase5-plan.md @@ -13,7 +13,7 @@ Research findings section below leans on that same precedent for M1's own C-stri | Item | Status | Commit | |---|---|---| | M1 — Checkpoint I/O: `MLXIO`, safetensors + GGUF (§1) | **Done** (`req/plans/phase5-m1-plan.md`'s own amendments record two runtime-discovered fixes beyond the original plan) | `5c85f8c` (PR #12) | -| M2 — Tokenizer integration (§2) | Desk-research spike done (D3 below); build-and-measure prototyping still pending -- needs a decision on installing a Rust toolchain first | — | +| M2 — Tokenizer integration (§2) | Desk-research spike done (D3 below): waiting for upstream `mlx-c` tokenizer support is ruled out, but the FFM-vs-pure-Java architecture choice itself is still open between two credible precedents -- neither prototyped yet | — | | M3 — Reference models: `LlamaModel`, `QwenModel` (§3) | Not started | — | ## Context @@ -146,6 +146,23 @@ uses for MLX itself.** Findings, each confirmed against a primary source rather - **No official C/C++ binding.** `huggingface/tokenizers` issue #185 (opened 2020, asking exactly this question) was closed as "not planned" via the stale-bot, with no maintainer commitment ever made. The crate remains Rust-native with official bindings only for Python (PyO3) and Node.js. +- **`mlx`/`mlx-c` adding tokenizer support upstream is very unlikely, and waiting for it is not a + viable deferral -- confirmed against the project's own stated scope and its own precedent across + its other two official language bindings, not assumed.** `ml-explore/mlx-c`'s own README states its + scope directly: "a C API for MLX... expands MLX to the C language... can be used standalone or as + a bridge to bind other languages to MLX" -- a low-level array/tensor API mirroring MLX's own core + scope (an array framework, not a text-processing one), with no tokenizer functionality anywhere in + it. More telling: Apple's own two higher-level official bindings already faced this exact problem + and both solved it the same way jmlx is now considering, not by asking MLX/mlx-c for it -- + `mlx-lm` (Python) depends directly on Hugging Face's own `transformers`/`tokenizers` packages, and + `mlx-swift-lm` (Swift) "integrates with a variety of tokenizer and downloader packages through + protocol conformance," naming `huggingface/swift-transformers`'s `Tokenizers` product specifically + (see below) as its default. All recent tokenizer-related engineering activity in the `ml-explore` + org (streaming-detokenizer fixes, chat-template handling, think-token NoneType fixes) is happening + in `mlx-lm` at the Python application layer, not in `mlx`/`mlx-c` core -- no roadmap item, issue, or + discussion found anywhere suggesting tokenizer support is planned for MLX's core C API. This is the + same architectural choice jmlx now faces (FFM-bind an external tokenizer implementation vs. write + one), not a problem MLX itself is going to solve on jmlx's behalf. - **A maintained, permissively-licensed, genuinely plain-C shim exists: `mlc-ai/tokenizers-cpp`.** Apache-2.0 (compatible with this repo's MIT `LICENSE` -- permissive, no copyleft obligation conflict), 503 stars, actively maintained (pushed 2026-05-20, per `gh api @@ -171,15 +188,30 @@ uses for MLX itself.** Findings, each confirmed against a primary source rather layer above it (`include/tokenizers_cpp.h`'s `Tokenizer::FromBlobSentencePiece`/ `FromBlobRWKVWorld`), which this project would not need to bind at all if only HF-JSON-format tokenizers are in scope for M3's reference models (Llama/Qwen both ship `tokenizer.json`). -- **Prior art already chose this exact path, independently.** DJL (Deep Java Library, the most - prominent Java ML library with an equivalent problem) does not reimplement HF tokenizers in pure - Java -- `extensions/tokenizers` wraps the same upstream `tokenizers` Rust crate via its own - hand-written native bridge (JNI in DJL's case, since that predates or simply didn't adopt FFM; - jmlx would use FFM instead, which needs no JNI ceremony at all -- one more reason to prefer - `tokenizers-cpp`'s plain-`extern "C"` shape over reusing DJL's crate directly, whose native - functions are JNI-shaped `Java_...` symbols, not callable via FFM). This is independent - confirmation that "FFM/JNI-bind the Rust crate" beats "reimplement in pure Java" for this exact - problem, from a team solving it for a different host language. +- **Prior art exists on both sides of this choice, from credible sources -- not a settled question.** + DJL (Deep Java Library, the most prominent Java ML library with an equivalent problem) does not + reimplement HF tokenizers in pure Java -- `extensions/tokenizers` wraps the same upstream + `tokenizers` Rust crate via its own hand-written native bridge (JNI in DJL's case, since that + predates or simply didn't adopt FFM; jmlx would use FFM instead, which needs no JNI ceremony at + all -- one more reason to prefer `tokenizers-cpp`'s plain-`extern "C"` shape over reusing DJL's + crate directly, whose native functions are JNI-shaped `Java_...` symbols, not callable via FFM). + + **Amendment: this bullet originally concluded DJL's choice was "independent confirmation that + FFM/JNI-bind beats reimplement" -- that overstated it.** Hugging Face itself did the opposite for + Swift: `huggingface/swift-transformers` (Apache-2.0, actively maintained -- pushed 2026-07-28, + confirmed via `gh api repos/huggingface/swift-transformers`) is a **from-scratch, pure-Swift** + reimplementation of the tokenizer pipeline, not a Rust FFI wrapper -- confirmed directly from its + `Sources/Tokenizers/` file list: `BPETokenizer.swift`, `UnigramTokenizer.swift`, + `BertTokenizer.swift`, `Normalizer.swift`, `PreTokenizer.swift`, `PostProcessor.swift`, + `TokenLattice.swift`, `Trie.swift`, with its own `TokenizersTests` suite. `mlx-swift-lm` (see the + bullet above) depends on exactly this package for its own tokenizer support. So the two most + directly comparable precedents split: DJL (Java, wraps Rust via JNI) vs. Hugging Face itself + (Swift, pure reimplementation, no Rust at all) -- both maintained, both production-used, for the + identical problem. What this actually changes for M2: "pure Java" is no longer the vaguely-scoped + fallback D3 originally framed it as -- there is now a concrete, battle-tested reference + implementation, maintained by the same organization that owns the `tokenizer.json` format, to port + from rather than reverse-engineer from spec documents alone. This does not resolve the choice; it + makes both sides of it equally concrete. - **Build shape: `tokenizers-c` is a Rust `staticlib`, not a `cdylib` -- jmlx cannot load it directly the way `NativeLoader` loads `libmlxc.dylib`.** `tokenizers-cpp/rust/Cargo.toml` declares `crate-type = ["staticlib"]`; producing something `System.load()`-able would need either (a) a small additional @@ -351,11 +383,12 @@ parameter (D2a above); `saveGguf` additionally builds and frees its own `mlx_io_ paragraph and architecture-diagram class list to name `MLXIO` as the fifth native-loading-guard class alongside `MLX`/`MLXScope`/`NativeOps`/`MLXGrad`. -### 2. Tokenizer integration (M2) — **DESK RESEARCH DONE, PROTOTYPE PENDING — see D3's amendment** +### 2. Tokenizer integration (M2) — **DESK RESEARCH DONE, ARCHITECTURE CHOICE STILL OPEN — see D3's amendment** -Not planned in detail here. Desk research (license, C-API existence, build shape, prior art, risks) -is written up as D3's amendment above; `req/plans/phase5-m2-plan.md` still should not be written -until the load-time-cost prototype D3 also asked for actually lands. +Not planned in detail here. Desk research (license, C-API existence, build shape, prior art on both +sides, risks) is written up as D3's amendment above; `req/plans/phase5-m2-plan.md` still should not +be written until both the FFM-vs-pure-Java choice is made and, if FFM is chosen, the load-time-cost +prototype D3 also asked for actually lands. ### 3. Reference models — `LlamaModel`, `QwenModel` (M3) — **NOT STARTED — blocked on M1 and M2** @@ -383,12 +416,20 @@ named scope boundary rather than left implicit. ## Open questions -- M2's architecture question (FFM-bind a plain-C shim over HF `tokenizers` vs. a pure-Java - implementation) is resolved in favor of the former — see D3's amendment. What remains open: - real load-time cost (needs an actual build-and-measure prototype, blocked on a Rust toolchain - decision), whether `onig` is actually needed for M3's target models, and whether to fork - `tokenizers-cpp/rust` or write a from-scratch minimal `cdylib` crate against the plain - `tokenizers` crate directly. +- **Resolved: waiting for `mlx`/`mlx-c` to add tokenizer support upstream is not a viable deferral + for M2.** See D3's amendment — MLX's own two other official language bindings (`mlx-lm`, + `mlx-swift-lm`) both already solved this by depending on an external tokenizer package rather than + MLX/mlx-c itself, and there is no roadmap signal anywhere suggesting that will change. +- **Still open: M2's architecture question itself (FFM-bind a plain-C shim over HF `tokenizers` vs. + a pure-Java reimplementation) is genuinely two-sided, not resolved.** An earlier pass through this + document concluded DJL's Rust/JNI choice settled it — that overstated things: Hugging Face's own + `swift-transformers` independently chose a from-scratch pure-Swift reimplementation for the + identical problem, so both directions now have a maintained, production-used precedent from a + credible source. What remains open regardless of which direction is chosen: real load-time cost for + the FFM path (needs an actual build-and-measure prototype, blocked on a Rust toolchain decision), + whether `onig` is actually needed for M3's target models if the FFM path is chosen, and how large a + port from `swift-transformers` would actually be if the pure-Java path is chosen instead (not yet + estimated — its `Sources/Tokenizers/` file list is known, its total size/complexity is not). No open question remains on the checkpoint-I/O (M1) side: `mlx_vector_string_get`'s ownership, the last unresolved item blocking `loadGguf`'s design, is settled — see Research findings above. From 3a2b880b88d0a09ea055c9434b472a9fa46b03e0 Mon Sep 17 00:00:00 2001 From: pernyf Date: Tue, 18 Aug 2026 16:40:38 +0200 Subject: [PATCH 04/11] Phase 5 M2 spike: measure the pure-Java port size instead of guessing it swift-transformers/Sources/Tokenizers/ totals ~3,850 lines across 12 files -- a real number, not a guess. Of Tokenizer.swift's 1041 lines, only AutoTokenizer's ~100 lines are Hub-download convenience code a port would drop (MLXIO's own file-loading precedent already covers "load from a local string" instead). The real complication found while measuring: applyChatTemplate is part of the core Tokenizer protocol, not an add-on, and depends on `import Jinja` -- swift-jinja, a full Jinja2 template engine, confirmed via Package.swift listing it as a direct (not test-only) dependency of the Tokenizers target. swift-jinja/Sources/Jinja/ totals ~6,660 lines on its own, larger than the tokenizer pipeline itself. So the honest total is ~3,850 lines (tokenize/detokenize only) or ~10,500 lines (with full chat-template fidelity), not a single number. Named but not resolved: M3's actual models (a known, small set, not arbitrary Hub models) may not need general Jinja2 evaluation at all -- hand-formatting their specific known chat templates would sidestep the ~6,660-line Jinja port, mirroring this codebase's existing ship-exactly-what's-needed convention. That scoping call depends on M3's own requirements, which this document explicitly defers (D4). Co-Authored-By: Claude Sonnet 5 --- req/phase5-plan.md | 47 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 42 insertions(+), 5 deletions(-) diff --git a/req/phase5-plan.md b/req/phase5-plan.md index 470924c..c5d2e00 100644 --- a/req/phase5-plan.md +++ b/req/phase5-plan.md @@ -212,6 +212,39 @@ uses for MLX itself.** Findings, each confirmed against a primary source rather implementation, maintained by the same organization that owns the `tokenizer.json` format, to port from rather than reverse-engineer from spec documents alone. This does not resolve the choice; it makes both sides of it equally concrete. +- **Port-size estimate for the pure-Java path, measured directly rather than guessed.** + `swift-transformers/Sources/Tokenizers/` totals **~3,850 lines** across 12 files + (`Tokenizer.swift` 1041, `Normalizer.swift` 376, `BertTokenizer.swift` 358, `PreTokenizer.swift` + 366, `BPETokenizer.swift` 339, `ByteEncoder.swift` 286, `Decoder.swift` 283, + `UnigramTokenizer.swift` 167, `TokenLattice.swift`/`String+PreTokenization.swift` 158 each, + `PostProcessor.swift` 214, `Trie.swift` 101); its own `TokenizersTests` totals **~2,290 lines** + (excluding `ChatTemplateTests.swift` and `YYJSONParserTests.swift`, neither of which tests the + tokenizer pipeline itself), reusable as a porting/validation reference even though XCTest + assertions don't translate mechanically to JUnit. Of `Tokenizer.swift`'s 1041 lines, only + `AutoTokenizer` (lines 846-948, ~100 lines, 11 of the file's `Hub.`-prefixed references) is + Hub-download convenience code a Java port would drop entirely -- `MLXIO`'s own file-loading + precedent already covers "load from a local string/path," so this isn't lost functionality, just + code that would never get ported. **The real complication: `applyChatTemplate` (part of the core + `Tokenizer` protocol, not an add-on) depends on `import Jinja` -- `huggingface/swift-jinja`, a full + Jinja2-template-engine port, confirmed via `Package.swift`'s `Tokenizers` target listing it as a + direct dependency, not merely a test dependency.** `swift-jinja/Sources/Jinja/` totals **~6,660 + lines** across 13 files (`Filters.swift` 2178, `Interpreter.swift` 903, `Value.swift` 891, + `Parser.swift` 777, `Lexer.swift` 484, plus 8 smaller files) -- larger than the tokenizer pipeline + itself. So the honest total for a pure-Java port with full chat-template fidelity is **~10,500 + lines** (~3,850 tokenizer + ~6,660 Jinja), not ~3,850 -- a materially different scoping question + than "port the tokenizer." Llama/Qwen instruct variants (M3's actual targets) need *some* chat + formatting to be usable at all, but not necessarily *general* Jinja2 evaluation: since M3 targets + a small, known set of model families rather than arbitrary Hub models, hand-formatting each + target model's own specific chat template as a small dedicated Java method (reading the same + `tokenizer_config.json`'s `chat_template` field just enough to detect which known template it is, + or simply hardcoding the known Llama-3/Qwen2 chat-format strings) would sidestep the ~6,660-line + Jinja port entirely -- at the cost of not generalizing to arbitrary future Hub models the way a + real Jinja engine would, mirroring this codebase's own established "ship exactly what's needed, + not the general case" convention (e.g. `req/plans/phase4-m4-plan.md`'s own deferrals). This + scoping choice is not made here -- it depends on M3's own requirements, which this document + explicitly declines to plan (D4) -- but it changes the comparison materially: ~3,850 lines (no + chat templates) or ~10,500 lines (full Jinja) against the FFM path's toolchain cost, not a single + fixed number either way. - **Build shape: `tokenizers-c` is a Rust `staticlib`, not a `cdylib` -- jmlx cannot load it directly the way `NativeLoader` loads `libmlxc.dylib`.** `tokenizers-cpp/rust/Cargo.toml` declares `crate-type = ["staticlib"]`; producing something `System.load()`-able would need either (a) a small additional @@ -425,11 +458,15 @@ named scope boundary rather than left implicit. document concluded DJL's Rust/JNI choice settled it — that overstated things: Hugging Face's own `swift-transformers` independently chose a from-scratch pure-Swift reimplementation for the identical problem, so both directions now have a maintained, production-used precedent from a - credible source. What remains open regardless of which direction is chosen: real load-time cost for - the FFM path (needs an actual build-and-measure prototype, blocked on a Rust toolchain decision), - whether `onig` is actually needed for M3's target models if the FFM path is chosen, and how large a - port from `swift-transformers` would actually be if the pure-Java path is chosen instead (not yet - estimated — its `Sources/Tokenizers/` file list is known, its total size/complexity is not). + credible source. The pure-Java port size is now measured, not guessed (see D3's amendment): ~3,850 + lines for tokenization alone, or ~10,500 lines if full Jinja2 chat-template fidelity is also + wanted — a real, quantified number to weigh against the FFM path's toolchain cost, not an open + unknown anymore. What remains genuinely open regardless of which direction is chosen: real + load-time cost for the FFM path (needs an actual build-and-measure prototype, blocked on a Rust + toolchain decision), whether `onig` is actually needed for M3's target models if the FFM path is + chosen, and whether M3's reference models need general Jinja2 chat-template evaluation at all or + can get away with hand-formatting a small, known set of chat templates instead (D4 explicitly + defers M3's own requirements, so this isn't decidable from this document alone). No open question remains on the checkpoint-I/O (M1) side: `mlx_vector_string_get`'s ownership, the last unresolved item blocking `loadGguf`'s design, is settled — see Research findings above. From ee95f0cb23d22464b2d6ad72bb463ebcb96d2a2e Mon Sep 17 00:00:00 2001 From: pernyf Date: Tue, 18 Aug 2026 16:46:39 +0200 Subject: [PATCH 05/11] Phase 5 M2 spike: no Java swift-jinja equivalent, but a smaller HF-maintained porting source and a third no-port option Direct answer: no official Java equivalent of swift-jinja exists. Jinjava (HubSpot, Apache-2.0) is the closest existing Java Jinja-syntax engine but was built for HubSpot's CMS content, not HF chat templates specifically -- its compatibility with real Llama/Qwen templates is unverified, and Maven Repository flags open CVEs against it (likely a different threat model than jmlx's trusted-input case, but asserted here, not verified). Two more useful findings while checking: - HF's own JS Jinja implementation (@huggingface/jinja, MIT, pushed within the last day) is ~3,860 lines -- ~42% smaller than swift-jinja's ~6,660 -- and its e2e test suite ships verbatim chat_template strings and expected-output goldens for meta-llama/Llama-3.1-8B-Instruct, Qwen/Qwen2.5-7B-Instruct, and Qwen/Qwen3-0.6B specifically -- M3's own named targets. Porting from this source instead of swift-jinja revises the full-fidelity estimate from ~10,500 down to ~7,700 lines. - A third option needs no Jinja port at all: GraalJS runs on stock OpenJDK (no GraalVM install, no native toolchain) via org.graalvm.polyglot:js-community Maven artifacts, and could run @huggingface/jinja's actual JS directly -- reusing HF's own tested implementation with zero compatibility-verification risk, at the cost of a genuinely heavyweight dependency for rendering one short template string per model load. Co-Authored-By: Claude Sonnet 5 --- req/phase5-plan.md | 64 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 56 insertions(+), 8 deletions(-) diff --git a/req/phase5-plan.md b/req/phase5-plan.md index c5d2e00..4e4c486 100644 --- a/req/phase5-plan.md +++ b/req/phase5-plan.md @@ -245,6 +245,49 @@ uses for MLX itself.** Findings, each confirmed against a primary source rather explicitly declines to plan (D4) -- but it changes the comparison materially: ~3,850 lines (no chat templates) or ~10,500 lines (full Jinja) against the FFM path's toolchain cost, not a single fixed number either way. +- **No official Java equivalent of `swift-jinja` exists; the three real options for the chat-template + half specifically each trade off differently, and one of them meaningfully revises the ~10,500-line + estimate above.** Hugging Face maintains minimalistic, purpose-built Jinja implementations for JS + (`@huggingface/jinja`, in `huggingface/huggingface.js`), Swift (`swift-jinja`), and (community, + JS-derived) PHP -- but not Java. + 1. **Jinjava (HubSpot)** is the only existing Java Jinja-syntax engine of note -- Apache-2.0, ~780 + stars, actively maintained (pushed 2026-08-03), used in production to render HubSpot's own CMS. + But it was built for "the subset of jinja in use in HubSpot content," not for HF chat templates + specifically -- unlike `swift-jinja`/`@huggingface/jinja`, which are purpose-built for exactly + this (custom globals/filters chat templates lean on, like `raise_exception`/`tojson`), Jinjava's + compatibility with real Llama/Qwen chat templates is unverified, not merely unproven -- would + need checking against actual template strings before trusting it. Maven Repository also flags + open CVEs against it (e.g. CVE-2026-25526) -- likely server-side-template-injection-from- + untrusted-input concerns relevant to HubSpot's threat model (rendering user-supplied templates + in a web CMS), not necessarily jmlx's (rendering a trusted `chat_template` string that ships + inside a model file the user already chose to load) -- but this distinction is asserted here, + not verified against the actual CVE text, and should be before depending on Jinjava for real. + 2. **`@huggingface/jinja` (the JS implementation) is a smaller, more current porting source than + `swift-jinja`, and already ships exact goldens for M3's own target models.** Measured directly: + `packages/jinja/src/` in `huggingface/huggingface.js` totals **~3,860 lines** across 7 files + (`runtime.ts` 1929, `parser.ts` 685, `lexer.ts` 413, `format.ts` 327, `ast.ts` 320, `utils.ts` + 125, `index.ts` 57) -- MIT-licensed, pushed within the last day at spike time (`gh api + repos/huggingface/huggingface.js` shows `pushed_at: "2026-08-18T03:21..."`), roughly the tokenizer + pipeline's own size and **~42% smaller than `swift-jinja`'s ~6,660 lines** for what is presumably + equivalent chat-template-rendering coverage (HF maintains both from the same org for the same + narrow purpose). Its own `test/e2e.test.js` contains verbatim `chat_template` strings and + expected-output goldens for, among others, `meta-llama/Llama-3.1-8B-Instruct`, + `Qwen/Qwen2.5-7B-Instruct`, `Qwen/Qwen3-0.6B`, and `Qwen/Qwen1.5-72B-Chat` -- M3's own named + targets, not generic examples -- which would port directly into JUnit goldens regardless of + which Jinja path is chosen. **This revises the earlier ~10,500-line full-fidelity estimate + downward to ~7,700 lines** (~3,850 tokenizer + ~3,860 Jinja) if porting from this source instead + of `swift-jinja`. + 3. **A third option avoids porting Jinja at all: embed GraalJS and run `@huggingface/jinja`'s actual + JS directly from the JVM.** Confirmed GraalJS runs on a stock OpenJDK, no GraalVM installation or + native toolchain required, via Maven Central artifacts (`org.graalvm.polyglot:polyglot` + + `org.graalvm.polyglot:js-community` specifically -- the plain `js` artifact defaults to Oracle's + more restrictive GFTC license, not a permissive OSS one). This reuses HF's actual, tested, + model-specific-validated implementation with zero hand-porting or compatibility-verification + risk, at the cost of a genuinely heavyweight dependency (a JS engine, non-trivial jar size) for + rendering one short template string per model load -- a real tension with this project's own + "pure, idiomatic Java... zero-copy FFM" framing in `CLAUDE.md`, which none of the tokenizer-side + options raise this sharply. Not evaluated further here: actual jar-size/startup-cost numbers, or + whether GraalJS's polyglot API composes cleanly with `MLXScope`'s confinement/lifecycle model. - **Build shape: `tokenizers-c` is a Rust `staticlib`, not a `cdylib` -- jmlx cannot load it directly the way `NativeLoader` loads `libmlxc.dylib`.** `tokenizers-cpp/rust/Cargo.toml` declares `crate-type = ["staticlib"]`; producing something `System.load()`-able would need either (a) a small additional @@ -459,14 +502,19 @@ named scope boundary rather than left implicit. `swift-transformers` independently chose a from-scratch pure-Swift reimplementation for the identical problem, so both directions now have a maintained, production-used precedent from a credible source. The pure-Java port size is now measured, not guessed (see D3's amendment): ~3,850 - lines for tokenization alone, or ~10,500 lines if full Jinja2 chat-template fidelity is also - wanted — a real, quantified number to weigh against the FFM path's toolchain cost, not an open - unknown anymore. What remains genuinely open regardless of which direction is chosen: real - load-time cost for the FFM path (needs an actual build-and-measure prototype, blocked on a Rust - toolchain decision), whether `onig` is actually needed for M3's target models if the FFM path is - chosen, and whether M3's reference models need general Jinja2 chat-template evaluation at all or - can get away with hand-formatting a small, known set of chat templates instead (D4 explicitly - defers M3's own requirements, so this isn't decidable from this document alone). + lines for tokenization alone, ~7,700 lines with full chat-template fidelity if porting Jinja from + HF's own smaller, more current JS implementation (`@huggingface/jinja`) rather than `swift-jinja`, + or ~10,500 from the latter — real, quantified numbers to weigh against the FFM path's toolchain + cost, not an open unknown anymore. No Java-native equivalent of `swift-jinja` exists; Jinjava + (HubSpot) is the closest existing library but is unverified against real HF chat templates, and + embedding GraalJS to run `@huggingface/jinja` directly (no porting at all) is a third option with + its own dependency-weight cost — see D3's amendment for all three. What remains genuinely open + regardless of which direction is chosen: real load-time cost for the FFM path (needs an actual + build-and-measure prototype, blocked on a Rust toolchain decision), whether `onig` is actually + needed for M3's target models if the FFM path is chosen, and whether M3's reference models need + general Jinja2 chat-template evaluation at all or can get away with hand-formatting a small, known + set of chat templates instead (D4 explicitly defers M3's own requirements, so this isn't decidable + from this document alone). No open question remains on the checkpoint-I/O (M1) side: `mlx_vector_string_get`'s ownership, the last unresolved item blocking `loadGguf`'s design, is settled — see Research findings above. From af3c896e31947d0f92ad50d8f25dd786ffce9c8a Mon Sep 17 00:00:00 2001 From: pernyf Date: Sun, 30 Aug 2026 14:57:12 +0200 Subject: [PATCH 06/11] Phase 5 M2 spike: hfjinja supersedes the Jinjava/port/GraalJS trade-off hfjinja (github.com/Alipsa/hfjinja) is a released, dependency-free Java 21+ port of @huggingface/jinja itself -- option 2 from the prior amendment, already done, by the same org as jmlx. This moots most of the port-vs-Jinjava-vs-GraalJS analysis: the ~3,860-line chat-template half of the pure-Java estimate doesn't need porting if adopted. Still unverified: its Maven coordinate and whether its differential test corpus covers M3's actual target models (Llama/Qwen/Mistral). Co-Authored-By: Claude Sonnet 5 --- req/phase5-plan.md | 37 +++++++++++++++++++++++++++---------- 1 file changed, 27 insertions(+), 10 deletions(-) diff --git a/req/phase5-plan.md b/req/phase5-plan.md index 4e4c486..a8c9c03 100644 --- a/req/phase5-plan.md +++ b/req/phase5-plan.md @@ -288,6 +288,20 @@ uses for MLX itself.** Findings, each confirmed against a primary source rather "pure, idiomatic Java... zero-copy FFM" framing in `CLAUDE.md`, which none of the tokenizer-side options raise this sharply. Not evaluated further here: actual jar-size/startup-cost numbers, or whether GraalJS's polyglot API composes cleanly with `MLXScope`'s confinement/lifecycle model. + + **Amendment: "no official Java equivalent of `swift-jinja` exists" is no longer true.** `hfjinja` + (`github.com/Alipsa/hfjinja`, MIT, same org as jmlx) has since been released -- a dependency-free + Java 21+ port of `@huggingface/jinja` specifically (option 2 above), pinned to upstream `0.5.9`, + explicitly scoped as "the Hugging Face chat-template Jinja subset," not a general-purpose or + Python-compatible Jinja2 engine -- the same narrow scope this document already argued for over + Jinjava. This effectively supersedes options 1-3 for M2/M3: it needs no hand-port of the ~3,860 + remaining Jinja lines (already done, against the smaller/more current source this document already + preferred over `swift-jinja`), no Jinjava compatibility-verification risk, and no GraalJS + dependency-weight cost. Not yet verified here: whether it's published to a resolvable Maven + coordinate yet (the repo alone doesn't confirm this), and whether its byte-exact-vs.-Node-output + differential corpus actually covers M3's target models (Llama/Qwen/Mistral) specifically -- both + worth checking before depending on it for real, but the "port or embed a JS engine" trade-off this + section spent most of its length on is likely moot now. - **Build shape: `tokenizers-c` is a Rust `staticlib`, not a `cdylib` -- jmlx cannot load it directly the way `NativeLoader` loads `libmlxc.dylib`.** `tokenizers-cpp/rust/Cargo.toml` declares `crate-type = ["staticlib"]`; producing something `System.load()`-able would need either (a) a small additional @@ -505,16 +519,19 @@ named scope boundary rather than left implicit. lines for tokenization alone, ~7,700 lines with full chat-template fidelity if porting Jinja from HF's own smaller, more current JS implementation (`@huggingface/jinja`) rather than `swift-jinja`, or ~10,500 from the latter — real, quantified numbers to weigh against the FFM path's toolchain - cost, not an open unknown anymore. No Java-native equivalent of `swift-jinja` exists; Jinjava - (HubSpot) is the closest existing library but is unverified against real HF chat templates, and - embedding GraalJS to run `@huggingface/jinja` directly (no porting at all) is a third option with - its own dependency-weight cost — see D3's amendment for all three. What remains genuinely open - regardless of which direction is chosen: real load-time cost for the FFM path (needs an actual - build-and-measure prototype, blocked on a Rust toolchain decision), whether `onig` is actually - needed for M3's target models if the FFM path is chosen, and whether M3's reference models need - general Jinja2 chat-template evaluation at all or can get away with hand-formatting a small, known - set of chat templates instead (D4 explicitly defers M3's own requirements, so this isn't decidable - from this document alone). + cost, not an open unknown anymore. **Largely mooted since: `hfjinja` + (`github.com/Alipsa/hfjinja`) is a released, dependency-free Java 21+ port of `@huggingface/jinja` + itself — i.e. option 2 already done, by the same org as jmlx — so the chat-template half of the + pure-Java estimate (~3,860 of the ~7,700 lines) doesn't need porting at all if adopted; see D3's + amendment for what's still unverified (Maven coordinate, model-coverage of its differential test + corpus).** What remains genuinely open regardless of which tokenization direction (FFM vs. + pure-Java) is chosen: real load-time cost for the FFM path (needs an actual build-and-measure + prototype, blocked on a Rust toolchain decision), whether `onig` is actually needed for M3's target + models if the FFM path is chosen, and whether M3's reference models need general Jinja2 + chat-template evaluation at all or can get away with hand-formatting a small, known set of chat + templates instead (D4 explicitly defers M3's own requirements, so this isn't decidable from this + document alone) — though if `hfjinja` is adopted, that last question loses most of its urgency too, + since the "port vs. hand-format" trade-off it was weighing is no longer a real port. No open question remains on the checkpoint-I/O (M1) side: `mlx_vector_string_get`'s ownership, the last unresolved item blocking `loadGguf`'s design, is settled — see Research findings above. From 7d25f1269e3123c677cd5642a024549e4df03001 Mon Sep 17 00:00:00 2001 From: pernyf Date: Sun, 30 Aug 2026 15:17:10 +0200 Subject: [PATCH 07/11] =?UTF-8?q?Phase=205=20M2=20spike:=20fix=20review=20?= =?UTF-8?q?findings=20=E2=80=94=20stale=20resolution=20claim,=20onig,=20pa?= =?UTF-8?q?nic/UB,=20hfjinja=20Maven=20gap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - D3's closing paragraph claimed the desk research "resolves the architecture question," contradicting the status table, the M2 section heading, and the open-questions section, all of which correctly say it's still open. Stale text from the first spike commit that the DJL-retraction commit missed; now says it narrows but resolves neither the architecture nor the cost question. - The onig escape-hatch bullet was wrong on both premise and mechanism: Llama-3/Qwen2 pretokenizers both use exactly the regex-lookahead split onig exists for (confirmed against llama.cpp's vocab regexes), default-features = false is already set (onig is an explicit opt-in, not a default to drop), and the only alternative engine (fancy-regex) is wasm-gated. Reframed as a hard requirement of the FFM path, not an escapable risk. - "Undefined behavior" was the wrong label for a Rust panic crossing extern "C" -- the Rustonomicon says it aborts safely; UB is the reverse direction. Corrected, and reframed the real distinction as interceptable (mlx-c, via NativeLoader) vs. not (Rust abort). - Added the int*/uint32_t* signedness mismatch between TokenizerEncodeResult and tokenizers_decode to the risk list. - hfjinja is confirmed absent from Maven Central (numFound: 0), so its "supersedes options 1-3" framing needed a caveat: adopting it today means JitPack, a source/composite build, or publishing it first. Co-Authored-By: Claude Sonnet 5 --- req/phase5-plan.md | 97 ++++++++++++++++++++++++++++------------------ 1 file changed, 60 insertions(+), 37 deletions(-) diff --git a/req/phase5-plan.md b/req/phase5-plan.md index a8c9c03..09c23c9 100644 --- a/req/phase5-plan.md +++ b/req/phase5-plan.md @@ -294,13 +294,17 @@ uses for MLX itself.** Findings, each confirmed against a primary source rather Java 21+ port of `@huggingface/jinja` specifically (option 2 above), pinned to upstream `0.5.9`, explicitly scoped as "the Hugging Face chat-template Jinja subset," not a general-purpose or Python-compatible Jinja2 engine -- the same narrow scope this document already argued for over - Jinjava. This effectively supersedes options 1-3 for M2/M3: it needs no hand-port of the ~3,860 - remaining Jinja lines (already done, against the smaller/more current source this document already - preferred over `swift-jinja`), no Jinjava compatibility-verification risk, and no GraalJS - dependency-weight cost. Not yet verified here: whether it's published to a resolvable Maven - coordinate yet (the repo alone doesn't confirm this), and whether its byte-exact-vs.-Node-output - differential corpus actually covers M3's target models (Llama/Qwen/Mistral) specifically -- both - worth checking before depending on it for real, but the "port or embed a JS engine" trade-off this + Jinjava. This effectively supersedes options 1-3 for M2/M3 on the porting/build-shape question: it + needs no hand-port of the ~3,860 remaining Jinja lines (already done, against the smaller/more + current source this document already preferred over `swift-jinja`), no Jinjava + compatibility-verification risk, and no GraalJS dependency-weight cost. **It does not, however, + supersede the adoption cost: Maven Central's search API returns `numFound: 0` for `hfjinja` -- + confirmed directly, not left as an open question -- so it is not currently resolvable as a normal + Gradle dependency.** Adopting it today means JitPack, a source/composite build, or publishing it + to Maven Central first; that real cost belongs in the comparison, not folded into "supersedes + everything." Still unverified: whether its byte-exact-vs.-Node-output differential corpus actually + covers M3's target models (Llama/Qwen/Mistral) specifically -- worth checking before depending on + it for real, but the "port or embed a JS engine" trade-off this section spent most of its length on is likely moot now. - **Build shape: `tokenizers-c` is a Rust `staticlib`, not a `cdylib` -- jmlx cannot load it directly the way `NativeLoader` loads `libmlxc.dylib`.** `tokenizers-cpp/rust/Cargo.toml` declares `crate-type @@ -331,24 +335,42 @@ uses for MLX itself.** Findings, each confirmed against a primary source rather - **One real risk, not previously visible from the plan text alone: Rust-side panics cross the FFI boundary as failures with no recoverable status.** `rust/src/lib.rs`'s wrapper calls `.unwrap()` on `Tokenizer::from_str`/`encode`/`decode` -- a malformed `tokenizer.json` or a decode error panics - inside Rust rather than returning a checkable error code. A Rust panic unwinding across an `extern - "C"` boundary without `catch_unwind` is undefined behavior, not a catchable `MLXException`-style - failure -- structurally worse than mlx-c's own error convention (`printf` + `exit(-1)`, which - `NativeLoader`'s custom handler already replaces) precisely because there is no error-handler hook - to intercept it the way `NativeLoader` intercepts mlx-c's. Whatever plan follows this spike needs + inside Rust rather than returning a checkable error code. **Correction: a Rust panic unwinding + across an `extern "C"` boundary without `catch_unwind` is not undefined behavior -- per the + Rustonomicon's FFI chapter, "panic will cause the process to safely abort" (UB is the reverse + direction: a foreign exception entering Rust). Both this and mlx-c's own error convention (`printf` + + `exit(-1)`) are immediate, defined process death; the genuine distinction is narrower than "UB vs. + catchable" -- mlx-c's exit path is interceptable, and `NativeLoader`'s custom handler already + intercepts it, while a Rust abort offers no hook at all.** Whatever plan follows this spike needs to either wrap every entry point in `catch_unwind` in a jmlx-owned fork of the Rust glue, or explicitly accept malformed-tokenizer-file input as an unrecoverable-crash case (unlike every other failure path in this codebase, which surfaces as a catchable `MLXException`). -- **A build-fragility note, not a blocker:** `tokenizers-cpp`'s `onig` Cargo feature (enabled in its - `Cargo.toml`, needed to replicate Python `regex`-module-exact Unicode splitting for GPT-2/GPT-4-style - BPE pretokenizers) pulls in `onig_sys`, which vendors and compiles an old bundled copy of the - Oniguruma C source when no system library is found via `pkg-config` -- known to hit compiler - compatibility issues on newer GCC (unconfirmed either way against Apple's clang on this repo's - actual macOS 26/Apple Silicon target, since that combination has not yet been built here). Whether - M2 needs `onig` at all depends on which reference models' tokenizers M3 actually targets: Llama/Qwen - both use byte-level BPE without the exact GPT-2 regex-split behavior `onig` exists for, so it may be - possible to build with `default-features = false` and skip `onig` entirely, avoiding this risk - rather than resolving it. +- **A second FFI hazard in the same header, independent of the panic risk above: a signedness + mismatch between encode and decode.** `TokenizerEncodeResult` declares `int* token_ids`, but + `tokenizers_decode` takes `const uint32_t* data` -- confirmed against the upstream header, not just + this document's own transcription of it. An FFM binding would read `token_ids` as + `ValueLayout.JAVA_INT` (signed) and need to pass it back as unsigned for decode; not exploitable + with current vocab sizes (no token ID reaches `Integer.MAX_VALUE`), but it's exactly the kind of + layout mismatch this codebase's own binding conventions otherwise pin down explicitly, and worth + naming alongside the panic risk rather than leaving implicit in the quoted header above. +- **A hard requirement of the FFM path for M3's actual target models, not an escapable build-fragility + note.** `tokenizers-cpp/rust/Cargo.toml` declares `tokenizers = { version = "0.21.2", + default-features = false, features = ["onig"] }` -- `onig` is already an explicit, deliberate + opt-in feature, not something inherited from upstream's own `default = ["progressbar", "onig", + "esaxx_fast"]`, so passing `default-features = false` (already set) does nothing further; skipping + `onig` means removing it from that explicit feature list, not flipping a flag. Doing so is not an + option for Llama/Qwen specifically: both use exactly the GPT-2-style regex split `onig` exists for, + confirmed against `llama.cpp/src/llama-vocab.cpp` -- QWEN2's pretokenizer regex is + `[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*` plus `\s+(?!\S)`, LLAMA3's differs only in + `\p{N}{1,3}` -- both rely on Unicode property classes and a negative lookahead, which is exactly + what Rust's `regex` crate cannot express and precisely why `onig` is there. There is also no + supported non-`onig` substitute reachable on this target: the only alternative engine, + `fancy-regex`, is gated behind upstream's `unstable_wasm = ["fancy-regex", "getrandom/wasm_js"]`, + which drags in a wasm `getrandom` backend -- wrong for `aarch64-apple-darwin`. `onig_sys` vendoring + and compiling a bundled Oniguruma C source when no system library is found via `pkg-config` (known + to hit compiler-compatibility issues on newer GCC, unconfirmed either way against Apple's clang on + this repo's actual macOS 26/Apple Silicon target) is therefore a real build-time cost the FFM path + must carry, not a risk it can build its way around. **Still open, deliberately not resolved by this desk-research pass:** actually building a minimal `cdylib` from a jmlx-owned fork of `tokenizers-cpp/rust` (or from scratch against the plain @@ -356,10 +378,9 @@ uses for MLX itself.** Findings, each confirmed against a primary source rather original "prototyping load-time cost" requirement. This machine has no Rust toolchain installed (`cargo`/`rustc` both absent); doing so is an environment change worth confirming with a human before taking, not something to do unilaterally mid-spike. `req/plans/phase5-m2-plan.md` should not be -written until that prototyping step also lands -- the desk research above resolves the *architecture* -question (FFM-bind a plain-C shim, most likely a jmlx-owned fork of `tokenizers-cpp/rust` scoped to -HF-JSON + byte-level-BPE only, skipping the C++/SentencePiece layer) but not the *cost* question D3 -also asked for. +written until that prototyping step also lands, nor until the FFM-vs-pure-Java architecture choice +itself is actually made -- the desk research above narrows both (see D3's amendment for the pure-Java +side, and this bullet's own onig/panic findings for the FFM side's real cost) but resolves neither. **D4 — Reference models (M3) are pure composition, deferred until M1 and M2 both land.** `LlamaModel`/`QwenModel` need nothing new at the tensor/module level: `se.alipsa.jmlx.nn` already @@ -519,19 +540,21 @@ named scope boundary rather than left implicit. lines for tokenization alone, ~7,700 lines with full chat-template fidelity if porting Jinja from HF's own smaller, more current JS implementation (`@huggingface/jinja`) rather than `swift-jinja`, or ~10,500 from the latter — real, quantified numbers to weigh against the FFM path's toolchain - cost, not an open unknown anymore. **Largely mooted since: `hfjinja` + cost, not an open unknown anymore. **The porting question is largely mooted since: `hfjinja` (`github.com/Alipsa/hfjinja`) is a released, dependency-free Java 21+ port of `@huggingface/jinja` itself — i.e. option 2 already done, by the same org as jmlx — so the chat-template half of the - pure-Java estimate (~3,860 of the ~7,700 lines) doesn't need porting at all if adopted; see D3's - amendment for what's still unverified (Maven coordinate, model-coverage of its differential test - corpus).** What remains genuinely open regardless of which tokenization direction (FFM vs. - pure-Java) is chosen: real load-time cost for the FFM path (needs an actual build-and-measure - prototype, blocked on a Rust toolchain decision), whether `onig` is actually needed for M3's target - models if the FFM path is chosen, and whether M3's reference models need general Jinja2 - chat-template evaluation at all or can get away with hand-formatting a small, known set of chat - templates instead (D4 explicitly defers M3's own requirements, so this isn't decidable from this - document alone) — though if `hfjinja` is adopted, that last question loses most of its urgency too, - since the "port vs. hand-format" trade-off it was weighing is no longer a real port. + pure-Java estimate (~3,860 of the ~7,700 lines) doesn't need porting at all if adopted. Its adoption + cost is not moot, though: it is confirmed absent from Maven Central (`numFound: 0`), so using it + today means JitPack, a source/composite build, or publishing it first — see D3's amendment.** What + remains genuinely open regardless of which tokenization direction (FFM vs. pure-Java) is chosen: + real load-time cost for the FFM path (needs an actual build-and-measure prototype, blocked on a + Rust toolchain decision); and, confirmed rather than open on the FFM side specifically, `onig` is a + hard requirement for Llama/Qwen's actual pretokenizer regex if that path is chosen, not an + escapable build-fragility note (see D3's amendment). Whether M3's reference models need general + Jinja2 chat-template evaluation at all or can get away with hand-formatting a small, known set of + chat templates instead is still deferred to M3's own requirements (D4), though if `hfjinja` is + adopted and its Maven-availability cost is paid, that question loses most of its urgency, since the + "port vs. hand-format" trade-off it was weighing is no longer a real port. No open question remains on the checkpoint-I/O (M1) side: `mlx_vector_string_get`'s ownership, the last unresolved item blocking `loadGguf`'s design, is settled — see Research findings above. From 49c0e8a418ec7f2ce2d298b41e2efc9abeb20f6b Mon Sep 17 00:00:00 2001 From: pernyf Date: Sun, 30 Aug 2026 15:23:28 +0200 Subject: [PATCH 08/11] =?UTF-8?q?Phase=205=20M2=20spike:=20onig=20fix=20ov?= =?UTF-8?q?er-corrected=20=E2=80=94=20fancy-regex=20is=20a=20real=20non-wa?= =?UTF-8?q?sm=20alternative?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous fix's "no supported non-onig substitute" claim was wrong: unstable_wasm bundles fancy-regex with a wasm getrandom backend, it doesn't gate fancy-regex itself. Verified against tokenizers 0.21.2's own Cargo.toml (fancy-regex is its own optional-dependency feature) and src/utils/mod.rs (cfg requires exactly one of onig/fancy-regex, confirming a regex engine is mandatory but onig specifically isn't). default-features = false, features = ["fancy-regex"] is a valid non-wasm config on aarch64-apple-darwin, and since fancy-regex is pure Rust, it would eliminate the onig_sys/Oniguruma C-vendoring fragility entirely -- the cheapest available build-cost reduction on the FFM path, not something to argue away. Left open: whether fancy-regex is byte-identical to onig on Llama-3/Qwen2's specific patterns, which belongs in the load-time-cost prototype this document already defers. Also: tightened "differs only in \p{N}{1,3}" to "differs mainly in" per llama.cpp's own wording, and noted the Rust fork proposal inherits an upstream tokenizers = "0.21.2" pin that should follow this repo's existing MLX_C_COMMIT / checkDependencies.zsh precedent. Co-Authored-By: Claude Sonnet 5 --- req/phase5-plan.md | 58 ++++++++++++++++++++++++++++------------------ 1 file changed, 36 insertions(+), 22 deletions(-) diff --git a/req/phase5-plan.md b/req/phase5-plan.md index 09c23c9..611621f 100644 --- a/req/phase5-plan.md +++ b/req/phase5-plan.md @@ -314,7 +314,11 @@ uses for MLX itself.** Findings, each confirmed against a primary source rather jmlx-owned fork of `rust/Cargo.toml` + `rust/src/lib.rs` with `crate-type = ["cdylib"]` instead -- the latter is simpler since it also sidesteps needing the C++/CMake/submodule machinery (`sentencepiece`, `msgpack`) that only the C++ layer requires, if HF-JSON-only scope (see above) - is accepted for M2. + is accepted for M2. Either way, this fork inherits an upstream pin the same way `bootstrap-native.sh` + pins `MLX_C_COMMIT`: `rust/Cargo.toml` pins `tokenizers = "0.21.2"` (see the `onig`/`fancy-regex` + bullet below), and a jmlx-owned fork would own re-pinning it on the same + `scripts/checkDependencies.zsh`/`scripts/updateMlx.zsh`-style discipline this repo already has for + mlx-c, not a new maintenance burden invented for M2. - **A Rust toolchain is unavoidable for this whole path -- there is no way to use `tokenizers-cpp` (or any fork of it) without one.** The actual tokenizer logic is the upstream `tokenizers` Rust crate itself; `tokenizers-cpp`'s C and C++ layers are thin wrapper headers that call into a Rust @@ -353,24 +357,32 @@ uses for MLX itself.** Findings, each confirmed against a primary source rather with current vocab sizes (no token ID reaches `Integer.MAX_VALUE`), but it's exactly the kind of layout mismatch this codebase's own binding conventions otherwise pin down explicitly, and worth naming alongside the panic risk rather than leaving implicit in the quoted header above. -- **A hard requirement of the FFM path for M3's actual target models, not an escapable build-fragility - note.** `tokenizers-cpp/rust/Cargo.toml` declares `tokenizers = { version = "0.21.2", - default-features = false, features = ["onig"] }` -- `onig` is already an explicit, deliberate - opt-in feature, not something inherited from upstream's own `default = ["progressbar", "onig", - "esaxx_fast"]`, so passing `default-features = false` (already set) does nothing further; skipping - `onig` means removing it from that explicit feature list, not flipping a flag. Doing so is not an - option for Llama/Qwen specifically: both use exactly the GPT-2-style regex split `onig` exists for, - confirmed against `llama.cpp/src/llama-vocab.cpp` -- QWEN2's pretokenizer regex is - `[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*` plus `\s+(?!\S)`, LLAMA3's differs only in - `\p{N}{1,3}` -- both rely on Unicode property classes and a negative lookahead, which is exactly - what Rust's `regex` crate cannot express and precisely why `onig` is there. There is also no - supported non-`onig` substitute reachable on this target: the only alternative engine, - `fancy-regex`, is gated behind upstream's `unstable_wasm = ["fancy-regex", "getrandom/wasm_js"]`, - which drags in a wasm `getrandom` backend -- wrong for `aarch64-apple-darwin`. `onig_sys` vendoring - and compiling a bundled Oniguruma C source when no system library is found via `pkg-config` (known - to hit compiler-compatibility issues on newer GCC, unconfirmed either way against Apple's clang on - this repo's actual macOS 26/Apple Silicon target) is therefore a real build-time cost the FFM path - must carry, not a risk it can build its way around. +- **Not an escapable build-fragility note, but not a hard requirement of the FFM path either: exactly + one of `onig`/`fancy-regex` must be enabled, and `fancy-regex` is a real, pure-Rust alternative.** + `tokenizers-cpp/rust/Cargo.toml` declares `tokenizers = { version = "0.21.2", default-features = + false, features = ["onig"] }` -- `onig` is already an explicit, deliberate opt-in feature, not + something inherited from upstream's own `default = ["progressbar", "onig", "esaxx_fast"]`, so + passing `default-features = false` (already set) does nothing further; upstream's own + `tokenizers/src/utils/mod.rs` requires one of the two (`#[cfg(not(any(feature = "onig", feature = + "fancy-regex")))] compile_error!(...)`), so "skip the regex engine entirely" -- this bullet's + original framing -- was never on the table. Skipping `onig` specifically is not an option for + Llama/Qwen: both use exactly the GPT-2-style regex split `onig`/`fancy-regex` exist for, confirmed + against `llama.cpp/src/llama-vocab.cpp` -- QWEN2's pretokenizer regex is + `[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*` plus `\s+(?!\S)`, LLAMA3's differs mainly + in the numeric portion (`\p{N}{1,3}`) -- both rely on Unicode property classes and a negative + lookahead, which Rust's plain `regex` crate cannot express. **But `fancy-regex` is a supported, + reachable substitute, not gated behind `unstable_wasm` as an earlier pass here claimed:** upstream's + `Cargo.toml` declares `fancy-regex = { version = "0.14", optional = true }` as its own + implicitly-named feature, and `unstable_wasm = ["fancy-regex", "getrandom/wasm_js"]` merely bundles + it with a wasm `getrandom` backend for a different (wasm) target -- it does not gate `fancy-regex` + itself. `default-features = false, features = ["fancy-regex"]` is a valid non-wasm configuration on + `aarch64-apple-darwin`, and since `fancy-regex` is pure Rust, choosing it would eliminate the + `onig_sys`/Oniguruma-C-vendoring/`pkg-config`/GCC-compatibility fragility entirely -- the cheapest + available reduction in the FFM path's build cost, not something this document should argue away. + Not verified here: whether `fancy-regex` is byte-identical to `onig` on Llama-3/Qwen2's specific + patterns -- both engines support the lookaround and `\p{...}` classes those patterns use, but + split-behavior equivalence for these exact regexes is unconfirmed, and belongs in the load-time-cost + prototype this document already defers, not asserted here. **Still open, deliberately not resolved by this desk-research pass:** actually building a minimal `cdylib` from a jmlx-owned fork of `tokenizers-cpp/rust` (or from scratch against the plain @@ -548,9 +560,11 @@ named scope boundary rather than left implicit. today means JitPack, a source/composite build, or publishing it first — see D3's amendment.** What remains genuinely open regardless of which tokenization direction (FFM vs. pure-Java) is chosen: real load-time cost for the FFM path (needs an actual build-and-measure prototype, blocked on a - Rust toolchain decision); and, confirmed rather than open on the FFM side specifically, `onig` is a - hard requirement for Llama/Qwen's actual pretokenizer regex if that path is chosen, not an - escapable build-fragility note (see D3's amendment). Whether M3's reference models need general + Rust toolchain decision), including which regex backend to build it with -- Llama/Qwen's + pretokenizer regexes need lookahead/Unicode-class support that only `onig` or `fancy-regex` provide + (plain `regex` is ruled out either way), but whether `fancy-regex`'s pure-Rust split behavior + actually matches `onig`'s on these specific patterns is unconfirmed and belongs in that same + prototype (see D3's amendment). Whether M3's reference models need general Jinja2 chat-template evaluation at all or can get away with hand-formatting a small, known set of chat templates instead is still deferred to M3's own requirements (D4), though if `hfjinja` is adopted and its Maven-availability cost is paid, that question loses most of its urgency, since the From 24cfc23d42b99cd9bb2fcd7d517c723bdc7c4ebb Mon Sep 17 00:00:00 2001 From: pernyf Date: Sun, 30 Aug 2026 15:34:42 +0200 Subject: [PATCH 09/11] Phase 5 M2 spike: restore dangling clang caveat, drop reviewer-voice, fix bullet reference - Line 333's forward reference to "the ambiguity noted below about onig's clang compatibility" pointed at nothing: 49c0e8a's rewrite compressed the onig bullet's original GCC/clang caveat away. Restored it (onig_sys vendors/compiles Oniguruma C via pkg-config, known GCC issues, unconfirmed against Apple's clang on this repo's actual target) so the pointer resolves again. - Dropped "not something this document should argue away" -- reviewer- reply voice with no reader, unlike this doc's established self-correcting **Amendment:**-style convention. - "this bullet's own onig/panic findings" misdescribed its own location -- the passage it's in is a standalone paragraph after the bullet list, not a bullet itself. Now "the onig and panic bullets above". Co-Authored-By: Claude Sonnet 5 --- req/phase5-plan.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/req/phase5-plan.md b/req/phase5-plan.md index 611621f..8783f99 100644 --- a/req/phase5-plan.md +++ b/req/phase5-plan.md @@ -376,10 +376,13 @@ uses for MLX itself.** Findings, each confirmed against a primary source rather implicitly-named feature, and `unstable_wasm = ["fancy-regex", "getrandom/wasm_js"]` merely bundles it with a wasm `getrandom` backend for a different (wasm) target -- it does not gate `fancy-regex` itself. `default-features = false, features = ["fancy-regex"]` is a valid non-wasm configuration on - `aarch64-apple-darwin`, and since `fancy-regex` is pure Rust, choosing it would eliminate the - `onig_sys`/Oniguruma-C-vendoring/`pkg-config`/GCC-compatibility fragility entirely -- the cheapest - available reduction in the FFM path's build cost, not something this document should argue away. - Not verified here: whether `fancy-regex` is byte-identical to `onig` on Llama-3/Qwen2's specific + `aarch64-apple-darwin`, and since `fancy-regex` is pure Rust, choosing it would eliminate entirely + the fragility `onig_sys` carries: it vendors and compiles a bundled Oniguruma C source when no + system library is found via `pkg-config`, known to hit compiler compatibility issues on newer GCC + and unconfirmed either way against Apple's clang on this repo's actual macOS 26/Apple Silicon + target, since that combination has not yet been built here. This is the cheapest available + reduction in the FFM path's build cost. Not verified here: whether `fancy-regex` is byte-identical + to `onig` on Llama-3/Qwen2's specific patterns -- both engines support the lookaround and `\p{...}` classes those patterns use, but split-behavior equivalence for these exact regexes is unconfirmed, and belongs in the load-time-cost prototype this document already defers, not asserted here. @@ -392,7 +395,7 @@ original "prototyping load-time cost" requirement. This machine has no Rust tool taking, not something to do unilaterally mid-spike. `req/plans/phase5-m2-plan.md` should not be written until that prototyping step also lands, nor until the FFM-vs-pure-Java architecture choice itself is actually made -- the desk research above narrows both (see D3's amendment for the pure-Java -side, and this bullet's own onig/panic findings for the FFM side's real cost) but resolves neither. +side, and the onig and panic bullets above for the FFM side's real cost) but resolves neither. **D4 — Reference models (M3) are pure composition, deferred until M1 and M2 both land.** `LlamaModel`/`QwenModel` need nothing new at the tensor/module level: `se.alipsa.jmlx.nn` already From 38bd70139545858dd2997922bef661e712f06c6f Mon Sep 17 00:00:00 2001 From: pernyf Date: Sun, 30 Aug 2026 15:45:27 +0200 Subject: [PATCH 10/11] Phase 5 M2 spike: onig bullet headline still claimed what its own body retracted "Not an escapable build-fragility note" was a holdover from the previous revision's withdrawn conclusion ("a real build-time cost the FFM path must carry") -- the body it introduces actually shows the opposite: fancy-regex eliminates the Oniguruma fragility entirely, so it is escapable. A reader skimming bullet headlines got the retracted answer. Rewrote the headline to match only what the body proves: a regex backend is mandatory, but onig specifically isn't, and fancy-regex avoids its build fragility. Also reflowed a mid-paragraph orphan line left over from the last two edits. Co-Authored-By: Claude Sonnet 5 --- req/phase5-plan.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/req/phase5-plan.md b/req/phase5-plan.md index 8783f99..ea59692 100644 --- a/req/phase5-plan.md +++ b/req/phase5-plan.md @@ -357,8 +357,9 @@ uses for MLX itself.** Findings, each confirmed against a primary source rather with current vocab sizes (no token ID reaches `Integer.MAX_VALUE`), but it's exactly the kind of layout mismatch this codebase's own binding conventions otherwise pin down explicitly, and worth naming alongside the panic risk rather than leaving implicit in the quoted header above. -- **Not an escapable build-fragility note, but not a hard requirement of the FFM path either: exactly - one of `onig`/`fancy-regex` must be enabled, and `fancy-regex` is a real, pure-Rust alternative.** +- **A regex backend is unavoidable, but `onig` specifically is not: exactly one of + `onig`/`fancy-regex` must be enabled, and `fancy-regex` is a real, pure-Rust alternative that + avoids the Oniguruma build fragility.** `tokenizers-cpp/rust/Cargo.toml` declares `tokenizers = { version = "0.21.2", default-features = false, features = ["onig"] }` -- `onig` is already an explicit, deliberate opt-in feature, not something inherited from upstream's own `default = ["progressbar", "onig", "esaxx_fast"]`, so @@ -382,8 +383,8 @@ uses for MLX itself.** Findings, each confirmed against a primary source rather and unconfirmed either way against Apple's clang on this repo's actual macOS 26/Apple Silicon target, since that combination has not yet been built here. This is the cheapest available reduction in the FFM path's build cost. Not verified here: whether `fancy-regex` is byte-identical - to `onig` on Llama-3/Qwen2's specific - patterns -- both engines support the lookaround and `\p{...}` classes those patterns use, but + to `onig` on Llama-3/Qwen2's specific patterns -- both engines support the lookaround and + `\p{...}` classes those patterns use, but split-behavior equivalence for these exact regexes is unconfirmed, and belongs in the load-time-cost prototype this document already defers, not asserted here. From 3fd2f975f8be69551235a8f6cc90669cc5d9a145 Mon Sep 17 00:00:00 2001 From: pernyf Date: Sun, 30 Aug 2026 16:21:30 +0200 Subject: [PATCH 11/11] Phase 5 M2 spike: onig bullet's mid-body clause still asserted the retracted conclusion Line 369 said "skipping onig specifically is not an option for Llama/Qwen," directly contradicting the bullet's own (already-fixed) headline and its later "fancy-regex is a supported, reachable substitute" clause -- a survivor of the withdrawn 49c0e8a conclusion that only got its trailing half patched. Replaced with what the preceding sentence already establishes: Llama/Qwen force the capability (lookaround + Unicode classes), not the crate. Reflowed the whole bullet as one block instead of patching lines in isolation, per review feedback that framing sentences had been edited on a different schedule than the body evidence -- also resolves the remaining mid-sentence orphan line from the last two passes. Co-Authored-By: Claude Sonnet 5 --- req/phase5-plan.md | 55 +++++++++++++++++++++++----------------------- 1 file changed, 27 insertions(+), 28 deletions(-) diff --git a/req/phase5-plan.md b/req/phase5-plan.md index ea59692..03e6626 100644 --- a/req/phase5-plan.md +++ b/req/phase5-plan.md @@ -359,34 +359,33 @@ uses for MLX itself.** Findings, each confirmed against a primary source rather naming alongside the panic risk rather than leaving implicit in the quoted header above. - **A regex backend is unavoidable, but `onig` specifically is not: exactly one of `onig`/`fancy-regex` must be enabled, and `fancy-regex` is a real, pure-Rust alternative that - avoids the Oniguruma build fragility.** - `tokenizers-cpp/rust/Cargo.toml` declares `tokenizers = { version = "0.21.2", default-features = - false, features = ["onig"] }` -- `onig` is already an explicit, deliberate opt-in feature, not - something inherited from upstream's own `default = ["progressbar", "onig", "esaxx_fast"]`, so - passing `default-features = false` (already set) does nothing further; upstream's own - `tokenizers/src/utils/mod.rs` requires one of the two (`#[cfg(not(any(feature = "onig", feature = - "fancy-regex")))] compile_error!(...)`), so "skip the regex engine entirely" -- this bullet's - original framing -- was never on the table. Skipping `onig` specifically is not an option for - Llama/Qwen: both use exactly the GPT-2-style regex split `onig`/`fancy-regex` exist for, confirmed - against `llama.cpp/src/llama-vocab.cpp` -- QWEN2's pretokenizer regex is - `[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*` plus `\s+(?!\S)`, LLAMA3's differs mainly - in the numeric portion (`\p{N}{1,3}`) -- both rely on Unicode property classes and a negative - lookahead, which Rust's plain `regex` crate cannot express. **But `fancy-regex` is a supported, - reachable substitute, not gated behind `unstable_wasm` as an earlier pass here claimed:** upstream's - `Cargo.toml` declares `fancy-regex = { version = "0.14", optional = true }` as its own - implicitly-named feature, and `unstable_wasm = ["fancy-regex", "getrandom/wasm_js"]` merely bundles - it with a wasm `getrandom` backend for a different (wasm) target -- it does not gate `fancy-regex` - itself. `default-features = false, features = ["fancy-regex"]` is a valid non-wasm configuration on - `aarch64-apple-darwin`, and since `fancy-regex` is pure Rust, choosing it would eliminate entirely - the fragility `onig_sys` carries: it vendors and compiles a bundled Oniguruma C source when no - system library is found via `pkg-config`, known to hit compiler compatibility issues on newer GCC - and unconfirmed either way against Apple's clang on this repo's actual macOS 26/Apple Silicon - target, since that combination has not yet been built here. This is the cheapest available - reduction in the FFM path's build cost. Not verified here: whether `fancy-regex` is byte-identical - to `onig` on Llama-3/Qwen2's specific patterns -- both engines support the lookaround and - `\p{...}` classes those patterns use, but - split-behavior equivalence for these exact regexes is unconfirmed, and belongs in the load-time-cost - prototype this document already defers, not asserted here. + avoids the Oniguruma build fragility.** `tokenizers-cpp/rust/Cargo.toml` declares `tokenizers = { + version = "0.21.2", default-features = false, features = ["onig"] }` -- `onig` is already an + explicit, deliberate opt-in feature, not something inherited from upstream's own `default = + ["progressbar", "onig", "esaxx_fast"]`, so passing `default-features = false` (already set) does + nothing further; upstream's own `tokenizers/src/utils/mod.rs` requires one of the two + (`#[cfg(not(any(feature = "onig", feature = "fancy-regex")))] compile_error!(...)`), so "skip the + regex engine entirely" -- this bullet's original framing -- was never on the table. What + Llama/Qwen force is the capability, not the crate: both use exactly the GPT-2-style regex split + `onig`/`fancy-regex` exist for, confirmed against `llama.cpp/src/llama-vocab.cpp` -- QWEN2's + pretokenizer regex is `[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*` plus `\s+(?!\S)`, + LLAMA3's differs mainly in the numeric portion (`\p{N}{1,3}`) -- both rely on Unicode property + classes and a negative lookahead, which Rust's plain `regex` crate cannot express. **But + `fancy-regex` is a supported, reachable substitute, not gated behind `unstable_wasm` as an earlier + pass here claimed:** upstream's `Cargo.toml` declares `fancy-regex = { version = "0.14", optional = + true }` as its own implicitly-named feature, and `unstable_wasm = ["fancy-regex", + "getrandom/wasm_js"]` merely bundles it with a wasm `getrandom` backend for a different (wasm) + target -- it does not gate `fancy-regex` itself. `default-features = false, features = + ["fancy-regex"]` is a valid non-wasm configuration on `aarch64-apple-darwin`, and since + `fancy-regex` is pure Rust, choosing it would eliminate entirely the fragility `onig_sys` carries: + it vendors and compiles a bundled Oniguruma C source when no system library is found via + `pkg-config`, known to hit compiler compatibility issues on newer GCC and unconfirmed either way + against Apple's clang on this repo's actual macOS 26/Apple Silicon target, since that combination + has not yet been built here. This is the cheapest available reduction in the FFM path's build + cost. Not verified here: whether `fancy-regex` is byte-identical to `onig` on Llama-3/Qwen2's + specific patterns -- both engines support the lookaround and `\p{...}` classes those patterns use, + but split-behavior equivalence for these exact regexes is unconfirmed, and belongs in the + load-time-cost prototype this document already defers, not asserted here. **Still open, deliberately not resolved by this desk-research pass:** actually building a minimal `cdylib` from a jmlx-owned fork of `tokenizers-cpp/rust` (or from scratch against the plain