Phase 5 M2 desk-research spike: tokenizer integration architecture - #13
Open
perNyfelt wants to merge 5 commits into
Open
Phase 5 M2 desk-research spike: tokenizer integration architecture#13perNyfelt wants to merge 5 commits into
perNyfelt wants to merge 5 commits into
Conversation
Resolves D3's architecture question (FFM-bind a C shim vs. pure Java) in favor of the former, based on primary-source research: - huggingface/tokenizers issue #185 confirms no official C/C++ API was ever planned (closed stale, no maintainer commitment). - mlc-ai/tokenizers-cpp provides a maintained (pushed 2026-05-20), Apache-2.0-licensed, genuinely plain-C shim (tokenizers_c.h) over the same Rust crate -- the same extern "C" shape mlx-c itself presents, covering HF tokenizer.json and byte-level BPE directly without needing its C++/SentencePiece layer for M3's Llama/Qwen targets. - DJL (Deep Java Library) independently chose the same wrap-the-Rust-crate approach over reimplementing in Java, confirming this from a team solving the identical problem for a different host language. - Two real risks are flagged for whatever plan follows: tokenizers-c is a staticlib (needs an extra link step or a forked cdylib crate-type), and its Rust glue calls .unwrap() with no catch_unwind, so a malformed tokenizer.json currently panics across the FFI boundary rather than surfacing as a catchable error. Also fixes this document's own stale M1 status (branch/PR/commit references predate PR #12's merge to main). Still open, deliberately not resolved here: an actual build-and-measure load-time prototype, which needs a Rust toolchain not currently installed on this machine -- confirming with the user before installing one rather than doing so unilaterally. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
tokenizers-cpp's C/C++ layers are thin wrappers around the actual tokenizer logic, which is Rust (the upstream tokenizers crate itself). Confirmed via CMakeLists.txt (explicitly targets aarch64-apple-darwin, shells out to cargo build) and its GitHub releases (both tags publish zero binary assets -- no prebuilt library to download the way bootstrap-native.sh already does for MLX itself). Rust becomes a new wherever-this-builds toolchain dependency this project doesn't otherwise have, unlike mlx-c's C/C++ requirement, which Xcode Command Line Tools already satisfies. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… own Swift port un-settles the architecture choice Directly answers whether deferring M2 until mlx-c adds tokenizer support is viable: no. mlx-c's own README scopes it as a low-level array/tensor C API mirroring MLX's own scope, and MLX's two other official language bindings (mlx-lm, mlx-swift-lm) both already solved tokenization by depending on an external tokenizer package rather than asking MLX/mlx-c for it -- no roadmap signal anywhere suggests that will change. While checking mlx-swift-lm's own tokenizer dependency (huggingface/swift-transformers), found it's a from-scratch pure-Swift reimplementation (BPETokenizer.swift, UnigramTokenizer.swift, Normalizer.swift, etc.), not a Rust FFI wrapper -- confirmed via its Sources/Tokenizers/ file list and its own TokenizersTests suite, actively maintained by Hugging Face themselves. This corrects an overstated conclusion from the prior commit: DJL's Rust+JNI choice was framed as "independent confirmation" that FFM-binding beats pure Java. It doesn't settle that -- Hugging Face chose the opposite, for the same problem, in a comparably-shaped language. Both directions now have a credible, maintained precedent. What actually changes: "pure Java" is no longer the vaguely-scoped option D3 originally framed it as -- there's a concrete reference implementation, from the organization that owns the tokenizer.json format, to port from instead of reverse-engineering from spec alone. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
swift-transformers/Sources/Tokenizers/ totals ~3,850 lines across 12 files -- a real number, not a guess. Of Tokenizer.swift's 1041 lines, only AutoTokenizer's ~100 lines are Hub-download convenience code a port would drop (MLXIO's own file-loading precedent already covers "load from a local string" instead). The real complication found while measuring: applyChatTemplate is part of the core Tokenizer protocol, not an add-on, and depends on `import Jinja` -- swift-jinja, a full Jinja2 template engine, confirmed via Package.swift listing it as a direct (not test-only) dependency of the Tokenizers target. swift-jinja/Sources/Jinja/ totals ~6,660 lines on its own, larger than the tokenizer pipeline itself. So the honest total is ~3,850 lines (tokenize/detokenize only) or ~10,500 lines (with full chat-template fidelity), not a single number. Named but not resolved: M3's actual models (a known, small set, not arbitrary Hub models) may not need general Jinja2 evaluation at all -- hand-formatting their specific known chat templates would sidestep the ~6,660-line Jinja port, mirroring this codebase's existing ship-exactly-what's-needed convention. That scoping call depends on M3's own requirements, which this document explicitly defers (D4). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…intained porting source and a third no-port option Direct answer: no official Java equivalent of swift-jinja exists. Jinjava (HubSpot, Apache-2.0) is the closest existing Java Jinja-syntax engine but was built for HubSpot's CMS content, not HF chat templates specifically -- its compatibility with real Llama/Qwen templates is unverified, and Maven Repository flags open CVEs against it (likely a different threat model than jmlx's trusted-input case, but asserted here, not verified). Two more useful findings while checking: - HF's own JS Jinja implementation (@huggingface/jinja, MIT, pushed within the last day) is ~3,860 lines -- ~42% smaller than swift-jinja's ~6,660 -- and its e2e test suite ships verbatim chat_template strings and expected-output goldens for meta-llama/Llama-3.1-8B-Instruct, Qwen/Qwen2.5-7B-Instruct, and Qwen/Qwen3-0.6B specifically -- M3's own named targets. Porting from this source instead of swift-jinja revises the full-fidelity estimate from ~10,500 down to ~7,700 lines. - A third option needs no Jinja port at all: GraalJS runs on stock OpenJDK (no GraalVM install, no native toolchain) via org.graalvm.polyglot:js-community Maven artifacts, and could run @huggingface/jinja's actual JS directly -- reusing HF's own tested implementation with zero compatibility-verification risk, at the cost of a genuinely heavyweight dependency for rendering one short template string per model load. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Desk-research spike for Phase 5 M2 (tokenizer integration), per Decision D3 in
req/phase5-plan.md(M2 requires its own research spike before an implementation plan can be written). This PR is documentation-only — noreq/plans/phase5-m2-plan.mdyet, and no code changes.mlc-ai/tokenizers-cpp(Apache-2.0) is the viable third-party C shim over the Rusttokenizerscrate, but ships as a Ruststaticlibwith no prebuilt binaries — Rust toolchain is unavoidable on this path.mlx-lm,mlx-swift-lm) solve tokenization via external dependencies rather than MLX proper. Waiting for upstream is ruled out.swift-transformers(used bymlx-swift-lm) is a pure-Swift reimplementation, not a Rust FFI wrapper — this cuts against an earlier draft's overstated claim that DJL's Rust/JNI choice "confirms" FFM-binding is the better path. The FFM-bind vs. pure-Java-port choice is genuinely open, not settled by precedent.swift-transformers's tokenizer core is ~3,850 lines; chat templates require Jinja (applyChatTemplateis non-optional on theTokenizerprotocol), andswift-jinjais ~6,660 lines, putting a full-fidelity port at ~10,500 lines.swift-jinjaexists. Jinjava (HubSpot) is the closest existing library but unverified against real chat templates and carries flagged CVEs. HF's own JS Jinja implementation (@huggingface/jinja, MIT) is smaller (~3,860 lines) and more current thanswift-jinja, with exact Llama/Qwen e2e test goldens already in hand — porting from it instead revises the full-fidelity estimate to ~7,700 lines. A third option (embed GraalJS, run@huggingface/jinja's actual JS with no port) trades porting effort for a heavyweight dependency.What's still open
onigbuild dependency (flagged as a fragility risk intokenizers-cpp's build).Test plan
req/phase5-plan.md🤖 Generated with Claude Code