feat: performance, adding pcre2 backend + regex-shards (5-15% speedup)

[michaelfeil]

Tips and tricks from a Blog post describing how this lib is 4-10x slower than it needs to be. https://www.crusoe.ai/resources/blog/reducing-ttft-by-cpumaxxing-tokenization

Performance: PCRE2 JIT regex backend

Benchmarked on LLaMA 3 tokenizer (data/llama-3-tokenizer.json) with data/big.txt (6.5 MB, 128K lines).

Setup

• Baseline: main branch with default onig (Oniguruma) regex backend
• Test: same branch with --no-default-features --features pcre2
• Hardware: Linux, Intel Xeon
• Benchmark: cargo bench --bench llama3_benchmark

Results

Benchmark	onig (main)	pcre2 JIT	Change
Sequential encode (full file)	1.82 s	1.63 s	-10 to -14%
Batch encode (1000 items)	737 ms	658 ms	~-5 to -11%
Offsets (batch + char offsets)	295 ms	288 ms	~-3%
BPE Train	~10 s	~10 s	no change

Sequential encode is the most stable measurement. Batch/offsets show improvement but have higher run-to-run variance due to thread scheduling.

Why

PCRE2 with JIT compiles regex patterns to native machine code at initialization. The GPT-2 byte-level pre-tokenization regex ('s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+) is the hot path for every encode() call and benefits directly from JIT compilation vs Oniguruma's interpreted matching.

Usage

# Cargo.toml
[dependencies]
tokenizers = { version = "0.22", features = ["pcre2"] }

The pcre2 feature is opt-in. When enabled, it takes priority over onig and fancy-regex. Cross-compilation works the same as onig — pcre2-sys bundles and compiles PCRE2 from C source via the cc crate. Not available on WASM (use fancy-regex there, as before).

[wheynelau]

@michaelfeil Thanks for providing more details! I didn't know that fasttokens used pcre2 as well, just learnt about it yesterday.

[wheynelau]

Curious as you are active on this, have you tried thread local caches to see if they help? Or other distributed caches like dashmap and parking lot? Thank you!

[michaelfeil]

@wheynelau thats the other PR. #1967

I think thread_local caches are most useful if re-used in a rayon pool, or if the threads are in some kind of pool.
For the L1 in the blog post, I tried to implement something, and gave it to claude, but both did not work. Implementing the shared lookup is a bit better IMO, since hashing is fast + you only need to loopup by hash, so the only place that uses RwLock is guarded.

[McPatate]

Good stuff! Didn't repro the benchmarks yet, but looking good so far! Thank you for taking the time.

[McPatate]

this doesn't need to be a Result or am I missing something?

[McPatate]

and pcre2-sys is built

which is always?

[McPatate]

same here for the Result ?

[McPatate]

I think we have the dependency on thiserror, would much rather have a proper defined error type

[McPatate]

or perhaps at least use crate::Result instead

[McPatate]

But compile is not thread safe I assume?

[McPatate]

why is one declared with the dep: prefix and not the other?

[McPatate]

I'd add at least one extra test to make sure the fallback logic is sane here.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.