Skip to content

Pull requests: huggingface/tokenizers

New pull request New
49 Open 776 Closed
49 Open 776 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Implementing Parity-aware BPE
#1974 opened Mar 21, 2026 by cimeister Loading…
Bump flatted from 3.3.1 to 3.4.2 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#1972 opened Mar 20, 2026 by dependabot bot Loading…
1
feat(benchmarks-only): adding longer-context llama3 benchmarks
#1971 opened Mar 19, 2026 by michaelfeil Loading…
Fix node-release: all platforms, zig cross-compilation, universal macOS
#1970 opened Mar 19, 2026 by MayCXC Loading…
2
feat: performance, adding pcre2 backend + regex-shards (5-15% speedup)
#1968 opened Mar 19, 2026 by michaelfeil Loading…
3
feat: Optimize BPE tokenization: sharded cache, packed merge keys, FxHash (10-15% speedup)
#1967 opened Mar 19, 2026 by michaelfeil Loading…
1
feat: add async decode batch type annotations in decoding
#1966 opened Mar 19, 2026 by michaelfeil Loading…
1
Fix type_ids not applied to overflow encodings
#1965 opened Mar 17, 2026 by joaquinhuigomez Loading…
5% BERT benchmark improvement: Remove unnecessary to_vec() from slice()
#1964 opened Mar 16, 2026 by jberg5 Loading…
1
feat: add pcre2 as optional feature
#1959 opened Mar 2, 2026 by wheynelau Loading…
3
feat: Add weakref support to Tokenizer class
#1958 opened Mar 2, 2026 by mrkm4ntr Loading…
3
Bump minimatch from 3.1.3 to 3.1.5 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#1956 opened Feb 28, 2026 by dependabot bot Loading…
1
Update release workflow for Python 3.14
#1952 opened Feb 19, 2026 by ngoldbaum Loading…
6
Add riscv64 build, make Linux wheel build matrix more explicit
#1951 opened Feb 19, 2026 by threexc Loading…
2
Fix identity comparison in toctree_tags.py
#1949 opened Feb 13, 2026 by llukito Loading…
1
Add get_special_tokens and is_special_token methods
#1945 opened Feb 5, 2026 by ArthurZucker Loading…
2 tasks done
1
Add post_process_tokens and post_process_ids methods
#1944 opened Feb 5, 2026 by ArthurZucker Loading…
3 tasks done
1
feat: add unk_token property to Unigram model
#1943 opened Feb 5, 2026 by ArthurZucker Loading…
4 tasks done
1
🚨 feat: add role_to_token field for special token metadata
#1942 opened Feb 5, 2026 by ArthurZucker Loading…
19
Fix broken source links in documentation
#1934 opened Jan 21, 2026 by Shivam-Bhardwaj Loading…
1
fix: added type hints in .py files
#1932 opened Jan 20, 2026 by ashmi8 Loading…
42
Include license file into python wheels
#1931 opened Jan 20, 2026 by justeph Loading…
feat: add progress_format option for machine-readable JSON output
#1921 opened Dec 26, 2025 by podarok Loading…
6 tasks done
1
Upgrade GitHub Actions for Node 24 compatibility
#1916 opened Dec 20, 2025 by salmanmkc Loading…
Use unicode-normalization instead of unicode-normalization-alignments
#1912 opened Dec 14, 2025 by IvanIsCoding Loading…
1
ProTip! Exclude everything labeled bug with -label:bug.