Handle byte_fallback configuration in BPETokenizer#44
Conversation
Co-authored-by: Gunnarguy <110250624+Gunnarguy@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @Gunnarguy, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
Pull request overview
Adds support for the Hugging Face byte_fallback (exposed as byteFallback) setting in the Swift BPETokenizer, and adjusts unknown-token handling to respect that configuration during tokenization.
Changes:
- Added
byteFallbacktoBPETokenizerand initialized it fromtokenizerData.model.byteFallback. - Updated
tokenize(text:)to emit the configured unknown token whenbyteFallbackis disabled, instead of always byte/hex-fallback encoding unknown tokens.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if !byteFallback, let unknownToken = unknownToken { | ||
| tokens.append(unknownToken) | ||
| } else { | ||
| tokens.append(contentsOf: hexaEncode(text: token)) | ||
| } |
Added a
byteFallbackproperty to theBPETokenizerclass that correctly extracts its value fromtokenizerData.model.byteFallback. Updated the fallback behavior for unknown tokens insidetokenize(text:)to correctly output the unknown token whenbyteFallbackis false, instead of incorrectly forcing a byte-level hexadecimal encoding.PR created automatically by Jules for task 2959915231792515435 started by @Gunnarguy