Repository navigation
FIX Write non-BMP characters as surrogate pairs in UnicodeReplacementConverter - #3132
Merged
Roman Lutz (romanlutz) merged 3 commits intoOct 14, 2026
Conversation
…Converter `:04x` is a minimum width, so a character above U+FFFF was written with five or six hex digits (`ὠ0` for U+1F600), which every `\u` decoder reads as a different character followed by a stray digit. Encode the word as UTF-16 code units so these characters become a surrogate pair (`😀`). BMP characters are unchanged. Fixes microsoft#3130 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Python str can hold an unpaired surrogate (for example from json.loads), and strict UTF-16 encoding raised "surrogates not allowed" for it. Encode with errors="surrogatepass" so a lone surrogate is written as its own code unit, as before, while non-BMP characters are still written as surrogate pairs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Roman Lutz (romanlutz)
approved these changes
Oct 14, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 14, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 14, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 14, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #3130.
UnicodeReplacementConverter.convert_word_asyncformatted each character as\u{ord(ch):04x}.:04xis a minimum width, so anything above U+FFFF came out with five or six hex digits, e.g.😀becameὠ0, which decodes toὠ0. A\uescape always holds exactly four hex digits.The word is now encoded as UTF-16 code units, so characters outside the BMP are written as their surrogate pair (
😀). This is the same formJsonStringConverteralready produces. Output for BMP characters, and theencode_spaceshandling, are unchanged.Tests and Documentation
test_unicode_replacement_non_bmp_uses_surrogate_pairs. It checks the exact output forhi 😀and thatjson.loadsdecodes it back to the input. It fails onmain(ὠ0) and passes with the change.pytest tests/unit/converter/test_unicode_replacement_converter.py tests/unit/converter/test_converter.py tests/unit/converter/test_seeded_converter_determinism.py: all pass.ruff check,ruff format --checkandty checkare clean on the changed files.🤖 Generated with Claude Code