Repository navigation
Conversation
The existing codec tests check each codec against fixed strings, and test_basics runs the incremental and stream interfaces on "abc123" only. This adds test_codecs_property.py, which runs every codec in all_unicode_encodings through the incremental codecs (any chunking, getstate/setstate round trips), StreamReader/StreamWriter, the error handlers (backslashreplace decoded back through unicode_escape, xmlcharrefreplace through html.unescape, surrogateescape round trips), charmap_decode/charmap_encode against a small oracle, name normalization in codecs.lookup, and the transform codecs, all on generated input and compared against the stateless codec functions. Six divergences turned up, each recorded as an expectedFailure with the minimal input as an explicit example and carved out of the broad tests only where the comments say so: - euc_jp, euc_jis_2004, euc_jisx0213 and gb18030 incremental decoders drop input after an incomplete trailing sequence under surrogateescape. - The ISO-2022 encoders emit U+001B as a bare ESC (iso2022_kr also SO and SI) which their decoders reject or misread. - StreamReader.read() never decodes with final=True, so an incomplete trailing character is silently dropped. - The punycode incremental codecs transform each chunk on its own. - euc_kr encodes U+3164 to the Hangul filler bytes it cannot decode alone. - iso2022_jp_3 decodes plane 2 0x7D3B with the 2004 table because the 2000 emulation is followed by "if" rather than "else if". The by-design differences (BOM-less utf-16/32 incrementally, utf-7 and lone surrogates, the many-to-one compatibility mappings in cp932, cp950, euc_jp and shift_jis, surrogateescape on non-ASCII-compatible codecs) are documented in the module rather than hidden in the strategies. With Hypothesis the file runs in about five seconds; 3000 examples per test pass as well. Without it the explicit examples replay through the stubs.
…xpected failures The StreamReader truncation is pythongh-56717, open since 2011 with the same diagnosis, and the euc_kr filler asymmetry was closed as not a bug in pythongh-101863; the punycode chunking is mentioned in pythongh-64331.
…ailures Previously the expectedFailure tests drew from the same broad strategies as the properties they were carved out of, so most of their generated inputs passed and only the explicit examples made them fail. Each one now generates just the known-failing shape (a trailing incomplete three- or four-byte sequence, a trailing ESC/SO/SI, a filler or the 0x7D3B character between printable ASCII, a chopped multi-byte character, two non-empty ASCII punycode chunks), so the whole test flips to passing once the bug is fixed. A search for a passing input in each strategy finds none.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #107862. This adds
Lib/test/test_codecs_property.py, Hypothesis property tests for the codec machinery, in the same shape as #159960 for the compression modules.The existing codec tests check each codec against fixed strings, and
test_basicsruns the incremental and stream interfaces on"abc123"only. The new file runs every codec inall_unicode_encodingson generated input and compares against the stateless codec functions:getstate()/setstate()round trips (the conditions documented forIncrementalDecoder.getstate());StreamWriterthenStreamReaderwith arbitrary write and read sizes, andreadline()/read()interleaving againststr.splitlines();backslashreplaceoutput decoded viaunicode_escape,xmlcharrefreplaceviahtml.unescape,surrogateescaperound trips on the ASCII-compatible single-byte codecs;charmap_decode/charmap_encodewith generated tables against a small oracle, andcharmap_buildmaps against equivalent dicts;escape_encode/escape_decode, the escape codecs with surrogates, the bytes transform codecs androt_13, and name normalization incodecs.lookup.With Hypothesis the file takes about five seconds; 3000 examples per test also pass. Without it the explicit examples replay through the stubs (27 tests, 6 expected failures in both modes).
The by-design differences are written down in the module rather than hidden in the strategies: BOM-less utf-16/32 is rejected incrementally (tested as such), utf-7 accepts lone surrogates and closes its shift sequence at every chunk, cp932/cp950/euc_jp/shift_jis fold a few compatibility characters, idna applies nameprep, raw_unicode_escape leaves backslashes alone, and
surrogateescapeonly round-trips on codecs where every byte decodes injectively.Six real divergences turned up, each pinned as an
expectedFailurewith the minimal input as an explicit@exampleand carved out of the broad tests only where the comments say so. Two are already on the tracker; issues for the others to follow:euc_jp,euc_jis_2004,euc_jisx0213andgb18030drop input after an incomplete trailing sequence undersurrogateescape:b'\x806'.decode('gb18030', 'surrogateescape')is'\udc806', the incremental decoder gives'\udc80'(also throughTextIOWrapper).'\x1b\xa0'.encode('iso2022_jp_2004').decode('iso2022_jp_2004')is'\x1b\x1b$(Q)"');iso2022_krdoes the same with SO and SI.codecs.StreamReader.read()never decodes withfinal=True, so an incomplete trailing character is silently dropped:codecs.getreader('utf-8')(io.BytesIO(b'ab\xe2')).read()is'ab', with no error in strict mode and no replacement character in replace mode. This is codecs.StreamReader doesn't pass final=1 to the UTF-8 codec #56717, open since 2011.'a'then'b'encodes tob'a-b-', notb'ab-').test_codecsalready lists punycode as "broken in stateful mode". Many incremental codecs don’t handle fragmented data #64331 mentions it in passing.'\u3164'.encode('euc_kr')gives the Hangul fillerb'\xa4\xd4', whicheuc_krcannot decode on its own (cp949can). This was euc_kr char '0x3164' decode('ksx1001') cause UnicodeDecodeError #101863, closed as not a bug; the test records the asymmetry either way.iso2022_jp_3encodes U+9B1D but decodes the result as U+9B1C: injisx0213_2000_2_decoderthe 2000-table emulation macro is followed byifinstead ofelse if, so the table lookup overwrites it.