Skip to content

gh-107862: Add Hypothesis property tests for the codecs machinery - #160155

Open
pganssle wants to merge 3 commits into
python:mainfrom
pganssle:codecs-hypothesis
Open

pganssle wants to merge 3 commits into
python:mainfrom
pganssle:codecs-hypothesis

Conversation

@pganssle

@pganssle pganssle commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Part of #107862. This adds Lib/test/test_codecs_property.py, Hypothesis property tests for the codec machinery, in the same shape as #159960 for the compression modules.

The existing codec tests check each codec against fixed strings, and test_basics runs the incremental and stream interfaces on "abc123" only. The new file runs every codec in all_unicode_encodings on generated input and compares against the stateless codec functions:

  • incremental encode and decode for any chunking of the input, under every error handler, and getstate()/setstate() round trips (the conditions documented for IncrementalDecoder.getstate());
  • StreamWriter then StreamReader with arbitrary write and read sizes, and readline()/read() interleaving against str.splitlines();
  • the error handlers run back through the codec: backslashreplace output decoded via unicode_escape, xmlcharrefreplace via html.unescape, surrogateescape round trips on the ASCII-compatible single-byte codecs;
  • charmap_decode/charmap_encode with generated tables against a small oracle, and charmap_build maps against equivalent dicts;
  • encode/decode round trips, escape_encode/escape_decode, the escape codecs with surrogates, the bytes transform codecs and rot_13, and name normalization in codecs.lookup.

With Hypothesis the file takes about five seconds; 3000 examples per test also pass. Without it the explicit examples replay through the stubs (27 tests, 6 expected failures in both modes).

The by-design differences are written down in the module rather than hidden in the strategies: BOM-less utf-16/32 is rejected incrementally (tested as such), utf-7 accepts lone surrogates and closes its shift sequence at every chunk, cp932/cp950/euc_jp/shift_jis fold a few compatibility characters, idna applies nameprep, raw_unicode_escape leaves backslashes alone, and surrogateescape only round-trips on codecs where every byte decodes injectively.

Six real divergences turned up, each pinned as an expectedFailure with the minimal input as an explicit @example and carved out of the broad tests only where the comments say so. Two are already on the tracker; issues for the others to follow:

  1. The incremental decoders of euc_jp, euc_jis_2004, euc_jisx0213 and gb18030 drop input after an incomplete trailing sequence under surrogateescape: b'\x806'.decode('gb18030', 'surrogateescape') is '\udc806', the incremental decoder gives '\udc80' (also through TextIOWrapper).
  2. The ISO-2022 encoders write U+001B as a bare ESC byte that their decoders reject or misread ('\x1b\xa0'.encode('iso2022_jp_2004').decode('iso2022_jp_2004') is '\x1b\x1b$(Q)"'); iso2022_kr does the same with SO and SI.
  3. codecs.StreamReader.read() never decodes with final=True, so an incomplete trailing character is silently dropped: codecs.getreader('utf-8')(io.BytesIO(b'ab\xe2')).read() is 'ab', with no error in strict mode and no replacement character in replace mode. This is codecs.StreamReader doesn't pass final=1 to the UTF-8 codec #56717, open since 2011.
  4. The punycode incremental codecs transform each chunk on its own ('a' then 'b' encodes to b'a-b-', not b'ab-'). test_codecs already lists punycode as "broken in stateful mode". Many incremental codecs don’t handle fragmented data #64331 mentions it in passing.
  5. '\u3164'.encode('euc_kr') gives the Hangul filler b'\xa4\xd4', which euc_kr cannot decode on its own (cp949 can). This was euc_kr char '0x3164' decode('ksx1001') cause UnicodeDecodeError  #101863, closed as not a bug; the test records the asymmetry either way.
  6. iso2022_jp_3 encodes U+9B1D but decodes the result as U+9B1C: in jisx0213_2000_2_decoder the 2000-table emulation macro is followed by if instead of else if, so the table lookup overwrites it.

The existing codec tests check each codec against fixed strings, and
test_basics runs the incremental and stream interfaces on "abc123" only.
This adds test_codecs_property.py, which runs every codec in
all_unicode_encodings through the incremental codecs (any chunking,
getstate/setstate round trips), StreamReader/StreamWriter, the error
handlers (backslashreplace decoded back through unicode_escape,
xmlcharrefreplace through html.unescape, surrogateescape round trips),
charmap_decode/charmap_encode against a small oracle, name normalization
in codecs.lookup, and the transform codecs, all on generated input and
compared against the stateless codec functions.

Six divergences turned up, each recorded as an expectedFailure with the
minimal input as an explicit example and carved out of the broad tests
only where the comments say so:

- euc_jp, euc_jis_2004, euc_jisx0213 and gb18030 incremental decoders
  drop input after an incomplete trailing sequence under surrogateescape.
- The ISO-2022 encoders emit U+001B as a bare ESC (iso2022_kr also SO and
  SI) which their decoders reject or misread.
- StreamReader.read() never decodes with final=True, so an incomplete
  trailing character is silently dropped.
- The punycode incremental codecs transform each chunk on its own.
- euc_kr encodes U+3164 to the Hangul filler bytes it cannot decode alone.
- iso2022_jp_3 decodes plane 2 0x7D3B with the 2004 table because the
  2000 emulation is followed by "if" rather than "else if".

The by-design differences (BOM-less utf-16/32 incrementally, utf-7 and
lone surrogates, the many-to-one compatibility mappings in cp932, cp950,
euc_jp and shift_jis, surrogateescape on non-ASCII-compatible codecs) are
documented in the module rather than hidden in the strategies.

With Hypothesis the file runs in about five seconds; 3000 examples per
test pass as well. Without it the explicit examples replay through the
stubs.
@bedevere-app bedevere-app Bot added the type-feature A feature request or enhancement label Oct 22, 2026
@bedevere-app bedevere-app Bot added awaiting core review tests Tests in the Lib/test dir labels Oct 22, 2026
…xpected failures

The StreamReader truncation is pythongh-56717, open since 2011 with the same
diagnosis, and the euc_kr filler asymmetry was closed as not a bug in
pythongh-101863; the punycode chunking is mentioned in pythongh-64331.
@pganssle pganssle removed the type-feature A feature request or enhancement label Oct 22, 2026
…ailures

Previously the expectedFailure tests drew from the same broad strategies
as the properties they were carved out of, so most of their generated
inputs passed and only the explicit examples made them fail. Each one
now generates just the known-failing shape (a trailing incomplete
three- or four-byte sequence, a trailing ESC/SO/SI, a filler or the
0x7D3B character between printable ASCII, a chopped multi-byte character,
two non-empty ASCII punycode chunks), so the whole test flips to passing
once the bug is fixed. A search for a passing input in each strategy
finds none.
@bedevere-app bedevere-app Bot added the type-feature A feature request or enhancement label Oct 22, 2026
@pganssle pganssle removed the type-feature A feature request or enhancement label Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting core review tests Tests in the Lib/test dir

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant