Skip to content

Fix base10 dropping leading zeros and corrupting some values on decode#46

Open
gaoflow wants to merge 1 commit into
dhondta:mainfrom
gaoflow:fix-base10-fast-path
Open

Fix base10 dropping leading zeros and corrupting some values on decode#46
gaoflow wants to merge 1 commit into
dhondta:mainfrom
gaoflow:fix-base10-fast-path

Conversation

@gaoflow

@gaoflow gaoflow commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

PR #44 taught the generic base_encode/base_decode to preserve leading null bytes, but base10 never reached that code: the n == 10 branch short-circuits through str(int) / int(str) / i2s() first. That fast-path has three separate round-trip bugs for base10 and its int/integer/dec/decimal aliases:

  • leading null bytes are dropped: encode(b'\x00abc') -> '6382179' -> decode -> b'abc' (base58 correctly gives '1ZiCa' -> b'\x00abc' since Fix Base58 dropping leading zero bytes #44);
  • i2s() does hex(i)[2:].rstrip("eL"), so any value whose integer ends in a 0x?e byte loses that nibble on decode: encode(b'd\xc6\xfe') -> '6604542' -> decode -> b'\x06Lo';
  • str()/int() hit CPython's integer string-conversion digit limit (default 4300 digits), so encode/decode raise ValueError on inputs larger than ~1780 bytes, while every other base handles them.

All three come from base10 bypassing the generic path. The fix keeps the fast-path only for a non-standard 10-character charset and lets the digits charset fall through to the generic loop, which already handles leading zeros, big integers and the 0x?e case correctly. The non-digits fast-path is untouched.

Added test_codec_base10 (base10 had no dedicated test) covering normal round-trips and the four aliases, leading-null preservation, a 0x?e-terminated value, and a 2 KiB input. Full suite green.

The n == 10 fast-path returned through str(int)/int(str)/i2s before reaching the
generic divmod path, so base10 (and its int/integer/dec/decimal aliases) both
missed the leading-zero handling added for the other big-integer bases and had
two more latent round-trip bugs:
- leading null bytes were dropped, e.g. b'\x00abc' -> '6382179' -> b'abc';
- i2s() does hex(i)[2:].rstrip("eL"), so any value whose integer ends in a 0x?e
  byte lost that nibble on decode: b'd\xc6\xfe' -> '6604542' -> b'\x06Lo';
- str()/int() hit CPython's integer string-conversion digit limit, raising
  ValueError on inputs larger than ~1780 bytes.

Keep the fast-path only for a non-standard 10-character charset and let the
digits charset use the generic loop, which already handles all three cases.
@mergify

mergify Bot commented Jul 19, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant