Repository navigation
Convert glibc guests' text the way glibc's iconv does - #16
Open
buyemune-sys wants to merge 2 commits into
Open
buyemune-sys wants to merge 2 commits into
buyemune-sys wants to merge 2 commits into
Conversation
Security updates replaced libapparmor1 4.0.1really4.0.1-0ubuntu0.24.04.7 with .9 and libexpat1 2.6.1-2ubuntu0.4 with ubuntu0.6, and the pool kept only the new files, so the cold-cache Fetch x86_64 job answers 404 and every x86-64 job behind it is skipped. Both pins move to the versions noble-updates ships now; their SHA256s match the archive's Packages index. The other twelve pins are still served.
Programs lean on corners of glibc's iconv that musl's converter does not
share, and the bridge passed iconv_open, iconv and iconv_close straight
to musl:
- A character the target charset lacks fails with EILSEQ at that
character. musl writes '*' and counts it as an irreversible
conversion, so text is altered behind the caller's back: Tarantool's
iconv module never sees the error its test expects.
- A code may carry //TRANSLIT and //IGNORE, or any other suffix glibc
disregards. musl rejects such a name, so `iconv_open("ASCII//TRANSLIT",
...)`, the usual idiom for lossy output, fails with EINVAL. //IGNORE
drops unconvertible characters and invalid input alike and reports
EILSEQ once the buffer is through; //TRANSLIT writes '?' where the
locale has no transliteration.
- UTF-16 and UTF-32 without a byte order mean a BOM and the host's order
on output and the host's order on unmarked input; UCS-2 is the host's
order outright and refuses characters beyond the BMP. musl means
big-endian throughout and writes U+FFFD for what UCS-2 cannot hold.
The bridge's descriptor now wraps musl's. Unicode targets convert
directly. The others convert a character at a time through wchar_t, so
the bridge sees each '*' musl substitutes and applies glibc's rule:
stop, skip, or write '?'.
The locale's transliteration tables stay out of reach: glibc spells
U+20AC as "EUR" even in the C locale and drops accents in UTF-8 locales,
where the bridge writes '?'.
The conformance battery checks each rule above against the values glibc
produces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
buyemune-sys
force-pushed
the
fix/glibc-iconv
branch
from
October 20, 2026 08:11
f041764 to
8dacf9e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
iconv_open,iconvandiconv_closewent straight to musl, whose converter differs from glibc's where programs rely on it:ASCII←"aéz"-1/EILSEQaté, outputarc=1, outputa*zASCII//TRANSLIT←"aéz"rc=1,a?ziconv_open: EINVALASCII//IGNORE←"aéz"-1/EILSEQafter the whole buffer,aziconv_open: EINVALUTF-8//IGNORE←"a\xffb"-1/EILSEQ,abiconv_open: EINVALUTF-16←"a"ff fe 61 00(BOM, host order)00 61UTF-32←"a"ff fe 00 00 61 00 00 0000 00 00 61UCS-2←"a"61 0000 61UCS-2← U+1F600-1/EILSEQfd ff,rc=0UTF-8←UTF-16"\x00a"(no BOM)aThe first row silently alters text: Tarantool's
iconvmodule never sees the error its test expects (app-tap/iconv.test.lua).//TRANSLITis the usual idiom for lossy output, andUTF-8//IGNOREthe usual one for dropping invalid bytes.The fix
The bridge's descriptor wraps musl's (
sh_iconv_open,lib/glibc_shim.cpp:3136).Name handling:
//SUFFIXpart of the codes is parsed off.TRANSLITandIGNOREare honored; other suffixes are ignored, as glibc does.UTF-16,UTF-32,UNICODE,UCS-2andUCS-4map to the byte orders glibc uses.Conversion:
wchar_t(sh_iconv_character,:3253). That is how the bridge sees each*musl substitutes and applies glibc's rule: stop with EILSEQ, skip, or write?.//IGNOREalso steps over invalid input by the source code unit and reports EILSEQ at the end.The locale's transliteration tables stay out of reach. glibc spells U+20AC as
EUReven in the C locale and drops accents in UTF-8 locales; the bridge writes?.Test
The conformance battery (
conversions()) checks every row above, plus E2BIG and EINVAL positions, a WINDOWS-1251 conversion and the BOM across a reset. All expected values were first taken from real glibc.On
main: 15 checks fail. With the fix:./build testpasses.