Skip to content

Normalize string encodings before StringArray assignment - #102

Open
quinnj wants to merge 1 commit into
mainfrom
fix/string-array-encoding
Open

quinnj wants to merge 1 commit into
mainfrom
fix/string-array-encoding

Conversation

@quinnj

@quinnj quinnj commented Sep 22, 2026

Copy link
Copy Markdown
Member

StringArray stores UTF-8 bytes, but assignment copied arbitrary AbstractString storage through pointer. UTF-16/UTF-32 WeakRefString values raised a MethodError, and pointerless strings also failed.

Normalize other string representations through String before copying. Keep direct storage access for String, SubString{String}, and WeakRefString{UInt8}, and restrict the buffer-alias shortcut to UTF-8 views. Preserve the copied data during pointer access.

Refs #59.

Validation: all 199 tests passed on Julia 1.12.6. New tests fail before the fix and cover UTF-16, UTF-32, empty/non-ASCII values, multidimensional assignment, and LazyString. Warmed String, SubString, and UTF-8 WeakRefString assignments each allocate 0 bytes when the destination buffer has capacity.

Co-authored by Codex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant