GH-51225: [C++] utf8_normalize: compose for NFC and NFKC

[singhpratech]

Rationale for this change

utf8_normalize with form=NFC or NFKC returned the decomposed forms (NFD, NFKD). Utf8NormalizeBase
builds the right utf8proc options for each form but only calls utf8proc_decompose(), which
decomposes regardless of UTF8PROC_COMPOSE; the composition step lives in
utf8proc_normalize_utf32(), which the kernel never called. See #51225.

What changes are included in this PR?

• After a successful utf8proc_decompose(), when the options include UTF8PROC_COMPOSE, call
  utf8proc_normalize_utf32() on the scratch buffer. It composes in place and returns the new code
  point count; the existing UTF-8 encode loop is unchanged. NFD and NFKD take the same path as before.
• Fix the json_composed fixture in scalar_string_test.cc: its bytes were the decomposed form
  (61 CC 81), the same string as json_decomposed, so the compose assertions were comparing a value
  with itself and passed with the bug. Thanks to @Santoshkumarpuppala for spotting that on the issue.
• Add composed/decomposed pairs (U+00E9, and a Hangul syllable with its jamo) to
  test_utf8_normalize in pyarrow; the existing input, U+00B2, is its own NFC.

Are these changes tested?

Yes. With the corrected fixture, TestStringKernels.Utf8Normalize fails on the unpatched kernel and
passes with this change; the pyarrow test covers the composed forms from Python.

Are there any user-facing changes?

Yes: utf8_normalize with NFC and NFKC now returns composed output. Callers that depended on the
previous (decomposed) result for those forms will see different bytes.

• GitHub Issue: [Python][C++] utf8_normalize ignores its form option: NFC and NFKC output is decomposed #51225

[github-actions]

⚠️ GitHub issue #51225 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #51225 has no components, please add labels for components.

[raulcd]

This is not a critical fix, can you update the description.

[github-actions]

⚠️ GitHub issue #51225 has no components, please add labels for components.

[Santoshkumarpuppala]

Checked the diff. All three parts land, and the kernel change is in the right place — after the regrow retry and the res < 0 check, so the buffer is already sized for res codepoints and composition only ever shrinks it. No spare-byte concern, and the existing UTF8EncodedLength/UTF8Encode loop is untouched.

It also matches the reference path exactly: utf8proc_map is decompose then reencode, and reencode is normalize_utf32 followed by the UTF-8 encode. Taking the first half and keeping Arrow's own encoder gives the same result. Gating on UTF8PROC_COMPOSE leaves NFD/NFKD alone.

The composed fixture is now "\xc3\xa1", so the compose loop asserts real composition instead of comparing a string to itself. That was the part that would have let this regress invisibly.

One small suggestion. The pyarrow case includes Hangul, which is the more interesting input — Hangul composition in utf8proc is algorithmic rather than table-driven, so it exercises a different path. The C++ suite still only covers the Latin pair. A Hangul pair there too would be worth having, since the C++ kernel is what the other bindings sit on.

Happy to look again once CI is green.

[singhpratech]

Added the Hangul pair to the C++ test: U+1112 U+1161 U+11AB against U+D55C, both directions for all
four forms, in the same shape as the Latin pair. With the kernel change reverted the new assertions
fail the same way the corrected Latin fixture does, so the table-driven and the algorithmic
composition paths are both covered now. Thanks for the careful read.

[pitrou]

Is this comment useful? This is mentioning an implementation detail of utf8proc, which doesn't seem relevant for our purposes.

(I've checked that Python agreed with this behavior)

[singhpratech]

Removed, the code point lines stay.

[pitrou]

Is it still useful that we call utf8proc_decompose first? Or should we just decode to UTF32 codepoints ourselves?

[singhpratech]

Yes, it is still needed. utf8proc_normalize_utf32() composes pairs (through the composition
table, Hangul by arithmetic) and applies the newline and control-character options; it does no
decomposition and no reordering of combining marks, both of which happen in
utf8proc_decompose(). NFC and NFKC are the full decomposition followed by canonical
composition. Skipping utf8proc_decompose() would leave singletons alone (U+212B ANGSTROM SIGN
normalizes to U+00C5, there is nothing to compose), would not reorder marks (U+00E1 U+0323 must
become U+1EA1 U+0301), and would drop all of NFKC's compatibility mappings, which happen in the
decompose call. utf8proc's own utf8proc_map() is the same sequence, utf8proc_decompose() then
utf8proc_reencode(), which calls utf8proc_normalize_utf32() before encoding; the kernel does
that into its scratch buffer instead of the malloc in utf8proc_map(). I added the U+212B
singleton to the test, since it is the case that only passes with both steps.

[pitrou]

Oh, that's interesting indeed. Thanks for the example.

>>> s = "\u212B"
>>> unicodedata.normalize('NFC', s) == s
False
>>> unicodedata.normalize('NFKC', s) == s
False
>>> unicodedata.normalize('NFD', s) == s
False
>>> unicodedata.normalize('NFKD', s) == s
False

[pitrou]

Thanks for this PR, and see already posted comments.

[pitrou]

Nit: I've literal-quoted @Santoshkumarpuppala in the PR description to avoid spurious GH notifications when commits propagate (merges, etc.).

[github-actions]

⚠️ GitHub issue #51225 has no components, please add labels for components.

[pitrou]

Thanks for the update and the explanations @singhpratech

[singhpratech]

Thanks for the review and the merge.