[ddehilster]

Problem

On Windows, NLPPlus (2.2.39 through 2.2.43) splits words at byte 0xA0 when that byte is part of a multi-byte UTF-8 character. "Voilà" (… C3 A0) comes out as Voil\xC3, and "오전" (EC 98 A4 EC A0 84) gets a space in place of A0. The output text and output files are then invalid UTF-8, and analyze() raises UnicodeDecodeError. The Linux wheel and nlp.exe are not affected.

Cause

The tokenizer is not at fault: dicttokz produces correct tokens, and every per-pass tree is valid UTF-8. The damage is done by clean_str() (lite/str.cpp), which phrasetext() and similar functions use. It tests each byte with _istspace → isspace, whose answer depends on the locale.

• nlp.exe runs in the "C" locale.
• Python sets LC_CTYPE from the user's environment at startup. On Windows that is a code page such as 1252, where 0xA0 is a no-break space, so isspace(0xA0) is true.
• On Linux, the locale is UTF-8, where isspace(0xA0) is false.

Forcing LC_CTYPE=C in the same Python process with the 2.2.39 wheel makes every output file valid.

Fix

bindings.cpp now wraps the constructor, analyze() and compile() in a small RAII guard. It switches the calling thread to the "C" locale, the same as nlp.exe, and restores the caller's locale on return. Windows uses _configthreadlocale; other platforms use uselocale. Other Python threads are not affected.

The guard also covers the other locale-dependent <ctype.h> calls in the engine (isalpha, toupper, …), which could fail the same way.

Test

tests/test_locale.py forces a single-byte locale and analyzes "Voilà, the form. 오전 9시." with parse-en-us. It checks that:

• every output file decodes as UTF-8;
• clause.txt has both phrases whole;
• the caller's locale is restored afterwards.

It skips where no single-byte locale exists, which includes Linux CI. publish.yml now runs it against every repaired Windows wheel through CIBW_TEST_COMMAND_WINDOWS.

Results:

• Local cp313 Windows wheel with the fix: both tests pass.
• NLPPlus 2.2.39: both tests fail.
• Linux/macOS: the non-Windows guard was compiled on its own with g++ 13 (-Wall -Wextra). No Linux or macOS wheel was built locally; CI covers that.

Notes

An engine-side fix in my_tchar.h (ASCII-only character tests) would also protect other programs that embed the engine. That belongs in a separate PR to nlp-engine.

🤖 Generated with Claude Code