The generated function of the Scanner AddCh() downcasts the int ch where the current character is stored with UTF-32 encoding to a 16 bit char that expects UTF-16 encoding.
This means that glyphs that require surrogate UTF-16 pairs are not being scanned corectly.

The generated function of the Scanner
AddCh()downcasts theint chwhere the current character is stored with UTF-32 encoding to a 16 bitcharthat expects UTF-16 encoding.This means that glyphs that require surrogate UTF-16 pairs are not being scanned corectly.