Invalid Unicode in unquoted atom

[RaymondLoranger]

Existing issue

• I have searched existing issues and could not find a duplicate.

Elixir and Erlang/OTP versions

Erlang/OTP 28 [erts-16.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]

Interactive Elixir (1.19.5) - press Ctrl+C to exit (type h() ENTER for help)

Operating system

Windows 10

Current behavior

When I compile hex package io_ansi_plus, I get error “Codepoint failed”. Why? and why are the “Got:” and “Hint:” codepoints identical? It seems in an unquoted atom that Pinyin character ǜ (Latin Small Letter U with diaeresis and grave) is invalid. However Pinyin character ǔ (Latin Small Letter U with caron) is valid. I am using Elixir v1.19.5 when getting this error. In version v1.18, the same source code compiled fine! Here is the error:

mix compile
Compiling 2 files (.ex)

== Compilation error in file lib/io/ansi/plus/ie.ex ==
** (SyntaxError) invalid syntax found on config/persist_colors.exs:2323:13:
error: unexpected token: Codepoint failed identifier tokenization, but a simpler form was found.

  Got:

      "cōng_lǜ" (code points 0x00063 0x0014D 0x0006E 0x00067 0x0005F 0x0006C 0x001DC)

  Hint: You could write the above in a similar way that is accepted by Elixir:

      "cōng_lǜ" (code points 0x00063 0x0014D 0x0006E 0x00067 0x0005F 0x0006C 0x001DC)

  See https://hexdocs.pm/elixir/unicode-syntax.html for more information.
  │

2323 │ :cōng_lǜ_green,
│ ^
│
└─ config/persist_colors.exs:2323:13
(elixir 1.19.5) lib/code.ex:628: Code.validated_eval_string/3
(elixir 1.19.5) lib/config.ex:326: Config.eval!/3
(elixir 1.19.5) lib/config/reader.ex:102: Config.Reader.read!/2
lib/io/ansi/plus/ie.ex:4: anonymous fn/1 in :elixir_compiler_2.MODULE/1
(elixir 1.19.5) lib/enum.ex:961: Enum."-each/2-lists^foreach/1-0-"/2
lib/io/ansi/plus/ie.ex:4: (module)

Expected behavior

Unquoted atom :cōng_lǜ_green should be valid.

[RaymondLoranger]

I agree that you should not suggest invalid characters but what makes a character invalid to be an atom?
As shown below, :ǜ (with diaeresis and grave) is invalid but :ǔ (with caron) is valid.
Is it because 2 accents are not accepted while 1 is? What is the criteria? I see no reason why :ǜ is invalid.
And as I said it used to work without issue in Elixir v.0.18!

iex(1)> :ǜ
** (SyntaxError) invalid syntax found on iex:1:1:
    error: unexpected token: ":" (column 1, code point U+003A)
    │
  1 │ :ǜ
    │ ^
    │
    └─ iex:1:1
    (iex 1.19.5) lib/iex/server.ex:402: IEx.Server.__parse__/3
    (stdlib 7.3) io_lib.erl:1600: :io_lib.get_until/4
    (kernel 10.6.1) group.erl:401: :group.xterm/3
    (stdlib 7.3) gen_statem.erl:3741: :gen_statem.loop_state_callback/11
    (stdlib 7.3) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
iex(1)> :ǔ
:ǔ
iex(2)>

[kipcole9]

I suspect (but have not confirmed) it’s related to Unicode security mechanism - particularly Confusable. They were updated in Unicode 17 - and elixir updates to then-current Unicode whenever a new Elixir is released.

[josevalim]

There are multiple ways of writing Unicode and not all of them are valid. In particular, Elixir normalizes the NFC form and refuses confusables. More information here: https://hexdocs.pm/elixir/unicode-syntax.html

[josevalim]

Btw, the reason this particular codepoint is not allowed is because Unicode itself denotes it as uncommon use:

01D5..01DC ; Uncommon_Use # 1.1 [8] LATIN CAPITAL LETTER U WITH DIAERESIS AND MACRON..LATIN SMALL LETTER U WITH DIAERESIS AND GRAVE

And therefore it falls outside of the profile allowed by Elixir. You can use the composed version and it will work, as it falls within the valid profiles: :cōng_lǜ.

I wish there was a simple answer. :(

[josevalim]

Better error message pushed to main.

[sabiwara]

I'm confess I'm a bit out of my depth with my limited unicode knowledge, but I think there might be some way to suggest a working alternative here:

• :ǜ is invalid ([476] - the NFKC)
• :ǜ is valid ([117, 776, 768] - the NFD) -> we could suggest this

iex> :unicode.characters_to_nfd_list([476])
[117, 776, 768]
iex> :unicode.characters_to_nfkc_list([117, 776, 768])
[476]

iex> String.Tokenizer.tokenize([476])
{:error, :empty}
iex> String.Tokenizer.tokenize([117, 776, 768])
{:identifier, [476], [], 3, false, [:nfkc]}

# uses :unicode.characters_to_nfd_list under the hood
iex> String.Tokenizer.Security.confusable_skeleton([476])
[117, 776, 768]
# but then we tokenize this again: {:identifier, [476], [], 3, false, [:nfkc]}

So for some reason, tokenize

• doesn't accept NFKC
• accepts NFD but returns NFKC (which isn't tokenizable)
• our suggestion was applying NFKC hence getting back the non-tokenizable version

Perhaps tokenizer should return something like {:identifier, [117, 776, 768], [], 3, false, [:nfd]} or sth here?

[josevalim]

Good find. I believe we need to validate the normalized version too. But then it means no version is supported. @RaymondLoranger does that appear in actual constructs/words/languages?

[josevalim]

Ok, so yes. After we convert the identifier, we need to validate it again. It will indeed reject the grapheme above, which may be valid in some cases, but that's unfortunately the Unicode Security recommendation.

[josevalim]

Keep in mind you can quote the atom :"..." and then have any character but then the atom is not normalized. This means :"cōng_lǜ" != :"cōng_lǜ", as they have different encodings.

[RaymondLoranger]

I decided to use the decomposed grapheme ([117, 776, 768]) in the unquoted atom and exclude the file from the formatter which would otherwise normalize it to the quoted atom :"cōng_lǜ_green":

Thank you so much Jean and José!!