Normalise non-ASCII variable names in __all__

[NateSoares]

BPO	30772
Nosy	@vstinner, @ezio-melotti, @stevendaprano, @Carreau

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-06-26.18:08:51.252>
labels = ['type-bug', '3.7', 'expert-unicode']
title = 'Normalise non-ASCII variable names in __all__'
updated_at = <Date 2017-07-05.13:36:35.910>
user = 'https://bugs.python.org/NateSoares'

bugs.python.org fields:

activity = <Date 2017-07-05.13:36:35.910>
actor = 'mbussonn'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2017-06-26.18:08:51.252>
creator = 'Nate Soares'
dependencies = []
files = []
hgrepos = []
issue_num = 30772
keywords = []
message_count = 6.0
messages = ['296928', '296934', '296935', '297284', '297333', '297739']
nosy_count = 6.0
nosy_names = ['vstinner', 'ezio.melotti', 'mrabarnett', 'steven.daprano', 'mbussonn', 'Nate Soares']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30772'
versions = ['Python 3.7']

Linked PRs

• gh-74955: Document that __all__ must contain strings in normalization form NFKC #144504
• [3.14] gh-74955: Document that __all__ must contain strings in normalization form NFKC (GH-144504) #144519
• [3.13] gh-74955: Document that __all__ must contain strings in normalization form NFKC (GH-144504) #144520

[NateSoares]

[NOTE: In this comment, I use BB to mean unicode character 0x1D539, b/c the issue tracker won't let me submit a comment with unicode characters in it.]

Directory structure:

repro/
  foo.py
  test_foo.py

Contents of foo.py:

BB = 1
__all__ = ['BB']

Contents of test_foo.py:

from .foo import *

Error message:

AttributeError: module 'repro.foo' has no attribute 'BB'

If I change foo.py to have __all__ = ['B'] (note that 'B' is not the same as 'BB'), then everything works "fine", modulo the fact that now foo.B is a thing and foo.BB is not a thing.

[Recall that in the above, BB is a placeholder for U+1D539, which the issuetracker prevents me from writing here.]

[ezio-melotti]

I can reproduce the issue:

$ cat foo.py 
𝔹𝔹 = 1
__all__ = ['𝔹𝔹']

$ python3 -c 'import foo; print(dir(foo)); from foo import *'
['BB', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__']
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'foo' has no attribute '𝔹𝔹

(Note the ascii 'BB' in the dir(foo))

There's also an easier way to reproduce it:

>>> 𝔹𝔹= 3
>>> 𝔹𝔹
3
>>> BB
3
>>> globals()['BB']
3
>>> globals()['𝔹𝔹']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: '𝔹𝔹'
>>> globals()
{'__name__': '__main__', '__spec__': None, '__builtins__': <module 'builtins' (built-in)>, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__doc__': None, 'BB': 3, '__package__': None}
>>> class Foo:
... 𝔹  𝔹= 3
... 
>>> Foo.𝔹𝔹
3
>>> Foo.BB
3

It seems the '𝔹𝔹' gets normalized to 'BB' when it's an identifier, but not when it's a string. I'm not sure why this happens though.

[mrabarnett]

See PEP-3131 -- Supporting Non-ASCII Identifiers

It says: """All identifiers are converted into the normal form NFKC while parsing; comparison of identifiers is based on NFKC."""

>>> import unicodedata
>>> unicodedata.name(unicodedata.normalize('NFKC', '\N{MATHEMATICAL DOUBLE-STRUCK CAPITAL B}'))
'LATIN CAPITAL LETTER B'

[NateSoares]

To be clear, the trouble I was trying to point at is that if foo.py didn't
have __all__, then it would still have a BB attribute. But if the module is
given __all__, the BB is normalized away into a B. This seems like pretty
strange/counterintuitive behavior. For instance, I found this bug when I
added __all__ to a mathy library, where other modules had previously been
happily importing BB and using <module>.BB etc. with no trouble.

In other words, I could accept "BB gets normalized to B always", but the
current behavior is "modules are allowed to have a BB attribute but only if
they don't use __all__, because __all__ requires putting the BB through a
process that normalizes it to B, and which otherwise doesn't get run".

If this is "working as intended" then w/e, I'll work around it, but I want
to make sure that we all understand the inconsistency before letting this
bug die in peace :-)

On Wed, Jun 28, 2017 at 10:55 AM Brett Cannon <report@bugs.python.org>
wrote:

Changes by Brett Cannon <brett@python.org>:

----------
resolution: -> not a bug
stage: -> resolved
status: open -> closed

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue30772\>

---

[stevendaprano]

I think that the names in __all__ should have the same NFKC normalisation applied as the identifiers.

Re-opening for 3.7.

[Carreau]

I think that the names in __all__ should have the same NFKC normalisation applied as the identifiers.

Does it make sens to add to this issue : Ensure that all elements of __all__ are str ? (At least emit a warning ?)

I have encounter a small number of libraries where some member of all are the actual objects. Easy mistake to make if you make a public decorator:

    __all__ = []

    def public(o):
        __all__.append(o)
        return o

    @public
    def bar():
        pass

Happy to open a different issue if deemed necessary. Thanks !