Add functions to get the width in columns of a character

@malemburg

BPO	12568
Nosy	@malemburg, @loewis, @terryjreedy, @vstinner, @benjaminp, @ezio-melotti, @merwok, @bitdancer, @serhiy-storchaka, @Vermeille, @ishigoya, @bianjp
Files	locale_width.patch width.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-11-08.21:45:08.936>
created_at = <Date 2011-07-14.22:43:56.356>
labels = ['type-feature', '3.8', 'expert-unicode']
title = 'Add functions to get the width in columns of a character'
updated_at = <Date 2018-11-08.21:45:08.934>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-11-08.21:45:08.934>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-08.21:45:08.936>
closer = 'vstinner'
components = ['Unicode']
creation = <Date 2011-07-14.22:43:56.356>
creator = 'vstinner'
dependencies = []
files = ['23401', '24773']
hgrepos = []
issue_num = 12568
keywords = ['patch']
message_count = 39.0
messages = ['140376', '140488', '141936', '145497', '145498', '145523', '145535', '145748', '145778', '155223', '155236', '155307', '155313', '155323', '155324', '155337', '155342', '155343', '155344', '155345', '155346', '155361', '155370', '155373', '155379', '155382', '156337', '156348', '181149', '238425', '255421', '297129', '297489', '297492', '297564', '297569', '298322', '323731', '329488']
nosy_count = 19.0
nosy_names = ['lemburg', 'loewis', 'terry.reedy', 'vstinner', 'benjamin.peterson', 'ezio.melotti', 'eric.araujo', 'Arfrever', 'r.david.murray', 'inigoserna', 'zeha', 'poq', 'Nicholas.Cole', 'tchrist', 'serhiy.storchaka', 'Socob', 'Guillaume Sanchez', 'ishigoya', 'bianjp']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue12568'
versions = ['Python 3.8']

Martin, I think you meant to write "if w == 'A':".
Some very common characters have ambiguous widths though (e.g. the Greek alphabet), so you can't just raise an error for them.

http://unicode.org/reports/tr11/ says:
"Ambiguous characters occur in East Asian legacy character sets as wide characters, but as narrow (i.e., normal-width) characters in non-East Asian usage."

So in practice applications can treat ambiguous characters as narrow by default, with a user setting to use legacy (wide) width.

As Tom pointed out there are also a bunch of zero width characters, and characters with special formatting like tab, soft hyphen, ...

I would encourage you to look at the Perl CPAN module Unicode::LineBreak,
which fully implements tr11. It includes Unicode::GCString, a class
that has a columns() method to determine the print columns. This is very
fancy in the case of Asian widths, but of course there are many other cases too.

If you'd like, I can show you a program that uses these, a rewrite the
standard Unix fmt(1) filter that works properly on Unicode column widths.

--tom

Marting and Poq: I think the sample code shows up a real problem. "Ambiguous" characters according to unicode may be rendered by curses in different ways.

Don't we need a function that actually reports how curses is going to print a given string, rather than just reporting what the unicode standard says?

Martin, I think you meant to write "if w == 'A':".
Some very common characters have ambiguous widths though (e.g. the Greek alphabet), so you can't just raise an error for them.

That's precisely why I don't think this should be in the library, but
in the application. Application developers who need that also need
to concern themselves with the border cases, and decide on how
they need to resolve them.

I would encourage you to look at the Perl CPAN module Unicode::LineBreak,
which fully implements tr11.

Thanks for the pointer!

If you'd like, I can show you a program that uses these, a rewrite the
standard Unix fmt(1) filter that works properly on Unicode column widths.

I believe there can't be any truly "proper" implementation, as you
can't be certain how the terminal will handle these itself. In any
case, anybody who is interested in contributing a patch should also
be capable of understanding the source of Unicode::LineBreak.

Martin v. L=C3=B6wis <martin@v.loewis.de> added the comment:

> Martin, I think you meant to write "if w =3D=3D 'A':".
> Some very common characters have ambiguous widths though (e.g. the Greek =
alphabet), so you can't just raise an error for them.

That's precisely why I don't think this should be in the library, but
in the application. Application developers who need that also need
to concern themselves with the border cases, and decide on how
they need to resolve them.

The column-width of a string is not an application issue. It is
well-defined by Unicode. Again, please see how we've done it in
Perl, where tr11 is fully implemented. The columns() method from
Unicode::GCString always gives the right answer per the Standard for
any string, even what you are calling ambiguous ones.

This is not an applications issue -- at all.

--tom

Don't we need a function that actually reports how curses is going to
print a given string, rather than just reporting what the unicode
standard says?

That may be useful, but

a) this patch doesn't provide that, and
b) it may not actually possible to implement such a change in a portable
way as there may be no function exposed by the curses implementation
that provides this information.

To put my closing this issue differently: I rejected the patch that
Victor initially submitted. If anybody wants to contribute a different
patch that uses a different strategy, please submit a new issue.

Martin v. L=C3=B6wis <martin@v.loewis.de> added the comment:

> I would encourage you to look at the Perl CPAN module Unicode::LineBreak,
> which fully implements tr11.

Thanks for the pointer!

> If you'd like, I can show you a program that uses these, a rewrite the
> standard Unix fmt(1) filter that works properly on Unicode column widths.

I believe there can't be any truly "proper" implementation, as you
can't be certain how the terminal will handle these itself.

Hm. I think we may not be talking about the same thing after all.

If we're talking about the Curses library, or something similar,
this is not the same. I do not think Curses has support for
combining characters, right to left text, wide characters, etc.

However, Unicode does, and defines the column width for those.

I have an illustration of what this looks like in the picture
in the very last recipe, #44, in

http://training.perl.com/scripts/perlunicook.html

That is what I have been talking about by print widths. It's running
in a Mac terminal emulator, and unlike the HTML which grabs from too
many fonts, the terminal program does the right thing with the widths.

Are we talking about different things?

--tom

It seems this is a bit of a minefield...

GNOME Terminal/libvte has an environment variable (VTE_CJK_WIDTH) to override the handling of ambiguous width characters. It bases its default on the locale (with the comment 'This is basically what GNU libc does').

urxvt just uses system wcwidth.

Xterm uses some voodoo to decide between system wcwidth and mk_wcwidth(_cjk): http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c

I think the simplest solution is to just expose libc's wc(s)width. It is widely used and is most likely to match the behaviour of the terminal.

FWIW I wrote a little script to test the widths of all Unicode characters, and came up with the following logic to match libvte behaviour:

def wcwidth(c, legacy_cjk=False):
	if c in u'\t\r\n\10\13\14': raise ValueError('character %r has no intrinsic width' % c)
	if c in u'\0\5\7\16\17': return 0
	if u'\u1160' <= c <= u'\u11ff': return 0 # hangul jamo
	if unicodedata.category(c) in ('Mn', 'Me', 'Cf') and c != u'\u00ad': return 0 # 00ad = soft hyphen
	eaw = unicodedata.east_asian_width(c)
	if eaw in ('F', 'W'): return 2
	if legacy_cjk and eaw == 'A': return 2
	return 1

Tom: I don't think Unicode::GCString implements UAX#11 correctly (but this is really out of scope of this issue). In particular, it contains an ad-hoc decision to introduce the EA_Z east-asian width that UAX#11 doesn't talk about.

In most cases, it's probably reasonable to introduce this EA_Z feature. However, there are some significant deviations from UAX#11 here:

combining characters are given EA_Z in sombok/data/custom.pl, even though UAX#11 assigns A or N. UAX#11 points out that the advance width depends on whether or not the terminal performs character combination or not. It's not clear whether Unicode::GCString aims for "strict" UAX#11, or "advance width".
control characters are also given EA_Z, even though UAX#11 gives them EA_N. In this case, it's neither UAX#11 width nor advance width since control characters will have various effects on the terminal (in particular for the tab character)

poq: I still remain opposed to exposing wcswidth, since it is just as incorrect as any of the other solutions that people have circulated. I could agree to it if it was called "wcswidth", making it clear that it does whatever the C library does, with whatever semantics the C library wants to give to it (and an availability that depends on whether the C library supports it or not).

That would probably cover the nurses use cases, except that it is not only incorrect with respect to Unicode, but also incorrect with respect to what the terminal may be doing. I guess users would use it anyway.

For Python's internal use, I could accept using the sombok algorithm. I wouldn't expose it, since it again would trick people into believing that it was correct in some sense. Perhaps calling it sombok_width might allow for exposing it.

Martin,

I agree that wcswidth is incorrect with respect to Unicode. However I don't think that's relevant at all. Python should only try to match the behaviour of the terminal.

Since terminals do slightly different things, trying to match them exactly - in all cases, on all systems - is virtually impossible. But AFAICT wcwidth should match the terminal behaviour on nearly all modern systems, so it makes sense to expose it.

Poq: I agree. Guessing from the Unicode standard is going to lead to users having to write some complicated code that people are going have to reinvent over and over, and is not going to be accurate with respect to curses. I'd favour exposing wcwidth.

Martin: I agree that there are going to be cases where it is not correct because the terminal does something strange, but what we need is something that gets as close as possible to what the terminal is likely to be doing (the Unicode standard itself is not really the issue for curses stuff). So whether it is called wcwidth or wcswidth I don't really mind, but I think it would be useful.

The other alternative is to include one of the other ideas that have been mentioned in this thread as part of the library, I suppose, so that people don't have to keep reinventing the wheel for themselves.

The one thing I really don't favour is shipping something that supports wide characters, but gives the users no way of guessing whether or not that is what they are printing, because that is surely going to break a lot of applications.

Martin: I agree that there are going to be cases where it is not
correct because the terminal does something strange, but what we
need is something that gets as close as possible to what the
terminal is likely to be doing

Can't we expose wcswidth() as locale.strwidth() with a recipe explaining how to use unicodedata to get a "correct" result? At least until everyone implements correctly Unicode and Unicode stops evolving? :-)

--

For unicodedata, a function to get the width of a string would be more convinient than unicodedata.east_asian_width():

>>> import unicodedata
>>> unicodedata.east_asian_width('abc')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: need a single Unicode character as parameter
>>> 'abc'.ljust(unicodedata.east_asian_width(' '))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'str' object cannot be interpreted as an integer

The function posted in msg155361 looks like east_asian_width() is not enough to get the width in columns of a single character.

Has anyone tested wcswidth on FreeBSD, old Solaris? With non-utf8 locales?

In this part of width.py,

        w = unicodedata.east_asian_width(c)
        if c == 'A':
            \# ambiguous
            raise ValueError("ambiguous character %x" % (ord(c)))

I presume that 'c' should be 'w'.

Since no consensus was found on the definition of the function, and this issue has no activity since 2 years, I close the issue as out of date.

I think this function would be very useful in many parts of interpreter core and standard library. From displaying tracebacks to formatting helps.

Otherwise we are doomed to implement imperfect variants in multiple places.

Since we failed to agree on this feature, I close the issue.

Interestingly, this just came up again in bpo-30717.

At least two other issues depend on this: bpo-17048 and bpo-24665.

If Victor lost interest in this issue I take it. I'm going to push at least imperfect solution which may be improved in time.

At least two other issues depend on this: bpo-17048 and bpo-24665.

I removed the dependency from bpo-24665 (CJK support for textwrap) to this issue, since its current PR uses unicodedata.east_asian_width(), not the C function wcswidth().

You need users who use CJK and understand locale issues especially the width of characters. Ask maybe Xiang Zhang and Naoki INADA?

Hello,

I come from bpo-30717 . I have a pending PR that needs review ( #2673 ) adding a function that breaks unicode strings into grapheme clusters (aka what one would intuitively call "a character"). It's based on the grapheme cluster breaking algorithm from TR29.

Let me know if this is of any relevance.

Quick demo:
>>> a=unicodedata.break_graphemes("lol")
>>> list(a)
['l', 'o', 'l']
>>> list(unicodedata.break_graphemes("lo\u0309l"))
['l', 'ỏ', 'l']
>>> list(unicodedata.break_graphemes("lo\u0309\u0301l"))
['l', 'ỏ́', 'l']
>>> list(unicodedata.break_graphemes("lo\u0301l"))
['l', 'ó', 'l']
>>> list(unicodedata.break_graphemes(""))
[]

I suggest reclosing this issue, for the same reason I suggested closure of bpo-24665 in msg321291: abstract unicode 'characters' (graphemes) do not, in general, have fixed physical widths of 0, 1, or 2 n-pixel columns (or spaces). I based that fairly long message on IDLE's multiscript font sample as displayed on Windows 10. In that context, for instance, the width of (fixed-pitch) East Asian characters is about 1.6, not 2.0, times the width of fixed-pitch Ascii characters. Variable-width Tamil characters average about the same. The exact ratio depends on the Latin font used.

I did more experiments with Python started from Command Prompt with code page 437 or 65001 and characters 20 pixels high. The Windows console only allows 'fixed pitch' fonts. East Asian characters, if displayed, are expanded to double width.

However, European characters are not reliably displayed in one column. The width depends on the both the font selected when a character is entered and the current font. The 20 latin1 characters in '¢£¥§©«®¶½ĞÀÁÂÃÄÅÇÐØß' usually display in 20 columns. But if they are entered with the font set to MSGothic, the '§' and '¶' are each displayed in the middle of 2 columns, for 22 total. If the font is changed to MSGothic after entry, the '§' and '¶' are shifted 1/2 column right to overlap the following '©' or '½' without changing the total width. Greek and Cyrillic characters also sometimes take two columns.

I did not test whether the font size (pixel height) affects horizontal column spacing.

I close the issue as WONTFIX.

I think that even imperfect solution is better than no solution. wcwidth() and wcswidth() are parts of Posix, they are used by modern terminals. There are several packages on PyPI:

They are virtually mandatory for any Python program with rich CLI or TUI.

But now the stdlib itself needs such function for internal use -- in REPL, calendar, argparse. We cannot depend on third-party packages.

@methane

But now the stdlib itself needs such function for internal use -- in REPL, calendar, argparse

On the "CJK support for textwrap issue", @methane wrote in 2018:

If someone really want this feature, please try it on PyPI.

And @merwok just closed the issue as "not planned": #68853

It seems like some core devs prefer to have a solution on PyPI rather than in the stdlib because of the complexity of the issue. Well, at least, the suggestion is to start the implementation on PyPI rather than starting in the stdlib.

I only changed the status from «closed as completed» to «closed as not planned» to better reflect the decision. But I support Serhiy’s position than something good enough is better than nothing, if the implementation is reasonable (i.e. using unicode data character classes if possible, not very long mappings like I saw in one of the third-party projects).

Interesting, that character with different from 1 is supported in formatting in C++23: https://timsong-cpp.github.io/cppwp/n4950/format.string.std

For example, while format("{:*^6}", 'x') returns "**x***" (like "{:*^6}".format('x') in Python), format("{:*^6}", "🤡🤡🤡") returns "🤡🤡🤡", because the 🤡 character has width 2.

So, we may end supporting this too, if even C++ supports this. Then width() may be just a method of str, as Unicode aware 'isspace()', tolower() and splitlines() which internally depend on the large Unicode database.

Unfortunately, even if this is a part of the Posix and C++ standards, the exact behavior is not specified. I am looking at the code in glibc/g++, llvm and the Python's wcwidth -- they are different, and even if they support the same rang (e.g. Hangul characters), they do it differently. We cannot have the implementation that matches the system behavior on Linux and macOS.

Thus, it may be reason to provide also locale.width(), like in the original patch. It should be used if we want absolute consistency with the system behavior.

Navigation Menu

Repository navigation

Add functions to get the width in columns of a character #56777

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Add functions to get the width in columns of a character #56777

Description

Activity

poq commented on Mar 10, 2012

tchrist commented on Mar 10, 2012

NicholasCole commented on Mar 10, 2012

loewis commented on Mar 10, 2012

loewis commented on Mar 10, 2012

tchrist commented on Mar 10, 2012

loewis commented on Mar 10, 2012

tchrist commented on Mar 10, 2012

poq commented on Mar 11, 2012

loewis commented on Mar 11, 2012

loewis commented on Mar 11, 2012

poq commented on Mar 11, 2012

NicholasCole commented on Mar 11, 2012

vstinner commented on Mar 19, 2012

serhiy-storchaka commented on Mar 19, 2012

terryjreedy commented on Feb 2, 2013

vstinner commented on Mar 18, 2015

serhiy-storchaka commented on Nov 26, 2015

vstinner commented on Jun 28, 2017

bitdancer commented on Jul 1, 2017

serhiy-storchaka commented on Jul 1, 2017

vstinner commented on Jul 3, 2017

vstinner commented on Jul 3, 2017

Vermeille commented on Jul 13, 2017

terryjreedy commented on Aug 18, 2018

vstinner commented on Nov 8, 2018

serhiy-storchaka commented on Dec 8, 2025

vstinner commented on Dec 9, 2025

merwok commented on Dec 9, 2025

serhiy-storchaka commented on Dec 10, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions