Localisation of more strings

[ic]

Problem and proposal

Using Click in a multi-lingual package, we would like to localise more strings than currently possible.

In our work before this PR, we reach like:

> export LANG=ja_JP.UTF-8
> artefacts hello
Usage: artefacts hello [OPTIONS] プロジェクト名
Try 'artefacts hello --help' for help.

Error: Missing argument 'プロジェクト名'.

The package we develop yields the artefacts command, based on Click, works alright to translate our strings, as well as a range of strings in Click wrapped with gettext.

With this PR, we get:

> export LANG=ja_JP.UTF-8
> artefacts hello
使用法:  artefacts hello [OPTIONS] プロジェクト名
ヘルプについては 'artefacts hello --help' を試してください。

エラー：引数がありません 'プロジェクト名'.

Which looks like possibly full coverage of meaningful strings in Click.

Note this PR addresses two issues:

1. Several strings where not wrapped into gettext
2. Libraries like PyBabel (most users?) cannot work with f-strings, so the pathological approach to use the format method, and to deactivate the PyUpgrade U032 rule (converts to f-strings).

Limitation, side-effects and discussion

It looks like part of this PR may be desired, as localisation is already in place---this mainly covers more of the strings (I'd say all the strings, but localisation does not always make sense).

However in the current approach I had to "deal" with f-strings and PyUpgrade, which may be undesired change for the project. On top of that it seems there is a bug in either Ruff or PyUpgrade in accepting ignore rules on multi-line commands. So a bunch of files get a file-global deactivation of the U032 rule, which means these files will not get checked for f-strings.

F-strings are assumedly desired, but given (1) PyBabel does not and may not support them, and (2) Click is library code, the project may (have to) accept the format method everywhere localisation is needed.

Recent discussions mention f-strings may work in PyBabel from Python 3.12. We did not confirm it, as we want to support down to the oldest supported Python3. So the changed proposed here may be transient for a couple years (then PyUpgrade may be reactivated and let work).

Related work

This PR only aims at localising more strings, so related to i18n issues.

On the way to this PR, we have considered a couple alternatives, notably trying to use the class API of Python's gettext. Some elements of discussion here may be useful to:

• Improve internationalization #2864 by @v--
• Better i18n with gettext: use class-based API #2706 by @carmenbianca

In fact, Carmen's post helped solve an issue in using catalogues from different domains at runtime (thanks!).

Checklist on CONTRIBUTING

• Add tests that demonstrate the correct behavior of the change. Tests should fail without the change.
• Add or update relevant docs, in the docs folder and in code.
• Add an entry in CHANGES.rst summarizing the change and linking to the issue.
• Add .. versionchanged:: entries in any relevant code docs.

At submission time, nothing checked here, as first would like to make sure this PR target is acceptable (likely not as-is). The tox-based checks all pass, though (i.e. running the tox command returns all green, except the skipped tests).

This PR supersedes #2890, because of a problem with GitHub.

[Rowlando13]

I don't think we are willing to limit the use of f strings in the project at this time. Thoughts @davidism?

[davidism]

None of these noqa marks are needed. Ruff is not trying to autoupgrade anything if I remove the comments, inline or file-level.

[ic]

Updated from former Ruff 0.8.1 to the latest 0.14.10, and as you reported, the UP032 do not trigger anymore.

All files cleaned from the comments, as well as the noqa marks.

[davidism]

We're only translating user-facing messages at this time, not developer-facing. Please remove all such translation markings.

[ic]

I have done a pass on all new translations and removed the ones that looked like dev-facing. How does it look now? I am not sure about a couple.

[davidism]

For user-facing messages, yes we can use the _("").format() pattern, f-strings can't be used for translation.

The # noqa: UP036 mark does not seem to be needed, please remove this.

We're only translating user-facing messages at this time, not developer-facing. Please remove all such translation markings.

[davidism]

This shouldn't have ever been marked for translation. It should be removed, since you've added the translation to the proper location above.

[ic]

Done!

[davidism]

Most of the changes here need to be rolled back, they are not part of the user-facing output.

[ic]

Hey thanks for the review and feedback. A quick note to let you know I expect to improve the work here soon, most likely in November. Sorry for anyone watching and expecting this sooner.

[AndreasBackx]

This looks fine now. Though I'd love for someone else to give this a final eye because I've never really touched too much gettext stuff.

[ic]

For the record and contribute to the review, we have been using this branch for a year in our CLI, without noticeable problem so far:

The CLI is for robotics engineering: pip install artefacts-cli

# export LANGUAGE=en_GB:en
# artefacts run-remote test_manipulation
Scanning source...
Job data compression: 5/5 Done
Job data uploading:   5/5 Done
Job meta uploading:   3/3 Done
Uploaded 42 MiB of compressed job data and 1 KiB of job metadata.
The new job will show up shortly at https://app.artefacts.com/artefacts/artefacts-toolkit-testsuite

# export LANGUAGE=ja_JP:ja
# artefacts run-remote test_manipulation
ソースをスキャン中...
ジョブデータの圧縮: 5/5 完了
ジョブデータのアップロード:   5/5 完了
ジョブメタデータのアップロード:   3/3 完了
圧縮されたジョブデータ 32 MiB 、およびジョブメタデータ 2 KiB のアップロードが完了しました。
新しいジョブは、まもなく https://app.artefacts.com/artefacts/artefacts-toolkit-testsuite に表示されます。

(if you try the package, the output will differ as working on it, but the translation is in use for a year already)

[kdeldycke]

@ic how hard would it be to add a couple of unittests? At least to prove and lockdown that Click is properly translating strings? With unittests we can force future contributors to be aware of this support.

Other than that this PR looks good for inclusion into 8.4.0.

[kdeldycke]

Grepping through the code I just realized some default strings are not translated, like in:

• click/src/click/decorators.py

  Lines 399 to 400 in 831c8f0

  	kwargs.setdefault("prompt", "Do you want to continue?")
  	kwargs.setdefault("help", "Confirm the action without prompting.")

• click/src/click/core.py

  Lines 1139 to 1143 in 831c8f0

  	deprecated_message = (
  	f"(DEPRECATED: {self.deprecated})"
  	if isinstance(self.deprecated, str)
  	else "(DEPRECATED)"
  	)

• click/src/click/core.py

  Lines 2786 to 2790 in 831c8f0

  	deprecated_message = (
  	f"(DEPRECATED: {deprecated})"
  	if isinstance(deprecated, str)
  	else "(DEPRECATED)"
  	)

• click/src/click/core.py

  Lines 2600 to 2607 in 831c8f0

  	message = _(
  	"DeprecationWarning: The {param_type} {name!r} is deprecated."
  	"{extra_message}"
  	).format(
  	param_type=self.param_type_name,
  	name=self.human_readable_name,
  	extra_message=extra_message,
  	)

• click/src/click/termui.py

  Line 258 in 831c8f0

  "y/n" if default is None else ("Y/n" if default else "y/N"),

• click/src/click/termui.py

  Lines 266 to 269 in 831c8f0

  	if value in ("y", "yes"):
  	rv = True
  	elif value in ("n", "no"):
  	rv = False

[ic]

@kdeldycke Thank you for the round of review.

1. Unit test, on it starting today. It looks like I can come up with some next week.
2. Translation strings like Usage: are often better be usage and let the code format and punctuate. Not done to minimise code change, but personal preference and suggestion, perhaps a later PR.
3. The default strings you list up should be included. Two questions:

• The PR is one year old, and some may be new strings. If acceptable, perhaps working first toward merging this PR, then iterate? As writing tests by next week, the ones you found can be done at the same time.
• An earlier comment by @davidism points out some strings are for developers only and should not be translated. It makes sense to me as dev is done in English anyway. It looks like only 2 of the ones you found enter this category. Best if we can confirm before I make us oscillating change/review.

[kdeldycke]

1. Unit test, on it starting today. It looks like I can come up with some next week.

No worries, take your time! :) There is a couple of other PRs aligned for 8.4.0 that are going to take more time to land so we're good here! :)

2. Translation strings like Usage: are often better be usage and let the code format and punctuate. Not done to minimise code change, but personal preference and suggestion, perhaps a later PR.

Your attention to punctuation and separation of concern is the right one. See my comment on the review discussion at: #2902 (comment) . My proposal is to simply reduce the diff to make the merge to upstream even more easier.

3. The default strings you list up should be included. Two questions:

• The PR is one year old, and some may be new strings. If acceptable, perhaps working first toward merging this PR, then iterate? As writing tests by next week, the ones you found can be done at the same time.

It's better to keep this PR self-contained. Now that this PR has both your attention and mine, I will keep tracking it and helping you so it lands in the upcoming 8.4.0. If you need more time no worries. Just tell us with a quick comment here. If you find the merge too hard to do let me know. I can take over this PR to cleanup and prepare it for merging.

• An earlier comment by @davidism points out some strings are for developers only and should not be translated. It makes sense to me as dev is done in English anyway. It looks like only 2 of the ones you found enter this category. Best if we can confirm before I make us oscillating change/review.

He is right that only user-facing strings should be translated. Re-reviewing the one I pointed out, yes, they should be translated as these are user-facing strings in the help screen: they're about options or parameters that are marked as deprecated.

[ic]

@kdeldycke Thank you for the detail. Proceeding accordingly.

A thought on unit testing: A couple generic tests on the _ utility works, and wondering whether a CI workflow to check for missing translation would not be useful? Done a couple times in other projects, where pybabel can list missing translations, and help failing some CI jobs and add GH labels for missing translations, etc. Retrospectively such CI is more informative and useful to me, rather than unit tests. How does it sound?

[davidism]

I don't really think this needs to be unit tested. Seems like a lot of work just to get this in. Maybe we can do that in another talk, but I'm not sure what those tests would reveal anyway.

[kdeldycke]

I don't really think this needs to be unit tested. Seems like a lot of work just to get this in. Maybe we can do that in another talk, but I'm not sure what those tests would reveal anyway.

What I was thinking about is a kind of test just doing what @ic is demonstrating in its comment #2902 (comment) . And have that test called test_translation or something generic like that, that is just a big and stupid string comparison of the output in Japanese. This would at least cover the high-level translation code path. Could be added to test_basic.py. Nobody cared about translation until @ic showed up. I can add that test later in a subsequent PR.

So @ic, no need to go to a complicated route like you proposed with pybabel deep introspection of the code. Just a high-level test as I describe is good enough, and better than nothing. Can you also squash your commits here?

[ic]

Starting to proceed with the changes.

1. Squashing commits can be done on merge, or you have something specific in mind?
2. [Done] I do now the missing messages, and
3. [Update] next on unit test if still wanted. Unit tests would amount to test unrelated gettext and ngettext functions from the gettext standard library. Can still do, but now thinking it is not useful to the test suite (it would be useful with custom use, but here plain use on purpose).
4. The CI approach is quite straight forward, with the advantage to be comprehensive. Something like (based on my working workflow):

pybabel extract src -o path/to/locales/base.pot # and other options
pybabel update  -i path/to/locales/base.pot -d path/to/locales
pybabel compile -d path/to/locales
msgattrib --untranslated path/to/locales/ja_JP/LC_MESSAGES/artefacts.po -o untranslated.po
if [[ -s untranslated.po ]]
then
  echo "Found  untranslated strings"
  echo "Please translate them and commit"
  gh pr edit ${{ github.event.pull_request.number }} --add-label "Empty translations"
  cat untranslated.po | grep msgid
  exit 1
else
  echo "All expected translations ready to go."
  gh pr edit ${{ github.event.pull_request.number }} --remove-label "Empty translations"
fi

Note: All tests pass according to CI here. They also pass on a local Linux machine. But on a Mac Intel, I have a single error with Python 3.14t, 3.13 stress, and tried also the random environment in Tox. It looks unrelated, but for reference:

tests/test_utils.py:438: in test_echo_via_pager
    assert pager == expected_pager, (
E   AssertionError: Unexpected pager output in test case 'Exception in generator function argument'
E   assert 'test' == ''
E
E     + test

[kdeldycke]

Maybe it's too much but why not split this into:

• {_("y")/_("n")}
• {_("y").upper()/_("n")}
• {_("y")/_("n").upper()}

The idea is to reuse the same _("y") and _("n") items you translated below? Or is this too much splitting?

[ThiefMaster]

Do people typically want Y/N in this type of CLI translated? E.g. do French speakaers really want O/N instead, or German speakers J/N? (I can tell you that as a German speakers I would never want this!)

[kdeldycke]

Do people typically want Y/N in this type of CLI translated? E.g. do French speakaers really want O/N instead, or German speakers J/N? (I can tell you that as a German speakers I would never want this!)

I personally agree with you and find that awkward. But it is user facing, so in the name of applying a blanket translatable policy, let's allow this to be translated and changed. In the end this is about taste. So at least with this we allow CLI developers choose to apply the translation or not. The capability exist and is consistent, the rest is developers preferences. 🤷

[ThiefMaster]

But does a CLI developer get to choose? If they enabled translations in general, they'd also get this one. Which is also kind of a breaking change because it changes the behavior of the CLI. I would make it opt-in somehow.

[ThiefMaster]

I just tested in a docker container with debian and a german translation and how the rm -i ... prompt behaves: it accepts j and ja as yes, but y and yes also work. likewise with french where o and oui are acccepted, but also the english defaults.

So I strongly suggest we do the same here and at least always accept the standard values on top of additional ones coming from translations.

[kdeldycke]

I just tested in a docker container with debian and a german translation and how the rm -i ... prompt behaves: it accepts j and ja as yes, but y and yes also work. likewise with french where o and oui are acccepted, but also the english defaults.

So I strongly suggest we do the same here and at least always accept the standard values on top of additional ones coming from translations.

Oh I see. Sorry I did not got your point in your first comment. Thanks for checking other CLIs and illustrating them. And so yes, I agree with you on that front and we should accept both the translated _("y") and _("yes") as well as the original y and yes as valid answer in the code below.

[ic]

Oh I got caught very lazy on this one. Thank you both for the feedback. I was actually wondering how people were workin in a different language... In non-alphabet languages, it seems to me developers (a tiny fraction of a population, right?) work with and expect y/n.

Interestingly asked Qwen (an LLM developed in China) how it is preferred there. It replies they expect y/n. The example it reports:

是否继续？(Y/n): 
确认安装？(y/N): 
删除文件？(Y/n): 

Same for Japanese. It goes as far as to list "best practices" like: "Never translate y/N or Y/n: Maintains cross-language consistency and avoids terminal width issues."

[kdeldycke]

4. The CI approach is quite straight forward, with the advantage to be comprehensive. Something like (based on my working workflow):

Ok let's see that point in a future PR.

Thanks for your feedback on the tests, we will see that topic later in a future PR.

I just have a last comment on the y/n format, and once you answer that, I will squash that PR and merge it.

[ic]

Experimented with translations on y/n, I concur with @ThiefMaster now. I did try to offer two options like y/n or custom, but there are really basic issues in general. For example "y" in Chinese is a single character (是/shi), but you need to type 2-3 keys before getting the character. Same in Japanese where はい/hai requires 3 keystrokes (and no one in Japan would associate "h" or "ha" to the idea of approval). Mismatch char/keys breaks the UX for some "popular" languages.

Another issue is default format. Language without notion of capitalisation do not match the y/N pattern. Many CLIs opt for the format y/n [n], with the convention that square brakets mean optional entries, with default. Stashed an implementation here:

# src/click/termui.py
# in the confirm function
def confirm(...):
    yes_mark = _("y")
    no_mark = _("n")
    def_mark = (
        "" if default is None else (f" ({yes_mark})" if default else f" ({no_mark})")
    )

    prompt = _build_prompt(
        text, prompt_suffix, show_default, f"{yes_mark}/{no_mark}{def_mark}"
    )

    while True:
        try:
            value = _readline_prompt(visible_prompt_func, prompt, err).lower().strip()
        except (KeyboardInterrupt, EOFError):
            raise Abort() from None
        if value in ("y", "yes", _("y"), _("yes")):   # <= here accepting y and translations
            rv = True
        elif value in ("n", "no", _("n"), _("no")):   # <= here accepting n and translations
            rv = False
...

This would lead to output like (chose round brackets for readability):

Prompt to stdin with no suffix [y/n (n)]

stdin へのプロンプト(サフィックスなし) [はい/いいえ (いいえ)]

标准输入提示(无后缀) [是/否 (否)]

As much as I'd like full localisation, this looks regular enough but not great. If Click users are essentially developers, y/n really looks the way to go. On option for the rare users who want customisation is best, though, perhaps out of scope of this PR (stashed the above, but need a mechanism to switch modes).

[ic]

@kdeldycke Conclusion on the geeking out here: I propose to rollback the y/n commit (0b75b01) for now, and freeze the PR scope.

The thinking:

• y/n translated is often not expected, or not working for non-alphabet languages (1 char is not 1 keystroke in many of them).
• Regular localisation would require really visible format change from y/N to y/n [n] to apply to most language.
• y/n can be future change, assuming Click users are essentially developers.

[kdeldycke]

I propose to rollback the y/n commit (0b75b01) for now, and freeze the PR scope.

Oh yes. That's full of edge-cases. Let's keep things simple for now. You can revert that commit, and I will merge after that. Thanks a lot for your deep tests!

[ic]

Oh yes. That's full of edge-cases. Let's keep things simple for now. You can revert that commit, and I will merge after that. Thanks a lot for your deep tests!

Rollback completed.

[kdeldycke]

Thanks @ic for your patience and for your work! Merged upstream and will be part of Click 8.4.