fix(release): decode distribution smoke output as UTF-8

[raymondginger2018-sudo]

Description

While checking the other subprocess call sites for the locale-decoding failure that PR #237 fixed for .git/info/exclude, I found the same pattern in scripts/verify_python_distribution.py, but with a worse failure mode than a crash.

_run() calls subprocess.run(..., capture_output=True, text=True) with no encoding, so the child's output is decoded with the locale codec. When the child emits a byte the locale cannot decode, the decode raises inside subprocess's _readerthread -- and that exception never reaches _run(). The thread dies, the process result stays clean, and the verifier reports the command with no output at all:

distribution smoke command failed: .../python.exe -c import sys; sys.stdout.buffer.write(b'partial \x91 output\n'); ...

The exit code is still 0. So a fail-closed gate (python-ci.yml, pypi-publish.yml) can stay green while its diagnostics are silently thrown away, which is the part that made me file this rather than a plain robustness tweak. I reproduced both readings on this machine (Windows, cp936) in the before/after section.

Three changes, reusing the subprocess policy the runtime already has instead of restating it:

• _run() takes its text kwargs from subprocess_text_kwargs() (encoding="utf-8", errors="replace") and builds the child environment with subprocess_env(...), which also exports PYTHONUTF8=1 and PYTHONIOENCODING=utf-8. Declaring UTF-8 on the read side is only correct if the child speaks it; without that, CJK output from pip or the CLI degrades into replacement characters instead of being readable.
• main() calls configure_utf8_stdio(). The failure path prints a report and then raises SystemExit, and with errors="replace" that report can contain U+FFFD -- which a cp936 console cannot encode, so the report would crash on the way out and the real failure would go unreported.
• The module adds the repository root to sys.path before importing core.platform_compat, the way scripts/verify_transparency_log.py already does. CI runs the script by path, so sys.path[0] is scripts/.

On UTF-8 locales nothing changes. I kept the policy in one place because duplicating it here is how the verifier and the runtime it checks would drift apart.

Related Issues

None that I could find. This is the same bug class as #237 (core/team/worktree.py) and cdd2681 (hooks loader), one layer down in the subprocess read path.

Changes Made

• scripts/verify_python_distribution.py: _run() uses subprocess_text_kwargs() and subprocess_env(); main() calls configure_utf8_stdio(); repository root is added to sys.path so core.platform_compat can be imported when the file is run as a script.
• tests/test_python_distribution_release.py: test_smoke_failure_reports_output_that_the_reader_cannot_decode runs a child that writes b'partial \x91 output\n' and exits non-zero, then asserts the raised DistributionVerificationError still carries the decoded text (and the replacement character). test_smoke_children_are_told_to_write_utf8 pins the contract the other direction: the kwargs it passes and the PYTHONIOENCODING the child receives. Plus the import subprocess both use.

Checklist

• Changes tested locally
• Code reviewed
• Documentation updated (if necessary)
• Unit tests added (if applicable)

Additional Notes

Before and after, Windows 11, Python 3.14, cp936 locale, python -m pytest tests/test_python_distribution_release.py:

• Unpatched script: 2 failed, 11 passed, 1 error. The two new tests fail, and the error is pytest escalating the dying reader thread (filterwarnings = error), which is the clearest statement of the bug:

  self = <Thread(Thread-1 (_readerthread), stopped daemon 21216)>
      def _readerthread(self, fh, buffer):
  >       buffer.append(fh.read())
  E       UnicodeDecodeError: 'gbk' codec can't decode byte 0x91 in position 8: illegal multibyte sequence
  C:\...\Lib\subprocess.py:1614: UnicodeDecodeError
  

  with fh = <_io.TextIOWrapper name=11 encoding='cp936'> and the assertion showing the report that lost its diagnostics.

• Patched script: 13 passed.

• python scripts/verify_python_distribution.py --help and a run against a missing --dist-dir both behave (the sys.path guard), and ruff check / ruff format --check are clean on both files.

CI coverage: scripts/verify_python_distribution.py is not in the documentation allowlist in scripts/ci_scope.py, so runtime_changed=true and the verifier step in python-ci.yml runs on this PR. The full smoke path (real wheel, pip install into a throwaway venv) only runs there -- locally I verified the decode and reporting path directly.

[Zongwei9888]

Merged into main as f4ca992. Thank you @raymondginger2018-sudo — a fail-closed gate that stays green while its diagnostics die in subprocess's reader thread is a nasty failure mode, and reusing subprocess_text_kwargs() / subprocess_env() instead of restating the policy keeps the verifier and the runtime from drifting. I checked that core/__init__.py has no imports and core.platform_compat is stdlib-only, so the pypi-publish.yml job (which installs only build/packaging/twine) can import it.