ENG-824: force UTF-8 in the scratchpad so non-UTF-8 host locales (GBK/CJK Windows) don't crash

[alecantu7]

What

On a host whose default code page isn't UTF-8 — GBK/cp936 on Chinese Windows, and likely other CJK locales — LocalScratchpadRuntime crashes before any cell runs, so the scratchpad (and thus most real tasks) is 100% unusable for those users.

Root cause (confirmed)

local.py crossed the parent↔scratchpad byte boundary using the host locale encoding everywhere. First failure: _BOOT_SCRIPT_PATH.read_text() decodes the 32 KB boot script with the locale default; the script contains …/— (a … at byte ~6382), so on GBK it throws 'gbk' codec can't decode byte 0xa6. Verified locally: the boot script's bytes decode fine as UTF-8 but raise under GBK — byte 0xa6 matches the reporter's error exactly.

Fix (interpreter-level + explicit at every boundary)

• Boot script read/written as UTF-8 (read_text(encoding="utf-8") / encode("utf-8")).
• Subprocess env forces UTF-8 mode — PYTHONUTF8=1 / PYTHONIOENCODING=utf-8 via _utf8_env (setdefault, so an explicit operator override still wins) — so the child's file I/O + stdio are UTF-8 regardless of host locale, covering user data (Chinese DB rows/CSVs) too.
• Cell payload + stdout/install output encoded/decoded as UTF-8 (errors="replace" on display output so stray bytes never crash the reader).

Why it can't break anything

All of this content is already UTF-8 on the wire (anton writes UTF-8). On macOS/Linux/English-Windows the default is already UTF-8 → behavior is identical. It only changes non-UTF-8-locale hosts — turning a hard crash into working. There's no working baseline to regress (it was a 100% crash there).

Testing

• New tests/test_scratchpad_utf8.py: _utf8_env forces UTF-8 / respects an explicit override; the boot script must be read as UTF-8 (its bytes are not GBK-decodable — the regression guard).
• 96 existing scratchpad tests pass (test_scratchpad.py, test_scratchpad_observer_dispatch.py).

Origin

Prod Cowork-desktop (Windows) user tried to test a PostgreSQL connection 3× today; every attempt died on this GBK codec error (Langfuse tool:report_failure ×3). See ENG-824.

Coordination

This is the scratchpad half. A companion cowork PR sets PYTHONUTF8=1 on the cowork-server sidecar launch, covering the non-scratchpad anton reads (workspace anton.md/.env, memory store) at the process level. The two are independent (either can merge first); together they close the whole locale surface. Both reach users on the next desktop build (cowork-server bumps its anton dep).

Security check

Encoding-only change; no new surface, no secrets touched.

Closes ENG-824 (scratchpad half).

[alecantu7]

Self-review (adversarial)

Major — found & fixed (0dc7261). I originally set both PYTHONUTF8=1 and PYTHONIOENCODING=utf-8. Verified empirically that a bare PYTHONIOENCODING=utf-8 downgrades the child's stdio error handler from surrogateescape → strict:

PYTHONUTF8=1                     → stdin/stdout errors = surrogateescape
PYTHONUTF8=1 PYTHONIOENCODING=utf-8 → stdin/stdout errors = strict   ← regression

It added nothing (UTF-8 mode already makes open()/filesystem/stdio UTF-8) and re-introduced a strict-encode crash on exotic output — the opposite of the goal. Dropped it; keep PYTHONUTF8=1 only (test updated to assert PYTHONIOENCODING is absent).

Low — noted, not changed:

• A few venv-management open()s (local.py ~271/300/311/324/334: .pth, requirements, python-version files) still use the locale default. Content is ASCII / same-locale round-trip, and at runtime the process runs under PYTHONUTF8=1 (from cowork#420, or _utf8_env for the child), so open() defaults to UTF-8 anyway. Left as-is to keep the diff focused; can sweep separately.
• test_boot_script_must_be_read_as_utf8 asserts the boot script is not GBK-decodable — intentionally brittle: if the script ever goes pure-ASCII the guard fires and prompts a conscious update.
• _utf8_env annotates os._Environ[str] (private type) — could tighten to Mapping[str, str]; cosmetic.

Electron/CLI paths unchanged; 96 existing scratchpad tests + 3 new pass.

[alecantu7]

/code-review skill pass (high) — outcome

Ran the review skill (complementing the earlier manual pass). No new correctness bugs; the substantive one (PYTHONIOENCODING strict-downgrade) was already found + fixed. One consistency gap surfaced and now fixed (a4ba80a):

• The dependency install subprocess (pip/uv) wasn't given _utf8_env, only the main scratchpad process was — so on a non-UTF-8 host locale its output could come back as mojibake. Now passes env=_utf8_env(os.environ) too.

Remaining findings are low/noted, not changing: forcing UTF-8 mode also flips the user's own open() default to UTF-8 (behavior note, not a regression — the scratchpad didn't start at all before); the decode("gbk") test guard is intentionally brittle; os._Environ annotation is cosmetic.

[ZoranPandovski]

@alecantu7 _setup_parent_site_packages still writes _parent_venv.pth with a plain open(pth_path, "w"), so it's locale-encoded, while the child (now in UTF-8 mode) reads .pth files as UTF-8.
Worth adding encoding="utf-8" there in this PR since it's the same class of bug.

[ZoranPandovski]

The boot script test checks the file's bytes rather than how we read it. If someone reverts read_text(encoding="utf-8") back to read_text(), the test still passes on UTF-8 CI.
Could we make it pin the actual read, e.g. monkeypatch Path.read_text and assert it's called with utf-8?

[alecantu7]

Thanks @ZoranPandovski — both addressed in d9406ce:

1. _parent_venv.pth write — now open(pth_path, "w", encoding="utf-8"). You're right it's the same class of bug: a locale-encoded write vs a UTF-8 read of .pth under UTF-8 mode would corrupt non-ASCII site-packages paths.
2. Test pins the read, not just the bytes — extracted _read_boot_script() and added test_boot_script_is_read_as_utf8, which spies on Path.read_text and asserts the boot-script read passes encoding="utf-8". Reverting to a bare read_text() now fails even on UTF-8 CI, which the bytes-only check couldn't catch.

All 4 tests green locally. Re-requesting review.

[alecantu7]

Self-review (adversarial) of the review-response commits

A — Asymmetric coverage → fixed (1fc202f). I'd pinned the boot-script read as UTF-8 but left the sibling _parent_venv.pth write with no regression test — asymmetric with the exact "same class of bug" concern raised here. Added test_parent_venv_pth_is_written_as_utf8, which spies on open() and asserts the .pth write passes encoding="utf-8". Verified it fails if the encoding is dropped and passes when restored. (Bypasses the heavy __init__ via object.__new__ so there's no network/subprocess — the method only reads self._venv_dir.)

B — Premise re-checked, holds. The write fix assumes the child reads .pth as UTF-8: the scratchpad subprocess always launches with PYTHONUTF8=1 (_utf8_env), and _setup_parent_site_packages runs in the parent writing into that child's venv — so parent-writes-UTF-8 + child-reads-UTF-8 is consistent. Real-world trigger is a CJK Windows user dir in getsitepackages().

C — Minor, accepted as-is. The read spy couples to the keyword-arg form (read_text(encoding=...)); a positional call would false-fail. Intentional — it pins the exact call form the codebase uses everywhere.

Full file: 5/5 tests green locally.