[JDProfresh]

A text upload that fails the UTF-8 check was refused with the same "File bytes do not match declared type" message as a renamed binary, so an ordinary Excel or Notepad export read like a corrupt file. This is the first of the two follow-ups you asked for when merging #621.

Problem

Since #621, a prompt upload in the text family (txt, md, csv, tsv, json, log, yaml, yml, xml) has to be UTF-8 with no NUL byte. That rule is right, but Excel's default "CSV" save writes Windows-1252, and Excel's "Unicode Text" and Notepad's "Unicode" write UTF-16LE with a BOM. Any of those got 415 File bytes do not match declared type .csv, and the toast showed 1 failed: File bytes do not match declared type .csv. Nothing in that tells the user the file is fine and only the encoding is wrong, or what to do about it.

Cause

The document branch of POST /api/sessions/:id/paste-image used one message for every imageMagicMatchesExt failure, signature and encoding alike. The log line already told the two apart (reason=nul-byte / reason=not-utf8), but the response did not.

Fix

Only the wording of the text-family refusal changes. Nothing new is accepted and nothing is transcoded.

• A text file that opens with a UTF-16 BOM (FF FE or FE FF) is refused as This .txt file is UTF-16 text, not UTF-8; save it as UTF-8 and attach it again.
• Any other non-UTF-8 text file (Windows-1252, or a NUL byte without a BOM) is refused as This .csv file is not UTF-8 text; save it as UTF-8 and attach it again.
• The office, zip and PDF signature refusals keep their old wording, and so does the image path.
• The log line gains reason=utf16-bom beside nul-byte and not-utf8, and it still never prints the bytes.

The client already shows the server's reason (#578): every surface goes through _uploadAndInsertImages in image-input.js (phone Attach, drop, the tile grid's drop, the Compose and paste-dialog pickers), so no frontend change was needed. An empty file still gets its own 400 Empty file, which the route checks before dispatch, so I left it alone.

I did not add a zh-CN entry. Server reasons reach the user only inside the count-prefixed toast (1 failed: <reason>), which i18n.js matches neither exactly nor by pattern, so a bare entry would never apply. Server error strings in that toast are untranslated today.

Docs: docs/api-reference.md (paste-image documents paragraph), docs/wiki/Working-With-Files.md (now says what the refusal looks like and to save as UTF-8, or "CSV UTF-8" in Excel) and the Prompt uploads paragraph in CLAUDE.md (the log reasons and the message). Invariant touched: architecture-invariants#buffers-uploads-and-terminal-history (the text check and the never-print-the-bytes log rule); its sentences stay true unchanged.

Tests

test/routes/session-routes.test.ts:

• New: refuses non-UTF-8 text by naming the encoding, UTF-16 when a BOM says so sends a UTF-16LE file with its BOM, a UTF-16BE file, a NUL byte without a BOM and a Latin-1 CSV. It checks the exact error and the log line's reason= for each.
• Narrowed: refuses document bytes that do not match the declared type now covers only the signature formats and pins their exact old wording.

Red-proofs: with BOM detection disabled, the UTF-16 rows fail (Received: "This .txt file is not UTF-8 text; ..."). With the old message restored, they fail too (Received: "File bytes do not match declared type .txt"). With the text wording applied to every document, the PDF row fails (Received: "This .pdf file is not UTF-8 text; ...").

Gate: typecheck, lint, format:check, check:frontend-syntax and check:browser-excludes are clean. npm test gives 582 files and 11584 tests passed.

Verified

I ran it on a throwaway instance (CODEMAN_INSTANCE, its own port, Node 26.2.0) and sent each file with curl to POST /api/sessions/:id/paste-image. The fixtures were a real UTF-16LE file with its BOM (what "Unicode Text" writes), a UTF-16BE file, a Windows-1252 CSV with é as byte E9, a file with a NUL byte, a UTF-8 CSV and an empty CSV.

Before, on upstream master ce809eac:

unicode-text.txt  415 {"error":"File bytes do not match declared type .txt"}
utf16be.csv       415 {"error":"File bytes do not match declared type .csv"}
excel-1252.csv    415 {"error":"File bytes do not match declared type .csv"}
nul.txt           415 {"error":"File bytes do not match declared type .txt"}
utf8.csv          200 saved as .codeman-uploads/paste-...-utf8.csv
empty.csv         400 {"error":"Empty file"}
log: reason=nul-byte (unicode-text.txt, utf16be.csv, nul.txt), reason=not-utf8 (excel-1252.csv)

After, on this branch:

unicode-text.txt  415 {"error":"This .txt file is UTF-16 text, not UTF-8; save it as UTF-8 and attach it again"}
utf16be.csv       415 {"error":"This .csv file is UTF-16 text, not UTF-8; save it as UTF-8 and attach it again"}
excel-1252.csv    415 {"error":"This .csv file is not UTF-8 text; save it as UTF-8 and attach it again"}
nul.txt           415 {"error":"This .txt file is not UTF-8 text; save it as UTF-8 and attach it again"}
utf8.csv          200 saved as .codeman-uploads/paste-...-utf8.csv
empty.csv         400 {"error":"Empty file"}
log: reason=utf16-bom (unicode-text.txt, utf16be.csv), reason=not-utf8 (excel-1252.csv), reason=nul-byte (nul.txt)

I also checked it in a browser: headless Chromium with an iPhone 13 profile on the same instance, using the phone toolbar's Attach. The observed toasts were 1 failed: This .txt file is UTF-16 text, not UTF-8; save it as UTF-8 and attach it again for the UTF-16 file, 1 failed: This .csv file is not UTF-8 text; save it as UTF-8 and attach it again for the Windows-1252 CSV, and 1 file ready for the UTF-8 CSV.

What I inferred and did not run: the drop, tile-grid drop, Compose and paste-dialog pickers show the same toast because they call the same _uploadAndInsertImages. I read that path; I only drove Attach. The fixtures were generated byte for byte (python3 str.encode('utf-16'), a printf with \xe9), not saved from Excel or Notepad themselves.

[Ark0N]

Thank you @JDProfresh, this is exactly the follow-up from #621: a text upload that fails the UTF-8 check now tells the user it is not UTF-8 (or that it is UTF-16 when it opens with a BOM) and to save it as UTF-8, instead of reading like a corrupt file.

I ran your new test against master's route code and it fails there with the old wording, and typecheck, lint, format and the full npm test are green here. Status and errorCode are unchanged, so the API contract does not move. I also checked that an Excel "CSV UTF-8" export (UTF-8 with a BOM) is accepted, so the advice in the wiki works as written.

The only thing I will touch at merge time is cosmetic: the reflow in docs/wiki/Working-With-Files.md:192 left one 120-character line, and I will rewrap it.

Ready to merge as is; it goes in with the next release.

[Ark0N]

Merged and shipped in 1.43.0. Thanks @JDProfresh, the UTF-16 BOM case is a nice touch. The only change on top was rewrapping that one long line in the wiki page.