decodeCsvText (documents.js, src/csv/text.ts) and markdown-codec's decode both use a fatal-mode UTF-8 TextDecoder, so any input that isn't well-formed UTF-8 throws (CsvInvalidUtf8Error for csv). That was a deliberate choice, so the boundary fails loudly instead of putting U+FFFD into everything downstream, but it turns away two common kinds of ordinary text file.
Excel's default "CSV (Comma delimited)" export on Windows writes the system ANSI codepage (windows-1252 in Western locales), and Notepad's "Unicode" save and PowerShell 5.1's > write UTF-16 with a BOM. A downstream consumer measured its own UTF-8-replacement-ratio check against those and refused a French staff list, a column of German surnames and the four bytes of café (11%, 15% and 25% replacement characters respectively).
What that consumer ended up doing, which seems a reasonable shape: honour a UTF-16 BOM first, then UTF-8 if it decodes cleanly, then windows-1252 only as a last resort behind a check that the result looks like Western text (windows-1252 maps almost every byte to something, so it can't be tested by trying it). It also separated "is this text at all" from "which encoding": a NUL byte or a high density of other C0 control bytes says binary regardless of encoding, whereas the replacement-character ratio was balancing on a boundary.
A UTF-16 BOM is unambiguous, so honouring it looks safe as a default. The windows-1252 fallback is the part that needs a decision, because a wrong guess corrupts text silently: throw as now, guess and report that it guessed, or take an explicit encoding option. The same decision applies to markdown-codec and to any plain-text entry point we add.
decodeCsvText(documents.js,src/csv/text.ts) and markdown-codec's decode both use a fatal-mode UTF-8TextDecoder, so any input that isn't well-formed UTF-8 throws (CsvInvalidUtf8Errorfor csv). That was a deliberate choice, so the boundary fails loudly instead of putting U+FFFD into everything downstream, but it turns away two common kinds of ordinary text file.Excel's default "CSV (Comma delimited)" export on Windows writes the system ANSI codepage (windows-1252 in Western locales), and Notepad's "Unicode" save and PowerShell 5.1's
>write UTF-16 with a BOM. A downstream consumer measured its own UTF-8-replacement-ratio check against those and refused a French staff list, a column of German surnames and the four bytes ofcafé(11%, 15% and 25% replacement characters respectively).What that consumer ended up doing, which seems a reasonable shape: honour a UTF-16 BOM first, then UTF-8 if it decodes cleanly, then windows-1252 only as a last resort behind a check that the result looks like Western text (windows-1252 maps almost every byte to something, so it can't be tested by trying it). It also separated "is this text at all" from "which encoding": a NUL byte or a high density of other C0 control bytes says binary regardless of encoding, whereas the replacement-character ratio was balancing on a boundary.
A UTF-16 BOM is unambiguous, so honouring it looks safe as a default. The windows-1252 fallback is the part that needs a decision, because a wrong guess corrupts text silently: throw as now, guess and report that it guessed, or take an explicit
encodingoption. The same decision applies to markdown-codec and to any plain-text entry point we add.