[ENG-4058] Screen response bodies in the character encoding a client reads

[patchstackdave]

Stacked on #297. Review that first; this PR's own changes are the last two commits.

Response screening reads text as UTF-8. This makes the reading match what the client will see, and reports a body that can't be read that way.

Character encodings

• A byte-order mark decides first, as it does for a client. Otherwise the declared charset does.
• A UTF-8 byte-order mark is left out of the text that's screened and put back in front of a rewritten body, so the client still reads the rewrite as UTF-8 whatever the declared charset says.
• A UTF-8 or ASCII label, or none, is screened as before.
• A body that can't be read as UTF-8 is sent unchanged and recorded as an unsupported-charset skip, with the charset, through onSkip and coverage(). That covers UTF-16/32 (declared, or marked by a byte-order mark) and any other charset whose bytes include non-ASCII, NUL, or ESC. Rules decided on headers alone ([ENG-4058] Apply header-only rules when the body is not screened #297) still apply to it on both paths.
• A legacy charset over plain ASCII bytes reads identically, so it is still screened. A rewrite that would add non-ASCII to it (a non-ASCII maskWith) gets the existing withheld response rather than being sent under the wrong charset.

JSON escapes

• A JSON body is screened with \uXXXX (printable ASCII) and \/ escapes read as a JSON parser reads them, so a value is matched as the client will see it.
• Escapes that keep the document well-formed (quote, backslash, control characters) or markup-safe (<, >, &, ', and \/ after <) stay escaped in a rewritten body, and it parses to the same values. Non-ASCII escapes are left as they are.
• A body that doesn't change is sent exactly as it was. A body that isn't valid JSON is not reinterpreted.

Decoding other charsets so they can be screened is not part of this change.

Tests: tests/protect/response-character-encoding.test.ts covers, on the fetch path and the Node path (headers set before the body and through writeHead()):

• each charset case;
• withheld and allowed non-ASCII rewrites;
• byte-order-mark precedence and preservation, including JSON behind a byte-order mark;
• header redaction and block rules on a body in an unsupported charset;
• JSON escape handling, including an escaped backslash, non-JSON bodies, and unmatched bodies.

Two tests in tests/protect/response-json-structure.test.ts now rewrite a non-ASCII escape (\u00e9) instead of a printable ASCII one. Screening reads a printable ASCII escape as its character, so a rewrite can no longer land inside one; a non-ASCII escape stays escaped, and still exercises the structure check.

Validation: full suite (3,971 passed, 7 skipped), typecheck, and build.

Part of ENG-4058.

🤖 Generated with Claude Code

[coderbuds]

Comprehensive charset-aware screening with robust JSON escape handling.

🎯 Quality: 100% Elite · 📦 Size: Large — consider splitting if possible

🛡️ Standards: Not checked — 451 lines changed, over your team's 400-line limit, and nothing checked before it was opened. Coding agents can call the assess-change-fit tool first, while a change this size is still cheap to split.

🤖 Authorship: Agent-written — Claude Code, going by its own attribution. Whether a person read it is unknown; coding agents can call the report-ai-usage tool to say.

📈 This month: Your 164th PR — above team average · Averaging Excellent

_{See how your team is trending →}

[patchstackdave]

/review