Skip to content

feat(ts): translate str::from_utf8 and Utf8Error - #159

Merged
kixelated merged 6 commits into
mainfrom
quest/m1/ts-utf8
Oct 28, 2026
Merged

kixelated merged 6 commits into
mainfrom
quest/m1/ts-utf8

Conversation

@kixelated

@kixelated kixelated commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

moq-dev/moq#6843 makes moq-net's Decoder::str call std::str::from_utf8, so just ts moq at the moq revision #144 moves to (c764ace) refuses std::str::from_utf8 and std::str::Utf8Error. Finishes quest/m1/ts-utf8.md.

Consumer: moq-net's coding::Decoder::str, and setup::decode_implementation and the like.

Approach

  • str::from_utf8 (both core::str::from_utf8 and str's own) is rt.strFromUtf8: the fatal TextDecoder on a view of the slice, which rejects exactly what std rejects (surrogates, overlongs, past U+10FFFF, truncated), keeping a leading BOM as std does.
  • On failure it reports std's valid_up_to and error_len from the scan String::from_utf8 already used.
  • Utf8Error is rt.Utf8Error, a plain { validUpTo, errorLen }; valid_up_to and error_len read its fields; Display and Debug are runtime functions.
  • rt.FromUtf8Error now holds an rt.Utf8Error as std's does, so the two share one Display and Debug.
  • Tests: a new lang op, utf8::decodes, decodes from every offset of raw bytes or of an alphabet of UTF-8 boundary bytes and prints valid_up_to, error_len, Display, Debug, and {:#?}. caller.ts adds named cases for the BOM, surrogates, overlongs, past U+10FFFF, and truncations at the end or cut by another byte. Mutation-checked: widening the surrogate range in the runtime yields 51 differences.

Impact

  • Translator output: std::str::from_utf8, str::from_utf8, Utf8Error::{valid_up_to, error_len}, and Utf8Error's Display/Debug now translate instead of being refused.
  • Runtime, breaking: rt.FromUtf8Error's validUpTo and errorLen move to e.error.validUpTo and e.error.errorLen (MIGRATE.md entry). New exports: Utf8Error, strFromUtf8, fmtUtf8ErrorDebug, fmtUtf8ErrorDisplay.

Alternatives

  • Keep FromUtf8Error's flat fields and duplicate the Debug: dropped for one shared representation matching std's.

Checks

  • just check: pass.
  • just ts test: pass (unit, fixtures incl. lang differential, runtime).
  • just ts moq target/moq at c764ace: pass (translate + tsc). The pin itself stays at 88448d9; feat(ffi): export moq's owned Path #144 moves it.
  • Not run: just ffi/driver/uniffi test (untouched).

Follow-ups

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 3 commits October 27, 2026 23:43
Quest-Claim: 739e227efa79db49
`str::from_utf8` decodes with the fatal `TextDecoder` and, on failure, reports std's
`valid_up_to` and `error_len`. `Utf8Error` is its own runtime type, which
`FromUtf8Error` now holds as std's does, so both share one `Display` and `Debug`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Strings from UTF-8 bytes feat(ts): translate str::from_utf8 and Utf8Error Oct 28, 2026
@kixelated
kixelated marked this pull request as ready for review October 28, 2026 07:17
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T09:16:29.268742Z 75689a5 New commits
🔒 Security Review ✅ Completed 2026-10-28T09:18:25.572495Z 75689a5 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 5d5abe4

No actionable bugs found in the changed code. The direction is sound: reuse the fatal decoder and existing UTF-8 error scan, share error formatting between borrowed and owned decoding, and document the runtime representation change rather than maintain duplicate error logic. The differential fixtures cover offsets, BOM preservation, invalid boundary sequences, truncation, and formatting.

Verification limits: static review of the diff and relevant runtime/lowering code through GitHub; I did not run the translator, differential tests, or consumer build.

# Conflicts:
#	ts/fixtures/lang/src/harness.rs
#	ts/fixtures/lang/src/lib.rs
# Conflicts:
#	quest/m1/ts-utf8.md
#	ts/fixtures/lang/caller.ts
Main's ops 55 and 56 keep their numbers; utf8::decodes moves to 57.
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merge summary:

  • Merged main twice to resolve conflicts with #158, #124, and #139.
  • lang op numbers: main appended wrappers (55) and edits (56) after skips (54). This PR had inserted utf8::decodes at 54, so it moves to 57 and main's numbers stay as they are. caller.ts and the harness.ts snapshot are updated to match.
  • quest/m1/ts-utf8.md stays deleted. fix(ts): view a Vec by its own element type #158 had only edited it to drop the vec-view link.
  • Checks: just check and just ts test pass locally on 75689a5; CI Check passed; Codex code and security reviews of 75689a5 found nothing.

(Written by Claude Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant