perf: encode the Typst source straight into native memory - #48
Open
msallin wants to merge 1 commit into
Open
Conversation
Encoding.UTF8.GetBytes copied the whole source onto the managed heap for the duration of one native call and left it as garbage afterwards; the Rust side copies the bytes into an owned String, so the array was only a staging buffer. A 200 KB source measured 200,024 bytes, on the large object heap. It is now encoded into the native buffer directly, which also removes the one-byte placeholder the empty-source case needed to avoid a null pointer. The native allocations moved inside the try whose finally releases them. They were made before it, so a throw from the caller-supplied font path sequence or from serialising the system inputs leaked everything allocated up to that point.
msallin
force-pushed
the
feat/source-utf8-native-encode
branch
from
September 6, 2026 19:51
9609ebc to
8312736
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Encoding.UTF8.GetBytescopied the whole source onto the managed heap for one native call and left it as garbage. The Rust side copies the bytes into an ownedString, so the array was only a staging buffer. A 200 KB source measured 200,024 bytes, on the large object heap; it is now encoded into the native buffer directly.The
(pointer, length)contract is unchanged, so embedded NUL bytes still survive. Thenew byte[1]placeholder goes away with it — it existed only becausefixedover an empty array yields a null pointer, which the native side reads as "no in-memory source".Also moves the native allocations inside the
trywhosefinallyreleases them. They were made before it, withfontPaths.ToList()over a caller-supplied lazy sequence andJsonSerializer.Serializein between; either could throw and leak everything allocated up to that point.Tests: empty source, which pins the null-pointer-versus-empty distinction the placeholder used to carry, and a source above 85 KB with multi-byte characters.
58/58 pass, clean on net8/9/10.