Skip to content

Commit 2e843f1

Browse files
committed
feat: add TextEncoder/TextDecoder and atob/btoa on a lazy-global tier
Mirrors NativeScript/ios#448: native WHATWG TextEncoder/TextDecoder (utf-8, utf-16le, utf-16be, windows-1252 with full label sets, streaming decode, exact replacement semantics) and forgiving-base64 atob/btoa, registered through a new lazy-global tier (LazyGlobals): each global is a SetLazyDataProperty on the global template, so the builtin behind it is compiled and run only on first read, once per isolate, with sibling names sharing the run through a RuntimeState slot. Unlike ios there is no metadata-interceptor decline hook — android has no global named-property interceptor, so none is needed. encodeInto registers a V8 Fast API overload (NATIVESCRIPT_ENABLE_FAST_API, default on), live on android's JIT tiers. Bumps the shared test suite for the 94 TextEncoding conformance specs and wires it into mainpage.js.
1 parent 6ebb265 commit 2e843f1

18 files changed

Lines changed: 1363 additions & 6 deletions

docs/README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,10 @@
1313
layered on the runtime's `EventTarget`, the GC contract (weak timers and
1414
`any()` links, listener-driven persistence), and the `DOMException`
1515
stand-in (name-patched `Error` reasons).
16+
- [TextEncoder / TextDecoder and atob / btoa](text-encoding.md) — the WHATWG
17+
encoding and base64 globals (`TextEncoder`, `TextDecoder`, `atob`, `btoa`),
18+
the supported encodings with their label sets, streaming decode semantics,
19+
and the lazy-global tier that runs their builtins only on first use.
1620
- [Error handling](error-handling.md) — global `error`/`unhandledrejection` events, `reportError`, catching Java exceptions in JS (`error.nativeException`), forwarding JS throws to Java callers (`interop.escapeException`), JS stacks on Java exceptions (`com.tns.JavaScriptStackTrace`), configuration flags, and crash-reporter integration.
1721
- [structuredClone](structured-clone.md) — the WHATWG `structuredClone(value, { transfer })` global: what clones, how graph identity and cycles are preserved, `ArrayBuffer` transfer, and the `DataCloneError`-named `Error` that stands in for `DOMException`.
1822
- [Implementing additional Chrome DevTools protocol Domains](extending-inspector.md)

docs/text-encoding.md

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# TextEncoder / TextDecoder and atob / btoa
2+
3+
Native, WHATWG-conformant `TextEncoder`, `TextDecoder`
4+
([Encoding Standard](https://encoding.spec.whatwg.org)) and `atob` / `btoa`
5+
([HTML Standard §8.3](https://html.spec.whatwg.org/multipage/webappapis.html#atob))
6+
globals, and the **lazy-global tier** they ride on.
7+
8+
## Lazy globals
9+
10+
These globals are registered on the global template as lazy data properties
11+
(`LazyGlobals`, `test-app/runtime/src/main/cpp/LazyGlobals.cpp`): the builtin
12+
behind a name is not compiled, run, or allocated until app code first reads it,
13+
and V8 then replaces the property with a plain data property so later reads
14+
cost nothing. Sibling names from one builtin (`TextEncoder` + `TextDecoder`)
15+
share a single run per isolate. Workers get the same globals — the tier is
16+
registered in every isolate's template. Assigning over one of these names
17+
before its first read replaces the global, like any other writable global.
18+
19+
The tier is the intended home for further web globals (`Blob`, `fetch`,
20+
`crypto`, `DOMException`, …) with zero cost when unused; see
21+
`test-app/runtime/src/main/cpp/js/README.md` for the rules a lazy builtin
22+
lives by.
23+
24+
## TextEncoder / TextDecoder
25+
26+
Node's split: `js/text-encoding.js` owns the WebIDL surface (brand checks via
27+
private fields, enumerable prototype members, `Symbol.toStringTag`),
28+
`TextEncoding.cpp` owns the bytes.
29+
30+
- **Encodings**: utf-8, utf-16le, utf-16be and windows-1252, each with its
31+
complete WHATWG label set; an unknown label throws `RangeError`. (Precedent:
32+
Node without ICU ships utf-8/utf-16le; utf-16be and windows-1252 are cheap,
33+
and windows-1252 covers the `ascii`/`latin1`/`iso-8859-1` aliases web code
34+
actually uses.)
35+
- **Streaming**: full `decode(…, { stream: true })` support. Incomplete
36+
sequences (split BOMs and split utf-16 code units included) carry across
37+
calls in a 16-byte `Uint8Array` the builtin owns — no per-instance native
38+
handle, no finalizer.
39+
- **Replacement semantics**: WHATWG utf-8 state machine with one U+FFFD per
40+
maximal invalid subpart; `fatal: true` throws `TypeError`; `ignoreBOM`
41+
honored.
42+
- `encode()` / `encodeInto()` with correct USV conversion and partial-write
43+
boundaries (`encodeInto` never splits an encoded code point).
44+
- **Fast paths**: pure-ASCII utf-8 and C1-free windows-1252 decode straight
45+
through `String::NewFromOneByte`; results downgrade to one-byte strings when
46+
possible. `encodeInto` registers a V8 Fast API overload
47+
(`NATIVESCRIPT_ENABLE_FAST_API`, default on), live once a call site tiers
48+
up.
49+
50+
## atob / btoa
51+
52+
WHATWG forgiving-base64 (`Base64.cpp`): whitespace stripping, padding rules,
53+
alphabet validation. With no `DOMException` in the runtime yet, failures throw
54+
the name-patched `Error` (`InvalidCharacterError`) stand-in the abort-signal
55+
and performance builtins already use; a follow-up will introduce
56+
`DOMException` and upgrade these.
57+
58+
## Tests
59+
60+
The shared suite (`test-app/app/src/main/assets/app/shared/TextEncoding`)
61+
holds the conformance specs, feature-detecting so runtimes without these
62+
globals report pending rather than failing; it was independently validated
63+
against Node 24 (full ICU) as a reference.

eslint.config.mjs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ const capturedStatics = [
3232

3333
// Captured constructors. A destructure from `primordials` shadows the global,
3434
// so these only fire on the unguarded reference.
35-
const restrictedGlobals = ['Date', 'FinalizationRegistry', 'Map', 'Number', 'Proxy', 'RangeError', 'Set', 'String', 'TypeError', 'WeakRef'].map((name) => ({
35+
const restrictedGlobals = ['Date', 'FinalizationRegistry', 'Map', 'Number', 'Proxy', 'RangeError', 'Set', 'String', 'TypeError', 'Uint8Array', 'Uint32Array', 'WeakRef'].map((name) => ({
3636
name,
3737
message: `Destructure ${name} from primordials — builtins must not read intrinsics off globals user code can replace.`,
3838
}));

test-app/app/src/main/assets/app/mainpage.js

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ shared.runRuntimeTests();
2020
shared.runWorkerTests();
2121
shared.runPerformanceTests();
2222
shared.runStructuredCloneTests();
23+
shared.runTextEncodingTests();
2324
require("./tests/testWebAssembly");
2425
require("./tests/testEventLoop");
2526
require("./tests/testMultithreadedJavascript");

test-app/runtime/CMakeLists.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ include_directories(
6969
set(RUNTIME_BUILTIN_JS_DIR ${PROJECT_SOURCE_DIR}/src/main/cpp/js)
7070
set(RUNTIME_BUILTIN_JS
7171
${RUNTIME_BUILTIN_JS_DIR}/abort-signal.js
72+
${RUNTIME_BUILTIN_JS_DIR}/base64.js
7273
${RUNTIME_BUILTIN_JS_DIR}/blob-url.js
7374
${RUNTIME_BUILTIN_JS_DIR}/error-events.js
7475
${RUNTIME_BUILTIN_JS_DIR}/events.js
@@ -84,6 +85,7 @@ set(RUNTIME_BUILTIN_JS
8485
${RUNTIME_BUILTIN_JS_DIR}/primordials.js
8586
${RUNTIME_BUILTIN_JS_DIR}/require-factory.js
8687
${RUNTIME_BUILTIN_JS_DIR}/structured-clone.js
88+
${RUNTIME_BUILTIN_JS_DIR}/text-encoding.js
8789
${RUNTIME_BUILTIN_JS_DIR}/weak-ref.js
8890
)
8991
set(RUNTIME_BUILTINS_GENERATED_DIR ${PROJECT_SOURCE_DIR}/src/main/cpp/generated)
@@ -168,6 +170,7 @@ add_library(
168170
src/main/cpp/ArrayElementAccessor.cpp
169171
src/main/cpp/ArrayHelper.cpp
170172
src/main/cpp/AssetExtractor.cpp
173+
src/main/cpp/Base64.cpp
171174
src/main/cpp/BuiltinLoader.cpp
172175
src/main/cpp/CallbackHandlers.cpp
173176
src/main/cpp/ConcurrentQueue.cpp
@@ -189,6 +192,7 @@ add_library(
189192
src/main/cpp/JsArgConverter.cpp
190193
src/main/cpp/JsArgToArrayConverter.cpp
191194
src/main/cpp/JSONObjectHelper.cpp
195+
src/main/cpp/LazyGlobals.cpp
192196
src/main/cpp/Logger.cpp
193197
src/main/cpp/ManualInstrumentation.cpp
194198
src/main/cpp/MetadataMethodInfo.cpp
@@ -215,6 +219,7 @@ add_library(
215219
src/main/cpp/SimpleProfiler.cpp
216220
src/main/cpp/StructuredClone.cpp
217221
src/main/cpp/StructuredSerialization.cpp
222+
src/main/cpp/TextEncoding.cpp
218223
src/main/cpp/Util.cpp
219224
src/main/cpp/V8GlobalHelpers.cpp
220225
src/main/cpp/V8StringConstants.cpp
Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
#include "Base64.h"
2+
3+
#include <vector>
4+
5+
#include "Util.h"
6+
7+
using namespace v8;
8+
9+
namespace tns {
10+
11+
namespace {
12+
13+
constexpr char kAlphabet[] =
14+
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
15+
16+
// 6-bit value per ASCII byte; 0xFF marks everything outside the alphabet.
17+
constexpr uint8_t kInvalid = 0xFF;
18+
19+
uint8_t SixBits(uint8_t c) {
20+
if (c >= 'A' && c <= 'Z') {
21+
return static_cast<uint8_t>(c - 'A');
22+
}
23+
if (c >= 'a' && c <= 'z') {
24+
return static_cast<uint8_t>(c - 'a' + 26);
25+
}
26+
if (c >= '0' && c <= '9') {
27+
return static_cast<uint8_t>(c - '0' + 52);
28+
}
29+
if (c == '+') {
30+
return 62;
31+
}
32+
if (c == '/') {
33+
return 63;
34+
}
35+
return kInvalid;
36+
}
37+
38+
bool IsAsciiWhitespace(uint8_t c) {
39+
return c == '\t' || c == '\n' || c == '\f' || c == '\r' || c == ' ';
40+
}
41+
42+
// The string's code units as bytes. Fails when any unit is above U+00FF,
43+
// which neither op can represent.
44+
bool GetLatin1Bytes(Isolate* isolate, Local<Value> value,
45+
std::vector<uint8_t>* out) {
46+
if (!value->IsString()) {
47+
return false;
48+
}
49+
Local<v8::String> str = value.As<v8::String>();
50+
if (!str->ContainsOnlyOneByte()) {
51+
return false;
52+
}
53+
const int length = str->Length();
54+
out->resize(static_cast<size_t>(length));
55+
if (length > 0) {
56+
str->WriteOneByteV2(isolate, 0, static_cast<uint32_t>(length), out->data());
57+
}
58+
return true;
59+
}
60+
61+
// btoa: base64-encode the input's code units.
62+
void BtoaCallback(const FunctionCallbackInfo<Value>& info) {
63+
Isolate* isolate = info.GetIsolate();
64+
std::vector<uint8_t> input;
65+
if (!GetLatin1Bytes(isolate, info[0], &input)) {
66+
info.GetReturnValue().SetNull();
67+
return;
68+
}
69+
70+
std::vector<uint8_t> out;
71+
out.reserve((input.size() + 2) / 3 * 4);
72+
size_t i = 0;
73+
for (; i + 3 <= input.size(); i += 3) {
74+
const uint32_t group = (static_cast<uint32_t>(input[i]) << 16) |
75+
(static_cast<uint32_t>(input[i + 1]) << 8) |
76+
input[i + 2];
77+
out.push_back(kAlphabet[(group >> 18) & 0x3F]);
78+
out.push_back(kAlphabet[(group >> 12) & 0x3F]);
79+
out.push_back(kAlphabet[(group >> 6) & 0x3F]);
80+
out.push_back(kAlphabet[group & 0x3F]);
81+
}
82+
const size_t remaining = input.size() - i;
83+
if (remaining == 1) {
84+
const uint32_t group = static_cast<uint32_t>(input[i]) << 16;
85+
out.push_back(kAlphabet[(group >> 18) & 0x3F]);
86+
out.push_back(kAlphabet[(group >> 12) & 0x3F]);
87+
out.push_back('=');
88+
out.push_back('=');
89+
} else if (remaining == 2) {
90+
const uint32_t group = (static_cast<uint32_t>(input[i]) << 16) |
91+
(static_cast<uint32_t>(input[i + 1]) << 8);
92+
out.push_back(kAlphabet[(group >> 18) & 0x3F]);
93+
out.push_back(kAlphabet[(group >> 12) & 0x3F]);
94+
out.push_back(kAlphabet[(group >> 6) & 0x3F]);
95+
out.push_back('=');
96+
}
97+
98+
if (out.empty()) {
99+
info.GetReturnValue().Set(v8::String::Empty(isolate));
100+
return;
101+
}
102+
Local<v8::String> result;
103+
if (v8::String::NewFromOneByte(isolate, out.data(), NewStringType::kNormal,
104+
static_cast<int>(out.size()))
105+
.ToLocal(&result)) {
106+
info.GetReturnValue().Set(result);
107+
}
108+
}
109+
110+
// atob: forgiving-base64 decode
111+
// (https://infra.spec.whatwg.org/#forgiving-base64-decode).
112+
void AtobCallback(const FunctionCallbackInfo<Value>& info) {
113+
Isolate* isolate = info.GetIsolate();
114+
std::vector<uint8_t> raw;
115+
if (!GetLatin1Bytes(isolate, info[0], &raw)) {
116+
info.GetReturnValue().SetNull();
117+
return;
118+
}
119+
120+
std::vector<uint8_t> data;
121+
data.reserve(raw.size());
122+
for (uint8_t c : raw) {
123+
if (!IsAsciiWhitespace(c)) {
124+
data.push_back(c);
125+
}
126+
}
127+
128+
if (data.size() % 4 == 0) {
129+
size_t strip = 0;
130+
while (strip < 2 && !data.empty() && data.back() == '=') {
131+
data.pop_back();
132+
strip++;
133+
}
134+
}
135+
if (data.size() % 4 == 1) {
136+
info.GetReturnValue().SetNull();
137+
return;
138+
}
139+
140+
std::vector<uint8_t> out;
141+
out.reserve(data.size() / 4 * 3 + 2);
142+
uint32_t accumulator = 0;
143+
uint32_t bits = 0;
144+
for (uint8_t c : data) {
145+
const uint8_t value = SixBits(c);
146+
if (value == kInvalid) {
147+
info.GetReturnValue().SetNull();
148+
return;
149+
}
150+
accumulator = (accumulator << 6) | value;
151+
bits += 6;
152+
if (bits >= 8) {
153+
bits -= 8;
154+
out.push_back(static_cast<uint8_t>((accumulator >> bits) & 0xFF));
155+
}
156+
}
157+
158+
if (out.empty()) {
159+
info.GetReturnValue().Set(v8::String::Empty(isolate));
160+
return;
161+
}
162+
Local<v8::String> result;
163+
if (v8::String::NewFromOneByte(isolate, out.data(), NewStringType::kNormal,
164+
static_cast<int>(out.size()))
165+
.ToLocal(&result)) {
166+
info.GetReturnValue().Set(result);
167+
}
168+
}
169+
170+
} // namespace
171+
172+
Local<Object> Base64::CreateBinding(Local<Context> context) {
173+
Isolate* isolate = v8::Isolate::GetCurrent();
174+
Local<Object> binding = Object::New(isolate);
175+
tns::SetMethodNoSideEffect(context, binding, "btoa", BtoaCallback);
176+
tns::SetMethodNoSideEffect(context, binding, "atob", AtobCallback);
177+
return binding;
178+
}
179+
180+
} // namespace tns
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
#ifndef BASE64_H_
2+
#define BASE64_H_
3+
4+
#include "v8.h"
5+
6+
namespace tns {
7+
8+
/*
9+
* Native ops behind the base64 builtin (internal/base64.js): the WHATWG
10+
* forgiving-base64 codec backing the atob / btoa globals. Both ops answer
11+
* null instead of throwing, so the builtin owns the error shape.
12+
*/
13+
class Base64 {
14+
public:
15+
static v8::Local<v8::Object> CreateBinding(v8::Local<v8::Context> context);
16+
};
17+
18+
} // namespace tns
19+
20+
#endif /* BASE64_H_ */

0 commit comments

Comments
 (0)