Skip to content

feat(core): deterministic sandbox hardening#3045

Open
NathanColosimo wants to merge 4 commits into
mainfrom
nathanc/vm-determinism-hardening
Open

feat(core): deterministic sandbox hardening#3045
NathanColosimo wants to merge 4 commits into
mainfrom
nathanc/vm-determinism-hardening

Conversation

@NathanColosimo

@NathanColosimo NathanColosimo commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

First of a 3-PR stack extracted from #2990 (retained-VM replay) for easier review. This PR contains only the sandbox determinism changes — every piece is a standalone win, independent of retention:

  • crypto.subtle.digest computes synchronously via node:crypto — byte-identical to WebCrypto (verified for SHA-1/256/384/512; createHash is stable and undeprecated through Node 26), so the promise settles at a deterministic point during replay instead of host threadpool timing. Full BufferSource contract: TypeError on non-buffer input, internal-slot view metadata reads, SharedArrayBuffer-backed views rejected.
  • Removed from the sandbox: Atomics.waitAsync (a wall-clock timer via SharedArrayBuffer), async WebAssembly compilation (compile/instantiate/…Streaming; the sync new WebAssembly.Module()/Instance() constructors remain), WeakRef, and FinalizationRegistry. Wall-clock timing and GC observation are unreplayable with or without retention.

No prototypes are frozen and no global bindings are pinned — workflow code can patch built-ins freely (Temporal / core-js polyfills keep working). Serialization safety is instead handled in part 3 of the stack, where serialization reads values through captured host intrinsics and reports when it had to execute workflow code.

Review focus

The user-facing semantic changes of the whole stack are isolated here: the compat question is "does deleting X break any real workflow bundle?"

Validation

  • pnpm --filter @workflow/core typecheck + build
  • full core unit suite green: 1,525 passed, 3 expected failures
  • sandbox tests cover: digest value parity vs WebCrypto + input validation, removed-API absence

Stack: this PR → #3046 (feature, primitives-gated) → #3047 (full argument-type support via taint-reporting hardened serialization). #2990 stays open as the reference implementation.

@NathanColosimo
NathanColosimo requested review from a team and ijjk as code owners July 22, 2026 02:32
@changeset-bot

changeset-bot Bot commented Jul 22, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 8e23cfa

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
Name Type
@workflow/core Patch
workflow Patch
@workflow/builders Patch
@workflow/cli Patch
@workflow/next Patch
@workflow/nitro Patch
@workflow/vitest Patch
@workflow/web-shared Patch
@workflow/web Patch
@workflow/world-testing Patch
@workflow/astro Patch
@workflow/nest Patch
@workflow/nuxt Patch
@workflow/rollup Patch
@workflow/sveltekit Patch
@workflow/vite Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
example-nextjs-workflow-turbopack Ready Ready Preview, Comment Jul 23, 2026 11:32pm
example-nextjs-workflow-webpack Ready Ready Preview, Comment Jul 23, 2026 11:32pm
example-workflow Ready Ready Preview, Comment Jul 23, 2026 11:32pm
workbench-astro-workflow Ready Ready Preview, Comment Jul 23, 2026 11:32pm
workbench-express-workflow Ready Ready Preview, Comment Jul 23, 2026 11:32pm
workbench-fastify-workflow Ready Ready Preview, Comment Jul 23, 2026 11:32pm
workbench-hono-workflow Ready Ready Preview, Comment Jul 23, 2026 11:32pm
workbench-nestjs-workflow Ready Ready Preview, Comment Jul 23, 2026 11:32pm
workbench-nitro-workflow Ready Ready Preview, Comment Jul 23, 2026 11:32pm
workbench-nuxt-workflow Ready Ready Preview, Comment Jul 23, 2026 11:32pm
workbench-sveltekit-workflow Ready Ready Preview, Comment Jul 23, 2026 11:32pm
workbench-tanstack-start-workflow Ready Ready Preview, Comment Jul 23, 2026 11:32pm
workbench-vite-workflow Ready Ready Preview, Comment Jul 23, 2026 11:32pm
workflow-docs Ready Ready Preview, Comment, Open in v0 Jul 23, 2026 11:32pm
workflow-swc-playground Ready Ready Preview, Comment Jul 23, 2026 11:32pm
workflow-tarballs Ready Ready Preview, Comment Jul 23, 2026 11:32pm
workflow-web Ready Ready Preview, Comment Jul 23, 2026 11:32pm

@NathanColosimo
NathanColosimo force-pushed the nathanc/vm-determinism-hardening branch from 02636d5 to cb05683 Compare July 22, 2026 02:32
@github-actions

github-actions Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

Passed Failed Skipped Total
✅ ▲ Vercel Production 1455 0 239 1694
✅ 💻 Local Development 1621 0 227 1848
✅ 📦 Local Production 1621 0 227 1848
✅ 🐘 Local Postgres 1621 0 227 1848
✅ 🪟 Windows 154 0 0 154
❌ 📋 Other 1019 1 212 1232
✅ vercel-multi-region 27 0 0 27
Total 7518 1 1132 8651

❌ Failed Tests

📋 Other (1 failed)

e2e-vercel-prod-nest (1 failed):

  • writableForwardedFromStepWorkflow | wrun_41KY8N8FA10GSBNVW003Q7W0VM

Details by Category

✅ ▲ Vercel Production
App Passed Failed Skipped
✅ astro 126 0 28
✅ example 126 0 28
✅ express 126 0 28
✅ fastify 126 0 28
✅ hono 126 0 28
✅ nextjs-turbopack 151 0 3
✅ nextjs-webpack 151 0 3
✅ nitro 126 0 28
✅ nuxt 126 0 28
✅ sveltekit 145 0 9
✅ vite 126 0 28
✅ 💻 Local Development
App Passed Failed Skipped
✅ astro-stable 128 0 26
✅ express-stable 128 0 26
✅ fastify-stable 128 0 26
✅ hono-stable 128 0 26
✅ nextjs-turbopack-canary 135 0 19
✅ nextjs-turbopack-stable 154 0 0
✅ nextjs-webpack-canary 135 0 19
✅ nextjs-webpack-stable 154 0 0
✅ nitro-stable 128 0 26
✅ nuxt-stable 128 0 26
✅ sveltekit-stable 147 0 7
✅ vite-stable 128 0 26
✅ 📦 Local Production
App Passed Failed Skipped
✅ astro-stable 128 0 26
✅ express-stable 128 0 26
✅ fastify-stable 128 0 26
✅ hono-stable 128 0 26
✅ nextjs-turbopack-canary 135 0 19
✅ nextjs-turbopack-stable 154 0 0
✅ nextjs-webpack-canary 135 0 19
✅ nextjs-webpack-stable 154 0 0
✅ nitro-stable 128 0 26
✅ nuxt-stable 128 0 26
✅ sveltekit-stable 147 0 7
✅ vite-stable 128 0 26
✅ 🐘 Local Postgres
App Passed Failed Skipped
✅ astro-stable 128 0 26
✅ express-stable 128 0 26
✅ fastify-stable 128 0 26
✅ hono-stable 128 0 26
✅ nextjs-turbopack-canary 135 0 19
✅ nextjs-turbopack-stable 154 0 0
✅ nextjs-webpack-canary 135 0 19
✅ nextjs-webpack-stable 154 0 0
✅ nitro-stable 128 0 26
✅ nuxt-stable 128 0 26
✅ sveltekit-stable 147 0 7
✅ vite-stable 128 0 26
✅ 🪟 Windows
App Passed Failed Skipped
✅ nextjs-turbopack 154 0 0
❌ 📋 Other
App Passed Failed Skipped
✅ e2e-local-dev-nest-stable 128 0 26
✅ e2e-local-dev-tanstack-start- 128 0 26
✅ e2e-local-postgres-nest-stable 128 0 26
✅ e2e-local-postgres-tanstack-start- 128 0 26
✅ e2e-local-prod-nest-stable 128 0 26
✅ e2e-local-prod-tanstack-start- 128 0 26
❌ e2e-vercel-prod-nest 125 1 28
✅ e2e-vercel-prod-tanstack-start 126 0 28
✅ vercel-multi-region
App Passed Failed Skipped
✅ nextjs-turbopack 27 0 0

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: success
  • Windows: success

Check the workflow run for details.

@github-actions

github-actions Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 8e23cfa · Thu, 23 Jul 2026 23:42:57 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 1230 (+252%) 🔻 1281 🔴 (+26%) 🔻 1333 🔴 (+29%) 🔻 1598 🔴 (+15%) 🔻 30
TTFS stream 1225 (+34%) 🔻 1277 🔴 (+22%) 🔻 1302 🔴 (+23%) 🔻 3379 🔴 (+134%) 🔻 30
TTFS hook + stream 505 (-55%) 💚 1556 🔴 (+16%) 🔻 1567 🔴 (+11%) 1704 🔴 (+12%) 30
STSO 1020 steps (1-20) 153 (-13%) 293 🔴 (+5.8%) 306 🔴 (-32%) 💚 543 🔴 (+21%) 🔻 19
STSO 1020 steps (101-120) 174 (-7.4%) 222 🔴 (-20%) 💚 237 🔴 (-27%) 💚 277 🔴 (-17%) 💚 19
STSO 1020 steps (1001-1020) 474 (+4.6%) 604 🔴 (+6.3%) 706 🔴 (+2.0%) 3411 🔴 (+309%) 🔻 19
WO 1020 steps 374335 (-4.7%) 374335 (-4.7%) 374335 (-4.7%) 374335 (-4.7%) 1
SL stream latency 79 (-1.3%) 122 🔴 (-3.2%) 129 🔴 (-12%) 142 🔴 (-66%) 💚 30
📜 Previous results (3)

9395b2d

Thu, 23 Jul 2026 23:12:23 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 221 (-37%) 💚 1277 🔴 (+25%) 🔻 1295 🔴 (+25%) 🔻 1366 🔴 (-1.7%) 30
TTFS stream 236 (-74%) 💚 1323 🔴 (+27%) 🔻 1330 🔴 (+25%) 🔻 1478 🔴 (+2.6%) 30
TTFS hook + stream 415 (-63%) 💚 1535 🔴 (+15%) 1574 🔴 (+11%) 1627 🔴 (+7.3%) 30
STSO 1020 steps (1-20) 169 (-4.0%) 280 🔴 (+1.1%) 340 🔴 (-24%) 💚 356 🔴 (-21%) 💚 19
STSO 1020 steps (101-120) 192 (+2.1%) 270 🔴 (-2.2%) 321 🔴 (-0.9%) 1123 🔴 (+236%) 🔻 19
STSO 1020 steps (1001-1020) 480 (+6.0%) 540 🔴 (-4.9%) 563 🔴 (-19%) 💚 616 🔴 (-26%) 💚 19
WO 1020 steps 403173 (+2.6%) 403173 (+2.6%) 403173 (+2.6%) 403173 (+2.6%) 1
SL stream latency 88 (+10%) 156 🔴 (+24%) 🔻 179 🔴 (+23%) 🔻 215 🔴 (-49%) 💚 30

b908241

Wed, 22 Jul 2026 17:59:42 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 167 (+4.4%) 331 🔴 (-18%) 💚 354 🔴 (-24%) 💚 555 (+5.9%) 30
TTFS stream 151 (-9.0%) 336 🔴 (+28%) 🔻 371 🔴 (+24%) 🔻 431 (-14%) 30
TTFS hook + stream 349 (+4.5%) 499 🔴 (-9.4%) 588 🔴 (-6.1%) 938 🔴 (+15%) 30
STSO 1020 steps (1-20) 212 (+10%) 353 🔴 (+17%) 🔻 377 🔴 (+7.7%) 561 🔴 (-19%) 💚 19
STSO 1020 steps (101-120) 250 (+1.6%) 392 🔴 (+3.7%) 425 🔴 (-16%) 💚 694 🔴 (+9.5%) 19
STSO 1020 steps (1001-1020) 688 (±0%) 977 🔴 (+11%) 1202 🔴 (+17%) 🔻 6811 🔴 (+561%) 🔻 19
WO 1020 steps 627015 (-6.1%) 627015 (-6.1%) 627015 (-6.1%) 627015 (-6.1%) 1
SL stream latency 96 (+16%) 🔻 145 🔴 (+21%) 🔻 200 🔴 (+9.9%) 436 🔴 (+27%) 🔻 30

cb05683

Wed, 22 Jul 2026 02:47:52 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 148 (+21%) 🔻 222 🔴 (-7.5%) 382 🔴 (+18%) 🔻 2459 🔴 (+471%) 🔻 30
TTFS stream 128 (-17%) 💚 210 🔴 (-38%) 💚 277 (-25%) 💚 582 (+26%) 🔻 30
TTFS hook + stream 332 (+7.8%) 518 🔴 (+12%) 539 🔴 (+6.1%) 672 🔴 (+8.2%) 30
STSO 1020 steps (1-20) 208 (+9.5%) 376 🔴 (+28%) 🔻 432 🔴 (+28%) 🔻 516 🔴 (+48%) 🔻 19
STSO 1020 steps (101-120) 256 (+15%) 353 🔴 (+10.0%) 431 🔴 (-11%) 474 🔴 (-5.4%) 19
STSO 1020 steps (1001-1020) 634 (+4.8%) 741 🔴 (-5.2%) 806 🔴 (-7.4%) 833 🔴 (-25%) 💚 19
WO 1020 steps 479397 (-9.3%) 479397 (-9.3%) 479397 (-9.3%) 479397 (-9.3%) 1
SL stream latency 107 (+19%) 🔻 151 🔴 (+0.7%) 184 🔴 (+1.7%) 336 🔴 (+47%) 🔻 30

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt)

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120

All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.

Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.

@vercel
vercel Bot temporarily deployed to Preview – workflow-docs July 22, 2026 02:32 Inactive
@NathanColosimo
NathanColosimo marked this pull request as draft July 22, 2026 05:14
- crypto.subtle.digest computes synchronously via node:crypto:
  byte-identical values, promise settles on a deterministic microtask,
  full BufferSource validation (internal-slot view reads, SAB rejection)
- Atomics.waitAsync (a wall-clock timer), async WebAssembly compilation,
  WeakRef, and FinalizationRegistry are removed from the sandbox — wall
  clock and GC observation are unreplayable; sync WebAssembly
  constructors remain
- freezeSerializationIntrinsics pins the universal dispatch surfaces:
  Object.prototype/Array.prototype/Function.prototype are frozen (every
  missed property read and hasInstance lookup terminates there) and
  serialization-referenced global bindings are non-writable. Value-type
  prototypes and constructor statics stay patchable so polyfills
  (Temporal's Date.prototype.toTemporalInstant, core-js Set.prototype
  .union / Object.groupBy) keep working — the retained-input gate
  verifies the members serialization executes per boundary instead.

Groundwork for retained-VM replay (#2990).
@NathanColosimo
NathanColosimo force-pushed the nathanc/vm-determinism-hardening branch from cb05683 to b908241 Compare July 22, 2026 17:41
…escence

The remaining async subtle methods reject immediately through the crypto
proxy (brand check — the receiver is not a real SubtleCrypto), so they can
never mint a host-timing promise. Narrow the quiescence comment to what the
code actually enforces and add a test so the unreachability is not silently
"fixed" later.

@TooTallNate TooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with the Slack thread context in mind (including the rescope that dropped prototype freezing / global pinning in favor of part-3 hardened serialization — which I think was the right call, and matches the compatibility concerns raised in the thread).

What I verified

  • Sync digest correctness: the captured-intrinsic view-metadata reads (buffer/byteOffset/byteLength via prototype getters + util.types brand checks) are exactly the right pattern — cross-realm safe, immune to shadowed own properties, and consistent with the hardened-serialization direction for part 3. Copying out of the pooled Buffer allocation is a nice catch. Value parity + BufferSource contract (TypeError on non-buffers, SAB rejection, subviews) are all test-covered.
  • The quiescence argument: I walked the sandbox surface looking for other host-timed promise sources — remaining crypto.subtle methods (brand-check rejection through the proxy receiver is deterministic and test-locked), dynamic import() (microtask rejection), injected web APIs (all sync), sync Atomics.wait (return value is deterministic in a single-agent realm — correctly left in). I found no gaps.
  • No internal usage: nothing in packages/core (VM-side runtime) or the workbench fixtures uses WeakRef/FinalizationRegistry/Atomics.waitAsync/async WASM.

Blocking

One item, inline: the changeset bump type should be major, not patch (details on the comment). Trivial fix.

Non-blocking notes

  • The E2E failure (writableForwardedFromStepWorkflow on nest) is an unrelated streams test — needs a rerun before merge but isn't caused by this diff.
  • digest({ name: 123 }) / digest({}) rejects with TypeError (from name.toUpperCase()), where WebCrypto's algorithm normalization does ToString first and rejects with NotSupportedError. Trivial divergence; fine to leave.
  • The /must be of type SubtleCrypto/ assertion matches V8's error wording, which isn't guaranteed stable across Node majors. Consider also accepting a looser match (e.g. asserting rejection with TypeError) so a Node bump doesn't break the suite — the important invariant is "rejects deterministically before scheduling crypto work", not the message.

@@ -0,0 +1,6 @@
---
'@workflow/core': patch
'workflow': patch

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this should be major, not patch.

Removing WeakRef, FinalizationRegistry, Atomics.waitAsync, and async WebAssembly compilation from the sandbox is a breaking change to the public workflow-globals surface (the thread itself frames it as one). On main the bump type doesn't affect beta numbering — but per the repo's versioning policy it does carry through to stable backports: as written, a backport would remove APIs from v4 in a patch release, which violates semver for anyone pinned to ~4.x.

Marking it major both signals the break correctly and forces the backport decision to be deliberate rather than auto-mergeable. (If the intent is that this ships v5-only, major also matches that: the human reviewing the auto-backport PR gets an obvious signal to close it.)

Suggested:

---
'@workflow/core': major
'workflow': major
---

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants