Step arguments fail to decrypt when a run spans two deployments: step handler resolves encryption key from runId only, and the error is silently swallowed by the queue retry callback

[Avishka0303]

Summary

When a workflow run is started by one deployment and its steps execute on another (common on preview branches,
where superseded deployments are deactivated and queue callbacks re-route to the branch's latest deployment),
every step attempt fails before user code runs with:

RuntimeDecryptionError: Encrypted data encountered but no encryption key is available.
Encryption is not configured or no key was provided for this run.

The run then fails as FatalError: Step "..." exceeded max retries (N retries) with a blank underlying stack,
because the per-attempt error is dropped (see "Why it's invisible" below). User-level try/catch inside the step
never executes.

Root cause

Two asymmetric key resolutions in @workflow/core 4.6.0:

• Workflow handler (dist/runtime.js, replay path) calls world.getEncryptionKeyForRun(workflowRun) with the run
  object. In @workflow/world-vercel (dist/encryption.js), run.deploymentId differs from VERCEL_DEPLOYMENT_ID, so
  it fetches the origin deployment's key from the Vercel API. Replay works and step inputs get
  dehydrated/encrypted with the origin deployment's run key.
• Step handler (dist/runtime/step-handler.js:356) calls world.getEncryptionKeyForRun?.(workflowRunId) with only
  the string. With no deploymentId context, the Vercel world takes the local-derivation branch
  (isServerlessRuntime && !deploymentId) and derives from the current deployment's VERCEL_DEPLOYMENT_KEY — the
  wrong key material for a cross-deployment run (or none). Hydrating the encrypted step input then throws
  RuntimeDecryptionError before stepFn.apply().

Still present in 5.0.0-beta.27: memoizeEncryptionKey(world, workflowRunId) is called with the bare string in
the beta step handler.

Why it's invisible

The pre-user-code error propagates to the queue handler, whose retry directive receives it and discards it
(@workflow/world-vercel/dist/queue.js):

retry: (_error, { deliveryCount }) => ({
afterSeconds: getHandlerErrorRetryAfterSeconds(deliveryCount),
}),

Nothing logs _error. The runtime's info/debug logs are also console-silent unless DEBUG=workflow:* is set, so
the operator sees only the final exceeded max retries with stack: step.error?.stack = undefined (no attempt
ever recorded a step error). Diagnosing this required patching the retry callback to print the swallowed error.

Reproduction

1. Next.js app on Vercel, workflow 4.6.0, on a git branch with a custom environment/domain (probably plain
   preview too).
2. Load the app, then push a new deployment of the branch.
3. Without reloading, trigger a workflow (skew protection routes the POST to the old deployment; queue
   callbacks execute on the new one).
4. First step of the run exhausts retries with the blank exceeded max retries; per-attempt
   RuntimeDecryptionError only visible with the logging patch.

Suggested fixes

1. In the step handler, resolve the key deployment-aware, e.g. fetch the run (world.runs.get(runId, {
   resolveData: 'none' })) and pass it to getEncryptionKeyForRun, matching the workflow handler. We've verified
   this patch fixes the failure in our app.
2. Log the error in the queue retry callback (or record it as a step attempt error) so retry-exhaustion is
   diagnosable.

Environment

• workflow 4.6.0 / @workflow/core 4.6.0 / @workflow/world-vercel 4.5.0 / @workflow/ai 4.1.2
• Next.js 16.0.10 (Turbopack), Node 24, Vercel Fluid Compute, iad1
• Run IDs available privately on request.

[VaguelySerious]

@Avishka0303 Aside from the encryption issue, a run continuing on another deployment is usually not safe due to potential code skew.

We should potentially fail the run with a clearer message instead, e.g. Run got scheduled on wrong deployment, exiting to protect against code skew issues. Is the original deployment still available?

[TooTallNate]

@Avishka0303 It should not be possible for a Vercel Queue message to be delivered to a different deployment. Would you be able to provide run IDs (you can email to nate@vercel.com)? Also just a quick sanity check - do you have the "Enable access to System Environment Variables" setting enabled for your project?