Skip to content

Give each run of a step its own instance key - #39

Merged
ancongui merged 10 commits into
mainfrom
feat/instance-identity
Oct 13, 2026
Merged

ancongui merged 10 commits into
mainfrom
feat/instance-identity

Conversation

@ancongui

Copy link
Copy Markdown
Contributor

Summary

The runtime now names each run of a step by its instance key, so that a step can later run more than once in one run (once per loop iteration, for example) without sharing state with its other runs. Outside loops an instance key is the step ID. Nothing changes for existing workflows: every recorded golden transition replays byte-identically.

  • Run state and history. Run state, step results, tasks, human tasks, deadlines, incidents and control commands carry instance keys. Every lookup into the workflow uses the step ID part of the key.
    • Branch frames are keyed by owner, block and the loop indexes of the steps they hold (checks/credit[3]).
    • Joins are keyed by their owner's instance key.
    • A next step keeps its predecessor's loop indexes.
    • Expressions still read /steps/<step ID>.
  • Proof without loops. Every golden scenario is replayed a second time with each instance key moved into iteration [3][0]. Each transition equals the recorded one with the same keys moved.
  • Deadlines and terminal controls. The deadline scanner and the terminal SQL match a deadline to its step through the step ID part of its key. Settling a deadline reads the step's kind, so a wait inside an iteration elapses instead of ending the run.
  • Operations.
    • Incident resolution finds the step and the task of a step instance.
    • Task output checks, exported history, replay and debug sessions find the step behind any canonical key.
    • A connector's idempotency key is {run_id}:{instance key}.
    • In debug sessions, a step mock and a step breakpoint apply to every instance of the step.
  • Human tasks. Task views add instance_key and iteration for tasks created inside a loop iteration. Both are omitted for every other task, so existing payloads keep their bytes.
  • Capacity checks. The capacity check of one event's work runs at its end and after every 64 steps or 1 MiB of new step output, instead of after every step. Large events no longer pay a full state walk per step. Work over capacity is still refused.

There is no migration, no new operation and no Studio change.

Verification

  • Local, offline (macOS): make check passed all 17 stages, including 6558 unit and contract tests, 208 agentic-worker tests and 29 files-worker tests. scripts/golden_corpus.py --check and scripts/language_fixtures.py --check exit 0.
  • Local, real PostgreSQL 17 (throwaway container): 203 tests passed. They cover:
    • the new instance-key cases: a wait elapsing in an iteration, policy-blocked and oversized signal timeouts, and retrying a failed step instance through incident resolution, with operation key {run}:work[2]
    • the existing incident, secret-admission, terminal-capacity, wait, duration-branch, parallel, history, human-task, run, run-lifecycle, unsupported-run, debug API, lease and restart suites
  • Mocked versus real:
    • Unit and contract tests run the kernel, the simulator and replay in memory.
    • Only the PostgreSQL suites run the repository, the deadline scanner SQL and the terminal SQL against a database.
    • Because no step creates suffixed keys yet, the PostgreSQL tests move stored rows to instance keys with SQL (pause to pause[2], work to work[2], approval to approval[2]). Those rows match what a loop body will store.
  • CI only:
    • Linux: offline, studio, Documentation and Acceptance (pr profile).
    • macOS and Windows: the cross-platform jobs run the golden and lifted corpus, the operations lookups and the contract tests.
    • CI does not run the PostgreSQL suites. The runtime unit tests run on Linux in CI and on macOS locally.

Decisions

  • A capacity overflow followed by a failing step in the same event ends in that step's incident. Capacity is now checked in batches within one event. Suppose an event grows its state past capacity before the next check, and a later step of the same event fails an expression. The run then opens a WV-EXPR-MISSING incident instead of failing with WV-RUNTIME-LIMIT. We accept this because:

    • The outcome is deterministic and stable on replay.
    • The rolled-back incident transition is itself checked, so nothing oversized is persisted or returned.
    • Successful transitions are unchanged.
    • Under the old behavior, raising the limit would only have led to the same incident.
    • Restoring the old order would cost a full capacity walk on every failure path.

    test_an_expression_failure_after_unadmitted_growth_ends_in_an_incident pins this behavior.

  • Signal receipts keep matching by step ID. Signal steps are not allowed inside loops, so a signal's key is always its step ID. Deadlines, which human tasks inside loops will also use, are matched through the step.

  • Incident views are unchanged here. Incident list and read views gain instance_key together with the incident view rework. Until then, an incident of a step instance would show its full key in node_id. No such incident can exist before loops.

  • In debug sessions, a human decision whose name is not a valid instance key (for example [2], #a or ~1) returns 422 WV-DEBUG-HUMAN-TASK, like an unknown or inactive human task.

Follow-ups

  • Loop steps and their runtime build on this. They will add:
    • an end-to-end replay and history-export test of a real loop run (an elapsed wait, a completed task and a resolved incident inside an iteration)
    • frame cleanup between sibling loops
  • Loop simulation will accept instance-key mocks and breakpoints. Its precedence test is node:send[2], then node:send, then action:.
  • Agent turns will add the handler for # keys, which until then refuse with WV-RUNTIME-EVENT.
  • When loops ship, the release notes ask users to upgrade CLIs and SDKs with the server, because older clients reject the new human task fields.

@ancongui
ancongui merged commit 659ff9a into main Oct 13, 2026
13 checks passed
@ancongui
ancongui deleted the feat/instance-identity branch October 13, 2026 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant