Run loops over items, with bounded work per event, recovery and replay

[ancongui]

Summary

Loops run. A Loop over items (forEach) step runs its body once per element, concurrency at a time, inside the same run, and its output lists one result per element in element order, whatever order the iterations finish in.

• Iterations. Iteration i runs in its own frame with /item, /index and /loops/<id>/…; its steps, tasks, human tasks, waits and incidents use instance keys such as send[2], so iterations never share state. Finished iterations are removed from the run state together with every block their body created; sibling loops keep theirs.
• Failures. An items value that is not a list, more elements than maxItems, or collected results over 1 MiB open WV-RUNTIME-LOOP-ITEMS, WV-RUNTIME-LOOP-LIMIT or WV-RUNTIME-LOOP-OUTPUT and roll back the event. A task incident in one iteration suspends the run; the other iterations' results wait and are applied after the decision; retry_safe retries only that iteration under the same idempotency key. A fail step fails the run and cancels the other iterations; canceling or timing out a run ends its loops, including runs too large to load.
• Large loops. One event starts at most 4,000 units of loop work (one per iteration plus one per body node) and reserves at most 500 of its 1,000 tasks and deadlines, nested loops included; a loop that must stop yields under <loop>~<n>, and the deadline scanner continues it about one poll later. A 10,000-element loop whose body only returns its item finishes in five events. Capabilities report the budget as operations.runtime.loop_units_per_turn.
• Linear cost. Expressions in a loop body and the capacity check of a long event no longer re-walk data that has not changed, with the same accept and refuse decisions (property tests compare both ways).
• Platform. The language manifest and capabilities list flow.forEach and forEach; publishing, importing, activating (activation readiness now derives its node kinds from the IR and the listed features) and debug sessions accept loops. Platforms without this change still refuse them with WV-IR-UNSUPPORTED.
• History and metrics. Loop commands carry the element position; yields are wait_elapsed events; exported history of loop runs replays as consistent. New counters weave.loop_iterations (status: ok per collected result, cancelled per iteration a run's end stopped) and weave.loop_yields.
• Golden corpus. A new corpus of loop transitions (tests/fixtures/golden/loops.json); the existing corpus is unchanged byte for byte.

Verification

• Local, offline: make check (every stage: source, docs, docs site, lint, format, types, 6,752 unit and contract tests, both workers, packaging and the installed wheel), the golden corpus check (existing corpus unchanged), the loop golden corpus check, the language fixture check, and Studio's type check, format check, 1,964 vitest tests and production build. Studio only gains the generated language manifest fixture; no screens changed, so there is no browser tour.
• Local, PostgreSQL 17 (throwaway container): the loop integration set (worker tasks at concurrency 3, human tasks, waits, a 10,000-element yielding loop, sibling frame cleanup, incident retry and replay, fail, cancellation on the normal and the oversized path, restart, counters) and the full integration suite with -n 4 --dist loadfile. Every module that runs without external services passes; the timing-sensitive capacity tests that failed under parallel load pass when rerun alone, on this branch and on main. Kafka, Keycloak, provider-inbox and connector-database modules need services the local run does not start.
• Benchmark: tests/benchmarks/test_loop_scaling.py (processor time, fastest of three, 1,000 vs 2,000 elements) was measured on a heavily loaded host, where ratios scattered between 1.3 and 4.0 on this branch and on the commit that introduced the linear path alike; 2,000 elements took 8–14 s against 31 s before the change. The 2.2× bound still needs a rerun on a quiet machine. The benchmark is not part of CI.
• CI only: Linux offline, studio, build and Acceptance, and cross-platform on macOS 15 and Windows 2022 (golden corpora, capability contract, compiler and debug-session tests). Kernel unit tests run on Linux in CI and on macOS locally. CI runs no PostgreSQL job.

Upgrade

Upgrade the CLI and SDK together with the server: alpha15 and earlier clients reject the new operations.runtime.loop_units_per_turn capability field (WV-WIRE on weave remote capabilities and the SDK's capabilities()), and human tasks inside loops now carry instance_key and iteration. Before rolling back to alpha15, let runs that use loops finish, or cancel them. No migration.

Decisions

• Publishing loops is enabled on merge. The platform lists flow.forEach as soon as its runtime ships, so publishing, importing, activating and debugging workflows with loops work immediately. Studio's loop editor is not there yet; loops are authored in source or the CLI until it ships.
• Intent reservation. A started iteration reserves the tasks, deadlines and signals its body can issue in one event (human tasks count two, nested loops one for their yield) instead of counting what it has issued, because iterations start in a batch before their bodies run. A wide loop of tasks starts up to 500 iterations per event and continues after a yield. The first iteration of a loop in an event always starts, and a loop with a pending yield starts no others until it wakes up.
• Linear-cost shortcuts. Inside loop bodies, expression contexts are checked from cached part sizes, and the capacity check of a long event re-measures only containers that grew (the event's steps and commands, the run input, each loop's items and outputs, and large JSON values). Accept and refuse decisions are the same as a full walk; property tests compare both ways. Contexts outside loops are measured as before, so existing workflows produce byte-identical transitions.
• Metric label status. weave.loop_iterations keeps the three bounded metric attributes: operation: loop, status: ok per collected result, and the new status cancelled per iteration still running when its run ends (the one holding a failing step included; the failure is counted once, as the run's outcome).
• Yields are not steps. A yield emits no step change and no step result row; its deadline and wait_elapsed event are its history.

Follow-ups

• Guard validation builds a JSON Schema validator per check (about 2 ms each); caching validators per schema would remove the largest constant left in loop cost (about 5.7 s of CPU per 1,000 pure iterations with three guarded steps).
• Nested loop instances and parallel branches inside an iteration copy the values visible at their outer loop; with a large earlier step and a wide outer loop the 32 MiB state limit is reached sooner than the element count suggests. Chaining the snapshot to the enclosing loop would remove the copy.
• Run facts: when this merges with the run facts projection, loop yield keys (each~1) must stay out of step facts.
• Loop simulation in debug sessions (loop views, instance breakpoints, per-instance mocks) and Studio's loop editor.
• Rerun the scaling benchmark on a quiet machine.