Run loops over items, with bounded work per event, recovery and replay

CHANGELOG.md

@@ -20,13 +20,35 @@ SPDX-License-Identifier: Apache-2.0
 
 ## Unreleased
 
+- Loops run. A **Loop over items** (`forEach`) step runs its body once per
+  element, `concurrency` at a time, in one run, and its output lists one result
+  per element in element order, whatever order the iterations finish in. Each
+  iteration's steps, tasks, human tasks, waits, and incidents use their own
+  instance key (`send[2]`); a task incident in one iteration suspends the run, a
+  `fail` step in one fails it, and canceling ends every iteration. Finished
+  iterations are removed from the run state. An `items` value that is not a list,
+  more elements than `maxItems`, or results over the 1 MiB payload limit open
+  `WV-RUNTIME-LOOP-ITEMS`, `WV-RUNTIME-LOOP-LIMIT`, or `WV-RUNTIME-LOOP-OUTPUT`.
+  A long loop yields after 4,000 units of work in one event and continues about
+  one scanner poll later; the capabilities response reports the budget as
+  `operations.runtime.loop_units_per_turn`. The platform now lists
+  `flow.forEach` and `forEach` in its language manifest and capabilities, so it
+  publishes, imports, and activates workflows with loops. Debug sessions run
+  loops with the same reducer. New counters `weave.loop_iterations` (`status:
+  ok` for each collected result, `cancelled` for each iteration still running
+  when its run ends) and `weave.loop_yields`. Upgrade the CLI and SDK together with the server: older
+  clients do not know the human task `instance_key` and `iteration` fields or
+  the new capability fields.
+- Evaluating an expression inside a loop body and admitting a long event no
+  longer walk data that has not changed since the event started, so a loop's
+  cost grows linearly with its number of elements.
 - The compiler checks and compiles **Loop over items** (`forEach`): `/item`,
   `/index`, and `/loops/<id>/…` inside a loop body, a typed list of results after
   it, `WV-COMP-LOOP_*` diagnostics for wrong items, scope, nesting, limits, and
   signals in a body (with a suggested `maxItems` for a loop budget that is too
   high), and an IR `forEach` node with an `each` edge and an `each` join that
-  requires the `flow.forEach` feature. This platform does not run loops yet, so
-  it refuses to publish, import, or activate such a workflow with
+  requires the `flow.forEach` feature. A platform that does not list the
+  feature refuses to publish, import, or activate such a workflow with
   `WV-IR-UNSUPPORTED`; workflows without loops compile to the same bytes as
   before. The compiler limits add `max_loop_items`, `max_loop_depth`, and
   `max_run_iterations`. Studio's local host answers `POST /studio/local/scope`

docs/concepts.md

@@ -306,7 +306,7 @@ Use this table to translate what you know:
 | Error boundary event | Not in the language | A failed Action retries when that is safe; otherwise the run is suspended with an incident for an operator. To branch on a business outcome, return it in the Action's output and test it with a Decision |
 | Message throw or end event | **Call an action**, or an integration event subscription | A subscription notifies another system when a run's status changes |
 | Call activity, subprocess | **Call a workflow** (`callWorkflow`) | Defined in the [workflow language](contracts.md#steps), but this version does not compile or run it: the compiler reports `WV-COMP-UNSUPPORTED_FEATURE`. Until then, keep the part inside the same workflow, or make it a separate workflow that your product starts |
-| Loop, multi-instance activity | **Loop over items** (`forEach`) | Defined in the [workflow language](contracts.md#steps). This version compiles and checks it, but the platform does not run it yet and refuses to publish it (`WV-IR-UNSUPPORTED`). Until then, use a Parallel with a fixed set of branches, or start one run per item |
+| Loop, multi-instance activity | **Loop over items** (`forEach`) | Runs its body once per element of a list, `concurrency` at a time, inside the same run, and collects one result per element in element order; see [loops at run time](reference/compiler.md#loops-at-run-time). A failure in one element stops the whole run, as in a Parallel |
 | Compensation | Not in the language | Model compensating Actions as ordinary steps; Weave never undoes an external effect |
 | Instance migration | Not available | A running run keeps its version; new runs use the new activation |
 | Process monitoring, cockpit | **Runs**, history, and incidents in Studio and the CLI; logs, metrics, and traces | Studio draws a run's graph and marks its current step **Now**; see [follow and manage runs](guides/studio.md#save-publish-activate-and-run), [execution management](guides/execution-management.md), and [observability](operations/observability.md) |

docs/contracts.md

@@ -217,10 +217,10 @@ other workflows are part of the language; compensation is not.
 
 **New language constructs.** `forEach`, `callWorkflow`, `concat`, and `join`
 each need a language feature: `flow.forEach`, `flow.callWorkflow`, `text.concat`,
-and `text.join`. This version compiles and runs `concat` and `join`, and compiles
-`forEach`: the compiler checks a loop's scope, types, and limits and lowers it,
-but this platform does not run loops yet, so it refuses to publish, import, or
-activate a workflow that uses one (`WV-IR-UNSUPPORTED`). The shape of
+and `text.join`. This version compiles and runs `concat`, `join`, and `forEach`;
+see [loops at run time](reference/compiler.md#loops-at-run-time). A platform
+that does not list `flow.forEach` refuses to publish, import, or activate a
+workflow that uses a loop (`WV-IR-UNSUPPORTED`). The shape of
 `callWorkflow` is final, so `load_definition` accepts it, but this version of the
 compiler reports `WV-COMP-UNSUPPORTED_FEATURE` at each use instead of compiling
 it. A platform runs a construct only when its
@@ -459,7 +459,7 @@ backend is absent. The `integration` and `e2e` pytest markers are registered.
 | `null` is rejected for an optional field | Optional fields may be omitted but not set to null | Remove the field |
 | A step's `retry` or `timeoutSeconds` is rejected | Retry and per-attempt timeout belong to the action definition | Move them to the action; use `spec.timeoutSeconds` for a workflow-wide timeout |
 | `WV-COMP-UNSUPPORTED_FEATURE` | The document uses `callWorkflow`, which this version of the compiler does not compile yet | Keep the document for a later version, or replace the construct with the steps the compiler supports |
-| `WV-IR-UNSUPPORTED` when publishing a workflow with `forEach` | The compiler accepts the loop, but this platform does not run loops yet, so it refuses to publish it; `result.missing_features` is `["flow.forEach"]` | Keep the workflow as a draft until the platform lists `flow.forEach` in its language manifest |
+| `WV-IR-UNSUPPORTED` when publishing a workflow with `forEach` | The platform you publish to is older and does not list `flow.forEach` in its language manifest; `result.missing_features` is `["flow.forEach"]` | Upgrade that platform, or keep the workflow as a draft until it lists the feature |
 | `WV-DECISION-OPERATOR` | A decision table rule uses `concat` or `join` | Build the text in a Transform step before the table and pass it in as input |
 | `WV-IR-UNSUPPORTED` on activation, a catalog read, or a run read | The platform does not list an IR version or a language feature the workflow uses; `result.missing_features` names the missing features and is empty when the IR version itself is unknown. A catalog list shows such a version as an unavailable item with `reason: ir_unsupported` and the same `missing_features`. A run that uses such a feature waits for an upgrade; signals and task results for it answer the same way, and run lists show it as unavailable | Upgrade the platform, or activate a version that does not use the feature |
 | `onFailure` is rejected on a call step | `onFailure` applies only when the call waits for its result | Remove `onFailure`, or set `mode: wait` |

docs/contributing/source-inventory.toml

@@ -291,6 +291,15 @@ copyright = "Copyright 2026 Firefly Software Foundation."
 author = "Firefly Software Foundation"
 license = "Apache-2.0"
 
+[[exceptions]]
+path = "tests/fixtures/golden/loops.json"
+kind = "generated"
+source = "scripts/golden_loops.py"
+reason = "Strict JSON golden corpus of compiled loop workflows and their kernel transition streams, recorded once by the release that started running loops; never regenerate it to make a change pass."
+copyright = "Copyright 2026 Firefly Software Foundation."
+author = "Firefly Software Foundation"
+license = "Apache-2.0"
+
 [[exceptions]]
 path = "tests/fixtures/public/c6-wire.json"
 kind = "commentless"

docs/operations/observability.md

@@ -167,10 +167,19 @@ seconds.
 
 | Type | Instruments |
 | --- | --- |
-| Counters | `weave.requests`, `weave.transitions`, `weave.recoveries`, `weave.provider_events`, `weave.outbox_deliveries`, `weave.secret_capacity`, `weave.compatibility`, `weave.retention`, `weave.telemetry_dropped`, `weave.queue_unknown` |
+| Counters | `weave.requests`, `weave.transitions`, `weave.recoveries`, `weave.provider_events`, `weave.outbox_deliveries`, `weave.secret_capacity`, `weave.compatibility`, `weave.retention`, `weave.loop_iterations`, `weave.loop_yields`, `weave.telemetry_dropped`, `weave.queue_unknown` |
 | Histograms | `weave.duration`, `weave.queue_age` |
 | Gauges | `weave.inventory`, `weave.inventory_freshness` |
 
+**Loops.** `weave.loop_iterations` counts finished loop iterations with
+`operation: loop`: `status: ok` for each result a loop collected, and `status:
+cancelled` for each iteration still running when its run ended, however it
+ended: failed (the iteration holding the failing step included), canceled, or
+timed out, through ordinary terminal control or the reduced terminal control of
+an unavailable or oversized run. The failure itself is counted once, as the
+run's outcome. `weave.loop_yields` counts the times a loop paused to continue
+in a later event.
+
 **Only three attributes.** Metrics carry `operation`, `status`, and
 `error_code`. Operations come from the public operation manifest plus a few fixed
 internal names; statuses and error codes come from short allowlists, and any

docs/operations/upgrades.md

@@ -418,6 +418,18 @@ before you reopen traffic on a server that includes them:
   reservation in the database, which grew with each run ever started. Usage
   figures and limits are unchanged. Apply the migration before you restart the
   server; a server that includes it refuses to start on the previous schema.
+- **Loops run; upgrade CLIs and SDKs with the server.** The server lists
+  `flow.forEach` in its language manifest and capabilities and runs workflows
+  with **Loop over items** (`forEach`) steps. The capabilities response gains
+  `operations.runtime.loop_units_per_turn`, and human tasks inside a loop
+  answer with `instance_key` and `iteration`. CLIs and SDKs check every answer
+  against the fields they know, so alpha15 and earlier clients fail
+  `capabilities.read` (`weave remote capabilities`, `capabilities()` in the SDK)
+  with `WV-WIRE` against such a server. Upgrade them together with the server. Before
+  you roll back to alpha15, let every run that uses a loop finish, or cancel it:
+  alpha15 does not list `flow.forEach`, so such runs wait until a server that
+  runs loops takes over, as described in
+  [runs that need a newer server](#runs-that-need-a-newer-server).
 - **AI test times use the database clock.** A recorded AI connection test is
   dated with the database's clock, the same clock readiness uses to measure its
   age. Tests recorded by an earlier server keep their API-host timestamps.

docs/reference/compiler.md

@@ -398,6 +398,50 @@ read a step's output as `/steps/<step ID>`. A branch frame is keyed by its owner
 step, its block name, and the loop indexes of the steps it holds: `checks/credit`
 outside loops, `checks/credit[3]` in iteration 3.
 
+### Loops at run time
+
+A `forEach` step evaluates `items` once. A value that is not a list opens the
+incident `WV-RUNTIME-LOOP-ITEMS`, and more elements than `maxItems` open
+`WV-RUNTIME-LOOP-LIMIT`; either way the event's work is rolled back and the run
+waits behind the incident, which an operator can only end with `terminate`. An
+empty list completes the loop at once with the output `[]`.
+
+- **Iterations.** Iteration `i` runs the body for element `i` in the frame
+  `<loop id>/body[i]`, with `/item`, `/index`, and `/loops/<id>/…` bound to the
+  current elements. The loop starts iterations in element order, never more than
+  `concurrency` at a time, and starts the next one as each finishes. Each step of
+  iteration `i` runs under the instance key `<step id>[i]`, so tasks, human tasks,
+  waits, and incidents of different iterations never share state.
+- **Results.** The body's output for element `i` is kept at position `i`; when
+  all iterations finish, the loop's output lists them in element order, whatever
+  order they finished in, without the `null` results under `collect: nonNull`.
+  The collected output is checked against the 1 MiB payload limit as each
+  iteration finishes, and `WV-RUNTIME-LOOP-OUTPUT` opens as soon as it would be
+  exceeded.
+- **Cleanup.** A finished iteration's frame, and every frame, join, and loop its
+  body created, are removed from the run state, so state does not grow with the
+  number of elements. Blocks of sibling loops with the same position are kept.
+- **Failures.** Retries belong to each action, per iteration, and keep the
+  iteration's connector idempotency key `{run_id}:<step id>[i]`. A task incident
+  in one iteration suspends the whole run; results of other iterations are kept
+  and applied after the incident is resolved. A `fail` step in any iteration
+  fails the run and cancels the other iterations, as in a parallel branch.
+  Canceling the run, or a run or human-task timeout, ends its loops the same way.
+- **Work per event.** One event starts at most 4,000 units of loop work: each
+  iteration costs one unit plus one per node of its body. A loop also stops
+  starting iterations once those started in the event could issue half of the
+  1,000 tasks and deadlines one event may hold. A loop that must stop with
+  iterations left **yields**: it registers a zero-duration wait under the key
+  `<loop instance>~<n>` (`notify~2`) that the deadline scanner wakes in a later
+  event, about one scanner poll later. While a yield is pending the loop starts
+  no other iterations, and the first iteration a loop starts in an event is
+  always allowed, so every loop progresses. The capabilities response reports the
+  budget as `operations.runtime.loop_units_per_turn`.
+- **History.** Loop commands carry the element position (`index`) on
+  `spawn_branch` and `accept_output`; yields appear as `wait_elapsed` events for
+  the yield key. Nothing new is stored for loops: iterations use the same run
+  state, step results, tasks, human tasks, and deadlines as other steps.
+
 ## Import an artifact
 
 `import_artifact(bytes_or_text_or_object, *, limits=ArtifactLimits(), features=None)`
@@ -563,6 +607,9 @@ is not classified. Safe ordinary artifacts keep their canonical digests.
 | `ok: false` after `weave workflow validate` with `validationOk: true` | Partial validation never produces an artifact | Compile with `--catalog` |
 | `WV-COMP-TYPE_MISMATCH` at `/spec/output` | An expression's type cannot satisfy the target schema | Change the expression or the schema; the authoring guide shows an example |
 | `WV-COMP-UNAVAILABLE_REFERENCE` | An expression reads a step that has not run yet in its scope, or a value private to another branch | Read the step only after it runs, or read the decision or parallel step's output |
+| Incident `WV-RUNTIME-LOOP-ITEMS` on a loop | `items` was not a list when the loop started | Terminate the run, then fix the step that produces the list or point `items` at a list |
+| Incident `WV-RUNTIME-LOOP-LIMIT` on a loop | `items` had more elements than the loop's `maxItems` | Terminate the run; raise `maxItems` (at most 10,000) or split the list before the loop |
+| Incident `WV-RUNTIME-LOOP-OUTPUT` on a loop | The collected results would exceed the 1 MiB payload limit | Terminate the run; return only what later steps need from the body, or use `collect: nonNull` to drop empty results |
 | `WV-COMP-UNKNOWN_ACTION` (or `_CONNECTOR`, `_TASK`, `_ADAPTER`) | The catalog does not contain that exact reference | Add the exact version to the catalog, or fix the reference |
 | `WV-COMP-UNSUPPORTED_FEATURE` | The workflow uses a construct whose language feature this compiler does not compile yet (`callWorkflow`); the message names the feature | Keep the document for a later version, or use the steps and operators this version compiles |
 | `WV-COMP-LOOP_ITEMS` at `…/items` | A loop's `items` can never be a list | Point `items` at a list, such as an array field of the input or of a step's output |

docs/reference/history-and-replay.md

@@ -208,6 +208,11 @@ sidecar, committed in the same transaction as the fact:
   spelling makes the history inconsistent. An export's `node_paths` stays keyed
   by node ID, so map an event's `node_id` to its path through the node ID part of
   the key (`node_of`).
+- A loop's history shows `spawn_branch` and `accept_output` commands with the
+  element position in `index`, its iterations' events under keys such as
+  `send[2]`, and each yield as a `wait_elapsed` event for its yield key
+  (`notify~1`). Replay checks these like any other fact, so the exported history
+  of a run with loops replays as `consistent`.
 - The expected transition is the SHA-256 of a canonical `classified-v1` safe
   projection of the kernel state, identified as `sha256-rfc8785-v1`; no
   duplicate snapshot is stored.

docs/reference/incident-operations.md

@@ -345,6 +345,8 @@ Some runs created before the current admission policy cannot be shown safely.
 | --- | --- | --- |
 | HTTP 409 `WV-RUNTIME-RECONCILIATION_REQUIRED` | `retry_safe` was used for an action that is not safe to repeat, the incident is not tied to an action (only `terminate` works), or a reconciled result lacks `output` or `evidence_reference` | Reconcile externally and use `accept_reconciled_result` with both fields, or `terminate` |
 | HTTP 409 `WV-INCIDENT-REVISION` | Another decision changed the incident since you read it | Read the incident again; decide only if it is still active |
+| A task incident keyed `send[2]:0` | A task failed in iteration 2 of a loop; the other iterations' results wait behind the incident | Decide it like any task incident; `retry_safe` retries only that iteration |
+| `WV-RUNTIME-LOOP-ITEMS`, `WV-RUNTIME-LOOP-LIMIT`, or `WV-RUNTIME-LOOP-OUTPUT` | A loop's list was not a list, was longer than `maxItems`, or its results exceeded the payload limit; see [loops at run time](compiler.md#loops-at-run-time) | Only `terminate` applies; fix the workflow and start a new run |
 | HTTP 409 `WV-RUNTIME-STATE` | The incident is no longer active, or no failed task matches it | Read the run and its incidents again |
 | HTTP 409 `WV-RUNTIME-DEADLINE` on `retry_safe` | The action's original deadline has passed | Use `terminate`, then start a linked retry of the whole run |
 | HTTP 422 `WV-RUNTIME-OUTPUT` | The reconciled `output` does not match the action's pinned output schema or exceeds the payload limit, or an `output` was sent with `retry_safe` or `terminate` | Fix the output to match that schema; leave `output` out for the other decisions |

examples/language/notify-overdue.workflow.yaml

@@ -14,8 +14,8 @@
 # Author: Firefly Software Foundation
 # SPDX-License-Identifier: Apache-2.0
 
-# Loop over items with a text template: one reminder per overdue invoice, four at a time. The compiler checks and
-# compiles the loop; a platform runs it once its language manifest lists the flow.forEach feature.
+# Loop over items with a text template: one reminder per overdue invoice, four at a time, collected in invoice order.
+# A platform runs it when its language manifest lists the flow.forEach feature.
 apiVersion: weave/v1alpha1
 kind: Workflow
 metadata: