Compile loops over items, with scope, types, limits and data suggestions

CHANGELOG.md

@@ -20,6 +20,17 @@ SPDX-License-Identifier: Apache-2.0
 
 ## Unreleased
 
+- The compiler checks and compiles **Loop over items** (`forEach`): `/item`,
+  `/index`, and `/loops/<id>/…` inside a loop body, a typed list of results after
+  it, `WV-COMP-LOOP_*` diagnostics for wrong items, scope, nesting, limits, and
+  signals in a body (with a suggested `maxItems` for a loop budget that is too
+  high), and an IR `forEach` node with an `each` edge and an `each` join that
+  requires the `flow.forEach` feature. This platform does not run loops yet, so
+  it refuses to publish, import, or activate such a workflow with
+  `WV-IR-UNSUPPORTED`; workflows without loops compile to the same bytes as
+  before. The compiler limits add `max_loop_items`, `max_loop_depth`, and
+  `max_run_iterations`. Studio's local host answers `POST /studio/local/scope`
+  with the data a field may use, including the current loop item and position.
 - Starting a run no longer slows down as executions accumulate. The usage accounting that runs on every run start
   and every change to quota-counted records read every reservation in the database, one per run ever started, and
   now finds its reservation and allocation entries by primary key. Usage figures and limits are unchanged. New

docs/concepts.md

@@ -306,7 +306,7 @@ Use this table to translate what you know:
 | Error boundary event | Not in the language | A failed Action retries when that is safe; otherwise the run is suspended with an incident for an operator. To branch on a business outcome, return it in the Action's output and test it with a Decision |
 | Message throw or end event | **Call an action**, or an integration event subscription | A subscription notifies another system when a run's status changes |
 | Call activity, subprocess | **Call a workflow** (`callWorkflow`) | Defined in the [workflow language](contracts.md#steps), but this version does not compile or run it: the compiler reports `WV-COMP-UNSUPPORTED_FEATURE`. Until then, keep the part inside the same workflow, or make it a separate workflow that your product starts |
-| Loop, multi-instance activity | **Loop over items** (`forEach`) | Defined in the [workflow language](contracts.md#steps), but this version does not compile or run it: the compiler reports `WV-COMP-UNSUPPORTED_FEATURE`. Until then, use a Parallel with a fixed set of branches, or start one run per item |
+| Loop, multi-instance activity | **Loop over items** (`forEach`) | Defined in the [workflow language](contracts.md#steps). This version compiles and checks it, but the platform does not run it yet and refuses to publish it (`WV-IR-UNSUPPORTED`). Until then, use a Parallel with a fixed set of branches, or start one run per item |
 | Compensation | Not in the language | Model compensating Actions as ordinary steps; Weave never undoes an external effect |
 | Instance migration | Not available | A running run keeps its version; new runs use the new activation |
 | Process monitoring, cockpit | **Runs**, history, and incidents in Studio and the CLI; logs, metrics, and traces | Studio draws a run's graph and marks its current step **Now**; see [follow and manage runs](guides/studio.md#save-publish-activate-and-run), [execution management](guides/execution-management.md), and [observability](operations/observability.md) |

docs/contracts.md

@@ -217,10 +217,13 @@ other workflows are part of the language; compensation is not.
 
 **New language constructs.** `forEach`, `callWorkflow`, `concat`, and `join`
 each need a language feature: `flow.forEach`, `flow.callWorkflow`, `text.concat`,
-and `text.join`. This version compiles and runs `concat` and `join`. The shape of
-`forEach` and `callWorkflow` is final, so `load_definition` accepts them, but this
-version of the compiler reports `WV-COMP-UNSUPPORTED_FEATURE` at each use instead
-of compiling it. A platform runs a construct only when its
+and `text.join`. This version compiles and runs `concat` and `join`, and compiles
+`forEach`: the compiler checks a loop's scope, types, and limits and lowers it,
+but this platform does not run loops yet, so it refuses to publish, import, or
+activate a workflow that uses one (`WV-IR-UNSUPPORTED`). The shape of
+`callWorkflow` is final, so `load_definition` accepts it, but this version of the
+compiler reports `WV-COMP-UNSUPPORTED_FEATURE` at each use instead of compiling
+it. A platform runs a construct only when its
 [language manifest](#language-manifest) lists the feature. Step IDs can never
 contain `[`, `#`, or `~`, which keeps
 [instance keys](reference/compiler.md#instance-keys) unambiguous.
@@ -229,6 +232,20 @@ Branches may contain zero steps, but each must declare its `output`. The compile
 not the shape check, enforces unique step IDs, unique signal names, branch scope,
 output compatibility, and concurrency budgets.
 
+**Loop scope.** Inside a `forEach` body, `/item` is the current element and
+`/index` its position from 0; an enclosing loop's are `/loops/<loop id>/item` and
+`/loops/<loop id>/index`. A body step sees what is visible at the loop plus the
+earlier steps of the same iteration, and `body.output` also sees the last body
+step. After the loop only `/steps/<loop id>/output` is visible: a list with one
+result per element, in element order, holding at most `maxItems` results (with
+`collect: nonNull`, results that are `null` are left out). `items` sees what is
+visible at the loop, never the loop's own item. Loops nest at most three deep, a
+body cannot wait for a signal, and all loops of a workflow together, each
+multiplied by the `maxItems` of the loops around it, may need at most 100,000
+iterations; the compiler suggests a lower `maxItems` for the inner loop when they
+need more. The [compiler reference](reference/compiler.md#troubleshooting) lists
+the diagnostics.
+
 **Human tasks** need an environment assignment binding at activation, and
 completing one requires the current claim and task permission. A workflow with a
 human task compiles to IR version `weave/ir-v1alpha2` or higher, because the
@@ -363,6 +380,9 @@ The operator's `Limits()` defaults bound compilation:
 | `max_document_nodes` | 100,000 | Parsed values, including the root, containers, and scalars (mapping keys excluded) |
 | `max_expression_nodes` | 10,000 | Expression nodes, counted by the compiler stage that owns expressions |
 | `max_diagnostics` | 100 | Diagnostics returned per result |
+| `max_loop_items` | 10,000 | The largest `maxItems` a loop may declare |
+| `max_loop_depth` | 3 | Loops nested inside one another |
+| `max_run_iterations` | 100,000 | Iterations all loops of one workflow may need, each loop multiplied by the `maxItems` of the loops around it |
 
 Definitions cannot change these budgets. Building a model alone does not apply
 them; the compiler does.
@@ -438,7 +458,8 @@ backend is absent. The `integration` and `e2e` pytest markers are registered.
 | A field is rejected although it looks right | Field names are camelCase and case-sensitive; snake_case keys are rejected | Use the published name, such as `timeoutSeconds` |
 | `null` is rejected for an optional field | Optional fields may be omitted but not set to null | Remove the field |
 | A step's `retry` or `timeoutSeconds` is rejected | Retry and per-attempt timeout belong to the action definition | Move them to the action; use `spec.timeoutSeconds` for a workflow-wide timeout |
-| `WV-COMP-UNSUPPORTED_FEATURE` | The document uses `forEach` or `callWorkflow`, which this version of the compiler does not compile yet | Keep the document for a later version, or replace the construct with the steps the compiler supports |
+| `WV-COMP-UNSUPPORTED_FEATURE` | The document uses `callWorkflow`, which this version of the compiler does not compile yet | Keep the document for a later version, or replace the construct with the steps the compiler supports |
+| `WV-IR-UNSUPPORTED` when publishing a workflow with `forEach` | The compiler accepts the loop, but this platform does not run loops yet, so it refuses to publish it; `result.missing_features` is `["flow.forEach"]` | Keep the workflow as a draft until the platform lists `flow.forEach` in its language manifest |
 | `WV-DECISION-OPERATOR` | A decision table rule uses `concat` or `join` | Build the text in a Transform step before the table and pass it in as input |
 | `WV-IR-UNSUPPORTED` on activation, a catalog read, or a run read | The platform does not list an IR version or a language feature the workflow uses; `result.missing_features` names the missing features and is empty when the IR version itself is unknown. A catalog list shows such a version as an unavailable item with `reason: ir_unsupported` and the same `missing_features`. A run that uses such a feature waits for an upgrade; signals and task results for it answer the same way, and run lists show it as unavailable | Upgrade the platform, or activate a version that does not use the feature |
 | `onFailure` is rejected on a call step | `onFailure` applies only when the call waits for its result | Remove `onFailure`, or set `mode: wait` |

docs/contributing/source-inventory.toml

@@ -469,6 +469,15 @@ copyright = "Copyright 2026 Firefly Software Foundation."
 author = "Firefly Software Foundation"
 license = "Apache-2.0"
 
+[[exceptions]]
+path = "studio/tests/fixtures/language/scope.json"
+kind = "generated"
+source = "scripts/language_fixtures.py"
+reason = "Strict JSON scope suggestion cases inside and around loops shared by the Python and Studio suites; regenerate with the generator, which also checks freshness."
+copyright = "Copyright 2026 Firefly Software Foundation."
+author = "Firefly Software Foundation"
+license = "Apache-2.0"
+
 [[exceptions]]
 path = "studio/tests/fixtures/language/template-segments.json"
 kind = "generated"

docs/reference/compiler.md

@@ -156,8 +156,25 @@ catalog:
 - Definite problems are still errors: `WV-COMP-UNAVAILABLE_REFERENCE` (a step
   reads a step that has not run yet in its scope), `WV-COMP-TYPE_MISMATCH`, and
   `WV-COMP-CONNECTION` for a step that names an undeclared connection slot.
+- A loop whose `items` read a pending output leaves `/item` unconstrained in the
+  same way.
 - Pending notes only use diagnostic capacity that real findings leave free.
 
+**Data suggestions.** For a field Studio is editing, the host answers
+`POST /studio/local/scope` with `{source, format, path}`, where `path` is the
+JSON Pointer of an expression in the definition. It returns
+`firefly_weave.compiler.scope.scope_at(document, path, *, catalog=None)`: `found`
+(false when the path is not inside an expression the compiler analyzes, or the
+document stops before flow analysis), `evaluated` (false when every path to it
+fails), `truncated`, and `entries`. Entries list the workflow input, then the
+current loop's item (`/item`, `source: item`) and position (`/index`, `source:
+index`), then enclosing loops by ID (`/loops/<id>/item`), then the visible step
+outputs in run order, each followed by its fields up to four levels deep and at
+most 200 entries in all. Each entry carries `ref`, `source`, `loop` or `stepId`
+and `stepKind`, `path`, `label`, `breadcrumb`, `schema`, `types`, `typeLabel`, and
+`optional`. Without a catalog, outputs of actions and decision tables stay
+unconstrained, as in offline validation.
+
 The result is always `partial=True`, `artifact=None`, and `ok=False`;
 `validation_ok` counts error-severity findings only. When Studio is connected and
 your account may compile, **Validate** compiles against the project catalog
@@ -213,9 +230,9 @@ Executables also define a sorted `features` list of the language features they
 need (`text.concat`, `text.join`, `flow.forEach`, `flow.callWorkflow`,
 `ai.agent`, `ai.memory`). It is omitted when empty, so existing digests do not
 change. A workflow that uses `concat` or `join` in any expression compiles to
-`weave/ir-v1alpha4` with `text.concat`, `text.join`, or both; v1alpha4 always
-has a non-empty list, and the validator recomputes the version and the list from
-the graph. The version is the highest level any construct needs, so a workflow
+`weave/ir-v1alpha4` with `text.concat`, `text.join`, or both, and a workflow with
+a `forEach` step lists `flow.forEach`; v1alpha4 always has a non-empty list, and
+the validator recomputes the version and the list from the graph. The version is the highest level any construct needs, so a workflow
 with a human task and `join` is v1alpha4, not v1alpha2. A platform imports and
 activates such an artifact only when its capabilities list every feature in
 `language_features`; otherwise it answers `ir_unsupported`. `ir.py` defines
@@ -315,6 +332,16 @@ flow as: start, decide, take one branch, join, end. The rules behind it:
   `fork` edges name the branches, `concurrency` carries the declared cap, and the
   join uses `mode: all`, collecting every named branch output into the step's
   output object.
+- **Loops.** A `forEach` node carries `items`, `concurrency`, `maxItems`, and
+  `collect` (defaults written out), its `target`, and its `join`. Its only edge is
+  an `each` edge with branch `body` to `target`: the first body step, or the
+  body's `branch-output` node when the body is empty. Body nodes have the scope
+  `[<loop id>, body]` after the loop's own scope. The body's `branch-output`
+  (`@branch:<loop id>:0`) runs `body.output` and always completes into the join
+  `@join:<loop id>`, which uses `mode: each` and continues with a `next` edge.
+  The graph stays acyclic: the runtime enters the body once per element. A
+  `loop_items` guard on `<step path>/items` (schema `{"type": "array"}`) is added
+  only when the compiler cannot prove that `items` is a list.
 - **Branch outputs.** Every branch has a `branch-output` node, even an empty one.
   Its output expression runs in the branch's scope; an empty branch enters it
   directly.
@@ -414,7 +441,7 @@ Each kind of input has its own policy, passed as a separate keyword:
 | `SchemaLimits` | Author schemas | See [schema profile limits](schema-profile.md#limits) |
 | `DEFAULT_CONTRACT_LIMITS` | Weave's generated outer contracts | Author limits with a larger work budget |
 | `ArtifactLimits` | The compiled artifact and its import | 33,554,432 bytes (32 MiB), 2,000,000 JSON value nodes, depth 128 |
-| `max_parallel_concurrency` | The largest `concurrency` a parallel step may declare | 1,000 |
+| `max_parallel_concurrency` | The largest `concurrency` a parallel or `forEach` step may declare | 1,000 |
 
 Limits are strict positive integers that source cannot raise. Artifact
 construction charges each fragment before adding it, then measures the whole
@@ -537,7 +564,17 @@ is not classified. Safe ordinary artifacts keep their canonical digests.
 | `WV-COMP-TYPE_MISMATCH` at `/spec/output` | An expression's type cannot satisfy the target schema | Change the expression or the schema; the authoring guide shows an example |
 | `WV-COMP-UNAVAILABLE_REFERENCE` | An expression reads a step that has not run yet in its scope, or a value private to another branch | Read the step only after it runs, or read the decision or parallel step's output |
 | `WV-COMP-UNKNOWN_ACTION` (or `_CONNECTOR`, `_TASK`, `_ADAPTER`) | The catalog does not contain that exact reference | Add the exact version to the catalog, or fix the reference |
-| `WV-COMP-UNSUPPORTED_FEATURE` | The workflow uses a construct whose language feature this compiler does not compile yet (`forEach` or `callWorkflow`); the message names the feature | Keep the document for a later version, or use the steps and operators this version compiles |
+| `WV-COMP-UNSUPPORTED_FEATURE` | The workflow uses a construct whose language feature this compiler does not compile yet (`callWorkflow`); the message names the feature | Keep the document for a later version, or use the steps and operators this version compiles |
+| `WV-COMP-LOOP_ITEMS` at `…/items` | A loop's `items` can never be a list | Point `items` at a list, such as an array field of the input or of a step's output |
+| `WV-COMP-UNKNOWN_COMPATIBILITY` at `…/items` | The compiler cannot prove that `items` is a list; the run checks it | Declare the field as an array where it comes from |
+| `WV-COMP-LOOP_SCOPE` at `…/ref` | `/item`, `/index`, or `/loops/<id>` is read outside a loop body, in the loop's own `items`, or names a loop that does not enclose the expression | Read loop data only inside the loop's body; use `/loops/<id>/item` for an enclosing loop |
+| `WV-COMP-LOOP_SIGNAL` | A loop body contains a **Wait for signal** step | Move the wait before or after the loop |
+| `WV-COMP-LOOP_BODY_INCOMPLETE` at `…/body` | Every path through the body ends in **Fail** | Let the body finish with a result for each element |
+| `WV-COMP-LOOP_DEPTH` | A loop is nested more than three deep | Flatten the inner loops |
+| `WV-COMP-LOOP_MAX_ITEMS` at `…/maxItems` | `maxItems` is above the platform's 10,000 | Lower `maxItems` |
+| `WV-COMP-LOOP_BUDGET` | All loops together, each multiplied by the `maxItems` of the loops around it, could need more than 100,000 iterations in one run | Apply the suggested edit, which lowers the inner loop's `maxItems`, or lower another loop's |
+| `WV-COMP-CONCURRENCY_LIMIT` at `…/concurrency` | A loop or parallel step runs more at a time than the platform allows (1,000) | Lower `concurrency` |
+| `WV-COMP-LOOP_FANOUT` (warning) | Nested loops together could start more than 1,000 tasks at once | Nothing to fix; the run spreads the work over several turns. Lower `concurrency` if the tasks must start together |
 | `WV-DECISION-OPERATOR` | A decision table rule uses `concat` or `join` | Build the text in a Transform step and pass it to the table as input |
 | `UnsupportedIR` on import, `WV-IR-UNSUPPORTED` on activation, a catalog read, or a run read | The artifact needs an IR version or language feature this platform does not list; `missing` names the missing features and is empty when the IR version itself is unknown. A run that needs it waits for an upgrade, and signals and task results for it answer the same way | Upgrade the platform, or use an artifact without that feature |
 | `WV-COMP-CATALOG_PENDING` notes in Studio | Studio works without the catalog and checks references later | Nothing; connect and **Validate** against the project catalog |

docs/reference/history-and-replay.md

@@ -205,7 +205,9 @@ sidecar, committed in the same transaction as the fact:
 - Facts and issued commands name each run of a step by its
   [instance key](compiler.md#instance-keys), which is the step ID outside loops.
   Replay finds the step from the step ID part of a canonical key; any other
-  spelling makes the history inconsistent.
+  spelling makes the history inconsistent. An export's `node_paths` stays keyed
+  by node ID, so map an event's `node_id` to its path through the node ID part of
+  the key (`node_of`).
 - The expected transition is the SHA-256 of a canonical `classified-v1` safe
   projection of the kernel state, identified as `sha256-rfc8785-v1`; no
   duplicate snapshot is stored.

examples/language/notify-overdue.workflow.yaml

@@ -14,8 +14,8 @@
 # Author: Firefly Software Foundation
 # SPDX-License-Identifier: Apache-2.0
 
-# Loop over items with a text template. This version compiles concat but reports "not yet supported" for
-# forEach; the document shape is final.
+# Loop over items with a text template: one reminder per overdue invoice, four at a time. The compiler checks and
+# compiles the loop; a platform runs it once its language manifest lists the flow.forEach feature.
 apiVersion: weave/v1alpha1
 kind: Workflow
 metadata: