Skip to content

Compile loops over items, with scope, types, limits and data suggestions - #42

Merged
ancongui merged 11 commits into
mainfrom
feat/loop-compiler
Oct 13, 2026
Merged

ancongui merged 11 commits into
mainfrom
feat/loop-compiler

Conversation

@ancongui

Copy link
Copy Markdown
Contributor

Summary

The compiler now checks and compiles Loop over items (forEach). Loops do not run yet: this platform refuses to publish, import or activate a workflow that uses one, and every workflow without loops compiles to the same bytes as before.

  • Scope. Inside a loop body, /item is the current element and /index its position from 0; an enclosing loop's are /loops/<loop id>/item and /loops/<loop id>/index. Body steps see what is visible at the loop plus earlier steps of the same iteration. After the loop only /steps/<loop id>/output is visible. Reading loop data anywhere else is WV-COMP-LOOP_SCOPE.
  • Types. The item type comes from the items list (tuple positions become anyOf), the position is a whole number below maxItems, and the loop's output is a list of one body result per element, at most maxItems long, without nulls for collect: nonNull.
  • Checks. WV-COMP-LOOP_ITEMS (items can never be a list; an unproved list warns and keeps a run-time loop_items guard), LOOP_BODY_INCOMPLETE, LOOP_SIGNAL (a body cannot wait for a signal), LOOP_DEPTH (three nested loops at most), LOOP_MAX_ITEMS (10,000), CONCURRENCY_LIMIT (shared with parallel), LOOP_BUDGET (all loops together, each multiplied by the loops around it, need at most 100,000 iterations; the diagnostic suggests a lower maxItems for an inner loop), and a LOOP_FANOUT warning for nested concurrency above 1,000 tasks. Limits gains max_loop_items, max_loop_depth and max_run_iterations.
  • IR. A forEach node with one each edge into its body, a body result node and an each join; the validator checks the shape, the body scope and that the body completes. A loop selects IR weave/ir-v1alpha4 with the flow.forEach feature, which this platform does not list yet.
  • Platform boundary. Publishing a workflow whose IR features the platform does not run now answers 422 WV-IR-UNSUPPORTED before anything is stored (import, catalog reads, activation and debug sessions already refused such artifacts). A runtime that still receives a loop opens an incident instead of skipping the step.
  • Tools. weave workflow explain prints loop settings and each edges; weave workflow graph draws compiled loops. Studio's local host answers POST /studio/local/scope with the data a field may use, from the compiler's own analysis, including the current loop item and position. The route requires a paired browser and the CSRF token, validates its request strictly, and reads no files, secrets or connections. Studio explains the loop diagnostics in plain language.
  • Shared fixtures. studio/tests/fixtures/language/scope.json, checked by the Python suite and by Studio's vitest suite against Studio's own type labels.

Verification

  • Local, offline (macOS): make check, all stages (6,664 unit and contract tests, worker suites, lint, format, mypy, source coverage, docs and packaging); scripts/golden_corpus.py --check and scripts/language_fixtures.py --check; Studio type check, format check, vitest (1,797 tests) and production build. A separate review rerun the unit and contract tests, lint, mypy and both checks on the final head, fuzzed scope_at at every position of every example and fixture, and probed the new host route (pairing, origin, CSRF, size and request validation).
  • Local, real PostgreSQL 17 (owned throwaway container): the new publication tests (refused without the loop feature; stored and read back with it) and the definitions, unsupported-run, wait and incident suites: 76 passed.
  • CI only: the cross-platform jobs on macOS and Windows run every compiler test of this change (IR, scope, typing, limits, scope_at, fixtures); runtime, CLI and Studio host unit tests run on Linux in CI and ran on macOS locally. The Acceptance workflow runs the pr profile. No integration tests run in CI.
  • Not run: no Studio screen changed, so no browser suite or visual tour was rerun.

Decisions

  • Publishing is refused until loops run. A workflow that needs a language feature this platform does not run is refused with 422 WV-IR-UNSUPPORTED before anything is stored, so the catalog never holds a version the platform cannot read back or activate. Drafts with loops save and validate normally.
  • Items not proved to be a list compile with a WV-COMP-UNKNOWN_COMPATIBILITY warning and a run-time loop_items guard; items that can never be a list are WV-COMP-LOOP_ITEMS. Once items has an error, references into /item are not checked further, so one mistake gives one diagnostic.
  • Loop limits are checked during flow analysis, so they appear next to type errors. The iteration budget is reported once, on an inner loop; the suggested edit is the largest maxItems that fits both the budget and the platform's item limit, on the inner loop with the most iterations that one edit can fix, or no edit when none can. The fan-out warning counts nested loop concurrency only and appears once, where it first exceeds 1,000 tasks.
  • Data suggestions list the workflow input, the current item and position, enclosing loops, then visible steps, each followed by its fields (four levels, at most 200 entries), with Studio's own type labels. A draft that cannot be analyzed answers found: false, not an error, because Studio asks while the author types.
  • A loop that reaches a runtime opens an incident (WV-RUNTIME-ARTIFACT) instead of being skipped; import already refuses such artifacts.
  • No language manifest change: forEach stays marked as not yet editable in Studio until Studio's loop editor ships.

Follow-ups

  • Running loops: iteration state, spawning within concurrency, collecting results in element order, yields that spread large loops over several turns, the loop_items and item-count checks at run time, cleanup of finished iterations, and listing flow.forEach (which replaces the kernel refusal and lifts the publication refusal for loops).
  • Loop simulation: per-iteration views, breakpoints and mocks in debug sessions, and refusing loop artifacts there with WV-IR-UNSUPPORTED instead of a generic validation error until then.
  • Studio: the loop editor (Items, At a time, Up to, Collect), the loop frame on the canvas, Studio's offline scope passing the shared fixtures, calling /studio/local/scope for suggestions, and warning before publishing to a platform that does not list a feature the workflow needs; the CLI gets the same pre-publish warning.
  • Separately: weave workflow graph fails on workflows with a decision table (existing issue).

@ancongui
ancongui merged commit d813e2e into main Oct 13, 2026
17 checks passed
@ancongui
ancongui deleted the feat/loop-compiler branch October 13, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant