Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions docs/breaking-changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,65 @@ description: "Breaking changes between flow releases, and what to do about each

This page documents changes that require you to update existing flow files, config, or scripts when upgrading between major versions.

## v2.3.0

### Argument Values Are Literals

**What changed:** A `$word` inside an argument value used to be expanded against flow's
environment and deleted when it did not resolve, so `flow run notify 'costs $5'` arrived as
`costs `. Argument values a user supplies are now literals, and pass through unchanged.

Expansion still applies to the strings you write in a flow file — an argument's `default`,
and a serial or parallel step's `args:` entries.

**How to migrate:** If you relied on a *caller* passing `$VAR` for flow to expand, expand it
in your shell instead (`flow run deploy "$TARGET"`), or move the reference into the
argument's `default`.

---

### Unresolved Variables Are Left As Written

**What changed:** A `$VAR` that does not resolve now stays in the string instead of becoming
an empty string. This affects every context flow expands: `dir` paths, `cmd` strings, an
argument's `default`, a step's `args:`, and a request's `url`, `headers` and `body`.

**Before:** `url: "https://$API_HOST/deploy"` with no `API_HOST` requested `https:///deploy`.

**After:** the same executable requests `https://$API_HOST/deploy`, which fails visibly.

`$$` now produces a literal `$`, which previously had no escape.

**How to migrate:** Nothing to change if your variables resolve. If you were depending on a
missing variable collapsing to nothing, give it an explicit empty default via a `param`.

---

### Request Bodies That Are JSON Are Sent As JSON

**What changed:** `body` was always evaluated as an Expr expression, and a JSON object is
valid Expr — it parses as a map literal and was stringified as
`map[environment:prod version:1.0]`. A body that is a JSON object or array is now sent as
written. Anything else is still an expression.

**How to migrate:** Nothing to change. A JSON body starts working; an expression body that
returns a string keeps working.

---

### Step `args:` Override Inherited Values

**What changed:** A serial or parallel step's `args:` list is now applied to the child, and
overrides values it would otherwise inherit from the parent. It was previously ignored when
the parent declared matching `args` — its mere presence, not its contents, was what enabled
inheritance.

**How to migrate:** Remove any placeholder `args:` entries added to force inheritance, such
as `args: ["-"]`. Inheritance no longer requires them, and a placeholder now reaches the
child as a real value.

---

## v2.0.0

### `fromFile` Import Field Removed
Expand Down
34 changes: 31 additions & 3 deletions docs/guides/advanced.md
Original file line number Diff line number Diff line change
Expand Up @@ -413,8 +413,20 @@ executables:
```

> [!NOTE]
> In serial and parallel executables, `params` and `args` that are defined at the parent level will apply to all
> child executables. Argument from the parent -> child executable should use matching `EnvKey` to ensure proper resolution.
> In serial and parallel executables, `params` and `args` defined at the parent level apply to all
> child executables, whether the child runs a command, a request, or a render. An argument passed
> from parent to child is matched by `envKey`, so both must declare the same one.
>
> A step's own `args:` list overrides what it would otherwise inherit, for the arguments it sets:
>
> ```yaml
> serial:
> args: [{pos: 1, envKey: TARGET}]
> execs:
> - ref: deploy service # inherits TARGET
> - ref: deploy service
> args: ["--target=staging"] # overrides it
> ```

**Resolution example:**
```shell
Expand All @@ -434,7 +446,23 @@ flow deploy app --param ENVIRONMENT=production -- --verbose=false

### Environment Variable Expansion

Environment variables are expanded in certain contexts:
`$VAR` and `${VAR}` are expanded in the parts of a flow file *you* write:

- `dir` paths
- `cmd` strings
- A request's `url`, `headers`, and `body`
- An argument's `default`
- A serial or parallel step's `args:` entries

They are **not** expanded in values a *user* supplies — a command-line argument, a prompt
response, or a `text` param. Those are literals, so `flow run notify 'costs $5'` sends
`costs $5`.

Two rules apply everywhere expansion happens:

- `$$` produces a literal `$`.
- A variable that does not resolve is left as written, so a missing `$API_HOST` shows up in
the command or URL rather than silently becoming an empty string.

**Directory paths:**
```yaml
Expand Down
48 changes: 45 additions & 3 deletions docs/guides/executables.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,9 @@ executables:
- `secretRef`: Reference to vault secret
- `prompt`: Interactive user input
- `text`: Static value
- `envFile`: Load environment variables from a file
- `envFile`: Load environment variables from a file. Relative paths resolve against the
directory holding the flow file (`FLOW_DEFINITION_DIR`), not the workspace root or your
current directory — setting `dir` on the executable does not change this.

### Arguments (`args`)

Expand Down Expand Up @@ -407,6 +409,14 @@ of the parent executable are inherited by the child executables.
- `retries`: Number of times to retry failed steps
- `reviewRequired`: Pause for user confirmation

Each step also accepts:
- `name`: A label for the step, shown in place of the ref in output
- `if`: An [Expr expression](./expressions) gating whether the step runs
- `interpreter`: The interpreter for a `cmd` step (a `ref` uses its own)
- `args`: Arguments to pass to a `ref`'d executable, written as you would type them
(`--flag=value` or a positional value). `$VAR` expands against the parent environment.
These override what the step would otherwise inherit.

### parallel - Concurrent Execution

Run multiple steps simultaneously:
Expand Down Expand Up @@ -434,6 +444,14 @@ of the parent executable are inherited by the child executables.
- `failFast`: Stop all operations on first failure (default: true)
- `retries`: Number of times to retry failed operations

Each step also accepts:
- `name`: A label for the step, shown in place of the ref in output
- `if`: An [Expr expression](./expressions) gating whether the step runs
- `interpreter`: The interpreter for a `cmd` step (a `ref` uses its own)
- `args`: Arguments to pass to a `ref`'d executable, written as you would type them
(`--flag=value` or a positional value). `$VAR` expands against the parent environment.
These override what the step would otherwise inherit.

### launch - Open Applications

Open files, URLs, or applications:
Expand Down Expand Up @@ -493,13 +511,32 @@ executables:
- `method`: HTTP method (GET, POST, PUT, PATCH, DELETE)
- `url`: Request URL (required)
- `headers`: Custom headers
- `body`: Request body
- `body`: Request body — sent as written when it is JSON, otherwise an [Expr expression](./expressions)
- `timeout`: Request timeout
- `validStatusCodes`: Acceptable status codes
- `logResponse`: Log response body
- `transformResponse`: Expr expression to reshape the response before output or file save
- `responseFile`: Save response to file

**Request bodies:**

`$VAR` references in `body` are expanded first, the same as in `url` and `headers`. What
happens next depends on the result:

- **A JSON object or array is sent as written.** This is the common case — write the body
as JSON and interpolate values with `$VAR`.
- **Anything else is evaluated as an [Expr expression](./expressions)** that must produce a
string.

The expression form is worth reaching for when a value needs escaping. `toJSON` quotes and
escapes its argument, so a prompt containing quotes or newlines survives intact — where
pasting it into a JSON template would produce a malformed body:

```yaml
body: >
'{"model":' + toJSON(env["MODEL"]) + ',"prompt":' + toJSON(env["PROMPT"]) + '}'
```

**Transforming responses with `transformResponse`:**

The `transformResponse` field is a single [Expr expression](./expressions) evaluated after the request completes. Its result replaces the raw response body in any output or `responseFile`. The expression has access to:
Expand All @@ -524,7 +561,8 @@ transformResponse: fromJSON(body)["name"]
transformResponse: upper(fromJSON(body)["status"])

# Format an array as newline-separated output
transformResponse: join(map(fromJSON(body)["items"], #["name"]), "\n")
# NOTE: quote any expression containing '#' - in YAML, a space then '#' starts a comment
transformResponse: 'join(map(fromJSON(body)["items"], #["name"]), "\n")'

# Conditional with fallback
transformResponse: code == 200 ? fromJSON(body)["result"] : "error " + string(code) + ": " + body
Expand Down Expand Up @@ -561,6 +599,10 @@ executables:
| `env` | `map[string]string` | Params and environment variables from the executable |
| `data` | `any` | Parsed contents of `templateDataFile` (nil if not set) |

By default a `render` opens an interactive viewer. To use one non-interactively — in CI, a
script, or piped into another command — set `DISABLE_FLOW_INTERACTIVE=true`, which makes it
write plain text to stdout. See [Interactive UI](./interactive#disabling-the-tui).

`data` is typed based on the file content — a JSON object becomes a map, a JSON array becomes a slice. Access fields with bracket notation: `data["key"]` or `data[0]["field"]`.

**Template file example** — given a `status-data.json`:
Expand Down
36 changes: 34 additions & 2 deletions docs/guides/expressions.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,21 @@ description: "The Expr-based expression language flow uses for conditionals and

# Expression Language

flow uses the [Expr language](https://expr-lang.org) for dynamic expressions and template logic. The same language appears in four places — learn it once and it works everywhere.
flow uses the [Expr language](https://expr-lang.org) for dynamic expressions and template logic. The same language appears in five places — learn it once and it works everywhere.

## Where Expressions Are Used

| Surface | Syntax | Context variables | Shell `$()` |
|---------|--------|-------------------|:-----------:|
| Step `if` conditions | Bare expression (no delimiters) | `os`, `arch`, `env`, `store`, `ctx` | ✓ |
| `transformResponse` | Bare expression (no delimiters) | `body`, `code`, `status`, `headers` | |
| Request `body` | Bare expression, **only when the body is not JSON** | `env` | |
| Template files (`.flow.tmpl`) | <span v-pre>`{{ expression }}`</span> delimiters | `name`, `form`, `env`, `os`, `arch`, … | |
| Render templates (render `.md`) | <span v-pre>`{{ expression }}`</span> delimiters | `env`, `data` | |

For the variables available in each surface, see the context-specific docs:
- **Step conditions** — [Advanced Workflows: Conditional Execution](./advanced#conditional-execution)
- **transformResponse** — [Executables: request](./executables#request---http-requests)
- **transformResponse** and **request body** — [Executables: request](./executables#request---http-requests)
- **Template files** — [Templates & Workflow Generation: Template Language](./templating#template-language)
- **Render templates** — [Executables: render](./executables#render---dynamic-documentation)

Expand All @@ -31,6 +32,37 @@ Expr is a sandboxed, typed, Go-native expression language. The key things that d
- `if` conditions use `==`, `not`, `and`, `or` — not `eq`, `ne`, `!`
- Shell execution via `$("command")` is available **only in step `if` conditions** — not in `transformResponse` or template surfaces

## Writing Expressions in a Flow File

Expressions live inside YAML, and YAML gets a say in what reaches flow. Three things bite:

**`#` starts a comment.** The closure shorthand used by `map()` and `filter()` is also YAML's
comment marker, so an unquoted expression is silently truncated at the first ` #`:

```yaml
# WRONG — YAML keeps only `join(map(fromJSON(body)["items"],`
transformResponse: join(map(fromJSON(body)["items"], #["name"]), "\n")

# RIGHT
transformResponse: 'join(map(fromJSON(body)["items"], #["name"]), "\n")'
```

The give-away is an error pointing past the end of what you wrote, like
`unexpected token EOF (1:32)`.

**A leading `{` starts a flow mapping.** Quote or use a block scalar for anything beginning
with `{` or `[`.

**A bare `: ` splits a key from a value.** Quote expressions containing one.

Single quotes are the safest wrapper, since Expr's own string literals are usually double
quoted. When an expression needs both, reach for a block scalar:

```yaml
if: >
env["DEPLOY_ENV"] == "production" and $("git branch --show-current") == "main"
```

## Core Syntax

### Operators
Expand Down
8 changes: 4 additions & 4 deletions docs/public/schemas/flowfile_schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -329,7 +329,7 @@
"type": "object",
"properties": {
"args": {
"description": "Arguments to pass to the executable.",
"description": "Arguments to pass to the executable, in the same form you would type them\n(`--flag=value` or a positional value). `$VAR` references are expanded against the\nparent's environment.\n\nThe parent's environment always reaches the executable; values listed here override\nwhat it would otherwise inherit for the arguments they set.\n",
"type": "array",
"default": [],
"items": {
Expand Down Expand Up @@ -357,7 +357,7 @@
},
"ref": {
"$ref": "#/definitions/ExecutableRef",
"description": "A reference to another executable to run in serial.\nOne of `cmd` or `ref` must be set.\n",
"description": "A reference to another executable to run in parallel.\nOne of `cmd` or `ref` must be set.\n",
"default": ""
},
"retries": {
Expand Down Expand Up @@ -460,7 +460,7 @@
"$ref": "#/definitions/ExecutableArgumentList"
},
"body": {
"description": "The body of the request.",
"description": "The body of the request. `$VAR` references are expanded first.\n\nIf the result is a JSON object or array, it is sent as written. Otherwise it is\nevaluated as an Expr expression that must produce a string, which lets you build\na body from values that need escaping: `'{\"prompt\":' + toJSON(env[\"PROMPT\"]) + '}'`.\n",
"type": "string",
"default": ""
},
Expand Down Expand Up @@ -582,7 +582,7 @@
"type": "object",
"properties": {
"args": {
"description": "Arguments to pass to the executable.",
"description": "Arguments to pass to the executable, in the same form you would type them\n(`--flag=value` or a positional value). `$VAR` references are expanded against the\nparent's environment.\n\nThe parent's environment always reaches the executable; values listed here override\nwhat it would otherwise inherit for the arguments they set.\n",
"type": "array",
"default": [],
"items": {
Expand Down
8 changes: 4 additions & 4 deletions docs/types/flowfile.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,12 +196,12 @@ Configuration for a parallel executable.

| Field | Type | Default | Required | Description |
| ----- | ---- | ------- | :------: | ----------- |
| `args` | `array` (`string`) | [] | | Arguments to pass to the executable. |
| `args` | `array` (`string`) | [] | | Arguments to pass to the executable, in the same form you would type them (`--flag=value` or a positional value). `$VAR` references are expanded against the parent's environment. The parent's environment always reaches the executable; values listed here override what it would otherwise inherit for the arguments they set. |
| `cmd` | `string` | | | The command to execute. One of `cmd` or `ref` must be set. |
| `if` | `string` | | | An expression that determines whether the executable should run, using the Expr language syntax. The expression is evaluated at runtime and must resolve to a boolean value. The expression has access to OS/architecture information (os, arch), environment variables (env), stored data (store), and context information (ctx) like workspace and paths. For example, `os == "darwin"` will only run on macOS, `len(store["feature"]) > 0` will run if a value exists in the store, and `env["CI"] == "true"` will run in CI environments. See the [Expr documentation](https://expr-lang.org/docs/language-definition) for more information. |
| `interpreter` | [ExecutableExecInterpreter](#executableexecinterpreter) | | | The interpreter used to run `cmd` for this step. Defaults to `sh`. Only applies to `cmd`; a `ref` uses the referenced executable's own interpreter. |
| `name` | `string` | | | A human-readable label for this step, used for display purposes. |
| `ref` | [ExecutableRef](#executableref) | | | A reference to another executable to run in serial. One of `cmd` or `ref` must be set. |
| `ref` | [ExecutableRef](#executableref) | | | A reference to another executable to run in parallel. One of `cmd` or `ref` must be set. |
| `retries` | `integer` | 0 | | The number of times to retry the executable if it fails. |

### ExecutableParallelRefConfigList
Expand Down Expand Up @@ -263,7 +263,7 @@ Makes an HTTP request.
| Field | Type | Default | Required | Description |
| ----- | ---- | ------- | :------: | ----------- |
| `args` | [ExecutableArgumentList](#executableargumentlist) | | | |
| `body` | `string` | | | The body of the request. |
| `body` | `string` | | | The body of the request. `$VAR` references are expanded first. If the result is a JSON object or array, it is sent as written. Otherwise it is evaluated as an Expr expression that must produce a string, which lets you build a body from values that need escaping: `'{"prompt":' + toJSON(env["PROMPT"]) + '}'`. |
| `headers` | `map` (`string` -> `string`) | map[] | | A map of headers to include in the request. |
| `logResponse` | `boolean` | false | | If set to true, the response will be logged as program output. |
| `method` | `string` | GET | | The HTTP method to use when making the request. |
Expand Down Expand Up @@ -305,7 +305,7 @@ Configuration for a serial executable.

| Field | Type | Default | Required | Description |
| ----- | ---- | ------- | :------: | ----------- |
| `args` | `array` (`string`) | [] | | Arguments to pass to the executable. |
| `args` | `array` (`string`) | [] | | Arguments to pass to the executable, in the same form you would type them (`--flag=value` or a positional value). `$VAR` references are expanded against the parent's environment. The parent's environment always reaches the executable; values listed here override what it would otherwise inherit for the arguments they set. |
| `cmd` | `string` | | | The command to execute. One of `cmd` or `ref` must be set. |
| `if` | `string` | | | An expression that determines whether the executable should run, using the Expr language syntax. The expression is evaluated at runtime and must resolve to a boolean value. The expression has access to OS/architecture information (os, arch), environment variables (env), stored data (store), and context information (ctx) like workspace and paths. For example, `os == "darwin"` will only run on macOS, `len(store["feature"]) > 0` will run if a value exists in the store, and `env["CI"] == "true"` will run in CI environments. See the [Expr documentation](https://expr-lang.org/docs/language-definition) for more information. |
| `interpreter` | [ExecutableExecInterpreter](#executableexecinterpreter) | | | The interpreter used to run `cmd` for this step. Defaults to `sh`. Only applies to `cmd`; a `ref` uses the referenced executable's own interpreter. |
Expand Down
Loading