Skip to content

feat(exec): add interpreter field with python support - #439

Merged
jahvon merged 1 commit into
mainfrom
feat/exec-interpreter
Aug 27, 2026
Merged

feat(exec): add interpreter field with python support#439
jahvon merged 1 commit into
mainfrom
feat/exec-interpreter

Conversation

@jahvon

@jahvon jahvon commented Aug 27, 2026

Copy link
Copy Markdown
Member

Part 1/5 of a stack adding Python execution to flow. Base: main.

Summary

flow could not run Python. exec.cmd is always parsed by the in-process mvdan.cc/sh POSIX interpreter, and exec.file only dispatched .sh / .bat / .cmd / .ps1 — a .py file was parsed as shell and failed.

This adds an interpreter field to the exec type, and teaches exec.file to dispatch .py by extension.

executables:
  - verb: run
    name: report
    exec:
      interpreter: python
      cmd: |
        import json, sys
        print(json.dumps({"python": sys.version_info[:2]}))

A .py file needs no interpreter at all — the extension implies it. Setting it explicitly overrides the extension.

Notable Changes

  • interpreter is a pointer. go-jsonschema runs with --only-models, so it emits plain structs with no UnmarshalJSON: neither the schema default nor its enum is ever applied in Go. Unset therefore stays distinguishable from an explicit sh (an unset interpreter is what lets a file extension decide), and the enum is re-checked in Executable.Validate the way ExecContainer.Validate re-checks its runtime.
  • Interpreter resolution prefers a project virtualenv, so a script gets the dependencies its repo installed: FLOW_PYTHON_BIN$VIRTUAL_ENV<workspace>/.venvpython3/python on PATH. The workspace root arrives via FLOW_WORKSPACE_PATH, already in the resolved env, so no run signature had to widen. An override that does not resolve fails rather than silently falling back.
  • Windows prefers python over python3, because python3.exe there is usually the Microsoft Store alias stub that prints an ad and exits non-zero.
  • Inline code runs from a 0600 temp file, not python -c — keeps user code out of the process table and preserves traceback line numbers.
  • Sets PYTHONUNBUFFERED (flow pipes stdout to a log writer, and CPython block-buffers to a pipe, so a long run would otherwise look hung) and PYTHONDONTWRITEBYTECODE (keeps __pycache__ out of workspaces). Both overridable.

Testing

  • Unit tests for resolution order, env defaults, temp-script handling, the type helpers, and runner routing.
  • tests/python_exec_e2e_test.go covers real execution, params through the environment, traceback line numbers, .py extension inference, and the not-found error.
  • CI pins an interpreter with actions/setup-python and adds a cross-platform test python-script job that runs a real .py through the built binary, asserting the env contract — so a resolution regression fails CI rather than silently degrading to shell.
  • flow validate passes (generate → lint → unit + e2e → schema validation), and generate produces no diff.

🤖 Generated with Claude Code

https://claude.ai/code/session_01R328pa3FUUfga4gYah1iQi

@codecov

codecov Bot commented Aug 27, 2026

Copy link
Copy Markdown

@jahvon
jahvon force-pushed the feat/exec-interpreter branch from 5e24130 to d91b0b0 Compare August 27, 2026 04:28
jahvon added a commit to flowexec/action that referenced this pull request Aug 27, 2026
Blocks flowexec/flow#446, which needs this to run its CI tasks with the
flow binary under review.

# Summary

Adds a `flow-binary` input so a caller can run its executables with a
flow binary it just built, instead of one installed from a release or
built from `main`.

```yaml
- run: go build -o ./bin/flow .
- uses: flowexec/action@v1
  with:
    executable: 'test unit'
    flow-binary: ./bin/flow
```

# Why

Without it, a repository whose CI runs flow tasks is always testing a
flow that predates the code under review. That is a silent gap in
general, and a hard block when a task depends on a flow feature that has
not shipped yet.

It surfaced concretely in flowexec/flow#439, which adds Python
execution. A `.execs` target running a `.py` script failed CI on every
platform:

```
Executing: flow test python-script
unable to parse file - 14:1: `foo(` must be followed by `)`
```

`flow-version: main` clones flow from GitHub and builds `main`, so the
branch adding `.py` support was discarded and the released flow parsed
the Python file as shell. No change to that PR could have made the job
pass — the capability has to be on `main` before CI can use it, which is
circular.

# Notable Changes

- `flow-binary` takes precedence over `flow-version`.
- The download cache step is skipped when it is set — there is nothing
to restore.
- A missing path **fails loudly** rather than falling back to a
download, so a typo cannot silently reintroduce the gap this exists to
close.
- Handles the Windows install path (`$HOME/bin/flow.exe`) alongside the
Unix one, matching the existing `main`-build branch.
- No behavior change when the input is unset.

# Testing

The action has no CI workflows to add a case to. Verified:

- `action.yaml` parses as valid YAML; `bash -n scripts/install-flow.sh`
is clean.
- The new branch is the first condition, so the cached / `latest` /
`main` / pinned-version paths are untouched when `flow-binary` is empty.
- End-to-end verification is flowexec/flow#446, which consumes this
input across its CI matrix (ubuntu, macos, windows). Worth landing this
and moving the `v1` tag before that PR goes green.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jahvon
jahvon force-pushed the feat/exec-interpreter branch from d91b0b0 to 682ce03 Compare August 27, 2026 05:48
@jahvon jahvon added the test:windows Run Windows PR validation label Aug 27, 2026
jahvon added a commit that referenced this pull request Aug 27, 2026
…451)

# Summary

Four specs asserted Unix-only behavior and failed on Windows. They have
been latent for a while: `windows-ci` only runs behind the
`test:windows` label, and no PR had carried it since the code that
introduced them. Labelling #439 ran them for the first time.

| Spec | Why it failed on Windows |
|---|---|
| `expands and mounts workspace-relative and absolute volumes` |
`/opt/data` is not absolute under `filepath.IsAbs` |
| `expands volume host paths [absolute]` | same |
| `writeEnvFile writes sorted 0600 entries` | mode read back as `0666` |
| `BoltDataStore` — **entire suite** | `ERROR_INVALID_NAME` in
`BeforeEach` |

**Volume paths.** `expandVolumeHost` gates on `filepath.IsAbs`, which
rejects `/opt/data` on Windows — there an absolute path needs a drive
letter. Two specs hardcoded it as the *host* side of a volume. They now
build the host path for the platform. The container side stays Unix,
since container paths are always Linux paths, and
`ExecContainerVolume.Parts` already handles the drive-letter colon.

**File mode.** Go maps Unix permission bits onto ACLs on Windows, so
`0600` is not expressible and reads back as `0666`. The assertion is
skipped there and still enforced everywhere it means something. The
content assertion is unchanged.

**Store suite.** This one took out *every* spec in `pkg/store`, not just
the offending one. The database filename was derived from the spec name:

```go
path := filepath.Join(GinkgoT().TempDir(), fmt.Sprintf("test_%s.db", GinkgoT().Name()))
```

One spec is named `... (running -> terminal)`, and `>` is not a legal
Windows filename character — so `BeforeEach` failed with
`ERROR_INVALID_NAME` (`0x7b`). `TempDir` is already unique per spec, so
the suffix bought nothing and is dropped.

# Scope

**Test-only.** The production paths were already Windows-capable —
`Parts` handles drive letters, and container paths are correctly
validated with Unix semantics. This fixes assertions, not behavior.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jahvon
jahvon force-pushed the feat/exec-interpreter branch from 682ce03 to c5c6b3f Compare August 27, 2026 06:24
@jahvon
jahvon force-pushed the feat/exec-interpreter branch from c5c6b3f to f59092a Compare August 27, 2026 06:44
Adds an `interpreter` field to the `exec` executable type so a command can run
as Python instead of shell, and teaches `exec.file` to dispatch `.py` by
extension.

Interpreter is a pointer because go-jsonschema runs with `--only-models`: it
emits plain structs with no UnmarshalJSON, so neither the schema `default` nor
its `enum` is ever applied in Go. Unset therefore stays distinguishable from an
explicit `sh` (an unset interpreter lets a file extension decide), and the enum
is re-checked in Executable.Validate the way ExecContainer.Validate re-checks
its runtime.

Interpreter resolution prefers a project's virtualenv over bare system python,
so a script gets the dependencies its repository installed: FLOW_PYTHON_BIN,
then $VIRTUAL_ENV, then <workspace>/.venv, then python3/python on PATH. The
workspace root arrives via FLOW_WORKSPACE_PATH, already in the resolved env, so
no run signature had to widen. Windows prefers `python` over `python3` because
`python3.exe` there is usually the Microsoft Store alias stub.

Inline code runs from a 0600 temp file rather than `python -c`, keeping user
code out of the process table and preserving traceback line numbers.
PYTHONUNBUFFERED and PYTHONDONTWRITEBYTECODE are set so output streams live and
workspaces stay free of __pycache__.

CI pins an interpreter with actions/setup-python and runs a cross-platform
`test python-script` integration target so a regression in resolution fails
rather than silently degrading to shell.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R328pa3FUUfga4gYah1iQi
@jahvon
jahvon force-pushed the feat/exec-interpreter branch from f59092a to b23bdf8 Compare August 27, 2026 06:49
@jahvon
jahvon merged commit c4070d4 into main Aug 27, 2026
23 checks passed
@jahvon
jahvon deleted the feat/exec-interpreter branch August 27, 2026 13:52
jahvon added a commit that referenced this pull request Aug 27, 2026
**Part 2/5.** Stacked on #439 — review that first; this PR's diff is
only its own change.

# Summary

Exposes Python execution to agents as a first-class MCP tool, alongside
`run_command`. This is the point of the stack: flow already acts as an
assistant's shell, and now it acts as their Python runtime too — same
workspace env and secrets, same captured logs, same attributable history
entry.

```
run_python({ code: "import pandas as pd\nprint(df.describe())", label: "summarize csv", dir: "/repo" })
```

Also adds the `--interpreter` flag on `flow exec` that backs it.

**Notable Changes**

- **`run_python` is its own tool rather than a parameter on
`run_command`.** Agents select tools by name, so a tool called
"run_command" is not what gets reached for when the task is Python. It
also sidesteps a real constraint: `run_command`'s multi-command form
builds serial/parallel steps, which carry no interpreter of their own.
- For that same reason the CLI rejects `--interpreter` with multiple
`--cmd` values, with a usage error. **Part 4 of this stack lifts that
restriction** once step configs gain the field — it is a deliberate
interim guard, not an oversight.
- The tool is a thin argv layer like its siblings, reusing
`runTransientTool`, `ExecutionOutput`, and the existing
provenance/progress plumbing.
- Updates the embedded MCP server instructions and
`docs/guides/ai-tools.md` so the tool ladder names it.
- Allowlists `mcp__flow__run_python` in this repo's own
`.claude/settings.json` and adds it to the `flow-context` skill, so the
agents working in this repo reach for it instead of shelling out to
`python -c`.

# Testing

- MCP tests assert the exact argv, workspace/sync forwarding, that
multi-line code survives as a single argument (line structure has to
stay intact or tracebacks point at the wrong line), and both
empty/missing `code` error paths. `run_python` added to the
registered-tool and output-schema assertions.
- E2E coverage for the flag, the unknown-interpreter rejection, and the
multi-command guard.
- `flow validate` passes; `generate` produces no diff. (Completion
scripts are unaffected — cobra generates them to query the binary at
runtime rather than embedding flag lists.)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01R328pa3FUUfga4gYah1iQi

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
jahvon added a commit that referenced this pull request Aug 27, 2026
# Summary

Reworks flowexec.io: a real design system instead of a 57-line override
file, a way to ask
questions the guides don't answer, and the SEO/social tags the site
never had.

Almost entirely docs and `tools/docsgen`. The one runtime change is a
second commit fixing three
flag usage strings that made `flow --help` print types that do not exist
— see the last section.

---

## Ask the codebase

`⌘K` local search is untouched and still instant. Alongside it, an **Ask
AI** button (`⌘I`) opens a
panel that queries DeepWiki's public MCP endpoint directly from the
browser — no proxy, no new
deploy surface, no secrets. The existing search modal also grows a
bridge row that hands the current
query across.

## Reference docs (`tools/docsgen`)

Cobra's markdown is a flat dump, and the **flag listings were the worst
of it** — fixed-width
columns that overflowed the content column and clipped their own
descriptions off the right edge.
A new post-processing pass rewrites them into tables, gives fences a
language, and turns the
command name into a real page title instead of an h2 with a section
rule.

The parser is deliberately conservative: anything it cannot parse falls
back to a verbatim code
block rather than a mangled table. It handles Cobra's backtick-derived
placeholders that contain
spaces and wrapped description lines.

## SEO

There were no Open Graph tags, no Twitter card, no canonical, and no
social image, so a link to
flowexec.io previewed as a bare URL.

## Guides

`integrations.md` split into **Containers** (the `exec.container` field
with its full option table,
plus running the CLI from the image) and **GitHub Actions** (inputs and
outputs read from
`action.yaml`). Both sit under a new Integrations group alongside AI
Tools. All inbound links and
`llms.txt` repointed.

Containers covers the Python support from #439#443 that landed while
this was in flight: the
interpreter-dependent `entrypoint` default, and the host Python env vars
that are dropped at the
container boundary regardless of `inheritEnv`.

Also: nav collapsed from five items to three (the two reference sections
became one menu), a
Dockery Labs footer, and `docs/public/demo.gif` deleted — 3 MB,
orphaned, nothing referenced it.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:windows Run Windows PR validation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant