feat(exec): run python inside containers - #441
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
jahvon
force-pushed
the
feat/python-container
branch
from
August 27, 2026 04:28
dce32d4 to
60cd979
Compare
jahvon
force-pushed
the
feat/python-container
branch
from
August 27, 2026 05:48
60cd979 to
c3b0edb
Compare
jahvon
force-pushed
the
feat/python-container
branch
from
August 27, 2026 06:24
c3b0edb to
56449b2
Compare
jahvon
force-pushed
the
feat/python-container
branch
from
August 27, 2026 06:44
56449b2 to
3782ca3
Compare
Combines `interpreter: python` with the `container:` block so a pinned Python toolchain needs no local install. Inline code is bind-mounted read-only as a script and passed via spec.Script rather than spec.Cmd: buildRunArgs maps Cmd to `<entrypoint> -c <code>`, which would put user code in the process table and cost traceback line numbers. The existing "script lives outside every mount" branch already did exactly this for files, so the generated temp script reuses it. The default entrypoint now follows the interpreter — sh for a shell command, python3 for a python one — resolved in the runner rather than on ExecContainer, keeping the types package unaware of how flow launches containers. An explicit container.entrypoint still wins, including the empty form. Host interpreter discovery deliberately stops at the container boundary: VIRTUAL_ENV, PYTHONPATH, PYTHONHOME, and FLOW_PYTHON_BIN are dropped from the container environment, since those host paths either do not exist inside it or resolve to an unrelated mounted directory. Verified through spec and argv unit tests; the live container run is covered by the e2e case, which needs a runtime and so only executes in CI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R328pa3FUUfga4gYah1iQi
jahvon
force-pushed
the
feat/python-container
branch
from
August 27, 2026 06:49
3782ca3 to
ff6c97c
Compare
jahvon
added a commit
that referenced
this pull request
Aug 27, 2026
**Part 4/5.** Stacked on #441. # Summary Inline `cmd` steps in `serial`/`parallel` executables now take their own `interpreter`, so one workflow can mix shell and Python without splitting into separate executables. ```yaml serial: execs: - cmd: ./fetch-data.sh - cmd: | import json print(json.load(open("data.json"))["total"]) interpreter: python ``` **Notable Changes** - Both step configs `$ref` the same `ExecInterpreter` definition the `exec` type uses, so there is one enum rather than three parallel ones. (Part 1 hoisted it to a top-level definition for exactly this.) - A step that omits `interpreter` runs under the shell as before; a `ref` step ignores the field — the referenced executable brings its own. - **This lifts the interim restriction from #440**: `--interpreter` now applies to every `--cmd` in an invocation, in serial and parallel mode alike. - `ExecutableForCmd` takes the interpreter as a new parameter (5 call sites). Its unused `int` parameter is left alone to keep the diff to the one concern. # Testing - E2E: a serial executable mixing a shell step and a Python step, each running under its own interpreter; a step without an interpreter staying on the shell (asserted via `echo`, a shell builtin, so it only passes if the step really stayed there); and `--interpreter` applied across a multi-command batch. - The e2e case that previously asserted the multi-command rejection is replaced with one asserting it now works. - `flow validate` passes; `generate` produces no diff. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01R328pa3FUUfga4gYah1iQi Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 3/5. Stacked on #440.
Summary
Combines
interpreter: pythonwith the existingcontainer:block, so a pinned Python toolchain needs no local install.Notable Changes
spec.Script, neverspec.Cmd.buildRunArgsmapsCmdto<entrypoint> -c <code>, which would put user code in the process table and cost traceback line numbers — the same trade-off Part 1 avoided on the host. The existing "script lives outside every mount" branch already did exactly this for files, so the generated temp script reuses it andinternal/services/run/container.goneeded no change at all.shfor a shell command,python3for a Python one. Resolved in the runner rather than onExecContainer, keeping the types package unaware of how flow launches containers. An explicitcontainer.entrypointstill wins, including the empty form (which, with Python, only works if the image's ownENTRYPOINTis an interpreter — documented).VIRTUAL_ENV,PYTHONPATH,PYTHONHOME, andFLOW_PYTHON_BINare dropped from the container environment: those host paths either do not exist inside it or, worse, resolve to an unrelated mounted directory.PYTHONUNBUFFEREDis set there too.ForceRemoveContainer, so a run abandoned by the runner's timeout goroutine does not leak it.Testing
Cmdempty,Scriptset to a.py, the script mount present and read-only, thepython3entrypoint default, explicit-entrypoint override, empty-entrypoint passthrough, that a shell command still getssh, and that host python env vars are dropped while unrelated ones survive.python:3.13-alpineimage, behind the suite's existing runtime guard.flow validatepasses;generateproduces no diff.Verification note: I have no container runtime on this machine, so the live container run was not executed locally — it is covered by the e2e case, which only executes where a runtime is available (CI). The spec and argv construction are verified by the unit tests above.
🤖 Generated with Claude Code
https://claude.ai/code/session_01R328pa3FUUfga4gYah1iQi