Feat/enclave job sandbox - #9485
Open
bitsofsteve wants to merge 4 commits into
Open
Conversation
Job code currently runs as a plain subprocess: root, same container as the supervisor, same network, inheriting the full environment. syft-restrict can constrain Python but not the compiled C++/CUDA that some private code ships as, so there is no mechanism today that prevents such code making network requests. Add syft_job.sandbox, which drops privileges, sets no_new_privs and installs a seccomp filter denying socket creation, then execs the job. Both restrictions are one-way -- the kernel offers no operation to remove a seccomp filter -- so the code being launched cannot lift them, and because the filter is enforced on syscall entry and survives execve it binds compiled binaries, not just Python. Denying every address family rather than only AF_INET is deliberate: a socket's network namespace is fixed at creation, so a process that can open a unix socket can be handed an already-connected one over SCM_RIGHTS by a co-resident helper. io_uring is denied for the same reason -- it performs network I/O without the syscalls a classic filter would see. Wired into both Popen sites behind SYFT_JOB_SANDBOX (off/on/require), defaulting to off: this runner also executes jobs on data owners' own machines, where sandboxing is neither expected nor always possible. When enabled the job also receives an allowlisted environment rather than the runner's full one, which today leaks bootstrap secrets. Not yet usable in production: run.sh as generated performs venv creation and dependency installation before the entrypoint, and uv itself needs local sockets (its async runtime uses a UnixStream for signal handling), so sandboxing the whole script breaks the install. Splitting install from execution is a prerequisite and follows in the next commit.
The sandbox denies every address family, including AF_UNIX, so it cannot wrap
run.sh as generated: that script builds a venv and installs dependencies before
reaching the entrypoint, and uv dies without local sockets -- its async runtime
opens a UnixStream for signal handling, which fails long before any network call
is attempted.
Rebuild the two phases from the submission's declared entrypoint and
dependencies instead of executing the submitted run.sh. Phase A installs with
the network available and unsandboxed; phase B runs only the entrypoint under
the lockdown. Because phase A runs unsandboxed and with network, it must not
execute code the submitter chose, so:
- syft-client is installed from this runner's own install source, which is
part of the attested enclave image, rather than from whatever the submission
declared. It may legitimately be a local path, so it is exempt from the
wheels-only rule.
- declared dependencies are installed --only-binary=:all:, since building a
source distribution runs its build backend; specs naming local paths or VCS
URLs are refused outright rather than built.
Bash submissions carry no entrypoint metadata to split on, so they are sandboxed
wholesale and will fail if they install anything -- acceptable while the flag is
opt-in.
Adds an integration test pair that runs a probe job through the real runner and
asserts it can open a socket with the sandbox off and cannot with require, so a
silently degraded sandbox fails the suite rather than passing quietly.
syft-job and syft-enclave suites: 143 passed with the sandbox off, 143 passed
with it on.
Only root can change user id, so apply_lockdown skipped the privilege drop when invoked as an ordinary user and installed just the seccomp filter. That is the less valuable half: the network is blocked, but the job keeps the invoking user's file access, so it can still read the Drive credential and modify the runner's own code. Worse, it happened silently, and under SYFT_JOB_SANDBOX= require -- where an operator has asked for the full guarantee. Make it an error by default. Callers wanting best-effort behaviour must pass --best-effort explicitly, which the runner does for "on" (documented as best-effort) and pointedly does not for "require". Existing tests missed this because they pass --uid $(id -u), so the requested uid already matched and no drop was needed. Added tests that request a different uid as a non-root user. Validated as root in the published enclave image, which is the configuration that actually ships: baseline uid=0 caps=a80425fb nnp=0 network ALLOWED token READABLE code WRITABLE sandboxed uid=1500 caps=0 nnp=1 network BLOCKED token BLOCKED code read-only syft-job and syft-enclave suites: 147 passed with the sandbox off and on.
A sandboxed job could not do useful work: it runs as an unprivileged account, so
it could neither create its virtualenv nor write outputs/ in a root-owned job
tree, nor read the datasets it was approved for -- the datasite lives under
/root, which is 0700 because the container runs as root and that is its home.
Hand the job's own working tree to the sandbox account, and open traversal and
read on the datasite without transferring ownership. Neither weakens the
lockdown: the Drive token and the runner's own source stay root-owned and
unreachable. It is a no-op when not running as root, which covers tests and data
owners' own machines.
Also add the syftjob account (uid 1500) to the enclave image, and allow
SYFT_JOB_SANDBOX{,_UID,_GID} through the Confidential Space env-override policy,
without which the sandbox cannot be enabled on a deployed enclave at all.
Verified as root inside an image built from this branch, driving the real
submit -> approve -> run -> distribute flow with SYFT_JOB_SANDBOX=require:
JOB REPORTED: uid=65534 user=nobody caps=0000000000000000 network=BLOCKED
and the trust split fired as intended, refusing the submitted local-path
dependency spec in favour of the runner's own install source.
147 passed with the sandbox off and on.
Contributor
PR title does not follow the required formatYour title must follow this pattern: Example titles:
Allowed types: Just edit your PR title above to fix this. The check will re-run automatically. See the PR guidelines for full details. |
pjwerneck
reviewed
Aug 3, 2026
pjwerneck
left a comment
pjwerneck
left a comment
Collaborator
There was a problem hiding this comment.
I'm not familiarized with the constraints of the enclaves and how we use them, but I left three comments on issues that could be relevant.
| # by default, since the container runs as root and that is its home. | ||
| for parent in list(syftbox_folder.parents)[:-1]: | ||
| try: | ||
| os.chmod(parent, os.stat(parent).st_mode | 0o011) |
Collaborator
There was a problem hiding this comment.
I couldn't find any code restoring this permission change to the original 0700 after the job. Is that intended to be permanent? is the tree ephemeral?
| os.chmod(parent, os.stat(parent).st_mode | 0o011) | ||
| except OSError: | ||
| pass | ||
| for root, dirs, files in os.walk(syftbox_folder): |
Collaborator
There was a problem hiding this comment.
This walk is granting read for the whole syftbox folder, not just the approved datasets. Are the enclaves single-tenant?
| for root, dirs, files in os.walk(submission_dir): | ||
| for name in dirs + files: | ||
| path = os.path.join(root, name) | ||
| if not os.path.islink(path): |
Collaborator
There was a problem hiding this comment.
This walk over submission_dir skips symlinks, but the one below, over syftbox_folder does not. Is that correct? Because if enclaves are not ephemeral, someone could use multiple job executions to intentionally or accidentally create symlinks that go where they shouldn't.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
spike