feat(run): add content-addressed source bundles - #36
Kevin Cho (chokevin) wants to merge 1 commit into
Conversation
|
Azure Pipelines: Some pipeline(s) encountered errors during trigger evaluation. |
|
Kevin Cho (@chokevin) this PR needs rebase |
| // MaxInputBytes is deliberately the same limit imposed on working_dir. | ||
| MaxInputBytes = 8 << 20 | ||
| DurableRoot = "/data/tau/source-bundles/sha256" | ||
| helperImage = "busybox:1.36.1@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662" |
There was a problem hiding this comment.
P0 This helper image bypasses the repository's MCR image policy (AGENTS.md: "Container images go through MCR"). Every real source-bundle submission creates this pod before the workload; clusters enforcing that policy reject or pull-block it, so staging aborts. Use an approved MCR-hosted helper image.
| target.mkdir(parents=True, exist_ok=True) | ||
| for info, normalized in members: | ||
| destination = (target / pathlib.PurePosixPath(normalized)).resolve() | ||
| if not destination.is_relative_to(target): |
There was a problem hiding this comment.
P1 Path.is_relative_to() requires Python 3.9, while the documented runtime contract only requires python3. Images on Python 3.8 raise an uncaught AttributeError in the bundle init container, so the workload never starts. Use a relative_to()/ValueError containment check, or explicitly require and validate Python 3.9+.
| } | ||
| return []string{ | ||
| "bash", "-c", | ||
| pythonPathPrefix + "exec " + shellQuote("./"+entrypoint) + argList.String() + extraFlagStr, |
There was a problem hiding this comment.
P1 This source-bundle path bypasses the existing shebang handling and directly executes the extracted file. A valid shebang entrypoint stored as mode 0644 is preserved as non-executable, so exec ./... fails with permission denied; a non-Python shebang in a .py file is also sent to Python. Restore shebang-based interpreter selection or stage executable mode consistently.
| return err | ||
| } | ||
| } | ||
| if err := stageRunSourceBundle(ctx, o.dryRun, runner, ns, pvcMount, request.Name, source); err != nil { |
There was a problem hiding this comment.
P1 This stages the archive only on the submission cluster before MultiKueue dispatch. The worker receives the workload object and path but not the archive bytes; with independently provisioned same-named PVCs, its init container cannot find the bundle and the run never starts. Reject this combination or stage/mirror the bundle on the selected worker; the Ray path has the same issue.
| moduleName := strings.TrimSuffix(o.ScriptName, ".py") | ||
| if o.SourceBundle != nil { | ||
| var err error | ||
| moduleName, err = entrypointModule(o.SourceBundle.Entrypoint) |
There was a problem hiding this comment.
P1 For a nested entrypoint such as pkg/train.py, this exports pkg.train, but the Tune driver loads it with bare __import__, which returns the top-level pkg. getattr(pkg, "train_func") then fails before Tune starts unless the package re-exports the function. Load it with importlib.import_module, or change the generated module contract.
2 participants
Summary
Add first-class content-addressed source bundles for direct
tau runJobs and RayJobs. Researchers can configure:Tau deterministically archives the local tree, stages it once at
/data/tau/source-bundles/sha256/<hex>.zipon the workspace data PVC, and records the digest/PVC/path in workload annotations, status, and lifecycle history. Job and Ray pods verify the exact digest and every ZIP member before use; Jobs safely extract into anemptyDir, while Ray uses the sharedfile://working-dir reference.The implementation fails closed for missing, corrupt, unsafe, or mismatched content; preserves safe executable bits; handles concurrent same-digest staging; excludes repository credentials and metadata; and never serializes source bytes into Kubernetes objects, Secrets, environment variables, or arguments. Client and server dry-runs validate and render without staging.
AI tooling assisted with implementation and adversarial review. All findings were independently reconciled and the resulting behavior was validated with the repository's existing test and build workflows.
Related issue
N/A
Validation
Compatibility and operational impact
run.working_dirbehavior and its fail-fast 64 KiB inline limit are unchanged.source_bundleis direct-run-only and mutually exclusive withworking_dir.storage.data_pvc, defaulting toblob-trainingwhen omitted. Multi-node RayJobs require aReadWriteManydata PVC.python3for bundle verification and Job extraction.pods/execcreate permissions for the short-lived staging helper. It gains no Secret, PVC-write-API, patch/update, or cluster-scoped permissions.Checklist