Skip to content

fix: run apko_image on remote executors - #411

Merged
shimmerjs merged 1 commit into
mainfrom
con-3638-drop-no-remote-exec
Oct 27, 2026
Merged

shimmerjs merged 1 commit into
mainfrom
con-3638-drop-no-remote-exec

Conversation

@shimmerjs

Copy link
Copy Markdown
Contributor

apko_image still pins its build action to local execution (no-remote-exec), so it never runs on a remote executor. Remote execution worked before #332, which added the pin as a stopgap for #331: apko failed to build --offline from the read-only cache that Bazel provides as an input.

apko no longer has that problem. Since v1.4.3 it verifies cached packages without writing to the cache directory, and this module pins v1.4.6. This deletes the pin.

Evidence

On darwin/arm64 I built the cache with this module's own repository rules (bazel build //:lock in e2e/smoke), made the copy read-only with chmod -R a-w, and ran the invocation apko_image uses: apko build apko.yaml lock:latest out --vcs=false --lockfile=apko.lock.json --cache-dir=cache --offline.

apko result
v1.2.11 (this module's pin when #332 landed) exit 1, mkdir .../expand-apk...: permission denied
v1.4.2 exit 1, failed installation from lockfile ... expanding {busybox ...}
v1.4.6 (current pin) exit 0, cache byte-identical before and after

v1.4.6 also exits 0 with HOME unset (env -i), so the action needs no writable home either.

bazel test //... in e2e/smoke on Bazel 9.1.0: 4/4 pass.

The new CI step runs bazel aquery //... --output=jsonproto in e2e/smoke and fails if any action has a no-remote-exec execution requirement. Against this tree it passes; against main (pin present) the same query reports the requirement.

Not verified

I have not run this on a real remote executor. The darwin runs show apko handles a read-only cache; they do not show the whole action working on a remote worker. The copy_to_workdir actions still call mkdir and cp from the executor's PATH, so an executor with no userspace would still fail there (#380 and #409 address that with a coreutils toolchain). A conventional RBE image has both.

apko_image carried a no-remote-exec requirement, added as a stopgap for
apko failing to build offline from a read-only cache (#331). apko has
handled that since v1.4.3 and this module pins v1.4.6. Checked on
darwin with the cache from this module's lock repository, made
read-only: v1.4.2 fails with permission denied, v1.4.6 builds and
leaves the cache unchanged, with HOME set or unset.

Add a CI step that fails if any action in e2e/smoke carries
no-remote-exec, so the requirement cannot come back unnoticed.

@codysoyland codysoyland left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by unblok using Codex · gpt-6-astra · medium reasoning effort

LGTM. Removing the execution restriction is supported by the current apko pin.

What I verified

  • apko/private/apko_image.bzl:104-112 no longer forces local execution; declared inputs, tools, outputs and --offline remain intact.
  • MODULE.bazel:20 pins apko v1.4.6. Its pkg/apk/expandapk/verify.go:165-176 falls back to os.TempDir() when a private data file cannot be created in the cache directory.
  • .github/workflows/ci.yaml:112-115 checks the smoke action graph for no-remote-exec; the smoke CI jobs passed on Bazel 7.6.1, 8.3.0, 8.6.0 and 9.1.0 at 49322ae.
  • e2e/smoke/BUILD:6-18 includes an actual apko_image build target. Real remote execution remains unverified; staging still needs mkdir/cp on the executor, as disclosed in the PR.

Security

  • ci: cleared. Workflow triggers and contents: read permissions are unchanged; the new step uses a literal command in the existing smoke workspace. A fresh reviewer confirmed the clearance.

@shimmerjs
shimmerjs merged commit 23806c6 into main Oct 27, 2026
25 checks passed
@shimmerjs
shimmerjs deleted the con-3638-drop-no-remote-exec branch October 27, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants