[jeffpignataro]

Summary

#332 disabled remote execution for the apko_image build action (no-remote-exec) to work around #331. This PR fixes the underlying incompatibilities so the action runs on remote executors again, and removes the opt-out. It also un-breaks setups that run --spawn_strategy=remote with no local fallback, which the no-remote-exec requirement made impossible (as noted in the #332 discussion).

Root cause of #331: apko writes into its --cache-dir even with --offline — it creates directories on cache lookup and expands packages into the cache (os.MkdirAll in pkg/apk/apk/package_getter.go). Bazel supplies the prepopulated cache tree artifact as a read-only action input, which remote executors enforce strictly; local sandboxes keep directories writable, which is why this only surfaced on RBE (unable to create cache directory ...: permission denied).

Changes:

• The action now copies the cache input into a writable scratch directory (mktemp -d) and points --cache-dir at it, so it works on read-only input trees. The flag is passed at the same precedence as before (after user args).
• HOME gets a writable fallback when the executor doesn't provide one (HOME="${HOME:-$SCRATCH/home}"); when the environment has HOME, behavior is unchanged.
• The apko_image and apko_show_config actions are bound to the resolved apko toolchain via toolchain = "@rules_apko//apko:toolchain_type", so the execution platform always matches the platform the apko binary was resolved for — under Bazel 9's platform ordering an executor could otherwise receive a wrong-OS binary.
• Removed execution_requirements = {"no-remote-exec": "1"}. Users who need local execution can still use tags = ["no-remote-exec"] or --strategy.
• New opt-in use_default_shell_env attribute on apko_image (default True, the existing behavior) for users who want a hermetic action environment. Existing targets are unaffected: the attribute name was previously rejected at load time, so no current usage can change meaning, and the flag only alters the action's environment — verified via aquery that command lines are byte-identical either way.

Test plan

• bazel test //..., all examples/, and e2e/smoke pass locally; e2e/smoke gains a lock_hermetic_env target covering use_default_shell_env = False.
• Remote execution verified against a Buildbarn-based remote executor from a macOS host with linux-amd64 workers: both e2e/smoke image targets built successfully with the apko actions executing remotely (3 processes: 1 internal, 2 remote). Repro shape: bazel build //:lock //:lock_hermetic_env --remote_executor=<endpoint> --extra_execution_platforms=<linux platform whose exec_properties match the worker pool>.
• Happy to follow up with an RBE CI job (e.g. BuildBuddy free tier) to keep this from regressing, per the maintainer request in fix: disable remote execution for apko_image build action #332.

[xnox]

@jeffpignataro - do you need to do additional testing before taking this out of draft?

[jeffpignataro]

@jeffpignataro - do you need to do additional testing before taking this out of draft?

I was just going to have @thesayyn review it on our end before marking it ready for review. I appreciate the proactive approval!

[xnox]

Seems to fail with a minimal remote executor. Let me show you.

[xnox]

I wonder if it is a bug in rules_apko or apko itself that it wants writable cache, and seems to unable to handle read-only cache. It sounds like a useful property, and with stock apko we do test this I think as part of regular image building but that is done with direct chainguard container, will check.

Separately instead of inventing shell, there seem to be utilities/binaries that can be used or something better basically.

[xnox]

Please see:

• ci: exercise apko_image remote execution against a local hermetic RBE worker jeffpignataro/rules_apko#1
• https://github.com/xnox/rules_apko/actions/runs/32969213066/job/98178726046

[xnox]

Data point from testing this against a different kind of executor, plus a suggestion. Not a blocker for this PR — as far as I can tell #379 is correct and does exactly what it claims.

I ran it against actiond, a local Remote Execution API worker that boots a Linux VM on the machine and runs each action inside an empty chroot in it. It works on GitHub-hosted ubuntu-24.04 runners with no account or secrets, which makes it a candidate for the RBE CI job discussed in #332 (x86_64 only — GitHub's arm64 Linux runners have no /dev/kvm, confirmed by GitHub staff, and macOS arm64 can't nest under Apple's Virtualization Framework).

This PR's core change verifies independently. Dropping no-remote-exec genuinely makes the apko action remote-eligible — Bazel schedules it ([Sched] Action examples/wolfi-base/wolfi-base) where before it never left the host. The read-only---cache-dir diagnosis also matches what I see.

What it surfaced. Two run_shell sites assume coreutils exist in the executor image. That holds on Buildbarn/BuildBuddy and doesn't on a minimal executor:

1. copy_to_workdir in apko/private/apko_config.bzl (mkdir -p … && cp -rfL …, untagged, so it goes remote) — unchanged by this PR:

ERROR: examples/wolfi-base/BUILD.bazel:6:11: Staging examples/wolfi-base/+apko+examples_wolfi_base/lockfile_copy
  failed: (Exit 127): bash failed: error executing CopyToWorkdir command
/bin/bash: line 2: mkdir: command not found

2. The apko action itself, once staging is forced local so it can be reached:

ERROR: ... Action examples/wolfi-base/wolfi-base failed: (Exit 127): bash failed: error executing Action command
  (cd /home/runner/.bazel/execroot/_main && \
  exec env - PATH=/bin:/usr/bin:/usr/local/bin \
  /bin/bash -c 'set -e
SCRATCH="$(mktemp -d)"
...
: line 2: mktemp: command not found

The runner has mktemp on that PATH, so "command not found" is confirmation the action really executed in the guest. Full run.

The mild irony is that the fix here is itself written in coreutils — mktemp, mkdir, find, ln, rm — so the apko action's dependency on the executor image grew from nothing to five binaries. That's a fine trade against conventional RBE, it just moves in the opposite direction for minimal ones.

Concrete way to close it, if it's of interest. aspect_bazel_lib is already a bazel_dep here, and its MODULE.bazel registers these toolchains in the non-dev section, so they're available with no new dependency and no registration:

• copy_to_directory replaces copy_to_workdir outright — it already does "list of files into one tree artifact with workspace-relative path remapping", which is what that function hand-rolls. Via ctx.actions.run it drops bash from those actions entirely.
• coreutils (@aspect_bazel_lib//lib:coreutils_toolchain_type, provider CoreUtilsInfo.bin) supplies mktemp/mkdir/ln/rm as a declared tool input rather than inheriting them from the image. The Linux assets are *-unknown-linux-musl, i.e. statically linked, so they need nothing from the executor.

One gap: uutils doesn't ship find (that's findutils), so the find … -exec ln -s mirror added in the last commit has no direct translation — copy_directory into the scratch dir is probably the better shape there anyway, being one purpose-built process instead of a fork per directory entry.

The general form is that anything a run_shell action calls has to be a declared tool input to survive an executor that doesn't ship a userspace. Happy to send this as a follow-up PR against this branch rather than expanding scope here. I've also opened jeffpignataro#1 with the actiond workflow itself if the CI angle is useful — it's currently red by design, as a reproducer rather than a gate.

🤖 Summary generated with Claude Code

[xnox]

@thesayyn @jeffpignataro could you please review #380 ?

[jeffpignataro]

Closing in favor of #380