Skip to content

Generate Apple Container provisioning, images, and MCP transport (stack layer 2) - #56131

Draft
lpcox wants to merge 1 commit into
lpcox-apple-container-runtime-contractfrom
lpcox-apple-container-runtime-provisioning
Draft

Generate Apple Container provisioning, images, and MCP transport (stack layer 2)#56131
lpcox wants to merge 1 commit into
lpcox-apple-container-runtime-contractfrom
lpcox-apple-container-runtime-provisioning

Conversation

@lpcox

@lpcox lpcox commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Stack layer 2 of 2. Targets lpcox-apple-container-runtime-contract, not main.

Depends on gh-aw-firewall#7764 (the selectable Apple Container backend) and gh-aw-firewall#7768 (the external loopback MCP gateway upstream contract). Review 28c9251..HEAD to see only this layer.

Runtime provisioning

Four steps now run before AWF, in dependency order, so an unusable host costs seconds rather than a multi-gigabyte pull:

  1. Host preflight — self-hosted provenance, macOS 26+, native arm64, kern.hv_support=1, a usable launchd user domain, bash 4+, and a working Docker daemon with Compose. arm64 is checked via RUNNER_ARCH and uname -m, so a runner registered as ARM64 but executing under Rosetta is caught rather than trusted.
  2. CLI setup — verifies a preinstalled container CLI inside AWF's validated range (>=0.4.0 <1.0.0). Only when runtime-install is not false does it install the pinned apple/container release, verified by SHA-256 and by its Developer ID Installer: Apple Inc. - Containerization signature before installer ever runs, over non-interactive sudo. No latest path, no unsigned fallback, and the post-install version is re-checked against the window.
  3. Service startcontainer system start --enable-kernel-install. The flag is not optional: the default prompts on stdin, and a headless runner would block until the step is cancelled. The known headless launchd failure is detected and reported with the actual remediation instead of an opaque bootstrap error.
  4. Image pull — populates Apple Container's own image store, which docker pull cannot reach. Only the agent and appleInit images, always --platform linux/arm64, and only when digest-pinned — the script re-rejects a floating reference rather than trusting its caller. Docker pre-pulls for Squid, the API proxy and the CLI proxy are untouched.

Note

The pin is 0.12.3, not the newest release. Every current apple/container release is 1.x, which is outside AWF's contract range: a major bump may relocate the real vminitd inside the init image and boot a guest with no capability relay at all. The digest, signing identity, and install paths were verified against the real signed package.

State is run-scoped under ${RUNNER_TEMP} by default so nothing an earlier job left on the persistent runner can influence this one. CONTAINER_APP_ROOT is exported so the service, the image pull, AWF, and teardown all address the same store — a split root would surface as AWF failing to verify an image that had just been pulled. Teardown runs with if: always(), stopping the guest, then the services, then removing run-scoped state, and never rewrites the agent's outcome.

MCP gateway transport

The gateway had no route into the guest: gh-aw runs awmg-mcpg as its own Docker container, every other runtime reaches it through network.topologyAttach, and AWF rejects topologyAttach for a NIC-less guest.

gh-aw now publishes the gateway on macOS loopback port 9100 — bound to 127.0.0.1 only, never 0.0.0.0 — and declares it to AWF as appleContainer.mcpGatewayUpstreamPort. AWF health-probes the port, then publishes mcp-gateway.sock into the guest, whose relay serves 127.0.0.1:8080. The generated MCP client config addresses that guest URL.

API-key authentication, allowed mount roots, safe-output path permissions, and containerized GitHub MCP startup are unchanged. Because the guest port is compiled into both halves of AWF's transport contract, a non-default sandbox.mcp.port is now rejected. No topologyAttach, host networking, Compose service, enclave flag, or guest Docker socket is introduced.

MCP is wired end to end, subject to the live-hardware gap below.

Security review

A read-only security specialist reviewed the staged diff. It confirmed the no-NIC guarantee, the absence of any Docker socket path into the guest, loopback-only binding, correct supply-chain ordering, and safe shell interpolation. It raised one HIGH finding, fixed in this PR:

The published host port and the gateway's own port had always been equal, so the Stop MCP Gateway step passed MCP_GATEWAY_PORT and POSTed to localhost:8080 with the gateway API key in an Authorization header. Under apple-container nothing gh-aw owns listens on host 8080, so that request would have handed the credential to whatever local process held the port on a long-lived self-hosted runner — silently, because the curl -f failure is tolerated. The gateway container holds the host Docker socket and GitHub tokens, so the API key is the only thing protecting it.

Fixed by threading the published port through as a step output, targeting it from the stop script (which now refuses to send the key to a non-numeric port), and re-anchoring every check_mcp_servers.sh probe on the caller-supplied origin instead of the port the gateway reports. Teardown additionally refuses to rm -rf any application root containing .., closing the one path where an un-normalised operator-supplied root could escape the prefix check.

Portability

Generated setup scripts that now have to run on macOS were made BSD- and bash-3.2-safe without changing Linux behaviour: a portable pull deadline instead of GNU timeout, a shasum fallback for sha256sum, BSD stat -Lf for Docker socket group discovery (so it works automatically on a Mac rather than requiring manual configuration), and a resolved millisecond clock instead of date +%s%3N. install_awf_binary.sh selects and verifies the Darwin arm64 AWF build, skips the Linux-only chroot cleanup on macOS rather than invoking sudo for nothing, and no longer uses the bash-4.4 ${var@Q} expansion. Because the wider script corpus still uses bash 4 features, the preflight gates on bash 4+ rather than letting an unrelated script fail with a syntax error mid-run.

Across all 294 compiled workflows the Linux delta is exactly two lines, both semantically identical to what they replace, because MCP_GATEWAY_HOST_PORT equals MCP_GATEWAY_PORT everywhere except apple-container:

MCP_GATEWAY_HOST_PORT: ${{ steps.start-mcp-gateway.outputs.gateway-host-port }}
export MCP_GATEWAY_HOST_PORT="8080"

Exact prerequisites

To run a workflow with sandbox.agent.runtime: apple-container:

  • runs-on: [self-hosted, macOS, ARM64] (extra pool labels allowed). Every GitHub-hosted macos-* label is rejected at compile time.
  • Bare-metal Apple Silicon, macOS 26+, sysctl kern.hv_support = 1.
  • The Actions runner installed as a per-user LaunchAgent under an auto-logged-in account (./svc.sh install), running as an unprivileged user. container system start registers a per-user service; a LaunchDaemon or bare SSH session cannot start it.
  • bash 4+ on PATH (brew install bash).
  • Docker daemon + Compose plugin — AWF still runs Squid, the API proxy, the CLI proxy and the MCP gateway on the host.
  • Passwordless sudo only if you want the workflow to install the pinned Apple Container package; otherwise preinstall apple/container 0.12.3 and set runtime-install: false.
  • sandbox.agent.version (or firewall.version) pinned to an AWF release containing Move JavaScript tests and production files to actions/setup/js #7764 and docs: fix campaign command description to match CLI behavior #7768. The gate stays at v0.28.9 as instructed; bump AWFAppleContainerMinVersion if docs: fix campaign command description to match CLI behavior #7768 ships later, and never lower it.
  • On the gh-aw-firewall side, an APPLE_VMINIT_IMAGE repository variable must be set before any release carries an apple-init image.

Validation

  • go build ./... clean; make fmt applied.
  • make golint0 issues. make golint-custom — no findings in the changed files (the pre-existing function-length finding from layer 1 was resolved by splitting the validator).
  • go test ./pkg/workflow ./pkg/constants — all pass except TestGeneratePackageLock_UsesNormalizedWorkflowDir, which is a pre-existing macOS-only failure (/var vs /private/var symlink resolution) reproduced on the unmodified base commit.
  • New shell test apple_container_setup_cli_test.sh (22 assertions on the CLI version window, including that 1.x is rejected and that 0.12.3 sorts above 0.4.0 numerically) wired into make test-scripts; resolve_docker_socket_gid_test.sh still green.
  • shellcheck -S warning on every new and modified script introduces no new warnings.
  • make recompile — 294/294 succeed, with the two-line delta above.
  • make update-golden / make update-wasm-golden regenerated; a new apple-container wasm-golden fixture pins the generated lock file for this runtime.

Live validation gap

Important

No part of this has run on real hardware. Apple Container requires Virtualization.framework, and every GitHub-hosted macOS runner reports kern.hv_support=0 — the preflight fails there by design, with no fallback. No bare-metal Apple Silicon runner is registered in this repository, so no live smoke workflow is added here; gh-aw-firewall owns the live suite (workflow_dispatch only, protected environment, maintainer-supplied SHA, persist-credentials: false).

Specifically unvalidated on hardware: the launchd LaunchAgent registration path and its failure detection, --enable-kernel-install behaviour on a fresh app root, virtiofs uid/gid mapping, Apple Container image-store pull throughput, and the end-to-end MCP relay hop. Everything is unit- and contract-tested, and the CLI surface (system start, system status, system kernel set, image pull/inspect/list, list, delete, stop) was verified against the real container 0.12.3 binary rather than assumed.

Treat the runtime as preview-only until it has run green on a real runner.



✨ PR Review Safe Output Test - Run 33029847256

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • clients2.google.com
  • mtalk.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "clients2.google.com"
    - "mtalk.google.com"

See Network Configuration for more information.

💥 [THE END] — Illustrated by Smoke Claude · claude · sonnet46 · 66.8 AIC · ⌖ 17.3 AIC · ⊞ 7.1K ·
Comment /smoke-claude to run again

Layer 1 defined the apple-container contract and failed closed on everything
statically knowable. This layer generates the steps that prepare a self-hosted
bare-metal Apple Silicon runner, and wires the one transport that had no route
into the NIC-less guest.

Runtime provisioning
--------------------

Four steps now run before AWF, in dependency order, so an unusable host costs
seconds rather than a multi-gigabyte pull:

1. Host preflight rejects an ineligible runner: self-hosted provenance, macOS 26+,
   native arm64 (checked via RUNNER_ARCH *and* uname -m, so a runner registered as
   ARM64 but executing under Rosetta is caught), kern.hv_support=1, a usable
   launchd user domain, bash 4+, and a working Docker daemon with Compose.
2. CLI setup verifies a preinstalled `container` CLI inside AWF's validated range
   (>=0.4.0 <1.0.0). Only when runtime-install is not false does it install the
   pinned apple/container release, verified by SHA-256 AND by its
   "Developer ID Installer: Apple Inc. - Containerization" signature before
   `installer` runs, over non-interactive sudo. No `latest` path, no unsigned
   fallback, and the post-install version is re-checked against the window.
3. Service start runs `container system start --enable-kernel-install`. The flag is
   not optional: the default prompts on stdin, and a headless runner would block
   until the step is cancelled. The known headless launchd failure is detected and
   reported with the actual remediation rather than an opaque bootstrap error.
4. Image pull populates Apple Container's own store, which `docker pull` cannot
   reach. Only the agent and appleInit images, always --platform linux/arm64, and
   only when digest-pinned -- the script re-rejects a floating reference rather
   than trusting its caller. Docker pre-pulls for Squid, the API proxy and the CLI
   proxy are untouched.

The pinned release is deliberately 0.12.3 rather than the newest one. Every
current apple/container release is 1.x, which is outside AWF's contract range: a
major bump may relocate the real vminitd inside the init image and boot a guest
with no capability relay at all.

State is run-scoped under ${RUNNER_TEMP} by default, so nothing an earlier job
left on the persistent runner can influence this one. CONTAINER_APP_ROOT is
exported so the service, the image pull, AWF and teardown all address the same
store -- a split root would surface as AWF failing to verify an image that had
just been pulled. Teardown runs with if: always(), stopping the guest, then the
services, then removing run-scoped state, and never rewrites the agent's outcome.

MCP gateway transport
---------------------

The gateway had no route into the guest. gh-aw runs awmg-mcpg as its own Docker
container, every other runtime reaches it through network.topologyAttach, and AWF
rejects topologyAttach for a NIC-less guest. gh-aw now publishes the gateway on
macOS loopback port 9100 -- bound to 127.0.0.1 only, never 0.0.0.0 -- and declares
it to AWF as appleContainer.mcpGatewayUpstreamPort (gh-aw-firewall#7768). AWF
health-probes the port, then publishes mcp-gateway.sock into the guest, whose
relay serves 127.0.0.1:8080.

The generated MCP client config addresses that guest URL. API-key authentication,
allowed mount roots, safe-output path permissions and containerized GitHub MCP
startup are unchanged. Because the guest port is compiled into both halves of
AWF's transport contract, a non-default sandbox.mcp.port is now rejected. No
topologyAttach, host networking, Compose service, enclave flag or guest Docker
socket is introduced.

Security review fix: the published host port and the gateway's own port had always
been equal, so the Stop MCP Gateway step passed MCP_GATEWAY_PORT and POSTed to
localhost:8080 with the gateway API key in an Authorization header. Under
apple-container nothing gh-aw owns listens on host 8080, so that request would
have handed the credential to whatever local process held the port on a long-lived
runner -- silently, because the curl failure is tolerated. The published port is
now threaded through as a step output, the stop script targets it and refuses to
send the key to a non-numeric port, and check_mcp_servers.sh re-anchors every
probe on the caller-supplied origin instead of the port the gateway reports.
Teardown additionally refuses to rm -rf any application root containing "..",
which closes the one path where an un-normalised operator-supplied root could
escape the prefix check.

Portability
-----------

Generated setup scripts that now have to run on macOS were made BSD- and
bash-3.2-safe without changing Linux behaviour: download_docker_images.sh resolves
a portable pull deadline instead of requiring GNU timeout,
compute_artifact_prefix.sh falls back to shasum, resolve_docker_socket_gid.sh also
tries BSD `stat -Lf` so socket group discovery works automatically on a Mac, and
the MCP gateway scripts resolve a millisecond clock instead of assuming
`date +%s%3N`. install_awf_binary.sh selects and verifies the Darwin arm64 AWF
build, skips the Linux-only chroot cleanup on macOS rather than invoking sudo for
nothing, and no longer uses the bash-4.4 ${var@Q} expansion. Because the wider
script corpus still uses bash 4 features, the preflight gates on bash 4+ rather
than letting an unrelated script fail with a syntax error mid-run.

Across all 294 compiled workflows the Linux delta is two lines, both
semantically identical to what they replace: MCP_GATEWAY_HOST_PORT equals
MCP_GATEWAY_PORT everywhere except apple-container.

Docs and tests
--------------

The agent-runtimes reference gains a full Apple Container section: runner labels
and prerequisites, why GitHub-hosted macOS can never work, the capability
transport table, provisioning, the APPLE_VMINIT_IMAGE publication dependency,
unsupported features, diagnostics and cleanup. Agent-runtime instructions are
updated so tools do not emit an invalid configuration.

Tests cover setup-step generation and ordering, the version/checksum/signature
gates, app-root consistency, the separate image stores in both directions, digest
pinning, the MCP endpoint and transport config, shell quoting, and the negative
compatibility cases. A wasm-golden fixture pins the generated lock file, and
end-to-end compiler tests assert the host bind, the AWF config field, the guest
URL, readiness ordering and topology absence.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

Hey @lpcox 👋 — excellent work on bringing the Apple Container runtime to production-ready status! This PR is well-structured, thoroughly documented, and clearly aligned with the project's roadmap.

What looks great:

  • The feature is tightly focused: provisioning, image preparation, MCP gateway transport, and macOS-portable setup — all working together as a cohesive layer.
  • Comprehensive validation: shell tests added (apple_container_setup_cli_test.sh), linting clean, Go tests passing, and 294 workflows recompiled successfully.
  • Security-hardened: API key port binding fixed after security review, safe shell interpolation verified, and supply-chain checks in place.
  • Extensive documentation in the PR body covering provisioning, MCP gateway, portability trade-offs, and the live hardware gap.

This is ready for maintainer review.

Generated by ✅ Contribution Check · copilot · auto · 52.6 AIC · ⌖ 14.5 AIC · ⊞ 9.3K ·

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • clients2.google.com
  • mtalk.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "clients2.google.com"
    - "mtalk.google.com"

See Network Configuration for more information.

💥 [THE END] — Illustrated by Smoke Claude · claude · sonnet46 · 66.8 AIC · ⌖ 17.3 AIC · ⊞ 7.1K
Comment /smoke-claude to run again

@@ -0,0 +1,24 @@
---
"gh-aw": minor
---

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Smoke test review comment #1 (Run 33029847256): The changeset file looks well-structured. Consider adding a link to the relevant issue for traceability.

@@ -25,13 +25,16 @@ Use these instructions when creating or updating workflows that mention Docker,
- Docker sbx requires KVM and normally does not work on ARC DinD because the sbx daemon must run on the runner host.
- Cloud Hypervisor requires `RUNNER_ENVIRONMENT=github-hosted`, Ubuntu Linux x86_64, and `/dev/kvm`; it is not supported on self-hosted or ARC DinD runners.
- Apple Container is the inverse: it requires a self-hosted bare-metal Apple Silicon runner and is never valid on a GitHub-hosted `macos-*` label, because those runners are virtual machines without nested virtualization.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Smoke test review comment #2 (Run 33029847256): Good documentation of runtime constraints. It would be helpful to add examples for the apple-container runtime configuration.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds runnable Apple Container provisioning and zero-NIC MCP transport to the compiler’s existing runtime contract.

Changes:

  • Generates host validation, CLI/service setup, image preparation, and teardown.
  • Routes MCP through a loopback-bound gateway and AWF capability socket.
  • Adds macOS portability support, documentation, and regression fixtures.
Show a summary per file
File Description
.changeset/apple-container-runtime-provisioning.md Records the runtime feature release.
.github/aw/agent-runtime-instructions.md Adds Apple Container authoring guidance.
.github/workflows/*.lock.yml (294 files) Regenerates host-port gateway wiring.
Makefile Runs the new CLI setup test.
actions/setup/sh/apple_container_host_preflight.sh Validates eligible macOS hosts.
actions/setup/sh/apple_container_pull_images.sh Populates Apple’s image store.
actions/setup/sh/apple_container_setup_cli.sh Verifies or installs the CLI.
actions/setup/sh/apple_container_setup_cli_test.sh Tests CLI version handling.
actions/setup/sh/apple_container_start_services.sh Starts services and selects state roots.
actions/setup/sh/apple_container_teardown.sh Stops services and removes state.
actions/setup/sh/check_mcp_servers.sh Anchors probes to the published gateway.
actions/setup/sh/compute_artifact_prefix.sh Adds macOS hashing fallback.
actions/setup/sh/download_docker_images.sh Adds portable pull deadlines.
actions/setup/sh/install_awf_binary.sh Supports Darwin AWF installation.
actions/setup/sh/resolve_docker_socket_gid.sh Supports BSD stat.
actions/setup/sh/start_mcp_gateway.sh Publishes and reports the host port.
actions/setup/sh/stop_mcp_gateway.sh Safely targets gateway shutdown.
actions/setup/sh/verify_mcp_gateway_health.sh Adds portable timing.
docs/src/content/docs/reference/agent-runtimes.md Documents the preview runtime.
pkg/constants/version_constants.go Defines runtime versions and ports.
pkg/workflow/apple_container.go Extends compatibility validation.
pkg/workflow/apple_container_install.go Generates provisioning steps.
pkg/workflow/apple_container_install_test.go Tests provisioning generation.
pkg/workflow/apple_container_mcp_test.go Tests MCP transport contracts.
pkg/workflow/apple_container_test.go Extends runtime validation tests.
pkg/workflow/awf_config.go Models upstream Apple configuration.
pkg/workflow/awf_config_build.go Emits Apple transport configuration.
pkg/workflow/codex_engine.go Adds provisioning to Codex.
pkg/workflow/compiler_yaml_ai_execution.go Emits post-agent teardown.
pkg/workflow/mcp_setup_gateway.go Separates guest and host ports.
pkg/workflow/nodejs.go Adds provisioning to Node engines.
pkg/workflow/sandbox_runtime_profile.go Enables runtime installation.
pkg/workflow/sandbox_runtime_profile_test.go Updates profile expectations.
pkg/workflow/schemas/awf-config.schema.json Adds the upstream MCP port field.
pkg/workflow/testdata/TestWasmGolden_AllEngines/claude.golden Updates generated Claude output.
pkg/workflow/testdata/TestWasmGolden_AllEngines/codex.golden Updates generated Codex output.
pkg/workflow/testdata/TestWasmGolden_AllEngines/copilot.golden Updates generated Copilot output.
pkg/workflow/testdata/TestWasmGolden_AllEngines/gemini.golden Updates generated Gemini output.
pkg/workflow/testdata/TestWasmGolden_AllEngines/pi.golden Updates generated Pi output.
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/apple-container.golden Pins complete runtime output.
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/basic-copilot.golden Updates gateway output wiring.
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/playwright-cli-mode.golden Updates gateway output wiring.
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/smoke-copilot.golden Updates gateway output wiring.
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/with-imports.golden Updates gateway output wiring.
pkg/workflow/testdata/wasm_golden/fixtures/apple-container.md Adds the runtime compile fixture.

Review details

  • Files reviewed: 300/338 changed files
  • Comments generated: 4
  • Review effort level: Balanced

Comment on lines +151 to +156
image := defaultAWFImageForRole(role, imageTag)
if manifest != nil {
if override, ok := manifest[role]; ok && override != "" {
image = override
}
}
Comment on lines +36 to +37
APP_ROOT="${GH_AW_APPLE_CONTAINER_APP_ROOT:-${CONTAINER_APP_ROOT:-}}"
PRESERVE="${GH_AW_APPLE_CONTAINER_PRESERVE:-false}"
Comment on lines +102 to +105
bash_major="${BASH_VERSINFO[0]:-0}"
if (( bash_major < 4 )); then
fail "bash ${BASH_VERSION:-unknown} is too old. gh-aw's generated setup scripts require bash 4 or newer, and macOS only ships bash 3.2. Install a newer bash on the runner (for example 'brew install bash') and make sure it precedes /bin/bash on PATH."
fi
Comment on lines 118 to 121
# Timeout produces exit code 124
if [ $exit_code -eq 124 ]; then
echo "docker pull timed out for $image after 5 minutes"
echo "docker pull timed out for $image after ${PULL_TIMEOUT_SECONDS}s"
return 1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants