[chruffins]

Summary

Third of six macOS guest PRs. Gives Darwin guests a system guest agent (a root LaunchDaemon serving the existing GuestService on vsock 2222). It also changes how macOS instances report readiness, run exec, stop, and boot, and it makes the exec exit path wait for the process group leader without reaping it.

Stack position: base is main, which now includes #498 and #499. Commits on top of main: 0b1dae2 (the guest agent, readiness, and exec), then the remediation commits from review.

• An image opts in with guest_agent: true in its platform config. Images without it keep the old behavior.
• With the agent, a macOS instance is Initializing until a readiness probe succeeds, then Running, like Linux guests. Without the agent, Running still means "VMM running".
• Exec, cp, and stat need the agent. Without it they return 501 before any WebSocket upgrade.
• Exec on macOS runs through the agent. A failed client WebSocket cancels the command and its process group.
• Stop gives macOS guests a 30-second minimum window and does not shorten it when the shutdown RPC fails.
• The guest display is now 3024x1964 at 160 ppi (1512x982 points at 2x). It was 1280x800 at 80 ppi (ebfa583).
• The direct Linux exec path reports the process exit status as on main: a timed-out command reports -1 (e2090c3). Only the macOS lifecycle reports 124 on a deadline.
• Agent versioning (23a38e6): a new GetAgentInfo RPC reports the agent's protocol version, build version and optional capabilities. The macOS readiness probe records it on the instance. An agent that predates it is protocol 0; an agent with a newer protocol than the host is refused. The host does not call it for Linux guests.
• Agent startup deadline (e85a8c7): on macOS hosts, a guest whose declared agent has not answered after 10 minutes in Initializing is stopped. The reason (with the last probe failure) goes in exit_message; disks and logs are kept and the restart policy applies.

Existing and new flows

Readiness (macOS only; Linux paths are unchanged):

Existing: start -> admission -> VMM starts -> state = Running
          (deriveRunningState: stored.MacOS != nil -> StateRunning)

New:      start -> admitMacOSBoot (boot lock, socket-path and identity checks) -> VMM starts
          if !image.guest_agent (GuestAgentEnabled false):   -> state = Running (unchanged result)
          if  image.guest_agent:
              loop until WaitForAgent:
                  ProbeExec: dial a new connection, exec "/usr/bin/true", close it
                  retryable connection error -> wait, then retry
                  success                    -> state = Running, transition at GuestAgentReadyAt
              no answer yet                  -> state = Initializing

Probes do not use the shared pooled connection. Before, a probe that failed evicted the pool for everyone else. KeepPooledConn is removed.

Exec exit path (macOS lifecycle; the Linux direct path is separate on purpose):

Existing (Linux direct path, exec.go):
  exec.CommandContext; buffer all stdout and stderr; cmd.Wait (reaps the leader);
  send buffered output in chunks; send exit code.
  timeout: Go kills the direct child; status 124.

New (macOS lifecycle, exec_lifecycle.go):
  Setpgid (no TTY); grp := processGroup(pid)
  watcher: if ctx ends before the lifecycle finishes -> grp.kill()  (never after reap)
  stream stdout and stderr through the sender as the process writes (or the PTY output)
  finishExec:
      waitLeaderExit(pid)          waitid WNOWAIT (Linux) / kqueue NOTE_EXIT (Darwin)
                                   leader is now a zombie; its group id is still reserved
      drainAfterExit(bound)        bounded; a deadline during the drain still kills the group
      grp.reap() -> cmd.Wait()     no kill is sent after this point
      execExitCode(deadline -> 124, else process exit code)
      sendExitCode(bounded)        the status is sent within the drain bound

Shared between both paths (exec_common.go): command setup, PTY start, the stdin and resize pump, and exit-status rules. The policy stays separate: the Linux path buffers output and waits without a bound, and the macOS lifecycle streams and bounds its drain.

Stop:

Existing: graceful shutdown RPC; if it was not sent and the wait exceeds
          shutdownFailureFallbackWait, shorten the wait to that fallback
          stop timeout = requested timeout

New:      same, except macOS keeps its full grace period (no shortening)
          macOS: stop timeout = max(requested, 30s)

Capability checks: readiness, probes, and shutdown use GuestAgentEnabled(), which also requires the macOS image declaration. Linux is unchanged, since there GuestAgentEnabled() equals !SkipGuestAgent.

Boot disk: a macOS instance's disk is always boot.raw. An instance created on main before this PR kept its disk at overlay.raw, and it will not start after this change (see Caveats).

Change size

Totals from GitHub for this PR against main: 60 files, +2,399 / −192.

Category	Files	Added	Removed
Implementation	36	1,069	180
Tests	21	1,200	5
Docs	3	130	7

Approximate meaningful implementation: about 650 lines. This is an estimate, not a Git count. Excluded: tests, docs, moved code (about 80 lines moved from exec.go into exec_common.go), mechanical call-site updates, and generated code (none).

Chunk	Main files	Approx. meaningful
Darwin guest agent: vsock listener, readiness probe, network, shutdown policy	vsock_darwin.go, vsock_darwin_nocgo.go, ready_darwin.go, network_darwin.go, shutdown_darwin.go, shutdown_policy.go, main.go	~200
Exec lifecycle: finish, drain, reap, and sender	exec_lifecycle.go	~250
Process group ownership and non-reaping wait	exec_group.go, exec_reap_linux.go, exec_reap_darwin.go	~110
Shared exec setup	exec_common.go	~60
Probes on their own connection	lib/guest/client.go	~40
Readiness, capability checks, boot disk, stop	query.go, guest_os.go, macos.go, stop.go, create.go	~150
Host-side exec, cp, and stat gating	cmd/api/api/exec.go, cp.go, guest_unsupported.go	~25
Display	cmd/vz-shim/macos_arm64.go	2

Behavior changes

• Readiness: an agent-enabled macOS instance is Initializing until the agent answers, so it reports Running later than before. MacOS images without the agent are unchanged.
• Exec on macOS without the agent: still 501, with a different message.
• Agent startup: an agent-enabled macOS guest that is still Initializing after 10 minutes is stopped instead of waiting forever.
• Shutdown: Darwin accepts only an orderly SIGTERM, and the agent must run as root. Otherwise the shutdown RPC fails.
• Display: the VM's device config changes, so a snapshot taken at 1280x800 is not expected to restore.
• Boot disk: boot.raw only. MacOS instances created before this PR keep their disk at overlay.raw and will not start.

Rollout

1. macOS guests 1/6: VZ runtime, machine images and instance lifecycle #498 and macOS guests 2/6: OCI machine-image integration #499 are merged. This PR's base is main, so it can merge as soon as review is done.
2. macOS guests 4/6: snapshots, schedules and stopped forks #506 (snapshots) and macOS guests 5/6: guest-port forwarding #514 (guest-port forwarding) rebase onto this PR. Snapshots and the display change interact, so review macOS guests 4/6: snapshots, schedules and stopped forks #506 with that in mind.
3. macOS guests 6/6: experimental machine-image builds #503 (machine-image builds) is not needed for v0.

Testing

• TestDirectExecReportsProcessExitCode: a direct exec that exits in time keeps its code when output delivery outlasts the deadline, and a timeout reports -1 (TTY and non-TTY). Fails on the previous head.
• TestGetAgentInfo, TestGuestServiceReportsAgentInfo, TestMacOSAgentInfoRecordedAtReadiness: current, legacy and too-new agents, and an RPC failure.
• TestMacOSAgentStartupDeadlineStopsGuest, TestMacOSAgentStartupOverdueRules: an overdue guest is stopped with its reason; one within the deadline is left alone.
• Full suite on the stack tip versus main: the same 51 failures on both (KVM, root networking, mkfs.erofs, Docker Hub).
• go test ./lib/system/guest_agent/ -count=1 -race — ok on Linux (14 s).
• Exec and lifecycle subset, -count=3 -race on Linux — ok (40 s), including TestExecTimeoutDuringDrainKillsBackgroundChild, TestExecTimeoutDuringPipeDrainKillsBackgroundChild, TestSendExitCodeBoundedWhenClientStopsReading, and TestCancellationAfterWaitDoesNotKillGroup.
• go test ./lib/guest/ -count=1 -race — ok, including TestProbeExecLeavesPooledConnectionInPlace.
• Darwin guest, test binary built with GOOS=darwin GOARCH=arm64 CGO_ENABLED=0 go test -c ./lib/system/guest_agent/ and run in the guest:
  • Full binary — PASS.
  • Exec, lifecycle, cancellation, and timeout subset — 22 pass, 1 skip. The skipped test is the TTY drain test. macOS revokes the terminal when the shell exits, so a background child cannot hold the drain open there. The pipe variant covers that drain on macOS.
• Darwin exit-wait probe, run in the guest before the change: the kqueue NOTE_EXIT event fired before the reap, the group was still signalable, kill(-pgid) reached a background child, and a watch on an already-exited child still fired.
• go test ./lib/instances/ with a targeted filter (guest, readiness, running, macOS, stop, health, VM storage path) — 77 pass, 7 skip, 3 fail. The 3 failures are Firecracker and Cloud Hypervisor fork and stop tests that need mkfs.erofs, which is not installed on the review VM. They are environmental.
• go vet on ./lib/instances/, ./lib/guest/, and ./lib/system/guest_agent/ — clean, using placeholder embed files that were removed afterwards.
• gofmt -l on the changed Go files — clean.
• Not run: go build ./..., the full ./lib/instances/ and ./cmd/api/... suites (embedded hypervisor binaries are missing from this checkout), and the cgo vz shim on Darwin. The display change has not been booted on a VM.
• Earlier native QA, from the original description and not re-run for this head: six runs on Apple silicon (macOS 27.0.1 host and guests), plus a VZ restore spike. The simplification pass after run 6 was not re-run natively.

Caveats / Not included

• Snapshots taken at the old display size are not expected to restore. Breaking changes are acceptable for macOS guests at this stage.
• MacOS instances created before this PR (disk at overlay.raw) do not start after it. No migration or fallback.
• Probes open one connection per attempt, so readiness listings cost more RPC setup than before.
• A group kill is never sent after the leader is reaped. The leader stays unreaped until the drain ends, so its group id stays reserved. Verified on a macOS guest.
• The 2x scale that 160 ppi produces has not been checked on a VM.
• Snapshots (macOS guests 4/6: snapshots, schedules and stopped forks #506), guest-port forwarding (macOS guests 5/6: guest-port forwarding #514), and machine-image builds (macOS guests 6/6: experimental machine-image builds #503) are out of scope here.
• Two output policies remain by design: the Linux direct path buffers output, and the macOS lifecycle streams it.

Review focus

• lib/system/guest_agent/exec_lifecycle.go (finishExec, executeLifecycleNoTTY) and exec_group.go: the reap ordering and the kill guard.
• lib/system/guest_agent/exec_reap_linux.go and exec_reap_darwin.go: the non-reaping wait.
• lib/guest/client.go (ProbeExec) and lib/instances/query.go: probes and the readiness state.

Stack

Each PR is one commit on the one below it.

1. macOS guests 1/6: VZ runtime, machine images and instance lifecycle #498 VZ runtime, machine images and instance lifecycle (merged)
2. macOS guests 2/6: OCI machine-image integration #499 OCI machine images (merged)
3. macOS guests 3/6: Darwin guest service, readiness and exec #500 Darwin guest agent, readiness and exec (this PR)
4. macOS guests 4/6: snapshots, schedules and stopped forks #506 snapshots, standby and stopped clones
5. macOS guests 5/6: guest-port forwarding #514 guest-port forwarding
6. macOS guests 6/6: experimental machine-image builds #503 machine-image builds (not needed for v0; can land later)

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 8f6dd1c. Configure here.}

[chruffins]

Real native Darwin GuestService2222 QA passed in an isolated cold-booted OCI guest using the current shared binary, user-launched at UID501 only. Host shared client verified id -u=501, synthetic file copy-to/from exact bytes, and client deadline cancellation of shell + sleep descendant (child PID no longer existed). Three repeated race runs passed. Shutdown correctly returned PermissionDenied for nonroot; guest remained running. Fixed two QA harness assumptions before the passing run: CopyFromInstance's destination is a directory, and deadline may surface as gRPC DeadlineExceeded rather than errors.Is(context.DeadlineExceeded).

Root daemon was not installed: guest noninteractive sudo is unavailable and no privileged provisioning was performed. System image capability remained false. These results prove native transport/user-level exec/files/cancellation and shutdown admission, not normal API system-agent installation/readiness or successful graceful root shutdown. Those remain gates. Original guest/API/storage untouched; isolated QA VM ended stopped.

[chruffins]

Simplification update at 243ce9a: common command setup/process-group cancellation and bounded wait/drain/final-status handling for both TTY modes. Permanent tests cover pre-cancelled starts and failed output sends cancelling/reaping commands. Full agent/client race suites passed 3 runs; combined post-restack focused suites also passed 3 runs. Base changed to feat/macos-oci-images so the image rollback fix and bundle consolidation are inherited once rather than duplicated. Root provisioning remains undone.

All PRs remain draft. This is source-level/synthetic validation, not a new live boot or production build proof. Default Codex independent review was attempted but is still blocked by authentication (HTTP401). Published atomically after checking reviewer heads with explicit per-ref force-with-lease; prior heads preserved locally.

[chruffins]

Root QA follow-up pushed as ae3685d; desktop restacked atomically with explicit leases. Manually installed shared system-role agent survived isolated normal-API cold boots. Repeated race-instrumented authenticated API tests verified UID0 exec (both TTY modes), root-owned0600 file copy round trips, and descendant cancellation. Live testing reproduced and fixed Darwin readiness using the nonexistent /bin/true; /usr/bin/true now probes successfully and normal GET persists GuestAgentReadyAt without a Linux workload marker. Fixed premature shutdown fallback after a lost Darwin RPC reply: configured grace period is retained while still requiring owned VMM exit; Linux policy is unchanged. Final normal API stop accepted the RPC and confirmed VMM exit/storage closure without forced fallback. Corrected stale opt-in test calls left after runtime simplification. Focused instances/API/agent/client tests pass race,count3; default independent autoreview remains401-blocked. Manual QA provisioning is not automatic image installation or end-to-end build/publication proof. PR remains draft.

[sjmiller609]

One nit here

linux regression: lib/system/guest_agent/exec.go:130,196 — slow output delivery can replace an already-completed command’s exit code with 124.

maybe if you want, consider adding RPC in the guest to report some versioning information, since in this scenario we have the guest agent being packaged with the image, not injected in initrd like it is in the case of linux. that addresses concerns like the below

agent compatibility: lib/images/types.go:39–41, lib/instances/query.go:443–460 — guest_agent: true plus executing true doesn’t establish compatibility. add protocol/capability negotiation and document image-based upgrades/rollback without breaking older linux agents.
startup timeout: lib/instances/query.go:299–339 — a missing/broken agent leaves managed macOS guests initializing indefinitely. add an overall deadline, retain an actionable failure reason, stop the VM, and preserve disks/logs. keep linux behavior unchanged.

[chruffins]

@sjmiller609 thanks, all three addressed:

• exit code: the direct (Linux) exec path reports the process exit status again, as on main (e2090c3). Only the macOS lifecycle reports 124 on a deadline. TestDirectExecReportsProcessExitCode covers slow output delivery and a real timeout.
• agent versioning: new GetAgentInfo RPC returning protocol version, build version and capabilities (23a38e6). The macOS readiness probe records it; agents that predate it are protocol 0, newer protocols are refused. Upgrade/rollback via images is in docs/macos-guest-agent.md. macOS guests 5/6: guest-port forwarding #514 adds a port-forward capability so old agents get a clear 501.
• startup deadline: on macOS hosts, a guest whose declared agent hasn't answered after 10 min in Initializing is stopped, with the reason (incl. last probe failure) in exit_message; disks/logs kept, restart policy applies (e85a8c7). Linux unchanged.