feat: microVM sandboxing backend — execution environments, guest image, vsock exec bridge - #873
Draft
panghy wants to merge 95 commits into
Draft
feat: microVM sandboxing backend — execution environments, guest image, vsock exec bridge#873panghy wants to merge 95 commits into
panghy wants to merge 95 commits into
Conversation
|
This pull request is abnormally large and would use a significant amount of tokens to review. If you still wish to review it, comment "augment review" and we will review it. |
panghy
force-pushed
the
feat/microvm-sandbox-backend
branch
from
August 3, 2026 12:21
dbec75f to
5455840
Compare
…nd microvm capability Introduces the sandbox settings group (sandbox.defaultType plus per-type direct/worktree/cow/microvm profiles with a microvm guest-image override), the sandbox.profiles.list / sandbox.profiles.update / sandbox.options router methods, the system.capabilities.microvmSupported field, and a one-time migration seeding sandbox.* from the legacy workspace.cowIsolation opt-in. Part of intent-hq/intent#1120 (EE-1).
Adds the guest-image sources (Dockerfile, intent-init, intent-vsock-exec) with the build script that produces a versioned rootfs + manifest, the guest-image-build / release-guest-image workflows (mirrored to intentd-releases like daemon assets), and the intent-services sandbox_image module that resolves an image ref (repo config, profile override, or built-in pin), downloads it, sha256-verifies it, and caches it under the data dir. Introduces the sandbox:image:* and sandbox:vm:* event constants consumed by this pipeline and the microVM orchestrator. Part of intent-hq/intent#1120 (EE-3).
New workspace crate wrapping libkrun: the daemon spawns this helper per agent VM to configure and boot the guest (rootfs, virtio-fs workspace mount, vsock) outside the daemon process. Ships the hypervisor entitlements plus the codesigning script used at packaging time. Part of intent-hq/intent#1120 (EE-4).
workspace.create accepts an explicit executionEnvironment (direct | worktree | cow | microvm), validated against the enabled sandbox profiles and host availability; explicit cow never silently falls back to a worktree, and microvm returns a structured not-implemented error until the orchestrator lands. The selection persists as Workspace.executionEnvironment (migration 0079; derived from the provisioning outcome when the param is omitted) and failures surface machine-readable execution-environment-unavailable / execution-environment-not-implemented error payloads. Also adds the repo-config executionEnvironment.image guest-image pin consumed by the sandbox_image resolver. Protocol version bumps to 3.3. Part of intent-hq/intent#1120 (EE-2).
…_manager integration Adds the intent-services microvm module: the orchestrator boots one VM per agent via the intentd-microvm-helper (guest image from the sandbox_image cache, per-agent reflink clone of the workspace mounted at /workspace), the vsock exec bridge drives the guest intent-vsock-exec agent, and the auth stager places short-lived credentials that are scrubbed on teardown. agent_manager spawns microvm-environment agents through this path (guest-facing MCP endpoints, VM handle ownership, teardown on reap), the ACP FileService gains a guest-path alias so /workspace/... requests rebase onto the host sandbox root, and the daemon wires its data dir through for the image cache and VM state. Part of intent-hq/intent#1120 (EE-5).
The exec, orchestrator, auth, and rootfs submodules use unix-only APIs (tokio::net::UnixStream, net::unix stream halves), breaking cargo check on x86_64-pc-windows-msvc. Compile them only on unix and provide a non-unix orchestrator stub with the same public surface whose entry points fail with a structured MicrovmError::Unsupported — mirroring the intentd-microvm-helper platform stub. microvm_platform_supported() already reports the capability as false on such targets, so the runtime never reaches the stub path.
panghy
force-pushed
the
feat/microvm-sandbox-backend
branch
from
August 3, 2026 12:32
5455840 to
1b9b23d
Compare
Services::provision_sandbox hard-errored with 'workspaces_root not configured' when no root was injected — but production daemons never call .with_workspaces_root() (only tests do; the binary configures the root via INTENTD_WORKSPACES_DIR). The microVM spawn path (ensure_started) is the first real-daemon caller of this method, so every microVM agent spawn failed immediately with an internal error. Resolve the root like every sibling consumer (the delegate CoW path, compute_cow_supported, compute_disk_usage): the injected workspaces_root, else default_workspaces_root(). This keeps microVM per-agent CoW clones under the same root as delegate CoW sandboxes. Adds a regression test that provisions a sandbox on a Services built without .with_workspaces_root(), and hoists the INTENTD_WORKSPACES_DIR env lock/guard to module scope so both env-mutating tests serialize.
Non-root tar cannot mknod on macOS, so device nodes baked into rootfs.tar.xz made the daemon's extraction fail and abort microVM spawn. - build-guest-image.sh: keep the mknod'd nodes only for the chroot smoke test; drop everything under /dev before the deterministic repack (the guest mounts devtmpfs over /dev at boot, so in-tarball nodes are never needed at runtime). - rootfs.rs ensure_extracted_tree: defensively --exclude './dev/*' and 'dev/*' on extraction so already-cached images built before this fix extract without a rebuild; recreate the empty /dev mountpoint after extraction (bsdtar drops the directory entry when its children are excluded). - intent-init: mkdir -p /dev guard before the devtmpfs mount. - regression test: fixture tarball with a plain file under ./dev/ must extract with the dev entry excluded and /dev left as an empty dir.
A deep daemon data dir pushed <vm_dir>/exec.sock past the 104-byte sockaddr_un limit on macOS, so libkrun's bind and the daemon's connect both failed and boot timed out after 60s. The exec socket now lives outside vm_dir at a short deterministic rendezvous path: <tmp>/intentd-vm-<uid>/<sha256(agent_id)[..12]>.sock. Because the socket grants arbitrary command execution inside the guest, the relocated path is hardened: - The parent dir is created 0700 and verified via lstat after creation: symlinks, non-directories, foreign-owned paths, and group/other mode bits are all hard errors (squat protection). The /tmp length fallback goes through the same private parent, never bare /tmp. - Pre-boot stale scrub removes the rendezvous path only when it is a socket owned by the current uid; anything else is a hard error. - The daemon chmods the socket to 0600 as soon as libkrun's bind materializes it (defense-in-depth; the bind happens inside krun_start_enter so the helper has no post-bind hook, and the brief window is covered by the 0700 parent). - Boot fails with a clear SUN_LEN error if even the fallback path is too long, and the helper CLI validates --vsock-listen path length up front (exit 64) instead of failing deep in a krun API call.
MicrovmVm's Drop scrubs the per-VM state dir (rootfs clone, console log)
and the temp-dir exec socket on a detached thread. When a teardown is
immediately followed by a re-spawn of the same agent id — e.g. the
silent redrive after a pre-output transport failure (monorepo#764 path)
— MicrovmVm::boot re-created the same vm_dir while the detached
remove_dir_all traversal was still running. The stale deletion then
gutted the freshly CoW-cloned rootfs, and the VM booted on a partial
tree: the guest exec of /usr/local/bin/intent-init failed with exit
status 127 ("No such file or directory").
Fix: register each teardown scrub thread in a registry keyed by vm_dir
(PENDING_SCRUBS); boot joins any pending scrub for its vm_dir before
provisioning. Since every boot goes through MicrovmVm::boot, this
closes the race for all re-entry paths (silent redrive, agent.retry,
a later sendMessage), and because the same scrub thread removes the
exec socket before the dir, the join also protects the new socket bind
against the old VM's socket scrub. Back-to-back scrubs of the same
path chain (the new thread joins the previous handle first), so
joining the newest handle always covers every in-flight deletion.
Regression tests: reprovision_awaits_inflight_teardown_scrub simulates
a slow in-flight deletion and asserts the re-provisioned rootfs stays
complete; chained_scrubs_join_previous_inflight_deletion covers the
chaining.
The provider runs inside the Linux guest, where the CoW sandbox is virtio-fs-mounted at /workspace (GUEST_WORKSPACE_DIR); the host sandbox path does not exist there. session/new and session/load carried the host cwd, so providers that validate the session cwd (auggie's workspace root) died before producing output, surfacing as "session/prompt transport closed before output". start_session now translates the cwd to GUEST_WORKSPACE_DIR when the agent handle owns a VM, covering all three session-open branches (resume, recreate, first open). Host-exec agents are unaffected. Refs intent-hq/intent#1120
…ox fields on discard discard_sandbox (and the GC / clone-failure / insert-failure cleanup paths) now remove the emptied <agentId>/ and sandboxes/ parent directories after deleting the sandbox, stopping at any non-empty level so sibling sandboxes survive. discard_sandbox also clears the agent session's sandbox_id/sandbox_path/ sandbox_branch (new scoped Store::clear_agent_session_sandbox UPDATE): a merged-and-discarded sandbox previously left session.sandbox_path pointing at the deleted directory, so a respawned microVM agent skipped re-provisioning and fell back to mounting the canonical repo — breaking isolation.
stage_file now compares the destination bytes before writing and returns false when identical — provider refreshers rewrite auth files on a timer without changing them, which spammed the rotation log and guest I/O. stage_all keeps host-file presence (not wrote-bytes) as the staged-entry criterion so already-matching files are still rotation-watched.
…erges Sandboxes (CoW delegate and microVM) now live for the agent's lifetime instead of being discarded on a successful merge-on-completion: - finalize_sandbox_merged keeps the sandbox directory, DB record, and session sandbox fields; it only marks the record merged, emits sandbox:cow:merged, and clears the retry count. - New sandbox.last_merged_commit_sha column (migration 0080) records the merged range's tip; merge_sandbox starts the next cherry-pick range there so repeat merges are incremental (no duplicate application). - provision_sandbox reuses an existing sandbox for the agent (respawn, daemon restart) instead of failing on the existing destination; a record whose directory vanished is dropped and re-provisioned. - agent.delete discards the agent's sandbox (directory + record) before the session row delete, since the FK cascade would otherwise strand the directory. Workspace deletion already sweeps the directory tree. - Tests updated for the persistent lifecycle; new coverage: repeat merge across two turns is incremental, provisioning reuses the existing sandbox, and agent.delete discards the sandbox.
Give parent agents control over child sandbox merge timing and visibility into sandbox state via the workspace_api surface, enabling fan-out of multiple sandboxed agents with a decide-later merge story. - ws.agent.delegate / ws.agent.create accept mergeOnTurnEnd (default true = today's behavior); the flag is stamped onto the child's session metadata and persisted on the sandbox record at provision time (migration 0081), surviving respawn/daemon restart - handle_sandbox_merge_on_completion skips the merge entirely for mergeOnTurnEnd=false sandboxes (no status transition, no bounce); completion propagates normally and the sandbox stays live. The merge retry sweep skips them too (skipped_manual_merge counter); manual sandbox.cow.merge remains the way to merge later - ws.agent.status surfaces sandboxStatus + mergeOnTurnEnd for sandboxed agents via the new WorkspaceApi::sandbox_get (one store lookup, only when metadata.sandboxPath is set); agent.list/status docs call out the metadata sandbox fields - workspace_api description gains sandbox doc clauses gated per-bridge on CoW capability (cow_capable_hint OR microVM workspace) via WorkspaceMcpServer::with_cow_capable; dispatch accepts and ignores mergeOnTurnEnd when not capable (advisory)
…backend # Conflicts: # crates/intent-acp/src/mcp_server.rs # crates/intent-acp/src/mcp_server/tools.rs # crates/intent-core/src/error.rs # crates/intent-core/src/lib.rs # crates/intent-services/src/agent_manager.rs # crates/intent-services/src/lib.rs # crates/intent-services/src/settings.rs # crates/intent-transport/src/catalog.rs # crates/intent-transport/src/catalog/tests.rs # crates/intent-transport/src/client/tests.rs # crates/intent-transport/src/control/tests.rs # crates/intent-transport/src/protocol.rs # crates/intentd/src/main.rs # crates/intentd/tests/wss_integration.rs
…d merge-outcome completion annotations - ws.agent.mergeSandbox(agentId): parent-facing MCP tool over the sandbox.cow.merge machinery (bindings/agent.rs prelude + dispatch), workspace-scoped via agent_get, documented in both tools.rs variants and scrubbed when the host is not CoW-capable (drift tests updated). - Dirty-sandbox policy keyed on the workspace's effective auto-commit: merge_sandbox_with(DirtyHandling) commits dirty state before merging (LLM-assisted message via the LNI-1 pipeline, deterministic fallback) when auto-commit is ON; when OFF the new MergeOutcome::Dirty refuses to snapshot/merge — the completion path bounces the agent with commit instructions (turn does not complete), the sweep treats it like blocked (no retry consumed), and the manual RPC returns status "dirty" + dirtyPaths. - Completion events now carry the merge outcome: agent:idle data gains sandboxMergeStatus (merged / merge_pending / unmerged), sandboxPath, and sandboxCommitRange; format_completion_wake and format_group_child_line render the outcome so parents always learn whether a child's work reached the canonical repo and where it lives when it did not.
- CoW: the workspace_aggregates cow_supported() choke point returns Some(false) on non-macOS without running the filesystem probe (and without creating the root dir), propagating to compute_cow_supported, cow_capable_hint, system.capabilities.cowSupported, Workspace.cowSupported, and the provisioning fallbacks. The direct probe call sites outside the choke point (workspace.create / workspace.duplicate provisioning, per-agent sandbox provisioning, the startup warm-up, and doctor) gate identically. - microVM: microvm_platform_supported() drops the Linux/KVM arm — Linux now reports 'microVM sandboxes are temporarily locked to macOS'; the macOS arms are unchanged. - Reason surfaces: sandbox.options and the explicit executionEnvironment validation name the temporary macOS lock on non-macOS instead of blaming the filesystem. - Tests: platform_capable assertions drop the Linux/KVM disjunct; CoW skip gates skip on non-macOS; new cow_supported_locked_off_macos unit test asserts the lock answers without touching disk. All lock sites are marked 'temporarily' for greppable unlocking.
…rrides and model-facing spawn hints
- Settings: sandbox.microvm.vcpus (integer, default 2, range 1-16) and
sandbox.microvm.memMib (integer, default 2048, min 128) join the
BE-owned sandbox.* typed-schema group; bounds mirror the microVM
helper's MAX_VCPUS/MIN_MEM_MIB and out-of-range values are rejected
by normal settings validation. Both ride the sandbox.profiles.*
microvm row like image (list echoes them, update validates
microvm-only + range and persists via the same enable batch).
- Per-agent override: optional vmResources { vcpus?, memMib? } on
ws.agent.delegate and ws.agent.create (new intent_core::VmResources
with helper-bounds validate()); parsed and validated at call time so
bad input errors at delegate/create, never at VM boot; persisted on
the child session metadata so respawns keep the size; partial
overrides merge over settings then built-in defaults. Advisory
(accepted-and-ignored) on non-microVM workspaces, like
mergeOnTurnEnd.
- Spawn site resolves session override > settings > 2/2048 at
ensure_started VM boot and logs the resolved size (info).
- Model-facing hints: microVM-workspace bridges inject a live
continuation line into the workspace_api delegate docs — resolved
guest image (repo config > settings override > built-in pin, labeled
by pin version or manifest URL; no network I/O at description build)
and the settings-resolved default VM size. Captured at bridge
creation (with_microvm_hints seam beside with_cow_capable), so
settings changes apply to new sessions only. vmResources doc clauses
ride the same cow_capable scrub as mergeOnTurnEnd.
- Tests: settings parse/range-reject + defaults drift, VmResources
bounds/wire-shape, delegate-time validation + metadata stamping,
profiles list/update sizing rows + microvm-only + range rejection,
hint injection/gating/composition/flattening, drift tests green.
Adds the daemon-global sandbox.image.check router method (protocol v4.8,
283 router / 320 dispatchable): fetches the guest-image manifest, verifies
the optional outer sha256 pin, and contract-checks it (schema/arch/rootfs/
vsock-exec) without downloading the rootfs or mutating the image cache.
Fetch/validation failures are results ({ valid: false, error }), not RPC
errors, so settings UIs can check-before-save a sandbox.microvm.image
override. Factors fetch_and_validate_manifest out of ensure_image as the
shared dry-run half.
… and merge phase timings
Root cause of the stranded-'merging' sandbox from isolation-lab live fire:
the turn-end merge path claimed the row (merge_pending -> merging) and then
ran the whole merge inline in the completion delivery loop with no drop/panic
protection. When the daemon task was cancelled mid-merge (harness restart),
no terminal status was ever written, leaving the row stranded in 'merging'
forever, and the inline merge also head-of-line-blocked every other agent's
completion delivery while it ran.
- MergeClaimGuard (RAII): resets merging -> merge_pending on drop unless
disarmed after a terminal status is persisted, in both the completion path
and the background sweep
- Completion delivery loop spawns a task per completion event so a slow
merge no longer stalls other agents' completion wakes
- Background sweep watchdog (reset_stale_merging_sandboxes) recovers rows
stuck in 'merging' beyond the stale threshold, covering daemon crashes
where no in-process guard survives
- sandbox.cow.merge on an already-claimed row returns a structured
{ status: "in_progress" } result instead of Error::Internal
- All git2/subprocess merge work moved onto spawn_blocking (merge_sandbox_git,
fetch_canonical_to_sandbox) so large repos cannot pin runtime workers
- Phase timings (claim, dirty-commit message, merge, finalize) logged per
turn-end merge
- Regression tests: guard drop/disarm, stale watchdog fresh-vs-stale rows,
manual merge on claimed row
…ge lanes, async sandbox_merge ack, and sandboxed host.exec cwd
- bounded conflict retries land a terminal 'conflict' status with
conflictingPaths persisted, a ReviewRequired attention event, and the
sandbox commits pushed to a sb/<agent-id>-recovery-<ts> branch in the
canonical repo so work is never stranded
- background sweep runs per-workspace merge lanes (DashMap of mutexes):
a wedged merge in one workspace no longer blocks unrelated workspaces
- sandbox.cow.merge returns an immediate { status: started | in_progress }
ack and runs the merge as a detached task; outcome surfaces via
agent.status sandboxStatus and completion/attention events
- documented status set (created/merging/merge_pending/merged/conflict/
failed) serialized snake_case everywhere; completion-event annotations
derive from the persisted row so they cannot disagree
- host.exec cwd for delegated sandboxed agents resolves inside the
caller's sandbox (callerAgentId stamped server-side; client-supplied
values stripped)
…backend # Conflicts: # crates/intent-acp/src/mcp_server/tools.rs # crates/intent-core/src/lib.rs # crates/intent-core/src/settings_file.rs # crates/intent-services/src/sandbox_ops.rs # crates/intent-transport/src/catalog.rs # crates/intent-transport/src/catalog/tests.rs # crates/intent-transport/src/client/tests.rs # crates/intent-transport/src/control/tests.rs # crates/intent-transport/src/protocol.rs # crates/intentd/tests/wss_integration.rs
… only persisting the message The bounce path called agent_send_message_op, which persists a transcript row but never resumes the agent, so a bounced agent sat idle until some unrelated wake. Route the bounce through deliver_wake_message so the conflict-reconciliation instructions start an actual turn. Regression: test_conflict_bounce_resumes_agent_via_runtime
…n_environment to new e2e workspace literal
…digest call sites off the removed LowerHex impl
- sandbox.direct.enabled is clamped: SettingsFile::validate rejects false (boot config, settings.update, and sandbox.profiles.update alike), and sandbox_type_enabled always reports direct enabled - workspace.create rejects executionEnvironment: worktree for githubUrl and isNewRepo creates (structured execution-environment-unavailable) — worktree needs a local repository copy to link against - direct in the hydration/new-repo flows no longer skips provisioning: the standalone checkout is still provisioned and worked in directly - explicit cow/microvm no longer silently degrades to a plain clone in the cache-hydration arm (probe-unsupported, probe-failed, and provisioning-time Unsupported all surface the structured payload) - omitted executionEnvironment now derives and persists from the provisioning outcome in the hydration (cow/direct) and isNewRepo (direct) arms, matching the local-repo arm - unit tests for the clamp, both flow rejections, and the isNewRepo derivation; WSS e2e arms for the flow rejections in Scenario H
assert_hydrated_checkout now checks execution_environment matches the checkout-mode mapping (Cow -> cow, Direct -> direct) in every hydration test, and create_hydrates_from_cache_on_miss reads the workspace back from the store to prove the derived value round-trips
…andbox migrations to 0101-0104 after main took 0100)
…own) into microvm-sandbox-backend Catch-up merge resolving 20 conflicting files: - protocol.rs: PROTOCOL_VERSION 7.4; merged §5.35 EE + §5.12 wake-trigger docs - migrations renumbered 0102-0105 (main claimed 0101); store tests updated - mcp_server: main's compact/full description split carries cow_capable + microvm_hints through full_workspace_api_description and the compact variant - agent_manager: main's child field rename (was _child) + vm handle kept; compact_tool_descriptions builder merged alongside with_microvm_hints - host_exec: sandbox-aware default cwd merged with main's monorepo#3231 workspace-root default (sandbox_root_for helper) - catalog: router methods 300, total 339 (branch +4, main +2 over base 294/333) - pedantic lint compliance for branch-side code (doc_markdown backticks, must_use, missing # Errors sections, format_collect, try_from casts)
panghy
marked this pull request as draft
August 26, 2026 04:31
…vm-sandbox-backend Catch-up merge #31 (281 commits). Sandbox migrations renumbered to 0115-0118, protocol re-seated at v9.8 on main's v9.7 baseline, cow_capable/microvm_hints re-threaded through the workspace_api description assembly (full + condensed), catalog goldens updated to 304 router / 345 total methods, and Workspace/AgentSession initializers updated for main's context_links/retired_at additions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backend implementation of microVM sandboxing (Waves 1+2 scope of the tracking issue), delivered as five scoped commits:
sandbox.*RPCs, and microvm capability — settings-file schema for sandbox profiles (sandbox.worktree/sandbox.cow/sandbox.microvm), thesandbox.profiles.list/sandbox.profiles.update/sandbox.optionsrouter methods, and thesystem.capabilities.microvmSupportedfield.guest-image/Dockerfile + init for the microVM guest rootfs, CI workflow to build/publish it, and thesandbox_imageresolver that downloads, checksums, and caches pinned guest images (repo-configexecutionEnvironment.imagepin).intentd-microvm-helperVMM helper crate — the privileged helper binary that owns the VMM (krun) lifecycle, plus signing script wiring.executionEnvironmentselection inworkspace.create— explicitdirect | worktree | cow | microvmselection validated against enabled profiles and host availability; persisted asWorkspace.executionEnvironment(migration 0079); structuredexecution-environment-unavailable/execution-environment-not-implementederror payloads.agent_managerintegration — the orchestrator that boots/supervises guest VMs, the vsock exec agent bridge for host→guest command execution, and agent-manager routing of agent processes into the sandbox.Protocol
Rebased on top of protocol 4.0 (
terminal.listenvelope change): the additive bumps in this PR land as 4.1 (sandbox RPC surface: 278 router methods, 315 dispatchable names,microvmSupportedcapability) and 4.2 (workspace.createexecutionEnvironmentparam + structured error payloads).Note: the corresponding
docs/PROTOCOL.mdupdate lives in the monorepodocs/tree and rides the later monorepo submodule-bump PR — no PROTOCOL.md copy exists in this repo.Verification
make check(fmt + clippy-D warnings+ build) — green from the monorepo root.make test— full suite run;intent-serviceslib suite passes 2279/2279 (two timing-flaky tests unrelated to this change failed once under full parallel load and pass in isolation and on re-run).Fixes intent-hq/intent#1120