From 46cb1eabda97e5967cc90142018d6f560947756a Mon Sep 17 00:00:00 2001 From: Tianyu Zhou Date: Wed, 9 Sep 2026 21:51:06 +0800 Subject: [PATCH 1/2] feat(firecracker): enable OCI and Nydus roots with virtio-fs Update sandboxd to b8f4656 and use its pinned Firecracker v1.16.1-akernel.3 artifacts. Build virtiofsd 1.14.0 from a pinned source revision with its release lockfile, and package the binary, runtime libraries, and licenses with the Firecracker payload. Enable read-only virtio-fs in standalone and Helm so Firecracker consumes OCI and Nydus roots directly from the image manager. Keep private ext4 writes and configure AsyncDirect with Writeback, documenting the host capabilities and checkpoint compatibility required by the updated stack. Allow the custom-image example and integration suite to select image roots, verify private writes across sandboxes, and install CA certificates for HTTPS checks. Document RAFS v5 preparation and the supported image and mount contracts. Signed-off-by: Tianyu Zhou --- AGENTS.md | 36 ++++++++++++++----- builder/node.Dockerfile | 21 +++++++++++ deploy/README.md | 28 ++++++++++++--- deploy/akernel/charts/core/values.yaml | 6 ++++ deploy/standalone/README.md | 8 +++++ deploy/standalone/config/sandboxd_config.toml | 6 ++++ sdk/python/README.md | 29 ++++++++++++--- sdk/python/examples/custom_image.py | 12 +++++-- sdk/python/tests/integration/test_sandbox.py | 22 ++++++++++-- src/sandboxd | 2 +- 10 files changed, 148 insertions(+), 22 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index fca1d2a..b018765 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -136,6 +136,12 @@ artifacts; installs the Firecracker VMM and guest kernel; and constructs the matching guest-agent initrd. Runc remains build-time optional, and `AKERNEL_ENABLE_FIRECRACKER=false` excludes the Firecracker payload. +AKernel builds virtiofsd 1.14.0 from the pinned source commit and release +Cargo.lock in `builder/node.Dockerfile`. Keep its shared-library dependencies +and licenses packaged with the Firecracker payload. Both standalone and Helm +enable read-only virtio-fs by default; disabling the Firecracker image payload +also excludes virtiofsd. + The sandboxd submodule's runtime manifest is the source of truth for the gVisor and Firecracker releases used by both sandboxd E2E and AKernel packaging. Test an unreleased runtime by checking out the sandboxd commit that @@ -199,16 +205,23 @@ scheduling with a no-resource error when no eligible node exists. Do not treat a configured runtime as an advertised runtime. Firecracker supports commands, files, PTYs, network policies, published ports, -reverse tunnels, read-only EROFS image roots and mounts, explicit `storage_mb` -quotas, and recovery across sandboxd restarts. Its root and filesystem image -mounts must be local or image-provider-backed regular EROFS files. It rejects -OCI/Nydus directory roots, directory mounts, writable live host binds, NVIDIA -GPUs, and nested KVM rather than weakening their semantics. +reverse tunnels, EROFS roots and mounts, OCI/Nydus directory roots and read-only +host directory mounts through virtio-fs, explicit `storage_mb` quotas, and +recovery across sandboxd restarts. OCI image mounts, writable live host binds, +NVIDIA GPUs, and nested KVM remain unsupported. Do not add Firecracker-specific directory conversion, image caching, or -artifact reference counting to sandboxd or its image manager. Build EROFS -before sandbox creation and distribute it through the existing local or S3 -imagefile paths. The bundled default runtime root already follows this model. +artifact reference counting to sandboxd or its image manager. Consume OCI/Nydus +directories directly from the image manager through read-only virtio-fs. +Explicit local/S3 imagefile roots and mounts must already be EROFS. The bundled +default runtime root also remains EROFS; sandbox writes use a private ext4 disk. + +The bundled Firecracker writable disk policy is `AsyncDirect` with `Writeback`. +Validate io_uring and `STATX_DIOALIGN` on the target host and filestore; use an +explicit `Async` or `Sync` policy on incompatible hosts, never silent fallback. +Keep standalone and Helm defaults synchronized. Drain before upgrading the +runtime stack: checkpoint compatibility includes VMM, kernel, initrd, and +virtiofsd digests, and restores retain the saved writable I/O engine. Runc is excluded from default image builds and from the default advertised runtime set. Guided cloud profiles use `make config ENABLE_RUNC=true`; this @@ -537,6 +550,13 @@ python sdk/python/benchmarks/sandbox_pressure.py \ --xpu gpu:a10:1 --storage-mb 256 --processes 1 --threads 1 ``` +Set `AKERNEL_TEST_IMAGE=ubuntu:24.04` with `AKERNEL_TEST_RUNTIME=firecracker` +to run integration and reload coverage against an OCI/Nydus image root. This +also verifies that two sandboxes using the same image have private writes. +Test both an ordinary OCI image and a Nydus image resolved through the deployed +image manager. The pinned distill-fs supports RAFS v5; use +`nydusify convert --fs-version 5` when preparing Nydus test images. + ## Maintenance Rules - Keep root README and repo-level agent guidance focused on current diff --git a/builder/node.Dockerfile b/builder/node.Dockerfile index 19669be..16634b0 100644 --- a/builder/node.Dockerfile +++ b/builder/node.Dockerfile @@ -33,6 +33,9 @@ ARG FIRECRACKER_BUILD_IMAGE=ubuntu:24.04 ARG FIRECRACKER_RELEASE ARG FIRECRACKER_AMD64_SHA256 ARG FIRECRACKER_AMD64_URL +ARG VIRTIOFSD_BUILD_IMAGE=rust:1.90.0-bookworm +# virtiofsd v1.14.0, including the release Cargo.lock. +ARG VIRTIOFSD_REVISION=c2540f8db14caba81c1e37fba23fc7bf2cd7f0dd ARG OTELCOL_CONTRIB_VERSION=0.120.0 ARG OTELCOL_CONTRIB_URL=https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTELCOL_CONTRIB_VERSION}/otelcol-contrib_${OTELCOL_CONTRIB_VERSION}_linux_amd64.tar.gz ARG AKERNEL_VERSION=unknown @@ -114,6 +117,19 @@ WORKDIR /src/sandboxd COPY ./src/sandboxd/ ./ RUN make release +FROM ${VIRTIOFSD_BUILD_IMAGE} AS virtiofsd-builder +ARG VIRTIOFSD_REVISION +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + ca-certificates git libcap-ng-dev libseccomp-dev pkg-config && \ + rm -rf /var/lib/apt/lists/* +WORKDIR /src/virtiofsd +RUN git init && \ + git fetch --depth=1 https://gitlab.com/virtio-fs/virtiofsd.git "${VIRTIOFSD_REVISION}" && \ + git checkout --detach FETCH_HEAD && \ + test "$(git rev-parse HEAD)" = "${VIRTIOFSD_REVISION}" && \ + cargo build --release --locked + FROM ${FIRECRACKER_BUILD_IMAGE} AS firecracker-runtime-true ARG FIRECRACKER_RELEASE ARG FIRECRACKER_AMD64_SHA256 @@ -154,6 +170,9 @@ RUN set -eux; \ /firecracker/opt/firecracker/; \ cp -a "${bundle}/licenses/." /firecracker/opt/firecracker/licenses/ +COPY --from=virtiofsd-builder /src/virtiofsd/target/release/virtiofsd /firecracker/usr/local/bin/virtiofsd +COPY --from=virtiofsd-builder /src/virtiofsd/LICENSE-APACHE /firecracker/opt/firecracker/licenses/virtiofsd-LICENSE-APACHE +COPY --from=virtiofsd-builder /src/virtiofsd/LICENSE-BSD-3-Clause /firecracker/opt/firecracker/licenses/virtiofsd-LICENSE-BSD-3-Clause COPY --from=sandboxd-builder /src/sandboxd/output/firecracker-agent /initrd/init RUN set -eux; \ chmod 0755 /initrd/init; \ @@ -247,7 +266,9 @@ RUN apt-get update && \ iptables \ jq \ kmod \ + libcap-ng0 \ libgcc-s1 \ + libseccomp2 \ logrotate \ mount \ openssl \ diff --git a/deploy/README.md b/deploy/README.md index 9ba6a48..63940e0 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -32,10 +32,30 @@ and do not advertise either VM runtime. If no eligible node advertises a requested runtime, `Sandbox(runtime="kata")` or `Sandbox(runtime="firecracker")` fails scheduling with a no-resource error. -The bundled Firecracker payload is selected by sandboxd's shared runtime -manifest. Set `AKERNEL_ENABLE_FIRECRACKER=false` while building to exclude it. -For supported operations and filesystem constraints, see the -[sandbox runtime comparison](https://github.com/inclusionAI/sandboxd/blob/1918fadb03b59bc6f540196e14b90a91bdf31b7d/doc/runtime.md). +The bundled Firecracker VMM and guest kernel are selected by sandboxd's shared +runtime manifest, and its guest-agent initrd is built from that same sandboxd +revision. AKernel also builds pinned virtiofsd 1.14.0 with its release lockfile +and enables read-only virtio-fs in standalone and Helm. OCI/Nydus image roots +use the directory provided by the image manager directly, without EROFS +conversion. The sandbox's writable layer remains a private ext4 image; +writable host sharing and OCI image mounts are unsupported. EROFS roots and +mounts remain supported. Set `AKERNEL_ENABLE_FIRECRACKER=false` while building +to exclude the VMM, kernel, virtiofsd, and initrd. + +The default writable disk policy is `AsyncDirect` with `Writeback`. Hosts must +provide usable `io_uring` and filesystem alignment queries through +`statx(STATX_DIOALIGN)` (normally ext4/XFS on Linux 6.1 or newer). Capability +checks, rather than the kernel version alone, determine compatibility. There +is no automatic buffered fallback. For older hosts, explicitly configure +`writable_io_engine="Async"`, or `"Sync"` without io_uring, under +`[plugin.runtime.firecracker]` in the standalone config or Helm's +`node.config.sandboxd.config`. Keep `writable_cache_type="Writeback"`. + +Drain sandboxes before replacing the Firecracker stack. Checkpoints record +VMM, kernel, initrd, and, when used, virtiofsd digests; mismatched stacks are +rejected on restore. Changing the writable I/O default does not convert the +engine saved in an existing checkpoint. For the full contract, see the +[sandbox runtime comparison](https://github.com/inclusionAI/sandboxd/blob/b8f4656c57432bce1b50111e766bcd21510ee482/doc/runtime.md). The native Linux runc backend is opt-in because it uses the host kernel. For a guided cloud profile, `make config ENABLE_RUNC=true` records both sides of the diff --git a/deploy/akernel/charts/core/values.yaml b/deploy/akernel/charts/core/values.yaml index ab638fd..b0cb659 100644 --- a/deploy/akernel/charts/core/values.yaml +++ b/deploy/akernel/charts/core/values.yaml @@ -494,6 +494,12 @@ node: firecracker="/usr/local/bin/firecracker" # AKERNEL_RUNTIME_RUNC + [plugin.runtime.firecracker] + virtiofs_enabled=true + virtiofsd_path="/usr/local/bin/virtiofsd" + writable_io_engine="AsyncDirect" + writable_cache_type="Writeback" + [plugin.runtime.kata] config_path="/opt/kata/share/defaults/kata-containers/runtime-rs/configuration-dragonball.toml" kvm_device="/dev/kvm" diff --git a/deploy/standalone/README.md b/deploy/standalone/README.md index b24912b..950a7df 100644 --- a/deploy/standalone/README.md +++ b/deploy/standalone/README.md @@ -18,6 +18,14 @@ Containers and Firecracker. Both `Sandbox(runtime="kata")` and virtualization on the Docker host. Nodes without KVM remain usable with runsc and do not advertise either VM runtime to the scheduler. +Firecracker enables read-only virtio-fs by default, so +`Sandbox(runtime="firecracker", image="ubuntu:24.04")` can use OCI or Nydus +roots directly. The image includes a pinned virtiofsd and matching VMM, +kernel, and guest agent. Its private writable disk uses `AsyncDirect` and +`Writeback`; the host must support io_uring and `STATX_DIOALIGN` on the +filestore filesystem. See the [deployment guide](../README.md) for older-host +configuration and checkpoint compatibility when upgrading the runtime stack. + See the maintained [runtime selection example](../../sdk/python/examples/sandbox_runtime.py) for client usage. diff --git a/deploy/standalone/config/sandboxd_config.toml b/deploy/standalone/config/sandboxd_config.toml index ea41dd1..d0bf138 100644 --- a/deploy/standalone/config/sandboxd_config.toml +++ b/deploy/standalone/config/sandboxd_config.toml @@ -50,6 +50,12 @@ kata="/usr/local/bin/containerd-shim-kata-v2" firecracker="/usr/local/bin/firecracker" # AKERNEL_RUNTIME_RUNC +[plugin.runtime.firecracker] +virtiofs_enabled=true +virtiofsd_path="/usr/local/bin/virtiofsd" +writable_io_engine="AsyncDirect" +writable_cache_type="Writeback" + [plugin.runtime.kata] config_path="/opt/kata/share/defaults/kata-containers/runtime-rs/configuration-dragonball.toml" kvm_device="/dev/kvm" diff --git a/sdk/python/README.md b/sdk/python/README.md index c18d5ae..e65fe88 100644 --- a/sdk/python/README.md +++ b/sdk/python/README.md @@ -289,8 +289,10 @@ prerequisites are available. Kata and Firecracker require at least one cluster node whose sandboxd instance successfully initialized the requested runtime with a usable `/dev/kvm` device. Nodes without KVM remain available for runsc workloads and do not -advertise either VM runtime. Firecracker accepts EROFS image roots and mounts -and rejects OCI/Nydus directories, directory mounts, GPUs, and nested KVM. +advertise either VM runtime. The bundled Firecracker configuration enables +read-only virtio-fs for OCI/Nydus image roots and read-only host directories, +while retaining EROFS image roots and mounts. OCI image mounts, writable host +binds, GPUs, and nested KVM remain unsupported. The all-in-one image can optionally package a native Linux `runc` backend. Operators build it with `AKERNEL_ENABLE_RUNC=true` and enable it explicitly @@ -561,9 +563,18 @@ runtime inside an S3 object. When neither source is supplied, AKernel sends only the selected isolation runtime and openYuanRong overlays it onto the rootfs configured by the deployed service. -For Firecracker, the deployed default or an explicit `rootfs` object must be a -raw EROFS image. `image="ubuntu:24.04"` produces an OCI/Nydus directory and is -therefore supported by runsc, runc, and Kata but rejected by Firecracker. +Firecracker supports `Sandbox(runtime="firecracker", image="ubuntu:24.04")` +with the bundled virtio-fs configuration. Both OCI and Nydus roots use the +image manager's directory directly, without conversion to EROFS. The shared +root stays read-only; sandbox writes use its private ext4 overlay. The deployed +default and explicit `rootfs` S3 objects continue to use raw EROFS images. +The bundled distill-fs supports Nydus RAFS v5. When preparing Nydus images with +`nydusify convert`, select `--fs-version 5` explicitly. +Custom deployments must enable `plugin.runtime.firecracker.virtiofs_enabled` +and install the matching Firecracker stack and virtiofsd. + +Run `python sdk/python/examples/custom_image.py --runtime firecracker` on a +KVM-capable deployment, or add `--image` to select another image. The same `S3Config` type can be used as a read-only mount source: @@ -702,6 +713,14 @@ PYTHONPATH=sdk/python \ python -m unittest discover -s sdk/python/tests/integration -t sdk/python -v ``` +Set `AKERNEL_TEST_RUNTIME=firecracker` and `AKERNEL_TEST_IMAGE=ubuntu:24.04` +to exercise the same suite, including checkpoint/reload, against a virtio-fs +image root. `AKERNEL_TEST_IMAGE` also accepts a Nydus image reference and +enables a check that writes stay private to each sandbox sharing the image. +The test image must provide an Ubuntu/Debian userspace with `apt-get` for the +checkpoint test's curl and CA certificate installation. Omit +`AKERNEL_TEST_IMAGE` to test the deployed default EROFS root. + Load and transfer benchmarks live under [`benchmarks/`](./benchmarks) and are not part of the default test suite. diff --git a/sdk/python/examples/custom_image.py b/sdk/python/examples/custom_image.py index ba76450..7229bb1 100644 --- a/sdk/python/examples/custom_image.py +++ b/sdk/python/examples/custom_image.py @@ -12,7 +12,9 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Launch a sandbox from a public OCI image.""" +"""Launch an OCI image; use --runtime firecracker on a KVM-capable node.""" + +import argparse from akernel_sdk import Sandbox @@ -20,7 +22,13 @@ def main() -> None: - with Sandbox(image=IMAGE, cpu=1000, memory=2048) as sandbox: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--runtime", default="runsc") + parser.add_argument("--image", default=IMAGE) + args = parser.parse_args() + with Sandbox( + image=args.image, runtime=args.runtime, cpu=1000, memory=2048 + ) as sandbox: result = sandbox.commands.run(". /etc/os-release && printf $PRETTY_NAME") assert result.exit_code == 0, result.stderr print(f"Sandbox {sandbox.id}: {result.stdout}") diff --git a/sdk/python/tests/integration/test_sandbox.py b/sdk/python/tests/integration/test_sandbox.py index 87ce95d..7387508 100644 --- a/sdk/python/tests/integration/test_sandbox.py +++ b/sdk/python/tests/integration/test_sandbox.py @@ -27,10 +27,12 @@ and bool(os.environ.get("AKERNEL_TOKEN")) ) _RUNTIME = os.environ.get("AKERNEL_TEST_RUNTIME", "runsc") +_IMAGE = os.environ.get("AKERNEL_TEST_IMAGE") or None _INSTALL_CURL_COMMAND = ( "apt-get update && " - "DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends curl" + "DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends " + "curl ca-certificates" ) _CHECKPOINT_COMMAND = ( @@ -69,7 +71,7 @@ def log_message(self, *_args): class SandboxIntegrationTest(unittest.TestCase): @classmethod def setUpClass(cls): - cls.sandbox = Sandbox(cpu=1000, memory=2048, runtime=_RUNTIME) + cls.sandbox = Sandbox(cpu=1000, memory=2048, runtime=_RUNTIME, image=_IMAGE) @classmethod def tearDownClass(cls): @@ -96,6 +98,21 @@ def test_filesystem(self): ) self.assertTrue(self.sandbox.files.exists("/tmp/akernel-integration.txt")) + @unittest.skipUnless(_IMAGE, "set AKERNEL_TEST_IMAGE to test an OCI/Nydus root") + def test_image_writes_are_private(self): + original = self.sandbox.files.read("/etc/os-release") + try: + self.sandbox.files.write("/etc/os-release", "AKERNEL_PRIVATE_ROOT\n") + with Sandbox( + cpu=1000, memory=2048, runtime=_RUNTIME, image=_IMAGE + ) as other: + self.assertEqual(other.files.read("/etc/os-release"), original) + self.assertEqual( + self.sandbox.files.read("/etc/os-release"), "AKERNEL_PRIVATE_ROOT\n" + ) + finally: + self.sandbox.files.write("/etc/os-release", original) + def test_pty(self): output = bytearray() with self.sandbox.pty.create(on_data=output.extend) as session: @@ -179,6 +196,7 @@ def test_internal_checkpoint_reload_and_reverse_tunnel(self): memory=2048, storage_mb=256, runtime=_RUNTIME, + image=_IMAGE, reverse_tunnel=tunnel, failover=True, ) diff --git a/src/sandboxd b/src/sandboxd index b892414..b8f4656 160000 --- a/src/sandboxd +++ b/src/sandboxd @@ -1 +1 @@ -Subproject commit b892414a3b21f3c2a4481322767362af14fc701f +Subproject commit b8f4656c57432bce1b50111e766bcd21510ee482 From 117b0142e53a22b9d25066fb687716a3566b1d60 Mon Sep 17 00:00:00 2001 From: Tianyu Zhou Date: Wed, 9 Sep 2026 22:08:26 +0800 Subject: [PATCH 2/2] refactor(examples): keep custom image launch minimal Restore the custom image example to its original fixed configuration so it remains a short illustration of the Sandbox API. Remove documentation for the extra command-line arguments while retaining Firecracker image support documentation and integration coverage. Signed-off-by: Tianyu Zhou --- sdk/python/README.md | 3 --- sdk/python/examples/custom_image.py | 12 ++---------- 2 files changed, 2 insertions(+), 13 deletions(-) diff --git a/sdk/python/README.md b/sdk/python/README.md index e65fe88..0e5bd15 100644 --- a/sdk/python/README.md +++ b/sdk/python/README.md @@ -573,9 +573,6 @@ The bundled distill-fs supports Nydus RAFS v5. When preparing Nydus images with Custom deployments must enable `plugin.runtime.firecracker.virtiofs_enabled` and install the matching Firecracker stack and virtiofsd. -Run `python sdk/python/examples/custom_image.py --runtime firecracker` on a -KVM-capable deployment, or add `--image` to select another image. - The same `S3Config` type can be used as a read-only mount source: ```python diff --git a/sdk/python/examples/custom_image.py b/sdk/python/examples/custom_image.py index 7229bb1..ba76450 100644 --- a/sdk/python/examples/custom_image.py +++ b/sdk/python/examples/custom_image.py @@ -12,9 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Launch an OCI image; use --runtime firecracker on a KVM-capable node.""" - -import argparse +"""Launch a sandbox from a public OCI image.""" from akernel_sdk import Sandbox @@ -22,13 +20,7 @@ def main() -> None: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--runtime", default="runsc") - parser.add_argument("--image", default=IMAGE) - args = parser.parse_args() - with Sandbox( - image=args.image, runtime=args.runtime, cpu=1000, memory=2048 - ) as sandbox: + with Sandbox(image=IMAGE, cpu=1000, memory=2048) as sandbox: result = sandbox.commands.run(". /etc/os-release && printf $PRETTY_NAME") assert result.exit_code == 0, result.stderr print(f"Sandbox {sandbox.id}: {result.stdout}")