feat(firecracker): enable OCI and Nydus roots with virtio-fs

AGENTS.md

@@ -136,6 +136,12 @@ artifacts; installs the Firecracker VMM and guest kernel; and constructs the
 matching guest-agent initrd. Runc remains build-time optional, and
 `AKERNEL_ENABLE_FIRECRACKER=false` excludes the Firecracker payload.
 
+AKernel builds virtiofsd 1.14.0 from the pinned source commit and release
+Cargo.lock in `builder/node.Dockerfile`. Keep its shared-library dependencies
+and licenses packaged with the Firecracker payload. Both standalone and Helm
+enable read-only virtio-fs by default; disabling the Firecracker image payload
+also excludes virtiofsd.
+
 The sandboxd submodule's runtime manifest is the source of truth for the
 gVisor and Firecracker releases used by both sandboxd E2E and AKernel
 packaging. Test an unreleased runtime by checking out the sandboxd commit that
@@ -199,16 +205,23 @@ scheduling with a no-resource error when no eligible node exists. Do not treat
 a configured runtime as an advertised runtime.
 
 Firecracker supports commands, files, PTYs, network policies, published ports,
-reverse tunnels, read-only EROFS image roots and mounts, explicit `storage_mb`
-quotas, and recovery across sandboxd restarts. Its root and filesystem image
-mounts must be local or image-provider-backed regular EROFS files. It rejects
-OCI/Nydus directory roots, directory mounts, writable live host binds, NVIDIA
-GPUs, and nested KVM rather than weakening their semantics.
+reverse tunnels, EROFS roots and mounts, OCI/Nydus directory roots and read-only
+host directory mounts through virtio-fs, explicit `storage_mb` quotas, and
+recovery across sandboxd restarts. OCI image mounts, writable live host binds,
+NVIDIA GPUs, and nested KVM remain unsupported.
 
 Do not add Firecracker-specific directory conversion, image caching, or
-artifact reference counting to sandboxd or its image manager. Build EROFS
-before sandbox creation and distribute it through the existing local or S3
-imagefile paths. The bundled default runtime root already follows this model.
+artifact reference counting to sandboxd or its image manager. Consume OCI/Nydus
+directories directly from the image manager through read-only virtio-fs.
+Explicit local/S3 imagefile roots and mounts must already be EROFS. The bundled
+default runtime root also remains EROFS; sandbox writes use a private ext4 disk.
+
+The bundled Firecracker writable disk policy is `AsyncDirect` with `Writeback`.
+Validate io_uring and `STATX_DIOALIGN` on the target host and filestore; use an
+explicit `Async` or `Sync` policy on incompatible hosts, never silent fallback.
+Keep standalone and Helm defaults synchronized. Drain before upgrading the
+runtime stack: checkpoint compatibility includes VMM, kernel, initrd, and
+virtiofsd digests, and restores retain the saved writable I/O engine.
 
 Runc is excluded from default image builds and from the default advertised
 runtime set. Guided cloud profiles use `make config ENABLE_RUNC=true`; this
@@ -537,6 +550,13 @@ python sdk/python/benchmarks/sandbox_pressure.py \
   --xpu gpu:a10:1 --storage-mb 256 --processes 1 --threads 1
 ```
 
+Set `AKERNEL_TEST_IMAGE=ubuntu:24.04` with `AKERNEL_TEST_RUNTIME=firecracker`
+to run integration and reload coverage against an OCI/Nydus image root. This
+also verifies that two sandboxes using the same image have private writes.
+Test both an ordinary OCI image and a Nydus image resolved through the deployed
+image manager. The pinned distill-fs supports RAFS v5; use
+`nydusify convert --fs-version 5` when preparing Nydus test images.
+
 ## Maintenance Rules
 
 - Keep root README and repo-level agent guidance focused on current

builder/node.Dockerfile

@@ -33,6 +33,9 @@ ARG FIRECRACKER_BUILD_IMAGE=ubuntu:24.04
 ARG FIRECRACKER_RELEASE
 ARG FIRECRACKER_AMD64_SHA256
 ARG FIRECRACKER_AMD64_URL
+ARG VIRTIOFSD_BUILD_IMAGE=rust:1.90.0-bookworm
+# virtiofsd v1.14.0, including the release Cargo.lock.
+ARG VIRTIOFSD_REVISION=c2540f8db14caba81c1e37fba23fc7bf2cd7f0dd
 ARG OTELCOL_CONTRIB_VERSION=0.120.0
 ARG OTELCOL_CONTRIB_URL=https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTELCOL_CONTRIB_VERSION}/otelcol-contrib_${OTELCOL_CONTRIB_VERSION}_linux_amd64.tar.gz
 ARG AKERNEL_VERSION=unknown
@@ -114,6 +117,19 @@ WORKDIR /src/sandboxd
 COPY ./src/sandboxd/ ./
 RUN make release
 
+FROM ${VIRTIOFSD_BUILD_IMAGE} AS virtiofsd-builder
+ARG VIRTIOFSD_REVISION
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+      ca-certificates git libcap-ng-dev libseccomp-dev pkg-config && \
+    rm -rf /var/lib/apt/lists/*
+WORKDIR /src/virtiofsd
+RUN git init && \
+    git fetch --depth=1 https://gitlab.com/virtio-fs/virtiofsd.git "${VIRTIOFSD_REVISION}" && \
+    git checkout --detach FETCH_HEAD && \
+    test "$(git rev-parse HEAD)" = "${VIRTIOFSD_REVISION}" && \
+    cargo build --release --locked
+
 FROM ${FIRECRACKER_BUILD_IMAGE} AS firecracker-runtime-true
 ARG FIRECRACKER_RELEASE
 ARG FIRECRACKER_AMD64_SHA256
@@ -154,6 +170,9 @@ RUN set -eux; \
       /firecracker/opt/firecracker/; \
     cp -a "${bundle}/licenses/." /firecracker/opt/firecracker/licenses/
 
+COPY --from=virtiofsd-builder /src/virtiofsd/target/release/virtiofsd /firecracker/usr/local/bin/virtiofsd
+COPY --from=virtiofsd-builder /src/virtiofsd/LICENSE-APACHE /firecracker/opt/firecracker/licenses/virtiofsd-LICENSE-APACHE
+COPY --from=virtiofsd-builder /src/virtiofsd/LICENSE-BSD-3-Clause /firecracker/opt/firecracker/licenses/virtiofsd-LICENSE-BSD-3-Clause
 COPY --from=sandboxd-builder /src/sandboxd/output/firecracker-agent /initrd/init
 RUN set -eux; \
     chmod 0755 /initrd/init; \
@@ -247,7 +266,9 @@ RUN apt-get update && \
         iptables \
         jq \
         kmod \
+        libcap-ng0 \
         libgcc-s1 \
+        libseccomp2 \
         logrotate \
         mount \
         openssl \

deploy/README.md

@@ -32,10 +32,30 @@ and do not advertise either VM runtime. If no eligible node advertises a
 requested runtime, `Sandbox(runtime="kata")` or
 `Sandbox(runtime="firecracker")` fails scheduling with a no-resource error.
 
-The bundled Firecracker payload is selected by sandboxd's shared runtime
-manifest. Set `AKERNEL_ENABLE_FIRECRACKER=false` while building to exclude it.
-For supported operations and filesystem constraints, see the
-[sandbox runtime comparison](https://github.com/inclusionAI/sandboxd/blob/1918fadb03b59bc6f540196e14b90a91bdf31b7d/doc/runtime.md).
+The bundled Firecracker VMM and guest kernel are selected by sandboxd's shared
+runtime manifest, and its guest-agent initrd is built from that same sandboxd
+revision. AKernel also builds pinned virtiofsd 1.14.0 with its release lockfile
+and enables read-only virtio-fs in standalone and Helm. OCI/Nydus image roots
+use the directory provided by the image manager directly, without EROFS
+conversion. The sandbox's writable layer remains a private ext4 image;
+writable host sharing and OCI image mounts are unsupported. EROFS roots and
+mounts remain supported. Set `AKERNEL_ENABLE_FIRECRACKER=false` while building
+to exclude the VMM, kernel, virtiofsd, and initrd.
+
+The default writable disk policy is `AsyncDirect` with `Writeback`. Hosts must
+provide usable `io_uring` and filesystem alignment queries through
+`statx(STATX_DIOALIGN)` (normally ext4/XFS on Linux 6.1 or newer). Capability
+checks, rather than the kernel version alone, determine compatibility. There
+is no automatic buffered fallback. For older hosts, explicitly configure
+`writable_io_engine="Async"`, or `"Sync"` without io_uring, under
+`[plugin.runtime.firecracker]` in the standalone config or Helm's
+`node.config.sandboxd.config`. Keep `writable_cache_type="Writeback"`.
+
+Drain sandboxes before replacing the Firecracker stack. Checkpoints record
+VMM, kernel, initrd, and, when used, virtiofsd digests; mismatched stacks are
+rejected on restore. Changing the writable I/O default does not convert the
+engine saved in an existing checkpoint. For the full contract, see the
+[sandbox runtime comparison](https://github.com/inclusionAI/sandboxd/blob/b8f4656c57432bce1b50111e766bcd21510ee482/doc/runtime.md).
 
 The native Linux runc backend is opt-in because it uses the host kernel. For a
 guided cloud profile, `make config ENABLE_RUNC=true` records both sides of the

deploy/akernel/charts/core/values.yaml

@@ -494,6 +494,12 @@ node:
         firecracker="/usr/local/bin/firecracker"
         # AKERNEL_RUNTIME_RUNC
 
+        [plugin.runtime.firecracker]
+        virtiofs_enabled=true
+        virtiofsd_path="/usr/local/bin/virtiofsd"
+        writable_io_engine="AsyncDirect"
+        writable_cache_type="Writeback"
+
         [plugin.runtime.kata]
         config_path="/opt/kata/share/defaults/kata-containers/runtime-rs/configuration-dragonball.toml"
         kvm_device="/dev/kvm"

deploy/standalone/README.md

@@ -18,6 +18,14 @@ Containers and Firecracker. Both `Sandbox(runtime="kata")` and
 virtualization on the Docker host. Nodes without KVM remain usable with runsc
 and do not advertise either VM runtime to the scheduler.
 
+Firecracker enables read-only virtio-fs by default, so
+`Sandbox(runtime="firecracker", image="ubuntu:24.04")` can use OCI or Nydus
+roots directly. The image includes a pinned virtiofsd and matching VMM,
+kernel, and guest agent. Its private writable disk uses `AsyncDirect` and
+`Writeback`; the host must support io_uring and `STATX_DIOALIGN` on the
+filestore filesystem. See the [deployment guide](../README.md) for older-host
+configuration and checkpoint compatibility when upgrading the runtime stack.
+
 See the maintained
 [runtime selection example](../../sdk/python/examples/sandbox_runtime.py) for
 client usage.

deploy/standalone/config/sandboxd_config.toml

@@ -50,6 +50,12 @@ kata="/usr/local/bin/containerd-shim-kata-v2"
 firecracker="/usr/local/bin/firecracker"
 # AKERNEL_RUNTIME_RUNC
 
+[plugin.runtime.firecracker]
+virtiofs_enabled=true
+virtiofsd_path="/usr/local/bin/virtiofsd"
+writable_io_engine="AsyncDirect"
+writable_cache_type="Writeback"
+
 [plugin.runtime.kata]
 config_path="/opt/kata/share/defaults/kata-containers/runtime-rs/configuration-dragonball.toml"
 kvm_device="/dev/kvm"

sdk/python/README.md

@@ -289,8 +289,10 @@ prerequisites are available.
 Kata and Firecracker require at least one cluster node whose sandboxd instance
 successfully initialized the requested runtime with a usable `/dev/kvm`
 device. Nodes without KVM remain available for runsc workloads and do not
-advertise either VM runtime. Firecracker accepts EROFS image roots and mounts
-and rejects OCI/Nydus directories, directory mounts, GPUs, and nested KVM.
+advertise either VM runtime. The bundled Firecracker configuration enables
+read-only virtio-fs for OCI/Nydus image roots and read-only host directories,
+while retaining EROFS image roots and mounts. OCI image mounts, writable host
+binds, GPUs, and nested KVM remain unsupported.
 
 The all-in-one image can optionally package a native Linux `runc` backend.
 Operators build it with `AKERNEL_ENABLE_RUNC=true` and enable it explicitly
@@ -561,9 +563,15 @@ runtime inside an S3 object. When neither source is supplied, AKernel sends
 only the selected isolation runtime and openYuanRong overlays it onto the
 rootfs configured by the deployed service.
 
-For Firecracker, the deployed default or an explicit `rootfs` object must be a
-raw EROFS image. `image="ubuntu:24.04"` produces an OCI/Nydus directory and is
-therefore supported by runsc, runc, and Kata but rejected by Firecracker.
+Firecracker supports `Sandbox(runtime="firecracker", image="ubuntu:24.04")`
+with the bundled virtio-fs configuration. Both OCI and Nydus roots use the
+image manager's directory directly, without conversion to EROFS. The shared
+root stays read-only; sandbox writes use its private ext4 overlay. The deployed
+default and explicit `rootfs` S3 objects continue to use raw EROFS images.
+The bundled distill-fs supports Nydus RAFS v5. When preparing Nydus images with
+`nydusify convert`, select `--fs-version 5` explicitly.
+Custom deployments must enable `plugin.runtime.firecracker.virtiofs_enabled`
+and install the matching Firecracker stack and virtiofsd.
 
 The same `S3Config` type can be used as a read-only mount source:
 
@@ -702,6 +710,14 @@ PYTHONPATH=sdk/python \
   python -m unittest discover -s sdk/python/tests/integration -t sdk/python -v
 ```
 
+Set `AKERNEL_TEST_RUNTIME=firecracker` and `AKERNEL_TEST_IMAGE=ubuntu:24.04`
+to exercise the same suite, including checkpoint/reload, against a virtio-fs
+image root. `AKERNEL_TEST_IMAGE` also accepts a Nydus image reference and
+enables a check that writes stay private to each sandbox sharing the image.
+The test image must provide an Ubuntu/Debian userspace with `apt-get` for the
+checkpoint test's curl and CA certificate installation. Omit
+`AKERNEL_TEST_IMAGE` to test the deployed default EROFS root.
+
 Load and transfer benchmarks live under [`benchmarks/`](./benchmarks) and are
 not part of the default test suite.
 

sdk/python/tests/integration/test_sandbox.py

@@ -27,10 +27,12 @@
     and bool(os.environ.get("AKERNEL_TOKEN"))
 )
 _RUNTIME = os.environ.get("AKERNEL_TEST_RUNTIME", "runsc")
+_IMAGE = os.environ.get("AKERNEL_TEST_IMAGE") or None
 
 _INSTALL_CURL_COMMAND = (
     "apt-get update && "
-    "DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends curl"
+    "DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends "
+    "curl ca-certificates"
 )
 
 _CHECKPOINT_COMMAND = (
@@ -69,7 +71,7 @@ def log_message(self, *_args):
 class SandboxIntegrationTest(unittest.TestCase):
     @classmethod
     def setUpClass(cls):
-        cls.sandbox = Sandbox(cpu=1000, memory=2048, runtime=_RUNTIME)
+        cls.sandbox = Sandbox(cpu=1000, memory=2048, runtime=_RUNTIME, image=_IMAGE)
 
     @classmethod
     def tearDownClass(cls):
@@ -96,6 +98,21 @@ def test_filesystem(self):
         )
         self.assertTrue(self.sandbox.files.exists("/tmp/akernel-integration.txt"))
 
+    @unittest.skipUnless(_IMAGE, "set AKERNEL_TEST_IMAGE to test an OCI/Nydus root")
+    def test_image_writes_are_private(self):
+        original = self.sandbox.files.read("/etc/os-release")
+        try:
+            self.sandbox.files.write("/etc/os-release", "AKERNEL_PRIVATE_ROOT\n")
+            with Sandbox(
+                cpu=1000, memory=2048, runtime=_RUNTIME, image=_IMAGE
+            ) as other:
+                self.assertEqual(other.files.read("/etc/os-release"), original)
+            self.assertEqual(
+                self.sandbox.files.read("/etc/os-release"), "AKERNEL_PRIVATE_ROOT\n"
+            )
+        finally:
+            self.sandbox.files.write("/etc/os-release", original)
+
     def test_pty(self):
         output = bytearray()
         with self.sandbox.pty.create(on_data=output.extend) as session:
@@ -179,6 +196,7 @@ def test_internal_checkpoint_reload_and_reverse_tunnel(self):
                 memory=2048,
                 storage_mb=256,
                 runtime=_RUNTIME,
+                image=_IMAGE,
                 reverse_tunnel=tunnel,
                 failover=True,
             )