From 14a80c78e60877defe95f3af0c775fb6f3bd64fe Mon Sep 17 00:00:00 2001 From: Tianyu Zhou Date: Thu, 10 Sep 2026 10:29:36 +0800 Subject: [PATCH 1/2] fix: declare container identity for systemd nodes Set container=oci in the final all-in-one image and explicitly pass it through Helm and standalone launchers so existing images also identify their PID 1 systemd as running inside a container. Without this identity, privileged shutdown can remount shared host filesystems read-only and leave replacement nodes unable to write logs or state. Check image metadata and PID 1 container detection in standalone CI, and document upgrade verification and the recovery order for mounts already affected by this issue. Signed-off-by: Tianyu Zhou --- .github/workflows/ci.yml | 14 ++++++ AGENTS.md | 7 +++ builder/node.Dockerfile | 3 ++ deploy/README.md | 49 +++++++++++++++++++ .../charts/core/templates/node/daemonset.yaml | 3 ++ deploy/standalone/start.sh | 1 + 6 files changed, 77 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5de3225..8c64cba 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -132,12 +132,26 @@ jobs: IMAGE_TAG="${GITHUB_SHA}" \ RUNTIME_PROFILE=rrt + - name: Verify systemd container identity in the image + run: | + docker image inspect "akernel-ci/all-in-one:${GITHUB_SHA}" \ + --format '{{json .Config.Env}}' | \ + python -c 'import json, sys; assert "container=oci" in json.load(sys.stdin)' + - name: Start standalone AKernel run: | IMAGE="akernel-ci/all-in-one:${GITHUB_SHA}" \ AKERNEL_NAT_BACKEND=iptables \ ./deploy/standalone/start.sh + - name: Verify PID 1 detects its container + run: | + docker exec akernel-node bash -euo pipefail -c ' + tr "\0" "\n" < /proc/1/environ | grep -x "container=oci" + test "$(cat /run/systemd/container)" = oci + systemd-detect-virt --container + ' + - name: Run SDK end-to-end examples run: | gateway_ip="$(docker inspect \ diff --git a/AGENTS.md b/AGENTS.md index b018765..64834a1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -183,6 +183,13 @@ Use [`deploy/README.md`](./deploy/README.md) as the deployment entry point. AKernel supports standalone, existing Kubernetes clusters via Helm, and Terraform-based cloud provisioning. +The all-in-one image and node launchers declare lowercase `container=oci` +for PID 1 systemd. Preserve this in the final image, Helm node environment, +and standalone launcher: without container detection, privileged systemd +shutdown can remount shared host filesystems read-only. See +[`deploy/README.md#systemd-container-identity`](./deploy/README.md#systemd-container-identity) +for upgrade verification and recovery of already affected mounts. + Aliyun's aggregate Pod PID budget is configurable independently of the per-sandbox limit; see `deploy/terraform/aliyun/README.md#pod-pid-budget`. diff --git a/builder/node.Dockerfile b/builder/node.Dockerfile index 16634b0..77ba145 100644 --- a/builder/node.Dockerfile +++ b/builder/node.Dockerfile @@ -233,6 +233,9 @@ COPY ./src/distill-fs/ ./ RUN cargo build --locked --release --bin distill_fs FROM ${AKERNEL_NODE_BASE_IMAGE} +# Let PID 1 systemd avoid remounting shared host filesystems during shutdown. +ENV container=oci + ARG AKERNEL_ENABLE_KATA ARG AKERNEL_ENABLE_RUNC ARG AKERNEL_ENABLE_FIRECRACKER diff --git a/deploy/README.md b/deploy/README.md index 63940e0..7942fa8 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -71,6 +71,55 @@ The node must support TC eBPF and bpffs. bpfnat does not manage host firewall policy, so custom host-network deployments must allow forwarding to and from the sandbox bridge when their `FORWARD` policy is `DROP`. +### systemd container identity + +The node and standalone roles run systemd as PID 1. The final all-in-one +image declares lowercase `container=oci`, and the Helm node template and +standalone launcher also pass it explicitly so older images receive the +same protection. Both Terraform providers use this Helm node template. +Keep this variable in custom launchers before starting systemd. + +Without container detection, privileged systemd shutdown can remount a +host-backed filesystem read-only at the superblock level, affecting the host +and replacement containers. `HostToContainer` mount propagation and masking +`systemd-remount-fs.service` do not prevent this shutdown path. See the +[systemd container interface](https://systemd.io/CONTAINER_INTERFACE/) and +[v255 shutdown implementation](https://github.com/systemd/systemd/blob/v255/src/shutdown/umount.c#L93-L121). + +Drain workloads before upgrading and replace the node Pod or standalone +container; exporting the variable in an exec shell does not change PID 1. +Inside the replacement container, verify: + +```bash +tr '\0' '\n' < /proc/1/environ | grep -x 'container=oci' +cat /run/systemd/container +systemd-detect-virt --container +``` + +The marker should contain `oci`; detection must succeed and may report +`container-other`. Verify the built image as well: + +```bash +docker image inspect --format '{{json .Config.Env}}' +``` + +Its environment must include `container=oci`. On the target container runtime, +verify normal shutdown, rolling replacement, and recreation with the actual +hostPath, filestore, and checkpoint mounts. Check both the mount flags and +the filesystem flags after ` - ` in `/proc/self/mountinfo`, confirm writes +still succeed, check sandboxd/YuanRong services and worker registration, and +run a basic sandbox create/execute/delete cycle. + +This fix does not restore an already read-only filesystem. First inspect +kernel logs and rule out disk I/O errors or filesystem damage. Replace the +old container with one that correctly identifies itself, since the old +container can still remount the filesystem during its final shutdown. Only +then, after confirming the affected mount and obtaining operator approval, +remount that specific filesystem read-write. Verify separate filestore and +checkpoint mounts too, and repeat normal replacement to confirm no second +repair is needed. Do not add unconditional startup remounts or clear sandbox +state as a workaround. + ### Network ACLs The bundled standalone, Helm, and Terraform sandboxd configurations enable diff --git a/deploy/akernel/charts/core/templates/node/daemonset.yaml b/deploy/akernel/charts/core/templates/node/daemonset.yaml index 01eb11d..a9e2561 100644 --- a/deploy/akernel/charts/core/templates/node/daemonset.yaml +++ b/deploy/akernel/charts/core/templates/node/daemonset.yaml @@ -76,6 +76,9 @@ spec: resources: {{- toYaml .Values.node.resources | nindent 10 }} env: + # Required for systemd shutdown safety, including with older images. + - name: container + value: "oci" - name: AKERNEL_ROLE value: "node" - name: RUNSC_AKERNEL diff --git a/deploy/standalone/start.sh b/deploy/standalone/start.sh index 8faed46..9d1a20a 100755 --- a/deploy/standalone/start.sh +++ b/deploy/standalone/start.sh @@ -361,6 +361,7 @@ start_node_container() { --privileged \ --net bridge \ --restart always \ + -e container=oci \ -e AKS_LOCAL_MODE="true" \ -e YR_RRT_CONTROL_SOCKET_PATH="/run/akernel" \ -e YR_IMAGE_PROCESS_CONFIG="${YR_IMAGE_PROCESS_CONFIG}" \ From 34c644446640fcfc3b37ebdb760dd10cc9110e29 Mon Sep 17 00:00:00 2001 From: Tianyu Zhou Date: Thu, 10 Sep 2026 10:33:44 +0800 Subject: [PATCH 2/2] docs: shorten systemd deployment guidance Keep the deployment guide focused on the container identity requirement, replacement requirement, and existing read-only filesystem limitation. Remove the detailed incident recovery and acceptance procedure, and update the agent guidance link description to match the shorter section. Signed-off-by: Tianyu Zhou --- AGENTS.md | 2 +- deploy/README.md | 51 +++++------------------------------------------- 2 files changed, 6 insertions(+), 47 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 64834a1..2c72a0e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -188,7 +188,7 @@ for PID 1 systemd. Preserve this in the final image, Helm node environment, and standalone launcher: without container detection, privileged systemd shutdown can remount shared host filesystems read-only. See [`deploy/README.md#systemd-container-identity`](./deploy/README.md#systemd-container-identity) -for upgrade verification and recovery of already affected mounts. +for deployment implications. Aliyun's aggregate Pod PID budget is configurable independently of the per-sandbox limit; see `deploy/terraform/aliyun/README.md#pod-pid-budget`. diff --git a/deploy/README.md b/deploy/README.md index 7942fa8..6788897 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -73,52 +73,11 @@ the sandbox bridge when their `FORWARD` policy is `DROP`. ### systemd container identity -The node and standalone roles run systemd as PID 1. The final all-in-one -image declares lowercase `container=oci`, and the Helm node template and -standalone launcher also pass it explicitly so older images receive the -same protection. Both Terraform providers use this Helm node template. -Keep this variable in custom launchers before starting systemd. - -Without container detection, privileged systemd shutdown can remount a -host-backed filesystem read-only at the superblock level, affecting the host -and replacement containers. `HostToContainer` mount propagation and masking -`systemd-remount-fs.service` do not prevent this shutdown path. See the -[systemd container interface](https://systemd.io/CONTAINER_INTERFACE/) and -[v255 shutdown implementation](https://github.com/systemd/systemd/blob/v255/src/shutdown/umount.c#L93-L121). - -Drain workloads before upgrading and replace the node Pod or standalone -container; exporting the variable in an exec shell does not change PID 1. -Inside the replacement container, verify: - -```bash -tr '\0' '\n' < /proc/1/environ | grep -x 'container=oci' -cat /run/systemd/container -systemd-detect-virt --container -``` - -The marker should contain `oci`; detection must succeed and may report -`container-other`. Verify the built image as well: - -```bash -docker image inspect --format '{{json .Config.Env}}' -``` - -Its environment must include `container=oci`. On the target container runtime, -verify normal shutdown, rolling replacement, and recreation with the actual -hostPath, filestore, and checkpoint mounts. Check both the mount flags and -the filesystem flags after ` - ` in `/proc/self/mountinfo`, confirm writes -still succeed, check sandboxd/YuanRong services and worker registration, and -run a basic sandbox create/execute/delete cycle. - -This fix does not restore an already read-only filesystem. First inspect -kernel logs and rule out disk I/O errors or filesystem damage. Replace the -old container with one that correctly identifies itself, since the old -container can still remount the filesystem during its final shutdown. Only -then, after confirming the affected mount and obtaining operator approval, -remount that specific filesystem read-write. Verify separate filestore and -checkpoint mounts too, and repeat normal replacement to confirm no second -repair is needed. Do not add unconditional startup remounts or clear sandbox -state as a workaround. +The all-in-one image, Helm node template, and standalone launcher set +`container=oci` so PID 1 systemd recognizes the container and does not remount +shared host filesystems read-only during shutdown. Preserve this variable in +custom launchers. Applying the fix requires replacing the node Pod or +standalone container; it does not repair an already read-only filesystem. ### Network ACLs