Multiple container VM support#254
Draft
dmcgowan wants to merge 22 commits into
Draft
Conversation
There was a problem hiding this comment.
Pull request overview
Adds per-pod “sandbox VM” support so multiple containers can run inside a single VM (enabling nerdbox to act as a real CRI RuntimeClass via containerd’s shim sandboxer). This spans host-side sandbox/TTRPC plumbing, guest init behavior, namespace + volume semantics, and updated conformance/stress coverage via vendored shimtest.
Changes:
- Introduces containerd sandbox shim API wiring (
runtime/sandbox/v1) and sandbox lifecycle support, including network-sandbox pinning + host netns entry for VM networking threads. - Implements shared pod-scoped guest namespaces (netns always; PID/IPC on-demand via a new
podnsTTRPC service and apod-pausePID1 anchor), plus pod config application (DNS/hostname/sysctls) and sandbox volume sharing through the shared virtiofs tree. - Expands testing/benchmarking with updated vendored
shimtest(sandbox suite + stress + benchmarks) and adds a CRI conformance harness undertest/critest.
Reviewed changes
Copilot reviewed 62 out of 100 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| vendor/modules.txt | Vendors new containerd sandbox API + k8s cri-api; bumps shimtest. |
| vendor/k8s.io/cri-api/pkg/apis/runtime/v1/constants.go | Adds vendored CRI constants used by pod config handling. |
| vendor/k8s.io/cri-api/LICENSE | Adds CRI API license text for vendored dependency. |
| vendor/github.com/containerd/shimtest/stress_suite.go | Adds sandbox stress option/override wiring. |
| vendor/github.com/containerd/shimtest/stress_sandbox_other.go | Skips sandbox stress on non-Linux. |
| vendor/github.com/containerd/shimtest/stress_sandbox_linux.go | Implements Linux sandbox churn + RSS/mount leak checks. |
| vendor/github.com/containerd/shimtest/sandbox_suite_volumes_linux.go | Adds sandbox member-container host volume contract test. |
| vendor/github.com/containerd/shimtest/sandbox_suite_shared_pidns_linux.go | Adds shared PID namespace contract test. |
| vendor/github.com/containerd/shimtest/sandbox_suite_shared_netns_linux.go | Adds shared netns loopback contract test. |
| vendor/github.com/containerd/shimtest/sandbox_suite_shared_ipcns_linux.go | Adds shared IPC namespace contract test. |
| vendor/github.com/containerd/shimtest/sandbox_suite_other.go | Skips sandbox suite on non-Linux. |
| vendor/github.com/containerd/shimtest/sandbox_suite_netns_linux.go | Adds “traffic scoped to provided network sandbox” test. |
| vendor/github.com/containerd/shimtest/sandbox_suite_member.go | Adds member-container contract tests (status, exec, UDS, outbound DNS, etc.). |
| vendor/github.com/containerd/shimtest/sandbox_suite_member_linux.go | Adds Linux-only member-container contract tests (netns pin/path, etc.). |
| vendor/github.com/containerd/shimtest/sandbox_bench_other.go | Skips sandbox benchmarks on non-Linux. |
| vendor/github.com/containerd/shimtest/sandbox_bench_linux.go | Adds sandbox benchmarks (per-container create cycle in shared VM). |
| vendor/github.com/containerd/shimtest/rootfs.go | Extends testbin commands used by new tests. |
| vendor/github.com/containerd/shimtest/README.md | Documents new sandbox/net features, tests, benchmarks, and skip options. |
| vendor/github.com/containerd/shimtest/helpers.go | Adds namespace + stdin request helpers. |
| vendor/github.com/containerd/shimtest/helpers_sandbox_other.go | Adds non-Linux sandbox helper stubs. |
| vendor/github.com/containerd/shimtest/helpers_realnetns_linux.go | Adds real netns tooling for traffic-origin tests. |
| vendor/github.com/containerd/shimtest/helpers_networksandbox_other.go | Adds non-Linux stubs for network sandbox helpers. |
| vendor/github.com/containerd/shimtest/helpers_networksandbox_linux.go | Adds Linux host-side network sandbox resource helper. |
| vendor/github.com/containerd/containerd/api/runtime/sandbox/v1/sandbox.proto | Vendors sandbox API proto. |
| vendor/github.com/containerd/containerd/api/runtime/sandbox/v1/sandbox_ttrpc.pb.go | Vendors generated TTRPC stubs for sandbox API. |
| vendor/github.com/containerd/containerd/api/runtime/sandbox/v1/doc.go | Adds sandbox package doc stub. |
| test/stress/stress_test.go | Enables sandbox stress and improves PATH handling to avoid stale libkrun. |
| test/shim/shim_test.go | Adds NetworkSuite + SandboxSuite and SandboxSuite benchmarks; improves PATH logic. |
| test/critest/cni-net.conflist | Adds CNI config for critest harness. |
| test/critest/build-dummy-pause.sh | Adds script to build a dummy pause image tar for CRI preflight. |
| test/critest/.gitignore | Ignores critest working dir and dummy pause tar. |
| plugins/shim/task/plugin.go | Wires SandboxService start-options callback into task service initialization. |
| plugins/shim/sandbox/service_plugin.go | Registers sandbox TTRPC service via dedicated TTRPCPlugin. |
| plugins/shim/sandbox/plugin.go | Introduces linux-only sandbox plugin and sandboxManager wrapper to avoid double TTRPC registration. |
| plugins/services/podns/service.go | Adds guest podns TTRPC service that ensures PID/IPC namespaces on demand. |
| pkg/vminit/initd/initd.go | Raises RLIMIT_NOFILE; creates shared pod netns; mounts optional shared containers virtiofs. |
| pkg/vminit/initd/containers_mount_linux.go | Mounts “containers” virtiofs share at /run/containers when available. |
| pkg/vm/vm.go | Adds StartOpts.NetNS + WithNetNS option and documents netns entry expectations. |
| pkg/vm/vm_test.go | Adds unit tests for WithNetNS. |
| pkg/shim/manager/mount_linux.go | Adjusts userns + mountns behavior to avoid ext4 mount restrictions for real root. |
| pkg/shim/manager/manager_windows.go | Makes missing config.json non-fatal in sandbox bundle start path (Windows). |
| pkg/shim/manager/manager_unix.go | Makes missing config.json non-fatal in sandbox bundle start path (Unix). |
| kernel/patches/0012-tsi-allow-hijacking-sockets-tsi_hijack.patch | Updates kernel patch metadata/content for TSI socket hijack behavior. |
| kernel/patches/0010-virtio-vsock-implement-has_data-for-DGRAM.patch | Adds vsock DGRAM SIOCINQ support plumbing via has_data. |
| kernel/patches/0009-vsock-Add-support-for-SIOCINQ-ioctl.patch | Adds upstream vsock SIOCINQ ioctl support patch. |
| internal/vminit/socketforward/socketforward.go | Ensures forwarded UDS sockets are connectable from user-namespaced processes (chmod). |
| internal/vminit/process/init.go | Removes guest bundle dir on container delete to prevent /run tmpfs growth. |
| internal/vminit/podpause/podpause.go | Adds pod-pause PID1 anchor process for shared PID namespaces. |
| internal/vminit/podns/podns.go | Implements on-demand guest IPC+PID namespace creation + bind-mounting. |
| internal/vminit/podnetns/podnetns_linux.go | Creates persistent guest shared pod netns and brings up loopback. |
| internal/vm/libkrun/krun.go | Improves naming/comments; documents thread inheritance requirements for libkrun start. |
| internal/vm/libkrun/krun_other.go | Adds non-Linux no-op netns entry hook. |
| internal/vm/libkrun/krun_linux.go | Implements netns entry with nsfs validation + EPERM handling. |
| internal/vm/libkrun/instance.go | Locks VM start goroutine to OS thread; enters pod netns before starting libkrun; validates netns consistency. |
| internal/vm/libkrun/instance_test.go | Tests netns consistency enforcement logic. |
| internal/shim/task/socketforward.go | Adds rootfs placeholder creation for UDS bind mount destinations. |
| internal/shim/task/sandboxvolumes.go | Rewrites bind mounts to use sandbox shared virtiofs tree via ShareVolume. |
| internal/shim/task/sandboxopts.go | Parses sandbox bundle to derive VM start options; injects resolv.conf defaults. |
| internal/shim/task/podns.go | Adds lazy guest RPC to ensure shared IPC/PID namespaces. |
| internal/shim/task/podnetns.go | Sanitizes host namespace paths; maps to guest shared namespaces as needed. |
| internal/shim/task/podnetns_test.go | Adds tests for namespace sanitization behavior. |
| internal/shim/task/podconfig.go | Applies PodSandboxConfig (hostname/sysctls) and decodes options payload. |
| internal/shim/task/podconfig_test.go | Adds tests for pod config decoding + application. |
| internal/shim/task/ctrnetworking.go | Generates resolv.conf from CRI DNSConfig; improves host resolv.conf selection with systemd-resolved handling. |
| internal/shim/task/apparmor.go | Strips host AppArmor profile from spec to avoid guest runtime failures. |
| internal/shim/task/apparmor_test.go | Adds tests for AppArmor profile stripping. |
| internal/shim/sandbox/vm/vm.go | Passes host netns path into VM start options (WithNetNS). |
| internal/shim/sandbox/sandbox.go | Adds NetnsPath option and WithNetnsPath setter. |
| internal/shim/sandbox/networksandbox.go | Adds cross-platform NetworkSandbox abstraction and factory hook. |
| internal/shim/sandbox/networksandbox_other.go | Implements non-Linux NetworkSandbox as no-op. |
| internal/shim/sandbox/networksandbox_linux.go | Pins host netns resource via open FD for sandbox lifetime. |
| internal/podns/podns.go | Defines shared guest IPC/PID namespace paths. |
| internal/podnetns/podnetns.go | Defines shared guest netns name/path constants. |
| go.sum | Adds checksums for shimtest bump and cri-api. |
| go.mod | Bumps shimtest and adds k8s cri-api dependency. |
| Dockerfile | Makes kernel patch application strict (--fuzz=0) and notes runtime /run/containers. |
| cmd/vminitd/main.go | Adds hidden pod-pause subcommand; registers podns service plugin. |
| api/services/podns/v1/podns_ttrpc.pb.go | Adds generated TTRPC service stubs for podns. |
| api/proto/nerdbox/services/podns/v1/podns.proto | Adds podns service definition (EnsureNamespaces). |
| api/next.txtpb | Updates Buf/descriptor snapshot for new podns proto. |
| .gitattributes | Disables whitespace checks for kernel patch files. |
Files not reviewed (2)
- api/services/podns/v1/podns.pb.go: Generated file
- api/services/podns/v1/podns_ttrpc.pb.go: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Init.delete unmounted the container's rootfs and removed crun's container state, but never removed the guest bundle directory itself (/run/bundles/<id>, created by the Bundle TTRPC service). /run is a size-limited tmpfs, so leaving these directories behind causes it to fill up over many container lifecycles on a single VM — previously unnoticeable on the legacy path (one container per VM, VM torn down after), but a real problem once a single VM can host many member containers created and deleted over its lifetime. Fix: once the rootfs has been unmounted and crun's own state removed, remove the bundle directory too. Signed-off-by: Derek McGowan <derek@mcg.dev>
Comment on lines
+52
to
+56
| // Unwrap the sandboxManager to get the underlying SandboxService. | ||
| type sandboxManagerUnwrapper interface { | ||
| Service() *intsandbox.SandboxService | ||
| } | ||
| svc := sbRaw.(sandboxManagerUnwrapper).Service() |
Comment on lines
+38
to
+44
| // Unwrap the sandboxManager to get the *SandboxService. | ||
| // The SandboxService implements both the Sandbox interface and the | ||
| // containerd TTRPCSandboxService. Returning it here (as a | ||
| // TTRPCPlugin) causes the shim framework to call RegisterTTRPC | ||
| // exactly once, registering the sandbox TTRPC service. | ||
| return sbRaw.(*sandboxManager).Service(), nil | ||
| }, |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 57 out of 94 changed files in this pull request and generated 1 comment.
Files not reviewed (2)
- api/services/podns/v1/podns.pb.go: Generated file
- api/services/podns/v1/podns_ttrpc.pb.go: Generated file
Comments suppressed due to low confidence (1)
plugins/shim/task/plugin.go:62
- The Sandbox plugin now returns
*internal/shim/sandbox.SandboxService, but this code expects the plugin to implement a customService() *SandboxServiceunwrapping interface. That assertion will fail at init time (and prevent the task plugin from wiringRegisterStartOptions), because*SandboxServicedoesn’t provideService().
Instead, treat the SandboxPlugin instance as the *intsandbox.SandboxService directly (type assert to it).
Comment on lines
+85
to
+95
| // GuestRootfsPath returns the in-guest path of the container's assembled | ||
| // rootfs, suitable for passing to the guest Task.Create as the rootfs source. | ||
| func GuestRootfsPath(containerID string) string { | ||
| return filepath.Join(GuestContainersDir, containerID, "rootfs") | ||
| } | ||
|
|
||
| // GuestVolumePath returns the in-guest path for volume mount n of the given | ||
| // container (0-indexed), suitable for bind-mounting into the container. | ||
| func GuestVolumePath(containerID string, n int) string { | ||
| return filepath.Join(GuestContainersDir, containerID, "volumes", fmt.Sprintf("%d", n)) | ||
| } |
Comment on lines
+71
to
+73
| if err := mountutil.All(ctx, hostRootfs, lmounts, mounts); err != nil { | ||
| return "", fmt.Errorf("mount container rootfs for %s: %w", containerID, err) | ||
| } |
Comment on lines
+161
to
+163
| } | ||
| f, err := os.OpenFile(destInRootfs, os.O_CREATE|os.O_EXCL, 0o666) | ||
| if err != nil && !os.IsExist(err) { |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 59 out of 96 changed files in this pull request and generated no new comments.
Files not reviewed (2)
- api/services/podns/v1/podns.pb.go: Generated file
- api/services/podns/v1/podns_ttrpc.pb.go: Generated file
Comments suppressed due to low confidence (1)
internal/shim/sandbox/sharedfs.go:95
- GuestRootfsPath/GuestVolumePath build guest (POSIX) paths using filepath.Join, which will emit backslashes on Windows builds. Even if sandbox mode is Linux-only today, keeping guest paths OS-independent avoids cross-compilation surprises and makes the helpers correct by construction.
// GuestRootfsPath returns the in-guest path of the container's assembled
// rootfs, suitable for passing to the guest Task.Create as the rootfs source.
func GuestRootfsPath(containerID string) string {
return filepath.Join(GuestContainersDir, containerID, "rootfs")
}
// GuestVolumePath returns the in-guest path for volume mount n of the given
// container (0-indexed), suitable for bind-mounting into the container.
func GuestVolumePath(containerID string, n int) string {
return filepath.Join(GuestContainersDir, containerID, "volumes", fmt.Sprintf("%d", n))
}
Comment on lines
+211
to
+214
| nofileLimit := unix.Rlimit{Cur: 65536, Max: 65536} | ||
| if err := unix.Setrlimit(unix.RLIMIT_NOFILE, &nofileLimit); err != nil { | ||
| log.G(ctx).WithError(err).Warn("failed to raise RLIMIT_NOFILE; FD exhaustion may occur under sustained load") | ||
| } |
Implements the containerd sandbox API (runtime/sandbox/v1) so that one
VM can host multiple containerd-managed containers ("member
containers") for the lifetime of a single pod, instead of nerdbox's
existing one-VM-per-container model. This is what lets nerdbox be used
as a real CRI RuntimeClass handler via containerd's built-in shim
sandboxer (sandboxer = "shim").
## SandboxService (internal/shim/sandbox/service.go, new)
Implements TTRPCSandboxService: CreateSandbox/StartSandbox — boots the
VM with resources/networking derived from the sandbox bundle via a
StartOptionsFunc callback registered by the task plugin, avoiding a
circular import between the sandbox and task packages —
StopSandbox/ShutdownSandbox, SandboxStatus (reporting the CRI v1
PodSandboxState enum's exact "SANDBOX_READY"/"SANDBOX_NOTREADY" names,
not an invented vocabulary, since containerd's CRI layer derives
PodSandboxStatus.State by looking the string up in that enum's
name-to-value map), Platform, and Ping.
Also owns pinning the sandbox's host-side network namespace for the
VM's lifetime (internal/shim/sandbox/networksandbox.go,
_linux/_other.go) and threading it down to libkrun: internal/vm/libkrun
serializes every FFI call for a VM context onto one dedicated,
permanently OS-thread-locked goroutine (vmExecutor), which is required
for correct namespace scoping — entering a network namespace via
setns(2) only affects the calling thread, and without a dedicated
thread the Go scheduler could run different krun_* calls (and the
worker threads libkrun spawns from them) in different namespaces.
SetNetnsPath enters the namespace on that thread before any other
configuration call. This ensures every container in the sandbox
actually uses the CNI-assigned network the sandbox was created with,
which is foundational regardless of any other pod feature.
## SharedFS (internal/shim/sandbox/sharedfs.go, new)
Member-container rootfs assembly happens entirely on the host: each
container's rootfs is assembled (overlay/erofs mounts, using
nerdbox's own internal/mountutil.All rather than the generic vendored
mount.All, which rejects nerdbox's custom "X-containerd.mkdir.*"
options) inside a per-sandbox directory tree exposed to the guest over
a single, persistent, pre-boot virtiofs share — avoiding the need to
hot-add a virtiofs device per container, which libkrun does not
support.
## Task service wiring (internal/shim/task/service.go)
Task.Create now branches on whether the sandbox API is in use
(IsSandboxed()): createSandboxedContainer resolves the rootfs via
SharedFS and drives the already-running VM's guest bundle/mount/task
RPCs directly, instead of createLegacyContainer's existing
boot-a-fresh-VM-per-container path (preserved unchanged for
non-sandboxed use). The per-sandbox VM event stream is now started
exactly once (eventStreamOnce) regardless of how many member
containers are created. internal/shim/task/socketforward.go gains
CreateRootfsPlaceholders, needed because a sandboxed container's
source rootfs must have UDS bind-mount placeholder files created
before SharedFS's read-only share is assembled (the legacy path's
rootfs isn't shared the same way, so this was never needed there).
## Plugin wiring
plugins/shim/sandbox: the SandboxPlugin now wraps the raw VM-backed
Sandbox in a SandboxService, exposed via a dedicated TTRPCPlugin
"sandbox" (service_plugin.go) rather than the SandboxPlugin itself, to
avoid a double-registration panic when the shim framework looks for
TTRPCService implementors. plugins/shim/task wires the task plugin's
NewTaskService to the same SandboxService instance and registers its
StartOptionsFunc callback.
## Guest side
pkg/vminit/initd/containers_mount_linux.go mounts the sandbox's shared
virtiofs tree at /run/containers at vminitd startup (best-effort:
absent/no-op on the legacy path, where the "containers" tag is never
registered by the host). initd.go also raises RLIMIT_NOFILE — the
kernel default (1024) is too low for a single VM now hosting many
container lifecycles' worth of inotify FDs from OOM monitoring under
sustained churn.
## Tests
test/shim/shim_test.go and test/stress/stress_test.go wire in
shimtest's new SandboxSuite (lifecycle, platform, ping, single/multiple
member containers, per-container independence, error cases) and its
stress/benchmark counterparts.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Adds the host-side and guest-side halves of per-sandbox network
namespace sharing for member containers, addressing two gaps found
while auditing how a member container's OCI spec network namespace is
handled:
1. No code anywhere sanitized spec.Linux.Namespaces before pushing a
container's bundle into the guest. In production CRI, a member
container's spec sets the network namespace entry's Path to a host
path (containerd's WithPodNamespaces sets it to
"/proc/<sandboxPid>/ns/net"), which is meaningless (or actively
wrong, if it happens to collide with something) once copied
verbatim into the guest — a different kernel with an unrelated
PID/namespace space entirely. The same applies to a host Path on
any other namespace type.
2. There was no mechanism to ensure all member containers of one
sandbox land in the same guest network namespace. Each container
got its own namespace (crun's default for an empty-Path entry, or
whatever the incoming host path happened to not-quite-mean), so two
containers in the same pod had no shared network identity inside
the guest.
internal/podnetns: shared Path/Name constants ("/run/netns/pod") used
by both halves below, kept in a separate no-build-tag package so
neither the host transformer nor the guest creator need to import
platform-specific code from the other.
internal/vminit/podnetns: guest-side creation. At vminitd startup,
creates a persistent, named network namespace at the well-known path
using the same "persistent netns" technique containerd's CRI plugin
uses on the host (see docs/sandbox-architecture.md, Layer 1): a
dedicated goroutine locks an OS thread, unshares a new network
namespace, and bind-mounts it — the bind-mount is what keeps it alive,
so the creating goroutine doesn't need to stay running. No explicit
teardown: the namespace is guest kernel state and disappears with the
VM.
internal/shim/task/podnetns.go: sanitizeNamespaces, a new bundle
transformer wired into createSandboxedContainer. Strips host Path
values from every namespace type (meaningless in the guest either
way), and rewrites/adds the network namespace entry to point at the
shared guest path — unless the container has its own dedicated
virtio-NIC (io.containerd.nerdbox.ctr.network.* annotation), in which
case it keeps its own separate namespace so the existing per-container
veth/bridge wiring in internal/vminit/ctrnetworking (which assumes a
container owns its namespace) is unaffected.
This is distinct from the host-side network namespace *pinning* that
SandboxService already performs for the VM as a whole (previous
commit): that ensures the VM's outbound traffic uses the CNI-assigned
network at all; this is about giving member containers of one pod a
shared L2/L3 view of *each other* inside the guest, matching real pod
semantics. Neither changes host reachability semantics: TSI is not
network-namespace-aware (documented below as "TSI ignores
guest-internal network namespaces" — its socket hijack triggers on
address family alone, before any netns-aware routing, and its vsock
channel to the host isn't real IP routing, so it cannot be scoped or
filtered by anything inside the guest; verified empirically, a
container placed in its own fresh guest network namespace still
reaches a host TCP listener via TSI unchanged). The guest kernel is
effectively a single network namespace with respect to host
reachability when TSI is enabled; the only real host-isolation
boundary is the host-side pod netns pinning, not anything configurable
inside the guest.
Verified end-to-end via a new shimtest test (MemberContainersShareNetwork):
two independently created member containers, one running an in-guest TCP
listener and one connecting to it via loopback, successfully exchange
data. New unit tests for sanitizeNamespaces in
internal/shim/task/podnetns_test.go.
Signed-off-by: Derek McGowan <derek@mcg.dev>
## Bind-mount volumes (internal/shim/sandbox/sharedfs.go, new
internal/shim/task/sandboxvolumes.go)
Member-container bind mounts (Kubernetes hostPath volumes, and CRI's
own injected UDS/sandbox-file mounts) previously went through
bindMounter, which shares each mount as a brand new virtiofs tag. That
only works on the legacy path, which boots a fresh VM per container and
can add the share before boot (sandbox.WithFS); a sandbox member
container is created against an already-running VM, and virtio-fs
shares cannot be hot-added after boot, so every such mount failed with
"mount ... fstype: virtiofs ... invalid argument".
Fix: SharedFS.ShareVolume bind-mounts the host source directly into the
sandbox's shared directory tree, which is already exposed to the guest
via the single, persistent, pre-boot "containers" virtiofs share (the
same one ShareRootfs uses) — no new virtiofs device, no extra
guest-side mount step. sandboxVolumeMounter (the sandboxed-path
counterpart to bindMounter) rewrites each bind mount's spec Source to
the resulting guest path and otherwise leaves Options untouched, so
crun's own bind mount from that path into the container enforces
whatever read-only/recursion/propagation the spec actually requested.
This uncovered a real bug in the first attempt: mounting read-only at
the *host* share layer (matching the container's own readonly flag)
looked reasonable but is actively wrong — Linux sets MNT_LOCKED on
every mount in a recursive read-only bind, and that lock cannot be
undone by any later mount (crun's own, or a fresh mount the container
creates inside it), so a *non-recursive* read-only container request
would become unintentionally, unremovably read-only all the way down.
Fixed by always sharing read-write at the host layer and leaving
read-only enforcement to crun's own (correctly-scoped) bind mount,
which is the only place it should happen exactly once.
New shimtest test (vendored): MemberContainerHostVolume, proving the
mount is a live share (a host-side update after the container starts
becomes visible inside it), not a one-time copy.
## Pod-level DNS/hostname/sysctls (new internal/shim/task/podconfig.go,
internal/shim/sandbox/service.go, internal/shim/task/ctrnetworking.go)
containerd's CRI layer normally writes resolv.conf/hostname/hosts files
into a sandbox root directory and bind-mounts them into every member
container (internal/cri/server's linuxContainerMounts +
podsandbox.Controller.setupSandboxFiles upstream) — but only the
podsandbox controller creates those files; the shim sandboxer path
(what this shim implements) gets none of them from containerd, so
those bind mounts are silently skipped and DNS/hostname/sysctls from
the pod's config never reached containers.
SandboxService now stores CreateSandboxRequest.Options verbatim
(deliberately uninterpreted — see its doc comment for why the sandbox
package stays CRI-agnostic) and exposes it via a new Options() getter.
The task package, which already owns other CRI-shaped concerns (DNS
annotations), unmarshals it as a k8s.io/cri-api PodSandboxConfig
(podSandboxConfig) and feeds DnsConfig/Hostname/Linux.Sysctls into:
- addResolvConf: now takes an optional pod DNSConfig, used when there
is no per-container DNS annotation override (existing priority
order otherwise unchanged).
- addHostname (new): sets spec.Hostname and bind-mounts /etc/hostname,
mirroring what the podsandbox controller does.
- addSysctls (new): merges pod sysctls into spec.Linux.Sysctl,
without overriding any per-container value already present.
Uses the k8s.io/cri-api v0.36.0 dependency added in the earlier vendor
commit (matching the k8s.io/cri-api version containerd v2.3.2 itself
pins) — the only way to unmarshal CreateSandboxRequest.Options into the
exact same generated protobuf type CRI marshaled it from.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Implements pod-level PID and IPC namespace sharing for sandbox member
containers, completing the pod namespace parity work started with the
shared network namespace (internal/podnetns) and pod volumes/DNS/
hostname/sysctls (previous commit).
containerd's WithPodNamespaces sets a host path (derived from the
sandbox's own host PID, e.g. "/proc/<sandboxPid>/ns/ipc") on the IPC
namespace entry of every member container's OCI spec (Kubernetes pods
share IPC by default), and on the PID namespace entry whenever the
pod's PID sharing mode isn't per-container. That host path is
meaningless in the guest, so the shim must recognize the request and
substitute a guest-side equivalent, the same way it already does for
the network namespace.
Guest side:
- internal/podns: well-known guest paths for the shared IPC
(/run/ipcns/pod) and PID (/run/pidns/pod) namespaces.
- internal/vminit/podns: Manager.EnsureNamespaces creates both
namespaces on demand (the first time any container asks for pod
namespace sharing), memoized with a sticky error. The IPC
namespace is created the same way as the shared network namespace
(unshare on a locked OS thread + bind-mount). The PID namespace
cannot be: unshare(CLONE_NEWPID) does not move the caller, only
the next forked child becomes PID 1, and a PID namespace is torn
down the instant its PID 1 exits. So the PID namespace is anchored
by a real, persistent process instead (internal/vminit/podpause):
vminitd re-execs itself with a hidden "pod-pause" argument and
CLONE_NEWPID, and that process reaps reparented orphans and
ignores every signal except SIGKILL for the sandbox's lifetime.
- plugins/services/podns: TTRPC plugin registration exposing
Manager.EnsureNamespaces as the PodNamespaces service (new proto:
api/proto/nerdbox/services/podns/v1).
Host side:
- internal/shim/task/podnetns.go: sanitizeNamespaces now also
rewrites IPC/PID namespace entries. Any non-empty incoming Path on
either type is treated as "share within this pod" and redirected
to the guest's shared namespace; an absent entry (the common case:
no pod-level sharing requested) keeps its own namespace and never
triggers the guest RPC. This deliberately does not distinguish
NamespaceMode_NODE (hostPID/hostIPC) from NamespaceMode_POD:
containerd derives the same host path for both, so the shim
cannot tell them apart from the data it receives, and both only
need cross-container-within-the-pod visibility to satisfy real
CRI conformance checks (see test/critest/README.md).
- internal/shim/task/podns.go: sharedNamespaces, a lazy,
sync.Once-memoized TTRPC client wrapper around the guest's
EnsureNamespaces call, so a container whose spec never asks for
PID/IPC sharing never pays for it.
- internal/shim/task/service.go: fetch the VM client earlier in
createSandboxedContainer so sanitizeNamespaces's transformer can
use it during bundle.Load.
shimtest: MemberContainersSharePID and MemberContainersShareIPC tests
(vendored from the shimtest module) verify cross-container PID
visibility via /proc and SysV shared memory visibility via new
pidscan/shmwrite/shmread testbin commands.
critest conformance improved from 81 passed / 8 failed / 24 skipped to
85 passed / 4 failed / 24 skipped. HostPID, HostIpc is false, and
PodPID now pass; the remaining IPC-related failure (HostIpc is true)
plants a SysV shm segment on the real host machine running critest
before creating the sandbox, which no VM-internal namespace can make
visible to guest processes -- the same class of limitation as the
pre-existing HostNetwork is true gap, now documented in
docs/sandbox-architecture.md's new "Pod PID and IPC namespace
sharing" section and test/critest/README.md.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Adds test/critest/, a self-contained harness that drives a dedicated containerd instance through a real CRI RuntimeClass-style runtime handler backed by containerd's built-in shim sandboxer (sandboxer = "shim", not podsandbox) — i.e. exercising the same CreateSandbox/StartSandbox/StopSandbox/ShutdownSandbox TTRPC path a real Kubernetes RuntimeClass would use, as opposed to shimtest's direct TTRPC harness or the ctr-driven examples in README.md/docs/. - run-critest.sh: generates a containerd v3 config (CRI runtime handler pointing at io.containerd.nerdbox.v1 with sandboxer=shim and snapshotter=erofs, alongside an unchanged "runc" handler as a comparison baseline; erofs snapshotter/differ load with no extra config on Linux, and containerd's transfer plugin already ships a built-in "erofs" unpack_config entry per runtime.snapshotter overrides — see plugins/transfer/plugin_defaults_linux.go upstream), starts/stops containerd, and provides up/down/smoke/critest/shell subcommands. - build-dummy-pause.sh: builds a deliberately non-functional OCI image (valid manifest+config, empty layer) used as the pinned CRI sandbox image. See the script's own comment and README.md for why a non-functional image is the right choice here: on the shim-sandboxer path the pause image only needs to resolve in containerd's image store (CRI's ensurePauseImageExists), never mount or run, so making it deliberately incapable of running proves nothing ever depends on it actually working. - cni-net.conflist: minimal bridge+portmap+loopback CNI config, isolated in this harness's own work dir (does not touch /etc/cni/net.d). - README.md: prerequisites, usage, the dummy-pause rationale, and a summary of the conformance run against everything implemented so far in this series (VM-per-pod sandboxing, shared network/PID/IPC namespaces, bind-mount volumes, and pod-level DNS/hostname/sysctls): 85 passed / 4 failed / 24 skipped. The 4 remaining failures are genuine architectural limitations of running each sandbox in its own VM kernel, not bugs, detailed in the README's "Known conformance gaps" section (mount propagation and non-recursive readonly mounts needing a live host kernel mount-graph virtiofs cannot represent, and HostNetwork/HostIpc "is true" checks needing the guest to observe state that only exists on the literal machine running critest, a different kernel entirely from the sandbox's own). Signed-off-by: Derek McGowan <derek@mcg.dev>
Adds a default --ginkgo.skip list to run-critest.sh's "critest"
subcommand covering the 4 specs confirmed to be permanent architectural
limitations of running each sandbox in its own VM kernel, not
implementation bugs (see README.md's "Known conformance gaps"):
- runtime should support HostIpc is true
- runtime should support HostNetwork is true
- mount with 'rshared' should support propagation from host to
container and vice versa
- should support non-recursive readonly mounts
A --no-skip flag runs the full, unfiltered suite (still 85/4/24, same
as before this change); the default run is now 85/0/28 -- a genuinely
green result rather than one that always requires reading past known
failures. This mirrors the posture of Kata Containers (the most mature
production VM-isolated CRI runtime), which excludes tests that assume
shared-kernel/host-visibility semantics rather than treating them as
bugs to chase -- documented in the new README.md comparison paragraph.
Root-caused the sibling 'rslave' propagation spec along the way: it
passes and needs no skip entry. This uncovered a more precise root
cause for 'rshared' than previously documented: the host->container
propagation direction actually works (confirmed by 'rslave', which
tests only that direction) -- the test's own setup marks the volume's
host source MS_SHARED, and per mount_namespaces(7) a later bind mount
taken from an already-shared mount joins the same peer group, which is
exactly what SharedFS.ShareVolume's plain bind mount does, so a mount
added on the host after container start lands in the same host-kernel
peer group as our virtiofs-shared copy and virtiofs simply serves the
updated content. Only the container->host direction is truly
impossible: a guest-internal mount(2) syscall is never relayed by
virtiofs's content-only protocol, so the host kernel can never observe
it, regardless of any peer-group configuration. README.md's explanation
of both mount-related failures is corrected accordingly.
Also fixed a latent, previously-undiscovered usage bug: the script's
own header comment recommended "critest -- ARGS", but a literal "--"
is consumed by critest's own (ginkgo/go test) flag parser as "stop
parsing flags", which silently disables every flag after it -- so the
documented usage pattern silently broke --ginkgo.focus/--ginkgo.skip.
Usage comments and README.md corrected to pass ARGS directly with no
separator.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Rewrites several sections that had accumulated development-history narrative (bug descriptions, fixes, "verified via test X", conformance run results) into plain descriptions of the current design, and removes the "Known limitation" framing from sections describing permanent, not-planned-to-change behavior (TSI's relationship to guest network namespaces and the host socket table, in favor of just describing what TSI does and does not do). The one-time TSI wire-protocol fix section is removed entirely, since it describes a resolved historical issue with no bearing on the current architecture. No content describing the current design or its rationale is removed; only the history/limitation framing around it. The one exception is "Future work", which is left as-is since it is genuinely forward-looking. Signed-off-by: Derek McGowan <derek@mcg.dev>
…hread
Previously, all libkrun FFI calls (including krun_create_ctx and every
krun_add_* configuration call) were serialized onto a dedicated,
permanently-locked "executor" OS thread, and loading libkrun plus
creating the VM context was deferred until the first such call so that
a namespace recorded by SetNetnsPath could be entered on that thread
before anything else ran on it.
This was overcautious. Auditing the vendored libkrun Rust source shows
it spawns no threads at load time or during any krun_set_*/krun_add_*
configuration call -- those only mutate a global context map. Every
thread relevant to networking (vCPU, virtio-net, vsock/TSI muxer and
reaper) is spawned exclusively inside krun_start_enter, as a
descendant of whichever thread calls it. So the only requirement is
that the goroutine calling krun_start_enter has entered the pod netns
first, via runtime.LockOSThread + setns immediately before the call --
there is no need to route every configuration call through that same
thread, and no need to defer loading the library or creating the
context.
Simplify accordingly:
- Remove vmExecutor and vmcontext.ensureLoaded entirely. vmcontext
methods call libkrun directly again; newvmcontext now creates the
krun context immediately (krun_create_ctx), returning an error on
failure like every other vmcontext method.
- NewInstance goes back to eagerly opening libkrun, initializing
logging, creating the VM context, and adding the reserved rootfs
disk, all synchronously, rather than deferring any of it.
- vmInstance.Start locks its goroutine to its OS thread with
runtime.LockOSThread (never unlocked -- Go retires the thread when
the goroutine exits) and, if a netns was requested, enters it via
setns immediately before calling krun_start_enter. The resulting
thread's netns inode is logged for cross-checking during
debugging.
- Instance.SetNetnsPath is removed from the pkg/vm interface. The
pod netns is now supplied via a new vm.WithNetNS StartOpt,
consumed by Start itself instead of a separate pre-Start call.
- vmInstance now guards against a Start retry silently changing the
requested netns: the first Start call (successful or not) records
the requested namespace; a later Start call with the same
namespace, or none, is a no-op, but a genuinely different,
non-empty namespace is rejected outright rather than silently
overriding the original request.
No behavioral change to which namespace ends up hosting the VM's
worker threads: krun_start_enter and everything it spawns still runs
strictly after the setns call, on the same locked thread.
Signed-off-by: Derek McGowan <derek@mcg.dev>
CRI sets Process.ApparmorProfile to the name of an AppArmor profile loaded on the host (via a pod's appArmorProfile field or the deprecated container.apparmor.security.beta.kubernetes.io annotation). That name is meaningless inside the VM guest -- the guest kernel may not have AppArmor enabled at all, and even if it does, it never loaded a profile by that name. Left unmodified, the guest's crun invocation fails outright trying to apply an unknown profile, so any pod requesting an AppArmor profile could never start. Add clearApparmorProfile, a bundle.Transformer that strips the field, and wire it into both the sandboxed and legacy container creation paths, alongside the existing host-namespace-path sanitization this shim already does for the same class of "host-specific setting that is meaningless in a nested guest kernel" problem. Co-authored-by: Kern Walster <kern.walster@gmail.com> Signed-off-by: Derek McGowan <derek@mcg.dev>
Signed-off-by: Derek McGowan <derek@mcg.dev>
go.mod pinned github.com/containerd/shimtest to v0.3.0 (the tagged release used by the network-suite work on main), but the sandbox work in this branch requires the sandbox-tests branch commit a8efaacdfbb80cad816b97198e4d8118a4be05ed, which adds SandboxSuite, StressSuite's sandbox stress test, and the network-sandbox test helpers. A prior rebase resolved a go.mod conflict between the two by keeping v0.3.0, silently dropping the pseudo-version pointing at that commit -- vendor was never regenerated to match, so every build referencing shimtest.NewSandboxSuite (test/shim/shim_test.go) or StressOptions.Sandbox/SandboxRSSGrowthOverride (test/stress/stress_test.go) failed. Fix: re-resolve the same commit to a pseudo-version based on its actual ancestry (v0.3.1-0.20260712075910-a8efaacdfbb8, now that v0.3.0 is an ancestor) via 'go get', then 'go mod tidy' and 'go mod vendor' to bring in the sandbox suite files that were missing from vendor. Signed-off-by: Derek McGowan <derek@mcg.dev>
CI reported several failures on this branch, all from the same root
cause: internal/shim/sandbox/service.go and sharedfs.go (SandboxService,
SharedFS, StartOptionsFunc) were gated //go:build linux despite having
no actual Linux-specific code (sharedfs.go's real mount logic is the
only genuinely Linux-only part), and plugins/shim/sandbox/plugin.go and
service_plugin.go carried the same tag with no non-Linux fallback file
at all. Every non-Linux build failed:
- Build (darwin/*), Build (windows/*): cmd/containerd-shim-nerdbox-v1
failed outright ("build constraints exclude all Go files in
plugins/shim/sandbox").
- Unit Tests (macos-latest, windows-latest): internal/shim/task
failed to compile (undefined: sandbox.SharedFS/SandboxService/
StartOptionsFunc), since it references these unconditionally.
Fix:
- Remove the Linux-only tag from service.go (nothing in it is
platform-specific).
- Split sharedfs.go: the struct, constants, and simple accessors stay
cross-platform; the real mount-based implementation of
ShareRootfs/ShareVolume/Unshare/UnshareAll moves to a new
sharedfs_linux.go, with a sharedfs_other.go stub returning a
not-supported error on other platforms (mirroring the existing
networksandbox_linux.go/_other.go split).
- Remove the Linux-only tag from the two plugin registration files;
their own dependencies (internal/shim/sandbox/vm, pkg/vm) were
already cross-platform.
Separately, Project Checks (file headers) failed on the same set of
files plus a few others (internal/vm/libkrun/krun_linux.go/krun_other.go,
internal/shim/task/sandboxopts.go, pkg/vminit/initd/
containers_mount_linux.go, test/critest/*.sh): they used a "//"
line-comment license header (and, where present, put a build tag
*after* the header) instead of the project's established "/* */"
block-comment form with any build tag placed *before* it. Reformatted
all of them to match every other file in the tree exactly (verified
byte-for-byte against the canonical header), and normalized the two
shell scripts' comment spacing (blank lines between paragraphs, per
convention) to match.
Signed-off-by: Derek McGowan <derek@mcg.dev>
linuxOpenNetworkSandbox's comment claimed to verify the given path "looks like a network namespace", but the code only did a plain unix.Stat existence check -- any regular file would be accepted and pinned as if it were a real netns. Add the nsfs-magic check (mirroring the identical, already-existing validation in internal/vm/libkrun's vmcontextSetNetns) via Fstatfs on the opened FD. A path that isn't backed by nsfs is still accepted rather than rejected: shimtest's non-root test suite deliberately pins a plain regular file in place of a real netns to avoid requiring CAP_SYS_ADMIN or kernel bind-mount support, and that technique must keep working. A debug log line now flags this case for visibility. Also fixes the file's license header to the project's standard block-comment form (flagged by the Project Checks CI job -- see the previous commit for the full explanation). Signed-off-by: Derek McGowan <derek@mcg.dev>
CreateRootfsPlaceholders joined sourceRootfs with entry.containerPath
(an OCI mount destination, attacker/spec-controlled) using a plain
filepath.Join. Go's filepath.Join cleans ".." segments as part of the
join, so a destination containing them (e.g. "/../../etc/passwd")
resolves outside sourceRootfs entirely -- filepath.Join("/root",
"../../etc/passwd") returns "/etc/passwd" -- letting a placeholder
file be created anywhere on the host the shim process can write to.
Fix: resolve the path with containerd/continuity/fs.RootPath instead,
which both cleans ".." components relative to the root (rather than
letting them escape it) and safely resolves any symlinks already
present inside sourceRootfs component-by-component, so a symlink
planted inside the shared rootfs tree can't be used to escape it
either.
Signed-off-by: Derek McGowan <derek@mcg.dev>
internal/vminit/podns: if the bind mount of the pod-pause anchor's PID namespace failed, the anchor process was already running and nothing would ever wait on or kill it -- the reaper goroutine was started before the mount was attempted, so it would block on cmd.Wait() forever with no way to reach it, leaking both the process and the goroutine for the VM's lifetime. Move the reaper goroutine to start only after the mount succeeds, and on failure kill the anchor and wait on it synchronously before returning the error. internal/vminit/socketforward: the forwarded UDS listener socket was chmod'd 0o777 to let user-namespaced container processes connect to it. Execute bits are meaningless for a UNIX socket; 0o666 (rw for all) is sufficient and slightly reduces the permissions granted. Signed-off-by: Derek McGowan <derek@mcg.dev>
…CPlugin
Mirrors the split already used by plugins/shim/sandbox: a TaskPlugin
registration ("manager") builds the *service via task.NewTaskService,
and a separate TTRPCPlugin registration ("task") wraps it in a thin
taskService adapter that implements shim.TTRPCService.RegisterTTRPC by
delegating to the manager's TTRPCTaskService. *service itself no longer
needs to implement shim.TTRPCService directly, so its RegisterTTRPC
method and the corresponding interface assertion are removed.
Both of the type assertions unwrapping a plugin.Type's registered
value (SandboxPlugin -> *sandboxManager, TaskPlugin ->
taskAPI.TTRPCTaskService) are ok-checked, returning a descriptive
error instead of panicking the shim if the wiring is ever wrong --
consistent with the same fix applied to the sandbox plugin's own
unwrapping in the previous cross-platform-build commit.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Signed-off-by: Derek McGowan <derek@mcg.dev>
cloneMntNs's non-root branch mapped the calling host UID to itself inside the new user namespace (e.g. host UID 1000 -> namespace UID 1000). This causes every capability to be cleared on exec: per Linux's exec-time capability recomputation, a process whose effective UID is non-zero *within its own current user namespace* has its capability sets cleared to empty when it execs, even though the namespace's creator normally holds a full capability set in it. Since the shim always execs itself into the new namespace (clone+exec, not unshare -- see the doc comment on why), the re-exec'd child ends up with no capabilities at all inside its own namespace: unable to perform the bind mounts SharedFS needs for sandboxed containers (mount(2) returning EPERM), and unable to even call getsockopt(2) on socket fds inherited across the namespace boundary (reproduced standalone by the existing script/userns-check). This is invisible to every build, lint, and unit test job, and to any CI job that happens to run privileged (hence "Unit Tests" and "Build" passing) -- it only surfaces as every sandboxed-path shimtest/CRI test failing at the actual mount syscall, exactly matching the "Integration Tests" CI failures on this PR (TestShim/SingleContainer and every other sandbox-suite case: "share rootfs ... err: operation not permitted"). Fix: map container UID/GID 0 to the real host UID/GID instead. This grants no additional real host privilege: every interaction with a resource outside the namespace is still translated back through the mapping to the real, unprivileged host UID for permission checks. It does keep the child's effective UID zero *inside its own namespace* across exec, which is what preserves the capability set and lets the actual mount()/getsockopt() calls the shim needs succeed. Updated script/userns-check to use the same mapping, so it continues to answer "does userns work at all here" for this code path. Verified: the full shimtest suite, which failed every sandboxed test with this exact "operation not permitted" error when run as a plain non-root user, now passes cleanly (0 failures) under the same non-root invocation that CI uses; full integration suite (12/12) and golangci-lint (linux/darwin x amd64/arm64, only the 5 known pre-existing gosec findings) also verified clean. Signed-off-by: Derek McGowan <derek@mcg.dev>
GuestRootfsPath and GuestVolumePath build paths that are always interpreted inside the (always Linux) guest, regardless of what OS this shim runs on. filepath.Join uses the host's path separator, so on a Windows host it would produce paths like "/run/containers\<id>\rootfs" using backslashes, which the guest can't resolve. Use path.Join instead, which always uses '/'. Signed-off-by: Derek McGowan <derek@mcg.dev>
All's cleanup defer was written after the mount loop, so it only takes effect for an error returned after the loop completes -- there is none; every error path is a direct return from inside the loop, before the defer statement itself ever executes. A failure partway through left every mount already established by that call active, with nothing tracking or unwinding them: SharedFS.Unshare only unmounts what its caller recorded as returned successfully, so those mounts leaked for the life of the shim process. Move the defer to before the loop so it covers every return path. Signed-off-by: Derek McGowan <derek@mcg.dev>
…files os.OpenFile was called with only os.O_CREATE|os.O_EXCL, no access mode flag, defaulting to O_RDONLY. Add O_WRONLY (the file is only ever created and closed, never read or written to afterward, so either explicit mode works, but O_WRONLY matches the create-a-placeholder intent) and tighten the permission from 0o666 to 0o644, since nothing needs to write to this file once created -- crun's own bind mount is what makes the container's socket visible at this path. Signed-off-by: Derek McGowan <derek@mcg.dev>
Comment on lines
+392
to
+412
| // UDS mounts are rewritten to bind mounts whose source is a socket | ||
| // file inside the VM and whose destination is a path in the container | ||
| // rootfs (e.g. /run/shared.sock). The OCI runtime requires the | ||
| // destination to already exist as a regular file. Since the rootfs | ||
| // will be bind-mounted read-only, we create empty placeholder files in | ||
| // the SOURCE rootfs directory now, while it is still writable. | ||
| for _, m := range r.Rootfs { | ||
| if m.Type == "bind" && m.Source != "" { | ||
| sfpr.CreateRootfsPlaceholders(ctx, m.Source) | ||
| break // placeholders are the same regardless of layer; one source suffices | ||
| } | ||
| } | ||
|
|
||
| // Assemble the container rootfs on the host inside the shared dir. | ||
| // Done after bundle loading so UDS placeholders are in place before the | ||
| // read-only bind mount is applied. | ||
| guestRootfs, err := fs.ShareRootfs(ctx, r.ID, r.Rootfs) | ||
| if err != nil { | ||
| fs.Unshare(ctx, r.ID) //nolint:errcheck | ||
| return nil, errgrpc.ToGRPC(fmt.Errorf("share rootfs for %s: %w", r.ID, err)) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds support for multiple containers per VM (one VM per pod), enabling
nerdbox to run as a real CRI
RuntimeClasshandler via containerd's shimsandboxer (
sandboxer = "shim"), instead of only one VM per container.What's new
SandboxService(TTRPC sandbox API), host-siderootfs assembly for member containers over a shared virtiofs tree,
cross-container UDS forwarding.
can share PID/IPC namespaces on demand (PID anchored by a
pod-pauseprocess, since a PID namespace needs a live PID 1).
hostPID/hostIPCand pod-level sharing are treated identically (containerd gives the shim
no way to tell them apart).
virtiofs tree; pod-level DNS/hostname/sysctls applied from
PodSandboxConfig.config.jsonin CRI sandbox bundles, real-rootuser-namespace demotion breaking ext4 mounts, guest bundle-dir leak on
delete, netns-entry-ordering race in libkrun init.
Testing
namespaces, volumes, error handling).
test/critest/harness: real containerd + shim sandboxer + CRIconformance suite. 85 passed / 4 failed / 24 skipped.
Known gaps
4 failures are inherent to per-pod VM isolation, not bugs: two mount-
propagation checks need a live host↔guest kernel mount graph; two
HostNetwork/HostIpc: truechecks need the guest to observe state thatonly exists on the real host machine. Pod-level sharing between member
containers (the common case) is unaffected.
This is currently Linux only. Cross platform is still a goal but changes higher up the stack is still needed to support non-Linux.