Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,3 +92,7 @@ virtio-fs staging and restored live-memory files, and runtime state. Every
virtio-fs export must remain read-only; never extend this path to writable host
sharing. Bounded read-only regular-file injection remains a separate
startup-metadata mechanism for files such as `resolv.conf`.

# Accelerator Contract

Ascend is an optional external adapter for physical 310P and 910 A2/A3 devices with runc. Keep driver-reported model normalization separate from runtime-family validation; do not add per-SKU allowlists. The SDK/scheduler matches the normalized reported model. Provider library directories use additive `SpecUpdates.PrependLibraryPaths`, preserving the application environment; device-visibility variables still use authoritative overrides.
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,7 @@ fmt: ## format Go code
go fmt ./...

check-fmt: ## verify Go code is gofmt-clean
@files="$$(gofmt -l .)" || exit $$?; \
@files="$$(gofmt -l $$(git ls-files '*.go'))" || exit $$?; \
test -z "$$files" || { printf '%s\n' "$$files" >&2; exit 1; }

vet: ## run go vet
Expand Down
21 changes: 14 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,15 @@ The `sbox` binary is an administrative CLI for managing sandboxes.

### NVIDIA GPU sandboxes

GPU support is experimental and currently uses gVisor runsc with nvproxy. The
scheduler passes concrete node-local device IDs through
GPU support is experimental. It supports gVisor runsc through nvproxy and
native runc through the NVIDIA OCI prestart hook. The scheduler passes concrete
node-local device IDs through
`StartRequest.xpu_allocations`; sandboxd resolves them to NVIDIA UUIDs and
maintains a local exclusive lease:

```bash
sbox start \
--runtime runc \
--rootfs /path/to/directory-rootfs \
--xpu-allocation gpu:0,2 \
/bin/sleep 300
Expand Down Expand Up @@ -103,6 +105,15 @@ make networkacl-test
make bpfnat-test
```

The Ascend OCI adapter is an optional external process and is not linked into
the default sandboxd binary. Build or download it from
[openYuanrong-mirror/ascend-oci-adapter](https://github.com/openYuanrong-mirror/ascend-oci-adapter).
The default trusted read-only driver mount profile remains
`configs/ascend/mounts.json`; deployments install it alongside the adapter.
Physical Ascend 310P and 910 A2/A3 support uses driver-reported model names
without a per-SKU allowlist. See [Sandbox runtimes](doc/runtime.md) for model
selection and library-path handling.

`networkacl-test` runs one backend-neutral conformance suite against native
iptables and TC eBPF enforcement in isolated network namespaces. It covers
allow and deny precedence, exact and wildcard peers, peer and sandbox ports,
Expand Down Expand Up @@ -162,11 +173,7 @@ tools/ pinned protobuf code-generation image
## Known limitations

- Kata Containers and Firecracker require a usable `/dev/kvm`; nodes without KVM continue to support gVisor. Firecracker additionally requires a compatible guest kernel/initrd and the ext4 image tool. Its root filesystem may be an immutable EROFS image or a directory exported through virtio-fs; directory-backed OCI/Nydus roots require `virtiofs_enabled` and a compatible virtiofsd.
- NVIDIA GPU sandboxes require runsc, a directory/lisafs-backed rootfs,
`nvidia-container-cli`, accessible NVIDIA devices and userspace driver
libraries, and a host driver supported by the pinned runsc nvproxy. Kata,
Firecracker, runc, MIG, fractional GPUs, and regular-file/EROFS rootfs are
not supported.
- NVIDIA GPU sandboxes require `nvidia-container-cli`, the executable NVIDIA OCI runtime hook, accessible NVIDIA devices and userspace driver libraries. Runsc additionally requires a host driver supported by its pinned nvproxy compatibility gate and a directory/lisafs-backed rootfs. Runc accepts the normal directory or EROFS rootfs through its writable host overlay. Kata, Firecracker, MIG, and fractional GPUs are not supported.
- sandboxd detects the local cgroup mode at startup. Legacy and hybrid hosts use cgroup v1; unified hosts use cgroup v2. The gRPC API and resource-cache behavior are identical in both modes.
- `[plugin.resource].disable_cgroup = true` enables an experimental/debug
compatibility mode for environments where sandboxd cannot write the
Expand Down
2 changes: 1 addition & 1 deletion api/runtime/v1/sandbox-api.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion api/runtime/v1/sandbox-api.proto
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,7 @@ message Mount {
// XpuAllocation describes concrete accelerator devices assigned by the
// scheduler on the selected node.
message XpuAllocation {
// Type is the normalized accelerator type, such as gpu.
// Type is the normalized accelerator type, such as gpu or npu.
string type = 1;
// DeviceIDs contains node-local physical device IDs selected by the scheduler.
repeated uint32 device_ids = 2;
Expand Down
14 changes: 14 additions & 0 deletions config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,23 @@ type PluginConfig struct {

NodeResourceConfig `toml:"node_resource" json:"nodeResource"`

XPUConfig `toml:"xpu" json:"xpu"`

ImageManagerConfig `toml:"image" json:"image"`
}

// XPUConfig contains optional vendor accelerator providers.
type XPUConfig struct {
Ascend AscendConfig `toml:"ascend" json:"ascend"`
Comment thread
mhsong1998-dot marked this conversation as resolved.
}

// AscendConfig enables the external Ascend OCI adapter for runc sandboxes.
type AscendConfig struct {
Enabled bool `toml:"enabled" json:"enabled"`
Adapter string `toml:"adapter" json:"adapter"`
MountProfile string `toml:"mount_profile" json:"mountProfile"`
}

// ImageManagerConfig configures image and mount lifecycle management.
type ImageManagerConfig struct {
ImageManagerRoot string `toml:"root" json:"root"`
Expand Down
14 changes: 14 additions & 0 deletions configs/ascend/mounts.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"default": [
{
"path": [
"/usr/local/Ascend/driver/lib64",
"/usr/local/Ascend/driver/include",
"/usr/local/dcmi",
"/usr/local/bin/npu-smi",
"/usr/sbin/dmidecode",
"/var/queue_schedule"
]
}
]
}
7 changes: 7 additions & 0 deletions configs/sandboxd.toml
Original file line number Diff line number Diff line change
Expand Up @@ -126,3 +126,10 @@ cgroup_memory_limit = "0"
# [plugin.node_resource]
# provider = "kubernetes" # or "cgroup"
# sock_path = "/run/sandboxd/resource.sock"

# Ascend is optional and fail-closed. Enabling it requires the runc runtime,
# a root-owned adapter, and a versioned read-only mount profile.
# [plugin.xpu.ascend]
# enabled = true
# adapter = "/usr/local/bin/ascend-oci-adapter"
# mount_profile = "/etc/akernel/ascend/mounts.json"
15 changes: 14 additions & 1 deletion doc/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,26 @@ binaries, boot artifacts, and host prerequisites pass validation.
| Published-port DNAT | Supported | Supported | Supported | Supported |
| Writable-layer quota | Supported | Not supported | Not supported | Supported |
| Checkpoint and restore | Supported (systrap and KVM) | Not supported | Not supported | Supported |
| NVIDIA GPU | Experimental nvproxy support | Not supported | Not supported | Not supported |
| NVIDIA GPU | Experimental nvproxy support | Experimental NVIDIA OCI-hook support | Not supported | Not supported |
| Ascend NPU | Not supported | Experimental physical 310P and 910 A2/A3 support | Not supported | Not supported |
| Cgroup-disabled mode | Experimental | Not supported | Not supported | Not supported |
| KVM | Optional execution platform; not exposed to the sandbox | Optional guest exposure | Required by the runtime | Required by the runtime; nested KVM is not exposed |

See [Checkpoint and restore](checkpoint-restore.md) for the API design,
artifact ownership, failure semantics, and compatibility requirements.

The NVIDIA provider owns a single physical-device inventory and exclusive
lease table shared by runsc and runc. Runsc remains subject to its nvproxy
driver-compatibility gate, while an unsupported nvproxy driver does not disable
runc GPU support. Both runtimes consume the provider's UUIDs and OCI environment
through the NVIDIA prestart hook.

The Ascend provider reports the normalized model name returned by the driver, without a per-model allowlist. Requests must match that reported name exactly; for example, a node reporting `ascend910` uses `npu:ascend910:1`, not an inferred SKU. The adapter independently resolves the driver device family to the supported physical 310P or 910 A2/A3 injection profile. Unsupported families and empty model names are rejected. Allocated devices must share a model and runtime family.

Ascend driver library paths are prepended to the final image/request `LD_LIBRARY_PATH`, preserving application and CANN directories while removing duplicates and empty entries. Provider-owned device visibility variables remain authoritative.

GPU capacity is not qualified by runtime in the scheduler. Deploy GPU nodes with consistent runtime capabilities and select `runtime="runc"` explicitly when their GPUs are available only through runc.

## Selection and configuration

A start request selects a runtime by name. Each adapter must have an entry
Expand Down
23 changes: 18 additions & 5 deletions internal/server/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -695,7 +695,9 @@ func NewSandboxService(root, configPath string) (result SandboxService, retErr e
}
xpuMgr := xpumanager.New(
cfg.RuntimeConfig.RuntimeBinary[config.RuntimeNameRunsc],
cfg.RuntimeConfig.RuntimeBinary[config.RuntimeNameRunc] != "",
sandboxRoot,
cfg.XPUConfig,
)

// The optional node-resource module comes up first so its external resource
Expand Down Expand Up @@ -1215,10 +1217,21 @@ func (h *sandboxService) Start(ctx context.Context, request *runtime.StartReques
return &runtime.StartResponse{Code: -1, Message: err.Error()},
errord.ToGRPC(errord.ErrInvalidArgument)
}
if len(startReq.XpuAllocations) > 0 && startReq.Runtime != config.RuntimeNameRunsc {
err := fmt.Errorf("XPU allocations require runtime %q", config.RuntimeNameRunsc)
return &runtime.StartResponse{Code: -1, Message: err.Error()},
errord.ToGRPC(errord.ErrInvalidArgument)
if len(startReq.XpuAllocations) > 0 {
if startReq.Runtime != config.RuntimeNameRunsc && startReq.Runtime != config.RuntimeNameRunc {
err := fmt.Errorf("XPU allocations require runtime %q or %q", config.RuntimeNameRunsc, config.RuntimeNameRunc)
return &runtime.StartResponse{Code: -1, Message: err.Error()},
errord.ToGRPC(errord.ErrInvalidArgument)
}
if h.xpuMgr == nil {
err := errors.New("XPU manager is not configured")
return &runtime.StartResponse{Code: -1, Message: err.Error()},
errord.ToGRPC(errord.ErrFailedPrecondition)
}
if err := h.xpuMgr.ValidateRuntime(startReq.Runtime, startReq.XpuAllocations); err != nil {
return &runtime.StartResponse{Code: -1, Message: err.Error()},
errord.ToGRPC(errord.ErrInvalidArgument)
}
}
if startReq.WritableLayerLimitBytes > 0 {
if startReq.Runtime != config.RuntimeNameRunsc &&
Expand Down Expand Up @@ -1387,7 +1400,7 @@ func (h *sandboxService) Start(ctx context.Context, request *runtime.StartReques
return &runtime.StartResponse{Code: -1, Message: err.Error()},
errord.ToGRPC(errord.ErrFailedPrecondition)
}
specUpdates, err = h.xpuMgr.Acquire(sandboxID, startReq.XpuAllocations)
specUpdates, err = h.xpuMgr.Acquire(sandboxID, startReq.Runtime, startReq.XpuAllocations)
if err != nil {
return &runtime.StartResponse{
Code: -1,
Expand Down
14 changes: 13 additions & 1 deletion internal/server/server_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,6 @@ func TestStartRejectsFirecrackerOCIImageBeforeFilesystemPrepare(t *testing.T) {

func TestStartRejectsXPUForUnsupportedRuntimes(t *testing.T) {
for _, runtimeName := range []string{
config.RuntimeNameRunc,
config.RuntimeNameKata,
config.RuntimeNameFirecracker,
} {
Expand All @@ -303,6 +302,19 @@ func TestStartRejectsXPUForUnsupportedRuntimes(t *testing.T) {
}
}

func TestStartRejectsXPUWhenManagerIsUnavailable(t *testing.T) {
s := newTestService(t, map[string]svc.Handler{
config.RuntimeNameRunc: svc.NewFakeRuntimeHandler(),
})
response, err := s.Start(context.Background(), &runtime.StartRequest{
Runtime: config.RuntimeNameRunc,
Rootfs: &runtime.RootfsConfig{},
XpuAllocations: []*runtime.XpuAllocation{{Type: "npu"}},
})
assert.Equal(t, codes.FailedPrecondition, status.Code(err))
assert.Contains(t, response.Message, "XPU manager is not configured")
}

func TestStartRejectsEnableKVMForRunsc(t *testing.T) {
s := newTestService(t, map[string]svc.Handler{
config.RuntimeNameRunsc: svc.NewFakeRuntimeHandler(),
Expand Down
11 changes: 11 additions & 0 deletions pkg/runtime/handler.go
Original file line number Diff line number Diff line change
Expand Up @@ -107,9 +107,20 @@ type StartConfig struct {
// this boundary so vendor-specific discovery and authorization do not leak
// into the runsc client.
type SpecUpdates struct {
// Envs carries provider-owned device visibility settings. Prestart is used
// by providers such as NVIDIA whose toolkit performs injection in an OCI
// hook; Ascend resolves the corresponding OCI objects before runc starts.
Envs []*runtime.KeyValue
Prestart []Hook
Annotations map[string]string
// LinuxDevices and DeviceCgroupRules authorize provider-selected device
// nodes. Mounts carries the provider's read-only driver/runtime files.
LinuxDevices []LinuxDevice
DeviceCgroupRules []LinuxDeviceCgroup
Mounts []Mount
// PrependLibraryPaths adds driver library directories ahead of the final
// image/request LD_LIBRARY_PATH without replacing application directories.
PrependLibraryPaths []string
// RequiresHostWritableRootfs requests a private writable rootfs view
// before provider hooks execute. It is separate from the writable layer
// visible to workloads after the sandbox starts.
Expand Down
70 changes: 70 additions & 0 deletions pkg/runtime/library_paths_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
// Copyright (c) 2026 Ant Group Corporation.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package runtime

import (
"reflect"
"testing"

api "github.com/inclusionAI/sandboxd/api/runtime/v1"
)

func TestGenerateOciPreservesApplicationLibrariesWithProviderPaths(t *testing.T) {
loader, err := NewBundleLoader("", t.TempDir())
if err != nil {
t.Fatal(err)
}
loader.baseSpec.Process.Env = []string{"LD_LIBRARY_PATH=/image/lib"}
_, spec, err := loader.GenerateOci(OciLoadOptions{
SandboxID: "sbox-libraries", CgroupPath: "/sandbox/libraries",
Config: StartConfig{
Rootfs: t.TempDir(), Resources: &api.LinuxSandboxResources{},
Envs: []*api.KeyValue{{Key: "LD_LIBRARY_PATH", Value: ":/opt/cann/lib64:/driver/lib:/application/lib::"}},
SpecUpdates: &SpecUpdates{
PrependLibraryPaths: []string{"/driver/lib", "", "/driver/lib"},
},
},
})
if err != nil {
t.Fatal(err)
}
want := "LD_LIBRARY_PATH=/driver/lib:/opt/cann/lib64:/application/lib"
if !containsString(spec.Process.Env, want) {
t.Fatalf("OCI env = %v, want %s", spec.Process.Env, want)
}
if containsString(spec.Process.Env, "LD_LIBRARY_PATH=/image/lib") {
t.Fatal("request must override image libraries before driver paths are merged")
}
}

func TestPrependLibraryPathsPreservesOtherEnvironment(t *testing.T) {
envs := []string{"CUSTOM=a=b", "LD_LIBRARY_PATH=/application/lib"}
got := prependLibraryPaths(envs, []string{"/driver/lib"})
want := []string{"CUSTOM=a=b", "LD_LIBRARY_PATH=/driver/lib:/application/lib"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("env = %v, want %v", got, want)
}
if got := prependLibraryPaths(envs, nil); !reflect.DeepEqual(got, envs) {
t.Fatalf("provider without library paths changed env: %v", got)
}
}

func TestPrependLibraryPathsWithoutApplicationPath(t *testing.T) {
got := prependLibraryPaths([]string{"PATH=/bin"}, []string{"", "/driver/lib", "/driver/lib", ""})
want := []string{"PATH=/bin", "LD_LIBRARY_PATH=/driver/lib"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("env = %v, want %v", got, want)
}
}
Loading