Skip to content

[Kata] Add an experimental Kata Containers backend - #1577

Draft
Eric Curtin (ericcurtin) wants to merge 13 commits into
microsoft:mainfrom
ericcurtin:feat/kata-interface
Draft

Eric Curtin (ericcurtin) wants to merge 13 commits into
microsoft:mainfrom
ericcurtin:feat/kata-interface

Conversation

@ericcurtin

@ericcurtin Eric Curtin (ericcurtin) commented Oct 26, 2026 •

Copy link
Copy Markdown

📖 Description

Experimental Linux kata backend: runs a request in a Kata microVM via containerd-shim-kata-v2 over ttrpc. Design and limits are in docs/backends/kata/kata-backend.md. Draft until #1610 and its spec are agreed.

🔗 References

🔍 Validation

Fedora x86-64, Kata 4.3.1 runtime-rs: scripts/ci/run_backend_validation_tests.sh kata passed 19, failed 0 (QEMU, Cloud Hypervisor, Dragonball, Firecracker). fmt and clippy clean. cargo test -p mxc-sdk --lib: only the 3 failures already on main.

Not verified: CI pool (/dev/kvm?), arm64, other Kata versions.

✅ Checklist

…ontract

Adds the experimental kata backend value and its kata section
(rootfs, hypervisor) to the 1.1.0-alpha contract, the runtime model,
and backend registration. The Kata backend itself follows in later
commits, so selecting it is still refused.

Published contracts are unchanged.
Adds a small ttrpc client and the protobuf messages that the Kata shim's
containerd.task.v2.Task service needs. The tests compare the encodings with
ones produced from containerd's own .proto files.

It adds no dependencies. The ttrpc and containerd-shim-protos crates would add
about 45 packages to Cargo.lock, including second versions of nix, thiserror,
and syn. Nothing uses the client yet.
The kata backend starts containerd-shim-kata-v2 itself and drives it over
ttrpc, with no containerd or Kubernetes. A request provisions a sandbox, runs
the command as an exec process in the started VM, and releases the sandbox on
every path. The hypervisor (qemu, clh, firecracker, dragonball) is chosen by
passing a Kata configuration file in the shim's runtime options.

The root filesystem is a directory (read-only overlay lower layer, writable
layer on a capped tmpfs) or an ext4 image (private copy on a loop device;
required for Firecracker). Filesystem policy becomes bind mounts, denied paths
become masks. The VM gets an empty network namespace, so only a deny-all
network policy is accepted; everything the backend cannot enforce is rejected
before a VM exists.

A request killed without cleaning up is reclaimed by the next one.

The streaming path now checks the experimental opt-in for this backend.
Lifecycle operations follow in a later commit.
Provision, start, exec, stop, and deprovision work for kata through the raw
exact 1.1.0-alpha JSON entry points. The sandbox id is kata:<token>; state is
kept on disk, so a later call can come from another process. Provision fixes
the filesystem and network policy. Stop keeps the writable layer, so a restart
sees earlier files. Typed SDK requests still do not reach kata.

Adds the kata provision root to the development contract and its generated
schema and TypeScript types. The .NET parity check treats kata like Windows
Sandbox: a lifecycle only the raw JSON lane reaches.
Adds the kata suite: lxc-exec runs on every hypervisor (guest kernel, exit
code, streams, deny-all network, filesystem policy, timeout, rejections), then
the prebuilt streaming_kata test covers streaming, kill, drop, a killed
request, and the container lifecycle. Every run checks that no sandbox
directory, mount, process, or loop device is left behind.

The build job ships streaming_kata with lxc-exec and runs the Kata unit tests
and the request-refusal tests, which need no root, KVM, or Kata. The matrix
runs kata on ubuntu-24.04 x64: the host prep installs a pinned Kata release
and Alpine root filesystem, and a missing KVM, Kata, or root filesystem fails
the job instead of skipping.

Documents the backend in docs/backends/kata and the schema, policy, and CI
docs.
Copilot AI balanced review requested due to automatic review settings October 26, 2026 19:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved lifecycle stream handling, timeout confirmation, cleanup failures, and an unprivileged CI test failure block approval.

7 open issues
What changed in this PR

Adds an experimental Linux Kata Containers backend to MXC’s execution engine, with create-and-run execution, streaming, and container lifecycle operations through the development JSON contract.

Changes:

  • Adds Kata runtime integration, filesystem isolation, deny-all networking, and resource cleanup.
  • Extends development contracts, generated artifacts, routing, and compatibility tests.
  • Adds Linux validation jobs, host preparation, and backend documentation.
File Description
tests/​scripts/​run_kata_all_tests.sh Adds multi-hypervisor functional validation and leak checks.
tests/​scripts/​README.md Documents the Kata suite and dispatcher.
src/​mxc-sdk/​tests/​mxc_contract/​version_boundaries/​experimental.rs Checks Kata settings require the development contract.
src/​mxc-sdk/​tests/​mxc_contract/​version_boundaries/​containment.rs Checks the Kata containment version boundary.
src/​mxc-sdk/​tests/​mxc_contract/​v1_1_0_alpha/​state_aware/​provision/​mod.rs Registers Kata provision tests.
src/​mxc-sdk/​tests/​mxc_contract/​v1_1_0_alpha/​state_aware/​provision/​kata.rs Tests provision request validation.
src/​mxc-sdk/​tests/​mxc_contract/​v1_1_0_alpha/​fixtures/​kata_provision/​valid/​minimal.json Adds a minimal valid provision fixture.
src/​mxc-sdk/​tests/​mxc_contract/​v1_1_0_alpha/​fixtures/​kata_provision/​valid/​full.json Adds a provision fixture with policies.
src/​mxc-sdk/​tests/​mxc_contract/​v1_1_0_alpha/​fixtures/​kata_provision/​invalid/​unknown_hypervisor.json Covers unsupported hypervisor values.
src/​mxc-sdk/​tests/​mxc_contract/​v1_1_0_alpha/​fixtures/​kata_provision/​invalid/​missing_kata.json Covers missing Kata configuration.
src/​mxc-sdk/​tests/​mxc_contract/​v1_1_0_alpha/​fixtures.rs Registers Kata fixture validation.
src/​mxc-sdk/​tests/​mxc_contract/​v1_1_0_alpha/​experimental/​root.rs Includes Kata in development-field parsing tests.
src/​mxc-sdk/​tests/​mxc_contract/​v1_1_0_alpha/​experimental/​kata.rs Tests Kata fields and hypervisor spellings.
src/​mxc-sdk/​tests/​mxc_contract/​v1_1_0_alpha/​experimental.rs Registers Kata configuration tests.
src/​mxc-sdk/​tests/​mxc_contract/​v1_1_0_alpha/​enums.rs Tests acceptance of Kata containment.
src/​mxc-sdk/​src/​lib.rs Exposes the Linux backend module internally.
src/​mxc-sdk/​src/​core/​mxc_engine/​state_aware.rs Routes Kata lifecycle operations.
src/​mxc-sdk/​src/​core/​mxc_engine/​run.rs Adds Linux runner selection and Windows rejection.
src/​mxc-sdk/​src/​core/​mxc_engine/​experimental.rs Includes Kata in experimental authorization tests.
src/​mxc-sdk/​src/​core/​mxc_engine/​dispatch.rs Adds experimental-gated streaming dispatch.
src/​mxc-sdk/​src/​core/​mxc_engine/​backend_registry.rs Registers Kata without discovery reporting.
src/​mxc-sdk/​src/​core/​mxc_contract/​registry.rs Registers the Kata provision contract root.
src/​mxc-sdk/​src/​core/​mxc_contract/​dev/​state_aware/​provision/​mod.rs Adds the Kata provision variant.
src/​mxc-sdk/​src/​core/​mxc_contract/​dev/​state_aware/​provision/​kata.rs Defines the Kata provision request.
src/​mxc-sdk/​src/​core/​mxc_contract/​dev/​state_aware/​provision/​containment.rs Recognizes Kata provision containment.
src/​mxc-sdk/​src/​core/​mxc_contract/​dev/​state_aware/​mod.rs Exports Kata lifecycle contract types.
src/​mxc-sdk/​src/​core/​mxc_contract/​dev/​schema.rs Adds Kata provision schema dispatch.
src/​mxc-sdk/​src/​core/​mxc_contract/​dev/​request.rs Parses Kata provision requests.
src/​mxc-sdk/​src/​core/​mxc_contract/​dev/​one_shot.rs Adds Kata create-and-run settings.
src/​mxc-sdk/​src/​core/​mxc_contract/​dev/​mod.rs Exports Kata development types.
src/​mxc-sdk/​src/​core/​mxc_contract/​dev/​experimental.rs Defines root filesystem and hypervisor settings.
src/​mxc-sdk/​src/​core/​mxc_common/​wire.rs Adds intermediate Kata configuration types.
src/​mxc-sdk/​src/​core/​mxc_common/​state_aware_operation.rs Carries Kata provision configuration.
src/​mxc-sdk/​src/​core/​mxc_common/​state_aware_dispatch.rs Resolves the kata: sandbox prefix.
src/​mxc-sdk/​src/​core/​mxc_common/​state_aware_binding.rs Binds lifecycle requests to Kata.
src/​mxc-sdk/​src/​core/​mxc_common/​state_aware_binding_tests.rs Tests Kata binding and configuration preservation.
src/​mxc-sdk/​src/​core/​mxc_common/​splice.rs Tests Kata command-source recognition.
src/​mxc-sdk/​src/​core/​mxc_common/​sdk_input.rs Keeps typed lifecycle requests from selecting Kata.
src/​mxc-sdk/​src/​core/​mxc_common/​policy_identity.rs Includes Kata settings in policy identity.
src/​mxc-sdk/​src/​core/​mxc_common/​models.rs Adds runtime backend and configuration models.
src/​mxc-sdk/​src/​core/​mxc_common/​config_parser.rs Normalizes and validates Kata request routing.
src/​mxc-sdk/​src/​core/​mxc_common/​config_contract_adapters/​v1_0/​state_aware.rs Leaves Kata absent in stable lifecycle adaptation.
src/​mxc-sdk/​src/​core/​mxc_common/​config_contract_adapters/​v1_0/​one_shot.rs Leaves Kata absent in stable execution adaptation.
src/​mxc-sdk/​src/​core/​mxc_common/​config_contract_adapters/​v0_9/​state_aware.rs Leaves Kata absent in older lifecycle adaptation.
src/​mxc-sdk/​src/​core/​mxc_common/​config_contract_adapters/​v0_9/​one_shot.rs Leaves Kata absent in older execution adaptation.
src/​mxc-sdk/​src/​core/​mxc_common/​config_contract_adapters/​dev/​state_aware.rs Adapts Kata provision configuration and policies.
src/​mxc-sdk/​src/​core/​mxc_common/​config_contract_adapters/​dev/​one_shot.rs Maps Kata settings into runtime input.
src/​mxc-sdk/​src/​core/​mxc_common/​config_contract_adapters/​dev/​one_shot_tests/​experimental.rs Tests Kata mapping and omitted-field preservation.
src/​mxc-sdk/​src/​core/​mxc_common/​common_request_ir.rs Carries optional Kata configuration.
src/​mxc-sdk/​src/​core/​mxc_common/​cmdline.rs Selects POSIX shell command handling for Kata.
src/​mxc-sdk/​src/​backends/​mod.rs Registers the Linux-only Kata module.
src/​mxc-sdk/​src/​backends/​kata/​common/​task_api.rs Implements shim Task API messages and calls.
src/​mxc-sdk/​src/​backends/​kata/​common/​state_aware.rs Implements lifecycle operations and exec handles.
src/​mxc-sdk/​src/​backends/​kata/​common/​rootfs.rs Prepares writable overlays and image copies.
src/​mxc-sdk/​src/​backends/​kata/​common/​protobuf.rs Adds the minimal protobuf codec.
src/​mxc-sdk/​src/​backends/​kata/​common/​oci.rs Builds container and process configurations.
src/​mxc-sdk/​src/​backends/​kata/​common/​netns.rs Creates and removes isolated network namespaces.
src/​mxc-sdk/​src/​backends/​kata/​common/​mounts.rs Finds and removes sandbox mounts.
src/​mxc-sdk/​src/​backends/​kata/​common/​mod.rs Organizes backend modules and runner exports.
src/​mxc-sdk/​src/​backends/​kata/​common/​loopdev.rs Attaches root images to autoclearing loop devices.
src/​mxc-sdk/​src/​backends/​kata/​common/​host.rs Locates Kata and checks host prerequisites.
src/​mxc-sdk/​src/​backends/​kata/​common/​execution.rs Manages guest processes, streams, waits, and cleanup.
src/​mxc-sdk/​src/​backends/​kata/​common/​error.rs Maps backend errors into MXC result types.
src/​ffi/​mxc_ffi/​tests/​ffi.rs Tests experimental authorization through the C API.
sdk/​node/​src/​generated/​v1_1_0_alpha/​wire.ts Adds generated Kata development-contract types.
scripts/​ci/​validation-test-matrix.json Schedules Kata validation on Ubuntu x64.
scripts/​ci/​run_backend_validation_tests.sh Dispatches strict Kata functional validation.
scripts/​ci/​prepare-linux-host.sh Installs pinned Kata and test root filesystems.
scripts/​check-dotnet-api-parity.js Excludes JSON-only Kata lifecycle from typed parity checks.
schemas/​dev/​mxc-config.schema.1.1.0-alpha.json Adds generated Kata schema definitions.
README.md Lists the experimental backend.
docs/​schema.md Documents Kata configuration and containment.
docs/​development/​build-and-test/​ci-validation-infrastructure.md Documents Kata build and validation coverage.
docs/​development/​architecture/​versioning.md Documents contract and experimental boundaries.
docs/​development/​architecture/​repository-architecture.md Records the backend’s architectural location.
docs/​development/​architecture/​container-lifecycle.md Documents JSON-only Kata lifecycle access.
docs/​containment-policy/​dev/​policy.md Defines supported Kata policy behavior.
docs/​backends/​kata/​kata-backend.md Adds setup, usage, lifecycle, and limitation guidance.
.github/​workflows/​Validation.Tests.Matrix.Job.yml Runs Kata validation with root privileges.
.github/​workflows/​Build.Linux.Job.yml Adds unit checks and packages live-test artifacts.

Review effort: Balanced

Comment thread src/mxc-sdk/src/backends/kata/common/netns.rs
Comment on lines +169 to +170
let stdout = execution.take_stdout().ok_or_else(|| missing("stdout"))?;
let stderr = execution.take_stderr().ok_or_else(|| missing("stderr"))?;

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 17cc774. Thank you!

Comment thread src/mxc-sdk/src/backends/kata/common/execution.rs Outdated
Comment thread src/mxc-sdk/src/backends/kata/common/execution.rs Outdated
Comment thread src/mxc-sdk/src/backends/kata/common/loopdev.rs Outdated
Comment thread src/mxc-sdk/src/backends/kata/common/state_aware.rs Outdated
Comment thread src/mxc-sdk/src/backends/kata/common/state_aware.rs
Copilot AI balanced review requested due to automatic review settings October 26, 2026 23:10
@ericcurtin

Copy link
Copy Markdown
Author

Addressed the review findings in 17cc774: relayed output no longer ends on EAGAIN (new test, red before), a failed wait thread now cleans up, SIGKILL needs a real Wait result, the loop device detaches if autoclear fails, and a relayed timeout is an error. Not changed: the netns test passes unprivileged here (umount2 returns ENOENT first), and CloseIO on native stdio transfer is a known limitation for a follow-up.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved containment, termination, and cleanup issues prevent safe approval.

6 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Forward Kata cleanup warnings from run-to-completion waits

src/​mxc-sdk/​src/​core/​mxc_engine/​run.rs:315

[P2] This run-to-completion route drops Kata cleanup warnings: KataProcess::finish records teardown failures in warnings(), but Runner::wait_for_child never reads them. As a result, lxc-exec can report success after leaving mounts or sandbox state behind without the warning promised by the backend guide. Forward the child's warnings through the shared runner's response or logger before dropping it, while preserving streaming warning propagation.

4 resolved since last review

Review effort: Balanced

Comment thread src/mxc-sdk/src/backends/kata/common/execution.rs
Comment thread src/mxc-sdk/src/backends/kata/common/execution.rs
Comment thread src/mxc-sdk/src/backends/kata/common/oci.rs
Copilot AI balanced review requested due to automatic review settings October 27, 2026 11:02
@ericcurtin

Copy link
Copy Markdown
Author

Pushed ec65cea: stdin goes through a cancellable pump, fixing the early CloseIO on native stdio and the stuck stdin write; descendant kill and guest-side denied paths are follow-ups.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved stream-blocking, cleanup-reporting, and timeout-handling defects prevent reliable execution and teardown.

3 open issues
3 previously missed issues

In code that hasn't changed since last review

Medium severity Runner drops sandbox cleanup warnings from lxc-exec

src/​mxc-sdk/​src/​backends/​kata/​common/​kata_runner.rs:263

[P2] Cleanup failures are only saved in SandboxProcess::warnings(), but the lxc-exec path uses Runner::wait_for_child, which never reads those warnings or forwards them to its Logger/ScriptResponse. Consequently an unmount or deprovision failure is silently discarded when the handle is dropped, and the CLI can exit successfully with resources left behind. Forward post-wait warnings through the shared Runner to the logger used by lxc-exec, including on timeout/error outcomes.

Medium severity Timeout kill fails to preserve timed-out result in wait and poll

src/​mxc-sdk/​src/​backends/​kata/​common/​kata_runner.rs:326

[P2] With no process.timeout configured, mxc_sandbox_kill_for_timeout uses the inherited kill implementation without recording that termination was requested for a timeout. Subsequent wait/poll therefore reports an ordinary exit rather than setting out_timed_out. Override kill_for_timeout to record a separate pending-timeout state, confirm termination and perform teardown, then preserve the timeout result in both wait and polling paths.

Medium severity Non-Linux spawn skips required experimental authorization check

src/​mxc-sdk/​src/​core/​mxc_engine/​dispatch.rs:210

[P2] This non-Linux arm skips the experimental authorization check performed by the Linux arm and by resolve_runner/run_state_aware. Consequently, mxc_spawn_json with Kata and experimental=0 returns unsupported_containment on Windows/macOS instead of the documented backend_unavailable experimental-opt-in error. Apply the same gate before returning the host-platform error so streaming preserves the other entry points' failure semantics.

5 resolved since last review

Review effort: Balanced

Comment thread src/mxc-sdk/src/backends/kata/common/kata_runner.rs Outdated
Comment thread src/mxc-sdk/src/backends/kata/common/kata_runner.rs Outdated
# Conflicts:
#	README.md
#	docs/containment-policy/dev/policy.md
#	docs/development/build-and-test/ci-validation-infrastructure.md
#	scripts/ci/validation-test-matrix.json
#	src/mxc-sdk/src/core/mxc_common/cmdline.rs
Copilot AI balanced review requested due to automatic review settings October 27, 2026 23:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Cleanup failures remain silent on the lxc-exec path because child warnings are not forwarded to Logger, allowing commands to report success despite leftover resources.

3 open issues
1 resolved since last review

Review effort: Balanced

Comment on lines +235 to +236
self.warnings
.extend(failures.into_iter().map(|f| format!("Kata cleanup: {f}")));

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 57b6b93, with a regression test. Thank you!

Copilot AI balanced review requested due to automatic review settings October 28, 2026 14:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread scripts/ci/prepare-linux-host.sh Outdated
Comment thread tests/scripts/run_kata_all_tests.sh Outdated

@bbonaby Branden Bonaby (bbonaby) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Eric Curtin (@ericcurtin), thanks for taking the time to put this together.

Before implementing a new feature or API, please first open a feature request and discuss the proposed design with the maintainers. This allows us to understand the motivating scenario, review the API surface, consider the impact across our supported platforms and SDKs, and agree on an approach before implementation begins.

Our contribution process is documented in the following guides:

MXC Contributor's Guide
Adding a New Feature

This PR does not currently have a corresponding issue, feature specification, or agreed-upon design. If you would like us to consider this feature, please open a feature request describing the scenario and proposed API. We can continue the design discussion there, and ultimately may ask for a feature spec PR first before any implementation.

Thanks again for your interest in contributing to MXC.

@microsoft-github-policy-service microsoft-github-policy-service Bot added the Needs-Author-Feedback Waiting for additional information or action from the issue or pull-request author. label Oct 28, 2026
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
Copilot AI balanced review requested due to automatic review settings October 28, 2026 22:23
@ericcurtin

Copy link
Copy Markdown
Author

Fixed the cached Kata check and the pipefail grep in cc805bb. Thank you!

@microsoft-github-policy-service microsoft-github-policy-service Bot added Needs-Attention Requires attention or a decision from the MXC maintainers. and removed Needs-Author-Feedback Waiting for additional information or action from the issue or pull-request author. labels Oct 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ericcurtin

Copy link
Copy Markdown
Author

Thank you Branden Bonaby (@bbonaby). Opened #1610 and kept this as a draft prototype until the issue and spec are agreed.

Signed-off-by: Eric Curtin <eric.curtin@docker.com>
Copilot AI balanced review requested due to automatic review settings October 29, 2026 22:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Incomplete rootfs caches can prevent recovery on retries, and relative Kata paths can break shim launches after working-directory changes.

4 open issues
1 resolved since last review

Review effort: Balanced

Comment on lines +326 to +334
if [[ ! -d "$KATA_ASSETS/rootfs" ]]; then
curl -fsSL --retry 3 -o "$work/alpine.tar.gz" \
"https://dl-cdn.alpinelinux.org/alpine/v${ALPINE_VERSION%.*}/releases/x86_64/alpine-minirootfs-${ALPINE_VERSION}-x86_64.tar.gz"
echo "${ALPINE_SHA256} $work/alpine.tar.gz" | sha256sum -c -
sudo mkdir "$KATA_ASSETS/rootfs"
sudo tar -xzf "$work/alpine.tar.gz" -C "$KATA_ASSETS/rootfs"
sudo rm -f "$KATA_ASSETS/rootfs.ext4"
sudo mke2fs -q -t ext4 -d "$KATA_ASSETS/rootfs" "$KATA_ASSETS/rootfs.ext4" 128M
fi
Comment on lines +166 to +171
locate_install(
hypervisor,
&process_env,
&|path: &Path| path.is_file(),
&path_dirs,
)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Needs-Attention Requires attention or a decision from the MXC maintainers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants