Skip to content

msft-main: 4.1.0 rebase - #535

Open
Saul Paredes (Redent0r) wants to merge 11 commits into
up/4.1.0from
saul/msft-main-rebase
Open

msft-main: 4.1.0 rebase#535
Saul Paredes (Redent0r) wants to merge 11 commits into
up/4.1.0from
saul/msft-main-rebase

Conversation

@Redent0r

@Redent0r Saul Paredes (Redent0r) commented Aug 21, 2026

Copy link
Copy Markdown

@Redent0r Saul Paredes (Redent0r) changed the title Saul/msft main rebase msft-main: 4.1.0 rebase Aug 21, 2026
@Redent0r

Copy link
Copy Markdown
Author

I'm discovering now that upstream kata-containers@a34c74a#diff-35ee5a7a73be1a70e4c5e4befc8c88a5cd494ffdec9f167290f6b7212a1d50fa may make our fork dd9707c redundant (or almost redundant). They adopted a very similar static resource management in that PR:

With runtime-rs static sandbox sizing, Kata uses:

- If workload limits are present:
  - `vm_vcpus = requested_vcpus + overhead_vcpus`
  - `vm_memory = requested_memory + overhead_memory`
- If workload limits are not present:
  - `vm_vcpus = default_vcpus`
  - `vm_memory = default_memory`

@Redent0r

Saul Paredes (Redent0r) commented Aug 24, 2026

Copy link
Copy Markdown
Author

I'm discovering now that upstream kata-containers@a34c74a#diff-35ee5a7a73be1a70e4c5e4befc8c88a5cd494ffdec9f167290f6b7212a1d50fa may make our fork dd9707c redundant (or almost redundant). They adopted a very similar static resource management in that PR:

With runtime-rs static sandbox sizing, Kata uses:

- If workload limits are present:
  - `vm_vcpus = requested_vcpus + overhead_vcpus`
  - `vm_memory = requested_memory + overhead_memory`
- If workload limits are not present:
  - `vm_vcpus = default_vcpus`
  - `vm_memory = default_memory`

I thought under this new behaviour if we could drop our runtime-rs resource management patch and set:

  • overhead_* to 0
  • default_* to what we use for static_defaults_* currently

And match our current fork behavior. This is:

  • if no limits set, use static defaults
  • if limits are set, use those (and don't add any extra)

But it's not straightforward. Particularly because upstream uses pod-level behaviour if self.resource.vcpu > 0.0 || self.resource.mem_mb > 0 https://github.com/kata-containers/kata-containers/pull/13173/changes#diff-35ee5a7a73be1a70e4c5e4befc8c88a5cd494ffdec9f167290f6b7212a1d50faR170 where if either CPU or memory are set, resource management will try to derive from the YAML limits. In the case where CPU is set but memory is not, the memory will be equal to the overhead

  • vm_memory = requested_memory (limits, in this case 0) + overhead_memory

This is even documented in the unit tests #[case::cpu_only_limit(3.0, 0.5, 1024, 128, 1.5, 0, 2.0, 128)] https://github.com/kata-containers/kata-containers/pull/13173/changes#diff-35ee5a7a73be1a70e4c5e4befc8c88a5cd494ffdec9f167290f6b7212a1d50faR612 . This reads: for CPU only limit set, the memory we expect (128) is just the memory overhead (128).

We use a memory overhead of 32Mi in AKS, so if we do above, every pod that only specifies a CPU limit, will get 32Mi of memory. This is too little.

Currently under this scenario, our guest gets 512. Why? Because we currently have per-resource behavior

        if self.resource.mem_mb == 0 {
            self.resource.mem_mb = config.runtime.static_sandbox_default_workload_mem;
        }

        if self.resource.vcpu == 0.0 {
            self.resource.vcpu = config.runtime.static_sandbox_default_workload_vcpus;
        }

dd9707c#diff-35ee5a7a73be1a70e4c5e4befc8c88a5cd494ffdec9f167290f6b7212a1d50faR173-R178, not pod-level-behavior. So in our fork, even if only CPU limits are set, we still separately check for memory and fallback to static_default if no limits are set.

So this tells me:

  • we can't just drop our patches
  • there's potentially an upstream fix to make it do resource-level- behaviour (it doesn't seem right to rely on pod overhead for pods that only specify a CPU limit. My understanding is pod overhead should be exclusively for overhead components (agent, guest kernel), not workload components)

edit: created ADO item https://dev.azure.com/mariner-org/container-runtime/_sprints/taskboard/container-runtime%20Team/container-runtime/MMM/Kr/CYC5?workitem=23451

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Rebases the Microsoft fork onto Kata Containers 4.1.0 while restoring fork-specific resource sizing, Azure Linux tooling, and CI exceptions.

Changes:

  • Adds static sandbox defaults and Cloud Hypervisor template resizing.
  • Adds Azure Linux UVM/IGVM build and deployment tooling.
  • Updates workflows, tests, webhook behavior, samples, and security documentation.

Reviewed changes

Copilot reviewed 31 out of 144 changed files in this pull request and generated 28 comments.

Show a summary per file
File Description
tools/testing/kata-webhook/main.go Enforces minimum container memory limits.
tools/testing/kata-webhook/deploy/webhook.yaml Configures the minimum-memory webhook deployment.
tools/testing/gatekeeper/skips.py Changes the default comparison branch.
tools/testing/gatekeeper/required-tests.yaml Disables unsupported fork CI requirements.
tools/osbuilder/README.md Links Azure Linux deployment documentation.
tools/osbuilder/node-builder/azure-linux/uvm_install.sh Installs UVM artifacts.
tools/osbuilder/node-builder/azure-linux/uvm_build.sh Builds Azure Linux UVM images.
tools/osbuilder/node-builder/azure-linux/README.md Documents Azure Linux builds and deployment.
tools/osbuilder/node-builder/azure-linux/package_tools_install.sh Packages UVM build tooling.
tools/osbuilder/node-builder/azure-linux/package_install.sh Installs runtime binaries and configuration.
tools/osbuilder/node-builder/azure-linux/package_build.sh Builds runtime, agent, and snapshotter components.
tools/osbuilder/node-builder/azure-linux/Makefile Defines Azure Linux build/deploy targets.
tools/osbuilder/node-builder/azure-linux/common.sh Centralizes Azure Linux build paths and settings.
tools/osbuilder/node-builder/azure-linux/clean.sh Cleans Azure Linux build outputs.
tools/osbuilder/Makefile Adds IGVM build and cleanup targets.
tools/osbuilder/igvm-builder/igvm_builder.sh Implements the IGVM builder entry point.
tools/osbuilder/igvm-builder/azure-linux/igvm_lib.sh Implements Azure Linux IGVM operations.
tools/osbuilder/igvm-builder/azure-linux/config.sh Defines IGVM image parameters.
tools/osbuilder/.gitignore Ignores generated Azure Linux artifacts.
tests/spellcheck/kata-dictionary.txt Adds fork-specific terminology.
tests/integration/nydus/nydus-sandbox.yaml Adds a sandbox memory limit.
tests/integration/kubernetes/runtimeclass_workloads/pod-sandbox-vcpus-allocation.yaml Disables the no-limit vCPU workload.
tests/integration/kubernetes/k8s-sandbox-vcpus-allocation.bats Adjusts vCPU test expectations.
tests/integration/cri-containerd/integration-tests.sh Adds pod memory limits and disables memory updates.
src/tools/genpolicy/policy_samples.json Defines policy-generation sample groups.
src/runtime/virtcontainers/sandbox.go Revises factory fallback and resource resizing.
src/runtime/virtcontainers/hypervisor.go Permits zero initial CPU and memory values.
src/runtime/virtcontainers/factory/factory_linux.go Delegates restored-VM resizing to the sandbox.
src/runtime/virtcontainers/clh.go Adds template memory-zone hotplug support.
src/runtime/virtcontainers/clh_test.go Tests snapshot handling for grown zones.
src/runtime/pkg/oci/utils.go Applies static workload defaults.
src/runtime/pkg/katautils/create.go Uses static factory configuration.
src/runtime/pkg/katautils/config.go Loads static workload defaults.
src/runtime/pkg/katautils/config-settings.go.in Changes the default vCPU count.
src/runtime/pkg/katautils/config_test.go Updates zero-memory configuration expectations.
src/runtime/Makefile Adds configurable static workload defaults.
src/runtime/config/configuration-clh.toml.in Exposes static workload settings.
src/runtime/cmd/kata-runtime/factory.go Uses static factory sizing.
src/runtime-rs/Makefile Adds runtime-rs static workload defaults.
src/runtime-rs/crates/runtimes/virt_container/src/factory/vm.rs Preserves zero memory for later sizing.
src/runtime-rs/crates/resource/src/cpu_mem/initial_size.rs Applies static defaults when limits are absent.
src/runtime-rs/config/configuration-dragonball.toml.in Adds Dragonball static defaults.
src/runtime-rs/config/configuration-clh-runtime-rs.toml.in Adds CLH static defaults.
src/runtime-rs/config/configuration-clh-azure-runtime-rs.toml.in Adds Azure CLH static defaults.
src/runtime-rs/arch/aarch64-options.mk Defines the aarch64 CLH binary.
src/libs/kata-types/src/config/runtime.rs Adds runtime static-default fields.
src/libs/kata-types/src/config/hypervisor/mod.rs Allows zero default memory.
src/libs/kata-types/src/config/hypervisor/ch.rs Removes CLH minimum-memory fallback.
src/libs/kata-types/src/config/default.rs Changes global CPU and CLH memory minima.
src/agent/samples/policy/yaml/secrets/pull-secrets.yaml Adds a pull-secret policy sample.
src/agent/samples/policy/yaml/kubernetes/unsupported-image/haproxyrc.yaml Adds an unsupported-image sample.
src/agent/samples/policy/yaml/kubernetes/unsupported-image/frontend-controller.yaml Adds a frontend policy sample.
src/agent/samples/policy/yaml/kubernetes/fixtures/quota.yaml Adds a quota fixture.
src/agent/samples/policy/yaml/kubernetes/fixtures/namespace.yaml Adds a namespace fixture.
src/agent/samples/policy/yaml/kubernetes/fixtures/limits.yaml Adds a limit-range fixture.
src/agent/samples/policy/yaml/dont-enable-kata-debug/pod-lots-of-layers.yaml Adds a large-layer policy sample.
SECURITY.md Adopts Microsoft vulnerability reporting guidance.
AGENTS.md Documents future rebase requirements.
.gitignore Adds Microsoft-specific generated artifacts.
.github/workflows/static-checks.yaml Changes the static-check comparison branch.
.github/workflows/static-checks-self-hosted.yaml Disables unavailable self-hosted runners.
.github/workflows/scorecard.yaml Retargets Scorecard pushes.
.github/workflows/release-s390x.yaml Retargets s390x branch releases.
.github/workflows/release-ppc64le.yaml Retargets ppc64le branch releases.
.github/workflows/release-arm64.yaml Retargets arm64 branch releases.
.github/workflows/release-amd64.yaml Retargets amd64 branch releases.
.github/workflows/push-oras-tarball-cache.yaml Retargets ORAS cache publishing.
.github/workflows/payload-after-push.yaml Retargets payload publishing.
.github/workflows/osv-scanner-scheduled.yaml Retargets scheduled OSV scanning.
.github/workflows/osv-scanner-pr.yaml Retargets PR OSV scanning.
.github/workflows/gatekeeper.yaml Changes the gatekeeper PR event.
.github/workflows/docs.yaml Retargets documentation publishing.
.github/workflows/commit-message-check.yaml Allows fixup commit subjects.
.github/workflows/codeql.yml Retargets CodeQL analysis.
.github/workflows/ci.yaml Disables unsupported fork jobs.
.github/workflows/ci-on-push.yaml Changes the primary CI PR trigger.
.github/workflows/build-kubectl-image.yaml Retargets kubectl image builds.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tools/testing/kata-webhook/main.go
Comment thread tools/osbuilder/Makefile
Comment thread tools/osbuilder/node-builder/azure-linux/package_build.sh
Comment thread tools/osbuilder/node-builder/azure-linux/Makefile
Comment thread src/runtime/pkg/oci/utils.go
Comment thread .github/workflows/static-checks.yaml
Comment thread tools/testing/gatekeeper/skips.py
Comment thread .github/workflows/push-oras-tarball-cache.yaml
Comment thread tests/integration/cri-containerd/integration-tests.sh Outdated
Comment thread tools/osbuilder/igvm-builder/azure-linux/igvm_lib.sh Outdated

@sprt Aurélien Bombo (sprt) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Redent0r
Saul Paredes (Redent0r) force-pushed the saul/msft-main-rebase branch 2 times, most recently from e60e6cf to 18ae7ad Compare August 25, 2026 21:06
Comment thread src/libs/kata-types/src/config/runtime.rs Outdated

@sprt Aurélien Bombo (sprt) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some nits but otherwise lgtm assuming green CI.

When you're please include links to green BB/image-build/conformance/perf runs in the PR description. 🙂

Comment thread tests/spellcheck/kata-dictionary.txt
Comment thread tools/testing/gatekeeper/required-tests.yaml Outdated
Comment thread tools/testing/gatekeeper/required-tests.yaml Outdated
Comment thread tools/testing/gatekeeper/required-tests.yaml Outdated
Manuel Huber and others added 11 commits August 26, 2026 16:49
* Update the default branch to msft-main in different places for
  the CI to work with our fork.
* Add the MSFT-required SECURITY.md and corresponding dictionary entries.

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
For runtime-go and runtime-rs. See below for details

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>

tools: Add initial igvm-builder and node-builder/azure-linux scripting

This branch starts introducing additional scripting to build, deploy
and evaluate the components used in AKS' Pod Sandboxing and
Confidential Containers preview features. This includes the capability
to build the IGVM file and its reference measurement file for remote
attestation.

Signed-off-by: Manuel Huber <mahuber@microsoft.com>

tools: Improve igvm-builder and node-builder/azure-linux scripting

- Support for Mariner 3 builds using OS_VERSION variable
- Improvements to IGVM build process and flow as described in README
- Adoption of using only cloud-hypervisor-cvm on CBL-Mariner

Signed-off-by: Manuel Huber <mahuber@microsoft.com>

tools: Add package-tools-install functionality

- Add script to install kata-containers(-cc)-tools bits
- Minor improvements in README.md
- Minor fix in package_install
- Remove echo outputs in package_build

Signed-off-by: Manuel Huber <mahuber@microsoft.com>

tools: Enable setting IGVM SVN

- Allow setting SVN parameter for IGVM build scripting

Signed-off-by: Manuel Huber <mahuber@microsoft.com>

node-builder: introduce BUILD_TYPE variable

This lets developers build and deploy Kata in debug mode without having to make
manual edits to the build scripts.

With BUILD_TYPE=debug (default is release):

 * The agent is built in debug mode.
 * The agent is built with a permissive policy (using allow-all.rego).
 * The shim debug config file is used, ie. we create the symlink
   configuration-clh-snp-debug.toml <- configuration-clh-snp.toml.

For example, building and deploying Kata-CC in debug mode is now as simple as:

   make BUILD_TYPE=debug all-confpods deploy-confpods

Also do note that make still lets you override the other variables even after
setting BUILD_TYPE. For example, you can use the production shim config with
BUILD_TYPE=debug:

   make BUILD_TYPE=debug SHIM_USE_DEBUG_CONFIG=no all-confpods deploy-confpods

Signed-off-by: Aurélien Bombo <abombo@microsoft.com>

node-builder: introduce SHIM_REDEPLOY_CONFIG

See README: when SHIM_REDEPLOY_CONFIG=no, the shim configuration is NOT
redeployed, so that potential config changes made directly on the host
during development aren't lost.

Signed-off-by: Aurélien Bombo <abombo@microsoft.com>

node-builder: Use img for Pod Sandboxing

Switch from UVM initrd to image format

Signed-off-by: Manuel Huber <mahuber@microsoft.com>

node-builder: Adapt README instructions

- Sanitize containerd config snippet
- Set podOverhead for Kata runtime class

Signed-off-by: Manuel Huber <mahuber@microsoft.com>

tools: Adapt AGENT_POLICY_FILE path

- Adapt path in uvm_build.sh script to comply
  with the usptream changes we pulled in

Signed-off-by: Manuel Huber <mahuber@microsoft.com>

node-builder: Use Azure Linux 3 as default path

- update recipe and node-builder scripting
- change default value on rootfs-builder

Signed-off-by: Manuel Huber <mahuber@microsoft.com>

node-builder: Deploy-only for AzL3 VMs

- split deployment sections in node-builder README.md
- install jq, curl dependencies within IGVM script
- add path parameter to UVM install script

Signed-off-by: Manuel Huber <mahuber@microsoft.com>

node-builder: Minor updates to README.md

- no longer install make package, is part of meta package
- remove superfluous popd
- add note on permissive policy for ConfPods UVM builds

Signed-off-by: Manuel Huber <mahuber@microsoft.com>

node-builder: Updates to README.md

- with the latest 3.2.0.azl4 package on PMC, can remove OS_VERSION parameter
  and use the make deploy calls instead of copying files by hand for variant
  I (now aligned with Variant II)
- with the latest changes on msft-main, set the podOverhead to 600Mi

Signed-off-by: Manuel Huber <mahuber@microsoft.com>

node-builder: Fix SHIM_USE_DEBUG_CONFIG behavior

Using a symlink would create a cycle after calling this script again when
copying the final configuration at line 74 so we just use cp instead.

Also, I moved this block to the end of the file to properly override the final
config file.

Signed-off-by: Aurélien Bombo <abombo@microsoft.com>

node-builder: Build and install debug configuration for pod sandboxing

For ease of debugging, install a configuration-clh-debug.toml for pod
sandboxing as we do in Conf pods.

Signed-off-by: Cameron Baird <cameronbaird@microsoft.com>

runtime: remove clh-snp config file usage in makefile

Not needed to build vanilla kata

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>

package_tools_install.sh: include nsdax.gpl.c

Include nsdax.gpl.c

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>

node-builder: fix typo in string comparison

This also fixes a shellcheck error and lets us require the
shellcheck-required job:

In ./tools/osbuilder/node-builder/azure-linux/uvm_build.sh line 34:
        if [ -z "${UVM_KERNEL_HEADER_DIR}}" ]; then
                                         ^-- SC2157 (error): Argument to -z is always false due to literal strings.

Signed-off-by: Aurélien Bombo <abombo@microsoft.com>

docs: node-builder: fix static check error

This fixes the below static check error to follow up on the infra fix from
kata-containers#11646:

2025-07-31T19:32:45.0031829Z time="2025-07-31T19:32:44.990004665Z" level=fatal msg="found 2 parse errors:\nfile=\"tools/osbuilder/node-builder/azure-linux/README.md\": duplicate heading: \"Set up environment\" (heading: {Name:Set up environment MDName:Set up environment LinkName:set-up-environment Level:2})\nfile=\"tools/osbuilder/node-builder/azure-linux/README.md\": duplicate heading: \"Install build dependencies\" (heading: {Name:Install build dependencies MDName:Install build dependencies LinkName:install-build-dependencies Level:2})" commit=1d17f56b1aa7a880468b8e25d14467c92dca8eeb name=kata-check-markdown pid=9075 source=check-markdown version=0.0.1

Note: that is likely flagged because having two headings with the same
name, even under different sections, makes it impossible to create a
canonical heading link in Markdown.

This should eventually be squashed into the node-builder commit.

Signed-off-by: Aurélien Bombo <abombo@microsoft.com>

docs: node-builder: Remove references to moby-containerd-cc

As we adopted containerd2, we remove references to our prior
forked containerd version.

Signed-off-by: Manuel Huber <mahuber@microsoft.com>

node-builder: 2Mb aligned guest image size

Build the mariner guest image using IMAGE_SIZE_ALIGNMENT_MB=2.

Signed-off-by: Dan Mihai <dmihai@microsoft.com>

to-squash: node-builder: add reference to README.md

This is needed to avoid the following static-checks error:

2025-08-05T21:27:20.0028337Z [static-checks.sh:808] ERROR: Document tools/osbuilder/node-builder/azure-linux/README.md is not referenced

This commit is to be squashed into the node-builder commit.

Signed-off-by: Aurélien Bombo <abombo@microsoft.com>

node-builder: build and install runtime-rs
Build and install both runtime-rs and runtime-go configs and binaries side by side:
  - runtime-go:
      /usr/local/bin/containerd-shim-kata-v2-go
      /usr/local/share/defaults/kata-containers/configuration-clh.toml
      /usr/local/share/defaults/kata-containers/configuration-clh-debug.toml

  - runtime-rs:
      /usr/local/bin/containerd-shim-kata-v2-rs
      /usr/local/share/defaults/kata-containers/configuration-cloud-hypervisor.toml
      /usr/local/share/defaults/kata-containers/configuration-cloud-hypervisor-debug.toml

Also add USE_RUNTIME_RS variable and default to "yes". This controls which runtime binary and configuration will be installed
to /usr/local/bin/containerd-shim-kata-v2 and /usr/local/share/defaults/kata-containers/configuration.toml respectively.

Also install kata-ctl (runtime-rs equivalent of kata-runtime) so we can exec into the UVM when using runtime-rs

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
- if no limits are specified, assign a default static amount of memory (512Mi)
and vcpu (1) to the UVM
- if limits are specified, use those limit values for the UVM resources (don't add any extra)

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>

runtime: Resolve high UVM memory footprint

Bug: https://microsoft.visualstudio.com/OS/_workitems/edit/43668151

Rationale: This is a temporary solution for optimizing memory usage for
the current mechanism of requesting resources through pod Limit
annotations:
- if no Limits are specified and hence WorkloadMemMB is 0, set a default
  value 'StaticWorkloadDefaultMem' to allocate a default amount of
  memory for use for containers in the sandbox in addition to the base
  memory
- if Limits are specified, the base memory and the sum of Limits are
  allocated. The end user needs to be aware of the minimum memory
  requirements for their pods, otherwise the pod will be stuck in the
  ContainerCreating state

Testing: Manual testing, creating pods with Limits and without limits,
and with two containers where each container has a limit, tested with
integration in a SPEC file where the config variables were set via
environment variables via the make command

Adapted by @mfrw from 3.1.0 to apply to 3.2.0

Signed-off-by: Muhammad Falak R Wani <mwani@microsoft.com>
Signed-off-by: Manuel Huber <mahuber@microsoft.com>

runtime: Remove unused VMM options for mem alloc

- We only ever tested these fork changes with CLH+MSHV
- Remove these options as we don't use QEMU/FC

Signed-off-by: Manuel Huber <mahuber@microsoft.com>

runtime: improved memory overhead management

After these changes:

1. The value of the K8s runtime class memory overhead:
   - Covers the memory usage from all the Host-side components (mainly
     the Kata Shim and the VMM).
   - Doesn't include the memory usage from any Guest-side components.

2. The value of a pod memory limit specified by the user:
   - Is equal to the memory size of the Pod VM.
   - Includes the memory usage from all the Guest-side components
     (mainly user's workload, the Guest kernel, and the Kata Agent)
   - Doesn't include the memory usage from any Host-side components.

Signed-off-by: Dan Mihai <dmihai@microsoft.com>

runtime: fix `make test`

This addresses the following errors from `make test` to allow us to require
that upstream CI:

https://github.com/microsoft/kata-containers/actions/runs/16656407213/job/47142422035?pr=392#step:13:53

Signed-off-by: Aurélien Bombo <abombo@microsoft.com>

runtime: Allocate default workload vcpus

- similar to the static_sandbox_default_workload_mem option,
  assign a default number of vcpus to the VM when no limits
  are given, 1 vcpu in this case
- similar to commit c7b8ee9, do not allocate additional vcpus
  when limits are provided

Signed-off-by: Manuel Huber <mahuber@microsoft.com>
- if no limits are specified, assign a default static amount of memory (512Mi) and vcpu (1) to the UVM
- if limits are specified, use those limit values for the UVM resources (don't add any extra)

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>

runtime-rs: Resolve high UVM memory footprint
This is a port from b03db3e into runtime-rs

Rationale: This is a temporary solution for optimizing memory usage for
the current mechanism of requesting resources through pod Limit
annotations:
- if no Limits are specified and hence WorkloadMemMB is 0, set a default
  value 'StaticWorkloadDefaultMem' to allocate a default amount of
  memory for use for containers in the sandbox in addition to the base
  memory
- if Limits are specified, the base memory and the sum of Limits are
  allocated. The end user needs to be aware of the minimum memory
  requirements for their pods, otherwise the pod will be stuck in the
  ContainerCreating state

Testing: Manual testing, creating pods with Limits and without limits,
and with two containers where each container has a limit, tested with
integration in a SPEC file where the config variables were set via
environment variables via the make command

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>

runtime-rs: improved memory overhead management

This is a port from 7ddec33 into runtime-rs

After these changes:

1. The value of the K8s runtime class memory overhead:
   - Covers the memory usage from all the Host-side components (mainly
     the Kata Shim and the VMM).
   - Doesn't include the memory usage from any Guest-side components.

2. The value of a pod memory limit specified by the user:
   - Is equal to the memory size of the Pod VM.
   - Includes the memory usage from all the Guest-side components
     (mainly user's workload, the Guest kernel, and the Kata Agent)
   - Doesn't include the memory usage from any Host-side components.

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>

runtime-rs: Allocate default workload vcpus

This is a port from 9af9844

Plus ports an existing behaviour from runtime-go to also add the vcpus. See
https://github.com/fidencio/kata-containers/blob/e2476f587c472d5d217df9c75cdb80193dd85994/src/runtime/pkg/oci/utils.go#L1232

- similar to the static_sandbox_default_workload_mem option,
  assign a default number of vcpus to the VM when no limits
  are given, 1 vcpu in this case
- similar to commit c7b8ee9, do not allocate additional vcpus when limits are provided

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>

runtime-rs: add test coverage for static resource management

If using static management and initial size manager uses 0 for CPU or memory,
we add default static values to the hv config

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
- tests that deploy pods with too small of a memory limit
- tests try to set a minimum memory limit for some containerd tests
- tests that use runners we don't have
- tests that depend on pushing to GHCR
- disable Kata Containers CI / kata-containers-ci-on-push / run-kata-deploy-tests / run-kata-deploy-tests (qemu, k3s)

Also disable these for runtime-rs that fail due to resource management patches:
- run-nerdctl-tests (dragonball)
- run-nydus (active, dragonball)
- run-nydus (lts, dragonball)

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
If memory limit is set and less than minimum, set it to minimum.

This is to to account for kata-containers@0ec3403

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
Add new documentation to assist with rebasing

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
For CLH, handle larger sizes for incoming deployments by resizing
the VM inherited from the template VM. i.e. incoming pod is
1024M and the template VM is 512M? Plug in a zone of size 512M.

Assisted-By: Claude Opus 4.8
Signed-Off-By: Cameron Baird <cameronbaird@microsoft.com>
This patch is for the msft fork only. It wires up the static size
configs such that factory respects the static_ fields.

This means we have the same VM size model as is in production today:
1. With no limits specified, the user gets the static size 512Mi,1vcpu
2. With limits specified, the user gets the exact specified size
	which is limits.memory,limits.vcpus for the deployment

Assisted-By: Clause Opus 4.8
Signed-Off-By: Cameron Baird <cameronbaird@microsoft.com>
Port the policy YAML samples and their test categorization from
release/3.15.0 for kata and kata-cc testing.

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
Temporary patch to test rebase (retargeted to up/4.1.0).

Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants