AI Cluster Runtime (AICR) makes it easy to stand up GPU-accelerated Kubernetes clusters. It captures known-good combinations of drivers, operators, kernels, and system configurations and publishes them as version-locked recipes — reproducible artifacts for Helm, Argo CD, Flux, and Helmfile.
Full documentation: docs.nvidia.com/aicr
Running GPU-accelerated Kubernetes clusters reliably is hard. Small differences in kernel versions, drivers, container runtimes, operators, and Kubernetes releases can cause failures that are difficult to diagnose and expensive to reproduce.
Historically, this knowledge has lived in internal validation pipelines and runbooks. AI Cluster Runtime makes it available to everyone.
Every AICR recipe is:
- Optimized — Tuned for a specific combination of hardware, cloud, OS, and workload intent.
- Validated — Passes automated constraint and compatibility checks before publishing.
- Reproducible — Same inputs produce identical deployments every time.
# Install the CLI (Homebrew)
brew tap NVIDIA/aicr
brew install aicr
# Or use the install script
curl -sfL https://raw.githubusercontent.com/NVIDIA/aicr/main/install | bash -s --
# Generate a recipe for your environment
aicr recipe --service eks --accelerator h100 --os ubuntu \
--intent training --platform kubeflow -o recipe.yaml
# Inspect any hydrated value (e.g., the resolved GPU driver version)
aicr query --service eks --accelerator h100 --os ubuntu --intent training --platform kubeflow \
--selector components.gpu-operator.values.driver.version
# Render it into deployment-ready bundles (helm, argocd, flux, or helmfile)
aicr bundle --recipe recipe.yaml --deployer argocd --output ./bundles
# After deploying the bundle, validate the running cluster against the recipe
aicr validate --recipe recipe.yamlThe contents of the bundles/ directory depend on the chosen --deployer: Helm values and a deploy.sh for helm, Argo CD Application manifests for argocd, HelmRelease and Kustomization manifests for flux, or a helmfile.yaml release graph for helmfile.
See the Installation Guide for manual installation, building from source, and container images.
| Feature | Description |
|---|---|
aicr CLI |
Single binary for the full workflow: snapshot, recipe, bundle, validate, verify, diff, and trust management. |
API Server (aicrd) |
REST API exposing the same capabilities as the CLI. Run in-cluster for CI/CD integration or air-gapped environments. |
| Snapshot Agent | Kubernetes Job that captures live cluster state (GPU hardware, drivers, kernel, OS, operators, K8s config) into a ConfigMap for validation against recipes. |
| Multi-Deployer Bundles | Render the same recipe into Helm, Argo CD, Flux, or Helmfile artifacts — pick whichever fits your GitOps pipeline. |
| Multi-Phase Validation | Deployment, performance (training and inference), and conformance phases — run all or one at a time. |
| Drift Detection | aicr diff compares two snapshots to surface configuration drift between clusters or over time. |
| Supply Chain Security | SLSA Level 3 provenance, signed SBOMs, image attestations (Cosign / Sigstore), and aicr verify for offline bundle verification. |
AICR recipes compose components from the following groups:
| Group | Examples |
|---|---|
| GPU stack | GPU Operator, DRA GPU Driver, Network Operator, NFD, NVSentinel |
| Cloud integration | AWS EFA, AWS EBS CSI, GKE NCCL TCPxO |
| Node tuning | Nodewright Operator and customizations, cert-manager |
| Observability | kube-prometheus-stack, Prometheus Operator CRDs, Prometheus Adapter, ephemeral-storage metrics |
| Training platforms | Kubeflow Trainer, Slinky Slurm Operator, KAI Scheduler, Kueue |
| Inference platforms | Dynamo, Grove, NIM Operator, Agent Gateway |
See the full Component Catalog for every component, pinned version, and source. Don't see what you need? Open an issue — feedback helps inform future validation priorities.
| Dimension | Values |
|---|---|
| Services | EKS, AKS, GKE, OKE, LKE, Kind |
| Accelerators | H100, GB200, B200, RTX PRO 6000 |
| Operating systems | Ubuntu, Talos, COS |
| Workload intents | Training, Inference |
| Platforms | Kubeflow, Slurm (Slinky), Dynamo, NIM |
A recipe is a version-locked configuration for a specific environment. You describe your target (cloud, GPU, OS, workload intent, optional platform), and the recipe engine matches it against a library of validated overlays — layered configurations that compose bottom-up from base defaults through cloud, accelerator, OS, and workload-specific tuning. Composable mixins carry shared fragments (OS constraints, platform components) so a leaf overlay only declares what is unique to it.
The bundler materializes a recipe into deployment-ready artifacts: one folder per component, each with Helm values, checksums, and a README. The validator compares a recipe against a live cluster snapshot — first checking declarative constraints, then optionally running deployment, performance, and conformance phases inside the cluster.
This separation means the same validated configuration works whether you deploy with Helm, Argo CD, Flux, Helmfile, or a custom pipeline.
- Not a Kubernetes distribution
- Not a cluster provisioner or lifecycle management system
- Not a managed control plane or hosted service
- Not a replacement for your cloud provider or OEM platform
- Not a generic configuration management platform
At its core, AICR is a cluster configuration generator. You bring your GPU-accelerated Kubernetes cluster and your deployment tooling; AICR generates the runtime configuration artifacts your tools deploy to the cluster. AICR can also validate that the configuration was correctly materialized and that it delivers the expected performance characteristics.
Full documentation lives at docs.nvidia.com/aicr. Key entry points:
- Installation — Install the
aicrCLI (script, manual, or build from source) - CLI Reference — Every command, flag, and example
- API Reference — REST API endpoints for
aicrd - Agent Deployment — Run the snapshot agent in your cluster
- Validation — Deployment, performance, and conformance phases
- Component Catalog — Every component that can appear in a recipe
- Recipe Development — Add or modify recipe metadata
- Automation Guide — CI/CD integration patterns
For contributors:
- Contributing Guide — Development setup, testing, and PR process
- Development Guide — Local development, Make targets, and tooling
- Architecture Overview — System design and packages
- Roadmap — Feature priorities and development timeline
- Security — Supply chain security, vulnerability reporting, and verification
- Releases — Binaries, SBOMs, and attestations
- Issues — Bugs, feature requests, and questions
- Slack — Join Kubernetes Slack and visit the #aicr channel
AI Cluster Runtime is Apache 2.0. Contributions are welcome: new recipes for environments we haven't covered, additional bundler formats, validation checks, or bug reports. See CONTRIBUTING.md for development setup and the PR process.