cluster-manager

The Agent Platform's MCP-only cluster write surface: node pools first (giantswarm/giantswarm#37637, epic giantswarm/giantswarm#37639).

cluster-manager is the third sibling of cluster-manager and agent-manager: a Go MCP server (streamable HTTP, an mcp-oauth resource server) registered with muster by its own MCPServer CR. Agents call its tools through muster, and so does the Dev Portal — as the signed-in person. Every Kubernetes call is presented to the API server with the caller's forwarded IdP token (forwardToken), so the caller's RBAC governs what a tool may read or write. There is no REST API.

Tools

Through muster the tools appear as x_cluster-manager_<tool>. docs/tools.md shows every tool's description and input schema as a model reads them (make tools-doc).

Tool	Purpose
get_info	Version, mode capabilities (apply, commit, and commitTools: the tools that take mode commit), the tool list and whether the installation serves the Cluster API (clusterApi)
list_clusters	The installation's clusters: name, organization, release version, own-cluster flag, the availability zones of the cluster's node subnets (zones[], from its AWSCluster — what a pool may be pinned to, the choices create_node_pool's zones is checked against; zonesNote when they cannot be read), GPU operator and serving presence with their provider (serving is present with a KServe controller — its Deployment, its release or the platform's discovery ConfigMap — never with the KServe CRDs alone, which Helm leaves behind when a serving layer goes: those are absent with the served APIs noted; a child HelmRelease of cluster-manager's slice release that is not Ready is named in the evidence with its condition's reason and message, since the slice's meta release reports Ready whether or not its children installed), the GPU pool releases (ready, and deleting while one is removed), the commit target; empty with a clusterApi note on an installation without the Cluster API. LLMInferenceServiceConfigs left terminating in the release namespace are named in serving.evidence. Beside status/provider/evidence both components carry readiness, structured, nothing to parse — gpuOperator.readiness: release{name, namespace, ready, reason, message, since, deletedAt} (the provider's operator release), clusterPolicy{name, state} (its status.state: ready, notReady, ignored), operands[]{name, namespace, desired, ready} (the device plugin and GPU feature discovery DaemonSets on the GPU nodes), operandsMessage (why operands is empty although a ClusterPolicy exists: the operator creates them once a GPU node joins, so a pool at scale-to-zero has none), operandsError (the DaemonSets could not be listed); serving.readiness: release (cluster-manager's slice release — ready only once every child release it renders is Ready too: the meta release reports Ready about 30 s before its children have installed, kserve-llmisvc-resources last, so until then ready is false with reason ChildNotReady and message naming each pending child with its condition), children[] (every child release of the slice with its Ready condition — the five KServe charts, the connectivity child, kserve-runtime-configs), controllers[]{name, namespace, available, replicas} (the KServe controller Deployments), configs{count} (the LLMInferenceServiceConfigs of the release namespace), backend{registered, backend, namespace, name, error} (the kserve backend cluster-manager registered with model-manager for this cluster: backend its name — kserve for the own cluster, kserve-<cluster> for a workload cluster —, namespace/name its ConfigMap; registered is true or false when the ConfigMap was read and omitted, with error set, when it could not be read — never a bare false for a failed read), presets{count} (the published serving presets), modelsGateway{name, namespace, ready, reason} (the Gateway listening on models.<domain>, its Programmed condition), webhook{configurations[], namespace, service, endpoints, ready, reason, message} (the llmisvc admission webhook: the configurations admitting LLMInferenceServices and the Service they call — ready only once the Service has a ready endpoint and every configuration carries its CA bundle, named in the evidence until then: the controller's release reads Ready and its Deployment counts an available replica before the webhook answers, and a load_model in that window fails admission), cache{enabled, claim} (the model cache as cluster-manager's slice release states it in its values: whether the predictors mount a claim and which — modelServing.cache.pvc.name, the chart's default hf-cache when the values name none; null where the slice release is not cluster-manager's), cacheClaims[]{namespace, name, phase, volume, zone, error} (every model cache PersistentVolumeClaim of the serving namespace — the one of before, hf-cache, and the claims per zone, hf-cache-<zone>, kept when the pool goes — each with the zone its bound volume's node affinity names, where a pool mounting it lands; empty for none, null when the cluster cannot be read); a count is null where the cluster cannot be read; for a cluster create_cluster made, admins{users, groups} it binds as cluster-admin, read from its values (adminsNote when they cannot be read). Every read is concurrent: the call stays inside the aggregator's deadline.
list_releases	The installation's releases (Release CRs), newest first per provider line: state (active, deprecated, wip, preview), Kubernetes version, the cluster chart the release pins and its release chart (release-<provider> at the release's version, the cluster chart renamed by the releases repository) with published, whether the registry lists the tag (null with a note when the registry cannot be read); offered says whether create_cluster creates a cluster of it — active, published, a provider line it offers (providers: aws, azure, eks) — and note why not; on an installation without the Release CRD an empty list with a note naming it
create_cluster	Create a workload cluster (giantswarm/giantswarm#37614): in org-<organization>, a HelmRelease named after the cluster installing the release chart through an OCIRepository at the release's tag (release defaults to the newest active one), reading the installation's cluster-app-installation-values first and the cluster's <name>-values ConfigMap second — the values given with the name, organization, description, release version and cloud identity composed on top. The cluster's apiserver trusts the installation's Dex: global.controlPlane.oidc is composed from the issuer this server validates tokens against, the audience muster requests at login (createdClusters.oidc.clientID, else the first of muster.mcpServer.auth.requiredAudiences), the email and groups claims and the Dex CA of a private certificate, so the tools that call the cluster as the caller are authenticated there; a caller's own global.controlPlane.oidc in values wins whole, and the answer's oidc names what the cluster trusts (installation, values or none). The creator and the organization's admins are cluster-admin on the new cluster: the cluster chart's rbac-bootstrap app binds the caller's email (the username the cluster takes from the email claim) and the users and groups org-<organization>'s RoleBindings bind to cluster-admin (rbac-operator's write-all-customer-group among them), after the bindings the release installs anyway; global.apps.rbacBootstrap.values.bindings in values replaces them whole ([] binds nobody), a caller who may not list the organization's RoleBindings is refused naming that key, and the answer's rbac names who is bound (composed, values or none, with a note on who is left out); list_clusters reads them back (admins). A cluster created before is brought along by a re-run with the same arguments, which updates its <name>-values. Refused before any write: a name that is not a DNS label of at most 20 characters or that a Cluster on the installation uses, a release that is not active or whose tag is not published, an organization without the installation's values, in mode apply an organization whose tenant ServiceAccount is not bound yet (rbac-operator's write-all-customer-sa RoleBinding follows the namespace within minutes), and values that, over the chart's defaults and the installation's, do not match the release chart's own values.schema.json (each violation named with its path). Mode apply as the caller; a re-run changes nothing; mode commit follows
delete_cluster	Remove a cluster create_cluster created, as the caller, in two calls with the same arguments. The first removes the kserve backend cluster-manager registered for the cluster, then the cluster's HelmRelease (helm-controller uninstalls the cluster and its default apps; the GPU pool, operator and slice releases go with the Cluster through their ownerReference), its OCIRepository and its <name>-values ConfigMap, listing those releases (withCluster) and the models served on the cluster (models). The second, once the Cluster is gone, removes what the uninstall left in org-<organization>: the cluster's HelmReleases and OCIRepositories rendered by Helm or cluster-manager, and cluster-manager's own source and values. A call that finds nothing of the cluster left answers not found with a structured {"notFound": {"cluster", "namespace", "nothingLeft": true}} block as a second text content of the tool error: the removal is complete, readable without parsing the text. Refused before any write: a release cluster-manager did not create, the installation's own cluster, a release in a Flux Kustomization's inventory, a release helm-controller has not reconciled since its last write. Mode apply; commit follows
list_node_pools	The pools of one cluster — its MachinePools, and the pool releases whose MachinePool is not there yet (right after create_node_pool) or gone already (while helm-controller uninstalls it) — with the pool's Kubernetes version and the control plane's as two fields, replicas, instance and accelerator types, the owning HelmRelease, and the pool's lifecycle: phase is creating (release not Ready, or MachinePool not created or not ready), ready (every step done; readyReplicas may be 0 at scale-to-zero), scaling (NodeClaims launching or terminating, nodes registering, or a node Karpenter could not launch), removing (release or MachinePool deleted, or a teardown re-run pending) or failed (release Ready=False, the reason in the step); steps[]{name, state, since, finishedAt, message} are release (the pool release Ready), machinePool (its Ready condition, else infrastructureReady) and nodes (Karpenter's NodeClaims of the pool on the cluster — launching, ready, terminating, with counts, a terminating node named with its deletion time: 0 NodeClaim(s) launching, 0 ready, 1 terminating — ip-… terminating since 2026-09-18T03:36:08Z (Karpenter drains the node and terminates its instance; the NodeClaim goes once the instance is terminated, minutes later) —, on a GPU pool of cluster-manager's the ready nodes that hold nothing (no GPU workload, no predictor) named idle since their last pod left (1 node(s) ready (aws:///…), 1 idle since 2026-09-17T22:55:41Z (ip-…): delete_node_pool removes an idle node with the pool) and a MachinePool still listing instances the cluster no longer has said so (0 nodes on the cluster: the MachinePool still lists 1 gone (…), its list follows within minutes), else readyReplicas of replicas where the cluster cannot be read; a node Karpenter could not launch keeps the step inProgress since the last refusal, never done with none, with the refusal in a line — every size and every zone refused: the pool's sizes and every zone it may use (its pool.zones pin, else the cluster's node-subnet zones, the AWSCluster's subnets tagged for nodes) when the reason is InsufficientCapacityError, since Karpenter gives a claim up for capacity only once every offering it asked for was refused and cuts its event after the first fleet error (#75); AWS's per-error hint of zones with capacity relayed only for zones outside the refused set, the next error of the same answer refusing the zone the first one named —, Karpenter's reason and message verbatim, and the way around it, naming what decided where the launch was tried (#65) and, as candidates for zones, only the cluster's node-subnet zones the answer did not refuse: 1 NodeClaim could not launch, the last (<pool>-5hpxb) at 2026-09-18T12:14:47Z — InsufficientInstanceCapacity for every size of the pool (g6e.2xlarge, g6e.4xlarge, g6e.8xlarge) in eu-central-1b, the zone the pool is pinned to (InsufficientCapacityError); Karpenter: "creating instance, insufficient capacity, …"; it retries while a pod waits — the pool is pinned to eu-central-1b — the model cache claim model-serving/hf-cache (volume pvc-…) lives in eu-central-1b and the pool's slice mounts it: re-run create_node_pool on the pool with zones naming one zone with capacity — not refused in this answer: eu-central-1a, eu-central-1c (the cluster's other node-subnet zones; capacity there is not promised) — with the cache on its slice then mounts that zone's model cache claim (the one Bound there, else hf-cache-<zone>, created there and kept; the weights downloaded once more), with cache false it serves from the node's local disk; wider sizes or another accelerator (a re-run of create_node_pool) give it more to choose from — for a pool with no pin … in every zone the pool may use (eu-central-1a, eu-central-1b, eu-central-1c — the cluster's node-subnet zones; the pool has no pin) …; it retries while a pod waits — every zone the pool may use was refused in the same answer, so there is no zone to move to (zones on a re-run would only pin the pool to one of them); Karpenter retries every few minutes (its unavailable-offering cache is three minutes) and the pool launches its node on its own once the family has capacity again; wider sizes or another accelerator … — read from the NodeClaims that carry the refusal (Launched or Registered False) and, for a claim Karpenter deleted at once for want of capacity, from its Warning event on the claim in the default namespace, which outlives the claim for the API server's event TTL; a refusal counts until Karpenter creates a claim after it), each pending, inProgress, done or failed, since the condition's lastTransitionTime or the object's creation (RFC3339), finishedAt for a done step. A pool created with prewarm carries a fourth step, prewarm: the placeholder Job's pod in the release namespace — pending while the pool release is not Ready and the Job does not exist (the chart creates it with the release's install; the release step says where the install stands), inProgress while the pod is pending (the first node launching, the scheduler's message beside it — or, when Karpenter could not launch it, the refusal in a line: the placeholder waits for the pool's first node, which Karpenter could not launch — InsufficientInstanceCapacity for g6e.2xlarge in eu-central-1a (the nodes step carries Karpenter's message)), holding node <name> or being preempted, done once preempted by the first workload or finished (the hold ended without one), failed when no node came within the Job's deadline (naming Karpenter's refusal when the nodes step has one), and done with a message when the Job is absent after the release installed (its TTL removed it, or prewarm was set on an existing pool); the step never decides the phase, and a pool without the option shows no such step. A pool under delete_node_pool stays listed with phase: removing, deleting: true and pending[] — the teardown's objects still present, pending or terminating — until its HelmRelease is gone, and by its MachinePool — removing, the terminating node named in the nodes step — until Karpenter's NodeClaim has gone with the terminated instance, minutes later (the release's uninstall does not wait for it; see create_node_pool). backend names the cluster's kserve backend with model-manager as list_clusters' serving.readiness.backend does: the backend a model on the cluster's pools is checked and loaded through. Every read is concurrent: the call stays inside the aggregator's deadline.
create_node_pool	Create a GPU node pool for a cluster, or update the pool of that name: the pool release of the gpu-node-pool chart (HelmRelease and OCIRepository in org-<org>, following the chart from compose.MinPoolChartVersion to the next major, or pinned exactly with chartVersion) with the Kubernetes version and Flatcar machine image of the cluster's current release — refused when the release runs ahead of the control plane — and a credential-free snapshot of the cluster's settings (registry credentials go into a valuesFrom Secret). A pool node boots with the NVIDIA driver from Flatcar's prebuilt, release-matched nvidia-drivers system extension (gpu-node-pool 0.7.4 pinned; the chart's default pool.nvidiaDriver.source: flatcar-sysext since 0.6.0; cluster-manager writes no nvidiaDriver block): the bootstrap enables the extension in /etc/flatcar/enabled-sysext.conf, the OS downloads it for its own version right after Ignition and merges it into /usr before the system starts, and the CDI unit loads the modules, creates the device nodes and writes the CDI specification the toolkit serves — no driver is built at boot. The extension ships with Flatcar 4344.0.0 and newer: a cluster whose release pins an older Flatcar is refused before anything is composed, naming its version and the way out — the pool needs a cluster release with a newer Flatcar (the chart's image-build source is not offered). Its /var/lib — containerd's image unpacks, the kubelet's directories, the pods' emptyDirs (a model's weights where no cache claim is mounted) and the containers' writable layers — is the node's NVMe instance store (gpu-node-pool 0.7.0, the chart's default pool.volumes.libSource: instance-store; cluster-manager writes no volumes block): the bootstrap formats the first Amazon EC2 NVMe Instance Storage device as the lib filesystem and mounts it at /var/lib, no gp3 volume is provisioned for it, and the EC2NodeClass carries instanceStorePolicy: RAID0 so Karpenter counts the store as the node's ephemeral storage. The store is local to the host, included in the instance price and goes with the node — nothing of a pool node outlives it. Every size of the curated families has one — 250 GB on an xlarge, 450 on a 2xlarge, 600 on a 4xlarge; the 8xlarge of g6 and g6e carry two devices of 450 GB and /var/lib gets the first, g5's and g4dn's 8xlarge one of 900; g4dn's smaller sizes 125 and 225 GB — so no size is refused, and the answer's sizes[] rows name it (instanceStoreGB across instanceStoreDisks devices of instanceStoreDiskGB; the last is what a node gives its pods). Its containerd runs with an unlimited locked-memory limit (gpu-node-pool 0.7.1, the bootstrap's memlock.conf drop-in, LimitMEMLOCK=infinity): a container inherits containerd's limit and has no CAP_IPC_LOCK to raise it, so under the unit's 8 MB default a serving runtime that mlock()s its weights died at engine start (giantswarm/agent-platform#564). mode: apply lands the objects as the caller, owned by the Cluster; dryRun returns the manifests and, on a re-run, the difference. A GitOps-owned object is never patched. When no GPU operator runs on the cluster (detected as the caller on the cluster itself), the <cluster>-gpu-operator release of the catalog's gpu-operator chart comes with the pool, configured from the two-row table read off the nodes (Flatcar: driver and toolkit off — the extension carries the driver, the image the toolkit, and the pool chart's bootstrap makes the toolkit serve them, see the chart's image contract; nvidia.com/gpu.deploy.driver=pre-installed: toolkit on; anything else refused), with Node Feature Discovery's worker pinned to the cluster's GPU pools (gpu-operator.node-feature-discovery.worker.affinity: giantswarm.io/machine-pool In [<cluster>-<pool>, …], the pool being created included; a re-run moves the pin) — the worker is what asserts nvidia.com/gpu.present, so GPU-family nodes of the general pool without a driver get neither the label nor the operands — and polling every 10 s (gpu-operator.node-feature-discovery.worker.config.core.sleepInterval: 10s; upstream's default is 60 s): the operator rolls the toolkit, device plugin and GPU feature discovery onto a node only once the worker has labelled it, so on a node that joins for one predictor a minute's poll was most of the ~100 s between Ready and nvidia.com/gpu allocatable; the rest of worker.config (the chart's PCI device class whitelist) merges in unchanged. On a pool node the release runs the validator's driver and toolkit validations (what proves the image's driver and CDI) without the validator's workload pods (validator.cuda.env and validator.plugin.env carry WITH_WORKLOAD=false; the chart's default starts a nvidia-cuda-validator-* pod on every fresh node beside the device plugin's start), the device plugin (what advertises nvidia.com/gpu, once the toolkit validation has written toolkit-ready) and GPU feature discovery (model-manager reads its nvidia.com/gpu.count, .memory and .product labels for its node budget) — and not mig-manager (migManager.enabled: false: the pool's accelerators have no MIG, and the operator schedules it on MIG-capable GPUs only) nor the DCGM exporter (dcgmExporter.enabled: false: nothing scrapes it out of the box, and on a fresh node it was an image pull and a pod initialising DCGM on the GPU while the device plugin brought the GPU up; an installation whose observability scrapes it turns it on with the chart's gpuOperator.dcgmExporter / --gpu-operator-dcgm-exporter). The worker and the operands tolerate the node's start-up taints that DaemonSets remove (gpu-operator.node-feature-discovery.worker.tolerations and gpu-operator.daemonsets.tolerations: the chart's defaults plus node.cluster.x-k8s.io/uninitialized and ebs.csi.aws.com/agent-not-ready, operator: Exists), so they are created as soon as the kubelet registers and the CNI is up and pull their images before the runtime pre-pull saturates the node — with the defaults alone the worker's pod waited for the taint remover and the EBS CSI node plugin to pull and run, 12–49 s after Ready on an installation, and nvidia.com/gpu was allocatable 73–108 s after it; not the Cilium taint (the operands use the pod network and need the CNI before their sandbox anyway) and never karpenter.sh/unregistered (Karpenter removes it itself at registration). A release cluster-manager does not compose keeps the chart's defaults; an operator the platform's release or a person provides is never re-created. When nothing provides serving on the cluster, the cluster's <cluster>-agent-platform slice release comes with the pool too (see enable_model_serving), updated in place on a re-run. After the pool, the cluster's own kserve backend is registered with model-manager — one per serving cluster: kserve (ConfigMap model-backend-kserve) for the installation's own cluster, kserve-<cluster> (model-backend-kserve-<cluster>) for a workload cluster, so pools on two workload clusters of one installation coexist, each cluster's models checked and loaded through its own backend (backend.backend in the answer); a workload cluster still registered under the shared kserve name by an earlier cluster-manager moves to its own name on the write, the shared document removed, and the own cluster's pool is refused while a workload cluster holds that name, naming the cluster and the re-run that frees it — the document naming every pool's instance shapes (the sizes below), so model-manager's check_fit judges a preset against a pool before it has a node and load_model refuses what no size hosts (model-manager 0.23.7 or newer): the one pool's as spec.kserve.gpuPool.instances, several pools' as spec.kserve.gpuPools, keyed by the pool's release name (the node label giantswarm.io/machine-pool, <cluster>-<name>), which model-manager judges and pins a model against (giantswarm/model-manager#152); delete_node_pool of a pool that is not the last re-writes it for the pools that remain. The answer lists the pool's sizes (sizes: the node as AWS lists it, its instance store and what it leaves a predictor after the kubelet's reservations and the fleet's daemonsets — a g6.xlarge leaves 3 vCPU / 11.9 GiB and gives its pods 250 GB of local disk) and, where the cluster publishes serving presets, the smallest size that hosts each (presetFit); a preset the accelerator could serve but no size of the pool hosts is a warning (warnings) — its predictor would sit Pending while Karpenter refuses every size (giantswarm/agent-platform#502). A size the family does not have is refused. prewarm: true launches the pool's first node with the release instead of with the first predictor (the chart's pool.prewarm.enabled): a one-shot placeholder Job in the release namespace holds one GPU at negative priority until the first workload preempts it, or its hold (the chart's default, 15 minutes) ends and Karpenter consolidates the empty node — a model named up front is served minutes earlier. The pool release does not wait for the chart's Jobs (install/upgrade.disableWaitForJobs: true; the operator and slice releases keep Helm's wait): Helm counts a Job as ready only once it completes, and the placeholder is meant to run until a workload preempts it — waited for, the install ran into its timeout and was uninstalled. Nor does its uninstall wait for the chart's objects to be gone (uninstall.disableWait: true; uninstall.timeout stays the default): the MachinePool goes only when EC2 has terminated the pool's last instance, minutes after its NodeClaim's delete and longer than helm-controller's timeout — waited for, the uninstall failed (UninstallFailed) and the release went with the retry minutes later; the MachinePool and the instance terminate on their own, list_node_pools naming the terminating node until they are gone. Only for the installation's own pool, where the release namespace is on the cluster the nodes join; a workload cluster's pool is refused, naming why. The Job is created with the release's install only: a re-run flips the value in place (would-update) but never launches a placeholder again; prewarm: false removes the block. Where the pool runs is the person's, and the zone brings its own cache (#65, #71): zones: ["eu-central-1a"] pins the pool's nodes to the named availability zone — pool.zones on the pool release, a topology.kubernetes.io/zone In requirement on the Karpenter NodePool, placeholder and predictors alike. The zone is checked against the cluster's node subnets (the AWSCluster's spec.network.subnets tagged giantswarm.io/role: nodes, the ones the chart's EC2NodeClass selects): a zone the cluster has no node subnet in is refused, naming the zones it has. And the pool's serving slice mounts that zone's model cache claim: a cache claim outlives the pool and is one EBS volume, bound in one zone by the first predictor that mounted it (#59) — a pool whose nodes come up elsewhere strands the next predictor Pending, and one zonal claim pinned every pool to a zone that had no capacity for the accelerator —, so the cache is one claim per zone: the slice release carries modelServing.cache.pvc.name with the claim of the zone — the hf-cache* claim Bound there (the one claim of before, hf-cache, in the zone its first predictor was in — reused, not doubled), else hf-cache-<zone> (<base>-<zone>, the base being --serving-cache-claim), which the connectivity chart creates where it does not exist and keeps: the first predictor of the pool binds it to a volume in the zone, a later pool in the zone reuses it, pools in other zones use theirs. Any combination of zones stands with the cache on, and the pool follows its cache (#79): with several zones the nodes are pinned to the zones named and the slice mounts the base claim, hf-cache, which the first predictor binds to a volume in its node's zone — one of the named — so that from then on every predictor mounting it runs there (the scheduler and Karpenter place a pod's node where its volume is), and a re-run of the same create_node_pool pins the pool's nodes to that zone; a claim Bound in one of the named zones already pins the pool to it at once, zonesNote naming the zones named, the one that stands and the ways out (cache: false to run across the zones, one zone alone to serve from it with its own claim). Claims Bound in several of the named zones are refused (refused{cacheClaims{claims, remedies}}: the slice mounts one, none is chosen for you), as is the base claim Bound outside every named zone (refused{cacheZone{claim, claimZone, zones, remedies}}: name its zone among the zones, one zone, or cache: false); a claim named after the one zone named but Bound elsewhere is refused as before (the ways out being to name the claim's zone, to pass cache: false, or to remove the claim). No claim is created per named zone up front: a claim binds only when a pod mounts it, and unbound claims are noise. Without zones the claims of the serving namespace decide: exactly one hf-cache* claim — Bound, the pool's nodes are pinned to its volume's zone (pool.zones: [eu-central-1b]) and the slice mounts it; not Bound yet, or its volume naming no zone (a network file system), nothing is pinned and zonesNote says what was found; its volume not readable as you, nothing is pinned and a warnings entry says why — never a guessed zone —; no claim, no zones block and nothing said (the chart creates hf-cache, bound by the first predictor); several claims are refused naming each with its zone and phase (refused{cacheClaims{claims, remedies}}): name zones. The claims not listable as you pin nothing and are a warnings entry. The answer names the pin and the claim: zones, zonesNote (nodes pinned to eu-central-1a, the zone named on create; the slice mounts the zone's model cache claim model-serving/hf-cache-eu-central-1a, which does not exist yet: … or nodes pinned to eu-central-1b: the model cache (claim model-serving/hf-cache, volume pvc-…) lives there …), cache{enabled, claim, note} (the claim the slice mounts and where it stands), cacheClaim{namespace, name, phase, volume, zone} (that claim as read on the cluster as you; null while it does not exist yet) and cacheClaims[] (every hf-cache* claim of the serving namespace as read); dryRun shows it, a re-run on an existing pool moves the pin and the claim (would-update, naming spec.values.modelServing.cache.pvc.name). The slice release is the cluster's one: its predictors mount the claim the last create_node_pool with the cache on named, so pools in several zones at once serve from the last-named zone's claim, and a predictor of another zone's pool restarted after that call waits for a node in the claim's zone. cache is off by default — without it the slice keeps its setting where the cluster's slice release runs, and a first slice serves without a cache (a claim is billed every month it exists and is never created unasked; cache: true is the opt-in) —; cache: false serves this pool's slice without the model cache: modelServing.cache.enabled: false on the <cluster>-agent-platform release, so no claim is applied or mounted, every predictor downloads its weights into its pod's ephemeral storage (the node's local disk) at each start, no zone follows from a claim — the zones named stand, several of them allowed, or nothing pins — and the existing claims are left as they are (Helm keeps them); the answer says so (cache{enabled, claim, note}, the note naming the existing claims' state), a re-run with a changed value is the slice's upgrade (would-update naming spec.values.modelServing.cache.enabled), and it is refused where someone else provides serving — the setting is that layer's. A family not offered in a pinned zone fails its launch visibly, named in list_node_pools' nodes step with what pinned the pool and the re-run that moves it. LLMInferenceServiceConfigs a serving layer that went left terminating in the release namespace are removed first, finalizer and all, when no llm-d controller runs (objects lists them, warnings says why), so the slice's release creates them afresh instead of adopting and losing them. The slice's serving runtime is the slim one (llm-d-slim/) when every GPU pool of the cluster is A10G, L4 or L40S, else the chart's llm-d-fast/. serving (default true) is whether the pool serves models: false creates a plain GPU pool — the pool and, where none runs, the GPU operator — composing no slice, registering no backend and reading nothing only serving needs (the ClusterIssuer, the serving runtime, the presets); refused with cache true and where cluster-manager serves on the cluster already. Where the slice is composed, a ClusterIssuer the person may not read is refused naming the right (get on clusterissuers.cert-manager.io at cluster scope) and the subjects the cluster binds as cluster-admin.
delete_node_pool	Remove what create_node_pool created, the pool's idle nodes first. The pool's nodes are read on the cluster as you — Karpenter's NodeClaims of the pool, the Nodes registered from them and the pods on each —, never from the MachinePool's provider IDs while the cluster is readable (that list lags a terminated instance by minutes). A node that holds nothing — no pod with a GPU or a KServe predictor's on it; a DaemonSet's pods, finished pods and pods preemptible by design (the pool's prewarm placeholder) do not count — is idle: its NodeClaim is deleted first, as you (Karpenter drains the node and terminates the instance within the pool's terminationGracePeriod), listed in objects, and the teardown completes in the same call — the pool release's uninstall does not wait for the MachinePool, which goes with the terminated instance minutes later, list_node_pools naming the terminating node meanwhile; dryRun lists the NodeClaims it would delete. Refused while a node of the pool is busy unless force — the refusal names each busy node with what holds it (node ip-… runs model-serving/qwen3-8b-fp8-kserve-… (1 GPU), a NodeClaim still launching, pods not readable as you) and the models served on the cluster (LLMInferenceService model-serving/qwen3-4b-instruct (Qwen/Qwen3-4B-Instruct-2507), a Pending one included) to unload first — and refused, with no busy node, while a model model-manager serves in the serving namespace has its predictor on no node (#59): every LLMInferenceService and InferenceService carrying model-manager's app.kubernetes.io/managed-by label counts, whatever its phase — one on a node of the pool is a busy node, one on no node is waiting for one (the kept cache's zone, or a size the family lacks there) and the delete would strand it: with the cluster's last pool the serving slice, its controller and the backend go too. The refusal names the model and its pod (LLMInferenceService model-serving/qwen3-8b (Qwen/Qwen3-8B): pod model-serving/qwen3-8b-kserve-…-xnkd2 Pending on no node, or no predictor pod yet) and the idle nodes that go with the pool once it is unloaded; a hand-made serving object does not count; a predictor waiting for another pool's node does not count while the cluster has another pool (its nodeSelector or required affinity names that pool's giantswarm.io/machine-pool, or Karpenter's latest Nominated event names a NodeClaim of that pool), and one that names no pool counts for every pool; the serving namespace's pods not readable as you is a refusal too. Beside the text the refusal carries a structured refused{nodes[], idle[], models[], unscheduled[], hint, readFrom} block as a second text content of the tool error ({"refused": {...}}), unscheduled naming the waiting models, readFrom being cluster, or machinePool when the cluster cannot be read as you and the MachinePool's replicas and provider IDs had to decide (the text says so). Refused for a pool cluster-manager did not create. With the cluster's last pool go the operator release, the slice release — unless another slice is on in it (sliceKept) — and the backend registration cluster-manager created; with a pool that is not the last, the backend document cluster-manager registered for the cluster is re-written for the pools that remain, right after the idle nodes (backend in the answer) — the one-pool form for the one left, keyed by release name for several. The slice goes in order (#28): its kserve-runtime-configs child release first, then every LLMInferenceServiceConfig of the release namespace is seen gone from the cluster — waited for while the llm-d controller runs to clear them, removed with serving.kserve.io/llmisvcconfig-finalizer taken off where nothing else will (the controller gone, force, or two minutes over; warnings says which) — and, the same moment, every model model-manager serves in the serving namespace is removed with serving.kserve.io/llmisvc-finalizer taken off (force took the controller away from under it, and a model stopped afterwards would sit terminating for good; its workload goes with it through its owner references; a serving object made by hand or through GitOps is named in warnings and left) — then the pool, the operator, the slice release and the backend, so nothing is left terminating for the next slice to adopt and lose. Without force a cluster that cannot be read as the caller is refused before anything is deleted.
enable_model_serving	Switch model serving on for a cluster, with or without a GPU pool: the cluster's one release of the agent-platform chart, <cluster>-agent-platform (HelmRelease and OCIRepository in org-<org>, delivered by the installation's Flux; the chart followed from the version the installation's own platform release runs — status.history[0].chartVersion of its HelmRelease, released by construction and known to work on the installation — to the next major (spec.ref.semver: ">=<version> <next major>", so a released agent-platform patch or minor reaches every slice without a re-run; an explicit --slice-chart-version is pinned exactly), with a floor of 4.27.0, the first chart whose serving slice places the predictors on a tainted GPU pool; a platform below it is refused, naming why; --slice-chart-version / serving.sliceChartVersion pins another version for a lab), with the serving slice on — the five KServe components and the llm-d well-known configs, modelServing with the nvidia RuntimeClass, the models Gateway at models.<domain> with the login issuer's JWT policy — its key set fetched from the platform's Dex service in-cluster on the installation's own cluster (dex.<namespace>.svc.cluster.local:5556 in plaintext, the namespace and port from the platform's gateway.jwksEgress, the source the platform's own JWT policies use — and gateway.jwksEgress itself in the slice's values, enabled with that namespace and port, the connectivity chart's precondition for an in-cluster host, without which the slice's connectivity child fails its render and serves no models Gateway; the chart's default, the public issuer on 443, yields no key set there and every id_token was refused as signed by an unknown key) and from the public issuer on a workload cluster, which cannot reach the installation's Service; the answer's slice.jwks names the source. global.domain, global.identity and the wildcard certificate where the platform names one, else a Certificate of models.<domain> from the fleet's ClusterIssuer — --models-certificate-issuer / serving.certificateIssuer, default letsencrypt-giantswarm are read from the installation's own platform release, never invented; every cluster gets the target knob (gitops.target.kubeConfig.secretRef.name: <cluster>-kubeconfig, see Delivery below); a workload cluster gets global.domain: <cluster>.<base domain> and components.agentgateway on, the installation's own cluster the platform's domain and agentgateway off (the platform's release owns the data plane); a cluster whose API serves no Gateway kind (a workload cluster created with kubectl-gs; read from the CRD gateways.gateway.networking.k8s.io) gets components.gateway-api-crds on, the fleet's Gateway API v1.6.1 CRDs (standard channel) ordered before the connectivity release and agentgateway, the slice then following agent-platform from at least 4.117.0, the first chart with the component, also where the installation's own platform release is held on an older chart; once composed they stay composed, and the slice's teardown (disable_model_serving, delete_node_pool of the last pool) removes them after the connectivity release and agentgateway are uninstalled unless Gateway API objects none of the slice's releases rendered remain, which keep them, named in warnings; a cluster that serves the Gateway API gets nothing composed and the answer's slice.gatewayAPI names the version found; a Gateway CRD that cannot be read is refused; the chart's Flux engine and model-manager are always off. With exactly one GPU pool on the cluster the predictors are pinned to it (modelServing.gpuPool.nodeSelector, and modelServing.serving.nodeSelector for the pinned chart). A release that exists is updated in place — the re-run moves the pin to the version the platform runs today; a release of the chart under another name is refused, naming it; a chart-provided or hand-installed serving layer is left alone. Registers the kserve backend, its document naming the pinned pool's instance shapes, read from the pool's release (see create_node_pool), when the cluster has exactly one pool. Stranded terminating LLMInferenceServiceConfigs are healed first, as create_node_pool does. With the cache on the slice mounts the model cache claim its release mounts already — the zone's claim the last create_node_pool named (modelServing.cache.pvc.name), kept by the re-run, which never moves the cache to another zone — else the chart's default hf-cache; the answer names it (cache{enabled, claim, note}). cache is off by default, as create_node_pool's (without it the release keeps its setting, a first slice serves without a cache); cache: false serves without the model cache: modelServing.cache.enabled: false on the release — no claim applied or mounted, the weights downloaded into each predictor pod's ephemeral storage, the existing claims left as they are — and the answer says so; a re-run with a changed value is the slice's upgrade.
disable_model_serving	Remove the slice release cluster-manager created and its backend registration. Refused while models are served on the cluster (named) unless force, and plainly when the cluster cannot be read as the caller; refused for a release cluster-manager did not create. The same ordered teardown as delete_node_pool's last pool, the served models and the configs removed finalizer and all once no controller runs, the slice release, then the serving namespace: removed once nothing is left in it, else marked retired (cluster-manager.giantswarm.io/retired, naming what holds it — the model cache claims are remove_model_cache's). A re-run once the slice is gone, or while its release is still being deleted, completes the namespace step (delete_node_pool of the pool removed since alike), a namespace Kubernetes is still emptying answered terminating, one gone not found; enable_model_serving and create_node_pool take a retired namespace back.
remove_model_cache	Remove a cluster's model cache: the hf-cache* claims of the serving namespace — gp3 volumes that outlive every pool and the slice release by design and are billed every month they exist, filled or not — deleted on the cluster as the caller, the volume going with the claim under the class's Delete reclaim policy (a Retain volume is a warning: it stays, and is billed, until deleted by hand). Where cluster-manager's slice release runs with the cache on it is upgraded to modelServing.cache.enabled: false first — the connectivity chart applies the claim from a hook on every upgrade, so a claim deleted under a slice with the cache on would come back — and every predictor of the cluster downloads its weights into its pod's ephemeral storage at each start from then on (about 90 s more per cold start); a later create_node_pool or enable_model_serving with cache: true creates the claim anew. claim names one claim; default every claim. Refused while a pod of the serving namespace mounts a claim to be removed (refused{models, hint}: the served models to unload first — a claim a pod mounts is not deleted until the pod is gone), plainly while the cluster cannot be read as the caller, and where someone else provides serving. The answer names the claims removed with their size, tier and monthly list price (removedClaims[]), every claim as read (cacheClaims[]), the slice's objects and cache.note. dryRun lists what would go.

The model cache is the cluster's setting with a standing cost (#83). The slice release is the cluster's one, so every pool serves from the claim it mounts, and a claim is a gp3 volume billed every month it exists — after every pool is removed too — until remove_model_cache deletes it. Every claim an answer names (list_clusters' serving.readiness.cacheClaims[], the writes' cacheClaim and cacheClaims[]) carries its capacity, StorageClass and tier (volume type, IOPS, throughput, read from the class or, the class gone, from the claim's tier annotations; tierSource says which), since when it stands, its monthly list price in the cluster's region (price{monthlyUSD, source, asOf}, or priceNote saying why there is none — a class that is gone, on a claim without tier annotations, leaves the tier unknown, and a storage-only figure is never passed off as the price) and, in list_clusters, whether the slice mounts it (mounted). The cache block of create_node_pool and enable_model_serving carries the same for the claim the slice mounts — as read (exists, since), or as the connectivity chart at the slice's version would create it (capacity, tier, monthlyPriceUSD from the chart's defaults) — and its note says the cost stands until the cache is removed. cache: false while the cluster's slice release runs with the cache on is refused (refused{cacheOn{claim, claimName, remedies}}): the flip would switch the cache off for the models served on every pool while the claim stays and keeps costing; leave the cache on, or remove it. A first slice with cache: false stands, and cache: true on a slice that ran without it is the slice's upgrade for every pool, said so in the note. The prices are AWS's published list prices (hack/aws-on-demand-prices.py regenerates internal/compose/prices_table.go, EC2 on-demand and EBS gp3, for the regions the fleet runs in).

Every write tool takes dryRun (the rendered manifests, or on a re-run the difference) and mode: apply | commit. apply lands the objects on the installation as the caller. commit (create_cluster, delete_cluster, create_node_pool and delete_node_pool) opens the pull request as the caller in the git repository that owns the cluster, or for a new cluster its organization, instead — see Commit mode below. An apply answers within the aggregator's deadline for a tool call: everything it reads is read once and concurrently before anything is composed, the objects are planned together (every refusal before the first write) and land one after the other — create_node_pool: pool, slice, the slice's backend registration, operator, the objects that do not exist yet before the updates. A write there is no budget left for (--apply-budget, default 8 s, counted from the call's start without the warm-up of the workload cluster's client — its first answer, the connection a call after a start opens cold —; the request's own deadline when earlier) is not started: the answer comes back with partial: true, the objects not reached as pending and nextStep naming the re-run, which writes them first. The deletes take the same budget: delete_node_pool's last pool and disable_model_serving remove the serving slice in order — the llm-d controller's child release; its well-known LLMInferenceServiceConfigs once no controller runs on the cluster (its webhook denies every delete of them while it does, and nothing clears their finalizer once it is gone, so cluster-manager removes them, finalizer and all); the configs' child release (a child whose uninstall failed before is asked to retry now); the operator, the backend registration, the slice release; and the pool's own objects last, its HelmRelease the very last, so the re-run finds the pool and continues where the teardown stands. Every phase is timed in the log at debug (--verbose).

A workload cluster's prerequisites

A GPU pool on a workload cluster needs two things of the cluster's values, which create_node_pool checks first — dry run included, before anything of the cluster is read — and refuses once, naming every one missing with the values to add:

• global.connectivity.baseDomain, the installation's base domain, for the pool's bootstrap. cluster-manager reads the cluster's effective values: its own (the HelmRelease's, or the App's values ConfigMaps) over the installation's cluster values that every organization's namespace carries (cluster-app-installation-values). A cluster made by kubectl gs template cluster gets them through app-operator's catalog layer, which cluster-manager does not read; their copy in the organization's namespace stands in for it. Neither carrying a base domain is the refusal.

• An apiserver that trusts the installation's identity provider: cluster-manager reads the cluster as you with the installation's Dex id_token, so the cluster's global.controlPlane.oidc names the Dex issuer with the audience the installation's apiserver accepts (--cluster-oidc-client-id) — the plain block, or with structuredAuthentication.enabled one entry of its issuers. kubectl gs template cluster sets none. Checked when the server knows its provider (Dex with --cluster-oidc-client-id); create_cluster composes it into every cluster it makes. For a cluster made otherwise, add to its user values (<name>-userconfig), with the issuer the refusal names:

  global:
    controlPlane:
      oidc:
        issuerUrl: https://dex.<installation base domain>
        clientId: dex-k8s-authenticator
        usernameClaim: email
        groupsClaim: groups

  The change rolls the cluster's control plane (about 15 minutes); re-run once it is done.

Delivery

Giant Swarm installations enforce Flux multi-tenancy on every HelmRelease outside the platform's namespaces (the flux-multi-tenancy Kyverno policy, admission-enforced, on org-<org> among others): a release carries spec.serviceAccountName or spec.kubeConfig.secretRef.name, and leaves its namespace (targetNamespace, storageNamespace) only through a kubeconfig. The releases cluster-manager composes take the two shapes the fleet's own releases in org-<org> take, by what they render — for the installation's own cluster exactly as for a workload cluster:

release	objects live	delivery
<cluster> (the cluster itself, create_cluster)	in org-<org> on the installation (the Cluster API objects the release chart renders)	serviceAccountName: automation, as the pool release
<cluster>-gpu-operator	on the cluster, in kube-system	kubeConfig.secretRef.name: <cluster>-kubeconfig — the cluster's Cluster API kubeconfig Secret, which exists for the installation's own cluster too and points at its own API server
<cluster>-<pool>	in org-<org> on the installation (the Cluster API objects)	serviceAccountName: automation — the org's tenant ServiceAccount (--tenant-service-account / flux.tenantServiceAccount; empty renders none, for a lab without the policy)
<cluster>-agent-platform	in org-<org> on the installation (the meta chart's child HelmReleases and OCIRepositories)	serviceAccountName: automation; the target knob gitops.target.kubeConfig.secretRef.name: <cluster>-kubeconfig in its values puts the kubeconfig on every child, which then installs into the cluster

The dry run cannot see admission, so the goldens hold the policy's constraint on every composed HelmRelease.

Running

cluster-manager serve --kubeconfig ~/.kube/config

Every flag has an environment variable named next to it in --help; flags win. The chart in helm/cluster-manager renders the Deployment, the Service, the MCPServer CR for muster and, with oauth.enabled, the mcp-oauth resource-server flags from the platform identity contract (global.identity).

Traces go out over OTLP when OTEL_EXPORTER_OTLP_ENDPOINT is set (the chart's observability.otel.endpoint); the standard OTEL_* variables configure the exporter and the sampler. Spans: POST /mcp, mcp.<method> per JSON-RPC request (gen_ai.tool.name on mcp.tools/call) and tool.<name> per tool handler.

Development

make build-linux-amd64   # the binary the Dockerfile expects
go test ./...
make helm-lint helm-template

Releases are automatic: every merge to main is tagged from Conventional Commits and CircleCI publishes the image to gsoci.azurecr.io/giantswarm/cluster-manager and the chart to the Giant Swarm catalog.

Commit mode

create_node_pool and delete_node_pool with mode: commit write the releases apply would land as files into the repository that owns the cluster and open one pull request as the caller:

• Where. The cluster's HelmRelease or App (named like the cluster) carries the labels of the Flux Kustomization that applies it; the Kustomization's GitRepository and spec.path name the repository, branch and directory (list_clusters' commitTarget). The files go to <path>/cluster-manager/: one file per release (<release>.yaml, the OCIRepository and the HelmRelease), the pool's registry credentials in <release>-values-secret.enc.yaml encrypted for the age recipients the repository's .sops.yaml names for the path (a repository without age recipients for it is refused), and a kustomization.yaml listing them. A parent directory with its own kustomization.yaml gets the cluster-manager entry; without one Flux generates it and includes the directory by itself.
• As whom. The server is registered with muster pinned to the GitHub App giantswarm-cluster-manager (chart github.enabled): muster runs the App's consent once per person and puts their user token on every call, which the server verifies with GET /user and opens the pull request with; auth.forwardIdentity puts the person's IdP ID token in X-Muster-Id-Token next to it, validated as a forwarded bearer is otherwise and used for Kubernetes, so apply mode is unchanged. One registration carries both. get_info answers modes.commit: true only then.
• What the answer says. commit{repository, base, directory, kustomization, prune, branch, files[], pullRequest, number, author, liveSteps[]}; dryRun shows every file with its content (a secret file without) and opens nothing; a re-run whose files the base already carries opens nothing. The kserve backend registration is model-manager's runtime state on the installation, not a file of the cluster's repository: it is written live, as in apply.
• A new cluster. create_cluster with mode: commit follows the cluster's Organization CR to the Flux Kustomization that reconciles it and that Kustomization's GitRepository and spec.path (a gitops-template root). The files go to organizations/<org>/workload-clusters/. <name>/ holds the objects apply would create, one file each, and its own kustomization.yaml. <name>.yaml is the cluster's Flux Kustomization <installation>-clusters-<name>, which builds <name>/ with the organization Kustomization's source, service account, interval, timeout and decryption, and prunes. The entry for <name>.yaml goes into workload-clusters/kustomization.yaml, which is created when absent. An Organization that no Kustomization reconciles is refused, and so is a root without organizations/<org>/. The cluster's node pools then go in mode commit to <name>/cluster-manager/, as for any cluster.
• A cluster's removal. delete_cluster with mode: commit removes <name>/, <name>.yaml and its entry in one pull request, and removes the kserve backend registration live. A cluster the repository does not carry is refused: remove it in mode apply. When the organization's Kustomization prunes, the merge removes the cluster's Kustomization, and with it the cluster. When it does not, liveSteps names delete_cluster in mode apply after the merge. That call deletes the cluster's Kustomization, which has left the organization's inventory, and its prune removes the release and so the cluster. The same call once more removes what the uninstall left.
• Removal. The removal pull request deletes the pool's files (with the last pool the operator's and the slice's too) and the kustomization entries; the nodes guard runs first as in apply. On a Kustomization that does not prune, liveSteps names delete_node_pool in mode apply after the merge, which tears the serving layer down in order. Apply mode refuses to change or delete a release Flux still applies from git (the Kustomization's inventory lists it) and removes what a merged removal left behind.