From 548a26a20462512d017eedfaf6ca1f259a9afc3b Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 21 Jul 2026 10:51:37 -0700 Subject: [PATCH 01/13] Add design for managed metrics collection Collecting a deployment's metrics is hand-wired: an operator writes a PodMonitor per deployment, keeps it in sync with the serving shape (leader/worker, prefill/decode), and removes it on teardown. This design has Modelplane compose the collection instead, per source it owns, the engine and the endpoint picker where present, on by default with an opt-out. It covers #269, with a section per source and a diagram. Signed-off-by: Dennis Ramdass --- design/metrics.md | 208 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 208 insertions(+) create mode 100644 design/metrics.md diff --git a/design/metrics.md b/design/metrics.md new file mode 100644 index 000000000..012a916f0 --- /dev/null +++ b/design/metrics.md @@ -0,0 +1,208 @@ +# Metrics collection + +**Status:** Draft +**Date:** July 2026 +**Author:** Dennis Ramdass + +This document proposes managed Prometheus metrics collection for a deployment's +serving stack. Modelplane composes a `PodMonitor` for each source it owns, the +engine and, where present, the endpoint picker, so collection comes up with the +deployment and is removed with it, with nothing to assemble by hand. It is on by +default. It builds on [design.md](./design.md) and addresses +[#269](https://github.com/modelplaneai/modelplane/issues/269). + +## Summary + +Collecting a deployment's metrics is hand-wired today. vLLM exposes `/metrics` on +its serving port, the serving stack runs a Prometheus on every workload cluster +with open `PodMonitor` discovery, and an operator writes a `PodMonitor` by hand to +scrape the engine pods (the [#264](https://github.com/modelplaneai/modelplane/issues/264) +example). Nothing owns that wiring. The operator builds it by hand and keeps it in +sync with the deployment's shape. They delete it on teardown. + +Instead, Modelplane composes the collection, per source: + +- **Engine metrics**, for every deployment. +- **Endpoint picker metrics**, wherever Modelplane composes an endpoint picker + (multi-pod Unified and prefill/decode). + +Both come up with the deployment, are reclaimed with it, and share one opt-out +field. On by default: + +```yaml +spec: + replicas: 1 + template: + spec: + metrics: + enabled: false # default true; collection is composed unless disabled + engines: + - name: qwen + # ... +``` + +The `ModelCache` and `ModelDeployment` specs are otherwise unchanged. + +## Why this is small + +Three existing pieces do the hard parts: + +- **The serving label already spans every shape.** `modelplane.ai/serving` is on + standalone pods, LeaderWorkerSet leaders, and both prefill/decode engines (it's + the label the InferencePool selects on). One selector on it follows the shape, + so the leader/worker and prefill/decode branching needs no special casing. +- **Prometheus already discovers openly.** `compose-serving-stack` runs + kube-prometheus-stack with `podMonitorSelectorNilUsesHelmValues: false` and an + empty namespace selector, so a composed `PodMonitor` is scraped with no operator + action. It already scrapes the gateway's Envoy proxies this way; the engine and + picker are the per-deployment sources still missing. +- **Modelplane owns the picker.** The EPP is Modelplane's own Deployment, so its + metrics port and flags are ours to set. + +## Engine metrics + +For every deployment, `compose-model-replica` composes a `PodMonitor` on the +workload cluster (the same place it composes the Service and InferencePool), +selecting the replica's serving pods by `modelplane.ai/serving`: + +```yaml +apiVersion: monitoring.coreos.com/v1 +kind: PodMonitor +metadata: + name: -metrics + namespace: default +spec: + selector: + matchLabels: + modelplane.ai/serving: + podMetricsEndpoints: + - port: http + path: /metrics + interval: 30s +``` + +It has no status of its own, so it uses default readiness (Ready once synced), +like the Service. No `mrap.yaml` change is needed: it's a provider-kubernetes +`Object`, already activated, and the CRD is installed by the Prometheus stack. + +### Naming the port + +The engine's serving port carries `/metrics`, so the backends give it a name (for +example `http`) in `native.py`, `llmd.py`, and the decode engine in `routing.py`. +The pd-sidecar's port stays unnamed. The `PodMonitor` then scrapes that port by +name, which matters for prefill/decode: the decode engine serves on 8001 because +the pd-sidecar takes 8000, so a `PodMonitor` matching port 8000 by number scrapes +the sidecar on decode pods, not the engine. Referencing the port by name scrapes +the engine on every pod regardless of its number. Naming is additive; the Service +and probes that reference the port by number are unaffected. + +### What gets scraped + +The `PodMonitor` ingests everything the engine exposes on `/metrics`. What the +engine emits is the user's, set through engine flags like everything else +(vLLM's `--disable-log-stats` and friends), so Modelplane doesn't decide which +metrics exist. It scrapes what's there. + +Filtering at ingestion, dropping specific metrics for cost or cardinality, is a +Prometheus `metricRelabelings` knob. It's out of scope for now, because a +selection surface reintroduces the per-deployment wiring #269 removes. The nested +`metrics` object leaves room to add it if a real need appears. + +## Endpoint picker metrics + +Multi-pod Unified and prefill/decode serving front the engines with an endpoint +picker (EPP) that Modelplane composes, today +`llm-d-router-endpoint-picker:v0.9.0`. The EPP makes the routing decisions the +engine metrics don't show: endpoint selection, prefix-cache-aware scoring, and the +prefill/decode split. It exposes a rich `llm_d_epp_*` metric set (request and +error totals, TTFT and per-output-token latency, in-flight requests, pool KV-cache +usage, and scheduler and disaggregation timings), the signal for whether that +routing is working. + +Because Modelplane owns the EPP Deployment and its args, collecting these is +straightforward. The EPP serves `/metrics` on a port set by `--metrics-port` +(default 9090), and its auth is a flag, `--metrics-endpoint-auth`. Modelplane sets +`--metrics-endpoint-auth=false` (the metrics are non-sensitive routing stats +reachable only in-cluster), declares the 9090 port on the Deployment, and composes +a plain `PodMonitor` for it, the same shape as the engine's. It's composed +alongside the EPP objects in `routing.py`, so it exists wherever an EPP does. No +ClusterRole, token, or TLS to manage. + +## The opt-out field + +`spec.template.spec.metrics.enabled` (boolean, default true), copied down to +`ModelReplica.spec`, governs both PodMonitors. A nested `metrics` object leaves +room for `interval` or `path` later; only `enabled` is defined now. +`compose-model-replica` composes the engine `PodMonitor`, and the EPP path +composes the picker's, unless `enabled` is false. + +## Architecture + +```mermaid +flowchart LR + subgraph rep["compose-model-replica composes"] + ENG["engine pods\n(label modelplane.ai/serving)"] + EPM["engine PodMonitor"] + EPP["endpoint picker\n(multi-pod / prefill-decode)"] + XPM["EPP PodMonitor"] + end + PROM["cluster Prometheus\n(open PodMonitor discovery)"] + EPM -->|"scrape :http /metrics"| ENG + XPM -->|"scrape :9090 /metrics"| EPP + PROM -->|discovers| EPM + PROM -->|discovers| XPM + classDef new fill:#ffb74d,stroke:#e65100,stroke-width:3px,color:#000; + class EPM,XPM new +``` + +## Alternatives considered + +### Opt-in, default off + +Closer to today, because the operator still has to remember a toggle. The issue +asks for collection that is managed for them, so the default is on. Anyone who +doesn't want scraping opts out. + +### Scrape the engine by port number, not name + +The manual example matches `targetPort: 8000`. That scrapes the pd-sidecar on +prefill/decode decode pods instead of the engine, and it re-couples the +`PodMonitor` to a number that moves with the shape. Naming the port removes both +problems. + +### Authenticate the EPP metrics endpoint + +The EPP can serve `/metrics` behind controller-runtime auth (a `ClusterRole` with +`nonResourceURLs: /metrics` plus a bearer token). Since Modelplane owns the EPP +args and the endpoint carries non-sensitive routing stats reachable only +in-cluster, `--metrics-endpoint-auth=false` collects them with a plain +`PodMonitor` and nothing to manage. Auth would add a `ClusterRole`, a binding, and +a token for no gain here. + +### One cluster-wide PodMonitor from the serving stack + +`compose-serving-stack` could install a single `PodMonitor` selecting all serving +pods (`modelplane.ai/serving` Exists), close to the manual example. It would have +no per-deployment lifecycle: it wouldn't come and go with a deployment, and it +couldn't be opted out per deployment. Composing per replica ties collection to the +thing it observes. + +### ServiceMonitor instead of PodMonitor + +A `ServiceMonitor` scrapes through a Service, so it needs a Service per scrape +target. The engine and picker pods are the targets, and a `PodMonitor` scrapes +them directly, matching the manual path. + +## Open questions + +- **Port name.** `http` (the serving port that also serves `/metrics`) versus + `metrics`. Leaning `http`, since it's the one serving port. +- **Configurability.** `interval` and `path` are fixed (30s, `/metrics`) for now. + The nested `metrics` object leaves room to add them if a real need appears. + +## Interaction with #264 + +The [#264](https://github.com/modelplaneai/modelplane/issues/264) example +documents the manual path. Once collection is composed, that example drops its +hand-written `podmonitor.yaml` and shows the opt-out field instead. + From 02c8cb685d5bd526d1e306f31e3e39d70ba5cf17 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 21 Jul 2026 13:31:53 -0700 Subject: [PATCH 02/13] Nest metrics under observability and address review scope Align the opt-out field with #77 by nesting it at observability.metrics.enabled, so traces and logs can join it later. Clarify that the 30s scrape is an observability default and not a routing input (the EPP scrapes engines on its own fast loop), that cache/PVC observability is out of scope, and that an existing hand-written PodMonitor must be deleted on upgrade. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/metrics.md | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-) diff --git a/design/metrics.md b/design/metrics.md index 012a916f0..af3e74c53 100644 --- a/design/metrics.md +++ b/design/metrics.md @@ -27,15 +27,17 @@ Instead, Modelplane composes the collection, per source: (multi-pod Unified and prefill/decode). Both come up with the deployment, are reclaimed with it, and share one opt-out -field. On by default: +field. This is serving metrics; `ModelCache` PVC and hydration observability is a +separate concern and out of scope here. On by default: ```yaml spec: replicas: 1 template: spec: - metrics: - enabled: false # default true; collection is composed unless disabled + observability: + metrics: + enabled: false # default true; collection is composed unless disabled engines: - name: qwen # ... @@ -108,6 +110,13 @@ Prometheus `metricRelabelings` knob. It's out of scope for now, because a selection surface reintroduces the per-deployment wiring #269 removes. The nested `metrics` object leaves room to add it if a real need appears. +This scrape feeds Prometheus for dashboards and alerting, so the 30s interval is +an observability default, not a routing input. The EPP scrapes engine `/metrics` +on its own fast internal loop for routing decisions, so routing latency doesn't +depend on this interval. An engine pod exposes on the order of dozens of series +(request and latency histograms, KV-cache, throughput); retention and storage +sizing are the serving stack's Prometheus configuration, not this composition. + ## Endpoint picker metrics Multi-pod Unified and prefill/decode serving front the engines with an endpoint @@ -130,11 +139,13 @@ ClusterRole, token, or TLS to manage. ## The opt-out field -`spec.template.spec.metrics.enabled` (boolean, default true), copied down to -`ModelReplica.spec`, governs both PodMonitors. A nested `metrics` object leaves -room for `interval` or `path` later; only `enabled` is defined now. -`compose-model-replica` composes the engine `PodMonitor`, and the EPP path -composes the picker's, unless `enabled` is false. +`spec.template.spec.observability.metrics.enabled` (boolean, default true), copied +down to `ModelReplica.spec`, governs both PodMonitors. The field sits under an +`observability` object to match [#77](https://github.com/modelplaneai/modelplane/issues/77), +which puts `observability.traces` on `ModelService`; traces and logs can slot +beside `metrics` later. The `metrics` object itself leaves room for `interval` or +`path`; only `enabled` is defined now. `compose-model-replica` composes the engine +`PodMonitor`, and the EPP path composes the picker's, unless `enabled` is false. ## Architecture @@ -205,4 +216,8 @@ them directly, matching the manual path. The [#264](https://github.com/modelplaneai/modelplane/issues/264) example documents the manual path. Once collection is composed, that example drops its hand-written `podmonitor.yaml` and shows the opt-out field instead. + +On upgrade, an existing hand-written `PodMonitor` has to be deleted, or it +double-scrapes the same pods alongside the composed one. This warrants a release +note. From 4e5a671445f1d7563bf97646b71a2e73e1795eda Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 22 Jul 2026 09:21:54 -0700 Subject: [PATCH 03/13] Reframe metrics around always-on collection and central aggregation Adopt Nic's direction: step back to what's worth monitoring (data plane, substrate, control plane, fleet roll-up), make PodMonitor collection always-on at every layer instead of a per-deployment opt-out, and add a central Modelplane Prometheus that the per-cluster instances feed and that also scrapes the control plane. Drops the observability.metrics.enabled toggle. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/metrics.md | 298 ++++++++++++++++++++++++---------------------- 1 file changed, 155 insertions(+), 143 deletions(-) diff --git a/design/metrics.md b/design/metrics.md index af3e74c53..0d61c9d66 100644 --- a/design/metrics.md +++ b/design/metrics.md @@ -4,68 +4,60 @@ **Date:** July 2026 **Author:** Dennis Ramdass -This document proposes managed Prometheus metrics collection for a deployment's -serving stack. Modelplane composes a `PodMonitor` for each source it owns, the -engine and, where present, the endpoint picker, so collection comes up with the -deployment and is removed with it, with nothing to assemble by hand. It is on by -default. It builds on [design.md](./design.md) and addresses -[#269](https://github.com/modelplaneai/modelplane/issues/269). - -## Summary - -Collecting a deployment's metrics is hand-wired today. vLLM exposes `/metrics` on -its serving port, the serving stack runs a Prometheus on every workload cluster -with open `PodMonitor` discovery, and an operator writes a `PodMonitor` by hand to -scrape the engine pods (the [#264](https://github.com/modelplaneai/modelplane/issues/264) -example). Nothing owns that wiring. The operator builds it by hand and keeps it in -sync with the deployment's shape. They delete it on teardown. - -Instead, Modelplane composes the collection, per source: - -- **Engine metrics**, for every deployment. -- **Endpoint picker metrics**, wherever Modelplane composes an endpoint picker - (multi-pod Unified and prefill/decode). - -Both come up with the deployment, are reclaimed with it, and share one opt-out -field. This is serving metrics; `ModelCache` PVC and hydration observability is a -separate concern and out of scope here. On by default: - -```yaml -spec: - replicas: 1 - template: - spec: - observability: - metrics: - enabled: false # default true; collection is composed unless disabled - engines: - - name: qwen - # ... -``` - -The `ModelCache` and `ModelDeployment` specs are otherwise unchanged. - -## Why this is small - -Three existing pieces do the hard parts: - -- **The serving label already spans every shape.** `modelplane.ai/serving` is on - standalone pods, LeaderWorkerSet leaders, and both prefill/decode engines (it's - the label the InferencePool selects on). One selector on it follows the shape, - so the leader/worker and prefill/decode branching needs no special casing. -- **Prometheus already discovers openly.** `compose-serving-stack` runs - kube-prometheus-stack with `podMonitorSelectorNilUsesHelmValues: false` and an - empty namespace selector, so a composed `PodMonitor` is scraped with no operator - action. It already scrapes the gateway's Envoy proxies this way; the engine and - picker are the per-deployment sources still missing. +This document proposes managed metrics collection and aggregation across a +Modelplane deployment. Modelplane composes a `PodMonitor` for each source it owns on +every cluster, always on, and feeds them into one central Prometheus an operator +points dashboards and alerting at. It builds on [design.md](./design.md) and +addresses [#269](https://github.com/modelplaneai/modelplane/issues/269). + +## What to monitor + +There are four things worth watching in a Modelplane deployment. + +1. **Inference signal (data plane).** vLLM's `/metrics`, the EPP's `llm_d_epp_*`, + and Envoy. TTFT, tokens per second, queue depth, KV-cache occupancy, and request + and error rates per model. It answers "is my model serving well, and is it + saturated?" +2. **Substrate health.** The stack Modelplane installs on each workload cluster: is + the gateway up, are cert-manager, the LeaderWorkerSet controller, and the NVIDIA + DRA driver healthy, are GPUs allocatable. "Is the machinery on this cluster + working?" +3. **Control-plane health.** Modelplane itself. Crossplane reconcile rates and + errors, function latency and panics, the fleet scheduler placing replicas, and XR + `Ready`/`Synced`. "Is the thing I operate working?" +4. **Fleet roll-up.** Across every cluster and deployment: total capacity, GPU + usage, how many deployments are degraded, and cost. + +An MD author cares about (1). The platform team owns the rest. A per-deployment +opt-in would cover only (1). Even then, the platform team has to plumb the metrics +somewhere the author can read them. So this design makes collection always on at +every layer and aggregates it centrally. It does not ask an MD author to manage a +toggle for something the platform team consumes. + +## Collection: always-on PodMonitors + +On each cluster, Modelplane composes a `PodMonitor` for every source it owns, with +no opt-in or opt-out. Three existing pieces make this cheap: + +- **The serving label spans every shape.** `modelplane.ai/serving` is on standalone + pods, LeaderWorkerSet leaders, and both prefill/decode engines (it's the label the + InferencePool selects on). One selector on it follows the shape, so the + leader/worker and prefill/decode branching needs no special casing. +- **Prometheus discovers openly.** `compose-serving-stack` runs kube-prometheus-stack + with `podMonitorSelectorNilUsesHelmValues: false` and an empty namespace selector, + so a composed `PodMonitor` is scraped with no operator action. It already scrapes + the gateway's Envoy proxies this way. - **Modelplane owns the picker.** The EPP is Modelplane's own Deployment, so its metrics port and flags are ours to set. -## Engine metrics +The sources are the engine (per deployment), the endpoint picker (wherever one +exists), and the serving-stack components (per cluster). + +### Engine metrics -For every deployment, `compose-model-replica` composes a `PodMonitor` on the -workload cluster (the same place it composes the Service and InferencePool), -selecting the replica's serving pods by `modelplane.ai/serving`: +`compose-model-replica` composes a `PodMonitor` on the workload cluster (the same +place it composes the Service and InferencePool), selecting the replica's serving +pods by `modelplane.ai/serving`: ```yaml apiVersion: monitoring.coreos.com/v1 @@ -83,141 +75,161 @@ spec: interval: 30s ``` -It has no status of its own, so it uses default readiness (Ready once synced), -like the Service. No `mrap.yaml` change is needed: it's a provider-kubernetes -`Object`, already activated, and the CRD is installed by the Prometheus stack. +It has no status of its own, so it uses default readiness (Ready once synced), like +the Service. No `mrap.yaml` change is needed: it's a provider-kubernetes `Object`, +already activated, and the CRD is installed by the Prometheus stack. ### Naming the port The engine's serving port carries `/metrics`, so the backends give it a name (for example `http`) in `native.py`, `llmd.py`, and the decode engine in `routing.py`. The pd-sidecar's port stays unnamed. The `PodMonitor` then scrapes that port by -name, which matters for prefill/decode: the decode engine serves on 8001 because -the pd-sidecar takes 8000, so a `PodMonitor` matching port 8000 by number scrapes -the sidecar on decode pods, not the engine. Referencing the port by name scrapes -the engine on every pod regardless of its number. Naming is additive; the Service -and probes that reference the port by number are unaffected. - -### What gets scraped +name, which matters for prefill/decode: the decode engine serves on 8001 because the +pd-sidecar takes 8000, so a `PodMonitor` matching port 8000 by number scrapes the +sidecar on decode pods, not the engine. Referencing the port by name scrapes the +engine on every pod regardless of its number. Naming is additive; the Service and +probes that reference the port by number are unaffected. -The `PodMonitor` ingests everything the engine exposes on `/metrics`. What the -engine emits is the user's, set through engine flags like everything else -(vLLM's `--disable-log-stats` and friends), so Modelplane doesn't decide which -metrics exist. It scrapes what's there. - -Filtering at ingestion, dropping specific metrics for cost or cardinality, is a -Prometheus `metricRelabelings` knob. It's out of scope for now, because a -selection surface reintroduces the per-deployment wiring #269 removes. The nested -`metrics` object leaves room to add it if a real need appears. - -This scrape feeds Prometheus for dashboards and alerting, so the 30s interval is -an observability default, not a routing input. The EPP scrapes engine `/metrics` -on its own fast internal loop for routing decisions, so routing latency doesn't -depend on this interval. An engine pod exposes on the order of dozens of series -(request and latency histograms, KV-cache, throughput); retention and storage -sizing are the serving stack's Prometheus configuration, not this composition. - -## Endpoint picker metrics +### Endpoint picker metrics Multi-pod Unified and prefill/decode serving front the engines with an endpoint picker (EPP) that Modelplane composes, today `llm-d-router-endpoint-picker:v0.9.0`. The EPP makes the routing decisions the engine metrics don't show: endpoint selection, prefix-cache-aware scoring, and the -prefill/decode split. It exposes a rich `llm_d_epp_*` metric set (request and -error totals, TTFT and per-output-token latency, in-flight requests, pool KV-cache -usage, and scheduler and disaggregation timings), the signal for whether that -routing is working. +prefill/decode split. It exposes a rich `llm_d_epp_*` metric set (request and error +totals, TTFT and per-output-token latency, in-flight requests, pool KV-cache usage, +and scheduler and disaggregation timings), the signal for whether that routing is +working. Because Modelplane owns the EPP Deployment and its args, collecting these is straightforward. The EPP serves `/metrics` on a port set by `--metrics-port` (default 9090), and its auth is a flag, `--metrics-endpoint-auth`. Modelplane sets `--metrics-endpoint-auth=false` (the metrics are non-sensitive routing stats -reachable only in-cluster), declares the 9090 port on the Deployment, and composes -a plain `PodMonitor` for it, the same shape as the engine's. It's composed -alongside the EPP objects in `routing.py`, so it exists wherever an EPP does. No -ClusterRole, token, or TLS to manage. +reachable only in-cluster), declares the 9090 port on the Deployment, and composes a +plain `PodMonitor` for it, the same shape as the engine's. It's composed alongside +the EPP objects in `routing.py`, so it exists wherever an EPP does. No ClusterRole, +token, or TLS to manage. + +### Serving-stack components -## The opt-out field +`compose-serving-stack` composes a `PodMonitor` for the substrate it installs: the +gateway's Envoy proxies (already scraped), cert-manager, the LeaderWorkerSet +controller, and the NVIDIA DRA driver. These are per-cluster and outlive any one +deployment, so they belong to the serving stack rather than to a replica. -`spec.template.spec.observability.metrics.enabled` (boolean, default true), copied -down to `ModelReplica.spec`, governs both PodMonitors. The field sits under an -`observability` object to match [#77](https://github.com/modelplaneai/modelplane/issues/77), -which puts `observability.traces` on `ModelService`; traces and logs can slot -beside `metrics` later. The `metrics` object itself leaves room for `interval` or -`path`; only `enabled` is defined now. `compose-model-replica` composes the engine -`PodMonitor`, and the EPP path composes the picker's, unless `enabled` is false. +### What gets scraped + +The `PodMonitor` ingests everything a source exposes on `/metrics`. What the engine +emits is the user's, set through engine flags like everything else (vLLM's +`--disable-log-stats` and friends), so Modelplane doesn't decide which metrics +exist. It scrapes what's there. + +This scrape feeds Prometheus for dashboards and alerting, so the 30s interval is an +observability default, not a routing input. The EPP scrapes engine `/metrics` on its +own fast internal loop for routing decisions, so routing latency doesn't depend on +this interval. An engine pod exposes on the order of dozens of series (request and +latency histograms, KV-cache, throughput); retention and storage sizing are the +Prometheus configuration below, not the composition of individual monitors. + +## Aggregation: a central Prometheus + +Each InferenceCluster runs a Prometheus already. This proposal adds a central +Modelplane Prometheus on the control plane. The per-cluster instances feed it, by +remote-write or by the central one federating them, so every cluster's series lands +in one place. The central instance also scrapes the control plane directly, for +Crossplane controller metrics, function latency and panics, the scheduler, and XR +`Ready`/`Synced`. Recording rules there roll up the fleet view of capacity, GPU +usage, and degraded deployments. The central Prometheus is then the single target an +operator points dashboards and an alerting stack at, for the whole deployment rather +than per cluster. ## Architecture ```mermaid flowchart LR - subgraph rep["compose-model-replica composes"] - ENG["engine pods\n(label modelplane.ai/serving)"] - EPM["engine PodMonitor"] - EPP["endpoint picker\n(multi-pod / prefill-decode)"] - XPM["EPP PodMonitor"] + subgraph icA["InferenceCluster A"] + PMA["PodMonitors\n(engine, EPP, serving stack)"] + PRA["Prometheus"] + end + subgraph icB["InferenceCluster B"] + PMB["PodMonitors"] + PRB["Prometheus"] + end + subgraph cpl["control plane"] + XP["Crossplane\n(functions, scheduler, XRs)"] + CENT["central Prometheus\n+ recording rules"] end - PROM["cluster Prometheus\n(open PodMonitor discovery)"] - EPM -->|"scrape :http /metrics"| ENG - XPM -->|"scrape :9090 /metrics"| EPP - PROM -->|discovers| EPM - PROM -->|discovers| XPM + OP["operator\ndashboards + alerting"] + PMA --> PRA + PMB --> PRB + PRA -->|remote-write| CENT + PRB -->|remote-write| CENT + XP -->|scraped by| CENT + CENT --> OP classDef new fill:#ffb74d,stroke:#e65100,stroke-width:3px,color:#000; - class EPM,XPM new + class CENT,PMA,PMB new ``` ## Alternatives considered -### Opt-in, default off +### A per-deployment opt-out field -Closer to today, because the operator still has to remember a toggle. The issue -asks for collection that is managed for them, so the default is on. Anyone who -doesn't want scraping opts out. +The earlier shape of this proposal put an `enabled` toggle on the deployment and +composed the engine and picker `PodMonitor`s unless it was false. It covers only the +data plane, and it asks an MD author to opt in or out of collection that the platform +team plumbs and consumes. Always-on collection at every layer fits the ownership +better, so the toggle is dropped. -### Scrape the engine by port number, not name +### Per-cluster Prometheus only, no central instance -The manual example matches `targetPort: 8000`. That scrapes the pd-sidecar on -prefill/decode decode pods instead of the engine, and it re-couples the -`PodMonitor` to a number that moves with the shape. Naming the port removes both -problems. - -### Authenticate the EPP metrics endpoint - -The EPP can serve `/metrics` behind controller-runtime auth (a `ClusterRole` with -`nonResourceURLs: /metrics` plus a bearer token). Since Modelplane owns the EPP -args and the endpoint carries non-sensitive routing stats reachable only -in-cluster, `--metrics-endpoint-auth=false` collects them with a plain -`PodMonitor` and nothing to manage. Auth would add a `ClusterRole`, a binding, and -a token for no gain here. +Leaving each InferenceCluster's Prometheus standalone is less to compose, but it +gives an operator no single place to query and no home for control-plane or +fleet-roll-up metrics. They'd federate the clusters by hand, which is the wiring +#269 is trying to remove. The central instance is the point. ### One cluster-wide PodMonitor from the serving stack `compose-serving-stack` could install a single `PodMonitor` selecting all serving -pods (`modelplane.ai/serving` Exists), close to the manual example. It would have -no per-deployment lifecycle: it wouldn't come and go with a deployment, and it -couldn't be opted out per deployment. Composing per replica ties collection to the -thing it observes. +pods (`modelplane.ai/serving` Exists). It would have no per-deployment lifecycle: it +wouldn't come and go with a deployment. Composing the engine monitor per replica ties +that collection to the thing it observes, while the serving-stack monitors, which are +per-cluster, are composed once with the stack. ### ServiceMonitor instead of PodMonitor A `ServiceMonitor` scrapes through a Service, so it needs a Service per scrape -target. The engine and picker pods are the targets, and a `PodMonitor` scrapes -them directly, matching the manual path. +target. The engine and picker pods are the targets, and a `PodMonitor` scrapes them +directly, matching the manual path. + +### Authenticate the EPP metrics endpoint + +The EPP can serve `/metrics` behind controller-runtime auth (a `ClusterRole` with +`nonResourceURLs: /metrics` plus a bearer token). Since Modelplane owns the EPP args +and the endpoint carries non-sensitive routing stats reachable only in-cluster, +`--metrics-endpoint-auth=false` collects them with a plain `PodMonitor` and nothing +to manage. Auth would add a `ClusterRole` and a bearer token for no gain here. ## Open questions +- **Feed mechanism.** Per-cluster remote-write into the central instance, or central + federation of each cluster. Remote-write is more timely and survives a cluster + Prometheus restart. Federation is simpler to stand up. +- **Central retention and sizing.** The central instance holds every cluster's + series, so its retention and storage need sizing that the per-cluster instances + don't. A sensible default belongs with the central instance's composition. +- **Phasing.** Data-plane and substrate collection (1 and 2) are the direct #269 + ask. Control-plane and fleet-roll-up (3 and 4) can land after, once the central + instance exists. - **Port name.** `http` (the serving port that also serves `/metrics`) versus `metrics`. Leaning `http`, since it's the one serving port. -- **Configurability.** `interval` and `path` are fixed (30s, `/metrics`) for now. - The nested `metrics` object leaves room to add them if a real need appears. ## Interaction with #264 -The [#264](https://github.com/modelplaneai/modelplane/issues/264) example -documents the manual path. Once collection is composed, that example drops its -hand-written `podmonitor.yaml` and shows the opt-out field instead. +The [#264](https://github.com/modelplaneai/modelplane/issues/264) example documents +the manual path: a hand-written `PodMonitor` plus the operator wiring to consume it. +Once collection is composed and aggregated, that example drops the hand-written +`podmonitor.yaml`. On upgrade, an existing hand-written `PodMonitor` has to be deleted, or it double-scrapes the same pods alongside the composed one. This warrants a release note. - From 7810e39ef98d5ee29f01e431ea19689e0cef3171 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Mon, 27 Jul 2026 08:56:14 -0700 Subject: [PATCH 04/13] Scope metrics to per-cluster collection plus a discoverable URL Pull the design back from a central aggregated Prometheus to what this layer should own: always-on PodMonitors inside each InferenceCluster and a Prometheus URL on the cluster status, so the platform team can scrape or federate without reaching into Modelplane internals. Leave cross-cluster aggregation and control-plane monitoring to the platform team, as the main open question. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/metrics.md | 273 +++++++++++++++++++++++----------------------- 1 file changed, 138 insertions(+), 135 deletions(-) diff --git a/design/metrics.md b/design/metrics.md index 0d61c9d66..baffc7a5b 100644 --- a/design/metrics.md +++ b/design/metrics.md @@ -4,60 +4,61 @@ **Date:** July 2026 **Author:** Dennis Ramdass -This document proposes managed metrics collection and aggregation across a -Modelplane deployment. Modelplane composes a `PodMonitor` for each source it owns on -every cluster, always on, and feeds them into one central Prometheus an operator -points dashboards and alerting at. It builds on [design.md](./design.md) and -addresses [#269](https://github.com/modelplaneai/modelplane/issues/269). +This document proposes managed metrics collection inside each `InferenceCluster`, with +a discoverable Prometheus URL on the cluster's status so the platform team can consume +it. Modelplane composes a `PodMonitor` for each source it owns on every cluster, always +on, and the cluster's own Prometheus scrapes them. Cross-cluster aggregation stays with +the platform team. It builds on [design.md](./design.md) and addresses +[#269](https://github.com/modelplaneai/modelplane/issues/269). ## What to monitor There are four things worth watching in a Modelplane deployment. -1. **Inference signal (data plane).** vLLM's `/metrics`, the EPP's `llm_d_epp_*`, - and Envoy. TTFT, tokens per second, queue depth, KV-cache occupancy, and request - and error rates per model. It answers "is my model serving well, and is it - saturated?" -2. **Substrate health.** The stack Modelplane installs on each workload cluster: is - the gateway up, are cert-manager, the LeaderWorkerSet controller, and the NVIDIA - DRA driver healthy, are GPUs allocatable. "Is the machinery on this cluster - working?" -3. **Control-plane health.** Modelplane itself. Crossplane reconcile rates and - errors, function latency and panics, the fleet scheduler placing replicas, and XR +1. **Inference signal (data plane).** vLLM's `/metrics`, the EPP's `llm_d_epp_*`, and + Envoy. TTFT, tokens per second, queue depth, KV-cache occupancy, and request and + error rates per model. It answers "is my model serving well, and is it saturated?" +2. **Substrate health.** The stack Modelplane installs on each workload cluster: is the + gateway up, are cert-manager, the LeaderWorkerSet controller, and the NVIDIA DRA + driver healthy, are GPUs allocatable. "Is the machinery on this cluster working?" +3. **Control-plane health.** Modelplane itself. Crossplane reconcile rates and errors, + function latency and panics, the fleet scheduler placing replicas, and XR `Ready`/`Synced`. "Is the thing I operate working?" -4. **Fleet roll-up.** Across every cluster and deployment: total capacity, GPU - usage, how many deployments are degraded, and cost. +4. **Fleet roll-up.** Across every cluster and deployment: total capacity, GPU usage, + how many deployments are degraded, and cost. -An MD author cares about (1). The platform team owns the rest. A per-deployment -opt-in would cover only (1). Even then, the platform team has to plumb the metrics -somewhere the author can read them. So this design makes collection always on at -every layer and aggregates it centrally. It does not ask an MD author to manage a -toggle for something the platform team consumes. +This proposal covers (1) and (2) inside each cluster and makes the data discoverable. +An MD author cares about (1), and the platform team owns the rest. A per-deployment +opt-in would cover only (1), and even then the platform team has to plumb the metrics +somewhere the author can read them. So collection is always on inside each cluster, +with no toggle for an MD author to manage. Aggregating across clusters and monitoring +the control plane stay with the platform team, and the end of this document returns to +them. ## Collection: always-on PodMonitors -On each cluster, Modelplane composes a `PodMonitor` for every source it owns, with -no opt-in or opt-out. Three existing pieces make this cheap: +On each cluster, Modelplane composes a `PodMonitor` for every source it owns, with no +opt-in or opt-out. Three existing pieces make this cheap: - **The serving label spans every shape.** `modelplane.ai/serving` is on standalone pods, LeaderWorkerSet leaders, and both prefill/decode engines (it's the label the - InferencePool selects on). One selector on it follows the shape, so the - leader/worker and prefill/decode branching needs no special casing. + InferencePool selects on). One selector on it follows the shape, so the leader/worker + and prefill/decode branching needs no special casing. - **Prometheus discovers openly.** `compose-serving-stack` runs kube-prometheus-stack - with `podMonitorSelectorNilUsesHelmValues: false` and an empty namespace selector, - so a composed `PodMonitor` is scraped with no operator action. It already scrapes - the gateway's Envoy proxies this way. + with `podMonitorSelectorNilUsesHelmValues: false` and an empty namespace selector, so + a composed `PodMonitor` is scraped with no operator action. It already scrapes the + gateway's Envoy proxies this way. - **Modelplane owns the picker.** The EPP is Modelplane's own Deployment, so its metrics port and flags are ours to set. -The sources are the engine (per deployment), the endpoint picker (wherever one -exists), and the serving-stack components (per cluster). +The sources are the engine (per deployment), the endpoint picker (wherever one exists), +and the serving-stack components (per cluster). ### Engine metrics -`compose-model-replica` composes a `PodMonitor` on the workload cluster (the same -place it composes the Service and InferencePool), selecting the replica's serving -pods by `modelplane.ai/serving`: +`compose-model-replica` composes a `PodMonitor` on the workload cluster (the same place +it composes the Service and InferencePool), selecting the replica's serving pods by +`modelplane.ai/serving`: ```yaml apiVersion: monitoring.coreos.com/v1 @@ -75,161 +76,163 @@ spec: interval: 30s ``` -It has no status of its own, so it uses default readiness (Ready once synced), like -the Service. No `mrap.yaml` change is needed: it's a provider-kubernetes `Object`, -already activated, and the CRD is installed by the Prometheus stack. +It has no status of its own, so it uses default readiness (Ready once synced), like the +Service. No `mrap.yaml` change is needed: it's a provider-kubernetes `Object`, already +activated, and the CRD is installed by the Prometheus stack. ### Naming the port The engine's serving port carries `/metrics`, so the backends give it a name (for -example `http`) in `native.py`, `llmd.py`, and the decode engine in `routing.py`. -The pd-sidecar's port stays unnamed. The `PodMonitor` then scrapes that port by -name, which matters for prefill/decode: the decode engine serves on 8001 because the -pd-sidecar takes 8000, so a `PodMonitor` matching port 8000 by number scrapes the -sidecar on decode pods, not the engine. Referencing the port by name scrapes the -engine on every pod regardless of its number. Naming is additive; the Service and -probes that reference the port by number are unaffected. +example `http`) in `native.py`, `llmd.py`, and the decode engine in `routing.py`. The +pd-sidecar's port stays unnamed. The `PodMonitor` then scrapes that port by name, which +matters for prefill/decode: the decode engine serves on 8001 because the pd-sidecar +takes 8000, so a `PodMonitor` matching port 8000 by number scrapes the sidecar on decode +pods, not the engine. Referencing the port by name scrapes the engine on every pod +regardless of its number. Naming is additive; the Service and probes that reference the +port by number are unaffected. ### Endpoint picker metrics -Multi-pod Unified and prefill/decode serving front the engines with an endpoint -picker (EPP) that Modelplane composes, today -`llm-d-router-endpoint-picker:v0.9.0`. The EPP makes the routing decisions the -engine metrics don't show: endpoint selection, prefix-cache-aware scoring, and the -prefill/decode split. It exposes a rich `llm_d_epp_*` metric set (request and error -totals, TTFT and per-output-token latency, in-flight requests, pool KV-cache usage, -and scheduler and disaggregation timings), the signal for whether that routing is -working. +Multi-pod Unified and prefill/decode serving front the engines with an endpoint picker +(EPP) that Modelplane composes, today `llm-d-router-endpoint-picker:v0.9.0`. The EPP +makes the routing decisions the engine metrics don't show: endpoint selection, +prefix-cache-aware scoring, and the prefill/decode split. It exposes a rich +`llm_d_epp_*` metric set (request and error totals, TTFT and per-output-token latency, +in-flight requests, pool KV-cache usage, and scheduler and disaggregation timings), the +signal for whether that routing is working. Because Modelplane owns the EPP Deployment and its args, collecting these is -straightforward. The EPP serves `/metrics` on a port set by `--metrics-port` -(default 9090), and its auth is a flag, `--metrics-endpoint-auth`. Modelplane sets -`--metrics-endpoint-auth=false` (the metrics are non-sensitive routing stats -reachable only in-cluster), declares the 9090 port on the Deployment, and composes a -plain `PodMonitor` for it, the same shape as the engine's. It's composed alongside -the EPP objects in `routing.py`, so it exists wherever an EPP does. No ClusterRole, -token, or TLS to manage. +straightforward. The EPP serves `/metrics` on a port set by `--metrics-port` (default +9090), and its auth is a flag, `--metrics-endpoint-auth`. Modelplane sets +`--metrics-endpoint-auth=false` (the metrics are non-sensitive routing stats reachable +only in-cluster), declares the 9090 port on the Deployment, and composes a plain +`PodMonitor` for it, the same shape as the engine's. It's composed alongside the EPP +objects in `routing.py`, so it exists wherever an EPP does. No ClusterRole, token, or +TLS to manage. ### Serving-stack components `compose-serving-stack` composes a `PodMonitor` for the substrate it installs: the -gateway's Envoy proxies (already scraped), cert-manager, the LeaderWorkerSet -controller, and the NVIDIA DRA driver. These are per-cluster and outlive any one -deployment, so they belong to the serving stack rather than to a replica. +gateway's Envoy proxies (already scraped), cert-manager, the LeaderWorkerSet controller, +and the NVIDIA DRA driver. These are per-cluster and outlive any one deployment, so they +belong to the serving stack rather than to a replica. ### What gets scraped The `PodMonitor` ingests everything a source exposes on `/metrics`. What the engine emits is the user's, set through engine flags like everything else (vLLM's -`--disable-log-stats` and friends), so Modelplane doesn't decide which metrics -exist. It scrapes what's there. +`--disable-log-stats` and friends), so Modelplane doesn't decide which metrics exist. It +scrapes what's there. This scrape feeds Prometheus for dashboards and alerting, so the 30s interval is an -observability default, not a routing input. The EPP scrapes engine `/metrics` on its -own fast internal loop for routing decisions, so routing latency doesn't depend on -this interval. An engine pod exposes on the order of dozens of series (request and -latency histograms, KV-cache, throughput); retention and storage sizing are the -Prometheus configuration below, not the composition of individual monitors. - -## Aggregation: a central Prometheus - -Each InferenceCluster runs a Prometheus already. This proposal adds a central -Modelplane Prometheus on the control plane. The per-cluster instances feed it, by -remote-write or by the central one federating them, so every cluster's series lands -in one place. The central instance also scrapes the control plane directly, for -Crossplane controller metrics, function latency and panics, the scheduler, and XR -`Ready`/`Synced`. Recording rules there roll up the fleet view of capacity, GPU -usage, and degraded deployments. The central Prometheus is then the single target an -operator points dashboards and an alerting stack at, for the whole deployment rather -than per cluster. +observability default, not a routing input. The EPP scrapes engine `/metrics` on its own +fast internal loop for routing decisions, so routing latency doesn't depend on this +interval. An engine pod exposes on the order of dozens of series (request and latency +histograms, KV-cache, throughput); retention and storage sizing are the cluster +Prometheus's configuration, not the composition of individual monitors. + +## Making the per-cluster Prometheus discoverable + +Each `InferenceCluster` already runs a Prometheus, installed by `compose-serving-stack` +(kube-prometheus-stack). With the `PodMonitor`s above composed, that Prometheus holds +all of the cluster's Modelplane sources, from the engines and EPPs to the serving-stack +components. Modelplane exposes its in-cluster URL on the `InferenceCluster` status: + +```yaml +status: + metrics: + prometheusURL: http://prometheus-operated.modelplane-system.svc:9090 +``` + +The platform team reads that URL to scrape or federate the cluster, without reaching +into Modelplane internals to find where the Prometheus runs or what it collects. This is +the line the proposal reaches: collection wired up and discoverable, so monitoring a +cluster's Modelplane workloads needs no knowledge of how they are composed. + +## Aggregation, and where this stops + +This proposal stops at per-cluster collection and the discoverable URL. Beyond that, the +platform team wires up cross-cluster aggregation and control-plane monitoring itself. +Whether Modelplane should later package a managed central Prometheus for the whole +deployment is out of scope here, and the main open question below. ## Architecture ```mermaid flowchart LR subgraph icA["InferenceCluster A"] - PMA["PodMonitors\n(engine, EPP, serving stack)"] - PRA["Prometheus"] + PMA["PodMonitors\n(engines, EPPs, serving stack)"] + PRA["cluster Prometheus"] + STA["status.metrics.prometheusURL"] end subgraph icB["InferenceCluster B"] - PMB["PodMonitors"] - PRB["Prometheus"] - end - subgraph cpl["control plane"] - XP["Crossplane\n(functions, scheduler, XRs)"] - CENT["central Prometheus\n+ recording rules"] + PRB["cluster Prometheus"] end - OP["operator\ndashboards + alerting"] + PT["platform team\nscrape / aggregate / alert"] PMA --> PRA - PMB --> PRB - PRA -->|remote-write| CENT - PRB -->|remote-write| CENT - XP -->|scraped by| CENT - CENT --> OP + PRA --> STA + STA -.discovered by.-> PT + PRB -.discovered by.-> PT classDef new fill:#ffb74d,stroke:#e65100,stroke-width:3px,color:#000; - class CENT,PMA,PMB new + class PMA,PRA,STA new ``` ## Alternatives considered ### A per-deployment opt-out field -The earlier shape of this proposal put an `enabled` toggle on the deployment and -composed the engine and picker `PodMonitor`s unless it was false. It covers only the -data plane, and it asks an MD author to opt in or out of collection that the platform -team plumbs and consumes. Always-on collection at every layer fits the ownership -better, so the toggle is dropped. +An earlier shape of this proposal put an `enabled` toggle on the deployment and composed +the engine and picker `PodMonitor`s unless it was false. It covers only the data plane, +and it asks an MD author to opt in or out of collection that the platform team plumbs and +consumes. Always-on collection at every layer fits the ownership better, so the toggle is +dropped. -### Per-cluster Prometheus only, no central instance +### A managed central Prometheus in this proposal -Leaving each InferenceCluster's Prometheus standalone is less to compose, but it -gives an operator no single place to query and no home for control-plane or -fleet-roll-up metrics. They'd federate the clusters by hand, which is the wiring -#269 is trying to remove. The central instance is the point. +Composing one central Prometheus for the whole deployment, federating every cluster and +scraping the control plane, would give an operator a single endpoint. It is more to own +than wiring up what each cluster already runs, so this proposal leaves aggregation to the +platform team and stops at the discoverable URL. ### One cluster-wide PodMonitor from the serving stack -`compose-serving-stack` could install a single `PodMonitor` selecting all serving -pods (`modelplane.ai/serving` Exists). It would have no per-deployment lifecycle: it -wouldn't come and go with a deployment. Composing the engine monitor per replica ties -that collection to the thing it observes, while the serving-stack monitors, which are +`compose-serving-stack` could install a single `PodMonitor` selecting all serving pods +(`modelplane.ai/serving` Exists). It would have no per-deployment lifecycle: it wouldn't +come and go with a deployment. Composing the engine monitor per replica ties that +collection to the thing it observes, while the serving-stack monitors, which are per-cluster, are composed once with the stack. ### ServiceMonitor instead of PodMonitor -A `ServiceMonitor` scrapes through a Service, so it needs a Service per scrape -target. The engine and picker pods are the targets, and a `PodMonitor` scrapes them -directly, matching the manual path. +A `ServiceMonitor` scrapes through a Service, so it needs a Service per scrape target. +The engine and picker pods are the targets, and a `PodMonitor` scrapes them directly, +matching the manual path. ### Authenticate the EPP metrics endpoint The EPP can serve `/metrics` behind controller-runtime auth (a `ClusterRole` with -`nonResourceURLs: /metrics` plus a bearer token). Since Modelplane owns the EPP args -and the endpoint carries non-sensitive routing stats reachable only in-cluster, -`--metrics-endpoint-auth=false` collects them with a plain `PodMonitor` and nothing -to manage. Auth would add a `ClusterRole` and a bearer token for no gain here. +`nonResourceURLs: /metrics` plus a bearer token). Since Modelplane owns the EPP args and +the endpoint carries non-sensitive routing stats reachable only in-cluster, +`--metrics-endpoint-auth=false` collects them with a plain `PodMonitor` and nothing to +manage. Auth would add a `ClusterRole` and a bearer token for no gain here. ## Open questions -- **Feed mechanism.** Per-cluster remote-write into the central instance, or central - federation of each cluster. Remote-write is more timely and survives a cluster - Prometheus restart. Federation is simpler to stand up. -- **Central retention and sizing.** The central instance holds every cluster's - series, so its retention and storage need sizing that the per-cluster instances - don't. A sensible default belongs with the central instance's composition. -- **Phasing.** Data-plane and substrate collection (1 and 2) are the direct #269 - ask. Control-plane and fleet-roll-up (3 and 4) can land after, once the central - instance exists. -- **Port name.** `http` (the serving port that also serves `/metrics`) versus - `metrics`. Leaning `http`, since it's the one serving port. +- **Central aggregation.** Whether Modelplane should package a managed central Prometheus + for the whole deployment, or leave cross-cluster aggregation and control-plane + monitoring to the platform team. This proposal leaves it out, and that is the main call + to make. +- **Discoverable field shape.** The `status.metrics.prometheusURL` name, and whether it + should also carry a service reference or scrape credentials the platform team needs. +- **Port name.** `http` (the serving port that also serves `/metrics`) versus `metrics`. + Leaning `http`, since it's the one serving port. ## Interaction with #264 -The [#264](https://github.com/modelplaneai/modelplane/issues/264) example documents -the manual path: a hand-written `PodMonitor` plus the operator wiring to consume it. -Once collection is composed and aggregated, that example drops the hand-written -`podmonitor.yaml`. +The [#264](https://github.com/modelplaneai/modelplane/issues/264) example documents the +manual path: a hand-written `PodMonitor` plus the operator wiring to consume it. Once +collection is composed and the cluster's Prometheus URL is exposed, that example drops +the hand-written `podmonitor.yaml`. -On upgrade, an existing hand-written `PodMonitor` has to be deleted, or it -double-scrapes the same pods alongside the composed one. This warrants a release -note. +On upgrade, an existing hand-written `PodMonitor` has to be deleted, or it double-scrapes +the same pods alongside the composed one. This warrants a release note. From db3c23b5a6b89c476c73bba87dac38fce8b9d44f Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 28 Jul 2026 06:26:52 -0700 Subject: [PATCH 05/13] Switch metrics to cluster-wide PodMonitors and state the proposal Agree the per-replica composition earns nothing once collection is always-on, so compose one cluster-wide PodMonitor per source in the serving stack instead. State the proposal and what approving it covers up front, answer how the per-cluster Prometheus is reached from outside (the platform team's existing cross-cluster path; Modelplane only publishes the URL), and drop the opt-in mention that was reacting to the earlier draft. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/metrics.md | 171 ++++++++++++++++++++++------------------------ 1 file changed, 83 insertions(+), 88 deletions(-) diff --git a/design/metrics.md b/design/metrics.md index baffc7a5b..bde872110 100644 --- a/design/metrics.md +++ b/design/metrics.md @@ -6,11 +6,20 @@ This document proposes managed metrics collection inside each `InferenceCluster`, with a discoverable Prometheus URL on the cluster's status so the platform team can consume -it. Modelplane composes a `PodMonitor` for each source it owns on every cluster, always -on, and the cluster's own Prometheus scrapes them. Cross-cluster aggregation stays with -the platform team. It builds on [design.md](./design.md) and addresses +it. It builds on [design.md](./design.md) and addresses [#269](https://github.com/modelplaneai/modelplane/issues/269). +Concretely, this proposes: + +1. cluster-wide `PodMonitor`s, composed once with the serving stack, that scrape every + Modelplane source on the cluster (engines, endpoint pickers, substrate), always on + with no per-deployment toggle +2. a `status.metrics.prometheusURL` on the `InferenceCluster`, so the platform team can + find the cluster's Prometheus without reaching into Modelplane internals + +Cross-cluster aggregation and control-plane monitoring stay with the platform team and +are out of scope. Approving this means agreeing to that scope. + ## What to monitor There are four things worth watching in a Modelplane deployment. @@ -27,18 +36,17 @@ There are four things worth watching in a Modelplane deployment. 4. **Fleet roll-up.** Across every cluster and deployment: total capacity, GPU usage, how many deployments are degraded, and cost. -This proposal covers (1) and (2) inside each cluster and makes the data discoverable. -An MD author cares about (1), and the platform team owns the rest. A per-deployment -opt-in would cover only (1), and even then the platform team has to plumb the metrics -somewhere the author can read them. So collection is always on inside each cluster, -with no toggle for an MD author to manage. Aggregating across clusters and monitoring -the control plane stay with the platform team, and the end of this document returns to -them. +This proposal covers (1) and (2) inside each cluster and makes the data discoverable. An +MD author cares about (1), and the platform team owns the rest. Aggregating across +clusters and monitoring the control plane stay with the platform team, and the end of +this document returns to them. -## Collection: always-on PodMonitors +## Collection: cluster-wide PodMonitors -On each cluster, Modelplane composes a `PodMonitor` for every source it owns, with no -opt-in or opt-out. Three existing pieces make this cheap: +The serving stack composes a small fixed set of `PodMonitor`s per cluster, always on, +with no opt-in or opt-out. Because there's no toggle and nothing per-deployment to track, +one cluster-wide selector per source is simpler than a monitor composed per replica. +Three existing pieces make it cheap: - **The serving label spans every shape.** `modelplane.ai/serving` is on standalone pods, LeaderWorkerSet leaders, and both prefill/decode engines (it's the label the @@ -48,81 +56,73 @@ opt-in or opt-out. Three existing pieces make this cheap: with `podMonitorSelectorNilUsesHelmValues: false` and an empty namespace selector, so a composed `PodMonitor` is scraped with no operator action. It already scrapes the gateway's Envoy proxies this way. -- **Modelplane owns the picker.** The EPP is Modelplane's own Deployment, so its - metrics port and flags are ours to set. - -The sources are the engine (per deployment), the endpoint picker (wherever one exists), -and the serving-stack components (per cluster). +- **Modelplane owns the picker.** The EPP is Modelplane's own Deployment, so its metrics + port and flags are ours to set. ### Engine metrics -`compose-model-replica` composes a `PodMonitor` on the workload cluster (the same place -it composes the Service and InferencePool), selecting the replica's serving pods by -`modelplane.ai/serving`: +`compose-serving-stack` composes one `PodMonitor` that selects every serving pod on the +cluster by `modelplane.ai/serving`, so it scrapes every engine of every deployment: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: - name: -metrics - namespace: default + name: modelplane-engines + namespace: modelplane-system spec: selector: - matchLabels: - modelplane.ai/serving: + matchExpressions: + - {key: modelplane.ai/serving, operator: Exists} podMetricsEndpoints: - port: http path: /metrics interval: 30s ``` -It has no status of its own, so it uses default readiness (Ready once synced), like the -Service. No `mrap.yaml` change is needed: it's a provider-kubernetes `Object`, already -activated, and the CRD is installed by the Prometheus stack. +One monitor covers every replica of every deployment. Collection is a cluster property, +composed once with the stack, not something that comes and goes with each +`ModelDeployment`. ### Naming the port -The engine's serving port carries `/metrics`, so the backends give it a name (for -example `http`) in `native.py`, `llmd.py`, and the decode engine in `routing.py`. The -pd-sidecar's port stays unnamed. The `PodMonitor` then scrapes that port by name, which -matters for prefill/decode: the decode engine serves on 8001 because the pd-sidecar -takes 8000, so a `PodMonitor` matching port 8000 by number scrapes the sidecar on decode -pods, not the engine. Referencing the port by name scrapes the engine on every pod -regardless of its number. Naming is additive; the Service and probes that reference the -port by number are unaffected. +The engine's serving port carries `/metrics`, so the backends give it a name such as +`http` in `native.py`, `llmd.py`, and the decode engine in `routing.py`. The pd-sidecar's +port stays unnamed. The `PodMonitor` scrapes that port by name, which matters for +prefill/decode: the decode engine serves on 8001 because the pd-sidecar takes 8000, so a +`PodMonitor` matching port 8000 by number scrapes the sidecar on decode pods, not the +engine. Referencing the port by name scrapes the engine on every pod regardless of its +number. Naming is additive; the Service and probes that reference the port by number are +unaffected. ### Endpoint picker metrics Multi-pod Unified and prefill/decode serving front the engines with an endpoint picker -(EPP) that Modelplane composes, today `llm-d-router-endpoint-picker:v0.9.0`. The EPP -makes the routing decisions the engine metrics don't show: endpoint selection, -prefix-cache-aware scoring, and the prefill/decode split. It exposes a rich -`llm_d_epp_*` metric set (request and error totals, TTFT and per-output-token latency, -in-flight requests, pool KV-cache usage, and scheduler and disaggregation timings), the -signal for whether that routing is working. - -Because Modelplane owns the EPP Deployment and its args, collecting these is -straightforward. The EPP serves `/metrics` on a port set by `--metrics-port` (default -9090), and its auth is a flag, `--metrics-endpoint-auth`. Modelplane sets +(EPP) that Modelplane composes, today `llm-d-router-endpoint-picker:v0.9.0`. The EPP makes +the routing decisions the engine metrics don't show: endpoint selection, +prefix-cache-aware scoring, and the prefill/decode split. It exposes a rich `llm_d_epp_*` +metric set (request and error totals, TTFT and per-output-token latency, in-flight +requests, pool KV-cache usage, and scheduler and disaggregation timings), the signal for +whether that routing is working. + +A second cluster-wide `PodMonitor` selects every EPP by its Modelplane label. Because +Modelplane owns the EPP Deployment and its args, this is cheap: it sets `--metrics-endpoint-auth=false` (the metrics are non-sensitive routing stats reachable -only in-cluster), declares the 9090 port on the Deployment, and composes a plain -`PodMonitor` for it, the same shape as the engine's. It's composed alongside the EPP -objects in `routing.py`, so it exists wherever an EPP does. No ClusterRole, token, or -TLS to manage. +only in-cluster) and declares the port (default 9090) on the Deployment. No ClusterRole, +token, or TLS to manage. -### Serving-stack components +### Substrate -`compose-serving-stack` composes a `PodMonitor` for the substrate it installs: the +`compose-serving-stack` also composes a `PodMonitor` for the substrate it installs: the gateway's Envoy proxies (already scraped), cert-manager, the LeaderWorkerSet controller, -and the NVIDIA DRA driver. These are per-cluster and outlive any one deployment, so they -belong to the serving stack rather than to a replica. +and the NVIDIA DRA driver. Same shape, same place, so all three sources are composed +together with the stack. ### What gets scraped -The `PodMonitor` ingests everything a source exposes on `/metrics`. What the engine -emits is the user's, set through engine flags like everything else (vLLM's -`--disable-log-stats` and friends), so Modelplane doesn't decide which metrics exist. It -scrapes what's there. +A `PodMonitor` ingests everything a source exposes on `/metrics`. What the engine emits is +the user's, set through engine flags like everything else (vLLM's `--disable-log-stats` +and friends), so Modelplane doesn't decide which metrics exist. It scrapes what's there. This scrape feeds Prometheus for dashboards and alerting, so the 30s interval is an observability default, not a routing input. The EPP scrapes engine `/metrics` on its own @@ -134,9 +134,9 @@ Prometheus's configuration, not the composition of individual monitors. ## Making the per-cluster Prometheus discoverable Each `InferenceCluster` already runs a Prometheus, installed by `compose-serving-stack` -(kube-prometheus-stack). With the `PodMonitor`s above composed, that Prometheus holds -all of the cluster's Modelplane sources, from the engines and EPPs to the serving-stack -components. Modelplane exposes its in-cluster URL on the `InferenceCluster` status: +(kube-prometheus-stack). With the `PodMonitor`s above composed, that Prometheus holds all +of the cluster's Modelplane sources, from the engines and EPPs to the substrate. +Modelplane exposes its in-cluster URL on the `InferenceCluster` status: ```yaml status: @@ -144,10 +144,10 @@ status: prometheusURL: http://prometheus-operated.modelplane-system.svc:9090 ``` -The platform team reads that URL to scrape or federate the cluster, without reaching -into Modelplane internals to find where the Prometheus runs or what it collects. This is -the line the proposal reaches: collection wired up and discoverable, so monitoring a -cluster's Modelplane workloads needs no knowledge of how they are composed. +Modelplane publishes where the Prometheus is, and does not expose it outside the cluster. +Reaching it across clusters stays the platform team's job, through whatever cross-cluster +path they already run, the same way they reach any in-cluster service. Publishing the URL +saves them from digging into Modelplane internals to locate the cluster's Prometheus. ## Aggregation, and where this stops @@ -161,7 +161,7 @@ deployment is out of scope here, and the main open question below. ```mermaid flowchart LR subgraph icA["InferenceCluster A"] - PMA["PodMonitors\n(engines, EPPs, serving stack)"] + PMA["cluster-wide PodMonitors\n(engines, EPPs, substrate)"] PRA["cluster Prometheus"] STA["status.metrics.prometheusURL"] end @@ -179,13 +179,18 @@ flowchart LR ## Alternatives considered +### A PodMonitor per replica + +`compose-model-replica` could compose a `PodMonitor` per replica, so collection comes and +goes with the deployment. With no opt-out and a cluster-wide Prometheus, that per-deployment +lifecycle buys nothing over one cluster-wide monitor, and it composes N monitors where one +does the same job. Collection is a cluster property, so it's composed once with the stack. + ### A per-deployment opt-out field -An earlier shape of this proposal put an `enabled` toggle on the deployment and composed -the engine and picker `PodMonitor`s unless it was false. It covers only the data plane, -and it asks an MD author to opt in or out of collection that the platform team plumbs and -consumes. Always-on collection at every layer fits the ownership better, so the toggle is -dropped. +An earlier shape put an `enabled` toggle on the deployment. It covers only the data plane +and asks an MD author to opt in or out of collection that the platform team plumbs and +consumes. Always-on collection fits the ownership better, so the toggle is dropped. ### A managed central Prometheus in this proposal @@ -194,19 +199,11 @@ scraping the control plane, would give an operator a single endpoint. It is more than wiring up what each cluster already runs, so this proposal leaves aggregation to the platform team and stops at the discoverable URL. -### One cluster-wide PodMonitor from the serving stack - -`compose-serving-stack` could install a single `PodMonitor` selecting all serving pods -(`modelplane.ai/serving` Exists). It would have no per-deployment lifecycle: it wouldn't -come and go with a deployment. Composing the engine monitor per replica ties that -collection to the thing it observes, while the serving-stack monitors, which are -per-cluster, are composed once with the stack. - ### ServiceMonitor instead of PodMonitor -A `ServiceMonitor` scrapes through a Service, so it needs a Service per scrape target. -The engine and picker pods are the targets, and a `PodMonitor` scrapes them directly, -matching the manual path. +A `ServiceMonitor` scrapes through a Service, so it needs a Service per scrape target. The +engine and picker pods are the targets, and a `PodMonitor` scrapes them directly, matching +the manual path. ### Authenticate the EPP metrics endpoint @@ -220,10 +217,8 @@ manage. Auth would add a `ClusterRole` and a bearer token for no gain here. - **Central aggregation.** Whether Modelplane should package a managed central Prometheus for the whole deployment, or leave cross-cluster aggregation and control-plane - monitoring to the platform team. This proposal leaves it out, and that is the main call - to make. -- **Discoverable field shape.** The `status.metrics.prometheusURL` name, and whether it - should also carry a service reference or scrape credentials the platform team needs. + monitoring to the platform team. This proposal leaves it out; that is the main call to + make. - **Port name.** `http` (the serving port that also serves `/metrics`) versus `metrics`. Leaning `http`, since it's the one serving port. @@ -231,8 +226,8 @@ manage. Auth would add a `ClusterRole` and a bearer token for no gain here. The [#264](https://github.com/modelplaneai/modelplane/issues/264) example documents the manual path: a hand-written `PodMonitor` plus the operator wiring to consume it. Once -collection is composed and the cluster's Prometheus URL is exposed, that example drops -the hand-written `podmonitor.yaml`. +collection is composed and the cluster's Prometheus URL is exposed, that example drops the +hand-written `podmonitor.yaml`. On upgrade, an existing hand-written `PodMonitor` has to be deleted, or it double-scrapes the same pods alongside the composed one. This warrants a release note. From 57aa27a99fc14c8481fc96feb873ff94cc3c420a Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Mon, 3 Aug 2026 09:12:46 -0700 Subject: [PATCH 06/13] Aggregate metrics up to a Modelplane-level view Reverse the earlier leave-aggregation-to-the-platform-team scope: aggregate every cluster's metrics up to one Modelplane store at the control plane, with recording rules rebranding every series under modelplane_* and a normalized label set. Weigh two collection mechanisms, the incumbent Prometheus stack and an OpenTelemetry collector, and lean toward the collector. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/metrics.md | 280 ++++++++++++++++++++-------------------------- 1 file changed, 124 insertions(+), 156 deletions(-) diff --git a/design/metrics.md b/design/metrics.md index bde872110..adefeed50 100644 --- a/design/metrics.md +++ b/design/metrics.md @@ -1,24 +1,27 @@ # Metrics collection **Status:** Draft -**Date:** July 2026 +**Date:** August 2026 **Author:** Dennis Ramdass -This document proposes managed metrics collection inside each `InferenceCluster`, with -a discoverable Prometheus URL on the cluster's status so the platform team can consume -it. It builds on [design.md](./design.md) and addresses +This document proposes collecting metrics on every cluster and aggregating them up to +one Modelplane-level view at the control plane, with every series rebranded under a +`modelplane_*` namespace. It builds on [design.md](./design.md) and addresses [#269](https://github.com/modelplaneai/modelplane/issues/269). Concretely, this proposes: -1. cluster-wide `PodMonitor`s, composed once with the serving stack, that scrape every - Modelplane source on the cluster (engines, endpoint pickers, substrate), always on - with no per-deployment toggle -2. a `status.metrics.prometheusURL` on the `InferenceCluster`, so the platform team can - find the cluster's Prometheus without reaching into Modelplane internals +1. per-cluster collection of every Modelplane source (engines, endpoint pickers, + substrate), always on, with no per-deployment toggle +2. aggregation up to a single Modelplane view at the control plane, so one query covers + the whole deployment +3. recording rules that rebrand every series under `modelplane_*` with normalized + labels, such as `modelplane_tokens_total{engine="vllm", cluster="...", + deployment="..."}` -Cross-cluster aggregation and control-plane monitoring stay with the platform team and -are out of scope. Approving this means agreeing to that scope. +Approving this means agreeing that central aggregation is Modelplane's job, not the +platform team's, and to the `modelplane_*` naming. The collection mechanism, a +Prometheus stack or an OpenTelemetry collector, is the main open choice, below. ## What to monitor @@ -36,189 +39,154 @@ There are four things worth watching in a Modelplane deployment. 4. **Fleet roll-up.** Across every cluster and deployment: total capacity, GPU usage, how many deployments are degraded, and cost. -This proposal covers (1) and (2) inside each cluster and makes the data discoverable. An -MD author cares about (1), and the platform team owns the rest. Aggregating across -clusters and monitoring the control plane stay with the platform team, and the end of -this document returns to them. +All four are in the central view. The data plane and substrate are collected on each +cluster and aggregated up; the control plane is scraped at the center; the fleet roll-up +is a set of recording rules over the aggregate. -## Collection: cluster-wide PodMonitors +## Collection: per cluster, always on -The serving stack composes a small fixed set of `PodMonitor`s per cluster, always on, -with no opt-in or opt-out. Because there's no toggle and nothing per-deployment to track, -one cluster-wide selector per source is simpler than a monitor composed per replica. -Three existing pieces make it cheap: +On each cluster, Modelplane collects from every source it owns, with no opt-in or +opt-out. The targets are the same whichever mechanism scrapes them: the engine pods, the +endpoint picker, and the substrate. Three existing pieces make it cheap. - **The serving label spans every shape.** `modelplane.ai/serving` is on standalone pods, LeaderWorkerSet leaders, and both prefill/decode engines (it's the label the InferencePool selects on). One selector on it follows the shape, so the leader/worker and prefill/decode branching needs no special casing. -- **Prometheus discovers openly.** `compose-serving-stack` runs kube-prometheus-stack - with `podMonitorSelectorNilUsesHelmValues: false` and an empty namespace selector, so - a composed `PodMonitor` is scraped with no operator action. It already scrapes the - gateway's Envoy proxies this way. +- **The stack already scrapes.** `compose-serving-stack` runs a metrics stack on every + workload cluster, so adding a scrape target is composition, not new infrastructure. It + already scrapes the gateway's Envoy proxies. - **Modelplane owns the picker.** The EPP is Modelplane's own Deployment, so its metrics port and flags are ours to set. -### Engine metrics - -`compose-serving-stack` composes one `PodMonitor` that selects every serving pod on the -cluster by `modelplane.ai/serving`, so it scrapes every engine of every deployment: - -```yaml -apiVersion: monitoring.coreos.com/v1 -kind: PodMonitor -metadata: - name: modelplane-engines - namespace: modelplane-system -spec: - selector: - matchExpressions: - - {key: modelplane.ai/serving, operator: Exists} - podMetricsEndpoints: - - port: http - path: /metrics - interval: 30s -``` - -One monitor covers every replica of every deployment. Collection is a cluster property, -composed once with the stack, not something that comes and goes with each -`ModelDeployment`. - -### Naming the port - -The engine's serving port carries `/metrics`, so the backends give it a name such as -`http` in `native.py`, `llmd.py`, and the decode engine in `routing.py`. The pd-sidecar's -port stays unnamed. The `PodMonitor` scrapes that port by name, which matters for -prefill/decode: the decode engine serves on 8001 because the pd-sidecar takes 8000, so a -`PodMonitor` matching port 8000 by number scrapes the sidecar on decode pods, not the -engine. Referencing the port by name scrapes the engine on every pod regardless of its -number. Naming is additive; the Service and probes that reference the port by number are -unaffected. - -### Endpoint picker metrics - -Multi-pod Unified and prefill/decode serving front the engines with an endpoint picker -(EPP) that Modelplane composes, today `llm-d-router-endpoint-picker:v0.9.0`. The EPP makes -the routing decisions the engine metrics don't show: endpoint selection, -prefix-cache-aware scoring, and the prefill/decode split. It exposes a rich `llm_d_epp_*` -metric set (request and error totals, TTFT and per-output-token latency, in-flight -requests, pool KV-cache usage, and scheduler and disaggregation timings), the signal for -whether that routing is working. - -A second cluster-wide `PodMonitor` selects every EPP by its Modelplane label. Because -Modelplane owns the EPP Deployment and its args, this is cheap: it sets -`--metrics-endpoint-auth=false` (the metrics are non-sensitive routing stats reachable -only in-cluster) and declares the port (default 9090) on the Deployment. No ClusterRole, -token, or TLS to manage. - -### Substrate - -`compose-serving-stack` also composes a `PodMonitor` for the substrate it installs: the -gateway's Envoy proxies (already scraped), cert-manager, the LeaderWorkerSet controller, -and the NVIDIA DRA driver. Same shape, same place, so all three sources are composed -together with the stack. - -### What gets scraped - -A `PodMonitor` ingests everything a source exposes on `/metrics`. What the engine emits is -the user's, set through engine flags like everything else (vLLM's `--disable-log-stats` -and friends), so Modelplane doesn't decide which metrics exist. It scrapes what's there. - -This scrape feeds Prometheus for dashboards and alerting, so the 30s interval is an -observability default, not a routing input. The EPP scrapes engine `/metrics` on its own -fast internal loop for routing decisions, so routing latency doesn't depend on this -interval. An engine pod exposes on the order of dozens of series (request and latency -histograms, KV-cache, throughput); retention and storage sizing are the cluster -Prometheus's configuration, not the composition of individual monitors. - -## Making the per-cluster Prometheus discoverable - -Each `InferenceCluster` already runs a Prometheus, installed by `compose-serving-stack` -(kube-prometheus-stack). With the `PodMonitor`s above composed, that Prometheus holds all -of the cluster's Modelplane sources, from the engines and EPPs to the substrate. -Modelplane exposes its in-cluster URL on the `InferenceCluster` status: - -```yaml -status: - metrics: - prometheusURL: http://prometheus-operated.modelplane-system.svc:9090 -``` - -Modelplane publishes where the Prometheus is, and does not expose it outside the cluster. -Reaching it across clusters stays the platform team's job, through whatever cross-cluster -path they already run, the same way they reach any in-cluster service. Publishing the URL -saves them from digging into Modelplane internals to locate the cluster's Prometheus. - -## Aggregation, and where this stops - -This proposal stops at per-cluster collection and the discoverable URL. Beyond that, the -platform team wires up cross-cluster aggregation and control-plane monitoring itself. -Whether Modelplane should later package a managed central Prometheus for the whole -deployment is out of scope here, and the main open question below. +### The scrape targets + +A cluster-wide selector on `modelplane.ai/serving` covers every engine of every +deployment, so collection is a cluster property rather than something composed per +replica. A second selector covers the endpoint pickers by their Modelplane label, and a +third covers the substrate `compose-serving-stack` installs (the gateway's Envoy proxies, +cert-manager, the LeaderWorkerSet controller, the NVIDIA DRA driver). + +The engine's serving port carries `/metrics`, so the backends name it (say `http`) in +`native.py`, `llmd.py`, and the decode engine in `routing.py`. The pd-sidecar's port +stays unnamed. Scraping the port by name matters for prefill/decode. The decode engine serves on 8001 +because the pd-sidecar takes 8000, so matching port 8000 by number would scrape the +sidecar. Referencing the port by name scrapes the engine on every pod regardless of its +number. + +What a source exposes on `/metrics` is the user's, set through engine flags like anything +else (vLLM's `--disable-log-stats` and friends). Modelplane scrapes what's there and +doesn't decide which metrics exist. + +## Aggregation: one Modelplane view + +Per-cluster collection is only half the ask. Each cluster's series aggregate up to a +single Modelplane store at the control plane, which also scrapes the control plane's own +metrics (Crossplane, the functions, the scheduler). One query then covers the whole +deployment, rather than a per-cluster island an operator has to stitch together by hand. + +Recording rules over the aggregate produce the `modelplane_*` surface. A raw +`vllm:num_requests_running` from one engine and an `llm_d_epp_*` from a picker become +`modelplane_*` series with a consistent label set (`engine`, `cluster`, `deployment`, +`model`), so a dashboard queries Modelplane's own vocabulary and doesn't track each +engine's native metric names. The fleet roll-up (capacity, GPU usage, degraded +deployments) is more recording rules over the same aggregate. + +## Mechanism: a Prometheus stack or an OpenTelemetry collector + +The targets and the `modelplane_*` surface are the same either way. The open choice is +what collects and forwards them. + +- **Prometheus stack.** Each cluster runs the kube-prometheus-stack `compose-serving-stack` + already installs, with composed `PodMonitor`s for the targets above. Each cluster + remote-writes to a central Prometheus (or Thanos, Mimir, or Cortex) at the control + plane, and recording rules there produce the `modelplane_*` series. It's the incumbent, + and recording rules and PromQL are standard. +- **OpenTelemetry collector.** A collector per cluster scrapes the same targets (the + Prometheus receiver), rebrands them in the pipeline (the transform processor), and + remote-writes up, with a central collector or store aggregating. It runs no full + Prometheus per cluster, does the `modelplane_*` rename in-pipeline rather than through + recording rules, and is something Upbound already runs elsewhere. + +The collector is the lighter and more familiar path, and worth taking unless the +per-cluster Prometheus earns a place for something aggregation doesn't need. This is the +call to settle. + +Whichever collects, the central layer reaches a cluster either because the cluster pushes +(remote-write) or because the center pulls. On the pull path Modelplane publishes each +cluster's endpoint on the `InferenceCluster` status; on the push path no such endpoint is +needed, and nothing is exposed outside the cluster. ## Architecture ```mermaid flowchart LR subgraph icA["InferenceCluster A"] - PMA["cluster-wide PodMonitors\n(engines, EPPs, substrate)"] - PRA["cluster Prometheus"] - STA["status.metrics.prometheusURL"] + SA["engines / EPPs / substrate"] + CA["collector\n(Prometheus or OTel)"] end subgraph icB["InferenceCluster B"] - PRB["cluster Prometheus"] + CB["collector"] end - PT["platform team\nscrape / aggregate / alert"] - PMA --> PRA - PRA --> STA - STA -.discovered by.-> PT - PRB -.discovered by.-> PT + subgraph cp["control plane"] + XP["Crossplane\n(functions, scheduler, XRs)"] + CENT["Modelplane store\n+ modelplane_* recording rules"] + end + OP["operator\ndashboards + alerting"] + SA --> CA + CA -->|remote-write| CENT + CB -->|remote-write| CENT + XP -->|scraped by| CENT + CENT --> OP classDef new fill:#ffb74d,stroke:#e65100,stroke-width:3px,color:#000; - class PMA,PRA,STA new + class CENT,CA,CB new ``` ## Alternatives considered -### A PodMonitor per replica +### Stop at per-cluster collection -`compose-model-replica` could compose a `PodMonitor` per replica, so collection comes and -goes with the deployment. With no opt-out and a cluster-wide Prometheus, that per-deployment -lifecycle buys nothing over one cluster-wide monitor, and it composes N monitors where one -does the same job. Collection is a cluster property, so it's composed once with the stack. +An earlier shape of this proposal collected on each cluster and left aggregation to the +platform team, publishing a Prometheus URL on the `InferenceCluster` status. Aggregating +up to a Modelplane view is the actual ask, so leaving it out just means everyone rebuilds +the same fleet view by hand. Per-cluster collection stays, but as the bottom half of the +pipeline, not the whole of it. -### A per-deployment opt-out field +### Raw engine metric names, no rebranding -An earlier shape put an `enabled` toggle on the deployment. It covers only the data plane -and asks an MD author to opt in or out of collection that the platform team plumbs and -consumes. Always-on collection fits the ownership better, so the toggle is dropped. +Aggregating the engines' native names (`vllm:*`, `llm_d_epp_*`) as-is is less work, but it +hands an operator a different vocabulary per engine and per component. The `modelplane_*` +surface is the point of aggregating in the first place: one set of names and labels for +the whole deployment. -### A managed central Prometheus in this proposal +### A PodMonitor per replica -Composing one central Prometheus for the whole deployment, federating every cluster and -scraping the control plane, would give an operator a single endpoint. It is more to own -than wiring up what each cluster already runs, so this proposal leaves aggregation to the -platform team and stops at the discoverable URL. +`compose-model-replica` could compose a `PodMonitor` per replica, so collection comes and +goes with the deployment. With no opt-out and a cluster-wide store, that per-deployment +lifecycle buys nothing over one cluster-wide selector, and it composes N monitors where +one does the same job. -### ServiceMonitor instead of PodMonitor +### A per-deployment opt-out field -A `ServiceMonitor` scrapes through a Service, so it needs a Service per scrape target. The -engine and picker pods are the targets, and a `PodMonitor` scrapes them directly, matching -the manual path. +An earlier shape put an `enabled` toggle on the deployment. It covers only the data plane +and asks an MD author to opt in or out of collection the platform team consumes. +Always-on collection fits the ownership better, so the toggle is dropped. ### Authenticate the EPP metrics endpoint The EPP can serve `/metrics` behind controller-runtime auth (a `ClusterRole` with `nonResourceURLs: /metrics` plus a bearer token). Since Modelplane owns the EPP args and the endpoint carries non-sensitive routing stats reachable only in-cluster, -`--metrics-endpoint-auth=false` collects them with a plain `PodMonitor` and nothing to -manage. Auth would add a `ClusterRole` and a bearer token for no gain here. +`--metrics-endpoint-auth=false` collects them with nothing to manage. Auth would add a +`ClusterRole` and a bearer token for no gain here. ## Open questions -- **Central aggregation.** Whether Modelplane should package a managed central Prometheus - for the whole deployment, or leave cross-cluster aggregation and control-plane - monitoring to the platform team. This proposal leaves it out; that is the main call to - make. +- **Prometheus stack or OpenTelemetry collector.** The mechanism above. The collector is + the lean, familiar default. Is there a reason to keep the per-cluster Prometheus? +- **Central store.** A single Prometheus is simplest to start; a horizontally scaled + backend (Thanos, Mimir, Cortex) is the answer once the aggregate outgrows one instance. + Which, and when. - **Port name.** `http` (the serving port that also serves `/metrics`) versus `metrics`. Leaning `http`, since it's the one serving port. @@ -226,8 +194,8 @@ manage. Auth would add a `ClusterRole` and a bearer token for no gain here. The [#264](https://github.com/modelplaneai/modelplane/issues/264) example documents the manual path: a hand-written `PodMonitor` plus the operator wiring to consume it. Once -collection is composed and the cluster's Prometheus URL is exposed, that example drops the -hand-written `podmonitor.yaml`. +collection is composed and aggregated, that example drops the hand-written +`podmonitor.yaml`. On upgrade, an existing hand-written `PodMonitor` has to be deleted, or it double-scrapes the same pods alongside the composed one. This warrants a release note. From 1e6e68b1d08d7b830336a1a3ee1486eb5817568b Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 5 Aug 2026 07:57:13 -0700 Subject: [PATCH 07/13] Lock the collector to OpenTelemetry and specify engine capture and normalization The doc left the collector as an open Prometheus-vs-OTel choice and asserted a modelplane_* namespace without saying how metrics from an engine Modelplane doesn't recognize become modelplane_* series. Commit to the OpenTelemetry collector, with the reasons (the GenAI conventions are the naming target, the rename runs in-pipeline, one pipeline carries metrics, traces, and logs). Add a capture section: an engine exposes Prometheus /metrics, an engine-type label picks a mapping from an extensible registry the way the GAIE picker already selects one for routing, and an unmapped engine degrades to raw names rather than a wrong guess. Add the normalized metric set and the per-engine mapping table with TTFT, ITL, TPOT, and the prefill/decode split. Move the Prometheus stack to a rejected alternative and re-tone throughout. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/metrics.md | 294 ++++++++++++++++++++++++++++------------------ 1 file changed, 181 insertions(+), 113 deletions(-) diff --git a/design/metrics.md b/design/metrics.md index adefeed50..2fb42b379 100644 --- a/design/metrics.md +++ b/design/metrics.md @@ -4,118 +4,177 @@ **Date:** August 2026 **Author:** Dennis Ramdass -This document proposes collecting metrics on every cluster and aggregating them up to -one Modelplane-level view at the control plane, with every series rebranded under a -`modelplane_*` namespace. It builds on [design.md](./design.md) and addresses +This document proposes collecting metrics on every cluster, normalizing them to a +`modelplane_*` namespace, and aggregating them up to one Modelplane view at the control +plane. It builds on [design.md](./design.md) and addresses [#269](https://github.com/modelplaneai/modelplane/issues/269). -Concretely, this proposes: +## Summary -1. per-cluster collection of every Modelplane source (engines, endpoint pickers, - substrate), always on, with no per-deployment toggle -2. aggregation up to a single Modelplane view at the control plane, so one query covers - the whole deployment -3. recording rules that rebrand every series under `modelplane_*` with normalized - labels, such as `modelplane_tokens_total{engine="vllm", cluster="...", - deployment="..."}` +I propose four things. -Approving this means agreeing that central aggregation is Modelplane's job, not the -platform team's, and to the `modelplane_*` naming. The collection mechanism, a -Prometheus stack or an OpenTelemetry collector, is the main open choice, below. +**Collect on every cluster, always on.** Modelplane collects from every source it owns, +the engines, the endpoint pickers, and the substrate, with no per-deployment toggle. + +**Normalize to `modelplane_*`.** Each engine names its metrics its own way (`vllm:*`, +`sglang:*`). The collector renames them to one Modelplane vocabulary, picked by an +engine-type label, so a dashboard reads Modelplane's names and not each engine's. + +**Aggregate to one view.** Every cluster's series roll up to a single store at the +control plane, so one query covers the whole deployment instead of a per-cluster island. + +**Collect with OpenTelemetry.** The collector is an OpenTelemetry collector, not a +per-cluster Prometheus. The section below gives the reasons. + +Approving this means agreeing that normalization and aggregation are Modelplane's job +rather than the platform team's, that collection is always on, and that the collector is +OpenTelemetry. ## What to monitor -There are four things worth watching in a Modelplane deployment. +Four things are worth watching in a Modelplane deployment. + +**Inference signal (data plane).** The engine's `/metrics`, the EPP's `llm_d_epp_*`, and +Envoy. TTFT, inter-token latency, tokens per second, queue depth, KV-cache occupancy, and +request and error rates per model. It answers "is my model serving well, and is it +saturated?" -1. **Inference signal (data plane).** vLLM's `/metrics`, the EPP's `llm_d_epp_*`, and - Envoy. TTFT, tokens per second, queue depth, KV-cache occupancy, and request and - error rates per model. It answers "is my model serving well, and is it saturated?" -2. **Substrate health.** The stack Modelplane installs on each workload cluster: is the - gateway up, are cert-manager, the LeaderWorkerSet controller, and the NVIDIA DRA - driver healthy, are GPUs allocatable. "Is the machinery on this cluster working?" -3. **Control-plane health.** Modelplane itself. Crossplane reconcile rates and errors, - function latency and panics, the fleet scheduler placing replicas, and XR - `Ready`/`Synced`. "Is the thing I operate working?" -4. **Fleet roll-up.** Across every cluster and deployment: total capacity, GPU usage, - how many deployments are degraded, and cost. +**Substrate health.** The stack Modelplane installs on each workload cluster. Is the +gateway up, are cert-manager, the LeaderWorkerSet controller, and the NVIDIA DRA driver +healthy, are GPUs allocatable. "Is the machinery on this cluster working?" + +**Control-plane health.** Modelplane itself. Crossplane reconcile rates and errors, +function latency and panics, the fleet scheduler placing replicas, and XR `Ready`/`Synced`. +"Is the thing I operate working?" + +**Fleet roll-up.** Across every cluster and deployment: total capacity, GPU usage, +degraded deployments, and cost. All four are in the central view. The data plane and substrate are collected on each -cluster and aggregated up; the control plane is scraped at the center; the fleet roll-up -is a set of recording rules over the aggregate. +cluster and aggregated up, the control plane is scraped at the center, and the fleet +roll-up is recording rules over the aggregate. -## Collection: per cluster, always on +## Collect on every cluster -On each cluster, Modelplane collects from every source it owns, with no opt-in or -opt-out. The targets are the same whichever mechanism scrapes them: the engine pods, the -endpoint picker, and the substrate. Three existing pieces make it cheap. +On each cluster Modelplane collects from every source it owns, with no opt-in or opt-out. +Three existing pieces make it cheap. -- **The serving label spans every shape.** `modelplane.ai/serving` is on standalone - pods, LeaderWorkerSet leaders, and both prefill/decode engines (it's the label the - InferencePool selects on). One selector on it follows the shape, so the leader/worker - and prefill/decode branching needs no special casing. +- **The serving label spans every shape.** `modelplane.ai/serving` is on standalone pods, + LeaderWorkerSet leaders, and both prefill and decode engines, since it's the label the + InferencePool selects on. One selector on it follows the shape, so leader/worker and + prefill/decode need no special casing. - **The stack already scrapes.** `compose-serving-stack` runs a metrics stack on every - workload cluster, so adding a scrape target is composition, not new infrastructure. It - already scrapes the gateway's Envoy proxies. + workload cluster and already scrapes the gateway's Envoy proxies, so adding a target is + composition, not new infrastructure. - **Modelplane owns the picker.** The EPP is Modelplane's own Deployment, so its metrics port and flags are ours to set. -### The scrape targets - A cluster-wide selector on `modelplane.ai/serving` covers every engine of every deployment, so collection is a cluster property rather than something composed per -replica. A second selector covers the endpoint pickers by their Modelplane label, and a -third covers the substrate `compose-serving-stack` installs (the gateway's Envoy proxies, -cert-manager, the LeaderWorkerSet controller, the NVIDIA DRA driver). - -The engine's serving port carries `/metrics`, so the backends name it (say `http`) in -`native.py`, `llmd.py`, and the decode engine in `routing.py`. The pd-sidecar's port -stays unnamed. Scraping the port by name matters for prefill/decode. The decode engine serves on 8001 -because the pd-sidecar takes 8000, so matching port 8000 by number would scrape the -sidecar. Referencing the port by name scrapes the engine on every pod regardless of its -number. - -What a source exposes on `/metrics` is the user's, set through engine flags like anything -else (vLLM's `--disable-log-stats` and friends). Modelplane scrapes what's there and -doesn't decide which metrics exist. - -## Aggregation: one Modelplane view - -Per-cluster collection is only half the ask. Each cluster's series aggregate up to a -single Modelplane store at the control plane, which also scrapes the control plane's own -metrics (Crossplane, the functions, the scheduler). One query then covers the whole -deployment, rather than a per-cluster island an operator has to stitch together by hand. - -Recording rules over the aggregate produce the `modelplane_*` surface. A raw -`vllm:num_requests_running` from one engine and an `llm_d_epp_*` from a picker become -`modelplane_*` series with a consistent label set (`engine`, `cluster`, `deployment`, -`model`), so a dashboard queries Modelplane's own vocabulary and doesn't track each -engine's native metric names. The fleet roll-up (capacity, GPU usage, degraded -deployments) is more recording rules over the same aggregate. - -## Mechanism: a Prometheus stack or an OpenTelemetry collector - -The targets and the `modelplane_*` surface are the same either way. The open choice is -what collects and forwards them. - -- **Prometheus stack.** Each cluster runs the kube-prometheus-stack `compose-serving-stack` - already installs, with composed `PodMonitor`s for the targets above. Each cluster - remote-writes to a central Prometheus (or Thanos, Mimir, or Cortex) at the control - plane, and recording rules there produce the `modelplane_*` series. It's the incumbent, - and recording rules and PromQL are standard. -- **OpenTelemetry collector.** A collector per cluster scrapes the same targets (the - Prometheus receiver), rebrands them in the pipeline (the transform processor), and - remote-writes up, with a central collector or store aggregating. It runs no full - Prometheus per cluster, does the `modelplane_*` rename in-pipeline rather than through - recording rules, and is something Upbound already runs elsewhere. - -The collector is the lighter and more familiar path, and worth taking unless the -per-cluster Prometheus earns a place for something aggregation doesn't need. This is the -call to settle. - -Whichever collects, the central layer reaches a cluster either because the cluster pushes -(remote-write) or because the center pulls. On the pull path Modelplane publishes each -cluster's endpoint on the `InferenceCluster` status; on the push path no such endpoint is -needed, and nothing is exposed outside the cluster. +replica. A second selector covers the endpoint pickers, and a third the substrate +`compose-serving-stack` installs. + +Scrape the engine port by name, not by number. The engine serves `/metrics` on its +serving port, which the backends name (say `http`) in `native.py`, `llmd.py`, and +`routing.py`. Under prefill/decode the decode engine serves on 8001 because the pd-sidecar +takes 8000, so matching 8000 by number would scrape the sidecar. By name, the scrape +follows the engine on every pod. + +## Capture from an opaque engine + +Modelplane doesn't know which engine a deployment runs. The ML team supplies an image and +args, and serving stays opaque to the engine inside. Normalization is the opposite. +`vllm:time_to_first_token_seconds` and `sglang:time_to_first_token_seconds` fold into one +`modelplane_*` series only if something knows which engine produced them. So we need just +enough engine identity to pick a mapping, and no more. + +The pattern is the one the [GAIE model-server-protocol](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/docs/proposals/003-model-server-protocol/README.md) +already uses, and that Modelplane's routing depends on: read a label, don't detect the +engine. The GAIE endpoint picker carries metric mappings for vLLM and SGLang and selects +one from an engine-type label on the pod. If Modelplane runs that picker for +KV-cache-aware routing, the label already exists, and normalization reuses it. One label, +one mapping registry, two consumers. + +- **A capture contract.** An engine exposes Prometheus `/metrics`. The required set + follows the GAIE protocol and the OpenTelemetry GenAI conventions: TTFT, time per output + token, queue depth, KV-cache occupancy. It's the metrics analogue of the OpenAI API + contract Modelplane already assumes for serving. +- **Selection by label.** An engine-type label (`modelplane.ai/engine: vllm`) picks the + mapping. The ML team already chose the engine in the image. Naming its kind for metrics + is one token and touches nothing about serving. +- **A mapping registry as data.** Modelplane provides mappings for the common engines + (vLLM, SGLang, Triton/TensorRT-LLM) as data, not code. A new or forked engine is a new + mapping entry and a label, with no Modelplane release, so a new engine doesn't wait on + us. +- **Graceful degradation.** An unlabelled or unmapped engine still gets scraped and + aggregated under its native names. The rename is skipped and Modelplane surfaces it + ("no mapping for `X`") rather than guessing a mapping and reporting the wrong thing. + +As engines emit the OpenTelemetry conventions directly (vLLM already emits OTLP traces, +and native OTLP metrics are in progress), each mapping shrinks toward identity and the +label becomes optional. + +## Normalize to `modelplane_*` + +The collector renames each engine's series to a `modelplane_*` surface with a consistent +label set (`engine`, `cluster`, `deployment`, `model`), so a dashboard reads one +vocabulary. Latency matters most. Measure it on P50/P90/P99 rather than the mean. The +distribution is right-skewed, so the mean hides the tail. Keep inference-only separate +from end-to-end. + +| `modelplane_*` | vLLM | SGLang | TRT-LLM / Triton | +| --- | --- | --- | --- | +| `time_to_first_token` | `vllm:time_to_first_token_seconds` | `sglang:time_to_first_token_seconds` | derived | +| `inter_token_latency` | `vllm:inter_token_latency_seconds` | `sglang:inter_token_latency_seconds` | derived | +| `time_per_output_token` | `vllm:time_per_output_token_seconds` | `sglang:time_per_output_token_seconds` | derived | +| `request_prefill_time` | `vllm:request_prefill_time_seconds` | `sglang:per_stage_req_latency_seconds` | `nv_trt_llm_*` | +| `request_decode_time` | `vllm:request_decode_time_seconds` | per-stage | `nv_trt_llm_*` | +| `e2e_request_latency` | `vllm:e2e_request_latency_seconds` | `sglang:e2e_request_latency_seconds` | `nv_inference_request_duration_us` | +| `requests_waiting` | `vllm:num_requests_waiting` | scheduler waiting | `nv_trt_llm_request_metrics` | +| `kv_cache_usage` | `vllm:kv_cache_usage_perc` | token usage | TRT-LLM KV metrics | +| `prefix_cache_hits` | `vllm:prefix_cache_hits` | cache hit | n/a | + +vLLM and SGLang map cleanly. Their names already nearly match, and both align to the +OpenTelemetry set. Triton and TensorRT-LLM expose batch-manager stats rather than native +TTFT and ITL histograms, so those rows are derived or wait on newer TensorRT-LLM metrics. +That gap is stated, not hidden. + +Inter-token latency and time per output token stay separate. ITL is the per-token gap a +streaming user feels. TPOT is the amortized decode rate. Only TPOT is in the OpenTelemetry +set, so we carry both. + +Under disaggregation the two roles show different health. A prefill worker is watched on +`modelplane_time_to_first_token` and prefill-queue depth. A decode worker is watched on +`modelplane_inter_token_latency` and `modelplane_kv_cache_usage`. A `role={prefill,decode}` +label carries the split, set from the same serving labels. + +## Aggregate to one view + +Per-cluster collection is half the ask. Each cluster's series roll up to a single +Modelplane store at the control plane, which also scrapes the control plane's own metrics +(Crossplane, the functions, the scheduler). One query then covers the whole deployment +rather than a per-cluster island an operator stitches together by hand. The fleet roll-up +(capacity, GPU usage, degraded deployments) is recording rules over the aggregate. + +## Collector: OpenTelemetry + +The collector is an OpenTelemetry collector. Three reasons settle it over a per-cluster +Prometheus. + +- **The normalization target is a standard.** The OpenTelemetry GenAI conventions already + define `time_to_first_token` and `time_per_output_token` as histograms with LLM-shaped + buckets. `modelplane_*` adopts those names rather than inventing them. +- **The rename happens in the pipeline.** The collector scrapes each engine's `/metrics` + with the Prometheus receiver. The transform processor renames the series to + `modelplane_*`, keyed by the engine label, before forwarding up. A Prometheus stack + pushes that rename into recording rules on every cluster and still needs its own + federation. +- **One pipeline carries three signals.** Metrics, the #77 traces, and logs travel + together, where a Prometheus stack is metrics only. + +Each cluster reaches the center by pushing (remote-write) or the center pulls. On the pull +path Modelplane publishes the cluster's endpoint on the `InferenceCluster` status. On the +push path nothing is exposed outside the cluster. ## Architecture @@ -123,14 +182,14 @@ needed, and nothing is exposed outside the cluster. flowchart LR subgraph icA["InferenceCluster A"] SA["engines / EPPs / substrate"] - CA["collector\n(Prometheus or OTel)"] + CA["OTel collector\n(scrape + rename to modelplane_*)"] end subgraph icB["InferenceCluster B"] - CB["collector"] + CB["OTel collector"] end subgraph cp["control plane"] XP["Crossplane\n(functions, scheduler, XRs)"] - CENT["Modelplane store\n+ modelplane_* recording rules"] + CENT["Modelplane store\n+ fleet roll-up rules"] end OP["operator\ndashboards + alerting"] SA --> CA @@ -144,33 +203,42 @@ flowchart LR ## Alternatives considered +### A Prometheus stack + +Each cluster runs the kube-prometheus-stack `compose-serving-stack` already installs, with +composed `PodMonitor`s, and remote-writes to a central Prometheus (or Thanos, Mimir, +Cortex). It's the incumbent and PromQL is standard. The collector wins for the reasons +above: it renames in the pipeline instead of through per-cluster recording rules, carries +traces and logs on the same path, and runs no full Prometheus per cluster. The central +store can still be Prometheus-compatible. + ### Stop at per-cluster collection -An earlier shape of this proposal collected on each cluster and left aggregation to the -platform team, publishing a Prometheus URL on the `InferenceCluster` status. Aggregating -up to a Modelplane view is the actual ask, so leaving it out just means everyone rebuilds -the same fleet view by hand. Per-cluster collection stays, but as the bottom half of the -pipeline, not the whole of it. +An earlier shape collected on each cluster and left aggregation to the platform team, +publishing a Prometheus URL on the `InferenceCluster` status. Aggregating up to a +Modelplane view is the actual ask, so leaving it out means everyone rebuilds the same +fleet view by hand. Per-cluster collection stays, but as the bottom half of the pipeline, +not the whole of it. -### Raw engine metric names, no rebranding +### Raw engine metric names, no normalization Aggregating the engines' native names (`vllm:*`, `llm_d_epp_*`) as-is is less work, but it hands an operator a different vocabulary per engine and per component. The `modelplane_*` -surface is the point of aggregating in the first place: one set of names and labels for -the whole deployment. +surface is the point of aggregating in the first place: one set of names and labels for the +whole deployment. ### A PodMonitor per replica `compose-model-replica` could compose a `PodMonitor` per replica, so collection comes and goes with the deployment. With no opt-out and a cluster-wide store, that per-deployment -lifecycle buys nothing over one cluster-wide selector, and it composes N monitors where -one does the same job. +lifecycle buys nothing over one cluster-wide selector, and it composes N monitors where one +does the same job. ### A per-deployment opt-out field An earlier shape put an `enabled` toggle on the deployment. It covers only the data plane -and asks an MD author to opt in or out of collection the platform team consumes. -Always-on collection fits the ownership better, so the toggle is dropped. +and asks an MD author to opt in or out of collection the platform team consumes. Always-on +collection fits the ownership better, so the toggle is dropped. ### Authenticate the EPP metrics endpoint @@ -182,11 +250,11 @@ the endpoint carries non-sensitive routing stats reachable only in-cluster, ## Open questions -- **Prometheus stack or OpenTelemetry collector.** The mechanism above. The collector is - the lean, familiar default. Is there a reason to keep the per-cluster Prometheus? -- **Central store.** A single Prometheus is simplest to start; a horizontally scaled - backend (Thanos, Mimir, Cortex) is the answer once the aggregate outgrows one instance. - Which, and when. +- **Central store.** A single Prometheus-compatible store is simplest to start. A + horizontally scaled backend (Thanos, Mimir, Cortex) is the answer once the aggregate + outgrows one instance. Which, and when. +- **Mapping registry shape.** How the per-engine mappings are packaged and extended, a + ConfigMap the collector reads or a small CRD. - **Port name.** `http` (the serving port that also serves `/metrics`) versus `metrics`. Leaning `http`, since it's the one serving port. From 7cece187608b9c0ef6eee68599cddb6d9df194fa Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 5 Aug 2026 09:28:55 -0700 Subject: [PATCH 08/13] Add sequence-length and disaggregation signals, and cluster scheduler metrics Add input and output sequence length to the normalized set, the two disaggregation bottleneck signals (queued prefill tokens, in-flight decode KV tokens), and a note that an autoscaler or SLA planner such as NVIDIA's Dynamo Planner reads these faster than a dashboard, so the scrape interval is a knob. Add SLO attainment to the fleet roll-up. Add a Cluster scheduler metrics section: the pod scheduler (kube-scheduler, or a gang scheduler like NVIDIA KAI or Volcano) is captured the way an engine is, a per-scheduler mapping normalized to modelplane_cluster_scheduler_*, with the name reserving modelplane_fleet_scheduler_* for a future fleet scheduler. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/metrics.md | 50 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 46 insertions(+), 4 deletions(-) diff --git a/design/metrics.md b/design/metrics.md index 2fb42b379..a1879c03c 100644 --- a/design/metrics.md +++ b/design/metrics.md @@ -40,8 +40,9 @@ request and error rates per model. It answers "is my model serving well, and is saturated?" **Substrate health.** The stack Modelplane installs on each workload cluster. Is the -gateway up, are cert-manager, the LeaderWorkerSet controller, and the NVIDIA DRA driver -healthy, are GPUs allocatable. "Is the machinery on this cluster working?" +gateway up, are cert-manager, the LeaderWorkerSet controller, the NVIDIA DRA driver, and +the pod scheduler healthy, are GPUs allocatable and gangs forming. "Is the machinery on +this cluster working?" **Control-plane health.** Modelplane itself. Crossplane reconcile rates and errors, function latency and panics, the fleet scheduler placing replicas, and XR `Ready`/`Synced`. @@ -133,6 +134,8 @@ from end-to-end. | `requests_waiting` | `vllm:num_requests_waiting` | scheduler waiting | `nv_trt_llm_request_metrics` | | `kv_cache_usage` | `vllm:kv_cache_usage_perc` | token usage | TRT-LLM KV metrics | | `prefix_cache_hits` | `vllm:prefix_cache_hits` | cache hit | n/a | +| `input_sequence_tokens` | `vllm:request_prompt_tokens` | prompt tokens | `nv_trt_llm_*` | +| `output_sequence_tokens` | `vllm:request_generation_tokens` | generation tokens | `nv_trt_llm_*` | vLLM and SGLang map cleanly. Their names already nearly match, and both align to the OpenTelemetry set. Triton and TensorRT-LLM expose batch-manager stats rather than native @@ -146,7 +149,45 @@ set, so we carry both. Under disaggregation the two roles show different health. A prefill worker is watched on `modelplane_time_to_first_token` and prefill-queue depth. A decode worker is watched on `modelplane_inter_token_latency` and `modelplane_kv_cache_usage`. A `role={prefill,decode}` -label carries the split, set from the same serving labels. +label carries the split, set from the same serving labels. The finer signals are the two +disaggregation bottlenecks, queued prefill tokens and in-flight decode KV tokens, exposed +per engine as forward-pass metrics. + +These series feed more than dashboards. An autoscaler or an SLA planner, with NVIDIA's +Dynamo Planner as the reference, reads the same normalized latency, sequence-length, and +queue series to size prefill against decode and to hold TTFT and ITL under target. Such a +consumer samples on the order of seconds, faster than a dashboard needs, so the scrape +interval is a knob rather than a fixed value. + +## Cluster scheduler metrics + +The engine is not the only pluggable component on a workload cluster. The pod scheduler +that places the engine pods is one too. By default it is kube-scheduler. For multi-node +gangs and GPU fairness a fleet may swap in a gang scheduler such as NVIDIA KAI or Volcano. +The collector already reaches these in-cluster pods. Modelplane treats a scheduler like an +engine, a per-scheduler mapping normalized to a `modelplane_cluster_scheduler_*` surface, +keyed by which scheduler is installed. The name says cluster because a +future Modelplane fleet scheduler, placing replicas across clusters rather than pods across +nodes, would get its own `modelplane_fleet_scheduler_*` surface. + +Five signals matter, and they answer whether a replica's pods reach GPUs and whether the +cluster's capacity is shared fairly across teams. + +- **Pending or unschedulable work.** kube-scheduler's `scheduler_pending_pods{queue}`, + Volcano's `volcano_unschedule_job_counts`, a KAI queue's waiting podgroups. +- **Scheduling latency.** `scheduler_scheduling_attempt_duration_seconds`, + `volcano_e2e_job_scheduling_latency_milliseconds`. +- **Gang readiness.** Whether a podgroup's pods can all start at once, + `volcano_queue_pod_group_pending_count` against `_running_count`. A gang that never forms + is a stuck multi-node deployment. +- **Per-queue GPU allocation against quota.** `kai_queue_allocated_gpus`, Volcano's + `volcano_queue_allocated_scalar_resources` against `_deserved_` and `_capacity_`, with + `volcano_queue_overused` for fairness. +- **Preemptions and evictions.** `scheduler_preemption_victims`, + `volcano_pod_preemption_victims`. + +The mapping and the degradation rule are the engine ones. An unmapped scheduler still gets +scraped under its native names, and Modelplane surfaces that rather than guessing. ## Aggregate to one view @@ -154,7 +195,8 @@ Per-cluster collection is half the ask. Each cluster's series roll up to a singl Modelplane store at the control plane, which also scrapes the control plane's own metrics (Crossplane, the functions, the scheduler). One query then covers the whole deployment rather than a per-cluster island an operator stitches together by hand. The fleet roll-up -(capacity, GPU usage, degraded deployments) is recording rules over the aggregate. +(capacity, GPU usage, degraded deployments, and SLO attainment such as the fraction of +requests under a TTFT target) is recording rules over the aggregate. ## Collector: OpenTelemetry From 93814128da20efe5f43f7820c4fb75d9ef6de6e6 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 5 Aug 2026 09:33:35 -0700 Subject: [PATCH 09/13] Stress-test cleanup: managed kube-scheduler, planner framing, error row Fix five things a close read surfaced. A managed cluster's kube-scheduler runs in the provider's control plane and may not be scrapable, so say so rather than claim the collector reaches it. Drop the Dynamo-specific forward-pass-metrics label from the general engine signal. Reword so the Dynamo Planner reads as the reference pattern for a consumer of the normalized series, not a consumer of them. Add a request-outcome row so the table matches the error rate the doc promises. Call the control-plane scheduler the fleet scheduler now that a cluster scheduler exists. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/metrics.md | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/design/metrics.md b/design/metrics.md index a1879c03c..c20c79b78 100644 --- a/design/metrics.md +++ b/design/metrics.md @@ -136,6 +136,7 @@ from end-to-end. | `prefix_cache_hits` | `vllm:prefix_cache_hits` | cache hit | n/a | | `input_sequence_tokens` | `vllm:request_prompt_tokens` | prompt tokens | `nv_trt_llm_*` | | `output_sequence_tokens` | `vllm:request_generation_tokens` | generation tokens | `nv_trt_llm_*` | +| `requests_total{outcome}` | `vllm:request_success_total` | request counters | Triton success/fail | vLLM and SGLang map cleanly. Their names already nearly match, and both align to the OpenTelemetry set. Triton and TensorRT-LLM expose batch-manager stats rather than native @@ -150,23 +151,24 @@ Under disaggregation the two roles show different health. A prefill worker is wa `modelplane_time_to_first_token` and prefill-queue depth. A decode worker is watched on `modelplane_inter_token_latency` and `modelplane_kv_cache_usage`. A `role={prefill,decode}` label carries the split, set from the same serving labels. The finer signals are the two -disaggregation bottlenecks, queued prefill tokens and in-flight decode KV tokens, exposed -per engine as forward-pass metrics. +disaggregation bottlenecks, queued prefill tokens and in-flight decode KV tokens, reported +by the engine's scheduler loop. -These series feed more than dashboards. An autoscaler or an SLA planner, with NVIDIA's -Dynamo Planner as the reference, reads the same normalized latency, sequence-length, and -queue series to size prefill against decode and to hold TTFT and ITL under target. Such a -consumer samples on the order of seconds, faster than a dashboard needs, so the scrape +These series feed more than dashboards. An autoscaler or an SLA planner reads the same +normalized latency, sequence-length, and queue series to size prefill against decode and +hold TTFT and ITL under target. NVIDIA's Dynamo Planner is the reference for such a +consumer. It samples on the order of seconds, faster than a dashboard needs, so the scrape interval is a knob rather than a fixed value. ## Cluster scheduler metrics The engine is not the only pluggable component on a workload cluster. The pod scheduler -that places the engine pods is one too. By default it is kube-scheduler. For multi-node -gangs and GPU fairness a fleet may swap in a gang scheduler such as NVIDIA KAI or Volcano. -The collector already reaches these in-cluster pods. Modelplane treats a scheduler like an -engine, a per-scheduler mapping normalized to a `modelplane_cluster_scheduler_*` surface, -keyed by which scheduler is installed. The name says cluster because a +that places the engine pods is one too. By default it is kube-scheduler. On a managed +cluster that scheduler sits in the provider's control plane and is often not scrapable. A +fleet running multi-node gangs or GPU fairness installs a gang scheduler instead, NVIDIA +KAI or Volcano. Those run as in-cluster pods the collector reaches. Modelplane treats such +a scheduler like an engine. A per-scheduler mapping, keyed by the one installed, normalizes +to a `modelplane_cluster_scheduler_*` surface. The name says cluster because a future Modelplane fleet scheduler, placing replicas across clusters rather than pods across nodes, would get its own `modelplane_fleet_scheduler_*` surface. @@ -193,7 +195,7 @@ scraped under its native names, and Modelplane surfaces that rather than guessin Per-cluster collection is half the ask. Each cluster's series roll up to a single Modelplane store at the control plane, which also scrapes the control plane's own metrics -(Crossplane, the functions, the scheduler). One query then covers the whole deployment +(Crossplane, the functions, the fleet scheduler). One query then covers the whole deployment rather than a per-cluster island an operator stitches together by hand. The fleet roll-up (capacity, GPU usage, degraded deployments, and SLO attainment such as the fraction of requests under a TTFT target) is recording rules over the aggregate. @@ -230,7 +232,7 @@ flowchart LR CB["OTel collector"] end subgraph cp["control plane"] - XP["Crossplane\n(functions, scheduler, XRs)"] + XP["Crossplane\n(functions, fleet scheduler, XRs)"] CENT["Modelplane store\n+ fleet roll-up rules"] end OP["operator\ndashboards + alerting"] From cbe4a2f9af62a651591e89b46879f4d9623b7815 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 5 Aug 2026 09:36:55 -0700 Subject: [PATCH 10/13] Resolve the open questions into decisions Take a stance on each rather than leave it open. The central store is a single Prometheus-compatible instance, moving to a horizontally scaled backend such as Mimir when one instance can't hold the fleet. The mapping registry is a ConfigMap the collector reads, with a CRD reserved for outside authors who need validation. The metrics port is http, since it is the one serving port. Drop the Open questions section. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/metrics.md | 30 +++++++++++++----------------- 1 file changed, 13 insertions(+), 17 deletions(-) diff --git a/design/metrics.md b/design/metrics.md index c20c79b78..897886e60 100644 --- a/design/metrics.md +++ b/design/metrics.md @@ -76,10 +76,11 @@ replica. A second selector covers the endpoint pickers, and a third the substrat `compose-serving-stack` installs. Scrape the engine port by name, not by number. The engine serves `/metrics` on its -serving port, which the backends name (say `http`) in `native.py`, `llmd.py`, and -`routing.py`. Under prefill/decode the decode engine serves on 8001 because the pd-sidecar -takes 8000, so matching 8000 by number would scrape the sidecar. By name, the scrape -follows the engine on every pod. +serving port, which the backends name `http` in `native.py`, `llmd.py`, and `routing.py`. +It is `http` and not `metrics` because it is the one serving port, not a dedicated metrics +one. Under prefill/decode the decode engine serves on 8001 because the pd-sidecar takes +8000, so matching 8000 by number would scrape the sidecar. By name, the scrape follows the +engine on every pod. ## Capture from an opaque engine @@ -104,9 +105,10 @@ one mapping registry, two consumers. mapping. The ML team already chose the engine in the image. Naming its kind for metrics is one token and touches nothing about serving. - **A mapping registry as data.** Modelplane provides mappings for the common engines - (vLLM, SGLang, Triton/TensorRT-LLM) as data, not code. A new or forked engine is a new - mapping entry and a label, with no Modelplane release, so a new engine doesn't wait on - us. + (vLLM, SGLang, Triton/TensorRT-LLM) as a ConfigMap the collector reads, not code. A new + or forked engine is a new entry and a label, with no Modelplane release, so a new engine + doesn't wait on us. A CRD makes sense only if outside parties author mappings and + need schema validation. - **Graceful degradation.** An unlabelled or unmapped engine still gets scraped and aggregated under its native names. The rename is skipped and Modelplane surfaces it ("no mapping for `X`") rather than guessing a mapping and reporting the wrong thing. @@ -200,6 +202,10 @@ rather than a per-cluster island an operator stitches together by hand. The flee (capacity, GPU usage, degraded deployments, and SLO attainment such as the fraction of requests under a TTFT target) is recording rules over the aggregate. +The store is a single Prometheus-compatible instance to start. When one instance can't hold +the whole fleet, it moves to a horizontally scaled backend such as Mimir, still +Prometheus-compatible so the queries and rules carry over. + ## Collector: OpenTelemetry The collector is an OpenTelemetry collector. Three reasons settle it over a per-cluster @@ -292,16 +298,6 @@ the endpoint carries non-sensitive routing stats reachable only in-cluster, `--metrics-endpoint-auth=false` collects them with nothing to manage. Auth would add a `ClusterRole` and a bearer token for no gain here. -## Open questions - -- **Central store.** A single Prometheus-compatible store is simplest to start. A - horizontally scaled backend (Thanos, Mimir, Cortex) is the answer once the aggregate - outgrows one instance. Which, and when. -- **Mapping registry shape.** How the per-engine mappings are packaged and extended, a - ConfigMap the collector reads or a small CRD. -- **Port name.** `http` (the serving port that also serves `/metrics`) versus `metrics`. - Leaning `http`, since it's the one serving port. - ## Interaction with #264 The [#264](https://github.com/modelplaneai/modelplane/issues/264) example documents the From 3007f9a98072ecbdc404943565fc13b52262bdfe Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 5 Aug 2026 09:39:15 -0700 Subject: [PATCH 11/13] Scope scaling out and make the mapping registry Crossplane-native The store is one Prometheus-compatible instance with a short retention window, and scaling it horizontally is out of scope, so drop the Mimir path. Move the mapping registry from a hand-edited ConfigMap to the serving-stack Composition, which renders the collector's config and versions the mappings with the package; a platform team extends the set through composition input. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/metrics.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/design/metrics.md b/design/metrics.md index 897886e60..a09e39f0c 100644 --- a/design/metrics.md +++ b/design/metrics.md @@ -104,11 +104,12 @@ one mapping registry, two consumers. - **Selection by label.** An engine-type label (`modelplane.ai/engine: vllm`) picks the mapping. The ML team already chose the engine in the image. Naming its kind for metrics is one token and touches nothing about serving. -- **A mapping registry as data.** Modelplane provides mappings for the common engines - (vLLM, SGLang, Triton/TensorRT-LLM) as a ConfigMap the collector reads, not code. A new - or forked engine is a new entry and a label, with no Modelplane release, so a new engine - doesn't wait on us. A CRD makes sense only if outside parties author mappings and - need schema validation. +- **A mapping registry as data.** Modelplane carries mappings for the common engines + (vLLM, SGLang, Triton/TensorRT-LLM) in the serving-stack Composition, which renders them + into the collector's config on each cluster. They are versioned data the package moves + as a set, not function code. A platform team adds a mapping for a new or forked engine + through composition input, with no fork, no hand-edited ConfigMap, and no wait on a + Modelplane release. - **Graceful degradation.** An unlabelled or unmapped engine still gets scraped and aggregated under its native names. The rename is skipped and Modelplane surfaces it ("no mapping for `X`") rather than guessing a mapping and reporting the wrong thing. @@ -202,9 +203,8 @@ rather than a per-cluster island an operator stitches together by hand. The flee (capacity, GPU usage, degraded deployments, and SLO attainment such as the fraction of requests under a TTFT target) is recording rules over the aggregate. -The store is a single Prometheus-compatible instance to start. When one instance can't hold -the whole fleet, it moves to a horizontally scaled backend such as Mimir, still -Prometheus-compatible so the queries and rules carry over. +The store is a single Prometheus-compatible instance with a short retention window. Scaling +it horizontally is out of scope here. ## Collector: OpenTelemetry @@ -256,8 +256,8 @@ flowchart LR ### A Prometheus stack Each cluster runs the kube-prometheus-stack `compose-serving-stack` already installs, with -composed `PodMonitor`s, and remote-writes to a central Prometheus (or Thanos, Mimir, -Cortex). It's the incumbent and PromQL is standard. The collector wins for the reasons +composed `PodMonitor`s, and remote-writes to a central Prometheus-compatible store. It's +the incumbent and PromQL is standard. The collector wins for the reasons above: it renames in the pipeline instead of through per-cluster recording rules, carries traces and logs on the same path, and runs no full Prometheus per cluster. The central store can still be Prometheus-compatible. From 50712425b2b1456ba1b3c678c38ba7470bcac404 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 5 Aug 2026 09:44:20 -0700 Subject: [PATCH 12/13] Clarify the mapping registry as a first-class MetricMapping resource Say plainly what a mapping is, a selector plus the engine's source names, the modelplane_* name each becomes, and the labels, and show a concrete vLLM MetricMapping. Make the registry first-class Modelplane resources rather than a ConfigMap or an EnvironmentConfig: compose-serving-stack reads every MetricMapping as a required resource, the way compose-model-deployment reads InferenceCluster and ModelCache, and renders them into the collector's config. A new engine is a new MetricMapping, validated on apply and discoverable, with no fork and no release. Schedulers use the same kind. Co-Authored-By: Claude Opus 4.8 --- design/metrics.md | 46 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 36 insertions(+), 10 deletions(-) diff --git a/design/metrics.md b/design/metrics.md index a09e39f0c..cb26c1a58 100644 --- a/design/metrics.md +++ b/design/metrics.md @@ -102,18 +102,43 @@ one mapping registry, two consumers. token, queue depth, KV-cache occupancy. It's the metrics analogue of the OpenAI API contract Modelplane already assumes for serving. - **Selection by label.** An engine-type label (`modelplane.ai/engine: vllm`) picks the - mapping. The ML team already chose the engine in the image. Naming its kind for metrics - is one token and touches nothing about serving. -- **A mapping registry as data.** Modelplane carries mappings for the common engines - (vLLM, SGLang, Triton/TensorRT-LLM) in the serving-stack Composition, which renders them - into the collector's config on each cluster. They are versioned data the package moves - as a set, not function code. A platform team adds a mapping for a new or forked engine - through composition input, with no fork, no hand-edited ConfigMap, and no wait on a - Modelplane release. + `MetricMapping`. The ML team already chose the engine in the image. Naming its kind for + metrics is one token and touches nothing about serving. +- **A registry of first-class resources.** Each mapping is a `MetricMapping`, a Modelplane + kind, not a ConfigMap or an EnvironmentConfig. Modelplane installs the built-in ones + (vLLM, SGLang, Triton/TensorRT-LLM). A platform team applies one more for a new or forked + engine. Being typed, it validates on apply and appears under `kubectl get metricmappings`, + and adding one is no fork and no Modelplane release. - **Graceful degradation.** An unlabelled or unmapped engine still gets scraped and aggregated under its native names. The rename is skipped and Modelplane surfaces it ("no mapping for `X`") rather than guessing a mapping and reporting the wrong thing. +A `MetricMapping` is small: a selector for the pods it applies to, the source names, the +`modelplane_*` name each becomes, and the labels to keep or add. The vLLM one: + +```yaml +apiVersion: modelplane.ai/v1alpha1 +kind: MetricMapping +metadata: + name: vllm +spec: + selector: + matchLabels: + modelplane.ai/engine: vllm + rename: + vllm:time_to_first_token_seconds: modelplane_time_to_first_token + vllm:inter_token_latency_seconds: modelplane_inter_token_latency + vllm:num_requests_waiting: modelplane_requests_waiting + labels: + add: { engine: vllm } +``` + +`compose-serving-stack` reads every `MetricMapping` as a required resource, the same way +`compose-model-deployment` reads `InferenceCluster` and `ModelCache`. It renders them into +the collector's config, the ConfigMap the OTel collector loads on each cluster. The +`rename` map becomes transform-processor rules. A new engine is a new `MetricMapping`, not a +package change. + As engines emit the OpenTelemetry conventions directly (vLLM already emits OTLP traces, and native OTLP metrics are in progress), each mapping shrinks toward identity and the label becomes optional. @@ -191,8 +216,9 @@ cluster's capacity is shared fairly across teams. - **Preemptions and evictions.** `scheduler_preemption_victims`, `volcano_pod_preemption_victims`. -The mapping and the degradation rule are the engine ones. An unmapped scheduler still gets -scraped under its native names, and Modelplane surfaces that rather than guessing. +A scheduler's mapping is a `MetricMapping` like an engine's, and the degradation rule +carries over. An unmapped scheduler still gets scraped under its native names, and +Modelplane surfaces that rather than guessing. ## Aggregate to one view From 31ea9d01bc9c9218ea7317b90d187f1463527c4e Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 5 Aug 2026 09:49:19 -0700 Subject: [PATCH 13/13] Stress-test cleanup: shared label not registry, throughput row, selector-scoped rename The picker and collector share the engine-type label, not a mapping registry, so say the label serves both rather than claim one shared registry. Add a tokens_total row so the table delivers the tokens-per-second the doc lists under what to monitor. State that a MetricMapping's rename applies only to metrics from pods its selector matches, which is what the selector is for. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Dennis Ramdass --- design/metrics.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/design/metrics.md b/design/metrics.md index cb26c1a58..f7eac0a18 100644 --- a/design/metrics.md +++ b/design/metrics.md @@ -94,8 +94,8 @@ The pattern is the one the [GAIE model-server-protocol](https://github.com/kuber already uses, and that Modelplane's routing depends on: read a label, don't detect the engine. The GAIE endpoint picker carries metric mappings for vLLM and SGLang and selects one from an engine-type label on the pod. If Modelplane runs that picker for -KV-cache-aware routing, the label already exists, and normalization reuses it. One label, -one mapping registry, two consumers. +KV-cache-aware routing, the engine-type label already exists, and normalization reuses it. +The label serves two consumers, the picker for routing and the collector for normalization. - **A capture contract.** An engine exposes Prometheus `/metrics`. The required set follows the GAIE protocol and the OpenTelemetry GenAI conventions: TTFT, time per output @@ -136,8 +136,8 @@ spec: `compose-serving-stack` reads every `MetricMapping` as a required resource, the same way `compose-model-deployment` reads `InferenceCluster` and `ModelCache`. It renders them into the collector's config, the ConfigMap the OTel collector loads on each cluster. The -`rename` map becomes transform-processor rules. A new engine is a new `MetricMapping`, not a -package change. +`rename` map becomes transform-processor rules, applied to metrics from the pods the +`selector` matches. A new engine is a new `MetricMapping`, not a package change. As engines emit the OpenTelemetry conventions directly (vLLM already emits OTLP traces, and native OTLP metrics are in progress), each mapping shrinks toward identity and the @@ -165,6 +165,7 @@ from end-to-end. | `input_sequence_tokens` | `vllm:request_prompt_tokens` | prompt tokens | `nv_trt_llm_*` | | `output_sequence_tokens` | `vllm:request_generation_tokens` | generation tokens | `nv_trt_llm_*` | | `requests_total{outcome}` | `vllm:request_success_total` | request counters | Triton success/fail | +| `tokens_total{kind}` | `vllm:prompt_tokens_total`, `vllm:generation_tokens_total` | token counters | Triton token counts | vLLM and SGLang map cleanly. Their names already nearly match, and both align to the OpenTelemetry set. Triton and TensorRT-LLM expose batch-manager stats rather than native