diff --git a/.github/upstream-projects.yaml b/.github/upstream-projects.yaml index 2b47be52..c0b36c46 100644 --- a/.github/upstream-projects.yaml +++ b/.github/upstream-projects.yaml @@ -44,7 +44,7 @@ projects: - id: toolhive repo: stacklok/toolhive - version: v0.34.0 + version: v0.35.0 # toolhive is a monorepo covering the CLI, the Kubernetes # operator, and the vMCP gateway. It also introduces cross- # cutting features that land in concepts/, integrations/, diff --git a/docs/toolhive/concepts/observability.mdx b/docs/toolhive/concepts/observability.mdx index c8f987c7..7d0e5237 100644 --- a/docs/toolhive/concepts/observability.mdx +++ b/docs/toolhive/concepts/observability.mdx @@ -227,6 +227,15 @@ monitoring backend operations, workflow executions, and optimizer performance. For details, see the [vMCP telemetry guide](../guides-vmcp/telemetry-and-metrics.mdx). +### Rate limit metrics + +When [rate limiting](../guides-k8s/rate-limiting.mdx) is configured on an +MCPServer or VirtualMCPServer, ToolHive emits metrics for bucket decisions, +Redis errors, and Redis Lua check latency so you can watch for rejections and +detect fail-open periods when Redis is unreachable. See +[Observe rate limit activity](../guides-k8s/rate-limiting.mdx#observe-rate-limit-activity) +for metric names, attributes, and example queries. + ## Trace context propagation ToolHive supports two methods of trace context propagation: diff --git a/docs/toolhive/guides-k8s/rate-limiting.mdx b/docs/toolhive/guides-k8s/rate-limiting.mdx index 1b2f2112..0b4fcadb 100644 --- a/docs/toolhive/guides-k8s/rate-limiting.mdx +++ b/docs/toolhive/guides-k8s/rate-limiting.mdx @@ -234,6 +234,65 @@ A request must pass both the server-level vMCP limit and the per-tool limit (if defined). Limits apply to the vMCP aggregator and are independent from any limits configured on the backend MCPServers it routes to. +## Observe rate limit activity + +When telemetry is enabled on the MCPServer or VirtualMCPServer (via a +[`MCPTelemetryConfig`](./telemetry-and-metrics.mdx) referenced by +`telemetryConfigRef`), the rate limiter emits OpenTelemetry metrics for each +Redis bucket check. Use them to watch for rejections, spot Redis health issues, +and track how long the atomic Lua check takes. + +All three metrics carry `namespace` and `server` attributes identifying the +resource that owns the bucket. Prometheus appends `_total` to counter names. The +latency histogram is exported with the `_seconds` unit suffix and the standard +`_bucket`, `_sum`, and `_count` series suffixes. + +| Metric | Type | Additional attributes | +| ----------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------- | +| `toolhive_rate_limit_decisions` | Counter | `decision` (`allowed` or `rejected`), `scope` (`shared` or `per_user`), `operation_type` (`server` or `tool`) | +| `toolhive_rate_limit_redis_errors` | Counter | `error_type` (`timeout`, `connection`, `auth`, or `other`) | +| `toolhive_rate_limit_check_latency` | Histogram | (none) | + +Counting semantics for `toolhive_rate_limit_decisions`: + +- An **allowed** request increments the counter once for every applicable + bucket. For example, a request that satisfies both a shared and a per-user + bucket increments the counter twice. +- A **rejected** request increments only for the first bucket rejected by the + atomic Redis check, so a single rejection is never double-counted across + scopes. +- Requests with no applicable bucket do not increment the counter. This includes + discovery methods that pass through unconditionally (`tools/list`, + `prompts/list`, `initialize`, `ping`) and unauthenticated requests against a + server that only has per-user buckets configured. + +The `toolhive_rate_limit_check_latency` histogram records the duration of each +Redis Lua call, whether the call succeeds or fails. Rate limit enforcement +[fails open](#how-rate-limiting-works) when Redis is unreachable, so watch +`toolhive_rate_limit_redis_errors` alongside decisions to detect fail-open +periods. + +Example PromQL queries against a Prometheus scrape of the server's `/metrics` +endpoint: + +```promql +# Rejection rate across all buckets, by server +sum by (namespace, server) ( + rate(toolhive_rate_limit_decisions_total{decision="rejected"}[5m]) +) + +# 95th-percentile Redis check latency +histogram_quantile( + 0.95, + sum by (le) (rate(toolhive_rate_limit_check_latency_seconds_bucket[5m])) +) + +# Redis errors by class. Non-zero timeout or connection rates warrant attention. +sum by (error_type) ( + rate(toolhive_rate_limit_redis_errors_total[5m]) +) +``` + :::info[Per-tool limits with the optimizer] When the [tool discovery optimizer](../guides-vmcp/optimizer.mdx) is enabled, @@ -281,6 +340,8 @@ spec: ## Next steps +- [Telemetry and metrics](./telemetry-and-metrics.mdx) to enable OpenTelemetry + collection and export for these metrics - [Token exchange](./token-exchange-k8s.mdx) to configure token exchange for upstream service authentication - [CRD reference](../reference/crds/index.mdx) for complete field definitions diff --git a/docs/toolhive/guides-vmcp/telemetry-and-metrics.mdx b/docs/toolhive/guides-vmcp/telemetry-and-metrics.mdx index 5e28c91d..2d38093b 100644 --- a/docs/toolhive/guides-vmcp/telemetry-and-metrics.mdx +++ b/docs/toolhive/guides-vmcp/telemetry-and-metrics.mdx @@ -167,6 +167,24 @@ These metrics track workflow execution across backends: | `toolhive_vmcp_workflow_errors` | Counter | Total workflow execution errors | | `toolhive_vmcp_workflow_duration` | Histogram | Duration of workflow executions | +### Rate limit metrics + +When [rate limiting](../guides-k8s/rate-limiting.mdx) is enabled on the +VirtualMCPServer, these metrics track token bucket decisions and the health of +the Redis backend that stores the buckets. Each metric carries `namespace` and +`server` attributes identifying the resource: + +| Metric | Type | Description | +| ----------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `toolhive_rate_limit_decisions` | Counter | Bucket decisions. Additional attributes: `decision` (`allowed` or `rejected`), `scope` (`shared` or `per_user`), `operation_type` (`server` or `tool`) | +| `toolhive_rate_limit_redis_errors` | Counter | Redis errors during rate limit checks. Additional attribute: `error_type` (`timeout`, `connection`, `auth`, or `other`) | +| `toolhive_rate_limit_check_latency` | Histogram | Duration of the atomic Redis Lua rate limit check, in seconds (each check, including failures) | + +For the counting semantics (allowed increments per bucket, rejected only for the +first rejected bucket) and example PromQL queries, see +[Observe rate limit activity](../guides-k8s/rate-limiting.mdx#observe-rate-limit-activity) +in the rate limiting guide. + ### Optimizer metrics When the vMCP optimizer is enabled, these metrics track tool-finding and diff --git a/static/api-specs/toolhive-api.yaml b/static/api-specs/toolhive-api.yaml index b70a631a..cc1db034 100644 --- a/static/api-specs/toolhive-api.yaml +++ b/static/api-specs/toolhive-api.yaml @@ -498,6 +498,13 @@ components: $ref: '#/components/schemas/github_com_stacklok_toolhive_pkg_authserver.DCRUpstreamConfig' identity_from_token: $ref: '#/components/schemas/github_com_stacklok_toolhive_pkg_authserver.IdentityFromTokenRunConfig' + insecure_allow_http: + description: |- + InsecureAllowHTTP permits plain-HTTP authorization and token endpoint URLs + for this upstream. Only for in-cluster development environments (e.g. an + OAuth2 provider served over HTTP in a kind cluster) where TLS is not + available. Never set this in production. + type: boolean redirect_uri: description: |- RedirectURI is the callback URL where the upstream IDP will redirect after authentication. @@ -552,6 +559,13 @@ components: ClientSecretFile is the path to a file containing the OAuth 2.0 client secret. Mutually exclusive with ClientSecretEnvVar. Optional for public clients using PKCE. type: string + insecure_allow_http: + description: |- + InsecureAllowHTTP permits a plain-HTTP issuer URL and HTTP discovery + endpoints for this upstream. Only for in-cluster development environments + (e.g. Dex served over HTTP in a kind cluster) where TLS is not available. + Never set this in production. + type: boolean issuer_url: description: |- IssuerURL is the OIDC issuer URL for automatic endpoint discovery.