From 88879edd8eed01472ba56fe53ae6c565a8cb705d Mon Sep 17 00:00:00 2001 From: Luke Knepper Date: Wed, 9 Sep 2026 16:26:57 -0700 Subject: [PATCH 1/2] Clarify Temporal Proxy fail-closed encryption --- .../temporal-proxy/encrypt-payloads.mdx | 25 +++++++++++++++++-- .../temporal-proxy/index.mdx | 6 +++-- 2 files changed, 27 insertions(+), 4 deletions(-) diff --git a/docs/production-deployment/temporal-proxy/encrypt-payloads.mdx b/docs/production-deployment/temporal-proxy/encrypt-payloads.mdx index 418c66b92c..f31efe0718 100644 --- a/docs/production-deployment/temporal-proxy/encrypt-payloads.mdx +++ b/docs/production-deployment/temporal-proxy/encrypt-payloads.mdx @@ -20,8 +20,9 @@ import { ReleaseNoteHeader } from '@site/src/components'; The proxy can encrypt Workflow and Activity payloads on the hop to an upstream and decrypt them on responses, set under the top-level `encryption` block. It is off by default. Workers and Clients keep exchanging cleartext with the gateway; -the proxy seals payloads before they leave and opens them on the way back, so the upstream Temporal Service only ever -stores ciphertext. Encryption is transparent, requiring no change to Worker or Client code. +the proxy seals codec-capable Payloads before they leave and opens them on the way back. New Payloads sent through the +proxy reach the upstream Temporal Service as ciphertext. Encryption is transparent, requiring no change to Worker or +Client code. It uses envelope encryption: a short-lived data encryption key (DEK) encrypts each payload with AES-256-GCM, and a KMS key you own wraps the DEK. The wrapped DEK and a reference to the key that wrapped it travel with the payload, so the @@ -77,6 +78,26 @@ encryption: - `cacheSize` bounds the in-memory cache of decrypted DEKs, which avoids a KMS call on every message. It must be non-negative. +:::note Scope of fail-closed behavior + +Fail-closed applies to outbound codec-capable Payloads on calls that pass through a proxy where `encryption.enabled` is +`true`. If the proxy cannot encrypt one of these Payloads, such as when the KMS is unavailable, it fails the request +before sending it upstream. It does not fall back to plaintext. + +The proxy cannot enforce encryption for a Client, Worker, Web UI, or CLI that connects directly to the upstream Temporal +Service. Restrict credentials and network paths when all application traffic must pass through the proxy. + +On responses, the proxy decrypts Payloads that carry its encryption metadata. It passes other Payloads through unchanged +so data written before proxy adoption remains readable during a migration. This behavior continues until existing +plaintext data ages out under the Namespace's retention period. + +Fail-closed does not mean that every request field is encrypted. Search Attributes remain unencrypted so the Temporal +Service can index them. Failure messages and call stacks are not codec-capable Payloads by default. See +[Codecs and Encryption](/production-deployment/data-encryption) for encryption coverage and +[Failure Converter](/failure-converter) for failure encoding. + +::: + `default` and each `overrides` entry are key policies with the same shape: | Field | Meaning | diff --git a/docs/production-deployment/temporal-proxy/index.mdx b/docs/production-deployment/temporal-proxy/index.mdx index 0377cd00c3..b4c2c79b44 100644 --- a/docs/production-deployment/temporal-proxy/index.mdx +++ b/docs/production-deployment/temporal-proxy/index.mdx @@ -78,8 +78,10 @@ For each request, the gateway: The per-upstream proxy then rewrites the local Namespace to the name the upstream expects, attaches that upstream's TLS and credentials, forwards to the Temporal Service, and translates the Namespace back on responses. When payload -encryption is enabled, it also seals payloads on the way out and opens them on the way back, so the upstream only ever -stores ciphertext. +encryption is enabled, it also seals codec-capable Payloads on the way out and opens them on the way back. New Payloads +sent through the proxy reach the upstream as ciphertext. See +[Encrypt payloads](/production-deployment/temporal-proxy/encrypt-payloads) for the scope and limits of this fail-closed +behavior. ### Terms From 5ebcbcbdda78a3bc24ae9a7739966d7e52a8c75c Mon Sep 17 00:00:00 2001 From: Luke Knepper Date: Thu, 10 Sep 2026 08:21:59 -0700 Subject: [PATCH 2/2] Reorganize Temporal Proxy documentation --- .../temporal-proxy/configure.mdx | 345 ++++-------------- .../temporal-proxy/deploy-kubernetes.mdx | 35 +- .../temporal-proxy/encrypt-payloads.mdx | 107 ++---- .../temporal-proxy/get-started-cloud.mdx | 171 +++++++++ .../temporal-proxy/index.mdx | 223 +++++------ .../temporal-proxy/manage-encryption-keys.mdx | 174 +++++++++ .../temporal-proxy/operate-troubleshoot.mdx | 158 ++++++++ .../temporal-proxy/plan-deployment.mdx | 136 +++++++ .../temporal-proxy/reference.mdx | 151 ++++++++ .../temporal-proxy/route-translate.mdx | 187 ++++++++++ .../temporal-proxy/secure-connections.mdx | 227 ++++++++++++ sidebars.js | 21 +- 12 files changed, 1435 insertions(+), 500 deletions(-) create mode 100644 docs/production-deployment/temporal-proxy/get-started-cloud.mdx create mode 100644 docs/production-deployment/temporal-proxy/manage-encryption-keys.mdx create mode 100644 docs/production-deployment/temporal-proxy/operate-troubleshoot.mdx create mode 100644 docs/production-deployment/temporal-proxy/plan-deployment.mdx create mode 100644 docs/production-deployment/temporal-proxy/reference.mdx create mode 100644 docs/production-deployment/temporal-proxy/route-translate.mdx create mode 100644 docs/production-deployment/temporal-proxy/secure-connections.mdx diff --git a/docs/production-deployment/temporal-proxy/configure.mdx b/docs/production-deployment/temporal-proxy/configure.mdx index fb170a76fc..33de92529d 100644 --- a/docs/production-deployment/temporal-proxy/configure.mdx +++ b/docs/production-deployment/temporal-proxy/configure.mdx @@ -1,8 +1,8 @@ --- id: configure -title: Configure the Temporal Proxy +title: Configure Temporal Proxy sidebar_label: Configure -description: Configure Temporal Proxy routing, upstreams, TLS, authentication, and payload encryption. +description: Assemble a Temporal Proxy configuration for routing, Namespace translation, access control, credentials, and payload encryption. toc_max_heading_level: 4 tags: - Temporal Service @@ -13,312 +13,117 @@ tags: import { ReleaseNoteHeader } from '@site/src/components'; - Temporal Proxy is under active development and evolving quickly. Behavior and configuration can change between - releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status and - the definitive configuration schema. + Temporal Proxy is under active development and is not ready for production use. Behavior and configuration can change + between releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for current status + and the definitive configuration schema. -The proxy reads a single YAML file. Three sections are the core of it: the gateway listener (`hostPort`), `routing`, and -the `upstreams` it forwards to. The optional `allowedServices`, `tls`, `auth`, `encryption`, and `extensionServers` -sections narrow which gRPC services the proxy forwards and add inbound TLS, inbound authentication and authorization, -and payload encryption. Values support `${VAR}` and `$VAR` environment variable expansion, and an upstream's `hostPort` -can be a template that resolves per request (for example, `{{ .RemoteNamespace }}`). +Temporal Proxy reads one YAML configuration file at startup. Begin with a gateway listener, routing policy, and one or +more upstreams. Add inbound access controls, upstream TLS and credentials, and payload encryption according to the +deployment boundary you planned. -The example below is the proxy's -[Temporal Cloud example](https://github.com/temporalio/temporal-proxy/tree/main/examples/cloud), which connects a Worker -to Temporal Cloud with an API key. The Worker carries no Cloud configuration: it talks plaintext to `127.0.0.1:7233`, -and the proxy adds TLS, the API key, and the Namespace rewrite on the way out. +This page shows how the configuration fits together. Follow the linked task pages for the complete behavior and +options of each section. + +## Assemble the configuration + +The following configuration exposes a plaintext local endpoint and forwards namespaced and Namespace-less calls to a +local Temporal Service: ```yaml -# Gateway: the local endpoint your Workers and Clients connect to (plaintext). hostPort: 127.0.0.1:7233 routing: - default: cloud # Namespaced requests. - system: system # Namespace-less requests (for example GetSystemInfo on connect). + default: local + system: local upstreams: - # Namespaced traffic. The host is derived per request from the translated - # Namespace, so one entry serves any number of Namespaces. - - name: cloud - hostPort: '{{ .RemoteNamespace }}.tmprl.cloud:7233' - tls: {} # Enable outbound TLS with defaults. - namespaces: - rules: - suffix: .$TEMPORAL_ACCOUNT # quickstart becomes quickstart. - credentials: - static: - apiKey: $TEMPORAL_API_KEY - - # Namespace-less calls have no Namespace to derive a host from, so they use a - # fixed endpoint. Any Namespace endpoint in the account answers them. - - name: system - hostPort: ${TEMPORAL_NAMESPACE}.${TEMPORAL_ACCOUNT}.tmprl.cloud:7233 - tls: {} - credentials: - static: - apiKey: ${TEMPORAL_API_KEY} -``` - -The [chart `values.yaml`](https://github.com/temporalio/helm-charts/tree/main/charts/temporal-proxy) and the -[temporal-proxy repository](https://github.com/temporalio/temporal-proxy) hold the complete, current set of options. - -## Restrict forwarded services - -The top-level `allowedServices` list names the gRPC services the proxy forwards, by proto full name. Omit it and the -proxy forwards `WorkflowService` and `OperatorService`, which is what an SDK Client, a Worker, the CLI, and the Web UI -need. - -```yaml -allowedServices: - - temporal.api.workflowservice.v1.WorkflowService - - temporal.api.operatorservice.v1.OperatorService - - grpc.reflection.v1.ServerReflection + - name: local + hostPort: 127.0.0.1:8233 ``` -| Service | Backs | In the default set | -| ------------------------------------------------- | ------------------------------------------------------------------------------- | ------------------ | -| `temporal.api.workflowservice.v1.WorkflowService` | Every SDK Client, Worker, and the Web UI | Yes | -| `temporal.api.operatorservice.v1.OperatorService` | `temporal operator` commands, and the Web UI's Search Attribute and Nexus views | Yes | -| `grpc.reflection.v1.ServerReflection` | Service discovery for tools such as `grpcurl` | No | +Build from that baseline in this order: -Those three are the whole set to select from. A name the proxy cannot forward fails the configuration at startup rather -than at the first request. +1. Define the gateway address with `hostPort` and, when callers cross an untrusted network, configure top-level `tls`. +2. Define each destination in `upstreams`, then select destinations with `routing`. +3. Add Namespace translation under an upstream's `namespaces` block when local and remote names differ. +4. Add an upstream `tls` block and `credentials` when the destination requires them. +5. Restrict inbound gRPC services with `allowedServices` and add `auth` when the gateway must authenticate or authorize + callers. +6. Add `encryption` when the proxy must encrypt supported Payload fields before forwarding them. -A service you leave out is never forwarded. The gateway answers a call to one with `Unimplemented` and -`unknown service ""`, decided before any upstream work, so the proxy behaves as a server that does not implement -it rather than passing the call through. +The configuration areas map to separate operating jobs: -Service discovery is opt-in, so name the reflection service only when you want tools such as `grpcurl` to probe the -gateway. Allowing `grpc.reflection.v1.ServerReflection` also allows the superseded -`grpc.reflection.v1alpha.ServerReflection`, because clients probe `v1` and fall back to it. The relation runs one way: -allowing only the superseded spelling does not allow the current one. +| Job | Configuration sections | Guide | +| --- | --- | --- | +| Select destinations and map Namespace names | `routing`, `upstreams[].hostPort`, `upstreams[].namespaces` | [Route requests and translate Namespaces](/production-deployment/temporal-proxy/route-translate) | +| Protect the gateway and upstream connections | `tls`, `allowedServices`, `auth`, `upstreams[].tls`, `upstreams[].credentials`, `extensionServers` | [Secure connections and authorize requests](/production-deployment/temporal-proxy/secure-connections) | +| Encrypt Payloads | `encryption`, optional KMS `extensionServers` | [Encrypt Payloads](/production-deployment/temporal-proxy/encrypt-payloads) | +| Rotate and retire encryption keys | `encryption.default`, `encryption.overrides`, `decryptURIs` | [Manage encryption keys](/production-deployment/temporal-proxy/manage-encryption-keys) | +| Mount configuration and secrets with Helm | Chart `config`, `env`, `envFrom`, and Secret references | [Deploy Temporal Proxy to Kubernetes](/production-deployment/temporal-proxy/deploy-kubernetes) | -### Health checks {/* #health-checks */} +See the [Temporal Proxy reference](/production-deployment/temporal-proxy/reference) for the top-level key map and the +source of truth for the complete schema. -The gateway serves the standard gRPC health service and publishes an entry for every allowed service under its proto -full name, alongside the usual empty service name. Each entry reports the same status, since the proxy's health is -process-wide. A Client health check that names a service, such as the Go SDK's `CheckHealth`, therefore gets a status, -and so does a per-service liveness probe. A service your configuration does not allow returns `NOT_FOUND` instead of -claiming a status. +## Supply environment-specific values -## Route requests - -The `routing` section selects an upstream for each request: - -- `default` is the fallback when no rule matches. It is optional; omit it to reject unmatched requests with an error. -- `system` is the upstream for Namespace-less requests, such as the SDK's `GetSystemInfo` and `GetClusterInfo` calls. It - is optional; when unset, those requests fall back to `default`. -- `rules` is an ordered list, evaluated top to bottom. The first match wins. - -Every upstream named by `default`, `system`, or a rule must exist in `upstreams`. Upstream `name` and `hostPort` values -must each be unique across the list: two upstreams sharing a name make a routing reference ambiguous, and two sharing an -address is a copy-paste error rather than a useful configuration. +Configuration string values support environment variable expansion in either `${VAR}` or `$VAR` form. Keep secrets +out of the YAML file and resolve them from the process environment: ```yaml -routing: - default: local # Fallback when no rule matches. - system: cloud # Namespace-less requests. - rules: - - match: - namespace: 'prod-*' - metadata: - x-tier: gold - upstream: cloud - - match: - namespace: '*-test' - upstream: local +upstreams: + - name: cloud + hostPort: ${TEMPORAL_NAMESPACE}.${TEMPORAL_ACCOUNT}.tmprl.cloud:7233 + tls: {} + credentials: + static: + apiKey: ${TEMPORAL_API_KEY} ``` -A rule matches when its Namespace matches and every metadata condition matches (AND logic). A `match` must set at least -one of `namespace` or `metadata`; an empty match is a configuration error, since that is what `default` is for. Routing -runs on the local Namespace, before translation. - -`namespace` is a string literal or a simple glob with a single leading or trailing `*`: - -| Pattern | Matches | -| ---------- | --------------------------- | -| `payments` | exactly `payments` | -| `prod-*` | names starting with `prod-` | -| `*-test` | names ending with `-test` | -| `*-test-*` | names containing `-test-` | -| `*` | any Namespace | +Undefined environment variables expand to an empty string, so verify that every required variable is present before +starting the proxy. On Kubernetes, use the chart's Secret integration instead of putting a credential in a ConfigMap. +See [Supply credentials from a Secret](/production-deployment/temporal-proxy/deploy-kubernetes#supply-credentials-from-a-secret). -A `*` in any other position, such as `a*b`, is invalid. - -`metadata` matches gRPC request metadata (headers). Keys are case-insensitive and do not support wildcards; values use -the same glob syntax as `namespace`. A key matches when any of the request's values for it match. - -## Translate Namespaces - -Applications connected to the proxy use short, local Namespace names. Each upstream rewrites those names to the ones its -Temporal Service expects, under `namespaces.rules`. The rewrite applies to requests and is reversed on responses, so -callers only ever see the local name. +An upstream `hostPort` and `tls.serverName` can also use Go templates that the proxy resolves for each request. For +example, the Temporal Cloud configuration can derive the endpoint from the translated Namespace: ```yaml upstreams: - name: cloud hostPort: '{{ .RemoteNamespace }}.tmprl.cloud:7233' tls: {} - namespaces: - rules: - prefix: '' # Optional string prepended to the local name. - suffix: .acct # payments becomes payments.acct - overrides: # Explicit pairs that bypass prefix and suffix. - - local: billing - remote: payments.acct -``` - -- `prefix` and `suffix` wrap every local Namespace: the remote name is `prefix + local + suffix`, and responses are - unwrapped back to the local name. -- `overrides` lists explicit `local` and `remote` pairs for names that do not follow the prefix and suffix convention. - An override takes precedence over the prefix and suffix rules. Each local name and each remote name may appear only - once. - -An upstream's `hostPort` and `tls.serverName` can be Go templates resolved per request, so one upstream can serve many -Namespaces. Available variables: - -- `{{ .LocalNamespace }}`: the Namespace before translation. -- `{{ .RemoteNamespace }}`: the Namespace after translation. -- `{{ .Metadata. }}` or `{{ index .Metadata "" }}`: a request metadata value. - -Upstreams with a static `hostPort` connect eagerly at startup; templated ones connect lazily on first use. - -## Authenticate and authorize inbound requests - -Inbound authentication runs on the gateway and is off by default: omit the top-level `auth` block to accept all -requests. When present, `auth` must select exactly one authenticator: `staticToken`, `jwks`, or `external`. The gateway -decides before the request is routed, so nothing reaches an upstream until the caller is admitted, and it strips the -credential before forwarding upstream. - -Compare an inbound bearer token against a fixed value with `staticToken`: - -```yaml -auth: - staticToken: - token: ${GATEWAY_TOKEN} # Required. The expected token value. - header: authorization # Header to read the token from. - scheme: Bearer # Scheme prefix to strip before comparing. ``` -Or verify a JWT's signature and claims against a JWKS endpoint with `jwks`: +Available template values and routing order are documented in +[Route requests and translate Namespaces](/production-deployment/temporal-proxy/route-translate). -```yaml -auth: - jwks: - url: https://issuer.example.com/.well-known/jwks.json # Required. Absolute https URL. - audiences: - - temporal-proxy - issuer: https://issuer.example.com/ - header: authorization - scheme: Bearer -``` - -`token` (for `staticToken`) and `url` (for `jwks`) are required; the remaining fields are optional. +## Start with the configuration -### Delegate the decision to an extension server +Pass the configuration path to `proxy serve`: -For an identity system neither built-in authenticator covers, or for a decision that turns on more than who the caller -is, point `auth.external` at an extension server you run. Declare the server under the top-level `extensionServers` -block, the same way you would a -[key management backend](/production-deployment/temporal-proxy/encrypt-payloads#plug-in-your-own-key-management-backend), -and the proxy asks it about every stream it accepts, forwarding only the ones it is told to admit. - -```yaml -extensionServers: - - name: authz - hostPort: 127.0.0.1:9444 - -auth: - external: - name: authz # Required. Names an entry in extensionServers. - credentialHeaders: - - authorization +```bash +proxy serve --config config.yaml ``` -`name` must match a configured extension server. `credentialHeaders` names the metadata headers that carry the caller's -credentials: the proxy lifts those into the request it sends the extension server and removes them from the stream it -forwards upstream, so a credential the server consumes never reaches the Temporal Service. - -Declaring no headers does not hide the caller's credentials from the extension server, since the proxy forwards the -caller's other metadata on the call either way. What you lose is the field naming them, so the server has to know which -metadata to read and cannot tell a header the proxy vouches for from any other. Nothing is stripped before forwarding -upstream either, so the caller's credential continues to the upstream alongside any credential configured for it. - -The server implements `api.auth.v1.AuthService`, one RPC defined in -[`api/auth/v1`](https://github.com/temporalio/temporal-proxy/tree/main/api/auth/v1). Each request carries what the call -is addressing and who is making it: - -| Field | Carries | -| ------------------ | ----------------------------------------------------------------------------------------------------------------- | -| `target.full_name` | The gRPC full method being invoked, leading slash included. Always set. | -| `target.namespace` | The Temporal Namespace the request names. | -| `credentials` | One entry per declared header the caller sent, each with the canonical header name and every value sent under it. | - -Both `target` fields come from the stream rather than from anything the caller claims, so a caller cannot forge either -by sending a header. `target.namespace` is empty when the method names no Namespace, when the caller sent no message to -read one from, or when the service is not one the proxy forwards. Empty means unknown rather than a Namespace called -nothing, so match a Namespace-scoped rule against `full_name` as well. `credentials` is empty when you declared no -headers or the caller sent none of them, which is an unauthenticated caller rather than a trusted one. - -Answer with a `decision`: - -- `DECISION_ALLOW` admits the caller and is the only value that does. A response left unfilled is - `DECISION_UNSPECIFIED`, which denies, so there is no way to admit a caller by omission. -- `DECISION_DENY` refuses the caller, who is told `PERMISSION_DENIED`. -- `reason` is optional and written for whoever operates the server. The proxy records it and keeps it out of what a - refused caller is told, so it can name internal systems or subjects. - -Return a gRPC error only for reaching no verdict at all, such as a backend the server cannot itself reach. The proxy -denies either way, but an error keeps its status code, so `UNAVAILABLE` or `DEADLINE_EXCEEDED` tells a Worker to retry -where a denial tells it not to bother. A server that cannot reach its own backend should report that rather than admit -the caller. - -The [authorization example](https://github.com/temporalio/temporal-proxy/tree/main/examples/authz) maps a JWT to claims -and then decides each call against them, the two steps Temporal Server splits across its `ClaimMapper` and `Authorizer`. -It includes a Worker that cannot reach a second Namespace and an auditor that can read Workflow History but not start a -Workflow. - -## Present credentials to upstreams - -Each upstream can present its own credential to the Temporal Service, set under `credentials`. `static` is the only -variant today; it injects a fixed API key as a bearer header on every outbound request, which is how you connect to -Temporal Cloud: - -```yaml -upstreams: - - name: cloud - hostPort: my-ns.acct.tmprl.cloud:7233 - tls: {} # Required whenever credentials are set. - credentials: - static: - apiKey: ${TEMPORAL_API_KEY} # Required. - header: authorization # Optional header override. - scheme: Bearer # Optional scheme override. -``` - -Credentials require TLS to the upstream. If you set `credentials` without a `tls` block, the configuration fails to -load. - -## Configure TLS {/* #configure-tls */} - -TLS is terminated in two independent places, both using the same keys: `ca`, `cert`, `key`, and `serverName`. `ca`, -`cert`, and `key` are paths to PEM files on disk, not inline PEM content. On Kubernetes, let the chart mount them from a -Secret and fill in the paths for you, as described in -[Supply TLS material](/production-deployment/temporal-proxy/deploy-kubernetes#supply-tls-material). +The `--config` flag also reads the `PROXY_CONFIG` environment variable. The proxy validates references and supported +values while it starts and exits when it finds an invalid configuration. It reads configuration only at startup, so +restart every proxy instance after a change. -**Inbound, on the gateway.** The top-level `tls` block secures connections from your applications. Set `cert` and `key` -for server TLS, and add `ca` to enforce mutual TLS, which requires each client to present a certificate signed by that -CA. Local development commonly omits `tls` and connects in plaintext. +Before directing application traffic to a new configuration: -**Outbound, per upstream.** Each upstream's `tls` block secures the connection from its proxy to the Temporal Service: +1. Start the proxy in an evaluation environment and resolve every startup error. +2. Check the gateway health service. +3. Connect a Client through the gateway and call both a Namespace-scoped operation and a Namespace-less operation. +4. Confirm the selected upstream, authentication result, and any encryption operations in logs and metrics. +5. Test direct upstream access when your design intends to prevent bypass. -- `tls: {}` verifies the upstream against the system root certificate pool and presents no client certificate. This is - what Temporal Cloud with an API key needs. -- `ca` alone verifies the upstream against a private trust anchor, still presenting no client certificate. -- `cert` and `key` together select mutual TLS and require `ca`. They must be set as a pair. +See [Operate and troubleshoot Temporal Proxy](/production-deployment/temporal-proxy/operate-troubleshoot) for health +checks, diagnostics, metrics, and common failure modes. -Set `serverName` when the host you dial does not match the common name or SAN on the server's certificate. +## Continue configuring -For payload encryption (envelope encryption, cloud KMS, and custom key backends), see -[Encrypt payloads](/production-deployment/temporal-proxy/encrypt-payloads). +- [Get started with Temporal Cloud](/production-deployment/temporal-proxy/get-started-cloud) +- [Plan a Temporal Proxy deployment](/production-deployment/temporal-proxy/plan-deployment) +- [Route requests and translate Namespaces](/production-deployment/temporal-proxy/route-translate) +- [Secure connections and authorize requests](/production-deployment/temporal-proxy/secure-connections) +- [Encrypt Payloads](/production-deployment/temporal-proxy/encrypt-payloads) diff --git a/docs/production-deployment/temporal-proxy/deploy-kubernetes.mdx b/docs/production-deployment/temporal-proxy/deploy-kubernetes.mdx index 544af9da89..1c5aa5a79e 100644 --- a/docs/production-deployment/temporal-proxy/deploy-kubernetes.mdx +++ b/docs/production-deployment/temporal-proxy/deploy-kubernetes.mdx @@ -2,7 +2,7 @@ id: deploy-kubernetes title: Deploy Temporal Proxy to Kubernetes sidebar_label: Deploy to Kubernetes -description: Deploy the Temporal Proxy with the Helm chart, including configuration, secrets, TLS, and KMS access. +description: Deploy Temporal Proxy with the Helm chart, mount configuration and secrets, configure probes and shutdown, and grant KMS access. toc_max_heading_level: 4 tags: - Temporal Service @@ -13,9 +13,9 @@ tags: import { ReleaseNoteHeader } from '@site/src/components'; - Temporal Proxy is under active development and evolving quickly. Behavior and configuration can change between - releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status and - the definitive configuration schema. + Temporal Proxy is under active development and is not ready for production use. Behavior and configuration can change + between releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for current status + and the definitive configuration schema. The [Helm chart](https://github.com/temporalio/helm-charts/tree/main/charts/temporal-proxy) provisions everything the @@ -23,6 +23,10 @@ proxy needs: a Deployment, a Service, the ConfigMap that holds your configuratio HorizontalPodAutoscaler and PodDisruptionBudget. The `extraObjects` value renders additional manifests as-is, for example an ExternalSecret that supplies upstream credentials. +Before choosing replica placement or exposing the gateway, see +[Plan a Temporal Proxy deployment](/production-deployment/temporal-proxy/plan-deployment) for topology, availability, +and trust-boundary guidance. + ## Supply the configuration `config` mirrors the proxy's own configuration schema. Whatever you put under it is what lands in the ConfigMap the @@ -95,9 +99,10 @@ The gateway's `config.tls` block and each `upstreams[].tls` block accept a `secr | `key` | `/tls.key`, or `/` if `keyKey` is set | | `ca` | `/`, only when `caKey` is set | -`ca` is opt-in because it changes behavior rather than just adding material. On the gateway, a `ca` enforces mutual TLS, -so only set `caKey` on `config.tls` when you intend to require client certificates. See -[Configure TLS](/production-deployment/temporal-proxy/configure#configure-tls) for what each combination means. +`ca` is opt-in because it changes connection behavior. On the gateway, a `ca` enforces mutual TLS, so only set `caKey` +on `config.tls` when you intend to require client certificates. See +[Secure upstream connections](/production-deployment/temporal-proxy/secure-connections#secure-upstream-connections) for +what each combination means. ```yaml config: @@ -135,21 +140,21 @@ readinessProbe: Naming a service is optional; omit it to check the empty service name. Either way the answer is the same, because the proxy's health is process-wide, and a service your configuration does not allow returns `NOT_FOUND`. See -[Health checks](/production-deployment/temporal-proxy/configure#health-checks) for what the gateway publishes. +[Check process health](/production-deployment/temporal-proxy/operate-troubleshoot#check-process-health) for what the +gateway publishes and what it does not test. Kubernetes gRPC probes connect in plaintext, so they cannot reach a gateway that terminates TLS. When `config.tls` is set, probe the proxy some other way, such as an exec probe running a client that presents the right certificate. ## Shut down cleanly -On `SIGTERM` the proxy stops accepting connections and drains in-flight RPCs. Each serving tier, meaning the gateway and -each upstream proxy, drains within five seconds, and the whole stop sequence is bounded at 30 seconds. Long-poll methods -block for much longer than that, so a rollout normally cuts some of them short: the proxy logs a warning naming the -deadline that ended the drain and still exits zero, because the callers re-poll. Treat that warning as ordinary on a -restart rather than a failed shutdown. +Set `terminationGracePeriodSeconds` to more than 30 seconds. On `SIGTERM`, the proxy stops admitting new requests and +drains each serving tier for up to five seconds; its complete shutdown is bounded at 30 seconds. A shorter Kubernetes +grace period can force-kill the process before its lifecycle hooks finish. Long polls can exceed a tier's drain window +and reconnect through another healthy replica. -An ordinary drain finishes in about five seconds, well inside the Kubernetes default `terminationGracePeriodSeconds` -of 30. +See [Shut down the proxy](/production-deployment/temporal-proxy/operate-troubleshoot#shut-down-the-proxy) for the full +shutdown sequence and expected warning. ## Grant access to your KMS key diff --git a/docs/production-deployment/temporal-proxy/encrypt-payloads.mdx b/docs/production-deployment/temporal-proxy/encrypt-payloads.mdx index f31efe0718..3af3dbaedb 100644 --- a/docs/production-deployment/temporal-proxy/encrypt-payloads.mdx +++ b/docs/production-deployment/temporal-proxy/encrypt-payloads.mdx @@ -1,8 +1,8 @@ --- id: encrypt-payloads -title: Encrypt payloads with Temporal Proxy -sidebar_label: Encrypt payloads -description: Configure Temporal Proxy payload encryption with envelope encryption and cloud or custom KMS backends. +title: Encrypt Payloads with Temporal Proxy +sidebar_label: Encrypt Payloads +description: Protect supported Temporal Payload fields with envelope encryption backed by AWS KMS, Azure Key Vault, Google Cloud KMS, or a custom service. toc_max_heading_level: 4 tags: - Temporal Service @@ -13,8 +13,8 @@ tags: import { ReleaseNoteHeader } from '@site/src/components'; - Temporal Proxy is under active development and evolving quickly. Behavior and configuration can change between - releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status + Temporal Proxy is under active development and is not ready for production use. Behavior and configuration can change + between releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for current status and the definitive configuration schema. @@ -26,8 +26,8 @@ Client code. It uses envelope encryption: a short-lived data encryption key (DEK) encrypts each payload with AES-256-GCM, and a KMS key you own wraps the DEK. The wrapped DEK and a reference to the key that wrapped it travel with the payload, so the -proxy never holds key material; it calls your KMS to wrap and unwrap DEKs. Supported key schemes are `awskms`, -`azurekeyvault`, and `gcpkms` for cloud KMS, `extension` for a +proxy does not receive the KMS key material. It calls your KMS to wrap and unwrap DEKs and can cache decrypted DEKs in +memory. Supported key schemes are `awskms`, `azurekeyvault`, and `gcpkms` for cloud KMS, `extension` for a [key management backend you run yourself](#plug-in-your-own-key-management-backend), and `testing` for local development only. @@ -52,7 +52,7 @@ flowchart LR Open -->|cleartext response| App ``` -**Configuration** +## Configure payload encryption ```yaml encryption: @@ -78,18 +78,22 @@ encryption: - `cacheSize` bounds the in-memory cache of decrypted DEKs, which avoids a KMS call on every message. It must be non-negative. +## Understand fail-closed behavior + :::note Scope of fail-closed behavior Fail-closed applies to outbound codec-capable Payloads on calls that pass through a proxy where `encryption.enabled` is `true`. If the proxy cannot encrypt one of these Payloads, such as when the KMS is unavailable, it fails the request before sending it upstream. It does not fall back to plaintext. -The proxy cannot enforce encryption for a Client, Worker, Web UI, or CLI that connects directly to the upstream Temporal -Service. Restrict credentials and network paths when all application traffic must pass through the proxy. +The proxy cannot enforce encryption for a Client, Worker, self-hosted Web UI, or CLI that connects directly to the +upstream Temporal Service. Restrict credentials and network paths when all application traffic must pass through the +proxy. On responses, the proxy decrypts Payloads that carry its encryption metadata. It passes other Payloads through unchanged -so data written before proxy adoption remains readable during a migration. This behavior continues until existing -plaintext data ages out under the Namespace's retention period. +so data written before proxy adoption remains readable during a migration. Closed Workflow histories normally age out +under the Namespace's retention period, but open Workflows, Archival, exports, backups, and other copies can preserve +plaintext data longer. Fail-closed does not mean that every request field is encrypted. Search Attributes remain unencrypted so the Temporal Service can index them. Failure messages and call stacks are not codec-capable Payloads by default. See @@ -98,6 +102,8 @@ Service can index them. Failure messages and call stacks are not codec-capable P ::: +## Define key policies + `default` and each `overrides` entry are key policies with the same shape: | Field | Meaning | @@ -107,57 +113,10 @@ Service can index them. Failure messages and call stacks are not codec-capable P | `duration` | How long a DEK is used for new encryption before it rotates. Must be greater than zero. | | `renewBefore` | Lead time before expiry at which the proxy pre-rotates the DEK. Must be at least zero and less than `duration`. | -DEK rotation is automatic on the `duration` and `renewBefore` schedule; you do not manage it. Older payloads stay -decryptable because their wrapped DEK and key reference travel with them. - -To change the active key without losing access to existing data, for example when switching providers, promote the new -key to `uri` and move the old one to `decryptURIs` in the same update. New payloads use the new key; older payloads -still resolve against the decrypt-only entry: - -```yaml -encryption: - default: - uri: gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v2 - decryptURIs: - - gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v1 -``` - -:::note Multi-region deployments need a multi-region key - -To stay available alongside a highly available upstream that fails over across regions, such as a Temporal Cloud High -Availability Namespace whose endpoint moves between regions, you typically run the proxy in more than one region. After -a failover, a payload sealed by the proxy in one region may be opened by the proxy in another, so every regional proxy -must be able to unwrap the others' DEKs. Back the key with a multi-region KMS key so the same key material is reachable -from every region. How you express that differs by provider: - -- **AWS:** use a multi-Region key (its ID starts with `mrk-`) and replicate it into each region. The replicas share key - material but each has its own regional ARN, so set each proxy's `uri` to its local ARN and list the other regions' - ARNs in `decryptURIs`: - - ```yaml - encryption: - default: - # Config for the us-east-1 proxy. - uri: awskms:///arn:aws:kms:us-east-1:123456789012:key/mrk-abcd1234?region=us-east-1 - decryptURIs: - - awskms:///arn:aws:kms:us-west-2:123456789012:key/mrk-abcd1234?region=us-west-2 - ``` - -- **Azure:** Key Vault keys are regional. Use a separate vault per region (or geo-replication), and, because each vault - gives the key a distinct URI, set each proxy's `uri` to its local vault and list the other vaults' keys in - `decryptURIs`. -- **GCP:** create the key in a multi-region location (for example, `locations/us`) or in `global`. The resource name is - the same everywhere, so every proxy uses the identical `uri` and no `decryptURIs` are needed: - `gcpkms://projects/my-project/locations/us/keyRings/codec/cryptoKeys/my-key`. - -For a single-region proxy deployment, a regional key is the right choice; reach for a multi-region key only when the -proxy itself spans regions. - -::: - -Decryption always runs on any payload that references a known key, regardless of `enabled`. Setting `enabled: false` -stops new encryption but keeps opening older payloads, which is a valid decrypt-only posture. The proxy reads encryption -configuration at startup, so restart it after changing keys or policies. +The proxy rotates DEKs automatically on the `duration` and `renewBefore` schedule. You remain responsible for changing, +retaining, and eventually retiring the KMS keys that wrap those DEKs. See +[Manage encryption keys](/production-deployment/temporal-proxy/manage-encryption-keys) for key migration, +multi-region access, decrypt-only operation, and safe retirement. The `testing://` scheme holds its key material in the configuration itself and provides no real security. Use it only for local development, and point production at `awskms`, `azurekeyvault`, `gcpkms`, or an extension server. @@ -300,23 +259,13 @@ Two constraints follow from that shape. `Decrypt` gets ciphertext and nothing el the local, pre-translation name, so a server keying on Namespace must use that name rather than the translated remote one. -The proxy builds the connection when it builds its key policies, so a malformed address, credentials without TLS, or an -unreadable `ca` file all fail at startup. Peer verification happens later, because gRPC connects lazily: a `ca` or -`serverName` that does not match the server's certificate is not discovered until the first payload needs a key, and it -surfaces as a hung request rather than a clear error. +The proxy validates and opens extension-server connections before the gateway starts accepting traffic. A malformed or +unreachable address, credentials without TLS, unreadable `ca` file, or certificate mismatch therefore fails startup. A +credential that the extension server rejects, or a dependency that fails after startup, surfaces when a request needs a +key operation and fails that request. The [KMS extension server example](https://github.com/temporalio/temporal-proxy/tree/main/examples/kms) runs the whole path on localhost, with a reference provider you can read as a starting point. -:::caution Deleting a key is irreversible - -Deleting a KMS key destroys everything encrypted under it, and neither Temporal nor your cloud provider can recover it. -Keep every key reachable, as `uri` or in `decryptURIs`, until every payload it wrapped is gone from every system the -proxy decrypts for, including Workflow history, visibility, and Archival storage. In practice, keep a key at least until -the Workflow Retention Period has elapsed. - -To retire a key, revoke the proxy's permissions rather than deleting it. Removing access is reversible: you can re-grant -it later if a payload still needs opening, whereas a deleted key is gone for good. Revoking only the encrypt permission -while leaving decrypt in place also lets you stop new encryption without stranding existing payloads. - -::: +After encryption works, define how keys change and how long old keys remain available. See +[Manage encryption keys](/production-deployment/temporal-proxy/manage-encryption-keys). diff --git a/docs/production-deployment/temporal-proxy/get-started-cloud.mdx b/docs/production-deployment/temporal-proxy/get-started-cloud.mdx new file mode 100644 index 0000000000..3ac2efbbff --- /dev/null +++ b/docs/production-deployment/temporal-proxy/get-started-cloud.mdx @@ -0,0 +1,171 @@ +--- +id: get-started-cloud +title: Get started with Temporal Proxy and Temporal Cloud +sidebar_label: Get started with Temporal Cloud +description: + Run the checked-in Temporal Proxy example to route a Worker to Temporal Cloud with TLS, credentials, and Namespace + translation handled by the proxy. +toc_max_heading_level: 4 +tags: + - Temporal Service + - Self-hosting + - Security +--- + +import { ReleaseNoteHeader } from '@site/src/components'; + + + Temporal Proxy is under active development and is not ready for production use. Behavior and configuration can change + between releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for current status + and the definitive configuration schema. + + +Run the checked-in [Temporal Cloud example](https://github.com/temporalio/temporal-proxy/tree/main/examples/cloud) to +connect a Worker and an SDK Client to Temporal Cloud through a local proxy. The Worker and Client use a short Namespace +name and carry no Temporal Cloud endpoint, TLS, or API key configuration. The proxy adds those settings before it sends +each request to Temporal Cloud. + +## Prerequisites + +Before you run the example, prepare the Temporal Cloud resources and local tools it uses: + +- A [Temporal Cloud Namespace](/cloud/namespaces) that allows API key authentication. Note its fully qualified name, + `.`. +- A [Temporal Cloud API key](/cloud/api-keys) whose identity can access the Namespace. +- [Git](https://git-scm.com/) and [Go](https://go.dev/) installed locally. +- Three terminal windows or tabs. + +The example runs the proxy from source and downloads its Go dependencies the first time you start it. + +## Get the example + +Clone the Temporal Proxy repository: + +```bash +git clone https://github.com/temporalio/temporal-proxy.git +``` + +The example is in the repository's `examples/cloud` directory. Its +[`config.yaml`](https://github.com/temporalio/temporal-proxy/blob/main/examples/cloud/config.yaml) defines a local +gateway on `127.0.0.1:7233`, a namespaced Temporal Cloud upstream, and a separate upstream for Namespace-less system +calls. + +## Run the example + +Run the proxy, Worker, and starter in separate terminals. Keeping the API key only in the proxy terminal demonstrates +that the application processes do not need the Temporal Cloud credential. + +### Start the proxy + +In the first terminal, set the short part of your fully qualified Namespace name. For example, use `quickstart` for +`quickstart.a1b2c`: + +```bash +export TEMPORAL_NAMESPACE=quickstart +``` + +Set the account identifier, which is the part after the dot in the fully qualified Namespace name: + +```bash +export TEMPORAL_ACCOUNT=a1b2c +``` + +Set the API key: + +```bash +export TEMPORAL_API_KEY='' +``` + +Change to the repository root: + +```bash +cd temporal-proxy +``` + +Start the proxy with the example configuration: + +```bash +go run ./cmd/proxy serve -c examples/cloud/config.yaml +``` + +The gateway-to-application connection and the proxy's internal sockets are plaintext in this local example. The +outbound connection from the proxy to Temporal Cloud uses TLS. + +### Start the Worker + +In the second terminal, change to the example directory: + +```bash +cd temporal-proxy/examples/cloud +``` + +Set only the short Namespace name. Do not set `TEMPORAL_ACCOUNT` or `TEMPORAL_API_KEY` in this terminal: + +```bash +export TEMPORAL_NAMESPACE=quickstart +``` + +Start the Worker: + +```bash +go run ./worker +``` + +The Worker connects to `localhost:7233` and polls the `cloud-example` Task Queue through the proxy. + +### Start a Workflow + +In the third terminal, change to the example directory: + +```bash +cd temporal-proxy/examples/cloud +``` + +Set only the short Namespace name: + +```bash +export TEMPORAL_NAMESPACE=quickstart +``` + +Run the starter: + +```bash +go run ./starter +``` + +The starter waits for the Workflow result and prints: + +```text +Hello, Temporal! +``` + +Press `CTRL+C` in the Worker and proxy terminals when you finish the example. + +## Verify the request path + +The greeting confirms that the starter, Worker, proxy, and Temporal Cloud completed a request-response cycle. You can +also open the Namespace in the [Temporal Cloud UI](https://cloud.temporal.io) and find the Workflow Id +`cloud-example-greeting`. + +The example separates application configuration from upstream configuration: + +| Process | Address and Namespace | Temporal Cloud TLS | Temporal Cloud API key | +| ------- | --------------------- | ------------------ | ---------------------- | +| Worker and starter | `localhost:7233`, short Namespace name | None | None | +| Temporal Proxy | Namespace endpoint derived from the translated Namespace name | Enabled | Read from `TEMPORAL_API_KEY` | + +For namespaced calls, the proxy appends `.$TEMPORAL_ACCOUNT` to the local Namespace and derives the endpoint from the +translated value. For Namespace-less calls such as `GetSystemInfo`, it uses the fixed `system` upstream. This split lets +one configuration serve more than one Namespace while still giving system calls an endpoint. + +## Continue configuring the proxy + +The example gateway accepts plaintext connections without inbound authentication, which is appropriate only for this +local evaluation. Before you place the gateway on a network, +[plan the deployment](/production-deployment/temporal-proxy/plan-deployment) and +[configure the proxy](/production-deployment/temporal-proxy/configure) for the callers and upstreams in your +environment. + +To keep new Workflow and Activity Payloads encrypted at the upstream, configure a production KMS provider under +[Encrypt payloads](/production-deployment/temporal-proxy/encrypt-payloads). To run replicated proxy instances with +Kubernetes, see [Deploy Temporal Proxy to Kubernetes](/production-deployment/temporal-proxy/deploy-kubernetes). diff --git a/docs/production-deployment/temporal-proxy/index.mdx b/docs/production-deployment/temporal-proxy/index.mdx index b4c2c79b44..14ede5d716 100644 --- a/docs/production-deployment/temporal-proxy/index.mdx +++ b/docs/production-deployment/temporal-proxy/index.mdx @@ -1,9 +1,7 @@ --- id: index title: Temporal Proxy -description: - Route requests between Temporal SDK clients and Workers and one or more upstream Temporal Services with the Temporal - Proxy, a gRPC proxy that handles namespace translation, TLS, authentication, and payload encryption. +description: Evaluate Temporal Proxy, understand its security and deployment model, and choose how to configure, deploy, and operate it. toc_max_heading_level: 4 tags: - Temporal Service @@ -14,46 +12,53 @@ tags: import { ReleaseNoteHeader } from '@site/src/components'; - Temporal Proxy is under active development and evolving quickly. Behavior and configuration can change between - releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status and - the definitive configuration schema. + Temporal Proxy is under active development and is not ready for production use. Behavior and configuration can change + between releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for current status + and the definitive configuration schema. -The Temporal Proxy is a gRPC proxy that sits between your Temporal SDK Clients, Workers, and the Temporal Web UI on one -side and one or more upstream Temporal Services on the other. It handles Namespace translation, TLS termination, and -optional payload encryption so your applications can target a single local endpoint while the proxy routes each request -to the right upstream, whether that is a local development Service, a self-hosted Service, or Temporal Cloud. +Temporal Proxy is a gRPC intermediary between your Temporal SDK Clients, Workers, CLI, or self-hosted Temporal Web UI +and one or more upstream Temporal Services. Applications connect to one gateway endpoint. The proxy can then route each +request, translate its Namespace, authenticate and authorize the caller, present upstream credentials, and encrypt +supported Payload fields. -## Why use it +Use these pages to evaluate the prerelease project, run it against Temporal Cloud, and understand the operational and +security work required before placing it in a request path. -Without the proxy, connection details leak into your application code. Every Worker and Client has to know the -upstream's host, TLS material, credentials, and the exact Namespace name the upstream expects. That couples your code to -an environment: moving between a local Service, a self-hosted deployment, and Temporal Cloud becomes a code change. +## Decide whether to use Temporal Proxy -The proxy owns that concern instead. Workers talk plaintext to a single local endpoint using a short Namespace name, and -the proxy adds TLS, credentials, and Namespace translation on the way out. Point a Worker at a different Namespace and -it reaches a different upstream, with no change to the Worker. +Temporal Proxy centralizes connection and data-handling concerns that would otherwise live in every application. It is +most useful when you need one or more of the following outcomes: -## How it works +| Goal | What the proxy does | Boundary to plan for | +| --- | --- | --- | +| Keep environment-specific connection details out of application configuration | Adds upstream TLS, credentials, and Namespace translation | Applications can still bypass the proxy unless you restrict their upstream credentials and network paths | +| Route applications to more than one Temporal Service | Selects an upstream by local Namespace and request metadata | The proxy becomes part of the request path and needs its own availability and capacity plan | +| Apply a common inbound access policy | Allows selected gRPC services and supports built-in or external authentication and authorization | The policy covers calls that enter the gateway, not direct calls to an upstream | +| Encrypt Payloads before they reach the Temporal Service | Applies envelope encryption to codec-capable Payload fields | Search Attributes and other non-Payload fields remain visible; existing plaintext Payloads remain readable during migration | -The proxy is built from a gateway and one proxy per upstream, connected by unix sockets: +If one of these controls must be mandatory, treat the proxy as one layer in a larger security boundary. Restrict direct +access to every upstream in addition to configuring the proxy. See +[Plan a Temporal Proxy deployment](/production-deployment/temporal-proxy/plan-deployment) before evaluating it with +production-shaped traffic. -- The **gateway** is the single inbound endpoint that every Worker, SDK Client, and the Web UI connects to. -- Each **upstream** has its own proxy that handles communication with that destination. +## Understand request handling + +The proxy consists of one gateway and one per-upstream proxy server, connected by Unix sockets: ```mermaid flowchart LR Worker[Worker] Client[SDK Client] - UI[Web UI] + UI[Self-hosted Web UI or CLI] subgraph Proxy[Temporal Proxy] direction LR - Gateway["Gateway
routes by Namespace
codec-transparent (no payload parsing)"] - ProxyA["Per-upstream proxy A
Namespace translation
payload encryption (optional)"] - ProxyB["Per-upstream proxy B
Namespace translation
payload encryption (optional)"] - Gateway -->|unix socket| ProxyA - Gateway -->|unix socket| ProxyB + Gateway["Gateway
admit and route calls"] + ProxyA["Upstream proxy A
translate and transform"] + ProxyB["Upstream proxy B
translate and transform"] + Gateway --> ProxyA + Gateway --> ProxyB end Cloud[Temporal Cloud] @@ -68,50 +73,52 @@ flowchart LR For each request, the gateway: -1. refuses the call if its gRPC service is not one the configuration allows, before any upstream work. By default that - allows `WorkflowService` and `OperatorService` and nothing else. -2. peeks the target Namespace without parsing the payload; it is codec-transparent and relays raw frames in both - directions. -3. picks an upstream: the first matching routing rule, otherwise the system upstream for Namespace-less calls, otherwise - the default. -4. hands the request to that upstream's proxy over a unix socket. - -The per-upstream proxy then rewrites the local Namespace to the name the upstream expects, attaches that upstream's TLS -and credentials, forwards to the Temporal Service, and translates the Namespace back on responses. When payload -encryption is enabled, it also seals codec-capable Payloads on the way out and opens them on the way back. New Payloads -sent through the proxy reach the upstream as ciphertext. See -[Encrypt payloads](/production-deployment/temporal-proxy/encrypt-payloads) for the scope and limits of this fail-closed -behavior. - -### Terms - -| Term | Meaning | -| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| gateway | The single inbound gRPC endpoint that every SDK Client, Worker, and the Web UI connects to. It routes each request to an upstream by Namespace and request metadata, and never parses payloads. | -| upstream | A configured destination the proxy forwards to: a Temporal Service (local dev, self-hosted, or Temporal Cloud), or another Temporal Proxy. | -| system upstream | The upstream that handles Namespace-less requests, such as the SDK's `GetSystemInfo` call on connect. | -| extension server | A gRPC service you run that the proxy calls out to for a capability it has no built-in backend for: wrapping data encryption keys, or deciding whether an inbound call may proceed. | -| Temporal Service | A Temporal frontend the proxy connects to. | - -## Prerequisites - -- One or more upstream Temporal Services to route to, such as a local development Service, a self-hosted Service, or - Temporal Cloud. -- The `hostPort` address for each upstream. -- Any credentials the upstreams require, such as a Temporal Cloud API key or mTLS certificates. -- Go installed, if you build the proxy from source. The container image and Helm chart do not require a local Go - toolchain. +1. Reads the target method and Namespace without decoding Payloads. +2. Applies configured inbound authentication or authorization to that target. +3. Rejects the call if its gRPC service is not allowed. +4. Selects the first matching route, the system route for Namespace-less calls, or the default route. +5. Sends the call to the selected per-upstream proxy. + +The per-upstream proxy translates the local Namespace, resolves the destination, attaches TLS and credentials, and +forwards the call. When payload encryption is enabled, it also encrypts supported Payloads on outbound calls and +decrypts recognized encrypted Payloads on responses. See +[Encrypt Payloads](/production-deployment/temporal-proxy/encrypt-payloads) for the exact encryption coverage and +fail-closed behavior. + +## Choose a deployment model + +You can run the gateway as a shared service or place a proxy beside each application deployment: + +| Model | Prefer it when | Account for | +| --- | --- | --- | +| Shared service | Several application deployments need the same routing and policy | Independent scaling, a larger failure domain, and securing the application-to-proxy network path | +| Sidecar or local proxy | You want a local endpoint and failure isolation per application deployment | More proxy instances, configuration distribution, and coordinated application rollouts | + +Both models require enough instances, capacity, and regional placement for the availability of the upstreams they +serve. [Plan a Temporal Proxy deployment](/production-deployment/temporal-proxy/plan-deployment) covers topology, +trust boundaries, bypass prevention, and high availability. + +## Choose your next step + +| If you want to | Start here | +| --- | --- | +| Run the checked-in example against Temporal Cloud | [Get started with Temporal Cloud](/production-deployment/temporal-proxy/get-started-cloud) | +| Decide where the proxy belongs and which traffic it must cover | [Plan a Temporal Proxy deployment](/production-deployment/temporal-proxy/plan-deployment) | +| Build a complete configuration | [Configure Temporal Proxy](/production-deployment/temporal-proxy/configure) | +| Route traffic and translate Namespace names | [Route requests and translate Namespaces](/production-deployment/temporal-proxy/route-translate) | +| Configure TLS, authentication, authorization, and credentials | [Secure connections and authorize requests](/production-deployment/temporal-proxy/secure-connections) | +| Encrypt supported Payload fields | [Encrypt Payloads](/production-deployment/temporal-proxy/encrypt-payloads) | +| Rotate or retire encryption keys | [Manage encryption keys](/production-deployment/temporal-proxy/manage-encryption-keys) | +| Install the Helm chart | [Deploy Temporal Proxy to Kubernetes](/production-deployment/temporal-proxy/deploy-kubernetes) | +| Monitor or diagnose a deployment | [Operate and troubleshoot Temporal Proxy](/production-deployment/temporal-proxy/operate-troubleshoot) | +| Look up flags, configuration sections, metrics, or compatibility | [Temporal Proxy reference](/production-deployment/temporal-proxy/reference) | ## Install the proxy -Install the `proxy` binary with Go: - -```bash -go install github.com/temporalio/temporal-proxy/cmd/proxy@latest -``` +Choose an installation method that matches your evaluation environment. Pin a release instead of using a moving tag +when you need repeatable behavior. -Pin an explicit version instead of `@latest`, using a tag from the -[releases page](https://github.com/temporalio/temporal-proxy/releases): +Install the `proxy` binary with Go: ```bash go install github.com/temporalio/temporal-proxy/cmd/proxy@vX.Y.Z @@ -120,86 +127,34 @@ go install github.com/temporalio/temporal-proxy/cmd/proxy@vX.Y.Z Pull the container image: ```bash -docker pull temporalio/temporal-proxy:latest +docker pull temporalio/temporal-proxy:vX.Y.Z ``` -Install with Helm from the Temporal Helm repo, optionally pinning a chart version with `--version`: +Install the Helm chart from the Temporal chart repository: ```bash helm install temporal-proxy temporal-proxy \ - --repo https://go.temporal.io/helm-charts + --repo https://go.temporal.io/helm-charts \ + --version ``` -Each chart release deploys a proxy version by default. Override it with `--set image.tag=vX.Y.Z`. Supply the proxy -configuration under the `config` key of a Helm values file, as described in -[Deploy to Kubernetes](/production-deployment/temporal-proxy/deploy-kubernetes). - -Run the proxy with a configuration file passed through the `-c` (or `--config`) flag: +See the [Temporal Proxy releases](https://github.com/temporalio/temporal-proxy/releases) and +[Helm chart releases](https://github.com/temporalio/helm-charts/releases) for available versions. The +[reference page](/production-deployment/temporal-proxy/reference) lists runtime flags and configuration sections. -```bash -proxy serve -c config.yaml -``` +## Review terminology -`--config` also reads the `PROXY_CONFIG` environment variable, which is how the Helm chart points the proxy at its -mounted configuration. See [Observability](#observability) for the remaining flags. - -## Observability {/* #observability */} - -The proxy serves Prometheus metrics at `/metrics` on `:9090` and logs JSON to stderr. The gateway also serves the -standard gRPC health service, with an entry for every allowed service, which is what a Client health check or a liveness -probe reads. See [Health checks](/production-deployment/temporal-proxy/configure#health-checks) for what it reports. - -Both the metrics listener and the log level are set with flags on `proxy serve`, each with an environment variable -equivalent: - -| Flag | Environment variable | Default | Sets | -| --------------------- | -------------------- | ------------- | -------------------------------------------- | -| `--config`, `-c` | `PROXY_CONFIG` | none | Path to the configuration file. Required. | -| `--level` | `LOG_LEVEL` | `info` | Log level: `debug`, `info`, `warn`, `error`. | -| `--metrics-addr` | `METRICS_ADDR` | `:9090` | The `host:port` serving `/metrics`. | -| `--metrics-namespace` | `METRICS_NAMESPACE` | `tmprl_proxy` | Prometheus namespace prefixed onto metrics. | - -Metric names are `__`, so with the default namespace the routing counter is -`tmprl_proxy_router_decisions_total`. There are three subsystems: - -| Subsystem | Metric | Labels | Reports | -| ------------ | -------------------------- | ---------------------------------- | ------------------------------------------------- | -| `server` | `requests_total` | `method`, `code` | RPCs served, by gRPC status code | -| `server` | `request_duration_seconds` | `method` | End-to-end time serving an RPC | -| `router` | `decisions_total` | `upstream`, `outcome` | Routing decisions, by chosen upstream | -| `router` | `forwarding_errors_total` | `upstream`, `reason` | Forwarding failures the router originated | -| `encryption` | `vault_ops_total` | `operation`, `result`, `namespace` | Envelope operations, sealing and opening payloads | -| `encryption` | `vault_ops_duration_secs` | `operation`, `namespace` | Time per envelope operation, end to end | -| `encryption` | `dek_ops_total` | `operation`, `result` | Outcome of the AES-256-GCM step alone | -| `encryption` | `dek_ops_duration_secs` | `operation` | Time in the AES-256-GCM step alone | -| `encryption` | `kek_ops_total` | `provider`, `operation`, `result` | DEK wrap and unwrap calls to your KMS | -| `encryption` | `kek_ops_duration_secs` | `provider`, `operation` | Time spent wrapping and unwrapping DEKs | -| `encryption` | `dek_rotations_total` | `reason` | DEK rotations, by why the DEK was replaced | -| `encryption` | `dek_cache_hits_total` | none | Reads served from the decrypted-DEK cache | -| `encryption` | `dek_cache_misses_total` | none | Reads that required a KMS unwrap | -| `encryption` | `dek_cache_size` | none | Current entries in the decrypted-DEK cache | - -The `encryption` metrics only move when [payload encryption](/production-deployment/temporal-proxy/encrypt-payloads) is -configured. They are layered, so pick the one that matches the question you are asking: - -- `vault_ops_*` is the whole envelope operation end to end, including any KEK call and cache lookup, and is the pair to - alert on. It carries the local Namespace. -- `dek_ops_*` is the symmetric AES-256-GCM step by itself, with the KEK work excluded. Its `result` is that step's own - outcome, so a payload that encrypts cleanly and then fails to wrap its DEK counts as a success here and an error under - `kek_ops_total`, which keeps the blame with the KMS. -- `kek_ops_*` is the calls to your KMS. Watch `kek_ops_total{result="error"}`, since a failure to wrap or unwrap a DEK - fails the request that needed it. - -`dek_rotations_total` splits by `reason`: `initial` for a Namespace's first DEK, `scheduled` for the `renewBefore` -pre-rotation, and `on_demand` for a DEK replaced at request time because no fresh one was ready. A rising `on_demand` -rate means rotation is falling behind, so raise `renewBefore`. Compare the cache counters against `cacheSize` to see -whether the cache is absorbing read traffic. +| Term | Meaning | +| --- | --- | +| gateway | The single inbound gRPC endpoint. It admits and routes requests without decoding Payloads. | +| upstream | A configured Temporal Service or another Temporal Proxy that receives forwarded calls. | +| system upstream | The upstream for Namespace-less calls such as `GetSystemInfo`. | +| per-upstream proxy | The internal server that applies translation, credentials, TLS, and optional payload encryption for one upstream definition. | +| extension server | A gRPC service you operate that wraps encryption keys or makes inbound authorization decisions. | ## Related - [Temporal Proxy repository](https://github.com/temporalio/temporal-proxy) - [Temporal Proxy Helm chart](https://github.com/temporalio/helm-charts/tree/main/charts/temporal-proxy) -- [Temporal Cloud example](https://github.com/temporalio/temporal-proxy/tree/main/examples/cloud) -- [KMS extension server example](https://github.com/temporalio/temporal-proxy/tree/main/examples/kms) - [Codecs and Encryption](/production-deployment/data-encryption) - [Self-hosted guide: Security](/self-hosted-guide/security) diff --git a/docs/production-deployment/temporal-proxy/manage-encryption-keys.mdx b/docs/production-deployment/temporal-proxy/manage-encryption-keys.mdx new file mode 100644 index 0000000000..13afa860ac --- /dev/null +++ b/docs/production-deployment/temporal-proxy/manage-encryption-keys.mdx @@ -0,0 +1,174 @@ +--- +id: manage-encryption-keys +title: Manage Temporal Proxy encryption keys +sidebar_label: Manage encryption keys +description: Temporal Proxy key lifecycle guidance covers staged rotation, decrypt-only transitions, multi-region access, and safe retirement. +toc_max_heading_level: 4 +tags: + - Temporal Service + - Self-hosting + - Security +--- + +import { ReleaseNoteHeader } from '@site/src/components'; + + + Temporal Proxy is under active development and is not ready for production use. Behavior and configuration can change + between releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for current status + and the definitive configuration schema. + + +Temporal Proxy uses short-lived data encryption keys (DEKs) to encrypt Payloads and a key encryption key (KEK) in your +KMS to wrap each DEK. The proxy rotates DEKs automatically. You manage the longer-lived KMS keys and must keep them +available for as long as encrypted Payloads refer to them. + +Before managing keys, [configure payload encryption](/production-deployment/temporal-proxy/encrypt-payloads) and grant +the proxy access to the selected AWS KMS, Azure Key Vault, Google Cloud KMS, or extension-server key. + +## Distinguish DEK rotation from KMS key rotation + +Each `default` or Namespace-specific `overrides` policy has the following lifecycle fields: + +| Field | Purpose | +| ------------- | ------------------------------------------------------------------------------------------------ | +| `uri` | Active KMS key used to wrap new DEKs. The key URI is recorded with every encrypted Payload. | +| `decryptURIs` | Earlier or staged KMS keys that the proxy can use only to unwrap existing DEKs. | +| `duration` | Time for which the proxy uses a DEK to encrypt new Payloads. | +| `renewBefore` | Lead time before `duration` expires when the proxy prepares a replacement DEK. | + +The proxy rotates a DEK when it reaches its renewal threshold. A DEK rotation does not rotate or create a KMS key. The +wrapped DEK and KMS key URI stored with each Payload let the proxy decrypt older Payloads after later DEK rotations. + +Use the encryption metrics to verify this lifecycle: + +- `dek_rotations_total{reason="scheduled"}` counts DEKs prepared by the background refresh. +- `dek_rotations_total{reason="on_demand"}` counts DEKs replaced on a request because a fresh DEK was not ready. +- `kek_ops_total` and `kek_ops_duration_secs` report the KMS operations that wrap and unwrap DEKs. + +A sustained increase in `on_demand` rotations indicates that refresh is falling behind. Increase `renewBefore`, while +keeping it less than `duration`, and check KMS latency and errors. See the +[metrics reference](/production-deployment/temporal-proxy/reference#inspect-prometheus-metrics) for the complete metric +list. + +## Change the active KMS key + +Treat a key change as a compatibility rollout. The key URI is part of the encrypted Payload metadata, so every proxy +instance that might read a Payload must recognize the URI that wrote it. The proxy reads its configuration at startup; +restart each instance after every configuration change. + +For a rolling deployment with more than one proxy instance, use two stages. + +First, register the new key for decryption while the old key remains active. This makes every updated instance recognize +both key URIs before any instance starts writing with the new key: + +```yaml +encryption: + enabled: true + default: + uri: gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v1 + decryptURIs: + - gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v2 + duration: 1h + renewBefore: 15m +``` + +Wait for every instance to run this configuration. Then promote the new key and retain the old key for decryption: + +```yaml +encryption: + enabled: true + default: + uri: gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v2 + decryptURIs: + - gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v1 + duration: 1h + renewBefore: 15m +``` + +Keep any keys from earlier migrations in `decryptURIs` during both stages. Repeat the process for every policy under +`overrides`; those map local, pre-translation Namespace names to key policies. + +After the second rollout, start a Workflow through each proxy instance and read both newly written and older Workflow +Histories through each instance. Watch `vault_ops_total{result="error"}` and `kek_ops_total{result="error"}` during the +test. Do not remove the old key because the rollout succeeds; retain it until no encrypted Payload refers to it. + +## Stop encrypting new Payloads + +To stop encrypting new outbound Payloads while keeping existing encrypted data readable, set `enabled: false` and keep +the key policies in the configuration: + +```yaml +encryption: + enabled: false + default: + uri: gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v2 + decryptURIs: + - gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v1 + duration: 1h + renewBefore: 15m +``` + +With this configuration, the proxy stops encrypting new outbound Payloads but continues decrypting Payloads marked with +its encryption metadata. Removing `default` removes the vault and prevents that decryption path from being created. + +This change does not rewrite data already stored by the upstream Temporal Service. Existing encrypted Payloads remain +encrypted at rest and require their original KMS keys whenever they are returned through the proxy. + +## Plan key access across regions + +Temporal Proxy does not replicate KMS keys or configuration. Every proxy instance that can receive traffic must be able +to unwrap every DEK that another instance can produce. This requirement applies during normal load balancing, regional +failover, rolling key changes, and rollbacks. + +Use these checks when the proxy runs in more than one region: + +1. Give every instance the same set of active and decrypt-only key URIs for the traffic it can serve. +2. Grant each instance's runtime identity decrypt access to every configured key URI. +3. Choose a KMS topology that stays reachable during the region failures your deployment is designed to tolerate. +4. Test reading data written in every region after removing network access to one region's KMS endpoint. + +The configured URI is the key identifier stored with an encrypted Payload. If two URIs address equivalent backing key +material, the proxy still treats them as different identifiers. Preserve the original URI in `decryptURIs` unless all +readers use the same stable URI. + +Provider choices differ: + +- **AWS KMS:** Use an AWS KMS multi-Region key or another design that lets every active region decrypt DEKs written in + every other region. Keep the configured URI stable across instances when possible, and test that the URI resolves to + usable key material after failover. See [AWS KMS multi-Region keys](https://docs.aws.amazon.com/kms/latest/developerguide/multi-region-keys-overview.html). +- **Azure Key Vault:** Place and replicate vaults according to your regional recovery design. Register every URI that + can appear in stored Payloads and grant each proxy identity access to all of them. See + [Azure Key Vault availability and redundancy](https://learn.microsoft.com/en-us/azure/key-vault/general/disaster-recovery-guidance). +- **Google Cloud KMS:** Use a global or multi-region key location when the same key must be reachable from several + regions, or ensure every instance can reach the configured regional location. See + [Cloud KMS locations](https://cloud.google.com/kms/docs/locations). +- **Custom KMS extension:** Keep the `extension:///` URI stable and make the extension service as available + as the proxy. Its `Decrypt` request contains the wrapped DEK but no Namespace, so the ciphertext returned by `Encrypt` + must carry enough information for the extension to select the correct key and version. + +## Retire an old KMS key + +Retention is not a sufficient retirement signal by itself. Open Workflow Executions can keep Event Histories beyond the +Namespace Retention Period, and archived or exported histories can outlive the source Namespace. A key is still needed +if any system from which data can be read contains a Payload that records its URI. + +Before removing a key from `decryptURIs`: + +1. Confirm that no active policy uses the key as `uri`. +2. Inventory open and closed Workflow Executions, Archival, exports, backups, and copied histories that might contain + Payloads encrypted with the key. +3. Exercise representative reads, replays, resets, and operational tooling through a proxy configured with the remaining + keys. +4. Remove the old URI from a canary instance and verify that it can read the same data before rolling out the removal. +5. Revoke access before scheduling deletion. Revocation is reversible if a missed Payload still needs the key. + +Keep KMS audit logs and the proxy's decrypt metrics during the retirement window. A decrypt failure after removing a URI +means that the original key must be restored to `decryptURIs` and made accessible again. + +:::caution Deleting a key is irreversible + +Deleting a KMS key permanently makes every Payload encrypted under it unreadable. Neither Temporal nor the KMS provider +can recover the plaintext. Keep the key in a recoverable, disabled state until your data inventory and recovery policy +prove that it is no longer required. + +::: diff --git a/docs/production-deployment/temporal-proxy/operate-troubleshoot.mdx b/docs/production-deployment/temporal-proxy/operate-troubleshoot.mdx new file mode 100644 index 0000000000..fe9cf2f263 --- /dev/null +++ b/docs/production-deployment/temporal-proxy/operate-troubleshoot.mdx @@ -0,0 +1,158 @@ +--- +id: operate-troubleshoot +title: Operate and troubleshoot Temporal Proxy +sidebar_label: Operate and troubleshoot +description: Monitor Temporal Proxy health, traffic, encryption, and shutdown behavior, and resolve common routing and connectivity failures. +toc_max_heading_level: 4 +tags: + - Temporal Service + - Self-hosting + - Security +--- + +import { ReleaseNoteHeader } from '@site/src/components'; + + + Temporal Proxy is under active development and is not ready for production use. Behavior and configuration can change + between releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for current status + and the definitive configuration schema. + + +Operate Temporal Proxy as part of the request path between applications and an upstream Temporal Service. Monitor the +gateway, each intended upstream path, and any KMS or extension server that a request depends on. The gateway health check +reports process state; request metrics and end-to-end tests cover the dependencies beyond the process. + +## Check process health + +The gateway implements the standard gRPC health service. It publishes the empty service name and one entry for each +service in `allowedServices`. All entries report the same process-wide state: + +- `SERVING` means the gateway process is accepting requests. +- `NOT_SERVING` is published when shutdown starts. +- `NOT_FOUND` means the health request named a service that is not in `allowedServices`. + +The health status does not continuously test upstream Temporal Services, KMS providers, or extension servers. Static +upstream addresses are connected before gateway startup, so an unreachable static upstream prevents startup. Templated +upstreams are resolved and connected on first use. After startup, the gateway can still report `SERVING` while one of +these dependencies is unavailable. + +Use the gRPC health check as a process liveness signal. For readiness, add a test that exercises the upstream route and +other dependencies required for the traffic that instance will receive. If the gateway terminates TLS or mTLS, the +health client must use matching transport credentials. + +For Kubernetes probe configuration, including the limitation of plaintext native gRPC probes, see +[Deploy Temporal Proxy to Kubernetes](/production-deployment/temporal-proxy/deploy-kubernetes#probe-the-gateway). + +## Monitor traffic and dependencies + +The proxy writes structured JSON logs to stderr. Set the log level with `--level` or `LOG_LEVEL`; the default is `info`. +Use `debug` temporarily to record routing decisions and resolved templated upstream targets: + +```bash +proxy serve --config config.yaml --level debug +``` + +Do not publish a configuration file, debug logs, or environment values without removing credentials and internal +addresses. A `testing://` encryption URI contains local test key material and must also be redacted. + +The proxy exposes Prometheus metrics at `/metrics`. The default address is `:9090`, and the default metric namespace is +`tmprl_proxy`; configure both under the top-level `metrics` block. Use the following signals for an initial dashboard: + +The metrics listener uses plaintext HTTP and has no authentication. Bind it to a private interface or restrict access +with your network and deployment controls. + +| Signal | What to check | +| ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | +| `server_requests_total` by `code` | Changes in non-`OK` gRPC results and the affected methods. | +| `server_request_duration_seconds` | End-to-end latency, including long-poll methods. | +| `router_decisions_total{outcome="unroutable"}` | Requests for which no route, system upstream, or default upstream was selected. | +| `router_forwarding_errors_total` | Missing connections or failures while creating an upstream stream. | +| `encryption_vault_ops_total{result="error"}` | Payload encrypt or decrypt failures. | +| `encryption_kek_ops_total{result="error"}` | KMS wrap or unwrap failures. | +| `encryption_dek_rotations_total{reason="on_demand"}` | Key refresh falling behind the configured renewal window. | + +Prefix each name in the table with the configured metrics namespace. For example, the default full request counter is +`tmprl_proxy_server_requests_total`. See the +[Temporal Proxy reference](/production-deployment/temporal-proxy/reference#inspect-prometheus-metrics) for all metric +names and labels. + +## Shut down the proxy + +On a termination signal, the proxy marks its gRPC health service `NOT_SERVING`, stops admitting new gateway requests, +and drains in-flight RPCs. The gateway drains before the per-upstream proxy tiers. Each serving tier has a five-second +drain budget, and the complete lifecycle stop is bounded at 30 seconds. + +Long-poll calls can outlast the drain budget. When the deadline expires, the proxy drops the remaining calls and logs +`Drain ended with calls in flight. Dropping them` as a warning. Temporal Clients and Workers re-poll, so this warning is +expected during some restarts. + +Give the process supervisor more than 30 seconds between its termination signal and a forced kill. This leaves time for +the proxy lifecycle hooks and KMS connections to close after the request drain. See +[Deploy Temporal Proxy to Kubernetes](/production-deployment/temporal-proxy/deploy-kubernetes#shut-down-cleanly) for +the Kubernetes setting. + +## Confirm that clients cannot bypass the proxy + +Routing Clients and Workers through the proxy does not stop them from connecting directly to the upstream Temporal +Service. A direct connection bypasses the proxy's inbound authentication, Namespace translation, outbound credential +handling, and payload encryption. + +When the proxy is the required access path: + +1. Configure all Clients, Workers, CLI profiles, and self-hosted Web UI instances to use the gateway endpoint. +2. Keep upstream API keys and client certificates out of application deployments. Give them only to the proxy runtime. +3. Restrict the upstream network path to the proxy's egress addresses, private network, or workload identity boundary. +4. From an application network and identity, verify that a direct upstream health or read request is rejected while the + same request through the proxy succeeds. +5. Compare expected application request volume with proxy metrics. Use this as a drift signal, not proof of enforcement; + requests that bypass the proxy produce no proxy metric. + +Repeat the direct-access check after credential, network, deployment, or Namespace changes. See +[Plan a Temporal Proxy deployment](/production-deployment/temporal-proxy/plan-deployment) for trust boundaries and +[Encrypt payloads](/production-deployment/temporal-proxy/encrypt-payloads#understand-fail-closed-behavior) for encryption +coverage. + +## Resolve startup failures + +The proxy validates its configuration before accepting gateway traffic. Configuration errors include the YAML path of +the invalid field when possible. Check these causes first: + +| Symptom or message | Action | +| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | +| The `--config` flag is required | Pass `--config ` or set `PROXY_CONFIG`. | +| `invalid configuration` | Correct the named field, duplicate name, or reference to an unknown upstream or extension server. | +| A certificate file cannot be read or verified | Check the `tls.ca`, `tls.cert`, and `tls.key` paths, PEM contents, expiry, and whether the certificate and key match. | +| `upstream connection not ready` | Check DNS, network policy, the static upstream `hostPort`, and its TLS configuration. | +| A listener address is already in use | Check gateway `hostPort`, `metrics.hostPort`, and other proxy processes on the host. | +| A KMS key cannot be registered | Check the key URI, provider credential chain, extension-server reference, and encrypt/decrypt permissions. | + +A static upstream is checked during startup. A templated `hostPort` is not, because its value depends on the request +Namespace or metadata. Test every templated route after the gateway starts. + +## Resolve request failures + +Start with the gRPC status and message returned to the caller, then correlate it with logs and metrics from the same +time window: + +| Status or symptom | Likely cause and action | +| ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `FAILED_PRECONDITION: no upstream matched...` | No routing rule matched and the applicable `routing.default` or `routing.system` value is absent. Add the intended fallback or matching rule. | +| `UNAVAILABLE` while forwarding to the selected upstream | The selected upstream has no usable connection. Check its address, TLS, credentials, DNS, and network path. | +| `UNIMPLEMENTED: unknown service...` | The service is outside `allowedServices`. Add a forwardable service only if callers should use it. | +| `UNAUTHENTICATED` from the gateway | The inbound static token or JWT is missing, malformed, expired, or does not match the configured issuer or audience. | +| `PERMISSION_DENIED` from external authorization | The extension server denied the resolved method or Namespace. Use the server-side reason in proxy logs; the caller does not receive that internal detail. | +| A templated upstream returns `INTERNAL` before dialing | The template failed or rendered an empty or invalid `hostPort` or `serverName`. Enable debug logs and inspect the local and translated Namespace. | +| `vault_ops_total{result="error"}` increases | The proxy could not encrypt or decrypt a Payload. Check the matching KMS metric, key URI, key access, and extension-server health. | +| Requests fail while the gRPC health check remains `SERVING` | A dependency failed after startup. Test the selected upstream, KMS, and extension service independently. | +| Some upstream Payloads remain plaintext | Check for direct-client bypass, data written before encryption was enabled, and fields outside Payload Codec coverage. | + +For routing and Namespace translation details, see +[Route and translate Namespaces](/production-deployment/temporal-proxy/route-translate). For gateway and upstream TLS +and credentials, see +[Secure connections and authorize requests](/production-deployment/temporal-proxy/secure-connections). Encryption +fails the request instead of forwarding a codec-capable Payload as plaintext; see +[Encrypt payloads](/production-deployment/temporal-proxy/encrypt-payloads). + +When opening an issue, include the output of `proxy --version`, the gRPC status and method, the relevant local Namespace +and selected upstream name, sanitized configuration, and a narrow log and metrics window. Do not include credentials, +private keys, JWTs, or Payload contents. diff --git a/docs/production-deployment/temporal-proxy/plan-deployment.mdx b/docs/production-deployment/temporal-proxy/plan-deployment.mdx new file mode 100644 index 0000000000..0be9362581 --- /dev/null +++ b/docs/production-deployment/temporal-proxy/plan-deployment.mdx @@ -0,0 +1,136 @@ +--- +id: plan-deployment +title: Plan a Temporal Proxy deployment +sidebar_label: Plan a deployment +description: + Choose a Temporal Proxy topology and plan traffic coverage, trust boundaries, availability, capacity, and operations + before deployment. +toc_max_heading_level: 4 +tags: + - Temporal Service + - Self-hosting + - Security +--- + +import { ReleaseNoteHeader } from '@site/src/components'; + + + Temporal Proxy is under active development and is not ready for production use. Behavior and configuration can change + between releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for current status + and the definitive configuration schema. + + +Plan where the proxy runs, which callers must use it, and how those callers keep access to the Temporal Service when a +proxy instance fails. Temporal Proxy becomes an additional gRPC hop for every connection routed through it. Use this +guidance for evaluation and preproduction design while the project remains prerelease, and check the repository's +current release status before production use. + +## Determine traffic coverage and trust boundaries + +List every Worker, SDK Client, CLI process, and self-hosted Web UI that connects to each upstream Temporal Service. For +each caller, decide whether the proxy provides convenience, such as Namespace translation, or enforces a required +control, such as payload encryption or inbound authorization. + +Proxy controls apply only to requests that pass through the proxy. A caller that retains an upstream endpoint and valid +upstream credentials can bypass its routing, authorization, and payload encryption. When a control is mandatory: + +- Give upstream credentials to the proxy rather than to application processes. +- Restrict network paths so application workloads can reach the gateway but cannot connect directly to the upstream. +- Configure every in-scope caller to use the gateway endpoint. +- Protect the gateway with network policy, inbound TLS or mTLS, and authentication appropriate to its callers. + +Plan the controls on each connection separately: + +| Connection | Data that crosses it | Controls to plan | +| ---------- | -------------------- | ---------------- | +| Application to gateway | Temporal gRPC requests and cleartext Payloads when proxy encryption is enabled | Network reachability, gateway TLS or mTLS, and inbound authentication and authorization | +| Proxy to upstream | Forwarded gRPC requests, upstream credentials, and optionally encrypted Payloads | Outbound TLS, upstream credential scope, routing, and Namespace translation | +| Proxy to KMS or extension server | Data encryption keys or authorization requests when those features are configured | Private reachability, TLS, service credentials, latency, and dependency availability | + +Payload encryption fails the outbound request when the proxy cannot encrypt a codec-capable Payload, but it does not +cover a direct connection to the upstream or every request field. Review the scope of +[fail-closed encryption](/production-deployment/temporal-proxy/encrypt-payloads#understand-fail-closed-behavior) before treating the proxy as a data +boundary. + +## Choose a deployment topology + +Use a shared service when several application deployments should use one gateway and configuration. Use a sidecar when +each Worker deployment should own its proxy instance and local connection. Both shapes run the same proxy binary, and +you can use different shapes for different workloads. + +| Consideration | Shared service | Sidecar with each Worker deployment | +| ------------- | -------------- | ----------------------------------- | +| Client endpoint | A stable network endpoint shared by many callers | A loopback or Pod-local endpoint | +| Configuration | Central rollout for all attached callers | Versioned and rolled out with each Worker deployment | +| Failure scope | A gateway outage can affect many workloads | A proxy outage is isolated to its Worker Pod | +| Scaling | Scale proxy replicas and the gateway path independently | Scale proxy instances with Worker replicas | +| Network security | Secure an application-to-gateway network hop | Keep the application-to-gateway hop inside the Pod | +| Other callers | SDK Clients, CLI processes, and a self-hosted Web UI can share the endpoint | Callers outside the Worker Pod need another proxy endpoint | +| Operational cost | Fewer instances and caches to operate | More instances, configuration copies, connections, and KMS caches | + +For a shared service, run more than one proxy replica behind an endpoint that supports long-lived gRPC connections. Use +the same routing, credentials, and encryption policies on every replica so a caller sees the same behavior after it +reconnects. For a sidecar, treat the Worker and proxy as one release unit and configure the Worker to connect or retry +only after the sidecar gateway is reachable. + +## Design for failures on the request path + +The proxy is on the request path. If callers cannot reach a healthy proxy instance, Workers cannot poll or complete +Tasks and Clients cannot send commands until connectivity returns. Existing Workflow state remains durable in the +Temporal Service, but Workflow progress can pause while the proxy path is unavailable. + +Design the failure behavior for the topology you selected: + +- Place shared replicas across the failure domains used by the attached applications, and maintain capacity while one + replica or domain is unavailable. +- Configure readiness, graceful termination, and disruption budgets so deployments remove an instance from service + before stopping it. Long-poll requests can reconnect to another healthy instance. +- Test proxy restarts, upstream connection failures, expired credentials, and invalid configuration before a cutover. +- Treat a KMS or external authorization service as part of the request path when you enable it. KMS cache hits reduce + calls to the KMS, but a request that needs a failed key operation does not fall back to plaintext. +- Keep configuration and secret rollouts reversible. The proxy reads its configuration at startup, so a change takes + effect when instances restart with the new version. + +If applications and a highly available upstream can move between regions, place a reachable proxy path in every region +where those applications can run. Every regional proxy that decrypts Payloads must be able to use the keys that other +regions used to encrypt them. See +[Manage encryption keys](/production-deployment/temporal-proxy/manage-encryption-keys) for multi-region key planning and +[Deploy to Kubernetes](/production-deployment/temporal-proxy/deploy-kubernetes) for replicas, probes, and shutdown +behavior. + +## Plan capacity and operations + +Size the proxy with a representative workload before a cutover. Include the number of Worker and Client connections, +concurrent long polls, request rate, Payload sizes, Namespace and upstream count, and the KMS operations generated by +cache misses and key rotation. No single request-per-second value represents every configuration. + +Establish an operating baseline and alert on changes in: + +- End-to-end request latency and gRPC error codes. +- Routing decisions and forwarding failures by upstream. +- Payload encryption failures, KMS errors and latency, and data encryption key cache misses. +- Proxy process availability, resource use, and replica count. + +The gateway's gRPC health service reports process-wide health. It does not replace a test that verifies the full path +through routing, upstream authentication, and any KMS or extension server. Use both health probes and a synthetic +request appropriate to your environment. For the available logs, metrics, and response checks, see +[Operate and troubleshoot Temporal Proxy](/production-deployment/temporal-proxy/operate-troubleshoot). + +Plan how you will rotate API keys, certificates, gateway tokens, and KMS permissions without updating application code. +Store credentials outside the proxy configuration file, limit access to them, and roll proxy instances after a change. + +## Complete the deployment checklist + +Use this checklist before directing an application workload through the proxy: + +- [ ] Record every caller, local Namespace, upstream Namespace, and upstream endpoint in scope. +- [ ] Decide which proxy controls are required and prevent in-scope callers from bypassing them. +- [ ] Choose a shared service, sidecar, or workload-specific combination and document its failure scope. +- [ ] Provide more than one proxy path wherever the application requires proxy fault tolerance. +- [ ] Configure inbound network controls, TLS, authentication, and the forwarded gRPC service allowlist. +- [ ] Scope upstream credentials to the Namespaces and actions each proxy deployment needs. +- [ ] Confirm that every replica has the same routing, translation, and encryption policies. +- [ ] Protect encryption keys from deletion and verify that every regional proxy can decrypt existing Payloads. +- [ ] Load test long polls, Payload sizes, KMS operations, and a replica or dependency failure. +- [ ] Configure health probes, metrics, logs, alerts, graceful shutdown, and a rollback procedure. +- [ ] Migrate callers in stages and verify the complete request path before removing direct upstream access. diff --git a/docs/production-deployment/temporal-proxy/reference.mdx b/docs/production-deployment/temporal-proxy/reference.mdx new file mode 100644 index 0000000000..9a46b78104 --- /dev/null +++ b/docs/production-deployment/temporal-proxy/reference.mdx @@ -0,0 +1,151 @@ +--- +id: reference +title: Temporal Proxy reference +sidebar_label: Reference +description: Look up Temporal Proxy release status, artifacts, command flags, configuration groups, forwarding scope, and Prometheus metrics. +toc_max_heading_level: 4 +tags: + - Temporal Service + - Self-hosting + - Security +--- + +import { ReleaseNoteHeader } from '@site/src/components'; + + + Temporal Proxy is under active development and is not ready for production use. Behavior and configuration can change + between releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for current status + and the definitive configuration schema. + + +Use this page to look up the current Temporal Proxy command interface, top-level configuration groups, supported gRPC +services, and proxy-defined Prometheus metrics. For configuration examples and operating guidance, start with +[Configure Temporal Proxy](/production-deployment/temporal-proxy/configure) and +[Operate and troubleshoot Temporal Proxy](/production-deployment/temporal-proxy/operate-troubleshoot). + +## Check release status and source + +Temporal Proxy is a prerelease project and does not yet provide a production-readiness or version-compatibility promise. +Configuration and behavior can change between releases. Pin the proxy version, test it with the exact SDK, CLI, +self-hosted Temporal Web UI, and upstream versions in your environment, and review changes before upgrading. + +The following sources define the current behavior: + +- [Releases](https://github.com/temporalio/temporal-proxy/releases) provide versioned release artifacts and notes. +- [`internal/config`](https://github.com/temporalio/temporal-proxy/tree/main/internal/config) defines and validates the + YAML configuration. +- [`cmd/proxy`](https://github.com/temporalio/temporal-proxy/tree/main/cmd/proxy) defines command-line flags and + environment-variable aliases. +- [The Helm chart](https://github.com/temporalio/helm-charts/tree/main/charts/temporal-proxy) defines Kubernetes-specific + values that render the proxy configuration and resources. + +Because links to `main` show unreleased source, use the repository tag matching your pinned version when verifying a +deployed binary. + +## Choose a distribution artifact + +| Artifact | Location | Versioning guidance | +| --------------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | +| Go binary | `github.com/temporalio/temporal-proxy/cmd/proxy` | Install a release tag with `go install ...@vX.Y.Z`. | +| Container image | [`temporalio/temporal-proxy`](https://hub.docker.com/r/temporalio/temporal-proxy) | Pin a version tag instead of relying on `latest`. | +| Helm chart | [Temporal Helm chart repository](https://github.com/temporalio/helm-charts/tree/main/charts/temporal-proxy) | Pin the chart version and review the proxy image tag selected by that chart release. | + +Run `proxy --version` to print the proxy version, build time, and Git commit SHA. + +## Use command flags + +The executable has one subcommand, `serve`: + +```bash +proxy serve --config config.yaml +``` + +| Flag | Alias | Environment variable | Default | Purpose | +| ---------------- | ----- | -------------------- | ------- | --------------------------------------------------- | +| `--config ` | `-c` | `PROXY_CONFIG` | None | Path to the required YAML configuration file. | +| `--level ` | None | `LOG_LEVEL` | `info` | Log level: `debug`, `info`, `warn`, or `error`. | + +Metrics are configured in the YAML file, not with command-line flags. + +## Use top-level configuration keys + +The proxy reads one YAML file and expands `$VAR` and `${VAR}` references from the process environment before parsing it. +An unset environment variable expands to an empty string, which can produce a validation error or an empty optional +value. Mount secrets through the environment or a secrets provider instead of committing credentials to the file. + +| Key | Required or default | Purpose | +| ------------------ | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | +| `hostPort` | Required when running the binary; the Helm chart can supply it from the Service port | Gateway `host:port` used by SDK Clients, Workers, CLI, and Web UI. | +| `tls` | Optional; plaintext when absent | TLS or mTLS material for inbound gateway connections. | +| `allowedServices` | Defaults to `WorkflowService` and `OperatorService` | Proto full names of gRPC services the gateway can forward. | +| `auth` | Optional; all requests admitted when absent | Selects exactly one inbound authenticator: `staticToken`, `jwks`, or `external`. | +| `encryption` | Optional; new Payload encryption disabled by default | Configures the DEK cache and default or per-Namespace KMS key policies. | +| `extensionServers` | Optional | Addresses external authorization or KMS gRPC services. | +| `metrics` | Defaults to `hostPort: :9090` and `namespace: tmprl_proxy` | Sets the HTTP metrics listener and the Prometheus metric-name prefix. | +| `routing` | No implicit upstream selection | Selects the default and system upstreams and ordered Namespace or metadata rules. | +| `upstreams` | At least one entry is required | Names upstream Temporal Services and configures addresses, TLS, credentials, and Namespace translation. | + +Each upstream and extension-server name must be unique, and each list must use unique `hostPort` values. Static upstreams +connect before the gateway starts. An upstream `hostPort` and TLS `serverName` can use request-time templates; +extension-server addresses cannot. + +For nested fields, validation rules, and defaults, consult the configuration source for the release you run and the +task-specific pages under [Configure Temporal Proxy](/production-deployment/temporal-proxy/configure). + +## Check forwarding scope and compatibility + +The proxy can forward the following gRPC services: + +| Service | Default | Used by | +| ------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------- | +| `temporal.api.workflowservice.v1.WorkflowService` | Yes | SDK Clients, Workers, the Temporal CLI, and self-hosted Temporal Web UI. | +| `temporal.api.operatorservice.v1.OperatorService` | Yes | `temporal operator` commands and self-hosted Web UI Search Attribute and Nexus Endpoint views. | +| `grpc.reflection.v1.ServerReflection` | No | Service discovery for tools such as `grpcurl`; also admits its `v1alpha` compatibility alias. | + +The `allowedServices` configuration can narrow this list but cannot add a service not compiled into the proxy. A call to +a service outside the allowlist returns `UNIMPLEMENTED` before upstream work begins. + +The gateway accepts Temporal gRPC traffic from SDK Clients, Workers, the Temporal CLI, and self-hosted Temporal Web UI. +It can route to local development, self-hosted, or Temporal Cloud upstream Services. This transport scope is not a +compatibility matrix. While the project is prerelease, verify each Client and upstream version combination with +representative Workflow, polling, Visibility, operator, and long-poll calls before deployment. + +## Inspect Prometheus metrics {/* #inspect-prometheus-metrics */} + +The proxy serves Prometheus metrics at `/metrics` on `metrics.hostPort`. Metric names use +`__`, where `metrics.namespace` defaults to `tmprl_proxy`. For example, +`router_decisions_total` is exposed as `tmprl_proxy_router_decisions_total` by default. + +The endpoint also includes collectors registered by the Go runtime and process. The following table lists metrics defined +by Temporal Proxy: + +| Subsystem | Metric | Labels | Reports | +| ------------ | -------------------------- | ---------------------------------- | ----------------------------------------------------------------------- | +| `server` | `requests_total` | `method`, `code` | Completed gateway RPCs by full gRPC method and status code. | +| `server` | `request_duration_seconds` | `method` | End-to-end RPC duration, with buckets extending through long polls. | +| `router` | `decisions_total` | `upstream`, `outcome` | Selected upstream and whether a rule, default, or system route chose it. | +| `router` | `forwarding_errors_total` | `upstream`, `reason` | Router-originated connection and stream-setup failures. | +| `encryption` | `vault_ops_total` | `operation`, `result`, `namespace` | Complete Payload encrypt and decrypt operations. | +| `encryption` | `vault_ops_duration_secs` | `operation`, `namespace` | End-to-end envelope-operation duration, including KMS and cache work. | +| `encryption` | `dek_ops_total` | `operation`, `result` | AES-256-GCM Payload operations and their own result. | +| `encryption` | `dek_ops_duration_secs` | `operation` | Time spent in the AES-256-GCM step. | +| `encryption` | `kek_ops_total` | `provider`, `operation`, `result` | KMS DEK wrap and unwrap operations. | +| `encryption` | `kek_ops_duration_secs` | `provider`, `operation` | Time spent wrapping and unwrapping DEKs. | +| `encryption` | `dek_rotations_total` | `reason` | DEK rotations by `initial`, `scheduled`, or `on_demand` reason. | +| `encryption` | `dek_cache_hits_total` | None | Reads served from the decrypted-DEK cache. | +| `encryption` | `dek_cache_misses_total` | None | Reads that required a KMS unwrap. | +| `encryption` | `dek_cache_size` | None | Current number of decrypted-DEK cache entries. | + +Encryption metrics are registered when a key policy creates a vault, including a decrypt-only configuration with +`encryption.enabled: false`. They move only when the corresponding key or Payload path is exercised. + +Label values include: + +- Router `outcome`: `match`, `default`, `system`, or `unroutable`. +- Router forwarding `reason`: `no_connection` or `stream_setup`. +- Encryption `operation`: `encrypt` or `decrypt` for vault and DEK metrics, and `wrap` or `unwrap` for KEK metrics. +- Operation `result`: `success` or `error`. +- DEK rotation `reason`: `initial`, `scheduled`, or `on_demand`. + +The `method` and `namespace` labels are not bounded by a fixed configuration list. Do not expose the gateway directly to +untrusted callers that can send arbitrary method names, and account for Namespace cardinality when storing metrics. diff --git a/docs/production-deployment/temporal-proxy/route-translate.mdx b/docs/production-deployment/temporal-proxy/route-translate.mdx new file mode 100644 index 0000000000..b7b42e183d --- /dev/null +++ b/docs/production-deployment/temporal-proxy/route-translate.mdx @@ -0,0 +1,187 @@ +--- +id: route-translate +title: Route requests and translate Namespaces +sidebar_label: Route and translate requests +description: Direct Temporal Proxy requests to upstream Temporal Services and map local Namespace names to each upstream's registered names. +toc_max_heading_level: 4 +tags: + - Temporal Service + - Self-hosting + - Security +--- + +import { ReleaseNoteHeader } from '@site/src/components'; + + + Temporal Proxy is under active development and is not ready for production use. Behavior and configuration can change + between releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for current status + and the definitive configuration schema. + + +Temporal Proxy can select an upstream Temporal Service for each request and present a different Namespace name to that +upstream. Applications connect to one gateway and continue to use local Namespace names. The proxy routes on the local +name, then translates it before forwarding the request. + +Start with [Configure Temporal Proxy](/production-deployment/temporal-proxy/configure) for the gateway listener and a +complete baseline configuration. + +## Define upstreams + +Each entry in `upstreams` describes one Temporal Service that the proxy can reach. Give every upstream a unique `name` +so that `routing` can refer to it, and a unique `hostPort` with the upstream's gRPC address. + +```yaml +upstreams: + - name: local + hostPort: temporal.internal.example.com:7233 + tls: + ca: /etc/temporal-proxy/tls/local-ca.pem + cert: /etc/temporal-proxy/tls/client.pem + key: /etc/temporal-proxy/tls/client-key.pem + + - name: cloud + hostPort: '{{ .RemoteNamespace }}.tmprl.cloud:7233' + tls: {} + namespaces: + rules: + suffix: .${TEMPORAL_ACCOUNT} + credentials: + static: + apiKey: ${TEMPORAL_API_KEY} +``` + +The proxy expands `$VAR` and `${VAR}` environment variables before parsing the YAML. It validates static addresses and +opens their connections at startup. If `hostPort` or `tls.serverName` contains a Go template action, the proxy resolves +the address for each request and connects on first use. + +Set `cloud: true` on an upstream when it connects to Temporal Cloud through an address and TLS server name that the +proxy cannot recognize as a Temporal Cloud endpoint. This enables Cloud-specific validation of translated Namespace +configuration, including suffix and override shapes. It does not reject every malformed Namespace produced after +request-time translation; verify translated Namespace names in your deployment tests. Addresses under `tmprl.cloud` are +detected automatically. + +For TLS and credentials on each upstream, see +[Secure Temporal Proxy connections](/production-deployment/temporal-proxy/secure-connections#secure-upstream-connections). + +## Route requests + +The `routing` block selects one upstream for each request. Rules are evaluated from top to bottom, and the first match +wins. When no rule matches, the proxy uses `system` for a Namespace-less request and `default` for any other request. + +```yaml +routing: + default: local + system: cloud + rules: + - match: + namespace: 'prod-*' + metadata: + x-tier: gold + upstream: cloud + - match: + namespace: '*-test' + upstream: local +``` + +- `default` is optional. Without it, an unmatched request fails with `FAILED_PRECONDITION`. +- `system` is optional. It handles calls that do not name a Namespace, such as `GetSystemInfo` and `GetClusterInfo`. + Without it, those calls use `default`. +- `rules` is optional. Each rule must contain `namespace`, `metadata`, or both, and must name a configured upstream. + +Rules run before the `system` and `default` fallbacks. A metadata-only rule can therefore match a Namespace-less +request. Selecting an upstream does not provide failover: if the selected upstream has no available connection, the +request fails instead of trying another rule or upstream. + +## Match Namespaces and metadata + +Routing matches the local Namespace name, before Namespace translation. A Namespace match supports a literal or one of +the following glob forms: + +| Pattern | Matches | +| ---------- | --------------------------- | +| `payments` | Exactly `payments` | +| `prod-*` | Names starting with `prod-` | +| `*-test` | Names ending with `-test` | +| `*-test-*` | Names containing `-test-` | +| `*` | Every Namespace | + +A wildcard in any other position, such as `prod-*-eu`, is invalid. + +Metadata matches gRPC request metadata. Keys are case-insensitive and do not support wildcards. Values support the +same glob forms as Namespace names. When a key has multiple values, the condition matches if any value matches. A rule +that specifies both Namespace and metadata matches only when every condition matches. + +:::caution Routing metadata is not authorization + +A caller that can connect to the gateway can set request metadata. Do not use a routing header by itself as a security +boundary. Authentication identifies the caller but does not make a caller-supplied header trustworthy. Use an extension +server to authorize the resolved request, or a trusted intermediary to set and sanitize the routing header. See +[Secure connections and authorize requests](/production-deployment/temporal-proxy/secure-connections). + +::: + +Routing rules cannot match the gRPC method, the translated Namespace name, or values inside a Payload. External +authorization can allow or deny a method for a caller, but it cannot select the upstream. Use separate gateways or +trusted routing metadata when the destination itself must vary by caller or method. + +## Translate Namespaces + +Namespace translation lets applications use a local name while an upstream receives its registered name. Rules belong +to an upstream because different Temporal Services can use different naming conventions. + +```yaml +upstreams: + - name: cloud + hostPort: '{{ .RemoteNamespace }}.tmprl.cloud:7233' + tls: {} + namespaces: + rules: + prefix: '' + suffix: .acct + overrides: + - local: billing + remote: payments.acct +``` + +- `prefix` and `suffix` wrap a local name on requests and are removed from the remote name on responses. In this + example, `orders` becomes `orders.acct`. +- `overrides` defines explicit `local` and `remote` pairs. An override takes precedence over `prefix` and `suffix`. + Each local name and each remote name can appear only once. + +For a Temporal Cloud upstream, a non-empty suffix must have the form `.`, and every override's remote name +must be a complete Cloud Namespace name. Translation does not register or rename a Namespace. The resulting remote +Namespace must already exist on the selected Temporal Service. + +The proxy translates recognized Namespace-name fields in Workflow Service and Operator Service protobuf messages, the +`temporal-namespace` request metadata header, and Namespace fields in typed gRPC error details. It does not rewrite +Namespace names embedded in Payload data, arbitrary metadata values, or free-text error messages. + +## Resolve templated upstreams + +Use a Go template in `hostPort` or `tls.serverName` when the address varies by Namespace or request metadata. These +values are available after the proxy selects an upstream and translates the Namespace: + +- `{{ .LocalNamespace }}` is the Namespace name received from the application. +- `{{ .RemoteNamespace }}` is the name after the selected upstream's translation rules run. +- `{{ .Metadata. }}` or `{{ index .Metadata "" }}` is the last value for a request metadata key. + +Use lowercase metadata keys in templates. gRPC normalizes metadata keys to lowercase before the proxy builds the +template context. + +For example, the following upstream directs each translated Cloud Namespace to its own endpoint: + +```yaml +upstreams: + - name: cloud + hostPort: '{{ .RemoteNamespace }}.tmprl.cloud:7233' + tls: {} + namespaces: + rules: + suffix: .${TEMPORAL_ACCOUNT} +``` + +An absent metadata key renders as an empty string. If that produces an empty or malformed `hostPort`, the request fails +without dialing an upstream. Template expressions are not supported in `extensionServers.hostPort`. + +For an end-to-end Cloud configuration that uses Namespace translation and a templated endpoint, see +[Get started with Temporal Cloud](/production-deployment/temporal-proxy/get-started-cloud). diff --git a/docs/production-deployment/temporal-proxy/secure-connections.mdx b/docs/production-deployment/temporal-proxy/secure-connections.mdx new file mode 100644 index 0000000000..269acf3e76 --- /dev/null +++ b/docs/production-deployment/temporal-proxy/secure-connections.mdx @@ -0,0 +1,227 @@ +--- +id: secure-connections +title: Secure connections and authorize requests +sidebar_label: Secure connections +description: Protect Temporal Proxy traffic with service restrictions, TLS, inbound authentication, authorization, and upstream credentials. +toc_max_heading_level: 4 +tags: + - Temporal Service + - Self-hosting + - Security +--- + +import { ReleaseNoteHeader } from '@site/src/components'; + + + Temporal Proxy is under active development and is not ready for production use. Behavior and configuration can change + between releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for current status + and the definitive configuration schema. + + +Temporal Proxy can restrict which Temporal gRPC services it forwards, protect inbound and outbound connections with +TLS, admit callers at the gateway, and present proxy-managed credentials to an upstream. Configure each control at the +trust boundary it protects. + +## Design the enforcement boundary + +Proxy controls apply only to traffic that enters the proxy gateway. A Client, Worker, CLI, or Web UI that can reach a +Temporal Service directly can bypass the proxy's service allowlist, authentication, authorization, routing, Namespace +translation, and Payload encryption. + +If the proxy is an enforcement point, combine its configuration with network controls that prevent application traffic +from reaching the upstream directly. Keep upstream credentials out of application configuration, restrict access to +the proxy's configuration and mounted secrets, and treat the proxy and its extension servers as trusted infrastructure. +See [Plan a Temporal Proxy deployment](/production-deployment/temporal-proxy/plan-deployment) for deployment topology +and availability considerations. + +The available controls protect different boundaries: + +| Boundary | Configuration | Effect | +| ---------------------------- | ------------------------------------------------- | ---------------------------------------------------------------------- | +| Caller to proxy | Top-level `tls` | Encrypts the connection and optionally verifies client certificates | +| Request admission | `allowedServices` and top-level `auth` | Limits services and admits or denies inbound calls | +| Proxy to extension server | `extensionServers[].tls` and `credentials` | Protects delegated authorization or key-management calls | +| Proxy to Temporal Service | `upstreams[].tls` and `credentials` | Encrypts the connection and supplies a proxy-managed upstream identity | +| Temporal Payload persistence | Top-level `encryption` | Encrypts supported Payloads before the upstream stores them | + +Transport TLS does not encrypt Payloads after the Temporal Service receives them. For that separate control, see +[Encrypt Payloads with Temporal Proxy](/production-deployment/temporal-proxy/encrypt-payloads). + +## Restrict forwarded services + +The top-level `allowedServices` list names each gRPC service the proxy forwards by its protobuf full name. If you omit +the list, the proxy forwards Workflow Service and Operator Service. An explicit empty list selects the same defaults; +it does not deny every service. + +```yaml +allowedServices: + - temporal.api.workflowservice.v1.WorkflowService + - temporal.api.operatorservice.v1.OperatorService + - grpc.reflection.v1.ServerReflection +``` + +| Service | Used by | In the default set | +| ------------------------------------------------- | ------------------------------------------------------------------------------- | ------------------ | +| `temporal.api.workflowservice.v1.WorkflowService` | SDK Clients, Workers, the CLI, and self-hosted Temporal Web UI | Yes | +| `temporal.api.operatorservice.v1.OperatorService` | `temporal operator` commands and self-hosted Web UI Search Attribute and Nexus Endpoint views | Yes | +| `grpc.reflection.v1.ServerReflection` | Service discovery for tools such as `grpcurl` | No | + +These are the services the proxy currently supports. An unknown or duplicate entry fails configuration validation at +startup. A call to a service you leave out fails with `UNIMPLEMENTED` before the proxy performs routing or contacts an +upstream. + +Reflection is opt-in. Allowing `grpc.reflection.v1.ServerReflection` also supports the superseded +`grpc.reflection.v1alpha.ServerReflection` endpoint used as a fallback by some clients. + +`allowedServices` works at service granularity, not at individual RPC or Namespace granularity. Use external +authorization to make decisions for a specific method, Namespace, or caller. + +## Secure inbound connections + +The top-level `tls` block protects connections from applications to the proxy gateway. `cert`, `key`, and `ca` are file +paths to PEM material on the proxy's filesystem. + +Configure server TLS with a certificate and its private key: + +```yaml +hostPort: 0.0.0.0:7233 +tls: + cert: /etc/temporal-proxy/tls/server.pem + key: /etc/temporal-proxy/tls/server-key.pem +``` + +Add `ca` to require mutual TLS. Each connecting application must then present a client certificate signed by that CA. + +```yaml +tls: + ca: /etc/temporal-proxy/tls/client-ca.pem + cert: /etc/temporal-proxy/tls/server.pem + key: /etc/temporal-proxy/tls/server-key.pem +``` + +Omit the top-level `tls` block to accept plaintext connections. If you include the block, `cert` and `key` are required +and must be set together. On Kubernetes, mount this material from a Secret as described in +[Supply TLS material](/production-deployment/temporal-proxy/deploy-kubernetes#supply-tls-material). + +Mutual TLS verifies that a client certificate chains to the configured CA. It does not replace a method- or +Namespace-aware authorization policy. + +## Authenticate inbound requests + +Inbound authentication is disabled when you omit the top-level `auth` block. When present, `auth` must select exactly +one of `staticToken`, `jwks`, or `external`. The proxy makes the decision before routing and strips the credential +header consumed by the authenticator before forwarding an admitted request. + +Use `staticToken` to compare a bearer token against one configured value: + +```yaml +auth: + staticToken: + token: ${GATEWAY_TOKEN} + header: authorization + scheme: Bearer +``` + +Use `jwks` to verify JWT signatures and claims against an HTTPS JWKS endpoint: + +```yaml +auth: + jwks: + url: https://issuer.example.com/.well-known/jwks.json + audiences: + - temporal-proxy + issuer: https://issuer.example.com/ + header: authorization + scheme: Bearer +``` + +`header` defaults to `authorization`, and `scheme` defaults to `Bearer`. A JWT must contain an expiration time. The +proxy checks `issuer` and whether the token contains one of `audiences` only when you configure those fields. A missing, +malformed, expired, or unverifiable token is not forwarded. + +These built-in authenticators determine whether a token is valid; they do not assign different permissions by method +or Namespace. Delegate the decision to an extension server when you need request-level authorization. + +## Delegate authorization to an extension server + +Configure `auth.external` to ask an operator-run server that implements `api.auth.v1.AuthService` about every inbound +stream. The extension server address must be static; templates are not supported for `extensionServers.hostPort`. + +```yaml +extensionServers: + - name: authz + hostPort: authz.internal.example.com:9444 + tls: + ca: /etc/temporal-proxy/tls/authz-ca.pem + cert: /etc/temporal-proxy/tls/authz-client.pem + key: /etc/temporal-proxy/tls/authz-client-key.pem + credentials: + static: + apiKey: ${AUTHZ_API_KEY} + +auth: + external: + name: authz + credentialHeaders: + - authorization +``` + +`name` must identify an entry in `extensionServers`. Each authorization request contains: + +| Field | Value | +| ------------------ | -------------------------------------------------------------------------------------------------------- | +| `target.full_name` | The gRPC full method name, including its leading slash | +| `target.namespace` | The local Namespace extracted before routing and translation, or empty when the Namespace is unknown | +| `credentials` | Every value received in each header named by `credentialHeaders` | + +The proxy also forwards the caller's remaining metadata to the extension server. It removes every declared credential +header before an admitted request is sent upstream. If `credentialHeaders` is empty, no caller header is identified as +a credential and no header is stripped. + +Only `DECISION_ALLOW` admits the request. `DECISION_DENY`, an unspecified or unrecognized decision, and an extension +server error all deny it. A denial returns `PERMISSION_DENIED`; an extension server error preserves its gRPC status +code so a Client or Worker can distinguish a retryable service failure from a rejected caller. + +Treat an empty `target.namespace` as unknown, not as a Namespace named with an empty string. Combine Namespace rules +with `target.full_name` so a Namespace-less or unreadable request cannot match a Namespace-scoped permission by +accident. The [authorization example](https://github.com/temporalio/temporal-proxy/tree/main/examples/authz) includes a +provider that maps a JWT to permissions for methods and Namespaces. + +Credentials on an extension server use the same static provider as upstream credentials. They require TLS. The proxy +removes any caller-supplied value in that credential header and sends its configured value instead. + +## Secure upstream connections + +Configure `tls` and `credentials` independently on each upstream: + +```yaml +upstreams: + - name: cloud + hostPort: my-namespace.my-account.tmprl.cloud:7233 + tls: {} + credentials: + static: + apiKey: ${TEMPORAL_API_KEY} + header: authorization + scheme: Bearer +``` + +An upstream TLS block supports these modes: + +- `tls: {}` verifies the upstream certificate with the system root certificates and does not present a client + certificate. Use this mode for Temporal Cloud with an API key. +- `ca` verifies the upstream against the CA or pinned trust anchor at that file path. +- `ca`, `cert`, and `key` enable mutual TLS. `cert` and `key` must be set together, and an outbound client certificate + requires `ca`. + +Set `serverName` when the address you dial does not match the DNS name on the upstream certificate. Both `hostPort` and +`tls.serverName` can use the per-request templates described in +[Resolve templated upstreams](/production-deployment/temporal-proxy/route-translate#resolve-templated-upstreams). + +`credentials.static` is the supported outbound credential provider. `apiKey` is required; `header` defaults to +`authorization`, and `scheme` defaults to `Bearer`. Credentials require TLS, and invalid combinations prevent the proxy +from starting. Before adding its credential, the proxy removes any caller-supplied value from the same outbound header. + +The upstream sees the proxy's configured identity, not the caller's consumed gateway credential. Preserve caller +identity in an independently trusted header only when the upstream is designed to validate it; forwarding an +unverified caller-supplied header does not establish identity. diff --git a/sidebars.js b/sidebars.js index ba42163641..67f4160008 100644 --- a/sidebars.js +++ b/sidebars.js @@ -1632,9 +1632,26 @@ module.exports = { id: 'production-deployment/temporal-proxy/index', }, items: [ - 'production-deployment/temporal-proxy/configure', - 'production-deployment/temporal-proxy/encrypt-payloads', + 'production-deployment/temporal-proxy/get-started-cloud', + 'production-deployment/temporal-proxy/plan-deployment', + { + type: 'category', + label: 'Configure', + collapsed: true, + link: { + type: 'doc', + id: 'production-deployment/temporal-proxy/configure', + }, + items: [ + 'production-deployment/temporal-proxy/route-translate', + 'production-deployment/temporal-proxy/secure-connections', + 'production-deployment/temporal-proxy/encrypt-payloads', + 'production-deployment/temporal-proxy/manage-encryption-keys', + ], + }, 'production-deployment/temporal-proxy/deploy-kubernetes', + 'production-deployment/temporal-proxy/operate-troubleshoot', + 'production-deployment/temporal-proxy/reference', ], }, ],