Skip to content

<WIP> Reorganize Temporal Proxy documentation #5298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lukeknep wants to merge 2 commits into
base: main
Choose a base branch
from
Open

<WIP> Reorganize Temporal Proxy documentation #5298

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
88879ed
Clarify Temporal Proxy fail-closed encryption
lukeknep Sep 9, 2026
5ebcbcb
Reorganize Temporal Proxy documentation
lukeknep Sep 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
345 changes: 75 additions & 270 deletions docs/production-deployment/temporal-proxy/configure.mdx
View file
Open in desktop

Large diffs are not rendered by default.

35 changes: 20 additions & 15 deletions docs/production-deployment/temporal-proxy/deploy-kubernetes.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
id: deploy-kubernetes
title: Deploy Temporal Proxy to Kubernetes
sidebar_label: Deploy to Kubernetes
description: Deploy the Temporal Proxy with the Helm chart, including configuration, secrets, TLS, and KMS access.
description: Deploy Temporal Proxy with the Helm chart, mount configuration and secrets, configure probes and shutdown, and grant KMS access.
toc_max_heading_level: 4
tags:
- Temporal Service
Expand All @@ -13,16 +13,20 @@ tags:
import { ReleaseNoteHeader } from '@site/src/components';

<ReleaseNoteHeader type="prerelease">
Temporal Proxy is under active development and evolving quickly. Behavior and configuration can change between
releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status and
the definitive configuration schema.
Temporal Proxy is under active development and is not ready for production use. Behavior and configuration can change
between releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for current status
and the definitive configuration schema.
</ReleaseNoteHeader>

The [Helm chart](https://github.com/temporalio/helm-charts/tree/main/charts/temporal-proxy) provisions everything the
proxy needs: a Deployment, a Service, the ConfigMap that holds your configuration, a ServiceAccount, and an optional
HorizontalPodAutoscaler and PodDisruptionBudget. The `extraObjects` value renders additional manifests as-is, for
example an ExternalSecret that supplies upstream credentials.

Before choosing replica placement or exposing the gateway, see
[Plan a Temporal Proxy deployment](/production-deployment/temporal-proxy/plan-deployment) for topology, availability,
and trust-boundary guidance.

## Supply the configuration

`config` mirrors the proxy's own configuration schema. Whatever you put under it is what lands in the ConfigMap the
Expand Down Expand Up @@ -95,9 +99,10 @@ The gateway's `config.tls` block and each `upstreams[].tls` block accept a `secr
| `key` | `<mount>/tls.key`, or `<mount>/<keyKey>` if `keyKey` is set |
| `ca` | `<mount>/<caKey>`, only when `caKey` is set |

`ca` is opt-in because it changes behavior rather than just adding material. On the gateway, a `ca` enforces mutual TLS,
so only set `caKey` on `config.tls` when you intend to require client certificates. See
[Configure TLS](/production-deployment/temporal-proxy/configure#configure-tls) for what each combination means.
`ca` is opt-in because it changes connection behavior. On the gateway, a `ca` enforces mutual TLS, so only set `caKey`
on `config.tls` when you intend to require client certificates. See
[Secure upstream connections](/production-deployment/temporal-proxy/secure-connections#secure-upstream-connections) for
what each combination means.

```yaml
config:
Expand Down Expand Up @@ -135,21 +140,21 @@ readinessProbe:

Naming a service is optional; omit it to check the empty service name. Either way the answer is the same, because the
proxy's health is process-wide, and a service your configuration does not allow returns `NOT_FOUND`. See
[Health checks](/production-deployment/temporal-proxy/configure#health-checks) for what the gateway publishes.
[Check process health](/production-deployment/temporal-proxy/operate-troubleshoot#check-process-health) for what the
gateway publishes and what it does not test.

Kubernetes gRPC probes connect in plaintext, so they cannot reach a gateway that terminates TLS. When `config.tls` is
set, probe the proxy some other way, such as an exec probe running a client that presents the right certificate.

## Shut down cleanly

On `SIGTERM` the proxy stops accepting connections and drains in-flight RPCs. Each serving tier, meaning the gateway and
each upstream proxy, drains within five seconds, and the whole stop sequence is bounded at 30 seconds. Long-poll methods
block for much longer than that, so a rollout normally cuts some of them short: the proxy logs a warning naming the
deadline that ended the drain and still exits zero, because the callers re-poll. Treat that warning as ordinary on a
restart rather than a failed shutdown.
Set `terminationGracePeriodSeconds` to more than 30 seconds. On `SIGTERM`, the proxy stops admitting new requests and
drains each serving tier for up to five seconds; its complete shutdown is bounded at 30 seconds. A shorter Kubernetes
grace period can force-kill the process before its lifecycle hooks finish. Long polls can exceed a tier's drain window
and reconnect through another healthy replica.

An ordinary drain finishes in about five seconds, well inside the Kubernetes default `terminationGracePeriodSeconds`
of 30.
See [Shut down the proxy](/production-deployment/temporal-proxy/operate-troubleshoot#shut-down-the-proxy) for the full
shutdown sequence and expected warning.

## Grant access to your KMS key

Expand Down
124 changes: 47 additions & 77 deletions docs/production-deployment/temporal-proxy/encrypt-payloads.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
id: encrypt-payloads
title: Encrypt payloads with Temporal Proxy
sidebar_label: Encrypt payloads
description: Configure Temporal Proxy payload encryption with envelope encryption and cloud or custom KMS backends.
title: Encrypt Payloads with Temporal Proxy
sidebar_label: Encrypt Payloads
description: Protect supported Temporal Payload fields with envelope encryption backed by AWS KMS, Azure Key Vault, Google Cloud KMS, or a custom service.
toc_max_heading_level: 4
tags:
- Temporal Service
Expand All @@ -13,20 +13,21 @@ tags:
import { ReleaseNoteHeader } from '@site/src/components';

<ReleaseNoteHeader type="prerelease">
Temporal Proxy is under active development and evolving quickly. Behavior and configuration can change between
releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status
Temporal Proxy is under active development and is not ready for production use. Behavior and configuration can change
between releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for current status
and the definitive configuration schema.
</ReleaseNoteHeader>

The proxy can encrypt Workflow and Activity payloads on the hop to an upstream and decrypt them on responses, set under
the top-level `encryption` block. It is off by default. Workers and Clients keep exchanging cleartext with the gateway;
the proxy seals payloads before they leave and opens them on the way back, so the upstream Temporal Service only ever
stores ciphertext. Encryption is transparent, requiring no change to Worker or Client code.
the proxy seals codec-capable Payloads before they leave and opens them on the way back. New Payloads sent through the
proxy reach the upstream Temporal Service as ciphertext. Encryption is transparent, requiring no change to Worker or
Client code.

It uses envelope encryption: a short-lived data encryption key (DEK) encrypts each payload with AES-256-GCM, and a KMS
key you own wraps the DEK. The wrapped DEK and a reference to the key that wrapped it travel with the payload, so the
proxy never holds key material; it calls your KMS to wrap and unwrap DEKs. Supported key schemes are `awskms`,
`azurekeyvault`, and `gcpkms` for cloud KMS, `extension` for a
proxy does not receive the KMS key material. It calls your KMS to wrap and unwrap DEKs and can cache decrypted DEKs in
memory. Supported key schemes are `awskms`, `azurekeyvault`, and `gcpkms` for cloud KMS, `extension` for a
[key management backend you run yourself](#plug-in-your-own-key-management-backend), and `testing` for local development
only.

Expand All @@ -51,7 +52,7 @@ flowchart LR
Open -->|cleartext response| App
```

**Configuration**
## Configure payload encryption

```yaml
encryption:
Expand All @@ -77,6 +78,32 @@ encryption:
- `cacheSize` bounds the in-memory cache of decrypted DEKs, which avoids a KMS call on every message. It must be
non-negative.

## Understand fail-closed behavior

:::note Scope of fail-closed behavior

Fail-closed applies to outbound codec-capable Payloads on calls that pass through a proxy where `encryption.enabled` is
`true`. If the proxy cannot encrypt one of these Payloads, such as when the KMS is unavailable, it fails the request
before sending it upstream. It does not fall back to plaintext.

The proxy cannot enforce encryption for a Client, Worker, self-hosted Web UI, or CLI that connects directly to the
upstream Temporal Service. Restrict credentials and network paths when all application traffic must pass through the
proxy.

On responses, the proxy decrypts Payloads that carry its encryption metadata. It passes other Payloads through unchanged
so data written before proxy adoption remains readable during a migration. Closed Workflow histories normally age out
under the Namespace's retention period, but open Workflows, Archival, exports, backups, and other copies can preserve
plaintext data longer.

Fail-closed does not mean that every request field is encrypted. Search Attributes remain unencrypted so the Temporal
Service can index them. Failure messages and call stacks are not codec-capable Payloads by default. See
[Codecs and Encryption](/production-deployment/data-encryption) for encryption coverage and
[Failure Converter](/failure-converter) for failure encoding.

:::

## Define key policies

`default` and each `overrides` entry are key policies with the same shape:

| Field | Meaning |
Expand All @@ -86,57 +113,10 @@ encryption:
| `duration` | How long a DEK is used for new encryption before it rotates. Must be greater than zero. |
| `renewBefore` | Lead time before expiry at which the proxy pre-rotates the DEK. Must be at least zero and less than `duration`. |

DEK rotation is automatic on the `duration` and `renewBefore` schedule; you do not manage it. Older payloads stay
decryptable because their wrapped DEK and key reference travel with them.

To change the active key without losing access to existing data, for example when switching providers, promote the new
key to `uri` and move the old one to `decryptURIs` in the same update. New payloads use the new key; older payloads
still resolve against the decrypt-only entry:

```yaml
encryption:
default:
uri: gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v2
decryptURIs:
- gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v1
```

:::note Multi-region deployments need a multi-region key

To stay available alongside a highly available upstream that fails over across regions, such as a Temporal Cloud High
Availability Namespace whose endpoint moves between regions, you typically run the proxy in more than one region. After
a failover, a payload sealed by the proxy in one region may be opened by the proxy in another, so every regional proxy
must be able to unwrap the others' DEKs. Back the key with a multi-region KMS key so the same key material is reachable
from every region. How you express that differs by provider:

- **AWS:** use a multi-Region key (its ID starts with `mrk-`) and replicate it into each region. The replicas share key
material but each has its own regional ARN, so set each proxy's `uri` to its local ARN and list the other regions'
ARNs in `decryptURIs`:

```yaml
encryption:
default:
# Config for the us-east-1 proxy.
uri: awskms:///arn:aws:kms:us-east-1:123456789012:key/mrk-abcd1234?region=us-east-1
decryptURIs:
- awskms:///arn:aws:kms:us-west-2:123456789012:key/mrk-abcd1234?region=us-west-2
```

- **Azure:** Key Vault keys are regional. Use a separate vault per region (or geo-replication), and, because each vault
gives the key a distinct URI, set each proxy's `uri` to its local vault and list the other vaults' keys in
`decryptURIs`.
- **GCP:** create the key in a multi-region location (for example, `locations/us`) or in `global`. The resource name is
the same everywhere, so every proxy uses the identical `uri` and no `decryptURIs` are needed:
`gcpkms://projects/my-project/locations/us/keyRings/codec/cryptoKeys/my-key`.

For a single-region proxy deployment, a regional key is the right choice; reach for a multi-region key only when the
proxy itself spans regions.

:::

Decryption always runs on any payload that references a known key, regardless of `enabled`. Setting `enabled: false`
stops new encryption but keeps opening older payloads, which is a valid decrypt-only posture. The proxy reads encryption
configuration at startup, so restart it after changing keys or policies.
The proxy rotates DEKs automatically on the `duration` and `renewBefore` schedule. You remain responsible for changing,
retaining, and eventually retiring the KMS keys that wrap those DEKs. See
[Manage encryption keys](/production-deployment/temporal-proxy/manage-encryption-keys) for key migration,
multi-region access, decrypt-only operation, and safe retirement.

The `testing://` scheme holds its key material in the configuration itself and provides no real security. Use it only
for local development, and point production at `awskms`, `azurekeyvault`, `gcpkms`, or an extension server.
Expand Down Expand Up @@ -279,23 +259,13 @@ Two constraints follow from that shape. `Decrypt` gets ciphertext and nothing el
the local, pre-translation name, so a server keying on Namespace must use that name rather than the translated remote
one.

The proxy builds the connection when it builds its key policies, so a malformed address, credentials without TLS, or an
unreadable `ca` file all fail at startup. Peer verification happens later, because gRPC connects lazily: a `ca` or
`serverName` that does not match the server's certificate is not discovered until the first payload needs a key, and it
surfaces as a hung request rather than a clear error.
The proxy validates and opens extension-server connections before the gateway starts accepting traffic. A malformed or
unreachable address, credentials without TLS, unreadable `ca` file, or certificate mismatch therefore fails startup. A
credential that the extension server rejects, or a dependency that fails after startup, surfaces when a request needs a
key operation and fails that request.

The [KMS extension server example](https://github.com/temporalio/temporal-proxy/tree/main/examples/kms) runs the whole
path on localhost, with a reference provider you can read as a starting point.

:::caution Deleting a key is irreversible

Deleting a KMS key destroys everything encrypted under it, and neither Temporal nor your cloud provider can recover it.
Keep every key reachable, as `uri` or in `decryptURIs`, until every payload it wrapped is gone from every system the
proxy decrypts for, including Workflow history, visibility, and Archival storage. In practice, keep a key at least until
the Workflow Retention Period has elapsed.

To retire a key, revoke the proxy's permissions rather than deleting it. Removing access is reversible: you can re-grant
it later if a payload still needs opening, whereas a deleted key is gone for good. Revoking only the encrypt permission
while leaving decrypt in place also lets you stop new encryption without stranding existing payloads.

:::
After encryption works, define how keys change and how long old keys remain available. See
[Manage encryption keys](/production-deployment/temporal-proxy/manage-encryption-keys).
Loading