Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/operator-public-documentation/preview/api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,8 @@ _Appears in:_
| `primary` _string_ | Primary is the name of the primary cluster for replication. | | |
| `clusterList` _[MemberCluster](#membercluster) array_ | ClusterList is the list of clusters participating in replication. | | |
| `highAvailability` _boolean_ | Whether or not to have replicas on the primary cluster. | | |
| `replicationTLSSecret` _string_ | ReplicationTLSSecret is the name of a Kubernetes Secret containing TLS certificates<br />for the streaming_replica user used in physical replication. The secret must contain<br />"tls.crt" and "tls.key" keys. When specified, the operator references this secret in<br />clusters participating in replication.<br />NOTE: It needs to be the same for all clusters | | MaxLength: 253 <br />Pattern: `^[a-z0-9]([-a-z0-9]*[a-z0-9])?$` <br />Optional: \{\} <br /> |
| `clientCASecret` _string_ | ClientCASecret is the name of a Kubernetes Secret containing the CA certificate<br />used to verify the streaming_replica client certificate. The secret must contain<br />a "ca.crt" key. When specified, the operator references this secret in<br />clusters participating in replication.<br />NOTE: It needs to be the same for all clusters | | MaxLength: 253 <br />Pattern: `^[a-z0-9]([-a-z0-9]*[a-z0-9])?$` <br />Optional: \{\} <br /> |


#### DocumentDB
Expand Down Expand Up @@ -169,6 +171,7 @@ _Appears in:_
| `bootstrap` _[BootstrapConfiguration](#bootstrapconfiguration)_ | Bootstrap configures the initialization of the DocumentDB cluster. | | Optional: \{\} <br /> |
| `backup` _[BackupConfiguration](#backupconfiguration)_ | Backup configures backup settings for DocumentDB. | | Optional: \{\} <br /> |
| `featureGates` _object (keys:string, values:boolean)_ | FeatureGates enables or disables optional DocumentDB features.<br />Keys are PascalCase feature names following the Kubernetes feature gate convention.<br />Example: \{"ChangeStreams": true\}<br />IMPORTANT: When adding a new feature gate, update ALL of the following:<br />1. Add a new FeatureGate* constant in documentdb_types.go<br />2. Add the key name to the XValidation CEL rule's allowed list below<br />3. Add a default entry in the featureGateDefaults map in documentdb_types.go | | Optional: \{\} <br /> |
| `schemaVersion` _string_ | SchemaVersion controls the desired schema version for the DocumentDB extension.<br />The operator never changes your database schema unless you ask:<br /> - Set documentDBVersion → updates the binary (safe to roll back)<br /> - Set schemaVersion → updates the database schema (irreversible)<br /> - Set schemaVersion: "auto" → schema auto-updates with binary<br />Once the schema has been updated, the operator blocks image rollback below the<br />installed schema version to prevent running an untested binary/schema combination.<br />Values:<br /> - "" (empty, default): Two-phase mode. Image upgrades happen automatically,<br /> but ALTER EXTENSION UPDATE does NOT run. Users must explicitly set this<br /> field to finalize the schema upgrade. This is the safest option for production<br /> as it allows rollback by reverting the image before committing the schema change.<br /> - "auto": Schema automatically updates to match the binary version whenever<br /> the binary is upgraded. This is the simplest mode but provides no rollback<br /> safety window.<br /> - "<version>" (e.g. "0.112.0"): Schema updates to exactly this version.<br /> Must be <= the binary version. | | Pattern: `^(auto\|[0-9]+\.[0-9]+\.[0-9]+)?$` <br />Optional: \{\} <br /> |
| `affinity` _[AffinityConfiguration](https://pkg.go.dev/github.com/cloudnative-pg/cloudnative-pg/api/v1#AffinityConfiguration)_ | Affinity/Anti-affinity rules for Pods (cnpg passthrough) | | Optional: \{\} <br /> |


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ an equal or greater volume of available storage compared to the primary.
Enable TLS for all connections:

- **Client-to-gateway:** Encrypt application connections (see [TLS configuration](../configuration/tls.md))
- **Replication traffic:** PostgreSQL SSL for inter-cluster replication
- **Replication traffic:** Cross-Kubernetes-cluster streaming replication is authenticated with mutual TLS using the `streaming_replica` PostgreSQL role. Configure `spec.clusterReplication.replicationTLSSecret` and optionally `spec.clusterReplication.clientCASecret` to wire up the client certificate and CA. See [Securing replication with TLS](setup.md#securing-replication-with-tls) for the complete setup.
- **Service mesh:** mTLS for cross-cluster service communication

### Authentication and authorization
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,47 @@ spec:
- name: member-westus3-cluster
```

#### Securing replication with TLS

Cross-Kubernetes-cluster streaming replication flows over the network between
member Kubernetes clusters, so the operator secures it with mutual TLS instead
of password or trust-based authentication. Each replica connects to the primary
as the dedicated `streaming_replica` PostgreSQL role and presents a client
certificate that the primary verifies against a shared certificate authority (CA).

!!! important "Insecure by default"
If no cert is provided, the operator defaults to trusting all external replication
connections

When you provide a replication cert for your multi-regional setup, the operator
configures PostgreSQL to only accepts replication connections over TLS with a valid
client certificate (`hostssl replication streaming_replica all cert` in `pg_hba.conf`).
Each member Kubernetes cluster must use the same replication certificate and CA,
so any replica can authenticate to any primary after a failover. Put the cert into
a Kubernetes Secret, then pass the name in using the following fields.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `replicationTLSSecret` | string | Yes for secure multi-region | Name of a Kubernetes Secret that contains the `streaming_replica` client certificate and key. Must contain `tls.crt` and `tls.key`. Must be the same name in every member Kubernetes cluster. |
| `clientCASecret` | string | Optional | Name of a Kubernetes Secret that contains the CA certificate (`ca.crt`) used to verify the client certificate. If omitted, the operator falls back to the CA embedded in `replicationTLSSecret`. Must be the same name in every member Kubernetes cluster. |

The operator looks up the secrets by name in the DocumentDB namespace on each member
Kubernetes cluster. Both the secret name and the certificate material must match
across Kubernetes clusters — otherwise the replica can't authenticate to the primary.

For a working KubeFleet example that propagates a Secret to every member Kubernetes
cluster via `ClusterResourcePlacement`, see [`documentdb-resource-crp.yaml`](https://github.com/documentdb/documentdb-kubernetes-operator/blob/main/documentdb-playground/aks-fleet-deployment/documentdb-resource-crp.yaml)
in the playground.

!!! tip "Single-region deployments"
The `replicationTLSSecret` and `clientCASecret` fields aren't required for
single-region clusters. Intra-Kubernetes-cluster replication between CloudNative-PG
pods is already secured by the certificates CloudNative-PG provisions for each
cluster.

See the [ClusterReplication API Reference](../api-reference.md#clusterreplication)
for the full field list.

## Deployment options

Choose a deployment approach based on your infrastructure and operational preferences.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,6 @@ helm upgrade --install hub-agent ./charts/hub-agent/ \

# Run the script.
chmod +x ./hack/membership/joinMC.sh
sed -i 's/--set namespace=fleet-system/--namespace=fleet-system --create-namespace/' hack/membership/joinMC.sh
./hack/membership/joinMC.sh $TAG $HUB_CLUSTER $MEMBER_CLUSTER_NAMES
popd

Expand All @@ -171,12 +170,17 @@ helm install hub-net-controller-manager ./charts/hub-net-controller-manager/ \
--set image.tag=$NETWORKING_TAG

HUB_CLUSTER_ADDRESS=$(kubectl config view -o jsonpath="{.clusters[?(@.name==\"$HUB_CLUSTER\")].cluster.server}")
HUB_CA=$(kubectl config view --raw -o jsonpath="{.clusters[?(@.name==\"$HUB_CLUSTER\")].cluster.certificate-authority-data}")

while read -r MEMBER_CLUSTER; do
kubectl config use-context $MEMBER_CLUSTER

kubectl apply -f config/crd/*

# ADD HUB CA to member cluster (temp fix while joinMC.sh is out of date)
kubectl -n fleet-system set env deploy/member-agent \
HUB_CERTIFICATE_AUTHORITY="$HUB_CA" -c member-agent

echo "Installing mcs-controller-manager..."
helm install mcs-controller-manager ./charts/mcs-controller-manager/ \
--set refreshtoken.repository=$REGISTRY/refresh-token \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,39 @@ stringData:

---

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: selfsigned-cross-region-issuer
namespace: documentdb-preview-ns
spec:
selfSigned: {}
---
apiVersion: v1
kind: Secret
metadata:
name: cross-region-client-cert
namespace: documentdb-preview-ns
labels:
cnpg.io/reload: ""
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: cross-region-client-cert
namespace: documentdb-preview-ns
spec:
secretName: cross-region-client-cert
usages:
- client auth
commonName: streaming_replica
issuerRef:
name: selfsigned-cross-region-issuer
kind: Issuer
group: cert-manager.io

---

apiVersion: documentdb.io/preview
kind: DocumentDB
metadata:
Expand All @@ -32,6 +65,8 @@ spec:
environment: aks
clusterReplication:
highAvailability: true
replicationTLSSecret: cross-region-client-cert
clientCASecret: cross-region-client-cert
crossCloudNetworkingStrategy: AzureFleet
primary: {{PRIMARY_CLUSTER}}
clusterList:
Expand Down Expand Up @@ -75,6 +110,10 @@ spec:
version: v1
kind: Secret
name: documentdb-credentials
- group: ""
version: v1
kind: Secret
name: cross-region-client-cert
policy:
placementType: PickAll
affinity:
Expand Down
31 changes: 28 additions & 3 deletions operator/documentdb-helm-chart/crds/documentdb.io_dbs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1091,6 +1091,16 @@ spec:
description: ClusterReplication configures cross-cluster replication
for DocumentDB.
properties:
clientCASecret:
description: |-
ClientCASecret is the name of a Kubernetes Secret containing the CA certificate
used to verify the streaming_replica client certificate. The secret must contain
a "ca.crt" key. When specified, the operator references this secret in
clusters participating in replication.
NOTE: It needs to be the same for all clusters
maxLength: 253
pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$
type: string
clusterList:
description: ClusterList is the list of clusters participating
in replication.
Expand Down Expand Up @@ -1131,6 +1141,16 @@ spec:
primary:
description: Primary is the name of the primary cluster for replication.
type: string
replicationTLSSecret:
description: |-
ReplicationTLSSecret is the name of a Kubernetes Secret containing TLS certificates
for the streaming_replica user used in physical replication. The secret must contain
"tls.crt" and "tls.key" keys. When specified, the operator references this secret in
clusters participating in replication.
NOTE: It needs to be the same for all clusters
maxLength: 253
pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$
type: string
required:
- clusterList
- primary
Expand Down Expand Up @@ -1268,8 +1288,13 @@ spec:
description: |-
SchemaVersion controls the desired schema version for the DocumentDB extension.

This field decouples the extension binary (image) update from the schema update
(ALTER EXTENSION documentdb UPDATE), providing a rollback-safe upgrade window.
The operator never changes your database schema unless you ask:
- Set documentDBVersion → updates the binary (safe to roll back)
- Set schemaVersion → updates the database schema (irreversible)
- Set schemaVersion: "auto" → schema auto-updates with binary

Once the schema has been updated, the operator blocks image rollback below the
installed schema version to prevent running an untested binary/schema combination.

Values:
- "" (empty, default): Two-phase mode. Image upgrades happen automatically,
Expand All @@ -1278,7 +1303,7 @@ spec:
as it allows rollback by reverting the image before committing the schema change.
- "auto": Schema automatically updates to match the binary version whenever
the binary is upgraded. This is the simplest mode but provides no rollback
safety window. Recommended for development and testing environments.
safety window.
- "<version>" (e.g. "0.112.0"): Schema updates to exactly this version.
Must be <= the binary version.
pattern: ^(auto|[0-9]+\.[0-9]+\.[0-9]+)?$
Expand Down
18 changes: 18 additions & 0 deletions operator/src/api/preview/documentdb_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,24 @@ type ClusterReplication struct {
ClusterList []MemberCluster `json:"clusterList"`
// Whether or not to have replicas on the primary cluster.
HighAvailability bool `json:"highAvailability,omitempty"`
// ReplicationTLSSecret is the name of a Kubernetes Secret containing TLS certificates
// for the streaming_replica user used in physical replication. The secret must contain
// "tls.crt" and "tls.key" keys. When specified, the operator references this secret in
// clusters participating in replication.
// NOTE: It needs to be the same for all clusters
// +optional
// +kubebuilder:validation:Pattern=`^[a-z0-9]([-a-z0-9]*[a-z0-9])?$`
// +kubebuilder:validation:MaxLength=253
ReplicationTLSSecret string `json:"replicationTLSSecret,omitempty"`
// ClientCASecret is the name of a Kubernetes Secret containing the CA certificate
// used to verify the streaming_replica client certificate. The secret must contain
// a "ca.crt" key. When specified, the operator references this secret in
// clusters participating in replication.
// NOTE: It needs to be the same for all clusters
// +optional
// +kubebuilder:validation:Pattern=`^[a-z0-9]([-a-z0-9]*[a-z0-9])?$`
// +kubebuilder:validation:MaxLength=253
ClientCASecret string `json:"clientCASecret,omitempty"`
}

type MemberCluster struct {
Expand Down
31 changes: 28 additions & 3 deletions operator/src/config/crd/bases/documentdb.io_dbs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1091,6 +1091,16 @@ spec:
description: ClusterReplication configures cross-cluster replication
for DocumentDB.
properties:
clientCASecret:
description: |-
ClientCASecret is the name of a Kubernetes Secret containing the CA certificate
used to verify the streaming_replica client certificate. The secret must contain
a "ca.crt" key. When specified, the operator references this secret in
clusters participating in replication.
NOTE: It needs to be the same for all clusters
maxLength: 253
pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$
type: string
clusterList:
description: ClusterList is the list of clusters participating
in replication.
Expand Down Expand Up @@ -1131,6 +1141,16 @@ spec:
primary:
description: Primary is the name of the primary cluster for replication.
type: string
replicationTLSSecret:
description: |-
ReplicationTLSSecret is the name of a Kubernetes Secret containing TLS certificates
for the streaming_replica user used in physical replication. The secret must contain
"tls.crt" and "tls.key" keys. When specified, the operator references this secret in
clusters participating in replication.
NOTE: It needs to be the same for all clusters
maxLength: 253
pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$
type: string
required:
- clusterList
- primary
Expand Down Expand Up @@ -1268,8 +1288,13 @@ spec:
description: |-
SchemaVersion controls the desired schema version for the DocumentDB extension.

This field decouples the extension binary (image) update from the schema update
(ALTER EXTENSION documentdb UPDATE), providing a rollback-safe upgrade window.
The operator never changes your database schema unless you ask:
- Set documentDBVersion → updates the binary (safe to roll back)
- Set schemaVersion → updates the database schema (irreversible)
- Set schemaVersion: "auto" → schema auto-updates with binary

Once the schema has been updated, the operator blocks image rollback below the
installed schema version to prevent running an untested binary/schema combination.

Values:
- "" (empty, default): Two-phase mode. Image upgrades happen automatically,
Expand All @@ -1278,7 +1303,7 @@ spec:
as it allows rollback by reverting the image before committing the schema change.
- "auto": Schema automatically updates to match the binary version whenever
the binary is upgraded. This is the simplest mode but provides no rollback
safety window. Recommended for development and testing environments.
safety window.
- "<version>" (e.g. "0.112.0"): Schema updates to exactly this version.
Must be <= the binary version.
pattern: ^(auto|[0-9]+\.[0-9]+\.[0-9]+)?$
Expand Down
5 changes: 2 additions & 3 deletions operator/src/internal/cnpg/cnpg_cluster.go
Original file line number Diff line number Diff line change
Expand Up @@ -115,9 +115,8 @@ func GetCnpgClusterSpec(req ctrl.Request, documentdb *dbpreview.DocumentDB, docu
return params
}(),
PgHBA: []string{
"host all all 0.0.0.0/0 trust",
"host all all ::0/0 trust",
"host replication all all trust",
"host all all localhost trust",
"hostssl replication streaming_replica all cert",
},
Comment thread
alaye-ms marked this conversation as resolved.
},
Bootstrap: getBootstrapConfiguration(documentdb, isPrimaryRegion, log),
Expand Down
2 changes: 1 addition & 1 deletion operator/src/internal/cnpg/cnpg_cluster_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ var _ = Describe("GetCnpgClusterSpec", func() {
Expect(result.Spec.PostgresConfiguration.Extensions[0].LdLibraryPath).To(Equal([]string{"lib", "system"}))
Expect(result.Spec.PostgresConfiguration.AdditionalLibraries).To(ConsistOf("pg_cron", "pg_documentdb_core", "pg_documentdb"))
Expect(result.Spec.PostgresConfiguration.Parameters).To(HaveKeyWithValue("cron.database_name", "postgres"))
Expect(result.Spec.PostgresConfiguration.PgHBA).To(HaveLen(3))
Expect(result.Spec.PostgresConfiguration.PgHBA).To(HaveLen(2))
Expect(result.Spec.PostgresUID).To(Equal(int64(0)))
Expect(result.Spec.PostgresGID).To(Equal(int64(0)))
})
Expand Down
1 change: 1 addition & 0 deletions operator/src/internal/cnpg/cnpg_patch.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ const (
PatchPathPlugins = "/spec/plugins"
PatchPathReplicationSlots = "/spec/replicationSlots"
PatchPathExternalClusters = "/spec/externalClusters"
PatchPathCertificates = "/spec/certificates"
PatchPathManagedServices = "/spec/managed/services/additional"
PatchPathSynchronous = "/spec/postgresql/synchronous"
PatchPathBootstrap = "/spec/bootstrap"
Expand Down
Loading
Loading