From 0bc53c378ee01db26bebd90137a52b01aa7b3c21 Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Thu, 18 Jun 2026 06:01:39 +0000 Subject: [PATCH 01/17] add PostgreSQL node migration and CloudBeaver solutions (MIDDLEWARE-31526) Precipitate historical internal KB PostgreSQL solutions, modernized to the current acid.zalan.do/v1 postgresql CR and verified live on ACP 4.2/4.3, bilingual (en + zh): - KB260515007 How to Migrate a PostgreSQL Instance to Another Node - KB260515008 How to Deploy CloudBeaver for PostgreSQL Management Co-Authored-By: Claude Opus 4.8 (1M context) --- ...y_CloudBeaver_for_PostgreSQL_Management.md | 159 +++++++++++++++++ ...e_a_PostgreSQL_Instance_to_Another_Node.md | 130 ++++++++++++++ ...y_CloudBeaver_for_PostgreSQL_Management.md | 160 ++++++++++++++++++ ...e_a_PostgreSQL_Instance_to_Another_Node.md | 105 ++++++++++++ 4 files changed, 554 insertions(+) create mode 100644 docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md create mode 100644 docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md create mode 100644 docs/zh/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md create mode 100644 docs/zh/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md diff --git a/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md b/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md new file mode 100644 index 000000000..4cd63077f --- /dev/null +++ b/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md @@ -0,0 +1,159 @@ +--- +kind: + - How To +products: + - Alauda Application Services +ProductsVersion: + - 4.0,4.1,4.2,4.3 +id: KB260515008 +--- + +# How to Deploy CloudBeaver for PostgreSQL Management + +## Issue + +You need a web-based SQL client for browsing and querying PostgreSQL instances managed by Alauda Application Services, without installing a desktop tool on every operator's workstation. CloudBeaver is the open-source web edition of DBeaver and runs as a single-pod Deployment on Kubernetes. This how-to deploys CloudBeaver and connects it to a PostgreSQL cluster managed by the PostgreSQL Operator. + +## Environment + +- Any Kubernetes cluster reachable to operators (NodePort exposure is used below; Ingress / Route works equally well) +- A `StorageClass` capable of provisioning ReadWriteOnce PVCs — used to persist CloudBeaver workspace state across pod restarts +- Network reachability from the CloudBeaver pod to the target PostgreSQL Service (`` on port 5432) + +## Resolution + +### 1. Prepare the manifest + +Save the following to `cloudbeaver.yaml`. Adjust `storageClassName`, image registry, and resource requests to match your environment: + +```yaml +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: cloudbeaver +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: sc-topolvm # replace with any RWO StorageClass available in the cluster + volumeMode: Filesystem +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: cloudbeaver +spec: + replicas: 1 + selector: + matchLabels: + app: cloudbeaver + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 1 + template: + metadata: + labels: + app: cloudbeaver + spec: + containers: + - name: cloudbeaver + image: docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest + imagePullPolicy: Always + ports: + - name: web + containerPort: 8978 + protocol: TCP + resources: + limits: + cpu: "1" + memory: 1Gi + requests: + cpu: 100m + memory: 256Mi + volumeMounts: + - name: cloudbeaver-data + mountPath: /opt/cloudbeaver/workspace + volumes: + - name: cloudbeaver-data + persistentVolumeClaim: + claimName: cloudbeaver +--- +apiVersion: v1 +kind: Service +metadata: + name: cloudbeaver +spec: + type: NodePort + selector: + app: cloudbeaver + ports: + - name: web + port: 8978 + targetPort: 8978 + protocol: TCP +``` + +> CloudBeaver stores its administrator password, saved connections, and query history under `/opt/cloudbeaver/workspace`. Without a persistent volume, everything is lost on every pod restart. Confirm the chosen `storageClassName` exists in the target cluster before applying. + +> **Air-gapped / IPv6-only clusters:** the public `docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest` image may be unreachable from the cluster. Mirror it into the cluster's own registry first and reference that path in the Deployment, for example `skopeo copy docker://docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest docker:///dbeaver/cloudbeaver:latest`. + +### 2. Deploy + +```bash +kubectl -n apply -f cloudbeaver.yaml +kubectl -n rollout status deploy/cloudbeaver +``` + +### 3. Discover the access URL + +The Service uses NodePort, so any node IP plus the allocated NodePort exposes the UI: + +```bash +namespace= +HOST=$(kubectl -n "$namespace" get pod -l app=cloudbeaver \ + -o jsonpath='{.items[0].status.hostIP}') +PORT=$(kubectl -n "$namespace" get svc cloudbeaver \ + -o jsonpath='{.spec.ports[0].nodePort}') +echo "http://$HOST:$PORT" +``` + +Open the URL in a browser. + +### 4. Initial setup + +1. On first launch CloudBeaver prompts you to set the administrator password. Choose a strong password and store it safely — this account governs all subsequent server-side configuration. +2. Log in with the administrator user. +3. (Optional) Switch the UI language under the user menu in the top-right. + +### 5. Connect to a PostgreSQL instance + +1. Click **New Connection** and choose **PostgreSQL**. +2. Fill **Host** with the cluster Service name and **Port** `5432`. From inside the cluster the host is `.` (the read-write Service); the `-repl` Service points at replicas. From outside the cluster, retrieve the NodePort or LoadBalancer address: + + ```bash + kubectl -n get svc + ``` + +3. Enter the database user and password. The `postgres` superuser password is available in the cluster as an environment variable / Secret: + + ```bash + kubectl exec -n -0 -c postgres -- \ + bash -c 'echo $PGPASSWORD_SUPERUSER' + ``` + +4. (Optional) Set **Database** to the target database; otherwise CloudBeaver connects to the default `postgres` database. +5. Under **Access Management**, grant the current CloudBeaver user permission to use the new connection. +6. Save and test the connection. SQL editors can now be opened from the browser and queries executed against the PostgreSQL cluster. + +### 6. Uninstall + +```bash +kubectl -n delete -f cloudbeaver.yaml +``` + +The PVC is removed alongside the rest of the manifest. To preserve CloudBeaver state for a future redeploy, delete only the Deployment and Service and re-attach the existing PVC during the next install. diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md new file mode 100644 index 000000000..b82b457bc --- /dev/null +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md @@ -0,0 +1,130 @@ +--- +kind: + - Solution +products: + - Alauda Application Services +ProductsVersion: + - 4.0,4.1,4.2,4.3 +id: KB260515007 +--- + +# How to Migrate a PostgreSQL Instance to Another Node + +## Issue + +A PostgreSQL cluster managed by the PostgreSQL Operator uses node-local storage +(for example TopoLVM). One or more instances must be moved off their current +node — because the node is being decommissioned, or an instance must run on a +specific compute node. Because the data lives in a node-local PersistentVolume, +the pod cannot simply be rescheduled; the volume is pinned to its node. + +## Environment + +- Alauda Application Services PostgreSQL Operator (Zalando-based, + `acid.zalan.do/v1` `postgresql` resource). +- Node-local storage such as TopoLVM (each PVC is bound to one node). +- At least one target node with enough free capacity. Because migration + re-clones data, the target node should have capacity for roughly **twice** the + instance's PVC size during the transition. + +## Resolution + +The migration relies on a property of the Operator: when an instance's PVC and +pod are deleted, the StatefulSet recreates the pod, a fresh PVC is provisioned +wherever the pod is scheduled, and Patroni re-clones the data from the current +leader. Data is preserved through streaming replication, not by moving the +volume. + +> Validated on ACP 4.2 and 4.3: after deleting a replica's PVC and pod, the +> member was recreated on a node, re-synced from the leader, and previously +> written rows were present on the resynced member. + +In the examples below set `$NAMESPACE` and `$CLUSTER_NAME` for the target +cluster. Replace placeholder node names with your own. + +### 1. Confirm the current placement + +```bash +kubectl get pod -n $NAMESPACE -o wide | grep $CLUSTER_NAME +kubectl get pvc -n $NAMESPACE -o wide | grep $CLUSTER_NAME +``` + +Note which member is the leader (do not delete the leader's volume first): + +```bash +kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- patronictl list +``` + +### 2. Restrict scheduling to the target node(s) + +So the recreated pod lands only on a desired node, cordon the other eligible +nodes (or use a `nodeSelector`/label that only the target nodes carry). Keep at +least the target node schedulable. + +```bash +# Example: label the source and target nodes, then select them +kubectl label node target=true --overwrite +``` + +If you use a label-based selector, set it on the instance: + +```bash +kubectl patch postgresql -n $NAMESPACE $CLUSTER_NAME --type merge \ + -p '{"spec":{"nodeSelector":{"target":"true"}}}' +``` + +### 3. Migrate one member at a time + +Always migrate a **non-leader** member first. Delete its PVC and pod together — +the PVC deletion blocks until the pod that mounts it is gone, so delete the pod +in parallel: + +```bash +# Delete the data PVC (it will stay in Terminating until the pod is gone) +kubectl delete pvc pgdata-$CLUSTER_NAME-1 -n $NAMESPACE --wait=false + +# Delete the pod to release the PVC +kubectl delete pod $CLUSTER_NAME-1 -n $NAMESPACE +``` + +The StatefulSet recreates `$CLUSTER_NAME-1`; a new PVC is provisioned on the +scheduled node and Patroni re-clones from the leader. + +### 4. Verify the member rejoined with its data + +```bash +kubectl get pod -n $NAMESPACE -o wide | grep $CLUSTER_NAME-1 +kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- patronictl list +``` + +The migrated member should return to role `Replica`, state `running`/`streaming` +with `Lag in MB` `0`. Spot-check data on the member (it is read-only / in +recovery): + +```bash +kubectl exec -n $NAMESPACE $CLUSTER_NAME-1 -c postgres -- \ + psql -U postgres -tAc "SELECT pg_is_in_recovery();" # expect t +``` + +### 5. Migrate the (former) leader if required + +To move the leader, first perform a switchover so another member becomes leader, +then repeat steps 3–4 for the old leader: + +```bash +kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- \ + patronictl switchover $CLUSTER_NAME --force +``` + +### 6. Restore scheduling + +Uncordon any nodes you cordoned and remove temporary labels/`nodeSelector` once +all members are on their intended nodes. + +## Notes + +- Migrate members one at a time and wait for each to fully re-sync (`Lag = 0`) + before moving the next, so the cluster always retains a healthy quorum. +- For a single-instance cluster, temporarily scale to two instances, let the new + member sync on the target node, switch over, then scale back to one — this + avoids downtime that a delete-and-reclone of the sole instance would cause. diff --git a/docs/zh/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md b/docs/zh/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md new file mode 100644 index 000000000..06360ae8e --- /dev/null +++ b/docs/zh/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md @@ -0,0 +1,160 @@ +--- +kind: + - How To +products: + - Alauda Application Services +ProductsVersion: + - '4.0,4.1,4.2,4.3' +id: KB260515008 +sourceSHA: e84115331a26da1f333d7e670c2f23f9970f7e06f98f168db22f6b0646ca7e5c +--- + +# 如何部署 CloudBeaver 以管理 PostgreSQL + +## 问题 + +你需要一个基于 Web 的 SQL 客户端，用于浏览和查询由 Alauda Application Services 管理的 PostgreSQL 实例，而无需在每位运维人员的工作站上安装桌面工具。CloudBeaver 是 DBeaver 的开源 Web 版本，作为单 Pod 的 Deployment 运行在 Kubernetes 上。本文档部署 CloudBeaver 并将其连接到由 PostgreSQL Operator 管理的 PostgreSQL 集群。 + +## 环境 + +- 任意运维人员可访问的 Kubernetes 集群（下文使用 NodePort 暴露；Ingress / Route 同样适用） +- 能够供给 ReadWriteOnce PVC 的 `StorageClass`——用于在 Pod 重启间持久化 CloudBeaver 工作区状态 +- 从 CloudBeaver Pod 到目标 PostgreSQL Service（`` 的 5432 端口）的网络可达性 + +## 解决方案 + +### 1. 准备清单 + +将以下内容保存为 `cloudbeaver.yaml`。根据环境调整 `storageClassName`、镜像仓库与资源请求： + +```yaml +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: cloudbeaver +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: sc-topolvm # 替换为集群中任意可用的 RWO StorageClass + volumeMode: Filesystem +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: cloudbeaver +spec: + replicas: 1 + selector: + matchLabels: + app: cloudbeaver + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 1 + template: + metadata: + labels: + app: cloudbeaver + spec: + containers: + - name: cloudbeaver + image: docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest + imagePullPolicy: Always + ports: + - name: web + containerPort: 8978 + protocol: TCP + resources: + limits: + cpu: "1" + memory: 1Gi + requests: + cpu: 100m + memory: 256Mi + volumeMounts: + - name: cloudbeaver-data + mountPath: /opt/cloudbeaver/workspace + volumes: + - name: cloudbeaver-data + persistentVolumeClaim: + claimName: cloudbeaver +--- +apiVersion: v1 +kind: Service +metadata: + name: cloudbeaver +spec: + type: NodePort + selector: + app: cloudbeaver + ports: + - name: web + port: 8978 + targetPort: 8978 + protocol: TCP +``` + +> CloudBeaver 将其管理员密码、已保存的连接和查询历史存储在 `/opt/cloudbeaver/workspace` 下。没有持久卷，所有内容都会在每次 Pod 重启时丢失。应用前请确认所选 `storageClassName` 在目标集群中存在。 + +> **隔离网络 / 仅 IPv6 集群：** 公共镜像 `docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest` 可能从集群内无法访问。请先将其镜像到集群自身的仓库，并在 Deployment 中引用该路径，例如 `skopeo copy docker://docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest docker:///dbeaver/cloudbeaver:latest`。 + +### 2. 部署 + +```bash +kubectl -n apply -f cloudbeaver.yaml +kubectl -n rollout status deploy/cloudbeaver +``` + +### 3. 获取访问地址 + +Service 使用 NodePort，因此任意节点 IP 加上分配的 NodePort 即可访问 UI： + +```bash +namespace= +HOST=$(kubectl -n "$namespace" get pod -l app=cloudbeaver \ + -o jsonpath='{.items[0].status.hostIP}') +PORT=$(kubectl -n "$namespace" get svc cloudbeaver \ + -o jsonpath='{.spec.ports[0].nodePort}') +echo "http://$HOST:$PORT" +``` + +在浏览器中打开该地址。 + +### 4. 初始设置 + +1. 首次启动时，CloudBeaver 会提示设置管理员密码。请选择强密码并妥善保存——该账户管理后续所有服务端配置。 +2. 使用管理员用户登录。 +3. （可选）在右上角用户菜单中切换 UI 语言。 + +### 5. 连接 PostgreSQL 实例 + +1. 点击 **New Connection** 并选择 **PostgreSQL**。 +2. **Host** 填写集群 Service 名称，**Port** 填写 `5432`。集群内部主机为 `.`（读写 Service）；`-repl` Service 指向副本。从集群外部访问时，获取 NodePort 或 LoadBalancer 地址： + + ```bash + kubectl -n get svc + ``` + +3. 输入数据库用户与密码。`postgres` 超级用户密码以环境变量 / Secret 形式存在于集群中： + + ```bash + kubectl exec -n -0 -c postgres -- \ + bash -c 'echo $PGPASSWORD_SUPERUSER' + ``` + +4. （可选）将 **Database** 设置为目标数据库；否则 CloudBeaver 连接到默认的 `postgres` 数据库。 +5. 在 **Access Management** 中，授予当前 CloudBeaver 用户使用该新连接的权限。 +6. 保存并测试连接。此后即可从浏览器打开 SQL 编辑器并对 PostgreSQL 集群执行查询。 + +### 6. 卸载 + +```bash +kubectl -n delete -f cloudbeaver.yaml +``` + +PVC 会随清单的其余部分一并删除。若要为将来重新部署保留 CloudBeaver 状态，仅删除 Deployment 与 Service，并在下次安装时重新挂载现有 PVC。 diff --git a/docs/zh/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md b/docs/zh/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md new file mode 100644 index 000000000..bcf155e80 --- /dev/null +++ b/docs/zh/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md @@ -0,0 +1,105 @@ +--- +kind: + - Solution +products: + - Alauda Application Services +ProductsVersion: + - '4.0,4.1,4.2,4.3' +id: KB260515007 +sourceSHA: 68545d5287dbfbc24261231c54b484401a71c97c16407e79865eea4f4aefd110 +--- + +# 如何将 PostgreSQL 实例迁移到其他节点 + +## 问题 + +由 PostgreSQL Operator 管理的 PostgreSQL 集群使用节点本地存储（例如 TopoLVM）。由于节点下线，或某个实例需要运行在指定的计算节点上，需要将一个或多个实例迁出当前节点。由于数据位于节点本地的 PersistentVolume 上，Pod 无法被直接重新调度——卷被固定在其所在节点上。 + +## 环境 + +- Alauda Application Services PostgreSQL Operator（基于 Zalando，`acid.zalan.do/v1` 的 `postgresql` 资源）。 +- 节点本地存储，例如 TopoLVM（每个 PVC 绑定到单个节点）。 +- 至少一个有足够空闲容量的目标节点。由于迁移会重新克隆数据，目标节点在迁移期间应具备约为实例 PVC 大小**两倍**的容量。 + +## 解决方案 + +迁移依赖 Operator 的一个特性：当某个实例的 PVC 与 Pod 被删除后，StatefulSet 会重建该 Pod，在 Pod 被调度到的节点上重新供给一个新的 PVC，Patroni 则从当前 Leader 重新克隆数据。数据通过流复制得以保留，而非通过移动卷。 + +> 已在 ACP 4.2 与 4.3 上验证：删除某副本的 PVC 与 Pod 后，该成员在节点上被重建，从 Leader 重新同步，先前写入的数据行在重新同步后的成员上依然存在。 + +以下示例中，请为目标集群设置 `$NAMESPACE` 与 `$CLUSTER_NAME`，并将占位节点名替换为实际节点名。 + +### 1. 确认当前分布 + +```bash +kubectl get pod -n $NAMESPACE -o wide | grep $CLUSTER_NAME +kubectl get pvc -n $NAMESPACE -o wide | grep $CLUSTER_NAME +``` + +确认哪个成员是 Leader（不要先删除 Leader 的卷）： + +```bash +kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- patronictl list +``` + +### 2. 将调度限制到目标节点 + +为使重建的 Pod 只落在期望的节点上，将其他可调度节点设置为不可调度（cordon），或使用仅目标节点具备的 `nodeSelector`/标签，并至少保留目标节点可调度。 + +```bash +# 示例：为源节点与目标节点打标签，然后基于标签选择 +kubectl label node target=true --overwrite +``` + +如使用基于标签的选择器，在实例上设置： + +```bash +kubectl patch postgresql -n $NAMESPACE $CLUSTER_NAME --type merge \ + -p '{"spec":{"nodeSelector":{"target":"true"}}}' +``` + +### 3. 逐个迁移成员 + +务必先迁移**非 Leader** 成员。同时删除其 PVC 与 Pod——PVC 的删除会阻塞，直到挂载它的 Pod 消失，因此需并行删除 Pod： + +```bash +# 删除数据 PVC（在 Pod 消失前会一直处于 Terminating） +kubectl delete pvc pgdata-$CLUSTER_NAME-1 -n $NAMESPACE --wait=false + +# 删除 Pod 以释放 PVC +kubectl delete pod $CLUSTER_NAME-1 -n $NAMESPACE +``` + +StatefulSet 会重建 `$CLUSTER_NAME-1`；在被调度的节点上供给新的 PVC，Patroni 从 Leader 重新克隆。 + +### 4. 验证成员已带数据重新加入 + +```bash +kubectl get pod -n $NAMESPACE -o wide | grep $CLUSTER_NAME-1 +kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- patronictl list +``` + +迁移后的成员应恢复为 `Replica` 角色，状态 `running`/`streaming`，`Lag in MB` 为 `0`。在该成员上抽查数据（它是只读 / 处于恢复状态）： + +```bash +kubectl exec -n $NAMESPACE $CLUSTER_NAME-1 -c postgres -- \ + psql -U postgres -tAc "SELECT pg_is_in_recovery();" # 期望为 t +``` + +### 5. 如需迁移（原）Leader + +要迁移 Leader，先执行切换（switchover）让其他成员成为 Leader，然后对原 Leader 重复第 3–4 步： + +```bash +kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- \ + patronictl switchover $CLUSTER_NAME --force +``` + +### 6. 恢复调度 + +待所有成员都位于期望节点后，取消对任何节点的 cordon，并移除临时标签/`nodeSelector`。 + +## 说明 + +- 逐个迁移成员，并在迁移下一个之前等待每个成员完全重新同步（`Lag = 0`），以使集群始终保持健康的法定数量。 +- 对于单实例集群，临时扩容到两个实例，让新成员在目标节点上完成同步，执行切换后再缩容回一个实例——以避免对唯一实例执行删除并重新克隆所导致的停机。 From 73e1b5d6dd08f355c0702c90b8845e3044ab094c Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Thu, 18 Jun 2026 07:16:34 +0000 Subject: [PATCH 02/17] docs: drop hand-written zh mirrors (pipeline auto-generates them) The zh translations are produced by the translation pipeline from the en sources (tracked via sourceSHA); they should not be committed by hand. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...y_CloudBeaver_for_PostgreSQL_Management.md | 160 ------------------ ...e_a_PostgreSQL_Instance_to_Another_Node.md | 105 ------------ 2 files changed, 265 deletions(-) delete mode 100644 docs/zh/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md delete mode 100644 docs/zh/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md diff --git a/docs/zh/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md b/docs/zh/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md deleted file mode 100644 index 06360ae8e..000000000 --- a/docs/zh/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md +++ /dev/null @@ -1,160 +0,0 @@ ---- -kind: - - How To -products: - - Alauda Application Services -ProductsVersion: - - '4.0,4.1,4.2,4.3' -id: KB260515008 -sourceSHA: e84115331a26da1f333d7e670c2f23f9970f7e06f98f168db22f6b0646ca7e5c ---- - -# 如何部署 CloudBeaver 以管理 PostgreSQL - -## 问题 - -你需要一个基于 Web 的 SQL 客户端，用于浏览和查询由 Alauda Application Services 管理的 PostgreSQL 实例，而无需在每位运维人员的工作站上安装桌面工具。CloudBeaver 是 DBeaver 的开源 Web 版本，作为单 Pod 的 Deployment 运行在 Kubernetes 上。本文档部署 CloudBeaver 并将其连接到由 PostgreSQL Operator 管理的 PostgreSQL 集群。 - -## 环境 - -- 任意运维人员可访问的 Kubernetes 集群（下文使用 NodePort 暴露；Ingress / Route 同样适用） -- 能够供给 ReadWriteOnce PVC 的 `StorageClass`——用于在 Pod 重启间持久化 CloudBeaver 工作区状态 -- 从 CloudBeaver Pod 到目标 PostgreSQL Service（`` 的 5432 端口）的网络可达性 - -## 解决方案 - -### 1. 准备清单 - -将以下内容保存为 `cloudbeaver.yaml`。根据环境调整 `storageClassName`、镜像仓库与资源请求： - -```yaml ---- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: cloudbeaver -spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 1Gi - storageClassName: sc-topolvm # 替换为集群中任意可用的 RWO StorageClass - volumeMode: Filesystem ---- -apiVersion: apps/v1 -kind: Deployment -metadata: - name: cloudbeaver -spec: - replicas: 1 - selector: - matchLabels: - app: cloudbeaver - strategy: - type: RollingUpdate - rollingUpdate: - maxSurge: 1 - maxUnavailable: 1 - template: - metadata: - labels: - app: cloudbeaver - spec: - containers: - - name: cloudbeaver - image: docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest - imagePullPolicy: Always - ports: - - name: web - containerPort: 8978 - protocol: TCP - resources: - limits: - cpu: "1" - memory: 1Gi - requests: - cpu: 100m - memory: 256Mi - volumeMounts: - - name: cloudbeaver-data - mountPath: /opt/cloudbeaver/workspace - volumes: - - name: cloudbeaver-data - persistentVolumeClaim: - claimName: cloudbeaver ---- -apiVersion: v1 -kind: Service -metadata: - name: cloudbeaver -spec: - type: NodePort - selector: - app: cloudbeaver - ports: - - name: web - port: 8978 - targetPort: 8978 - protocol: TCP -``` - -> CloudBeaver 将其管理员密码、已保存的连接和查询历史存储在 `/opt/cloudbeaver/workspace` 下。没有持久卷，所有内容都会在每次 Pod 重启时丢失。应用前请确认所选 `storageClassName` 在目标集群中存在。 - -> **隔离网络 / 仅 IPv6 集群：** 公共镜像 `docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest` 可能从集群内无法访问。请先将其镜像到集群自身的仓库，并在 Deployment 中引用该路径，例如 `skopeo copy docker://docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest docker:///dbeaver/cloudbeaver:latest`。 - -### 2. 部署 - -```bash -kubectl -n apply -f cloudbeaver.yaml -kubectl -n rollout status deploy/cloudbeaver -``` - -### 3. 获取访问地址 - -Service 使用 NodePort，因此任意节点 IP 加上分配的 NodePort 即可访问 UI： - -```bash -namespace= -HOST=$(kubectl -n "$namespace" get pod -l app=cloudbeaver \ - -o jsonpath='{.items[0].status.hostIP}') -PORT=$(kubectl -n "$namespace" get svc cloudbeaver \ - -o jsonpath='{.spec.ports[0].nodePort}') -echo "http://$HOST:$PORT" -``` - -在浏览器中打开该地址。 - -### 4. 初始设置 - -1. 首次启动时，CloudBeaver 会提示设置管理员密码。请选择强密码并妥善保存——该账户管理后续所有服务端配置。 -2. 使用管理员用户登录。 -3. （可选）在右上角用户菜单中切换 UI 语言。 - -### 5. 连接 PostgreSQL 实例 - -1. 点击 **New Connection** 并选择 **PostgreSQL**。 -2. **Host** 填写集群 Service 名称，**Port** 填写 `5432`。集群内部主机为 `.`（读写 Service）；`-repl` Service 指向副本。从集群外部访问时，获取 NodePort 或 LoadBalancer 地址： - - ```bash - kubectl -n get svc - ``` - -3. 输入数据库用户与密码。`postgres` 超级用户密码以环境变量 / Secret 形式存在于集群中： - - ```bash - kubectl exec -n -0 -c postgres -- \ - bash -c 'echo $PGPASSWORD_SUPERUSER' - ``` - -4. （可选）将 **Database** 设置为目标数据库；否则 CloudBeaver 连接到默认的 `postgres` 数据库。 -5. 在 **Access Management** 中，授予当前 CloudBeaver 用户使用该新连接的权限。 -6. 保存并测试连接。此后即可从浏览器打开 SQL 编辑器并对 PostgreSQL 集群执行查询。 - -### 6. 卸载 - -```bash -kubectl -n delete -f cloudbeaver.yaml -``` - -PVC 会随清单的其余部分一并删除。若要为将来重新部署保留 CloudBeaver 状态，仅删除 Deployment 与 Service，并在下次安装时重新挂载现有 PVC。 diff --git a/docs/zh/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md b/docs/zh/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md deleted file mode 100644 index bcf155e80..000000000 --- a/docs/zh/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md +++ /dev/null @@ -1,105 +0,0 @@ ---- -kind: - - Solution -products: - - Alauda Application Services -ProductsVersion: - - '4.0,4.1,4.2,4.3' -id: KB260515007 -sourceSHA: 68545d5287dbfbc24261231c54b484401a71c97c16407e79865eea4f4aefd110 ---- - -# 如何将 PostgreSQL 实例迁移到其他节点 - -## 问题 - -由 PostgreSQL Operator 管理的 PostgreSQL 集群使用节点本地存储（例如 TopoLVM）。由于节点下线，或某个实例需要运行在指定的计算节点上，需要将一个或多个实例迁出当前节点。由于数据位于节点本地的 PersistentVolume 上，Pod 无法被直接重新调度——卷被固定在其所在节点上。 - -## 环境 - -- Alauda Application Services PostgreSQL Operator（基于 Zalando，`acid.zalan.do/v1` 的 `postgresql` 资源）。 -- 节点本地存储，例如 TopoLVM（每个 PVC 绑定到单个节点）。 -- 至少一个有足够空闲容量的目标节点。由于迁移会重新克隆数据，目标节点在迁移期间应具备约为实例 PVC 大小**两倍**的容量。 - -## 解决方案 - -迁移依赖 Operator 的一个特性：当某个实例的 PVC 与 Pod 被删除后，StatefulSet 会重建该 Pod，在 Pod 被调度到的节点上重新供给一个新的 PVC，Patroni 则从当前 Leader 重新克隆数据。数据通过流复制得以保留，而非通过移动卷。 - -> 已在 ACP 4.2 与 4.3 上验证：删除某副本的 PVC 与 Pod 后，该成员在节点上被重建，从 Leader 重新同步，先前写入的数据行在重新同步后的成员上依然存在。 - -以下示例中，请为目标集群设置 `$NAMESPACE` 与 `$CLUSTER_NAME`，并将占位节点名替换为实际节点名。 - -### 1. 确认当前分布 - -```bash -kubectl get pod -n $NAMESPACE -o wide | grep $CLUSTER_NAME -kubectl get pvc -n $NAMESPACE -o wide | grep $CLUSTER_NAME -``` - -确认哪个成员是 Leader（不要先删除 Leader 的卷）： - -```bash -kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- patronictl list -``` - -### 2. 将调度限制到目标节点 - -为使重建的 Pod 只落在期望的节点上，将其他可调度节点设置为不可调度（cordon），或使用仅目标节点具备的 `nodeSelector`/标签，并至少保留目标节点可调度。 - -```bash -# 示例：为源节点与目标节点打标签，然后基于标签选择 -kubectl label node target=true --overwrite -``` - -如使用基于标签的选择器，在实例上设置： - -```bash -kubectl patch postgresql -n $NAMESPACE $CLUSTER_NAME --type merge \ - -p '{"spec":{"nodeSelector":{"target":"true"}}}' -``` - -### 3. 逐个迁移成员 - -务必先迁移**非 Leader** 成员。同时删除其 PVC 与 Pod——PVC 的删除会阻塞，直到挂载它的 Pod 消失，因此需并行删除 Pod： - -```bash -# 删除数据 PVC（在 Pod 消失前会一直处于 Terminating） -kubectl delete pvc pgdata-$CLUSTER_NAME-1 -n $NAMESPACE --wait=false - -# 删除 Pod 以释放 PVC -kubectl delete pod $CLUSTER_NAME-1 -n $NAMESPACE -``` - -StatefulSet 会重建 `$CLUSTER_NAME-1`；在被调度的节点上供给新的 PVC，Patroni 从 Leader 重新克隆。 - -### 4. 验证成员已带数据重新加入 - -```bash -kubectl get pod -n $NAMESPACE -o wide | grep $CLUSTER_NAME-1 -kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- patronictl list -``` - -迁移后的成员应恢复为 `Replica` 角色，状态 `running`/`streaming`，`Lag in MB` 为 `0`。在该成员上抽查数据（它是只读 / 处于恢复状态）： - -```bash -kubectl exec -n $NAMESPACE $CLUSTER_NAME-1 -c postgres -- \ - psql -U postgres -tAc "SELECT pg_is_in_recovery();" # 期望为 t -``` - -### 5. 如需迁移（原）Leader - -要迁移 Leader，先执行切换（switchover）让其他成员成为 Leader，然后对原 Leader 重复第 3–4 步： - -```bash -kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- \ - patronictl switchover $CLUSTER_NAME --force -``` - -### 6. 恢复调度 - -待所有成员都位于期望节点后，取消对任何节点的 cordon，并移除临时标签/`nodeSelector`。 - -## 说明 - -- 逐个迁移成员，并在迁移下一个之前等待每个成员完全重新同步（`Lag = 0`），以使集群始终保持健康的法定数量。 -- 对于单实例集群，临时扩容到两个实例，让新成员在目标节点上完成同步，执行切换后再缩容回一个实例——以避免对唯一实例执行删除并重新克隆所导致的停机。 From d80a95e9c477be9ad0b24213b879f098760cb8a8 Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Thu, 18 Jun 2026 07:34:30 +0000 Subject: [PATCH 03/17] docs: address CodeRabbit review on PG solutions - CloudBeaver: retrieve superuser password from the operator-created Secret (postgres..credentials...) instead of a non-existent PGPASSWORD_SUPERUSER env var - node migration: label only the target node to avoid rescheduling onto the source Co-Authored-By: Claude Opus 4.8 (1M context) --- ...How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md | 8 +++++--- ...ow_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md | 8 ++++++-- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md b/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md index 4cd63077f..cbd1cb248 100644 --- a/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md +++ b/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md @@ -139,13 +139,15 @@ Open the URL in a browser. kubectl -n get svc ``` -3. Enter the database user and password. The `postgres` superuser password is available in the cluster as an environment variable / Secret: +3. Enter the database user and password. The Operator stores each role's credentials in a Secret named `..credentials.postgresql.acid.zalan.do`. Retrieve the `postgres` superuser password from its Secret: ```bash - kubectl exec -n -0 -c postgres -- \ - bash -c 'echo $PGPASSWORD_SUPERUSER' + kubectl get secret postgres..credentials.postgresql.acid.zalan.do \ + -n -o jsonpath='{.data.password}' | base64 -d; echo ``` + The same Secret also carries `username`, `host` and `port` keys. + 4. (Optional) Set **Database** to the target database; otherwise CloudBeaver connects to the default `postgres` database. 5. Under **Access Management**, grant the current CloudBeaver user permission to use the new connection. 6. Save and test the connection. SQL editors can now be opened from the browser and queries executed against the PostgreSQL cluster. diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md index b82b457bc..efb7e789f 100644 --- a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md @@ -62,10 +62,14 @@ nodes (or use a `nodeSelector`/label that only the target nodes carry). Keep at least the target node schedulable. ```bash -# Example: label the source and target nodes, then select them -kubectl label node target=true --overwrite +# Label ONLY the target node(s) so the recreated pod can land there and nowhere else +kubectl label node target=true --overwrite ``` +Do **not** label the source node — labeling both source and target would let the +pod reschedule back onto the source. If you prefer cordoning over labels, cordon +all non-target nodes instead and skip the `nodeSelector` step below. + If you use a label-based selector, set it on the instance: ```bash From 3164b7f850320eb8dd0ff05336b74ee763a16893 Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Mon, 20 Jul 2026 04:27:45 +0000 Subject: [PATCH 04/17] docs: add PostgreSQL cross-cluster instance migration guide (KB260720001) Validated end-to-end on operator v4.3.3 across two ACP platforms (NodePort pairing, two-phase switchover, checksum-gated cutover). Includes troubleshooting for the ECO-703 cross-version XCR schema drift and standby-recreate secret cleanup. Co-Authored-By: Claude Fable 5 --- ...e_a_PostgreSQL_Instance_Across_Clusters.md | 305 ++++++++++++++++++ 1 file changed, 305 insertions(+) create mode 100644 docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md new file mode 100644 index 000000000..1e4682215 --- /dev/null +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md @@ -0,0 +1,305 @@ +--- +kind: + - Solution +products: + - Alauda Application Services +ProductsVersion: + - 4.3 +id: KB260720001 +--- + +# PostgreSQL Instance Cross-Cluster Migration Guide (Operator v4.3.3) + +## Background + +### The Challenge + +Sometimes a running PostgreSQL instance must move to a different Kubernetes cluster — cluster decommissioning, hardware refresh, moving a workload between ACP platforms, or consolidating environments. Dump-and-restore migrations require downtime proportional to database size and lose writes made after the dump. + +### The Solution + +This guide migrates a PostgreSQL instance across clusters using the operator's hot standby (cross-cluster replication) feature: the target instance is created as a *standby cluster* that bootstraps directly from the source via streaming replication, stays continuously in sync, and is then promoted in a controlled two-phase switchover. Downtime is limited to the switchover itself (seconds to a couple of minutes), and data integrity is verified with checksums before and after cutover. + +This procedure was validated end-to-end with PostgreSQL Operator v4.3.3 on two separate ACP platforms (source and target in different platforms, connected via NodePort). It builds on the [PostgreSQL Hot Standby Cluster Configuration Guide](./How_to_Use_PostgreSQL_Hot_Standby_Cluster.md) (KB251000009); read that first for concept background. + +## Environment Information + +- PostgreSQL Operator: v4.3.3 on **both** source and target clusters (see [Important Limitations](#important-limitations)) +- ACP: any 4.x cluster able to run the v4.3.3 operator; source and target may be in different ACP platforms +- PostgreSQL: same major version on both sides (this guide uses 16) + +## Important Limitations + +- **The operator version must match on both sides.** Cross-version pairing (e.g. a v4.2+/v4.3 standby against a primary managed by v4.1.x) fails with `pq: column "external_ip" does not exist` and — dangerously — leaves the "standby" running as an empty independent primary (tracked as ECO-703). See [Troubleshooting](#troubleshooting). +- Source and target must run the same PostgreSQL major version. +- The standby cluster must initially be created with `numberOfInstances: 1`; scale it up after promotion. +- `replSvcType` must be identical on both clusters. +- The target cluster must be able to reach the source cluster's node IP + NodePort (standby pulls from primary). Verify this before starting — see Step 2. +- On mixed-architecture target clusters, pin the instance (and ideally the operator) to nodes matching the source architecture. Streaming replication copies the data directory bit-for-bit; PostgreSQL does not support mixed-architecture replication. + +## Migration Overview + +``` +Step 1 Prepare source: enable clusterReplication, record NodePort, baseline checksums +Step 2 Preflight: network reachability, version/arch checks +Step 3 Create standby on target (bootstraps from source, stays streaming) +Step 4 Verify sync: identity, lag, checksums +Step 5 Cutover: two-phase switchover (demote source, promote target) +Step 6 Post-migration: repoint clients; keep reverse standby or dismantle +``` + +## Step 1: Prepare the Source Instance + +If the source instance does not yet have cluster replication enabled, patch it (this is an online change; the operator creates the replication metadata and exposes the master service): + +```bash +SRC_NS="pg-migrate" +SRC_CLUSTER="acid-mig" + +kubectl -n $SRC_NS patch postgresql $SRC_CLUSTER --type merge -p '{ + "spec": { + "clusterReplication": {"enabled": true, "replSvcType": "NodePort"}, + "postgresql": {"parameters": {"max_slot_wal_keep_size": "10GB"}} + } +}' +``` + +> `max_slot_wal_keep_size` bounds WAL retained for the replication slot if the standby disconnects. Size it to your volume: it must fit in the instance's free disk space, and it defines how long a standby outage you can tolerate before a re-bootstrap is needed. + +Record the connection coordinates the standby will use: + +```bash +# NodePort of the source master service +kubectl -n $SRC_NS get svc $SRC_CLUSTER -o jsonpath='{.spec.ports[0].nodePort}{"\n"}' + +# A node IP hosting the instance (any node IP of the cluster works for NodePort) +kubectl -n $SRC_NS get pod -l cluster-name=$SRC_CLUSTER -o jsonpath='{range .items[*]}{.status.hostIP}{"\n"}{end}' +``` + +Confirm the source registered itself in the replication metadata (role `primary`, correct `node_port`): + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -x \ + -c "SELECT * FROM sys_operator.multi_cluster_info;" +``` + +Take a data integrity baseline. Adapt the checksum query to your schema — the point is a number you can compare after migration: + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -d -tA -c " +SELECT relname, n_live_tup FROM pg_stat_user_tables ORDER BY relname;" +# Example per-table checksum: +# SELECT count(*), sum(hashtext(id::text || payload)) FROM your_table; +``` + +Finally, force a checkpoint so the standby's basebackup starts from a consistent point: + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -c "CHECKPOINT;" +``` + +## Step 2: Preflight Checks on the Target + +**Network reachability** — the standby (operator pod *and* PostgreSQL pods) must reach the source NodePort. Test from a pod on the overlay network of the target cluster, on the node(s) where the instance will run: + +```bash +# From any pod with bash on the target cluster: +kubectl exec -- bash -c 'timeout 4 bash -c "echo > /dev/tcp//" && echo OPEN || echo CLOSED' +``` + +If this prints `CLOSED`, stop and fix connectivity first. Note that reachability can differ per node (broken egress on individual nodes has been observed); test from the nodes you will schedule onto. + +**Version check** — both operators must be v4.3.3 (or at least the same version): + +```bash +kubectl -n operators get csv | grep postgres-operator +``` + +**Architecture** — on mixed-architecture targets, decide the node set now and pin with `nodeAffinity` (shown in Step 3). + +## Step 3: Create the Standby on the Target Cluster + +Create the namespace and a bootstrap secret holding the **source** cluster's admin credentials: + +```bash +TGT_NS="pg-migrate" + +# Read the admin password from the source cluster: +kubectl --context -n $SRC_NS get secret \ + postgres.${SRC_CLUSTER}.credentials.postgresql.acid.zalan.do \ + -o jsonpath='{.data.password}' | base64 -d +``` + +```yaml +kind: Secret +apiVersion: v1 +metadata: + name: standby-bootstrap-secret + namespace: pg-migrate +type: kubernetes.io/basic-auth +stringData: + username: postgres + password: "" +``` + +Create the standby instance. Keep PostgreSQL version, parameters, and volume size aligned with the source; start with a single replica: + +```yaml +apiVersion: acid.zalan.do/v1 +kind: postgresql +metadata: + name: acid-mig # name may differ from the source + namespace: pg-migrate +spec: + teamId: acid + numberOfInstances: 1 # required initially for standby clusters + postgresql: + version: "16" # must match the source major version + parameters: + max_slot_wal_keep_size: '10GB' + volume: + size: 5Gi # same capacity as source + storageClass: + # Only needed on mixed-architecture clusters — match the source arch: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/arch + operator: In + values: ["arm64"] + clusterReplication: + enabled: true + isReplica: true + peerHost: "" + peerPort: + replSvcType: NodePort + bootstrapSecret: standby-bootstrap-secret +``` + +What happens on creation: the operator connects to the source, copies the database user credential secrets into the target namespace, creates a local `-xcr` service whose endpoints point at the source nodes, and bootstraps the pod with `pg_basebackup` from the source. `patronictl list` shows `creating replica` during the basebackup (duration depends on database size and link bandwidth), then `Standby Leader | streaming`. + +## Step 4: Verify Synchronization + +All three checks below must pass before cutover. + +**1. Streaming state and shared identity** — the cluster identifier printed by `patronictl list` must be *identical* on source and target (it is the PostgreSQL system identifier). If the target shows a different identifier, it bootstrapped as an independent cluster and contains none of your data — see [Troubleshooting](#troubleshooting). + +```bash +# Target — expect: Standby Leader | streaming +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- patronictl list + +# Source — expect the standby connected, plus the slot active: +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -c \ + "SELECT application_name, client_addr, state FROM pg_stat_replication; + SELECT slot_name, active FROM pg_replication_slots WHERE slot_name='xdc_hotstandby';" +``` + +**2. Replication lag** — write on the source, confirm it appears on the target within seconds, and compare LSNs: + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -tA -c "SELECT pg_current_wal_lsn();" +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -tA -c "SELECT pg_last_wal_replay_lsn();" +``` + +**3. Data checksums** — rerun the Step 1 baseline queries on the target; every value must match. + +## Step 5: Cutover (Two-Phase Switchover) + +Perform the switchover in the documented order — demote first, then promote — so there is never a moment with two writable primaries. + +1. **Stop application writes** to the source (scale writers down, or hold traffic at the application layer). + +2. **Confirm zero lag** (Step 4, check 2 — the two LSNs must be equal once writes stop). + +3. **Phase 1 — demote the source** to a standby: + +```bash +kubectl --context -n $SRC_NS patch postgresql $SRC_CLUSTER --type merge \ + -p '{"spec":{"clusterReplication":{"isReplica":true},"numberOfInstances":1}}' +``` + +Wait until the source shows `Standby Leader` (it may pass through `stopped` briefly). The demoted source finds the target automatically through the replication metadata — no `peerHost` needs to be added to its spec. + +4. **Phase 2 — promote the target** and scale it to full size: + +```bash +kubectl --context -n $TGT_NS patch postgresql acid-mig --type merge \ + -p '{"spec":{"clusterReplication":{"isReplica":false},"numberOfInstances":2}}' +``` + +5. **Gate on the real promotion signal.** Do not rely on `pg_is_in_recovery()` alone — during promotion and scale-up the cluster status can read `Running` while Patroni is still converting roles. Wait until **all** of the following hold: + +```bash +# a) Patroni shows a Leader in state running (a timeline increase here is normal): +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- patronictl list +# b) the CR reports Running: +kubectl -n $TGT_NS get postgresql acid-mig -o jsonpath='{.status.PostgresClusterStatus}{"\n"}' +# c) the new replica is streaming (after scale-up): +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -c \ + "SELECT application_name, state FROM pg_stat_replication;" +``` + +6. **Verify writes and integrity** on the new primary: insert a marker row, rerun the checksum queries, compare with the baseline. + +> A leadership change between the target's pods during promote+scale-up (with an extra timeline bump) is normal operator rolling behavior and does not indicate a problem. + +## Step 6: Post-Migration + +- **Repoint clients** to the target cluster's service (and update any external access such as NodePort/LoadBalancer/ingress used by applications). +- The demoted source is now a live **reverse DR standby** of the target: writes on the target replicate back to it. Choose one: + - **Keep it** as disaster-recovery / rollback insurance (recommended for at least a soak period). Rolling back is the same two-phase switchover in the opposite direction. + - **Dismantle it**: delete the source `postgresql` CR, then its PVCs (PVCs are retained on CR deletion and must be removed manually if you want the storage back). +- Scale/tune the target (replica count, resources, backup schedule, monitoring) to match what the source had. + +## Troubleshooting + +### Standby fails with `pq: column "external_ip" does not exist` — and runs as an empty independent primary + +**Cause:** operator version mismatch between source and target (e.g. source primary managed by v4.1.x, target operator v4.2+/v4.3). The replication metadata table schema differs between these lines and is never migrated (ECO-703). The standby create hook aborts midway and the pod falls through to a normal bootstrap: it comes up as a *fresh, empty, writable* primary while its CR still says `isReplica: true`. + +**Detection:** compare the cluster identifier in `patronictl list` on both sides — different identifier means independent cluster, not a standby. + +**Fix:** upgrade both operators to the same version (preferred). If the source operator cannot be upgraded immediately, add the missing column on the source primary (safe for both versions — all statements reference columns by name): + +```sql +ALTER TABLE sys_operator.multi_cluster_info ADD COLUMN IF NOT EXISTS external_ip CHAR(64) DEFAULT ''; +``` + +Then recreate the standby cleanly — see the next item. + +### Recreating a failed standby: create hook fails with secret "already exists" + +The operator copies the source's credential secrets into the target namespace *before* bootstrapping, and the copy step fails hard if they already exist from a previous attempt. To retry a standby creation from scratch, delete all of: + +```bash +kubectl -n $TGT_NS delete postgresql acid-mig +kubectl -n $TGT_NS delete pvc -l cluster-name=acid-mig # data from the failed attempt must go +kubectl -n $TGT_NS delete secret \ + postgres.acid-mig.credentials.postgresql.acid.zalan.do \ + standby.acid-mig.credentials.postgresql.acid.zalan.do +``` + +then re-apply the standby manifest. + +### Standby stuck in `creating replica` + +The basebackup is either still copying (large database — check network throughput) or cannot connect. Verify Step 2 reachability *from the node the standby pod landed on*; per-node egress differences are a real failure mode. Also confirm the bootstrap secret contains the current source admin password. + +### Replication slot errors (`TypeError ... 'int' and 'NoneType'`) in standby logs + +Known Patroni issue; replication generally continues. See the [Hot Standby guide's troubleshooting section](./How_to_Use_PostgreSQL_Hot_Standby_Cluster.md#troubleshooting) — drop the `xdc_hotstandby` slot if needed. + +## Verification Checklist + +| Check | When | Pass condition | +|---|---|---| +| Network preflight | before Step 3 | target overlay pod reaches `SRC_NODE_IP:NODEPORT` | +| Operator versions | before Step 3 | identical on both clusters | +| Shared system identifier | after Step 3 | `patronictl list` identifier equal on both sides | +| Streaming | after Step 3 | target `Standby Leader / streaming`; source slot `xdc_hotstandby` active | +| Checksums (pre-cutover) | Step 4 | all values match baseline | +| LSN parity | Step 5, writes stopped | `pg_current_wal_lsn()` == `pg_last_wal_replay_lsn()` | +| Promotion | Step 5 | Patroni Leader running + CR Running + replica streaming | +| Checksums (post-cutover) | Step 5 | all values match baseline; new write on target succeeds | +| Reverse replication (if source kept) | Step 6 | target write appears on demoted source | From f06cb0eec4f2fd3729b8c75729d29f976006a3e1 Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Mon, 20 Jul 2026 04:38:55 +0000 Subject: [PATCH 05/17] docs: add full source-dismantle + XCR teardown steps to migration guide Ordered teardown (standby first, then remove clusterReplication from the promoted primary), operator-side cleanup behavior (sys_operator schema drop, xdc_hotstandby slot removal), leftover-object cleanup, and verification. Co-Authored-By: Claude Fable 5 --- ...e_a_PostgreSQL_Instance_Across_Clusters.md | 45 ++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md index 1e4682215..a4acec194 100644 --- a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md @@ -249,9 +249,52 @@ kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -c \ - **Repoint clients** to the target cluster's service (and update any external access such as NodePort/LoadBalancer/ingress used by applications). - The demoted source is now a live **reverse DR standby** of the target: writes on the target replicate back to it. Choose one: - **Keep it** as disaster-recovery / rollback insurance (recommended for at least a soak period). Rolling back is the same two-phase switchover in the opposite direction. - - **Dismantle it**: delete the source `postgresql` CR, then its PVCs (PVCs are retained on CR deletion and must be removed manually if you want the storage back). + - **Dismantle it** and remove the replication setup entirely — see below. - Scale/tune the target (replica count, resources, backup schedule, monitoring) to match what the source had. +### Dismantling the Source and Removing the Replication Configuration + +Perform these steps **in order** — the standby must be gone before the primary's replication configuration is removed, otherwise the standby loses its upstream while still streaming. + +**1. Delete the demoted source instance** (on the source cluster): + +```bash +kubectl --context -n $SRC_NS delete postgresql $SRC_CLUSTER +# PVCs are retained on CR deletion — remove them to reclaim storage: +kubectl --context -n $SRC_NS delete pvc -l cluster-name=$SRC_CLUSTER +``` + +The operator deletes the credential secrets it owns along with the CR. + +**2. Convert the target to a normal (non-replicated) instance** by removing the `clusterReplication` block: + +```bash +kubectl --context -n $TGT_NS patch postgresql acid-mig --type json \ + -p '[{"op":"remove","path":"/spec/clusterReplication"}]' +``` + +This transition is fully handled by the operator: it re-applies normal-primary Patroni configuration, drops the `sys_operator` metadata schema from the database, and removes the `xdc_hotstandby` replication slot. (The reverse — converting an existing normal instance *into* a standby — is not a supported transition; standbys must be created as standbys.) + +**3. Clean up leftover objects** on the target that the operator does not own: + +```bash +kubectl --context -n $TGT_NS delete secret standby-bootstrap-secret +# The -xcr service is owner-referenced to the CR; verify it is gone and +# delete it manually only if it lingers: +kubectl --context -n $TGT_NS get svc acid-mig-xcr +``` + +**4. Verify the target is a clean standalone instance:** + +```bash +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -tA -c \ + "SELECT count(*) FROM pg_replication_slots WHERE slot_name='xdc_hotstandby'; + SELECT count(*) FROM pg_namespace WHERE nspname='sys_operator';" +# Both counts must be 0 +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- patronictl list +# Expect a plain Leader/Replica cluster, no Standby Leader +``` + ## Troubleshooting ### Standby fails with `pq: column "external_ip" does not exist` — and runs as an empty independent primary From 054eafdc2a84ff8c1b3e6177101bf14c1e390862 Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Mon, 20 Jul 2026 05:42:37 +0000 Subject: [PATCH 06/17] docs: correct teardown per live validation (PVC retention, lingering slot/xcr svc) Ran the full teardown on the validated v4.3.3 pair: sys_operator schema drop confirmed automatic; PVC retention is config-dependent (deleted with the CR on v4.3.3); the physical xdc_hotstandby slot and the -xcr service linger and need manual removal. Co-Authored-By: Claude Fable 5 --- ...e_a_PostgreSQL_Instance_Across_Clusters.md | 24 +++++++++++++------ 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md index a4acec194..d9be5392e 100644 --- a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md @@ -260,8 +260,10 @@ Perform these steps **in order** — the standby must be gone before the primary ```bash kubectl --context -n $SRC_NS delete postgresql $SRC_CLUSTER -# PVCs are retained on CR deletion — remove them to reclaim storage: -kubectl --context -n $SRC_NS delete pvc -l cluster-name=$SRC_CLUSTER +# PVC retention on CR deletion depends on operator configuration — check and +# remove any leftovers to reclaim storage: +kubectl --context -n $SRC_NS get pvc -l cluster-name=$SRC_CLUSTER +kubectl --context -n $SRC_NS delete pvc -l cluster-name=$SRC_CLUSTER --ignore-not-found ``` The operator deletes the credential secrets it owns along with the CR. @@ -273,15 +275,23 @@ kubectl --context -n $TGT_NS patch postgresql acid-mig --type json -p '[{"op":"remove","path":"/spec/clusterReplication"}]' ``` -This transition is fully handled by the operator: it re-applies normal-primary Patroni configuration, drops the `sys_operator` metadata schema from the database, and removes the `xdc_hotstandby` replication slot. (The reverse — converting an existing normal instance *into* a standby — is not a supported transition; standbys must be created as standbys.) +The operator handles this transition: it re-applies normal-primary Patroni configuration, drops the `sys_operator` metadata schema from the database, and removes the `xdc_hotstandby` slot from the replication configuration. (The reverse — converting an existing normal instance *into* a standby — is not a supported transition; standbys must be created as standbys.) -**3. Clean up leftover objects** on the target that the operator does not own: +**3. Clean up leftover objects** that the operator does not remove: ```bash +# Bootstrap secret (user-created, never operator-owned): kubectl --context -n $TGT_NS delete secret standby-bootstrap-secret -# The -xcr service is owner-referenced to the CR; verify it is gone and -# delete it manually only if it lingers: -kubectl --context -n $TGT_NS get svc acid-mig-xcr + +# The -xcr service lingers after the role change (it is only removed +# with the CR itself) — delete it: +kubectl --context -n $TGT_NS delete svc acid-mig-xcr --ignore-not-found + +# The physical replication slot can also linger even though it was removed +# from the Patroni configuration — drop it if the Step 4 check finds it: +kubectl --context -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -c \ + "SELECT pg_drop_replication_slot('xdc_hotstandby') + WHERE EXISTS (SELECT 1 FROM pg_replication_slots WHERE slot_name='xdc_hotstandby' AND NOT active);" ``` **4. Verify the target is a clean standalone instance:** From d4206cbe7b707eb5c4e0bda44c214cc3b9bec81f Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Tue, 21 Jul 2026 02:07:28 +0000 Subject: [PATCH 07/17] docs: add workstation-relayed logical migration for isolated clusters For topologies with no inter-cluster network where the workstation reaches both API servers: pg_dump piped between two kubectl exec sessions. Validated live (100k rows, exact checksum parity, clean exit); documents the version-matched-binary and spilo-schema-exclusion requirements found during validation. Co-Authored-By: Claude Fable 5 --- ...e_a_PostgreSQL_Instance_Across_Clusters.md | 45 ++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md index d9be5392e..78f441870 100644 --- a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md @@ -34,7 +34,7 @@ This procedure was validated end-to-end with PostgreSQL Operator v4.3.3 on two s - Source and target must run the same PostgreSQL major version. - The standby cluster must initially be created with `numberOfInstances: 1`; scale it up after promotion. - `replSvcType` must be identical on both clusters. -- The target cluster must be able to reach the source cluster's node IP + NodePort (standby pulls from primary). Verify this before starting — see Step 2. +- The target cluster must be able to reach the source cluster's node IP + NodePort (standby pulls from primary). Verify this before starting — see Step 2. If there is **no network path between the clusters**, the streaming approach cannot work — use the [workstation-relayed logical migration](#alternative-migration-without-inter-cluster-connectivity) instead. - On mixed-architecture target clusters, pin the instance (and ideally the operator) to nodes matching the source architecture. Streaming replication copies the data directory bit-for-bit; PostgreSQL does not support mixed-architecture replication. ## Migration Overview @@ -305,6 +305,49 @@ kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- patronictl list # Expect a plain Leader/Replica cluster, no Standby Leader ``` +## Alternative: Migration Without Inter-Cluster Connectivity + +Use this approach when the clusters have **no network path between them** (isolated networks, firewalled environments) but your workstation can reach both Kubernetes API servers. All data flows through the workstation over the two `kubectl exec` channels — the clusters never talk to each other. + +**Trade-offs versus the streaming approach:** + +| Aspect | Streaming (XCR) | Workstation-relayed logical | +|---|---|---| +| Downtime | Seconds (switchover only) | Full duration of dump+restore | +| Data after copy starts | Continuously replicated | Lost — stop writes first | +| Network requirement | Target reaches source NodePort | Workstation reaches both API servers only | +| Version constraints | Same operator + PG major | None (works across PG majors, operator versions, and CPU architectures) | +| Transfer speed | Cluster-to-cluster bandwidth | Limited by the workstation's bandwidth to both API servers | + +**Procedure:** + +1. **Create the target instance** as a plain instance (no `clusterReplication`). Define application users and databases in the target CR spec (`spec.users` / `spec.databases`) — do not attempt to dump global objects; the operator manages roles. + +2. **Stop application writes** on the source and take the checksum baseline (Step 1 method). + +3. **Relay each database** through the workstation with a single pipe (no intermediate file needed for small/medium databases): + +```bash +kubectl --context -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- \ + /usr/lib/postgresql/16/bin/pg_dump -U postgres -Fc \ + -N metric_helpers -N user_management \ +| kubectl --context -n $TGT_NS exec -i -c postgres -- \ + /usr/lib/postgresql/16/bin/pg_restore -U postgres -d --no-owner -x +``` + + Command details (all validated; omitting them produces noisy or failing restores): + - **Use the version-matched binaries** (`/usr/lib/postgresql//bin/`). The image's default `pg_dump` can be a newer major whose output (e.g. `SET transaction_timeout`) the older target server rejects. + - **`-N metric_helpers -N user_management`** excludes the schemas the operator image pre-creates in every database; without this the restore reports ~20 "already exists" errors. + - **`--no-owner -x`** skips ownership/ACL statements that reference operator-managed roles; objects are owned by the connecting user. Re-grant application permissions afterwards if you use fine-grained ACLs. + - The pipe returns exit code 0 on a clean restore — treat any non-zero exit as a failed migration, drop the target database (`DROP DATABASE` must run as its own single statement), and re-run. + - For large databases, dump to a compressed file on the workstation first (`... pg_dump -Fc ... > db.dump`) for resumability, then restore with `kubectl exec -i ... pg_restore ... < db.dump`. Parallel restore (`-j`) requires copying the file into the target pod rather than streaming stdin. + +4. **Verify checksums** on the target against the baseline, then repoint clients. + +5. **Teardown** is trivial: delete the source instance (and PVCs if retained) — there is no replication configuration to remove. + +Validation reference: a 100k-row database relayed this way completed in under 6 seconds with exact checksum parity (workstation on a proxied connection to both platforms). + ## Troubleshooting ### Standby fails with `pq: column "external_ip" does not exist` — and runs as an empty independent primary From ad657b6cdef9fb361c6520ff06884f32c06afd44 Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Tue, 21 Jul 2026 02:17:12 +0000 Subject: [PATCH 08/17] docs: split network-isolated migration into standalone solution KB260721001 Workstation-relayed logical migration as its own doc, tested verbatim end-to-end before commit. Testing corrected the restore identity: restore as the CR-defined application user (plus --no-comments), otherwise objects land owned by postgres and the app user cannot query them. Migration guide now links to the standalone doc instead of embedding the procedure. Co-Authored-By: Claude Fable 5 --- ...e_a_PostgreSQL_Instance_Across_Clusters.md | 41 +--- ...tance_Between_Network_Isolated_Clusters.md | 212 ++++++++++++++++++ 2 files changed, 214 insertions(+), 39 deletions(-) create mode 100644 docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md index 78f441870..0b0624988 100644 --- a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md @@ -307,46 +307,9 @@ kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- patronictl list ## Alternative: Migration Without Inter-Cluster Connectivity -Use this approach when the clusters have **no network path between them** (isolated networks, firewalled environments) but your workstation can reach both Kubernetes API servers. All data flows through the workstation over the two `kubectl exec` channels — the clusters never talk to each other. +When the clusters have **no network path between them** but your workstation can reach both Kubernetes API servers, the migration can be relayed through the workstation as a logical dump/restore piped between two `kubectl exec` sessions — the clusters never talk to each other. Downtime equals the full copy duration (versus seconds for the streaming switchover), but there are no operator-version, PostgreSQL-major, or CPU-architecture constraints. -**Trade-offs versus the streaming approach:** - -| Aspect | Streaming (XCR) | Workstation-relayed logical | -|---|---|---| -| Downtime | Seconds (switchover only) | Full duration of dump+restore | -| Data after copy starts | Continuously replicated | Lost — stop writes first | -| Network requirement | Target reaches source NodePort | Workstation reaches both API servers only | -| Version constraints | Same operator + PG major | None (works across PG majors, operator versions, and CPU architectures) | -| Transfer speed | Cluster-to-cluster bandwidth | Limited by the workstation's bandwidth to both API servers | - -**Procedure:** - -1. **Create the target instance** as a plain instance (no `clusterReplication`). Define application users and databases in the target CR spec (`spec.users` / `spec.databases`) — do not attempt to dump global objects; the operator manages roles. - -2. **Stop application writes** on the source and take the checksum baseline (Step 1 method). - -3. **Relay each database** through the workstation with a single pipe (no intermediate file needed for small/medium databases): - -```bash -kubectl --context -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- \ - /usr/lib/postgresql/16/bin/pg_dump -U postgres -Fc \ - -N metric_helpers -N user_management \ -| kubectl --context -n $TGT_NS exec -i -c postgres -- \ - /usr/lib/postgresql/16/bin/pg_restore -U postgres -d --no-owner -x -``` - - Command details (all validated; omitting them produces noisy or failing restores): - - **Use the version-matched binaries** (`/usr/lib/postgresql//bin/`). The image's default `pg_dump` can be a newer major whose output (e.g. `SET transaction_timeout`) the older target server rejects. - - **`-N metric_helpers -N user_management`** excludes the schemas the operator image pre-creates in every database; without this the restore reports ~20 "already exists" errors. - - **`--no-owner -x`** skips ownership/ACL statements that reference operator-managed roles; objects are owned by the connecting user. Re-grant application permissions afterwards if you use fine-grained ACLs. - - The pipe returns exit code 0 on a clean restore — treat any non-zero exit as a failed migration, drop the target database (`DROP DATABASE` must run as its own single statement), and re-run. - - For large databases, dump to a compressed file on the workstation first (`... pg_dump -Fc ... > db.dump`) for resumability, then restore with `kubectl exec -i ... pg_restore ... < db.dump`. Parallel restore (`-j`) requires copying the file into the target pod rather than streaming stdin. - -4. **Verify checksums** on the target against the baseline, then repoint clients. - -5. **Teardown** is trivial: delete the source instance (and PVCs if retained) — there is no replication configuration to remove. - -Validation reference: a 100k-row database relayed this way completed in under 6 seconds with exact checksum parity (workstation on a proxied connection to both platforms). +The full validated procedure is a separate solution: [How to Migrate a PostgreSQL Instance Between Network-Isolated Clusters](./How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md) (KB260721001). ## Troubleshooting diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md new file mode 100644 index 000000000..1f7dd878c --- /dev/null +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md @@ -0,0 +1,212 @@ +--- +kind: + - Solution +products: + - Alauda Application Services +ProductsVersion: + - 4.x +id: KB260721001 +--- + +# How to Migrate a PostgreSQL Instance Between Network-Isolated Clusters + +## Background + +### The Challenge + +The streaming-replication migration described in the [PostgreSQL Instance Cross-Cluster Migration Guide](./How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md) requires the target cluster to reach the source cluster's network. In many real deployments the clusters are network-isolated — separate platforms, firewalled sites, disconnected security zones — and no such path exists or can be opened. + +### The Solution + +When an administrator workstation can reach **both** Kubernetes API servers, the migration can be relayed through the workstation: `pg_dump` runs in the source pod, `pg_restore` runs in the target pod, and the data flows between them through two `kubectl exec` channels piped together on the workstation. The clusters never exchange a packet. + +Because the transfer is logical (SQL-level), this method has no version-matching constraints: it works across different operator versions, different PostgreSQL major versions (same or newer on the target), and different CPU architectures. + +**Trade-off:** unlike streaming replication, this is a point-in-time copy. Writes made on the source after the dump starts are not transferred — application writes must be stopped for the whole dump+restore window, so downtime equals the full copy duration. + +## Environment Information + +- PostgreSQL Operator: any 4.x version on each side (versions do **not** need to match) +- PostgreSQL: target major version equal to or newer than the source +- Workstation: `kubectl` access (kubeconfig/context) to both clusters + +## Prerequisites + +- Two kubeconfig contexts on the workstation, one per cluster; verify both work: + +```bash +SRC_CTX=""; SRC_NS=""; SRC_CLUSTER="" +TGT_CTX=""; TGT_NS=""; TGT_CLUSTER="" + +kubectl --context $SRC_CTX -n $SRC_NS get postgresql $SRC_CLUSTER +kubectl --context $TGT_CTX -n $TGT_NS get postgresql $TGT_CLUSTER 2>/dev/null || echo "target instance not created yet" +``` + +- Enough workstation bandwidth to both API servers for the database size (all data flows through the workstation). +- A maintenance window sized to the dump+restore duration. + +## Step 1: Create the Target Instance + +Create the target as a plain instance (no `clusterReplication`), sized like the source. Define the application databases and their owner users **in the CR spec** — the operator creates the roles and databases and manages their credentials; do not attempt to dump global objects (roles) from the source: + +```yaml +apiVersion: acid.zalan.do/v1 +kind: postgresql +metadata: + name: acid-target # $TGT_CLUSTER + namespace: target-namespace +spec: + teamId: acid + numberOfInstances: 2 + postgresql: + version: "16" # same as source, or newer major + users: + appuser: [] # owner role for the migrated database + databases: + appdb: appuser # database -> owner + volume: + size: 5Gi # at least the source's data size + storageClass: +``` + +Wait for status `Running`: + +```bash +kubectl --context $TGT_CTX -n $TGT_NS get postgresql $TGT_CLUSTER \ + -o jsonpath='{.status.PostgresClusterStatus}{"\n"}' +``` + +## Step 2: Stop Writes and Take a Baseline + +Stop application writes on the source, then record row counts (and, for stronger guarantees, per-table checksums) to compare after the restore: + +```bash +SRC_POD=$(kubectl --context $SRC_CTX -n $SRC_NS get pod \ + -l spilo-role=master,cluster-name=$SRC_CLUSTER -o jsonpath='{.items[0].metadata.name}') + +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -d appdb -tA -c \ + "SELECT relname, n_live_tup FROM pg_stat_user_tables ORDER BY relname;" +# Example per-table checksum: +# SELECT count(*), sum(hashtext(id::text || payload)) FROM your_table; +``` + +## Step 3: Relay the Database Through the Workstation + +Identify the target master pod and read the application user's password from the operator-managed secret, then run the pipe. One pipe per database: + +```bash +TGT_POD=$(kubectl --context $TGT_CTX -n $TGT_NS get pod \ + -l spilo-role=master,cluster-name=$TGT_CLUSTER -o jsonpath='{.items[0].metadata.name}') + +APP_PW=$(kubectl --context $TGT_CTX -n $TGT_NS get secret \ + appuser.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do \ + -o jsonpath='{.data.password}' | base64 -d) + +PG_BIN=/usr/lib/postgresql/16/bin # match the SOURCE server major (see notes) + +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + $PG_BIN/pg_dump -U postgres -Fc --no-comments \ + -N metric_helpers -N user_management appdb \ +| kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ + env PGPASSWORD="$APP_PW" \ + $PG_BIN/pg_restore -U appuser -h localhost -d appdb --no-owner -x +echo "pipe exit: $?" +``` + +The pipe must exit `0`. Command details — each flag below was added because omitting it produced a failing, noisy, or broken-permissions restore during validation: + +- **Restore as the application user** (`-U appuser` with its password), not as `postgres`: with `--no-owner`, restored objects are owned by the connecting user. Restoring as `postgres` leaves every table owned by `postgres`, and the application user cannot even `SELECT` afterwards (`permission denied for table ...`). Restoring as the CR-defined owner lands the ownership correctly with no post-step. +- **Version-matched binaries** (`/usr/lib/postgresql//bin/`): the container's default `pg_dump` can be a newer major whose output (e.g. `SET transaction_timeout`) an older target server rejects. The images ship binaries for every supported major, so pick the path explicitly. When migrating to a *newer* target major, use the **target** major's `pg_dump` (also present in the source pod's image). +- **`--no-comments`**: the dump captures `COMMENT ON EXTENSION` for the extensions the image pre-installs (`pg_stat_statements`, `pg_stat_kcache`, `set_user`); executing those requires superuser, which the application user is not. Object comments are dropped — restore them separately as `postgres` if you rely on them. +- **`-N metric_helpers -N user_management`**: the operator image pre-creates these schemas in every database; without the exclusion the restore reports ~20 `already exists` errors. +- **`-x`**: skips ACL statements that reference roles managed by the source operator and absent on the target. Re-grant privileges for any additional (non-owner) users on the target afterwards. + +### Large databases + +A single pipe has no resume capability. For large databases, dump to a compressed file on the workstation first, then restore from it: + +```bash +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + $PG_BIN/pg_dump -U postgres -Fc --no-comments \ + -N metric_helpers -N user_management appdb > appdb.dump + +kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ + env PGPASSWORD="$APP_PW" \ + $PG_BIN/pg_restore -U appuser -h localhost -d appdb --no-owner -x < appdb.dump +``` + +Parallel restore (`pg_restore -j N`) cannot read from stdin; to use it, copy the dump file into the target pod (`kubectl cp`) and restore from the local path. + +## Step 4: Verify + +Rerun the Step 2 queries on the target and compare — counts/checksums must match exactly: + +```bash +kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ + psql -U postgres -d appdb -tA -c \ + "SELECT relname, n_live_tup FROM pg_stat_user_tables ORDER BY relname;" +``` + +Then confirm the application user can actually query its data with the target-managed credentials (this catches ownership mistakes immediately): + +```bash +kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ + env PGPASSWORD="$APP_PW" psql -U appuser -h localhost -d appdb -tA -c \ + "SELECT current_user, count(*) FROM ;" +``` + +## Step 5: Cut Over and Clean Up + +- Repoint applications to the target instance's service and re-enable writes. +- Delete the source instance when satisfied. There is no replication configuration to tear down: + +```bash +kubectl --context $SRC_CTX -n $SRC_NS delete postgresql $SRC_CLUSTER +# PVC retention depends on operator configuration — remove leftovers: +kubectl --context $SRC_CTX -n $SRC_NS delete pvc -l cluster-name=$SRC_CLUSTER --ignore-not-found +``` + +## Troubleshooting + +### Application user gets `permission denied for table ...` after a successful restore + +The restore was run as `postgres` instead of the application user, so all objects are owned by `postgres`. Either rerun the restore as the application user (Step 3), or transfer ownership in place: + +```sql +DO $$ +DECLARE r record; +BEGIN + FOR r IN SELECT tablename FROM pg_tables WHERE schemaname = 'public' LOOP + EXECUTE format('ALTER TABLE public.%I OWNER TO appuser', r.tablename); + END LOOP; + FOR r IN SELECT sequencename FROM pg_sequences WHERE schemaname = 'public' LOOP + EXECUTE format('ALTER SEQUENCE public.%I OWNER TO appuser', r.sequencename); + END LOOP; +END $$; +``` + +### Restore reports `must be owner of extension pg_stat_statements` (and similar) + +`--no-comments` was omitted from `pg_dump`; the extension comments require superuser. The data restores fine — the errors only break the clean exit code. Rerun with `--no-comments` for a verifiable result. + +### Restore reports `unrecognized configuration parameter "transaction_timeout"` + +The dump was taken with a `pg_dump` newer than the target server (typically the image's default binary). Rerun using the explicit version-matched binary path (Step 3). + +### Restore reports `schema "metric_helpers" already exists` (and similar) + +The `-N metric_helpers -N user_management` exclusions were omitted. These errors are harmless for the excluded schemas' objects, but rerun with the exclusions for a clean, verifiable exit code. + +### Rerunning after a failed restore + +Drop and recreate the target database first — a partial restore leaves objects that collide (`relation already exists`, duplicate-key COPY failures). Note `DROP DATABASE` must be executed as its own single statement: + +```bash +kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- psql -U postgres -c "DROP DATABASE appdb" +kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- psql -U postgres -c "CREATE DATABASE appdb OWNER appuser" +``` + +### Pipe is slow + +Throughput is bounded by the workstation's link to both API servers (every byte traverses it twice: exec stream in, exec stream out). Run the relay from a machine with good connectivity to both platforms (e.g. a jump host), or use the file-based variant to at least make progress resumable. From e89eec2033a00fdfcd300dfcc32b67615aa06a50 Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Tue, 21 Jul 2026 03:04:28 +0000 Subject: [PATCH 09/17] docs: address review findings on KB260721001 - Verify with exact per-table counts (query_to_xml) instead of the n_live_tup estimate, in both baseline and post-restore steps - Run the relay pipe under pipefail and report per-side PIPESTATUS so a source-side dump failure cannot be masked - Add extension preflight: inventory pg_extension on the source and pre-create missing extensions as postgres on the target; new troubleshooting entry for 'permission denied to create extension' - Scope the ownership guidance to the single-owner operator layout and extend the fallback ALTER note beyond public tables/sequences - Add security notes: password visibility in local process listings, dump-file permissions/retention, TLS transport - List required Kubernetes permissions and Spilo image assumptions in Prerequisites - Reword the 'no version-matching constraints' claim (here and in the companion guide's pointer) to same-or-newer target major, no downgrades Co-Authored-By: Claude Fable 5 --- ...e_a_PostgreSQL_Instance_Across_Clusters.md | 2 +- ...tance_Between_Network_Isolated_Clusters.md | 58 +++++++++++++++---- 2 files changed, 49 insertions(+), 11 deletions(-) diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md index 0b0624988..db2fb7e8b 100644 --- a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md @@ -307,7 +307,7 @@ kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- patronictl list ## Alternative: Migration Without Inter-Cluster Connectivity -When the clusters have **no network path between them** but your workstation can reach both Kubernetes API servers, the migration can be relayed through the workstation as a logical dump/restore piped between two `kubectl exec` sessions — the clusters never talk to each other. Downtime equals the full copy duration (versus seconds for the streaming switchover), but there are no operator-version, PostgreSQL-major, or CPU-architecture constraints. +When the clusters have **no network path between them** but your workstation can reach both Kubernetes API servers, the migration can be relayed through the workstation as a logical dump/restore piped between two `kubectl exec` sessions — the clusters never talk to each other. Downtime equals the full copy duration (versus seconds for the streaming switchover), but there are no operator-version or CPU-architecture constraints, and the PostgreSQL major only needs to be the same or newer on the target. The full validated procedure is a separate solution: [How to Migrate a PostgreSQL Instance Between Network-Isolated Clusters](./How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md) (KB260721001). diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md index 1f7dd878c..cf101a4ad 100644 --- a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md @@ -20,7 +20,7 @@ The streaming-replication migration described in the [PostgreSQL Instance Cross- When an administrator workstation can reach **both** Kubernetes API servers, the migration can be relayed through the workstation: `pg_dump` runs in the source pod, `pg_restore` runs in the target pod, and the data flows between them through two `kubectl exec` channels piped together on the workstation. The clusters never exchange a packet. -Because the transfer is logical (SQL-level), this method has no version-matching constraints: it works across different operator versions, different PostgreSQL major versions (same or newer on the target), and different CPU architectures. +Because the transfer is logical (SQL-level), this method has no same-version requirement: it works across different operator versions, across CPU architectures, and from an older PostgreSQL major to the same or a newer major on the target. Migrating to an **older** PostgreSQL major (a downgrade) is not supported. **Trade-off:** unlike streaming replication, this is a point-in-time copy. Writes made on the source after the dump starts are not transferred — application writes must be stopped for the whole dump+restore window, so downtime equals the full copy duration. @@ -44,6 +44,8 @@ kubectl --context $TGT_CTX -n $TGT_NS get postgresql $TGT_CLUSTER 2>/dev/null || - Enough workstation bandwidth to both API servers for the database size (all data flows through the workstation). - A maintenance window sized to the dump+restore duration. +- Sufficient Kubernetes permissions in both namespaces: `get` on `postgresqls.acid.zalan.do`, `pods`, and `secrets`; `create` on `pods/exec`; plus `create` on `postgresqls` (target side) and, for the final cleanup, `delete` on `postgresqls` and `persistentvolumeclaims` (source side). +- The commands assume the operator's standard Spilo image: pod labels `spilo-role`/`cluster-name`, database container named `postgres`, and client binaries for every supported major under `/usr/lib/postgresql//bin/`. Custom or non-Spilo images require adapting these labels and paths. ## Step 1: Create the Target Instance @@ -76,47 +78,69 @@ kubectl --context $TGT_CTX -n $TGT_NS get postgresql $TGT_CLUSTER \ -o jsonpath='{.status.PostgresClusterStatus}{"\n"}' ``` -## Step 2: Stop Writes and Take a Baseline +## Step 2: Stop Writes, Inventory Extensions, and Take a Baseline -Stop application writes on the source, then record row counts (and, for stronger guarantees, per-table checksums) to compare after the restore: +Stop application writes on the source, then record **exact** row counts (and, for stronger guarantees, per-table checksums) to compare after the restore. Do not use `pg_stat_user_tables.n_live_tup` for this comparison — it is a planner statistics *estimate*, not an exact count: ```bash SRC_POD=$(kubectl --context $SRC_CTX -n $SRC_NS get pod \ -l spilo-role=master,cluster-name=$SRC_CLUSTER -o jsonpath='{.items[0].metadata.name}') +# Exact row count of every user table: kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ psql -U postgres -d appdb -tA -c \ - "SELECT relname, n_live_tup FROM pg_stat_user_tables ORDER BY relname;" + "SELECT relname, (xpath('/row/c/text()', query_to_xml(format( + 'SELECT count(*) AS c FROM %I.%I', schemaname, relname), false, true, '')))[1]::text::bigint + FROM pg_stat_user_tables ORDER BY relname;" # Example per-table checksum: # SELECT count(*), sum(hashtext(id::text || payload)) FROM your_table; ``` +Also inventory the extensions the database uses — extension creation typically requires superuser, so any extension that the image does not pre-install must be created on the target ahead of the restore (Step 3): + +```bash +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -d appdb -tA -c "SELECT extname, extversion FROM pg_extension ORDER BY 1;" +``` + ## Step 3: Relay the Database Through the Workstation -Identify the target master pod and read the application user's password from the operator-managed secret, then run the pipe. One pipe per database: +Identify the target master pod, and pre-create (as `postgres`) any extension from the Step 2 inventory that is not already present on the target — the restore runs as the application user, which cannot create extensions: ```bash TGT_POD=$(kubectl --context $TGT_CTX -n $TGT_NS get pod \ -l spilo-role=master,cluster-name=$TGT_CLUSTER -o jsonpath='{.items[0].metadata.name}') +kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ + psql -U postgres -d appdb -tA -c "SELECT extname FROM pg_extension ORDER BY 1;" +# For each extension the source has but the target lacks (requires the extension +# packages to be present in the target image): +kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ + psql -U postgres -d appdb -c "CREATE EXTENSION IF NOT EXISTS ;" +``` + +Read the application user's password from the operator-managed secret, then run the pipe — in a shell with `pipefail` set, otherwise a source-side `pg_dump`/`kubectl` failure is masked by the exit status of the last command. One pipe per database: + +```bash APP_PW=$(kubectl --context $TGT_CTX -n $TGT_NS get secret \ appuser.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do \ -o jsonpath='{.data.password}' | base64 -d) PG_BIN=/usr/lib/postgresql/16/bin # match the SOURCE server major (see notes) +set -o pipefail kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ $PG_BIN/pg_dump -U postgres -Fc --no-comments \ -N metric_helpers -N user_management appdb \ | kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ env PGPASSWORD="$APP_PW" \ $PG_BIN/pg_restore -U appuser -h localhost -d appdb --no-owner -x -echo "pipe exit: $?" +echo "pipe exit: $? (dump=${PIPESTATUS[0]}, restore=${PIPESTATUS[1]})" ``` -The pipe must exit `0`. Command details — each flag below was added because omitting it produced a failing, noisy, or broken-permissions restore during validation: +The pipe must exit `0` (with `pipefail`, a non-zero status from *either* side surfaces; `PIPESTATUS` shows which side failed). Command details — each flag below was added because omitting it produced a failing, noisy, or broken-permissions restore during validation: -- **Restore as the application user** (`-U appuser` with its password), not as `postgres`: with `--no-owner`, restored objects are owned by the connecting user. Restoring as `postgres` leaves every table owned by `postgres`, and the application user cannot even `SELECT` afterwards (`permission denied for table ...`). Restoring as the CR-defined owner lands the ownership correctly with no post-step. +- **Restore as the application user** (`-U appuser` with its password), not as `postgres`: with `--no-owner`, restored objects are owned by the connecting user. Restoring as `postgres` leaves every table owned by `postgres`, and the application user cannot even `SELECT` afterwards (`permission denied for table ...`). Restoring as the CR-defined owner lands the ownership correctly with no post-step. **Scope note:** this pattern fits the operator's standard layout — one application database wholly owned by one CR-defined user. If your database has multiple owning roles, `SECURITY DEFINER` functions, or objects in non-`public` schemas with distinct owners, `--no-owner` collapses all ownership onto the connecting user; plan an explicit post-restore ownership and grant pass for those objects. - **Version-matched binaries** (`/usr/lib/postgresql//bin/`): the container's default `pg_dump` can be a newer major whose output (e.g. `SET transaction_timeout`) an older target server rejects. The images ship binaries for every supported major, so pick the path explicitly. When migrating to a *newer* target major, use the **target** major's `pg_dump` (also present in the source pod's image). - **`--no-comments`**: the dump captures `COMMENT ON EXTENSION` for the extensions the image pre-installs (`pg_stat_statements`, `pg_stat_kcache`, `set_user`); executing those requires superuser, which the application user is not. Object comments are dropped — restore them separately as `postgres` if you rely on them. - **`-N metric_helpers -N user_management`**: the operator image pre-creates these schemas in every database; without the exclusion the restore reports ~20 `already exists` errors. @@ -138,14 +162,22 @@ kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ Parallel restore (`pg_restore -j N`) cannot read from stdin; to use it, copy the dump file into the target pod (`kubectl cp`) and restore from the local path. +### Security notes + +- While the restore runs, `$APP_PW` is expanded into the workstation's `kubectl` argument list and is visible in local process listings (`ps`). Run the relay from a trusted, single-user workstation, and do not enable shell tracing (`set -x`) around these commands. `unset APP_PW` when finished. +- A dump file is a full plaintext logical copy of the database. Create it with restrictive permissions (`umask 077` before dumping, or `chmod 600` immediately after), encrypt it at rest if your policy requires, and delete it once the migration is verified. +- The data itself only traverses the two TLS-protected `kubectl exec` channels; nothing is exposed on the network beyond the two Kubernetes API connections. + ## Step 4: Verify -Rerun the Step 2 queries on the target and compare — counts/checksums must match exactly: +Rerun the Step 2 baseline queries on the target and compare — exact counts/checksums must match exactly: ```bash kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ psql -U postgres -d appdb -tA -c \ - "SELECT relname, n_live_tup FROM pg_stat_user_tables ORDER BY relname;" + "SELECT relname, (xpath('/row/c/text()', query_to_xml(format( + 'SELECT count(*) AS c FROM %I.%I', schemaname, relname), false, true, '')))[1]::text::bigint + FROM pg_stat_user_tables ORDER BY relname;" ``` Then confirm the application user can actually query its data with the target-managed credentials (this catches ownership mistakes immediately): @@ -186,6 +218,12 @@ BEGIN END $$; ``` +This block covers tables and sequences in the `public` schema only. Views, functions, types, and objects in other schemas need analogous `ALTER ... OWNER TO` statements (enumerate them via `pg_views`, `pg_proc`, `pg_type`, or `\dn`/`\df` in psql). + +### Restore reports `permission denied to create extension ...` + +The source database uses an extension that is not pre-installed on the target, and the application user cannot create it. Create the extension as `postgres` on the target database (Step 3 pre-create), then rerun the restore. If `CREATE EXTENSION` itself fails with a missing-file error, the extension's packages are not present in the target image at all — it must be added to the image/instance before this migration can carry that database. + ### Restore reports `must be owner of extension pg_stat_statements` (and similar) `--no-comments` was omitted from `pg_dump`; the extension comments require superuser. The data restores fine — the errors only break the clean exit code. Rerun with `--no-comments` for a verifiable result. From 4e10e288475490905c5e99a467405368c9719c8b Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Wed, 22 Jul 2026 01:40:43 +0000 Subject: [PATCH 10/17] docs: make per-database scope explicit in KB260721001 Executor feedback: the examples read as if 'appdb' were the whole instance. Step 1 now enumerates the source's application databases (pg_database query, validated live) and states that Steps 2-4 repeat once per database with the name/owner substituted; Step 2 and the Step 3 pipe restate it at the point of use. The postgres maintenance database is explicitly excluded from migration. Co-Authored-By: Claude Fable 5 --- ...tance_Between_Network_Isolated_Clusters.md | 23 ++++++++++++++----- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md index cf101a4ad..2e0bfd135 100644 --- a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md @@ -47,7 +47,21 @@ kubectl --context $TGT_CTX -n $TGT_NS get postgresql $TGT_CLUSTER 2>/dev/null || - Sufficient Kubernetes permissions in both namespaces: `get` on `postgresqls.acid.zalan.do`, `pods`, and `secrets`; `create` on `pods/exec`; plus `create` on `postgresqls` (target side) and, for the final cleanup, `delete` on `postgresqls` and `persistentvolumeclaims` (source side). - The commands assume the operator's standard Spilo image: pod labels `spilo-role`/`cluster-name`, database container named `postgres`, and client binaries for every supported major under `/usr/lib/postgresql//bin/`. Custom or non-Spilo images require adapting these labels and paths. -## Step 1: Create the Target Instance +## Step 1: Enumerate Source Databases and Create the Target Instance + +An instance can hold several application databases, and **the whole procedure migrates one database at a time**. First list what the source actually holds: + +```bash +SRC_POD=$(kubectl --context $SRC_CTX -n $SRC_NS get pod \ + -l spilo-role=master,cluster-name=$SRC_CLUSTER -o jsonpath='{.items[0].metadata.name}') + +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -tA -c \ + "SELECT datname, pg_get_userbyid(datdba) AS owner FROM pg_database + WHERE NOT datistemplate AND datname <> 'postgres' ORDER BY 1;" +``` + +Every database in this list must be declared in the target CR below (`spec.databases`, with its owner in `spec.users`), and Steps 2–4 are then **repeated once per database**, substituting the database name and owner each round. The examples throughout use a single database `appdb` owned by `appuser`. The `postgres` maintenance database is managed by the operator/image on each side and is not migrated. Create the target as a plain instance (no `clusterReplication`), sized like the source. Define the application databases and their owner users **in the CR spec** — the operator creates the roles and databases and manages their credentials; do not attempt to dump global objects (roles) from the source: @@ -80,12 +94,9 @@ kubectl --context $TGT_CTX -n $TGT_NS get postgresql $TGT_CLUSTER \ ## Step 2: Stop Writes, Inventory Extensions, and Take a Baseline -Stop application writes on the source, then record **exact** row counts (and, for stronger guarantees, per-table checksums) to compare after the restore. Do not use `pg_stat_user_tables.n_live_tup` for this comparison — it is a planner statistics *estimate*, not an exact count: +Run Steps 2–4 once per database from the Step 1 list; the commands below use `appdb`. Stop application writes on the source, then record **exact** row counts (and, for stronger guarantees, per-table checksums) to compare after the restore. Do not use `pg_stat_user_tables.n_live_tup` for this comparison — it is a planner statistics *estimate*, not an exact count: ```bash -SRC_POD=$(kubectl --context $SRC_CTX -n $SRC_NS get pod \ - -l spilo-role=master,cluster-name=$SRC_CLUSTER -o jsonpath='{.items[0].metadata.name}') - # Exact row count of every user table: kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ psql -U postgres -d appdb -tA -c \ @@ -119,7 +130,7 @@ kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ psql -U postgres -d appdb -c "CREATE EXTENSION IF NOT EXISTS ;" ``` -Read the application user's password from the operator-managed secret, then run the pipe — in a shell with `pipefail` set, otherwise a source-side `pg_dump`/`kubectl` failure is masked by the exit status of the last command. One pipe per database: +Read the application user's password from the operator-managed secret, then run the pipe — in a shell with `pipefail` set, otherwise a source-side `pg_dump`/`kubectl` failure is masked by the exit status of the last command. This transfers **one database**; run it once per database from the Step 1 list, substituting the database name and its owner user (in the secret name, `-U`, and `-d`) each time: ```bash APP_PW=$(kubectl --context $TGT_CTX -n $TGT_NS get secret \ From 8e2cb49be62d18231ca9b565219b0e010195c020 Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Wed, 22 Jul 2026 02:00:42 +0000 Subject: [PATCH 11/17] docs: add scripted whole-instance migration to KB260721001 Reduce the procedure from repeat-Steps-2-4-per-database to one script run: enumerate all application databases, auto-create ones missing on the target (e.g. postgres-owned databases created outside the CR), pre-create extensions, transfer each database as its owner, verify exact per-table counts, and report PASS/FAIL per database. Step 1 also gains a generator for the target CR users:/databases: sections. Validated end-to-end on the longterm-c -> longterm-b pair with a 3-database fixture (two CR-owned + one stray postgres-owned): all databases PASS first try, ownership and per-user access verified. Co-Authored-By: Claude Fable 5 --- ...tance_Between_Network_Isolated_Clusters.md | 104 +++++++++++++++++- 1 file changed, 103 insertions(+), 1 deletion(-) diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md index 2e0bfd135..409fa3e6f 100644 --- a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md @@ -61,7 +61,24 @@ kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ WHERE NOT datistemplate AND datname <> 'postgres' ORDER BY 1;" ``` -Every database in this list must be declared in the target CR below (`spec.databases`, with its owner in `spec.users`), and Steps 2–4 are then **repeated once per database**, substituting the database name and owner each round. The examples throughout use a single database `appdb` owned by `appuser`. The `postgres` maintenance database is managed by the operator/image on each side and is not migrated. +Every database in this list must be declared in the target CR below (`spec.databases`, with its owner in `spec.users`), and Steps 2–4 are then **repeated once per database**, substituting the database name and owner each round — or run all of them in one pass with the [whole-instance script](#scripted-whole-instance-migration) below. The examples throughout use a single database `appdb` owned by `appuser`. The `postgres` maintenance database is managed by the operator/image on each side and is not migrated. + +The `users:`/`databases:` sections of the target CR can be generated directly from the source instead of hand-written (databases owned by `postgres` are deliberately excluded — the migration script creates those on the fly): + +```bash +{ + echo " users:" + kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -tA -c \ + "SELECT DISTINCT ' ' || pg_get_userbyid(datdba) || ': []' FROM pg_database + WHERE NOT datistemplate AND datname <> 'postgres' AND pg_get_userbyid(datdba) <> 'postgres';" + echo " databases:" + kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -tA -c \ + "SELECT ' ' || datname || ': ' || pg_get_userbyid(datdba) FROM pg_database + WHERE NOT datistemplate AND datname <> 'postgres' AND pg_get_userbyid(datdba) <> 'postgres' ORDER BY 1;" +} +``` Create the target as a plain instance (no `clusterReplication`), sized like the source. Define the application databases and their owner users **in the CR spec** — the operator creates the roles and databases and manages their credentials; do not attempt to dump global objects (roles) from the source: @@ -210,6 +227,91 @@ kubectl --context $SRC_CTX -n $SRC_NS delete postgresql $SRC_CLUSTER kubectl --context $SRC_CTX -n $SRC_NS delete pvc -l cluster-name=$SRC_CLUSTER --ignore-not-found ``` +## Scripted Whole-Instance Migration + +The manual steps above migrate one database at a time. The script below performs the same procedure for **every** application database of the source instance in a single run: it enumerates the databases, pre-creates extensions, transfers each database as its owner, and verifies exact per-table row counts — reporting `PASS`/`FAIL` per database and exiting non-zero if anything failed. + +Before running it: the target instance must exist and be `Running` (Step 1), and **application writes on the source must already be stopped** (Step 2) — the script does not stop them for you. Databases missing on the target (e.g. created outside the CR spec, owned by `postgres`) are created on the fly, provided their owner role exists on the target. For stronger guarantees than row counts, add per-table checksum comparison (Step 2) for your critical tables afterwards. + +```bash +#!/usr/bin/env bash +# Whole-instance PostgreSQL migration relayed through the workstation. +# Migrates every application database of $SRC_CLUSTER into $TGT_CLUSTER. +set -u -o pipefail + +SRC_CTX=""; SRC_NS=""; SRC_CLUSTER="" +TGT_CTX=""; TGT_NS=""; TGT_CLUSTER="" + +SRC_POD=$(kubectl --context "$SRC_CTX" -n "$SRC_NS" get pod \ + -l spilo-role=master,cluster-name="$SRC_CLUSTER" -o jsonpath='{.items[0].metadata.name}') +TGT_POD=$(kubectl --context "$TGT_CTX" -n "$TGT_NS" get pod \ + -l spilo-role=master,cluster-name="$TGT_CLUSTER" -o jsonpath='{.items[0].metadata.name}') + +srcsql() { local db=$1; shift; kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- psql -U postgres -d "$db" -tA -v ON_ERROR_STOP=1 "$@"; } +tgtsql() { local db=$1; shift; kubectl --context "$TGT_CTX" -n "$TGT_NS" exec "$TGT_POD" -c postgres -- psql -U postgres -d "$db" -tA -v ON_ERROR_STOP=1 "$@"; } + +# Client binaries matching the SOURCE server major (present in both Spilo images). +SRC_MAJOR=$(srcsql postgres -c "SHOW server_version;" | cut -d. -f1) +PG_BIN=/usr/lib/postgresql/$SRC_MAJOR/bin +echo "Source PostgreSQL major: $SRC_MAJOR (client binaries: $PG_BIN)" + +COUNT_QUERY="SELECT relname, (xpath('/row/c/text()', query_to_xml(format( + 'SELECT count(*) AS c FROM %I.%I', schemaname, relname), false, true, '')))[1]::text::bigint + FROM pg_stat_user_tables ORDER BY relname;" + +FAILED="" +while read -r DB OWNER; do + [ -z "$DB" ] && continue + echo "=== migrating database: $DB (owner: $OWNER) ===" + + # Ensure the database exists on the target (covers databases created + # outside the CR spec; the owner role must already exist on the target). + if [ "$(tgtsql postgres -c "SELECT 1 FROM pg_database WHERE datname = '$DB';")" != "1" ]; then + echo " creating database $DB on target" + tgtsql postgres -c "CREATE DATABASE \"$DB\" OWNER \"$OWNER\";" >/dev/null || { FAILED="$FAILED $DB(create)"; continue; } + fi + + # Pre-create the source's extensions (extension creation needs superuser). + for EXT in $(srcsql "$DB" -c "SELECT extname FROM pg_extension;"); do + tgtsql "$DB" -c "CREATE EXTENSION IF NOT EXISTS \"$EXT\";" >/dev/null 2>&1 \ + || echo " WARNING: could not create extension $EXT on target — restore may fail" + done + + # Baseline on the source (exact counts). + SRC_COUNTS=$(srcsql "$DB" -c "$COUNT_QUERY") + + # Restore as the database's owner, with the target-managed password. + OWNER_PW=$(kubectl --context "$TGT_CTX" -n "$TGT_NS" get secret \ + "$OWNER.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do" \ + -o jsonpath='{.data.password}' | base64 -d) || { FAILED="$FAILED $DB(secret)"; continue; } + + kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- \ + "$PG_BIN/pg_dump" -U postgres -Fc --no-comments \ + -N metric_helpers -N user_management "$DB" \ + | kubectl --context "$TGT_CTX" -n "$TGT_NS" exec -i "$TGT_POD" -c postgres -- \ + env PGPASSWORD="$OWNER_PW" \ + "$PG_BIN/pg_restore" -U "$OWNER" -h localhost -d "$DB" --no-owner -x + RC=$? + + # Verify: exact counts must match the baseline. + TGT_COUNTS=$(tgtsql "$DB" -c "$COUNT_QUERY") + if [ "$RC" -eq 0 ] && [ "$SRC_COUNTS" = "$TGT_COUNTS" ]; then + echo " PASS: $DB (tables/rows identical)" + else + echo " FAIL: $DB (pipe exit $RC; counts $( [ "$SRC_COUNTS" = "$TGT_COUNTS" ] && echo match || echo DIFFER ))" + FAILED="$FAILED $DB" + fi +done < <(srcsql postgres -F' ' -c \ + "SELECT datname, pg_get_userbyid(datdba) FROM pg_database + WHERE NOT datistemplate AND datname <> 'postgres' ORDER BY 1;") + +echo +if [ -n "$FAILED" ]; then echo "MIGRATION INCOMPLETE — failed:$FAILED"; exit 1; fi +echo "MIGRATION COMPLETE — all databases verified." +``` + +With this, the human operations reduce to: create the target CR (Step 1, with the generated `users:`/`databases:` fragment), stop application writes, run the script, and cut over (Step 5). The per-database sections above remain the reference for what each part of the script does and for troubleshooting a `FAIL`. + ## Troubleshooting ### Application user gets `permission denied for table ...` after a successful restore From dc53cdafe6c81bab40c2c31c21d5441ef7d91f04 Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Wed, 22 Jul 2026 02:10:20 +0000 Subject: [PATCH 12/17] docs: restructure KB260721001 script-first MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scripted whole-instance migration is now the main procedure (Steps 1-5: create target CR, stop writes, run script, verify app access, cut over). The per-database manual steps move to a 'Reference: Manual Single-Database Transfer' section — kept because debugging a per-database FAIL, single-database/partial migrations, and the resumable file-based variant for very large databases all need the standalone commands and flag rationale. Troubleshooting pointers retargeted; added explicit warning not to rerun the whole script after a partial success (re-restoring populated databases collides). Co-Authored-By: Claude Fable 5 --- ...tance_Between_Network_Isolated_Clusters.md | 250 +++++++++--------- 1 file changed, 124 insertions(+), 126 deletions(-) diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md index 409fa3e6f..3ce730167 100644 --- a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md @@ -20,6 +20,8 @@ The streaming-replication migration described in the [PostgreSQL Instance Cross- When an administrator workstation can reach **both** Kubernetes API servers, the migration can be relayed through the workstation: `pg_dump` runs in the source pod, `pg_restore` runs in the target pod, and the data flows between them through two `kubectl exec` channels piped together on the workstation. The clusters never exchange a packet. +The procedure is four human operations: create the target instance (Step 1), stop application writes (Step 2), run the migration script (Step 3), and cut over (Steps 4–5). The script migrates **every** application database of the instance and verifies each one. + Because the transfer is logical (SQL-level), this method has no same-version requirement: it works across different operator versions, across CPU architectures, and from an older PostgreSQL major to the same or a newer major on the target. Migrating to an **older** PostgreSQL major (a downgrade) is not supported. **Trade-off:** unlike streaming replication, this is a point-in-time copy. Writes made on the source after the dump starts are not transferred — application writes must be stopped for the whole dump+restore window, so downtime equals the full copy duration. @@ -28,7 +30,7 @@ Because the transfer is logical (SQL-level), this method has no same-version req - PostgreSQL Operator: any 4.x version on each side (versions do **not** need to match) - PostgreSQL: target major version equal to or newer than the source -- Workstation: `kubectl` access (kubeconfig/context) to both clusters +- Workstation: `bash` and `kubectl` access (kubeconfig/context) to both clusters ## Prerequisites @@ -47,9 +49,9 @@ kubectl --context $TGT_CTX -n $TGT_NS get postgresql $TGT_CLUSTER 2>/dev/null || - Sufficient Kubernetes permissions in both namespaces: `get` on `postgresqls.acid.zalan.do`, `pods`, and `secrets`; `create` on `pods/exec`; plus `create` on `postgresqls` (target side) and, for the final cleanup, `delete` on `postgresqls` and `persistentvolumeclaims` (source side). - The commands assume the operator's standard Spilo image: pod labels `spilo-role`/`cluster-name`, database container named `postgres`, and client binaries for every supported major under `/usr/lib/postgresql//bin/`. Custom or non-Spilo images require adapting these labels and paths. -## Step 1: Enumerate Source Databases and Create the Target Instance +## Step 1: Create the Target Instance -An instance can hold several application databases, and **the whole procedure migrates one database at a time**. First list what the source actually holds: +List the application databases and owners the source actually holds: ```bash SRC_POD=$(kubectl --context $SRC_CTX -n $SRC_NS get pod \ @@ -61,9 +63,9 @@ kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ WHERE NOT datistemplate AND datname <> 'postgres' ORDER BY 1;" ``` -Every database in this list must be declared in the target CR below (`spec.databases`, with its owner in `spec.users`), and Steps 2–4 are then **repeated once per database**, substituting the database name and owner each round — or run all of them in one pass with the [whole-instance script](#scripted-whole-instance-migration) below. The examples throughout use a single database `appdb` owned by `appuser`. The `postgres` maintenance database is managed by the operator/image on each side and is not migrated. +Every CR-managed database in this list must be declared in the target CR (`spec.databases`, with its owner in `spec.users`) — the operator creates the roles and databases and manages their credentials; do not attempt to dump global objects (roles) from the source. The `postgres` maintenance database is managed by the operator/image on each side and is not migrated. Databases owned by `postgres` (created outside the CR spec) do not go into the CR — the migration script creates them on the target automatically. -The `users:`/`databases:` sections of the target CR can be generated directly from the source instead of hand-written (databases owned by `postgres` are deliberately excluded — the migration script creates those on the fly): +The `users:`/`databases:` sections of the target CR can be generated directly from the source instead of hand-written: ```bash { @@ -80,7 +82,7 @@ The `users:`/`databases:` sections of the target CR can be generated directly fr } ``` -Create the target as a plain instance (no `clusterReplication`), sized like the source. Define the application databases and their owner users **in the CR spec** — the operator creates the roles and databases and manages their credentials; do not attempt to dump global objects (roles) from the source: +Create the target as a plain instance (no `clusterReplication`), sized like the source: ```yaml apiVersion: acid.zalan.do/v1 @@ -94,9 +96,9 @@ spec: postgresql: version: "16" # same as source, or newer major users: - appuser: [] # owner role for the migrated database + appuser: [] # generated above databases: - appdb: appuser # database -> owner + appdb: appuser # generated above volume: size: 5Gi # at least the source's data size storageClass: @@ -109,129 +111,23 @@ kubectl --context $TGT_CTX -n $TGT_NS get postgresql $TGT_CLUSTER \ -o jsonpath='{.status.PostgresClusterStatus}{"\n"}' ``` -## Step 2: Stop Writes, Inventory Extensions, and Take a Baseline - -Run Steps 2–4 once per database from the Step 1 list; the commands below use `appdb`. Stop application writes on the source, then record **exact** row counts (and, for stronger guarantees, per-table checksums) to compare after the restore. Do not use `pg_stat_user_tables.n_live_tup` for this comparison — it is a planner statistics *estimate*, not an exact count: - -```bash -# Exact row count of every user table: -kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ - psql -U postgres -d appdb -tA -c \ - "SELECT relname, (xpath('/row/c/text()', query_to_xml(format( - 'SELECT count(*) AS c FROM %I.%I', schemaname, relname), false, true, '')))[1]::text::bigint - FROM pg_stat_user_tables ORDER BY relname;" -# Example per-table checksum: -# SELECT count(*), sum(hashtext(id::text || payload)) FROM your_table; -``` - -Also inventory the extensions the database uses — extension creation typically requires superuser, so any extension that the image does not pre-install must be created on the target ahead of the restore (Step 3): - -```bash -kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ - psql -U postgres -d appdb -tA -c "SELECT extname, extversion FROM pg_extension ORDER BY 1;" -``` - -## Step 3: Relay the Database Through the Workstation - -Identify the target master pod, and pre-create (as `postgres`) any extension from the Step 2 inventory that is not already present on the target — the restore runs as the application user, which cannot create extensions: - -```bash -TGT_POD=$(kubectl --context $TGT_CTX -n $TGT_NS get pod \ - -l spilo-role=master,cluster-name=$TGT_CLUSTER -o jsonpath='{.items[0].metadata.name}') - -kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ - psql -U postgres -d appdb -tA -c "SELECT extname FROM pg_extension ORDER BY 1;" -# For each extension the source has but the target lacks (requires the extension -# packages to be present in the target image): -kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ - psql -U postgres -d appdb -c "CREATE EXTENSION IF NOT EXISTS ;" -``` - -Read the application user's password from the operator-managed secret, then run the pipe — in a shell with `pipefail` set, otherwise a source-side `pg_dump`/`kubectl` failure is masked by the exit status of the last command. This transfers **one database**; run it once per database from the Step 1 list, substituting the database name and its owner user (in the secret name, `-U`, and `-d`) each time: - -```bash -APP_PW=$(kubectl --context $TGT_CTX -n $TGT_NS get secret \ - appuser.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do \ - -o jsonpath='{.data.password}' | base64 -d) - -PG_BIN=/usr/lib/postgresql/16/bin # match the SOURCE server major (see notes) - -set -o pipefail -kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ - $PG_BIN/pg_dump -U postgres -Fc --no-comments \ - -N metric_helpers -N user_management appdb \ -| kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ - env PGPASSWORD="$APP_PW" \ - $PG_BIN/pg_restore -U appuser -h localhost -d appdb --no-owner -x -echo "pipe exit: $? (dump=${PIPESTATUS[0]}, restore=${PIPESTATUS[1]})" -``` - -The pipe must exit `0` (with `pipefail`, a non-zero status from *either* side surfaces; `PIPESTATUS` shows which side failed). Command details — each flag below was added because omitting it produced a failing, noisy, or broken-permissions restore during validation: - -- **Restore as the application user** (`-U appuser` with its password), not as `postgres`: with `--no-owner`, restored objects are owned by the connecting user. Restoring as `postgres` leaves every table owned by `postgres`, and the application user cannot even `SELECT` afterwards (`permission denied for table ...`). Restoring as the CR-defined owner lands the ownership correctly with no post-step. **Scope note:** this pattern fits the operator's standard layout — one application database wholly owned by one CR-defined user. If your database has multiple owning roles, `SECURITY DEFINER` functions, or objects in non-`public` schemas with distinct owners, `--no-owner` collapses all ownership onto the connecting user; plan an explicit post-restore ownership and grant pass for those objects. -- **Version-matched binaries** (`/usr/lib/postgresql//bin/`): the container's default `pg_dump` can be a newer major whose output (e.g. `SET transaction_timeout`) an older target server rejects. The images ship binaries for every supported major, so pick the path explicitly. When migrating to a *newer* target major, use the **target** major's `pg_dump` (also present in the source pod's image). -- **`--no-comments`**: the dump captures `COMMENT ON EXTENSION` for the extensions the image pre-installs (`pg_stat_statements`, `pg_stat_kcache`, `set_user`); executing those requires superuser, which the application user is not. Object comments are dropped — restore them separately as `postgres` if you rely on them. -- **`-N metric_helpers -N user_management`**: the operator image pre-creates these schemas in every database; without the exclusion the restore reports ~20 `already exists` errors. -- **`-x`**: skips ACL statements that reference roles managed by the source operator and absent on the target. Re-grant privileges for any additional (non-owner) users on the target afterwards. +## Step 2: Stop Writes -### Large databases +Stop application writes on the source — the transfer is a point-in-time copy, and anything written after the dump starts is lost. -A single pipe has no resume capability. For large databases, dump to a compressed file on the workstation first, then restore from it: +The migration script (Step 3) records and compares exact per-table row counts automatically. For stronger guarantees on critical tables, additionally record per-table checksums now and re-check them in Step 4: ```bash kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ - $PG_BIN/pg_dump -U postgres -Fc --no-comments \ - -N metric_helpers -N user_management appdb > appdb.dump - -kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ - env PGPASSWORD="$APP_PW" \ - $PG_BIN/pg_restore -U appuser -h localhost -d appdb --no-owner -x < appdb.dump -``` - -Parallel restore (`pg_restore -j N`) cannot read from stdin; to use it, copy the dump file into the target pod (`kubectl cp`) and restore from the local path. - -### Security notes - -- While the restore runs, `$APP_PW` is expanded into the workstation's `kubectl` argument list and is visible in local process listings (`ps`). Run the relay from a trusted, single-user workstation, and do not enable shell tracing (`set -x`) around these commands. `unset APP_PW` when finished. -- A dump file is a full plaintext logical copy of the database. Create it with restrictive permissions (`umask 077` before dumping, or `chmod 600` immediately after), encrypt it at rest if your policy requires, and delete it once the migration is verified. -- The data itself only traverses the two TLS-protected `kubectl exec` channels; nothing is exposed on the network beyond the two Kubernetes API connections. - -## Step 4: Verify - -Rerun the Step 2 baseline queries on the target and compare — exact counts/checksums must match exactly: - -```bash -kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ - psql -U postgres -d appdb -tA -c \ - "SELECT relname, (xpath('/row/c/text()', query_to_xml(format( - 'SELECT count(*) AS c FROM %I.%I', schemaname, relname), false, true, '')))[1]::text::bigint - FROM pg_stat_user_tables ORDER BY relname;" -``` - -Then confirm the application user can actually query its data with the target-managed credentials (this catches ownership mistakes immediately): - -```bash -kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ - env PGPASSWORD="$APP_PW" psql -U appuser -h localhost -d appdb -tA -c \ - "SELECT current_user, count(*) FROM ;" -``` - -## Step 5: Cut Over and Clean Up - -- Repoint applications to the target instance's service and re-enable writes. -- Delete the source instance when satisfied. There is no replication configuration to tear down: - -```bash -kubectl --context $SRC_CTX -n $SRC_NS delete postgresql $SRC_CLUSTER -# PVC retention depends on operator configuration — remove leftovers: -kubectl --context $SRC_CTX -n $SRC_NS delete pvc -l cluster-name=$SRC_CLUSTER --ignore-not-found + psql -U postgres -d -tA -c \ + "SELECT count(*), sum(hashtext(id::text || payload)) FROM ;" ``` -## Scripted Whole-Instance Migration +## Step 3: Run the Migration Script -The manual steps above migrate one database at a time. The script below performs the same procedure for **every** application database of the source instance in a single run: it enumerates the databases, pre-creates extensions, transfers each database as its owner, and verifies exact per-table row counts — reporting `PASS`/`FAIL` per database and exiting non-zero if anything failed. +The script performs the whole migration in one run: it enumerates the source's application databases, creates any that are missing on the target (e.g. `postgres`-owned databases created outside the CR spec — their owner role must exist on the target), pre-creates each database's extensions, transfers each database as its owner, and verifies exact per-table row counts — reporting `PASS`/`FAIL` per database and exiting non-zero if anything failed. -Before running it: the target instance must exist and be `Running` (Step 1), and **application writes on the source must already be stopped** (Step 2) — the script does not stop them for you. Databases missing on the target (e.g. created outside the CR spec, owned by `postgres`) are created on the fly, provided their owner role exists on the target. For stronger guarantees than row counts, add per-table checksum comparison (Step 2) for your critical tables afterwards. +Fill in the six variables at the top, then run it with `bash`: ```bash #!/usr/bin/env bash @@ -310,13 +206,113 @@ if [ -n "$FAILED" ]; then echo "MIGRATION INCOMPLETE — failed:$FAILED"; exit 1 echo "MIGRATION COMPLETE — all databases verified." ``` -With this, the human operations reduce to: create the target CR (Step 1, with the generated `users:`/`databases:` fragment), stop application writes, run the script, and cut over (Step 5). The per-database sections above remain the reference for what each part of the script does and for troubleshooting a `FAIL`. +If a database reports `FAIL`, see [Troubleshooting](#troubleshooting) and the [manual single-database transfer](#reference-manual-single-database-transfer) reference — it explains what each part of the script does, so the failing database can be transferred and debugged in isolation. For very large databases, prefer the file-based variant described there (a straight pipe is not resumable). + +### Security notes + +- While a restore runs, the owner password is expanded into the workstation's `kubectl` argument list and is visible in local process listings (`ps`). Run the migration from a trusted, single-user workstation, and do not enable shell tracing (`set -x`) around these commands. +- A dump file (file-based variant) is a full plaintext logical copy of the database. Create it with restrictive permissions (`umask 077` before dumping, or `chmod 600` immediately after), encrypt it at rest if your policy requires, and delete it once the migration is verified. +- The data itself only traverses the two TLS-protected `kubectl exec` channels; nothing is exposed on the network beyond the two Kubernetes API connections. + +## Step 4: Verify Application Access + +The script has already verified per-table row counts. Confirm in addition that each application user can actually query its data with the **target**-managed credentials (this catches ownership problems immediately), and re-check any Step 2 checksums: + +```bash +APP_PW=$(kubectl --context $TGT_CTX -n $TGT_NS get secret \ + .$TGT_CLUSTER.credentials.postgresql.acid.zalan.do \ + -o jsonpath='{.data.password}' | base64 -d) + +kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ + env PGPASSWORD="$APP_PW" psql -U -h localhost -d -tA -c \ + "SELECT current_user, count(*) FROM ;" +``` + +## Step 5: Cut Over and Clean Up + +- Repoint applications to the target instance's service and re-enable writes. +- Delete the source instance when satisfied. There is no replication configuration to tear down: + +```bash +kubectl --context $SRC_CTX -n $SRC_NS delete postgresql $SRC_CLUSTER +# PVC retention depends on operator configuration — remove leftovers: +kubectl --context $SRC_CTX -n $SRC_NS delete pvc -l cluster-name=$SRC_CLUSTER --ignore-not-found +``` + +## Reference: Manual Single-Database Transfer + +This section is the script's inner loop as standalone commands. Use it to debug a database the script reported as `FAIL`, to migrate a single database only, or to handle very large databases with the resumable file-based variant. The examples transfer a database `appdb` owned by `appuser`. + +First inventory the source database's extensions and pre-create (as `postgres`) any that are missing on the target — the restore runs as the application user, which cannot create extensions: + +```bash +TGT_POD=$(kubectl --context $TGT_CTX -n $TGT_NS get pod \ + -l spilo-role=master,cluster-name=$TGT_CLUSTER -o jsonpath='{.items[0].metadata.name}') + +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -d appdb -tA -c "SELECT extname, extversion FROM pg_extension ORDER BY 1;" +kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ + psql -U postgres -d appdb -c "CREATE EXTENSION IF NOT EXISTS ;" +``` + +Record the exact row counts on the source (do not use `pg_stat_user_tables.n_live_tup` for this — it is a planner statistics *estimate*): + +```bash +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -d appdb -tA -c \ + "SELECT relname, (xpath('/row/c/text()', query_to_xml(format( + 'SELECT count(*) AS c FROM %I.%I', schemaname, relname), false, true, '')))[1]::text::bigint + FROM pg_stat_user_tables ORDER BY relname;" +``` + +Read the application user's password from the operator-managed secret, then run the pipe — in a shell with `pipefail` set, otherwise a source-side `pg_dump`/`kubectl` failure is masked by the exit status of the last command: + +```bash +APP_PW=$(kubectl --context $TGT_CTX -n $TGT_NS get secret \ + appuser.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do \ + -o jsonpath='{.data.password}' | base64 -d) + +PG_BIN=/usr/lib/postgresql/16/bin # match the SOURCE server major (see notes) + +set -o pipefail +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + $PG_BIN/pg_dump -U postgres -Fc --no-comments \ + -N metric_helpers -N user_management appdb \ +| kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ + env PGPASSWORD="$APP_PW" \ + $PG_BIN/pg_restore -U appuser -h localhost -d appdb --no-owner -x +echo "pipe exit: $? (dump=${PIPESTATUS[0]}, restore=${PIPESTATUS[1]})" +``` + +The pipe must exit `0` (with `pipefail`, a non-zero status from *either* side surfaces; `PIPESTATUS` shows which side failed). Rerun the count query on the target afterwards and compare with the baseline. Command details — each flag below was added because omitting it produced a failing, noisy, or broken-permissions restore during validation: + +- **Restore as the application user** (`-U appuser` with its password), not as `postgres`: with `--no-owner`, restored objects are owned by the connecting user. Restoring as `postgres` leaves every table owned by `postgres`, and the application user cannot even `SELECT` afterwards (`permission denied for table ...`). Restoring as the CR-defined owner lands the ownership correctly with no post-step. **Scope note:** this pattern fits the operator's standard layout — one application database wholly owned by one CR-defined user. If your database has multiple owning roles, `SECURITY DEFINER` functions, or objects in non-`public` schemas with distinct owners, `--no-owner` collapses all ownership onto the connecting user; plan an explicit post-restore ownership and grant pass for those objects. +- **Version-matched binaries** (`/usr/lib/postgresql//bin/`): the container's default `pg_dump` can be a newer major whose output (e.g. `SET transaction_timeout`) an older target server rejects. The images ship binaries for every supported major, so pick the path explicitly (the script auto-detects this from `SHOW server_version`). When migrating to a *newer* target major, use the **target** major's `pg_dump` (also present in the source pod's image). +- **`--no-comments`**: the dump captures `COMMENT ON EXTENSION` for the extensions the image pre-installs (`pg_stat_statements`, `pg_stat_kcache`, `set_user`); executing those requires superuser, which the application user is not. Object comments are dropped — restore them separately as `postgres` if you rely on them. +- **`-N metric_helpers -N user_management`**: the operator image pre-creates these schemas in every database; without the exclusion the restore reports ~20 `already exists` errors. +- **`-x`**: skips ACL statements that reference roles managed by the source operator and absent on the target. Re-grant privileges for any additional (non-owner) users on the target afterwards. + +### Large databases + +A single pipe has no resume capability. For large databases, dump to a compressed file on the workstation first, then restore from it: + +```bash +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + $PG_BIN/pg_dump -U postgres -Fc --no-comments \ + -N metric_helpers -N user_management appdb > appdb.dump + +kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ + env PGPASSWORD="$APP_PW" \ + $PG_BIN/pg_restore -U appuser -h localhost -d appdb --no-owner -x < appdb.dump +``` + +Parallel restore (`pg_restore -j N`) cannot read from stdin; to use it, copy the dump file into the target pod (`kubectl cp`) and restore from the local path. ## Troubleshooting ### Application user gets `permission denied for table ...` after a successful restore -The restore was run as `postgres` instead of the application user, so all objects are owned by `postgres`. Either rerun the restore as the application user (Step 3), or transfer ownership in place: +The restore was run as `postgres` instead of the application user, so all objects are owned by `postgres`. Either rerun the restore as the application user (see the [reference section](#reference-manual-single-database-transfer)), or transfer ownership in place: ```sql DO $$ @@ -335,7 +331,7 @@ This block covers tables and sequences in the `public` schema only. Views, funct ### Restore reports `permission denied to create extension ...` -The source database uses an extension that is not pre-installed on the target, and the application user cannot create it. Create the extension as `postgres` on the target database (Step 3 pre-create), then rerun the restore. If `CREATE EXTENSION` itself fails with a missing-file error, the extension's packages are not present in the target image at all — it must be added to the image/instance before this migration can carry that database. +The source database uses an extension that is not pre-installed on the target, and the application user cannot create it. The script warns about this (`WARNING: could not create extension`). Create the extension as `postgres` on the target database, then rerun the restore. If `CREATE EXTENSION` itself fails with a missing-file error, the extension's packages are not present in the target image at all — it must be added to the image/instance before this migration can carry that database. ### Restore reports `must be owner of extension pg_stat_statements` (and similar) @@ -343,7 +339,7 @@ The source database uses an extension that is not pre-installed on the target, a ### Restore reports `unrecognized configuration parameter "transaction_timeout"` -The dump was taken with a `pg_dump` newer than the target server (typically the image's default binary). Rerun using the explicit version-matched binary path (Step 3). +The dump was taken with a `pg_dump` newer than the target server (typically the image's default binary). Rerun using the explicit version-matched binary path (the script auto-detects it; in the manual commands set `PG_BIN` to the source server's major). ### Restore reports `schema "metric_helpers" already exists` (and similar) @@ -358,6 +354,8 @@ kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- psql -U postg kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- psql -U postgres -c "CREATE DATABASE appdb OWNER appuser" ``` +Then redo **only the failed database** with the [manual transfer](#reference-manual-single-database-transfer). Do not rerun the whole script after a partial success: it would restore into the databases that already transferred, and the resulting `already exists` collisions would mark them as failed. + ### Pipe is slow Throughput is bounded by the workstation's link to both API servers (every byte traverses it twice: exec stream in, exec stream out). Run the relay from a machine with good connectivity to both platforms (e.g. a jump host), or use the file-based variant to at least make progress resumable. From 9bf2b25f07c162d242df3d6b4d3862cdfff7d520 Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Wed, 22 Jul 2026 02:29:55 +0000 Subject: [PATCH 13/17] docs: preserve application passwords across the migration (KB260721001) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 1 gains a 'Preserve application credentials' sub-step: copy each application user's credential secret from the source into the target namespace BEFORE creating the target CR — the operator adopts existing secrets and sets role passwords from them instead of generating new ones, so applications keep their credentials and only the endpoint changes at cutover. System users (postgres/standby) are skipped. New troubleshooting entry for the CR-created-first case: patch the secret AND ALTER ROLE together. Live-validated on the C->B pair: pre-staged secrets adopted (target secret == source password, login with source password OK), full scripted migration PASS, migrated data queried on the target with the source-era passwords for both users. Co-Authored-By: Claude Fable 5 --- ...tance_Between_Network_Isolated_Clusters.md | 37 ++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md index 3ce730167..f47980e21 100644 --- a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md @@ -82,6 +82,24 @@ The `users:`/`databases:` sections of the target CR can be generated directly fr } ``` +### Preserve application credentials + +By default the target operator generates **new** passwords for the CR users, which would force every application to be re-configured at cutover. To keep the source passwords, copy each application user's credential secret into the target namespace **before creating the CR** — when the operator finds an existing credential secret, it adopts it and sets the role's password from it instead of generating a new one: + +```bash +for SEC in $(kubectl --context $SRC_CTX -n $SRC_NS get secret \ + -l application=spilo,cluster-name=$SRC_CLUSTER -o name); do + U=$(kubectl --context $SRC_CTX -n $SRC_NS get $SEC -o jsonpath='{.data.username}' | base64 -d) + case "$U" in postgres|standby) continue ;; esac # system users stay operator-managed + PW=$(kubectl --context $SRC_CTX -n $SRC_NS get $SEC -o jsonpath='{.data.password}' | base64 -d) + kubectl --context $TGT_CTX -n $TGT_NS create secret generic \ + "$U.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do" \ + --from-literal=username="$U" --from-literal=password="$PW" +done +``` + +The `postgres` and `standby` system users are deliberately skipped — they belong to each instance's own operator. The ordering matters: pre-stage the secrets first, then create the CR (see [Troubleshooting](#troubleshooting) if the CR was created first). + Create the target as a plain instance (no `clusterReplication`), sized like the source: ```yaml @@ -216,7 +234,7 @@ If a database reports `FAIL`, see [Troubleshooting](#troubleshooting) and the [m ## Step 4: Verify Application Access -The script has already verified per-table row counts. Confirm in addition that each application user can actually query its data with the **target**-managed credentials (this catches ownership problems immediately), and re-check any Step 2 checksums: +The script has already verified per-table row counts. Confirm in addition that each application user can actually query its data with the **target**-managed credentials (this catches ownership problems immediately), and re-check any Step 2 checksums. If you pre-staged the credential secrets in Step 1, this password is identical to the source one — applications keep their existing credentials and only the connection endpoint changes at cutover: ```bash APP_PW=$(kubectl --context $TGT_CTX -n $TGT_NS get secret \ @@ -356,6 +374,23 @@ kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- psql -U postg Then redo **only the failed database** with the [manual transfer](#reference-manual-single-database-transfer). Do not rerun the whole script after a partial success: it would restore into the databases that already transferred, and the resulting `already exists` collisions would mark them as failed. +### Application passwords changed after the migration + +The target CR was created **before** the credential secrets were pre-staged (Step 1), so the operator generated new passwords. To restore the source credentials after the fact, update both the secret and the role together — the secret alone is not enough, and an `ALTER ROLE` alone would be reverted by the operator's next sync from the secret: + +```bash +SRC_PW=$(kubectl --context $SRC_CTX -n $SRC_NS get secret \ + appuser.$SRC_CLUSTER.credentials.postgresql.acid.zalan.do -o jsonpath='{.data.password}' | base64 -d) + +kubectl --context $TGT_CTX -n $TGT_NS patch secret \ + appuser.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do \ + -p "{\"stringData\":{\"password\":\"$SRC_PW\"}}" +kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ + psql -U postgres -c "ALTER ROLE appuser PASSWORD '$SRC_PW';" +``` + +Verify with a login test as in Step 4. + ### Pipe is slow Throughput is bounded by the workstation's link to both API servers (every byte traverses it twice: exec stream in, exec stream out). Run the relay from a machine with good connectivity to both platforms (e.g. a jump host), or use the file-based variant to at least make progress resumable. From 2407ef0d5a35a1c522215e6d29ee5507ed21985a Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Wed, 22 Jul 2026 02:44:05 +0000 Subject: [PATCH 14/17] docs: drop the manual-transfer reference section from KB260721001 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The script now accepts database names as arguments (bash migrate.sh appdb) to migrate a subset — validated live with two sequential single-database runs — which was the reference section's main job (redoing one FAIL). Its remaining unique content is folded in place: the file-based large-database variant becomes a compact Step 3 subsection, the -x re-grant note moves to Step 4, and the multi-owner scope note plus newer-major binary tip move into the troubleshooting entries that need them. Doc shrinks by a net 40 lines with a single flow: 5 steps + troubleshooting. Co-Authored-By: Claude Fable 5 --- ...tance_Between_Network_Isolated_Clusters.md | 108 ++++++------------ 1 file changed, 34 insertions(+), 74 deletions(-) diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md index f47980e21..76a99f71d 100644 --- a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md @@ -145,7 +145,7 @@ kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ The script performs the whole migration in one run: it enumerates the source's application databases, creates any that are missing on the target (e.g. `postgres`-owned databases created outside the CR spec — their owner role must exist on the target), pre-creates each database's extensions, transfers each database as its owner, and verifies exact per-table row counts — reporting `PASS`/`FAIL` per database and exiting non-zero if anything failed. -Fill in the six variables at the top, then run it with `bash`: +Fill in the six variables at the top, then run it with `bash`. Without arguments it migrates every database; pass database names (`bash migrate.sh appdb`) to migrate only those — used to redo a single database after a failure: ```bash #!/usr/bin/env bash @@ -176,6 +176,10 @@ COUNT_QUERY="SELECT relname, (xpath('/row/c/text()', query_to_xml(format( FAILED="" while read -r DB OWNER; do [ -z "$DB" ] && continue + # With arguments, migrate only the named databases (e.g. to redo one FAIL). + if [ "$#" -gt 0 ]; then + case " $* " in *" $DB "*) ;; *) continue ;; esac + fi echo "=== migrating database: $DB (owner: $OWNER) ===" # Ensure the database exists on the target (covers databases created @@ -224,7 +228,25 @@ if [ -n "$FAILED" ]; then echo "MIGRATION INCOMPLETE — failed:$FAILED"; exit 1 echo "MIGRATION COMPLETE — all databases verified." ``` -If a database reports `FAIL`, see [Troubleshooting](#troubleshooting) and the [manual single-database transfer](#reference-manual-single-database-transfer) reference — it explains what each part of the script does, so the failing database can be transferred and debugged in isolation. For very large databases, prefer the file-based variant described there (a straight pipe is not resumable). +If a database reports `FAIL`, see [Troubleshooting](#troubleshooting); after fixing the cause, redo just that database by passing its name to the script. + +### Large databases + +A straight pipe is not resumable. For a large database, dump to a file on the workstation first, then restore from it — same flags as the script; `` is the owner's password from the target secret: + +```bash +PG_BIN=/usr/lib/postgresql//bin + +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + $PG_BIN/pg_dump -U postgres -Fc --no-comments \ + -N metric_helpers -N user_management > db.dump + +kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ + env PGPASSWORD="" \ + $PG_BIN/pg_restore -U -h localhost -d --no-owner -x < db.dump +``` + +Parallel restore (`pg_restore -j N`) cannot read from stdin; to use it, copy the dump file into the target pod (`kubectl cp`) and restore from the local path. ### Security notes @@ -237,6 +259,9 @@ If a database reports `FAIL`, see [Troubleshooting](#troubleshooting) and the [m The script has already verified per-table row counts. Confirm in addition that each application user can actually query its data with the **target**-managed credentials (this catches ownership problems immediately), and re-check any Step 2 checksums. If you pre-staged the credential secrets in Step 1, this password is identical to the source one — applications keep their existing credentials and only the connection endpoint changes at cutover: ```bash +TGT_POD=$(kubectl --context $TGT_CTX -n $TGT_NS get pod \ + -l spilo-role=master,cluster-name=$TGT_CLUSTER -o jsonpath='{.items[0].metadata.name}') + APP_PW=$(kubectl --context $TGT_CTX -n $TGT_NS get secret \ .$TGT_CLUSTER.credentials.postgresql.acid.zalan.do \ -o jsonpath='{.data.password}' | base64 -d) @@ -246,6 +271,8 @@ kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ "SELECT current_user, count(*) FROM ;" ``` +The dump deliberately skips ACL statements (`-x`) because they reference roles managed by the source operator — if additional (non-owner) users need privileges on the migrated data, re-grant them on the target now. + ## Step 5: Cut Over and Clean Up - Repoint applications to the target instance's service and re-enable writes. @@ -257,80 +284,11 @@ kubectl --context $SRC_CTX -n $SRC_NS delete postgresql $SRC_CLUSTER kubectl --context $SRC_CTX -n $SRC_NS delete pvc -l cluster-name=$SRC_CLUSTER --ignore-not-found ``` -## Reference: Manual Single-Database Transfer - -This section is the script's inner loop as standalone commands. Use it to debug a database the script reported as `FAIL`, to migrate a single database only, or to handle very large databases with the resumable file-based variant. The examples transfer a database `appdb` owned by `appuser`. - -First inventory the source database's extensions and pre-create (as `postgres`) any that are missing on the target — the restore runs as the application user, which cannot create extensions: - -```bash -TGT_POD=$(kubectl --context $TGT_CTX -n $TGT_NS get pod \ - -l spilo-role=master,cluster-name=$TGT_CLUSTER -o jsonpath='{.items[0].metadata.name}') - -kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ - psql -U postgres -d appdb -tA -c "SELECT extname, extversion FROM pg_extension ORDER BY 1;" -kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ - psql -U postgres -d appdb -c "CREATE EXTENSION IF NOT EXISTS ;" -``` - -Record the exact row counts on the source (do not use `pg_stat_user_tables.n_live_tup` for this — it is a planner statistics *estimate*): - -```bash -kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ - psql -U postgres -d appdb -tA -c \ - "SELECT relname, (xpath('/row/c/text()', query_to_xml(format( - 'SELECT count(*) AS c FROM %I.%I', schemaname, relname), false, true, '')))[1]::text::bigint - FROM pg_stat_user_tables ORDER BY relname;" -``` - -Read the application user's password from the operator-managed secret, then run the pipe — in a shell with `pipefail` set, otherwise a source-side `pg_dump`/`kubectl` failure is masked by the exit status of the last command: - -```bash -APP_PW=$(kubectl --context $TGT_CTX -n $TGT_NS get secret \ - appuser.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do \ - -o jsonpath='{.data.password}' | base64 -d) - -PG_BIN=/usr/lib/postgresql/16/bin # match the SOURCE server major (see notes) - -set -o pipefail -kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ - $PG_BIN/pg_dump -U postgres -Fc --no-comments \ - -N metric_helpers -N user_management appdb \ -| kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ - env PGPASSWORD="$APP_PW" \ - $PG_BIN/pg_restore -U appuser -h localhost -d appdb --no-owner -x -echo "pipe exit: $? (dump=${PIPESTATUS[0]}, restore=${PIPESTATUS[1]})" -``` - -The pipe must exit `0` (with `pipefail`, a non-zero status from *either* side surfaces; `PIPESTATUS` shows which side failed). Rerun the count query on the target afterwards and compare with the baseline. Command details — each flag below was added because omitting it produced a failing, noisy, or broken-permissions restore during validation: - -- **Restore as the application user** (`-U appuser` with its password), not as `postgres`: with `--no-owner`, restored objects are owned by the connecting user. Restoring as `postgres` leaves every table owned by `postgres`, and the application user cannot even `SELECT` afterwards (`permission denied for table ...`). Restoring as the CR-defined owner lands the ownership correctly with no post-step. **Scope note:** this pattern fits the operator's standard layout — one application database wholly owned by one CR-defined user. If your database has multiple owning roles, `SECURITY DEFINER` functions, or objects in non-`public` schemas with distinct owners, `--no-owner` collapses all ownership onto the connecting user; plan an explicit post-restore ownership and grant pass for those objects. -- **Version-matched binaries** (`/usr/lib/postgresql//bin/`): the container's default `pg_dump` can be a newer major whose output (e.g. `SET transaction_timeout`) an older target server rejects. The images ship binaries for every supported major, so pick the path explicitly (the script auto-detects this from `SHOW server_version`). When migrating to a *newer* target major, use the **target** major's `pg_dump` (also present in the source pod's image). -- **`--no-comments`**: the dump captures `COMMENT ON EXTENSION` for the extensions the image pre-installs (`pg_stat_statements`, `pg_stat_kcache`, `set_user`); executing those requires superuser, which the application user is not. Object comments are dropped — restore them separately as `postgres` if you rely on them. -- **`-N metric_helpers -N user_management`**: the operator image pre-creates these schemas in every database; without the exclusion the restore reports ~20 `already exists` errors. -- **`-x`**: skips ACL statements that reference roles managed by the source operator and absent on the target. Re-grant privileges for any additional (non-owner) users on the target afterwards. - -### Large databases - -A single pipe has no resume capability. For large databases, dump to a compressed file on the workstation first, then restore from it: - -```bash -kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ - $PG_BIN/pg_dump -U postgres -Fc --no-comments \ - -N metric_helpers -N user_management appdb > appdb.dump - -kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ - env PGPASSWORD="$APP_PW" \ - $PG_BIN/pg_restore -U appuser -h localhost -d appdb --no-owner -x < appdb.dump -``` - -Parallel restore (`pg_restore -j N`) cannot read from stdin; to use it, copy the dump file into the target pod (`kubectl cp`) and restore from the local path. - ## Troubleshooting ### Application user gets `permission denied for table ...` after a successful restore -The restore was run as `postgres` instead of the application user, so all objects are owned by `postgres`. Either rerun the restore as the application user (see the [reference section](#reference-manual-single-database-transfer)), or transfer ownership in place: +The restore was run as `postgres` instead of the application user, so all objects are owned by `postgres` (the script always restores as the owner; this typically comes from a hand-run restore). Either redo the database — drop and recreate it, then rerun the script with the database name — or transfer ownership in place: ```sql DO $$ @@ -347,6 +305,8 @@ END $$; This block covers tables and sequences in the `public` schema only. Views, functions, types, and objects in other schemas need analogous `ALTER ... OWNER TO` statements (enumerate them via `pg_views`, `pg_proc`, `pg_type`, or `\dn`/`\df` in psql). +Related scope note: the migration's `--no-owner` model assumes the operator's standard layout — one database wholly owned by one CR-defined user. If a database has multiple owning roles, `SECURITY DEFINER` functions, or objects in non-`public` schemas with distinct owners, ownership collapses onto the restoring user; plan an explicit post-restore ownership and grant pass for those objects. + ### Restore reports `permission denied to create extension ...` The source database uses an extension that is not pre-installed on the target, and the application user cannot create it. The script warns about this (`WARNING: could not create extension`). Create the extension as `postgres` on the target database, then rerun the restore. If `CREATE EXTENSION` itself fails with a missing-file error, the extension's packages are not present in the target image at all — it must be added to the image/instance before this migration can carry that database. @@ -357,7 +317,7 @@ The source database uses an extension that is not pre-installed on the target, a ### Restore reports `unrecognized configuration parameter "transaction_timeout"` -The dump was taken with a `pg_dump` newer than the target server (typically the image's default binary). Rerun using the explicit version-matched binary path (the script auto-detects it; in the manual commands set `PG_BIN` to the source server's major). +The dump was taken with a `pg_dump` newer than the target server (typically the image's default binary — a hand-run dump without the explicit `PG_BIN` path; the script auto-detects it from `SHOW server_version`). Rerun with the version-matched binary path. When migrating to a *newer* target major, use the **target** major's binaries instead — they are present in the source pod's image too. ### Restore reports `schema "metric_helpers" already exists` (and similar) @@ -372,7 +332,7 @@ kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- psql -U postg kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- psql -U postgres -c "CREATE DATABASE appdb OWNER appuser" ``` -Then redo **only the failed database** with the [manual transfer](#reference-manual-single-database-transfer). Do not rerun the whole script after a partial success: it would restore into the databases that already transferred, and the resulting `already exists` collisions would mark them as failed. +Then rerun the script with **only the failed database** as an argument (`bash migrate.sh `). Do not rerun it without arguments after a partial success: restoring into the databases that already transferred produces `already exists` collisions that mark them as failed. ### Application passwords changed after the migration From dc60b425752843d6c6213663bf63b836d345b7c5 Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Wed, 22 Jul 2026 03:13:38 +0000 Subject: [PATCH 15/17] docs: fix codex round-2 findings on KB260721001 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1 (blocker): the script can no longer report success after migrating zero databases — pod lookups and the source-major probe are guarded with early exits, and an ATTEMPTED counter turns empty enumeration or a non-matching subset argument into an explicit error. Validated: bogus cluster name exits 1 at preflight, full run reports the count, 'bash migrate.sh nosuchdb' exits 1. 2 (major): the credential pre-stage loop derives the user from the secret's NAME (the CR user the target operator looks up) instead of the decoded username field, copies credential bytes base64-verbatim, skips pooler alongside postgres/standby, and documents the password-rotation limitation. 3 (major): the password recovery is injection-safe — base64 merge patch for the secret and psql :'pw' literal quoting for ALTER ROLE. Validated with a password containing quotes, backslash, and SQL. 4 (minor): the verification query schema-qualifies table names and excludes the schemas the dump excludes. Co-Authored-By: Claude Fable 5 --- ...tance_Between_Network_Isolated_Clusters.md | 59 +++++++++++++------ 1 file changed, 40 insertions(+), 19 deletions(-) diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md index 76a99f71d..450c78c6b 100644 --- a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md @@ -87,18 +87,26 @@ The `users:`/`databases:` sections of the target CR can be generated directly fr By default the target operator generates **new** passwords for the CR users, which would force every application to be re-configured at cutover. To keep the source passwords, copy each application user's credential secret into the target namespace **before creating the CR** — when the operator finds an existing credential secret, it adopts it and sets the role's password from it instead of generating a new one: ```bash +SUFFIX="credentials.postgresql.acid.zalan.do" for SEC in $(kubectl --context $SRC_CTX -n $SRC_NS get secret \ -l application=spilo,cluster-name=$SRC_CLUSTER -o name); do - U=$(kubectl --context $SRC_CTX -n $SRC_NS get $SEC -o jsonpath='{.data.username}' | base64 -d) - case "$U" in postgres|standby) continue ;; esac # system users stay operator-managed - PW=$(kubectl --context $SRC_CTX -n $SRC_NS get $SEC -o jsonpath='{.data.password}' | base64 -d) - kubectl --context $TGT_CTX -n $TGT_NS create secret generic \ - "$U.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do" \ - --from-literal=username="$U" --from-literal=password="$PW" + NAME=${SEC#secret/} + U=${NAME%.$SRC_CLUSTER.$SUFFIX} + [ "$U" = "$NAME" ] && continue # not a credential secret + case "$U" in postgres|standby|pooler) continue ;; esac # operator-managed system users + kubectl --context $TGT_CTX -n $TGT_NS apply -f - <"; TGT_NS=""; TGT_CLUSTER="" SRC_POD=$(kubectl --context "$SRC_CTX" -n "$SRC_NS" get pod \ - -l spilo-role=master,cluster-name="$SRC_CLUSTER" -o jsonpath='{.items[0].metadata.name}') + -l spilo-role=master,cluster-name="$SRC_CLUSTER" -o jsonpath='{.items[0].metadata.name}') \ + && [ -n "$SRC_POD" ] || { echo "ERROR: cannot find source master pod for $SRC_CLUSTER" >&2; exit 1; } TGT_POD=$(kubectl --context "$TGT_CTX" -n "$TGT_NS" get pod \ - -l spilo-role=master,cluster-name="$TGT_CLUSTER" -o jsonpath='{.items[0].metadata.name}') + -l spilo-role=master,cluster-name="$TGT_CLUSTER" -o jsonpath='{.items[0].metadata.name}') \ + && [ -n "$TGT_POD" ] || { echo "ERROR: cannot find target master pod for $TGT_CLUSTER" >&2; exit 1; } srcsql() { local db=$1; shift; kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- psql -U postgres -d "$db" -tA -v ON_ERROR_STOP=1 "$@"; } tgtsql() { local db=$1; shift; kubectl --context "$TGT_CTX" -n "$TGT_NS" exec "$TGT_POD" -c postgres -- psql -U postgres -d "$db" -tA -v ON_ERROR_STOP=1 "$@"; } # Client binaries matching the SOURCE server major (present in both Spilo images). SRC_MAJOR=$(srcsql postgres -c "SHOW server_version;" | cut -d. -f1) +case "$SRC_MAJOR" in ''|*[!0-9]*) echo "ERROR: cannot determine source PostgreSQL major (psql via $SRC_POD failed)" >&2; exit 1 ;; esac PG_BIN=/usr/lib/postgresql/$SRC_MAJOR/bin echo "Source PostgreSQL major: $SRC_MAJOR (client binaries: $PG_BIN)" -COUNT_QUERY="SELECT relname, (xpath('/row/c/text()', query_to_xml(format( +# Count every user table outside the schemas the dump excludes. +COUNT_QUERY="SELECT schemaname||'.'||relname, (xpath('/row/c/text()', query_to_xml(format( 'SELECT count(*) AS c FROM %I.%I', schemaname, relname), false, true, '')))[1]::text::bigint - FROM pg_stat_user_tables ORDER BY relname;" + FROM pg_stat_user_tables + WHERE schemaname NOT IN ('metric_helpers','user_management') ORDER BY 1;" -FAILED="" +FAILED=""; ATTEMPTED=0 while read -r DB OWNER; do [ -z "$DB" ] && continue # With arguments, migrate only the named databases (e.g. to redo one FAIL). if [ "$#" -gt 0 ]; then case " $* " in *" $DB "*) ;; *) continue ;; esac fi + ATTEMPTED=$((ATTEMPTED+1)) echo "=== migrating database: $DB (owner: $OWNER) ===" # Ensure the database exists on the target (covers databases created @@ -224,8 +238,12 @@ done < <(srcsql postgres -F' ' -c \ WHERE NOT datistemplate AND datname <> 'postgres' ORDER BY 1;") echo +if [ "$ATTEMPTED" -eq 0 ]; then + echo "ERROR: no databases migrated — source enumeration failed, or no database matched the arguments" >&2 + exit 1 +fi if [ -n "$FAILED" ]; then echo "MIGRATION INCOMPLETE — failed:$FAILED"; exit 1; fi -echo "MIGRATION COMPLETE — all databases verified." +echo "MIGRATION COMPLETE — $ATTEMPTED database(s) verified." ``` If a database reports `FAIL`, see [Troubleshooting](#troubleshooting); after fixing the cause, redo just that database by passing its name to the script. @@ -336,17 +354,20 @@ Then rerun the script with **only the failed database** as an argument (`bash mi ### Application passwords changed after the migration -The target CR was created **before** the credential secrets were pre-staged (Step 1), so the operator generated new passwords. To restore the source credentials after the fact, update both the secret and the role together — the secret alone is not enough, and an `ALTER ROLE` alone would be reverted by the operator's next sync from the secret: +The target CR was created **before** the credential secrets were pre-staged (Step 1), so the operator generated new passwords. To restore the source credentials after the fact, update both the secret and the role together — the secret alone is not enough, and an `ALTER ROLE` alone would be reverted by the operator's next sync from the secret. The commands below stay safe for passwords containing quotes, backslashes, or JSON metacharacters: the secret is patched with the base64 value (JSON-safe by construction), and the SQL uses psql's `:'pw'` literal quoting instead of string interpolation: ```bash -SRC_PW=$(kubectl --context $SRC_CTX -n $SRC_NS get secret \ - appuser.$SRC_CLUSTER.credentials.postgresql.acid.zalan.do -o jsonpath='{.data.password}' | base64 -d) +PW_B64=$(kubectl --context $SRC_CTX -n $SRC_NS get secret \ + appuser.$SRC_CLUSTER.credentials.postgresql.acid.zalan.do -o jsonpath='{.data.password}') kubectl --context $TGT_CTX -n $TGT_NS patch secret \ appuser.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do \ - -p "{\"stringData\":{\"password\":\"$SRC_PW\"}}" -kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ - psql -U postgres -c "ALTER ROLE appuser PASSWORD '$SRC_PW';" + --type merge -p "{\"data\":{\"password\":\"$PW_B64\"}}" + +kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ + psql -U postgres -v pw="$(printf %s "$PW_B64" | base64 -d)" <<'SQL' +ALTER ROLE appuser PASSWORD :'pw'; +SQL ``` Verify with a login test as in Step 4. From abdd3e6a2cd274a35960dfc540607621c12a60b3 Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Wed, 22 Jul 2026 03:28:11 +0000 Subject: [PATCH 16/17] docs: unify large-database handling into the script (DUMP_DIR file mode) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove the last manual side-path: instead of a hand-run dump/restore for large databases, the script now has two modes through the same code — default pipe, or DUMP_DIR=/path for per-database dump files with atomic completion (.partial + mv) and reuse on rerun, making transfers resumable. The failed-restore redo simplifies to DROP DATABASE + rerun with the database name (the script recreates the database with the correct owner). Validated live: file-mode full run, resume run reusing a completed dump after DROP, and pipe mode unchanged. Co-Authored-By: Claude Fable 5 --- ...tance_Between_Network_Isolated_Clusters.md | 68 ++++++++++--------- 1 file changed, 36 insertions(+), 32 deletions(-) diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md index 450c78c6b..bd4a874bb 100644 --- a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md @@ -153,7 +153,10 @@ kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ The script performs the whole migration in one run: it enumerates the source's application databases, creates any that are missing on the target (e.g. `postgres`-owned databases created outside the CR spec — their owner role must exist on the target), pre-creates each database's extensions, transfers each database as its owner, and verifies exact per-table row counts — reporting `PASS`/`FAIL` per database and exiting non-zero if anything failed. -Fill in the six variables at the top, then run it with `bash`. Without arguments it migrates every database; pass database names (`bash migrate.sh appdb`) to migrate only those — used to redo a single database after a failure: +Fill in the six variables at the top, then run it with `bash`. Without arguments it migrates every database; pass database names (`bash migrate.sh appdb`) to migrate only those — used to redo a single database after a failure. Every transfer goes through the same script in one of two modes: + +- **Pipe mode** (default): source streams straight into the target — no intermediate storage. +- **File mode** (`DUMP_DIR=/path bash migrate.sh`): each database is dumped to `$DUMP_DIR/.dump` first, then restored from the file. A rerun **reuses completed dumps**, making the transfer resumable — use this for large databases or unreliable links. ```bash #!/usr/bin/env bash @@ -217,13 +220,33 @@ while read -r DB OWNER; do "$OWNER.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do" \ -o jsonpath='{.data.password}' | base64 -d) || { FAILED="$FAILED $DB(secret)"; continue; } - kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- \ - "$PG_BIN/pg_dump" -U postgres -Fc --no-comments \ - -N metric_helpers -N user_management "$DB" \ - | kubectl --context "$TGT_CTX" -n "$TGT_NS" exec -i "$TGT_POD" -c postgres -- \ - env PGPASSWORD="$OWNER_PW" \ - "$PG_BIN/pg_restore" -U "$OWNER" -h localhost -d "$DB" --no-owner -x - RC=$? + # Transfer: straight pipe by default; with DUMP_DIR set, dump to a file + # first and restore from it (resumable — completed dumps are reused). + if [ -n "${DUMP_DIR:-}" ]; then + mkdir -p "$DUMP_DIR" + DUMP_FILE="$DUMP_DIR/$DB.dump" + if [ -s "$DUMP_FILE" ]; then + echo " reusing existing dump $DUMP_FILE" + else + kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- \ + "$PG_BIN/pg_dump" -U postgres -Fc --no-comments \ + -N metric_helpers -N user_management "$DB" > "$DUMP_FILE.partial" \ + && mv "$DUMP_FILE.partial" "$DUMP_FILE" \ + || { rm -f "$DUMP_FILE.partial"; FAILED="$FAILED $DB(dump)"; continue; } + fi + kubectl --context "$TGT_CTX" -n "$TGT_NS" exec -i "$TGT_POD" -c postgres -- \ + env PGPASSWORD="$OWNER_PW" \ + "$PG_BIN/pg_restore" -U "$OWNER" -h localhost -d "$DB" --no-owner -x < "$DUMP_FILE" + RC=$? + else + kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- \ + "$PG_BIN/pg_dump" -U postgres -Fc --no-comments \ + -N metric_helpers -N user_management "$DB" \ + | kubectl --context "$TGT_CTX" -n "$TGT_NS" exec -i "$TGT_POD" -c postgres -- \ + env PGPASSWORD="$OWNER_PW" \ + "$PG_BIN/pg_restore" -U "$OWNER" -h localhost -d "$DB" --no-owner -x + RC=$? + fi # Verify: exact counts must match the baseline. TGT_COUNTS=$(tgtsql "$DB" -c "$COUNT_QUERY") @@ -246,30 +269,12 @@ if [ -n "$FAILED" ]; then echo "MIGRATION INCOMPLETE — failed:$FAILED"; exit 1 echo "MIGRATION COMPLETE — $ATTEMPTED database(s) verified." ``` -If a database reports `FAIL`, see [Troubleshooting](#troubleshooting); after fixing the cause, redo just that database by passing its name to the script. - -### Large databases - -A straight pipe is not resumable. For a large database, dump to a file on the workstation first, then restore from it — same flags as the script; `` is the owner's password from the target secret: - -```bash -PG_BIN=/usr/lib/postgresql//bin - -kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ - $PG_BIN/pg_dump -U postgres -Fc --no-comments \ - -N metric_helpers -N user_management > db.dump - -kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ - env PGPASSWORD="" \ - $PG_BIN/pg_restore -U -h localhost -d --no-owner -x < db.dump -``` - -Parallel restore (`pg_restore -j N`) cannot read from stdin; to use it, copy the dump file into the target pod (`kubectl cp`) and restore from the local path. +If a database reports `FAIL`, see [Troubleshooting](#troubleshooting); after fixing the cause, redo just that database by passing its name to the script (in file mode, its completed dump is reused). ### Security notes - While a restore runs, the owner password is expanded into the workstation's `kubectl` argument list and is visible in local process listings (`ps`). Run the migration from a trusted, single-user workstation, and do not enable shell tracing (`set -x`) around these commands. -- A dump file (file-based variant) is a full plaintext logical copy of the database. Create it with restrictive permissions (`umask 077` before dumping, or `chmod 600` immediately after), encrypt it at rest if your policy requires, and delete it once the migration is verified. +- In file mode, `$DUMP_DIR` holds full plaintext logical copies of the databases. Point it at a directory with restrictive permissions (`umask 077` before running, or `chmod 700` on the directory), encrypt at rest if your policy requires, and delete the dumps once the migration is verified. - The data itself only traverses the two TLS-protected `kubectl exec` channels; nothing is exposed on the network beyond the two Kubernetes API connections. ## Step 4: Verify Application Access @@ -343,14 +348,13 @@ The `-N metric_helpers -N user_management` exclusions were omitted. These errors ### Rerunning after a failed restore -Drop and recreate the target database first — a partial restore leaves objects that collide (`relation already exists`, duplicate-key COPY failures). Note `DROP DATABASE` must be executed as its own single statement: +Drop the target database first — a partial restore leaves objects that collide (`relation already exists`, duplicate-key COPY failures). Note `DROP DATABASE` must be executed as its own single statement: ```bash kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- psql -U postgres -c "DROP DATABASE appdb" -kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- psql -U postgres -c "CREATE DATABASE appdb OWNER appuser" ``` -Then rerun the script with **only the failed database** as an argument (`bash migrate.sh `). Do not rerun it without arguments after a partial success: restoring into the databases that already transferred produces `already exists` collisions that mark them as failed. +Then rerun the script with **only the failed database** as an argument (`bash migrate.sh `) — it recreates the database with the correct owner, and in file mode it reuses the completed dump. Do not rerun it without arguments after a partial success: restoring into the databases that already transferred produces `already exists` collisions that mark them as failed. ### Application passwords changed after the migration @@ -374,4 +378,4 @@ Verify with a login test as in Step 4. ### Pipe is slow -Throughput is bounded by the workstation's link to both API servers (every byte traverses it twice: exec stream in, exec stream out). Run the relay from a machine with good connectivity to both platforms (e.g. a jump host), or use the file-based variant to at least make progress resumable. +Throughput is bounded by the workstation's link to both API servers (every byte traverses it twice: exec stream in, exec stream out). Run the migration from a machine with good connectivity to both platforms (e.g. a jump host), and use file mode (`DUMP_DIR`) so progress is resumable. If a very large restore needs parallelism, note `pg_restore -j N` cannot read from stdin — copy the dump file into the target pod (`kubectl cp`) and restore from the local path there. From a12836f88222792b94c9dc48b00a099bf33f431d Mon Sep 17 00:00:00 2001 From: Jinpei Su Date: Wed, 22 Jul 2026 03:37:23 +0000 Subject: [PATCH 17/17] docs: gate dump reuse behind REUSE_DUMPS (codex round-3) Finding 1 (major): a leftover completed dump could be silently reused across maintenance windows, migrating stale data past the count check. File mode now takes a fresh dump on every run by default; REUSE_DUMPS=1 opts into reuse, documented as safe only while source writes have stayed stopped since the dump was taken. Validated live: after a post-window source write, the default rerun re-dumped and migrated the new row; REUSE_DUMPS=1 reuses the completed dump. Finding 2 (nit): FAIL message says 'transfer exit' instead of 'pipe exit', correct for both modes. Co-Authored-By: Claude Fable 5 --- ...QL_Instance_Between_Network_Isolated_Clusters.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md index bd4a874bb..725f20d0f 100644 --- a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md @@ -156,7 +156,7 @@ The script performs the whole migration in one run: it enumerates the source's a Fill in the six variables at the top, then run it with `bash`. Without arguments it migrates every database; pass database names (`bash migrate.sh appdb`) to migrate only those — used to redo a single database after a failure. Every transfer goes through the same script in one of two modes: - **Pipe mode** (default): source streams straight into the target — no intermediate storage. -- **File mode** (`DUMP_DIR=/path bash migrate.sh`): each database is dumped to `$DUMP_DIR/.dump` first, then restored from the file. A rerun **reuses completed dumps**, making the transfer resumable — use this for large databases or unreliable links. +- **File mode** (`DUMP_DIR=/path bash migrate.sh`): each database is dumped to `$DUMP_DIR/.dump` first, then restored from the file — use this for large databases or unreliable links. By default every run takes a **fresh** dump; add `REUSE_DUMPS=1` to reuse completed dumps (resumable). Reuse is only safe **within the same stopped-write maintenance window** — a dump taken before source writes resumed would silently migrate stale data. ```bash #!/usr/bin/env bash @@ -221,11 +221,12 @@ while read -r DB OWNER; do -o jsonpath='{.data.password}' | base64 -d) || { FAILED="$FAILED $DB(secret)"; continue; } # Transfer: straight pipe by default; with DUMP_DIR set, dump to a file - # first and restore from it (resumable — completed dumps are reused). + # first and restore from it. REUSE_DUMPS=1 reuses completed dumps — only + # safe while source writes have stayed stopped since the dump was taken. if [ -n "${DUMP_DIR:-}" ]; then mkdir -p "$DUMP_DIR" DUMP_FILE="$DUMP_DIR/$DB.dump" - if [ -s "$DUMP_FILE" ]; then + if [ -n "${REUSE_DUMPS:-}" ] && [ -s "$DUMP_FILE" ]; then echo " reusing existing dump $DUMP_FILE" else kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- \ @@ -253,7 +254,7 @@ while read -r DB OWNER; do if [ "$RC" -eq 0 ] && [ "$SRC_COUNTS" = "$TGT_COUNTS" ]; then echo " PASS: $DB (tables/rows identical)" else - echo " FAIL: $DB (pipe exit $RC; counts $( [ "$SRC_COUNTS" = "$TGT_COUNTS" ] && echo match || echo DIFFER ))" + echo " FAIL: $DB (transfer exit $RC; counts $( [ "$SRC_COUNTS" = "$TGT_COUNTS" ] && echo match || echo DIFFER ))" FAILED="$FAILED $DB" fi done < <(srcsql postgres -F' ' -c \ @@ -269,7 +270,7 @@ if [ -n "$FAILED" ]; then echo "MIGRATION INCOMPLETE — failed:$FAILED"; exit 1 echo "MIGRATION COMPLETE — $ATTEMPTED database(s) verified." ``` -If a database reports `FAIL`, see [Troubleshooting](#troubleshooting); after fixing the cause, redo just that database by passing its name to the script (in file mode, its completed dump is reused). +If a database reports `FAIL`, see [Troubleshooting](#troubleshooting); after fixing the cause, redo just that database by passing its name to the script (in file mode, add `REUSE_DUMPS=1` to skip re-dumping if writes have stayed stopped). ### Security notes @@ -354,7 +355,7 @@ Drop the target database first — a partial restore leaves objects that collide kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- psql -U postgres -c "DROP DATABASE appdb" ``` -Then rerun the script with **only the failed database** as an argument (`bash migrate.sh `) — it recreates the database with the correct owner, and in file mode it reuses the completed dump. Do not rerun it without arguments after a partial success: restoring into the databases that already transferred produces `already exists` collisions that mark them as failed. +Then rerun the script with **only the failed database** as an argument (`bash migrate.sh `) — it recreates the database with the correct owner; in file mode, add `REUSE_DUMPS=1` to restore from the completed dump instead of re-dumping (only if source writes have stayed stopped). Do not rerun it without arguments after a partial success: restoring into the databases that already transferred produces `already exists` collisions that mark them as failed. ### Application passwords changed after the migration