docs(server): complete helm upgrade, migration hooks, and recovery runbooks

[blue4209211]

Summary

Overhauls doc-server/docs/installation/server/upgrade.md with code-grounded operational runbooks derived from the umbrella chart architecture in nudgebee-oss:

1. Pre-Flight Planning & Stored Values Merging:

   • Chart version pinning invariant (--version <TARGET_VERSION>).
   • Deep explanation of Helm stored-values merging (helm get values --all vs helm get values) and avoiding stale parameters from --reuse-values.
   • Manifest diffing pre-checks via helm diff upgrade and fallback native helm get manifest + helm template.
   • Security invariant: preserving nudgebee_secret.NUDGEBEE_ENCRYPTION_KEY across upgrades.
   • Pre-upgrade database snapshot commands (bundled PostgreSQL & AWS RDS).

2. Database Migration Lifecycle & Failure Diagnostics:

   • Sequence diagram illustrating Helm upgrade hooks and migration pod flow.
   • Complete breakdown of postgres-migration-job hook (helm.sh/hook: post-install,post-upgrade, weight: -5, backoffLimit: 0).
   • Init container db-check polling pg_isready and main container run-migrations.sh.
   • Troubleshooting init container deadlocks, DDL table locks, and hook cleanup semantics.
   • Temporal schema migrations (temporal.schema.useHelmHooks: false) and shard count invariant (numHistoryShards: 512).

3. StatefulSet & PVC Storage Diagnostics:

   • StorageClass volume expansion prerequisites (allowVolumeExpansion: true).
   • ReadWriteOnce (RWO) multi-attach error resolution across node transitions.
   • StatefulSet immutable field violations and safe recreation via kubectl delete statefulset --cascade=orphan.

4. Bundled vs. External Infrastructure Topology:

   • Architecture diagram and complete values overrides for externalizing PostgreSQL (RDS/Aurora), RabbitMQ (CloudAMQP), Redis (ElastiCache), and ClickHouse.

5. Rollback & Recovery Runbook:

   • Helm revision rollback commands.
   • Database rollback realities (forward-only SQL migrations) and point-in-time snapshot restore procedures.

Verification

• npm run build executed cleanly in doc-server with zero broken links or broken anchors.

[blue4209211]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request comprehensively rewrites the NudgeBee Server upgrade documentation, adding detailed operational procedures for pre-flight checks, manifest comparisons, database migration hooks, storage diagnostics, and rollback runbooks. The review feedback highlights several necessary corrections to the documentation's command-line examples. These include prefixing Kubernetes resource names (deployments, pods, StatefulSets) with the standard release name, using the correct database name (nudgebee instead of postgres), avoiding TTY allocation (-it) when piping kubectl exec output to prevent file corruption, adding AWS Aurora-specific snapshot commands, and defining environment variables before they are referenced.

[gemini-code-assist]

Code Review

This pull request comprehensively updates the NudgeBee Server upgrade documentation, transforming it from a basic checklist into an in-depth operational guide. It adds detailed sections on pre-flight checks, manifest comparison, database migration lifecycles, StatefulSet/PVC diagnostics, external infrastructure topologies, rollback runbooks, and post-upgrade verification. Feedback on the changes suggests removing the -t (TTY) flag from the kubectl exec database backup command to prevent potential archive corruption or execution failures in non-interactive environments.

[blue4209211]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request significantly expands and improves the NudgeBee Server upgrade documentation, adding detailed sections on pre-flight checks, manifest comparisons, database migration lifecycles, StatefulSet/PVC storage diagnostics, external infrastructure topologies, rollback runbooks, and post-upgrade verification. The review comments provide valuable improvements to the backup and restore commands, suggesting the use of pg_dumpall instead of pg_dump to ensure all databases (including Temporal databases) are fully backed up and restored.

[blue4209211]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request significantly expands the NudgeBee Server upgrade documentation, adding comprehensive guides on pre-flight checks, manifest comparison, database migration lifecycles, storage diagnostics, external infrastructure topologies, and rollback runbooks. The review feedback suggests several improvements to the documented commands, including explicitly targeting the PostgreSQL container in kubectl exec commands to avoid multi-container failures, adding missing password fields to the external Temporal configuration example, and dropping existing databases before restoring backups during a rollback to prevent relation conflicts.

[blue4209211]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request significantly expands the NudgeBee Server upgrade documentation, adding comprehensive guides on pre-flight checks, manifest comparisons, database migration lifecycles, storage diagnostics, and rollback runbooks. The reviewer suggested a critical improvement to the database rollback instructions, recommending that active connections be terminated using pg_terminate_backend before attempting to drop the databases to prevent execution failures.

[blue4209211]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request significantly expands and restructures the NudgeBee Server upgrade documentation, adding comprehensive, code-grounded operational procedures such as pre-flight checks, database backups, manifest comparisons, migration lifecycle details, StatefulSet/PVC troubleshooting, external infrastructure topology, rollback runbooks, and post-upgrade verification. The review feedback identifies two configuration errors in the documentation: an incorrect YAML nesting structure for Temporal's persistence configuration and an incorrect configuration key reference for Temporal's schema Helm hooks.

[blue4209211]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request significantly expands the NudgeBee Server upgrade documentation, adding comprehensive guides on pre-flight checks, database backups, manifest comparisons, migration lifecycles, storage diagnostics, external infrastructure topologies, and rollback procedures. The review feedback highlights a few critical improvements for the rollback and configuration sections: correcting the Temporal Helm chart hook parameters, ensuring Temporal services are scaled down and up during database restores to avoid connection conflicts, and addressing rollback procedures for external databases.

[blue4209211]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request significantly expands the NudgeBee Server upgrade documentation, adding detailed guides on pre-flight checks, database migration lifecycles, storage diagnostics, external infrastructure topology, and rollback runbooks. The review feedback points out two issues in the updated documentation: first, the Temporal SQL persistence configuration example is missing the required pluginName: "postgres" parameter; second, the postgres-migration-job is incorrectly documented as a post-upgrade hook when it needs to be a pre-upgrade hook to run before the application pods roll over.

[blue4209211]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request significantly expands the NudgeBee Server upgrade documentation, adding detailed operational procedures for pre-flight checks, manifest comparison, database migrations, storage diagnostics, external infrastructure topology, and rollback runbooks. The review feedback suggests several corrections to the documentation, including fixing an inconsistency in the database migration sequence diagram, replacing a wildcard in the database restore command to prevent potential stream corruption, adding a necessary step to create a DB instance when restoring an Aurora cluster, and correcting a redundant argument in the verification kubectl run command.

[blue4209211]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request significantly expands the upgrade documentation for NudgeBee Server, transforming it into a comprehensive, production-grade guide. It now includes detailed instructions on pre-flight checks, database backups, manifest comparisons, migration diagnostics, storage troubleshooting, external topology setups, and database rollback runbooks. The reviewer feedback addresses two issues in the documentation's code snippets: correcting the Temporal Helm chart hook parameters from useHelmHooks to registerWithHelmHooks, and splitting the DROP DATABASE command into separate -c flags in the PostgreSQL restore runbook to prevent transaction block errors.

[blue4209211]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request significantly expands the NudgeBee Server upgrade documentation, adding detailed operational procedures for pre-flight checks, manifest comparisons, database migrations, storage diagnostics, external topologies, and rollback runbooks. The review feedback suggests several improvements to the documentation, including adding TLS configuration parameters for Temporal datastores, ensuring collector deployments are properly scaled down and up during database restores to prevent connection conflicts, and specifying network configuration options when restoring RDS/Aurora snapshots.

[blue4209211]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request significantly expands the NudgeBee Server upgrade documentation, transforming it into a comprehensive, production-grade guide that covers pre-flight checks, database backups, manifest comparisons, StatefulSet/PVC diagnostics, external infrastructure topologies, and rollback runbooks. The review feedback highlights critical issues in the rollback and troubleshooting commands, including incorrect rollback ordering for bundled PostgreSQL (which risks database incompatibility issues), redundant manual scaling steps that override production replica counts, missing Helm release prefixes on migration job commands, and the need for a note explaining benign errors during database restores.