From 1f81af863020dbc48567815cf9c1bd927ff4f2d1 Mon Sep 17 00:00:00 2001 From: Himanshu Verma Date: Fri, 18 Sep 2026 23:41:07 +0530 Subject: [PATCH 1/9] doc: add Helm chart deployment page for Kubernetes Documents the distributed Helm chart from apache/hugegraph#3218: install with the three values presets, the auth Secrets model, PD health vs readiness probing, upgrade and scale-down caveats, uninstall behavior, and current limitations. EN and CN pages under quickstart/hugegraph, weight 4, matching the sibling page skeleton. --- .../quickstart/hugegraph/hugegraph-helm.md | 236 ++++++++++++++++ .../quickstart/hugegraph/hugegraph-helm.md | 263 ++++++++++++++++++ 2 files changed, 499 insertions(+) create mode 100644 content/cn/docs/quickstart/hugegraph/hugegraph-helm.md create mode 100644 content/en/docs/quickstart/hugegraph/hugegraph-helm.md diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md new file mode 100644 index 0000000000..72911f369c --- /dev/null +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md @@ -0,0 +1,236 @@ +--- +title: "使用 Helm 在 Kubernetes 上部署" +linkTitle: "Kubernetes 部署 (Helm)" +weight: 4 +search_keywords: + - helm + - kubernetes + - k8s +--- + +### 1 概述 + +Helm chart 在 Kubernetes 上部署一套分布式 HugeGraph 集群：PD、Store、Server，以及可选的 Hubble UI。chart 位于主仓库的 +[`helm/hugegraph`](https://github.com/apache/hugegraph/tree/master/helm/hugegraph) 目录。 + +| 组件 | 工作负载 | 默认副本数 | 作用 | +|------|----------|------------|------| +| PD | StatefulSet + PVC | 3 | 元数据管理：以 Raft 组跟踪 Store 与分区 | +| Store | StatefulSet + PVC | 3 | 图数据存储 (HStore) | +| Server | Deployment | 3 | Gremlin 与 REST 查询层 | +| Hubble | Deployment | 0（默认关闭） | Web UI，通过 `hubble.enabled=true` 启用 | + +分布式 HugeGraph 集群有一套启动约定（Server 不执行 `init-store`、每个 Server 都通过 PD 读写图元数据、Store 等待 PD +形成多数派、一个 PD REST 密钥由三个组件共用）。chart 把这套约定固化下来，运维人员无需手工处理；细节见 +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#chart-details)。 + +```mermaid +flowchart LR + subgraph Kubernetes 集群 + PD[PD StatefulSet
3 节点 Raft 组，PVC] + Store[Store StatefulSet
3 副本，PVC] -->|注册、心跳| PD + Server[Server Deployment
3 副本] -->|元数据、服务发现| PD + Server -->|gRPC 读写| Store + Hubble[Hubble Deployment
可选 UI] -->|通过 PD 发现 Server| PD + end + Client[客户端 / hugegraph-client] -->|REST / Gremlin| Server +``` + +启动顺序由 chart 保证，而不是由运维人员保证：PD 先选出 leader，每个 Store Pod 的 init 容器等到多数 PD 汇报就绪后才启动， +Server 则反复等待存储层，直到 Store 完成注册。全新安装无需手工干预即可收敛。 + +### 2 前置条件 + +- Kubernetes 1.23 及以上（chart 渲染 `autoscaling/v2` 和 `policy/v1`） +- Helm 3；升级一节提到的 `--reset-then-reuse-values` 需要 Helm 3.14 及以上 +- 动态卷供给：有默认 StorageClass，或为 PD 和 Store 显式指定 `storageClassName` +- 默认拓扑要运行九个 JVM 进程，内存需留足；见安装一节的资源说明 + +chart 要求组件镜像包含 PD 就绪探测端点和 PD REST 认证（两者都已合入 1.7.0 之后的版本）。默认镜像 tag 指向的构建已包含 +这些改动；1.7.0 镜像不受支持。 + +### 3 安装 + +#### 3.1 获取 chart + +chart 尚未发布到 chart 仓库，从源码树安装： + +```bash +git clone https://github.com/apache/hugegraph.git +cd hugegraph +``` + +#### 3.2 使用默认值安装 + +先确认 `kubectl` 指向目标集群，且集群能供给存储卷。因缺少 StorageClass 而卡在 `Pending` 的 PVC 是最常见的首次安装 +故障： + +```bash +kubectl config current-context +kubectl get storageclass +``` + +然后安装： + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace --wait --timeout 15m +``` + +`--wait` 让 Helm 阻塞到所有工作负载就绪。对分布式集群来说，这个信号意味着 PD 已选出 leader、Store 已注册、Server 已 +启动；不加它，`helm install` 在对象创建完成后就返回。全新集群一般几分钟内收敛，15 分钟超时是给缓慢的镜像拉取留的余量。 + +继续之前需要了解两个默认值： + +- **默认不设置 resources。** 每个 Pod 都是 BestEffort，JVM 按节点总内存计算堆大小。单节点没有问题；多节点集群上各个堆 + 会超订节点内存导致进程中止。超出笔记本范围的部署请使用 `values-cluster.yaml`，或按组件设置 `resources`。 +- **镜像 tag 跟踪 `latest`**，且 `pullPolicy: Always`，直到下一个 HugeGraph 版本发布。生产环境请固定 tag 或 digest。 + +#### 3.3 拓扑预设 + +chart 附带三个 values 文件： + +| 文件 | 拓扑 | 适用场景 | +|------|------|----------| +| `values.yaml` | 3 PD + 3 Store + 3 Server | 默认；preferred 反亲和，认证开启，Hubble 关闭 | +| `values-single.yaml` | 1 + 1 + 1 | 单节点开发与 CI；PVC 更小 | +| `values-cluster.yaml` | 3 + 3 + 3 | 生产起点：JVM 堆与资源设置、PD/Store PodDisruptionBudget、`required` 反亲和 | + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace \ + -f helm/hugegraph/values-single.yaml --wait --timeout 15m +``` + +`values-cluster.yaml` 是起点而非容量保证：请按图规模和流量重新核算资源。完整参数参考（每个组件的探针、调度、Secret +配置项）见 [chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration)。 + +#### 3.4 验证安装 + +```bash +helm test hugegraph --namespace hugegraph +``` + +读取自动生成的 admin 密码并调用 API： + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin -o jsonpath='{.data.password}' | base64 --decode)" +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +``` + +以上命令假定 release 名为 `hugegraph`；用其他名字时，请替换成带 release 前缀的资源名（`kubectl get svc,secret -n +` 可以列出）。`helm install` 结束时打印的说明里包含填好名字的同样命令。 + +### 4 认证与 Secret + +认证默认开启，chart 管理三个 Secret。每个凭据按同一顺序取值：你预先创建的 `existingSecret` 优先，其次是内联值，最后 +是安装时随机生成。 + +| Secret | 键 | 用途 | 自带凭据的配置项 | +|--------|-----|------|------------------| +| `-admin` | `password` | Server admin 账号、Hubble 登录 | `server.auth.admin.existingSecret` | +| `-auth-token` | `token_secret` | 所有 Server 副本共用的 JWT 签名密钥 | `server.auth.token.existingSecret` | +| `-pd-auth` | `secret-key` | PD REST 认证，由 PD、Server、Hubble 读取 | `pd.auth.existingSecret` | + +要自行管理凭据，请在安装前创建 Secret 并把对应的 `existingSecret` 指向它；chart 不会改动任何不是它创建的 Secret。取值 +约束：admin 密码不能包含换行、回车或反斜杠；JWT 密钥至少 32 字节；PD 密钥必须是可打印 ASCII。非法值会在渲染时或启动 +包装脚本中被拒绝，不会被悄悄截断。 + +chart 管理的 Secret 在卸载时保留，同名 release 再次安装会复用它们。 + +
+轮换与注意事项 + +- admin 密码只在认证元数据首次创建时生效，之后修改 Secret 不会轮换已有集群的密码。请改用 Server 的 auth API 轮换。 +- 轮换 PD REST Secret 会在下一次 `helm upgrade` 时同时滚动 PD、Server 和 Hubble，保证三者持有的副本一致。纯模板流水线 + （`helm template`、GitOps 渲染器）看不到集群里的 Secret，因此在那里检测轮换的注解不起作用。 +- 三个 Secret 即使从不读取也都存在：安装结束打印的说明里有读取 admin 密码和 PD 密钥的 `kubectl get secret` 命令。 +
+ +### 5 健康检查与启动顺序 + +PD 暴露两个健康端点，chart 有意同时使用两者： + +- `/v1/health` 在 REST 监听建立后立刻返回 200。它不查询 Raft，因此感知不到多数派丢失。 +- `/v1/ready` 在 PD Raft 组选出 leader 之前返回 503，反映的是多数派状态而不只是进程存活。 + +chart 把 PD 的**就绪探测**和 Store init 容器的等待放在 `/v1/ready` 上：Store 只有在多数 PD 成为多数派成员后才启动， +失去 leader 的 PD 会退出 Service 端点，直到 leader 恢复。PD 的**启动和存活探测**则特意留在 `/v1/health` 上：只是失去 +leader 的 PD 仍是健康的 Raft 成员，重启它只会让故障恶化。 + +Server 的启动获得相同的预算：镜像默认会在 120 秒后杀掉仍在启动的 Server，chart 因此从启动探针推导 +`HG_SERVER_STARTUP_TIMEOUT_S`（默认 450 秒），配置的探针预算低于该下限时会被抬高。如果存储层启动更慢，调大 +`server.probes.startup`，镜像的预算会跟着变。 + +### 6 启用 Hubble UI + +Hubble 默认关闭，纯 API 集群因此更精简。在运行中的 release 上启用： + +```bash +helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values --set hubble.enabled=true +``` + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-hubble 8088:8088 +``` + +打开 `http://127.0.0.1:8088`，用 3.4 节的 admin 密码以 `admin` 身份登录。Hubble 通过 PD 发现 Server，集群运维视图无需 +额外配置即可工作。Hubble 只提供明文 HTTP：请通过 port-forward 或做 HTTPS 终结的 Ingress 访问，绝不要直接暴露在不可信 +网络上。在集群外运行 Hubble 也可行，但配置更多；见 chart README 的 +[Reaching Hubble](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#reaching-hubble-pick-one-path)。 + +### 7 升级 + +```bash +helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values +``` + +`--reuse-values` 保留 release 的既有覆盖值；不加它，升级会以 chart 默认值重建 release。任何改变 Pod 模板的升级都会让 +对应工作负载滚动一次。需要提前规划的几点： + +- **全新安装后的第一次升级会让 PD、Server、Hubble 各滚动一次**，因为跟踪 Secret 的注解第一次观察到安装时创建的 + Secret。Store 不受影响。 +- **Store 的滚动更新以监听检查推进，而不是以分片恢复推进**，因此控制器可能在上一个 Store 尚未重新加入分片组时就替换 + 下一个。生产环境滚动镜像时，设置 `store.updateStrategy.type=OnDelete`，逐个替换 Store Pod，并在替换下一个之前确认上 + 一个在 PD 中显示 `Up`；PD 本来就逐个重启，`pd.updateStrategy.type=OnDelete` 为维护窗口提供同样的手工控制。 +- **升级不能修改 PVC 大小**：Kubernetes 禁止修改 StatefulSet 的 `volumeClaimTemplates`，带新 `storage.size` 的升级会 + 被整体拒绝。chart README 记录了支持卷扩容的 StorageClass 上的扩容步骤。 + +Server 的扩缩容是普通的 values 变更（`server.replicas` 或 `server.hpa`）。**缩容 PD 或 Store 不是**：Raft 与分片成员 +关系是持久化的，删除 Pod 不会重新配置它们，PD 从 3 缩到 1 会永久失去多数派。chart 会拒绝副本数低于线上 StatefulSet +的升级；先迁移再缩容的手工步骤见 chart README 的 +[Scaling](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#scaling)。 + +### 8 卸载 + +```bash +helm uninstall hugegraph --namespace hugegraph +``` + +有两类状态是有意保留的。StatefulSet 创建的 PersistentVolumeClaim 会保留（Kubernetes 行为），确认数据不再需要后请显式 +删除。chart 管理的 Secret 也会保留，同名 release 再次安装时凭据不变。 + +### 9 限制 + +- 尚无 NetworkPolicy 资源。PD 的 Raft IP 白名单在集群内被关闭（Pod IP 会变化；白名单只在启动时解析一次对端，之后就会 + 拦截它们），集群内的网络访问控制目前由运维方负责。 +- 镜像 tag 跟踪 `latest`，直到下一个 HugeGraph 版本发布带版本号的镜像；长期运行的环境请固定 tag 或 digest。 +- 创建图之后，其他 Server 副本在短暂窗口内可能尚未收敛，路由到这类副本的查询可能报 `Could not rebind [g]` 之类的 + 错误。请带退避重试，或对"创建后立即查询"的流程使用会话粘滞路由；集群级图就绪在 + [#3137](https://github.com/apache/hugegraph/issues/3137) 跟踪。 +- 当前版本的 Store 恢复由运维人员触发：Store 丢失后的副本重建、leader 均衡、分区再均衡都只在调用 PD 的 REST API 时 + 执行。chart README 的 + [Disaster Recovery](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#disaster-recovery) 一节即操作 + 手册。 +- 集群内无 TLS 终结，无备份工具，无 Operator，无内置监控栈。 + +### 10 排障 + +| 现象 | 先查什么 | +|------|----------| +| Store Pod 卡在 `Init:0/1` | PD 未就绪：`kubectl logs -c wait-for-pd`，再看 PD Pod | +| PVC 停在 `Pending` | 没有默认 StorageClass，或供给器故障：`kubectl get sc` | +| 多节点上 Pod 被 OOM 杀掉或反复重启 | 未设置 resources，JVM 按节点内存取堆：用 `values-cluster.yaml` | +| 建图后立刻查询失败 | 副本收敛窗口：见上文"限制" | + +每种情况的完整排查步骤见 +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#troubleshooting)。 diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md new file mode 100644 index 0000000000..f0b2d28e38 --- /dev/null +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md @@ -0,0 +1,263 @@ +--- +title: "Deploy on Kubernetes with Helm" +linkTitle: "Deploy on Kubernetes (Helm)" +weight: 4 +search_keywords: + - helm + - kubernetes + - k8s +--- + +### 1 Overview + +The Helm chart deploys a distributed HugeGraph cluster on Kubernetes: PD, Store, and Server, plus the optional +Hubble UI. It lives in the main repository under +[`helm/hugegraph`](https://github.com/apache/hugegraph/tree/master/helm/hugegraph). + +| Component | Workload | Default replicas | Purpose | +|-----------|----------|------------------|---------| +| PD | StatefulSet + PVC | 3 | Placement driver: a Raft group tracking Stores and partitions | +| Store | StatefulSet + PVC | 3 | Graph data storage (HStore) | +| Server | Deployment | 3 | Gremlin and REST query layer | +| Hubble | Deployment | 0 (off) | Web UI, enabled with `hubble.enabled=true` | + +A distributed HugeGraph cluster has a startup contract (no `init-store` on Server, every Server using PD for graph +metadata, Store waiting for a PD quorum, one PD REST secret shared by three readers). The chart encodes that +contract so operators do not have to; the details are in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#chart-details). + +```mermaid +flowchart LR + subgraph Kubernetes cluster + PD[PD StatefulSet
Raft group of 3, PVC] + Store[Store StatefulSet
3 replicas, PVC] -->|register, heartbeat| PD + Server[Server Deployment
3 replicas] -->|metadata, discovery| PD + Server -->|gRPC read/write| Store + Hubble[Hubble Deployment
optional UI] -->|discovers Servers via PD| PD + end + Client[Client / hugegraph-client] -->|REST / Gremlin| Server +``` + +Startup order is enforced by the chart, not by the operator: PD pods elect a leader first, each Store pod waits in +an init container until a majority of PD peers report ready, and Servers keep restarting their storage wait until +Stores have registered. A fresh install converges without manual steps. + +### 2 Prerequisites + +- Kubernetes 1.23 or later (the chart renders `autoscaling/v2` and `policy/v1`) +- Helm 3; the `--reset-then-reuse-values` flag mentioned under Upgrade needs Helm 3.14 or later +- Dynamic volume provisioning: a default StorageClass, or an explicit `storageClassName` for PD and Store +- Memory for nine JVMs in the default topology; see the resource note under Install + +The chart requires component images that carry the PD readiness endpoint and PD REST authentication (both merged +for the release after 1.7.0). The default image tags already point at builds that include them; 1.7.0 images are +not supported. + +### 3 Install + +#### 3.1 Get the chart + +The chart is not published to a chart repository yet, so install it from the source tree: + +```bash +git clone https://github.com/apache/hugegraph.git +cd hugegraph +``` + +#### 3.2 Install with default values + +First confirm `kubectl` points at the intended cluster and that it can provision volumes. PVCs stuck in `Pending` +for want of a StorageClass are the most common first-run failure: + +```bash +kubectl config current-context +kubectl get storageclass +``` + +Then install: + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace --wait --timeout 15m +``` + +`--wait` makes Helm block until every workload is ready, which for a distributed cluster is the signal that PD +elected a leader, Stores registered, and Servers came up; without it `helm install` returns as soon as the objects +are created. A fresh cluster normally converges in a few minutes; the 15 minute timeout leaves room for slow image +pulls. + +Two defaults to know before going further: + +- **No resources are set.** Every pod is BestEffort and each JVM sizes its heap against total node memory. That is + fine on a single node; on a multi-node cluster the heaps oversubscribe the nodes and pods abort. Use + `values-cluster.yaml` or set `resources` per component for anything beyond a laptop. +- **Image tags track `latest`** until the next HugeGraph release is published, with `pullPolicy: Always`. Pin tags + or digests for production. + +#### 3.3 Topology presets + +The chart ships three values files: + +| File | Topology | Intended use | +|------|----------|--------------| +| `values.yaml` | 3 PD + 3 Store + 3 Server | Default; preferred anti-affinity, auth on, Hubble off | +| `values-single.yaml` | 1 + 1 + 1 | Single-node development and CI; smaller PVCs | +| `values-cluster.yaml` | 3 + 3 + 3 | Production starting point: JVM heap and resource settings, PD/Store PodDisruptionBudgets, `required` anti-affinity | + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace \ + -f helm/hugegraph/values-single.yaml --wait --timeout 15m +``` + +`values-cluster.yaml` is a starting point, not a capacity guarantee: recalculate resources for your graph size and +traffic. The full parameter reference (every component, probe, scheduling, and Secret knob) is kept in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration). + +#### 3.4 Verify the install + +```bash +helm test hugegraph --namespace hugegraph +``` + +Read the generated admin password and call the API: + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin -o jsonpath='{.data.password}' | base64 --decode)" +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +``` + +The commands above assume the release is named `hugegraph`; with another name, substitute the release-prefixed +resource names (`kubectl get svc,secret -n ` lists them). The post-install notes printed by +`helm install` repeat these commands with the right names filled in. + +### 4 Authentication and Secrets + +Authentication is on by default and the chart manages three Secrets. Each credential resolves in the same order: +an `existingSecret` you created wins, then an inline value, then a random value generated at install time. + +| Secret | Key | Used for | Bring your own with | +|--------|-----|----------|---------------------| +| `-admin` | `password` | Server admin account, Hubble login | `server.auth.admin.existingSecret` | +| `-auth-token` | `token_secret` | JWT signing key shared by all Server replicas | `server.auth.token.existingSecret` | +| `-pd-auth` | `secret-key` | PD REST authentication, read by PD, Server, and Hubble | `pd.auth.existingSecret` | + +To manage a credential yourself, create the Secret before installing and point the matching `existingSecret` value +at it; the chart never modifies a Secret it did not create. Value constraints: the admin password must not contain +newlines, carriage returns, or backslashes; the JWT key must be at least 32 bytes; the PD secret must be printable +ASCII. Invalid values are rejected at render time or by the startup wrapper, not silently truncated. + +Chart-managed Secrets are kept on uninstall and reused by a later install under the same release name. + +
+Rotation and caveats + +- The admin password is applied only when the auth metadata is first created, so changing the Secret later does not + rotate an existing cluster's password. Rotate it through the Server's auth API instead. +- Rotating the PD REST Secret rolls PD, Server, and Hubble together on the next `helm upgrade`, which keeps their + copies in step. Template-only pipelines (`helm template`, GitOps renderers) cannot see live Secrets, so there the + rotation-detecting annotation is inert. +- All three Secrets exist even if you only ever read one: the post-install notes print the exact `kubectl get + secret` commands for the admin password and the PD secret. +
+ +### 5 Health checks and startup order + +PD exposes two health endpoints, and the chart deliberately uses both: + +- `/v1/health` answers 200 as soon as the REST listener is up. It never consults Raft, so it cannot see a lost + quorum. +- `/v1/ready` answers 503 until the PD Raft group has a leader, so it reports quorum, not just a live process. + +The chart puts PD **readiness** and the Store init-container wait on `/v1/ready`: a Store only starts once a +majority of PD peers are quorum members, and a PD that lost its leader drops out of Service endpoints until a +leader is back. PD **startup and liveness** stay on `/v1/health` on purpose: a PD that merely lost its leader is +still a healthy Raft member, and restarting it would make the outage worse. + +Server startup gets a matching budget: the image would normally kill a Server still starting after 120 seconds, so +the chart derives `HG_SERVER_STARTUP_TIMEOUT_S` from the startup probe (450 seconds by default) and raises a lower +configured probe budget to that floor. Raise `server.probes.startup` if your storage takes longer to come up, and +the image budget follows. + +### 6 Enable the Hubble UI + +Hubble is off by default so API-only clusters stay lean. Enable it on a running release: + +```bash +helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values --set hubble.enabled=true +``` + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-hubble 8088:8088 +``` + +Open `http://127.0.0.1:8088` and log in as `admin` with the admin password from Section 3.4. Hubble discovers the +Servers through PD, so the cluster operations view works without extra wiring. Hubble serves plain HTTP: reach it +through a port-forward or an HTTPS-terminating Ingress, never directly from an untrusted network. Running Hubble +outside the cluster is possible but takes more wiring; see +[Reaching Hubble](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#reaching-hubble-pick-one-path) in +the chart README. + +### 7 Upgrade + +```bash +helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values +``` + +`--reuse-values` keeps the release's existing overrides; without it the upgrade rebuilds the release from chart +defaults. Any upgrade that changes a Pod template rolls that workload once. Points worth planning around: + +- **The first upgrade after a fresh install rolls PD, Server, and Hubble once**, when the Secret-tracking + annotations first observe the install-created Secrets. Store is untouched. +- **Store rolling updates advance on a listener check, not on shard recovery**, so the controller can replace the + next Store while the previous one is still rejoining its shard groups. For a production image roll, set + `store.updateStrategy.type=OnDelete` and replace Store pods one at a time, confirming the previous Store shows + `Up` in PD before the next; PD restarts are one pod at a time either way, and `pd.updateStrategy.type=OnDelete` + gives the same manual control for maintenance windows. +- **PVC sizes cannot be changed by upgrade**: Kubernetes forbids changing StatefulSet `volumeClaimTemplates`, so an + upgrade with a new `storage.size` is rejected in full. The chart README documents the resize procedure for + StorageClasses that support volume expansion. + +Scaling Server up and down is a values change (`server.replicas`, or `server.hpa`). Scaling **PD or Store down is +not**: Raft and shard membership are persisted, deleting pods does not reconfigure them, and a 3-to-1 PD shrink +permanently loses quorum. The chart rejects an upgrade whose replica count is below the live StatefulSet; the +manual drain-then-scale procedure is in the chart README under +[Scaling](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#scaling). + +### 8 Uninstall + +```bash +helm uninstall hugegraph --namespace hugegraph +``` + +Two kinds of state survive on purpose. PersistentVolumeClaims created by the StatefulSets are kept (Kubernetes +behavior); delete them explicitly once the data is no longer needed. The chart-managed Secrets are also kept, so a +later install under the same release name comes back with the same credentials. + +### 9 Limitations + +- No NetworkPolicy resources yet. With the PD Raft IP allowlist disabled in-cluster (pod IPs change; the allowlist + resolves peers once at boot and then blocks them), in-cluster network access control is currently the operator's + responsibility. +- Image tags track `latest` until the next HugeGraph release publishes versioned images; pin tags or digests for + anything long-lived. +- After creating a graph, other Server replicas can lag for a short window before they serve queries for it, so a + query routed to a not-yet-converged replica can fail with an error such as `Could not rebind [g]`. Retry with + backoff, or use sticky routing for create-then-query flows; cluster-wide graph readiness is tracked in + [#3137](https://github.com/apache/hugegraph/issues/3137). +- Store recovery is operator-triggered on current builds: re-replication after Store loss, leader balancing, and + partition rebalancing run only when called through PD's REST API. The chart README's + [Disaster Recovery](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#disaster-recovery) section is + the runbook. +- No TLS termination inside the cluster, no backup tooling, no Operator, and no bundled monitoring stack. + +### 10 Troubleshooting + +| Symptom | First check | +|---------|-------------| +| Store pods stuck in `Init:0/1` | PD is not ready: `kubectl logs -c wait-for-pd`, then the PD pods | +| PVCs stay `Pending` | No default StorageClass, or the provisioner is down: `kubectl get sc` | +| Pods OOM killed or restarting on multi-node | No resources set, JVM heaps sized to node memory: use `values-cluster.yaml` | +| Query fails right after creating a graph | Replica convergence window: see Limitations above | + +Longer walkthroughs for each case are in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#troubleshooting). From 1da7486448f0607c385abb35bffc894657a7fd0a Mon Sep 17 00:00:00 2001 From: Himanshu Verma Date: Fri, 18 Sep 2026 23:55:56 +0530 Subject: [PATCH 2/9] doc: register the Helm page in the version route map The versioned build validates that every canonical Docs page has a logical ID in data/version_routes.json; the two new pages were missing from it, failing the Build latest job. Adds en: and cn: entries with latest populated and the four older versions null, the same shape as the other pages introduced after 1.7. --- data/version_routes.json | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/data/version_routes.json b/data/version_routes.json index bd7e4351ef..ab687468dc 100644 --- a/data/version_routes.json +++ b/data/version_routes.json @@ -715,6 +715,13 @@ "1.3": null, "1.0": null }, + "cn:quickstart/hugegraph/hugegraph-helm": { + "latest": "cn/docs/quickstart/hugegraph/hugegraph-helm/", + "1.7": null, + "1.5": null, + "1.3": null, + "1.0": null + }, "cn:quickstart/hugegraph/hugegraph-server": { "latest": "cn/docs/quickstart/hugegraph/hugegraph-server/", "1.7": "cn/docs/quickstart/hugegraph/hugegraph-server/", @@ -1422,6 +1429,13 @@ "1.3": null, "1.0": null }, + "en:quickstart/hugegraph/hugegraph-helm": { + "latest": "docs/quickstart/hugegraph/hugegraph-helm/", + "1.7": null, + "1.5": null, + "1.3": null, + "1.0": null + }, "en:quickstart/hugegraph/hugegraph-server": { "latest": "docs/quickstart/hugegraph/hugegraph-server/", "1.7": "docs/quickstart/hugegraph/hugegraph-server/", From c4ed3619c4286bf8415994ad6448dec4299c9109 Mon Sep 17 00:00:00 2001 From: Himanshu Verma Date: Thu, 24 Sep 2026 21:47:05 +0530 Subject: [PATCH 3/9] doc: refresh the Helm page and add a Kubernetes operations page Bring the deployment page level with the chart at 8603cdbb3: the values-cluster preset now ships NetworkPolicy and 5Gi/8Gi Store memory, PD startup and liveness derive from the replica count (pd.livenessPath, single-PD /v1/ready per apache/hugegraph#3222), the Store roll procedure no longer treats Up in PD as the between-pods check, and the Limitations follow the merged fixes, including in-place empty-PVC Store recovery on images carrying apache/hugegraph#3234. Add an operations page (EN and CN, registered in the version route map) rewritten for operators from the chart README: ports and health, scheduling, partition sharding, NetworkPolicy with the per-CNI NodePort client behavior, safe Store rolls, the disaster recovery runbook with the post-#3234 procedure, scaling including the Store drain steps, running Hubble outside the cluster, and the two causes of "Could not rebind". Verified: scripts/hugo.sh build passes and both new pages render with their cross-links in EN and CN. --- .../hugegraph/hugegraph-helm-operations.md | 356 +++++++++++++ .../quickstart/hugegraph/hugegraph-helm.md | 41 +- .../hugegraph/hugegraph-helm-operations.md | 483 ++++++++++++++++++ .../quickstart/hugegraph/hugegraph-helm.md | 45 +- data/version_routes.json | 14 + 5 files changed, 913 insertions(+), 26 deletions(-) create mode 100644 content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md create mode 100644 content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md new file mode 100644 index 0000000000..1b5d66dcb3 --- /dev/null +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -0,0 +1,356 @@ +--- +title: "在 Kubernetes 上运维 HugeGraph" +linkTitle: "Kubernetes 运维 (Helm)" +weight: 5 +search_keywords: + - helm + - kubernetes + - operations + - disaster recovery + - networkpolicy +--- + +### 1 适用范围 + +本页面向已经用 Helm chart 部署好 HugeGraph 集群的运维场景。安装、升级、卸载见 +[部署页](/cn/docs/quickstart/hugegraph/hugegraph-helm/)；完整参数参考仍在 +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration)。 + +下文命令假定 release 名为 `hugegraph`、namespace 为 `hugegraph`，请按需替换。两个凭据会反复用到，都从 +chart 管理的 Secret 读取： + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin \ + -o jsonpath='{.data.password}' | base64 --decode)" +PD_SECRET="$(kubectl get secret -n hugegraph hugegraph-pd-auth \ + -o jsonpath='{.data.secret-key}' | base64 --decode)" +``` + +### 2 端口与健康 + +| 组件 | 端口 | 用途 | +|------|------|------| +| PD | `8686` | gRPC（Store 与 Server 客户端） | +| PD | `8620` | REST / 健康探测 | +| PD | `8610` | Raft | +| Store | `8500` | gRPC | +| Store | `8510` | Raft | +| Store | `8520` | REST / 健康探测 | +| Server | `8080` | Gremlin 与 REST API | + +所有端口都可以通过 values 配置；修改 `server.port` 会同时更新监听、容器端口和 Service。进程存活但停止响应的组件由 +存活探测终结：默认 20 秒周期、3 次失败阈值把一个卡死 Store 的影响时间限制在约一分钟内，重启后 raft 几秒内就会转移 +它持有的分区 leader。 + +通过 port-forward 访问 API： + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/graphs +``` + +### 3 调度 + +每个组件（`pd`、`store`、`server`、`hubble`）都提供 `nodeSelector`、`tolerations`、`affinity`、 +`topologySpreadConstraints` 和 `priorityClassName`。例如把 Store 固定到打了标签的节点： + +```yaml +store: + nodeSelector: + hugegraph/role: storage +``` + +`antiAffinity`（`required` | `preferred` | `disabled`）为 `pd`、`store`、`server` 渲染按主机名的 +pod 反亲和预设；Hubble 设计为单副本，没有这个配置项。设置原生 `affinity` 会整体替换预设。三者默认都是 +`preferred`，因此节点数少于副本数的集群也能调度。代价是：节点资源紧张时调度器可能把副本放到同一节点，单个节点 +故障就可能同时带走多个 PD 或 Store 副本。节点足够的生产集群应把 `pd.antiAffinity` 和 `store.antiAffinity` +固定为 `required`，`values-cluster.yaml` 即如此。 + +### 4 分区分片 + +全新安装会在 `store.replicas` 不低于 3 时把 PD 的分片数种子设为 3，否则设为 1；没有这个种子，PD 镜像会把 +`partition.default-shard-count` 固定为 1，chart 部署的集群就没有 Store 级高可用。chart 通过 PD 的 +`JAVA_OPTS` 传入 `-Dpartition.default-shard-count`，不影响镜像的 JVM 自动调优。 + +**种子只在首次引导时生效。** PD 第一次以空存储启动时把分片数持久化到自己的元数据里，此后存储的值才是权威：之后修改 +`pd.partition.defaultShardCount` 或跨过推导边界地伸缩 `store.replicas`，对已初始化的集群都没有效果。要修改运行中 +集群的分片数，请使用 PD 自己的配置 API（只接受不超过存活 Store 数的奇数），然后触发 +`GET /v1/task/patrolPartitions`，并预期分片组会发生再分配。 + +分片数还决定初始分区数：`store.replicas x storeMaxShardCount / shardCount`，只在引导时计算一次。按镜像默认的 +`store-max-shard-count`（12），默认 3 Store 安装得到 12 个分区；需要更多分区时调大 +`pd.partition.storeMaxShardCount`（同样只在首次引导时生效）。 + +显式设置的 `pd.partition.defaultShardCount` 必须是奇数且不超过 `store.replicas`；其他值会在渲染时被 chart +拒绝，因为 PD 会悄悄钳制或拒绝它们，渲染通过并不等于设置生效。 + +### 5 NetworkPolicy + +`networkPolicy.enabled` 为每个组件（PD、Store、Server，启用时还有 Hubble）渲染一个 NetworkPolicy。每个策略 +都对自己的 Pod 做双向隔离并列出所需流量，因此应用顺序无关紧要。`values.yaml` 默认关闭（基础配置无法预知你的 +客户端是谁），`values-cluster.yaml` 默认开启。 + +它只在集群网络插件强制执行 NetworkPolicy 时才生效（kind v0.25 及以上、k3s、Calico、Cilium）；其他插件会接受 +对象但不执行。验证方法：在另一个 namespace 起一个不带 chart 标签的 Pod，`curl` PD 客户端 Service 的 REST +端口，必须超时。 + +策略开启后，release 只放行自己的流量： + +| 目标 | 来源与端口 | +|---|---| +| PD | PD：raft、gRPC。Store、Server、`pd` 模式的 Hubble：gRPC、REST | +| Store | Store：raft。Server：gRPC、REST。`pd` 模式的 Hubble：REST | +| Server | Hubble 和 `helm test` Pod：`server.port` | +| Hubble | 无（port-forward 走回环，不需要规则） | + +每个组件还可以走 53 端口解析 DNS。release 之外的任何来源都必须列入 +`networkPolicy..extraIngress` 才被放行，包括 Ingress 控制器和 NodePort / LoadBalancer Service +的客户端。在 `extraIngress` 为空时暴露 PD、Server 或 Hubble，或设置 `server.advertiseUrl`，渲染会直接失败 +而不是打开端口。该检查只看 chart 自己创建的暴露；你自行添加的 Service、Gateway 路由或代理需要自己的条目。 + +两个出向事实需要规划。PD、Store、Server 除 DNS 外不访问 release 以外的任何地址，因此需要外呼的功能在策略开启时 +不可用；默认唯一的外呼方是 Store 镜像，它每次启动都从 github.com 下载 `libjemalloc.so`。策略开启时该连接约两分 +钟后超时，Store 不带 jemalloc 继续启动（kind 上实测：就绪时间从 11 秒变为 151 秒）；无外网的集群同理。另外 +`helm test` Pod 不被任何 chart 策略选中，因此在你自己的 namespace 级默认拒绝策略下，要给它放行到 `server.port` +和 DNS 的出向流量。 + +**放行其他工作负载。** 每条规则的 `from` 必须指明对端；要放行任意地址，请显式写 `0.0.0.0/0` 这样的 +`ipBlock`。同一个对端里的 `namespaceSelector` 和 `podSelector` 必须同时匹配；写成两个对端则满足其一即可。 +NodePort / LoadBalancer 客户端在 Pod 看来是什么地址，取决于网络插件、`externalTrafficPolicy` 和请求到达的 +节点。在双节点 kind 集群上用 NodePort Server 实测： + +- kindnet，以及开 kube-proxy replacement 的 Cilium：请求发到 Server 所在节点时带客户端地址；经另一节点转发时 + 带那个节点的地址。 +- Calico：经另一节点转发的请求来自该节点在 Pod CIDR 里的隧道地址。 +- 用 kube-proxy 的 Cilium：任何 `ipBlock` 规则都放行不了 NodePort 流量，因为 Cilium 用自己的节点身份而不是 + CIDR 识别节点地址。 + +请用你实际运行的插件测试，并按看到的来源地址写 CIDR。放行 Ingress 控制器、应用 namespace 和 Prometheus 的 +完整示例： + +```yaml +networkPolicy: + server: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: ingress-nginx + podSelector: + matchLabels: + app.kubernetes.io/name: ingress-nginx + ports: + - port: 8080 + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: apps + ports: + - port: 8080 + pd: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: monitoring + ports: + - port: 8620 +``` + +### 6 安全地滚动 Store 镜像 + +Store 的滚动更新以监听检查推进，而不是以分片恢复推进，控制器可能在上一个 Store 尚未重新加入分片组时就替换下一 +个。生产环境滚动镜像时，设置 `store.updateStrategy.type=OnDelete`，逐个删除 Store Pod，并在两次删除之间做检查。 + +PD 里的 `Up` 不是这个检查。PD 在注册时就把 Store 标为 `Up`，此时它还没恢复任何分区；已停止的 Store 也会在 +keep-alive 记录过期前（当前镜像为 300 秒）一直保持 `Up` 并留在所有分片组里：窗口内删除又回来的 Pod 根本不会离开 +`Up` 状态。检查要从 Pod 开始： + +```bash +kubectl -n hugegraph wait --for=condition=Ready \ + pod/hugegraph-store- --timeout=10m +``` + +然后从 PD leader 逐组检查分片成员与 leader（找 leader 的方法见下文"灾难恢复"）： + +```bash +curl -s -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/shardGroups | jq ' + .shardGroups[] | {id: (.id // 0), + shards: [.shards[] | {storeId, role}], + leaders: [.shards[] | select(.role=="Leader")] | length}' +``` + +只有当被替换的 Pod 已 `Ready`、它的 Store id 在 `/v1/stores` 里显示新的 `lastHeartBeat`、且每个分片组都报告 +完整分片数和恰好一个 `Leader` 时，才删除下一个 Store。 + +要清楚这证明不了什么：分片列表是 PD 的成员记录，不代表该 Store 已追上 raft 日志。当前镜像没有任何端点报告"恢复 +完成"。想看得更近，port-forward 被替换的 Store，读它自己对某个分片组的视图：`GET :8520/v1/partition/` +返回该 Store 持有的 raft 角色、term 和已提交 index，Store 停机时会失败；term 和 index 要与对端 Store 上的同一 +组对比着看，不要单独读。复数形式的 `GET :8520/v1/partitions` 在 +[apache/hugegraph#3232](https://github.com/apache/hugegraph/pull/3232)（2026-09-24 合入）之前构建的镜像上， +对任何跟随分片组的 Store 返回 500；之后的镜像对每个 Store 都返回 200，被跟随分组的 `conf` 和 `peers` 为 +null。逐组路径在两类镜像上都可用。 + +成员检查通过后留出余量再删下一个 Pod，把 `store.pdb.minAvailable` 保持在 `replicas - 1`，意外的第二次驱逐会被 +拒绝；任何缺分片或没有 leader 的分组都应视为停止信号。真正的"分区恢复完成"信号是上游工作，在 +[apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229) 跟踪。 + +### 7 灾难恢复 + +当前版本 PD 自动做的事很少：60 秒一次的巡检只把停止心跳的 Store 标为 `Offline`。没有自动副本重建；丢失 Store +的副本重新放置、按分片数校正分片组、处理 Tombstone Store，都只在显式触发分区巡检时执行。 + +任务端点在收到请求的那台 PD 上本地执行，follower 会返回空的成功响应但什么也不做。port-forward 客户端 Service +选中的 PD 是任意的，所以先找 leader，再 port-forward 那个 Pod（forward 在前台运行，需要第二个终端）： + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-pd-client 8620:8620 +# 读 .data.pdLeader.raftUrl，其主机名即 leader Pod。 +curl -su "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/members +# 停掉 Service 的 forward，改为 forward leader Pod。 +kubectl port-forward -n hugegraph pod/ 8620:8620 +# 校正分片组并处理 Tombstone Store。 +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/patrolPartitions +# 先摊平 Raft leader，再摊平分区数据。 +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balanceLeaders +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balancePartitions +``` + +任务跑完后再读一次 `/v1/members`：如果中途 leader 迁移，后面的任务其实跑在 follower 上、什么也没做。 +`balancePartitions` 之后至少等 180 秒再执行 `balanceLeaders`：`balancePartitions` 即使什么都没搬也会设置 +180 秒的 balance-shard 标志，窗口内的 `balanceLeaders` 会被拒绝。在 +[apache/hugegraph#3233](https://github.com/apache/hugegraph/pull/3233)（2026-09-24 合入）之前构建的镜像上， +拒绝表现为裸的 HTTP 500，原因只在 PD 日志里；之后的镜像把原因放进响应体： +`{"status":1001,"error":"balance shard is processing, please try later!"}`。 + +要分辨"真的执行了"和"空跑"，得看 PD leader 的日志，响应本身分辨不了：`patrolPartitions` 无论有没有修复、在 +leader 还是 follower 上，都返回同样的空成功（看日志里的 `reallocShards`、`shardOffline`、`storeTurnoff`，或 +对比前后的 `/v1/shardGroups`）；`balancePartitions` 在 leader 上返回 `{}`，在 follower 上返回空响应体。只有 +`balanceLeaders` 的响应体携带工作内容。可区分的任务响应是上游工作，在 +[apache/hugegraph#3231](https://github.com/apache/hugegraph/issues/3231) 跟踪。 + +替换一个回不来的 Store 之后执行 `patrolPartitions`，集群稳定后执行 `balancePartitions`，重启导致 leader 分布 +倾斜后执行 `balanceLeaders`。 + +**丢失 Store 卷。** 带着空 PVC 重建的 Store，在携带 +[apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234)（2026-09-24 合入）的镜像上可以原地 +恢复；更早的镜像，包括所有已发布版本的镜像，都不行。两种情况下替换者都会以**新的 Store ID** 注册，而 Pod 名、 +DNS 名、raft 地址不变，`/v1/stores` 会在同一地址下列出两个 ID。 + +在含 #3234 的镜像上，退役流程是可用的：在 `/v1/stores` 里找到旧 ID（被替换 Pod 地址下不是新注册的那一行），在 +PD leader 上 `POST /v1/store/` 并携带 `{"storeState":"Tombstone"}`，执行 +`GET /v1/task/patrolPartitions`，然后等待；确认每个分片组都恢复到完整分片数且恰好一个 leader、没有分组再引用 +旧 ID、被替换 Store 自己的 `:8520/v1/partition/` 对每个分组都返回 200；最后 +`DELETE /v1/store/` 清除退役记录。在 3+3+3 安装上用 `master` `dbb6663a` 构建的镜像实测：替换者 156 秒 +就绪，Tombstone 加巡检之后 1 秒内 12 个分组全部收敛到新 ID（空 Store 通过 raft 快照安装追平数据），持续写入 +没有丢失任何已确认的写。 + +在不含 #3234 的镜像上，同样的退役流程会执行但修不好分组：PD 发起配置变更，但分组 leader 看到该地址已在组内， +jraft 无可添加，分组继续记录旧 ID。实测：20 分钟、三次巡检之后，12 个分组仍全部引用退役 ID，替换 Store 不持有 +任何分区。健康面板完全看不出这个故障：`/v1/stores`、集群状态、Hubble、Pod 就绪全部显示健康，而每个分片组实际 +只运行在两个存活副本上。唯一能看出问题的检查是被替换 Store 自己的 `:8520/v1/partition/`。 + +因此默认做法不变：替换 Store Pod 时保留它的 PVC（Store id 存在数据目录里，Pod 会以同一 id 回来）。空 PVC 替换 +只作为含 #3234 镜像上的恢复手段；在已发布镜像上，卷真的丢了就当集群已降级处理，准备重建而不是原地恢复。 + +周期性 leader 均衡在 [apache/hugegraph#3135](https://github.com/apache/hugegraph/issues/3135) 跟踪； +灾难恢复指标在 [apache/hugegraph#3136](https://github.com/apache/hugegraph/issues/3136) 跟踪。 + +### 8 伸缩 + +PD 和 Store 在资源名里为最大 StatefulSet 序号预留了空间，伸缩不会改名 PersistentVolumeClaim、不会移动 Pod +身份；两者上限都是 99 副本。Server 通过 `server.replicas` 或 `server.hpa` 自由伸缩（HPA 开启时 Deployment +省略 `spec.replicas`，升级不会覆盖自动扩缩的副本数）。 + +分阶段上线没法写进 values 文件（schema 要求每个组件至少一个副本）；用 `kubectl scale statefulset +hugegraph-store --replicas=0` 分阶段，准备好后再扩回来。Server 会保持未就绪等待 Store 注册，下一次 +`helm upgrade` 会按 values 恢复完整拓扑。 + +修改线上 release 的 PD 或 Store 副本数不是普通的 values 变更：raft 与分片成员关系是持久化的，Pod 本身不会 +重新配置它们。chart 通过读取线上 StatefulSet 拒绝 PD 的双向变更和 Store 的缩容，全新安装不受影响。 + +**PD，双向。** chart 渲染的对端列表只作为 raft 的引导配置生效，已初始化的组会忽略它：3 扩到 5 只是多起两个 +PD，投票配置仍是三个；3 缩到 1 直接失去多数派。成员变更走 PD 客户端 API（没有 REST 路由），chart 无法代劳。 +先通过 PD 改成员、在 `/v1/members` 确认新配置、`kubectl scale` 线上 StatefulSet、再用匹配的 values 执行 +`helm upgrade`；在你自己的构建上验证过这套流程之前，请按打算长期保留的 PD 数量安装。 + +**Store，缩容。** 迁出是状态转换，不是均衡：`patrolPartitions` 和 `balancePartitions` 都不会退役健康的 +Store。像灾难恢复退役被替换 Store 那样退役要下线的 Store： + +1. 确认剩余 Store 仍能承载持久化的分片数（`pd.partition.defaultShardCount`；留空时 `store.replicas` 不低于 + 3 推导为 3）。 +2. 通过 `/v1/stores` 按 Pod 地址把要删除的序号（最大的几个）映射到 Store id。 +3. 对每个要下线的 id `POST /v1/store/`，携带 `{"storeState":"Tombstone"}`。 +4. 等到 `/v1/shardGroups` 不再列出这些 id，且每个分组都报告完整分片数和一个 leader。 +5. `kubectl scale` 线上 StatefulSet，再用匹配的 values 执行 `helm upgrade`。 + +删除被移除序号的 PVC 是独立且不可逆的操作；只在第 4 步确认数据已迁走之后再做。 + +### 9 在集群外运行 Hubble + +集群内 Hubble（`hubble.enabled=true` 加 port-forward）是推荐路径，部署页已覆盖。Hubble 必须在集群外运行时有 +两条路径。 + +**直连 Server URL**（图、schema、数据、Gremlin；不需要 PD 发现）：关闭 chart 内的 Hubble，暴露 Server +（`server.service.type` NodePort/LoadBalancer，或 Ingress），用如下配置运行独立的 Hubble 镜像： + +```properties +pd.enabled=false +server.direct_url=https://: +``` + +配置文件挂载到镜像内的 `/hubble/conf/hugegraph-hubble.properties`（工作目录是 `/hubble`）。 +`server.direct_url` 要用 HTTPS 或可信通道：登录会把 Server 凭据发到这个 URL。 + +**PD 发现**（集群外的 Hubble 向 PD 询问 Server 地址）：`*.svc` 这类集群内名字在外部不可解析，chart 提供两个 +配置项。把 `server.advertiseUrl` 设为外部 Hubble 发现后要使用的绝对 `http(s)://` URL，chart 会用它替代集群内 +Service URL 注册到 PD。再暴露 PD 客户端 Service（`pd.service.type` NodePort/LoadBalancer），这需要 +`pd.service.allowInsecureExposure=true`，因为 PD gRPC 没有认证；先限制谁能访问它（chart 的 NetworkPolicy +开启时，调用方没有列入 `networkPolicy.pd.extraIngress` 前渲染会拒绝这种暴露）。然后用如下配置运行独立 +Hubble： + +```properties +pd.enabled=true +pd.peers=: +pd.server=: +``` + +取舍：设置 `server.advertiseUrl` 后，每个 Server 副本注册的都是同一个逻辑 URL，PD 会把它返回给所有发现客户 +端，包括集群内的 Hubble。留空则走默认的集群内路径，每个 Server Pod 注册自己的 IP。 + +本机快速验证（集群和 Hubble 在同一台机器）：port-forward Server `8080` 和 PD 客户端 `8620`/`8686`，设置 +`server.advertiseUrl=http://127.0.0.1:8080`，用 `--network host` 和上面的 PD 配置运行独立 Hubble，再打开 +`8088` 端口。 + +### 10 Gremlin 报 "Could not rebind" 时 + +这个报错有两个成因，持续时间不同。 + +**建图后的收敛窗口。** 处理 `CreateGraph` 的 Server 会等自己的 Gremlin 绑定完成才返回 HTTP 200 +（[#3138](https://github.com/apache/hugegraph/pull/3138)），所以在同一个 Server 上建图后立即查询是可靠的。 +其他副本独立收敛，在完成之前，经负载均衡 Service 路由到未收敛副本的 Gremlin 查询可能报 400，如 +`Could not rebind [g]`。带退避重试（窗口通常几秒内关闭），对"建图后立即验证"的流程用会话粘滞或 +port-forward，或先在每个副本上轮询 `/graphs` 再放开查询流量。集群级就绪在 +[#3137](https://github.com/apache/hugegraph/issues/3137) 跟踪。 + +**在 PD 滚动期间启动的 Server Pod。** Gremlin Server 只在启动时实例化一次图；那一刻 PD 客户端连不上的话， +这个 Pod 会终身通过就绪探测、正常提供 REST，而每个发到它的 Gremlin 请求都报 `Could not rebind [graph]`。 +它的 `hugegraph-server.log` 会写明： + +``` +Graph [DEFAULT-hugegraph] configured at [...] could not be instantiated and +will not be available in Gremlin Server +``` + +任何同时滚动了 PD 和 Server 的升级之后，逐个检查 Server Pod 的 Gremlin（镜像不带 curl，逐个 port-forward）： + +```bash +kubectl port-forward -n hugegraph pod/ 8080:8080 +curl -s --compressed -u "admin:${PASSWORD}" -H 'Content-Type: application/json' \ + -X POST http://127.0.0.1:8080/gremlin \ + -d '{"gremlin":"graph.traversal().V().limit(1).count()","aliases":{"graph":"DEFAULT-hugegraph"}}' +``` + +健康的 Pod 返回 `result.data`；返回 `Could not rebind` 的 Pod 直接删除。只要 PD 稳定，替换者会正常完成绑定 +（实测：与 PD 滚动重叠的 12 次 Server 启动中 4 次命中，每次删除后都恢复）。 diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md index 72911f369c..15c001fecc 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md @@ -22,7 +22,9 @@ Helm chart 在 Kubernetes 上部署一套分布式 HugeGraph 集群：PD、Store 分布式 HugeGraph 集群有一套启动约定（Server 不执行 `init-store`、每个 Server 都通过 PD 读写图元数据、Store 等待 PD 形成多数派、一个 PD REST 密钥由三个组件共用）。chart 把这套约定固化下来，运维人员无需手工处理；细节见 -[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#chart-details)。 +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#chart-details)。日常运维（NetworkPolicy +细节、灾难恢复、伸缩、安全滚动 Store、在集群外运行 Hubble）见 +[运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/)。 ```mermaid flowchart LR @@ -93,15 +95,17 @@ chart 附带三个 values 文件： |------|------|----------| | `values.yaml` | 3 PD + 3 Store + 3 Server | 默认；preferred 反亲和，认证开启，Hubble 关闭 | | `values-single.yaml` | 1 + 1 + 1 | 单节点开发与 CI；PVC 更小 | -| `values-cluster.yaml` | 3 + 3 + 3 | 生产起点：JVM 堆与资源设置、PD/Store PodDisruptionBudget、`required` 反亲和 | +| `values-cluster.yaml` | 3 + 3 + 3 | 生产起点：JVM 堆与资源设置、PD/Store PodDisruptionBudget、`required` 反亲和、NetworkPolicy 开启 | ```bash helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace \ -f helm/hugegraph/values-single.yaml --wait --timeout 15m ``` -`values-cluster.yaml` 是起点而非容量保证：请按图规模和流量重新核算资源。完整参数参考（每个组件的探针、调度、Secret -配置项）见 [chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration)。 +`values-cluster.yaml` 是起点而非容量保证：请按图规模和流量重新核算资源。其中一个数字值得说明：它给每个 Store 申请 +5Gi、限制 8Gi 内存，远高于 1Gi 的堆，因为 Store 的 RocksDB 缓存在 JVM 堆之外（4Gi 的限制在约 1 GB 数据后就把 +Store OOM 杀掉了）。两个数字要随数据量一起放大。完整参数参考（每个组件的探针、调度、Secret 配置项）见 +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration)。 #### 3.4 验证安装 @@ -154,8 +158,11 @@ PD 暴露两个健康端点，chart 有意同时使用两者： - `/v1/ready` 在 PD Raft 组选出 leader 之前返回 503，反映的是多数派状态而不只是进程存活。 chart 把 PD 的**就绪探测**和 Store init 容器的等待放在 `/v1/ready` 上：Store 只有在多数 PD 成为多数派成员后才启动， -失去 leader 的 PD 会退出 Service 端点，直到 leader 恢复。PD 的**启动和存活探测**则特意留在 `/v1/health` 上：只是失去 -leader 的 PD 仍是健康的 Raft 成员，重启它只会让故障恶化。 +失去 leader 的 PD 会退出 Service 端点，直到 leader 恢复。PD 的**启动和存活探测**按副本数推导（`pd.livenessPath` +可覆盖）。多副本时特意留在 `/v1/health` 上：只是失去 leader 的 PD 仍是健康的 Raft 成员，重启它只会让故障恶化。 +单副本 PD 是例外，推导为 `/v1/ready`：它没有选举可失去，而一个永久退位的 PD（例如磁盘写满导致 Raft 快照失败之后， +[apache/hugegraph#3222](https://github.com/apache/hugegraph/issues/3222)）会一直用 `/v1/health` 返回 200 却不再 +服务写入；改用 `/v1/ready` 让 kubelet 把它重启。 Server 的启动获得相同的预算：镜像默认会在 120 秒后杀掉仍在启动的 Server，chart 因此从启动探针推导 `HG_SERVER_STARTUP_TIMEOUT_S`（默认 450 秒），配置的探针预算低于该下限时会被抬高。如果存储层启动更慢，调大 @@ -190,8 +197,11 @@ helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values - **全新安装后的第一次升级会让 PD、Server、Hubble 各滚动一次**，因为跟踪 Secret 的注解第一次观察到安装时创建的 Secret。Store 不受影响。 - **Store 的滚动更新以监听检查推进，而不是以分片恢复推进**，因此控制器可能在上一个 Store 尚未重新加入分片组时就替换 - 下一个。生产环境滚动镜像时，设置 `store.updateStrategy.type=OnDelete`，逐个替换 Store Pod，并在替换下一个之前确认上 - 一个在 PD 中显示 `Up`；PD 本来就逐个重启，`pd.updateStrategy.type=OnDelete` 为维护窗口提供同样的手工控制。 + 下一个。生产环境滚动镜像时，设置 `store.updateStrategy.type=OnDelete`，逐个替换 Store Pod。PD 里的 `Up` 不是两次 + 替换之间的检查：PD 在注册时就把 Store 标为 `Up`，此时分区尚未恢复，已停止的 Store 也要等 300 秒 keep-alive 过期 + 才离开 `Up`。应等被替换的 Pod 变为 `Ready`，再确认每个分片组都报告完整分片数和一个 leader；完整流程见 + [运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/)。PD 本来就逐个重启， + `pd.updateStrategy.type=OnDelete` 为维护窗口提供同样的手工控制。 - **升级不能修改 PVC 大小**：Kubernetes 禁止修改 StatefulSet 的 `volumeClaimTemplates`，带新 `storage.size` 的升级会 被整体拒绝。chart README 记录了支持卷扩容的 StorageClass 上的扩容步骤。 @@ -211,16 +221,21 @@ helm uninstall hugegraph --namespace hugegraph ### 9 限制 -- 尚无 NetworkPolicy 资源。PD 的 Raft IP 白名单在集群内被关闭（Pod IP 会变化；白名单只在启动时解析一次对端，之后就会 - 拦截它们），集群内的网络访问控制目前由运维方负责。 +- `networkPolicy.enabled` 为每个组件渲染只放行本 release 流量的 NetworkPolicy；默认关闭，`values-cluster.yaml` + 中开启。它之所以重要，是因为 chart 在集群内关闭了 PD 的 Raft IP 白名单（Pod IP 会变化；白名单只在启动时解析一次 + 对端，之后就会拦截它们），raft 与 gRPC 端口的访问限制由这些策略承担。策略需要强制执行 NetworkPolicy 的网络插件 + （Calico、Cilium、kind v0.25 及以上、k3s），所有外部调用方（包括 Ingress 控制器）都必须列入 + `networkPolicy..extraIngress`，否则渲染失败。细节见 + [运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/)。 - 镜像 tag 跟踪 `latest`，直到下一个 HugeGraph 版本发布带版本号的镜像；长期运行的环境请固定 tag 或 digest。 - 创建图之后，其他 Server 副本在短暂窗口内可能尚未收敛，路由到这类副本的查询可能报 `Could not rebind [g]` 之类的 错误。请带退避重试，或对"创建后立即查询"的流程使用会话粘滞路由；集群级图就绪在 [#3137](https://github.com/apache/hugegraph/issues/3137) 跟踪。 - 当前版本的 Store 恢复由运维人员触发：Store 丢失后的副本重建、leader 均衡、分区再均衡都只在调用 PD 的 REST API 时 - 执行。chart README 的 - [Disaster Recovery](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#disaster-recovery) 一节即操作 - 手册。 + 执行。丢失了卷的 Store 只有在携带 + [apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234)（2026-09-24 合入，尚未进入任何发布版本） + 的镜像上才能原地恢复；更早的镜像上，替换 Store Pod 时请保留它的 PVC。操作手册见 + [运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/)的灾难恢复一节。 - 集群内无 TLS 终结，无备份工具，无 Operator，无内置监控栈。 ### 10 排障 diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md new file mode 100644 index 0000000000..85d79fa566 --- /dev/null +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -0,0 +1,483 @@ +--- +title: "Operate HugeGraph on Kubernetes" +linkTitle: "Operations on Kubernetes (Helm)" +weight: 5 +search_keywords: + - helm + - kubernetes + - operations + - disaster recovery + - networkpolicy +--- + +### 1 Scope + +This page is for operating a HugeGraph cluster that the Helm chart already +deployed. Installing, upgrading, and uninstalling are on the +[deployment page](/docs/quickstart/hugegraph/hugegraph-helm/); the full +parameter reference stays in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration). + +Commands below assume the release is named `hugegraph` in namespace +`hugegraph`; substitute your own names. Two credentials come up repeatedly, +both read from chart-managed Secrets: + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin \ + -o jsonpath='{.data.password}' | base64 --decode)" +PD_SECRET="$(kubectl get secret -n hugegraph hugegraph-pd-auth \ + -o jsonpath='{.data.secret-key}' | base64 --decode)" +``` + +### 2 Ports and health + +| Component | Port | Purpose | +|------|-------------|---------| +| PD | `8686` | gRPC (Store and Server clients) | +| PD | `8620` | REST / health probes | +| PD | `8610` | Raft | +| Store | `8500` | gRPC | +| Store | `8510` | Raft | +| Store | `8520` | REST / health probes | +| Server | `8080` | Gremlin and REST API | + +All ports are configurable through values; changing `server.port` updates the +listener, container port, and Service together. A stalled component (process +alive but frozen) is ended by its liveness probe: the default 20 s period and +3-failure threshold bound the blast radius of a stalled Store at roughly one +minute, and raft moves its partition leaders within seconds of the restart. + +Reach the API through a port-forward: + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/graphs +``` + +### 3 Scheduling + +Every component (`pd`, `store`, `server`, `hubble`) exposes `nodeSelector`, +`tolerations`, `affinity`, `topologySpreadConstraints`, and +`priorityClassName`. Pinning Store to labeled nodes is just: + +```yaml +store: + nodeSelector: + hugegraph/role: storage +``` + +`antiAffinity` (`required` | `preferred` | `disabled`) renders a hostname +pod-anti-affinity preset for `pd`, `store`, and `server`; Hubble is +single-replica by design and has no such key. Setting a raw `affinity` +replaces the preset entirely. All three default to `preferred`, so the chart +schedules on clusters with fewer nodes than replicas. The trade: under node +pressure the scheduler may co-locate replicas, and a single node failure can +then take more than one PD or Store replica with it. Production clusters with +enough nodes should pin `pd.antiAffinity` and `store.antiAffinity` to +`required`, as `values-cluster.yaml` does. + +### 4 Partition sharding + +A fresh install seeds PD with a partition shard count of 3 when +`store.replicas` is at least 3, and 1 otherwise; without the seed, the PD +image pins `partition.default-shard-count` to 1 and chart-deployed clusters +would run without store-level HA. The chart passes the value as +`-Dpartition.default-shard-count` in PD's `JAVA_OPTS`, so the image's JVM +auto-sizing is unaffected. + +**The seed applies at first bootstrap only.** PD persists the shard count the +first time it starts with empty storage; from then on the stored value is +authoritative, and changing `pd.partition.defaultShardCount` or scaling +`store.replicas` later has no effect on an initialized cluster. To change +the shard count of a running cluster, use PD's own config API (odd values +only, at most the live store count), then trigger +`GET /v1/task/patrolPartitions` and expect shard-group reallocation. + +The shard count also fixes the initial partition count: +`store.replicas x storeMaxShardCount / shardCount`, computed once at +bootstrap. With the image default `store-max-shard-count` of 12, a default +3-store install gets 12 partitions; raise +`pd.partition.storeMaxShardCount` (likewise seeded once) when more +partitions are wanted. + +An explicit `pd.partition.defaultShardCount` must be odd and at most +`store.replicas`; the chart rejects other values at render time because PD +would otherwise clamp or refuse them silently. + +### 5 NetworkPolicy + +`networkPolicy.enabled` renders one NetworkPolicy per component (PD, Store, +Server, and Hubble when enabled). Each policy isolates its own Pods in both +directions and lists the traffic they need, so apply order does not matter. +It is off in `values.yaml`, because the base values cannot know who your +clients are, and on in `values-cluster.yaml`. + +It only works when the cluster's network plugin enforces NetworkPolicy (kind +v0.25 or later, k3s, Calico, Cilium); other plugins accept the objects and +enforce nothing. To check, run a Pod without chart labels in another +namespace and `curl` the PD client Service on the REST port: it must time +out. + +With the policies on, the release admits only its own traffic: + +| To | From, ports | +|---|---| +| PD | PD: raft, gRPC. Store, Server, and Hubble in `pd` mode: gRPC, REST | +| Store | Store: raft. Server: gRPC, REST. Hubble in `pd` mode: REST | +| Server | Hubble and the `helm test` Pod: `server.port` | +| Hubble | nothing (port-forward uses loopback and needs no rule) | + +Every component may also resolve DNS on port 53. Nothing outside the release +is admitted unless it is listed in `networkPolicy..extraIngress`, +including the Ingress controller and the clients of a NodePort or +LoadBalancer Service. Exposing PD, Server or Hubble that way, or setting +`server.advertiseUrl`, with an empty `extraIngress` fails the render instead +of opening the port. The check sees only exposure the chart creates; a +Service, Gateway route or proxy you add yourself needs its own entry. + +Two egress facts to plan around. PD, Store and Server reach nothing outside +the release except DNS, so features that call out do not work with the +policies on; the one default caller is the Store image, which downloads +`libjemalloc.so` from github.com on every start. With the policies on that +connection times out after about two minutes and the Store starts without +jemalloc (measured on kind: Ready after 151 s instead of 11 s); the same +happens on any cluster without internet access. And the `helm test` Pod is +selected by no chart policy, so under a namespace-wide default-deny policy +of your own you must allow it egress to `server.port` and DNS. + +**Letting other workloads in.** Every rule must name its peers in `from`; +to admit any address, write an `ipBlock` such as `0.0.0.0/0` explicitly. A +`namespaceSelector` and a `podSelector` in the same peer must both match; +as two separate peers, either one is enough. What a NodePort or +LoadBalancer client looks like from the Pod depends on the network plugin, +`externalTrafficPolicy`, and the node the request arrives on. Measured with +a NodePort Server on two-node kind clusters: + +- kindnet, and Cilium with kube-proxy replacement: a call to the Server's + own node arrived with the client address; through the other node it + arrived with that node's address. +- Calico: through the other node the call arrived from that node's tunnel + address inside the Pod CIDR. +- Cilium with kube-proxy: no `ipBlock` rule admitted NodePort traffic, + because Cilium identifies node addresses by its own node identities + rather than by CIDR. + +Test with the plugin you run and name the CIDR you see arriving. A worked +example admitting an Ingress controller, an application namespace, and +Prometheus: + +```yaml +networkPolicy: + server: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: ingress-nginx + podSelector: + matchLabels: + app.kubernetes.io/name: ingress-nginx + ports: + - port: 8080 + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: apps + ports: + - port: 8080 + pd: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: monitoring + ports: + - port: 8620 +``` + +### 6 Rolling Store images safely + +Store rolling updates advance on a listener check, not on shard recovery, +so the controller can replace the next Store while the previous one is +still rejoining its shard groups. For a production image roll, set +`store.updateStrategy.type=OnDelete` and delete Store Pods one at a time, +checking between deletions. + +`Up` in PD is not that check. PD marks a Store `Up` at registration, before +the Store has restored any partition, and a stopped Store stays `Up` in +every shard group until its keep-alive entry expires (300 s on current +images): a Pod deleted and back inside that window never leaves `Up` at +all. Start with the Pod instead: + +```bash +kubectl -n hugegraph wait --for=condition=Ready \ + pod/hugegraph-store- --timeout=10m +``` + +Then check shard membership and leadership per group, read from the PD +leader (see Disaster Recovery below for finding the leader): + +```bash +curl -s -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/shardGroups | jq ' + .shardGroups[] | {id: (.id // 0), + shards: [.shards[] | {storeId, role}], + leaders: [.shards[] | select(.role=="Leader")] | length}' +``` + +Delete the next Store only when the replaced Pod is `Ready`, its Store id +shows a fresh `lastHeartBeat` in `/v1/stores`, and every group reports its +full shard count with exactly one `Leader`. + +Know what this does not prove: the shard list is PD's membership record, +not a statement that the Store caught up on the raft log. No endpoint on +current images reports restoration-complete. For a closer look, +port-forward the replaced Store and read its own view of a group: +`GET :8520/v1/partition/` returns the raft role, term, and +committed index that Store holds, and fails while the Store is down; +compare term and index with a peer Store rather than reading them alone. +The plural `GET :8520/v1/partitions` answers 500 on any Store that follows +a group on images built before +[apache/hugegraph#3232](https://github.com/apache/hugegraph/pull/3232) +(merged 2026-09-24); after it, it answers 200 on every Store, with `conf` +and `peers` null for followed groups. The per-group path works on both. + +Leave a margin after the membership check, keep `store.pdb.minAvailable` +at `replicas - 1` so an accidental second eviction is refused, and treat a +group that is short a shard or has no leader as a stop. A real +restoration-complete signal is upstream work, tracked in +[apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229). + +### 7 Disaster recovery + +What PD automates on current builds is narrow: a 60-second patrol only +marks Stores that stopped sending heartbeats as `Offline`. There is no +automatic re-replication; re-placing the replicas of a lost Store, +reconciling shard groups, and processing tombstoned Stores run only when a +partition patrol is triggered explicitly. + +The task endpoints execute locally on the PD that receives them, and a +follower answers with an empty success while doing nothing. +Port-forwarding the client Service selects an arbitrary PD, so identify +the leader first, then port-forward that Pod (two terminals: the forward +runs in the foreground): + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-pd-client 8620:8620 +# Read .data.pdLeader.raftUrl; its host names the leader Pod. +curl -su "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/members +# Stop the Service forward, then forward the leader Pod instead. +kubectl port-forward -n hugegraph pod/ 8620:8620 +# Reconcile shard groups and process tombstoned Stores. +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/patrolPartitions +# Spread Raft leaders, then partition data. +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balanceLeaders +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balancePartitions +``` + +Read `/v1/members` again after the tasks: if leadership moved mid-sequence, +the later tasks ran on a follower and did nothing. Wait at least 180 s +before rerunning `balanceLeaders` after a `balancePartitions` call: +`balancePartitions` sets a balance-shard flag for 180 s even when it moves +nothing, and `balanceLeaders` inside that window is refused. On images +built before +[apache/hugegraph#3233](https://github.com/apache/hugegraph/pull/3233) +(merged 2026-09-24) the refusal is a bare HTTP 500 whose reason appears +only in the PD log; after it, the reason comes back in the body as +`{"status":1001,"error":"balance shard is processing, please try later!"}`. + +Telling a real run from a no-op takes the PD leader's log, because the +responses do not: `patrolPartitions` answers the same empty success on the +leader and on a follower, whether or not it repaired anything (look for +`reallocShards`, `shardOffline` or `storeTurnoff` lines, or diff +`/v1/shardGroups` before and after), and `balancePartitions` answers `{}` +on the leader and an empty body on a follower. `balanceLeaders` is the one +call whose body carries the work. Distinguishable task responses are +upstream work, tracked in +[apache/hugegraph#3231](https://github.com/apache/hugegraph/issues/3231). + +Run `patrolPartitions` after replacing a Store that is not coming back, +`balancePartitions` once the cluster is stable again, and `balanceLeaders` +after restarts that skewed leader placement. + +**Losing a Store volume.** A Store rebuilt with an empty PVC recovers in +place on images carrying +[apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234) +(merged 2026-09-24); on every earlier image, including all published +release images, it does not. In both cases the replacement registers under +a new Store ID while its Pod name, DNS name, and raft address are +unchanged, so `/v1/stores` lists two IDs at one address. + +On post-#3234 images the retirement works: find the old ID in `/v1/stores` +(the row at the replaced Pod's address that is not the newly registered +one), `POST /v1/store/` with `{"storeState":"Tombstone"}` on the PD +leader, run `GET /v1/task/patrolPartitions`, and wait; verify every shard +group is back to full shard count with one leader, no group names the old +ID, and the replaced Store's own `:8520/v1/partition/` answers +200 for every group; then `DELETE /v1/store/` to erase the retired +record. Measured on a 3+3+3 install with images built from `master` at +`dbb6663a`: the replacement was Ready in 156 s, all 12 groups converged +onto the new ID 1 s after the Tombstone and patrol (the empty Store caught +up by raft snapshot install), and a continuous writer lost 0 acknowledged +writes. + +On images without #3234 the same retirement runs and does not repair the +groups: PD fires the configuration change, but the group leader sees the +address already in the group, so jraft has nothing to add and the group +keeps the old ID. Measured: 20 minutes and three patrols later, all 12 +groups still listed the retired ID and the replacement Store held no +partitions. Nothing in the health surface shows the failure; `/v1/stores`, +cluster state, Hubble, and Pod readiness all read healthy while every +shard group runs on two live replicas. The one check that shows it is the +replaced Store's own `:8520/v1/partition/`. + +So the default remains: replace a Store Pod, keep its PVC (the Store id +lives in the data path, and the Pod comes back under the same id). Treat +the empty-PVC procedure as recovery for post-#3234 images only; on a +released image a genuinely lost volume leaves the cluster degraded, and +expect to rebuild rather than to recover in place. + +Periodic leader balancing is tracked in +[apache/hugegraph#3135](https://github.com/apache/hugegraph/issues/3135); +disaster-recovery metrics in +[apache/hugegraph#3136](https://github.com/apache/hugegraph/issues/3136). + +### 8 Scaling + +PD and Store reserve the maximum StatefulSet ordinal in their resource +names, so scaling never renames a PersistentVolumeClaim or shifts a Pod +identity; both are capped at 99 replicas. Server scales freely through +`server.replicas` or `server.hpa` (with HPA on, the Deployment omits +`spec.replicas` so upgrades do not overwrite the autoscaler). + +A staged rollout cannot be written in a values file (the schema requires +one replica per component); stage it with `kubectl scale statefulset +hugegraph-store --replicas=0` and scale back up when ready. The Servers +wait, not ready, until Stores register, and the next `helm upgrade` +restores the full topology from values. + +Changing PD or Store replicas on a live release is not a values change: +raft and shard membership are persisted, and Pods alone do not reconfigure +them. The chart rejects both directions for PD and a shrink for Store by +reading the live StatefulSet, so fresh installs at any count are +unaffected. + +**PD, either direction.** The rendered peer list reaches raft only as its +bootstrap configuration, which an initialized group ignores; a 3-to-5 +upgrade leaves the voting configuration at three, and a 3-to-1 shrink +loses quorum outright. Membership changes go through the PD client API +(no REST route exposes them), which the chart cannot wrap. Change the +membership through PD, confirm it in `/v1/members`, scale the live +StatefulSet, then `helm upgrade` with the matching value; until that +sequence is verified on your own build, install the PD count you intend +to keep. + +**Store, shrinking.** Draining is a state transition, not a balance: +neither `patrolPartitions` nor `balancePartitions` retires a healthy +Store. Retire the leaving Stores the way Disaster Recovery retires a +replaced one: + +1. Check the remaining Stores still cover the persisted shard count + (`pd.partition.defaultShardCount`; empty derives 3 when + `store.replicas` is at least 3). +2. Map the ordinals the shrink will delete (the highest ones) to Store + ids through `/v1/stores`, matching on the Pod address. +3. `POST /v1/store/` with `{"storeState":"Tombstone"}` for each + leaving id. +4. Wait until `/v1/shardGroups` no longer lists those ids and every group + reports its full shard count with one leader. +5. Scale the live StatefulSet, then `helm upgrade` with the matching + value. + +Deleting the PVCs of the removed ordinals is separate and permanent; do it +only after step 4 reports the data moved. + +### 9 Running Hubble outside the cluster + +In-cluster Hubble (`hubble.enabled=true` plus a port-forward) is the +recommended path and is covered on the deployment page. Two paths exist +for a Hubble that must run outside the cluster. + +**Direct Server URL** (graph, schema, data, and Gremlin; no PD +discovery): leave in-chart Hubble off, expose Server +(`server.service.type` NodePort/LoadBalancer, or Ingress), and run a +standalone Hubble image with: + +```properties +pd.enabled=false +server.direct_url=https://: +``` + +Mount the file at `/hubble/conf/hugegraph-hubble.properties` (the image +workdir is `/hubble`). Use HTTPS or a trusted channel for +`server.direct_url`: login sends the Server credentials over that URL. + +**PD discovery** (the outside Hubble asks PD for the Server address): +in-cluster names such as `*.svc` are not reachable from outside, so the +chart provides two knobs. Set `server.advertiseUrl` to the absolute +`http(s)://` URL the outside Hubble will use after discovery; the chart +registers it with PD in place of the in-cluster Service URL. And expose +the PD client Service (`pd.service.type` NodePort/LoadBalancer), which +requires `pd.service.allowInsecureExposure=true` because PD gRPC has no +authentication; restrict who can reach it first (with the chart's +NetworkPolicy on, the render refuses the exposure until its callers are +listed in `networkPolicy.pd.extraIngress`). Then run standalone Hubble +with: + +```properties +pd.enabled=true +pd.peers=: +pd.server=: +``` + +Trade-off: when `server.advertiseUrl` is set, every Server replica +registers that same logical URL and PD returns it to every discovery +client, including an in-cluster Hubble. Leave it empty for the default +in-cluster path, where each Server Pod registers its own IP. + +Local quick test (cluster and Hubble on one machine): port-forward Server +`8080` and PD client `8620`/`8686`, set +`server.advertiseUrl=http://127.0.0.1:8080`, run standalone Hubble with +`--network host` and the properties above, then open Hubble on `8088`. + +### 10 When Gremlin fails with "Could not rebind" + +The error has two causes with different lifetimes. + +**A convergence window after creating a graph.** The Server that handles +`CreateGraph` waits for its own Gremlin binding before returning HTTP 200 +([#3138](https://github.com/apache/hugegraph/pull/3138)), so +create-then-query on the same Server is reliable. Other replicas converge +independently, and until they finish, a Gremlin query routed through the +load-balanced Service to a not-yet-converged replica can fail with a 400 +such as `Could not rebind [g]`. Retry with backoff (the window normally +closes in seconds), use sticky routing or a port-forward for +create-then-verify flows, or poll `/graphs` on each replica before +opening query traffic. Cluster-wide readiness is tracked in +[#3137](https://github.com/apache/hugegraph/issues/3137). + +**A Server Pod that started while PD was rolling.** Gremlin Server +instantiates the graph once at startup; if the PD client cannot connect +at that moment, the Pod passes readiness and serves REST while every +Gremlin request on it fails with `Could not rebind [graph]`, for the life +of the Pod. Its `hugegraph-server.log` names it: + +``` +Graph [DEFAULT-hugegraph] configured at [...] could not be instantiated and +will not be available in Gremlin Server +``` + +After any upgrade that rolled PD and Server together, check Gremlin on +each Server Pod (the image ships no curl, so port-forward each Pod): + +```bash +kubectl port-forward -n hugegraph pod/ 8080:8080 +curl -s --compressed -u "admin:${PASSWORD}" -H 'Content-Type: application/json' \ + -X POST http://127.0.0.1:8080/gremlin \ + -d '{"gremlin":"graph.traversal().V().limit(1).count()","aliases":{"graph":"DEFAULT-hugegraph"}}' +``` + +A healthy Pod answers with `result.data`; delete a Pod that answers +`Could not rebind`. Its replacement binds normally once PD is stable +(measured: 4 of 12 Server starts that overlapped a PD roll hit this, and +every deletion recovered). diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md index f0b2d28e38..b2d4feb228 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md @@ -25,6 +25,9 @@ A distributed HugeGraph cluster has a startup contract (no `init-store` on Serve metadata, Store waiting for a PD quorum, one PD REST secret shared by three readers). The chart encodes that contract so operators do not have to; the details are in the [chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#chart-details). +Day-2 work (NetworkPolicy details, disaster recovery, scaling, safe Store +rolls, running Hubble outside the cluster) is on the +[operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/). ```mermaid flowchart LR @@ -101,7 +104,7 @@ The chart ships three values files: |------|----------|--------------| | `values.yaml` | 3 PD + 3 Store + 3 Server | Default; preferred anti-affinity, auth on, Hubble off | | `values-single.yaml` | 1 + 1 + 1 | Single-node development and CI; smaller PVCs | -| `values-cluster.yaml` | 3 + 3 + 3 | Production starting point: JVM heap and resource settings, PD/Store PodDisruptionBudgets, `required` anti-affinity | +| `values-cluster.yaml` | 3 + 3 + 3 | Production starting point: JVM heap and resource settings, PD/Store PodDisruptionBudgets, `required` anti-affinity, NetworkPolicy on | ```bash helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace \ @@ -109,7 +112,10 @@ helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace ``` `values-cluster.yaml` is a starting point, not a capacity guarantee: recalculate resources for your graph size and -traffic. The full parameter reference (every component, probe, scheduling, and Secret knob) is kept in the +traffic. One of its numbers deserves a note: it requests 5Gi and limits at 8Gi of memory per Store, well above the +1Gi heap, because the Store's RocksDB caches live outside the JVM heap (a 4Gi limit OOM-killed Stores after about +1 GB of data). Scale both numbers with data size. The full parameter reference (every component, probe, +scheduling, and Secret knob) is kept in the [chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration). #### 3.4 Verify the install @@ -170,8 +176,12 @@ PD exposes two health endpoints, and the chart deliberately uses both: The chart puts PD **readiness** and the Store init-container wait on `/v1/ready`: a Store only starts once a majority of PD peers are quorum members, and a PD that lost its leader drops out of Service endpoints until a -leader is back. PD **startup and liveness** stay on `/v1/health` on purpose: a PD that merely lost its leader is -still a healthy Raft member, and restarting it would make the outage worse. +leader is back. PD **startup and liveness** derive from the replica count (`pd.livenessPath` overrides the +choice). With more than one PD they stay on `/v1/health` on purpose: a PD that merely lost its leader is still a +healthy Raft member, and restarting it would make the outage worse. A single PD is the exception and derives to +`/v1/ready`: it has no election to lose, and one that steps down for good, as after a failed Raft snapshot on a +full disk ([apache/hugegraph#3222](https://github.com/apache/hugegraph/issues/3222)), would answer `/v1/health` +forever while serving no writes; the kubelet restarts it instead. Server startup gets a matching budget: the image would normally kill a Server still starting after 120 seconds, so the chart derives `HG_SERVER_STARTUP_TIMEOUT_S` from the startup probe (450 seconds by default) and raises a lower @@ -210,9 +220,12 @@ defaults. Any upgrade that changes a Pod template rolls that workload once. Poin annotations first observe the install-created Secrets. Store is untouched. - **Store rolling updates advance on a listener check, not on shard recovery**, so the controller can replace the next Store while the previous one is still rejoining its shard groups. For a production image roll, set - `store.updateStrategy.type=OnDelete` and replace Store pods one at a time, confirming the previous Store shows - `Up` in PD before the next; PD restarts are one pod at a time either way, and `pd.updateStrategy.type=OnDelete` - gives the same manual control for maintenance windows. + `store.updateStrategy.type=OnDelete` and replace Store pods one at a time. `Up` in PD is not the between-pods + check: PD marks a Store `Up` at registration, before it restores partitions, and a stopped Store stays `Up` + until a 300 s keep-alive expires. Wait for the replaced Pod to be `Ready`, then confirm every shard group + reports its full shard count with one leader; the full procedure is on the + [operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/). PD restarts are one pod at a time + either way, and `pd.updateStrategy.type=OnDelete` gives the same manual control for maintenance windows. - **PVC sizes cannot be changed by upgrade**: Kubernetes forbids changing StatefulSet `volumeClaimTemplates`, so an upgrade with a new `storage.size` is rejected in full. The chart README documents the resize procedure for StorageClasses that support volume expansion. @@ -235,9 +248,13 @@ later install under the same release name comes back with the same credentials. ### 9 Limitations -- No NetworkPolicy resources yet. With the PD Raft IP allowlist disabled in-cluster (pod IPs change; the allowlist - resolves peers once at boot and then blocks them), in-cluster network access control is currently the operator's - responsibility. +- `networkPolicy.enabled` renders one NetworkPolicy per component that admits only the release's own traffic; it + is off by default and on in `values-cluster.yaml`. It matters because the chart disables the PD Raft IP + allowlist in-cluster (pod IPs change; the allowlist resolves peers once at boot and then blocks them), so the + policies are what restricts the raft and gRPC ports. They need a network plugin that enforces NetworkPolicy + (Calico, Cilium, kind v0.25 or later, k3s), and every outside client, the Ingress controller included, must be + listed in `networkPolicy..extraIngress` or the render fails. Details are on the + [operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/). - Image tags track `latest` until the next HugeGraph release publishes versioned images; pin tags or digests for anything long-lived. - After creating a graph, other Server replicas can lag for a short window before they serve queries for it, so a @@ -245,9 +262,11 @@ later install under the same release name comes back with the same credentials. backoff, or use sticky routing for create-then-query flows; cluster-wide graph readiness is tracked in [#3137](https://github.com/apache/hugegraph/issues/3137). - Store recovery is operator-triggered on current builds: re-replication after Store loss, leader balancing, and - partition rebalancing run only when called through PD's REST API. The chart README's - [Disaster Recovery](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#disaster-recovery) section is - the runbook. + partition rebalancing run only when called through PD's REST API. A Store whose volume is lost can be recovered + in place only on images carrying + [apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234) (merged 2026-09-24, in no release yet); + on earlier images, keep a Store's PVC when replacing its Pod. The Disaster Recovery section of the + [operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/) is the runbook. - No TLS termination inside the cluster, no backup tooling, no Operator, and no bundled monitoring stack. ### 10 Troubleshooting diff --git a/data/version_routes.json b/data/version_routes.json index ab687468dc..6207b4b3c7 100644 --- a/data/version_routes.json +++ b/data/version_routes.json @@ -722,6 +722,13 @@ "1.3": null, "1.0": null }, + "cn:quickstart/hugegraph/hugegraph-helm-operations": { + "latest": "cn/docs/quickstart/hugegraph/hugegraph-helm-operations/", + "1.7": null, + "1.5": null, + "1.3": null, + "1.0": null + }, "cn:quickstart/hugegraph/hugegraph-server": { "latest": "cn/docs/quickstart/hugegraph/hugegraph-server/", "1.7": "cn/docs/quickstart/hugegraph/hugegraph-server/", @@ -1436,6 +1443,13 @@ "1.3": null, "1.0": null }, + "en:quickstart/hugegraph/hugegraph-helm-operations": { + "latest": "docs/quickstart/hugegraph/hugegraph-helm-operations/", + "1.7": null, + "1.5": null, + "1.3": null, + "1.0": null + }, "en:quickstart/hugegraph/hugegraph-server": { "latest": "docs/quickstart/hugegraph/hugegraph-server/", "1.7": "docs/quickstart/hugegraph/hugegraph-server/", From 5e4cbd25c8445271f0d02b5d3d3c2fedbffdfbc5 Mon Sep 17 00:00:00 2001 From: Himanshu Verma Date: Fri, 2 Oct 2026 17:42:23 +0530 Subject: [PATCH 4/9] doc: sync the Helm pages with chart 0e754b98c Describe the chart as of apache/hugegraph#3218 at 0e754b98c, in EN and CN: - helm test now queries Gremlin on every Ready Server Pod through the headless Service, with the replica floor and the 150 s retry window - values-cluster.yaml sets Store updateStrategy OnDelete; the other presets keep RollingUpdate, and OnDelete is not a safety check by itself - rollingUpdate.maxUnavailable accepts only the integer 1 - Server startup budget: guaranteed time is (failureThreshold - 1) * periodSeconds, default 91 x 5, HG_SERVER_STARTUP_TIMEOUT_S 150 s - PD secret and admin password whitespace and backslash constraints, and the Hubble egress render guard with server.advertiseUrl - Limitations and troubleshooting for apache/hugegraph#3226 and #3228, #3229 referenced from the Upgrade section, and the Scaling link moved from the chart README to the operations page where the procedure lives --- .../hugegraph/hugegraph-helm-operations.md | 19 +++-- .../quickstart/hugegraph/hugegraph-helm.md | 59 ++++++++++---- .../hugegraph/hugegraph-helm-operations.md | 33 +++++--- .../quickstart/hugegraph/hugegraph-helm.md | 76 ++++++++++++++----- 4 files changed, 140 insertions(+), 47 deletions(-) diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 1b5d66dcb3..4fbf2ac0c7 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -107,7 +107,9 @@ pod 反亲和预设；Hubble 设计为单副本，没有这个配置项。设置 每个组件还可以走 53 端口解析 DNS。release 之外的任何来源都必须列入 `networkPolicy..extraIngress` 才被放行，包括 Ingress 控制器和 NodePort / LoadBalancer Service 的客户端。在 `extraIngress` 为空时暴露 PD、Server 或 Hubble，或设置 `server.advertiseUrl`，渲染会直接失败 -而不是打开端口。该检查只看 chart 自己创建的暴露；你自行添加的 Service、Gateway 路由或代理需要自己的条目。 +而不是打开端口。该检查只看 chart 自己创建的暴露；你自行添加的 Service、Gateway 路由或代理需要自己的条目。集群内 +`pd` 模式的 Hubble 与 `server.advertiseUrl` 同时使用时，在 `networkPolicy.hubble.extraEgress` 放行该对外地址之前， +渲染同样会失败：PD 会把这个 URL 交给 Hubble 用于发现，而 Hubble 的出向策略默认只能访问本 release 的 Pod。 两个出向事实需要规划。PD、Store、Server 除 DNS 外不访问 release 以外的任何地址，因此需要外呼的功能在策略开启时 不可用；默认唯一的外呼方是 Store 镜像，它每次启动都从 github.com 下载 `libjemalloc.so`。策略开启时该连接约两分 @@ -161,7 +163,9 @@ networkPolicy: ### 6 安全地滚动 Store 镜像 Store 的滚动更新以监听检查推进，而不是以分片恢复推进，控制器可能在上一个 Store 尚未重新加入分片组时就替换下一 -个。生产环境滚动镜像时，设置 `store.updateStrategy.type=OnDelete`，逐个删除 Store Pod，并在两次删除之间做检查。 +个。`values-cluster.yaml` 因此设置了 `store.updateStrategy.type=OnDelete`（其他生产 values 请自行设置）：升级只更新 +StatefulSet，不替换任何 Store Pod，由你逐个删除 Store Pod，并在两次删除之间做检查。`OnDelete` 只是停止自动推进； +删除 Pod 时不做下面的检查，风险相同。 PD 里的 `Up` 不是这个检查。PD 在注册时就把 Store 标为 `Up`，此时它还没恢复任何分区；已停止的 Store 也会在 keep-alive 记录过期前（当前镜像为 300 秒）一直保持 `Up` 并留在所有分片组里：窗口内删除又回来的 Pod 根本不会离开 @@ -317,7 +321,8 @@ pd.server=: ``` 取舍：设置 `server.advertiseUrl` 后，每个 Server 副本注册的都是同一个逻辑 URL，PD 会把它返回给所有发现客户 -端，包括集群内的 Hubble。留空则走默认的集群内路径，每个 Server Pod 注册自己的 IP。 +端，包括集群内的 Hubble（开启 chart 的 NetworkPolicy 时，它还需要 `networkPolicy.hubble.extraEgress`，见上文 +NetworkPolicy 一节）。留空则走默认的集群内路径，每个 Server Pod 注册自己的 IP。 本机快速验证（集群和 Hubble 在同一台机器）：port-forward Server `8080` 和 PD 客户端 `8620`/`8686`，设置 `server.advertiseUrl=http://127.0.0.1:8080`，用 `--network host` 和上面的 PD 配置运行独立 Hubble，再打开 @@ -335,15 +340,17 @@ port-forward，或先在每个副本上轮询 `/graphs` 再放开查询流量。 [#3137](https://github.com/apache/hugegraph/issues/3137) 跟踪。 **在 PD 滚动期间启动的 Server Pod。** Gremlin Server 只在启动时实例化一次图；那一刻 PD 客户端连不上的话， -这个 Pod 会终身通过就绪探测、正常提供 REST，而每个发到它的 Gremlin 请求都报 `Could not rebind [graph]`。 -它的 `hugegraph-server.log` 会写明： +这个 Pod 会终身通过就绪探测、正常提供 REST，而每个发到它的 Gremlin 请求都报 `Could not rebind [graph]` +（[apache/hugegraph#3228](https://github.com/apache/hugegraph/issues/3228)）。它的 `hugegraph-server.log` 会写明： ``` Graph [DEFAULT-hugegraph] configured at [...] could not be instantiated and will not be available in Gremlin Server ``` -任何同时滚动了 PD 和 Server 的升级之后，逐个检查 Server Pod 的 Gremlin（镜像不带 curl，逐个 port-forward）： +任何同时滚动了 PD 和 Server 的升级之后，运行 `helm test`：它通过 headless Service `hugegraph-server-headless` 向每个 +Ready 的 Server Pod 发送下面这条绑定图的 Gremlin 查询，最多重试 150 秒，遇到处于这种状态的 Pod 会以 +`Gremlin failed on ` 失败。手工检查单个 Pod 时，对它做 port-forward（镜像不带 curl）： ```bash kubectl port-forward -n hugegraph pod/ 8080:8080 diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md index 15c001fecc..ff1dc4b203 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md @@ -95,7 +95,7 @@ chart 附带三个 values 文件： |------|------|----------| | `values.yaml` | 3 PD + 3 Store + 3 Server | 默认；preferred 反亲和，认证开启，Hubble 关闭 | | `values-single.yaml` | 1 + 1 + 1 | 单节点开发与 CI；PVC 更小 | -| `values-cluster.yaml` | 3 + 3 + 3 | 生产起点：JVM 堆与资源设置、PD/Store PodDisruptionBudget、`required` 反亲和、NetworkPolicy 开启 | +| `values-cluster.yaml` | 3 + 3 + 3 | 生产起点：JVM 堆与资源设置、PD/Store PodDisruptionBudget、`required` 反亲和、NetworkPolicy 开启、Store 使用 `OnDelete` 更新 | ```bash helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace \ @@ -113,6 +113,13 @@ Store OOM 杀掉了）。两个数字要随数据量一起放大。完整参数 helm test hugegraph --namespace hugegraph ``` +测试先经 Server Service 调用 `/versions` 和 `/graphs`，再解析 headless Service `hugegraph-server-headless`，向它列出的 +每个 Server Pod 发送一条带认证、绑定 `DEFAULT-hugegraph` 图的 Gremlin 查询，并要求至少有 `server.replicas` 个 Pod +（开启 HPA 时为 `server.hpa.minReplicas`）。headless Service 只列出 Ready 的 Pod，也就是 Server Service 实际转发流量的 +那些 Pod，未 Ready 的 Pod 不会被查询。失败的 Pod 每 5 秒重试一次，最多 150 秒，短暂的 PD 选举因此不会被误判为 Pod +损坏；超时后测试失败，并打印每个失败 Pod 的 IP 和 HTTP 状态码。能通过就绪探测、能提供 REST、但每个 Gremlin 调用都 +失败的 Server，正是靠这一步发现的（见"限制"）。HPA 规模较大时，`helm test --timeout` 可能需要高于默认的 5 分钟。 + 读取自动生成的 admin 密码并调用 API： ```bash @@ -136,7 +143,8 @@ curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions | `-pd-auth` | `secret-key` | PD REST 认证，由 PD、Server、Hubble 读取 | `pd.auth.existingSecret` | 要自行管理凭据，请在安装前创建 Secret 并把对应的 `existingSecret` 指向它；chart 不会改动任何不是它创建的 Secret。取值 -约束：admin 密码不能包含换行、回车或反斜杠；JWT 密钥至少 32 字节；PD 密钥必须是可打印 ASCII。非法值会在渲染时或启动 +约束：admin 密码不能包含换行、回车或反斜杠，也不能以空白开头或结尾；JWT 密钥至少 32 字节；PD 密钥必须是不含反斜杠、 +首尾没有空格的可打印 ASCII。非法值会在渲染时或启动 包装脚本中被拒绝，不会被悄悄截断。 chart 管理的 Secret 在卸载时保留，同名 release 再次安装会复用它们。 @@ -164,9 +172,12 @@ chart 把 PD 的**就绪探测**和 Store init 容器的等待放在 `/v1/ready` [apache/hugegraph#3222](https://github.com/apache/hugegraph/issues/3222)）会一直用 `/v1/health` 返回 200 却不再 服务写入；改用 `/v1/ready` 让 kubelet 把它重启。 -Server 的启动获得相同的预算：镜像默认会在 120 秒后杀掉仍在启动的 Server，chart 因此从启动探针推导 -`HG_SERVER_STARTUP_TIMEOUT_S`（默认 450 秒），配置的探针预算低于该下限时会被抬高。如果存储层启动更慢，调大 -`server.probes.startup`，镜像的预算会跟着变。 +Server 启动获得至少 450 秒的预算，足够覆盖镜像入口脚本在启动命令之前执行的 300 秒存储等待和进程启动。chart 按 +`(failureThreshold - 1) * periodSeconds` 计算有保证的探测时间，因为 kubelet 可能在容器启动后立刻执行第一次探测； +配置的 `failureThreshold` 低于该下限时会被抬高（默认 5 秒周期下为 91，也就是默认值）。镜像默认会在 120 秒后杀掉 +仍在启动的 Server，chart 因此把 `HG_SERVER_STARTUP_TIMEOUT_S` 设为有保证的探测时间减去 300 秒存储等待：默认 150 秒， +不低于镜像的 120 秒。这样启动命令与探针同时放弃。如果存储层启动更慢，调大 `server.probes.startup`，超时会跟着变； +该变量由 chart 管理，请修改探针，不要在 `server.extraEnv` 里设置它。 ### 6 启用 Hubble UI @@ -195,20 +206,27 @@ helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values 对应工作负载滚动一次。需要提前规划的几点： - **全新安装后的第一次升级会让 PD、Server、Hubble 各滚动一次**，因为跟踪 Secret 的注解第一次观察到安装时创建的 - Secret。Store 不受影响。 + Secret。Store 不受影响。在 PD 滚动期间重启的 Server 可能丢失 Gremlin 绑定（见"限制"），因此任何同时滚动了 PD 和 + Server 的升级之后，请运行 `helm test`。 - **Store 的滚动更新以监听检查推进，而不是以分片恢复推进**，因此控制器可能在上一个 Store 尚未重新加入分片组时就替换 - 下一个。生产环境滚动镜像时，设置 `store.updateStrategy.type=OnDelete`，逐个替换 Store Pod。PD 里的 `Up` 不是两次 - 替换之间的检查：PD 在注册时就把 Store 标为 `Up`，此时分区尚未恢复，已停止的 Store 也要等 300 秒 keep-alive 过期 - 才离开 `Up`。应等被替换的 Pod 变为 `Ready`，再确认每个分片组都报告完整分片数和一个 leader；完整流程见 - [运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/)。PD 本来就逐个重启， - `pd.updateStrategy.type=OnDelete` 为维护窗口提供同样的手工控制。 + 下一个。`values-cluster.yaml` 因此设置了 `store.updateStrategy.type=OnDelete`；`values.yaml` 和 `values-single.yaml` + 仍为 `RollingUpdate`，其他生产 values 请自行设置。`OnDelete` 下升级只更新 StatefulSet，不替换任何 Store Pod，由你逐个 + 删除 Store Pod。`OnDelete` 只是停止自动推进：删除 Pod 时不在两次删除之间做检查，风险相同。PD 里的 `Up` 不是这个检查： + PD 在注册时就把 Store 标为 `Up`，此时分区尚未恢复，已停止的 Store 也要等 300 秒 keep-alive 过期才离开 `Up` + （真正的恢复完成信号在 [apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229) 跟踪）。应等被替换的 + Pod 变为 `Ready`，再确认每个分片组都报告完整分片数和一个 leader；完整流程见 + [运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/#6-安全地滚动-store-镜像)。 +- **控制器每次最多滚动一个 PD 或 Store Pod。** chart 不设置 `updateStrategy.rollingUpdate.maxUnavailable`，且该字段只接受 + 整数 `1`：更大的数字或百分比会导致渲染失败，因为它会让控制器同时停掉三成员 Raft 组或分片中的两个成员，而 + PodDisruptionBudget 不约束控制器发起的滚动。PD 的维护窗口可用 `pd.updateStrategy.type=OnDelete` 获得与 Store 相同的 + 手工控制。 - **升级不能修改 PVC 大小**：Kubernetes 禁止修改 StatefulSet 的 `volumeClaimTemplates`，带新 `storage.size` 的升级会 被整体拒绝。chart README 记录了支持卷扩容的 StorageClass 上的扩容步骤。 Server 的扩缩容是普通的 values 变更（`server.replicas` 或 `server.hpa`）。**缩容 PD 或 Store 不是**：Raft 与分片成员 关系是持久化的，删除 Pod 不会重新配置它们，PD 从 3 缩到 1 会永久失去多数派。chart 会拒绝副本数低于线上 StatefulSet -的升级；先迁移再缩容的手工步骤见 chart README 的 -[Scaling](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#scaling)。 +的升级；先迁移再缩容的手工步骤见 +[运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/#8-伸缩)。 ### 8 卸载 @@ -231,6 +249,17 @@ helm uninstall hugegraph --namespace hugegraph - 创建图之后，其他 Server 副本在短暂窗口内可能尚未收敛，路由到这类副本的查询可能报 `Could not rebind [g]` 之类的 错误。请带退避重试，或对"创建后立即查询"的流程使用会话粘滞路由；集群级图就绪在 [#3137](https://github.com/apache/hugegraph/issues/3137) 跟踪。 +- 在 PD 不可达时（例如 PD 滚动期间）启动的 Server，可能在整个 Pod 生命周期内都没有 Gremlin 绑定：它通过就绪探测、 + 正常提供 REST，而每个发到它的 Gremlin 请求都报 `Could not rebind [graph]` + （[apache/hugegraph#3228](https://github.com/apache/hugegraph/issues/3228)）。就绪探测调用的是 `/versions`，看不到 + 这种状态。`helm test` 会在每个 Ready 的 Server Pod 上查询 Gremlin 并指出处于这种状态的 Pod；删除该 Pod，PD 稳定后 + 替换者会正常绑定。见 + [Gremlin 报 "Could not rebind" 时](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/#10-gremlin-报-could-not-rebind-时)。 +- 启动时打不开自己 RocksDB 存储的 PD（例如上一个进程仍持有存储的 `LOCK` 文件），只记录一次 `Failed to open RocksDB`， + 之后继续运行，既不重试也不退出（[apache/hugegraph#3226](https://github.com/apache/hugegraph/issues/3226)）。 + `/v1/ready` 返回 503 和 `STATE_UNINITIALIZED`，因此 Pod 不会 Ready，并退出 Service 端点。但多副本 PD 的启动和存活 + 探测使用仍返回 200 的 `/v1/health`，kubelet 永远不会重启它，它会一直卡住，直到运维人员删除该 Pod。不要改为删除 + `LOCK` 文件：它们防的是可能仍在运行的另一个进程。单副本 PD 的存活探测使用 `/v1/ready`，kubelet 会重启它。 - 当前版本的 Store 恢复由运维人员触发：Store 丢失后的副本重建、leader 均衡、分区再均衡都只在调用 PD 的 REST API 时 执行。丢失了卷的 Store 只有在携带 [apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234)（2026-09-24 合入，尚未进入任何发布版本） @@ -246,6 +275,8 @@ helm uninstall hugegraph --namespace hugegraph | PVC 停在 `Pending` | 没有默认 StorageClass，或供给器故障：`kubectl get sc` | | 多节点上 Pod 被 OOM 杀掉或反复重启 | 未设置 resources，JVM 按节点内存取堆：用 `values-cluster.yaml` | | 建图后立刻查询失败 | 副本收敛窗口：见上文"限制" | +| `helm test` 打印 `Gremlin failed on ` | 该 Server Pod 丢失了 Gremlin 绑定：删除它（见上文"限制"） | +| 某个 PD Pod 一直不 Ready，其 `/v1/ready` 报 `STATE_UNINITIALIZED` | 它的 RocksDB 存储没有打开：在日志中查找 `Failed to open RocksDB`，然后删除该 Pod（见上文"限制"） | -每种情况的完整排查步骤见 +其中多数情况的完整排查步骤见 [chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#troubleshooting)。 diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 85d79fa566..33501e3ee0 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -134,7 +134,11 @@ including the Ingress controller and the clients of a NodePort or LoadBalancer Service. Exposing PD, Server or Hubble that way, or setting `server.advertiseUrl`, with an empty `extraIngress` fails the render instead of opening the port. The check sees only exposure the chart creates; a -Service, Gateway route or proxy you add yourself needs its own entry. +Service, Gateway route or proxy you add yourself needs its own entry. An +in-cluster Hubble in `pd` mode combined with `server.advertiseUrl` also +fails the render until `networkPolicy.hubble.extraEgress` admits the +advertised destination: PD hands Hubble that URL for discovery, and +Hubble's egress policy otherwise reaches only the release's own Pods. Two egress facts to plan around. PD, Store and Server reach nothing outside the release except DNS, so features that call out do not work with the @@ -200,9 +204,12 @@ networkPolicy: Store rolling updates advance on a listener check, not on shard recovery, so the controller can replace the next Store while the previous one is -still rejoining its shard groups. For a production image roll, set -`store.updateStrategy.type=OnDelete` and delete Store Pods one at a time, -checking between deletions. +still rejoining its shard groups. `values-cluster.yaml` therefore sets +`store.updateStrategy.type=OnDelete` (set it yourself on any other +production values): an upgrade then updates the StatefulSet without +replacing any Store Pod, and you delete Store Pods one at a time, checking +between deletions. `OnDelete` only stops automatic advancement; deleting +Pods without the checks below carries the same risk. `Up` in PD is not that check. PD marks a Store `Up` at registration, before the Store has restored any partition, and a stopped Store stays `Up` in @@ -432,8 +439,10 @@ pd.server=: Trade-off: when `server.advertiseUrl` is set, every Server replica registers that same logical URL and PD returns it to every discovery -client, including an in-cluster Hubble. Leave it empty for the default -in-cluster path, where each Server Pod registers its own IP. +client, including an in-cluster Hubble (which, with the chart's +NetworkPolicy on, then needs `networkPolicy.hubble.extraEgress`; see +NetworkPolicy above). Leave it empty for the default in-cluster path, +where each Server Pod registers its own IP. Local quick test (cluster and Hubble on one machine): port-forward Server `8080` and PD client `8620`/`8686`, set @@ -460,15 +469,21 @@ opening query traffic. Cluster-wide readiness is tracked in instantiates the graph once at startup; if the PD client cannot connect at that moment, the Pod passes readiness and serves REST while every Gremlin request on it fails with `Could not rebind [graph]`, for the life -of the Pod. Its `hugegraph-server.log` names it: +of the Pod +([apache/hugegraph#3228](https://github.com/apache/hugegraph/issues/3228)). +Its `hugegraph-server.log` names it: ``` Graph [DEFAULT-hugegraph] configured at [...] could not be instantiated and will not be available in Gremlin Server ``` -After any upgrade that rolled PD and Server together, check Gremlin on -each Server Pod (the image ships no curl, so port-forward each Pod): +After any upgrade that rolled PD and Server together, run `helm test`: +it sends the graph-bound Gremlin query below to every Ready Server Pod +through the headless Service `hugegraph-server-headless`, retries for up +to 150 seconds, and fails with `Gremlin failed on ` for a Pod in +this state. To check one Pod by hand, port-forward it (the image ships no +curl): ```bash kubectl port-forward -n hugegraph pod/ 8080:8080 diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md index b2d4feb228..6cf241527b 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md @@ -104,7 +104,7 @@ The chart ships three values files: |------|----------|--------------| | `values.yaml` | 3 PD + 3 Store + 3 Server | Default; preferred anti-affinity, auth on, Hubble off | | `values-single.yaml` | 1 + 1 + 1 | Single-node development and CI; smaller PVCs | -| `values-cluster.yaml` | 3 + 3 + 3 | Production starting point: JVM heap and resource settings, PD/Store PodDisruptionBudgets, `required` anti-affinity, NetworkPolicy on | +| `values-cluster.yaml` | 3 + 3 + 3 | Production starting point: JVM heap and resource settings, PD/Store PodDisruptionBudgets, `required` anti-affinity, NetworkPolicy on, Store `OnDelete` updates | ```bash helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace \ @@ -124,6 +124,15 @@ scheduling, and Secret knob) is kept in the helm test hugegraph --namespace hugegraph ``` +The test first calls `/versions` and `/graphs` through the Server Service. It then resolves the headless Service +`hugegraph-server-headless` and sends an authenticated Gremlin query bound to the `DEFAULT-hugegraph` graph to +each Server Pod it lists, and it requires at least `server.replicas` Pods (`server.hpa.minReplicas` with HPA on). +The headless Service lists only Ready Pods, the same Pods the Server Service routes to, so a Pod that is not Ready +is not queried. A failing Pod is retried every 5 seconds for up to 150 seconds, so a short PD election is not +reported as a broken Pod; after that the test fails and prints each failing Pod IP with its HTTP status. This is +the check that catches a Server that passes readiness and serves REST while every Gremlin call on it fails (see +Limitations). Large HPA fleets may need `helm test --timeout` above the 5-minute default. + Read the generated admin password and call the API: ```bash @@ -149,8 +158,9 @@ an `existingSecret` you created wins, then an inline value, then a random value To manage a credential yourself, create the Secret before installing and point the matching `existingSecret` value at it; the chart never modifies a Secret it did not create. Value constraints: the admin password must not contain -newlines, carriage returns, or backslashes; the JWT key must be at least 32 bytes; the PD secret must be printable -ASCII. Invalid values are rejected at render time or by the startup wrapper, not silently truncated. +newlines, carriage returns, or backslashes, or start or end with whitespace; the JWT key must be at least 32 bytes; +the PD secret must be printable ASCII with no backslashes and no leading or trailing space. Invalid values are +rejected at render time or by the startup wrapper, not silently truncated. Chart-managed Secrets are kept on uninstall and reused by a later install under the same release name. @@ -183,10 +193,15 @@ healthy Raft member, and restarting it would make the outage worse. A single PD full disk ([apache/hugegraph#3222](https://github.com/apache/hugegraph/issues/3222)), would answer `/v1/health` forever while serving no writes; the kubelet restarts it instead. -Server startup gets a matching budget: the image would normally kill a Server still starting after 120 seconds, so -the chart derives `HG_SERVER_STARTUP_TIMEOUT_S` from the startup probe (450 seconds by default) and raises a lower -configured probe budget to that floor. Raise `server.probes.startup` if your storage takes longer to come up, and -the image budget follows. +Server startup gets a budget of at least 450 seconds, enough for the 300-second storage wait the image entrypoint +runs before the start command, plus process startup. The chart counts the guaranteed probe time as +`(failureThreshold - 1) * periodSeconds`, because the kubelet may run the first probe as soon as the container +starts, and raises a lower configured `failureThreshold` to that floor (91 at the default 5-second period, the +value `values.yaml` ships). The image would kill a Server still starting after 120 seconds, so the chart also sets +`HG_SERVER_STARTUP_TIMEOUT_S` to the guaranteed probe time minus the 300-second storage wait: 150 seconds by +default, never less than the image's 120. The start command and the probe then give up together. Raise +`server.probes.startup` if your storage takes longer to come up, and the timeout follows; the variable is +chart-managed, so change the probe rather than setting it in `server.extraEnv`. ### 6 Enable the Hubble UI @@ -217,15 +232,25 @@ helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values defaults. Any upgrade that changes a Pod template rolls that workload once. Points worth planning around: - **The first upgrade after a fresh install rolls PD, Server, and Hubble once**, when the Secret-tracking - annotations first observe the install-created Secrets. Store is untouched. + annotations first observe the install-created Secrets. Store is untouched. A Server that restarts while PD is + rolling can lose its Gremlin binding (see Limitations), so run `helm test` after any upgrade that rolled PD and + Server together. - **Store rolling updates advance on a listener check, not on shard recovery**, so the controller can replace the - next Store while the previous one is still rejoining its shard groups. For a production image roll, set - `store.updateStrategy.type=OnDelete` and replace Store pods one at a time. `Up` in PD is not the between-pods - check: PD marks a Store `Up` at registration, before it restores partitions, and a stopped Store stays `Up` - until a 300 s keep-alive expires. Wait for the replaced Pod to be `Ready`, then confirm every shard group - reports its full shard count with one leader; the full procedure is on the - [operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/). PD restarts are one pod at a time - either way, and `pd.updateStrategy.type=OnDelete` gives the same manual control for maintenance windows. + next Store while the previous one is still rejoining its shard groups. `values-cluster.yaml` therefore sets + `store.updateStrategy.type=OnDelete`; `values.yaml` and `values-single.yaml` keep `RollingUpdate`, so set it + yourself on any other production values. Under `OnDelete` an upgrade updates the StatefulSet but replaces no + Store Pod, and you delete Store Pods one at a time. `OnDelete` only stops automatic advancement: deleting Pods + without checking between them carries the same risk. `Up` in PD is not that check: PD marks a Store `Up` at + registration, before it restores partitions, and a stopped Store stays `Up` until a 300 s keep-alive expires + ([apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229) tracks a real restoration signal). + Wait for the replaced Pod to be `Ready`, then confirm every shard group reports its full shard count with one + leader; the full procedure is on the + [operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/#6-rolling-store-images-safely). +- **The controller never rolls more than one PD or Store Pod at a time.** The chart leaves + `updateStrategy.rollingUpdate.maxUnavailable` unset and accepts only the integer `1` there: a larger number or + a percentage fails the render, because it would let the controller take down two members of a three-member + Raft group or shard at once, and a PodDisruptionBudget does not limit controller rollouts. For PD maintenance + windows, `pd.updateStrategy.type=OnDelete` gives the same manual control as for Store. - **PVC sizes cannot be changed by upgrade**: Kubernetes forbids changing StatefulSet `volumeClaimTemplates`, so an upgrade with a new `storage.size` is rejected in full. The chart README documents the resize procedure for StorageClasses that support volume expansion. @@ -233,8 +258,8 @@ defaults. Any upgrade that changes a Pod template rolls that workload once. Poin Scaling Server up and down is a values change (`server.replicas`, or `server.hpa`). Scaling **PD or Store down is not**: Raft and shard membership are persisted, deleting pods does not reconfigure them, and a 3-to-1 PD shrink permanently loses quorum. The chart rejects an upgrade whose replica count is below the live StatefulSet; the -manual drain-then-scale procedure is in the chart README under -[Scaling](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#scaling). +manual drain-then-scale procedure is on the +[operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/#8-scaling). ### 8 Uninstall @@ -261,6 +286,19 @@ later install under the same release name comes back with the same credentials. query routed to a not-yet-converged replica can fail with an error such as `Could not rebind [g]`. Retry with backoff, or use sticky routing for create-then-query flows; cluster-wide graph readiness is tracked in [#3137](https://github.com/apache/hugegraph/issues/3137). +- A Server that starts while PD is unreachable, as during a PD roll, can come up without its Gremlin binding for + the life of the Pod: it passes readiness and serves REST while every Gremlin request on it fails with + `Could not rebind [graph]` ([apache/hugegraph#3228](https://github.com/apache/hugegraph/issues/3228)). The + readiness probe calls `/versions` and cannot see this. `helm test` queries Gremlin on every Ready Server Pod and + names a Pod in this state; delete that Pod, and its replacement binds once PD is stable. See + [When Gremlin fails with "Could not rebind"](/docs/quickstart/hugegraph/hugegraph-helm-operations/#10-when-gremlin-fails-with-could-not-rebind). +- A PD that fails to open its RocksDB store at startup, as when the previous process still holds the store's + `LOCK` file, logs `Failed to open RocksDB` once and keeps running without retrying or exiting + ([apache/hugegraph#3226](https://github.com/apache/hugegraph/issues/3226)). `/v1/ready` answers 503 with + `STATE_UNINITIALIZED`, so the Pod is not Ready and leaves the Service endpoints. With more than one PD, though, + startup and liveness probe `/v1/health`, which still answers 200, so the kubelet never restarts it and it stays + stuck until an operator deletes the Pod. Do not delete the `LOCK` files instead: they guard against a second + process that may still be running. A single PD probes `/v1/ready` for liveness, so the kubelet restarts it. - Store recovery is operator-triggered on current builds: re-replication after Store loss, leader balancing, and partition rebalancing run only when called through PD's REST API. A Store whose volume is lost can be recovered in place only on images carrying @@ -277,6 +315,8 @@ later install under the same release name comes back with the same credentials. | PVCs stay `Pending` | No default StorageClass, or the provisioner is down: `kubectl get sc` | | Pods OOM killed or restarting on multi-node | No resources set, JVM heaps sized to node memory: use `values-cluster.yaml` | | Query fails right after creating a graph | Replica convergence window: see Limitations above | +| `helm test` prints `Gremlin failed on ` | That Server Pod lost its Gremlin binding: delete it (see Limitations above) | +| A PD Pod stays not Ready and its `/v1/ready` reports `STATE_UNINITIALIZED` | Its RocksDB store did not open: look for `Failed to open RocksDB` in its log, then delete the Pod (see Limitations above) | -Longer walkthroughs for each case are in the +Longer walkthroughs for most of these cases are in the [chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#troubleshooting). From b14b53dd1fc13ac1824271a261e7690f01179dc3 Mon Sep 17 00:00:00 2001 From: Himanshu Verma Date: Fri, 2 Oct 2026 17:47:46 +0530 Subject: [PATCH 5/9] doc: list the Helm pages in the docs navigation Since the OINK migration the sidebar comes from data/docs_nav.json, and the two Helm pages were not in it, so no navigation led to them. Add both under the HugeGraph components entry after PD, regenerate the derived maps with materialize_docs_navigation, and update the pinned navigation stats: latest gains two pages, and each archived version, which lacks the pages, counts two more removed entries. --- data/docs_nav.json | 30 +++++++++++++++++++++++++++++- scripts/versioning.py | 12 ++++++------ 2 files changed, 35 insertions(+), 7 deletions(-) diff --git a/data/docs_nav.json b/data/docs_nav.json index d4b4125433..4a5503ecee 100644 --- a/data/docs_nav.json +++ b/data/docs_nav.json @@ -34,6 +34,12 @@ }, { "page": "/docs/quickstart/hugegraph/hugegraph-pd" + }, + { + "page": "/docs/quickstart/hugegraph/hugegraph-helm" + }, + { + "page": "/docs/quickstart/hugegraph/hugegraph-helm-operations" } ] }, @@ -406,6 +412,16 @@ "page": "/docs/quickstart/hugegraph/hugegraph-pd", "url": "/docs/quickstart/hugegraph/hugegraph-pd/", "children": [] + }, + { + "page": "/docs/quickstart/hugegraph/hugegraph-helm", + "url": "/docs/quickstart/hugegraph/hugegraph-helm/", + "children": [] + }, + { + "page": "/docs/quickstart/hugegraph/hugegraph-helm-operations", + "url": "/docs/quickstart/hugegraph/hugegraph-helm-operations/", + "children": [] } ] }, @@ -906,6 +922,16 @@ "/docs/quickstart/hugegraph/", "/docs/quickstart/hugegraph/hugegraph-pd/" ], + "/docs/quickstart/hugegraph/hugegraph-helm/": [ + "@group/components", + "/docs/quickstart/hugegraph/", + "/docs/quickstart/hugegraph/hugegraph-helm/" + ], + "/docs/quickstart/hugegraph/hugegraph-helm-operations/": [ + "@group/components", + "/docs/quickstart/hugegraph/", + "/docs/quickstart/hugegraph/hugegraph-helm-operations/" + ], "/docs/quickstart/toolchain/": [ "@group/components", "/docs/quickstart/toolchain/" @@ -1347,7 +1373,9 @@ ], "/docs/quickstart/hugegraph/": [ "/docs/quickstart/hugegraph/hugegraph-hstore", - "/docs/quickstart/hugegraph/hugegraph-pd" + "/docs/quickstart/hugegraph/hugegraph-pd", + "/docs/quickstart/hugegraph/hugegraph-helm", + "/docs/quickstart/hugegraph/hugegraph-helm-operations" ], "/docs/quickstart/toolchain/": [ "/docs/quickstart/toolchain/visualization", diff --git a/scripts/versioning.py b/scripts/versioning.py index e812d8c0fc..622a3695bd 100644 --- a/scripts/versioning.py +++ b/scripts/versioning.py @@ -130,36 +130,36 @@ DOCS_NAV_EXPECTED_STATS = { "latest": { "groups": 5, - "pages": 92, + "pages": 94, "removed": 4, "scopedLinks": 0, - "treeSha256": "15de4de385857f877b46083de8e65d2887769d811687c4fe7ade3069401fc929", + "treeSha256": "c610cc9128f26a85e0f60843f9e5255f95daa8ba7d1d3a62d19a07a799a3e520", }, "1.7": { "groups": 5, "pages": 85, - "removed": 5, + "removed": 7, "scopedLinks": 10, "treeSha256": "c87538b82b0e3506eef59417411686ddc188e0f33d0e3a1cda687de8d6f88747", }, "1.5": { "groups": 5, "pages": 77, - "removed": 13, + "removed": 15, "scopedLinks": 10, "treeSha256": "ac29ab8f0e7020496a3a1afe480687043d751e32c93ffb848e2210e3500cd483", }, "1.3": { "groups": 5, "pages": 68, - "removed": 22, + "removed": 24, "scopedLinks": 10, "treeSha256": "e8c9b0a64dac6dd3e47827ec5072697eb7d3ed79659e0b5c4a9c5df8e1408d38", }, "1.0": { "groups": 5, "pages": 59, - "removed": 21, + "removed": 23, "scopedLinks": 10, "treeSha256": "bd45db511d9b6728e1523bd3c5ebd9d18912ee85aeabf79c77bcca7e511f977a", }, From ed90090e54cf2779677210875d967e476be2d3d3 Mon Sep 17 00:00:00 2001 From: Himanshu Verma Date: Fri, 2 Oct 2026 17:59:03 +0530 Subject: [PATCH 6/9] doc: describe the Helm limitations by behavior State each limitation as the behavior a user sees, with the issue link kept as a reference. The Store roll now says plainly that no endpoint reports restore completion. The uninitialized-PD entry separates what was observed outside Kubernetes from what the chart's probes do, and no longer offers an untested recovery step. helm test prints the failing Pod's IP, not its name. --- .../quickstart/hugegraph/hugegraph-helm.md | 21 ++++++++------- .../quickstart/hugegraph/hugegraph-helm.md | 27 ++++++++++--------- 2 files changed, 26 insertions(+), 22 deletions(-) diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md index ff1dc4b203..e0b78ec48a 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md @@ -212,9 +212,9 @@ helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values 下一个。`values-cluster.yaml` 因此设置了 `store.updateStrategy.type=OnDelete`；`values.yaml` 和 `values-single.yaml` 仍为 `RollingUpdate`，其他生产 values 请自行设置。`OnDelete` 下升级只更新 StatefulSet，不替换任何 Store Pod，由你逐个 删除 Store Pod。`OnDelete` 只是停止自动推进：删除 Pod 时不在两次删除之间做检查，风险相同。PD 里的 `Up` 不是这个检查： - PD 在注册时就把 Store 标为 `Up`，此时分区尚未恢复，已停止的 Store 也要等 300 秒 keep-alive 过期才离开 `Up` - （真正的恢复完成信号在 [apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229) 跟踪）。应等被替换的 - Pod 变为 `Ready`，再确认每个分片组都报告完整分片数和一个 leader；完整流程见 + PD 在注册时就把 Store 标为 `Up`，此时分区尚未恢复，已停止的 Store 也要等 300 秒 keep-alive 过期才离开 `Up`。当前 + 镜像上没有任何端点报告 Store 已完成分区恢复（[apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229)）， + 因此只能间接检查。应等被替换的 Pod 变为 `Ready`，再确认每个分片组都报告完整分片数和一个 leader；完整流程见 [运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/#6-安全地滚动-store-镜像)。 - **控制器每次最多滚动一个 PD 或 Store Pod。** chart 不设置 `updateStrategy.rollingUpdate.maxUnavailable`，且该字段只接受 整数 `1`：更大的数字或百分比会导致渲染失败，因为它会让控制器同时停掉三成员 Raft 组或分片中的两个成员，而 @@ -252,14 +252,15 @@ helm uninstall hugegraph --namespace hugegraph - 在 PD 不可达时（例如 PD 滚动期间）启动的 Server，可能在整个 Pod 生命周期内都没有 Gremlin 绑定：它通过就绪探测、 正常提供 REST，而每个发到它的 Gremlin 请求都报 `Could not rebind [graph]` （[apache/hugegraph#3228](https://github.com/apache/hugegraph/issues/3228)）。就绪探测调用的是 `/versions`，看不到 - 这种状态。`helm test` 会在每个 Ready 的 Server Pod 上查询 Gremlin 并指出处于这种状态的 Pod；删除该 Pod，PD 稳定后 + 这种状态。`helm test` 会在每个 Ready 的 Server Pod 上查询 Gremlin，并打印处于这种状态的 Pod 的 IP；删除该 Pod，PD 稳定后 替换者会正常绑定。见 [Gremlin 报 "Could not rebind" 时](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/#10-gremlin-报-could-not-rebind-时)。 -- 启动时打不开自己 RocksDB 存储的 PD（例如上一个进程仍持有存储的 `LOCK` 文件），只记录一次 `Failed to open RocksDB`， - 之后继续运行，既不重试也不退出（[apache/hugegraph#3226](https://github.com/apache/hugegraph/issues/3226)）。 - `/v1/ready` 返回 503 和 `STATE_UNINITIALIZED`，因此 Pod 不会 Ready，并退出 Service 端点。但多副本 PD 的启动和存活 - 探测使用仍返回 200 的 `/v1/health`，kubelet 永远不会重启它，它会一直卡住，直到运维人员删除该 Pod。不要改为删除 - `LOCK` 文件：它们防的是可能仍在运行的另一个进程。单副本 PD 的存活探测使用 `/v1/ready`，kubelet 会重启它。 +- 启动时打不开自己 RocksDB 存储的 PD（例如另一个进程仍持有存储的 `LOCK` 文件）既不重试也不退出：它只记录一次 + `Failed to open RocksDB`，之后继续运行，`/v1/ready` 返回 503 和 `STATE_UNINITIALIZED`，而 `/v1/health` 返回 200。 + 这是在 Kubernetes 之外的 PD 进程上观察到的（[apache/hugegraph#3226](https://github.com/apache/hugegraph/issues/3226)）。 + 在 chart 的探针下，这样的 Pod 不会 Ready，并退出 Service 端点。多副本 PD 的启动和存活探测使用 `/v1/health`，kubelet + 不会重启它；单副本 PD 使用 `/v1/ready`，kubelet 会重启容器。这种状态在 Kubernetes 上如何出现，以及重启或删除 Pod + 能否消除它，都尚未测试。删除 `LOCK` 文件不是解决办法：它们保护存储不被可能仍在运行的另一个进程同时打开。 - 当前版本的 Store 恢复由运维人员触发：Store 丢失后的副本重建、leader 均衡、分区再均衡都只在调用 PD 的 REST API 时 执行。丢失了卷的 Store 只有在携带 [apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234)（2026-09-24 合入，尚未进入任何发布版本） @@ -276,7 +277,7 @@ helm uninstall hugegraph --namespace hugegraph | 多节点上 Pod 被 OOM 杀掉或反复重启 | 未设置 resources，JVM 按节点内存取堆：用 `values-cluster.yaml` | | 建图后立刻查询失败 | 副本收敛窗口：见上文"限制" | | `helm test` 打印 `Gremlin failed on ` | 该 Server Pod 丢失了 Gremlin 绑定：删除它（见上文"限制"） | -| 某个 PD Pod 一直不 Ready，其 `/v1/ready` 报 `STATE_UNINITIALIZED` | 它的 RocksDB 存储没有打开：在日志中查找 `Failed to open RocksDB`，然后删除该 Pod（见上文"限制"） | +| 某个 PD Pod 一直不 Ready，其 `/v1/ready` 报 `STATE_UNINITIALIZED` | 它的 RocksDB 存储可能没有打开：在日志中查找 `Failed to open RocksDB`（见上文"限制"） | 其中多数情况的完整排查步骤见 [chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#troubleshooting)。 diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md index 6cf241527b..bab7e1b2ae 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md @@ -241,9 +241,10 @@ defaults. Any upgrade that changes a Pod template rolls that workload once. Poin yourself on any other production values. Under `OnDelete` an upgrade updates the StatefulSet but replaces no Store Pod, and you delete Store Pods one at a time. `OnDelete` only stops automatic advancement: deleting Pods without checking between them carries the same risk. `Up` in PD is not that check: PD marks a Store `Up` at - registration, before it restores partitions, and a stopped Store stays `Up` until a 300 s keep-alive expires - ([apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229) tracks a real restoration signal). - Wait for the replaced Pod to be `Ready`, then confirm every shard group reports its full shard count with one + registration, before it restores partitions, and a stopped Store stays `Up` until a 300 s keep-alive expires. + No endpoint on current images reports that a Store has finished restoring its partitions + ([apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229)), so the check is indirect. Wait for + the replaced Pod to be `Ready`, then confirm every shard group reports its full shard count with one leader; the full procedure is on the [operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/#6-rolling-store-images-safely). - **The controller never rolls more than one PD or Store Pod at a time.** The chart leaves @@ -290,15 +291,17 @@ later install under the same release name comes back with the same credentials. the life of the Pod: it passes readiness and serves REST while every Gremlin request on it fails with `Could not rebind [graph]` ([apache/hugegraph#3228](https://github.com/apache/hugegraph/issues/3228)). The readiness probe calls `/versions` and cannot see this. `helm test` queries Gremlin on every Ready Server Pod and - names a Pod in this state; delete that Pod, and its replacement binds once PD is stable. See + prints the IP of a Pod in this state; delete that Pod, and its replacement binds once PD is stable. See [When Gremlin fails with "Could not rebind"](/docs/quickstart/hugegraph/hugegraph-helm-operations/#10-when-gremlin-fails-with-could-not-rebind). -- A PD that fails to open its RocksDB store at startup, as when the previous process still holds the store's - `LOCK` file, logs `Failed to open RocksDB` once and keeps running without retrying or exiting - ([apache/hugegraph#3226](https://github.com/apache/hugegraph/issues/3226)). `/v1/ready` answers 503 with - `STATE_UNINITIALIZED`, so the Pod is not Ready and leaves the Service endpoints. With more than one PD, though, - startup and liveness probe `/v1/health`, which still answers 200, so the kubelet never restarts it and it stays - stuck until an operator deletes the Pod. Do not delete the `LOCK` files instead: they guard against a second - process that may still be running. A single PD probes `/v1/ready` for liveness, so the kubelet restarts it. +- A PD that cannot open its RocksDB store at startup, for example because another process still holds the + store's `LOCK` file, neither retries nor exits: it logs `Failed to open RocksDB` once and keeps running, with + `/v1/ready` answering 503 and `STATE_UNINITIALIZED` while `/v1/health` answers 200. This was observed on PD + processes outside Kubernetes ([apache/hugegraph#3226](https://github.com/apache/hugegraph/issues/3226)). Under + the chart's probes such a Pod is not Ready and leaves the Service endpoints. With more than one PD, startup and + liveness use `/v1/health`, so the kubelet does not restart it; with a single PD they use `/v1/ready`, so the + kubelet restarts the container. How this state arises on Kubernetes, and whether restarting or deleting the Pod + clears it, has not been tested. Deleting the `LOCK` files is not a fix: they protect the store from a second + process that may still be running. - Store recovery is operator-triggered on current builds: re-replication after Store loss, leader balancing, and partition rebalancing run only when called through PD's REST API. A Store whose volume is lost can be recovered in place only on images carrying @@ -316,7 +319,7 @@ later install under the same release name comes back with the same credentials. | Pods OOM killed or restarting on multi-node | No resources set, JVM heaps sized to node memory: use `values-cluster.yaml` | | Query fails right after creating a graph | Replica convergence window: see Limitations above | | `helm test` prints `Gremlin failed on ` | That Server Pod lost its Gremlin binding: delete it (see Limitations above) | -| A PD Pod stays not Ready and its `/v1/ready` reports `STATE_UNINITIALIZED` | Its RocksDB store did not open: look for `Failed to open RocksDB` in its log, then delete the Pod (see Limitations above) | +| A PD Pod stays not Ready and its `/v1/ready` reports `STATE_UNINITIALIZED` | Its RocksDB store may not have opened: look for `Failed to open RocksDB` in its log (see Limitations above) | Longer walkthroughs for most of these cases are in the [chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#troubleshooting). From 686d3b930f782e14be5b4cf712457896dae920ab Mon Sep 17 00:00:00 2001 From: Himanshu Verma Date: Fri, 2 Oct 2026 18:39:21 +0530 Subject: [PATCH 7/9] doc: run Helm port-forwards in a separate terminal kubectl port-forward stays in the foreground, so the curl commands that followed it in the same block never ran. Split the install check, the operations health check and the per-Pod Gremlin check into a forward block and a call block, in English and Chinese. --- .../hugegraph/hugegraph-helm-operations.md | 15 +++++++++++++-- .../docs/quickstart/hugegraph/hugegraph-helm.md | 9 +++++++-- .../hugegraph/hugegraph-helm-operations.md | 17 ++++++++++++++--- .../docs/quickstart/hugegraph/hugegraph-helm.md | 9 +++++++-- 4 files changed, 41 insertions(+), 9 deletions(-) diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 4fbf2ac0c7..7bd1c639f9 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -42,10 +42,15 @@ PD_SECRET="$(kubectl get secret -n hugegraph hugegraph-pd-auth \ 存活探测终结：默认 20 秒周期、3 次失败阈值把一个卡死 Store 的影响时间限制在约一分钟内，重启后 raft 几秒内就会转移 它持有的分区 leader。 -通过 port-forward 访问 API： +通过 port-forward 访问 API。forward 在前台运行，请让它留在一个终端里： ```bash kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +``` + +再在第二个终端里调用 API（`PASSWORD` 按上文设置）： + +```bash curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/graphs ``` @@ -350,10 +355,16 @@ will not be available in Gremlin Server 任何同时滚动了 PD 和 Server 的升级之后，运行 `helm test`：它通过 headless Service `hugegraph-server-headless` 向每个 Ready 的 Server Pod 发送下面这条绑定图的 Gremlin 查询，最多重试 150 秒，遇到处于这种状态的 Pod 会以 -`Gremlin failed on ` 失败。手工检查单个 Pod 时，对它做 port-forward（镜像不带 curl）： +`Gremlin failed on ` 失败。手工检查单个 Pod 时，在一个终端里对它做 port-forward（镜像不带 curl，forward +在前台运行）： ```bash kubectl port-forward -n hugegraph pod/ 8080:8080 +``` + +再在第二个终端里发送查询： + +```bash curl -s --compressed -u "admin:${PASSWORD}" -H 'Content-Type: application/json' \ -X POST http://127.0.0.1:8080/gremlin \ -d '{"gremlin":"graph.traversal().V().limit(1).count()","aliases":{"graph":"DEFAULT-hugegraph"}}' diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md index e0b78ec48a..34a02c917f 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md @@ -120,11 +120,16 @@ helm test hugegraph --namespace hugegraph 损坏；超时后测试失败，并打印每个失败 Pod 的 IP 和 HTTP 状态码。能通过就绪探测、能提供 REST、但每个 Gremlin 调用都 失败的 Server，正是靠这一步发现的（见"限制"）。HPA 规模较大时，`helm test --timeout` 可能需要高于默认的 5 分钟。 -读取自动生成的 admin 密码并调用 API： +调用 API 前，先在一个终端里启动 port-forward；它会一直在前台运行，直到你停止它： ```bash -PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin -o jsonpath='{.data.password}' | base64 --decode)" kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +``` + +然后在第二个终端里读取自动生成的 admin 密码并调用 API： + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin -o jsonpath='{.data.password}' | base64 --decode)" curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions ``` diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 33501e3ee0..06da371da2 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -47,10 +47,16 @@ alive but frozen) is ended by its liveness probe: the default 20 s period and 3-failure threshold bound the blast radius of a stalled Store at roughly one minute, and raft moves its partition leaders within seconds of the restart. -Reach the API through a port-forward: +Reach the API through a port-forward. It runs in the foreground, so keep +it in one terminal: ```bash kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +``` + +and call the API from a second terminal, with `PASSWORD` set as above: + +```bash curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/graphs ``` @@ -482,11 +488,16 @@ After any upgrade that rolled PD and Server together, run `helm test`: it sends the graph-bound Gremlin query below to every Ready Server Pod through the headless Service `hugegraph-server-headless`, retries for up to 150 seconds, and fails with `Gremlin failed on ` for a Pod in -this state. To check one Pod by hand, port-forward it (the image ships no -curl): +this state. To check one Pod by hand, port-forward it in one terminal +(the image ships no curl, and the forward runs in the foreground): ```bash kubectl port-forward -n hugegraph pod/ 8080:8080 +``` + +and send the query from a second: + +```bash curl -s --compressed -u "admin:${PASSWORD}" -H 'Content-Type: application/json' \ -X POST http://127.0.0.1:8080/gremlin \ -d '{"gremlin":"graph.traversal().V().limit(1).count()","aliases":{"graph":"DEFAULT-hugegraph"}}' diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md index bab7e1b2ae..f18c2819bd 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md @@ -133,11 +133,16 @@ reported as a broken Pod; after that the test fails and prints each failing Pod the check that catches a Server that passes readiness and serves REST while every Gremlin call on it fails (see Limitations). Large HPA fleets may need `helm test --timeout` above the 5-minute default. -Read the generated admin password and call the API: +To call the API, start a port-forward in one terminal; it runs in the foreground until you stop it: ```bash -PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin -o jsonpath='{.data.password}' | base64 --decode)" kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +``` + +Then, in a second terminal, read the generated admin password and call the API: + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin -o jsonpath='{.data.password}' | base64 --decode)" curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions ``` From 795d5f8355d6c9d90bbb68ce87ea1ac0948e42ec Mon Sep 17 00:00:00 2001 From: Himanshu Verma Date: Fri, 2 Oct 2026 18:39:21 +0530 Subject: [PATCH 8/9] doc: correct Helm preset, scaling and recovery details - values-cluster.yaml adds the Server PodDisruptionBudget; values.yaml already enables the PD and Store budgets. - Each JVM sets its heap to half of the free node memory, so the single-node preset needs memory sizing too. - Explain that --reuse-values keeps old values as the base and how to adopt new defaults; Prerequisites referred to this. - The chart rejects PD growth as well as PD and Store shrinks. - The Store per-group partition route never fills term. - patrolPartitions also sets the 180 s balance-shard flag when it reallocates a shard group. - PD exposes POST /v1/members/change behind its REST authentication. - A retained PVC of a retired Store ordinal cannot register again. --- .../hugegraph/hugegraph-helm-operations.md | 18 +++++--- .../quickstart/hugegraph/hugegraph-helm.md | 19 ++++---- .../hugegraph/hugegraph-helm-operations.md | 44 ++++++++++++------- .../quickstart/hugegraph/hugegraph-helm.md | 25 ++++++----- 4 files changed, 66 insertions(+), 40 deletions(-) diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 7bd1c639f9..a9089cf0fe 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -195,8 +195,9 @@ curl -s -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/shardGroups | jq ' 要清楚这证明不了什么：分片列表是 PD 的成员记录，不代表该 Store 已追上 raft 日志。当前镜像没有任何端点报告"恢复 完成"。想看得更近，port-forward 被替换的 Store，读它自己对某个分片组的视图：`GET :8520/v1/partition/` -返回该 Store 持有的 raft 角色、term 和已提交 index，Store 停机时会失败；term 和 index 要与对端 Store 上的同一 -组对比着看，不要单独读。复数形式的 `GET :8520/v1/partitions` 在 +返回该 Store 持有的 raft 角色、leader 和已提交 index（`logIndex`），Store 停机时会失败；index 要与对端 Store 上的 +同一组对比着看，不要单独读。这个路由从不填写 `term` 字段，所以这里的 `term` 总是 0。 +复数形式的 `GET :8520/v1/partitions` 在 [apache/hugegraph#3232](https://github.com/apache/hugegraph/pull/3232)（2026-09-24 合入）之前构建的镜像上， 对任何跟随分片组的 Store 返回 500；之后的镜像对每个 Store 都返回 200，被跟随分组的 `conf` 和 `peers` 为 null。逐组路径在两类镜像上都可用。 @@ -221,14 +222,16 @@ curl -su "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/members kubectl port-forward -n hugegraph pod/ 8620:8620 # 校正分片组并处理 Tombstone Store。 curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/patrolPartitions +# 如果巡检修复了分片组，先等 180 秒（见下文）。 # 先摊平 Raft leader，再摊平分区数据。 curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balanceLeaders curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balancePartitions ``` 任务跑完后再读一次 `/v1/members`：如果中途 leader 迁移，后面的任务其实跑在 follower 上、什么也没做。 -`balancePartitions` 之后至少等 180 秒再执行 `balanceLeaders`：`balancePartitions` 即使什么都没搬也会设置 -180 秒的 balance-shard 标志，窗口内的 `balanceLeaders` 会被拒绝。在 +PD 在两种情况下设置 180 秒的 balance-shard 标志：`patrolPartitions` 每次重新分配分片数不对的分片组时都会设置， +`balancePartitions` 即使什么都没搬也会设置。窗口内 `balanceLeaders` 会被拒绝，`balancePartitions` 什么也不做就返回， +因此在修复了分片组的巡检之后、以及 `balancePartitions` 之后，至少等 180 秒再执行下一次均衡。在 [apache/hugegraph#3233](https://github.com/apache/hugegraph/pull/3233)（2026-09-24 合入）之前构建的镜像上， 拒绝表现为裸的 HTTP 500，原因只在 PD 日志里；之后的镜像把原因放进响应体： `{"status":1001,"error":"balance shard is processing, please try later!"}`。 @@ -280,7 +283,8 @@ hugegraph-store --replicas=0` 分阶段，准备好后再扩回来。Server 会 重新配置它们。chart 通过读取线上 StatefulSet 拒绝 PD 的双向变更和 Store 的缩容，全新安装不受影响。 **PD，双向。** chart 渲染的对端列表只作为 raft 的引导配置生效，已初始化的组会忽略它：3 扩到 5 只是多起两个 -PD，投票配置仍是三个；3 缩到 1 直接失去多数派。成员变更走 PD 客户端 API（没有 REST 路由），chart 无法代劳。 +PD，投票配置仍是三个；3 缩到 1 直接失去多数派。成员变更走 PD 的对端列表变更，PD 客户端 API 和 PD REST 路由 +`POST /v1/members/change` 都提供它；该路由与其他 PD REST 路由一样需要 Basic 认证，chart 两者都不封装。 先通过 PD 改成员、在 `/v1/members` 确认新配置、`kubectl scale` 线上 StatefulSet、再用匹配的 values 执行 `helm upgrade`；在你自己的构建上验证过这套流程之前，请按打算长期保留的 PD 数量安装。 @@ -294,7 +298,9 @@ Store。像灾难恢复退役被替换 Store 那样退役要下线的 Store： 4. 等到 `/v1/shardGroups` 不再列出这些 id，且每个分组都报告完整分片数和一个 leader。 5. `kubectl scale` 线上 StatefulSet，再用匹配的 values 执行 `helm upgrade`。 -删除被移除序号的 PVC 是独立且不可逆的操作；只在第 4 步确认数据已迁走之后再做。 +chart 会保留被移除序号的 PVC（`store.persistentVolumeClaimRetentionPolicy.whenScaled: Retain`），保留下来的 PVC +里仍是已退役的 Store id。PD 拒绝注册处于 Tombstone 或已删除的 Store id，所以之后在这个 PVC 上启动的 Store（例如把 +同一序号重新扩容回来）无法加入。请在第 4 步确认数据已迁走之后、复用这些序号之前删除它们的 PVC；删除不可逆。 ### 9 在集群外运行 Hubble diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md index 34a02c917f..d5063bd4a2 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md @@ -83,8 +83,9 @@ helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace 继续之前需要了解两个默认值： -- **默认不设置 resources。** 每个 Pod 都是 BestEffort，JVM 按节点总内存计算堆大小。单节点没有问题；多节点集群上各个堆 - 会超订节点内存导致进程中止。超出笔记本范围的部署请使用 `values-cluster.yaml`，或按组件设置 `resources`。 +- **默认不设置 resources。** 每个 Pod 都是 BestEffort，每个 JVM 在启动时把堆上限设为它看到的节点空闲内存的一半（各组件另有上限）， + 因此同一节点上的几个 JVM 合起来可能占用超过节点总量的内存。单节点上的 `values-single.yaml` 也是如此。每次安装都请 + 按组件设置 `resources`，单节点预设也不例外；`values-cluster.yaml` 为多节点集群设置了这些值。 - **镜像 tag 跟踪 `latest`**，且 `pullPolicy: Always`，直到下一个 HugeGraph 版本发布。生产环境请固定 tag 或 digest。 #### 3.3 拓扑预设 @@ -95,7 +96,7 @@ chart 附带三个 values 文件： |------|------|----------| | `values.yaml` | 3 PD + 3 Store + 3 Server | 默认；preferred 反亲和，认证开启，Hubble 关闭 | | `values-single.yaml` | 1 + 1 + 1 | 单节点开发与 CI；PVC 更小 | -| `values-cluster.yaml` | 3 + 3 + 3 | 生产起点：JVM 堆与资源设置、PD/Store PodDisruptionBudget、`required` 反亲和、NetworkPolicy 开启、Store 使用 `OnDelete` 更新 | +| `values-cluster.yaml` | 3 + 3 + 3 | 生产起点：JVM 堆与资源设置、Server PodDisruptionBudget、`required` 反亲和、NetworkPolicy 开启、Store 使用 `OnDelete` 更新 | ```bash helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace \ @@ -207,7 +208,9 @@ kubectl port-forward -n hugegraph svc/hugegraph-hubble 8088:8088 helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values ``` -`--reuse-values` 保留 release 的既有覆盖值；不加它，升级会以 chart 默认值重建 release。任何改变 Pod 模板的升级都会让 +`--reuse-values` 保留 release 的既有覆盖值；不加它，升级会以 chart 默认值重建 release。它还会把旧 values 当作完整的基准， +因此由早期 chart 版本创建的 release 不会获得新的默认值（例如加固后的 `securityContext`）；要采用它们，请用 `-f` 传入 +自己的 values，或使用 `--reset-then-reuse-values`。任何改变 Pod 模板的升级都会让 对应工作负载滚动一次。需要提前规划的几点： - **全新安装后的第一次升级会让 PD、Server、Hubble 各滚动一次**，因为跟踪 Secret 的注解第一次观察到安装时创建的 @@ -228,9 +231,9 @@ helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values - **升级不能修改 PVC 大小**：Kubernetes 禁止修改 StatefulSet 的 `volumeClaimTemplates`，带新 `storage.size` 的升级会 被整体拒绝。chart README 记录了支持卷扩容的 StorageClass 上的扩容步骤。 -Server 的扩缩容是普通的 values 变更（`server.replicas` 或 `server.hpa`）。**缩容 PD 或 Store 不是**：Raft 与分片成员 -关系是持久化的，删除 Pod 不会重新配置它们，PD 从 3 缩到 1 会永久失去多数派。chart 会拒绝副本数低于线上 StatefulSet -的升级；先迁移再缩容的手工步骤见 +Server 的扩缩容是普通的 values 变更（`server.replicas` 或 `server.hpa`）。**双向改变 PD 数量、或缩容 Store 都不是**： +Raft 与分片成员关系是持久化的，Pod 本身不会重新配置它们，因此 PD 从 3 缩到 1 会永久失去多数派，新增的 +PD Pod 也不会加入投票配置。chart 读取线上 StatefulSet，拒绝改变 PD 数量或降低 Store 数量的升级；手工步骤见 [运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/#8-伸缩)。 ### 8 卸载 @@ -279,7 +282,7 @@ helm uninstall hugegraph --namespace hugegraph |------|----------| | Store Pod 卡在 `Init:0/1` | PD 未就绪：`kubectl logs -c wait-for-pd`，再看 PD Pod | | PVC 停在 `Pending` | 没有默认 StorageClass，或供给器故障：`kubectl get sc` | -| 多节点上 Pod 被 OOM 杀掉或反复重启 | 未设置 resources，JVM 按节点内存取堆：用 `values-cluster.yaml` | +| Pod 被 OOM 杀掉或反复重启 | 未设置 resources，JVM 按节点内存取堆：设置 `resources`（见"安装"） | | 建图后立刻查询失败 | 副本收敛窗口：见上文"限制" | | `helm test` 打印 `Gremlin failed on ` | 该 Server Pod 丢失了 Gremlin 绑定：删除它（见上文"限制"） | | 某个 PD Pod 一直不 Ready，其 `/v1/ready` 报 `STATE_UNINITIALIZED` | 它的 RocksDB 存储可能没有打开：在日志中查找 `Failed to open RocksDB`（见上文"限制"） | diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 06da371da2..ed9e9871fa 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -246,11 +246,12 @@ Know what this does not prove: the shard list is PD's membership record, not a statement that the Store caught up on the raft log. No endpoint on current images reports restoration-complete. For a closer look, port-forward the replaced Store and read its own view of a group: -`GET :8520/v1/partition/` returns the raft role, term, and -committed index that Store holds, and fails while the Store is down; -compare term and index with a peer Store rather than reading them alone. -The plural `GET :8520/v1/partitions` answers 500 on any Store that follows -a group on images built before +`GET :8520/v1/partition/` returns the raft role, leader, and +committed index (`logIndex`) that Store holds, and fails while the Store +is down; compare the index with a peer Store rather than reading it +alone. This route never fills its `term` field, so `term` always reads 0 +there. The plural `GET :8520/v1/partitions` answers 500 on any Store that +follows a group on images built before [apache/hugegraph#3232](https://github.com/apache/hugegraph/pull/3232) (merged 2026-09-24); after it, it answers 200 on every Store, with `conf` and `peers` null for followed groups. The per-group path works on both. @@ -283,17 +284,21 @@ curl -su "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/members kubectl port-forward -n hugegraph pod/ 8620:8620 # Reconcile shard groups and process tombstoned Stores. curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/patrolPartitions +# If the patrol repaired a shard group, wait 180 s first (see below). # Spread Raft leaders, then partition data. curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balanceLeaders curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balancePartitions ``` Read `/v1/members` again after the tasks: if leadership moved mid-sequence, -the later tasks ran on a follower and did nothing. Wait at least 180 s -before rerunning `balanceLeaders` after a `balancePartitions` call: -`balancePartitions` sets a balance-shard flag for 180 s even when it moves -nothing, and `balanceLeaders` inside that window is refused. On images -built before +the later tasks ran on a follower and did nothing. PD sets a +balance-shard flag for 180 s in two cases: `patrolPartitions` sets it +whenever it reallocates a shard group whose shard count is wrong, and +`balancePartitions` sets it even when it moves nothing. Inside that +window `balanceLeaders` is refused and `balancePartitions` returns +without doing anything, so wait at least 180 s after a patrol that +repaired a group, and after `balancePartitions`, before the next +balancing call. On images built before [apache/hugegraph#3233](https://github.com/apache/hugegraph/pull/3233) (merged 2026-09-24) the refusal is a bare HTTP 500 whose reason appears only in the PD log; after it, the reason comes back in the body as @@ -378,10 +383,12 @@ unaffected. **PD, either direction.** The rendered peer list reaches raft only as its bootstrap configuration, which an initialized group ignores; a 3-to-5 upgrade leaves the voting configuration at three, and a 3-to-1 shrink -loses quorum outright. Membership changes go through the PD client API -(no REST route exposes them), which the chart cannot wrap. Change the -membership through PD, confirm it in `/v1/members`, scale the live -StatefulSet, then `helm upgrade` with the matching value; until that +loses quorum outright. Membership changes go through PD's peer-list +change, which the PD client API and the PD REST route +`POST /v1/members/change` both expose; the route needs the same Basic +authentication as the other PD REST routes, and the chart wraps neither. +Change the membership through PD, confirm it in `/v1/members`, scale the +live StatefulSet, then `helm upgrade` with the matching value; until that sequence is verified on your own build, install the PD count you intend to keep. @@ -402,8 +409,13 @@ replaced one: 5. Scale the live StatefulSet, then `helm upgrade` with the matching value. -Deleting the PVCs of the removed ordinals is separate and permanent; do it -only after step 4 reports the data moved. +The chart keeps the PVCs of removed ordinals +(`store.persistentVolumeClaimRetentionPolicy.whenScaled: Retain`), and a +retained PVC still holds the retired Store id. PD refuses to register a +Store id that is Tombstone or deleted, so a Store that later starts on +that PVC, as when you scale the same ordinal back up, cannot join. +Delete the PVCs of the removed ordinals after step 4 reports the data +moved and before reusing those ordinals; the deletion is permanent. ### 9 Running Hubble outside the cluster diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md index f18c2819bd..96218bb076 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md @@ -90,9 +90,10 @@ pulls. Two defaults to know before going further: -- **No resources are set.** Every pod is BestEffort and each JVM sizes its heap against total node memory. That is - fine on a single node; on a multi-node cluster the heaps oversubscribe the nodes and pods abort. Use - `values-cluster.yaml` or set `resources` per component for anything beyond a laptop. +- **No resources are set.** Every pod is BestEffort, and each JVM sets its maximum heap to half of the memory it + sees free on the node at start (up to a per-component ceiling), so the JVMs on one node can together claim more + memory than the node has. That holds for `values-single.yaml` on one node as well. Set `resources` per component on every install, the single-node + preset included; `values-cluster.yaml` sets them for a multi-node cluster. - **Image tags track `latest`** until the next HugeGraph release is published, with `pullPolicy: Always`. Pin tags or digests for production. @@ -104,7 +105,7 @@ The chart ships three values files: |------|----------|--------------| | `values.yaml` | 3 PD + 3 Store + 3 Server | Default; preferred anti-affinity, auth on, Hubble off | | `values-single.yaml` | 1 + 1 + 1 | Single-node development and CI; smaller PVCs | -| `values-cluster.yaml` | 3 + 3 + 3 | Production starting point: JVM heap and resource settings, PD/Store PodDisruptionBudgets, `required` anti-affinity, NetworkPolicy on, Store `OnDelete` updates | +| `values-cluster.yaml` | 3 + 3 + 3 | Production starting point: JVM heap and resource settings, a Server PodDisruptionBudget, `required` anti-affinity, NetworkPolicy on, Store `OnDelete` updates | ```bash helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace \ @@ -234,7 +235,10 @@ helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values ``` `--reuse-values` keeps the release's existing overrides; without it the upgrade rebuilds the release from chart -defaults. Any upgrade that changes a Pod template rolls that workload once. Points worth planning around: +defaults. It also keeps the old values as the complete base, so a release created by an earlier chart revision does +not pick up new defaults such as the hardened `securityContext`; pass your own values with `-f`, or use +`--reset-then-reuse-values`, to adopt them. Any upgrade that changes a Pod template rolls that workload once. +Points worth planning around: - **The first upgrade after a fresh install rolls PD, Server, and Hubble once**, when the Secret-tracking annotations first observe the install-created Secrets. Store is untouched. A Server that restarts while PD is @@ -261,10 +265,11 @@ defaults. Any upgrade that changes a Pod template rolls that workload once. Poin upgrade with a new `storage.size` is rejected in full. The chart README documents the resize procedure for StorageClasses that support volume expansion. -Scaling Server up and down is a values change (`server.replicas`, or `server.hpa`). Scaling **PD or Store down is -not**: Raft and shard membership are persisted, deleting pods does not reconfigure them, and a 3-to-1 PD shrink -permanently loses quorum. The chart rejects an upgrade whose replica count is below the live StatefulSet; the -manual drain-then-scale procedure is on the +Scaling Server up and down is a values change (`server.replicas`, or `server.hpa`). Changing the **PD count in +either direction, or shrinking Store, is not**: Raft and shard membership are persisted and Pods alone do not +reconfigure them, so a 3-to-1 PD shrink permanently loses quorum and new PD Pods do not join the voting +configuration. The chart reads the live StatefulSet and rejects an upgrade that changes the PD count or +lowers the Store count; the manual procedures are on the [operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/#8-scaling). ### 8 Uninstall @@ -321,7 +326,7 @@ later install under the same release name comes back with the same credentials. |---------|-------------| | Store pods stuck in `Init:0/1` | PD is not ready: `kubectl logs -c wait-for-pd`, then the PD pods | | PVCs stay `Pending` | No default StorageClass, or the provisioner is down: `kubectl get sc` | -| Pods OOM killed or restarting on multi-node | No resources set, JVM heaps sized to node memory: use `values-cluster.yaml` | +| Pods OOM killed or restarting | No resources set, JVM heaps sized to node memory: set `resources` (see Install) | | Query fails right after creating a graph | Replica convergence window: see Limitations above | | `helm test` prints `Gremlin failed on ` | That Server Pod lost its Gremlin binding: delete it (see Limitations above) | | A PD Pod stays not Ready and its `/v1/ready` reports `STATE_UNINITIALIZED` | Its RocksDB store may not have opened: look for `Failed to open RocksDB` in its log (see Limitations above) | From a5fadccb09ab6470577c6fedb7b7dfe5036343b0 Mon Sep 17 00:00:00 2001 From: Himanshu Verma Date: Sat, 3 Oct 2026 22:38:35 +0530 Subject: [PATCH 9/9] doc: align the Helm pages with the final chart behavior The admin password must now be printable ASCII with no spaces, colons or backslashes, and a bring-your-own Secret is checked at Pod start. A no-change upgrade rolls nothing, and a template-only pipeline rolls the Pods that read a generated credential on every sync instead of leaving the annotation inert. Document the Server and Hubble Service exposure acknowledgements, the install-time identity and bootstrap-only values, and why helm rollback bypasses the chart's guards. EN and CN updated together. --- .../hugegraph/hugegraph-helm-operations.md | 5 +- .../quickstart/hugegraph/hugegraph-helm.md | 38 ++++++++++---- .../hugegraph/hugegraph-helm-operations.md | 8 +-- .../quickstart/hugegraph/hugegraph-helm.md | 51 ++++++++++++++----- 4 files changed, 76 insertions(+), 26 deletions(-) diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md index a9089cf0fe..0d23fa507b 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -307,8 +307,9 @@ chart 会保留被移除序号的 PVC（`store.persistentVolumeClaimRetentionPol 集群内 Hubble（`hubble.enabled=true` 加 port-forward）是推荐路径，部署页已覆盖。Hubble 必须在集群外运行时有 两条路径。 -**直连 Server URL**（图、schema、数据、Gremlin；不需要 PD 发现）：关闭 chart 内的 Hubble，暴露 Server -（`server.service.type` NodePort/LoadBalancer，或 Ingress），用如下配置运行独立的 Hubble 镜像： +**直连 Server URL**（图、schema、数据、Gremlin；不需要 PD 发现）：关闭 chart 内的 Hubble，通过做 HTTPS 终结的 +Ingress 或 NodePort/LoadBalancer 类型的 `server.service.type` 暴露 Server，并运行独立的 Hubble 镜像。这类 Service +承载明文 HTTP API，chart 因此要求设置 `server.service.allowInsecureExposure=true`；请先限制谁能访问它。Hubble 配置如下： ```properties pd.enabled=false diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md index d5063bd4a2..a0f8a77fd4 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md @@ -149,9 +149,10 @@ curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions | `-pd-auth` | `secret-key` | PD REST 认证，由 PD、Server、Hubble 读取 | `pd.auth.existingSecret` | 要自行管理凭据，请在安装前创建 Secret 并把对应的 `existingSecret` 指向它；chart 不会改动任何不是它创建的 Secret。取值 -约束：admin 密码不能包含换行、回车或反斜杠，也不能以空白开头或结尾；JWT 密钥至少 32 字节；PD 密钥必须是不含反斜杠、 -首尾没有空格的可打印 ASCII。非法值会在渲染时或启动 -包装脚本中被拒绝，不会被悄悄截断。 +约束：admin 密码必须是不含空格、冒号和反斜杠的可打印 ASCII（镜像会在空格前写入反斜杠，Server 又按每个冒号拆分 +Basic 认证凭据，两者都会让账号无法用 Secret 中的值登录）；JWT 密钥至少 32 字节；PD 密钥必须是不含反斜杠、 +首尾没有空格的可打印 ASCII。非法的内联值在渲染时被拒绝，`existingSecret` 的值在 Pod 启动时被启动包装脚本拒绝， +不会被悄悄截断。 chart 管理的 Secret 在卸载时保留，同名 release 再次安装会复用它们。 @@ -159,8 +160,12 @@ chart 管理的 Secret 在卸载时保留，同名 release 再次安装会复用 轮换与注意事项 - admin 密码只在认证元数据首次创建时生效，之后修改 Secret 不会轮换已有集群的密码。请改用 Server 的 auth API 轮换。 -- 轮换 PD REST Secret 会在下一次 `helm upgrade` 时同时滚动 PD、Server 和 Hubble，保证三者持有的副本一致。纯模板流水线 - （`helm template`、GitOps 渲染器）看不到集群里的 Secret，因此在那里检测轮换的注解不起作用。 +- 轮换凭据会在下一次 `helm upgrade` 时让读取它的 Pod 滚动一次：PD REST Secret 同时滚动 PD、Server 和 Hubble，保证三者 + 持有的副本一致；admin 密码或 JWT 密钥只滚动 Server。 +- 纯模板流水线（`helm template`、Argo CD、模板模式的 Flux）看不到集群里的 Secret，因此 chart 生成的凭据每次渲染都会得到 + 新值：每次同步都会改动 Secret、滚动读取它的 Pod，并让首次启动时创建的 admin 账号与 Secret 不再一致。这类流水线请先 + 创建三个 Secret 并设置对应的 `existingSecret`；见 chart README 的 + [Template-only pipelines](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#template-only-pipelines-gitops)。 - 三个 Secret 即使从不读取也都存在：安装结束打印的说明里有读取 admin 密码和 PD 密钥的 `kubectl get secret` 命令。 @@ -199,7 +204,8 @@ kubectl port-forward -n hugegraph svc/hugegraph-hubble 8088:8088 打开 `http://127.0.0.1:8088`，用 3.4 节的 admin 密码以 `admin` 身份登录。Hubble 通过 PD 发现 Server，集群运维视图无需 额外配置即可工作。Hubble 只提供明文 HTTP：请通过 port-forward 或做 HTTPS 终结的 Ingress 访问，绝不要直接暴露在不可信 -网络上。在集群外运行 Hubble 也可行，但配置更多；见 chart README 的 +网络上；NodePort 或 LoadBalancer 类型的 Hubble Service 需要设置 `hubble.service.allowInsecureExposure=true` 确认后才会 +渲染。在集群外运行 Hubble 也可行，但配置更多；见 chart README 的 [Reaching Hubble](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#reaching-hubble-pick-one-path)。 ### 7 升级 @@ -213,9 +219,10 @@ helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values 自己的 values，或使用 `--reset-then-reuse-values`。任何改变 Pod 模板的升级都会让 对应工作负载滚动一次。需要提前规划的几点： -- **全新安装后的第一次升级会让 PD、Server、Hubble 各滚动一次**，因为跟踪 Secret 的注解第一次观察到安装时创建的 - Secret。Store 不受影响。在 PD 滚动期间重启的 Server 可能丢失 Gremlin 绑定（见"限制"），因此任何同时滚动了 PD 和 - Server 的升级之后，请运行 `helm test`。 +- **没有变更的升级不会滚动任何工作负载。** 跟踪 Secret 的注解对每个凭据的值做哈希（`existingSecret` 则取其线上 + `resourceVersion`），只有 Pod 读取的凭据变化时它才会滚动。确实同时滚动 PD 和 Server 的升级（例如轮换 PD REST Secret， + 或同时更换两者的镜像）可能让 Server 丢失 Gremlin 绑定（见"限制"），因此任何同时滚动了 PD 和 Server 的升级之后， + 请运行 `helm test`。 - **Store 的滚动更新以监听检查推进，而不是以分片恢复推进**，因此控制器可能在上一个 Store 尚未重新加入分片组时就替换 下一个。`values-cluster.yaml` 因此设置了 `store.updateStrategy.type=OnDelete`；`values.yaml` 和 `values-single.yaml` 仍为 `RollingUpdate`，其他生产 values 请自行设置。`OnDelete` 下升级只更新 StatefulSet，不替换任何 Store Pod，由你逐个 @@ -230,6 +237,15 @@ helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values 手工控制。 - **升级不能修改 PVC 大小**：Kubernetes 禁止修改 StatefulSet 的 `volumeClaimTemplates`，带新 `storage.size` 的升级会 被整体拒绝。chart README 记录了支持卷扩容的 StorageClass 上的扩容步骤。 +- **部分值在 release 初始化后就固定。** `nameOverride`、`fullnameOverride`、PD 与 Store 的 raft 端口以及存储设置属于 + 安装时身份：chart 会对照线上 StatefulSet 拒绝修改 override 或 raft 端口，Kubernetes 拒绝修改存储设置。 + `server.auth.admin.*` 和分区分片数只在初始化时生效，已初始化的集群会忽略新值。chart README 按 + [生命周期](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#settings-by-lifecycle)对各个值分了类。 +- **`helm rollback` 不运行 chart 的任何检查。** 回滚（包括 `--atomic` 升级失败后的自动回滚）直接重新应用之前保存的 + 清单，不渲染 chart，因此跨越 PD 或 Store 副本数变更的回滚会把 StatefulSet 直接缩放到旧数量，可能让 PD 失去多数派。 + 不支持跨越成员关系或身份变更的回滚：要回到更早的 chart 或镜像，请用保留当前拓扑 values 的正向升级，且改变副本数的 + 升级不要加 `--atomic`。任何回滚之后要做的检查见 chart README 的 + [Rollback](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#rollback)。 Server 的扩缩容是普通的 values 变更（`server.replicas` 或 `server.hpa`）。**双向改变 PD 数量、或缩容 Store 都不是**： Raft 与分片成员关系是持久化的，Pod 本身不会重新配置它们，因此 PD 从 3 缩到 1 会永久失去多数派，新增的 @@ -274,6 +290,10 @@ helm uninstall hugegraph --namespace hugegraph [apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234)（2026-09-24 合入，尚未进入任何发布版本） 的镜像上才能原地恢复；更早的镜像上，替换 Store Pod 时请保留它的 PVC。操作手册见 [运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/)的灾难恢复一节。 +- Server 和 Hubble 只提供明文 HTTP，PD gRPC 没有认证。它们中任何一个的 NodePort 或 LoadBalancer Service，都要设置对应的 + `pd.service.allowInsecureExposure`、`server.service.allowInsecureExposure` 或 `hubble.service.allowInsecureExposure` + 才会渲染；不带 `tls` 的 Server 或 Hubble Ingress 也要设置各自的 `allowPlainHttp`。优先使用 port-forward 或做 HTTPS + 终结的 Ingress，暴露 Service 前先限制谁能访问它。 - 集群内无 TLS 终结，无备份工具，无 Operator，无内置监控栈。 ### 10 排障 diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md index ed9e9871fa..a8f63d62ad 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -424,9 +424,11 @@ recommended path and is covered on the deployment page. Two paths exist for a Hubble that must run outside the cluster. **Direct Server URL** (graph, schema, data, and Gremlin; no PD -discovery): leave in-chart Hubble off, expose Server -(`server.service.type` NodePort/LoadBalancer, or Ingress), and run a -standalone Hubble image with: +discovery): leave in-chart Hubble off, expose Server through an +HTTPS-terminating Ingress or a NodePort/LoadBalancer `server.service.type`, +and run a standalone Hubble image. The Service carries the plain-HTTP API, +so the chart requires `server.service.allowInsecureExposure=true` for it; +restrict who can reach it first. The Hubble properties are: ```properties pd.enabled=false diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md index 96218bb076..836cee94f3 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md @@ -163,10 +163,12 @@ an `existingSecret` you created wins, then an inline value, then a random value | `-pd-auth` | `secret-key` | PD REST authentication, read by PD, Server, and Hubble | `pd.auth.existingSecret` | To manage a credential yourself, create the Secret before installing and point the matching `existingSecret` value -at it; the chart never modifies a Secret it did not create. Value constraints: the admin password must not contain -newlines, carriage returns, or backslashes, or start or end with whitespace; the JWT key must be at least 32 bytes; -the PD secret must be printable ASCII with no backslashes and no leading or trailing space. Invalid values are -rejected at render time or by the startup wrapper, not silently truncated. +at it; the chart never modifies a Secret it did not create. Value constraints: the admin password must be printable +ASCII with no spaces, colons, or backslashes (the image stores a space with a backslash before it, and the Server splits +Basic-auth credentials on every colon, so either would leave the account unable to log in with the Secret value); the +JWT key must be at least 32 bytes; the PD secret must be printable ASCII with no backslashes and no leading or trailing +space. Invalid inline values are rejected at render time, and an `existingSecret` by the startup wrapper when the Pod +starts, not silently truncated. Chart-managed Secrets are kept on uninstall and reused by a later install under the same release name. @@ -175,9 +177,14 @@ Chart-managed Secrets are kept on uninstall and reused by a later install under - The admin password is applied only when the auth metadata is first created, so changing the Secret later does not rotate an existing cluster's password. Rotate it through the Server's auth API instead. -- Rotating the PD REST Secret rolls PD, Server, and Hubble together on the next `helm upgrade`, which keeps their - copies in step. Template-only pipelines (`helm template`, GitOps renderers) cannot see live Secrets, so there the - rotation-detecting annotation is inert. +- Rotating a credential rolls the Pods that read it once, on the next `helm upgrade`: PD, Server, and Hubble together + for the PD REST Secret, which keeps their copies in step, and Server for the admin password or JWT key. +- Template-only pipelines (`helm template`, Argo CD, Flux in template mode) cannot see live Secrets, so a + chart-generated credential gets a new value on every render: each sync changes the Secret, rolls the Pods that read + it, and leaves the admin account created at first start out of step with the Secret. For such a pipeline, create the + three Secrets first and set the `existingSecret` values; see + [Template-only pipelines](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#template-only-pipelines-gitops) + in the chart README. - All three Secrets exist even if you only ever read one: the post-install notes print the exact `kubectl get secret` commands for the admin password and the PD secret. @@ -223,7 +230,8 @@ kubectl port-forward -n hugegraph svc/hugegraph-hubble 8088:8088 Open `http://127.0.0.1:8088` and log in as `admin` with the admin password from Section 3.4. Hubble discovers the Servers through PD, so the cluster operations view works without extra wiring. Hubble serves plain HTTP: reach it -through a port-forward or an HTTPS-terminating Ingress, never directly from an untrusted network. Running Hubble +through a port-forward or an HTTPS-terminating Ingress, never directly from an untrusted network; a NodePort or +LoadBalancer Hubble Service is refused unless `hubble.service.allowInsecureExposure=true` acknowledges it. Running Hubble outside the cluster is possible but takes more wiring; see [Reaching Hubble](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#reaching-hubble-pick-one-path) in the chart README. @@ -240,10 +248,11 @@ not pick up new defaults such as the hardened `securityContext`; pass your own v `--reset-then-reuse-values`, to adopt them. Any upgrade that changes a Pod template rolls that workload once. Points worth planning around: -- **The first upgrade after a fresh install rolls PD, Server, and Hubble once**, when the Secret-tracking - annotations first observe the install-created Secrets. Store is untouched. A Server that restarts while PD is - rolling can lose its Gremlin binding (see Limitations), so run `helm test` after any upgrade that rolled PD and - Server together. +- **A no-change upgrade rolls nothing.** The Secret-tracking annotations hash each credential's value, or the live + `resourceVersion` of an `existingSecret`, so a Pod rolls only when a credential it reads changes. An upgrade that + does roll PD and Server together, such as a PD REST Secret rotation or an image change on both, can leave a Server + without its Gremlin binding (see Limitations), so run `helm test` after any upgrade that rolled PD and Server + together. - **Store rolling updates advance on a listener check, not on shard recovery**, so the controller can replace the next Store while the previous one is still rejoining its shard groups. `values-cluster.yaml` therefore sets `store.updateStrategy.type=OnDelete`; `values.yaml` and `values-single.yaml` keep `RollingUpdate`, so set it @@ -264,6 +273,19 @@ Points worth planning around: - **PVC sizes cannot be changed by upgrade**: Kubernetes forbids changing StatefulSet `volumeClaimTemplates`, so an upgrade with a new `storage.size` is rejected in full. The chart README documents the resize procedure for StorageClasses that support volume expansion. +- **Some values are fixed once the release is initialized.** `nameOverride`, `fullnameOverride`, the PD and Store + raft ports, and the storage settings are install-time identity: the chart refuses an override or raft port change + against the live StatefulSets, and Kubernetes refuses the storage change. `server.auth.admin.*` and the partition + shard counts apply at bootstrap only, so an initialized cluster ignores a new value. The chart README sorts the + values by + [lifecycle](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#settings-by-lifecycle). +- **`helm rollback` runs none of the chart's guards.** A rollback, including the automatic one after a failed + `--atomic` upgrade, reapplies an earlier stored manifest without rendering the chart, so a rollback across a PD or + Store replica change scales the StatefulSet straight to the old count and can drop PD below quorum. Rollback across + a membership or identity change is unsupported: return to an earlier chart or image with a forward upgrade that + keeps the current topology values, and keep `--atomic` off for upgrades that change replicas. The chart README + covers the checks to run after any + [rollback](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#rollback). Scaling Server up and down is a values change (`server.replicas`, or `server.hpa`). Changing the **PD count in either direction, or shrinking Store, is not**: Raft and shard membership are persisted and Pods alone do not @@ -318,6 +340,11 @@ later install under the same release name comes back with the same credentials. [apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234) (merged 2026-09-24, in no release yet); on earlier images, keep a Store's PVC when replacing its Pod. The Disaster Recovery section of the [operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/) is the runbook. +- The Server and Hubble serve plain HTTP, and PD gRPC has no authentication. A NodePort or LoadBalancer Service for + any of them is refused unless the matching `pd.service.allowInsecureExposure`, `server.service.allowInsecureExposure`, + or `hubble.service.allowInsecureExposure` is set, and a Server or Hubble Ingress without `tls` is refused unless its + `allowPlainHttp` is set. Prefer a port-forward or an HTTPS-terminating Ingress, and restrict who can reach an + exposed Service first. - No TLS termination inside the cluster, no backup tooling, no Operator, and no bundled monitoring stack. ### 10 Troubleshooting