diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md new file mode 100644 index 0000000000..e97b747441 --- /dev/null +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -0,0 +1,381 @@ +--- +title: "在 Kubernetes 上运维 HugeGraph" +linkTitle: "Kubernetes 运维 (Helm)" +weight: 5 +search_keywords: + - helm + - kubernetes + - operations + - disaster recovery + - networkpolicy +--- + +### 1 适用范围 + +本页面向已经用 Helm chart 部署好 HugeGraph 集群的运维场景。安装、升级、卸载见 +[部署页](/cn/docs/quickstart/hugegraph/hugegraph-helm/)；完整参数参考仍在 +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration)。 + +下文命令假定 release 名为 `hugegraph`、namespace 为 `hugegraph`，请按需替换。两个凭据会反复用到，都从 +chart 管理的 Secret 读取： + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin \ + -o jsonpath='{.data.password}' | base64 --decode)" +PD_SECRET="$(kubectl get secret -n hugegraph hugegraph-pd-auth \ + -o jsonpath='{.data.secret-key}' | base64 --decode)" +``` + +### 2 端口与健康 + +| 组件 | 端口 | 用途 | +|------|------|------| +| PD | `8686` | gRPC（Store 与 Server 客户端） | +| PD | `8620` | REST / 健康探测 | +| PD | `8610` | Raft | +| Store | `8500` | gRPC | +| Store | `8510` | Raft | +| Store | `8520` | REST / 健康探测 | +| Server | `8080` | Gremlin 与 REST API | + +所有端口都可以通过 values 配置；修改 `server.port` 会同时更新监听、容器端口和 Service。进程存活但停止响应的组件由 +存活探测终结：默认 20 秒周期、3 次失败阈值把一个卡死 Store 的影响时间限制在约一分钟内，重启后 raft 几秒内就会转移 +它持有的分区 leader。 + +通过 port-forward 访问 API。forward 在前台运行，请让它留在一个终端里： + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +``` + +再在第二个终端里调用 API（`PASSWORD` 按上文设置）： + +```bash +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/graphs +``` + +### 3 调度 + +每个组件（`pd`、`store`、`server`、`hubble`）都提供 `nodeSelector`、`tolerations`、`affinity`、 +`topologySpreadConstraints` 和 `priorityClassName`。例如把 Store 固定到打了标签的节点： + +```yaml +store: + nodeSelector: + hugegraph/role: storage +``` + +`antiAffinity`（`required` | `preferred` | `disabled`）为 `pd`、`store`、`server` 渲染按主机名的 +pod 反亲和预设；Hubble 设计为单副本，没有这个配置项。设置原生 `affinity` 会整体替换预设。三者默认都是 +`preferred`，因此节点数少于副本数的集群也能调度。代价是：节点资源紧张时调度器可能把副本放到同一节点，单个节点 +故障就可能同时带走多个 PD 或 Store 副本。节点足够的生产集群应把 `pd.antiAffinity` 和 `store.antiAffinity` +固定为 `required`，`values-cluster.yaml` 即如此。 + +### 4 分区分片 + +全新安装会在 `store.replicas` 不低于 3 时把 PD 的分片数种子设为 3，否则设为 1；没有这个种子，PD 镜像会把 +`partition.default-shard-count` 固定为 1，chart 部署的集群就没有 Store 级高可用。chart 通过 PD 的 +`JAVA_OPTS` 传入 `-Dpartition.default-shard-count`，不影响镜像的 JVM 自动调优。 + +**种子只在首次引导时生效。** PD 第一次以空存储启动时把分片数持久化到自己的元数据里，此后存储的值才是权威：之后修改 +`pd.partition.defaultShardCount` 或跨过推导边界地伸缩 `store.replicas`，对已初始化的集群都没有效果。要修改运行中 +集群的分片数，请使用 PD 自己的配置 API（只接受不超过存活 Store 数的奇数），然后触发 +`GET /v1/task/patrolPartitions`，并预期分片组会发生再分配。 + +分片数还决定初始分区数：`store.replicas x storeMaxShardCount / shardCount`，只在引导时计算一次。按镜像默认的 +`store-max-shard-count`（12），默认 3 Store 安装得到 12 个分区；需要更多分区时调大 +`pd.partition.storeMaxShardCount`（同样只在首次引导时生效）。 + +显式设置的 `pd.partition.defaultShardCount` 必须是奇数且不超过 `store.replicas`；其他值会在渲染时被 chart +拒绝，因为 PD 会悄悄钳制或拒绝它们，渲染通过并不等于设置生效。 + +### 5 NetworkPolicy + +`networkPolicy.enabled` 为每个组件（PD、Store、Server，启用时还有 Hubble）渲染一个 NetworkPolicy。每个策略 +都对自己的 Pod 做双向隔离并列出所需流量，因此应用顺序无关紧要。`values.yaml` 默认关闭（基础配置无法预知你的 +客户端是谁），`values-cluster.yaml` 默认开启。 + +它只在集群网络插件强制执行 NetworkPolicy 时才生效（kind v0.25 及以上、k3s、Calico、Cilium）；其他插件会接受 +对象但不执行。验证方法：在另一个 namespace 起一个不带 chart 标签的 Pod，`curl` PD 客户端 Service 的 REST +端口，必须超时。 + +策略开启后，release 只放行自己的流量： + +| 目标 | 来源与端口 | +|---|---| +| PD | PD：raft、gRPC。Store、Server、`pd` 模式的 Hubble：gRPC、REST | +| Store | Store：raft。Server：gRPC、REST。`pd` 模式的 Hubble：REST | +| Server | Hubble 和 `helm test` Pod：`server.port` | +| Hubble | 无（port-forward 走回环，不需要规则） | + +每个组件还可以走 53 端口解析 DNS。release 之外的任何来源都必须列入 +`networkPolicy..extraIngress` 才被放行，包括 Ingress 控制器和 NodePort / LoadBalancer Service +的客户端。在 `extraIngress` 为空时暴露 PD、Server 或 Hubble，或设置 `server.advertiseUrl`，渲染会直接失败 +而不是打开端口。该检查只看 chart 自己创建的暴露；你自行添加的 Service、Gateway 路由或代理需要自己的条目。集群内 +`pd` 模式的 Hubble 与 `server.advertiseUrl` 同时使用时，在 `networkPolicy.hubble.extraEgress` 放行该对外地址之前， +渲染同样会失败：PD 会把这个 URL 交给 Hubble 用于发现，而 Hubble 的出向策略默认只能访问本 release 的 Pod。 + +两个出向事实需要规划。PD、Store、Server 除 DNS 外不访问 release 以外的任何地址，因此需要外呼的功能在策略开启时 +不可用；默认唯一的外呼方是 Store 镜像，它每次启动都从 github.com 下载 `libjemalloc.so`。策略开启时该连接约两分 +钟后超时，Store 不带 jemalloc 继续启动（kind 上实测：就绪时间从 11 秒变为 151 秒）；无外网的集群同理。另外 +`helm test` Pod 不被任何 chart 策略选中，因此在你自己的 namespace 级默认拒绝策略下，要给它放行到 `server.port` +和 DNS 的出向流量。 + +**放行其他工作负载。** 每条规则的 `from` 必须指明对端；要放行任意地址，请显式写 `0.0.0.0/0` 这样的 +`ipBlock`。同一个对端里的 `namespaceSelector` 和 `podSelector` 必须同时匹配；写成两个对端则满足其一即可。 +NodePort / LoadBalancer 客户端在 Pod 看来是什么地址，取决于网络插件、`externalTrafficPolicy` 和请求到达的 +节点。在双节点 kind 集群上用 NodePort Server 实测： + +- kindnet，以及开 kube-proxy replacement 的 Cilium：请求发到 Server 所在节点时带客户端地址；经另一节点转发时 + 带那个节点的地址。 +- Calico：经另一节点转发的请求来自该节点在 Pod CIDR 里的隧道地址。 +- 用 kube-proxy 的 Cilium：任何 `ipBlock` 规则都放行不了 NodePort 流量，因为 Cilium 用自己的节点身份而不是 + CIDR 识别节点地址。 + +请用你实际运行的插件测试，并按看到的来源地址写 CIDR。放行 Ingress 控制器、应用 namespace 和 Prometheus 的 +完整示例： + +```yaml +networkPolicy: + server: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: ingress-nginx + podSelector: + matchLabels: + app.kubernetes.io/name: ingress-nginx + ports: + - port: 8080 + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: apps + ports: + - port: 8080 + pd: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: monitoring + ports: + - port: 8620 +``` + +### 6 安全地滚动 Store 镜像 + +Store 的滚动更新以监听检查推进，而不是以分片恢复推进，控制器可能在上一个 Store 尚未重新加入分片组时就替换下一 +个。`values-cluster.yaml` 因此设置了 `store.updateStrategy.type=OnDelete`（其他生产 values 请自行设置）：升级只更新 +StatefulSet，不替换任何 Store Pod，由你逐个删除 Store Pod，并在两次删除之间做检查。`OnDelete` 只是停止自动推进； +删除 Pod 时不做下面的检查，风险相同。 + +PD 里的 `Up` 不是这个检查。PD 在注册时就把 Store 标为 `Up`，此时它还没恢复任何分区；已停止的 Store 也会在 +keep-alive 记录过期前（当前镜像为 300 秒）一直保持 `Up` 并留在所有分片组里：窗口内删除又回来的 Pod 根本不会离开 +`Up` 状态。检查要从 Pod 开始： + +```bash +kubectl -n hugegraph wait --for=condition=Ready \ + pod/hugegraph-store- --timeout=10m +``` + +然后从 PD leader 逐组检查分片成员与 leader（找 leader 的方法见下文"灾难恢复"）： + +```bash +curl -s -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/shardGroups | jq ' + .shardGroups[] | {id: (.id // 0), + shards: [.shards[] | {storeId, role}], + leaders: [.shards[] | select(.role=="Leader")] | length}' +``` + +只有当被替换的 Pod 已 `Ready`、它的 Store id 在 `/v1/stores` 里显示新的 `lastHeartBeat`、且每个分片组都报告 +完整分片数和恰好一个 `Leader` 时，才删除下一个 Store。 + +要清楚这证明不了什么：分片列表是 PD 的成员记录，不代表该 Store 已追上 raft 日志。当前镜像没有任何端点报告"恢复 +完成"。想看得更近，port-forward 被替换的 Store，读它自己对某个分片组的视图：`GET :8520/v1/partition/` +返回该 Store 持有的 raft 角色、leader 和已提交 index（`logIndex`），Store 停机时会失败；index 要与对端 Store 上的 +同一组对比着看，不要单独读。这个路由从不填写 `term` 字段，所以这里的 `term` 总是 0。 +复数形式的 `GET :8520/v1/partitions` 在 +[apache/hugegraph#3232](https://github.com/apache/hugegraph/pull/3232)（2026-09-24 合入）之前构建的镜像上， +对任何跟随分片组的 Store 返回 500；之后的镜像对每个 Store 都返回 200，被跟随分组的 `conf` 和 `peers` 为 +null。逐组路径在两类镜像上都可用。 + +成员检查通过后留出余量再删下一个 Pod，把 `store.pdb.minAvailable` 保持在 `replicas - 1`，意外的第二次驱逐会被 +拒绝；任何缺分片或没有 leader 的分组都应视为停止信号。真正的"分区恢复完成"信号是上游工作，在 +[apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229) 跟踪。 + +### 7 灾难恢复 + +当前版本 PD 自动做的事很少：60 秒一次的巡检只把停止心跳的 Store 标为 `Offline`。没有自动副本重建；丢失 Store +的副本重新放置、按分片数校正分片组、处理 Tombstone Store，都只在显式触发分区巡检时执行。 + +任务端点在收到请求的那台 PD 上本地执行，follower 会返回空的成功响应但什么也不做。port-forward 客户端 Service +选中的 PD 是任意的，所以先找 leader，再 port-forward 那个 Pod（forward 在前台运行，需要第二个终端）： + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-pd-client 8620:8620 +# 读 .data.pdLeader.raftUrl，其主机名即 leader Pod。 +curl -su "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/members +# 停掉 Service 的 forward，改为 forward leader Pod。 +kubectl port-forward -n hugegraph pod/ 8620:8620 +# 校正分片组并处理 Tombstone Store。 +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/patrolPartitions +# 如果巡检修复了分片组，先等 180 秒（见下文）。 +# 先摊平 Raft leader，再摊平分区数据。 +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balanceLeaders +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balancePartitions +``` + +任务跑完后再读一次 `/v1/members`：如果中途 leader 迁移，后面的任务其实跑在 follower 上、什么也没做。 +PD 在两种情况下设置 180 秒的 balance-shard 标志：`patrolPartitions` 每次重新分配分片数不对的分片组时都会设置， +`balancePartitions` 即使什么都没搬也会设置。窗口内 `balanceLeaders` 会被拒绝，`balancePartitions` 什么也不做就返回， +因此在修复了分片组的巡检之后、以及 `balancePartitions` 之后，至少等 180 秒再执行下一次均衡。在 +[apache/hugegraph#3233](https://github.com/apache/hugegraph/pull/3233)（2026-09-24 合入）之前构建的镜像上， +拒绝表现为裸的 HTTP 500，原因只在 PD 日志里；之后的镜像把原因放进响应体： +`{"status":1001,"error":"balance shard is processing, please try later!"}`。 + +要分辨"真的执行了"和"空跑"，得看 PD leader 的日志，响应本身分辨不了：`patrolPartitions` 无论有没有修复、在 +leader 还是 follower 上，都返回同样的空成功（看日志里的 `reallocShards`、`shardOffline`、`storeTurnoff`，或 +对比前后的 `/v1/shardGroups`）；`balancePartitions` 在 leader 上返回 `{}`，在 follower 上返回空响应体。只有 +`balanceLeaders` 的响应体携带工作内容。可区分的任务响应是上游工作，在 +[apache/hugegraph#3231](https://github.com/apache/hugegraph/issues/3231) 跟踪。 + +替换一个回不来的 Store 之后执行 `patrolPartitions`，集群稳定后执行 `balancePartitions`，重启导致 leader 分布 +倾斜后执行 `balanceLeaders`。 + +**丢失 Store 卷。** 带着空 PVC 重建的 Store，在携带 +[apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234)（2026-09-24 合入）的镜像上可以原地 +恢复；更早的镜像，包括所有已发布版本的镜像，都不行。两种情况下替换者都会以**新的 Store ID** 注册，而 Pod 名、 +DNS 名、raft 地址不变，`/v1/stores` 会在同一地址下列出两个 ID。 + +在含 #3234 的镜像上，退役流程是可用的：在 `/v1/stores` 里找到旧 ID（被替换 Pod 地址下不是新注册的那一行），在 +PD leader 上 `POST /v1/store/` 并携带 `{"storeState":"Tombstone"}`，执行 +`GET /v1/task/patrolPartitions`，然后等待；确认每个分片组都恢复到完整分片数且恰好一个 leader、没有分组再引用 +旧 ID、被替换 Store 自己的 `:8520/v1/partition/` 对每个分组都返回 200；最后 +`DELETE /v1/store/` 清除退役记录。在 3+3+3 安装上用 `master` `dbb6663a` 构建的镜像实测：替换者 156 秒 +就绪，Tombstone 加巡检之后 1 秒内 12 个分组全部收敛到新 ID（空 Store 通过 raft 快照安装追平数据），持续写入 +没有丢失任何已确认的写。 + +在不含 #3234 的镜像上，同样的退役流程会执行但修不好分组：PD 发起配置变更，但分组 leader 看到该地址已在组内， +jraft 无可添加，分组继续记录旧 ID。实测：20 分钟、三次巡检之后，12 个分组仍全部引用退役 ID，替换 Store 不持有 +任何分区。健康面板完全看不出这个故障：`/v1/stores`、集群状态、Hubble、Pod 就绪全部显示健康，而每个分片组实际 +只运行在两个存活副本上。唯一能看出问题的检查是被替换 Store 自己的 `:8520/v1/partition/`。 + +因此默认做法不变：替换 Store Pod 时保留它的 PVC（Store id 存在数据目录里，Pod 会以同一 id 回来）。空 PVC 替换 +只作为含 #3234 镜像上的恢复手段；在已发布镜像上，卷真的丢了就当集群已降级处理，准备重建而不是原地恢复。 + +周期性 leader 均衡在 [apache/hugegraph#3135](https://github.com/apache/hugegraph/issues/3135) 跟踪； +灾难恢复指标在 [apache/hugegraph#3136](https://github.com/apache/hugegraph/issues/3136) 跟踪。 + +### 8 伸缩 + +PD 和 Store 在资源名里为最大 StatefulSet 序号预留了空间，伸缩不会改名 PersistentVolumeClaim、不会移动 Pod +身份；两者上限都是 99 副本。Server 通过 `server.replicas` 或 `server.hpa` 自由伸缩（HPA 开启时 Deployment +省略 `spec.replicas`，升级不会覆盖自动扩缩的副本数）。 + +分阶段上线没法写进 values 文件（schema 要求每个组件至少一个副本）；用 `kubectl scale statefulset +hugegraph-store --replicas=0` 分阶段，准备好后再扩回来。Server 会保持未就绪等待 Store 注册，下一次 +`helm upgrade` 会按 values 恢复完整拓扑。 + +修改线上 release 的 PD 或 Store 副本数不是普通的 values 变更：raft 与分片成员关系是持久化的，Pod 本身不会 +重新配置它们。chart 通过读取线上 StatefulSet 拒绝 PD 的双向变更和 Store 的缩容，全新安装不受影响。 + +**PD，双向。** chart 渲染的对端列表只作为 raft 的引导配置生效，已初始化的组会忽略它：3 扩到 5 只是多起两个 +PD，投票配置仍是三个；3 缩到 1 直接失去多数派。成员变更走 PD 的对端列表变更，PD 客户端 API 和 PD REST 路由 +`POST /v1/members/change` 都提供它；该路由与其他 PD REST 路由一样需要 Basic 认证，chart 两者都不封装。 +先通过 PD 改成员、在 `/v1/members` 确认新配置、`kubectl scale` 线上 StatefulSet、再用匹配的 values 执行 +`helm upgrade`；在你自己的构建上验证过这套流程之前，请按打算长期保留的 PD 数量安装。 + +**Store，缩容。** 迁出是状态转换，不是均衡：`patrolPartitions` 和 `balancePartitions` 都不会退役健康的 +Store。像灾难恢复退役被替换 Store 那样退役要下线的 Store： + +1. 确认剩余 Store 仍能承载持久化的分片数（`pd.partition.defaultShardCount`；留空时 `store.replicas` 不低于 + 3 推导为 3）。 +2. 通过 `/v1/stores` 按 Pod 地址把要删除的序号（最大的几个）映射到 Store id。 +3. 对每个要下线的 id `POST /v1/store/`，携带 `{"storeState":"Tombstone"}`。 +4. 等到 `/v1/shardGroups` 不再列出这些 id，且每个分组都报告完整分片数和一个 leader。 +5. `kubectl scale` 线上 StatefulSet，再用匹配的 values 执行 `helm upgrade`。 + +chart 会保留被移除序号的 PVC（`store.persistentVolumeClaimRetentionPolicy.whenScaled: Retain`），保留下来的 PVC +里仍是已退役的 Store id。PD 拒绝注册处于 Tombstone 或已删除的 Store id，所以之后在这个 PVC 上启动的 Store（例如把 +同一序号重新扩容回来）无法加入。请在第 4 步确认数据已迁走之后、复用这些序号之前删除它们的 PVC；删除不可逆。 + +### 9 在集群外运行 Hubble + +集群内 Hubble（`hubble.enabled=true` 加 port-forward）是推荐路径，部署页已覆盖。Hubble 必须在集群外运行时有 +两条路径。 + +**直连 Server URL**（图、schema、数据、Gremlin；不需要 PD 发现）：关闭 chart 内的 Hubble，通过做 HTTPS 终结的 +Ingress 或 NodePort/LoadBalancer 类型的 `server.service.type` 暴露 Server，并运行独立的 Hubble 镜像。这类 Service +承载明文 HTTP API，chart 因此要求设置 `server.service.allowInsecureExposure=true`；请先限制谁能访问它。Hubble 配置如下： + +```properties +pd.enabled=false +server.direct_url=https://: +``` + +配置文件挂载到镜像内的 `/hubble/conf/hugegraph-hubble.properties`（工作目录是 `/hubble`）。 +`server.direct_url` 要用 HTTPS 或可信通道：登录会把 Server 凭据发到这个 URL。 + +**PD 发现**（集群外的 Hubble 向 PD 询问 Server 地址）：`*.svc` 这类集群内名字在外部不可解析，chart 提供两个 +配置项。把 `server.advertiseUrl` 设为外部 Hubble 发现后要使用的绝对 `http(s)://` URL，chart 会用它替代集群内 +Service URL 注册到 PD。再暴露 PD 客户端 Service（`pd.service.type` NodePort/LoadBalancer），这需要 +`pd.service.allowInsecureExposure=true`，因为 PD gRPC 没有认证；先限制谁能访问它（chart 的 NetworkPolicy +开启时，调用方没有列入 `networkPolicy.pd.extraIngress` 前渲染会拒绝这种暴露）。然后用如下配置运行独立 +Hubble： + +```properties +pd.enabled=true +pd.peers=: +pd.server=: +``` + +取舍：设置 `server.advertiseUrl` 后，每个 Server 副本注册的都是同一个逻辑 URL，PD 会把它返回给所有发现客户 +端，包括集群内的 Hubble（开启 chart 的 NetworkPolicy 时，它还需要 `networkPolicy.hubble.extraEgress`，见上文 +NetworkPolicy 一节）。留空则走默认的集群内路径，每个 Server Pod 注册自己的 IP。 + +本机快速验证（集群和 Hubble 在同一台机器）：port-forward Server `8080` 和 PD 客户端 `8620`/`8686`，设置 +`server.advertiseUrl=http://127.0.0.1:8080`，用 `--network host` 和上面的 PD 配置运行独立 Hubble，再打开 +`8088` 端口。 + +### 10 Gremlin 报 "Could not rebind" 时 + +这个报错有两个成因，持续时间不同。 + +**建图后的收敛窗口。** 处理 `CreateGraph` 的 Server 会等自己的 Gremlin 绑定完成才返回 HTTP 200 +（[#3138](https://github.com/apache/hugegraph/pull/3138)），所以在同一个 Server 上建图后立即查询是可靠的。 +其他副本独立收敛，在完成之前，经负载均衡 Service 路由到未收敛副本的 Gremlin 查询可能报 400，如 +`Could not rebind [g]`。带退避重试（窗口通常几秒内关闭），对"建图后立即验证"的流程用会话粘滞或 +port-forward，或先在每个副本上轮询 `/graphs` 再放开查询流量。集群级就绪在 +[#3137](https://github.com/apache/hugegraph/issues/3137) 跟踪。 + +**在 PD 滚动期间启动的 Server Pod。** Gremlin Server 只在启动时实例化一次图；那一刻 PD 客户端连不上的话， +这个 Pod 会终身通过就绪探测、正常提供 REST，而每个发到它的 Gremlin 请求都报 `Could not rebind [graph]` +（[apache/hugegraph#3228](https://github.com/apache/hugegraph/issues/3228)）。它的 `hugegraph-server.log` 会写明： + +``` +Graph [DEFAULT-hugegraph] configured at [...] could not be instantiated and +will not be available in Gremlin Server +``` + +任何同时滚动了 PD 和 Server 的升级或回滚之后，运行 +`helm test hugegraph --namespace hugegraph --logs --timeout 5m`：它通过 headless Service `hugegraph-server-headless` 向每个 +Ready 的 Server Pod 发送下面这条绑定图的 Gremlin 查询，最多重试 150 秒，遇到处于这种状态的 Pod 会以 +`Gremlin failed on ` 失败。手工检查单个 Pod 时，在一个终端里对它做 port-forward（forward 在前台运行）： + +```bash +kubectl port-forward -n hugegraph pod/ 8080:8080 +``` + +再在第二个终端里发送查询： + +```bash +curl -s --compressed -u "admin:${PASSWORD}" -H 'Content-Type: application/json' \ + -X POST http://127.0.0.1:8080/gremlin \ + -d '{"gremlin":"graph.traversal().V().limit(1).count()","aliases":{"graph":"DEFAULT-hugegraph"}}' +``` + +健康的 Pod 返回 `result.data`。PD 稳定后，删除受影响的 Pod，并重新运行测试 hook 验证替换副本。 +若启用了 Hubble，还需验证新会话登录和图查询。 diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md new file mode 100644 index 0000000000..a051724234 --- /dev/null +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm.md @@ -0,0 +1,316 @@ +--- +title: "使用 Helm 在 Kubernetes 上部署" +linkTitle: "Kubernetes 部署 (Helm)" +weight: 4 +search_keywords: + - helm + - kubernetes + - k8s +--- + +### 1 概述 + +Helm chart 在 Kubernetes 上部署一套分布式 HugeGraph 集群：PD、Store、Server，以及可选的 Hubble UI。chart 位于主仓库的 +[`helm/hugegraph`](https://github.com/apache/hugegraph/tree/master/helm/hugegraph) 目录。 + +| 组件 | 工作负载 | 默认副本数 | 作用 | +|------|----------|------------|------| +| PD | StatefulSet + PVC | 3 | 元数据管理：以 Raft 组跟踪 Store 与分区 | +| Store | StatefulSet + PVC | 3 | 图数据存储 (HStore) | +| Server | Deployment | 3 | Gremlin 与 REST 查询层 | +| Hubble | Deployment | 0（默认关闭） | Web UI，通过 `hubble.enabled=true` 启用 | + +分布式 HugeGraph 集群有一套启动约定（Server 不执行 `init-store`、每个 Server 都通过 PD 读写图元数据、Store 等待 PD +形成多数派、一个 PD REST 密钥由三个组件共用）。chart 把这套约定固化下来，运维人员无需手工处理；细节见 +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#chart-details)。日常运维（NetworkPolicy +细节、灾难恢复、伸缩、安全滚动 Store、在集群外运行 Hubble）见 +[运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/)。 + +```mermaid +flowchart LR + subgraph Kubernetes 集群 + PD[PD StatefulSet
3 节点 Raft 组，PVC] + Store[Store StatefulSet
3 副本，PVC] -->|注册、心跳| PD + Server[Server Deployment
3 副本] -->|元数据、服务发现| PD + Server -->|gRPC 读写| Store + Hubble[Hubble Deployment
可选 UI] -->|通过 PD 发现 Server| PD + end + Client[客户端 / hugegraph-client] -->|REST / Gremlin| Server +``` + +启动顺序由 chart 保证，而不是由运维人员保证：PD 先选出 leader，每个 Store Pod 的 init 容器等到多数 PD 汇报就绪后才启动， +Server 则反复等待存储层，直到 Store 完成注册。全新安装无需手工干预即可收敛。 + +### 2 前置条件 + +- Kubernetes 1.23 及以上（chart 渲染 `autoscaling/v2` 和 `policy/v1`） +- Helm 3；升级一节提到的 `--reset-then-reuse-values` 需要 Helm 3.14 及以上 +- 动态卷供给：有默认 StorageClass，或为 PD 和 Store 显式指定 `storageClassName` +- 默认拓扑要运行九个 JVM 进程，内存需留足；见安装一节的资源说明 + +chart 要求组件镜像包含 PD 就绪探测端点和 PD REST 认证（两者都已合入 1.7.0 之后的版本）。默认镜像 tag 指向的构建已包含 +这些改动；1.7.0 镜像不受支持。 + +### 3 安装 + +#### 3.1 获取 chart + +chart 尚未发布到 chart 仓库，从源码树安装： + +```bash +git clone https://github.com/apache/hugegraph.git +cd hugegraph +``` + +#### 3.2 使用默认值安装 + +先确认 `kubectl` 指向目标集群，且集群能供给存储卷。因缺少 StorageClass 而卡在 `Pending` 的 PVC 是最常见的首次安装 +故障： + +```bash +kubectl config current-context +kubectl get storageclass +``` + +然后安装： + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace --wait --timeout 15m +``` + +`--wait` 等待 Kubernetes 就绪探测通过，不代表业务流程可用。全新集群一般几分钟内收敛，15 分钟超时为缓慢的镜像拉取留出余量。 +安装后仍需验证图查询；不加 `--wait`，Helm 在对象创建完成后就返回。 + +继续之前需要了解两个默认值： + +- **默认不设置 resources。** 每个 Pod 都是 BestEffort，每个 JVM 在启动时把堆上限设为它看到的节点空闲内存的一半（各组件另有上限）， + 因此同一节点上的几个 JVM 合起来可能占用超过节点总量的内存。单节点上的 `values-single.yaml` 也是如此。每次安装都请 + 按组件设置 `resources`，单节点预设也不例外；`values-cluster.yaml` 为多节点集群设置了这些值。 +- **镜像 tag 跟踪 `latest`**，且 `pullPolicy: Always`，直到下一个 HugeGraph 版本发布。生产环境请固定 tag 或 digest。 + +#### 3.3 拓扑预设 + +chart 附带三个 values 文件： + +| 文件 | 拓扑 | 适用场景 | +|------|------|----------| +| `values.yaml` | 3 PD + 3 Store + 3 Server | 默认；preferred 反亲和，认证开启，Hubble 关闭 | +| `values-single.yaml` | 1 + 1 + 1 | 单节点开发与 CI；PVC 更小 | +| `values-cluster.yaml` | 3 + 3 + 3 | 生产起点：JVM 堆与资源设置、Server PodDisruptionBudget、`required` 反亲和、NetworkPolicy 开启、Store 使用 `OnDelete` 更新 | + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace \ + -f helm/hugegraph/values-single.yaml --wait --timeout 15m +``` + +`values-cluster.yaml` 是起点而非容量保证：请按图规模和流量重新核算资源。其中一个数字值得说明：它给每个 Store 申请 +5Gi、限制 8Gi 内存，远高于 1Gi 的堆，因为 Store 的 RocksDB 缓存在 JVM 堆之外（4Gi 的限制在约 1 GB 数据后就把 +Store OOM 杀掉了）。两个数字要随数据量一起放大。完整参数参考（每个组件的探针、调度、Secret 配置项）见 +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration)。 + +#### 3.4 验证安装 + +```bash +helm test hugegraph --namespace hugegraph --logs --timeout 5m +``` + +测试先经 Server Service 调用 `/versions` 和 `/graphs`，再解析 headless Service `hugegraph-server-headless`，向它列出的 +每个 Server Pod 发送一条带认证、绑定 `DEFAULT-hugegraph` 图的 Gremlin 查询，并要求至少有 `server.replicas` 个 Pod +（开启 HPA 时为 `server.hpa.minReplicas`）。headless Service 只列出 Ready 的 Pod，也就是 Server Service 实际转发流量的 +那些 Pod，未 Ready 的 Pod 不会被查询。失败的 Pod 每 5 秒重试一次，最多 150 秒，短暂的 PD 选举因此不会被误判为 Pod +损坏；超时后测试失败，并打印每个失败 Pod 的 IP 和 HTTP 状态码。能通过就绪探测、能提供 REST、但每个 Gremlin 调用都 +失败的 Server，正是靠这一步发现的（见"限制"）。HPA 规模较大时，`helm test --timeout` 可能需要高于默认的 5 分钟。 + +测试 hook 验证 Server Gremlin，不验证 Hubble 登录。启用 Hubble 时，安装或恢复后还需验证新会话登录和图查询； +仅 `/actuator/health` 通过不能证明后端连接可用。 + +调用 API 前，先在一个终端里启动 port-forward；它会一直在前台运行，直到你停止它： + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +``` + +然后在第二个终端里读取自动生成的 admin 密码并调用 API： + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin -o jsonpath='{.data.password}' | base64 --decode)" +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +``` + +以上命令假定 release 名为 `hugegraph`；用其他名字时，请替换成带 release 前缀的资源名（`kubectl get svc,secret -n +` 可以列出）。`helm install` 结束时打印的说明里包含填好名字的同样命令。 + +### 4 认证与 Secret + +认证默认开启，chart 管理三个 Secret。每个凭据按同一顺序取值：你预先创建的 `existingSecret` 优先，其次是内联值，最后 +是安装时随机生成。 + +| Secret | 键 | 用途 | 自带凭据的配置项 | +|--------|-----|------|------------------| +| `-admin` | `password` | Server admin 账号、Hubble 登录 | `server.auth.admin.existingSecret` | +| `-auth-token` | `token_secret` | 所有 Server 副本共用的 JWT 签名密钥 | `server.auth.token.existingSecret` | +| `-pd-auth` | `secret-key` | PD REST 认证，由 PD、Server、Hubble 读取 | `pd.auth.existingSecret` | + +要自行管理凭据，请在安装前创建 Secret 并把对应的 `existingSecret` 指向它；chart 不会改动任何不是它创建的 Secret。取值 +约束：admin 密码必须是不含空格、冒号和反斜杠的可打印 ASCII（镜像会在空格前写入反斜杠，Server 又按每个冒号拆分 +Basic 认证凭据，两者都会让账号无法用 Secret 中的值登录）；JWT 密钥至少 32 字节；PD 密钥必须是不含反斜杠、 +首尾没有空格的可打印 ASCII。非法的内联值在渲染时被拒绝，`existingSecret` 的值在 Pod 启动时被启动包装脚本拒绝， +不会被悄悄截断。 + +chart 管理的 Secret 在卸载时保留，同名 release 再次安装会复用它们。 + +
+轮换与注意事项 + +- admin 密码只在认证元数据首次创建时生效，之后修改 Secret 不会轮换已有集群的密码。请改用 Server 的 auth API 轮换。 +- 轮换凭据会在下一次 `helm upgrade` 时让读取它的 Pod 滚动一次：PD REST Secret 同时滚动 PD、Server 和 Hubble，保证三者 + 持有的副本一致；admin 密码或 JWT 密钥只滚动 Server。 +- 纯模板流水线（`helm template`、Argo CD、模板模式的 Flux）看不到集群里的 Secret，因此 chart 生成的凭据每次渲染都会得到 + 新值：每次同步都会改动 Secret、滚动读取它的 Pod，并让首次启动时创建的 admin 账号与 Secret 不再一致。这类流水线请先 + 创建三个 Secret 并设置对应的 `existingSecret`；见 chart README 的 + [Template-only pipelines](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#template-only-pipelines-gitops)。 +- 三个 Secret 即使从不读取也都存在：安装结束打印的说明里有读取 admin 密码和 PD 密钥的 `kubectl get secret` 命令。 +
+ +### 5 健康检查与启动顺序 + +PD 暴露两个健康端点，chart 有意同时使用两者： + +- `/v1/health` 在 REST 监听建立后立刻返回 200。它不查询 Raft，因此感知不到多数派丢失。 +- `/v1/ready` 在 PD Raft 组选出 leader 之前返回 503，反映的是多数派状态而不只是进程存活。 + +chart 把 PD 的**就绪探测**和 Store init 容器的等待放在 `/v1/ready` 上：Store 只有在多数 PD 成为多数派成员后才启动， +失去 leader 的 PD 会退出 Service 端点，直到 leader 恢复。PD 的**启动和存活探测**按副本数推导（`pd.livenessPath` +可覆盖）。多副本时特意留在 `/v1/health` 上：只是失去 leader 的 PD 仍是健康的 Raft 成员，重启它只会让故障恶化。 +单副本 PD 是例外，推导为 `/v1/ready`：它没有选举可失去，而一个永久退位的 PD（例如磁盘写满导致 Raft 快照失败之后， +[apache/hugegraph#3222](https://github.com/apache/hugegraph/issues/3222)）会一直用 `/v1/health` 返回 200 却不再 +服务写入；改用 `/v1/ready` 让 kubelet 把它重启。 + +Server 启动获得至少 450 秒的预算，足够覆盖镜像入口脚本在启动命令之前执行的 300 秒存储等待和进程启动。chart 按 +`(failureThreshold - 1) * periodSeconds` 计算有保证的探测时间，因为 kubelet 可能在容器启动后立刻执行第一次探测； +配置的 `failureThreshold` 低于该下限时会被抬高（默认 5 秒周期下为 91，也就是默认值）。镜像默认会在 120 秒后杀掉 +仍在启动的 Server，chart 因此把 `HG_SERVER_STARTUP_TIMEOUT_S` 设为有保证的探测时间减去 300 秒存储等待：默认 150 秒， +不低于镜像的 120 秒。这样启动命令与探针同时放弃。如果存储层启动更慢，调大 `server.probes.startup`，超时会跟着变； +该变量由 chart 管理，请修改探针，不要在 `server.extraEnv` 里设置它。 + +### 6 启用 Hubble UI + +Hubble 默认关闭，纯 API 集群因此更精简。在运行中的 release 上启用： + +```bash +helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values \ + --set hubble.enabled=true --wait --timeout 15m +``` + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-hubble 8088:8088 +``` + +打开 `http://127.0.0.1:8088`，用 3.4 节的 admin 密码以 `admin` 身份登录。Hubble 通过 PD 发现 Server，集群运维视图无需 +额外配置即可工作。Hubble 只提供明文 HTTP：请通过 port-forward 或做 HTTPS 终结的 Ingress 访问，绝不要直接暴露在不可信 +网络上；NodePort 或 LoadBalancer 类型的 Hubble Service 需要设置 `hubble.service.allowInsecureExposure=true` 确认后才会 +渲染。在集群外运行 Hubble 也可行，但配置更多；见 chart README 的 +[Reaching Hubble](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#reaching-hubble-pick-one-path)。 + +### 7 升级 + +```bash +helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values +``` + +`--reuse-values` 保留 release 的既有覆盖值；不加它，升级会以 chart 默认值重建 release。它还会把旧 values 当作完整的基准， +因此由早期 chart 版本创建的 release 不会获得新的默认值（例如加固后的 `securityContext`）；要采用它们，请用 `-f` 传入 +自己的 values，或使用 `--reset-then-reuse-values`。任何改变 Pod 模板的升级都会让 +对应工作负载滚动一次。需要提前规划的几点： + +- **没有变更的升级不会滚动任何工作负载。** 跟踪 Secret 的注解对每个凭据的值做哈希（`existingSecret` 则取其线上 + `resourceVersion`），只有 Pod 读取的凭据变化时它才会滚动。确实同时滚动 PD 和 Server 的升级（例如轮换 PD REST Secret， + 或同时更换两者的镜像）可能让 Server 丢失 Gremlin 绑定（见"限制"）。应将 `pd.auth` 轮换视为联合维护，而非无人干预的常规变更： + 完成后运行 `helm test --logs`，并按[运维流程](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/#10-gremlin-报-could-not-rebind-时) + 恢复受影响副本。回滚若同时滚动两者，也需要相同验证。 +- **Store 的滚动更新以监听检查推进，而不是以分片恢复推进**，因此控制器可能在上一个 Store 尚未重新加入分片组时就替换 + 下一个。`values-cluster.yaml` 因此设置了 `store.updateStrategy.type=OnDelete`；`values.yaml` 和 `values-single.yaml` + 仍为 `RollingUpdate`，其他生产 values 请自行设置。`OnDelete` 下升级只更新 StatefulSet，不替换任何 Store Pod，由你逐个 + 删除 Store Pod。`OnDelete` 只是停止自动推进：删除 Pod 时不在两次删除之间做检查，风险相同。PD 里的 `Up` 不是这个检查： + PD 在注册时就把 Store 标为 `Up`，此时分区尚未恢复，已停止的 Store 也要等 300 秒 keep-alive 过期才离开 `Up`。当前 + 镜像上没有任何端点报告 Store 已完成分区恢复（[apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229)）， + 因此只能间接检查。应等被替换的 Pod 变为 `Ready`，再确认每个分片组都报告完整分片数和一个 leader；完整流程见 + [运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/#6-安全地滚动-store-镜像)。 +- **控制器每次最多滚动一个 PD 或 Store Pod。** chart 不设置 `updateStrategy.rollingUpdate.maxUnavailable`，且该字段只接受 + 整数 `1`：更大的数字或百分比会导致渲染失败，因为它会让控制器同时停掉三成员 Raft 组或分片中的两个成员，而 + PodDisruptionBudget 不约束控制器发起的滚动。PD 的维护窗口可用 `pd.updateStrategy.type=OnDelete` 获得与 Store 相同的 + 手工控制。 +- **升级不能修改 PVC 大小**：Kubernetes 禁止修改 StatefulSet 的 `volumeClaimTemplates`，带新 `storage.size` 的升级会 + 被整体拒绝。chart README 记录了支持卷扩容的 StorageClass 上的扩容步骤。 +- **部分值在 release 初始化后就固定。** `nameOverride`、`fullnameOverride`、PD 与 Store 的 raft 端口以及存储设置属于 + 安装时身份：chart 会对照线上 StatefulSet 拒绝修改 override 或 raft 端口，Kubernetes 拒绝修改存储设置。 + `server.auth.admin.*` 和分区分片数只在初始化时生效，已初始化的集群会忽略新值。chart README 按 + [生命周期](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#settings-by-lifecycle)对各个值分了类。 +- **`helm rollback` 不运行 chart 的任何检查。** 回滚（包括 `--atomic` 升级失败后的自动回滚）直接重新应用之前保存的 + 清单，不渲染 chart，因此跨越 PD 或 Store 副本数变更的回滚会把 StatefulSet 直接缩放到旧数量，可能让 PD 失去多数派。 + 不支持跨越成员关系或身份变更的回滚：要回到更早的 chart 或镜像，请用保留当前拓扑 values 的正向升级，且改变副本数的 + 升级不要加 `--atomic`。任何回滚之后要做的检查见 chart README 的 + [Rollback](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#rollback)。 + +Server 的扩缩容是普通的 values 变更（`server.replicas` 或 `server.hpa`）。**双向改变 PD 数量、或缩容 Store 都不是**： +Raft 与分片成员关系是持久化的，Pod 本身不会重新配置它们，因此 PD 从 3 缩到 1 会永久失去多数派，新增的 +PD Pod 也不会加入投票配置。chart 读取线上 StatefulSet，拒绝改变 PD 数量或降低 Store 数量的升级；手工步骤见 +[运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/#8-伸缩)。 + +### 8 卸载 + +```bash +helm uninstall hugegraph --namespace hugegraph +``` + +有两类状态是有意保留的。StatefulSet 创建的 PersistentVolumeClaim 会保留（Kubernetes 行为），确认数据不再需要后请显式 +删除。chart 管理的 Secret 也会保留，同名 release 再次安装时凭据不变。 + +### 9 限制 + +- `networkPolicy.enabled` 为每个组件渲染只放行本 release 流量的 NetworkPolicy；默认关闭，`values-cluster.yaml` + 中开启。它之所以重要，是因为 chart 在集群内关闭了 PD 的 Raft IP 白名单（Pod IP 会变化；白名单只在启动时解析一次 + 对端，之后就会拦截它们），raft 与 gRPC 端口的访问限制由这些策略承担。策略需要强制执行 NetworkPolicy 的网络插件 + （Calico、Cilium、kind v0.25 及以上、k3s），所有外部调用方（包括 Ingress 控制器）都必须列入 + `networkPolicy..extraIngress`，否则渲染失败。细节见 + [运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/)。 +- 镜像 tag 跟踪 `latest`，直到下一个 HugeGraph 版本发布带版本号的镜像；长期运行的环境请固定 tag 或 digest。 +- 创建图之后，其他 Server 副本在短暂窗口内可能尚未收敛，路由到这类副本的查询可能报 `Could not rebind [g]` 之类的 + 错误。请带退避重试，或对"创建后立即查询"的流程使用会话粘滞路由；集群级图就绪在 + [#3137](https://github.com/apache/hugegraph/issues/3137) 跟踪。 +- 在 PD 不可达时（例如 PD 滚动期间）启动的 Server，可能在整个 Pod 生命周期内都没有 Gremlin 绑定：它通过就绪探测、 + 正常提供 REST，而每个发到它的 Gremlin 请求都报 `Could not rebind [graph]` + （[apache/hugegraph#3228](https://github.com/apache/hugegraph/issues/3228)）。就绪探测调用的是 `/versions`，看不到 + 这种状态。`helm test` 会在每个 Ready 的 Server Pod 上查询 Gremlin，并打印处于这种状态的 Pod 的 IP；删除该 Pod，PD 稳定后 + 替换者会正常绑定。见 + [Gremlin 报 "Could not rebind" 时](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/#10-gremlin-报-could-not-rebind-时)。 +- 启动时打不开自己 RocksDB 存储的 PD（例如另一个进程仍持有存储的 `LOCK` 文件）既不重试也不退出：它只记录一次 + `Failed to open RocksDB`，之后继续运行，`/v1/ready` 返回 503 和 `STATE_UNINITIALIZED`，而 `/v1/health` 返回 200。 + 这是在 Kubernetes 之外的 PD 进程上观察到的（[apache/hugegraph#3226](https://github.com/apache/hugegraph/issues/3226)）。 + 在 chart 的探针下，这样的 Pod 不会 Ready，并退出 Service 端点。多副本 PD 的启动和存活探测使用 `/v1/health`，kubelet + 不会重启它；单副本 PD 使用 `/v1/ready`，kubelet 会重启容器。这种状态在 Kubernetes 上如何出现，以及重启或删除 Pod + 能否消除它，都尚未测试。删除 `LOCK` 文件不是解决办法：它们保护存储不被可能仍在运行的另一个进程同时打开。 +- 当前版本的 Store 恢复由运维人员触发：Store 丢失后的副本重建、leader 均衡、分区再均衡都只在调用 PD 的 REST API 时 + 执行。丢失了卷的 Store 只有在携带 + [apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234)（2026-09-24 合入，尚未进入任何发布版本） + 的镜像上才能原地恢复；更早的镜像上，替换 Store Pod 时请保留它的 PVC。操作手册见 + [运维页](/cn/docs/quickstart/hugegraph/hugegraph-helm-operations/)的灾难恢复一节。 +- Server 和 Hubble 只提供明文 HTTP，PD gRPC 没有认证。它们中任何一个的 NodePort 或 LoadBalancer Service，都要设置对应的 + `pd.service.allowInsecureExposure`、`server.service.allowInsecureExposure` 或 `hubble.service.allowInsecureExposure` + 才会渲染；不带 `tls` 的 Server 或 Hubble Ingress 也要设置各自的 `allowPlainHttp`。优先使用 port-forward 或做 HTTPS + 终结的 Ingress，暴露 Service 前先限制谁能访问它。 +- 集群内无 TLS 终结，无备份工具，无 Operator，无内置监控栈。 + +### 10 排障 + +| 现象 | 先查什么 | +|------|----------| +| Store Pod 卡在 `Init:0/1` | PD 未就绪：`kubectl logs -c wait-for-pd`，再看 PD Pod | +| PVC 停在 `Pending` | 没有默认 StorageClass，或供给器故障：`kubectl get sc` | +| Pod 被 OOM 杀掉或反复重启 | 未设置 resources，JVM 按节点内存取堆：设置 `resources`（见"安装"） | +| 建图后立刻查询失败 | 副本收敛窗口：见上文"限制" | +| `helm test` 打印 `Gremlin failed on ` | 该 Server Pod 丢失了 Gremlin 绑定：删除它（见上文"限制"） | +| 某个 PD Pod 一直不 Ready，其 `/v1/ready` 报 `STATE_UNINITIALIZED` | 它的 RocksDB 存储可能没有打开：在日志中查找 `Failed to open RocksDB`（见上文"限制"） | + +其中多数情况的完整排查步骤见 +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#troubleshooting)。 diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md new file mode 100644 index 0000000000..6c8bba8f93 --- /dev/null +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -0,0 +1,523 @@ +--- +title: "Operate HugeGraph on Kubernetes" +linkTitle: "Operations on Kubernetes (Helm)" +weight: 5 +search_keywords: + - helm + - kubernetes + - operations + - disaster recovery + - networkpolicy +--- + +### 1 Scope + +This page is for operating a HugeGraph cluster that the Helm chart already +deployed. Installing, upgrading, and uninstalling are on the +[deployment page](/docs/quickstart/hugegraph/hugegraph-helm/); the full +parameter reference stays in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration). + +Commands below assume the release is named `hugegraph` in namespace +`hugegraph`; substitute your own names. Two credentials come up repeatedly, +both read from chart-managed Secrets: + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin \ + -o jsonpath='{.data.password}' | base64 --decode)" +PD_SECRET="$(kubectl get secret -n hugegraph hugegraph-pd-auth \ + -o jsonpath='{.data.secret-key}' | base64 --decode)" +``` + +### 2 Ports and health + +| Component | Port | Purpose | +|------|-------------|---------| +| PD | `8686` | gRPC (Store and Server clients) | +| PD | `8620` | REST / health probes | +| PD | `8610` | Raft | +| Store | `8500` | gRPC | +| Store | `8510` | Raft | +| Store | `8520` | REST / health probes | +| Server | `8080` | Gremlin and REST API | + +All ports are configurable through values; changing `server.port` updates the +listener, container port, and Service together. A stalled component (process +alive but frozen) is ended by its liveness probe: the default 20 s period and +3-failure threshold bound the blast radius of a stalled Store at roughly one +minute, and raft moves its partition leaders within seconds of the restart. + +Reach the API through a port-forward. It runs in the foreground, so keep +it in one terminal: + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +``` + +and call the API from a second terminal, with `PASSWORD` set as above: + +```bash +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/graphs +``` + +### 3 Scheduling + +Every component (`pd`, `store`, `server`, `hubble`) exposes `nodeSelector`, +`tolerations`, `affinity`, `topologySpreadConstraints`, and +`priorityClassName`. Pinning Store to labeled nodes is just: + +```yaml +store: + nodeSelector: + hugegraph/role: storage +``` + +`antiAffinity` (`required` | `preferred` | `disabled`) renders a hostname +pod-anti-affinity preset for `pd`, `store`, and `server`; Hubble is +single-replica by design and has no such key. Setting a raw `affinity` +replaces the preset entirely. All three default to `preferred`, so the chart +schedules on clusters with fewer nodes than replicas. The trade: under node +pressure the scheduler may co-locate replicas, and a single node failure can +then take more than one PD or Store replica with it. Production clusters with +enough nodes should pin `pd.antiAffinity` and `store.antiAffinity` to +`required`, as `values-cluster.yaml` does. + +### 4 Partition sharding + +A fresh install seeds PD with a partition shard count of 3 when +`store.replicas` is at least 3, and 1 otherwise; without the seed, the PD +image pins `partition.default-shard-count` to 1 and chart-deployed clusters +would run without store-level HA. The chart passes the value as +`-Dpartition.default-shard-count` in PD's `JAVA_OPTS`, so the image's JVM +auto-sizing is unaffected. + +**The seed applies at first bootstrap only.** PD persists the shard count the +first time it starts with empty storage; from then on the stored value is +authoritative, and changing `pd.partition.defaultShardCount` or scaling +`store.replicas` later has no effect on an initialized cluster. To change +the shard count of a running cluster, use PD's own config API (odd values +only, at most the live store count), then trigger +`GET /v1/task/patrolPartitions` and expect shard-group reallocation. + +The shard count also fixes the initial partition count: +`store.replicas x storeMaxShardCount / shardCount`, computed once at +bootstrap. With the image default `store-max-shard-count` of 12, a default +3-store install gets 12 partitions; raise +`pd.partition.storeMaxShardCount` (likewise seeded once) when more +partitions are wanted. + +An explicit `pd.partition.defaultShardCount` must be odd and at most +`store.replicas`; the chart rejects other values at render time because PD +would otherwise clamp or refuse them silently. + +### 5 NetworkPolicy + +`networkPolicy.enabled` renders one NetworkPolicy per component (PD, Store, +Server, and Hubble when enabled). Each policy isolates its own Pods in both +directions and lists the traffic they need, so apply order does not matter. +It is off in `values.yaml`, because the base values cannot know who your +clients are, and on in `values-cluster.yaml`. + +It only works when the cluster's network plugin enforces NetworkPolicy (kind +v0.25 or later, k3s, Calico, Cilium); other plugins accept the objects and +enforce nothing. To check, run a Pod without chart labels in another +namespace and `curl` the PD client Service on the REST port: it must time +out. + +With the policies on, the release admits only its own traffic: + +| To | From, ports | +|---|---| +| PD | PD: raft, gRPC. Store, Server, and Hubble in `pd` mode: gRPC, REST | +| Store | Store: raft. Server: gRPC, REST. Hubble in `pd` mode: REST | +| Server | Hubble and the `helm test` Pod: `server.port` | +| Hubble | nothing (port-forward uses loopback and needs no rule) | + +Every component may also resolve DNS on port 53. Nothing outside the release +is admitted unless it is listed in `networkPolicy..extraIngress`, +including the Ingress controller and the clients of a NodePort or +LoadBalancer Service. Exposing PD, Server or Hubble that way, or setting +`server.advertiseUrl`, with an empty `extraIngress` fails the render instead +of opening the port. The check sees only exposure the chart creates; a +Service, Gateway route or proxy you add yourself needs its own entry. An +in-cluster Hubble in `pd` mode combined with `server.advertiseUrl` also +fails the render until `networkPolicy.hubble.extraEgress` admits the +advertised destination: PD hands Hubble that URL for discovery, and +Hubble's egress policy otherwise reaches only the release's own Pods. + +Two egress facts to plan around. PD, Store and Server reach nothing outside +the release except DNS, so features that call out do not work with the +policies on; the one default caller is the Store image, which downloads +`libjemalloc.so` from github.com on every start. With the policies on that +connection times out after about two minutes and the Store starts without +jemalloc (measured on kind: Ready after 151 s instead of 11 s); the same +happens on any cluster without internet access. And the `helm test` Pod is +selected by no chart policy, so under a namespace-wide default-deny policy +of your own you must allow it egress to `server.port` and DNS. + +**Letting other workloads in.** Every rule must name its peers in `from`; +to admit any address, write an `ipBlock` such as `0.0.0.0/0` explicitly. A +`namespaceSelector` and a `podSelector` in the same peer must both match; +as two separate peers, either one is enough. What a NodePort or +LoadBalancer client looks like from the Pod depends on the network plugin, +`externalTrafficPolicy`, and the node the request arrives on. Measured with +a NodePort Server on two-node kind clusters: + +- kindnet, and Cilium with kube-proxy replacement: a call to the Server's + own node arrived with the client address; through the other node it + arrived with that node's address. +- Calico: through the other node the call arrived from that node's tunnel + address inside the Pod CIDR. +- Cilium with kube-proxy: no `ipBlock` rule admitted NodePort traffic, + because Cilium identifies node addresses by its own node identities + rather than by CIDR. + +Test with the plugin you run and name the CIDR you see arriving. A worked +example admitting an Ingress controller, an application namespace, and +Prometheus: + +```yaml +networkPolicy: + server: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: ingress-nginx + podSelector: + matchLabels: + app.kubernetes.io/name: ingress-nginx + ports: + - port: 8080 + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: apps + ports: + - port: 8080 + pd: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: monitoring + ports: + - port: 8620 +``` + +### 6 Rolling Store images safely + +Store rolling updates advance on a listener check, not on shard recovery, +so the controller can replace the next Store while the previous one is +still rejoining its shard groups. `values-cluster.yaml` therefore sets +`store.updateStrategy.type=OnDelete` (set it yourself on any other +production values): an upgrade then updates the StatefulSet without +replacing any Store Pod, and you delete Store Pods one at a time, checking +between deletions. `OnDelete` only stops automatic advancement; deleting +Pods without the checks below carries the same risk. + +`Up` in PD is not that check. PD marks a Store `Up` at registration, before +the Store has restored any partition, and a stopped Store stays `Up` in +every shard group until its keep-alive entry expires (300 s on current +images): a Pod deleted and back inside that window never leaves `Up` at +all. Start with the Pod instead: + +```bash +kubectl -n hugegraph wait --for=condition=Ready \ + pod/hugegraph-store- --timeout=10m +``` + +Then check shard membership and leadership per group, read from the PD +leader (see Disaster Recovery below for finding the leader): + +```bash +curl -s -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/shardGroups | jq ' + .shardGroups[] | {id: (.id // 0), + shards: [.shards[] | {storeId, role}], + leaders: [.shards[] | select(.role=="Leader")] | length}' +``` + +Delete the next Store only when the replaced Pod is `Ready`, its Store id +shows a fresh `lastHeartBeat` in `/v1/stores`, and every group reports its +full shard count with exactly one `Leader`. + +Know what this does not prove: the shard list is PD's membership record, +not a statement that the Store caught up on the raft log. No endpoint on +current images reports restoration-complete. For a closer look, +port-forward the replaced Store and read its own view of a group: +`GET :8520/v1/partition/` returns the raft role, leader, and +committed index (`logIndex`) that Store holds, and fails while the Store +is down; compare the index with a peer Store rather than reading it +alone. This route never fills its `term` field, so `term` always reads 0 +there. The plural `GET :8520/v1/partitions` answers 500 on any Store that +follows a group on images built before +[apache/hugegraph#3232](https://github.com/apache/hugegraph/pull/3232) +(merged 2026-09-24); after it, it answers 200 on every Store, with `conf` +and `peers` null for followed groups. The per-group path works on both. + +Leave a margin after the membership check, keep `store.pdb.minAvailable` +at `replicas - 1` so an accidental second eviction is refused, and treat a +group that is short a shard or has no leader as a stop. A real +restoration-complete signal is upstream work, tracked in +[apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229). + +### 7 Disaster recovery + +What PD automates on current builds is narrow: a 60-second patrol only +marks Stores that stopped sending heartbeats as `Offline`. There is no +automatic re-replication; re-placing the replicas of a lost Store, +reconciling shard groups, and processing tombstoned Stores run only when a +partition patrol is triggered explicitly. + +The task endpoints execute locally on the PD that receives them, and a +follower answers with an empty success while doing nothing. +Port-forwarding the client Service selects an arbitrary PD, so identify +the leader first, then port-forward that Pod (two terminals: the forward +runs in the foreground): + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-pd-client 8620:8620 +# Read .data.pdLeader.raftUrl; its host names the leader Pod. +curl -su "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/members +# Stop the Service forward, then forward the leader Pod instead. +kubectl port-forward -n hugegraph pod/ 8620:8620 +# Reconcile shard groups and process tombstoned Stores. +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/patrolPartitions +# If the patrol repaired a shard group, wait 180 s first (see below). +# Spread Raft leaders, then partition data. +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balanceLeaders +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balancePartitions +``` + +Read `/v1/members` again after the tasks: if leadership moved mid-sequence, +the later tasks ran on a follower and did nothing. PD sets a +balance-shard flag for 180 s in two cases: `patrolPartitions` sets it +whenever it reallocates a shard group whose shard count is wrong, and +`balancePartitions` sets it even when it moves nothing. Inside that +window `balanceLeaders` is refused and `balancePartitions` returns +without doing anything, so wait at least 180 s after a patrol that +repaired a group, and after `balancePartitions`, before the next +balancing call. On images built before +[apache/hugegraph#3233](https://github.com/apache/hugegraph/pull/3233) +(merged 2026-09-24) the refusal is a bare HTTP 500 whose reason appears +only in the PD log; after it, the reason comes back in the body as +`{"status":1001,"error":"balance shard is processing, please try later!"}`. + +Telling a real run from a no-op takes the PD leader's log, because the +responses do not: `patrolPartitions` answers the same empty success on the +leader and on a follower, whether or not it repaired anything (look for +`reallocShards`, `shardOffline` or `storeTurnoff` lines, or diff +`/v1/shardGroups` before and after), and `balancePartitions` answers `{}` +on the leader and an empty body on a follower. `balanceLeaders` is the one +call whose body carries the work. Distinguishable task responses are +upstream work, tracked in +[apache/hugegraph#3231](https://github.com/apache/hugegraph/issues/3231). + +Run `patrolPartitions` after replacing a Store that is not coming back, +`balancePartitions` once the cluster is stable again, and `balanceLeaders` +after restarts that skewed leader placement. + +**Losing a Store volume.** A Store rebuilt with an empty PVC recovers in +place on images carrying +[apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234) +(merged 2026-09-24); on every earlier image, including all published +release images, it does not. In both cases the replacement registers under +a new Store ID while its Pod name, DNS name, and raft address are +unchanged, so `/v1/stores` lists two IDs at one address. + +On post-#3234 images the retirement works: find the old ID in `/v1/stores` +(the row at the replaced Pod's address that is not the newly registered +one), `POST /v1/store/` with `{"storeState":"Tombstone"}` on the PD +leader, run `GET /v1/task/patrolPartitions`, and wait; verify every shard +group is back to full shard count with one leader, no group names the old +ID, and the replaced Store's own `:8520/v1/partition/` answers +200 for every group; then `DELETE /v1/store/` to erase the retired +record. Measured on a 3+3+3 install with images built from `master` at +`dbb6663a`: the replacement was Ready in 156 s, all 12 groups converged +onto the new ID 1 s after the Tombstone and patrol (the empty Store caught +up by raft snapshot install), and a continuous writer lost 0 acknowledged +writes. + +On images without #3234 the same retirement runs and does not repair the +groups: PD fires the configuration change, but the group leader sees the +address already in the group, so jraft has nothing to add and the group +keeps the old ID. Measured: 20 minutes and three patrols later, all 12 +groups still listed the retired ID and the replacement Store held no +partitions. Nothing in the health surface shows the failure; `/v1/stores`, +cluster state, Hubble, and Pod readiness all read healthy while every +shard group runs on two live replicas. The one check that shows it is the +replaced Store's own `:8520/v1/partition/`. + +So the default remains: replace a Store Pod, keep its PVC (the Store id +lives in the data path, and the Pod comes back under the same id). Treat +the empty-PVC procedure as recovery for post-#3234 images only; on a +released image a genuinely lost volume leaves the cluster degraded, and +expect to rebuild rather than to recover in place. + +Periodic leader balancing is tracked in +[apache/hugegraph#3135](https://github.com/apache/hugegraph/issues/3135); +disaster-recovery metrics in +[apache/hugegraph#3136](https://github.com/apache/hugegraph/issues/3136). + +### 8 Scaling + +PD and Store reserve the maximum StatefulSet ordinal in their resource +names, so scaling never renames a PersistentVolumeClaim or shifts a Pod +identity; both are capped at 99 replicas. Server scales freely through +`server.replicas` or `server.hpa` (with HPA on, the Deployment omits +`spec.replicas` so upgrades do not overwrite the autoscaler). + +A staged rollout cannot be written in a values file (the schema requires +one replica per component); stage it with `kubectl scale statefulset +hugegraph-store --replicas=0` and scale back up when ready. The Servers +wait, not ready, until Stores register, and the next `helm upgrade` +restores the full topology from values. + +Changing PD or Store replicas on a live release is not a values change: +raft and shard membership are persisted, and Pods alone do not reconfigure +them. The chart rejects both directions for PD and a shrink for Store by +reading the live StatefulSet, so fresh installs at any count are +unaffected. + +**PD, either direction.** The rendered peer list reaches raft only as its +bootstrap configuration, which an initialized group ignores; a 3-to-5 +upgrade leaves the voting configuration at three, and a 3-to-1 shrink +loses quorum outright. Membership changes go through PD's peer-list +change, which the PD client API and the PD REST route +`POST /v1/members/change` both expose; the route needs the same Basic +authentication as the other PD REST routes, and the chart wraps neither. +Change the membership through PD, confirm it in `/v1/members`, scale the +live StatefulSet, then `helm upgrade` with the matching value; until that +sequence is verified on your own build, install the PD count you intend +to keep. + +**Store, shrinking.** Draining is a state transition, not a balance: +neither `patrolPartitions` nor `balancePartitions` retires a healthy +Store. Retire the leaving Stores the way Disaster Recovery retires a +replaced one: + +1. Check the remaining Stores still cover the persisted shard count + (`pd.partition.defaultShardCount`; empty derives 3 when + `store.replicas` is at least 3). +2. Map the ordinals the shrink will delete (the highest ones) to Store + ids through `/v1/stores`, matching on the Pod address. +3. `POST /v1/store/` with `{"storeState":"Tombstone"}` for each + leaving id. +4. Wait until `/v1/shardGroups` no longer lists those ids and every group + reports its full shard count with one leader. +5. Scale the live StatefulSet, then `helm upgrade` with the matching + value. + +The chart keeps the PVCs of removed ordinals +(`store.persistentVolumeClaimRetentionPolicy.whenScaled: Retain`), and a +retained PVC still holds the retired Store id. PD refuses to register a +Store id that is Tombstone or deleted, so a Store that later starts on +that PVC, as when you scale the same ordinal back up, cannot join. +Delete the PVCs of the removed ordinals after step 4 reports the data +moved and before reusing those ordinals; the deletion is permanent. + +### 9 Running Hubble outside the cluster + +In-cluster Hubble (`hubble.enabled=true` plus a port-forward) is the +recommended path and is covered on the deployment page. Two paths exist +for a Hubble that must run outside the cluster. + +**Direct Server URL** (graph, schema, data, and Gremlin; no PD +discovery): leave in-chart Hubble off, expose Server through an +HTTPS-terminating Ingress or a NodePort/LoadBalancer `server.service.type`, +and run a standalone Hubble image. The Service carries the plain-HTTP API, +so the chart requires `server.service.allowInsecureExposure=true` for it; +restrict who can reach it first. The Hubble properties are: + +```properties +pd.enabled=false +server.direct_url=https://: +``` + +Mount the file at `/hubble/conf/hugegraph-hubble.properties` (the image +workdir is `/hubble`). Use HTTPS or a trusted channel for +`server.direct_url`: login sends the Server credentials over that URL. + +**PD discovery** (the outside Hubble asks PD for the Server address): +in-cluster names such as `*.svc` are not reachable from outside, so the +chart provides two knobs. Set `server.advertiseUrl` to the absolute +`http(s)://` URL the outside Hubble will use after discovery; the chart +registers it with PD in place of the in-cluster Service URL. And expose +the PD client Service (`pd.service.type` NodePort/LoadBalancer), which +requires `pd.service.allowInsecureExposure=true` because PD gRPC has no +authentication; restrict who can reach it first (with the chart's +NetworkPolicy on, the render refuses the exposure until its callers are +listed in `networkPolicy.pd.extraIngress`). Then run standalone Hubble +with: + +```properties +pd.enabled=true +pd.peers=: +pd.server=: +``` + +Trade-off: when `server.advertiseUrl` is set, every Server replica +registers that same logical URL and PD returns it to every discovery +client, including an in-cluster Hubble (which, with the chart's +NetworkPolicy on, then needs `networkPolicy.hubble.extraEgress`; see +NetworkPolicy above). Leave it empty for the default in-cluster path, +where each Server Pod registers its own IP. + +Local quick test (cluster and Hubble on one machine): port-forward Server +`8080` and PD client `8620`/`8686`, set +`server.advertiseUrl=http://127.0.0.1:8080`, run standalone Hubble with +`--network host` and the properties above, then open Hubble on `8088`. + +### 10 When Gremlin fails with "Could not rebind" + +The error has two causes with different lifetimes. + +**A convergence window after creating a graph.** The Server that handles +`CreateGraph` waits for its own Gremlin binding before returning HTTP 200 +([#3138](https://github.com/apache/hugegraph/pull/3138)), so +create-then-query on the same Server is reliable. Other replicas converge +independently, and until they finish, a Gremlin query routed through the +load-balanced Service to a not-yet-converged replica can fail with a 400 +such as `Could not rebind [g]`. Retry with backoff (the window normally +closes in seconds), use sticky routing or a port-forward for +create-then-verify flows, or poll `/graphs` on each replica before +opening query traffic. Cluster-wide readiness is tracked in +[#3137](https://github.com/apache/hugegraph/issues/3137). + +**A Server Pod that started while PD was rolling.** Gremlin Server +instantiates the graph once at startup; if the PD client cannot connect +at that moment, the Pod passes readiness and serves REST while every +Gremlin request on it fails with `Could not rebind [graph]`, for the life +of the Pod +([apache/hugegraph#3228](https://github.com/apache/hugegraph/issues/3228)). +Its `hugegraph-server.log` names it: + +``` +Graph [DEFAULT-hugegraph] configured at [...] could not be instantiated and +will not be available in Gremlin Server +``` + +After an upgrade or rollback that rolled PD and Server together, run +`helm test hugegraph --namespace hugegraph --logs --timeout 5m`: +it sends the graph-bound Gremlin query below to every Ready Server Pod +through the headless Service `hugegraph-server-headless`, retries for up +to 150 seconds, and fails with `Gremlin failed on ` for a Pod in +this state. To check one Pod by hand, port-forward it in one terminal +(the forward runs in the foreground): + +```bash +kubectl port-forward -n hugegraph pod/ 8080:8080 +``` + +and send the query from a second: + +```bash +curl -s --compressed -u "admin:${PASSWORD}" -H 'Content-Type: application/json' \ + -X POST http://127.0.0.1:8080/gremlin \ + -d '{"gremlin":"graph.traversal().V().limit(1).count()","aliases":{"graph":"DEFAULT-hugegraph"}}' +``` + +A healthy Pod answers with `result.data`. Once PD is stable, delete the affected +Pod and rerun the hook to verify its replacement. If Hubble is enabled, also +verify a fresh login and graph query. diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md new file mode 100644 index 0000000000..ca61de9bf3 --- /dev/null +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm.md @@ -0,0 +1,367 @@ +--- +title: "Deploy on Kubernetes with Helm" +linkTitle: "Deploy on Kubernetes (Helm)" +weight: 4 +search_keywords: + - helm + - kubernetes + - k8s +--- + +### 1 Overview + +The Helm chart deploys a distributed HugeGraph cluster on Kubernetes: PD, Store, and Server, plus the optional +Hubble UI. It lives in the main repository under +[`helm/hugegraph`](https://github.com/apache/hugegraph/tree/master/helm/hugegraph). + +| Component | Workload | Default replicas | Purpose | +|-----------|----------|------------------|---------| +| PD | StatefulSet + PVC | 3 | Placement driver: a Raft group tracking Stores and partitions | +| Store | StatefulSet + PVC | 3 | Graph data storage (HStore) | +| Server | Deployment | 3 | Gremlin and REST query layer | +| Hubble | Deployment | 0 (off) | Web UI, enabled with `hubble.enabled=true` | + +A distributed HugeGraph cluster has a startup contract (no `init-store` on Server, every Server using PD for graph +metadata, Store waiting for a PD quorum, one PD REST secret shared by three readers). The chart encodes that +contract so operators do not have to; the details are in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#chart-details). +Day-2 work (NetworkPolicy details, disaster recovery, scaling, safe Store +rolls, running Hubble outside the cluster) is on the +[operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/). + +```mermaid +flowchart LR + subgraph Kubernetes cluster + PD[PD StatefulSet
Raft group of 3, PVC] + Store[Store StatefulSet
3 replicas, PVC] -->|register, heartbeat| PD + Server[Server Deployment
3 replicas] -->|metadata, discovery| PD + Server -->|gRPC read/write| Store + Hubble[Hubble Deployment
optional UI] -->|discovers Servers via PD| PD + end + Client[Client / hugegraph-client] -->|REST / Gremlin| Server +``` + +Startup order is enforced by the chart, not by the operator: PD pods elect a leader first, each Store pod waits in +an init container until a majority of PD peers report ready, and Servers keep restarting their storage wait until +Stores have registered. A fresh install converges without manual steps. + +### 2 Prerequisites + +- Kubernetes 1.23 or later (the chart renders `autoscaling/v2` and `policy/v1`) +- Helm 3; the `--reset-then-reuse-values` flag mentioned under Upgrade needs Helm 3.14 or later +- Dynamic volume provisioning: a default StorageClass, or an explicit `storageClassName` for PD and Store +- Memory for nine JVMs in the default topology; see the resource note under Install + +The chart requires component images that carry the PD readiness endpoint and PD REST authentication (both merged +for the release after 1.7.0). The default image tags already point at builds that include them; 1.7.0 images are +not supported. + +### 3 Install + +#### 3.1 Get the chart + +The chart is not published to a chart repository yet, so install it from the source tree: + +```bash +git clone https://github.com/apache/hugegraph.git +cd hugegraph +``` + +#### 3.2 Install with default values + +First confirm `kubectl` points at the intended cluster and that it can provision volumes. PVCs stuck in `Pending` +for want of a StorageClass are the most common first-run failure: + +```bash +kubectl config current-context +kubectl get storageclass +``` + +Then install: + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace --wait --timeout 15m +``` + +`--wait` waits for Kubernetes readiness probes, not application workflows. A fresh cluster normally converges in +a few minutes; the 15-minute timeout leaves room for slow image pulls. Verify graph queries after installation. +Without `--wait`, Helm returns as soon as the objects are created. + +Two defaults to know before going further: + +- **No resources are set.** Every pod is BestEffort, and each JVM sets its maximum heap to half of the memory it + sees free on the node at start (up to a per-component ceiling), so the JVMs on one node can together claim more + memory than the node has. That holds for `values-single.yaml` on one node as well. Set `resources` per component on every install, the single-node + preset included; `values-cluster.yaml` sets them for a multi-node cluster. +- **Image tags track `latest`** until the next HugeGraph release is published, with `pullPolicy: Always`. Pin tags + or digests for production. + +#### 3.3 Topology presets + +The chart ships three values files: + +| File | Topology | Intended use | +|------|----------|--------------| +| `values.yaml` | 3 PD + 3 Store + 3 Server | Default; preferred anti-affinity, auth on, Hubble off | +| `values-single.yaml` | 1 + 1 + 1 | Single-node development and CI; smaller PVCs | +| `values-cluster.yaml` | 3 + 3 + 3 | Production starting point: JVM heap and resource settings, a Server PodDisruptionBudget, `required` anti-affinity, NetworkPolicy on, Store `OnDelete` updates | + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace \ + -f helm/hugegraph/values-single.yaml --wait --timeout 15m +``` + +`values-cluster.yaml` is a starting point, not a capacity guarantee: recalculate resources for your graph size and +traffic. One of its numbers deserves a note: it requests 5Gi and limits at 8Gi of memory per Store, well above the +1Gi heap, because the Store's RocksDB caches live outside the JVM heap (a 4Gi limit OOM-killed Stores after about +1 GB of data). Scale both numbers with data size. The full parameter reference (every component, probe, +scheduling, and Secret knob) is kept in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration). + +#### 3.4 Verify the install + +```bash +helm test hugegraph --namespace hugegraph --logs --timeout 5m +``` + +The test first calls `/versions` and `/graphs` through the Server Service. It then resolves the headless Service +`hugegraph-server-headless` and sends an authenticated Gremlin query bound to the `DEFAULT-hugegraph` graph to +each Server Pod it lists, and it requires at least `server.replicas` Pods (`server.hpa.minReplicas` with HPA on). +The headless Service lists only Ready Pods, the same Pods the Server Service routes to, so a Pod that is not Ready +is not queried. A failing Pod is retried every 5 seconds for up to 150 seconds, so a short PD election is not +reported as a broken Pod; after that the test fails and prints each failing Pod IP with its HTTP status. This is +the check that catches a Server that passes readiness and serves REST while every Gremlin call on it fails (see +Limitations). Large HPA fleets may need `helm test --timeout` above the 5-minute default. + +The hook verifies Server Gremlin, not Hubble login. With Hubble enabled, also verify a fresh login and graph query +after installation or recovery; `/actuator/health` alone does not establish backend connectivity. + +To call the API, start a port-forward in one terminal; it runs in the foreground until you stop it: + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +``` + +Then, in a second terminal, read the generated admin password and call the API: + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin -o jsonpath='{.data.password}' | base64 --decode)" +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +``` + +The commands above assume the release is named `hugegraph`; with another name, substitute the release-prefixed +resource names (`kubectl get svc,secret -n ` lists them). The post-install notes printed by +`helm install` repeat these commands with the right names filled in. + +### 4 Authentication and Secrets + +Authentication is on by default and the chart manages three Secrets. Each credential resolves in the same order: +an `existingSecret` you created wins, then an inline value, then a random value generated at install time. + +| Secret | Key | Used for | Bring your own with | +|--------|-----|----------|---------------------| +| `-admin` | `password` | Server admin account, Hubble login | `server.auth.admin.existingSecret` | +| `-auth-token` | `token_secret` | JWT signing key shared by all Server replicas | `server.auth.token.existingSecret` | +| `-pd-auth` | `secret-key` | PD REST authentication, read by PD, Server, and Hubble | `pd.auth.existingSecret` | + +To manage a credential yourself, create the Secret before installing and point the matching `existingSecret` value +at it; the chart never modifies a Secret it did not create. Value constraints: the admin password must be printable +ASCII with no spaces, colons, or backslashes (the image stores a space with a backslash before it, and the Server splits +Basic-auth credentials on every colon, so either would leave the account unable to log in with the Secret value); the +JWT key must be at least 32 bytes; the PD secret must be printable ASCII with no backslashes and no leading or trailing +space. Invalid inline values are rejected at render time, and an `existingSecret` by the startup wrapper when the Pod +starts, not silently truncated. + +Chart-managed Secrets are kept on uninstall and reused by a later install under the same release name. + +
+Rotation and caveats + +- The admin password is applied only when the auth metadata is first created, so changing the Secret later does not + rotate an existing cluster's password. Rotate it through the Server's auth API instead. +- Rotating a credential rolls the Pods that read it once, on the next `helm upgrade`: PD, Server, and Hubble together + for the PD REST Secret, which keeps their copies in step, and Server for the admin password or JWT key. +- Template-only pipelines (`helm template`, Argo CD, Flux in template mode) cannot see live Secrets, so a + chart-generated credential gets a new value on every render: each sync changes the Secret, rolls the Pods that read + it, and leaves the admin account created at first start out of step with the Secret. For such a pipeline, create the + three Secrets first and set the `existingSecret` values; see + [Template-only pipelines](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#template-only-pipelines-gitops) + in the chart README. +- All three Secrets exist even if you only ever read one: the post-install notes print the exact `kubectl get + secret` commands for the admin password and the PD secret. +
+ +### 5 Health checks and startup order + +PD exposes two health endpoints, and the chart deliberately uses both: + +- `/v1/health` answers 200 as soon as the REST listener is up. It never consults Raft, so it cannot see a lost + quorum. +- `/v1/ready` answers 503 until the PD Raft group has a leader, so it reports quorum, not just a live process. + +The chart puts PD **readiness** and the Store init-container wait on `/v1/ready`: a Store only starts once a +majority of PD peers are quorum members, and a PD that lost its leader drops out of Service endpoints until a +leader is back. PD **startup and liveness** derive from the replica count (`pd.livenessPath` overrides the +choice). With more than one PD they stay on `/v1/health` on purpose: a PD that merely lost its leader is still a +healthy Raft member, and restarting it would make the outage worse. A single PD is the exception and derives to +`/v1/ready`: it has no election to lose, and one that steps down for good, as after a failed Raft snapshot on a +full disk ([apache/hugegraph#3222](https://github.com/apache/hugegraph/issues/3222)), would answer `/v1/health` +forever while serving no writes; the kubelet restarts it instead. + +Server startup gets a budget of at least 450 seconds, enough for the 300-second storage wait the image entrypoint +runs before the start command, plus process startup. The chart counts the guaranteed probe time as +`(failureThreshold - 1) * periodSeconds`, because the kubelet may run the first probe as soon as the container +starts, and raises a lower configured `failureThreshold` to that floor (91 at the default 5-second period, the +value `values.yaml` ships). The image would kill a Server still starting after 120 seconds, so the chart also sets +`HG_SERVER_STARTUP_TIMEOUT_S` to the guaranteed probe time minus the 300-second storage wait: 150 seconds by +default, never less than the image's 120. The start command and the probe then give up together. Raise +`server.probes.startup` if your storage takes longer to come up, and the timeout follows; the variable is +chart-managed, so change the probe rather than setting it in `server.extraEnv`. + +### 6 Enable the Hubble UI + +Hubble is off by default so API-only clusters stay lean. Enable it on a running release: + +```bash +helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values \ + --set hubble.enabled=true --wait --timeout 15m +``` + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-hubble 8088:8088 +``` + +Open `http://127.0.0.1:8088` and log in as `admin` with the admin password from Section 3.4. Hubble discovers the +Servers through PD, so the cluster operations view works without extra wiring. Hubble serves plain HTTP: reach it +through a port-forward or an HTTPS-terminating Ingress, never directly from an untrusted network; a NodePort or +LoadBalancer Hubble Service is refused unless `hubble.service.allowInsecureExposure=true` acknowledges it. Running Hubble +outside the cluster is possible but takes more wiring; see +[Reaching Hubble](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#reaching-hubble-pick-one-path) in +the chart README. + +### 7 Upgrade + +```bash +helm upgrade hugegraph ./helm/hugegraph --namespace hugegraph --reuse-values +``` + +`--reuse-values` keeps the release's existing overrides; without it the upgrade rebuilds the release from chart +defaults. It also keeps the old values as the complete base, so a release created by an earlier chart revision does +not pick up new defaults such as the hardened `securityContext`; pass your own values with `-f`, or use +`--reset-then-reuse-values`, to adopt them. Any upgrade that changes a Pod template rolls that workload once. +Points worth planning around: + +- **A no-change upgrade rolls nothing.** The Secret-tracking annotations hash each credential's value, or the live + `resourceVersion` of an `existingSecret`, so a Pod rolls only when a credential it reads changes. An upgrade that + does roll PD and Server together, such as a PD REST Secret rotation or an image change on both, can leave a Server + without its Gremlin binding (see Limitations). Treat `pd.auth` rotation as coordinated maintenance, not an + unattended routine change: run `helm test --logs` afterwards and recover affected replicas as documented on the + [operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/#10-when-gremlin-fails-with-could-not-rebind). + A rollback that rolls both components needs the same verification. +- **Store rolling updates advance on a listener check, not on shard recovery**, so the controller can replace the + next Store while the previous one is still rejoining its shard groups. `values-cluster.yaml` therefore sets + `store.updateStrategy.type=OnDelete`; `values.yaml` and `values-single.yaml` keep `RollingUpdate`, so set it + yourself on any other production values. Under `OnDelete` an upgrade updates the StatefulSet but replaces no + Store Pod, and you delete Store Pods one at a time. `OnDelete` only stops automatic advancement: deleting Pods + without checking between them carries the same risk. `Up` in PD is not that check: PD marks a Store `Up` at + registration, before it restores partitions, and a stopped Store stays `Up` until a 300 s keep-alive expires. + No endpoint on current images reports that a Store has finished restoring its partitions + ([apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229)), so the check is indirect. Wait for + the replaced Pod to be `Ready`, then confirm every shard group reports its full shard count with one + leader; the full procedure is on the + [operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/#6-rolling-store-images-safely). +- **The controller never rolls more than one PD or Store Pod at a time.** The chart leaves + `updateStrategy.rollingUpdate.maxUnavailable` unset and accepts only the integer `1` there: a larger number or + a percentage fails the render, because it would let the controller take down two members of a three-member + Raft group or shard at once, and a PodDisruptionBudget does not limit controller rollouts. For PD maintenance + windows, `pd.updateStrategy.type=OnDelete` gives the same manual control as for Store. +- **PVC sizes cannot be changed by upgrade**: Kubernetes forbids changing StatefulSet `volumeClaimTemplates`, so an + upgrade with a new `storage.size` is rejected in full. The chart README documents the resize procedure for + StorageClasses that support volume expansion. +- **Some values are fixed once the release is initialized.** `nameOverride`, `fullnameOverride`, the PD and Store + raft ports, and the storage settings are install-time identity: the chart refuses an override or raft port change + against the live StatefulSets, and Kubernetes refuses the storage change. `server.auth.admin.*` and the partition + shard counts apply at bootstrap only, so an initialized cluster ignores a new value. The chart README sorts the + values by + [lifecycle](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#settings-by-lifecycle). +- **`helm rollback` runs none of the chart's guards.** A rollback, including the automatic one after a failed + `--atomic` upgrade, reapplies an earlier stored manifest without rendering the chart, so a rollback across a PD or + Store replica change scales the StatefulSet straight to the old count and can drop PD below quorum. Rollback across + a membership or identity change is unsupported: return to an earlier chart or image with a forward upgrade that + keeps the current topology values, and keep `--atomic` off for upgrades that change replicas. The chart README + covers the checks to run after any + [rollback](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#rollback). + +Scaling Server up and down is a values change (`server.replicas`, or `server.hpa`). Changing the **PD count in +either direction, or shrinking Store, is not**: Raft and shard membership are persisted and Pods alone do not +reconfigure them, so a 3-to-1 PD shrink permanently loses quorum and new PD Pods do not join the voting +configuration. The chart reads the live StatefulSet and rejects an upgrade that changes the PD count or +lowers the Store count; the manual procedures are on the +[operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/#8-scaling). + +### 8 Uninstall + +```bash +helm uninstall hugegraph --namespace hugegraph +``` + +Two kinds of state survive on purpose. PersistentVolumeClaims created by the StatefulSets are kept (Kubernetes +behavior); delete them explicitly once the data is no longer needed. The chart-managed Secrets are also kept, so a +later install under the same release name comes back with the same credentials. + +### 9 Limitations + +- `networkPolicy.enabled` renders one NetworkPolicy per component that admits only the release's own traffic; it + is off by default and on in `values-cluster.yaml`. It matters because the chart disables the PD Raft IP + allowlist in-cluster (pod IPs change; the allowlist resolves peers once at boot and then blocks them), so the + policies are what restricts the raft and gRPC ports. They need a network plugin that enforces NetworkPolicy + (Calico, Cilium, kind v0.25 or later, k3s), and every outside client, the Ingress controller included, must be + listed in `networkPolicy..extraIngress` or the render fails. Details are on the + [operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/). +- Image tags track `latest` until the next HugeGraph release publishes versioned images; pin tags or digests for + anything long-lived. +- After creating a graph, other Server replicas can lag for a short window before they serve queries for it, so a + query routed to a not-yet-converged replica can fail with an error such as `Could not rebind [g]`. Retry with + backoff, or use sticky routing for create-then-query flows; cluster-wide graph readiness is tracked in + [#3137](https://github.com/apache/hugegraph/issues/3137). +- A Server that starts while PD is unreachable, as during a PD roll, can come up without its Gremlin binding for + the life of the Pod: it passes readiness and serves REST while every Gremlin request on it fails with + `Could not rebind [graph]` ([apache/hugegraph#3228](https://github.com/apache/hugegraph/issues/3228)). The + readiness probe calls `/versions` and cannot see this. `helm test` queries Gremlin on every Ready Server Pod and + prints the IP of a Pod in this state; delete that Pod, and its replacement binds once PD is stable. See + [When Gremlin fails with "Could not rebind"](/docs/quickstart/hugegraph/hugegraph-helm-operations/#10-when-gremlin-fails-with-could-not-rebind). +- A PD that cannot open its RocksDB store at startup, for example because another process still holds the + store's `LOCK` file, neither retries nor exits: it logs `Failed to open RocksDB` once and keeps running, with + `/v1/ready` answering 503 and `STATE_UNINITIALIZED` while `/v1/health` answers 200. This was observed on PD + processes outside Kubernetes ([apache/hugegraph#3226](https://github.com/apache/hugegraph/issues/3226)). Under + the chart's probes such a Pod is not Ready and leaves the Service endpoints. With more than one PD, startup and + liveness use `/v1/health`, so the kubelet does not restart it; with a single PD they use `/v1/ready`, so the + kubelet restarts the container. How this state arises on Kubernetes, and whether restarting or deleting the Pod + clears it, has not been tested. Deleting the `LOCK` files is not a fix: they protect the store from a second + process that may still be running. +- Store recovery is operator-triggered on current builds: re-replication after Store loss, leader balancing, and + partition rebalancing run only when called through PD's REST API. A Store whose volume is lost can be recovered + in place only on images carrying + [apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234) (merged 2026-09-24, in no release yet); + on earlier images, keep a Store's PVC when replacing its Pod. The Disaster Recovery section of the + [operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/) is the runbook. +- The Server and Hubble serve plain HTTP, and PD gRPC has no authentication. A NodePort or LoadBalancer Service for + any of them is refused unless the matching `pd.service.allowInsecureExposure`, `server.service.allowInsecureExposure`, + or `hubble.service.allowInsecureExposure` is set, and a Server or Hubble Ingress without `tls` is refused unless its + `allowPlainHttp` is set. Prefer a port-forward or an HTTPS-terminating Ingress, and restrict who can reach an + exposed Service first. +- No TLS termination inside the cluster, no backup tooling, no Operator, and no bundled monitoring stack. + +### 10 Troubleshooting + +| Symptom | First check | +|---------|-------------| +| Store pods stuck in `Init:0/1` | PD is not ready: `kubectl logs -c wait-for-pd`, then the PD pods | +| PVCs stay `Pending` | No default StorageClass, or the provisioner is down: `kubectl get sc` | +| Pods OOM killed or restarting | No resources set, JVM heaps sized to node memory: set `resources` (see Install) | +| Query fails right after creating a graph | Replica convergence window: see Limitations above | +| `helm test` prints `Gremlin failed on ` | That Server Pod lost its Gremlin binding: delete it (see Limitations above) | +| A PD Pod stays not Ready and its `/v1/ready` reports `STATE_UNINITIALIZED` | Its RocksDB store may not have opened: look for `Failed to open RocksDB` in its log (see Limitations above) | + +Longer walkthroughs for most of these cases are in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#troubleshooting). diff --git a/data/docs_nav.json b/data/docs_nav.json index d4b4125433..4a5503ecee 100644 --- a/data/docs_nav.json +++ b/data/docs_nav.json @@ -34,6 +34,12 @@ }, { "page": "/docs/quickstart/hugegraph/hugegraph-pd" + }, + { + "page": "/docs/quickstart/hugegraph/hugegraph-helm" + }, + { + "page": "/docs/quickstart/hugegraph/hugegraph-helm-operations" } ] }, @@ -406,6 +412,16 @@ "page": "/docs/quickstart/hugegraph/hugegraph-pd", "url": "/docs/quickstart/hugegraph/hugegraph-pd/", "children": [] + }, + { + "page": "/docs/quickstart/hugegraph/hugegraph-helm", + "url": "/docs/quickstart/hugegraph/hugegraph-helm/", + "children": [] + }, + { + "page": "/docs/quickstart/hugegraph/hugegraph-helm-operations", + "url": "/docs/quickstart/hugegraph/hugegraph-helm-operations/", + "children": [] } ] }, @@ -906,6 +922,16 @@ "/docs/quickstart/hugegraph/", "/docs/quickstart/hugegraph/hugegraph-pd/" ], + "/docs/quickstart/hugegraph/hugegraph-helm/": [ + "@group/components", + "/docs/quickstart/hugegraph/", + "/docs/quickstart/hugegraph/hugegraph-helm/" + ], + "/docs/quickstart/hugegraph/hugegraph-helm-operations/": [ + "@group/components", + "/docs/quickstart/hugegraph/", + "/docs/quickstart/hugegraph/hugegraph-helm-operations/" + ], "/docs/quickstart/toolchain/": [ "@group/components", "/docs/quickstart/toolchain/" @@ -1347,7 +1373,9 @@ ], "/docs/quickstart/hugegraph/": [ "/docs/quickstart/hugegraph/hugegraph-hstore", - "/docs/quickstart/hugegraph/hugegraph-pd" + "/docs/quickstart/hugegraph/hugegraph-pd", + "/docs/quickstart/hugegraph/hugegraph-helm", + "/docs/quickstart/hugegraph/hugegraph-helm-operations" ], "/docs/quickstart/toolchain/": [ "/docs/quickstart/toolchain/visualization", diff --git a/data/version_routes.json b/data/version_routes.json index 7f6e062577..a92a982749 100644 --- a/data/version_routes.json +++ b/data/version_routes.json @@ -715,6 +715,20 @@ "1.3": null, "1.0": null }, + "cn:quickstart/hugegraph/hugegraph-helm": { + "latest": "cn/docs/quickstart/hugegraph/hugegraph-helm/", + "1.7": null, + "1.5": null, + "1.3": null, + "1.0": null + }, + "cn:quickstart/hugegraph/hugegraph-helm-operations": { + "latest": "cn/docs/quickstart/hugegraph/hugegraph-helm-operations/", + "1.7": null, + "1.5": null, + "1.3": null, + "1.0": null + }, "cn:quickstart/hugegraph/hugegraph-server": { "latest": "cn/docs/quickstart/hugegraph/hugegraph-server/", "1.7": "cn/docs/quickstart/hugegraph/hugegraph-server/", @@ -1429,6 +1443,20 @@ "1.3": null, "1.0": null }, + "en:quickstart/hugegraph/hugegraph-helm": { + "latest": "docs/quickstart/hugegraph/hugegraph-helm/", + "1.7": null, + "1.5": null, + "1.3": null, + "1.0": null + }, + "en:quickstart/hugegraph/hugegraph-helm-operations": { + "latest": "docs/quickstart/hugegraph/hugegraph-helm-operations/", + "1.7": null, + "1.5": null, + "1.3": null, + "1.0": null + }, "en:quickstart/hugegraph/hugegraph-server": { "latest": "docs/quickstart/hugegraph/hugegraph-server/", "1.7": "docs/quickstart/hugegraph/hugegraph-server/", diff --git a/scripts/versioning.py b/scripts/versioning.py index 2e37d3a435..5ff92f9030 100644 --- a/scripts/versioning.py +++ b/scripts/versioning.py @@ -133,36 +133,36 @@ DOCS_NAV_EXPECTED_STATS = { "latest": { "groups": 5, - "pages": 92, + "pages": 94, "removed": 4, "scopedLinks": 0, - "treeSha256": "15de4de385857f877b46083de8e65d2887769d811687c4fe7ade3069401fc929", + "treeSha256": "c610cc9128f26a85e0f60843f9e5255f95daa8ba7d1d3a62d19a07a799a3e520", }, "1.7": { "groups": 5, "pages": 85, - "removed": 5, + "removed": 7, "scopedLinks": 10, "treeSha256": "c87538b82b0e3506eef59417411686ddc188e0f33d0e3a1cda687de8d6f88747", }, "1.5": { "groups": 5, "pages": 77, - "removed": 13, + "removed": 15, "scopedLinks": 10, "treeSha256": "ac29ab8f0e7020496a3a1afe480687043d751e32c93ffb848e2210e3500cd483", }, "1.3": { "groups": 5, "pages": 68, - "removed": 22, + "removed": 24, "scopedLinks": 10, "treeSha256": "e8c9b0a64dac6dd3e47827ec5072697eb7d3ed79659e0b5c4a9c5df8e1408d38", }, "1.0": { "groups": 5, "pages": 59, - "removed": 21, + "removed": 23, "scopedLinks": 10, "treeSha256": "bd45db511d9b6728e1523bd3c5ebd9d18912ee85aeabf79c77bcca7e511f977a", },