diff --git a/docs/en/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md b/docs/en/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md new file mode 100644 index 000000000..332b6ae4d --- /dev/null +++ b/docs/en/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md @@ -0,0 +1,103 @@ +--- +products: + - Alauda Container Platform +kind: + - Troubleshooting +ProductsVersion: + - 4.3.0,4.3.1,4.3.2 +id: TBD +--- + +# VPC Egress Gateway remains Ready with zero BFD sessions + +## Issue + +On Alauda Container Platform 4.3.0, 4.3.1, or 4.3.2, a VPC Egress Gateway (VEG) has BFD enabled and application Pods selected by the VEG cannot access external networks. The VEG Pod remains `Running` and `Ready`, but the `bfdd` container has no local BFD session: + +```text +There are 0 sessions: +``` + +This article applies only when `bfdd-control status` reports `There are 0 sessions:` while the VEG Pod remains healthy. + +## Root Cause + +The existing `bfdd` health probes run `bfdd-control status` directly. The command returns exit code `0` when it successfully communicates with the local daemon, even if the daemon reports no BFD sessions: + +| BFD state | Example output | Exit code | +| --- | --- | --- | +| A session exists | `There are 1 sessions:` | `0` | +| No session exists | `There are 0 sessions:` | `0` | + +Kubernetes exec probes evaluate only the command exit code and do not interpret its output. The kubelet therefore continues to consider `bfdd` healthy when the local session table is empty and does not restart the container to reinitialize its BFD configuration. Without a BFD session, the corresponding VEG next hop can become unavailable and interrupt application outbound traffic. + +## Resolution + +Upgrade the cluster to ACP 4.3.3 or later, which includes the fix for this issue. No manual health-probe change is required after the upgrade. + +If the cluster must remain on ACP 4.3.0, 4.3.1, or 4.3.2, use the following temporary workaround. + +Set the VEG namespace and name, and obtain the generated Deployment name: + +```bash +NAMESPACE="" +VEG_NAME="" + +DEPLOYMENT="$(kubectl -n "${NAMESPACE}" get deployment \ + -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ + -o jsonpath='{.items[0].metadata.name}')" +``` + +Edit the generated Deployment: + +```bash +kubectl -n "${NAMESPACE}" edit deployment "${DEPLOYMENT}" +``` + +Find the container named `bfdd` and replace only its existing `livenessProbe` with the following configuration. Keep the readiness probe and all other Deployment fields unchanged: + +```yaml +livenessProbe: + exec: + command: + - bash + - -ec + - 'output="$(bfdd-control status)"; ! grep -q "^There are 0 sessions:" <<< "${output}"' + initialDelaySeconds: 1 + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 3 + successThreshold: 1 +``` + +Wait for the Deployment rollout to finish: + +```bash +kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ + --timeout=180s +``` + +:::warning +Updating the Deployment recreates VEG Pods. For a single-replica VEG, schedule a maintenance window because application outbound traffic can be interrupted during the rollout. + +The VEG Deployment is generated by the Kube-OVN controller. This manual probe change can be overwritten when the VEG specification or replica count changes, Kube-OVN is upgraded, or the VEG or Deployment is recreated. Upgrade to ACP 4.3.3 or later instead of reapplying the workaround when an upgrade is available. +::: + +## Diagnostic Steps + +Check the local BFD session in each VEG Pod: + +```bash +for pod in $(kubectl -n "${NAMESPACE}" get pods \ + -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ + -o jsonpath='{.items[*].metadata.name}'); do + echo "=== ${pod} ===" + kubectl -n "${NAMESPACE}" exec "${pod}" -c bfdd -- bfdd-control status + kubectl -n "${NAMESPACE}" get pod "${pod}" \ + -o jsonpath='bfdd restartCount={.status.containerStatuses[?(@.name=="bfdd")].restartCount}{"\n"}' +done +``` + +When the temporary probe detects `There are 0 sessions:` three consecutive times, the kubelet restarts `bfdd`. The existing startup probe then configures the BFD parameters and registers the peer again. Confirm that a new session is created and eventually reaches `Up`. + +If `bfdd` continues to report zero sessions after repeated restarts, investigate the VPC BFD port, OVN-side BFD state, peer reachability, and the network path. Restarting the local container cannot repair an unavailable peer or a broken network path. diff --git a/docs/zh/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md b/docs/zh/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md new file mode 100644 index 000000000..ed9c8cf4e --- /dev/null +++ b/docs/zh/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md @@ -0,0 +1,104 @@ +--- +products: + - Alauda Container Platform +kind: + - Troubleshooting +ProductsVersion: + - 4.3.0,4.3.1,4.3.2 +id: TBD +sourceSHA: 876c743ff16cecb573589b3657be2ed43b666921c72c20ac0be4b41553a9eca6 +--- + +# VPC 出口网关 BFD 会话为空但 Pod 仍为 Ready + +## 问题现象 + +在 Alauda Container Platform 4.3.0、4.3.1 或 4.3.2 中，VPC 出口网关（VEG）已启用 BFD，被 VEG 选中的业务 Pod 无法访问外部网络。此时 VEG Pod 仍处于 `Running` 和 `Ready` 状态，但 `bfdd` 容器中没有本地 BFD 会话： + +```text +There are 0 sessions: +``` + +本文仅适用于 `bfdd-control status` 显示 `There are 0 sessions:`，但 VEG Pod 仍然健康的场景。 + +## 根因 + +现有 `bfdd` 健康探针直接执行 `bfdd-control status`。该命令只要能够成功连接本地守护进程，就会返回退出码 `0`，即使本地已经不存在 BFD 会话： + +| BFD 状态 | 示例输出 | 退出码 | +| --- | --- | --- | +| 存在 BFD 会话 | `There are 1 sessions:` | `0` | +| 不存在 BFD 会话 | `There are 0 sessions:` | `0` | + +Kubernetes exec 探针只判断命令退出码，不解析命令输出。因此，本地会话表为空时，kubelet 仍然认为 `bfdd` 健康，不会重启容器并重新初始化 BFD 配置。没有 BFD 会话时，对应的 VEG 下一跳可能变为不可用，导致业务出网流量中断。 + +## 解决方案 + +将集群升级到 ACP 4.3.3 或更高版本，该版本已包含此问题的修复。升级后不需要手工修改健康探针。 + +如果集群暂时必须继续使用 ACP 4.3.0、4.3.1 或 4.3.2，请使用以下临时方案。 + +设置 VEG 所在的命名空间和名称，并获取控制器生成的 Deployment 名称： + +```bash +NAMESPACE="" +VEG_NAME="" + +DEPLOYMENT="$(kubectl -n "${NAMESPACE}" get deployment \ + -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ + -o jsonpath='{.items[0].metadata.name}')" +``` + +编辑控制器生成的 Deployment： + +```bash +kubectl -n "${NAMESPACE}" edit deployment "${DEPLOYMENT}" +``` + +找到名为 `bfdd` 的容器，只将其现有 `livenessProbe` 替换为以下配置。保持就绪探针和 Deployment 中的其他字段不变： + +```yaml +livenessProbe: + exec: + command: + - bash + - -ec + - 'output="$(bfdd-control status)"; ! grep -q "^There are 0 sessions:" <<< "${output}"' + initialDelaySeconds: 1 + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 3 + successThreshold: 1 +``` + +等待 Deployment 完成滚动更新： + +```bash +kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ + --timeout=180s +``` + +:::warning +更新 Deployment 会重建 VEG Pod。对于单副本 VEG，请安排维护窗口，因为滚动更新期间业务出网可能暂时中断。 + +VEG Deployment 由 Kube-OVN 控制器生成。修改 VEG spec 或副本数、升级 Kube-OVN，或者重建 VEG 或 Deployment 时，手工修改的探针可能被覆盖。如果可以升级，请直接升级到 ACP 4.3.3 或更高版本，不要重复应用临时方案。 +::: + +## 诊断步骤 + +检查每个 VEG Pod 中的本地 BFD 会话： + +```bash +for pod in $(kubectl -n "${NAMESPACE}" get pods \ + -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ + -o jsonpath='{.items[*].metadata.name}'); do + echo "=== ${pod} ===" + kubectl -n "${NAMESPACE}" exec "${pod}" -c bfdd -- bfdd-control status + kubectl -n "${NAMESPACE}" get pod "${pod}" \ + -o jsonpath='bfdd restartCount={.status.containerStatuses[?(@.name=="bfdd")].restartCount}{"\n"}' +done +``` + +临时探针连续三次检测到 `There are 0 sessions:` 后，kubelet 会重启 `bfdd`。现有启动探针随后会重新配置 BFD 参数并注册对端。确认新的 BFD 会话已经创建并最终进入 `Up` 状态。 + +如果 `bfdd` 在反复重启后仍然显示零个会话，需要检查 VPC BFD 端口、OVN 侧 BFD 状态、对端连通性和网络路径。重启本地容器无法修复对端不可用或网络路径异常。