Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
---
products:
- Alauda Container Platform
kind:
- Troubleshooting
ProductsVersion:
- 4.3.0,4.3.1,4.3.2
id: TBD
---

# VPC Egress Gateway remains Ready with zero BFD sessions

## Issue

On Alauda Container Platform 4.3.0, 4.3.1, or 4.3.2, a VPC Egress Gateway (VEG) has BFD enabled and application Pods selected by the VEG cannot access external networks. The VEG Pod remains `Running` and `Ready`, but the `bfdd` container has no local BFD session:

```text
There are 0 sessions:
```

This article applies only when `bfdd-control status` reports `There are 0 sessions:` while the VEG Pod remains healthy.

## Root Cause

The existing `bfdd` health probes run `bfdd-control status` directly. The command returns exit code `0` when it successfully communicates with the local daemon, even if the daemon reports no BFD sessions:

| BFD state | Example output | Exit code |
| --- | --- | --- |
| A session exists | `There are 1 sessions:` | `0` |
| No session exists | `There are 0 sessions:` | `0` |

Kubernetes exec probes evaluate only the command exit code and do not interpret its output. The kubelet therefore continues to consider `bfdd` healthy when the local session table is empty and does not restart the container to reinitialize its BFD configuration. Without a BFD session, the corresponding VEG next hop can become unavailable and interrupt application outbound traffic.

## Resolution

Upgrade the cluster to ACP 4.3.3 or later, which includes the fix for this issue. No manual health-probe change is required after the upgrade.

If the cluster must remain on ACP 4.3.0, 4.3.1, or 4.3.2, use the following temporary workaround.

Set the VEG namespace and name, and obtain the generated Deployment name:

```bash
NAMESPACE="<veg-namespace>"
VEG_NAME="<veg-name>"

DEPLOYMENT="$(kubectl -n "${NAMESPACE}" get deployment \
-l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \
-o jsonpath='{.items[0].metadata.name}')"
```

Edit the generated Deployment:

```bash
kubectl -n "${NAMESPACE}" edit deployment "${DEPLOYMENT}"
```

Find the container named `bfdd` and replace only its existing `livenessProbe` with the following configuration. Keep the readiness probe and all other Deployment fields unchanged:

```yaml
livenessProbe:
exec:
command:
- bash
- -ec
- 'output="$(bfdd-control status)"; ! grep -q "^There are 0 sessions:" <<< "${output}"'
initialDelaySeconds: 1
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 3
successThreshold: 1
```

Wait for the Deployment rollout to finish:

```bash
kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \
--timeout=180s
```

:::warning
Updating the Deployment recreates VEG Pods. For a single-replica VEG, schedule a maintenance window because application outbound traffic can be interrupted during the rollout.

The VEG Deployment is generated by the Kube-OVN controller. This manual probe change can be overwritten when the VEG specification or replica count changes, Kube-OVN is upgraded, or the VEG or Deployment is recreated. Upgrade to ACP 4.3.3 or later instead of reapplying the workaround when an upgrade is available.
:::

## Diagnostic Steps

Check the local BFD session in each VEG Pod:

```bash
for pod in $(kubectl -n "${NAMESPACE}" get pods \
-l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \
-o jsonpath='{.items[*].metadata.name}'); do
echo "=== ${pod} ==="
kubectl -n "${NAMESPACE}" exec "${pod}" -c bfdd -- bfdd-control status
kubectl -n "${NAMESPACE}" get pod "${pod}" \
-o jsonpath='bfdd restartCount={.status.containerStatuses[?(@.name=="bfdd")].restartCount}{"\n"}'
done
```

When the temporary probe detects `There are 0 sessions:` three consecutive times, the kubelet restarts `bfdd`. The existing startup probe then configures the BFD parameters and registers the peer again. Confirm that a new session is created and eventually reaches `Up`.

If `bfdd` continues to report zero sessions after repeated restarts, investigate the VPC BFD port, OVN-side BFD state, peer reachability, and the network path. Restarting the local container cannot repair an unavailable peer or a broken network path.
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
---
products:
- Alauda Container Platform
kind:
- Troubleshooting
ProductsVersion:
- 4.3.0,4.3.1,4.3.2
id: TBD
sourceSHA: 876c743ff16cecb573589b3657be2ed43b666921c72c20ac0be4b41553a9eca6
---

# VPC 出口网关 BFD 会话为空但 Pod 仍为 Ready

## 问题现象

在 Alauda Container Platform 4.3.0、4.3.1 或 4.3.2 中,VPC 出口网关(VEG)已启用 BFD,被 VEG 选中的业务 Pod 无法访问外部网络。此时 VEG Pod 仍处于 `Running` 和 `Ready` 状态,但 `bfdd` 容器中没有本地 BFD 会话:

```text
There are 0 sessions:
```

本文仅适用于 `bfdd-control status` 显示 `There are 0 sessions:`,但 VEG Pod 仍然健康的场景。

## 根因

现有 `bfdd` 健康探针直接执行 `bfdd-control status`。该命令只要能够成功连接本地守护进程,就会返回退出码 `0`,即使本地已经不存在 BFD 会话:

| BFD 状态 | 示例输出 | 退出码 |
| --- | --- | --- |
| 存在 BFD 会话 | `There are 1 sessions:` | `0` |
| 不存在 BFD 会话 | `There are 0 sessions:` | `0` |

Kubernetes exec 探针只判断命令退出码,不解析命令输出。因此,本地会话表为空时,kubelet 仍然认为 `bfdd` 健康,不会重启容器并重新初始化 BFD 配置。没有 BFD 会话时,对应的 VEG 下一跳可能变为不可用,导致业务出网流量中断。

## 解决方案

将集群升级到 ACP 4.3.3 或更高版本,该版本已包含此问题的修复。升级后不需要手工修改健康探针。

如果集群暂时必须继续使用 ACP 4.3.0、4.3.1 或 4.3.2,请使用以下临时方案。

设置 VEG 所在的命名空间和名称,并获取控制器生成的 Deployment 名称:

```bash
NAMESPACE="<veg-namespace>"
VEG_NAME="<veg-name>"

DEPLOYMENT="$(kubectl -n "${NAMESPACE}" get deployment \
-l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \
-o jsonpath='{.items[0].metadata.name}')"
```

编辑控制器生成的 Deployment:

```bash
kubectl -n "${NAMESPACE}" edit deployment "${DEPLOYMENT}"
```

找到名为 `bfdd` 的容器,只将其现有 `livenessProbe` 替换为以下配置。保持就绪探针和 Deployment 中的其他字段不变:

```yaml
livenessProbe:
exec:
command:
- bash
- -ec
- 'output="$(bfdd-control status)"; ! grep -q "^There are 0 sessions:" <<< "${output}"'
initialDelaySeconds: 1
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 3
successThreshold: 1
```

等待 Deployment 完成滚动更新:

```bash
kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \
--timeout=180s
```

:::warning
更新 Deployment 会重建 VEG Pod。对于单副本 VEG,请安排维护窗口,因为滚动更新期间业务出网可能暂时中断。

VEG Deployment 由 Kube-OVN 控制器生成。修改 VEG spec 或副本数、升级 Kube-OVN,或者重建 VEG 或 Deployment 时,手工修改的探针可能被覆盖。如果可以升级,请直接升级到 ACP 4.3.3 或更高版本,不要重复应用临时方案。
:::

## 诊断步骤

检查每个 VEG Pod 中的本地 BFD 会话:

```bash
for pod in $(kubectl -n "${NAMESPACE}" get pods \
-l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \
-o jsonpath='{.items[*].metadata.name}'); do
echo "=== ${pod} ==="
kubectl -n "${NAMESPACE}" exec "${pod}" -c bfdd -- bfdd-control status
kubectl -n "${NAMESPACE}" get pod "${pod}" \
-o jsonpath='bfdd restartCount={.status.containerStatuses[?(@.name=="bfdd")].restartCount}{"\n"}'
done
```

临时探针连续三次检测到 `There are 0 sessions:` 后,kubelet 会重启 `bfdd`。现有启动探针随后会重新配置 BFD 参数并注册对端。确认新的 BFD 会话已经创建并最终进入 `Up` 状态。

如果 `bfdd` 在反复重启后仍然显示零个会话,需要检查 VPC BFD 端口、OVN 侧 BFD 状态、对端连通性和网络路径。重启本地容器无法修复对端不可用或网络路径异常。
Loading