fix: isolate suspect server state per client

[hhlgt]

Synchronize suspect cleanup with roster refresh and make stale cleanup idempotent.

Summary

Fix the repeated RouteTableRefresher::doCheckAliveTask fail warnings that may occur after an Observer is taken offline.

This change isolates suspect-server state per ObTableClient, makes concurrent cleanup idempotent, and reconciles suspect state when the authoritative tenant roster is refreshed.

Fixes issue #436.

Background

After an Observer is replaced or taken offline, existing RPC connections or stale requests may report the old server as unavailable.

The client then adds the address to the suspect-server set and runs a JDBC keep-alive check every second. Normally, the server should be evicted after three consecutive failures.

With multiple ObTableClient instances in the same JVM, however, the suspect state was shared globally. Concurrent background refreshers could remove or process the same entry at the same time, causing cleanup exceptions and repeated warnings such as:

RouteTableRefresher::doCheckAliveTask fail, failed server: ObServerAddr{...}

Solution Description

This fix improves suspect-server lifecycle management with minimal behavioral changes:

• Isolates suspect-server state per ObTableClient instance instead of sharing it globally across clients.
• Uses thread-safe storage for server access timestamps.
• Tracks the active tenant server roster and prevents removed servers from being re-added to the suspect list by late failure callbacks.
• Cleans suspect entries, locks, and access timestamps when a server is removed from the tenant roster.
• Makes suspect-server removal idempotent and concurrency-safe, avoiding exceptions when entries or locks have already been removed.
• Adds null checks and safer exception handling to prevent concurrent cleanup from causing NullPointerException or repeated doCheckAliveTask fail warnings.
• Clears all instance-level suspect state when the route refresher is closed.
• Routes connection failure reports through the owning TableRoute, ensuring each client updates only its own suspect state.

The existing failure threshold, health-check interval, request-routing behavior, RS-list refresh policy, and tenant-roster refresh policy remain unchanged. No additional metadata queries or background tasks are introduced.

[CLAassistant]

All committers have signed the CLA.

[hhlgt]

AI Code Review 报告

审查提交：eeddf25 fix: isolate suspect server state per client
分支：bugfix/issue-436-suspect-isolation

审查结论

核心故障链路修复正确，未发现阻止解决 Issue #436 的功能性缺陷。实例级 suspect 状态、幂等清理、空值保护以及 inactive server 拦截能够避免原有每秒持续 WARN。

建议合入前确认 1 项 API 兼容性问题；其余为已知边界和低风险改进项。

Review Findings

[P1] 确认公开方法的二进制兼容性

原有公开静态方法：

public static void addIntoSuspectIPs(SuspectObServer server)

已被替换为：

public void addIntoSuspectIPs(ObServerAddr addr)

如果外部应用直接使用过旧方法，升级后会出现编译失败；未经重新编译的应用可能出现 NoSuchMethodError。

虽然该方法看起来属于客户端内部实现，但 Java 可见性是 public。合入前应确认 location.model 是否被视为非公开 API；如果不是，需要在版本说明中明确记录兼容性变化或设计兼容策略。

[P2] API_MOVE 新地址存在 active roster 状态窗口

[ObTableClient.java:940]可以根据服务端 reroute 响应动态创建 ObTable，并注入失败回调；但 activeServers 只从 serverRoster 初始化或在完整 tenant roster 刷新后更新。

如果 MOVE 返回的地址尚未进入当前 serverRoster，该地址的连接失败会被当作 inactive server 忽略。结果是该节点不会进入后台探活，直到后续元数据刷新收敛。

这不会重新引发 Issue #436 的持续 WARN，但可能削弱动态迁移窗口内的失败节点清理能力。可以作为已知边界记录，并补充对应测试。

非阻塞风险

• suspectLocks 为避免并发线程使用不同锁，被保留到 refresher 关闭。长生命周期、频繁节点替换的客户端会累计历史地址的 lock 对象，但每个地址只有一个轻量对象，当前风险较低。
• RS 集合不变但 tenant observer 已替换时，activeServers 可能暂时保留旧地址。持续的迟到失败上报可能每隔 20 秒触发一轮有限探活，但幂等清理可以避免每秒持续 WARN。
• 新增单元测试主要通过反射调用内部清理方法，尚未直接覆盖 JDBC checkAlive、scheduler 与 roster 刷新的并发交错；真实 OB 验证可补足主要行为，但建议后续增加确定性并发单测。

正向评价

• 将静态状态改为实例状态，正确消除了多个客户端之间的 suspect 状态污染。
• serverLastAccessTimestamps 改为 ConcurrentHashMap，修复并发访问风险。
• failure callback 携带所属 TableRoute，owner 关系明确。
• inactive server 在加锁前后均进行校验，可阻止迟到上报重新激活已下线节点。
• 清理操作支持 lock 缺失、条目已删除和并发移除，不再抛出原有异常。
• calcFailureOrClearCache 的空值保护解决并发删除后的 NPE。
• close() 幂等并恢复线程中断标志，生命周期处理合理。
• SQLException 与 ObTableEntryRefreshException 改为互斥分支，避免一次异常被重复处理。
• git diff --check 未发现空白或格式问题。

最终建议：确认 P1 API 兼容性后可以合入；P2 和其他边界可以记录为后续增强项，不阻塞本次 Issue #436 核心修复。

[WeiXinChan]

LGTM