locate: preserve connector TLS for store liveness probes

[idy]

Transactional RPC clients can use connector-specific TLS via tikv.WithSecurity, but store recovery health checks construct a new gRPC client using config.GetGlobalConfig().Security. When the connector uses mTLS and global security is empty, normal reads and writes succeed until a transient transport failure starts liveness checking. The health connection then uses plaintext, fails with EOF, and the store remains unreachable even after its transport recovers.

This adds optional tikv.WithStoreLivenessSecurity(security) for tikv.NewKVStore, allowing recovery probes to use the connector's CA and client identity without mutating global configuration. Omitting the option retains the existing global-security default; an explicitly empty policy selects plaintext. Each store cache/security policy owns its singleflight group, so connectors with different credentials cannot share an authenticated health result for the same address.

To reproduce the original problem:

1. Run a PD/TiKV cluster requiring a private CA and client certificates.
2. Construct the PD/RPC clients with that connector's credentials while leaving SDK global security empty. Initial transactions succeed.
3. Temporarily interrupt the TiKV transport to trigger store liveness checking, then restore it.
4. Observe plaintext health attempts and continued store-unreachable/no-replica errors. With the new store option, the same client recovers using mTLS.

Validation passed for the TLS fix on merged master 138b7bc; the subsequent test-only listener-context correction also passed its focused race and full lint checks:

• go test -tags=intest ./...
• go test -race -tags=intest ./internal/locate ./tikv
• go test -race -tags=intest ./internal/locate -run '^TestStoreLiveness' -count=1 after the test-listener correction
• golangci-lint v2.8.0 with the repository configuration: 0 issues
• git diff --check

New loopback gRPC mTLS regressions cover independent CAs, wrong CA and missing client certificate rejection, explicit plaintext versus the legacy global default, concurrent same-address credential isolation, changing credentials while a probe is in flight, and background NOT_SERVING-to-SERVING recovery.

A downstream real PD/TiKV 8.5.6 test with two independent CAs also passed: after an injected transport interruption, the original client recovered, the other connector remained available, no plaintext connections were observed, and SDK global security stayed unchanged. That fixture is downstream validation and is not included in this PR; the reproducible SDK-level regressions are included here.

Existing TiDB/TiCDC/BR/CSE consumers retain their current behavior unless they opt in. The patch changes neither key encoding nor transaction semantics and does not disable certificate or hostname verification.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign zhongzc for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

Hi @idy. Thanks for your PR.

I'm waiting for a tikv member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

Welcome @idy!

It looks like this is your first PR to tikv/client-go 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!



Thank you, and welcome to tikv/client-go. 😃

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 9de8ba58-ce46-4fc7-b5ad-023e3eb52c81

📥 Commits

Reviewing files that changed from the base of the PR and between 138b7bc and 628e775.

📒 Files selected for processing: 1

• internal/locate/store_cache_tls_test.go

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 0 remain after this review.

📝 Walkthrough

Store liveness probes can use connector-specific TLS settings. Probe deduplication is scoped to each security policy. The new option and documentation describe explicit plaintext behavior and the global-security default when the option is omitted.

Changes: Store Liveness TLS

Layer / File(s)	Summary
Expose store liveness security configuration tikv/kv.go, internal/locate/region_cache.go, README.md	Adds WithStoreLivenessSecurity and a region-cache setter. The README describes connector-specific probe security, explicit plaintext settings, and the global default when the option is omitted.
Apply security per liveness policy internal/locate/store_cache.go	Gives each liveness policy its own singleflight group. Probes use the selected policy's security settings. A nil policy uses global security; an explicitly supplied policy without a CA uses plaintext.
Validate TLS and probe isolation internal/locate/store_cache_tls_test.go	Adds tests for mTLS, plaintext overrides, isolation of in-flight probes across policies, and recovery after a health service resumes serving.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant Connector
  participant WithStoreLivenessSecurity
  participant RegionCache
  participant storeCache
  participant createKVHealthClient
  Connector->>WithStoreLivenessSecurity: provide security settings
  WithStoreLivenessSecurity->>RegionCache: set store liveness security
  RegionCache->>storeCache: install policy
  storeCache->>createKVHealthClient: create probe client with policy security

Loading

Merge Risk: ⚪ Minimal · up to 628e7

The connector-specific option preserves the documented security behavior, and no actionable regression is established. The change appears mergeable subject to normal checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 15 functions across 4 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: preserving connector-specific TLS for store liveness probes.

• Fix all pre-merge checks with AI

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}