[blocked on #3191] Lag-aware availability checks for geo-redundant failover: findings and plan

[mgravell]

Important

Blocked on #3191, and not only a planning branch.

This currently adds notes and no product code. It is opened as a draft now so the investigation is
reviewable while the dependency clears — this branch becomes the feature branch once
#3191 merges, with the
implementation landing on top of these same commits rather than starting again elsewhere.

The plan is a proposal. Where it leans, it says so; where the evidence does not decide, it leaves
the question open rather than inventing an answer.

What this is

Investigating whether Redis Enterprise's
lag-aware database-availability API
belongs in the geo-redundant failover health checks in src/StackExchange.Redis/Availability/
(currently behind SER007), using
Lettuce's LagAwareStrategy
as the reference implementation.

file	what it is
notes/lag-aware/findings.md	the research — evidence, with sources
notes/lag-aware/plan.md	the sequence that follows from it — proposed, blocked on step 0
notes/readme.md	the notes/ convention (see "overlap" below)

Findings worth surfacing

• This is a cross-client feature, not a Java curiosity. Lettuce and Jedis have
  LagAwareStrategy; redis-py has LagAwareHealthCheck. It is the client-side geographic failover
  family. The check is not liveness — extend_check=lag asks whether a replica is synchronised
  enough to be a safe failover target, which is what stops a failback onto a region that is
  reachable but stale.
• Our abstractions already fit, nearly one-to-one. HealthCheckProbe ↔ AbstractHealthCheck,
  AllSuccess/AnySuccess/MajoritySuccess ↔ HEALTHY_ALL/ANY/MAJORITY, Ping default,
  circuit breaker, per-member configuration. Two details make it land cleanly: HealthCheckProbe
  has no internal abstract members, so it is already subclassable from outside the assembly; and
  ConnectionGroupMember already carries a per-member HealthCheck override, which is exactly what
  a per-cluster REST endpoint needs.
• The defaults disagree across clients. 100 ms from the cluster default, the REST documentation's
  explicit recommendation, and redis-py — but 5000 ms in Lettuce. Fifty times more permissive
  than its own documentation advises. The plan measures rather than copies.
• We are better placed than Lettuce on one point. HealthCheckResult has Inconclusive;
  HealthStatus has two states, and Lettuce maps a failed REST call to UNHEALTHY — so a
  management-plane outage is indistinguishable from the data plane being down, and can fail over a
  database that is serving traffic perfectly. Proposed as a deliberate, documented deviation.
• src/ may need no changes at all. The probe is subclassable, RawConfig and
  ConfigurationOptions.Tunnel are public. The only candidate core change is an optional Tunnel
  addition so the management plane honours tunnels — and if we do that, it should return a stream,
  composed into SocketsHttpHandler.ConnectCallback by the consumer, so no System.Net.Http type
  enters the public API.

Corrections made while investigating

Recorded rather than quietly fixed, because the reasoning is the useful part:

1. I assumed a real cluster could not be made to show "reachable but stale" on demand, making fakes
   the only route to the scenario that matters. Wrong twice: Redis ships a fault injection service
   for exactly this, and we already have a test tier built on it — in Server-native maintenance notifications: opt in, react, hand off, and recover when nothing is announced #3191.
2. I framed the HTTP dependency as a significant cost. It is not: System.Net.Http needs no package
   reference on any target we ship. System.Text.Json is the only genuinely new dependency, and it
   is avoidable. So "where does this live" is a judgement call about scope, not a technical
   constraint.

Why step 0 is "wait"

#3191 carries tests/StackExchange.Redis.FaultInjector.Tests, and with it ClusterRestClient —
already a Redis Enterprise REST client on 9443 doing Basic auth, CA pinning and
GET /v1/bdbs?fields=… — plus a FaultInjectorClient that can be asked which lag-inducing effects
a deployment supports. Starting now would mean writing a second REST client against the same cluster
and reconciling them at merge, for a feature whose implementation is smaller than the client that
tests it.

That tier also settled three things empirically that I had only from documentation: the management
certificate is self-signed per environment (so the probe needs CA configuration, not ambient
trust), auth really is Basic, and the bdbs listing returns the requested fields.

The one thing that can start earlier is validating the availability API against a real deployment,
because it produces notes rather than code — and #3191's own summary is the argument for doing it
first: "the specifications are prose, the payloads were never published, and nearly every assumption
I started with was wrong in some way that mattered."

Deliberately undecided

• Where the code ships — core, a separate Enterprise package, or documented extension point. The
  plan leans toward a separate package and says why, but this is a scope judgement.
• The tolerance default — measured in step 1, not copied.
• Whether the Tunnel addition happens at all — separable; the probe works without it.

Overlap to be aware of

notes/readme.md also appears in
#3241, which introduced the
notes/ convention. Whichever merges second gets a one-line conflict in its Topics list; the rest of
the file is identical.

Draft until #3191 lands.