Add retry mechanism and dynamic model loading to SONIC framework

[kakwok]

PR description:

Introduces a pluggable retry mechanism for SONIC inference clients and dynamic
model loading/unloading on the Triton fallback server. This enables automatic
recovery when a remote Triton server becomes unavailable during event processing.

Retry mechanism (SonicCore)

• RetryActionBase: Plugin factory base class for retry strategies, with
  retry(), start(), finish() interface
• RetrySameServerAction: Retries inference on the same server (configurable
  allowedTries)
• SonicClientBase::finish(): Drives retry loop — on retryable failure
  (finish(false)), iterates through registered retry actions; on non-retryable
  failure (finish(false, eptr)), propagates exception directly

Retry with server failover (SonicTriton)

• RetryActionDiffServer: On failure, queries TritonService::getBestServer()
  for an alternative healthy remote server, calls updateServer() + eval() to
  retry on the new server
• RetryFallbackServerAction: Last-resort action — when all other retries are
  exhausted, lazily starts the fallback server (idempotent), dynamically loads the
  model to the fallback server via TritonClient::switchToFallback(), and retries locally. Fires at most
  once per inference call.
• Server health tracking: TritonService maintains per-server health stats
  (liveness, readiness, failure count, queue time) via updateServerHealth(),
  used by getBestServer() to select the best candidate
• Server selection preference: resolveServerName() prefers remote (non-fallback)
  servers when multiple servers provide the same model; getBestServer() excludes
  fallback servers from candidate pool

Async retry redesign (holder-based)

• evaluate() (Async mode) creates an inner WaitingTaskWithArenaHolder per
  inference attempt — the gRPC callback calls doneWaiting() and returns
  immediately; finish()/retry()/updateServer() are scheduled as TBB tasks

Dynamic model loading (TritonService)

• loadModel()/unloadModel() with reference counting for the fallback server
• startFallbackServer() is idempotent; only started at preBeginJob for
  unassigned models (no remote server available)
• Fallback server health entry properly registered in serversHealth_

Configuration

• Retry actions configurable via Retry VPSet in client config
  (retryType, allowedTries)
• customize.py updated with retry options, defaulted to add RetryFallbackServerAction as the last action.

PR validation:

retry_diffServer_fallback.log

[kpedro88]

is the explicit RetryActionPtr() needed here? usually for emplace_back(), it shouldn't be

[kakwok]

Yes, because I think the RetryAction needs a custom deleter here, which is needed because it's a friend class of the SonicClient.
https://github.com/kakwok/cmssw/blob/SonicRetryDML_CMSSW_17_0_0_pre2/HeterogeneousCore/SonicCore/interface/SonicClientBase.h#L63-L69

Here's some more details from Claude:
RetryActionFactory::create() returns a std::unique_ptr<RetryActionBase> using the default deleter (std::default_delete<RetryActionBase>), since that's what edmplugin::PluginFactory hands back.
But retryActions_ stores your custom
RetryActionPtr = std::unique_ptr<RetryActionBase, RetryDeleter> — a different type, because RetryDeleter is a distinct functor from std::default_delete<RetryActionBase>.

[kpedro88]

do these tests work without a driver script to enable/disable servers?

[kakwok]

They will pass but they does not do what they suggest to test.
The retryDiff test will require some fiddling because the server choice is still undeterministic among remote servers.
I can commit a retry same script retry_action_same.sh for now.

[kakwok]

done in b9a4f8e

[kpedro88]

We will also have to update the various reco/miniaod algorithm configuration files that currently set allowedTries = cms.untracked.uint32(0)

[kakwok]

Configs are updated as well Update allowedTries config

[kakwok]

@kpedro88 On the topic of configuration, should we add a RetryFallbackServerAction to all configurations as default?