SDK: safe request-path defaults for verify() (1s timeout + fail-closed) - #2
Merged
Merged
Conversation
verify()/verify_batch() sit inline with the caller's traffic, but the client applied the same 10s timeout as control-plane calls and raised on any failure -- so a slow or down verifier stalled requests for up to 10s and turned an outage into an exception on the hot path. The docs already told users to fix this by hand (timeout=1.0, catch-and-fall-through); the SDK default now matches that advice. - verify_timeout (default 1.0) separate from timeout (still 10.0 for feedback/finetune/monitoring). Passed per-request so control-plane calls are unaffected. - On timeout / connection error / 5xx, verify() logs a warning and returns a synthetic VerifyResult(degraded=True, approved=<fail_open>) instead of raising. fail_open defaults to False (fall through to the LLM, same as a cache miss). 4xx still raises CacheVerifierError. - VerifyResult gains `degraded`; model_version is "verify_unavailable" on a synthesized result. - GPTCache adapter takes verify_timeout / fail_open and returns 0.0 on an outage by default. v0.2.0 -> v0.3.0. Co-authored-by: xin <xin@xindeMacBook-Pro.local>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
verify()/verify_batch()are on the caller's request path, but the clienttreated them like every other call: the 10s control-plane timeout, and an
exception on any failure. A slow or unreachable verifier stalled requests for up
to 10s and turned an outage into a hot-path exception. The docs already told
users to work around this by hand (
timeout=1.0+ catch-and-fall-through) — thismakes the SDK default match that advice.
Changes
verify_timeout(default1.0), separate fromtimeout(still10.0,used for feedback / fine-tune / monitoring). Applied per-request, so
control-plane calls are unchanged.
verify()logs a warning on the
cacheverifierlogger and returns a syntheticVerifyResult(degraded=True, approved=False)instead of raising — the callerfalls through to their LLM exactly as on a cache miss.
4xx(bad key, badrequest, rate limit) still raises
CacheVerifierError.fail_open=Trueflips the degraded result toapproved=True, for trafficwhere a stale-or-near-miss answer beats a regeneration.
VerifyResultgainsdegraded: bool;model_versionis"verify_unavailable"on a synthesized result.
verify_timeout/fail_open; returns0.0(don't reuse) on a verifier outage by default.
v0.2.0→v0.3.0. New tests cover the tight timeout, fail-closed/open ontimeout + 5xx, 4xx still raising, and batch degrading every pair.
ruff,mypy cacheverifier, fullpytestgreen locally.Compat note
Callers that wrapped
verify()intry/except CacheVerifierErrorfor transportfailures will now instead see
result.degraded is True— theexceptstillcatches real 4xx. Behavior on success is unchanged.