Skip to content

SDK: safe request-path defaults for verify() (1s timeout + fail-closed) - #2

Merged
imxinchengyou merged 1 commit into
mainfrom
sdk-request-path-defaults
Sep 4, 2026
Merged

SDK: safe request-path defaults for verify() (1s timeout + fail-closed)#2
imxinchengyou merged 1 commit into
mainfrom
sdk-request-path-defaults

Conversation

@imxinchengyou

Copy link
Copy Markdown
Owner

What

verify() / verify_batch() are on the caller's request path, but the client
treated them like every other call: the 10s control-plane timeout, and an
exception on any failure. A slow or unreachable verifier stalled requests for up
to 10s and turned an outage into a hot-path exception. The docs already told
users to work around this by hand (timeout=1.0 + catch-and-fall-through) — this
makes the SDK default match that advice.

Changes

  • verify_timeout (default 1.0), separate from timeout (still 10.0,
    used for feedback / fine-tune / monitoring). Applied per-request, so
    control-plane calls are unchanged.
  • Fail-closed by default. On a timeout / connection error / 5xx, verify()
    logs a warning on the cacheverifier logger and returns a synthetic
    VerifyResult(degraded=True, approved=False) instead of raising — the caller
    falls through to their LLM exactly as on a cache miss. 4xx (bad key, bad
    request, rate limit) still raises CacheVerifierError.
  • fail_open=True flips the degraded result to approved=True, for traffic
    where a stale-or-near-miss answer beats a regeneration.
  • VerifyResult gains degraded: bool; model_version is "verify_unavailable"
    on a synthesized result.
  • GPTCache adapter takes the same verify_timeout / fail_open; returns 0.0
    (don't reuse) on a verifier outage by default.

v0.2.0v0.3.0. New tests cover the tight timeout, fail-closed/open on
timeout + 5xx, 4xx still raising, and batch degrading every pair. ruff,
mypy cacheverifier, full pytest green locally.

Compat note

Callers that wrapped verify() in try/except CacheVerifierError for transport
failures will now instead see result.degraded is True — the except still
catches real 4xx. Behavior on success is unchanged.

verify()/verify_batch() sit inline with the caller's traffic, but the
client applied the same 10s timeout as control-plane calls and raised on
any failure -- so a slow or down verifier stalled requests for up to 10s
and turned an outage into an exception on the hot path. The docs already
told users to fix this by hand (timeout=1.0, catch-and-fall-through); the
SDK default now matches that advice.

- verify_timeout (default 1.0) separate from timeout (still 10.0 for
  feedback/finetune/monitoring). Passed per-request so control-plane
  calls are unaffected.
- On timeout / connection error / 5xx, verify() logs a warning and
  returns a synthetic VerifyResult(degraded=True, approved=<fail_open>)
  instead of raising. fail_open defaults to False (fall through to the
  LLM, same as a cache miss). 4xx still raises CacheVerifierError.
- VerifyResult gains `degraded`; model_version is "verify_unavailable" on
  a synthesized result.
- GPTCache adapter takes verify_timeout / fail_open and returns 0.0 on an
  outage by default.

v0.2.0 -> v0.3.0.

Co-authored-by: xin <xin@xindeMacBook-Pro.local>
@imxinchengyou
imxinchengyou merged commit 8013043 into main Sep 4, 2026
7 checks passed
@imxinchengyou
imxinchengyou deleted the sdk-request-path-defaults branch September 4, 2026 05:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant