feat(proxy): deterministic substitution/failover engine (v0.9.1)#38
Merged
Conversation
Extends the Reliability Proxy /proxy/{slug} from single-hop into a config-driven,
multi-hop substitution engine. Deterministic layer only — ML-ranked selection is
a later phase.
- core/substitution.py: group loader (cached; ordered by wri_score when present,
else curated manual_rank — never sorts on nulls, never random), FailoverPolicy,
category validators, run_with_failover (classify -> retry-first -> idempotency
gate -> chain -> bill -> log), substitution_events writer.
- Substitution GROUPS are config-driven (substitution_groups table, seeded
web-search + llm-inference; weather/maps config-ready, no peers yet). A provider
is NEVER substituted outside its group.
- Failure detection distinguishes pre_send (conn refused, connect-timeout, 5xx,
429) vs post_send_ambiguous (read-timeout-after-send, invalid-body-after-200)
via httpx exception types — new _try_execute_managed_ex in execute.py.
- Billing: charges ONLY the served provider (+ its embedded 1.5% routing fee);
every failed hop nets to zero (deduct+refund, per-hop idempotency keys); the
surviving ledger row is the real served provider.
- Idempotency (#5, FLAGGED FOR REVIEW): pre-send always fails over; post-send on
the MANAGED rail fails over (user always refunded) under a cost cap, with a
same-provider retry-first and full instrumentation (duplicate_upstream_cost_
possible + measured second_upstream_cost on each event); x402/on-chain is STRICT
(no post-send failover). failover_post_send defaults true, flippable to strict.
- substitution_events log (migration 064, shared DB) — future learned-layer
training signal; nothing consumes it yet.
- Visible self-heal: X-Wayforth-Served-By + X-Wayforth-Fallback headers (+ wrap
envelope served_by/fallback). Group exhaustion -> clean 502 listing providers
tried + reasons.
- Guardrails: depth cap; per-provider creds via key_var (missing -> skip);
PRIMARY SUCCESS PATH UNTOUCHED (engine only invoked on the failure branch).
Migration 064 applied + verified on prod (additive/idempotent). 13 engine unit
tests green. VERSION 0.9.0 -> 0.9.1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n idempotency key Merge-review fixes (billing-critical, #5 guarantee): - Settlement matrix: WriteError/WriteTimeout move pre_send -> post_send_ambiguous (a partial write may have reached + been processed upstream). PoolTimeout moves post_send -> pre_send (no connection was ever acquired → request never sent). Now: ConnectError/ConnectTimeout/PoolTimeout=pre_send; ReadTimeout/WriteError/ WriteTimeout=post_send; 5xx/429 received=pre_send (fail-over-safe); unclassifiable=post_send (fail safe). Direct parametrized test of the matrix. - Retry-first idempotency gate: a same-provider retry after a POST-send failure only runs when the provider honors an idempotency key (_IDEMPOTENCY_KEY_PROVIDERS, empty today) — otherwise it would create the duplicate cost it's meant to prevent, so it is SKIPPED (straight to the cost-capped, instrumented substitution). pre_send retry is always safe. - Tests: classification matrix (httpx + http-status), post-send-retry-skipped, pre-send-retry-succeeds, cap-exceeded refund safety (primary still refunded). 24 engine tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends the Reliability Proxy
/proxy/{slug}from single-hop into a config-driven, multi-hop substitution/failover engine. Deterministic layer only — ML-ranked selection is a later phase. Built inapps/api(co-located with /proxy + billing; zero cross-service hop), per the approved plan.failover_post_send_max_cost, default 25 cr), preceded by a same-provider retry-first, with full instrumentation (duplicate_upstream_cost_possible+ measuredsecond_upstream_cost_creditsper event).failover_post_senddefaults true, flippable to strict via env if the measured leak turns material. The residual risk is Wayforth's duplicate upstream cost — now measured insubstitution_events, not assumed.What it does
substitution_groupstable, seeded web-search + llm-inference; weather/maps config-ready, no peers yet). A provider is never substituted outside its group. Ordered bywri_scorewhen present, else curatedmanual_rank— never sorts on nulls, never random (verified live: tavily wri 85.5 ranks first, null-wri peers fall to manual_rank)._try_execute_managed_exdistinguishes httpx ConnectError/ConnectTimeout/5xx/429 (pre_send) vs ReadTimeout/PoolTimeout/200-invalid-body (post_send).X-Wayforth-Served-By+X-Wayforth-Fallbackheaders (+ wrap envelope). Group exhaustion → clean 502 listing providers tried + reasons.key_var(missing → skip); primary success path untouched (engine only invoked on the failure branch — zero overhead).Verification
substitution_events(15 cols) created. Loader ordering validated live against real wri data.Not auto-merging — please review the idempotency (#5) policy above first.
🤖 Generated with Claude Code