Self-improving site routing: curated rules + failure classification + residential probe loop

[myleshorton]

Self-improving site routing

Wick now closes a feedback loop: it learns which sites it fails on from the public stats, empirically tests access methods through residential proxies, and maintains a shared curated "known behaviors" list so every client routes correctly on its first visit — instead of each machine independently re-paying the Cronet‑fail‑then‑escalate cost.

sequenceDiagram
    autonumber
    participant C as Wick client<br/>fetch.rs
    participant W as Worker<br/>index.js
    participant H as Probe harness<br/>bench/probe.sh
    participant R as Residential proxy<br/>oxylabs

    C->>W: POST /v1/events host, strategy, ok, error_kind
    Note over W: aggregate per host:<br/>success_rate + error_kind_dist
    H->>W: GET /v1/stats/summary
    Note over H: select site-side failing hosts<br/>drop error_kind=offline ⚠️
    H->>R: fetch via cronet / cronet+res / cef
    R-->>H: per-strategy status + bytes
    Note over H: derive render + needs_residential
    H->>W: POST /v1/site-rules merged seed + measured
    C->>W: GET /v1/site-rules daily refresh
    W-->>C: curated rules
    Note over C: site_rules consulted ABOVE site_cache<br/>right method on first visit ✅

Loading

What's in this PR

1. Curated site-rules (the evolving list) — rust/src/site_rules.rs + rust/data/site-rules.json

• Consulted above the per-machine site_cache: a curated render: cef / needs_residential rule routes a first-time visit correctly with no local learning.
• Bundled seed (include_str!) overlaid by ~/.wick/site-rules.json, refreshed daily from the Worker — so the list evolves without a reinstall.
• fetch.rs now threads the rule's residential flag + selector into every CEF path (the old hardcoded use_residential: false is gone).

2. Transport-failure classification — the data-quality foundation the loop depends on

• The old cronet-transport-error bucket couldn't tell a user disconnect from a site actively blocking us — so a user whose wifi died 25× looked identical to a hard site and would poison the rules.
• cronet::on_failed now captures the real net-error (was discarded); classify_transport_error gates every non-definitive cause on a connectivity probe → offline / dns / timeout / reset / refused / unreachable / quic / connect / other.
• Worker aggregates error_kind_dist alongside status_dist.

3. The probe harness — bench/probe.sh, bench/publish-rules.sh, bench/PROBE.md

• Reads /v1/stats/summary, selects site-side failing hosts (drops offline-dominated so user disconnects aren't chased), probes a cronet | cronet+residential | cef matrix via wick fetch --json, derives render + needs_residential.
• publish-rules.sh merges measured verdicts with the seed (measured wins, correcting over-aggressive seeds) → POST /v1/site-rules.

4. Propagation — Worker GET /v1/site-rules (public, cached) + POST /v1/site-rules/:key (auth); client site_rules::refresh_if_stale() (once/process, background, atomic, opt-out-aware).

Verified

• Built and tested in both the default and --features cronet configs (the real shipping path); 28 unit tests pass, including rule-precedence and cause-taxonomy tests. Worker passes node --check.
• Ran live against oxylabs US residential (3/3 clean exits). The loop auto-corrected 5 over-aggressive hand-seeds (reuters/cfr/tradingview/apkmirror/apkcombo all work on plain Cronet from a clean vantage — the telemetry "100% failures" were vantage/noise) and preserved the seed for apkpure (DataDome — failed every testable cell).
• A prior 3-agent adversarial review of part 1 surfaced and fixed 6 issues (CEF daemon residential-mode reuse, universal offline gate, proxy-aware probe, worker key-cardinality, dup host-walk, dead code).

Not in this PR / follow-ups

• Worker is not yet deployed (npx wrangler deploy activates error_kind_dist + the site-rules endpoints) and no rules are published yet.
• Methodology caveat (see PROBE.md): the cronet baseline cell uses the operator's own IP — run the harness from a datacenter VM to detect needs_residential faithfully. And --proxy routes Cronet, not CEF (CEF residential is a WireGuard preload), so cef+residential isn't tested here.
• PR4: a weekly Claude curation agent to review anomalies and propose genuinely new methods.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR implements a closed-loop “self-improving routing” system: clients emit richer failure telemetry, the Worker aggregates it (including transport-failure causes), a probe harness measures best strategies (Cronet/Cronet+proxy/CEF), and curated per-site rules are published for clients to consume on first visit.

Changes:

• Add curated site-rules: bundled seed + on-disk overlay refreshed daily from the Worker, consulted above the per-machine site_cache.
• Add transport-failure cause classification (with a connectivity probe gate) and propagate error_kind_dist through Worker aggregation.
• Add probe + publish scripts to measure and push merged rules to the Worker, plus CLI --json output for deterministic probing.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
worker/src/index.js	Adds error_kind_dist allowlisting/aggregation and new public/private site-rules endpoints.
rust/src/site_rules.rs	Introduces curated site-rules (seed + overlay), lookup semantics, and background refresh logic.
rust/src/site_cache.rs	Refactors parent-domain walk into shared helper for consistent cache/rules scoping.
rust/src/main.rs	Adds wick fetch --json output mode for probe harness consumption.
rust/src/fetch.rs	Consults curated rules in routing decisions; adds transport error classification with connectivity probe.
rust/src/cronet/mod.rs	Captures Cronet net-error codes into error messages for downstream classification.
rust/src/cronet/ffi.rs	Exposes Cronet error-code accessor needed for failure classification.
rust/src/cef.rs	Ensures CEF daemon is respawned when residential mode changes (singleton mode correctness).
rust/src/analytics.rs	Adds report_transport_error including error_kind for Worker aggregation.
rust/data/site-rules.json	Adds initial bundled seed rules document.
bench/publish-rules.sh	Publishes merged seed+measured rules to Worker (measured wins).
bench/proxy-providers.sh	Updates Oxylabs proxy scheme to HTTP CONNECT for compatibility.
bench/probe.sh	Adds probing harness: candidate selection from stats + strategy matrix + measured rules output.
bench/PROBE.md	Documents harness usage, scheduling, and methodology caveats.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cloudflare-workers-and-pages]

Deploying wickproject with    Cloudflare Pages

Latest commit:	2e061b0
Status:	✅  Deploy successful!
Preview URL:	https://cf42b53e.wickproject.pages.dev
Branch Preview URL:	https://self-improving-site-rules.wickproject.pages.dev

View logs

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.