Skip to content

Weekly site-rules curation agent (self-improvement loop, part 4)#10

Open
myleshorton wants to merge 2 commits into
self-improving-site-rulesfrom
wick-curate-agent
Open

Weekly site-rules curation agent (self-improvement loop, part 4)#10
myleshorton wants to merge 2 commits into
self-improving-site-rulesfrom
wick-curate-agent

Conversation

@myleshorton

Copy link
Copy Markdown
Member

Weekly site-rules curation agent — the "invent new methods" layer

The judgment layer of the self-improvement loop, on top of the deterministic probe harness (#9). The harness measures a fixed matrix (cronet | cronet+residential | cef) and corrects rules from it; this agent looks at what the matrix couldn't crack and reasons about methods it doesn't try.

Stacked on #9 (base branch self-improving-site-rules) — review/merge #9 first.

What it adds

  • bench/curate-inputs.sh — read-only, creds-free. Gathers the weekly inputs into one JSON:
    • failing — site-side failing hosts with a per-cause breakdown (reset/refused/timeout/403…), so the agent reasons about why.
    • hard — hosts where the latest probe sweep failed every cell (verified live: apkpure.com).
    • published_rules — what clients are currently served.
  • agent-skill/wick-curate/SKILL.md — reasons about anomalies and proposes + tests new methods in priority order: different residential country (geo-blocks), wait_for_selector (un-hydrated SPAs), URL rewrites (à la old.reddit), CEF+residential (DataDome-class), else file an issue.

Guardrails

  • Never publish a rule not backed by a passing probe — a hypothesis is not a rule.
  • Respect the offline signal — high user-offline fraction means it's the users' network, not the site; excluded from "hard sites."
  • Composes with the existing scripts (probe.sh, publish-rules.sh, /residential-proxy); methodology caveats in bench/PROBE.md.

🤖 Generated with Claude Code

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 26, 2026

Copy link
Copy Markdown

Deploying wickproject with  Cloudflare Pages  Cloudflare Pages

Latest commit: 14edda0
Status: ✅  Deploy successful!
Preview URL: https://d02ad879.wickproject.pages.dev
Branch Preview URL: https://wick-curate-agent.wickproject.pages.dev

View logs

@myleshorton myleshorton force-pushed the wick-curate-agent branch 2 times, most recently from f1135e1 to 9e2107a Compare June 26, 2026 22:04
Adds the judgment layer of the self-improvement loop on top of the
deterministic probe harness. bench/curate-inputs.sh gathers the weekly inputs
(stats with per-cause failure breakdown, latest probe traces, currently
published rules) into one JSON; agent-skill/wick-curate/SKILL.md reasons about
what the fixed matrix couldn't crack — sites failing EVERY cell (e.g. apkpure),
high user-offline noise, seed/measured conflicts — and proposes + tests
genuinely new methods (different residential country, wait_for_selector, URL
rewrites, CEF+residential), then re-probes and republishes.

Guardrail: never publish a rule that isn't backed by a passing probe.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Same belt-and-suspenders as probe.sh for the host aggregation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant