Cap claude CLI usage and filter scraped API noise from discovery - #2
Open
msohailse wants to merge 2 commits into
Open
Cap claude CLI usage and filter scraped API noise from discovery#2msohailse wants to merge 2 commits into
msohailse wants to merge 2 commits into
Conversation
The CLI path had no ceiling on how many subprocess calls a single `applypilot run` could make. Score, judge, tailor, and cover all share one LLMClient, so a bad batch (e.g. a pile of junk scraped "jobs") could burn through the entire subscription window before the first stage even finished, with no way to stop it short of killing the process. - LLMClient tracks claude_cli_call_count against a CLAUDE_CLI_CALL_BUDGET env var (default 40); _chat_claude_cli raises ClaudeUsageLimitError before spawning another subprocess once the budget is hit, so the cap applies uniformly to every stage without each one having to check it. - scorer.py and cover_letter.py were catching ClaudeUsageLimitError inside a blind `except Exception`, converting it into a fake per-job error and continuing -- so even a real limit hit never stopped the loop, it just burned through every remaining job. Both now let it propagate and break the loop, flushing whatever's already done (same pattern tailor.py already had). - Bumped the CLI subprocess timeout from 120s to 240s -- the tailoring prompt (skills boundary, hard rules, full resume) is large enough that a cold `claude -p` call was hitting the old timeout on legitimate requests, not just rate limits.
collect_page_intelligence()'s response listener captures anything with "/api/" in the URL, which also catches telemetry, auth, and consent endpoints a job board's own frontend fires on every page load (e.g. talent.com's /api/auth/get-session and /api/telemetry/web-vitals, onetrust.com's geolocation lookup). These were reaching the LLM judge and, on judge error, getting kept and stored as "jobs" -- so a chunk of every run's LLM budget (see previous commit) went toward scoring/judging garbage that was never a job posting. - Added a denylist (_is_noise_url) applied at capture time, before a response is even considered for judging, and again as a backstop right before the DB insert in case anything slips through. - judge_api_responses() was also fail-open: on any error (LLM or otherwise) it kept the response rather than dropping it, which is how a real usage-limit hit turned into "keep everything for the rest of the batch." Now it fails closed and stops the judging pass entirely on a usage-limit hit instead of grinding through the rest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up safety work for the Claude CLI provider path (upstream Pickle-Pixel#96), kept as its own PR/diff so it's reviewable separately from the provider addition itself. Based on
add-anthropic-claude-providersince these commits modify_chat_claude_cli/ClaudeUsageLimitError, which only exist there until Pickle-Pixel#96 merges -- will retarget tomainonce it does.LLMClientnow enforces a sharedCLAUDE_CLI_CALL_BUDGET(default 40) across every stage (score/judge/tailor/cover), raising before spawning another subprocess once hit, instead of relying on each stage to remember to check. Also fixesscorer.py/cover_letter.pyswallowing the usage-limit exception inside a blindexcept Exceptionand burning through every remaining job anyway. Bumped the CLI subprocess timeout 120s -> 240s (legitimate tailoring prompts were hitting the old timeout, not just rate limits).smartextract.pywas capturing any response with/api/in the URL, including telemetry/auth/consent endpoints job boards fire on page load. These were reaching the LLM judge and, on judge error (fail-open), getting stored as jobs -- wasting a chunk of every run's LLM budget on garbage. Added a denylist at capture time and as a DB-insert backstop, and made the judge fail closed instead of open.Test plan
ruff check/ruff format --checkon all changed files -- no new findings beyond pre-existing debt in files touchedCLAUDE_CLI_CALL_BUDGETlow and running the pipeline live against my own Claude CLI subscriptionscorer.py/cover_letter.pynow stop cleanly on a real usage-limit hit instead of looping through every remaining job🤖 Generated with Claude Code