Cap claude CLI usage and filter scraped API noise from discovery - #99
Closed
msohailse wants to merge 3 commits into
Closed
Cap claude CLI usage and filter scraped API noise from discovery#99msohailse wants to merge 3 commits into
msohailse wants to merge 3 commits into
Conversation
Adds two ways to use Claude for scoring/tailoring, alongside the existing Gemini/OpenAI/local auto-detection in llm.py: - ANTHROPIC_API_KEY: native Anthropic Messages API integration. Needed its own code path rather than reusing the OpenAI-compatible one, since Anthropic takes `system` as a top-level field (not a message role) and uses x-api-key/anthropic-version headers with a different response shape. Defaults to claude-haiku-4-5, matching the existing cost-conscious defaults for the other providers. - USE_CLAUDE_CLI: shells out to the `claude` CLI in print mode instead of hitting the API directly, so scoring/tailoring rides an existing Claude subscription rather than metered per-token billing. `claude -p` takes a single prompt string, not a chat history, so multi-turn messages are flattened (this app's LLM calls are already single-turn); --disallowedTools "*" keeps it a pure text completion with no file/bash access. Both paths tested against the live app's llm.py: - ANTHROPIC_API_KEY: request reached Anthropic and was accepted as well-formed (rejected only for zero account credit balance, not a bad request) -- confirms the request/response wiring is correct. - USE_CLAUDE_CLI: full round trip succeeded end-to-end through get_client().chat(), including system-prompt splitting, using the existing `claude` CLI login. .env.example documents both new options. ruff-checked and ruff-formatted (line-length=120 per pyproject.toml) -- clean on the added code; two pre-existing findings elsewhere in the file (an unused `exc` binding and some formatting drift) predate this change and are left alone to keep the diff scoped to this feature. Also wires the new providers into tier detection: get_tier()/check_tier() in config.py and the `doctor` command in cli.py previously only recognized GEMINI_API_KEY, OPENAI_API_KEY, and LLM_URL, so setting ANTHROPIC_API_KEY or USE_CLAUDE_CLI alone left the CLI reporting Tier 1 and blocking scoring/tailoring/apply even though llm.py fully supported those providers. .env.example | 6 ++- src/applypilot/cli.py | 13 ++++- src/applypilot/config.py | 9 +- src/applypilot/llm.py | 118 +++++++++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 141 insertions(+), 5 deletions(-)
The CLI path had no ceiling on how many subprocess calls a single `applypilot run` could make. Score, judge, tailor, and cover all share one LLMClient, so a bad batch (e.g. a pile of junk scraped "jobs") could burn through the entire subscription window before the first stage even finished, with no way to stop it short of killing the process. - LLMClient tracks claude_cli_call_count against a CLAUDE_CLI_CALL_BUDGET env var (default 40); _chat_claude_cli raises ClaudeUsageLimitError before spawning another subprocess once the budget is hit, so the cap applies uniformly to every stage without each one having to check it. - scorer.py and cover_letter.py were catching ClaudeUsageLimitError inside a blind `except Exception`, converting it into a fake per-job error and continuing -- so even a real limit hit never stopped the loop, it just burned through every remaining job. Both now let it propagate and break the loop, flushing whatever's already done (same pattern tailor.py already had). - Bumped the CLI subprocess timeout from 120s to 240s -- the tailoring prompt (skills boundary, hard rules, full resume) is large enough that a cold `claude -p` call was hitting the old timeout on legitimate requests, not just rate limits.
collect_page_intelligence()'s response listener captures anything with "/api/" in the URL, which also catches telemetry, auth, and consent endpoints a job board's own frontend fires on every page load (e.g. talent.com's /api/auth/get-session and /api/telemetry/web-vitals, onetrust.com's geolocation lookup). These were reaching the LLM judge and, on judge error, getting kept and stored as "jobs" -- so a chunk of every run's LLM budget (see previous commit) went toward scoring/judging garbage that was never a job posting. - Added a denylist (_is_noise_url) applied at capture time, before a response is even considered for judging, and again as a backstop right before the DB insert in case anything slips through. - judge_api_responses() was also fail-open: on any error (LLM or otherwise) it kept the response rather than dropping it, which is how a real usage-limit hit turned into "keep everything for the rest of the batch." Now it fails closed and stops the judging pass entirely on a usage-limit hit instead of grinding through the rest.
Author
|
Reopening as a fork-internal PR (base = add-anthropic-claude-provider) to get a pure 2-commit diff without the provider commit mixed in. Will retarget to main once #96 merges. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up safety work for the Claude CLI provider path (#96). This branch is built on top of
add-anthropic-claude-provider, so the diff below includes that commit plus 2 new ones -- sincemaindoesn't have the provider code yet, GitHub will show all 3 until #96 merges. The 2 new commits are the actual content of this PR:LLMClientnow enforces a sharedCLAUDE_CLI_CALL_BUDGET(default 40) across every stage (score/judge/tailor/cover), raising before spawning another subprocess once hit, instead of relying on each stage to remember to check. Also fixesscorer.py/cover_letter.pyswallowing the usage-limit exception inside a blindexcept Exceptionand burning through every remaining job anyway. Bumped the CLI subprocess timeout 120s -> 240s (legitimate tailoring prompts were hitting the old timeout, not just rate limits).smartextract.pywas capturing any response with/api/in the URL, including telemetry/auth/consent endpoints job boards fire on page load. These were reaching the LLM judge and, on judge error (fail-open), getting stored as jobs -- wasting a chunk of every run's LLM budget on garbage. Added a denylist at capture time and as a DB-insert backstop, and made the judge fail closed instead of open.Test plan
ruff check/ruff format --checkon all changed files -- no new findings beyond pre-existing debt in files touchedCLAUDE_CLI_CALL_BUDGETlow and running the pipeline live against my own Claude CLI subscriptionscorer.py/cover_letter.pynow stop cleanly on a real usage-limit hit instead of looping through every remaining job🤖 Generated with Claude Code