Add cacheverifier healthcheck -- offline stock-vs-fine-tuned evaluation - #1
Merged
Conversation
…tion `cacheverifier healthcheck traffic.jsonl` runs the same held-out AUC evaluation as the hosted Health Check Report (POST /v1/finetune/dry-run), but entirely on the user's machine -- for teams that can't upload real traffic to evaluate fit. Nothing is sent; `--emit-summary` writes an aggregate-only JSON file (AUCs, counts, rates -- no query/answer text). - New `cacheverifier` console script (`cacheverifier.__main__:main`), an argparse dispatcher. `--version` / `--help` work on the base install. - New `healthcheck` extra (numpy<2, torch, sentence-transformers<4); the base client stays httpx-only. Running the subcommand without the extra prints the pip install line and exits 2. - `cacheverifier/_healthcheck/_finetune.py`: a trimmed, self-contained port of the service's verifier_core.finetune -- chronological train/calibrate/test split, rank-based AUC, Youden's-J threshold, label-noise and ceiling diagnostics. Drops CRC certification, cost-ratio grids, and drift monitoring (server-only concerns). - CI: base `test` matrix covers dispatch + the missing-extra path; a new `healthcheck` job installs the extra (CPU-only torch) and runs the real fine-tune end to end. - Version 0.1.0 -> 0.2.0; README + CHANGELOG.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wraps
cacheverifier-service'sscripts/local_health_check.pyas a first-class command in this package.What
cacheverifier healthcheck traffic.jsonlruns the same stock-vs-fine-tuned held-out AUC evaluation as the hosted Health Check Report (POST /v1/finetune/dry-run), but entirely offline — for teams who can't upload real traffic to evaluate whether the product fits. Queries and answers never leave the machine;--emit-summarywrites an aggregate-only JSON file (AUCs, counts, rates — no text) that's safe to share for a human read.pip install "cacheverifier[healthcheck]" cacheverifier healthcheck traffic.jsonl --emit-summary summary.jsonInput: a JSON array or JSONL of
{query, candidate_answer, was_correct}rows in arrival order (optionalstale: trueper row). The train/calibrate/test split is chronological, matching the hosted service so the numbers are comparable.How it's packaged
cacheverifierconsole script (cacheverifier.__main__:main) — a small argparse dispatcher.--version/--helpwork on the base install.healthcheckextra (numpy<2,torch>=2.2,sentence-transformers<4). The base client stayshttpx-only, py3.9+. Runningcacheverifier healthcheckwithout the extra prints thepip installline and exits 2.cacheverifier/_healthcheck/_finetune.pyis a trimmed, self-contained port of the service'sverifier_core.finetune— chronological split, rank-based AUC, Youden's-J threshold, label-noise + ceiling diagnostics. Drops the server-only machinery (Conformal Risk Control certification, cost-ratio threshold grids, drift monitoring).numpy<2: the torch builds these versions resolve are compiled against the numpy 1.x ABI (Failed to initialize NumPy: _ARRAY_API not foundon numpy 2). Same pin the hosted service uses.CI
testmatrix (py3.9–3.13,.[dev]) exercises CLI dispatch, parsing, and the missing-extra path.healthcheckjob installs.[healthcheck,dev]with CPU-only torch (--index-url https://download.pytorch.org/whl/cpu— the default wheel drags in ~3 GB of unused CUDA libs) and runs the real fine-tune end to end.Testing
Local, both environments:
ruff✓,mypy✓,pytest18 passed / 1 skipped (e2e training ran),python -m build+twine check✓ruff✓,mypy✓,pytest18 passed / 1 skipped (e2e skipped, missing-extra path ran)Version bumped 0.1.0 → 0.2.0; README + CHANGELOG updated.
publish.ymlis release-triggered and version-agnostic — no change.