Add cacheverifier healthcheck -- offline stock-vs-fine-tuned evaluation

.github/workflows/ci.yml

@@ -24,6 +24,27 @@ jobs:
       - run: mypy cacheverifier
         if: matrix.python-version != '3.9'  # mypy config targets 3.10+
 
+  # `cacheverifier healthcheck` -- the base `test` job above installs only
+  # `.[dev]`, so it exercises the CLI dispatch and the missing-extra path;
+  # this job installs `.[healthcheck,dev]` and runs the real fine-tune
+  # end-to-end. One Python version is enough (the training path isn't
+  # version-sensitive) and CPU-only torch keeps the install lean -- the
+  # default wheel drags in ~3 GB of unused CUDA libraries whose import-time
+  # mmap alone can OOM a small runner.
+  healthcheck:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - run: python -m pip install --upgrade pip
+      - run: pip install torch --index-url https://download.pytorch.org/whl/cpu
+      - run: pip install -e ".[healthcheck,dev]"
+      - run: ruff check .
+      - run: mypy cacheverifier
+      - run: pytest -q tests/test_healthcheck.py
+
   build:
     runs-on: ubuntu-latest
     steps:

CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## 0.2.0
+
+Add a `cacheverifier` console script with a `healthcheck` subcommand:
+`cacheverifier healthcheck traffic.jsonl` runs the hosted Health Check
+Report's stock-vs-fine-tuned held-out AUC evaluation (`POST /v1/finetune/dry-run`)
+entirely offline — no queries or answers leave the machine. Needs the new
+`healthcheck` extra (`pip install "cacheverifier[healthcheck]"`: torch,
+sentence-transformers, numpy); the base client stays `httpx`-only.
+`--emit-summary` writes an aggregate-only JSON file with no text.
+
 ## 0.1.0
 
 Initial release: thin `httpx`-based client for `/v1/verify`, `/v1/verify/batch`,

README.md

@@ -23,9 +23,12 @@ zone", where a plain threshold match might be wrong.
 pip install cacheverifier
 # with the GPTCache adapter:
 pip install "cacheverifier[gptcache]"
+# with the offline Health Check (adds torch + sentence-transformers):
+pip install "cacheverifier[healthcheck]"
 ```
 
-Requires Python 3.9+. The only runtime dependency is `httpx`.
+Requires Python 3.9+. The only runtime dependency is `httpx` — the extras above
+are opt-in.
 
 ## Quickstart
 
@@ -95,6 +98,41 @@ if job.get("result_model_version"):
 `cv.dry_run([...])` reports the same baseline-vs-tuned AUC on examples you pass directly,
 without writing anything or deploying a model.
 
+## Local Health Check (offline)
+
+`cv.dry_run()` still uploads your examples to the API. If that's a blocker — a
+compliance review, or just not wanting production traffic to leave your network —
+run the identical stock-vs-fine-tuned evaluation entirely on your own machine:
+
+```bash
+pip install "cacheverifier[healthcheck]"
+
+cacheverifier healthcheck traffic.jsonl
+cacheverifier healthcheck traffic.jsonl --emit-summary summary.json
+```
+
+`traffic.jsonl` is a JSON array or JSONL of `{"query", "candidate_answer", "was_correct"}`
+rows **in arrival order** (the train/calibrate/test split is chronological, matching the
+hosted service so the numbers are comparable). Optional per row: `"stale": true`.
+
+Nothing is sent anywhere — the base model downloads once from Hugging Face, then it's
+fully offline. `--emit-summary` writes an aggregate-only JSON file (AUCs, counts, rates —
+no query or answer text) that's safe to share for a human read.
+
+```
+results
+------------------------------------------------------------------
+  train / calibrate / test:         3349 / 419 / 419
+  stock verifier   held-out AUC:    0.6120
+  fine-tuned       held-out AUC:    0.7080   (delta +0.0960)
+  label-noise proxy (disagreement): 11.4%
+  ceiling status:                   still_improvable
+
+verdict
+------------------------------------------------------------------
+  IMPROVED   -- fine-tuning on your own data helps this traffic
+```
+
 ## API surface
 
 | method | endpoint |

cacheverifier/__init__.py

@@ -18,5 +18,5 @@
 
 from cacheverifier.client import CacheVerifier, CacheVerifierError, VerifyResult
 
-__version__ = "0.1.0"
+__version__ = "0.2.0"
 __all__ = ["CacheVerifier", "CacheVerifierError", "VerifyResult", "__version__"]

cacheverifier/__main__.py

@@ -0,0 +1,44 @@
+"""`python -m cacheverifier` / the `cacheverifier` console script.
+
+A thin argparse dispatcher. The base install (httpx only) provides
+`--version` and `--help`; the `healthcheck` subcommand additionally needs
+the `healthcheck` extra and imports nothing heavy until it runs.
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+
+from cacheverifier import __version__
+
+
+def _build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="cacheverifier",
+        description="CacheVerifier -- hosted semantic-cache verification (https://www.cacheverifier.com).",
+    )
+    parser.add_argument("--version", action="version", version=f"cacheverifier {__version__}")
+    subparsers = parser.add_subparsers(dest="command", metavar="<command>")
+
+    # Import lazily and defensively: a missing healthcheck extra must not
+    # break `cacheverifier --help` or `--version`. cli.add_subparser itself
+    # only touches argparse.
+    from cacheverifier._healthcheck import cli as healthcheck_cli
+
+    healthcheck_cli.add_subparser(subparsers)
+
+    return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = _build_parser()
+    args = parser.parse_args(argv)
+    if not getattr(args, "command", None):
+        parser.print_help()
+        return 1
+    return int(args.func(args))
+
+
+if __name__ == "__main__":
+    sys.exit(main())

cacheverifier/_healthcheck/__init__.py

@@ -0,0 +1,7 @@
+"""`cacheverifier healthcheck` -- run the hosted Health Check Report's
+stock-vs-fine-tuned AUC evaluation entirely on your own machine.
+
+Everything under here needs the `healthcheck` extra
+(`pip install "cacheverifier[healthcheck]"`: torch, sentence-transformers,
+scikit-learn, numpy) and is imported only when the subcommand runs.
+"""