Security, invariant, and test-coverage fixes from a repo gap analysis

.github/workflows/test.yml

@@ -0,0 +1,41 @@
+# The CPU test suite, on every push and PR.
+#
+# tests/gpu/ is excluded — it needs a real CUDA box and is run with
+# `make gpu-test`. Everything under tests/ runs here.
+
+name: test
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+concurrency:
+  group: test-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  pytest:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.12"]  # the floor and the current default
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+
+      - name: Install (batteries included — the same stack users get)
+        run: |
+          python -m pip install --quiet --upgrade pip
+          python -m pip install --quiet -e '.[dev]'
+
+      - name: Run the CPU suite
+        run: python -m pytest tests/ --ignore=tests/gpu -q
+
+      - name: Byte-compile the package
+        run: python -m compileall -q shadowlm

CLAUDE.md

@@ -28,19 +28,22 @@ make check             # compileall the package + `tsc -b` the frontend
 make build             # build the frontend, then the wheel+sdist, then twine check
 make release           # bump patch (or BUMP=minor/major, V=x.y.z), build, tag, push
 
-pytest                                       # the CPU test suite (tests/, excludes gpu/)
+make test              # the CPU test suite — exactly what CI runs
 pytest tests/test_more_plus_router.py        # a single test file
 pytest tests/test_more_plus_router.py::test_name -x   # a single test, stop on first fail
 make gpu-test          # the CUDA verification suite — run on a GPU box only
 ```
 
 `tests/gpu/` (CUDA, real GPU) is **not** part of the default `pytest` run — it's
-invoked explicitly via `make gpu-test` or `python tests/gpu/test_cuda.py`.
+invoked explicitly via `make gpu-test` or `python tests/gpu/test_cuda.py`, and
+it exits non-zero if every test skipped (no CUDA) rather than reading as a pass.
 
-Releases publish to PyPI via `.github/workflows/publish.yml` on a `v*` tag. The
-CI gate requires the version to match in **three** places: the git tag,
-`pyproject.toml`, and `shadowlm/__init__.py`. `make bump`/`make release` keep
-the latter two in sync — never edit only one.
+Two workflows: `.github/workflows/test.yml` runs the CPU suite on 3.10 + 3.12
+for every push and PR; `.github/workflows/publish.yml` builds on every push to
+main and publishes to PyPI on a `v*` tag. The publish gate requires the version
+to match in **three** places: the git tag, `pyproject.toml`, and
+`shadowlm/__init__.py`. `make bump`/`make release` keep the latter two in sync —
+never edit only one.
 
 ## Architecture
 
@@ -88,10 +91,16 @@ touches one file and no others.
   records an unmodified agent's traffic, reconstructing message-level
   trajectories (calls that extend a prior call's message prefix merge into one
   episode; use an `x-session-id` header to disambiguate interleaved conversations).
+- `traces.py` — the offline sibling of `capture.py`: ingests OpenTelemetry GenAI
+  spans (OTLP JSON, event/spec/OpenInference/raw-wire shapes), groups them into
+  conversations, and `to_dataset()`s them. For agents already instrumented —
+  no proxy in the path.
 - `rl.py` — `Trajectory` / `TrajectoryGroup` / `judge_group` (LLM-judge scoring),
   fed into `method="grpo"`.
 - `apo.py` — `optimize_prompt()`: optimize the prompt instead of weights, same
   capture/judge front end, no GPU.
+- `eval.py` — `slm.evaluate()` / `shadowlm eval`: score a model on a held-out set
+  (exact / contains / numeric / JSON / LLM-judge scorers).
 
 ### Signature methods (MoRE)
 
@@ -105,14 +114,27 @@ routing; its run progress is one step per unit (see `resolve_total_steps`).
 - `serve.py` — `python -m shadowlm.serve` / `shadowlm serve`. Pure-stdlib
   (`http.server` + threads) reference server: trains on **this machine's real
   backend** (no mock), streams metrics, ships adapters as tar.gz, serves the
-  built React UI from `_static`. One job at a time — honest reference tier.
+  built React UI from `_static`. It runs one local job at a time, and doubles as
+  the **hub** for the worker tier below (per-worker inboxes, job routing,
+  inference forwarding).
+- `worker.py` + `ws.py` — the fleet tier. `shadowlm worker --hub <url>` makes a
+  NAT'd machine dial *out* and hold one websocket open: jobs arrive on it, logs
+  and metrics stream back up it, cancels land mid-step, and playground
+  chat/generate is proxied to whichever worker holds the adapter. The trained
+  adapter ships home over plain HTTP (`POST /v1/workers/<n>/jobs/<id>/artifact`).
+  `ws.py` is a minimal RFC 6455 implementation — no fragmentation, no
+  extensions, capped frame size.
 - `remote.py` — the typed client for that JSON protocol (`/v1/finetunes`, …).
-  Same protocol backs `backend="remote"` and ShadowLM Studio.
+  Same protocol backs `backend="remote"` and ShadowLM Studio. `SHADOWLM_API_URL`
+  may list several servers; `pick()` binds to the least-busy reachable one for
+  the session (client-side routing, deliberately not a scheduler).
 - `frontend/` — React 19 + Vite + Tailwind v4 studio. `npm run build` outputs to
   `../shadowlm/_static` (the wheel ships the compiled UI; end users never need
   node). `frontend/src/api.ts` is the typed mirror of the remote protocol. The
-  pages (Datasets → Models → Train → Runs → Playground) are the capture→train→own
-  loop as a UI. Auth: studio routes are gated by username/password.
+  pages (Dashboard · Datasets → Models → Train → Runs → Playground · Machines)
+  are the capture→train→own loop as a UI. Auth has three modes — `password`,
+  `apikey`, or `none` (`GET /v1/auth` reports which) — plus long-lived, hashed,
+  individually-revocable **machine tokens** that workers authenticate with.
 
 ### The shadow accelerator (`accel.py`)
 

Makefile

@@ -62,6 +62,11 @@ check:  ## compile the package + typecheck the frontend
 	$(PY) -m compileall -q shadowlm
 	cd frontend && npx tsc -b
 
+.PHONY: test
+test: $(PY)  ## the CPU test suite (what CI runs; tests/gpu needs a GPU box)
+	$(PIP) install -q pytest
+	$(PY) -m pytest tests/ --ignore=tests/gpu -q
+
 .PHONY: gpu-test
 gpu-test:  ## the CUDA verification suite (run on a GPU box)
 	$(PY) tests/gpu/test_cuda.py

README.md

@@ -72,6 +72,7 @@ The whole **capture → judge → train → own a shadowLM** loop runs on these:
 | Block | What it does | API |
 |-------|--------------|-----|
 | **Capture proxy** | drop-in OpenAI endpoint that records your agent's traffic into trajectories — agent unchanged | `slm.capture()` |
+| **Trace ingestion** | already have OpenTelemetry GenAI spans? turn an OTLP dump into a training set, no proxy | `slm.traces.to_dataset()` |
 | **13 methods** | LoRA · QLoRA · DoRA · full · CPT · DPO · GRPO · MoRE · MoRE+ · BitFit · prompt · p-tuning · adapter | `method=` |
 | **Judge → train** | score episodes with an LLM judge, train with trajectory-GRPO or DPO | `judge_group` |
 | **APO** | optimize the *prompt* instead of weights — same capture/judge front end, no GPU | `slm.optimize_prompt()` |
@@ -80,7 +81,9 @@ The whole **capture → judge → train → own a shadowLM** loop runs on these:
 | **Any hardware** | CUDA · TPU · Trainium · Intel · Apple · CPU (whatever HF accelerate targets) | `device=` |
 | **Shadow accelerator** | 4-bit, grad checkpointing, flash-attn, fused optimizer, optional Liger kernels — logged, never silent | `accelerator="shadow"` |
 | **Checkpoints** | save every N steps, then load or A/B any version — `step 200` vs `final` — in the playground | `save_steps=` · `run.checkpoint_at(step)` |
+| **Eval** | score a model on a held-out set — exact/contains/numeric/JSON/LLM-judge scorers | `slm.evaluate()` · `shadowlm eval` |
 | **Remote + server** | train on a GPU box or fleet over one JSON protocol; metrics stream back | `backend="remote"` · `shadowlm serve` |
+| **Worker fleet** | a NAT'd GPU box dials the hub over one websocket and takes jobs; the studio routes to it | `shadowlm worker --hub …` |
 | **Studio** | datasets → models → guided train → live runs (charts + console) → playground compare | `shadowlm serve` → `/` |
 | **CLI** | finetune / runs / plot / chat / export / methods from the shell | `shadowlm …` |
 | **Own the weights** | adapter/merged export, run records that survive restarts, nothing leaves your box | `model.save()` |
@@ -139,8 +142,9 @@ curl -fsSL https://install.shadowlm.sh | sh
 It detects your hardware and installs the matching stack — Apple Silicon → mlx,
 NVIDIA → torch + Liger fused kernels, otherwise torch CPU — into an isolated env
 in `~/.shadowlm/venv`, then launches `shadowlm serve` at `http://127.0.0.1:8329`.
-Re-run any time to upgrade. Override with `SHADOWLM_EXTRAS=cli` (UI only),
-`SHADOWLM_PORT=…`, or `SHADOWLM_NO_SERVE=1` (install without launching).
+Re-run any time to upgrade. Override with `SHADOWLM_EXTRAS=…` (pip extras —
+NVIDIA gets `kernels` by default), `SHADOWLM_PORT=…`, or `SHADOWLM_NO_SERVE=1`
+(install without launching).
 
 Or with pip — `pip install shadowlm` ships the full training stack (torch +
 HuggingFace, retrieval, CLI). On Apple Silicon the mlx dev backend is pulled in
@@ -154,12 +158,12 @@ automatically. Two extras stay opt-in for specialized hardware:
 ```bash
 git clone https://github.com/open-gitagent/shadowLM && cd shadowLM
 python3 -m venv .venv && source .venv/bin/activate && pip install -e .
-python examples/quickstart.py    # datasets → finetune → inference, end to end
+python examples/mlx/lora.py      # datasets → finetune → inference, end to end
 ```
 
-No hardware handy? Test-drive the whole thing — checkpoints, faiss MoRE, APO —
-on a free Colab GPU:
-[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/open-gitagent/shadowLM/blob/main/examples/colab_test_drive.ipynb)
+`examples/` has one runnable script per (backend × method) — `mlx/` runs a 0.5B
+model in seconds, `torch/` and `remote/` use Qwen3-8B. See
+[examples/README.md](./examples/README.md) for the full matrix.
 
 Run output (mlx, a 0.5B model, ~3.5s):
 
@@ -204,16 +208,17 @@ The engine ships first; **ShadowLM Studio** (the hosted tier) wraps this exact
 API — nothing reimplemented — to turn the blocks into a one-click migration:
 
 - **Decision inbox** — captured traces surfaced for human approve/correct into chosen-vs-rejected pairs (today: auto-scored by an LLM judge).
-- **Eval gates** — advance only when quality holds *and* savings beat cost: task-level evals + cost-per-task on the run records.
+- **Cost gates** — `slm.evaluate` already scores a model on a held-out set; the gate that ties quality to cost-per-task and blocks the switch is Studio's.
 - **Shadow router** — the capture proxy evolved: run the shadowLM in parallel behind the live agent, then shift traffic % frontier → owned.
-- **Fleet + teams** — GPU job queue, shared run history, dataset/adapter registry.
+- **Teams** — shared run history, dataset/adapter registry, org auth on top of the fleet below.
 
 ```
 [x] SDK — datasets → finetune → inference on mlx / torch / remote
 [x] 13 methods incl. MoRE, MoRE+ (decoupled MoE), trajectory GRPO, judge rewards
-[x] Capture proxy · shadow accelerator · any-hardware
+[x] Capture proxy · OTLP trace ingestion · shadow accelerator · any-hardware
 [x] Remote backend + reference server + the studio dashboard + CLI
-[ ] Studio orchestration — decision inbox · eval gates · shadow router · switch
+[x] Eval scorers (`slm.evaluate`, `shadowlm eval`) · worker fleet (`shadowlm worker`)
+[ ] Studio orchestration — decision inbox · cost gates · shadow router · switch
 ```
 
 ## Contributing

frontend/src/api.ts

@@ -38,7 +38,8 @@ export interface MethodInfo {
   description: string;
   default_lr: number;
   trainer: string;
-  adapter: string;  // lora | dora | more | bottleneck | bitfit | prompt | ptuning | none
+  // lora | dora | more | more_plus | bottleneck | bitfit | prompt | ptuning | none
+  adapter: string;
 }
 
 export interface JobSummary {
@@ -101,6 +102,8 @@ export interface AuthInfo { auth_required: boolean; mode: "password" | "apikey"
 export const getAuthInfo = () =>
   fetch("/v1/auth").then((r) => r.json() as Promise<AuthInfo>);
 
+export interface LoginResponse { token: string; user: string; expires: number; }
+
 export async function login(username: string, password: string): Promise<void> {
   const r = await fetch("/v1/login", {
     method: "POST", headers: { "Content-Type": "application/json" },
@@ -110,14 +113,23 @@ export async function login(username: string, password: string): Promise<void> {
     const d = await r.json().catch(() => ({} as { error?: string }));
     throw new Error(d.error || "login failed");
   }
-  const { token } = (await r.json()) as { token: string };
+  const { token } = (await r.json()) as LoginResponse;
   apiKey.set(token);
 }
 
 export const logout = () => apiKey.clear();
 
-export const getHealth = () =>
-  api<{ ok: boolean; backend: string; version: string }>("/v1/health");
+// the server spreads capacity() into this — fleet depth, not just liveness
+export interface Health {
+  ok: boolean;
+  backend: string;
+  version: string;
+  gpus: number;
+  running: number;
+  pending: number;
+  workers: number;
+}
+export const getHealth = () => api<Health>("/v1/health");
 export const getDatasets = () =>
   api<{ datasets: DatasetMeta[] }>("/v1/datasets");
 export const getDataset = (id: string) => api<DatasetMeta>(`/v1/datasets/${id}`);
@@ -194,7 +206,22 @@ export const getCheckpoints = (id: string) =>
   api<{ checkpoints: Checkpoint[] }>(`/v1/finetunes/${id}/checkpoints`);
 export const cancelJob = (id: string) =>
   api<{ ok: boolean }>(`/v1/finetunes/${id}/cancel`, { method: "POST" });
-export const submitFinetune = (body: object) =>
+export interface DatasetPayload { rows: unknown[]; format?: string }
+
+// what POST /v1/finetunes accepts — either `dataset` rows or a `dataset_id`.
+// eval_dataset is rows, a holdout ("20%" or a 0–1 fraction), or null for none.
+export interface FinetuneRequest {
+  base_model: string;
+  config: Record<string, unknown>;
+  name?: string;
+  dataset?: DatasetPayload | null;
+  dataset_id?: string | null;
+  eval_dataset?: DatasetPayload | string | number | null;
+  worker?: string | null;
+  load_in_4bit?: boolean;
+  max_seq_length?: number;
+}
+export const submitFinetune = (body: FinetuneRequest) =>
   api<{ job_id: string }>("/v1/finetunes", { method: "POST", body: JSON.stringify(body) });
 export const prewarm = (model: string, adapter: string | null, checkpoint: number | null = null) =>
   api<{ ready: boolean; error?: string }>("/v1/prewarm", {

frontend/src/pages/Train.tsx

@@ -215,7 +215,7 @@ export default function Train({ methods }: { methods: MethodInfo[] }) {
   }
 
   async function start() {
-    if (!ready || busy) return;
+    if (!ready || busy || !model || !ds) return;  // `ready` covers this; tsc can't see it
     setErr(""); setBusy(true);
     try {
       const out = await submitFinetune({

install.sh

@@ -6,7 +6,7 @@
 # Installs shadowlm into an isolated venv (~/.shadowlm/venv), picks the right
 # training backend for your machine, and opens the studio. Re-run any time to
 # upgrade. Override with env vars:
-#   SHADOWLM_EXTRAS=cli      # lighter: just the UI/CLI, add a backend later
+#   SHADOWLM_EXTRAS=kernels  # pip extras to add (NVIDIA gets this by default)
 #   SHADOWLM_PORT=8329       # studio port
 #   SHADOWLM_NO_SERVE=1      # install only, don't launch
 set -eu

pyproject.toml

@@ -55,6 +55,9 @@ dependencies = [
     # mixture of retrieval experts (method="more") — embeddings + faiss index
     "sentence-transformers>=3.0",
     "faiss-cpu>=1.8",
+    # more.py / more_plus.py use it directly (index math, routing scores) rather
+    # than relying on it arriving via torch/faiss
+    "numpy>=1.24",
     # the shadowlm CLI (Typer + Rich)
     "typer>=0.12",
     "rich>=13",
@@ -83,6 +86,10 @@ kernels = [
 verl = [
     "verl>=0.4",
 ]
+# Everything `make test` / the CI test job needs on top of the base install.
+dev = [
+    "pytest>=8",
+]
 # Back-compat aliases — the training stack now ships in the base install, so
 # these resolve to nothing but keep older `shadowlm[all]` / `[torch]` commands working.
 all = []

shadowlm/_cli_entry.py

@@ -1,9 +1,8 @@
 """Console-script entry point — pure stdlib, always importable.
 
-The real CLI (`shadowlm.cli`) is built on Typer + Rich, which ship in the
-`[cli]` extra (and in `[all]` / `[mlx-all]`). This shim lets the bare
-`shadowlm` command degrade to a friendly message instead of a traceback when
-those aren't installed.
+The real CLI (`shadowlm.cli`) is built on Typer + Rich, which the base install
+ships. This shim lets the bare `shadowlm` command degrade to a friendly message
+instead of a traceback if they're somehow missing (a partial or pruned install).
 """
 
 from __future__ import annotations
@@ -17,9 +16,9 @@ def main() -> int:
     except ModuleNotFoundError as e:
         if e.name in ("typer", "rich", "click", "click.core"):
             print(
-                "The shadowlm CLI needs Typer + Rich:\n\n"
-                "    pip install 'shadowlm[cli]'\n\n"
-                "(included in 'shadowlm[all]' and 'shadowlm[mlx-all]')",
+                f"The shadowlm CLI needs Typer + Rich ({e.name} is missing).\n\n"
+                "    pip install --upgrade --force-reinstall shadowlm\n\n"
+                "They ship in the base install, so this means a partial one.",
                 file=sys.stderr,
             )
             return 1