Skip to content

Docs drift: README references missing files, roadmap lists shipped features, numpy undeclared #10

Description

@thedataengineer

Docs lag shipped code by about six weeks. Concrete items:

README points at files that do not exist:

  • README.md:157python examples/quickstart.py; no such file (examples/ has evaluate.py, shadow_from_traces.py, and per-backend method examples only).
  • README.md:162 — Colab badge links examples/colab_test_drive.ipynb; no .ipynb exists anywhere in the repo.

README roadmap lists shipped features as future:

  • README.md:207 "Eval gates" is under the road ahead, but eval.py shipped 2026-06-20 with a CLI command (shadowlm eval), an example, and tests. Zero README mentions.
  • README.md:209 "Fleet + teams — GPU job queue" is under the road ahead with an unticked checkbox, but the worker/hub tier shipped 2026-07-13 (worker.py, ws.py, shadowlm worker, Machines page, machine tokens, device routing). Zero README mentions.
  • traces.py (OTLP GenAI ingestion, 555 lines, exported as slm.traces, with example + sample data + 15 tests) has zero README mentions. It is the offline sibling of capture() and arguably the second headline feature.

Misleading install guidance:

  • README.md:142 and install.sh:9 describe SHADOWLM_EXTRAS=cli as "lighter: just the UI/CLI", but [cli] resolves to [] (pyproject.toml:94), so it installs the identical full stack.
  • _cli_entry.py:23-25 tells the user to pip install 'shadowlm[cli]' to fix a ModuleNotFoundError; that extra is empty, so the suggested fix is a no-op that cannot resolve the error it is raised for.

Source docstring drift:

  • models.py:148-149 lists 12 built-in methods; more_plus is missing (registry has 13, methods/__init__.py:40).
  • models.py:295,305 document backend: as "auto" | "mlx" | "torch"; the actual accepted set adds remote and verl (backends/__init__.py:79). The same omission appears in five CLI help strings (cli.py:211, 305, 323, 369, 496).

Packaging:

  • numpy is imported directly (more.py:61,82,116,133, more_plus.py:50,169,278) but is not in dependencies (pyproject.toml:41-65); it resolves only transitively through torch/faiss.
  • bitsandbytes is Linux-only (pyproject.toml:54) while TrainConfig defaults optim="adamw_8bit"; the README fused-optimizer claim (README.md:81,194) silently does not apply on macOS/Windows.

CLAUDE.md (contributor doc) predates the worker tier:

  • Still describes serve.py as "One job at a time — honest reference tier"; it is now also a job router with per-worker inboxes and inference forwarding.
  • Missing entirely: worker.py, ws.py, traces.py, eval.py, bottleneck.py, the Dashboard and Machines pages, the three-mode auth (password|apikey|none) plus hashed machine tokens, and the multi-server pool pick() in remote.py.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions