serve: model registry with hot-swap and per-request model logging

CHANGELOG.md

@@ -12,6 +12,38 @@ changed. Each entry names the section to read for the numbers behind it.
 
 ### Fixed
 
+- **`"loaded"` on `GET /v1/models` now means residency, as
+  `POST /v1/models/load` already said it did.** The registry listing set
+  `loaded: true` only for the model being served, so under
+  `--keep-previous` a model the server still holds an open `waste_ctx`
+  for was listed `loaded: false` — indistinguishable from a container
+  that was never opened — while the load response's `models` field
+  counted it as resident, and a generation naming it was refused as "not
+  loaded". The two endpoints now agree: `loaded` is `mid in engines`,
+  the same set the load response reports, and `GET /v1/models/{id}` says
+  the same for one entry. The 409 for a model that is resident but not
+  current says "resident but not the model being served" rather than
+  "registered but not loaded"; the `model_not_loaded` type and status
+  are unchanged. The `waste` shape still travels only on the current
+  entry — the per-model facts move with the current slot and are
+  re-derived when a swap makes a resident model current again. Tests in
+  `tests/serve/test_server.py`.
+
+- **A single-container server serves any model name again.** The model
+  registry made `check_model_request` strict for every deployment: with
+  no `--models`, the registry holds only the loaded model, so a request
+  naming anything else was a 404 — including the fixed name most OpenAI
+  clients are configured with and the `"model": "waste"` example in
+  `serve --help` itself, neither of which a client can easily change.
+  The server's own id defaults to the container's file name
+  (`k3.waste` → `k3`), so even the "correct" name was one most clients
+  did not know. What an existing deployment guaranteed was "any name is
+  answered by the one container"; that is restored: absent, the loaded
+  id, or any other string all serve. Strict validation — a 404 for a
+  name the registry does not know, a 409 for one it knows but has not
+  loaded — is what `--models` opts into, and is unchanged there. Tests
+  for both behaviours in `tests/serve/test_server.py`.
+
 - **The K2 tool-grammar check defaulted to a path on an external volume**,
   which is a description of one machine rather than a default: everywhere
   else — CI, a fresh clone, this machine with the disk unplugged — it read
@@ -27,6 +59,40 @@ changed. Each entry names the section to read for the numbers behind it.
 
 ### Added
 
+- **`--models` now has to prove its swap fits, and a load that would not
+  is a 507.** A swap opens the new container before closing the old one —
+  the order that makes a failed open leave the server serving what it was
+  serving — so two contexts are resident at once, and `docs/SERVE.md`
+  asked the operator to size `--budget` so that moment fits while nothing
+  checked. The default made it worse than "the sum of the two": with
+  `--budget 0` each context sizes itself to as much as 3/4 of
+  `waste_usable_ram()`, so a swap ran at ~1.5x what the process may use —
+  a paging run, not a slow one.
+
+  `--models` now requires an explicit `--budget`, and the server refuses
+  to start unless `2 x budget` fits, naming the largest budget that does;
+  the same lines are printed under the startup banner and by `--plan`,
+  which is where a budget gets chosen. At runtime the resident set is
+  counted at every load — each engine's budget, or the floor and expert
+  cache `waste_memory_used` reports for one that chose its own — and a
+  load that would not fit is refused with **507**
+  (`insufficient_memory`) *before* the container is opened: nothing is
+  closed, nothing is half-loaded, the previous model keeps serving, and
+  the message says what it needed next to what was already held. 507
+  rather than 503 because asking again cannot help.
+
+  That is also what bounds `--keep-previous`, whose resident set grows
+  with every model ever switched to and which no startup check can price
+  in advance: the cap is derived from the budgets rather than from a
+  separate `--max-resident` count that could disagree with them, and a
+  slot move to a model that is already resident is never refused because
+  it allocates nothing. Evicting a resident model to make room was the
+  alternative and is not done — changing what is resident behind a
+  client's back is the failure `--keep-previous` exists to prevent.
+
+  `serve/` goes to 144 server tests (from 135) plus a new
+  `tests/serve/test_main.py` of 12, for the arithmetic on its own.
+
 - **A strict CI job for the K2 tool protocol**, the one GLM has had and
   which `ci.yml` used to have to exempt K2 from in so many words: "the same
   ground-truth rule the K2 template check in tests/run.sh applies, except

docs/SERVE.md

@@ -268,7 +268,8 @@ transcribed in this repo. That is what is left of
 | endpoint | notes |
 |---|---|
 | `GET /health` | liveness; never requires the API key |
-| `GET /v1/models`, `GET /v1/models/{id}` | reports the container's real shape under a `waste` key |
+| `GET /v1/models`, `GET /v1/models/{id}` | the registry: `loaded` is residency — with `--keep-previous`, a model the server has swapped away from answers `loaded: true` too. The model being served is listed first and is the only entry with its real shape under a `waste` key |
+| `POST /v1/models/load` | swap models; see "Swapping models" below |
 | `POST /v1/chat/completions` | streaming and not, tools, images |
 | `POST /v1/completions` | raw continuation, no chat template |
 
@@ -318,6 +319,91 @@ one lock, and requests queue. On a model streaming experts off an SSD at a
 few tokens a second, the wait for the lock is small next to the wait for
 the answer.
 
+### Swapping models
+
+By default the process serves the one container it was started with, and
+a request may name it by any model string: absent, the container's own
+id — the file name, `k3.waste` → `k3` — or anything else a client sends,
+including the fixed name most OpenAI clients are configured with. There
+is nothing to be strict about until there is a set to be strict against.
+
+`--models PATH[=ID]` (repeatable) registers additional containers a
+client may switch to, and opts into strict validation: a request naming a
+model the registry does not know is a 404, rather than being silently
+served by the loaded model, which is what made `model` a decorative
+string before the registry existed.
+
+    python3 -m serve ~/models/k3.waste \
+        --models ~/models/glm53.waste --models ~/models/deepseek41.waste=ds41
+
+    curl localhost:8000/v1/models                 # the registry, with
+                                                  # "loaded" per entry
+    curl localhost:8000/v1/models/load \
+        -d '{"model":"glm53"}'                    # swap, then answer
+
+What a swap does:
+
+- It waits for the engine lock, so a generation in flight finishes before
+  the model moves under it.
+- It opens the new container, re-derives everything the container decides —
+  reply format, markers, stop tokens, the thinking default, `/v1/models`'s
+  shape — and only then **unloads the previous model** (`waste_close`).
+  One model resident at a time is the default, and the flag to change it
+  is `--keep-previous`.
+- The new container is opened before the old one is unloaded, so a swap
+  that fails — a truncated container, an `--exclusive-open` conflict —
+  leaves the server serving what it was serving and answers 500 with the
+  engine's own reason. The cost of that guarantee is a moment where both
+  containers are resident, and that moment is checked rather than
+  assumed: `--models` **requires `--budget`**, and `2 x budget` has to
+  fit in `waste_usable_ram()` or the server refuses to start. The reason
+  is the default. With `--budget 0` each context sizes itself to as much
+  as 3/4 of usable RAM, so two of them at once is ~1.5x what the process
+  may use — paging, not slowness. At startup the numbers are printed:
+
+      registry  glm53, ds41
+                glm53  floor 12.3 GB, recommended 30.1 GB
+                ds41   floor 18.7 GB, recommended 44.2 GB
+                two at once: 48.0 GB against 64.0 GB usable — fits
+
+  `--plan` prints the same lines without starting anything, which is how
+  to choose the budget in the first place.
+- A load that would not fit is refused **before the container is
+  opened**, with **507** and `type: insufficient_memory`: nothing is
+  closed, nothing is half-loaded, the previous model keeps serving, and
+  the message says what it needed next to what was already held. 507
+  rather than 503 because asking again cannot help — an operator has to
+  lower `--budget`, drop `--keep-previous`, or restart.
+- Generation always serves the current model. A request naming a
+  registered-but-not-loaded model is a 409, telling the client to
+  `POST /v1/models/load` first, rather than an unnoticed multi-gigabyte
+  swap in the middle of a conversation. A request with no model, or
+  naming the current one, is untouched — which is every client that does
+  not know about the registry.
+- A swap discards the previous model's KV state. That is inherent:
+  `waste_close` frees the context. An agent harness should treat a swap
+  as rare and expensive, not as a per-turn choice.
+
+`--keep-previous` keeps the replaced model resident instead of unloading
+it. Switching back to it is then a slot move rather than a reopen — its
+`waste_ctx` and the state it holds are still there — and a slot move is
+never refused, because it allocates nothing. The RAM two resident
+contexts need is the sum of their budgets; on the machines this engine
+targets that is usually the difference between working and paging, which
+is why unloading is the default.
+
+It also makes the total unbounded by anything a startup check can know,
+since every model switched to stays resident: three models at a budget of
+20 GB under a 64 GB machine is not a pair anyone can validate in advance.
+So the resident set is counted at each load — every engine's budget, or,
+when it chose its own, the floor and expert cache `waste_memory_used`
+reports for it — and the load that would put the sum over
+`waste_usable_ram()` is refused with the same 507. There is no separate
+`--max-resident` number to keep in step with the budgets: the cap is
+derived from them. A model that does not fit is not evicted to make room
+either — changing what is resident behind a client's back is the failure
+`--keep-previous` exists to prevent.
+
 Streaming is written straight from the token callback, on the thread
 holding the lock. A client hanging up propagates back as a return value the
 engine understands — the callback says stop, `waste_generate` unwinds, the
@@ -380,6 +466,20 @@ nothing while the model reasons — which, on a model whose reasoning can be
 most of the reply, looks like a server that has stopped. `--no-thinking`
 makes the default answer-only, and a request can still ask for reasoning.
 
+## Request log
+
+With request logging on (the default; `--no-log-requests` turns it off),
+each response line names the model it concerns:
+
+```
+127.0.0.1 - "POST /v1/chat/completions HTTP/1.1" 200 -  [model=glm53]
+127.0.0.1 - "POST /v1/chat/completions HTTP/1.1" 409 -  [model=tiny]
+```
+
+A success line names the model that served it, a 409 or 404 names the
+model that was refused, and a `/v1/models/load` line names the model it
+switched to. GET lines carry no model annotation.
+
 ## Security
 
 - `--host` defaults to `127.0.0.1`. Binding anywhere else without
@@ -429,7 +529,9 @@ K3_DIR=/Volumes/WasteDisk/k3 python3 tools/gen_xtml_goldens.py
 python3 -m serve MODEL [options]
 
   --host, --port, --model-id, --api-key
-  --budget SIZE      hard RAM ceiling, e.g. 48G (0 = the engine chooses)
+  --budget SIZE      hard RAM ceiling, e.g. 48G (0 = the engine chooses,
+                     which is up to 3/4 of usable RAM per context — so
+                     --models needs this set explicitly)
   --ctx N            context tokens
   --threads N        compute threads (0 = one per core)
   --cpus LIST        restrict them to a cpu list, e.g. 0-5 or 0-2,6-8;
@@ -444,5 +546,15 @@ python3 -m serve MODEL [options]
   --max-tokens N     default cap when a request does not set one (4096)
   --no-thinking      answer without the think channel unless asked
   --allow-local-images
+  --models PATH[=ID] additional containers a client may switch to with
+                     POST /v1/models/load (repeatable; switching unloads
+                     the model it replaces). Requires --budget, because a
+                     swap holds two contexts at once: 2 x budget has to
+                     fit in RAM or the server refuses to start
+  --keep-previous    keep the replaced model resident instead of unloading
+                     it; every model switched to stays resident, so the
+                     load that would put the set over the machine's RAM is
+                     refused with 507
   --plan             print the memory plan and exit
+  --no-log-requests  silence the per-request log lines
 ```