fix: idle work must not starve real requests — the stuck context probe

.abcd/work/DECISIONS.md

@@ -383,3 +383,5 @@
 - 2026-09-21 — **Debug logging for one model until it is restarted is built, reviewed and shipped on one branch (`feat/2609062346` merged onto `integrate/2609062346`; itd-2609062346072707 shipped, spc-2609201007359229 closed with `--impact additive`)**, the second pilot of the autonomous run and the first run of the outer loop as a script: two fresh implementer sessions in sequence, two Sonnet reviews, the fixes by resuming the second implementer, the spec closed in the same change. What holds each acceptance criterion: arming touches no running process, `TestArmingTouchesNoRunningProcess`; DEBUG once then INFO, `TestAnArmedLaunchIsAtDebugAndTheNextIsAtInfo` and `TestALaunchThatFailsToSpawnLeavesTheMarkArmed`; the panel's words, `TestThePanelSaysWhatDebugLoggingWrites`; armed and running distinguished, `TestTheSnapshotCarriesTheDebugState` and `TestTheCardDrawsTheDebugPills`; the mark never from the statistics switch, `TestTheDebugMarkIsNamedOnlyByItsReaders`, `TestDebugMarkReadersAllExist` and `TestTheDebugMarkIsDerivedFromNothingElse`; the launcher test amended not deleted, `TestTheModelServerLevelComesOnlyFromThePerModelDebugMark`; the bound, `TestTheDebugLogStopsAtItsBound` and `TestAnArmedLaunchStopsItsLogAtTheBound`; the kept previous file, `TestALaunchKeepsThePreviousRunsLog` and `TestTwoConsecutiveLaunchesKeepOnlyOnePreviousLog`; the docs sentence, `TestTheLoggingPageDescribesThePerModelDebugAction`; the refusal, `TestAnExceptedModelRefusesTheDebugArm`; the ADR, the link-integrity check and adr-2609201008477513 cited by the amended test's comment and the spec (row 11, checked by hand). The reviews: the ruthless review's one finding — a mark armed for a model then deleted survived a later download of the same repo id — fixed with `Pool.Remove` and `App.Delete`, `TestDeleteDropsTheDebugLoggingMark` and `TestRemovingAModelDropsItsMark` watched red first; the security review APPROVE, its one low finding (the bounded writer swallows write errors) captured as iss-2609210903529294; the adjudication step skipped by the loop's own rule (two reviews, two findings). The docs-currency review found every claim on `docs/logging.md` and `docs/posture-reference.md` current. Row 4 checked by hand on the scratch root with the small model: arming appears in `/api/state` on a fresh read; the next request loads the model at DEBUG (`resident[].debug_log: true`, the model's log at DEBUG with the prompt in it) and spends the mark; unload and a second request load it at INFO with no prompt content in the new log, the DEBUG run kept as `.previous.log`; the bound of 64 MB stated in code and docs; the server stopped and proven gone. Row 10's end-to-end check is owed to itd-2609091715089488, whose per-model field the predicate reads. The Fable design review the big run's plan asks for was not stood in for: no such review ran, and nothing here claims one did.
 - 2026-09-21 — **The 0.9.2 cut, codename Prellerhaus**, by hand on the 0.9.1 precedent. The cut ships debug logging for one model (itd-2609062346072707, PR 139, `impact: additive`) and nothing else since v0.9.1, so the version is a patch; `build/CODENAME` is unchanged. The two open majors are re-deferred with `deferred_after: "v0.9.2"`: iss-2609200815308397 and iss-2609190242198542, because the maintainer tests by hand against this release and neither is of this cut's class. Cut by the second pilot's operator at the loop's `release` step, which the script marks manual; the fidelity verdict of the shipped intent (MET 8, MET_WITH_CONCERNS 3, the concerns captured) rides in the same change. Retention after verification deletes the v0.9.1 release and keeps its tag.
 - 2026-09-21 — **Some models keep no transcript even while recording is on is built, reviewed and merged on one branch (`feat/2609091715` merged onto `integrate/2609091715`; itd-2609091715089488 shipped, spc-2609201007367486 closed with `--impact additive`), by the outer loop as a script (pilot 3) — and merged WITHOUT a release, because the lane was built before its parent.** Two fresh implementer sessions in sequence (the spec is over the split line), two Sonnet reviews, the fixes by resuming the second implementer, the spec closed in the same change, one planted merge conflict as the run's stop-and-resume test. The 2026-09-20 ordering line says transcript recording (itd-2609091707499248) lands before this intent; the maintainer's pilot-3 prompt chose this intent regardless, the loop ran it, and the ruthless review's one high finding is exactly the consequence: without the parent's switch `gateway.Options.TranscriptOn` is never set, so every models-list entry carries `recording: false` and every card and picker row reads "keeps no transcript" — which is true on a tree where nothing is recorded, but leaves criteria 9 and 10 (a visible difference between an excepted and a recorded model) unmet until the parent lands. The fix session captured it rather than changed the value rule: iss-2609211218478273, major, which refuses the release cut while open; the maintainer chose (at the pr step, 2026-09-21) to merge and not release: v0.9.3 follows the parent. Departures are rendered on the closed spec from the reports (the seam `Options.TranscriptOn` nil-is-off; `Control.TranscriptExcepted` removed in favour of `Config.NoTranscript` read per request; `docs/transcript.md` created rather than gained as a section; the form posts every per-model box explicitly because the merged save reads an absent key as "keep"). What holds each criterion: the field and its folded fail-closed reader, `TestNoTranscriptIsReadFoldedAndFailsClosed`; the field-by-field merge of the per-model map, `TestAPerModelFieldThePanelDidNotRenderSurvivesASave` (table-driven over every field) and `TestTheMergedPerModelMapHoldsExactlyTheKeysTheBodyNames`; the untouched save stays accepted, `TestASaveOfAnUneditedFormIsAccepted`; the form posts every box it draws, `TestSettingsFormPostsTheTranscriptBox` and `TestSettingsFormPostsThePerModelMapWhole`; settable before download, `TestTheExceptionIsSettableOnAModelNotYetDownloaded`; the debug arm refused, `TestAnExceptedModelRefusesTheDebugArm` and `TestExceptingAModelInSettingsRefusesItsNextDebugArm`, with both sentences in the markup, `TestBothPanelsSayHowTheExceptionMeetsDebugLogging`; `recording` in the base entry to every client, the two pinned-field-set tests and `TestModelsListReferenceDocumentsEveryFieldServed`; the icon in the client and the card, `TestChatClientPickerShowsTheTranscriptStateWithWords`, `TestChatClientTranscriptStateRule` (the Swift unit tier, 7 checks) and `TestTheCardDrawsTheTranscriptPillWithItsWords`; the bridge, `TestTheModelCommandOmitsAnExceptedModel`, `TestTheModelCommandRefusesAnExceptedModel` and `TestAChannelExceptedAfterItChoseIsRefusedAtTheNextMessage`. Not held on this tree: criteria 6 and 8's gateway tests (a mixed message array recorded whole; the first served request of an excepted model writing nothing) need the parent's store and are owed to its integration, where `Gateway.recorded` is called on the completions path. Reviews: ruthless FIX_FIRST with the one high finding above; security APPROVE, 0 findings (the merge cannot smuggle two spellings of one model past `validateModels`; the debug-arm read is per request, closing a stale-cache window the old seam had; the bridge has no path to an excepted model; `recording` renders only a fixed two-word vocabulary). Adjudication skipped on the count rule (1 < 3); docs currency CURRENT over `docs/transcript.md`, `docs/models-list.md` and `docs/discord-bridge.md`. Hand checks on the scratch root (integration build `v0.9.2-14-g17cfcc2c`, port 11999, loopback): row 10 of the debug-logging spec, owed since pilot 2 — arming succeeds before the exception (200), and once the model is excepted it is refused 409 with the reason, under the folded spelling too; through the real panel, two clicks on Debug logging post the 409 and the panel shows the reason; the panel's Transcript box posts `no_transcript: true` and every other box of the row as an explicit zero, and clearing it posts `false`; a stale form (its snapshot taken before `served_context` and `pinned` were hand-planted in `config.json` and the server restarted) saving only `no_transcript` leaves both planted fields on disk; both sentences are served (the debug control's "A model that keeps no transcript refuses this.", the transcript control's naming the bridge and the refusal); the card's icon carries the label "keeps no transcript" — before and after the exception alike, the finding made visible; pilot 2's owed DOM half of its row 4 (iss-2609210922572240): two clicks arm debug logging from the card (200, `DEBUG ARMED` pill, "Stop debug logging"), two more disarm it. Not checked by hand: the chat client's picker against a live server (the client compiles; its archtest and unit tier hold the words), and the Discord bridge (no bridge on the scratch root; the three bridge tests hold each arm). Not stood in for: the parent's landing, and the Fable design review the big run's plan asks for.
+- 2026-09-21 — **The stuck context probe is three bug fixes, and pre-emption stays a draft** (branch `fix/stuck-probe`; iss-2609211334563318, iss-2609211334570516 and iss-2609211334576018 resolved; itd-2609211335097114 untouched). The live server (v0.9.1) refused every chat request 503 for six hours because the probe queued `mlx-community/GLM-OCR-bf16` — an image-to-text model, `chat: false` — and loaded it thirty-two times: the child raised `ValueError: Model type glm_ocr not supported` in its generate thread on the first request while its httpd answered `/health`, so the pool waited its ten-minute readiness timeout each time, and while loading the model was charged the whole budget (its default served window is worked out to fill what the budget has, so a 2.2 GB model with no served-window setting is charged ≈ the budget from the moment its entry exists — a property of the charge, not of loading, and not changed here). (1) The probe considers only models the server offers to chat: `Candidates()` and `MeasureNow` read `registry.Model.CanChat` with the rule in force. (2) A load the child has given up on fails in seconds: the pool watches the per-model child log while it waits for readiness (`LoadLogger`, which the real launcher's process satisfies) and ends the wait on a traceback whose terminal line is a ValueError, ModuleNotFoundError or ImportError, the line bounded and stripped of anything path-shaped; BrokenPipeError and the like are not in the set because the child survives them. A load that never became ready is recorded on the model (`registry.LoadFailure`: reason and provenance — runtime, budget, concurrency, served window); while it stands the probe and the self-test skip the model, a queued probe of it is dropped, and a request for it is refused at once as a NotReadyError carrying the reason; it is lifted by a moved provenance (through `RefreshStaleness`), a re-download, Load or Measure now. The pool tells the observer whether a failure was the model's own or interrupted (the entry taken out of the pool meanwhile), and only the former is recorded. The probe's ten-minute step floor is deliberately left: it is the gateway's own prefill base, which the probe's timer must not undercut or a slow step is filed as the deadline's, and a step's request includes the cold load the pool allows ten minutes for. (3) The refusal names the holder to an entitled client only: the gateway is handed the idle loop's status (which gains `since`), and a no-room refusal to a loopback or key-admitted client — the same clients the models list tells what is resident — names the model the job holds, the job, for how long, and that Unload releases it; the pool's own refusal still names no model and unentitled clients still get the generic sentence. The card's pill says "loading", "loading for the context probe", "held by the self-test". What would show these wrong: a chat model the probe now skips; a genuine load — a slow cold load — that a traceback line in the set fails early; a load failure that survives a runtime change; a keyless network client that reads a model id in a 503. Pre-emption — a real request taking the memory idle work holds — is itd-2609211335097114, a draft with no acceptance criteria, and is not implemented or approximated here.
+- 2026-09-21 — **Corrections to the stuck-probe line above, from its two adversarial reviews** (an append-only ledger corrects by superseding; both lines stand and this one governs where they differ). (1) A load failure that is the pool's own bound — the readiness timeout, an exit by signal — is recorded as `transient`: it stands for this process (idle work skips the model, a client is told why at once) and is dropped at the next start, because a slow load on a busy Mac says nothing about the next one; the child's own traceback and a non-signal exit status stand until the provenance moves or a person retries, as the line above says. (2) The refusal's promise that Unload releases a model an idle job holds was false while the job's own request was in flight (the pool refused it as busy — the maintainer's two 409s on the live box): the idle loop now exposes `Runner.Interrupt(model)`, the run yields as it does for a client, and the panel's Unload asks for that first and waits, bounded, for the model to go (`TestUnloadFromThePanelTakesTheModelBackFromTheProbe`, watched red on the 409). (3) The child-log reader takes a terminal line only straight after a traceback's frames and only from whole lines; its open refuses a link and a FIFO; the sanitiser drops control characters and blanks a path with spaces as one path. (4) A failure reported after a hand retry has started a fresh load is not written over it; a Measure now arriving between Due's candidate snapshot and its pruning is not pruned. Accepted without change: a local process that can reach the child's loopback port can write a line the reader takes (the same trust class that can already plant registry.json; no new privilege); the provenance is read when the failure is recorded rather than when the load began.

.abcd/work/issues/open/iss-2609211334563318-the-context-probe-measures-models-that-are-not-chat-models-i.md → .abcd/work/issues/resolved/iss-2609211334563318-the-context-probe-measures-models-that-are-not-chat-models-i.md

@@ -9,6 +9,10 @@ found_during: "live server v0.9.1 on 2026-09-21, 503 for every chat request"
 origin: researcher-authored
 production_mode: hand-written
 found_at: "internal/app/contextprobe.go"
+resolution: "Candidates() and Measure now read registry.Model.CanChat with the rule in force; an image-to-text model with chat: false is never a candidate (TestTheProbeConsidersOnlyChatModels)"
+impact: fix
+resolved_by:
+  commit: "52dcef7a"
 ---
 
 The context probe measures models that are not chat models: it takes every Ready() model as a candidate (internal/app/contextprobe.go Candidates), so on the live server it picked mlx-community/GLM-OCR-bf16 (pipeline image-to-text, chat: false in the models list), spawned an mlx_lm server for it and POSTed /v1/chat/completions, which never answered (the child idle at 0% CPU answering /health in under a millisecond, in_flight 1 for over ten minutes at 'calibrating at 1024 tokens'). A served window is only meaningful for a model the server offers to chat; the probe should skip chat: false models, and a step whose child answers /health but not a completion should fail fast rather than wait for the step timeout.

.abcd/work/issues/open/iss-2609211334570516-while-the-context-probe-calibrates-a-model-the-model-is-load.md → .abcd/work/issues/resolved/iss-2609211334570516-while-the-context-probe-calibrates-a-model-the-model-is-load.md

@@ -9,6 +9,10 @@ found_during: "live server v0.9.1 on 2026-09-21, 503 for every chat request"
 origin: researcher-authored
 production_mode: hand-written
 found_at: "internal/contextprobe/probe.go"
+resolution: "The pool watches the child's log while it waits for readiness and ends the wait on a fatal traceback line with that reason (TestAFatalLineInTheChildLogEndsTheLoadWaitAtOnce); a load failure is recorded on the model with its provenance, skipped by the probe and the self-test, refused with the reason at once, and lifted by a moved provenance, a re-download, Load or Measure now (TestAFailedModelIsSkippedByIdleWorkAndRefusedWithItsReason). The step floor is left: it is the gateway's own base and covers the cold load."
+impact: fix
+resolved_by:
+  commit: "190d7172"
 ---
 
 While the context probe calibrates a model, the model is loading/in_flight and charged the whole memory budget (charge_bytes = budget: 82.46 GB for a 2.2 GB model), so it is not evictable and every real request for another model is refused 503 for the length of the step; the step timeout floor is eleven minutes (defaultStepTimeout in internal/contextprobe/probe.go: at least ten minutes plus a minute's margin), and after a failed step the probe can pick the same model again. Idle work starved real requests on the live server for the whole period the maintainer tried several chat clients. An idle job should yield the budget to a real request (abort the step, release the model) rather than the request yielding to the job.

.abcd/work/issues/open/iss-2609211334576018-the-503-a-client-gets-while-the-budget-is-held-by-idle-work.md → .abcd/work/issues/resolved/iss-2609211334576018-the-503-a-client-gets-while-the-budget-is-held-by-idle-work.md

@@ -9,6 +9,10 @@ found_during: "live server v0.9.1 on 2026-09-21, 503 for every chat request"
 origin: researcher-authored
 production_mode: hand-written
 found_at: "internal/gateway"
+resolution: "The gateway is handed the idle loop's status and a no-room refusal to an entitled client names the model an idle job holds, the job, and for how long (TestARefusalNamesTheModelAnIdleJobIsHolding); the card's pill names the job (TestTheResidencyPillNamesTheJobHoldingTheModel)"
+impact: fix
+resolved_by:
+  commit: "c9f75c6b"
 ---
 
 The 503 a client gets while the budget is held by idle work reads 'not enough memory to load another model, and no model in memory can be freed (limit 76.8 GB)': true, but it names neither what holds the memory (a loading model under the context probe) nor that the holder is idle work the server started itself, so the person reads it as their model being too big. The refusal should name the holder and the job, and the panel should show the same (the card says only 'loading').

CHANGELOG.md

@@ -40,6 +40,33 @@ GitHub release notes.
   every per-model setting; a save of an untouched form is still accepted,
   and a cross-field refusal still names a field the save changed.
 
+### Fixed
+
+- **A model the server cannot load no longer holds every chat request off
+  for hours.** `impact: fix`. On a server with the context probe switched
+  on, the probe picked a model the runtime has no support for — an OCR
+  model, which the models list already said was not a chat model — and
+  loaded it thirty-two times over six hours, each time waiting the whole
+  ten-minute readiness timeout while the model's charge filled the memory
+  budget, so every request Alice sent was refused for want of memory.
+  Three things change. The probe measures only models the server offers to
+  chat, and **Measure now** on any other says so. A load the model server
+  has already given up on — its own log says the model type is not
+  supported, or a module is missing — fails in seconds with that reason,
+  not after ten minutes; a model that failed to load is marked on its card
+  ("did not load", with the reason) and is left alone by the probe and the
+  self-test, and a request for it is refused at once with the reason, until
+  the runtime, the memory budget or its served window changes, or Alice
+  presses **Load** or **Measure now** to try it again (a mark that is only
+  Dessau's own timeout also goes at the next restart). And when a request
+  is refused for want of memory while the server's own idle work holds it,
+  the refusal says so to a client on this Mac or one holding the API key —
+  which model, which job, for how long, and that **Unload** on its card
+  releases it — which it now does even while the job's own request is in
+  flight, where it used to answer that the model was busy — and the card
+  says "loading for the context probe" rather than "loaded". A stranger on
+  the network is still told nothing.
+
 ## [0.9.2] - 2026-09-21
 
 ### Added