From 482ec414abbe1661aff0e449de618f440547595b Mon Sep 17 00:00:00 2001 From: Graffioh Date: Tue, 1 Sep 2026 16:43:26 +0200 Subject: [PATCH 1/2] docs: define serving and engine component boundaries --- README.md | 1 + docs/specs/engine-components.md | 569 ++++++++++++++++++++++++++++++++ 2 files changed, 570 insertions(+) create mode 100644 docs/specs/engine-components.md diff --git a/README.md b/README.md index 64f999206..f00c0a3e2 100644 --- a/README.md +++ b/README.md @@ -222,6 +222,7 @@ curl -s http://127.0.0.1:8216/v1/chat/completions \ | DeepSeek V4 single-device and heterogeneous profiles | [DeepSeek V4 guide](server/docs/DS4.md) | | Environment variables | [Environment reference](server/docs/ENVIRONMENT.md) | | Server internals | [Architecture](server/docs/ARCHITECTURE.md) | +| Serving and engine component design | [Engine components](docs/specs/engine-components.md) | | Client integration and qualification | [Harness guide](harness/README.md) | Benchmarks stay with each implementation: [DFlash](server/RESULTS.md), [PFlash](optimizations/pflash/), [Spark](optimizations/spark/), [KVFlash](optimizations/kvflash/), and [Megakernel](optimizations/megakernel/). diff --git a/docs/specs/engine-components.md b/docs/specs/engine-components.md new file mode 100644 index 000000000..0652db4c0 --- /dev/null +++ b/docs/specs/engine-components.md @@ -0,0 +1,569 @@ +# Serving and engine components + +Status: Draft + +Source snapshot: `upstream/main` at `298031aa` + +## Summary + +Lucebox has two local generation paths behind one HTTP server: + +- The classic worker runs one complete request through `ModelBackend`. +- The concurrent scheduler advances several requests through `SeqEngine`. + +Both paths are valid. A full-request backend call and an iteration-level engine +step solve different problems, so this design does not combine them behind a +new universal interface. It instead defines the ownership boundaries between +the HTTP edge, serving coordination, model execution, architecture runtime, +GGML, auxiliary processes, and the client response. + +The first implementation work should preserve the established types and split +the large `http_server.cpp` translation unit by responsibility. Later changes +can share narrowly defined response and error state without forcing the two +execution paths into the same lifecycle. + +## Goals + +- Make a request traceable from the socket to GGML and back to the client. +- State which component owns transport, request policy, scheduling, model + state, tensor execution, and response formatting. +- Keep JSON and HTTP concerns out of model and GGML code. +- Preserve the distinction between whole-request and iteration-level + execution. +- Reduce the amount of server state a reader must hold in mind at once. +- Reuse current names when they already describe their responsibility. +- Introduce new types only when they remove duplicated state or make failure + handling explicit. +- Provide an incremental migration that can be reviewed and verified in small + changes. + +## Non-goals + +- Replacing `ModelBackend` and `SeqEngine` with one engine interface. +- Copying the process topology of vLLM or SGLang. +- Moving the normal local request path behind IPC. +- Renaming every request, slot, callback, or result type. +- Splitting every optional `ModelBackend` capability at once. +- Changing scheduling policy, cache behavior, or API output in the first file + split. +- Defining a request-wide `GenerationPlan` type. +- Hiding architecture-specific graph and cache state behind generic maps or + untyped payloads. + +## Current request paths + +### Startup + +`server_main.cpp` resolves configuration, creates the selected backend and +tokenizer, constructs `HttpServer`, and calls `HttpServer::run()`. + +At startup, `HttpServer::run()` selects the execution loop: + +- If `ModelBackend::seq_engine()` returns `nullptr`, it starts `worker_loop()`. +- If the backend exposes a `SeqEngine`, it starts `scheduler_loop()`. +- The upstream proxy path remains an HTTP concern and bypasses local model + execution for the forwarded request. + +Today, Qwen 3.5 is the worked concurrent implementation under +`server/src/qwen35/concurrency/`. Other model families use the classic path. + +### Relationship to inference configuration + +The configuration design in +[PR #688](https://github.com/Luce-Org/lucebox/pull/688) owns startup input, +validation, and backend construction. This design begins after the backend and +server configuration have been resolved. + +The two changes should meet at existing constructor boundaries. The component +work should not move startup parsing into `HttpServer`, and the configuration +work should not introduce request-lifecycle types. The initial server file +split uses `ServerConfig` and `ModelBackend` as they exist, so it can be stacked +or merged independently of the configuration refactor. + +### Common HTTP edge + +The request enters through these existing components: + +```text +client socket + -> HttpServer::handle_client() + -> HttpServer::route_request() + -> request parsing, chat rendering, and tokenization + -> ParsedRequest + -> ServerJob + -> server queue +``` + +`ParsedRequest` is the normalized, model-ready representation of one supported +HTTP request. It contains prompt tokens, sampling settings, output limits, +streaming mode, tools, stop sequences, and the API fields needed to construct +the response. + +`ServerJob` connects that request to its client socket and completion wait. The +client thread owns the job on its stack. The worker or scheduler borrows it +until it signals completion. + +### Classic execution + +The classic worker owns the complete lifecycle of one request: + +```text +ServerJob + -> HttpServer::process_job() + -> prepare prompt and cache state + -> GenerateRequest + -> ModelBackend::generate() or restore_and_generate() + -> architecture-specific prefill and decode + -> GGML graph execution + -> GenerateResult +``` + +`process_job()` also coordinates FlowKV, PFlash, prefix snapshots, draft +residency, agent-turn memory, status reporting, and final response delivery. +These are serving policies around generation. They are not responsibilities of +GGML. + +`ModelBackend::generate()` represents one complete prefill and decode cycle. +The backend selects autoregressive, speculative, or other model-specific +execution. `GenerateRequest` and `GenerateResult` are the typed boundary for +that call. + +`DaemonIO` carries the legacy daemon stream descriptor together with token, +cancellation, and observation callbacks. The HTTP server uses the callbacks; +the stdin daemon protocol still uses the descriptor. This mixed role should be +reduced only after the two callers can be migrated independently. + +### Concurrent execution + +The concurrent path keeps request policy in the scheduler and model state in +the engine: + +```text +ServerJob + -> HttpServer::scheduler_loop() + -> SchedSlot + -> SeqEngine::admit() + -> SeqEngine::step() + -> architecture-specific batched forward and sampling + -> GGML graph execution + -> SeqEngine::StepResult +``` + +The scheduler owns admission order, fairness, output caps, stop conditions, +thinking-budget token substitution, client backpressure, and retirement. Its +`SchedSlot` contains only server-side request progress and response state. + +The engine owns slot allocation, prompt progress, paged KV blocks, recurrent +state, graph shapes, tensor inputs, batched forward execution, and sampling. +Those details remain inside the architecture implementation. The scheduler +sees only slot identifiers and the existing `SeqEngine` inputs and outputs. + +`SeqEngine::StepPlan` is intentionally limited to one scheduler iteration. It +is not a request-wide configuration object and should remain named for the +single step it describes. + +### Response path + +Both local execution paths use the same response components: + +```text +token or terminal result + -> SseEmitter + -> API-specific events or complete JSON + -> direct socket write in the classic worker + or ClientSendBuffer in the concurrent scheduler + -> client +``` + +`SseEmitter` currently owns two concerns: + +1. Semantic response state, including reasoning, content, tool calls, stop + sequences, and finish reason. +2. OpenAI, Anthropic, and Responses API event formatting. + +This combination keeps behavior consistent today, but it makes non-streaming +and streaming response construction harder to reason about. A later extraction +can move the first concern into `ResponseState` while keeping `SseEmitter` as +the established wire-format component. + +`ClientSendBuffer` is specific to the shared concurrent loop. It prevents a +slow reader from blocking other active sequences. The classic worker can write +directly because it serves only one generation at a time. + +### Auxiliary process execution + +IPC is an optional branch inside backend and optimization implementations. It +is not the normal boundary between `HttpServer` and `ModelBackend`. + +```text +ModelBackend or architecture runtime + -> role-specific IPC client + -> BackendIpcProcess + -> draft, compression, shard, or expert subprocess +``` + +`BackendIpcProcess` owns process launch, pipes, shared payload setup, status +transport, scratch paths, and shutdown. Role-specific clients such as +`DFlashDraftIpcClient`, `PFlashDrafterIpcClient`, and +`TargetShardIpcSession` own their payload protocols. + +This distinction matters for both naming and failure handling. A transport +failure belongs to the IPC session. A generation or model failure belongs to a +typed engine result. The server should not need to decode subprocess strings. + +## Target ownership + +The dependency direction should remain one way: + +```text +HTTP and API edge + -> serving request state + -> classic worker or concurrent scheduler + -> ModelBackend or SeqEngine + -> architecture runtime + -> GGML + +architecture runtime + -> optional role-specific IPC client + -> BackendIpcProcess + +GGML result + -> typed backend or engine result + -> response state + -> API events or response JSON + -> socket or ClientSendBuffer +``` + +### HTTP and API edge + +Owned by `server/src/server/`. + +Responsibilities: + +- Accept sockets and parse HTTP framing. +- Route supported endpoints. +- Validate JSON and map API aliases into one internal representation. +- Apply chat templates and tokenize prompts. +- Construct API-specific success and error responses. +- Detect client disconnects and manage streaming headers. + +The edge may depend on tokenizer, chat-template, and API-format code. Model and +GGML code must not depend on HTTP status codes, SSE frames, or request JSON. + +### Serving request state + +Keep these current names: + +- `ParsedRequest` for the normalized request accepted by local serving. +- `ServerJob` for the socket-bound queued unit and its completion wait. +- `GenerationInputs` for the classic worker's private aggregate. +- `SchedSlot` for one request's server-side concurrent state. + +These names match their scope. Promoting `GenerationInputs` into a shared +engine contract would be a mistake because the classic and concurrent paths do +not consume the same unit of work. + +The main improvement is ownership, not renaming. Fields should move out of +these structures only when another component becomes their clear owner. + +### Serving coordination + +The classic worker and concurrent scheduler should remain separate loops. +They can share pure policy functions and response construction, but not a +synthetic lifecycle interface. + +The classic worker owns: + +- Whole-request prompt preparation and cache restore. +- Request-scoped draft residency. +- One call to `ModelBackend::generate()` or `restore_and_generate()`. +- Whole-request cache finalization and performance reporting. + +The concurrent scheduler owns: + +- Admission and fairness. +- Prefill slice selection. +- Batched decode iteration order. +- Per-slot stop and retirement decisions. +- Non-blocking response delivery. + +Small shared functions should describe the value they resolve, for example +`resolve_generation_cap()`, rather than collecting unrelated decisions into a +new request-wide object. + +### Model execution + +Keep the two current contracts: + +- `ModelBackend` owns backend lifetime, whole-request generation, snapshots, + and optional capabilities. +- `SeqEngine` is the optional interface for backends that can keep several + live sequences and execute scheduler-selected work together. + +`ModelBackend::seq_engine()` is enough to select the concurrent path. A third +base class named `Engine`, `ServingEngine`, or similar would add indirection +without removing either existing contract. + +`ModelBackend` is broad, but its optional methods should be extracted only when +there is a concrete caller and more than one useful implementation. Until +then, grouped methods and capability checks are easier to follow than a set of +one-method interfaces. + +Concurrent failures should eventually reuse `GenerateError` and +`GenerateErrorCode` rather than add another string-based error family. The +exact migration can update `AdmitResult`, `DecodeOutput`, `PrefillOutput`, and +`StepResult` independently while preserving each result's current scope. + +### Architecture runtime and GGML + +Owned by model-family directories such as `qwen35/`, `gemma4/`, `laguna/`, +and `deepseek4/`, together with genuinely model-neutral helpers in `common/`. + +Responsibilities: + +- Load weights and choose device placement. +- Own model-specific KV, recurrent, and speculative state. +- Build graphs and bind tensor inputs. +- Execute GGML backends and read outputs. +- Implement model-specific prefill, decode, and sampling mechanisms. + +The engine boundary should expose tokens, sampling configuration, progress, +timings, and typed failures. It should not expose graph tensors, allocator +handles, block-table layouts, or architecture-specific state to the server. + +Code belongs in `common/` only when at least two model families can use the +same semantics. Similar graph code is not automatically the same component if +the model invariants differ. + +### Response state and transport + +Keep these current names: + +- `SseEmitter` for API event construction and SSE framing. +- `ClientSendBuffer` for non-blocking concurrent socket output. + +Introduce `ResponseState` only when semantic accumulation is physically moved +out of `SseEmitter`. It should own content, reasoning, tool-call parsing, stop +matching, and finish reason. It should not know about SSE syntax, HTTP status, +or sockets. + +The draft terminal-error change in +[PR #689](https://github.com/Luce-Org/lucebox/pull/689) is the first part of +this boundary. It gives the API layer one response error representation while +preserving `GenerateError` as the backend-facing failure. + +### IPC transport + +Keep `BackendIpcProcess` as the process and transport owner. Keep payload +semantics in role-specific clients. + +An IPC client should either complete one protocol transaction or invalidate +the session. This prevents a partial read or write from being mistaken for the +next response. A small shared helper for marking a process unusable is +preferable to a generic request envelope shared by unrelated IPC modes. + +## Source organization + +The first structural change should split `http_server.cpp` without changing +the `HttpServer` class or request behavior: + +| File | Responsibility | +|---|---| +| `http_server.cpp` | Server lifetime, accept loop, socket I/O, job queue, and disconnect monitoring | +| `http_routes.cpp` | Endpoint routing, request validation, chat rendering, tokenization, and `ParsedRequest` construction | +| `generation_worker.cpp` | Classic `worker_loop()`, `process_job()`, prompt preparation, cache lifecycle, and backend call | +| `scheduler.cpp` | Concurrent admission, iteration policy, slot lifecycle, and buffered delivery | +| `sse_emitter.cpp` | Existing semantic stream state and API event formatting until `ResponseState` is extracted | + +This split changes file ownership, not public interfaces. Existing +`HttpServer` member functions can be defined across the translation units. +Tests and the server target should compile the same source set. + +After the split, helpers that are used by only one file should move into that +file's anonymous namespace. Helpers shared by classic and concurrent serving +should have narrow typed signatures in a server-local header. + +## Migration order + +### 1. Make terminal results explicit + +Land the API-facing terminal error boundary from PR #689. Both execution paths +must branch on success or failure before success finalization, cache updates, +or HTTP 200 responses. + +### 2. Split the server translation unit + +Create `http_routes.cpp` and `generation_worker.cpp`, then move existing +functions with no behavior or naming changes. Update both the server target and +model-free test target together. + +This change creates reviewable component boundaries before adding new types. + +### 3. Extract semantic response state + +Move content, reasoning, tool-call, stop-sequence, and finish-reason state from +`SseEmitter` into `ResponseState`. + +Both streaming and non-streaming builders should consume the same state. +`SseEmitter` should remain responsible for API events and SSE framing. This +keeps the familiar name and removes the current accumulation ambiguity. + +### 4. Share only common request policy + +Extract pure helpers for values that classic and concurrent serving must +derive identically, beginning with generation cap, thinking budget, EOS +classification, and terminal error mapping. + +Do not create a shared request executor. Each path should call the helper at +the point where it owns the relevant decision. + +### 5. Type concurrent engine failures + +Replace free-form `SeqEngine` failure strings with `GenerateError` values. +Keep admission, per-row, and whole-step results distinct. The scheduler can +then map both classic and concurrent failures through the same response error +function. + +### 6. Narrow `DaemonIO` + +Move the legacy file-descriptor behavior behind the daemon caller. Let HTTP +generation pass only the existing token callback, cancellation probe, and +inference observer responsibilities needed by the backend. + +Migrate all callers in the same change before deleting unused fields. Do not +add a compatibility wrapper that preserves both shapes indefinitely. + +### 7. Harden IPC transactions + +Centralize the rule that a framing, payload, or subprocess failure closes the +affected `BackendIpcProcess`. Keep the error returned to generation typed and +specific to the role-specific client. + +### 8. Revisit optional backend capabilities + +After call sites are narrow and covered by tests, measure whether snapshots, +compression, parking, or remote draft support benefit from separate capability +interfaces. Extract only the groups that reduce real coupling. + +## Naming rules + +New names should reveal both scope and owner: + +- Use `Request` and `Response` for API or whole-request data. +- Use `Admission`, `Slot`, `Prefill`, `Decode`, and `Step` for scheduler and + concurrent engine data. +- Use `Backend` for model-family lifetime and capability ownership. +- Use `Ipc` plus the remote role for subprocess clients and sessions. +- Use `State` only for data that persists across calls. +- Use `Result` for a completed operation with success or failure. + +Avoid names that hide the unit of work. In particular, do not introduce +`GenerationPlan`, `ApiRequest`, `ServingRequest`, `OwnedGenerationPlan`, +`GenerationCallbacks`, `RequestSlotState`, or `ApiStreamEncoder` as renames of +the current types. The established names are clearer when their ownership is +documented and their files are smaller. + +## Verification + +Each migration step should leave both execution paths usable. + +Required checks: + +- Build `dflash_server` for the enabled CUDA or HIP configuration. +- Build and run the model-free server unit tests. +- Run `test_seq_engine_contract` for each concurrent engine. +- Exercise streaming and non-streaming success responses for all supported API + formats. +- Exercise classic and concurrent backend failures and verify non-200 or SSE + error termination. +- Exercise client disconnects during prefill and decode. +- Exercise a slow concurrent reader and verify other slots continue. +- Exercise prefix restore, request-scoped draft residency, and upstream proxy + paths after moving the classic worker. +- Check that model-family and `common/` sources do not construct HTTP JSON or + SSE frames. +- Check that server sources do not depend on architecture-specific graph or KV + structures. + +The file split should produce no wire-format or scheduling changes. A useful +review technique is to compare the moved function bodies before and after the +split and keep functional edits in later commits. + +## Alternatives rejected + +### One universal engine interface + +`ModelBackend::generate()` completes a request. `SeqEngine::step()` advances a +selected batch by one iteration. A common interface would either expose both +lifecycle models or reduce one to callbacks around the other. Neither result +simplifies the caller. + +### An internal event bus + +The request path is direct and performance-sensitive. Typed calls and results +make ownership and failure propagation visible. An event bus would obscure +ordering, lifetime, and cancellation without adding a current deployment +benefit. + +### IPC between HTTP and every backend + +vLLM and SGLang use process boundaries to support their deployment and +scheduling designs. Lucebox currently has a direct in-process local path plus +targeted subprocesses for heterogeneous execution. Mandatory IPC would add +serialization and lifecycle work without resolving the current source +organization problem. + +### A broad rename pass + +The current names mostly identify real units: parsed request, server job, +backend request and result, scheduler slot, engine step, SSE emitter, send +buffer, and IPC process. Moving responsibilities first gives any future rename +concrete evidence and keeps review focused. + +### Splitting all `ModelBackend` capabilities now + +Small interfaces are useful when they let callers depend on less. Creating one +interface per optional method before narrowing callers would increase the +number of types without changing ownership. + +### JSON below the server layer + +JSON is part of the external API contract. Passing it into backend or engine +code couples model execution to current endpoints and makes non-HTTP callers +harder to support. + +## Related designs + +These projects support the ownership direction in this document, but Lucebox +should not copy their process layouts. + +- [vLLM architecture overview](https://github.com/vllm-project/vllm/blob/main/docs/design/arch_overview.md) + separates API input and output processing from an engine core that owns + scheduling, KV cache management, and worker coordination. Its API and engine + core communicate across ZMQ because vLLM chooses a multi-process topology. +- [SGLang manager I/O structures](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/managers/io_struct.py) + distinguish external generation input from tokenized scheduler input. The + useful lesson is the typed boundary between stages, not the number of + manager processes. +- [llama.cpp server developer guide](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README-dev.md) + separates HTTP context, routes, tasks, queues, inference slots, task results, + and response readers. It also keeps JSON formatting and chat templates in + the HTTP layer and passes native C++ types to inference slots. + +Lucebox differs in two important ways. It supports a classic whole-request +backend path beside continuous batching, and it uses auxiliary IPC inside +specific heterogeneous execution features. The target structure must make +those differences explicit. + +## Open questions + +- Should `ResponseState` be extracted immediately after the file split, or + should terminal error parity land in both paths first? +- Which serving policies must reach parity before another model family exposes + `SeqEngine`? +- Can the daemon protocol stop sharing `DaemonIO` with the HTTP server without + affecting external scripts? +- Should all concurrent failures use `GenerateError`, or is a smaller subset + sufficient for `SeqEngine`? +- When this design is implemented, should `server/docs/ARCHITECTURE.md` become + an operational overview that links here for component ownership? From ff828e1fffa16d19576ef9c2ce8cb86cfdba75c8 Mon Sep 17 00:00:00 2001 From: Graffioh Date: Wed, 2 Sep 2026 09:23:22 +0200 Subject: [PATCH 2/2] docs: redesign engine ownership around LuceEngine --- README.md | 2 +- docs/specs/engine-components.md | 1040 +++++++++++++++++++------------ 2 files changed, 631 insertions(+), 411 deletions(-) diff --git a/README.md b/README.md index f00c0a3e2..be2625d1e 100644 --- a/README.md +++ b/README.md @@ -222,8 +222,8 @@ curl -s http://127.0.0.1:8216/v1/chat/completions \ | DeepSeek V4 single-device and heterogeneous profiles | [DeepSeek V4 guide](server/docs/DS4.md) | | Environment variables | [Environment reference](server/docs/ENVIRONMENT.md) | | Server internals | [Architecture](server/docs/ARCHITECTURE.md) | -| Serving and engine component design | [Engine components](docs/specs/engine-components.md) | | Client integration and qualification | [Harness guide](harness/README.md) | +| LuceEngine component design | [Engine components](docs/specs/engine-components.md) | Benchmarks stay with each implementation: [DFlash](server/RESULTS.md), [PFlash](optimizations/pflash/), [Spark](optimizations/spark/), [KVFlash](optimizations/kvflash/), and [Megakernel](optimizations/megakernel/). diff --git a/docs/specs/engine-components.md b/docs/specs/engine-components.md index 0652db4c0..a7b786273 100644 --- a/docs/specs/engine-components.md +++ b/docs/specs/engine-components.md @@ -1,4 +1,4 @@ -# Serving and engine components +# LuceEngine component design Status: Draft @@ -6,564 +6,784 @@ Source snapshot: `upstream/main` at `298031aa` ## Summary -Lucebox has two local generation paths behind one HTTP server: +Lucebox currently exposes two local generation paths to `HttpServer`: -- The classic worker runs one complete request through `ModelBackend`. -- The concurrent scheduler advances several requests through `SeqEngine`. +- A worker runs a complete request through `ModelBackend::generate()`. +- A scheduler advances concurrent requests through `SeqEngine`. -Both paths are valid. A full-request backend call and an iteration-level engine -step solve different problems, so this design does not combine them behind a -new universal interface. It instead defines the ownership boundaries between -the HTTP edge, serving coordination, model execution, architecture runtime, -GGML, auxiliary processes, and the client response. +This split let continuous batching land without rewriting every model family. +It now makes the HTTP layer responsible for model scheduling and gives +request-level generation two unrelated shapes. -The first implementation work should preserve the established types and split -the large `http_server.cpp` translation unit by responsibility. Later changes -can share narrowly defined response and error state without forcing the two -execution paths into the same lifecycle. +The target has one request boundary: + +```text +HTTP or daemon adapter + -> LuceEngine::generate() + -> request lifecycle and scheduling + -> model execution capability + -> model-family state + -> GGML + <- token events and one terminal result + -> protocol formatting + -> client +``` + +The boundary is structural. C++ access modifiers do not define it. +`LuceEngine` owns request scheduling, model access, cancellation, and +completion. `HttpServer` owns HTTP and sockets. Model-family code owns weights, +caches, graphs, and GGML objects. + +There is one request-level function named `generate()`. The final design does +not retain `ModelBackend::generate()` below `LuceEngine::generate()`, and it +does not present `ModelBackend` and `SeqEngine` as peer engines. ## Goals -- Make a request traceable from the socket to GGML and back to the client. -- State which component owns transport, request policy, scheduling, model - state, tensor execution, and response formatting. -- Keep JSON and HTTP concerns out of model and GGML code. -- Preserve the distinction between whole-request and iteration-level - execution. -- Reduce the amount of server state a reader must hold in mind at once. -- Reuse current names when they already describe their responsibility. -- Introduce new types only when they remove duplicated state or make failure - handling explicit. -- Provide an incremental migration that can be reviewed and verified in small - changes. +- Give HTTP and daemon generation one operation with one lifecycle. +- Keep scheduling mode out of request callers. +- Define each component by the decisions and state it owns. +- Keep HTTP, JSON, SSE, and socket types out of engine and model execution. +- Keep GGML, graph, cache-layout, and device types out of HTTP and engine + request types. +- Preserve the concurrency invariants introduced by PR #594. +- Preserve serial generation behavior, including speculative empty-output + retry, snapshot restore, cache behavior, and cancellation. +- Make request data safe to retain after `generate()` returns. +- Give shutdown and model control one serialization point. +- Migrate in steps that leave one usable request path after each change. ## Non-goals -- Replacing `ModelBackend` and `SeqEngine` with one engine interface. -- Copying the process topology of vLLM or SGLang. -- Moving the normal local request path behind IPC. -- Renaming every request, slot, callback, or result type. -- Splitting every optional `ModelBackend` capability at once. -- Changing scheduling policy, cache behavior, or API output in the first file - split. -- Defining a request-wide `GenerationPlan` type. -- Hiding architecture-specific graph and cache state behind generic maps or - untyped payloads. +- Copying the multi-process topology of vLLM or SGLang. +- Moving normal local generation behind IPC. +- Making every model family implement continuous batching. +- Forcing complete-request and batch-step execution into the same low-level + method. +- Passing `ParsedRequest` or `ServerConfig` into model-family code. +- Adding a request-wide `GenerationPlan` type. +- Renaming existing types without moving ownership. +- Splitting every optional backend capability before its callers move. -## Current request paths +## Why the current split exists -### Startup +`ModelBackend` and `SeqEngine` were created for different callers and at +different times. -`server_main.cpp` resolves configuration, creates the selected backend and -tokenizer, constructs `HttpServer`, and calls `HttpServer::run()`. +Commit `3d1dcad85` introduced `ModelBackend` as the shared adapter for the +line-oriented daemon. Its whole-request operation matched the daemon command: +prefill, decode, stream tokens, and return a result. The same interface also +collected park, snapshot, compression, and command hooks. -At startup, `HttpServer::run()` selects the execution loop: +Commits `35784218` and `9daa25c2`, followed by +[PR #594](https://github.com/Luce-Org/lucebox/pull/594), added concurrent Qwen +serving in two parts. `SeqEngine` first defined model-side slot execution. +`scheduler_loop()` then added admission, fair prefill, cancellation, and +non-blocking client delivery. `HttpServer::run()` selected the scheduler only +when a backend returned a sequence engine. -- If `ModelBackend::seq_engine()` returns `nullptr`, it starts `worker_loop()`. -- If the backend exposes a `SeqEngine`, it starts `scheduler_loop()`. -- The upstream proxy path remains an HTTP concern and bypasses local model - execution for the forwarded request. +That history explains the asymmetry. It does not require the HTTP server to +keep choosing between the two paths. -Today, Qwen 3.5 is the worked concurrent implementation under -`server/src/qwen35/concurrency/`. Other model families use the classic path. +The useful boundary from PR #594 remains: -### Relationship to inference configuration +- Request coordination owns admission order, fairness, stop decisions, + cancellation, client progress, and retirement. +- Model execution owns slot allocation, KV blocks, recurrent state, graph + shapes, batched forward execution, and sampling. -The configuration design in -[PR #688](https://github.com/Luce-Org/lucebox/pull/688) owns startup input, -validation, and backend construction. This design begins after the backend and -server configuration have been resolved. +The new structure moves request coordination into `LuceEngine`. It does not +move it into GGML or model-family code. -The two changes should meet at existing constructor boundaries. The component -work should not move startup parsing into `HttpServer`, and the configuration -work should not introduce request-lifecycle types. The initial server file -split uses `ServerConfig` and `ModelBackend` as they exist, so it can be stacked -or merged independently of the configuration refactor. +## Current flow -### Common HTTP edge +### Startup -The request enters through these existing components: +`server_main.cpp` constructs a backend, a tokenizer, and `HttpServer`. +`HttpServer::run()` then checks `ModelBackend::seq_engine()` once: ```text -client socket - -> HttpServer::handle_client() - -> HttpServer::route_request() - -> request parsing, chat rendering, and tokenization - -> ParsedRequest - -> ServerJob - -> server queue +seq_engine() == nullptr -> worker_loop() +seq_engine() != nullptr -> scheduler_loop() ``` -`ParsedRequest` is the normalized, model-ready representation of one supported -HTTP request. It contains prompt tokens, sampling settings, output limits, -streaming mode, tools, stop sequences, and the API fields needed to construct -the response. - -`ServerJob` connects that request to its client socket and completion wait. The -client thread owns the job on its stack. The worker or scheduler borrows it -until it signals completion. - -### Classic execution +The upstream proxy forces the worker path even when the backend has a +`SeqEngine`. -The classic worker owns the complete lifecycle of one request: +### Complete-request execution ```text -ServerJob - -> HttpServer::process_job() - -> prepare prompt and cache state +HttpServer::process_job() + -> prompt preparation and cache selection -> GenerateRequest -> ModelBackend::generate() or restore_and_generate() - -> architecture-specific prefill and decode + -> model-specific prefill and decode -> GGML graph execution + -> DaemonIO token callbacks -> GenerateResult + -> cache and response finalization ``` -`process_job()` also coordinates FlowKV, PFlash, prefix snapshots, draft -residency, agent-turn memory, status reporting, and final response delivery. -These are serving policies around generation. They are not responsibilities of -GGML. +`process_job()` owns HTTP state and model-serving policy in one function. It +coordinates PFlash, FlowKV, prefix snapshots, draft residency, generation +limits, status, token delivery, and response completion. -`ModelBackend::generate()` represents one complete prefill and decode cycle. -The backend selects autoregressive, speculative, or other model-specific -execution. `GenerateRequest` and `GenerateResult` are the typed boundary for -that call. - -`DaemonIO` carries the legacy daemon stream descriptor together with token, -cancellation, and observation callbacks. The HTTP server uses the callbacks; -the stdin daemon protocol still uses the descriptor. This mixed role should be -reduced only after the two callers can be migrated independently. - -### Concurrent execution - -The concurrent path keeps request policy in the scheduler and model state in -the engine: +### Batch execution ```text -ServerJob - -> HttpServer::scheduler_loop() +HttpServer::scheduler_loop() -> SchedSlot -> SeqEngine::admit() + -> build StepPlan -> SeqEngine::step() - -> architecture-specific batched forward and sampling - -> GGML graph execution - -> SeqEngine::StepResult + -> apply token and stop policy + -> buffer client output + -> SeqEngine::retire() ``` -The scheduler owns admission order, fairness, output caps, stop conditions, -thinking-budget token substitution, client backpressure, and retirement. Its -`SchedSlot` contains only server-side request progress and response state. +`SchedSlot` contains two kinds of state. Request progress, slot identity, token +history, and cancellation belong to generation. Sockets, `SseEmitter`, and +`ClientSendBuffer` belong to HTTP response delivery. -The engine owns slot allocation, prompt progress, paged KV blocks, recurrent -state, graph shapes, tensor inputs, batched forward execution, and sampling. -Those details remain inside the architecture implementation. The scheduler -sees only slot identifiers and the existing `SeqEngine` inputs and outputs. +`Qwen35SeqEngine` is not a peer of `Qwen35Backend`. The backend owns the +sequence engine, paged KV pool, weights, graphs, and GGML backends. The +sequence engine borrows those resources to run batched steps. -`SeqEngine::StepPlan` is intentionally limited to one scheduler iteration. It -is not a request-wide configuration object and should remain named for the -single step it describes. +## Target ownership -### Response path +### Composition and lifetime -Both local execution paths use the same response components: +The configuration design in +[PR #688](https://github.com/Luce-Org/lucebox/pull/688) and its implementation +follow-up own launch-time input, validation, and model construction. The engine +design begins after backend construction succeeds: ```text -token or terminal result - -> SseEmitter - -> API-specific events or complete JSON - -> direct socket write in the classic worker - or ClientSendBuffer in the concurrent scheduler - -> client +BackendArgs + BackendAdmissionContext + -> BackendPlan + -> create_backend(plan) + -> BackendRuntime + -> LuceEngine + -> HttpServer ``` -`SseEmitter` currently owns two concerns: +`BackendPlan` remains launch-time data. It is not a request or scheduler plan. +`BackendRuntime` owns the validated plan and the model-family resources whose +configuration points into it. + +`LuceEngine` takes ownership of `BackendRuntime`. `HttpServer` borrows +`LuceEngine` for the duration of server execution. This ownership gives the +shutdown order a concrete shape: + +1. `HttpServer` stops accepting work and cancels its live generations. +2. `LuceEngine` rejects new requests, completes or retires active work, and + joins its execution thread. +3. `BackendRuntime` releases model and device resources as part of + `LuceEngine` destruction. + +`ServerConfig` remains the HTTP and request-default configuration. Construct +`LuceEngine` with only scheduler, cache, and output-channel values that the +engine owns. Do not pass the full `ServerConfig` into backend preparation, +backend construction, or model code. + +### Caller view + +HTTP prepares an owned model request and receives a generation handle: + +```cpp +GenerateRequest request = make_generate_request(parsed_request); +Generation generation = engine.generate(std::move(request)); + +for (;;) { + GenerateEvent event = generation.next(); + if (const auto * tokens = std::get_if(&event)) { + response.write(tokens->tokens); + continue; + } + if (const auto * progress = std::get_if(&event)) { + status.update(*progress); + continue; + } + + response.finish(std::get(event).result); + break; +} +``` -1. Semantic response state, including reasoning, content, tool calls, stop - sequences, and finish reason. -2. OpenAI, Anthropic, and Responses API event formatting. +The daemon uses the same operation and maps events to its descriptor protocol. +Neither caller checks concurrency, engine slots, or model type. -This combination keeps behavior consistent today, but it makes non-streaming -and streaming response construction harder to reason about. A later extraction -can move the first concern into `ResponseState` while keeping `SseEmitter` as -the established wire-format component. +Upstream forwarding remains an HTTP transport operation. It does not pretend +to be local generation. -`ClientSendBuffer` is specific to the shared concurrent loop. It prevents a -slow reader from blocking other active sequences. The classic worker can write -directly because it serves only one generation at a time. +### Request and completion -### Auxiliary process execution +The existing names remain where their meaning still fits: -IPC is an optional branch inside backend and optimization implementations. It -is not the normal boundary between `HttpServer` and `ModelBackend`. +```cpp +struct GenerateRequest { + std::vector prompt; + int n_gen; + SamplerCfg sampler; + BudgetHook budget_hook; + std::vector stop_sequences; + std::optional restore_from; + std::optional capture; -```text -ModelBackend or architecture runtime - -> role-specific IPC client - -> BackendIpcProcess - -> draft, compression, shard, or expert subprocess -``` + // These become owned values instead of pointers into an HTTP stack frame. + std::vector hint_tokens; + std::vector stall_tool_prefix_tokens; + std::vector stall_action_suffix_tokens; + std::vector stall_skip_tokens; +}; -`BackendIpcProcess` owns process launch, pipes, shared payload setup, status -transport, scratch paths, and shutdown. Role-specific clients such as -`DFlashDraftIpcClient`, `PFlashDrafterIpcClient`, and -`TargetShardIpcSession` own their payload protocols. +struct TokenBatch { + std::vector tokens; +}; -This distinction matters for both naming and failure handling. A transport -failure belongs to the IPC session. A generation or model failure belongs to a -typed engine result. The server should not need to decode subprocess strings. +struct GenerationProgress { + GenerationPhase phase; + int processed_tokens; +}; -## Target ownership +struct GenerateCompleted { + GenerateResult result; +}; -The dependency direction should remain one way: +using GenerateEvent = + std::variant; +``` -```text -HTTP and API edge - -> serving request state - -> classic worker or concurrent scheduler - -> ModelBackend or SeqEngine - -> architecture runtime - -> GGML - -architecture runtime - -> optional role-specific IPC client - -> BackendIpcProcess - -GGML result - -> typed backend or engine result - -> response state - -> API events or response JSON - -> socket or ClientSendBuffer +`GenerateRequest` contains model-ready, transport-free input. It does not +contain a file descriptor, JSON value, HTTP format, response identifier, or +socket callback. + +The request owns normalized stop strings. `LuceEngine` receives the model's +token decoder as a construction dependency and keeps one incremental matcher +per request. A stop string may cross token boundaries. It is not reduced to a +list of independently tokenized suffixes. + +The first migration keeps the existing complete token vector in +`GenerateResult`. The generation operation owns that vector. Token events are +bounded in-flight copies used for streaming. This preserves daemon, cache, and +telemetry callers while making the memory cost explicit. + +### `Generation` + +```cpp +class Generation { +public: + GenerateEvent next(); + void cancel(GenerationCancelReason reason); + ~Generation(); +}; + +class LuceEngine { +public: + LuceEngine(std::unique_ptr runtime, + const TokenDecoder & decoder, + LuceEngineConfig config); + + Generation generate(GenerateRequest request); + ControlResult control(ControlCommand command); + void shutdown(); +}; ``` -### HTTP and API edge +`Generation` is move-only. It owns a reference to channel and terminal state. +Dropping an unfinished handle cancels its request. + +Each request has a bounded token channel and a separate terminal cell. Model +execution never waits for a socket. When a consumer stops draining and the +channel fills, `LuceEngine` cancels only that request with a typed output +backpressure result. The terminal cell remains writable even when the token +channel is full. + +The request queue is bounded. When the queue is full, or shutdown has started, +`generate()` returns an already-completed `Generation` with a typed rejection. +It does not block or throw. HTTP calls `generate()` before committing streaming +headers, so it can map an immediate rejection to a normal HTTP status. + +`shutdown()` is idempotent. It wakes every waiting `Generation`, publishes one +terminal result for each request, retires model state on the engine thread, +and joins before returning. A `Generation` handle may outlive `LuceEngine`, but +after shutdown it contains only detached channel and terminal state. It cannot +retain a backend, executor, observer, or engine pointer. + +### `LuceEngine` + +`LuceEngine` owns: + +- The bounded request queue and engine thread. +- Request IDs and operation state. +- Selection of serial or continuous-batch coordination at construction. +- Admission order, fair prefill selection, generation limits, cancellation, + retry decisions, and retirement. +- Incremental stop matching and thinking-budget token decisions before the + token returns to model KV. +- Typed terminal completion and output-channel backpressure. +- Prefix-cache policy and coordination with physical snapshots. +- Typed model controls serialized with live generation. +- Backend lifetime and orderly shutdown. + +`LuceEngine` does not own HTTP routes, API response formats, sockets, GGML +graphs, block tables, or architecture-specific cache layouts. + +The serial and batch coordinators are implementation components inside this +ownership boundary. They are not alternate entry points. Both consume the +same engine request state and publish through the same `Generation` channel. + +### Model execution capabilities + +Complete-request and batch-step execution remain different below +`LuceEngine`. Their contracts use names on the same axis: + +```cpp +class SingleRequestExecutor { +public: + virtual GenerateResult execute( + const GenerateRequest & request, + ExecutionHooks & hooks) = 0; +}; + +class BatchExecutor { +public: + virtual SlotClaim claim(const GenerateRequest & request) = 0; + virtual StepPlanLimits step_limits(int decode_rows) const = 0; + virtual BatchStepResult step(const BatchStepPlan & plan) = 0; + virtual void retire(SlotId slot) = 0; + virtual bool token_is_eos(int32_t token) const = 0; +}; +``` -Owned by `server/src/server/`. +`ExecutionHooks` is transport-free. It supplies cancellation, progress, and a +synchronous pre-commit token decision. For each sampled token, the +single-request executor asks the hook whether to accept, replace, or stop +before it writes that token to reusable KV state. Accepted tokens then enter +the bounded `Generation` channel. Progress reports are model-neutral and may +be coalesced when a consumer falls behind. -Responsibilities: +The batch coordinator performs the same decision directly because +`BatchExecutor::step()` already returns pending tokens before the next step +commits them. This keeps one owner for stop matching and force-close policy +without making a complete-request backend expose a batch-step lifecycle. -- Accept sockets and parse HTTP framing. -- Route supported endpoints. -- Validate JSON and map API aliases into one internal representation. -- Apply chat templates and tokenize prompts. -- Construct API-specific success and error responses. -- Detect client disconnects and manage streaming headers. +A model backend supplies exactly one execution capability. The type selected +at construction is fixed for the engine lifetime. -The edge may depend on tokenizer, chat-template, and API-format code. Model and -GGML code must not depend on HTTP status codes, SSE frames, or request JSON. +`ModelBackend::generate()` does not exist in the final structure. Existing +`generate_impl()` bodies move to `SingleRequestExecutor::execute()`. Existing +`SeqEngine` implementations move to `BatchExecutor` after the scheduler sits +behind `LuceEngine`. -### Serving request state +This is not a universal low-level engine interface. It records the two real +model execution units without making HTTP understand either one. -Keep these current names: +### Model-family and GGML boundary -- `ParsedRequest` for the normalized request accepted by local serving. -- `ServerJob` for the socket-bound queued unit and its completion wait. -- `GenerationInputs` for the classic worker's private aggregate. -- `SchedSlot` for one request's server-side concurrent state. +`BackendRuntime` and the model-family backend own: -These names match their scope. Promoting `GenerationInputs` into a shared -engine contract would be a mistake because the classic and concurrent paths do -not consume the same unit of work. +- Loaded weights and model configuration. +- Target and draft GGML backends. +- KV, recurrent, and speculative state. +- Graph construction and tensor binding. +- Model-specific prefill, decode, verification, and sampling. +- Scratch buffers and device placement. +- Physical snapshot capture and restore. +- Role-specific IPC clients. -The main improvement is ownership, not renaming. Fields should move out of -these structures only when another component becomes their clear owner. +No `ggml_*` type, graph tensor, allocator, block table, or model-specific state +crosses into `LuceEngine`, `Generation`, or `HttpServer`. -### Serving coordination +`ModelBackend` remains the migration name for this resource owner. Do not +rename it until generation and daemon concerns have moved out and its final +responsibility can be judged from the remaining code. -The classic worker and concurrent scheduler should remain separate loops. -They can share pure policy functions and response construction, but not a -synthetic lifecycle interface. +### Stop conditions and token commitment -The classic worker owns: +Stop conditions that affect model progress belong to generation policy, not +wire formatting. `LuceEngine` owns EOS, generation limits, normalized stop +sequences, cancellation, and thinking-budget force-close. -- Whole-request prompt preparation and cache restore. -- Request-scoped draft residency. -- One call to `ModelBackend::generate()` or `restore_and_generate()`. -- Whole-request cache finalization and performance reporting. +The engine must decide whether to continue before a sampled token is fed back +into KV. This preserves the current batch invariant that a force-close or stop +decision can replace or reject a pending token before the next step commits +it. API-specific reasoning, tool-call, and event formatting remain in +`SseEmitter` or a later transport-neutral semantic response component. -The concurrent scheduler owns: +The incremental stop matcher and budget hook live in engine request state. The +single-request executor calls `ExecutionHooks` before commit. The batch +coordinator applies the same decision to each pending token before it builds +the next step. HTTP never decides whether model execution continues. It only +formats token events already accepted by generation policy. -- Admission and fairness. -- Prefill slice selection. -- Batched decode iteration order. -- Per-slot stop and retirement decisions. -- Non-blocking response delivery. +### Retry ownership -Small shared functions should describe the value they resolve, for example -`resolve_generation_cap()`, rather than collecting unrelated decisions into a -new request-wide object. +`LuceEngine` owns the common empty speculative-output decision now implemented +by `ModelBackend::generate()` and `restore_and_generate()`. -### Model execution +The single-request coordinator preserves the current retry and result-merge +behavior. The token policy withholds suppressed EOS-only output, so an attempt +marked `empty_visible_output` cannot leak token events before the retry. The +coordinator retries once with autoregressive decode and merges timing, +restored-prefix, speculation, budget-close, and degeneration metadata exactly +as the current wrapper does. -Keep the two current contracts: +A batch retry is allowed only when the executor reports that no visible token +was published and the attempt can be reset without reusing partially mutated +state. Otherwise the engine returns a typed failure. Retiring and re-admitting +a slot is not sufficient proof of rollback. -- `ModelBackend` owns backend lifetime, whole-request generation, snapshots, - and optional capabilities. -- `SeqEngine` is the optional interface for backends that can keep several - live sequences and execute scheduler-selected work together. +### Cache and snapshot boundary -`ModelBackend::seq_engine()` is enough to select the concurrent path. A third -base class named `Engine`, `ServingEngine`, or similar would add indirection -without removing either existing contract. +Prefix policy and physical model state have different owners: -`ModelBackend` is broad, but its optional methods should be extracted only when -there is a concrete caller and more than one useful implementation. Until -then, grouped methods and capability checks are easier to follow than a set of -one-method interfaces. +```text +LuceEngine cache component + -> prefix keys, selection, eviction, request association + -> opaque SnapshotId and backend-neutral metadata -Concurrent failures should eventually reuse `GenerateError` and -`GenerateErrorCode` rather than add another string-based error family. The -exact migration can update `AdmitResult`, `DecodeOutput`, `PrefillOutput`, and -`StepResult` independently while preserving each result's current scope. +model-family snapshot component + -> physical images, release, import, and export + -> GGML buffers and architecture-specific representation +``` -### Architecture runtime and GGML +The model backend supplies a snapshot capability beside its execution +capability: -Owned by model-family directories such as `qwen35/`, `gemma4/`, `laguna/`, -and `deepseek4/`, together with genuinely model-neutral helpers in `common/`. +```cpp +class SnapshotStore { +public: + virtual SnapshotResult release(SnapshotId id) = 0; + virtual SnapshotExport export_snapshot(SnapshotId id) = 0; + virtual SnapshotResult import_snapshot(SnapshotImport image) = 0; +}; +``` -Responsibilities: +`LuceEngine` is the only coordinator of this capability and invokes it at an +engine safe point. The selected execution capability applies snapshots to its +own state. `SingleRequestExecutor::execute()` handles `restore_from` before +prefill. `BatchExecutor::claim()` binds `restore_from` to the claimed slot as +one atomic admission operation. A batch executor that cannot restore returns a +typed unsupported result without claiming a slot. -- Load weights and choose device placement. -- Own model-specific KV, recurrent, and speculative state. -- Build graphs and bind tensor inputs. -- Execute GGML backends and read outputs. -- Implement model-specific prefill, decode, and sampling mechanisms. +`GenerateRequest::capture` asks the selected executor to capture its request +state at the specified position and return an opaque `SnapshotId`. The +executor and the model-family snapshot component share the physical +representation below the engine boundary. Unsupported operations never fall +back to a second generation path. -The engine boundary should expose tokens, sampling configuration, progress, -timings, and typed failures. It should not expose graph tensors, allocator -handles, block-table layouts, or architecture-specific state to the server. +`DiskPrefixCache` currently reads `ModelBackend::SnapshotRef`, including raw +GGML handles. The target replaces that crossing with an owned snapshot +transaction or stream. A failed import must leave no adopted partial state. -Code belongs in `common/` only when at least two model families can use the -same semantics. Similar graph code is not automatically the same component if -the model invariants differ. +Model-cache validity must depend on engine facts such as accepted tokens and +terminal state. API or socket visibility must not enter model execution. +Client-visible conversation memory can remain a response-layer concern when +its validity depends on what reached the client. -### Response state and transport +### Control and daemon boundary -Keep these current names: +Park, unpark, explicit snapshots, compression, bootstrap, and daemon-specific +commands are control operations. They do not become `GenerateRequest` modes or +extra `generate()` functions. -- `SseEmitter` for API event construction and SSE framing. -- `ClientSendBuffer` for non-blocking concurrent socket output. +The daemon adapter parses its line protocol into typed commands. `LuceEngine` +serializes those commands through the engine thread so they cannot race model +execution. Each command declares one lifecycle rule: reject while busy, queue +behind active work, drain active work, or cancel active work. Model-family code +implements the mechanism after the engine establishes the safe state. -Introduce `ResponseState` only when semantic accumulation is physically moved -out of `SseEmitter`. It should own content, reasoning, tool-call parsing, stop -matching, and finish reason. It should not know about SSE syntax, HTTP status, -or sockets. +`DaemonIO`, string commands, and file descriptors stop at the daemon adapter. +Model execution receives `ExecutionHooks` and typed control values. Model +progress enters those hooks and becomes coalesced +`GenerationProgress` events. This preserves live status without passing a +daemon or HTTP observer into model-family code. -The draft terminal-error change in -[PR #689](https://github.com/Luce-Org/lucebox/pull/689) is the first part of -this boundary. It gives the API layer one response error representation while -preserving `GenerateError` as the backend-facing failure. +### Response and transport boundary -### IPC transport +`HttpServer` owns: -Keep `BackendIpcProcess` as the process and transport owner. Keep payload -semantics in role-specific clients. +- HTTP framing, routes, and request validation. +- API aliases, chat rendering, and tokenization. +- Sockets, disconnect detection, CORS, and heartbeats. +- SSE and non-streaming response formatting. +- Translation of typed engine failures into HTTP or stream errors. -An IPC client should either complete one protocol transaction or invalidate -the session. This prevents a partial read or write from being mistaken for the -next response. A small shared helper for marking a process unusable is -preferable to a generic request envelope shared by unrelated IPC modes. +Each client thread consumes its `Generation` and remains the only code that +writes that socket. This removes `SocketHandle`, `ServerJob`, `SseEmitter`, and +`ClientSendBuffer` from engine scheduler state. -## Source organization +Streaming headers may already be committed when generation fails. The HTTP +adapter maps the same typed terminal error to either a normal HTTP error or the +matching stream error based on its own connection state. This keeps the +terminal boundary in +[PR #689](https://github.com/Luce-Org/lucebox/pull/689) intact. -The first structural change should split `http_server.cpp` without changing -the `HttpServer` class or request behavior: +Keep the name `SseEmitter` until semantic accumulation is physically moved out +of it. A later extraction should separate response meaning from SSE framing, +but that change is not required to introduce `LuceEngine`. -| File | Responsibility | -|---|---| -| `http_server.cpp` | Server lifetime, accept loop, socket I/O, job queue, and disconnect monitoring | -| `http_routes.cpp` | Endpoint routing, request validation, chat rendering, tokenization, and `ParsedRequest` construction | -| `generation_worker.cpp` | Classic `worker_loop()`, `process_job()`, prompt preparation, cache lifecycle, and backend call | -| `scheduler.cpp` | Concurrent admission, iteration policy, slot lifecycle, and buffered delivery | -| `sse_emitter.cpp` | Existing semantic stream state and API event formatting until `ResponseState` is extracted | +### IPC boundary -This split changes file ownership, not public interfaces. Existing -`HttpServer` member functions can be defined across the translation units. -Tests and the server target should compile the same source set. +`BackendIpcProcess` remains the owner of process launch, pipes, shared payload +setup, framing, and shutdown. Role-specific clients continue to own their +payload protocols. -After the split, helpers that are used by only one file should move into that -file's anonymous namespace. Helpers shared by classic and concurrent serving -should have narrow typed signatures in a server-local header. +IPC stays below the model execution capability. `LuceEngine` receives typed +execution or control outcomes. It does not receive pipe descriptors or decode +subprocess strings. -## Migration order +## Final dependency direction -### 1. Make terminal results explicit +```text +server_main + -> BackendPlan + -> BackendRuntime + -> LuceEngine + -> request coordinator + -> SingleRequestExecutor + or BatchExecutor + -> model-family resources + -> optional role-specific IPC + -> GGML + -> HttpServer + -> Generation + -> response formatting + -> client socket +``` -Land the API-facing terminal error boundary from PR #689. Both execution paths -must branch on success or failure before success finalization, cache updates, -or HTTP 200 responses. +The model backend and batch executor are not alternative top-level paths. +`BackendRuntime` owns the model resources. `LuceEngine` owns one request +lifecycle and uses the execution capability supplied by those resources. -### 2. Split the server translation unit +## Source organization + +The target directory structure follows owned state: + +| Path | Responsibility | +|---|---| +| `server/src/engine/luce_engine.{h,cpp}` | Request submission, `Generation`, lifecycle, cancellation, and shutdown | +| `server/src/engine/coordinator.{h,cpp}` | Serial and continuous-batch request coordination and shared policy | +| `server/src/engine/generation_channel.h` | Bounded token delivery and one terminal result | +| `server/src/engine/stop_matcher.{h,cpp}` | Incremental text stops and pre-commit token decisions | +| `server/src/engine/prefix_cache.{h,cpp}` | Prefix policy and opaque snapshot association | +| `server/src/common/generation_executor.h` | `SingleRequestExecutor` and `BatchExecutor` contracts | +| `server/src/server/http_server.{h,cpp}` | Listener, client lifetime, routes, and HTTP framing | +| `server/src/server/response_writer.{h,cpp}` | `SseEmitter`, JSON responses, and typed error mapping | +| `server/src/common/daemon_loop.{h,cpp}` | Legacy command parsing and protocol output during migration | +| `server/src//` | Execution capability and physical model state | + +Do not move files only to match this table. Move a file when the state and +decisions listed here move with it. -Create `http_routes.cpp` and `generation_worker.cpp`, then move existing -functions with no behavior or naming changes. Update both the server target and -model-free test target together. +## Migration -This change creates reviewable component boundaries before adding new types. +### 1. Add owned request and generation output -### 3. Extract semantic response state +Make `GenerateRequest` own its optional token vectors. Add `Generation` with a +bounded request queue, bounded token channel, typed terminal result, idempotent +cancellation, immediate rejection, and shutdown tests. -Move content, reasoning, tool-call, stop-sequence, and finish-reason state from -`SseEmitter` into `ResponseState`. +### 2. Put both current loops behind `LuceEngine` -Both streaming and non-streaming builders should consume the same state. -`SseEmitter` should remain responsible for API events and SSE framing. This -keeps the familiar name and removes the current accumulation ambiguity. +Move the job queue and worker ownership out of `HttpServer`. Wrap the current +worker and scheduler without changing model mechanics. Switch HTTP and daemon +generation callers to `LuceEngine::generate()`. -### 4. Share only common request policy +At this point callers have one request operation, but transitional lower +interfaces still exist. -Extract pure helpers for values that classic and concurrent serving must -derive identically, beginning with generation cap, thinking budget, EOS -classification, and terminal error mapping. +### 3. Split scheduler state from transport state -Do not create a shared request executor. Each path should call the helper at -the point where it owns the relevant decision. +Move request IDs, admission, prefill progress, pending tokens, cancellation, +generation limits, retry state, and retirement into engine-owned state. Keep +sockets, `SseEmitter`, heartbeats, and wire formatting in HTTP-owned state. -### 5. Type concurrent engine failures +Delete `ClientSendBuffer` from scheduler state after client threads consume +bounded `Generation` channels directly. -Replace free-form `SeqEngine` failure strings with `GenerateError` values. -Keep admission, per-row, and whole-step results distinct. The scheduler can -then map both classic and concurrent failures through the same response error -function. +### 4. Rename lower execution operations by role -### 6. Narrow `DaemonIO` +Move existing whole-request implementations to +`SingleRequestExecutor::execute()`. Move `SeqEngine` implementations to +`BatchExecutor` only after the scheduler no longer lives in `HttpServer`. -Move the legacy file-descriptor behavior behind the daemon caller. Let HTTP -generation pass only the existing token callback, cancellation probe, and -inference observer responsibilities needed by the backend. +Update HTTP support runs, cache staging, and daemon callers in the same wave. +Delete `ModelBackend::generate()`, `restore_and_generate()`, `generate_impl()`, +and `seq_engine()` when their final callers migrate. -Migrate all callers in the same change before deleting unused fields. Do not -add a compatibility wrapper that preserves both shapes indefinitely. +### 5. Move shared generation policy once -### 7. Harden IPC transactions +Move generation limits, stop decisions, thinking-budget substitution, +speculative empty-output retry, cancellation reasons, and terminal completion +into `LuceEngine`. Preserve the current serial behavior with focused tests. +Add `ExecutionHooks` before moving the serial stop and force-close decisions. -Centralize the rule that a framing, payload, or subprocess failure closes the -affected `BackendIpcProcess`. Keep the error returned to generation typed and -specific to the role-specific client. +### 6. Move cache and control ownership -### 8. Revisit optional backend capabilities +Separate prefix policy from physical snapshots. Replace raw snapshot handles +with `SnapshotStore` and an owned transaction. Parse daemon commands at the +adapter and serialize typed controls through `LuceEngine::control()`. -After call sites are narrow and covered by tests, measure whether snapshots, -compression, parking, or remote draft support benefit from separate capability -interfaces. Extract only the groups that reduce real coupling. +### 7. Delete compatibility structure -## Naming rules +Remove the HTTP execution-mode branch, the old worker and scheduler entry +points, `DaemonIO` from HTTP generation, transport fields in scheduler slots, +and temporary adapters. The migration is incomplete while any caller can +bypass `LuceEngine::generate()` for local generation. -New names should reveal both scope and owner: +## Required invariants -- Use `Request` and `Response` for API or whole-request data. -- Use `Admission`, `Slot`, `Prefill`, `Decode`, and `Step` for scheduler and - concurrent engine data. -- Use `Backend` for model-family lifetime and capability ownership. -- Use `Ipc` plus the remote role for subprocess clients and sessions. -- Use `State` only for data that persists across calls. -- Use `Result` for a completed operation with success or failure. +The migration must preserve these facts from PR #594: -Avoid names that hide the unit of work. In particular, do not introduce -`GenerationPlan`, `ApiRequest`, `ServingRequest`, `OwnedGenerationPlan`, -`GenerationCallbacks`, `RequestSlotState`, or `ApiStreamEncoder` as renames of -the current types. The established names are clearer when their ownership is -documented and their files are smaller. +- One engine thread is the only caller that mutates model execution state. +- Admission claims capacity without running model compute. +- Request and slot identity remain stable until retirement. +- Each decode step covers every selected live row exactly once. +- Prefill work is bounded, fair, and may progress beside active decode. +- A pending token may be substituted before the executor commits it to KV. +- Each selected row returns one explicit outcome. +- A possibly mutating step failure retires the affected cohort before another + step. +- Temporary capacity pressure preserves FIFO order. +- Permanent capacity failure returns a typed terminal result. +- A slow or disconnected consumer cannot block progress for other requests. +- Retirement releases all model-owned request state and is safe after failure. + +The migration must also preserve these complete-request facts: + +- Empty speculative output retries through autoregressive decode exactly once. +- Restored-prefix and timing metadata survive retry. +- Cancellation is checked during prefill and decode. +- Prefix restore and cache staging cannot race live model work. +- Draft residency changes occur only at an engine safe point. +- Streaming and non-streaming calls receive one terminal success or failure. ## Verification -Each migration step should leave both execution paths usable. - -Required checks: - -- Build `dflash_server` for the enabled CUDA or HIP configuration. -- Build and run the model-free server unit tests. -- Run `test_seq_engine_contract` for each concurrent engine. -- Exercise streaming and non-streaming success responses for all supported API - formats. -- Exercise classic and concurrent backend failures and verify non-200 or SSE - error termination. -- Exercise client disconnects during prefill and decode. -- Exercise a slow concurrent reader and verify other slots continue. -- Exercise prefix restore, request-scoped draft residency, and upstream proxy - paths after moving the classic worker. -- Check that model-family and `common/` sources do not construct HTTP JSON or - SSE frames. -- Check that server sources do not depend on architecture-specific graph or KV - structures. - -The file split should produce no wire-format or scheduling changes. A useful -review technique is to compare the moved function bodies before and after the -split and keep functional edits in later commits. +Each implementation step must include checks at the boundary it changes: + +- Model-free tests for `Generation` ownership, cancellation, bounded output, + immediate overload rejection, terminal delivery, shutdown, and a handle + destroyed after its engine. +- Existing `SeqEngine` contract tests against `BatchExecutor` after migration. +- Serial empty-output retry and result-merge tests. +- Streaming and non-streaming success and failure for every API format. +- Disconnects during admission, prefill, decode, and final output. +- A slow client while other batch slots continue. +- Stop and force-close decisions before the next KV commit. +- Cohort retirement after a partially mutating batch failure. +- Prefix restore, disk snapshot failure cleanup, and cache staging. +- Park, unpark, and shutdown while requests are queued or active. +- Coalesced progress delivery without an HTTP or daemon observer below the + engine boundary. +- Static dependency checks that keep HTTP types out of engine and model code, + and GGML types out of HTTP and engine request headers. + +## Lessons from other engines + +The useful comparison is component ownership, not process count. + +- [vLLM](https://github.com/vllm-project/vllm/blob/main/docs/design/arch_overview.md) + keeps HTTP input and output processing separate from an engine core that + owns scheduling, KV management, and worker coordination. Lucebox can use the + ownership split without adopting ZMQ or one worker process per GPU. +- [SGLang](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/managers/io_struct.py) + uses distinct typed messages between API, tokenization, scheduling, and + model execution. Lucebox needs fewer stages, but it benefits from the same + rule that wire objects stop at the API boundary. +- [llama.cpp](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README-dev.md) + separates HTTP routes, task queues, inference slots, task results, and + response readers. Lucebox differs because several model families still own + complete-request execution, so the lower capability boundary must admit + both honest execution shapes. ## Alternatives rejected -### One universal engine interface +### Keep both paths under a facade + +`LuceEngine::generate()` calling another request-level +`ModelBackend::generate()` would add a pass-through layer. It would hide the +name conflict without moving retry, cache, cancellation, or lifecycle +ownership. + +### Force every backend into `admit()`, `step()`, and `retire()` + +A single step protocol looks uniform, but it makes every complete-request +backend expose resumable state only to satisfy the abstraction. The target +keeps one request interface and two honest model execution capabilities. -`ModelBackend::generate()` completes a request. `SeqEngine::step()` advances a -selected batch by one iteration. A common interface would either expose both -lifecycle models or reduce one to callbacks around the other. Neither result -simplifies the caller. +### Rename `ModelBackend` to `LuceEngine` -### An internal event bus +The current type owns daemon commands, resource controls, complete-request +execution, snapshots, and optional batch execution. Renaming it would not move +the HTTP scheduler or split its responsibilities. -The request path is direct and performance-sensitive. Typed calls and results -make ownership and failure propagation visible. An event bus would obscure -ordering, lifetime, and cancellation without adding a current deployment -benefit. +### Keep scheduling in `HttpServer` -### IPC between HTTP and every backend +This would preserve duplicate request state, cancellation, shutdown, and +completion behavior. HTTP would still depend on model execution mode. -vLLM and SGLang use process boundaries to support their deployment and -scheduling designs. Lucebox currently has a direct in-process local path plus -targeted subprocesses for heterogeneous execution. Mandatory IPC would add -serialization and lifecycle work without resolving the current source -organization problem. +### Put sockets in `LuceEngine` -### A broad rename pass +Socket ownership would couple scheduling to HTTP and retain separate blocking +and non-blocking send paths. The existing client thread can consume a bounded +`Generation` without exposing transport to the engine. -The current names mostly identify real units: parsed request, server job, -backend request and result, scheduler slot, engine step, SSE emitter, send -buffer, and IPC process. Moving responsibilities first gives any future rename -concrete evidence and keeps review focused. +### Add mandatory IPC -### Splitting all `ModelBackend` capabilities now +An in-process boundary is sufficient. IPC adds serialization and subprocess +lifecycle but does not improve source ownership. -Small interfaces are useful when they let callers depend on less. Creating one -interface per optional method before narrowing callers would increase the -number of types without changing ownership. +## Synthesis decision -### JSON below the server layer +Two target shapes were compared. -JSON is part of the external API contract. Passing it into backend or engine -code couples model execution to current endpoints and makes non-HTTP callers -harder to support. +The selected design keeps `SingleRequestExecutor` and `BatchExecutor` as +separate model capabilities under one `LuceEngine`. This matches the existing +model state machines and avoids making serial backends imitate continuous +batching. -## Related designs +The rejected design used one scheduler and one `admit/step/retire` contract for +every model, with serial represented as capacity one. Its final diagram was +smaller, but the migration required every model family to become resumable and +moved model differences into a wider shared step protocol. -These projects support the ownership direction in this document, but Lucebox -should not copy their process layouts. +The selected design takes four details from that alternative: an explicit +split of `SchedSlot` state, a bounded generation channel consumed by client +threads, the PR #594 invariant checklist, and a migration that first moves +both loops behind `LuceEngine` before renaming lower contracts. -- [vLLM architecture overview](https://github.com/vllm-project/vllm/blob/main/docs/design/arch_overview.md) - separates API input and output processing from an engine core that owns - scheduling, KV cache management, and worker coordination. Its API and engine - core communicate across ZMQ because vLLM chooses a multi-process topology. -- [SGLang manager I/O structures](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/managers/io_struct.py) - distinguish external generation input from tokenized scheduler input. The - useful lesson is the typed boundary between stages, not the number of - manager processes. -- [llama.cpp server developer guide](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README-dev.md) - separates HTTP context, routes, tasks, queues, inference slots, task results, - and response readers. It also keeps JSON formatting and chat templates in - the HTTP layer and passes native C++ types to inference slots. +## Tradeoffs -Lucebox differs in two important ways. It supports a classic whole-request -backend path beside continuous batching, and it uses auxiliary IPC inside -specific heterogeneous execution features. The target structure must make -those differences explicit. +- We accept two model-execution capabilities in exchange for one honest + request lifecycle and no fake universal step method. +- We accept one engine thread and per-request channels in serial mode in + exchange for the same cancellation, control, output, and shutdown behavior + for every model. +- We accept owned copies of optional request token vectors in exchange for + safe asynchronous lifetime. +- We cancel a request whose output channel remains full so one client cannot + stop shared model progress. +- We move backend ownership into `LuceEngine` so model controls cannot bypass + the engine's single-writer rule. ## Open questions -- Should `ResponseState` be extracted immediately after the file split, or - should terminal error parity land in both paths first? -- Which serving policies must reach parity before another model family exposes - `SeqEngine`? -- Can the daemon protocol stop sharing `DaemonIO` with the HTTP server without - affecting external scripts? -- Should all concurrent failures use `GenerateError`, or is a smaller subset - sufficient for `SeqEngine`? -- When this design is implemented, should `server/docs/ARCHITECTURE.md` become - an operational overview that links here for component ownership? +- Which park and compression controls reject, queue, drain, or cancel active + requests? +- Should snapshot export materialize one owned image or stream chunks to the + disk cache? +- Which cache policies depend on client-visible output and therefore must stay + above model-state caching? +- Can every batched speculative implementation prove safe retry after an empty + output, or should some return a typed non-retryable failure? +- Should `server/docs/ARCHITECTURE.md` become the short operational overview + that links to this target design? + +## Next implementation step + +Add an owned `GenerateRequest` and a model-free `Generation` channel test. The +test must prove bounded publication, one terminal result, idempotent +cancellation, overload behavior, and shutdown wakeup before model code moves.