Conversation
stikves
added a commit
to stikves/coreai-models
that referenced
this pull request
Sep 22, 2026
Add a vision-language path to llm-server so a kind=vlm bundle (e.g. muse-glimmer-30b) can be served over the OpenAI-compatible chat endpoint with image input. - LLMServerMain builds CoreAISequentialVLMEngine directly for kind=vlm bundles (EngineFactory only produces text engines) and skips the text-shaped warmup. - ServerState carries the VisionConfig and exposes the engine as a MultimodalInferenceEngine. - runVLMCompletion decodes one image (base64 data URL or local path), runs the vision encoder, expands the image placeholder in the prompt, and generates through the multimodal engine. It is routed from the shared runChatCompletion core, so both the HTTP non-streaming handler and --replay exercise it. - Output reuses the existing think-tag / tool-call parsing, so reasoning models return a clean final answer with the analysis separated into reasoning_content. - /v1/models reports supports_vision for VLM bundles. Validated with --replay on the muse-glimmer-30b VLM bundle: an image referenced by path is transcribed accurately, with the harmony analysis stripped from the answer. Stacks on the seed (apple#265) and --replay (apple#266) branches since the VLM core lives in the shared runChatCompletion; the diff includes those commits until they merge. Deferred: multiple images, video, http(s) URL fetch, guided/constrained decoding on the multimodal path, VLM prefix reuse, and the HTTP streaming VLM path.
Adds a JSONL replay path so the real generation stack can be exercised without binding a socket (sandboxed CI, locked-down machines), and consolidates serving and replay onto one code path. Extract runChatCompletion from the non-streaming HTTP handler: it returns the assembled ChatCompletionResponse plus per-request instrumentation (ChatCompletionOutcome). Extract the streaming core the same way: prepareStreaming does preflight/guards, runStreamingLoop drives the incremental think/tool parse loop and delivers each delta to an emit closure. The SSE handler and --replay both call these, so serve and replay share one generation core per mode. Shared format in CoreAILMCommon/ReplayTypes.swift: ReplayRequest wraps the exact ChatCompletionRequest plus a session id and arrival timestamp; ReplayResult is a parseable JSONL line with content, reasoning_content, tool_calls, finish_reason, system_fingerprint, token counts, prefix_reuse_tokens, ttft/total/decode timings, and a streamed flag. ReplayIO parses JSONL (skips blanks/comments, orders by timestamp, stable on ties) and serializes results. Replay honors the request's stream field: stream:false runs runChatCompletion, stream:true drives runStreamingLoop and folds the emitted deltas back into the result via ReplayStreamAggregator, so streaming requests exercise the incremental parse path. Requests run in arrival order; sequential by construction (single-active engine), matching the real server. llm-server --replay <in.jsonl> [--replay-output <out.jsonl>] loads and warms the model exactly as serve does, then runs requests and writes results JSONL, never binding the port. Tests: ReplayTypesTests cover request decode, JSONL parse/order/skip, result encoding (including reasoning_content/tool_calls/streamed), the isStreaming decision, and ReplayStreamAggregator delta folding. Runner execution over a live engine is not unit-tested (needs model assets); the pure pieces it composes are.
stikves
force-pushed
the
sukru/server-replay-mode
branch
from
September 22, 2026 14:39
b34f8b8 to
f20684b
Compare
…rate CRLF in JSONL
stikves
marked this pull request as ready for review
September 22, 2026 19:34
stikves
requested review from
alejandro-isaza,
carinapeng,
gokulkrishna98,
guru-desh,
kevchengcodes,
srjoglekar246 and
tjia1818
September 22, 2026 19:35
| public let id: String? | ||
| public let session: String? | ||
| public let t: Double? | ||
| public let content: String? |
Contributor
There was a problem hiding this comment.
why not keep the ChatCompletionRequest format, where we'd have a [ChatMessage] instead of pulling out the content and toolCalls? Would this work if there are multiple messages for one replay result?
stikves
added a commit
to stikves/coreai-models
that referenced
this pull request
Sep 23, 2026
Add a vision-language path to llm-server so a kind=vlm bundle (e.g. muse-glimmer-30b) can be served over the OpenAI-compatible chat endpoint with image input. - LLMServerMain builds CoreAISequentialVLMEngine directly for kind=vlm bundles (EngineFactory only produces text engines) and skips the text-shaped warmup. - ServerState carries the VisionConfig and exposes the engine as a MultimodalInferenceEngine. - runVLMCompletion decodes one image (base64 data URL or local path), runs the vision encoder, expands the image placeholder in the prompt, and generates through the multimodal engine. It is routed from the shared runChatCompletion core, so both the HTTP non-streaming handler and --replay exercise it. - Output reuses the existing think-tag / tool-call parsing, so reasoning models return a clean final answer with the analysis separated into reasoning_content. - /v1/models reports supports_vision for VLM bundles. Validated with --replay on the muse-glimmer-30b VLM bundle: an image referenced by path is transcribed accurately, with the harmony analysis stripped from the answer. Stacks on the seed (apple#265) and --replay (apple#266) branches since the VLM core lives in the shared runChatCompletion; the diff includes those commits until they merge. Deferred: multiple images, video, http(s) URL fetch, guided/constrained decoding on the multimodal path, VLM prefix reuse, and the HTTP streaming VLM path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
--replaymode to the LLM server: it reads a JSONL file of chat requests, runs them through the same request-handling core as the HTTP path, and writes a JSONL of results. This enables offline, functional testing of the server without opening a port.What this adds
ReplayTypes(shared request/result JSONL format) inCoreAILMCommon.ReplayRunnerand a--replay <input> --replay-output <output>flag on the server.Testing
ReplayTypesTestscovers the JSONL round-trip.swift format lint --strictpass.