Conversation
stikves
force-pushed
the
sukru/vlm-server-image-input
branch
2 times, most recently
from
September 23, 2026 01:09
b033b63 to
111c4a0
Compare
Adds a JSONL replay path so the real generation stack can be exercised without binding a socket (sandboxed CI, locked-down machines), and consolidates serving and replay onto one code path. Extract runChatCompletion from the non-streaming HTTP handler: it returns the assembled ChatCompletionResponse plus per-request instrumentation (ChatCompletionOutcome). Extract the streaming core the same way: prepareStreaming does preflight/guards, runStreamingLoop drives the incremental think/tool parse loop and delivers each delta to an emit closure. The SSE handler and --replay both call these, so serve and replay share one generation core per mode. Shared format in CoreAILMCommon/ReplayTypes.swift: ReplayRequest wraps the exact ChatCompletionRequest plus a session id and arrival timestamp; ReplayResult is a parseable JSONL line with content, reasoning_content, tool_calls, finish_reason, system_fingerprint, token counts, prefix_reuse_tokens, ttft/total/decode timings, and a streamed flag. ReplayIO parses JSONL (skips blanks/comments, orders by timestamp, stable on ties) and serializes results. Replay honors the request's stream field: stream:false runs runChatCompletion, stream:true drives runStreamingLoop and folds the emitted deltas back into the result via ReplayStreamAggregator, so streaming requests exercise the incremental parse path. Requests run in arrival order; sequential by construction (single-active engine), matching the real server. llm-server --replay <in.jsonl> [--replay-output <out.jsonl>] loads and warms the model exactly as serve does, then runs requests and writes results JSONL, never binding the port. Tests: ReplayTypesTests cover request decode, JSONL parse/order/skip, result encoding (including reasoning_content/tool_calls/streamed), the isStreaming decision, and ReplayStreamAggregator delta folding. Runner execution over a live engine is not unit-tested (needs model assets); the pure pieces it composes are.
…rate CRLF in JSONL
Replace ReplayResult's flat content/reasoning_content/tool_calls/finish_reason with a choices array reusing ChatCompletionResponse.Choice, so results mirror the response type and handle multiple messages. The streaming aggregator now folds deltas into a single assistant Choice.
Add a vision-language path to llm-server so a kind=vlm bundle (e.g. muse-glimmer-30b) can be served over the OpenAI-compatible chat endpoint with image input. - LLMServerMain builds CoreAISequentialVLMEngine directly for kind=vlm bundles (EngineFactory only produces text engines) and skips the text-shaped warmup. - ServerState carries the VisionConfig and exposes the engine as a MultimodalInferenceEngine. - runVLMCompletion decodes one image (base64 data URL or local path), runs the vision encoder, expands the image placeholder in the prompt, and generates through the multimodal engine. It is routed from the shared runChatCompletion core, so both the HTTP non-streaming handler and --replay exercise it. - Output reuses the existing think-tag / tool-call parsing, so reasoning models return a clean final answer with the analysis separated into reasoning_content. - /v1/models reports supports_vision for VLM bundles. Validated with --replay on the muse-glimmer-30b VLM bundle: an image referenced by path is transcribed accurately, with the harmony analysis stripped from the answer. Stacks on the seed (apple#265) and --replay (apple#266) branches since the VLM core lives in the shared runChatCompletion; the diff includes those commits until they merge. Deferred: multiple images, video, http(s) URL fetch, guided/constrained decoding on the multimodal path, VLM prefix reuse, and the HTTP streaming VLM path.
Add a --file-access flag (off by default, subdirs) controlling whether image_url content parts may read local files. off rejects all bare paths and file:// URLs; subdirs allows only canonicalized real paths inside the server working directory, rejecting .. traversal, absolute-outside paths, and symlink escapes. Thread the policy through ServerConfig into decodeImage. Reject stream:true requests carrying an image with a 400 instead of silently answering on the text path without the image.
Replace the inline kind == .vlm CoreAISequentialVLMEngine build in LLMServerMain with the shared EngineFactory bundle entry point, which routes VLM bundles to the sequential VLM engine and text bundles to the text engines. Chunking overrides pass through engineOptions; the VLM warmup skip is unchanged.
stikves
force-pushed
the
sukru/vlm-server-image-input
branch
from
September 23, 2026 20:17
111c4a0 to
b957b43
Compare
stikves
marked this pull request as ready for review
September 23, 2026 20:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a vision-language path to
llm-serverso akind=vlmbundle (e.g. muse-glimmer-30b) can be served over the OpenAI-compatible chat endpoint with image input.What it does
LLMServerMainbuildsCoreAISequentialVLMEnginedirectly forkind=vlmbundles (EngineFactoryonly produces text engines) and skips the text-shaped warmup.ServerStatecarries theVisionConfigand exposes the engine as aMultimodalInferenceEngine.runVLMCompletiondecodes one image (base64 data URL or local path), runs the vision encoder, expands the image placeholder in the prompt, and generates through the multimodal engine. It is routed from the sharedrunChatCompletioncore, so the HTTP non-streaming handler and--replayboth exercise it.reasoning_content./v1/modelsreportssupports_visionfor VLM bundles.File access & streaming
--file-accessflag controls whether animage_urlmay read local files. Defaultoffrejects all bare paths andfile://URLs (onlydata:andhttp(s)://are accepted);subdirsallows local files only when the canonicalized real path resolves inside the server working directory subtree, rejecting..traversal, absolute-outside paths, and symlink escapes. This closes an unrestricted local-file-read hole (a client could previously pass/etc/hosts).stream:truerequest that carries an image now returns a 400 ("Streaming is not supported for image (VLM) requests") instead of silently falling through to the text path and answering without the image.Testing
LLMServerTeststarget:decodeImageon a data URL and on a remote URL,buildPromptTokensplaceholder expansion, and file-access policy coverage (offrejects/etc/hostsandfile:///etc/hosts;subdirsallows inside-CWD paths and rejects traversal, absolute-outside, base-dir, and symlink-escape).--replayon the muse-glimmer-30b VLM bundle: an image referenced by path is transcribed accurately, with the harmony analysis stripped from the answer.Stacking
Stacks on the seed (#265) and
--replay(#266) branches since the VLM core lives in the sharedrunChatCompletion; the diff includes those commits until they merge.Deferred
Multiple images, video, http(s) URL fetch, guided/constrained decoding on the multimodal path, VLM prefix reuse, and the HTTP streaming VLM path (currently rejected with a 400).