Switch openCodeMlx default model to Qwen3.8-27B (mlx-vlm)

[perNyfelt]

Summary

• Switches demo/openCodeMlx's default MLX_MODEL to mlx-community/Qwen3.8-27B-4bit, a vision-language checkpoint that plain mlx_lm.server can't load (unsupported architecture).
• Adds detect_model_backend(), which inspects the downloaded model's config.json for a vision_config key to auto-pick mlx_vlm vs mlx_lm as the serving backend — so future MLX_MODEL swaps between text-only and vision-capable checkpoints keep working with the existing "swap one line" UX, no second env var to keep in sync.
• Adds ensure_mlx_vlm_current() (mirrors ensure_mlx_lm_current's not-installed→hard-error / already-installed→warn-and-continue pattern), only invoked when the detected backend needs it.
• start_mlx_server_instance() now branches between mlx_lm.server (--prompt-cache-bytes) and python3 -m mlx_vlm.server (--max-kv-size, --trust-remote-code) based on the detected backend. The small background model (Qwen2.5-Coder-1.5B-Instruct-4bit) is explicitly pinned to mlx_lm and unaffected.

Known tradeoff (accepted): mlx-vlm's prompt-prefix caching is limited, so opencode's system prompt may get reprocessed every turn instead of cached like the mlx_lm.server path — a documented upstream opencode+mlx-vlm behavior. This is a real latency cost for the coding-agent workflow, not fixed here.

Also in this PR: LCA model-config consolidation, GPU tuning, and /benchmark

Unrelated to the demo/openCodeMlx work above, but landed on this branch:

• Consolidates scattered Ollama model config (names, context sizes, custom model recipes) into src/main/bin/lca as the single canonical source. models.sh now derives its model list from lca via a targeted grep+eval instead of duplicating it by hand, and application.properties picks the same values up through LCA_* env vars that lca exports at launch (with literal fallbacks for ./run.sh/mvnw test/IDE runs that bypass lca).
• Switches the chat/review models from the mlx variant to llama.cpp, raises the review model's context window to 128k, and applies num_gpu/num_batch Modelfile tuning to the main model for full GPU offload on Apple Silicon.
• Fixes rebuild_custom_model_if_changed() in lca to fingerprint the full desired Modelfile recipe (base model id + context size + extra params) instead of just the base model's id — previously a context- or parameter-only change went undetected, so a stale custom model kept running silently after a config edit.
• Adds a /benchmark command (ModelRegistry.benchmark(), calling Ollama's /api/generate directly since Spring AI/Embabel doesn't surface Ollama's raw timing fields) to measure real prompt-eval/generation tokens-per-second for a given model — optionally against a real file's contents via --prompt-file, so future model or config changes can be measured instead of guessed at.
• Fixes two intent-routing bugs surfaced while wiring /benchmark in:
  • Every typed line (even a literal /health) was being routed through the LLM-based intent classifier before dispatch; a command absent from its hand-curated description list could get reinterpreted as something else entirely (observed: /benchmark --model x was misrouted to /run, which then tried to execute benchmark as a literal shell binary). Literal, well-formed slash commands CommandExecutor already recognizes now dispatch directly, skipping the classifier.
  • CommandExecutor's command-parsing regex used \w+, which doesn't match hyphens, so /git-apply was silently truncated to /git and could never dispatch by name; /git-push was additionally never wired into the dispatch switch at all despite having a working ShellCommands.gitPush(...) handler. Both are fixed.

Follow-up fixes from code review

• demo/openCodeMlx:
  • ensure_mlx_vlm_current() now carries the same transformers>=5.7,<5.13 pin as ensure_mlx_lm_current — both packages share one venv, so an unconstrained pip install mlx-vlm could upgrade transformers past the version mlx-lm's AutoTokenizer.register call tolerates, silently breaking mlx_lm.server (still used unconditionally for the small background model regardless of the main model's backend).
  • MLX_DRAFT_MODEL is no longer downloaded when the main model isn't served via mlx_vlm (speculative decoding is only wired up there); a mismatched quant level between MLX_DRAFT_MODEL and MLX_MODEL is now rejected up front with a clear error instead of silently serving a mismatched pair.
  • The mlx_vlm path's --max-kv-size OOM guard is now overridable via MLX_VLM_MAX_KV_SIZE, with the comment now honestly stating that reusing MLX_CONTEXT_LIMIT as a stand-in is an unverified guess, not a measured bound.
  • cleanup()'s shutdown message now names the actually-detected backend (mlx_vlm.server (main) vs mlx_lm.server (main)) instead of always saying mlx_lm.server.
• REPL command dispatch: wires /model, /context, /version, /stage, /revert, /commit-suggest, and /applyBlocks into CommandExecutor's dispatch switch and KNOWN_COMMANDS set — these already worked as ShellCommands methods (and were documented/allow-listed) but fell through to "Unknown command" when typed directly, the same class of gap /benchmark hit. While wiring their --confirm/--dry-run/--secret-scan flags, fixed the same Groovy truthiness trap already fixed once for /benchmark's --max-tokens 0: parseBoolean(x) ?: true was silently turning an explicit --flag false back into true.

Second round of review fixes

• demo/openCodeMlx:
  • _quant_suffix()'s regex used a bare [[ ... ]] && echo ..., so a non-matching model id returned exit status 1; assigned via a bare x=$(_quant_suffix ...) under main()'s set -e, that silently killed the whole script right after the multi-GB model download with no error message. Now always returns 0.
  • _quant_suffix() also now matches DWQ/AWQ-style qualified ids (...-4bit-DWQ, ...-8bit_dwq), which the original ([0-9]+bit)$ missed, letting the draft/main quant-mismatch guard silently no-op for them.
  • Reverted an unrelated, unsubstantiated LOCAL_MODEL_PROMPT rewrite that had been bundled into the drafter-perf commit — it removed exploration-budget/response-length/no-subagents rules with no evidence backing the change, on the exact 4-bit model this project's own history already flags as unable to follow prompt-level fixes reliably. The drafter perf fix itself (disabling speculative decoding by default) is unaffected.
• REPL command dispatch: closed the last three parseBoolean(x) ?: true / parseInt(x) ?: N truthiness-trap sites in CommandExecutor — /review --log-review false, /search --headless false, /run --confirm false (all silently forced back to their default), and /context --padding 0 (silently reset to 2).
• Testing: added DemoTestScriptsSpec so every demo/test_*.sh now runs as part of ./mvnw test — the _quant_suffix set -e regression above had a passing hand-written test the whole time, but nothing executed it automatically.

Third round of review fixes

• Demo test determinism: test_updates_mlxvlm.sh, test_updates_mlxlm.sh and test_updates_opencode.sh's "not installed" scenarios all silently depended on a tool's absence from the real system PATH — a property of the machine running them, not of the code under test. Verified by reproducing the exact flip (stubbing python3 to succeed made test_updates_mlxvlm.sh assert 1, get 0). Harmless while nobody ran these by hand; load-bearing now that they're wired into ./mvnw test. Fixed by stubbing absence explicitly (mlx_lm.server/python3) where possible, and by adding a real-environment guard for test_updates_opencode.sh (opencode itself is what's being checked, so its "not installed" scenarios now skip with a clear message if a real opencode is actually resolvable on the test's PATH, rather than silently running a real opencode upgrade).
• DemoTestScriptsSpec hang risk: process.waitFor() had no timeout, and one of these scripts can reach a real network call on a misconfigured machine. Output is now collected on a background thread concurrently with a bounded waitFor(120s) plus destroyForcibly() on expiry — a timeout on waitFor() alone wouldn't have been enough, since the prior blocking process.inputStream.text read would hang before ever reaching it.
• DemoTestScriptsSpec discovery scope: pinned to git ls-files demo output so an untracked scratch script can't silently join the build (verified by dropping in an untracked failing script and confirming it's excluded).

Fourth round of review fixes

• DemoTestScriptsSpec's own git subprocess had the same bug it had just fixed: gitTrackedDemoTestScripts()'s git ls-files call blocked on process.inputStream.text before its bounded waitFor(30s) — a hung git (index lock, credential prompt) would hang there regardless of the timeout. Extracted a shared runBounded() helper (concurrent reader thread + bounded wait + forced kill) used by both process invocations, so there's one place this gets fixed instead of two to keep in sync. Also switched the shared output buffer from StringBuilder to StringBuffer — reader.join(5000) only gives a happens-before guarantee when the reader actually finishes in time, and on the timeout path it might not have.
• Silent zero-coverage risk: an empty git-tracked-names result would filter every script out. Verified first that Spock actually already throws "Data provider has no data" on an empty where: list (doesn't silently pass), then added an explicit assertion anyway for a much clearer diagnosis pointing at the tracked-files lookup specifically.
• The opencode "not installed" skip was invisible to the build: DemoTestScriptsSpec only asserts exit 0 and no FAIL:, and the guard's skip path satisfied both — so on a machine with a real opencode on REAL_PATH, those three scenarios would quietly vanish from coverage while the build stayed green. Narrowed the PATH those scenarios use from /usr/bin:/bin:/usr/local/bin to /usr/bin:/bin (verified mkdir/cat/chmod/bash/sh all still resolve there) — the official opencode installer already targets ~/.opencode/bin, off REAL_PATH entirely, so this converts an environment-dependent skip that could fire on a normal Homebrew setup into one that essentially never fires, while keeping the guard itself as a defense-in-depth fallback.

Test plan

• bash -n demo/openCodeMlx syntax check
• demo/test_updates_opencode.sh — 8/8 passing (no regression)
• demo/test_updates_mlxvlm.sh (new) — 9/9 passing, asserts the shared transformers pin survives both install and update paths
• demo/test_quant_suffix.sh (new) — 5/5 passing
• Ran the real script end-to-end on Apple Silicon: full ~27GB ModelScope download succeeded, backend auto-detected as mlx_vlm, mlx-vlm auto-installed
• Confirmed real mlx_vlm.server --help output matches the flags used in the script
• Started the actual server against the real downloaded model; /v1/models responds (what the script polls)
• Sent a real tool-calling chat-completions request and got back a correct tool_calls response, confirming the agentic use case works end-to-end
• ./mvnw test — full suite green (1180 tests) across all four review rounds, including demo/test_*.sh now running as bounded, deterministic JVM tests via DemoTestScriptsSpec with no unbounded subprocess waits anywhere in that spec
• Manually verified /benchmark --model qwen3.8-review:latest --prompt-file <real source file> reports sensible tokens/sec against the tuned model

🤖 Generated with Claude Code

[perNyfelt]

Reviewed PR #54 ✅

Verified against the actual environment:

• \ bash -n demo/openCodeMlx — syntax OK
• Ran all demo test suites: test_updates_opencode 8/8, test_updates_model 10/10, test_updates_mlxlm 8/8, test_main_update 6/6, test_main_guard 4/4 — no regressions
• Confirmed real python3 -m mlx_vlm.server --help exposes exactly the flags the new branch passes: --model, --host, --port, --prefill-step-size, --max-kv-size, --trust-remote-code
• Small model path correctly pinned to mlx_lm with --prompt-cache-bytes unchanged

Review notes (non-blocking):

1. Double opt-in is clumsy: the new default makes the active line MLX_MODEL="${MLX_MODEL:-mlx-community/Qwen3.8-27B-8bit} and the very next line a commented-out #MLX_MODEL="${MLX_MODEL:-...Coder-30B...}". The documented pattern is one active MLX_MODEL=line (the other model lines below are already bare#MLX_...comments). The new Qwen3 line should be the active one and the old default commented out as a plain#MLX_MODEL\u003d...— as written, this file has twoMLX_MODEL=` assignments at lines 13–14.
2. ensure_mlx_vlm_current pip line: pip install --upgrade pip mlx-vlm — consider matching the pip install ... spelling used two lines above in ensure_mlx_lm_current for consistency (cosmetic).
3. The name is ensure_mlx_vlm_current but the body mirrors ensure_mlx_lm_current; fine, just noting it's deliberately the same install-or-update shape.
4. detect_model_backend falling back to mlx_lm on any read/parse failure is a sensible fail-safe for existing text-only models — agreed with the design.
5. --max-kv-size reuses `` (128k tokens) as the token-count analogue of the byte cap — a reasonable mapping given mlx-vlm exposes no byte-based knob.

[perNyfelt]

Follow-up: the 8-bit model was too slow in practice, so this branch now switches to mlx-community/Qwen3.8-27B-4bit with real speculative decoding via the mlx-community/Qwen3.8-27B-MTP-4bit draft head (--draft-model/--draft-kind mtp).

Also fixed model downloads to bind to a specific network interface (MLX_DOWNLOAD_INTERFACE, default en7) — GlobalProtect VPN was resetting every connection to huggingface.co outright.

Live-verified on real hardware:

• Both models download successfully via en7 (bypassing the VPN block)
• mlx_vlm.server starts with the draft model attached, /v1/models responds
• Tool-calling smoke test works correctly with speculative decoding active
• Decode throughput: 17.8 tok/s (195-token generation, ~77% draft-token acceptance) vs the previous ~12 tok/s 8-bit baseline