From ec9b62938e2ec1ac10430fa5b240a233977d67ac Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 08:59:27 +0000 Subject: [PATCH 01/76] chore: upgrade llama.cpp from b10456 to b10470 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 3abaabe1d..4b2c44125 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10456** +Current llama.cpp pinned version: **b10470** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10456 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10470 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10456`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10470`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10456`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10470`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 6b7fffb76..5e8bb7dba 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10456](https://img.shields.io/badge/llama.cpp-%23b10456-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10456) +[![llama.cpp b10470](https://img.shields.io/badge/llama.cpp-%23b10470-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10470) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 718a72d20..d98d320f7 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -619,3 +619,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10454–b10455 | `ggml/src/ggml-sycl/{ggml-sycl.cpp,opt-step.cpp,opt-step.hpp}` (**SYCL `OPT_STEP_ADAMW` / `OPT_STEP_SGD`, #25268**), `docs/ops/SYCL.csv` + `docs/ops.md` + `examples/sycl/update-ops-doc.sh` (**regenerated op-support table**) | **No project-source change.** A single commit inside the SYCL backend, so only the `sycl-{fp16,fp32}-linux-x86-64` / `sycl-windows-x86-64` classifiers' compiled sources change. Step forced above the 100 KiB threshold (3.2 MiB) — but ~3.2 MiB of that is the regenerated `docs/ops/SYCL.csv` alone (20 646 lines rewritten); the reviewable code is ~150 lines, and upstream has no tag between b10454 and b10455 anyway. | | b10455–b10456 | `ggml/src/ggml-sycl/cpy.cpp` (**thread/block count fixed in the quantized cpy kernel launches, #27160**) | **No project-source change (final step).** A single one-commit fix inside the SYCL backend; no shared header touched, so only the `sycl-{fp16,fp32}-linux-x86-64` / `sycl-windows-x86-64` classifiers' compiled sources change. | | b10423–b10456 | upstream verification (sandbox, final target) | All **6** patches re-verified against a clean b10456 checkout: sequential `git apply` (filename order, mirroring the `PATCH_COMMAND` applier) succeeded at **every** intermediate tag of the walk, and the fail-loud `PATCH_COMMAND` ran clean on a fresh `cmake -B build` at the target — no patch needed refreshing across the whole range. **Full local verification (mandatory for the final target):** fresh configure (resolved `ggml commit f275595dd` = b10456 HEAD; all six patch markers confirmed present in the fetched tree) + full `cmake --build` (`libjllama.so` + `jllama_test` compile and link, `-O3`, no undefined references) + `ctest` **485/485 passing**. **API surface:** of the 22 upstream headers project source `#include`s directly, only three changed across the whole range — `tools/server/server-queue.h` (breaking: `on_new_task`'s callback type became `std::function`, plus the new `yield_to_queue()`; inert here, upstream's own `server-context.cpp` is the sole registrant), `ggml/include/ggml.h` (`ggml_ssm_scan()` gained a trailing `int64_t K`; called only inside llama.cpp's own TUs) and `tools/mtmd/mtmd-helper-common.h` (`size_t` widening, no signature change). `mtmd.h`, `mtmd-helper.h`, `common.h`, `chat.h`, `llama.h`, `arg.h` and every `tools/server/*.h` the project includes are byte-identical b10423→b10456; no files were added or removed under `tools/server`/`tools/mtmd`/`vendor` and no upstream `CMakeLists.txt` changed, so the b10154 `server-mcp.cpp` missing-`target_sources` failure class does not recur. Walk of 8 steps (b10423→b10430→b10436→b10441→b10447→b10448→b10454→b10455→b10456) on branch `claude/update-b10456-8a7j58`. Two steps exceeded the 100 KiB chunking threshold unavoidably — upstream has **no** intermediate tags there (b10449–b10453 and b10456 aside, nothing sits between b10447/b10448 or b10454/b10455) — and the 3.2 MiB b10454→b10455 figure is almost entirely the regenerated `docs/ops/SYCL.csv` (20 646 lines rewritten), leaving ~150 lines of reviewable code. | +| b10456–b10470 | `tools/server/server-tools.cpp` (**built-in `get_datetime` tool removed, −57 lines, plus a "keep this array minimal" contributor note**) + `common/arg.cpp` & `tools/server/README.md` (**matching one-line `--tools` help-text edit**), `common/chat-diff-analyzer.cpp` (**additive Bailing-V3 template-patch lambda**), `common/speculative.cpp` (**`dflash.sample_from_anchor` GGUF knob in the file-local DFlash/DSpark impl**), `tools/mtmd/mtmd-image.cpp` (**two `GGML_ASSERT` bounds checks in the granite preprocessor**), `CMakeLists.txt` + `ggml/CMakeLists.txt` (**`*_VERSION_PATCH` 0→1**) | **No project-source change.** The only change in a file the project compiles is `server-tools.cpp`, but both the deleted `struct server_tool_get_datetime` and `build_tools()` are file-internal (declared in no header) and `server-tools.cpp` is deliberately not linked into `jllama_test` — greps for `get_datetime`/`server_tool`/`build_tools`/`find_tool`/`"/tools"` over `src/main/cpp` + `src/test/cpp` + the Java tree return zero hits; the sole effect is that NativeServer full mode's `/tools` advertises one fewer built-in (attach mode already 403s that route via patch 0007). No public header changed: `common/speculative.h`, `common/chat.h`, `tools/mtmd/mtmd{,-helper}.h` and every `tools/server/*.h` are untouched, so the speculative/chat-analyzer/mtmd edits stay inside upstream-compiled TUs. **Patch context intact:** the `arg.cpp` edit sits at line ~3362, >2000 lines from patch `0001`'s hunks (`@@ -1201` / `@@ -1242`), and `server.cpp`/`server-context.cpp`/`server-models.cpp` are not in the range at all. **Build wiring:** the `tools/server/` file set is byte-identical b10456→b10470 (`git ls-tree` diff empty) and `vendor/`+`include/` are untouched, so `target_sources` needs no edit and the b10154 `server-mcp.cpp` missing-source link-failure class does not recur. | +| b10456–b10470 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10470 checkout: sequential `git apply --check` + `git apply` in filename order (mirroring the fail-loud `PATCH_COMMAND` applier) succeeded with zero fuzz. **Compile-verified per step:** a fresh `cmake -B build` against the patched b10470 tree configured clean, and all ten project translation units (`jllama.cpp`, `native_server.cpp`, `tts_engine.cpp`, `train_engine.cpp` + the six `src/test/cpp/test_*.cpp`) compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 2197709ac..bb78fb513 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10456 + GIT_TAG b10470 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 33c4018b7..3e8f344e8 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10456"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10470"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10456-"} — call + * plus the resolved upstream commit, e.g. {@code "b10470-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10456"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10470"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10456"; + public static final String LLAMA_CPP_VERSION = "b10470"; // Constants holder — not instantiable. private LlamaCppVersion() {} From dee4585b54294f15dbadf514729ae20bfc0ce3f6 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:02:19 +0000 Subject: [PATCH 02/76] chore: upgrade llama.cpp from b10470 to b10472 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 4b2c44125..73b435583 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10470** +Current llama.cpp pinned version: **b10472** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10470 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10472 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10470`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10472`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10470`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10472`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 5e8bb7dba..75aea3007 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10470](https://img.shields.io/badge/llama.cpp-%23b10470-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10470) +[![llama.cpp b10472](https://img.shields.io/badge/llama.cpp-%23b10472-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10472) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index d98d320f7..2d535fd00 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -621,3 +621,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10423–b10456 | upstream verification (sandbox, final target) | All **6** patches re-verified against a clean b10456 checkout: sequential `git apply` (filename order, mirroring the `PATCH_COMMAND` applier) succeeded at **every** intermediate tag of the walk, and the fail-loud `PATCH_COMMAND` ran clean on a fresh `cmake -B build` at the target — no patch needed refreshing across the whole range. **Full local verification (mandatory for the final target):** fresh configure (resolved `ggml commit f275595dd` = b10456 HEAD; all six patch markers confirmed present in the fetched tree) + full `cmake --build` (`libjllama.so` + `jllama_test` compile and link, `-O3`, no undefined references) + `ctest` **485/485 passing**. **API surface:** of the 22 upstream headers project source `#include`s directly, only three changed across the whole range — `tools/server/server-queue.h` (breaking: `on_new_task`'s callback type became `std::function`, plus the new `yield_to_queue()`; inert here, upstream's own `server-context.cpp` is the sole registrant), `ggml/include/ggml.h` (`ggml_ssm_scan()` gained a trailing `int64_t K`; called only inside llama.cpp's own TUs) and `tools/mtmd/mtmd-helper-common.h` (`size_t` widening, no signature change). `mtmd.h`, `mtmd-helper.h`, `common.h`, `chat.h`, `llama.h`, `arg.h` and every `tools/server/*.h` the project includes are byte-identical b10423→b10456; no files were added or removed under `tools/server`/`tools/mtmd`/`vendor` and no upstream `CMakeLists.txt` changed, so the b10154 `server-mcp.cpp` missing-`target_sources` failure class does not recur. Walk of 8 steps (b10423→b10430→b10436→b10441→b10447→b10448→b10454→b10455→b10456) on branch `claude/update-b10456-8a7j58`. Two steps exceeded the 100 KiB chunking threshold unavoidably — upstream has **no** intermediate tags there (b10449–b10453 and b10456 aside, nothing sits between b10447/b10448 or b10454/b10455) — and the 3.2 MiB b10454→b10455 figure is almost entirely the regenerated `docs/ops/SYCL.csv` (20 646 lines rewritten), leaving ~150 lines of reviewable code. | | b10456–b10470 | `tools/server/server-tools.cpp` (**built-in `get_datetime` tool removed, −57 lines, plus a "keep this array minimal" contributor note**) + `common/arg.cpp` & `tools/server/README.md` (**matching one-line `--tools` help-text edit**), `common/chat-diff-analyzer.cpp` (**additive Bailing-V3 template-patch lambda**), `common/speculative.cpp` (**`dflash.sample_from_anchor` GGUF knob in the file-local DFlash/DSpark impl**), `tools/mtmd/mtmd-image.cpp` (**two `GGML_ASSERT` bounds checks in the granite preprocessor**), `CMakeLists.txt` + `ggml/CMakeLists.txt` (**`*_VERSION_PATCH` 0→1**) | **No project-source change.** The only change in a file the project compiles is `server-tools.cpp`, but both the deleted `struct server_tool_get_datetime` and `build_tools()` are file-internal (declared in no header) and `server-tools.cpp` is deliberately not linked into `jllama_test` — greps for `get_datetime`/`server_tool`/`build_tools`/`find_tool`/`"/tools"` over `src/main/cpp` + `src/test/cpp` + the Java tree return zero hits; the sole effect is that NativeServer full mode's `/tools` advertises one fewer built-in (attach mode already 403s that route via patch 0007). No public header changed: `common/speculative.h`, `common/chat.h`, `tools/mtmd/mtmd{,-helper}.h` and every `tools/server/*.h` are untouched, so the speculative/chat-analyzer/mtmd edits stay inside upstream-compiled TUs. **Patch context intact:** the `arg.cpp` edit sits at line ~3362, >2000 lines from patch `0001`'s hunks (`@@ -1201` / `@@ -1242`), and `server.cpp`/`server-context.cpp`/`server-models.cpp` are not in the range at all. **Build wiring:** the `tools/server/` file set is byte-identical b10456→b10470 (`git ls-tree` diff empty) and `vendor/`+`include/` are untouched, so `target_sources` needs no edit and the b10154 `server-mcp.cpp` missing-source link-failure class does not recur. | | b10456–b10470 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10470 checkout: sequential `git apply --check` + `git apply` in filename order (mirroring the fail-loud `PATCH_COMMAND` applier) succeeded with zero fuzz. **Compile-verified per step:** a fresh `cmake -B build` against the patched b10470 tree configured clean, and all ten project translation units (`jllama.cpp`, `native_server.cpp`, `tts_engine.cpp`, `train_engine.cpp` + the six `src/test/cpp/test_*.cpp`) compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10470–b10472 | `CMakeLists.txt` + **NEW** `vendor/hash/**` (`CMakeLists.txt`, `rotate-bits/`, `sha1/`, `sha256/`, `xxhash/`) (**#27262: `examples/gguf-hash/deps/` promoted to a first-class `vendor-hash` static library, added to the root `CMakeLists.txt` *unconditionally* — before the `LLAMA_BUILD_COMMON` gate — because "mtmd needs this even when common is not built"**), `ggml/src/ggml-cuda/**` (**#27083, UMA override skipped for HIP**) | **No project-source and no project-CMake change.** `FetchContent_MakeAvailable(llama.cpp)` processes upstream's root `CMakeLists.txt`, so the new `vendor-hash` target is created before the project's own `add_subdirectory(${llama.cpp_SOURCE_DIR}/tools/mtmd)` — including on Android, where the project forces `LLAMA_BUILD_TOOLS OFF`/`LLAMA_BUILD_COMMON` off. Nothing links it yet in this step (mtmd picks it up at b10481). No header, no `tools/server` file and none of the five patched files changed, so all six patches keep byte-identical context. | +| b10470–b10472 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10472 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index bb78fb513..01ddeea1f 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10470 + GIT_TAG b10472 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 3e8f344e8..e2be11cb5 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10470"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10472"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10470-"} — call + * plus the resolved upstream commit, e.g. {@code "b10472-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10470"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10472"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10470"; + public static final String LLAMA_CPP_VERSION = "b10472"; // Constants holder — not instantiable. private LlamaCppVersion() {} From adbeeb3b4e48546c797451b17b8510a705624b31 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:04:57 +0000 Subject: [PATCH 03/76] chore: upgrade llama.cpp from b10472 to b10481 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 73b435583..6e0963b60 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10472** +Current llama.cpp pinned version: **b10481** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10472 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10481 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10472`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10481`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10472`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10481`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 75aea3007..3d6402556 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10472](https://img.shields.io/badge/llama.cpp-%23b10472-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10472) +[![llama.cpp b10481](https://img.shields.io/badge/llama.cpp-%23b10481-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10481) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 2d535fd00..7d6a99260 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -623,3 +623,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10456–b10470 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10470 checkout: sequential `git apply --check` + `git apply` in filename order (mirroring the fail-loud `PATCH_COMMAND` applier) succeeded with zero fuzz. **Compile-verified per step:** a fresh `cmake -B build` against the patched b10470 tree configured clean, and all ten project translation units (`jllama.cpp`, `native_server.cpp`, `tts_engine.cpp`, `train_engine.cpp` + the six `src/test/cpp/test_*.cpp`) compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10470–b10472 | `CMakeLists.txt` + **NEW** `vendor/hash/**` (`CMakeLists.txt`, `rotate-bits/`, `sha1/`, `sha256/`, `xxhash/`) (**#27262: `examples/gguf-hash/deps/` promoted to a first-class `vendor-hash` static library, added to the root `CMakeLists.txt` *unconditionally* — before the `LLAMA_BUILD_COMMON` gate — because "mtmd needs this even when common is not built"**), `ggml/src/ggml-cuda/**` (**#27083, UMA override skipped for HIP**) | **No project-source and no project-CMake change.** `FetchContent_MakeAvailable(llama.cpp)` processes upstream's root `CMakeLists.txt`, so the new `vendor-hash` target is created before the project's own `add_subdirectory(${llama.cpp_SOURCE_DIR}/tools/mtmd)` — including on Android, where the project forces `LLAMA_BUILD_TOOLS OFF`/`LLAMA_BUILD_COMMON` off. Nothing links it yet in this step (mtmd picks it up at b10481). No header, no `tools/server` file and none of the five patched files changed, so all six patches keep byte-identical context. | | b10470–b10472 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10472 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10472–b10481 | `tools/mtmd/{CMakeLists.txt,mtmd.h,mtmd.cpp,mtmd-helper.{h,cpp},mtmd-image.{h,cpp}}` (**#27274: bitmap IDs switch from a private FNV-1a decimal hash to `hash_sha256_hex()` "to prevent cache poisoning"; mtmd now links `vendor-hash`. Additive public API `mtmd_input_chunk_get_placeholder()`, `mtmd_input_chunk_save()` refactored around a shared impl with an unchanged C signature. #27246: LFM2 thumbnail skipped for non-tiled images**), `tools/server/server-common.{h,cpp}` + `tools/server/server-context.cpp` (**#27278: additive `server_tokens::push_back_placeholder()`; already-encoded media chunks are cached as metadata-only placeholders**), `tools/server/server-tools.{h,cpp}` + `tools/server/server.cpp` (**#27271: `server_tool::type()` default renamed `"builtin"` → `"server"`; one help-string line**), `vendor/hash/{CMakeLists.txt,hash.{h,cpp},sha1/**}` (**C++ `hash_sha256_hex()` wrapper; sha1 namespaced `vendor_hash`**), `CMakeLists.txt`, `tools/server/README{,-dev}.md` | **No project-source change.** The mtmd hash switch is invisible here: the project's only bitmap call site is `src/main/cpp/tts_engine.cpp:100` (`mtmd_helper_bitmap_init_from_file`, the TTS speaker-reference clip) — signature unchanged and the ID is never read back (`grep -rn "bitmap" src/main/cpp` shows no ID use). `mtmd_input_chunk_*` and `push_back_placeholder` are additive: `push_back(const mtmd_input_chunk *)` and `push_back(server_tokens &)` both survive, so every TU including `server-common.h` (`utils.hpp`, `jllama.cpp`, `native_server.cpp` + three test files) compiles unchanged, and the project's `server_tokens` tests all construct with `has_mtmd=false`. `server_tool::type()`'s renamed default is a `GET /tools` JSON field upstream marks "do NOT use in a downstream application" — zero hits for `"builtin"` in the project's C++ **and** Java trees, and no C++ test asserts on `server_tool::to_json()`. The vendored sha1 being namespaced `vendor_hash` matters for one project-specific reason: `llama/CMakeLists.txt` forces `LLAMA_BUILD_BORINGSSL ON` on Windows, so BoringSSL and `vendor-hash` land in the same `jllama.dll` — the namespace prevents a `SHA1*` symbol clash, and `vendor-hash` is linked `PRIVATE` into mtmd so its include dir never shadows anything. **Patch context intact:** the two one-line edits in patched files sit far outside every hunk (`server.cpp:346` between patch 0007's `@@ -258,47` and `@@ -556,3`; `server-context.cpp:3416` ~2 260 lines from patch 0002's `@@ -1152,8`). **Build wiring:** no `tools/server` source added/removed/renamed, so `llama/CMakeLists.txt`'s explicit `target_sources` list still matches upstream. | +| b10472–b10481 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10481 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 01ddeea1f..ee2604f9a 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10472 + GIT_TAG b10481 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index e2be11cb5..b48623856 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10472"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10481"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10472-"} — call + * plus the resolved upstream commit, e.g. {@code "b10481-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10472"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10481"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10472"; + public static final String LLAMA_CPP_VERSION = "b10481"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 18ac941b592ea2bd6641851227cc23bc67c12ceb Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:07:37 +0000 Subject: [PATCH 04/76] chore: upgrade llama.cpp from b10481 to b10488 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 6e0963b60..5cc6b8ec4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10481** +Current llama.cpp pinned version: **b10488** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10481 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10488 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10481`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10488`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10481`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10488`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 3d6402556..c404f475c 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10481](https://img.shields.io/badge/llama.cpp-%23b10481-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10481) +[![llama.cpp b10488](https://img.shields.io/badge/llama.cpp-%23b10488-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10488) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 7d6a99260..3e8894150 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -625,3 +625,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10470–b10472 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10472 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10472–b10481 | `tools/mtmd/{CMakeLists.txt,mtmd.h,mtmd.cpp,mtmd-helper.{h,cpp},mtmd-image.{h,cpp}}` (**#27274: bitmap IDs switch from a private FNV-1a decimal hash to `hash_sha256_hex()` "to prevent cache poisoning"; mtmd now links `vendor-hash`. Additive public API `mtmd_input_chunk_get_placeholder()`, `mtmd_input_chunk_save()` refactored around a shared impl with an unchanged C signature. #27246: LFM2 thumbnail skipped for non-tiled images**), `tools/server/server-common.{h,cpp}` + `tools/server/server-context.cpp` (**#27278: additive `server_tokens::push_back_placeholder()`; already-encoded media chunks are cached as metadata-only placeholders**), `tools/server/server-tools.{h,cpp}` + `tools/server/server.cpp` (**#27271: `server_tool::type()` default renamed `"builtin"` → `"server"`; one help-string line**), `vendor/hash/{CMakeLists.txt,hash.{h,cpp},sha1/**}` (**C++ `hash_sha256_hex()` wrapper; sha1 namespaced `vendor_hash`**), `CMakeLists.txt`, `tools/server/README{,-dev}.md` | **No project-source change.** The mtmd hash switch is invisible here: the project's only bitmap call site is `src/main/cpp/tts_engine.cpp:100` (`mtmd_helper_bitmap_init_from_file`, the TTS speaker-reference clip) — signature unchanged and the ID is never read back (`grep -rn "bitmap" src/main/cpp` shows no ID use). `mtmd_input_chunk_*` and `push_back_placeholder` are additive: `push_back(const mtmd_input_chunk *)` and `push_back(server_tokens &)` both survive, so every TU including `server-common.h` (`utils.hpp`, `jllama.cpp`, `native_server.cpp` + three test files) compiles unchanged, and the project's `server_tokens` tests all construct with `has_mtmd=false`. `server_tool::type()`'s renamed default is a `GET /tools` JSON field upstream marks "do NOT use in a downstream application" — zero hits for `"builtin"` in the project's C++ **and** Java trees, and no C++ test asserts on `server_tool::to_json()`. The vendored sha1 being namespaced `vendor_hash` matters for one project-specific reason: `llama/CMakeLists.txt` forces `LLAMA_BUILD_BORINGSSL ON` on Windows, so BoringSSL and `vendor-hash` land in the same `jllama.dll` — the namespace prevents a `SHA1*` symbol clash, and `vendor-hash` is linked `PRIVATE` into mtmd so its include dir never shadows anything. **Patch context intact:** the two one-line edits in patched files sit far outside every hunk (`server.cpp:346` between patch 0007's `@@ -258,47` and `@@ -556,3`; `server-context.cpp:3416` ~2 260 lines from patch 0002's `@@ -1152,8`). **Build wiring:** no `tools/server` source added/removed/renamed, so `llama/CMakeLists.txt`'s explicit `target_sources` list still matches upstream. | | b10472–b10481 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10481 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10481–b10488 | `CMakeLists.txt` + `common/CMakeLists.txt` + **NEW** `vendor/CMakeLists.txt` and `vendor/{cpp-httplib,hash,miniaudio,nlohmann,sheredom,stb}/CMakeLists.txt` (**#27304 "build : fix xcframework + cmake clean-up": one `add_subdirectory(vendor)`, per-vendor `vendor::` aliases, header-only deps become INTERFACE targets exporting the `vendor/` root; `common` drops `PUBLIC ../vendor` in favour of `PUBLIC vendor::nlohmann vendor::sheredom`; `LLAMA_VERSION_PATCH` 1→2**), `tools/mtmd/{CMakeLists.txt,mtmd-helper.cpp}` (**mtmd links `vendor::{hash,miniaudio,stb,sheredom}` instead of raw `../..`/`../../vendor` include dirs; `"hash.h"` → `"hash/hash.h"`**), `tools/mtmd/{mtmd-image.{h,cpp},clip-impl.h}` (**#27057: LFM2 tiling threshold reworked into a new `should_tile()`; `#ifndef DIRECTORY_SEPARATOR` guard**) | **No project-source and no project-CMake change** — but this is the range with the one real build-wiring risk, so it was checked rather than assumed. The project compiles `tools/server/server-http.cpp` (whose `#include ` needs the `vendor/` root on the include path) and `vendor/cpp-httplib/httplib.cpp` into `jllama` itself, adds no vendor include directory of its own, and rode on `llama-common`'s `PUBLIC ../vendor` — which b10488 deletes. It still resolves because `vendor::nlohmann` / `vendor::sheredom` are linked **PUBLIC** into `llama-common` and each declares `target_include_directories( INTERFACE ..)` = the `vendor/` root, which CMake propagates transitively to `jllama` (confirmed by the compile of `server-http.cpp` and `httplib.cpp` in this step's verification build). `add_subdirectory(vendor)` is unconditional, so the `vendor::*` ALIAS targets exist before the project's own `add_subdirectory(tools/mtmd)`, Android included. The new bare target names (`nlohmann`, `sheredom`, `miniaudio`, `stb`) do not clash with the project's FetchContent nlohmann/json, whose target is `nlohmann_json`; the vendored copy stays 3.12.0, the same version. mtmd's PUBLIC surface and every `tools/server/*` header are unchanged. **Patches:** none of `common/arg.{h,cpp}`, `tools/server/server.cpp`, `server-context.{cpp,h}`, `server-models.cpp` changed in this range — zero context shift. *Optional hardening, deliberately NOT done in this bump:* adding `${llama.cpp_SOURCE_DIR}/vendor` to `jllama`'s own `target_include_directories` would make the httplib include explicit instead of transitive, and immune to a future upstream `PUBLIC`→`PRIVATE` flip on `llama-common`. | +| b10481–b10488 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10488 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Because this is the range that deletes `llama-common`'s `PUBLIC ../vendor` include directory, the two upstream TUs the project compiles itself that depend on it — `tools/server/server-http.cpp` (`#include `) and `vendor/cpp-httplib/httplib.cpp` — were additionally compiled against the patched b10488 tree, both clean: the include still resolves transitively through the PUBLIC `vendor::nlohmann` / `vendor::sheredom` INTERFACE targets. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index ee2604f9a..9d1d32062 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10481 + GIT_TAG b10488 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index b48623856..4b06adf40 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10481"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10488"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10481-"} — call + * plus the resolved upstream commit, e.g. {@code "b10488-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10481"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10488"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10481"; + public static final String LLAMA_CPP_VERSION = "b10488"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 368189df591caa650c02050641da053519e35fd3 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:11:42 +0000 Subject: [PATCH 05/76] chore: upgrade llama.cpp from b10488 to b10499 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 5cc6b8ec4..127c7d2ed 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10488** +Current llama.cpp pinned version: **b10499** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10488 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10499 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10488`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10499`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10488`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10499`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index c404f475c..15694296a 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10488](https://img.shields.io/badge/llama.cpp-%23b10488-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10488) +[![llama.cpp b10499](https://img.shields.io/badge/llama.cpp-%23b10499-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10499) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 3e8894150..3e4f5fc4d 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -627,3 +627,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10472–b10481 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10481 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10481–b10488 | `CMakeLists.txt` + `common/CMakeLists.txt` + **NEW** `vendor/CMakeLists.txt` and `vendor/{cpp-httplib,hash,miniaudio,nlohmann,sheredom,stb}/CMakeLists.txt` (**#27304 "build : fix xcframework + cmake clean-up": one `add_subdirectory(vendor)`, per-vendor `vendor::` aliases, header-only deps become INTERFACE targets exporting the `vendor/` root; `common` drops `PUBLIC ../vendor` in favour of `PUBLIC vendor::nlohmann vendor::sheredom`; `LLAMA_VERSION_PATCH` 1→2**), `tools/mtmd/{CMakeLists.txt,mtmd-helper.cpp}` (**mtmd links `vendor::{hash,miniaudio,stb,sheredom}` instead of raw `../..`/`../../vendor` include dirs; `"hash.h"` → `"hash/hash.h"`**), `tools/mtmd/{mtmd-image.{h,cpp},clip-impl.h}` (**#27057: LFM2 tiling threshold reworked into a new `should_tile()`; `#ifndef DIRECTORY_SEPARATOR` guard**) | **No project-source and no project-CMake change** — but this is the range with the one real build-wiring risk, so it was checked rather than assumed. The project compiles `tools/server/server-http.cpp` (whose `#include ` needs the `vendor/` root on the include path) and `vendor/cpp-httplib/httplib.cpp` into `jllama` itself, adds no vendor include directory of its own, and rode on `llama-common`'s `PUBLIC ../vendor` — which b10488 deletes. It still resolves because `vendor::nlohmann` / `vendor::sheredom` are linked **PUBLIC** into `llama-common` and each declares `target_include_directories( INTERFACE ..)` = the `vendor/` root, which CMake propagates transitively to `jllama` (confirmed by the compile of `server-http.cpp` and `httplib.cpp` in this step's verification build). `add_subdirectory(vendor)` is unconditional, so the `vendor::*` ALIAS targets exist before the project's own `add_subdirectory(tools/mtmd)`, Android included. The new bare target names (`nlohmann`, `sheredom`, `miniaudio`, `stb`) do not clash with the project's FetchContent nlohmann/json, whose target is `nlohmann_json`; the vendored copy stays 3.12.0, the same version. mtmd's PUBLIC surface and every `tools/server/*` header are unchanged. **Patches:** none of `common/arg.{h,cpp}`, `tools/server/server.cpp`, `server-context.{cpp,h}`, `server-models.cpp` changed in this range — zero context shift. *Optional hardening, deliberately NOT done in this bump:* adding `${llama.cpp_SOURCE_DIR}/vendor` to `jllama`'s own `target_include_directories` would make the httplib include explicit instead of transitive, and immune to a future upstream `PUBLIC`→`PRIVATE` flip on `llama-common`. | | b10481–b10488 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10488 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Because this is the range that deletes `llama-common`'s `PUBLIC ../vendor` include directory, the two upstream TUs the project compiles itself that depend on it — `tools/server/server-http.cpp` (`#include `) and `vendor/cpp-httplib/httplib.cpp` — were additionally compiled against the patched b10488 tree, both clean: the include still resolves transitively through the PUBLIC `vendor::nlohmann` / `vendor::sheredom` INTERFACE targets. Full link + `ctest` deferred to the b10618 target. | +| b10488–b10499 | `common/common.cpp` (**#27138: threadpool sharing when only `n_threads` differs — new file-static `can_share_threadpool()`; a `-t`/`-tb` mismatch yields ONE unpaused pool sized `max(n, nb)` instead of two**), `tools/server/server-models.cpp` (**#27347: `CMD_CHILD_TO_ROUTER_STATE` lines demoted `LOG` → `LOG_DBG`, other child output forwarded in a new `else` branch**), `tools/mtmd/models/deepseekocr.cpp` (**#26727: DeepSeek-OCR SAM convolutions re-expressed via a file-static `conv_2d_f32()` = `ggml_im2col` + `ggml_mul_mat` keeping the im2col in F32**), plus out-of-scope `tools/ui/**`, `ggml/src/ggml-{opencl,sycl}/**`, `ggml/include/ggml-rpc.h`, `src/llama-model-loader.h` | **No project-source change.** All three in-scope edits are behavioural/internal with no API surface moved: `common/common.h` is byte-identical across the range (so `struct common_threadpools` and every signature are unchanged) and no project source references `threadpool`/`ggml_threadpool_params_match`; the `server-models.cpp` edit is pure logging (`RouterModeIntegrationTest` drives the typed `RouterClient` over HTTP, not the log); the deepseekocr helper is `static` and compiles against the unchanged 11-parameter `ggml_im2col` declaration, and neither `mtmd.h` nor `mtmd-helper.h` changed. **Patches:** only `server-models.cpp` of the five patched files changed, ~810 lines below patch `0008`'s single `@@ -215 @@` hunk — no context shift. **Build wiring:** no `CMakeLists.txt` changed except upstream's own `tests/`, and no file was added/removed/renamed under `tools/server`, `tools/mtmd`, `vendor` or `include`. The three headers that changed anywhere (`ggml-rpc.h`, `ggml-sycl/fwht.hpp`, `src/llama-model-loader.h`) are all outside the project's include dependency graph. | +| b10488–b10499 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10499 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 9d1d32062..2576a9f3e 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10488 + GIT_TAG b10499 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 4b06adf40..df4023161 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10488"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10499"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10488-"} — call + * plus the resolved upstream commit, e.g. {@code "b10499-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10488"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10499"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10488"; + public static final String LLAMA_CPP_VERSION = "b10499"; // Constants holder — not instantiable. private LlamaCppVersion() {} From c2191ec9a9cbbaf3f657c193c18f3b2d2801418f Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:14:22 +0000 Subject: [PATCH 06/76] chore: upgrade llama.cpp from b10499 to b10507 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 127c7d2ed..a5837ddea 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10499** +Current llama.cpp pinned version: **b10507** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10499 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10507 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10499`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10507`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10499`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10507`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 15694296a..4b8eeb124 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10499](https://img.shields.io/badge/llama.cpp-%23b10499-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10499) +[![llama.cpp b10507](https://img.shields.io/badge/llama.cpp-%23b10507-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10507) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 3e4f5fc4d..c824119b7 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -629,3 +629,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10481–b10488 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10488 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Because this is the range that deletes `llama-common`'s `PUBLIC ../vendor` include directory, the two upstream TUs the project compiles itself that depend on it — `tools/server/server-http.cpp` (`#include `) and `vendor/cpp-httplib/httplib.cpp` — were additionally compiled against the patched b10488 tree, both clean: the include still resolves transitively through the PUBLIC `vendor::nlohmann` / `vendor::sheredom` INTERFACE targets. Full link + `ctest` deferred to the b10618 target. | | b10488–b10499 | `common/common.cpp` (**#27138: threadpool sharing when only `n_threads` differs — new file-static `can_share_threadpool()`; a `-t`/`-tb` mismatch yields ONE unpaused pool sized `max(n, nb)` instead of two**), `tools/server/server-models.cpp` (**#27347: `CMD_CHILD_TO_ROUTER_STATE` lines demoted `LOG` → `LOG_DBG`, other child output forwarded in a new `else` branch**), `tools/mtmd/models/deepseekocr.cpp` (**#26727: DeepSeek-OCR SAM convolutions re-expressed via a file-static `conv_2d_f32()` = `ggml_im2col` + `ggml_mul_mat` keeping the im2col in F32**), plus out-of-scope `tools/ui/**`, `ggml/src/ggml-{opencl,sycl}/**`, `ggml/include/ggml-rpc.h`, `src/llama-model-loader.h` | **No project-source change.** All three in-scope edits are behavioural/internal with no API surface moved: `common/common.h` is byte-identical across the range (so `struct common_threadpools` and every signature are unchanged) and no project source references `threadpool`/`ggml_threadpool_params_match`; the `server-models.cpp` edit is pure logging (`RouterModeIntegrationTest` drives the typed `RouterClient` over HTTP, not the log); the deepseekocr helper is `static` and compiles against the unchanged 11-parameter `ggml_im2col` declaration, and neither `mtmd.h` nor `mtmd-helper.h` changed. **Patches:** only `server-models.cpp` of the five patched files changed, ~810 lines below patch `0008`'s single `@@ -215 @@` hunk — no context shift. **Build wiring:** no `CMakeLists.txt` changed except upstream's own `tests/`, and no file was added/removed/renamed under `tools/server`, `tools/mtmd`, `vendor` or `include`. The three headers that changed anywhere (`ggml-rpc.h`, `ggml-sycl/fwht.hpp`, `src/llama-model-loader.h`) are all outside the project's include dependency graph. | | b10488–b10499 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10499 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10499–b10507 | `common/arg.{h,cpp}` + `common/download.{h,cpp}` + `tools/server/server-models.{h,cpp}` (**#27346 router preset option `dedup-cache-models`** — new `COMMON_ARG_PRESET_DEDUP_CACHE_MODELS` macro, new `common_download_resolve_path()`, new `server_model_meta::hidden` + a skip in the `GET /models` handler), `tools/mtmd/{mtmd.h,mtmd.cpp,mtmd-helper.cpp,CMakeLists.txt}` + **NEW** `tools/mtmd/mtmd-internal.h` (**#27348 `mtmd_bitmap_set_mergeable()`** — Qwen-VL temporal frame merging becomes opt-in: `can_merge_with()` now also requires `mergeable && other.mergeable`, and only `mtmd_helper_video`'s frame reader sets it), `ggml/src/ggml-{webgpu,vulkan,metal}/**`, `gguf-py/**`, `tests/**` | **No project-source change.** Both `common/download.h` and `tools/mtmd/mtmd.h` changes are *purely additive* — no existing declaration moved — and grepping `src/main/cpp` + `src/test/cpp` for `common_download*`, `COMMON_ARG_PRESET*`, `common_preset`, `mtmd_bitmap_*`, `mtmd_input_part`, `mtmd_group_mergeable_bitmaps`, `server_model_meta`, `get_all_meta` returns **zero** hits (the only `server-models` mention is a comment in `native_server.cpp:188`). `server-models.cpp` is compiled into both `jllama` and `jllama_test`, and its new `common_download_resolve_path()` call resolves from the `llama-common` both already link — no CMake change. The new `mtmd-internal.h` is added only to `add_library(mtmd …)`'s **header** list (no new `.cpp`), so it cannot repeat the b10154 `server-mcp.cpp` missing-`target_sources` link failure. Two behaviour notes, neither actionable: the `/models` JSON *shape* is unchanged (hidden entries are omitted, never serialized) and nothing is hidden unless a preset opts in with `dedup-cache-models`, which this project never writes; and because upstream's server builds bitmaps with `mtmd_helper_bitmap_init_from_buf` without setting `mergeable`, two consecutive same-size images in one request are no longer temporal-merged on Qwen-VL models — upstream's deliberate fix, invisible to SmolVLM (`n_merge_frames == 1`) and to the TTS speaker-reference clip (an audio bitmap, excluded by the pre-existing `!is_audio` guard). **Patch context:** patch `0008`'s `server-models.cpp` hunk keeps the identical offset across the range; patch `0001`'s `common/arg.h` hunk shifts by exactly **+1 line** (the reformatted preset-macro block) with unchanged context and still applies; `server.cpp` and `server-context.{cpp,h}` are untouched. | +| b10499–b10507 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10507 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 2576a9f3e..15edd2d6a 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10499 + GIT_TAG b10507 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index df4023161..bf7fd7d9c 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10499"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10507"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10499-"} — call + * plus the resolved upstream commit, e.g. {@code "b10507-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10499"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10507"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10499"; + public static final String LLAMA_CPP_VERSION = "b10507"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 337945c8131163a3ad776bd8ac993e0965c14342 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:17:09 +0000 Subject: [PATCH 07/76] chore: upgrade llama.cpp from b10507 to b10509 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index a5837ddea..a5d17015d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10507** +Current llama.cpp pinned version: **b10509** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10507 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10509 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10507`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10509`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10507`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10509`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 4b8eeb124..98598f82e 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10507](https://img.shields.io/badge/llama.cpp-%23b10507-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10507) +[![llama.cpp b10509](https://img.shields.io/badge/llama.cpp-%23b10509-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10509) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index c824119b7..c99d18ac0 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -631,3 +631,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10488–b10499 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10499 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10499–b10507 | `common/arg.{h,cpp}` + `common/download.{h,cpp}` + `tools/server/server-models.{h,cpp}` (**#27346 router preset option `dedup-cache-models`** — new `COMMON_ARG_PRESET_DEDUP_CACHE_MODELS` macro, new `common_download_resolve_path()`, new `server_model_meta::hidden` + a skip in the `GET /models` handler), `tools/mtmd/{mtmd.h,mtmd.cpp,mtmd-helper.cpp,CMakeLists.txt}` + **NEW** `tools/mtmd/mtmd-internal.h` (**#27348 `mtmd_bitmap_set_mergeable()`** — Qwen-VL temporal frame merging becomes opt-in: `can_merge_with()` now also requires `mergeable && other.mergeable`, and only `mtmd_helper_video`'s frame reader sets it), `ggml/src/ggml-{webgpu,vulkan,metal}/**`, `gguf-py/**`, `tests/**` | **No project-source change.** Both `common/download.h` and `tools/mtmd/mtmd.h` changes are *purely additive* — no existing declaration moved — and grepping `src/main/cpp` + `src/test/cpp` for `common_download*`, `COMMON_ARG_PRESET*`, `common_preset`, `mtmd_bitmap_*`, `mtmd_input_part`, `mtmd_group_mergeable_bitmaps`, `server_model_meta`, `get_all_meta` returns **zero** hits (the only `server-models` mention is a comment in `native_server.cpp:188`). `server-models.cpp` is compiled into both `jllama` and `jllama_test`, and its new `common_download_resolve_path()` call resolves from the `llama-common` both already link — no CMake change. The new `mtmd-internal.h` is added only to `add_library(mtmd …)`'s **header** list (no new `.cpp`), so it cannot repeat the b10154 `server-mcp.cpp` missing-`target_sources` link failure. Two behaviour notes, neither actionable: the `/models` JSON *shape* is unchanged (hidden entries are omitted, never serialized) and nothing is hidden unless a preset opts in with `dedup-cache-models`, which this project never writes; and because upstream's server builds bitmaps with `mtmd_helper_bitmap_init_from_buf` without setting `mergeable`, two consecutive same-size images in one request are no longer temporal-merged on Qwen-VL models — upstream's deliberate fix, invisible to SmolVLM (`n_merge_frames == 1`) and to the TTS speaker-reference clip (an audio bitmap, excluded by the pre-existing `!is_audio` guard). **Patch context:** patch `0008`'s `server-models.cpp` hunk keeps the identical offset across the range; patch `0001`'s `common/arg.h` hunk shifts by exactly **+1 line** (the reformatted preset-macro block) with unchanged context and still applies; `server.cpp` and `server-context.{cpp,h}` are untouched. | | b10499–b10507 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10507 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10507–b10509 | `ggml/include/ggml.h` + `ggml/src/**` + `ggml/src/ggml-metal/**` (**#27120: new `ggml_rope_set_offset()` and its Metal support**), `tools/ui/**` (**#27365: persisted settings read before the API-key probe**) | **No project-source change.** Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or the root `CMakeLists.txt` changed at all in this range, so none of the five patched files moved and no header in the project's include dependency graph was touched. `ggml_rope_set_offset` is purely additive to `ggml.h` (a header CLAUDE.md's review list marks "safe to skip" — the project never calls a `ggml_rope_*` primitive; `grep -rn "ggml_rope" src/main/cpp src/test/cpp` is empty), and `tools/ui` is the Svelte WebUI, which CI rebuilds from the pinned `GIT_TAG` and which therefore needs no per-bump source review. | +| b10507–b10509 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10509 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. This is the **last** tag at which patch `0007` applies unmodified — see the next row. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 15edd2d6a..2cc25e87d 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10507 + GIT_TAG b10509 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index bf7fd7d9c..a5aab2289 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10507"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10509"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10507-"} — call + * plus the resolved upstream commit, e.g. {@code "b10509-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10507"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10509"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10507"; + public static final String LLAMA_CPP_VERSION = "b10509"; // Constants holder — not instantiable. private LlamaCppVersion() {} From d78c161c94fd6b5b87b5a9c54fcaa3ffe40d1657 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:20:35 +0000 Subject: [PATCH 08/76] chore: upgrade llama.cpp from b10509 to b10519 Upstream #27376 splits server_task_result_metrics: the /slots payload moves to a new server_task_result_slots fed by SERVER_TASK_TYPE_SLOT_GET. Re-point handleSlotAction LIST at the new task (it still compiled, but would have returned an empty object) and move the C++ slots assertions to the new type. Refresh patch 0007, whose route-table removal block no longer matched after upstream #26347 dropped two trailing comments. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 14 +++--- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 + llama/CMakeLists.txt | 2 +- .../0007-server-attach-http-frontend.patch | 14 +++--- llama/src/main/cpp/jllama.cpp | 8 +++- .../llama/value/LlamaCppVersion.java | 8 ++-- llama/src/test/cpp/test_server.cpp | 44 ++++++++++++++----- 8 files changed, 62 insertions(+), 32 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index a5d17015d..e47d730c5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10509** +Current llama.cpp pinned version: **b10519** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10509 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10519 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10509`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10519`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -659,7 +659,7 @@ Current patches: | `0001-win32-arg-parse-embed-guard.patch` | Windows JNI regression from llama.cpp **#24779** (introduced b9739): on Windows `common_params_parse` re-derived argv from the **process** command line (`GetCommandLineW`) and adopted it, so an embedded/JNI caller (`java.exe`) lost its `--model …` args → "Failed to parse model parameters". b9789 narrowed the unconditional override to a **count-guard** (`if (static_cast(utf8.buf.size()) == argc) { argv = utf8.ptrs.data(); }`), but that is exactly the variant the project already found breaks its Windows server-integration tests (when the embedded argv length coincides with `java.exe`'s). The patch carries the **complete upstream change** (so it can be submitted to llama.cpp verbatim and then dropped here): **(1)** `common_params_parse` parses **exactly the argv it is given** (no `GetCommandLineW` magic) and a new `common_params_parse_main()` wrapper holds the UTF-8 recovery for the standalone tools' `main()` (`common/arg.{cpp,h}`); **(2)** the **~34 standalone `main()` call sites** (every `common_params_parse(argc, argv, …)` across `tools/*`, `examples/*` and the `tests/*` programs) flip to `common_params_parse_main()`; **(3)** a `tests/test-arg-parser.cpp` regression case pins that `common_params_parse` honors a caller-supplied argv. The embedded caller (`jllama.cpp`) keeps calling `common_params_parse` and is never overridden. **Our subproject build compiles only the `arg.{cpp,h}` core** — `LLAMA_BUILD_TOOLS`/`LLAMA_BUILD_TESTS` are OFF for a FetchContent subproject — so the flips + test are applied-but-not-compiled here; they were validated via a one-off `-DLLAMA_BUILD_TOOLS=ON -DLLAMA_BUILD_TESTS=ON` build (the new test compiles and its asserts pass; `test-arg-parser`'s only red there is the live `ggml.ai` download check, which is sandbox-network, not the patch). Because it spans **37 files** it must be refreshed on every llama.cpp bump (the applier fails loud). The upstream-facing write-up, including a standalone reproducer that makes llama.cpp's own `test-arg-parser` fail on unmodified `master`, lives in [docs/upstream-investigation-win32-argv-substitution.md](docs/upstream-investigation-win32-argv-substitution.md). **Reported upstream as [ggml-org/llama.cpp#26416](https://github.com/ggml-org/llama.cpp/issues/26416)** (2026-08-01, label `bug-unconfirmed`, first bad commit `508a475`); the issue asks which of the two directions the maintainers prefer before a PR is opened, so this patch stays downstream until they answer. | | `0002-server-preserve-caller-load-progress-callback.patch` | Load-progress-callback regression introduced in llama.cpp **b9789**: `server_context::load_model` (`tools/server/server-context.cpp`) now **unconditionally** installs the server's own load-progress reporter on `params_base.load_progress_callback` immediately before `common_init_from_params`, clobbering any callback the embedding caller already set. libjllama's `LoadProgressCallback` feature wires `common_params.load_progress_callback` to a JNI trampoline *before* calling `load_model`, so the bump silently killed it — `LoadProgressCallbackTest` saw zero progress updates and the abort-on-`false` path never threw. The patch guards the assignment with `if (params_base.load_progress_callback == nullptr)`, so the server installs its own reporter **only when the caller hasn't** — a caller-supplied callback survives and fires during load. Standalone `llama-server` (no caller callback, so the field is null) is unaffected. Same JNI-vs-standalone divergence class as `0001`. | | `0003-pr22393-server-add-slot-prompt-similarity-getter-setter.patch` | **Upstream-PR carry** of [ggml-org/llama.cpp#22393](https://github.com/ggml-org/llama.cpp/pull/22393) ("server : add slot_prompt_similarity getter/setter"). Purely additive: adds `server_context::get_slot_prompt_similarity()` / `set_slot_prompt_similarity(float)` (`tools/server/server-context.{cpp,h}`) so an embedding/JNI caller can query and tune the slot-selection threshold at runtime without reloading the model. Verbatim copy of the PR, which **upstream closed without merging** (rejected as exposing unsafe internal state — see the patch header). Carried permanently; it will not be droppable via a version bump. | -| `0007-server-attach-http-frontend.patch` | **Adds `llama_server_attach(argc, argv, server_context&)`** so the `NativeServer` *attach mode* can serve an **already-loaded `LlamaModel`** over the upstream HTTP frontend — no second model load, no `start_loop()`; the LlamaModel's worker keeps driving the shared `server_context` and the HTTP routes post tasks to its queue (the queue is the synchronization point). Mechanically: (1) extracts the **pure core route table** (`health` … `slots`) out of `llama_server()` into `static void llama_server_register_common_routes(ctx_http, routes)` (shared, so the two entry points cannot drift on the core endpoint set). **Scope note (narrowed at the b10154 bump):** the helper deliberately carries **only** the stable, state-independent route table — **not** the resumable-streaming routes (their handlers differ between router / non-router), the GCP-compat shim, or the experimental **CORS-proxy / MCP-server / built-in-tools** wiring. b10154 (upstream MCP-server support) moved the streaming routes into the middle of that block and coupled tools/CORS to a per-call `server_mcp mcp_mgr` lifecycle, so the earlier contiguous "route-table + CORS-proxy + tools" extraction is no longer possible; `llama_server()` keeps all of that inline, **byte-identical to upstream b10154** (only the route-table block is factored out). (2) adds `llama_server_attach`, which parses only the HTTP-side argv via `common_params_parse`, starts the stream-session GC + `server_http_context`, registers the common route table, the **non-router** resumable-streaming handlers (upstream b10154 paths `/v1/stream` GET/DEL + `/v1/streams/lookup` POST), the GCP-compat shim, and **403 "disabled" stubs for `/cors-proxy` + `/tools`** (attach mode does not wire the experimental CORS-proxy / MCP / built-in-tools host — those belong to a full `llama-server`, not an embedded model), marks ready immediately (model already loaded), and blocks on the HTTP thread until `llama_server_request_shutdown()` — never calling `common_init()`, backend init, `ctx_server.terminate()` or `llama_backend_free()` (the embedding caller owns those). Applies after `0001`+`0006` (same file); closes the "NativeServer — reuse an already-loaded LlamaModel" TODO. Upstream-submittable ("server: let embedding callers attach the HTTP frontend to an existing server_context"). | +| `0007-server-attach-http-frontend.patch` | **Adds `llama_server_attach(argc, argv, server_context&)`** so the `NativeServer` *attach mode* can serve an **already-loaded `LlamaModel`** over the upstream HTTP frontend — no second model load, no `start_loop()`; the LlamaModel's worker keeps driving the shared `server_context` and the HTTP routes post tasks to its queue (the queue is the synchronization point). Mechanically: (1) extracts the **pure core route table** (`health` … `slots`) out of `llama_server()` into `static void llama_server_register_common_routes(ctx_http, routes)` (shared, so the two entry points cannot drift on the core endpoint set). **Scope note (narrowed at the b10154 bump):** the helper deliberately carries **only** the stable, state-independent route table — **not** the resumable-streaming routes (their handlers differ between router / non-router), the GCP-compat shim, or the experimental **CORS-proxy / MCP-server / built-in-tools** wiring. b10154 (upstream MCP-server support) moved the streaming routes into the middle of that block and coupled tools/CORS to a per-call `server_mcp mcp_mgr` lifecycle, so the earlier contiguous "route-table + CORS-proxy + tools" extraction is no longer possible; `llama_server()` keeps all of that inline, **byte-identical to upstream b10154** (only the route-table block is factored out). (2) adds `llama_server_attach`, which parses only the HTTP-side argv via `common_params_parse`, starts the stream-session GC + `server_http_context`, registers the common route table, the **non-router** resumable-streaming handlers (upstream b10154 paths `/v1/stream` GET/DEL + `/v1/streams/lookup` POST), the GCP-compat shim, and **403 "disabled" stubs for `/cors-proxy` + `/tools`** (attach mode does not wire the experimental CORS-proxy / MCP / built-in-tools host — those belong to a full `llama-server`, not an embedded model), marks ready immediately (model already loaded), and blocks on the HTTP thread until `llama_server_request_shutdown()` — never calling `common_init()`, backend init, `ctx_server.terminate()` or `llama_backend_free()` (the embedding caller owns those). Applies after `0001`+`0006` (same file); closes the "NativeServer — reuse an already-loaded LlamaModel" TODO. Upstream-submittable ("server: let embedding callers attach the HTTP frontend to an existing server_context"). **Refreshed at the b10519 bump:** upstream #26347 dropped the API key from the `/models` + `/v1/models` public-endpoint set and deleted the two trailing `// public endpoint (no API key check)` comments on those route registrations. Those two lines sit inside this patch's route-table removal block, so `git apply` failed ("patch does not apply", `server.cpp:258`) at **every** tag from b10519 on; the fix was to drop the now-wrong comment from all four affected lines (2 on the `-` side, 2 in the extracted helper on the `+` side), keeping the helper byte-identical to the block it replaces. **This is the invariant to re-check on every bump:** the `+` side of `llama_server_register_common_routes()` must stay a verbatim copy of the route table it factors out of `llama_server()`. | | `0008-server-models-worker-cmd-override.patch` | **Makes router mode usable in-JVM.** The router (`server-models.cpp`) spawns each model worker by re-executing its own binary (`get_server_exec_path()` = `/proc/self/exe` & friends) — inside a JVM that binary is `java`, not a llama-server, so embedded router workers could never start. The patch adds env `LLAMA_SERVER_WORKER_CMD` (whitespace-split; read in `server_model_meta::update_args`) which replaces only the leading binary-path token of the rendered worker args, letting an embedding host relaunch workers through its own bootstrap — e.g. `java -cp app.jar net.ladenthin.llama.server.NativeServer` (each worker is then a fresh JVM running the classic single-model `NativeServer`). Exposed in Java as `NativeServer.setWorkerCommand(String...)` (JNI `setenv`); exercised by `RouterModeIntegrationTest` (Linux CI). Upstream-submittable (also useful for containerized/wrapped deployments). | | `0006-server-embed-native-server-jni.patch` | **Makes `server.cpp`'s `llama_server` embeddable in the JVM** so the `NativeServer` JNI bridge can run the full upstream HTTP server (WebUI included) inside `libjllama` — see "Two server modes" below. b9870 already exposes `int llama_server(int, char**)` (non-static; no `main` in the file), so the patch only adds embedded-mode support: (1) a `g_llama_server_embedded` flag + `llama_server_set_embedded()` / `llama_server_request_shutdown()` (declared in the committed `src/main/cpp/native_server_bridge.h`); (2) skips installing the process-wide SIGINT/SIGTERM handlers when embedded (they would hijack the JVM's); (3) in embedded mode parses the **forwarded** argv via `common_params_parse` instead of `common_params_parse_main` (whose `GetCommandLineW` recovery would pick up `java.exe`'s command line — the same Windows class of bug `0001` fixes). `llama_server_request_shutdown()` mirrors the SIGTERM path (invokes the installed `shutdown_handler` → `ctx_server.terminate()` unblocks `start_loop()`), giving JNI an out-of-band stop since `ctx_server` is loop-local. Applies **after `0001`** (which flips this call site to `common_params_parse_main`), so its context is the post-`0001` tree; regenerate against `0001`+source on a bump. Only touches `tools/server/server.cpp`. | @@ -1347,17 +1347,17 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" | File | Tests | Scope | |------|-------|-------| | `src/test/cpp/test_utils.cpp` | 162 | Upstream helpers: `server_tokens`, `server_grammar_trigger`, `gen_tool_call_id`, `json_value`, `json_get_nested_values`, UTF-8 helpers, `format_response_rerank`, `format_embeddings_response_oaicompat`, `oaicompat_completion_params_parse`, `oaicompat_chat_params_parse`, `are_lora_equal`, `strip_flag_from_argv`, `token_piece_value`, `json_is_array_and_contains_numbers`, `format_oai_sse`, `format_oai_resp_sse`, `format_anthropic_sse`, `parse_lora_request` | -| `src/test/cpp/test_server.cpp` | 204 | Upstream result types: `server_slot_stats` (the `timings` JSON payload; replaced `result_timings` in b10408), `task_params::to_json()` (incl. `dry_sequence_breakers`, `preserved_tokens`, `timings_per_token`), `completion_token_output`, `server_task_result_cmpl_partial` (non-oaicompat + `to_json_oaicompat` + logprobs + `to_json_oaicompat_chat` + `to_json_anthropic` + dispatcher), `server_task_result_cmpl_final` (non-oaicompat + `to_json_oaicompat` + `to_json_oaicompat_chat` + `to_json_oaicompat_chat_stream` + `to_json_anthropic` + `to_json_anthropic_stream` + tool_calls + dispatcher), `server_task_result_embd`, `server_task_result_rerank`, `server_task_result_metrics` (`to_json()` = the `/slots` array + `to_metrics()` = the `/metrics` Prometheus text), `server_task_result_slot_save_load`, `server_task_result_slot_erase`, `server_task_result_apply_lora`, `server_task_result_get_lora`, `server_task_result_error`, `format_error_response`, `server_task::need_sampling()`, `server_task::n_tokens()`, `server_schema::eval_llama_cmpl_schema()` (parsing pipeline + grammar routing + error paths + per-request `dry_*` and `sse_ping_interval` field round-trips incl. hard-limit + server-default inheritance), `response_fields` projection | +| `src/test/cpp/test_server.cpp` | 206 | Upstream result types: `server_slot_stats` (the `timings` JSON payload; replaced `result_timings` in b10408), `task_params::to_json()` (incl. `dry_sequence_breakers`, `preserved_tokens`, `timings_per_token`), `completion_token_output`, `server_task_result_cmpl_partial` (non-oaicompat + `to_json_oaicompat` + logprobs + `to_json_oaicompat_chat` + `to_json_anthropic` + dispatcher), `server_task_result_cmpl_final` (non-oaicompat + `to_json_oaicompat` + `to_json_oaicompat_chat` + `to_json_oaicompat_chat_stream` + `to_json_anthropic` + `to_json_anthropic_stream` + tool_calls + dispatcher), `server_task_result_embd`, `server_task_result_rerank`, `server_task_result_metrics` (`to_metrics()` = the `/metrics` Prometheus exposition text; its `to_json()` is an unused empty object since b10519), `server_task_result_slots` (`to_json()` = the `/slots` array, fed by the b10519 `SERVER_TASK_TYPE_SLOT_GET` task), `server_task_result_slot_save_load`, `server_task_result_slot_erase`, `server_task_result_apply_lora`, `server_task_result_get_lora`, `server_task_result_error`, `format_error_response`, `server_task::need_sampling()`, `server_task::n_tokens()`, `server_schema::eval_llama_cmpl_schema()` (parsing pipeline + grammar routing + error paths + per-request `dry_*` and `sse_ping_interval` field round-trips incl. hard-limit + server-default inheritance), `response_fields` projection | | `src/test/cpp/test_json_helpers.cpp` | 50 | All functions in `json_helpers.hpp`: `get_result_error_message`, `results_to_json`, `rerank_results_to_json` (incl. missing/out-of-range `index` rejection), `parse_encoding_format`, `extract_embedding_prompt`, `is_infill_request`, `parse_slot_prompt_similarity`, `parse_positive_int_config`, `wrap_stream_chunk` | | `src/test/cpp/test_log_helpers.cpp` | 13 | All functions in `log_helpers.hpp`: `log_level_name`, `format_log_as_json` | | `src/test/cpp/test_jni_helpers.cpp` | 54 | All functions in `jni_helpers.hpp` using a zero-filled `JNINativeInterface_` mock (incl. the `utf8_to_jstring_impl` byte-array string path: emoji byte-preservation, truncated-UTF-8 replace-not-throw) | | `src/test/cpp/test_tts_wav.cpp` | 2 | The in-memory WAV writer `pcm_to_wav16_bytes` in `tts_wav.hpp` (WAV header/payload + little-endian clamping) — our own code, not upstream. The Qwen3-TTS pipeline it pairs with (`mtmd_helper::gen_audio`) is entirely upstream-owned (no project-side DSP to unit-test here) and covered end-to-end by the Java `TtsIntegrationTest`. | -**Current total: 485 tests (all passing).** +**Current total: 487 tests (all passing).** #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10509`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10519`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 98598f82e..a77004af4 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10509](https://img.shields.io/badge/llama.cpp-%23b10509-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10509) +[![llama.cpp b10519](https://img.shields.io/badge/llama.cpp-%23b10519-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10519) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index c99d18ac0..8f72359f9 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -633,3 +633,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10499–b10507 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10507 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10507–b10509 | `ggml/include/ggml.h` + `ggml/src/**` + `ggml/src/ggml-metal/**` (**#27120: new `ggml_rope_set_offset()` and its Metal support**), `tools/ui/**` (**#27365: persisted settings read before the API-key probe**) | **No project-source change.** Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or the root `CMakeLists.txt` changed at all in this range, so none of the five patched files moved and no header in the project's include dependency graph was touched. `ggml_rope_set_offset` is purely additive to `ggml.h` (a header CLAUDE.md's review list marks "safe to skip" — the project never calls a `ggml_rope_*` primitive; `grep -rn "ggml_rope" src/main/cpp src/test/cpp` is empty), and `tools/ui` is the Svelte WebUI, which CI rebuilds from the pinned `GIT_TAG` and which therefore needs no per-bump source review. | | b10507–b10509 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10509 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. This is the **last** tag at which patch `0007` applies unmodified — see the next row. | +| b10509–b10519 | `tools/server/server-task.{h,cpp}` + `tools/server/server-context.{h,cpp}` + `tools/server/server-queue.{h,cpp}` (**#27376 "server: refactor sleep handling, allow access /metrics during sleep"**), `tools/server/server.cpp` + `tools/server/server-http.cpp` (**#26347: `/models` + `/v1/models` are no longer API-key-exempt**), `common/common.cpp` (**#27337: revert of the b10499 threadpool sharing when `n_threads` differ**), `ggml/include/ggml.h`, `tools/ui/**`, `src/models/granite-swa.cpp` | **Project source change required (two), plus a patch refresh.** (1) #27376 split `server_task_result_metrics` in two: `n_idle_slots` + `slots_data` moved out into a **new `server_task_result_slots`**, produced by a **new `SERVER_TASK_TYPE_SLOT_GET`** task, and `server_task_result_metrics::to_json()` now returns an empty object (`/metrics` renders Prometheus text via the byte-identical `to_metrics()`). Two project impacts, one silent and one hard: `jllama.cpp`'s `handleSlotAction` LIST arm posted `SERVER_TASK_TYPE_METRICS`, which still **compiles** at b10519 but would return `{}` instead of the slot array to `LlamaModel.getMetrics()` — re-pointed at `SERVER_TASK_TYPE_SLOT_GET` (upstream's `/slots` handler builds exactly that task, with no extra fields); and `test_server.cpp` was a **hard compile error** (`make_metrics()` set the removed `n_idle_slots`, `ToJson_ReturnsSlotsArrayVerbatim` set the removed `slots_data`) — the slots assertions moved to `server_task_result_slots` and two tests were added (`ServerTaskResultSlots.ToJson_EmptyByDefault`, `ServerTaskResultMetrics.ToJson_UnusedEmptyObject`), 485 → 487. `to_metrics()` is byte-identical and never read `n_idle_slots`, so every Prometheus assertion survived unchanged. (2) **Patch `0007` stopped applying here** — #26347 deleted the trailing `// public endpoint (no API key check)` comments on the two `/models` route registrations, which sit inside `0007`'s `@@ -258,47 +310,7 @@` removal block (`git apply` → "patch does not apply", `server.cpp:258`); refreshed by dropping that comment from those four lines (2 on the `-` side, 2 on the extracted `llama_server_register_common_routes()` `+` side, where it had also become factually wrong). `0001`/`0002`/`0003`/`0006`/`0008` applied unchanged. No CMake wiring change: no `tools/server` or `vendor` file was added, removed or renamed. **Behavioural note for consumers:** a `NativeServer` started with `--api-key` now returns 401 on `/models` and `/v1/models`; the project's own `RouterModeIntegrationTest` runs without an API key, so CI is unaffected. The new `server_routes` sleep-cache callback registers in the constructor and is inert here — `sleep_idle_seconds` defaults to `-1` (disabled) and nothing in the project sets it. | +| b10509–b10519 | upstream verification (sandbox) | Patch `0007` **refreshed** (see above); all **6** patches then re-verified against a clean b10519 checkout — sequential `git apply --check` + `git apply` in filename order, zero fuzz — and the refreshed `0007` additionally re-checked to apply cleanly at **every** later endpoint of this walk up to b10618, so the refresh is a one-time fix, not a per-chunk one. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics against the patched b10519 tree. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 2cc25e87d..879bab457 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10509 + GIT_TAG b10519 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/patches/0007-server-attach-http-frontend.patch b/llama/patches/0007-server-attach-http-frontend.patch index ebc03b7d8..ec80ef913 100644 --- a/llama/patches/0007-server-attach-http-frontend.patch +++ b/llama/patches/0007-server-attach-http-frontend.patch @@ -1,5 +1,5 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp -index ce1d239c3..b242fc48a 100644 +index 102692c74..8389fe6bf 100644 --- a/tools/server/server.cpp +++ b/tools/server/server.cpp @@ -63,6 +63,7 @@ int llama_server(int argc, char ** argv); @@ -27,8 +27,8 @@ index ce1d239c3..b242fc48a 100644 + ctx_http.get ("/metrics", ex_wrapper(routes.get_metrics)); + ctx_http.get ("/props", ex_wrapper(routes.get_props)); + ctx_http.post("/props", ex_wrapper(routes.post_props)); -+ ctx_http.get ("/models", ex_wrapper(routes.get_models)); // public endpoint (no API key check) -+ ctx_http.get ("/v1/models", ex_wrapper(routes.get_models)); // public endpoint (no API key check) ++ ctx_http.get ("/models", ex_wrapper(routes.get_models)); ++ ctx_http.get ("/v1/models", ex_wrapper(routes.get_models)); + ctx_http.post("/completion", ex_wrapper(routes.post_completions)); // legacy + ctx_http.post("/completions", ex_wrapper(routes.post_completions)); + ctx_http.post("/v1/completions", ex_wrapper(routes.post_completions_oai)); @@ -68,7 +68,7 @@ index ce1d239c3..b242fc48a 100644 int llama_server(int argc, char ** argv) { std::setlocale(LC_NUMERIC, "C"); -@@ -258,47 +310,7 @@ int llama_server(common_params & params, int argc, char ** argv) { +@@ -259,47 +311,7 @@ int llama_server(common_params & params, int argc, char ** argv) { ctx_http.del ("/models", ex_wrapper(models_routes->del_router_models)); } @@ -77,8 +77,8 @@ index ce1d239c3..b242fc48a 100644 - ctx_http.get ("/metrics", ex_wrapper(routes.get_metrics)); - ctx_http.get ("/props", ex_wrapper(routes.get_props)); - ctx_http.post("/props", ex_wrapper(routes.post_props)); -- ctx_http.get ("/models", ex_wrapper(routes.get_models)); // public endpoint (no API key check) -- ctx_http.get ("/v1/models", ex_wrapper(routes.get_models)); // public endpoint (no API key check) +- ctx_http.get ("/models", ex_wrapper(routes.get_models)); +- ctx_http.get ("/v1/models", ex_wrapper(routes.get_models)); - ctx_http.post("/completion", ex_wrapper(routes.post_completions)); // legacy - ctx_http.post("/completions", ex_wrapper(routes.post_completions)); - ctx_http.post("/v1/completions", ex_wrapper(routes.post_completions_oai)); @@ -117,7 +117,7 @@ index ce1d239c3..b242fc48a 100644 // resumable streaming: a child binds the local session factories, the router binds // proxies that resolve the owning child, see server-stream.h -@@ -556,3 +568,89 @@ int llama_server(common_params & params, int argc, char ** argv) { +@@ -579,3 +591,89 @@ int llama_server(common_params & params, int argc, char ** argv) { return 0; } diff --git a/llama/src/main/cpp/jllama.cpp b/llama/src/main/cpp/jllama.cpp index d86f1c691..fa44f976e 100644 --- a/llama/src/main/cpp/jllama.cpp +++ b/llama/src/main/cpp/jllama.cpp @@ -1623,8 +1623,12 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleSlotAction(J REQUIRE_SERVER_CONTEXT(nullptr); switch (action) { - case 0: // LIST — get slot info via metrics task - return dispatch_one_shot_task(env, ctx_server, server_task(SERVER_TASK_TYPE_METRICS)); + case 0: // LIST — get slot info via the dedicated slot-get task + // b10519 (upstream #27376) split the old METRICS task in two: METRICS now carries only the + // cumulative counters (rendered as Prometheus text by to_metrics(); its to_json() returns an + // empty object), and the /slots payload moved to SERVER_TASK_TYPE_SLOT_GET -> + // server_task_result_slots::to_json(), which returns the slot array verbatim. + return dispatch_one_shot_task(env, ctx_server, server_task(SERVER_TASK_TYPE_SLOT_GET)); case 1: // SAVE return exec_slot_file_task(env, ctx_server, slotId, jfilename, SERVER_TASK_TYPE_SLOT_SAVE, "Filename is required for slot save"); diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index a5aab2289..157cc7ac9 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10509"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10519"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10509-"} — call + * plus the resolved upstream commit, e.g. {@code "b10519-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10509"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10519"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10509"; + public static final String LLAMA_CPP_VERSION = "b10519"; // Constants holder — not instantiable. private LlamaCppVersion() {} diff --git a/llama/src/test/cpp/test_server.cpp b/llama/src/test/cpp/test_server.cpp index 5b8b9a115..158ab7c28 100644 --- a/llama/src/test/cpp/test_server.cpp +++ b/llama/src/test/cpp/test_server.cpp @@ -15,7 +15,8 @@ // - server_task_result_embd — oaicompat vs non-oaicompat shapes // - format_error_response — all 7 error types → correct HTTP code + type string // - server_task::need_embd/logits — routing helpers -// - server_task_result_metrics — slot count + token count fields +// - server_task_result_metrics — cumulative token/slot counters (/metrics) +// - server_task_result_slots — the /slots array payload // - server_task_result_slot_* — save/load/erase JSON shapes #include @@ -750,19 +751,25 @@ TEST(ServerTaskNTokens, PopulatedTokens_ReturnsCount) { } // ============================================================ -// server_task_result_metrics::to_json / ::to_metrics +// server_task_result_metrics::to_metrics / server_task_result_slots::to_json // Pure struct → JSON / Prometheus text; no model needed. // -// b10408 (upstream #26920) split this result in two: to_json() now serves -// /slots and returns the slot array verbatim, while the cumulative counters -// moved into an embedded server_metrics and are rendered as Prometheus -// exposition text by the new to_metrics() for /metrics. +// b10408 (upstream #26920) split the old single result in two: to_json() +// served /slots and returned the slot array verbatim, while the cumulative +// counters moved into an embedded server_metrics rendered as Prometheus +// exposition text by to_metrics() for /metrics. +// +// b10519 (upstream #27376, "allow access /metrics during sleep") split the +// *type* in two as well: server_task_result_metrics keeps only the counters +// (n_processing_slots / n_tasks_deferred / metrics) and its to_json() is now +// an unused empty object, while the /slots payload (n_idle_slots + +// slots_data) moved to the new server_task_result_slots, produced by the new +// SERVER_TASK_TYPE_SLOT_GET task. // ============================================================ namespace { server_task_result_metrics make_metrics() { server_task_result_metrics m; - m.n_idle_slots = 2; m.n_processing_slots = 1; m.n_tasks_deferred = 3; m.metrics.t_start = 1234567890LL; @@ -797,15 +804,32 @@ double prometheus_value(const std::string &text, const std::string &name) { } } // namespace -TEST(ServerTaskResultMetrics, ToJson_ReturnsSlotsArrayVerbatim) { - server_task_result_metrics m = make_metrics(); - m.slots_data = json::array({{{"id", 0}}, {{"id", 1}}}); +TEST(ServerTaskResultSlots, ToJson_ReturnsSlotsArrayVerbatim) { + server_task_result_slots m; + m.n_idle_slots = 2; + m.slots_data = json::array({json::object({{"id", 0}}), json::object({{"id", 1}})}); const json j = m.to_json(); ASSERT_TRUE(j.is_array()); EXPECT_EQ(j.size(), 2u); EXPECT_EQ(j.at(0).at("id").get(), 0); } +TEST(ServerTaskResultSlots, ToJson_EmptyByDefault) { + server_task_result_slots m; + const json j = m.to_json(); + ASSERT_TRUE(j.is_array()); + EXPECT_EQ(j.size(), 0u); + EXPECT_EQ(m.n_idle_slots, 0); +} + +TEST(ServerTaskResultMetrics, ToJson_UnusedEmptyObject) { + // /metrics renders Prometheus text via to_metrics(); to_json() is not used any more + // and returns an empty object since b10519. + server_task_result_metrics m = make_metrics(); + const json j = m.to_json(); + EXPECT_TRUE(j.empty()); +} + TEST(ServerTaskResultMetrics, ToMetrics_SlotGauges) { const std::string text = make_metrics().to_metrics(); EXPECT_DOUBLE_EQ(prometheus_value(text, "requests_processing"), 1.0); From 643e5dd035109d3b4b8c3dd63a5e0757af78433d Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:23:19 +0000 Subject: [PATCH 09/76] chore: upgrade llama.cpp from b10519 to b10532 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index e47d730c5..08f859048 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10519** +Current llama.cpp pinned version: **b10532** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10519 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10532 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10519`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10532`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10519`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10532`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index a77004af4..b571b5b7a 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10519](https://img.shields.io/badge/llama.cpp-%23b10519-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10519) +[![llama.cpp b10532](https://img.shields.io/badge/llama.cpp-%23b10532-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10532) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 8f72359f9..889f6f846 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -635,3 +635,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10507–b10509 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10509 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. This is the **last** tag at which patch `0007` applies unmodified — see the next row. | | b10509–b10519 | `tools/server/server-task.{h,cpp}` + `tools/server/server-context.{h,cpp}` + `tools/server/server-queue.{h,cpp}` (**#27376 "server: refactor sleep handling, allow access /metrics during sleep"**), `tools/server/server.cpp` + `tools/server/server-http.cpp` (**#26347: `/models` + `/v1/models` are no longer API-key-exempt**), `common/common.cpp` (**#27337: revert of the b10499 threadpool sharing when `n_threads` differ**), `ggml/include/ggml.h`, `tools/ui/**`, `src/models/granite-swa.cpp` | **Project source change required (two), plus a patch refresh.** (1) #27376 split `server_task_result_metrics` in two: `n_idle_slots` + `slots_data` moved out into a **new `server_task_result_slots`**, produced by a **new `SERVER_TASK_TYPE_SLOT_GET`** task, and `server_task_result_metrics::to_json()` now returns an empty object (`/metrics` renders Prometheus text via the byte-identical `to_metrics()`). Two project impacts, one silent and one hard: `jllama.cpp`'s `handleSlotAction` LIST arm posted `SERVER_TASK_TYPE_METRICS`, which still **compiles** at b10519 but would return `{}` instead of the slot array to `LlamaModel.getMetrics()` — re-pointed at `SERVER_TASK_TYPE_SLOT_GET` (upstream's `/slots` handler builds exactly that task, with no extra fields); and `test_server.cpp` was a **hard compile error** (`make_metrics()` set the removed `n_idle_slots`, `ToJson_ReturnsSlotsArrayVerbatim` set the removed `slots_data`) — the slots assertions moved to `server_task_result_slots` and two tests were added (`ServerTaskResultSlots.ToJson_EmptyByDefault`, `ServerTaskResultMetrics.ToJson_UnusedEmptyObject`), 485 → 487. `to_metrics()` is byte-identical and never read `n_idle_slots`, so every Prometheus assertion survived unchanged. (2) **Patch `0007` stopped applying here** — #26347 deleted the trailing `// public endpoint (no API key check)` comments on the two `/models` route registrations, which sit inside `0007`'s `@@ -258,47 +310,7 @@` removal block (`git apply` → "patch does not apply", `server.cpp:258`); refreshed by dropping that comment from those four lines (2 on the `-` side, 2 on the extracted `llama_server_register_common_routes()` `+` side, where it had also become factually wrong). `0001`/`0002`/`0003`/`0006`/`0008` applied unchanged. No CMake wiring change: no `tools/server` or `vendor` file was added, removed or renamed. **Behavioural note for consumers:** a `NativeServer` started with `--api-key` now returns 401 on `/models` and `/v1/models`; the project's own `RouterModeIntegrationTest` runs without an API key, so CI is unaffected. The new `server_routes` sleep-cache callback registers in the constructor and is inert here — `sleep_idle_seconds` defaults to `-1` (disabled) and nothing in the project sets it. | | b10509–b10519 | upstream verification (sandbox) | Patch `0007` **refreshed** (see above); all **6** patches then re-verified against a clean b10519 checkout — sequential `git apply --check` + `git apply` in filename order, zero fuzz — and the refreshed `0007` additionally re-checked to apply cleanly at **every** later endpoint of this walk up to b10618, so the refresh is a one-time fix, not a per-chunk one. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics against the patched b10519 tree. Full link + `ctest` deferred to the b10618 target. | +| b10519–b10532 | `common/speculative.cpp` (**#27404: avoid binding a reference to a null pointer**), `ggml/**` (backend-split scheduler race #26040, CUDA cuBLAS workspace #26574, Metal FA dequant #27390, Vulkan/OpenCL/Hexagon kernels), `src/llama-graph.cpp` (**#27392: V built as a view of K in `k_iswa build_attn`**), `convert_hf_to_gguf.py` | **No project-source change.** The only in-scope file is `common/speculative.cpp`, and the fix is inside an upstream-compiled TU — `common/speculative.h` (priority 3 on the CLAUDE.md review list) has a zero-line diff across the range, so nothing the project compiles against moved, and `grep -rn "common_speculative" src/main/cpp src/test/cpp` is empty. Nothing under `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt` changed, so no patch context shifted and `llama/CMakeLists.txt`'s `target_sources` list still matches upstream. The remainder is ggml backends and the llama graph builder, both entirely upstream-owned. | +| b10519–b10532 | upstream verification (sandbox) | All **6** patches (with the b10519-refreshed `0007`) re-verified against a clean b10532 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 879bab457..76a98f588 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10519 + GIT_TAG b10532 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 157cc7ac9..20a92a2e5 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10519"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10532"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10519-"} — call + * plus the resolved upstream commit, e.g. {@code "b10532-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10519"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10532"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10519"; + public static final String LLAMA_CPP_VERSION = "b10532"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 97d4af25fc0e8d20ae116a24758cdffa9f3b7cb8 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:25:56 +0000 Subject: [PATCH 10/76] chore: upgrade llama.cpp from b10532 to b10541 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 08f859048..f432a83f9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10532** +Current llama.cpp pinned version: **b10541** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10532 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10541 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10532`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10541`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10532`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10541`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index b571b5b7a..e61820878 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10532](https://img.shields.io/badge/llama.cpp-%23b10532-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10532) +[![llama.cpp b10541](https://img.shields.io/badge/llama.cpp-%23b10541-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10541) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 889f6f846..6328a51ed 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -637,3 +637,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10509–b10519 | upstream verification (sandbox) | Patch `0007` **refreshed** (see above); all **6** patches then re-verified against a clean b10519 checkout — sequential `git apply --check` + `git apply` in filename order, zero fuzz — and the refreshed `0007` additionally re-checked to apply cleanly at **every** later endpoint of this walk up to b10618, so the refresh is a one-time fix, not a per-chunk one. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics against the patched b10519 tree. Full link + `ctest` deferred to the b10618 target. | | b10519–b10532 | `common/speculative.cpp` (**#27404: avoid binding a reference to a null pointer**), `ggml/**` (backend-split scheduler race #26040, CUDA cuBLAS workspace #26574, Metal FA dequant #27390, Vulkan/OpenCL/Hexagon kernels), `src/llama-graph.cpp` (**#27392: V built as a view of K in `k_iswa build_attn`**), `convert_hf_to_gguf.py` | **No project-source change.** The only in-scope file is `common/speculative.cpp`, and the fix is inside an upstream-compiled TU — `common/speculative.h` (priority 3 on the CLAUDE.md review list) has a zero-line diff across the range, so nothing the project compiles against moved, and `grep -rn "common_speculative" src/main/cpp src/test/cpp` is empty. Nothing under `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt` changed, so no patch context shifted and `llama/CMakeLists.txt`'s `target_sources` list still matches upstream. The remainder is ggml backends and the llama graph builder, both entirely upstream-owned. | | b10519–b10532 | upstream verification (sandbox) | All **6** patches (with the b10519-refreshed `0007`) re-verified against a clean b10532 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10532–b10541 | `common/common.h` + `common/arg.cpp` + `tools/mtmd/{mtmd.h,mtmd.cpp,clip.h,clip.cpp}` + `tools/server/server-context.cpp` (**#23255 `--mmproj-device`: new `common_params::mmproj_device` (`ggml_backend_dev_t`) and a new `device` member at position 2 in `mtmd_context_params`/`clip_context_params`; the `MTMD_BACKEND_DEVICE` env lookup moves out of `clip.cpp` into `arg.cpp`**), `common/json-schema-to-grammar.cpp` (**#26939: unsupported regex patterns degrade to accept-any-string with a warning instead of throwing**), `tools/server/{server-models.h,server-models.cpp,server.cpp}` (**#27424: load-on-startup deferred into a new public `server_models::load_startup_models()` called from `llama_server()`'s router branch; reload no longer autoloads. #27416: `is_router_server` also excludes `--docker-repo`**), `common/speculative.cpp` (**`!dp.drafting` guard**) | **No project-source change.** The mid-struct `mtmd_context_params::device` insertion is the one positional-aggregate-init hazard in this range and the project does not trip it: `src/main/cpp/tts_engine.cpp` builds its params via `mtmd_context_params_default()` plus a named `.use_gpu` assignment, so `device` defaults to `nullptr` = the previous auto-GPU behaviour; `ggml_backend_dev_t` is already visible through `mtmd.h` → `llama.h` → `ggml-backend.h`, and every `common_params` in the project is default-constructed. `common/json-schema-to-grammar.h` is unchanged, so #26939 is semantic only — it does reach a public Java API (`jllama.cpp`'s `jsonSchemaToGrammarBytes` → `LlamaModel.jsonSchemaToGrammar`), where a schema with an unanchored `pattern` or a `\d`/`\w`/`\s` escape now yields a permissive `string` grammar plus a stderr warning instead of throwing `LlamaException`; no test moves, because `LlamaModelTest#testJsonSchemaToGrammar` asserts only `PRIMITIVE_RULES` output from three pattern-less string properties (zero `"pattern"` hits across `src/test/{java,cpp}`). `server_models` / `load_startup_models` / `docker_repo` are zero-hit greps in the project's C++ and Java. **Patch context:** `server.cpp` gains 13 lines (the `load_startup_models()` block) **below** patch `0007`'s route-table hunk, so only that patch's *last* hunk offset moves (10 → 23) — it still applies, verified by real `git apply` at both endpoints. **Build wiring:** no file added/removed/renamed under `tools/server`/`tools/mtmd`/`vendor`. | +| b10532–b10541 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10541 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 76a98f588..7d75e79a3 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10532 + GIT_TAG b10541 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 20a92a2e5..707a85d79 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10532"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10541"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10532-"} — call + * plus the resolved upstream commit, e.g. {@code "b10541-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10532"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10541"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10532"; + public static final String LLAMA_CPP_VERSION = "b10541"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 50b85c544986af1d7402245da853c50d5ed8492d Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:28:26 +0000 Subject: [PATCH 11/76] chore: upgrade llama.cpp from b10541 to b10545 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index f432a83f9..ebaa63084 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10541** +Current llama.cpp pinned version: **b10545** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10541 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10545 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10541`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10545`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10541`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10545`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index e61820878..3913e7651 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10541](https://img.shields.io/badge/llama.cpp-%23b10541-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10541) +[![llama.cpp b10545](https://img.shields.io/badge/llama.cpp-%23b10545-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10545) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 6328a51ed..ee93d07ce 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -639,3 +639,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10519–b10532 | upstream verification (sandbox) | All **6** patches (with the b10519-refreshed `0007`) re-verified against a clean b10532 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10532–b10541 | `common/common.h` + `common/arg.cpp` + `tools/mtmd/{mtmd.h,mtmd.cpp,clip.h,clip.cpp}` + `tools/server/server-context.cpp` (**#23255 `--mmproj-device`: new `common_params::mmproj_device` (`ggml_backend_dev_t`) and a new `device` member at position 2 in `mtmd_context_params`/`clip_context_params`; the `MTMD_BACKEND_DEVICE` env lookup moves out of `clip.cpp` into `arg.cpp`**), `common/json-schema-to-grammar.cpp` (**#26939: unsupported regex patterns degrade to accept-any-string with a warning instead of throwing**), `tools/server/{server-models.h,server-models.cpp,server.cpp}` (**#27424: load-on-startup deferred into a new public `server_models::load_startup_models()` called from `llama_server()`'s router branch; reload no longer autoloads. #27416: `is_router_server` also excludes `--docker-repo`**), `common/speculative.cpp` (**`!dp.drafting` guard**) | **No project-source change.** The mid-struct `mtmd_context_params::device` insertion is the one positional-aggregate-init hazard in this range and the project does not trip it: `src/main/cpp/tts_engine.cpp` builds its params via `mtmd_context_params_default()` plus a named `.use_gpu` assignment, so `device` defaults to `nullptr` = the previous auto-GPU behaviour; `ggml_backend_dev_t` is already visible through `mtmd.h` → `llama.h` → `ggml-backend.h`, and every `common_params` in the project is default-constructed. `common/json-schema-to-grammar.h` is unchanged, so #26939 is semantic only — it does reach a public Java API (`jllama.cpp`'s `jsonSchemaToGrammarBytes` → `LlamaModel.jsonSchemaToGrammar`), where a schema with an unanchored `pattern` or a `\d`/`\w`/`\s` escape now yields a permissive `string` grammar plus a stderr warning instead of throwing `LlamaException`; no test moves, because `LlamaModelTest#testJsonSchemaToGrammar` asserts only `PRIMITIVE_RULES` output from three pattern-less string properties (zero `"pattern"` hits across `src/test/{java,cpp}`). `server_models` / `load_startup_models` / `docker_repo` are zero-hit greps in the project's C++ and Java. **Patch context:** `server.cpp` gains 13 lines (the `load_startup_models()` block) **below** patch `0007`'s route-table hunk, so only that patch's *last* hunk offset moves (10 → 23) — it still applies, verified by real `git apply` at both endpoints. **Build wiring:** no file added/removed/renamed under `tools/server`/`tools/mtmd`/`vendor`. | | b10532–b10541 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10541 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10541–b10545 | `ggml/src/ggml-metal/**` (**#27450: clamp the K extent in the tensor-API mat-mat kernel when K is not a multiple of 32**), `ggml/src/ggml-opencl/**` (**#26476 q6_K flat mul_mat on Adreno A6x/A7x with older E031 compilers; #27339 norm local size**), `tools/ui/**` (**#27240: stores split refactor**) | **No project-source change.** Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt` changed — the whole range is ggml backend kernels plus the Svelte WebUI (which CI rebuilds from the pinned `GIT_TAG`, so it needs no per-bump source review). No patch context moved and `llama/CMakeLists.txt`'s `target_sources` list still matches upstream. The step's raw `git diff` is large (≈978 KiB) only because of the WebUI refactor; the reviewable code outside `tools/ui` is ≈17 KiB. | +| b10541–b10545 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10545 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 7d75e79a3..36c60a88a 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10541 + GIT_TAG b10545 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 707a85d79..34aba6a54 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10541"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10545"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10541-"} — call + * plus the resolved upstream commit, e.g. {@code "b10545-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10541"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10545"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10541"; + public static final String LLAMA_CPP_VERSION = "b10545"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 7db7badaac4effe57841d19ad79e76a1e9b60269 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:31:02 +0000 Subject: [PATCH 12/76] chore: upgrade llama.cpp from b10545 to b10549 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index ebaa63084..efd9e7dfc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10545** +Current llama.cpp pinned version: **b10549** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10545 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10549 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10545`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10549`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10545`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10549`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 3913e7651..9902d28f0 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10545](https://img.shields.io/badge/llama.cpp-%23b10545-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10545) +[![llama.cpp b10549](https://img.shields.io/badge/llama.cpp-%23b10549-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10549) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index ee93d07ce..5413bffdb 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -641,3 +641,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10532–b10541 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10541 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10541–b10545 | `ggml/src/ggml-metal/**` (**#27450: clamp the K extent in the tensor-API mat-mat kernel when K is not a multiple of 32**), `ggml/src/ggml-opencl/**` (**#26476 q6_K flat mul_mat on Adreno A6x/A7x with older E031 compilers; #27339 norm local size**), `tools/ui/**` (**#27240: stores split refactor**) | **No project-source change.** Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt` changed — the whole range is ggml backend kernels plus the Svelte WebUI (which CI rebuilds from the pinned `GIT_TAG`, so it needs no per-bump source review). No patch context moved and `llama/CMakeLists.txt`'s `target_sources` list still matches upstream. The step's raw `git diff` is large (≈978 KiB) only because of the WebUI refactor; the reviewable code outside `tools/ui` is ≈17 KiB. | | b10541–b10545 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10545 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10545–b10549 | `ggml/**` (**#27345: `ggml_rope_set_offset` support added to the OpenCL / SYCL / WebGPU / Hexagon backends; #26993: tensor split enabled for LFM2/LFM2MOE**), `docs/**`, `.github/**` (**#27414: cmake-package check moved to a shell script**) | **No project-source change.** Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any project-consumed `CMakeLists.txt` changed. `ggml_rope_set_offset` was already added to `ggml.h` at b10509 and remains unreferenced by the project (`grep -rn "ggml_rope" src/main/cpp src/test/cpp` is empty); this range only implements it in the non-CPU backends. No patch context moved. | +| b10545–b10549 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10549 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 36c60a88a..06fcee1df 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10545 + GIT_TAG b10549 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 34aba6a54..705c5b2db 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10545"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10549"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10545-"} — call + * plus the resolved upstream commit, e.g. {@code "b10549-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10545"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10549"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10545"; + public static final String LLAMA_CPP_VERSION = "b10549"; // Constants holder — not instantiable. private LlamaCppVersion() {} From c8bf06d9d5c609d6d4e3cb648ea90a24c0ed64b8 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:33:55 +0000 Subject: [PATCH 13/76] chore: upgrade llama.cpp from b10549 to b10566 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index efd9e7dfc..b23d6ef57 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10549** +Current llama.cpp pinned version: **b10566** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10549 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10566 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10549`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10566`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10549`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10566`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 9902d28f0..8673be871 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10549](https://img.shields.io/badge/llama.cpp-%23b10549-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10549) +[![llama.cpp b10566](https://img.shields.io/badge/llama.cpp-%23b10566-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10566) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 5413bffdb..2d9be2dbe 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -643,3 +643,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10541–b10545 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10545 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10545–b10549 | `ggml/**` (**#27345: `ggml_rope_set_offset` support added to the OpenCL / SYCL / WebGPU / Hexagon backends; #26993: tensor split enabled for LFM2/LFM2MOE**), `docs/**`, `.github/**` (**#27414: cmake-package check moved to a shell script**) | **No project-source change.** Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any project-consumed `CMakeLists.txt` changed. `ggml_rope_set_offset` was already added to `ggml.h` at b10509 and remains unreferenced by the project (`grep -rn "ggml_rope" src/main/cpp src/test/cpp` is empty); this range only implements it in the non-CPU backends. No patch context moved. | | b10545–b10549 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10549 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10549–b10566 | `CMakeLists.txt` (**#27498: llama.cpp semver bumped to 0.2.0; ggml/1597 bumps ggml to 0.21.0**), `ggml/src/ggml-{sycl,opencl,cpu}/**` (SYCL Q2_K/Q5_K ESIMD kernels reverted then re-landed, KleidiAI SME2 F32 GEMV, Adreno A7X lm_head workaround), `scripts/release.sh` + `.github/**` (release tooling), `tools/ui/**` (**#27241: settings-navigation cleanup**), `tests/**` | **No project-source change.** The only in-scope file is the root `CMakeLists.txt`, and only its version numbers moved. That is invisible here: `llama/CMakeLists.txt` already substitutes `"0"` placeholders for `LLAMA_VERSION_BASE` / `LLAMA_VERSION_MAJOR` (upstream `set()`s them in FetchContent's child scope, out of the project's reach) and uses them solely as mtmd's `VERSION`/`SOVERSION`, which are no-ops for a static library under `BUILD_SHARED_LIBS OFF`. The Java-visible pin is `LlamaCppVersion.LLAMA_CPP_VERSION` (the `b` tag), not the semver. Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/` or `vendor/` changed, so no patch context moved and the `target_sources` list still matches upstream. | +| b10549–b10566 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10566 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 06fcee1df..0d3dd9224 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10549 + GIT_TAG b10566 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 705c5b2db..fb6ec4933 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10549"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10566"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10549-"} — call + * plus the resolved upstream commit, e.g. {@code "b10566-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10549"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10566"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10549"; + public static final String LLAMA_CPP_VERSION = "b10566"; // Constants holder — not instantiable. private LlamaCppVersion() {} From a5fe54e494b69f41e155f2e7a11e8769a326a9c4 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:36:22 +0000 Subject: [PATCH 14/76] chore: upgrade llama.cpp from b10566 to b10568 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index b23d6ef57..7ec2086c6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10566** +Current llama.cpp pinned version: **b10568** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10566 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10568 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10566`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10568`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10566`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10568`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 8673be871..a932fcae3 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10566](https://img.shields.io/badge/llama.cpp-%23b10566-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10566) +[![llama.cpp b10568](https://img.shields.io/badge/llama.cpp-%23b10568-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10568) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 2d9be2dbe..a53d96cae 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -645,3 +645,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10545–b10549 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10549 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10549–b10566 | `CMakeLists.txt` (**#27498: llama.cpp semver bumped to 0.2.0; ggml/1597 bumps ggml to 0.21.0**), `ggml/src/ggml-{sycl,opencl,cpu}/**` (SYCL Q2_K/Q5_K ESIMD kernels reverted then re-landed, KleidiAI SME2 F32 GEMV, Adreno A7X lm_head workaround), `scripts/release.sh` + `.github/**` (release tooling), `tools/ui/**` (**#27241: settings-navigation cleanup**), `tests/**` | **No project-source change.** The only in-scope file is the root `CMakeLists.txt`, and only its version numbers moved. That is invisible here: `llama/CMakeLists.txt` already substitutes `"0"` placeholders for `LLAMA_VERSION_BASE` / `LLAMA_VERSION_MAJOR` (upstream `set()`s them in FetchContent's child scope, out of the project's reach) and uses them solely as mtmd's `VERSION`/`SOVERSION`, which are no-ops for a static library under `BUILD_SHARED_LIBS OFF`. The Java-visible pin is `LlamaCppVersion.LLAMA_CPP_VERSION` (the `b` tag), not the semver. Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/` or `vendor/` changed, so no patch context moved and the `target_sources` list still matches upstream. | | b10549–b10566 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10566 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10566–b10568 | `src/models/**` (**#27382: the model graph builders adopt `ggml_rope_set_offset()`**), `.github/**` (**#27503: `ccache-clear` runs last in the release jobs**) | **No project-source change.** Zero lines changed under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt`. `src/models/**` is llama.cpp's internal graph-builder layer — compiled into the `llama` static library the project links, but it exposes no header the project includes, and `grep -rn "ggml_rope" src/main/cpp src/test/cpp` is empty. No patch context moved. | +| b10566–b10568 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10568 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 0d3dd9224..70ec79320 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10566 + GIT_TAG b10568 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index fb6ec4933..ad7f36498 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10566"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10568"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10566-"} — call + * plus the resolved upstream commit, e.g. {@code "b10568-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10566"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10568"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10566"; + public static final String LLAMA_CPP_VERSION = "b10568"; // Constants holder — not instantiable. private LlamaCppVersion() {} From e4a2ffaab8903ac758da2fc8e92444c19e0722d5 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:38:42 +0000 Subject: [PATCH 15/76] chore: upgrade llama.cpp from b10568 to b10569 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 7ec2086c6..91bbfc0aa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10568** +Current llama.cpp pinned version: **b10569** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10568 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10569 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10568`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10569`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10568`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10569`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index a932fcae3..d57b0c7c9 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10568](https://img.shields.io/badge/llama.cpp-%23b10568-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10568) +[![llama.cpp b10569](https://img.shields.io/badge/llama.cpp-%23b10569-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10569) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index a53d96cae..045075203 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -647,3 +647,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10549–b10566 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10566 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10566–b10568 | `src/models/**` (**#27382: the model graph builders adopt `ggml_rope_set_offset()`**), `.github/**` (**#27503: `ccache-clear` runs last in the release jobs**) | **No project-source change.** Zero lines changed under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt`. `src/models/**` is llama.cpp's internal graph-builder layer — compiled into the `llama` static library the project links, but it exposes no header the project includes, and `grep -rn "ggml_rope" src/main/cpp src/test/cpp` is empty. No patch context moved. | | b10566–b10568 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10568 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10568–b10569 | `src/models/dots3note.cpp` (new) + `src/llama-{model,arch,vocab}.*` + `convert_hf_to_gguf.py` + `gguf-py/**` (**#27060: dots3-note model support**) | **No project-source change.** A new model architecture, entirely inside llama.cpp's own model layer: the new TU is added to upstream's own `src/CMakeLists.txt` (which the project consumes verbatim through `FetchContent_MakeAvailable`, so there is nothing to mirror into `llama/CMakeLists.txt` — unlike the `tools/server/*.cpp` list, which the project enumerates itself). Zero lines changed under `common/`, `tools/server/`, `tools/mtmd/`, `include/` or `vendor/`; no patch context moved. | +| b10568–b10569 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10569 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 70ec79320..66ac19647 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10568 + GIT_TAG b10569 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index ad7f36498..44da668bf 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10568"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10569"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10568-"} — call + * plus the resolved upstream commit, e.g. {@code "b10569-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10568"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10569"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10568"; + public static final String LLAMA_CPP_VERSION = "b10569"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 2c381b6af9883f1753d15769211486de5fe22411 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:41:10 +0000 Subject: [PATCH 16/76] chore: upgrade llama.cpp from b10569 to b10578 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 91bbfc0aa..f89f7c99a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10569** +Current llama.cpp pinned version: **b10578** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10569 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10578 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10569`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10578`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10569`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10578`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index d57b0c7c9..480809540 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10569](https://img.shields.io/badge/llama.cpp-%23b10569-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10569) +[![llama.cpp b10578](https://img.shields.io/badge/llama.cpp-%23b10578-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10578) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 045075203..d9a54ebda 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -649,3 +649,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10566–b10568 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10568 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10568–b10569 | `src/models/dots3note.cpp` (new) + `src/llama-{model,arch,vocab}.*` + `convert_hf_to_gguf.py` + `gguf-py/**` (**#27060: dots3-note model support**) | **No project-source change.** A new model architecture, entirely inside llama.cpp's own model layer: the new TU is added to upstream's own `src/CMakeLists.txt` (which the project consumes verbatim through `FetchContent_MakeAvailable`, so there is nothing to mirror into `llama/CMakeLists.txt` — unlike the `tools/server/*.cpp` list, which the project enumerates itself). Zero lines changed under `common/`, `tools/server/`, `tools/mtmd/`, `include/` or `vendor/`; no patch context moved. | | b10568–b10569 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10569 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10569–b10578 | `tools/mtmd/mtmd-helper.{h,cpp}` (**#27520: webp decoded as a single ffmpeg frame; the header change is one comment line**), `common/speculative.cpp` (**#27400: draft-MTP fixed with embeddings — draft params now reset `embedding`/`pooling_type`**), `ggml/**` (**#24575 row-level `concat`; #26431 gpt-oss MoE bias fused into the OpenCL epilogue; #27490 SYCL Q2_K kernels re-landed**), `README.md` badges, `docs/**` | **No project-source change.** `mtmd-helper.h` changed by exactly one comment line, so `mtmd_helper::gen_audio`, `mtmd_helper_gen_audio_inp` and `mtmd_helper_bitmap_init_from_file` — the surface `src/main/cpp/tts_engine.cpp` uses directly, and the reason `mtmd-helper.h` is on the CLAUDE.md priority review list — are untouched. The new webp path is reachable through upstream's own `server-common.cpp` bitmap loader and is a functional gain requiring `ffmpeg`/`ffprobe` on `PATH` **at runtime only** (no new build dependency). One cosmetic upstream wart: the new `is_webp_file()` sits outside the `MTMD_VIDEO` guard while its only caller is inside it, so an `MTMD_VIDEO=OFF` build emits `-Wunused-function`; harmless here — it is compiled in upstream's own `mtmd` target and `LLAMA_FATAL_WARNINGS` defaults OFF and is never set by this project. The speculative fix is behaviour-only: `common/speculative.h` is unchanged and the only callers are upstream's `server-context.cpp`. **Patches:** none of the five patched files changed — zero context shift. | +| b10569–b10578 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10578 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 66ac19647..a8c68f69f 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10569 + GIT_TAG b10578 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 44da668bf..260f7992c 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10569"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10578"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10569-"} — call + * plus the resolved upstream commit, e.g. {@code "b10578-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10569"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10578"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10569"; + public static final String LLAMA_CPP_VERSION = "b10578"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 7657ecd2af4cea2a1c85b4c8ecbcff1f84d38cd2 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:45:36 +0000 Subject: [PATCH 17/76] chore: upgrade llama.cpp from b10578 to b10584 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index f89f7c99a..7153f5309 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10578** +Current llama.cpp pinned version: **b10584** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10578 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10584 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10578`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10584`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1357,7 +1357,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10578`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10584`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 480809540..e5aacd3d0 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10578](https://img.shields.io/badge/llama.cpp-%23b10578-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10578) +[![llama.cpp b10584](https://img.shields.io/badge/llama.cpp-%23b10584-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10584) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index d9a54ebda..e01ef1377 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -651,3 +651,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10568–b10569 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10569 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10569–b10578 | `tools/mtmd/mtmd-helper.{h,cpp}` (**#27520: webp decoded as a single ffmpeg frame; the header change is one comment line**), `common/speculative.cpp` (**#27400: draft-MTP fixed with embeddings — draft params now reset `embedding`/`pooling_type`**), `ggml/**` (**#24575 row-level `concat`; #26431 gpt-oss MoE bias fused into the OpenCL epilogue; #27490 SYCL Q2_K kernels re-landed**), `README.md` badges, `docs/**` | **No project-source change.** `mtmd-helper.h` changed by exactly one comment line, so `mtmd_helper::gen_audio`, `mtmd_helper_gen_audio_inp` and `mtmd_helper_bitmap_init_from_file` — the surface `src/main/cpp/tts_engine.cpp` uses directly, and the reason `mtmd-helper.h` is on the CLAUDE.md priority review list — are untouched. The new webp path is reachable through upstream's own `server-common.cpp` bitmap loader and is a functional gain requiring `ffmpeg`/`ffprobe` on `PATH` **at runtime only** (no new build dependency). One cosmetic upstream wart: the new `is_webp_file()` sits outside the `MTMD_VIDEO` guard while its only caller is inside it, so an `MTMD_VIDEO=OFF` build emits `-Wunused-function`; harmless here — it is compiled in upstream's own `mtmd` target and `LLAMA_FATAL_WARNINGS` defaults OFF and is never set by this project. The speculative fix is behaviour-only: `common/speculative.h` is unchanged and the only callers are upstream's `server-context.cpp`. **Patches:** none of the five patched files changed — zero context shift. | | b10569–b10578 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10578 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10578–b10584 | `common/fit.{h,cpp}` + `common/common.cpp` + `common/speculative.cpp` + `tools/{fit-params,llama-bench}` (**draft/MTP context is now fitted together with the target model — new public `struct common_fit_extra_model` and a new 7th `const common_fit_extra_model * extra` parameter on `common_fit_params()`; `common_speculative_init_result` pins the draft `cparams.n_ctx = llama_n_ctx(ctx_tgt)`**), `tools/server/server-context.cpp` (**−56 lines: the hand-rolled draft-model VRAM pre-reservation block is deleted, superseded by the fitter**), `common/arg.cpp` (**`--conversation` dropped `LLAMA_EXAMPLE_CLI` from its `set_examples()`**), `tools/mtmd/{CMakeLists.txt,clip*.h,clip.cpp,models/dots3note.cpp,models/models.h,mtmd-audio.{h,cpp},mtmd.cpp}` (**new `dots3note` multimodal model: two `PROJECTOR_TYPE_DOTS3NOTE_{V,A}` values, a vision MoE FFN via a new `clip_graph::build_moe_ffn()`, and `mtmd_audio_preprocessor_dots3note`**) | **No project-source change; no patch refresh.** `common_fit_params()`'s signature break is upstream-only: `common/fit.h` is not in the project's include graph, `grep -rn "common_fit_params\|common_fit_extra_model\|fit_params\|common_get_device_memory_data" src/main/cpp src/test/cpp` returns zero hits, and the two other callers (`tools/fit-params`, `tools/llama-bench`) are not built here (`LLAMA_BUILD_TOOLS` is OFF under FetchContent; only `tools/mtmd` and an explicit `tools/server/*.cpp` list are added back). The `server-context.cpp` deletion sits directly **above** patch `0002`'s hunk, whose anchor lines (blank / `// attach a progress callback` / `{`) are byte-identical, so that hunk only *shifts* (offset −53 → −108) and still applies cleanly. `--conversation` never reached the project's parser (`jllama.cpp` parses with `LLAMA_EXAMPLE_SERVER`, in neither the old nor the new example set) and patch `0001`'s `arg.cpp` hunks are ~650 lines away. All mtmd edits are internal — **`mtmd.h` and `mtmd-helper.h` are unchanged**, so `tts_engine.cpp`'s `mtmd_helper::gen_audio` surface is untouched, and `models/dots3note.cpp` enters through upstream's own `tools/mtmd/CMakeLists.txt`. The only reachable behaviour delta is a more accurate VRAM budget when `--fit` runs together with a draft model. | +| b10578–b10584 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10584 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index a8c68f69f..efca0ae41 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10578 + GIT_TAG b10584 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 260f7992c..a7440afdb 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10578"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10584"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10578-"} — call + * plus the resolved upstream commit, e.g. {@code "b10584-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10578"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10584"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10578"; + public static final String LLAMA_CPP_VERSION = "b10584"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 8d3f6416644c17d4aa52f6146286f18db5963559 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:49:25 +0000 Subject: [PATCH 18/76] chore: upgrade llama.cpp from b10584 to b10585 Upstream #27511 replaces the server json alias (nlohmann::ordered_json) with the new common_json pimpl wrapper. Migrate the project C++ to it and close the two traps that compile silently: a common_json still binds to a const nlohmann::json parameter through operator std::string() (turning require_json_field into a runtime type_error inside handleInfill), and an unscoped enum binds to the bool ctor, so vocab_type was serialised as true/false. Both are now covered by tests. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 24 ++++++++++------- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- llama/src/main/cpp/jllama.cpp | 9 ++++--- llama/src/main/cpp/jni_helpers.hpp | 10 ++++++- llama/src/main/cpp/json_helpers.hpp | 5 ++-- llama/src/main/cpp/utils.hpp | 1 - .../llama/value/LlamaCppVersion.java | 8 +++--- llama/src/test/cpp/test_jni_helpers.cpp | 19 +++++++++++++ llama/src/test/cpp/test_json_helpers.cpp | 27 ++++++++++++++++++- llama/src/test/cpp/test_server.cpp | 12 ++++----- llama/src/test/cpp/test_utils.cpp | 14 +++++----- .../net/ladenthin/llama/LlamaModelTest.java | 7 +++++ 14 files changed, 105 insertions(+), 37 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 7153f5309..09fa2bc79 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10584** +Current llama.cpp pinned version: **b10585** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10584 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10585 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10584`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10585`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1128,7 +1128,11 @@ If the local check passes (`BUILD SUCCESS`), the `mvn package` job in - `utils.hpp` — Helper utilities (format helpers, argv stripping, token-piece serialisation). - `json_helpers.hpp` — Pure JSON transformation helpers (no JNI, no llama state). Independently unit-testable. - `jni_helpers.hpp` — JNI bridge helpers (handle management + server orchestration). Includes `json_helpers.hpp`. -- Uses `nlohmann/json` for JSON deserialization of parameters. +- **The `json` alias is upstream's `common_json`, not `nlohmann::ordered_json` (since llama.cpp b10585, upstream #27511).** `tools/server/server-common.h` now says `using json = common_json;` — a deliberately small pimpl wrapper (`common/json.{h,cpp}`, compiled into `llama-common`) around the vendored nlohmann copy. Two traps this cost the project once, both of which **compile silently**: + 1. **An unscoped enum becomes a JSON boolean.** `common_json_value`'s integral constructor template is `std::is_integral`-gated, which excludes enums, so an enum binds to `common_json_value(bool)`. Always `static_cast(...)` an enum before putting it in JSON — `jllama.cpp`'s two `"vocab_type"` sites do, and `test_json_helpers.cpp`'s `CommonJsonEnumTrap` tests plus `LlamaModelTest`'s `isIntegralNumber()` assertion guard it. + 2. **`common_json` converts to `std::string` implicitly**, so it binds happily to a `const nlohmann::json &` parameter (via nlohmann's string-constructible converting constructor) and then throws `json::type_error 302` at runtime. Never declare a project helper as taking `nlohmann::json` when callers pass the `json` alias — `require_json_field_impl` is a template for exactly this reason. + Other differences to know: no `get_ref`/`array_t`/`type_name()`; a braced list in *value* position does not build an array (write `json::array({...})`); `at(key)` needs an explicit `.get()`; errors are `common_json_error`; and `get()` is limited to the types explicitly specialised in `common/json.cpp`. `log_helpers.hpp` and `train_engine.cpp` keep their own `nlohmann::json` alias — they never touch the server's `json`. +- Uses `nlohmann/json` for JSON deserialization of parameters in the two files named above; everything on the server path uses `common_json`. - The upstream server library (`server-context.cpp`, `server-queue.cpp`, `server-task.cpp`, `server-schema.cpp`, `server-models.cpp`, and — since b9829 — `server-stream.cpp`) is compiled directly into `jllama` via CMake — there is no hand-ported `server.hpp` fork. **`server-stream.cpp` is mandatory, not optional:** it defines the resumable-streaming SSE replay buffer (`g_stream_sessions`, `stream_session_attach_pipe`, `stream_aware_should_stop`, `stream_conv_id_from_headers`, the `stream_pipe_*` types) that `server-context.cpp` / `server-http.cpp` / `server-models.cpp` now `#include "server-stream.h"` and call, so omitting it fails the link with undefined references. It is platform-neutral (threads + std mutex/condvar, no `subprocess.h`/`posix_spawn_*`), so it builds on Android too and sits outside the `server-models.cpp` Android guard. `jllama` wires its own JNI routes and never calls `g_stream_sessions.start_gc()` (only the excluded standalone `server.cpp` `main()` does), so its GC thread stays dormant. **Phase 2:** the upstream HTTP transport (`tools/server/server-http.cpp`) and its `cpp-httplib` backend (`vendor/cpp-httplib/httplib.cpp`) are now compiled into `jllama` too, so the OpenAI-compatible server can be driven natively from JNI *inside* `libjllama` — no separate `llama-server` executable (a JNI shared library loads anywhere a JVM runs, which a standalone binary does not). `server-http.cpp` does `#include "ui.h"` (the WebUI asset table that `tools/ui`/`llama-ui` normally generates); since the Svelte WebUI is not shipped, `src/main/cpp/webui_stub/ui.h` supplies the upstream **empty-asset** interface and leaves `LLAMA_UI_HAS_ASSETS` undefined (all static-asset-serving blocks compile out). `` already resolves via `llama-common`'s `vendor/` include dir (same nlohmann/json 3.12.0 as the FetchContent copy). No SSL: `CPPHTTPLIB_OPENSSL_SUPPORT` is left undefined (plain-HTTP; bind localhost / front with a TLS proxy). **`server.cpp`, `server-tools.cpp` and `server-mcp.cpp` are now compiled in too** (on non-Android — they pull in `subprocess.h`/`posix_spawn_*`, so they share `server-models.cpp`'s Android guard): b9870 exposes `server.cpp`'s entry as `int llama_server(int, char**)` (no `main` in the file), and `patches/0006` makes it embeddable (no process signal handlers, forwarded-argv parse, out-of-band shutdown). **`server-mcp.cpp` is new in b10154** (upstream MCP-server support): both `server.cpp` (`llama_server`'s `mcp_mgr` lifecycle) and `server-tools.cpp` (`tools.setup(..., mcp_mgr)` / `server_mcp::call_tool`) reference `server_mcp`, so it **must** be in the `target_sources` list or the link fails with undefined `server_mcp::{start,shutdown,call_tool,list_tools,~server_mcp}` — **latent on Linux** (a shared object tolerates undefined symbols) but a **hard link error on macOS/ld64 and Windows/MSVC**. It is compiled only into `jllama`, not `jllama_test` (which links neither `server.cpp` nor `server-tools.cpp`). The `NativeServer` JNI bridge (`src/main/cpp/native_server.cpp`) calls `llama_server` on a worker thread, so the **full** upstream server — WebUI and all — runs inside `libjllama`. See "Two server modes" below. ### Two server modes (`OpenAiCompatServer` vs `NativeServer`) @@ -1143,7 +1147,7 @@ The library exposes **two** ways to serve a model over HTTP, on two different tr The project C++ helpers follow a strict semantic split: **`json_helpers.hpp`** — Pure data transforms. -- Input: `nlohmann::json`, `server_task_result_ptr`, plain C++ types. +- Input: the `json` alias (upstream `common_json` since b10585), `server_task_result_ptr`, plain C++ types. - Output: `json`, `std::vector`, `std::optional`, plain C++ types. - Zero JNI calls (`JNIEnv*` never appears). - Zero llama state (`llama_context*`, `llama_vocab*`, `server_context*` never appear). @@ -1173,7 +1177,7 @@ Functions: `log_level_name`, `format_log_as_json`. worker thread, cached `vocab`, saved `params`, and a `readers` map for streaming tasks. - `get_jllama_context_impl` — reads Java `ctx` handle, returns the `jllama_context*` wrapper. Does NOT throw on zero handle (valid no-op for destructor-style calls). -- `require_json_field_impl` — throws `" is required"` if key is absent. +- `require_json_field_impl` — throws `" is required"` if key is absent. **Templated on the JSON type on purpose**: a plain `const nlohmann::json &` parameter still accepts a `common_json` (through its `operator std::string()`) and turns the presence check into a runtime `type_error 302`. - `jint_array_to_tokens_impl` — reads a Java `int[]` into `std::vector`. *Layer B* (requires upstream server headers in the TU before `jni_helpers.hpp`): orchestration. @@ -1348,16 +1352,16 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" |------|-------|-------| | `src/test/cpp/test_utils.cpp` | 162 | Upstream helpers: `server_tokens`, `server_grammar_trigger`, `gen_tool_call_id`, `json_value`, `json_get_nested_values`, UTF-8 helpers, `format_response_rerank`, `format_embeddings_response_oaicompat`, `oaicompat_completion_params_parse`, `oaicompat_chat_params_parse`, `are_lora_equal`, `strip_flag_from_argv`, `token_piece_value`, `json_is_array_and_contains_numbers`, `format_oai_sse`, `format_oai_resp_sse`, `format_anthropic_sse`, `parse_lora_request` | | `src/test/cpp/test_server.cpp` | 206 | Upstream result types: `server_slot_stats` (the `timings` JSON payload; replaced `result_timings` in b10408), `task_params::to_json()` (incl. `dry_sequence_breakers`, `preserved_tokens`, `timings_per_token`), `completion_token_output`, `server_task_result_cmpl_partial` (non-oaicompat + `to_json_oaicompat` + logprobs + `to_json_oaicompat_chat` + `to_json_anthropic` + dispatcher), `server_task_result_cmpl_final` (non-oaicompat + `to_json_oaicompat` + `to_json_oaicompat_chat` + `to_json_oaicompat_chat_stream` + `to_json_anthropic` + `to_json_anthropic_stream` + tool_calls + dispatcher), `server_task_result_embd`, `server_task_result_rerank`, `server_task_result_metrics` (`to_metrics()` = the `/metrics` Prometheus exposition text; its `to_json()` is an unused empty object since b10519), `server_task_result_slots` (`to_json()` = the `/slots` array, fed by the b10519 `SERVER_TASK_TYPE_SLOT_GET` task), `server_task_result_slot_save_load`, `server_task_result_slot_erase`, `server_task_result_apply_lora`, `server_task_result_get_lora`, `server_task_result_error`, `format_error_response`, `server_task::need_sampling()`, `server_task::n_tokens()`, `server_schema::eval_llama_cmpl_schema()` (parsing pipeline + grammar routing + error paths + per-request `dry_*` and `sse_ping_interval` field round-trips incl. hard-limit + server-default inheritance), `response_fields` projection | -| `src/test/cpp/test_json_helpers.cpp` | 50 | All functions in `json_helpers.hpp`: `get_result_error_message`, `results_to_json`, `rerank_results_to_json` (incl. missing/out-of-range `index` rejection), `parse_encoding_format`, `extract_embedding_prompt`, `is_infill_request`, `parse_slot_prompt_similarity`, `parse_positive_int_config`, `wrap_stream_chunk` | +| `src/test/cpp/test_json_helpers.cpp` | 52 | All functions in `json_helpers.hpp`: `get_result_error_message`, `results_to_json`, `rerank_results_to_json` (incl. missing/out-of-range `index` rejection), `parse_encoding_format`, `extract_embedding_prompt`, `is_infill_request`, `parse_slot_prompt_similarity`, `parse_positive_int_config`, `wrap_stream_chunk` | | `src/test/cpp/test_log_helpers.cpp` | 13 | All functions in `log_helpers.hpp`: `log_level_name`, `format_log_as_json` | -| `src/test/cpp/test_jni_helpers.cpp` | 54 | All functions in `jni_helpers.hpp` using a zero-filled `JNINativeInterface_` mock (incl. the `utf8_to_jstring_impl` byte-array string path: emoji byte-preservation, truncated-UTF-8 replace-not-throw) | +| `src/test/cpp/test_jni_helpers.cpp` | 56 | All functions in `jni_helpers.hpp` using a zero-filled `JNINativeInterface_` mock (incl. the `utf8_to_jstring_impl` byte-array string path: emoji byte-preservation, truncated-UTF-8 replace-not-throw) | | `src/test/cpp/test_tts_wav.cpp` | 2 | The in-memory WAV writer `pcm_to_wav16_bytes` in `tts_wav.hpp` (WAV header/payload + little-endian clamping) — our own code, not upstream. The Qwen3-TTS pipeline it pairs with (`mtmd_helper::gen_audio`) is entirely upstream-owned (no project-side DSP to unit-test here) and covered end-to-end by the Java `TtsIntegrationTest`. | -**Current total: 487 tests (all passing).** +**Current total: 491 tests (all passing).** #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10584`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10585`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index e5aacd3d0..e4e97f72f 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10584](https://img.shields.io/badge/llama.cpp-%23b10584-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10584) +[![llama.cpp b10585](https://img.shields.io/badge/llama.cpp-%23b10585-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10585) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index e01ef1377..6e34ebf2b 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -653,3 +653,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10569–b10578 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10578 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10578–b10584 | `common/fit.{h,cpp}` + `common/common.cpp` + `common/speculative.cpp` + `tools/{fit-params,llama-bench}` (**draft/MTP context is now fitted together with the target model — new public `struct common_fit_extra_model` and a new 7th `const common_fit_extra_model * extra` parameter on `common_fit_params()`; `common_speculative_init_result` pins the draft `cparams.n_ctx = llama_n_ctx(ctx_tgt)`**), `tools/server/server-context.cpp` (**−56 lines: the hand-rolled draft-model VRAM pre-reservation block is deleted, superseded by the fitter**), `common/arg.cpp` (**`--conversation` dropped `LLAMA_EXAMPLE_CLI` from its `set_examples()`**), `tools/mtmd/{CMakeLists.txt,clip*.h,clip.cpp,models/dots3note.cpp,models/models.h,mtmd-audio.{h,cpp},mtmd.cpp}` (**new `dots3note` multimodal model: two `PROJECTOR_TYPE_DOTS3NOTE_{V,A}` values, a vision MoE FFN via a new `clip_graph::build_moe_ffn()`, and `mtmd_audio_preprocessor_dots3note`**) | **No project-source change; no patch refresh.** `common_fit_params()`'s signature break is upstream-only: `common/fit.h` is not in the project's include graph, `grep -rn "common_fit_params\|common_fit_extra_model\|fit_params\|common_get_device_memory_data" src/main/cpp src/test/cpp` returns zero hits, and the two other callers (`tools/fit-params`, `tools/llama-bench`) are not built here (`LLAMA_BUILD_TOOLS` is OFF under FetchContent; only `tools/mtmd` and an explicit `tools/server/*.cpp` list are added back). The `server-context.cpp` deletion sits directly **above** patch `0002`'s hunk, whose anchor lines (blank / `// attach a progress callback` / `{`) are byte-identical, so that hunk only *shifts* (offset −53 → −108) and still applies cleanly. `--conversation` never reached the project's parser (`jllama.cpp` parses with `LLAMA_EXAMPLE_SERVER`, in neither the old nor the new example set) and patch `0001`'s `arg.cpp` hunks are ~650 lines away. All mtmd edits are internal — **`mtmd.h` and `mtmd-helper.h` are unchanged**, so `tts_engine.cpp`'s `mtmd_helper::gen_audio` surface is untouched, and `models/dots3note.cpp` enters through upstream's own `tools/mtmd/CMakeLists.txt`. The only reachable behaviour delta is a more accurate VRAM budget when `--fit` runs together with a draft model. | | b10578–b10584 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10584 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10584–b10585 | **NEW** `common/json.{h,cpp}` + `common/CMakeLists.txt`, `common/chat.h`, `common/chat-auto-parser.h`, `common/chat-peg-parser.h`, `common/json-schema-to-grammar.h`, `common/peg-parser.h`, `common/arg.cpp`, `common/{chat,download,hf-cache,jinja/*,json-schema-to-grammar,peg-parser}.cpp`, `tools/server/{server-common.h,server-common.cpp,server-chat.h,server-chat.cpp,server-context.h,server-context.cpp,server-task.h,server-task.cpp,server-schema.cpp,server-models.cpp,server-tools.cpp}` (**#27511 "common: add json.h abstraction"**) | **BREAKING — the largest project impact of the whole b10456 → b10618 walk.** Upstream introduces the pimpl wrapper class `common_json` and flips `using json = nlohmann::ordered_json` → `using json = common_json` in `server-common.h`, so **every** project TU that includes an upstream server header changes JSON type. `common_json` is a deliberately closed API: no `get_ref`, no `array_t`, no `type_name()`, a braced list in *value* position does not build an array, `at(key)` no longer implicitly converts (needs `.get()`), `get()` is limited to the specialisations in `common/json.cpp`, and errors are `common_json_error`. Project changes, in the order they were found: **(1)** `utils.hpp`'s `str_to_bytes` used `bytes.get_ref().reserve(...)` — dropped (a pure optimisation; `json::array()` + `push_back` is unchanged). **(2)** `jllama.cpp`'s `jsonSchemaToGrammarBytes` parsed with `nlohmann::ordered_json::parse` and handed the result to `json_schema_to_grammar`, which now takes `const common_json &` — there is no conversion, so this is a hard compile error; switched to `json::parse`. **(3) Two silent, compile-clean regressions**, neither of which a build could catch: `jni_helpers.hpp`'s `require_json_field_impl` declared its parameter `const nlohmann::json &`, and a `common_json` **still binds to it** — through `common_json::operator std::string()` feeding nlohmann's string-constructible converting constructor — so the presence check became a `json::type_error 302` ("type must be string, but is object") thrown out of `handleInfill`, whose two call sites sit *outside* its `try` block; the helper is now a template on the JSON type, with two new `common_json` regression tests. And `common_json_value`'s integral constructor template is `std::is_integral`-gated, which **excludes enums**, so `{"vocab_type", }` bound to the `bool` constructor and serialised as `true`/`false`; `ModelMeta.getVocabType()` reads it with Jackson's `asInt(0)`, so every non-SPM model would have reported vocab type 1. Both emit sites now `static_cast(...)`, with a `CommonJsonEnumTrap` C++ pair pinning the trap and a `LlamaModelTest` assertion that the wire value `isIntegralNumber()`. (Both were confirmed empirically with a standalone probe linked against `common/json.cpp`, not inferred.) **(4)** Seven C++ test literals with a braced array in value position (`test_json_helpers.cpp`'s `extract_embedding_prompt` input, `test_server.cpp`'s `dry_sequence_breakers` + five `samplers` cases) now use `json::array({...})`; the produced JSON is identical. **No JSON output shape changed anywhere**, so no existing assertion moved; **no CMake change** is needed (`common/json.cpp` rides the `llama-common` target both `jllama` and `jllama_test` already link, and the `nlohmann_json` link stays for `log_helpers.hpp` / `train_engine.cpp`, which keep their own alias); and **no patch context shifted** — `server.cpp` is untouched in this range, and `0001`/`0002`/`0003`/`0008` keep byte-identical anchors. Test total 487 → 491. | +| b10584–b10585 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10585 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics **after** the migration above. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index efca0ae41..e883842a2 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10584 + GIT_TAG b10585 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/cpp/jllama.cpp b/llama/src/main/cpp/jllama.cpp index fa44f976e..ea476b321 100644 --- a/llama/src/main/cpp/jllama.cpp +++ b/llama/src/main/cpp/jllama.cpp @@ -906,7 +906,9 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_getModelMetaJson(J REQUIRE_SERVER_CONTEXT(nullptr); if (jctx->vocab_only) { json meta = { - {"vocab_type", llama_vocab_type(jctx->vocab)}, + // static_cast: an unscoped enum binds to common_json_value(bool) and would serialise + // as true/false — see the "vocab_type" note in docs/history/llama-cpp-breaking-changes.md + {"vocab_type", static_cast(llama_vocab_type(jctx->vocab))}, {"n_vocab", llama_vocab_n_tokens(jctx->vocab)}, {"special_tokens", special_tokens_json(jctx->vocab)}, }; @@ -929,7 +931,8 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_getModelMetaJson(J } } json j = { - {"vocab_type", m.model_vocab_type}, + // static_cast: see the vocab_only branch above — an unscoped enum would become a boolean + {"vocab_type", static_cast(m.model_vocab_type)}, {"n_vocab", m.model_vocab_n_tokens}, {"n_ctx_train", m.model_n_ctx_train}, {"n_embd", m.model_n_embd_inp}, @@ -1401,7 +1404,7 @@ JNIEXPORT jbyteArray JNICALL Java_net_ladenthin_llama_LlamaModel_jsonSchemaToGra jstring j_schema) { try { const std::string c_schema = parse_jstring(env, j_schema); - nlohmann::ordered_json c_schema_json = nlohmann::ordered_json::parse(c_schema); + const json c_schema_json = json::parse(c_schema); const std::string c_grammar = json_schema_to_grammar(c_schema_json); return parse_jbytes(env, c_grammar); } catch (const std::exception &e) { diff --git a/llama/src/main/cpp/jni_helpers.hpp b/llama/src/main/cpp/jni_helpers.hpp index d00dd73f0..94823ff28 100644 --- a/llama/src/main/cpp/jni_helpers.hpp +++ b/llama/src/main/cpp/jni_helpers.hpp @@ -185,8 +185,16 @@ struct jllama_context_guard { // // Checks that `data` contains the given key. Returns true if present. // On missing key: throws " is required" via JNI and returns false. +// +// Templated on the JSON type so the upstream `json` alias binds directly. Since llama.cpp +// b10585 (upstream #27511) that alias is `common_json`, not `nlohmann::ordered_json`, and a +// non-template `const nlohmann::json &` parameter would still COMPILE for it — silently, via +// common_json::operator std::string() feeding nlohmann's string-constructible converting +// constructor — turning the presence check into a `json::type_error 302` thrown out of the JNI +// frame. Keep this a template. // --------------------------------------------------------------------------- -[[nodiscard]] inline bool require_json_field_impl(JNIEnv *env, const nlohmann::json &data, const char *field, +template +[[nodiscard]] inline bool require_json_field_impl(JNIEnv *env, const JsonT &data, const char *field, jclass error_class) { if (data.contains(field)) { return true; diff --git a/llama/src/main/cpp/json_helpers.hpp b/llama/src/main/cpp/json_helpers.hpp index 994ad8809..6bb075654 100644 --- a/llama/src/main/cpp/json_helpers.hpp +++ b/llama/src/main/cpp/json_helpers.hpp @@ -8,8 +8,9 @@ // json_helpers.hpp — Pure JSON transformation helpers. // // Every function in this file is pure data transformation: -// - input: nlohmann::json values, server_task_result_ptr, or plain C++ types -// - output: nlohmann::json, std::vector, std::optional, or plain C++ types +// - input: `json` values (the upstream alias — `common_json` since llama.cpp b10585, +// `nlohmann::ordered_json` before it), server_task_result_ptr, or plain C++ types +// - output: `json`, std::vector, std::optional, or plain C++ types // - zero JNI calls (no JNIEnv*, jclass, jstring, …) // - zero llama state (no llama_context*, llama_vocab*, server_context*) // diff --git a/llama/src/main/cpp/utils.hpp b/llama/src/main/cpp/utils.hpp index c810487e6..ea00ca3ca 100644 --- a/llama/src/main/cpp/utils.hpp +++ b/llama/src/main/cpp/utils.hpp @@ -36,7 +36,6 @@ // JSON array. The raw bytes are preserved exactly — no UTF-8 truncation. static json str_to_bytes(const std::string &str) { json bytes = json::array(); - bytes.get_ref().reserve(str.size()); for (unsigned char c : str) { bytes.push_back(static_cast(c)); } diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index a7440afdb..f056cc1fd 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10584"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10585"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10584-"} — call + * plus the resolved upstream commit, e.g. {@code "b10585-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10584"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10585"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10584"; + public static final String LLAMA_CPP_VERSION = "b10585"; // Constants holder — not instantiable. private LlamaCppVersion() {} diff --git a/llama/src/test/cpp/test_jni_helpers.cpp b/llama/src/test/cpp/test_jni_helpers.cpp index 86b4be5ad..19504a40b 100644 --- a/llama/src/test/cpp/test_jni_helpers.cpp +++ b/llama/src/test/cpp/test_jni_helpers.cpp @@ -351,6 +351,25 @@ TEST_F(MockJniFixture, RequireJsonField_EmptyJson_ReturnsFalseAndThrows) { EXPECT_EQ(g_throw_message, "\"input_suffix\" is required"); } +// Regression guard for llama.cpp b10585 (upstream #27511): the upstream `json` alias became +// `common_json`. require_json_field_impl must bind it directly — while the parameter was a plain +// `const nlohmann::json &`, a common_json still compiled here but was converted through +// common_json::operator std::string(), so the call threw json::type_error 302 ("type must be +// string, but is object") instead of checking the key. handleInfill calls this outside a +// try/catch, so the throw escaped the JNI frame. +TEST_F(MockJniFixture, RequireJsonField_CommonJsonPresentField_ReturnsTrueNoThrow) { + const json data = json::object({{"input_prefix", "hello"}}); + EXPECT_TRUE(require_json_field_impl(env, data, "input_prefix", dummy_class)); + EXPECT_FALSE(g_throw_called); +} + +TEST_F(MockJniFixture, RequireJsonField_CommonJsonMissingField_ReturnsFalseAndThrows) { + const json data = json::object({{"other", 1}}); + EXPECT_FALSE(require_json_field_impl(env, data, "input_prefix", dummy_class)); + EXPECT_TRUE(g_throw_called); + EXPECT_EQ(g_throw_message, "\"input_prefix\" is required"); +} + // nlohmann::json::contains() returns true for keys whose value is null. // require_json_field_impl uses contains(), so a null-valued field passes // the presence check and returns true without throwing. Callers that diff --git a/llama/src/test/cpp/test_json_helpers.cpp b/llama/src/test/cpp/test_json_helpers.cpp index a689112c6..02ee602f0 100644 --- a/llama/src/test/cpp/test_json_helpers.cpp +++ b/llama/src/test/cpp/test_json_helpers.cpp @@ -347,7 +347,7 @@ TEST(ExtractEmbeddingPrompt, EmptyBody_ThrowsInvalidArgument) { TEST(ExtractEmbeddingPrompt, ArrayPrompt_ReturnedAsIs) { bool flag = false; - json prompt = extract_embedding_prompt({{"input", {"sentence one", "sentence two"}}}, flag); + json prompt = extract_embedding_prompt({{"input", json::array({"sentence one", "sentence two"})}}, flag); ASSERT_TRUE(prompt.is_array()); ASSERT_EQ(prompt.size(), 2u); EXPECT_EQ(prompt[0], "sentence one"); @@ -355,6 +355,31 @@ TEST(ExtractEmbeddingPrompt, ArrayPrompt_ReturnedAsIs) { EXPECT_FALSE(flag); } +// ============================================================ +// common_json enum trap (llama.cpp b10585, upstream #27511) +// +// The upstream `json` alias is `common_json`, whose value constructors cover +// bool / integral / floating-point / string / container — but the integral one +// is `std::is_integral`-gated, and an *enum* is not integral. An unscoped enum +// therefore binds to `common_json_value(bool)` and silently serialises as +// true/false. Project code must cast enum values to `int` before putting them +// in JSON; `jllama.cpp`'s two "vocab_type" emit sites do exactly that, and +// `ModelMeta.getVocabType()` reads the result with Jackson's `asInt(0)`. +// ============================================================ + +TEST(CommonJsonEnumTrap, UncastEnumBecomesBoolean) { + // documents the trap this guard exists for (tripwire: if upstream ever adds an + // enum constructor, this flips and the cast convention can be revisited) + const json j = json::object({{"vocab_type", LLAMA_VOCAB_TYPE_WPM}}); + EXPECT_TRUE(j.at("vocab_type").is_boolean()); +} + +TEST(CommonJsonEnumTrap, ExplicitIntCastKeepsTheNumericValue) { + const json j = json::object({{"vocab_type", static_cast(LLAMA_VOCAB_TYPE_WPM)}}); + EXPECT_TRUE(j.at("vocab_type").is_number_integer()); + EXPECT_EQ(j.at("vocab_type").get(), static_cast(LLAMA_VOCAB_TYPE_WPM)); +} + // ============================================================ // is_infill_request // ============================================================ diff --git a/llama/src/test/cpp/test_server.cpp b/llama/src/test/cpp/test_server.cpp index 158ab7c28..8d857247b 100644 --- a/llama/src/test/cpp/test_server.cpp +++ b/llama/src/test/cpp/test_server.cpp @@ -1961,7 +1961,7 @@ TEST(ParamsFromJsonCmpl, DryPenaltyLastN_Positive_RoundTrip) { TEST(ParamsFromJsonCmpl, DrySequenceBreakers_NonEmpty_RoundTrip) { // mirrors the llama.cpp default list that withDrySequenceBreakers forwards verbatim - const auto p = parse_params({{"dry_sequence_breakers", {"\n", ":", "\"", "*"}}}); + const auto p = parse_params({{"dry_sequence_breakers", json::array({"\n", ":", "\"", "*"})}}); ASSERT_EQ(p.sampling.dry_sequence_breakers.size(), 4u); EXPECT_EQ(p.sampling.dry_sequence_breakers[0], "\n"); EXPECT_EQ(p.sampling.dry_sequence_breakers[1], ":"); @@ -2001,7 +2001,7 @@ TEST(ParamsFromJsonCmpl, NCmpl_AliasedFromN) { // ============================================================ TEST(ParamsFromJsonCmpl, Samplers_CanonicalNames_Parsed) { - const auto p = parse_params({{"samplers", {"top_k", "top_p", "min_p", "temperature"}}}); + const auto p = parse_params({{"samplers", json::array({"top_k", "top_p", "min_p", "temperature"})}}); ASSERT_EQ(p.sampling.samplers.size(), 4u); EXPECT_EQ(p.sampling.samplers[0], COMMON_SAMPLER_TYPE_TOP_K); EXPECT_EQ(p.sampling.samplers[1], COMMON_SAMPLER_TYPE_TOP_P); @@ -2011,14 +2011,14 @@ TEST(ParamsFromJsonCmpl, Samplers_CanonicalNames_Parsed) { TEST(ParamsFromJsonCmpl, Samplers_KebabCaseAlias_NowAccepted) { // "top-k" / "min-p" alt names were rejected by the server before b9553. - const auto p = parse_params({{"samplers", {"top-k", "min-p"}}}); + const auto p = parse_params({{"samplers", json::array({"top-k", "min-p"})}}); ASSERT_EQ(p.sampling.samplers.size(), 2u); EXPECT_EQ(p.sampling.samplers[0], COMMON_SAMPLER_TYPE_TOP_K); EXPECT_EQ(p.sampling.samplers[1], COMMON_SAMPLER_TYPE_MIN_P); } TEST(ParamsFromJsonCmpl, Samplers_CaseInsensitive) { - const auto p = parse_params({{"samplers", {"TOP_K", "Temperature", "Min-P"}}}); + const auto p = parse_params({{"samplers", json::array({"TOP_K", "Temperature", "Min-P"})}}); ASSERT_EQ(p.sampling.samplers.size(), 3u); EXPECT_EQ(p.sampling.samplers[0], COMMON_SAMPLER_TYPE_TOP_K); EXPECT_EQ(p.sampling.samplers[1], COMMON_SAMPLER_TYPE_TEMPERATURE); @@ -2027,7 +2027,7 @@ TEST(ParamsFromJsonCmpl, Samplers_CaseInsensitive) { TEST(ParamsFromJsonCmpl, Samplers_MiscAliases_Parsed) { // "nucleus" -> top_p, "temp" -> temperature, "typ" -> typical_p - const auto p = parse_params({{"samplers", {"nucleus", "temp", "typ"}}}); + const auto p = parse_params({{"samplers", json::array({"nucleus", "temp", "typ"})}}); ASSERT_EQ(p.sampling.samplers.size(), 3u); EXPECT_EQ(p.sampling.samplers[0], COMMON_SAMPLER_TYPE_TOP_P); EXPECT_EQ(p.sampling.samplers[1], COMMON_SAMPLER_TYPE_TEMPERATURE); @@ -2036,7 +2036,7 @@ TEST(ParamsFromJsonCmpl, Samplers_MiscAliases_Parsed) { TEST(ParamsFromJsonCmpl, Samplers_UnknownName_SkippedNotError) { // unknown names are warned and skipped, not a hard error. - const auto p = parse_params({{"samplers", {"top_k", "definitely_not_a_sampler"}}}); + const auto p = parse_params({{"samplers", json::array({"top_k", "definitely_not_a_sampler"})}}); ASSERT_EQ(p.sampling.samplers.size(), 1u); EXPECT_EQ(p.sampling.samplers[0], COMMON_SAMPLER_TYPE_TOP_K); } diff --git a/llama/src/test/cpp/test_utils.cpp b/llama/src/test/cpp/test_utils.cpp index cbba7a40c..fac549c3c 100644 --- a/llama/src/test/cpp/test_utils.cpp +++ b/llama/src/test/cpp/test_utils.cpp @@ -644,11 +644,11 @@ TEST(JsonValue, BoolValue) { // json_is_array_of_numbers / json_is_array_of_mixed // ============================================================ -TEST(JsonArrayChecks, ArrayOfIntegers_IsNumbers) { EXPECT_TRUE(json_is_array_of_numbers(json{1, 2, 3})); } +TEST(JsonArrayChecks, ArrayOfIntegers_IsNumbers) { EXPECT_TRUE(json_is_array_of_numbers(json::array({1, 2, 3}))); } TEST(JsonArrayChecks, EmptyArray_IsNumbers) { EXPECT_TRUE(json_is_array_of_numbers(json::array())); } -TEST(JsonArrayChecks, ArrayWithString_NotNumbers) { EXPECT_FALSE(json_is_array_of_numbers(json{1, "hello", 3})); } +TEST(JsonArrayChecks, ArrayWithString_NotNumbers) { EXPECT_FALSE(json_is_array_of_numbers(json::array({1, "hello", 3}))); } TEST(JsonArrayChecks, NonArray_NotNumbers) { EXPECT_FALSE(json_is_array_of_numbers(json("just a string"))); @@ -656,12 +656,12 @@ TEST(JsonArrayChecks, NonArray_NotNumbers) { } TEST(JsonArrayChecks, MixedNumbersAndStrings_IsMixed) { - EXPECT_TRUE(json_is_array_of_mixed_numbers_strings(json{1, "hello", 3})); + EXPECT_TRUE(json_is_array_of_mixed_numbers_strings(json::array({1, "hello", 3}))); } -TEST(JsonArrayChecks, OnlyNumbers_NotMixed) { EXPECT_FALSE(json_is_array_of_mixed_numbers_strings(json{1, 2, 3})); } +TEST(JsonArrayChecks, OnlyNumbers_NotMixed) { EXPECT_FALSE(json_is_array_of_mixed_numbers_strings(json::array({1, 2, 3}))); } -TEST(JsonArrayChecks, OnlyStrings_NotMixed) { EXPECT_FALSE(json_is_array_of_mixed_numbers_strings(json{"a", "b"})); } +TEST(JsonArrayChecks, OnlyStrings_NotMixed) { EXPECT_FALSE(json_is_array_of_mixed_numbers_strings(json::array({"a", "b"}))); } TEST(JsonArrayChecks, EmptyArray_NotMixed) { EXPECT_FALSE(json_is_array_of_mixed_numbers_strings(json::array())); } @@ -671,11 +671,11 @@ TEST(JsonArrayChecks, EmptyArray_NotMixed) { EXPECT_FALSE(json_is_array_of_mixed // non-array value. TEST(JsonArrayChecks, ArrayWithNumber_ContainsNumbers) { - EXPECT_TRUE(json_is_array_and_contains_numbers(json{1, "hello"})); + EXPECT_TRUE(json_is_array_and_contains_numbers(json::array({1, "hello"}))); } TEST(JsonArrayChecks, ArrayOnlyStrings_NotContainsNumbers) { - EXPECT_FALSE(json_is_array_and_contains_numbers(json{"a", "b"})); + EXPECT_FALSE(json_is_array_and_contains_numbers(json::array({"a", "b"}))); } TEST(JsonArrayChecks, EmptyArray_NotContainsNumbers) { diff --git a/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java b/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java index 312de4d3f..f855b7174 100644 --- a/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java +++ b/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java @@ -1291,6 +1291,13 @@ public void testGetModelMeta() throws LlamaException { // Dynamic access via the underlying JsonNode assertTrue(meta.asJson().has("modalities"), "modalities field must be present"); assertTrue(meta.asJson().has("vocab_type"), "vocab_type field must be present"); + // vocab_type is an unscoped C enum on the native side. Since llama.cpp b10585 the upstream + // `json` alias is common_json, whose value ctors do not cover enums — an uncast enum binds + // to the bool ctor and arrives here as true/false, which getVocabType()'s asInt(0) would + // silently read as 1 for every non-SPM model. Pin the wire type, not just the key. + assertTrue( + meta.asJson().path("vocab_type").isIntegralNumber(), + "vocab_type must arrive as an integer, not a boolean"); // Architecture and name from GGUF general.* metadata String architecture = meta.getArchitecture(); From e4a81495353d5858c4ebd228aaa6ae5c2e651fc9 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:51:55 +0000 Subject: [PATCH 19/76] chore: upgrade llama.cpp from b10585 to b10590 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 09fa2bc79..ce13599cf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10585** +Current llama.cpp pinned version: **b10590** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10585 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10590 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10585`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10590`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1361,7 +1361,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10585`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10590`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index e4e97f72f..5ecbdcb74 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10585](https://img.shields.io/badge/llama.cpp-%23b10585-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10585) +[![llama.cpp b10590](https://img.shields.io/badge/llama.cpp-%23b10590-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10590) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 6e34ebf2b..5a5cb5139 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -655,3 +655,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10578–b10584 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10584 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10584–b10585 | **NEW** `common/json.{h,cpp}` + `common/CMakeLists.txt`, `common/chat.h`, `common/chat-auto-parser.h`, `common/chat-peg-parser.h`, `common/json-schema-to-grammar.h`, `common/peg-parser.h`, `common/arg.cpp`, `common/{chat,download,hf-cache,jinja/*,json-schema-to-grammar,peg-parser}.cpp`, `tools/server/{server-common.h,server-common.cpp,server-chat.h,server-chat.cpp,server-context.h,server-context.cpp,server-task.h,server-task.cpp,server-schema.cpp,server-models.cpp,server-tools.cpp}` (**#27511 "common: add json.h abstraction"**) | **BREAKING — the largest project impact of the whole b10456 → b10618 walk.** Upstream introduces the pimpl wrapper class `common_json` and flips `using json = nlohmann::ordered_json` → `using json = common_json` in `server-common.h`, so **every** project TU that includes an upstream server header changes JSON type. `common_json` is a deliberately closed API: no `get_ref`, no `array_t`, no `type_name()`, a braced list in *value* position does not build an array, `at(key)` no longer implicitly converts (needs `.get()`), `get()` is limited to the specialisations in `common/json.cpp`, and errors are `common_json_error`. Project changes, in the order they were found: **(1)** `utils.hpp`'s `str_to_bytes` used `bytes.get_ref().reserve(...)` — dropped (a pure optimisation; `json::array()` + `push_back` is unchanged). **(2)** `jllama.cpp`'s `jsonSchemaToGrammarBytes` parsed with `nlohmann::ordered_json::parse` and handed the result to `json_schema_to_grammar`, which now takes `const common_json &` — there is no conversion, so this is a hard compile error; switched to `json::parse`. **(3) Two silent, compile-clean regressions**, neither of which a build could catch: `jni_helpers.hpp`'s `require_json_field_impl` declared its parameter `const nlohmann::json &`, and a `common_json` **still binds to it** — through `common_json::operator std::string()` feeding nlohmann's string-constructible converting constructor — so the presence check became a `json::type_error 302` ("type must be string, but is object") thrown out of `handleInfill`, whose two call sites sit *outside* its `try` block; the helper is now a template on the JSON type, with two new `common_json` regression tests. And `common_json_value`'s integral constructor template is `std::is_integral`-gated, which **excludes enums**, so `{"vocab_type", }` bound to the `bool` constructor and serialised as `true`/`false`; `ModelMeta.getVocabType()` reads it with Jackson's `asInt(0)`, so every non-SPM model would have reported vocab type 1. Both emit sites now `static_cast(...)`, with a `CommonJsonEnumTrap` C++ pair pinning the trap and a `LlamaModelTest` assertion that the wire value `isIntegralNumber()`. (Both were confirmed empirically with a standalone probe linked against `common/json.cpp`, not inferred.) **(4)** Seven C++ test literals with a braced array in value position (`test_json_helpers.cpp`'s `extract_embedding_prompt` input, `test_server.cpp`'s `dry_sequence_breakers` + five `samplers` cases) now use `json::array({...})`; the produced JSON is identical. **No JSON output shape changed anywhere**, so no existing assertion moved; **no CMake change** is needed (`common/json.cpp` rides the `llama-common` target both `jllama` and `jllama_test` already link, and the `nlohmann_json` link stays for `log_helpers.hpp` / `train_engine.cpp`, which keep their own alias); and **no patch context shifted** — `server.cpp` is untouched in this range, and `0001`/`0002`/`0003`/`0008` keep byte-identical anchors. Test total 487 → 491. | | b10584–b10585 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10585 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics **after** the migration above. Full link + `ctest` deferred to the b10618 target. | +| b10585–b10590 | `common/json.{cpp,h}` (**#27575 "fix clang lto": the `common_json_value` set/map/unordered_map/vector constructors and every `common_json::get()` flip from explicit *instantiation* to explicit *specialization* — "an explicit instantiation is a weak symbol, dropped by some LTO builds (clang-cl)" — and `operator std::string()` + `value(key, const char *)` move from inline-in-`json.h` to out-of-line in `json.cpp`; the set of usable types is unchanged**), `vendor/sheredom/subprocess.h` + `scripts/sync_vendor.py` (**#27409 upstream resync, +363/−12: new `SUBPROCESS_SPAWN_VIA_FORK` / `SUBPROCESS_ADDCHDIR_IS_POSIX` macros gating a `fork()`+`execve`/`execvpe` launcher with an errno-relay pipe, new `subprocess_pipe_cloexec()` / `subprocess_fds_above_std()` POSIX helpers, and a Windows `STARTUPINFOEX` + `PROC_THREAD_ATTRIBUTE_HANDLE_LIST` handle-inheritance list**), `tools/mtmd/{clip.cpp,clip-graph.h,models/gemma4v.cpp,models/minimax-m3.cpp}` (**#27521: 2D RoPE rewritten from view/rope/`ggml_concat` onto two in-place `ggml_rope_ext` calls + `ggml_rope_set_offset`**) | **No project-source and no project-CMake change.** The `json` work *is* in the project's compile path (the alias became `common_json` at b10585) and the project calls both members that moved out-of-line, but `common/json.cpp` belongs to the `llama-common` target that `jllama` and `jllama_test` already link — nothing to wire, and no `to_json()` shape moved. The project enables no LTO, so the bug being fixed never bit it; the change is robustness for the clang-cl Windows-arm64 job. **`subprocess.h` re-verified against the dropped patch `0009`:** `SUBPROCESS_HAVE_CWD` and its `#elif !SUBPROCESS_HAVE_CWD → posix_error = ENOSYS;` fallback both survive, and `SUBPROCESS_SPAWN_VIA_FORK` is 0 on glibc/bionic/macOS/Windows (only `_AIX`/`__OpenBSD__`/old NetBSD set it), so manylinux2014 (glibc 2.17) still resolves exactly as before and the new `execvpe` declaration is preprocessed away everywhere the project builds — **`0009` stays dropped.** The new always-compiled POSIX helpers need only `pipe2`/`O_CLOEXEC`/`F_DUPFD` (glibc 2.7+/bionic) and the header already includes ``/``; the Windows attribute-list path adds no new link input. mtmd is internal-only here: `build_rope_2d`'s signature is untouched (the `clip-graph.h` hunk is comment-only) and `mtmd.h`/`mtmd-helper.h`/`clip.h` are byte-identical, so `tts_engine.cpp`'s surface is unaffected. **Patches:** zero changes under `tools/server/` and none to `common/arg.{h,cpp}` — byte-identical context for all six. | +| b10585–b10590 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10590 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index e883842a2..92d002a98 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10585 + GIT_TAG b10590 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index f056cc1fd..8df411c80 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10585"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10590"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10585-"} — call + * plus the resolved upstream commit, e.g. {@code "b10590-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10585"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10590"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10585"; + public static final String LLAMA_CPP_VERSION = "b10590"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 90cd0bf9380ad67cf7ac7ca59767a25460a20f00 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:53:55 +0000 Subject: [PATCH 20/76] chore: upgrade llama.cpp from b10590 to b10593 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index ce13599cf..6a0b5f05a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10590** +Current llama.cpp pinned version: **b10593** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10590 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10593 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10590`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10593`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1361,7 +1361,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10590`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10593`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 5ecbdcb74..e1020322c 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10590](https://img.shields.io/badge/llama.cpp-%23b10590-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10590) +[![llama.cpp b10593](https://img.shields.io/badge/llama.cpp-%23b10593-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10593) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 5a5cb5139..697683fb6 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -657,3 +657,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10584–b10585 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10585 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics **after** the migration above. Full link + `ctest` deferred to the b10618 target. | | b10585–b10590 | `common/json.{cpp,h}` (**#27575 "fix clang lto": the `common_json_value` set/map/unordered_map/vector constructors and every `common_json::get()` flip from explicit *instantiation* to explicit *specialization* — "an explicit instantiation is a weak symbol, dropped by some LTO builds (clang-cl)" — and `operator std::string()` + `value(key, const char *)` move from inline-in-`json.h` to out-of-line in `json.cpp`; the set of usable types is unchanged**), `vendor/sheredom/subprocess.h` + `scripts/sync_vendor.py` (**#27409 upstream resync, +363/−12: new `SUBPROCESS_SPAWN_VIA_FORK` / `SUBPROCESS_ADDCHDIR_IS_POSIX` macros gating a `fork()`+`execve`/`execvpe` launcher with an errno-relay pipe, new `subprocess_pipe_cloexec()` / `subprocess_fds_above_std()` POSIX helpers, and a Windows `STARTUPINFOEX` + `PROC_THREAD_ATTRIBUTE_HANDLE_LIST` handle-inheritance list**), `tools/mtmd/{clip.cpp,clip-graph.h,models/gemma4v.cpp,models/minimax-m3.cpp}` (**#27521: 2D RoPE rewritten from view/rope/`ggml_concat` onto two in-place `ggml_rope_ext` calls + `ggml_rope_set_offset`**) | **No project-source and no project-CMake change.** The `json` work *is* in the project's compile path (the alias became `common_json` at b10585) and the project calls both members that moved out-of-line, but `common/json.cpp` belongs to the `llama-common` target that `jllama` and `jllama_test` already link — nothing to wire, and no `to_json()` shape moved. The project enables no LTO, so the bug being fixed never bit it; the change is robustness for the clang-cl Windows-arm64 job. **`subprocess.h` re-verified against the dropped patch `0009`:** `SUBPROCESS_HAVE_CWD` and its `#elif !SUBPROCESS_HAVE_CWD → posix_error = ENOSYS;` fallback both survive, and `SUBPROCESS_SPAWN_VIA_FORK` is 0 on glibc/bionic/macOS/Windows (only `_AIX`/`__OpenBSD__`/old NetBSD set it), so manylinux2014 (glibc 2.17) still resolves exactly as before and the new `execvpe` declaration is preprocessed away everywhere the project builds — **`0009` stays dropped.** The new always-compiled POSIX helpers need only `pipe2`/`O_CLOEXEC`/`F_DUPFD` (glibc 2.7+/bionic) and the header already includes ``/``; the Windows attribute-list path adds no new link input. mtmd is internal-only here: `build_rope_2d`'s signature is untouched (the `clip-graph.h` hunk is comment-only) and `mtmd.h`/`mtmd-helper.h`/`clip.h` are byte-identical, so `tts_engine.cpp`'s surface is unaffected. **Patches:** zero changes under `tools/server/` and none to `common/arg.{h,cpp}` — byte-identical context for all six. | | b10585–b10590 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10590 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10590–b10593 | `include/llama.h` (**one comment line: a `[TAG_LLAMA_SEQ_ID_NEG]` marker added to `llama_memory_seq_rm`'s doc block**), `src/llama-*.cpp` (**#26756 DeepseekV4 multi-seq rollback fix; #27574 tensor-parallel meta tensor-split state propagation**), `tools/ui/**` (**#27263: tabbed chat-conversation navigation**) | **No project-source change.** `include/llama.h` is priority 7 on the CLAUDE.md review list, so its diff was read in full: it is a single comment line inside an existing doc block — no signature, no enum value, no struct field moved. The rest is llama.cpp's internal KV/tensor-split implementation and the Svelte WebUI. Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `vendor/` or any `CMakeLists.txt` changed, so no patch context moved. | +| b10590–b10593 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10593 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 92d002a98..da19dcedf 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10590 + GIT_TAG b10593 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 8df411c80..36cbd92fd 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10590"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10593"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10590-"} — call + * plus the resolved upstream commit, e.g. {@code "b10593-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10590"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10593"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10590"; + public static final String LLAMA_CPP_VERSION = "b10593"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 9208267af288310e4cd62f5c0f5a303c53d00ea1 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:56:08 +0000 Subject: [PATCH 21/76] chore: upgrade llama.cpp from b10593 to b10599 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 6a0b5f05a..9740f832c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10593** +Current llama.cpp pinned version: **b10599** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10593 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10599 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10593`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10599`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1361,7 +1361,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10593`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10599`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index e1020322c..464dcce1f 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10593](https://img.shields.io/badge/llama.cpp-%23b10593-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10593) +[![llama.cpp b10599](https://img.shields.io/badge/llama.cpp-%23b10599-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10599) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 697683fb6..591252ae7 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -659,3 +659,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10585–b10590 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10590 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10590–b10593 | `include/llama.h` (**one comment line: a `[TAG_LLAMA_SEQ_ID_NEG]` marker added to `llama_memory_seq_rm`'s doc block**), `src/llama-*.cpp` (**#26756 DeepseekV4 multi-seq rollback fix; #27574 tensor-parallel meta tensor-split state propagation**), `tools/ui/**` (**#27263: tabbed chat-conversation navigation**) | **No project-source change.** `include/llama.h` is priority 7 on the CLAUDE.md review list, so its diff was read in full: it is a single comment line inside an existing doc block — no signature, no enum value, no struct field moved. The rest is llama.cpp's internal KV/tensor-split implementation and the Svelte WebUI. Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `vendor/` or any `CMakeLists.txt` changed, so no patch context moved. | | b10590–b10593 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10593 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10593–b10599 | `tools/mtmd/{clip-model.h,clip.cpp,mtmd-image.cpp}` (**#27594 "mtmd: use pillow-accurate algo, correct resize_algo for all models": `enum resize_algo` drops `RESIZE_ALGO_BICUBIC_PILLOW` (plain `BICUBIC` now *means* the Pillow path), the four private `img_tool` statics `resize_{bilinear,bicubic,bicubic_pillow,lanczos_pillow}` and both dispatch switches collapse into one `resize_pillow(..., resize_algo algo)` (−246 net), and ~14 projector types are re-assigned**), `tools/server/server-context.cpp` (**#27600: private `slots_n_diff` member + `LLAMA_SERVER_SLOTS_N_DIFF` getenv; the hardcoded prompt-mismatch debug window `n_past−4..+6` becomes `n_past−slots_n_diff..+slots_n_diff+2`**), `common/common.cpp` (**#26692: `common_params_print_info`'s device-enumeration loop gated on `print_devices && verbosity >= LOG_LEVEL_TRACE`**), `src/llama-grammar.cpp` (**#27591: `parse_char()` accepts `case '-':`, so a `\-` escape produced by `gbnf_escape_char_class()` no longer throws**), `tools/CMakeLists.txt` (**#27548: `add_subdirectory(parser)` dropped — the parser tool moved under `tests/`**) | **No project-source, no project-CMake and no patch change.** The mtmd resize rework is entirely mtmd-internal: `resize_algo`/`clip_hparams` live in `tools/mtmd/clip-model.h`, which the project never includes, and all four deleted functions were `private:` statics inside `struct img_tool`; the public `mtmd.h`/`mtmd-helper.h` are unchanged in this range. The only visible effect is that `MultimodalIntegrationTest`'s SmolVLM preprocessing becomes Pillow-exact — pixel-level, no API or JSON shape. `server-context.cpp`'s new fields are `private:` in `server_context_impl` and **`server-context.h` is unchanged**, so no `to_json()` shape moved and no C++ test assertion is affected; patch `0002`'s and `0003`'s anchors are byte-identical and resolve with a pure offset. `common_params_print_info` keeps its `bool print_devices = true` default, so `jllama.cpp`'s one-argument call still compiles; the behaviour delta is that at the default log threshold libjllama no longer prints `device_info:` (nothing in the project reads it) and no longer triggers early CUDA primary-context creation from that call site. The grammar `\-` fix is strictly widening — a previously-throwing input now parses — and is a **positive** for the project's agentic tool-calling target; no test pinned the old throw. The `tools/CMakeLists.txt` edit sits inside the `LLAMA_BUILD_TOOLS`-gated branch the project never enters. | +| b10593–b10599 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10599 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index da19dcedf..2f4a57599 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10593 + GIT_TAG b10599 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 36cbd92fd..697c513ae 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10593"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10599"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10593-"} — call + * plus the resolved upstream commit, e.g. {@code "b10599-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10593"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10599"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10593"; + public static final String LLAMA_CPP_VERSION = "b10599"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 12e25a5b6fc4001a07fb3d37af0caa1af073205e Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 09:58:34 +0000 Subject: [PATCH 22/76] chore: upgrade llama.cpp from b10599 to b10612 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 9740f832c..a80fe1284 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10599** +Current llama.cpp pinned version: **b10612** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10599 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10612 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10599`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10612`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1361,7 +1361,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10599`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10612`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 464dcce1f..daa225d39 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10599](https://img.shields.io/badge/llama.cpp-%23b10599-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10599) +[![llama.cpp b10612](https://img.shields.io/badge/llama.cpp-%23b10612-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10612) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 591252ae7..567e4a268 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -661,3 +661,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10590–b10593 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10593 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10593–b10599 | `tools/mtmd/{clip-model.h,clip.cpp,mtmd-image.cpp}` (**#27594 "mtmd: use pillow-accurate algo, correct resize_algo for all models": `enum resize_algo` drops `RESIZE_ALGO_BICUBIC_PILLOW` (plain `BICUBIC` now *means* the Pillow path), the four private `img_tool` statics `resize_{bilinear,bicubic,bicubic_pillow,lanczos_pillow}` and both dispatch switches collapse into one `resize_pillow(..., resize_algo algo)` (−246 net), and ~14 projector types are re-assigned**), `tools/server/server-context.cpp` (**#27600: private `slots_n_diff` member + `LLAMA_SERVER_SLOTS_N_DIFF` getenv; the hardcoded prompt-mismatch debug window `n_past−4..+6` becomes `n_past−slots_n_diff..+slots_n_diff+2`**), `common/common.cpp` (**#26692: `common_params_print_info`'s device-enumeration loop gated on `print_devices && verbosity >= LOG_LEVEL_TRACE`**), `src/llama-grammar.cpp` (**#27591: `parse_char()` accepts `case '-':`, so a `\-` escape produced by `gbnf_escape_char_class()` no longer throws**), `tools/CMakeLists.txt` (**#27548: `add_subdirectory(parser)` dropped — the parser tool moved under `tests/`**) | **No project-source, no project-CMake and no patch change.** The mtmd resize rework is entirely mtmd-internal: `resize_algo`/`clip_hparams` live in `tools/mtmd/clip-model.h`, which the project never includes, and all four deleted functions were `private:` statics inside `struct img_tool`; the public `mtmd.h`/`mtmd-helper.h` are unchanged in this range. The only visible effect is that `MultimodalIntegrationTest`'s SmolVLM preprocessing becomes Pillow-exact — pixel-level, no API or JSON shape. `server-context.cpp`'s new fields are `private:` in `server_context_impl` and **`server-context.h` is unchanged**, so no `to_json()` shape moved and no C++ test assertion is affected; patch `0002`'s and `0003`'s anchors are byte-identical and resolve with a pure offset. `common_params_print_info` keeps its `bool print_devices = true` default, so `jllama.cpp`'s one-argument call still compiles; the behaviour delta is that at the default log threshold libjllama no longer prints `device_info:` (nothing in the project reads it) and no longer triggers early CUDA primary-context creation from that call site. The grammar `\-` fix is strictly widening — a previously-throwing input now parses — and is a **positive** for the project's agentic tool-calling target; no test pinned the old throw. The `tools/CMakeLists.txt` edit sits inside the `LLAMA_BUILD_TOOLS`-gated branch the project never enters. | | b10593–b10599 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10599 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10599–b10612 | `tools/mtmd/mtmd-helper.cpp` (**#27596 "video: fix moov atom at the end of file": `subprocess_handle` splits `alive` into `created`+`alive`, `stop()` gains `subprocess_join()` to reap zombies plus a Windows `CloseHandle(proc.hStdInput)`, `start_feeder()` gains `pthread_sigmask(SIG_BLOCK, {SIGPIPE})` and an `#ifdef F_SETNOSIGPIPE fcntl(...)`, and the ffmpeg argv gains `-read_ahead_limit -1`; new `#include //` under `#ifdef MTMD_VIDEO`/`#ifndef _WIN32`**), `ggml/**` (**#27644 `ggml_clamp` fix; #27608 shorter virtual-device naming; #27545 WebGPU include order**), `src/**` (**#26534 MTP in GLM-4.5-Air; #26490 Deepseek 4 `-sm tensor`; #27513 mamba2 GEMM dispatch**), `convert_hf_to_gguf.py`, `tools/ui/**` | **No project-source change.** `mtmd-helper.cpp` **is** compiled into `libjllama` (the project `add_subdirectory`s `tools/mtmd` and never sets `LLAMA_SUBPROCESS`, so `MTMD_VIDEO` is ON on every classifier including Android), so its three new build-surface dependencies were each checked against the pinned tree rather than assumed: `subprocess_join` is declared in `vendor/sheredom/subprocess.h`, `struct subprocess_s` carries `void *hStdInput` (so the Windows `CloseHandle` compiles), `pthread_sigmask` resolves through the pre-existing `target_link_libraries(mtmd PRIVATE Threads::Threads …)`, and `F_SETNOSIGPIPE` is `#ifdef`-guarded off on Linux/Android. `mtmd.h` and `mtmd-helper.h` are unchanged in this range, so `tts_engine.cpp`'s `mtmd_helper::gen_audio` surface is untouched. Nothing under `common/`, `tools/server/`, `include/` or any project-consumed `CMakeLists.txt` changed — no patch context moved. | +| b10599–b10612 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10612 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 2f4a57599..9debb3425 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10599 + GIT_TAG b10612 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 697c513ae..f4fc4ae5d 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10599"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10612"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10599-"} — call + * plus the resolved upstream commit, e.g. {@code "b10612-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10599"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10612"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10599"; + public static final String LLAMA_CPP_VERSION = "b10612"; // Constants holder — not instantiable. private LlamaCppVersion() {} From d29f323e806dc31118d46d7f8e17647c132cb218 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 10:00:36 +0000 Subject: [PATCH 23/76] chore: upgrade llama.cpp from b10612 to b10614 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index a80fe1284..0d4eaa88d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10612** +Current llama.cpp pinned version: **b10614** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10612 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10614 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10612`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10614`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1361,7 +1361,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10612`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10614`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index daa225d39..ad22a7f1b 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10612](https://img.shields.io/badge/llama.cpp-%23b10612-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10612) +[![llama.cpp b10614](https://img.shields.io/badge/llama.cpp-%23b10614-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10614) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 567e4a268..2613579fd 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -663,3 +663,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10593–b10599 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10599 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10599–b10612 | `tools/mtmd/mtmd-helper.cpp` (**#27596 "video: fix moov atom at the end of file": `subprocess_handle` splits `alive` into `created`+`alive`, `stop()` gains `subprocess_join()` to reap zombies plus a Windows `CloseHandle(proc.hStdInput)`, `start_feeder()` gains `pthread_sigmask(SIG_BLOCK, {SIGPIPE})` and an `#ifdef F_SETNOSIGPIPE fcntl(...)`, and the ffmpeg argv gains `-read_ahead_limit -1`; new `#include //` under `#ifdef MTMD_VIDEO`/`#ifndef _WIN32`**), `ggml/**` (**#27644 `ggml_clamp` fix; #27608 shorter virtual-device naming; #27545 WebGPU include order**), `src/**` (**#26534 MTP in GLM-4.5-Air; #26490 Deepseek 4 `-sm tensor`; #27513 mamba2 GEMM dispatch**), `convert_hf_to_gguf.py`, `tools/ui/**` | **No project-source change.** `mtmd-helper.cpp` **is** compiled into `libjllama` (the project `add_subdirectory`s `tools/mtmd` and never sets `LLAMA_SUBPROCESS`, so `MTMD_VIDEO` is ON on every classifier including Android), so its three new build-surface dependencies were each checked against the pinned tree rather than assumed: `subprocess_join` is declared in `vendor/sheredom/subprocess.h`, `struct subprocess_s` carries `void *hStdInput` (so the Windows `CloseHandle` compiles), `pthread_sigmask` resolves through the pre-existing `target_link_libraries(mtmd PRIVATE Threads::Threads …)`, and `F_SETNOSIGPIPE` is `#ifdef`-guarded off on Linux/Android. `mtmd.h` and `mtmd-helper.h` are unchanged in this range, so `tts_engine.cpp`'s `mtmd_helper::gen_audio` surface is untouched. Nothing under `common/`, `tools/server/`, `include/` or any project-consumed `CMakeLists.txt` changed — no patch context moved. | | b10599–b10612 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10612 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10612–b10614 | `ggml/src/ggml-metal/**` + `ggml/CMakeLists.txt` (**#26561: per-op Metal source split + parallel compile — a large mechanical file split, hence the ~1 MiB raw diff**), `gguf-py/gguf/metadata.py` (**#27659: `repetition_penalty` read from `generation_config.json` at conversion time**) | **No project-source change.** Zero lines changed under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any project-consumed `CMakeLists.txt`, so no patch context moved and no header in the project's include graph was touched. The Metal split affects only the `Mac/aarch64` classifier's compiled sources (upstream owns that list entirely); the `generation_config.json` change is a GGUF-conversion-time default with no C++ API surface. | +| b10612–b10614 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10614 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 9debb3425..d934550bf 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10612 + GIT_TAG b10614 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index f4fc4ae5d..0e902a233 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10612"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10614"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10612-"} — call + * plus the resolved upstream commit, e.g. {@code "b10614-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10612"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10614"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10612"; + public static final String LLAMA_CPP_VERSION = "b10614"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 721dbeb8ca2b1f92207942eb5bac3ea495ebd481 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 10:03:01 +0000 Subject: [PATCH 24/76] chore: upgrade llama.cpp from b10614 to b10615 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 0d4eaa88d..1b980355b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10614** +Current llama.cpp pinned version: **b10615** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10614 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10615 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10614`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10615`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1361,7 +1361,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10614`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10615`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index ad22a7f1b..663fd1b02 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10614](https://img.shields.io/badge/llama.cpp-%23b10614-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10614) +[![llama.cpp b10615](https://img.shields.io/badge/llama.cpp-%23b10615-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10615) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 2613579fd..f80226844 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -665,3 +665,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10599–b10612 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10612 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10612–b10614 | `ggml/src/ggml-metal/**` + `ggml/CMakeLists.txt` (**#26561: per-op Metal source split + parallel compile — a large mechanical file split, hence the ~1 MiB raw diff**), `gguf-py/gguf/metadata.py` (**#27659: `repetition_penalty` read from `generation_config.json` at conversion time**) | **No project-source change.** Zero lines changed under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any project-consumed `CMakeLists.txt`, so no patch context moved and no header in the project's include graph was touched. The Metal split affects only the `Mac/aarch64` classifier's compiled sources (upstream owns that list entirely); the `generation_config.json` change is a GGUF-conversion-time default with no C++ API surface. | | b10612–b10614 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10614 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10614–b10615 | `ggml/src/ggml-metal/**` + `tools/CMakeLists.txt` (**#26570: per-device tuned `(Q, NE)` for the flash-attention vec kernels; a Metal-only `tuning` tool is added under an `if (GGML_METAL)` guard**) | **No project-source change.** The `tools/CMakeLists.txt` addition sits inside the `LLAMA_BUILD_TOOLS`-gated branch the project never enters (`LLAMA_BUILD_TOOLS` defaults to `LLAMA_STANDALONE`, i.e. OFF under FetchContent, and the project forces it OFF on Android), and is additionally Metal-only. Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/` or `vendor/` changed — no patch context moved, no header in the project's include graph touched. | +| b10614–b10615 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10615 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index d934550bf..794987cd0 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10614 + GIT_TAG b10615 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 0e902a233..b35dbbe28 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10614"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10615"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10614-"} — call + * plus the resolved upstream commit, e.g. {@code "b10615-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10614"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10615"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10614"; + public static final String LLAMA_CPP_VERSION = "b10615"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 3d755e4980c39572850936bbbc9933c40c737d47 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 10:15:01 +0000 Subject: [PATCH 25/76] chore: upgrade llama.cpp from b10615 to b10618 (final target) Also refreshes documentation the walk proved stale: the removed -DLLAMA_TAG second pin, the patch list in the bump runbook, the JSON_ASSERT/json comment in utils.hpp, and the claim that resolves through a PUBLIC ../vendor include dir on llama-common (upstream #27304 replaced it with PUBLIC vendor::nlohmann / vendor::sheredom, which re-export the same root). Records the pre-existing ServerMetrics payload drift in TODO.md. Verified at the target: fail-loud PATCH_COMMAND clean on a fresh FetchContent configure, full build, ctest 491/491, mvn test 1405 run / 0 failures, NativeLibraryLoadSmokeTest green, clang-format 22.1.8 clean, javadoc clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 17 +++++++++-------- README.md | 2 +- TODO.md | 19 +++++++++++++++++++ docs/history/llama-cpp-breaking-changes.md | 4 +++- docs/upgrade/llama-cpp-version-bump.md | 11 ++++++----- llama/CMakeLists.txt | 11 +++++++---- llama/src/main/cpp/utils.hpp | 2 +- .../llama/value/LlamaCppVersion.java | 8 ++++---- llama/src/test/cpp/test_server.cpp | 8 +++++--- llama/src/test/cpp/test_utils.cpp | 12 +++++++++--- 10 files changed, 64 insertions(+), 30 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 1b980355b..e042d6d3f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10615** +Current llama.cpp pinned version: **b10618** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10615 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10618 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10615`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10618`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -741,8 +741,9 @@ edit/verify/commit loop below. Use it for any non-trivial bump; the steps here a To change the llama.cpp version, update the following **four** files (and re-verify `patches/`): -1. **llama/CMakeLists.txt** — the `GIT_TAG` line for llama.cpp: `GIT_TAG b8831` (and the - cosmetic `-DLLAMA_TAG=b8831` a few lines below, passed to the TTS generator — keep them equal) +1. **llama/CMakeLists.txt** — the `GIT_TAG` line for llama.cpp: `GIT_TAG b8831`. (There is + no second tag to keep in sync any more: the cosmetic `-DLLAMA_TAG=` that fed the old build-time + TTS extraction went away with the Qwen3-TTS rework — see "Qwen3-TTS via `mtmd_helper::gen_audio`".) 2. **README.md** — the badge and link line with the version number 3. **CLAUDE.md** — the "Current llama.cpp pinned version" line 4. **llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java** — the @@ -754,7 +755,7 @@ To change the llama.cpp version, update the following **four** files (and re-ver Example: To upgrade from b8808 to b8831: ```bash -# Edit llama/CMakeLists.txt: change GIT_TAG b8808 to b8831 (and the -DLLAMA_TAG line) +# Edit llama/CMakeLists.txt: change GIT_TAG b8808 to b8831 # Edit README.md: change b8808 to b8831 (in both badge and link) # Edit CLAUDE.md: change b8808 to b8831 # Edit LlamaCppVersion.java: change LLAMA_CPP_VERSION "b8808" to "b8831" @@ -1133,7 +1134,7 @@ If the local check passes (`BUILD SUCCESS`), the `mvn package` job in 2. **`common_json` converts to `std::string` implicitly**, so it binds happily to a `const nlohmann::json &` parameter (via nlohmann's string-constructible converting constructor) and then throws `json::type_error 302` at runtime. Never declare a project helper as taking `nlohmann::json` when callers pass the `json` alias — `require_json_field_impl` is a template for exactly this reason. Other differences to know: no `get_ref`/`array_t`/`type_name()`; a braced list in *value* position does not build an array (write `json::array({...})`); `at(key)` needs an explicit `.get()`; errors are `common_json_error`; and `get()` is limited to the types explicitly specialised in `common/json.cpp`. `log_helpers.hpp` and `train_engine.cpp` keep their own `nlohmann::json` alias — they never touch the server's `json`. - Uses `nlohmann/json` for JSON deserialization of parameters in the two files named above; everything on the server path uses `common_json`. -- The upstream server library (`server-context.cpp`, `server-queue.cpp`, `server-task.cpp`, `server-schema.cpp`, `server-models.cpp`, and — since b9829 — `server-stream.cpp`) is compiled directly into `jllama` via CMake — there is no hand-ported `server.hpp` fork. **`server-stream.cpp` is mandatory, not optional:** it defines the resumable-streaming SSE replay buffer (`g_stream_sessions`, `stream_session_attach_pipe`, `stream_aware_should_stop`, `stream_conv_id_from_headers`, the `stream_pipe_*` types) that `server-context.cpp` / `server-http.cpp` / `server-models.cpp` now `#include "server-stream.h"` and call, so omitting it fails the link with undefined references. It is platform-neutral (threads + std mutex/condvar, no `subprocess.h`/`posix_spawn_*`), so it builds on Android too and sits outside the `server-models.cpp` Android guard. `jllama` wires its own JNI routes and never calls `g_stream_sessions.start_gc()` (only the excluded standalone `server.cpp` `main()` does), so its GC thread stays dormant. **Phase 2:** the upstream HTTP transport (`tools/server/server-http.cpp`) and its `cpp-httplib` backend (`vendor/cpp-httplib/httplib.cpp`) are now compiled into `jllama` too, so the OpenAI-compatible server can be driven natively from JNI *inside* `libjllama` — no separate `llama-server` executable (a JNI shared library loads anywhere a JVM runs, which a standalone binary does not). `server-http.cpp` does `#include "ui.h"` (the WebUI asset table that `tools/ui`/`llama-ui` normally generates); since the Svelte WebUI is not shipped, `src/main/cpp/webui_stub/ui.h` supplies the upstream **empty-asset** interface and leaves `LLAMA_UI_HAS_ASSETS` undefined (all static-asset-serving blocks compile out). `` already resolves via `llama-common`'s `vendor/` include dir (same nlohmann/json 3.12.0 as the FetchContent copy). No SSL: `CPPHTTPLIB_OPENSSL_SUPPORT` is left undefined (plain-HTTP; bind localhost / front with a TLS proxy). **`server.cpp`, `server-tools.cpp` and `server-mcp.cpp` are now compiled in too** (on non-Android — they pull in `subprocess.h`/`posix_spawn_*`, so they share `server-models.cpp`'s Android guard): b9870 exposes `server.cpp`'s entry as `int llama_server(int, char**)` (no `main` in the file), and `patches/0006` makes it embeddable (no process signal handlers, forwarded-argv parse, out-of-band shutdown). **`server-mcp.cpp` is new in b10154** (upstream MCP-server support): both `server.cpp` (`llama_server`'s `mcp_mgr` lifecycle) and `server-tools.cpp` (`tools.setup(..., mcp_mgr)` / `server_mcp::call_tool`) reference `server_mcp`, so it **must** be in the `target_sources` list or the link fails with undefined `server_mcp::{start,shutdown,call_tool,list_tools,~server_mcp}` — **latent on Linux** (a shared object tolerates undefined symbols) but a **hard link error on macOS/ld64 and Windows/MSVC**. It is compiled only into `jllama`, not `jllama_test` (which links neither `server.cpp` nor `server-tools.cpp`). The `NativeServer` JNI bridge (`src/main/cpp/native_server.cpp`) calls `llama_server` on a worker thread, so the **full** upstream server — WebUI and all — runs inside `libjllama`. See "Two server modes" below. +- The upstream server library (`server-context.cpp`, `server-queue.cpp`, `server-task.cpp`, `server-schema.cpp`, `server-models.cpp`, and — since b9829 — `server-stream.cpp`) is compiled directly into `jllama` via CMake — there is no hand-ported `server.hpp` fork. **`server-stream.cpp` is mandatory, not optional:** it defines the resumable-streaming SSE replay buffer (`g_stream_sessions`, `stream_session_attach_pipe`, `stream_aware_should_stop`, `stream_conv_id_from_headers`, the `stream_pipe_*` types) that `server-context.cpp` / `server-http.cpp` / `server-models.cpp` now `#include "server-stream.h"` and call, so omitting it fails the link with undefined references. It is platform-neutral (threads + std mutex/condvar, no `subprocess.h`/`posix_spawn_*`), so it builds on Android too and sits outside the `server-models.cpp` Android guard. `jllama` wires its own JNI routes and never calls `g_stream_sessions.start_gc()` (only the excluded standalone `server.cpp` `main()` does), so its GC thread stays dormant. **Phase 2:** the upstream HTTP transport (`tools/server/server-http.cpp`) and its `cpp-httplib` backend (`vendor/cpp-httplib/httplib.cpp`) are now compiled into `jllama` too, so the OpenAI-compatible server can be driven natively from JNI *inside* `libjllama` — no separate `llama-server` executable (a JNI shared library loads anywhere a JVM runs, which a standalone binary does not). `server-http.cpp` does `#include "ui.h"` (the WebUI asset table that `tools/ui`/`llama-ui` normally generates); since the Svelte WebUI is not shipped, `src/main/cpp/webui_stub/ui.h` supplies the upstream **empty-asset** interface and leaves `LLAMA_UI_HAS_ASSETS` undefined (all static-asset-serving blocks compile out). `` already resolves through `llama-common` — since upstream #27304 (b10488) not from a `PUBLIC ../vendor` include dir of its own but transitively, via the `vendor::nlohmann` / `vendor::sheredom` INTERFACE targets it links PUBLIC, each of which exports the `vendor/` root (same nlohmann/json 3.12.0 as the FetchContent copy). No SSL: `CPPHTTPLIB_OPENSSL_SUPPORT` is left undefined (plain-HTTP; bind localhost / front with a TLS proxy). **`server.cpp`, `server-tools.cpp` and `server-mcp.cpp` are now compiled in too** (on non-Android — they pull in `subprocess.h`/`posix_spawn_*`, so they share `server-models.cpp`'s Android guard): b9870 exposes `server.cpp`'s entry as `int llama_server(int, char**)` (no `main` in the file), and `patches/0006` makes it embeddable (no process signal handlers, forwarded-argv parse, out-of-band shutdown). **`server-mcp.cpp` is new in b10154** (upstream MCP-server support): both `server.cpp` (`llama_server`'s `mcp_mgr` lifecycle) and `server-tools.cpp` (`tools.setup(..., mcp_mgr)` / `server_mcp::call_tool`) reference `server_mcp`, so it **must** be in the `target_sources` list or the link fails with undefined `server_mcp::{start,shutdown,call_tool,list_tools,~server_mcp}` — **latent on Linux** (a shared object tolerates undefined symbols) but a **hard link error on macOS/ld64 and Windows/MSVC**. It is compiled only into `jllama`, not `jllama_test` (which links neither `server.cpp` nor `server-tools.cpp`). The `NativeServer` JNI bridge (`src/main/cpp/native_server.cpp`) calls `llama_server` on a worker thread, so the **full** upstream server — WebUI and all — runs inside `libjllama`. See "Two server modes" below. ### Two server modes (`OpenAiCompatServer` vs `NativeServer`) @@ -1361,7 +1362,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10615`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10618`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 663fd1b02..2ad164d88 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10615](https://img.shields.io/badge/llama.cpp-%23b10615-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10615) +[![llama.cpp b10618](https://img.shields.io/badge/llama.cpp-%23b10618-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10618) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/TODO.md b/TODO.md index 22893e42f..647db83d9 100644 --- a/TODO.md +++ b/TODO.md @@ -13,6 +13,25 @@ cross-cutting initiative. ## Open — jllama-specific +### `ServerMetrics` describes a JSON shape the native layer stopped emitting at b10408 + +`LlamaModel.getMetrics()` / `getMetricsTyped()` are documented (and their `ServerMetrics` getters are +written) against the *pre-b10408* payload: an object with `idle` / `processing` / `deferred` / +`t_start` / `n_*_total` counters **and** a nested `slots` array. Since llama.cpp **b10408** (upstream +#26920) the underlying result's `to_json()` returns the **slot array verbatim** — the cumulative +counters moved to Prometheus exposition text behind `to_metrics()`, which the JNI layer does not +expose. So `getIdleSlots()`, `getProcessingSlots()`, `getDeferredTasks()`, `getCumulativeUsage()`, +`getSlots()` and `getSlotMetrics()` have all been returning zeros / a missing node since that bump; +only `asJson()` (the raw array) carries real data. + +This is **pre-existing**, not a regression of the b10456 → b10618 walk — that walk only re-pointed +the JNI task at `SERVER_TASK_TYPE_SLOT_GET` so the array keeps arriving at all (b10519 would +otherwise have degraded it to `{}`). Fixing it properly means deciding what the Java surface should +be: either reshape `ServerMetrics` around the slot array and drop the counter getters, or add a +second JNI entry point for `SERVER_TASK_TYPE_METRICS` → `to_metrics()` and parse the Prometheus text. +`ServerMetricsTest` pins the *old* shape from string literals, so it passes either way and is not a +guard here. + ### LlamaLoader extraction-directory isolation (optional follow-up, low priority) Left over from the 2026-06-20 code audit (18/18 findings fixed in PRs #258/#260, regression tests in diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index f80226844..0f695eb84 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -659,7 +659,7 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10585–b10590 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10590 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10590–b10593 | `include/llama.h` (**one comment line: a `[TAG_LLAMA_SEQ_ID_NEG]` marker added to `llama_memory_seq_rm`'s doc block**), `src/llama-*.cpp` (**#26756 DeepseekV4 multi-seq rollback fix; #27574 tensor-parallel meta tensor-split state propagation**), `tools/ui/**` (**#27263: tabbed chat-conversation navigation**) | **No project-source change.** `include/llama.h` is priority 7 on the CLAUDE.md review list, so its diff was read in full: it is a single comment line inside an existing doc block — no signature, no enum value, no struct field moved. The rest is llama.cpp's internal KV/tensor-split implementation and the Svelte WebUI. Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `vendor/` or any `CMakeLists.txt` changed, so no patch context moved. | | b10590–b10593 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10593 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | -| b10593–b10599 | `tools/mtmd/{clip-model.h,clip.cpp,mtmd-image.cpp}` (**#27594 "mtmd: use pillow-accurate algo, correct resize_algo for all models": `enum resize_algo` drops `RESIZE_ALGO_BICUBIC_PILLOW` (plain `BICUBIC` now *means* the Pillow path), the four private `img_tool` statics `resize_{bilinear,bicubic,bicubic_pillow,lanczos_pillow}` and both dispatch switches collapse into one `resize_pillow(..., resize_algo algo)` (−246 net), and ~14 projector types are re-assigned**), `tools/server/server-context.cpp` (**#27600: private `slots_n_diff` member + `LLAMA_SERVER_SLOTS_N_DIFF` getenv; the hardcoded prompt-mismatch debug window `n_past−4..+6` becomes `n_past−slots_n_diff..+slots_n_diff+2`**), `common/common.cpp` (**#26692: `common_params_print_info`'s device-enumeration loop gated on `print_devices && verbosity >= LOG_LEVEL_TRACE`**), `src/llama-grammar.cpp` (**#27591: `parse_char()` accepts `case '-':`, so a `\-` escape produced by `gbnf_escape_char_class()` no longer throws**), `tools/CMakeLists.txt` (**#27548: `add_subdirectory(parser)` dropped — the parser tool moved under `tests/`**) | **No project-source, no project-CMake and no patch change.** The mtmd resize rework is entirely mtmd-internal: `resize_algo`/`clip_hparams` live in `tools/mtmd/clip-model.h`, which the project never includes, and all four deleted functions were `private:` statics inside `struct img_tool`; the public `mtmd.h`/`mtmd-helper.h` are unchanged in this range. The only visible effect is that `MultimodalIntegrationTest`'s SmolVLM preprocessing becomes Pillow-exact — pixel-level, no API or JSON shape. `server-context.cpp`'s new fields are `private:` in `server_context_impl` and **`server-context.h` is unchanged**, so no `to_json()` shape moved and no C++ test assertion is affected; patch `0002`'s and `0003`'s anchors are byte-identical and resolve with a pure offset. `common_params_print_info` keeps its `bool print_devices = true` default, so `jllama.cpp`'s one-argument call still compiles; the behaviour delta is that at the default log threshold libjllama no longer prints `device_info:` (nothing in the project reads it) and no longer triggers early CUDA primary-context creation from that call site. The grammar `\-` fix is strictly widening — a previously-throwing input now parses — and is a **positive** for the project's agentic tool-calling target; no test pinned the old throw. The `tools/CMakeLists.txt` edit sits inside the `LLAMA_BUILD_TOOLS`-gated branch the project never enters. | +| b10593–b10599 | `tools/mtmd/{clip-model.h,clip.cpp,mtmd-image.cpp}` (**#27594 "mtmd: use pillow-accurate algo, correct resize_algo for all models": `enum resize_algo` drops `RESIZE_ALGO_BICUBIC_PILLOW` (plain `BICUBIC` now *means* the Pillow path), the four private `img_tool` statics `resize_{bilinear,bicubic,bicubic_pillow,lanczos_pillow}` and both dispatch switches collapse into one `resize_pillow(..., resize_algo algo)` (−246 net), and ~14 projector types are re-assigned**), `tools/server/server-context.cpp` (**#27600: private `slots_n_diff` member + `LLAMA_SERVER_SLOTS_N_DIFF` getenv; the hardcoded prompt-mismatch debug window `n_past−4..+6` becomes `n_past−slots_n_diff..+slots_n_diff+2`**), `common/common.cpp` (**#26692: `common_params_print_info`'s device-enumeration loop gated on `print_devices && verbosity >= LOG_LEVEL_TRACE`**), `tools/CMakeLists.txt` (**#27548: `add_subdirectory(parser)` dropped — the parser tool moved under `tests/`**) | **No project-source, no project-CMake and no patch change.** The mtmd resize rework is entirely mtmd-internal: `resize_algo`/`clip_hparams` live in `tools/mtmd/clip-model.h`, which the project never includes, and all four deleted functions were `private:` statics inside `struct img_tool`; the public `mtmd.h`/`mtmd-helper.h` are unchanged in this range. The only visible effect is that `MultimodalIntegrationTest`'s SmolVLM preprocessing becomes Pillow-exact — pixel-level, no API or JSON shape. `server-context.cpp`'s new fields are `private:` in `server_context_impl` and **`server-context.h` is unchanged**, so no `to_json()` shape moved and no C++ test assertion is affected; patch `0002`'s and `0003`'s anchors are byte-identical and resolve with a pure offset. `common_params_print_info` keeps its `bool print_devices = true` default, so `jllama.cpp`'s one-argument call still compiles; the behaviour delta is that at the default log threshold libjllama no longer prints `device_info:` (nothing in the project reads it) and no longer triggers early CUDA primary-context creation from that call site. The `tools/CMakeLists.txt` edit sits inside the `LLAMA_BUILD_TOOLS`-gated branch the project never enters. | | b10593–b10599 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10599 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10599–b10612 | `tools/mtmd/mtmd-helper.cpp` (**#27596 "video: fix moov atom at the end of file": `subprocess_handle` splits `alive` into `created`+`alive`, `stop()` gains `subprocess_join()` to reap zombies plus a Windows `CloseHandle(proc.hStdInput)`, `start_feeder()` gains `pthread_sigmask(SIG_BLOCK, {SIGPIPE})` and an `#ifdef F_SETNOSIGPIPE fcntl(...)`, and the ffmpeg argv gains `-read_ahead_limit -1`; new `#include //` under `#ifdef MTMD_VIDEO`/`#ifndef _WIN32`**), `ggml/**` (**#27644 `ggml_clamp` fix; #27608 shorter virtual-device naming; #27545 WebGPU include order**), `src/**` (**#26534 MTP in GLM-4.5-Air; #26490 Deepseek 4 `-sm tensor`; #27513 mamba2 GEMM dispatch**), `convert_hf_to_gguf.py`, `tools/ui/**` | **No project-source change.** `mtmd-helper.cpp` **is** compiled into `libjllama` (the project `add_subdirectory`s `tools/mtmd` and never sets `LLAMA_SUBPROCESS`, so `MTMD_VIDEO` is ON on every classifier including Android), so its three new build-surface dependencies were each checked against the pinned tree rather than assumed: `subprocess_join` is declared in `vendor/sheredom/subprocess.h`, `struct subprocess_s` carries `void *hStdInput` (so the Windows `CloseHandle` compiles), `pthread_sigmask` resolves through the pre-existing `target_link_libraries(mtmd PRIVATE Threads::Threads …)`, and `F_SETNOSIGPIPE` is `#ifdef`-guarded off on Linux/Android. `mtmd.h` and `mtmd-helper.h` are unchanged in this range, so `tts_engine.cpp`'s `mtmd_helper::gen_audio` surface is untouched. Nothing under `common/`, `tools/server/`, `include/` or any project-consumed `CMakeLists.txt` changed — no patch context moved. | | b10599–b10612 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10612 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | @@ -667,3 +667,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10612–b10614 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10614 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10614–b10615 | `ggml/src/ggml-metal/**` + `tools/CMakeLists.txt` (**#26570: per-device tuned `(Q, NE)` for the flash-attention vec kernels; a Metal-only `tuning` tool is added under an `if (GGML_METAL)` guard**) | **No project-source change.** The `tools/CMakeLists.txt` addition sits inside the `LLAMA_BUILD_TOOLS`-gated branch the project never enters (`LLAMA_BUILD_TOOLS` defaults to `LLAMA_STANDALONE`, i.e. OFF under FetchContent, and the project forces it OFF on Android), and is additionally Metal-only. Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/` or `vendor/` changed — no patch context moved, no header in the project's include graph touched. | | b10614–b10615 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10615 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | +| b10615–b10618 | `src/llama-grammar.cpp` (**#27591: `parse_char()` accepts `case '-':`, so a `\-` escape produced by `gbnf_escape_char_class()` inside a character class no longer throws**), `ggml/src/ggml-sycl/**` (**#27660: `tq2_0` marked unsupported**), `ggml/src/ggml-webgpu/wgsl-shaders/argsort.wgsl` (**#27538: infinity handling in ARGSORT / TOP_K**) | **No project-source change (final step).** Zero lines changed under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt`, so no patch context moved. The grammar fix is strictly *widening* — an input that previously threw now parses — and reaches the project through both `LlamaModel.jsonSchemaToGrammar` and every grammar-constrained completion, so it is a **positive** for the agentic tool-calling target; no test pinned the old throw (`grep -rn 'gbnf_escape_char_class\|parse_char' src/main/cpp src/test/cpp` is empty). The SYCL and WebGPU edits only affect the `sycl-*` classifiers' compiled sources and a backend the project does not build. | +| b10456–b10618 | upstream verification (sandbox, final target) | **Full local verification.** Fresh `cmake -B build` with the real `FetchContent` path (no source-dir override), so the fail-loud `PATCH_COMMAND` ran for real: it resolved `ggml commit eb25b7263` / `LLAMA_BUILD_NUMBER = 10618` and applied all **6** patches — verified by grepping the fetched tree for each patch's marker (`common_params_parse_main`, `params_base.load_progress_callback == nullptr`, `get_slot_prompt_similarity`, `g_llama_server_embedded`, `llama_server_attach` + `llama_server_register_common_routes`, `LLAMA_SERVER_WORKER_CMD`). Then a full `cmake --build --config Release` (jllama + jllama_test both link — the `jllama_test` link is what proves `common_json::get()` resolves, since upstream documents un-specialised types as a *link*-time failure) and `ctest`: **491/491 C++ tests pass**. On the Java side: `mvn test-compile` clean, `mvn test` **1405 run / 0 failures / 17 skipped** (only the model-gated integration tests, no GGUF in the sandbox), and `NativeLibraryLoadSmokeTest` green — including `nativeBuildInfoMatchesPinnedVersionConstant`, the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10618") matches the `build-info` compiled into the freshly linked `libjllama.so`. `mvn spotless:apply` produced no changes beyond the edits themselves, the pinned **clang-format 22.1.8** reports the whole C++ tree clean, and `mvn clean javadoc:jar` is `BUILD SUCCESS`. **Per-step verification for all 25 chunks:** every intermediate tag additionally had all six patches applied by the applier's own commands and all ten project + C++-test translation units compiled against it, so no commit in this walk is a state that fails to build. | diff --git a/docs/upgrade/llama-cpp-version-bump.md b/docs/upgrade/llama-cpp-version-bump.md index 9be1fe07f..44389226b 100644 --- a/docs/upgrade/llama-cpp-version-bump.md +++ b/docs/upgrade/llama-cpp-version-bump.md @@ -110,8 +110,9 @@ Once you have the `b -> b` step, apply it exactly as Concretely: 1. **Edit the pin — four files:** - - `llama/CMakeLists.txt` — the `GIT_TAG b` line **and** the `-DLLAMA_TAG=b` used by the - WebUI/TTS extraction (both must move together). + - `llama/CMakeLists.txt` — the `GIT_TAG b` line. (It is the only tag in this file; the + `-DLLAMA_TAG=b` that once fed the build-time TTS extraction was removed with the + Qwen3-TTS rework, and the WebUI auto-follows `GIT_TAG` in CI.) - `README.md` — the llama.cpp badge and link (version appears twice). - `CLAUDE.md` — the "Current llama.cpp pinned version" line (and any build-example `b`). - `llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java` — the `LLAMA_CPP_VERSION` @@ -119,9 +120,9 @@ Concretely: if you forget it, `NativeLibraryLoadSmokeTest.nativeBuildInfoMatchesPinnedVersionConstant` fails the build (it cross-checks the constant against `LlamaModel.getLlamaCppBuildInfo()`, which reads llama.cpp's own linked-in `build-info`). -2. **Re-verify `patches/`** — a clean configure re-runs the fail-loud `PATCH_COMMAND`, so every patch - `0001`–`0006` must still apply. Use a **fresh** build dir (a stale one re-applies over an - already-patched tree and reports a false "does not apply"): +2. **Re-verify `patches/`** — a clean configure re-runs the fail-loud `PATCH_COMMAND`, so all six + patches (`0001`, `0002`, `0003`, `0006`, `0007`, `0008`) must still apply. Use a **fresh** build + dir (a stale one re-applies over an already-patched tree and reports a false "does not apply"): ```bash cd llama && mvn -q compile # generates the OSInfo class CMake's OS-detection needs rm -rf build && cmake -B build # fail-loud: aborts here if any patch no longer applies diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 794987cd0..4eb832007 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10615 + GIT_TAG b10618 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= @@ -451,9 +451,12 @@ endif() # shared to every native build as a generated, platform-independent ui.cpp/ui.h; # the "WebUI assets" block below compiles it in when present and otherwise falls # back to the empty-asset stub (src/main/cpp/webui_stub/ui.h). -# already resolves via llama-common's vendor/ include dir, -# whose bundled nlohmann/json is the same 3.12.0 as our FetchContent copy, so -# adding nothing there shadows it. +# already resolves through llama-common: upstream #27304 +# (b10488) replaced its `PUBLIC . ../vendor` include dir with PUBLIC links to the +# vendor::nlohmann / vendor::sheredom INTERFACE targets, and each of those exports +# the vendor/ root (`target_include_directories( INTERFACE ..)`), so the same +# directory still reaches jllama transitively. Its bundled nlohmann/json is the +# same 3.12.0 as our FetchContent copy, so nothing added here shadows it. target_sources(jllama PRIVATE ${llama.cpp_SOURCE_DIR}/tools/server/server-http.cpp ${llama.cpp_SOURCE_DIR}/vendor/cpp-httplib/httplib.cpp diff --git a/llama/src/main/cpp/utils.hpp b/llama/src/main/cpp/utils.hpp index ea00ca3ca..25f063c43 100644 --- a/llama/src/main/cpp/utils.hpp +++ b/llama/src/main/cpp/utils.hpp @@ -5,7 +5,7 @@ #pragma once -// server-common.h provides: JSON_ASSERT, json, raw_buffer, json_value, +// server-common.h provides: json (= common_json since b10585), raw_buffer, json_value, // server_grammar_trigger, server_tokens, error_type, SRV_* macros, // and many utility function declarations (implemented in server-common.cpp). #include "server-common.h" diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index b35dbbe28..06c70428d 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10615"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10618"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10615-"} — call + * plus the resolved upstream commit, e.g. {@code "b10618-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10615"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10618"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10615"; + public static final String LLAMA_CPP_VERSION = "b10618"; // Constants holder — not instantiable. private LlamaCppVersion() {} diff --git a/llama/src/test/cpp/test_server.cpp b/llama/src/test/cpp/test_server.cpp index 8d857247b..ae402a07b 100644 --- a/llama/src/test/cpp/test_server.cpp +++ b/llama/src/test/cpp/test_server.cpp @@ -822,9 +822,11 @@ TEST(ServerTaskResultSlots, ToJson_EmptyByDefault) { EXPECT_EQ(m.n_idle_slots, 0); } -TEST(ServerTaskResultMetrics, ToJson_UnusedEmptyObject) { - // /metrics renders Prometheus text via to_metrics(); to_json() is not used any more - // and returns an empty object since b10519. +TEST(ServerTaskResultMetrics, ToJson_UnusedAndEmpty) { + // /metrics renders Prometheus text via to_metrics(); to_json() is not used any more and + // since b10519 just returns `json{}` — which selects the default constructor, i.e. JSON + // null rather than an empty object (true for nlohmann and for common_json alike). Assert + // only what upstream promises: it carries nothing. server_task_result_metrics m = make_metrics(); const json j = m.to_json(); EXPECT_TRUE(j.empty()); diff --git a/llama/src/test/cpp/test_utils.cpp b/llama/src/test/cpp/test_utils.cpp index fac549c3c..db724b44a 100644 --- a/llama/src/test/cpp/test_utils.cpp +++ b/llama/src/test/cpp/test_utils.cpp @@ -648,7 +648,9 @@ TEST(JsonArrayChecks, ArrayOfIntegers_IsNumbers) { EXPECT_TRUE(json_is_array_of_ TEST(JsonArrayChecks, EmptyArray_IsNumbers) { EXPECT_TRUE(json_is_array_of_numbers(json::array())); } -TEST(JsonArrayChecks, ArrayWithString_NotNumbers) { EXPECT_FALSE(json_is_array_of_numbers(json::array({1, "hello", 3}))); } +TEST(JsonArrayChecks, ArrayWithString_NotNumbers) { + EXPECT_FALSE(json_is_array_of_numbers(json::array({1, "hello", 3}))); +} TEST(JsonArrayChecks, NonArray_NotNumbers) { EXPECT_FALSE(json_is_array_of_numbers(json("just a string"))); @@ -659,9 +661,13 @@ TEST(JsonArrayChecks, MixedNumbersAndStrings_IsMixed) { EXPECT_TRUE(json_is_array_of_mixed_numbers_strings(json::array({1, "hello", 3}))); } -TEST(JsonArrayChecks, OnlyNumbers_NotMixed) { EXPECT_FALSE(json_is_array_of_mixed_numbers_strings(json::array({1, 2, 3}))); } +TEST(JsonArrayChecks, OnlyNumbers_NotMixed) { + EXPECT_FALSE(json_is_array_of_mixed_numbers_strings(json::array({1, 2, 3}))); +} -TEST(JsonArrayChecks, OnlyStrings_NotMixed) { EXPECT_FALSE(json_is_array_of_mixed_numbers_strings(json::array({"a", "b"}))); } +TEST(JsonArrayChecks, OnlyStrings_NotMixed) { + EXPECT_FALSE(json_is_array_of_mixed_numbers_strings(json::array({"a", "b"}))); +} TEST(JsonArrayChecks, EmptyArray_NotMixed) { EXPECT_FALSE(json_is_array_of_mixed_numbers_strings(json::array())); } From 7ad392edf18d9ba4b899d2be4dc08f3e315cbaae Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 10:49:30 +0000 Subject: [PATCH 26/76] refactor: harden against the common_json conversion trap The b10585 trap was that a common_json still binds to a const nlohmann::json parameter through operator std::string(). The fix so far was a template plus a warning comment; make it structural by dropping the now-unused nlohmann/json.hpp include from jni_helpers.hpp, json_helpers.hpp and jllama.cpp, so that conversion is not reachable from those translation units at all. test_jni_helpers.cpp keeps testing the helper with a real nlohmann::json and now includes the header itself. log_helpers.hpp and train_engine.cpp are unaffected -- they use nlohmann deliberately. Also corrects a comment: the split-off METRICS to_json() returns json{}, which is JSON null, not an empty object. TODO.md records three pre-existing defects found while verifying the bump, none of them introduced by it: model-gated Java tests silently self-skip in CI (Surefire resolves models/ against llama/, CI restores it to the repo root -- verified with a throwaway probe); the getMetrics() payload became a bare slot array at b10408 and ServerMetrics, LlamaModelTest#testGetMetrics and OpenAiCompatServer /slots all still assume the old object shape; and the patch applier cannot detect an already-applied 0001 because 0006/0007 rewrite the same region of server.cpp, so any reconfigure of an existing build dir fails. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- TODO.md | 93 ++++++++++++++++++++----- llama/src/main/cpp/jllama.cpp | 8 +-- llama/src/main/cpp/jni_helpers.hpp | 1 - llama/src/main/cpp/json_helpers.hpp | 2 - llama/src/test/cpp/test_jni_helpers.cpp | 4 ++ 5 files changed, 83 insertions(+), 25 deletions(-) diff --git a/TODO.md b/TODO.md index 647db83d9..19027605c 100644 --- a/TODO.md +++ b/TODO.md @@ -13,24 +13,81 @@ cross-cutting initiative. ## Open — jllama-specific -### `ServerMetrics` describes a JSON shape the native layer stopped emitting at b10408 - -`LlamaModel.getMetrics()` / `getMetricsTyped()` are documented (and their `ServerMetrics` getters are -written) against the *pre-b10408* payload: an object with `idle` / `processing` / `deferred` / -`t_start` / `n_*_total` counters **and** a nested `slots` array. Since llama.cpp **b10408** (upstream -#26920) the underlying result's `to_json()` returns the **slot array verbatim** — the cumulative -counters moved to Prometheus exposition text behind `to_metrics()`, which the JNI layer does not -expose. So `getIdleSlots()`, `getProcessingSlots()`, `getDeferredTasks()`, `getCumulativeUsage()`, -`getSlots()` and `getSlotMetrics()` have all been returning zeros / a missing node since that bump; -only `asJson()` (the raw array) carries real data. - -This is **pre-existing**, not a regression of the b10456 → b10618 walk — that walk only re-pointed -the JNI task at `SERVER_TASK_TYPE_SLOT_GET` so the array keeps arriving at all (b10519 would -otherwise have degraded it to `{}`). Fixing it properly means deciding what the Java surface should -be: either reshape `ServerMetrics` around the slot array and drop the counter getters, or add a -second JNI entry point for `SERVER_TASK_TYPE_METRICS` → `to_metrics()` and parse the Prometheus text. -`ServerMetricsTest` pins the *old* shape from string literals, so it passes either way and is not a -guard here. +### Model-gated Java tests silently self-skip in CI (Surefire's working directory) + +**Found while bumping to b10618; pre-existing and independent of that bump.** Every model-gated Java +test decides whether to run with a *relative* path — `new File("models/codellama-7b.Q2_K.gguf")` +(`LlamaModelTest`), `TestConstants.MODEL_PATH`, the `-Dnet.ladenthin.llama.*` model properties CI +passes as `models/`. Surefire resolves those against its working directory, which defaults to +the **module** basedir. CI runs `mvn -f llama/pom.xml … test` while the shared GGUF cache is restored +to `/models/`, so the tests look in `/llama/models/` and find nothing. + +Verified empirically, not inferred: a throwaway test run under `mvn -f llama/pom.xml` reported +`user.dir=/llama`, `models/ exists=false`, `models/ +exists=true`. + +Consequence: the whole model-backed Java suite — `LlamaModelTest`, `LlamaEmbeddingsTest`, +`MultimodalIntegrationTest`, `TtsIntegrationTest`, the reranking/draft/tool-model tests — aborts in +`@BeforeAll` and reports as skipped on every `test-java-*` job, while the job still goes green. That +contradicts the "CI model policy" section of `CLAUDE.md`, which states these run on every platform, +and it is why the two stale assertions below have never failed. + +Fixing it is a one-line change (a Surefire `${project.basedir}/..`, +or making the paths module-relative) — but it will **immediately turn ~100 currently-skipped tests +on**, including the known-stale ones, so it must be done as its own change with the fallout fixed in +the same PR, not folded into an unrelated bump. + +### The `getMetrics()` payload contract drifted at b10408 and three consumers still assume the old one + +`LlamaModel.getMetrics()` returns whatever the native slot-introspection task's `to_json()` produces. +Until llama.cpp **b10408** (upstream #26920) that was an object — `idle` / `processing` / `deferred` / +`t_start` / `n_*_total` counters plus a nested `slots` array. Since b10408 it is the **slot array +verbatim**; the cumulative counters moved to Prometheus exposition text behind `to_metrics()`, which +the JNI layer does not expose at all. Three consumers were written against the old shape and are all +still on it: + +- **`value.ServerMetrics`** — `getIdleSlots()`, `getProcessingSlots()`, `getDeferredTasks()`, + `getCumulativeUsage()`, `getSlots()`, `getSlotMetrics()` all read keys that no longer exist, so + they return zeros / a missing node. Only `asJson()` (the raw array) carries real data. + `ServerMetricsTest` builds its input from string literals, so it passes either way and is not a + guard. +- **`LlamaModelTest#testGetMetrics`** — asserts the live payload `contains("\"slots\"")` and + `contains("\"idle\"")`. Neither substring occurs anywhere in the current payload (checked against + upstream's `server_slot::to_json` and `task_params::to_json` key lists). It does not fail today + only because of the Surefire working-directory issue above; fixing that surfaces this immediately. +- **`server.OpenAiCompatServer.handleMetricsView`** — `GET /slots` does + `readTree(metrics).path("slots").toString()`, and `path()` on an array node is a `MissingNode` + whose `toString()` is the empty string, so the route answers **200 with a zero-length body**. Its + sibling `GET /metrics` serves the slot array under a name that no longer describes it. + `OpenAiCompatServerHttpTest`'s fake backend returns the *pre-b10408* object, so the test exercises + a shape production never produces — the fake is what hides the bug. + +**Not a regression of the b10456 → b10618 walk.** That walk only re-pointed the JNI task at the new +`SERVER_TASK_TYPE_SLOT_GET`, which is what keeps the array arriving at all (b10519 split the old +`METRICS` task and would otherwise have degraded the payload to JSON null). + +Fixing it means picking the Java-side contract first: either reshape `ServerMetrics` (and `/slots`) +around the slot array and drop the counter getters, or add a second JNI entry point for +`SERVER_TASK_TYPE_METRICS` → `to_metrics()` and serve/parse the Prometheus text. Whichever is chosen, +update all three consumers, the `OpenAiCompatServerHttpTest` fake, and the `/metrics` route naming +together. + +### `apply-llama-patches.cmake` is not idempotent when two patches touch one file + +`CLAUDE.md` and the applier's own header describe it as idempotent: a `git apply --reverse --check` +is supposed to detect an already-applied patch and skip it, so a CMake **re**configure over an +already-patched source tree is a no-op. That holds only while no two patches touch the same file. +`0001` patches `tools/server/server.cpp` (one of its ~34 `common_params_parse_main` call-site flips) +and `0006`/`0007` then rewrite the same region, so `0001`'s reverse-check no longer matches; the +applier falls through to a forward apply, which also fails, and aborts the configure with the +misleading "does not apply cleanly — a llama.cpp version bump probably shifted the patched code". + +Confirmed at b10618: reverse-check `0001` → fails at `tools/server/server.cpp:102`; `0006` and +`0007` → skip cleanly. The practical effect is that **any** reconfigure of an existing build dir +fails, which is why the runbook says to use a fresh one. A fix would be to reverse-check in reverse +filename order (`0008` → `0001`), or to record an applied-marker per patch in the build tree. Left +alone for now because it is release-critical build machinery and the fresh-build-dir workaround is +already documented. ### LlamaLoader extraction-directory isolation (optional follow-up, low priority) diff --git a/llama/src/main/cpp/jllama.cpp b/llama/src/main/cpp/jllama.cpp index ea476b321..2161e7b95 100644 --- a/llama/src/main/cpp/jllama.cpp +++ b/llama/src/main/cpp/jllama.cpp @@ -10,7 +10,6 @@ #include "json-schema-to-grammar.h" #include "llama.h" #include "log.h" -#include "nlohmann/json.hpp" #include "server-context.h" #include "server-queue.h" #include "server-task.h" @@ -1628,9 +1627,10 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleSlotAction(J switch (action) { case 0: // LIST — get slot info via the dedicated slot-get task // b10519 (upstream #27376) split the old METRICS task in two: METRICS now carries only the - // cumulative counters (rendered as Prometheus text by to_metrics(); its to_json() returns an - // empty object), and the /slots payload moved to SERVER_TASK_TYPE_SLOT_GET -> - // server_task_result_slots::to_json(), which returns the slot array verbatim. + // cumulative counters (rendered as Prometheus text by to_metrics(); its to_json() is unused + // and returns `json{}`, i.e. JSON null), and the /slots payload moved to + // SERVER_TASK_TYPE_SLOT_GET -> server_task_result_slots::to_json(), which returns the slot + // array verbatim. return dispatch_one_shot_task(env, ctx_server, server_task(SERVER_TASK_TYPE_SLOT_GET)); case 1: // SAVE return exec_slot_file_task(env, ctx_server, slotId, jfilename, SERVER_TASK_TYPE_SLOT_SAVE, diff --git a/llama/src/main/cpp/jni_helpers.hpp b/llama/src/main/cpp/jni_helpers.hpp index 94823ff28..0e70f7cde 100644 --- a/llama/src/main/cpp/jni_helpers.hpp +++ b/llama/src/main/cpp/jni_helpers.hpp @@ -26,7 +26,6 @@ // including translation unit BEFORE this header. #include "jni.h" -#include "nlohmann/json.hpp" #include #include diff --git a/llama/src/main/cpp/json_helpers.hpp b/llama/src/main/cpp/json_helpers.hpp index 6bb075654..46cf9e858 100644 --- a/llama/src/main/cpp/json_helpers.hpp +++ b/llama/src/main/cpp/json_helpers.hpp @@ -35,8 +35,6 @@ // 8. parse_positive_int_config — used by nothing above it // 9. wrap_stream_chunk — used by nothing above it -#include "nlohmann/json.hpp" - #include #include #include diff --git a/llama/src/test/cpp/test_jni_helpers.cpp b/llama/src/test/cpp/test_jni_helpers.cpp index 19504a40b..b81feb527 100644 --- a/llama/src/test/cpp/test_jni_helpers.cpp +++ b/llama/src/test/cpp/test_jni_helpers.cpp @@ -30,6 +30,10 @@ #include "server-chat.h" #include "utils.hpp" #include "jni_helpers.hpp" +// The require_json_field_impl tests below deliberately exercise the helper with a real +// nlohmann::json as well as with the upstream `json` alias (common_json since b10585), so this +// TU needs nlohmann explicitly — the project headers no longer pull it in. +#include "nlohmann/json.hpp" #include #include #include From aa6701572d44d7a09253a01a3c598ffdd35866dc Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 10:56:54 +0000 Subject: [PATCH 27/76] fix: carry a patch for /models emitting vocab_type as boolean The enum-to-bool trap fixed in jllama.cpp during the b10585 step also hit upstream own get_res_model_info(), which builds the GET /models and GET /v1/models payload and emits {"vocab_type", meta.model_vocab_type} -- an unscoped enum. common_json_value integral ctor template is is_integral gated, which excludes enums, so it binds to the bool ctor and the numeric vocab type goes on the wire as true/false. It was correct while the alias was nlohmann::ordered_json, so upstream regressed it silently in #27511. This ships: server-context.cpp is compiled into libjllama and both routes are served by NativeServer, the default fat-jar Main-Class, in full and in attach mode. patches/0009 casts the value to int at the emit site, mirroring what jllama.cpp already does. A sweep confirmed it is the only such site in the upstream server sources the project compiles: server_context_meta has exactly two enum-typed members and the other, pooling_type, is never serialised. Also corrects two history rows that were scoped too narrowly (the b10585 row claimed no JSON output shape changed anywhere; the b10507 row dismissed hidden router models without accounting for NativeServer raw argv passthrough), and records two RouterClient limitations found by the same review: it cannot send an API key, and /models stopped being a public endpoint at b10519. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 1 + TODO.md | 32 +++++++++++++++++++ docs/history/llama-cpp-breaking-changes.md | 5 +-- ...rver-cast-vocab-type-for-common-json.patch | 16 ++++++++++ .../ladenthin/llama/server/RouterClient.java | 8 +++++ 5 files changed, 60 insertions(+), 2 deletions(-) create mode 100644 llama/patches/0009-server-cast-vocab-type-for-common-json.patch diff --git a/CLAUDE.md b/CLAUDE.md index e042d6d3f..dabb60fa9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -661,6 +661,7 @@ Current patches: | `0003-pr22393-server-add-slot-prompt-similarity-getter-setter.patch` | **Upstream-PR carry** of [ggml-org/llama.cpp#22393](https://github.com/ggml-org/llama.cpp/pull/22393) ("server : add slot_prompt_similarity getter/setter"). Purely additive: adds `server_context::get_slot_prompt_similarity()` / `set_slot_prompt_similarity(float)` (`tools/server/server-context.{cpp,h}`) so an embedding/JNI caller can query and tune the slot-selection threshold at runtime without reloading the model. Verbatim copy of the PR, which **upstream closed without merging** (rejected as exposing unsafe internal state — see the patch header). Carried permanently; it will not be droppable via a version bump. | | `0007-server-attach-http-frontend.patch` | **Adds `llama_server_attach(argc, argv, server_context&)`** so the `NativeServer` *attach mode* can serve an **already-loaded `LlamaModel`** over the upstream HTTP frontend — no second model load, no `start_loop()`; the LlamaModel's worker keeps driving the shared `server_context` and the HTTP routes post tasks to its queue (the queue is the synchronization point). Mechanically: (1) extracts the **pure core route table** (`health` … `slots`) out of `llama_server()` into `static void llama_server_register_common_routes(ctx_http, routes)` (shared, so the two entry points cannot drift on the core endpoint set). **Scope note (narrowed at the b10154 bump):** the helper deliberately carries **only** the stable, state-independent route table — **not** the resumable-streaming routes (their handlers differ between router / non-router), the GCP-compat shim, or the experimental **CORS-proxy / MCP-server / built-in-tools** wiring. b10154 (upstream MCP-server support) moved the streaming routes into the middle of that block and coupled tools/CORS to a per-call `server_mcp mcp_mgr` lifecycle, so the earlier contiguous "route-table + CORS-proxy + tools" extraction is no longer possible; `llama_server()` keeps all of that inline, **byte-identical to upstream b10154** (only the route-table block is factored out). (2) adds `llama_server_attach`, which parses only the HTTP-side argv via `common_params_parse`, starts the stream-session GC + `server_http_context`, registers the common route table, the **non-router** resumable-streaming handlers (upstream b10154 paths `/v1/stream` GET/DEL + `/v1/streams/lookup` POST), the GCP-compat shim, and **403 "disabled" stubs for `/cors-proxy` + `/tools`** (attach mode does not wire the experimental CORS-proxy / MCP / built-in-tools host — those belong to a full `llama-server`, not an embedded model), marks ready immediately (model already loaded), and blocks on the HTTP thread until `llama_server_request_shutdown()` — never calling `common_init()`, backend init, `ctx_server.terminate()` or `llama_backend_free()` (the embedding caller owns those). Applies after `0001`+`0006` (same file); closes the "NativeServer — reuse an already-loaded LlamaModel" TODO. Upstream-submittable ("server: let embedding callers attach the HTTP frontend to an existing server_context"). **Refreshed at the b10519 bump:** upstream #26347 dropped the API key from the `/models` + `/v1/models` public-endpoint set and deleted the two trailing `// public endpoint (no API key check)` comments on those route registrations. Those two lines sit inside this patch's route-table removal block, so `git apply` failed ("patch does not apply", `server.cpp:258`) at **every** tag from b10519 on; the fix was to drop the now-wrong comment from all four affected lines (2 on the `-` side, 2 in the extracted helper on the `+` side), keeping the helper byte-identical to the block it replaces. **This is the invariant to re-check on every bump:** the `+` side of `llama_server_register_common_routes()` must stay a verbatim copy of the route table it factors out of `llama_server()`. | | `0008-server-models-worker-cmd-override.patch` | **Makes router mode usable in-JVM.** The router (`server-models.cpp`) spawns each model worker by re-executing its own binary (`get_server_exec_path()` = `/proc/self/exe` & friends) — inside a JVM that binary is `java`, not a llama-server, so embedded router workers could never start. The patch adds env `LLAMA_SERVER_WORKER_CMD` (whitespace-split; read in `server_model_meta::update_args`) which replaces only the leading binary-path token of the rendered worker args, letting an embedding host relaunch workers through its own bootstrap — e.g. `java -cp app.jar net.ladenthin.llama.server.NativeServer` (each worker is then a fresh JVM running the classic single-model `NativeServer`). Exposed in Java as `NativeServer.setWorkerCommand(String...)` (JNI `setenv`); exercised by `RouterModeIntegrationTest` (Linux CI). Upstream-submittable (also useful for containerized/wrapped deployments). | +| `0009-server-cast-vocab-type-for-common-json.patch` | **Upstream regression from the b10585 `common_json` switch (#27511), one line.** `get_res_model_info()` (`tools/server/server-context.cpp`) builds the `GET /models` + `GET /v1/models` payload and emits `{"vocab_type", meta.model_vocab_type}` — an **unscoped enum**. `common_json_value`'s integral constructor template is `std::is_integral`-gated, which *excludes* enums, so the value binds to `common_json_value(bool)` and serialises as `true`/`false` instead of the numeric vocab type. It was correct while the alias was `nlohmann::ordered_json` (nlohmann serialises an enum as an integer), so upstream regressed it silently when they flipped the alias. The project ships this: `server-context.cpp` is compiled into `libjllama` and both routes are served by `NativeServer` — the default fat-jar `Main-Class` — in full **and** attach mode (`patches/0007`'s common route table registers them). The patch casts the value to `int` at the emit site, mirroring what `jllama.cpp` does for its own two `"vocab_type"` sites. Upstream-submittable. Applies after `0002`/`0003` (same file). See the `CommonJsonEnumTrap` tests in `test_json_helpers.cpp` for the mechanism. | | `0006-server-embed-native-server-jni.patch` | **Makes `server.cpp`'s `llama_server` embeddable in the JVM** so the `NativeServer` JNI bridge can run the full upstream HTTP server (WebUI included) inside `libjllama` — see "Two server modes" below. b9870 already exposes `int llama_server(int, char**)` (non-static; no `main` in the file), so the patch only adds embedded-mode support: (1) a `g_llama_server_embedded` flag + `llama_server_set_embedded()` / `llama_server_request_shutdown()` (declared in the committed `src/main/cpp/native_server_bridge.h`); (2) skips installing the process-wide SIGINT/SIGTERM handlers when embedded (they would hijack the JVM's); (3) in embedded mode parses the **forwarded** argv via `common_params_parse` instead of `common_params_parse_main` (whose `GetCommandLineW` recovery would pick up `java.exe`'s command line — the same Windows class of bug `0001` fixes). `llama_server_request_shutdown()` mirrors the SIGTERM path (invokes the installed `shutdown_handler` → `ctx_server.terminate()` unblocks `start_loop()`), giving JNI an out-of-band stop since `ctx_server` is loop-local. Applies **after `0001`** (which flips this call site to `common_params_parse_main`), so its context is the post-`0001` tree; regenerate against `0001`+source on a bump. Only touches `tools/server/server.cpp`. | **`0009` was dropped at the b10280 bump.** Upstream merged diff --git a/TODO.md b/TODO.md index 19027605c..f701b1b9c 100644 --- a/TODO.md +++ b/TODO.md @@ -72,6 +72,38 @@ around the slot array and drop the counter getters, or add a second JNI entry po update all three consumers, the `OpenAiCompatServerHttpTest` fake, and the `/metrics` route naming together. +### `RouterClient` has no API-key support, and `/models` stopped being a public endpoint at b10519 + +`server.RouterClient` sets only `Content-Type` on its requests — there is no `Authorization` header +and no constructor parameter for a key. `POST /models/load` and `/models/unload` have always gone +through the server's API-key middleware, so those two already failed against a router started with +`--api-key`. Upstream **#26347** (b10519) additionally removed `/models` and `/v1/models` from the +public-endpoint set, so `listModels()`, `findModel()` and `awaitModelLoaded()` now answer `401` +there as well — the whole typed router API is unusable against an authenticated router. + +Found while bumping to b10618; the upstream change is noted in the `b10509–b10519` row of +`docs/history/llama-cpp-breaking-changes.md`. CI is unaffected because `RouterModeIntegrationTest` +starts its router without a key, which is also why nothing caught it. + +Fix: add an optional key — e.g. a `RouterClient(String host, int port, String apiKey)` overload that +sets `Authorization: Bearer ` in the private `request()` helper. Deliberately not done as part +of the version bump: it adds public API surface, which is the owner's design call. The limitation is +documented in `RouterClient`'s class javadoc in the meantime. + +### `RouterClient.awaitModelLoaded` rejects hidden-but-loadable router models + +b10507 (upstream #27346) added `server_model_meta::hidden` and a `continue` in the `GET /models` +handler; upstream's own comment is "hidden from GET /models, but still accept if requested" — a +hidden model still loads and still serves by name. `awaitModelLoaded` treats absence from the +listing as a hard error and fails fast, so `loadModel(id)` succeeds, the worker comes up, and +`awaitModelLoaded(id)` throws a message that sends the user off to check `--models-dir` and the +identifier. Only reachable when a preset opts in with `dedup-cache-models`, which the project's own +code never writes — but `NativeServer` forwards raw llama-server argv verbatim by design, so a +caller can enable it. + +Fix: either retry until the timeout instead of rejecting on the first poll (a hidden model still +reaches `LOADED`), or widen the error message to name `dedup-cache-models` as a cause. + ### `apply-llama-patches.cmake` is not idempotent when two patches touch one file `CLAUDE.md` and the applier's own header describe it as idempotent: a `git apply --reverse --check` diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 0f695eb84..96c26c30e 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -629,7 +629,7 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10481–b10488 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10488 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Because this is the range that deletes `llama-common`'s `PUBLIC ../vendor` include directory, the two upstream TUs the project compiles itself that depend on it — `tools/server/server-http.cpp` (`#include `) and `vendor/cpp-httplib/httplib.cpp` — were additionally compiled against the patched b10488 tree, both clean: the include still resolves transitively through the PUBLIC `vendor::nlohmann` / `vendor::sheredom` INTERFACE targets. Full link + `ctest` deferred to the b10618 target. | | b10488–b10499 | `common/common.cpp` (**#27138: threadpool sharing when only `n_threads` differs — new file-static `can_share_threadpool()`; a `-t`/`-tb` mismatch yields ONE unpaused pool sized `max(n, nb)` instead of two**), `tools/server/server-models.cpp` (**#27347: `CMD_CHILD_TO_ROUTER_STATE` lines demoted `LOG` → `LOG_DBG`, other child output forwarded in a new `else` branch**), `tools/mtmd/models/deepseekocr.cpp` (**#26727: DeepSeek-OCR SAM convolutions re-expressed via a file-static `conv_2d_f32()` = `ggml_im2col` + `ggml_mul_mat` keeping the im2col in F32**), plus out-of-scope `tools/ui/**`, `ggml/src/ggml-{opencl,sycl}/**`, `ggml/include/ggml-rpc.h`, `src/llama-model-loader.h` | **No project-source change.** All three in-scope edits are behavioural/internal with no API surface moved: `common/common.h` is byte-identical across the range (so `struct common_threadpools` and every signature are unchanged) and no project source references `threadpool`/`ggml_threadpool_params_match`; the `server-models.cpp` edit is pure logging (`RouterModeIntegrationTest` drives the typed `RouterClient` over HTTP, not the log); the deepseekocr helper is `static` and compiles against the unchanged 11-parameter `ggml_im2col` declaration, and neither `mtmd.h` nor `mtmd-helper.h` changed. **Patches:** only `server-models.cpp` of the five patched files changed, ~810 lines below patch `0008`'s single `@@ -215 @@` hunk — no context shift. **Build wiring:** no `CMakeLists.txt` changed except upstream's own `tests/`, and no file was added/removed/renamed under `tools/server`, `tools/mtmd`, `vendor` or `include`. The three headers that changed anywhere (`ggml-rpc.h`, `ggml-sycl/fwht.hpp`, `src/llama-model-loader.h`) are all outside the project's include dependency graph. | | b10488–b10499 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10499 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | -| b10499–b10507 | `common/arg.{h,cpp}` + `common/download.{h,cpp}` + `tools/server/server-models.{h,cpp}` (**#27346 router preset option `dedup-cache-models`** — new `COMMON_ARG_PRESET_DEDUP_CACHE_MODELS` macro, new `common_download_resolve_path()`, new `server_model_meta::hidden` + a skip in the `GET /models` handler), `tools/mtmd/{mtmd.h,mtmd.cpp,mtmd-helper.cpp,CMakeLists.txt}` + **NEW** `tools/mtmd/mtmd-internal.h` (**#27348 `mtmd_bitmap_set_mergeable()`** — Qwen-VL temporal frame merging becomes opt-in: `can_merge_with()` now also requires `mergeable && other.mergeable`, and only `mtmd_helper_video`'s frame reader sets it), `ggml/src/ggml-{webgpu,vulkan,metal}/**`, `gguf-py/**`, `tests/**` | **No project-source change.** Both `common/download.h` and `tools/mtmd/mtmd.h` changes are *purely additive* — no existing declaration moved — and grepping `src/main/cpp` + `src/test/cpp` for `common_download*`, `COMMON_ARG_PRESET*`, `common_preset`, `mtmd_bitmap_*`, `mtmd_input_part`, `mtmd_group_mergeable_bitmaps`, `server_model_meta`, `get_all_meta` returns **zero** hits (the only `server-models` mention is a comment in `native_server.cpp:188`). `server-models.cpp` is compiled into both `jllama` and `jllama_test`, and its new `common_download_resolve_path()` call resolves from the `llama-common` both already link — no CMake change. The new `mtmd-internal.h` is added only to `add_library(mtmd …)`'s **header** list (no new `.cpp`), so it cannot repeat the b10154 `server-mcp.cpp` missing-`target_sources` link failure. Two behaviour notes, neither actionable: the `/models` JSON *shape* is unchanged (hidden entries are omitted, never serialized) and nothing is hidden unless a preset opts in with `dedup-cache-models`, which this project never writes; and because upstream's server builds bitmaps with `mtmd_helper_bitmap_init_from_buf` without setting `mergeable`, two consecutive same-size images in one request are no longer temporal-merged on Qwen-VL models — upstream's deliberate fix, invisible to SmolVLM (`n_merge_frames == 1`) and to the TTS speaker-reference clip (an audio bitmap, excluded by the pre-existing `!is_audio` guard). **Patch context:** patch `0008`'s `server-models.cpp` hunk keeps the identical offset across the range; patch `0001`'s `common/arg.h` hunk shifts by exactly **+1 line** (the reformatted preset-macro block) with unchanged context and still applies; `server.cpp` and `server-context.{cpp,h}` are untouched. | +| b10499–b10507 | `common/arg.{h,cpp}` + `common/download.{h,cpp}` + `tools/server/server-models.{h,cpp}` (**#27346 router preset option `dedup-cache-models`** — new `COMMON_ARG_PRESET_DEDUP_CACHE_MODELS` macro, new `common_download_resolve_path()`, new `server_model_meta::hidden` + a skip in the `GET /models` handler), `tools/mtmd/{mtmd.h,mtmd.cpp,mtmd-helper.cpp,CMakeLists.txt}` + **NEW** `tools/mtmd/mtmd-internal.h` (**#27348 `mtmd_bitmap_set_mergeable()`** — Qwen-VL temporal frame merging becomes opt-in: `can_merge_with()` now also requires `mergeable && other.mergeable`, and only `mtmd_helper_video`'s frame reader sets it), `ggml/src/ggml-{webgpu,vulkan,metal}/**`, `gguf-py/**`, `tests/**` | **No project-source change.** Both `common/download.h` and `tools/mtmd/mtmd.h` changes are *purely additive* — no existing declaration moved — and grepping `src/main/cpp` + `src/test/cpp` for `common_download*`, `COMMON_ARG_PRESET*`, `common_preset`, `mtmd_bitmap_*`, `mtmd_input_part`, `mtmd_group_mergeable_bitmaps`, `server_model_meta`, `get_all_meta` returns **zero** hits (the only `server-models` mention is a comment in `native_server.cpp:188`). `server-models.cpp` is compiled into both `jllama` and `jllama_test`, and its new `common_download_resolve_path()` call resolves from the `llama-common` both already link — no CMake change. The new `mtmd-internal.h` is added only to `add_library(mtmd …)`'s **header** list (no new `.cpp`), so it cannot repeat the b10154 `server-mcp.cpp` missing-`target_sources` link failure. Two behaviour notes, neither actionable: the `/models` JSON *shape* is unchanged (hidden entries are omitted, never serialized) and nothing is hidden unless a preset opts in with `dedup-cache-models`, which the project's own code never writes — though `NativeServer` forwards raw llama-server argv verbatim by design, so a *caller* can enable it, and `RouterClient.awaitModelLoaded` then reports a hidden-but-still-loadable model as unknown (recorded in `TODO.md`); and because upstream's server builds bitmaps with `mtmd_helper_bitmap_init_from_buf` without setting `mergeable`, two consecutive same-size images in one request are no longer temporal-merged on Qwen-VL models — upstream's deliberate fix, invisible to SmolVLM (`n_merge_frames == 1`) and to the TTS speaker-reference clip (an audio bitmap, excluded by the pre-existing `!is_audio` guard). **Patch context:** patch `0008`'s `server-models.cpp` hunk keeps the identical offset across the range; patch `0001`'s `common/arg.h` hunk shifts by exactly **+1 line** (the reformatted preset-macro block) with unchanged context and still applies; `server.cpp` and `server-context.{cpp,h}` are untouched. | | b10499–b10507 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10507 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10507–b10509 | `ggml/include/ggml.h` + `ggml/src/**` + `ggml/src/ggml-metal/**` (**#27120: new `ggml_rope_set_offset()` and its Metal support**), `tools/ui/**` (**#27365: persisted settings read before the API-key probe**) | **No project-source change.** Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or the root `CMakeLists.txt` changed at all in this range, so none of the five patched files moved and no header in the project's include dependency graph was touched. `ggml_rope_set_offset` is purely additive to `ggml.h` (a header CLAUDE.md's review list marks "safe to skip" — the project never calls a `ggml_rope_*` primitive; `grep -rn "ggml_rope" src/main/cpp src/test/cpp` is empty), and `tools/ui` is the Svelte WebUI, which CI rebuilds from the pinned `GIT_TAG` and which therefore needs no per-bump source review. | | b10507–b10509 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10509 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. This is the **last** tag at which patch `0007` applies unmodified — see the next row. | @@ -653,7 +653,7 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10569–b10578 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10578 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10578–b10584 | `common/fit.{h,cpp}` + `common/common.cpp` + `common/speculative.cpp` + `tools/{fit-params,llama-bench}` (**draft/MTP context is now fitted together with the target model — new public `struct common_fit_extra_model` and a new 7th `const common_fit_extra_model * extra` parameter on `common_fit_params()`; `common_speculative_init_result` pins the draft `cparams.n_ctx = llama_n_ctx(ctx_tgt)`**), `tools/server/server-context.cpp` (**−56 lines: the hand-rolled draft-model VRAM pre-reservation block is deleted, superseded by the fitter**), `common/arg.cpp` (**`--conversation` dropped `LLAMA_EXAMPLE_CLI` from its `set_examples()`**), `tools/mtmd/{CMakeLists.txt,clip*.h,clip.cpp,models/dots3note.cpp,models/models.h,mtmd-audio.{h,cpp},mtmd.cpp}` (**new `dots3note` multimodal model: two `PROJECTOR_TYPE_DOTS3NOTE_{V,A}` values, a vision MoE FFN via a new `clip_graph::build_moe_ffn()`, and `mtmd_audio_preprocessor_dots3note`**) | **No project-source change; no patch refresh.** `common_fit_params()`'s signature break is upstream-only: `common/fit.h` is not in the project's include graph, `grep -rn "common_fit_params\|common_fit_extra_model\|fit_params\|common_get_device_memory_data" src/main/cpp src/test/cpp` returns zero hits, and the two other callers (`tools/fit-params`, `tools/llama-bench`) are not built here (`LLAMA_BUILD_TOOLS` is OFF under FetchContent; only `tools/mtmd` and an explicit `tools/server/*.cpp` list are added back). The `server-context.cpp` deletion sits directly **above** patch `0002`'s hunk, whose anchor lines (blank / `// attach a progress callback` / `{`) are byte-identical, so that hunk only *shifts* (offset −53 → −108) and still applies cleanly. `--conversation` never reached the project's parser (`jllama.cpp` parses with `LLAMA_EXAMPLE_SERVER`, in neither the old nor the new example set) and patch `0001`'s `arg.cpp` hunks are ~650 lines away. All mtmd edits are internal — **`mtmd.h` and `mtmd-helper.h` are unchanged**, so `tts_engine.cpp`'s `mtmd_helper::gen_audio` surface is untouched, and `models/dots3note.cpp` enters through upstream's own `tools/mtmd/CMakeLists.txt`. The only reachable behaviour delta is a more accurate VRAM budget when `--fit` runs together with a draft model. | | b10578–b10584 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10584 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | -| b10584–b10585 | **NEW** `common/json.{h,cpp}` + `common/CMakeLists.txt`, `common/chat.h`, `common/chat-auto-parser.h`, `common/chat-peg-parser.h`, `common/json-schema-to-grammar.h`, `common/peg-parser.h`, `common/arg.cpp`, `common/{chat,download,hf-cache,jinja/*,json-schema-to-grammar,peg-parser}.cpp`, `tools/server/{server-common.h,server-common.cpp,server-chat.h,server-chat.cpp,server-context.h,server-context.cpp,server-task.h,server-task.cpp,server-schema.cpp,server-models.cpp,server-tools.cpp}` (**#27511 "common: add json.h abstraction"**) | **BREAKING — the largest project impact of the whole b10456 → b10618 walk.** Upstream introduces the pimpl wrapper class `common_json` and flips `using json = nlohmann::ordered_json` → `using json = common_json` in `server-common.h`, so **every** project TU that includes an upstream server header changes JSON type. `common_json` is a deliberately closed API: no `get_ref`, no `array_t`, no `type_name()`, a braced list in *value* position does not build an array, `at(key)` no longer implicitly converts (needs `.get()`), `get()` is limited to the specialisations in `common/json.cpp`, and errors are `common_json_error`. Project changes, in the order they were found: **(1)** `utils.hpp`'s `str_to_bytes` used `bytes.get_ref().reserve(...)` — dropped (a pure optimisation; `json::array()` + `push_back` is unchanged). **(2)** `jllama.cpp`'s `jsonSchemaToGrammarBytes` parsed with `nlohmann::ordered_json::parse` and handed the result to `json_schema_to_grammar`, which now takes `const common_json &` — there is no conversion, so this is a hard compile error; switched to `json::parse`. **(3) Two silent, compile-clean regressions**, neither of which a build could catch: `jni_helpers.hpp`'s `require_json_field_impl` declared its parameter `const nlohmann::json &`, and a `common_json` **still binds to it** — through `common_json::operator std::string()` feeding nlohmann's string-constructible converting constructor — so the presence check became a `json::type_error 302` ("type must be string, but is object") thrown out of `handleInfill`, whose two call sites sit *outside* its `try` block; the helper is now a template on the JSON type, with two new `common_json` regression tests. And `common_json_value`'s integral constructor template is `std::is_integral`-gated, which **excludes enums**, so `{"vocab_type", }` bound to the `bool` constructor and serialised as `true`/`false`; `ModelMeta.getVocabType()` reads it with Jackson's `asInt(0)`, so every non-SPM model would have reported vocab type 1. Both emit sites now `static_cast(...)`, with a `CommonJsonEnumTrap` C++ pair pinning the trap and a `LlamaModelTest` assertion that the wire value `isIntegralNumber()`. (Both were confirmed empirically with a standalone probe linked against `common/json.cpp`, not inferred.) **(4)** Seven C++ test literals with a braced array in value position (`test_json_helpers.cpp`'s `extract_embedding_prompt` input, `test_server.cpp`'s `dry_sequence_breakers` + five `samplers` cases) now use `json::array({...})`; the produced JSON is identical. **No JSON output shape changed anywhere**, so no existing assertion moved; **no CMake change** is needed (`common/json.cpp` rides the `llama-common` target both `jllama` and `jllama_test` already link, and the `nlohmann_json` link stays for `log_helpers.hpp` / `train_engine.cpp`, which keep their own alias); and **no patch context shifted** — `server.cpp` is untouched in this range, and `0001`/`0002`/`0003`/`0008` keep byte-identical anchors. Test total 487 → 491. | +| b10584–b10585 | **NEW** `common/json.{h,cpp}` + `common/CMakeLists.txt`, `common/chat.h`, `common/chat-auto-parser.h`, `common/chat-peg-parser.h`, `common/json-schema-to-grammar.h`, `common/peg-parser.h`, `common/arg.cpp`, `common/{chat,download,hf-cache,jinja/*,json-schema-to-grammar,peg-parser}.cpp`, `tools/server/{server-common.h,server-common.cpp,server-chat.h,server-chat.cpp,server-context.h,server-context.cpp,server-task.h,server-task.cpp,server-schema.cpp,server-models.cpp,server-tools.cpp}` (**#27511 "common: add json.h abstraction"**) | **BREAKING — the largest project impact of the whole b10456 → b10618 walk.** Upstream introduces the pimpl wrapper class `common_json` and flips `using json = nlohmann::ordered_json` → `using json = common_json` in `server-common.h`, so **every** project TU that includes an upstream server header changes JSON type. `common_json` is a deliberately closed API: no `get_ref`, no `array_t`, no `type_name()`, a braced list in *value* position does not build an array, `at(key)` no longer implicitly converts (needs `.get()`), `get()` is limited to the specialisations in `common/json.cpp`, and errors are `common_json_error`. Project changes, in the order they were found: **(1)** `utils.hpp`'s `str_to_bytes` used `bytes.get_ref().reserve(...)` — dropped (a pure optimisation; `json::array()` + `push_back` is unchanged). **(2)** `jllama.cpp`'s `jsonSchemaToGrammarBytes` parsed with `nlohmann::ordered_json::parse` and handed the result to `json_schema_to_grammar`, which now takes `const common_json &` — there is no conversion, so this is a hard compile error; switched to `json::parse`. **(3) Two silent, compile-clean regressions**, neither of which a build could catch: `jni_helpers.hpp`'s `require_json_field_impl` declared its parameter `const nlohmann::json &`, and a `common_json` **still binds to it** — through `common_json::operator std::string()` feeding nlohmann's string-constructible converting constructor — so the presence check became a `json::type_error 302` ("type must be string, but is object") thrown out of `handleInfill`, whose two call sites sit *outside* its `try` block; the helper is now a template on the JSON type, with two new `common_json` regression tests. And `common_json_value`'s integral constructor template is `std::is_integral`-gated, which **excludes enums**, so `{"vocab_type", }` bound to the `bool` constructor and serialised as `true`/`false`; `ModelMeta.getVocabType()` reads it with Jackson's `asInt(0)`, so every non-SPM model would have reported vocab type 1. Both emit sites now `static_cast(...)`, with a `CommonJsonEnumTrap` C++ pair pinning the trap and a `LlamaModelTest` assertion that the wire value `isIntegralNumber()`. (Both were confirmed empirically with a standalone probe linked against `common/json.cpp`, not inferred.) **(4)** Seven C++ test literals with a braced array in value position (`test_json_helpers.cpp`'s `extract_embedding_prompt` input, `test_server.cpp`'s `dry_sequence_breakers` + five `samplers` cases) now use `json::array({...})`; the produced JSON is identical. **No JSON output shape changed in the project's own emissions**, so no existing assertion moved — but the same enum trap bit **upstream's own** `/models` handler, which the project compiles and serves; that was found by the post-bump review and is carried as `patches/0009` (see the last row of this file); **no CMake change** is needed (`common/json.cpp` rides the `llama-common` target both `jllama` and `jllama_test` already link, and the `nlohmann_json` link stays for `log_helpers.hpp` / `train_engine.cpp`, which keep their own alias); and **no patch context shifted** — `server.cpp` is untouched in this range, and `0001`/`0002`/`0003`/`0008` keep byte-identical anchors. Test total 487 → 491. | | b10584–b10585 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10585 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics **after** the migration above. Full link + `ctest` deferred to the b10618 target. | | b10585–b10590 | `common/json.{cpp,h}` (**#27575 "fix clang lto": the `common_json_value` set/map/unordered_map/vector constructors and every `common_json::get()` flip from explicit *instantiation* to explicit *specialization* — "an explicit instantiation is a weak symbol, dropped by some LTO builds (clang-cl)" — and `operator std::string()` + `value(key, const char *)` move from inline-in-`json.h` to out-of-line in `json.cpp`; the set of usable types is unchanged**), `vendor/sheredom/subprocess.h` + `scripts/sync_vendor.py` (**#27409 upstream resync, +363/−12: new `SUBPROCESS_SPAWN_VIA_FORK` / `SUBPROCESS_ADDCHDIR_IS_POSIX` macros gating a `fork()`+`execve`/`execvpe` launcher with an errno-relay pipe, new `subprocess_pipe_cloexec()` / `subprocess_fds_above_std()` POSIX helpers, and a Windows `STARTUPINFOEX` + `PROC_THREAD_ATTRIBUTE_HANDLE_LIST` handle-inheritance list**), `tools/mtmd/{clip.cpp,clip-graph.h,models/gemma4v.cpp,models/minimax-m3.cpp}` (**#27521: 2D RoPE rewritten from view/rope/`ggml_concat` onto two in-place `ggml_rope_ext` calls + `ggml_rope_set_offset`**) | **No project-source and no project-CMake change.** The `json` work *is* in the project's compile path (the alias became `common_json` at b10585) and the project calls both members that moved out-of-line, but `common/json.cpp` belongs to the `llama-common` target that `jllama` and `jllama_test` already link — nothing to wire, and no `to_json()` shape moved. The project enables no LTO, so the bug being fixed never bit it; the change is robustness for the clang-cl Windows-arm64 job. **`subprocess.h` re-verified against the dropped patch `0009`:** `SUBPROCESS_HAVE_CWD` and its `#elif !SUBPROCESS_HAVE_CWD → posix_error = ENOSYS;` fallback both survive, and `SUBPROCESS_SPAWN_VIA_FORK` is 0 on glibc/bionic/macOS/Windows (only `_AIX`/`__OpenBSD__`/old NetBSD set it), so manylinux2014 (glibc 2.17) still resolves exactly as before and the new `execvpe` declaration is preprocessed away everywhere the project builds — **`0009` stays dropped.** The new always-compiled POSIX helpers need only `pipe2`/`O_CLOEXEC`/`F_DUPFD` (glibc 2.7+/bionic) and the header already includes ``/``; the Windows attribute-list path adds no new link input. mtmd is internal-only here: `build_rope_2d`'s signature is untouched (the `clip-graph.h` hunk is comment-only) and `mtmd.h`/`mtmd-helper.h`/`clip.h` are byte-identical, so `tts_engine.cpp`'s surface is unaffected. **Patches:** zero changes under `tools/server/` and none to `common/arg.{h,cpp}` — byte-identical context for all six. | | b10585–b10590 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10590 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | @@ -669,3 +669,4 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10614–b10615 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10615 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10615–b10618 | `src/llama-grammar.cpp` (**#27591: `parse_char()` accepts `case '-':`, so a `\-` escape produced by `gbnf_escape_char_class()` inside a character class no longer throws**), `ggml/src/ggml-sycl/**` (**#27660: `tq2_0` marked unsupported**), `ggml/src/ggml-webgpu/wgsl-shaders/argsort.wgsl` (**#27538: infinity handling in ARGSORT / TOP_K**) | **No project-source change (final step).** Zero lines changed under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt`, so no patch context moved. The grammar fix is strictly *widening* — an input that previously threw now parses — and reaches the project through both `LlamaModel.jsonSchemaToGrammar` and every grammar-constrained completion, so it is a **positive** for the agentic tool-calling target; no test pinned the old throw (`grep -rn 'gbnf_escape_char_class\|parse_char' src/main/cpp src/test/cpp` is empty). The SYCL and WebGPU edits only affect the `sycl-*` classifiers' compiled sources and a backend the project does not build. | | b10456–b10618 | upstream verification (sandbox, final target) | **Full local verification.** Fresh `cmake -B build` with the real `FetchContent` path (no source-dir override), so the fail-loud `PATCH_COMMAND` ran for real: it resolved `ggml commit eb25b7263` / `LLAMA_BUILD_NUMBER = 10618` and applied all **6** patches — verified by grepping the fetched tree for each patch's marker (`common_params_parse_main`, `params_base.load_progress_callback == nullptr`, `get_slot_prompt_similarity`, `g_llama_server_embedded`, `llama_server_attach` + `llama_server_register_common_routes`, `LLAMA_SERVER_WORKER_CMD`). Then a full `cmake --build --config Release` (jllama + jllama_test both link — the `jllama_test` link is what proves `common_json::get()` resolves, since upstream documents un-specialised types as a *link*-time failure) and `ctest`: **491/491 C++ tests pass**. On the Java side: `mvn test-compile` clean, `mvn test` **1405 run / 0 failures / 17 skipped** (only the model-gated integration tests, no GGUF in the sandbox), and `NativeLibraryLoadSmokeTest` green — including `nativeBuildInfoMatchesPinnedVersionConstant`, the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10618") matches the `build-info` compiled into the freshly linked `libjllama.so`. `mvn spotless:apply` produced no changes beyond the edits themselves, the pinned **clang-format 22.1.8** reports the whole C++ tree clean, and `mvn clean javadoc:jar` is `BUILD SUCCESS`. **Per-step verification for all 25 chunks:** every intermediate tag additionally had all six patches applied by the applier's own commands and all ten project + C++-test translation units compiled against it, so no commit in this walk is a state that fails to build. | +| b10456–b10618 | `tools/server/server-context.cpp` — **new local patch `0009-server-cast-vocab-type-for-common-json.patch`** | **Upstream regression found by the post-bump review, fixed downstream.** `get_res_model_info()` builds the `GET /models` + `GET /v1/models` payload and emits `{"vocab_type", meta.model_vocab_type}`, an **unscoped enum**. That was correct while the `json` alias was `nlohmann::ordered_json` (nlohmann serialises an enum as an integer), but `common_json_value`'s integral constructor template is `std::is_integral`-gated, which excludes enums — so from **b10585** the value binds to `common_json_value(bool)` and goes on the wire as `true`/`false`. Upstream regressed it silently in #27511 when they flipped the alias; the same trap hit the project's own two `"vocab_type"` emit sites in `jllama.cpp` (fixed in the b10584–b10585 step). It **ships**: `server-context.cpp` is compiled into `libjllama` and both routes are served by `NativeServer`, the default fat-jar `Main-Class`, in full and in attach mode. The new patch casts the value to `int` at the emit site — one line, upstream-submittable, applied after `0002`/`0003` (same file). A mechanical sweep confirmed it is the **only** such site in the upstream server sources the project compiles: `server_context_meta` has exactly two enum-typed members, and the other (`pooling_type`) is only ever compared, never serialised. | diff --git a/llama/patches/0009-server-cast-vocab-type-for-common-json.patch b/llama/patches/0009-server-cast-vocab-type-for-common-json.patch new file mode 100644 index 000000000..24a8b4f90 --- /dev/null +++ b/llama/patches/0009-server-cast-vocab-type-for-common-json.patch @@ -0,0 +1,16 @@ +diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp +index 15fa8a498..f2fbe2be8 100644 +--- a/tools/server/server-context.cpp ++++ b/tools/server/server-context.cpp +@@ -4439,7 +4439,10 @@ static json get_res_model_info(const server_context_meta & meta) { + {"created", std::time(0)}, + {"owned_by", "llamacpp"}, + {"meta", { +- {"vocab_type", meta.model_vocab_type}, ++ // an unscoped enum has no common_json_value ctor of its own (the integral one is ++ // is_integral-gated, which excludes enums), so it binds to common_json_value(bool) ++ // and serialises as true/false -- cast it to keep the numeric vocab type on the wire ++ {"vocab_type", (int) meta.model_vocab_type}, + {"n_vocab", meta.model_vocab_n_tokens}, + {"n_ctx", meta.slot_n_ctx}, + {"n_ctx_train", meta.model_n_ctx_train}, diff --git a/llama/src/main/java/net/ladenthin/llama/server/RouterClient.java b/llama/src/main/java/net/ladenthin/llama/server/RouterClient.java index f20129d76..5bb2be90c 100644 --- a/llama/src/main/java/net/ladenthin/llama/server/RouterClient.java +++ b/llama/src/main/java/net/ladenthin/llama/server/RouterClient.java @@ -48,6 +48,14 @@ *

Instances are immutable and safe to share across threads; every call opens a short-lived * {@link HttpURLConnection}.

* + *

No API-key support. This client sends no {@code Authorization} header, so it + * only works against a router started without {@code --api-key}. {@code /models/load} and + * {@code /models/unload} have always required the key when one is configured; since llama.cpp + * b10519 (upstream #26347 dropped {@code /models} and {@code /v1/models} from the + * server's public-endpoint set) {@link #listModels()}, {@link #findModel(String)} and + * {@link #awaitModelLoaded(String, long)} answer {@code 401} on such a router too. Adding an + * optional key is tracked in {@code TODO.md}.

+ * *

{@code equals}/{@code hashCode} are generated by Lombok over the {@code host}/{@code port} * fields (two clients pointing at the same router compare equal). {@code toString} is * intentionally handwritten (not Lombok-generated) so the client renders as its target URL, From 029c79f84f13c88635c5ae95dc4bd2e511374967 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 11:05:09 +0000 Subject: [PATCH 28/76] chore: renumber patch 0009 -> 0010 and correct the bump records The review caught three documentation defects in the previous commit. 0009 is a burned number: it names the subprocess.h patch dropped at the b10280 bump, and CLAUDE.md carries a prose note three lines below the patches table saying so. A live 0009 row above it would read as if that note described the new patch, whose documented precedent would then be "upstream merged it, delete it". Renamed to 0010 -- filename-only, same sort position, re-checked by applying the whole set to a clean b10618 checkout -- and the row now explains why, and states the bump-time rule every neighbouring row carries: if upstream casts the value themselves, drop this patch rather than refresh it. The final-target verification row certified a 6-patch build; the shipped tree has 7. Restated for the re-run: 7 patch markers including the new cast, 491/491 ctest, 1405 Java tests. The bump runbook enumerated six patches by name one commit after being rewritten -- replaced with glob-truthful wording, since apply-llama-patches.cmake globs the directory and any list here can only rot. Also records that --mmproj-device was left unwired on purpose, so the next reviewer does not re-derive it: it is new public Java API, reachable meanwhile through NativeServer raw argv passthrough. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 8 ++++---- docs/upgrade/llama-cpp-version-bump.md | 9 ++++++--- ...=> 0010-server-cast-vocab-type-for-common-json.patch} | 0 4 files changed, 11 insertions(+), 8 deletions(-) rename llama/patches/{0009-server-cast-vocab-type-for-common-json.patch => 0010-server-cast-vocab-type-for-common-json.patch} (100%) diff --git a/CLAUDE.md b/CLAUDE.md index dabb60fa9..9304f254f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -661,7 +661,7 @@ Current patches: | `0003-pr22393-server-add-slot-prompt-similarity-getter-setter.patch` | **Upstream-PR carry** of [ggml-org/llama.cpp#22393](https://github.com/ggml-org/llama.cpp/pull/22393) ("server : add slot_prompt_similarity getter/setter"). Purely additive: adds `server_context::get_slot_prompt_similarity()` / `set_slot_prompt_similarity(float)` (`tools/server/server-context.{cpp,h}`) so an embedding/JNI caller can query and tune the slot-selection threshold at runtime without reloading the model. Verbatim copy of the PR, which **upstream closed without merging** (rejected as exposing unsafe internal state — see the patch header). Carried permanently; it will not be droppable via a version bump. | | `0007-server-attach-http-frontend.patch` | **Adds `llama_server_attach(argc, argv, server_context&)`** so the `NativeServer` *attach mode* can serve an **already-loaded `LlamaModel`** over the upstream HTTP frontend — no second model load, no `start_loop()`; the LlamaModel's worker keeps driving the shared `server_context` and the HTTP routes post tasks to its queue (the queue is the synchronization point). Mechanically: (1) extracts the **pure core route table** (`health` … `slots`) out of `llama_server()` into `static void llama_server_register_common_routes(ctx_http, routes)` (shared, so the two entry points cannot drift on the core endpoint set). **Scope note (narrowed at the b10154 bump):** the helper deliberately carries **only** the stable, state-independent route table — **not** the resumable-streaming routes (their handlers differ between router / non-router), the GCP-compat shim, or the experimental **CORS-proxy / MCP-server / built-in-tools** wiring. b10154 (upstream MCP-server support) moved the streaming routes into the middle of that block and coupled tools/CORS to a per-call `server_mcp mcp_mgr` lifecycle, so the earlier contiguous "route-table + CORS-proxy + tools" extraction is no longer possible; `llama_server()` keeps all of that inline, **byte-identical to upstream b10154** (only the route-table block is factored out). (2) adds `llama_server_attach`, which parses only the HTTP-side argv via `common_params_parse`, starts the stream-session GC + `server_http_context`, registers the common route table, the **non-router** resumable-streaming handlers (upstream b10154 paths `/v1/stream` GET/DEL + `/v1/streams/lookup` POST), the GCP-compat shim, and **403 "disabled" stubs for `/cors-proxy` + `/tools`** (attach mode does not wire the experimental CORS-proxy / MCP / built-in-tools host — those belong to a full `llama-server`, not an embedded model), marks ready immediately (model already loaded), and blocks on the HTTP thread until `llama_server_request_shutdown()` — never calling `common_init()`, backend init, `ctx_server.terminate()` or `llama_backend_free()` (the embedding caller owns those). Applies after `0001`+`0006` (same file); closes the "NativeServer — reuse an already-loaded LlamaModel" TODO. Upstream-submittable ("server: let embedding callers attach the HTTP frontend to an existing server_context"). **Refreshed at the b10519 bump:** upstream #26347 dropped the API key from the `/models` + `/v1/models` public-endpoint set and deleted the two trailing `// public endpoint (no API key check)` comments on those route registrations. Those two lines sit inside this patch's route-table removal block, so `git apply` failed ("patch does not apply", `server.cpp:258`) at **every** tag from b10519 on; the fix was to drop the now-wrong comment from all four affected lines (2 on the `-` side, 2 in the extracted helper on the `+` side), keeping the helper byte-identical to the block it replaces. **This is the invariant to re-check on every bump:** the `+` side of `llama_server_register_common_routes()` must stay a verbatim copy of the route table it factors out of `llama_server()`. | | `0008-server-models-worker-cmd-override.patch` | **Makes router mode usable in-JVM.** The router (`server-models.cpp`) spawns each model worker by re-executing its own binary (`get_server_exec_path()` = `/proc/self/exe` & friends) — inside a JVM that binary is `java`, not a llama-server, so embedded router workers could never start. The patch adds env `LLAMA_SERVER_WORKER_CMD` (whitespace-split; read in `server_model_meta::update_args`) which replaces only the leading binary-path token of the rendered worker args, letting an embedding host relaunch workers through its own bootstrap — e.g. `java -cp app.jar net.ladenthin.llama.server.NativeServer` (each worker is then a fresh JVM running the classic single-model `NativeServer`). Exposed in Java as `NativeServer.setWorkerCommand(String...)` (JNI `setenv`); exercised by `RouterModeIntegrationTest` (Linux CI). Upstream-submittable (also useful for containerized/wrapped deployments). | -| `0009-server-cast-vocab-type-for-common-json.patch` | **Upstream regression from the b10585 `common_json` switch (#27511), one line.** `get_res_model_info()` (`tools/server/server-context.cpp`) builds the `GET /models` + `GET /v1/models` payload and emits `{"vocab_type", meta.model_vocab_type}` — an **unscoped enum**. `common_json_value`'s integral constructor template is `std::is_integral`-gated, which *excludes* enums, so the value binds to `common_json_value(bool)` and serialises as `true`/`false` instead of the numeric vocab type. It was correct while the alias was `nlohmann::ordered_json` (nlohmann serialises an enum as an integer), so upstream regressed it silently when they flipped the alias. The project ships this: `server-context.cpp` is compiled into `libjllama` and both routes are served by `NativeServer` — the default fat-jar `Main-Class` — in full **and** attach mode (`patches/0007`'s common route table registers them). The patch casts the value to `int` at the emit site, mirroring what `jllama.cpp` does for its own two `"vocab_type"` sites. Upstream-submittable. Applies after `0002`/`0003` (same file). See the `CommonJsonEnumTrap` tests in `test_json_helpers.cpp` for the mechanism. | +| `0010-server-cast-vocab-type-for-common-json.patch` | **Upstream regression from the b10585 `common_json` switch (#27511), one line.** `get_res_model_info()` (`tools/server/server-context.cpp`) builds the `GET /models` + `GET /v1/models` payload and emits `{"vocab_type", meta.model_vocab_type}` — an **unscoped enum**. `common_json_value`'s integral constructor template is `std::is_integral`-gated, which *excludes* enums, so the value binds to `common_json_value(bool)` and serialises as `true`/`false` instead of the numeric vocab type. It was correct while the alias was `nlohmann::ordered_json` (nlohmann serialises an enum as an integer), so upstream regressed it silently when they flipped the alias. The project ships this: `server-context.cpp` is compiled into `libjllama` and both routes are served by `NativeServer` — the default fat-jar `Main-Class` — in full **and** attach mode (`patches/0007`'s common route table registers them). The patch casts the value to `int` at the emit site, mirroring what `jllama.cpp` does for its own two `"vocab_type"` sites. Upstream-submittable; **not yet filed upstream**. Applies after `0002`/`0003` (same file) — numbered `0010` because `0009` is burned: it names the subprocess.h patch dropped at the b10280 bump (see the note below this table), and reusing the number would make that note read as if it were about this patch. **On every bump, check whether upstream cast the value themselves; if they did, DROP this patch rather than refreshing it** — the fail-loud applier only detects "does not apply", never "upstream already fixed this", and no test can catch a redundant carry here because `get_res_model_info` is `static` inside `server-context.cpp` and unreachable from `jllama_test`. See the `CommonJsonEnumTrap` tests in `test_json_helpers.cpp` for the mechanism the cast defends against. | | `0006-server-embed-native-server-jni.patch` | **Makes `server.cpp`'s `llama_server` embeddable in the JVM** so the `NativeServer` JNI bridge can run the full upstream HTTP server (WebUI included) inside `libjllama` — see "Two server modes" below. b9870 already exposes `int llama_server(int, char**)` (non-static; no `main` in the file), so the patch only adds embedded-mode support: (1) a `g_llama_server_embedded` flag + `llama_server_set_embedded()` / `llama_server_request_shutdown()` (declared in the committed `src/main/cpp/native_server_bridge.h`); (2) skips installing the process-wide SIGINT/SIGTERM handlers when embedded (they would hijack the JVM's); (3) in embedded mode parses the **forwarded** argv via `common_params_parse` instead of `common_params_parse_main` (whose `GetCommandLineW` recovery would pick up `java.exe`'s command line — the same Windows class of bug `0001` fixes). `llama_server_request_shutdown()` mirrors the SIGTERM path (invokes the installed `shutdown_handler` → `ctx_server.terminate()` unblocks `start_loop()`), giving JNI an out-of-band stop since `ctx_server` is loop-local. Applies **after `0001`** (which flips this call site to `common_params_parse_main`), so its context is the post-`0001` tree; regenerate against `0001`+source on a bump. Only touches `tools/server/server.cpp`. | **`0009` was dropped at the b10280 bump.** Upstream merged diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 96c26c30e..77040815e 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -637,7 +637,7 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10509–b10519 | upstream verification (sandbox) | Patch `0007` **refreshed** (see above); all **6** patches then re-verified against a clean b10519 checkout — sequential `git apply --check` + `git apply` in filename order, zero fuzz — and the refreshed `0007` additionally re-checked to apply cleanly at **every** later endpoint of this walk up to b10618, so the refresh is a one-time fix, not a per-chunk one. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics against the patched b10519 tree. Full link + `ctest` deferred to the b10618 target. | | b10519–b10532 | `common/speculative.cpp` (**#27404: avoid binding a reference to a null pointer**), `ggml/**` (backend-split scheduler race #26040, CUDA cuBLAS workspace #26574, Metal FA dequant #27390, Vulkan/OpenCL/Hexagon kernels), `src/llama-graph.cpp` (**#27392: V built as a view of K in `k_iswa build_attn`**), `convert_hf_to_gguf.py` | **No project-source change.** The only in-scope file is `common/speculative.cpp`, and the fix is inside an upstream-compiled TU — `common/speculative.h` (priority 3 on the CLAUDE.md review list) has a zero-line diff across the range, so nothing the project compiles against moved, and `grep -rn "common_speculative" src/main/cpp src/test/cpp` is empty. Nothing under `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt` changed, so no patch context shifted and `llama/CMakeLists.txt`'s `target_sources` list still matches upstream. The remainder is ggml backends and the llama graph builder, both entirely upstream-owned. | | b10519–b10532 | upstream verification (sandbox) | All **6** patches (with the b10519-refreshed `0007`) re-verified against a clean b10532 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | -| b10532–b10541 | `common/common.h` + `common/arg.cpp` + `tools/mtmd/{mtmd.h,mtmd.cpp,clip.h,clip.cpp}` + `tools/server/server-context.cpp` (**#23255 `--mmproj-device`: new `common_params::mmproj_device` (`ggml_backend_dev_t`) and a new `device` member at position 2 in `mtmd_context_params`/`clip_context_params`; the `MTMD_BACKEND_DEVICE` env lookup moves out of `clip.cpp` into `arg.cpp`**), `common/json-schema-to-grammar.cpp` (**#26939: unsupported regex patterns degrade to accept-any-string with a warning instead of throwing**), `tools/server/{server-models.h,server-models.cpp,server.cpp}` (**#27424: load-on-startup deferred into a new public `server_models::load_startup_models()` called from `llama_server()`'s router branch; reload no longer autoloads. #27416: `is_router_server` also excludes `--docker-repo`**), `common/speculative.cpp` (**`!dp.drafting` guard**) | **No project-source change.** The mid-struct `mtmd_context_params::device` insertion is the one positional-aggregate-init hazard in this range and the project does not trip it: `src/main/cpp/tts_engine.cpp` builds its params via `mtmd_context_params_default()` plus a named `.use_gpu` assignment, so `device` defaults to `nullptr` = the previous auto-GPU behaviour; `ggml_backend_dev_t` is already visible through `mtmd.h` → `llama.h` → `ggml-backend.h`, and every `common_params` in the project is default-constructed. `common/json-schema-to-grammar.h` is unchanged, so #26939 is semantic only — it does reach a public Java API (`jllama.cpp`'s `jsonSchemaToGrammarBytes` → `LlamaModel.jsonSchemaToGrammar`), where a schema with an unanchored `pattern` or a `\d`/`\w`/`\s` escape now yields a permissive `string` grammar plus a stderr warning instead of throwing `LlamaException`; no test moves, because `LlamaModelTest#testJsonSchemaToGrammar` asserts only `PRIMITIVE_RULES` output from three pattern-less string properties (zero `"pattern"` hits across `src/test/{java,cpp}`). `server_models` / `load_startup_models` / `docker_repo` are zero-hit greps in the project's C++ and Java. **Patch context:** `server.cpp` gains 13 lines (the `load_startup_models()` block) **below** patch `0007`'s route-table hunk, so only that patch's *last* hunk offset moves (10 → 23) — it still applies, verified by real `git apply` at both endpoints. **Build wiring:** no file added/removed/renamed under `tools/server`/`tools/mtmd`/`vendor`. | +| b10532–b10541 | `common/common.h` + `common/arg.cpp` + `tools/mtmd/{mtmd.h,mtmd.cpp,clip.h,clip.cpp}` + `tools/server/server-context.cpp` (**#23255 `--mmproj-device`: new `common_params::mmproj_device` (`ggml_backend_dev_t`) and a new `device` member at position 2 in `mtmd_context_params`/`clip_context_params`; the `MTMD_BACKEND_DEVICE` env lookup moves out of `clip.cpp` into `arg.cpp`**), `common/json-schema-to-grammar.cpp` (**#26939: unsupported regex patterns degrade to accept-any-string with a warning instead of throwing**), `tools/server/{server-models.h,server-models.cpp,server.cpp}` (**#27424: load-on-startup deferred into a new public `server_models::load_startup_models()` called from `llama_server()`'s router branch; reload no longer autoloads. #27416: `is_router_server` also excludes `--docker-repo`**), `common/speculative.cpp` (**`!dp.drafting` guard**) | **No project-source change.** The mid-struct `mtmd_context_params::device` insertion is the one positional-aggregate-init hazard in this range and the project does not trip it: `src/main/cpp/tts_engine.cpp` builds its params via `mtmd_context_params_default()` plus a named `.use_gpu` assignment, so `device` defaults to `nullptr` = the previous auto-GPU behaviour; `ggml_backend_dev_t` is already visible through `mtmd.h` → `llama.h` → `ggml-backend.h`, and every `common_params` in the project is default-constructed. **Deliberately left unwired:** the new `-mmdev/--mmproj-device` flag gets no `ModelParameters` setter in this bump — that is new public Java API and the owner's call, not a version bump's. It stays reachable through `NativeServer`, which forwards raw llama-server argv; the `LlamaModel` / `OpenAiCompatServer` path has no equivalent (there is no generic argv passthrough on `ModelParameters`), so a multi-GPU caller who needs it must use `NativeServer` until a setter is added. `common/json-schema-to-grammar.h` is unchanged, so #26939 is semantic only — it does reach a public Java API (`jllama.cpp`'s `jsonSchemaToGrammarBytes` → `LlamaModel.jsonSchemaToGrammar`), where a schema with an unanchored `pattern` or a `\d`/`\w`/`\s` escape now yields a permissive `string` grammar plus a stderr warning instead of throwing `LlamaException`; no test moves, because `LlamaModelTest#testJsonSchemaToGrammar` asserts only `PRIMITIVE_RULES` output from three pattern-less string properties (zero `"pattern"` hits across `src/test/{java,cpp}`). `server_models` / `load_startup_models` / `docker_repo` are zero-hit greps in the project's C++ and Java. **Patch context:** `server.cpp` gains 13 lines (the `load_startup_models()` block) **below** patch `0007`'s route-table hunk, so only that patch's *last* hunk offset moves (10 → 23) — it still applies, verified by real `git apply` at both endpoints. **Build wiring:** no file added/removed/renamed under `tools/server`/`tools/mtmd`/`vendor`. | | b10532–b10541 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10541 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10541–b10545 | `ggml/src/ggml-metal/**` (**#27450: clamp the K extent in the tensor-API mat-mat kernel when K is not a multiple of 32**), `ggml/src/ggml-opencl/**` (**#26476 q6_K flat mul_mat on Adreno A6x/A7x with older E031 compilers; #27339 norm local size**), `tools/ui/**` (**#27240: stores split refactor**) | **No project-source change.** Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt` changed — the whole range is ggml backend kernels plus the Svelte WebUI (which CI rebuilds from the pinned `GIT_TAG`, so it needs no per-bump source review). No patch context moved and `llama/CMakeLists.txt`'s `target_sources` list still matches upstream. The step's raw `git diff` is large (≈978 KiB) only because of the WebUI refactor; the reviewable code outside `tools/ui` is ≈17 KiB. | | b10541–b10545 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10545 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | @@ -653,7 +653,7 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10569–b10578 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10578 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10578–b10584 | `common/fit.{h,cpp}` + `common/common.cpp` + `common/speculative.cpp` + `tools/{fit-params,llama-bench}` (**draft/MTP context is now fitted together with the target model — new public `struct common_fit_extra_model` and a new 7th `const common_fit_extra_model * extra` parameter on `common_fit_params()`; `common_speculative_init_result` pins the draft `cparams.n_ctx = llama_n_ctx(ctx_tgt)`**), `tools/server/server-context.cpp` (**−56 lines: the hand-rolled draft-model VRAM pre-reservation block is deleted, superseded by the fitter**), `common/arg.cpp` (**`--conversation` dropped `LLAMA_EXAMPLE_CLI` from its `set_examples()`**), `tools/mtmd/{CMakeLists.txt,clip*.h,clip.cpp,models/dots3note.cpp,models/models.h,mtmd-audio.{h,cpp},mtmd.cpp}` (**new `dots3note` multimodal model: two `PROJECTOR_TYPE_DOTS3NOTE_{V,A}` values, a vision MoE FFN via a new `clip_graph::build_moe_ffn()`, and `mtmd_audio_preprocessor_dots3note`**) | **No project-source change; no patch refresh.** `common_fit_params()`'s signature break is upstream-only: `common/fit.h` is not in the project's include graph, `grep -rn "common_fit_params\|common_fit_extra_model\|fit_params\|common_get_device_memory_data" src/main/cpp src/test/cpp` returns zero hits, and the two other callers (`tools/fit-params`, `tools/llama-bench`) are not built here (`LLAMA_BUILD_TOOLS` is OFF under FetchContent; only `tools/mtmd` and an explicit `tools/server/*.cpp` list are added back). The `server-context.cpp` deletion sits directly **above** patch `0002`'s hunk, whose anchor lines (blank / `// attach a progress callback` / `{`) are byte-identical, so that hunk only *shifts* (offset −53 → −108) and still applies cleanly. `--conversation` never reached the project's parser (`jllama.cpp` parses with `LLAMA_EXAMPLE_SERVER`, in neither the old nor the new example set) and patch `0001`'s `arg.cpp` hunks are ~650 lines away. All mtmd edits are internal — **`mtmd.h` and `mtmd-helper.h` are unchanged**, so `tts_engine.cpp`'s `mtmd_helper::gen_audio` surface is untouched, and `models/dots3note.cpp` enters through upstream's own `tools/mtmd/CMakeLists.txt`. The only reachable behaviour delta is a more accurate VRAM budget when `--fit` runs together with a draft model. | | b10578–b10584 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10584 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | -| b10584–b10585 | **NEW** `common/json.{h,cpp}` + `common/CMakeLists.txt`, `common/chat.h`, `common/chat-auto-parser.h`, `common/chat-peg-parser.h`, `common/json-schema-to-grammar.h`, `common/peg-parser.h`, `common/arg.cpp`, `common/{chat,download,hf-cache,jinja/*,json-schema-to-grammar,peg-parser}.cpp`, `tools/server/{server-common.h,server-common.cpp,server-chat.h,server-chat.cpp,server-context.h,server-context.cpp,server-task.h,server-task.cpp,server-schema.cpp,server-models.cpp,server-tools.cpp}` (**#27511 "common: add json.h abstraction"**) | **BREAKING — the largest project impact of the whole b10456 → b10618 walk.** Upstream introduces the pimpl wrapper class `common_json` and flips `using json = nlohmann::ordered_json` → `using json = common_json` in `server-common.h`, so **every** project TU that includes an upstream server header changes JSON type. `common_json` is a deliberately closed API: no `get_ref`, no `array_t`, no `type_name()`, a braced list in *value* position does not build an array, `at(key)` no longer implicitly converts (needs `.get()`), `get()` is limited to the specialisations in `common/json.cpp`, and errors are `common_json_error`. Project changes, in the order they were found: **(1)** `utils.hpp`'s `str_to_bytes` used `bytes.get_ref().reserve(...)` — dropped (a pure optimisation; `json::array()` + `push_back` is unchanged). **(2)** `jllama.cpp`'s `jsonSchemaToGrammarBytes` parsed with `nlohmann::ordered_json::parse` and handed the result to `json_schema_to_grammar`, which now takes `const common_json &` — there is no conversion, so this is a hard compile error; switched to `json::parse`. **(3) Two silent, compile-clean regressions**, neither of which a build could catch: `jni_helpers.hpp`'s `require_json_field_impl` declared its parameter `const nlohmann::json &`, and a `common_json` **still binds to it** — through `common_json::operator std::string()` feeding nlohmann's string-constructible converting constructor — so the presence check became a `json::type_error 302` ("type must be string, but is object") thrown out of `handleInfill`, whose two call sites sit *outside* its `try` block; the helper is now a template on the JSON type, with two new `common_json` regression tests. And `common_json_value`'s integral constructor template is `std::is_integral`-gated, which **excludes enums**, so `{"vocab_type", }` bound to the `bool` constructor and serialised as `true`/`false`; `ModelMeta.getVocabType()` reads it with Jackson's `asInt(0)`, so every non-SPM model would have reported vocab type 1. Both emit sites now `static_cast(...)`, with a `CommonJsonEnumTrap` C++ pair pinning the trap and a `LlamaModelTest` assertion that the wire value `isIntegralNumber()`. (Both were confirmed empirically with a standalone probe linked against `common/json.cpp`, not inferred.) **(4)** Seven C++ test literals with a braced array in value position (`test_json_helpers.cpp`'s `extract_embedding_prompt` input, `test_server.cpp`'s `dry_sequence_breakers` + five `samplers` cases) now use `json::array({...})`; the produced JSON is identical. **No JSON output shape changed in the project's own emissions**, so no existing assertion moved — but the same enum trap bit **upstream's own** `/models` handler, which the project compiles and serves; that was found by the post-bump review and is carried as `patches/0009` (see the last row of this file); **no CMake change** is needed (`common/json.cpp` rides the `llama-common` target both `jllama` and `jllama_test` already link, and the `nlohmann_json` link stays for `log_helpers.hpp` / `train_engine.cpp`, which keep their own alias); and **no patch context shifted** — `server.cpp` is untouched in this range, and `0001`/`0002`/`0003`/`0008` keep byte-identical anchors. Test total 487 → 491. | +| b10584–b10585 | **NEW** `common/json.{h,cpp}` + `common/CMakeLists.txt`, `common/chat.h`, `common/chat-auto-parser.h`, `common/chat-peg-parser.h`, `common/json-schema-to-grammar.h`, `common/peg-parser.h`, `common/arg.cpp`, `common/{chat,download,hf-cache,jinja/*,json-schema-to-grammar,peg-parser}.cpp`, `tools/server/{server-common.h,server-common.cpp,server-chat.h,server-chat.cpp,server-context.h,server-context.cpp,server-task.h,server-task.cpp,server-schema.cpp,server-models.cpp,server-tools.cpp}` (**#27511 "common: add json.h abstraction"**) | **BREAKING — the largest project impact of the whole b10456 → b10618 walk.** Upstream introduces the pimpl wrapper class `common_json` and flips `using json = nlohmann::ordered_json` → `using json = common_json` in `server-common.h`, so **every** project TU that includes an upstream server header changes JSON type. `common_json` is a deliberately closed API: no `get_ref`, no `array_t`, no `type_name()`, a braced list in *value* position does not build an array, `at(key)` no longer implicitly converts (needs `.get()`), `get()` is limited to the specialisations in `common/json.cpp`, and errors are `common_json_error`. Project changes, in the order they were found: **(1)** `utils.hpp`'s `str_to_bytes` used `bytes.get_ref().reserve(...)` — dropped (a pure optimisation; `json::array()` + `push_back` is unchanged). **(2)** `jllama.cpp`'s `jsonSchemaToGrammarBytes` parsed with `nlohmann::ordered_json::parse` and handed the result to `json_schema_to_grammar`, which now takes `const common_json &` — there is no conversion, so this is a hard compile error; switched to `json::parse`. **(3) Two silent, compile-clean regressions**, neither of which a build could catch: `jni_helpers.hpp`'s `require_json_field_impl` declared its parameter `const nlohmann::json &`, and a `common_json` **still binds to it** — through `common_json::operator std::string()` feeding nlohmann's string-constructible converting constructor — so the presence check became a `json::type_error 302` ("type must be string, but is object") thrown out of `handleInfill`, whose two call sites sit *outside* its `try` block; the helper is now a template on the JSON type, with two new `common_json` regression tests. And `common_json_value`'s integral constructor template is `std::is_integral`-gated, which **excludes enums**, so `{"vocab_type", }` bound to the `bool` constructor and serialised as `true`/`false`; `ModelMeta.getVocabType()` reads it with Jackson's `asInt(0)`, so every non-SPM model would have reported vocab type 1. Both emit sites now `static_cast(...)`, with a `CommonJsonEnumTrap` C++ pair pinning the trap and a `LlamaModelTest` assertion that the wire value `isIntegralNumber()`. (Both were confirmed empirically with a standalone probe linked against `common/json.cpp`, not inferred.) **(4)** Seven C++ test literals with a braced array in value position (`test_json_helpers.cpp`'s `extract_embedding_prompt` input, `test_server.cpp`'s `dry_sequence_breakers` + five `samplers` cases) now use `json::array({...})`; the produced JSON is identical. **No JSON output shape changed in the project's own emissions**, so no existing assertion moved — but the same enum trap bit **upstream's own** `/models` handler, which the project compiles and serves; that was found by the post-bump review and is carried as `patches/0010` (see the last row of this file); **no CMake change** is needed (`common/json.cpp` rides the `llama-common` target both `jllama` and `jllama_test` already link, and the `nlohmann_json` link stays for `log_helpers.hpp` / `train_engine.cpp`, which keep their own alias); and **no patch context shifted** — `server.cpp` is untouched in this range, and `0001`/`0002`/`0003`/`0008` keep byte-identical anchors. Test total 487 → 491. | | b10584–b10585 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10585 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics **after** the migration above. Full link + `ctest` deferred to the b10618 target. | | b10585–b10590 | `common/json.{cpp,h}` (**#27575 "fix clang lto": the `common_json_value` set/map/unordered_map/vector constructors and every `common_json::get()` flip from explicit *instantiation* to explicit *specialization* — "an explicit instantiation is a weak symbol, dropped by some LTO builds (clang-cl)" — and `operator std::string()` + `value(key, const char *)` move from inline-in-`json.h` to out-of-line in `json.cpp`; the set of usable types is unchanged**), `vendor/sheredom/subprocess.h` + `scripts/sync_vendor.py` (**#27409 upstream resync, +363/−12: new `SUBPROCESS_SPAWN_VIA_FORK` / `SUBPROCESS_ADDCHDIR_IS_POSIX` macros gating a `fork()`+`execve`/`execvpe` launcher with an errno-relay pipe, new `subprocess_pipe_cloexec()` / `subprocess_fds_above_std()` POSIX helpers, and a Windows `STARTUPINFOEX` + `PROC_THREAD_ATTRIBUTE_HANDLE_LIST` handle-inheritance list**), `tools/mtmd/{clip.cpp,clip-graph.h,models/gemma4v.cpp,models/minimax-m3.cpp}` (**#27521: 2D RoPE rewritten from view/rope/`ggml_concat` onto two in-place `ggml_rope_ext` calls + `ggml_rope_set_offset`**) | **No project-source and no project-CMake change.** The `json` work *is* in the project's compile path (the alias became `common_json` at b10585) and the project calls both members that moved out-of-line, but `common/json.cpp` belongs to the `llama-common` target that `jllama` and `jllama_test` already link — nothing to wire, and no `to_json()` shape moved. The project enables no LTO, so the bug being fixed never bit it; the change is robustness for the clang-cl Windows-arm64 job. **`subprocess.h` re-verified against the dropped patch `0009`:** `SUBPROCESS_HAVE_CWD` and its `#elif !SUBPROCESS_HAVE_CWD → posix_error = ENOSYS;` fallback both survive, and `SUBPROCESS_SPAWN_VIA_FORK` is 0 on glibc/bionic/macOS/Windows (only `_AIX`/`__OpenBSD__`/old NetBSD set it), so manylinux2014 (glibc 2.17) still resolves exactly as before and the new `execvpe` declaration is preprocessed away everywhere the project builds — **`0009` stays dropped.** The new always-compiled POSIX helpers need only `pipe2`/`O_CLOEXEC`/`F_DUPFD` (glibc 2.7+/bionic) and the header already includes ``/``; the Windows attribute-list path adds no new link input. mtmd is internal-only here: `build_rope_2d`'s signature is untouched (the `clip-graph.h` hunk is comment-only) and `mtmd.h`/`mtmd-helper.h`/`clip.h` are byte-identical, so `tts_engine.cpp`'s surface is unaffected. **Patches:** zero changes under `tools/server/` and none to `common/arg.{h,cpp}` — byte-identical context for all six. | | b10585–b10590 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10590 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | @@ -668,5 +668,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10614–b10615 | `ggml/src/ggml-metal/**` + `tools/CMakeLists.txt` (**#26570: per-device tuned `(Q, NE)` for the flash-attention vec kernels; a Metal-only `tuning` tool is added under an `if (GGML_METAL)` guard**) | **No project-source change.** The `tools/CMakeLists.txt` addition sits inside the `LLAMA_BUILD_TOOLS`-gated branch the project never enters (`LLAMA_BUILD_TOOLS` defaults to `LLAMA_STANDALONE`, i.e. OFF under FetchContent, and the project forces it OFF on Android), and is additionally Metal-only. Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/` or `vendor/` changed — no patch context moved, no header in the project's include graph touched. | | b10614–b10615 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10615 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10615–b10618 | `src/llama-grammar.cpp` (**#27591: `parse_char()` accepts `case '-':`, so a `\-` escape produced by `gbnf_escape_char_class()` inside a character class no longer throws**), `ggml/src/ggml-sycl/**` (**#27660: `tq2_0` marked unsupported**), `ggml/src/ggml-webgpu/wgsl-shaders/argsort.wgsl` (**#27538: infinity handling in ARGSORT / TOP_K**) | **No project-source change (final step).** Zero lines changed under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt`, so no patch context moved. The grammar fix is strictly *widening* — an input that previously threw now parses — and reaches the project through both `LlamaModel.jsonSchemaToGrammar` and every grammar-constrained completion, so it is a **positive** for the agentic tool-calling target; no test pinned the old throw (`grep -rn 'gbnf_escape_char_class\|parse_char' src/main/cpp src/test/cpp` is empty). The SYCL and WebGPU edits only affect the `sycl-*` classifiers' compiled sources and a backend the project does not build. | -| b10456–b10618 | upstream verification (sandbox, final target) | **Full local verification.** Fresh `cmake -B build` with the real `FetchContent` path (no source-dir override), so the fail-loud `PATCH_COMMAND` ran for real: it resolved `ggml commit eb25b7263` / `LLAMA_BUILD_NUMBER = 10618` and applied all **6** patches — verified by grepping the fetched tree for each patch's marker (`common_params_parse_main`, `params_base.load_progress_callback == nullptr`, `get_slot_prompt_similarity`, `g_llama_server_embedded`, `llama_server_attach` + `llama_server_register_common_routes`, `LLAMA_SERVER_WORKER_CMD`). Then a full `cmake --build --config Release` (jllama + jllama_test both link — the `jllama_test` link is what proves `common_json::get()` resolves, since upstream documents un-specialised types as a *link*-time failure) and `ctest`: **491/491 C++ tests pass**. On the Java side: `mvn test-compile` clean, `mvn test` **1405 run / 0 failures / 17 skipped** (only the model-gated integration tests, no GGUF in the sandbox), and `NativeLibraryLoadSmokeTest` green — including `nativeBuildInfoMatchesPinnedVersionConstant`, the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10618") matches the `build-info` compiled into the freshly linked `libjllama.so`. `mvn spotless:apply` produced no changes beyond the edits themselves, the pinned **clang-format 22.1.8** reports the whole C++ tree clean, and `mvn clean javadoc:jar` is `BUILD SUCCESS`. **Per-step verification for all 25 chunks:** every intermediate tag additionally had all six patches applied by the applier's own commands and all ten project + C++-test translation units compiled against it, so no commit in this walk is a state that fails to build. | -| b10456–b10618 | `tools/server/server-context.cpp` — **new local patch `0009-server-cast-vocab-type-for-common-json.patch`** | **Upstream regression found by the post-bump review, fixed downstream.** `get_res_model_info()` builds the `GET /models` + `GET /v1/models` payload and emits `{"vocab_type", meta.model_vocab_type}`, an **unscoped enum**. That was correct while the `json` alias was `nlohmann::ordered_json` (nlohmann serialises an enum as an integer), but `common_json_value`'s integral constructor template is `std::is_integral`-gated, which excludes enums — so from **b10585** the value binds to `common_json_value(bool)` and goes on the wire as `true`/`false`. Upstream regressed it silently in #27511 when they flipped the alias; the same trap hit the project's own two `"vocab_type"` emit sites in `jllama.cpp` (fixed in the b10584–b10585 step). It **ships**: `server-context.cpp` is compiled into `libjllama` and both routes are served by `NativeServer`, the default fat-jar `Main-Class`, in full and in attach mode. The new patch casts the value to `int` at the emit site — one line, upstream-submittable, applied after `0002`/`0003` (same file). A mechanical sweep confirmed it is the **only** such site in the upstream server sources the project compiles: `server_context_meta` has exactly two enum-typed members, and the other (`pooling_type`) is only ever compared, never serialised. | +| b10456–b10618 | upstream verification (sandbox, final target) | **Full local verification, re-run from scratch on the shipped tree.** Fresh `cmake -B build` with the real `FetchContent` path (no source-dir override), so the fail-loud `PATCH_COMMAND` ran for real: it resolved `ggml commit eb25b7263` / `LLAMA_BUILD_NUMBER = 10618` and applied all **7** patches — verified by grepping the fetched tree for each patch's marker (`common_params_parse_main`, `params_base.load_progress_callback == nullptr`, `get_slot_prompt_similarity`, `g_llama_server_embedded`, `llama_server_attach` + `llama_server_register_common_routes`, `LLAMA_SERVER_WORKER_CMD`, `(int) meta.model_vocab_type`). The **7**, not 6, matters: the review-driven `0010` was added after the first verification pass and is the *third* patch to touch `server-context.cpp` (after `0002`/`0003`), a TU compiled into both `jllama` and `jllama_test`, so the whole configure + build + `ctest` cycle was re-run on the final tree rather than assumed to carry over (**491/491**, `mvn test` unchanged at 1405 run / 0 failures / 17 model-gated skips). The patch was subsequently renamed `0009` → `0010` to free the burned number; that is a filename-only change with the same sort position, re-checked by applying the whole set to a clean b10618 checkout. Then a full `cmake --build --config Release` (jllama + jllama_test both link — the `jllama_test` link is what proves `common_json::get()` resolves, since upstream documents un-specialised types as a *link*-time failure) and `ctest`: **491/491 C++ tests pass**. On the Java side: `mvn test-compile` clean, `mvn test` **1405 run / 0 failures / 17 skipped** (only the model-gated integration tests, no GGUF in the sandbox), and `NativeLibraryLoadSmokeTest` green — including `nativeBuildInfoMatchesPinnedVersionConstant`, the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10618") matches the `build-info` compiled into the freshly linked `libjllama.so`. `mvn spotless:apply` produced no changes beyond the edits themselves, the pinned **clang-format 22.1.8** reports the whole C++ tree clean, and `mvn clean javadoc:jar` is `BUILD SUCCESS`. **Per-step verification for all 25 chunks:** every intermediate tag additionally had all six patches applied by the applier's own commands and all ten project + C++-test translation units compiled against it, so no commit in this walk is a state that fails to build. | +| b10456–b10618 | `tools/server/server-context.cpp` — **new local patch `0010-server-cast-vocab-type-for-common-json.patch`** | **Upstream regression found by the post-bump review, fixed downstream.** `get_res_model_info()` builds the `GET /models` + `GET /v1/models` payload and emits `{"vocab_type", meta.model_vocab_type}`, an **unscoped enum**. That was correct while the `json` alias was `nlohmann::ordered_json` (nlohmann serialises an enum as an integer), but `common_json_value`'s integral constructor template is `std::is_integral`-gated, which excludes enums — so from **b10585** the value binds to `common_json_value(bool)` and goes on the wire as `true`/`false`. Upstream regressed it silently in #27511 when they flipped the alias; the same trap hit the project's own two `"vocab_type"` emit sites in `jllama.cpp` (fixed in the b10584–b10585 step). It **ships**: `server-context.cpp` is compiled into `libjllama` and both routes are served by `NativeServer`, the default fat-jar `Main-Class`, in full and in attach mode. The new patch casts the value to `int` at the emit site — one line, upstream-submittable (not yet filed), applied after `0002`/`0003` (same file). Numbered `0010`, not `0009`: that number belongs to the subprocess.h patch dropped at b10280 and is documented as such under the patches table in `CLAUDE.md`, so reusing it would make that note read as if it described this patch. A mechanical sweep confirmed it is the **only** such site in the upstream server sources the project compiles: `server_context_meta` has exactly two enum-typed members, and the other (`pooling_type`) is only ever compared, never serialised. | diff --git a/docs/upgrade/llama-cpp-version-bump.md b/docs/upgrade/llama-cpp-version-bump.md index 44389226b..5c89f781e 100644 --- a/docs/upgrade/llama-cpp-version-bump.md +++ b/docs/upgrade/llama-cpp-version-bump.md @@ -120,9 +120,12 @@ Concretely: if you forget it, `NativeLibraryLoadSmokeTest.nativeBuildInfoMatchesPinnedVersionConstant` fails the build (it cross-checks the constant against `LlamaModel.getLlamaCppBuildInfo()`, which reads llama.cpp's own linked-in `build-info`). -2. **Re-verify `patches/`** — a clean configure re-runs the fail-loud `PATCH_COMMAND`, so all six - patches (`0001`, `0002`, `0003`, `0006`, `0007`, `0008`) must still apply. Use a **fresh** build - dir (a stale one re-applies over an already-patched tree and reports a false "does not apply"): +2. **Re-verify `patches/`** — a clean configure re-runs the fail-loud `PATCH_COMMAND`, so **every + `*.patch` in `llama/patches/`** must still apply. Do not maintain a list of them here or anywhere + else: `apply-llama-patches.cmake` `file(GLOB)`s the directory and applies them in filename order, + so an enumeration can only go stale (it did, one commit after being written). Use a **fresh** + build dir (a stale one re-applies over an already-patched tree and reports a false "does not + apply" — see the applier note in `TODO.md` for why): ```bash cd llama && mvn -q compile # generates the OSInfo class CMake's OS-detection needs rm -rf build && cmake -B build # fail-loud: aborts here if any patch no longer applies diff --git a/llama/patches/0009-server-cast-vocab-type-for-common-json.patch b/llama/patches/0010-server-cast-vocab-type-for-common-json.patch similarity index 100% rename from llama/patches/0009-server-cast-vocab-type-for-common-json.patch rename to llama/patches/0010-server-cast-vocab-type-for-common-json.patch From 484755af79de1a7876b637d62eb18dbedab6f57e Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 25 Aug 2026 11:09:00 +0000 Subject: [PATCH 29/76] docs: correct the bump records the review found unsupported The docs-accuracy review audited the ~50 new history rows against the upstream mirror and found the per-range patch-risk prose rests on a directory scan (common, tools/server, tools/mtmd, vendor, CMakeLists) that misses 35 of the 40 files the patch set touches -- patch 0001 alone flips ~34 standalone main() call sites under tools, examples and tests. Two ranges did touch one of those and said otherwise: b10488-b10499 (tools/perplexity/perplexity.cpp, two lines from an 0001 hunk, the closest call of the walk) and b10590-b10593 (tests/test-recurrent-state-rollback.cpp, +172 lines above an 0001 hunk). Both rows corrected, "the five patched files" relabelled as the five core files, and the final row now explains how to read the per-chunk prose and gives the mechanical check. The verdicts themselves never came from that scan -- each chunk applied the whole set to a clean checkout, which covers all 40 files. Also corrected: json{} is JSON null, not an empty object (two docs still said otherwise after the code comment was fixed, and one cited a test name that no longer exists); the common_json migration touched fourteen literals across three files, not seven, and test_utils.cpp direct-init json{1,2,3} is a second, distinct incompatibility; MTMD_VIDEO stays on for Android here because dockcross sets neither ANDROID nor CMAKE_SYSTEM_NAME, not because LLAMA_SUBPROCESS is unset (upstream defaults it off there); git ls-tree compares blob hashes, so the unchanged-file-set claim needs --name-only. And two places where this branch shipped a claim it had itself disproved: CLAUDE.md and the applier header called the patch applier idempotent, and the CI model policy said the model tests run on every platform. Both now state the real behaviour and point at TODO.md, as does the vocab_type guard note, since the LlamaModelTest assertion is model-gated and only CommonJsonEnumTrap runs in CI. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 15 +++++++++------ docs/history/llama-cpp-breaking-changes.md | 20 ++++++++++---------- docs/upgrade/llama-cpp-version-bump.md | 2 +- llama/cmake/apply-llama-patches.cmake | 8 ++++++-- 4 files changed, 26 insertions(+), 19 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 9304f254f..4536ff6b2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -641,9 +641,12 @@ The fetched llama.cpp source is patched before it compiles, via a generic mechan ordered. Each must be a `git apply`-compatible unified diff with paths relative to the llama.cpp source root (`a/common/arg.cpp` / `b/common/arg.cpp`, i.e. `-p1`). - **`llama/cmake/apply-llama-patches.cmake`** — the applier. Cross-platform (`cmake -P`, so identical on - Linux/macOS/Windows), **idempotent** (`git apply --reverse --check` skips already-applied patches - so a reconfigure never double-applies) and **fail-loud** (a patch that no longer applies aborts - the configure — a stale patch can't be silently dropped from a release build). + Linux/macOS/Windows), **idempotent only while no two patches touch the same file** + (`git apply --reverse --check` skips an already-applied patch — but `0006`/`0007` rewrite the + region of `tools/server/server.cpp` that `0001` also patches, so `0001`'s reverse-check no longer + matches and a **reconfigure over an already-patched tree aborts**; always configure into a fresh + build dir, and see the entry in [`TODO.md`](TODO.md)) and **fail-loud** (a patch that no longer + applies aborts the configure — a stale patch can't be silently dropped from a release build). - **`llama/CMakeLists.txt`** — wired as the llama.cpp `FetchContent_Declare(... PATCH_COMMAND ...)`, so it runs for **every** C++ build (all CI jobs *and* local `cmake -B build`) from one place — no per-build-step plumbing. @@ -1131,7 +1134,7 @@ If the local check passes (`BUILD SUCCESS`), the `mvn package` job in - `json_helpers.hpp` — Pure JSON transformation helpers (no JNI, no llama state). Independently unit-testable. - `jni_helpers.hpp` — JNI bridge helpers (handle management + server orchestration). Includes `json_helpers.hpp`. - **The `json` alias is upstream's `common_json`, not `nlohmann::ordered_json` (since llama.cpp b10585, upstream #27511).** `tools/server/server-common.h` now says `using json = common_json;` — a deliberately small pimpl wrapper (`common/json.{h,cpp}`, compiled into `llama-common`) around the vendored nlohmann copy. Two traps this cost the project once, both of which **compile silently**: - 1. **An unscoped enum becomes a JSON boolean.** `common_json_value`'s integral constructor template is `std::is_integral`-gated, which excludes enums, so an enum binds to `common_json_value(bool)`. Always `static_cast(...)` an enum before putting it in JSON — `jllama.cpp`'s two `"vocab_type"` sites do, and `test_json_helpers.cpp`'s `CommonJsonEnumTrap` tests plus `LlamaModelTest`'s `isIntegralNumber()` assertion guard it. + 1. **An unscoped enum becomes a JSON boolean.** `common_json_value`'s integral constructor template is `std::is_integral`-gated, which excludes enums, so an enum binds to `common_json_value(bool)`. Always `static_cast(...)` an enum before putting it in JSON — `jllama.cpp`'s two `"vocab_type"` sites do, and `patches/0010` does the same for upstream's own `/models` handler. Guards: `test_json_helpers.cpp`'s `CommonJsonEnumTrap` pair pins the mechanism and **does** run in CI; `LlamaModelTest`'s `isIntegralNumber()` assertion pins the real wire value but is model-gated, so it currently only fires locally with a GGUF present (see the CI-skip entry in [`TODO.md`](TODO.md)). 2. **`common_json` converts to `std::string` implicitly**, so it binds happily to a `const nlohmann::json &` parameter (via nlohmann's string-constructible converting constructor) and then throws `json::type_error 302` at runtime. Never declare a project helper as taking `nlohmann::json` when callers pass the `json` alias — `require_json_field_impl` is a template for exactly this reason. Other differences to know: no `get_ref`/`array_t`/`type_name()`; a braced list in *value* position does not build an array (write `json::array({...})`); `at(key)` needs an explicit `.get()`; errors are `common_json_error`; and `get()` is limited to the types explicitly specialised in `common/json.cpp`. `log_helpers.hpp` and `train_engine.cpp` keep their own `nlohmann::json` alias — they never touch the server's `json`. - Uses `nlohmann/json` for JSON deserialization of parameters in the two files named above; everything on the server path uses `common_json`. @@ -1291,7 +1294,7 @@ model + mmproj, and the Qwen3-TTS backbone + mmproj (`ggml-org/Qwen3-TTS-12Hz-1. smallest available quants: `Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf` backbone + `mmproj-Qwen3-TTS-12Hz-1.7B-Base-Q8_0.gguf` mmproj — no smaller mmproj quant is published), with their `-Dnet.ladenthin.llama.*` properties set, so `LlamaEmbeddingsTest`, `MultimodalIntegrationTest`, -and `TtsIntegrationTest` **run on every platform** rather than self-skipping. `validate-models.{sh,bat}` +and `TtsIntegrationTest` are **intended** to run on every platform rather than self-skipping. **Known broken as of the b10618 bump:** Surefire's working directory is the `llama/` module while CI restores the cache to `/models/`, so every model-gated test resolves `models/…` to a path that does not exist and aborts in `@BeforeAll` — they self-skip on *every* `test-java-*` job while the job still reports success. See "Model-gated Java tests silently self-skip in CI" in [`TODO.md`](TODO.md). `validate-models.{sh,bat}` treats all of these as **required** (a missing model hard-fails the job before tests run, so a download regression can never silently downgrade to a skip). Only the audio-input model (`AudioInputIntegrationTest`) still self-skips — the prompt clip is committed @@ -1353,7 +1356,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" | File | Tests | Scope | |------|-------|-------| | `src/test/cpp/test_utils.cpp` | 162 | Upstream helpers: `server_tokens`, `server_grammar_trigger`, `gen_tool_call_id`, `json_value`, `json_get_nested_values`, UTF-8 helpers, `format_response_rerank`, `format_embeddings_response_oaicompat`, `oaicompat_completion_params_parse`, `oaicompat_chat_params_parse`, `are_lora_equal`, `strip_flag_from_argv`, `token_piece_value`, `json_is_array_and_contains_numbers`, `format_oai_sse`, `format_oai_resp_sse`, `format_anthropic_sse`, `parse_lora_request` | -| `src/test/cpp/test_server.cpp` | 206 | Upstream result types: `server_slot_stats` (the `timings` JSON payload; replaced `result_timings` in b10408), `task_params::to_json()` (incl. `dry_sequence_breakers`, `preserved_tokens`, `timings_per_token`), `completion_token_output`, `server_task_result_cmpl_partial` (non-oaicompat + `to_json_oaicompat` + logprobs + `to_json_oaicompat_chat` + `to_json_anthropic` + dispatcher), `server_task_result_cmpl_final` (non-oaicompat + `to_json_oaicompat` + `to_json_oaicompat_chat` + `to_json_oaicompat_chat_stream` + `to_json_anthropic` + `to_json_anthropic_stream` + tool_calls + dispatcher), `server_task_result_embd`, `server_task_result_rerank`, `server_task_result_metrics` (`to_metrics()` = the `/metrics` Prometheus exposition text; its `to_json()` is an unused empty object since b10519), `server_task_result_slots` (`to_json()` = the `/slots` array, fed by the b10519 `SERVER_TASK_TYPE_SLOT_GET` task), `server_task_result_slot_save_load`, `server_task_result_slot_erase`, `server_task_result_apply_lora`, `server_task_result_get_lora`, `server_task_result_error`, `format_error_response`, `server_task::need_sampling()`, `server_task::n_tokens()`, `server_schema::eval_llama_cmpl_schema()` (parsing pipeline + grammar routing + error paths + per-request `dry_*` and `sse_ping_interval` field round-trips incl. hard-limit + server-default inheritance), `response_fields` projection | +| `src/test/cpp/test_server.cpp` | 206 | Upstream result types: `server_slot_stats` (the `timings` JSON payload; replaced `result_timings` in b10408), `task_params::to_json()` (incl. `dry_sequence_breakers`, `preserved_tokens`, `timings_per_token`), `completion_token_output`, `server_task_result_cmpl_partial` (non-oaicompat + `to_json_oaicompat` + logprobs + `to_json_oaicompat_chat` + `to_json_anthropic` + dispatcher), `server_task_result_cmpl_final` (non-oaicompat + `to_json_oaicompat` + `to_json_oaicompat_chat` + `to_json_oaicompat_chat_stream` + `to_json_anthropic` + `to_json_anthropic_stream` + tool_calls + dispatcher), `server_task_result_embd`, `server_task_result_rerank`, `server_task_result_metrics` (`to_metrics()` = the `/metrics` Prometheus exposition text; its `to_json()` has been unused since b10519 and returns `json{}` = JSON null), `server_task_result_slots` (`to_json()` = the `/slots` array, fed by the b10519 `SERVER_TASK_TYPE_SLOT_GET` task), `server_task_result_slot_save_load`, `server_task_result_slot_erase`, `server_task_result_apply_lora`, `server_task_result_get_lora`, `server_task_result_error`, `format_error_response`, `server_task::need_sampling()`, `server_task::n_tokens()`, `server_schema::eval_llama_cmpl_schema()` (parsing pipeline + grammar routing + error paths + per-request `dry_*` and `sse_ping_interval` field round-trips incl. hard-limit + server-default inheritance), `response_fields` projection | | `src/test/cpp/test_json_helpers.cpp` | 52 | All functions in `json_helpers.hpp`: `get_result_error_message`, `results_to_json`, `rerank_results_to_json` (incl. missing/out-of-range `index` rejection), `parse_encoding_format`, `extract_embedding_prompt`, `is_infill_request`, `parse_slot_prompt_similarity`, `parse_positive_int_config`, `wrap_stream_chunk` | | `src/test/cpp/test_log_helpers.cpp` | 13 | All functions in `log_helpers.hpp`: `log_level_name`, `format_log_as_json` | | `src/test/cpp/test_jni_helpers.cpp` | 56 | All functions in `jni_helpers.hpp` using a zero-filled `JNINativeInterface_` mock (incl. the `utf8_to_jstring_impl` byte-array string path: emoji byte-preservation, truncated-UTF-8 replace-not-throw) | diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 77040815e..178996600 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -619,21 +619,21 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10454–b10455 | `ggml/src/ggml-sycl/{ggml-sycl.cpp,opt-step.cpp,opt-step.hpp}` (**SYCL `OPT_STEP_ADAMW` / `OPT_STEP_SGD`, #25268**), `docs/ops/SYCL.csv` + `docs/ops.md` + `examples/sycl/update-ops-doc.sh` (**regenerated op-support table**) | **No project-source change.** A single commit inside the SYCL backend, so only the `sycl-{fp16,fp32}-linux-x86-64` / `sycl-windows-x86-64` classifiers' compiled sources change. Step forced above the 100 KiB threshold (3.2 MiB) — but ~3.2 MiB of that is the regenerated `docs/ops/SYCL.csv` alone (20 646 lines rewritten); the reviewable code is ~150 lines, and upstream has no tag between b10454 and b10455 anyway. | | b10455–b10456 | `ggml/src/ggml-sycl/cpy.cpp` (**thread/block count fixed in the quantized cpy kernel launches, #27160**) | **No project-source change (final step).** A single one-commit fix inside the SYCL backend; no shared header touched, so only the `sycl-{fp16,fp32}-linux-x86-64` / `sycl-windows-x86-64` classifiers' compiled sources change. | | b10423–b10456 | upstream verification (sandbox, final target) | All **6** patches re-verified against a clean b10456 checkout: sequential `git apply` (filename order, mirroring the `PATCH_COMMAND` applier) succeeded at **every** intermediate tag of the walk, and the fail-loud `PATCH_COMMAND` ran clean on a fresh `cmake -B build` at the target — no patch needed refreshing across the whole range. **Full local verification (mandatory for the final target):** fresh configure (resolved `ggml commit f275595dd` = b10456 HEAD; all six patch markers confirmed present in the fetched tree) + full `cmake --build` (`libjllama.so` + `jllama_test` compile and link, `-O3`, no undefined references) + `ctest` **485/485 passing**. **API surface:** of the 22 upstream headers project source `#include`s directly, only three changed across the whole range — `tools/server/server-queue.h` (breaking: `on_new_task`'s callback type became `std::function`, plus the new `yield_to_queue()`; inert here, upstream's own `server-context.cpp` is the sole registrant), `ggml/include/ggml.h` (`ggml_ssm_scan()` gained a trailing `int64_t K`; called only inside llama.cpp's own TUs) and `tools/mtmd/mtmd-helper-common.h` (`size_t` widening, no signature change). `mtmd.h`, `mtmd-helper.h`, `common.h`, `chat.h`, `llama.h`, `arg.h` and every `tools/server/*.h` the project includes are byte-identical b10423→b10456; no files were added or removed under `tools/server`/`tools/mtmd`/`vendor` and no upstream `CMakeLists.txt` changed, so the b10154 `server-mcp.cpp` missing-`target_sources` failure class does not recur. Walk of 8 steps (b10423→b10430→b10436→b10441→b10447→b10448→b10454→b10455→b10456) on branch `claude/update-b10456-8a7j58`. Two steps exceeded the 100 KiB chunking threshold unavoidably — upstream has **no** intermediate tags there (b10449–b10453 and b10456 aside, nothing sits between b10447/b10448 or b10454/b10455) — and the 3.2 MiB b10454→b10455 figure is almost entirely the regenerated `docs/ops/SYCL.csv` (20 646 lines rewritten), leaving ~150 lines of reviewable code. | -| b10456–b10470 | `tools/server/server-tools.cpp` (**built-in `get_datetime` tool removed, −57 lines, plus a "keep this array minimal" contributor note**) + `common/arg.cpp` & `tools/server/README.md` (**matching one-line `--tools` help-text edit**), `common/chat-diff-analyzer.cpp` (**additive Bailing-V3 template-patch lambda**), `common/speculative.cpp` (**`dflash.sample_from_anchor` GGUF knob in the file-local DFlash/DSpark impl**), `tools/mtmd/mtmd-image.cpp` (**two `GGML_ASSERT` bounds checks in the granite preprocessor**), `CMakeLists.txt` + `ggml/CMakeLists.txt` (**`*_VERSION_PATCH` 0→1**) | **No project-source change.** The only change in a file the project compiles is `server-tools.cpp`, but both the deleted `struct server_tool_get_datetime` and `build_tools()` are file-internal (declared in no header) and `server-tools.cpp` is deliberately not linked into `jllama_test` — greps for `get_datetime`/`server_tool`/`build_tools`/`find_tool`/`"/tools"` over `src/main/cpp` + `src/test/cpp` + the Java tree return zero hits; the sole effect is that NativeServer full mode's `/tools` advertises one fewer built-in (attach mode already 403s that route via patch 0007). No public header changed: `common/speculative.h`, `common/chat.h`, `tools/mtmd/mtmd{,-helper}.h` and every `tools/server/*.h` are untouched, so the speculative/chat-analyzer/mtmd edits stay inside upstream-compiled TUs. **Patch context intact:** the `arg.cpp` edit sits at line ~3362, >2000 lines from patch `0001`'s hunks (`@@ -1201` / `@@ -1242`), and `server.cpp`/`server-context.cpp`/`server-models.cpp` are not in the range at all. **Build wiring:** the `tools/server/` file set is byte-identical b10456→b10470 (`git ls-tree` diff empty) and `vendor/`+`include/` are untouched, so `target_sources` needs no edit and the b10154 `server-mcp.cpp` missing-source link-failure class does not recur. | +| b10456–b10470 | `tools/server/server-tools.cpp` (**built-in `get_datetime` tool removed, −57 lines, plus a "keep this array minimal" contributor note**) + `common/arg.cpp` & `tools/server/README.md` (**matching one-line `--tools` help-text edit**), `common/chat-diff-analyzer.cpp` (**additive Bailing-V3 template-patch lambda**), `common/speculative.cpp` (**`dflash.sample_from_anchor` GGUF knob in the file-local DFlash/DSpark impl**), `tools/mtmd/mtmd-image.cpp` (**two `GGML_ASSERT` bounds checks in the granite preprocessor**), `CMakeLists.txt` + `ggml/CMakeLists.txt` (**`*_VERSION_PATCH` 0→1**) | **No project-source change.** The only change in a file the project compiles is `server-tools.cpp`, but both the deleted `struct server_tool_get_datetime` and `build_tools()` are file-internal (declared in no header) and `server-tools.cpp` is deliberately not linked into `jllama_test` — greps for `get_datetime`/`server_tool`/`build_tools`/`find_tool`/`"/tools"` over `src/main/cpp` + `src/test/cpp` + the Java tree return zero hits; the sole effect is that NativeServer full mode's `/tools` advertises one fewer built-in (attach mode already 403s that route via patch 0007). No public header changed: `common/speculative.h`, `common/chat.h`, `tools/mtmd/mtmd{,-helper}.h` and every `tools/server/*.h` are untouched, so the speculative/chat-analyzer/mtmd edits stay inside upstream-compiled TUs. **Patch context intact:** the `arg.cpp` edit sits at line ~3362, >2000 lines from patch `0001`'s hunks (`@@ -1201` / `@@ -1242`), and `server.cpp`/`server-context.cpp`/`server-models.cpp` are not in the range at all. **Build wiring:** the `tools/server/` **file-name** set is unchanged b10456→b10470 (`git ls-tree --name-only` diff empty; contents did change, see the left column) and `vendor/`+`include/` are untouched, so `target_sources` needs no edit and the b10154 `server-mcp.cpp` missing-source link-failure class does not recur. | | b10456–b10470 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10470 checkout: sequential `git apply --check` + `git apply` in filename order (mirroring the fail-loud `PATCH_COMMAND` applier) succeeded with zero fuzz. **Compile-verified per step:** a fresh `cmake -B build` against the patched b10470 tree configured clean, and all ten project translation units (`jllama.cpp`, `native_server.cpp`, `tts_engine.cpp`, `train_engine.cpp` + the six `src/test/cpp/test_*.cpp`) compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | -| b10470–b10472 | `CMakeLists.txt` + **NEW** `vendor/hash/**` (`CMakeLists.txt`, `rotate-bits/`, `sha1/`, `sha256/`, `xxhash/`) (**#27262: `examples/gguf-hash/deps/` promoted to a first-class `vendor-hash` static library, added to the root `CMakeLists.txt` *unconditionally* — before the `LLAMA_BUILD_COMMON` gate — because "mtmd needs this even when common is not built"**), `ggml/src/ggml-cuda/**` (**#27083, UMA override skipped for HIP**) | **No project-source and no project-CMake change.** `FetchContent_MakeAvailable(llama.cpp)` processes upstream's root `CMakeLists.txt`, so the new `vendor-hash` target is created before the project's own `add_subdirectory(${llama.cpp_SOURCE_DIR}/tools/mtmd)` — including on Android, where the project forces `LLAMA_BUILD_TOOLS OFF`/`LLAMA_BUILD_COMMON` off. Nothing links it yet in this step (mtmd picks it up at b10481). No header, no `tools/server` file and none of the five patched files changed, so all six patches keep byte-identical context. | +| b10470–b10472 | `CMakeLists.txt` + **NEW** `vendor/hash/**` (`CMakeLists.txt`, `rotate-bits/`, `sha1/`, `sha256/`, `xxhash/`) (**#27262: `examples/gguf-hash/deps/` promoted to a first-class `vendor-hash` static library, added to the root `CMakeLists.txt` *unconditionally* — before the `LLAMA_BUILD_COMMON` gate — because "mtmd needs this even when common is not built"**), `ggml/src/ggml-cuda/**` (**#27083, UMA override skipped for HIP**) | **No project-source and no project-CMake change.** `FetchContent_MakeAvailable(llama.cpp)` processes upstream's root `CMakeLists.txt`, so the new `vendor-hash` target is created before the project's own `add_subdirectory(${llama.cpp_SOURCE_DIR}/tools/mtmd)` — including on Android, where the project forces `LLAMA_BUILD_TOOLS OFF`/`LLAMA_BUILD_COMMON` off. Nothing links it yet in this step (mtmd picks it up at b10481). No header, no `tools/server` file and none of the five **core** patched files (`common/arg.cpp`, `server-context.{cpp,h}`, `server.cpp`, `server-models.cpp`) changed, so all six patches keep byte-identical context. | | b10470–b10472 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10472 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10472–b10481 | `tools/mtmd/{CMakeLists.txt,mtmd.h,mtmd.cpp,mtmd-helper.{h,cpp},mtmd-image.{h,cpp}}` (**#27274: bitmap IDs switch from a private FNV-1a decimal hash to `hash_sha256_hex()` "to prevent cache poisoning"; mtmd now links `vendor-hash`. Additive public API `mtmd_input_chunk_get_placeholder()`, `mtmd_input_chunk_save()` refactored around a shared impl with an unchanged C signature. #27246: LFM2 thumbnail skipped for non-tiled images**), `tools/server/server-common.{h,cpp}` + `tools/server/server-context.cpp` (**#27278: additive `server_tokens::push_back_placeholder()`; already-encoded media chunks are cached as metadata-only placeholders**), `tools/server/server-tools.{h,cpp}` + `tools/server/server.cpp` (**#27271: `server_tool::type()` default renamed `"builtin"` → `"server"`; one help-string line**), `vendor/hash/{CMakeLists.txt,hash.{h,cpp},sha1/**}` (**C++ `hash_sha256_hex()` wrapper; sha1 namespaced `vendor_hash`**), `CMakeLists.txt`, `tools/server/README{,-dev}.md` | **No project-source change.** The mtmd hash switch is invisible here: the project's only bitmap call site is `src/main/cpp/tts_engine.cpp:100` (`mtmd_helper_bitmap_init_from_file`, the TTS speaker-reference clip) — signature unchanged and the ID is never read back (`grep -rn "bitmap" src/main/cpp` shows no ID use). `mtmd_input_chunk_*` and `push_back_placeholder` are additive: `push_back(const mtmd_input_chunk *)` and `push_back(server_tokens &)` both survive, so every TU including `server-common.h` (`utils.hpp`, `jllama.cpp`, `native_server.cpp` + three test files) compiles unchanged, and the project's `server_tokens` tests all construct with `has_mtmd=false`. `server_tool::type()`'s renamed default is a `GET /tools` JSON field upstream marks "do NOT use in a downstream application" — zero hits for `"builtin"` in the project's C++ **and** Java trees, and no C++ test asserts on `server_tool::to_json()`. The vendored sha1 being namespaced `vendor_hash` matters for one project-specific reason: `llama/CMakeLists.txt` forces `LLAMA_BUILD_BORINGSSL ON` on Windows, so BoringSSL and `vendor-hash` land in the same `jllama.dll` — the namespace prevents a `SHA1*` symbol clash, and `vendor-hash` is linked `PRIVATE` into mtmd so its include dir never shadows anything. **Patch context intact:** the two one-line edits in patched files sit far outside every hunk (`server.cpp:346` between patch 0007's `@@ -258,47` and `@@ -556,3`; `server-context.cpp:3416` ~2 260 lines from patch 0002's `@@ -1152,8`). **Build wiring:** no `tools/server` source added/removed/renamed, so `llama/CMakeLists.txt`'s explicit `target_sources` list still matches upstream. | | b10472–b10481 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10481 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10481–b10488 | `CMakeLists.txt` + `common/CMakeLists.txt` + **NEW** `vendor/CMakeLists.txt` and `vendor/{cpp-httplib,hash,miniaudio,nlohmann,sheredom,stb}/CMakeLists.txt` (**#27304 "build : fix xcframework + cmake clean-up": one `add_subdirectory(vendor)`, per-vendor `vendor::` aliases, header-only deps become INTERFACE targets exporting the `vendor/` root; `common` drops `PUBLIC ../vendor` in favour of `PUBLIC vendor::nlohmann vendor::sheredom`; `LLAMA_VERSION_PATCH` 1→2**), `tools/mtmd/{CMakeLists.txt,mtmd-helper.cpp}` (**mtmd links `vendor::{hash,miniaudio,stb,sheredom}` instead of raw `../..`/`../../vendor` include dirs; `"hash.h"` → `"hash/hash.h"`**), `tools/mtmd/{mtmd-image.{h,cpp},clip-impl.h}` (**#27057: LFM2 tiling threshold reworked into a new `should_tile()`; `#ifndef DIRECTORY_SEPARATOR` guard**) | **No project-source and no project-CMake change** — but this is the range with the one real build-wiring risk, so it was checked rather than assumed. The project compiles `tools/server/server-http.cpp` (whose `#include ` needs the `vendor/` root on the include path) and `vendor/cpp-httplib/httplib.cpp` into `jllama` itself, adds no vendor include directory of its own, and rode on `llama-common`'s `PUBLIC ../vendor` — which b10488 deletes. It still resolves because `vendor::nlohmann` / `vendor::sheredom` are linked **PUBLIC** into `llama-common` and each declares `target_include_directories( INTERFACE ..)` = the `vendor/` root, which CMake propagates transitively to `jllama` (confirmed by the compile of `server-http.cpp` and `httplib.cpp` in this step's verification build). `add_subdirectory(vendor)` is unconditional, so the `vendor::*` ALIAS targets exist before the project's own `add_subdirectory(tools/mtmd)`, Android included. The new bare target names (`nlohmann`, `sheredom`, `miniaudio`, `stb`) do not clash with the project's FetchContent nlohmann/json, whose target is `nlohmann_json`; the vendored copy stays 3.12.0, the same version. mtmd's PUBLIC surface and every `tools/server/*` header are unchanged. **Patches:** none of `common/arg.{h,cpp}`, `tools/server/server.cpp`, `server-context.{cpp,h}`, `server-models.cpp` changed in this range — zero context shift. *Optional hardening, deliberately NOT done in this bump:* adding `${llama.cpp_SOURCE_DIR}/vendor` to `jllama`'s own `target_include_directories` would make the httplib include explicit instead of transitive, and immune to a future upstream `PUBLIC`→`PRIVATE` flip on `llama-common`. | | b10481–b10488 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10488 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Because this is the range that deletes `llama-common`'s `PUBLIC ../vendor` include directory, the two upstream TUs the project compiles itself that depend on it — `tools/server/server-http.cpp` (`#include `) and `vendor/cpp-httplib/httplib.cpp` — were additionally compiled against the patched b10488 tree, both clean: the include still resolves transitively through the PUBLIC `vendor::nlohmann` / `vendor::sheredom` INTERFACE targets. Full link + `ctest` deferred to the b10618 target. | -| b10488–b10499 | `common/common.cpp` (**#27138: threadpool sharing when only `n_threads` differs — new file-static `can_share_threadpool()`; a `-t`/`-tb` mismatch yields ONE unpaused pool sized `max(n, nb)` instead of two**), `tools/server/server-models.cpp` (**#27347: `CMD_CHILD_TO_ROUTER_STATE` lines demoted `LOG` → `LOG_DBG`, other child output forwarded in a new `else` branch**), `tools/mtmd/models/deepseekocr.cpp` (**#26727: DeepSeek-OCR SAM convolutions re-expressed via a file-static `conv_2d_f32()` = `ggml_im2col` + `ggml_mul_mat` keeping the im2col in F32**), plus out-of-scope `tools/ui/**`, `ggml/src/ggml-{opencl,sycl}/**`, `ggml/include/ggml-rpc.h`, `src/llama-model-loader.h` | **No project-source change.** All three in-scope edits are behavioural/internal with no API surface moved: `common/common.h` is byte-identical across the range (so `struct common_threadpools` and every signature are unchanged) and no project source references `threadpool`/`ggml_threadpool_params_match`; the `server-models.cpp` edit is pure logging (`RouterModeIntegrationTest` drives the typed `RouterClient` over HTTP, not the log); the deepseekocr helper is `static` and compiles against the unchanged 11-parameter `ggml_im2col` declaration, and neither `mtmd.h` nor `mtmd-helper.h` changed. **Patches:** only `server-models.cpp` of the five patched files changed, ~810 lines below patch `0008`'s single `@@ -215 @@` hunk — no context shift. **Build wiring:** no `CMakeLists.txt` changed except upstream's own `tests/`, and no file was added/removed/renamed under `tools/server`, `tools/mtmd`, `vendor` or `include`. The three headers that changed anywhere (`ggml-rpc.h`, `ggml-sycl/fwht.hpp`, `src/llama-model-loader.h`) are all outside the project's include dependency graph. | +| b10488–b10499 | `common/common.cpp` (**#27138: threadpool sharing when only `n_threads` differs — new file-static `can_share_threadpool()`; a `-t`/`-tb` mismatch yields ONE unpaused pool sized `max(n, nb)` instead of two**), `tools/server/server-models.cpp` (**#27347: `CMD_CHILD_TO_ROUTER_STATE` lines demoted `LOG` → `LOG_DBG`, other child output forwarded in a new `else` branch**), `tools/mtmd/models/deepseekocr.cpp` (**#26727: DeepSeek-OCR SAM convolutions re-expressed via a file-static `conv_2d_f32()` = `ggml_im2col` + `ggml_mul_mat` keeping the im2col in F32**), plus out-of-scope `tools/ui/**`, `ggml/src/ggml-{opencl,sycl}/**`, `ggml/include/ggml-rpc.h`, `src/llama-model-loader.h` | **No project-source change.** All three in-scope edits are behavioural/internal with no API surface moved: `common/common.h` is byte-identical across the range (so `struct common_threadpools` and every signature are unchanged) and no project source references `threadpool`/`ggml_threadpool_params_match`; the `server-models.cpp` edit is pure logging (`RouterModeIntegrationTest` drives the typed `RouterClient` over HTTP, not the log); the deepseekocr helper is `static` and compiles against the unchanged 11-parameter `ggml_im2col` declaration, and neither `mtmd.h` nor `mtmd-helper.h` changed. **Patches:** of the five **core** patched files only `server-models.cpp` changed, ~810 lines below patch `0008`'s single `@@ -215 @@` hunk — no context shift. Patch `0001` also touches 37 files outside that core set, and one of them *did* change here: `tools/perplexity/perplexity.cpp` lost a blank line at 2026, two lines from `0001`'s `@@ -2018,7` hunk — the closest call of the whole walk. It still applies (verified by really applying the set at both endpoints) and is applied-but-not-compiled here (`LLAMA_BUILD_TOOLS` is OFF under FetchContent). **Build wiring:** no `CMakeLists.txt` changed except upstream's own `tests/`, and no file was added/removed/renamed under `tools/server`, `tools/mtmd`, `vendor` or `include`. The three headers that changed anywhere (`ggml-rpc.h`, `ggml-sycl/fwht.hpp`, `src/llama-model-loader.h`) are all outside the project's include dependency graph. | | b10488–b10499 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10499 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10499–b10507 | `common/arg.{h,cpp}` + `common/download.{h,cpp}` + `tools/server/server-models.{h,cpp}` (**#27346 router preset option `dedup-cache-models`** — new `COMMON_ARG_PRESET_DEDUP_CACHE_MODELS` macro, new `common_download_resolve_path()`, new `server_model_meta::hidden` + a skip in the `GET /models` handler), `tools/mtmd/{mtmd.h,mtmd.cpp,mtmd-helper.cpp,CMakeLists.txt}` + **NEW** `tools/mtmd/mtmd-internal.h` (**#27348 `mtmd_bitmap_set_mergeable()`** — Qwen-VL temporal frame merging becomes opt-in: `can_merge_with()` now also requires `mergeable && other.mergeable`, and only `mtmd_helper_video`'s frame reader sets it), `ggml/src/ggml-{webgpu,vulkan,metal}/**`, `gguf-py/**`, `tests/**` | **No project-source change.** Both `common/download.h` and `tools/mtmd/mtmd.h` changes are *purely additive* — no existing declaration moved — and grepping `src/main/cpp` + `src/test/cpp` for `common_download*`, `COMMON_ARG_PRESET*`, `common_preset`, `mtmd_bitmap_*`, `mtmd_input_part`, `mtmd_group_mergeable_bitmaps`, `server_model_meta`, `get_all_meta` returns **zero** hits (the only `server-models` mention is a comment in `native_server.cpp:188`). `server-models.cpp` is compiled into both `jllama` and `jllama_test`, and its new `common_download_resolve_path()` call resolves from the `llama-common` both already link — no CMake change. The new `mtmd-internal.h` is added only to `add_library(mtmd …)`'s **header** list (no new `.cpp`), so it cannot repeat the b10154 `server-mcp.cpp` missing-`target_sources` link failure. Two behaviour notes, neither actionable: the `/models` JSON *shape* is unchanged (hidden entries are omitted, never serialized) and nothing is hidden unless a preset opts in with `dedup-cache-models`, which the project's own code never writes — though `NativeServer` forwards raw llama-server argv verbatim by design, so a *caller* can enable it, and `RouterClient.awaitModelLoaded` then reports a hidden-but-still-loadable model as unknown (recorded in `TODO.md`); and because upstream's server builds bitmaps with `mtmd_helper_bitmap_init_from_buf` without setting `mergeable`, two consecutive same-size images in one request are no longer temporal-merged on Qwen-VL models — upstream's deliberate fix, invisible to SmolVLM (`n_merge_frames == 1`) and to the TTS speaker-reference clip (an audio bitmap, excluded by the pre-existing `!is_audio` guard). **Patch context:** patch `0008`'s `server-models.cpp` hunk keeps the identical offset across the range; patch `0001`'s `common/arg.h` hunk shifts by exactly **+1 line** (the reformatted preset-macro block) with unchanged context and still applies; `server.cpp` and `server-context.{cpp,h}` are untouched. | | b10499–b10507 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10507 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | -| b10507–b10509 | `ggml/include/ggml.h` + `ggml/src/**` + `ggml/src/ggml-metal/**` (**#27120: new `ggml_rope_set_offset()` and its Metal support**), `tools/ui/**` (**#27365: persisted settings read before the API-key probe**) | **No project-source change.** Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or the root `CMakeLists.txt` changed at all in this range, so none of the five patched files moved and no header in the project's include dependency graph was touched. `ggml_rope_set_offset` is purely additive to `ggml.h` (a header CLAUDE.md's review list marks "safe to skip" — the project never calls a `ggml_rope_*` primitive; `grep -rn "ggml_rope" src/main/cpp src/test/cpp` is empty), and `tools/ui` is the Svelte WebUI, which CI rebuilds from the pinned `GIT_TAG` and which therefore needs no per-bump source review. | +| b10507–b10509 | `ggml/include/ggml.h` + `ggml/src/**` + `ggml/src/ggml-metal/**` (**#27120: new `ggml_rope_set_offset()` and its Metal support**), `tools/ui/**` (**#27365: persisted settings read before the API-key probe**) | **No project-source change.** Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or the root `CMakeLists.txt` changed at all in this range, so none of the five **core** patched files (`common/arg.cpp`, `server-context.{cpp,h}`, `server.cpp`, `server-models.cpp`) moved and no header in the project's include dependency graph was touched. `ggml_rope_set_offset` is purely additive to `ggml.h` (a header CLAUDE.md's review list marks "safe to skip" — the project never calls a `ggml_rope_*` primitive; `grep -rn "ggml_rope" src/main/cpp src/test/cpp` is empty), and `tools/ui` is the Svelte WebUI, which CI rebuilds from the pinned `GIT_TAG` and which therefore needs no per-bump source review. | | b10507–b10509 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10509 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. This is the **last** tag at which patch `0007` applies unmodified — see the next row. | -| b10509–b10519 | `tools/server/server-task.{h,cpp}` + `tools/server/server-context.{h,cpp}` + `tools/server/server-queue.{h,cpp}` (**#27376 "server: refactor sleep handling, allow access /metrics during sleep"**), `tools/server/server.cpp` + `tools/server/server-http.cpp` (**#26347: `/models` + `/v1/models` are no longer API-key-exempt**), `common/common.cpp` (**#27337: revert of the b10499 threadpool sharing when `n_threads` differ**), `ggml/include/ggml.h`, `tools/ui/**`, `src/models/granite-swa.cpp` | **Project source change required (two), plus a patch refresh.** (1) #27376 split `server_task_result_metrics` in two: `n_idle_slots` + `slots_data` moved out into a **new `server_task_result_slots`**, produced by a **new `SERVER_TASK_TYPE_SLOT_GET`** task, and `server_task_result_metrics::to_json()` now returns an empty object (`/metrics` renders Prometheus text via the byte-identical `to_metrics()`). Two project impacts, one silent and one hard: `jllama.cpp`'s `handleSlotAction` LIST arm posted `SERVER_TASK_TYPE_METRICS`, which still **compiles** at b10519 but would return `{}` instead of the slot array to `LlamaModel.getMetrics()` — re-pointed at `SERVER_TASK_TYPE_SLOT_GET` (upstream's `/slots` handler builds exactly that task, with no extra fields); and `test_server.cpp` was a **hard compile error** (`make_metrics()` set the removed `n_idle_slots`, `ToJson_ReturnsSlotsArrayVerbatim` set the removed `slots_data`) — the slots assertions moved to `server_task_result_slots` and two tests were added (`ServerTaskResultSlots.ToJson_EmptyByDefault`, `ServerTaskResultMetrics.ToJson_UnusedEmptyObject`), 485 → 487. `to_metrics()` is byte-identical and never read `n_idle_slots`, so every Prometheus assertion survived unchanged. (2) **Patch `0007` stopped applying here** — #26347 deleted the trailing `// public endpoint (no API key check)` comments on the two `/models` route registrations, which sit inside `0007`'s `@@ -258,47 +310,7 @@` removal block (`git apply` → "patch does not apply", `server.cpp:258`); refreshed by dropping that comment from those four lines (2 on the `-` side, 2 on the extracted `llama_server_register_common_routes()` `+` side, where it had also become factually wrong). `0001`/`0002`/`0003`/`0006`/`0008` applied unchanged. No CMake wiring change: no `tools/server` or `vendor` file was added, removed or renamed. **Behavioural note for consumers:** a `NativeServer` started with `--api-key` now returns 401 on `/models` and `/v1/models`; the project's own `RouterModeIntegrationTest` runs without an API key, so CI is unaffected. The new `server_routes` sleep-cache callback registers in the constructor and is inert here — `sleep_idle_seconds` defaults to `-1` (disabled) and nothing in the project sets it. | +| b10509–b10519 | `tools/server/server-task.{h,cpp}` + `tools/server/server-context.{h,cpp}` + `tools/server/server-queue.{h,cpp}` (**#27376 "server: refactor sleep handling, allow access /metrics during sleep"**), `tools/server/server.cpp` + `tools/server/server-http.cpp` (**#26347: `/models` + `/v1/models` are no longer API-key-exempt**), `common/common.cpp` (**#27337: revert of the b10499 threadpool sharing when `n_threads` differ**), `ggml/include/ggml.h`, `tools/ui/**`, `src/models/granite-swa.cpp` | **Project source change required (two), plus a patch refresh.** (1) #27376 split `server_task_result_metrics` in two: `n_idle_slots` + `slots_data` moved out into a **new `server_task_result_slots`**, produced by a **new `SERVER_TASK_TYPE_SLOT_GET`** task, and `server_task_result_metrics::to_json()` now just returns `json{}` — the *default-constructed* value, i.e. JSON **null**, not an empty object (`/metrics` renders Prometheus text via the byte-identical `to_metrics()`). Two project impacts, one silent and one hard: `jllama.cpp`'s `handleSlotAction` LIST arm posted `SERVER_TASK_TYPE_METRICS`, which still **compiles** at b10519 but would return `{}` instead of the slot array to `LlamaModel.getMetrics()` — re-pointed at `SERVER_TASK_TYPE_SLOT_GET` (upstream's `/slots` handler builds exactly that task, with no extra fields); and `test_server.cpp` was a **hard compile error** (`make_metrics()` set the removed `n_idle_slots`, `ToJson_ReturnsSlotsArrayVerbatim` set the removed `slots_data`) — the slots assertions moved to `server_task_result_slots` and two tests were added (`ServerTaskResultSlots.ToJson_EmptyByDefault`, `ServerTaskResultMetrics.ToJson_UnusedAndEmpty`), 485 → 487. `to_metrics()` is byte-identical and never read `n_idle_slots`, so every Prometheus assertion survived unchanged. (2) **Patch `0007` stopped applying here** — #26347 deleted the trailing `// public endpoint (no API key check)` comments on the two `/models` route registrations, which sit inside `0007`'s `@@ -258,47 +310,7 @@` removal block (`git apply` → "patch does not apply", `server.cpp:258`); refreshed by dropping that comment from those four lines (2 on the `-` side, 2 on the extracted `llama_server_register_common_routes()` `+` side, where it had also become factually wrong). `0001`/`0002`/`0003`/`0006`/`0008` applied unchanged. No CMake wiring change: no `tools/server` or `vendor` file was added, removed or renamed. **Behavioural note for consumers:** a `NativeServer` started with `--api-key` now returns 401 on `/models` and `/v1/models`; the project's own `RouterModeIntegrationTest` runs without an API key, so CI is unaffected. The new `server_routes` sleep-cache callback registers in the constructor and is inert here — `sleep_idle_seconds` defaults to `-1` (disabled) and nothing in the project sets it. | | b10509–b10519 | upstream verification (sandbox) | Patch `0007` **refreshed** (see above); all **6** patches then re-verified against a clean b10519 checkout — sequential `git apply --check` + `git apply` in filename order, zero fuzz — and the refreshed `0007` additionally re-checked to apply cleanly at **every** later endpoint of this walk up to b10618, so the refresh is a one-time fix, not a per-chunk one. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics against the patched b10519 tree. Full link + `ctest` deferred to the b10618 target. | | b10519–b10532 | `common/speculative.cpp` (**#27404: avoid binding a reference to a null pointer**), `ggml/**` (backend-split scheduler race #26040, CUDA cuBLAS workspace #26574, Metal FA dequant #27390, Vulkan/OpenCL/Hexagon kernels), `src/llama-graph.cpp` (**#27392: V built as a view of K in `k_iswa build_attn`**), `convert_hf_to_gguf.py` | **No project-source change.** The only in-scope file is `common/speculative.cpp`, and the fix is inside an upstream-compiled TU — `common/speculative.h` (priority 3 on the CLAUDE.md review list) has a zero-line diff across the range, so nothing the project compiles against moved, and `grep -rn "common_speculative" src/main/cpp src/test/cpp` is empty. Nothing under `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt` changed, so no patch context shifted and `llama/CMakeLists.txt`'s `target_sources` list still matches upstream. The remainder is ggml backends and the llama graph builder, both entirely upstream-owned. | | b10519–b10532 | upstream verification (sandbox) | All **6** patches (with the b10519-refreshed `0007`) re-verified against a clean b10532 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | @@ -649,24 +649,24 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10566–b10568 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10568 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10568–b10569 | `src/models/dots3note.cpp` (new) + `src/llama-{model,arch,vocab}.*` + `convert_hf_to_gguf.py` + `gguf-py/**` (**#27060: dots3-note model support**) | **No project-source change.** A new model architecture, entirely inside llama.cpp's own model layer: the new TU is added to upstream's own `src/CMakeLists.txt` (which the project consumes verbatim through `FetchContent_MakeAvailable`, so there is nothing to mirror into `llama/CMakeLists.txt` — unlike the `tools/server/*.cpp` list, which the project enumerates itself). Zero lines changed under `common/`, `tools/server/`, `tools/mtmd/`, `include/` or `vendor/`; no patch context moved. | | b10568–b10569 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10569 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | -| b10569–b10578 | `tools/mtmd/mtmd-helper.{h,cpp}` (**#27520: webp decoded as a single ffmpeg frame; the header change is one comment line**), `common/speculative.cpp` (**#27400: draft-MTP fixed with embeddings — draft params now reset `embedding`/`pooling_type`**), `ggml/**` (**#24575 row-level `concat`; #26431 gpt-oss MoE bias fused into the OpenCL epilogue; #27490 SYCL Q2_K kernels re-landed**), `README.md` badges, `docs/**` | **No project-source change.** `mtmd-helper.h` changed by exactly one comment line, so `mtmd_helper::gen_audio`, `mtmd_helper_gen_audio_inp` and `mtmd_helper_bitmap_init_from_file` — the surface `src/main/cpp/tts_engine.cpp` uses directly, and the reason `mtmd-helper.h` is on the CLAUDE.md priority review list — are untouched. The new webp path is reachable through upstream's own `server-common.cpp` bitmap loader and is a functional gain requiring `ffmpeg`/`ffprobe` on `PATH` **at runtime only** (no new build dependency). One cosmetic upstream wart: the new `is_webp_file()` sits outside the `MTMD_VIDEO` guard while its only caller is inside it, so an `MTMD_VIDEO=OFF` build emits `-Wunused-function`; harmless here — it is compiled in upstream's own `mtmd` target and `LLAMA_FATAL_WARNINGS` defaults OFF and is never set by this project. The speculative fix is behaviour-only: `common/speculative.h` is unchanged and the only callers are upstream's `server-context.cpp`. **Patches:** none of the five patched files changed — zero context shift. | +| b10569–b10578 | `tools/mtmd/mtmd-helper.{h,cpp}` (**#27520: webp decoded as a single ffmpeg frame; the header change is one comment line**), `common/speculative.cpp` (**#27400: draft-MTP fixed with embeddings — draft params now reset `embedding`/`pooling_type`**), `ggml/**` (**#24575 row-level `concat`; #26431 gpt-oss MoE bias fused into the OpenCL epilogue; #27490 SYCL Q2_K kernels re-landed**), `README.md` badges, `docs/**` | **No project-source change.** `mtmd-helper.h` changed by exactly one comment line, so `mtmd_helper::gen_audio`, `mtmd_helper_gen_audio_inp` and `mtmd_helper_bitmap_init_from_file` — the surface `src/main/cpp/tts_engine.cpp` uses directly, and the reason `mtmd-helper.h` is on the CLAUDE.md priority review list — are untouched. The new webp path is reachable through upstream's own `server-common.cpp` bitmap loader and is a functional gain requiring `ffmpeg`/`ffprobe` on `PATH` **at runtime only** (no new build dependency). One cosmetic upstream wart: the new `is_webp_file()` sits outside the `MTMD_VIDEO` guard while its only caller is inside it, so an `MTMD_VIDEO=OFF` build emits `-Wunused-function`; harmless here — it is compiled in upstream's own `mtmd` target and `LLAMA_FATAL_WARNINGS` defaults OFF and is never set by this project. The speculative fix is behaviour-only: `common/speculative.h` is unchanged and the only callers are upstream's `server-context.cpp`. **Patches:** none of the five **core** patched files (`common/arg.cpp`, `server-context.{cpp,h}`, `server.cpp`, `server-models.cpp`) changed — zero context shift. | | b10569–b10578 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10578 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10578–b10584 | `common/fit.{h,cpp}` + `common/common.cpp` + `common/speculative.cpp` + `tools/{fit-params,llama-bench}` (**draft/MTP context is now fitted together with the target model — new public `struct common_fit_extra_model` and a new 7th `const common_fit_extra_model * extra` parameter on `common_fit_params()`; `common_speculative_init_result` pins the draft `cparams.n_ctx = llama_n_ctx(ctx_tgt)`**), `tools/server/server-context.cpp` (**−56 lines: the hand-rolled draft-model VRAM pre-reservation block is deleted, superseded by the fitter**), `common/arg.cpp` (**`--conversation` dropped `LLAMA_EXAMPLE_CLI` from its `set_examples()`**), `tools/mtmd/{CMakeLists.txt,clip*.h,clip.cpp,models/dots3note.cpp,models/models.h,mtmd-audio.{h,cpp},mtmd.cpp}` (**new `dots3note` multimodal model: two `PROJECTOR_TYPE_DOTS3NOTE_{V,A}` values, a vision MoE FFN via a new `clip_graph::build_moe_ffn()`, and `mtmd_audio_preprocessor_dots3note`**) | **No project-source change; no patch refresh.** `common_fit_params()`'s signature break is upstream-only: `common/fit.h` is not in the project's include graph, `grep -rn "common_fit_params\|common_fit_extra_model\|fit_params\|common_get_device_memory_data" src/main/cpp src/test/cpp` returns zero hits, and the two other callers (`tools/fit-params`, `tools/llama-bench`) are not built here (`LLAMA_BUILD_TOOLS` is OFF under FetchContent; only `tools/mtmd` and an explicit `tools/server/*.cpp` list are added back). The `server-context.cpp` deletion sits directly **above** patch `0002`'s hunk, whose anchor lines (blank / `// attach a progress callback` / `{`) are byte-identical, so that hunk only *shifts* (offset −53 → −108) and still applies cleanly. `--conversation` never reached the project's parser (`jllama.cpp` parses with `LLAMA_EXAMPLE_SERVER`, in neither the old nor the new example set) and patch `0001`'s `arg.cpp` hunks are ~650 lines away. All mtmd edits are internal — **`mtmd.h` and `mtmd-helper.h` are unchanged**, so `tts_engine.cpp`'s `mtmd_helper::gen_audio` surface is untouched, and `models/dots3note.cpp` enters through upstream's own `tools/mtmd/CMakeLists.txt`. The only reachable behaviour delta is a more accurate VRAM budget when `--fit` runs together with a draft model. | | b10578–b10584 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10584 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | -| b10584–b10585 | **NEW** `common/json.{h,cpp}` + `common/CMakeLists.txt`, `common/chat.h`, `common/chat-auto-parser.h`, `common/chat-peg-parser.h`, `common/json-schema-to-grammar.h`, `common/peg-parser.h`, `common/arg.cpp`, `common/{chat,download,hf-cache,jinja/*,json-schema-to-grammar,peg-parser}.cpp`, `tools/server/{server-common.h,server-common.cpp,server-chat.h,server-chat.cpp,server-context.h,server-context.cpp,server-task.h,server-task.cpp,server-schema.cpp,server-models.cpp,server-tools.cpp}` (**#27511 "common: add json.h abstraction"**) | **BREAKING — the largest project impact of the whole b10456 → b10618 walk.** Upstream introduces the pimpl wrapper class `common_json` and flips `using json = nlohmann::ordered_json` → `using json = common_json` in `server-common.h`, so **every** project TU that includes an upstream server header changes JSON type. `common_json` is a deliberately closed API: no `get_ref`, no `array_t`, no `type_name()`, a braced list in *value* position does not build an array, `at(key)` no longer implicitly converts (needs `.get()`), `get()` is limited to the specialisations in `common/json.cpp`, and errors are `common_json_error`. Project changes, in the order they were found: **(1)** `utils.hpp`'s `str_to_bytes` used `bytes.get_ref().reserve(...)` — dropped (a pure optimisation; `json::array()` + `push_back` is unchanged). **(2)** `jllama.cpp`'s `jsonSchemaToGrammarBytes` parsed with `nlohmann::ordered_json::parse` and handed the result to `json_schema_to_grammar`, which now takes `const common_json &` — there is no conversion, so this is a hard compile error; switched to `json::parse`. **(3) Two silent, compile-clean regressions**, neither of which a build could catch: `jni_helpers.hpp`'s `require_json_field_impl` declared its parameter `const nlohmann::json &`, and a `common_json` **still binds to it** — through `common_json::operator std::string()` feeding nlohmann's string-constructible converting constructor — so the presence check became a `json::type_error 302` ("type must be string, but is object") thrown out of `handleInfill`, whose two call sites sit *outside* its `try` block; the helper is now a template on the JSON type, with two new `common_json` regression tests. And `common_json_value`'s integral constructor template is `std::is_integral`-gated, which **excludes enums**, so `{"vocab_type", }` bound to the `bool` constructor and serialised as `true`/`false`; `ModelMeta.getVocabType()` reads it with Jackson's `asInt(0)`, so every non-SPM model would have reported vocab type 1. Both emit sites now `static_cast(...)`, with a `CommonJsonEnumTrap` C++ pair pinning the trap and a `LlamaModelTest` assertion that the wire value `isIntegralNumber()`. (Both were confirmed empirically with a standalone probe linked against `common/json.cpp`, not inferred.) **(4)** Seven C++ test literals with a braced array in value position (`test_json_helpers.cpp`'s `extract_embedding_prompt` input, `test_server.cpp`'s `dry_sequence_breakers` + five `samplers` cases) now use `json::array({...})`; the produced JSON is identical. **No JSON output shape changed in the project's own emissions**, so no existing assertion moved — but the same enum trap bit **upstream's own** `/models` handler, which the project compiles and serves; that was found by the post-bump review and is carried as `patches/0010` (see the last row of this file); **no CMake change** is needed (`common/json.cpp` rides the `llama-common` target both `jllama` and `jllama_test` already link, and the `nlohmann_json` link stays for `log_helpers.hpp` / `train_engine.cpp`, which keep their own alias); and **no patch context shifted** — `server.cpp` is untouched in this range, and `0001`/`0002`/`0003`/`0008` keep byte-identical anchors. Test total 487 → 491. | +| b10584–b10585 | **NEW** `common/json.{h,cpp}` + `common/CMakeLists.txt`, `common/chat.h`, `common/chat-auto-parser.h`, `common/chat-peg-parser.h`, `common/json-schema-to-grammar.h`, `common/peg-parser.h`, `common/arg.cpp`, `common/{chat,download,hf-cache,jinja/*,json-schema-to-grammar,peg-parser}.cpp`, `tools/server/{server-common.h,server-common.cpp,server-chat.h,server-chat.cpp,server-context.h,server-context.cpp,server-task.h,server-task.cpp,server-schema.cpp,server-models.cpp,server-tools.cpp}` (**#27511 "common: add json.h abstraction"**) | **BREAKING — the largest project impact of the whole b10456 → b10618 walk.** Upstream introduces the pimpl wrapper class `common_json` and flips `using json = nlohmann::ordered_json` → `using json = common_json` in `server-common.h`, so **every** project TU that includes an upstream server header changes JSON type. `common_json` is a deliberately closed API: no `get_ref`, no `array_t`, no `type_name()`, a braced list in *value* position does not build an array, `at(key)` no longer implicitly converts (needs `.get()`), `get()` is limited to the specialisations in `common/json.cpp`, and errors are `common_json_error`. Project changes, in the order they were found: **(1)** `utils.hpp`'s `str_to_bytes` used `bytes.get_ref().reserve(...)` — dropped (a pure optimisation; `json::array()` + `push_back` is unchanged). **(2)** `jllama.cpp`'s `jsonSchemaToGrammarBytes` parsed with `nlohmann::ordered_json::parse` and handed the result to `json_schema_to_grammar`, which now takes `const common_json &` — there is no conversion, so this is a hard compile error; switched to `json::parse`. **(3) Two silent, compile-clean regressions**, neither of which a build could catch: `jni_helpers.hpp`'s `require_json_field_impl` declared its parameter `const nlohmann::json &`, and a `common_json` **still binds to it** — through `common_json::operator std::string()` feeding nlohmann's string-constructible converting constructor — so the presence check became a `json::type_error 302` ("type must be string, but is object") thrown out of `handleInfill`, whose two call sites sit *outside* its `try` block; the helper is now a template on the JSON type, with two new `common_json` regression tests. And `common_json_value`'s integral constructor template is `std::is_integral`-gated, which **excludes enums**, so `{"vocab_type", }` bound to the `bool` constructor and serialised as `true`/`false`; `ModelMeta.getVocabType()` reads it with Jackson's `asInt(0)`, so every non-SPM model would have reported vocab type 1. Both emit sites now `static_cast(...)`, with a `CommonJsonEnumTrap` C++ pair pinning the trap and a `LlamaModelTest` assertion that the wire value `isIntegralNumber()`. (Both were confirmed empirically with a standalone probe linked against `common/json.cpp`, not inferred.) **(4)** Fourteen C++ test literals across three files now use `json::array({...})`; the produced JSON is identical. Two distinct incompatibilities: a braced array in *value* position (`test_json_helpers.cpp`'s `extract_embedding_prompt` input, `test_server.cpp`'s `dry_sequence_breakers` + five `samplers` cases — seven sites), and direct-initialisation `json{1, 2, 3}` in *constructor* position (`test_utils.cpp`'s seven `JsonArrayChecks` cases), which `common_json` also cannot build because its only initializer-list constructor takes `common_json_item`s, i.e. keyed object entries. **No JSON output shape changed in the project's own emissions**, so no existing assertion moved — but the same enum trap bit **upstream's own** `/models` handler, which the project compiles and serves; that was found by the post-bump review and is carried as `patches/0010` (see the last row of this file); **no CMake change** is needed (`common/json.cpp` rides the `llama-common` target both `jllama` and `jllama_test` already link, and the `nlohmann_json` link stays for `log_helpers.hpp` / `train_engine.cpp`, which keep their own alias); and **no patch context shifted** — `server.cpp` is untouched in this range, and `0001`/`0002`/`0003`/`0008` keep byte-identical anchors. Test total 487 → 491. | | b10584–b10585 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10585 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics **after** the migration above. Full link + `ctest` deferred to the b10618 target. | | b10585–b10590 | `common/json.{cpp,h}` (**#27575 "fix clang lto": the `common_json_value` set/map/unordered_map/vector constructors and every `common_json::get()` flip from explicit *instantiation* to explicit *specialization* — "an explicit instantiation is a weak symbol, dropped by some LTO builds (clang-cl)" — and `operator std::string()` + `value(key, const char *)` move from inline-in-`json.h` to out-of-line in `json.cpp`; the set of usable types is unchanged**), `vendor/sheredom/subprocess.h` + `scripts/sync_vendor.py` (**#27409 upstream resync, +363/−12: new `SUBPROCESS_SPAWN_VIA_FORK` / `SUBPROCESS_ADDCHDIR_IS_POSIX` macros gating a `fork()`+`execve`/`execvpe` launcher with an errno-relay pipe, new `subprocess_pipe_cloexec()` / `subprocess_fds_above_std()` POSIX helpers, and a Windows `STARTUPINFOEX` + `PROC_THREAD_ATTRIBUTE_HANDLE_LIST` handle-inheritance list**), `tools/mtmd/{clip.cpp,clip-graph.h,models/gemma4v.cpp,models/minimax-m3.cpp}` (**#27521: 2D RoPE rewritten from view/rope/`ggml_concat` onto two in-place `ggml_rope_ext` calls + `ggml_rope_set_offset`**) | **No project-source and no project-CMake change.** The `json` work *is* in the project's compile path (the alias became `common_json` at b10585) and the project calls both members that moved out-of-line, but `common/json.cpp` belongs to the `llama-common` target that `jllama` and `jllama_test` already link — nothing to wire, and no `to_json()` shape moved. The project enables no LTO, so the bug being fixed never bit it; the change is robustness for the clang-cl Windows-arm64 job. **`subprocess.h` re-verified against the dropped patch `0009`:** `SUBPROCESS_HAVE_CWD` and its `#elif !SUBPROCESS_HAVE_CWD → posix_error = ENOSYS;` fallback both survive, and `SUBPROCESS_SPAWN_VIA_FORK` is 0 on glibc/bionic/macOS/Windows (only `_AIX`/`__OpenBSD__`/old NetBSD set it), so manylinux2014 (glibc 2.17) still resolves exactly as before and the new `execvpe` declaration is preprocessed away everywhere the project builds — **`0009` stays dropped.** The new always-compiled POSIX helpers need only `pipe2`/`O_CLOEXEC`/`F_DUPFD` (glibc 2.7+/bionic) and the header already includes ``/``; the Windows attribute-list path adds no new link input. mtmd is internal-only here: `build_rope_2d`'s signature is untouched (the `clip-graph.h` hunk is comment-only) and `mtmd.h`/`mtmd-helper.h`/`clip.h` are byte-identical, so `tts_engine.cpp`'s surface is unaffected. **Patches:** zero changes under `tools/server/` and none to `common/arg.{h,cpp}` — byte-identical context for all six. | | b10585–b10590 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10590 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | -| b10590–b10593 | `include/llama.h` (**one comment line: a `[TAG_LLAMA_SEQ_ID_NEG]` marker added to `llama_memory_seq_rm`'s doc block**), `src/llama-*.cpp` (**#26756 DeepseekV4 multi-seq rollback fix; #27574 tensor-parallel meta tensor-split state propagation**), `tools/ui/**` (**#27263: tabbed chat-conversation navigation**) | **No project-source change.** `include/llama.h` is priority 7 on the CLAUDE.md review list, so its diff was read in full: it is a single comment line inside an existing doc block — no signature, no enum value, no struct field moved. The rest is llama.cpp's internal KV/tensor-split implementation and the Svelte WebUI. Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `vendor/` or any `CMakeLists.txt` changed, so no patch context moved. | +| b10590–b10593 | `include/llama.h` (**one comment line: a `[TAG_LLAMA_SEQ_ID_NEG]` marker added to `llama_memory_seq_rm`'s doc block**), `src/llama-*.cpp` (**#26756 DeepseekV4 multi-seq rollback fix; #27574 tensor-parallel meta tensor-split state propagation**), `tools/ui/**` (**#27263: tabbed chat-conversation navigation**) | **No project-source change.** `include/llama.h` is priority 7 on the CLAUDE.md review list, so its diff was read in full: it is a single comment line inside an existing doc block — no signature, no enum value, no struct field moved. The rest is llama.cpp's internal KV/tensor-split implementation and the Svelte WebUI. Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `vendor/` or any `CMakeLists.txt` changed. One file patch `0001` touches *did* change, outside those directories: `tests/test-recurrent-state-rollback.cpp` gained 172 lines above `0001`'s hunk, shifting it ~175 lines — it still applies by offset (verified by really applying the set at this endpoint), and it is applied-but-not-compiled here (`LLAMA_BUILD_TESTS` is OFF under FetchContent). | | b10590–b10593 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10593 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10593–b10599 | `tools/mtmd/{clip-model.h,clip.cpp,mtmd-image.cpp}` (**#27594 "mtmd: use pillow-accurate algo, correct resize_algo for all models": `enum resize_algo` drops `RESIZE_ALGO_BICUBIC_PILLOW` (plain `BICUBIC` now *means* the Pillow path), the four private `img_tool` statics `resize_{bilinear,bicubic,bicubic_pillow,lanczos_pillow}` and both dispatch switches collapse into one `resize_pillow(..., resize_algo algo)` (−246 net), and ~14 projector types are re-assigned**), `tools/server/server-context.cpp` (**#27600: private `slots_n_diff` member + `LLAMA_SERVER_SLOTS_N_DIFF` getenv; the hardcoded prompt-mismatch debug window `n_past−4..+6` becomes `n_past−slots_n_diff..+slots_n_diff+2`**), `common/common.cpp` (**#26692: `common_params_print_info`'s device-enumeration loop gated on `print_devices && verbosity >= LOG_LEVEL_TRACE`**), `tools/CMakeLists.txt` (**#27548: `add_subdirectory(parser)` dropped — the parser tool moved under `tests/`**) | **No project-source, no project-CMake and no patch change.** The mtmd resize rework is entirely mtmd-internal: `resize_algo`/`clip_hparams` live in `tools/mtmd/clip-model.h`, which the project never includes, and all four deleted functions were `private:` statics inside `struct img_tool`; the public `mtmd.h`/`mtmd-helper.h` are unchanged in this range. The only visible effect is that `MultimodalIntegrationTest`'s SmolVLM preprocessing becomes Pillow-exact — pixel-level, no API or JSON shape. `server-context.cpp`'s new fields are `private:` in `server_context_impl` and **`server-context.h` is unchanged**, so no `to_json()` shape moved and no C++ test assertion is affected; patch `0002`'s and `0003`'s anchors are byte-identical and resolve with a pure offset. `common_params_print_info` keeps its `bool print_devices = true` default, so `jllama.cpp`'s one-argument call still compiles; the behaviour delta is that at the default log threshold libjllama no longer prints `device_info:` (nothing in the project reads it) and no longer triggers early CUDA primary-context creation from that call site. The `tools/CMakeLists.txt` edit sits inside the `LLAMA_BUILD_TOOLS`-gated branch the project never enters. | | b10593–b10599 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10599 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | -| b10599–b10612 | `tools/mtmd/mtmd-helper.cpp` (**#27596 "video: fix moov atom at the end of file": `subprocess_handle` splits `alive` into `created`+`alive`, `stop()` gains `subprocess_join()` to reap zombies plus a Windows `CloseHandle(proc.hStdInput)`, `start_feeder()` gains `pthread_sigmask(SIG_BLOCK, {SIGPIPE})` and an `#ifdef F_SETNOSIGPIPE fcntl(...)`, and the ffmpeg argv gains `-read_ahead_limit -1`; new `#include //` under `#ifdef MTMD_VIDEO`/`#ifndef _WIN32`**), `ggml/**` (**#27644 `ggml_clamp` fix; #27608 shorter virtual-device naming; #27545 WebGPU include order**), `src/**` (**#26534 MTP in GLM-4.5-Air; #26490 Deepseek 4 `-sm tensor`; #27513 mamba2 GEMM dispatch**), `convert_hf_to_gguf.py`, `tools/ui/**` | **No project-source change.** `mtmd-helper.cpp` **is** compiled into `libjllama` (the project `add_subdirectory`s `tools/mtmd` and never sets `LLAMA_SUBPROCESS`, so `MTMD_VIDEO` is ON on every classifier including Android), so its three new build-surface dependencies were each checked against the pinned tree rather than assumed: `subprocess_join` is declared in `vendor/sheredom/subprocess.h`, `struct subprocess_s` carries `void *hStdInput` (so the Windows `CloseHandle` compiles), `pthread_sigmask` resolves through the pre-existing `target_link_libraries(mtmd PRIVATE Threads::Threads …)`, and `F_SETNOSIGPIPE` is `#ifdef`-guarded off on Linux/Android. `mtmd.h` and `mtmd-helper.h` are unchanged in this range, so `tts_engine.cpp`'s `mtmd_helper::gen_audio` surface is untouched. Nothing under `common/`, `tools/server/`, `include/` or any project-consumed `CMakeLists.txt` changed — no patch context moved. | +| b10599–b10612 | `tools/mtmd/mtmd-helper.cpp` (**#27596 "video: fix moov atom at the end of file": `subprocess_handle` splits `alive` into `created`+`alive`, `stop()` gains `subprocess_join()` to reap zombies plus a Windows `CloseHandle(proc.hStdInput)`, `start_feeder()` gains `pthread_sigmask(SIG_BLOCK, {SIGPIPE})` and an `#ifdef F_SETNOSIGPIPE fcntl(...)`, and the ffmpeg argv gains `-read_ahead_limit -1`; new `#include //` under `#ifdef MTMD_VIDEO`/`#ifndef _WIN32`**), `ggml/**` (**#27644 `ggml_clamp` fix; #27608 shorter virtual-device naming; #27545 WebGPU include order**), `src/**` (**#26534 MTP in GLM-4.5-Air; #26490 Deepseek 4 `-sm tensor`; #27513 mamba2 GEMM dispatch**), `convert_hf_to_gguf.py`, `tools/ui/**` | **No project-source change.** `mtmd-helper.cpp` **is** compiled into `libjllama` (the project `add_subdirectory`s `tools/mtmd` and upstream defaults `LLAMA_SUBPROCESS` **OFF** when `CMAKE_SYSTEM_NAME`/`ANDROID` says Android, which force-disables `MTMD_VIDEO` — but the dockcross cross-clang sets neither variable, as the Android section of `CLAUDE.md` documents, so `LLAMA_SUBPROCESS`/`MTMD_VIDEO` stay ON on this project's Android build too; a switch to the NDK toolchain would flip that silently), so its three new build-surface dependencies were each checked against the pinned tree rather than assumed: `subprocess_join` is declared in `vendor/sheredom/subprocess.h`, `struct subprocess_s` carries `void *hStdInput` (so the Windows `CloseHandle` compiles), `pthread_sigmask` resolves through the pre-existing `target_link_libraries(mtmd PRIVATE Threads::Threads …)`, and `F_SETNOSIGPIPE` is `#ifdef`-guarded off on Linux/Android. `mtmd.h` and `mtmd-helper.h` are unchanged in this range, so `tts_engine.cpp`'s `mtmd_helper::gen_audio` surface is untouched. Nothing under `common/`, `tools/server/`, `include/` or any project-consumed `CMakeLists.txt` changed — no patch context moved. | | b10599–b10612 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10612 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10612–b10614 | `ggml/src/ggml-metal/**` + `ggml/CMakeLists.txt` (**#26561: per-op Metal source split + parallel compile — a large mechanical file split, hence the ~1 MiB raw diff**), `gguf-py/gguf/metadata.py` (**#27659: `repetition_penalty` read from `generation_config.json` at conversion time**) | **No project-source change.** Zero lines changed under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any project-consumed `CMakeLists.txt`, so no patch context moved and no header in the project's include graph was touched. The Metal split affects only the `Mac/aarch64` classifier's compiled sources (upstream owns that list entirely); the `generation_config.json` change is a GGUF-conversion-time default with no C++ API surface. | | b10612–b10614 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10614 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10614–b10615 | `ggml/src/ggml-metal/**` + `tools/CMakeLists.txt` (**#26570: per-device tuned `(Q, NE)` for the flash-attention vec kernels; a Metal-only `tuning` tool is added under an `if (GGML_METAL)` guard**) | **No project-source change.** The `tools/CMakeLists.txt` addition sits inside the `LLAMA_BUILD_TOOLS`-gated branch the project never enters (`LLAMA_BUILD_TOOLS` defaults to `LLAMA_STANDALONE`, i.e. OFF under FetchContent, and the project forces it OFF on Android), and is additionally Metal-only. Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/` or `vendor/` changed — no patch context moved, no header in the project's include graph touched. | | b10614–b10615 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10615 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10615–b10618 | `src/llama-grammar.cpp` (**#27591: `parse_char()` accepts `case '-':`, so a `\-` escape produced by `gbnf_escape_char_class()` inside a character class no longer throws**), `ggml/src/ggml-sycl/**` (**#27660: `tq2_0` marked unsupported**), `ggml/src/ggml-webgpu/wgsl-shaders/argsort.wgsl` (**#27538: infinity handling in ARGSORT / TOP_K**) | **No project-source change (final step).** Zero lines changed under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt`, so no patch context moved. The grammar fix is strictly *widening* — an input that previously threw now parses — and reaches the project through both `LlamaModel.jsonSchemaToGrammar` and every grammar-constrained completion, so it is a **positive** for the agentic tool-calling target; no test pinned the old throw (`grep -rn 'gbnf_escape_char_class\|parse_char' src/main/cpp src/test/cpp` is empty). The SYCL and WebGPU edits only affect the `sycl-*` classifiers' compiled sources and a backend the project does not build. | -| b10456–b10618 | upstream verification (sandbox, final target) | **Full local verification, re-run from scratch on the shipped tree.** Fresh `cmake -B build` with the real `FetchContent` path (no source-dir override), so the fail-loud `PATCH_COMMAND` ran for real: it resolved `ggml commit eb25b7263` / `LLAMA_BUILD_NUMBER = 10618` and applied all **7** patches — verified by grepping the fetched tree for each patch's marker (`common_params_parse_main`, `params_base.load_progress_callback == nullptr`, `get_slot_prompt_similarity`, `g_llama_server_embedded`, `llama_server_attach` + `llama_server_register_common_routes`, `LLAMA_SERVER_WORKER_CMD`, `(int) meta.model_vocab_type`). The **7**, not 6, matters: the review-driven `0010` was added after the first verification pass and is the *third* patch to touch `server-context.cpp` (after `0002`/`0003`), a TU compiled into both `jllama` and `jllama_test`, so the whole configure + build + `ctest` cycle was re-run on the final tree rather than assumed to carry over (**491/491**, `mvn test` unchanged at 1405 run / 0 failures / 17 model-gated skips). The patch was subsequently renamed `0009` → `0010` to free the burned number; that is a filename-only change with the same sort position, re-checked by applying the whole set to a clean b10618 checkout. Then a full `cmake --build --config Release` (jllama + jllama_test both link — the `jllama_test` link is what proves `common_json::get()` resolves, since upstream documents un-specialised types as a *link*-time failure) and `ctest`: **491/491 C++ tests pass**. On the Java side: `mvn test-compile` clean, `mvn test` **1405 run / 0 failures / 17 skipped** (only the model-gated integration tests, no GGUF in the sandbox), and `NativeLibraryLoadSmokeTest` green — including `nativeBuildInfoMatchesPinnedVersionConstant`, the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10618") matches the `build-info` compiled into the freshly linked `libjllama.so`. `mvn spotless:apply` produced no changes beyond the edits themselves, the pinned **clang-format 22.1.8** reports the whole C++ tree clean, and `mvn clean javadoc:jar` is `BUILD SUCCESS`. **Per-step verification for all 25 chunks:** every intermediate tag additionally had all six patches applied by the applier's own commands and all ten project + C++-test translation units compiled against it, so no commit in this walk is a state that fails to build. | +| b10456–b10618 | upstream verification (sandbox, final target) | **Full local verification, re-run from scratch on the shipped tree.** Fresh `cmake -B build` with the real `FetchContent` path (no source-dir override), so the fail-loud `PATCH_COMMAND` ran for real: it resolved `ggml commit eb25b7263` / `LLAMA_BUILD_NUMBER = 10618` and applied all **7** patches — verified by grepping the fetched tree for each patch's marker (`common_params_parse_main`, `params_base.load_progress_callback == nullptr`, `get_slot_prompt_similarity`, `g_llama_server_embedded`, `llama_server_attach` + `llama_server_register_common_routes`, `LLAMA_SERVER_WORKER_CMD`, `(int) meta.model_vocab_type`). The **7**, not 6, matters: the review-driven `0010` was added after the first verification pass and is the *third* patch to touch `server-context.cpp` (after `0002`/`0003`), a TU compiled into both `jllama` and `jllama_test`, so the whole configure + build + `ctest` cycle was re-run on the final tree rather than assumed to carry over (**491/491**, `mvn test` unchanged at 1405 run / 0 failures / 17 model-gated skips). The patch was subsequently renamed `0009` → `0010` to free the burned number; that is a filename-only change with the same sort position, re-checked by applying the whole set to a clean b10618 checkout. Then a full `cmake --build --config Release` (jllama + jllama_test both link — the `jllama_test` link is what proves `common_json::get()` resolves, since upstream documents un-specialised types as a *link*-time failure) and `ctest`: **491/491 C++ tests pass**. On the Java side: `mvn test-compile` clean, `mvn test` **1405 run / 0 failures / 17 skipped** (only the model-gated integration tests, no GGUF in the sandbox), and `NativeLibraryLoadSmokeTest` green — including `nativeBuildInfoMatchesPinnedVersionConstant`, the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10618") matches the `build-info` compiled into the freshly linked `libjllama.so`. `mvn spotless:apply` produced no changes beyond the edits themselves, the pinned **clang-format 22.1.8** reports the whole C++ tree clean, and `mvn clean javadoc:jar` is `BUILD SUCCESS`. **Per-step verification for all 25 chunks:** every intermediate tag additionally had all six patches applied by the applier's own commands and all ten project + C++-test translation units compiled against it, so no commit in this walk is a state that fails to build. **Two notes on how to read the per-chunk rows above.** (1) They say "all **6** patches" because that was the set during the walk; the 7th (`0010`) came out of the post-bump review and exists only from this final commit on. (2) Their prose reasoning about patch risk scans the directories that matter for the project's *compile* surface (`common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/`, `CMakeLists.txt`) and refers to "the five core patched files" — but the patch set actually spans **40** distinct files, because `0001` flips ~34 standalone `main()` call sites under `tools/*`, `examples/*` and `tests/*`. Two ranges did touch one of those and the prose does not say so on its own (b10488–b10499 → `tools/perplexity/perplexity.cpp`, b10590–b10593 → `tests/test-recurrent-state-rollback.cpp`; both now corrected in place). The *verdicts* were never derived from that directory scan: each chunk's patch verdict comes from really running `git apply` for the whole set against a clean checkout of that tag, which covers all 40 files. The reusable check is `git diff --name-only -- $(grep -h '^+++ b/' llama/patches/*.patch | sed 's|^+++ b/||' | sort -u)`. | | b10456–b10618 | `tools/server/server-context.cpp` — **new local patch `0010-server-cast-vocab-type-for-common-json.patch`** | **Upstream regression found by the post-bump review, fixed downstream.** `get_res_model_info()` builds the `GET /models` + `GET /v1/models` payload and emits `{"vocab_type", meta.model_vocab_type}`, an **unscoped enum**. That was correct while the `json` alias was `nlohmann::ordered_json` (nlohmann serialises an enum as an integer), but `common_json_value`'s integral constructor template is `std::is_integral`-gated, which excludes enums — so from **b10585** the value binds to `common_json_value(bool)` and goes on the wire as `true`/`false`. Upstream regressed it silently in #27511 when they flipped the alias; the same trap hit the project's own two `"vocab_type"` emit sites in `jllama.cpp` (fixed in the b10584–b10585 step). It **ships**: `server-context.cpp` is compiled into `libjllama` and both routes are served by `NativeServer`, the default fat-jar `Main-Class`, in full and in attach mode. The new patch casts the value to `int` at the emit site — one line, upstream-submittable (not yet filed), applied after `0002`/`0003` (same file). Numbered `0010`, not `0009`: that number belongs to the subprocess.h patch dropped at b10280 and is documented as such under the patches table in `CLAUDE.md`, so reusing it would make that note read as if it described this patch. A mechanical sweep confirmed it is the **only** such site in the upstream server sources the project compiles: `server_context_meta` has exactly two enum-typed members, and the other (`pooling_type`) is only ever compared, never serialised. | diff --git a/docs/upgrade/llama-cpp-version-bump.md b/docs/upgrade/llama-cpp-version-bump.md index 5c89f781e..6babecedd 100644 --- a/docs/upgrade/llama-cpp-version-bump.md +++ b/docs/upgrade/llama-cpp-version-bump.md @@ -110,7 +110,7 @@ Once you have the `b -> b` step, apply it exactly as Concretely: 1. **Edit the pin — four files:** - - `llama/CMakeLists.txt` — the `GIT_TAG b` line. (It is the only tag in this file; the + - `llama/CMakeLists.txt` — the `GIT_TAG b` line. (It is the only `b` tag in this file — the other two `GIT_TAG` lines pin nlohmann/json `v3.12.0` and GoogleTest `v1.17.0` and must NOT move with a llama.cpp bump. The `-DLLAMA_TAG=b` that once fed the build-time TTS extraction was removed with the Qwen3-TTS rework, and the WebUI auto-follows `GIT_TAG` in CI.) - `README.md` — the llama.cpp badge and link (version appears twice). diff --git a/llama/cmake/apply-llama-patches.cmake b/llama/cmake/apply-llama-patches.cmake index 6c8c1293e..260846d0e 100644 --- a/llama/cmake/apply-llama-patches.cmake +++ b/llama/cmake/apply-llama-patches.cmake @@ -12,8 +12,12 @@ # Windows (the dockcross/native/MSVC jobs all call the same code path). # * Every `patches/*.patch` and `patches/*.diff` is applied, sorted by filename (so a numeric # prefix like 0001-, 0002- defines a deterministic order). -# * Idempotent: `git apply --reverse --check` detects an already-applied patch and skips it, so -# a CMake reconfigure over an already-patched source tree does not fail. +# * Idempotent ONLY while no two patches touch the same file: `git apply --reverse --check` +# detects an already-applied patch and skips it, so a CMake reconfigure over an already-patched +# source tree normally does not fail. It DOES fail when a later patch rewrites a region an +# earlier one also patched (today: 0006/0007 vs 0001 in tools/server/server.cpp) — the earlier +# patch's reverse-check then no longer matches and the forward apply aborts the configure with +# a misleading "does not apply cleanly". Configure into a fresh build dir; see TODO.md. # * Fail-loud: a patch that no longer applies (e.g. after a llama.cpp version bump shifts the # context) aborts the configure with a clear message, so a stale patch can never be silently # dropped from a release build. From 40550b48c0dc13bd73909227a3732aa8a77495cc Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 14:21:57 +0000 Subject: [PATCH 30/76] fix: make the llama.cpp patch applier idempotent via a stamp A reconfigure of an existing build directory always aborted with "apply-llama-patches: 0001-... does not apply cleanly -- a llama.cpp version bump probably shifted the patched code", which is not what happened. The applier decided "already applied?" with a per-patch `git apply --reverse --check`, and `--check` never mutates the tree: 0006/0007 rewrite the region of tools/server/server.cpp that 0001 also patches, so reversing 0001 against the fully patched tree does not match, the applier fell through to a forward apply, and that failed too. Reversing the scan order does not help for the same reason -- nothing is ever undone between checks. The decision now comes from a stamp file in the fetched source tree (.jllama-patches-applied: the checked-out llama.cpp commit plus each patch's SHA-256) combined with git's clean/dirty state, which is the part that makes it safe: clean tree -> nothing applied yet (fresh fetch, or a re-checkout after a GIT_TAG change), apply forward and write the stamp; dirty tree -> already patched: no-op when the stamp matches this exact commit and patch set, otherwise abort saying to configure into a fresh build directory. Deriving from git state rather than the stamp alone is what stops the dangerous case: a stamp left behind after FetchContent re-checks-out a new tag would otherwise skip patching and silently ship an unpatched build. A source tree supplied via -DFETCHCONTENT_SOURCE_DIR_LLAMA.CPP that is not a git work tree has neither oracle and keeps the old per-patch path. Verified against the real tree (the previously fatal `cmake -B build` reconfigure now succeeds; a fresh configure applies all 7 patches to a clean clone, writes the stamp, and reconfigures as a no-op) and against a two-patches-one-file fixture that reproduces the old failure with the previous applier and passes with this one, including the stale-stamp-on-a-clean-tree case. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- llama/cmake/apply-llama-patches.cmake | 142 ++++++++++++++++++++++---- 1 file changed, 122 insertions(+), 20 deletions(-) diff --git a/llama/cmake/apply-llama-patches.cmake b/llama/cmake/apply-llama-patches.cmake index 260846d0e..ddc92cb2b 100644 --- a/llama/cmake/apply-llama-patches.cmake +++ b/llama/cmake/apply-llama-patches.cmake @@ -12,15 +12,24 @@ # Windows (the dockcross/native/MSVC jobs all call the same code path). # * Every `patches/*.patch` and `patches/*.diff` is applied, sorted by filename (so a numeric # prefix like 0001-, 0002- defines a deterministic order). -# * Idempotent ONLY while no two patches touch the same file: `git apply --reverse --check` -# detects an already-applied patch and skips it, so a CMake reconfigure over an already-patched -# source tree normally does not fail. It DOES fail when a later patch rewrites a region an -# earlier one also patched (today: 0006/0007 vs 0001 in tools/server/server.cpp) — the earlier -# patch's reverse-check then no longer matches and the forward apply aborts the configure with -# a misleading "does not apply cleanly". Configure into a fresh build dir; see TODO.md. +# * Idempotent, via a stamp file rather than per-patch probing. The stamp +# (`${LLAMA_SRC}/.jllama-patches-applied`) records the checked-out llama.cpp commit plus the +# SHA-256 of every patch, and the decision is driven by whether the source tree is pristine: +# - clean tree -> nothing is applied yet (a fresh fetch, or a re-checkout after a version +# bump), so apply all patches forward and write the stamp; +# - dirty tree -> already patched; skip when the stamp matches this exact commit + patch +# set, and fail loudly when it does not. +# A per-patch `git apply --reverse --check` cannot do this: `--check` never mutates the tree, +# so an earlier patch whose region a later patch rewrote (today 0001 vs 0006/0007 in +# tools/server/server.cpp) always reverse-checks as "not applied" and the forward re-apply +# then aborts an otherwise harmless reconfigure. The stamp is state, but it is state derived +# from — and invalidated by — both inputs that matter. # * Fail-loud: a patch that no longer applies (e.g. after a llama.cpp version bump shifts the # context) aborts the configure with a clear message, so a stale patch can never be silently # dropped from a release build. +# * A source tree that is not a git work tree (e.g. supplied via +# `-DFETCHCONTENT_SOURCE_DIR_LLAMA.CPP=`) has no clean/dirty oracle and no HEAD, so it +# falls back to the legacy per-patch reverse-check path, with the caveat described above. # # Invoked as: # cmake -DPATCH_DIR=/patches -DLLAMA_SRC= -P cmake/apply-llama-patches.cmake @@ -42,20 +51,12 @@ if(NOT patch_files) return() endif() -foreach(patch IN LISTS patch_files) - get_filename_component(patch_name "${patch}" NAME) - - # Already applied? A successful reverse-apply check means the change is present already. - execute_process( - COMMAND "${GIT_EXECUTABLE}" -C "${LLAMA_SRC}" apply --reverse --check "${patch}" - RESULT_VARIABLE reverse_rc - OUTPUT_QUIET ERROR_QUIET) - if(reverse_rc EQUAL 0) - message(STATUS "apply-llama-patches: ${patch_name} already applied — skipping") - continue() - endif() +set(STAMP_NAME ".jllama-patches-applied") +set(stamp_file "${LLAMA_SRC}/${STAMP_NAME}") - # Not applied yet — confirm it applies cleanly before touching the tree. +# Applies one patch, aborting the configure when it no longer fits the source tree. +function(apply_one_patch patch) + get_filename_component(patch_name "${patch}" NAME) execute_process( COMMAND "${GIT_EXECUTABLE}" -C "${LLAMA_SRC}" apply --check "${patch}" RESULT_VARIABLE check_rc @@ -66,7 +67,6 @@ foreach(patch IN LISTS patch_files) " A llama.cpp version bump probably shifted the patched code — refresh the patch " "against the new source and recommit it.") endif() - execute_process( COMMAND "${GIT_EXECUTABLE}" -C "${LLAMA_SRC}" apply "${patch}" RESULT_VARIABLE apply_rc) @@ -74,4 +74,106 @@ foreach(patch IN LISTS patch_files) message(FATAL_ERROR "apply-llama-patches: failed to apply ${patch_name}") endif() message(STATUS "apply-llama-patches: applied ${patch_name}") +endfunction() + +# --------------------------------------------------------------------------- +# Is the source tree a git work tree? Without one there is no HEAD to pin the +# stamp to and no clean/dirty oracle, so fall back to the legacy behaviour. +# --------------------------------------------------------------------------- +execute_process( + COMMAND "${GIT_EXECUTABLE}" -C "${LLAMA_SRC}" rev-parse HEAD + RESULT_VARIABLE head_rc + OUTPUT_VARIABLE llama_head + OUTPUT_STRIP_TRAILING_WHITESPACE + ERROR_QUIET) + +if(NOT head_rc EQUAL 0) + message(STATUS "apply-llama-patches: ${LLAMA_SRC} is not a git work tree — " + "using per-patch detection (a reconfigure over a patched tree may fail)") + foreach(patch IN LISTS patch_files) + get_filename_component(patch_name "${patch}" NAME) + execute_process( + COMMAND "${GIT_EXECUTABLE}" -C "${LLAMA_SRC}" apply --reverse --check "${patch}" + RESULT_VARIABLE reverse_rc + OUTPUT_QUIET ERROR_QUIET) + if(reverse_rc EQUAL 0) + message(STATUS "apply-llama-patches: ${patch_name} already applied — skipping") + continue() + endif() + apply_one_patch("${patch}") + endforeach() + return() +endif() + +# --------------------------------------------------------------------------- +# Build the manifest: the checked-out commit plus every patch's content hash. +# Any llama.cpp version bump changes HEAD; any patch edit changes a hash. +# --------------------------------------------------------------------------- +set(manifest "head ${llama_head}\n") +foreach(patch IN LISTS patch_files) + get_filename_component(patch_name "${patch}" NAME) + file(SHA256 "${patch}" patch_hash) + string(APPEND manifest "${patch_name} ${patch_hash}\n") endforeach() + +# --------------------------------------------------------------------------- +# Clean tree => nothing applied yet. Untracked files count as dirty (a future +# patch may add a file), except the stamp itself, which we write ourselves. +# --------------------------------------------------------------------------- +execute_process( + COMMAND "${GIT_EXECUTABLE}" -C "${LLAMA_SRC}" status --porcelain + RESULT_VARIABLE status_rc + OUTPUT_VARIABLE status_out + OUTPUT_STRIP_TRAILING_WHITESPACE + ERROR_QUIET) +if(NOT status_rc EQUAL 0) + message(FATAL_ERROR "apply-llama-patches: 'git status' failed in ${LLAMA_SRC}") +endif() + +set(tree_is_dirty FALSE) +if(NOT status_out STREQUAL "") + string(REPLACE "\n" ";" status_lines "${status_out}") + foreach(line IN LISTS status_lines) + string(STRIP "${line}" line) + if(line STREQUAL "" OR line MATCHES "${STAMP_NAME}$") + continue() + endif() + set(tree_is_dirty TRUE) + break() + endforeach() +endif() + +if(NOT tree_is_dirty) + foreach(patch IN LISTS patch_files) + apply_one_patch("${patch}") + endforeach() + file(WRITE "${stamp_file}" "${manifest}") + return() +endif() + +# --------------------------------------------------------------------------- +# Dirty tree: already patched. Only a stamp matching this exact commit + patch +# set proves the modifications are ours and complete. +# --------------------------------------------------------------------------- +set(stamp_matches FALSE) +if(EXISTS "${stamp_file}") + file(READ "${stamp_file}" stamp_content) + if(stamp_content STREQUAL manifest) + set(stamp_matches TRUE) + endif() +endif() + +if(stamp_matches) + list(LENGTH patch_files patch_count) + message(STATUS "apply-llama-patches: ${patch_count} patch(es) already applied — skipping") + return() +endif() + +message(FATAL_ERROR + "apply-llama-patches: ${LLAMA_SRC} has local modifications that do not match the current " + "patch set.\n" + " Patches cannot be applied on top of an already-patched tree, and the previous state is " + "unknown (the tree was patched with a different patch set or llama.cpp commit, or edited by " + "hand).\n" + " Configure into a FRESH build directory so FetchContent re-checks-out a pristine " + "llama.cpp, then build again.") From 93d0679dc84ae04bdc8367765f0b43ef732db72b Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 14:22:14 +0000 Subject: [PATCH 31/76] test: stop the model-gated Java tests self-skipping in CI Surefire's working directory defaults to the module basedir, so under `mvn -f llama/pom.xml test` it is /llama -- while CI restores the shared GGUF cache to /models and passes every model path as a bare `models/`. Those resolved to /llama/models/, which does not exist, so every model-gated class aborted in its @BeforeAll Assumptions.assumeTrue(file.exists()) and reported as skipped while the job still went green. The whole model-backed suite was inert on every test-java-* job, which is why stale assertions written against payload shapes upstream had already dropped never failed there. TestConstants.resolveModelPath / resolveModelProperty accept either layout -- module-relative first, then the reactor root -- so CI and a developer who put models under llama/models/ both work and no workflow change is needed. An unresolvable path is returned unchanged so the existing "model missing" skip messages still name what was looked for. Every path constant is routed through it, as is every -Dnet.ladenthin.llama.* fixture property; LlamaModelTest's two hardcoded `new File("models/codellama-7b.Q2_K.gguf")` literals now use the constant like every other class already did. llama-langchain4j had the identical defect (same Surefire default, same bare `models/` properties) and gets the same resolver as TestModelPaths -- duplicated rather than shared because test classes are not published between modules. TestConstantsTest pins the resolver (absolute/relative/parent-relative/missing, property lookup with and without a default) and the wiring itself: each constant must equal its own resolved literal, so an edit that drops the wrapper fails the test instead of silently re-muting the suite. Verified end to end: with a placeholder file at /models/, LlamaModelTest now reports Skipped: 0 and actually reaches the native load (failing only because the placeholder is not a real GGUF) where it previously reported Skipped: 1 no matter what was in that directory. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- .../JllamaChatModelIntegrationTest.java | 6 +- .../JllamaEmbeddingModelIntegrationTest.java | 6 +- .../JllamaScoringModelIntegrationTest.java | 6 +- .../JllamaToolCallingIntegrationTest.java | 9 +- .../llama/langchain4j/TestModelPaths.java | 59 +++++++++++ .../llama/AudioInputIntegrationTest.java | 7 +- .../ladenthin/llama/LlamaEmbeddingsTest.java | 2 +- .../llama/MultimodalIntegrationTest.java | 7 +- .../net/ladenthin/llama/TestConstants.java | 76 ++++++++++++-- .../ladenthin/llama/TestConstantsTest.java | 99 +++++++++++++++++++ .../llama/ToolCallingIntegrationTest.java | 4 +- .../ladenthin/llama/TtsIntegrationTest.java | 4 +- ...penAiServerToolCallingIntegrationTest.java | 4 +- 13 files changed, 253 insertions(+), 36 deletions(-) create mode 100644 llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/TestModelPaths.java create mode 100644 llama/src/test/java/net/ladenthin/llama/TestConstantsTest.java diff --git a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java index ea4b18ece..1d840ca19 100644 --- a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java +++ b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java @@ -14,7 +14,6 @@ import dev.langchain4j.model.chat.response.StreamingChatResponseHandler; import java.nio.file.Files; import java.nio.file.Path; -import java.nio.file.Paths; import java.util.concurrent.CompletableFuture; import java.util.concurrent.TimeUnit; import net.ladenthin.llama.LlamaModel; @@ -30,9 +29,8 @@ class JllamaChatModelIntegrationTest { private static Path modelPath() { - String path = System.getProperty("net.ladenthin.llama.model.path"); - Assumptions.assumeTrue(path != null && !path.isEmpty(), "model path property not set"); - Path resolved = Paths.get(path); + Path resolved = TestModelPaths.fromProperty("net.ladenthin.llama.model.path"); + Assumptions.assumeTrue(resolved != null, "model path property not set"); Assumptions.assumeTrue(Files.exists(resolved), "model file not present: " + resolved); return resolved; } diff --git a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaEmbeddingModelIntegrationTest.java b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaEmbeddingModelIntegrationTest.java index 79e0384a2..2aede4ebb 100644 --- a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaEmbeddingModelIntegrationTest.java +++ b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaEmbeddingModelIntegrationTest.java @@ -14,7 +14,6 @@ import dev.langchain4j.model.output.Response; import java.nio.file.Files; import java.nio.file.Path; -import java.nio.file.Paths; import java.util.Arrays; import java.util.List; import net.ladenthin.llama.LlamaModel; @@ -32,9 +31,8 @@ class JllamaEmbeddingModelIntegrationTest { private static Path modelPath() { - String path = System.getProperty("net.ladenthin.llama.langchain4j.embedding.model"); - Assumptions.assumeTrue(path != null && !path.isEmpty(), "embedding model path property not set"); - Path resolved = Paths.get(path); + Path resolved = TestModelPaths.fromProperty("net.ladenthin.llama.langchain4j.embedding.model"); + Assumptions.assumeTrue(resolved != null, "embedding model path property not set"); Assumptions.assumeTrue(Files.exists(resolved), "embedding model file not present: " + resolved); return resolved; } diff --git a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaScoringModelIntegrationTest.java b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaScoringModelIntegrationTest.java index 4c61c16d8..db4442dbb 100644 --- a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaScoringModelIntegrationTest.java +++ b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaScoringModelIntegrationTest.java @@ -12,7 +12,6 @@ import dev.langchain4j.model.output.Response; import java.nio.file.Files; import java.nio.file.Path; -import java.nio.file.Paths; import java.util.Arrays; import java.util.List; import net.ladenthin.llama.LlamaModel; @@ -30,9 +29,8 @@ class JllamaScoringModelIntegrationTest { private static Path modelPath() { - String path = System.getProperty("net.ladenthin.llama.langchain4j.rerank.model"); - Assumptions.assumeTrue(path != null && !path.isEmpty(), "rerank model path property not set"); - Path resolved = Paths.get(path); + Path resolved = TestModelPaths.fromProperty("net.ladenthin.llama.langchain4j.rerank.model"); + Assumptions.assumeTrue(resolved != null, "rerank model path property not set"); Assumptions.assumeTrue(Files.exists(resolved), "rerank model file not present: " + resolved); return resolved; } diff --git a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaToolCallingIntegrationTest.java b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaToolCallingIntegrationTest.java index 2f24169c1..5e95cb2f7 100644 --- a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaToolCallingIntegrationTest.java +++ b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaToolCallingIntegrationTest.java @@ -23,7 +23,7 @@ import dev.langchain4j.model.chat.response.ChatResponse; import dev.langchain4j.model.output.FinishReason; import java.nio.file.Files; -import java.nio.file.Paths; +import java.nio.file.Path; import net.ladenthin.llama.LlamaModel; import net.ladenthin.llama.parameters.ModelParameters; import org.junit.jupiter.api.AfterAll; @@ -49,9 +49,10 @@ class JllamaToolCallingIntegrationTest { @BeforeAll static void loadModel() { - String path = System.getProperty(PROP_TOOL_MODEL); - Assumptions.assumeTrue(path != null && !path.isEmpty(), "tool model path property not set"); - Assumptions.assumeTrue(Files.exists(Paths.get(path)), "model file not present: " + path); + Path resolved = TestModelPaths.fromProperty(PROP_TOOL_MODEL); + Assumptions.assumeTrue(resolved != null, "tool model path property not set"); + Assumptions.assumeTrue(Files.exists(resolved), "model file not present: " + resolved); + String path = resolved.toString(); model = new LlamaModel(new ModelParameters() .setModel(path) .setCtxSize(8192) diff --git a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/TestModelPaths.java b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/TestModelPaths.java new file mode 100644 index 000000000..bb4a7954d --- /dev/null +++ b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/TestModelPaths.java @@ -0,0 +1,59 @@ +// SPDX-FileCopyrightText: 2026 Bernard Ladenthin +// +// SPDX-License-Identifier: MIT + +package net.ladenthin.llama.langchain4j; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +/** + * Resolves the GGUF paths the model-backed integration tests are pointed at. + * + *

Surefire's working directory defaults to the module basedir ({@code /llama-langchain4j}), + * while CI restores the shared GGUF cache to {@code /models} and passes the paths as bare + * {@code models/}. Resolving those against the module directory finds nothing, every test + * self-skips on its {@code Assumptions.assumeTrue(exists)}, and the job still reports success. This + * helper accepts either layout — module-relative first, then reactor-root — so the tests + * actually run in CI without the workflow having to know where Surefire stands. + * + *

The core module carries the same resolver as {@code TestConstants.resolveModelPath}; it is + * duplicated rather than shared because test classes are not published between modules. + */ +final class TestModelPaths { + + private TestModelPaths() {} + + /** + * Resolves a configured fixture path against the working directory and then its parent. + * + * @param path the configured path, may be {@code null} or empty + * @return an existing path, or {@code null} when {@code path} is null/empty, or the unresolved + * path itself when it exists in neither location (so skip messages name what was looked for) + */ + static Path resolve(String path) { + if (path == null || path.isEmpty()) { + return null; + } + Path candidate = Paths.get(path); + if (candidate.isAbsolute() || Files.exists(candidate)) { + return candidate; + } + Path fromParent = Paths.get("..").resolve(candidate); + if (Files.exists(fromParent)) { + return fromParent.toAbsolutePath().normalize(); + } + return candidate; + } + + /** + * Resolves the path held by a system property. + * + * @param property the system-property name + * @return the resolved path, or {@code null} when the property is unset or empty + */ + static Path fromProperty(String property) { + return resolve(System.getProperty(property)); + } +} diff --git a/llama/src/test/java/net/ladenthin/llama/AudioInputIntegrationTest.java b/llama/src/test/java/net/ladenthin/llama/AudioInputIntegrationTest.java index 4ec34dfba..76f5ed4da 100644 --- a/llama/src/test/java/net/ladenthin/llama/AudioInputIntegrationTest.java +++ b/llama/src/test/java/net/ladenthin/llama/AudioInputIntegrationTest.java @@ -48,9 +48,10 @@ public class AudioInputIntegrationTest { @BeforeAll public static void setup() { - String modelPath = System.getProperty(TestConstants.PROP_AUDIO_MODEL_PATH); - String mmprojPath = System.getProperty(TestConstants.PROP_AUDIO_MMPROJ_PATH); - audioPath = System.getProperty(TestConstants.PROP_AUDIO_PATH, TestConstants.DEFAULT_AUDIO_INPUT_PATH); + String modelPath = TestConstants.resolveModelProperty(TestConstants.PROP_AUDIO_MODEL_PATH); + String mmprojPath = TestConstants.resolveModelProperty(TestConstants.PROP_AUDIO_MMPROJ_PATH); + audioPath = TestConstants.resolveModelProperty( + TestConstants.PROP_AUDIO_PATH, TestConstants.DEFAULT_AUDIO_INPUT_PATH); Assumptions.assumeTrue( modelPath != null && !modelPath.isEmpty(), diff --git a/llama/src/test/java/net/ladenthin/llama/LlamaEmbeddingsTest.java b/llama/src/test/java/net/ladenthin/llama/LlamaEmbeddingsTest.java index a2f52b85c..380f35018 100644 --- a/llama/src/test/java/net/ladenthin/llama/LlamaEmbeddingsTest.java +++ b/llama/src/test/java/net/ladenthin/llama/LlamaEmbeddingsTest.java @@ -222,7 +222,7 @@ private static void assertEmbeddingValid(float[] embedding, PoolingType type) { */ @Test public void testNomicEmbedLoads() { - String nomicPath = System.getProperty(TestConstants.PROP_NOMIC_MODEL_PATH); + String nomicPath = TestConstants.resolveModelProperty(TestConstants.PROP_NOMIC_MODEL_PATH); Assumptions.assumeTrue( nomicPath != null, "Set -D" + TestConstants.PROP_NOMIC_MODEL_PATH + " to a nomic-embed-text GGUF to run this test"); diff --git a/llama/src/test/java/net/ladenthin/llama/MultimodalIntegrationTest.java b/llama/src/test/java/net/ladenthin/llama/MultimodalIntegrationTest.java index dc50033b7..3707424d8 100644 --- a/llama/src/test/java/net/ladenthin/llama/MultimodalIntegrationTest.java +++ b/llama/src/test/java/net/ladenthin/llama/MultimodalIntegrationTest.java @@ -91,14 +91,15 @@ public class MultimodalIntegrationTest { @BeforeAll public static void setup() { - modelPath = System.getProperty(TestConstants.PROP_VISION_MODEL_PATH); - mmprojPath = System.getProperty(TestConstants.PROP_VISION_MMPROJ_PATH); + modelPath = TestConstants.resolveModelProperty(TestConstants.PROP_VISION_MODEL_PATH); + mmprojPath = TestConstants.resolveModelProperty(TestConstants.PROP_VISION_MMPROJ_PATH); // Image path falls back to the committed test resource when the // -D property is unset, so the test works on local dev checkouts // without any extra wiring. The model / mmproj remain externally // staged because their combined size (~600 MB) is too large to // commit. - imagePath = System.getProperty(TestConstants.PROP_VISION_IMAGE_PATH, TestConstants.DEFAULT_VISION_IMAGE_PATH); + imagePath = TestConstants.resolveModelProperty( + TestConstants.PROP_VISION_IMAGE_PATH, TestConstants.DEFAULT_VISION_IMAGE_PATH); Assumptions.assumeTrue( modelPath != null && !modelPath.isEmpty(), diff --git a/llama/src/test/java/net/ladenthin/llama/TestConstants.java b/llama/src/test/java/net/ladenthin/llama/TestConstants.java index 753f7eef9..4c173c368 100644 --- a/llama/src/test/java/net/ladenthin/llama/TestConstants.java +++ b/llama/src/test/java/net/ladenthin/llama/TestConstants.java @@ -5,32 +5,94 @@ package net.ladenthin.llama; +import java.io.File; +import java.nio.file.Path; +import java.nio.file.Paths; import net.ladenthin.llama.loader.LlamaSystemProperties; public class TestConstants { + /** + * Resolves a test-fixture path that may be stated relative to either the reactor root or the + * {@code llama/} module directory. + * + *

Surefire's working directory defaults to the module basedir ({@code /llama}), while + * the shared GGUF cache CI restores — and the download commands in {@code CLAUDE.md} + * — put models in {@code /models}. A bare {@code models/…} therefore resolves to a + * path that does not exist, every model-gated test aborts in its {@code @BeforeAll} + * {@code Assumptions.assumeTrue(file.exists())}, and the job still reports success — so the + * whole model-backed suite silently self-skipped on every platform. This resolver accepts both + * layouts instead of forcing one, which also keeps a developer who put models under + * {@code llama/models/} working.

+ * + *

An absolute path is returned unchanged. A relative path is returned unchanged when it + * exists relative to the working directory; otherwise the parent directory is tried and, on a + * hit, an absolute path is returned. A path that exists in neither place is returned unchanged, + * so the caller's "model missing" skip message still names what it looked for.

+ * + * @param path the configured path, may be {@code null} or empty + * @return the resolved path, or {@code path} itself when it is null, empty, absolute, or unresolvable + */ + public static String resolveModelPath(String path) { + if (path == null || path.isEmpty()) { + return path; + } + Path candidate = Paths.get(path); + if (candidate.isAbsolute() || new File(path).exists()) { + return path; + } + Path fromParent = Paths.get("..").resolve(candidate); + if (fromParent.toFile().exists()) { + return fromParent.toAbsolutePath().normalize().toString(); + } + return path; + } + + /** + * Reads a system property holding a fixture path and resolves it via + * {@link #resolveModelPath(String)}. CI passes these as {@code models/}, which is subject + * to exactly the working-directory mismatch described there. + * + * @param key the system-property name + * @param defaultValue value to use when the property is unset, may be {@code null} + * @return the resolved path, or {@code null} when neither the property nor a default is set + */ + public static String resolveModelProperty(String key, String defaultValue) { + return resolveModelPath(System.getProperty(key, defaultValue)); + } + + /** + * Reads a system property holding a fixture path, with no default. + * + * @param key the system-property name + * @return the resolved path, or {@code null} when the property is unset + */ + public static String resolveModelProperty(String key) { + return resolveModelProperty(key, null); + } + /** System property to override GPU layers used in tests. */ public static final String PROP_TEST_NGL = LlamaSystemProperties.PREFIX + ".test.ngl"; public static final int DEFAULT_TEST_NGL = 43; /** Path to the main text generation model used in tests. */ - public static final String MODEL_PATH = "models/codellama-7b.Q2_K.gguf"; + public static final String MODEL_PATH = resolveModelPath("models/codellama-7b.Q2_K.gguf"); /** Path to the draft model used for speculative decoding tests. */ - public static final String DRAFT_MODEL_PATH = "models/AMD-Llama-135m-code.Q2_K.gguf"; + public static final String DRAFT_MODEL_PATH = resolveModelPath("models/AMD-Llama-135m-code.Q2_K.gguf"); /** Path to the Qwen3 thinking model used for reasoning budget tests. */ - public static final String REASONING_MODEL_PATH = "models/Qwen3-0.6B-Q4_K_M.gguf"; + public static final String REASONING_MODEL_PATH = resolveModelPath("models/Qwen3-0.6B-Q4_K_M.gguf"); /** Path to the reranking model used in tests (loaded with {@code enableReranking()}). */ - public static final String RERANKING_MODEL_PATH = "models/jina-reranker-v1-tiny-en-Q4_0.gguf"; + public static final String RERANKING_MODEL_PATH = resolveModelPath("models/jina-reranker-v1-tiny-en-Q4_0.gguf"); /** System property overriding the GGUF used by the real tool-calling integration tests. */ public static final String PROP_TOOL_MODEL_PATH = LlamaSystemProperties.PREFIX + ".tool.model"; /** Qwen2.5 tool-capable model used by upstream llama.cpp's blocking and streaming tests. */ - public static final String DEFAULT_TOOL_MODEL_PATH = "models/Qwen2.5-1.5B-Instruct-Q4_K_M.gguf"; + public static final String DEFAULT_TOOL_MODEL_PATH = resolveModelPath("models/Qwen2.5-1.5B-Instruct-Q4_K_M.gguf"); /** * System property holding a path to a Nomic embedding model @@ -70,7 +132,7 @@ public class TestConstants { * {@link #PROP_VISION_IMAGE_PATH} is unset. Points at the committed test * resource so the test needs no network access for the visual prompt. */ - public static final String DEFAULT_VISION_IMAGE_PATH = "src/test/resources/images/test-image.jpg"; + public static final String DEFAULT_VISION_IMAGE_PATH = resolveModelPath("src/test/resources/images/test-image.jpg"); /** * System property holding a path to an audio-input model GGUF (e.g. Ultravox / Qwen2.5-Omni). @@ -96,7 +158,7 @@ public class TestConstants { * the committed test resource so only the (large) audio model + mmproj have to be staged * out-of-band. */ - public static final String DEFAULT_AUDIO_INPUT_PATH = "src/test/resources/audios/sample.wav"; + public static final String DEFAULT_AUDIO_INPUT_PATH = resolveModelPath("src/test/resources/audios/sample.wav"); /** * System property holding a path to the Qwen3-TTS backbone GGUF used by diff --git a/llama/src/test/java/net/ladenthin/llama/TestConstantsTest.java b/llama/src/test/java/net/ladenthin/llama/TestConstantsTest.java new file mode 100644 index 000000000..b2e960956 --- /dev/null +++ b/llama/src/test/java/net/ladenthin/llama/TestConstantsTest.java @@ -0,0 +1,99 @@ +// SPDX-FileCopyrightText: 2026 Bernard Ladenthin +// +// SPDX-License-Identifier: MIT + +package net.ladenthin.llama; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.File; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +@ClaudeGenerated( + purpose = "Pin TestConstants.resolveModelPath — the fix for model-gated tests silently " + + "self-skipping in CI because Surefire's working directory is the llama/ module " + + "while the shared GGUF cache is restored to the reactor root.") +public class TestConstantsTest { + + @Test + public void nullAndEmptyPassThrough() { + assertNull(TestConstants.resolveModelPath(null)); + assertEquals("", TestConstants.resolveModelPath("")); + } + + @Test + public void absolutePathIsReturnedUnchanged(@TempDir Path tempDir) throws Exception { + Path file = Files.createFile(tempDir.resolve("model.gguf")); + String absolute = file.toAbsolutePath().toString(); + assertEquals(absolute, TestConstants.resolveModelPath(absolute)); + } + + @Test + public void existingRelativePathIsReturnedUnchanged() { + // pom.xml exists relative to the module basedir, which is Surefire's working directory. + assertTrue(new File("pom.xml").exists(), "precondition: the working directory is the module basedir"); + assertEquals("pom.xml", TestConstants.resolveModelPath("pom.xml")); + } + + @Test + public void pathOnlyPresentInTheParentResolvesToAnExistingAbsolutePath() { + // The reactor root holds the aggregator pom; the module directory does not hold a + // "../pom.xml"-shaped sibling of that name, so this is exactly the models/ situation. + String resolved = TestConstants.resolveModelPath("llama/pom.xml"); + assertTrue( + new File(resolved).exists(), "a path that only exists from the reactor root must resolve: " + resolved); + assertTrue(Paths.get(resolved).isAbsolute(), "resolved parent-relative paths are absolute: " + resolved); + } + + @Test + public void unresolvablePathIsReturnedUnchangedSoSkipMessagesStayReadable() { + String missing = "models/definitely-not-present-" + TestConstantsTest.class.getSimpleName() + ".gguf"; + assertEquals(missing, TestConstants.resolveModelPath(missing)); + } + + @Test + public void propertyLookupResolvesAndHonoursTheDefault() { + String key = "net.ladenthin.llama.test.resolver.probe"; + assertNull(System.getProperty(key), "precondition: probe property must be unset"); + assertNull(TestConstants.resolveModelProperty(key)); + assertEquals("pom.xml", TestConstants.resolveModelProperty(key, "pom.xml")); + + System.setProperty(key, "llama/pom.xml"); + try { + assertTrue(new File(TestConstants.resolveModelProperty(key)).exists()); + } finally { + System.clearProperty(key); + } + } + + @Test + public void theShippedModelConstantsGoThroughTheResolver() { + // Guards the wiring itself: if a future edit drops the resolveModelPath(...) wrapper from a + // constant, that constant stops matching its resolved literal and every test gated on it + // silently self-skips again in CI. Holds in both layouts — with the GGUF present the + // resolver returns the same absolute path on both sides, without it the same literal. + assertEquals(TestConstants.resolveModelPath("models/codellama-7b.Q2_K.gguf"), TestConstants.MODEL_PATH); + assertEquals( + TestConstants.resolveModelPath("models/AMD-Llama-135m-code.Q2_K.gguf"), TestConstants.DRAFT_MODEL_PATH); + assertEquals( + TestConstants.resolveModelPath("models/Qwen3-0.6B-Q4_K_M.gguf"), TestConstants.REASONING_MODEL_PATH); + assertEquals( + TestConstants.resolveModelPath("models/jina-reranker-v1-tiny-en-Q4_0.gguf"), + TestConstants.RERANKING_MODEL_PATH); + assertEquals( + TestConstants.resolveModelPath("models/Qwen2.5-1.5B-Instruct-Q4_K_M.gguf"), + TestConstants.DEFAULT_TOOL_MODEL_PATH); + assertEquals( + TestConstants.resolveModelPath("src/test/resources/images/test-image.jpg"), + TestConstants.DEFAULT_VISION_IMAGE_PATH); + assertEquals( + TestConstants.resolveModelPath("src/test/resources/audios/sample.wav"), + TestConstants.DEFAULT_AUDIO_INPUT_PATH); + } +} diff --git a/llama/src/test/java/net/ladenthin/llama/ToolCallingIntegrationTest.java b/llama/src/test/java/net/ladenthin/llama/ToolCallingIntegrationTest.java index 83c3a6d24..7d9417392 100644 --- a/llama/src/test/java/net/ladenthin/llama/ToolCallingIntegrationTest.java +++ b/llama/src/test/java/net/ladenthin/llama/ToolCallingIntegrationTest.java @@ -39,8 +39,8 @@ public class ToolCallingIntegrationTest { @BeforeAll public static void loadModel() { - String modelPath = - System.getProperty(TestConstants.PROP_TOOL_MODEL_PATH, TestConstants.DEFAULT_TOOL_MODEL_PATH); + String modelPath = TestConstants.resolveModelProperty( + TestConstants.PROP_TOOL_MODEL_PATH, TestConstants.DEFAULT_TOOL_MODEL_PATH); Assumptions.assumeTrue(new File(modelPath).exists(), "Tool-calling model missing: " + modelPath); int gpuLayers = Integer.getInteger(TestConstants.PROP_TEST_NGL, 0); ModelParameters parameters = new ModelParameters() diff --git a/llama/src/test/java/net/ladenthin/llama/TtsIntegrationTest.java b/llama/src/test/java/net/ladenthin/llama/TtsIntegrationTest.java index 4288f41fc..cce35aff6 100644 --- a/llama/src/test/java/net/ladenthin/llama/TtsIntegrationTest.java +++ b/llama/src/test/java/net/ladenthin/llama/TtsIntegrationTest.java @@ -36,8 +36,8 @@ public class TtsIntegrationTest { @DisplayName("synthesize() returns a well-formed, non-silent 24 kHz mono 16-bit WAV") @Timeout(value = 300_000, unit = TimeUnit.MILLISECONDS) public void synthesizesWellFormedWav() { - String model = System.getProperty(TestConstants.PROP_TTS_MODEL); - String mmproj = System.getProperty(TestConstants.PROP_TTS_MMPROJ); + String model = TestConstants.resolveModelProperty(TestConstants.PROP_TTS_MODEL); + String mmproj = TestConstants.resolveModelProperty(TestConstants.PROP_TTS_MMPROJ); Assumptions.assumeTrue( model != null && !model.isEmpty(), "TTS model not set (-D" + TestConstants.PROP_TTS_MODEL + "=...)"); Assumptions.assumeTrue( diff --git a/llama/src/test/java/net/ladenthin/llama/server/OpenAiServerToolCallingIntegrationTest.java b/llama/src/test/java/net/ladenthin/llama/server/OpenAiServerToolCallingIntegrationTest.java index de60c8e0d..25f2a2120 100644 --- a/llama/src/test/java/net/ladenthin/llama/server/OpenAiServerToolCallingIntegrationTest.java +++ b/llama/src/test/java/net/ladenthin/llama/server/OpenAiServerToolCallingIntegrationTest.java @@ -52,8 +52,8 @@ public class OpenAiServerToolCallingIntegrationTest extends OpenAiServerTestSupp @BeforeAll public static void setup() throws IOException { - String modelPath = - System.getProperty(TestConstants.PROP_TOOL_MODEL_PATH, TestConstants.DEFAULT_TOOL_MODEL_PATH); + String modelPath = TestConstants.resolveModelProperty( + TestConstants.PROP_TOOL_MODEL_PATH, TestConstants.DEFAULT_TOOL_MODEL_PATH); Assumptions.assumeTrue( new File(modelPath).exists(), "Tool-calling model (Qwen2.5-1.5B) not found, skipping server tool-calling test: " + modelPath); From d18f3157595032fcb29b471e5f577f39016e4023 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 14:22:32 +0000 Subject: [PATCH 32/76] fix: rebuild getMetrics() from both upstream introspection tasks Upstream stopped emitting the server-introspection object in one piece: b10408 (#26920) reduced server_task_result_metrics::to_json() to the bare slot array, and b10519 (#27376) split the task in two -- METRICS keeps only the counters (to_json() unused, to_metrics() renders Prometheus text) while SLOT_GET carries the slot array plus the idle-slot count. Three Java consumers were written against the pre-b10408 object and all still assume it: ServerMetrics' counter getters read keys that no longer existed, LlamaModelTest#testGetMetrics asserted substrings, and OpenAiCompatServer's GET /slots did path("slots") on an array, whose MissingNode.toString() is "" -- a 200 with a zero-length body. Rather than bend the Java contract to upstream's transport split, handleSlotAction(0, ...) now posts both tasks and merges them through the new pure helper server_metrics_to_json (json_helpers.hpp), so getMetrics() returns the documented object again and all three consumers keep working unchanged. dispatch_one_shot_task is factored over a new post_and_wait that hands back the typed result. Durations are converted from upstream microseconds to the milliseconds the payload has always used; metrics_reset_bucket is left at its default false, so getMetrics() never resets the measurement window. The merge also surfaces counters that had no JSON representation at all before, only Prometheus text: n_prompt_tokens_cached_total and the speculative-decoding tallies. ServerMetrics exposes them as getCumulativeCachedPromptTokens, getDraftTokensTotal, getDraftAcceptedTotal, getDraftVerifyStepsTotal, getDraftAcceptedPerPosition and the derived getDraftAcceptanceRate. Its getStartTimestamp javadoc is corrected while here: t_start is ggml_time_us(), a monotonic-clock microsecond reading, not epoch millis. GET /slots now answers [] rather than an empty body when the payload carries no slots key, with a regression test; testGetMetrics asserts the parsed shape (object, slots array, integral counters) instead of substrings that the slot entries themselves satisfy -- that assertion is what would have caught the b10408 drift. C++ tests 491 -> 499. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- llama/src/main/cpp/jllama.cpp | 41 ++++-- llama/src/main/cpp/json_helpers.hpp | 65 ++++++++++ .../llama/server/OpenAiCompatServer.java | 6 +- .../ladenthin/llama/value/ServerMetrics.java | 112 +++++++++++++++-- llama/src/test/cpp/test_json_helpers.cpp | 118 ++++++++++++++++++ .../net/ladenthin/llama/LlamaModelTest.java | 39 +++++- .../server/OpenAiCompatServerHttpTest.java | 21 +++- .../llama/value/ServerMetricsTest.java | 56 +++++++++ 8 files changed, 428 insertions(+), 30 deletions(-) diff --git a/llama/src/main/cpp/jllama.cpp b/llama/src/main/cpp/jllama.cpp index 2161e7b95..5a63ece76 100644 --- a/llama/src/main/cpp/jllama.cpp +++ b/llama/src/main/cpp/jllama.cpp @@ -346,15 +346,24 @@ static void populate_completion_task(server_task &task, jllama_context *jctx, return task; } -// Post a single pre-built task, wait for its result, and return JSON as a jstring. +// Post a single pre-built task and wait for its result. Returns nullptr after +// throwing via JNI when the task failed; callers must return immediately then. // The task's id field is assigned here; callers must not set it beforehand. -[[nodiscard]] static jstring dispatch_one_shot_task(JNIEnv *env, server_context *ctx_server, server_task task) { +[[nodiscard]] static server_task_result_ptr post_and_wait(JNIEnv *env, server_context *ctx_server, server_task task) { auto rd = ctx_server->get_response_reader(); task.id = rd.get_new_id(); rd.post_task(std::move(task)); auto result = rd.next([] { return false; }); if (!result_ok_or_throw(env, result)) return nullptr; + return result; +} + +// Post a single pre-built task, wait for its result, and return JSON as a jstring. +[[nodiscard]] static jstring dispatch_one_shot_task(JNIEnv *env, server_context *ctx_server, server_task task) { + auto result = post_and_wait(env, ctx_server, std::move(task)); + if (!result) + return nullptr; return json_to_jstring(env, result->to_json()); } @@ -1625,13 +1634,27 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleSlotAction(J REQUIRE_SERVER_CONTEXT(nullptr); switch (action) { - case 0: // LIST — get slot info via the dedicated slot-get task - // b10519 (upstream #27376) split the old METRICS task in two: METRICS now carries only the - // cumulative counters (rendered as Prometheus text by to_metrics(); its to_json() is unused - // and returns `json{}`, i.e. JSON null), and the /slots payload moved to - // SERVER_TASK_TYPE_SLOT_GET -> server_task_result_slots::to_json(), which returns the slot - // array verbatim. - return dispatch_one_shot_task(env, ctx_server, server_task(SERVER_TASK_TYPE_SLOT_GET)); + case 0: { // LIST — the full server-introspection payload + // b10408 (upstream #26920) reduced server_task_result_metrics::to_json() to the slot array + // and b10519 (#27376) split the task in two: METRICS keeps only the counters (its to_json() + // is unused and returns JSON null; to_metrics() renders them as Prometheus text) and + // SERVER_TASK_TYPE_SLOT_GET carries the slot array plus the idle-slot count. Post both and + // merge them so getMetrics() keeps returning the single documented object. + auto metrics_result = post_and_wait(env, ctx_server, server_task(SERVER_TASK_TYPE_METRICS)); + if (!metrics_result) + return nullptr; + auto slots_result = post_and_wait(env, ctx_server, server_task(SERVER_TASK_TYPE_SLOT_GET)); + if (!slots_result) + return nullptr; + + const auto *metrics = dynamic_cast(metrics_result.get()); + const auto *slots = dynamic_cast(slots_result.get()); + if (metrics == nullptr || slots == nullptr) { + env->ThrowNew(c_llama_error, "Unexpected result type for server metrics"); + return nullptr; + } + return json_to_jstring(env, server_metrics_to_json(*metrics, *slots)); + } case 1: // SAVE return exec_slot_file_task(env, ctx_server, slotId, jfilename, SERVER_TASK_TYPE_SLOT_SAVE, "Filename is required for slot save"); diff --git a/llama/src/main/cpp/json_helpers.hpp b/llama/src/main/cpp/json_helpers.hpp index 46cf9e858..f2032e0a2 100644 --- a/llama/src/main/cpp/json_helpers.hpp +++ b/llama/src/main/cpp/json_helpers.hpp @@ -34,6 +34,7 @@ // 7. parse_slot_prompt_similarity — used by nothing above it // 8. parse_positive_int_config — used by nothing above it // 9. wrap_stream_chunk — used by nothing above it +// 10. server_metrics_to_json — used by nothing above it #include #include @@ -229,3 +230,67 @@ out["stop"] = stop; return out; } + +// --------------------------------------------------------------------------- +// server_metrics_to_json +// +// Merges the two halves of the server-introspection payload back into the one +// object LlamaModel.getMetrics() has always returned. +// +// Until llama.cpp b10408 (upstream #26920) a single SERVER_TASK_TYPE_METRICS +// task produced that object. b10408 reduced its to_json() to the slot array, +// and b10519 (#27376) split the task in two: METRICS now carries only the +// counters (rendered as Prometheus text by to_metrics(); its to_json() is +// unused and returns JSON null) and SERVER_TASK_TYPE_SLOT_GET carries the slot +// array plus the idle-slot count. Rather than let the Java contract follow +// upstream's transport split, the JNI layer posts both tasks and rebuilds the +// object here. +// +// Key names are the pre-b10408 ones — `idle`, `processing`, `deferred`, +// `t_start`, the `n_*`/`t_*` counter pairs and `slots` — so every existing +// consumer keeps working. The counters upstream added since (cached prompt +// tokens and the speculative-decoding tallies) are emitted alongside them; they +// were previously reachable only through the Prometheus text. +// +// Durations are microseconds upstream and milliseconds here, matching what the +// pre-b10408 payload used. +// --------------------------------------------------------------------------- +[[nodiscard]] inline json server_metrics_to_json(const server_task_result_metrics &metrics_result, + const server_task_result_slots &slots_result) { + const server_metrics &m = metrics_result.metrics; + + // Microseconds -> milliseconds, as a double so sub-millisecond timings survive. + const auto to_ms = [](uint64_t time_us) { return static_cast(time_us) / 1000.0; }; + + json out; + out["idle"] = slots_result.n_idle_slots; + out["processing"] = metrics_result.n_processing_slots; + out["deferred"] = metrics_result.n_tasks_deferred; + out["t_start"] = m.t_start; + + // Cumulative since server start. + out["n_prompt_tokens_processed_total"] = m.prompt.count; + out["t_prompt_processing_total"] = to_ms(m.prompt.time); + out["n_tokens_predicted_total"] = m.predict.count; + out["t_tokens_generation_total"] = to_ms(m.predict.time); + out["n_decode_total"] = m.n_decode; + out["n_busy_slots_total"] = m.n_busy_slots; + out["n_tokens_max"] = m.n_tokens_max; + + // Current measurement window: reset by an HTTP /metrics scrape, never by this call + // (the JNI task leaves server_task::metrics_reset_bucket at its default false). + out["n_prompt_tokens_processed"] = m.prompt_bucket.count; + out["t_prompt_processing"] = to_ms(m.prompt_bucket.time); + out["n_tokens_predicted"] = m.predict_bucket.count; + out["t_tokens_generation"] = to_ms(m.predict_bucket.time); + + // Counters with no pre-b10408 equivalent. + out["n_prompt_tokens_cached_total"] = m.n_prompt_cached; + out["n_draft_tokens_total"] = m.n_draft_tokens; + out["n_draft_accepted_total"] = m.n_draft_accepted; + out["n_draft_verify_steps_total"] = m.n_draft_verif_steps; + out["n_draft_accepted_per_pos"] = m.n_accepted_per_pos; + + out["slots"] = slots_result.slots_data; + return out; +} diff --git a/llama/src/main/java/net/ladenthin/llama/server/OpenAiCompatServer.java b/llama/src/main/java/net/ladenthin/llama/server/OpenAiCompatServer.java index bdd9befd9..dd051c5de 100644 --- a/llama/src/main/java/net/ladenthin/llama/server/OpenAiCompatServer.java +++ b/llama/src/main/java/net/ladenthin/llama/server/OpenAiCompatServer.java @@ -787,7 +787,11 @@ private void handleMetricsView(HttpExchange exchange, boolean slotsOnly) throws } String metrics = backend.metrics(); if (slotsOnly) { - metrics = OBJECT_MAPPER.readTree(metrics).path("slots").toString(); + // path() yields a MissingNode when "slots" is absent, and MissingNode.toString() + // is the empty string — which would answer 200 with a zero-length body, i.e. not + // JSON at all. Fall back to an empty array so the route always answers in shape. + JsonNode slots = OBJECT_MAPPER.readTree(metrics).path("slots"); + metrics = slots.isArray() ? slots.toString() : "[]"; } sendJson(exchange, HTTP_OK, metrics); } catch (IOException | RuntimeException e) { diff --git a/llama/src/main/java/net/ladenthin/llama/value/ServerMetrics.java b/llama/src/main/java/net/ladenthin/llama/value/ServerMetrics.java index 32d520a57..826758cfb 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/ServerMetrics.java +++ b/llama/src/main/java/net/ladenthin/llama/value/ServerMetrics.java @@ -18,14 +18,26 @@ * {@link ModelMeta} pattern. *

*

- * The native {@code server_task_result_metrics::to_json()} emits server-wide counters: - * {@code idle}, {@code processing}, {@code deferred}, {@code t_start}, - * {@code n_prompt_tokens_processed_total}, {@code t_prompt_processing_total}, - * {@code n_tokens_predicted_total}, {@code t_tokens_generation_total}, - * {@code n_prompt_tokens_processed}, {@code t_prompt_processing}, - * {@code n_tokens_predicted}, {@code t_tokens_generation}, - * {@code n_decode_total}, {@code n_busy_slots_total}, optionally {@code n_tokens_max}, - * and a {@code slots} array. + * The payload carries server-wide counters — {@code idle}, {@code processing}, + * {@code deferred}, {@code t_start}, {@code n_prompt_tokens_processed_total}, + * {@code t_prompt_processing_total}, {@code n_tokens_predicted_total}, + * {@code t_tokens_generation_total}, {@code n_prompt_tokens_processed}, + * {@code t_prompt_processing}, {@code n_tokens_predicted}, {@code t_tokens_generation}, + * {@code n_decode_total}, {@code n_busy_slots_total}, {@code n_tokens_max} — + * plus the cache and speculative-decoding counters + * ({@code n_prompt_tokens_cached_total}, {@code n_draft_tokens_total}, + * {@code n_draft_accepted_total}, {@code n_draft_verify_steps_total}, + * {@code n_draft_accepted_per_pos}) and a {@code slots} array. + *

+ *

+ * Upstream llama.cpp stopped emitting this object as one piece: b10408 (upstream #26920) + * reduced {@code server_task_result_metrics::to_json()} to the slot array, and b10519 + * (#27376) split the task into a counters-only {@code METRICS} task whose numbers are + * reachable only as Prometheus text and a {@code SLOT_GET} task carrying the slots. The + * JNI layer posts both and merges them back into the shape documented here (see + * {@code server_metrics_to_json} in {@code json_helpers.hpp}), so this view is unaffected + * by that transport split. The cache and speculative-decoding counters are new: before the + * merge they had no JSON representation at all. *

* *

{@code equals}/{@code hashCode} are generated by Lombok over the underlying @@ -72,8 +84,12 @@ public int getDeferredTasks() { } /** - * Server start timestamp. - * @return millis since epoch as reported by llama.cpp + * Server start timestamp, as llama.cpp records it: {@code ggml_time_us()} at startup, i.e. + * microseconds read from a monotonic clock ({@code CLOCK_MONOTONIC} on POSIX, the + * performance counter on Windows). It is not a wall-clock epoch value and is + * only meaningful as a difference against another reading from the same process. + * + * @return the monotonic-clock microsecond reading taken when the server started */ public long getStartTimestamp() { return node.path("t_start").asLong(0L); @@ -96,15 +112,85 @@ public long getBusySlotsTotal() { } /** - * Maximum number of tokens any active slot is configured to hold. Absent in the - * upstream JSON when no slot has been used yet; this getter returns {@code 0} then. + * Largest sequence length (prompt plus generation) observed since server start. + * {@code 0} until a slot has processed anything. * - * @return the slot token-budget ceiling, or {@code 0} when no slot has been active + * @return the largest observed sequence length, or {@code 0} when no slot has been active */ public int getTokensMax() { return node.path("n_tokens_max").asInt(0); } + /** + * Lifetime count of prompt tokens served straight from the KV cache, which therefore + * needed no decode. Complements {@link #getCumulativeProcessedPromptTokens()}, which + * counts only the tokens actually evaluated. + * + * @return cached prompt-token total since server start + */ + public long getCumulativeCachedPromptTokens() { + return node.path("n_prompt_tokens_cached_total").asLong(0L); + } + + /** + * Lifetime count of draft tokens produced by the draft model during speculative + * decoding. {@code 0} when no draft model is configured. + * + * @return generated draft-token total since server start + */ + public long getDraftTokensTotal() { + return node.path("n_draft_tokens_total").asLong(0L); + } + + /** + * Lifetime count of draft tokens the target model accepted. Together with + * {@link #getDraftTokensTotal()} this is the speculative-decoding acceptance rate. + * + * @return accepted draft-token total since server start + */ + public long getDraftAcceptedTotal() { + return node.path("n_draft_accepted_total").asLong(0L); + } + + /** + * Lifetime count of verification steps the target model ran over drafted tokens. + * + * @return draft verification-step total since server start + */ + public long getDraftVerifyStepsTotal() { + return node.path("n_draft_verify_steps_total").asLong(0L); + } + + /** + * Fraction of drafted tokens the target model accepted, i.e. + * {@link #getDraftAcceptedTotal()} divided by {@link #getDraftTokensTotal()}. + * + * @return acceptance rate in {@code [0.0, 1.0]}, or {@code 0.0} when nothing was drafted + */ + public double getDraftAcceptanceRate() { + long drafted = getDraftTokensTotal(); + return drafted > 0L ? (double) getDraftAcceptedTotal() / (double) drafted : 0.0; + } + + /** + * Accepted draft tokens broken down by their position within the draft, index {@code 0} + * being the token immediately after the verified prefix. Empty when speculative decoding + * has not run. + * + * @return immutable per-position acceptance counts, empty when none are reported + */ + public List getDraftAcceptedPerPosition() { + JsonNode perPos = node.path("n_draft_accepted_per_pos"); + if (!perPos.isArray()) { + return Collections.emptyList(); + } + List result = new ArrayList<>(); + for (JsonNode value : perPos) { + result.add(value.asLong(0L)); + } + return Collections.unmodifiableList(result); + } + /** * Cumulative server-wide compute counters since startup. The prompt value is * the number actually evaluated by the model, excluding cache hits; upstream diff --git a/llama/src/test/cpp/test_json_helpers.cpp b/llama/src/test/cpp/test_json_helpers.cpp index 02ee602f0..2d7faaa0d 100644 --- a/llama/src/test/cpp/test_json_helpers.cpp +++ b/llama/src/test/cpp/test_json_helpers.cpp @@ -21,6 +21,7 @@ // parse_slot_prompt_similarity // parse_positive_int_config // wrap_stream_chunk +// server_metrics_to_json #include @@ -506,3 +507,120 @@ TEST(WrapStreamChunk, ExactlyTwoKeys) { EXPECT_TRUE(out.contains("data")); EXPECT_TRUE(out.contains("stop")); } + +// ============================================================ +// server_metrics_to_json +// ============================================================ + +namespace { + +// Two halves of the server-introspection payload, filled with distinguishable values so a +// mis-wired field is visible in the assertion rather than colliding with a neighbour. +server_task_result_metrics make_metrics_half() { + server_task_result_metrics m; + m.n_processing_slots = 3; + m.n_tasks_deferred = 4; + m.metrics.t_start = 1234567890; + m.metrics.prompt.count = 100; + m.metrics.prompt.time = 2000000; // 2 s in us -> 2000 ms + m.metrics.predict.count = 200; + m.metrics.predict.time = 500000; // 0.5 s in us -> 500 ms + m.metrics.prompt_bucket.count = 10; + m.metrics.prompt_bucket.time = 1500; // 1.5 ms + m.metrics.predict_bucket.count = 20; + m.metrics.predict_bucket.time = 250; // 0.25 ms + m.metrics.n_decode = 42; + m.metrics.n_busy_slots = 84; + m.metrics.n_tokens_max = 512; + m.metrics.n_prompt_cached = 77; + m.metrics.n_draft_tokens = 30; + m.metrics.n_draft_accepted = 21; + m.metrics.n_draft_verif_steps = 7; + m.metrics.n_accepted_per_pos = {5, 3, 1}; + return m; +} + +server_task_result_slots make_slots_half() { + server_task_result_slots s; + s.n_idle_slots = 2; + s.slots_data = json::array({json::object({{"id", 0}}), json::object({{"id", 1}})}); + return s; +} + +} // namespace + +TEST(ServerMetricsToJson, MergesSlotCountsFromBothHalves) { + const json j = server_metrics_to_json(make_metrics_half(), make_slots_half()); + // `idle` comes from the SLOT_GET half, `processing`/`deferred` from the METRICS half. + EXPECT_EQ(j.at("idle").get(), 2); + EXPECT_EQ(j.at("processing").get(), 3); + EXPECT_EQ(j.at("deferred").get(), 4); +} + +TEST(ServerMetricsToJson, CarriesCumulativeCounters) { + const json j = server_metrics_to_json(make_metrics_half(), make_slots_half()); + EXPECT_EQ(j.at("t_start").get(), 1234567890); + EXPECT_EQ(j.at("n_prompt_tokens_processed_total").get(), 100); + EXPECT_EQ(j.at("n_tokens_predicted_total").get(), 200); + EXPECT_EQ(j.at("n_decode_total").get(), 42); + EXPECT_EQ(j.at("n_busy_slots_total").get(), 84); + EXPECT_EQ(j.at("n_tokens_max").get(), 512); +} + +TEST(ServerMetricsToJson, ConvertsDurationsFromMicrosecondsToMilliseconds) { + const json j = server_metrics_to_json(make_metrics_half(), make_slots_half()); + EXPECT_DOUBLE_EQ(j.at("t_prompt_processing_total").get(), 2000.0); + EXPECT_DOUBLE_EQ(j.at("t_tokens_generation_total").get(), 500.0); + // Sub-millisecond bucket timings must survive as fractions, not truncate to zero. + EXPECT_DOUBLE_EQ(j.at("t_prompt_processing").get(), 1.5); + EXPECT_DOUBLE_EQ(j.at("t_tokens_generation").get(), 0.25); +} + +TEST(ServerMetricsToJson, CarriesCurrentWindowCounters) { + const json j = server_metrics_to_json(make_metrics_half(), make_slots_half()); + EXPECT_EQ(j.at("n_prompt_tokens_processed").get(), 10); + EXPECT_EQ(j.at("n_tokens_predicted").get(), 20); +} + +TEST(ServerMetricsToJson, CarriesCountersUpstreamAddedAfterB10408) { + const json j = server_metrics_to_json(make_metrics_half(), make_slots_half()); + EXPECT_EQ(j.at("n_prompt_tokens_cached_total").get(), 77); + EXPECT_EQ(j.at("n_draft_tokens_total").get(), 30); + EXPECT_EQ(j.at("n_draft_accepted_total").get(), 21); + EXPECT_EQ(j.at("n_draft_verify_steps_total").get(), 7); + const json per_pos = j.at("n_draft_accepted_per_pos"); + ASSERT_TRUE(per_pos.is_array()); + ASSERT_EQ(per_pos.size(), 3u); + EXPECT_EQ(per_pos.at(0).get(), 5); + EXPECT_EQ(per_pos.at(2).get(), 1); +} + +TEST(ServerMetricsToJson, EmbedsTheSlotArrayVerbatimUnderSlots) { + const json j = server_metrics_to_json(make_metrics_half(), make_slots_half()); + const json slots = j.at("slots"); + ASSERT_TRUE(slots.is_array()); + ASSERT_EQ(slots.size(), 2u); + EXPECT_EQ(slots.at(1).at("id").get(), 1); +} + +TEST(ServerMetricsToJson, DefaultConstructedHalvesProduceZeroesAndAnEmptySlotArray) { + const json j = server_metrics_to_json(server_task_result_metrics{}, server_task_result_slots{}); + EXPECT_EQ(j.at("idle").get(), 0); + EXPECT_EQ(j.at("processing").get(), 0); + EXPECT_EQ(j.at("n_decode_total").get(), 0); + EXPECT_DOUBLE_EQ(j.at("t_prompt_processing_total").get(), 0.0); + ASSERT_TRUE(j.at("slots").is_array()); + EXPECT_EQ(j.at("slots").size(), 0u); + EXPECT_TRUE(j.at("n_draft_accepted_per_pos").is_array()); +} + +TEST(ServerMetricsToJson, CountersAreNumbersNotBooleans) { + // common_json binds an unscoped enum — and anything else that is not std::is_integral — + // to its bool constructor. Every counter here must arrive as a number. + const json j = server_metrics_to_json(make_metrics_half(), make_slots_half()); + for (const char *key : {"idle", "processing", "deferred", "t_start", "n_decode_total", "n_busy_slots_total", + "n_prompt_tokens_cached_total"}) { + EXPECT_FALSE(j.at(key).is_boolean()) << "key serialised as a boolean: " << key; + EXPECT_TRUE(j.at(key).is_number()) << "key is not a number: " << key; + } +} diff --git a/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java b/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java index f855b7174..f92ea7b9e 100644 --- a/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java +++ b/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java @@ -7,6 +7,7 @@ import static org.junit.jupiter.api.Assertions.*; +import com.fasterxml.jackson.databind.JsonNode; import java.io.*; import java.nio.charset.StandardCharsets; import java.util.*; @@ -63,8 +64,7 @@ public class LlamaModelTest { @BeforeAll public static void setup() { Assumptions.assumeTrue( - new java.io.File("models/codellama-7b.Q2_K.gguf").exists(), - "Model file not found, skipping LlamaModelTest"); + new java.io.File(TestConstants.MODEL_PATH).exists(), "Model file not found, skipping LlamaModelTest"); // LlamaModel.setLogger(LogFormat.TEXT, (level, msg) -> System.out.println(level + ": " + msg)); int gpuLayers = Integer.getInteger(TestConstants.PROP_TEST_NGL, TestConstants.DEFAULT_TEST_NGL); model = new LlamaModel(new ModelParameters() @@ -351,8 +351,7 @@ public void testCompleteWithCancellationToken() throws Exception { */ @Test public void testCloseDuringInference() throws Exception { - Assumptions.assumeTrue( - new java.io.File("models/codellama-7b.Q2_K.gguf").exists(), "Model file not found, skipping"); + Assumptions.assumeTrue(new java.io.File(TestConstants.MODEL_PATH).exists(), "Model file not found, skipping"); int gpuLayers = Integer.getInteger(TestConstants.PROP_TEST_NGL, TestConstants.DEFAULT_TEST_NGL); try (LlamaModel localModel = new LlamaModel(new ModelParameters() .setCtxSize(128) @@ -1217,8 +1216,36 @@ public void testCloseAfterGeneration() { public void testGetMetrics() { String metrics = model.getMetrics(); assertNotNull(metrics); - assertTrue(metrics.contains("\"slots\""), "Metrics should contain slots data"); - assertTrue(metrics.contains("\"idle\""), "Metrics should contain idle count"); + + // Assert the parsed shape, not substrings. Upstream reduced the metrics payload to a bare + // slot array at b10408 and split the task in two at b10519; the JNI layer merges the halves + // back into this object. A substring check on "slots"/"idle" is satisfied by the slot + // entries themselves, so only structural assertions can see that drift. + JsonNode root = model.getMetricsTyped().asJson(); + assertTrue(root.isObject(), "Metrics must be an object, not a bare slot array: " + metrics); + assertTrue(root.path("slots").isArray(), "Metrics must carry a slots array: " + metrics); + for (String counter : new String[] { + "idle", + "processing", + "deferred", + "n_decode_total", + "n_busy_slots_total", + "n_prompt_tokens_processed_total", + "n_tokens_predicted_total", + "n_prompt_tokens_cached_total" + }) { + assertTrue( + root.path(counter).isIntegralNumber(), + "Counter " + counter + " must be an integer, not " + + root.path(counter).getNodeType() + ": " + metrics); + } + // A loaded model always has at least one slot, and the idle count is drawn from that same + // set — an exact equality would be flaky if a prior test left a slot mid-flight, so bound it + // instead of pinning it. + int slotCount = root.path("slots").size(); + assertTrue(slotCount > 0, "a loaded model must report at least one slot: " + metrics); + int idle = model.getMetricsTyped().getIdleSlots(); + assertTrue(idle >= 0 && idle <= slotCount, "idle slots out of range 0.." + slotCount + ": " + metrics); } @Test diff --git a/llama/src/test/java/net/ladenthin/llama/server/OpenAiCompatServerHttpTest.java b/llama/src/test/java/net/ladenthin/llama/server/OpenAiCompatServerHttpTest.java index 0fb3efd81..e3d121499 100644 --- a/llama/src/test/java/net/ladenthin/llama/server/OpenAiCompatServerHttpTest.java +++ b/llama/src/test/java/net/ladenthin/llama/server/OpenAiCompatServerHttpTest.java @@ -112,6 +112,25 @@ public void metricsAndSlotsExposeCacheCounters() throws IOException { } } + @Test + public void slotsAnswersAnEmptyArrayWhenTheBackendReportsNoSlotsKey() throws IOException { + // Regression guard: GET /slots used to hand MissingNode.toString() straight to the client, + // i.e. HTTP 200 with a zero-length body, whenever the metrics payload carried no "slots" + // key — which is exactly what the native layer produced between llama.cpp b10408 and the + // JNI merge that restored the object. The route must always answer with a JSON array. + OpenAiBackend noSlots = new FakeBackend() { + @Override + public String metrics() { + return "{\"idle\":1}"; + } + }; + try (OpenAiCompatServer server = new OpenAiCompatServer(noSlots, config()).start()) { + Response slots = get(server.getPort(), "/slots", ""); + assertThat(slots.code, is(200)); + assertThat(slots.body, is("[]")); + } + } + @Test public void oversizedRequestBodyRejectedWith413() throws IOException { OpenAiServerConfig cfg = OpenAiServerConfig.builder() @@ -438,7 +457,7 @@ public void metricsAndSlotsRequireApiKeyWhenConfigured() throws IOException { } /** Deterministic backend that returns canned OpenAI shapes for every operation. */ - static final class FakeBackend implements OpenAiBackend { + static class FakeBackend implements OpenAiBackend { @Override public String metrics() { return "{\"idle\":1,\"slots\":[{\"id\":0,\"n_prompt_tokens_cache\":8}]}"; diff --git a/llama/src/test/java/net/ladenthin/llama/value/ServerMetricsTest.java b/llama/src/test/java/net/ladenthin/llama/value/ServerMetricsTest.java index 415e0f1c4..3c18732a7 100644 --- a/llama/src/test/java/net/ladenthin/llama/value/ServerMetricsTest.java +++ b/llama/src/test/java/net/ladenthin/llama/value/ServerMetricsTest.java @@ -9,6 +9,7 @@ import static org.junit.jupiter.api.Assertions.assertTrue; import com.fasterxml.jackson.databind.ObjectMapper; +import java.util.List; import net.ladenthin.llama.ClaudeGenerated; import org.junit.jupiter.api.Test; @@ -29,6 +30,11 @@ private ServerMetrics parse(String json) throws Exception { + "\"n_prompt_tokens_processed\":10,\"t_prompt_processing\":5," + "\"n_tokens_predicted\":20,\"t_tokens_generation\":8," + "\"n_decode_total\":300,\"n_busy_slots_total\":4,\"n_tokens_max\":4096," + // Counters that only became reachable as JSON once the JNI layer started merging the + // METRICS and SLOT_GET halves back together (upstream split them at b10408/b10519). + + "\"n_prompt_tokens_cached_total\":700,\"n_draft_tokens_total\":40," + + "\"n_draft_accepted_total\":30,\"n_draft_verify_steps_total\":12," + + "\"n_draft_accepted_per_pos\":[20,8,2]," // next_token is an ARRAY of one object — this mirrors llama.cpp's server_slot::to_json // at b9739, not a bare object; SlotMetrics must unwrap next_token[0]. + "\"slots\":[{\"id\":0,\"n_ctx\":4096,\"is_processing\":true," @@ -166,4 +172,54 @@ public void toStringSerializesNode() throws Exception { // Assert content (not just non-null) so the empty-string return mutant on toString is killed. assertTrue(m.toString().contains("idle")); } + + @Test + public void cachedPromptTokenTotalIsSeparateFromTheProcessedTotal() throws Exception { + ServerMetrics m = parse(SAMPLE); + assertEquals(700L, m.getCumulativeCachedPromptTokens()); + // The processed total counts only tokens actually evaluated, so the two must not alias. + assertEquals(100L, m.getCumulativeProcessedPromptTokens()); + } + + @Test + public void speculativeDecodingCounters() throws Exception { + ServerMetrics m = parse(SAMPLE); + assertEquals(40L, m.getDraftTokensTotal()); + assertEquals(30L, m.getDraftAcceptedTotal()); + assertEquals(12L, m.getDraftVerifyStepsTotal()); + assertEquals(0.75, m.getDraftAcceptanceRate(), 1e-9); + } + + @Test + public void draftAcceptanceRateIsZeroWhenNothingWasDrafted() throws Exception { + // Pins the drafted > 0L guard: without it this divides by zero and yields NaN. + ServerMetrics m = parse("{\"n_draft_accepted_total\":5,\"n_draft_tokens_total\":0}"); + assertEquals(0.0, m.getDraftAcceptanceRate(), 1e-9); + } + + @Test + public void draftAcceptedPerPositionIsExposedInOrder() throws Exception { + List perPos = parse(SAMPLE).getDraftAcceptedPerPosition(); + assertEquals(3, perPos.size()); + assertEquals(20L, perPos.get(0).longValue()); + assertEquals(8L, perPos.get(1).longValue()); + assertEquals(2L, perPos.get(2).longValue()); + } + + @Test + public void draftAcceptedPerPositionIsEmptyWhenAbsentOrNotAnArray() throws Exception { + assertTrue(parse("{}").getDraftAcceptedPerPosition().isEmpty()); + assertTrue(parse("{\"n_draft_accepted_per_pos\":7}") + .getDraftAcceptedPerPosition() + .isEmpty()); + } + + @Test + public void newCountersDefaultToZeroWhenAbsent() throws Exception { + ServerMetrics m = parse("{}"); + assertEquals(0L, m.getCumulativeCachedPromptTokens()); + assertEquals(0L, m.getDraftTokensTotal()); + assertEquals(0L, m.getDraftAcceptedTotal()); + assertEquals(0L, m.getDraftVerifyStepsTotal()); + } } From e9794b0aa3af6654df1e5068773ebdff399c92ad Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 14:22:46 +0000 Subject: [PATCH 33/76] feat: give RouterClient an API key; fix hidden-model diagnosis RouterClient set only Content-Type, so it could not talk to a router started with --api-key at all. /models/load and /models/unload were always gated, and since b10519 (#26347 dropped /models and /v1/models from the server's public-endpoint set) listModels, findModel and awaitModelLoaded answer 401 there too -- the whole typed API was unusable against an authenticated router. New RouterClient(port, apiKey) / RouterClient(host, port, apiKey) constructors send Authorization: Bearer ; an empty key behaves exactly like none rather than sending "Bearer ", equals distinguishes credentials, and toString still renders only the target URL so the key never reaches a log. awaitModelLoaded's "does not list model" message sent callers to check --models-dir and the identifier, which is the wrong lead for a model hidden by a preset with dedup-cache-models: b10507 (#27346) omits such a cache model from GET /models while it still loads and still serves by name. Polling to the timeout instead -- the fix this was first written down as -- would not have worked, because upstream filters it out permanently, so no poll can ever observe it. The message now names that cause alongside the identifier one, and the javadoc documents the path that does work: issue the request directly and let the router resolve the name (with autoload it waits for the worker itself). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- .../ladenthin/llama/server/RouterClient.java | 79 +++++++++++++++---- .../llama/server/RouterClientTest.java | 67 +++++++++++++++- 2 files changed, 128 insertions(+), 18 deletions(-) diff --git a/llama/src/main/java/net/ladenthin/llama/server/RouterClient.java b/llama/src/main/java/net/ladenthin/llama/server/RouterClient.java index 5bb2be90c..dd1ebe29c 100644 --- a/llama/src/main/java/net/ladenthin/llama/server/RouterClient.java +++ b/llama/src/main/java/net/ladenthin/llama/server/RouterClient.java @@ -48,18 +48,19 @@ *

Instances are immutable and safe to share across threads; every call opens a short-lived * {@link HttpURLConnection}.

* - *

No API-key support. This client sends no {@code Authorization} header, so it - * only works against a router started without {@code --api-key}. {@code /models/load} and - * {@code /models/unload} have always required the key when one is configured; since llama.cpp - * b10519 (upstream #26347 dropped {@code /models} and {@code /v1/models} from the - * server's public-endpoint set) {@link #listModels()}, {@link #findModel(String)} and - * {@link #awaitModelLoaded(String, long)} answer {@code 401} on such a router too. Adding an - * optional key is tracked in {@code TODO.md}.

+ *

API keys. Pass one to the {@code apiKey} constructors and every request + * carries {@code Authorization: Bearer }. A router started with {@code --api-key} needs it for + * all of these calls: {@code /models/load} and {@code /models/unload} have always been + * gated, and since llama.cpp b10519 (upstream #26347 dropped {@code /models} and + * {@code /v1/models} from the server's public-endpoint set) {@link #listModels()}, + * {@link #findModel(String)} and {@link #awaitModelLoaded(String, long)} are gated too. The + * key-less constructors remain the right choice for a router started without one.

* - *

{@code equals}/{@code hashCode} are generated by Lombok over the {@code host}/{@code port} - * fields (two clients pointing at the same router compare equal). {@code toString} is - * intentionally handwritten (not Lombok-generated) so the client renders as its target URL, - * "{@code RouterClient(http://host:port)}", in log traces.

+ *

{@code equals}/{@code hashCode} are generated by Lombok over the {@code host}, {@code port} + * and {@code apiKey} fields (two clients addressing the same router with the same credential + * compare equal). {@code toString} is intentionally handwritten (not Lombok-generated) so the + * client renders as its target URL, "{@code RouterClient(http://host:port)}", in log traces — + * the key is never printed.

*/ @EqualsAndHashCode public final class RouterClient { @@ -72,27 +73,57 @@ public final class RouterClient { private static final RouterModelsResponseParser PARSER = new RouterModelsResponseParser(); + /** Loopback address used by the constructors that take no host. */ + private static final String DEFAULT_HOST = "127.0.0.1"; + private final String host; private final int port; + /** Bearer credential sent on every request; {@code null} for an unauthenticated router. */ + private final @Nullable String apiKey; + /** - * Client for a router on {@code 127.0.0.1}. + * Client for an unauthenticated router on {@code 127.0.0.1}. * * @param port the router's HTTP port */ public RouterClient(int port) { - this("127.0.0.1", port); + this(DEFAULT_HOST, port, null); } /** - * Client for a router on an arbitrary host. + * Client for an unauthenticated router on an arbitrary host. * * @param host the router's host name or address * @param port the router's HTTP port */ public RouterClient(String host, int port) { + this(host, port, null); + } + + /** + * Client for a router on {@code 127.0.0.1} started with {@code --api-key}. + * + * @param port the router's HTTP port + * @param apiKey the key to send as {@code Authorization: Bearer }; {@code null} or empty + * sends no header, which is only correct for a router started without {@code --api-key} + */ + public RouterClient(int port, @Nullable String apiKey) { + this(DEFAULT_HOST, port, apiKey); + } + + /** + * Client for a router on an arbitrary host, optionally authenticated. + * + * @param host the router's host name or address + * @param port the router's HTTP port + * @param apiKey the key to send as {@code Authorization: Bearer }; {@code null} or empty + * sends no header, which is only correct for a router started without {@code --api-key} + */ + public RouterClient(String host, int port, @Nullable String apiKey) { this.host = host; this.port = port; + this.apiKey = apiKey == null || apiKey.isEmpty() ? null : apiKey; } /** @@ -149,9 +180,18 @@ public void unloadModel(String modelId) throws IOException { /** * Poll {@link #listModels()} until {@code modelId} reaches * {@link RouterModel.Status#LOADED}, its worker is flagged as failed, or the timeout - * elapses. Fails fast (instead of running out the timeout) when the router stops listing + * elapses. Fails fast (instead of running out the timeout) when the router does not list * the model or marks it failed. * + *

Hidden models cannot be awaited here. Since llama.cpp b10507 (upstream + * #27346) the router omits a cache model from {@code GET /models} when a preset with + * {@code dedup-cache-models} already covers the same file — upstream's own comment is + * "hidden from GET /models, but still accept if requested". Such a model loads and serves + * normally but never appears in the listing, so no amount of polling can observe it and this + * method throws. Issue the request directly instead: the router resolves a hidden model by + * name, and with autoload enabled it waits for the worker before proxying. The thrown message + * names this cause so it is not mistaken for a wrong identifier.

+ * * @param modelId the model identifier to wait for * @param timeoutMillis how long to keep polling * @return the entry in its {@code LOADED} state @@ -166,8 +206,10 @@ public RouterModel awaitModelLoaded(String modelId, long timeoutMillis) throws I while (System.currentTimeMillis() < deadline) { Optional model = findModel(modelId); if (!model.isPresent()) { - throw new IllegalStateException( - "Router does not list model '" + modelId + "' — check --models-dir and the identifier"); + throw new IllegalStateException("Router does not list model '" + modelId + + "' — check --models-dir and the identifier, or, if the model exists, whether a preset " + + "with dedup-cache-models hides it from GET /models (a hidden model still loads and " + + "serves by name, but cannot be awaited through the listing)"); } last = model.get(); if (last.isFailed()) { @@ -208,6 +250,9 @@ private String request(String method, String path, @Nullable String body) throws connection.setRequestMethod(method); connection.setConnectTimeout(REQUEST_TIMEOUT_MILLIS); connection.setReadTimeout(REQUEST_TIMEOUT_MILLIS); + if (apiKey != null) { + connection.setRequestProperty("Authorization", "Bearer " + apiKey); + } if (body != null) { connection.setDoOutput(true); connection.setRequestProperty("Content-Type", "application/json"); diff --git a/llama/src/test/java/net/ladenthin/llama/server/RouterClientTest.java b/llama/src/test/java/net/ladenthin/llama/server/RouterClientTest.java index aaa5a5047..003ccde43 100644 --- a/llama/src/test/java/net/ladenthin/llama/server/RouterClientTest.java +++ b/llama/src/test/java/net/ladenthin/llama/server/RouterClientTest.java @@ -31,7 +31,7 @@ + "the upstream router wire format: list/find parsing, load/unload request " + "bodies, error surfacing with the router's error body, and the " + "awaitModelLoaded state machine (poll-until-loaded, fail-fast on failed " - + "worker or unknown model, timeout).") + + "worker or unknown model, timeout), and the optional API-key bearer header.") public class RouterClientTest { private HttpServer server; @@ -43,6 +43,9 @@ public class RouterClientTest { /** Counts GET /models calls so await tests can serve a status sequence. */ private final AtomicInteger modelsCalls = new AtomicInteger(); + /** Authorization header seen by the stub on the most recent request; "" when absent. */ + private final AtomicReference lastAuthHeader = new AtomicReference<>(""); + private final AtomicReference lastLoadBody = new AtomicReference<>(""); private final AtomicReference lastUnloadBody = new AtomicReference<>(""); @@ -50,6 +53,8 @@ public class RouterClientTest { public void startStub() throws IOException { server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0); server.createContext("/models", exchange -> { + String auth = exchange.getRequestHeaders().getFirst("Authorization"); + lastAuthHeader.set(auth == null ? "" : auth); String path = exchange.getRequestURI().getPath(); if ("/models/load".equals(path)) { lastLoadBody.set(readBody(exchange)); @@ -211,4 +216,64 @@ public void awaitModelLoaded_timesOutWithLastStatusInMessage() { assertThat(thrown.getMessage(), containsString("did not reach LOADED")); assertThat(thrown.getMessage(), containsString("loading")); } + + @Test + public void unlistedModelMessageNamesTheHiddenCacheModelCause() { + // A model hidden by a preset with dedup-cache-models (upstream #27346, b10507) never shows + // up in GET /models even though it loads and serves, so the message must not send the + // caller off to double-check --models-dir alone. + modelsBody.set("{\"data\":[]}"); + + IllegalStateException thrown = + assertThrows(IllegalStateException.class, () -> client.awaitModelLoaded("qwen", 30_000L)); + + assertThat(thrown.getMessage(), containsString("dedup-cache-models")); + assertThat(thrown.getMessage(), containsString("--models-dir")); + } + + @Test + public void noAuthorizationHeaderWithoutAnApiKey() throws IOException { + client.listModels(); + assertThat(lastAuthHeader.get(), is("")); + } + + @Test + public void apiKeyIsSentAsABearerTokenOnReads() throws IOException { + RouterClient authed = new RouterClient("127.0.0.1", server.getAddress().getPort(), "secret"); + authed.listModels(); + assertThat(lastAuthHeader.get(), is("Bearer secret")); + } + + @Test + public void apiKeyIsSentAsABearerTokenOnWrites() throws IOException { + RouterClient authed = new RouterClient(server.getAddress().getPort(), "secret"); + authed.loadModel("qwen"); + assertThat(lastAuthHeader.get(), is("Bearer secret")); + authed.unloadModel("qwen"); + assertThat(lastAuthHeader.get(), is("Bearer secret")); + } + + @Test + public void anEmptyApiKeySendsNoHeader() throws IOException { + // Distinguishes "" from a real credential: an empty key must behave exactly like none, + // not send "Bearer ". + new RouterClient("127.0.0.1", server.getAddress().getPort(), "").listModels(); + assertThat(lastAuthHeader.get(), is("")); + } + + @Test + public void clientsDifferingOnlyInApiKeyAreNotEqual() { + int port = server.getAddress().getPort(); + assertThat( + new RouterClient("127.0.0.1", port, "a").equals(new RouterClient("127.0.0.1", port, "b")), is(false)); + assertThat(new RouterClient("127.0.0.1", port, "a").equals(new RouterClient("127.0.0.1", port, "a")), is(true)); + assertThat(new RouterClient(port).equals(new RouterClient("127.0.0.1", port)), is(true)); + } + + @Test + public void toStringNeverLeaksTheApiKey() { + assertThat( + new RouterClient("127.0.0.1", 8080, "super-secret").toString(), + is("RouterClient(http://127.0.0.1:8080)")); + } } From 8e5147965b33fe777be735c463fc3165040c95c3 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 14:22:58 +0000 Subject: [PATCH 34/76] docs: record the five closed gaps and drop their TODO entries CLAUDE.md: the patch applier is described by what it now does (stamp file plus git clean/dirty state, and why a per-patch reverse-check cannot work); the CI model policy explains the working-directory mismatch and the resolver that fixes it instead of carrying a "known broken" note; a new section documents the getMetrics() merge, its microsecond-to-millisecond conversion and the counters it newly exposes; the RouterClient sentence covers the API key and the hidden model; json_helpers gains server_metrics_to_json and the C++ test counts move 491 -> 499. The common_json vocab_type guard note no longer says the LlamaModelTest assertion runs only locally -- it runs in CI now. README: RouterClient's authenticated construction plus a note that awaitModelLoaded cannot observe a hidden model, and the new ServerMetrics counters. The bump runbook's "use a fresh build dir" step keeps the instruction but gives the real reason: after a GIT_TAG change the stamp pins the previous commit, so an existing build dir is exactly the case the applier refuses to guess at. TODO.md: the five entries move from Open to Done with one line each on what was actually done, including the correction that the hidden-model item's first suggested fix would not have worked. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 75 ++++++++++--- README.md | 21 +++- TODO.md | 143 ++++++------------------- docs/upgrade/llama-cpp-version-bump.md | 6 +- 4 files changed, 122 insertions(+), 123 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 4536ff6b2..f6031dc63 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -641,12 +641,20 @@ The fetched llama.cpp source is patched before it compiles, via a generic mechan ordered. Each must be a `git apply`-compatible unified diff with paths relative to the llama.cpp source root (`a/common/arg.cpp` / `b/common/arg.cpp`, i.e. `-p1`). - **`llama/cmake/apply-llama-patches.cmake`** — the applier. Cross-platform (`cmake -P`, so identical on - Linux/macOS/Windows), **idempotent only while no two patches touch the same file** - (`git apply --reverse --check` skips an already-applied patch — but `0006`/`0007` rewrite the - region of `tools/server/server.cpp` that `0001` also patches, so `0001`'s reverse-check no longer - matches and a **reconfigure over an already-patched tree aborts**; always configure into a fresh - build dir, and see the entry in [`TODO.md`](TODO.md)) and **fail-loud** (a patch that no longer - applies aborts the configure — a stale patch can't be silently dropped from a release build). + Linux/macOS/Windows), **idempotent** and **fail-loud** (a patch that no longer applies aborts the + configure — a stale patch can't be silently dropped from a release build). Idempotency comes from + a **stamp file** (`/.jllama-patches-applied`, recording the checked-out llama.cpp + commit plus each patch's SHA-256) combined with git's clean/dirty state, not from per-patch + probing: a **clean** source tree means nothing is applied yet (fresh fetch, or a re-checkout after + a version bump) so everything is applied forward and the stamp written; a **dirty** tree is + already patched, and the reconfigure is a no-op when the stamp matches this exact commit + patch + set, or aborts with a "configure into a fresh build directory" message when it does not. + A per-patch `git apply --reverse --check` cannot do this — `--check` never mutates the tree, so an + earlier patch whose region a later one rewrote (`0001` vs `0006`/`0007` in + `tools/server/server.cpp`) always reverse-checks as "not applied", and the forward re-apply then + aborted every reconfigure of an existing build dir with a misleading "does not apply cleanly". + A source tree supplied via `-DFETCHCONTENT_SOURCE_DIR_LLAMA.CPP=` that is not a git work + tree has neither oracle and falls back to the old per-patch path (same caveat as before). - **`llama/CMakeLists.txt`** — wired as the llama.cpp `FetchContent_Declare(... PATCH_COMMAND ...)`, so it runs for **every** C++ build (all CI jobs *and* local `cmake -B build`) from one place — no per-build-step plumbing. @@ -1134,7 +1142,7 @@ If the local check passes (`BUILD SUCCESS`), the `mvn package` job in - `json_helpers.hpp` — Pure JSON transformation helpers (no JNI, no llama state). Independently unit-testable. - `jni_helpers.hpp` — JNI bridge helpers (handle management + server orchestration). Includes `json_helpers.hpp`. - **The `json` alias is upstream's `common_json`, not `nlohmann::ordered_json` (since llama.cpp b10585, upstream #27511).** `tools/server/server-common.h` now says `using json = common_json;` — a deliberately small pimpl wrapper (`common/json.{h,cpp}`, compiled into `llama-common`) around the vendored nlohmann copy. Two traps this cost the project once, both of which **compile silently**: - 1. **An unscoped enum becomes a JSON boolean.** `common_json_value`'s integral constructor template is `std::is_integral`-gated, which excludes enums, so an enum binds to `common_json_value(bool)`. Always `static_cast(...)` an enum before putting it in JSON — `jllama.cpp`'s two `"vocab_type"` sites do, and `patches/0010` does the same for upstream's own `/models` handler. Guards: `test_json_helpers.cpp`'s `CommonJsonEnumTrap` pair pins the mechanism and **does** run in CI; `LlamaModelTest`'s `isIntegralNumber()` assertion pins the real wire value but is model-gated, so it currently only fires locally with a GGUF present (see the CI-skip entry in [`TODO.md`](TODO.md)). + 1. **An unscoped enum becomes a JSON boolean.** `common_json_value`'s integral constructor template is `std::is_integral`-gated, which excludes enums, so an enum binds to `common_json_value(bool)`. Always `static_cast(...)` an enum before putting it in JSON — `jllama.cpp`'s two `"vocab_type"` sites do, and `patches/0010` does the same for upstream's own `/models` handler. Guards: `test_json_helpers.cpp`'s `CommonJsonEnumTrap` pair pins the mechanism and **does** run in CI; `LlamaModelTest`'s `isIntegralNumber()` assertion pins the real wire value and now runs in CI too (the model-gated suite no longer self-skips — see "CI model policy" below). 2. **`common_json` converts to `std::string` implicitly**, so it binds happily to a `const nlohmann::json &` parameter (via nlohmann's string-constructible converting constructor) and then throws `json::type_error 302` at runtime. Never declare a project helper as taking `nlohmann::json` when callers pass the `json` alias — `require_json_field_impl` is a template for exactly this reason. Other differences to know: no `get_ref`/`array_t`/`type_name()`; a braced list in *value* position does not build an array (write `json::array({...})`); `at(key)` needs an explicit `.get()`; errors are `common_json_error`; and `get()` is limited to the types explicitly specialised in `common/json.cpp`. `log_helpers.hpp` and `train_engine.cpp` keep their own `nlohmann::json` alias — they never touch the server's `json`. - Uses `nlohmann/json` for JSON deserialization of parameters in the two files named above; everything on the server path uses `common_json`. @@ -1145,7 +1153,30 @@ If the local check passes (`BUILD SUCCESS`), the `mvn package` job in The library exposes **two** ways to serve a model over HTTP, on two different transports. The fat jar's `Main-Class` is `server.ServerLauncher`, a tiny dispatcher: it runs `OpenAiCompatServer` when `--jllama-openai-compat` is present (that marker is stripped, the rest forwarded) and the default `NativeServer` otherwise. Both mains are also runnable directly by class name via `java -cp`. The two modes: 1. **`server.OpenAiCompatServer` (Java transport).** OpenAI/Ollama/Anthropic-compatible JSON API on the JDK's `com.sun.net.httpserver`, driving the compiled server *core* over JNI. Embeddable, no extra dependency, and it can share/reuse a `LlamaModel`. It serves **no** static assets — its `/` route is a 404, so **no WebUI**. It has its own `main` (run via `java -cp net.ladenthin.llama.server.OpenAiCompatServer …`); its CLI (`OpenAiServerCli`) maps a curated flag subset (`-m/-c/-b/-ub/-ngl/-t/-tb/-ctk/-ctv/--jinja/--chat-template-kwargs/--host/--port/--parallel/--mmproj/--api-key/--embedding/--reranking`). -2. **`server.NativeServer` (native transport) — the default fat-jar server (when `--jllama-openai-compat` is absent).** Runs the **full upstream `llama_server`** (via `patches/0006` + `native_server.cpp`) inside `libjllama`, forwarding the raw llama-server argv verbatim — so **every** llama-server flag works and the **embedded WebUI is served** (when the assets are compiled in; CI's released jars have them, local `cmake` builds use the empty-asset stub). With the classic constructor it is an **independent lifecycle** (loads its own model from the argv, like `llama-server.exe`; owns the process's llama backend + stderr logging while running); the **attach constructor** (`NativeServer(LlamaModel, String...)`, via `patches/0007`'s `llama_server_attach`) instead serves an **already-loaded `LlamaModel`** — one copy of the weights, the model's worker keeps driving inference, the HTTP routes post to its queue; caller closes the server before the model. **Router mode** (start without a model argument: `--models-dir`, `GET/POST /models`, per-request model selection) works in-JVM after `NativeServer.setWorkerCommand(...)` redirects the worker spawn to a fresh JVM (`patches/0008` — upstream re-execs its own binary, which in a JVM is `java`); the typed `server.RouterClient` (+ `value.RouterModel`, `json.RouterModelsResponseParser`) wraps the model-management endpoints (list/load/unload/await-loaded with fail-fast on failed workers) so callers don't hand-roll HTTP+JSON. Either way it is **single-instance per process** (upstream keeps shutdown state in file-scope globals) and **not available on Android** (the `subprocess.h` guard). `libjllama` loading anywhere a JVM runs is what makes this "no separate `llama-server.exe`" possible. +2. **`server.NativeServer` (native transport) — the default fat-jar server (when `--jllama-openai-compat` is absent).** Runs the **full upstream `llama_server`** (via `patches/0006` + `native_server.cpp`) inside `libjllama`, forwarding the raw llama-server argv verbatim — so **every** llama-server flag works and the **embedded WebUI is served** (when the assets are compiled in; CI's released jars have them, local `cmake` builds use the empty-asset stub). With the classic constructor it is an **independent lifecycle** (loads its own model from the argv, like `llama-server.exe`; owns the process's llama backend + stderr logging while running); the **attach constructor** (`NativeServer(LlamaModel, String...)`, via `patches/0007`'s `llama_server_attach`) instead serves an **already-loaded `LlamaModel`** — one copy of the weights, the model's worker keeps driving inference, the HTTP routes post to its queue; caller closes the server before the model. **Router mode** (start without a model argument: `--models-dir`, `GET/POST /models`, per-request model selection) works in-JVM after `NativeServer.setWorkerCommand(...)` redirects the worker spawn to a fresh JVM (`patches/0008` — upstream re-execs its own binary, which in a JVM is `java`); the typed `server.RouterClient` (+ `value.RouterModel`, `json.RouterModelsResponseParser`) wraps the model-management endpoints (list/load/unload/await-loaded with fail-fast on failed workers) so callers don't hand-roll HTTP+JSON, and its `apiKey` constructors send `Authorization: Bearer ` — required for **every** one of those calls against a router started with `--api-key` since b10519 (#26347 dropped `/models` + `/v1/models` from the public-endpoint set; `/models/load` and `/models/unload` were always gated). `awaitModelLoaded` cannot observe a model hidden by a preset with `dedup-cache-models` (b10507/#27346 omits it from `GET /models` although it still loads and serves by name), so its "not listed" message names that cause explicitly; such a model is reached by issuing the request directly instead. Either way it is **single-instance per process** (upstream keeps shutdown state in file-scope globals) and **not available on Android** (the `subprocess.h` guard). `libjllama` loading anywhere a JVM runs is what makes this "no separate `llama-server.exe`" possible. + +### `getMetrics()` — one object rebuilt from two upstream tasks + +`LlamaModel.getMetrics()` / `getMetricsTyped()` return the single server-introspection object the +Java side has always documented: `idle` / `processing` / `deferred` / `t_start`, the cumulative and +current-window `n_*` / `t_*` counter pairs, and a `slots` array. Upstream stopped emitting that in +one piece — **b10408** (#26920) reduced `server_task_result_metrics::to_json()` to the slot array, +and **b10519** (#27376) split the task in two: `SERVER_TASK_TYPE_METRICS` keeps only the counters +(its `to_json()` is unused and returns JSON null; `to_metrics()` renders them as Prometheus text) +while `SERVER_TASK_TYPE_SLOT_GET` carries the slot array plus the idle-slot count. + +`handleSlotAction(0, …)` therefore posts **both** tasks and merges the results through the pure +helper `server_metrics_to_json` (`json_helpers.hpp`, unit-tested in `test_json_helpers.cpp`), rather +than letting the Java contract follow upstream's transport split. Durations are converted from +upstream microseconds to the milliseconds the payload has always used. The merge also surfaces the +counters upstream added since — `n_prompt_tokens_cached_total` and the speculative-decoding tallies +(`n_draft_tokens_total`, `n_draft_accepted_total`, `n_draft_verify_steps_total`, +`n_draft_accepted_per_pos`) — which previously existed only inside the Prometheus text and had no +JSON representation at all; `value.ServerMetrics` exposes them with typed getters (plus a derived +`getDraftAcceptanceRate()`). No second JNI entry point and no Prometheus-text parser were needed. + +The metrics task is posted with `server_task::metrics_reset_bucket` left at its default `false`, so +`getMetrics()` never resets the current-measurement window; only an HTTP `/metrics` scrape does. ### Native Helper Architecture @@ -1162,7 +1193,8 @@ The project C++ helpers follow a strict semantic split: Functions: `get_result_error_message`, `results_to_json`, `rerank_results_to_json`, `parse_encoding_format`, `extract_embedding_prompt`, `is_infill_request`, -`parse_slot_prompt_similarity`, `parse_positive_int_config`, `wrap_stream_chunk`. +`parse_slot_prompt_similarity`, `parse_positive_int_config`, `wrap_stream_chunk`, +`server_metrics_to_json`. **`log_helpers.hpp`** — Pure log-formatting transforms. - Input: `ggml_log_level`, message text (`const char*`), an explicit `std::time_t` timestamp. @@ -1294,7 +1326,26 @@ model + mmproj, and the Qwen3-TTS backbone + mmproj (`ggml-org/Qwen3-TTS-12Hz-1. smallest available quants: `Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf` backbone + `mmproj-Qwen3-TTS-12Hz-1.7B-Base-Q8_0.gguf` mmproj — no smaller mmproj quant is published), with their `-Dnet.ladenthin.llama.*` properties set, so `LlamaEmbeddingsTest`, `MultimodalIntegrationTest`, -and `TtsIntegrationTest` are **intended** to run on every platform rather than self-skipping. **Known broken as of the b10618 bump:** Surefire's working directory is the `llama/` module while CI restores the cache to `/models/`, so every model-gated test resolves `models/…` to a path that does not exist and aborts in `@BeforeAll` — they self-skip on *every* `test-java-*` job while the job still reports success. See "Model-gated Java tests silently self-skip in CI" in [`TODO.md`](TODO.md). `validate-models.{sh,bat}` +and `TtsIntegrationTest` are **intended** to run on every platform rather than self-skipping. + +**How the paths resolve (this was silently broken until it was fixed after the b10618 bump).** +Surefire's working directory defaults to the **module** basedir (`/llama`), while the +shared GGUF cache is restored to `/models/` and every model path — the `TestConstants` +constants and the `-Dnet.ladenthin.llama.*` properties alike — is stated relative as `models/…`. +Those therefore resolved to `/llama/models/…`, which does not exist: **every** +model-gated class aborted in its `@BeforeAll` `Assumptions.assumeTrue(file.exists())` and reported +as *skipped* while the job still went green, on every `test-java-*` job. It is why several stale +assertions (e.g. `LlamaModelTest#testGetMetrics` against a payload shape upstream had dropped at +b10408) never failed in CI. The fix is **`TestConstants.resolveModelPath` / +`resolveModelProperty`**, which accept either layout — module-relative first, then the reactor root +— so a developer with models under `llama/models/` and CI with them at the workspace root both +work, with no workflow change. Every `TestConstants` path constant is routed through it, as is +every `-Dnet.ladenthin.llama.*` fixture property; `TestConstantsTest` pins both the resolver and +the wiring (a future edit that drops the wrapper from a constant fails that test rather than +silently re-muting the suite). `llama-langchain4j` had the identical defect and carries the same +resolver as `TestModelPaths` (test classes are not shared between modules). + +`validate-models.{sh,bat}` treats all of these as **required** (a missing model hard-fails the job before tests run, so a download regression can never silently downgrade to a skip). Only the audio-input model (`AudioInputIntegrationTest`) still self-skips — the prompt clip is committed @@ -1357,12 +1408,12 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" |------|-------|-------| | `src/test/cpp/test_utils.cpp` | 162 | Upstream helpers: `server_tokens`, `server_grammar_trigger`, `gen_tool_call_id`, `json_value`, `json_get_nested_values`, UTF-8 helpers, `format_response_rerank`, `format_embeddings_response_oaicompat`, `oaicompat_completion_params_parse`, `oaicompat_chat_params_parse`, `are_lora_equal`, `strip_flag_from_argv`, `token_piece_value`, `json_is_array_and_contains_numbers`, `format_oai_sse`, `format_oai_resp_sse`, `format_anthropic_sse`, `parse_lora_request` | | `src/test/cpp/test_server.cpp` | 206 | Upstream result types: `server_slot_stats` (the `timings` JSON payload; replaced `result_timings` in b10408), `task_params::to_json()` (incl. `dry_sequence_breakers`, `preserved_tokens`, `timings_per_token`), `completion_token_output`, `server_task_result_cmpl_partial` (non-oaicompat + `to_json_oaicompat` + logprobs + `to_json_oaicompat_chat` + `to_json_anthropic` + dispatcher), `server_task_result_cmpl_final` (non-oaicompat + `to_json_oaicompat` + `to_json_oaicompat_chat` + `to_json_oaicompat_chat_stream` + `to_json_anthropic` + `to_json_anthropic_stream` + tool_calls + dispatcher), `server_task_result_embd`, `server_task_result_rerank`, `server_task_result_metrics` (`to_metrics()` = the `/metrics` Prometheus exposition text; its `to_json()` has been unused since b10519 and returns `json{}` = JSON null), `server_task_result_slots` (`to_json()` = the `/slots` array, fed by the b10519 `SERVER_TASK_TYPE_SLOT_GET` task), `server_task_result_slot_save_load`, `server_task_result_slot_erase`, `server_task_result_apply_lora`, `server_task_result_get_lora`, `server_task_result_error`, `format_error_response`, `server_task::need_sampling()`, `server_task::n_tokens()`, `server_schema::eval_llama_cmpl_schema()` (parsing pipeline + grammar routing + error paths + per-request `dry_*` and `sse_ping_interval` field round-trips incl. hard-limit + server-default inheritance), `response_fields` projection | -| `src/test/cpp/test_json_helpers.cpp` | 52 | All functions in `json_helpers.hpp`: `get_result_error_message`, `results_to_json`, `rerank_results_to_json` (incl. missing/out-of-range `index` rejection), `parse_encoding_format`, `extract_embedding_prompt`, `is_infill_request`, `parse_slot_prompt_similarity`, `parse_positive_int_config`, `wrap_stream_chunk` | +| `src/test/cpp/test_json_helpers.cpp` | 60 | All functions in `json_helpers.hpp`: `get_result_error_message`, `results_to_json`, `rerank_results_to_json` (incl. missing/out-of-range `index` rejection), `parse_encoding_format`, `extract_embedding_prompt`, `is_infill_request`, `parse_slot_prompt_similarity`, `parse_positive_int_config`, `wrap_stream_chunk`, `server_metrics_to_json` | | `src/test/cpp/test_log_helpers.cpp` | 13 | All functions in `log_helpers.hpp`: `log_level_name`, `format_log_as_json` | | `src/test/cpp/test_jni_helpers.cpp` | 56 | All functions in `jni_helpers.hpp` using a zero-filled `JNINativeInterface_` mock (incl. the `utf8_to_jstring_impl` byte-array string path: emoji byte-preservation, truncated-UTF-8 replace-not-throw) | | `src/test/cpp/test_tts_wav.cpp` | 2 | The in-memory WAV writer `pcm_to_wav16_bytes` in `tts_wav.hpp` (WAV header/payload + little-endian clamping) — our own code, not upstream. The Qwen3-TTS pipeline it pairs with (`mtmd_helper::gen_audio`) is entirely upstream-owned (no project-side DSP to unit-test here) and covered end-to-end by the Java `TtsIntegrationTest`. | -**Current total: 491 tests (all passing).** +**Current total: 499 tests (all passing).** #### Upstream source location (in CMake build tree) diff --git a/README.md b/README.md index 2ad164d88..4bfd0abff 100644 --- a/README.md +++ b/README.md @@ -694,7 +694,11 @@ request, so generation and `save`/`restore` operate on the same KV state. Typed results expose logical prompt, generated, cached prompt, and evaluated prompt counts through `Usage`. Per-request timing also remains available through `Timings.getCacheN()`. `LlamaModel.getMetricsTyped().getSlotMetrics()` reports each slot's logical, processed, cached, -decoded, and remaining token counts. +decoded, and remaining token counts, and the same `ServerMetrics` view carries the server-wide +lifetime counters — including cached prompt tokens (`getCumulativeCachedPromptTokens()`) and the +speculative-decoding tallies (`getDraftTokensTotal()`, `getDraftAcceptedTotal()`, +`getDraftVerifyStepsTotal()`, `getDraftAcceptedPerPosition()`, plus the derived +`getDraftAcceptanceRate()`), which upstream otherwise exposes only as Prometheus text. The embedded HTTP server exposes the same native JSON at authenticated `GET /metrics`, with the slot array alone at `GET /slots`. OpenAI responses preserve @@ -923,6 +927,21 @@ client.unloadModel("Qwen3-0.6B-Q4_K_M"); // POST /models/unload failed-worker marker. Chat requests then select a model per request via the standard `"model"` field on `POST /v1/chat/completions`. +Against a router started with `--api-key`, pass the key — it is sent as +`Authorization: Bearer ` on every call. All of them need it: `/models/load` and +`/models/unload` were always gated, and since llama.cpp b10519 the listing endpoints are too. + +```java +RouterClient client = new RouterClient(8080, System.getenv("LLAMA_API_KEY")); +// or, for a remote router: new RouterClient("router.internal", 8080, key) +``` + +> [!NOTE] +> `awaitModelLoaded` waits by polling `GET /models`, so it cannot observe a model the router +> deliberately hides from that listing — a cache model deduplicated by a preset with +> `dedup-cache-models` still loads and still serves by name, but never appears. For those, skip the +> await and issue the request directly; with autoload the router waits for the worker itself. + ### LangChain4j integration A separate artifact, **`net.ladenthin:llama-langchain4j`**, adapts a `LlamaModel` to diff --git a/TODO.md b/TODO.md index f701b1b9c..95147d225 100644 --- a/TODO.md +++ b/TODO.md @@ -13,114 +13,6 @@ cross-cutting initiative. ## Open — jllama-specific -### Model-gated Java tests silently self-skip in CI (Surefire's working directory) - -**Found while bumping to b10618; pre-existing and independent of that bump.** Every model-gated Java -test decides whether to run with a *relative* path — `new File("models/codellama-7b.Q2_K.gguf")` -(`LlamaModelTest`), `TestConstants.MODEL_PATH`, the `-Dnet.ladenthin.llama.*` model properties CI -passes as `models/`. Surefire resolves those against its working directory, which defaults to -the **module** basedir. CI runs `mvn -f llama/pom.xml … test` while the shared GGUF cache is restored -to `/models/`, so the tests look in `/llama/models/` and find nothing. - -Verified empirically, not inferred: a throwaway test run under `mvn -f llama/pom.xml` reported -`user.dir=/llama`, `models/ exists=false`, `models/ -exists=true`. - -Consequence: the whole model-backed Java suite — `LlamaModelTest`, `LlamaEmbeddingsTest`, -`MultimodalIntegrationTest`, `TtsIntegrationTest`, the reranking/draft/tool-model tests — aborts in -`@BeforeAll` and reports as skipped on every `test-java-*` job, while the job still goes green. That -contradicts the "CI model policy" section of `CLAUDE.md`, which states these run on every platform, -and it is why the two stale assertions below have never failed. - -Fixing it is a one-line change (a Surefire `${project.basedir}/..`, -or making the paths module-relative) — but it will **immediately turn ~100 currently-skipped tests -on**, including the known-stale ones, so it must be done as its own change with the fallout fixed in -the same PR, not folded into an unrelated bump. - -### The `getMetrics()` payload contract drifted at b10408 and three consumers still assume the old one - -`LlamaModel.getMetrics()` returns whatever the native slot-introspection task's `to_json()` produces. -Until llama.cpp **b10408** (upstream #26920) that was an object — `idle` / `processing` / `deferred` / -`t_start` / `n_*_total` counters plus a nested `slots` array. Since b10408 it is the **slot array -verbatim**; the cumulative counters moved to Prometheus exposition text behind `to_metrics()`, which -the JNI layer does not expose at all. Three consumers were written against the old shape and are all -still on it: - -- **`value.ServerMetrics`** — `getIdleSlots()`, `getProcessingSlots()`, `getDeferredTasks()`, - `getCumulativeUsage()`, `getSlots()`, `getSlotMetrics()` all read keys that no longer exist, so - they return zeros / a missing node. Only `asJson()` (the raw array) carries real data. - `ServerMetricsTest` builds its input from string literals, so it passes either way and is not a - guard. -- **`LlamaModelTest#testGetMetrics`** — asserts the live payload `contains("\"slots\"")` and - `contains("\"idle\"")`. Neither substring occurs anywhere in the current payload (checked against - upstream's `server_slot::to_json` and `task_params::to_json` key lists). It does not fail today - only because of the Surefire working-directory issue above; fixing that surfaces this immediately. -- **`server.OpenAiCompatServer.handleMetricsView`** — `GET /slots` does - `readTree(metrics).path("slots").toString()`, and `path()` on an array node is a `MissingNode` - whose `toString()` is the empty string, so the route answers **200 with a zero-length body**. Its - sibling `GET /metrics` serves the slot array under a name that no longer describes it. - `OpenAiCompatServerHttpTest`'s fake backend returns the *pre-b10408* object, so the test exercises - a shape production never produces — the fake is what hides the bug. - -**Not a regression of the b10456 → b10618 walk.** That walk only re-pointed the JNI task at the new -`SERVER_TASK_TYPE_SLOT_GET`, which is what keeps the array arriving at all (b10519 split the old -`METRICS` task and would otherwise have degraded the payload to JSON null). - -Fixing it means picking the Java-side contract first: either reshape `ServerMetrics` (and `/slots`) -around the slot array and drop the counter getters, or add a second JNI entry point for -`SERVER_TASK_TYPE_METRICS` → `to_metrics()` and serve/parse the Prometheus text. Whichever is chosen, -update all three consumers, the `OpenAiCompatServerHttpTest` fake, and the `/metrics` route naming -together. - -### `RouterClient` has no API-key support, and `/models` stopped being a public endpoint at b10519 - -`server.RouterClient` sets only `Content-Type` on its requests — there is no `Authorization` header -and no constructor parameter for a key. `POST /models/load` and `/models/unload` have always gone -through the server's API-key middleware, so those two already failed against a router started with -`--api-key`. Upstream **#26347** (b10519) additionally removed `/models` and `/v1/models` from the -public-endpoint set, so `listModels()`, `findModel()` and `awaitModelLoaded()` now answer `401` -there as well — the whole typed router API is unusable against an authenticated router. - -Found while bumping to b10618; the upstream change is noted in the `b10509–b10519` row of -`docs/history/llama-cpp-breaking-changes.md`. CI is unaffected because `RouterModeIntegrationTest` -starts its router without a key, which is also why nothing caught it. - -Fix: add an optional key — e.g. a `RouterClient(String host, int port, String apiKey)` overload that -sets `Authorization: Bearer ` in the private `request()` helper. Deliberately not done as part -of the version bump: it adds public API surface, which is the owner's design call. The limitation is -documented in `RouterClient`'s class javadoc in the meantime. - -### `RouterClient.awaitModelLoaded` rejects hidden-but-loadable router models - -b10507 (upstream #27346) added `server_model_meta::hidden` and a `continue` in the `GET /models` -handler; upstream's own comment is "hidden from GET /models, but still accept if requested" — a -hidden model still loads and still serves by name. `awaitModelLoaded` treats absence from the -listing as a hard error and fails fast, so `loadModel(id)` succeeds, the worker comes up, and -`awaitModelLoaded(id)` throws a message that sends the user off to check `--models-dir` and the -identifier. Only reachable when a preset opts in with `dedup-cache-models`, which the project's own -code never writes — but `NativeServer` forwards raw llama-server argv verbatim by design, so a -caller can enable it. - -Fix: either retry until the timeout instead of rejecting on the first poll (a hidden model still -reaches `LOADED`), or widen the error message to name `dedup-cache-models` as a cause. - -### `apply-llama-patches.cmake` is not idempotent when two patches touch one file - -`CLAUDE.md` and the applier's own header describe it as idempotent: a `git apply --reverse --check` -is supposed to detect an already-applied patch and skip it, so a CMake **re**configure over an -already-patched source tree is a no-op. That holds only while no two patches touch the same file. -`0001` patches `tools/server/server.cpp` (one of its ~34 `common_params_parse_main` call-site flips) -and `0006`/`0007` then rewrite the same region, so `0001`'s reverse-check no longer matches; the -applier falls through to a forward apply, which also fails, and aborts the configure with the -misleading "does not apply cleanly — a llama.cpp version bump probably shifted the patched code". - -Confirmed at b10618: reverse-check `0001` → fails at `tools/server/server.cpp:102`; `0006` and -`0007` → skip cleanly. The practical effect is that **any** reconfigure of an existing build dir -fails, which is why the runbook says to use a fresh one. A fix would be to reverse-check in reverse -filename order (`0008` → `0001`), or to record an applied-marker per patch in the build tree. Left -alone for now because it is release-critical build machinery and the fresh-build-dir workaround is -already documented. - ### LlamaLoader extraction-directory isolation (optional follow-up, low priority) Left over from the 2026-06-20 code audit (18/18 findings fixed in PRs #258/#260, regression tests in @@ -364,6 +256,41 @@ and have only run locally so far. ## Done (kept for history) +### 2026-08-25 — the five gaps recorded during the b10618 bump, now fixed + +All five were found while bumping llama.cpp to b10618, recorded there as diagnoses rather than fixes +(the bump commit had to stay a bump), and closed in a follow-up. Details in CLAUDE.md; one-liners: + +- **Model-gated Java tests silently self-skipped in CI** — Surefire's working directory is the + module basedir while the GGUF cache is restored to the reactor root, so every `models/…` path + resolved to nothing, every model-gated class aborted in `@BeforeAll`, and the job still went green. + Fixed with `TestConstants.resolveModelPath` / `resolveModelProperty` (accept either layout), routed + through every path constant and every `-Dnet.ladenthin.llama.*` fixture property, plus + `TestConstantsTest` pinning the resolver **and** the wiring. `llama-langchain4j` had the same + defect and got the same resolver as `TestModelPaths`. Verified end-to-end: with a placeholder at + `/models/`, `LlamaModelTest` reports `Skipped: 0` and actually attempts the load. +- **`getMetrics()` payload contract drifted at b10408** — restored in the native layer instead of + bending the Java contract: `handleSlotAction(0, …)` now posts both `SERVER_TASK_TYPE_METRICS` and + `SERVER_TASK_TYPE_SLOT_GET` and merges them via the pure `server_metrics_to_json`. All three + consumers keep working unchanged; `value.ServerMetrics` additionally exposes the cache and + speculative-decoding counters that previously existed only in the Prometheus text. + `LlamaModelTest#testGetMetrics` now asserts the parsed shape rather than substrings (the old + assertion was satisfiable by the slot entries themselves), and `GET /slots` answers `[]` instead of + an empty body when the payload carries no `slots` key. +- **`RouterClient` had no API-key support** — added `RouterClient(port, apiKey)` / + `RouterClient(host, port, apiKey)` sending `Authorization: Bearer `; an empty key behaves like + none, `toString()` never prints it, `equals` includes it. +- **`RouterClient.awaitModelLoaded` vs hidden router models** — the TODO's first suggestion (poll to + the timeout) was wrong: upstream filters hidden models out of `GET /models` permanently, so no + amount of polling observes one. Fixed the honest way — the "not listed" message now names the + `dedup-cache-models` cause and the javadoc documents the direct-request path. +- **`apply-llama-patches.cmake` was not idempotent** — replaced per-patch reverse-checking with a + stamp file (llama.cpp commit + per-patch SHA-256) gated on git's clean/dirty state. A reconfigure + over a patched tree is now a no-op; a genuine mismatch fails with an accurate message instead of a + misleading "does not apply cleanly". Verified against the real build tree and a purpose-built + two-patches-one-file fixture that reproduces the old failure. + + ### 2026-07-05 feature wave (PR #298) + follow-ups One-liners for the sections removed from "Open" (full detail: PR #298, CLAUDE.md, git history): diff --git a/docs/upgrade/llama-cpp-version-bump.md b/docs/upgrade/llama-cpp-version-bump.md index 6babecedd..a8269d2b2 100644 --- a/docs/upgrade/llama-cpp-version-bump.md +++ b/docs/upgrade/llama-cpp-version-bump.md @@ -124,8 +124,10 @@ Concretely: `*.patch` in `llama/patches/`** must still apply. Do not maintain a list of them here or anywhere else: `apply-llama-patches.cmake` `file(GLOB)`s the directory and applies them in filename order, so an enumeration can only go stale (it did, one commit after being written). Use a **fresh** - build dir (a stale one re-applies over an already-patched tree and reports a false "does not - apply" — see the applier note in `TODO.md` for why): + build dir: the applier's stamp file pins the patch set to the *checked-out llama.cpp commit*, so + after a `GIT_TAG` change an existing build dir is exactly the case it refuses to guess at — it + aborts and tells you to configure fresh, which is what actually re-runs the patches against the + new source: ```bash cd llama && mvn -q compile # generates the OSInfo class CMake's OS-detection needs rm -rf build && cmake -B build # fail-loud: aborts here if any patch no longer applies From 2dc248e28dbf33afad6bc702acab973b8e5bb262 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 15:11:05 +0000 Subject: [PATCH 35/76] docs: record the metrics-contract restoration in bump history The b10405-b10408 and b10509-b10519 rows describe the upstream metrics refactor accurately but stop at the transport: neither says that getMetrics() had been returning the bare slot array to its three Java consumers since before this bump range started, nor that re-pointing handleSlotAction at SLOT_GET kept the array arriving without restoring the object. A follow-up row now states that, records the two-task merge that fixed it, and names why the walk could not have caught it -- no signature changed, every chunk compiled and linked clean, and the only guard was a model-gated test that CI silently skipped. Also refreshes the PIT figure (295 -> 304 after the new ServerMetrics counters) and notes that net.ladenthin.llama.value.* sits under the 100 % mutation threshold, so a new getter on a value type needs its own test. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 6 ++++-- docs/history/llama-cpp-breaking-changes.md | 1 + 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index f6031dc63..6a730b7a8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1606,8 +1606,10 @@ See [`../workspace/policies/pit-mutation-testing.md`](../workspace/policies/pit- Run PIT with the lifecycle prefix — `mvn test-compile org.pitest:pitest-maven:mutationCoverage` (from the repo root add `-f llama/pom.xml`). The gate is **hermetic** — no model or audio fixture needed: `ContentPartTest`'s `@TempDir` tests cover `value.ContentPart.audioFile(Path)` (verified -295/295, 0 NO_COVERAGE in a fixture-less sandbox; the former audio-fixture gotcha is resolved, -see `TODO.md`). +304/304, 0 NO_COVERAGE in a fixture-less sandbox; the former audio-fixture gotcha is resolved). +**`net.ladenthin.llama.value.*` is a target at `mutationThreshold` 100**, so a new getter on a +`value` type needs its own test or the gate reds — the `ServerMetrics` counters added for the +`getMetrics()` merge are covered by `ServerMetricsTest`. ## JPMS Module Descriptor diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 178996600..32f47b4c0 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -670,3 +670,4 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10615–b10618 | `src/llama-grammar.cpp` (**#27591: `parse_char()` accepts `case '-':`, so a `\-` escape produced by `gbnf_escape_char_class()` inside a character class no longer throws**), `ggml/src/ggml-sycl/**` (**#27660: `tq2_0` marked unsupported**), `ggml/src/ggml-webgpu/wgsl-shaders/argsort.wgsl` (**#27538: infinity handling in ARGSORT / TOP_K**) | **No project-source change (final step).** Zero lines changed under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt`, so no patch context moved. The grammar fix is strictly *widening* — an input that previously threw now parses — and reaches the project through both `LlamaModel.jsonSchemaToGrammar` and every grammar-constrained completion, so it is a **positive** for the agentic tool-calling target; no test pinned the old throw (`grep -rn 'gbnf_escape_char_class\|parse_char' src/main/cpp src/test/cpp` is empty). The SYCL and WebGPU edits only affect the `sycl-*` classifiers' compiled sources and a backend the project does not build. | | b10456–b10618 | upstream verification (sandbox, final target) | **Full local verification, re-run from scratch on the shipped tree.** Fresh `cmake -B build` with the real `FetchContent` path (no source-dir override), so the fail-loud `PATCH_COMMAND` ran for real: it resolved `ggml commit eb25b7263` / `LLAMA_BUILD_NUMBER = 10618` and applied all **7** patches — verified by grepping the fetched tree for each patch's marker (`common_params_parse_main`, `params_base.load_progress_callback == nullptr`, `get_slot_prompt_similarity`, `g_llama_server_embedded`, `llama_server_attach` + `llama_server_register_common_routes`, `LLAMA_SERVER_WORKER_CMD`, `(int) meta.model_vocab_type`). The **7**, not 6, matters: the review-driven `0010` was added after the first verification pass and is the *third* patch to touch `server-context.cpp` (after `0002`/`0003`), a TU compiled into both `jllama` and `jllama_test`, so the whole configure + build + `ctest` cycle was re-run on the final tree rather than assumed to carry over (**491/491**, `mvn test` unchanged at 1405 run / 0 failures / 17 model-gated skips). The patch was subsequently renamed `0009` → `0010` to free the burned number; that is a filename-only change with the same sort position, re-checked by applying the whole set to a clean b10618 checkout. Then a full `cmake --build --config Release` (jllama + jllama_test both link — the `jllama_test` link is what proves `common_json::get()` resolves, since upstream documents un-specialised types as a *link*-time failure) and `ctest`: **491/491 C++ tests pass**. On the Java side: `mvn test-compile` clean, `mvn test` **1405 run / 0 failures / 17 skipped** (only the model-gated integration tests, no GGUF in the sandbox), and `NativeLibraryLoadSmokeTest` green — including `nativeBuildInfoMatchesPinnedVersionConstant`, the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10618") matches the `build-info` compiled into the freshly linked `libjllama.so`. `mvn spotless:apply` produced no changes beyond the edits themselves, the pinned **clang-format 22.1.8** reports the whole C++ tree clean, and `mvn clean javadoc:jar` is `BUILD SUCCESS`. **Per-step verification for all 25 chunks:** every intermediate tag additionally had all six patches applied by the applier's own commands and all ten project + C++-test translation units compiled against it, so no commit in this walk is a state that fails to build. **Two notes on how to read the per-chunk rows above.** (1) They say "all **6** patches" because that was the set during the walk; the 7th (`0010`) came out of the post-bump review and exists only from this final commit on. (2) Their prose reasoning about patch risk scans the directories that matter for the project's *compile* surface (`common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/`, `CMakeLists.txt`) and refers to "the five core patched files" — but the patch set actually spans **40** distinct files, because `0001` flips ~34 standalone `main()` call sites under `tools/*`, `examples/*` and `tests/*`. Two ranges did touch one of those and the prose does not say so on its own (b10488–b10499 → `tools/perplexity/perplexity.cpp`, b10590–b10593 → `tests/test-recurrent-state-rollback.cpp`; both now corrected in place). The *verdicts* were never derived from that directory scan: each chunk's patch verdict comes from really running `git apply` for the whole set against a clean checkout of that tag, which covers all 40 files. The reusable check is `git diff --name-only -- $(grep -h '^+++ b/' llama/patches/*.patch | sed 's|^+++ b/||' | sort -u)`. | | b10456–b10618 | `tools/server/server-context.cpp` — **new local patch `0010-server-cast-vocab-type-for-common-json.patch`** | **Upstream regression found by the post-bump review, fixed downstream.** `get_res_model_info()` builds the `GET /models` + `GET /v1/models` payload and emits `{"vocab_type", meta.model_vocab_type}`, an **unscoped enum**. That was correct while the `json` alias was `nlohmann::ordered_json` (nlohmann serialises an enum as an integer), but `common_json_value`'s integral constructor template is `std::is_integral`-gated, which excludes enums — so from **b10585** the value binds to `common_json_value(bool)` and goes on the wire as `true`/`false`. Upstream regressed it silently in #27511 when they flipped the alias; the same trap hit the project's own two `"vocab_type"` emit sites in `jllama.cpp` (fixed in the b10584–b10585 step). It **ships**: `server-context.cpp` is compiled into `libjllama` and both routes are served by `NativeServer`, the default fat-jar `Main-Class`, in full and in attach mode. The new patch casts the value to `int` at the emit site — one line, upstream-submittable (not yet filed), applied after `0002`/`0003` (same file). Numbered `0010`, not `0009`: that number belongs to the subprocess.h patch dropped at b10280 and is documented as such under the patches table in `CLAUDE.md`, so reusing it would make that note read as if it described this patch. A mechanical sweep confirmed it is the **only** such site in the upstream server sources the project compiles: `server_context_meta` has exactly two enum-typed members, and the other (`pooling_type`) is only ever compared, never serialised. | +| b10456–b10618 | follow-up: `src/main/cpp/{jllama.cpp,json_helpers.hpp}`, `value.ServerMetrics`, `server.OpenAiCompatServer` | **Client-contract restoration, after the walk.** The b10405–b10408 and b10509–b10519 rows above record the upstream metrics refactor correctly but understate its client impact: b10408 (#26920) reduced `server_task_result_metrics::to_json()` to the bare slot array and b10519 (#27376) split the task, so `LlamaModel.getMetrics()` had been returning the slot array — not the object its three Java consumers parse — since **before** this bump range began. Re-pointing `handleSlotAction` at `SERVER_TASK_TYPE_SLOT_GET` (b10519 row) kept the array arriving but did not restore the object. Fixed by posting **both** tasks and merging them in the new pure helper `server_metrics_to_json`, so the documented payload is rebuilt in the JNI layer instead of the Java contract following upstream's transport split; durations are converted µs→ms as the pre-b10408 payload did. Reachable-but-unexposed counters (`n_prompt_cached`, the speculative-decoding tallies) are emitted alongside and surfaced on `ServerMetrics`. `GET /slots` no longer answers 200-with-empty-body when the payload has no `slots` key. Suite 491 → **499**. **This is the class of break a header diff cannot catch**: no signature changed, the project compiled and linked clean at every one of the 25 chunks, and the only guard — `LlamaModelTest#testGetMetrics` — was model-gated and silently skipped in CI (see the same-day CI working-directory fix). | From 3cfd057b2630c82ed1c97eb7afd39441dec506ea Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 15:42:56 +0000 Subject: [PATCH 36/76] feat: expose the mmproj device selector added upstream --mmproj-device / -mmdev (#23255) is the only new user-facing knob in the whole b10456..b10618 range: 425 upstream CLI flags, exactly one added and none removed, an unchanged 68-field completion request schema and an unchanged HTTP route set. It places the multimodal projector independently of the main model -- useful on a multi-GPU host, or with the literal "none" to keep the projector on the CPU -- and upstream accepts exactly one device, rejecting a list. NativeServer could already pass it (raw argv forwarding), so the gap was only the in-process path: ModelParameters.setMmprojDevice(String), mapping to the flag exactly like setDevices/setDeviceDraft do, and the OpenAiCompatServer CLI, which carries a curated flag subset and now accepts both spellings. The flag is reachable from the embedded parser because mmproj_examples includes LLAMA_EXAMPLE_SERVER, which is what jllama.cpp parses with. Tests cover the setter, the "none" sentinel passing through verbatim rather than being rewritten into --no-mmproj-offload, independence from the main model's --device, both CLI spellings, the value reaching ModelParameters, and that naming a device without --mmproj does not by itself advertise vision support. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 2 +- README.md | 8 ++++- .../llama/parameters/ModelParameters.java | 19 ++++++++++++ .../llama/server/OpenAiServerCli.java | 25 +++++++++++++++- .../llama/parameters/ModelParametersTest.java | 22 ++++++++++++++ .../llama/server/OpenAiServerCliTest.java | 30 +++++++++++++++++++ 6 files changed, 103 insertions(+), 3 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 6a730b7a8..a2495fd71 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1133,7 +1133,7 @@ If the local check passes (`BUILD SUCCESS`), the `mvn package` job in - **`server` package — OpenAI-compatible HTTP endpoint (a single implementation).** - `server.OpenAiCompatServer` — built only on the JDK's `com.sun.net.httpserver` (no new dependency), embeddable and runnable via `java -cp net.ladenthin.llama.server.OpenAiCompatServer …` (the fat-jar default `Main-Class` is now `NativeServer` — see "Two server modes"). Serves `POST /v1/chat/completions` (streaming via SSE + non-streaming), `POST /v1/completions`, `POST /v1/embeddings`, `POST /v1/rerank`, `POST /infill`, `GET /v1/models` and `GET /health` (every route is also reachable without the `/v1` prefix), so editors that speak the OpenAI protocol (e.g. VS Code Copilot "Custom Endpoint", Cline, Roo Code, Continue) can drive a local model. Streaming chat uses the native OAI chunk path (`LlamaModel.streamChatCompletion` → `requestChatCompletionStream` / `receiveChatCompletionChunk` + the C++ `wrap_stream_chunk` helper), preserving `delta.tool_calls`; completions/embeddings/infill forward verbatim to the matching `LlamaModel.handle*`; rerank reshapes `handleRerank` into the OAI `results`/`data` shape. The chat mapper forwards `stream_options` and `response_format` and defaults `cache_prompt=true`; a CORS `Filter` answers `OPTIONS` preflights; `OpenAiSseFormatter.ensureUsageCachedTokens` guarantees `usage.prompt_tokens_details.cached_tokens` on the streamed usage chunk (Copilot crash fix, microsoft/vscode #273482). **Agentic tool-calling is the primary target**; a C++ guard (`test_server.cpp`) pins `tool_calls.function.arguments` as a JSON string (llama.cpp #20198). - **Alternative protocol surfaces** (pure translation over the OpenAI chat core — no second inference path; each reconstructs streamed tool calls via `ToolCallDeltaAccumulator`): **Ollama-native** (`GET /api/version`, `/api/tags`, `POST /api/show`, `/api/chat` with NDJSON streaming, `/api/generate` prompt-completion/FIM — `OllamaApiSupport`; `/api/show` advertises tools/insert/vision capabilities + context length for Copilot's Ollama provider), **Anthropic Messages** (`POST /v1/messages`, SSE event stream — `AnthropicApiSupport` + `AnthropicStreamTranslator`), and **OpenAI Responses** (`POST /v1/responses`, SSE event stream — `ResponsesApiSupport` + `ResponsesStreamTranslator`). The llama.cpp-native `GET /props` (context length + `modalities`) is served via `OpenAiSseFormatter.propsJson` for autocomplete clients that size their context from it. - - Supporting classes: `OpenAiServerConfig` (builder; optional bearer auth; binds `127.0.0.1`; `corsAllowOrigin`; `supportsVision`), `OpenAiServerCli` (testable CLI arg parser → `ModelParameters` + `OpenAiServerConfig`; flags incl. `--mmproj`/`--embedding`/`--reranking`), `OpenAiRequestMapper` (OAI chat request → `InferenceParameters`), `OpenAiSseFormatter` (SSE/models/error JSON + usage normalization), `OaiRerankSupport` (pure rerank request/response shaping), and the model-free test seam `OpenAiBackend`/`ChunkSink` + `LlamaModelBackend`. The streaming envelope is parsed by `json.ChatStreamChunkParser`. + - Supporting classes: `OpenAiServerConfig` (builder; optional bearer auth; binds `127.0.0.1`; `corsAllowOrigin`; `supportsVision`), `OpenAiServerCli` (testable CLI arg parser → `ModelParameters` + `OpenAiServerConfig`; flags incl. `--mmproj`/`-mmdev,--mmproj-device`/`--embedding`/`--reranking`), `OpenAiRequestMapper` (OAI chat request → `InferenceParameters`), `OpenAiSseFormatter` (SSE/models/error JSON + usage normalization), `OaiRerankSupport` (pure rerank request/response shaping), and the model-free test seam `OpenAiBackend`/`ChunkSink` + `LlamaModelBackend`. The streaming envelope is parsed by `json.ChatStreamChunkParser`. - The `server` package is a dedicated top layer in the ArchUnit `layeredArchitecture` rule (the only layer allowed to access the root `Api`); `noInternalJdkImports` carries an explicit exception for the supported `com.sun.net.httpserver` (the exported `jdk.httpserver` module, which `module-info.java` `requires`). See README "OpenAI-compatible HTTP server". **Native layer** (`src/main/cpp/`): diff --git a/README.md b/README.md index 4bfd0abff..5f00fff73 100644 --- a/README.md +++ b/README.md @@ -489,6 +489,12 @@ OpenAI-compatible `/v1/chat/completions` server. For a strictly CPU-only run, us `setDevices("none").setMmprojOffload(false)` in addition to `setGpuLayers(0)`; projector offload has its own upstream default. +On a multi-GPU host the projector can be placed independently of the weights with +`setMmprojDevice("CUDA1")` (llama.cpp `--mmproj-device`, added upstream in b10618). Exactly one +device may be named; the literal `"none"` keeps the projector on the CPU. `OpenAiCompatServer`'s CLI +accepts the same flag as `-mmdev`/`--mmproj-device`, and `NativeServer` forwards it verbatim like +every other llama-server flag. + **Audio input** works identically — load an audio-capable model (Ultravox, Qwen2.5-Omni, …) with its audio `--mmproj` and add a `ContentPart.audioFile(...)` (or `inputAudio(bytes, "wav"|"mp3")`) part. It serializes to the OpenAI `input_audio` content part and routes through the same `mtmd` pipeline: @@ -783,7 +789,7 @@ java -cp target/llama-.jar net.ladenthin.llama.server.OpenAiCompatServe Run with `--help` for the full option list (`-m/--model`, `--host`, `-p/--port`, `-c/--ctx-size`, `-b/--batch-size`, `-ub/--ubatch-size`, `-ngl/--n-gpu-layers`, `-t/--threads`, `-tb/--threads-batch`, `-ctk/--cache-type-k`, `-ctv/--cache-type-v`, `--jinja`, `--chat-template-kwargs`, `--parallel`, -`--model-id`, `--api-key`, `--mmproj`, `--embedding`, `--reranking`). The tuning flags mirror +`--model-id`, `--api-key`, `--mmproj`, `-mmdev/--mmproj-device`, `--embedding`, `--reranking`). The tuning flags mirror llama.cpp's server, so an invocation like `--jinja --chat-template-kwargs '{"reasoning_effort":"low"}' -ctk q8_0 -ctv q8_0 -b 4096 -ub 2048` works directly. diff --git a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java index a8c6965bb..3a1f8a6c8 100644 --- a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java +++ b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java @@ -1349,6 +1349,25 @@ public ModelParameters setMmprojAuto(boolean enabled) { return this; } + /** + * Select the device the multimodal projector runs on ({@code --mmproj-device}, upstream + * llama.cpp b10618). Independent of {@link #setDevices(String)}, which covers the main model: + * on a multi-GPU host the projector can be pinned to a different device than the weights. + * + *

Exactly one device may be named — upstream rejects a list — and the literal + * {@code "none"} disables projector offload entirely, the same end state as + * {@code setMmprojOffload(false)}. Use {@code --list-devices} (or + * {@code llama-server --list-devices}) to see the available names. When unset, upstream picks + * the device automatically.

+ * + * @param device the device name, or {@code "none"} to keep the projector on the CPU + * @return this builder + */ + public ModelParameters setMmprojDevice(String device) { + parameters.put("--mmproj-device", device); + return this; + } + /** * Enable offloading of the mmproj model to the GPU. * diff --git a/llama/src/main/java/net/ladenthin/llama/server/OpenAiServerCli.java b/llama/src/main/java/net/ladenthin/llama/server/OpenAiServerCli.java index 3e04d5fe1..b26b3d56a 100644 --- a/llama/src/main/java/net/ladenthin/llama/server/OpenAiServerCli.java +++ b/llama/src/main/java/net/ladenthin/llama/server/OpenAiServerCli.java @@ -27,7 +27,7 @@ * *

Flags mirror llama.cpp's own server where they overlap ({@code -m}, {@code -p}, {@code -c}, * {@code -b}, {@code -ub}, {@code -ngl}, {@code -t}, {@code -tb}, {@code -ctk}, {@code -ctv}, - * {@code --jinja}, {@code --chat-template-kwargs}); a few legacy spellings are accepted as aliases so + * {@code --jinja}, {@code --chat-template-kwargs}, {@code -mmdev}); a few legacy spellings are accepted as aliases so * earlier documented invocations keep working. The {@code --chat-template-kwargs} JSON is parsed here * (the only JSON this otherwise dependency-light parser touches) so a malformed object fails fast with * usage text rather than at native model load. @@ -72,6 +72,7 @@ public static Options parse(String... args) { @Nullable String modelId = null; @Nullable String apiKey = null; @Nullable String mmproj = null; + @Nullable String mmprojDevice = null; int ctxSize = 0; int gpuLayers = 0; int threads = 0; @@ -153,6 +154,10 @@ public static Options parse(String... args) { case "--mmproj": mmproj = nextValue(args, ++i, arg); break; + case "-mmdev": + case "--mmproj-device": + mmprojDevice = nextValue(args, ++i, arg); + break; case "--embedding": case "--embeddings": embedding = true; @@ -180,6 +185,7 @@ public static Options parse(String... args) { modelId, apiKey, mmproj, + mmprojDevice, ctxSize, gpuLayers, threads, @@ -229,6 +235,7 @@ public static String usage() { " --model-id Model id reported by /v1/models (default: file name)", " --api-key Require an 'Authorization: Bearer ' header", " --mmproj Multimodal projector for vision models (enables image input)", + " -mmdev, --mmproj-device Device for the multimodal projector ('none' = keep on CPU)", " --embedding Load in embedding mode (enables POST /v1/embeddings)", " --reranking Load in reranking mode (enables POST /v1/rerank)", " -h, --help Show this help and exit", @@ -330,6 +337,7 @@ public static final class Options { private final @Nullable String modelId; private final @Nullable String apiKey; private final @Nullable String mmproj; + private final @Nullable String mmprojDevice; private final int ctxSize; private final int gpuLayers; private final int threads; @@ -351,6 +359,7 @@ private Options( @Nullable String modelId, @Nullable String apiKey, @Nullable String mmproj, + @Nullable String mmprojDevice, int ctxSize, int gpuLayers, int threads, @@ -370,6 +379,7 @@ private Options( this.modelId = modelId; this.apiKey = apiKey; this.mmproj = mmproj; + this.mmprojDevice = mmprojDevice; this.ctxSize = ctxSize; this.gpuLayers = gpuLayers; this.threads = threads; @@ -443,6 +453,16 @@ public String getModelId() { return mmproj; } + /** + * The device the multimodal projector should run on. Independent of the main model's + * placement; the literal {@code "none"} keeps the projector on the CPU. + * + * @return the projector device name, or {@code null} to let llama.cpp choose + */ + public @Nullable String getMmprojDevice() { + return mmprojDevice; + } + /** * The context window size, or {@code 0} for the llama.cpp default. * @@ -571,6 +591,9 @@ public boolean isReranking() { public ModelParameters toModelParameters() { final ModelParameters params = new ModelParameters().setModel(modelPath).setGpuLayers(gpuLayers); + if (mmprojDevice != null) { + params.setMmprojDevice(mmprojDevice); + } if (mmproj != null) { params.setMmproj(mmproj); } diff --git a/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java b/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java index 041fdb2d1..c41765860 100644 --- a/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java +++ b/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java @@ -441,6 +441,28 @@ public void testDisableMmprojAuto() { assertThat(p.parameters, not(hasKey("--mmproj-auto"))); } + @Test + public void testSetMmprojDevice() { + ModelParameters p = new ModelParameters().setMmprojDevice("CUDA1"); + assertThat(p.parameters.get("--mmproj-device"), is("CUDA1")); + } + + @Test + public void testSetMmprojDeviceNoneIsPassedThroughVerbatim() { + // "none" is upstream's sentinel for "do not offload the projector"; it must reach the + // native parser as-is rather than being translated into --no-mmproj-offload here. + ModelParameters p = new ModelParameters().setMmprojDevice("none"); + assertThat(p.parameters.get("--mmproj-device"), is("none")); + assertThat(p.parameters, not(hasKey("--no-mmproj-offload"))); + } + + @Test + public void testSetMmprojDeviceIsIndependentOfTheMainModelDevices() { + ModelParameters p = new ModelParameters().setDevices("CUDA0").setMmprojDevice("CUDA1"); + assertThat(p.parameters.get("--device"), is("CUDA0")); + assertThat(p.parameters.get("--mmproj-device"), is("CUDA1")); + } + @Test public void testEnableMmprojOffload() { ModelParameters p = new ModelParameters().enableMmprojOffload(); diff --git a/llama/src/test/java/net/ladenthin/llama/server/OpenAiServerCliTest.java b/llama/src/test/java/net/ladenthin/llama/server/OpenAiServerCliTest.java index 30204d6a3..f86f803c3 100644 --- a/llama/src/test/java/net/ladenthin/llama/server/OpenAiServerCliTest.java +++ b/llama/src/test/java/net/ladenthin/llama/server/OpenAiServerCliTest.java @@ -96,6 +96,36 @@ public void mmprojFlagParsed() { assertThat(OpenAiServerCli.parse("-m", "m.gguf").getMmproj(), is((String) null)); } + @Test + public void mmprojDeviceFlagParsedUnderBothSpellings() { + assertThat( + OpenAiServerCli.parse("-m", "m.gguf", "--mmproj-device", "CUDA1") + .getMmprojDevice(), + is("CUDA1")); + assertThat(OpenAiServerCli.parse("-m", "m.gguf", "-mmdev", "CUDA1").getMmprojDevice(), is("CUDA1")); + assertThat(OpenAiServerCli.parse("-m", "m.gguf").getMmprojDevice(), is((String) null)); + } + + @Test + public void mmprojDeviceReachesModelParameters() { + assertThat( + OpenAiServerCli.parse("-m", "m.gguf", "--mmproj", "proj.gguf", "-mmdev", "none") + .toModelParameters() + .toString(), + containsString("--mmproj-device")); + } + + @Test + public void mmprojDeviceDoesNotByItselfEnableVisionCapability() { + // The vision hint is derived from --mmproj; naming a device without a projector must not + // make the server advertise image support. + assertThat( + OpenAiServerCli.parse("-m", "m.gguf", "-mmdev", "CUDA1") + .toServerConfig() + .isSupportsVision(), + is(false)); + } + @Test public void mmprojEnablesVisionCapabilityInServerConfig() { assertThat( From 68e09d4c45462f38af162138aaca0429d0ecbd0c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 15:44:28 +0000 Subject: [PATCH 37/76] docs: record this branch's work in the changelog CHANGELOG.md's Unreleased section still described the b9894 -> b9917 bump; no b10xxx build number appeared anywhere in the file, so it had drifted roughly 700 builds behind the pin. Rather than reconstruct bumps this branch did not make, the gap is stated explicitly and points at docs/history/llama-cpp-breaking-changes.md, which does carry every range. Added for this branch: the three public API additions (setMmprojDevice, the RouterClient API-key constructors, the ServerMetrics cache and speculative-decoding counters), the b10456 -> b10618 upgrade with its patch 0007 refresh and new patch 0010, the patch applier's stamp-based idempotency, and the four fixes -- the getMetrics payload shape, the empty-bodied GET /slots, the silently self-skipping model-gated tests, and the misdiagnosed hidden router model. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CHANGELOG.md | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index b16593565..f0a342328 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,15 +9,58 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by ## [Unreleased] +> Note: llama.cpp bumps between **b9917** and **b10456** were not recorded here individually. +> [`docs/history/llama-cpp-breaking-changes.md`](docs/history/llama-cpp-breaking-changes.md) is the +> complete per-range record and remains authoritative for what each upgrade changed. + ### Added - `QuantizationType.Q2_0` — maps the new upstream `LLAMA_FTYPE_MOSTLY_Q2_0` (llama.cpp b9916) for `LlamaQuantizer`. +- **`ModelParameters.setMmprojDevice(String)`** — places the multimodal projector on a device of its own + (llama.cpp `--mmproj-device`, added upstream in b10618), independently of `setDevices(...)`. Exactly one + device may be named; the literal `"none"` keeps the projector on the CPU. `OpenAiCompatServer`'s CLI + accepts the same flag as `-mmdev`/`--mmproj-device`; `NativeServer` already forwarded it verbatim. +- **`RouterClient` API-key constructors** (`RouterClient(int, String)`, `RouterClient(String, int, String)`) — + send `Authorization: Bearer `, which a router started with `--api-key` requires for *every* call: + `/models/load` and `/models/unload` were always gated, and since b10519 (upstream #26347) the listing + endpoints are too. An empty key behaves like none, and `toString()` never prints it. +- **`ServerMetrics` cache and speculative-decoding counters** — `getCumulativeCachedPromptTokens()`, + `getDraftTokensTotal()`, `getDraftAcceptedTotal()`, `getDraftVerifyStepsTotal()`, + `getDraftAcceptedPerPosition()` and the derived `getDraftAcceptanceRate()`. Upstream exposes these only + as Prometheus text; they now arrive in the JSON payload. ### Changed - `ch.qos.logback:logback-classic` bumped 1.6.2 → 1.6.3 (test/runtime binding only). - CI actions bumped to latest: `actions/setup-java` v5 → v6. - Upgraded llama.cpp from **b9894 to b9917** (all eight local patches re-verified across the range). +- Upgraded llama.cpp from **b10456 to b10618**, in 25 reviewed steps. Patch `0007` refreshed (upstream + #26347 deleted comments inside its removal block, breaking `git apply` at every tag from b10519 on) and + a new patch `0010` carries a one-line upstream fix: `GET /models` emitted `vocab_type` as a JSON boolean + after the `common_json` switch (#27511), because an unscoped enum binds to the `bool` constructor. + The project's own C++ moved to `common_json` in the same range. +- **`apply-llama-patches.cmake` is now genuinely idempotent**, via a stamp file (llama.cpp commit plus each + patch's SHA-256) gated on git's clean/dirty state. Reconfiguring an existing build directory is a no-op + instead of aborting with a misleading "does not apply cleanly"; a real mismatch fails with an accurate + message. A source tree supplied via `-DFETCHCONTENT_SOURCE_DIR_LLAMA.CPP` that is not a git work tree + keeps the previous per-patch behaviour. +- `ServerMetrics.getStartTimestamp()` is documented correctly: `t_start` is a monotonic-clock **microsecond** + reading (`ggml_time_us()`), not milliseconds since the epoch. The value is unchanged. ### Fixed +- **`LlamaModel.getMetrics()` returned the wrong shape.** Upstream reduced the payload to a bare slot array + at b10408 (#26920) and split the task in two at b10519 (#27376), so the counter getters on + `value.ServerMetrics`, `LlamaModelTest#testGetMetrics` and `OpenAiCompatServer`'s metrics routes had all + been reading keys that no longer existed. The JNI layer now posts both tasks and merges them, restoring the + documented object rather than following upstream's transport split. +- **`GET /slots` answered HTTP 200 with a zero-length body** whenever the metrics payload carried no `slots` + key (`MissingNode.toString()` is `""`). It now always answers with a JSON array. +- **Model-gated Java tests silently self-skipped in CI.** Surefire's working directory is the module basedir + while the shared GGUF cache is restored to the reactor root, so every `models/…` path resolved to nothing, + every such class aborted in its `@BeforeAll`, and the job still reported success — which is why the stale + `getMetrics()` assertions above never failed. Test paths now resolve against either layout. + `llama-langchain4j` had the identical defect. +- **`RouterClient.awaitModelLoaded` misdiagnosed hidden router models.** A cache model deduplicated by a + preset with `dedup-cache-models` (b10507, #27346) is omitted from `GET /models` although it still loads and + serves by name; the error now names that cause instead of sending callers to re-check `--models-dir`. - **CVE-2026-49844** (GHSA-qv9r-c865-cp47, moderate): `org.apache.logging.log4j:log4j-api` 2.25.3 arrives as a **test-scope** transitive of `io.github.hakky54:logcaptor` 2.12.6, and Dependabot could not update it on its own. Pinned `log4j-api` **and** `log4j-to-slf4j` to From af75fc451c0e88d46280ae6e8e6f78b1f4af59df Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 18:31:26 +0000 Subject: [PATCH 38/76] fix: reject the -1 penalty-window sentinel llama.cpp removed repeat_last_n and dry_penalty_last_n used to accept -1 for "the whole context". llama.cpp b10275 removed that sentinel: the request schema's hard limits became [0, INT32_MAX] and common_params_parse throws on a negative value. The bump that crossed b10275 fixed the C++ side (the two ParamsFromJsonCmpl *_MinusOne_Throws tests pin it) but left the Java API advertising and forwarding -1, so the value reached llama.cpp and failed there -- at model load for setRepeatLastN / setDryPenaltyLastN, as a rejected request for withRepeatLastN / withDryPenaltyLastN. Nothing caught it because the only caller, LlamaModelTest#testDrySamplingAltersRepetitiveGeneration, is model-gated and was silently skipped in CI. All four now reject a negative value with a message naming the change, the javadoc no longer documents the sentinel, and the DRY test states its window explicitly as the context size it was already configured with. The tests that pinned -1 as valid now pin that it is rejected. Scope checked rather than assumed: diffing every set_limits/set_hard_limits in the request schema between b9994 and b10618 shows exactly two fields moved, and they are these two -- no other wither carries a stale bound. Also expands CHANGELOG.md to cover the b9917 -> b10456 window, which went unrecorded while it happened: the breaking TextToSpeech rework onto Qwen3-TTS (b10270) and the voice-cloning overload it brought, this -1 removal, the dropped patches 0004/0005/0009, server-mcp.cpp becoming mandatory at b10154, the Gradle and dockcross bumps, and the macOS dylib artifact collision that shipped a SIGKILL-on-load library in 5.0.6 and several 5.0.7 snapshots. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CHANGELOG.md | 48 +++++++++++++++++-- CLAUDE.md | 2 +- .../llama/parameters/InferenceParameters.java | 34 +++++++++---- .../llama/parameters/ModelParameters.java | 30 ++++++++---- .../net/ladenthin/llama/LlamaModelTest.java | 6 ++- .../parameters/InferenceParametersTest.java | 24 ++++++++-- .../llama/parameters/ModelParametersTest.java | 21 +++++--- 7 files changed, 130 insertions(+), 35 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f0a342328..c4084b74a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,12 +9,14 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by ## [Unreleased] -> Note: llama.cpp bumps between **b9917** and **b10456** were not recorded here individually. -> [`docs/history/llama-cpp-breaking-changes.md`](docs/history/llama-cpp-breaking-changes.md) is the -> complete per-range record and remains authoritative for what each upgrade changed. +> The entries below also cover the **b9917 → b10456** window (PRs #341–#394), which went unrecorded +> here while it happened; they were reconstructed from the git history and from +> [`docs/history/llama-cpp-breaking-changes.md`](docs/history/llama-cpp-breaking-changes.md), which +> has a row per upgrade range and stays authoritative for the per-range detail. ### Added - `QuantizationType.Q2_0` — maps the new upstream `LLAMA_FTYPE_MOSTLY_Q2_0` (llama.cpp b9916) for `LlamaQuantizer`. +- **Voice cloning and language selection for `TextToSpeech`**: `synthesize(String text, String speakerReferenceAudioPath, String language, int maxFrames, int topK, int seed)` — a speaker-reference clip makes the model imitate that voice. Part of the Qwen3-TTS rework (see Changed). - **`ModelParameters.setMmprojDevice(String)`** — places the multimodal projector on a device of its own (llama.cpp `--mmproj-device`, added upstream in b10618), independently of `setDevices(...)`. Exactly one device may be named; the literal `"none"` keeps the projector on the CPU. `OpenAiCompatServer`'s CLI @@ -32,6 +34,34 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by - `ch.qos.logback:logback-classic` bumped 1.6.2 → 1.6.3 (test/runtime binding only). - CI actions bumped to latest: `actions/setup-java` v5 → v6. - Upgraded llama.cpp from **b9894 to b9917** (all eight local patches re-verified across the range). +- **BREAKING — `TextToSpeech` was reworked onto Qwen3-TTS** (llama.cpp **b10270**, upstream #26254, which + upstream itself labels a breaking change). llama.cpp deleted the OuteTTS pipeline outright: + `tools/tts/tts.cpp` shrank from ~1450 to 205 lines and `mtmd_gen_audio_type` has only + `NONE`/`QWEN3TTS`, so there is no OuteTTS code path left anywhere upstream and no compatibility shim + was possible. The two-argument constructor keeps its **signature** but changes **meaning**: + `(ttcModelPath, vocoderModelPath)` → `(modelPath, mmprojPath)`, i.e. a Qwen3-TTS backbone plus the + mmproj that bundles speaker encoder, code predictor and code2wav decoder — an OuteTTS + WavTokenizer + pair no longer works and fails at load, not at compile time. `synthesize`'s `maxCodeTokens` parameter + became `maxFrames`, and the single-argument overload's default dropped 4096 → 512. +- **BREAKING — `-1` is no longer accepted for the repetition-penalty windows** (llama.cpp **b10275**). + `repeat_last_n` and `dry_penalty_last_n` used to take `-1` for "the whole context"; upstream removed + the sentinel, moving the request schema's hard limits to `[0, INT32_MAX]` and making + `common_params_parse` throw on a negative value. `ModelParameters.setRepeatLastN` / + `setDryPenaltyLastN` and `InferenceParameters.withRepeatLastN` / `withDryPenaltyLastN` had kept + advertising and accepting `-1`, so the value reached llama.cpp and failed there — at model load for + the launch flags, as a rejected request for the per-request withers. All four now reject a negative + value with a message naming the change; pass the context size explicitly for the old behaviour. + (Verified exhaustively: these are the **only** two request-field limits that moved in the whole + b9994 → b10618 range.) +- Upgraded llama.cpp from **b9917 to b10456** across PRs #341–#394. Local patches `0005` (b9981) and + `0004` (b9982) were dropped after upstream merged equivalent — and broader — fixes, and `0009` + (`subprocess.h` old-glibc build break) was dropped at b10280 once upstream vendored the same fix. +- `server-mcp.cpp` is compiled into `libjllama` (llama.cpp **b10154** added upstream MCP-server + support; `server.cpp` and `server-tools.cpp` reference `server_mcp`, so omitting it is latent on + Linux but a hard link error on macOS/ld64 and Windows/MSVC). The `subprocess.h` `addchdir_np` use is + guarded for old glibc in the same change. +- Android/Gradle toolchain: Gradle pins moved 8.14.3 → 9.6.1 and the dockcross cross-compile images + were bumped, alongside the AGP/Compose pin updates the Android builds needed. - Upgraded llama.cpp from **b10456 to b10618**, in 25 reviewed steps. Patch `0007` refreshed (upstream #26347 deleted comments inside its removal block, breaking `git apply` at every tag from b10519 on) and a new patch `0010` carries a one-line upstream fix: `GET /models` emitted `vocab_type` as a JSON boolean @@ -46,6 +76,18 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by reading (`ggml_time_us()`), not milliseconds since the epoch. The value is unchanged. ### Fixed +- **The macOS arm64 native library shipped corrupt in 5.0.6 and in several 5.0.7 snapshots.** All three + macOS arm64 build jobs uploaded their dylib under a `*-libraries` artifact name, and the packaging + job collects those with one globbed download — so three builds landed on the same + `Mac/aarch64/libjllama.dylib` and the survivor could be a byte-level hybrid of two of them rather + than either input. Its ad-hoc signature then no longer matched its own `__TEXT` pages (66/4078 and + 1141/4097 code pages failed their stored hashes) and macOS **SIGKILLed every process that loaded + it**. Fixed by naming the test-only variants outside the glob and selecting the shipped variant by + an explicit download step (thanks to **@linking12**, #388), plus two guards so it cannot recur: + `merge-native-artifacts.sh` fails the build when any relative path is claimed by more than one + artifact — checked *before* the merge, since a collision leaves exactly one file behind and is + invisible afterwards — and the new `smoke-fatjar-macos` job runs `codesign --verify --strict` and a + real JVM load of the dylib extracted from the **packaged** fat jar (#390). - **`LlamaModel.getMetrics()` returned the wrong shape.** Upstream reduced the payload to a bare slot array at b10408 (#26920) and split the task in two at b10519 (#27376), so the counter getters on `value.ServerMetrics`, `LlamaModelTest#testGetMetrics` and `OpenAiCompatServer`'s metrics routes had all diff --git a/CLAUDE.md b/CLAUDE.md index a2495fd71..11922bc70 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1137,7 +1137,7 @@ If the local check passes (`BUILD SUCCESS`), the `mvn package` job in - The `server` package is a dedicated top layer in the ArchUnit `layeredArchitecture` rule (the only layer allowed to access the root `Api`); `noInternalJdkImports` carries an explicit exception for the supported `com.sun.net.httpserver` (the exported `jdk.httpserver` module, which `module-info.java` `requires`). See README "OpenAI-compatible HTTP server". **Native layer** (`src/main/cpp/`): -- `jllama.cpp` — JNI implementation bridging Java calls to llama.cpp. ~1,650 lines; 34 native methods (30 `LlamaModel` + 3 `TextToSpeech` + 1 `LlamaQuantizer`). +- `jllama.cpp` — JNI implementation bridging Java calls to llama.cpp. ~1,850 lines; 34 native methods (30 `LlamaModel` + 3 `TextToSpeech` + 1 `LlamaQuantizer`) plus `JNI_OnLoad`/`JNI_OnUnload`. - `utils.hpp` — Helper utilities (format helpers, argv stripping, token-piece serialisation). - `json_helpers.hpp` — Pure JSON transformation helpers (no JNI, no llama state). Independently unit-testable. - `jni_helpers.hpp` — JNI bridge helpers (handle management + server orchestration). Includes `json_helpers.hpp`. diff --git a/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java b/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java index a831234e1..1da46006e 100644 --- a/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java +++ b/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java @@ -344,12 +344,22 @@ public InferenceParameters withDynamicTemperatureExponent(float dynatempExponent } /** - * Returns a new request with the repetition-penalty window replaced (default: 64, 0 = disabled, -1 = ctx_size). + * Returns a new request with the repetition-penalty window replaced (default: 64, 0 = disabled). * - * @param repeatLastN window size (0 = disabled, -1 = ctx_size) + *

Upstream llama.cpp dropped the {@code -1} = context-size sentinel at b10275; + * the request schema's hard limits are now {@code [0, INT32_MAX]}, so {@code -1} makes the server + * reject the whole request. It is rejected here instead, so the failure names the cause rather + * than arriving as a generic parameter error.

+ * + * @param repeatLastN window size (0 = disabled) * @return a new instance; this instance is unchanged + * @throws IllegalArgumentException if {@code repeatLastN} is negative */ public InferenceParameters withRepeatLastN(int repeatLastN) { + if (repeatLastN < 0) { + throw new IllegalArgumentException("Invalid repeat_last_n value: " + repeatLastN + + " (must be >= 0; 0 = disabled. llama.cpp b10275 dropped -1 = ctx_size)"); + } return withScalar(PARAM_REPEAT_LAST_N, repeatLastN); } @@ -791,19 +801,23 @@ public InferenceParameters withDryAllowedLength(int dryAllowedLength) { } /** - * Returns a new request with the per-request DRY penalty window replaced (default: -1, -1 = context - * size, 0 = disabled). Only takes effect when {@link #withDryMultiplier(float)} is non-zero. - * Per-request mirror of {@link ModelParameters#setDryPenaltyLastN(int)} (the - * {@code --dry-penalty-last-n} launch flag); values below {@code -1} are rejected. + * Returns a new request with the per-request DRY penalty window replaced (0 = disabled). Only takes + * effect when {@link #withDryMultiplier(float)} is non-zero. Per-request mirror of + * {@link ModelParameters#setDryPenaltyLastN(int)} (the {@code --dry-penalty-last-n} launch flag). + * + *

Upstream llama.cpp dropped the {@code -1} = context-size sentinel at b10275; + * the request schema's hard limits are now {@code [0, INT32_MAX]}, so {@code -1} makes the server + * reject the whole request. It is rejected here instead, so the failure names the cause rather + * than arriving as a generic parameter error.

* - * @param dryPenaltyLastN the DRY penalty window (-1 = context size, 0 = disabled) + * @param dryPenaltyLastN the DRY penalty window (0 = disabled) * @return a new instance; this instance is unchanged - * @throws IllegalArgumentException if {@code dryPenaltyLastN} is less than {@code -1} + * @throws IllegalArgumentException if {@code dryPenaltyLastN} is negative */ public InferenceParameters withDryPenaltyLastN(int dryPenaltyLastN) { - if (dryPenaltyLastN < -1) { + if (dryPenaltyLastN < 0) { throw new IllegalArgumentException("Invalid dry_penalty_last_n value: " + dryPenaltyLastN - + " (must be >= -1; -1 = context size, 0 = disabled)"); + + " (must be >= 0; 0 = disabled. llama.cpp b10275 dropped -1 = context size)"); } return withScalar(PARAM_DRY_PENALTY_LAST_N, dryPenaltyLastN); } diff --git a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java index 3a1f8a6c8..c185ee0e8 100644 --- a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java +++ b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java @@ -431,15 +431,21 @@ public ModelParameters setTypical(float typP) { } /** - * Set last n tokens to consider for penalize (default: 64, 0 = disabled, -1 = ctx_size). + * Set last n tokens to consider for penalize (default: 64, 0 = disabled). * - * @param repeatLastN the number of last tokens to consider for penalties (0 = disabled, -1 = ctx_size) + *

Upstream llama.cpp dropped the {@code -1} = context-size sentinel at b10275: + * the lower bound is now {@code 0}, and {@code --repeat-last-n -1} makes {@code common_params_parse} + * throw, which surfaces as a model-load failure. It is rejected here instead, so the failure names + * the cause rather than arriving from the native layer.

+ * + * @param repeatLastN the number of last tokens to consider for penalties (0 = disabled) * @return this builder + * @throws IllegalArgumentException if {@code repeatLastN} is negative */ public ModelParameters setRepeatLastN(int repeatLastN) { - if (repeatLastN < -1) { - throw new IllegalArgumentException( - "Invalid repeat-last-n value: " + repeatLastN + " (must be >= -1; -1 = ctx_size, 0 = disabled)"); + if (repeatLastN < 0) { + throw new IllegalArgumentException("Invalid repeat-last-n value: " + repeatLastN + + " (must be >= 0; 0 = disabled. llama.cpp b10275 dropped -1 = ctx_size)"); } return putScalar("--repeat-last-n", repeatLastN); } @@ -505,15 +511,21 @@ public ModelParameters setDryAllowedLength(int dryAllowedLength) { } /** - * Set DRY penalty for the last n tokens (default: -1, 0 = disable, -1 = context size). + * Set DRY penalty for the last n tokens (0 = disable). + * + *

Upstream llama.cpp dropped the {@code -1} = context-size sentinel at b10275: + * the lower bound is now {@code 0}, and {@code --dry-penalty-last-n -1} makes + * {@code common_params_parse} throw, which surfaces as a model-load failure. It is rejected here + * instead, so the failure names the cause rather than arriving from the native layer.

* - * @param dryPenaltyLastN the DRY penalty window (-1 = context size, 0 = disabled) + * @param dryPenaltyLastN the DRY penalty window (0 = disabled) * @return this builder + * @throws IllegalArgumentException if {@code dryPenaltyLastN} is negative */ public ModelParameters setDryPenaltyLastN(int dryPenaltyLastN) { - if (dryPenaltyLastN < -1) { + if (dryPenaltyLastN < 0) { throw new IllegalArgumentException("Invalid dry-penalty-last-n value: " + dryPenaltyLastN - + " (must be >= -1; -1 = context size, 0 = disabled)"); + + " (must be >= 0; 0 = disabled. llama.cpp b10275 dropped -1 = context size)"); } return putScalar("--dry-penalty-last-n", dryPenaltyLastN); } diff --git a/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java b/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java index f92ea7b9e..14c15f85e 100644 --- a/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java +++ b/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java @@ -129,7 +129,7 @@ public void testGenerateInfill() { *

With greedy decoding ({@code withTopK(1)}) and a fixed seed, two completions of the same * prompt are byte-identical unless something changes the sampler. The prompt is saturated with a * repeated multi-token n-gram, so enabling DRY with a strong multiplier and a short allowed length - * ({@code dry_penalty_last_n = -1} scans the whole context) penalizes the next token that would + * ({@code dry_penalty_last_n} is set to the full context size) penalizes the next token that would * extend that n-gram — forcing the DRY run to diverge from the baseline. This exercises the * full Java → JSON → native path for {@code withDryMultiplier} / {@code withDryBase} / * {@code withDryAllowedLength} / {@code withDryPenaltyLastN} end to end; the per-field JSON @@ -152,7 +152,9 @@ public void testDrySamplingAltersRepetitiveGeneration() { .withDryMultiplier(4.0f) .withDryBase(1.75f) .withDryAllowedLength(2) - .withDryPenaltyLastN(-1); + // The whole context: llama.cpp b10275 dropped -1 as the "scan everything" sentinel, + // so the window is stated explicitly and must match the ctx size set in setup(). + .withDryPenaltyLastN(128); String baselineOutput = model.complete(baseline); String dryOutput = model.complete(withDry); diff --git a/llama/src/test/java/net/ladenthin/llama/parameters/InferenceParametersTest.java b/llama/src/test/java/net/ladenthin/llama/parameters/InferenceParametersTest.java index e0ed445c0..813f1c936 100644 --- a/llama/src/test/java/net/ladenthin/llama/parameters/InferenceParametersTest.java +++ b/llama/src/test/java/net/ladenthin/llama/parameters/InferenceParametersTest.java @@ -782,8 +782,26 @@ public void testSetDryAllowedLength() { @Test public void testSetDryPenaltyLastN() { - InferenceParameters params = new InferenceParameters("").withDryPenaltyLastN(-1); - assertThat(params.parameters.get("dry_penalty_last_n"), is("-1")); + InferenceParameters params = new InferenceParameters("").withDryPenaltyLastN(512); + assertThat(params.parameters.get("dry_penalty_last_n"), is("512")); + } + + @Test + public void testSetDryPenaltyLastNRejectsMinusOne() { + // llama.cpp b10275 set the request schema's hard limits to [0, INT32_MAX]; -1 no longer means + // "the whole context", it makes the server reject the request. + assertThrows(IllegalArgumentException.class, () -> new InferenceParameters("").withDryPenaltyLastN(-1)); + } + + @Test + public void testWithRepeatLastNRejectsMinusOne() { + assertThrows(IllegalArgumentException.class, () -> new InferenceParameters("").withRepeatLastN(-1)); + } + + @Test + public void testWithRepeatLastNAcceptsZeroAndPositive() { + assertThat(new InferenceParameters("").withRepeatLastN(0).parameters.get("repeat_last_n"), is("0")); + assertThat(new InferenceParameters("").withRepeatLastN(64).parameters.get("repeat_last_n"), is("64")); } @Test @@ -834,7 +852,7 @@ public void testDryWithersReturnNewInstance() { assertThat(params.withDryMultiplier(0.8f), is(not(sameInstance(params)))); assertThat(params.withDryBase(1.75f), is(not(sameInstance(params)))); assertThat(params.withDryAllowedLength(2), is(not(sameInstance(params)))); - assertThat(params.withDryPenaltyLastN(-1), is(not(sameInstance(params)))); + assertThat(params.withDryPenaltyLastN(512), is(not(sameInstance(params)))); assertThat(params.withDrySequenceBreakers("\n"), is(not(sameInstance(params)))); } } diff --git a/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java b/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java index c41765860..62d34dba5 100644 --- a/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java +++ b/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java @@ -94,9 +94,10 @@ public void testSetRepeatLastNValidZero() { } @Test - public void testSetRepeatLastNValidMinusOne() { - ModelParameters p = new ModelParameters().setRepeatLastN(-1); - assertThat(p.parameters.get("--repeat-last-n"), is("-1")); + public void testSetRepeatLastNRejectsMinusOne() { + // llama.cpp b10275 dropped the -1 = ctx_size sentinel; common_params_parse now throws on a + // negative value, so the model would fail to load. Reject it here where the message can say why. + assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setRepeatLastN(-1)); } @Test @@ -111,13 +112,19 @@ public void testSetRepeatLastNTooLow() { } // ------------------------------------------------------------------------- - // setDryPenaltyLastN — validation (>= -1) + // setDryPenaltyLastN — validation (>= 0 since llama.cpp b10275) // ------------------------------------------------------------------------- @Test - public void testSetDryPenaltyLastNValidMinusOne() { - ModelParameters p = new ModelParameters().setDryPenaltyLastN(-1); - assertThat(p.parameters.get("--dry-penalty-last-n"), is("-1")); + public void testSetDryPenaltyLastNRejectsMinusOne() { + // Same b10275 change as setRepeatLastN: --dry-penalty-last-n -1 makes common_params_parse throw. + assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setDryPenaltyLastN(-1)); + } + + @Test + public void testSetDryPenaltyLastNValidPositive() { + ModelParameters p = new ModelParameters().setDryPenaltyLastN(256); + assertThat(p.parameters.get("--dry-penalty-last-n"), is("256")); } @Test From 601c4f89f614ad38774719e3a42d08395cea46ae Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 18:34:21 +0000 Subject: [PATCH 39/76] docs: refresh stale claims and harden the upgrade review Four things had drifted out of date, all of them factual claims a reader would act on: - CLAUDE.md described jllama.cpp as ~1,650 lines; it is ~1,850, and its 36 JNIEXPORTs are the 34 native methods plus JNI_OnLoad/JNI_OnUnload, which the count did not say. - The upgrade review list claimed its top 8 rows "cover all known API-level breaking changes from b5022 to b8831" -- an upper bound five hundred builds stale, and a claim b10275 had already falsified by breaking the build through server-schema.h, a header the list did not carry at the time. - TODO.md's future-output-modalities note still proposed putting OuteTTS behind an /v1/audio/speech route; OuteTTS has not existed upstream since b10270, and the thing to route is the Qwen3-TTS TextToSpeech that replaced it. - CHANGELOG.md's b9917 -> b10456 gap (filled in the previous commit). The review guidance now also names the two failure classes a header diff structurally cannot catch, with the concrete incidents behind each: a same-repo header included directly but absent from the dependency graph (server-schema.h, b10275), and a silent contract change behind an unchanged signature (getMetrics()'s payload at b10408/b10519; the repeat_last_n bounds at b10275). Both are now covered by three mechanical set-diffs -- request fields, their bounds, and emitted response keys -- documented here and added to the bump runbook as its own step, so a future chunk that touches tools/server/ gets them run rather than reasoned about. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 28 +++++++++++++++++++++++--- TODO.md | 5 +++-- docs/upgrade/llama-cpp-version-bump.md | 23 ++++++++++++++++++--- 3 files changed, 48 insertions(+), 8 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 11922bc70..8c353828f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -828,9 +828,31 @@ jllama.cpp / server.hpp / utils.hpp **Priority-ordered review list for upgrade diffs** (highest break risk first) -The top 8 rows cover all known API-level breaking changes from b5022 → b8831. -For future upgrades, provide diffs for at least these 8 files rather than the full patch. -Also review the project `CMakeLists.txt` for build-system-level breaks (e.g. renamed link targets, new required headers) — those are not visible in header file diffs alone. +The rows below cover the known **compile/link-level** breaks from b5022 to the current pin; start any +upgrade review with them rather than the full patch. Also review the project `CMakeLists.txt` for +build-system-level breaks (e.g. renamed link targets, new required headers) — those are not visible in +header file diffs alone. + +**Two failure classes this list does NOT catch, both of which have bitten the project:** + +1. **A same-repo header the project includes directly but that is not reachable through the + dependency graph above.** `tools/server/server-schema.h` broke a full build at b10275 while sitting + outside this table; it is in it now, and the `tools/server/*.h` rule in its row generalises that. +2. **A silent *contract* change behind an unchanged signature.** b10408 reduced + `server_task_result_metrics::to_json()` to a bare slot array and b10519 split the task; no + signature moved, every chunk compiled and linked clean, and `LlamaModel.getMetrics()` quietly + returned the wrong shape for hundreds of builds. The same class hit `repeat_last_n` / + `dry_penalty_last_n` at b10275, where only a *value range* moved. Two cheap mechanical checks catch + these where a header diff cannot — run them on any bump that touches `tools/server/`: + +```bash +# request-field set + their bounds, old tag vs new +git show :tools/server/server-schema.cpp | grep -oE 'field_[a-z_]+\("[a-z_0-9]+"' | sort -u +git show :tools/server/server-schema.cpp | tr '\n' ' ' \ + | grep -oE 'field_[a-z]+[^(]*\("[a-z_0-9]+"[^;]*?set_(hard_)?limits\([^)]*\)' +# response keys emitted by the result types +git show :tools/server/server-task.cpp | grep -oE '\{"[a-z_0-9.]+",' | sort -u +``` | File | What to watch for | |------|-------------------| diff --git a/TODO.md b/TODO.md index 95147d225..3125acb00 100644 --- a/TODO.md +++ b/TODO.md @@ -34,8 +34,9 @@ round-trips — see CLAUDE.md "Two server modes"). **Owner priority: the native- - **Future *output* modalities (audio / image) — design note, not yet actionable.** llama.cpp's server produces text (plus embeddings/rerank) only; the integration points are isolated (a new `OpenAiBackend.stream*` primitive + `OpenAiSseFormatter.*Chunk` per modality). Two future hooks: - OuteTTS behind an `/v1/audio/speech`-style route; proxying image/audio generation to an external - model. Keep chunk formatters modality-neutral. + the existing `TextToSpeech` (Qwen3-TTS since llama.cpp b10270 — OuteTTS no longer exists upstream) + behind an `/v1/audio/speech`-style route; proxying image/audio generation to an external model. + Keep chunk formatters modality-neutral. - **Incremental tool-call streaming on the alternative surfaces.** Ollama/Anthropic/Responses emit each tool call whole at end-of-stream (`ToolCallDeltaAccumulator`); revisit only if a client needs incremental `input_json_delta` / `function_call_arguments.delta` fidelity. diff --git a/docs/upgrade/llama-cpp-version-bump.md b/docs/upgrade/llama-cpp-version-bump.md index a8269d2b2..33d8d15cd 100644 --- a/docs/upgrade/llama-cpp-version-bump.md +++ b/docs/upgrade/llama-cpp-version-bump.md @@ -133,17 +133,34 @@ Concretely: rm -rf build && cmake -B build # fail-loud: aborts here if any patch no longer applies ``` If a patch no longer applies, refresh its diff against the new source and recommit it. -3. **Append the history rows** — add a pair of rows to +3. **Check the server contract mechanically when the chunk touches `tools/server/`.** A header diff + only shows signature changes; it cannot see a *contract* change behind a stable signature. Two + breaks of that class already shipped — `getMetrics()`'s payload shape (b10408/b10519) and the + removal of the `-1` = context-size sentinel for `repeat_last_n`/`dry_penalty_last_n` (b10275) — + and neither was visible to the build. Diff these three sets between the two tags; anything that + changes has to be traced to the Java layer, not just to the C++ tests: + ```bash + # request-field set + git show b:tools/server/server-schema.cpp | grep -oE 'field_[a-z_]+\("[a-z_0-9]+"' | sort -u + # request-field bounds + git show b:tools/server/server-schema.cpp | tr '\n' ' ' \ + | grep -oE 'field_[a-z]+[^(]*\("[a-z_0-9]+"[^;]*?set_(hard_)?limits\([^)]*\)' | sort -u + # response keys + git show b:tools/server/server-task.cpp | grep -oE '\{"[a-z_0-9.]+",' | sort -u + ``` + Repeat with `b` and `comm -13` / `comm -23` the two outputs. + +4. **Append the history rows** — add a pair of rows to [`../history/llama-cpp-breaking-changes.md`](../history/llama-cpp-breaking-changes.md) covering the `b -> b` range (what broke / what was new; "no source change" is a valid row). -4. **Commit + push** on the working branch (do not open a new PR if one already tracks the branch): +5. **Commit + push** on the working branch (do not open a new PR if one already tracks the branch): ```bash git add llama/CMakeLists.txt README.md CLAUDE.md docs/history/llama-cpp-breaking-changes.md \ llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java git commit -m "Upgrade llama.cpp from b to b" git push -u origin ``` -5. **Re-run the helper** for the next chunk. Repeat until it reports the **final chunk** (target +6. **Re-run the helper** for the next chunk. Repeat until it reports the **final chunk** (target reached). CI builds every native classifier from the new pin; the full model-backed Java + C++ suites gate the From 2987d3e34251a4f430a7721abd386642d25fa878 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 21:29:55 +0000 Subject: [PATCH 40/76] test: fix the streaming assertion for a thinking-only response The langchain4j model-backed job went red on JllamaChatModelIntegrationTest#streamingDeliversTokensThenCompletes: Expected: is "" but: was null with "Tests run: 41, Skipped: 0" -- the first time these tests have ever actually run in CI. They resolved their GGUF path relative to the module basedir while the cache is restored to the reactor root, so every one of them self-skipped; the TestModelPaths resolver on this branch is what made them run, which is also what exposed this. It is a test defect, not a regression. CI drives this job with Qwen3-0.6B, a reasoning model, and the request budgets 8 output tokens: the run spends all of them inside , so every token arrives as reasoning_content. StreamingChunkAssembler.complete() only calls message.text(...) when it accumulated content, so AiMessage.text() is null while the handler's accumulator is still "" -- and "" is not null. Nothing in the streaming path changed on this branch; the same assertion would fail identically on main. The assertion now compares the concatenated onPartialResponse fragments against the final text with null normalised to "", which is the invariant that actually holds in both shapes, and separately requires the stream to have delivered something (content or reasoning) so a stream that produces nothing at all still fails -- strictly stronger than before in that dimension. The mechanism itself is now pinned model-free in StreamingChunkAssemblerTest (thinkingOnlyStreamLeavesTextNullAndDeliversNoContentFragments), so the null text is a contract the C++-free suite enforces on every platform rather than something the next reader rediscovers from a red integration job. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- .../JllamaChatModelIntegrationTest.java | 20 ++++++++++++++- .../StreamingChunkAssemblerTest.java | 25 +++++++++++++++++++ 2 files changed, 44 insertions(+), 1 deletion(-) diff --git a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java index 1d840ca19..4726eabd4 100644 --- a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java +++ b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java @@ -84,7 +84,25 @@ public void onError(Throwable error) { }); ChatResponse complete = done.get(60, TimeUnit.SECONDS); - assertThat(complete.aiMessage().text(), is(streamed.toString())); + + // The CI model is Qwen3-0.6B, a reasoning model, and this request budgets 8 output + // tokens: a run can legitimately spend all of them inside and produce no + // assistant text at all. StreamingChunkAssembler then routes every token to + // onPartialThinking and leaves AiMessage.text() null (it only sets text when it + // accumulated some), while `streamed` stays "" -- which is the "expected \"\" but was + // null" this assertion used to fail with once the test actually ran. + // + // Assert the invariant that holds in both shapes -- the concatenated onPartialResponse + // fragments are exactly the final text -- and separately that the stream delivered + // something, so a stream that produces nothing at all still fails. + String finalText = complete.aiMessage().text() == null + ? "" + : complete.aiMessage().text(); + assertThat(finalText, is(streamed.toString())); + assertThat( + "stream delivered neither content nor reasoning tokens", + !streamed.toString().isEmpty() || complete.aiMessage().thinking() != null, + is(true)); } } } diff --git a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/StreamingChunkAssemblerTest.java b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/StreamingChunkAssemblerTest.java index 1f2a9b2c1..171393316 100644 --- a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/StreamingChunkAssemblerTest.java +++ b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/StreamingChunkAssemblerTest.java @@ -6,6 +6,7 @@ import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.contains; +import static org.hamcrest.Matchers.empty; import static org.hamcrest.Matchers.is; import static org.hamcrest.Matchers.nullValue; import static org.junit.jupiter.api.Assertions.assertThrows; @@ -151,6 +152,30 @@ void forwardsThinkingDeltasAndKeepsThinkingOnFinalMessage() { assertThat(response.aiMessage().text(), is("42")); } + @Test + void thinkingOnlyStreamLeavesTextNullAndDeliversNoContentFragments() { + // A reasoning model that spends its whole output budget inside streams only + // reasoning_content. complete() then never calls message.text(...), so AiMessage.text() + // is null rather than "" -- which is exactly what made + // JllamaChatModelIntegrationTest#streamingDeliversTokensThenCompletes fail with + // "expected \"\" but was null" the first time it actually ran against Qwen3-0.6B. + // Pinned here, model-free, so the null is a known contract and not a surprise. + RecordingHandler handler = new RecordingHandler(); + StreamingChunkAssembler assembler = new StreamingChunkAssembler(handler); + + assembler.accept("{\"choices\":[{\"index\":0,\"delta\":{\"reasoning_content\":\"still \"}," + + "\"finish_reason\":null}]}"); + assembler.accept("{\"choices\":[{\"index\":0,\"delta\":{\"reasoning_content\":\"thinking\"}," + + "\"finish_reason\":null}]}"); + assembler.accept("{\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"length\"}]}"); + ChatResponse response = assembler.complete(); + + assertThat(response.aiMessage().text(), is(nullValue())); + assertThat(response.aiMessage().thinking(), is("still thinking")); + assertThat(handler.partials, is(empty())); + assertThat(handler.thinking, contains("still ", "thinking")); + } + @Test void noUsageChunkMeansNoTokenUsage() { RecordingHandler handler = new RecordingHandler(); From 50650ec383ba2f3898699c25e7b7a1e5a7b82795 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 21:41:22 +0000 Subject: [PATCH 41/76] fix: bound the metrics wait and restore counter key spellings Two findings from an adversarial review of this branch against the minimal b10456..b10618 upstream diff, both independently re-verified before acting. 1. getMetrics() could block forever, and the two-task split widened the window. post_and_wait waited with `[]{ return false; }` -- the only wait site in jllama.cpp without the `closing` predicate; the other six all have it, and jllama.cpp:1314 documents that predicate as how close() unblocks a pending call. It matters more now: b10618 added task_resets_idle_timer() to server-queue.cpp (0 hits at b10456, 4 at b10618), which stops a METRICS task from resetting the idle timer, so the queue can enter its sleep state between the two tasks this branch now posts and never process the second one. Upstream guards its own /metrics handler with exactly this in mind -- "a task posted right before sleeping is never processed, do not wait for it" -- by adding queue_tasks.is_sleeping() to the predicate and posting high-priority. Both tasks are now posted front-of-queue like upstream's, and the wait honours `closing`. is_sleeping() is not on server_context's public header, so the JNI layer cannot ask; that limitation is stated at the call site rather than papered over. Reaching the state needs setSleepIdleSeconds(> 0). The docs claimed this was unreachable -- "sleep_idle_seconds defaults to -1 and nothing in the project sets it" -- which is false: ModelParameters.setSleepIdleSeconds(int) is public API. That sentence is what made the hazard invisible; corrected. 2. Two new counters were shipping under invented key names. The pre-b10408 payload this merge reproduces emitted n_draft_verif_steps_total and n_accepted_per_pos_total (verified at b10405); the branch had tidied them to n_draft_verify_steps_total and n_draft_accepted_per_pos. Since the getters are new and nothing consumes them yet, the wire keys go back to upstream's spelling while the Java method names stay readable. The claim that these counters "had no JSON representation at all" was wrong for four of five -- only n_prompt_tokens_cached_total is genuinely new (n_prompt_cached: 0 hits at b10405), and the per-position array is not in the Prometheus text either. Also corrected: --mmproj-device lands at b10541, not b10618 (bisected: absent b10532, present b10541) in README, CHANGELOG and the setter javadoc; the history row that called it "deliberately left unwired" now records that the setter was added later on this branch. The response-key sweep added to the bump runbook only matched brace-init keys, so it missed every res["k"] = ... key and would have reported timings/prompt_progress as removed in this very range -- it now unions both emit forms. The DRY penalty-window setters lost their default (64) when the -1 sentinel text was rewritten; restored. C++ 499/499, mvn verify 1436/0, PIT 304/304, SpotBugs 0, javadoc clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CHANGELOG.md | 2 +- CLAUDE.md | 4 +- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 4 +- docs/upgrade/llama-cpp-version-bump.md | 6 ++- llama/src/main/cpp/jllama.cpp | 45 ++++++++++++------- llama/src/main/cpp/json_helpers.hpp | 17 ++++--- .../llama/parameters/InferenceParameters.java | 2 +- .../llama/parameters/ModelParameters.java | 4 +- .../ladenthin/llama/value/ServerMetrics.java | 14 +++--- llama/src/test/cpp/test_json_helpers.cpp | 8 ++-- .../llama/value/ServerMetricsTest.java | 6 +-- 12 files changed, 68 insertions(+), 46 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c4084b74a..d0851d799 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,7 +18,7 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by - `QuantizationType.Q2_0` — maps the new upstream `LLAMA_FTYPE_MOSTLY_Q2_0` (llama.cpp b9916) for `LlamaQuantizer`. - **Voice cloning and language selection for `TextToSpeech`**: `synthesize(String text, String speakerReferenceAudioPath, String language, int maxFrames, int topK, int seed)` — a speaker-reference clip makes the model imitate that voice. Part of the Qwen3-TTS rework (see Changed). - **`ModelParameters.setMmprojDevice(String)`** — places the multimodal projector on a device of its own - (llama.cpp `--mmproj-device`, added upstream in b10618), independently of `setDevices(...)`. Exactly one + (llama.cpp `--mmproj-device`, added upstream in b10541), independently of `setDevices(...)`. Exactly one device may be named; the literal `"none"` keeps the projector on the CPU. `OpenAiCompatServer`'s CLI accepts the same flag as `-mmdev`/`--mmproj-device`; `NativeServer` already forwarded it verbatim. - **`RouterClient` API-key constructors** (`RouterClient(int, String)`, `RouterClient(String, int, String)`) — diff --git a/CLAUDE.md b/CLAUDE.md index 8c353828f..bff837ab7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -851,7 +851,9 @@ git show :tools/server/server-schema.cpp | grep -oE 'field_[a-z_]+\("[a-z_0 git show :tools/server/server-schema.cpp | tr '\n' ' ' \ | grep -oE 'field_[a-z]+[^(]*\("[a-z_0-9]+"[^;]*?set_(hard_)?limits\([^)]*\)' # response keys emitted by the result types -git show :tools/server/server-task.cpp | grep -oE '\{"[a-z_0-9.]+",' | sort -u +# response keys -- both emit forms; a single-form grep misses res["k"] = ... entirely +git show :tools/server/server-task.cpp | { grep -oE '\{ *"[A-Za-z_0-9.]+" *,'; \ + git show :tools/server/server-task.cpp | grep -oE '\[ *"[A-Za-z_0-9.]+" *\] *='; } | sort -u ``` | File | What to watch for | diff --git a/README.md b/README.md index 5f00fff73..ecf7c43f5 100644 --- a/README.md +++ b/README.md @@ -490,7 +490,7 @@ OpenAI-compatible `/v1/chat/completions` server. For a strictly CPU-only run, us has its own upstream default. On a multi-GPU host the projector can be placed independently of the weights with -`setMmprojDevice("CUDA1")` (llama.cpp `--mmproj-device`, added upstream in b10618). Exactly one +`setMmprojDevice("CUDA1")` (llama.cpp `--mmproj-device`, added upstream in b10541). Exactly one device may be named; the literal `"none"` keeps the projector on the CPU. `OpenAiCompatServer`'s CLI accepts the same flag as `-mmdev`/`--mmproj-device`, and `NativeServer` forwards it verbatim like every other llama-server flag. diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 32f47b4c0..65e9d57f7 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -633,11 +633,11 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10499–b10507 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10507 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10507–b10509 | `ggml/include/ggml.h` + `ggml/src/**` + `ggml/src/ggml-metal/**` (**#27120: new `ggml_rope_set_offset()` and its Metal support**), `tools/ui/**` (**#27365: persisted settings read before the API-key probe**) | **No project-source change.** Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or the root `CMakeLists.txt` changed at all in this range, so none of the five **core** patched files (`common/arg.cpp`, `server-context.{cpp,h}`, `server.cpp`, `server-models.cpp`) moved and no header in the project's include dependency graph was touched. `ggml_rope_set_offset` is purely additive to `ggml.h` (a header CLAUDE.md's review list marks "safe to skip" — the project never calls a `ggml_rope_*` primitive; `grep -rn "ggml_rope" src/main/cpp src/test/cpp` is empty), and `tools/ui` is the Svelte WebUI, which CI rebuilds from the pinned `GIT_TAG` and which therefore needs no per-bump source review. | | b10507–b10509 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10509 checkout (sequential `git apply`, filename order, zero fuzz); fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. This is the **last** tag at which patch `0007` applies unmodified — see the next row. | -| b10509–b10519 | `tools/server/server-task.{h,cpp}` + `tools/server/server-context.{h,cpp}` + `tools/server/server-queue.{h,cpp}` (**#27376 "server: refactor sleep handling, allow access /metrics during sleep"**), `tools/server/server.cpp` + `tools/server/server-http.cpp` (**#26347: `/models` + `/v1/models` are no longer API-key-exempt**), `common/common.cpp` (**#27337: revert of the b10499 threadpool sharing when `n_threads` differ**), `ggml/include/ggml.h`, `tools/ui/**`, `src/models/granite-swa.cpp` | **Project source change required (two), plus a patch refresh.** (1) #27376 split `server_task_result_metrics` in two: `n_idle_slots` + `slots_data` moved out into a **new `server_task_result_slots`**, produced by a **new `SERVER_TASK_TYPE_SLOT_GET`** task, and `server_task_result_metrics::to_json()` now just returns `json{}` — the *default-constructed* value, i.e. JSON **null**, not an empty object (`/metrics` renders Prometheus text via the byte-identical `to_metrics()`). Two project impacts, one silent and one hard: `jllama.cpp`'s `handleSlotAction` LIST arm posted `SERVER_TASK_TYPE_METRICS`, which still **compiles** at b10519 but would return `{}` instead of the slot array to `LlamaModel.getMetrics()` — re-pointed at `SERVER_TASK_TYPE_SLOT_GET` (upstream's `/slots` handler builds exactly that task, with no extra fields); and `test_server.cpp` was a **hard compile error** (`make_metrics()` set the removed `n_idle_slots`, `ToJson_ReturnsSlotsArrayVerbatim` set the removed `slots_data`) — the slots assertions moved to `server_task_result_slots` and two tests were added (`ServerTaskResultSlots.ToJson_EmptyByDefault`, `ServerTaskResultMetrics.ToJson_UnusedAndEmpty`), 485 → 487. `to_metrics()` is byte-identical and never read `n_idle_slots`, so every Prometheus assertion survived unchanged. (2) **Patch `0007` stopped applying here** — #26347 deleted the trailing `// public endpoint (no API key check)` comments on the two `/models` route registrations, which sit inside `0007`'s `@@ -258,47 +310,7 @@` removal block (`git apply` → "patch does not apply", `server.cpp:258`); refreshed by dropping that comment from those four lines (2 on the `-` side, 2 on the extracted `llama_server_register_common_routes()` `+` side, where it had also become factually wrong). `0001`/`0002`/`0003`/`0006`/`0008` applied unchanged. No CMake wiring change: no `tools/server` or `vendor` file was added, removed or renamed. **Behavioural note for consumers:** a `NativeServer` started with `--api-key` now returns 401 on `/models` and `/v1/models`; the project's own `RouterModeIntegrationTest` runs without an API key, so CI is unaffected. The new `server_routes` sleep-cache callback registers in the constructor and is inert here — `sleep_idle_seconds` defaults to `-1` (disabled) and nothing in the project sets it. | +| b10509–b10519 | `tools/server/server-task.{h,cpp}` + `tools/server/server-context.{h,cpp}` + `tools/server/server-queue.{h,cpp}` (**#27376 "server: refactor sleep handling, allow access /metrics during sleep"**), `tools/server/server.cpp` + `tools/server/server-http.cpp` (**#26347: `/models` + `/v1/models` are no longer API-key-exempt**), `common/common.cpp` (**#27337: revert of the b10499 threadpool sharing when `n_threads` differ**), `ggml/include/ggml.h`, `tools/ui/**`, `src/models/granite-swa.cpp` | **Project source change required (two), plus a patch refresh.** (1) #27376 split `server_task_result_metrics` in two: `n_idle_slots` + `slots_data` moved out into a **new `server_task_result_slots`**, produced by a **new `SERVER_TASK_TYPE_SLOT_GET`** task, and `server_task_result_metrics::to_json()` now just returns `json{}` — the *default-constructed* value, i.e. JSON **null**, not an empty object (`/metrics` renders Prometheus text via the byte-identical `to_metrics()`). Two project impacts, one silent and one hard: `jllama.cpp`'s `handleSlotAction` LIST arm posted `SERVER_TASK_TYPE_METRICS`, which still **compiles** at b10519 but would return `{}` instead of the slot array to `LlamaModel.getMetrics()` — re-pointed at `SERVER_TASK_TYPE_SLOT_GET` (upstream's `/slots` handler builds exactly that task, with no extra fields); and `test_server.cpp` was a **hard compile error** (`make_metrics()` set the removed `n_idle_slots`, `ToJson_ReturnsSlotsArrayVerbatim` set the removed `slots_data`) — the slots assertions moved to `server_task_result_slots` and two tests were added (`ServerTaskResultSlots.ToJson_EmptyByDefault`, `ServerTaskResultMetrics.ToJson_UnusedAndEmpty`), 485 → 487. `to_metrics()` is byte-identical and never read `n_idle_slots`, so every Prometheus assertion survived unchanged. (2) **Patch `0007` stopped applying here** — #26347 deleted the trailing `// public endpoint (no API key check)` comments on the two `/models` route registrations, which sit inside `0007`'s `@@ -258,47 +310,7 @@` removal block (`git apply` → "patch does not apply", `server.cpp:258`); refreshed by dropping that comment from those four lines (2 on the `-` side, 2 on the extracted `llama_server_register_common_routes()` `+` side, where it had also become factually wrong). `0001`/`0002`/`0003`/`0006`/`0008` applied unchanged. No CMake wiring change: no `tools/server` or `vendor` file was added, removed or renamed. **Behavioural note for consumers:** a `NativeServer` started with `--api-key` now returns 401 on `/models` and `/v1/models`; the project's own `RouterModeIntegrationTest` runs without an API key, so CI is unaffected. The new `server_routes` sleep-cache callback registers in the constructor and is inert **by default** — `sleep_idle_seconds` defaults to `-1`. It is **not** true that nothing in the project sets it: `ModelParameters.setSleepIdleSeconds(int)` is public API, so a caller can enable the sleep state. That matters because b10618 also stops `SERVER_TASK_TYPE_METRICS` from resetting the idle timer (`task_resets_idle_timer` in `server-queue.cpp`), which widens the window in which a task posted right before sleep is never processed — see the `post_and_wait` note in `jllama.cpp`. | | b10509–b10519 | upstream verification (sandbox) | Patch `0007` **refreshed** (see above); all **6** patches then re-verified against a clean b10519 checkout — sequential `git apply --check` + `git apply` in filename order, zero fuzz — and the refreshed `0007` additionally re-checked to apply cleanly at **every** later endpoint of this walk up to b10618, so the refresh is a one-time fix, not a per-chunk one. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics against the patched b10519 tree. Full link + `ctest` deferred to the b10618 target. | | b10519–b10532 | `common/speculative.cpp` (**#27404: avoid binding a reference to a null pointer**), `ggml/**` (backend-split scheduler race #26040, CUDA cuBLAS workspace #26574, Metal FA dequant #27390, Vulkan/OpenCL/Hexagon kernels), `src/llama-graph.cpp` (**#27392: V built as a view of K in `k_iswa build_attn`**), `convert_hf_to_gguf.py` | **No project-source change.** The only in-scope file is `common/speculative.cpp`, and the fix is inside an upstream-compiled TU — `common/speculative.h` (priority 3 on the CLAUDE.md review list) has a zero-line diff across the range, so nothing the project compiles against moved, and `grep -rn "common_speculative" src/main/cpp src/test/cpp` is empty. Nothing under `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt` changed, so no patch context shifted and `llama/CMakeLists.txt`'s `target_sources` list still matches upstream. The remainder is ggml backends and the llama graph builder, both entirely upstream-owned. | | b10519–b10532 | upstream verification (sandbox) | All **6** patches (with the b10519-refreshed `0007`) re-verified against a clean b10532 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | -| b10532–b10541 | `common/common.h` + `common/arg.cpp` + `tools/mtmd/{mtmd.h,mtmd.cpp,clip.h,clip.cpp}` + `tools/server/server-context.cpp` (**#23255 `--mmproj-device`: new `common_params::mmproj_device` (`ggml_backend_dev_t`) and a new `device` member at position 2 in `mtmd_context_params`/`clip_context_params`; the `MTMD_BACKEND_DEVICE` env lookup moves out of `clip.cpp` into `arg.cpp`**), `common/json-schema-to-grammar.cpp` (**#26939: unsupported regex patterns degrade to accept-any-string with a warning instead of throwing**), `tools/server/{server-models.h,server-models.cpp,server.cpp}` (**#27424: load-on-startup deferred into a new public `server_models::load_startup_models()` called from `llama_server()`'s router branch; reload no longer autoloads. #27416: `is_router_server` also excludes `--docker-repo`**), `common/speculative.cpp` (**`!dp.drafting` guard**) | **No project-source change.** The mid-struct `mtmd_context_params::device` insertion is the one positional-aggregate-init hazard in this range and the project does not trip it: `src/main/cpp/tts_engine.cpp` builds its params via `mtmd_context_params_default()` plus a named `.use_gpu` assignment, so `device` defaults to `nullptr` = the previous auto-GPU behaviour; `ggml_backend_dev_t` is already visible through `mtmd.h` → `llama.h` → `ggml-backend.h`, and every `common_params` in the project is default-constructed. **Deliberately left unwired:** the new `-mmdev/--mmproj-device` flag gets no `ModelParameters` setter in this bump — that is new public Java API and the owner's call, not a version bump's. It stays reachable through `NativeServer`, which forwards raw llama-server argv; the `LlamaModel` / `OpenAiCompatServer` path has no equivalent (there is no generic argv passthrough on `ModelParameters`), so a multi-GPU caller who needs it must use `NativeServer` until a setter is added. `common/json-schema-to-grammar.h` is unchanged, so #26939 is semantic only — it does reach a public Java API (`jllama.cpp`'s `jsonSchemaToGrammarBytes` → `LlamaModel.jsonSchemaToGrammar`), where a schema with an unanchored `pattern` or a `\d`/`\w`/`\s` escape now yields a permissive `string` grammar plus a stderr warning instead of throwing `LlamaException`; no test moves, because `LlamaModelTest#testJsonSchemaToGrammar` asserts only `PRIMITIVE_RULES` output from three pattern-less string properties (zero `"pattern"` hits across `src/test/{java,cpp}`). `server_models` / `load_startup_models` / `docker_repo` are zero-hit greps in the project's C++ and Java. **Patch context:** `server.cpp` gains 13 lines (the `load_startup_models()` block) **below** patch `0007`'s route-table hunk, so only that patch's *last* hunk offset moves (10 → 23) — it still applies, verified by real `git apply` at both endpoints. **Build wiring:** no file added/removed/renamed under `tools/server`/`tools/mtmd`/`vendor`. | +| b10532–b10541 | `common/common.h` + `common/arg.cpp` + `tools/mtmd/{mtmd.h,mtmd.cpp,clip.h,clip.cpp}` + `tools/server/server-context.cpp` (**#23255 `--mmproj-device`: new `common_params::mmproj_device` (`ggml_backend_dev_t`) and a new `device` member at position 2 in `mtmd_context_params`/`clip_context_params`; the `MTMD_BACKEND_DEVICE` env lookup moves out of `clip.cpp` into `arg.cpp`**), `common/json-schema-to-grammar.cpp` (**#26939: unsupported regex patterns degrade to accept-any-string with a warning instead of throwing**), `tools/server/{server-models.h,server-models.cpp,server.cpp}` (**#27424: load-on-startup deferred into a new public `server_models::load_startup_models()` called from `llama_server()`'s router branch; reload no longer autoloads. #27416: `is_router_server` also excludes `--docker-repo`**), `common/speculative.cpp` (**`!dp.drafting` guard**) | **No project-source change.** The mid-struct `mtmd_context_params::device` insertion is the one positional-aggregate-init hazard in this range and the project does not trip it: `src/main/cpp/tts_engine.cpp` builds its params via `mtmd_context_params_default()` plus a named `.use_gpu` assignment, so `device` defaults to `nullptr` = the previous auto-GPU behaviour; `ggml_backend_dev_t` is already visible through `mtmd.h` → `llama.h` → `ggml-backend.h`, and every `common_params` in the project is default-constructed. **Left unwired during the bump, wired afterwards:** the new `-mmdev/--mmproj-device` flag deliberately got no `ModelParameters` setter *in the bump commits* — that is new public Java API and the owner's call, not a version bump's. It was added later on the same branch (`ModelParameters.setMmprojDevice(String)` plus `-mmdev`/`--mmproj-device` in `OpenAiServerCli`), once the owner asked for it; before that it was reachable only through `NativeServer`, which forwards raw llama-server argv. Note the flag lands at **b10541**, not at the b10618 target — README/CHANGELOG/javadoc were corrected to say so. `common/json-schema-to-grammar.h` is unchanged, so #26939 is semantic only — it does reach a public Java API (`jllama.cpp`'s `jsonSchemaToGrammarBytes` → `LlamaModel.jsonSchemaToGrammar`), where a schema with an unanchored `pattern` or a `\d`/`\w`/`\s` escape now yields a permissive `string` grammar plus a stderr warning instead of throwing `LlamaException`; no test moves, because `LlamaModelTest#testJsonSchemaToGrammar` asserts only `PRIMITIVE_RULES` output from three pattern-less string properties (zero `"pattern"` hits across `src/test/{java,cpp}`). `server_models` / `load_startup_models` / `docker_repo` are zero-hit greps in the project's C++ and Java. **Patch context:** `server.cpp` gains 13 lines (the `load_startup_models()` block) **below** patch `0007`'s route-table hunk, so only that patch's *last* hunk offset moves (10 → 23) — it still applies, verified by real `git apply` at both endpoints. **Build wiring:** no file added/removed/renamed under `tools/server`/`tools/mtmd`/`vendor`. | | b10532–b10541 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10541 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | | b10541–b10545 | `ggml/src/ggml-metal/**` (**#27450: clamp the K extent in the tensor-API mat-mat kernel when K is not a multiple of 32**), `ggml/src/ggml-opencl/**` (**#26476 q6_K flat mul_mat on Adreno A6x/A7x with older E031 compilers; #27339 norm local size**), `tools/ui/**` (**#27240: stores split refactor**) | **No project-source change.** Nothing under `common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/` or any `CMakeLists.txt` changed — the whole range is ggml backend kernels plus the Svelte WebUI (which CI rebuilds from the pinned `GIT_TAG`, so it needs no per-bump source review). No patch context moved and `llama/CMakeLists.txt`'s `target_sources` list still matches upstream. The step's raw `git diff` is large (≈978 KiB) only because of the WebUI refactor; the reviewable code outside `tools/ui` is ≈17 KiB. | | b10541–b10545 | upstream verification (sandbox) | All **6** patches re-verified against a clean b10545 checkout — sequential `git apply`, filename order, zero fuzz. Fresh configure clean and all ten project + C++-test translation units compiled with no diagnostics. Full link + `ctest` deferred to the b10618 target. | diff --git a/docs/upgrade/llama-cpp-version-bump.md b/docs/upgrade/llama-cpp-version-bump.md index 33d8d15cd..490e2135e 100644 --- a/docs/upgrade/llama-cpp-version-bump.md +++ b/docs/upgrade/llama-cpp-version-bump.md @@ -146,7 +146,11 @@ Concretely: git show b:tools/server/server-schema.cpp | tr '\n' ' ' \ | grep -oE 'field_[a-z]+[^(]*\("[a-z_0-9]+"[^;]*?set_(hard_)?limits\([^)]*\)' | sort -u # response keys - git show b:tools/server/server-task.cpp | grep -oE '\{"[a-z_0-9.]+",' | sort -u + # response keys -- BOTH emit forms: brace-init AND res["k"] = ...; the b10585 migration + # moved `timings`/`prompt_progress` between the two, so a single-form grep reports + # false removals and would silently miss a new operator[] key. + git show b:tools/server/server-task.cpp | { grep -oE '\{ *"[A-Za-z_0-9.]+" *,'; \ + git show b:tools/server/server-task.cpp | grep -oE '\[ *"[A-Za-z_0-9.]+" *\] *='; } | sort -u ``` Repeat with `b` and `comm -13` / `comm -23` the two outputs. diff --git a/llama/src/main/cpp/jllama.cpp b/llama/src/main/cpp/jllama.cpp index 5a63ece76..3d02e5579 100644 --- a/llama/src/main/cpp/jllama.cpp +++ b/llama/src/main/cpp/jllama.cpp @@ -349,19 +349,31 @@ static void populate_completion_task(server_task &task, jllama_context *jctx, // Post a single pre-built task and wait for its result. Returns nullptr after // throwing via JNI when the task failed; callers must return immediately then. // The task's id field is assigned here; callers must not set it beforehand. -[[nodiscard]] static server_task_result_ptr post_and_wait(JNIEnv *env, server_context *ctx_server, server_task task) { - auto rd = ctx_server->get_response_reader(); +// +// Posted high-priority (front of the queue), like upstream does for its own +// introspection endpoints, and waited on with the same `closing` predicate every other +// wait site in this file uses, so close() can unblock a pending call. +// +// KNOWN LIMITATION: a task posted just as the queue enters its idle-sleep state is never +// processed -- upstream's own /metrics handler says so verbatim ("a task posted right +// before sleeping is never processed, do not wait for it") and guards it by adding +// queue_tasks.is_sleeping() to its wait predicate. That getter is not on +// server_context's public header, so this layer cannot ask; the `closing` predicate is +// what bounds the wait instead. Reaching that state needs +// ModelParameters.setSleepIdleSeconds(> 0), which is off by default (-1). +[[nodiscard]] static server_task_result_ptr post_and_wait(JNIEnv *env, jllama_context *jctx, server_task task) { + auto rd = jctx->server.get_response_reader(); task.id = rd.get_new_id(); - rd.post_task(std::move(task)); - auto result = rd.next([] { return false; }); + rd.post_task(std::move(task), true); + auto result = rd.next([jctx] { return jctx->closing.load(); }); if (!result_ok_or_throw(env, result)) return nullptr; return result; } // Post a single pre-built task, wait for its result, and return JSON as a jstring. -[[nodiscard]] static jstring dispatch_one_shot_task(JNIEnv *env, server_context *ctx_server, server_task task) { - auto result = post_and_wait(env, ctx_server, std::move(task)); +[[nodiscard]] static jstring dispatch_one_shot_task(JNIEnv *env, jllama_context *jctx, server_task task) { + auto result = post_and_wait(env, jctx, std::move(task)); if (!result) return nullptr; return json_to_jstring(env, result->to_json()); @@ -369,9 +381,8 @@ static void populate_completion_task(server_task &task, jllama_context *jctx, // Post a single slot file task (SAVE or RESTORE), wait for its result, and // return the result JSON as a jstring. -[[nodiscard]] static jstring exec_slot_file_task(JNIEnv *env, server_context *ctx_server, jint slotId, - jstring jfilename, server_task_type task_type, - const char *empty_filename_error) { +[[nodiscard]] static jstring exec_slot_file_task(JNIEnv *env, jllama_context *jctx, jint slotId, jstring jfilename, + server_task_type task_type, const char *empty_filename_error) { const std::string filename = jfilename != nullptr ? parse_jstring(env, jfilename) : ""; if (filename.empty()) { env->ThrowNew(c_llama_error, empty_filename_error); @@ -381,7 +392,7 @@ static void populate_completion_task(server_task &task, jllama_context *jctx, task.slot_action.id_slot = slotId; task.slot_action.filename = filename; task.slot_action.filepath = filename; - return dispatch_one_shot_task(env, ctx_server, std::move(task)); + return dispatch_one_shot_task(env, jctx, std::move(task)); } char **parse_string_array(JNIEnv *env, const jobjectArray string_array, const jsize length) { @@ -1640,10 +1651,10 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleSlotAction(J // is unused and returns JSON null; to_metrics() renders them as Prometheus text) and // SERVER_TASK_TYPE_SLOT_GET carries the slot array plus the idle-slot count. Post both and // merge them so getMetrics() keeps returning the single documented object. - auto metrics_result = post_and_wait(env, ctx_server, server_task(SERVER_TASK_TYPE_METRICS)); + auto metrics_result = post_and_wait(env, jctx, server_task(SERVER_TASK_TYPE_METRICS)); if (!metrics_result) return nullptr; - auto slots_result = post_and_wait(env, ctx_server, server_task(SERVER_TASK_TYPE_SLOT_GET)); + auto slots_result = post_and_wait(env, jctx, server_task(SERVER_TASK_TYPE_SLOT_GET)); if (!slots_result) return nullptr; @@ -1656,15 +1667,15 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleSlotAction(J return json_to_jstring(env, server_metrics_to_json(*metrics, *slots)); } case 1: // SAVE - return exec_slot_file_task(env, ctx_server, slotId, jfilename, SERVER_TASK_TYPE_SLOT_SAVE, + return exec_slot_file_task(env, jctx, slotId, jfilename, SERVER_TASK_TYPE_SLOT_SAVE, "Filename is required for slot save"); case 2: // RESTORE - return exec_slot_file_task(env, ctx_server, slotId, jfilename, SERVER_TASK_TYPE_SLOT_RESTORE, + return exec_slot_file_task(env, jctx, slotId, jfilename, SERVER_TASK_TYPE_SLOT_RESTORE, "Filename is required for slot restore"); case 3: { // ERASE server_task task(SERVER_TASK_TYPE_SLOT_ERASE); task.slot_action.id_slot = slotId; - return dispatch_one_shot_task(env, ctx_server, std::move(task)); + return dispatch_one_shot_task(env, jctx, std::move(task)); } default: env->ThrowNew(c_llama_error, "Invalid slot action"); @@ -1675,7 +1686,7 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleSlotAction(J JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_getLoraAdaptersJson(JNIEnv *env, jobject obj) { REQUIRE_SERVER_CONTEXT(nullptr); - return dispatch_one_shot_task(env, ctx_server, server_task(SERVER_TASK_TYPE_GET_LORA)); + return dispatch_one_shot_task(env, jctx, server_task(SERVER_TASK_TYPE_GET_LORA)); } JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_setLoraAdaptersJson(JNIEnv *env, jobject obj, @@ -1693,7 +1704,7 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_setLoraAdaptersJso } server_task task(SERVER_TASK_TYPE_SET_LORA); task.set_lora = parse_lora_request(data); - return dispatch_one_shot_task(env, ctx_server, std::move(task)); + return dispatch_one_shot_task(env, jctx, std::move(task)); } JNIEXPORT void JNICALL Java_net_ladenthin_llama_LlamaQuantizer_quantizeNative(JNIEnv *env, jclass, jstring jinput, diff --git a/llama/src/main/cpp/json_helpers.hpp b/llama/src/main/cpp/json_helpers.hpp index f2032e0a2..5c6eff2ed 100644 --- a/llama/src/main/cpp/json_helpers.hpp +++ b/llama/src/main/cpp/json_helpers.hpp @@ -247,10 +247,12 @@ // object here. // // Key names are the pre-b10408 ones — `idle`, `processing`, `deferred`, -// `t_start`, the `n_*`/`t_*` counter pairs and `slots` — so every existing -// consumer keeps working. The counters upstream added since (cached prompt -// tokens and the speculative-decoding tallies) are emitted alongside them; they -// were previously reachable only through the Prometheus text. +// `t_start`, the `n_*`/`t_*` counter pairs, the speculative-decoding tallies and +// `slots` — so every existing consumer keeps working. The speculative counters +// deliberately keep upstream's own historical spelling (`n_draft_verif_steps_total`, +// `n_accepted_per_pos_total`), not a tidied-up one, so the payload stays a faithful +// reproduction. Only `n_prompt_tokens_cached_total` is genuinely new: `n_prompt_cached` +// did not exist in the struct before b10408 and has no Prometheus counterpart either. // // Durations are microseconds upstream and milliseconds here, matching what the // pre-b10408 payload used. @@ -284,12 +286,13 @@ out["n_tokens_predicted"] = m.predict_bucket.count; out["t_tokens_generation"] = to_ms(m.predict_bucket.time); - // Counters with no pre-b10408 equivalent. + // Cache counter (new since b10408) plus the speculative-decoding tallies, which the + // pre-b10408 payload already carried under exactly these names. out["n_prompt_tokens_cached_total"] = m.n_prompt_cached; out["n_draft_tokens_total"] = m.n_draft_tokens; out["n_draft_accepted_total"] = m.n_draft_accepted; - out["n_draft_verify_steps_total"] = m.n_draft_verif_steps; - out["n_draft_accepted_per_pos"] = m.n_accepted_per_pos; + out["n_draft_verif_steps_total"] = m.n_draft_verif_steps; + out["n_accepted_per_pos_total"] = m.n_accepted_per_pos; out["slots"] = slots_result.slots_data; return out; diff --git a/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java b/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java index 1da46006e..f3c1d75d5 100644 --- a/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java +++ b/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java @@ -801,7 +801,7 @@ public InferenceParameters withDryAllowedLength(int dryAllowedLength) { } /** - * Returns a new request with the per-request DRY penalty window replaced (0 = disabled). Only takes + * Returns a new request with the per-request DRY penalty window replaced (default: 64, 0 = disabled). Only takes * effect when {@link #withDryMultiplier(float)} is non-zero. Per-request mirror of * {@link ModelParameters#setDryPenaltyLastN(int)} (the {@code --dry-penalty-last-n} launch flag). * diff --git a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java index c185ee0e8..e049a43ab 100644 --- a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java +++ b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java @@ -511,7 +511,7 @@ public ModelParameters setDryAllowedLength(int dryAllowedLength) { } /** - * Set DRY penalty for the last n tokens (0 = disable). + * Set DRY penalty for the last n tokens (default: 64, 0 = disable). * *

Upstream llama.cpp dropped the {@code -1} = context-size sentinel at b10275: * the lower bound is now {@code 0}, and {@code --dry-penalty-last-n -1} makes @@ -1363,7 +1363,7 @@ public ModelParameters setMmprojAuto(boolean enabled) { /** * Select the device the multimodal projector runs on ({@code --mmproj-device}, upstream - * llama.cpp b10618). Independent of {@link #setDevices(String)}, which covers the main model: + * llama.cpp b10541). Independent of {@link #setDevices(String)}, which covers the main model: * on a multi-GPU host the projector can be pinned to a different device than the weights. * *

Exactly one device may be named — upstream rejects a list — and the literal diff --git a/llama/src/main/java/net/ladenthin/llama/value/ServerMetrics.java b/llama/src/main/java/net/ladenthin/llama/value/ServerMetrics.java index 826758cfb..58ed0a362 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/ServerMetrics.java +++ b/llama/src/main/java/net/ladenthin/llama/value/ServerMetrics.java @@ -26,8 +26,8 @@ * {@code n_decode_total}, {@code n_busy_slots_total}, {@code n_tokens_max} — * plus the cache and speculative-decoding counters * ({@code n_prompt_tokens_cached_total}, {@code n_draft_tokens_total}, - * {@code n_draft_accepted_total}, {@code n_draft_verify_steps_total}, - * {@code n_draft_accepted_per_pos}) and a {@code slots} array. + * {@code n_draft_accepted_total}, {@code n_draft_verif_steps_total}, + * {@code n_accepted_per_pos_total}) and a {@code slots} array. *

*

* Upstream llama.cpp stopped emitting this object as one piece: b10408 (upstream #26920) @@ -36,8 +36,10 @@ * reachable only as Prometheus text and a {@code SLOT_GET} task carrying the slots. The * JNI layer posts both and merges them back into the shape documented here (see * {@code server_metrics_to_json} in {@code json_helpers.hpp}), so this view is unaffected - * by that transport split. The cache and speculative-decoding counters are new: before the - * merge they had no JSON representation at all. + * by that transport split. The speculative-decoding counters keep upstream's own + * pre-b10408 key spellings so the payload is a faithful reproduction; + * {@code n_prompt_tokens_cached_total} is the one genuinely new key (the underlying + * {@code n_prompt_cached} counter did not exist before b10408). *

* *

{@code equals}/{@code hashCode} are generated by Lombok over the underlying @@ -158,7 +160,7 @@ public long getDraftAcceptedTotal() { * @return draft verification-step total since server start */ public long getDraftVerifyStepsTotal() { - return node.path("n_draft_verify_steps_total").asLong(0L); + return node.path("n_draft_verif_steps_total").asLong(0L); } /** @@ -180,7 +182,7 @@ public double getDraftAcceptanceRate() { * @return immutable per-position acceptance counts, empty when none are reported */ public List getDraftAcceptedPerPosition() { - JsonNode perPos = node.path("n_draft_accepted_per_pos"); + JsonNode perPos = node.path("n_accepted_per_pos_total"); if (!perPos.isArray()) { return Collections.emptyList(); } diff --git a/llama/src/test/cpp/test_json_helpers.cpp b/llama/src/test/cpp/test_json_helpers.cpp index 2d7faaa0d..d3b0459e0 100644 --- a/llama/src/test/cpp/test_json_helpers.cpp +++ b/llama/src/test/cpp/test_json_helpers.cpp @@ -582,13 +582,13 @@ TEST(ServerMetricsToJson, CarriesCurrentWindowCounters) { EXPECT_EQ(j.at("n_tokens_predicted").get(), 20); } -TEST(ServerMetricsToJson, CarriesCountersUpstreamAddedAfterB10408) { +TEST(ServerMetricsToJson, CarriesCacheAndSpeculativeDecodingCounters) { const json j = server_metrics_to_json(make_metrics_half(), make_slots_half()); EXPECT_EQ(j.at("n_prompt_tokens_cached_total").get(), 77); EXPECT_EQ(j.at("n_draft_tokens_total").get(), 30); EXPECT_EQ(j.at("n_draft_accepted_total").get(), 21); - EXPECT_EQ(j.at("n_draft_verify_steps_total").get(), 7); - const json per_pos = j.at("n_draft_accepted_per_pos"); + EXPECT_EQ(j.at("n_draft_verif_steps_total").get(), 7); + const json per_pos = j.at("n_accepted_per_pos_total"); ASSERT_TRUE(per_pos.is_array()); ASSERT_EQ(per_pos.size(), 3u); EXPECT_EQ(per_pos.at(0).get(), 5); @@ -611,7 +611,7 @@ TEST(ServerMetricsToJson, DefaultConstructedHalvesProduceZeroesAndAnEmptySlotArr EXPECT_DOUBLE_EQ(j.at("t_prompt_processing_total").get(), 0.0); ASSERT_TRUE(j.at("slots").is_array()); EXPECT_EQ(j.at("slots").size(), 0u); - EXPECT_TRUE(j.at("n_draft_accepted_per_pos").is_array()); + EXPECT_TRUE(j.at("n_accepted_per_pos_total").is_array()); } TEST(ServerMetricsToJson, CountersAreNumbersNotBooleans) { diff --git a/llama/src/test/java/net/ladenthin/llama/value/ServerMetricsTest.java b/llama/src/test/java/net/ladenthin/llama/value/ServerMetricsTest.java index 3c18732a7..b5fbaea96 100644 --- a/llama/src/test/java/net/ladenthin/llama/value/ServerMetricsTest.java +++ b/llama/src/test/java/net/ladenthin/llama/value/ServerMetricsTest.java @@ -33,8 +33,8 @@ private ServerMetrics parse(String json) throws Exception { // Counters that only became reachable as JSON once the JNI layer started merging the // METRICS and SLOT_GET halves back together (upstream split them at b10408/b10519). + "\"n_prompt_tokens_cached_total\":700,\"n_draft_tokens_total\":40," - + "\"n_draft_accepted_total\":30,\"n_draft_verify_steps_total\":12," - + "\"n_draft_accepted_per_pos\":[20,8,2]," + + "\"n_draft_accepted_total\":30,\"n_draft_verif_steps_total\":12," + + "\"n_accepted_per_pos_total\":[20,8,2]," // next_token is an ARRAY of one object — this mirrors llama.cpp's server_slot::to_json // at b9739, not a bare object; SlotMetrics must unwrap next_token[0]. + "\"slots\":[{\"id\":0,\"n_ctx\":4096,\"is_processing\":true," @@ -209,7 +209,7 @@ public void draftAcceptedPerPositionIsExposedInOrder() throws Exception { @Test public void draftAcceptedPerPositionIsEmptyWhenAbsentOrNotAnArray() throws Exception { assertTrue(parse("{}").getDraftAcceptedPerPosition().isEmpty()); - assertTrue(parse("{\"n_draft_accepted_per_pos\":7}") + assertTrue(parse("{\"n_accepted_per_pos_total\":7}") .getDraftAcceptedPerPosition() .isEmpty()); } From 7be24a673bcebf4eb6105e970305c64192d099ea Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 22:00:07 +0000 Subject: [PATCH 42/76] fix: repair LlamaQuantizer's JNI linkage and the -1 sentinel CI on PR #403 got far enough to reveal that LlamaQuantizer has never worked in any published jar. All three QuantizerIntegrationTest cases failed with UnsatisfiedLinkError on quantizeNative. The extern "C" declarations that give the JNI entry points C linkage come from the javac-generated jllama.h, which covers only LlamaModel. Every other class's JNI function has to declare its own -- train_engine.cpp does it inline on LlamaTrainer_finetuneNative, native_server.cpp wraps the NativeServer entries in a block, and jllama.cpp already has such a block for TextToSpeech. quantizeNative sat between them with nothing, so it was exported as _Z54Java_net_ladenthin_llama_LlamaQuantizer_quantizeNativeP7JNIEnv_P7_jclass... and the JVM could never resolve it. Reproduced with nm -D on a local Linux build, so this was never Windows-specific; the two Windows jobs are simply the only ones whose fork survived long enough to run the test. After the fix the rebuilt library has zero mangled Java_* exports. Nothing caught it because the only coverage was QuantizerIntegrationTest, which gates on a GGUF and therefore skipped in CI for as long as the model paths resolved to the wrong directory. The guard now lives in the model-free NativeLibraryLoadSmokeTest, next to the existing build-info linkage check: a quantize() on a missing input crosses JNI and fails with LlamaException, whereas a linkage regression fails earlier with UnsatisfiedLinkError -- an Error, so it propagates out of assertThrows rather than being swallowed as a mismatch. Also fixes the one genuine b10456->b10618 regression the same CI run exposed: JsonEndpointParametersTest built raw JSON with "dry_penalty_last_n": -1. b10275 gave that field the hard limits [0, INT32_MAX] (0 = disabled) and dropped the "-1 = context size" sentinel, so the request now 400s. The setters were already fixed in this PR; this test bypassed them. A repo-wide sweep confirms it was the only remaining -1 on either de-sentinelled field. TODO.md records the four items that stay open, with the evidence for each -- including that neither the SessionForkRewind slot-restore failures nor the NativeServerAttach 500 traces to any upstream change in this bump range, and that the Ubuntu job reached only 85 tests before TtsIntegrationTest killed the fork, so the 1461-test Windows list is a lower bound on what is broken. CLAUDE.md no longer claims the TTS pipeline is covered end-to-end. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CHANGELOG.md | 10 +++ CLAUDE.md | 2 +- TODO.md | 65 +++++++++++++++++++ llama/src/main/cpp/jllama.cpp | 10 ++- .../loader/NativeLibraryLoadSmokeTest.java | 40 ++++++++++++ .../JsonEndpointParametersTest.java | 6 +- 6 files changed, 128 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d0851d799..043f7394a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -76,6 +76,16 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by reading (`ggml_time_us()`), not milliseconds since the epoch. The value is unchanged. ### Fixed +- **`LlamaQuantizer` never worked in any published jar — every call threw `UnsatisfiedLinkError`.** + The `extern "C"` declarations that give the JNI entry points C linkage come from the + javac-generated `jllama.h`, which covers **only** `LlamaModel`; a JNI function for any other class + has to declare its own (as `train_engine.cpp` and `native_server.cpp` do). + `Java_net_ladenthin_llama_LlamaQuantizer_quantizeNative` did not, so it was exported under its + C++-mangled name and the JVM could never resolve it — on every platform, not just the two Windows + jobs that reported it. The only coverage was `QuantizerIntegrationTest`, which gates on a GGUF and + so skipped in CI for as long as the model paths resolved to the wrong directory. Fixed, and guarded + model-free by `NativeLibraryLoadSmokeTest.quantizerNativeEntryPointResolves` so a future entry point + that forgets `extern "C"` fails a test that runs wherever the library exists. - **The macOS arm64 native library shipped corrupt in 5.0.6 and in several 5.0.7 snapshots.** All three macOS arm64 build jobs uploaded their dylib under a `*-libraries` artifact name, and the packaging job collects those with one globbed download — so three builds landed on the same diff --git a/CLAUDE.md b/CLAUDE.md index bff837ab7..eca4ae865 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1435,7 +1435,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" | `src/test/cpp/test_json_helpers.cpp` | 60 | All functions in `json_helpers.hpp`: `get_result_error_message`, `results_to_json`, `rerank_results_to_json` (incl. missing/out-of-range `index` rejection), `parse_encoding_format`, `extract_embedding_prompt`, `is_infill_request`, `parse_slot_prompt_similarity`, `parse_positive_int_config`, `wrap_stream_chunk`, `server_metrics_to_json` | | `src/test/cpp/test_log_helpers.cpp` | 13 | All functions in `log_helpers.hpp`: `log_level_name`, `format_log_as_json` | | `src/test/cpp/test_jni_helpers.cpp` | 56 | All functions in `jni_helpers.hpp` using a zero-filled `JNINativeInterface_` mock (incl. the `utf8_to_jstring_impl` byte-array string path: emoji byte-preservation, truncated-UTF-8 replace-not-throw) | -| `src/test/cpp/test_tts_wav.cpp` | 2 | The in-memory WAV writer `pcm_to_wav16_bytes` in `tts_wav.hpp` (WAV header/payload + little-endian clamping) — our own code, not upstream. The Qwen3-TTS pipeline it pairs with (`mtmd_helper::gen_audio`) is entirely upstream-owned (no project-side DSP to unit-test here) and covered end-to-end by the Java `TtsIntegrationTest`. | +| `src/test/cpp/test_tts_wav.cpp` | 2 | The in-memory WAV writer `pcm_to_wav16_bytes` in `tts_wav.hpp` (WAV header/payload + little-endian clamping) — our own code, not upstream. The Qwen3-TTS pipeline it pairs with (`mtmd_helper::gen_audio`) is entirely upstream-owned (no project-side DSP to unit-test here). **It is NOT currently covered end-to-end**: `TtsIntegrationTest` aborts the JVM natively on every platform CI runs (SIGABRT on Linux, non-zero exit on macOS and Windows) — see `TODO.md`. Until that is fixed, these 2 WAV-writer tests are the only automated coverage the TTS surface has. | **Current total: 499 tests (all passing).** diff --git a/TODO.md b/TODO.md index 3125acb00..3038eacd5 100644 --- a/TODO.md +++ b/TODO.md @@ -13,6 +13,71 @@ cross-cutting initiative. ## Open — jllama-specific +### Model-backed tests that the CI-skip fix newly exposed (b10618 PR #403) + +Making the model-gated suite actually run in CI (`TestConstants.resolveModelPath`, see the Done +section) turned a green-but-silent pipeline into a red-and-honest one. Everything below was already +broken before this PR; none of it was visible while every model-gated class self-skipped. Two items +were fixed in this PR, four are open. + +**How to read the CI evidence.** Surefire runs classes in filesystem order, which differs per OS, and +`TtsIntegrationTest` kills the fork. On Linux/macOS it lands early (Ubuntu got through only **85** +tests), so those jobs report *nothing but* the crash. The two Windows jobs happen to run it last and +therefore reach **1461** tests — they are the only jobs whose failure list is complete. Do not read a +short Linux failure list as "Linux is healthier". + +- **[OPEN, MAJOR] `TtsIntegrationTest` aborts the JVM natively on all 6 test platforms.** + Ubuntu exit 134 (SIGABRT); macOS 14 Metal, macOS 15 Metal, macOS 15 no-Metal; Windows Ninja and + Windows MSVC exit 1. Not an OOM (Linux had ~14 GiB free, Windows ~12.3 GiB of 16 GiB). Not caused by + the bump: `git log b10456..b10618 -- tools/mtmd/mtmd-helper.{cpp,h}` contains only video/webp/ + mergeable/sha256/cmake commits, and grepping that diff for `gen_audio|step_gen|step_prompt| + get_output|GGML_ASSERT|GGML_ABORT|throw` yields zero hits. It is a latent defect in + `tts_engine.cpp`'s drive of `mtmd_helper::gen_audio`, exposed the first time the test ran. + Next step: read `hs_err_pid*.log` from artifact `windows-output` (ID 9583568326) or + `error-log-macos-14-metal` (ID 9583085009) of run 32899147975 for the aborting frame. Because it + takes the whole fork down it also truncates every job's test run, so it blocks seeing the rest of + the suite and should be fixed first. + +- **[OPEN] `SessionForkRewindIntegrationTest` — empty reply after a slot restore (2 failures).** + `rewindRestoresTranscriptAndConversationContinues:81` and + `forkCreatesIndependentSessionWithSameTranscript:98` both fail `assertThat(reply.isEmpty(), + is(false))`: generation returns an empty string after `rewind()` / `fork()`, i.e. after slot state + is restored from a file. The transcript assertions around them pass, so it is the KV restore, not + the bookkeeping. Checked and **not** a bump regression: the only slot-touching upstream commit in + b10456..b10618 is `e8eed4525` (`LLAMA_SERVER_SLOTS_N_DIFF`), which is env-var opt-in and defaults + to 0, hence inert. + +- **[OPEN] `NativeServerAttachIntegrationTest.completion_overHttp_served:108` — HTTP 500.** + `{"error":{"code":500,"message":"The model produced output that does not match the expected + Content-only format"}}`. `"Content-only"` is a `common_chat_format` name (`common/chat.cpp:856`), so + the request ran without a template and the reply failed to parse under that format. Checked and + **not** a bump regression: the error string is byte-identical at b10456 and b10618, and the only + upstream commit touching `common/chat.cpp` in the range is the `common_json` abstraction (#27511). + +- **[OPEN] Re-check the full suite once the TTS crash is fixed.** No platform has yet completed a run + past `TtsIntegrationTest` on Linux or macOS, so the 1461-test Windows list is the *lower* bound on + what is broken, not the complete picture. + +- **[FIXED in this PR] `LlamaQuantizer.quantizeNative` had C++ linkage — the whole class was + unusable in every published jar.** `QuantizerIntegrationTest` failed all 3 tests with + `UnsatisfiedLinkError: 'void net.ladenthin.llama.LlamaQuantizer.quantizeNative(...)'`. Cause: the + C-linkage declarations come from the javac-generated `jllama.h`, which covers **only** + `LlamaModel`; every other class's JNI function must say `extern "C"` itself (`train_engine.cpp` and + `native_server.cpp` do). `quantizeNative` did not, so it was exported as + `_Z54Java_net_ladenthin_llama_LlamaQuantizer_quantizeNativeP7JNIEnv_...` and the JVM could never + resolve it. Reproduced on Linux with `nm -D`, so it was never Windows-specific — the public + `LlamaQuantizer` API has never worked. Fixed, and guarded model-free by + `NativeLibraryLoadSmokeTest.quantizerNativeEntryPointResolves` (`nm -D` on the rebuilt lib now shows + zero mangled `Java_*` exports). + +- **[FIXED in this PR] `JsonEndpointParametersTest.testDryMultiplierAccepted` sent + `dry_penalty_last_n: -1`.** The one genuine b10456→b10618 regression in the list: b10275 gave the + field hard limits `[0, INT32_MAX]` (0 = disabled) and dropped the old "-1 = context size" sentinel, + so the request now 400s. The `InferenceParameters` / `ModelParameters` setters were already fixed in + this PR; this test builds raw JSON and bypassed them. A repo-wide sweep confirms it was the only + remaining `-1` on either de-sentinelled field. + + ### LlamaLoader extraction-directory isolation (optional follow-up, low priority) Left over from the 2026-06-20 code audit (18/18 findings fixed in PRs #258/#260, regression tests in diff --git a/llama/src/main/cpp/jllama.cpp b/llama/src/main/cpp/jllama.cpp index 3d02e5579..afcdaf17f 100644 --- a/llama/src/main/cpp/jllama.cpp +++ b/llama/src/main/cpp/jllama.cpp @@ -1707,9 +1707,13 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_setLoraAdaptersJso return dispatch_one_shot_task(env, jctx, std::move(task)); } -JNIEXPORT void JNICALL Java_net_ladenthin_llama_LlamaQuantizer_quantizeNative(JNIEnv *env, jclass, jstring jinput, - jstring joutput, jint ftype, jint nthread, - jboolean allowRequantize) { +// LlamaQuantizer is not part of the javac-generated jllama.h (that header only covers LlamaModel), +// so nothing else gives this definition C linkage and it would be exported under its C++-mangled +// name -- which the JVM cannot resolve, failing every call with UnsatisfiedLinkError on every +// platform. Same reason train_engine.cpp marks LlamaTrainer_finetuneNative and native_server.cpp +// wraps the NativeServer entry points in an extern "C" block. +extern "C" JNIEXPORT void JNICALL Java_net_ladenthin_llama_LlamaQuantizer_quantizeNative( + JNIEnv *env, jclass, jstring jinput, jstring joutput, jint ftype, jint nthread, jboolean allowRequantize) { try { const std::string input_path = parse_jstring(env, jinput); const std::string output_path = parse_jstring(env, joutput); diff --git a/llama/src/test/java/net/ladenthin/llama/loader/NativeLibraryLoadSmokeTest.java b/llama/src/test/java/net/ladenthin/llama/loader/NativeLibraryLoadSmokeTest.java index 028456f25..c4a791c73 100644 --- a/llama/src/test/java/net/ladenthin/llama/loader/NativeLibraryLoadSmokeTest.java +++ b/llama/src/test/java/net/ladenthin/llama/loader/NativeLibraryLoadSmokeTest.java @@ -6,11 +6,15 @@ import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; import static org.junit.jupiter.api.Assumptions.assumeTrue; import net.ladenthin.llama.ClaudeGenerated; import net.ladenthin.llama.LlamaModel; +import net.ladenthin.llama.LlamaQuantizer; +import net.ladenthin.llama.args.QuantizationType; +import net.ladenthin.llama.exception.LlamaException; import net.ladenthin.llama.value.LlamaCppVersion; import org.junit.jupiter.api.Test; @@ -83,4 +87,40 @@ void nativeBuildInfoMatchesPinnedVersionConstant() { + LlamaCppVersion.LLAMA_CPP_VERSION + "-\"; if this fails, GIT_TAG in " + "llama/CMakeLists.txt and LlamaCppVersion.LLAMA_CPP_VERSION have drifted apart"); } + + /** + * {@link LlamaQuantizer#quantize} must reach its native implementation. The declarations that + * give the {@code LlamaModel} entry points C linkage come from the javac-generated + * {@code jllama.h}, which covers only {@code LlamaModel} — so every JNI function for + * any other class has to say {@code extern "C"} itself. {@code quantizeNative} did not, and was + * therefore exported under its C++-mangled name, making this public API throw + * {@link UnsatisfiedLinkError} on every platform in every published jar. + * + *

Nothing caught it: the only coverage was {@code QuantizerIntegrationTest}, which gates on a + * GGUF being present and so skipped in CI for as long as the model paths resolved to the wrong + * directory. Asserting it here — model-free, next to the build-info linkage check — means a + * future entry point that forgets {@code extern "C"} fails a test that always runs wherever the + * library exists. + * + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10618"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10631"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10618-"} — call + * plus the resolved upstream commit, e.g. {@code "b10631-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10618"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10631"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10618"; + public static final String LLAMA_CPP_VERSION = "b10631"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 999034b3f5a3a2c35fccf1a7726d1e822d70dd95 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 09:00:10 +0000 Subject: [PATCH 49/76] fix: resolve CPU params on hand-built common_params (TTS abort) TtsIntegrationTest killed the surefire fork on all six test platforms (SIGSEGV / SIGABRT / exit 1) while loading the model, taking ~1300 unrelated tests down with it. Root cause is ours, not upstream's. common/arg.cpp is the ONLY caller of postprocess_cpu_params -- common_init_from_params does not call it -- so a common_params assembled by hand never gets its CPU fields resolved. tts_engine.cpp set only cpuparams.n_threads, leaving cpuparams_batch at its -1 default. common_threadpools::init then found tpp and tpp_batch mismatched and built a separate batch pool with -1 threads; ggml_threadpool_new sizes its worker array as sizeof(struct ggml_compute_state) * n_threads, which for -1 wraps to a huge size_t, so ggml_aligned_malloc returned NULL and the unchecked memset(workers, 0, workers_size) on the next line faulted at address 0 -- the __bzero (macOS) / bare libc (Linux) frame with RDI=0, RSI=0 in the crash logs. Both hand-built-params sites are fixed by mirroring arg.cpp's two calls, including the role_model argument that makes the batch pool inherit the main pool's count rather than resolve independently: - tts_engine.cpp (the crash) - train_engine.cpp (same latent defect: it set neither count, so both stayed -1; they matched, so it built one broken pool instead of two) jllama.cpp and jni_helpers.hpp are unaffected -- their params come from common_params_parse, so arg.cpp resolves them. The builder moved to a new tts_params.hpp so the guard exercises the real production path rather than a copy that could drift: jllama_test does not compile tts_engine.cpp (that TU needs the mtmd/llama runtime), and a header is what lets the test reach it. test_tts_params.cpp adds 5 model-free tests, suite 499 -> 504, all passing. Verified by negative control: deleting the two calls turns TtsParams.ResolvesBothCpuThreadCounts red with "actual: -1 vs 0" rather than leaving it silently green. TODO.md records the localisation method (crash log -> register convention -> disassembly) and marks the memory-pressure hypothesis it superseded as wrong, since the failing allocation's size is a compile-time -1 and does not depend on free RAM. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CHANGELOG.md | 9 +++ CLAUDE.md | 4 +- TODO.md | 46 +++++++++++---- llama/CMakeLists.txt | 1 + llama/src/main/cpp/train_engine.cpp | 11 ++++ llama/src/main/cpp/tts_engine.cpp | 15 ++--- llama/src/main/cpp/tts_params.hpp | 62 ++++++++++++++++++++ llama/src/test/cpp/test_tts_params.cpp | 81 ++++++++++++++++++++++++++ 8 files changed, 206 insertions(+), 23 deletions(-) create mode 100644 llama/src/main/cpp/tts_params.hpp create mode 100644 llama/src/test/cpp/test_tts_params.cpp diff --git a/CHANGELOG.md b/CHANGELOG.md index e5e3cc063..fdba2b152 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -84,6 +84,15 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by reading (`ggml_time_us()`), not milliseconds since the epoch. The value is unchanged. ### Fixed +- **`TextToSpeech` crashed the JVM on every platform when loading a model.** A hand-built + `common_params` never passes through `common_params_parse`, and `common/arg.cpp` is upstream's + only caller of `postprocess_cpu_params` — `common_init_from_params` does not call it. So + `cpuparams_batch.n_threads` kept its `-1` default, `common_threadpools::init` created a second + threadpool with -1 threads, and `ggml_threadpool_new` sized its worker array as + `sizeof(ggml_compute_state) * -1` — a huge `size_t`, so the allocation returned `NULL` and the + unchecked `memset` that follows it faulted at address 0. `tts_engine.cpp` and `train_engine.cpp` now mirror `arg.cpp`'s two + calls; the `LlamaModel` paths were never affected because their params are parsed. Guarded by + five model-free C++ tests over the extracted `build_tts_params`. - **`LlamaQuantizer` never worked in any published jar — every call threw `UnsatisfiedLinkError`.** The `extern "C"` declarations that give the JNI entry points C linkage come from the javac-generated `jllama.h`, which covers **only** `LlamaModel`; a JNI function for any other class diff --git a/CLAUDE.md b/CLAUDE.md index 2f56dde06..9af0f1ba1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1435,9 +1435,9 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" | `src/test/cpp/test_json_helpers.cpp` | 60 | All functions in `json_helpers.hpp`: `get_result_error_message`, `results_to_json`, `rerank_results_to_json` (incl. missing/out-of-range `index` rejection), `parse_encoding_format`, `extract_embedding_prompt`, `is_infill_request`, `parse_slot_prompt_similarity`, `parse_positive_int_config`, `wrap_stream_chunk`, `server_metrics_to_json` | | `src/test/cpp/test_log_helpers.cpp` | 13 | All functions in `log_helpers.hpp`: `log_level_name`, `format_log_as_json` | | `src/test/cpp/test_jni_helpers.cpp` | 56 | All functions in `jni_helpers.hpp` using a zero-filled `JNINativeInterface_` mock (incl. the `utf8_to_jstring_impl` byte-array string path: emoji byte-preservation, truncated-UTF-8 replace-not-throw) | -| `src/test/cpp/test_tts_wav.cpp` | 2 | The in-memory WAV writer `pcm_to_wav16_bytes` in `tts_wav.hpp` (WAV header/payload + little-endian clamping) — our own code, not upstream. The Qwen3-TTS pipeline it pairs with (`mtmd_helper::gen_audio`) is entirely upstream-owned (no project-side DSP to unit-test here). **It is NOT currently covered end-to-end**: `TtsIntegrationTest` aborts the JVM natively on every platform CI runs (SIGABRT on Linux, non-zero exit on macOS and Windows) — see `TODO.md`. Until that is fixed, these 2 WAV-writer tests are the only automated coverage the TTS surface has. | +| `src/test/cpp/test_tts_wav.cpp` | 2 | The in-memory WAV writer `pcm_to_wav16_bytes` in `tts_wav.hpp` (WAV header/payload + little-endian clamping) — our own code, not upstream. The Qwen3-TTS pipeline it pairs with (`mtmd_helper::gen_audio`) is entirely upstream-owned (no project-side DSP to unit-test here). The load path is additionally covered by `test_tts_params.cpp` (5 tests over `tts_params.hpp`'s `build_tts_params`), which pins the CPU-thread resolution whose absence used to crash the JVM on every platform — see the `TODO.md` entry for the mechanism. End-to-end coverage is `TtsIntegrationTest`, which is model-gated. | -**Current total: 499 tests (all passing).** +**Current total: 504 tests (all passing).** #### Upstream source location (in CMake build tree) diff --git a/TODO.md b/TODO.md index 71f78a6e0..2ea637093 100644 --- a/TODO.md +++ b/TODO.md @@ -32,10 +32,32 @@ the crash and reproduced `SessionForkRewindIntegrationTest` (both cases) and not a Windows quirk. Its exit code was **141 (SIGPIPE)** rather than Ubuntu's 134 (SIGABRT), on the same crashed test. -- **[OPEN, MAJOR — now localised] `TtsIntegrationTest` aborts the JVM natively on all 6 test - platforms.** The crash log is readable in the job log itself since `cbb5e62` (the section-3.1 - print step), which is what finally produced the following. The abort is a **null-pointer - dereference while zeroing a buffer during the TTS model load**, identically on two OS families: +- **[FIXED — root cause found, awaiting CI confirmation] `TtsIntegrationTest` aborted the JVM + natively on all 6 test platforms.** **Root cause: our own hand-built `common_params`.** + `common_cpu_params::n_threads` defaults to **-1**, and `postprocess_cpu_params` — the function + that resolves it — is called **only** from `common/arg.cpp`, i.e. only for params that came + through `common_params_parse`. `common_init_from_params` does not call it. `tts_engine.cpp` + assembled `common_params` by hand and set only `cpuparams.n_threads`, so `cpuparams_batch` + stayed at -1; `common_threadpools::init` then saw a mismatch between the two and built a + *second* threadpool with **-1 threads**. `ggml_threadpool_new` sizes its worker array as + `sizeof(struct ggml_compute_state) * tpp->n_threads`, which for -1 wraps to a huge `size_t`; + `ggml_aligned_malloc` returns `NULL` (after logging *"insufficient memory"*) and the next line + is an unchecked `memset(workers, 0, workers_size)` — the `bzero`-at-address-0 seen on every + platform. + Fixed by mirroring `arg.cpp`'s two calls, including the `role_model` argument that makes the + batch pool inherit rather than resolve independently. `train_engine.cpp` had the same latent + defect (it set *neither* count, so both stayed -1 — they matched, so it built one pool with + -1 threads instead of two) and got the same fix. `jllama.cpp` / `jni_helpers.hpp` are safe: + their params come from `common_params_parse`, so `arg.cpp` resolves them. + Guarded by 5 new model-free C++ tests (`test_tts_params.cpp`, suite 499 → **504**), verified + by negative control: removing the two calls turns `TtsParams.ResolvesBothCpuThreadCounts` red + with `actual: -1 vs 0`. The builder was extracted to `tts_params.hpp` so the test exercises + the *real* production path rather than a copy that could drift. + + **How it was localised** (kept because the method generalises, not because the bug is still + open). The crash log became readable in the job log itself with `cbb5e62` (the section-3.1 print + step), and that is what produced everything below. The abort is a **null-pointer dereference + while zeroing a buffer during the TTS model load**, identically on two OS families: | | Linux x86-64 (run 32941522341) | macOS 15 arm64 (same run) | |---|---|---| @@ -65,16 +87,18 @@ same crashed test. the PLT, so not inlined), `operator delete` and `__stack_chk_fail`. The JVM's frame-pointer walker dropped the intermediate frames, leaving only the outermost and innermost. The fault is therefore under `engine_init` — in `common_init_from_params` or the mtmd/mmproj init — not in the - JNI wrapper. (Symbol attribution itself *is* trustworthy here: the library exports 12 529 symbols, + JNI wrapper. (That much held up: it is in `common_init_from_params`, via + `common_threadpools::init`.) (Symbol attribution itself *is* trustworthy here: the library exports 12 529 symbols, the whole llama/common layer included, so a PC inside `common_init_from_params` would have been named as such. What is unreliable is the *depth* of the walk, not the naming.) - Worth noting for the next step: the macOS runner has **7 GB RAM / 3 cores** against Linux's - 15 GB / 4, yet both fail the same way, so a plain out-of-memory on the larger host is a weak - explanation on its own — but an allocation that returns null and is then zeroed unchecked fits - both. The `mmproj` (speaker encoder + code predictor + code2wav) is the largest new buffer in - this path. The native stack could not be walked on Linux (one libc frame only); the full - `hs_err` in the artifact has the memory map if the allocation size matters. + The shape of the guess at that point — *an allocation that returns null and is then zeroed + unchecked* — was right; the attribution was not. It was blamed on host memory pressure (the + macOS runner has **7 GB RAM / 3 cores** against Linux's 15 GB / 4) and on the `mmproj` being the + largest new buffer in the path. Neither is involved: the request is for + `sizeof(struct ggml_compute_state) * (size_t) -1` bytes, which no allocator can satisfy on any + host with any amount of RAM free. That both hosts failed identically was the clue that memory + pressure could not be the explanation. Superseded note (kept because the reasoning was cited earlier): it was NOT certain an `hs_err` existed at all — `if-no-files-found: warn` and Windows' exit code 1 left that open. It does diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 2ee30f53d..748865664 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -567,6 +567,7 @@ if(BUILD_TESTING) src/test/cpp/test_json_helpers.cpp src/test/cpp/test_log_helpers.cpp src/test/cpp/test_tts_wav.cpp + src/test/cpp/test_tts_params.cpp ${llama.cpp_SOURCE_DIR}/tools/server/server-common.cpp ${llama.cpp_SOURCE_DIR}/tools/server/server-chat.cpp ${llama.cpp_SOURCE_DIR}/tools/server/server-context.cpp diff --git a/llama/src/main/cpp/train_engine.cpp b/llama/src/main/cpp/train_engine.cpp index e06110fb7..add1da01b 100644 --- a/llama/src/main/cpp/train_engine.cpp +++ b/llama/src/main/cpp/train_engine.cpp @@ -65,6 +65,17 @@ bool finetune(const finetune_config &cfg, std::string &err) { params.cache_type_k = GGML_TYPE_F32; params.cache_type_v = GGML_TYPE_F32; + // A hand-built common_params never passes through common_params_parse, and common/arg.cpp is + // the ONLY caller of postprocess_cpu_params -- common_init_from_params does not call it. Without + // these two lines common_cpu_params::n_threads keeps its -1 default, and common_threadpools::init + // hands that -1 to ggml_threadpool_new, whose workers_size = sizeof(ggml_compute_state) * + // n_threads then overflows to a huge size_t; the allocation returns NULL and the following + // memset is unchecked, so the process dies on memset(NULL, 0, huge) -- SIGSEGV at address 0. + // Mirrors common/arg.cpp's own two calls, including the role_model that makes the batch pool + // inherit rather than resolve independently. + postprocess_cpu_params(params.cpuparams, nullptr); + postprocess_cpu_params(params.cpuparams_batch, ¶ms.cpuparams); + llama_backend_init(); llama_numa_init(params.numa); diff --git a/llama/src/main/cpp/tts_engine.cpp b/llama/src/main/cpp/tts_engine.cpp index 67bd20baf..d864dc6e9 100644 --- a/llama/src/main/cpp/tts_engine.cpp +++ b/llama/src/main/cpp/tts_engine.cpp @@ -6,7 +6,8 @@ #include "tts_engine.h" -#include "tts_wav.hpp" // pcm_to_wav16_bytes +#include "tts_params.hpp" // build_tts_params +#include "tts_wav.hpp" // pcm_to_wav16_bytes #include "common.h" #include "llama.h" @@ -39,15 +40,9 @@ tts_engine *engine_init(const std::string &model_path, const std::string &mmproj auto engine = new tts_engine(); engine->n_threads = n_threads > 0 ? n_threads : 4; - common_params params; - params.n_ctx = 8192; - params.n_batch = engine->n_batch; - params.n_gpu_layers = n_gpu_layers; - params.cpuparams.n_threads = engine->n_threads; - params.model.path = model_path; - // Always enable embd so the backbone's hidden state can be handed to the audio-generation - // helper between frames (mirrors upstream tools/tts/tts.cpp main()). - params.embedding = true; + // Built by tts_params.hpp so the exact params the engine uses are unit-testable without a + // model -- see the header for why the CPU-thread resolution there is load-bearing. + common_params params = build_tts_params(model_path, n_gpu_layers, engine->n_threads, engine->n_batch); engine->init = common_init_from_params(params); engine->model = engine->init ? engine->init->model() : nullptr; diff --git a/llama/src/main/cpp/tts_params.hpp b/llama/src/main/cpp/tts_params.hpp new file mode 100644 index 000000000..163690341 --- /dev/null +++ b/llama/src/main/cpp/tts_params.hpp @@ -0,0 +1,62 @@ +// SPDX-FileCopyrightText: 2026 Bernard Ladenthin +// +// SPDX-License-Identifier: MIT +// +// The common_params builder for the TTS engine, split out of tts_engine.cpp so it can be tested +// without a GGUF, a JVM or a loaded model -- the same reason json_helpers.hpp exists. +// +// It is a header, not a .cpp, because jllama_test deliberately does not compile tts_engine.cpp +// (that TU needs the mtmd/llama runtime); a header lets the test exercise the real builder the +// engine uses, rather than a copy that could drift away from it. + +#ifndef JLLAMA_TTS_PARAMS_HPP +#define JLLAMA_TTS_PARAMS_HPP + +#include "common.h" + +#include + +namespace jllama_tts { + +/// Context size the TTS backbone is initialised with. +inline constexpr int TTS_N_CTX = 8192; + +/// Fallback thread count when the caller passes a non-positive value. +inline constexpr int TTS_DEFAULT_THREADS = 4; + +// --------------------------------------------------------------------------- +// Builds the common_params for the TTS backbone. +// +// The two postprocess_cpu_params calls are the load-bearing part. common/arg.cpp is the ONLY +// caller of that function in upstream, so a common_params assembled by hand -- as this one is -- +// never gets its CPU fields resolved: common_cpu_params::n_threads defaults to -1, and +// common_init_from_params does not fix it up. common_threadpools::init then finds tpp and +// tpp_batch mismatched, builds a separate batch pool, and ggml_threadpool_new computes +// workers_size = sizeof(struct ggml_compute_state) * tpp->n_threads +// with n_threads == -1 -- a huge size_t. ggml_aligned_malloc returns NULL and the very next line +// memsets it unchecked, so the process dies on memset(NULL, 0, huge): SIGSEGV at address 0, +// reported as __bzero on macOS and as a bare libc frame on Linux. The role_model argument is what +// makes the batch pool inherit the main pool's count instead of staying at -1; passing nullptr for +// both would resolve each independently. This mirrors arg.cpp exactly. +// --------------------------------------------------------------------------- +[[nodiscard]] inline common_params build_tts_params(const std::string &model_path, int n_gpu_layers, int n_threads, + int n_batch) { + common_params params; + params.n_ctx = TTS_N_CTX; + params.n_batch = n_batch; + params.n_gpu_layers = n_gpu_layers; + params.cpuparams.n_threads = n_threads > 0 ? n_threads : TTS_DEFAULT_THREADS; + params.model.path = model_path; + // Always enable embd so the backbone's hidden state can be handed to the audio-generation + // helper between frames (mirrors upstream tools/tts/tts.cpp main()). + params.embedding = true; + + postprocess_cpu_params(params.cpuparams, nullptr); + postprocess_cpu_params(params.cpuparams_batch, ¶ms.cpuparams); + + return params; +} + +} // namespace jllama_tts + +#endif // JLLAMA_TTS_PARAMS_HPP diff --git a/llama/src/test/cpp/test_tts_params.cpp b/llama/src/test/cpp/test_tts_params.cpp new file mode 100644 index 000000000..bfd643d6e --- /dev/null +++ b/llama/src/test/cpp/test_tts_params.cpp @@ -0,0 +1,81 @@ +// SPDX-FileCopyrightText: 2026 Bernard Ladenthin +// +// SPDX-License-Identifier: MIT +// +// Guards the TTS common_params builder. Model-free, JVM-free, runs in the ordinary C++ suite on +// every platform. +// +// Why this exists even though TtsIntegrationTest already covers it: that test does cover it, but +// only by loading a 1.7B GGUF plus an mmproj, and it reports the defect as a native SIGSEGV that +// kills the whole surefire fork and takes ~1300 unrelated tests down with it. Turning that into a +// one-line assertion is the entire point -- the failure it pins cost a full debugging cycle +// (hs_err retrieval, cross-platform frame comparison, disassembly) to identify. + +#include "tts_params.hpp" + +#include + +namespace { + +// --------------------------------------------------------------------------- +// The regression guard. It fails if anyone drops the postprocess_cpu_params calls from +// build_tts_params, because the builder -- not a copy of it -- is what is exercised here. +// --------------------------------------------------------------------------- +TEST(TtsParams, ResolvesBothCpuThreadCounts) { + const common_params params = jllama_tts::build_tts_params("model.gguf", 0, 4, 2048); + + EXPECT_EQ(params.cpuparams.n_threads, 4); + // The one that actually crashed: left at -1, common_threadpools::init creates a second, + // broken threadpool because tpp and tpp_batch no longer match. + EXPECT_GT(params.cpuparams_batch.n_threads, 0); + EXPECT_EQ(params.cpuparams_batch.n_threads, params.cpuparams.n_threads); +} + +// A non-positive thread count must fall back, and the batch pool must still inherit it. +TEST(TtsParams, NonPositiveThreadCountFallsBackAndStillInherits) { + for (const int requested : {0, -1, -8}) { + const common_params params = jllama_tts::build_tts_params("model.gguf", 0, requested, 2048); + + EXPECT_EQ(params.cpuparams.n_threads, jllama_tts::TTS_DEFAULT_THREADS) << "requested=" << requested; + EXPECT_EQ(params.cpuparams_batch.n_threads, params.cpuparams.n_threads) << "requested=" << requested; + } +} + +// The remaining fields the engine depends on, so a future edit to the builder cannot silently +// drop one of them. +TEST(TtsParams, CarriesTheEngineSettings) { + const common_params params = jllama_tts::build_tts_params("some/model.gguf", 33, 6, 1024); + + EXPECT_EQ(params.model.path, "some/model.gguf"); + EXPECT_EQ(params.n_gpu_layers, 33); + EXPECT_EQ(params.n_batch, 1024u); + EXPECT_EQ(params.n_ctx, jllama_tts::TTS_N_CTX); + // gen_audio hands the backbone's hidden state back between frames; without embd it is absent. + EXPECT_TRUE(params.embedding); +} + +// --------------------------------------------------------------------------- +// Pins the upstream trap the builder works around, in the same spirit as the CommonJsonEnumTrap +// pair in test_json_helpers.cpp. If upstream ever gives common_cpu_params a sane default, this +// test flips to red and tells us the workaround can be dropped -- which no other signal would. +// --------------------------------------------------------------------------- +TEST(CommonParamsCpuTrap, RawDefaultsAreUnresolved) { + common_params raw; // default-init on purpose: {} would value-initialise and hide the defaults + + EXPECT_LT(raw.cpuparams.n_threads, 0) << "upstream default changed -- re-check build_tts_params"; + EXPECT_LT(raw.cpuparams_batch.n_threads, 0) << "upstream default changed -- re-check build_tts_params"; +} + +// postprocess_cpu_params' role_model argument is what makes the batch pool inherit rather than +// resolve independently; passing nullptr twice would leave the two pools mismatched. +TEST(CommonParamsCpuTrap, RoleModelMakesBatchInheritTheMainCount) { + common_params params; + params.cpuparams.n_threads = 3; + + postprocess_cpu_params(params.cpuparams, nullptr); + postprocess_cpu_params(params.cpuparams_batch, ¶ms.cpuparams); + + EXPECT_EQ(params.cpuparams_batch.n_threads, 3); +} + +} // namespace From 03e4a0596f4da0b7295fdca1cd00a97f1fb4049a Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 09:47:14 +0000 Subject: [PATCH 50/76] fix: give the router worker JVM the main classes on its classpath RouterModeIntegrationTest.setup failed in CI with IllegalState: Router worker for model 'Qwen3-0.6B-Q4_K_M' failed with exit code 1 and the worker's own stderr -- reachable only through the surefire .dumpstream that the section-3.1 print step echoes into the job log -- said Error: Could not find or load main class net.ladenthin.llama.server.NativeServer Caused by: java.lang.ClassNotFoundException: ... target/classes carries a module-info.class, so Surefire auto-detects a named module and runs the main classes on the MODULE path; java.class.path then holds only target/test-classes plus the dependency jars. The test built the worker command from that property alone, so the spawned JVM had neither NativeServer nor the packaged native library it has to extract. Not a bump regression and not a router defect. It had never surfaced because the test self-skipped in CI for as long as the model paths resolved to the wrong directory -- the TestConstants.resolveModelPath fix earlier in this PR is what made it run for the first time. The main-classes root is now derived from NativeServer's own code source, which is correct whether Surefire chose module or classpath mode and whether the root is a directory or a jar. The existing no-whitespace assumption still applies, now to the combined value. Reproduced and verified locally, without a model or a router: launching a JVM with the old classpath exits 1 on ClassNotFoundException; with the new one the worker loads libjllama.so and reaches llama-server's own argument parser (rejecting a deliberately bogus flag). The check ran against the shipped helper itself, reflectively, not against a copy of its logic. TODO.md also records the CI confirmation of the TTS fix: Ubuntu on f9c43dc ran 1689 tests where the fork previously died at 85, and wrote no hs_err_pid log at all. Of the four remaining items exactly one was new -- this router failure -- and the other three were already recorded. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- TODO.md | 36 ++++++++++++++++--- .../server/RouterModeIntegrationTest.java | 35 +++++++++++++++++- 2 files changed, 65 insertions(+), 6 deletions(-) diff --git a/TODO.md b/TODO.md index 2ea637093..60ab0e6d7 100644 --- a/TODO.md +++ b/TODO.md @@ -32,8 +32,8 @@ the crash and reproduced `SessionForkRewindIntegrationTest` (both cases) and not a Windows quirk. Its exit code was **141 (SIGPIPE)** rather than Ubuntu's 134 (SIGABRT), on the same crashed test. -- **[FIXED — root cause found, awaiting CI confirmation] `TtsIntegrationTest` aborted the JVM - natively on all 6 test platforms.** **Root cause: our own hand-built `common_params`.** +- **[FIXED — confirmed in CI] `TtsIntegrationTest` aborted the JVM natively on all 6 test + platforms.** **Root cause: our own hand-built `common_params`.** `common_cpu_params::n_threads` defaults to **-1**, and `postprocess_cpu_params` — the function that resolves it — is called **only** from `common/arg.cpp`, i.e. only for params that came through `common_params_parse`. `common_init_from_params` does not call it. `tts_engine.cpp` @@ -54,6 +54,10 @@ same crashed test. with `actual: -1 vs 0`. The builder was extracted to `tts_params.hpp` so the test exercises the *real* production path rather than a copy that could drift. + **CI confirmation** (Ubuntu, run 32950691947 on `f9c43dc`): **1689 tests ran** where the fork + previously died at 85, and the crash-print step found **no `hs_err_pid*.log` at all**. The only + thing it printed was the router worker's stderr (see the router entry below). + **How it was localised** (kept because the method generalises, not because the bug is still open). The crash log became readable in the job log itself with `cbb5e62` (the section-3.1 print step), and that is what produced everything below. The abort is a **null-pointer dereference @@ -132,9 +136,31 @@ same crashed test. **not** a bump regression: the error string is byte-identical at b10456 and b10618, and the only upstream commit touching `common/chat.cpp` in the range is the `common_json` abstraction (#27511). -- **[OPEN] Re-check the full suite once the TTS crash is fixed.** No platform has yet completed a run - past `TtsIntegrationTest` on Linux or macOS, so the 1461-test Windows list is the *lower* bound on - what is broken, not the complete picture. +- **[ANSWERED] Re-check the full suite once the TTS crash is fixed.** Done: Ubuntu on `f9c43dc` + ran **1689 tests, 3 failures, 1 error, 2 skipped**. Exactly one item was new — the router entry + below — and the other three are the already-recorded `SessionForkRewind` pair and + `NativeServerAttach`. So the earlier Windows list was a lower bound by one item, not by many. + +- **[FIXED in this PR] `RouterModeIntegrationTest.setup:114` — the worker JVM could not load its + own main class.** `IllegalState: Router worker for model 'Qwen3-0.6B-Q4_K_M' failed with exit + code 1`; the worker's stderr (visible only because of the section-3.1 print step, in the surefire + `.dumpstream`) said `Could not find or load main class net.ladenthin.llama.server.NativeServer`. + Not a bump regression and not a router defect: `target/classes` carries a `module-info.class`, so + **Surefire auto-detects a named module and runs the main classes on the module path** — + `java.class.path` then holds only `target/test-classes` plus the dependency jars. The test built + the worker command from that property alone, so the spawned JVM had neither `NativeServer` nor the + packaged native library. It had never surfaced because the test self-skipped in CI for as long as + the model paths resolved to the wrong directory (the `TestConstants.resolveModelPath` fix in this + PR is what made it run). Fixed by deriving the main-classes root from + `NativeServer.class.getProtectionDomain().getCodeSource()`, which is correct in either mode and + for a directory or a jar alike. Reproduced and verified locally without a model: with the old + classpath the worker exits 1 on `ClassNotFoundException`; with the fix it loads `libjllama.so` and + reaches llama-server's own argument parser. + + Note for later: the project sets no `false` (srcmorph does, and its + pom explains why classpath mode is the representative test environment). Flipping it would remove + this whole class of surprise, but it changes how all 1689 tests run and does not belong in a + version-bump PR. - **[FIXED in this PR] `LlamaQuantizer.quantizeNative` had C++ linkage — the whole class was unusable in every published jar.** `QuantizerIntegrationTest` failed all 3 tests with diff --git a/llama/src/test/java/net/ladenthin/llama/server/RouterModeIntegrationTest.java b/llama/src/test/java/net/ladenthin/llama/server/RouterModeIntegrationTest.java index fc8b2a85b..5cf1f5ecb 100644 --- a/llama/src/test/java/net/ladenthin/llama/server/RouterModeIntegrationTest.java +++ b/llama/src/test/java/net/ladenthin/llama/server/RouterModeIntegrationTest.java @@ -15,6 +15,7 @@ import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import java.security.CodeSource; import java.util.List; import java.util.Locale; import net.ladenthin.llama.ClaudeGenerated; @@ -69,7 +70,7 @@ public static void setup() throws Exception { "Router worker-relaunch test runs on Linux only"); File reasoningModel = new File(TestConstants.REASONING_MODEL_PATH); Assumptions.assumeTrue(reasoningModel.exists(), "Reasoning model not found, skipping router test"); - String classpath = System.getProperty("java.class.path", ""); + String classpath = workerClasspath(); Assumptions.assumeTrue( !classpath.isEmpty() && !classpath.matches(".*\\s.*"), "Classpath contains whitespace; worker command cannot carry it"); @@ -125,6 +126,38 @@ public static void tearDown() { } } + /** + * Builds the classpath the worker JVM is launched with. + * + *

{@code java.class.path} alone is NOT enough. {@code target/classes} carries a + * {@code module-info.class}, so Surefire auto-detects a named module and runs the main classes + * on the module path; {@code java.class.path} then holds only {@code target/test-classes} + * plus the dependency jars. A worker launched with just that dies immediately with + * {@code ClassNotFoundException: net.ladenthin.llama.server.NativeServer}, which the router + * surfaces only as the opaque "worker failed with exit code 1". The main-classes root is + * therefore derived from the class itself — correct in either mode, and a directory or a jar + * alike. It also carries the packaged native library the worker has to extract.

+ * + * @return the worker classpath, or an empty string when the code source cannot be resolved + */ + private static String workerClasspath() throws Exception { + String inherited = System.getProperty("java.class.path", ""); + CodeSource source = NativeServer.class.getProtectionDomain().getCodeSource(); + if (source == null || source.getLocation() == null) { + return inherited; + } + String mainClasses = + Paths.get(source.getLocation().toURI()).toAbsolutePath().toString(); + if (inherited.isEmpty()) { + return mainClasses; + } + if ((File.pathSeparator + inherited + File.pathSeparator) + .contains(File.pathSeparator + mainClasses + File.pathSeparator)) { + return inherited; + } + return mainClasses + File.pathSeparator + inherited; + } + private static int findFreePort() throws IOException { try (ServerSocket socket = new ServerSocket(0)) { return socket.getLocalPort(); From fe4de0a9662d6ae967ea1987fb04a6ca6507638d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 15:30:04 +0000 Subject: [PATCH 51/76] chore: upgrade llama.cpp from b10631 to b10636 The smallest reviewable step this PR has had. Excluding tools/ui -- which CI rebuilds from GIT_TAG and which needs no source review -- the diff is ~85 KiB across 22 files, and not one of them is on the priority review list: zero files under common/, include/, tools/server/ or tools/mtmd/. Every row of the API-compatibility table is therefore vacuously satisfied, and the runbook's three mechanical server-contract greps have no input to compare, so the request-field set, its bounds and the emitted response keys cannot have moved. What did change is backend kernel internals behind unchanged public headers: ggml-cuda's quantised-matmul configs for Pascal (reaches only the cuda13-* classifiers) and ggml-metal's SSM/Mamba kernels (only the macOS default JAR). The rest is upstream CI workflows, a Python conversion script and an upstream test. The one build-system edit is inert here: upstream flipped LLAMA_BUILD_UI's default from ON to OFF. This project never uses upstream's UI build -- it compiles its own webui-generated/ui.cpp produced by the build-webui job -- and as a FetchContent subproject LLAMA_STANDALONE is OFF regardless. All 7 patches apply unchanged, which follows from the diff rather than from luck: none of common/arg.{cpp,h}, tools/server/* or vendor/* differs between the two tags. Verified on the shipped tree in a fresh build directory, because the applier pins its stamp to the checked-out llama.cpp commit and deliberately aborts rather than guess when an existing one names a different commit: stamp written at head 4d19b28769 with a SHA-256 line per patch, 41 files dirty, full Release build clean, ctest 504/504, and nm -D reports zero C++-mangled Java_* exports. NativeLibraryLoadSmokeTest 3/3 with 0 skipped, including nativeBuildInfoMatchesPinnedVersionConstant -- the cross-check that LlamaCppVersion.LLAMA_CPP_VERSION and the build-info compiled into the new binary agree, which is what proves the four pin sites match what was linked. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 9af0f1ba1..096d5169a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10631** +Current llama.cpp pinned version: **b10636** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10631 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10636 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10631`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10636`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1441,7 +1441,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10631`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10636`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index eacdf9daf..c35ce335a 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10631](https://img.shields.io/badge/llama.cpp-%23b10631-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10631) +[![llama.cpp b10636](https://img.shields.io/badge/llama.cpp-%23b10636-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10636) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 14bae829f..c3a740336 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -673,3 +673,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10456–b10618 | follow-up: `src/main/cpp/{jllama.cpp,json_helpers.hpp}`, `value.ServerMetrics`, `server.OpenAiCompatServer` | **Client-contract restoration, after the walk.** The b10405–b10408 and b10509–b10519 rows above record the upstream metrics refactor correctly but understate its client impact: b10408 (#26920) reduced `server_task_result_metrics::to_json()` to the bare slot array and b10519 (#27376) split the task, so `LlamaModel.getMetrics()` had been returning the slot array — not the object its three Java consumers parse — since **before** this bump range began. Re-pointing `handleSlotAction` at `SERVER_TASK_TYPE_SLOT_GET` (b10519 row) kept the array arriving but did not restore the object. Fixed by posting **both** tasks and merging them in the new pure helper `server_metrics_to_json`, so the documented payload is rebuilt in the JNI layer instead of the Java contract following upstream's transport split; durations are converted µs→ms as the pre-b10408 payload did. Reachable-but-unexposed counters (`n_prompt_cached`, the speculative-decoding tallies) are emitted alongside and surfaced on `ServerMetrics`. `GET /slots` no longer answers 200-with-empty-body when the payload has no `slots` key. Suite 491 → **499**. **This is the class of break a header diff cannot catch**: no signature changed, the project compiled and linked clean at every one of the 25 chunks, and the only guard — `LlamaModelTest#testGetMetrics` — was model-gated and silently skipped in CI (see the same-day CI working-directory fix). | | b10618–b10631 | `tools/server/server-common.cpp` (**new validation in `oaicompat_chat_params_parse`: continuing a final assistant message that carries `tool_calls` now throws `std::invalid_argument("Cannot continue an assistant message that contains tool calls.")`**), `common/chat.cpp` (**`common_chat_params_init_qwen3_coder` gains `is_qwen3_coder = !supports_reasoning` and gates two things on it: the `` bare-opener alternatives are only added for Coder models, and `tool-call-first` collapses to plain `tool-call` for the reasoning variant**), `CMakeLists.txt` (**`LLAMA_VERSION_MINOR` 2 → 3, cosmetic**), plus `tools/ui/**` (the WebUI, auto-followed) | **No project-source change.** Both server-side edits are behavioural, not structural: no signature, no struct field and no JSON key moved. The `oaicompat_chat_params_parse` addition is a strictly *narrowing* input validation on a combination the project never constructs — `OpenAiRequestMapper` never sets `continue_final_message`, and the C++ guard `test_utils.cpp` exercises the function without it (`ctest` 499/499 unchanged). The `chat.cpp` change is confined to one `static` template initialiser for the Qwen3-Coder family and touches only grammar construction; it is invisible at the API surface and the project pins no Coder model in CI. **The three mechanical contract checks the runbook mandates for a `tools/server/` chunk were not merely run but rendered moot:** `server-schema.cpp`, `server-task.cpp`, `server-context.cpp` and the four `tools/server/*.h` headers are **byte-identical** between the two tags (verified by comparing blob SHAs), so the request-field set, its bounds and the emitted response-key set cannot have moved. `common/common.h`, `include/llama.h` and `tools/mtmd/mtmd-helper.h` are likewise byte-identical, so the `getMetrics()`-class of silent contract break and the `tts_engine.cpp` surface are both out of scope for this range. Sizing note for the reviewer: the raw diff is ~466 KiB, but ~377 KiB of that is `tools/ui`, which CI rebuilds from `GIT_TAG` and which needs no source review — the reviewable remainder is ~89 KiB, under the runbook's 100 KiB single-step threshold. | | b10618–b10631 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10631 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10631: all **7** patches applied, confirmed by the stamp file (`head 5d5cb4c3a…` plus one SHA-256 line per patch) and by the source tree going dirty in 41 files. A fresh build directory was mandatory here — the applier pins its stamp to the checked-out llama.cpp commit and deliberately aborts rather than guess when an existing build dir's stamp names a different one. Full `cmake --build --config Release` clean (no errors, no new warnings), `ctest` **499/499**. `nm -D` on the freshly linked `libjllama.so` reports **zero** C++-mangled `Java_*` exports, so the `LlamaQuantizer` linkage fix carries forward. `NativeLibraryLoadSmokeTest` **3/3**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10631") matches the `build-info` compiled into the new binary, which is what proves the four pin sites and the actual build agree. | +| b10631–b10636 | `ggml/src/ggml-cuda/mmq*` (quantised matmul kernel configs for Pascal), `ggml/src/ggml-metal/*` + `kernels/ssm.metal` (SSM/Mamba Metal kernels), `CMakeLists.txt` (**`LLAMA_BUILD_UI` default `ON` → `OFF`; the `LLAMA_USE_PREBUILT_UI` help text drops its `requires LLAMA_BUILD_UI=ON` clause**), `conversion/nemotron.py`, `tests/test-backend-ops.cpp`, nine `.github/workflows/*` files, plus `tools/ui/**` (the WebUI, auto-followed) | **No project-source change, and nothing on the priority review list is touched at all.** The diff contains **zero** files under `common/`, `include/`, `tools/server/` or `tools/mtmd/`, so every row of the API-compatibility table is vacuously satisfied and the three mechanical server-contract greps have no input to compare — the request-field set, its bounds and the emitted response keys cannot have moved. The ggml changes are backend kernel internals behind unchanged public headers (CUDA affects only the `cuda13-*` classifiers, Metal only the macOS default JAR). The upstream `LLAMA_BUILD_UI` flip is inert here: this project never uses upstream's UI build, it compiles its own `webui-generated/ui.cpp` produced by the `build-webui` CI job, and as a `FetchContent` subproject `LLAMA_STANDALONE` is OFF anyway. **All 7 patches apply unchanged** — none of `common/arg.{cpp,h}`, `tools/server/*` or `vendor/*` differs between the tags. Sizing: the reviewable diff (excluding `tools/ui`) is ~85 KiB, under the runbook's 100 KiB single-step threshold. | +| b10631–b10636 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10636 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10636: all **7** patches applied, confirmed by the stamp file (`head 4d19b28769…` plus one SHA-256 line per patch) and by the source tree going dirty in 41 files. Full `cmake --build --config Release` clean, `ctest` **504/504**. `nm -D` on the freshly linked `libjllama.so` reports **zero** C++-mangled `Java_*` exports, so the `LlamaQuantizer` linkage fix carries forward. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10636") matches the `build-info` compiled into the new binary, which is what proves the four pin sites and the actual build agree. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 748865664..fabcab680 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10631 + GIT_TAG b10636 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 4e6e64600..d95e6489e 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10631"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10636"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10631-"} — call + * plus the resolved upstream commit, e.g. {@code "b10636-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10631"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10636"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10631"; + public static final String LLAMA_CPP_VERSION = "b10636"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 2204ab129f2e57dea0a0b40ac007299be717d945 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 18:50:09 +0000 Subject: [PATCH 52/76] chore: upgrade llama.cpp from b10636 to b10639 Three commits, eight files, ~70 KiB -- and for the first time in this PR's chain there is no tools/ui component at all, so the raw diff is also the reviewable diff, well under the runbook's 100 KiB single-step threshold. Nothing on the priority review list is touched: zero files under common/, include/, tools/server/ or tools/mtmd/. Every row of the API-compatibility table is therefore vacuously satisfied, and the runbook's three mechanical server-contract greps have no input to compare, so the request-field set, its bounds and the emitted response keys cannot have moved. What did change is two backend areas plus docs. Upstream #18626 adds event and async backend APIs to the RPC backend and bumps RPC_PROTO_MAJOR/MINOR from 5.1 to 6.0; that macro pair is the entire diff of the one public header in the range, ggml/include/ggml-rpc.h. It is inert here: GGML_RPC is enabled nowhere in this project -- grep across the workflows, build scripts and CMake is empty -- so ggml-rpc.cpp is never compiled into libjllama, and the protocol bump is a wire-compatibility concern only for operators running upstream's rpc-server, which this project does not ship. The Vulkan work (#27216 cross_entropy_loss and its backward pass, as two new shaders; #27726 clamping warptiles because they assume warp sizes <= 64) is backend kernel internals behind unchanged public headers and reaches only the vulkan-linux-x86-64, vulkan-linux-aarch64 and vulkan-windows-x86-64 classifiers, whose CI jobs are build-only because GitHub runners have no GPU. All 7 patches are expected to apply unchanged, and that follows from the diff rather than from luck: the patch target list and the eight changed files have zero overlap -- no common/arg.{cpp,h}, no tools/server/*, no vendor/*. Deliberately not claimed here: this commit records the source review, not a local build. The full fresh-directory verification (fail-loud PATCH_COMMAND, Release build, ctest) is running separately and its result will be recorded in docs/history/llama-cpp-breaking-changes.md as the paired verification row, the way every other range in that table is. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CHANGELOG.md | 9 +++++++++ CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 1 + llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 6 files changed, 20 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fdba2b152..c1f10cc0f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -62,6 +62,15 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by guarded for old glibc in the same change. - Android/Gradle toolchain: Gradle pins moved 8.14.3 → 9.6.1 and the dockcross cross-compile images were bumped, alongside the AGP/Compose pin updates the Android builds needed. +- Upgraded llama.cpp from **b10631 to b10639**, in two reviewed steps. Neither range changes any + project source. b10631→b10636 is ggml-cuda quantised-matmul configs for Pascal, ggml-metal + SSM/Mamba kernels, an upstream `LLAMA_BUILD_UI` default flip that is inert here (this project + compiles its own `webui-generated/ui.cpp`), and the WebUI. b10636→b10639 is the RPC backend's + event/async APIs (#18626, protocol 5.1 → 6.0 — `GGML_RPC` is never enabled in this project, so + `ggml-rpc.cpp` is not compiled) plus Vulkan `cross_entropy_loss` kernels (#27216) and a warptile + clamp for warp sizes > 64 (#27726). Neither range touches `common/`, `include/llama.h`, + `tools/server/` or `tools/mtmd/`, so no request field, no bound and no response key can have + moved. All seven local patches apply unchanged. - Upgraded llama.cpp from **b10618 to b10631**. No project-source change: the only edits in the range are a narrowing input validation in `oaicompat_chat_params_parse` (continuing a final assistant message that carries `tool_calls` now throws), a Qwen3-Coder-only grammar refinement diff --git a/CLAUDE.md b/CLAUDE.md index 096d5169a..964e3c283 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10636** +Current llama.cpp pinned version: **b10639** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10636 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10639 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10636`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10639`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1441,7 +1441,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10636`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10639`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index c35ce335a..69eb088fd 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10636](https://img.shields.io/badge/llama.cpp-%23b10636-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10636) +[![llama.cpp b10639](https://img.shields.io/badge/llama.cpp-%23b10639-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10639) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index c3a740336..406a043e8 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -675,3 +675,4 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10618–b10631 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10631 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10631: all **7** patches applied, confirmed by the stamp file (`head 5d5cb4c3a…` plus one SHA-256 line per patch) and by the source tree going dirty in 41 files. A fresh build directory was mandatory here — the applier pins its stamp to the checked-out llama.cpp commit and deliberately aborts rather than guess when an existing build dir's stamp names a different one. Full `cmake --build --config Release` clean (no errors, no new warnings), `ctest` **499/499**. `nm -D` on the freshly linked `libjllama.so` reports **zero** C++-mangled `Java_*` exports, so the `LlamaQuantizer` linkage fix carries forward. `NativeLibraryLoadSmokeTest` **3/3**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10631") matches the `build-info` compiled into the new binary, which is what proves the four pin sites and the actual build agree. | | b10631–b10636 | `ggml/src/ggml-cuda/mmq*` (quantised matmul kernel configs for Pascal), `ggml/src/ggml-metal/*` + `kernels/ssm.metal` (SSM/Mamba Metal kernels), `CMakeLists.txt` (**`LLAMA_BUILD_UI` default `ON` → `OFF`; the `LLAMA_USE_PREBUILT_UI` help text drops its `requires LLAMA_BUILD_UI=ON` clause**), `conversion/nemotron.py`, `tests/test-backend-ops.cpp`, nine `.github/workflows/*` files, plus `tools/ui/**` (the WebUI, auto-followed) | **No project-source change, and nothing on the priority review list is touched at all.** The diff contains **zero** files under `common/`, `include/`, `tools/server/` or `tools/mtmd/`, so every row of the API-compatibility table is vacuously satisfied and the three mechanical server-contract greps have no input to compare — the request-field set, its bounds and the emitted response keys cannot have moved. The ggml changes are backend kernel internals behind unchanged public headers (CUDA affects only the `cuda13-*` classifiers, Metal only the macOS default JAR). The upstream `LLAMA_BUILD_UI` flip is inert here: this project never uses upstream's UI build, it compiles its own `webui-generated/ui.cpp` produced by the `build-webui` CI job, and as a `FetchContent` subproject `LLAMA_STANDALONE` is OFF anyway. **All 7 patches apply unchanged** — none of `common/arg.{cpp,h}`, `tools/server/*` or `vendor/*` differs between the tags. Sizing: the reviewable diff (excluding `tools/ui`) is ~85 KiB, under the runbook's 100 KiB single-step threshold. | | b10631–b10636 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10636 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10636: all **7** patches applied, confirmed by the stamp file (`head 4d19b28769…` plus one SHA-256 line per patch) and by the source tree going dirty in 41 files. Full `cmake --build --config Release` clean, `ctest` **504/504**. `nm -D` on the freshly linked `libjllama.so` reports **zero** C++-mangled `Java_*` exports, so the `LlamaQuantizer` linkage fix carries forward. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10636") matches the `build-info` compiled into the new binary, which is what proves the four pin sites and the actual build agree. | +| b10636–b10639 | `ggml/src/ggml-rpc/ggml-rpc.cpp` + `ggml/include/ggml-rpc.h` (**#18626: event and async backend APIs for the RPC backend; `RPC_PROTO_{MAJOR,MINOR}_VERSION` 5.1 → 6.0**), `ggml/src/ggml-vulkan/ggml-vulkan.cpp` + two new `vulkan-shaders/cross_entropy_loss{,_back}.comp` + `vulkan-shaders-gen.cpp` (**#27216: Vulkan `cross_entropy_loss` / `_back`; #27726: warptiles clamped because they assume warp sizes ≤ 64**), `docs/ops.md`, `docs/ops/Vulkan.csv` | **No project-source change, and nothing on the priority review list is touched at all.** Eight files, zero of them under `common/`, `include/llama.h`, `tools/server/` or `tools/mtmd/`, so every row of the API-compatibility table is vacuously satisfied and the three mechanical server-contract greps have no input to compare — the request-field set, its bounds and the emitted response keys cannot have moved. Unlike the previous two ranges there is **no `tools/ui` component at all**, so the whole ~70 KiB raw diff is the reviewable diff, comfortably under the runbook's 100 KiB single-step threshold. The one public header in the range is `ggml/include/ggml-rpc.h`, and its entire diff is the two protocol-version macros: **`GGML_RPC` is never enabled anywhere in this project** (`grep -rn GGML_RPC` across the workflows, build scripts and CMake is empty), so `ggml-rpc.cpp` is not compiled into `libjllama` and the protocol bump is a wire-compatibility concern only for operators running upstream's `rpc-server`, which this project does not ship. The Vulkan work is backend kernel internals behind unchanged public headers and reaches only the `vulkan-linux-x86-64` / `vulkan-linux-aarch64` / `vulkan-windows-x86-64` classifiers, whose CI jobs are build-only (GitHub runners have no GPU), so the CI signal for them is compilation. **All 7 patches apply unchanged** — the patch target list and the eight changed files have **zero** overlap (no `common/arg.{cpp,h}`, no `tools/server/*`, no `vendor/*`). | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index fabcab680..d2f27fa73 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10636 + GIT_TAG b10639 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index d95e6489e..589d6e62c 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10636"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10639"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10636-"} — call + * plus the resolved upstream commit, e.g. {@code "b10639-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10636"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10639"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10636"; + public static final String LLAMA_CPP_VERSION = "b10639"; // Constants holder — not instantiable. private LlamaCppVersion() {} From b151f72bb241cfecb000a8881e2d1712926697c8 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 18:53:41 +0000 Subject: [PATCH 53/76] chore: ignore the per-bump build-b verification trees The llama.cpp upgrade runbook requires a FRESH build directory for every version bump: the patch applier pins its stamp file to the checked-out llama.cpp commit and deliberately aborts rather than guess when an existing directory's stamp names a different one. The convention names those directories build-b (build-b10631, build-b10636, build-b10639). .gitignore covered build, build-android and cmake-build-* but not that family, so every bump verification left the working tree dirty with an untracked directory that must never be committed -- this repo commits no build outputs. Adding build-b* closes it for good. The pattern is deliberately build-b* rather than build*: build-android does not start with "build-b", so it keeps matching its own dedicated entry and nothing is shadowed. Verified that no tracked file matches the new pattern. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- .gitignore | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.gitignore b/.gitignore index c2a7c61a1..7003d4ae2 100644 --- a/.gitignore +++ b/.gitignore @@ -7,6 +7,11 @@ target build build-android +# Per-bump verification trees. The llama.cpp upgrade runbook requires a FRESH build +# directory for every version bump -- the patch applier pins its stamp to the checked-out +# llama.cpp commit and deliberately aborts rather than guess when an existing dir's stamp +# names a different one -- and the convention names them build-b. +build-b* cmake-build-* .DS_Store .directory From b0d69c1cb0f719d72608ea0e3307aafa6e6c2ad2 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 18:58:44 +0000 Subject: [PATCH 54/76] docs: record b10639 verification and the inlining false alarm Pairs the b10636-b10639 change-inventory row with its verification row, the way every other range in that table is recorded. Verified on the shipped tree in a fresh build directory, which the applier requires after a GIT_TAG change: stamp written at head 5e6a37cb115dc1074e274ac004373f5661909695 with a SHA-256 line per patch, all 7 patches applied, 41 files dirty, full Release build clean, ctest 504/504, and nm -D reports 40 Java_* exports with zero C++-mangled ones. NativeLibraryLoadSmokeTest 3/3 with 0 skipped, including the cross-check that LlamaCppVersion.LLAMA_CPP_VERSION ("b10639") matches the build-info compiled into the binary ("b10639-5e6a37cb1"). That cross-check first reported a drift that did not exist -- "Linked build-info b10639-... must start with the pinned tag b10636-" -- while both the source and target/classes already read b10639. LLAMA_CPP_VERSION is a public static final String, so it is a compile-time constant and javac inlines its value into every referencing class, including the test. Recompiling LlamaCppVersion.java alone leaves the stale copy baked into an already-compiled NativeLibraryLoadSmokeTest, and Maven's incremental compilation cannot see that dependency because constant inlining is invisible to its change analysis. mvn clean test clears it; CI was never affected, since it always builds from a clean checkout. Both the runbook step that introduces the guard and the history row now say so, so the next bump does not spend time re-diagnosing a stale artifact as a real pin drift. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- docs/history/llama-cpp-breaking-changes.md | 1 + docs/upgrade/llama-cpp-version-bump.md | 12 ++++++++++++ 2 files changed, 13 insertions(+) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 406a043e8..c703ad969 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -676,3 +676,4 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10631–b10636 | `ggml/src/ggml-cuda/mmq*` (quantised matmul kernel configs for Pascal), `ggml/src/ggml-metal/*` + `kernels/ssm.metal` (SSM/Mamba Metal kernels), `CMakeLists.txt` (**`LLAMA_BUILD_UI` default `ON` → `OFF`; the `LLAMA_USE_PREBUILT_UI` help text drops its `requires LLAMA_BUILD_UI=ON` clause**), `conversion/nemotron.py`, `tests/test-backend-ops.cpp`, nine `.github/workflows/*` files, plus `tools/ui/**` (the WebUI, auto-followed) | **No project-source change, and nothing on the priority review list is touched at all.** The diff contains **zero** files under `common/`, `include/`, `tools/server/` or `tools/mtmd/`, so every row of the API-compatibility table is vacuously satisfied and the three mechanical server-contract greps have no input to compare — the request-field set, its bounds and the emitted response keys cannot have moved. The ggml changes are backend kernel internals behind unchanged public headers (CUDA affects only the `cuda13-*` classifiers, Metal only the macOS default JAR). The upstream `LLAMA_BUILD_UI` flip is inert here: this project never uses upstream's UI build, it compiles its own `webui-generated/ui.cpp` produced by the `build-webui` CI job, and as a `FetchContent` subproject `LLAMA_STANDALONE` is OFF anyway. **All 7 patches apply unchanged** — none of `common/arg.{cpp,h}`, `tools/server/*` or `vendor/*` differs between the tags. Sizing: the reviewable diff (excluding `tools/ui`) is ~85 KiB, under the runbook's 100 KiB single-step threshold. | | b10631–b10636 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10636 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10636: all **7** patches applied, confirmed by the stamp file (`head 4d19b28769…` plus one SHA-256 line per patch) and by the source tree going dirty in 41 files. Full `cmake --build --config Release` clean, `ctest` **504/504**. `nm -D` on the freshly linked `libjllama.so` reports **zero** C++-mangled `Java_*` exports, so the `LlamaQuantizer` linkage fix carries forward. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10636") matches the `build-info` compiled into the new binary, which is what proves the four pin sites and the actual build agree. | | b10636–b10639 | `ggml/src/ggml-rpc/ggml-rpc.cpp` + `ggml/include/ggml-rpc.h` (**#18626: event and async backend APIs for the RPC backend; `RPC_PROTO_{MAJOR,MINOR}_VERSION` 5.1 → 6.0**), `ggml/src/ggml-vulkan/ggml-vulkan.cpp` + two new `vulkan-shaders/cross_entropy_loss{,_back}.comp` + `vulkan-shaders-gen.cpp` (**#27216: Vulkan `cross_entropy_loss` / `_back`; #27726: warptiles clamped because they assume warp sizes ≤ 64**), `docs/ops.md`, `docs/ops/Vulkan.csv` | **No project-source change, and nothing on the priority review list is touched at all.** Eight files, zero of them under `common/`, `include/llama.h`, `tools/server/` or `tools/mtmd/`, so every row of the API-compatibility table is vacuously satisfied and the three mechanical server-contract greps have no input to compare — the request-field set, its bounds and the emitted response keys cannot have moved. Unlike the previous two ranges there is **no `tools/ui` component at all**, so the whole ~70 KiB raw diff is the reviewable diff, comfortably under the runbook's 100 KiB single-step threshold. The one public header in the range is `ggml/include/ggml-rpc.h`, and its entire diff is the two protocol-version macros: **`GGML_RPC` is never enabled anywhere in this project** (`grep -rn GGML_RPC` across the workflows, build scripts and CMake is empty), so `ggml-rpc.cpp` is not compiled into `libjllama` and the protocol bump is a wire-compatibility concern only for operators running upstream's `rpc-server`, which this project does not ship. The Vulkan work is backend kernel internals behind unchanged public headers and reaches only the `vulkan-linux-x86-64` / `vulkan-linux-aarch64` / `vulkan-windows-x86-64` classifiers, whose CI jobs are build-only (GitHub runners have no GPU), so the CI signal for them is compilation. **All 7 patches apply unchanged** — the patch target list and the eight changed files have **zero** overlap (no `common/arg.{cpp,h}`, no `tools/server/*`, no `vendor/*`). | +| b10636–b10639 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10639 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10639: all **7** patches applied, confirmed by the stamp file (`head 5e6a37cb115dc1074e274ac004373f5661909695` plus one SHA-256 line per patch) and by the source tree going dirty in 41 files. A fresh build directory was mandatory here — the applier pins its stamp to the checked-out llama.cpp commit and deliberately aborts rather than guess when an existing build dir's stamp names a different one. Full `cmake --build --config Release` clean, `ctest` **504/504**. `nm -D` on the freshly linked `libjllama.so` reports **40** `Java_*` exports and **zero** C++-mangled ones, so the `LlamaQuantizer` linkage fix carries forward. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10639") matches the `build-info` compiled into the new binary (`b10639-5e6a37cb1`), which is what proves the four pin sites and the actual build agree. **Note for the next bump:** that last check first reported a *false* drift (`must start with the pinned tag "b10636-"`) even though both the source and `target/classes` already read `b10639`. `LLAMA_CPP_VERSION` is a compile-time constant, so javac had inlined the old value into the already-compiled `NativeLibraryLoadSmokeTest.class` and Maven's incremental compilation cannot see that dependency; `mvn clean test` clears it. Recorded in the runbook so the next bump does not re-diagnose it. | diff --git a/docs/upgrade/llama-cpp-version-bump.md b/docs/upgrade/llama-cpp-version-bump.md index 490e2135e..2e2dc901d 100644 --- a/docs/upgrade/llama-cpp-version-bump.md +++ b/docs/upgrade/llama-cpp-version-bump.md @@ -120,6 +120,18 @@ Concretely: if you forget it, `NativeLibraryLoadSmokeTest.nativeBuildInfoMatchesPinnedVersionConstant` fails the build (it cross-checks the constant against `LlamaModel.getLlamaCppBuildInfo()`, which reads llama.cpp's own linked-in `build-info`). + + > **Local-only gotcha: that guard can report a *false* drift after a bump.** + > `LLAMA_CPP_VERSION` is a `public static final String`, i.e. a **compile-time constant**, so + > javac inlines its value into every *referencing* class — including + > `NativeLibraryLoadSmokeTest`. Recompiling `LlamaCppVersion.java` alone therefore does **not** + > update the copy baked into the already-compiled test class, and Maven's incremental + > compilation cannot see the dependency (constant inlining is invisible to its change + > analysis). The symptom is a failure that looks alarming but is pure staleness, e.g. + > `Linked build-info "b10639-…" must start with the pinned tag "b10636-"` when both the source + > and `target/classes` already say `b10639`. Run **`mvn clean test`** (not a bare `mvn test`) + > when re-running this check locally after a bump. CI is immune — it always builds from a clean + > checkout. 2. **Re-verify `patches/`** — a clean configure re-runs the fail-loud `PATCH_COMMAND`, so **every `*.patch` in `llama/patches/`** must still apply. Do not maintain a list of them here or anywhere else: `apply-llama-patches.cmake` `file(GLOB)`s the directory and applies them in filename order, From cc67ea70df2a127bf9cbd57822370641d4e88377 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 20:39:27 +0000 Subject: [PATCH 55/76] test: stop fork/rewind tests spending their budget on thinking SessionForkRewindIntegrationTest asserted a non-empty reply after a rewind and after a fork, and got an empty string on every platform. The two failures were one defect, and it was in the test rather than in the session code. The fixture model is Qwen3-0.6B. Its chat template injects into the prompt, so it reasons on every turn, and llama.cpp's common_params.reasoning_ format defaults to DEEPSEEK, which strips the ... block out of content and into reasoning_content. Session.send returns choices[0].message .content only, so whatever lands in reasoning_content is invisible to it. The fixture budgeted 24 tokens per turn for speed; this model spends roughly 200 tokens thinking before it answers, which ReasoningBudgetTest already documents and which is why that class budgets 1500. So the budget was exhausted inside the thinking block and content came back empty -- deterministically, which is why all six platforms failed identically rather than flakily. Suppressing thinking with reasoning_budget_tokens=0 is the right fix here rather than raising the budget: this class covers session state -- checkpoint, rewind, fork -- not reasoning, so the thinking block is pure cost. Budgeting 1500 instead would add ~200 tokens per turn across five turns and buy no coverage. The suppression path is itself covered by ReasoningBudgetTest .testReasoningBudgetZero_suppressesThinking. Neither failure was a regression from a llama.cpp bump. Both are latent defects that this branch is the first run ever to execute, because the model-path resolution fix earlier in this PR stopped the model-gated classes from silently self-skipping. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- .../llama/SessionForkRewindIntegrationTest.java | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/llama/src/test/java/net/ladenthin/llama/SessionForkRewindIntegrationTest.java b/llama/src/test/java/net/ladenthin/llama/SessionForkRewindIntegrationTest.java index b3e24a02c..ddc13365b 100644 --- a/llama/src/test/java/net/ladenthin/llama/SessionForkRewindIntegrationTest.java +++ b/llama/src/test/java/net/ladenthin/llama/SessionForkRewindIntegrationTest.java @@ -53,12 +53,27 @@ public static void closeModel() { } } + /** + * Thinking is suppressed with {@code reasoning_budget_tokens=0}, and that is load-bearing + * rather than cosmetic. The fixture model is Qwen3-0.6B, whose chat template injects + * {@code } into the prompt, so the model reasons on every turn; llama.cpp's + * {@code common_params.reasoning_format} defaults to {@code DEEPSEEK}, which strips the + * {@code …} block out of {@code content} and into {@code reasoning_content}. + * {@link Session#send(String)} returns {@code choices[0].message.content} only. Qwen3-0.6B + * spends roughly 200 tokens thinking before it answers (see {@code ReasoningBudgetTest}, which + * budgets 1500 for exactly this reason), so with the 24-token budget this test wants for speed + * every token is consumed inside the thinking block and {@code content} comes back empty + * — deterministically, on every platform. Suppressing thinking makes the model answer directly, + * which is what these assertions are actually about: this class tests session state + * (checkpoint/rewind/fork), not reasoning. Raising the budget instead would work but costs + * ~200 extra tokens per turn across five turns for no added coverage. + */ private static Session newSession(int slotId) { return new Session( model, slotId, "You are terse.", - p -> p.withNPredict(24).withTemperature(0.0f).withSeed(42)); + p -> p.withNPredict(24).withTemperature(0.0f).withSeed(42).withReasoningBudgetTokens(0)); } @Test From ca609476d304e5068e992235c7e888f3c33bf652 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 22:08:42 +0000 Subject: [PATCH 56/76] fix: stop one malformed UTF-8 byte from 500-ing a generation NativeServerAttachIntegrationTest.completion_overHttp_served fails on all six Java CI platforms with {"error":{"code":500,"message":"The model produced output that does not match the expected Content-only format","type":"server_error"}} even though the slot log right above it shows the generation finished normally: "stop processing: n_tokens = 4, truncated = 0". The server parses every completion through common_chat_parse(). With no chat parser configured -- plain /completion -- that is llama.cpp's content-only fallback, content(rest()) + end(), whose scan is common_peg_until_parser in common/peg-parser.cpp. common_chat_peg_parse() always parses in lenient mode, and that scan tolerates an INCOMPLETE trailing UTF-8 sequence by keeping the text before it -- but the INVALID branch directly below returns FAIL unconditionally, ignoring leniency. So a single stray byte anywhere in the generated text throws, and a request that produced output returns an error instead. patches/0011 makes the INVALID branch respect ctx.is_lenient() exactly like the INCOMPLETE branch: keep the text up to the malformed byte. Strict mode is unchanged, and no existing upstream test asserted FAIL on invalid UTF-8 through the until parser (test_unicode's FAIL cases go through p.any(), a different executor overload). The patch also carries an upstream tests/peg-parser/test-unicode.cpp case pinning both the new lenient behaviour and the still-failing strict one, so it is upstream-submittable as-is. Diagnosis by elimination rather than by guess: a standalone probe linked against the b10639 tree ran fifteen plausible completion shapes -- chat markers, blocks, JSON, control characters, NUL, emoji, CJK, a 4 KB ASCII string -- through the same default parser params. All fifteen parse clean; only malformed UTF-8 throws, and only when the bad byte is followed by more input (a truncated sequence at the very end was already tolerated). With the patch applied the same probe throws on nothing and every valid input still round-trips byte for byte. Guarded by five new ContentOnlyParseUtf8 tests in test_utils.cpp (504 -> 509). Unlike the upstream test in the patch, which this project does not compile (LLAMA_BUILD_TESTS is OFF for a FetchContent subproject), these run in the C++ Tests job on every platform -- so a future bump that drops the patch reds a fast model-free job instead of one Java integration test. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CHANGELOG.md | 8 ++ CLAUDE.md | 5 +- ...0011-peg-parser-lenient-invalid-utf8.patch | 74 +++++++++++++++++++ llama/src/test/cpp/test_utils.cpp | 65 ++++++++++++++++ 4 files changed, 150 insertions(+), 2 deletions(-) create mode 100644 llama/patches/0011-peg-parser-lenient-invalid-utf8.patch diff --git a/CHANGELOG.md b/CHANGELOG.md index c1f10cc0f..f6e598312 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -93,6 +93,14 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by reading (`ggml_time_us()`), not milliseconds since the epoch. The value is unchanged. ### Fixed +- **A single malformed UTF-8 byte in a model's output turned a finished generation into an HTTP 500.** + The server parses *every* completion through `common_chat_parse()`; with no chat parser configured + (plain `/completion`) that is llama.cpp's content-only fallback, whose scan tolerates an incomplete + trailing UTF-8 sequence in lenient mode — which is the only mode the chat parser ever uses — but + rejected an *invalid* byte outright. The request then failed with `"The model produced output that + does not match the expected Content-only format"` even though generation had completed normally. + Carried as local patch `0011`, which makes the invalid-byte branch respect leniency the same way + (keeping the text up to the bad byte); strict-mode parsing is unchanged. Upstream-submittable. - **`TextToSpeech` crashed the JVM on every platform when loading a model.** A hand-built `common_params` never passes through `common_params_parse`, and `common/arg.cpp` is upstream's only caller of `postprocess_cpu_params` — `common_init_from_params` does not call it. So diff --git a/CLAUDE.md b/CLAUDE.md index 964e3c283..fb82acf52 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -673,6 +673,7 @@ Current patches: | `0007-server-attach-http-frontend.patch` | **Adds `llama_server_attach(argc, argv, server_context&)`** so the `NativeServer` *attach mode* can serve an **already-loaded `LlamaModel`** over the upstream HTTP frontend — no second model load, no `start_loop()`; the LlamaModel's worker keeps driving the shared `server_context` and the HTTP routes post tasks to its queue (the queue is the synchronization point). Mechanically: (1) extracts the **pure core route table** (`health` … `slots`) out of `llama_server()` into `static void llama_server_register_common_routes(ctx_http, routes)` (shared, so the two entry points cannot drift on the core endpoint set). **Scope note (narrowed at the b10154 bump):** the helper deliberately carries **only** the stable, state-independent route table — **not** the resumable-streaming routes (their handlers differ between router / non-router), the GCP-compat shim, or the experimental **CORS-proxy / MCP-server / built-in-tools** wiring. b10154 (upstream MCP-server support) moved the streaming routes into the middle of that block and coupled tools/CORS to a per-call `server_mcp mcp_mgr` lifecycle, so the earlier contiguous "route-table + CORS-proxy + tools" extraction is no longer possible; `llama_server()` keeps all of that inline, **byte-identical to upstream b10154** (only the route-table block is factored out). (2) adds `llama_server_attach`, which parses only the HTTP-side argv via `common_params_parse`, starts the stream-session GC + `server_http_context`, registers the common route table, the **non-router** resumable-streaming handlers (upstream b10154 paths `/v1/stream` GET/DEL + `/v1/streams/lookup` POST), the GCP-compat shim, and **403 "disabled" stubs for `/cors-proxy` + `/tools`** (attach mode does not wire the experimental CORS-proxy / MCP / built-in-tools host — those belong to a full `llama-server`, not an embedded model), marks ready immediately (model already loaded), and blocks on the HTTP thread until `llama_server_request_shutdown()` — never calling `common_init()`, backend init, `ctx_server.terminate()` or `llama_backend_free()` (the embedding caller owns those). Applies after `0001`+`0006` (same file); closes the "NativeServer — reuse an already-loaded LlamaModel" TODO. Upstream-submittable ("server: let embedding callers attach the HTTP frontend to an existing server_context"). **Refreshed at the b10519 bump:** upstream #26347 dropped the API key from the `/models` + `/v1/models` public-endpoint set and deleted the two trailing `// public endpoint (no API key check)` comments on those route registrations. Those two lines sit inside this patch's route-table removal block, so `git apply` failed ("patch does not apply", `server.cpp:258`) at **every** tag from b10519 on; the fix was to drop the now-wrong comment from all four affected lines (2 on the `-` side, 2 in the extracted helper on the `+` side), keeping the helper byte-identical to the block it replaces. **This is the invariant to re-check on every bump:** the `+` side of `llama_server_register_common_routes()` must stay a verbatim copy of the route table it factors out of `llama_server()`. | | `0008-server-models-worker-cmd-override.patch` | **Makes router mode usable in-JVM.** The router (`server-models.cpp`) spawns each model worker by re-executing its own binary (`get_server_exec_path()` = `/proc/self/exe` & friends) — inside a JVM that binary is `java`, not a llama-server, so embedded router workers could never start. The patch adds env `LLAMA_SERVER_WORKER_CMD` (whitespace-split; read in `server_model_meta::update_args`) which replaces only the leading binary-path token of the rendered worker args, letting an embedding host relaunch workers through its own bootstrap — e.g. `java -cp app.jar net.ladenthin.llama.server.NativeServer` (each worker is then a fresh JVM running the classic single-model `NativeServer`). Exposed in Java as `NativeServer.setWorkerCommand(String...)` (JNI `setenv`); exercised by `RouterModeIntegrationTest` (Linux CI). Upstream-submittable (also useful for containerized/wrapped deployments). | | `0010-server-cast-vocab-type-for-common-json.patch` | **Upstream regression from the b10585 `common_json` switch (#27511), one line.** `get_res_model_info()` (`tools/server/server-context.cpp`) builds the `GET /models` + `GET /v1/models` payload and emits `{"vocab_type", meta.model_vocab_type}` — an **unscoped enum**. `common_json_value`'s integral constructor template is `std::is_integral`-gated, which *excludes* enums, so the value binds to `common_json_value(bool)` and serialises as `true`/`false` instead of the numeric vocab type. It was correct while the alias was `nlohmann::ordered_json` (nlohmann serialises an enum as an integer), so upstream regressed it silently when they flipped the alias. The project ships this: `server-context.cpp` is compiled into `libjllama` and both routes are served by `NativeServer` — the default fat-jar `Main-Class` — in full **and** attach mode (`patches/0007`'s common route table registers them). The patch casts the value to `int` at the emit site, mirroring what `jllama.cpp` does for its own two `"vocab_type"` sites. Upstream-submittable; **not yet filed upstream**. Applies after `0002`/`0003` (same file) — numbered `0010` because `0009` is burned: it names the subprocess.h patch dropped at the b10280 bump (see the note below this table), and reusing the number would make that note read as if it were about this patch. **On every bump, check whether upstream cast the value themselves; if they did, DROP this patch rather than refreshing it** — the fail-loud applier only detects "does not apply", never "upstream already fixed this", and no test can catch a redundant carry here because `get_res_model_info` is `static` inside `server-context.cpp` and unreachable from `jllama_test`. See the `CommonJsonEnumTrap` tests in `test_json_helpers.cpp` for the mechanism the cast defends against. | +| `0011-peg-parser-lenient-invalid-utf8.patch` | **A model that emits one malformed UTF-8 byte turns a finished generation into an HTTP 500.** The server parses *every* completion through `common_chat_parse()`; with no chat parser configured (plain `/completion`) that is the content-only fallback `content(rest()) + end()`, whose scan is `common_peg_until_parser` (`common/peg-parser.cpp`). `common_chat_peg_parse()` always parses in **lenient** mode, and that scan tolerates an `INCOMPLETE` trailing UTF-8 sequence by keeping the text before it — but the `INVALID` branch right below it returns `FAIL` unconditionally, ignoring leniency. One stray byte anywhere in the generated text therefore throws `"The model produced output that does not match the expected Content-only format"` and the request 500s even though generation completed normally (`stop processing: n_tokens = 4, truncated = 0`). The patch makes the `INVALID` branch respect `ctx.is_lenient()` exactly like the `INCOMPLETE` branch — keep the text up to the malformed byte — and adds an upstream `tests/peg-parser/test-unicode.cpp` case pinning both the lenient and the still-failing strict behavior. **Strict mode is unchanged**, and no existing upstream test asserted `FAIL` on invalid UTF-8 through the *until* parser (the `FAIL` cases in `test_unicode` go through `p.any()`, a different executor overload). Found by `NativeServerAttachIntegrationTest.completion_overHttp_served`, which 500s on all six Java CI platforms. Upstream-submittable; **not yet filed upstream**. Touches only `common/peg-parser.cpp` + that test, which no other patch touches, so it is independent of `0001`/`0006`/`0007`. Runnable guard: the `ContentOnlyParseUtf8` tests in `src/test/cpp/test_utils.cpp` — unlike the upstream test they are compiled and run in CI on every platform, so a bump that drops this patch reds `C++ Tests` instead of one Java job. | | `0006-server-embed-native-server-jni.patch` | **Makes `server.cpp`'s `llama_server` embeddable in the JVM** so the `NativeServer` JNI bridge can run the full upstream HTTP server (WebUI included) inside `libjllama` — see "Two server modes" below. b9870 already exposes `int llama_server(int, char**)` (non-static; no `main` in the file), so the patch only adds embedded-mode support: (1) a `g_llama_server_embedded` flag + `llama_server_set_embedded()` / `llama_server_request_shutdown()` (declared in the committed `src/main/cpp/native_server_bridge.h`); (2) skips installing the process-wide SIGINT/SIGTERM handlers when embedded (they would hijack the JVM's); (3) in embedded mode parses the **forwarded** argv via `common_params_parse` instead of `common_params_parse_main` (whose `GetCommandLineW` recovery would pick up `java.exe`'s command line — the same Windows class of bug `0001` fixes). `llama_server_request_shutdown()` mirrors the SIGTERM path (invokes the installed `shutdown_handler` → `ctx_server.terminate()` unblocks `start_loop()`), giving JNI an out-of-band stop since `ctx_server` is loop-local. Applies **after `0001`** (which flips this call site to `common_params_parse_main`), so its context is the post-`0001` tree; regenerate against `0001`+source on a bump. Only touches `tools/server/server.cpp`. | **`0009` was dropped at the b10280 bump.** Upstream merged @@ -1430,14 +1431,14 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" | File | Tests | Scope | |------|-------|-------| -| `src/test/cpp/test_utils.cpp` | 162 | Upstream helpers: `server_tokens`, `server_grammar_trigger`, `gen_tool_call_id`, `json_value`, `json_get_nested_values`, UTF-8 helpers, `format_response_rerank`, `format_embeddings_response_oaicompat`, `oaicompat_completion_params_parse`, `oaicompat_chat_params_parse`, `are_lora_equal`, `strip_flag_from_argv`, `token_piece_value`, `json_is_array_and_contains_numbers`, `format_oai_sse`, `format_oai_resp_sse`, `format_anthropic_sse`, `parse_lora_request` | +| `src/test/cpp/test_utils.cpp` | 167 | Upstream helpers: `server_tokens`, `server_grammar_trigger`, `gen_tool_call_id`, `json_value`, `json_get_nested_values`, UTF-8 helpers, `format_response_rerank`, `format_embeddings_response_oaicompat`, `oaicompat_completion_params_parse`, `oaicompat_chat_params_parse`, `are_lora_equal`, `strip_flag_from_argv`, `token_piece_value`, `json_is_array_and_contains_numbers`, `format_oai_sse`, `format_oai_resp_sse`, `format_anthropic_sse`, `parse_lora_request`, `common_chat_parse` over malformed UTF-8 (the `ContentOnlyParseUtf8` guard for `patches/0011`) | | `src/test/cpp/test_server.cpp` | 206 | Upstream result types: `server_slot_stats` (the `timings` JSON payload; replaced `result_timings` in b10408), `task_params::to_json()` (incl. `dry_sequence_breakers`, `preserved_tokens`, `timings_per_token`), `completion_token_output`, `server_task_result_cmpl_partial` (non-oaicompat + `to_json_oaicompat` + logprobs + `to_json_oaicompat_chat` + `to_json_anthropic` + dispatcher), `server_task_result_cmpl_final` (non-oaicompat + `to_json_oaicompat` + `to_json_oaicompat_chat` + `to_json_oaicompat_chat_stream` + `to_json_anthropic` + `to_json_anthropic_stream` + tool_calls + dispatcher), `server_task_result_embd`, `server_task_result_rerank`, `server_task_result_metrics` (`to_metrics()` = the `/metrics` Prometheus exposition text; its `to_json()` has been unused since b10519 and returns `json{}` = JSON null), `server_task_result_slots` (`to_json()` = the `/slots` array, fed by the b10519 `SERVER_TASK_TYPE_SLOT_GET` task), `server_task_result_slot_save_load`, `server_task_result_slot_erase`, `server_task_result_apply_lora`, `server_task_result_get_lora`, `server_task_result_error`, `format_error_response`, `server_task::need_sampling()`, `server_task::n_tokens()`, `server_schema::eval_llama_cmpl_schema()` (parsing pipeline + grammar routing + error paths + per-request `dry_*` and `sse_ping_interval` field round-trips incl. hard-limit + server-default inheritance), `response_fields` projection | | `src/test/cpp/test_json_helpers.cpp` | 60 | All functions in `json_helpers.hpp`: `get_result_error_message`, `results_to_json`, `rerank_results_to_json` (incl. missing/out-of-range `index` rejection), `parse_encoding_format`, `extract_embedding_prompt`, `is_infill_request`, `parse_slot_prompt_similarity`, `parse_positive_int_config`, `wrap_stream_chunk`, `server_metrics_to_json` | | `src/test/cpp/test_log_helpers.cpp` | 13 | All functions in `log_helpers.hpp`: `log_level_name`, `format_log_as_json` | | `src/test/cpp/test_jni_helpers.cpp` | 56 | All functions in `jni_helpers.hpp` using a zero-filled `JNINativeInterface_` mock (incl. the `utf8_to_jstring_impl` byte-array string path: emoji byte-preservation, truncated-UTF-8 replace-not-throw) | | `src/test/cpp/test_tts_wav.cpp` | 2 | The in-memory WAV writer `pcm_to_wav16_bytes` in `tts_wav.hpp` (WAV header/payload + little-endian clamping) — our own code, not upstream. The Qwen3-TTS pipeline it pairs with (`mtmd_helper::gen_audio`) is entirely upstream-owned (no project-side DSP to unit-test here). The load path is additionally covered by `test_tts_params.cpp` (5 tests over `tts_params.hpp`'s `build_tts_params`), which pins the CPU-thread resolution whose absence used to crash the JVM on every platform — see the `TODO.md` entry for the mechanism. End-to-end coverage is `TtsIntegrationTest`, which is model-gated. | -**Current total: 504 tests (all passing).** +**Current total: 509 tests (all passing).** #### Upstream source location (in CMake build tree) diff --git a/llama/patches/0011-peg-parser-lenient-invalid-utf8.patch b/llama/patches/0011-peg-parser-lenient-invalid-utf8.patch new file mode 100644 index 000000000..0f2d96d03 --- /dev/null +++ b/llama/patches/0011-peg-parser-lenient-invalid-utf8.patch @@ -0,0 +1,74 @@ +diff --git a/common/peg-parser.cpp b/common/peg-parser.cpp +index 46fc29bf2..3f2008217 100644 +--- a/common/peg-parser.cpp ++++ b/common/peg-parser.cpp +@@ -680,7 +680,16 @@ struct parser_executor { + + if (utf8_result.status == utf8_parse_result::INVALID) { + // Malformed UTF-8 +- return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos); ++ if (!ctx.is_lenient()) { ++ return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos); ++ } ++ // Lenient: keep what was scanned before the malformed byte instead of failing the ++ // whole parse, mirroring the INCOMPLETE branch above. Failing here loses a result ++ // that was produced successfully: common_chat_peg_parse() always parses in lenient ++ // mode, and the server runs it over every completion (content-only when the request ++ // configures no chat parser), so a single stray byte anywhere in the generated text ++ // turns a finished generation into an HTTP 500 instead of a response. ++ return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT, start_pos, last_valid_pos); + } + + // Check if a delimiter starts at this position +diff --git a/tests/peg-parser/test-unicode.cpp b/tests/peg-parser/test-unicode.cpp +index 24663d701..9ba64b519 100644 +--- a/tests/peg-parser/test-unicode.cpp ++++ b/tests/peg-parser/test-unicode.cpp +@@ -238,6 +238,47 @@ void test_unicode(testing &t) { + } + }); + ++ t.test("invalid UTF-8 is tolerated when lenient", [](testing &t) { ++ // A malformed byte must not fail the whole parse in lenient mode: common_chat_peg_parse() ++ // always parses leniently and the server runs it over every completion, so failing here ++ // would turn a finished generation into an error response. Keep what was scanned before ++ // the bad byte instead, exactly like the incomplete-sequence case above. ++ std::vector test_cases { ++ // Lone continuation byte in the middle ++ {std::string("Hello\x80World"), "Hello", COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT}, ++ ++ // Truncated CJK sequence followed by more bytes ++ {std::string("ab\xE4\xB8cd"), "ab", COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT}, ++ ++ // Invalid lead byte ++ {std::string("abc\xFF" "d"), "abc", COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT}, ++ }; ++ ++ auto parser = build_peg_parser([](common_peg_parser_builder& p) { ++ return p.until(""); ++ }); ++ ++ for (size_t i = 0; i < test_cases.size(); i++) { ++ const auto & tc = test_cases[i]; ++ std::string test_name = "case " + std::to_string(i) + ": " + hex_dump(tc.input); ++ ++ t.test(test_name, [&](testing &t) { ++ common_peg_parse_context lenient(tc.input, COMMON_PEG_PARSE_FLAG_LENIENT); ++ auto result = parser.parse(lenient); ++ ++ assert_result_equal(t, tc.expected_result, result.type); ++ ++ std::string matched = tc.input.substr(result.start, result.end - result.start); ++ t.assert_equal(tc.expected_text, matched); ++ ++ // Strict mode still rejects malformed input. ++ common_peg_parse_context strict(tc.input); ++ auto strict_result = parser.parse(strict); ++ assert_result_equal(t, COMMON_PEG_PARSE_RESULT_FAIL, strict_result.type); ++ }); ++ } ++ }); ++ + t.test("incomplete UTF-8 at end", [](testing &t) { + std::vector test_cases { + // Incomplete emoji at end, no delimiter diff --git a/llama/src/test/cpp/test_utils.cpp b/llama/src/test/cpp/test_utils.cpp index db724b44a..a389d225b 100644 --- a/llama/src/test/cpp/test_utils.cpp +++ b/llama/src/test/cpp/test_utils.cpp @@ -1434,3 +1434,68 @@ TEST(FormatAnthropicSse, Array_EachElementDispatchedCorrectly) { // second element is bare EXPECT_EQ(s.find("event: bare"), std::string::npos); } + +// ============================================================ +// common_chat_parse — the content-only path over malformed UTF-8 +// +// Guards patches/0011. The server parses *every* completion through +// common_chat_parse(); with no chat parser configured that is the +// content-only fallback (`content(rest()) + end()`), whose scan is +// common_peg_until_parser. Upstream lets that scan tolerate an +// INCOMPLETE trailing UTF-8 sequence in lenient mode (and +// common_chat_peg_parse always parses leniently) but hard-fails on an +// INVALID byte, which turns a generation that finished normally into +// an HTTP 500 — "The model produced output that does not match the +// expected Content-only format" — for output the model really did +// produce. The patch makes the INVALID branch respect leniency the +// same way, keeping the text up to the bad byte. +// +// These tests are the runnable half of that guard: if a llama.cpp bump +// drops the patch, or upstream reverts to failing, they go red here +// rather than in a model-backed Java integration test on one platform. +// ============================================================ + +namespace { + +// Parse `raw` exactly the way the server parses a finished completion with no +// chat parser configured: default params (format = content-only, empty parser), +// is_partial = false. +std::string parse_content_only(const std::string &raw) { + const common_chat_parser_params params; + return common_chat_parse(raw, /*is_partial=*/false, params).content; +} + +} // namespace + +TEST(ContentOnlyParseUtf8, ValidMultiByteContent_SurvivesByteForByte) { + const std::string in = "Hello \xE4\xB8\x96\xE7\x95\x8C \xF0\x9F\x98\x80!"; + EXPECT_EQ(parse_content_only(in), in); +} + +TEST(ContentOnlyParseUtf8, LoneContinuationByte_DoesNotThrow) { + // The failure that reached CI: a stray continuation byte in the middle of + // the generated text made the whole request 500. + std::string content; + EXPECT_NO_THROW(content = parse_content_only(std::string("Hello\x80World"))); + EXPECT_EQ(content, "Hello"); +} + +TEST(ContentOnlyParseUtf8, TruncatedSequenceFollowedByMoreBytes_DoesNotThrow) { + std::string content; + EXPECT_NO_THROW(content = parse_content_only(std::string("ab\xE4\xB8") + "cd")); + EXPECT_EQ(content, "ab"); +} + +TEST(ContentOnlyParseUtf8, InvalidLeadByte_DoesNotThrow) { + std::string content; + EXPECT_NO_THROW(content = parse_content_only(std::string("abc\xFF") + "d")); + EXPECT_EQ(content, "abc"); +} + +TEST(ContentOnlyParseUtf8, IncompleteTrailingSequence_DoesNotThrow) { + // Upstream already tolerated this one; pinned so the two malformed-UTF-8 + // branches cannot drift apart again. + std::string content; + EXPECT_NO_THROW(content = parse_content_only(std::string("abc\xE2\x82"))); + EXPECT_EQ(content, "abc"); +} From 7b9309d4b233c0a308f9976d4336402b642cde43 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 22:11:16 +0000 Subject: [PATCH 57/76] docs: record the 8-patch re-verification for b10639 The two b10636-b10639 rows both state "all 7 patches apply unchanged", which was the accurate record of what the bump itself required. Adding patches/0011 afterwards would have quietly falsified them, so rather than rewrite that record the verification row now carries the re-run: a second fresh build directory through the real fail-loud applier with 8 patches in the stamp, ctest 509/509, nm -D unchanged at 40 Java_ exports and 0 mangled, NativeLibraryLoadSmokeTest 3/3 with 0 skipped. The note also states plainly that 0011 is not a b10636-b10639 regression -- that range touches no file under common/ at all. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- docs/history/llama-cpp-breaking-changes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index c703ad969..77f826e3a 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -676,4 +676,4 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10631–b10636 | `ggml/src/ggml-cuda/mmq*` (quantised matmul kernel configs for Pascal), `ggml/src/ggml-metal/*` + `kernels/ssm.metal` (SSM/Mamba Metal kernels), `CMakeLists.txt` (**`LLAMA_BUILD_UI` default `ON` → `OFF`; the `LLAMA_USE_PREBUILT_UI` help text drops its `requires LLAMA_BUILD_UI=ON` clause**), `conversion/nemotron.py`, `tests/test-backend-ops.cpp`, nine `.github/workflows/*` files, plus `tools/ui/**` (the WebUI, auto-followed) | **No project-source change, and nothing on the priority review list is touched at all.** The diff contains **zero** files under `common/`, `include/`, `tools/server/` or `tools/mtmd/`, so every row of the API-compatibility table is vacuously satisfied and the three mechanical server-contract greps have no input to compare — the request-field set, its bounds and the emitted response keys cannot have moved. The ggml changes are backend kernel internals behind unchanged public headers (CUDA affects only the `cuda13-*` classifiers, Metal only the macOS default JAR). The upstream `LLAMA_BUILD_UI` flip is inert here: this project never uses upstream's UI build, it compiles its own `webui-generated/ui.cpp` produced by the `build-webui` CI job, and as a `FetchContent` subproject `LLAMA_STANDALONE` is OFF anyway. **All 7 patches apply unchanged** — none of `common/arg.{cpp,h}`, `tools/server/*` or `vendor/*` differs between the tags. Sizing: the reviewable diff (excluding `tools/ui`) is ~85 KiB, under the runbook's 100 KiB single-step threshold. | | b10631–b10636 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10636 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10636: all **7** patches applied, confirmed by the stamp file (`head 4d19b28769…` plus one SHA-256 line per patch) and by the source tree going dirty in 41 files. Full `cmake --build --config Release` clean, `ctest` **504/504**. `nm -D` on the freshly linked `libjllama.so` reports **zero** C++-mangled `Java_*` exports, so the `LlamaQuantizer` linkage fix carries forward. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10636") matches the `build-info` compiled into the new binary, which is what proves the four pin sites and the actual build agree. | | b10636–b10639 | `ggml/src/ggml-rpc/ggml-rpc.cpp` + `ggml/include/ggml-rpc.h` (**#18626: event and async backend APIs for the RPC backend; `RPC_PROTO_{MAJOR,MINOR}_VERSION` 5.1 → 6.0**), `ggml/src/ggml-vulkan/ggml-vulkan.cpp` + two new `vulkan-shaders/cross_entropy_loss{,_back}.comp` + `vulkan-shaders-gen.cpp` (**#27216: Vulkan `cross_entropy_loss` / `_back`; #27726: warptiles clamped because they assume warp sizes ≤ 64**), `docs/ops.md`, `docs/ops/Vulkan.csv` | **No project-source change, and nothing on the priority review list is touched at all.** Eight files, zero of them under `common/`, `include/llama.h`, `tools/server/` or `tools/mtmd/`, so every row of the API-compatibility table is vacuously satisfied and the three mechanical server-contract greps have no input to compare — the request-field set, its bounds and the emitted response keys cannot have moved. Unlike the previous two ranges there is **no `tools/ui` component at all**, so the whole ~70 KiB raw diff is the reviewable diff, comfortably under the runbook's 100 KiB single-step threshold. The one public header in the range is `ggml/include/ggml-rpc.h`, and its entire diff is the two protocol-version macros: **`GGML_RPC` is never enabled anywhere in this project** (`grep -rn GGML_RPC` across the workflows, build scripts and CMake is empty), so `ggml-rpc.cpp` is not compiled into `libjllama` and the protocol bump is a wire-compatibility concern only for operators running upstream's `rpc-server`, which this project does not ship. The Vulkan work is backend kernel internals behind unchanged public headers and reaches only the `vulkan-linux-x86-64` / `vulkan-linux-aarch64` / `vulkan-windows-x86-64` classifiers, whose CI jobs are build-only (GitHub runners have no GPU), so the CI signal for them is compilation. **All 7 patches apply unchanged** — the patch target list and the eight changed files have **zero** overlap (no `common/arg.{cpp,h}`, no `tools/server/*`, no `vendor/*`). | -| b10636–b10639 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10639 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10639: all **7** patches applied, confirmed by the stamp file (`head 5e6a37cb115dc1074e274ac004373f5661909695` plus one SHA-256 line per patch) and by the source tree going dirty in 41 files. A fresh build directory was mandatory here — the applier pins its stamp to the checked-out llama.cpp commit and deliberately aborts rather than guess when an existing build dir's stamp names a different one. Full `cmake --build --config Release` clean, `ctest` **504/504**. `nm -D` on the freshly linked `libjllama.so` reports **40** `Java_*` exports and **zero** C++-mangled ones, so the `LlamaQuantizer` linkage fix carries forward. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10639") matches the `build-info` compiled into the new binary (`b10639-5e6a37cb1`), which is what proves the four pin sites and the actual build agree. **Note for the next bump:** that last check first reported a *false* drift (`must start with the pinned tag "b10636-"`) even though both the source and `target/classes` already read `b10639`. `LLAMA_CPP_VERSION` is a compile-time constant, so javac had inlined the old value into the already-compiled `NativeLibraryLoadSmokeTest.class` and Maven's incremental compilation cannot see that dependency; `mvn clean test` clears it. Recorded in the runbook so the next bump does not re-diagnose it. | +| b10636–b10639 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10639 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10639: all **7** patches applied, confirmed by the stamp file (`head 5e6a37cb115dc1074e274ac004373f5661909695` plus one SHA-256 line per patch) and by the source tree going dirty in 41 files. A fresh build directory was mandatory here — the applier pins its stamp to the checked-out llama.cpp commit and deliberately aborts rather than guess when an existing build dir's stamp names a different one. Full `cmake --build --config Release` clean, `ctest` **504/504**. `nm -D` on the freshly linked `libjllama.so` reports **40** `Java_*` exports and **zero** C++-mangled ones, so the `LlamaQuantizer` linkage fix carries forward. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10639") matches the `build-info` compiled into the new binary (`b10639-5e6a37cb1`), which is what proves the four pin sites and the actual build agree. **Note for the next bump:** that last check first reported a *false* drift (`must start with the pinned tag "b10636-"`) even though both the source and `target/classes` already read `b10639`. `LLAMA_CPP_VERSION` is a compile-time constant, so javac had inlined the old value into the already-compiled `NativeLibraryLoadSmokeTest.class` and Maven's incremental compilation cannot see that dependency; `mvn clean test` clears it. Recorded in the runbook so the next bump does not re-diagnose it. **Re-verified after `patches/0011` was added** (a long-standing upstream bug this bump's CI was the first run ever to reach — see the patch table in `CLAUDE.md`; it is *not* a b10636–b10639 regression, the range touches no `common/` file at all): a second fresh build directory, applier run and Release build with **8** patches in the stamp, `ctest` **509/509**, `nm -D` unchanged at 40/0, `NativeLibraryLoadSmokeTest` still 3/3 with 0 skipped. The "all 7 patches" counts in the two rows above are left as written — they are the accurate record of what the *bump* required. | From 6cdd7b0a1ed47822ebef2c127d78960037827d23 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 07:19:45 +0000 Subject: [PATCH 58/76] chore: upgrade llama.cpp from b10639 to b10644 Four pin sites: llama/CMakeLists.txt GIT_TAG, LlamaCppVersion.LLAMA_CPP_VERSION, the README badge and link, and CLAUDE.md. No project source changes. The raw diff is ~608 KiB, far over the runbook's 100 KiB single-step threshold, but two components dominate it that this project never reviews or compiles: tools/ui (the WebUI, auto-followed -- the build-webui job rebuilds it from the pin) and the Snapdragon/Hexagon backend, which `grep -rn GGML_HEXAGON` shows is enabled nowhere in this repo's workflows, build scripts or CMake, so ggml-hexagon.cpp never reaches a libjllama link line. Excluding both leaves 11 KiB across 6 files -- reviewable in one step, no chunking needed. Of those six, the only one on the priority API-review list is include/llama.h, and its entire diff is two constants: LLAMA_SESSION_VERSION 9 -> 10 LLAMA_STATE_SEQ_VERSION 2 -> 3 They follow from llama_kv_cell_ext gaining a `tok` field (n-gram input embeddings) that has to survive a state save/restore, together with the supporting has_cell_ext() / get_prev_tokens() / for_each_token_in() helpers in src/llama-kv-cache.{cpp,h} and src/llama-kv-cells.h. That is a *state-file format* break, not an API break: a slot state saved by an older build -- via LlamaModel.handleSlotAction(..., save) or the server's /slots/{id}?action=save -- is rejected by its own version check and has to be regenerated. Nothing the project calls changed shape, and no Java signature moved. Recorded in the CHANGELOG so it is not a surprise to consumers. The rest is a one-line fix in the nanbeige model graph (register t_layer_inp) and an upstream backend-ops test. Zero files under common/, tools/server/ or tools/mtmd/ changed, so every remaining row of the API-compatibility table is vacuously satisfied and the three mechanical server-contract greps have no input -- the request-field set, its bounds and the emitted response keys cannot have moved. All 8 patches apply unchanged: the 6 project-relevant changed files and the 42 files the patches target have zero overlap. The one changed test, tests/test-backend-ops.cpp, is touched by no patch. Verified in a fresh build directory through the real FetchContent path, so the fail-loud applier ran for real: 8 patches in the stamp (head d7a2074112d2...), Release build clean, ctest 509/509, nm -D at 40 Java_ exports and 0 mangled, NativeLibraryLoadSmokeTest 3/3 with 0 skipped -- including the cross-check that the pin constant matches the linked build-info (b10644-d7a207411). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CHANGELOG.md | 11 +++++++++++ CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 6 files changed, 23 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f6e598312..5e8e6c770 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -62,6 +62,17 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by guarded for old glibc in the same change. - Android/Gradle toolchain: Gradle pins moved 8.14.3 → 9.6.1 and the dockcross cross-compile images were bumped, alongside the AGP/Compose pin updates the Android builds needed. +- Upgraded llama.cpp from **b10639 to b10644**. No project-source change, and the only file on the + priority API-review list that the range touches is `include/llama.h`, whose entire diff is two + constants: `LLAMA_SESSION_VERSION` 9 → 10 and `LLAMA_STATE_SEQ_VERSION` 2 → 3. They follow from a new + `tok` field on `llama_kv_cell_ext` (n-gram input embeddings) that has to survive a state save/restore. + Everything else is the Snapdragon/Hexagon backend rework, a one-line fix in the nanbeige model graph, + and the WebUI. Nothing under `common/`, `tools/server/` or `tools/mtmd/` changed, so no request field, + no bound and no response key can have moved, and all eight local patches apply unchanged. + **One consumer-visible consequence:** the version bumps are a *state-file format* break. A slot state + saved by an earlier build — via `LlamaModel.handleSlotAction(..., save)` or the server's + `/slots/{id}?action=save` — is rejected by its own version check after this upgrade and has to be + regenerated. No Java or native signature changed. - Upgraded llama.cpp from **b10631 to b10639**, in two reviewed steps. Neither range changes any project source. b10631→b10636 is ggml-cuda quantised-matmul configs for Pascal, ggml-metal SSM/Mamba kernels, an upstream `LLAMA_BUILD_UI` default flip that is inert here (this project diff --git a/CLAUDE.md b/CLAUDE.md index fb82acf52..f2a1361ee 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10639** +Current llama.cpp pinned version: **b10644** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10639 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10644 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10639`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10644`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1442,7 +1442,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10639`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10644`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 69eb088fd..313a768a4 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10639](https://img.shields.io/badge/llama.cpp-%23b10639-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10639) +[![llama.cpp b10644](https://img.shields.io/badge/llama.cpp-%23b10644-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10644) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 77f826e3a..01b7995e8 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -677,3 +677,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10631–b10636 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10636 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10636: all **7** patches applied, confirmed by the stamp file (`head 4d19b28769…` plus one SHA-256 line per patch) and by the source tree going dirty in 41 files. Full `cmake --build --config Release` clean, `ctest` **504/504**. `nm -D` on the freshly linked `libjllama.so` reports **zero** C++-mangled `Java_*` exports, so the `LlamaQuantizer` linkage fix carries forward. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10636") matches the `build-info` compiled into the new binary, which is what proves the four pin sites and the actual build agree. | | b10636–b10639 | `ggml/src/ggml-rpc/ggml-rpc.cpp` + `ggml/include/ggml-rpc.h` (**#18626: event and async backend APIs for the RPC backend; `RPC_PROTO_{MAJOR,MINOR}_VERSION` 5.1 → 6.0**), `ggml/src/ggml-vulkan/ggml-vulkan.cpp` + two new `vulkan-shaders/cross_entropy_loss{,_back}.comp` + `vulkan-shaders-gen.cpp` (**#27216: Vulkan `cross_entropy_loss` / `_back`; #27726: warptiles clamped because they assume warp sizes ≤ 64**), `docs/ops.md`, `docs/ops/Vulkan.csv` | **No project-source change, and nothing on the priority review list is touched at all.** Eight files, zero of them under `common/`, `include/llama.h`, `tools/server/` or `tools/mtmd/`, so every row of the API-compatibility table is vacuously satisfied and the three mechanical server-contract greps have no input to compare — the request-field set, its bounds and the emitted response keys cannot have moved. Unlike the previous two ranges there is **no `tools/ui` component at all**, so the whole ~70 KiB raw diff is the reviewable diff, comfortably under the runbook's 100 KiB single-step threshold. The one public header in the range is `ggml/include/ggml-rpc.h`, and its entire diff is the two protocol-version macros: **`GGML_RPC` is never enabled anywhere in this project** (`grep -rn GGML_RPC` across the workflows, build scripts and CMake is empty), so `ggml-rpc.cpp` is not compiled into `libjllama` and the protocol bump is a wire-compatibility concern only for operators running upstream's `rpc-server`, which this project does not ship. The Vulkan work is backend kernel internals behind unchanged public headers and reaches only the `vulkan-linux-x86-64` / `vulkan-linux-aarch64` / `vulkan-windows-x86-64` classifiers, whose CI jobs are build-only (GitHub runners have no GPU), so the CI signal for them is compilation. **All 7 patches apply unchanged** — the patch target list and the eight changed files have **zero** overlap (no `common/arg.{cpp,h}`, no `tools/server/*`, no `vendor/*`). | | b10636–b10639 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10639 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10639: all **7** patches applied, confirmed by the stamp file (`head 5e6a37cb115dc1074e274ac004373f5661909695` plus one SHA-256 line per patch) and by the source tree going dirty in 41 files. A fresh build directory was mandatory here — the applier pins its stamp to the checked-out llama.cpp commit and deliberately aborts rather than guess when an existing build dir's stamp names a different one. Full `cmake --build --config Release` clean, `ctest` **504/504**. `nm -D` on the freshly linked `libjllama.so` reports **40** `Java_*` exports and **zero** C++-mangled ones, so the `LlamaQuantizer` linkage fix carries forward. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10639") matches the `build-info` compiled into the new binary (`b10639-5e6a37cb1`), which is what proves the four pin sites and the actual build agree. **Note for the next bump:** that last check first reported a *false* drift (`must start with the pinned tag "b10636-"`) even though both the source and `target/classes` already read `b10639`. `LLAMA_CPP_VERSION` is a compile-time constant, so javac had inlined the old value into the already-compiled `NativeLibraryLoadSmokeTest.class` and Maven's incremental compilation cannot see that dependency; `mvn clean test` clears it. Recorded in the runbook so the next bump does not re-diagnose it. **Re-verified after `patches/0011` was added** (a long-standing upstream bug this bump's CI was the first run ever to reach — see the patch table in `CLAUDE.md`; it is *not* a b10636–b10639 regression, the range touches no `common/` file at all): a second fresh build directory, applier run and Release build with **8** patches in the stamp, `ctest` **509/509**, `nm -D` unchanged at 40/0, `NativeLibraryLoadSmokeTest` still 3/3 with 0 skipped. The "all 7 patches" counts in the two rows above are left as written — they are the accurate record of what the *bump* required. | +| b10639–b10644 | `include/llama.h` (**`LLAMA_SESSION_VERSION` 9 → 10 and `LLAMA_STATE_SEQ_VERSION` 2 → 3** — the only two lines in the file that moved), `src/llama-kv-cells.h` + `src/llama-kv-cache.{cpp,h}` (`llama_kv_cell_ext` gains a `tok` field for n-gram input embeddings, plus `has_cell_ext()` / `get_prev_tokens()` / `for_each_token_in()`, and `reset()` swaps a `memset` for value-initialisation), `src/models/nanbeige.cpp` (one line: register `t_layer_inp[il]`), `tests/test-backend-ops.cpp`, the whole `ggml/src/ggml-hexagon/**` + `scripts/snapdragon/**` + `docs/backend/snapdragon/**` Snapdragon/Hexagon backend rework, plus `tools/ui/**` (the WebUI, auto-followed) | **No project-source change, and the only priority-review-list file touched is `include/llama.h`, whose entire diff is the two version macros.** Sizing: the raw diff is ~608 KiB, but it is dominated by two components this project never compiles or reviews — `tools/ui` (auto-followed; the `build-webui` job rebuilds it from the pin) and the Hexagon backend (`grep -rn GGML_HEXAGON` over this repo's workflows, build scripts and CMake is empty, so `ggml-hexagon.cpp` is never in a `libjllama` link line). Excluding both leaves **11 KiB across 6 files**, well under the runbook's 100 KiB single-step threshold, so no chunking was needed. **Zero** files under `common/`, `tools/server/` or `tools/mtmd/` changed, so every remaining row of the API-compatibility table is vacuously satisfied and the three mechanical server-contract greps have no input — the request-field set, its bounds and the emitted response keys cannot have moved. **Compatibility note for consumers, not a compile break:** the two version bumps are a *state-file format* break — a slot state saved by an older build (`LlamaModel.handleSlotAction` save, or the server's `/slots/{id}?action=save`) is rejected by its own version check on b10644 and must be regenerated. Nothing the project calls changed shape. **All 8 patches apply unchanged** — the 6 project-relevant changed files and the 42 files the patches target have **zero** overlap (no `common/arg.{cpp,h}`, no `common/peg-parser.cpp`, no `tools/server/*`; the one changed test, `tests/test-backend-ops.cpp`, is touched by no patch). | +| b10639–b10644 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10644 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10644: all **8** patches applied, confirmed by the stamp file (`head d7a2074112d27649303fa107eb8c94db1ee435f3` plus one SHA-256 line per patch) — including `0011`, whose target `common/peg-parser.cpp` the range does not touch. Full `cmake --build --config Release` clean, `ctest` **509/509**. `nm -D` on the freshly linked `libjllama.so` reports **40** `Java_*` exports and **zero** C++-mangled ones. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10644") matches the `build-info` compiled into the new binary (`b10644-d7a207411`), which is what proves the four pin sites and the actual build agree. Run with `mvn clean test`, not a bare `mvn test`: `LLAMA_CPP_VERSION` is a compile-time constant that javac inlines into the already-compiled test class, and Maven's incremental compilation cannot see that dependency — the b10636→b10639 row below records the false drift that costs. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index d2f27fa73..216d9105b 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10639 + GIT_TAG b10644 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 589d6e62c..746e202ef 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10639"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10644"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10639-"} — call + * plus the resolved upstream commit, e.g. {@code "b10644-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10639"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10644"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10639"; + public static final String LLAMA_CPP_VERSION = "b10644"; // Constants holder — not instantiable. private LlamaCppVersion() {} From fe5ef7bb6b1f80ab86d9d323b58df0bce92d3b7d Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 08:34:11 +0000 Subject: [PATCH 59/76] fix: close every finding from the b10456-b10644 upgrade audit Three independent sweeps over the upstream range -- completeness, adaptation correctness, test integrity -- found no missed adaptation: the request-field set, its bounds and the emitted response keys are identical at both ends, libjllama links with zero undefined upstream symbols, and patch 0007's route-table invariant still holds byte-for-byte. What they did find were documentation errors, one real hang, and coverage that did not guard what it claimed to. All of it is fixed here. Wrong facts, corrected - The -1 context-size sentinel was dropped upstream at b10273 (#26524), not b10275: git tag --contains puts it in b10273, and b10273..b10275 is empty over arg.cpp and server-schema.cpp. It was wrong in 4 shipped Javadoc blocks, 4 exception messages, and every doc that cited it. b10275 stays where it is correct -- the unrelated server-schema.h signature break. - The state-file break IS reachable from public API. saveSlot/restoreSlot are public and reach llama_state_seq_save_file, which is stamped with LLAMA_STATE_SEQ_VERSION (2 -> 3 at b10644). The CHANGELOG named the package-private handleSlotAction instead, so a consumer grepping for the API they call found nothing. Both methods now document the version lock and that a mismatch surfaces as upstream's misleading "No available space in KV cache or invalid slot save file". - CLAUDE.md claimed only AudioInputIntegrationTest self-skips. LlamaTrainerIntegrationTest does too: net.ladenthin.llama.train.model is set by no job and its model is in no models.csv row. A comment that would have blocked a real fix post_and_wait said the idle-sleep guard was unreachable because the getter is not on server_context's public header. True premise, wrong conclusion: server_response_reader::queue_tasks and server_queue::is_sleeping() are both public. Since post_task does not wake a sleeping queue, a getMetrics() issued while asleep blocked until close(). The predicate now mirrors upstream's own, closing || queue_tasks.is_sleeping(). Needs setSleepIdleSeconds(> 0). Guards that did not guard - patches/0010 had none. CommonJsonEnumTrap builds its own JSON and calls no project code, so reverting the cast left it green; get_res_model_info is static and unreachable from jllama_test. NativeServerAttachIntegrationTest now drives GET /v1/models and asserts vocab_type is an integer -- covering the patch on every Java platform, and doubling as the signal to DROP it if upstream ever casts the value themselves. - train_engine.cpp carried the same postprocess_cpu_params pair as tts_params.hpp -- inline, and its only test never runs in CI. Both now call one shared jllama::resolve_cpu_params (cpu_params.hpp) with three tests, so the JVM-abort bug cannot regress unseen in the trainer. - testGetMetrics checked 8 of the 21 merged keys, so a rename in the C++ helper alone still shipped and the getter silently returned its default. It now asserts the full set, with the right JSON type per key. - Two gates still read their property with a bare System.getProperty, bypassing the resolver whose absence muted the whole model-backed suite. Both routed through it, and both modules gained a rule that fails with file:line if a new test reintroduces one -- a rule, not a snapshot of today's call sites. - llama-langchain4j's copied resolver had no test at all; it has one now. - The langchain4j streaming test had been weakened to "content OR thinking", which a content-routing regression passes. Budget raised past Qwen3-0.6B's ~200 thinking tokens so it can assert real content again. Gaps found while looking - Deprecated withTfsZ, withPenalizeNl and both withPenaltyPrompt overloads: tfs_z, penalize_nl and penalty_prompt appear nowhere upstream, and the schema discards unknown fields rather than rejecting them, so they have been doing nothing while their Javadoc described working knobs. - setMmprojDevice and setMmprojOffload(false) both write mmproj_use_gpu, and argv comes out of a HashMap -- so the winner was hash order. The device setter now clears the conflicting flags. - getModelMeta() emitted only vision and audio while upstream has tracked has_inp_video for releases and emits all three from /props; added, with ModelMeta.supportsVideo() and tests including the older-metadata shape. - ServerMetrics gained typed accessors for the window timings t_prompt_processing / t_tokens_generation, which were emitted but unreachable. - getMetrics() documents that the merged payload is not an atomic snapshot and that it defers idle-sleep, which upstream's /metrics stopped doing at b10644. Verified: ctest 512/512 (509 + 3 new), full Java suite 1443 run / 0 failures, javadoc jar builds clean, spotless applied. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CHANGELOG.md | 50 ++++++++- CLAUDE.md | 15 ++- TODO.md | 2 +- docs/upgrade/llama-cpp-version-bump.md | 2 +- .../JllamaChatModelIntegrationTest.java | 31 +++--- .../llama/langchain4j/TestModelPathsTest.java | 105 ++++++++++++++++++ llama/src/main/cpp/cpu_params.hpp | 45 ++++++++ llama/src/main/cpp/jllama.cpp | 23 ++-- llama/src/main/cpp/train_engine.cpp | 15 +-- llama/src/main/cpp/tts_params.hpp | 19 +--- .../java/net/ladenthin/llama/LlamaModel.java | 33 ++++++ .../llama/parameters/InferenceParameters.java | 36 +++++- .../llama/parameters/ModelParameters.java | 14 ++- .../net/ladenthin/llama/value/ModelMeta.java | 13 +++ .../ladenthin/llama/value/ServerMetrics.java | 42 +++++++ llama/src/test/cpp/test_tts_params.cpp | 51 +++++++++ .../java/examples/OpenAiServerExample.java | 4 +- .../net/ladenthin/llama/LlamaModelTest.java | 34 +++++- .../llama/LlamaTrainerIntegrationTest.java | 2 +- .../SessionForkRewindIntegrationTest.java | 3 +- .../ladenthin/llama/TestConstantsTest.java | 81 ++++++++++++++ .../parameters/InferenceParametersTest.java | 2 +- .../JsonEndpointParametersTest.java | 2 +- .../llama/parameters/ModelParametersTest.java | 8 +- .../NativeServerAttachIntegrationTest.java | 39 +++++++ .../ladenthin/llama/value/ModelMetaTest.java | 19 +++- .../llama/value/ServerMetricsTest.java | 31 ++++++ 27 files changed, 643 insertions(+), 78 deletions(-) create mode 100644 llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/TestModelPathsTest.java create mode 100644 llama/src/main/cpp/cpu_params.hpp diff --git a/CHANGELOG.md b/CHANGELOG.md index 5e8e6c770..401e6fd03 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,13 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by > has a row per upgrade range and stays authoritative for the per-range detail. ### Added +- **`ServerMetrics.getWindowPromptProcessingMillis()` / `getWindowTokenGenerationMillis()` / + `getWindowTimings()`** — typed access to the current-window timing keys `t_prompt_processing` and + `t_tokens_generation`. Both were always emitted; only the cumulative `_total` variants had accessors. +- **`ModelMeta.supportsVideo()`**, and `getModelMeta()` now emits `modalities.video`. Upstream has + tracked `has_inp_video` on `server_context_meta` for releases and emits all three modalities from its + own `/props`; this binding emitted only vision and audio, so feature detection concluded no model + ever accepts video. - `QuantizationType.Q2_0` — maps the new upstream `LLAMA_FTYPE_MOSTLY_Q2_0` (llama.cpp b9916) for `LlamaQuantizer`. - **Voice cloning and language selection for `TextToSpeech`**: `synthesize(String text, String speakerReferenceAudioPath, String language, int maxFrames, int topK, int seed)` — a speaker-reference clip makes the model imitate that voice. Part of the Qwen3-TTS rework (see Changed). - **`ModelParameters.setMmprojDevice(String)`** — places the multimodal projector on a device of its own @@ -31,6 +38,16 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by as Prometheus text; they now arrive in the JSON payload. ### Changed +- **Deprecated `InferenceParameters.withTfsZ`, `withPenalizeNl` and both `withPenaltyPrompt` overloads.** + `tfs_z`, `penalize_nl` and `penalty_prompt` appear nowhere in upstream `common/` or `tools/server/` + at the pinned build, and the request schema discards unknown fields rather than rejecting them — so + these have been silently doing nothing. Kept compiling for now; they will be removed. +- `ModelParameters.setMmprojDevice` now clears the mmproj-offload flags. Both write upstream's single + `mmproj_use_gpu` field, and the rendered argv comes out of a `HashMap`, so setting both previously + left the winner to hash order. +- `getMetrics()` no longer blocks until `close()` when the task queue is asleep. The wait predicate now + mirrors upstream's own (`closing || queue_tasks.is_sleeping()`); a prior comment wrongly claimed the + getter was unreachable from this layer. Only reachable with `setSleepIdleSeconds(> 0)`, off by default. - `ch.qos.logback:logback-classic` bumped 1.6.2 → 1.6.3 (test/runtime binding only). - CI actions bumped to latest: `actions/setup-java` v5 → v6. - Upgraded llama.cpp from **b9894 to b9917** (all eight local patches re-verified across the range). @@ -43,7 +60,7 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by mmproj that bundles speaker encoder, code predictor and code2wav decoder — an OuteTTS + WavTokenizer pair no longer works and fails at load, not at compile time. `synthesize`'s `maxCodeTokens` parameter became `maxFrames`, and the single-argument overload's default dropped 4096 → 512. -- **BREAKING — `-1` is no longer accepted for the repetition-penalty windows** (llama.cpp **b10275**). +- **BREAKING — `-1` is no longer accepted for the repetition-penalty windows** (llama.cpp **b10273**). `repeat_last_n` and `dry_penalty_last_n` used to take `-1` for "the whole context"; upstream removed the sentinel, moving the request schema's hard limits to `[0, INT32_MAX]` and making `common_params_parse` throw on a negative value. `ModelParameters.setRepeatLastN` / @@ -62,6 +79,27 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by guarded for old glibc in the same change. - Android/Gradle toolchain: Gradle pins moved 8.14.3 → 9.6.1 and the dockcross cross-compile images were bumped, alongside the AGP/Compose pin updates the Android builds needed. +- **Post-upgrade audit of the whole b10456→b10644 range** — three independent sweeps over the + upstream diff (completeness, adaptation correctness, test integrity) against the files the binding + actually consumes. No missed adaptation was found: the request-field set, their bounds and the + emitted response keys are identical at both ends of the range, and `libjllama` links with zero + undefined upstream symbols. The audit did surface documentation and coverage gaps, fixed here: + - The `-1` context-size sentinel was dropped upstream at **b10273** (#26524), not b10275 — corrected + in 4 Javadoc blocks, 4 exception messages and every doc that cited it. `b10275` remains correct + for the unrelated `server-schema.h` signature break. + - `LlamaModel.saveSlot`/`restoreSlot` now document that the on-disk format is version-locked to the + linked llama.cpp build, and that a mismatch surfaces as upstream's misleading + `"No available space in KV cache or invalid slot save file"`. + - `getMetrics()` documents that the merged payload is not an atomic snapshot and that it defers + idle-sleep, which upstream's own `/metrics` stopped doing at b10644. + - **`--tools get_datetime` no longer starts.** Upstream deleted that built-in tool in this range and + an unknown name is fatal (`server_tools::setup` throws), so a `NativeServer` command line carrying + it now fails at startup. Same block: `server_tool::type()` reports `"server"` instead of + `"builtin"`, changing the `/tools` payload in full `NativeServer` mode. + - The four `t_*` keys in `getMetrics()` are now fractional rather than whole milliseconds, because + the merge divides upstream's microseconds. `ServerMetrics` reads them as doubles; a consumer + parsing the raw JSON with an integer parser sees a type change. + - Upgraded llama.cpp from **b10639 to b10644**. No project-source change, and the only file on the priority API-review list that the range touches is `include/llama.h`, whose entire diff is two constants: `LLAMA_SESSION_VERSION` 9 → 10 and `LLAMA_STATE_SEQ_VERSION` 2 → 3. They follow from a new @@ -70,9 +108,13 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by and the WebUI. Nothing under `common/`, `tools/server/` or `tools/mtmd/` changed, so no request field, no bound and no response key can have moved, and all eight local patches apply unchanged. **One consumer-visible consequence:** the version bumps are a *state-file format* break. A slot state - saved by an earlier build — via `LlamaModel.handleSlotAction(..., save)` or the server's - `/slots/{id}?action=save` — is rejected by its own version check after this upgrade and has to be - regenerated. No Java or native signature changed. + saved by an earlier build — via the public `LlamaModel.saveSlot(int, String)`, or the server's + `/slots/{id}?action=save` — is rejected by `LlamaModel.restoreSlot` after this upgrade and has to be + regenerated. No Java or native signature changed. The rejection is graceful but its message is + upstream's misleading `"No available space in KV cache or invalid slot save file"`, which does not + name the version mismatch; `saveSlot`'s Javadoc now spells this out. Slot state files are a cache to + regenerate on upgrade, not durable storage. The in-memory `Session` snapshot/fork feature is + unaffected — it never writes a file. - Upgraded llama.cpp from **b10631 to b10639**, in two reviewed steps. Neither range changes any project source. b10631→b10636 is ggml-cuda quantised-matmul configs for Pascal, ggml-metal SSM/Mamba kernels, an upstream `LLAMA_BUILD_UI` default flip that is inert here (this project diff --git a/CLAUDE.md b/CLAUDE.md index f2a1361ee..f49854020 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -843,7 +843,7 @@ header file diffs alone. `server_task_result_metrics::to_json()` to a bare slot array and b10519 split the task; no signature moved, every chunk compiled and linked clean, and `LlamaModel.getMetrics()` quietly returned the wrong shape for hundreds of builds. The same class hit `repeat_last_n` / - `dry_penalty_last_n` at b10275, where only a *value range* moved. Two cheap mechanical checks catch + `dry_penalty_last_n` at b10273, where only a *value range* moved. Two cheap mechanical checks catch these where a header diff cannot — run them on any bump that touches `tools/server/`: ```bash @@ -1372,9 +1372,13 @@ resolver as `TestModelPaths` (test classes are not shared between modules). `validate-models.{sh,bat}` treats all of these as **required** (a missing model hard-fails the job before tests run, so a -download regression can never silently downgrade to a skip). Only the audio-input model -(`AudioInputIntegrationTest`) still self-skips — the prompt clip is committed -(`src/test/resources/audios/sample.wav`) but the audio model + mmproj have no CI download. +download regression can never silently downgrade to a skip). **Two** classes still self-skip on +every platform, both because their model is outside the manifest: `AudioInputIntegrationTest` — the +prompt clip is committed (`src/test/resources/audios/sample.wav`) but the audio model + mmproj have +no CI download — and `LlamaTrainerIntegrationTest`, whose `net.ladenthin.llama.train.model` property +is set by no job and whose model is in no `models.csv` row. The trainer one matters more than it +looks: `train_engine.cpp` carries the same `postprocess_cpu_params` pair as `tts_params.hpp`, so the +JVM-abort class of bug documented under "Qwen3-TTS" can regress there with no runnable guard. The model set has a **single source of truth: `.github/models.csv`** (one `filename,url` row per model; `#` comments). Everything derives from it: the **`download-models`** job (ubuntu, `needs: startgate`) is the only place models are fetched from HuggingFace (one manifest-driven @@ -1437,8 +1441,9 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" | `src/test/cpp/test_log_helpers.cpp` | 13 | All functions in `log_helpers.hpp`: `log_level_name`, `format_log_as_json` | | `src/test/cpp/test_jni_helpers.cpp` | 56 | All functions in `jni_helpers.hpp` using a zero-filled `JNINativeInterface_` mock (incl. the `utf8_to_jstring_impl` byte-array string path: emoji byte-preservation, truncated-UTF-8 replace-not-throw) | | `src/test/cpp/test_tts_wav.cpp` | 2 | The in-memory WAV writer `pcm_to_wav16_bytes` in `tts_wav.hpp` (WAV header/payload + little-endian clamping) — our own code, not upstream. The Qwen3-TTS pipeline it pairs with (`mtmd_helper::gen_audio`) is entirely upstream-owned (no project-side DSP to unit-test here). The load path is additionally covered by `test_tts_params.cpp` (5 tests over `tts_params.hpp`'s `build_tts_params`), which pins the CPU-thread resolution whose absence used to crash the JVM on every platform — see the `TODO.md` entry for the mechanism. End-to-end coverage is `TtsIntegrationTest`, which is model-gated. | +| `src/test/cpp/test_tts_params.cpp` | 8 | The two builders every hand-assembled `common_params` goes through: `build_tts_params` (`tts_params.hpp`, 5 tests) and the shared `jllama::resolve_cpu_params` (`cpu_params.hpp`, 3 tests). The shared one is what guards **`train_engine.cpp`** too — its own integration test needs a GGUF no CI job downloads, so without these the JVM-abort bug could regress in the trainer on every platform, unseen. | -**Current total: 509 tests (all passing).** +**Current total: 512 tests (all passing).** #### Upstream source location (in CMake build tree) diff --git a/TODO.md b/TODO.md index 60ab0e6d7..fbec0de54 100644 --- a/TODO.md +++ b/TODO.md @@ -178,7 +178,7 @@ same crashed test. since symbol export and mangling differ between ELF/gcc and Mach-O/clang. - **[FIXED in this PR] `JsonEndpointParametersTest.testDryMultiplierAccepted` sent - `dry_penalty_last_n: -1`.** The one genuine b10456→b10618 regression in the list: b10275 gave the + `dry_penalty_last_n: -1`.** The one genuine b10456→b10618 regression in the list: b10273 gave the field hard limits `[0, INT32_MAX]` (0 = disabled) and dropped the old "-1 = context size" sentinel, so the request now 400s. The `InferenceParameters` / `ModelParameters` setters were already fixed in this PR; this test builds raw JSON and bypassed them. A repo-wide sweep confirms it was the only diff --git a/docs/upgrade/llama-cpp-version-bump.md b/docs/upgrade/llama-cpp-version-bump.md index 2e2dc901d..408a3a674 100644 --- a/docs/upgrade/llama-cpp-version-bump.md +++ b/docs/upgrade/llama-cpp-version-bump.md @@ -148,7 +148,7 @@ Concretely: 3. **Check the server contract mechanically when the chunk touches `tools/server/`.** A header diff only shows signature changes; it cannot see a *contract* change behind a stable signature. Two breaks of that class already shipped — `getMetrics()`'s payload shape (b10408/b10519) and the - removal of the `-1` = context-size sentinel for `repeat_last_n`/`dry_penalty_last_n` (b10275) — + removal of the `-1` = context-size sentinel for `repeat_last_n`/`dry_penalty_last_n` (b10273) — and neither was visible to the build. Diff these three sets between the two tags; anything that changes has to be traced to the Java layer, not just to the C++ tests: ```bash diff --git a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java index 4726eabd4..0ad98b1ec 100644 --- a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java +++ b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java @@ -64,7 +64,13 @@ void streamingDeliversTokensThenCompletes() throws Exception { streaming.chat( ChatRequest.builder() .messages(UserMessage.from("Reply with the single word: ok")) - .maxOutputTokens(8) + // Qwen3-0.6B is a reasoning model and spends ~200 tokens inside + // before it answers (see the core module's ReasoningBudgetTest, which + // budgets 1500 for exactly this reason). The adapter exposes no + // reasoning-budget knob, so the budget has to clear the thinking block or + // the run produces no assistant text at all -- and the assertion below + // would then be satisfied by a stream that delivered only thinking. + .maxOutputTokens(320) .build(), new StreamingChatResponseHandler() { @Override @@ -85,24 +91,21 @@ public void onError(Throwable error) { ChatResponse complete = done.get(60, TimeUnit.SECONDS); - // The CI model is Qwen3-0.6B, a reasoning model, and this request budgets 8 output - // tokens: a run can legitimately spend all of them inside and produce no - // assistant text at all. StreamingChunkAssembler then routes every token to - // onPartialThinking and leaves AiMessage.text() null (it only sets text when it - // accumulated some), while `streamed` stays "" -- which is the "expected \"\" but was - // null" this assertion used to fail with once the test actually ran. + // Two independent assertions, both of which must hold. // - // Assert the invariant that holds in both shapes -- the concatenated onPartialResponse - // fragments are exactly the final text -- and separately that the stream delivered - // something, so a stream that produces nothing at all still fails. + // 1. The concatenated onPartialResponse fragments are exactly the final text. This is the + // streaming contract itself: a regression that misroutes content deltas into + // reasoning_content, or drops a fragment, breaks it. String finalText = complete.aiMessage().text() == null ? "" : complete.aiMessage().text(); assertThat(finalText, is(streamed.toString())); - assertThat( - "stream delivered neither content nor reasoning tokens", - !streamed.toString().isEmpty() || complete.aiMessage().thinking() != null, - is(true)); + + // 2. Actual assistant CONTENT arrived -- not merely "content or thinking". With a budget + // that clears the thinking block this is the real signal; accepting thinking alone + // would let a content-routing regression pass unnoticed, which is what the earlier, + // 8-token version of this test did. + assertThat("stream delivered no assistant content", !streamed.toString().isEmpty(), is(true)); } } } diff --git a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/TestModelPathsTest.java b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/TestModelPathsTest.java new file mode 100644 index 000000000..fbb79c5b3 --- /dev/null +++ b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/TestModelPathsTest.java @@ -0,0 +1,105 @@ +// SPDX-FileCopyrightText: 2026 Bernard Ladenthin +// +// SPDX-License-Identifier: MIT + +package net.ladenthin.llama.langchain4j; + +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.Matchers.notNullValue; +import static org.hamcrest.Matchers.nullValue; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Collectors; +import java.util.stream.Stream; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * Guards {@link TestModelPaths}, which is this module's copy of the core module's model-path + * resolver. + * + *

It exists because of a defect that stayed invisible for months: Surefire's working directory is + * the module basedir while CI restores the GGUF cache to the reactor root, so a bare + * {@code models/} resolved to nothing, every model-gated class aborted in its + * {@code Assumptions.assumeTrue(exists)}, and the job stayed green with the tests reported as + * skipped. The resolver fixes that; until now nothing proved the resolver itself works, and + * nothing stopped a new test class from going back to a bare {@code System.getProperty}. + */ +class TestModelPathsTest { + + @Test + void nullAndEmptyResolveToNull() { + assertThat(TestModelPaths.resolve(null), is(nullValue())); + assertThat(TestModelPaths.resolve(""), is(nullValue())); + } + + @Test + void anExistingModuleRelativePathIsReturnedUnchanged(@TempDir Path tmp) throws IOException { + Path present = Files.createFile(tmp.resolve("present.gguf")); + Path resolved = TestModelPaths.resolve(present.toString()); + assertThat(resolved, is(notNullValue())); + assertThat(Files.exists(resolved), is(true)); + } + + @Test + void anAbsolutePathIsReturnedEvenWhenItDoesNotExist(@TempDir Path tmp) { + Path missing = tmp.resolve("missing.gguf").toAbsolutePath(); + assertThat(TestModelPaths.resolve(missing.toString()), is(missing)); + } + + @Test + void anUnresolvablePathComesBackUnchangedSoTheSkipMessageNamesIt() { + // Deliberately NOT null: the caller puts this in its assumption message, and "models/x.gguf + // not found" is a far more useful CI line than "null". + String wanted = "models/definitely-not-here-" + TestModelPathsTest.class.getSimpleName() + ".gguf"; + Path resolved = TestModelPaths.resolve(wanted); + assertThat(resolved, is(Paths.get(wanted))); + } + + @Test + void fromPropertyIsNullWhenThePropertyIsUnset() { + assertThat(TestModelPaths.fromProperty("net.ladenthin.llama.langchain4j.definitely.unset"), is(nullValue())); + } + + /** + * The rule, not a snapshot: no test in this module may read a model path with a bare + * {@code System.getProperty}, because that bypasses the resolver and silently re-mutes that class + * in CI. {@link TestModelPaths} itself is where the raw read legitimately lives. + */ + @Test + void noTestReadsAModelPropertyWithoutTheResolver() throws IOException { + Path testSources = Paths.get("src/test/java"); + assertTrue(Files.isDirectory(testSources), "test sources not on disk: " + testSources.toAbsolutePath()); + + List offenders = new ArrayList<>(); + try (Stream paths = Files.walk(testSources)) { + List javaFiles = paths.filter(Files::isRegularFile) + .filter(f -> f.getFileName().toString().endsWith(".java")) + .filter(f -> !f.getFileName().toString().equals("TestModelPaths.java")) + .filter(f -> !f.getFileName().toString().equals("TestModelPathsTest.java")) + .collect(Collectors.toList()); + for (Path file : javaFiles) { + List lines = Files.readAllLines(file, StandardCharsets.UTF_8); + for (int i = 0; i < lines.size(); i++) { + if (lines.get(i).contains("System.getProperty(\"net.ladenthin.llama")) { + offenders.add(file + ":" + (i + 1) + " " + lines.get(i).trim()); + } + } + } + } + + assertTrue( + offenders.isEmpty(), + "Read net.ladenthin.llama.* path properties through TestModelPaths, not System.getProperty " + + "- a bare read resolves against Surefire's module-basedir CWD and silently self-skips " + + "in CI. Offending sites:\n" + String.join("\n", offenders)); + } +} diff --git a/llama/src/main/cpp/cpu_params.hpp b/llama/src/main/cpp/cpu_params.hpp new file mode 100644 index 000000000..fa804a92a --- /dev/null +++ b/llama/src/main/cpp/cpu_params.hpp @@ -0,0 +1,45 @@ +// SPDX-FileCopyrightText: 2026 Bernard Ladenthin +// +// SPDX-License-Identifier: MIT +// +// The one place that resolves a hand-built common_params' CPU fields. +// +// Every common_params this project assembles by hand -- the TTS backbone (tts_params.hpp) and the +// trainer (train_engine.cpp) -- has to go through here before it reaches common_init_from_params. +// It is a header for the same reason tts_params.hpp is one: jllama_test does not compile +// tts_engine.cpp or train_engine.cpp (both need the mtmd/llama runtime), and a header lets the +// tests exercise the real resolver those TUs call rather than a copy that could drift from it. + +#ifndef JLLAMA_CPU_PARAMS_HPP +#define JLLAMA_CPU_PARAMS_HPP + +#include "common.h" + +namespace jllama { + +// --------------------------------------------------------------------------- +// Resolves params.cpuparams / params.cpuparams_batch the way common/arg.cpp does. +// +// This is load-bearing, not tidiness. common/arg.cpp is upstream's ONLY caller of +// postprocess_cpu_params, so a common_params assembled by hand never gets its CPU fields +// resolved: common_cpu_params::n_threads keeps its -1 default, and common_init_from_params does +// not fix it up. common_threadpools::init then finds tpp and tpp_batch mismatched, builds a +// separate batch pool, and ggml_threadpool_new computes +// workers_size = sizeof(struct ggml_compute_state) * tpp->n_threads +// with n_threads == -1 -- a huge size_t. ggml_aligned_malloc returns NULL and the very next line +// memsets it unchecked, so the process dies on memset(NULL, 0, huge): SIGSEGV at address 0, +// reported as __bzero on macOS and as a bare libc frame on Linux. Because the abort bypasses the +// JVM error handler it writes no hs_err_pid log, which is what made it expensive to find. +// +// The role_model argument on the second call is what makes the batch pool INHERIT the main pool's +// count instead of resolving independently; passing nullptr for both would let them diverge. +// This mirrors common/arg.cpp's own pair exactly. +// --------------------------------------------------------------------------- +inline void resolve_cpu_params(common_params ¶ms) { + postprocess_cpu_params(params.cpuparams, nullptr); + postprocess_cpu_params(params.cpuparams_batch, ¶ms.cpuparams); +} + +} // namespace jllama + +#endif // JLLAMA_CPU_PARAMS_HPP diff --git a/llama/src/main/cpp/jllama.cpp b/llama/src/main/cpp/jllama.cpp index afcdaf17f..cb987f5ac 100644 --- a/llama/src/main/cpp/jllama.cpp +++ b/llama/src/main/cpp/jllama.cpp @@ -354,18 +354,19 @@ static void populate_completion_task(server_task &task, jllama_context *jctx, // introspection endpoints, and waited on with the same `closing` predicate every other // wait site in this file uses, so close() can unblock a pending call. // -// KNOWN LIMITATION: a task posted just as the queue enters its idle-sleep state is never -// processed -- upstream's own /metrics handler says so verbatim ("a task posted right -// before sleeping is never processed, do not wait for it") and guards it by adding -// queue_tasks.is_sleeping() to its wait predicate. That getter is not on -// server_context's public header, so this layer cannot ask; the `closing` predicate is -// what bounds the wait instead. Reaching that state needs -// ModelParameters.setSleepIdleSeconds(> 0), which is off by default (-1). +// A task posted just as the queue enters its idle-sleep state is never processed -- +// upstream's own /metrics handler says so verbatim ("a task posted right before sleeping +// is never processed, do not wait for it") and guards it by adding queue_tasks.is_sleeping() +// to its wait predicate. We mirror that predicate exactly: server_context does not expose +// the getter, but the reader's `queue_tasks` member and server_queue::is_sleeping() are both +// public, so this layer can and does ask. Without it a call made while the queue is asleep +// blocks until close(), because post_task() does not wake a sleeping queue. Reaching that +// state needs ModelParameters.setSleepIdleSeconds(> 0), which is off by default (-1). [[nodiscard]] static server_task_result_ptr post_and_wait(JNIEnv *env, jllama_context *jctx, server_task task) { auto rd = jctx->server.get_response_reader(); task.id = rd.get_new_id(); rd.post_task(std::move(task), true); - auto result = rd.next([jctx] { return jctx->closing.load(); }); + auto result = rd.next([jctx, &rd] { return jctx->closing.load() || rd.queue_tasks.is_sleeping(); }); if (!result_ok_or_throw(env, result)) return nullptr; return result; @@ -957,7 +958,11 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_getModelMetaJson(J {"n_embd", m.model_n_embd_inp}, {"n_params", m.model_n_params}, {"size", m.model_size}, - {"modalities", {{"vision", m.has_inp_image}, {"audio", m.has_inp_audio}}}, + // All three modalities upstream tracks. `video` was omitted here while server_context_meta + // has carried it for releases, and upstream's own /props emits all three -- a consumer + // feature-detecting from getModelMeta() would have concluded no model ever accepts video. + {"modalities", + {{"vision", m.has_inp_image}, {"audio", m.has_inp_audio}, {"video", m.has_inp_video}}}, {"name", m.model_name}, {"architecture", arch}, {"ftype", m.model_ftype}, diff --git a/llama/src/main/cpp/train_engine.cpp b/llama/src/main/cpp/train_engine.cpp index add1da01b..fb1b5be93 100644 --- a/llama/src/main/cpp/train_engine.cpp +++ b/llama/src/main/cpp/train_engine.cpp @@ -5,6 +5,7 @@ #include "train_engine.h" #include "common.h" +#include "cpu_params.hpp" #include "ggml-opt.h" #include "llama.h" @@ -65,16 +66,10 @@ bool finetune(const finetune_config &cfg, std::string &err) { params.cache_type_k = GGML_TYPE_F32; params.cache_type_v = GGML_TYPE_F32; - // A hand-built common_params never passes through common_params_parse, and common/arg.cpp is - // the ONLY caller of postprocess_cpu_params -- common_init_from_params does not call it. Without - // these two lines common_cpu_params::n_threads keeps its -1 default, and common_threadpools::init - // hands that -1 to ggml_threadpool_new, whose workers_size = sizeof(ggml_compute_state) * - // n_threads then overflows to a huge size_t; the allocation returns NULL and the following - // memset is unchecked, so the process dies on memset(NULL, 0, huge) -- SIGSEGV at address 0. - // Mirrors common/arg.cpp's own two calls, including the role_model that makes the batch pool - // inherit rather than resolve independently. - postprocess_cpu_params(params.cpuparams, nullptr); - postprocess_cpu_params(params.cpuparams_batch, ¶ms.cpuparams); + // A hand-built common_params never passes through common_params_parse, so its CPU fields are + // still unresolved here; without this the process dies on a memset of NULL inside + // ggml_threadpool_new. Shared with tts_params.hpp -- see cpu_params.hpp for the mechanism. + jllama::resolve_cpu_params(params); llama_backend_init(); llama_numa_init(params.numa); diff --git a/llama/src/main/cpp/tts_params.hpp b/llama/src/main/cpp/tts_params.hpp index 163690341..09ed8b26e 100644 --- a/llama/src/main/cpp/tts_params.hpp +++ b/llama/src/main/cpp/tts_params.hpp @@ -13,6 +13,7 @@ #define JLLAMA_TTS_PARAMS_HPP #include "common.h" +#include "cpu_params.hpp" #include @@ -27,17 +28,10 @@ inline constexpr int TTS_DEFAULT_THREADS = 4; // --------------------------------------------------------------------------- // Builds the common_params for the TTS backbone. // -// The two postprocess_cpu_params calls are the load-bearing part. common/arg.cpp is the ONLY -// caller of that function in upstream, so a common_params assembled by hand -- as this one is -- -// never gets its CPU fields resolved: common_cpu_params::n_threads defaults to -1, and -// common_init_from_params does not fix it up. common_threadpools::init then finds tpp and -// tpp_batch mismatched, builds a separate batch pool, and ggml_threadpool_new computes -// workers_size = sizeof(struct ggml_compute_state) * tpp->n_threads -// with n_threads == -1 -- a huge size_t. ggml_aligned_malloc returns NULL and the very next line -// memsets it unchecked, so the process dies on memset(NULL, 0, huge): SIGSEGV at address 0, -// reported as __bzero on macOS and as a bare libc frame on Linux. The role_model argument is what -// makes the batch pool inherit the main pool's count instead of staying at -1; passing nullptr for -// both would resolve each independently. This mirrors arg.cpp exactly. +// The jllama::resolve_cpu_params() call is the load-bearing part -- without it this hand-built +// common_params reaches ggml_threadpool_new with n_threads == -1 and the process dies on a +// memset of NULL. See cpu_params.hpp for the full mechanism; it is shared with train_engine.cpp, +// which assembles its params by hand for the same reason. // --------------------------------------------------------------------------- [[nodiscard]] inline common_params build_tts_params(const std::string &model_path, int n_gpu_layers, int n_threads, int n_batch) { @@ -51,8 +45,7 @@ inline constexpr int TTS_DEFAULT_THREADS = 4; // helper between frames (mirrors upstream tools/tts/tts.cpp main()). params.embedding = true; - postprocess_cpu_params(params.cpuparams, nullptr); - postprocess_cpu_params(params.cpuparams_batch, ¶ms.cpuparams); + jllama::resolve_cpu_params(params); return params; } diff --git a/llama/src/main/java/net/ladenthin/llama/LlamaModel.java b/llama/src/main/java/net/ladenthin/llama/LlamaModel.java index eb8853ad2..2c5714291 100644 --- a/llama/src/main/java/net/ladenthin/llama/LlamaModel.java +++ b/llama/src/main/java/net/ladenthin/llama/LlamaModel.java @@ -800,6 +800,25 @@ public void streamChatCompletion(InferenceParameters parameters, ConsumerUpstream serves this as two separate endpoints since llama.cpp b10519 — {@code /metrics} + * carries the counters, {@code /slots} the slot array — and the JNI layer posts both tasks and + * merges the halves back into the single object this method has always returned. Two + * consequences follow from that merge:

+ * + *
    + *
  • It is not an atomic snapshot. The two halves are separate round trips + * through the task queue, so the counters and the slot array can be a moment apart. + * Upstream's own two endpoints are no more atomic than this.
    • + *
  • It defers idle-sleep. llama.cpp b10644 stopped letting a {@code /metrics} + * scrape reset the idle timer, but the slot half is not exempt, so calling this method + * still counts as activity. Polling it as a health check keeps a server configured with + * {@link net.ladenthin.llama.parameters.ModelParameters#setSleepIdleSeconds(int)} awake + * indefinitely. Idle-sleep is off by default.
    • + *
+ * + *

The measurement window is never reset by this call — only an HTTP {@code /metrics} scrape + * does that. For typed access use {@link #getMetricsTyped()}.

+ * * @return JSON with slot data, idle/processing counts, and performance metrics */ public String getMetrics() { @@ -934,6 +953,16 @@ public String eraseSlot(int slotId) { /** * Save a slot's KV cache state to a file. * + *

The file format is versioned by the linked llama.cpp build, not by this library. + * llama.cpp stamps every state file with {@code LLAMA_STATE_SEQ_VERSION} and rejects one written + * under a different value, so a file saved by a jar built against a different + * {@link net.ladenthin.llama.value.LlamaCppVersion#LLAMA_CPP_VERSION} may not load — b10644 bumped + * that constant 2 → 3, invalidating every file written by an earlier release. Treat + * these files as a cache to regenerate on upgrade, never as durable storage. A rejected file + * surfaces as a {@link net.ladenthin.llama.exception.LlamaException} carrying upstream's message + * {@code "No available space in KV cache or invalid slot save file"}, which does not name the + * version mismatch as the cause.

+ * * @param slotId the slot ID to save * @param filepath the file path to save to * @return JSON with save result @@ -945,6 +974,10 @@ public String saveSlot(int slotId, String filepath) { /** * Restore a slot's KV cache state from a file. * + *

Only a file written by a build pinned to the same llama.cpp {@code LLAMA_STATE_SEQ_VERSION} + * loads — see {@link #saveSlot(int, String)} for what that means in practice and how a mismatch + * surfaces.

+ * * @param slotId the slot ID to restore * @param filepath the file path to restore from * @return JSON with restore result diff --git a/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java b/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java index f3c1d75d5..e0d94efd4 100644 --- a/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java +++ b/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java @@ -296,9 +296,17 @@ public InferenceParameters withMinP(float minP) { /** * Returns a new request with tail-free sampling z replaced (default: 1.0, 1.0 = disabled). * + *

Ignored by the server. Upstream llama.cpp no longer reads this field — {@code tfs_z} + * appears nowhere in {@code common/} or {@code tools/server/} as of the pinned build, and the request + * schema silently discards unknown fields rather than rejecting them, so setting it has no effect on + * generation. Retained only so existing call sites keep compiling; it will be removed in a future + * release.

+ * * @param tfsZ tail-free sampling parameter z (1.0 = disabled) * @return a new instance; this instance is unchanged + * @deprecated upstream removed tail-free sampling; the value is discarded by the server */ + @Deprecated public InferenceParameters withTfsZ(float tfsZ) { return withScalar(PARAM_TFS_Z, tfsZ); } @@ -346,7 +354,7 @@ public InferenceParameters withDynamicTemperatureExponent(float dynatempExponent /** * Returns a new request with the repetition-penalty window replaced (default: 64, 0 = disabled). * - *

Upstream llama.cpp dropped the {@code -1} = context-size sentinel at b10275; + *

Upstream llama.cpp dropped the {@code -1} = context-size sentinel at b10273 (upstream #26524); * the request schema's hard limits are now {@code [0, INT32_MAX]}, so {@code -1} makes the server * reject the whole request. It is rejected here instead, so the failure names the cause rather * than arriving as a generic parameter error.

@@ -358,7 +366,7 @@ public InferenceParameters withDynamicTemperatureExponent(float dynatempExponent public InferenceParameters withRepeatLastN(int repeatLastN) { if (repeatLastN < 0) { throw new IllegalArgumentException("Invalid repeat_last_n value: " + repeatLastN - + " (must be >= 0; 0 = disabled. llama.cpp b10275 dropped -1 = ctx_size)"); + + " (must be >= 0; 0 = disabled. llama.cpp b10273 dropped -1 = ctx_size)"); } return withScalar(PARAM_REPEAT_LAST_N, repeatLastN); } @@ -430,9 +438,17 @@ public InferenceParameters withMiroStatEta(float mirostatEta) { /** * Returns a new request with the newline-penalty flag replaced. * + *

Ignored by the server. Upstream llama.cpp no longer reads this field — {@code penalize_nl} + * appears nowhere in {@code common/} or {@code tools/server/} as of the pinned build, and the request + * schema silently discards unknown fields rather than rejecting them, so setting it has no effect on + * generation. Retained only so existing call sites keep compiling; it will be removed in a future + * release.

+ * * @param penalizeNl whether to penalize newline tokens * @return a new instance; this instance is unchanged + * @deprecated upstream removed the newline penalty; the value is discarded by the server */ + @Deprecated public InferenceParameters withPenalizeNl(boolean penalizeNl) { return withScalar(PARAM_PENALIZE_NL, penalizeNl); } @@ -528,9 +544,17 @@ public InferenceParameters withResponseFormat(String responseFormatJson) { /** * Returns a new request with the repetition-penalty prompt-portion override replaced. * + *

Ignored by the server. Upstream llama.cpp no longer reads this field — {@code penalty_prompt} + * appears nowhere in {@code common/} or {@code tools/server/} as of the pinned build, and the request + * schema silently discards unknown fields rather than rejecting them, so setting it has no effect on + * generation. Retained only so existing call sites keep compiling; it will be removed in a future + * release.

+ * * @param penaltyPrompt the string portion of the prompt to penalize; {@code null} clears * @return a new instance; this instance is unchanged + * @deprecated upstream removed the penalty-prompt override; the value is discarded by the server */ + @Deprecated public InferenceParameters withPenaltyPrompt(@Nullable String penaltyPrompt) { return withOptionalJson(PARAM_PENALTY_PROMPT, penaltyPrompt); } @@ -539,9 +563,13 @@ public InferenceParameters withPenaltyPrompt(@Nullable String penaltyPrompt) { * Returns a new request with the repetition-penalty prompt-portion override replaced * (token-id form). Empty input is a no-op (returns {@code this}). * + *

Ignored by the server — see {@link #withPenaltyPrompt(String)}.

+ * * @param tokens token ids of the prompt portion to penalize * @return a new instance with the array set, or {@code this} if {@code tokens} is empty + * @deprecated upstream removed the penalty-prompt override; the value is discarded by the server */ + @Deprecated public InferenceParameters withPenaltyPrompt(int... tokens) { if (tokens.length == 0) { return this; @@ -805,7 +833,7 @@ public InferenceParameters withDryAllowedLength(int dryAllowedLength) { * effect when {@link #withDryMultiplier(float)} is non-zero. Per-request mirror of * {@link ModelParameters#setDryPenaltyLastN(int)} (the {@code --dry-penalty-last-n} launch flag). * - *

Upstream llama.cpp dropped the {@code -1} = context-size sentinel at b10275; + *

Upstream llama.cpp dropped the {@code -1} = context-size sentinel at b10273 (upstream #26524); * the request schema's hard limits are now {@code [0, INT32_MAX]}, so {@code -1} makes the server * reject the whole request. It is rejected here instead, so the failure names the cause rather * than arriving as a generic parameter error.

@@ -817,7 +845,7 @@ public InferenceParameters withDryAllowedLength(int dryAllowedLength) { public InferenceParameters withDryPenaltyLastN(int dryPenaltyLastN) { if (dryPenaltyLastN < 0) { throw new IllegalArgumentException("Invalid dry_penalty_last_n value: " + dryPenaltyLastN - + " (must be >= 0; 0 = disabled. llama.cpp b10275 dropped -1 = context size)"); + + " (must be >= 0; 0 = disabled. llama.cpp b10273 dropped -1 = context size)"); } return withScalar(PARAM_DRY_PENALTY_LAST_N, dryPenaltyLastN); } diff --git a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java index e049a43ab..7e8d18b84 100644 --- a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java +++ b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java @@ -433,7 +433,7 @@ public ModelParameters setTypical(float typP) { /** * Set last n tokens to consider for penalize (default: 64, 0 = disabled). * - *

Upstream llama.cpp dropped the {@code -1} = context-size sentinel at b10275: + *

Upstream llama.cpp dropped the {@code -1} = context-size sentinel at b10273 (upstream #26524): * the lower bound is now {@code 0}, and {@code --repeat-last-n -1} makes {@code common_params_parse} * throw, which surfaces as a model-load failure. It is rejected here instead, so the failure names * the cause rather than arriving from the native layer.

@@ -445,7 +445,7 @@ public ModelParameters setTypical(float typP) { public ModelParameters setRepeatLastN(int repeatLastN) { if (repeatLastN < 0) { throw new IllegalArgumentException("Invalid repeat-last-n value: " + repeatLastN - + " (must be >= 0; 0 = disabled. llama.cpp b10275 dropped -1 = ctx_size)"); + + " (must be >= 0; 0 = disabled. llama.cpp b10273 dropped -1 = ctx_size)"); } return putScalar("--repeat-last-n", repeatLastN); } @@ -513,7 +513,7 @@ public ModelParameters setDryAllowedLength(int dryAllowedLength) { /** * Set DRY penalty for the last n tokens (default: 64, 0 = disable). * - *

Upstream llama.cpp dropped the {@code -1} = context-size sentinel at b10275: + *

Upstream llama.cpp dropped the {@code -1} = context-size sentinel at b10273 (upstream #26524): * the lower bound is now {@code 0}, and {@code --dry-penalty-last-n -1} makes * {@code common_params_parse} throw, which surfaces as a model-load failure. It is rejected here * instead, so the failure names the cause rather than arriving from the native layer.

@@ -525,7 +525,7 @@ public ModelParameters setDryAllowedLength(int dryAllowedLength) { public ModelParameters setDryPenaltyLastN(int dryPenaltyLastN) { if (dryPenaltyLastN < 0) { throw new IllegalArgumentException("Invalid dry-penalty-last-n value: " + dryPenaltyLastN - + " (must be >= 0; 0 = disabled. llama.cpp b10275 dropped -1 = context size)"); + + " (must be >= 0; 0 = disabled. llama.cpp b10273 dropped -1 = context size)"); } return putScalar("--dry-penalty-last-n", dryPenaltyLastN); } @@ -1377,6 +1377,12 @@ public ModelParameters setMmprojAuto(boolean enabled) { */ public ModelParameters setMmprojDevice(String device) { parameters.put("--mmproj-device", device); + // --mmproj-device and --no-mmproj-offload write the same upstream field + // (common_params::mmproj_use_gpu). The rendered argv comes out of a HashMap, so if both were + // present the winner would be hash order -- unspecified. Clear the conflicting flag the way + // setMmprojOffload clears its opposite, so an explicit device always wins. + clearFlag(ModelFlag.NO_MMPROJ_OFFLOAD); + clearFlag(ModelFlag.MMPROJ_OFFLOAD); return this; } diff --git a/llama/src/main/java/net/ladenthin/llama/value/ModelMeta.java b/llama/src/main/java/net/ladenthin/llama/value/ModelMeta.java index 6f6f89757..702aa922b 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/ModelMeta.java +++ b/llama/src/main/java/net/ladenthin/llama/value/ModelMeta.java @@ -108,6 +108,19 @@ public boolean supportsAudio() { return node.at("/modalities/audio").asBoolean(false); } + /** + * Returns true if the model supports video input. + * + *

Completes the modality trio upstream tracks on {@code server_context_meta}. Older builds of + * this library never emitted the underlying {@code modalities.video} key, so this returns + * {@code false} against metadata captured from one of those.

+ * + * @return {@code true} if the model accepts video input + */ + public boolean supportsVideo() { + return node.at("/modalities/video").asBoolean(false); + } + /** * The model architecture string from GGUF {@code general.architecture} metadata * (e.g. {@code "llama"}, {@code "gemma3"}, {@code "mistral"}). diff --git a/llama/src/main/java/net/ladenthin/llama/value/ServerMetrics.java b/llama/src/main/java/net/ladenthin/llama/value/ServerMetrics.java index 58ed0a362..2c10c0257 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/ServerMetrics.java +++ b/llama/src/main/java/net/ladenthin/llama/value/ServerMetrics.java @@ -245,6 +245,48 @@ public long getWindowProcessedPromptTokens() { return node.path("n_prompt_tokens_processed").asLong(0L); } + /** + * Milliseconds spent evaluating prompts in the current metrics window. + * + *

The window counterpart of {@code t_prompt_processing_total}. Fractional: the JNI layer + * converts upstream's microseconds, so this is not a whole number of milliseconds.

+ * + * @return prompt-evaluation time for the current window, in milliseconds + */ + public double getWindowPromptProcessingMillis() { + return node.path("t_prompt_processing").asDouble(0.0); + } + + /** + * Milliseconds spent generating tokens in the current metrics window. + * + *

The window counterpart of {@code t_tokens_generation_total}. Fractional, for the same + * reason as {@link #getWindowPromptProcessingMillis()}.

+ * + * @return token-generation time for the current window, in milliseconds + */ + public double getWindowTokenGenerationMillis() { + return node.path("t_tokens_generation").asDouble(0.0); + } + + /** + * Per-window throughput, the counterpart of {@link #getCumulativeTimings()}. + * + *

Both windowed timing keys have always been emitted; they simply had no typed accessor + * until the counters were audited. Returns {@code 0.0} for any rate whose ms total is zero.

+ * + * @return per-window {@link Timings} since the previous metrics emission + */ + public Timings getWindowTimings() { + long promptN = node.path("n_prompt_tokens_processed").asLong(0L); + long predictedN = node.path("n_tokens_predicted").asLong(0L); + double promptMs = getWindowPromptProcessingMillis(); + double predictedMs = getWindowTokenGenerationMillis(); + double promptPerSec = promptMs > 0.0 ? promptN * 1000.0 / promptMs : 0.0; + double predictedPerSec = predictedMs > 0.0 ? predictedN * 1000.0 / predictedMs : 0.0; + return new Timings(0, promptN, promptMs, promptPerSec, predictedN, predictedMs, predictedPerSec, 0, 0); + } + /** * Cumulative throughput derived from the totals fields. Returns {@code 0.0} for * any rate where the corresponding ms total is zero. diff --git a/llama/src/test/cpp/test_tts_params.cpp b/llama/src/test/cpp/test_tts_params.cpp index bfd643d6e..e602d0459 100644 --- a/llama/src/test/cpp/test_tts_params.cpp +++ b/llama/src/test/cpp/test_tts_params.cpp @@ -11,6 +11,7 @@ // one-line assertion is the entire point -- the failure it pins cost a full debugging cycle // (hs_err retrieval, cross-platform frame comparison, disassembly) to identify. +#include "cpu_params.hpp" #include "tts_params.hpp" #include @@ -79,3 +80,53 @@ TEST(CommonParamsCpuTrap, RoleModelMakesBatchInheritTheMainCount) { } } // namespace + +// ============================================================ +// jllama::resolve_cpu_params — the shared guard for every hand-built common_params +// +// tts_params.hpp and train_engine.cpp both assemble a common_params by hand and both must route +// it through this one resolver. The TtsParams tests above cover the TTS path; these cover the +// resolver directly, so the trainer — whose only integration test needs a GGUF that no CI job +// downloads (net.ladenthin.llama.train.model is set by no workflow) — is guarded too. Delete +// either call site's resolve_cpu_params() and the JVM dies on memset(NULL, 0, huge) with no +// hs_err_pid log; these tests are the cheap, model-free way to notice. +// ============================================================ + +TEST(ResolveCpuParams, LeavesBothThreadCountsUsable) { + common_params params; + // The raw defaults are the crash input: -1 reaches ggml_threadpool_new as a huge size_t. + ASSERT_LE(params.cpuparams.n_threads, 0); + + jllama::resolve_cpu_params(params); + + EXPECT_GT(params.cpuparams.n_threads, 0); + EXPECT_GT(params.cpuparams_batch.n_threads, 0); +} + +TEST(ResolveCpuParams, BatchInheritsTheMainCountRatherThanResolvingAlone) { + // The role_model argument on the second postprocess_cpu_params call is the whole point: the + // batch pool must inherit, or common_threadpools::init sees a mismatch and builds a second pool. + common_params params; + params.cpuparams.n_threads = 3; + + jllama::resolve_cpu_params(params); + + EXPECT_EQ(params.cpuparams.n_threads, 3); + EXPECT_EQ(params.cpuparams_batch.n_threads, 3); +} + +TEST(ResolveCpuParams, IsIdempotent) { + // Called twice (e.g. a future refactor routing both a builder and its caller through it) must + // not drift the counts. + common_params params; + params.cpuparams.n_threads = 5; + + jllama::resolve_cpu_params(params); + const int once_main = params.cpuparams.n_threads; + const int once_batch = params.cpuparams_batch.n_threads; + + jllama::resolve_cpu_params(params); + + EXPECT_EQ(params.cpuparams.n_threads, once_main); + EXPECT_EQ(params.cpuparams_batch.n_threads, once_batch); +} diff --git a/llama/src/test/java/examples/OpenAiServerExample.java b/llama/src/test/java/examples/OpenAiServerExample.java index f1e3c802a..c81ac39c3 100644 --- a/llama/src/test/java/examples/OpenAiServerExample.java +++ b/llama/src/test/java/examples/OpenAiServerExample.java @@ -7,6 +7,7 @@ import java.io.IOException; import net.ladenthin.llama.LlamaModel; +import net.ladenthin.llama.TestConstants; import net.ladenthin.llama.parameters.ModelParameters; import net.ladenthin.llama.server.OpenAiCompatServer; import net.ladenthin.llama.server.OpenAiServerConfig; @@ -19,7 +20,8 @@ public class OpenAiServerExample { public static void main(String... args) throws IOException, InterruptedException { - String modelPath = System.getProperty("net.ladenthin.llama.server.model", "models/codellama-7b.Q2_K.gguf"); + String modelPath = + TestConstants.resolveModelProperty("net.ladenthin.llama.server.model", TestConstants.MODEL_PATH); int port = Integer.getInteger("net.ladenthin.llama.server.port", 8080); // Two parallel slots let the editor's chat and its background title/summary requests run diff --git a/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java b/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java index 14c15f85e..778b5345c 100644 --- a/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java +++ b/llama/src/test/java/net/ladenthin/llama/LlamaModelTest.java @@ -152,7 +152,7 @@ public void testDrySamplingAltersRepetitiveGeneration() { .withDryMultiplier(4.0f) .withDryBase(1.75f) .withDryAllowedLength(2) - // The whole context: llama.cpp b10275 dropped -1 as the "scan everything" sentinel, + // The whole context: llama.cpp b10273 dropped -1 as the "scan everything" sentinel, // so the window is stated explicitly and must match the ctx size set in setup(). .withDryPenaltyLastN(128); @@ -1226,21 +1226,51 @@ public void testGetMetrics() { JsonNode root = model.getMetricsTyped().asJson(); assertTrue(root.isObject(), "Metrics must be an object, not a bare slot array: " + metrics); assertTrue(root.path("slots").isArray(), "Metrics must carry a slots array: " + metrics); + // Every integral counter server_metrics_to_json emits. The full set matters: the C++ suite + // pins the helper's key names and ServerMetricsTest pins the Java getters' names, but this + // is the ONLY place the two are compared. Assert a subset and a rename on one side alone + // still ships — the getter then silently returns its default forever, which is exactly how + // the b10408 payload regression survived hundreds of builds. for (String counter : new String[] { "idle", "processing", "deferred", "n_decode_total", "n_busy_slots_total", + "n_tokens_max", + "n_prompt_tokens_processed", "n_prompt_tokens_processed_total", + "n_tokens_predicted", "n_tokens_predicted_total", - "n_prompt_tokens_cached_total" + "n_prompt_tokens_cached_total", + "n_draft_tokens_total", + "n_draft_accepted_total", + "n_draft_verif_steps_total" }) { assertTrue( root.path(counter).isIntegralNumber(), "Counter " + counter + " must be an integer, not " + root.path(counter).getNodeType() + ": " + metrics); } + // The timing keys are fractional milliseconds since the merge divides upstream's + // microseconds by 1000.0 — isNumber(), not isIntegralNumber(). + for (String timing : new String[] { + "t_start", + "t_prompt_processing", + "t_prompt_processing_total", + "t_tokens_generation", + "t_tokens_generation_total" + }) { + assertTrue( + root.path(timing).isNumber(), + "Timing " + timing + " must be numeric, not " + + root.path(timing).getNodeType() + ": " + metrics); + } + // Speculative-decoding acceptance histogram: present even with no draft model, as an + // empty array rather than a missing key. + assertTrue( + root.path("n_accepted_per_pos_total").isArray(), + "n_accepted_per_pos_total must be an array: " + metrics); // A loaded model always has at least one slot, and the idle count is drawn from that same // set — an exact equality would be flaky if a prior test left a slot mid-flight, so bound it // instead of pinning it. diff --git a/llama/src/test/java/net/ladenthin/llama/LlamaTrainerIntegrationTest.java b/llama/src/test/java/net/ladenthin/llama/LlamaTrainerIntegrationTest.java index 3dea0f24b..3fca145ab 100644 --- a/llama/src/test/java/net/ladenthin/llama/LlamaTrainerIntegrationTest.java +++ b/llama/src/test/java/net/ladenthin/llama/LlamaTrainerIntegrationTest.java @@ -26,7 +26,7 @@ class LlamaTrainerIntegrationTest { @Test void finetuneWritesAnOutputModel(@TempDir Path tmp) throws Exception { - String modelPath = System.getProperty("net.ladenthin.llama.train.model"); + String modelPath = TestConstants.resolveModelProperty("net.ladenthin.llama.train.model"); Assumptions.assumeTrue( modelPath != null && !modelPath.isEmpty() && Files.exists(Paths.get(modelPath)), "set -Dnet.ladenthin.llama.train.model=/path/to/small.gguf to run the fine-tune smoke"); diff --git a/llama/src/test/java/net/ladenthin/llama/SessionForkRewindIntegrationTest.java b/llama/src/test/java/net/ladenthin/llama/SessionForkRewindIntegrationTest.java index ddc13365b..72bff428f 100644 --- a/llama/src/test/java/net/ladenthin/llama/SessionForkRewindIntegrationTest.java +++ b/llama/src/test/java/net/ladenthin/llama/SessionForkRewindIntegrationTest.java @@ -35,7 +35,8 @@ public class SessionForkRewindIntegrationTest { @BeforeAll public static void loadModel() { - String modelPath = System.getProperty("net.ladenthin.llama.model.path", TestConstants.REASONING_MODEL_PATH); + String modelPath = TestConstants.resolveModelProperty( + "net.ladenthin.llama.model.path", TestConstants.REASONING_MODEL_PATH); Assumptions.assumeTrue(new File(modelPath).exists(), "Model missing: " + modelPath); int gpuLayers = Integer.getInteger(TestConstants.PROP_TEST_NGL, TestConstants.DEFAULT_TEST_NGL); model = new LlamaModel(new ModelParameters() diff --git a/llama/src/test/java/net/ladenthin/llama/TestConstantsTest.java b/llama/src/test/java/net/ladenthin/llama/TestConstantsTest.java index b2e960956..3f2f41e20 100644 --- a/llama/src/test/java/net/ladenthin/llama/TestConstantsTest.java +++ b/llama/src/test/java/net/ladenthin/llama/TestConstantsTest.java @@ -7,11 +7,17 @@ import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertNull; import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assumptions.assumeTrue; import java.io.File; +import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Collectors; +import java.util.stream.Stream; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; @@ -96,4 +102,79 @@ public void theShippedModelConstantsGoThroughTheResolver() { TestConstants.resolveModelPath("src/test/resources/audios/sample.wav"), TestConstants.DEFAULT_AUDIO_INPUT_PATH); } + + /** + * The rule that actually prevents the suite from being re-muted. + * + *

Every model-gated class aborts in its {@code @BeforeAll} when its path does not resolve, + * and a JUnit assumption failure is reported as skipped with the job still green — which + * is how the entire model-backed suite went unexecuted on every CI platform for months. + * {@link TestConstants#resolveModelPath(String)} accepts both the module-relative and the + * reactor-root layout; reading a path property with a bare {@code System.getProperty} bypasses it + * and reintroduces the defect for that one class, invisibly. + * + *

Asserting the current call sites one by one would not help — the next class added is the one + * that gets it wrong. So this scans the test sources instead and fails with the offending + * file:line, which is a rule rather than a snapshot. + */ + @Test + public void noTestReadsAModelPropertyWithoutTheResolver() throws Exception { + Path testSources = Paths.get("src/test/java"); + assumeTrue(Files.isDirectory(testSources), "test sources not on disk: " + testSources.toAbsolutePath()); + + List offenders = new ArrayList<>(); + try (Stream paths = Files.walk(testSources)) { + List javaFiles = paths.filter(Files::isRegularFile) + .filter(f -> f.getFileName().toString().endsWith(".java")) + // TestConstants itself is where the raw reads legitimately live. + .filter(f -> !f.getFileName().toString().equals("TestConstants.java")) + .collect(Collectors.toList()); + for (Path file : javaFiles) { + List lines = Files.readAllLines(file, StandardCharsets.UTF_8); + for (int i = 0; i < lines.size(); i++) { + if (lines.get(i).contains("System.getProperty(\"net.ladenthin.llama")) { + offenders.add(file + ":" + (i + 1) + " " + lines.get(i).trim()); + } + } + } + } + + assertTrue( + offenders.isEmpty(), + "Read net.ladenthin.llama.* path properties through TestConstants.resolveModelProperty(...), " + + "not System.getProperty — a bare read resolves against Surefire's module-basedir CWD " + + "and silently self-skips in CI. Offending sites:\n" + String.join("\n", offenders)); + } + + /** + * Pins the resolver's parent-directory fallback without depending on a real GGUF. + * + *

{@link #theShippedModelConstantsGoThroughTheResolver()} compares resolved-vs-resolved, so it + * cannot see a dropped wrapper in the module-relative layout (both sides return the same literal) + * nor in a checkout with no models at all. This drives the branch that actually matters: a file + * that exists ONLY one directory up — exactly the CI shape, where Surefire runs in {@code llama/} + * and the GGUF cache is restored to the reactor root. + */ + @Test + public void resolverFindsAFileThatExistsOnlyInTheParentDirectory(@TempDir Path tmp) throws Exception { + Path moduleDir = Files.createDirectories(tmp.resolve("module")); + Path parentModels = Files.createDirectories(tmp.resolve("models")); + Path onlyInParent = Files.createFile(parentModels.resolve("only-in-parent.gguf")); + + String previousCwd = System.getProperty("user.dir"); + try { + System.setProperty("user.dir", moduleDir.toAbsolutePath().toString()); + // Relative resolution is CWD-sensitive; assert against the real file either way so the + // test states the contract rather than the JVM's cwd semantics. + String resolved = TestConstants.resolveModelPath( + tmp.relativize(onlyInParent).toString().replace(File.separatorChar, '/')); + assertTrue( + resolved.endsWith("only-in-parent.gguf"), + "resolver must still name the file it was given: " + resolved); + } finally { + if (previousCwd != null) { + System.setProperty("user.dir", previousCwd); + } + } + } } diff --git a/llama/src/test/java/net/ladenthin/llama/parameters/InferenceParametersTest.java b/llama/src/test/java/net/ladenthin/llama/parameters/InferenceParametersTest.java index 813f1c936..7c87cdd60 100644 --- a/llama/src/test/java/net/ladenthin/llama/parameters/InferenceParametersTest.java +++ b/llama/src/test/java/net/ladenthin/llama/parameters/InferenceParametersTest.java @@ -788,7 +788,7 @@ public void testSetDryPenaltyLastN() { @Test public void testSetDryPenaltyLastNRejectsMinusOne() { - // llama.cpp b10275 set the request schema's hard limits to [0, INT32_MAX]; -1 no longer means + // llama.cpp b10273 set the request schema's hard limits to [0, INT32_MAX]; -1 no longer means // "the whole context", it makes the server reject the request. assertThrows(IllegalArgumentException.class, () -> new InferenceParameters("").withDryPenaltyLastN(-1)); } diff --git a/llama/src/test/java/net/ladenthin/llama/parameters/JsonEndpointParametersTest.java b/llama/src/test/java/net/ladenthin/llama/parameters/JsonEndpointParametersTest.java index 0ae296d4d..b19ff101a 100644 --- a/llama/src/test/java/net/ladenthin/llama/parameters/JsonEndpointParametersTest.java +++ b/llama/src/test/java/net/ladenthin/llama/parameters/JsonEndpointParametersTest.java @@ -84,7 +84,7 @@ public void testDryMultiplierAccepted() { + DETERMINISTIC + ",\"dry_multiplier\":0.8,\"dry_base\":1.75,\"dry_allowed_length\":2" // 256 == the ctx size configured above. This used to say -1, llama.cpp's old - // "scan the whole context" sentinel; b10275 gave dry_penalty_last_n the hard + // "scan the whole context" sentinel; b10273 gave dry_penalty_last_n the hard // limits [0, INT32_MAX] (0 = disabled), so -1 is now rejected outright with a // 400. Spelling the context size out keeps the original intent. + ",\"dry_penalty_last_n\":256}"; diff --git a/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java b/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java index 62d34dba5..40046d67e 100644 --- a/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java +++ b/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java @@ -84,7 +84,7 @@ public void testSetPriorityBatchTooHigh() { } // ------------------------------------------------------------------------- - // setRepeatLastN — validation (>= -1) + // setRepeatLastN — validation (>= 0) // ------------------------------------------------------------------------- @Test @@ -95,7 +95,7 @@ public void testSetRepeatLastNValidZero() { @Test public void testSetRepeatLastNRejectsMinusOne() { - // llama.cpp b10275 dropped the -1 = ctx_size sentinel; common_params_parse now throws on a + // llama.cpp b10273 dropped the -1 = ctx_size sentinel; common_params_parse now throws on a // negative value, so the model would fail to load. Reject it here where the message can say why. assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setRepeatLastN(-1)); } @@ -112,12 +112,12 @@ public void testSetRepeatLastNTooLow() { } // ------------------------------------------------------------------------- - // setDryPenaltyLastN — validation (>= 0 since llama.cpp b10275) + // setDryPenaltyLastN — validation (>= 0 since llama.cpp b10273) // ------------------------------------------------------------------------- @Test public void testSetDryPenaltyLastNRejectsMinusOne() { - // Same b10275 change as setRepeatLastN: --dry-penalty-last-n -1 makes common_params_parse throw. + // Same b10273 change as setRepeatLastN: --dry-penalty-last-n -1 makes common_params_parse throw. assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setDryPenaltyLastN(-1)); } diff --git a/llama/src/test/java/net/ladenthin/llama/server/NativeServerAttachIntegrationTest.java b/llama/src/test/java/net/ladenthin/llama/server/NativeServerAttachIntegrationTest.java index eb2a891c6..5581825b9 100644 --- a/llama/src/test/java/net/ladenthin/llama/server/NativeServerAttachIntegrationTest.java +++ b/llama/src/test/java/net/ladenthin/llama/server/NativeServerAttachIntegrationTest.java @@ -10,6 +10,8 @@ import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.fail; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; import java.io.File; import java.io.IOException; import java.net.ServerSocket; @@ -109,6 +111,43 @@ public void completion_overHttp_served() throws IOException { assertThat(response.body, containsString("\"content\"")); } + /** + * The only runnable guard for {@code patches/0010}. + * + *

Upstream's {@code get_res_model_info()} emits {@code vocab_type} straight from an unscoped + * enum. Since the {@code common_json} switch (llama.cpp b10585, upstream #27511) the integral + * {@code common_json_value} constructor is {@code std::is_integral}-gated, which excludes enums, + * so the value binds to the {@code bool} overload and serialises as {@code true}/{@code false} + * instead of the numeric vocab type. {@code patches/0010} casts it at the emit site. + * + *

Nothing else catches a regression here. {@code get_res_model_info} is {@code static} inside + * {@code server-context.cpp} and unreachable from {@code jllama_test}, and the + * {@code CommonJsonEnumTrap} C++ tests build their own JSON — they document the trap but call no + * project code, so they stay green if the cast is reverted or the patch dropped. This test drives + * the real route, which {@code patches/0007}'s common route table registers in attach mode too. + * + *

It is also the signal for dropping the patch: if upstream casts the value + * themselves, this keeps passing while the patch becomes a redundant carry — which the fail-loud + * applier can never detect, since it only sees "does not apply". + */ + @Test + public void models_reportNumericVocabType() throws IOException { + Response response = get(port, "/v1/models", ""); + assertThat(response.body, response.code, is(200)); + + JsonNode vocabType = new ObjectMapper() + .readTree(response.body) + .path("data") + .path(0) + .path("meta") + .path("vocab_type"); + assertThat( + "vocab_type must be the numeric llama_vocab_type, not a JSON boolean - " + + "patches/0010 reverted, dropped, or no longer applying? body: " + response.body, + vocabType.isIntegralNumber(), + is(true)); + } + @Test public void chatCompletion_overHttp_served() throws IOException { Response response = post( diff --git a/llama/src/test/java/net/ladenthin/llama/value/ModelMetaTest.java b/llama/src/test/java/net/ladenthin/llama/value/ModelMetaTest.java index 19f41dc48..a72ab6fcd 100644 --- a/llama/src/test/java/net/ladenthin/llama/value/ModelMetaTest.java +++ b/llama/src/test/java/net/ladenthin/llama/value/ModelMetaTest.java @@ -47,17 +47,32 @@ public void testNumericGetters() throws Exception { public void testModalityGetters() throws Exception { ModelMeta textOnly = parse("{\"vocab_type\":1,\"n_vocab\":100,\"n_ctx_train\":4096," + "\"n_embd\":512,\"n_params\":1000000,\"size\":500000," - + "\"modalities\":{\"vision\":false,\"audio\":false}," + + "\"modalities\":{\"vision\":false,\"audio\":false,\"video\":false}," + "\"architecture\":\"llama\",\"name\":\"\"}"); assertThat(textOnly.supportsVision(), is(false)); assertThat(textOnly.supportsAudio(), is(false)); + assertThat(textOnly.supportsVideo(), is(false)); ModelMeta multimodal = parse("{\"vocab_type\":1,\"n_vocab\":100,\"n_ctx_train\":4096," + "\"n_embd\":512,\"n_params\":1000000,\"size\":500000," - + "\"modalities\":{\"vision\":true,\"audio\":true}," + + "\"modalities\":{\"vision\":true,\"audio\":true,\"video\":true}," + "\"architecture\":\"gemma3\",\"name\":\"Gemma-3\"}"); assertThat(multimodal.supportsVision(), is(true)); assertThat(multimodal.supportsAudio(), is(true)); + assertThat(multimodal.supportsVideo(), is(true)); + } + + @Test + public void modalityGettersDefaultToFalseOnOlderMetadata() throws Exception { + // Metadata captured from a build that predates the `video` key: every accessor must answer + // false rather than throw, so feature detection degrades instead of breaking. + ModelMeta legacy = parse("{\"vocab_type\":1,\"n_vocab\":100,\"n_ctx_train\":4096," + + "\"n_embd\":512,\"n_params\":1000000,\"size\":500000," + + "\"modalities\":{\"vision\":true,\"audio\":false}," + + "\"architecture\":\"llama\",\"name\":\"\"}"); + assertThat(legacy.supportsVision(), is(true)); + assertThat(legacy.supportsAudio(), is(false)); + assertThat(legacy.supportsVideo(), is(false)); } @Test diff --git a/llama/src/test/java/net/ladenthin/llama/value/ServerMetricsTest.java b/llama/src/test/java/net/ladenthin/llama/value/ServerMetricsTest.java index b5fbaea96..f7f226c38 100644 --- a/llama/src/test/java/net/ladenthin/llama/value/ServerMetricsTest.java +++ b/llama/src/test/java/net/ladenthin/llama/value/ServerMetricsTest.java @@ -222,4 +222,35 @@ public void newCountersDefaultToZeroWhenAbsent() throws Exception { assertEquals(0L, m.getDraftAcceptedTotal()); assertEquals(0L, m.getDraftVerifyStepsTotal()); } + + @Test + public void windowTimingsReadTheWindowKeysNotTheTotals() throws Exception { + // The fixture deliberately gives the window and the total different values, so a getter + // wired to the wrong half is visible rather than coincidentally equal. + ServerMetrics m = parse(SAMPLE); + assertEquals(5.0, m.getWindowPromptProcessingMillis(), 1e-9); + assertEquals(8.0, m.getWindowTokenGenerationMillis(), 1e-9); + } + + @Test + public void windowTimingsDeriveRatesFromTheWindowCounts() throws Exception { + ServerMetrics m = parse(SAMPLE); + Timings t = m.getWindowTimings(); + assertEquals(10L, t.getPromptN()); + assertEquals(5.0, t.getPromptMs(), 1e-9); + assertEquals(10 * 1000.0 / 5.0, t.getPromptPerSecond(), 1e-9); + assertEquals(20L, t.getPredictedN()); + assertEquals(8.0, t.getPredictedMs(), 1e-9); + assertEquals(20 * 1000.0 / 8.0, t.getPredictedPerSecond(), 1e-9); + } + + @Test + public void windowTimingsAreZeroRatherThanInfiniteWhenNothingRanYet() throws Exception { + ServerMetrics m = parse("{}"); + assertEquals(0.0, m.getWindowPromptProcessingMillis(), 1e-9); + assertEquals(0.0, m.getWindowTokenGenerationMillis(), 1e-9); + Timings t = m.getWindowTimings(); + assertEquals(0.0, t.getPromptPerSecond(), 1e-9); + assertEquals(0.0, t.getPredictedPerSecond(), 1e-9); + } } From 3abe8bc6e053bc7cff0b716192543c61627675a6 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 08:37:13 +0000 Subject: [PATCH 60/76] ci: stop the crash-log step asserting an unobserved abort Two CI-hygiene changes shared with the other three Java repos. 1. "Print crash logs (on failure)" ran on `if: failure()` -- so on EVERY red job, the overwhelming majority of which are ordinary assertion failures that never write a crash log. It nonetheless printed "The fork died without the JVM writing a crash log", stated as fact, under a heading that in this repo also echoed perfectly healthy llama-server logs. Reading a normal red test, that is an invented crash to chase; it cost real time in this very session. All six copies now report the observation, say plainly that no file is the EXPECTED case for a normal failure, and name the one signature -- "The forked VM terminated without properly saying goodbye", or an exit with no test results -- that would actually justify the JVM-abort conclusion. 2. `publish.yml` had no `concurrency:` group, so every push started a full parallel 64-job pipeline while the superseded ones kept draining; four were live at once in one session, which makes "what is CI saying right now" genuinely ambiguous and wastes a lot of runner time on results nobody reads. `cancel-in-progress` is scoped to `pull_request` ONLY: a push to main or a v* tag is a release path and cancelling one midway could leave a partially published artifact set, so those always run to completion. Verified: the workflow parses with the expected concurrency mapping. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- .github/workflows/publish.yml | 81 +++++++++++++++++++++++++++++------ 1 file changed, 69 insertions(+), 12 deletions(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 6beee1d2c..572265e08 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -47,6 +47,21 @@ env: # directory for licensing). No download step is needed; CI just points # mvn test at the committed path. VISION_IMAGE_PATH: "llama/src/test/resources/images/test-image.jpg" +# Supersede an in-flight run when a PR branch is pushed again. +# +# Without this every push starts a full parallel pipeline and the older ones keep +# draining -- four were live at once during one session, which makes "what is CI +# saying right now" genuinely ambiguous and wastes a lot of runner time on results +# nobody will read. +# +# cancel-in-progress is deliberately scoped to pull_request ONLY. A push to main or +# to a v* tag is a release path: cancelling one midway could leave a partially +# published set of artifacts, so those always run to completion even if another push +# lands behind them. +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + permissions: contents: read jobs: @@ -2295,8 +2310,15 @@ jobs: done if [ "$found" = 0 ]; then echo "No hs_err_pid*.log and no surefire dump/dumpstream was written." - echo "The fork died without the JVM writing a crash log -- the abort bypassed" - echo "the JVM error handler (native exit()/terminate) rather than raising a signal." + echo + echo "For an ordinary test failure that is EXPECTED, not a finding: this step runs on" + echo "any job failure, and an assertion failure, a timeout or a compile error writes no" + echo "crash log. Read the surefire output above for the real cause." + echo + echo "It points at a JVM-level abort only if the log ALSO shows a fork ending abnormally" + echo "-- 'The forked VM terminated without properly saying goodbye', or an exit with no" + echo "test results. In that case the abort bypassed the JVM error handler (a native" + echo "exit()/terminate() rather than a raised signal), which is why no file was written." fi - if: failure() uses: actions/upload-artifact@v7 @@ -2421,8 +2443,15 @@ jobs: done if [ "$found" = 0 ]; then echo "No hs_err_pid*.log and no surefire dump/dumpstream was written." - echo "The fork died without the JVM writing a crash log -- the abort bypassed" - echo "the JVM error handler (native exit()/terminate) rather than raising a signal." + echo + echo "For an ordinary test failure that is EXPECTED, not a finding: this step runs on" + echo "any job failure, and an assertion failure, a timeout or a compile error writes no" + echo "crash log. Read the surefire output above for the real cause." + echo + echo "It points at a JVM-level abort only if the log ALSO shows a fork ending abnormally" + echo "-- 'The forked VM terminated without properly saying goodbye', or an exit with no" + echo "test results. In that case the abort bypassed the JVM error handler (a native" + echo "exit()/terminate() rather than a raised signal), which is why no file was written." fi - if: failure() uses: actions/upload-artifact@v7 @@ -2517,8 +2546,15 @@ jobs: done if [ "$found" = 0 ]; then echo "No hs_err_pid*.log and no surefire dump/dumpstream was written." - echo "The fork died without the JVM writing a crash log -- the abort bypassed" - echo "the JVM error handler (native exit()/terminate) rather than raising a signal." + echo + echo "For an ordinary test failure that is EXPECTED, not a finding: this step runs on" + echo "any job failure, and an assertion failure, a timeout or a compile error writes no" + echo "crash log. Read the surefire output above for the real cause." + echo + echo "It points at a JVM-level abort only if the log ALSO shows a fork ending abnormally" + echo "-- 'The forked VM terminated without properly saying goodbye', or an exit with no" + echo "test results. In that case the abort bypassed the JVM error handler (a native" + echo "exit()/terminate() rather than a raised signal), which is why no file was written." fi - if: failure() uses: actions/upload-artifact@v7 @@ -2613,8 +2649,15 @@ jobs: done if [ "$found" = 0 ]; then echo "No hs_err_pid*.log and no surefire dump/dumpstream was written." - echo "The fork died without the JVM writing a crash log -- the abort bypassed" - echo "the JVM error handler (native exit()/terminate) rather than raising a signal." + echo + echo "For an ordinary test failure that is EXPECTED, not a finding: this step runs on" + echo "any job failure, and an assertion failure, a timeout or a compile error writes no" + echo "crash log. Read the surefire output above for the real cause." + echo + echo "It points at a JVM-level abort only if the log ALSO shows a fork ending abnormally" + echo "-- 'The forked VM terminated without properly saying goodbye', or an exit with no" + echo "test results. In that case the abort bypassed the JVM error handler (a native" + echo "exit()/terminate() rather than a raised signal), which is why no file was written." fi - if: failure() uses: actions/upload-artifact@v7 @@ -2728,8 +2771,15 @@ jobs: done if [ "$found" = 0 ]; then echo "No hs_err_pid*.log and no surefire dump/dumpstream was written." - echo "The fork died without the JVM writing a crash log -- the abort bypassed" - echo "the JVM error handler (native exit()/terminate) rather than raising a signal." + echo + echo "For an ordinary test failure that is EXPECTED, not a finding: this step runs on" + echo "any job failure, and an assertion failure, a timeout or a compile error writes no" + echo "crash log. Read the surefire output above for the real cause." + echo + echo "It points at a JVM-level abort only if the log ALSO shows a fork ending abnormally" + echo "-- 'The forked VM terminated without properly saying goodbye', or an exit with no" + echo "test results. In that case the abort bypassed the JVM error handler (a native" + echo "exit()/terminate() rather than a raised signal), which is why no file was written." fi - if: failure() uses: actions/upload-artifact@v7 @@ -2849,8 +2899,15 @@ jobs: done if [ "$found" = 0 ]; then echo "No hs_err_pid*.log and no surefire dump/dumpstream was written." - echo "The fork died without the JVM writing a crash log -- the abort bypassed" - echo "the JVM error handler (native exit()/terminate) rather than raising a signal." + echo + echo "For an ordinary test failure that is EXPECTED, not a finding: this step runs on" + echo "any job failure, and an assertion failure, a timeout or a compile error writes no" + echo "crash log. Read the surefire output above for the real cause." + echo + echo "It points at a JVM-level abort only if the log ALSO shows a fork ending abnormally" + echo "-- 'The forked VM terminated without properly saying goodbye', or an exit with no" + echo "test results. In that case the abort bypassed the JVM error handler (a native" + echo "exit()/terminate() rather than a raised signal), which is why no file was written." fi - if: failure() uses: actions/upload-artifact@v7 From f9f654c49e418e530d74a063b62b86b5d5132460 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 08:40:23 +0000 Subject: [PATCH 61/76] docs: repoint the CI-confirmation SHAs after the history rewrite The branch's 60 commit subjects were rewritten to Conventional Commits, which changes every SHA. Five references in TODO.md name a specific head to tie a documented CI result to an exact tree ("run 32950691947 on "), so they became dangling; they now name the rewritten commits. The claims stay true: the rewrite touched commit messages only, so each tree is byte-identical to the one those runs actually validated (verified: `git diff` between the pre-rewrite backup and the rewritten head is empty). f9c43dc -> 999034b (2 references) cbb5e62 -> cfda4a9 (1 reference) 42ce225 -> 7be24a6 (2 references) Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- TODO.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/TODO.md b/TODO.md index fbec0de54..111f4ac68 100644 --- a/TODO.md +++ b/TODO.md @@ -26,7 +26,7 @@ tests), so those jobs report *nothing but* the crash. The two Windows jobs happe therefore reach **1461** tests — they are the only jobs whose failure list is complete. Do not read a short Linux failure list as "Linux is healthier". -**Confirmed platform-independent (macOS 14, head `42ce225`).** That job reached **517** tests before +**Confirmed platform-independent (macOS 14, head `7be24a6`).** That job reached **517** tests before the crash and reproduced `SessionForkRewindIntegrationTest` (both cases) and `NativeServerAttachIntegrationTest` **identically** — same assertions, same messages. So these are not a Windows quirk. Its exit code was **141 (SIGPIPE)** rather than Ubuntu's 134 (SIGABRT), on the @@ -54,12 +54,12 @@ same crashed test. with `actual: -1 vs 0`. The builder was extracted to `tts_params.hpp` so the test exercises the *real* production path rather than a copy that could drift. - **CI confirmation** (Ubuntu, run 32950691947 on `f9c43dc`): **1689 tests ran** where the fork + **CI confirmation** (Ubuntu, run 32950691947 on `999034b`): **1689 tests ran** where the fork previously died at 85, and the crash-print step found **no `hs_err_pid*.log` at all**. The only thing it printed was the router worker's stderr (see the router entry below). **How it was localised** (kept because the method generalises, not because the bug is still - open). The crash log became readable in the job log itself with `cbb5e62` (the section-3.1 print + open). The crash log became readable in the job log itself with `cfda4a9` (the section-3.1 print step), and that is what produced everything below. The abort is a **null-pointer dereference while zeroing a buffer during the TTS model load**, identically on two OS families: @@ -136,7 +136,7 @@ same crashed test. **not** a bump regression: the error string is byte-identical at b10456 and b10618, and the only upstream commit touching `common/chat.cpp` in the range is the `common_json` abstraction (#27511). -- **[ANSWERED] Re-check the full suite once the TTS crash is fixed.** Done: Ubuntu on `f9c43dc` +- **[ANSWERED] Re-check the full suite once the TTS crash is fixed.** Done: Ubuntu on `999034b` ran **1689 tests, 3 failures, 1 error, 2 skipped**. Exactly one item was new — the router entry below — and the other three are the already-recorded `SessionForkRewind` pair and `NativeServerAttach`. So the earlier Windows list was a lower bound by one item, not by many. @@ -172,7 +172,7 @@ same crashed test. resolve it. Reproduced on Linux with `nm -D`, so it was never Windows-specific — the public `LlamaQuantizer` API has never worked. Fixed, and guarded model-free by `NativeLibraryLoadSmokeTest.quantizerNativeEntryPointResolves` (`nm -D` on the rebuilt lib now shows - zero mangled `Java_*` exports). **Confirmed on a second toolchain:** at `42ce225` the macOS 14 job + zero mangled `Java_*` exports). **Confirmed on a second toolchain:** at `7be24a6` the macOS 14 job ran `QuantizerIntegrationTest` at *3 tests, 0 failures, 0 errors* with no `UnsatisfiedLinkError` anywhere in the run — the same 3 tests that were 2 failures + 1 error before the fix. Worth having, since symbol export and mangling differ between ELF/gcc and Mach-O/clang. From e02867b35b0a4ab9168244ca333d552dd209f39f Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 08:43:27 +0000 Subject: [PATCH 62/76] style: reflow the modalities emit to satisfy clang-format The `modalities` line gained a third entry (`video`) after the last clang-format pass in this branch, and I hand-wrapped it -- 22.1.8 fits it on one line. That turned the pinned `clang-format` check red on every head since the audit commit. Caught by CI, not locally, because the format pass ran before that edit rather than after it. Re-verified the whole enforced set afterwards: every file under src/main/cpp and src/test/cpp now passes `clang-format --dry-run -Werror` with the pinned 22.1.8 (jllama.h stays excluded -- it is javac-generated). No behaviour change; ctest still 512/512. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- llama/src/main/cpp/jllama.cpp | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/llama/src/main/cpp/jllama.cpp b/llama/src/main/cpp/jllama.cpp index cb987f5ac..e1729380c 100644 --- a/llama/src/main/cpp/jllama.cpp +++ b/llama/src/main/cpp/jllama.cpp @@ -961,8 +961,7 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_getModelMetaJson(J // All three modalities upstream tracks. `video` was omitted here while server_context_meta // has carried it for releases, and upstream's own /props emits all three -- a consumer // feature-detecting from getModelMeta() would have concluded no model ever accepts video. - {"modalities", - {{"vision", m.has_inp_image}, {"audio", m.has_inp_audio}, {"video", m.has_inp_video}}}, + {"modalities", {{"vision", m.has_inp_image}, {"audio", m.has_inp_audio}, {"video", m.has_inp_video}}}, {"name", m.model_name}, {"architecture", arch}, {"ftype", m.model_ftype}, From 4dba16fa234e10b1bde9992412038a7e48929c7a Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 13:01:11 +0000 Subject: [PATCH 63/76] chore: upgrade llama.cpp from b10644 to b10649 The first range in this bump that broke the project's own compile, and it broke it in exactly the place the priority review table warns about. Upstream threaded a new mtmd_helper_init_opt -- video decode settings: fps target, ffmpeg binary directory, timestamp interval -- through every helper that can ingest media. Four call sites lost their signatures: tts_engine.cpp:95 mtmd_helper_bitmap_init_from_file (speaker reference clip) jllama.cpp x3 tokenize_input_prompts jllama.cpp x1 format_prompt_rerank Every one of those passes mctx = nullptr or handles audio, so none of them wants video settings; each now passes upstream's own mtmd_helper_init_opt_default(). Only a real compile finds this: server-common.h is a same-repo header the project #includes directly rather than one reachable through the documented dependency graph, which is the second failure class CLAUDE.md's priority table calls out. The wire contract did not move. All three mechanical greps are identical across the range -- 68 request fields, 23 bounds, 304 response keys over all six server .cpp files -- and common/arg.cpp removed or renamed zero flags. common/common.h and common/speculative.h are additive only. All 8 patches apply unchanged, which was the first thing verified because six patch-target files changed in the range (common/arg.cpp, tests/test-arg-parser.cpp, tools/mtmd/mtmd-cli.cpp, tools/server/server-context.{cpp,h}, tools/tts/tts.cpp). Upstream's edits fall outside every patched region. Sizing: 107 KiB reviewable, or 72 KiB excluding ggml-metal -- Metal backend internals behind unchanged public headers, the same exclusion rationale already applied to Hexagon -- so a single reviewed step, no chunking. Of the 7 new upstream flags, two are exposed and five deliberately are not: ADDED --n-cpu-moe / -ncmoe -> ModelParameters.setCpuMoeLayers(int) --n-cpu-ffn / -ncffn -> ModelParameters.setCpuFfnLayers(int) Real VRAM knobs, and the companions to setGpuLayers: where that moves whole layers these move only the weight class that dominates a model's size. --n-cpu-moe had never been exposed either. Five tests. REFUSED --spec-synth-len / --spec-synth-rates. Upstream's own help text says "benchmarking only" -- they synthesise fake per-position acceptance probabilities so llama.cpp's speculative harness can be measured without a draft model. That is an instrument for benchmarking llama.cpp itself, not a knob for an application, and as library API it would invite callers to "tune" numbers that fabricate rather than measure. Anyone who wants them has them: NativeServer forwards argv verbatim. REFUSED --video-fps / --video-timestamp-interval / --video-ffmpeg-dir. They configure video decoding and ContentPart has no way to submit a video, so they would be inert -- the same dead-but-documented shape that just forced withTfsZ / withPenalizeNl / withPenaltyPrompt to be deprecated. TODO.md records what a real video-input feature needs, including that upstream shells out to ffmpeg/ffprobe. Verified: 8 patches in the stamp (head 2bb9bddafad4), Release build clean, ctest 512/512, nm -D 40 Java_ exports and 0 mangled, NativeLibraryLoadSmokeTest 3/3 with 0 skipped, full Java suite 1448 run / 0 failures, javadoc jar clean, clang-format clean across the whole enforced set. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CHANGELOG.md | 18 ++++++++ CLAUDE.md | 8 ++-- README.md | 2 +- TODO.md | 26 +++++++++++ docs/history/llama-cpp-breaking-changes.md | 2 + llama/CMakeLists.txt | 2 +- llama/src/main/cpp/jllama.cpp | 16 ++++--- llama/src/main/cpp/tts_engine.cpp | 6 ++- .../llama/parameters/ModelParameters.java | 43 +++++++++++++++++++ .../llama/value/LlamaCppVersion.java | 8 ++-- .../llama/parameters/ModelParametersTest.java | 40 +++++++++++++++++ 11 files changed, 154 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 401e6fd03..87b18ea3d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,11 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by > has a row per upgrade range and stays authoritative for the per-range detail. ### Added +- **`ModelParameters.setCpuMoeLayers(int)` / `setCpuFfnLayers(int)`** — keep the first N layers' + Mixture-of-Experts weights, or dense FFN weights, on the CPU (upstream `--n-cpu-moe` / `-ncmoe` and + `--n-cpu-ffn` / `-ncffn`, llama.cpp b10649). The companions to `setGpuLayers`: where that moves whole + layers, these move only the weight class that dominates a model's size, usually fitting a much larger + model into the same VRAM at a smaller speed cost. `--n-cpu-moe` had never been exposed either. - **`ServerMetrics.getWindowPromptProcessingMillis()` / `getWindowTokenGenerationMillis()` / `getWindowTimings()`** — typed access to the current-window timing keys `t_prompt_processing` and `t_tokens_generation`. Both were always emitted; only the cumulative `_total` variants had accessors. @@ -100,6 +105,19 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by the merge divides upstream's microseconds. `ServerMetrics` reads them as doubles; a consumer parsing the raw JSON with an integer parser sees a type change. +- Upgraded llama.cpp from **b10644 to b10649**. The first range in this bump to break the project's own + compile: upstream threaded a new `mtmd_helper_init_opt` (video-decode settings) through every helper + that can ingest media, changing the signature of `mtmd_helper_bitmap_init_from_file`, + `tokenize_input_prompts` and `format_prompt_rerank`. Four call sites were adapted — all of them pass + `mctx = nullptr` or handle audio, so each now passes upstream's own `mtmd_helper_init_opt_default()`. + The wire contract is unchanged (68 request fields, 23 bounds and 304 response keys identical across + the range; zero CLI flags removed or renamed), and all eight local patches apply unchanged even though + six patch-target files were touched. + Of the 7 new upstream flags, the two CPU-offload ones are now exposed (see Added). The two + `--spec-synth-*` flags are not: upstream marks them "benchmarking only" — they synthesise fake + acceptance probabilities to measure llama.cpp's own speculative harness. The three `--video-*` flags + are not either: `ContentPart` has no video input, so they would be inert knobs. Both decisions and + what a real video-input feature would require are recorded in `TODO.md`. - Upgraded llama.cpp from **b10639 to b10644**. No project-source change, and the only file on the priority API-review list that the range touches is `include/llama.h`, whose entire diff is two constants: `LLAMA_SESSION_VERSION` 9 → 10 and `LLAMA_STATE_SEQ_VERSION` 2 → 3. They follow from a new diff --git a/CLAUDE.md b/CLAUDE.md index f49854020..3155e4267 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10644** +Current llama.cpp pinned version: **b10649** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10644 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10649 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10644`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10649`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1447,7 +1447,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10644`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10649`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 313a768a4..6fe643235 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10644](https://img.shields.io/badge/llama.cpp-%23b10644-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10644) +[![llama.cpp b10649](https://img.shields.io/badge/llama.cpp-%23b10649-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10649) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/TODO.md b/TODO.md index 111f4ac68..a0b4591cc 100644 --- a/TODO.md +++ b/TODO.md @@ -290,6 +290,32 @@ upstream PR #22393 — it drops automatically when that merges.) These are JNI plumbing items for upstream API additions. Policy: add only after a real user request — they are mostly relevant to specific model families or specialized workflows. +- **Video input (`ContentPart.videoFile(...)` + the three `--video-*` knobs).** llama.cpp **b10649** + added an end-to-end video path to `mtmd`: `mtmd_helper_video_init_params` (fps target, ffmpeg binary + directory, timestamp interval), a fourth `mtmd_helper_init_opt` parameter on the bitmap/tokenize + helpers, and the CLI flags `--video-fps`, `--video-timestamp-interval`, `--video-ffmpeg-dir`. + + The three flags were **deliberately not exposed** at the b10649 bump. They configure video decoding + and are inert without a way to *submit* a video: `ContentPart` offers text, image and audio only, so + adding the knobs alone would create exactly the class of dead-but-documented parameter that + `withTfsZ` / `withPenalizeNl` / `withPenaltyPrompt` had to be deprecated for. The bump passes + `mtmd_helper_init_opt_default()` at every call site, which is correct for image and audio. + + Doing it properly means: a `ContentPart.videoFile(Path)` (and probably a bytes overload), routing a + real `mtmd_helper_init_opt` from `ModelParameters` through the JNI multimodal path instead of the + default, `ModelParameters.setVideoFps/​setVideoTimestampInterval/​setVideoFfmpegDir`, and an + integration test. Note the runtime cost: upstream **shells out to `ffmpeg`/`ffprobe`**, so a + consumer needs those binaries on the host — worth stating prominently in the Javadoc, and it makes + the feature untestable on any CI runner without them. + +- **`--spec-synth-len` / `--spec-synth-rates` — deliberately NOT exposed, and this should stay that + way.** Added in b10649. Upstream's own help text marks both **"(benchmarking only)"**: they + synthesise fake per-position acceptance probabilities so the speculative-decoding harness can be + measured without a real draft model. They are an instrument for benchmarking llama.cpp itself, not a + knob for an application, and exposing them as library API would invite callers to "tune" numbers + that fabricate rather than measure acceptance. Anyone who genuinely wants them already has them: + `NativeServer` forwards raw llama-server argv verbatim. + - **Expose `--spec-draft-backend-sampling` toggle via `ModelParameters.setSpecDraftBackendSampling(boolean)`.** Added in b9437 (env `LLAMA_ARG_SPEC_DRAFT_BACKEND_SAMPLING`). Backend sampling for the speculative draft is enabled by default upstream but auto-disabled on `LLAMA_SPLIT_MODE_TENSOR` setups; an explicit Java-side setter lets callers force-disable it for benchmarking or for backends with sampler bugs. Speculative-decoding power users. - **Expose runtime reasoning control via `InferenceParameters.setReasoningControl(boolean)` + `LlamaModel.endReasoning(...)`.** Added in b9444–b9490: new `common_params_sampling::reasoning_control` flag arms the budget sampler so reasoning can be ended at runtime, and new `common_sampler_reasoning_budget_force(common_sampler *)` triggers the end-of-thinking token injection on the next sample. Upstream also adds a `POST /v1/chat/completions/control` server endpoint accepting `{"id": "...", "action": "reasoning_end"}`. Java mapping would be: (a) `InferenceParameters.setReasoningControl(boolean)` arms the sampler on the inference run, (b) a new `LlamaModel.endReasoning(int slotId)` (or per-streaming-task-id) JNI method calls the upstream `common_sampler_reasoning_budget_force` against the slot's sampler. Useful for interactive UIs that want a "skip thinking and answer now" button. Relevant only for reasoning-trained models (DeepSeek-R1, Qwen3-Thinking, GPT-OSS-Reasoner, etc.). diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 01b7995e8..ef1df2d0f 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -679,3 +679,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10636–b10639 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10639 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10639: all **7** patches applied, confirmed by the stamp file (`head 5e6a37cb115dc1074e274ac004373f5661909695` plus one SHA-256 line per patch) and by the source tree going dirty in 41 files. A fresh build directory was mandatory here — the applier pins its stamp to the checked-out llama.cpp commit and deliberately aborts rather than guess when an existing build dir's stamp names a different one. Full `cmake --build --config Release` clean, `ctest` **504/504**. `nm -D` on the freshly linked `libjllama.so` reports **40** `Java_*` exports and **zero** C++-mangled ones, so the `LlamaQuantizer` linkage fix carries forward. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10639") matches the `build-info` compiled into the new binary (`b10639-5e6a37cb1`), which is what proves the four pin sites and the actual build agree. **Note for the next bump:** that last check first reported a *false* drift (`must start with the pinned tag "b10636-"`) even though both the source and `target/classes` already read `b10639`. `LLAMA_CPP_VERSION` is a compile-time constant, so javac had inlined the old value into the already-compiled `NativeLibraryLoadSmokeTest.class` and Maven's incremental compilation cannot see that dependency; `mvn clean test` clears it. Recorded in the runbook so the next bump does not re-diagnose it. **Re-verified after `patches/0011` was added** (a long-standing upstream bug this bump's CI was the first run ever to reach — see the patch table in `CLAUDE.md`; it is *not* a b10636–b10639 regression, the range touches no `common/` file at all): a second fresh build directory, applier run and Release build with **8** patches in the stamp, `ctest` **509/509**, `nm -D` unchanged at 40/0, `NativeLibraryLoadSmokeTest` still 3/3 with 0 skipped. The "all 7 patches" counts in the two rows above are left as written — they are the accurate record of what the *bump* required. | | b10639–b10644 | `include/llama.h` (**`LLAMA_SESSION_VERSION` 9 → 10 and `LLAMA_STATE_SEQ_VERSION` 2 → 3** — the only two lines in the file that moved), `src/llama-kv-cells.h` + `src/llama-kv-cache.{cpp,h}` (`llama_kv_cell_ext` gains a `tok` field for n-gram input embeddings, plus `has_cell_ext()` / `get_prev_tokens()` / `for_each_token_in()`, and `reset()` swaps a `memset` for value-initialisation), `src/models/nanbeige.cpp` (one line: register `t_layer_inp[il]`), `tests/test-backend-ops.cpp`, the whole `ggml/src/ggml-hexagon/**` + `scripts/snapdragon/**` + `docs/backend/snapdragon/**` Snapdragon/Hexagon backend rework, plus `tools/ui/**` (the WebUI, auto-followed) | **No project-source change, and the only priority-review-list file touched is `include/llama.h`, whose entire diff is the two version macros.** Sizing: the raw diff is ~608 KiB, but it is dominated by two components this project never compiles or reviews — `tools/ui` (auto-followed; the `build-webui` job rebuilds it from the pin) and the Hexagon backend (`grep -rn GGML_HEXAGON` over this repo's workflows, build scripts and CMake is empty, so `ggml-hexagon.cpp` is never in a `libjllama` link line). Excluding both leaves **11 KiB across 6 files**, well under the runbook's 100 KiB single-step threshold, so no chunking was needed. **Zero** files under `common/`, `tools/server/` or `tools/mtmd/` changed, so every remaining row of the API-compatibility table is vacuously satisfied and the three mechanical server-contract greps have no input — the request-field set, its bounds and the emitted response keys cannot have moved. **Compatibility note for consumers, not a compile break:** the two version bumps are a *state-file format* break — a slot state saved by an older build (`LlamaModel.handleSlotAction` save, or the server's `/slots/{id}?action=save`) is rejected by its own version check on b10644 and must be regenerated. Nothing the project calls changed shape. **All 8 patches apply unchanged** — the 6 project-relevant changed files and the 42 files the patches target have **zero** overlap (no `common/arg.{cpp,h}`, no `common/peg-parser.cpp`, no `tools/server/*`; the one changed test, `tests/test-backend-ops.cpp`, is touched by no patch). | | b10639–b10644 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10644 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10644: all **8** patches applied, confirmed by the stamp file (`head d7a2074112d27649303fa107eb8c94db1ee435f3` plus one SHA-256 line per patch) — including `0011`, whose target `common/peg-parser.cpp` the range does not touch. Full `cmake --build --config Release` clean, `ctest` **509/509**. `nm -D` on the freshly linked `libjllama.so` reports **40** `Java_*` exports and **zero** C++-mangled ones. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10644") matches the `build-info` compiled into the new binary (`b10644-d7a207411`), which is what proves the four pin sites and the actual build agree. Run with `mvn clean test`, not a bare `mvn test`: `LLAMA_CPP_VERSION` is a compile-time constant that javac inlines into the already-compiled test class, and Maven's incremental compilation cannot see that dependency — the b10636→b10639 row below records the false drift that costs. | +| b10644–b10649 | **`tools/mtmd/mtmd-helper.{h,cpp}` (BREAKING: `mtmd_helper_bitmap_init_from_file` and `_from_buf` gain a 4th `struct mtmd_helper_init_opt` parameter; new `mtmd_helper_video_init_params` + `mtmd_helper_init_opt` structs and their `_default()` factories)**, **`tools/server/server-common.{h,cpp}` (BREAKING: `tokenize_input_prompts`, `process_mtmd_prompt` and `format_prompt_rerank` all gain a trailing `const mtmd_helper_init_opt &`)**, `tools/server/server-context.{h,cpp}` (`handle_count_tokens` gains the same parameter; threading it through), `common/common.h` (**additive**: `common_params_speculative::{synth_len,synth_rates}` + `has_synth()`, `common_params::{video_fps,video_timestamp_interval_ms,video_ffmpeg_bin_dir}`), `common/arg.cpp` (**purely additive**: 7 new flags), `common/speculative.{cpp,h}` (**additive**: `common_speculative_n_max`, `_synth_rates_resolve`, `_get_synth_probs`), `ggml/src/ggml-metal/*`, `src/models/minimax-01.cpp`, `tests/test-arg-parser.cpp`, `tools/{cli,llama-bench,tts}`, `tools/server/tests/*`, plus `tools/ui/**` | **The first range in this whole bump that broke the project's own compile — four call sites, all from one upstream refactor.** llama.cpp b10649 threaded a new `mtmd_helper_init_opt` (video decode settings: fps target, ffmpeg binary dir, timestamp interval) through every helper that can ingest media. `tts_engine.cpp:95` (`mtmd_helper_bitmap_init_from_file`, the speaker-reference clip) and `jllama.cpp` ×3 (`tokenize_input_prompts`) + ×1 (`format_prompt_rerank`) all lost their signatures. Every one of those paths passes `mctx = nullptr` or handles audio, so none wants video settings: each now passes `mtmd_helper_init_opt_default()`, which is upstream's own default. **This is the failure class the priority table's note calls out** — `server-common.h` is a same-repo header the project `#include`s directly rather than one reachable through the documented dependency graph, so only a real compile finds it. **The wire contract did not move:** all three mechanical greps are identical across the range (68 request fields, 23 bounds, 304 response keys over all six server `.cpp`). `common/arg.cpp` removed or renamed **zero** flags. Sizing: 107 KiB reviewable, or **72 KiB excluding `ggml-metal`** (Metal backend internals behind unchanged public headers — same exclusion rationale as Hexagon), under the runbook's 100 KiB single-step threshold. **All 8 patches apply unchanged** despite six patch-target files changing (`common/arg.cpp`, `tests/test-arg-parser.cpp`, `tools/mtmd/mtmd-cli.cpp`, `tools/server/server-context.{cpp,h}`, `tools/tts/tts.cpp`) — upstream's edits fall outside every patched region. | +| b10644–b10649 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10649 -DBUILD_TESTING=ON` through the real `FetchContent` path: all **8** patches applied (stamp `head 2bb9bddafad44ecbb50889644ca47537ec11841b`), which was the first thing checked because six patch-target files changed in the range. The first build failed loud with the four signature breaks above; after adapting the call sites, Release build clean, `ctest` **512/512**, `nm -D` **40** `Java_*` exports and **zero** C++-mangled, `NativeLibraryLoadSmokeTest` **3/3, 0 skipped** including the pin cross-check against the linked `build-info`. `ModelParametersTest` 73/73 with the five new CPU-offload cases. **Feature exposure decided deliberately, not by default:** of the 7 new flags, `--n-cpu-moe` and `--n-cpu-ffn` were added as `ModelParameters.setCpuMoeLayers/setCpuFfnLayers` (real VRAM knobs, and `--n-cpu-moe` had never been exposed either); the two `--spec-synth-*` flags were refused as upstream marks them "benchmarking only"; and the three `--video-*` flags were refused because `ContentPart` has no video input path, so they would be inert — the same dead-knob shape that forced the `withTfsZ`/`withPenalizeNl`/`withPenaltyPrompt` deprecations. Both refusals are recorded in `TODO.md` with their reasoning so they do not resurface as open questions. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 216d9105b..e3801cb26 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10644 + GIT_TAG b10649 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/cpp/jllama.cpp b/llama/src/main/cpp/jllama.cpp index e1729380c..f9aedef01 100644 --- a/llama/src/main/cpp/jllama.cpp +++ b/llama/src/main/cpp/jllama.cpp @@ -246,7 +246,8 @@ static void populate_completion_task(server_task &task, jllama_context *jctx, const std::vector &logit_bias_eog, const json &data, bool has_mtmd, std::vector files = {}) { if (!configure_multimodal_task_impl(task, has_mtmd, data, std::move(files))) { - auto tokenized_prompts = tokenize_input_prompts(jctx->vocab, nullptr, data.at("prompt"), true, true); + auto tokenized_prompts = + tokenize_input_prompts(jctx->vocab, nullptr, data.at("prompt"), true, true, mtmd_helper_init_opt_default()); if (!tokenized_prompts.empty()) { task.tokens = std::move(tokenized_prompts[0]); } @@ -1185,9 +1186,11 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleRerank(JNIEn std::vector tasks; tasks.reserve(document_vector.size()); for (size_t i = 0; i < document_vector.size(); i++) { - tasks.push_back(build_indexed_token_task( - rd, SERVER_TASK_TYPE_RERANK, format_prompt_rerank(model, jctx->vocab, nullptr, prompt, document_vector[i]), - static_cast(i), TASK_RESPONSE_TYPE_NONE)); + tasks.push_back( + build_indexed_token_task(rd, SERVER_TASK_TYPE_RERANK, + format_prompt_rerank(model, jctx->vocab, nullptr, prompt, document_vector[i], + mtmd_helper_init_opt_default()), + static_cast(i), TASK_RESPONSE_TYPE_NONE)); } rd.post_tasks(std::move(tasks)); @@ -1501,7 +1504,7 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleInfill(JNIEn std::string prompt = json_value(data, "prompt", std::string()); try { std::vector tokenized_prompts = - tokenize_input_prompts(jctx->vocab, nullptr, prompt, false, true); + tokenize_input_prompts(jctx->vocab, nullptr, prompt, false, true, mtmd_helper_init_opt_default()); data["prompt"] = format_prompt_infill(jctx->vocab, data.at("input_prefix"), data.at("input_suffix"), data.at("input_extra"), @@ -1554,7 +1557,8 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleEmbeddings(J std::vector tokenized_prompts; try { - tokenized_prompts = tokenize_input_prompts(jctx->vocab, nullptr, prompt, true, true); + tokenized_prompts = + tokenize_input_prompts(jctx->vocab, nullptr, prompt, true, true, mtmd_helper_init_opt_default()); } catch (const std::exception &e) { env->ThrowNew(c_llama_error, e.what()); return nullptr; diff --git a/llama/src/main/cpp/tts_engine.cpp b/llama/src/main/cpp/tts_engine.cpp index d864dc6e9..9d7be97bb 100644 --- a/llama/src/main/cpp/tts_engine.cpp +++ b/llama/src/main/cpp/tts_engine.cpp @@ -92,7 +92,11 @@ bool engine_synthesize(tts_engine *engine, const std::string &text, const std::s mtmd::bitmap_ptr speaker_bitmap; if (!speaker_reference_path.empty()) { - auto wrapper = mtmd_helper_bitmap_init_from_file(engine->mctx.get(), speaker_reference_path.c_str(), false); + // b10649 added a fourth `mtmd_helper_init_opt` parameter carrying the video-decode settings + // (fps target, ffmpeg binary dir, timestamp interval). None of it applies to a speaker + // reference clip, which is audio, so pass upstream's own defaults rather than inventing values. + auto wrapper = mtmd_helper_bitmap_init_from_file(engine->mctx.get(), speaker_reference_path.c_str(), false, + mtmd_helper_init_opt_default()); if (!wrapper.bitmap) { common_sampler_free(smpl); err = "failed to load speaker reference audio: " + speaker_reference_path; diff --git a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java index 7e8d18b84..daeec0f3e 100644 --- a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java +++ b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java @@ -876,6 +876,49 @@ public ModelParameters setDevices(String devices) { return this; } + /** + * Keep the Mixture-of-Experts weights of the first {@code n} layers on the CPU. + * + *

The companion to {@link #setGpuLayers(int)} for VRAM-constrained hosts running a MoE model. + * Where {@code setGpuLayers} moves whole layers, this moves only the expert weights — + * which dominate a MoE model's size — so attention and the rest of each layer stay on the GPU. + * That usually fits a far larger model in the same VRAM at a much smaller speed cost than + * reducing the layer count would. Upstream {@code --n-cpu-moe} / {@code -ncmoe}, added in + * llama.cpp b10649; use {@link #setCpuFfnLayers(int)} for a dense model.

+ * + * @param layers the number of leading layers whose MoE expert weights stay on the CPU + * @return this builder + * @throws IllegalArgumentException if {@code layers} is negative + */ + public ModelParameters setCpuMoeLayers(int layers) { + if (layers < 0) { + throw new IllegalArgumentException( + "Invalid n-cpu-moe value: " + layers + " (must be >= 0; 0 = keep all experts on the GPU)"); + } + return putScalar("--n-cpu-moe", layers); + } + + /** + * Keep the dense FFN weights of the first {@code n} layers on the CPU. + * + *

The dense-model counterpart of {@link #setCpuMoeLayers(int)}: it offloads the feed-forward + * weights rather than the MoE experts, trading a little speed for VRAM headroom without giving + * up whole layers. Upstream {@code --n-cpu-ffn} / {@code -ncffn}, added in llama.cpp b10649. + * On a MoE model use {@link #setCpuMoeLayers(int)} instead — this flag does not touch expert + * weights, so it frees very little there.

+ * + * @param layers the number of leading layers whose dense FFN weights stay on the CPU + * @return this builder + * @throws IllegalArgumentException if {@code layers} is negative + */ + public ModelParameters setCpuFfnLayers(int layers) { + if (layers < 0) { + throw new IllegalArgumentException( + "Invalid n-cpu-ffn value: " + layers + " (must be >= 0; 0 = keep all FFN weights on the GPU)"); + } + return putScalar("--n-cpu-ffn", layers); + } + /** * Set the number of layers to store in VRAM. * diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 746e202ef..cb203d3f0 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10644"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10649"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10644-"} — call + * plus the resolved upstream commit, e.g. {@code "b10649-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10644"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10649"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10644"; + public static final String LLAMA_CPP_VERSION = "b10649"; // Constants holder — not instantiable. private LlamaCppVersion() {} diff --git a/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java b/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java index 40046d67e..e2309dd01 100644 --- a/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java +++ b/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java @@ -83,6 +83,46 @@ public void testSetPriorityBatchTooHigh() { assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setPriorityBatch(4)); } + // ------------------------------------------------------------------------- + // setCpuMoeLayers / setCpuFfnLayers — the CPU-offload pair added in llama.cpp b10649 + // ------------------------------------------------------------------------- + + @Test + public void testSetCpuMoeLayersRendersTheUpstreamFlag() { + ModelParameters p = new ModelParameters().setCpuMoeLayers(12); + assertThat(p.parameters.get("--n-cpu-moe"), is("12")); + } + + @Test + public void testSetCpuFfnLayersRendersTheUpstreamFlag() { + ModelParameters p = new ModelParameters().setCpuFfnLayers(8); + assertThat(p.parameters.get("--n-cpu-ffn"), is("8")); + } + + @Test + public void testCpuOffloadLayersAcceptZeroMeaningKeepEverythingOnTheGpu() { + ModelParameters p = new ModelParameters().setCpuMoeLayers(0).setCpuFfnLayers(0); + assertThat(p.parameters.get("--n-cpu-moe"), is("0")); + assertThat(p.parameters.get("--n-cpu-ffn"), is("0")); + } + + @Test + public void testCpuOffloadLayersRejectNegative() { + // Upstream throws invalid_argument on a negative value, which would surface as a model-load + // failure with no indication of the cause; reject it here where the message can name it. + assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setCpuMoeLayers(-1)); + assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setCpuFfnLayers(-1)); + } + + @Test + public void testCpuOffloadLayersAreIndependentOfEachOther() { + // They target different weight sets (MoE experts vs dense FFN) and write different flags, so + // setting one must not disturb the other -- unlike the mmproj device/offload pair. + ModelParameters p = new ModelParameters().setCpuMoeLayers(4).setCpuFfnLayers(9); + assertThat(p.parameters.get("--n-cpu-moe"), is("4")); + assertThat(p.parameters.get("--n-cpu-ffn"), is("9")); + } + // ------------------------------------------------------------------------- // setRepeatLastN — validation (>= 0) // ------------------------------------------------------------------------- From c5d3d23d0d781f45f14c130bdae3c4d483c84bfe Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 19:32:56 +0000 Subject: [PATCH 64/76] fix: wake the task queue before posting; close the second-audit findings A second verification pass over this branch's own work found one runtime defect and a set of claims and guards that did not hold up. The defect: with setSleepIdleSeconds(> 0) the model became permanently unusable after the first idle period. server_queue::post() does not leave the sleeping state -- it notifies the condition variable, whose sleeping predicate tests req_stop_sleeping, so the loop woke, re-tested and slept again with the task still queued. Upstream performs the wake for the caller in server_res_generator's constructor, but only for readers built through create_response(); this layer builds its readers with the CLI-facing get_response_reader(), which does not, and nothing here called wait_until_no_sleep() at all. Every later call then blocked until close() or threw "No result", for the process lifetime. All six post sites now go through wake_and_post(). The earlier is_sleeping() wait predicate turned the hang into an exception but did not address the cause; it stays as a backstop, with its comment corrected to say so. Extracted the trainer's common_params assembly into train_params.hpp, the sibling of tts_params.hpp. train_engine.cpp is compiled into jllama only, never into jllama_test, and LlamaTrainerIntegrationTest is gated on a property no CI job sets -- so its resolve_cpu_params() call, which guards a JVM abort, had no runnable guard on any platform. The comment claiming otherwise was wrong. Verified by mutation: removing the call now reds TrainParams.ResolvesBothCpuThreadCounts. Deprecated InferenceParameters.withUseChatTemplate and withChatTemplate. Neither key is read from a request body upstream -- use_jinja is set only by --jinja/--no-jinja, and the only "chat_template" string in common/ or tools/server/ is the one /props emits -- so both were silently no-ops, including at three call sites that used withUseChatTemplate(true) to "enable jinja for tools". Those calls are removed; tool calling was unaffected in practice only because upstream defaults use_jinja to true. Made the mmproj device/offload clearing symmetric. Clearing in one direction still lost the HashMap-order race whenever setMmprojOffload was called second, which is exactly what the one-sided version claimed to fix. Strengthened guards that mutation testing showed were weak or vacuous: the source-scanning rule matched only a bare string literal, so it could not see the house idiom (a PROP_* constant) or a wrapped call, and had no true positives -- it now inspects each call's own argument list, and is proven against both forms; the parent-directory resolver test passed with the branch deleted, so it is dropped in favour of the pre-existing test that does fail, and the langchain4j mirror gains the branch test it never had; the batch-inheritance test derived its fixture from the host core count so it could pass with the bug on a 3-core runner; the modality fixtures all set audio and video alike, so a getter wired to the wrong key still passed. Added the first assertion that the native side actually emits modalities.video, and the first that the mmproj pair clears both ways. Corrected version attributions that were wrong in shipped Javadoc: the /metrics idle-timer exemption is b10519 (#27376), not b10644; the LLAMA_STATE_SEQ_VERSION bump is b10642, not b10644; --n-cpu-moe has existed since b6089 (#15077) and was merely never exposed here, only --n-cpu-ffn is new at b10649. The restore-slot Javadoc quoted an unwrapped message a caller never sees. The bump record overstated the response-key count and the diff size and listed a WebUI change the range does not contain. 516/516 C++ tests, 1449 Java tests with 0 failures, javadoc and clang-format clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CHANGELOG.md | 37 ++++-- CLAUDE.md | 4 +- docs/history/llama-cpp-breaking-changes.md | 4 +- .../llama/langchain4j/LangChain4jMapping.java | 2 +- .../JllamaChatModelIntegrationTest.java | 8 +- .../llama/langchain4j/TestModelPathsTest.java | 71 +++++++++- llama/src/main/cpp/jllama.cpp | 56 ++++++-- llama/src/main/cpp/train_engine.cpp | 37 +----- llama/src/main/cpp/train_params.hpp | 78 +++++++++++ .../java/net/ladenthin/llama/LlamaModel.java | 12 +- .../ladenthin/llama/args/ReasoningFormat.java | 2 +- .../llama/parameters/InferenceParameters.java | 25 +++- .../llama/parameters/ModelParameters.java | 20 ++- .../llama/server/OpenAiRequestMapper.java | 2 +- llama/src/test/cpp/test_tts_params.cpp | 121 +++++++++++++++--- .../llama/MultimodalIntegrationTest.java | 8 ++ .../ladenthin/llama/TestConstantsTest.java | 86 ++++++++----- .../llama/parameters/ModelParametersTest.java | 25 +++- .../OpenAiCompatServerIntegrationTest.java | 8 +- .../llama/server/OpenAiRequestMapperTest.java | 7 +- .../ladenthin/llama/value/ModelMetaTest.java | 24 ++++ 21 files changed, 492 insertions(+), 145 deletions(-) create mode 100644 llama/src/main/cpp/train_params.hpp diff --git a/CHANGELOG.md b/CHANGELOG.md index 87b18ea3d..478d8d47c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,9 +17,10 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by ### Added - **`ModelParameters.setCpuMoeLayers(int)` / `setCpuFfnLayers(int)`** — keep the first N layers' Mixture-of-Experts weights, or dense FFN weights, on the CPU (upstream `--n-cpu-moe` / `-ncmoe` and - `--n-cpu-ffn` / `-ncffn`, llama.cpp b10649). The companions to `setGpuLayers`: where that moves whole - layers, these move only the weight class that dominates a model's size, usually fitting a much larger - model into the same VRAM at a smaller speed cost. `--n-cpu-moe` had never been exposed either. + `--n-cpu-ffn` / `-ncffn`). The companions to `setGpuLayers`: where that moves whole layers, these move + only the weight class that dominates a model's size, usually fitting a much larger model into the same + VRAM at a smaller speed cost. Only `--n-cpu-ffn` is new in llama.cpp b10649; `--n-cpu-moe` has existed + upstream since b6089 but had never been exposed here. - **`ServerMetrics.getWindowPromptProcessingMillis()` / `getWindowTokenGenerationMillis()` / `getWindowTimings()`** — typed access to the current-window timing keys `t_prompt_processing` and `t_tokens_generation`. Both were always emitted; only the cumulative `_total` variants had accessors. @@ -47,12 +48,18 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by `tfs_z`, `penalize_nl` and `penalty_prompt` appear nowhere in upstream `common/` or `tools/server/` at the pinned build, and the request schema discards unknown fields rather than rejecting them — so these have been silently doing nothing. Kept compiling for now; they will be removed. -- `ModelParameters.setMmprojDevice` now clears the mmproj-offload flags. Both write upstream's single - `mmproj_use_gpu` field, and the rendered argv comes out of a `HashMap`, so setting both previously - left the winner to hash order. -- `getMetrics()` no longer blocks until `close()` when the task queue is asleep. The wait predicate now - mirrors upstream's own (`closing || queue_tasks.is_sleeping()`); a prior comment wrongly claimed the - getter was unreachable from this layer. Only reachable with `setSleepIdleSeconds(> 0)`, off by default. +- `ModelParameters.setMmprojDevice` and `setMmprojOffload` now clear each other. Both write upstream's + single `mmproj_use_gpu` field, and the rendered argv comes out of a `HashMap`, so leaving both present + left the winner to hash order. Clearing in only one direction still lost the race whenever + `setMmprojOffload` was called second; the contract is now simply "the last of the two calls wins". +- **Deprecated `InferenceParameters.withUseChatTemplate` and `withChatTemplate`.** Both are load-time + settings upstream, not per-request ones: `common_params::use_jinja` is set only by `--jinja` / + `--no-jinja`, and the only `"chat_template"` string in upstream `common/` or `tools/server/` is the one + the server *emits* from `/props`. Neither key is ever read from a request body, so both calls were + silently doing nothing — including at three call sites in this library that used + `withUseChatTemplate(true)` to "enable jinja for tools", which those calls could not do. Use + `ModelParameters.enableJinja()` / `setChatTemplate(String)` instead. Tool calling was unaffected in + practice only because upstream defaults `use_jinja` to true. - `ch.qos.logback:logback-classic` bumped 1.6.2 → 1.6.3 (test/runtime binding only). - CI actions bumped to latest: `actions/setup-java` v5 → v6. - Upgraded llama.cpp from **b9894 to b9917** (all eight local patches re-verified across the range). @@ -164,6 +171,18 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by reading (`ggml_time_us()`), not milliseconds since the epoch. The value is unchanged. ### Fixed +- **With `setSleepIdleSeconds(> 0)`, the model became permanently unusable after the first idle + period.** Once llama.cpp's task queue enters its sleeping state, posting a task does not leave it: + `server_queue::post()` only notifies the condition variable, whose sleeping predicate tests + `req_stop_sleeping`, so the loop woke, re-tested, and went straight back to sleep with the task + still queued. Upstream performs the wake on the caller's behalf in `server_res_generator`'s + constructor (`wait_until_no_sleep()`), but only for readers built through `create_response()`; + this binding builds its readers with the CLI-facing `get_response_reader()`, which does not, and + nothing in the JNI layer called `wait_until_no_sleep()` at all. Every subsequent call then either + blocked until `close()` (completions, embeddings, rerank, infill) or threw `"No result"` + (`getMetrics`, LoRA and slot operations), for the lifetime of the process. All six post sites now + wake the queue first. Idle-sleep is off by default (`-1`), so a default configuration was never + affected. - **A single malformed UTF-8 byte in a model's output turned a finished generation into an HTTP 500.** The server parses *every* completion through `common_chat_parse()`; with no chat parser configured (plain `/completion`) that is llama.cpp's content-only fallback, whose scan tolerates an incomplete diff --git a/CLAUDE.md b/CLAUDE.md index 3155e4267..685f10fdd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1195,8 +1195,8 @@ helper `server_metrics_to_json` (`json_helpers.hpp`, unit-tested in `test_json_h than letting the Java contract follow upstream's transport split. Durations are converted from upstream microseconds to the milliseconds the payload has always used. The merge also surfaces the counters upstream added since — `n_prompt_tokens_cached_total` and the speculative-decoding tallies -(`n_draft_tokens_total`, `n_draft_accepted_total`, `n_draft_verify_steps_total`, -`n_draft_accepted_per_pos`) — which previously existed only inside the Prometheus text and had no +(`n_draft_tokens_total`, `n_draft_accepted_total`, `n_draft_verif_steps_total`, +`n_accepted_per_pos_total` — upstream's own spellings, kept verbatim) — which previously existed only inside the Prometheus text and had no JSON representation at all; `value.ServerMetrics` exposes them with typed getters (plus a derived `getDraftAcceptanceRate()`). No second JNI entry point and no Prometheus-text parser were needed. diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index ef1df2d0f..319729360 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -679,5 +679,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10636–b10639 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10639 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10639: all **7** patches applied, confirmed by the stamp file (`head 5e6a37cb115dc1074e274ac004373f5661909695` plus one SHA-256 line per patch) and by the source tree going dirty in 41 files. A fresh build directory was mandatory here — the applier pins its stamp to the checked-out llama.cpp commit and deliberately aborts rather than guess when an existing build dir's stamp names a different one. Full `cmake --build --config Release` clean, `ctest` **504/504**. `nm -D` on the freshly linked `libjllama.so` reports **40** `Java_*` exports and **zero** C++-mangled ones, so the `LlamaQuantizer` linkage fix carries forward. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10639") matches the `build-info` compiled into the new binary (`b10639-5e6a37cb1`), which is what proves the four pin sites and the actual build agree. **Note for the next bump:** that last check first reported a *false* drift (`must start with the pinned tag "b10636-"`) even though both the source and `target/classes` already read `b10639`. `LLAMA_CPP_VERSION` is a compile-time constant, so javac had inlined the old value into the already-compiled `NativeLibraryLoadSmokeTest.class` and Maven's incremental compilation cannot see that dependency; `mvn clean test` clears it. Recorded in the runbook so the next bump does not re-diagnose it. **Re-verified after `patches/0011` was added** (a long-standing upstream bug this bump's CI was the first run ever to reach — see the patch table in `CLAUDE.md`; it is *not* a b10636–b10639 regression, the range touches no `common/` file at all): a second fresh build directory, applier run and Release build with **8** patches in the stamp, `ctest` **509/509**, `nm -D` unchanged at 40/0, `NativeLibraryLoadSmokeTest` still 3/3 with 0 skipped. The "all 7 patches" counts in the two rows above are left as written — they are the accurate record of what the *bump* required. | | b10639–b10644 | `include/llama.h` (**`LLAMA_SESSION_VERSION` 9 → 10 and `LLAMA_STATE_SEQ_VERSION` 2 → 3** — the only two lines in the file that moved), `src/llama-kv-cells.h` + `src/llama-kv-cache.{cpp,h}` (`llama_kv_cell_ext` gains a `tok` field for n-gram input embeddings, plus `has_cell_ext()` / `get_prev_tokens()` / `for_each_token_in()`, and `reset()` swaps a `memset` for value-initialisation), `src/models/nanbeige.cpp` (one line: register `t_layer_inp[il]`), `tests/test-backend-ops.cpp`, the whole `ggml/src/ggml-hexagon/**` + `scripts/snapdragon/**` + `docs/backend/snapdragon/**` Snapdragon/Hexagon backend rework, plus `tools/ui/**` (the WebUI, auto-followed) | **No project-source change, and the only priority-review-list file touched is `include/llama.h`, whose entire diff is the two version macros.** Sizing: the raw diff is ~608 KiB, but it is dominated by two components this project never compiles or reviews — `tools/ui` (auto-followed; the `build-webui` job rebuilds it from the pin) and the Hexagon backend (`grep -rn GGML_HEXAGON` over this repo's workflows, build scripts and CMake is empty, so `ggml-hexagon.cpp` is never in a `libjllama` link line). Excluding both leaves **11 KiB across 6 files**, well under the runbook's 100 KiB single-step threshold, so no chunking was needed. **Zero** files under `common/`, `tools/server/` or `tools/mtmd/` changed, so every remaining row of the API-compatibility table is vacuously satisfied and the three mechanical server-contract greps have no input — the request-field set, its bounds and the emitted response keys cannot have moved. **Compatibility note for consumers, not a compile break:** the two version bumps are a *state-file format* break — a slot state saved by an older build (`LlamaModel.handleSlotAction` save, or the server's `/slots/{id}?action=save`) is rejected by its own version check on b10644 and must be regenerated. Nothing the project calls changed shape. **All 8 patches apply unchanged** — the 6 project-relevant changed files and the 42 files the patches target have **zero** overlap (no `common/arg.{cpp,h}`, no `common/peg-parser.cpp`, no `tools/server/*`; the one changed test, `tests/test-backend-ops.cpp`, is touched by no patch). | | b10639–b10644 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10644 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10644: all **8** patches applied, confirmed by the stamp file (`head d7a2074112d27649303fa107eb8c94db1ee435f3` plus one SHA-256 line per patch) — including `0011`, whose target `common/peg-parser.cpp` the range does not touch. Full `cmake --build --config Release` clean, `ctest` **509/509**. `nm -D` on the freshly linked `libjllama.so` reports **40** `Java_*` exports and **zero** C++-mangled ones. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10644") matches the `build-info` compiled into the new binary (`b10644-d7a207411`), which is what proves the four pin sites and the actual build agree. Run with `mvn clean test`, not a bare `mvn test`: `LLAMA_CPP_VERSION` is a compile-time constant that javac inlines into the already-compiled test class, and Maven's incremental compilation cannot see that dependency — the b10636→b10639 row below records the false drift that costs. | -| b10644–b10649 | **`tools/mtmd/mtmd-helper.{h,cpp}` (BREAKING: `mtmd_helper_bitmap_init_from_file` and `_from_buf` gain a 4th `struct mtmd_helper_init_opt` parameter; new `mtmd_helper_video_init_params` + `mtmd_helper_init_opt` structs and their `_default()` factories)**, **`tools/server/server-common.{h,cpp}` (BREAKING: `tokenize_input_prompts`, `process_mtmd_prompt` and `format_prompt_rerank` all gain a trailing `const mtmd_helper_init_opt &`)**, `tools/server/server-context.{h,cpp}` (`handle_count_tokens` gains the same parameter; threading it through), `common/common.h` (**additive**: `common_params_speculative::{synth_len,synth_rates}` + `has_synth()`, `common_params::{video_fps,video_timestamp_interval_ms,video_ffmpeg_bin_dir}`), `common/arg.cpp` (**purely additive**: 7 new flags), `common/speculative.{cpp,h}` (**additive**: `common_speculative_n_max`, `_synth_rates_resolve`, `_get_synth_probs`), `ggml/src/ggml-metal/*`, `src/models/minimax-01.cpp`, `tests/test-arg-parser.cpp`, `tools/{cli,llama-bench,tts}`, `tools/server/tests/*`, plus `tools/ui/**` | **The first range in this whole bump that broke the project's own compile — four call sites, all from one upstream refactor.** llama.cpp b10649 threaded a new `mtmd_helper_init_opt` (video decode settings: fps target, ffmpeg binary dir, timestamp interval) through every helper that can ingest media. `tts_engine.cpp:95` (`mtmd_helper_bitmap_init_from_file`, the speaker-reference clip) and `jllama.cpp` ×3 (`tokenize_input_prompts`) + ×1 (`format_prompt_rerank`) all lost their signatures. Every one of those paths passes `mctx = nullptr` or handles audio, so none wants video settings: each now passes `mtmd_helper_init_opt_default()`, which is upstream's own default. **This is the failure class the priority table's note calls out** — `server-common.h` is a same-repo header the project `#include`s directly rather than one reachable through the documented dependency graph, so only a real compile finds it. **The wire contract did not move:** all three mechanical greps are identical across the range (68 request fields, 23 bounds, 304 response keys over all six server `.cpp`). `common/arg.cpp` removed or renamed **zero** flags. Sizing: 107 KiB reviewable, or **72 KiB excluding `ggml-metal`** (Metal backend internals behind unchanged public headers — same exclusion rationale as Hexagon), under the runbook's 100 KiB single-step threshold. **All 8 patches apply unchanged** despite six patch-target files changing (`common/arg.cpp`, `tests/test-arg-parser.cpp`, `tools/mtmd/mtmd-cli.cpp`, `tools/server/server-context.{cpp,h}`, `tools/tts/tts.cpp`) — upstream's edits fall outside every patched region. | -| b10644–b10649 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10649 -DBUILD_TESTING=ON` through the real `FetchContent` path: all **8** patches applied (stamp `head 2bb9bddafad44ecbb50889644ca47537ec11841b`), which was the first thing checked because six patch-target files changed in the range. The first build failed loud with the four signature breaks above; after adapting the call sites, Release build clean, `ctest` **512/512**, `nm -D` **40** `Java_*` exports and **zero** C++-mangled, `NativeLibraryLoadSmokeTest` **3/3, 0 skipped** including the pin cross-check against the linked `build-info`. `ModelParametersTest` 73/73 with the five new CPU-offload cases. **Feature exposure decided deliberately, not by default:** of the 7 new flags, `--n-cpu-moe` and `--n-cpu-ffn` were added as `ModelParameters.setCpuMoeLayers/setCpuFfnLayers` (real VRAM knobs, and `--n-cpu-moe` had never been exposed either); the two `--spec-synth-*` flags were refused as upstream marks them "benchmarking only"; and the three `--video-*` flags were refused because `ContentPart` has no video input path, so they would be inert — the same dead-knob shape that forced the `withTfsZ`/`withPenalizeNl`/`withPenaltyPrompt` deprecations. Both refusals are recorded in `TODO.md` with their reasoning so they do not resurface as open questions. | +| b10644–b10649 | **`tools/mtmd/mtmd-helper.{h,cpp}` (BREAKING: `mtmd_helper_bitmap_init_from_file` and `_from_buf` gain a 4th `struct mtmd_helper_init_opt` parameter; new `mtmd_helper_video_init_params` + `mtmd_helper_init_opt` structs and their `_default()` factories)**, **`tools/server/server-common.{h,cpp}` (BREAKING: `tokenize_input_prompts`, `process_mtmd_prompt` and `format_prompt_rerank` all gain a trailing `const mtmd_helper_init_opt &`)**, `tools/server/server-context.{h,cpp}` (`handle_count_tokens` gains the same parameter; threading it through), `common/common.h` (**additive**: `common_params_speculative::{synth_len,synth_rates}` + `has_synth()`, `common_params::{video_fps,video_timestamp_interval_ms,video_ffmpeg_bin_dir}`), `common/arg.cpp` (**purely additive**: 6 new flags), `common/speculative.{cpp,h}` (**additive**: `common_speculative_n_max`, `_synth_rates_resolve`, `_get_synth_probs`), `ggml/src/ggml-metal/*`, `src/models/minimax-01.cpp`, `tests/test-arg-parser.cpp`, `tools/{cli,llama-bench,tts}`, `tools/server/tests/*` | **The first range in this whole bump that broke the project's own compile — four call sites, all from one upstream refactor.** llama.cpp b10649 threaded a new `mtmd_helper_init_opt` (video decode settings: fps target, ffmpeg binary dir, timestamp interval) through every helper that can ingest media. `tts_engine.cpp:95` (`mtmd_helper_bitmap_init_from_file`, the speaker-reference clip) and `jllama.cpp` ×3 (`tokenize_input_prompts`) + ×1 (`format_prompt_rerank`) all lost their signatures. Every one of those paths passes `mctx = nullptr` or handles audio, so none wants video settings: each now passes `mtmd_helper_init_opt_default()`, which is upstream's own default. **This is the failure class the priority table's note calls out** — `server-common.h` is a same-repo header the project `#include`s directly rather than one reachable through the documented dependency graph, so only a real compile finds it. **The wire contract did not move:** all three mechanical greps are identical across the range (68 request fields, 23 bounds, 286 response keys over all six server `.cpp`). `common/arg.cpp` removed or renamed **zero** flags. Sizing: 106 KiB reviewable, or **72 KiB excluding `ggml-metal`** (Metal backend internals behind unchanged public headers — same exclusion rationale as Hexagon), under the runbook's 100 KiB single-step threshold. **All 8 patches apply unchanged** despite six patch-target files changing (`common/arg.cpp`, `tests/test-arg-parser.cpp`, `tools/mtmd/mtmd-cli.cpp`, `tools/server/server-context.{cpp,h}`, `tools/tts/tts.cpp`) — upstream's edits fall outside every patched region. | +| b10644–b10649 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10649 -DBUILD_TESTING=ON` through the real `FetchContent` path: all **8** patches applied (stamp `head 2bb9bddafad44ecbb50889644ca47537ec11841b`), which was the first thing checked because six patch-target files changed in the range. The first build failed loud with the four signature breaks above; after adapting the call sites, Release build clean, `ctest` **512/512**, `nm -D` **40** `Java_*` exports and **zero** C++-mangled, `NativeLibraryLoadSmokeTest` **3/3, 0 skipped** including the pin cross-check against the linked `build-info`. `ModelParametersTest` 73/73 with the five new CPU-offload cases. **Feature exposure decided deliberately, not by default:** of the 6 new flags, only `--n-cpu-ffn` is genuinely new upstream, and it was added as `ModelParameters.setCpuFfnLayers`; `setCpuMoeLayers` was added alongside it for `--n-cpu-moe`, which has existed upstream since **b6089** (#15077, `ec428b02c`) but had never been exposed here — b10649 only refactored its lambda onto the shared `llm_add_n_cpu_ffn_overrides` helper; the two `--spec-synth-*` flags were refused as upstream marks them "benchmarking only"; and the three `--video-*` flags were refused because `ContentPart` has no video input path, so they would be inert — the same dead-knob shape that forced the `withTfsZ`/`withPenalizeNl`/`withPenaltyPrompt` deprecations. Both refusals are recorded in `TODO.md` with their reasoning so they do not resurface as open questions. | diff --git a/llama-langchain4j/src/main/java/net/ladenthin/llama/langchain4j/LangChain4jMapping.java b/llama-langchain4j/src/main/java/net/ladenthin/llama/langchain4j/LangChain4jMapping.java index c137b51b1..550a6c2c3 100644 --- a/llama-langchain4j/src/main/java/net/ladenthin/llama/langchain4j/LangChain4jMapping.java +++ b/llama-langchain4j/src/main/java/net/ladenthin/llama/langchain4j/LangChain4jMapping.java @@ -96,7 +96,7 @@ static InferenceParameters toStreamingParameters(ChatRequest request) { InferenceParameters.empty().withMessagesJson(jllama.buildMessagesJson()); java.util.Optional toolsJson = jllama.buildToolsJson(); if (toolsJson.isPresent()) { - params = params.withToolsJson(toolsJson.get()).withUseChatTemplate(true); + params = params.withToolsJson(toolsJson.get()); java.util.Optional toolChoice = jllama.getToolChoice(); if (toolChoice.isPresent()) { params = params.withToolChoice(toolChoice.get()); diff --git a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java index 0ad98b1ec..20860efbb 100644 --- a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java +++ b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java @@ -45,11 +45,17 @@ void chatReturnsAssistantText() { chat.chat( ChatRequest.builder() .messages(UserMessage.from("Reply with the single word: ok")) - .maxOutputTokens(8) + // Same budget as the streaming sibling, for the same reason: + // Qwen3-0.6B spends ~200 tokens inside first, so a small + // budget yields an EMPTY assistant text -- which a bare + // notNullValue() assertion would happily accept. + .maxOutputTokens(320) .build()); assertThat(response.aiMessage(), is(notNullValue())); assertThat(response.aiMessage().text(), is(notNullValue())); + assertThat("the model must produce assistant text, not only a thinking block", + response.aiMessage().text().trim().isEmpty(), is(false)); } } diff --git a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/TestModelPathsTest.java b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/TestModelPathsTest.java index fbb79c5b3..56e689774 100644 --- a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/TestModelPathsTest.java +++ b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/TestModelPathsTest.java @@ -8,6 +8,7 @@ import static org.hamcrest.Matchers.is; import static org.hamcrest.Matchers.notNullValue; import static org.hamcrest.Matchers.nullValue; +import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; import java.io.IOException; @@ -64,6 +65,18 @@ void anUnresolvablePathComesBackUnchangedSoTheSkipMessageNamesIt() { assertThat(resolved, is(Paths.get(wanted))); } + @Test + void aPathOnlyPresentFromTheReactorRootResolvesToAnExistingFile() { + // The branch this resolver exists for, and the only one the other cases never reach: + // Surefire's working directory is this module's basedir, so "llama-langchain4j/pom.xml" + // exists only when looked up from the parent. Exactly the models/ situation. + Path resolved = TestModelPaths.resolve("llama-langchain4j/pom.xml"); + assertThat(resolved, is(notNullValue())); + assertThat("a path that only exists from the reactor root must resolve: " + resolved, + Files.exists(resolved), is(true)); + assertThat(resolved.isAbsolute(), is(true)); + } + @Test void fromPropertyIsNullWhenThePropertyIsUnset() { assertThat(TestModelPaths.fromProperty("net.ladenthin.llama.langchain4j.definitely.unset"), is(nullValue())); @@ -86,13 +99,14 @@ void noTestReadsAModelPropertyWithoutTheResolver() throws IOException { .filter(f -> !f.getFileName().toString().equals("TestModelPaths.java")) .filter(f -> !f.getFileName().toString().equals("TestModelPathsTest.java")) .collect(Collectors.toList()); + assertFalse(javaFiles.isEmpty(), "the scan matched no sources under " + testSources.toAbsolutePath()); for (Path file : javaFiles) { - List lines = Files.readAllLines(file, StandardCharsets.UTF_8); - for (int i = 0; i < lines.size(); i++) { - if (lines.get(i).contains("System.getProperty(\"net.ladenthin.llama")) { - offenders.add(file + ":" + (i + 1) + " " + lines.get(i).trim()); - } - } + String source = new String(Files.readAllBytes(file), StandardCharsets.UTF_8); + // Inspect each call's own argument list rather than the surrounding line: a property + // is normally named through a constant, either form may be wrapped across lines, and + // scoping to the arguments keeps a neighbouring comment mentioning a + // net.ladenthin.llama class from reading as a violation. + offenders.addAll(rawModelPropertyReads(file, source)); } } @@ -102,4 +116,49 @@ void noTestReadsAModelPropertyWithoutTheResolver() throws IOException { + "- a bare read resolves against Surefire's module-basedir CWD and silently self-skips " + "in CI. Offending sites:\n" + String.join("\n", offenders)); } + + /** + * Finds every {@code System.getProperty(...)} / {@code System.getenv(...)} call in {@code source} + * whose own argument list names a {@code net.ladenthin.llama.*} property, either as a literal or + * through a {@code PROP_*} constant. + * + * @param file the file being scanned, used only to label a finding + * @param source the file's full text + * @return one entry per offending call site; empty when the file is clean + */ + private static List rawModelPropertyReads(Path file, String source) { + List found = new ArrayList<>(); + for (String call : new String[] {"System.getProperty(", "System.getenv("}) { + int from = 0; + while (true) { + int start = source.indexOf(call, from); + if (start < 0) { + break; + } + int open = start + call.length() - 1; + int depth = 0; + int end = open; + while (end < source.length()) { + char c = source.charAt(end); + if (c == '(') { + depth++; + } else if (c == ')') { + depth--; + if (depth == 0) { + break; + } + } + end++; + } + String arguments = source.substring(open, Math.min(end + 1, source.length())); + if (arguments.contains("net.ladenthin.llama") || arguments.contains("PROP_")) { + found.add(file + " " + source.substring(start, Math.min(end + 1, source.length())) + .replaceAll("\\s+", " ")); + } + from = start + call.length(); + } + } + return found; + } + } diff --git a/llama/src/main/cpp/jllama.cpp b/llama/src/main/cpp/jllama.cpp index f9aedef01..3edd0c790 100644 --- a/llama/src/main/cpp/jllama.cpp +++ b/llama/src/main/cpp/jllama.cpp @@ -256,6 +256,33 @@ static void populate_completion_task(server_task &task, jllama_context *jctx, configure_task_slot_impl(task, data); } +// Wake the queue before posting, then post. +// +// With ModelParameters.setSleepIdleSeconds(> 0) the queue enters a sleeping state after the +// configured idle period, and server_queue::post() does NOT leave it: post() only calls +// condition_tasks.notify_one(), while the sleeping wait predicate is +// `(!running || req_stop_sleeping)`. The loop wakes, re-tests, and goes straight back to +// sleep with the task still queued. Only wait_until_no_sleep() sets req_stop_sleeping. +// +// Upstream does this wake for the caller: server_res_generator's constructor calls +// queue_tasks.wait_until_no_sleep() for every route built with create_response(). Exactly +// four routes opt out via create_response(true) -- health, metrics, props and models -- and +// each answers from cache instead of waiting. This layer builds its readers with +// server_context::get_response_reader(), the CLI-facing accessor, which performs no wake -- +// so the wake has to happen here or not at all. +// +// Used at every post site in this file. Sleeping is off by default (-1), in which case +// wait_until_no_sleep() sees a non-sleeping queue and returns immediately. +static void wake_and_post(server_response_reader &rd, server_task &&task, bool front = false) { + rd.queue_tasks.wait_until_no_sleep(); + rd.post_task(std::move(task), front); +} + +static void wake_and_post(server_response_reader &rd, std::vector &&tasks) { + rd.queue_tasks.wait_until_no_sleep(); + rd.post_tasks(std::move(tasks)); +} + [[nodiscard]] static jint dispatch_streaming_completion(JNIEnv *env, jllama_context *jctx, const json &data, server_task_type task_type, task_response_type res_type, std::vector files = {}) { @@ -268,7 +295,7 @@ static void populate_completion_task(server_task &task, jllama_context *jctx, task.id = tid; populate_completion_task(task, jctx, meta.logit_bias_eog, data, meta.has_mtmd, std::move(files)); task.params.res_type = res_type; - rd->post_task(std::move(task)); + wake_and_post(*rd, std::move(task)); } catch (const std::exception &e) { delete rd; throw_invalid_request(env, e); @@ -301,7 +328,7 @@ static void populate_completion_task(server_task &task, jllama_context *jctx, return nullptr; } task.params.res_type = res_type; - rd.post_task(std::move(task)); + wake_and_post(rd, std::move(task)); auto br = rd.wait_for_all([jctx] { return jctx->closing.load(); }); if (!batch_ok_or_throw(env, br)) return nullptr; @@ -355,18 +382,19 @@ static void populate_completion_task(server_task &task, jllama_context *jctx, // introspection endpoints, and waited on with the same `closing` predicate every other // wait site in this file uses, so close() can unblock a pending call. // -// A task posted just as the queue enters its idle-sleep state is never processed -- -// upstream's own /metrics handler says so verbatim ("a task posted right before sleeping -// is never processed, do not wait for it") and guards it by adding queue_tasks.is_sleeping() -// to its wait predicate. We mirror that predicate exactly: server_context does not expose -// the getter, but the reader's `queue_tasks` member and server_queue::is_sleeping() are both -// public, so this layer can and does ask. Without it a call made while the queue is asleep -// blocks until close(), because post_task() does not wake a sleeping queue. Reaching that -// state needs ModelParameters.setSleepIdleSeconds(> 0), which is off by default (-1). +// wake_and_post() has already left any idle-sleep state, so the task is guaranteed to be +// seen by a running loop. The is_sleeping() disjunct below is only a backstop for the +// narrow window in which the queue could fall asleep again between the wake and this wait +// -- it takes a full idle period (>= 1s) of quiet to re-enter, so it needs the calling +// thread to be descheduled for longer than that. Should it ever happen, the result is a +// thrown "No result" rather than a block until close(); upstream's /metrics handler adds +// the same is_sleeping() guard for the same reason ("a task posted right before sleeping +// is never processed, do not wait for it"), differing only in that it can fall back to a +// cached response where this has nothing to fall back to. [[nodiscard]] static server_task_result_ptr post_and_wait(JNIEnv *env, jllama_context *jctx, server_task task) { auto rd = jctx->server.get_response_reader(); task.id = rd.get_new_id(); - rd.post_task(std::move(task), true); + wake_and_post(rd, std::move(task), true); auto result = rd.next([jctx, &rd] { return jctx->closing.load() || rd.queue_tasks.is_sleeping(); }); if (!result_ok_or_throw(env, result)) return nullptr; @@ -1139,7 +1167,7 @@ JNIEXPORT jfloatArray JNICALL Java_net_ladenthin_llama_LlamaModel_embed(JNIEnv * task.id = rd.get_new_id(); task.tokens = server_tokens(tokens, false); task.index = 0; - rd.post_task(std::move(task)); + wake_and_post(rd, std::move(task)); auto br = rd.wait_for_all([jctx] { return jctx->closing.load(); }); if (!batch_ok_or_throw(env, br)) @@ -1192,7 +1220,7 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleRerank(JNIEn mtmd_helper_init_opt_default()), static_cast(i), TASK_RESPONSE_TYPE_NONE)); } - rd.post_tasks(std::move(tasks)); + wake_and_post(rd, std::move(tasks)); auto br = rd.wait_for_all([jctx] { return jctx->closing.load(); }); if (!batch_ok_or_throw(env, br)) @@ -1579,7 +1607,7 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleEmbeddings(J server_tokens(tokenized_prompts[i].get_tokens(), false), static_cast(i), res_type)); } - rd.post_tasks(std::move(tasks)); + wake_and_post(rd, std::move(tasks)); auto br = rd.wait_for_all([jctx] { return jctx->closing.load(); }); if (!batch_ok_or_throw(env, br)) diff --git a/llama/src/main/cpp/train_engine.cpp b/llama/src/main/cpp/train_engine.cpp index fb1b5be93..9dcd9820e 100644 --- a/llama/src/main/cpp/train_engine.cpp +++ b/llama/src/main/cpp/train_engine.cpp @@ -5,9 +5,9 @@ #include "train_engine.h" #include "common.h" -#include "cpu_params.hpp" #include "ggml-opt.h" #include "llama.h" +#include "train_params.hpp" #include @@ -24,27 +24,7 @@ using json = nlohmann::json; namespace jllama_train { bool finetune(const finetune_config &cfg, std::string &err) { - common_params params; - params.escape = false; - params.model.path = cfg.model_path; - params.out_file = cfg.output_path; - params.n_ctx = cfg.n_ctx; - params.n_gpu_layers = cfg.n_gpu_layers; - params.val_split = cfg.val_split; - if (cfg.n_batch > 0) { - params.n_batch = cfg.n_batch; - } - if (cfg.n_ubatch > 0) { - params.n_ubatch = cfg.n_ubatch; - } - - params.optimizer = cfg.optimizer == 1 ? GGML_OPT_OPTIMIZER_TYPE_SGD : GGML_OPT_OPTIMIZER_TYPE_ADAMW; - params.lr.lr0 = cfg.learning_rate; - params.lr.lr_min = cfg.lr_min; - params.lr.decay_epochs = cfg.decay_epochs; - params.lr.wd = cfg.weight_decay; - params.lr.epochs = static_cast(cfg.epochs > 0 ? cfg.epochs : 1); - params.lr.init(); // required after setting lr fields, before the optimizer reads get_lr() + common_params params = build_train_params(cfg); // The corpus is either read from a file or supplied inline. if (!cfg.training_file.empty()) { @@ -58,19 +38,6 @@ bool finetune(const finetune_config &cfg, std::string &err) { params.prompt = cfg.training_text; } - // Training needs writable weights (mmap yields read-only pointers) and an f32 KV cache - // (OUT_PROD has no f16 support) — same forced settings as upstream finetune.cpp. - // b10107 replaced the use_mmap/use_mlock/use_direct_io booleans with a single load_mode - // enum; LLAMA_LOAD_MODE_NONE disables mmap so the weight pointers stay writable. - params.load_mode = LLAMA_LOAD_MODE_NONE; - params.cache_type_k = GGML_TYPE_F32; - params.cache_type_v = GGML_TYPE_F32; - - // A hand-built common_params never passes through common_params_parse, so its CPU fields are - // still unresolved here; without this the process dies on a memset of NULL inside - // ggml_threadpool_new. Shared with tts_params.hpp -- see cpu_params.hpp for the mechanism. - jllama::resolve_cpu_params(params); - llama_backend_init(); llama_numa_init(params.numa); diff --git a/llama/src/main/cpp/train_params.hpp b/llama/src/main/cpp/train_params.hpp new file mode 100644 index 000000000..1e411c0ed --- /dev/null +++ b/llama/src/main/cpp/train_params.hpp @@ -0,0 +1,78 @@ +// SPDX-FileCopyrightText: 2026 Bernard Ladenthin +// +// SPDX-License-Identifier: MIT +// +// The common_params builder for the fine-tuning engine, split out of train_engine.cpp so it can be +// tested without a GGUF, a JVM or a loaded model -- the sibling of tts_params.hpp, and for the same +// reason. +// +// It is a header, not a .cpp, because jllama_test deliberately does not compile train_engine.cpp +// (that TU needs the llama runtime); a header lets the test exercise the real builder the engine +// uses, rather than a copy that could drift away from it. Before this split there was NO runnable +// guard on the trainer's jllama::resolve_cpu_params() call at all: train_engine.cpp is in the +// jllama target only, and LlamaTrainerIntegrationTest is gated on a system property no CI job sets +// and a model in no models.csv row -- so the JVM-abort documented in cpu_params.hpp could have +// regressed here, on every platform, unseen. +// +// Deliberately excluded: reading the training corpus. That does file I/O and can fail, so it stays +// in finetune(), which owns the error string. + +#ifndef JLLAMA_TRAIN_PARAMS_HPP +#define JLLAMA_TRAIN_PARAMS_HPP + +#include "common.h" +#include "cpu_params.hpp" +#include "ggml-opt.h" +#include "llama.h" +#include "train_engine.h" + +namespace jllama_train { + +// --------------------------------------------------------------------------- +// Maps one finetune_config onto the common_params the training run is driven with. +// +// The jllama::resolve_cpu_params() call is the load-bearing part -- without it this hand-built +// common_params reaches ggml_threadpool_new with n_threads == -1 and the process dies on a +// memset of NULL. See cpu_params.hpp for the full mechanism. +// --------------------------------------------------------------------------- +[[nodiscard]] inline common_params build_train_params(const finetune_config &cfg) { + common_params params; + params.escape = false; + params.model.path = cfg.model_path; + params.out_file = cfg.output_path; + params.n_ctx = cfg.n_ctx; + params.n_gpu_layers = cfg.n_gpu_layers; + params.val_split = cfg.val_split; + if (cfg.n_batch > 0) { + params.n_batch = cfg.n_batch; + } + if (cfg.n_ubatch > 0) { + params.n_ubatch = cfg.n_ubatch; + } + + params.optimizer = cfg.optimizer == 1 ? GGML_OPT_OPTIMIZER_TYPE_SGD : GGML_OPT_OPTIMIZER_TYPE_ADAMW; + params.lr.lr0 = cfg.learning_rate; + params.lr.lr_min = cfg.lr_min; + params.lr.decay_epochs = cfg.decay_epochs; + params.lr.wd = cfg.weight_decay; + params.lr.epochs = static_cast(cfg.epochs > 0 ? cfg.epochs : 1); + params.lr.init(); // required after setting lr fields, before the optimizer reads get_lr() + + // Training needs writable weights (mmap yields read-only pointers) and an f32 KV cache + // (OUT_PROD has no f16 support) — same forced settings as upstream finetune.cpp. + // b10107 replaced the use_mmap/use_mlock/use_direct_io booleans with a single load_mode + // enum; LLAMA_LOAD_MODE_NONE disables mmap so the weight pointers stay writable. + params.load_mode = LLAMA_LOAD_MODE_NONE; + params.cache_type_k = GGML_TYPE_F32; + params.cache_type_v = GGML_TYPE_F32; + + // A hand-built common_params never passes through common_params_parse, so its CPU fields are + // still unresolved here. Shared with tts_params.hpp -- see cpu_params.hpp for the mechanism. + jllama::resolve_cpu_params(params); + + return params; +} + +} // namespace jllama_train + +#endif // JLLAMA_TRAIN_PARAMS_HPP diff --git a/llama/src/main/java/net/ladenthin/llama/LlamaModel.java b/llama/src/main/java/net/ladenthin/llama/LlamaModel.java index 2c5714291..5a1608c4d 100644 --- a/llama/src/main/java/net/ladenthin/llama/LlamaModel.java +++ b/llama/src/main/java/net/ladenthin/llama/LlamaModel.java @@ -635,7 +635,7 @@ public ChatResponse chat(ChatRequest request) { InferenceParameters params = InferenceParameters.empty().withMessagesJson(request.buildMessagesJson()); Optional toolsJsonOpt = request.buildToolsJson(); if (toolsJsonOpt.isPresent()) { - params = params.withToolsJson(toolsJsonOpt.get()).withUseChatTemplate(true); + params = params.withToolsJson(toolsJsonOpt.get()); Optional toolChoice = request.getToolChoice(); if (toolChoice.isPresent()) { params = params.withToolChoice(toolChoice.get()); @@ -809,7 +809,7 @@ public void streamChatCompletion(InferenceParameters parameters, ConsumerIt is not an atomic snapshot. The two halves are separate round trips * through the task queue, so the counters and the slot array can be a moment apart. * Upstream's own two endpoints are no more atomic than this. - *
  • It defers idle-sleep. llama.cpp b10644 stopped letting a {@code /metrics} + *
  • It defers idle-sleep. llama.cpp b10519 stopped letting a {@code /metrics} * scrape reset the idle timer, but the slot half is not exempt, so calling this method * still counts as activity. Polling it as a health check keeps a server configured with * {@link net.ladenthin.llama.parameters.ModelParameters#setSleepIdleSeconds(int)} awake @@ -956,12 +956,12 @@ public String eraseSlot(int slotId) { *

    The file format is versioned by the linked llama.cpp build, not by this library. * llama.cpp stamps every state file with {@code LLAMA_STATE_SEQ_VERSION} and rejects one written * under a different value, so a file saved by a jar built against a different - * {@link net.ladenthin.llama.value.LlamaCppVersion#LLAMA_CPP_VERSION} may not load — b10644 bumped + * {@link net.ladenthin.llama.value.LlamaCppVersion#LLAMA_CPP_VERSION} may not load — b10642 bumped * that constant 2 → 3, invalidating every file written by an earlier release. Treat * these files as a cache to regenerate on upgrade, never as durable storage. A rejected file - * surfaces as a {@link net.ladenthin.llama.exception.LlamaException} carrying upstream's message - * {@code "No available space in KV cache or invalid slot save file"}, which does not name the - * version mismatch as the cause.

    + * surfaces as a {@link net.ladenthin.llama.exception.LlamaException} whose message is upstream's + * wrapped form, {@code "Unable to restore slot: No available space in KV cache or invalid slot + * save file"} — it does not name the version mismatch as the cause.

    * * @param slotId the slot ID to save * @param filepath the file path to save to diff --git a/llama/src/main/java/net/ladenthin/llama/args/ReasoningFormat.java b/llama/src/main/java/net/ladenthin/llama/args/ReasoningFormat.java index d46791fd4..3d26026a7 100644 --- a/llama/src/main/java/net/ladenthin/llama/args/ReasoningFormat.java +++ b/llama/src/main/java/net/ladenthin/llama/args/ReasoningFormat.java @@ -11,7 +11,7 @@ * *

    Passed as {@code "reasoning_format"} in inference requests. Only meaningful when the model * uses a thinking tag (e.g. {@code ...}) and chat-template rendering is active - * ({@link net.ladenthin.llama.parameters.InferenceParameters#withUseChatTemplate(boolean)}). + * ({@link net.ladenthin.llama.parameters.ModelParameters#enableJinja()}, a load-time option). */ public enum ReasoningFormat implements CliArg { diff --git a/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java b/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java index e0d94efd4..49f1fa419 100644 --- a/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java +++ b/llama/src/main/java/net/ladenthin/llama/parameters/InferenceParameters.java @@ -92,7 +92,6 @@ public final class InferenceParameters extends JsonParameters { private static final String PARAM_STOP = "stop"; private static final String PARAM_SAMPLERS = "samplers"; private static final String PARAM_STREAM = "stream"; - private static final String PARAM_USE_CHAT_TEMPLATE = "use_chat_template"; private static final String PARAM_CHAT_TEMPLATE = "chat_template"; private static final String PARAM_USE_JINJA = "use_jinja"; private static final String PARAM_CHAT_TEMPLATE_KWARGS = "chat_template_kwargs"; @@ -680,9 +679,21 @@ public InferenceParameters withSamplers(Sampler... samplers) { /** * Returns a new request with the chat-template flag replaced. * + *

    Ignored by the server. Jinja templating is a launch-time setting, + * not a per-request one: {@code common_params::use_jinja} is set only by the {@code --jinja} / + * {@code --no-jinja} arguments, and the string {@code "use_jinja"} appears nowhere in + * {@code common/} or {@code tools/server/} as a request key on the pinned build. The request + * schema silently discards unknown fields, so this neither enables nor disables anything. + * Use {@link net.ladenthin.llama.parameters.ModelParameters#enableJinja()} when loading the + * model instead. Retained only so existing call sites keep compiling; it will be removed in a + * future release.

    + * * @param useChatTemplate whether to apply a chat template * @return a new instance; this instance is unchanged + * @deprecated jinja is a load-time option; the request field is discarded by the server. Use + * {@link net.ladenthin.llama.parameters.ModelParameters#enableJinja()} */ + @Deprecated public InferenceParameters withUseChatTemplate(boolean useChatTemplate) { return withScalar(PARAM_USE_JINJA, useChatTemplate); } @@ -690,9 +701,21 @@ public InferenceParameters withUseChatTemplate(boolean useChatTemplate) { /** * Returns a new request with the chat-template string replaced. * + *

    Ignored by the server. The chat template is chosen when the model is + * loaded, not per request: on the pinned build the only {@code "chat_template"} string in + * {@code common/} or {@code tools/server/} is the one the server emits in its + * {@code /props} response, and nothing reads it from a request body. The request schema + * silently discards unknown fields, so a template passed here is never applied. Use + * {@link net.ladenthin.llama.parameters.ModelParameters#setChatTemplate(String)} instead. + * Retained only so existing call sites keep compiling; it will be removed in a future + * release.

    + * * @param chatTemplate the Jinja-style chat template to use; {@code null} clears * @return a new instance; this instance is unchanged + * @deprecated the chat template is a load-time option; the request field is discarded by the + * server. Use {@link net.ladenthin.llama.parameters.ModelParameters#setChatTemplate(String)} */ + @Deprecated public InferenceParameters withChatTemplate(@Nullable String chatTemplate) { return withOptionalJson(PARAM_CHAT_TEMPLATE, chatTemplate); } diff --git a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java index daeec0f3e..0f95ea621 100644 --- a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java +++ b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java @@ -884,7 +884,8 @@ public ModelParameters setDevices(String devices) { * which dominate a MoE model's size — so attention and the rest of each layer stay on the GPU. * That usually fits a far larger model in the same VRAM at a much smaller speed cost than * reducing the layer count would. Upstream {@code --n-cpu-moe} / {@code -ncmoe}, added in - * llama.cpp b10649; use {@link #setCpuFfnLayers(int)} for a dense model.

    + * llama.cpp b6089 but never exposed here until now; use {@link #setCpuFfnLayers(int)} for a + * dense model.

    * * @param layers the number of leading layers whose MoE expert weights stay on the CPU * @return this builder @@ -1420,10 +1421,11 @@ public ModelParameters setMmprojAuto(boolean enabled) { */ public ModelParameters setMmprojDevice(String device) { parameters.put("--mmproj-device", device); - // --mmproj-device and --no-mmproj-offload write the same upstream field - // (common_params::mmproj_use_gpu). The rendered argv comes out of a HashMap, so if both were - // present the winner would be hash order -- unspecified. Clear the conflicting flag the way - // setMmprojOffload clears its opposite, so an explicit device always wins. + // --mmproj-device and --{no-,}mmproj-offload write the same upstream field + // (common_params::mmproj_use_gpu), so upstream resolves a clash by argv order. Our argv is + // rendered from a HashMap, where that order is unspecified -- so the pair must never both be + // present. setMmprojOffload clears this key in turn, making the rule simply "the last of the + // two calls wins". clearFlag(ModelFlag.NO_MMPROJ_OFFLOAD); clearFlag(ModelFlag.MMPROJ_OFFLOAD); return this; @@ -1442,12 +1444,20 @@ public ModelParameters enableMmprojOffload() { * Enable or disable GPU offload for the multimodal projector. This is independent of * {@link #setGpuLayers(int)} because upstream enables projector offload by default. * + *

    Clears any device previously named by {@link #setMmprojDevice(String)}: both write + * upstream's single {@code mmproj_use_gpu} field, so only one of them may appear in the + * rendered argv. Between the two, the last call wins — call {@code setMmprojDevice} + * afterwards to pin a device again.

    + * * @param enabled {@code true} to offload the projector, {@code false} to keep it on CPU * @return this builder */ public ModelParameters setMmprojOffload(boolean enabled) { setFlag(enabled ? ModelFlag.MMPROJ_OFFLOAD : ModelFlag.NO_MMPROJ_OFFLOAD); clearFlag(enabled ? ModelFlag.NO_MMPROJ_OFFLOAD : ModelFlag.MMPROJ_OFFLOAD); + // Also drop any --mmproj-device: it writes the same upstream field, and with a HashMap- + // rendered argv the winner between the two would be hash order. See setMmprojDevice. + parameters.remove("--mmproj-device"); return this; } diff --git a/llama/src/main/java/net/ladenthin/llama/server/OpenAiRequestMapper.java b/llama/src/main/java/net/ladenthin/llama/server/OpenAiRequestMapper.java index 6db868a0d..5bddfea81 100644 --- a/llama/src/main/java/net/ladenthin/llama/server/OpenAiRequestMapper.java +++ b/llama/src/main/java/net/ladenthin/llama/server/OpenAiRequestMapper.java @@ -53,7 +53,7 @@ InferenceParameters toInferenceParameters(JsonNode request) { // Tools are chat-only. JsonNode tools = request.path("tools"); if (tools.isArray() && tools.size() > 0) { - params = params.withToolsJson(tools.toString()).withUseChatTemplate(true); + params = params.withToolsJson(tools.toString()); JsonNode toolChoice = request.path("tool_choice"); if (toolChoice.isTextual()) { params = params.withToolChoice(toolChoice.asText()); diff --git a/llama/src/test/cpp/test_tts_params.cpp b/llama/src/test/cpp/test_tts_params.cpp index e602d0459..f5f0b480f 100644 --- a/llama/src/test/cpp/test_tts_params.cpp +++ b/llama/src/test/cpp/test_tts_params.cpp @@ -12,6 +12,7 @@ // (hs_err retrieval, cross-platform frame comparison, disassembly) to identify. #include "cpu_params.hpp" +#include "train_params.hpp" #include "tts_params.hpp" #include @@ -84,12 +85,12 @@ TEST(CommonParamsCpuTrap, RoleModelMakesBatchInheritTheMainCount) { // ============================================================ // jllama::resolve_cpu_params — the shared guard for every hand-built common_params // -// tts_params.hpp and train_engine.cpp both assemble a common_params by hand and both must route -// it through this one resolver. The TtsParams tests above cover the TTS path; these cover the -// resolver directly, so the trainer — whose only integration test needs a GGUF that no CI job -// downloads (net.ladenthin.llama.train.model is set by no workflow) — is guarded too. Delete -// either call site's resolve_cpu_params() and the JVM dies on memset(NULL, 0, huge) with no -// hs_err_pid log; these tests are the cheap, model-free way to notice. +// tts_params.hpp and train_params.hpp both assemble a common_params by hand and both must route +// it through this one resolver. Delete either call site's resolve_cpu_params() and the JVM dies +// on memset(NULL, 0, huge) with no hs_err_pid log; these tests are the cheap, model-free way to +// notice. Each builder has its own guard above/below, because testing the resolver alone does +// NOT cover the call sites: train_engine.cpp is compiled into jllama only, and its integration +// test is gated on net.ladenthin.llama.train.model, which no CI job sets. // ============================================================ TEST(ResolveCpuParams, LeavesBothThreadCountsUsable) { @@ -106,27 +107,107 @@ TEST(ResolveCpuParams, LeavesBothThreadCountsUsable) { TEST(ResolveCpuParams, BatchInheritsTheMainCountRatherThanResolvingAlone) { // The role_model argument on the second postprocess_cpu_params call is the whole point: the // batch pool must inherit, or common_threadpools::init sees a mismatch and builds a second pool. + // + // The requested count is derived from the host rather than hardcoded: dropping role_model makes + // the batch pool resolve to the host's own core count, so a fixture that happened to equal that + // count (a 3-core runner, a 3-CPU container) would pass with the bug present. + common_params probe; + jllama::resolve_cpu_params(probe); + const int host_default = probe.cpuparams.n_threads; + ASSERT_GT(host_default, 0); + const int requested = host_default + 1; + common_params params; - params.cpuparams.n_threads = 3; + params.cpuparams.n_threads = requested; jllama::resolve_cpu_params(params); - EXPECT_EQ(params.cpuparams.n_threads, 3); - EXPECT_EQ(params.cpuparams_batch.n_threads, 3); + EXPECT_EQ(params.cpuparams.n_threads, requested); + EXPECT_EQ(params.cpuparams_batch.n_threads, requested); + EXPECT_NE(params.cpuparams_batch.n_threads, host_default) << "batch pool resolved alone instead of inheriting"; } -TEST(ResolveCpuParams, IsIdempotent) { - // Called twice (e.g. a future refactor routing both a builder and its caller through it) must - // not drift the counts. - common_params params; - params.cpuparams.n_threads = 5; +// ============================================================ +// build_train_params — the trainer's builder +// +// The sibling of the TtsParams block. This is the only runnable guard on the trainer's +// resolve_cpu_params() call: train_engine.cpp is in the jllama target, not jllama_test, and +// LlamaTrainerIntegrationTest self-skips on every platform because its model is in no +// models.csv row. +// ============================================================ - jllama::resolve_cpu_params(params); - const int once_main = params.cpuparams.n_threads; - const int once_batch = params.cpuparams_batch.n_threads; +namespace { - jllama::resolve_cpu_params(params); +jllama_train::finetune_config minimal_train_config() { + jllama_train::finetune_config cfg{}; + cfg.model_path = "base.gguf"; + cfg.output_path = "tuned.gguf"; + cfg.epochs = 2; + cfg.learning_rate = 1e-5f; + cfg.lr_min = -1.0f; + cfg.decay_epochs = 0.0f; + cfg.weight_decay = 0.0f; + cfg.optimizer = 0; + cfg.n_ctx = 512; + cfg.n_gpu_layers = 0; + cfg.val_split = 0.0f; + return cfg; +} + +} // namespace + +TEST(TrainParams, ResolvesBothCpuThreadCounts) { + const common_params params = jllama_train::build_train_params(minimal_train_config()); + + // The one that crashed the JVM: left at -1, ggml_threadpool_new memsets NULL. + EXPECT_GT(params.cpuparams.n_threads, 0); + EXPECT_GT(params.cpuparams_batch.n_threads, 0); + EXPECT_EQ(params.cpuparams_batch.n_threads, params.cpuparams.n_threads); +} + +TEST(TrainParams, ForcesTheSettingsTrainingRequires) { + const common_params params = jllama_train::build_train_params(minimal_train_config()); + + // Weights must stay writable (mmap yields read-only pointers) and the KV cache must be f32 + // (OUT_PROD has no f16 support). Losing either turns into a runtime failure deep in ggml-opt. + EXPECT_EQ(params.load_mode, LLAMA_LOAD_MODE_NONE); + EXPECT_EQ(params.cache_type_k, GGML_TYPE_F32); + EXPECT_EQ(params.cache_type_v, GGML_TYPE_F32); + EXPECT_FALSE(params.escape); +} + +TEST(TrainParams, MapsTheConfigOntoTheOptimizerFields) { + jllama_train::finetune_config cfg = minimal_train_config(); + cfg.optimizer = 1; + cfg.epochs = 7; + cfg.n_batch = 64; + cfg.n_ubatch = 16; + + const common_params params = jllama_train::build_train_params(cfg); + + EXPECT_EQ(params.optimizer, GGML_OPT_OPTIMIZER_TYPE_SGD); + EXPECT_EQ(params.lr.epochs, 7u); + EXPECT_EQ(params.n_batch, 64); + EXPECT_EQ(params.n_ubatch, 16); + EXPECT_EQ(params.model.path, "base.gguf"); + EXPECT_EQ(params.out_file, "tuned.gguf"); +} + +TEST(TrainParams, NonPositiveBatchSizesKeepTheNativeDefaults) { + const common_params defaults{}; + jllama_train::finetune_config cfg = minimal_train_config(); + cfg.n_batch = 0; + cfg.n_ubatch = 0; + + const common_params params = jllama_train::build_train_params(cfg); + + EXPECT_EQ(params.n_batch, defaults.n_batch); + EXPECT_EQ(params.n_ubatch, defaults.n_ubatch); +} + +TEST(TrainParams, EpochsBelowOneAreClampedSoTheOptimizerRunsAtLeastOnce) { + jllama_train::finetune_config cfg = minimal_train_config(); + cfg.epochs = 0; - EXPECT_EQ(params.cpuparams.n_threads, once_main); - EXPECT_EQ(params.cpuparams_batch.n_threads, once_batch); + EXPECT_EQ(jllama_train::build_train_params(cfg).lr.epochs, 1u); } diff --git a/llama/src/test/java/net/ladenthin/llama/MultimodalIntegrationTest.java b/llama/src/test/java/net/ladenthin/llama/MultimodalIntegrationTest.java index 3707424d8..bebe952fa 100644 --- a/llama/src/test/java/net/ladenthin/llama/MultimodalIntegrationTest.java +++ b/llama/src/test/java/net/ladenthin/llama/MultimodalIntegrationTest.java @@ -125,6 +125,14 @@ public static void setup() { } model = new LlamaModel(parameters); assertTrue(model.getModelMeta().supportsVision(), "loaded model + mmproj must advertise vision input"); + // The native side must emit all three modality keys. ModelMetaTest feeds ModelMeta its own + // JSON, so nothing there would notice getModelMetaJson() dropping one; this is the only + // place the real emitter is observed. Presence, not value -- whether this particular model + // accepts video is the model's business. + JsonNode modalities = model.getModelMeta().asJson().path("modalities"); + for (String key : new String[] {"vision", "audio", "video"}) { + assertTrue(modalities.has(key), "getModelMetaJson() must emit modalities." + key + ": " + modalities); + } } @AfterAll diff --git a/llama/src/test/java/net/ladenthin/llama/TestConstantsTest.java b/llama/src/test/java/net/ladenthin/llama/TestConstantsTest.java index 3f2f41e20..88ef8bc5b 100644 --- a/llama/src/test/java/net/ladenthin/llama/TestConstantsTest.java +++ b/llama/src/test/java/net/ladenthin/llama/TestConstantsTest.java @@ -5,9 +5,9 @@ package net.ladenthin.llama; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNull; import static org.junit.jupiter.api.Assertions.assertTrue; -import static org.junit.jupiter.api.Assumptions.assumeTrue; import java.io.File; import java.nio.charset.StandardCharsets; @@ -120,22 +120,27 @@ public void theShippedModelConstantsGoThroughTheResolver() { @Test public void noTestReadsAModelPropertyWithoutTheResolver() throws Exception { Path testSources = Paths.get("src/test/java"); - assumeTrue(Files.isDirectory(testSources), "test sources not on disk: " + testSources.toAbsolutePath()); + // assert, not assume: a rule against silently skipping must not silently skip itself. + assertTrue(Files.isDirectory(testSources), "test sources not on disk: " + testSources.toAbsolutePath()); List offenders = new ArrayList<>(); try (Stream paths = Files.walk(testSources)) { List javaFiles = paths.filter(Files::isRegularFile) .filter(f -> f.getFileName().toString().endsWith(".java")) - // TestConstants itself is where the raw reads legitimately live. + // TestConstants itself is where the raw reads legitimately live, and this file + // necessarily contains the very text the rule searches for. .filter(f -> !f.getFileName().toString().equals("TestConstants.java")) + .filter(f -> !f.getFileName().toString().equals("TestConstantsTest.java")) .collect(Collectors.toList()); + assertFalse(javaFiles.isEmpty(), "the scan matched no sources under " + testSources.toAbsolutePath()); for (Path file : javaFiles) { - List lines = Files.readAllLines(file, StandardCharsets.UTF_8); - for (int i = 0; i < lines.size(); i++) { - if (lines.get(i).contains("System.getProperty(\"net.ladenthin.llama")) { - offenders.add(file + ":" + (i + 1) + " " + lines.get(i).trim()); - } - } + String source = new String(Files.readAllBytes(file), StandardCharsets.UTF_8); + // Inspect each call's own argument list rather than the surrounding line or + // statement: the house idiom names the property through a TestConstants.PROP_* + // constant, not a literal, and either form may be wrapped across lines. Scoping to + // the parenthesised arguments also keeps a neighbouring comment that happens to + // mention a net.ladenthin.llama class from reading as a violation. + offenders.addAll(rawModelPropertyReads(file, source)); } } @@ -147,34 +152,47 @@ public void noTestReadsAModelPropertyWithoutTheResolver() throws Exception { } /** - * Pins the resolver's parent-directory fallback without depending on a real GGUF. + * Finds every {@code System.getProperty(...)} / {@code System.getenv(...)} call in {@code source} + * whose own argument list names a {@code net.ladenthin.llama.*} property, either as a literal or + * through a {@code PROP_*} constant. * - *

    {@link #theShippedModelConstantsGoThroughTheResolver()} compares resolved-vs-resolved, so it - * cannot see a dropped wrapper in the module-relative layout (both sides return the same literal) - * nor in a checkout with no models at all. This drives the branch that actually matters: a file - * that exists ONLY one directory up — exactly the CI shape, where Surefire runs in {@code llama/} - * and the GGUF cache is restored to the reactor root. + * @param file the file being scanned, used only to label a finding + * @param source the file's full text + * @return one entry per offending call site; empty when the file is clean */ - @Test - public void resolverFindsAFileThatExistsOnlyInTheParentDirectory(@TempDir Path tmp) throws Exception { - Path moduleDir = Files.createDirectories(tmp.resolve("module")); - Path parentModels = Files.createDirectories(tmp.resolve("models")); - Path onlyInParent = Files.createFile(parentModels.resolve("only-in-parent.gguf")); - - String previousCwd = System.getProperty("user.dir"); - try { - System.setProperty("user.dir", moduleDir.toAbsolutePath().toString()); - // Relative resolution is CWD-sensitive; assert against the real file either way so the - // test states the contract rather than the JVM's cwd semantics. - String resolved = TestConstants.resolveModelPath( - tmp.relativize(onlyInParent).toString().replace(File.separatorChar, '/')); - assertTrue( - resolved.endsWith("only-in-parent.gguf"), - "resolver must still name the file it was given: " + resolved); - } finally { - if (previousCwd != null) { - System.setProperty("user.dir", previousCwd); + private static List rawModelPropertyReads(Path file, String source) { + List found = new ArrayList<>(); + for (String call : new String[] {"System.getProperty(", "System.getenv("}) { + int from = 0; + while (true) { + int start = source.indexOf(call, from); + if (start < 0) { + break; + } + int open = start + call.length() - 1; + int depth = 0; + int end = open; + while (end < source.length()) { + char c = source.charAt(end); + if (c == '(') { + depth++; + } else if (c == ')') { + depth--; + if (depth == 0) { + break; + } + } + end++; + } + String arguments = source.substring(open, Math.min(end + 1, source.length())); + if (arguments.contains("net.ladenthin.llama") || arguments.contains("PROP_")) { + found.add(file + " " + + source.substring(start, Math.min(end + 1, source.length())) + .replaceAll("\\s+", " ")); + } + from = start + call.length(); } } + return found; } } diff --git a/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java b/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java index e2309dd01..846eba3be 100644 --- a/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java +++ b/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java @@ -84,7 +84,8 @@ public void testSetPriorityBatchTooHigh() { } // ------------------------------------------------------------------------- - // setCpuMoeLayers / setCpuFfnLayers — the CPU-offload pair added in llama.cpp b10649 + // setCpuMoeLayers / setCpuFfnLayers — the CPU-offload pair. --n-cpu-ffn is new in llama.cpp + // b10649; --n-cpu-moe has existed upstream since b6089 but was never exposed here until now. // ------------------------------------------------------------------------- @Test @@ -503,6 +504,28 @@ public void testSetMmprojDeviceNoneIsPassedThroughVerbatim() { assertThat(p.parameters, not(hasKey("--no-mmproj-offload"))); } + @Test + public void testMmprojDeviceAndOffloadClearEachOtherInBothDirections() { + // Both write upstream's single mmproj_use_gpu field, and our argv is rendered from a HashMap + // whose iteration order is unspecified -- so if both keys were present the winner would be + // hash order. The contract is "the last of the two calls wins". + ModelParameters deviceLast = + new ModelParameters().setMmprojOffload(false).setMmprojDevice("CUDA1"); + assertThat(deviceLast.parameters.get("--mmproj-device"), is("CUDA1")); + assertThat(deviceLast.parameters, not(hasKey("--no-mmproj-offload"))); + assertThat(deviceLast.parameters, not(hasKey("--mmproj-offload"))); + + ModelParameters offloadLast = + new ModelParameters().setMmprojDevice("CUDA1").setMmprojOffload(false); + assertThat(offloadLast.parameters, hasKey("--no-mmproj-offload")); + assertThat(offloadLast.parameters, not(hasKey("--mmproj-device"))); + + ModelParameters enabledLast = + new ModelParameters().setMmprojDevice("CUDA1").setMmprojOffload(true); + assertThat(enabledLast.parameters, hasKey("--mmproj-offload")); + assertThat(enabledLast.parameters, not(hasKey("--mmproj-device"))); + } + @Test public void testSetMmprojDeviceIsIndependentOfTheMainModelDevices() { ModelParameters p = new ModelParameters().setDevices("CUDA0").setMmprojDevice("CUDA1"); diff --git a/llama/src/test/java/net/ladenthin/llama/server/OpenAiCompatServerIntegrationTest.java b/llama/src/test/java/net/ladenthin/llama/server/OpenAiCompatServerIntegrationTest.java index 645bfd03d..53c4360c1 100644 --- a/llama/src/test/java/net/ladenthin/llama/server/OpenAiCompatServerIntegrationTest.java +++ b/llama/src/test/java/net/ladenthin/llama/server/OpenAiCompatServerIntegrationTest.java @@ -96,9 +96,11 @@ public void streamingChatEmitsChunksAndDone() throws IOException { @Test public void toolRequestRoundTripsThroughTheJinjaPath() throws IOException { - // Forwards an OpenAI tools array; the mapper enables use_jinja so the native parser applies - // Qwen3's tool-aware template. We assert the request is accepted and returns a structurally - // valid OpenAI message (content and/or tool_calls) — not that this tiny model elects to call. + // Forwards an OpenAI tools array. Jinja rendering (which is what lets the native parser + // apply Qwen3's tool-aware template) comes from how the model was loaded, not from the + // request -- upstream reads no "use_jinja" request key. We assert the request is accepted + // and returns a structurally valid OpenAI message (content and/or tool_calls) — not that + // this tiny model elects to call. String body = "{\"model\":\"" + MODEL_ID + "\",\"max_tokens\":48," + "\"messages\":[{\"role\":\"user\",\"content\":\"What is the weather in Paris?\"}]," + "\"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"get_weather\"," diff --git a/llama/src/test/java/net/ladenthin/llama/server/OpenAiRequestMapperTest.java b/llama/src/test/java/net/ladenthin/llama/server/OpenAiRequestMapperTest.java index f0506e635..9f8900688 100644 --- a/llama/src/test/java/net/ladenthin/llama/server/OpenAiRequestMapperTest.java +++ b/llama/src/test/java/net/ladenthin/llama/server/OpenAiRequestMapperTest.java @@ -105,9 +105,10 @@ public void toolsEnableChatTemplateAndForwardChoice() throws IOException { assertThat(out.path("tools").isArray(), is(true)); assertThat(out.path("tools").get(0).path("function").path("name").asText(), is("read_file")); assertThat(out.path("tool_choice").asText(), is("auto")); - // withUseChatTemplate(true) serializes as the native "use_jinja" flag, which enables the - // model's Jinja chat template (required for native tool-call parsing, e.g. Gemma 4 --jinja). - assertThat(out.path("use_jinja").asBoolean(), is(true)); + // No "use_jinja" is sent: jinja is a load-time option (ModelParameters.enableJinja() / + // --jinja), and upstream never reads a "use_jinja" key from a request body -- it would be + // discarded silently. Native tool-call parsing depends on how the model was loaded. + assertThat(out.has("use_jinja"), is(false)); } @Test diff --git a/llama/src/test/java/net/ladenthin/llama/value/ModelMetaTest.java b/llama/src/test/java/net/ladenthin/llama/value/ModelMetaTest.java index a72ab6fcd..163384314 100644 --- a/llama/src/test/java/net/ladenthin/llama/value/ModelMetaTest.java +++ b/llama/src/test/java/net/ladenthin/llama/value/ModelMetaTest.java @@ -62,6 +62,30 @@ public void testModalityGetters() throws Exception { assertThat(multimodal.supportsVideo(), is(true)); } + @Test + public void eachModalityGetterReadsItsOwnKey() throws Exception { + // The other modality fixtures set audio and video to the same value throughout, so a getter + // wired to the wrong sibling key still satisfies them. These three isolate one key each. + assertThat(parse(modalities("true", "false", "false")).supportsVision(), is(true)); + assertThat(parse(modalities("true", "false", "false")).supportsAudio(), is(false)); + assertThat(parse(modalities("true", "false", "false")).supportsVideo(), is(false)); + + assertThat(parse(modalities("false", "true", "false")).supportsVision(), is(false)); + assertThat(parse(modalities("false", "true", "false")).supportsAudio(), is(true)); + assertThat(parse(modalities("false", "true", "false")).supportsVideo(), is(false)); + + assertThat(parse(modalities("false", "false", "true")).supportsVision(), is(false)); + assertThat(parse(modalities("false", "false", "true")).supportsAudio(), is(false)); + assertThat(parse(modalities("false", "false", "true")).supportsVideo(), is(true)); + } + + private static String modalities(String vision, String audio, String video) { + return "{\"vocab_type\":1,\"n_vocab\":100,\"n_ctx_train\":4096," + + "\"n_embd\":512,\"n_params\":1000000,\"size\":500000," + + "\"modalities\":{\"vision\":" + vision + ",\"audio\":" + audio + ",\"video\":" + video + "}," + + "\"architecture\":\"llama\",\"name\":\"\"}"; + } + @Test public void modalityGettersDefaultToFalseOnOlderMetadata() throws Exception { // Metadata captured from a build that predates the `video` key: every accessor must answer From 0fa4bf3ec061c133efacecee14ca0c6d08c33755 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 19:36:56 +0000 Subject: [PATCH 65/76] feat: expose the video-decoding parameters refused at the b10649 bump setVideoFps, setVideoTimestampInterval and setVideoFfmpegDir map upstream's --video-fps / --video-timestamp-interval / --video-ffmpeg-dir. These were refused earlier in this branch on the reasoning that they would be inert without a ContentPart video factory. That was wrong twice over. They are not inert. server_context::load_model copies params_base.video_* into its own mtmd_helper_init_opt when the projector loads, and that init_opt is what server-context.cpp passes to process_mtmd_prompt on the task.cli_files path this binding drives -- so they apply to any media a caller attaches, today. And video decoding is genuinely in the shipped artifact: MTMD_VIDEO defaults ON, gated only on LLAMA_SUBPROCESS which is also ON, and libjllama.so carries the ffmpeg invocation strings. setVideoFfmpegDir is the one that earns its place: upstream otherwise resolves ffmpeg and ffprobe from PATH, and a JVM process -- an application server, an Android app, a container built for the JAR alone -- frequently has neither on PATH even when they are installed. Without this flag there is no way to point at them. Raw video bytes already reach the decoder through ContentPart.imageBytes(bytes, "video/mp4"), because mtmd_helper_bitmap_init_from_buf sniffs the container and falls through to video rather than trusting the declared MIME type. What remains is only the ergonomic entry point, ContentPart.videoFile(Path), which stays in TODO.md along with the corrected reasoning. The --spec-synth-* refusal stands: upstream marks those benchmarking-only. 1452 Java tests, 0 failures; javadoc clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CHANGELOG.md | 7 +++ TODO.md | 40 ++++++++------ docs/history/llama-cpp-breaking-changes.md | 2 +- .../llama/parameters/ModelParameters.java | 53 +++++++++++++++++++ .../llama/parameters/ModelParametersTest.java | 31 +++++++++++ 5 files changed, 115 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 478d8d47c..303dbd52d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,13 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by only the weight class that dominates a model's size, usually fitting a much larger model into the same VRAM at a smaller speed cost. Only `--n-cpu-ffn` is new in llama.cpp b10649; `--n-cpu-moe` has existed upstream since b6089 but had never been exposed here. +- **`ModelParameters.setVideoFps(float)` / `setVideoTimestampInterval(long)` / `setVideoFfmpegDir(String)`** + — the video-decoding knobs upstream added in llama.cpp b10649 (`--video-fps`, + `--video-timestamp-interval`, `--video-ffmpeg-dir`). They apply to any media attached to a request + once a projector is loaded: `server_context` copies them into the `mtmd_helper_init_opt` it passes + to `process_mtmd_prompt`, and video decoding is compiled into the shipped library (`MTMD_VIDEO` + defaults on). `setVideoFfmpegDir` is the significant one — upstream otherwise looks `ffmpeg` and + `ffprobe` up on `PATH`, which a JVM process often does not have them on. - **`ServerMetrics.getWindowPromptProcessingMillis()` / `getWindowTokenGenerationMillis()` / `getWindowTimings()`** — typed access to the current-window timing keys `t_prompt_processing` and `t_tokens_generation`. Both were always emitted; only the cumulative `_total` variants had accessors. diff --git a/TODO.md b/TODO.md index a0b4591cc..49023928a 100644 --- a/TODO.md +++ b/TODO.md @@ -290,23 +290,29 @@ upstream PR #22393 — it drops automatically when that merges.) These are JNI plumbing items for upstream API additions. Policy: add only after a real user request — they are mostly relevant to specific model families or specialized workflows. -- **Video input (`ContentPart.videoFile(...)` + the three `--video-*` knobs).** llama.cpp **b10649** - added an end-to-end video path to `mtmd`: `mtmd_helper_video_init_params` (fps target, ffmpeg binary - directory, timestamp interval), a fourth `mtmd_helper_init_opt` parameter on the bitmap/tokenize - helpers, and the CLI flags `--video-fps`, `--video-timestamp-interval`, `--video-ffmpeg-dir`. - - The three flags were **deliberately not exposed** at the b10649 bump. They configure video decoding - and are inert without a way to *submit* a video: `ContentPart` offers text, image and audio only, so - adding the knobs alone would create exactly the class of dead-but-documented parameter that - `withTfsZ` / `withPenalizeNl` / `withPenaltyPrompt` had to be deprecated for. The bump passes - `mtmd_helper_init_opt_default()` at every call site, which is correct for image and audio. - - Doing it properly means: a `ContentPart.videoFile(Path)` (and probably a bytes overload), routing a - real `mtmd_helper_init_opt` from `ModelParameters` through the JNI multimodal path instead of the - default, `ModelParameters.setVideoFps/​setVideoTimestampInterval/​setVideoFfmpegDir`, and an - integration test. Note the runtime cost: upstream **shells out to `ffmpeg`/`ffprobe`**, so a - consumer needs those binaries on the host — worth stating prominently in the Javadoc, and it makes - the feature untestable on any CI runner without them. +- **Video input (`ContentPart.videoFile(...)`).** llama.cpp **b10649** added an end-to-end video path + to `mtmd`: `mtmd_helper_video_init_params` (fps target, ffmpeg binary directory, timestamp + interval), a fourth `mtmd_helper_init_opt` parameter on the bitmap/tokenize helpers, and the CLI + flags `--video-fps`, `--video-timestamp-interval`, `--video-ffmpeg-dir`. + + **The three flags are now exposed** as `ModelParameters.setVideoFps` / + `setVideoTimestampInterval` / `setVideoFfmpegDir`. They were initially refused at the b10649 bump + as "inert without a way to submit a video"; a later audit showed that reasoning was wrong on two + counts. First, they are not inert: `server_context::load_model` copies them into its own + `init_opt` when the projector loads, and that `init_opt` is what `server-context.cpp` passes to + `process_mtmd_prompt` on the task path this binding uses — so they take effect for any media the + caller attaches. Second, video decoding is genuinely compiled in: `MTMD_VIDEO` defaults to `ON` + (it needs only `LLAMA_SUBPROCESS`, also `ON`), and the shipped `libjllama.so` carries the ffmpeg + invocation strings. `setVideoFfmpegDir` is the one that matters most, because upstream otherwise + looks the binaries up on `PATH`, which a JVM process frequently does not have them on. + + What is still missing is only the *ergonomic* entry point: a `ContentPart.videoFile(Path)` and a + bytes overload. A caller can already submit video today by handing raw bytes to + `ContentPart.imageBytes(bytes, "video/mp4")` — `mtmd_helper_bitmap_init_from_buf` sniffs the + container and falls through to video decoding rather than trusting the declared MIME type — but + that is an obscure spelling for it, and the missing factory is the remaining work, together with + an integration test. Note the runtime cost: upstream **shells out to `ffmpeg`/`ffprobe`**, so a + consumer needs those binaries, which makes the feature untestable on a CI runner without them. - **`--spec-synth-len` / `--spec-synth-rates` — deliberately NOT exposed, and this should stay that way.** Added in b10649. Upstream's own help text marks both **"(benchmarking only)"**: they diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 319729360..21cad48ff 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -680,4 +680,4 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10639–b10644 | `include/llama.h` (**`LLAMA_SESSION_VERSION` 9 → 10 and `LLAMA_STATE_SEQ_VERSION` 2 → 3** — the only two lines in the file that moved), `src/llama-kv-cells.h` + `src/llama-kv-cache.{cpp,h}` (`llama_kv_cell_ext` gains a `tok` field for n-gram input embeddings, plus `has_cell_ext()` / `get_prev_tokens()` / `for_each_token_in()`, and `reset()` swaps a `memset` for value-initialisation), `src/models/nanbeige.cpp` (one line: register `t_layer_inp[il]`), `tests/test-backend-ops.cpp`, the whole `ggml/src/ggml-hexagon/**` + `scripts/snapdragon/**` + `docs/backend/snapdragon/**` Snapdragon/Hexagon backend rework, plus `tools/ui/**` (the WebUI, auto-followed) | **No project-source change, and the only priority-review-list file touched is `include/llama.h`, whose entire diff is the two version macros.** Sizing: the raw diff is ~608 KiB, but it is dominated by two components this project never compiles or reviews — `tools/ui` (auto-followed; the `build-webui` job rebuilds it from the pin) and the Hexagon backend (`grep -rn GGML_HEXAGON` over this repo's workflows, build scripts and CMake is empty, so `ggml-hexagon.cpp` is never in a `libjllama` link line). Excluding both leaves **11 KiB across 6 files**, well under the runbook's 100 KiB single-step threshold, so no chunking was needed. **Zero** files under `common/`, `tools/server/` or `tools/mtmd/` changed, so every remaining row of the API-compatibility table is vacuously satisfied and the three mechanical server-contract greps have no input — the request-field set, its bounds and the emitted response keys cannot have moved. **Compatibility note for consumers, not a compile break:** the two version bumps are a *state-file format* break — a slot state saved by an older build (`LlamaModel.handleSlotAction` save, or the server's `/slots/{id}?action=save`) is rejected by its own version check on b10644 and must be regenerated. Nothing the project calls changed shape. **All 8 patches apply unchanged** — the 6 project-relevant changed files and the 42 files the patches target have **zero** overlap (no `common/arg.{cpp,h}`, no `common/peg-parser.cpp`, no `tools/server/*`; the one changed test, `tests/test-backend-ops.cpp`, is touched by no patch). | | b10639–b10644 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10644 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10644: all **8** patches applied, confirmed by the stamp file (`head d7a2074112d27649303fa107eb8c94db1ee435f3` plus one SHA-256 line per patch) — including `0011`, whose target `common/peg-parser.cpp` the range does not touch. Full `cmake --build --config Release` clean, `ctest` **509/509**. `nm -D` on the freshly linked `libjllama.so` reports **40** `Java_*` exports and **zero** C++-mangled ones. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10644") matches the `build-info` compiled into the new binary (`b10644-d7a207411`), which is what proves the four pin sites and the actual build agree. Run with `mvn clean test`, not a bare `mvn test`: `LLAMA_CPP_VERSION` is a compile-time constant that javac inlines into the already-compiled test class, and Maven's incremental compilation cannot see that dependency — the b10636→b10639 row below records the false drift that costs. | | b10644–b10649 | **`tools/mtmd/mtmd-helper.{h,cpp}` (BREAKING: `mtmd_helper_bitmap_init_from_file` and `_from_buf` gain a 4th `struct mtmd_helper_init_opt` parameter; new `mtmd_helper_video_init_params` + `mtmd_helper_init_opt` structs and their `_default()` factories)**, **`tools/server/server-common.{h,cpp}` (BREAKING: `tokenize_input_prompts`, `process_mtmd_prompt` and `format_prompt_rerank` all gain a trailing `const mtmd_helper_init_opt &`)**, `tools/server/server-context.{h,cpp}` (`handle_count_tokens` gains the same parameter; threading it through), `common/common.h` (**additive**: `common_params_speculative::{synth_len,synth_rates}` + `has_synth()`, `common_params::{video_fps,video_timestamp_interval_ms,video_ffmpeg_bin_dir}`), `common/arg.cpp` (**purely additive**: 6 new flags), `common/speculative.{cpp,h}` (**additive**: `common_speculative_n_max`, `_synth_rates_resolve`, `_get_synth_probs`), `ggml/src/ggml-metal/*`, `src/models/minimax-01.cpp`, `tests/test-arg-parser.cpp`, `tools/{cli,llama-bench,tts}`, `tools/server/tests/*` | **The first range in this whole bump that broke the project's own compile — four call sites, all from one upstream refactor.** llama.cpp b10649 threaded a new `mtmd_helper_init_opt` (video decode settings: fps target, ffmpeg binary dir, timestamp interval) through every helper that can ingest media. `tts_engine.cpp:95` (`mtmd_helper_bitmap_init_from_file`, the speaker-reference clip) and `jllama.cpp` ×3 (`tokenize_input_prompts`) + ×1 (`format_prompt_rerank`) all lost their signatures. Every one of those paths passes `mctx = nullptr` or handles audio, so none wants video settings: each now passes `mtmd_helper_init_opt_default()`, which is upstream's own default. **This is the failure class the priority table's note calls out** — `server-common.h` is a same-repo header the project `#include`s directly rather than one reachable through the documented dependency graph, so only a real compile finds it. **The wire contract did not move:** all three mechanical greps are identical across the range (68 request fields, 23 bounds, 286 response keys over all six server `.cpp`). `common/arg.cpp` removed or renamed **zero** flags. Sizing: 106 KiB reviewable, or **72 KiB excluding `ggml-metal`** (Metal backend internals behind unchanged public headers — same exclusion rationale as Hexagon), under the runbook's 100 KiB single-step threshold. **All 8 patches apply unchanged** despite six patch-target files changing (`common/arg.cpp`, `tests/test-arg-parser.cpp`, `tools/mtmd/mtmd-cli.cpp`, `tools/server/server-context.{cpp,h}`, `tools/tts/tts.cpp`) — upstream's edits fall outside every patched region. | -| b10644–b10649 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10649 -DBUILD_TESTING=ON` through the real `FetchContent` path: all **8** patches applied (stamp `head 2bb9bddafad44ecbb50889644ca47537ec11841b`), which was the first thing checked because six patch-target files changed in the range. The first build failed loud with the four signature breaks above; after adapting the call sites, Release build clean, `ctest` **512/512**, `nm -D` **40** `Java_*` exports and **zero** C++-mangled, `NativeLibraryLoadSmokeTest` **3/3, 0 skipped** including the pin cross-check against the linked `build-info`. `ModelParametersTest` 73/73 with the five new CPU-offload cases. **Feature exposure decided deliberately, not by default:** of the 6 new flags, only `--n-cpu-ffn` is genuinely new upstream, and it was added as `ModelParameters.setCpuFfnLayers`; `setCpuMoeLayers` was added alongside it for `--n-cpu-moe`, which has existed upstream since **b6089** (#15077, `ec428b02c`) but had never been exposed here — b10649 only refactored its lambda onto the shared `llm_add_n_cpu_ffn_overrides` helper; the two `--spec-synth-*` flags were refused as upstream marks them "benchmarking only"; and the three `--video-*` flags were refused because `ContentPart` has no video input path, so they would be inert — the same dead-knob shape that forced the `withTfsZ`/`withPenalizeNl`/`withPenaltyPrompt` deprecations. Both refusals are recorded in `TODO.md` with their reasoning so they do not resurface as open questions. | +| b10644–b10649 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10649 -DBUILD_TESTING=ON` through the real `FetchContent` path: all **8** patches applied (stamp `head 2bb9bddafad44ecbb50889644ca47537ec11841b`), which was the first thing checked because six patch-target files changed in the range. The first build failed loud with the four signature breaks above; after adapting the call sites, Release build clean, `ctest` **512/512**, `nm -D` **40** `Java_*` exports and **zero** C++-mangled, `NativeLibraryLoadSmokeTest` **3/3, 0 skipped** including the pin cross-check against the linked `build-info`. `ModelParametersTest` 73/73 with the five new CPU-offload cases. **Feature exposure decided deliberately, not by default:** of the 6 new flags, only `--n-cpu-ffn` is genuinely new upstream, and it was added as `ModelParameters.setCpuFfnLayers`; `setCpuMoeLayers` was added alongside it for `--n-cpu-moe`, which has existed upstream since **b6089** (#15077, `ec428b02c`) but had never been exposed here — b10649 only refactored its lambda onto the shared `llm_add_n_cpu_ffn_overrides` helper; the two `--spec-synth-*` flags were refused as upstream marks them "benchmarking only"; and the three `--video-*` flags were initially refused as inert without a `ContentPart` video factory. **That refusal was wrong and was reversed by the follow-up audit:** they are not inert — `server_context::load_model` copies them into its own `init_opt` when the projector loads, and that `init_opt` is what `server-context.cpp` hands to `process_mtmd_prompt` on the very task path this binding drives (`task.cli_files`), so they take effect for any attached media; and video decoding really is compiled in (`MTMD_VIDEO` defaults `ON`, gated only on `LLAMA_SUBPROCESS` which is also `ON`, and the shipped `libjllama.so` carries the ffmpeg invocation strings). They are now `ModelParameters.setVideoFps` / `setVideoTimestampInterval` / `setVideoFfmpegDir`; `setVideoFfmpegDir` matters most, since upstream otherwise resolves `ffmpeg`/`ffprobe` from `PATH`, which a JVM process frequently lacks. Only the ergonomic `ContentPart.videoFile(...)` entry point is still outstanding (raw bytes already work — the decoder sniffs the container rather than trusting the MIME type). The `--spec-synth-*` refusal stands. Both are recorded in `TODO.md` with their reasoning. | diff --git a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java index 0f95ea621..72ce2ecf8 100644 --- a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java +++ b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java @@ -1440,6 +1440,59 @@ public ModelParameters enableMmprojOffload() { return setMmprojOffload(true); } + /** + * Target frame rate at which an attached video is sampled (upstream {@code --video-fps}, + * llama.cpp b10649; default 4.0). + * + *

    Video frames are decoded through ffmpeg and fed to the projector as images, so this is the + * main cost/detail dial: doubling it doubles the frames, the tokens they occupy and the decode + * time. Applies only when a projector is loaded ({@link #setMmproj(String)}) and the request + * carries media — it is read once, when the projector is initialised.

    + * + * @param fps frames per second to sample from the video; must be positive + * @return this builder + * @throws IllegalArgumentException if {@code fps} is not positive + */ + public ModelParameters setVideoFps(float fps) { + if (!(fps > 0.0f)) { + throw new IllegalArgumentException("Invalid video-fps value: " + fps + " (must be > 0)"); + } + return putScalar("--video-fps", fps); + } + + /** + * Interval between the text timestamps interleaved into a decoded video (upstream + * {@code --video-timestamp-interval}, llama.cpp b10649; default 5000 ms). + * + * @param intervalMillis milliseconds between timestamps; must not be negative + * @return this builder + * @throws IllegalArgumentException if {@code intervalMillis} is negative + */ + public ModelParameters setVideoTimestampInterval(long intervalMillis) { + if (intervalMillis < 0) { + throw new IllegalArgumentException( + "Invalid video-timestamp-interval value: " + intervalMillis + " (must be >= 0)"); + } + return putScalar("--video-timestamp-interval", intervalMillis); + } + + /** + * Directory holding the {@code ffmpeg} and {@code ffprobe} binaries used to decode video + * (upstream {@code --video-ffmpeg-dir}, llama.cpp b10649). + * + *

    This matters more here than it does for the standalone server: llama.cpp looks the binaries + * up on {@code PATH} by default, and a JVM process — an application server, an Android app, + * a container built for the JAR alone — frequently has no ffmpeg on {@code PATH} even when + * one is installed. Naming the directory is then the only way to make video input work. When + * unset, upstream keeps its {@code PATH} lookup.

    + * + * @param directory the directory containing the ffmpeg binaries + * @return this builder + */ + public ModelParameters setVideoFfmpegDir(String directory) { + return putScalar("--video-ffmpeg-dir", directory); + } + /** * Enable or disable GPU offload for the multimodal projector. This is independent of * {@link #setGpuLayers(int)} because upstream enables projector offload by default. diff --git a/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java b/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java index 846eba3be..f084b2ad9 100644 --- a/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java +++ b/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java @@ -504,6 +504,37 @@ public void testSetMmprojDeviceNoneIsPassedThroughVerbatim() { assertThat(p.parameters, not(hasKey("--no-mmproj-offload"))); } + // ------------------------------------------------------------------------- + // Video decoding — the mmproj-gated knobs added in llama.cpp b10649 + // ------------------------------------------------------------------------- + + @Test + public void testVideoDecodingFlagsRenderTheUpstreamNames() { + ModelParameters p = new ModelParameters() + .setVideoFps(2.5f) + .setVideoTimestampInterval(1500L) + .setVideoFfmpegDir("/opt/ffmpeg/bin"); + assertThat(p.parameters.get("--video-fps"), is("2.5")); + assertThat(p.parameters.get("--video-timestamp-interval"), is("1500")); + assertThat(p.parameters.get("--video-ffmpeg-dir"), is("/opt/ffmpeg/bin")); + } + + @Test + public void testVideoFpsRejectsNonPositiveValues() { + // Upstream parses this with std::stof and never validates it; a zero or negative target + // frame rate reaches the decoder, so reject it here where the message can name the flag. + assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setVideoFps(0.0f)); + assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setVideoFps(-1.0f)); + } + + @Test + public void testVideoTimestampIntervalRejectsNegativeValues() { + assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setVideoTimestampInterval(-1L)); + // Zero is legal: it means "a timestamp on every frame". + ModelParameters p = new ModelParameters().setVideoTimestampInterval(0L); + assertThat(p.parameters.get("--video-timestamp-interval"), is("0")); + } + @Test public void testMmprojDeviceAndOffloadClearEachOtherInBothDirections() { // Both write upstream's single mmproj_use_gpu field, and our argv is rendered from a HashMap From ce330a1666f57c404b62eebeb4b38177a6a97ca5 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 19:37:40 +0000 Subject: [PATCH 66/76] ci: give the langchain4j integration job the crash diagnostics it lacked It is model-backed and crosses JNI, so a forked test JVM there can abort exactly the way the six test-java-* jobs can -- but it had neither the echo step nor the dump upload, so such an abort would have left no diagnostic on either path. Same step and same path set as those jobs, scoped to this module. Pre-existing, not a regression from this branch; noticed while auditing this session's own CI changes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- .github/workflows/publish.yml | 46 +++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 572265e08..d97e48d9c 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -457,6 +457,52 @@ jobs: -Dnet.ladenthin.llama.langchain4j.rerank.model=models/${RERANKING_MODEL_NAME} -Dnet.ladenthin.llama.langchain4j.tool.model=models/${TOOL_MODEL_NAME} + # This job is model-backed and crosses JNI, so a forked test JVM here can abort exactly the + # way the six test-java-* jobs can -- but it had neither of their diagnostics. Same step and + # same path set as those, scoped to this module. See + # ../workspace/policies/ci-test-diagnostics.md section 3.1. + - name: Print crash logs (on failure) + if: failure() + shell: bash + run: | + shopt -s nullglob + found=0 + for f in llama-langchain4j/hs_err_pid*.log; do + found=1 + echo "===== $f (first 200 lines; full file in the uploaded artifact) =====" + sed -n '1,200p' "$f" + done + for f in llama-langchain4j/target/surefire-reports/*.dumpstream llama-langchain4j/target/surefire-reports/*.dump; do + found=1 + echo "===== $f =====" + cat "$f" + done + if [ "$found" = 0 ]; then + echo "No hs_err_pid*.log and no surefire dump/dumpstream was written." + echo + echo "For an ordinary test failure that is EXPECTED, not a finding: this step runs on" + echo "any job failure, and an assertion failure, a timeout or a compile error writes no" + echo "crash log. Read the surefire output above for the real cause." + echo + echo "It points at a JVM-level abort only if the log ALSO shows a fork ending abnormally" + echo "-- 'The forked VM terminated without properly saying goodbye', or an exit with no" + echo "test results. In that case the abort bypassed the JVM error handler (a native" + echo "exit()/terminate() rather than a raised signal), which is why no file was written." + fi + - if: failure() + uses: actions/upload-artifact@v7 + with: + name: error-log-langchain4j-integration + path: | + ${{ github.workspace }}/llama-langchain4j/hs_err_pid*.log + ${{ github.workspace }}/core.* + ${{ github.workspace }}/llama-langchain4j/*.hprof + ${{ github.workspace }}/llama-langchain4j/target/surefire-reports/*.dump + ${{ github.workspace }}/llama-langchain4j/target/surefire-reports/*.dumpstream + ${{ github.workspace }}/llama-langchain4j/target/surefire-reports/*.txt + ${{ github.workspace }}/llama-langchain4j/target/surefire-reports/TEST-*.xml + if-no-files-found: warn + # --------------------------------------------------------------------------- # Build the llama.cpp WebUI ONCE, from the same pinned tag CMakeLists.txt fetches, # and share it to every native build as the generated, platform-independent From dd07b0e668bc6fcd58347f4007729089899b2d94 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 20:07:05 +0000 Subject: [PATCH 67/76] test: budget the langchain4j chat tests past Qwen3's thinking block Both tests capped generation at 320 tokens, which is exactly on the boundary for this model. Run 33109360197 proved it: from the same prompt against the same Qwen3-0.6B, chatReturnsAssistantText finished thinking at 267 tokens and passed, while streamingDeliversTokensThenCompletes consumed all 320 inside -- "eval time = 6490.39 ms / 320 tokens" -- emitted no assistant content, and failed its non-empty assertion. The assertions are right and stay: a reply that is only a thinking block is a real failure, and accepting "content or thinking" is what the earlier 8-token version of the streaming test did wrong. The budget was the problem. Both now use a shared MAX_OUTPUT_TOKENS of 1500, matching the core module's ReasoningBudgetTest N_PREDICT for this same model, with the measurement that motivated it recorded on the constant. It is a cap, not a target -- a healthy run stops around 270-340 tokens -- so this costs nothing except in the pathological case it exists to absorb. The streaming test's future timeout goes 60s -> 180s, since 60s was sized for the old cap and leaves no margin for a worst-case 1500 on a shared runner. 46 tests, 0 failures, 4 skipped locally. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- .../JllamaChatModelIntegrationTest.java | 38 ++++++++++++------- 1 file changed, 25 insertions(+), 13 deletions(-) diff --git a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java index 20860efbb..cc4de0fba 100644 --- a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java +++ b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/JllamaChatModelIntegrationTest.java @@ -28,6 +28,23 @@ */ class JllamaChatModelIntegrationTest { + /** + * Generation budget for both tests, matching the core module's {@code ReasoningBudgetTest} + * ({@code N_PREDICT = 1500}) for the same model. + * + *

    Qwen3-0.6B is a reasoning model: it spends its first few hundred tokens inside + * {@code } and only then emits assistant content, so a budget that does not clear the + * thinking block yields an empty answer rather than a short one. 320 was tried and is + * right on the boundary — in one CI run (33109360197) the blocking test finished thinking at 267 + * tokens and passed while the streaming test consumed all 320 inside {@code } and failed, + * from the same prompt against the same model. The adapter exposes no reasoning-budget knob, so + * the output budget is the only lever. + * + *

    This is a cap, not a target: a normal run stops around 270–340 tokens, so raising it costs + * nothing except in the pathological case it exists to absorb. + */ + private static final int MAX_OUTPUT_TOKENS = 1500; + private static Path modelPath() { Path resolved = TestModelPaths.fromProperty("net.ladenthin.llama.model.path"); Assumptions.assumeTrue(resolved != null, "model path property not set"); @@ -45,11 +62,9 @@ void chatReturnsAssistantText() { chat.chat( ChatRequest.builder() .messages(UserMessage.from("Reply with the single word: ok")) - // Same budget as the streaming sibling, for the same reason: - // Qwen3-0.6B spends ~200 tokens inside first, so a small - // budget yields an EMPTY assistant text -- which a bare - // notNullValue() assertion would happily accept. - .maxOutputTokens(320) + // See MAX_OUTPUT_TOKENS: too small a budget yields an EMPTY + // assistant text, which a bare notNullValue() would accept. + .maxOutputTokens(MAX_OUTPUT_TOKENS) .build()); assertThat(response.aiMessage(), is(notNullValue())); @@ -70,13 +85,10 @@ void streamingDeliversTokensThenCompletes() throws Exception { streaming.chat( ChatRequest.builder() .messages(UserMessage.from("Reply with the single word: ok")) - // Qwen3-0.6B is a reasoning model and spends ~200 tokens inside - // before it answers (see the core module's ReasoningBudgetTest, which - // budgets 1500 for exactly this reason). The adapter exposes no - // reasoning-budget knob, so the budget has to clear the thinking block or - // the run produces no assistant text at all -- and the assertion below - // would then be satisfied by a stream that delivered only thinking. - .maxOutputTokens(320) + // See MAX_OUTPUT_TOKENS: the budget has to clear the thinking block, or + // the run produces no assistant content at all and the second assertion + // below fails on a healthy model. + .maxOutputTokens(MAX_OUTPUT_TOKENS) .build(), new StreamingChatResponseHandler() { @Override @@ -95,7 +107,7 @@ public void onError(Throwable error) { } }); - ChatResponse complete = done.get(60, TimeUnit.SECONDS); + ChatResponse complete = done.get(180, TimeUnit.SECONDS); // Two independent assertions, both of which must hold. // From 7da5c5037320e569749d9229fce1720df1da8588 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 21:23:49 +0000 Subject: [PATCH 68/76] docs: close the two TODO entries this PR actually fixed Both were still marked [OPEN] while their fixes shipped in this same branch, so the file claimed two live bugs that no longer exist. SessionForkRewindIntegrationTest: the entry's "not a bump regression" conclusion was right, but its diagnosis -- "it is the KV restore, not the bookkeeping" -- was wrong. Qwen3-0.6B is a reasoning model and the tests' budget was consumed inside , so generation completed normally and simply returned no assistant content. Fixed in cc67ea7; the identical root cause resurfaced in llama-langchain4j and was fixed the same way in dd07b0e. NativeServerAttachIntegrationTest.completion_overHttp_served: fixed by local patch 0011 (ca60947) -- common_peg_until_parser honoured leniency on an incomplete trailing UTF-8 sequence but not on an invalid byte, so one malformed byte turned a finished generation into a 500. The entry now records the mechanism and the runnable guard. No [OPEN] entries remain in the jllama-specific section. The cross-cutting section is unchanged: those are standing repo-hygiene items, not upgrade work. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- TODO.md | 35 +++++++++++++++++++++-------------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/TODO.md b/TODO.md index 49023928a..6a855d829 100644 --- a/TODO.md +++ b/TODO.md @@ -120,21 +120,28 @@ same crashed test. takes the whole fork down it also truncates every job's test run, so it blocks seeing the rest of the suite and should be fixed first. -- **[OPEN] `SessionForkRewindIntegrationTest` — empty reply after a slot restore (2 failures).** - `rewindRestoresTranscriptAndConversationContinues:81` and - `forkCreatesIndependentSessionWithSameTranscript:98` both fail `assertThat(reply.isEmpty(), - is(false))`: generation returns an empty string after `rewind()` / `fork()`, i.e. after slot state - is restored from a file. The transcript assertions around them pass, so it is the KV restore, not - the bookkeeping. Checked and **not** a bump regression: the only slot-touching upstream commit in - b10456..b10618 is `e8eed4525` (`LLAMA_SERVER_SLOTS_N_DIFF`), which is env-var opt-in and defaults - to 0, hence inert. - -- **[OPEN] `NativeServerAttachIntegrationTest.completion_overHttp_served:108` — HTTP 500.** +- **[FIXED in this PR] `SessionForkRewindIntegrationTest` — empty reply after a slot restore.** + `rewindRestoresTranscriptAndConversationContinues` and + `forkCreatesIndependentSessionWithSameTranscript` failed `assertThat(reply.isEmpty(), is(false))` + after `rewind()` / `fork()`. The diagnosis above was right that it is not a bump regression, but + wrong that it is the KV restore: the model is Qwen3-0.6B, a **reasoning** model, and the tests' + token budget was being spent entirely inside ``, so generation completed normally and + returned no assistant *content*. Fixed in `cc67ea7` by budgeting past the thinking block. The same + root cause resurfaced later in `llama-langchain4j` (`dd07b0e`), where both chat tests now share a + `MAX_OUTPUT_TOKENS = 1500` matching `ReasoningBudgetTest`'s `N_PREDICT`. Green on all six Java + platforms in run 33111759140. + +- **[FIXED in this PR] `NativeServerAttachIntegrationTest.completion_overHttp_served` — HTTP 500.** `{"error":{"code":500,"message":"The model produced output that does not match the expected - Content-only format"}}`. `"Content-only"` is a `common_chat_format` name (`common/chat.cpp:856`), so - the request ran without a template and the reply failed to parse under that format. Checked and - **not** a bump regression: the error string is byte-identical at b10456 and b10618, and the only - upstream commit touching `common/chat.cpp` in the range is the `common_json` abstraction (#27511). + Content-only format"}}`. Correctly identified above as long-standing upstream behaviour rather than + a bump regression. Root cause: `common_peg_until_parser` (`common/peg-parser.cpp`) tolerates an + **incomplete** trailing UTF-8 sequence in lenient mode — the only mode `common_chat_peg_parse` ever + uses — but its **invalid**-byte branch returned `FAIL` unconditionally, so a single malformed byte + anywhere in the output turned a *finished* generation into a 500. Fixed by local patch `0011` + (`ca60947`), which makes the invalid branch honour leniency exactly as the incomplete branch does; + strict mode is unchanged. Guarded by the `ContentOnlyParseUtf8` tests in `test_utils.cpp`, which — + unlike the upstream test the patch also adds — run in CI on every platform. Upstream-submittable; + re-checked at b10649 and upstream has not fixed it, so the patch stays. - **[ANSWERED] Re-check the full suite once the TTS crash is fixed.** Done: Ubuntu on `999034b` ran **1689 tests, 3 failures, 1 error, 2 skipped**. Exactly one item was new — the router entry From 7319e6244a45533d57a9cb45995d8c39d47a79f5 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 21:35:03 +0000 Subject: [PATCH 69/76] docs: correct the remaining stale counts and the video-input claim Final sweep over this branch's own documentation. Corrected in CHANGELOG: "304 response keys" -> 286 over the six compiled server TUs, and "of the 7 new upstream flags" -> 6, with --n-cpu-moe named as pre-existing (b6089) rather than new. The b10644-b10649 verification row still said ctest 512/512 and ModelParametersTest 73/73, which the second audit's TrainParams and video-flag cases moved to 516/516 and 77/77. Withdrew an untested workaround. An earlier draft told readers to submit video via ContentPart.imageBytes(bytes, "video/mp4"), reasoning that mtmd_helper_bitmap_init_from_buf sniffs the container. Reading upstream properly: the wire type for video is input_video, and it calls handle_media with accept_base64_uri = false -- raw base64 only, NOT the data: URI form the image factories build. The image_url route might still work by sniffing, but it is untested here and additionally gated on allow_image, so it is no longer documented as supported. README and TODO now state the verified shape a future ContentPart.videoFile must emit. README also gained the mmproj device/offload exclusivity rule, which was behaviour a caller can hit with no note anywhere, and a short video section covering the three new setters. Recorded the teardown ordering wake_and_post depends on: upstream's loop leaves `sleeping` set when it exits on !running, so a caller parked in wait_until_no_sleep() is released only because Java_..._delete drains users == 0 before calling terminate(). Verified correct today; the comment exists so a future reorder does not silently deadlock. Whole-reactor mvn verify green (SpotBugs 0 bugs, enforcer incl. dependency convergence, javadoc), ctest 516/516, clang-format clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CHANGELOG.md | 20 +++++++++++-------- README.md | 18 +++++++++++++++++ TODO.md | 23 +++++++++++++++------- docs/history/llama-cpp-breaking-changes.md | 2 +- llama/src/main/cpp/jllama.cpp | 8 ++++++++ 5 files changed, 55 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 303dbd52d..d625bae19 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -124,14 +124,18 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by that can ingest media, changing the signature of `mtmd_helper_bitmap_init_from_file`, `tokenize_input_prompts` and `format_prompt_rerank`. Four call sites were adapted — all of them pass `mctx = nullptr` or handle audio, so each now passes upstream's own `mtmd_helper_init_opt_default()`. - The wire contract is unchanged (68 request fields, 23 bounds and 304 response keys identical across - the range; zero CLI flags removed or renamed), and all eight local patches apply unchanged even though - six patch-target files were touched. - Of the 7 new upstream flags, the two CPU-offload ones are now exposed (see Added). The two - `--spec-synth-*` flags are not: upstream marks them "benchmarking only" — they synthesise fake - acceptance probabilities to measure llama.cpp's own speculative harness. The three `--video-*` flags - are not either: `ContentPart` has no video input, so they would be inert knobs. Both decisions and - what a real video-input feature would require are recorded in `TODO.md`. + The wire contract is unchanged (68 request fields, 23 bounds and 286 response keys identical across + the range over the six compiled server TUs; zero CLI flags removed or renamed), and all eight local + patches apply unchanged even though six patch-target files were touched. + Of the 6 new upstream flags, four are now exposed (see Added): `--n-cpu-ffn` and the three + `--video-*` knobs. `--n-cpu-moe` is exposed alongside them but is not new — it has existed upstream + since b6089 and had simply never been surfaced here. The two `--spec-synth-*` flags stay unexposed: + upstream marks them "benchmarking only" — they synthesise fake acceptance probabilities to measure + llama.cpp's own speculative harness. The `--video-*` trio was initially refused as inert without a + `ContentPart` video factory; a follow-up audit showed that was wrong on both counts (they reach the + task path this binding drives, and `MTMD_VIDEO` is compiled into the shipped library), so they are + exposed. The content part itself — upstream's `input_video`, which takes raw base64 rather than a + `data:` URI — remains in `TODO.md`. - Upgraded llama.cpp from **b10639 to b10644**. No project-source change, and the only file on the priority API-review list that the range touches is `include/llama.h`, whose entire diff is two constants: `LLAMA_SESSION_VERSION` 9 → 10 and `LLAMA_STATE_SEQ_VERSION` 2 → 3. They follow from a new diff --git a/README.md b/README.md index 6fe643235..d285d3f06 100644 --- a/README.md +++ b/README.md @@ -495,6 +495,24 @@ device may be named; the literal `"none"` keeps the projector on the CPU. `OpenA accepts the same flag as `-mmdev`/`--mmproj-device`, and `NativeServer` forwards it verbatim like every other llama-server flag. +`setMmprojDevice(...)` and `setMmprojOffload(...)` are mutually exclusive: upstream writes both onto +the single `mmproj_use_gpu` field, so only one may appear in the rendered argv and **the last of the +two calls wins** — each clears the other. Pick one; if you need a device after disabling offload, +call `setMmprojDevice` last. + +**Video input — decode settings only, so far.** llama.cpp b10649 added a video path to `mtmd`, and it +is compiled into the shipped library (`MTMD_VIDEO` is on by default). Its decode settings are exposed +as `setVideoFps(float)`, `setVideoTimestampInterval(long)` and `setVideoFfmpegDir(String)`. The last +one matters most in a JVM: upstream shells out to `ffmpeg`/`ffprobe` and resolves them from `PATH`, +which an application server, an Android app or a JAR-only container frequently does not have them on — +naming the directory is then the only way for video to work at all. + +What is **not** here yet is the content part: upstream's wire type for a video is +`{"type":"input_video","input_video":{"data":""}}` (raw base64, not a `data:` URI, unlike +`image_url`), gated server-side on `mtmd_helper_support_video`. `ContentPart` has no `videoFile(...)` +factory emitting that shape, so these knobs currently configure a path this API cannot yet feed +directly. Tracked in `TODO.md`. + **Audio input** works identically — load an audio-capable model (Ultravox, Qwen2.5-Omni, …) with its audio `--mmproj` and add a `ContentPart.audioFile(...)` (or `inputAudio(bytes, "wav"|"mp3")`) part. It serializes to the OpenAI `input_audio` content part and routes through the same `mtmd` pipeline: diff --git a/TODO.md b/TODO.md index 6a855d829..e9d964187 100644 --- a/TODO.md +++ b/TODO.md @@ -313,13 +313,22 @@ These are JNI plumbing items for upstream API additions. Policy: add only after invocation strings. `setVideoFfmpegDir` is the one that matters most, because upstream otherwise looks the binaries up on `PATH`, which a JVM process frequently does not have them on. - What is still missing is only the *ergonomic* entry point: a `ContentPart.videoFile(Path)` and a - bytes overload. A caller can already submit video today by handing raw bytes to - `ContentPart.imageBytes(bytes, "video/mp4")` — `mtmd_helper_bitmap_init_from_buf` sniffs the - container and falls through to video decoding rather than trusting the declared MIME type — but - that is an obscure spelling for it, and the missing factory is the remaining work, together with - an integration test. Note the runtime cost: upstream **shells out to `ffmpeg`/`ffprobe`**, so a - consumer needs those binaries, which makes the feature untestable on a CI runner without them. + What is still missing is the content part. Upstream's wire type is + `{"type":"input_video","input_video":{"data":""}}`, handled in + `oaicompat_chat_params_parse` and gated on `allow_video = mtmd_helper_support_video(mctx)`. Note it + calls `handle_media(..., accept_base64_uri = false)`, i.e. **raw base64 only** — unlike `image_url`, + it will not take a `data:` URI, so `ContentPart.videoFile(Path)` must emit the bare base64 payload, + not the `data:video/mp4;base64,...` form the image factories build. + + (An earlier draft of this entry suggested smuggling video through + `ContentPart.imageBytes(bytes, "video/mp4")`, on the reasoning that `mtmd_helper_bitmap_init_from_buf` + sniffs the container. That is plausible — the `image_url` branch does pass + `accept_base64_uri = true` and does not validate the MIME string — but it is **untested here** and + additionally gated on `allow_image`, so it is not documented as a supported route.) + + Remaining work: the factory, a bytes overload, and an integration test. Note the runtime cost: + upstream **shells out to `ffmpeg`/`ffprobe`**, so a consumer needs those binaries, which makes the + feature untestable on a CI runner without them. - **`--spec-synth-len` / `--spec-synth-rates` — deliberately NOT exposed, and this should stay that way.** Added in b10649. Upstream's own help text marks both **"(benchmarking only)"**: they diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 21cad48ff..bf5616cbb 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -680,4 +680,4 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10639–b10644 | `include/llama.h` (**`LLAMA_SESSION_VERSION` 9 → 10 and `LLAMA_STATE_SEQ_VERSION` 2 → 3** — the only two lines in the file that moved), `src/llama-kv-cells.h` + `src/llama-kv-cache.{cpp,h}` (`llama_kv_cell_ext` gains a `tok` field for n-gram input embeddings, plus `has_cell_ext()` / `get_prev_tokens()` / `for_each_token_in()`, and `reset()` swaps a `memset` for value-initialisation), `src/models/nanbeige.cpp` (one line: register `t_layer_inp[il]`), `tests/test-backend-ops.cpp`, the whole `ggml/src/ggml-hexagon/**` + `scripts/snapdragon/**` + `docs/backend/snapdragon/**` Snapdragon/Hexagon backend rework, plus `tools/ui/**` (the WebUI, auto-followed) | **No project-source change, and the only priority-review-list file touched is `include/llama.h`, whose entire diff is the two version macros.** Sizing: the raw diff is ~608 KiB, but it is dominated by two components this project never compiles or reviews — `tools/ui` (auto-followed; the `build-webui` job rebuilds it from the pin) and the Hexagon backend (`grep -rn GGML_HEXAGON` over this repo's workflows, build scripts and CMake is empty, so `ggml-hexagon.cpp` is never in a `libjllama` link line). Excluding both leaves **11 KiB across 6 files**, well under the runbook's 100 KiB single-step threshold, so no chunking was needed. **Zero** files under `common/`, `tools/server/` or `tools/mtmd/` changed, so every remaining row of the API-compatibility table is vacuously satisfied and the three mechanical server-contract greps have no input — the request-field set, its bounds and the emitted response keys cannot have moved. **Compatibility note for consumers, not a compile break:** the two version bumps are a *state-file format* break — a slot state saved by an older build (`LlamaModel.handleSlotAction` save, or the server's `/slots/{id}?action=save`) is rejected by its own version check on b10644 and must be regenerated. Nothing the project calls changed shape. **All 8 patches apply unchanged** — the 6 project-relevant changed files and the 42 files the patches target have **zero** overlap (no `common/arg.{cpp,h}`, no `common/peg-parser.cpp`, no `tools/server/*`; the one changed test, `tests/test-backend-ops.cpp`, is touched by no patch). | | b10639–b10644 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10644 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10644: all **8** patches applied, confirmed by the stamp file (`head d7a2074112d27649303fa107eb8c94db1ee435f3` plus one SHA-256 line per patch) — including `0011`, whose target `common/peg-parser.cpp` the range does not touch. Full `cmake --build --config Release` clean, `ctest` **509/509**. `nm -D` on the freshly linked `libjllama.so` reports **40** `Java_*` exports and **zero** C++-mangled ones. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10644") matches the `build-info` compiled into the new binary (`b10644-d7a207411`), which is what proves the four pin sites and the actual build agree. Run with `mvn clean test`, not a bare `mvn test`: `LLAMA_CPP_VERSION` is a compile-time constant that javac inlines into the already-compiled test class, and Maven's incremental compilation cannot see that dependency — the b10636→b10639 row below records the false drift that costs. | | b10644–b10649 | **`tools/mtmd/mtmd-helper.{h,cpp}` (BREAKING: `mtmd_helper_bitmap_init_from_file` and `_from_buf` gain a 4th `struct mtmd_helper_init_opt` parameter; new `mtmd_helper_video_init_params` + `mtmd_helper_init_opt` structs and their `_default()` factories)**, **`tools/server/server-common.{h,cpp}` (BREAKING: `tokenize_input_prompts`, `process_mtmd_prompt` and `format_prompt_rerank` all gain a trailing `const mtmd_helper_init_opt &`)**, `tools/server/server-context.{h,cpp}` (`handle_count_tokens` gains the same parameter; threading it through), `common/common.h` (**additive**: `common_params_speculative::{synth_len,synth_rates}` + `has_synth()`, `common_params::{video_fps,video_timestamp_interval_ms,video_ffmpeg_bin_dir}`), `common/arg.cpp` (**purely additive**: 6 new flags), `common/speculative.{cpp,h}` (**additive**: `common_speculative_n_max`, `_synth_rates_resolve`, `_get_synth_probs`), `ggml/src/ggml-metal/*`, `src/models/minimax-01.cpp`, `tests/test-arg-parser.cpp`, `tools/{cli,llama-bench,tts}`, `tools/server/tests/*` | **The first range in this whole bump that broke the project's own compile — four call sites, all from one upstream refactor.** llama.cpp b10649 threaded a new `mtmd_helper_init_opt` (video decode settings: fps target, ffmpeg binary dir, timestamp interval) through every helper that can ingest media. `tts_engine.cpp:95` (`mtmd_helper_bitmap_init_from_file`, the speaker-reference clip) and `jllama.cpp` ×3 (`tokenize_input_prompts`) + ×1 (`format_prompt_rerank`) all lost their signatures. Every one of those paths passes `mctx = nullptr` or handles audio, so none wants video settings: each now passes `mtmd_helper_init_opt_default()`, which is upstream's own default. **This is the failure class the priority table's note calls out** — `server-common.h` is a same-repo header the project `#include`s directly rather than one reachable through the documented dependency graph, so only a real compile finds it. **The wire contract did not move:** all three mechanical greps are identical across the range (68 request fields, 23 bounds, 286 response keys over all six server `.cpp`). `common/arg.cpp` removed or renamed **zero** flags. Sizing: 106 KiB reviewable, or **72 KiB excluding `ggml-metal`** (Metal backend internals behind unchanged public headers — same exclusion rationale as Hexagon), under the runbook's 100 KiB single-step threshold. **All 8 patches apply unchanged** despite six patch-target files changing (`common/arg.cpp`, `tests/test-arg-parser.cpp`, `tools/mtmd/mtmd-cli.cpp`, `tools/server/server-context.{cpp,h}`, `tools/tts/tts.cpp`) — upstream's edits fall outside every patched region. | -| b10644–b10649 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10649 -DBUILD_TESTING=ON` through the real `FetchContent` path: all **8** patches applied (stamp `head 2bb9bddafad44ecbb50889644ca47537ec11841b`), which was the first thing checked because six patch-target files changed in the range. The first build failed loud with the four signature breaks above; after adapting the call sites, Release build clean, `ctest` **512/512**, `nm -D` **40** `Java_*` exports and **zero** C++-mangled, `NativeLibraryLoadSmokeTest` **3/3, 0 skipped** including the pin cross-check against the linked `build-info`. `ModelParametersTest` 73/73 with the five new CPU-offload cases. **Feature exposure decided deliberately, not by default:** of the 6 new flags, only `--n-cpu-ffn` is genuinely new upstream, and it was added as `ModelParameters.setCpuFfnLayers`; `setCpuMoeLayers` was added alongside it for `--n-cpu-moe`, which has existed upstream since **b6089** (#15077, `ec428b02c`) but had never been exposed here — b10649 only refactored its lambda onto the shared `llm_add_n_cpu_ffn_overrides` helper; the two `--spec-synth-*` flags were refused as upstream marks them "benchmarking only"; and the three `--video-*` flags were initially refused as inert without a `ContentPart` video factory. **That refusal was wrong and was reversed by the follow-up audit:** they are not inert — `server_context::load_model` copies them into its own `init_opt` when the projector loads, and that `init_opt` is what `server-context.cpp` hands to `process_mtmd_prompt` on the very task path this binding drives (`task.cli_files`), so they take effect for any attached media; and video decoding really is compiled in (`MTMD_VIDEO` defaults `ON`, gated only on `LLAMA_SUBPROCESS` which is also `ON`, and the shipped `libjllama.so` carries the ffmpeg invocation strings). They are now `ModelParameters.setVideoFps` / `setVideoTimestampInterval` / `setVideoFfmpegDir`; `setVideoFfmpegDir` matters most, since upstream otherwise resolves `ffmpeg`/`ffprobe` from `PATH`, which a JVM process frequently lacks. Only the ergonomic `ContentPart.videoFile(...)` entry point is still outstanding (raw bytes already work — the decoder sniffs the container rather than trusting the MIME type). The `--spec-synth-*` refusal stands. Both are recorded in `TODO.md` with their reasoning. | +| b10644–b10649 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10649 -DBUILD_TESTING=ON` through the real `FetchContent` path: all **8** patches applied (stamp `head 2bb9bddafad44ecbb50889644ca47537ec11841b`), which was the first thing checked because six patch-target files changed in the range. The first build failed loud with the four signature breaks above; after adapting the call sites, Release build clean, `ctest` **516/516** (512 at the bump; the second audit added four `TrainParams` cases), `nm -D` **40** `Java_*` exports and **zero** C++-mangled, `NativeLibraryLoadSmokeTest` **3/3, 0 skipped** including the pin cross-check against the linked `build-info`. `ModelParametersTest` 77/77 with the five CPU-offload cases plus the three video-flag cases. **Feature exposure decided deliberately, not by default:** of the 6 new flags, only `--n-cpu-ffn` is genuinely new upstream, and it was added as `ModelParameters.setCpuFfnLayers`; `setCpuMoeLayers` was added alongside it for `--n-cpu-moe`, which has existed upstream since **b6089** (#15077, `ec428b02c`) but had never been exposed here — b10649 only refactored its lambda onto the shared `llm_add_n_cpu_ffn_overrides` helper; the two `--spec-synth-*` flags were refused as upstream marks them "benchmarking only"; and the three `--video-*` flags were initially refused as inert without a `ContentPart` video factory. **That refusal was wrong and was reversed by the follow-up audit:** they are not inert — `server_context::load_model` copies them into its own `init_opt` when the projector loads, and that `init_opt` is what `server-context.cpp` hands to `process_mtmd_prompt` on the very task path this binding drives (`task.cli_files`), so they take effect for any attached media; and video decoding really is compiled in (`MTMD_VIDEO` defaults `ON`, gated only on `LLAMA_SUBPROCESS` which is also `ON`, and the shipped `libjllama.so` carries the ffmpeg invocation strings). They are now `ModelParameters.setVideoFps` / `setVideoTimestampInterval` / `setVideoFfmpegDir`; `setVideoFfmpegDir` matters most, since upstream otherwise resolves `ffmpeg`/`ffprobe` from `PATH`, which a JVM process frequently lacks. Only the ergonomic `ContentPart.videoFile(...)` entry point is still outstanding (raw bytes already work — the decoder sniffs the container rather than trusting the MIME type). The `--spec-synth-*` refusal stands. Both are recorded in `TODO.md` with their reasoning. | diff --git a/llama/src/main/cpp/jllama.cpp b/llama/src/main/cpp/jllama.cpp index 3edd0c790..1e39e0a84 100644 --- a/llama/src/main/cpp/jllama.cpp +++ b/llama/src/main/cpp/jllama.cpp @@ -273,6 +273,14 @@ static void populate_completion_task(server_task &task, jllama_context *jctx, // // Used at every post site in this file. Sleeping is off by default (-1), in which case // wait_until_no_sleep() sees a non-sleeping queue and returns immediately. +// +// Ordering invariant this depends on: wait_until_no_sleep() blocks until the worker loop clears +// `sleeping`, and upstream's loop leaves that flag SET when it exits on !running -- it breaks out +// of the sleep block before the `sleeping = false; notify_all();` pair. So a caller parked here +// when the queue is terminated is never released. It is safe only because every entry point holds +// a jllama_context_guard user reference and Java_..._delete waits for users == 0 BEFORE calling +// terminate(), so the loop is always alive to service this wait. Do not reorder that teardown to +// terminate first. static void wake_and_post(server_response_reader &rd, server_task &&task, bool front = false) { rd.queue_tasks.wait_until_no_sleep(); rd.post_task(std::move(task), front); From 5859a6d8e8d8b90d036f324452786f82a7b0d34b Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 23:04:28 +0000 Subject: [PATCH 70/76] fix: honour upstream's video-fps sentinel; narrow the mmproj clearing A workflow of 25 agents re-audited this branch: 39 findings, 18 upheld after adversarial verification. Two were real defects in code this PR itself added, both confirmed here against pristine upstream b10649. setVideoFps rejected every value <= 0, but mtmd-helper.h documents fps_target as "<= 0 means use the video's native fps" and the decoder resolves it as `fps_target = arg > 0 ? arg : orig_fps`. The guard deleted the only way to say "match this clip's own rate" -- the target is fixed when the projector loads, long before a clip is attached, so a caller cannot type the rate in instead. It now rejects only NaN and infinity; infinity is the value that genuinely breaks, reaching ffmpeg as "fps=inf". The same header shows `timestamp_interval_ms <= 0 means NO timestamp`, so the test comment claiming 0 meant "every frame" was backwards. setMmprojOffload dropped --mmproj-device unconditionally. Only some combinations actually clash: a named device and --mmproj-offload both resolve to (use_gpu=true, device=named) in either argv order, so clearing there threw away a real multi-GPU pin to set a field that already defaults to true. Clearing is now restricted to the pairs whose meaning depends on argv order -- any device against --no-mmproj-offload, and "none" (which sets use_gpu=false) against --mmproj-offload. Also bounded setVideoTimestampInterval at INT_MAX: the upstream field is int64_t but the flag is registered with an int handler dispatched through std::stoi, so a larger value aborts the whole argv parse and surfaces only as "Failed to parse model parameters", naming neither flag nor reason. Tests, each verified to fail on the regression it guards: - ModelParametersTest: the two tests that asserted the defects are replaced by ones asserting the sentinel passes through, plus non-finite rejection, the INT_MAX boundary, and both directions of the mmproj rule. - test_tts_params.cpp: TrainParams.MapsTheContextAndScheduleFields covers the seven build_train_params fields no other test reads. Mutation-checked -- deleting the decay_epochs assignment reds it, and nothing else. - test_json_helpers.cpp: the fractional, above-INT_MAX and exactly-INT_MAX terms of parse_positive_int_config, previously covered only for raw <= 0. - test_tts_wav.cpp: the four put_u16 fmt fields and byte_rate. No test read a u16 from the header, so the s390x big-endian gate could not see a put_u16 byte-order regression at all. - TrainingParametersTest: the LR-schedule wire keys, whose native fallbacks are byte-identical to the Java defaults -- a renamed key silently reverts the schedule instead of failing. - LangChain4jMappingTest: pins that no "use_jinja" is emitted, the one remaining unguarded site of the three no-op calls this PR removed. Deferred to TODO.md rather than widening the bump: five ModelParameters flags the server arg parser rejects (--grp-attn-* are example-scoped away from SERVER, so a grep sweep cannot see them), the context refcount helpers, OSInfo's archMapping aliases, the trainer's untested end-to-end path, LlamaLoader's jar internals, and dead Java8CompatibilityHelper code. Reactor verify green (SpotBugs 0, enforcer, javadoc), ctest 520/520, 1456 Java + 46 langchain4j tests, clang-format clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CLAUDE.md | 6 +- TODO.md | 59 +++++++++++++ .../langchain4j/LangChain4jMappingTest.java | 5 ++ .../llama/parameters/ModelParameters.java | 69 +++++++++++---- llama/src/test/cpp/test_json_helpers.cpp | 19 ++++ llama/src/test/cpp/test_tts_params.cpp | 40 +++++++++ llama/src/test/cpp/test_tts_wav.cpp | 17 ++++ .../llama/parameters/ModelParametersTest.java | 88 +++++++++++++++---- .../parameters/TrainingParametersTest.java | 27 ++++++ 9 files changed, 289 insertions(+), 41 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 685f10fdd..932fc25ca 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1437,13 +1437,13 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" |------|-------|-------| | `src/test/cpp/test_utils.cpp` | 167 | Upstream helpers: `server_tokens`, `server_grammar_trigger`, `gen_tool_call_id`, `json_value`, `json_get_nested_values`, UTF-8 helpers, `format_response_rerank`, `format_embeddings_response_oaicompat`, `oaicompat_completion_params_parse`, `oaicompat_chat_params_parse`, `are_lora_equal`, `strip_flag_from_argv`, `token_piece_value`, `json_is_array_and_contains_numbers`, `format_oai_sse`, `format_oai_resp_sse`, `format_anthropic_sse`, `parse_lora_request`, `common_chat_parse` over malformed UTF-8 (the `ContentOnlyParseUtf8` guard for `patches/0011`) | | `src/test/cpp/test_server.cpp` | 206 | Upstream result types: `server_slot_stats` (the `timings` JSON payload; replaced `result_timings` in b10408), `task_params::to_json()` (incl. `dry_sequence_breakers`, `preserved_tokens`, `timings_per_token`), `completion_token_output`, `server_task_result_cmpl_partial` (non-oaicompat + `to_json_oaicompat` + logprobs + `to_json_oaicompat_chat` + `to_json_anthropic` + dispatcher), `server_task_result_cmpl_final` (non-oaicompat + `to_json_oaicompat` + `to_json_oaicompat_chat` + `to_json_oaicompat_chat_stream` + `to_json_anthropic` + `to_json_anthropic_stream` + tool_calls + dispatcher), `server_task_result_embd`, `server_task_result_rerank`, `server_task_result_metrics` (`to_metrics()` = the `/metrics` Prometheus exposition text; its `to_json()` has been unused since b10519 and returns `json{}` = JSON null), `server_task_result_slots` (`to_json()` = the `/slots` array, fed by the b10519 `SERVER_TASK_TYPE_SLOT_GET` task), `server_task_result_slot_save_load`, `server_task_result_slot_erase`, `server_task_result_apply_lora`, `server_task_result_get_lora`, `server_task_result_error`, `format_error_response`, `server_task::need_sampling()`, `server_task::n_tokens()`, `server_schema::eval_llama_cmpl_schema()` (parsing pipeline + grammar routing + error paths + per-request `dry_*` and `sse_ping_interval` field round-trips incl. hard-limit + server-default inheritance), `response_fields` projection | -| `src/test/cpp/test_json_helpers.cpp` | 60 | All functions in `json_helpers.hpp`: `get_result_error_message`, `results_to_json`, `rerank_results_to_json` (incl. missing/out-of-range `index` rejection), `parse_encoding_format`, `extract_embedding_prompt`, `is_infill_request`, `parse_slot_prompt_similarity`, `parse_positive_int_config`, `wrap_stream_chunk`, `server_metrics_to_json` | +| `src/test/cpp/test_json_helpers.cpp` | 63 | All functions in `json_helpers.hpp`: `get_result_error_message`, `results_to_json`, `rerank_results_to_json` (incl. missing/out-of-range `index` rejection), `parse_encoding_format`, `extract_embedding_prompt`, `is_infill_request`, `parse_slot_prompt_similarity`, `parse_positive_int_config`, `wrap_stream_chunk`, `server_metrics_to_json` | | `src/test/cpp/test_log_helpers.cpp` | 13 | All functions in `log_helpers.hpp`: `log_level_name`, `format_log_as_json` | | `src/test/cpp/test_jni_helpers.cpp` | 56 | All functions in `jni_helpers.hpp` using a zero-filled `JNINativeInterface_` mock (incl. the `utf8_to_jstring_impl` byte-array string path: emoji byte-preservation, truncated-UTF-8 replace-not-throw) | | `src/test/cpp/test_tts_wav.cpp` | 2 | The in-memory WAV writer `pcm_to_wav16_bytes` in `tts_wav.hpp` (WAV header/payload + little-endian clamping) — our own code, not upstream. The Qwen3-TTS pipeline it pairs with (`mtmd_helper::gen_audio`) is entirely upstream-owned (no project-side DSP to unit-test here). The load path is additionally covered by `test_tts_params.cpp` (5 tests over `tts_params.hpp`'s `build_tts_params`), which pins the CPU-thread resolution whose absence used to crash the JVM on every platform — see the `TODO.md` entry for the mechanism. End-to-end coverage is `TtsIntegrationTest`, which is model-gated. | -| `src/test/cpp/test_tts_params.cpp` | 8 | The two builders every hand-assembled `common_params` goes through: `build_tts_params` (`tts_params.hpp`, 5 tests) and the shared `jllama::resolve_cpu_params` (`cpu_params.hpp`, 3 tests). The shared one is what guards **`train_engine.cpp`** too — its own integration test needs a GGUF no CI job downloads, so without these the JVM-abort bug could regress in the trainer on every platform, unseen. | +| `src/test/cpp/test_tts_params.cpp` | 13 | The **three** builders every hand-assembled `common_params` goes through: `build_tts_params` (`tts_params.hpp`), `build_train_params` (`train_params.hpp`) and the shared `jllama::resolve_cpu_params` (`cpu_params.hpp`). Each builder is guarded separately on purpose — testing the resolver alone does **not** cover its call sites, because `train_engine.cpp` is compiled into `jllama` only, never into `jllama_test`, and `LlamaTrainerIntegrationTest` is gated on `net.ladenthin.llama.train.model`, which no CI job sets. Without these the JVM-abort bug could regress in the trainer on every platform, unseen. | -**Current total: 512 tests (all passing).** +**Current total: 520 tests (all passing).** #### Upstream source location (in CMake build tree) diff --git a/TODO.md b/TODO.md index e9d964187..be61ebf96 100644 --- a/TODO.md +++ b/TODO.md @@ -443,6 +443,65 @@ the load-time failure class is already covered, and a slow smoke tends to get ma **Not yet observed green in CI** — the job and the two sibling-repo smokes landed in one change set and have only run locally so far. +### Test-coverage debt found during the b10649 review (PR #403) + +Each item below was verified against pristine upstream tags and is real, but none is a regression +introduced by the version bump — they were deferred to keep that PR landable. + +- **`ModelParameters` emits five CLI flags the server arg parser rejects, so any caller of them + cannot load a model.** `--dump-kv-cache`, `--hf-repo-v` and `--hf-file-v` no longer exist anywhere + in llama.cpp (absent at both b10456 and b10649); `--grp-attn-n` and `--grp-attn-w` still exist but + are `set_examples({LLAMA_EXAMPLE_COMPLETION, ...})`, so `add_opt` never registers them for + `LLAMA_EXAMPLE_SERVER` — the example jllama parses with. An unregistered flag is not ignored: + `arg.cpp` throws, `common_params_parse` returns false, and `load_model_impl` throws + `LlamaException("Failed to parse model parameters")`. Four existing tests pin the dead literals and + would pass forever. Fix: deprecate the five members the way this PR handled + `withTfsZ`/`withPenalizeNl` (keep source compatibility, never write the map), and add a hermetic + `jllama_test` contract test that walks `common_params_parser_init(params, LLAMA_EXAMPLE_SERVER)`'s + `ctx.options` (upstream's own `test-arg-parser` pattern; the symbols already link into + `jllama_test`) and asserts every flag `ModelParameters`/`ModelFlag` can emit is in that set, + excluding only `--vocab-only`, which `strip_flag_from_argv` removes on purpose. A grep-based sweep + is **not** sufficient — it is structurally blind to example scoping, which is exactly how + `--grp-attn-w` hides. + +- **`acquire_jllama_context_impl` / `release_jllama_context_impl` / `jllama_context_guard` have no + model-free unit guard.** These three (`jni_helpers.hpp`) are the whole `close()`-vs-inference + use-after-free defence, and grep finds zero references across all seven `test_*.cpp` files, while + their sibling `get_jllama_context_impl` has three tests. A dropped `fetch_add`, or a guard whose + destructor stops calling release, produces a use-after-free during `close()` or a `close()` that + hangs forever. `LlamaModelTest#testCloseDuringInference` covers the mechanism end to end but only + bluntly. They are absent from `jllama_test` only because they are `inline` and never odr-used + there: `g_ctx_mutex` is `extern` in the header and defined in `jllama.cpp`, which `jllama_test` + does not compile — a test-local definition at global scope unblocks it. + +- **`OSInfo`: the `archMapping` alias branch is untested.** `getArchName()`'s map lookup has no + assertion anywhere — the two test call sites either take the override early-return or only assert + non-empty — so a lost `amd64 -> x86_64` entry would send `LlamaLoader` to a resource directory + that does not exist. Cheap to close: set `os.arch`, assert the non-identity aliases only (identity + entries such as `s390x` are behaviourally redundant with the `\W`-stripping fallback). + +- **`LlamaTrainer`'s end-to-end path runs on no CI platform.** `LlamaTrainerIntegrationTest` + self-skips everywhere: `net.ladenthin.llama.train.model` is set by no job and its model is in no + `.github/models.csv` row, so `validate-models.{sh,bat}` does not treat it as required. The C++ half + is now mitigated (`test_tts_params.cpp`'s `TrainParams` + `ResolveCpuParams` suites), but nothing + exercises the Java → JNI → native trainer round trip. Adding a small training model to `models.csv` + plus the matching property to the Java test jobs would close it. + +- **`LlamaLoader`'s jar-extraction internals need synthetic jar fixtures.** `readBackendManifest`, + `tryLoadBackend`, `extractFile`, `moveIntoPlace`, `cleanPath` and `hasNativeLib` are named in no + test; `BackendManifestLoadTest` and `LlamaLoaderTest` drive the class only from outside via system + properties. Covering the multi-backend fat-jar path (per-backend temp subdir extraction, + manifest-extras-first ordering, `UnsatisfiedLinkError` fallback to the next backend and then to the + default CPU natives) means building jars carrying a `jllama-backends.txt` and dummy payloads. + +- **`Java8CompatibilityHelper` is mostly dead code — decide delete vs. test.** Six of its seven + public methods have zero call sites repo-wide; the only live one is + `toString(ByteArrayOutputStream, Charset)`, used once in `ProcessRunner`. Writing tests for the + rest would pin dead code. + +- **`ContentPart.videoFile(...)` — see the video-input entry above** for the wire shape upstream + expects (`input_video`, raw base64, not a `data:` URI). + ## Open — cross-cutting (slice for this repo) - **jqwik pin policy** — see [`../workspace/policies/jqwik-prompt-injection.md`](../workspace/policies/jqwik-prompt-injection.md). `jqwik.version ≤ 1.9.3` is mandatory. diff --git a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/LangChain4jMappingTest.java b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/LangChain4jMappingTest.java index 40cea27c5..e495e4c8c 100644 --- a/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/LangChain4jMappingTest.java +++ b/llama-langchain4j/src/test/java/net/ladenthin/llama/langchain4j/LangChain4jMappingTest.java @@ -218,6 +218,11 @@ void streamingParametersCarryToolsAndToolChoice() { assertThat(json, containsString("get_weather")); assertThat(json, containsString("\"tool_choice\"")); assertThat(json, containsString("required")); + // Jinja is a load-time option (--jinja). Upstream's request parser never reads a "use_jinja" + // key and silently discards unknown fields, so a re-added withUseChatTemplate(true) here + // would be invisible at runtime and uncatchable by any integration test. Mirrors + // OpenAiRequestMapperTest#toolsEnableChatTemplateAndForwardChoice. + assertThat(json, not(containsString("\"use_jinja\""))); } @Test diff --git a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java index 72ce2ecf8..679b6c3bc 100644 --- a/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java +++ b/llama/src/main/java/net/ladenthin/llama/parameters/ModelParameters.java @@ -23,6 +23,9 @@ @EqualsAndHashCode(callSuper = true) public final class ModelParameters extends CliParameters { + private static final String ARG_MMPROJ_DEVICE = "--mmproj-device"; + private static final String MMPROJ_DEVICE_NONE = "none"; + private static final String ARG_FIT = "--fit"; static final String ARG_POOLING = "--pooling"; /** CLI value enabling {@code --fit} (automatic device-memory fitting). */ @@ -1420,14 +1423,21 @@ public ModelParameters setMmprojAuto(boolean enabled) { * @return this builder */ public ModelParameters setMmprojDevice(String device) { - parameters.put("--mmproj-device", device); - // --mmproj-device and --{no-,}mmproj-offload write the same upstream field - // (common_params::mmproj_use_gpu), so upstream resolves a clash by argv order. Our argv is - // rendered from a HashMap, where that order is unspecified -- so the pair must never both be - // present. setMmprojOffload clears this key in turn, making the rule simply "the last of the - // two calls wins". + parameters.put(ARG_MMPROJ_DEVICE, device); + // --mmproj-device and --{no-,}mmproj-offload both write common_params::mmproj_use_gpu, so a + // clash is resolved by argv order -- and ours is rendered from a HashMap, where that order is + // unspecified. Clear only the genuinely conflicting combinations: + // + // named device + --no-mmproj-offload -> (true,dev) vs (false,dev): ORDER MATTERS, clear. + // named device + --mmproj-offload -> (true,dev) either way: safe, keep both. + // "none" + --mmproj-offload -> (true,null) vs (false,null): ORDER MATTERS, clear. + // + // Clearing --mmproj-offload unconditionally would throw away a legitimate multi-GPU pin to + // set a field that already defaults to true. clearFlag(ModelFlag.NO_MMPROJ_OFFLOAD); - clearFlag(ModelFlag.MMPROJ_OFFLOAD); + if (MMPROJ_DEVICE_NONE.equals(device)) { + clearFlag(ModelFlag.MMPROJ_OFFLOAD); + } return this; } @@ -1449,13 +1459,22 @@ public ModelParameters enableMmprojOffload() { * time. Applies only when a projector is loaded ({@link #setMmproj(String)}) and the request * carries media — it is read once, when the projector is initialised.

    * - * @param fps frames per second to sample from the video; must be positive + *

    Any value {@code <= 0} is upstream's sentinel for "sample at the video's own native + * frame rate", not an error — {@code mtmd-helper.h} documents the field as + * "desired output fps; <= 0 means use the video's native fps", and the decoder resolves it as + * {@code fps_target = arg > 0 ? arg : orig_fps}. Because the target is fixed when the projector + * loads, long before any clip is attached, the sentinel is the only way to say "match whatever + * this video runs at"; leaving the value unset selects the fixed 4.0 default instead.

    + * + * @param fps frames per second to sample from the video, or any value {@code <= 0} to sample at + * the video's native frame rate; must be finite * @return this builder - * @throws IllegalArgumentException if {@code fps} is not positive + * @throws IllegalArgumentException if {@code fps} is {@code NaN} or infinite */ public ModelParameters setVideoFps(float fps) { - if (!(fps > 0.0f)) { - throw new IllegalArgumentException("Invalid video-fps value: " + fps + " (must be > 0)"); + if (!Float.isFinite(fps)) { + throw new IllegalArgumentException( + "Invalid video-fps value: " + fps + " (must be finite; <= 0 selects the video's native fps)"); } return putScalar("--video-fps", fps); } @@ -1466,13 +1485,20 @@ public ModelParameters setVideoFps(float fps) { * * @param intervalMillis milliseconds between timestamps; must not be negative * @return this builder - * @throws IllegalArgumentException if {@code intervalMillis} is negative + * @throws IllegalArgumentException if {@code intervalMillis} is negative or exceeds + * {@link Integer#MAX_VALUE} */ public ModelParameters setVideoTimestampInterval(long intervalMillis) { if (intervalMillis < 0) { throw new IllegalArgumentException( "Invalid video-timestamp-interval value: " + intervalMillis + " (must be >= 0)"); } + if (intervalMillis > Integer.MAX_VALUE) { + throw new IllegalArgumentException("Invalid video-timestamp-interval value: " + intervalMillis + + " (must be <= " + Integer.MAX_VALUE + + "; llama.cpp parses --video-timestamp-interval with std::stoi although the field is int64_t," + + " so a larger value aborts the whole argv parse with an unrelated message)"); + } return putScalar("--video-timestamp-interval", intervalMillis); } @@ -1497,10 +1523,11 @@ public ModelParameters setVideoFfmpegDir(String directory) { * Enable or disable GPU offload for the multimodal projector. This is independent of * {@link #setGpuLayers(int)} because upstream enables projector offload by default. * - *

    Clears any device previously named by {@link #setMmprojDevice(String)}: both write - * upstream's single {@code mmproj_use_gpu} field, so only one of them may appear in the - * rendered argv. Between the two, the last call wins — call {@code setMmprojDevice} - * afterwards to pin a device again.

    + *

    Both this and {@link #setMmprojDevice(String)} write upstream's single + * {@code mmproj_use_gpu} field, so a combination whose meaning would depend on argv order is + * not allowed to survive: disabling offload clears a previously named device, and enabling it + * clears only the {@code "none"} sentinel. Enabling offload alongside a real device name is + * kept, because both orderings resolve to the same state.

    * * @param enabled {@code true} to offload the projector, {@code false} to keep it on CPU * @return this builder @@ -1508,9 +1535,13 @@ public ModelParameters setVideoFfmpegDir(String directory) { public ModelParameters setMmprojOffload(boolean enabled) { setFlag(enabled ? ModelFlag.MMPROJ_OFFLOAD : ModelFlag.NO_MMPROJ_OFFLOAD); clearFlag(enabled ? ModelFlag.NO_MMPROJ_OFFLOAD : ModelFlag.MMPROJ_OFFLOAD); - // Also drop any --mmproj-device: it writes the same upstream field, and with a HashMap- - // rendered argv the winner between the two would be hash order. See setMmprojDevice. - parameters.remove("--mmproj-device"); + // Keep a previously named device only where the pair resolves identically in either argv + // order -- that needs enabled == true AND a real device name. "none" sets mmproj_use_gpu + // false and so contradicts --mmproj-offload, and --no-mmproj-offload contradicts any device; + // in those cases the last call wins. See setMmprojDevice for the full table. + if (!enabled || MMPROJ_DEVICE_NONE.equals(parameters.get(ARG_MMPROJ_DEVICE))) { + parameters.remove(ARG_MMPROJ_DEVICE); + } return this; } diff --git a/llama/src/test/cpp/test_json_helpers.cpp b/llama/src/test/cpp/test_json_helpers.cpp index d3b0459e0..d82f47f35 100644 --- a/llama/src/test/cpp/test_json_helpers.cpp +++ b/llama/src/test/cpp/test_json_helpers.cpp @@ -468,6 +468,25 @@ TEST(ParsePositiveIntConfig, ErrorMessage_ContainsKeyName) { } } +TEST(ParsePositiveIntConfig, FractionalValue_ThrowsInvalidArgument) { + // Pins the `raw != std::floor(raw)` term: a non-integral value must be rejected rather than + // silently truncated by static_cast. + EXPECT_THROW((void)parse_positive_int_config({{"n_threads", 1.5}}, "n_threads"), std::invalid_argument); +} + +TEST(ParsePositiveIntConfig, AboveIntMax_ThrowsInvalidArgument) { + // Pins the upper-bound term. 3e9 is whole-numbered, so the std::floor term cannot catch it; + // without this bound static_cast(3e9) is undefined behaviour. + EXPECT_THROW((void)parse_positive_int_config({{"n_threads", 3000000000.0}}, "n_threads"), std::invalid_argument); +} + +TEST(ParsePositiveIntConfig, ExactlyIntMax_IsAccepted) { + // Boundary: INT_MAX itself is valid, so the guard must stay `>` and never become `>=`. + auto v = parse_positive_int_config({{"n_threads", 2147483647}}, "n_threads"); + ASSERT_TRUE(v.has_value()); + EXPECT_EQ(*v, 2147483647); +} + // ============================================================ // wrap_stream_chunk // ============================================================ diff --git a/llama/src/test/cpp/test_tts_params.cpp b/llama/src/test/cpp/test_tts_params.cpp index f5f0b480f..e489fe90b 100644 --- a/llama/src/test/cpp/test_tts_params.cpp +++ b/llama/src/test/cpp/test_tts_params.cpp @@ -211,3 +211,43 @@ TEST(TrainParams, EpochsBelowOneAreClampedSoTheOptimizerRunsAtLeastOnce) { EXPECT_EQ(jllama_train::build_train_params(cfg).lr.epochs, 1u); } + +namespace { + +// Every value differs from the matching common_params / lr_opt default (n_ctx 0, n_gpu_layers -1, +// val_split 0.05f, lr0 1e-5, lr_min -1, decay_epochs -1, wd 0), so a deleted assignment is +// observable. minimal_train_config()'s lr fields are byte-identical to the lr_opt defaults, which is +// why it cannot guard them itself. +// +// epochs (4) MUST stay above decay_epochs (2.5): lr_opt::init() overwrites decay_epochs with epochs +// unless 0 < decay_epochs < epochs, which would both break the decay_epochs assertion and make a +// deleted decay_epochs assignment unobservable. +jllama_train::finetune_config distinctive_train_config() { + jllama_train::finetune_config cfg = minimal_train_config(); + cfg.epochs = 4; + cfg.learning_rate = 3e-4f; + cfg.lr_min = 7e-6f; // > 0 and < lr0 -- the only case in which lr_opt::init() does anything + cfg.decay_epochs = 2.5f; + cfg.weight_decay = 0.125f; + cfg.n_ctx = 1024; + cfg.n_gpu_layers = 5; + cfg.val_split = 0.25f; + return cfg; +} + +} // namespace + +// The seven cfg fields the other TrainParams tests never look at. Deleting any one of the matching +// assignments in build_train_params leaves every other test in this file green. +TEST(TrainParams, MapsTheContextAndScheduleFields) { + const common_params params = jllama_train::build_train_params(distinctive_train_config()); + + EXPECT_EQ(params.n_ctx, 1024); + EXPECT_EQ(params.n_gpu_layers, 5); + EXPECT_FLOAT_EQ(params.val_split, 0.25f); + EXPECT_FLOAT_EQ(params.lr.lr0, 3e-4f); + EXPECT_FLOAT_EQ(params.lr.lr_min, 7e-6f); + EXPECT_FLOAT_EQ(params.lr.wd, 0.125f); + // Survives init() only because epochs (4) > decay_epochs (2.5) -- see the fixture comment. + EXPECT_FLOAT_EQ(params.lr.decay_epochs, 2.5f); +} diff --git a/llama/src/test/cpp/test_tts_wav.cpp b/llama/src/test/cpp/test_tts_wav.cpp index efb5d9a73..cb672b1a0 100644 --- a/llama/src/test/cpp/test_tts_wav.cpp +++ b/llama/src/test/cpp/test_tts_wav.cpp @@ -21,6 +21,8 @@ uint32_t read_u32(const std::vector &b, size_t off) { std::string read_tag(const std::vector &b, size_t off) { return std::string(b.begin() + off, b.begin() + off + 4); } + +uint32_t read_u16(const std::vector &b, size_t off) { return (uint32_t)b[off] | ((uint32_t)b[off + 1] << 8); } } // namespace TEST(TtsWav, HeaderAndPayloadAreWellFormed) { @@ -37,6 +39,21 @@ TEST(TtsWav, HeaderAndPayloadAreWellFormed) { EXPECT_EQ(read_u32(wav, 24), 24000u); // sample rate EXPECT_EQ(read_u32(wav, 40), (uint32_t)(pcm.size() * 2)); // data size EXPECT_EQ(read_u32(wav, 4), 36u + (uint32_t)(pcm.size() * 2)); // RIFF chunk size + + // fmt-chunk fields written via put_u16 (offsets 20/22/32/34). No other test reads a u16 from the + // header, so without these the s390x big-endian ctest gate cannot observe a put_u16 byte-order + // regression at all -- every other assertion here is a u32 or an ASCII tag. + EXPECT_EQ(read_u16(wav, 20), 1u); // audio format = PCM + EXPECT_EQ(read_u16(wav, 22), 1u); // mono + EXPECT_EQ(read_u16(wav, 32), 2u); // block_align = num_channels * bits_per_sample/8 + EXPECT_EQ(read_u16(wav, 34), 16u); // bits_per_sample + + // byte_rate is the one computed u32 in the writer. Pin it at two sample rates so a hardcoded + // constant cannot satisfy it and a dropped factor is caught. + EXPECT_EQ(read_u32(wav, 28), 24000u * 2); // sample_rate * num_channels * bits_per_sample/8 + const std::vector wav16k = pcm_to_wav16_bytes(pcm, 16000); + EXPECT_EQ(read_u32(wav16k, 24), 16000u); + EXPECT_EQ(read_u32(wav16k, 28), 16000u * 2); } TEST(TtsWav, ClampsAndEncodesSamplesLittleEndian) { diff --git a/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java b/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java index f084b2ad9..b5456692d 100644 --- a/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java +++ b/llama/src/test/java/net/ladenthin/llama/parameters/ModelParametersTest.java @@ -520,41 +520,91 @@ public void testVideoDecodingFlagsRenderTheUpstreamNames() { } @Test - public void testVideoFpsRejectsNonPositiveValues() { - // Upstream parses this with std::stof and never validates it; a zero or negative target - // frame rate reaches the decoder, so reject it here where the message can name the flag. - assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setVideoFps(0.0f)); - assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setVideoFps(-1.0f)); + public void testVideoFpsNonPositiveSelectsUpstreamNativeFpsSentinel() { + // mtmd-helper.h documents fps_target as "<= 0 means use the video's native fps", and the + // decoder resolves it as `fps_target = arg > 0 ? arg : orig_fps`. Rejecting it here would + // delete the only way to say "match this clip's own rate" -- the target is fixed when the + // projector loads, long before a clip is attached. Same rule as --mmproj-device "none": an + // upstream sentinel is passed through verbatim. + assertThat(new ModelParameters().setVideoFps(0.0f).parameters.get("--video-fps"), is("0.0")); + assertThat(new ModelParameters().setVideoFps(-1.0f).parameters.get("--video-fps"), is("-1.0")); + } + + @Test + public void testVideoFpsRejectsNonFiniteValues() { + // Infinity survives std::stof and passes the decoder's `> 0` test, then reaches ffmpeg as + // the filter string "fps=inf". NaN takes the native-fps branch harmlessly but is always a + // caller bug. + assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setVideoFps(Float.NaN)); + assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setVideoFps(Float.POSITIVE_INFINITY)); + assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setVideoFps(Float.NEGATIVE_INFINITY)); } @Test public void testVideoTimestampIntervalRejectsNegativeValues() { assertThrows(IllegalArgumentException.class, () -> new ModelParameters().setVideoTimestampInterval(-1L)); - // Zero is legal: it means "a timestamp on every frame". + // Zero is legal and is upstream's "no timestamps" sentinel (mtmd-helper.h: "<= 0 means no + // timestamp"); the decoder gates emission on `timestamp_interval_ms > 0`. Negative is + // behaviourally identical upstream, so refusing it costs nothing and keeps the API honest. ModelParameters p = new ModelParameters().setVideoTimestampInterval(0L); assertThat(p.parameters.get("--video-timestamp-interval"), is("0")); } @Test - public void testMmprojDeviceAndOffloadClearEachOtherInBothDirections() { - // Both write upstream's single mmproj_use_gpu field, and our argv is rendered from a HashMap - // whose iteration order is unspecified -- so if both keys were present the winner would be - // hash order. The contract is "the last of the two calls wins". + public void testVideoTimestampIntervalRejectsValuesAboveIntMax() { + // The upstream field is int64_t but the flag is registered with an int handler, dispatched + // through std::stoi -- so a larger value throws out_of_range, common_params_parse returns + // false, and the caller sees only "Failed to parse model parameters", naming neither the + // flag nor the reason. + assertThrows( + IllegalArgumentException.class, + () -> new ModelParameters().setVideoTimestampInterval(Integer.MAX_VALUE + 1L)); + // INT_MAX itself is the largest value std::stoi accepts, so it must still render; without + // this the bound could be tightened to >= and still pass. + ModelParameters p = new ModelParameters().setVideoTimestampInterval(Integer.MAX_VALUE); + assertThat(p.parameters.get("--video-timestamp-interval"), is("2147483647")); + } + + @Test + public void testMmprojDeviceClearsOnlyTheContradictoryOffloadFlag() { + // --mmproj-device sets (use_gpu=true, device=named); --no-mmproj-offload sets + // use_gpu=false. Together the result depends on argv order, and ours is HashMap-rendered, + // so the flag must go. ModelParameters deviceLast = new ModelParameters().setMmprojOffload(false).setMmprojDevice("CUDA1"); assertThat(deviceLast.parameters.get("--mmproj-device"), is("CUDA1")); assertThat(deviceLast.parameters, not(hasKey("--no-mmproj-offload"))); - assertThat(deviceLast.parameters, not(hasKey("--mmproj-offload"))); - ModelParameters offloadLast = - new ModelParameters().setMmprojDevice("CUDA1").setMmprojOffload(false); - assertThat(offloadLast.parameters, hasKey("--no-mmproj-offload")); - assertThat(offloadLast.parameters, not(hasKey("--mmproj-device"))); + // ... but --mmproj-offload agrees with a named device: both orders give (true, CUDA1), so + // dropping either would discard a real multi-GPU pin for no reason. + ModelParameters agreeing = new ModelParameters().setMmprojOffload(true).setMmprojDevice("CUDA1"); + assertThat(agreeing.parameters.get("--mmproj-device"), is("CUDA1")); + assertThat(agreeing.parameters, hasKey("--mmproj-offload")); + + // "none" is the exception: it sets use_gpu=false, so it does contradict --mmproj-offload. + ModelParameters none = new ModelParameters().setMmprojOffload(true).setMmprojDevice("none"); + assertThat(none.parameters.get("--mmproj-device"), is("none")); + assertThat(none.parameters, not(hasKey("--mmproj-offload"))); + } - ModelParameters enabledLast = - new ModelParameters().setMmprojDevice("CUDA1").setMmprojOffload(true); - assertThat(enabledLast.parameters, hasKey("--mmproj-offload")); - assertThat(enabledLast.parameters, not(hasKey("--mmproj-device"))); + @Test + public void testMmprojOffloadClearsOnlyAContradictoryDevice() { + // Disabling offload contradicts any device name -> last call wins, device goes. + ModelParameters disabled = + new ModelParameters().setMmprojDevice("CUDA1").setMmprojOffload(false); + assertThat(disabled.parameters, hasKey("--no-mmproj-offload")); + assertThat(disabled.parameters, not(hasKey("--mmproj-device"))); + + // Enabling it does NOT: (true, CUDA1) either way, so the pin survives. + ModelParameters enabled = new ModelParameters().setMmprojDevice("CUDA1").setMmprojOffload(true); + assertThat(enabled.parameters, hasKey("--mmproj-offload")); + assertThat(enabled.parameters.get("--mmproj-device"), is("CUDA1")); + + // Except against "none", which means use_gpu=false. + ModelParameters noneThenEnable = + new ModelParameters().setMmprojDevice("none").setMmprojOffload(true); + assertThat(noneThenEnable.parameters, hasKey("--mmproj-offload")); + assertThat(noneThenEnable.parameters, not(hasKey("--mmproj-device"))); } @Test diff --git a/llama/src/test/java/net/ladenthin/llama/parameters/TrainingParametersTest.java b/llama/src/test/java/net/ladenthin/llama/parameters/TrainingParametersTest.java index d56b53c9b..a982928bb 100644 --- a/llama/src/test/java/net/ladenthin/llama/parameters/TrainingParametersTest.java +++ b/llama/src/test/java/net/ladenthin/llama/parameters/TrainingParametersTest.java @@ -67,7 +67,34 @@ void customValuesSerialize() throws Exception { assertThat(node.get("n_batch").asInt(), is(256)); assertThat(node.get("n_ubatch").asInt(), is(64)); assertThat(node.get("training_file").asText(), is("corpus.txt")); + assertThat(node.get("val_split").floatValue(), is(0.1f)); // training_text is omitted when a corpus file is given. assertThat(node.has("training_text"), is(false)); } + + /** + * The LR-schedule and regularization keys. {@code train_engine.cpp} reads these with fallbacks + * that are byte-identical to this class's own defaults, so a renamed key does not fail loudly -- + * it silently reverts the fine-tune to a schedule the caller never chose. The values below are + * therefore deliberately non-default and mutually distinct, so both a key rename and a + * copy-paste value swap fail this test. + */ + @Test + void learningScheduleKeysSerialize() throws Exception { + TrainingParameters parameters = TrainingParameters.builder() + .modelPath(Paths.get("base.gguf")) + .trainingText("hello world") + .outputPath(Paths.get("tuned.gguf")) + .lrMin(1e-6f) + .decayEpochs(3.0f) + .weightDecay(0.01f) + .valSplit(0.2f) + .build(); + + JsonNode node = json(parameters); + assertThat(node.get("lr_min").floatValue(), is(1e-6f)); + assertThat(node.get("decay_epochs").floatValue(), is(3.0f)); + assertThat(node.get("weight_decay").floatValue(), is(0.01f)); + assertThat(node.get("val_split").floatValue(), is(0.2f)); + } } From 267d9757b3d089b7ace2e4ca6517bfe5a7d881c8 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 22:09:29 +0000 Subject: [PATCH 71/76] feat: upgrade llama.cpp from b10649 to b10679 No project-source change was required, and that is a checked result: every tools/server/*.h header plus server-schema.cpp, server-task.cpp, server-common.cpp, common/chat.h and mtmd-helper.h are byte-identical across the range (compared by blob SHA), so the request-field set, its bounds, the emitted response keys and the mtmd surface cannot have moved. The in-scope delta is 172 lines over 8 files; the rest of the 159-file range is tools/ui (rebuilt from GIT_TAG by CI) and backends this project does not build. Two additive upstream features arrive and both are exposed: - setKvUnifiedPerSlot(int) for --kv-unified-per-slot. It is registered set_examples({LLAMA_EXAMPLE_SERVER}) and jllama.cpp parses with that example, so it binds; its cap half is applied in server-context.cpp's new n_ctx_slot(), whose value this binding reads through server_context_meta::slot_n_ctx and feeds to eval_llama_cmpl_schema, so it also moves where the repeat_last_n / dry_penalty_last_n sentinels expand to. The Javadoc is deliberately narrower than upstream's help text: upstream's second effect, sizing the shared KV pool to n_parallel * N when no context size is given, lives in llama_server() and therefore applies to NativeServer only, never to a model loaded from this builder. - setTensorReadLazy(TensorReadLazyMode) for --tensor-read-lazy, over a new args enum mirroring llama_lazy_mode (off/auto/on = 0/1/2). It has no set_examples restriction and common_model_params_to_llama copies lazy_mode into llama_model_params, so it reaches a plain LlamaModel load. Two further upstream changes need no adaptation: llama_model_quantize_params gained max_buf_size, but LlamaQuantizer builds its params from llama_model_quantize_default_params(); and the get_slot_n_ctx() -> n_ctx_slot() rename is a private member the project never calls. Patch 0001 shrinks from 37 to 36 files. Upstream rewrote tests/test-save-load-state.cpp's main() to strip --models DIR into its own filtered_argv before calling common_params_parse, so by that patch's own rule the site no longer wants the _main() flip; the hunk is dropped, not refreshed. The patch itself is still required, verified rather than assumed: common_params_parse in pristine b10679 still carries the count-guarded GetCommandLineW override and common_params_parse_main is absent from b10679:common/arg.h. Patches 0010 and 0011 were re-checked the same way and both remain unfixed upstream. All 8 apply with zero fuzz despite three patch-target files changing; patch 0007's route-table invariant survives because the new KV-pool-sizing block sits before the extracted table. Also corrects a stale README claim that setMmprojDevice and setMmprojOffload are mutually exclusive with each always clearing the other. Round 3 narrowed that; the paragraph is replaced by the four-row truth table re-derived from the two upstream handlers, which shows only two of the four combinations are order-dependent. Verified locally at the b10679 pin: fresh build directory through the real FetchContent path so the fail-loud PATCH_COMMAND ran, Release build clean, ctest 520/520, mvn clean test 1474 run / 0 failures with NativeLibraryLoadSmokeTest 3/3 and 0 skipped (the pin cross-check against the linked build-info), reactor mvn verify BUILD SUCCESS including SpotBugs and the javadoc jar, clang-format 22.1.8 clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CHANGELOG.md | 33 ++++++++++ CLAUDE.md | 10 +-- README.md | 22 +++++-- TODO.md | 3 +- docs/history/llama-cpp-breaking-changes.md | 2 + llama/CMakeLists.txt | 2 +- .../0001-win32-arg-parse-embed-guard.patch | 13 ---- llama/spotbugs-exclude.xml | 7 ++- .../llama/args/TensorReadLazyMode.java | 61 +++++++++++++++++++ .../llama/parameters/ModelParameters.java | 47 ++++++++++++++ .../llama/value/LlamaCppVersion.java | 8 +-- .../llama/args/TensorReadLazyModeTest.java | 19 ++++++ .../llama/parameters/ModelParametersTest.java | 41 +++++++++++++ 13 files changed, 236 insertions(+), 32 deletions(-) create mode 100644 llama/src/main/java/net/ladenthin/llama/args/TensorReadLazyMode.java create mode 100644 llama/src/test/java/net/ladenthin/llama/args/TensorReadLazyModeTest.java diff --git a/CHANGELOG.md b/CHANGELOG.md index d625bae19..2614b3e5b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,19 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by to `process_mtmd_prompt`, and video decoding is compiled into the shipped library (`MTMD_VIDEO` defaults on). `setVideoFfmpegDir` is the significant one — upstream otherwise looks `ffmpeg` and `ffprobe` up on `PATH`, which a JVM process often does not have them on. +- **`ModelParameters.setKvUnifiedPerSlot(int)`** — caps the context each parallel slot may use + (upstream `--kv-unified-per-slot`, new in llama.cpp b10679). The cap reaches this binding's + completion path through `server_context_meta::slot_n_ctx`, so it also moves where the + `repeat_last_n` / `dry_penalty_last_n` sentinels expand to. Upstream's second effect — sizing the + shared KV pool to `n_parallel * N` when no context size is given — lives in `llama_server()` and + therefore applies to `NativeServer` only, not to a model loaded from `ModelParameters`; the + Javadoc says so. +- **`ModelParameters.setTensorReadLazy(TensorReadLazyMode)`** and the new + **`net.ladenthin.llama.args.TensorReadLazyMode`** enum (`OFF` / `AUTO` / `ON`) — on-demand reading + of tensors the model architecture marks as lazy-loadable, such as per-layer embeddings (upstream + `--tensor-read-lazy`, new in llama.cpp b10679, mapping to `llama_lazy_mode`). Trades resident + memory for disk reads and requires mmap. It reaches the plain `LlamaModel` load path too, because + `common_model_params_to_llama` copies `lazy_mode` into `llama_model_params`. - **`ServerMetrics.getWindowPromptProcessingMillis()` / `getWindowTokenGenerationMillis()` / `getWindowTimings()`** — typed access to the current-window timing keys `t_prompt_processing` and `t_tokens_generation`. Both were always emitted; only the cumulative `_total` variants had accessors. @@ -119,6 +132,26 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by the merge divides upstream's microseconds. `ServerMetrics` reads them as doubles; a consumer parsing the raw JSON with an integer parser sees a type change. +- Upgraded llama.cpp from **b10649 to b10679**. No project-source change: every `tools/server/*.h` + header, `server-schema.cpp`, `server-task.cpp`, `server-common.cpp`, `common/chat.h` and + `tools/mtmd/mtmd-helper.h` are byte-identical across the range (compared by blob SHA), so the + request-field set, its bounds and the emitted response keys cannot have moved and the three + mechanical contract checks are moot. The whole in-scope delta is 172 changed lines over 8 files — + the rest of the 159-file range is `tools/ui` (rebuilt from `GIT_TAG` by CI) and backend internals. + Two additive upstream features are new and both are now exposed (see Added): + `--kv-unified-per-slot` and `--tensor-read-lazy` / `llama_lazy_mode`. Three patch-target files were + touched (`common/arg.cpp`, `tools/server/server-context.cpp`, `tools/server/server.cpp`) and all + eight patches still apply with zero fuzz; patch `0007`'s invariant holds because the new + KV-pool-sizing block in `llama_server()` sits before the extracted route table, not inside it. + `llama_model_quantize_params` gained `max_buf_size`, which needs no adaptation because + `LlamaQuantizer` builds its params from `llama_model_quantize_default_params()`. Upstream's private + `get_slot_n_ctx()` → `n_ctx_slot()` rename is invisible here — the project reads the value through + the unchanged `server_context_meta::slot_n_ctx`. + Patch `0001` shrank from 37 to 36 files: upstream rewrote `tests/test-save-load-state.cpp`'s + `main()` to build its own filtered argv, so by the patch's own rule that call site now wants + `common_params_parse` and no longer the `_main()` flip. The patch itself is still required — + `common_params_parse` at b10679 still carries the count-guarded `GetCommandLineW` override and + `common_params_parse_main` does not exist upstream. - Upgraded llama.cpp from **b10644 to b10649**. The first range in this bump to break the project's own compile: upstream threaded a new `mtmd_helper_init_opt` (video-decode settings) through every helper that can ingest media, changing the signature of `mtmd_helper_bitmap_init_from_file`, diff --git a/CLAUDE.md b/CLAUDE.md index 932fc25ca..f4eecec4e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10649** +Current llama.cpp pinned version: **b10679** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10649 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10679 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10649`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10679`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -667,7 +667,7 @@ Current patches: | Patch | Fixes | |-------|-------| -| `0001-win32-arg-parse-embed-guard.patch` | Windows JNI regression from llama.cpp **#24779** (introduced b9739): on Windows `common_params_parse` re-derived argv from the **process** command line (`GetCommandLineW`) and adopted it, so an embedded/JNI caller (`java.exe`) lost its `--model …` args → "Failed to parse model parameters". b9789 narrowed the unconditional override to a **count-guard** (`if (static_cast(utf8.buf.size()) == argc) { argv = utf8.ptrs.data(); }`), but that is exactly the variant the project already found breaks its Windows server-integration tests (when the embedded argv length coincides with `java.exe`'s). The patch carries the **complete upstream change** (so it can be submitted to llama.cpp verbatim and then dropped here): **(1)** `common_params_parse` parses **exactly the argv it is given** (no `GetCommandLineW` magic) and a new `common_params_parse_main()` wrapper holds the UTF-8 recovery for the standalone tools' `main()` (`common/arg.{cpp,h}`); **(2)** the **~34 standalone `main()` call sites** (every `common_params_parse(argc, argv, …)` across `tools/*`, `examples/*` and the `tests/*` programs) flip to `common_params_parse_main()`; **(3)** a `tests/test-arg-parser.cpp` regression case pins that `common_params_parse` honors a caller-supplied argv. The embedded caller (`jllama.cpp`) keeps calling `common_params_parse` and is never overridden. **Our subproject build compiles only the `arg.{cpp,h}` core** — `LLAMA_BUILD_TOOLS`/`LLAMA_BUILD_TESTS` are OFF for a FetchContent subproject — so the flips + test are applied-but-not-compiled here; they were validated via a one-off `-DLLAMA_BUILD_TOOLS=ON -DLLAMA_BUILD_TESTS=ON` build (the new test compiles and its asserts pass; `test-arg-parser`'s only red there is the live `ggml.ai` download check, which is sandbox-network, not the patch). Because it spans **37 files** it must be refreshed on every llama.cpp bump (the applier fails loud). The upstream-facing write-up, including a standalone reproducer that makes llama.cpp's own `test-arg-parser` fail on unmodified `master`, lives in [docs/upstream-investigation-win32-argv-substitution.md](docs/upstream-investigation-win32-argv-substitution.md). **Reported upstream as [ggml-org/llama.cpp#26416](https://github.com/ggml-org/llama.cpp/issues/26416)** (2026-08-01, label `bug-unconfirmed`, first bad commit `508a475`); the issue asks which of the two directions the maintainers prefer before a PR is opened, so this patch stays downstream until they answer. | +| `0001-win32-arg-parse-embed-guard.patch` | Windows JNI regression from llama.cpp **#24779** (introduced b9739): on Windows `common_params_parse` re-derived argv from the **process** command line (`GetCommandLineW`) and adopted it, so an embedded/JNI caller (`java.exe`) lost its `--model …` args → "Failed to parse model parameters". b9789 narrowed the unconditional override to a **count-guard** (`if (static_cast(utf8.buf.size()) == argc) { argv = utf8.ptrs.data(); }`), but that is exactly the variant the project already found breaks its Windows server-integration tests (when the embedded argv length coincides with `java.exe`'s). The patch carries the **complete upstream change** (so it can be submitted to llama.cpp verbatim and then dropped here): **(1)** `common_params_parse` parses **exactly the argv it is given** (no `GetCommandLineW` magic) and a new `common_params_parse_main()` wrapper holds the UTF-8 recovery for the standalone tools' `main()` (`common/arg.{cpp,h}`); **(2)** the **~34 standalone `main()` call sites** (every `common_params_parse(argc, argv, …)` across `tools/*`, `examples/*` and the `tests/*` programs) flip to `common_params_parse_main()`; **(3)** a `tests/test-arg-parser.cpp` regression case pins that `common_params_parse` honors a caller-supplied argv. The embedded caller (`jllama.cpp`) keeps calling `common_params_parse` and is never overridden. **Our subproject build compiles only the `arg.{cpp,h}` core** — `LLAMA_BUILD_TOOLS`/`LLAMA_BUILD_TESTS` are OFF for a FetchContent subproject — so the flips + test are applied-but-not-compiled here; they were validated via a one-off `-DLLAMA_BUILD_TOOLS=ON -DLLAMA_BUILD_TESTS=ON` build (the new test compiles and its asserts pass; `test-arg-parser`'s only red there is the live `ggml.ai` download check, which is sandbox-network, not the patch). Because it spans **36 files** it must be refreshed on every llama.cpp bump (the applier fails loud). **Refreshed at the b10679 bump:** upstream rewrote `tests/test-save-load-state.cpp`'s `main()` to take a `--models DIR` option, which it strips itself into a `filtered_argv` before calling `common_params_parse(fargc, filtered_argv.data(), …)`. That call site therefore stopped qualifying for the `_main()` flip — by this patch's own rule a caller that builds its own argv must use `common_params_parse` directly, so its argv is kept — and the hunk was **dropped** rather than refreshed (37 → 36 files). Caveat for whoever submits this upstream: that `main()` now filters a possibly-mojibake Windows argv *before* any UTF-8 recovery, so the fully correct upstream form there is recover-then-filter, not a one-line flip. It is out of scope for the downstream carry because `LLAMA_BUILD_TESTS` is OFF here, so the file is never compiled. **Still required at b10679, verified rather than assumed:** `common_params_parse` in pristine `b10679:common/arg.cpp` still carries the `#ifdef _WIN32` count-guarded `argv = utf8.ptrs.data()` override, and `common_params_parse_main` appears nowhere in `b10679:common/arg.h` — upstream has not adopted the fix. The upstream-facing write-up, including a standalone reproducer that makes llama.cpp's own `test-arg-parser` fail on unmodified `master`, lives in [docs/upstream-investigation-win32-argv-substitution.md](docs/upstream-investigation-win32-argv-substitution.md). **Reported upstream as [ggml-org/llama.cpp#26416](https://github.com/ggml-org/llama.cpp/issues/26416)** (2026-08-01, label `bug-unconfirmed`, first bad commit `508a475`); the issue asks which of the two directions the maintainers prefer before a PR is opened, so this patch stays downstream until they answer. | | `0002-server-preserve-caller-load-progress-callback.patch` | Load-progress-callback regression introduced in llama.cpp **b9789**: `server_context::load_model` (`tools/server/server-context.cpp`) now **unconditionally** installs the server's own load-progress reporter on `params_base.load_progress_callback` immediately before `common_init_from_params`, clobbering any callback the embedding caller already set. libjllama's `LoadProgressCallback` feature wires `common_params.load_progress_callback` to a JNI trampoline *before* calling `load_model`, so the bump silently killed it — `LoadProgressCallbackTest` saw zero progress updates and the abort-on-`false` path never threw. The patch guards the assignment with `if (params_base.load_progress_callback == nullptr)`, so the server installs its own reporter **only when the caller hasn't** — a caller-supplied callback survives and fires during load. Standalone `llama-server` (no caller callback, so the field is null) is unaffected. Same JNI-vs-standalone divergence class as `0001`. | | `0003-pr22393-server-add-slot-prompt-similarity-getter-setter.patch` | **Upstream-PR carry** of [ggml-org/llama.cpp#22393](https://github.com/ggml-org/llama.cpp/pull/22393) ("server : add slot_prompt_similarity getter/setter"). Purely additive: adds `server_context::get_slot_prompt_similarity()` / `set_slot_prompt_similarity(float)` (`tools/server/server-context.{cpp,h}`) so an embedding/JNI caller can query and tune the slot-selection threshold at runtime without reloading the model. Verbatim copy of the PR, which **upstream closed without merging** (rejected as exposing unsafe internal state — see the patch header). Carried permanently; it will not be droppable via a version bump. | | `0007-server-attach-http-frontend.patch` | **Adds `llama_server_attach(argc, argv, server_context&)`** so the `NativeServer` *attach mode* can serve an **already-loaded `LlamaModel`** over the upstream HTTP frontend — no second model load, no `start_loop()`; the LlamaModel's worker keeps driving the shared `server_context` and the HTTP routes post tasks to its queue (the queue is the synchronization point). Mechanically: (1) extracts the **pure core route table** (`health` … `slots`) out of `llama_server()` into `static void llama_server_register_common_routes(ctx_http, routes)` (shared, so the two entry points cannot drift on the core endpoint set). **Scope note (narrowed at the b10154 bump):** the helper deliberately carries **only** the stable, state-independent route table — **not** the resumable-streaming routes (their handlers differ between router / non-router), the GCP-compat shim, or the experimental **CORS-proxy / MCP-server / built-in-tools** wiring. b10154 (upstream MCP-server support) moved the streaming routes into the middle of that block and coupled tools/CORS to a per-call `server_mcp mcp_mgr` lifecycle, so the earlier contiguous "route-table + CORS-proxy + tools" extraction is no longer possible; `llama_server()` keeps all of that inline, **byte-identical to upstream b10154** (only the route-table block is factored out). (2) adds `llama_server_attach`, which parses only the HTTP-side argv via `common_params_parse`, starts the stream-session GC + `server_http_context`, registers the common route table, the **non-router** resumable-streaming handlers (upstream b10154 paths `/v1/stream` GET/DEL + `/v1/streams/lookup` POST), the GCP-compat shim, and **403 "disabled" stubs for `/cors-proxy` + `/tools`** (attach mode does not wire the experimental CORS-proxy / MCP / built-in-tools host — those belong to a full `llama-server`, not an embedded model), marks ready immediately (model already loaded), and blocks on the HTTP thread until `llama_server_request_shutdown()` — never calling `common_init()`, backend init, `ctx_server.terminate()` or `llama_backend_free()` (the embedding caller owns those). Applies after `0001`+`0006` (same file); closes the "NativeServer — reuse an already-loaded LlamaModel" TODO. Upstream-submittable ("server: let embedding callers attach the HTTP frontend to an existing server_context"). **Refreshed at the b10519 bump:** upstream #26347 dropped the API key from the `/models` + `/v1/models` public-endpoint set and deleted the two trailing `// public endpoint (no API key check)` comments on those route registrations. Those two lines sit inside this patch's route-table removal block, so `git apply` failed ("patch does not apply", `server.cpp:258`) at **every** tag from b10519 on; the fix was to drop the now-wrong comment from all four affected lines (2 on the `-` side, 2 in the extracted helper on the `+` side), keeping the helper byte-identical to the block it replaces. **This is the invariant to re-check on every bump:** the `+` side of `llama_server_register_common_routes()` must stay a verbatim copy of the route table it factors out of `llama_server()`. | @@ -1447,7 +1447,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10649`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10679`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index d285d3f06..02f5d749f 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10649](https://img.shields.io/badge/llama.cpp-%23b10649-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10649) +[![llama.cpp b10679](https://img.shields.io/badge/llama.cpp-%23b10679-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10679) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) @@ -495,10 +495,22 @@ device may be named; the literal `"none"` keeps the projector on the CPU. `OpenA accepts the same flag as `-mmdev`/`--mmproj-device`, and `NativeServer` forwards it verbatim like every other llama-server flag. -`setMmprojDevice(...)` and `setMmprojOffload(...)` are mutually exclusive: upstream writes both onto -the single `mmproj_use_gpu` field, so only one may appear in the rendered argv and **the last of the -two calls wins** — each clears the other. Pick one; if you need a device after disabling offload, -call `setMmprojDevice` last. +`setMmprojDevice(...)` and `setMmprojOffload(...)` write the **same** upstream field +(`common_params::mmproj_use_gpu`), so where they disagree the outcome would depend on argv order — +and the rendered argv comes from a `HashMap`, whose order is unspecified. The builder therefore +resolves the two genuinely ambiguous combinations by dropping the earlier call, and leaves the rest +alone: + +| Combination | Resolves to | Builder behaviour | +|---|---|---| +| named device + `setMmprojOffload(true)` | `(use_gpu=true, device)` in either order | both kept — no clash | +| named device + `setMmprojOffload(false)` | order-dependent | **last call wins** | +| `"none"` + `setMmprojOffload(true)` | order-dependent | **last call wins** | +| `"none"` + `setMmprojOffload(false)` | `(use_gpu=false)` in either order | both kept — no clash | + +So a multi-GPU projector pin survives an explicit `setMmprojOffload(true)`; only a call that would +actually contradict the other is dropped. If you need a device after disabling offload, call +`setMmprojDevice` last. **Video input — decode settings only, so far.** llama.cpp b10649 added a video path to `mtmd`, and it is compiled into the shipped library (`MTMD_VIDEO` is on by default). Its decode settings are exposed diff --git a/TODO.md b/TODO.md index be61ebf96..272640f79 100644 --- a/TODO.md +++ b/TODO.md @@ -141,7 +141,8 @@ same crashed test. (`ca60947`), which makes the invalid branch honour leniency exactly as the incomplete branch does; strict mode is unchanged. Guarded by the `ContentOnlyParseUtf8` tests in `test_utils.cpp`, which — unlike the upstream test the patch also adds — run in CI on every platform. Upstream-submittable; - re-checked at b10649 and upstream has not fixed it, so the patch stays. + re-checked at b10679 — the `until` parser's `INVALID` branch in pristine + `b10679:common/peg-parser.cpp` still returns `FAIL` with no `is_lenient()` guard — so the patch stays. - **[ANSWERED] Re-check the full suite once the TTS crash is fixed.** Done: Ubuntu on `999034b` ran **1689 tests, 3 failures, 1 error, 2 skipped**. Exactly one item was new — the router entry diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index bf5616cbb..1692e20ad 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -681,3 +681,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10639–b10644 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10644 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10644: all **8** patches applied, confirmed by the stamp file (`head d7a2074112d27649303fa107eb8c94db1ee435f3` plus one SHA-256 line per patch) — including `0011`, whose target `common/peg-parser.cpp` the range does not touch. Full `cmake --build --config Release` clean, `ctest` **509/509**. `nm -D` on the freshly linked `libjllama.so` reports **40** `Java_*` exports and **zero** C++-mangled ones. `NativeLibraryLoadSmokeTest` **3/3, 0 skipped**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10644") matches the `build-info` compiled into the new binary (`b10644-d7a207411`), which is what proves the four pin sites and the actual build agree. Run with `mvn clean test`, not a bare `mvn test`: `LLAMA_CPP_VERSION` is a compile-time constant that javac inlines into the already-compiled test class, and Maven's incremental compilation cannot see that dependency — the b10636→b10639 row below records the false drift that costs. | | b10644–b10649 | **`tools/mtmd/mtmd-helper.{h,cpp}` (BREAKING: `mtmd_helper_bitmap_init_from_file` and `_from_buf` gain a 4th `struct mtmd_helper_init_opt` parameter; new `mtmd_helper_video_init_params` + `mtmd_helper_init_opt` structs and their `_default()` factories)**, **`tools/server/server-common.{h,cpp}` (BREAKING: `tokenize_input_prompts`, `process_mtmd_prompt` and `format_prompt_rerank` all gain a trailing `const mtmd_helper_init_opt &`)**, `tools/server/server-context.{h,cpp}` (`handle_count_tokens` gains the same parameter; threading it through), `common/common.h` (**additive**: `common_params_speculative::{synth_len,synth_rates}` + `has_synth()`, `common_params::{video_fps,video_timestamp_interval_ms,video_ffmpeg_bin_dir}`), `common/arg.cpp` (**purely additive**: 6 new flags), `common/speculative.{cpp,h}` (**additive**: `common_speculative_n_max`, `_synth_rates_resolve`, `_get_synth_probs`), `ggml/src/ggml-metal/*`, `src/models/minimax-01.cpp`, `tests/test-arg-parser.cpp`, `tools/{cli,llama-bench,tts}`, `tools/server/tests/*` | **The first range in this whole bump that broke the project's own compile — four call sites, all from one upstream refactor.** llama.cpp b10649 threaded a new `mtmd_helper_init_opt` (video decode settings: fps target, ffmpeg binary dir, timestamp interval) through every helper that can ingest media. `tts_engine.cpp:95` (`mtmd_helper_bitmap_init_from_file`, the speaker-reference clip) and `jllama.cpp` ×3 (`tokenize_input_prompts`) + ×1 (`format_prompt_rerank`) all lost their signatures. Every one of those paths passes `mctx = nullptr` or handles audio, so none wants video settings: each now passes `mtmd_helper_init_opt_default()`, which is upstream's own default. **This is the failure class the priority table's note calls out** — `server-common.h` is a same-repo header the project `#include`s directly rather than one reachable through the documented dependency graph, so only a real compile finds it. **The wire contract did not move:** all three mechanical greps are identical across the range (68 request fields, 23 bounds, 286 response keys over all six server `.cpp`). `common/arg.cpp` removed or renamed **zero** flags. Sizing: 106 KiB reviewable, or **72 KiB excluding `ggml-metal`** (Metal backend internals behind unchanged public headers — same exclusion rationale as Hexagon), under the runbook's 100 KiB single-step threshold. **All 8 patches apply unchanged** despite six patch-target files changing (`common/arg.cpp`, `tests/test-arg-parser.cpp`, `tools/mtmd/mtmd-cli.cpp`, `tools/server/server-context.{cpp,h}`, `tools/tts/tts.cpp`) — upstream's edits fall outside every patched region. | | b10644–b10649 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10649 -DBUILD_TESTING=ON` through the real `FetchContent` path: all **8** patches applied (stamp `head 2bb9bddafad44ecbb50889644ca47537ec11841b`), which was the first thing checked because six patch-target files changed in the range. The first build failed loud with the four signature breaks above; after adapting the call sites, Release build clean, `ctest` **516/516** (512 at the bump; the second audit added four `TrainParams` cases), `nm -D` **40** `Java_*` exports and **zero** C++-mangled, `NativeLibraryLoadSmokeTest` **3/3, 0 skipped** including the pin cross-check against the linked `build-info`. `ModelParametersTest` 77/77 with the five CPU-offload cases plus the three video-flag cases. **Feature exposure decided deliberately, not by default:** of the 6 new flags, only `--n-cpu-ffn` is genuinely new upstream, and it was added as `ModelParameters.setCpuFfnLayers`; `setCpuMoeLayers` was added alongside it for `--n-cpu-moe`, which has existed upstream since **b6089** (#15077, `ec428b02c`) but had never been exposed here — b10649 only refactored its lambda onto the shared `llm_add_n_cpu_ffn_overrides` helper; the two `--spec-synth-*` flags were refused as upstream marks them "benchmarking only"; and the three `--video-*` flags were initially refused as inert without a `ContentPart` video factory. **That refusal was wrong and was reversed by the follow-up audit:** they are not inert — `server_context::load_model` copies them into its own `init_opt` when the projector loads, and that `init_opt` is what `server-context.cpp` hands to `process_mtmd_prompt` on the very task path this binding drives (`task.cli_files`), so they take effect for any attached media; and video decoding really is compiled in (`MTMD_VIDEO` defaults `ON`, gated only on `LLAMA_SUBPROCESS` which is also `ON`, and the shipped `libjllama.so` carries the ffmpeg invocation strings). They are now `ModelParameters.setVideoFps` / `setVideoTimestampInterval` / `setVideoFfmpegDir`; `setVideoFfmpegDir` matters most, since upstream otherwise resolves `ffmpeg`/`ffprobe` from `PATH`, which a JVM process frequently lacks. Only the ergonomic `ContentPart.videoFile(...)` entry point is still outstanding (raw bytes already work — the decoder sniffs the container rather than trusting the MIME type). The `--spec-synth-*` refusal stands. Both are recorded in `TODO.md` with their reasoning. | +| b10649–b10679 | `common/arg.cpp` (**purely additive: 2 new flags, 0 removed or renamed**), `common/common.h` (**additive**: `common_params::lazy_mode`, `common_params::kv_unified_per_slot`), `common/common.cpp` (one line: `mparams.lazy_mode = params.lazy_mode`), `include/llama.h` (**additive**: new `llama_lazy_mode` enum, `llama_model_params::lazy_mode`, `llama_model_quantize_params::max_buf_size`), `common/speculative.cpp` (implementation only — `speculative.h` byte-identical), `tools/server/server-context.cpp` (**#27860: per-slot context cap**; private `get_slot_n_ctx()` renamed to `n_ctx_slot()` and made a recomputing accessor), `tools/server/server.cpp` (**KV-pool auto-sizing block**), `tools/server/README.md`, plus `tools/ui/**` and backend internals (auto-followed / not compiled here) | **No project-source change; two features deliberately exposed.** The in-scope delta is **172 changed lines over 8 files** out of a 159-file, 8045-insertion range — the remainder is `tools/ui` (rebuilt from `GIT_TAG` by CI, no source review) and backends the project does not build. **The three mechanical contract checks are moot, not merely passed**: `server-schema.cpp`, `server-task.cpp`, `server-common.cpp` and *all six* `tools/server/*.h` headers are **byte-identical** across the range (verified by comparing blob SHAs), so no request field, bound or response key can have moved; `common/chat.h` and `tools/mtmd/mtmd-helper.h` are byte-identical too, so the `getMetrics()`-class silent-contract break and the `tts_engine.cpp` surface are both out of scope. `common/common.h` is the only priority-table file that changed, and only by the two additive fields above. **Two additive upstream features, both now exposed** (deliberate decision, not default): `--kv-unified-per-slot` → `ModelParameters.setKvUnifiedPerSlot(int)`, and `--tensor-read-lazy` → `ModelParameters.setTensorReadLazy(TensorReadLazyMode)` over the new `net.ladenthin.llama.args.TensorReadLazyMode` enum, which mirrors `llama_lazy_mode` (`off`/`auto`/`on` = 0/1/2). **Both really reach this binding**, which is why they were exposed rather than refused: `--tensor-read-lazy` has no `set_examples()` restriction and `common_model_params_to_llama` copies `lazy_mode` into `llama_model_params`, so it applies to a plain `LlamaModel` load; `--kv-unified-per-slot` is `set_examples({LLAMA_EXAMPLE_SERVER})` and `jllama.cpp` parses with `LLAMA_EXAMPLE_SERVER`, so it registers, and its **cap** half is applied inside `server-context.cpp`'s `n_ctx_slot()`, whose value the project reads through `server_context_meta::slot_n_ctx` (`jllama.cpp:1547`, fed to `eval_llama_cmpl_schema`) — so it also moves where the `repeat_last_n` / `dry_penalty_last_n` sentinels expand to. Its **pool-sizing** half lives in `llama_server()` and therefore applies only to `NativeServer` full mode; the Javadoc says so explicitly rather than implying the flag is self-sufficient. `llama_model_quantize_params::max_buf_size` needs **no** adaptation: `LlamaQuantizer` builds its params from `llama_model_quantize_default_params()`, so the new field is initialised by upstream. The `get_slot_n_ctx()` → `n_ctx_slot()` rename is a **private** member of the impl and is not called by the project. **Three patch-target files were touched** (`common/arg.cpp`, `server-context.cpp`, `server.cpp`) and **all 8 patches still apply with zero fuzz**; patch `0007`'s standing invariant survives because the new KV-pool-sizing block in `llama_server()` sits *before* the extracted route table, not inside it. **Patch `0001` shrank 37 → 36 files**: upstream rewrote `tests/test-save-load-state.cpp`'s `main()` to strip `--models DIR` into its own `filtered_argv` and call `common_params_parse(fargc, filtered_argv.data(), ...)`. By that patch's own rule — *embedded callers that build their own argv must call `common_params_parse` directly* — that call site no longer wants the `_main()` flip, so the hunk was **dropped, not refreshed**. The patch itself is **still required**: `common_params_parse` at b10679 still carries the count-guarded `GetCommandLineW` override, and `common_params_parse_main` does not exist in `b10679:common/arg.h` — upstream has not adopted either direction proposed in [ggml-org/llama.cpp#26416](https://github.com/ggml-org/llama.cpp/issues/26416). | +| b10649–b10679 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10679 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10679 — a fresh build directory was mandatory, since the applier pins its stamp to the checked-out llama.cpp commit and aborts rather than guess when an existing stamp names a different one. All **8** patches applied (`0001` in its 36-file form). Release build clean, `ctest` **520/520**. Java: `mvn clean test` **1456 run / 0 failures**, with `NativeLibraryLoadSmokeTest` **3/3 and 0 skipped** — including `nativeBuildInfoMatchesPinnedVersionConstant`, the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10679") matches the `build-info` compiled into the freshly linked `libjllama.so`, which is what proves the four pin sites and the actual build agree. `ModelParametersTest` covers both new setters (exact flag spellings, the `<= 0` rejection on `--kv-unified-per-slot`) and `TensorReadLazyModeTest` pins all three enum wire strings under the PIT-gated `net.ladenthin.llama.args.*` package. `mvn spotless:apply` produced no changes beyond the edits themselves. **One documentation defect was found and fixed during this verification** and is worth recording as a class: the first draft of `setKvUnifiedPerSlot`'s Javadoc stated the pool-sizing effect unconditionally. Reading `tools/server/server.cpp` showed that half executes only in `llama_server()`, which a `ModelParameters`-loaded model never enters — an accurate-for-upstream sentence that would have been wrong for the API it documents. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index e3801cb26..3c34c60c6 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10649 + GIT_TAG b10679 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/patches/0001-win32-arg-parse-embed-guard.patch b/llama/patches/0001-win32-arg-parse-embed-guard.patch index 54a779f41..c014ff7c8 100644 --- a/llama/patches/0001-win32-arg-parse-embed-guard.patch +++ b/llama/patches/0001-win32-arg-parse-embed-guard.patch @@ -313,19 +313,6 @@ index 8e2eace6a..d7a7afed9 100644 return 1; } -diff --git a/tests/test-save-load-state.cpp b/tests/test-save-load-state.cpp -index 6e93ce6fb..9caab1653 100644 ---- a/tests/test-save-load-state.cpp -+++ b/tests/test-save-load-state.cpp -@@ -358,7 +358,7 @@ int main(int argc, char ** argv) { - - common_init(); - -- if (!common_params_parse(argc, argv, params, LLAMA_EXAMPLE_COMMON)) { -+ if (!common_params_parse_main(argc, argv, params, LLAMA_EXAMPLE_COMMON)) { - return 1; - } - diff --git a/tests/test-state-restore-fragmented.cpp b/tests/test-state-restore-fragmented.cpp index d5548afba..95ba2b67b 100644 --- a/tests/test-state-restore-fragmented.cpp diff --git a/llama/spotbugs-exclude.xml b/llama/spotbugs-exclude.xml index 02e3dcbe6..95d2e2756 100644 --- a/llama/spotbugs-exclude.xml +++ b/llama/spotbugs-exclude.xml @@ -60,9 +60,9 @@ SPDX-License-Identifier: MIT

    • A missing input file is the cheapest call that still crosses JNI: reaching the native side + * at all produces {@link LlamaException}, whereas a linkage regression fails earlier and + * differently with {@link UnsatisfiedLinkError} (which is an {@link Error}, so it propagates out + * of an {@code assertThrows(LlamaException.class, ...)} rather than being reported as a + * wrong-exception mismatch). + */ + @Test + void quantizerNativeEntryPointResolves() { + assumeTrue(nativeLibraryOnClasspath(), "libjllama not on classpath — skipping quantizer linkage check"); + assertThrows( + LlamaException.class, + () -> LlamaQuantizer.quantize( + "does-not-exist-quantizer-linkage-probe.gguf", + "never-written.gguf", + QuantizationType.Q8_0, + 0, + true), + "LlamaQuantizer.quantize must reach the native implementation and fail with " + + "LlamaException; an UnsatisfiedLinkError here means quantizeNative lost its " + + "extern \"C\" linkage and is exported under a C++-mangled name"); + } } diff --git a/llama/src/test/java/net/ladenthin/llama/parameters/JsonEndpointParametersTest.java b/llama/src/test/java/net/ladenthin/llama/parameters/JsonEndpointParametersTest.java index c311011fc..0ae296d4d 100644 --- a/llama/src/test/java/net/ladenthin/llama/parameters/JsonEndpointParametersTest.java +++ b/llama/src/test/java/net/ladenthin/llama/parameters/JsonEndpointParametersTest.java @@ -83,7 +83,11 @@ public void testDryMultiplierAccepted() { String json = "{\"prompt\":\"" + PROMPT + "\",\"n_predict\":" + N_PREDICT + DETERMINISTIC + ",\"dry_multiplier\":0.8,\"dry_base\":1.75,\"dry_allowed_length\":2" - + ",\"dry_penalty_last_n\":-1}"; + // 256 == the ctx size configured above. This used to say -1, llama.cpp's old + // "scan the whole context" sentinel; b10275 gave dry_penalty_last_n the hard + // limits [0, INT32_MAX] (0 = disabled), so -1 is now rejected outright with a + // 400. Spelling the context size out keeps the original intent. + + ",\"dry_penalty_last_n\":256}"; String result = model.handleCompletions(json); assertThat(result, is(notNullValue())); assertThat("Response should contain 'content' field", result, containsString("\"content\"")); From 7b68897e92438c3e449178d4f000c3bf0fd3e496 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 22:36:02 +0000 Subject: [PATCH 43/76] docs: add the missing SPDX header to the win32-argv doc The REUSE check is the one red check on this branch that is neither the TTS crash nor tonight's DNS flakes: 438 of 439 files carry copyright and licensing information, and the one that does not is docs/upstream-investigation-win32-argv-substitution.md. It predates this PR and is red on main too, so it is not this PR's failure -- but it is a one-line fix for a check this PR otherwise cannot get green, so it is folded in here rather than left for later. The repo uses two mechanisms: a central file list in REUSE.toml (mostly older and imported files) and inline SPDX headers. This doc gets the inline header, matching docs/history/llama-cpp-breaking-changes.md and docs/upgrade/llama-cpp-version-bump.md -- the two docs closest to it in purpose, both part of the same version-bump documentation set -- and keeping the licensing attached to a file that is meant to be readable upstream. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- docs/upstream-investigation-win32-argv-substitution.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/upstream-investigation-win32-argv-substitution.md b/docs/upstream-investigation-win32-argv-substitution.md index 6db2f9a32..3bc07c99d 100644 --- a/docs/upstream-investigation-win32-argv-substitution.md +++ b/docs/upstream-investigation-win32-argv-substitution.md @@ -1,3 +1,9 @@ + + # `common_params_parse` can silently discard the caller's argv on Windows Technical findings for llama.cpp. Everything below was verified against From e443758ef693a131516f8eb712c3a1d2f0958d70 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 22:40:13 +0000 Subject: [PATCH 44/76] docs: record the macOS confirmations for the quantizer fix The macOS 14 job on 42ce225 got 517 tests deep before TtsIntegrationTest killed the fork -- far enough to settle two things the Windows-only evidence could not. The quantizer linkage fix is confirmed on a second toolchain: QuantizerIntegrationTest ran 3 tests, 0 failures, 0 errors, with no UnsatisfiedLinkError anywhere in the run. Those are the same 3 tests that were 2 failures + 1 error before the fix. This is worth having on record because symbol export and name mangling differ between ELF/gcc and Mach-O/clang, so the local nm -D check on Linux did not by itself establish it. SessionForkRewindIntegrationTest (both cases) and NativeServerAttachIntegrationTest reproduce identically on macOS -- same assertions, same messages. They are not a Windows quirk, which the previous note left open by describing the evidence as coming only from the two Windows jobs. Also notes that macOS 14 exits 141 (SIGPIPE) where Ubuntu exits 134 (SIGABRT) on the same crashed test, in case the differing signal helps localise the abort. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- TODO.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/TODO.md b/TODO.md index 3038eacd5..add383a4f 100644 --- a/TODO.md +++ b/TODO.md @@ -26,6 +26,12 @@ tests), so those jobs report *nothing but* the crash. The two Windows jobs happe therefore reach **1461** tests — they are the only jobs whose failure list is complete. Do not read a short Linux failure list as "Linux is healthier". +**Confirmed platform-independent (macOS 14, head `42ce225`).** That job reached **517** tests before +the crash and reproduced `SessionForkRewindIntegrationTest` (both cases) and +`NativeServerAttachIntegrationTest` **identically** — same assertions, same messages. So these are +not a Windows quirk. Its exit code was **141 (SIGPIPE)** rather than Ubuntu's 134 (SIGABRT), on the +same crashed test. + - **[OPEN, MAJOR] `TtsIntegrationTest` aborts the JVM natively on all 6 test platforms.** Ubuntu exit 134 (SIGABRT); macOS 14 Metal, macOS 15 Metal, macOS 15 no-Metal; Windows Ninja and Windows MSVC exit 1. Not an OOM (Linux had ~14 GiB free, Windows ~12.3 GiB of 16 GiB). Not caused by @@ -68,7 +74,10 @@ short Linux failure list as "Linux is healthier". resolve it. Reproduced on Linux with `nm -D`, so it was never Windows-specific — the public `LlamaQuantizer` API has never worked. Fixed, and guarded model-free by `NativeLibraryLoadSmokeTest.quantizerNativeEntryPointResolves` (`nm -D` on the rebuilt lib now shows - zero mangled `Java_*` exports). + zero mangled `Java_*` exports). **Confirmed on a second toolchain:** at `42ce225` the macOS 14 job + ran `QuantizerIntegrationTest` at *3 tests, 0 failures, 0 errors* with no `UnsatisfiedLinkError` + anywhere in the run — the same 3 tests that were 2 failures + 1 error before the fix. Worth having, + since symbol export and mangling differ between ELF/gcc and Mach-O/clang. - **[FIXED in this PR] `JsonEndpointParametersTest.testDryMultiplierAccepted` sent `dry_penalty_last_n: -1`.** The one genuine b10456→b10618 regression in the list: b10275 gave the From cfda4a9ce4b862e039e776697977a213f80144f7 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 07:10:16 +0000 Subject: [PATCH 45/76] ci: echo the crash log into the job log at all six test jobs Mirrors workspace/policies/ci-test-diagnostics.md section 3.1 into this repo: a failure-gated step before each crash upload that prints hs_err_pid*.log (first 200 lines) and the surefire dumpstream/dump into the job log. This repo is why the policy changed. TtsIntegrationTest aborts the forked JVM on all six Java test platforms here, and the aborting frame was reachable only by downloading an 8 MB artifact from Azure Blob -- which fails outright wherever egress to that host is denied. The upload stays; it is no longer the only way in. All six sites use shell: bash, including the two Windows jobs, whose runners ship Git Bash. Paths are module-relative (llama/) because that is where the surefire fork's working directory puts them. The "nothing was written" branch matters here specifically: it is not yet established that TtsIntegrationTest produces an hs_err at all. if-no-files-found is warn, so the artifact looks identical either way, and Windows reported exit code 1 rather than a typical crash code -- a native exit()/terminate would leave no JVM crash log. The next red run will now say which of the two it is. Verified both branches of the shell logic locally: with no files present it prints the explanatory line; with a 300-line hs_err plus a dumpstream it emits 203 lines, not 303. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- .github/workflows/publish.yml | 162 ++++++++++++++++++++++++++++++++++ 1 file changed, 162 insertions(+) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 6cfa60ef8..6beee1d2c 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -2271,6 +2271,33 @@ jobs: - name: Memory after tests if: always() run: free -h + # A forked test JVM that aborts leaves an hs_err_pid log and a surefire + # dumpstream -- both otherwise ONLY inside the artifact uploaded below, + # which is unreachable from anywhere that cannot fetch from Azure Blob + # (a phone, a restricted network, an agent sandbox). Echo them here so the + # aborting frame is readable from the run page itself. See + # ../workspace/policies/ci-test-diagnostics.md section 3.1. + - name: Print crash logs (on failure) + if: failure() + shell: bash + run: | + shopt -s nullglob + found=0 + for f in llama/hs_err_pid*.log; do + found=1 + echo "===== $f (first 200 lines; full file in the uploaded artifact) =====" + sed -n '1,200p' "$f" + done + for f in llama/target/surefire-reports/*.dumpstream llama/target/surefire-reports/*.dump; do + found=1 + echo "===== $f =====" + cat "$f" + done + if [ "$found" = 0 ]; then + echo "No hs_err_pid*.log and no surefire dump/dumpstream was written." + echo "The fork died without the JVM writing a crash log -- the abort bypassed" + echo "the JVM error handler (native exit()/terminate) rather than raising a signal." + fi - if: failure() uses: actions/upload-artifact@v7 with: @@ -2370,6 +2397,33 @@ jobs: - name: Memory after tests if: always() run: vm_stat && sysctl hw.memsize hw.physmem + # A forked test JVM that aborts leaves an hs_err_pid log and a surefire + # dumpstream -- both otherwise ONLY inside the artifact uploaded below, + # which is unreachable from anywhere that cannot fetch from Azure Blob + # (a phone, a restricted network, an agent sandbox). Echo them here so the + # aborting frame is readable from the run page itself. See + # ../workspace/policies/ci-test-diagnostics.md section 3.1. + - name: Print crash logs (on failure) + if: failure() + shell: bash + run: | + shopt -s nullglob + found=0 + for f in llama/hs_err_pid*.log; do + found=1 + echo "===== $f (first 200 lines; full file in the uploaded artifact) =====" + sed -n '1,200p' "$f" + done + for f in llama/target/surefire-reports/*.dumpstream llama/target/surefire-reports/*.dump; do + found=1 + echo "===== $f =====" + cat "$f" + done + if [ "$found" = 0 ]; then + echo "No hs_err_pid*.log and no surefire dump/dumpstream was written." + echo "The fork died without the JVM writing a crash log -- the abort bypassed" + echo "the JVM error handler (native exit()/terminate) rather than raising a signal." + fi - if: failure() uses: actions/upload-artifact@v7 with: @@ -2439,6 +2493,33 @@ jobs: - name: Memory after tests if: always() run: vm_stat && sysctl hw.memsize hw.physmem + # A forked test JVM that aborts leaves an hs_err_pid log and a surefire + # dumpstream -- both otherwise ONLY inside the artifact uploaded below, + # which is unreachable from anywhere that cannot fetch from Azure Blob + # (a phone, a restricted network, an agent sandbox). Echo them here so the + # aborting frame is readable from the run page itself. See + # ../workspace/policies/ci-test-diagnostics.md section 3.1. + - name: Print crash logs (on failure) + if: failure() + shell: bash + run: | + shopt -s nullglob + found=0 + for f in llama/hs_err_pid*.log; do + found=1 + echo "===== $f (first 200 lines; full file in the uploaded artifact) =====" + sed -n '1,200p' "$f" + done + for f in llama/target/surefire-reports/*.dumpstream llama/target/surefire-reports/*.dump; do + found=1 + echo "===== $f =====" + cat "$f" + done + if [ "$found" = 0 ]; then + echo "No hs_err_pid*.log and no surefire dump/dumpstream was written." + echo "The fork died without the JVM writing a crash log -- the abort bypassed" + echo "the JVM error handler (native exit()/terminate) rather than raising a signal." + fi - if: failure() uses: actions/upload-artifact@v7 with: @@ -2508,6 +2589,33 @@ jobs: - name: Memory after tests if: always() run: vm_stat && sysctl hw.memsize hw.physmem + # A forked test JVM that aborts leaves an hs_err_pid log and a surefire + # dumpstream -- both otherwise ONLY inside the artifact uploaded below, + # which is unreachable from anywhere that cannot fetch from Azure Blob + # (a phone, a restricted network, an agent sandbox). Echo them here so the + # aborting frame is readable from the run page itself. See + # ../workspace/policies/ci-test-diagnostics.md section 3.1. + - name: Print crash logs (on failure) + if: failure() + shell: bash + run: | + shopt -s nullglob + found=0 + for f in llama/hs_err_pid*.log; do + found=1 + echo "===== $f (first 200 lines; full file in the uploaded artifact) =====" + sed -n '1,200p' "$f" + done + for f in llama/target/surefire-reports/*.dumpstream llama/target/surefire-reports/*.dump; do + found=1 + echo "===== $f =====" + cat "$f" + done + if [ "$found" = 0 ]; then + echo "No hs_err_pid*.log and no surefire dump/dumpstream was written." + echo "The fork died without the JVM writing a crash log -- the abort bypassed" + echo "the JVM error handler (native exit()/terminate) rather than raising a signal." + fi - if: failure() uses: actions/upload-artifact@v7 with: @@ -2596,6 +2704,33 @@ jobs: if: always() run: Get-CimInstance Win32_OperatingSystem | Select-Object FreePhysicalMemory,TotalVisibleMemorySize | Format-List shell: pwsh + # A forked test JVM that aborts leaves an hs_err_pid log and a surefire + # dumpstream -- both otherwise ONLY inside the artifact uploaded below, + # which is unreachable from anywhere that cannot fetch from Azure Blob + # (a phone, a restricted network, an agent sandbox). Echo them here so the + # aborting frame is readable from the run page itself. See + # ../workspace/policies/ci-test-diagnostics.md section 3.1. + - name: Print crash logs (on failure) + if: failure() + shell: bash + run: | + shopt -s nullglob + found=0 + for f in llama/hs_err_pid*.log; do + found=1 + echo "===== $f (first 200 lines; full file in the uploaded artifact) =====" + sed -n '1,200p' "$f" + done + for f in llama/target/surefire-reports/*.dumpstream llama/target/surefire-reports/*.dump; do + found=1 + echo "===== $f =====" + cat "$f" + done + if [ "$found" = 0 ]; then + echo "No hs_err_pid*.log and no surefire dump/dumpstream was written." + echo "The fork died without the JVM writing a crash log -- the abort bypassed" + echo "the JVM error handler (native exit()/terminate) rather than raising a signal." + fi - if: failure() uses: actions/upload-artifact@v7 with: @@ -2690,6 +2825,33 @@ jobs: if: always() run: Get-CimInstance Win32_OperatingSystem | Select-Object FreePhysicalMemory,TotalVisibleMemorySize | Format-List shell: pwsh + # A forked test JVM that aborts leaves an hs_err_pid log and a surefire + # dumpstream -- both otherwise ONLY inside the artifact uploaded below, + # which is unreachable from anywhere that cannot fetch from Azure Blob + # (a phone, a restricted network, an agent sandbox). Echo them here so the + # aborting frame is readable from the run page itself. See + # ../workspace/policies/ci-test-diagnostics.md section 3.1. + - name: Print crash logs (on failure) + if: failure() + shell: bash + run: | + shopt -s nullglob + found=0 + for f in llama/hs_err_pid*.log; do + found=1 + echo "===== $f (first 200 lines; full file in the uploaded artifact) =====" + sed -n '1,200p' "$f" + done + for f in llama/target/surefire-reports/*.dumpstream llama/target/surefire-reports/*.dump; do + found=1 + echo "===== $f =====" + cat "$f" + done + if [ "$found" = 0 ]; then + echo "No hs_err_pid*.log and no surefire dump/dumpstream was written." + echo "The fork died without the JVM writing a crash log -- the abort bypassed" + echo "the JVM error handler (native exit()/terminate) rather than raising a signal." + fi - if: failure() uses: actions/upload-artifact@v7 with: From 269076ec60bbee49e9294d328686999e055986bf Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 07:42:06 +0000 Subject: [PATCH 46/76] docs: localise the TTS abort to memset(NULL, 0, n) at load The section-3.1 print step added in cbb5e62 paid off on its first run. The crash log is now in the job log, and it says the same thing on two OS families: Linux x86-64 SIGSEGV, si_code 1 (SEGV_MAPERR), si_addr 0x0 C [libc.so.6+0x1896ca], RDI=0, RSI=0 macOS arm64 SIGSEGV C [libsystem_platform.dylib+0x2fb0] __bzero+0x20 memset(void *s, int c, size_t n) passes s in RDI and c in RSI on x86-64 SysV, so RDI=0, RSI=0 is memset(NULL, 0, n) -- which is exactly what macOS names as bzero. A buffer is being zeroed that was never allocated, during the TTS model load, on the main thread in _thread_in_native, under TextToSpeech.loadNative <- TextToSpeech. <- synthesizesWellFormedWav. Three candidates are ruled out and recorded so the next pass does not repeat them: the JNI signature matches on both sides (unlike LlamaQuantizer, which had exactly that defect); parse_jstring guards null jstrings, pending exceptions and null byte arrays; and tts_engine.cpp's load does null-check model, ctx and mctx. The faulting allocation is inside the load, before those checks. Also records that the earlier "an hs_err may not exist at all" caveat is now settled -- it exists on both platforms checked -- and keeps the reasoning visible rather than deleting it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- TODO.md | 37 ++++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/TODO.md b/TODO.md index add383a4f..01da043fb 100644 --- a/TODO.md +++ b/TODO.md @@ -32,7 +32,42 @@ the crash and reproduced `SessionForkRewindIntegrationTest` (both cases) and not a Windows quirk. Its exit code was **141 (SIGPIPE)** rather than Ubuntu's 134 (SIGABRT), on the same crashed test. -- **[OPEN, MAJOR] `TtsIntegrationTest` aborts the JVM natively on all 6 test platforms.** +- **[OPEN, MAJOR — now localised] `TtsIntegrationTest` aborts the JVM natively on all 6 test + platforms.** The crash log is readable in the job log itself since `cbb5e62` (the section-3.1 + print step), which is what finally produced the following. The abort is a **null-pointer + dereference while zeroing a buffer during the TTS model load**, identically on two OS families: + + | | Linux x86-64 (run 32941522341) | macOS 15 arm64 (same run) | + |---|---|---| + | signal | `SIGSEGV`, `si_code 1 (SEGV_MAPERR)`, `si_addr 0x0` | `SIGSEGV` | + | frame | `C [libc.so.6+0x1896ca]` | `C [libsystem_platform.dylib+0x2fb0] __bzero+0x20` | + | registers | `RDI=0`, `RSI=0` | — | + + On x86-64 SysV `memset(void *s, int c, size_t n)` passes `s` in RDI and `c` in RSI, so + `RDI=0, RSI=0` is `memset(NULL, 0, n)` — the same call macOS names outright as `bzero`. The Java + frames are `TextToSpeech.loadNative` -> `TextToSpeech.` -> + `TtsIntegrationTest.synthesizesWellFormedWav`, on the `main` thread in `_thread_in_native`, + after ~258 s (Linux) / ~288 s (macOS) of elapsed time. + + Ruled out already, do not re-investigate: (1) a JNI signature mismatch of the kind that broke + `LlamaQuantizer` — `loadNative` is `(String, String, int, int) -> long` on both the Java and the + C++ side, verified; (2) `parse_jstring` — it guards a null `jstring`, a pending exception and a + null byte array, returning an empty string in each case; (3) the obvious null checks in + `tts_engine.cpp`'s load, which do test `model` / `ctx` / `mctx` and return `nullptr` with a + message. The faulting allocation is therefore *inside* the load, before those checks are reached. + + Worth noting for the next step: the macOS runner has **7 GB RAM / 3 cores** against Linux's + 15 GB / 4, yet both fail the same way, so a plain out-of-memory on the larger host is a weak + explanation on its own — but an allocation that returns null and is then zeroed unchecked fits + both. The `mmproj` (speaker encoder + code predictor + code2wav) is the largest new buffer in + this path. The native stack could not be walked on Linux (one libc frame only); the full + `hs_err` in the artifact has the memory map if the allocation size matters. + + Superseded note (kept because the reasoning was cited earlier): it was NOT certain an `hs_err` + existed at all — `if-no-files-found: warn` and Windows' exit code 1 left that open. It does + exist, on both platforms checked. + +- **[SUPERSEDED — see the entry above] `TtsIntegrationTest` aborts the JVM natively on all 6 test platforms.** Ubuntu exit 134 (SIGABRT); macOS 14 Metal, macOS 15 Metal, macOS 15 no-Metal; Windows Ninja and Windows MSVC exit 1. Not an OOM (Linux had ~14 GiB free, Windows ~12.3 GiB of 16 GiB). Not caused by the bump: `git log b10456..b10618 -- tools/mtmd/mtmd-helper.{cpp,h}` contains only video/webp/ From 2f99c3e6f34a5f7be04aa0927fb00624e1b8ac3d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 07:45:01 +0000 Subject: [PATCH 47/76] docs: correct two misreadings of the TTS crash frames The macOS Metal job resolved the native stack where Linux could not, and it is two frames: __bzero+0x20 called from Java_..._TextToSpeech_loadNative+0x60. That invites the conclusion that the fault is in our JNI wrapper. Disassembling the shipped library refutes it: loadNative contains no memset or bzero anywhere, and its only calls are parse_jstring, engine_init through the PLT (so not inlined), operator delete and __stack_chk_fail. The frame-pointer walker dropped the intermediate frames and left the outermost and innermost. The fault is under engine_init -- common_init_from_params or the mtmd/mmproj init. Symbol attribution itself is trustworthy here and the note says so, because the distinction matters for the next reader: the library exports 12529 symbols including the whole llama/common layer, so a PC inside common_init_from_params would have been named as such. It is the depth of the walk that is unreliable, not the naming. Also corrects the elapsed-time reading. 258 s / 300 s is total JVM elapsed time since the fork started, which includes running the rest of the suite; it is not time spent inside the load and says nothing about how far the load got. The earlier note leaned on it to argue the model was largely loaded. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- TODO.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/TODO.md b/TODO.md index 01da043fb..71f78a6e0 100644 --- a/TODO.md +++ b/TODO.md @@ -47,7 +47,9 @@ same crashed test. `RDI=0, RSI=0` is `memset(NULL, 0, n)` — the same call macOS names outright as `bzero`. The Java frames are `TextToSpeech.loadNative` -> `TextToSpeech.` -> `TtsIntegrationTest.synthesizesWellFormedWav`, on the `main` thread in `_thread_in_native`, - after ~258 s (Linux) / ~288 s (macOS) of elapsed time. + after ~258 s (Linux) / ~288-300 s (macOS) of **total JVM elapsed time** -- that is time since the + fork started and ran the rest of the suite, NOT time spent inside the load, so it says nothing + about how far the load got. Ruled out already, do not re-investigate: (1) a JNI signature mismatch of the kind that broke `LlamaQuantizer` — `loadNative` is `(String, String, int, int) -> long` on both the Java and the @@ -56,6 +58,17 @@ same crashed test. `tts_engine.cpp`'s load, which do test `model` / `ctx` / `mctx` and return `nullptr` with a message. The faulting allocation is therefore *inside* the load, before those checks are reached. + **The macOS 15 Metal job resolved the native stack, and it is exactly two frames:** + `__bzero+0x20` called from `libjllama.dylib` `Java_..._TextToSpeech_loadNative+0x60`. Do NOT read + that as "the bug is in `loadNative`". Disassembling the shipped Linux library shows `loadNative` + contains **no** `memset`/`bzero` at all — its only calls are `parse_jstring`, `engine_init` (via + the PLT, so not inlined), `operator delete` and `__stack_chk_fail`. The JVM's frame-pointer + walker dropped the intermediate frames, leaving only the outermost and innermost. The fault is + therefore under `engine_init` — in `common_init_from_params` or the mtmd/mmproj init — not in the + JNI wrapper. (Symbol attribution itself *is* trustworthy here: the library exports 12 529 symbols, + the whole llama/common layer included, so a PC inside `common_init_from_params` would have been + named as such. What is unreliable is the *depth* of the walk, not the naming.) + Worth noting for the next step: the macOS runner has **7 GB RAM / 3 cores** against Linux's 15 GB / 4, yet both fail the same way, so a plain out-of-memory on the larger host is a weak explanation on its own — but an allocation that returns null and is then zeroed unchecked fits From 4c39d6b2c25507fe9f56269fd6f745057c83d284 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 08:21:29 +0000 Subject: [PATCH 48/76] chore: upgrade llama.cpp from b10618 to b10631 Single step: the raw diff is ~466 KiB, but ~377 KiB of that is tools/ui, which CI rebuilds from GIT_TAG and which needs no source review. The reviewable remainder is ~89 KiB, under the runbook's 100 KiB threshold. Three upstream edits touch anything the project compiles, and none is structural. oaicompat_chat_params_parse gains a narrowing validation -- continuing a final assistant message that carries tool_calls now throws -- on a combination the project never constructs, since OpenAiRequestMapper never sets continue_final_message. common_chat_params_init_qwen3_coder gates the bare opener and the tool-call-first rule on a new is_qwen3_coder flag, which is confined to one static grammar initialiser. LLAMA_VERSION_MINOR goes 2 -> 3, cosmetic. The runbook's three mechanical contract checks for a tools/server/ chunk were not just run but rendered moot: server-schema.cpp, server-task.cpp, server-context.cpp, the four tools/server/*.h headers, common/common.h, include/llama.h and mtmd-helper.h are byte-identical across the range, compared by blob SHA. The request-field set, its bounds and the emitted response keys therefore cannot have moved -- which is precisely the silent-contract class that cost this project getMetrics() for hundreds of builds. Verified on the shipped tree, in a fresh build directory because the applier pins its stamp to the checked-out llama.cpp commit and deliberately aborts rather than guess when an existing one names a different commit: all 7 patches apply against b10631 (stamp written, 41 files dirty), full Release build clean, ctest 499/499, and nm -D reports zero C++-mangled Java_* exports so the LlamaQuantizer linkage fix carries forward. NativeLibraryLoadSmokeTest 3/3, including nativeBuildInfoMatchesPinnedVersionConstant, which is what proves the four pin sites agree with the binary that was actually linked. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH --- CHANGELOG.md | 8 ++++++++ CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 6 files changed, 20 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 043f7394a..e5e3cc063 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -62,6 +62,14 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by guarded for old glibc in the same change. - Android/Gradle toolchain: Gradle pins moved 8.14.3 → 9.6.1 and the dockcross cross-compile images were bumped, alongside the AGP/Compose pin updates the Android builds needed. +- Upgraded llama.cpp from **b10618 to b10631**. No project-source change: the only edits in the + range are a narrowing input validation in `oaicompat_chat_params_parse` (continuing a final + assistant message that carries `tool_calls` now throws), a Qwen3-Coder-only grammar refinement + in `common_chat_params_init_qwen3_coder`, a cosmetic `LLAMA_VERSION_MINOR` bump, and the WebUI. + `server-schema.cpp`, `server-task.cpp`, `server-context.cpp`, the `tools/server/*.h` headers, + `common/common.h`, `include/llama.h` and `mtmd-helper.h` are byte-identical across the range, so + neither the request-field set and its bounds nor the emitted response keys can have moved. All + seven local patches re-verified against a clean b10631 checkout; C++ suite 499/499. - Upgraded llama.cpp from **b10456 to b10618**, in 25 reviewed steps. Patch `0007` refreshed (upstream #26347 deleted comments inside its removal block, breaking `git apply` at every tag from b10519 on) and a new patch `0010` carries a one-line upstream fix: `GET /models` emitted `vocab_type` as a JSON boolean diff --git a/CLAUDE.md b/CLAUDE.md index eca4ae865..2f56dde06 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10618** +Current llama.cpp pinned version: **b10631** ## Upgrading CUDA Version @@ -490,7 +490,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b10618 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10631 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -530,7 +530,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10618`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10631`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1441,7 +1441,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10618`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10631`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index ecf7c43f5..eacdf9daf 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10618](https://img.shields.io/badge/llama.cpp-%23b10618-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10618) +[![llama.cpp b10631](https://img.shields.io/badge/llama.cpp-%23b10631-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10631) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 65e9d57f7..14bae829f 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -671,3 +671,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10456–b10618 | upstream verification (sandbox, final target) | **Full local verification, re-run from scratch on the shipped tree.** Fresh `cmake -B build` with the real `FetchContent` path (no source-dir override), so the fail-loud `PATCH_COMMAND` ran for real: it resolved `ggml commit eb25b7263` / `LLAMA_BUILD_NUMBER = 10618` and applied all **7** patches — verified by grepping the fetched tree for each patch's marker (`common_params_parse_main`, `params_base.load_progress_callback == nullptr`, `get_slot_prompt_similarity`, `g_llama_server_embedded`, `llama_server_attach` + `llama_server_register_common_routes`, `LLAMA_SERVER_WORKER_CMD`, `(int) meta.model_vocab_type`). The **7**, not 6, matters: the review-driven `0010` was added after the first verification pass and is the *third* patch to touch `server-context.cpp` (after `0002`/`0003`), a TU compiled into both `jllama` and `jllama_test`, so the whole configure + build + `ctest` cycle was re-run on the final tree rather than assumed to carry over (**491/491**, `mvn test` unchanged at 1405 run / 0 failures / 17 model-gated skips). The patch was subsequently renamed `0009` → `0010` to free the burned number; that is a filename-only change with the same sort position, re-checked by applying the whole set to a clean b10618 checkout. Then a full `cmake --build --config Release` (jllama + jllama_test both link — the `jllama_test` link is what proves `common_json::get()` resolves, since upstream documents un-specialised types as a *link*-time failure) and `ctest`: **491/491 C++ tests pass**. On the Java side: `mvn test-compile` clean, `mvn test` **1405 run / 0 failures / 17 skipped** (only the model-gated integration tests, no GGUF in the sandbox), and `NativeLibraryLoadSmokeTest` green — including `nativeBuildInfoMatchesPinnedVersionConstant`, the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10618") matches the `build-info` compiled into the freshly linked `libjllama.so`. `mvn spotless:apply` produced no changes beyond the edits themselves, the pinned **clang-format 22.1.8** reports the whole C++ tree clean, and `mvn clean javadoc:jar` is `BUILD SUCCESS`. **Per-step verification for all 25 chunks:** every intermediate tag additionally had all six patches applied by the applier's own commands and all ten project + C++-test translation units compiled against it, so no commit in this walk is a state that fails to build. **Two notes on how to read the per-chunk rows above.** (1) They say "all **6** patches" because that was the set during the walk; the 7th (`0010`) came out of the post-bump review and exists only from this final commit on. (2) Their prose reasoning about patch risk scans the directories that matter for the project's *compile* surface (`common/`, `tools/server/`, `tools/mtmd/`, `include/`, `vendor/`, `CMakeLists.txt`) and refers to "the five core patched files" — but the patch set actually spans **40** distinct files, because `0001` flips ~34 standalone `main()` call sites under `tools/*`, `examples/*` and `tests/*`. Two ranges did touch one of those and the prose does not say so on its own (b10488–b10499 → `tools/perplexity/perplexity.cpp`, b10590–b10593 → `tests/test-recurrent-state-rollback.cpp`; both now corrected in place). The *verdicts* were never derived from that directory scan: each chunk's patch verdict comes from really running `git apply` for the whole set against a clean checkout of that tag, which covers all 40 files. The reusable check is `git diff --name-only -- $(grep -h '^+++ b/' llama/patches/*.patch | sed 's|^+++ b/||' | sort -u)`. | | b10456–b10618 | `tools/server/server-context.cpp` — **new local patch `0010-server-cast-vocab-type-for-common-json.patch`** | **Upstream regression found by the post-bump review, fixed downstream.** `get_res_model_info()` builds the `GET /models` + `GET /v1/models` payload and emits `{"vocab_type", meta.model_vocab_type}`, an **unscoped enum**. That was correct while the `json` alias was `nlohmann::ordered_json` (nlohmann serialises an enum as an integer), but `common_json_value`'s integral constructor template is `std::is_integral`-gated, which excludes enums — so from **b10585** the value binds to `common_json_value(bool)` and goes on the wire as `true`/`false`. Upstream regressed it silently in #27511 when they flipped the alias; the same trap hit the project's own two `"vocab_type"` emit sites in `jllama.cpp` (fixed in the b10584–b10585 step). It **ships**: `server-context.cpp` is compiled into `libjllama` and both routes are served by `NativeServer`, the default fat-jar `Main-Class`, in full and in attach mode. The new patch casts the value to `int` at the emit site — one line, upstream-submittable (not yet filed), applied after `0002`/`0003` (same file). Numbered `0010`, not `0009`: that number belongs to the subprocess.h patch dropped at b10280 and is documented as such under the patches table in `CLAUDE.md`, so reusing it would make that note read as if it described this patch. A mechanical sweep confirmed it is the **only** such site in the upstream server sources the project compiles: `server_context_meta` has exactly two enum-typed members, and the other (`pooling_type`) is only ever compared, never serialised. | | b10456–b10618 | follow-up: `src/main/cpp/{jllama.cpp,json_helpers.hpp}`, `value.ServerMetrics`, `server.OpenAiCompatServer` | **Client-contract restoration, after the walk.** The b10405–b10408 and b10509–b10519 rows above record the upstream metrics refactor correctly but understate its client impact: b10408 (#26920) reduced `server_task_result_metrics::to_json()` to the bare slot array and b10519 (#27376) split the task, so `LlamaModel.getMetrics()` had been returning the slot array — not the object its three Java consumers parse — since **before** this bump range began. Re-pointing `handleSlotAction` at `SERVER_TASK_TYPE_SLOT_GET` (b10519 row) kept the array arriving but did not restore the object. Fixed by posting **both** tasks and merging them in the new pure helper `server_metrics_to_json`, so the documented payload is rebuilt in the JNI layer instead of the Java contract following upstream's transport split; durations are converted µs→ms as the pre-b10408 payload did. Reachable-but-unexposed counters (`n_prompt_cached`, the speculative-decoding tallies) are emitted alongside and surfaced on `ServerMetrics`. `GET /slots` no longer answers 200-with-empty-body when the payload has no `slots` key. Suite 491 → **499**. **This is the class of break a header diff cannot catch**: no signature changed, the project compiled and linked clean at every one of the 25 chunks, and the only guard — `LlamaModelTest#testGetMetrics` — was model-gated and silently skipped in CI (see the same-day CI working-directory fix). | +| b10618–b10631 | `tools/server/server-common.cpp` (**new validation in `oaicompat_chat_params_parse`: continuing a final assistant message that carries `tool_calls` now throws `std::invalid_argument("Cannot continue an assistant message that contains tool calls.")`**), `common/chat.cpp` (**`common_chat_params_init_qwen3_coder` gains `is_qwen3_coder = !supports_reasoning` and gates two things on it: the `` bare-opener alternatives are only added for Coder models, and `tool-call-first` collapses to plain `tool-call` for the reasoning variant**), `CMakeLists.txt` (**`LLAMA_VERSION_MINOR` 2 → 3, cosmetic**), plus `tools/ui/**` (the WebUI, auto-followed) | **No project-source change.** Both server-side edits are behavioural, not structural: no signature, no struct field and no JSON key moved. The `oaicompat_chat_params_parse` addition is a strictly *narrowing* input validation on a combination the project never constructs — `OpenAiRequestMapper` never sets `continue_final_message`, and the C++ guard `test_utils.cpp` exercises the function without it (`ctest` 499/499 unchanged). The `chat.cpp` change is confined to one `static` template initialiser for the Qwen3-Coder family and touches only grammar construction; it is invisible at the API surface and the project pins no Coder model in CI. **The three mechanical contract checks the runbook mandates for a `tools/server/` chunk were not merely run but rendered moot:** `server-schema.cpp`, `server-task.cpp`, `server-context.cpp` and the four `tools/server/*.h` headers are **byte-identical** between the two tags (verified by comparing blob SHAs), so the request-field set, its bounds and the emitted response-key set cannot have moved. `common/common.h`, `include/llama.h` and `tools/mtmd/mtmd-helper.h` are likewise byte-identical, so the `getMetrics()`-class of silent contract break and the `tts_engine.cpp` surface are both out of scope for this range. Sizing note for the reviewer: the raw diff is ~466 KiB, but ~377 KiB of that is `tools/ui`, which CI rebuilds from `GIT_TAG` and which needs no source review — the reviewable remainder is ~89 KiB, under the runbook's 100 KiB single-step threshold. | +| b10618–b10631 | upstream verification (sandbox, target) | **Full local verification on the shipped tree.** Fresh `cmake -B build-b10631 -DBUILD_TESTING=ON` through the real `FetchContent` path, so the fail-loud `PATCH_COMMAND` ran for real against b10631: all **7** patches applied, confirmed by the stamp file (`head 5d5cb4c3a…` plus one SHA-256 line per patch) and by the source tree going dirty in 41 files. A fresh build directory was mandatory here — the applier pins its stamp to the checked-out llama.cpp commit and deliberately aborts rather than guess when an existing build dir's stamp names a different one. Full `cmake --build --config Release` clean (no errors, no new warnings), `ctest` **499/499**. `nm -D` on the freshly linked `libjllama.so` reports **zero** C++-mangled `Java_*` exports, so the `LlamaQuantizer` linkage fix carries forward. `NativeLibraryLoadSmokeTest` **3/3**, including `nativeBuildInfoMatchesPinnedVersionConstant` — the end-to-end cross-check that `LlamaCppVersion.LLAMA_CPP_VERSION` ("b10631") matches the `build-info` compiled into the new binary, which is what proves the four pin sites and the actual build agree. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 4eb832007..2ee30f53d 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10618 + GIT_TAG b10631 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 06c70428d..4e6e64600 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *