From 229903cbf93722864cb9458731983b29cebafe60 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 1 Sep 2026 08:18:12 +0000 Subject: [PATCH 1/3] feat: wire flashAttn up for real, now that the binding can express it The knob has been documented and settable since it was introduced, and could never be forwarded. --flash-attn takes a mandatory on|off|auto, the binding offered only a bare-flag setter, and emitting the key alone made llama.cpp's parser consume the next argv token -- so the load died naming a flag the user never set. 1.2.0 shipped a refusal at plan time rather than that diagnostic. net.ladenthin:llama 5.2.0 adds ModelParameters.setFlashAttn(FlashAttn), so the provider now emits --flash-attn on when the knob is set. false keeps emitting nothing, which leaves llama.cpp's own auto default in force: that is what an unconfigured run should get, and it is what every other optional knob here does with its sentinel. Both refusals go, along with the three tests that pinned them, and the DEFAULT_FLASH_ATTN javadoc stops describing a knob that cannot be used. The history is kept in that javadoc rather than deleted -- the reason the value matters is exactly why the knob spent a release unusable. The part that is more than bookkeeping: flashAttn moves out of the knob sweep's NOT_SWEPT list into a real sweep case, so it is now set against a real model on every run. Until now only the refusal had ever been executed, which is what the removed TODO asked for. 36 sweep cases green, plus EngineSupport, AiGenerationConfig and the factory mapping -- 105 tests. llama.version moves to 5.2.0-SNAPSHOT deliberately: the binding change is not published yet, so CI stays red until it is. The CHANGELOG says so, so a red build is not mistaken for a regression. --- CHANGELOG.md | 28 ++++++++++ TODO.md | 27 ---------- srcmorph/pom.xml | 2 +- .../srcmorph/config/AiGenerationConfig.java | 23 +++++---- .../srcmorph/engine/EngineSupport.java | 25 --------- .../LlamaCppJniAiGenerationProvider.java | 43 +++------------- .../srcmorph/engine/EngineSupportTest.java | 51 ------------------- .../provider/LlamaCppJniKnobSweepTest.java | 42 +++------------ 8 files changed, 56 insertions(+), 185 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cb6fda20..f17dee98 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,34 @@ The release procedure (prompt template and step-by-step instructions) lives in [ --- +## [Unreleased] + +### Added +- **`flashAttn` works.** The knob was documented and settable since it was introduced, but could not + be forwarded: `--flash-attn` takes a mandatory `on|off|auto` and `net.ladenthin:llama` offered only + a bare-flag setter, which emitted the key alone and made llama.cpp's parser consume the following + argv token — the load then died naming a flag the user never set. 1.2.0 shipped a refusal at plan + time rather than that diagnostic. + + `net.ladenthin:llama` **5.2.0** adds `ModelParameters.setFlashAttn(FlashAttn)`, so the provider now + emits `--flash-attn on` when the knob is set. `false` still emits nothing, which leaves llama.cpp's + own `auto` default in force — that is the correct behaviour for an unconfigured run, and it is what + the knob's javadoc now says. + + **The knob is exercised against a real model for the first time.** `flashAttn` moves out of + `LlamaCppJniKnobSweepTest`'s `NOT_SWEPT` list into a real sweep case; until now only the refusal had + ever been executed. + +### Removed +- The plan-time and provider-side refusals (`EngineSupport.validateFlashAttnIsNotRequested`, + `LlamaCppJniAiGenerationProvider.FLASH_ATTN_UNSUPPORTED_MESSAGE`) and the three tests that pinned + them. + +### Changed +- **`net.ladenthin:llama` 5.1.0 → 5.2.0-SNAPSHOT.** Deliberately a snapshot: the binding change this + release depends on is not yet published. Building srcmorph therefore requires that snapshot to be + resolvable, and CI stays red until it is — recorded here so nobody mistakes it for a regression. + ## [1.2.0] - 2026-09-01 ### Added diff --git a/TODO.md b/TODO.md index 7c813f52..a89c3d11 100644 --- a/TODO.md +++ b/TODO.md @@ -55,33 +55,6 @@ recorded in git history and `crossrepostatus.md`, not here. not dropping the idea. Deliberately out of scope for 1.2.0: it is a build-time question, not a correctness one. -- **Wire `flashAttn` up for real after `net.ladenthin:llama` 5.2.0 ships.** 1.2.0 *refuses* the knob - rather than forwarding it, in two places (`LlamaCppJniAiGenerationProvider.model()` for the direct - API path, `EngineSupport.validateFlashAttnIsNotRequested` for the plan phase), because the binding - cannot currently spell it: `-fa` takes a mandatory `[on|off|auto]`, `enableFlashAttn()` routes - through `setFlag` which stores `null`, and the argv renderer then emits the key with no token after - it — so llama.cpp swallows the *next* argv token and the load dies naming a flag the user never set - (`error: unknown value for --flash-attn: '--reasoning-format'`, reproduced against the shipped fat - jar). No workaround exists downstream: `putScalar` is `protected`. - - When 5.2.0 lands with a value-taking setter: bump `llama.version`, replace both guards with the real - call, delete `FLASH_ATTN_UNSUPPORTED_MESSAGE` and the four tests that pin the refusal (two in - `EngineSupportTest`, one in `LlamaCppJniAiGenerationProviderTest`, and - `LlamaCppJniKnobSweepTest#flashAttn_isRefusedRatherThanSilentlyDropped`), and restore the - `flashAttn`/`cacheTypeV` guidance in `srcmorph-maven-plugin/README.md`. Then move `flashAttn` out of - `LlamaCppJniKnobSweepTest.NOT_SWEPT` into a real sweep case — that is the step that finally sets the - knob against a real model, which the refusal has never done. The sweep's completeness check keeps - the bookkeeping honest: it fails if the knob is neither swept nor excluded, so it cannot be left - half-migrated. - - **The trap to avoid: bumping `llama.version` alone changes nothing visible.** The guards keep - throwing, the knob keeps looking broken, and every gate stays green — so this has to be an explicit - line on the bump checklist rather than something a version bump surfaces on its own. The upstream - fix is tracked in java-llama.cpp (`ModelParameters` needs `setFlashAttn(on|off|auto)` mirroring the - `CacheType` enum pattern, `enableFlashAttn()` deprecated, and - `ModelParametersExtendedTest.testToArrayComplexCombination` corrected — it currently pins the broken - 9-token argv shape as correct). - - **`enable_thinking` is sent unconditionally, including at its own default.** `LlamaCppJniAiGenerationProvider.model()` always puts `enable_thinking` into `chatTemplateKwargs`, at whatever `chatTemplateEnableThinking` says — and its default is `true`. diff --git a/srcmorph/pom.xml b/srcmorph/pom.xml index c0d176cd..4a9ae65d 100644 --- a/srcmorph/pom.xml +++ b/srcmorph/pom.xml @@ -50,7 +50,7 @@ SPDX-License-Identifier: Apache-2.0 templating is unchanged. The rest of the provider's API surface (LlamaModel, InferenceParameters, ModelParameters, ChatResponse/Timings/Pair, ChatResponseParser, ReasoningFormat) is untouched by 5.1.0. --> - 5.1.0 + 5.2.0-SNAPSHOT - 5.2.0-SNAPSHOT + ReasoningFormat) is untouched by 5.1.0. + 5.2.0 is a BREAKING bump this module needs, not an optional one: the binding removed + ModelParameters.enableFlashAttn()/ModelFlag.FLASH_ATTN (--flash-attn takes a mandatory + on|off|auto value since llama.cpp b10273, so the valueless flag emitted a broken argv) + and renamed setTensorReadLazy/TensorReadLazyMode to setLazyMode/LazyMode (llama.cpp + b10731 renamed --tensor-read-lazy to --lazy-mode with no alias). The provider calls + both, so it cannot compile against 5.1.0 any more. + This deliberately names the RELEASE, not 5.2.0-SNAPSHOT: a snapshot would need a + entry for the Central snapshot channel (this repo declares none -- the + distributionManagement/snapshotRepository below is deploy-only) AND would block every + srcmorph release, since Central rejects a release with a -SNAPSHOT dependency. The + build therefore stays red until net.ladenthin:llama:5.2.0 is published, and goes green + with no further edit the moment it is. --> + 5.2.0