Repository navigation
docs(readme): tool calling is verified per family, not "experimental / not reliable yet" - #470
Merged
Merged
Conversation
…/ not reliable yet" The README still carried the May 2026 status wording that tool calling "is not reliable yet" and "may fail to trigger or parse even when plain generation works". Since 0.53.0 the NATIVE providers are verified end-to-end on real checkpoints by model-gated smoke tests (CHANGELOG "Verified — tool calling against engine 0.53.0", and the Qwen3 chat-template rewrite in the same release). Reword the feature bullet, the status warning and the near-term roadmap item accordingly; the GENERIC fallback for unknown families stays described as best-effort. Proof, re-run on 2026-10-09 against engine 0.57.0 (develop @ 2f80182, Linux x86_64, i7-9750H, JDK 25): QWEN_MODEL_PATH=.../Qwen3-0.6B-Q8_0.gguf ./gradlew :llm-runtime:kllama:jvmTest --tests QwenToolCallSmokeTest --rerun tests=2 failures=0 skipped=0 (61.9 s) metadata: family=qwen, arch=qwen3, hints=[<tool_call>, <tool_response>, <|im_start|>] model output: <tool_call>{"name": "calculator", "arguments": {"expression": "17 * 23"}}</tool_call> -> parsed, calculator executed by the agent loop, final answer produced FUNCTIONGEMMA_GGUF=.../functiongemma-270m-it-Q8_0.gguf ./gradlew :llm-runtime:kgemma:jvmTest --tests FunctionGemmaOfficialGgufTest --rerun tests=3 failures=0 skipped=0 (12.1 s) model output: <start_function_call>call:get_weather{location:<escape>Paris<escape>}<end_function_call> -> parsed get_weather call; template tokens and prompt tokenization pinned against llama.cpp Not re-run here (checkpoints not cached on this machine): Gemma4E2BToolCallSmokeTest, SmolLM2ToolCallSmokeTest; both are listed as green in the 0.53.0 changelog entry. A Llama 3 real-checkpoint smoke test does not exist yet and the roadmap now says so.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The README still carried the May 2026 status wording that tool calling "is not reliable yet" and "may fail to trigger or parse even when plain generation works". Since 0.53.0 the NATIVE providers are verified end-to-end on real checkpoints by model-gated smoke tests (CHANGELOG: Verified — tool calling against engine 0.53.0, plus the Qwen3 chat-template rewrite in the same release). The wording undersells what the changelog already documents.
What
Proof
Re-run 2026-10-09 against engine 0.57.0 (develop @ 2f80182, Linux x86_64, i7-9750H, JDK 25). JUnit XML checked: no test case skipped.
:llm-runtime:kllamaQwenToolCallSmokeTest:llm-runtime:kgemmaFunctionGemmaOfficialGgufTestQwen3 output through the auto-detected
qwenprovider (family=qwen, arch=qwen3):parsed, calculator executed by the agent loop, final answer produced.
FunctionGemma output (OFFICIAL style):
parsed
get_weathercall; template tokens and prompt tokenization pinned against the llama.cpp reference.Not re-run here because the checkpoints are not cached on this machine:
Gemma4E2BToolCallSmokeTest,SmolLM2ToolCallSmokeTest; both are recorded green in the 0.53.0 changelog entry.Scope
README only, no code change.