Skip to content

docs(readme): tool calling is verified per family, not "experimental / not reliable yet" - #470

Merged
michalharakal merged 1 commit into
developfrom
docs/readme-tool-calling-status
Oct 10, 2026
Merged

michalharakal merged 1 commit into
developfrom
docs/readme-tool-calling-status

Conversation

@michalharakal

Copy link
Copy Markdown
Contributor

Why

The README still carried the May 2026 status wording that tool calling "is not reliable yet" and "may fail to trigger or parse even when plain generation works". Since 0.53.0 the NATIVE providers are verified end-to-end on real checkpoints by model-gated smoke tests (CHANGELOG: Verified — tool calling against engine 0.53.0, plus the Qwen3 chat-template rewrite in the same release). The wording undersells what the changelog already documents.

What

  • Feature bullet: drop "(experimental)" and "not reliable yet"; list the families with NATIVE providers, name the smoke tests that verify them on real checkpoints, keep the GENERIC fallback described as best-effort, and note that zero-shot tool-call quality is still the model's (Qwen2.5-0.5B figures from 0.57.0).
  • Status warning: keep the disclaimer, but scope the "can fail to trigger or parse" example to the GENERIC fallback and link the native-vs-generic table.
  • Near-term roadmap: drop "including tool calling"; state the one real gap, there is no Llama 3 real-checkpoint smoke test yet.

Proof

Re-run 2026-10-09 against engine 0.57.0 (develop @ 2f80182, Linux x86_64, i7-9750H, JDK 25). JUnit XML checked: no test case skipped.

Test Model Result
:llm-runtime:kllama QwenToolCallSmokeTest Qwen3-0.6B Q8_0 2 passed, 0 skipped, 61.9 s
:llm-runtime:kgemma FunctionGemmaOfficialGgufTest FunctionGemma 270M Q8_0 3 passed, 0 skipped, 12.1 s

Qwen3 output through the auto-detected qwen provider (family=qwen, arch=qwen3):

<tool_call>
{"name": "calculator", "arguments": {"expression": "17 * 23"}}
</tool_call>

parsed, calculator executed by the agent loop, final answer produced.

FunctionGemma output (OFFICIAL style):

<start_function_call>call:get_weather{location:<escape>Paris<escape>}<end_function_call>

parsed get_weather call; template tokens and prompt tokenization pinned against the llama.cpp reference.

Not re-run here because the checkpoints are not cached on this machine: Gemma4E2BToolCallSmokeTest, SmolLM2ToolCallSmokeTest; both are recorded green in the 0.53.0 changelog entry.

Scope

README only, no code change.

…/ not reliable yet"

The README still carried the May 2026 status wording that tool calling "is not
reliable yet" and "may fail to trigger or parse even when plain generation
works". Since 0.53.0 the NATIVE providers are verified end-to-end on real
checkpoints by model-gated smoke tests (CHANGELOG "Verified — tool calling
against engine 0.53.0", and the Qwen3 chat-template rewrite in the same
release). Reword the feature bullet, the status warning and the near-term
roadmap item accordingly; the GENERIC fallback for unknown families stays
described as best-effort.

Proof, re-run on 2026-10-09 against engine 0.57.0 (develop @ 2f80182,
Linux x86_64, i7-9750H, JDK 25):

  QWEN_MODEL_PATH=.../Qwen3-0.6B-Q8_0.gguf
  ./gradlew :llm-runtime:kllama:jvmTest --tests QwenToolCallSmokeTest --rerun
    tests=2 failures=0 skipped=0 (61.9 s)
    metadata: family=qwen, arch=qwen3, hints=[<tool_call>, <tool_response>, <|im_start|>]
    model output: <tool_call>{"name": "calculator", "arguments": {"expression": "17 * 23"}}</tool_call>
    -> parsed, calculator executed by the agent loop, final answer produced

  FUNCTIONGEMMA_GGUF=.../functiongemma-270m-it-Q8_0.gguf
  ./gradlew :llm-runtime:kgemma:jvmTest --tests FunctionGemmaOfficialGgufTest --rerun
    tests=3 failures=0 skipped=0 (12.1 s)
    model output: <start_function_call>call:get_weather{location:<escape>Paris<escape>}<end_function_call>
    -> parsed get_weather call; template tokens and prompt tokenization pinned against llama.cpp

Not re-run here (checkpoints not cached on this machine): Gemma4E2BToolCallSmokeTest,
SmolLM2ToolCallSmokeTest; both are listed as green in the 0.53.0 changelog entry.
A Llama 3 real-checkpoint smoke test does not exist yet and the roadmap now says so.
@michalharakal
michalharakal merged commit 4cc241a into develop Oct 10, 2026
2 checks passed
@michalharakal
michalharakal deleted the docs/readme-tool-calling-status branch October 10, 2026 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant