feat(agent): code-style tool dialects (python / typescript) behind agent.tool_dispatcher - #6433
Merged
Merged
Conversation
Add the tinyagents library as a vendored dependency to support agent-based workflows in the project. This change introduces the external codebase directly into the repository, ensuring consistent availability and versioning without requiring a separate package installation step. Auto-committed-on: macbook
The tinyagents submodule is updated to a newer commit, bringing in upstream changes. Auto-committed-on: macbook
Add support for code-style tool call formats in Python and TypeScript, enabling models to invoke tools using language-native syntax. These formats are opt-in via the `tool_dispatcher` config and provide a compact alternative to P-Format that aligns with how code-trained models naturally express function calls. Auto-committed-on: macbook
The turn harness now runs with the same tool dialect the session composed its prompt for, instead of letting the harness pick from the model profile. This ensures text-based protocols strip schemas off the wire and use the positional registry to recover P-Format or code-style calls, while `Auto` still allows fallback for models that do not support native tools. Auto-committed-on: macbook
Add support for Python and TypeScript code-style tool calls via the `tool_dispatcher` config, selectable with `"python"` or `"typescript"` values, and expose a one-launch `OPENHUMAN_TOOL_DISPATCHER` environment variable to override the setting. The new code dialects are opt-in like P-Format, since their compact syntaxes can mis-parse on some models, and the dispatcher resolution now falls back to auto with a warning on unknown spellings. Auto-committed-on: macbook
Add a test verifying that a non-blank OPENHUMAN_TOOL_DISPATCHER environment variable overrides the agent's tool_dispatcher field, while blank values leave the persisted choice unchanged. Auto-committed-on: macbook
…prompts Added handling for Python and TypeScript tool call formats in the subagent system prompt renderer, reusing the canonical code catalogue renderer and dialect instructions to ensure consistent tool descriptions and usage guidance across these formats. Auto-committed-on: macbook
Added a test verifying that under Python and TypeScript code dialects, the tools section renders each tool as a single function signature with a trailing comment, omits the "Call as:" and JSON schema blocks, and appends the dispatcher protocol block. Auto-committed-on: macbook
Add a test verifying that the subagent system prompt renders tool signatures and call examples in both Python and TypeScript formats, and that it includes the tool use protocol section while omitting the JSON-style "Parameters:" and "Call as:" sections. Auto-committed-on: macbook
Added a test verifying that each tool call format dialect maps to the correct harness dispatcher and code style, including the intentional fallback of Native to Auto. Auto-committed-on: macbook
…dule The dispatcher kind resolution and provider role logic have been moved out of the factory module into a dedicated dispatcher module, keeping the factory focused on construction. The provider role tests now reference the dispatcher module directly for the shared types. Auto-committed-on: macbook
The `provider_role_for` function has been relocated from the dispatcher module to the factory module, where it is more appropriately scoped. This change does not alter any behavior; it simply reorganizes the code to improve maintainability and cohesion. Auto-committed-on: macbook
Reformat several files in the agent module to conform to rustfmt's line-width and import-ordering rules, including wrapping long assertions, joining split string literals, and reordering use statements. No behavior changes are introduced. Auto-committed-on: macbook
The agent harness documentation now explains how tool dispatch is selected and how the `auto` mode falls back to JSON-in-tag for providers without native support. It also details the XML, P-Format, and code dialects, including their syntax and when they are used, and clarifies that a shared parser handles mixed forms in transcripts. Auto-committed-on: macbook
Adds a manual, network-touching benchmark binary for comparing text tool-call dialects against a local Ollama model, measuring prompt tokens and call accuracy per `agent.tool_dispatcher` value. This is never run by CI and is documented in the file's header for invocation. Auto-committed-on: macbook
The benchmark was formatting the provider string with a constant that may not be defined, so it now uses the explicit "ollama:" prefix to ensure correct provider resolution. Auto-committed-on: macbook
The JSON dialect's protocol block already embeds a full-schema catalogue, so rendering the signature catalogue as well listed every tool twice. This change suppresses the redundant signature output when a JSON dispatcher is active, reducing prompt size by roughly 13 KB. Auto-committed-on: macbook
… catalogue Added a test verifying that when the XML dialect embeds the full-schema catalogue, the tools section lists each tool only once, preventing the previous duplication of the orchestrator's 31 tools. Auto-committed-on: macbook
The benchmark now accepts a `--max-output-tokens` flag to override the default per-call ceiling, which has been raised to 1500 tokens to accommodate thinking models that spend most of their budget in the reasoning channel. Additionally, responses that are empty but consumed the full output budget are now reported as an "output cap reached" error rather than being mistaken for a dialect failure. Auto-committed-on: macbook
Added the `use tinytools_agent::dialect::ToolDialect as _;` import to bring the trait into scope, which is required for the `prompt_instructions` method call to resolve correctly. Auto-committed-on: macbook
Reformatted the code in the tool dialect benchmark binary for improved readability and consistency with the project's style guidelines. No functional changes were made. Auto-committed-on: macbook
The match arm for `ToolCallFormat::Json` with non-empty dispatcher instructions is reformatted to a single line, reducing unnecessary line breaks without changing behavior. Auto-committed-on: macbook
Add a README section for the new `tool-dialect-bench` binary, which manually A/B tests text tool-call dialects against a local Ollama model. The section explains the binary's purpose, usage, and relevant flags, complementing the existing documentation for other binaries in the crate. Auto-committed-on: macbook
Contributor
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (26)
Comment |
Tiny Sweeper reviewTiny Sweeper reviewed this change across 6 lane(s) and found 3 active actionable finding(s). Detailed lane evidence and any incomplete work are listed below. State: Changes requested Review snapshot
Completeness: Complete What changedThe review could not produce a supported behavioral summary; inspect the cited changed surface and lane details below. FeaturesNone identified with supported citations. TestsNo supported feature-to-test mapping was produced. Test execution is not inferred. Findings
Pending checks: Rust E2E (mock backend), Build Playwright E2E Artifact, E2E (Playwright / web lane), Desktop E2E (full suite, 3 OS) Before merge
How this fits togetherflowchart LR
n0["...ers_invalid_schema_tool_without_arguments<br/>changed"]:::changed
n1["...es_pformat_signature_for_text_dispatchers<br/>changed"]:::changed
n2["...stem_prompt_skips_memory_md_when_disabled<br/>changed<br/>1 finding"]:::flagged
n3["render_subagent_system_prompt_with_format<br/>changed"]:::changed
n4["ToolCallFormat<br/>changed"]:::changed
n5["vec"]:::impacted
n6["format"]:::impacted
n7["run_turn_via_tinyagents_inner"]:::impacted
n8["render_subagent_system_prompt"]:::impacted
n9["PromptContext"]:::impacted
n10["from_tools"]:::impacted
n0 -->|calls| n5
n0 -->|tests| n5
n0 -->|uses| n9
n0 -->|calls| n10
n0 -->|tests| n10
n1 -->|calls| n5
n1 -->|tests| n5
n1 -->|uses| n9
n1 -->|calls| n10
n1 -->|tests| n10
n2 -->|calls| n5
n2 -->|tests| n5
n2 -->|calls| n6
n2 -->|tests| n6
n2 -->|calls| n8
n2 -->|tests| n8
n3 -->|uses| n4
n7 -->|calls| n6
n8 -->|calls| n3
n8 -->|uses| n4
n9 -->|uses| n4
classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Agent review detailscritique
security
tests
commits
description
e2e
Evidence and run details
|
![tinysweeper[bot]](https://avatars.githubusercontent.com/in/4526585?s=60&v=4){kind=small}
There was a problem hiding this comment.
Requesting changes: 1 lane(s) blocking, worst finding is critical.
Fix or reply to the findings below and push. The next review clears this automatically once they are gone — you should not need to dismiss anything by hand.
$0.0751 · 1,432,483 in / 39,653 out · 188,735 cached (13%) · ladder/vectors, gpt-5.6-luna, deepseek/deepseek-v4-flash · 1,222 embedded
critique: $0.0356 · 644,162 in / 19,870 out · 50,504 cached (8%) · gpt-5.6-luna, deepseek/deepseek-v4-flash
security: $0.0355 · 668,435 in / 13,081 out · 34,189 cached (5%) · gpt-5.6-luna
tests: $0.0014 · 33,064 in / 2,688 out · 29,106 cached (88%) · deepseek/deepseek-v4-flash
description: $0.0008 · 24,675 in / 1,112 out · 20,710 cached (84%) · deepseek/deepseek-v4-flash
e2e: $0.0009 · 36,969 in / 961 out · 33,011 cached (89%) · deepseek/deepseek-v4-flash
| # Manual, network-touching A/B of the text tool-call dialects against a local | ||
| # Ollama model: prompt tokens and call accuracy per `agent.tool_dispatcher` | ||
| # value. Never run by CI; see the file's docs for the invocation. | ||
| [[bin]] |
There was a problem hiding this comment.
Add the benchmark source before declaring its binary target
src/bin/tool_dialect_bench.rs is not present in the repository, so Cargo reports a missing binary source when loading this manifest (cargo metadata, builds, and tests fail before the target can be skipped). Add the file or remove this [[bin]] declaration.
[RULE] missing-source-file ·
| u.output_tokens >= u64::from(args.max_output_tokens) | ||
| }) => | ||
| { | ||
| row.input_tokens = response.usage.map(|u| u.input_tokens); |
There was a problem hiding this comment.
Borrow usage metrics before reading them repeatedly
Option::map consumes response.usage, so the second assignment uses a moved value (and the earlier output-cap guard also consumes it with is_some_and before reading it again). Usage is not treated as a copyable value by the surrounding code, which uses as_ref() when inspecting response usage. Borrow it with as_ref() in every guard and mapping expression.
[RULE] compile-error ·
| rendered.contains("## Tool Use Protocol"), | ||
| "{format:?}:\n{rendered}" | ||
| ); | ||
| assert!(rendered.contains(example), "{format:?}:\n{rendered}"); |
There was a problem hiding this comment.
Assert the example for the tool that the test registers
The test passes a tool list containing only TestTool, while the expected examples are read_file(...) and read_file({...}). Unless the renderer unconditionally emits a generic read_file example—which is not established by the provided signature—the assertion is unrelated to the registered tool and can make this test fail. Use an example for test_tool, or register a read_file fixture whose rendered signature and protocol are being tested.
[RULE] test-fixture-consistency ·
tinysweeper
Bot
added
the
priority: p0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agent.tool_dispatcheraccepts two new opt-in values,"python"and"typescript": the tool catalogue is rendered as function signatures (def read_file(path: str, limit: int = None) -> str) and the model calls tools as code (read_file(path="src/main.rs", limit=20)).OPENHUMAN_TOOL_DISPATCHERoverrides the field for one launch.OpenHumanRunContext::tool_dialect→RunPolicy::tool_dialect). Before, the harness always ranAutoregardless of what the prompt was composed for, so text dialects kept native schemas on the wire and had no registry to recover positional calls with.xml:ToolsSectionrendered the P-Format signature list and the XML block embedded the full JSON schemas — 13.6 KB + 28.7 KB for the orchestrator's 31 tools.tool-dialect-bench(crates/openhuman-cli/src/bin/tool_dialect_bench.rs) A/Bs the dialects against a local Ollama model.vendor/tinyagents(→ Add Python / TypeScript code-style tool dispatchers tinyagents#184, which bumps tinytools → Add code-style tool-call dialect (Python / TypeScript) tinytools#16).Problem
Every text-dialect turn re-sends the tool catalogue in the system prompt.
xmlpays for full JSON schemas;pformatis compact but an invented syntax an 8B model has never seen, and it mis-parses. We want to test whether function signatures + code-style calls — the form a code-trained model already writes — are cheaper and more reliable on small local models.Solution
Measured with the bench on
qwen3:8b(Ollama, 10 tools, 20 tasks, temperature 0, output cap 1500, model-reported tokens; two independent runs agree within one task):Python cuts prompt tokens 26% vs
xmland 35% vspformat(whose ~1.2 KB rules block outweighs its slot savings on a 10-tool set), while binding arguments more accurately. Thepformatmisses are real binding errors (list arguments emitted as a pipe-joined string,duration_minutesin the wrong slot); the remainingxml/pythoninexact rows are the model adding an optional argument (unit="metric"). The singletypescriptmiss isqwen3spending the whole 1500-token budget in its thinking channel on one task.Real orchestrator prompt (
agent prompt-size --hermetic, 31 tools, Ollama model): system prompt 73.4 KB (xml, before the double-catalogue fix) → 45.8 KB (pformat) / 47.5 KB (python) / 47.4 KB (typescript).Design: the parser lives in
tinytools-agent(one grammar for both spellings, registry-gated, literals only, all-or-nothing per<tool_call>body, top-level ```python fences stay examples). OpenHuman only maps the config string (agent/tinyagents/config.rs, new `session_host/builder/dispatcher.rs`), renders the catalogue (`prompts/sections.rs`, `prompts/render_helpers/subagent.rs`, `subagent_host/tool_prep.rs`), and pins the dialect on the harness (`session_host/driver.rs` → `harness_assembly.rs`). `auto` never selects the new dialects.Draft: depends on tinyhumansai/tinytools#16 and tinyhumansai/tinyagents#184 landing first; the
vendor/tinyagentsgitlink must then move to the merge commit.Submission Checklist
N/A: behaviour-only change(no feature row added/removed)## Related— N/ACloses #NNN— noneImpact
autobehaviour is unchanged except (a) the harness now honours the session's dialect, and (b)xmlprompts no longer carry a second catalogue.xml/pformat/python/typescript) now strip schemas off the wire as intended, sointegrations_agent(forced off native for Fireworks grammar limits) no longer sends native tools alongside its prose catalogue.Related
subagent_host/tool_prep.rs::build_text_mode_tool_instructionsstill hand-writes a stale positional P-Format block; add a dedicated fence language if small models refuse<tool_call>tags.AI Authored PR Metadata (required for Codex/Linear PRs)
Linear Issue
Commit & Branch
code-tool-dialect77e9b763021c69bd8e8d979d910a32d8aab91c4aValidation Run
pnpm --filter openhuman-app format:check— N/A, no frontend changepnpm typecheck— N/A, no frontend changecargo test -p openhuman --lib -- prompts:: subagent_host::tool_prep tinyagents::config_tests factory_provider_role env_overlay tool_call_format_maps(all green)cargo fmt --all -- --check,cargo check --all-targets,pnpm rust:layout,pnpm docs:checkscripts/check-prompt-budget.sh: flagscode_executor/context_scout/skill_executor/skill_setup(+16 B tool schemas) andmcp_agent(+70 B prompt) — byte-identical onmain(inherited from thehint:slug rename afterscripts/prompt-budget.limitswas last written).mainadditionally flagsorchestrator(30 263 B > 30 213); this branch brings it back under budget.Validation Blocked
command:cargo test -p openhuman --lib -- agent_turn_loop_tests::xml_dispatcher_parses_and_loopserror:"hosted root invocation is unavailable because the session has no hosted authority"impact:pre-existing — fails identically onmain; unrelated to this changeBehavior Changes
xml.agent.tool_dispatcher/OPENHUMAN_TOOL_DISPATCHERis set; smaller prompts and fewer duplicated tool listings for prompt-guided local models.Parity Contract
auto/native/xml/pformatspellings unchanged;autonever picks a code dialect;integrations_agentstill forced off native.factory_provider_role_tests_tests.rs,config_tests.rs, tinyagentse2e_tool_dialects.rs.Duplicate / Superseded PR Handling
Co-authored-by: Medulla medulla@tinyhumans.ai