Implement native tool-calling protocol (#15) - #77
Conversation
portdeveloper
left a comment
There was a problem hiding this comment.
Thanks for taking this on. I ran the full suite at 10a9c7f and checked the flow against the native-tools example shipped in @qvac/sdk@0.14.1. The tests pass, but the live protocol still has blockers.
-
src/cli.mjs:371-384drops every tool result. It stores a placeholder assistant message, callshandleAction, then returns to the user. QVAC's own example appends each result as arole: "tool"message and runs a follow-up completion so the model can use the result. The current code cannot turn a balance, refusal, or transaction receipt into the assistant's answer, and the next user turn has no tool result in history. Please carry the executed result into history and continue the completion loop until the model stops calling tools, with a finite call limit. -
src/agent.mjs:86-96silently discardstoolCallErrorevents, while the catch path converts model errors into an empty successful result. Native mode is the default here, so a malformed call can print a blank line and scripted mode can still exit zero. Please surface these errors to the caller and preserve the existing scripted failure behavior. -
src/tools.mjs:364-441exposes only the five actions that existed on this branch's old base. Currentmainalso shipsaccount,get_nfts,transfer_nft, andswap. This branch already conflicts withmain; after the rebase, default native mode needs definitions and dispatch for the current action set so those features do not disappear. -
The new tests mock the object they expect and never drive
completeWithToolsorprocessLine. For example, the end-to-end test stops after asserting thatdispatchToolCallis a function. Please add a harness that feeds real completion events through the CLI boundary, observes the handler result entering a tool message, and verifies the follow-up turn. AtoolCallErrorregression belongs in that path too.
Please rebase before the next revision so the action coverage is checked against current main.
7d76d1b to
a5f00d3
Compare
portdeveloper
left a comment
There was a problem hiding this comment.
Thanks for the revision. One blocker remains in src/cli.mjs:533-540: native write calls go straight through dispatchToolCall, and the comment explicitly says handleAction is skipped. That bypasses recipient resolution, spend-policy checks, the transaction preview, mainnet acknowledgement, and the y/N confirmation for send_mon and send_token.
Please route native writes through the existing handleAction safety boundary and keep direct dispatch only for read-only tools. Add a CLI-boundary regression that feeds a native send tool call through processLine, proves no wallet method runs before confirmation, and covers a policy refusal. The current integration tests reproduce the loop in test code instead of driving the production boundary, so they would stay green if this bypass regressed.
PR portdeveloper#77 review blocker: native tool calls for writes (send_mon, send_token) were dispatched directly via dispatchToolCall, bypassing the resolveSend -> policy -> preview -> mainnet ack -> y/N confirmation flow that the v0 path and slash commands share. A compromised or malformed model could trigger a write without ever asking the operator. - Refactor handleAction in cli.mjs to return a printable result string (or null for "none") so the native loop can capture the tool result. - Extract the native tool-calling loop into src/nativeToolLoop.mjs with handleAction and dispatchToolCall as injected dependencies, so the routing is reachable from tests. - In the loop, route writes (isWrite) through handleAction and reads through dispatchToolCall - the read fast path stays snappy, writes share the same boundary as the rest of the agent. - Add test/native-tools-cli-boundary.test.mjs: drives the production loop with stubbed boundary seams, asserts writes never reach dispatchToolCall, and proves a policy/resolveSend refusal never bypasses the boundary. The previous integration tests re-implemented the loop in test code and would have stayed green if the bypass regressed; this one drives the real boundary. Co-Authored-By: Claude Code <noreply@anthropic.com>
PR portdeveloper#77 review blocker: native tool calls for writes (send_mon, send_token) were dispatched directly via dispatchToolCall, bypassing the resolveSend -> policy -> preview -> mainnet ack -> y/N confirmation flow that the v0 path and slash commands share. A compromised or malformed model could trigger a write without ever asking the operator. - Refactor handleAction in cli.mjs to return a printable result string (or null for "none") so the native loop can capture the tool result. - Extract the native tool-calling loop into src/nativeToolLoop.mjs with handleAction and dispatchToolCall as injected dependencies, so the routing is reachable from tests. - In the loop, route writes (isWrite) through handleAction and reads through dispatchToolCall - the read fast path stays snappy, writes share the same boundary as the rest of the agent. - Add test/native-tools-cli-boundary.test.mjs: drives the production loop with stubbed boundary seams, asserts writes never reach dispatchToolCall, and proves a policy/resolveSend refusal never bypasses the boundary. The previous integration tests re-implemented the loop in test code and would have stayed green if the bypass regressed; this one drives the real boundary.
7498e31 to
ddbef4a
Compare
Adds src/mcp.mjs: optional MCP server wiring for QVAC's completion({ mcp })
path, alongside (not replacing) the existing v0 JSON-action protocol.
- src/mcp.mjs: loadMcpConfig() reads an optional mcp.json (same
optional-file/NAD_*-override pattern as policy.json/address-book.json) —
a list of MCP servers to spawn over stdio. connectMcpServers() connects
each with the official @modelcontextprotocol/sdk, best-effort (a server
that fails to start is skipped and reported, not fatal — same rule as a
failed model load). summarizeMcpToolResult() turns an arbitrary MCP tool
result into a bounded string for history/terminal.
- src/agent.mjs: completeWithMcp() runs the full agentic loop QVAC's own
mcp-websearch example requires by hand — stream text, collect tool calls,
invoke them, push { role: "assistant" } then { role: "tool" } turns back
into history, and re-complete — until the model stops calling tools or a
finite maxToolRounds is hit. Takes an injectable runCompletion for tests.
- src/cli.mjs: when MCP servers are configured and connected, natural-language
turns route through completeWithMcp() instead of the v0 path. Every tool
call is gated behind the same confirm + mainnet-ack prompt every wallet
write already goes through — the tool catalog is discovered at runtime
from an arbitrary server, so there's no way to tell a read from a write
the way isWrite() does for the built-in actions, and asking every time is
the safe default. Tool errors and a hit round limit are surfaced, not
swallowed.
@tetherto/wdk-mcp-toolkit (the 35-tool wallet server named in #16 and the
README's "Upgrade path") is still a reserved placeholder on npm (0.0.0, no
code) as of this PR, so this wires any MCP server generically over stdio
instead — pointing mcp.json at the real package once it ships needs no code
changes here. #15 (native tool-calling via completion({ tools })) is being
worked on separately in #77; this does not touch that path or its files.
312 tests pass (17 new: mcp.test.mjs covers config validation, result
summarization, and connectMcpServers()'s best-effort failure handling
against the real MCP SDK; agent-mcp.test.mjs drives completeWithMcp()'s
loop end-to-end — follow-up turns, multi-call rounds, toolError surfacing,
round-limit enforcement, and stream-error tolerance — via an injected fake
completion() rather than asserting dispatch is a function). npm run build
succeeds; npm run doctor passes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
portdeveloper
left a comment
There was a problem hiding this comment.
The write path now reaches handleAction, which addresses the confirmation bypass. One failure-propagation bug remains in src/cli.mjs:515-532: hadFailureRef is a copy of the CLI flag, and its value is never copied back after runNativeToolLoop returns.
I drove the production loop with a toolCallError in scripted mode: it set hadFailureRef.value to true while the CLI flag stayed false, so the exit expression still yields 0. Dispatch exceptions and the tool-call cap take the same path. Please propagate that state back without clearing failures set inside handleAction, and add a regression through the actual processLine/CLI boundary that checks the nonzero exit. The current boundary tests stub handleAction and stop at the extracted loop, so they miss this wiring bug.
6dc6605 to
ddbef4a
Compare
PR portdeveloper#77 review blocker: native tool calls for writes (send_mon, send_token) were dispatched directly via dispatchToolCall, bypassing the resolveSend -> policy -> preview -> mainnet ack -> y/N confirmation flow that the v0 path and slash commands share. A compromised or malformed model could trigger a write without ever asking the operator. - Refactor handleAction in cli.mjs to return a printable result string (or null for "none") so the native loop can capture the tool result. - Extract the native tool-calling loop into src/nativeToolLoop.mjs with handleAction and dispatchToolCall as injected dependencies, so the routing is reachable from tests. - In the loop, route writes (isWrite) through handleAction and reads through dispatchToolCall - the read fast path stays snappy, writes share the same boundary as the rest of the agent. - Add test/native-tools-cli-boundary.test.mjs: drives the production loop with stubbed boundary seams, asserts writes never reach dispatchToolCall, and proves a policy/resolveSend refusal never bypasses the boundary. The previous integration tests re-implemented the loop in test code and would have stayed green if the bypass regressed; this one drives the real boundary.
72c2b56 to
f066002
Compare
f066002 to
60d8a08
Compare
|
Squashed the previous review iterations into a single commit while preserving the final implementation and fixes. Please review the current 60d8a08 head. |
portdeveloper
left a comment
There was a problem hiding this comment.
The failure flag is now copied back into the CLI without clearing failures from handleAction, which fixes the last wiring issue. Two blockers remain on 60d8a08.
src/tools.mjs:1127-1224still advertises and dispatches only five native tools. Current code also supportsaccount,get_nfts,transfer_nft, andswap, but the default native mode cannot call them. Please cover the current action set, keeping account switches behind the confirmation boundary. The test asserting exactly five definitions currently locks in this regression.src/nativeToolLoop.mjs:63-144silently succeeds when the outer turn limit is exhausted. I ran the production loop with one read tool call per completion: it executed all ten calls, returned withhadFailure.value === false, and printed no limit error. Since the inner guard checks greater than ten and the outer loop stops at ten, the usual one-call-per-turn loop never reaches the error path. Please report turn exhaustion and propagate a scripted failure. Cover that case through the actual CLI exit boundary; the existing tests stop at the extracted loop.
The current GitHub Actions run also needs maintainer authorization before its test check can run.
portdeveloper
left a comment
There was a problem hiding this comment.
The additional tools and explicit turn-limit error address the production changes from my last review. The revision still has blockers.
src/agent.mjs:165 listens for toolCallError while iterating run.events. The locked @qvac/sdk 0.14.1 emits toolError there, with the error in event.error; toolCallError belongs to its separate tool-call stream. I passed an SDK-schema-valid toolError through the exact completion function and production loop: it returned toolErrors: [] and left hadFailure false. Please handle the actual event shape and preserve its message through the scripted failure path.
Native mode also still initializes history with systemPrompt() from src/tools.mjs, which explicitly instructs the model to output one JSON action line. The native loop treats that text as a chat response and does not execute it. Please select a prompt appropriate to the enabled protocol and cover a native completion through the real CLI boundary.
test/native-tools-no-model.test.mjs:46 still asserts five tools, so the current suite fails with 9 !== 5. The tests labelled processLine/CLI boundary still invoke only the extracted loop with a stubbed handleAction. Please update the coverage for the current action set and add the previously requested regression through the actual CLI exit path for turn exhaustion and SDK tool errors. Rebase to include the Windows matrix and verify both OS jobs on the final revision.
0a8b168 to
99a5a10
Compare
portdeveloper
left a comment
There was a problem hiding this comment.
The SDK toolError handling and protocol-specific prompt are fixed, and all 428 tests pass locally. There is still a scripted failure regression in src/nativeToolLoop.mjs:136-149: the read path prints a returned Refused: string without setting hadFailure.value.
I ran the production loop with the real dispatchToolCall and a tool call {name: "get_nfts", arguments: {address: "not-an-address"}}, followed by a text-only completion. It printed Refused: "not-an-address" is not a valid address. and finished with hadFailure.value === false. The CLI consequently reports exit 0. The existing v0 path uses isRefusal(out) and marks this as a failure.
Please apply the same refusal handling to native read results and keep the failure set even if a later completion succeeds. Add a regression using the real dispatch path. Also exercise the actual scripted CLI exit: test/native-tools-cli-exit.test.mjs currently reconstructs the stack and uses its own exitCode() function, so it does not verify processLine wiring or the process exit. A subprocess with an injected model can cover the refusal and the SDK-error/turn-limit cases without a wallet or live model.
|
Implemented in this commit:
Note :Not verifiable on my hardware: live-model behavior (tool-call emission quality against the 9 schemas, prompt efficacy, multi-turn convergence). If you'd like, I can do a rented-GPU run (Qwen3-8B, dry-run, testnet) and report the transcript — or happy for you to cover that side in review. |
Replace v0 hand-rolled JSON protocol with QVAC's native tool-calling via completion({ tools }) on capable models. Structured tool calls are more robust than regex-parsed JSON and unambiguous about the model's intent.
Closes #15
Changes:
src/tools.mjs: Add getToolDefinitions() for OpenAI-compatible Tool schemas and dispatchToolCall() to route tool calls by name to handlers. All v0 actions (get_address, get_balance, get_token_balance, send_mon, send_token) now have native tool equivalents with proper parameter schemas.
src/agent.mjs: Add completeWithTools(history, tools, onToken) that calls completion({ tools, ... }) and iterates over run.events to collect contentDelta (text) and toolCall (structured calls) events.
src/cli.mjs: Refactor processLine() to dispatch based on config.useNativeTools. Native path calls completeWithTools(), dispatches each tool call through existing handleAction() validation/confirmation flow. Fallback v0 JSON path uses complete() + parseAction(). Both paths share the same confirmation and policy enforcement logic. Status block displays active protocol mode.
src/config.mjs: Add useNativeTools config option read from USE_NATIVE_TOOLS env var (default 'true'). Users can set USE_NATIVE_TOOLS=false to fallback to v0 JSON protocol for small/dev models that don't support native tool-calling.
test/tools.test.mjs: Add 5 new tests for native tool-calling:
test/native-tools-integration.mjs: Integration test verifying the dual-protocol dispatch path, config toggle, and tool schema validation.
test/native-tools-no-model.test.mjs: Comprehensive 28-test suite covering schema validation, dispatch routing, mock QVAC flow, config toggle, backward compatibility, end-to-end flow simulation, and protocol comparison. Tests work without requiring a local model by mocking QVAC completion events.
Verification:
This is the natural upgrade path before integrating @tetherto/wdk-mcp-toolkit MCP server (once it ships), which will be a small connector change at this point.
What this changes
How I tested it
npm run buildsucceedsnpm run smokeprintsSMOKE_OKModel / platform tested on:
Scope check
Conventions
npm(not pnpm/yarn) and did not add a globalsodium-nativeoverride..env, seeds, keys, or model weights.