fix(core): normalize claude-code MCP tool names to bare endpoint - #178
Open
Rodriguespn wants to merge 3 commits into
Open
fix(core): normalize claude-code MCP tool names to bare endpoint#178Rodriguespn wants to merge 3 commits into
Rodriguespn wants to merge 3 commits into
Conversation
Claude Code names MCP tools `mcp__<server>__<tool>`, so the parser recorded `originalName` (the scorer-facing endpoint) as e.g. `mcp__supabase-mcp__query_logs`, while the Codex parser records the bare `query_logs`. A scorer doing `tc.endpoint === "query_logs"` therefore matches Codex but silently never matches Claude Code, so tool-selection evals false-fail on Claude Code for behavior that actually happened. Strip the `mcp__<server>__` prefix in the claude-code parser so the endpoint is agent-agnostic, consistent with Codex. The raw name is still passed to `normalizeToolName`, so `name` stays `tool_use`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
… parsers
Replaces the flat scorer-facing `endpoint` string with a structured
`tool: ToolCall` — `{ kind: 'mcp'; server; toolName } | { kind: 'other'; toolName }`
— produced by each agent parser from what its transcript encodes:
- claude-code: split `mcp__<server>__<tool>` structurally.
- codex: `mcp_tool_call` items carry the server (explicit `item.server`, else
the sole configured MCP server from ParseContext); bare `item.tool` name.
- opencode: `<server>_<tool>` is lossy, so recover the server by matching the
configured names' sanitized prefix (opencode's own `sanitize`) via a new
optional `ParseContext.mcpServerNames`, threaded from the engine.
- ai-sdk executor: attribute via a tool→server map built from the MCP handles.
`toolName` is the bare, agent-agnostic name (no server prefix); `originalName`
stays raw for tracing. This lets scorers disambiguate our MCP server's tool
from a same-named native/hosted tool or another server's tool, e.g.
`tc.tool.kind === 'mcp' && tc.tool.server === 'supabase-mcp' && tc.tool.toolName === 'query_logs'`.
Supersedes the earlier claude-code-only prefix strip. Updates consumers
(docs-results, resolve-database-001 scorer) and all parser tests; adds MCP
attribution coverage for codex (explicit + sole-server) and opencode (prefix).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ults tests docs-results.test.ts re-derived the mcp__<server>__<tool> split instead of reusing claude-code/parser's parseClaudeCodeToolCall, which already owns that format. Export and reuse it instead of duplicating the parsing logic.
mattrossman
reviewed
Aug 10, 2026
| // treat a missing status as unknown (not success). `item.tool` is the | ||
| // bare tool name; `item.server` names the MCP server when present. | ||
| const bare = | ||
| str(item.tool) ?? str(item.name) ?? str(item.server) ?? 'mcp_tool_call'; |
Collaborator
There was a problem hiding this comment.
I know it's not new in this PR, but this fallback chain looks strange. I wouldn't expect the tool name to fall back to server name, maybe better to keep it undefined or generic fallback text if it's truly something that can go missing (Codex source seems to indicate it's not optional)
Collaborator
|
Sanity checked running all 3 harnesses, 
|
mattrossman
reviewed
Aug 11, 2026
| str(item.tool) ?? str(item.name) ?? str(item.server) ?? 'mcp_tool_call'; | ||
| // Prefer the explicit server field; fall back to the sole configured MCP | ||
| // server when the (unpinned) shape omits it. | ||
| const server = str(item.server) ?? soleServer; |
Collaborator
There was a problem hiding this comment.
I think we can remove the concept of "soleServer" as a fallback entirely, seems like dead code given the version of Codex we're on. I'd assume it will always tell us which server an MCP tool call was associated with, otherwise I'd want it fail validation in some visible way so we can stay aware of parser behaviors.
mattrossman
approved these changes
Aug 11, 2026
mattrossman
left a comment
mattrossman
left a comment
Collaborator
There was a problem hiding this comment.
Approving to unblock, see notes about about Codex fallback cleanup opportunities
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The claude-code parser recorded MCP tool calls'
originalName(the scorer-facingendpoint) as Claude Code's rawmcp__<server>__<tool>(e.g.mcp__supabase-mcp__query_logs), while the Codex parser records the barequery_logs. So a scorer doingtc.endpoint === "query_logs"matches Codex but never matches claude-code — tool-selection evals false-fail on claude-code for behavior that actually happened.This strips the
mcp__<server>__prefix in the claude-code parser so the endpoint is agent-agnostic, matching Codex. The raw name is still passed tonormalizeToolName, sonamestaystool_use.Why it's in the parser
Each agent parser owns normalizing its own tool names; Codex already emits the bare MCP tool name, claude-code was the outlier.
Verification
investigate-logs-002-query-logs-tool-selectionwithclaude-code-sonnet-5against the mcp#333 preview build went FAIL (1/2) → PASS (2/2); Showing a false negative (agent was called the tool but the eval check ignored it)