Skip to content

test(tool): pin every tool's parameters schema before migration#23244

Open
kitlangton wants to merge 1 commit intodevfrom
kit/tools-schema
Open

test(tool): pin every tool's parameters schema before migration#23244
kitlangton wants to merge 1 commit intodevfrom
kit/tools-schema

Conversation

@kitlangton
Copy link
Copy Markdown
Contributor

Summary

Pre-migration safety net for the upcoming tool-by-tool zod → Schema conversion. Every tool's parameters schema now has:

  1. A JSON Schema snapshot (z.toJSONSchema with io: "input") — captures exactly what the LLM sees at tool registration time, so any drift caused by a future migration fails the snapshot.
  2. Parse-accept / parse-reject assertions per tool pinning the user-visible contract (required fields, refinement bounds, enum membership, default values).

Refactor

To make the snapshots possible without standing up each tool's full Effect runtime, every tool file now exports its parameters schema as Parameters at module scope:

  • 9 tools already had a module-level const — just added export, and standardised the name to Parameters (uppercase) where it was previously parameters.
  • 9 tools had their schema inline inside Tool.define(...) — hoisted to module scope under the same Parameters name and wired back through.

The tool framework's Tool.define(...) call site is unchanged — it just references Parameters by name instead of an inline z.object({...}).

Coverage

18 JSON Schema snapshots and 43 explicit parse/reject assertions for:

apply_patch, bash, codesearch, edit, glob, grep, invalid, lsp, multiedit, plan, question, read, skill, task, todo, webfetch, websearch, write

Verification

  • bun typecheck clean
  • bun run test test/ — 2054 pass / 0 fail (full suite)
  • SDK byte-identical (types.gen.ts and openapi.json)

Next

With this in place, each subsequent PR can migrate one tool's Parameters from z.object({...}) to Schema.Struct({...}) (or similar), let the effect-zod walker emit the zod the tool framework still consumes, and rely on these tests to catch any JSON Schema or parse behaviour drift.

Pre-migration safety net for the upcoming tool-by-tool zod\u2192Schema
conversion. Every tool's parameters schema now has:

1. A JSON Schema snapshot (`z.toJSONSchema` with `io: "input"`) \u2014 this
   captures exactly what the LLM sees at tool registration time, so any
   drift caused by a future migration fails the snapshot.
2. Parse-accept/parse-reject assertions per tool pinning the
   user-visible behavioural contract (required fields, refinement
   bounds, enum membership, default values).

To make the snapshots possible without standing up each tool's full
Effect runtime, every tool file now exports its parameters schema as
`Parameters` at module scope:

- 9 tools already had a module-level const \u2014 just added `export`, and
  standardised the name to `Parameters` (uppercase) where it was
  previously `parameters`.
- 9 tools had their schema inline inside `Tool.define` \u2014 hoisted to
  module scope under the same `Parameters` name and wired back through.

Zero behaviour change: Tool.define still sees the same schema, runtime
validation path is identical, SDK (types.gen.ts + openapi.json) is
byte-identical, and the full 2054-test suite passes.

18 JSON Schema snapshots and 43 explicit parse/reject assertions for the
18 built-in tools (apply_patch, bash, codesearch, edit, glob, grep,
invalid, lsp, multiedit, plan, question, read, skill, task, todo,
webfetch, websearch, write).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant