Skip to content

Add inverse and bulk assertions for tool list inspection #30

@othercodes

Description

@othercodes

Context

When using AssertableMCP.lists_tools() to inspect a tools/list response, the current public API supports:

  • contains_tool(name) / contains_tool(name, callback) — assert a tool exists, optionally with per-tool assertions via callback
  • does_not_contain_tool(name) — assert a tool is absent
  • with_count(n) — assert the list size
  • Inside the callback (AssertableToolDef): accepts(params), accepts_optional(params), documented(), has_output_schema()

Two gaps come up in practice:

  1. No way to assert that a tool does NOT advertise an optional property. accepts_optional([\"x\"]) asserts the property is present, but there is no inverse — useful when a server transforms its tool schema based on auth context (e.g., stripping internal-only parameters for restricted clients).

  2. No way to apply the same assertion to every tool in the list. contains_tool is per-name, so verifying a uniform invariant ("no tool exposes property X") requires enumerating every tool name in the test, which couples the test to the full catalog.

Proposed API

AssertableToolDef.does_not_accept_optional(params)

response.assert_mcp().lists_tools().contains_tool(
    "search",
    lambda t: t.does_not_accept_optional([\"internal_user_id\"]),
)

Raises AssertionError if any of the listed params are present in inputSchema.properties.

AssertableToolList.every_tool(callback)

response.assert_mcp().lists_tools().every_tool(
    lambda t: t.does_not_accept_optional([\"internal_user_id\"])
)

Applies the callback to every AssertableToolDef in the list. Useful for asserting catalog-wide invariants (uniform schema transformations, no leaked internals, etc.) without naming each tool.

Suggested implementation

class AssertableToolDef:
    def does_not_accept_optional(self, params: list[str]) -> Self:
        properties = list((self._definition.get(\"inputSchema\") or {}).get(\"properties\") or {})
        present = [p for p in params if p in properties]
        if present:
            raise AssertionError(
                f\"Tool '{self._name}' should not expose properties {present!r}; \"
                f\"properties={properties!r}\"
            )
        return self


class AssertableToolList:
    def every_tool(self, callback: Callable[[AssertableToolDef], Any]) -> Self:
        for tool in self._tools:
            callback(AssertableToolDef(tool))
        return self

Use case

Servers that conditionally rewrite their tool catalog per-caller (auth scopes, feature flags, public-vs-admin views) need to assert both the positive ("admin sees X") and negative ("public does not see X") sides of the transformation. The current API supports the positive side cleanly; the negative side requires either enumerating every tool by name or dropping to private attributes like _definition.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions