feat(eval): query_logs tool selection on hosted-shaped platform - #100
Closed
jordienr wants to merge 3 commits into
Closed
feat(eval): query_logs tool selection on hosted-shaped platform#100jordienr wants to merge 3 commits into
jordienr wants to merge 3 commits into
Conversation
…hosted) Two MCP-mode evals asserting description-driven log tool selection: - hosted project -> query_logs - self-hosted project -> get_logs From supabase/mcp#333 review feedback (Rodriguespn). Blocked on query_logs shipping in a released @supabase/mcp-server-supabase version and bumping MCP_SERVER_VERSION in this repo; not runnable until then.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Rodriguespn
added
the
run-evals-changed
Rodriguespn
removed
the
run-evals-changed
6 tasks
The self-hosted scenario assumed the evals harness could simulate a platform without ClickHouse-backed query_logs support, but supabaseMcpServer() always spawns @supabase/mcp-server-supabase against api-platform.ts, which implements queryLogs unconditionally regardless of --api-url. There is no way to exercise the get_logs-only arm through this harness; that behavior is already covered by supabase/mcp's own unit tests (server.test.ts, debugging-tools.test.ts). Keep and rename the hosted scenario, since query_logs is now the only listed logs tool on the platform-lite fixture (get_logs is hidden), so this verifies query_logs works end-to-end rather than a two-tool choice.
jordienr
changed the title
Rodriguespn
changed the base branch from
main
to
fix/claude-code-mcp-tool-endpoint
August 7, 2026 14:13
jordienr
added a commit
to supabase/mcp
that referenced
this pull request
Aug 10, 2026
## What Adds a new `query_logs` tool to the `debugging` feature group. It runs a custom **ClickHouse SQL query** against a project's unified logs stream, for cases where the `get_logs` service presets are too coarse (filtering, aggregating, or joining across log fields). Tracks **AI-701**. Builds on the ClickHouse logs endpoint work in O11Y-1813 and the `get_logs` ClickHouse migration (#326). ## Breaking change This ships as `feat!:` per team alignment (see PR discussion): on platforms that implement ClickHouse-backed querying (hosted/production), `query_logs` is the tool discoverable via `tools/list`, and `get_logs` is hidden from discovery — though it remains callable via `tools/call` for any client still holding an older tool list. On platforms without ClickHouse support (CLI/self-hosted), the reverse holds: `get_logs` stays listed and `query_logs` is not registered at all. `DebuggingOperations.queryLogs` is optional on the `SupabasePlatform` type (published via the `./platform` subpath export). Implementers outside this repo (CLI, studio, the hosted controller) are unaffected if they don't implement it — they simply keep `get_logs` as their only logs tool. ## How it addresses the observability team's concerns 1. **Prompt injection / security** — log content is user-controllable (same class of risk as `execute_sql`), so the result is wrapped in `wrapWithUntrustedDataBoundary`, the same best-effort guardrail `execute_sql` uses. Not foolproof, but consistent with the existing arbitrary-query tool. 2. **Cost / scalability** — starts conservative: defaults to a 24h window, and both the client (`resolveLogWindow`) and the analytics endpoint cap the requested range at 24h. No unbounded queries. 3. **ClickHouse, not BigQuery** — hits `GET /v1/projects/{ref}/analytics/endpoints/logs` (ClickHouse / logs.all.otel) and takes raw ClickHouse-dialect SQL. Does not touch the deprecated BigQuery-backed `logs.all`. 4. **No POST on v1** — uses `GET` with `sql` as a query param. No new POST handler. 5. **CLI / self-hosted** — `query_logs` is only registered when the platform implements `queryLogs`; CLI/self-hosted platforms don't, so they keep `get_logs` as their only logs tool and never see `query_logs` at all (see Breaking change above). ## Details - `query_logs` params: `project_id`, `sql`, optional `iso_timestamp_start`/`iso_timestamp_end`. - Timestamps are validated as ISO 8601 with an explicit UTC `Z` suffix or offset (`z.iso.datetime({ offset: true })`), so the constraint is visible in the tool's JSON schema and offset-less (ambiguous local-time) timestamps are rejected up front. - `resolveLogWindow` (shared by `get_logs` and `query_logs`) anchors the default start to 24h before the resolved end, normalizes accepted timestamps to canonical UTC ISO strings, and rejects an inverted/equal window or one exceeding 24h — unit-tested directly in `debugging-tools.test.ts`. - Read-only enforcement of the SQL itself is left to the backend; no client-side SQL parsing. - Tool descriptions carry no environment-specific wording (hosted/self-hosted/CLI) — routing is handled structurally via registration/hiding, not prose the model has to reason about. ## Verification - `tsc --noEmit` clean, `biome ci` clean. - All unit + integration suites pass (incl. `logs.test.ts`, `server.test.ts`, `debugging-tools.test.ts`). The 5 failing `|e2e|` checks need live `SUPABASE_ACCESS_TOKEN` / `ANTHROPIC_API_KEY`, which this fork PR doesn't receive — confirmed passing on an identical-commit mirror pushed directly to the repo (#341). - Manually verified `query_logs` end-to-end against a real staging project (ClickHouse query executed, results returned, untrusted-data wrapping applied). - Eval coverage: supabase/evals#100 (draft, blocked on this shipping in a released `@supabase/mcp-server-supabase` version) verifies the agent successfully uses `query_logs` on a hosted-shaped fixture where it's the only logs tool listed. ## Post-merge follow-ups Per Rodriguespn — this adds a tool and changes `get_logs`'s discoverability, so per [CONTRIBUTING's breaking-change guidance](https://github.com/supabase/mcp/blob/main/CONTRIBUTING.md#breaking-changes): - [ ] Release a new `@supabase/mcp-server-supabase` version (automatic via release-please once merged as `feat!:`) - [ ] Update the tool list at [supabase.com/mcp](https://supabase.com/mcp#available-tools) - [ ] Bump `MCP_SERVER_VERSION` in `supabase/evals` and un-draft evals#100 - [ ] Open a PR on `platform` to add `query_logs` to the remote MCP server - [ ] Update the `@supabase/mcp-server-supabase` version pin inside `studio` - [ ] Resubmit the ChatGPT app to pick up the new tool and reworded `get_logs` description (frozen fields) > [!NOTE] > Per CONTRIBUTING, feature PRs should track an accepted issue — this tracks AI-701.
Contributor
|
Closing as supabase/mcp#333 was merged and these evals were just to test the change |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
One MCP-mode eval verifying the agent successfully uses
query_logsto investigate edge function logs on a hosted-shaped platform-lite fixture:investigate-logs-002-query-logs-tool-selection— assertsquery_logswas called (andget_logswas not), scored deterministically onctx.toolCalls[].endpoint.Why
Review feedback on supabase/mcp#333 (Rodriguespn): "curious if the agent will correctly call
query_logswhen working with the hosted version ... we should have an eval scenario." That PR now hidesget_logsfromtools/listwhenever the platform implementsqueryLogs(registration-driven, not description-driven), so this exercisesquery_logsend-to-end on the fixture where it's the only logs tool listed.Dropped: the self-hosted arm
The original version of this PR also had
investigate-logs-003-tool-selection-self-hosted, asserting the agent usesget_logs(notquery_logs) on a self-hosted-shaped platform. That scenario is untestable through this harness and has been removed:supabaseMcpServer()(packages/core/src/index.ts) always spawnsnpx @supabase/mcp-server-supabase@<version> --api-url <backend>, which always constructs its platform viacreateSupabaseApiPlatform()(packages/mcp-server-supabase/src/platform/api-platform.ts). That implementation definesqueryLogsunconditionally — it doesn't vary by--api-url, including when pointed at platform-lite. The actual self-hosted/CLI behavior (get_logs-only) comes from a different, externalSupabasePlatformimplementation (the CLI's own adapter) that this harness never spawns. There's no way to make the spawned server lackqueryLogsfrom evals today.That registration behavior is already covered by
supabase/mcp's own unit tests (server.test.ts: "debugging tools show get_logs when the platform does not implement query_logs";debugging-tools.test.ts). If we want a real evals-level self-hosted check in the future,@supabase/mcp-server-supabasewould need some way to force aqueryLogs-less platform for testing (e.g. a flag or a local-stack-shaped runtime), which is out of scope here.Still blocked — not runnable yet
query_logsonly exists in unmerged supabase/mcp#333.MCP_SERVER_VERSIONhere is pinned to a version without it, so this eval can't pass until:@supabase/mcp-server-supabaseversion shipsquery_logs, andMCP_SERVER_VERSIONis bumped to that version inpackages/core/src/index.ts.Validated with
pnpm eval:dry(loads, parses, plans asmode=tools runtime=platform-lite). Not executed — needs the version bump above.