Skip to content

perf(catalog): ERD neighbourhood and MCP lookups read only the tables they need (#1572) - #1662

Merged
ZhuchkaTriplesix merged 6 commits into
devfrom
issue/1572-catalog-table-filter
Oct 17, 2026
Merged

ZhuchkaTriplesix merged 6 commits into
devfrom
issue/1572-catalog-table-filter

Conversation

@ZhuchkaTriplesix

Copy link
Copy Markdown
Member

Closes #1572.

Before (measured)

  • ERD neighbourhood (ErdCatalog.loadNeighbourhood): for every schema in the neighbourhood it ran the column catalog query over all tables of that schema and kept the few it needed in Dart. On PostgreSQL that query has per-column correlated subqueries (unique index, enum labels, comments, defaults), so its cost follows the size of the schema. With catalogRowLimit = 200000, a large schema was cut before filtering, and the table the user opened could come back without columns.
  • MCP describe_table / sample_rows: to find one table, three catalog queries over the whole schema (columns, keys, view columns) on every call. sample_rows used them only to check the name.

SQLite (no PostgreSQL here; the app's own SQL run with Python's sqlite3), tables with 17 columns:

tables whole-schema columns (before) filtered to the 3 tables of a neighbourhood (after)
500 35 ms, 8,499 rows 0.3 ms, 51 rows
3,000 225 ms, 50,999 rows 0.7 ms, 51 rows

On PostgreSQL the gap should be wider (the subqueries run per column), and the rows also travel to the app and get decoded.

Change

  • columnsSql / viewColumnsSql(…, tables:) add AND <name> IN ('a', 'b') (c.relname / c.table_name / m.name), literal-escaped like the rest of the catalog SQL. An empty list matches nothing; null is the old whole-schema query.
  • loadNeighbourhood: keys first (unchanged; the walk needs them), then for each PostgreSQL schema only its tables of the neighbourhood. schema.table asks that schema for table, which also gives the fallback for a key-less table its bare name.
  • MCP: tableNamesSql (names of tables and views only) resolves the name with the same rule as before (exact, else the single case-insensitive match). describe_table then loads that table's columns and view columns; sample_rows needs only the name. list_tables and the schema:// resource still read the whole catalog, as they should.

Tests

  • erd_test.dart:
    • the neighbourhood query carries the IN filter;
    • a schema whose whole-schema query is truncated before the target still gives the target's columns (and is no longer marked truncated);
    • the filter is emitted for each dialect, quotes are escaped, an empty list matches nothing;
    • existing fakes recognise the query with or without the filter.
  • mcp_query_service_test.dart: describeTable sends only filtered column queries, and sampleRows('USERS') resolves users without any column query. The fakes now answer the names query.

Not run with flutter test locally; CI decides.

… they need (#1572)

loadNeighbourhood ran the column catalog query for every column of each
schema in the neighbourhood and filtered in Dart; describe_table and
sample_rows loaded the columns, keys and view columns of the whole schema
to find one table. The cost followed the schema, not the request, and a
schema over the 200,000-row limit was cut before filtering, so a table
could come back without its columns.

- columnsSql / viewColumnsSql take tables: an IN filter on the table
  name (relname, table_name, m.name), literal-escaped.
- loadNeighbourhood walks the keys first, then asks each schema only for
  its tables in the neighbourhood.
- MCP resolves the name with tableNamesSql (names only, exact match else
  the single case-insensitive one, as before), then describe_table loads
  that table's columns and view columns; sample_rows needs only the name.
  list_tables and the schema:// resource still read the whole catalog.

SQLite, 3,000 tables x 17 columns: the neighbourhood of one table read
51,000 column rows in 225 ms; filtered, 51 rows in 0.7 ms.
@github-actions github-actions Bot added performance Theme parser epic label: performance core Core library logic and services backend Backend database driver execution and queries P3 Low priority / Polish & Enhancements labels Oct 17, 2026
…ltered catalog (#1572)

describe_table loaded the columns of the asked table only, and
ErdSchema drops a relation to a table it has no columns for, so
orders.user_id lost its users.id reference. The keys are read first and
the tables they point to are loaded with it.

Tests: the MCP check now picks column queries by format_type (the key
query also joins pg_attribute), the ERD layout test's fake recognises the
filtered column query, and an unused import is gone.
@ZhuchkaTriplesix
ZhuchkaTriplesix merged commit 865f7f0 into dev Oct 17, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend Backend database driver execution and queries core Core library logic and services P3 Low priority / Polish & Enhancements performance Theme parser epic label: performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant