[escherize]

A green corpus test does not mean the lint matches Metabase. 23 of the 41 cases, from 10 causes, record a place where the lint and Metabase disagree today (known in the case file). Their tests pass while the lint stays wrong in exactly that way, so that a fix, or a new difference, shows up. Read the table under "What the first corpus says" as open bugs: grep -l "^known: [^n]" corpus/cases/*.yaml lists them.

Why

The lint has one job: give the answer Metabase gives. Today the live parity check (scripts/lint-parity.ts) compares the two on one folder, examples/v1, plus a fixed list of mutations. A review of this stack found 17 places where the lint and Metabase may disagree, and none of them show up in that one folder.

This PR adds a corpus: many small folders, each with what a real Metabase did with it, replayed against the lint in bun test.

How it works

• Case: corpus/base/ (the transforms part of examples/v1) plus a few edited files, and Metabase's recorded answers:
  • whether the import succeeded
  • the tables each transform reads, from a test probe with no inputs
  • whether Metabase accepts each transform test
• Recording (scripts/record-corpus.ts, the only writer of cases): it commits each case to a branch of the git sync repository, imports that branch, and probes as lint-parity.ts does. So the expected answers always come from Metabase, never from the lint. A server error becomes an answer only when Metabase gives it twice.
• Generation (scripts/corpus-gen.ts, fast-check): it rewrites transform SQL and edits test inputs and expectations, using the shapes the lint has to get right:
  • case, quoting and aliases
  • CTEs and CTEs that shadow a table
  • subqueries, UNION, several statements
  • comments and string literals
  • optional clauses with and without template tags, nested
    The recorder shrinks a disagreeing case to the fewest edits that still disagree. It keeps an agreeing case only when the case covers a shape, spelling or answer that no case kept so far covers.
• Replay (scripts/corpus.test.ts): it lints each case and compares the result with the recorded answers, using lint-parity.ts's comparisons plus three more: a failed import must be explained by a lint error on the entity Metabase names, every test must have an answer, and a run where every answer was skipped fails. A case the lint disagrees with stores the exact differences in known, so a fix or a new difference fails the test. scripts/refresh-known.ts updates known after a lint change, without touching the recorded answers.
• Drift: the Import workflow runs record-corpus.ts --check, which records every case again against the latest Metabase build and fails when an answer changed.
• lint-parity.ts's Metabase calls moved unchanged to scripts/metabase-api.ts, so both scripts share them.

What the first corpus says

25 cases, recorded on Metabase enterprise head (2026-11-02): 5 hand-written from review findings, and 20 kept from 150 generated with seed 7. In the first recording, 12 cases carried known differences, from 4 causes (the first commit message says six; that is wrong):

Case	Metabase	Lint
Nested optional clause; outer tag has a default, inner has none (src/sql-tables.ts:72)	reads the outer clause's table, refuses the test with missing-inputs	misses both. 3 cases: the hand-written one, and 2 the generator found on its own.
MBQL source table with a null schema, test input public.orders (src/input-match.ts:22)	accepts the test	missing-input and unused-input errors. 1 case.
[[...]] with no tag	refuses with unparseable-source	only a reads-not-checked warning, although its message already says Metabase cannot parse it. 7 cases, in transforms and in expectations.
Unterminated string literal	answers HTTP 500 (sqlglot call failed: TokenError)	a Metabase bug, kept as a pinned answer. 1 case.

The self-review of #40 at 6405d92 (report) ran no live Metabase. Six of its findings are corpus cases, and Metabase confirms all six:

Self-review	Metabase	Lint
2.1 Python source table table_id: null, as Metabase writes it	imports it	schema error
2.2 field-filter tag on orders.created_at, input orders	accepts the test	missing-input
2.2 field-filter tag on products.category, plus an input for products	refuses with unused-inputs	no error
2.3 files holding only [] or ''	imports the folder	schema error on each
3.1 FROM "Orders", input orders	refuses with unremapped-reference	no error
3.2 source.type: native	refuses the import	no error

A second generator run (seed 11) found two more causes on its own:

• The MBQL form of 3.1: an MBQL test whose input is PUBLIC.ORDERS. Metabase refuses it with unremapped-reference, and the lint passes it.
• Expectation SQL with {{tag}} or [[...]]. Metabase refuses it with unparseable-source, and the lint is silent.

Two suspected differences turned out to agree with Metabase: the tables read by multi-statement SQL, and by INSERT ... SELECT.

Validation

• type-check, eslint, format, bun test (824 pass) and build pass.
• Each new check fails when it should:
  • a case marked agreeing that disagrees fails
  • a known that doesn't match exactly fails
  • a failed import with only an unrelated lint error fails
  • a missing test answer fails
  • an observation where every answer is skipped fails
• --check against the same Metabase answered all cases the same.
• An independent review agent read the diff. I fixed what it found:
  • known cases passed on any difference
  • a missing test answer passed
  • the failed-import rule was too weak
  • the keeper counted shapes from failed imports, which compare nothing
  • transient errors could become answers
• After those fixes, a live run on Metabase enterprise head:
  • --check answered all 25 cases the same, including the HTTP 500, which came back twice.
  • --findings rewrote the 5 finding cases byte for byte.
  • A 12-case generation run shrank its disagreements and kept 7 cases.
  • No case branch was left behind in the sync repository.
• bun test prints the cases that hold a known difference after the run, and their test names start with KNOWN DIFFERENCE.

Not covered yet (follow-ups)

• Answers this recorder doesn't capture yet:
  • reading back a stored value: YAML merge keys, BOM
  • running a test: column checks, row keys
  • the transform API refusals that Compare transform API refusals with the lint's target rules #51 compares (the base doesn't edit targets yet)
• A Redshift warehouse, for case folding.
• Subgraph testing: supplying input CSVs and output queries for a few transforms, running the graph, and checking the result.
• More entities (questions, dashboards, collections, snippets), each with recorded answers.

Also seen while recording:

• The Import job of Compare lint results with Metabase in import workflow #47-Compare transform API refusals with the lint's target rules #51 is red at "Query every card".
• On today's Metabase head, examples/v1 fails to import: Metabase refuses the data-app action "Update order".