Skip to content

Compare transform API refusals with the lint's target rules - #51

Closed
galdre wants to merge 1 commit into
galdre/lint-9-columnsfrom
galdre/lint-parity-api-probes
Closed

galdre wants to merge 1 commit into
galdre/lint-9-columnsfrom
galdre/lint-parity-api-probes

Conversation

@galdre

@galdre galdre commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

This PR extends the CI comparison between the lint and Metabase to three rules that Metabase's transform API enforces and an import does not: missing-target-schema, missing-incremental-table-tag and missing-target-database. With the new flag --probe-transforms, scripts/lint-parity.ts asks Metabase to save transforms shaped by the mutations of these rules, and compares each answer with the lint's errors on that transform. The import workflow passes the flag.

What users of the package see

Nothing. The PR changes scripts/, .github/workflows/import.yml and src/test-support/lint-mutations.ts; the package ships none of them.

What to review

  • The probes. For each rule, the script applies the mutation of the same name from src/test-support/lint-mutations.ts to a copy of the folder, and finds each transform the mutation changes (changedTransforms, by entity id). It sends POST /api/transform twice for each such transform: once as imported, and once as mutated. Each answer must match the lint's errors on that transform, in the folder or in the mutated copy. A mutation that changes no transform fails, and so does a rule for which no answer was compared.
  • The request body. The API takes ids where a file names databases and fields. So the body starts from the transform that GET /api/transform/:id returns, whose query the import has already resolved. From the file it takes the target's type, schema and database, and the source's incremental strategy. A mutation that changes any other part of the transform fails the probe. The script resolves a database by GET /api/database, and a checkpoint field by GET /api/database/:id/metadata, once per database.
  • Request details.
    • Every probe writes to the table lint_parity_probe, which no transform of the folder writes. Metabase refuses, with 409, a target table the warehouse already holds.
    • A blank target schema is left out of the request, because the endpoint's own schema refuses it before the check the lint predicts.
    • An incremental target gets target-incremental-strategy: {type: append}, because the API requires one and a transform file need not give it.
  • Answers. A 400 is a refusal. Its plain-text message maps to the rule that predicts it (RULE_OF_TRANSFORM_REFUSAL in scripts/parity.ts), and a message no rule predicts fails. Metabase stops at its first refusal, in the order of TRANSFORM_RULE_ORDER, so the lint's errors on later rules are not compared. A 200 is an acceptance, and the script deletes the transform it created; a deletion that does not answer 204 fails. A 402 (feature not licensed) is skipped. Any other answer fails.
  • Locale. Metabase's transform API refuses with a message and no error code, and translates the message. The comparison matches the English messages, so the workflow relies on the site locale en set at setup.
  • mutateExamples takes an optional source folder; the mutations still edit files by their path in examples/v1, so --folder must be examples/v1 or a copy of it. The usage line says so.

How it was checked

  • scripts/lint-parity.test.ts runs the script against a local stand-in for Metabase. The stand-in answers POST /api/transform with the checks the probes reach, in Metabase's order: the target database exists, the checkpoint field exists, the warehouse holds no table of the target's name (409), the target names a schema on a database with schemas, and an incremental native query has a table template tag for the checkpoint field's table, or any table template tag when no checkpoint field is set. The warehouse holds the target of each example transform, so a probe that wrote to its transform's own target would get 409. The tests cover agreement (and that every transform the script creates is deleted), a refusal the lint does not predict, an acceptance the lint calls an error, a probe Metabase does not answer, a failed field lookup, a blank target schema, findings on another transform, a table tag for another table than the checkpoint field's and for the same table, a mutation that changes no transform, a mutation that changes a transform beyond what a probe carries, a folder the mutation cannot take, a skipped 402, and mutateExamples with a source folder.
  • scripts/parity.test.ts checks transformVerdictDiffers, the order of checks included, and changedTransforms, which tells a change the probe carries from one it does not.
  • On this branch, bun install --frozen-lockfile, bun run type-check, lint-eslint, lint-format, build and validate-schema pass, and so does CI=true bun test --timeout 2800. src/index.ts, the main entry, is unchanged from main. The built main entry has no lint; experimental.js has it. lint --experimental on examples/v1 prints 113 files, 0 errors, 0 warnings and exits 0, and lint without the flag exits 2.

Base: galdre/lint-9-columns. Last branch of the stack.

@galdre
galdre force-pushed the galdre/lint-9-columns branch from 0f91870 to c0f7cf1 Compare October 30, 2026 23:23
@galdre
galdre force-pushed the galdre/lint-parity-api-probes branch from ff1f245 to 6c34413 Compare October 30, 2026 23:23
@galdre
galdre force-pushed the galdre/lint-9-columns branch from c0f7cf1 to 6835335 Compare November 2, 2026 23:49
@galdre
galdre force-pushed the galdre/lint-parity-api-probes branch from 6c34413 to e85d5a8 Compare November 2, 2026 23:49
@galdre
galdre force-pushed the galdre/lint-9-columns branch from 6835335 to 4eb366c Compare November 3, 2026 17:27
@galdre
galdre force-pushed the galdre/lint-parity-api-probes branch from e85d5a8 to f723b2a Compare November 3, 2026 17:27
escherize added a commit that referenced this pull request Nov 3, 2026
galdre re-cut the lint stack, so this merges galdre/lint-parity-api-probes
at f723b2a rather than rebasing (no force push).

- Every file outside this PR's own takes the re-cut's version. A merge
  would otherwise bring back 59 example files the re-cut reverted.
- lint-parity.ts keeps the re-cut's changes (default-schema parsing, the
  missing-folder check, "compared nothing" for an empty folder) on top of
  the extraction into metabase-api.ts, which now uses the shared
  isStringArray from src/values.ts.
- import.yml keeps the pinned setup-bun SHA from #41 with the corpus
  step's API_KEY condition.
- refresh-known.ts recomputed each case's `known` against the re-cut lint.
  No difference grew: 8 cases now agree with Metabase
  (finding-44-nested-optional-clause, finding-46-mbql-null-schema,
  gen-11-131, gen-7-137, both self-review-2.2 cases, self-review-2.3,
  self-review-3.1), gen-11-107 and gen-7-18 each lose one difference, and
  the rest change only wording ("no error rule predicts"). 15 of 41 cases
  now hold a known difference, down from 23.
@galdre
galdre force-pushed the galdre/lint-9-columns branch from 4eb366c to c887c22 Compare November 3, 2026 19:15
@galdre
galdre force-pushed the galdre/lint-parity-api-probes branch from f723b2a to e74100d Compare November 3, 2026 19:15
@galdre
galdre force-pushed the galdre/lint-9-columns branch from c887c22 to 44e71f8 Compare November 4, 2026 05:06
@galdre
galdre force-pushed the galdre/lint-parity-api-probes branch from e74100d to 4632623 Compare November 4, 2026 05:06
@galdre
galdre force-pushed the galdre/lint-9-columns branch from 44e71f8 to 54f95fb Compare November 4, 2026 20:22
@galdre
galdre force-pushed the galdre/lint-parity-api-probes branch from 4632623 to 7dbe8ca Compare November 4, 2026 20:22
@galdre
galdre force-pushed the galdre/lint-9-columns branch from 54f95fb to 420e3cd Compare November 4, 2026 21:32
@galdre
galdre force-pushed the galdre/lint-parity-api-probes branch from 7dbe8ca to 625f394 Compare November 4, 2026 21:32
@galdre galdre closed this Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants