Skip to content

fix(drift): read skip index arguments from type_full and ignore stored parens - #211

Merged
KeKs0r merged 2 commits into
obsessiondb:mainfrom
victor-software-house:fix/drift-index-arguments
Sep 26, 2026
Merged

KeKs0r merged 2 commits into
obsessiondb:mainfrom
victor-software-house:fix/drift-index-arguments

Conversation

@any-victor

@any-victor any-victor commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

chkit drift reports index_mismatch for a parameterised skip index right after chkit migrate created it. Two causes:

  1. Arguments are lost on introspection. system.data_skipping_indices.type holds only the index name (ngrambf_v1). The arguments are in type_full (ngrambf_v1(3, 4096, 2, 0)). parseIndexType therefore parsed every argument as 0, so set(0), bloom_filter(0.01), tokenbf_v1(...), and ngrambf_v1(...) never matched the schema. The TypeScript introspection, the ObsessionDB remote executor, and chkit-py now select type_full AS type.
  2. Stored parentheses. chkit renders INDEX name (expr), and ClickHouse keeps those parentheses in expr, so the live expression is (lower(email)) against lower(email) in the schema. The index comparison now drops one pair of parentheses only when it encloses the whole expression, so (a) || (b) stays as written. The TypeScript and chkit-py comparisons both apply it.

Reproduction (ClickHouse 26.9.1.1629)

CREATE TABLE default.t (s String, INDEX a lower(s) TYPE ngrambf_v1(3, 4096, 2, 0) GRANULARITY 1) ENGINE = MergeTree ORDER BY s;
SELECT name, type, type_full, expr FROM system.data_skipping_indices WHERE table = 't';
-- a  ngrambf_v1  ngrambf_v1(3, 4096, 2, 0)  lower(s)

After chkit migrate, expr reads (lower(s)) because of the rendered parentheses.

Relation to #201

#201 canonicalises expressions through formatQuerySingleLineOrNull. That formatter keeps the wrapping pair: SELECT (lower(email)) formats as SELECT (lower(email)) on 26.9. So the parenthesis handling here is still needed alongside #201, and type_full is independent of it. If #201 lands first, the strip applies after canonicalisation.

Test plan

  • New e2e case in drift.e2e.test.ts: a table with set(0), bloom_filter(0.01), and ngrambf_v1(3, 4096, 2, 0) indexes reports no drift after migrate. It failed before this change with index_mismatch on all three, and passes after it.
  • New unit case in drift.test.ts: stored enclosing parentheses read clean; a changed expression still reports index_mismatch.
  • bun run typecheck and bun run lint clean; bun run build green.
  • Full test run against a local ClickHouse 26.9.1.1629: all packages green except four chkit e2e tests (three backfill runtime tests and the dictionary lifecycle), which time out at about 33 s here and fail identically on an unmodified main.
  • chkit-py: two new cases in test_drift_compare.py (stored enclosing parentheses read clean; (a) || (b) still reports index_mismatch); 1,185 non-e2e tests pass.
  • chkit-py end to end against ClickHouse 26.9.1.1629: generate, migrate --apply, then drift --live reports no drift for a table with an ngrambf_v1 index, a toIntervalDay TTL, and a TO materialized view. Before the chkit-py fix it reported index_mismatch on that index.

any-victor added a commit to victor-software-house/chkit that referenced this pull request Sep 25, 2026
Downstream-Reason: Drift and pull read index arguments from system.data_skipping_indices.type_full; type holds only the name, so every parameterised index showed as changed.
Upstream-Status: submitted
Drop-When: obsessiondb#211 (obsessiondb#211) is merged and the base is rebased past it.
any-victor added a commit to victor-software-house/chkit that referenced this pull request Sep 25, 2026
Downstream-Reason: Drift and pull read index arguments from system.data_skipping_indices.type_full; type holds only the name, so every parameterised index showed as changed.
Upstream-Status: submitted
Drop-When: obsessiondb#211 (obsessiondb#211) is merged and the base is rebased past it.
function normalizeIndexShape(index: SkipIndexDefinition): string {
return [
`expr=${normalizeSQLFragment(index.expression)}`,
`expr=${stripEnclosingParens(normalizeSQLFragment(index.expression))}`,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't we just take the normalizeClause Function?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, but the regex for that has cases that would lead to issues.
e.g. (a) || (b) -> a) || (b

…d parens

system.data_skipping_indices.type holds only the index name, so introspection parsed every argument of set, bloom_filter, tokenbf_v1, and ngrambf_v1 as 0 and drift reported index_mismatch right after migrate. type_full carries the arguments.

chkit renders INDEX name (expr), and ClickHouse keeps those parentheses in expr, so the comparison also drops one pair when it encloses the whole expression.
The TypeScript comparison already drops one pair of parentheses that
encloses the whole index expression. chkit-py still compared the stored
`(lower(x))` against `lower(x)`, so `chkit drift --live` reported
index_mismatch for every skip index right after migrate.
@KeKs0r
KeKs0r force-pushed the fix/drift-index-arguments branch from 427e508 to c6625d9 Compare September 26, 2026 19:28
@KeKs0r
KeKs0r merged commit 75d15e9 into obsessiondb:main Sep 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants