fix(drift): read skip index arguments from type_full and ignore stored parens - #211
Merged
KeKs0r merged 2 commits intoSep 26, 2026
Merged
Conversation
any-victor
added a commit
to victor-software-house/chkit
that referenced
this pull request
Sep 25, 2026
Downstream-Reason: Drift and pull read index arguments from system.data_skipping_indices.type_full; type holds only the name, so every parameterised index showed as changed. Upstream-Status: submitted Drop-When: obsessiondb#211 (obsessiondb#211) is merged and the base is rebased past it.
6 tasks
any-victor
added a commit
to victor-software-house/chkit
that referenced
this pull request
Sep 25, 2026
Downstream-Reason: Drift and pull read index arguments from system.data_skipping_indices.type_full; type holds only the name, so every parameterised index showed as changed. Upstream-Status: submitted Drop-When: obsessiondb#211 (obsessiondb#211) is merged and the base is rebased past it.
Pracxion
reviewed
Sep 25, 2026
| function normalizeIndexShape(index: SkipIndexDefinition): string { | ||
| return [ | ||
| `expr=${normalizeSQLFragment(index.expression)}`, | ||
| `expr=${stripEnclosingParens(normalizeSQLFragment(index.expression))}`, |
There was a problem hiding this comment.
Couldn't we just take the normalizeClause Function?
Member
There was a problem hiding this comment.
Good idea, but the regex for that has cases that would lead to issues.
e.g. (a) || (b) -> a) || (b
…d parens system.data_skipping_indices.type holds only the index name, so introspection parsed every argument of set, bloom_filter, tokenbf_v1, and ngrambf_v1 as 0 and drift reported index_mismatch right after migrate. type_full carries the arguments. chkit renders INDEX name (expr), and ClickHouse keeps those parentheses in expr, so the comparison also drops one pair when it encloses the whole expression.
The TypeScript comparison already drops one pair of parentheses that encloses the whole index expression. chkit-py still compared the stored `(lower(x))` against `lower(x)`, so `chkit drift --live` reported index_mismatch for every skip index right after migrate.
KeKs0r
force-pushed
the
fix/drift-index-arguments
branch
from
September 26, 2026 19:28
427e508 to
c6625d9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
chkit driftreportsindex_mismatchfor a parameterised skip index right afterchkit migratecreated it. Two causes:system.data_skipping_indices.typeholds only the index name (ngrambf_v1). The arguments are intype_full(ngrambf_v1(3, 4096, 2, 0)).parseIndexTypetherefore parsed every argument as 0, soset(0),bloom_filter(0.01),tokenbf_v1(...), andngrambf_v1(...)never matched the schema. The TypeScript introspection, the ObsessionDB remote executor, and chkit-py now selecttype_full AS type.INDEX name (expr), and ClickHouse keeps those parentheses inexpr, so the live expression is(lower(email))againstlower(email)in the schema. The index comparison now drops one pair of parentheses only when it encloses the whole expression, so(a) || (b)stays as written. The TypeScript and chkit-py comparisons both apply it.Reproduction (ClickHouse 26.9.1.1629)
After
chkit migrate,exprreads(lower(s))because of the rendered parentheses.Relation to #201
#201 canonicalises expressions through
formatQuerySingleLineOrNull. That formatter keeps the wrapping pair:SELECT (lower(email))formats asSELECT (lower(email))on 26.9. So the parenthesis handling here is still needed alongside #201, andtype_fullis independent of it. If #201 lands first, the strip applies after canonicalisation.Test plan
drift.e2e.test.ts: a table withset(0),bloom_filter(0.01), andngrambf_v1(3, 4096, 2, 0)indexes reports no drift after migrate. It failed before this change withindex_mismatchon all three, and passes after it.drift.test.ts: stored enclosing parentheses read clean; a changed expression still reportsindex_mismatch.bun run typecheckandbun run lintclean;bun run buildgreen.chkite2e tests (three backfill runtime tests and the dictionary lifecycle), which time out at about 33 s here and fail identically on an unmodifiedmain.test_drift_compare.py(stored enclosing parentheses read clean;(a) || (b)still reportsindex_mismatch); 1,185 non-e2e tests pass.generate,migrate --apply, thendrift --livereports no drift for a table with anngrambf_v1index, atoIntervalDayTTL, and aTOmaterialized view. Before the chkit-py fix it reportedindex_mismatchon that index.