feat(schema): add the ClickHouse text skip index type - #212
Closed
any-victor wants to merge 3 commits into
Closed
any-victor wants to merge 3 commits into
any-victor wants to merge 3 commits into
Conversation
…d parens system.data_skipping_indices.type holds only the index name, so introspection parsed every argument of set, bloom_filter, tokenbf_v1, and ngrambf_v1 as 0 and drift reported index_mismatch right after migrate. type_full carries the arguments. chkit renders INDEX name (expr), and ClickHouse keeps those parentheses in expr, so the comparison also drops one pair when it encloses the whole expression.
any-victor
added a commit
to victor-software-house/chkit
that referenced
this pull request
Sep 25, 2026
Downstream-Reason: Model the ClickHouse text index (GA since 26.2) as a skip index type so schemas can declare, pull, migrate, and drift-check it. Upstream-Status: submitted Drop-When: obsessiondb#212 (obsessiondb#212) is merged and the base is rebased past it.
any-victor
added a commit
to victor-software-house/chkit
that referenced
this pull request
Sep 25, 2026
Downstream-Reason: Model the ClickHouse text index (GA since 26.2) as a skip index type so schemas can declare, pull, migrate, and drift-check it. Upstream-Status: submitted Drop-When: obsessiondb#212 (obsessiondb#212) is merged and the base is rebased past it.
The TypeScript comparison already drops one pair of parentheses that encloses the whole index expression. chkit-py still compared the stored `(lower(x))` against `lower(x)`, so `chkit drift --live` reported index_mismatch for every skip index right after migrate.
ClickHouse 26.2 made the text index generally available, and chkit could not declare it: the skip index union had no `text` variant, so a schema with a full-text index could not be generated, pulled, or drift-checked. `type: 'text'` takes a required `tokenizer` and the optional preprocessor, postprocessor, phrase search, dictionary, and posting list parameters. chkit renders them in one fixed order. ClickHouse keeps them in the order the DDL was written, so introspection parses `type_full` by key and drift compares the re-rendered form. Pull writes the fields back, validation reports a missing tokenizer, and chkit-py carries the same support as `SkipIndexText`.
any-victor
force-pushed
the
feat/text-skip-index
branch
from
September 25, 2026 00:35
fc3eaec to
37d0e41
Compare
Member
|
thanks a lot for this. There were some missing pieaces and I needed to rebase. I merged your proposal in a new PR: #215 Thanks for the input and let me know if there are other missing pieces or general feedback. Would love to learn how you are using ChKit. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ClickHouse 26.2 made the
textindex generally available. chkit had notextvariant in the skip index union, so a schema could not declare, generate, pull, or drift-check a full-text index.type: 'text'takes a requiredtokenizer(SQL, for example'splitByNonAlpha'or'ngrams(3)') and the optionalpreprocessor,postprocessor,supportPhraseSearch,dictionaryBlockSize,dictionaryBlockFrontcodingCompression,postingListBlockSize, andpostingListCodec.TYPE text(tokenizer = ..., ...)with the parameters in one fixed order (packages/core/src/text-index.ts).type_fullby key, and drift compares the re-rendered form, so the order a table was created with does not show as drift.chkit pullwrites the fields back. Validation reportstext_index_missing_tokenizerfor an empty tokenizer.SkipIndexText, with the same render order and parser (chkit_python/src/chkit/core/text_index.py).Depends on #211
This branch is stacked on #211 (
fix/drift-index-arguments). The text drift check readstype_full, which #211 introduces. Until #211 merges, the diff here includes its commit; the text change is the top commit, 37d0e41.Test plan
drift.e2e.test.ts:lower(email)withngrams(3), and a column withsplitByString, a preprocessor,dictionary_block_size = 512, andposting_list_codec = 'bitpacking'. Drift reports nothing right after migrate.sql-validatione2e: atextcase in both the CREATE and the ALTER ADD INDEX lists, in TypeScript and chkit-py.bun run typecheck,bun run lint, andbun run buildclean.ruff checkandmypy srcclean.