Skip to content

feat(file-index): support additional data types - #271

Open
zjw1111 wants to merge 1 commit into
apache:mainfrom
zjw1111:codex/file-index-extra-types
Open

feat(file-index): support additional data types#271
zjw1111 wants to merge 1 commit into
apache:mainfrom
zjw1111:codex/file-index-extra-types

Conversation

@zjw1111

@zjw1111 zjw1111 commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Linked issue: #173

Extend file-index read/write support while keeping the serialized data compatible with Java Paimon:

  • Support FLOAT and DOUBLE in bitmap indexes.
  • Support DECIMAL in BSI indexes.
  • Support STRING, DECIMAL, and TIMESTAMP in range-bitmap indexes.
  • Add a range-bitmap type adapter for logical-to-storage type and literal conversion.
  • Add variable-length dictionary chunks for string keys and avoid eagerly reading chunk bodies.
  • Cover canonical and payload NaNs, signed zero, infinities, and Java-generated bitmap bytes.

Tests

  • pre-commit run --files <all changed files>
  • git diff --check upstream/main...HEAD
  • Compiled all 12 changed C++ translation units with the repository Debug compile commands (-Werror, ASan).
  • BitmapIndexTest.* (14 tests)
  • BitSliceIndexBitmapIndexReaderTest.*:ChunkedDictionaryTest.*:RangeBitmapFileIndexTest.*:RangeBitmapTypeAdapterTest.* (55 tests)
  • SizeLimitAndCardinality/* (70 tests)

API and Format

No public API under include/ is changed. The existing bitmap, BSI, and range-bitmap formats are extended to the additional logical types without changing their format versions. Floating-point bitmap keys are verified against Java-generated V1/V2 bytes.

Documentation

No documentation changes.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant