Conversation
### What problem does this PR solve? Issue Number: close apache#68309 Problem Summary: BE serializes a bitmap holding at most 32 values as `BitmapTypeCode::SET` (flag 5) whenever `enable_set_in_bitmap_value` is on, which has been the default since apache#35730. The FE reader in `fe-common`'s `BitmapValue.deserialize()` only knew the type codes 0 through 4, so any BE-produced SET bitmap reaching an FE read path hit the default branch and failed with `RuntimeException: unknown bitmap type 5`. Affected paths are the ones that route BE bytes through `fe-common`, for example the Hive UDF binary bitmap argument. Reproduction: with `enable_set_in_bitmap_value=true`, `SELECT bitmap_to_base64(bitmap_from_array([1, 3, 5]))` on BE returns `BQMBAAAAAAAAAAMAAAAAAAAABQAAAAAAAAA=` (type 5, count 3, then little-endian 1, 3, 5). Feeding those bytes to `BitmapValue.deserialize()` throws instead of yielding `{1, 3, 5}`. Fix: add a `SET` case that reads the count as an unsigned byte, rejects a count above `SET_TYPE_THRESHOLD` (32, matching BE's own reader and inline SET capacity), and reads each value as a little-endian uint64. Instead of introducing a distinct in-memory SET representation, the values are folded through `add()`, which produces exactly the same set semantics in the existing single/bitmap representation and leaves every downstream consumer of `BitmapValue` unchanged. Scope note: this fixes flag 5, the code BE emits on the default path. The FE reader still does not know BE's `SET_V2` (10), `BITMAP32_V2` (12) or `BITMAP64_V2` (13) codes, which require `bitmap_serialize_version` other than 1 on both sides; supporting those is deliberately out of scope here. ### Release note Fixed the FE `BitmapValue` reader so it can deserialize SET-format bitmaps (flag 5) produced by the BE, removing the `unknown bitmap type 5` failure on FE paths that read BE-serialized bitmaps. ### Check List (For Author) - Test: Unit Test - `BitmapValueTest` covers hard-coded cross-language fixtures exported from BE via `bitmap_to_base64` (rather than another Java writer-to-reader round trip, since the Java serializer never emits SET): a normal SET, the 32-element boundary, values of 0, 2^32, Long.MAX_VALUE and UINT64_MAX, an over-threshold count, truncated input, and duplicate values. - `mvn -pl fe-foundation,fe-common test -Dtest=BitmapValueTest` passes (16 tests). - Behavior changed: Yes (the FE reader now accepts flag 5 instead of throwing) - Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Author
|
run buildall |
Author
|
/review |
Contributor
TPC-H: Total hot run time: 27862 ms |
Contributor
TPC-DS: Total hot run time: 152649 ms |
Contributor
ClickBench: Total hot run time: 24 s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #68309
Problem Summary:
BE serializes a bitmap holding at most 32 values as
BitmapTypeCode::SET(flag 5) wheneverenable_set_in_bitmap_valueis on, which has been the default since #35730. The FE reader infe-common'sBitmapValue.deserialize()only knew the type codes 0 through 4, so any BE-produced SET bitmap reaching an FE read path hit the default branch and failed withRuntimeException: unknown bitmap type 5. Affected paths are the ones that route BE bytes throughfe-common, for example the Hive UDF binary bitmap argument.Reproduction: with
enable_set_in_bitmap_value=true,SELECT bitmap_to_base64(bitmap_from_array([1, 3, 5]))on BE returnsBQMBAAAAAAAAAAMAAAAAAAAABQAAAAAAAAA=(type 5, count 3, then little-endian 1, 3, 5). Feeding those bytes toBitmapValue.deserialize()throws instead of yielding{1, 3, 5}.Fix: add a
SETcase that reads the count as an unsigned byte, rejects a count aboveSET_TYPE_THRESHOLD(32, matching BE's own reader and inline SET capacity), and reads each value as a little-endian uint64. Instead of introducing a distinct in-memory SET representation, the values are folded throughadd(), which produces exactly the same set semantics in the existing single/bitmap representation and leaves every downstream consumer ofBitmapValueunchanged.Scope note: this fixes flag 5, the code BE emits on the default path. The FE reader still does not know BE's
SET_V2(10),BITMAP32_V2(12) orBITMAP64_V2(13) codes, which requirebitmap_serialize_versionother than 1 on both sides; supporting those is deliberately out of scope here.Release note
Fixed the FE
BitmapValuereader so it can deserialize SET-format bitmaps (flag 5) produced by the BE, removing theunknown bitmap type 5failure on FE paths that read BE-serialized bitmaps.Check List (For Author)
Test
Behavior changed:
RuntimeException.Does this need documentation?
Check List (For Reviewer who merge this PR)