feat(predicate): optimize IN and NOT IN evaluation with a literal lookup set

[lucasfang]

Purpose

Linked issue: close #262

Evaluating IN / NOT IN predicates currently goes through MultiLiteralsLeafFunction, which materializes the whole column into Literal objects (one heap allocation per row) and linearly scans all literals for every row, i.e. O(rows × literals) per batch. This PR introduces LiteralSet, an immutable, type-specialized lookup structure built once when the predicate is constructed, so arrow arrays are probed in O(rows) with no per-row allocation:

• Integer family (TINYINT/SMALLINT/INT/BIGINT/DATE, widened to int64): a dense bitmap when the value span is small and close to the literal count, otherwise a hash set, with a min/max range check in front. The span computation guards against the full int64 range, where the unsigned span would wrap to zero and corrupt the dense bitmap.
• BOOLEAN: two flags.
• STRING/BINARY: a hash set of string_views backed by owned storage, filtered by length range and a first-byte bitmap before hashing. Dictionary-encoded arrays are probed once per dictionary and then only follow indices, with null dictionary entries probed as empty strings to keep exact parity with the fallback converter.
• NULL semantics match In / NotIn exactly: null literals are ignored by IN, a null literal makes NOT IN false for every row, and null column values never match.

Unsupported types (FLOAT/DOUBLE, TIMESTAMP, DECIMAL), heterogeneous literal types, and mismatched arrow array types all fall back to the existing LeafFunction implementation, so observable behavior including error reporting stays unchanged. The structure is immutable after construction and shared via std::shared_ptr<const LiteralSet> on NewLeafPredicate rebinds, keeping concurrent evaluation lock-free and avoiding rebuilds per reader.

Tests

New unit tests in src/paimon/common/predicate/literal_set_test.cpp:

• TestDenseIntegers, TestSparseIntegers, TestInt64FullSpan (regression for the full int64 range that used to crash when building the dense bitmap), TestTinyIntAndSmallInt, TestDate
• TestBoolean, TestString, TestStringWithoutEmptyLiteral, TestBinary
• TestDictionaryString, TestDictionaryWithNullValue (regression for null dictionary entries matching empty-string semantics of the fallback path)
• TestNullLiteralIgnoredForIn, TestOnlyNullLiterals, TestCreateOrNullUnsupported, TestMatchesArrowTypeMismatch, TestValueNull, TestSlicedArray, TestOutputBufferValidation

New end-to-end tests in src/paimon/common/predicate/predicate_test.cpp:

• TestLargeStringIn: 1000-literal STRING IN / NOT IN over a plain array including empty string and null
• TestInAfterRebind: verifies rebinding by field name/index shares the prebuilt lookup and keeps results identical
• TestInt64BoundaryIn: IN / NOT IN with INT64_MIN and INT64_MAX literals through PredicateBuilder, guarding the construction-time crash path

Verified with the full paimon-common-test suite (1515 tests, all passed).

API and Format

No. All changes are internal to src/paimon/common/predicate/; no public header under include/, storage format, or protocol is touched.

Documentation

No. This is a performance optimization with no user-visible behavior change.

Generative AI tooling

Generated-by: Qoder