[Feature] Optimize IN and NOT IN predicate evaluation with a dedicated literal lookup set

[lucasfang]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Evaluating an IN / NOT IN predicate on a batch currently goes through MultiLiteralsLeafFunction, which materializes the whole column into Literal objects (one heap allocation per row, and non-owning string literals still require building the Literal wrapper) and then linearly scans all literals for every row. The cost is O(rows × literals) plus O(rows) heap allocations per batch. With large IN lists (e.g. thousands of partition keys pushed down from SQL), predicate evaluation becomes a measurable part of scan time even though membership testing only needs set lookups.

Solution

Build an immutable, type-specialized lookup structure once when the predicate is constructed, and probe arrow arrays in O(rows) without any per-row allocation:

• Introduce LiteralSet in src/paimon/common/predicate/, created from the predicate literals at construction time and owned by LeafPredicateImpl. Unsupported types (FLOAT/DOUBLE with NaN semantics, TIMESTAMP with unit conversion, DECIMAL with cross-scale comparison) or heterogeneous literals make the construction return null, and every path falls back to the existing In / NotIn implementation, so observable behavior stays identical.
• Integer family (TINYINT/SMALLINT/INT/BIGINT/DATE, widened to int64): a dense bitmap when the value span is small and close to the literal count, otherwise a hash set, with a min/max range check rejecting out-of-range values before any lookup.
• BOOLEAN: two flags.
• STRING/BINARY: a hash set of string_views backed by owned storage, with length-range and first-byte bitmap filters applied before hashing; dictionary-encoded arrays are probed once per dictionary plus O(rows) index follows.
• NULL semantics match the existing functions exactly: null literals are ignored by IN, a null literal makes NOT IN false for every row, and null column values never match.
• The structure is immutable after construction and shared through std::shared_ptr<const LiteralSet> when predicates are rebound to other schemas, so concurrent evaluation stays lock-free and the structure is not rebuilt per reader.

Anything else?

No public API, storage format, or protocol change; everything is internal to src/paimon/common/predicate/. The statistics/min-max pruning path keeps using the existing LeafFunction implementation.

Are you willing to submit a PR?

• I'm willing to submit a PR!