Skip to content

[python] Add batch numeric extraction for Variant fields - #997

Merged
JingsongLi merged 3 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/variant-numeric-batch
Sep 30, 2026
Merged

JingsongLi merged 3 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/variant-numeric-batch

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Wide numeric Variant workloads currently materialize many Python dictionaries and scalar objects before building numeric arrays.

Changes

  • Extract ordered, literal top-level numeric fields into Arrow FixedSizeList<Float32>.
  • Reuse the Variant metadata and exact object layout within a batch while validating object offsets and selected child boundaries.
  • Preserve SQL null rows and represent missing, Variant-null, or non-object-root fields as child nulls.
  • Expose pypaimon_rust.data.variant_get_numeric_fields and release the GIL during extraction.
  • Return NotImplementedError for non-numeric selected values so an integration can explicitly fall back.

Scope and trade-offs

  • This reduces decoding and Python object materialization; it does not reduce Parquet or object-store I/O.
  • Supported values are Variant integer, float, double, and decimal. Conversion to float32 may lose precision.
  • The result follows the requested field order. Field names are literal, not Variant path expressions.
  • Automatic PyPaimon selection and application SDK integration are intentionally outside this PR.

Tests

  • cargo test -p paimon --lib variant (46 passed)
  • pytest -q bindings/python/tests/test_variant.py (1 passed)
  • cargo clippy -p paimon --lib -p pypaimon_rust -- -D warnings
  • cargo fmt --all --check

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review September 30, 2026 11:44
@JingsongLi
JingsongLi merged commit c9cf6c8 into apache:main Sep 30, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants