Skip to content

[FLINK-40435][python] Add explode support to DataFrame API - #29196

Open
beetle0915 wants to merge 1 commit into
apache:masterfrom
beetle0915:FLINK-40435
Open

beetle0915 wants to merge 1 commit into
apache:masterfrom
beetle0915:FLINK-40435

Conversation

@beetle0915

@beetle0915 beetle0915 commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Add DataFrame.explode() as specified in FLIP-591 and FLINK-40435, exposing collection expansion through the DataFrame API.

Brief change log

  • Expand ARRAY, MAP and MULTISET columns using existing SQL UNNEST, including ROW-valued elements and duplicate occurrences.
  • Preserve empty/null collections by default with LEFT JOIN; use CROSS JOIN when ignore_empty_and_null=True.
  • Validate expressions and output names, preserve retained columns, and return a lazy DataFrame without modifying the input.
  • Add API documentation and 20 tests covering argument validation, schemas, aliases, computed expressions, identifier quoting, batch execution and streaming changelog multiplicities.

Verifying this change

Local validation uses Python 3.12.11, Java 17.0.14 and Maven 3.9.16. Both Python source and Java artifacts come from this PR's commit 447edf29025, based on master at bd0a65afb94. The distribution and Python test dependencies were freshly built in an isolated Maven repository. Loaded Table API/runtime JAR paths and the extracted Planner hash were checked against the new build.

  • ./mvnw -B -T2 -pl flink-dist,flink-python -am -DskipTests -Pskip-webui-build install with the isolated local repository: BUILD SUCCESS, 106 modules (16m57s). This builds artifacts and test classes; it does not execute the Java tests or build the Web UI.
  • python -m pytest -q --tb=short pyflink/dataframe/tests against the new distribution: 371 passed, 8 existing deprecation warnings (109.37s).
  • ./mvnw -B -pl flink-table/flink-table-planner -Dtest=UnnestTest,LogicalUnnestRuleTest,UnnestITCase test with the same local repository: 175 passed, no failures, errors or skips; covers logical rules, batch/stream plans and batch/stream execution.
  • python -m flake8 --config=tox.ini pyflink/dataframe: passed.
  • python -m mypy --config-file tox.ini: passed for 83 source files.
  • Sphinx HTML build with -W --keep-going: passed.
  • git diff origin/master --check: passed.

Full-repository Maven verification and community CI validation remain outstanding; this PR remains a draft.

The existing UNNEST behavior that drops null ROW elements is tracked separately in FLINK-40658. It reproduces through direct Table API/SQL without this change. This PR does not fix that behavior; the separate reproduction is in that ticket.

Does this pull request potentially affect one of the following parts:

  • Dependencies: no.
  • Public API: yes, adds an @PublicEvolving DataFrame method specified by FLIP-591.
  • Serializers: no.
  • Runtime per-record code paths: no changes; delegates to existing UNNEST execution.
  • Deployment or recovery: no.
  • S3 file system connector: no.

Documentation

  • New feature: yes.
  • Documented in the method docstring and DataFrame API reference autosummary.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: OpenAI Codex CLI 0.154.0-alpha.6.2

Generated-by: OpenAI Codex CLI 0.154.0-alpha.6.2
@flinkbot

flinkbot commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@beetle0915
beetle0915 marked this pull request as ready for review September 15, 2026 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants