Skip to content

feat: add LazyPartition trait for LazyMemoryExec, migrate generate_series to partitions#20369

Open
ethan-tyler wants to merge 4 commits intoapache:mainfrom
ethan-tyler:lazy-partition-migration
Open

feat: add LazyPartition trait for LazyMemoryExec, migrate generate_series to partitions#20369
ethan-tyler wants to merge 4 commits intoapache:mainfrom
ethan-tyler:lazy-partition-migration

Conversation

@ethan-tyler
Copy link
Contributor

Which issue does this PR close?

Rationale for this change

LazyMemoryExec currently uses generator closures (LazyBatchGenerator) as its partition interface. This replaces with a LazyPartition trait that gives partitions a proper abstraction while keeping the old generator path working through a legacy adapter.

generate_series/range are migrated to native LazyPartition as the first consumer. Proto roundtrip is updated to handle both native and legacy forms.

What changes are included in this PR?

  • LazyPartition trait as the native partition interface for LazyMemoryExec
  • LazyBatchGeneratorPartition adapter allowing existing generator to work
  • Deprecated compatibility APIs preserved: LazyMemoryExec::try_new, LazyMemoryExec::generators, GenerateSeriesTable::as_generator.
  • generate_series/range migrated to native GenerateSeriesPartition
  • Shared helper for generate series state construction (dedupes native and deprecated paths)
  • Proto roundtrip updated for native GenerateSeriesPartition with legacy adapter fallback
  • Core coop test helper updated to use LazyPartition
  • Proto regression test coverage added

Are these changes tested?

yes

  • cargo test -p datafusion-physical-plan lazy_memory_tests --lib
  • cargo test -p datafusion-functions-table generate_series_tests --lib
  • cargo test -p datafusion-proto roundtrip_generate_series --test proto_integration
  • cargo test -p datafusion execution::coop --test core_integration
  • cargo check -p datafusion-physical-plan -p datafusion-functions-table -p datafusion-proto -p datafusion

Are there any user-facing changes?

No. LazyBatchGenerator and existing public APIs remain available (deprecated)

…tions

Signed-off-by: Ethan Urbanski <ethanurbanski@gmail.com>
Signed-off-by: Ethan Urbanski <ethanurbanski@gmail.com>
Signed-off-by: Ethan Urbanski <ethanurbanski@gmail.com>
Signed-off-by: Ethan Urbanski <ethanurbanski@gmail.com>
@github-actions github-actions bot added core Core DataFusion crate proto Related to proto crate functions Changes to functions implementation physical-plan Changes to the physical-plan crate labels Feb 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate functions Changes to functions implementation physical-plan Changes to the physical-plan crate proto Related to proto crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Simplify LazyMemoryExec with SendableRecordBatchStream

1 participant