feat: pxf fdw support parallel scan#61
Conversation
.../pxf-service/src/main/java/org/apache/cloudberry/pxf/service/controller/ReadServiceImpl.java
Outdated
Show resolved
Hide resolved
|
@ostinru My approach to kernel parallel processing is still too simplistic. Maybe I will change or refactor later. |
|
All deployments are local.
When exploring parallelization, the good news is that parallelization does indeed improve efficiency. For small data volumes, the improvement is not obvious and may even be less efficient than non-parallel processing. Only when the data volume is large does it show a noticeable improvement. However, the current improvement still falls short of the expected level. Theoretically, the speedup factor should be almost equal to the number of workers. The reason it hasn’t reached the expected level may be due to bottlenecks in I/O or CPU. Further exploration will be conducted in the future. |
MisterRaindrop
force-pushed
the
liuxiaoyu/paralle_fdw_2
branch
from
a83213f to
9f68aec
Compare
- Added PxfBridgeImportStartVirtual function to manage imports with virtual segment IDs. - Updated PxfFdwScanState structure to include fields for gang-parallel execution. - Enhanced foreign scan functions to support gang-parallel mode, ensuring unique fragment distribution among workers. - Implemented initialization and cleanup routines for gang-parallel state management.
MisterRaindrop
force-pushed
the
liuxiaoyu/paralle_fdw_2
branch
from
9f68aec to
71f0bed
Compare
| EXPLAIN SELECT count(*) FROM pxf_parallel_enabled; | ||
| QUERY PLAN | ||
| ---------------------------------------------------------------------------------------------- | ||
| Gather Motion 1:1 (slice1; segments: 1) (cost=XXX rows=NNN width=NN) |
There was a problem hiding this comment.
The plan seems not correct?
3 participants
#58
Change logs
Currently, parallel FDW is supported. This implementation depends on the kernel's commit.
The current code is not yet ready for the review stage. This current commit is only an exploratory submission for FDW parallelization. More importantly, I need to ensure that the core part of the kernel is solid first.
apache/cloudberry#1571
Contributor's checklist
Here are some reminders before you submit your pull request: