[tzphh]

Purpose

Reduce repeated data-file reads in batched primary-key vector queries and support distributed result lookup through Ray Data.

Changes

• search_vectors(...).to_arrow() groups matches by partition, bucket, and data file. It reads the union of matched physical row positions once per file and distributes the rows back to each query. This applies to both local and Ray candidate search.
• search(...).to_ray() and search_vectors(...).to_ray() reuse snapshot-pinned IndexedSplits. Ray workers read the projected rows lazily when the Dataset is consumed. Batch search returns one Dataset per input vector, and each Dataset executes its lookup independently.

Tests

• Verified that three queries with overlapping matches open one data-file reader and return the same rows as independent lookups.
• Verified that Ray workers open data files when the Dataset is consumed. Covered single and batch queries, local and Ray candidate search, projections, filters, and empty result schemas.

[JingsongLi]

Reusing file reads for overlapping Top-K results and deferring Ray payload reads to workers both have end-to-end value. All 119 targeted tests passed on Python 3.11 with Ray 2.54.0. The 13 Dataset cases also passed with Ray 2.59.1.

The published guide still says primary-key vector queries cannot return Ray Datasets. Please update it to describe candidate execution, lazy snapshot-pinned row lookup on workers, and independent execution of each batch Dataset. Distributed Vector Search

[JingsongLi]

[P2] Preserve bounded multi-file read parallelism

Each to_arrow call receives one split, which bypasses the existing read.parallelism path. With read.parallelism=3, I exercised two disjoint queries over two compacted Parquet files each. Independent lookups opened the same four files on reader workers, while the shared path opened all four serially on the caller thread. Disjoint Top-K results save no opens, so remote-file latency can regress. Please keep the existing path when no file is reused or use bounded parallelism across the shared file groups.

[tzphh]

Thanks for catching this. In cee7adf, batches with no shared data files now use the existing per-query lookup path, preserving multi-file reads under read.parallelism. I added a regression test where two queries each hit two distinct files. Each lookup receives both splits in one call and returns the same rows as an independent lookup.

I also updated the PyPaimon guide to explain that execution controls where candidate search runs, Ray workers fetch rows lazily from the search snapshot, and each Dataset returned by a batch query runs its own lookup.