Skip to content

feat(fts): allow external indexed input for hybrid queries - #9847

Draft
BubbleCal wants to merge 1 commit into
mainfrom
yang/partial-index-hybrid-input
Draft

BubbleCal wants to merge 1 commit into
mainfrom
yang/partial-index-hybrid-input

Conversation

@BubbleCal

Copy link
Copy Markdown
Contributor

Distributed planners cannot replace the indexed work inside HybridCompoundQueryExec, so an appended unindexed tail keeps the entire indexed search on the query node.

Add with_indexed_input to accept a bounded, single-partition result stream and its committed-corpus BM25 scorer. The hybrid node continues to tokenize and score residual rows, then merges both arms with the existing score/row-ID ordering and limit. With an external input, it opens only the representative segment needed for the residual tokenizer. Physical child replacement preserves the input and scorer; schema, partition count, result cardinality and null checks enforce the input contract.

Validation on AWS with the repository's Rust 1.98.1 toolchain:

  • 343 FTS tests, two scanner compound tests and the new API doctest pass. The seven new API tests cover native result parity, child replacement and invalid input contracts.
  • cargo fmt --all -- --check passes.
  • Standard full-workspace Clippy reproduces 16 errors in unchanged lance-io on both this change and its pinned baseline 0fd26effc79c62c244a5baf92cdda06e301baf21. The same full-workspace check passes with command-line allowances for those two existing lints: -A clippy::single_range_in_vec_init -A clippy::implicit_clone. No source or repository lint settings were suppressed.
  • The Sophon integration passes 45 focused regressions and 1,628 complete related tests, with one existing ignored test, plus scoped Clippy on Rust 1.97.1 in release mode.

This commit is based on Sophon's existing Lance pin; the affected source file is unchanged on current Lance main. It adds no storage or wire-format changes.

@github-actions github-actions Bot added the enhancement New feature or request label Oct 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant