[opt](lance) skip full-snapshot row-id prefilters and add profile counters

[Gabriel39]

What problem does this PR solve?

An unfiltered Lance vector search explicitly scoped to every fragment in its snapshot can still scan all indexed row IDs and construct a redundant allow set. This adds work to every query, particularly under concurrency, and the existing profile cannot isolate set construction from input scanning.

Integrate the narrowly scoped Lance v11 fix in lance-format/lance#9460 through the existing lance-c patch chain. Pin every Lance crate and the lockfile to commit f75f3343b5e125c42da1bd7acc8d8217cd5660a6; the commit is fetchable from the official repository. The upstream PR is pending review. Full-snapshot unfiltered queries omit the row-ID loader; fragment subsets, real predicates, deletion/overlay masks, indexed segment selection, and unindexed fallback retain their semantics. The explicit fragment list is preserved.

Expose seven profile metrics: LancePrefilterLoads, LancePrefilterInputRows, LancePrefilterInputBatches, LancePrefilterRowIds, LancePrefilterLoadTime, LancePrefilterInputTime, and LancePrefilterBuildTime. Document their scope and overlapping wall-time semantics, and extend indexed multi-vector reader tests to check filtered/non-filtered metrics after close. There is no C ABI or FE protocol change.

ANN result compatibility

Removing an all-row prefilter activates Lance's unfiltered distance calculation path. For 4-bit IVF_PQ, that path additionally quantizes the distance table and can change ANN candidates at a fixed refinement budget. The frozen dot fixture reproduces a Top-5 recall change from 4/5 to 3/5; this PR does not change the underlying PQ algorithm.

As a test-only workaround, the dot-score golden probes all partitions and uses refine_factor=256, covering all 1024 fixture rows before exact reranking. Update its golden to the exact dot results. Existing index-selection and single-partition ANN checks retain their original settings. This workaround does not establish unchanged recall for production queries.

Release note

Avoid redundant row-ID prefilter materialization for full-snapshot Lance vector searches and add detailed prefilter profile counters.

Validation

• Lance: 78 prefilter tests and 4 fragment-scope/segment-intersection/FTS tests passed, including execution-summary export, deleted rows, old snapshots, stable row IDs, pinned segments, and unindexed appends.
• Lance: workspace rustfmt and cargo clippy --profile ci --all --tests --benches -- -D warnings passed.
• Doris: syntax compilation of both changed C++ files passed with Lance enabled; clang-format 16 and shell syntax checks passed.
• Applied the full eight-patch chain to a fresh lance-c v0.1.9 archive with --fuzz=0; Rust 1.91 locked Linux dependency metadata resolved all 21 Lance crates to the pinned commit without updating the lockfile.
• Test workaround: Groovy compilation and 16 fixture checks against Lance 11.0.0 passed across filtered/unfiltered paths, including all five score goldens and the three single-partition discriminators. The full Doris regression suite was not run locally.
• The full Doris BE unit-test binary and production concurrency benchmark were not run locally. CI and a benchmark at fixed recall are still needed; no end-to-end latency reduction is claimed.

Check List (For Author)

• Unit Test: extend the existing Lance reader test and add upstream planner/loader tests.
• Behavior changed: redundant full-snapshot prefilter work is skipped; 4-bit PQ ANN candidates may change as described above.
• Documentation: docs/lance-prefilter-profile.md describes the counters and validation method.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: Codex completed, but no new pull request review was submitted for the current head SHA.
Workflow run: https://github.com/apache/doris/actions/runs/35701136444

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Sep 28th, 2026 2:03 AM.
Workflow run: https://github.com/apache/doris/actions/runs/35720095312

The selected account is excluded until 2026-09-28T02:03:00Z. Please trigger /review again; another configured account may be available.

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 100.00% (21/21) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.58% (31527/42274)
Line Coverage	58.81% (351488/597652)
Region Coverage	55.49% (293090/528150)
Branch Coverage	56.40% (132415/234761)