Repository navigation
perf(core): rank vector search rows in the database and count search bytes - #2074
Conversation
…bytes Built-in vector indexes (pgvector, sqlite-vec) now rank chunks, collapse them to their nearest search rows, and return at most `limit` rows in one query. A filtered window is read once: no widening loop, no manifest hydration statement. Excerpts count each returned row's chunks, then read only its best five chunk keys instead of every chunk key of the note. tests/statement_traffic.py records rows and bytes per statement. test-int/test_search_traffic.py runs searches through the HTTP route over 40 KB notes, asserts a byte budget and at most two nearest-neighbour queries per search, and has a benchmark that reports per-statement bytes. Refs #1746 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CMYVL4TcbWdXhdvAzxfqjo Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 31f709f61d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| not self.vector.external | ||
| or not query.has_filters | ||
| or window.admitted >= row_target |
There was a problem hiding this comment.
For filtered SQLite searches, returning rows does not mean the underlying chunk scan is exhausted: search_rows() collapses only the nearest min(row_limit * 10, 4096) chunks. With the default semantic_vector_k=100, five matching notes containing 300 chunks each, and chunks clustered by note similarity, a five-result request scans 1,000 chunks and returns only four notes. This unconditional exit then hides the fifth matching note, whereas the previous filtered loop widened to 1,600 chunks and returned all five. Large PostgreSQL admissions using HNSW have the same limitation. Preserve a way to fill row_target when the underlying chunk scan was truncated, rather than treating every short built-in row window as the complete answer.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 8fc72f1. search_rows now returns VectorRowSearch(matches, truncated). The collapse query counts its candidates, and truncated means the scan read every chunk it was allowed. While a truncated scan has fewer than row_target rows, the reader doubles chunk_scan up to the index's max_chunk_scan (4,096 for sqlite-vec, 50,000 for pgvector HNSW). An exact pgvector ranking is never truncated. Each round still returns at most row_limit rows. Regression tests: test_a_truncated_row_scan_widens_until_the_page_fills (your five notes of 300 chunks: scans 1,000, then 2,000 chunks, returns all five) and test_a_row_scan_stops_at_the_index_limit_or_when_it_saw_every_chunk; the real sqlite-vec test checks the flag.
A built-in row search collapses only the nearest `chunk_scan` chunks. When matching notes hold hundreds of chunks each, that scan can reach fewer rows than the page needs while more admitted rows sit past it, and the reader treated the short window as the whole answer. `search_rows` now returns `VectorRowSearch(matches, truncated)`: the scan counts its candidates, and `truncated` means it read as many chunks as it was allowed. While a truncated scan returns fewer than `row_target` rows, the reader doubles the chunk scan up to the index's `max_chunk_scan` (4,096 on sqlite-vec, 50,000 on pgvector HNSW). An exact ranking is never truncated. Each round still returns at most `row_limit` rows, not chunk keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CMYVL4TcbWdXhdvAzxfqjo Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
The per-search `search.vector_rows` log event is gone. Rounds, chunk scan, rows, row target and truncation are attributes on the one `search.vector_query` span every vector search already writes, since Logfire bills per record. The span is now opened explicitly so the row scan can set them; the external-index overfetch loop moved into `_run_external_query` unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CMYVL4TcbWdXhdvAzxfqjo Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 49f9f850b2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| while ( | ||
| search.truncated | ||
| and len(search.matches) < row_target | ||
| and chunk_scan < index.max_chunk_scan |
There was a problem hiding this comment.
For unfiltered hybrid searches with reranking and long multi-chunk notes, stopping at row_target can change the reranked prefix between pages. _prefix_rows() still uses _rerank_candidate_limit() as its position bound, but positions now count rows rather than chunks, so a deeper scan can add vector evidence inside that supposedly fixed prefix. With vector_k=8, two rerank candidates, and 40 clustered chunks per note, a two-result first page scans 80 chunks, while the next page scans 280. Reproducing this through the hybrid pipeline produced rerank pools [100, 1] and [3, 100], returning pages [100, 1] and [1, 2]: note 1 repeats and note 3 is skipped. Preserve fixed hybrid-prefix membership when switching from chunk windows to row windows.
Useful? React with 👍 / 👎.
`search.vector_window` and `search.materialize` were log records written on every search. Each is now a span around its unit of work with the outcome as attributes, so search writes no per-search log records. The window carries its read count so the span can report rounds. Drops the telemetry-shape test added with the row scan span. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CMYVL4TcbWdXhdvAzxfqjo Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Why
Basic Memory Cloud bills Neon transfer to customers as of 2026-11-07, and search is most of that transfer (#1746). Cloud already runs #1755 (late text fetch) and #2062, yet production still moves about 18 MB per search.
A new byte counter shows that most of what a vector or hybrid search still reads is chunk keys, not note text:
What
search_rows. The database ranks chunks, keeps each search row's bestchunks_per_row, and returns at mostlimitrows. The collapse SQL is shared by both engines (collapse_to_rows_sql).chunks_per_row=5over the returned rows.candidate_limitrows instead ofcandidate_limitchunks.Measuring it
tests/statement_traffic.pyrecords rows and bytes returned per statement.test-int/test_search_traffic.pyruns searches through the HTTP route over 40 KB notes (production's average is 41 KB). Its regular test asserts a byte budget and at most two nearest-neighbour queries per search.test_benchmark_search_traffic(-m benchmark) prints per-statement bytes.BASIC_MEMORY_SEARCH_TRAFFIC_NOTESsets its size.Bytes per search, page of 10,
main(7f7a219) → this branch. Each note is cut into about 129 chunks.Rows per vector search fell from about 3,200 to about 350–500. On Postgres I used 100 notes because seeding takes about 9 s per note; the 300-note Postgres baseline timed out after 190.
What a 300-note search still reads:
content(4,000 chars per row)The fixture cannot show the production-scale gain. There, the widening loop read up to 50,000 chunk rows per round across several rounds; it now reads one window of rows.
Latency. Indicative only: every timed run shared the machine with the test suite.
One plan change to watch: the HNSW path (more than 50,000 admitted chunks, or no filters) now reads up to 10 chunks per requested row, 1,000 at the default 100 rows, where main read 100 chunks per round. I could not exercise it at that size locally.
Tests
test_search_rows_collapses_chunks_to_their_nearest_rows(real sqlite-vec), pgvector unit tests for the exact and HNSW plans, a reader trace test for row matches, and the traffic test on both backends.test_a_built_in_index_reads_its_row_window_oncereplaces the long-notes widening test).just fast-checkclean.just fast-test: 6,881 passed.tests/repositoryplus the traffic test pass on SQLite; the traffic test passes on Postgres.Not in this PR
contentfor vector-matched rows (a response change).Refs #1746
🤖 Generated with Claude Code
https://claude.ai/code/session_01CMYVL4TcbWdXhdvAzxfqjo