fix: Tier-2 vector recall applies unordered LIMIT before cosine ranking, causing false negatives

[Shelliemayinforest]

Pre-submission checklist | 提交前检查

• I have searched existing issues and this hasn't been mentioned before
• I have read the project documentation and confirmed this issue doesn't already exist
• This issue is specific to MemOS and not a general software issue

Bug Description | 问题描述

Tier-2 vector recall applies an unordered SQL LIMIT before cosine ranking. As a result, the globally best vector can be excluded from similarity computation solely because it appears after the physical scan prefix.

This is not a claim that every retrieval must scan the full table. A bounded scan is a reasonable performance safeguard. The correctness problem is that the bounded candidate set is not selected by an ANN index, vector relevance, deterministic recency order, or another semantic pre-filter: scanAndTopK() executes a query of the following shape:

SELECT ... FROM traces WHERE ... LIMIT <hardCap>

There is no ORDER BY. runTier2() then passes hardCap: vecPoolSize * 4. Therefore records outside the arbitrary SQLite prefix have zero probability of vector recall, even if their cosine similarity is 1.0.

With the v2.0.14 defaults, tier2TopK=5 and candidatePoolFactor=4, so vecPoolSize=20 and the effective vector scan cap is only 80 qualifying trace rows per vector channel.

Expected behavior: a Tier-2 vector Top-K search should either select candidates using a relevance-aware strategy, or explicitly expose/document approximate bounded-search semantics with a deterministic, meaningful candidate policy. It should not silently treat an unordered SQL prefix as the vector search corpus.

Actual behavior: the best vector is deterministically missed when it is immediately outside the prefix. Increasing tier2TopK by one changes the cap and suddenly makes the same vector rank first, although neither the data nor query changed.

How to Reproduce | 如何重现

The reproduction directly imports the compiled implementation from the official @memtensor/memos-local-plugin@2.0.14 npm tarball.

1. Create an in-memory SQLite traces table with 400 qualifying rows and the same priority/timestamp.
2. Create an index on priority so the chosen SQL prefix is observable and reproducible.
3. Store vector [1, 0] at row 335. Use query vector [1, 0], so its cosine is exactly 1.0.
4. Store vectors with cosine 0.3 in every other row. This remains above the v2.0.14 minTraceSim=0.25 default.
5. Call the official scanAndTopK() with hardCap=320, then with hardCap=400.
6. Call the official runTier2() with tier2TopK=20, then with tier2TopK=21, keeping every other input/config value unchanged.

Observed output:

scanAndTopK(hardCap=320): trace-335 absent; best returned cosine ~= 0.3
scanAndTopK(hardCap=400): trace-335 is rank 1; cosine = 1.0

runTier2(tier2TopK=20): observed hardCap=320; trace-335 absent
runTier2(tier2TopK=21): observed hardCap=336; trace-335 present at rank 1

The corresponding query plan is:

SEARCH traces USING INDEX idx_traces_priority (priority>?)

This boundary test isolates the cause:

• The target vector is valid and is the unique global best match.
• It satisfies the same SQL filters and similarity threshold as the other rows.
• The target becomes rank 1 as soon as the scan boundary includes it.
• Therefore the miss happens before cosine Top-K ranking, not in embedding generation, thresholding, hydration, or later re-ranking.

The same mechanism was independently reproduced against the official v2.0.10 package as well.

Environment | 环境信息

• Package: @memtensor/memos-local-plugin@2.0.14
• Official npm tarball SHA-1: 32639d241918c7da8d536e52eac7e0a7c42c312e
• Tarball SHA-256: 575af9121f86ad3eacfc6aca9d8a3e3e0856b2caa91a7d537a3c37ad3ee43907
• npm integrity: sha512-yEAroCSBfdf7urP47Hyr2MzTg4BPLIWqlno5r0imHb69s8fh7uXZRuPK23IWCzDFIWuPK/SuZfk8u3MdGQOzLg==
• Node.js: v22.23.0
• Database driver: better-sqlite3
• Operating system: Linux, aarch64 container

Additional Context | 其他信息

This may have been introduced or made more visible by the safeguards added after reports of synchronous full-table vector scans causing high CPU/RSS. The performance motivation is valid, but the current cap trades away recall based on physical/query-plan order rather than an explicit approximation strategy.

Possible directions, without prescribing a particular fix:

• use an ANN/vector index;
• preselect candidates using a documented semantic or temporal policy before cosine ranking;
• make the scan bound configurable and observable, with explicit approximate-recall semantics;
• avoid an unordered LIMIT as the sole candidate generator.

Willingness to Implement | 实现意愿

• I would like someone else to implement this