fix(tesserae): legacy SimHash backfill + open-time FTS self-heal + explicit embedding degradation - #22
Merged
Merged
Conversation
…mbedding degradation Two follow-ups to ticket 8 and the embedding fallback chain, found by live DB forensics: all 45 pre-ticket-8 tesserae had no drift fingerprint (only write-time stamping existed), and a configured-but-broken embedding provider degraded every search silently (construction failures never arm the cooldown, so searches reported a healthy lexical mode for months). Tesserae backfill (src/cli/service.ts, src/core/store/retrieval.ts): - One batched pass per project root, per process, on the first search that knows the root: stamps 16-hex SimHash of current file content onto rows still missing file_simhash (bounded 200/pass, oldest first). - updateTesseraSimhash is update-at-most-once (WHERE file_simhash IS NULL) — a fingerprint is the file as written, never a moving baseline. - Files unreadable now stay honestly unstamped; the pass never fails a search. Live DB: 34/46 stamped, 9 point at deleted files (memory-anchors rename history), 3 hit transient concurrent-writer windows and will stamp on the next search after the daemon restarts on new code. Embedding degradation visibility (src/integration/*): - #embeddingDegradedReason now reports a configured provider whose client failed to construct (e.g. missing API key) as a persistent degraded reason; previously such a daemon claimed degraded:false forever. - compactSearchContext keeps retrieval metadata when degraded, and the search surface renders an explicit [degraded retrieval: ...] line so the agent (and the user behind it) sees why vector ranking is unavailable. Tests: legacy-row backfill survives a file move; backfill skips missing files without inventing fingerprints; a missing-key gemini provider degrades explicitly (this is exactly the live-daemon shape). The compact-JSON bounded contract test now pins a clean environment — under the inherited NMG_EMBED_PROVIDER=gemini (no key) it correctly sees the new degraded field.
Live-DB forensics after the daemon restart: the remaining unstamped
tesserae were not contention. 12 of 45 legacy bookmark rows existed in the
content table without a matching tesserae_fts docsize entry (written by a
build whose schema predates the FTS sync triggers). Consequences: the rows
were invisible to tesserae search, and any UPDATE touching them failed with
SQLITE_CORRUPT ("database disk image is malformed") because the sync
trigger's FTS5 'delete' command found no docsize entry for the rowid —
deterministic, invisible to PRAGMA integrity_check, and it also broke the
first backfill pass mid-way.
- Open-time self-heal (ensureTesseraFtsSynced): two COUNTs at store open
detect content/index divergence and run FTS5 'rebuild', re-deriving the
index from the authoritative content table. Applied live: 34 -> 46
docsize entries, integrity-check ok, 9 bookmarks searchable again.
- Backfill pass convergence: per-row try/catch and the root is marked done
only when a pass has nothing stampable left, so a pass cut short retries
on a later search instead of being lost for the process lifetime.
- Live store converged to 43/46 stamped; the 3 unstamped rows point at a
deleted file (memory-anchors rename history) and stay honestly unstamped.
Tests: a legacy-shaped database (content table, no FTS/triggers) heals on
open, becomes searchable, and is then backfilled; temp-dir removal retries
briefly for Windows handle-release timing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
变更描述
What（改了什么）：
修复工单 8（#21）落地后的两个数据/可观测性缺口：① 全部 45 条存量 tessera 书签没有 drift 指纹（当时只有写时计算，无回填路径），且其中 12 条因 FTS 索引发散对书签搜索完全不可见、任何 UPDATE 触碰即 SQLITE_CORRUPT；② 配置了但构造失败的 embedding provider（如缺 API key）让所有搜索以
degraded: false静默降级为词法检索——本仓库活体 daemon 正是以此状态运行了约三周。Why（为什么改 / 解决什么问题）：
SimHash 漂移容忍只对"写时有指纹"的新书签生效，存量书签在文件编辑/移动后直接 stale；而 embedding 降级不可见意味着 agent 与用户都无法察觉向量检索已死（status 里有 reason，但搜索表面丢弃了它）。live DB 取证发现第三层问题：FTS external-content 索引与 content 表发散（content 46 行 vs docsize 34 条），
PRAGMA integrity_check看不到发散，但 sync 触发器的 FTS5'delete'命令读不到 docsize 条目即确定性抛 "database disk image is malformed"。Changes（关键改动点，按文件或模块列出）：
src/cli/service.ts—#backfillTesseraSimhashes：每个 project root 一次批量回填 pass（首个带 projectDir 的 search 触发，限 200 行/批，最旧优先），逐行 try/catch；root 仅在"本 pass 无可 stamp 项"时标记 done，被打断的 pass 由下一次 search 重试（收敛语义）src/core/store/schema.ts—ensureTesseraFtsSynced开库自愈：两个 COUNT 检测 content/FTS docsize 计数发散 → FTS5'rebuild'从权威 content 表重推索引（活体库已修复：docsize 34→46，integrity-check ok）src/core/store/retrieval.ts—tesseraeMissingSimhash()（回填扫描）+updateTesseraSimhash()（update-at-most-once：WHERE file_simhash IS NULL，指纹永不改写为移动基线）src/integration/search-projection.ts/agent-surface.ts— degraded 检索元数据穿透 compact 投影并在搜索表面渲染[degraded retrieval: mode=lexical; reason=…]行tests/core/tesserae-simhash.test.ts— 新增 3 测试：legacy 行回填后扛文件移动；缺文件诚实不 stamp；老格式库（content 表无 FTS/触发器）开库自愈→可搜索→回填tests/cli/service.test.ts— 新增：缺 key gemini provider 显式 degraded（正是活体 daemon 形态）tests/cli/process.test.ts— compact JSON 有界契约测试钉住干净环境（继承的NMG_EMBED_PROVIDER=gemini下现在正确看到新的 degraded 字段）验证：tsc 通过；CLI 套件 66/66、tesserae 7/7；全量
test:product1008/1009（唯一失败为预存在的 WindowsrmSyncEPERM 临时目录清理噪声，与本次改动无关，单测复跑通过）；活体 LTG 库收敛 43/46 stamped（余 3 条指向已删除文件，诚实保持 unstamped）。完成检查项
本地质量检查
npm run verify:static中 tsc/prompts:check 通过（完整 verify:static 由 CI 静态 job 覆盖）npm run test:product1008/1009（唯一失败为预存在 Windows rmSync EPERM 噪声，单测复跑通过；相关套件全绿）#backfillTesseraSimhashes≈6，自愈函数 ≈3）RCP（Repository Control Plane）
npm run agent:verify已跑并写入证据（相关路由全绿；两次全量运行中剩余失败均为并发噪声集漂移，串行 65/65 + 全量 1008/1009 复核）git status核对）CI 完成确认