Skip to content

Silent index degradation — status:"indexed" but only ~500 nodes for 72k LOC Rust codebase #333

Closed
Closed
Silent index degradation — status:"indexed" but only ~500 nodes for 72k LOC Rust codebase#333
Labels
bugSomething isn't workingparsing/qualityGraph extraction bugs, false positives, missing edgespriority/highNeeds near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker.
Milestone
0.10.0-rc

Description

@nguiaSoren
nguiaSoren
opened on May 11, 2026

Title: Silent index degradation — status:"indexed" but only ~500 nodes for 72k LOC Rust codebase

Version: codebase-memory-mcp 0.6.0
Platform: macOS (Apple Silicon)
Project size: 72k LOC, 9 Rust files (single Cargo project)

Symptom

index_repository(mode="full") returns status:"indexed" but produces only ~500 nodes and ~500 edges for a 72k LOC Rust codebase. Expected ~5,000–10,000+ symbol nodes based on comparable smaller projects.

The status:"ready" / status:"indexed" response is misleading — the rebuild silently fails on something but reports success.

Reproduction

  1. Kill any existing codebase-memory-mcp processes:
    ps aux | grep codebase-memory then kill <PID>
  2. Delete cached index:
    rm ~/.local/share/codebase-memory-mcp/Users-soren-Desktop-new_simulator.db
  3. Verify directory is empty of project cache
  4. Restart Claude Code (spawns fresh MCP server process)
  5. In Claude Code: index_repository(mode="full")
  6. Run index_status — shows ~500 nodes / ~500 edges, status:"indexed"

Multiple fresh rebuilds produce the same low node count (469/462, 540/531, 572/555 across attempts).

Comparison with healthy project

Same machine, same MCP server binary:

  • X project (~6k LOC): 6,436 nodes / 10,308 edges, 14 MB .db file ✓ healthy
  • Y project (72k LOC): ~500 nodes / ~500 edges, 2 MB .db file ✗ degraded

The 2 MB file size is suspiciously exactly 2 × 1024 × 1024 bytes — suggests the database is being truncated or capped at a fixed boundary.

Known-symbol verification

search_graph(name_pattern="apply_knight_hero_effects") returns 0 results.

This function exists at src/hero_system.rs (verified via grep). Should be indexed as a function symbol but is missing from the graph.

Workaround

Direct file reads + grep work fine. The bug appears specific to the graph-based query path (search_graph, trace_path).

Diagnostic information requested

Happy to provide:

  • Sample file from the codebase that may be triggering the issue (if I can isolate which file)
  • Full index_repository output
  • Any debug logging the indexer emits

Activity

  1. DeusData
    added
    bugSomething isn't working
    parsing/qualityGraph extraction bugs, false positives, missing edges
    on May 30, 2026

  2. DeusData commented on May 31, 2026

    @DeusData
    DeusData
    on May 31, 2026
    Owner

    This looks like the same durability issue as #334 (note the near-identical counts — 469/540/572 — and 9-Rust-file setup). The varying low node count across fresh rebuilds points to non-deterministic write loss (uncheckpointed WAL under synchronous=OFF, often from a hard-killed sibling process), not a deterministic parser cap — I checked, there's no per-file size/node cap that would deterministically clip a large file.

    v0.7.0 (#277) checkpoints the WAL on close and recovers a stale WAL on open, which should resolve the stranded-write loss. Could you retest on v0.7.0? If a clean full index still yields ~500 nodes for the 72k-LOC project, please share ls -l of the cache dir right after (so I can see if data is stranded in .db-wal) and, if possible, the repo or a representative large Rust file — then I can add a deterministic regression test. I'm also proposing an extracted-vs-persisted plausibility check over in #334 to make this fail loudly instead of silently; your input there welcome. Leaving open pending retest. 🙏

  3. DeusData commented on May 31, 2026

    @DeusData
    DeusData
    on May 31, 2026
    Owner

    Thanks for the data point. Important context here: Rust is currently a tree-sitter-only language in codebase-memory-mcp — it is not one of the hybrid-LSP languages (Python, TypeScript/JavaScript, PHP, C#, Go, C/C++). That means Rust gets structural definition extraction from the vendored tree-sitter grammar, but not the deeper LSP-backed symbol resolution and cross-file call-graph edges the hybrid languages get. So a lower node/edge count on a large Rust codebase is expected to a degree — though ~500 nodes for 72k LOC does sound lower than tree-sitter alone should yield, which points at gaps in the Rust extraction query (e.g. impl methods, trait methods, associated functions, macro-defined items).

    This is a parsing/extraction completeness gap rather than the index-corruption class the rest of this umbrella tracks — the index isn't corrupt, Rust extraction is just shallow. To target it I'd like to know, on v0.7.0: roughly how many nodes you get now, and which symbol kinds are missing (free functions? impl methods? structs/enums? trait defs?). A pointer to the (public) Rust repo you tested would let me reproduce and tighten the tree-sitter query directly. Reclassifying this as a Rust-extraction-depth issue and keeping it open on that basis. Thanks!

  4. DeusData commented on Jun 1, 2026

    @DeusData
    DeusData
    on Jun 1, 2026
    Owner

    I dug into this empirically — built v0.7.0 and indexed a large pure-Rust OSS repo (ripgrep, 100 .rs files / 52k LOC) to test whether Rust extraction itself under-produces:

    nodes=4581  edges=13509  errors=0   →  ~88 nodes/kLOC

    Label breakdown was healthy: 2,039 Method, 697 Function, 356 Class (structs), 70 Enum, 608 Field, 7 Interface (traits). The single largest file (7,779 LOC) alone produced 985 nodes (~1 node / 8 LOC). So the Rust tree-sitter extraction is not generically broken — your ~500 nodes for 72k LOC is ~7 nodes/kLOC, 12× lower than a normal Rust repo, which points at something specific to your codebase rather than the parser.

    Two likely causes, and one is easy to confirm:

    1. File-discovery exclusion — if a .gitignore/.cbmignore pattern (or the workspace layout) is excluding most of your .rs files, discover only sees a few. Check your index log for pipeline.discover files=N — if N is small (tens, not hundreds), that's it.
    2. Macro-heavy code — tree-sitter cannot see definitions that live inside macro invocations (it doesn't expand macros). I confirmed this on ripgrep itself: its macro-based test files (rgtest!(name, …)) extracted 0 functions despite 1,700+ LOC. If your codebase leans heavily on framework/proc macros (actix-web, diesel, rocket, sqlx, impl_trait!-style DSLs, generated code), large swaths become invisible and the node count collapses.

    Could you paste these two lines from your index run? They pinpoint it immediately:

    • pipeline.discover ... files=…
    • parallel.extract ... done nodes=… errors=…

    If files= is correct but nodes= is ~500, it's the macro blind spot (a known tree-sitter limitation we can discuss heuristics for). If files= is tiny, it's a discovery/ignore issue we can fix directly. Thanks — this'll let us nail it without your private repo.

  5. DeusData commented on Jun 1, 2026

    @DeusData
    DeusData
    on Jun 1, 2026
    Owner

    If its 2.: I already have the Rust LSP Hybrid in the works. Could potentially fix this, but we will need to double check, not sure though when I can ship this. Could take one or two weeks

  6. DeusData commented on Jun 12, 2026

    @DeusData
    DeusData
    on Jun 12, 2026
    Owner

    Update: the Rust LSP hybrid mentioned above shipped in v0.8.0 (current is v0.8.1) — Rust extraction now runs through the same hybrid-LSP path as TypeScript/Python/Go instead of tree-sitter-only.

    Could you reindex the 72k-LOC Rust repo on v0.8.1 and post the resulting node/edge counts? If it still lands around ~500 nodes, the index log (pipeline.done and gbuf.dump nodes=… edges=… lines) would pin down where extraction stops. Thanks!

  7. nguiaSoren commented on Jun 13, 2026

    @nguiaSoren
    nguiaSoren
    on Jun 13, 2026
    ContributorAuthor

    Retested on v0.8.1 (was still on 0.6.0 — sorry for the delay). Clean full index of the 72k-LOC Rust repo:

    pipeline.discover files=948
gbuf.dump nodes=29594 edges=46487
pipeline.done ... status=indexed

    So it's not reproducible — 29,594 nodes, and the symbol I reported missing (apply_knight_hero_effects) now resolves with full metadata.

    Most useful data point: I think your first comment had it right — this was the WAL-durability class (#334), not extraction depth. Two reasons:

    1. Before I even upgraded, the same v0.6.0 binary had already self-recovered to ~29.6k nodes (from the original ~500) with no code or version change. A deterministic extraction gap can't self-heal like that — but stranded-WAL write loss can, which matches the varying 469/540/572 counts you flagged.
    2. I found 12 orphaned codebase-memory-mcp processes still running on macOS from rapid session churn — the exact killed-sibling / uncheckpointed-WAL scenario. Clearing them + a clean rebuild fixed it.

    On the extraction-depth hypotheses from comments 2–3, for the record on this codebase: files=948 rules out discovery exclusion, and ground-truth grep -c 'fn ' vs the graph shows every production .rs file fully captured (graph symbols ≥ raw fn count in all of them — e.g. combat.rs 205 graph / 71 fns over 19.7k LOC; it's just a monolith of very large functions). It's also a free-function-heavy codebase (almost no impl blocks), so the impl/trait-method concern doesn't apply here.

    Net: #387's WAL checkpoint-on-close looks like the correct fix. v0.8.1 node count is flat vs v0.6.0 (29,598 → 29,594) but edges jumped 36,233 → 46,487 (+28%) from the LSP hybrid's cross-file edges — nice bonus. Happy to keep an eye out for any recurrence under session churn now that the orphan-process pileup is cleared.

  8. DeusData
    added a commit that references this issue on Jun 25, 2026

    test(repro): Rust trait-method extraction depth for #333

    78d1aff

  9. DeusData commented on Jun 28, 2026

    @DeusData
    DeusData
    on Jun 28, 2026
    Owner

    This issue now has a first-class, durable reproduction in the cumulative bug-reproduction suite landing in #667 — a test that pins the exact failing behaviour, so it's tracked and guarded against silent regression. Where #667 also lands the root-cause fix, that's called out in the PR; otherwise the reproduction makes the eventual fix straightforward and verifiable.

    The reproduction goes live once #667 merges. (Posted from the release branch's QA pass.)

  10. DeusData
    added
    priority/highNeeds near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker.
    on Jul 1, 2026
  11. DeusData
    added this to the 0.9.1-rc milestone on Jul 8, 2026

  12. DeusData commented on Sep 26, 2026

    @DeusData
    DeusData
    on Sep 26, 2026
    Owner

    Thank you so much, @nguiaSoren, for coming back with such a thorough retest! The before/after counts, the orphan-process finding and the per-file fn cross-check made the diagnosis unusually clear.

    Your conclusion is right: this was the WAL-durability class (#334), and the checkpoint-on-close and stale-WAL recovery work fixed it. Rust extraction depth was not the cause. Since you confirmed it no longer reproduces, I'm closing this as fixed. If a node count ever collapses again under session churn, please reopen with the pipeline.discover and gbuf.dump lines.

  13. DeusData
    closed this as completedon Sep 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingparsing/qualityGraph extraction bugs, false positives, missing edgespriority/highNeeds near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker.

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions