Skip to content

fix: SCIP enrichment duplicate-PK crash, COPY bad-record retry, ENOSPC preflight - #44

Open
pradeepmouli wants to merge 1 commit into
intuit:mainfrom
pradeepmouli:fix/scip-symbol-crash-fixes
Open

fix: SCIP enrichment duplicate-PK crash, COPY bad-record retry, ENOSPC preflight#44
pradeepmouli wants to merge 1 commit into
intuit:mainfrom
pradeepmouli:fix/scip-symbol-crash-fixes

Conversation

@pradeepmouli

Copy link
Copy Markdown
Contributor

Summary

Root-causes a crash observed in a real-world SCIP-enrichment run: scip_sym_to_name had two bugs that produced colliding or garbled symbol names, which crashed the bulk COPY insert with a primary-key collision.

  • scip_sym_to_name (crates/infigraph-core/src/scip/mod.rs) didn't handle:

    • a double-quote-quoted descriptor with a trailing disambiguator digit (e.g. "'"0, used by scip-typescript for non-identifier term names like string-literal object keys)
    • a chained non-empty (...) disambiguator group after a method's own empty parens (e.g. findNestedSeparator().(rule).), which fell through to returning the raw, unparsed SCIP symbol string as the "name"

    Both now fold the disambiguator into the returned name instead of discarding it — dropping it collapsed genuinely distinct symbols onto the same extracted name, causing the primary-key collision that crashed the COPY.

  • Bad-record-drop-and-retry heuristic. Replaced the "COPY fails → fall back to slow per-row UNWIND for the whole batch" pattern everywhere it occurs (SCIP's Symbol/CALLS/INHERITS inserts, the main indexer's edge-table loop, resolve/calls.rs, resolve/inherits.rs) with a bounded drop-bad-record-and-retry-COPY heuristic: parse the offending value out of Kuzu's own COPY error text, drop just that record, and retry the bulk COPY (up to 20 attempts) before falling back to UNWIND. Avoids paying the slow per-row path for an entire batch over a handful of bad rows.

  • Disk-headroom preflight before SCIP's bulk writes. Kuzu throws an uncaught C++ TransactionManagerException on ENOSPC mid-transaction, aborting the whole process rather than surfacing a Result. Added a preflight check (GraphStore::db_dir + store_util::check_disk_headroom, using the already-present fs2 dependency) that refuses gracefully with a normal error instead.

  • Also hardened a pre-existing flaky assertion in crates/infigraph-mcp/tests/groups_watch_perf.rs (group_index can legitimately report "Indexed 0 repos" if background auto-watch already indexed both repos by the time the explicit call runs — both outcomes mean the data is indexed).

Test plan

  • cargo test -p infigraph-core --lib — 284/284 passing (includes 4 new regression tests for the two scip_sym_to_name bugs, plus tests for the new extract_bad_copy_value/check_disk_headroom helpers)
  • cargo test -p infigraph-mcp --test groups_watch_perf -- --ignored — verified across multiple runs after the assertion fix
  • cargo test -p infigraph-core --test write_lock_perf -- --ignored — passing
  • cargo test -p infigraph-core --test index_perf -- --ignored — passing
  • cargo fmt --all -- --check and cargo clippy --all-targets -- -D warnings — clean

🤖 Generated with Claude Code

https://claude.ai/code/session_01SjmvwHuwV5r7ZeZpJLp5oR

…C preflight

Root-causes the sittir 43GB SCIP-enrichment bloat/crash incident:

- scip_sym_to_name (crates/infigraph-core/src/scip/mod.rs) had two bugs that
  produced colliding or garbled symbol names: it didn't handle a
  double-quote-quoted descriptor with a trailing disambiguator digit (e.g.
  `"'"0`), and it didn't handle a chained non-empty `(...)` disambiguator
  group after a method's own empty parens (e.g. `findNestedSeparator().(rule).`),
  which fell through to returning the raw unparsed SCIP symbol string as the
  name. Both now fold the disambiguator into the returned name instead of
  discarding it, preventing the primary-key collision that crashed the COPY.

- Replaced the "COPY fails -> fall back to per-row UNWIND for the whole
  batch" pattern everywhere it occurred (SCIP's Symbol/CALLS/INHERITS, the
  main indexer's edge-table loop, resolve/calls.rs, resolve/inherits.rs)
  with a bounded drop-bad-record-and-retry-COPY heuristic: parse the
  offending value out of Kuzu's own COPY error text, drop just that record,
  and retry the bulk COPY (up to 20 attempts) before falling back to UNWIND.

- Added a disk-headroom preflight before SCIP's bulk writes (GraphStore::db_dir
  + store_util::check_disk_headroom, using the existing fs2 dependency).
  lbug throws an uncaught C++ TransactionManagerException on ENOSPC
  mid-transaction, aborting the whole process rather than surfacing a Rust
  Result -- this preflight turns that into a graceful error instead.

Also confirmed (not a code change): all 4 write-capable paths -- SCIP
enrichment, full reindex, incremental index/watch, and cross-file resolve --
already acquire GraphStore::write_lock() independently, so the incident was
not a locking gap.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SjmvwHuwV5r7ZeZpJLp5oR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant