Skip to content

refactor(indexer): split pipeline.rs (2374 lines) into 7-module per-concern tree#17

Merged
sdsrss merged 1 commit into
mainfrom
refactor/split-pipeline-rs
May 9, 2026
Merged

refactor(indexer): split pipeline.rs (2374 lines) into 7-module per-concern tree#17
sdsrss merged 1 commit into
mainfrom
refactor/split-pipeline-rs

Conversation

@sdsrss
Copy link
Copy Markdown
Owner

@sdsrss sdsrss commented May 9, 2026

Summary

File Lines Concern
mod.rs 237 entry points + IndexStats/IndexResult/ProgressFn + collect_dirty_node_ids
embed.rs 71 embed_and_store_batch + sequential fallback path
context.rs 197 categorize_edges + format_route_from_metadata + regenerate_context_strings + repair_null_context_strings
python_modules.rs 73 build_python_module_map + resolve_python_module_targets
resolve.rs 204 refine_ambiguous_targets + resolve_pending_calls
index_files.rs 827 Phase 0..3 orchestrator + FileIndexed
tests.rs 884 all #[cfg(test)] tests

Public surface preserved (crate::indexer::pipeline::{run_full_index, ensure_file_indexed, run_incremental_index, run_incremental_index_cached, embed_and_store_batch, repair_null_context_strings, IndexStats, IndexResult, ProgressFn}). External callers in cli.rs / mcp/server / tests/ / benches/ need no edits.

Submodule items use pub(super); embed_and_store_batch and repair_null_context_strings get pub use'd at mod.rs to keep their existing public path.

Test plan

  • cargo check — clean
  • cargo +1.95.0 clippy --no-default-features -- -D warnings — clean
  • cargo +1.95.0 clippy --all-targets -- -D warnings — clean
  • cargo test --release — 292 lib + 129 integration = 421 tests, 0 failed (1 pre-existing #[ignore])
  • CI green on PR

🤖 Generated with Claude Code

src/indexer/pipeline.rs (2374 lines) → src/indexer/pipeline/{mod,embed,
context,python_modules,resolve,index_files,tests}.rs. mod.rs (237 lines)
keeps the public entry points (run_full_index, ensure_file_indexed,
run_incremental_index{,_cached}) + IndexStats/IndexResult/ProgressFn +
collect_dirty_node_ids glue. The Phase-0..3 orchestrator stays whole in
index_files.rs (827 lines) — its phases share local transaction/atomics/
batch_parsed/name_to_ids state that splitting would have to thread back
in via large arg lists.

Per-concern submodules:
  embed.rs          (71)  — embed_and_store_batch + sequential fallback
  context.rs        (197) — categorize_edges + format_route_from_metadata
                            + regenerate_context_strings + repair_null_*
  python_modules.rs (73)  — build_python_module_map + resolve_python_*
  resolve.rs        (204) — refine_ambiguous_targets + resolve_pending_calls
  index_files.rs    (827) — Phase 0..3 orchestrator + FileIndexed
  tests.rs          (884) — all #[cfg(test)] tests

Public surface preserved (`crate::indexer::pipeline::{run_full_index,
ensure_file_indexed, run_incremental_index, run_incremental_index_cached,
embed_and_store_batch, repair_null_context_strings, IndexStats,
IndexResult, ProgressFn}`). External callers in cli.rs / mcp/server /
tests / benches / claude-plugin all keep their imports unchanged.

Verification:
- cargo check: clean
- cargo +1.95.0 clippy --no-default-features -- -D warnings: clean
- cargo +1.95.0 clippy --all-targets -- -D warnings: clean
- cargo test --release: 292 lib + 6 + 44 + 19 + 6 + 54 = 421 tests, 0 failed
  (1 pre-existing #[ignore])

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 9, 2026

Warning

Rate limit exceeded

@sdsrss has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 18 minutes and 6 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4bc9aad1-c2b7-4734-9673-994c46f113f5

📥 Commits

Reviewing files that changed from the base of the PR and between 2dfbab9 and 968f63b.

📒 Files selected for processing (8)
  • src/indexer/pipeline.rs
  • src/indexer/pipeline/context.rs
  • src/indexer/pipeline/embed.rs
  • src/indexer/pipeline/index_files.rs
  • src/indexer/pipeline/mod.rs
  • src/indexer/pipeline/python_modules.rs
  • src/indexer/pipeline/resolve.rs
  • src/indexer/pipeline/tests.rs
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch refactor/split-pipeline-rs

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@sdsrss sdsrss merged commit aef55b2 into main May 9, 2026
9 checks passed
@sdsrss sdsrss deleted the refactor/split-pipeline-rs branch May 9, 2026 21:17
sdsrss added a commit that referenced this pull request May 9, 2026
…/ pipeline)

Pure refactor release — three biggest source files (8049 lines as monoliths)
decomposed into 26 per-concern submodules. Zero behavior change, public
surface preserved across all three splits.

- src/storage/queries.rs (2892 lines) → src/storage/queries/ (10 files, #15)
- src/parser/relations.rs (2783 lines) → src/parser/relations/ (9 files, #16)
- src/indexer/pipeline.rs (2374 lines) → src/indexer/pipeline/ (7 files, #17)

External callers in cli.rs / mcp/server / tests / benches / claude-plugin
keep their imports unchanged. Two orchestrator-style functions kept whole
deliberately (walk_for_relations ~650 lines, index_files ~770 lines) — their
arms/phases share local state that splitting would either duplicate or thread
back via large arg lists.

Verification: cargo check + cargo +1.95.0 clippy (no-default-features +
all-targets, -D warnings) + cargo test --release (292 lib + 129 integration
= 421 tests, 0 failed). Pre-merge CI green on all three PRs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant