Section nodes don't index body text — BM25 can't search markdown content

Problem

Section nodes extracted from markdown files store only the heading text in the name field. The prose content below each heading is discarded. This means BM25 search cannot find terms that appear inside markdown body text — only in headings.

For markdown-heavy projects (skill definitions, documentation workspaces, knowledge bases), this makes search_graph largely ineffective for content discovery.

Reproduction

Given a file .kiro/skills/qa-run/SKILL.md containing:

## BROWSER AGENT (Playwright MCP — runs first)

Before writing ANY test file, explore the live application using Playwright MCP.
This is non-negotiable for web applications. You MUST see the real app first.

The graph creates a Section node with:

name: "BROWSER AGENT (Playwright MCP — runs first)"
lines: "" (empty)
No content, bt, or docstring field

Searching for "Playwright" or "browser testing" or "test file" via BM25 returns 0 results, because those terms only appear in the body text which is never stored or indexed.

search_graph(query="QA testing playwright browser") → 0 results
search_graph(query="quality assurance test automation") → 0 results

Meanwhile, Function nodes DO have docstring and bt (body tokens) fields that BM25 indexes, making function content fully searchable.

Impact

Tested on a 65-file workspace (29 markdown files, 10 Python functions):

Query type	BM25 accuracy	Notes
Function by name/docstring	5/5	Works perfectly
Function by body token	5/5	`bt` field indexed
Markdown by heading text	5/5	Heading = node name
Markdown by body content	0/5	Body text not stored

This means for documentation-heavy projects, search_graph is blind to ~80% of the content.

Proposed Solution

Add a content field (or reuse bt) on Section nodes containing the first N characters (e.g., 500) of the body text below each heading. Index this field in FTS5 alongside name and docstring.

Expected changes

src/pipeline/ — During markdown extraction, capture body text between the current heading and the next heading (or end of file). Store as content property on Section nodes.
src/store/ — Add content to the FTS5 index so BM25 searches it.
tests/test_pipeline.c — Add test cases verifying Section body content is extracted and searchable.

Considerations

Size: 500 chars per section × 209 sections = ~100KB additional storage (negligible vs typical graph size of 37MB+)
Backward compat: New field, no existing field changes
Performance: FTS5 handles the additional text with no measurable impact at this scale

Evidence

Full evaluation results with 12 test cases comparing CBM vs baseline tools on a markdown-heavy workspace: the 4 tests where CBM scores 3/5 or below (T3 Architecture, T7 Semantic Search, T9 Persistence quality, T12 Cross-Domain Search) all fail specifically because markdown body content is not indexed.

Environment

codebase-memory-mcp (latest, full index mode)
631 nodes, 634 edges indexed
Node labels: Variable(260), Section(209), File(60), Module(59), Folder(29), Function(10)
Section properties available: name, qualified_name, file_path, start_line, end_line, complexity, is_entry_point, is_exported, is_test, lines
Section properties missing: content, bt, docstring (any body text field)

Hey, thx will check ASAP

Hi 👋 opened a PR for this (and its sibling #519 ): #617 .

What it does

Both issues trace back to the same gap: BM25 (search_graph with query) only indexed node names, so the prose under a markdown heading and a YAML/JSON description value were never searchable. The PR:

Adds a body column to the nodes_fts FTS5 table, backfilled from each node's docstring property (guarded by json_valid()), through a single cbm_store_fts_rebuild() that both backfill paths now share.
Captures the markdown section body beneath each heading (this issue) and promotes a top-level description/summary/purpose value onto the file's Module node (META.yaml/frontmatter description values not indexed for BM25 search #519 ), both reuse the existing docstring property, so they flow through serialization and every pipeline pass unchanged.

One thing worth flagging 🔍

Indexing the content alone wasn't enough, bm25_search in mcp.c was explicitly excluding Section and Module from results (label NOT IN (…'Section','Module'…)), so even an indexed body would never surface. The PR drops those two from the exclusion list. They carry no rank boost, so code symbols still sort first and ordering for code queries is unchanged.

Two decisions I'd appreciate your take on

I reused docstring rather than adding a new content field (the issue noted "add a content field (or reuse bt)") , it needed zero serialization changes and already survives every pass. Easy to switch to a dedicated field if you'd rather.
Section nodes don't index body text — BM25 can't search markdown content #518 and META.yaml/frontmatter description values not indexed for BM25 search #519 share the FTS body infrastructure (META.yaml/frontmatter description values not indexed for BM25 search #519 can't work without it), so I bundled them in one PR. Happy to split into two stacked PRs if that suits your workflow better.

Bonus: since body comes from docstring, function/method docstrings are now BM25-searchable too, at no extra cost. If you'd want function bodies searchable, indexing bt (body_tokens) the same way would be a small follow-up.

Tests: 7 extraction + 3 store FTS cases. Backward compatible, additive column, and legacy 4-column FTS DBs upgrade automatically on the next index. Thanks for the great project!

feat(search): index prose content for BM25 full-text search

test(repro): grammar batteries — scientific/shaders + markup/docs

This issue now has a first-class, durable reproduction in the cumulative bug-reproduction suite landing in #667 — a test that pins the exact failing behaviour, so it's tracked and guarded against silent regression. Where #667 also lands the root-cause fix, that's called out in the PR; otherwise the reproduction makes the eventual fix straightforward and verifiable.

The reproduction goes live once #667 merges. (Posted from the release branch's QA pass.)

feat(search): index prose content for BM25 full-text search

** #617 Rebased onto main (post-#667) , all tests green ✅**

Rebased on top of the #667 merge. One minor conflict in extract_defs.c: upstream added qn_safe_segment() immediately before push_simple_class_def as part of the markdown QN slugification work. Resolution was additive, kept both the new helper and the docstring parameter this PR ( #617 ) adds.

Verification against the #518 repro in the bug suite:

The #667 markup battery (repro_grammar_markup_markdown) now correctly expects "Section" as the label for headings, which is exactly what this PR ( #617 ) emits. Running the 7 targeted extraction tests locally confirms the rebased code is still correct:

markdown_section_body_captured          PASS  ← Playwright body captured, sibling excluded
markdown_section_no_body                PASS  ← empty heading → no docstring
markdown_section_body_capped            PASS  ← body ≤ 500 bytes
yaml_description_promoted_to_module     PASS
yaml_summary_promoted_to_module         PASS
json_description_promoted_to_module     PASS
yaml_no_description_leaves_module_bare  PASS

feat(search): index prose content for BM25 full-text search

test(bench): isolate the nodes_fts body column cost (DeusData#518)

style(store): split CBM_SQL_FTS_BODY_EXPR across continuation lines

feat(search): index prose in BM25 via a nodes_fts body column

test(search): end-to-end guard from source prose to a findable body m…

Navigation Menu

Repository navigation

Section nodes don't index body text — BM25 can't search markdown content #518

Section nodes don't index body text — BM25 can't search markdown content

Problem

Reproduction

Impact

Proposed Solution

Expected changes

Considerations

Evidence

Environment

What it does

One thing worth flagging 🔍

Two decisions I'd appreciate your take on

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Section nodes don't index body text — BM25 can't search markdown content #518

Description

Section nodes don't index body text — BM25 can't search markdown content

Problem

Reproduction

Impact

Proposed Solution

Expected changes

Considerations

Evidence

Environment

Activity

DeusData commented on Jun 21, 2026

ShauryaaSharma commented on Jun 24, 2026

What it does

One thing worth flagging 🔍

Two decisions I'd appreciate your take on

DeusData commented on Jun 28, 2026

ShauryaaSharma commented on Jun 29, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions