Repository navigation
META.yaml/frontmatter description values not indexed for BM25 search #519
Description
META.yaml/frontmatter description values not indexed for BM25 search
Problem
When indexing YAML files (e.g., META.yaml with skill metadata), the description field value is stored as a Variable node with name: "description" — but the actual description text (e.g., "8-agent QA loop using Playwright MCP for browser testing") is not stored in any searchable property.
This means BM25 cannot find skills/modules by their description content.
Reproduction
Given .kiro/skills/qa-run/META.yaml:
name: qa-run
description: "8-agent QA loop: browser exploration via Playwright MCP, then analyze, plan, test, audit, heal, expand, snapshot. Quality gate score >= 85 to pass."The graph creates:
- Module node:
name: ".kiro/skills/qa-run/META.yaml" - Variable node:
name: "description"(just the key name, not the value)
Searching:
search_graph(query="QA loop Playwright browser testing quality gate") → 0 results
search_graph(query="8-agent QA") → 0 results
The description text is never stored or indexed anywhere in the graph.
Expected Behaviour
BM25 search for "Playwright browser testing" should match the qa-run module because its META.yaml description contains those terms.
Proposed Solution
For YAML/JSON files with known metadata fields (description, summary, purpose), store the field value as a searchable property on the parent Module node — e.g., as a docstring or description property that FTS5 indexes.
Scope of change
src/pipeline/— When extracting Variable nodes from YAML/JSON, detectdescription/summaryfields and promote their values to a searchable property on the parent Module node.src/store/— Ensure the new property is included in the FTS5 full-text index.tests/test_pipeline.c— Add test verifying YAML description values are BM25-searchable.
Considerations
- Only promote specific well-known keys (
description,summary,purpose) — not all Variable values - Small storage overhead (~100-200 bytes per module with a description)
- No schema-breaking change — adds a property to existing Module nodes
Context
In projects with many YAML-configured modules (skills, agents, pipelines), the description field is the primary way users think about and search for components. Without indexing these values, BM25 search is limited to file paths and node names — missing the most semantically-rich metadata available.
Environment
- codebase-memory-mcp (latest, full index mode)
- Tested on a workspace with 12 skills, each with META.yaml containing
descriptionfields - 260 Variable nodes indexed, none with searchable values
Activity
DeusData commentedon Jun 21, 2026
Hey, thx for reporting, will check ASAP
ShauryaaSharma commentedon Jun 24, 2026
Hi 👋 opened a PR for this (and its sibling #518 ): #617 .
What it does
Both issues trace back to the same gap: BM25 (
search_graphwithquery) only indexed node names, so the prose under a markdown heading and a YAML/JSONdescriptionvalue were never searchable. The PR:bodycolumn to thenodes_ftsFTS5 table, backfilled from each node'sdocstringproperty (guarded byjson_valid()), through a singlecbm_store_fts_rebuild()that both backfill paths now share.description/summary/purposevalue onto the file'sModulenode (this one), both reuse the existingdocstringproperty, so they flow through serialization and every pipeline pass unchanged.One thing worth flagging 🔍
Indexing the content alone wasn't enough,
bm25_searchinmcp.cwas explicitly excludingSectionandModulefrom results (label NOT IN (…'Section','Module'…)), so even an indexed body would never surface. The PR drops those two from the exclusion list. They carry no rank boost, so code symbols still sort first and ordering for code queries is unchanged.Two decisions I'd appreciate your take on
docstringrather than adding a newcontentfield (the issue noted "add acontentfield (or reusebt)") , it needed zero serialization changes and already survives every pass. Easy to switch to a dedicated field if you'd rather.bodyinfrastructure (this issue can't work without it), so I bundled them in one PR. Happy to split into two stacked PRs if that suits your workflow better.Bonus: since
bodycomes fromdocstring, function/method docstrings are now BM25-searchable too, at no extra cost. If you'd want function bodies searchable, indexingbt(body_tokens) the same way would be a small follow-up.Tests: 7 extraction + 3 store FTS cases. Backward compatible, additive column, and legacy 4-column FTS DBs upgrade automatically on the next index. Thanks for the great project!
feat(search): index prose content for BM25 full-text search
feat(search): index prose content for BM25 full-text search
feat(search): index prose content for BM25 full-text search
Russdataart commentedon Jun 29, 2026
ShauryaaSharma commentedon Jun 29, 2026
Hey @DeusData ,
just wanted to bump this for visibility, Russell (the original reporter) has reviewed the approach and is happy with both design decisions (reusing docstring and keeping the two issues in one PR). Thanks @Russdataart for the kind words and for validating the approach! Would love to get your eyes on it when you get a chance @DeusData!
DeusData commentedon Jun 29, 2026
Hey @ShauryaaSharma, yeah thanks man for the contribution!. Sorry that I am not able to review as quick as you deserve. Right now the shop is full, trying to work my way through everything. Will give you feedback as soon as I can
ShauryaaSharma commentedon Jun 29, 2026
No worries at all @DeusData, take your time, there's no rush on my end! Happy to wait and appreciate you acknowledging it. 🙂
feat(search): index prose content for BM25 full-text search
feat(search): index prose content for BM25 full-text search
feat(search): index prose in BM25 via a nodes_fts body column
test(search): end-to-end guard from source prose to a findable body m…