feat: heading-aware chunker and SQLite FTS5 index with git-SHA provenance #2586

Goal

Index Base documentation locally with deterministic, offline lexical search, and expose it through basectl docs.

Background

For well-structured Markdown with stable headings and finding IDs, lexical BM25 retrieval is deterministic, dependency-free (stdlib sqlite3 with FTS5), and easy to test. Embeddings are deferred until the #2585 eval shows lexical misses. Index placement must follow the cache ownership document.

Next-release planning review of 2026-10-14/15 against main at 7faf6dfb. Tracking: #2555.

Prerequisites: #2585.

Scope

• Corpus definition: docs/**, root Markdown, .ai-context/, schemas; docs/superpowers/ excluded by default.
• Heading-aware chunks with stable anchors and metadata (path, heading path, stability tier, git SHA).
• basectl docs index and basectl docs search <query> [--limit] [--format json]; bare basectl docs keeps opening the URL.
• Incremental rebuild by content hash; clear error when FTS5 is unavailable.

Acceptance Criteria

• The same commit produces an identical index digest.
• Search results carry anchors and the index SHA.
• Recall@5 on the test: documentation question eval set with token baselines #2585 set is recorded in the PR.
• No network access at index or search time.

Validation

BASE_CLI_SOURCE_DIR=../base-cli/lib/python \
PYTHONPATH=../base-cli/lib/python:lib/python:cli/python \
python -m pytest cli/python/base_docs/tests <new docs tests>
git diff --check

Non-Goals

• No embeddings.
• No server (see the MCP issues).

Project Fields

• Status: Backlog
• Priority: P1
• Area: CLI
• Initiative: Agentic Coding Platform
• Size: M
• Milestone: v1.12.0
• Category: enhancement

Agent Assignment

Assignee: codeforester. Blocked on #2585.