test: documentation question eval set with token baselines #2585

Goal

Create the measuring stick for documentation retrieval before building it.

Background

Base's docs are about 115K words across 73 files (excluding docs/superpowers/), roughly 150K tokens. basectl export-context returns the curated .ai-context/ pack (about 8K words, roughly 11K tokens), and basectl docs only opens a URL. The stated goal is answering documentation questions without burning tokens; that is unprovable without a question set and baselines.

Next-release planning review of 2026-10-14/15 against main at 7faf6dfb. Tracking: #2555.

Scope

• At least 50 real questions mined from closed support/question issues, FAQ.md, first-run troubleshooting, and maintainer history, covering command lookup, finding IDs, concepts, procedures, and troubleshooting.
• Each with expected source anchors (path#heading) and required facts.
• Baselines: .ai-context pack, full docs, and an exploratory-agent reading cost.
• A deterministic offline harness computing recall@k, MRR, and returned-context tokens.

Acceptance Criteria

• Question set and baselines are committed.
• Harness runs offline and deterministically in CI.
• The set includes questions the .ai-context pack cannot answer.

Validation

BASE_CLI_SOURCE_DIR=../base-cli/lib/python \
PYTHONPATH=../base-cli/lib/python:lib/python:cli/python \
python -m pytest tests/<new eval harness tests>
git diff --check

Non-Goals

• No retrieval implementation here.
• No LLM judge required for CI.

Project Fields

• Status: Ready
• Priority: P1
• Area: Docs
• Initiative: Agentic Coding Platform
• Size: M
• Milestone: v1.12.0
• Category: ci

Agent Assignment

Assignee: codeforester. Implementation-ready; use an issue-backed worktree.