feat(core): add context-efficient retrieval

README.md

@@ -77,7 +77,8 @@ pnpm exec rgr doctor
 ```
 
 Setup writes local helper files for the selected agents. The MCP surface is intentionally bounded
-and read-focused.
+and read-focused. Agents can request compact evidence first, then expand one returned citation
+without opening a second index or reading arbitrary files.
 
 ### Audit a knowledge base
 

docs/agent-integration.md

@@ -28,9 +28,17 @@ Use `--scope user` only when you intentionally want a user-wide installation. Pr
 
 ## MCP tools
 
-The server exposes `ragmir_status`, `ragmir_route_prompt`, `ragmir_search`, `ragmir_ask`, `ragmir_research`, `ragmir_audit`, `ragmir_evaluate`, `ragmir_usage_report`, and `ragmir_security_audit`.
-
-Use compact search output when context is limited. `ragmir_ask` returns cited evidence, not a model generated answer. A cloud agent can receive returned passages, so choose that handoff only when it matches the corpus’s confidentiality requirements.
+The server exposes `ragmir_status`, `ragmir_route_prompt`, `ragmir_search`, `ragmir_ask`,
+`ragmir_research`, `ragmir_expand`, `ragmir_audit`, `ragmir_evaluate`, `ragmir_usage_report`, and
+`ragmir_security_audit`.
+
+Use compact retrieval first, then pass a returned citation to `ragmir_expand` when the agent needs
+the exact chunk or a bounded neighbor window. Search, ask, research, and expansion accept `maxBytes`;
+the server also enforces the configured `mcpMaxOutputBytes` ceiling. Their single JSON text result
+stays parseable, while `_meta["ragmir/output"]` reports retrieved bytes, returned bytes, compacting,
+and truncation. `ragmir_ask` returns cited evidence, not a model generated answer. A cloud agent can
+receive returned passages, so choose that handoff only when it matches the corpus’s confidentiality
+requirements.
 
 ## Verify
 

docs/api-reference.md

@@ -29,11 +29,16 @@ All paths resolve from `cwd` or the current working directory. Retrieval results
 | `search(query, options?)` | Return ranked cited passages. |
 | `ask(query, options?)` | Return cited retrieval context without calling an LLM. |
 | `research(query, options?)` | Run audit-backed multi-query retrieval. |
+| `expandCitation(citation, options?)` | Read one exact indexed chunk and a bounded neighbor window. |
 | `compactSearchResults(results)` | Produce compact search output for limited contexts. |
 | `compactResearchReport(report)` | Remove verbose evidence text from a research report. |
 | `evaluateGoldenQueries(options)` | Score retrieval against a local golden-query file. |
 
 `SearchOptions` accepts `cwd`, `topK`, `contextRadius`, `includePaths`, and `excludePaths`.
+`ExpandCitationOptions` accepts `cwd` and a `contextRadius` clamped to three chunks.
+Search results expose a `contextPath` derived from Markdown headings or JSON structure. This field
+improves retrieval, while `text`, offsets, and citations continue to reference the exact source
+passage.
 
 ## Operations and privacy
 
@@ -70,3 +75,8 @@ All paths resolve from `cwd` or the current working directory. Retrieval results
 
 `kbCommand` and `ragmirCommand` remain compatibility helpers. New integration code should use
 `rgrCommand` and the `rgr` CLI name.
+
+The MCP retrieval tools accept an optional `maxBytes` below the configured `mcpMaxOutputBytes`
+ceiling. `ragmir_search` and `ragmir_research` also accept `compact`; `ragmir_ask` supports the same
+flag. Oversized full output is compacted and then reduced as valid JSON. Output metrics are available
+under `_meta["ragmir/output"]` and in the metadata-only usage report.

docs/configuration.md

@@ -21,11 +21,14 @@ edit JSON only for a real need.
 | `retrievalProfile` | `balanced` | Use `fast`, `quality`, or `custom` for different search budgets. |
 | `embeddingProvider` | `local-hash` | Set `transformers` only after an explicit preload. |
 | `topK` | `8` | Change the default number of returned passages. |
+| `mcpMaxOutputBytes` | `32768` | Cap each retrieval tool's serialized MCP text output. |
 | `chunkSize` / `chunkOverlap` | `1200` / `200` | Tune chunking, then rebuild the index. |
 | `maxFileBytes` | `50000000` | Raise only when the target corpus justifies it. |
 | `includeExtensions` | `[]` | Add safe custom text extensions. |
 
 Changing an embedding provider, model, or chunking field requires `rgr ingest --rebuild`.
+Ragmir also preserves Markdown heading paths and JSON or JSONL structure as retrieval-only context.
+Rebuild indexes created by an older Ragmir version to populate that structural context.
 
 ## Privacy profiles
 
@@ -61,7 +64,8 @@ they run without a shell and must print text to stdout.
 Use `RAGMIR_*` variables for local experiments, for example:
 
 ```bash
-RAGMIR_TOP_K=5 RAGMIR_CHUNK_SIZE=1000 rgr search "migration"
+RAGMIR_TOP_K=5 rgr search "migration"
+RAGMIR_MCP_MAX_OUTPUT_BYTES=16384 rgr mcp
 ```
 
 Environment overrides cover selected runtime settings such as models, retrieval limits, access logs,

packages/ragmir-core/README.md

@@ -89,7 +89,7 @@ Frequently used exports:
 | --- | --- |
 | `setupProject`, `addSourceEntries` | Initialize project state and select files |
 | `ingest`, `audit` | Build the index and compare it with files on disk |
-| `search`, `ask`, `research` | Retrieve cited passages at different levels of breadth |
+| `search`, `ask`, `research`, `expandCitation` | Retrieve or expand cited passages |
 | `doctor`, `securityAudit` | Inspect readiness and local privacy posture |
 | `serveMcp` | Start the read-focused local MCP server |
 | `configurePdfOcr`, `inspectPdfOcr` | Configure and inspect local PDF OCR |
@@ -105,8 +105,9 @@ npx rgr doctor
 ```
 
 Ragmir writes helper files for the selected clients and points them at the current project. MCP
-exposes status, search, ask, research, audit, evaluation, usage, and security tools. It does not
-expose index deletion.
+exposes status, search, ask, research, exact citation expansion, audit, evaluation, usage, and
+security tools. Retrieval responses have a global byte ceiling and expose metadata-only output
+metrics. The server does not expose index deletion.
 
 ## Retrieval modes
 
@@ -147,7 +148,8 @@ local executable, never a shell string or cloud OCR service.
 - Generated indexes, models, reports, audio, and access logs belong under ignored `.ragmir/` state.
 - Redaction runs before indexing when configured, but it is not a compliance certification.
 - External extractors and model downloads are opt-in system boundaries.
-- MCP retrieval is bounded and access logs contain metadata, not query text or retrieved passages.
+- MCP retrieval is bounded by `mcpMaxOutputBytes`; access logs contain byte metrics, not query text
+  or retrieved passages.
 
 Read [configuration and privacy](https://github.com/jcode-works/jcode-ragmir/blob/main/docs/configuration.md)
 and [security hardening](https://github.com/jcode-works/jcode-ragmir/blob/main/SECURITY-HARDENING.md)

packages/ragmir-core/scripts/mcp-smoke.mjs

@@ -17,6 +17,7 @@ const requiredTools = [
   "ragmir_search",
   "ragmir_ask",
   "ragmir_research",
+  "ragmir_expand",
   "ragmir_audit",
   "ragmir_evaluate",
   "ragmir_usage_report",
@@ -61,6 +62,9 @@ try {
   if (status.chunksIndexed < 1) {
     throw new Error("MCP status reported an empty index.")
   }
+  if (status.mcpMaxOutputBytes !== 32_768) {
+    throw new Error(`MCP status reported an unexpected output budget: ${JSON.stringify(status)}`)
+  }
   if (
     status.ingestionLimits?.maxFileBytes !== 50_000_000 ||
     status.ingestionLimits?.maxFiles !== null ||
@@ -90,6 +94,47 @@ try {
     throw new Error("MCP compact search did not return snippet-only results.")
   }
 
+  const boundedSearchResult = await client.callTool(
+    {
+      name: "ragmir_search",
+      arguments: {
+        query: "offline retrieval approval",
+        topK: 2,
+        compact: true,
+        maxBytes: 1_024,
+      },
+    },
+    undefined,
+    { timeout: 10_000 },
+  )
+  const boundedSearch = parseJsonToolResult(boundedSearchResult, "ragmir_search")
+  const outputMeta = boundedSearchResult._meta?.["ragmir/output"]
+  if (
+    !Array.isArray(boundedSearch) ||
+    typeof outputMeta?.returnedBytes !== "number" ||
+    outputMeta.returnedBytes > 1_024 ||
+    outputMeta.budgetBytes !== 1_024
+  ) {
+    throw new Error(
+      `MCP search did not enforce its byte budget: ${JSON.stringify(boundedSearchResult)}`,
+    )
+  }
+
+  const expanded = await callJsonTool(client, "ragmir_expand", {
+    citation: searchResults[0].citation,
+    contextRadius: 1,
+    maxBytes: 1_024,
+  })
+  if (
+    expanded.found !== true ||
+    !Array.isArray(expanded.passages) ||
+    !expanded.passages.some((passage) => passage.citation === searchResults[0].citation)
+  ) {
+    throw new Error(
+      `MCP citation expansion returned an unexpected result: ${JSON.stringify(expanded)}`,
+    )
+  }
+
   const filteredSearchResults = await callJsonTool(client, "ragmir_search", {
     query: "offline retrieval approval",
     topK: 5,
@@ -108,10 +153,14 @@ try {
   const answer = await callJsonTool(client, "ragmir_ask", {
     query: "What evidence supports offline operation?",
     topK: 2,
+    compact: true,
   })
   if (!Array.isArray(answer.sources) || answer.sources.length < 1) {
     throw new Error("MCP ask returned no cited sources.")
   }
+  if (!("snippet" in answer.sources[0]) || "text" in answer.sources[0]) {
+    throw new Error("MCP compact ask did not return snippet-only sources.")
+  }
 
   const research = await callJsonTool(client, "ragmir_research", {
     query: "offline retrieval approval",
@@ -140,6 +189,9 @@ try {
   if (typeof usage.averageResultCountByAction?.search !== "number") {
     throw new Error(`MCP usage report omitted per-action averages: ${JSON.stringify(usage)}`)
   }
+  if (usage.mcpOutput?.responses < 1 || usage.mcpOutput.returnedBytes < 1) {
+    throw new Error(`MCP usage report omitted output metrics: ${JSON.stringify(usage)}`)
+  }
   if (JSON.stringify(usage).includes(demoRoot)) {
     throw new Error("MCP usage report should not expose local project paths.")
   }
@@ -152,6 +204,7 @@ try {
         tools: requiredTools,
         chunksIndexed: status.chunksIndexed,
         searchResults: searchResults.length,
+        expandedPassages: expanded.passages.length,
         askSources: answer.sources.length,
         researchEvidence: research.evidence.length,
         evaluationRecall: evaluation.recall,
@@ -182,6 +235,10 @@ function runCliJson(args) {
 
 async function callJsonTool(client, name, args) {
   const result = await client.callTool({ name, arguments: args }, undefined, { timeout: 10_000 })
+  return parseJsonToolResult(result, name)
+}
+
+function parseJsonToolResult(result, name) {
   if (result.isError) {
     throw new Error(`${name} returned an MCP error.`)
   }

packages/ragmir-core/skills/ragmir/SKILL.md

@@ -48,6 +48,7 @@ acceptable. Normal confidential indexing keeps remote model loading disabled.
 | Check unindexed or unsupported files | `rgr audit --unsupported` |
 | Check privacy posture | `rgr security-audit` |
 | Retrieve exact passages | `rgr search "query" --compact` or `ragmir_search` |
+| Expand one returned citation | `ragmir_expand` |
 | Gather broad cited evidence | `rgr research "topic" --compact` or `ragmir_research` |
 | Return deterministic context | `rgr ask "question"` or `ragmir_ask` |
 | Measure retrieval recall | `rgr evaluate --golden <file>` or `ragmir_evaluate` |
@@ -99,6 +100,11 @@ Cline. A generic server configuration is:
 When a client cannot set `cwd`, set `RAGMIR_PROJECT_ROOT` for the server process. Prefer MCP tools
 when available. Use CLI commands when they are not.
 
+Prefer compact search, ask, or research output first. Call `ragmir_expand` with a returned citation
+only when the exact chunk or neighboring context is needed. Retrieval tools accept `maxBytes`, but
+the configured `mcpMaxOutputBytes` remains the hard ceiling. Inspect `_meta["ragmir/output"]` to see
+whether the response was compacted or truncated.
+
 MCP is read-focused. Only remove an index with the explicit shell command:
 
 ```sh

packages/ragmir-core/src/access-log.test.ts

@@ -2,7 +2,7 @@ import { appendFile, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/prom
 import os from "node:os"
 import path from "node:path"
 import { afterEach, describe, expect, it } from "vitest"
-import { accessLogUsageReport, recordAccess } from "./access-log.js"
+import { accessLogUsageReport, recordAccess, recordMcpOutput } from "./access-log.js"
 import { loadConfig } from "./config.js"
 import { initProject } from "./init.js"
 
@@ -77,6 +77,71 @@ describe("accessLogUsageReport", () => {
     expect(report.averageResultCountByAction.ask).toBe(2)
     expect(report.averageResultCountByAction.evaluate).toBeNull()
   })
+
+  it("should aggregate metadata-only MCP output metrics without storing corpus details", async () => {
+    const root = await mkdtemp(path.join(os.tmpdir(), "ragmir-mcp-output-log-"))
+    tempDirs.push(root)
+    await initProject(root)
+    const config = await loadConfig(root)
+
+    await recordMcpOutput(config, {
+      tool: "ragmir_search",
+      retrievedBytes: 8_000,
+      returnedBytes: 2_000,
+      compacted: true,
+      truncated: false,
+    })
+    await recordMcpOutput(config, {
+      tool: "ragmir_expand",
+      retrievedBytes: 3_000,
+      returnedBytes: 1_000,
+      compacted: false,
+      truncated: true,
+    })
+
+    const report = await accessLogUsageReport({ cwd: root, days: 7 })
+    const rawLog = await readFile(config.accessLogPath, "utf8")
+
+    expect(report.mcpOutput).toEqual({
+      responses: 2,
+      retrievedBytes: 11_000,
+      returnedBytes: 3_000,
+      savedBytes: 8_000,
+      reductionRatio: 8 / 11,
+      compactedResponses: 1,
+      truncatedResponses: 1,
+    })
+    expect(rawLog).not.toContain("private query")
+    expect(rawLog).not.toContain("raw/policy.md")
+    expect(rawLog).not.toContain(root)
+  })
+
+  it("should reject malformed MCP metric lines while reading older access entries", async () => {
+    const root = await mkdtemp(path.join(os.tmpdir(), "ragmir-mcp-output-invalid-"))
+    tempDirs.push(root)
+    await initProject(root)
+    const config = await loadConfig(root)
+    await recordAccess(config, { action: "search", query: "known query", resultCount: 1 })
+    await appendFile(
+      config.accessLogPath,
+      `${JSON.stringify({
+        timestamp: new Date().toISOString(),
+        kind: "mcp-output",
+        tool: "ragmir_search",
+        retrievedBytes: -1,
+        returnedBytes: 1,
+        compacted: false,
+        truncated: false,
+      })}\n`,
+      "utf8",
+    )
+
+    const report = await accessLogUsageReport({ cwd: root, days: 7 })
+
+    expect(report.eventsByAction.search).toBe(1)
+    expect(report.mcpOutput.responses).toBe(0)
+    expect(report.invalidLines).toBe(1)
+  })
 })
 
 describe("recordAccess retention", () => {