Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
- Keep `docs/` concise and task-oriented: CLI, API, configuration, agent integration, troubleshooting, local chat, and local TTS. Remove future plans and obsolete surfaces rather than documenting them.
- Package READMEs are brief npm entrypoints that link to the root README.
- Keep `llms.txt` and `context7.json` aligned with public documentation and generated-output exclusions.
- When code changes public behavior, commands, configuration, supported formats, architecture, or product claims, update the relevant docs and landing in the same change. For internal-only changes, verify both surfaces and leave them unchanged when no update is needed.

## Validation

Expand Down
3 changes: 3 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ Use `pnpm --filter @jcode.labs/ragmir <script>` for Core-only work. The pinned N

Generated `dist/`, `.astro/`, `release-artifacts/`, and `.ragmir/` directories are ignored. Do not
commit them. The root README is the canonical documentation entrypoint; keep package READMEs short.
When code changes public behavior, commands, configuration, supported formats, architecture, or
product claims, update the relevant docs and landing in the same change. For internal-only changes,
verify both surfaces and leave them unchanged when no update is needed.

## Boundaries

Expand Down
40 changes: 35 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,18 +78,44 @@ pnpm exec rgr doctor

Setup writes local helper files for the selected agents. The MCP surface is intentionally bounded
and read-focused. Agents can request compact evidence first, then expand one returned citation
without opening a second index or reading arbitrary files.
without opening a second index or reading arbitrary files. MCP clients can read `ragmir://context`
for a compact base, readiness, freshness, and capability overview before choosing a tool.

### Route knowledge in a monorepo

```bash
pnpm exec rgr bases
pnpm exec rgr --project-root apps/web search "checkout contract"
```

Ragmir selects the nearest `.ragmir/config.json` from the working directory. A monorepo can keep a
root base for shared knowledge and isolated bases in individual apps. `rgr bases` shows the active
base, generated MCP helpers pin their project root, and nested bases receive distinct server names
so agents do not silently query the wrong index.

### Audit a knowledge base

```bash
pnpm exec rgr preview --path docs --max-chunks 3
pnpm exec rgr audit --unsupported
pnpm exec rgr security-audit
pnpm exec rgr research "release obligations" --compact
```

Use this path for policies, runbooks, specifications, contracts, and other corpora where the answer
must remain traceable to evidence.
must remain traceable to evidence. `preview` shows redacted chunks, structural context, citations,
and size distributions without writing an index.

### Explain retrieval decisions

```bash
pnpm exec rgr search "release approval" --explain
pnpm exec rgr search "release approval" --context-path "Operations > Release"
```

Explanations expose reciprocal-rank-fusion contributions, vector and lexical ranks, backend scores,
and matched terms without changing default ranking. Structural filters can target Markdown heading
paths or JSON paths before candidate retrieval.

### Enable semantic retrieval

Expand Down Expand Up @@ -130,15 +156,19 @@ import { ingest, search } from "@jcode.labs/ragmir"

await ingest({ cwd: process.cwd() })

const results = await search("Which decision changed the rollout?", { topK: 5 })
const results = await search("Which decision changed the rollout?", {
topK: 5,
explain: true,
})

for (const result of results) {
console.log(result.citation, result.text)
}
```

Core also exports `ask`, `research`, `audit`, `doctor`, `securityAudit`, `serveMcp`, and setup
helpers. See the [API reference](./docs/api-reference.md) for the public surface.
Core also exports `previewChunks`, `ask`, `research`, `audit`, `doctor`, `securityAudit`,
`discoverKnowledgeBases`, bounded context helpers, `serveMcp`, and setup helpers. See the
[API reference](./docs/api-reference.md) for the public surface.

## Privacy boundaries

Expand Down
5 changes: 4 additions & 1 deletion context7.json
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@
"The `local-hash` embedding provider (default) is a lexical sha256 embedding, not semantic; use `transformers` for semantic retrieval.",
"Switching `embeddingProvider` requires `rgr ingest --rebuild`, since the two providers produce incompatible vectors.",
"Run `rgr doctor --fix` after upgrading or misconfiguration to repair scaffolding, `.gitignore` entries, and the agent skill install.",
"Config resolves from the caller's working directory (`.ragmir/config.json`), never from the package install path."
"Config resolves from the caller's working directory (`.ragmir/config.json`), never from the package install path.",
"In a monorepo, the nearest configured ancestor is active; use `rgr bases --json` or explicit `--project-root` to verify routing.",
"Use `rgr preview` to inspect redacted chunks without writing an index, and `search(..., { explain: true })` only when rank diagnostics are needed.",
"MCP resources `ragmir://context` and `ragmir://sources` are bounded agent-orientation surfaces; retrieval still happens through the read-focused tools."
]
}
28 changes: 28 additions & 0 deletions docs/agent-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,39 @@ rgr install-agent --agents codex,claude

Use `--scope user` only when you intentionally want a user-wide installation. Project scope is the default. `--mode copy` is a fallback for filesystems that cannot follow symlinks.

## Monorepo bases

A monorepo can run one root knowledge base plus isolated bases in individual apps. From the app or
file currently in scope, run:

```bash
rgr bases --json
```

The nearest `.ragmir/config.json` is `activeId`. Use the root base for shared architecture and
cross-app decisions; use the nearest app base for app-specific questions. Generated MCP helpers set
`RAGMIR_PROJECT_ROOT` explicitly. Nested bases also receive deterministic names such as
`ragmir-apps-web`, avoiding collisions with the root `ragmir` server. If an agent can see more than
one Ragmir server, call `ragmir_status` and verify `knowledgeBaseId` before retrieval. Keep evidence
from different bases labeled rather than silently merging citations.

## MCP tools

The server exposes `ragmir_status`, `ragmir_route_prompt`, `ragmir_search`, `ragmir_ask`,
`ragmir_research`, `ragmir_expand`, `ragmir_audit`, `ragmir_evaluate`, `ragmir_usage_report`, and
`ragmir_security_audit`.

It also exposes two bounded resources:

| Resource | Use |
| --- | --- |
| `ragmir://context` | Active base identity, readiness, freshness, coverage, and available operations. |
| `ragmir://sources` | Source coverage, skipped-file counts, and index drift, with per-file lists capped at 50. |

Read `ragmir://context` first when the client supports resources. This gives an agent enough context
to choose the next operation without chaining status, doctor, and audit calls. Totals in
`ragmir://sources` stay complete even when detail lists are truncated.

Use compact retrieval first, then pass a returned citation to `ragmir_expand` when the agent needs
the exact chunk or a bounded neighbor window. Search, ask, research, and expansion accept `maxBytes`;
the server also enforces the configured `mcpMaxOutputBytes` ceiling. Their single JSON text result
Expand All @@ -45,6 +72,7 @@ requirements.
```bash
rgr doctor
rgr status --json
rgr bases --json
rgr search "known phrase" --compact
```

Expand Down
18 changes: 17 additions & 1 deletion docs/api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@ All paths resolve from `cwd` or the current working directory. Retrieval results
| `initProject(cwd?)` | Create local Ragmir state and ignore rules. |
| `setupProject(options?)` | Initialize sources, helpers, and optional semantic retrieval. |
| `loadConfig(start?)` | Resolve and validate effective configuration. |
| `knowledgeBaseIdentity(start?)` | Identify the nearest base relative to the outer configured workspace. |
| `discoverKnowledgeBases(start?)` | List root and nested bases and mark the active one. |
| `getKnowledgeBaseContext(cwd?)` | Return bounded identity, readiness, freshness, and capability context. |
| `getKnowledgeBaseSourceCatalog(cwd?)` | Return bounded source coverage with complete totals. |
| `listSourceEntries(cwd?)` | Read configured source entries. |
| `addSourceEntries(options)` | Add source paths or exclusions. |

Expand All @@ -26,6 +30,7 @@ All paths resolve from `cwd` or the current working directory. Retrieval results
| --- | --- |
| `ingest(options?)` | Incrementally parse, redact, chunk, embed, and store files. |
| `audit(cwd?)` | Compare files on disk with the current index. |
| `previewChunks(options?)` | Return redacted structured chunks without writing an index. |
| `search(query, options?)` | Return ranked cited passages. |
| `ask(query, options?)` | Return cited retrieval context without calling an LLM. |
| `research(query, options?)` | Run audit-backed multi-query retrieval. |
Expand All @@ -34,12 +39,19 @@ All paths resolve from `cwd` or the current working directory. Retrieval results
| `compactResearchReport(report)` | Remove verbose evidence text from a research report. |
| `evaluateGoldenQueries(options)` | Score retrieval against a local golden-query file. |

`SearchOptions` accepts `cwd`, `topK`, `contextRadius`, `includePaths`, and `excludePaths`.
`SearchOptions` accepts `cwd`, `topK`, `contextRadius`, `includePaths`, `excludePaths`,
`contextPaths`, and `explain`. When explanation is enabled, each result includes RRF contributions,
one-based vector and lexical ranks, vector distance, lexical backend score, and matched query terms.
`ExpandCitationOptions` accepts `cwd` and a `contextRadius` clamped to three chunks.
Search results expose a `contextPath` derived from Markdown headings or JSON structure. This field
improves retrieval, while `text`, offsets, and citations continue to reference the exact source
passage.

`PreviewReport` includes per-file redaction counts, citations, offsets, structural context, omitted
counts, and chunk distributions. `AuditReport.chunkStats` provides corpus-wide character
distributions and structural-context coverage. Golden-query cases may include `contextPaths` to
evaluate one heading or structured-data branch.

## Operations and privacy

| Export | Use |
Expand Down Expand Up @@ -76,6 +88,10 @@ passage.
`kbCommand` and `ragmirCommand` remain compatibility helpers. New integration code should use
`rgrCommand` and the `rgr` CLI name.

In a monorepo, all public operations still resolve the nearest configured ancestor from `cwd`.
`KnowledgeBaseInventory` exposes deterministic relative IDs, while each base keeps its own source,
storage, manifest, and access-log paths.

The MCP retrieval tools accept an optional `maxBytes` below the configured `mcpMaxOutputBytes`
ceiling. `ragmir_search` and `ragmir_research` also accept `compact`; `ragmir_ask` supports the same
flag. Oversized full output is compacted and then reduced as valid JSON. Output metrics are available
Expand Down
26 changes: 24 additions & 2 deletions docs/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,13 @@ rgr search "release decision"
| `setup [--semantic]` | Initialize `.ragmir/`, agent helpers, and optionally preload embeddings. |
| `init` | Create basic local configuration only. |
| `doctor [--fix]` | Check setup, index freshness, and safe repairs. |
| `preview` | Parse, redact, and chunk selected sources without writing the index. |
| `ingest [--rebuild]` | Index configured sources; rebuild after provider or chunking changes. |
| `search <query>` | Return ranked cited passages. |
| `ask <query>` | Return cited context without model synthesis. |
| `research <query>` | Run an audit-backed multi-query retrieval pass. |
| `audit [--unsupported]` | Compare sources with the index and list skipped files. |
| `bases` | List root and nested monorepo bases and mark the active one. |
| `status` | Show configuration and indexed chunk count. |
| `security-audit [--strict]` | Check local privacy and Git-ignore posture. |

Expand All @@ -29,13 +31,33 @@ rgr search "release decision"
```bash
rgr sources add "docs/**/*.md" "!docs/archive/**"
rgr sources list
rgr preview --path docs --max-files 5 --max-chunks 3
rgr search "migration" --top-k 5 --context-radius 1
rgr search "migration" --include-path docs --exclude-path docs/archive
rgr search "migration" --context-path "Guide > Migration" --explain
```

`sources add` accepts paths, globs, and `!` exclusions. Search, ask, and research accept `--top-k`,
`--include-path`, and `--exclude-path`. Use `--compact` on search or research when agent context is
limited.
`--include-path`, `--exclude-path`, and repeatable `--context-path`. Search and ask accept
`--explain`; the optional score object reports RRF contributions, retriever ranks, raw backend
scores, and matched query terms without changing ranking. Use `--compact` on search or research when
agent context is limited.

`preview` uses the active redaction and chunking configuration but never writes storage. `audit`
reports min, mean, p50, p95, and max chunk sizes plus structural-context coverage.

## Monorepos

```bash
cd apps/web/src
rgr bases --json
rgr search "app-specific contract"
rgr --project-root /absolute/path/to/monorepo search "shared architecture"
```

Commands resolve the nearest configured ancestor. Use the root base for shared or cross-app
knowledge and an app base for app-specific evidence. `--project-root` overrides the working
directory deterministically. Root and nested bases use separate storage and never share index rows.

## Optional local features

Expand Down
9 changes: 8 additions & 1 deletion llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ store.

- Install: `pnpm add -D @jcode.labs/ragmir`
- First use: `pnpm exec rgr setup && pnpm exec rgr ingest`
- CLI: `rgr search "query"`, `rgr doctor`, `rgr audit`, and `rgr security-audit`
- CLI: `rgr preview`, `rgr search "query" --explain`, `rgr bases`, `rgr doctor`, `rgr audit`, and
`rgr security-audit`
- Core behavior: cited retrieval only. `ask` returns evidence, while synthesis belongs to the calling
agent or optional local chat add-on.
- Default retrieval: local-hash, with no model download. Semantic embeddings and local GGUF chat are
Expand All @@ -20,6 +21,12 @@ store.
when the user chooses that workflow, under that provider's data policy.
- OCR boundary: embedded PDF text comes first; a configured local executable runs only for blank
pages. Ragmir has no cloud OCR integration.
- Retrieval inspection: preview redacted chunks without writing storage; optionally expose RRF
contributions and filter by source or structural context.
- Monorepos: the nearest configured base is active, nested indexes remain isolated, and generated MCP
helpers pin a distinct base root.
- MCP resources: `ragmir://context` gives bounded readiness and capability context;
`ragmir://sources` gives capped source detail with complete totals.

## Documentation

Expand Down
10 changes: 10 additions & 0 deletions packages/ragmir-core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,10 @@ npx rgr audit --unsupported
npx rgr security-audit

# Retrieval
npx rgr preview --path docs --max-chunks 3
npx rgr search "deployment decision"
npx rgr search "deployment decision" --explain
npx rgr search "deployment decision" --context-path "Operations > Deployment"
npx rgr ask "What evidence supports the deployment decision?"
npx rgr research "deployment obligations" --compact

Expand Down Expand Up @@ -88,7 +91,10 @@ Frequently used exports:
| Export | Purpose |
| --- | --- |
| `setupProject`, `addSourceEntries` | Initialize project state and select files |
| `discoverKnowledgeBases`, `knowledgeBaseIdentity` | Route root and nested monorepo bases |
| `getKnowledgeBaseContext`, `getKnowledgeBaseSourceCatalog` | Give agents bounded readiness and source context |
| `ingest`, `audit` | Build the index and compare it with files on disk |
| `previewChunks` | Inspect redacted chunks and distributions without writing storage |
| `search`, `ask`, `research`, `expandCitation` | Retrieve or expand cited passages |
| `doctor`, `securityAudit` | Inspect readiness and local privacy posture |
| `serveMcp` | Start the read-focused local MCP server |
Expand All @@ -109,6 +115,10 @@ exposes status, search, ask, research, exact citation expansion, audit, evaluati
security tools. Retrieval responses have a global byte ceiling and expose metadata-only output
metrics. The server does not expose index deletion.

For a monorepo with root and app-specific bases, run `rgr bases --json` before retrieval when scope
is unclear. The nearest configured base is active, nested MCP helpers get unique names, and every
helper pins `RAGMIR_PROJECT_ROOT` so clients cannot drift to a sibling index.

## Retrieval modes

| Mode | Network requirement | Best for |
Expand Down
Loading