kb

A 6MB Go binary that builds and maintains structured wikis from any source. Code, docs, research papers, meeting notes, web pages. Feed it text, it gives you a searchable, interlinked knowledge base.

No embedding model or vector database inside: search is BM25 over the compiled articles. Vectors are optional, and you supply them.

The idea

This is an implementation of Karpathy's LLM Wiki concept. The core insight: instead of retrieving raw document chunks at query time (RAG), have the LLM compile your sources into a persistent wiki that accumulates knowledge over time.

Three layers:

• Raw sources -- your files, immutable, never modified
• The wiki -- LLM-generated articles with summaries, concepts, categories, and cross-references
• The index -- concept graph, backlinks, categories. BM25 search over the compiled articles

Three operations:

• Ingest -- add a source, the LLM reads it, writes a structured article, updates the index
• Search -- query the wiki, not the raw files. 10ms, not 10 seconds
• Lint -- health-check for contradictions, orphan concepts, missing cross-references

The LLM does the tedious work that kills human-maintained wikis: summarizing, cross-referencing, keeping things consistent. You curate sources and ask questions.

Which LLM is up to you. kb ships with a built-in Anthropic client: set ANTHROPIC_API_KEY and kb build compiles your sources. Applications that already run a model (an agent backend, a LiteLLM gateway, Claude Code) can bring their own instead through the compiler extension: a --compiler command, agent mode (kb prepare / kb accept), or kb ingest --article-json. Either way the understanding happens at write time. See Compiling articles.

Not just for code

kb started as a codebase wiki builder, and it's good at that. AST parsing for Go, Python, and TypeScript means the LLM gets structural context alongside raw source.

But the input is text and the output is articles. Anything that produces text can feed kb:

# Codebases
kb build ./src --scope myapp --pattern "*.go,*.py,*.ts"

# Research papers
pdftotext paper.pdf - | kb ingest --scope research --source "arxiv-2401.1234"

# Web pages and documentation
curl -s https://docs.example.com/guide | kb ingest --scope docs --source "setup-guide"

# Meeting notes
cat standup-notes.md | kb ingest --scope team --source "standup-04-07"

# Anything with text
cat slack-export.json | jq -r '.messages[].text' | kb ingest --scope comms --source "slack-general"

Each new source gets compiled into a wiki article, linked to existing concepts, and made searchable. The wiki grows and gets more useful over time.

How it compares to RAG

	kb (LLM wiki)	Traditional RAG
When understanding happens	At write time (once)	At query time (every time)
What you search	Compiled wiki articles	Raw document chunks
Search method	BM25 with concept weighting (~10ms)	Vector similarity + reranking (~200ms)
What you store	Readable markdown files	Opaque vector embeddings
Infrastructure	One 6MB binary	Python + embedding model + vector DB
Incremental updates	SHA256 cache, 0.03s warm rebuild	Re-embed changed documents
Offline	Yes, after first build	Needs embedding model running
Maintenance	LLM handles cross-refs and consistency	You manage chunking and embedding quality

RAG is better for fuzzy semantic matching over huge unstructured text where you can't compile everything upfront. kb is better when you want accumulated, structured knowledge that improves as you add sources.

Numbers

Real codebases, compiled by claude-haiku-4-5-20251001 through the built-in client. Not synthetic benchmarks. With your own compiler, build time and cost depend on the model and endpoint you pick; Compiling articles has measured numbers for each recipe.

Build speed

Source	Type	Files	Cold build	Per file	Warm build
litestream	Go codebase	129	355s	2.75s	0.03s
flask	Python codebase	83	78s	3.25s	0.01s
Small corpus	Go/Py/TS	10	9.5s	1.90s	0.01s

Cold builds run 5 compilations at a time. Warm builds check SHA256 hashes and skip unchanged files.

Search accuracy

Corpus	Queries	Correct first result	Latency
Go examples	6	6/6	9.8ms
Python examples	4	4/4	10.0ms
TypeScript examples	2	2/2	10.0ms
litestream (129 articles)	5	5/5	~11ms
flask (24 articles)	5	5/5	~11ms

22 for 22. BM25 with title weighting (3x) and concept weighting (2x).

Test queries: examples/golden/search_relevance.json

LongMemEval (external benchmark)

LongMemEval is a 500-question benchmark for long-term conversation memory retrieval (ICLR 2025). Each question has ~50 conversation sessions as a haystack and you need to find the one that contains the answer.

We ran kb's BM25 search over raw session text (no wiki compilation, just tokenize and score):

Method	R@1	R@5	R@10	Time	Dependencies
kb BM25	85.0%	95.0%	96.6%	0.5s	zero
MemPalace (raw mode)	—	96.6%	—	minutes	ChromaDB + ONNX + sentence-transformers

95% recall at 5 with nothing but string tokenization and BM25 scoring. The 1.6-point gap to MemPalace is the gap between a full ML pipeline and a single Go binary.

Breakdown by question type:

Type	R@5	Notes
knowledge-update	100%	BM25 handles fact changes well
single-session-user	98.6%	Strong keyword overlap
multi-session	95.5%	Cross-session works via shared entities
temporal-reasoning	95.5%	Date-adjacent keywords help
single-session-assistant	92.9%	Harder — user-only corpus, assistant answers implicit
single-session-preference	76.7%	Semantic gap — "cocktail" vs "gin and tonic"

Reproduce: go test -v -run TestLongMemEval_BM25_Small -timeout 120s ./internal/search (requires downloading the dataset from HuggingFace).

Full benchmark harness and error analysis in benchmarks/longmemeval/.

Raw throughput

Offline, Apple M2 Pro. No model needed to reproduce.

Operation	Speed
Go AST parse	17,586 files/sec
Python parse	4,783 files/sec
TypeScript parse	6,136 files/sec
BM25 search (1K articles)	11ms
BM25 search (5K articles)	55ms
Index rebuild (1K articles)	0.8ms
Content hash (10KB file)	6 microseconds

What comes out

	litestream (129 files)	flask (83 files)
Articles	129	24
Total words	74,931	13,893
Words per article	580 avg	578 avg
Concepts	990	263
Concepts per article	7.6 avg	10.9 avg
Categories	353	112

Each article has a title, summary, body, concepts, categories, and backlinks. Full documents, not fragments.

Browse pre-built output: examples/output/

Install

# Download binary (macOS ARM)
curl -L https://github.com/qbtrix/kb-go/releases/download/v0.1.0/kb-darwin-arm64 -o kb && chmod +x kb

# Build from source
go install github.com/qbtrix/kb-go@latest

# Clone and build
git clone https://github.com/qbtrix/kb-go && cd kb-go && go build -o kb .

Binaries for macOS and Linux (ARM and x86) on the releases page.

For standalone use, set an Anthropic API key:

export ANTHROPIC_API_KEY="sk-..."

Or bring your own model (see Compiling articles):

• A compiler command. --compiler "<command>" (or KB_COMPILER): kb pipes each prompt into the command and reads one JSON article back. Use your Claude Code login, a LiteLLM proxy, LM Studio, Ollama, or anything else.
• Your agent. Running inside Claude Code, Cursor or Codex? Use agent mode: kb prepare hands out the prompts, the agent writes the articles, kb accept stores them. No API key.

Search, show, list, stats, structural lint, glossary and kb serve never need a model.

Quick start

# Built-in client (or set KB_COMPILER to bring your own, see below)
export ANTHROPIC_API_KEY="sk-..."

# Build a wiki from a codebase
kb build ./src --scope myapp --pattern "*.go,*.py,*.ts"

# Search it
kb search "auth middleware" --scope myapp

# Rebuild (only changed files recompile)
kb build ./src --scope myapp --pattern "*.go"

# Export the wiki as markdown
kb build ./src --scope myapp --output docs/wiki/

# Watch for changes
kb watch ./src --scope myapp --pattern "*.go"

# See what compiling cost (tokens per article)
kb stats --scope myapp

Compiling articles

kb builds the compile prompt; an LLM turns it into an article. There are four ways to supply that LLM.

Choosing a path

Path	How	Use it when
Built-in client	export ANTHROPIC_API_KEY=..., then kb build (--model picks the model)	You run kb on its own: a laptop, CI, a cron job
Built-in client through a proxy	also export ANTHROPIC_BASE_URL=http://localhost:4000	Spend must go through a LiteLLM (or other) gateway with budgets and logs
Your own compiler	--compiler "<command>" or KB_COMPILER	You want another model or provider, or your Claude Code login instead of an API key
Agent mode	kb prepare → your agent compiles → kb accept	kb runs as a skill inside Claude Code, Cursor, Codex
Article JSON	kb ingest --article-json	An application (PocketPaw, a LiteLLM-based app) compiles one document itself and hands kb the result

For the commands that compile (build, ingest, recompile, watch, lint --llm), kb picks the path in this order:

1. --compiler "<command>"
2. KB_COMPILER
3. the built-in client, when ANTHROPIC_API_KEY is set
4. none: the command exits 2 and lists these options

--model only applies to the built-in client. Passing it with a compiler (flag or KB_COMPILER) is a usage error, exit 2: the compiler picks its own model. Agent mode and --article-json never call a model from kb, so they need no key.

The built-in client

kb calls the Anthropic Messages API directly (no SDK). The default model is claude-haiku-4-5-20251001; --model overrides it. ANTHROPIC_BASE_URL follows the Anthropic SDK convention: it replaces the root (default https://api.anthropic.com) and kb posts to <base>/v1/messages. Point it at a LiteLLM proxy and every compile is metered there:

export ANTHROPIC_BASE_URL=http://localhost:4000   # LiteLLM proxy with an Anthropic-format route
export ANTHROPIC_API_KEY=sk-litellm-virtual-key
kb build ./src --scope myapp --pattern "*.go"

Each article records the tokens the API reported (see Usage metadata); compiled_with is the model name. kb does not compute a cost for the built-in path: it carries no price table, and a stale one would report wrong numbers. Use your gateway's spend logs, or a compiler that reports cost_usd.

Bring your own compiler: the contract

--compiler "<command>" (or the KB_COMPILER environment variable; the flag wins) is used by build, ingest, recompile, watch and lint --llm, and wins over the built-in client. For each item kb:

1. runs the command through the platform shell: sh -c "<command>" on Linux/macOS, cmd /S /C "<command>" on Windows. Write the command exactly as you would type it in that shell. On Windows, " quotes work and ' does not; run a POSIX script there through bash script.sh, or use python script.py;
2. writes the prompt to the command's stdin;
3. reads stdout and parses one JSON object: title, summary, content, concepts, categories, plus optional compiled_with and usage. Markdown code fences and a stray line around the object are tolerated.

The command's stderr is passed through, prefixed with [compiler <source>]. Each run has a timeout (--compiler-timeout, default 300s; accepts 90s, 2m or plain seconds); on timeout kb kills the command's whole process tree. --concurrency N (default 5) runs that many at once during build.

A non-zero exit, a timeout, or output without a title and content fails that item loudly: an Error: line on stderr naming the source, no article, no cache entry (the next build retries it), and exit code 1 once the rest of the build is saved. kb never stores the raw text as the article in its place. The only verbatim path is the explicit kb ingest --allow-fallback. The built-in client fails the same way on a network error, a non-200 response, or a reply that isn't one article.

Glossary files and unchanged (cached) files need no LLM, so kb build only asks for a key or a compiler when something actually needs compiling.

Usage metadata

The built-in client records model, input_tokens and output_tokens from every API response. A compiler, kb accept, or kb ingest --article-json can report the same, plus a cost:

{"title": "...", "summary": "...", "content": "...", "concepts": [], "categories": [],
 "usage": {"model": "claude-haiku-4-5-20251001", "input_tokens": 931, "output_tokens": 526, "cost_usd": 0.0036}}

Every field is optional and unknown keys are ignored. kb stores it in the article frontmatter as usage, uses usage.model as compiled_with when you didn't send one (a compiler without either is recorded as compiler:<first word of the command>), and replaces it when the article is recompiled or re-accepted. kb stats adds the totals when any article has usage:

  Compile usage (3 articles): 2364 input + 1295 output tokens, $0.0088

kb stats --json carries the same numbers under "usage": {"articles", "input_tokens", "output_tokens", "cost_usd"}. kb build --json keeps its input_tokens / output_tokens / total_tokens fields, summed from the built-in client's or the compiler's reports, and adds failed.

Recipe: headless Claude Code (your Claude login, no API key)

For one person on their own machine; it bills to your Claude plan. The flags strip everything a compile doesn't need: no tools, no MCP servers, no CLAUDE.md, skills or user settings (which can switch on an advisor model), no extended thinking, no saved session.

kb build ./docs --scope docs --pattern "*.md" --terse --compiler 'claude -p --safe-mode --setting-sources "" --strict-mcp-config --tools "" --settings "{\"alwaysThinkingEnabled\":false}" --no-session-persistence --model haiku --system-prompt "You are a knowledge compiler. Output only valid JSON. No markdown fences."'

The inner command (everything inside the single quotes) is shell-neutral: it runs unchanged under sh -c and cmd /S /C. Tested on Windows with Claude Code 2.1.289.

That one-liner records compiled_with: compiler:claude and no usage. For per-article tokens and cost, use the wrapper, which runs the same flags with --output-format json and maps Claude Code's usage block (input includes cache reads and writes) into usage:

kb build ./docs --scope docs --pattern "*.md" --terse --compiler "python examples/compilers/claude_code.py"
# KB_COMPILE_MODEL=sonnet to change the model; CLAUDE_BIN to point at another claude binary

Measured on three small C4 docs (0.3 to 2.5 KB) with --terse and Haiku 4.5, one at a time: 488 to 945 input tokens, 361 to 526 output tokens, about 6 s and $0.002 to $0.004 per doc. The same three in parallel (default concurrency) took 7.5 s. Drop --safe-mode --setting-sources "" and a call made inside a repo picks up its CLAUDE.md (about 8K extra input tokens per call in our workspace) plus any advisor model from your settings (one call went from $0.003 to $0.19).

Recipe: LiteLLM proxy or any OpenAI-compatible endpoint

examples/compilers/openai_compatible.py (Python 3.8+, standard library only) posts the prompt to $OPENAI_BASE_URL/chat/completions and prints the article with usage filled from the response. Behind a LiteLLM proxy, cost_usd comes from its x-litellm-response-cost header; elsewhere set KB_COMPILE_COST_IN / KB_COMPILE_COST_OUT (USD per million tokens) if you want a cost.

export OPENAI_BASE_URL=http://localhost:4000/v1     # your LiteLLM proxy
export OPENAI_API_KEY=sk-litellm-virtual-key        # optional bearer token
export KB_COMPILE_MODEL=claude-haiku                 # a model name the proxy knows
kb build ./src --scope myapp --pattern "*.go" --compiler "python examples/compilers/openai_compatible.py"

Tested against LM Studio's OpenAI-compatible server (OPENAI_BASE_URL=http://localhost:1234/v1, KB_COMPILE_MODEL=zai-org/glm-4.6v-flash on a local GPU): three docs compiled, about 26 s per doc, 1,454 input and 3,323 output tokens in total (the output count includes the model's reasoning), no cost reported. The bearer token and the LiteLLM cost header were checked against a stub proxy.

Recipe: Ollama

The same script, pointed at Ollama's OpenAI-compatible endpoint (untested here):

export OPENAI_BASE_URL=http://localhost:11434/v1 KB_COMPILE_MODEL=llama3.1:8b
kb build ./src --scope myapp --pattern "*.py" --compiler "python examples/compilers/openai_compatible.py"

Small local models often break the JSON contract. kb fails those items loudly and retries them on the next build, so watch stderr.

Recipe: inside a Claude Code (or any agent) conversation

No compiler at all: the agent you are already talking to writes the articles. See Agent mode explained.

kb prepare ./src --scope myapp --pattern "*.go" --terse > prompts.json   # prompts, nothing compiled
# the agent answers each item's "prompt" with the article JSON
echo '{"scope":"myapp","articles":[...]}' | kb accept --scope myapp       # optional "usage" per article

Use with AI agents

kb works in two modes: search an existing wiki, or build the wiki using the agent's own LLM (no API key or compiler needed).

Claude Code

Option A: Install as a skill (recommended)

npx skills add qbtrix/kb-go

Then ask Claude Code to "build a knowledge base from ./src" and it'll follow the skill instructions.

Option B: Add to CLAUDE.md

Drop this in your project's CLAUDE.md:

## Knowledge base

This project has a knowledge base. Use it before reading raw files.

### Search (existing wiki)
\`\`\`bash
kb search "<topic>" --scope myapp --context
\`\`\`

### Build wiki (agent mode)
To build or update the wiki, use agent mode:
1. Run `kb prepare ./src --scope myapp --pattern "*.go,*.py,*.ts" --json`
2. For each item in the output, process the `prompt` field yourself
3. Collect results and pipe to: `echo '<json>' | kb accept --scope myapp`

The accept input format:
\`\`\`json
{"scope":"myapp","articles":[{"source":"file.go","hash":"from prepare","raw_id":"from prepare","title":"...","summary":"...","content":"...","concepts":["..."],"categories":["..."]}]}
\`\`\`

### Other commands
- `kb show <article-id> --scope myapp` — full article
- `kb stats --scope myapp` — overview
- `kb lint --scope myapp` — health check

Claude Code loads CLAUDE.md at session start, so it picks up the kb commands automatically.

OpenAI Codex

Add to AGENTS.md in your project root:

## Knowledge base

This project uses `kb` for structured knowledge. Binary must be on PATH.

### Search
\`\`\`bash
kb search "<topic>" --scope myapp --context
\`\`\`

### Build (agent mode)
1. `kb prepare ./src --scope myapp --pattern "*.py" --json` — get compilation prompts
2. Process each item's `prompt` field, output JSON with: title, summary, content, concepts, categories
3. Pipe results: `echo '<json>' | kb accept --scope myapp`

Cursor

Add to .cursorrules:

This project has a knowledge base. Before searching files, run:
  kb search "<topic>" --scope myapp --context
To build/update the wiki with your own model, use agent mode:
  kb prepare ./src --scope myapp --pattern "*.go" --json
  Then compile each prompt and pipe results to: kb accept --scope myapp

Any other agent

If the agent can run shell commands, the same pattern applies. kb search "topic" --scope myapp --context returns formatted text you can inject into prompts. kb prepare + kb accept handles builds. Add --json to any command for machine-readable output.

Agent mode explained

If kb is running inside an agent, the agent already has LLM access. Agent mode lets it do the compiling by splitting the build into two steps:

# Step 1: Scan files, check cache, output prompts (no LLM call)
kb prepare ./src --scope myapp --pattern "*.go,*.py,*.ts"
# Returns: {"items": [{"source": "main.go", "hash": "...", "raw_id": "...", "prompt": "..."}], ...}

# Step 2: Agent compiles each prompt using its own LLM
# (This is the part where YOUR agent does the work)

# Step 3: Feed compiled results back
echo '<json>' | kb accept --scope myapp
# Returns: {"accepted": 5, "articles": 12, "concepts": 47}

accept reads JSON from stdin. It accepts a wrapped object {"scope":"...","articles":[...]}, a bare array [{...}], or a single article {...}.

Each article needs: source, hash, raw_id (from prepare output), plus title, summary, content, concepts, categories (from your LLM compilation). Optional: compiled_with and usage (see Usage metadata).

Cache works across both modes. Run prepare again and unchanged files are skipped.

Programmatic access

--context returns formatted text for prompt injection:

CONTEXT=$(kb search "authentication flow" --scope myapp --context)

Each hit becomes a ## <Title> block, and blocks are joined by a --- line. An article that fits the per-article budget comes back whole. A longer one is cut down to the sections that match the query (kept in document order, with their headings, … where text was skipped). Tables are never split mid-row. Budgets are in bytes:

• --context-chars N: per-article budget (default 4000)
• --context-total N: total budget (default 8000). The top hit is trimmed to fit, never dropped.

Article bodies can contain their own --- rules, so splitting the text output on the separator is fragile. Add --json to get the same excerpts as an array:

kb search "shoe sizes" --scope shop --context --json
# [{"id": "size-guide", "title": "Size Guide", "text": "<excerpt, no title line>", "truncated": false}]

--json on every command for machine consumption:

kb search "auth" --scope myapp --json
kb stats --scope myapp --json
kb show auth-service --scope myapp --json

Concept graph export

Every wiki already has a concept graph built in — articles are connected via shared concepts. kb graph exports that graph in portable formats so you can visualize it with whatever tool you want.

# Overview: top N concepts by article count, edges for co-occurring concepts
kb graph --scope myapp --limit 30

# Focus on one concept (one-hop neighborhood)
kb graph --scope myapp --concept "authentication"

# Focus on one article and its concepts
kb graph --scope myapp --article auth-service

# Other formats
kb graph --scope myapp --format dot | dot -Tpng > graph.png
kb graph --scope myapp --format json > graph.json

Mermaid output works directly in GitHub READMEs, Obsidian, Notion, and anything that renders Mermaid. DOT output pipes into Graphviz. JSON is raw node/edge data for custom tooling.

Flags:

• --format mermaid|dot|json — output format (default mermaid)
• --concept <name> — one-hop subgraph around a concept
• --article <id> — article and all its concepts
• --limit N — cap concepts in overview (default 30)
• --min-articles N — only include concepts appearing in ≥N articles (default 2)

Domain glossary

Every project carries vocabulary that LLMs misread by default — a "Pocket" that is not clothing, a "Soul" that is not religious, a "Fabric" that is not textile. Burying those definitions in chat-time prompts doesn't scale past five or six terms. kb-go hosts the full glossary as a first-class article type that round-trips through the wiki without LLM rewriting.

Placement

Drop hand-curated markdown files in any directory whose immediate parent is named glossary. kb build recognizes the convention and skips LLM compilation for those files.

docs/
  wiki/
    glossary/
      pocket.md
      soul.md
      fabric.md
    ...
src/
  ... (your code)

Frontmatter shape

---
{"id":"pocket","title":"Pocket","kind":"glossary","term":"Pocket","aliases":["pkt","pocket"],"category":"workspace-primitives","related":["Soul","Fabric"]}
---

A Pocket is a workspace container that holds agents, data, tools, connectors,
and rules. The "app" primitive of the agentic OS.

**Not:** clothing.
**Related:** Soul, Fabric.

Glossary-only fields:

Field	Type	Purpose
kind	string	"glossary" marks this article as hand-curated (auto-set when parent dir is glossary/)
term	string	Canonical name surfaced in list, matched in show, boosted in search
aliases	string[]	Alternative names — all also match in show and search
category	string	Bucket label (e.g. workspace-primitives); shown in list
related	string[]	Cross-references to other terms or aliases; validated by glossary validate

The body below the frontmatter is preserved byte-for-byte. No LLM call.

Commands

# Enumerate the glossary
kb glossary list --scope myproject

# TERM     CATEGORY              ALIASES        FILE
# Pocket   workspace-primitives  pkt,pocket     pocket.md
# Soul     identity              spirit         soul.md
# Fabric   ontology              -              fabric.md

# Print one entry (matches Term or any Alias, case-insensitive)
kb glossary show Pocket --scope myproject
kb glossary show pkt    --scope myproject   # alias works
kb glossary show POCKET --scope myproject   # case-insensitive

# Structural and semantic checks
kb glossary validate --scope myproject
# Reports: duplicate terms, duplicate aliases, alias↔term collisions,
# dangling `related` references, and cross-source contradictions.
# Exit code 2 when issues are found.

Cross-source contradictions

Structural checks catch a term defined twice. They don't catch the worse case: two sources that both define the same term but disagree on what it means. That's the failure mode the glossary exists to prevent — a "Soul" that one source treats as spiritual and another as the agent-identity layer. Left alone, the build picks a winner by save order and the loser disappears with no warning.

kb build now scans glossary definitions for these conflicts and flags them instead of merging silently:

kb build ./docs --scope myproject --pattern "*.md"
# ...
# 1 glossary contradiction(s) — sources disagree on a definition:
#   contradiction: term "Soul" defined differently across sources [...]

The tool flags, it does not resolve. A human decides which definition is canonical. Behavior:

• Detected during kb build and re-listed by kb glossary validate.
• kb build exits 3 when contradictions are found (CI-gateable); --json emits a contradictions array instead of exiting non-zero.
• Threshold is tunable via --contradiction-mode:
  • strict (default) — flags when the definitions' first sentences differ.
  • loose — flags only when the full definition bodies differ.
  • off — skip the scan.
• Fully offline. No LLM, no network. An optional LLM-assisted similarity pass can layer on later as a flag, never a hard dependency.

Caveat — this compares wording, not meaning. The scan keys on the normalized text (first sentence in strict, full body in loose), so it measures whether two definitions are worded differently, not whether they actually disagree. Two sources that mean the same thing but phrase it differently — a paraphrase, a reordered clause, a synonym — will be flagged as a contradiction even though they agree. Because kb build exits 3 on a finding, a paraphrase-only divergence can fail a CI gate. If strict is too aggressive for your build, drop to loose (which only fires when the whole body differs) or off, and lean on kb glossary validate for a non-gating review instead.

Search ranking

Query and article text are lowercased and stemmed the same way before scoring. The stemmer only undoes inflection (step 1 of the Porter stemmer: plurals and -ed/-ing), so "open" matches "opens" and "opening", and "returning" matches "returns", while "general" and "generating" stay distinct.

kb search applies a 10× boost on top of the normal BM25 score whenever a query token matches a glossary article's Term or any Alias (case-insensitive, after the same stemming, so the plural Term "Connectors" fires on "connector"). Hand-curated definitions outrank module articles that merely mention the term in passing.

Pairing with Soul Protocol

kb and Soul Protocol solve different halves of the same problem. Wire them together in your agent pipeline and you get something most AI tools don't have: memory of what happened plus a structured view of what exists now.

The split

	kb	soul
What it stores	Articles compiled from sources	Episodic, semantic, procedural memory tied to an agent's identity
Where content comes from	Files (code, docs, text)	Conversations, decisions, experiences
Regenerable?	Yes — re-run kb build	No — memory is earned, not recomputed
Decays?	Never	Yes — via ACT-R activation and significance gating
Search	BM25 over compiled articles	Relevance + recency + importance weighting
Portable format	Markdown + JSON frontmatter	.soul ZIP file

kb answers "how does X work?" from the current codebase. soul answers "what did we decide about X, and why?" from accumulated conversations.

When to use which

• Ask kb when the answer is derivable from source files: implementation details, architecture, API shapes, which module does what
• Ask soul when the answer requires remembering a conversation: past decisions, user preferences, relationship context, why something was built a certain way
• Ask both when you want the full picture: why was this built (soul) AND how does it work now (kb)

The integration pattern

Keep both as standalone tools. Bridge them in your agent pipeline, not in their code. The cleanest place is wherever your agent builds its system prompt — inject soul-recalled memories alongside kb-searched articles in the same context-building step.

Rough shape (pseudocode):

# In your agent's context builder, run both queries in parallel
soul_memories = await soul.recall(query, limit=5)
kb_articles = subprocess.run(["kb", "search", query, "--scope", project, "--context"])

# Merge into the system prompt with clear framing
system_prompt += f"""
## What we've discussed before (from memory)
{format_soul_memories(soul_memories)}

## What the codebase currently says (from knowledge base)
{kb_articles.stdout}
"""

For example, in any agent framework with a context-builder step — PocketPaw's src/pocketpaw/bootstrap/context_builder.py is one such place — drop the kb lookup alongside existing memory/soul providers. One extra step in the pipeline, no new CLI, no tight coupling.

Workflow patterns

1. Pipe soul memories into kb

Make episodic memories searchable alongside code articles. Useful when you want BM25 search to surface past conversations by concept, not just by date.

# Pull recent important memories from soul, compile into kb as structured articles
soul recall .soul/my-agent.soul --recent 20 --min-importance 7 --format text \
  | kb ingest --scope my-sessions --source "soul-$(date +%Y-%m-%d)"

Now kb search "authentication decisions" returns both "this is how auth works now" (code article) and "here's when we decided JWT and why" (compiled soul memory).

2. Soul references kb articles

Instead of storing long facts in soul, store short memories with kb pointers. Soul stays lean, the agent still has access to the rich structured knowledge.

# Soul stores the decision, kb has the details
soul remember .soul/my-agent.soul "Decided JWT auth. Details: kb://myapp/auth-core" --importance 8

On recall, your agent sees the memory, follows the kb:// pointer, and runs kb show auth-core --scope myapp to get the current state.

3. Unified context injection

Both queries run on every agent request. The agent gets what was decided (soul) plus what the code currently looks like (kb). This is the pattern that matters most — it's the difference between an agent that feels like a coworker versus one that has to be re-briefed every session.

Why not merge them into one tool?

Different failure modes. kb needs to rebuild from source whenever code changes — it's a cache of your files. Soul can't be rebuilt — losing a .soul file means losing experiences that only happened once. Keeping them separate means each one can be backed up, versioned, shared, or deleted independently. The 6MB binary stays a 6MB binary. The .soul file stays portable across platforms.

The bridge is thin by design. Any agent pipeline can wire them together in ~20 lines.

Commands

Command	Description
kb build <path>	Scan, parse AST, compile pending files with the LLM (built-in client or --compiler), build wiki. Exits 2 with neither when something needs compiling, 1 if any file failed to compile
kb prepare <path>	Output compilation prompts as JSON (agent mode)
kb accept	Read compiled articles from stdin (agent mode)
kb graph	Export concept graph (mermaid, dot, or json)
kb search <query>	BM25 search over articles
kb ingest [file]	Ingest a file or piped stdin, compiled with the LLM (built-in client or --compiler). Fails (exit 1) if the compile fails — the raw doc is kept, no article is written. --allow-fallback stores the raw text verbatim instead (no LLM needed). --article-json reads {"raw_text", "article"} from stdin so the caller supplies the compiled article. Exits 2 with none of these
kb show <id>	Print a full article
kb list	List all articles
kb stats	Counts for articles, concepts, words, plus compile usage totals when reported
kb lint	Structural checks; --llm for an LLM review (built-in client or --compiler); --normalize-categories [--apply] to collapse variant labels ("CLI"/"cli") into one
kb recompile <id>	Recompile from raw source (--all for everything)
kb watch <path>	Auto-rebuild on file changes
kb convo ingest <file>	Parse a conversation transcript, extract entities/decisions/topics, create wiki articles
kb convo search <query>	Search conversation articles
kb convo list	List conversation articles
kb glossary list	List hand-curated glossary entries in the scope
kb glossary show <term>	Print a glossary entry's body (matches Term or Alias, case-insensitive)
kb glossary validate	Check for duplicate terms, alias collisions, dangling references, and cross-source contradictions
kb clear	Wipe all data for a scope
kb version	Module version (go install …@vX), or dev plus the VCS revision for local builds

Flags

Flag	Default	Description
--scope	default	Scope name, for multi-tenant use
--json	off	JSON output
--model	claude-haiku-4-5-20251001	Model for the built-in client. Exit 2 if combined with a compiler
--compiler	$KB_COMPILER	Bring your own compiler: a command that compiles one article (prompt on stdin, JSON article on stdout). Wins over the built-in client. See Compiling articles
--compiler-timeout	300s	Per-item compiler timeout (90s, 2m, or seconds)
--pattern	*.py	File patterns, comma-separated
--concurrency	5	Parallel compilations
--output		Export wiki to directory
--contradiction-mode	strict	Glossary contradiction threshold on kb build: strict, loose, or off
--lang	auto	Language for stdin ingest
--allow-fallback	off	On kb ingest: store the raw text verbatim as an article when there is no key or compiler, or the compile fails
--article-json	off	On kb ingest: read {"raw_text": "...", "article": {"title", "summary", "content", "concepts", "categories", "source", "compiled_with", "usage"}} from stdin; kb saves raw doc + article without calling an LLM

AST parsing

Source files are parsed for structure before the LLM sees them. The compilation prompt includes both the raw code and a structural summary, which tends to produce better articles.

Language	Parser	Extracts
Go	go/parser (stdlib)	packages, structs, interfaces, methods, functions, constants
Python	regex	classes, methods, async markers, imports, docstrings, constants
TypeScript/JS	regex	classes, interfaces, enums, types, functions, arrow functions

How it works

Source (any text)
    ↓
AST parse if code (Go: go/ast, Python: regex, TS: regex)
    ↓
LLM compile: built-in Anthropic client or your --compiler command (5 concurrent),
             or your agent via prepare/accept
    ↓
Wiki article (markdown + JSON frontmatter)
    ↓
BM25 inverted index (versioned postings: term → (doc, tf), weighted by title and concepts)
    ↓
~/.knowledge-base/{scope}/
├── raw/       original source, kept for recompilation
├── wiki/      compiled articles as .md files
├── cache/     SHA256 hashes + inverted search index (versioned; old-format files are ignored and rebuilt on the next search or index write)
├── index.json concept graph, backlinks, categories
└── vectors.bin optional embeddings (binary; a vectors.json from kb v0.4.0 and
               earlier is converted on the first read and kept until the next
               vector write, so an older kb still finds it after a downgrade)

Articles are plain markdown with JSON frontmatter. cat them, grep them, commit them.

Architecture

A thin root main.go calls internal/cli (dispatch, one cmd_<name>.go per command); the library is one folder package per concern under internal/: store (storage), search (index, BM25, context excerpts, vector/hybrid), compile (the built-in Anthropic client and the --compiler hook), parse, lint, export, glossary, contradiction, convo (conversation mode) and mcp (the MCP server). Packages are layered and import only lower layers. One dependency: fsnotify for watch mode.

Component	~Lines	What it does
Data models	100	RawDoc, WikiArticle, Concept, KnowledgeIndex, Cache
AST parsers	400	Go (stdlib), Python (regex), TypeScript (regex)
Storage	200	File CRUD, markdown with JSON frontmatter
BM25 search	100	Weighted scoring from an inverted postings index
Compilation	450	Prompt building, the built-in Anthropic client (direct HTTP, no SDK), and the --compiler hook (platform shell, timeout, one-JSON-article parsing, usage)
Lint	150	Structural checks and an LLM review through the hook
CLI	400	Commands, flags, JSON output
Watch	80	fsnotify, 3s debounce

Testing

69 unit tests, 13 performance benchmarks. No test dependencies beyond Go stdlib.

go test -v ./...       # Unit tests
go test -bench=. ./... # Benchmarks
./bench.sh small       # Full pipeline (needs ANTHROPIC_API_KEY or KB_COMPILER)

Coverage

Area	Tests	What it checks
Storage	6	Round-trip for articles, raw docs, index; frontmatter parsing
Search	5	Ranking, empty inputs, limits
Cache	3	Hits, misses, persistence
AST parsers	6	Go, Python, TypeScript extraction
Lint	3	Empty KB, missing concepts, broken backlinks
Scanning	3	Patterns, skip lists, multi-pattern
Helpers	8	Slugify, tokenize, hashing, word count
Compat	1	Python format round-trip
Benchmarks	10	All core operations at multiple scales

Reproduce the benchmarks

go test -bench=. -benchmem ./...              # Offline, seconds
./bench.sh small                              # 10 files, ~30s
./examples/fetch.sh all && ./bench.sh medium  # 129 files, ~6 min
cat bench_results.json

Examples

examples/
├── small/              10 source files (Go, Python, TypeScript)
│   ├── go/             server, handler, middleware, models, config
│   ├── python/         service, models, utils
│   └── typescript/     api, types
├── golden/             Expected search results and concept extraction
├── output/             Pre-built wiki (browse without compiling anything)
├── fetch.sh            Download litestream + flask for benchmarking
└── README.md

Source files · Expected results · Compiled output

Inspired by

Karpathy's LLM Wiki -- the idea that LLMs should maintain persistent, structured knowledge instead of re-deriving it from raw sources on every query. kb is that concept as a CLI tool.

License

MIT