Cairn

A retrieval pipeline that answers from your documents, cites the exact span it used, abstains with a calibrated guarantee when the evidence is thin, and proves all three with a number.

A cairn is a stack of stones marking a route across ground that offers nothing to follow. It points the way, and it proves somebody walked here before you and checked.

The numbers

Every row is one configuration, measured by one protocol, on the same corpus and the same 120-question golden set. The table is the output of cairn bench with the deterministic stub provider: no API key, no network, and CI reproduces it on every commit.

config	recall@5	nDCG@10	token F1	judge	citation validity	answer rate	error among answered	cost/q	p95 ms
bm25	0.963	0.895	0.319	0.914	1.000	0.500	0.194	0.00000	2
default	0.870	0.846	0.334	0.902	1.000	0.347	0.240	0.00000	2
hybrid-hashing	0.870	0.846	0.334	0.902	1.000	0.347	0.240	0.00000	2
dense-hashing	0.815	0.745	0.340	0.898	1.000	0.472	0.206	0.00000	2

What the table says that a demo would not:

• Lexical search wins on this corpus. BM25 alone finds the supporting passage in the top five for 96 percent of questions; the hashing hybrid manages 87. A hashing embedder is not semantic, and the golden questions reuse the documents' own vocabulary, so fusing in its rankings dilutes BM25 instead of helping it. default is hybrid-hashing under its shipped name.
• The guarantee is stated and met. Every configuration asks for at most 35 percent error among the questions it chooses to answer, and every one lands under it. Why 35 and not 10: with 47 calibration questions the tightest 95 percent upper bound on zero observed errors is about 0.31, so a smaller target is not certifiable at this sample size, and the pipeline refuses to pretend otherwise. The comment at the top of configs/default.yaml has the arithmetic.
• Citation validity is 1.000 because a quote that is not a verbatim substring of the chunk it cites never reaches the reader. The verifier catches it first.

With a real embedder, installed by the local extra (weights download on first use, so CI does not run these; see reports/leaderboard-local.md):

config	recall@5	nDCG@10	token F1	judge	citation validity	answer rate	error among answered	cost/q	p95 ms
dense-local	0.926	0.873	0.000	n/a	0.000	0.000	0.000	0.00000	10
hybrid-local	0.926	0.903	0.000	n/a	0.000	0.000	0.000	0.00000	266
hybrid-local-norerank	0.926	0.892	0.000	n/a	0.000	0.000	0.000	0.00000	11

Semantic retrieval lifts dense recall from 0.815 to 0.926, and the cross encoder gives the best ranking of all at roughly a hundred times the latency. The answer columns are zero, and that is the guarantee working rather than failing: with these confidence signals the calibration slice could not certify the 0.35 bound at any threshold, so the record abstains on everything instead of answering at whatever confidence happened to be highest. A larger calibration slice or a real answering model is what unlocks these rows, and the harness will say so in numbers when it happens.

The adversarial suite asked fourteen questions against six poisoned documents carrying injected instructions and canary strings. All six were retrieved; none leaked. The report is reports/attack-default.md, and cairn attack exits 1 the moment that changes.

Quickstart

pip install "cairn-pipeline[dev]"
cairn init          # writes configs/ and generates data/sample/ from code
cairn ingest        # builds the index and prints its manifest
cairn ask "What was the average fuel use across the fleet in March 2026?"
cairn calibrate     # fits the abstention threshold and states the guarantee
cairn eval          # measures everything; writes reports/latest/default.json
cairn attack        # the poisoned corpus; exit 1 on any leak

Average fuel use across the fleet in March 2026 was 9.4 litres per 100 kilometres.

1. Average fuel use across the fleet in March 2026 was 9.4 litres per 100 kilometres. (verified)
   meeting-minutes-2026-04-15.txt page 1: 'Average fuel use across the fleet in March 2026 was 9.4 litres per 100 kilometres.'

status answered, confidence 0.720, threshold 0.523, 219 ms, 0.00000 USD

Every command defaults to configs/default.yaml. The sample corpus belongs to a fictional food distributor, Halcyon Provisions; cairn init writes it from code and checks it against a pinned digest, so the copy on your machine is the copy the numbers above were measured on. Nothing under data/ is committed or packaged.

Then make it worse on purpose and watch the gate refuse:

cairn eval --out reports/good.json
CAIRN_STUB_DEGRADE=0.5 cairn eval --out reports/degraded.json
cairn gate --baseline reports/good.json --candidate reports/degraded.json

roll back: default against default on judge_score
- judge_score dropped by 0.329 (baseline 0.902, candidate 0.573); the allowed drop is 0.000.
- the candidate's abstention guarantee is not met: error rate among answered is 0.520 against alpha 0.350.

That exit code is the whole idea. The quickstart walks through each step with its real output.

What it does

documents  ->  parse  ->  chunk  ->  embed  ->  hybrid search  ->  rerank
                  |                                                   |
             provenance                                        injection scrub
             (page, span)                                             |
                  |                                          generate with
             versioned index                                 structured citations
                  |                                                   |
                  |                                            confidence score
                  |                                                   |
                  |                              above threshold -> answer and spans
                  |                              below threshold -> abstain and passages
                  v
          evaluation harness  ->  report  ->  promotion gate

A pipeline and a small library, not an application. It ingests a corpus in five formats with page and span provenance, builds a versioned index that refuses to load against a configuration it was not built for, answers with citations or abstains with the closest passages, and produces a report you can compare against last week's. It runs three ways: entirely local with a local model and embedder, against a hosted model, or with a deterministic stub standing in for the model, which is how the tests and the published numbers run.

The three claims

Better than what we had. A frozen golden set, one protocol, and a promotion gate that exits non-zero when a candidate is worse on quality, citations, cost or latency. A change to the chunk size stops being a matter of taste. CI keeps the committed report as the baseline and gates every commit against it.

Not in the documents. The abstention threshold is chosen by conformal risk control: the lowest confidence whose finite-sample Clopper-Pearson bound on the error among answered questions sits at or below the target. The record states the guarantee in a sentence, the report says whether it held, and when no threshold can be certified the pipeline abstains on everything rather than promising what the data cannot support.

Where it came from. Every claim carries a span-level citation: source, page, and the quote itself. The quote must be a verbatim substring of the chunk it cites and must actually support the claim, and a citation that fails either check is marked rather than shown as fact.

And one thing that is not a claim but a stance: retrieved text is data, never instruction. The prompt says so, a scrub removes hidden characters and lines shaped like commands to a model, and a poisoned copy of the corpus is part of the test suite.

Configurations

File	Retrieval	Embedder	Reranker	Answers with	Needs
default.yaml, hybrid-hashing.yaml	hybrid	hashing, 512-d	none	stub	nothing
bm25.yaml	BM25	none	none	stub	nothing
dense-hashing.yaml	dense	hashing, 512-d	none	stub	nothing
dense-local.yaml	dense	all-MiniLM-L6-v2	none	stub	local
hybrid-local-norerank.yaml	hybrid	all-MiniLM-L6-v2	none	stub	local
hybrid-local.yaml	hybrid	all-MiniLM-L6-v2	cross encoder	stub	local
hybrid-local-rewrite.yaml	hybrid, query rewriting	all-MiniLM-L6-v2	cross encoder	stub	local
hybrid-ollama.yaml	hybrid	all-MiniLM-L6-v2	cross encoder	Ollama, llama3.2	local, a running Ollama
hybrid-openai.yaml	hybrid	text-embedding-3-small	none	OpenAI	openai, OPENAI_API_KEY
hybrid-anthropic.yaml	hybrid	all-MiniLM-L6-v2	cross encoder	Anthropic, with prompt caching	anthropic, local, ANTHROPIC_API_KEY

A configuration is one YAML file naming a component for each stage; two files that differ in one line are one experiment. cairn bench runs every file it is given, skips the ones whose extra or key is missing, and writes the leaderboard. Keys come from the environment only; .env.example lists every variable.

Serve

pip install "cairn-pipeline[serve]"
cairn serve --config configs/default.yaml

POST /answer returns the answer with its citations or the abstention with its passages, GET /health reports the index manifest, the calibration in force and a drift signal, and GET /metrics exposes Prometheus counters and histograms per stage. Every request carries an OpenTelemetry span per stage.

Development

git clone https://github.com/rhs2/cairn
cd cairn
make install      # virtualenv with the dev extra
make check        # ruff, mypy, the offline test suite
make bench        # every configuration; writes reports/leaderboard.md
make attack       # the adversarial suite
make docs         # the documentation site, live

CI runs the same three checks on Python 3.10 and 3.12, then generates the corpus, runs the measured pipeline, and gates the new report against the committed baseline. A separate job scans the full history for secrets. Releases are cut from a v* tag and published to PyPI by trusted publishing, so no token is stored anywhere.

What is committed: the code, the configurations, the generator, and the reports under reports/latest/ that CI compares against. What is not: anything under data/, any index or calibration record, any .env.

Documentation

rhs2.github.io/cairn: quickstart, the concepts behind each claim, guides for evaluation, configuration, components and serving, and the reference for the CLI, the configuration file, the golden set and the report.

Licence

Apache-2.0.