ragcheck

ragcheck is a Go CLI for evaluating retrieval and RAG runs offline.

It gives you fast Precision@k, Recall@k, HitRate@k, MRR@k, MAP@k, nDCG@k, and lightweight answer/context judges without spinning up notebooks or Python-based eval tooling.

Support

Quickstart

Install

brew install itamaker/tap/ragcheck

You can also download binaries from GitHub Releases.

Current release archives:

• macOS (Apple Silicon/arm64): ragcheck_0.2.0_darwin_arm64.tar.gz
• macOS (Intel/x86_64): ragcheck_0.2.0_darwin_amd64.tar.gz
• Linux (arm64): ragcheck_0.2.0_linux_arm64.tar.gz
• Linux (x86_64): ragcheck_0.2.0_linux_amd64.tar.gz

Each archive contains a single executable: ragcheck.

First Run

Run:

ragcheck

This launches the interactive Bubble Tea terminal UI.

You can still use the direct command form:

ragcheck score -qrels examples/qrels.json -run examples/run.json -k 3

Judge answer quality and grounding:

ragcheck judge -input examples/judge.json

Requirements

• Go 1.22+

Run

go run . score -qrels examples/qrels.json -run examples/run.json -k 3

Supported metrics:

• Precision@k
• Recall@k
• HitRate@k
• MRR@k
• MAP@k
• nDCG@k

Build From Source

make build

go build -o dist/ragcheck .

What It Does

1. Loads qrels and retrieval run files from JSON.
2. Matches retrieved document IDs against relevant sets.
3. Computes standard and graded top-k retrieval metrics.
4. Can judge answer relevance, context relevance, groundedness, and reference coverage for RAG outputs.
5. Prints quick offline evaluation summaries for engineering and research loops.

Notes

• Keep qrels and run files aligned on query_id.
• Maintainer release steps live in PUBLISHING.md.

Contributors ✨

Zhaoyang Jia

License

MIT