Status: v0.4, in active development. Interfaces still move between releases.
Memrank is a tool for reproducible, auditable evaluation of memory systems.
Compare memory systems and their versions on tasks that matter to your agent or application. You choose the tasks and success criteria; Memrank runs the evaluation, reports scores and timings, and records responses and errors so you can investigate differences.
Start here | Compare memory systems and their versions | Understand results
Check the installation with a small local retrieval evaluation. Installation needs the network; the Python block below needs no key, service or network. This checks that Memrank works in your project; it does not establish how a memory system will perform on your workload.
Use Python >= 3.10 and an existing project or virtualenv. Installation instructions cover creating one.
uv add memrank # or, into a virtualenv you already have: pip install memrankfrom memrank.evaluations import SQuAD
from memrank.systems import TFIDF
evaluation = SQuAD()
result = evaluation.run(system=TFIDF())
print(result)TFIDF is keyword search
weighted by how rare each word is.
SQuAD supplies
32 bundled passages and 64 questions. The squad-score measures full-passage retrieval recall,
not answer-span or end-to-end answer correctness. Check that the output names the expected system
and evaluation and says 64 recorded, 0 with errors.
Read the output.
Or paste this to your coding agent:
Install memrank in this project and run its smoke evaluation, following
https://github.com/atomicstrata/memrank/blob/main/docs/install.md. Check the prerequisites
that page lists before you change anything, install into this project only, and do not
install anything globally or edit my shell configuration. When the run finishes, show me the
`system:`, `evaluation:` and `traces:` lines it printed. Stop and ask me if any step fails.
Run candidates on the same evaluation, inspect coverage and failures, and read each measure's
meaning before interpreting a gap. The memory systems comparison guide
shows how to read the differences, with an offline toy example of task-level pairing.
It also explains when to compare summaries: memrank.paired does not compare aggregate
scores such as squad-score.
Use a shipped client
or connect your own system.
A memory implements prepare, ingest, retrieve and cleanup so Memrank can give it context,
ask questions and clear state between independent cases. An external
engine
may need a configured service and credentials.
Express your evaluation as tasks, expected outcomes and measures. You decide which cases represent your problem and what counts as success; Memrank applies those rules and records the evidence.
Understand results shows how to inspect a task's trace, distinguish missing values from zero, and save a result for later use. Write a measure to read something new from stored traces without rerunning the system.
Control examples show what happens when retrieval returns no documents or all documents. These checks help expose what a measure rewards; they do not prove that the evaluation represents your workload. Methodology states the measurement rules and limits on claims.
| Guide | Task |
|---|---|
| Start here | Start using Memrank |
| Memory systems comparison guide | Compare memory systems or their versions |
| Understand results | Interpret and save measurements |
| Systems | Find available integrations |
| Evaluations | Find available task sets |
| Reference | Look up Python contracts |
| Documentation index | Find every guide |
The guides above use Python. Memrank also provides a command line for tracked and placed runs, and a translator contract for memory systems implemented in other languages.
Contributing and getting help links the issue tracker and development checks. Questions about your setup are easier to reproduce with the package version, a small example and the error text. Do not include keys or private data.
Memrank is maintained by AtomicStrata under a vendor-neutral charter: anyone may submit a system, results are published as measured, and methodology changes go through public proposal and comment. The commitments are in SPEC.md section 7. AtomicStrata also develops AtomicMemory, one of the engines Memrank can evaluate. Comparisons should be assessed through their method, configuration and recorded evidence.
Memrank's code is Apache-2.0. The bundled SQuAD subset is CC BY-SA 4.0; its notice credits the creators and passage sources and records the selection and reformatting.