Your manifesto makes the accuracy/speed/cost case really well, and the "we want datasets that stress memory in ways the current set doesn't" line is what brought me here.
Here is one axis that AMB, LoCoMo, and LongMemEval all leave uncovered, and that accuracy can't see: integrity under correction. When a fact gets updated, two things a recall benchmark never tests:
- Echo resurrection. After "X is now B", the user later restates the old value ("I think it's A"), not maliciously, just a forgotten preference or a stray transcript line. Does A come back from the dead?
- Value-obscuring revert. The user says "go back to what we had" and names no value. Can the store honor that unmarked command, and should it?
A system can score 90% on retrieval and still fail both. These are behavior properties, not retrieval accuracy.
I put together a small open harness for exactly this: https://github.com/DanceNitra/agent-memory-integrity. It's an adapter interface (~15 lines to add a store) plus a shared, ground-truth-blind judge run on each system's own recall surface, so nothing is scored on a home-field instrument. The mnemo adapter runs out of the box (pip install agora-mnemo + any OpenAI-compatible judge, defaulting to a local Ollama so it needs no API key). The repo also carries the measured cross-system results — mem0 and Graphiti in their own native configs (mem0 on gpt-4o-mini + text-embedding-3-small, Graphiti on a live neo4j) — with the full methodology, though those two backends need their own keys/services to reproduce.
It's deliberately narrow and adversarial, not "which library is best," and it leads with the honest reading: mem0 and Graphiti correctly retain the corrected value but expose no revert operation, so that cell is a capability gap rather than a tuning gap; and I disclose the fairness fix that dropped my own system's headline from a flattering 1.00 to 0.75 once I stopped scoring it on home turf.
If it's useful, I'd be glad to contribute this integrity axis into AMB itself — a PR in your format, as a dataset + scoring cell you can run across the systems you already benchmark. The ground-truth-blind judge is the piece I'd most want a second opinion on. Either way, thank you for putting the multi-dimensional framing out there; the cost axis especially is one more people should be forced to look at.
Your manifesto makes the accuracy/speed/cost case really well, and the "we want datasets that stress memory in ways the current set doesn't" line is what brought me here.
Here is one axis that AMB, LoCoMo, and LongMemEval all leave uncovered, and that accuracy can't see: integrity under correction. When a fact gets updated, two things a recall benchmark never tests:
A system can score 90% on retrieval and still fail both. These are behavior properties, not retrieval accuracy.
I put together a small open harness for exactly this: https://github.com/DanceNitra/agent-memory-integrity. It's an adapter interface (~15 lines to add a store) plus a shared, ground-truth-blind judge run on each system's own recall surface, so nothing is scored on a home-field instrument. The mnemo adapter runs out of the box (
pip install agora-mnemo+ any OpenAI-compatible judge, defaulting to a local Ollama so it needs no API key). The repo also carries the measured cross-system results — mem0 and Graphiti in their own native configs (mem0 on gpt-4o-mini + text-embedding-3-small, Graphiti on a live neo4j) — with the full methodology, though those two backends need their own keys/services to reproduce.It's deliberately narrow and adversarial, not "which library is best," and it leads with the honest reading: mem0 and Graphiti correctly retain the corrected value but expose no revert operation, so that cell is a capability gap rather than a tuning gap; and I disclose the fairness fix that dropped my own system's headline from a flattering 1.00 to 0.75 once I stopped scoring it on home turf.
If it's useful, I'd be glad to contribute this integrity axis into AMB itself — a PR in your format, as a dataset + scoring cell you can run across the systems you already benchmark. The ground-truth-blind judge is the piece I'd most want a second opinion on. Either way, thank you for putting the multi-dimensional framing out there; the cost axis especially is one more people should be forced to look at.