An autonomous research agent that has to show its evidence.
You give it a research question. It searches academic databases and code hosts, forms hypotheses, hunts for evidence, reads actual source files to check whether a mechanism is really implemented, and writes a report — where every claim carries an evidence class it had to earn.
The name is from the Royal Society's motto, nullius in verba: take nobody's word for it. Including the agent's own.
┌─────────────┐
research question ─────────────▶│ Initializer │ plans queries + sources
└──────┬──────┘
▼
┌──────────────────────────────────────────────────────────┐
│ Retrieval → Generation → Reflection → Verification │
│ ▲ │ │
│ └──────── new evidence makes ──────────┘ │
│ old conclusions stale │
└──────────────────────────────────────────────────────────┘
▼
┌─────────────┐
│ MetaReview │ report + verdict
└─────────────┘
Most research agents will happily tell you they verified something. This one is built so that it cannot — unless it can point at the file, the line, and the symbol.
The distinction it enforces everywhere:
"we did not find it" is not "it is not there."
That sounds obvious. It was violated in four different subsystems of this codebase, and each time it looked like a completely different bug. The whole story is below.
- A web UI to create runs, watch agents work live, and read reports
- Evidence classes on every claim, from
HYPOTHESISup toREPO_ARTIFACT_VERIFIED - Code inspection that opens real repository files and quotes the matching line
- A hard budget ceiling — runs stop at a dollar amount you set, enforced twice
- Durable everything — every run, task, source, claim and verdict is a database row, so you can audit what actually happened rather than what the agent says happened
- Node.js 20+
- pnpm 9 —
corepack enable && corepack prepare pnpm@9.12.0 --activate - Docker — for PostgreSQL and Redis
- An API key for DeepSeek or Anthropic
git clone https://github.com/krapcys1-maker/nullius.git
cd nullius
pnpm installcp .env.example .envOpen .env and fill in, at minimum:
DATABASE_URL=postgresql://postgres:postgres@localhost:55432/researchos
REDIS_URL=redis://localhost:56379
NEXTAUTH_SECRET= # any long random string
LLM_PROVIDER=deepseek # or "anthropic"
DEEPSEEK_API_KEY= # your keyTwo optional ones that matter more than they look:
GITHUB_TOKEN= # unauthenticated GitHub allows 60 API calls/hour,
# which one code inspection exhausts. A token with
# no scopes at all raises this to 5000/hour.
OPENALEX_EMAIL= # gets you into OpenAlex's polite pool — faster,
# more reliable, and it is the courteous thing to do.env is gitignored. Keep it that way.
docker compose up -d # PostgreSQL + Redis
pnpm db:migrate # create the schema
pnpm db:seed # a demo user and projectTwo processes, two terminals:
pnpm dev # web UI on http://localhost:3000pnpm worker # the agent runtime — nothing happens without thisOpen http://localhost:3000, create a run, and watch the agents work. Set a
maxModelCostUsd on the run. It is enforced, and you will want it.
pnpm test501 tests. No API key is needed for any of them — nothing in the suite calls a model. Services are a different matter, and the numbers are exact because they are measured on every push:
| what is running | tests passing |
|---|---|
| nothing at all | 412 |
| PostgreSQL | 495 |
| PostgreSQL + Redis | 499 |
| + Hindsight (memory) | 500 (+1 skipped by design) |
CI runs the third row — it has no Hindsight container, so the one memory
round-trip test does not run there. It is exercised locally by
pnpm tsx scripts/smoke-reopen-condition.ts, which is worth running after any
change to how memory records are stored: it caught a field that was written
correctly, passed its gate, and silently failed to come back on recall.
So you can run most of the suite before touching Docker. If you see failures
mentioning @prisma/client did not initialize, run pnpm db:generate once — under
pnpm the client generated during postinstall occasionally lands where the runtime
does not look for it.
Every run takes maxModelCostUsd. A pre-dispatch estimator refuses to start a task
it cannot afford, rather than noticing the overspend afterwards. For extra safety
there is a second, independent guard that polls actual cost and force-cancels:
pnpm tsx scripts/cost-guard.ts <runId> 0.25 15000
# run cap poll msA focused run over four sources costs roughly $0.20.
In the run's constraints:
{ "requiredEvidenceClasses": ["CODE_VERIFIED", "EXTERNAL_MEASURED"] }This is not a hint to the model. Requiring CODE_VERIFIED mechanically obliges the
pipeline to search a code host, and the final report cannot conclude "no qualifying
candidate exists" until that actually happened. If it did not, the verdict is
overridden to INCOMPLETE_SOURCE_COVERAGE and the override is recorded as a visible
event.
CANDIDATES_FOUND found something that meets the bar
NO_CANDIDATES_IN_SEARCHED_SOURCES looked properly, found nothing
ZERO_QUALIFYING_CANDIDATES ...and coverage was complete
INCOMPLETE_SOURCE_COVERAGE a required source was never searched
RETRIEVAL_FAILED a required source failed
EVIDENCE_NOT_VERIFIED candidates exist, evidence does not support them
The middle three are the point. A negative result you have not earned is worse than no result, because you will act on it.
REPO_ARTIFACT_VERIFIED implementation + eval scripts + results, all read
CODE_VERIFIED a named symbol in a named file, read and quoted
EXTERNAL_MEASURED a benchmark number someone else measured, retrieved
REPO_CLAIM a repository describing itself (README)
PAPER_CLAIM a paper's own abstract
INFERENCE reasoning over the above
HYPOTHESIS asserted, unverified
CODE_VERIFIED requires either a multi-word mechanism phrase or two distinct
mechanism-specific terms, matched in source that was actually fetched — and the verdict
must name file, line, symbol and matched term. Star counts are not evidence of anything
and may not be cited as support.
A match inside a comment or docstring does not count on its own. That is the
repository describing itself, which is REPO_CLAIM. It is promoted only when a real
line of code in the same file uses the mechanism, and then the code line is what
gets cited — the comment was how the site was found, not the evidence.
Full detail: docs/EVIDENCE_CLASSES.md. Pipeline internals: docs/ARCHITECTURE.md.
Every serious defect found while building this was the same mistake in a new disguise.
In search. The pipeline required code-level evidence, never queried a code host, and reported that no qualifying candidate existed. A verdict about the world, drawn from a hole in its own coverage.
In code inspection. Search terms came from the research goal — a task
description. The most frequent words of "identify systems that address exactly one
of three failure modes with measured evidence" are
["failure", "exactly", "loss", "remedy", "evidence"]. Searching source code for those
granted CODE_VERIFIED to a literature-search engine, because one C++ file contained
the word "exactly".
In the verdict. A finding was justified by the first matching line instead of the strongest: a docstring bullet on line 700 got cited while the real implementation sat at line 2191 in the same match list.
In memory. The worst, because it compounded. Claims marked unsupported (no source
backs this) were stored alongside contradicts (a source argues against this), then
recalled to every later run under the heading PRIOR NEGATIVE RESULTS. Across the
entire history: 111 unsupported, zero contradicts. So everything memory offered
as "already failed" was merely unverified — including true statements the system was
then told, run after run, had been tried and refuted.
In the gate itself — found by hand, after the four above were fixed. Running
the inspector against a real repository returned CODE_VERIFIED on a phrase matched
at HippoRAG.py:700. That line is - Personalized PageRank based re-ranking: a
bullet in a docstring. The gate had already computed that the match was prose — it
appended "(in a comment — prose mention, not an implementation site)" to its own
reason — and returned CODE_VERIFIED anyway. It knew, said so out loud, and graded
prose as code.
Nothing in 477 passing tests caught it, because every fixture in the suite looked
like def f(): / # personalized pagerank / pass. The mechanism named in a
comment, the body doing nothing. Those fixtures were written to test other things —
depth, artifacts, reason strings — and in passing they encoded "a comment is enough"
as the expected behaviour. The suite was not merely silent about the bug. It asserted
it.
A prose match now has to be corroborated by a line of real code in the same file
using the mechanism, and the verdict cites that line. Same repository, after:
matched mechanism phrase "personalized pagerank" at src/hipporag/HippoRAG.py:2168 in run_ppr. The implementation, not the advertisement.
In the query. Found by running the system, after four rounds of code audit had found nothing here. Every run was sending English sentences to GitHub's repository search:
"code-verified implementations of bitemporal memory extraction with
epoch-day normalization" -> 0 results
"measured evaluations of multi-anchor retrieval fusion compared to
top-1-anchored escalation" -> 0 results
"existing LLM agent memory systems code" -> 1 result
Nineteen of twenty queries returned nothing. GitHub matches names, descriptions and
topics and ANDs every term; prose cannot match. The queries also carried this
system's own evidence vocabulary — code-verified, external benchmark are
internal class names no repository description contains, so the run was asking a code
host for the answer rather than the material.
CODE_VERIFIED can only come from reading repository code. So a run required it,
retrieved one repository out of sixteen sources, and reported that no candidate
existed for all three targets. The emptiness was structural, and three consecutive
runs read it as a fact about the world. The systems those runs were looking for are
findable in one keyword: graphiti returns getzep/graphiti, hipporag returns
OSU-NLP-Group/HippoRAG. Nobody had asked in a way that could work.
In the fix for one of the others — the most uncomfortable of the seven. The
context block was being cut by character count, so a fix was written to drop whole
records instead. It dropped lines. Records contain newlines, and the renderer emits
two or three lines per record on purpose, so it stripped REJECTED BECAUSE: off a
refuted hypothesis and left a bare bullet under a heading that calls it refuted —
reintroducing, verbatim, a defect fixed hours earlier by the commit it cited in its
own comment. Every test in that file still passed. It was caught by reading the code
adversarially, not by running it.
Absence of evidence is not evidence of absence. An agent that cannot hold that distinction will confidently hand you a negative result it never earned — and, as the fifth case shows, a positive one too. The last two are the reason this list keeps growing: the defect survives being understood, and it survives being fixed.
Public on purpose. Each links to an issue with the run id, the stored rows and the file:line that produced it. See Issues.
- The inspection depth field records the verdict, not the depth, and is false
every time it is set.
README_READis only ever assigned to an inspection that did open source files and found no match — it never means "we only read the README". Three repositories that read 36 files between them are all stored as though their code was never opened. (#8) - Recall reports "the bank had nothing" when it was the resolver that dropped
everything. Fourteen candidates returned, twelve unresolvable,
empty: true— whose contract says "a healthy provider responded and found nothing". (#9) - A run can lose every memory record it wrote and report
Retained 18/18. The canonical write swallows its errors, and a write-gate rejection is aninfoevent that gates nothing. (#10) - Code inspection is invisible to the report, and to the model that concludes no candidate exists. A 24,521-character report mentions none of it. (#11)
- The largest file in a repository is always skipped, and it is the likeliest place for the mechanism. A memory repository's 336 KB store implementation was never opened, on five passes. (#12)
- Two evidence classes cannot be produced by any code path, and a run may still
require them. Coverage now refuses to call that complete;
verifiedArtifactsis still a permanently-zero field printed in every report. (#13) - The relevance gate is lexical. A query about conversational turn-level memory once retrieved a paper on turn-level traffic flow prediction at road intersections. Partly fixed by anchoring on the run's domain; still lexical. (#1)
- Recall finds mechanism families reliably and specific known systems unreliably. A structural cause was found and removed — see below — but the held-out score predates the fix and has not been re-measured. (#4)
Recently closed: a successful code inspection now does raise the evidence class it supports (#2), and the memory bank is protected by a Postgres advisory lock (#3).
You will see the name ResearchOS in the running application, in log lines and in
the Docker container names. That was this project's working name and it is still what
the code calls itself; nullius is the name it was published under. Renaming 42 files
of UI copy to make a repository look tidier is not worth the risk of breaking
something, so the inconsistency stays until there is a reason to touch those files
anyway.
This is a derivative work of
arnavbathla/ai-scientist by Arnav
Bathla (MIT). The multi-agent pipeline, the agent roles, the supervisor loop and the
application shell come from there. Its copyright notice is preserved in
LICENSE, as MIT requires.
What this project added on top — the evidence-class ladder and its gate, code inspection, source-coverage gating, staleness-driven re-verification, the inquiry loop, the claim ledger, persistent research memory, circuit breaking and budget enforcement — is listed file by file in NOTICE.
Memory is Hindsight by Vectorize (MIT),
run as a container and reached over HTTP. Models come from DeepSeek or Anthropic.
Sources are OpenAlex, Crossref, PubMed and GitHub, each under its own terms — set
OPENALEX_EMAIL and NCBI_EMAIL to identify yourself, as those services ask.
Also using: PostgreSQL · Redis · Prisma · BullMQ · Next.js · React · Vitest · Playwright
docs/WHERE_TO_START.md lists real, currently-open work with the data behind each one — three that need no pipeline knowledge, three that change what the agent can do.
Bug reports that show the system claiming more than its evidence supports are the most valuable thing you can send. One house rule: any change touching an evidence decision needs a test that fails when the fix is reverted. See CONTRIBUTING.md for why that rule exists — it was learned the hard way.
MIT licensed.