Skip to content
2 changes: 1 addition & 1 deletion .claude/skills/fact-check/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ with `refs_manifest.json` and per-source ingests under `case/ingest/<slug>/`.
running head): pick phrases unique within the block.

6. **Crop the evidence.**
`papertrace highlight case/ --claim <id>` (or the library call) →
`papertrace highlight -c <case> --claim <id>` (or the library call) →
writes `case/out/evidence/claim_<id>_<slug>_p<page>.png` with red boxes on
the anchor phrases. If zero boxes were drawn, your anchor phrases don't
match the page text — fix them (ligatures, hyphenation) rather than
Expand Down
30 changes: 21 additions & 9 deletions .claude/skills/review/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,15 +58,26 @@ proceeding; report anything unreadable immediately, not five steps later.

## 2 · Inventory — first wow

**Settle the case folder before anything else.** One case folder per paper:
left alone, `papertrace` names it after the paper's own file and puts it beside
the paper (`<the paper's folder>/<paper stem>/`), and `-c` overrides that. Fix
it once, tell the user the path, and pass the same `-c <case>` to every later
step — the stages that take no paper (`scout`, `check`, `highlight`, `report`)
have no name to derive and refuse rather than guess between two audits. If the
folder already holds an audit of this paper, the CLI asks whether to amend it
(picking up sources added since) or start a numbered sibling; with nobody at a
terminal it amends and says so.

Run ingest and echo back what you actually found, as a tidy table:

```bash
papertrace ingest <paper.pdf> -o case/ingest/manuscript
papertrace ingest <paper.pdf> -o <case>/ingest/manuscript
```

Report: pages, blocks, the detected section headings, how many numbered
references the paper cites (`papertrace refs --parse-only`), and how many of
those the user's sources folder already covers. One table, no prose padding.
references the paper cites (`papertrace refs <paper.pdf> --parse-only`), and
how many of those the user's sources folder already covers. One table, no
prose padding.
If anything looks off (no References section found, scanned/no text layer),
say so now and ask.

Expand All @@ -75,7 +86,7 @@ say so now and ask.
Collect what the user actually wants answered. Read any screenshots with the
Read tool and extract the exact fields — numbered questions, dropdowns,
word-limited boxes; take typed questions as they are. Write the combined
checklist to `case/out/questions.md`, then tell the user:
checklist to `<case>/out/questions.md`, then tell the user:

> You're asking **N questions**. Every one of them will be answered in the
> final write-up — here's the list so you can correct me now if I misread
Expand All @@ -88,7 +99,7 @@ literature, methods and results consistency — and say you did.
## 4 · Retrieval — live ticker

```bash
papertrace refs <paper.pdf> --provided <sources_dir> -o case/
papertrace refs <paper.pdf> --provided <sources_dir> -c <case>
```

Stream the per-reference ticker as it runs (✓ retrieved via unpaywall · ✓
Expand All @@ -98,7 +109,7 @@ unpublished)”**, and remind the user they can drop more PDFs into the sources
folder at any point; you'll pick them up on request.

For a published paper, also run the literature scout
(`papertrace scout -c case/`, `--doi` if the title lookup misses) and show
(`papertrace scout -c <case>`, `--doi` if the title lookup misses) and show
its two registers: published since, and existed-but-uncited. Candidates for
the user's judgement, not accusations.

Expand Down Expand Up @@ -157,10 +168,10 @@ unsure.

## 7 · Outputs

Write to `case/out/`:
Write to `<case>/out/`:

- `results.json` — every claim with verdict + anchors (schema in `schemas/`)
- `fact_check_report.md` + rendered looks: `papertrace report case/`
- `fact_check_report.md` + rendered looks: `papertrace report -c <case>`
- `questions.md` — the user's questions, now answered
- `findings.md` — the audit narrative: what holds, what doesn't, what
couldn't be checked, what the scout surfaced
Expand All @@ -187,7 +198,8 @@ a busy reader to look at first. Close with:
- Scout hits and uncited-literature candidates are search-based leads, not
findings — present them as questions, never as misconduct claims.
- Do not reproduce >15 consecutive words of the audited paper in any output.
- Everything stays in the local `case/` folder, which is gitignored.
- Everything stays in the local case folder, which carries a `.gitignore` of
its own — a folder named after a manuscript is not covered by the repo's.
- When the session is a peer review, additionally: no accept/reject
recommendation in author-facing text (that reasoning goes only to the
editor, and even there as reasoning, not a verdict); never sign with the
Expand Down
127 changes: 127 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,18 @@ All notable changes to PaperTrace are documented here. The format follows

### Added

- **A resumed reference list says so, in all three formats.** Crossing a section
boundary to finish a bibliography is a judgement, and the numbering of the
later entries rests on it — so `RefManifest.references_resumed` records it
(optional, in `schemas/refs_manifest.schema.json`) and a run-level disclosure
carries it into markdown, editor and terminal; `refs` also says so on the
console in amber, so the crossed boundary is visible before the report is
read. The wording points both ways on
purpose: the guess could be wrong, but not making it was the previous
behaviour and that failed silently. The terminal template needed an explicit
branch, and the mechanical guard added in 0.4.0 caught the omission before any
parity test did — its first live catch.

- **`evals/` — an offline evaluation harness and its design.**
[`evals/DESIGN.md`](evals/DESIGN.md) specifies the evaluation unit, paired
faithful/altered cases, the metric definitions (verdict accuracy, per-class
Expand Down Expand Up @@ -83,6 +95,22 @@ All notable changes to PaperTrace are documented here. The format follows

### Changed

- **The README claimed a figure's contents get checked, which it could not
support.** "A claim that lives in a table cell **or inside a figure** is found,
checked, and shown like any other" was true of the red box — a figure's numbers
are in the PDF text layer, so text search finds them on the real page — and
unsupported for the judging half: under the layout backend a figure region
reaches the model as `[FIGURE: <caption>]`, and in-figure text arrives only
where docling's layout model found a text region inside the figure. On the one
paper measured it found none: of 9 figures, 5 carried text in the text layer
and no docling text block landed inside any figure region, while the flat-text
backend did carry that text. The section now names which backend each half
holds for, states the measurement as one paper rather than a rate, and says the
two illustrating crops come from **cited sources** — which batch mode always
reads as flat text. The same section's claim that a flat-text source delivers
"its figures not at all" was wrong in the other direction and now says what it
does deliver: loose words with no figure attached.

- **A co-cited claim is judged against every source it cites, not just the
first.** Batch mode used to pick `avail[0]`, judge against that, and file
every other co-citation as never opened. Co-citation is an offer of support,
Expand Down Expand Up @@ -163,6 +191,105 @@ All notable changes to PaperTrace are documented here. The format follows

### Fixed

- **A URL-only reference was resolved to an unrelated paper.** Reference [8] of a
real audited manuscript is an ACR news page with no DOI. With no DOI to look
up, `resolve_entry` fell through to a Crossref *bibliographic title search* —
which always returns something — and that something was `10.1002/acr2.11538`:
ACR Open Rheumatology, American College of *Rheumatology*, not Radiology.
Unpaywall served Solomon et al.'s editorial on authorship and ChatGPT, the
title check passed it at 6/15, and two claims were reported `not_addressed`
against a rheumatology editorial. Two of those six matches were `chatgpt` and
`source`, harvested from the `?utm_source=chatgpt.com` tracking parameter in
the reference's own URL — the tracking parameter is what made a ChatGPT
editorial look like a title match. A reference whose identity is carried by a
URL — no DOI, no volume, no page range, no identifier — now terminates at
`no_doi` with no request sent: a news page was never retrievable as a PDF, so
the honest gap costs nothing that was ever on offer. The gate keys on the
absence of article structure rather than the presence of a link, because
publishers' own reference styles print a URL beside the volume and those
references resolve well. Separately the title check no longer takes tokens
from a URL, and `verified` now needs four distinct matched words rather than a
ratio a three-word reference clears on generic domain vocabulary — falling
short reads `unverifiable`, never `mismatch`, since too few words to tell is
not evidence of a different paper.
- **Every audit defaulted into one folder called `case`, in whatever directory the
user was standing in.** A first-time user ran a batch audit from the root of a
git clone and the output landed in `PaperTrace/case/` — not named for the paper,
and the same folder every subsequent paper would have used. `run` and `refs` now
default to a folder named after the paper, beside the paper: the one location
stable across invocations, so a re-run finds its own case without a flag. An
explicit `-c` still wins, unconditionally. `check`, `highlight`, `report` and
`scout` have no paper to take a name from, so they are given no default at all —
`./case` is still used when it exists, and otherwise they refuse, listing the
folders in this directory that look like audits rather than picking one. Because
a case folder is no longer named `case`, `.gitignore`'s name-based guardrail no
longer covers it, so the folder is created carrying a `.gitignore` of its own.
- **Re-running a paper reused its case folder silently, including when that was not
what the user meant.** Adding source PDFs and re-running is the intended flow, but
so is auditing a revised draft, and nothing distinguished them: `_guard_case`
refuses a *different* paper and permits the same one without a word. A derived
case folder already holding an audit of this paper now asks — amend it
(references are resolved again, so sources added since are picked up) or start a
numbered sibling, leaving the first untouched. Only for a folder the tool named
itself; `-c` is an instruction, not a suggestion. With nobody at a terminal it
amends and says so, never blocking on stdin: amend is what a re-run did before,
it deletes nothing, and it keeps the report's path predictable, where a fresh
folder would move a scripted caller's output somewhere it never named. A
different paper in the folder is still exit 2, unchanged.
- **The skills documented four commands that do not exist.** `papertrace refs …
-o case/` (`refs` has no `-o`), `papertrace report case/` and `papertrace
highlight case/ --claim <id>` (neither takes a positional argument), and a prose
`papertrace refs --parse-only` with no manuscript — each a usage error, in files
an agent executes verbatim. Every `papertrace` line in every skill is now parsed
against the real Typer commands by a test. Appending `--help` would not have
done: `--help` is eager and fires before click reports an unexpected extra
argument, so `report case/ --help` exits 0 and the check would have passed a
broken line. It parses each documented line into a click context instead, which
validates arguments without invoking anything. Scanning one skill is how the
fourth command survived while the other three were fixed, so the test walks
`.claude/skills/**/*.md`.
- **docling deleted a hyphen that belonged to the word.** It joins a word split
across two lines and drops the hyphen, which is *right* far more often than it
is wrong — `approxi-` / `mately` is the single word "approximately", and 87 of
94 breaks on the paper measured were of that kind — and wrong when the hyphen
is the word's own: `Non-` / `Hispanic` arrived as `NonHispanic`, `thin-fat` as
`thinfat`. Latent: it corrupts the text the model reads and the phrases it can
quote. The adapter now repairs a join **only where the paper writes that
compound out unbroken somewhere else** — the document's own evidence, never a
lower→upper junction, which proves nothing (`HbA1c` and `PaperTrace` have one;
`Timedependent` has one and is broken). Unproven joins are left exactly as
docling produced them rather than repaired by guesswork: on the measured paper
that is 4 blocks rewritten, not the 46 a blanket fix touched.
- **An anchor phrase could be unboxable for typesetting reasons alone, and this
one was observed.** `page.search_for` reads a hyphenated line break as a
space, so a page printing `Non-` / `Hispanic` carries only `Non- Hispanic` —
neither the compound the paper means nor docling's join is on it, and a model
tidying a quoted `develop- ing` to `developing` was searching for a string no
page has. In a real audit one of two quoted phrases (`sohn-2022` p2) matched
nothing, and that claim kept its box only because its second phrase matched.
`highlight` now retries a missed phrase in forms the **page** dictates:
whitespace beside a dash the phrase already carries, then the phrase rewritten
with the page's own line-break hyphenation. Still exact text search, still no
similarity matching — a phrase the page does not carry returns no box and
`anchor_located = False`, as before. Measured over that audit's 20 anchor
phrases: 18 located verbatim, 19 with the retry.
- **A reference list split by an intervening section was parsed short.** A real
pre-proof put refs 1-9 on page 7, a `Declaration of interests` section next,
then refs 10-15 on page 8. `references_section` stops at the following section
header — the guard that keeps the reference list from swallowing the rest of
the paper — so six references were never parsed, never retrieved and never
mentioned: the audit reported 9 references on a paper citing 15. The list is
now picked up again after an interruption, gated on two independent signals
because neither alone separates a split bibliography from an appendix: the
entries must be `list`-typed, and the resumed run must be at least two blocks.
Restricted to `list` and never `text` on purpose — under the flat backend
reference entries are `text`, the same type as every paragraph, so resuming
there would swallow the Discussion of any paper whose references are not last.
The cost of that asymmetry is that a flat-ingested split list is still parsed
short. Only reference-shaped runs are collected, which matters more than it
sounds: `_parse_bulleted` appends a non-bullet line to the *previous* entry, so
a stray paragraph corrupts a reference rather than merely adding noise.

- **A bracketed label inside a reference could steal the next entry, and its
DOI.** The mid-line marker rule exists because Elsevier PDFs run entries
together — requiring a line start once collapsed 34 references into one. But
Expand Down
40 changes: 31 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ and batching its questions.
```bash
pip install -e ".[full]" # standard install (see matrix below)
export PAPERTRACE_EMAIL="you@example.org" # Unpaywall asks for a contact
papertrace run paper.pdf --provided ./my_pdfs -c case
papertrace run paper.pdf --provided ./my_pdfs # case folder: ./paper/ beside the PDF
```

Install options:
Expand Down Expand Up @@ -284,24 +284,46 @@ accusations.

## Tables and figures are evidence too

With the standard install, a claim that lives in a table cell or inside a
figure is found, checked, and shown like any other — cell and in-figure
numbers boxed by text search on the real page:
A number in a table cell, or drawn inside a figure, is still in the PDF's text
layer — so the red box lands on it whichever backend read the document.
`highlight` searches the real page, never the extracted text:

<p align="center">
<img src="https://raw.githubusercontent.com/defraction0/PaperTrace/main/docs/table_figure_evidence.png" width="85%" alt="Two evidence crops: a table cell (N = 8382, 84.3%) and a number inside a flow-chart figure (97%), each boxed in red">
</p>

Both crops above come from **cited sources** whose block types (`table block`,
`picture block`) come from ingesting those sources with the layout backend by
hand — in batch mode `check` reads a cited source as flat text.

Whether such a number can be *claimed and checked* in the first place is a
different question, decided by what the backend hands the model:

| | a table cell | text drawn inside a figure |
|---|---|---|
| **flat text** (`pymupdf`) | reaches the model linearised — the row and column it belongs to are lost | reaches the model as loose words, with no figure to belong to |
| **layout-aware** (`docling`; standard install, audited paper only) | reaches the model as a GFM table | the figure arrives as `[FIGURE: <caption>]`; in-figure text arrives only where docling's layout model found a text region inside the figure |

On the one paper measured for this, it found none: of 9 figures, 5 carried text
in the PDF's text layer, and docling emitted no text block anywhere inside a
figure region — so that text reached the model nowhere, while the flat-text
backend did carry it. One paper is not a rate and none is claimed; what is
claimed is only that in-figure text is **not guaranteed** on the layout path.

So the box is equally trustworthy either way, and the claim behind it is not:
a table-cell claim is strongest under the layout backend, and an in-figure claim
is the weakest evidence this tool produces — under the layout backend the judge
may never have seen the number, and under flat text it saw the number without
the figure that gives it meaning.

That layout fidelity is spent on the **audited paper**. In batch mode a cited
source that has **not yet been ingested** is ingested with the fast flat-text
backend, so its tables reach the judge linearised and its figures not at all.
backend, so its tables reach the judge linearised and its figures only as
whatever loose words sat inside them.
`check` reuses an existing `case/ingest/<slug>/annotated.md` if one is already
there — so a source you ingested yourself with `papertrace ingest --backend
docling` keeps its layout, and the report does **not** currently distinguish
the two cases. The
red box still lands correctly either way — PDF text search doesn't care about
layout — but a claim resting on a figure inside a *cited source* is weaker
evidence than one resting on its prose.
the two cases.

## Try the demo yourself

Expand Down
4 changes: 4 additions & 0 deletions schemas/refs_manifest.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,10 @@
}
}
}
},
"references_resumed": {
"type": "boolean",
"description": "True when the reference list was picked up again after an intervening section, so the entry numbering spans a boundary the parser chose to cross. Gated on block type and run length, but still a judgement — the reader should be able to check it."
}
}
}
Loading
Loading