diff --git a/.claude/skills/fact-check/SKILL.md b/.claude/skills/fact-check/SKILL.md index 99be1cb..4a74839 100644 --- a/.claude/skills/fact-check/SKILL.md +++ b/.claude/skills/fact-check/SKILL.md @@ -43,7 +43,7 @@ with `refs_manifest.json` and per-source ingests under `case/ingest//`. running head): pick phrases unique within the block. 6. **Crop the evidence.** - `papertrace highlight case/ --claim ` (or the library call) → + `papertrace highlight -c --claim ` (or the library call) → writes `case/out/evidence/claim___p.png` with red boxes on the anchor phrases. If zero boxes were drawn, your anchor phrases don't match the page text — fix them (ligatures, hyphenation) rather than diff --git a/.claude/skills/review/SKILL.md b/.claude/skills/review/SKILL.md index 27e5ab4..181e263 100644 --- a/.claude/skills/review/SKILL.md +++ b/.claude/skills/review/SKILL.md @@ -58,15 +58,26 @@ proceeding; report anything unreadable immediately, not five steps later. ## 2 · Inventory — first wow +**Settle the case folder before anything else.** One case folder per paper: +left alone, `papertrace` names it after the paper's own file and puts it beside +the paper (`//`), and `-c` overrides that. Fix +it once, tell the user the path, and pass the same `-c ` to every later +step — the stages that take no paper (`scout`, `check`, `highlight`, `report`) +have no name to derive and refuse rather than guess between two audits. If the +folder already holds an audit of this paper, the CLI asks whether to amend it +(picking up sources added since) or start a numbered sibling; with nobody at a +terminal it amends and says so. + Run ingest and echo back what you actually found, as a tidy table: ```bash -papertrace ingest -o case/ingest/manuscript +papertrace ingest -o /ingest/manuscript ``` Report: pages, blocks, the detected section headings, how many numbered -references the paper cites (`papertrace refs --parse-only`), and how many of -those the user's sources folder already covers. One table, no prose padding. +references the paper cites (`papertrace refs --parse-only`), and +how many of those the user's sources folder already covers. One table, no +prose padding. If anything looks off (no References section found, scanned/no text layer), say so now and ask. @@ -75,7 +86,7 @@ say so now and ask. Collect what the user actually wants answered. Read any screenshots with the Read tool and extract the exact fields — numbered questions, dropdowns, word-limited boxes; take typed questions as they are. Write the combined -checklist to `case/out/questions.md`, then tell the user: +checklist to `/out/questions.md`, then tell the user: > You're asking **N questions**. Every one of them will be answered in the > final write-up — here's the list so you can correct me now if I misread @@ -88,7 +99,7 @@ literature, methods and results consistency — and say you did. ## 4 · Retrieval — live ticker ```bash -papertrace refs --provided -o case/ +papertrace refs --provided -c ``` Stream the per-reference ticker as it runs (✓ retrieved via unpaywall · ✓ @@ -98,7 +109,7 @@ unpublished)”**, and remind the user they can drop more PDFs into the sources folder at any point; you'll pick them up on request. For a published paper, also run the literature scout -(`papertrace scout -c case/`, `--doi` if the title lookup misses) and show +(`papertrace scout -c `, `--doi` if the title lookup misses) and show its two registers: published since, and existed-but-uncited. Candidates for the user's judgement, not accusations. @@ -157,10 +168,10 @@ unsure. ## 7 · Outputs -Write to `case/out/`: +Write to `/out/`: - `results.json` — every claim with verdict + anchors (schema in `schemas/`) -- `fact_check_report.md` + rendered looks: `papertrace report case/` +- `fact_check_report.md` + rendered looks: `papertrace report -c ` - `questions.md` — the user's questions, now answered - `findings.md` — the audit narrative: what holds, what doesn't, what couldn't be checked, what the scout surfaced @@ -187,7 +198,8 @@ a busy reader to look at first. Close with: - Scout hits and uncited-literature candidates are search-based leads, not findings — present them as questions, never as misconduct claims. - Do not reproduce >15 consecutive words of the audited paper in any output. -- Everything stays in the local `case/` folder, which is gitignored. +- Everything stays in the local case folder, which carries a `.gitignore` of + its own — a folder named after a manuscript is not covered by the repo's. - When the session is a peer review, additionally: no accept/reject recommendation in author-facing text (that reasoning goes only to the editor, and even there as reasoning, not a verdict); never sign with the diff --git a/CHANGELOG.md b/CHANGELOG.md index d6e02b2..3155842 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,18 @@ All notable changes to PaperTrace are documented here. The format follows ### Added +- **A resumed reference list says so, in all three formats.** Crossing a section + boundary to finish a bibliography is a judgement, and the numbering of the + later entries rests on it — so `RefManifest.references_resumed` records it + (optional, in `schemas/refs_manifest.schema.json`) and a run-level disclosure + carries it into markdown, editor and terminal; `refs` also says so on the + console in amber, so the crossed boundary is visible before the report is + read. The wording points both ways on + purpose: the guess could be wrong, but not making it was the previous + behaviour and that failed silently. The terminal template needed an explicit + branch, and the mechanical guard added in 0.4.0 caught the omission before any + parity test did — its first live catch. + - **`evals/` — an offline evaluation harness and its design.** [`evals/DESIGN.md`](evals/DESIGN.md) specifies the evaluation unit, paired faithful/altered cases, the metric definitions (verdict accuracy, per-class @@ -83,6 +95,22 @@ All notable changes to PaperTrace are documented here. The format follows ### Changed +- **The README claimed a figure's contents get checked, which it could not + support.** "A claim that lives in a table cell **or inside a figure** is found, + checked, and shown like any other" was true of the red box — a figure's numbers + are in the PDF text layer, so text search finds them on the real page — and + unsupported for the judging half: under the layout backend a figure region + reaches the model as `[FIGURE: ]`, and in-figure text arrives only + where docling's layout model found a text region inside the figure. On the one + paper measured it found none: of 9 figures, 5 carried text in the text layer + and no docling text block landed inside any figure region, while the flat-text + backend did carry that text. The section now names which backend each half + holds for, states the measurement as one paper rather than a rate, and says the + two illustrating crops come from **cited sources** — which batch mode always + reads as flat text. The same section's claim that a flat-text source delivers + "its figures not at all" was wrong in the other direction and now says what it + does deliver: loose words with no figure attached. + - **A co-cited claim is judged against every source it cites, not just the first.** Batch mode used to pick `avail[0]`, judge against that, and file every other co-citation as never opened. Co-citation is an offer of support, @@ -163,6 +191,105 @@ All notable changes to PaperTrace are documented here. The format follows ### Fixed +- **A URL-only reference was resolved to an unrelated paper.** Reference [8] of a + real audited manuscript is an ACR news page with no DOI. With no DOI to look + up, `resolve_entry` fell through to a Crossref *bibliographic title search* — + which always returns something — and that something was `10.1002/acr2.11538`: + ACR Open Rheumatology, American College of *Rheumatology*, not Radiology. + Unpaywall served Solomon et al.'s editorial on authorship and ChatGPT, the + title check passed it at 6/15, and two claims were reported `not_addressed` + against a rheumatology editorial. Two of those six matches were `chatgpt` and + `source`, harvested from the `?utm_source=chatgpt.com` tracking parameter in + the reference's own URL — the tracking parameter is what made a ChatGPT + editorial look like a title match. A reference whose identity is carried by a + URL — no DOI, no volume, no page range, no identifier — now terminates at + `no_doi` with no request sent: a news page was never retrievable as a PDF, so + the honest gap costs nothing that was ever on offer. The gate keys on the + absence of article structure rather than the presence of a link, because + publishers' own reference styles print a URL beside the volume and those + references resolve well. Separately the title check no longer takes tokens + from a URL, and `verified` now needs four distinct matched words rather than a + ratio a three-word reference clears on generic domain vocabulary — falling + short reads `unverifiable`, never `mismatch`, since too few words to tell is + not evidence of a different paper. +- **Every audit defaulted into one folder called `case`, in whatever directory the + user was standing in.** A first-time user ran a batch audit from the root of a + git clone and the output landed in `PaperTrace/case/` — not named for the paper, + and the same folder every subsequent paper would have used. `run` and `refs` now + default to a folder named after the paper, beside the paper: the one location + stable across invocations, so a re-run finds its own case without a flag. An + explicit `-c` still wins, unconditionally. `check`, `highlight`, `report` and + `scout` have no paper to take a name from, so they are given no default at all — + `./case` is still used when it exists, and otherwise they refuse, listing the + folders in this directory that look like audits rather than picking one. Because + a case folder is no longer named `case`, `.gitignore`'s name-based guardrail no + longer covers it, so the folder is created carrying a `.gitignore` of its own. +- **Re-running a paper reused its case folder silently, including when that was not + what the user meant.** Adding source PDFs and re-running is the intended flow, but + so is auditing a revised draft, and nothing distinguished them: `_guard_case` + refuses a *different* paper and permits the same one without a word. A derived + case folder already holding an audit of this paper now asks — amend it + (references are resolved again, so sources added since are picked up) or start a + numbered sibling, leaving the first untouched. Only for a folder the tool named + itself; `-c` is an instruction, not a suggestion. With nobody at a terminal it + amends and says so, never blocking on stdin: amend is what a re-run did before, + it deletes nothing, and it keeps the report's path predictable, where a fresh + folder would move a scripted caller's output somewhere it never named. A + different paper in the folder is still exit 2, unchanged. +- **The skills documented four commands that do not exist.** `papertrace refs … + -o case/` (`refs` has no `-o`), `papertrace report case/` and `papertrace + highlight case/ --claim ` (neither takes a positional argument), and a prose + `papertrace refs --parse-only` with no manuscript — each a usage error, in files + an agent executes verbatim. Every `papertrace` line in every skill is now parsed + against the real Typer commands by a test. Appending `--help` would not have + done: `--help` is eager and fires before click reports an unexpected extra + argument, so `report case/ --help` exits 0 and the check would have passed a + broken line. It parses each documented line into a click context instead, which + validates arguments without invoking anything. Scanning one skill is how the + fourth command survived while the other three were fixed, so the test walks + `.claude/skills/**/*.md`. +- **docling deleted a hyphen that belonged to the word.** It joins a word split + across two lines and drops the hyphen, which is *right* far more often than it + is wrong — `approxi-` / `mately` is the single word "approximately", and 87 of + 94 breaks on the paper measured were of that kind — and wrong when the hyphen + is the word's own: `Non-` / `Hispanic` arrived as `NonHispanic`, `thin-fat` as + `thinfat`. Latent: it corrupts the text the model reads and the phrases it can + quote. The adapter now repairs a join **only where the paper writes that + compound out unbroken somewhere else** — the document's own evidence, never a + lower→upper junction, which proves nothing (`HbA1c` and `PaperTrace` have one; + `Timedependent` has one and is broken). Unproven joins are left exactly as + docling produced them rather than repaired by guesswork: on the measured paper + that is 4 blocks rewritten, not the 46 a blanket fix touched. +- **An anchor phrase could be unboxable for typesetting reasons alone, and this + one was observed.** `page.search_for` reads a hyphenated line break as a + space, so a page printing `Non-` / `Hispanic` carries only `Non- Hispanic` — + neither the compound the paper means nor docling's join is on it, and a model + tidying a quoted `develop- ing` to `developing` was searching for a string no + page has. In a real audit one of two quoted phrases (`sohn-2022` p2) matched + nothing, and that claim kept its box only because its second phrase matched. + `highlight` now retries a missed phrase in forms the **page** dictates: + whitespace beside a dash the phrase already carries, then the phrase rewritten + with the page's own line-break hyphenation. Still exact text search, still no + similarity matching — a phrase the page does not carry returns no box and + `anchor_located = False`, as before. Measured over that audit's 20 anchor + phrases: 18 located verbatim, 19 with the retry. +- **A reference list split by an intervening section was parsed short.** A real + pre-proof put refs 1-9 on page 7, a `Declaration of interests` section next, + then refs 10-15 on page 8. `references_section` stops at the following section + header — the guard that keeps the reference list from swallowing the rest of + the paper — so six references were never parsed, never retrieved and never + mentioned: the audit reported 9 references on a paper citing 15. The list is + now picked up again after an interruption, gated on two independent signals + because neither alone separates a split bibliography from an appendix: the + entries must be `list`-typed, and the resumed run must be at least two blocks. + Restricted to `list` and never `text` on purpose — under the flat backend + reference entries are `text`, the same type as every paragraph, so resuming + there would swallow the Discussion of any paper whose references are not last. + The cost of that asymmetry is that a flat-ingested split list is still parsed + short. Only reference-shaped runs are collected, which matters more than it + sounds: `_parse_bulleted` appends a non-bullet line to the *previous* entry, so + a stray paragraph corrupts a reference rather than merely adding noise. + - **A bracketed label inside a reference could steal the next entry, and its DOI.** The mid-line marker rule exists because Elsevier PDFs run entries together — requiring a line start once collapsed 34 references into one. But diff --git a/README.md b/README.md index a1e60d4..bec6b79 100644 --- a/README.md +++ b/README.md @@ -182,7 +182,7 @@ and batching its questions. ```bash pip install -e ".[full]" # standard install (see matrix below) export PAPERTRACE_EMAIL="you@example.org" # Unpaywall asks for a contact -papertrace run paper.pdf --provided ./my_pdfs -c case +papertrace run paper.pdf --provided ./my_pdfs # case folder: ./paper/ beside the PDF ``` Install options: @@ -284,24 +284,46 @@ accusations. ## Tables and figures are evidence too -With the standard install, a claim that lives in a table cell or inside a -figure is found, checked, and shown like any other — cell and in-figure -numbers boxed by text search on the real page: +A number in a table cell, or drawn inside a figure, is still in the PDF's text +layer — so the red box lands on it whichever backend read the document. +`highlight` searches the real page, never the extracted text:

Two evidence crops: a table cell (N = 8382, 84.3%) and a number inside a flow-chart figure (97%), each boxed in red

+Both crops above come from **cited sources** whose block types (`table block`, +`picture block`) come from ingesting those sources with the layout backend by +hand — in batch mode `check` reads a cited source as flat text. + +Whether such a number can be *claimed and checked* in the first place is a +different question, decided by what the backend hands the model: + +| | a table cell | text drawn inside a figure | +|---|---|---| +| **flat text** (`pymupdf`) | reaches the model linearised — the row and column it belongs to are lost | reaches the model as loose words, with no figure to belong to | +| **layout-aware** (`docling`; standard install, audited paper only) | reaches the model as a GFM table | the figure arrives as `[FIGURE: ]`; in-figure text arrives only where docling's layout model found a text region inside the figure | + +On the one paper measured for this, it found none: of 9 figures, 5 carried text +in the PDF's text layer, and docling emitted no text block anywhere inside a +figure region — so that text reached the model nowhere, while the flat-text +backend did carry it. One paper is not a rate and none is claimed; what is +claimed is only that in-figure text is **not guaranteed** on the layout path. + +So the box is equally trustworthy either way, and the claim behind it is not: +a table-cell claim is strongest under the layout backend, and an in-figure claim +is the weakest evidence this tool produces — under the layout backend the judge +may never have seen the number, and under flat text it saw the number without +the figure that gives it meaning. + That layout fidelity is spent on the **audited paper**. In batch mode a cited source that has **not yet been ingested** is ingested with the fast flat-text -backend, so its tables reach the judge linearised and its figures not at all. +backend, so its tables reach the judge linearised and its figures only as +whatever loose words sat inside them. `check` reuses an existing `case/ingest//annotated.md` if one is already there — so a source you ingested yourself with `papertrace ingest --backend docling` keeps its layout, and the report does **not** currently distinguish -the two cases. The -red box still lands correctly either way — PDF text search doesn't care about -layout — but a claim resting on a figure inside a *cited source* is weaker -evidence than one resting on its prose. +the two cases. ## Try the demo yourself diff --git a/schemas/refs_manifest.schema.json b/schemas/refs_manifest.schema.json index 7fe914b..fabbafb 100644 --- a/schemas/refs_manifest.schema.json +++ b/schemas/refs_manifest.schema.json @@ -105,6 +105,10 @@ } } } + }, + "references_resumed": { + "type": "boolean", + "description": "True when the reference list was picked up again after an intervening section, so the entry numbering spans a boundary the parser chose to cross. Gated on block type and run length, but still a judgement — the reader should be able to check it." } } } diff --git a/src/papertrace/cli.py b/src/papertrace/cli.py index 1385197..f77e26e 100644 --- a/src/papertrace/cli.py +++ b/src/papertrace/cli.py @@ -10,11 +10,13 @@ import datetime import os +import re import sys from pathlib import Path import typer from rich.console import Console +from rich.prompt import Prompt from .models import ClaimResult, RefManifest, RunResults, manuscript_fingerprint @@ -40,6 +42,82 @@ } +CASE_NAME_MAX = 60 # a folder name, not a title — some journals' stems run long + + +def default_case(manuscript: Path) -> Path: + """Where this paper's audit lives when `-c` was not given: beside the paper, + named after it. + + Not the working directory, for two reasons a real first run hit at once. A + folder literally named `case` is the *same* folder for every paper, so a + second audit lands on the first unless the user remembers `-c`. And it + appears wherever the user happened to be standing — for that user, the root + of a git clone, which `.gitignore` covers only under the name `case/`. The + paper's own folder is the one location that is stable across invocations, so + a re-run finds its case again without a flag. + """ + name = re.sub(r"[^A-Za-z0-9._-]+", "-", manuscript.stem).strip("-.")[:CASE_NAME_MAX] + parent = manuscript.parent + if not os.access(parent, os.W_OK): + # a read-only volume (a mounted share, an email attachment folder): say + # where the audit went instead, never fail for want of a default + console.print( + f"[yellow]⚠ {parent}/ is not writable, so the audit cannot sit beside the " + f"paper — keeping it in {Path.cwd()}/ instead.[/yellow]" + ) + parent = Path.cwd() + return parent / (name or "case") + + +def _sibling_case(case: Path) -> Path: + """`-2`, `-3`, … — the first that does not exist yet.""" + n = 2 + while (candidate := case.with_name(f"{case.name}-{n}")).exists(): + n += 1 + return candidate + + +def _open_case(case: Path) -> Path: + """Create the case folder, ignoring itself. + + `.gitignore` blocks `case/`, `cases/` and `demo_case/` by name; a folder + named after a manuscript matches none of them, so the guardrail travels + inside the folder rather than depending on where it was created. + """ + case.mkdir(parents=True, exist_ok=True) + marker = case / ".gitignore" + if not marker.exists(): + marker.write_text("*\n") + return case + + +def _stage_case(case: Path | None) -> Path: + """Which case folder a stage that has no manuscript works on. + + `check`, `highlight`, `report` and `scout` take no paper, so they have no + name to derive and there is nothing honest to default to: picking one of + several audits in the current directory is exactly the mixing `_guard_case` + exists to prevent. `case/` is still accepted when it is there, because it + was the default through 0.4.0 — anything else is refused with the folders + that do look like audits, rather than guessed. + """ + if case is not None: + return case + if (legacy := Path("case")).is_dir(): + return legacy + found = sorted( + p.name for p in Path().iterdir() if p.is_dir() and (p / "refs_manifest.json").exists() + ) + hint = ( + " audits in this folder: " + ", ".join(f"[cyan]-c {n}[/cyan]" for n in found[:8]) + if found + else " no case folder found here — `papertrace run ` makes one." + ) + console.print(f"[red]which audit? this step needs [bold]-c [/bold].[/red]\n{hint}") + raise typer.Exit(2) + + def _case_conflict(case: Path, manuscript: Path) -> tuple[str | None, str]: """A case folder belongs to one paper. Returns the previous paper's name when `case` already holds an audit of a different one (else None), and what @@ -89,6 +167,45 @@ def _guard_case(case: Path, manuscript: Path) -> str: return basis +def _resolve_case(case: Path | None, manuscript: Path) -> Path: + """The case folder this invocation works in, asking only when it chose the name. + + An explicit `-c` is returned untouched — including when it already holds + this paper, which is a legitimate re-run and what the guard's own message + tells people to do. The question is put only for a *derived* folder, where + the tool picked the name and the user has no reason to expect a collision. + """ + if case is not None: + return case + case = default_case(manuscript) + if not (case / "refs_manifest.json").exists(): + # an absent or half-ingested folder holds no audit, so there is nothing + # to amend and nothing to lose — only the name is worth stating + console.print(f"[dim]case folder: {case}/ — named after the paper; -c chooses another[/dim]") + return case + previous, _ = _case_conflict(case, manuscript) + if previous: + return case # a different paper: `_guard_case` refuses it, and says why + console.print(f"[yellow]{case}/ already holds an audit of this paper.[/yellow]") + fresh = _sibling_case(case) + if not sys.stdin.isatty(): + # A pipe, a cron job or CI has nobody to answer, and must never sit on + # stdin. Amend is the documented choice: it is what re-running the same + # paper did through 0.4.0, it deletes nothing, and it keeps the report's + # path predictable — `fresh` would move the output somewhere the caller + # never named. + console.print(f" [dim]no terminal to ask, so amending {case}/ — pass -c to choose[/dim]") + return case + console.print( + f" [bold]amend[/bold] reuse it — references are resolved again, so source PDFs " + f"you have added since are picked up\n" + f" [bold]fresh[/bold] audit this paper from scratch in {fresh}/, leaving " + f"{case}/ untouched" + ) + answer = Prompt.ask(" amend or fresh?", choices=["amend", "fresh"], default="amend") + return case if answer == "amend" else fresh + + def _verdict_line(c: dict[str, int]) -> str: """The one-line tally, built so a reader's own arithmetic works. @@ -215,13 +332,16 @@ def start() -> None: @app.command(rich_help_panel="Utilities") def init(case: Path = typer.Argument(Path("case"), help="Case folder to create")) -> None: """Create a case folder skeleton (gitignored by design — keep manuscripts local).""" + _open_case(case) for sub in ("sources", "form", "ingest", "out/evidence"): (case / sub).mkdir(parents=True, exist_ok=True) - (case / ".gitignore").write_text("*\n") console.print(BANNER) console.print(f"case folder ready: [cyan]{case}/[/cyan]") console.print(" put reference PDFs you already have into [cyan]sources/[/cyan]") console.print(" put your questions or form-field screenshots into [cyan]form/[/cyan]") + # `run` and `refs` name their own folder after the paper, so a hand-made one + # is only used if it is passed - saying so here beats orphaned sources/ + console.print(f" [dim]hand this folder to every step: [cyan]-c {case}[/cyan][/dim]") @app.command(rich_help_panel="Pipeline stages — `run` calls these in order") @@ -240,7 +360,7 @@ def ingest( # -c means the same thing here as in every other subcommand; `papertrace # ingest -c foo` used to fail with "No such option: -c" while its # neighbours all took it. --out stays authoritative and unchanged. - out = out or (case or Path("case")) / "ingest" / pdf.stem + out = out or (case or default_case(pdf)) / "ingest" / pdf.stem smap = ingest_pdf(pdf, out, backend=backend) by_type = {t: sum(1 for b in smap.blocks if b.type == t) for t in ("sectionheader", "text", "table", "picture", "list")} @@ -271,7 +391,10 @@ def ingest( @app.command(rich_help_panel="Pipeline stages — `run` calls these in order") def refs( manuscript: Path = typer.Argument(..., exists=True), - case: Path = typer.Option(Path("case"), "--case", "-c"), + case: Path = typer.Option( + None, "--case", "-c", + help="Case folder (default: a folder named after the paper, beside the paper)", + ), provided: Path = typer.Option( None, "--provided", help="Folder of reference PDFs you already have; files match by name " @@ -282,14 +405,16 @@ def refs( backend: str = typer.Option("auto", "--backend", help="auto | docling | pymupdf"), ) -> None: """Parse the References section, then retrieve open-access copies with an honest manifest.""" - from .ingest import ingest_pdf, references_section + from .ingest import ingest_pdf, references_span from .models import SourceMap from .refs import parse_references, resolve_all + case = _resolve_case(case, manuscript) # named after the paper unless -c said otherwise # identity first — the cached source map below is a manuscript-derived # artifact, and reading it before the guard is how references from one paper # ended up in a manifest stamped with another paper's hash basis = _guard_case(case, manuscript) + _open_case(case) # before ingest writes into it, so the folder is never briefly untracked ingest_dir = case / "ingest" / "manuscript" cached = ingest_dir / "source_map.json" @@ -300,11 +425,20 @@ def refs( else: smap = ingest_pdf(manuscript, ingest_dir, backend=backend) - entries = parse_references(references_section(smap)) + refs_text, references_resumed = references_span(smap) + entries = parse_references(refs_text) if not entries: console.print("[red]No numbered references found — is there a References section?[/red]") raise typer.Exit(1) console.print(f"parsed [bold]{len(entries)}[/bold] numbered references") + if references_resumed: + # a list interrupted by another section used to end at the interruption: + # 9 of 15 references parsed, and the last 6 never retrieved or checked + console.print( + "[yellow]⚠ the reference list continues past an intervening section and was " + "picked up again — a boundary was crossed, so check the tail of the list " + "above against the paper.[/yellow]" + ) if parse_only: for e in entries: console.print(f" [{e.num:>3}] {e.raw[:90]}") @@ -335,8 +469,8 @@ def tick(e): manuscript=manuscript.name, entries=entries, manuscript_sha256=manuscript_fingerprint(manuscript), # identity, not the name + references_resumed=references_resumed, ) - case.mkdir(parents=True, exist_ok=True) manifest.to_json(case / "refs_manifest.json") ok = len(manifest.retrieved) misses = len(entries) - ok @@ -350,13 +484,17 @@ def tick(e): @app.command(rich_help_panel="Pipeline stages — `run` calls these in order") def scout( - case: Path = typer.Option(Path("case"), "--case", "-c"), + case: Path = typer.Option( + None, "--case", "-c", + help="Case folder holding the audit (required unless ./case exists)", + ), doi: str = typer.Option(None, "--doi", help="DOI of the paper itself (skips the title lookup)"), email: str = typer.Option(None, "--email", envvar=["PAPERTRACE_EMAIL", "MANUSCRIPTAGENT_EMAIL"]), ) -> None: """Scan Europe PMC for literature the reference list doesn't know.""" from .scout import scout_case + case = _stage_case(case) if not (case / "refs_manifest.json").exists(): console.print("[red]refs_manifest.json not found[/red] — run `papertrace refs` first") raise typer.Exit(1) @@ -394,12 +532,16 @@ def scout( @app.command(rich_help_panel="Pipeline stages — `run` calls these in order") def check( - case: Path = typer.Option(Path("case"), "--case", "-c"), + case: Path = typer.Option( + None, "--case", "-c", + help="Case folder holding the audit (required unless ./case exists)", + ), model: str = typer.Option(None, "--model", help="Model override for claude -p"), ) -> None: """Extract citation-backed claims and judge each against its cited source (claude -p).""" from .check import Truncations, check_claims, claude_available, extract_claims + case = _stage_case(case) if not claude_available(): console.print( "[red]The `claude` CLI is required for batch checking[/red] — " @@ -482,12 +624,16 @@ def fail(slug, msg): @app.command(rich_help_panel="Pipeline stages — `run` calls these in order") def highlight( - case: Path = typer.Option(Path("case"), "--case", "-c"), + case: Path = typer.Option( + None, "--case", "-c", + help="Case folder holding the audit (required unless ./case exists)", + ), claim: int = typer.Option(None, "--claim", help="Only this claim id"), ) -> None: """Produce red-box evidence crops for every claim with a page anchor.""" from .highlight import crop_for_anchor, source_page_count + case = _stage_case(case) results = RunResults.from_json(case / "out" / "results.json") out_dir = case / "out" / "evidence" done = 0 @@ -542,7 +688,10 @@ def highlight( @app.command(rich_help_panel="Pipeline stages — `run` calls these in order") def report( - case: Path = typer.Option(Path("case"), "--case", "-c"), + case: Path = typer.Option( + None, "--case", "-c", + help="Case folder holding the audit (required unless ./case exists)", + ), png: bool = typer.Option( False, "--png/--no-png", help="Also export PNG images of the report looks (one-time: playwright install chromium)", @@ -552,6 +701,7 @@ def report( from .models import ScoutResults from .report import write_reports + case = _stage_case(case) results = RunResults.from_json(case / "out" / "results.json") manifest_path = case / "refs_manifest.json" manifest = RefManifest.from_json(manifest_path) if manifest_path.exists() else None @@ -569,7 +719,10 @@ def report( @app.command(rich_help_panel="Start here") def run( manuscript: Path = typer.Argument(..., exists=True), - case: Path = typer.Option(Path("case"), "--case", "-c"), + case: Path = typer.Option( + None, "--case", "-c", + help="Case folder (default: a folder named after the paper, beside the paper)", + ), provided: Path = typer.Option( None, "--provided", help="Folder of reference PDFs you already have; files match by name " @@ -591,7 +744,11 @@ def run( """Full pipeline: ingest → refs → scout → check → highlight → report.""" console.print(BANNER) email = _email(email) # fail fast — before the ingest models load, not after + # resolved once, here, and passed down by keyword: a stage that re-derived + # its own folder could put the same question six times, or disagree + case = _resolve_case(case, manuscript) _guard_case(case, manuscript) # one case folder per paper — never mix two audits + _open_case(case) # KEYWORDS ONLY, deliberately. These stages are Typer commands called as # plain functions, and Typer's declared defaults are OptionInfo objects # rather than the values they display. A positional call therefore breaks diff --git a/src/papertrace/disclosures.py b/src/papertrace/disclosures.py index 7c399cb..04cb2b8 100644 --- a/src/papertrace/disclosures.py +++ b/src/papertrace/disclosures.py @@ -30,6 +30,7 @@ ANCHOR_NOT_LOCATED_TOKEN = "no anchor phrase was found on this page" ANCHOR_UNKNOWN_TOKEN = "anchor match not recorded" SOURCE_IDENTITY_TOKEN = "identity was never confirmed" +REFERENCES_RESUMED_TOKEN = "reference list continued past a section break" @dataclass(frozen=True) @@ -298,6 +299,28 @@ def _source_identity(unverified: list, mismatched: list) -> Disclosure: ) +def _references_resumed(total: int) -> Disclosure: + """The reference list continued past an intervening section. + + Worth the reader's eye in both directions. Crossing the boundary is a guess, + so the numbering of the later entries could be wrong — but *not* crossing it + was the previous behaviour, and that failed silently: a real pre-proof put + refs 1-9 on one page, a declaration section next, then refs 10-15, and the + audit simply reported nine references and never attempted the other six. + """ + return Disclosure( + key="references_resumed", + level="warn", + token=REFERENCES_RESUMED_TOKEN, + text=( + f"The {REFERENCES_RESUMED_TOKEN}, and the parser followed it — all {total} " + "entries here span that boundary. Check the numbering of the later entries " + "against the paper: a list read short would instead have gone unmentioned." + ), + short=f"{REFERENCES_RESUMED_TOKEN} — {total} entries, numbering worth a check", + ) + + def run_disclosures(results, manifest=None) -> list[Disclosure]: """Every run-level disclosure this RunResults owes its reader. @@ -328,6 +351,8 @@ def run_disclosures(results, manifest=None) -> list[Disclosure]: mismatched = [e for e in in_use if e.title_check == "mismatch"] if unverified or mismatched: out.append(_source_identity(unverified, mismatched)) + if getattr(manifest, "references_resumed", False): + out.append(_references_resumed(len(manifest.entries))) return out diff --git a/src/papertrace/highlight.py b/src/papertrace/highlight.py index f137b87..365c0a0 100644 --- a/src/papertrace/highlight.py +++ b/src/papertrace/highlight.py @@ -3,10 +3,17 @@ Boxes come from text search on the PDF — never hand-placed — so a box always sits where the evidence actually is. The crop region defaults to the anchored block's bbox (from the source's own source_map), padded for context. + +A quote can miss the page for typesetting reasons alone: a hyphenated line break +reads as "Non- Hispanic" to `search_for`, and the text the model read says +"Non-Hispanic" or "NonHispanic". `_search` retries such a phrase, but only in +forms the page itself dictates and always by exact search — never by similarity, +so an unlocated anchor still comes back unlocated. """ from __future__ import annotations +import re from pathlib import Path try: @@ -20,6 +27,66 @@ BOX_PAD = 2.0 # breathing room around a matched phrase TARGET_W = 2400 # rendered crop width in pixels (retina-ish) +_DASHES = "-\u2010\u2011" # hyphen-minus, Unicode hyphen, non-breaking hyphen +_MAX_VARIANTS = 8 # a phrase full of dashes is not worth a combinatorial search +_LINE_BREAK_HYPHEN = re.compile(rf"(\w+)[{_DASHES}\u00ad]\n(\w+)") + + +def _dash_variants(phrase: str) -> list[str]: + """The phrase, then one-space-at-one-dash variants of it. + + A page that breaks a hyphenated word across lines reads as "Non- Hispanic" + to `search_for` — the line break is a space — and a model quoting that + passage tidies it to "Non-Hispanic", which that occurrence does not carry. + Only + whitespace beside a dash the phrase already has moves here: no character is + invented, no similarity is computed, and every candidate is still located by + exact text search, so a hit sits on text the page really carries. + """ + variants = [phrase] + for i, ch in enumerate(phrase): + if ch not in _DASHES or i == 0 or i == len(phrase) - 1: + continue + rest = phrase[i + 1:] + variant = phrase[: i + 1] + (rest.lstrip(" ") if rest[0] == " " else " " + rest) + if variant not in variants: + variants.append(variant) + if len(variants) >= _MAX_VARIANTS: + break + return variants + + +def _as_the_page_breaks_it(page, phrase: str) -> str: + """`phrase` with every word this page splits at a hyphen put back as it reads. + + The ingest text says "approximately" and the page says "approxi-" / newline + "mately"; `search_for` reads that break as a space, so the phrase is on the + page only as "approxi- mately". The rewrite is dictated by the page — no + split position is guessed, and a word the page does not break is untouched. + """ + for tail, head in _LINE_BREAK_HYPHEN.findall(page.get_text()): + joined = tail + head + if joined in phrase: + phrase = re.sub(rf"(? list: + """Locate `phrase` on `page`; retry the page's own hyphenation of it. + + Every candidate is still an exact `search_for`, and every difference from + the model's phrase comes from the page: whitespace beside a dash the phrase + already carries, or a break the page itself makes. Returns [] when the page + carries none of them — a genuine miss stays a miss, and the caller records + anchor_located = False rather than boxing something that resembles the quote. + """ + for candidate in _dash_variants(phrase): + hits = page.search_for(candidate, clip=clip) + if hits: + return hits + broken = _as_the_page_breaks_it(page, phrase) # last, it costs a text extraction + return page.search_for(broken, clip=clip) if broken != phrase else [] + def source_page_count(pdf_path: Path) -> int: """Pages in a resolved source — for bounds-checking a page the model named.""" @@ -62,7 +129,7 @@ def crop_evidence( boxes = 0 for phrase in phrases: - for hit in page.search_for(phrase, clip=rect): + for hit in _search(page, phrase, clip=rect): x0 = (hit.x0 - BOX_PAD - rect.x0) * zoom y0 = (hit.y0 - BOX_PAD - rect.y0) * zoom x1 = (hit.x1 + BOX_PAD - rect.x0) * zoom @@ -116,7 +183,7 @@ def crop_for_anchor(anchor, claim_id: int, sources_dir: Path, ingest_root: Path, # fall back to the union of phrase hits on the page, padded doc = fitz.open(pdf) page = doc[anchor.source_page - 1] - hits = [h for p in anchor.anchor_phrases for h in page.search_for(p)] + hits = [h for p in anchor.anchor_phrases for h in _search(page, p)] doc.close() if not hits: # the phrases were searched against the real page and matched diff --git a/src/papertrace/ingest/__init__.py b/src/papertrace/ingest/__init__.py index ecc093c..9ba12c7 100644 --- a/src/papertrace/ingest/__init__.py +++ b/src/papertrace/ingest/__init__.py @@ -19,9 +19,9 @@ from pathlib import Path from ..models import Block, SourceMap -from .pymupdf_ import ingest_blocks_pymupdf, references_section +from .pymupdf_ import ingest_blocks_pymupdf, references_section, references_span -__all__ = ["ingest_pdf", "references_section", "available_backends"] +__all__ = ["ingest_pdf", "references_section", "references_span", "available_backends"] def _docling_available() -> bool: diff --git a/src/papertrace/ingest/docling_.py b/src/papertrace/ingest/docling_.py index 583cf0f..081953b 100644 --- a/src/papertrace/ingest/docling_.py +++ b/src/papertrace/ingest/docling_.py @@ -9,6 +9,13 @@ Coordinate note: docling reports bounding boxes with a BOTTOMLEFT origin; PyMuPDF (and our highlight step) use TOPLEFT. `_to_top_left` converts using the page height — getting this wrong mirrors every red box vertically. + +Fidelity note: docling deletes the hyphen when it joins a word split across two +lines, which is right for a syllabic break ("approxi-" / "mately") and wrong for +a lexical one — a page reading "Non-" / "Hispanic" arrives as "NonHispanic". +`hyphen_joins` separates the two by evidence from the document itself, never by +guessing at the joined text; `restore_hyphen_joins` repairs only what it proved. +Locating a quote on the page is a separate job, and stays in `highlight.py`. """ from __future__ import annotations @@ -17,14 +24,26 @@ import os import re from contextlib import contextmanager +from dataclasses import replace from pathlib import Path +try: + import pymupdf as fitz # PyMuPDF >= 1.24 module name (the bare `fitz` import is deprecated) +except ImportError: # pragma: no cover — older PyMuPDF exposes only `fitz` + import fitz + from ..models import Block _HEADING_LABELS = {"section_header", "title"} _LIST_LABELS = {"list_item"} _SKIP_LABELS = {"page_header", "page_footer", "footnote"} +# the characters docling treats as a line-break hyphen and deletes: ASCII +# hyphen-minus (its `\x02` soft-hyphen marker is rewritten to this before the +# join), Unicode hyphen, non-breaking hyphen, soft hyphen +_JOIN_DASHES = "-\u2010\u2011\u00ad" +_LINE_BREAK_HYPHEN = re.compile(rf"(\w+)[{_JOIN_DASHES}]\n(\w+)") + def _to_top_left(bbox, page_height: float) -> tuple[float, float, float, float]: """Convert a docling BoundingBox to top-left-origin (x0, y0, x1, y1).""" @@ -119,6 +138,65 @@ def blocks_from_docling(doc, page_heights: dict[int, float]) -> list[Block]: return blocks +def hyphen_joins(pdf_path: Path) -> dict[int, dict[str, str]]: + """docling's dehyphenated word → the compound the document itself proves, + per 1-based page number. + + docling deletes the hyphen whenever it joins a word split across two lines. + That is *right* far more often than it is wrong — "approxi-" / "mately" is + the single word "approximately", and on the paper this was measured against + 87 of 94 breaks were of that kind — and wrong when the hyphen belongs to the + word: "Non-" / "Hispanic" is "Non-Hispanic", never "NonHispanic". + + Nothing in the joined text tells the two apart. A lower→upper junction is + not evidence (`HbA1c`, `PaperTrace`, `Timedependent` all have one, and only + the last is broken), and "missioncritical" has no junction at all. So the + document is asked instead: an entry appears only where the hyphenated form + occurs somewhere in the PDF **unbroken**, which is the author writing the + compound out. No evidence, no entry — the text then stays exactly as docling + produced it rather than being repaired by guesswork. + """ + doc = fitz.open(pdf_path) + try: + pages = [doc[index].get_text() for index in range(doc.page_count)] + finally: + doc.close() + whole = "\n".join(pages) # the compound may be written out on any page + + joins: dict[int, dict[str, str]] = {} + for number, raw in enumerate(pages, start=1): + found: dict[str, str] = {} + for tail, head in _LINE_BREAK_HYPHEN.findall(raw): + joined, compound = tail + head, f"{tail}-{head}" + if compound not in whole: + continue # nowhere written out: no evidence the hyphen is lexical + if re.search(rf"(? list[Block]: + """Put back a hyphen docling deleted, for the joins `hyphen_joins` proved. + + Word-bounded so a short join can never land inside an unrelated word, and + scoped to the block's page so one page's break cannot rewrite another's. + A block docling got right is returned untouched. + """ + out: list[Block] = [] + for block in blocks: + text = block.text + for joined, on_page in joins.get(block.page, {}).items(): + if joined in text: + text = re.sub(rf"(? None: """docling's model stack (RapidOCR, torch dynamo, transformers) floods the terminal with INFO/WARNING logs, tqdm weight-loading bars and torch @@ -207,4 +285,8 @@ def ingest_blocks_docling(pdf_path: Path) -> tuple[int, list[Block], str]: pages = len(page_heights) or 1 version = getattr(docling, "__version__", "?") - return pages, blocks_from_docling(doc, page_heights), version + blocks = blocks_from_docling(doc, page_heights) + # only where the paper writes the compound out somewhere: docling's join is + # the author's word for a syllabic break, and wrong for a lexical hyphen + blocks = restore_hyphen_joins(blocks, hyphen_joins(pdf_path)) + return pages, blocks, version diff --git a/src/papertrace/ingest/pymupdf_.py b/src/papertrace/ingest/pymupdf_.py index 27c7db2..1c475ef 100644 --- a/src/papertrace/ingest/pymupdf_.py +++ b/src/papertrace/ingest/pymupdf_.py @@ -73,8 +73,23 @@ def ingest_blocks_pymupdf(pdf_path: Path) -> tuple[int, list[Block]]: return pages, blocks -def references_section(smap: SourceMap) -> str: - """Return the text of the References/Bibliography section, if found. +# A reference list interrupted by another section is not necessarily over, but +# resuming across the break is a guess, so it takes two independent signals. +# +# `list` only, never `text`: docling types reference entries as `list`, which is +# structurally distinct from prose. Under the flat backend they are `text` — +# the same type as every paragraph in the paper — so resuming on a run of +# `text` would swallow the Discussion of any paper whose references are not +# last. That asymmetry is the whole reason this is restricted rather than +# general, and it means a *flat-ingested* split list is still parsed short. +_RESUMABLE_TYPE = "list" +# one bulleted block after an unrelated heading is far more likely a sentence +# than the tail of a bibliography +_MIN_RESUME_RUN = 2 + + +def references_span(smap: SourceMap) -> tuple[str, bool]: + """Reference-list text, and whether it was resumed across a section break. A block whose *entire* text is the heading word counts as the heading even when ingest typed it as body text. Flat-text ingest guesses headings from @@ -84,18 +99,57 @@ def references_section(smap: SourceMap) -> str: Requiring the whole block to be the word, not merely to start with it, is what keeps "References were checked by hand" from swallowing the paper. + + The second return value says the list was picked up again after an + intervening section. That is worth surfacing rather than hiding: a real + pre-proof put refs 1-9 on page 7, `Declaration of interests` next, then refs + 10-15 on page 8, and stopping at the first header lost six sources without + saying so. Only reference-shaped runs are collected, so the prose of the + intervening section never enters the list — which matters because + `_parse_bulleted` appends a non-bullet line to the *previous* entry, so a + stray paragraph corrupts a reference rather than merely adding noise. """ - started = False - out: list[str] = [] - for b in smap.blocks: + blocks = smap.blocks + start = next( # the SAME rule coverage_audit uses — see models.is_references_heading - if is_references_heading(b.type, b.text): - if started: - break - started = True + (i + 1 for i, b in enumerate(blocks) if is_references_heading(b.type, b.text)), + None, + ) + if start is None: + return "", False + + out: list[str] = [] + entry_type: str | None = None + i = start + while i < len(blocks): + b = blocks[i] + if b.type == "sectionheader" or is_references_heading(b.type, b.text): + break # the next real heading ends the contiguous run + if entry_type is None: + entry_type = b.type + out.append(b.text) + i += 1 + + if not out or entry_type != _RESUMABLE_TYPE: + return "\n".join(out), False + + resumed = False + rest = blocks[i:] + k = 0 + while k < len(rest): + if rest[k].type != entry_type: + k += 1 continue - if b.type == "sectionheader" and started: - break # the next real heading ends the list - if started: - out.append(b.text) - return "\n".join(out) + j = k + while j < len(rest) and rest[j].type == entry_type: + j += 1 + if j - k >= _MIN_RESUME_RUN: + out.extend(b.text for b in rest[k:j]) + resumed = True + k = j + return "\n".join(out), resumed + + +def references_section(smap: SourceMap) -> str: + """The reference-list text. See `references_span` for the resume signal.""" + return references_span(smap)[0] diff --git a/src/papertrace/models.py b/src/papertrace/models.py index 0b2373c..6ca7d52 100644 --- a/src/papertrace/models.py +++ b/src/papertrace/models.py @@ -161,6 +161,12 @@ class RefManifest: # before content hashing — those fall back to comparing the file name, and # say so; they self-heal on the next `papertrace refs`. manuscript_sha256: str | None = None + # the reference list was picked up again after an intervening section, so + # the entry numbering spans a boundary the parser chose to cross. It is a + # guess — a defensible one, gated on block type and run length — and the + # reader has to be able to check it, because the alternative failure is + # silent: a list parsed short simply reports fewer references. + references_resumed: bool = False @property def retrieved(self) -> list[RefEntry]: @@ -174,6 +180,7 @@ def to_json(self, path: Path) -> None: payload = { "manuscript": self.manuscript, "manuscript_sha256": self.manuscript_sha256, + "references_resumed": self.references_resumed, "summary": { "total": len(self.entries), "available": len(self.retrieved), @@ -192,6 +199,8 @@ def from_json(cls, path: Path) -> RefManifest: manuscript=data["manuscript"], entries=[RefEntry(**e) for e in data["entries"]], manuscript_sha256=data.get("manuscript_sha256"), + # .get: a manifest written before this field must still load + references_resumed=bool(data.get("references_resumed", False)), ) diff --git a/src/papertrace/refs.py b/src/papertrace/refs.py index f8532b6..fb02177 100644 --- a/src/papertrace/refs.py +++ b/src/papertrace/refs.py @@ -218,6 +218,31 @@ def _download_pdf(client: httpx.Client, url: str, dest: Path) -> bool: "elsevier", "springer", "wiley", "volume", "press", "https"} ) +_URL_RE = re.compile(r"(?:https?://|www\.)\S+", re.I) + + +def _title_tokens(raw: str) -> set[str]: + """The reference's own distinctive words — URLs removed first. + + A URL is not part of a title, and a *tracking parameter* least of all: + `?utm_source=chatgpt.com` on a cited news page contributed `chatgpt` and + `source` to this set, and the wrong paper Crossref returned was an + editorial about ChatGPT. Path segments do the same from the other side, + inflating the denominator with `firstmedical`, `assuranceprogram` and + `publications` — words no first page will carry, so they dilute the ratio + the check is measured on. + """ + return set(re.findall(r"[a-z]{5,}", _URL_RE.sub(" ", raw).lower())) - _TITLE_STOPWORDS + + +# Four distinct words, not three. The observed false positive cleared the 0.35 +# ratio on `artificial`, `intelligence` and `medical` — three words that are the +# subject of most papers in this field, so no stopword list can retire them +# without rejecting correct matches. Falling below the floor yields +# `unverifiable`, never `mismatch`: too few words to tell is not evidence of a +# different paper, and a `mismatch` would discard a possibly-correct download. +_TITLE_MIN_MATCHES = 4 + # the three answers the check can give. "unverifiable" used to share `None` # with "verified", so a scanned PDF someone supplied by hand was reported as @@ -237,11 +262,17 @@ def _title_check_text(raw: str, page_text: str) -> tuple[str, str]: page = re.sub(r"\s+", " ", page_text).lower() if not page.strip(): return TITLE_UNVERIFIABLE, "no readable text on its first page (scanned or image-only)" - tokens = set(re.findall(r"[a-z]{5,}", raw.lower())) - _TITLE_STOPWORDS + tokens = _title_tokens(raw) if not tokens: return TITLE_UNVERIFIABLE, "the reference string has no distinctive words to match on" found = sum(1 for t in tokens if t in page) if found / len(tokens) >= 0.35: + if found < _TITLE_MIN_MATCHES: + return ( + TITLE_UNVERIFIABLE, + f"only {found} of the reference's {len(tokens)} distinctive words appear on its " + "first page — too few to tell this paper from another on the same subject", + ) return TITLE_VERIFIED, f"{found}/{len(tokens)} reference tokens on its first page" return ( TITLE_MISMATCH, @@ -334,6 +365,56 @@ def _match_provided(entry: RefEntry, provided_dir: Path | None) -> Path | None: return candidates[0] if candidates else None +# The structural marks of a journal article besides a DOI: an identifier, a +# volume, a page range, a `volume:page` pair. Their ABSENCE is what the webpage +# gate keys on, so this set is deliberately small — every pattern added here +# sends one more reference into a title search. +_ARTICLE_SIGNAL_RE = re.compile( + r"\bdois?\b" + r"|\bpm(?:id|cid)\b" + r"|\barxiv\b|\bbiorxiv\b|\bmedrxiv\b|\bssrn\b|\bisbn\b" + r"|\bvol(?:ume)?\b\.?\s*\d" # vol. 12 / volume 12 + r"|\bpp?\b\.\s*\d" # p. 225 / pp. 225-232 + r"|\b\d+\s*\(\s*\d+\s*\)\s*[:,]?\s*\d" # 89(1061):225 + r"|\b\d+\s*:\s*e?\d" # 11:2624 / 5:e230024 + # a page range: 1068-1083. Two four-digit years either side of the dash are + # a date span in a headline ("digital health 2020-2025"), not pages, and a + # journal citation that really does span 1981-1990 carries its volume with + # it — `388:1981` matches the pattern above. + r"|(? bool: + """Is this reference a web page rather than an article? + + A URL plus none of the structural marks of an article. Both halves matter: + publishers' own reference styles print a link beside the volume and page + range, and those references resolve well — while a reference with no URL at + all is exactly what a bibliographic search is for. + + Two alternatives were weighed and rejected. Judging by how much of the + string is URL measures nothing: a news page cited with a long headline and a + short link scores low, a journal reference carrying a long publisher link + scores high. A domain or TLD list is an arms race with every press office, + newsroom and society website in existence. + + The error this accepts is the harmless one. A wrongly gated article ends at + `no_doi` — a recorded gap; a wrongly searched web page ends with a real + paper downloaded, title-checked against a news headline and judged for + claims it never made. + """ + if not _URL_RE.search(raw): + return False # no link: nothing here suggests a web page + if DOI_RE.search(raw): + return False # a DOI anywhere counts, including inside the link itself + # the rest of the marks are looked for with the URL removed, so that a path + # segment or a query string cannot impersonate a volume or a page range + return not _ARTICLE_SIGNAL_RE.search(_URL_RE.sub(" ", raw)) + + def resolve_entry( entry: RefEntry, dest_dir: Path, @@ -377,6 +458,22 @@ def resolve_entry( if _accept(entry, client, url, dest, "arxiv", "arXiv"): return entry + # A bibliographic title search always returns *something*, and for a web + # page that something is a confident wrong answer: `ACR launches first + # medical practice artificial intelligence QA program` fetched an ACR + # Open Rheumatology editorial (American College of Rheumatology, not + # Radiology), which then passed the title check on the shared vocabulary. + # A news page was never retrievable as a PDF anyway, so nothing is lost. + if not entry.doi and _is_webpage_reference(entry.raw): + entry.status = "no_doi" + entry.reason = ( + "this reference is a web page, not an article — no DOI, and no volume, " + "page range or identifier to look one up with. Not searched by title: " + "Crossref would answer with the closest-looking journal article, and " + "judging a claim against that is worse than recording the gap" + ) + return entry + if not entry.doi: try: entry.doi = _crossref_doi(client, entry.raw, email) diff --git a/src/papertrace/templates/report_terminal.html.j2 b/src/papertrace/templates/report_terminal.html.j2 index a7609f0..62d8c85 100644 --- a/src/papertrace/templates/report_terminal.html.j2 +++ b/src/papertrace/templates/report_terminal.html.j2 @@ -80,6 +80,9 @@ {% endfor %} {% for d in disclosures if d.key == "source_identity" %}
▸ resolve⚠ {{ d.short }}
+{% endfor %} +{% for d in disclosures if d.key == "references_resumed" %} +
▸ resolve⚠ {{ d.short }}
{% endfor %}
▸ check{{ r.claims|length }} citation-backed claims · reading each against its cited page{% if r.uncited %} · {{ r.uncited|length }} uncited assertions flagged{% endif %}
{% if scout and not scout.error %} diff --git a/src/papertrace/wizard.py b/src/papertrace/wizard.py index b21af34..912cb42 100644 --- a/src/papertrace/wizard.py +++ b/src/papertrace/wizard.py @@ -258,9 +258,15 @@ def equivalent_command( def _suggest_case(pdf: Path) -> str: - """A case-folder name the user can accept with one keystroke.""" - stem = re.sub(r"[^A-Za-z0-9._-]+", "-", pdf.stem).strip("-.") - return stem[:60] or "case" + """The folder batch mode would pick, offered for one keystroke. + + Delegated rather than reimplemented: two answers to "where does this audit + live" is how the wizard and `papertrace run` came to disagree in the first + place. Imported inside the function — `cli` imports this module. + """ + from .cli import default_case + + return str(default_case(pdf)) def _ask_paper() -> Path: diff --git a/tests/test_case_identity.py b/tests/test_case_identity.py index e50af8e..70c43cd 100644 --- a/tests/test_case_identity.py +++ b/tests/test_case_identity.py @@ -1,4 +1,4 @@ -"""One case folder, one paper — including when the folder predates hashing. +"""One case folder, one paper — which folder that is, and that it holds one paper. A case folder is the unit of work and `_guard_case` is what keeps two audits from mixing. The gap this file pins is an ordering one: `refs` used to parse @@ -6,10 +6,18 @@ stamp the supplied manuscript's hash onto the resulting manifest. A legacy case plus a same-named different PDF therefore produced a manifest that looked content-verified while describing the previous paper. + +The second half of the file covers *which* folder an audit lands in: every +command used to default to a folder literally named `case`, so consecutive +audits of different papers piled into one folder unless the user remembered +`-c`, and the folder appeared in whatever directory they happened to be in. +The last test checks the commands the `/review` skill tells an agent to run — +documentation is the only interface those lines have to the CLI. """ import json import sys +import types from pathlib import Path import pymupdf @@ -112,3 +120,164 @@ def test_a_different_paper_in_a_hashed_case_is_still_refused(tmp_path, offline): cli.refs(manuscript=two, case=case, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") assert e.value.exit_code == 2 + + +@pytest.fixture() +def terminal(monkeypatch): + """A tty whose answers are scripted; an unscripted question is an error. + + Returned list is the answer queue — leaving it empty asserts that nothing + was asked, because `Prompt.ask` then raises IndexError. + """ + monkeypatch.setattr(cli.sys, "stdin", types.SimpleNamespace(isatty=lambda: True)) + answers: list[str] = [] + + class _Prompt: + @staticmethod + def ask(*_a, **_kw): + return answers.pop(0) + + monkeypatch.setattr(cli, "Prompt", _Prompt) + return answers + + +@pytest.fixture() +def piped(monkeypatch): + """No tty: a pipe, a cron job, CI. Nothing may block on stdin.""" + monkeypatch.setattr(cli.sys, "stdin", types.SimpleNamespace(isatty=lambda: False)) + + +def test_the_case_folder_defaults_to_the_papers_own_name(tmp_path, offline, monkeypatch): + """The reported defect: a batch run wrote into `case/` in whatever directory + the user stood in — a git clone's root, in the report — so a second paper + landed on the first. The folder is named after the paper and sits beside it. + """ + (elsewhere := tmp_path / "elsewhere").mkdir() + monkeypatch.chdir(elsewhere) # the audit must not follow the user's cwd + pdf = _paper(tmp_path / "papers" / "PIIS0720048X2600522X.pdf", "ONE", "10.1000/one") + + cli.refs(manuscript=pdf, case=None, provided=None, email="test@example.org", + parse_only=False, backend="pymupdf") + + derived = tmp_path / "papers" / "PIIS0720048X2600522X" + assert (derived / "refs_manifest.json").exists(), "audit did not land in the derived folder" + assert not (Path.cwd() / "case").exists(), "still scattering a `case/` into the cwd" + # the folder is named after a manuscript, so .gitignore's `case/` no longer + # covers it — a case folder created for the user ignores itself + assert (derived / ".gitignore").read_text() == "*\n" + + +def test_an_explicit_case_flag_still_wins(tmp_path, offline, monkeypatch, terminal): + """`-c` is an instruction, not a suggestion — and never asks a question.""" + monkeypatch.chdir(tmp_path) + pdf = _paper(tmp_path / "papers" / "alpha.pdf", "ALPHA", "10.1000/alpha") + chosen = tmp_path / "mycase" + + for _ in range(2): # twice: an explicit re-run is not interrogated either + cli.refs(manuscript=pdf, case=chosen, provided=None, email="test@example.org", + parse_only=False, backend="pymupdf") + + assert (chosen / "refs_manifest.json").exists() + assert not (tmp_path / "papers" / "alpha").exists() + + +def test_a_rerun_of_the_same_paper_can_amend_its_case(tmp_path, offline, monkeypatch, terminal): + """The "I added more source PDFs" case: reuse the folder, pick the new ones up.""" + monkeypatch.chdir(tmp_path) + pdf = _paper(tmp_path / "papers" / "beta.pdf", "BETA", "10.1000/beta") + derived = tmp_path / "papers" / "beta" + + cli.refs(manuscript=pdf, case=None, provided=None, email="test@example.org", + parse_only=False, backend="pymupdf") + terminal.append("amend") + cli.refs(manuscript=pdf, case=None, provided=None, email="test@example.org", + parse_only=False, backend="pymupdf") + + assert terminal == [], "the collision was not put to the user" + assert (derived / "refs_manifest.json").exists() + assert not (tmp_path / "papers" / "beta-2").exists(), "amend must not open a second folder" + + +def test_a_rerun_can_start_a_fresh_numbered_case(tmp_path, offline, monkeypatch, terminal): + """The other branch: leave the first audit intact, start the paper over.""" + monkeypatch.chdir(tmp_path) + pdf = _paper(tmp_path / "papers" / "gamma.pdf", "GAMMA", "10.1000/gamma") + + cli.refs(manuscript=pdf, case=None, provided=None, email="test@example.org", + parse_only=False, backend="pymupdf") + first = (tmp_path / "papers" / "gamma" / "refs_manifest.json").read_bytes() + terminal.append("fresh") + cli.refs(manuscript=pdf, case=None, provided=None, email="test@example.org", + parse_only=False, backend="pymupdf") + + assert (tmp_path / "papers" / "gamma-2" / "refs_manifest.json").exists() + assert (tmp_path / "papers" / "gamma" / "refs_manifest.json").read_bytes() == first + + +def test_a_rerun_without_a_terminal_amends_and_says_so(tmp_path, offline, monkeypatch, piped, + capsys): + """A pipe, a cron job or CI has nobody to answer. The documented choice is + amend: it is what the previous default did for a re-run of the same paper, + it destroys nothing, and it keeps the output path predictable — `fresh` + would silently move the report somewhere a caller cannot name. + """ + monkeypatch.chdir(tmp_path) + pdf = _paper(tmp_path / "papers" / "delta.pdf", "DELTA", "10.1000/delta") + + cli.refs(manuscript=pdf, case=None, provided=None, email="test@example.org", + parse_only=False, backend="pymupdf") + capsys.readouterr() + cli.refs(manuscript=pdf, case=None, provided=None, email="test@example.org", + parse_only=False, backend="pymupdf") + out = capsys.readouterr().out + + assert (tmp_path / "papers" / "delta" / "refs_manifest.json").exists() + assert not (tmp_path / "papers" / "delta-2").exists() + assert "amend" in out, out + + +def test_a_replaced_paper_of_the_same_name_is_still_refused(tmp_path, offline, monkeypatch, + terminal): + """The derived name collides only when the file itself was replaced — v2 + saved over v1. That is a different paper in an existing case, so the hard + guard owns it: exit 2, and no amend/fresh question (`terminal` is empty). + """ + monkeypatch.chdir(tmp_path) + path = tmp_path / "papers" / "epsilon.pdf" + _paper(path, "FIRST", "10.1000/first") + cli.refs(manuscript=path, case=None, provided=None, email="test@example.org", + parse_only=False, backend="pymupdf") + + _paper(path, "SECOND", "10.1000/second") # same name, different paper + with pytest.raises(typer.Exit) as e: + cli.refs(manuscript=path, case=None, provided=None, email="test@example.org", + parse_only=False, backend="pymupdf") + assert e.value.exit_code == 2 + + +def test_run_derives_one_case_folder_and_hands_it_to_every_stage(tmp_path, offline, monkeypatch): + """`run` resolves once and passes the result down by keyword, so no stage + re-derives a folder of its own and the collision is put once, not six times. + """ + monkeypatch.chdir(tmp_path) + pdf = _paper(tmp_path / "papers" / "zeta.pdf", "ZETA", "10.1000/zeta") + seen: dict[str, dict] = {} + for name in ("ingest", "refs", "scout", "check", "highlight", "report"): + monkeypatch.setattr(cli, name, (lambda n: lambda **kw: seen.__setitem__(n, kw))(name)) + + cli.run(manuscript=pdf, case=None, provided=None, email="test@example.org", model=None, + png=False, backend="pymupdf", with_scout=True, doi=None) + + derived = tmp_path / "papers" / "zeta" + assert set(seen) == {"ingest", "refs", "scout", "check", "highlight", "report"} + assert {n: kw["case"] for n, kw in seen.items()} == dict.fromkeys(seen, derived) + + +def test_the_wizard_suggests_the_folder_batch_mode_would_use(tmp_path): + """One answer to "where does this audit live", not two.""" + from papertrace import wizard + + pdf = tmp_path / "papers" / "eta.pdf" + pdf.parent.mkdir(parents=True) + pdf.write_bytes(b"%PDF-1.4\n") + assert wizard._suggest_case(pdf) == str(cli.default_case(pdf)) diff --git a/tests/test_disclosure_parity.py b/tests/test_disclosure_parity.py index d75a606..31af3df 100644 --- a/tests/test_disclosure_parity.py +++ b/tests/test_disclosure_parity.py @@ -224,3 +224,71 @@ def test_the_terminal_template_names_every_disclosure_key_that_exists(): f"report_terminal.html.j2 renders no branch for {sorted(missing)} — " "its filter is an allow-list, so a new disclosure is dropped, not surfaced" ) + + +# --- a reference list read across a section break says so ------------------ + + +def test_a_resumed_reference_list_is_disclosed_in_all_three_formats(tmp_path): + """Crossing a section boundary to finish the bibliography is a guess, and + the numbering of the later entries depends on it. The previous behaviour — + stopping at the first heading — failed the other way and failed silently: + a real paper reported 9 references and never attempted the other 6.""" + from papertrace.models import RefEntry, RefManifest + + manifest = RefManifest( + manuscript="m.pdf", + entries=[RefEntry(num=str(i), raw=f"Author {i}. A paper. 2020.", status="paywalled") + for i in range(1, 16)], + references_resumed=True, + ) + results = RunResults(manuscript="m.pdf", converter="pymupdf", claims=[_claim()]) + + write_reports(results, manifest, tmp_path, png=False) + rendered = {name: (tmp_path / name).read_text() for name in FORMATS} + + d = next((x for x in run_disclosures(results, manifest) if x.key == "references_resumed"), None) + assert d is not None, [x.key for x in run_disclosures(results, manifest)] + for name, body in rendered.items(): + assert d.token in body, f"token {d.token!r} missing from {name}" + + +def test_an_uninterrupted_reference_list_adds_no_warning(tmp_path): + """An ordinary paper must not grow a caveat it has not earned.""" + from papertrace.models import RefEntry, RefManifest + + manifest = RefManifest( + manuscript="m.pdf", + entries=[RefEntry(num="1", raw="Author. A paper. 2020.", status="retrieved")], + ) + fired = run_disclosures( + RunResults(manuscript="m.pdf", converter="pymupdf", claims=[_claim()]), manifest + ) + assert not any(d.key == "references_resumed" for d in fired) + + +def test_references_resumed_round_trips_and_older_manifests_still_load(tmp_path): + """New field ⇒ schema update plus a round-trip test, and absent-safe: a + manifest written before this field has no such key.""" + import json as _json + + import jsonschema + + from papertrace.models import RefEntry, RefManifest + + path = tmp_path / "refs_manifest.json" + RefManifest( + manuscript="m.pdf", + entries=[RefEntry(num="1", raw="Author. A paper. 2020.", status="retrieved")], + references_resumed=True, + ).to_json(path) + + schema_path = Path(__file__).resolve().parent.parent / "schemas" / "refs_manifest.schema.json" + payload = _json.loads(path.read_text()) + jsonschema.validate(payload, _json.loads(schema_path.read_text())) + assert RefManifest.from_json(path).references_resumed is True + + del payload["references_resumed"] # a pre-field manifest + path.write_text(_json.dumps(payload)) + jsonschema.validate(_json.loads(path.read_text()), _json.loads(schema_path.read_text())) + assert RefManifest.from_json(path).references_resumed is False diff --git a/tests/test_ingest_backends.py b/tests/test_ingest_backends.py index 0f19df7..2497c99 100644 --- a/tests/test_ingest_backends.py +++ b/tests/test_ingest_backends.py @@ -12,10 +12,21 @@ sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) +from papertrace.highlight import crop_evidence # noqa: E402 from papertrace.ingest import ingest_pdf, write_outputs # noqa: E402 -from papertrace.ingest.docling_ import _to_top_left, blocks_from_docling # noqa: E402 +from papertrace.ingest.docling_ import ( # noqa: E402 + _to_top_left, + blocks_from_docling, + hyphen_joins, + restore_hyphen_joins, +) from papertrace.models import Block, SourceMap # noqa: E402 +try: + import pymupdf as fitz # noqa: E402 +except ImportError: # pragma: no cover — older PyMuPDF exposes only `fitz` + import fitz # noqa: E402 + # --------------------------------------------------------------------------- # stub docling objects # --------------------------------------------------------------------------- @@ -129,6 +140,144 @@ def test_blocks_from_docling_full_taxonomy(): assert [b.id for b in blocks] == [f"block_{i:04d}" for i in range(1, 6)] +def test_a_picture_reaches_the_model_as_its_caption_only(): + """Characterization, not a fix: a `picture` item's own text is not read. + + docling carries in-figure text — when its layout model finds a text region + inside the figure — as separate nested text items, never on the picture + item, so the adapter has nothing to lose here. Pinned because the README + now states what a figure region delivers to the judge. + """ + pic = StubPicture(prov=_prov(t=590, b=400)) + pic.text = "97% completed follow-up" # docling's PictureItem has no such field + blocks = blocks_from_docling(StubDoc([pic]), {1: 842.0}) + assert [b.text for b in blocks] == ["[FIGURE: Figure 3. Forest plot.]"] + + +def test_in_figure_text_survives_when_docling_emits_it_as_a_text_item(): + """The flat adapter keeps a text item that sits inside a figure's region. + + Whether docling emits one is docling's decision, not ours — on the paper + this was measured against it emitted none for any of nine figures. + """ + doc = StubDoc( + [ + StubPicture(prov=_prov(t=590, b=400)), + StubText("DocItemLabel.TEXT", "97% completed follow-up", _prov(t=520, b=500)), + ] + ) + blocks = blocks_from_docling(doc, {1: 842.0}) + assert [b.type for b in blocks] == ["picture", "text"] + assert blocks[1].text == "97% completed follow-up" + + +# --------------------------------------------------------------------------- +# hyphenated line breaks — docling deletes the hyphen, the page still has one +# --------------------------------------------------------------------------- + + +def _hyphen_pdf(tmp_path): + """A page that breaks words at a hyphen across lines, as typeset pages do. + + Line 1 writes "Non-Hispanic" out unbroken; lines 2-3 break it. That pairing + is the whole evidence rule. + """ + doc = fitz.open() + page = doc.new_page() + page.insert_text((72, 100), "Regarding race, white Non-Hispanic patients were", fontsize=11) + page.insert_text((72, 114), "prevalent in each subgroup, followed by Asian, Non-", fontsize=11) + page.insert_text((72, 128), "Hispanic; and HbA1c was measured in PaperTrace.", fontsize=11) + page.insert_text((72, 170), "The cohort was assessed approxi-", fontsize=11) + page.insert_text((72, 184), "mately once a year at https://x.org/Data-Science-and-", fontsize=11) + page.insert_text((72, 198), "Informatics/report as agreed.", fontsize=11) + # page 2: the joined form is also a real word here, so the break is ambiguous + page2 = doc.new_page() + page2.insert_text((72, 100), "The column NonHispanic holds the Non-Hispanic flag, and", fontsize=11) + page2.insert_text((72, 114), "white Non-", fontsize=11) + page2.insert_text((72, 128), "Hispanic is its label.", fontsize=11) + pdf = tmp_path / "hyphen.pdf" + doc.save(pdf) + doc.close() + return pdf + + +def test_hyphen_joins_takes_the_compound_from_the_document(tmp_path): + joins = hyphen_joins(_hyphen_pdf(tmp_path)) + # the paper writes "Non-Hispanic" out on line 1, so the break on line 2 is + # a hyphen docling should not have eaten + assert joins[1]["NonHispanic"] == "Non-Hispanic" + # a syllabic break: docling's "approximately" IS the author's word, and + # "approxi-mately" appears nowhere — no entry, nothing rewritten + assert "approximately" not in joins[1] + # the URL wrap breaks at a hyphen that belongs to the URL, but the document + # never writes it out unbroken — unproven, so not claimed + assert "andInformatics" not in joins[1] + # mid-line hyphens and camel-cased words are not breaks at all + assert "HbA1c" not in joins[1] and "PaperTrace" not in joins[1] + # page 2 carries "NonHispanic" as a word of its own: which of the two a + # rewrite would hit is unknowable, so nothing is claimed there + assert "NonHispanic" not in joins.get(2, {}) + + +def test_restore_hyphen_joins_rewrites_only_the_proven_join(tmp_path): + joins = {1: {"NonHispanic": "Non-Hispanic"}} + blocks = [ + Block("block_0001", "text", 1, (0, 0, 1, 1), [], + "Asian, NonHispanic; white Non-Hispanic; HbA1c in PaperTrace"), + Block("block_0002", "text", 2, (0, 0, 1, 1), [], "Asian, NonHispanic follow"), + ] + out = restore_hyphen_joins(blocks, joins) + assert out[0].text == "Asian, Non-Hispanic; white Non-Hispanic; HbA1c in PaperTrace" + # page 2 has no proven join — a page's joins never leak onto another page + assert out[1].text == "Asian, NonHispanic follow" + + +def test_the_repaired_block_reads_as_the_author_wrote_it(tmp_path): + pdf = _hyphen_pdf(tmp_path) + blocks = restore_hyphen_joins( + [Block("block_0001", "text", 1, (0, 0, 1, 1), [], "followed by Asian, NonHispanic; and")], + hyphen_joins(pdf), + ) + assert "Asian, Non-Hispanic" in blocks[0].text + + +# --------------------------------------------------------------------------- +# the anchor half of the same defect: what `search_for` can actually find +# +# These belong beside a highlight test, but the pairing is the point — the join +# decides what the model can quote, and the page decides what can be boxed. +# `search_for` reads a line break as a space, so neither the compound +# ("Non-Hispanic") nor docling's join ("NonHispanic") is on the page: only +# "Non- Hispanic" is. +# --------------------------------------------------------------------------- + + +def test_a_compound_quote_boxes_the_broken_line_on_the_page(tmp_path): + pdf = _hyphen_pdf(tmp_path) + page = fitz.open(pdf)[0] + assert page.search_for("Asian, Non-Hispanic") == [] # not on the page verbatim + boxes = crop_evidence(pdf, 1, (60, 90, 560, 210), ["Asian, Non-Hispanic"], + tmp_path / "crop.png") + assert boxes >= 1 + + +def test_a_dehyphenated_quote_boxes_the_broken_line_on_the_page(tmp_path): + """The syllabic case, which no phrase-only rule can repair: the page is asked.""" + pdf = _hyphen_pdf(tmp_path) + page = fitz.open(pdf)[0] + assert page.search_for("assessed approximately once") == [] + boxes = crop_evidence(pdf, 1, (60, 90, 560, 210), ["assessed approximately once"], + tmp_path / "crop2.png") + assert boxes >= 1 + + +def test_the_retry_never_invents_a_box(tmp_path): + pdf = _hyphen_pdf(tmp_path) + boxes = crop_evidence(pdf, 1, (60, 90, 560, 210), ["Pacific-Islander patients were"], + tmp_path / "crop3.png") + assert boxes == 0 + + # --------------------------------------------------------------------------- # writers render tables/figures usefully for the LLM # --------------------------------------------------------------------------- diff --git a/tests/test_reference_list.py b/tests/test_reference_list.py new file mode 100644 index 0000000..d2ae453 --- /dev/null +++ b/tests/test_reference_list.py @@ -0,0 +1,131 @@ +"""Where the reference list starts, and where it really ends. + +A real Elsevier pre-proof put refs 1-9 on page 7, then a `Declaration of +interests` section, then refs 10-15 on page 8. `references_section` stops at the +next section header — the guard that stops the reference list swallowing the +rest of the paper — so six references were never parsed and never retrieved. The +audit reported 9 references on a paper that cites 15, with nothing saying so. +""" + +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from papertrace.ingest.pymupdf_ import references_section, references_span # noqa: E402 +from papertrace.models import Block, SourceMap # noqa: E402 + + +def _map(*blocks: tuple[str, int, str]) -> SourceMap: + """(type, page, text) triples → a SourceMap, ids in order.""" + return SourceMap( + doc="p.pdf", pages=max(p for _, p, _ in blocks), converter="docling 2.123.1", + blocks=[ + Block(id=f"block_{i:04d}", type=t, page=p, bbox=(0, 0, 10, 10), + heading_path=[], text=x) + for i, (t, p, x) in enumerate(blocks, start=1) + ], + ) + + +# the real shape, reduced: nine entries, an interrupting section, six more +_SPLIT = _map( + ("sectionheader", 7, "Conclusion"), + ("text", 7, "The narrative surrounding AI in radiology has often centered on replacement."), + ("sectionheader", 7, "References"), + ("list", 7, "- Jing, A. B., Garg, N. & Brown, J. J. AI solutions to the radiology gap. 2025."), + ("list", 7, "- Tejani, A. S., Cook, T. S. & Hussain, M. Integrating AI. Radiology 2024."), + ("list", 7, "- Korfiatis, P. et al. Implementing artificial intelligence algorithms. 2025."), + ("sectionheader", 8, "Declaration of interests"), + ("text", 8, "☒ The authors declare that they have no known competing interests."), + ("sectionheader", 8, "Journal Pre-proofs"), + ("list", 8, "- Dean, G. et al. Real-world monitoring of AI in radiology. 2025."), + ("list", 8, "- Assess-AI algorithm performance monitoring. https://www.acr.org/Data-Science"), + ("list", 8, "- Kitamura, F. et al. Teaching AI for Radiology applications. 2025."), + ("text", 8, "_"), +) + + +def test_a_reference_list_split_by_a_section_is_parsed_whole(): + """The reported defect. Six of nine entries here sit after an interrupting + `Declaration of interests` section, and stopping at the first header lost + them — six sources never even attempted for retrieval.""" + text = references_section(_SPLIT) + + for surname in ("Jing", "Tejani", "Korfiatis", "Dean", "Assess-AI", "Kitamura"): + assert surname in text, f"{surname} missing — the list was cut short" + assert text.count("- ") == 6, text + + +def test_the_interrupting_section_is_not_pulled_into_the_list(): + """`_parse_bulleted` glues a non-bullet line onto the previous entry, so a + stray prose block does not merely add noise — it corrupts a reference.""" + text = references_section(_SPLIT) + + assert "competing interests" not in text + assert "Declaration" not in text + assert "Journal Pre-proofs" not in text + assert "narrative surrounding" not in text + + +def test_resuming_across_a_section_break_is_reported(): + """Crossing a section boundary to continue a list is a guess the reader + should be able to check, so the fact travels with the text.""" + _, resumed = references_span(_SPLIT) + assert resumed is True + + plain = _map( + ("sectionheader", 7, "References"), + ("list", 7, "- Jing, A. B. AI solutions to the radiology gap. 2025."), + ("list", 7, "- Tejani, A. S. Integrating AI. Radiology 2024."), + ) + _, resumed = references_span(plain) + assert resumed is False, "an ordinary list must not claim it was resumed" + + +def test_prose_after_the_references_is_still_never_swallowed(): + """The guard this defect sits behind. Text following the reference list must + stay out, whatever its position — that is what stops "References were + checked by hand" taking the rest of the paper with it.""" + m = _map( + ("sectionheader", 7, "References"), + ("list", 7, "- Jing, A. B. AI solutions to the radiology gap. 2025."), + ("sectionheader", 8, "Appendix A"), + ("text", 8, "References were checked by hand against the originals."), + ("text", 8, "Further discussion of the cohort follows in three paragraphs."), + ) + text = references_section(m) + assert "Jing" in text + assert "checked by hand" not in text + assert "Further discussion" not in text + + +def test_a_single_stray_item_does_not_resume_the_list(): + """One list block after an unrelated heading is far more likely to be a + bulleted sentence than the tail of a bibliography, so it takes a run.""" + m = _map( + ("sectionheader", 7, "References"), + ("list", 7, "- Jing, A. B. AI solutions to the radiology gap. 2025."), + ("list", 7, "- Tejani, A. S. Integrating AI. Radiology 2024."), + ("sectionheader", 8, "Acknowledgements"), + ("list", 8, "- Supported by a grant from the institute, 2019."), + ) + text, resumed = references_span(m) + assert "Supported by a grant" not in text, text + assert resumed is False + + +def test_a_differently_typed_run_does_not_resume_the_list(): + """The reference blocks were `list`; a following run of `text` blocks is a + different structure and must not be appended to the bibliography.""" + m = _map( + ("sectionheader", 7, "References"), + ("list", 7, "- Jing, A. B. AI solutions to the radiology gap. 2025."), + ("list", 7, "- Tejani, A. S. Integrating AI. Radiology 2024."), + ("sectionheader", 8, "Appendix"), + ("text", 8, "The first supplementary consideration, discussed in 2020."), + ("text", 8, "The second supplementary consideration, discussed in 2021."), + ) + text, resumed = references_span(m) + assert "supplementary consideration" not in text, text + assert resumed is False diff --git a/tests/test_refs.py b/tests/test_refs.py index 6bbd3d6..8c1a60b 100644 --- a/tests/test_refs.py +++ b/tests/test_refs.py @@ -630,3 +630,163 @@ def test_the_user_agent_reports_the_real_version(): assert f"PaperTrace/{__version__}" in UA assert f"PaperTrace/{__version__}" in UA_CONTACT.format(email="a@b.org") + + +# --- a web page is not an article, and a title search will not admit that --- +# +# Reference [8] of a real audited paper is an ACR news page with no DOI. With no +# DOI to look up, `resolve_entry` fell through to a Crossref *bibliographic +# title search*, which answered `10.1002/acr2.11538` — ACR Open Rheumatology +# (American College of Rheumatology, not Radiology). Unpaywall served that +# journal's editorial about ChatGPT, the title check passed it at 6/15, and two +# claims were judged `not_addressed` against a rheumatology editorial. + +ACR_WEBPAGE_REF = ( + "ACR launches first medical practice artificial intelligence QA program. " + "https://www.acr.org/News-and-Publications/Media-Center/2024/ACR-Launches-FirstMedical-" + "Practice-Artificial-Intelligence-Quality-AssuranceProgram?utm_source=chatgpt.com." +) + +# Verbatim excerpt of the wrong paper's first page as PyMuPDF extracts it — +# ligatures and all. Inlined so the test stays offline and self-contained. +ACR_EDITORIAL_FIRST_PAGE = ( + "E D I T O R I A L\n" + "ChatGPT, et al … Artificial Intelligence, Authorship, and Medical Publishing\n" + "Daniel H. Solomon,1 Kelli D. Allen,2 Patricia Katz,3 Amr H. Sawalha,4 and Ed Yelin3\n" + "If you have not yet heard of ChatGPT, you will! This artificial intelligence " + "(AI)-based chatbot is making waves in medicine, education, academic publishing, " + "and more widely. GPT, generative pretrained transformer, describes the next " + "generation in AI-powered chatbots that not only construct full sentences on topic " + "but now synthesize information from many fields, from many sources, and with " + "tremendous nuance. The American College of Rheumatology (ACR) journal editors and " + "the ACR Committee on Journal Publications have agreed that co-authorship is not " + "appropriate, since authorship according to the International Committee of Medical " + "Journal Editors requires that authors agree to be accountable. " + "This is an open access article under the terms of the Creative Commons " + "Attribution-NonCommercial-NoDerivs License, provided the original work is properly " + "cited. ChatGPT, et al … Artificial Intelligence, Authorship, and Medical Publishing" +) + + +def test_a_url_only_reference_never_enters_a_bibliographic_title_search(tmp_path): + """Crossref's bibliographic search always returns *something*; for a news + page that something is a confident wrong answer. `no_doi` costs nothing + real — the page was never retrievable as a PDF — and it is the only honest + answer, so the refusal happens before any request goes out.""" + from papertrace.refs import resolve_entry + + e = RefEntry(num="8", raw=ACR_WEBPAGE_REF, slug="acr-2024") + calls: list[str] = [] + client = httpx.Client(transport=httpx.MockTransport( + lambda r: calls.append(str(r.url)) or httpx.Response( + 200, json={"message": {"items": [{"DOI": "10.1002/acr2.11538"}]}} + ) + )) + out = resolve_entry(entry=e, dest_dir=tmp_path, email="a@b.org", + client=client, provided_dir=None) + + assert calls == [], f"a web page reached the network: {calls}" + assert out.status == "no_doi" + assert out.doi is None, "a title search must not attach a DOI to a web page" + assert out.pdf_path is None + assert "web page" in out.reason.lower(), out.reason + + +def test_a_journal_reference_that_merely_includes_a_url_still_resolves(tmp_path): + """The gate keys on the *absence* of article structure, not the presence of + a URL — publishers' own reference styles print a link beside the volume and + page range, and those references are exactly what Crossref answers well.""" + from papertrace.refs import resolve_entry + + raw = ( + "Smith A, Jones B (2021) Deep learning for chest radiographs: a systematic " + "review. Radiology 298:120-130. Available at: " + "https://pubs.rsna.org/journal/radiology" + ) + page = ( + "Deep learning for chest radiographs: a systematic review. " + "A. Smith, B. Jones. Radiology 2021; 298:120-130." + ) + calls: list[str] = [] + + def handler(request: httpx.Request) -> httpx.Response: + calls.append(str(request.url)) + if "api.crossref.org" in str(request.url): + return httpx.Response(200, json={"message": {"items": [{"DOI": "10.1148/r.2021"}]}}) + if "api.unpaywall.org" in str(request.url): + return httpx.Response( + 200, json={"best_oa_location": {"url_for_pdf": "https://x/oa.pdf"}} + ) + return httpx.Response(200, content=_real_pdf_bytes(page)) + + e = RefEntry(num="9", raw=raw, slug="smith-2021") + out = resolve_entry(entry=e, dest_dir=tmp_path, email="a@b.org", + client=httpx.Client(transport=httpx.MockTransport(handler)), + provided_dir=None) + + assert any("api.crossref.org" in c for c in calls), f"never asked Crossref: {calls}" + assert out.status == "retrieved" and out.doi == "10.1148/r.2021" + + +def test_a_year_span_in_a_headline_is_not_a_page_range(): + """The gate reads a page range as proof of an article, and `2020-2025` in a + policy document's title has that shape. Two four-digit years either side of + a dash are a date span, not pages — the pattern that clears the gate has to + be able to tell the difference, or every government report with a date + range in its title goes to a title search.""" + from papertrace.refs import _is_webpage_reference + + assert _is_webpage_reference( + "World Health Organization. Global strategy on digital health 2020-2025. " + "https://www.who.int/publications/i/item/9789240020924" + ) + # and a real page range still clears it + assert not _is_webpage_reference( + "Smith A. Deep learning triage. Clin Radiol. 2022; pages 1068-1083. " + "https://www.clinicalradiologyonline.net/toc" + ) + + +def test_a_doi_that_lives_only_inside_the_link_is_still_a_doi(): + """URLs are stripped before the article signals are looked for, which hides + a DOI written only as `https://doi.org/10.…`. `resolve_entry` happens not to + ask in that case — it consults the gate only when no DOI was found — but a + predicate that is wrong on its own is a trap for the next caller.""" + from papertrace.refs import _is_webpage_reference + + assert not _is_webpage_reference( + "Zenodo dataset for the segmentation challenge. " + "Available at: https://doi.org/10.5281/zenodo.1234567" + ) + + +def test_a_tracking_parameter_is_not_part_of_a_title(): + """`?utm_source=chatgpt.com` put `chatgpt` and `source` into the reference's + token set, and the wrong paper is an editorial about ChatGPT — so the URL + supplied two of the six matches that passed it. URL fragments inflate both + the numerator and the denominator; neither belongs to a title.""" + from papertrace.refs import _title_tokens + + tokens = _title_tokens(ACR_WEBPAGE_REF) + assert "chatgpt" not in tokens, "a tracking parameter matched the wrong paper's subject" + assert "source" not in tokens + assert not any(t in tokens for t in ("firstmedical", "assuranceprogram", "publications")) + assert {"launches", "practice", "artificial"} <= tokens, "the title's own words survive" + + +def test_three_generic_domain_words_are_not_an_identity_check(): + """The last line of defence, in case a URL-only reference reaches it by some + other route: with the URL stripped the ACR news page still scores 3/7 = + 0.43 against the rheumatology editorial, on `artificial`, `intelligence` + and `medical` alone. A ratio is trivially cleared by a short reference full + of generic domain vocabulary, and in this field that vocabulary is most + papers' subject.""" + from papertrace.refs import _title_check_text + + state, detail = _title_check_text(ACR_WEBPAGE_REF, ACR_EDITORIAL_FIRST_PAGE) + assert state != "verified", detail + # and not laundered the other way either: nobody established this is a + # different paper, only that the check cannot tell + assert state == "unverifiable", detail + # a page with none of the reference's words is still called wrong outright + assert _title_check_text(ACR_WEBPAGE_REF, RIGHT_PAGE)[0] == "mismatch" diff --git a/tests/test_skill_commands.py b/tests/test_skill_commands.py new file mode 100644 index 0000000..f1da965 --- /dev/null +++ b/tests/test_skill_commands.py @@ -0,0 +1,93 @@ +"""Every `papertrace` command a skill documents must actually parse. + +A skill is documentation an agent executes verbatim, so a stale flag in one is a +broken command rather than a typo. Four were: `refs ... -o case/` (`refs` has no +`-o`), `report case/` and `highlight case/ --claim ` (neither takes a +positional argument), and a prose `refs --parse-only` with no manuscript. +""" + +import re +import shlex +import sys +from pathlib import Path + +import pytest +import typer + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from papertrace import cli # noqa: E402 + +_SKILLS = Path(__file__).resolve().parent.parent / ".claude" / "skills" +_CMD = re.compile(r"papertrace [^`\n]+") + + +def _skill_command_lines() -> list[tuple[str, str]]: + """(skill file, command) for every `papertrace ...` line in every skill. + + Every skill, not just `review/`: scanning one file is why a fourth broken + command — `papertrace highlight case/ --claim ` in `fact-check/` — sat + there while the other three were being fixed. + """ + if not _SKILLS.is_dir(): # an installed wheel carries no skills + pytest.skip("repo-only directory") + out: list[tuple[str, str]] = [] + for md in sorted(_SKILLS.rglob("*.md")): + rel = str(md.relative_to(_SKILLS)) + out += [(rel, m.group(0).strip().rstrip(").,")) for m in _CMD.finditer(md.read_text())] + return out + + +def test_every_papertrace_command_in_every_skill_parses(tmp_path): + """The skill is documentation an agent executes verbatim, so a stale flag in + it is a broken command, not a typo. Two were: `refs ... -o case/` (refs has + no -o) and `report case/` (report has no positional argument). + + Parsed with `make_context`, which validates arguments without invoking + anything. Appending `--help` instead would not do: `--help` is eager, fires + before click reports an unexpected extra argument, and would have called + `report case/` fine — masking one of the two bugs this test exists for. + """ + paper = tmp_path / "paper.pdf" + paper.write_bytes(b"%PDF-1.4\n") + sources = tmp_path / "sources" + sources.mkdir() + + def literal(token: str) -> str: + if not (token.startswith("<") and token.endswith(">")): + return token + name = token[1:-1].lower() + if "pdf" in name or "paper" in name or "manuscript" in name: + return str(paper) + if "dir" in name or "folder" in name or "source" in name or "case" in name: + return str(sources) + return "1" # , and friends: any scalar will do + + group = typer.main.get_command(cli.app) + # `group.context_class`, not `click.Context`: typer 0.27 vendors click and + # declares no dependency on it, so a top-level `import click` is a + # collection error wherever only typer is installed — and a collection + # error interrupts the whole pytest session, losing every test in it. + root = group.context_class(group, info_name="papertrace") + lines = _skill_command_lines() + assert lines, "no papertrace commands found — did the extraction regex rot?" + + # every broken line at once: stopping at the first would have hidden the + # second bug behind the first for as long as one stayed unfixed + broken = [] + for where, line in lines: + argv = [literal(t) for t in shlex.split(line)[1:]] + sub = group.get_command(root, argv[0]) if argv else None + if sub is None: + broken.append(f"{where}: {line!r}: no such command") + continue + try: + # typer 0.26 raises its own vendored click exceptions, so catch broadly: + # any failure to parse is the documented line being unrunnable + with sub.make_context(argv[0], argv[1:], parent=root): + pass + except SystemExit as e: + broken.append(f"{where}: {line!r}: exits during parsing ({e.code})") + except Exception as e: # noqa: BLE001 - the failure is the finding + broken.append(f"{where}: {line!r}: {type(e).__name__}: {e}") + assert not broken, "documented commands that do not parse:\n " + "\n ".join(broken)