diff --git a/.claude/skills/fact-check/SKILL.md b/.claude/skills/fact-check/SKILL.md index 4a74839..b0cebcb 100644 --- a/.claude/skills/fact-check/SKILL.md +++ b/.claude/skills/fact-check/SKILL.md @@ -18,8 +18,25 @@ with `refs_manifest.json` and per-source ingests under `case/ingest//`. forbidden move. Your training knowledge of a paper is NOT evidence; only the PDF on disk is. + **A reference may carry `supplements`.** Each is a separate document with + its own slug, and each is judged **separately** — one verdict, one anchor, + one crop apiece, exactly as two co-cited references would be. Do not merge + an appendix's evidence into the article's verdict. The claim's headline is + the most adverse across all of them. + + The manuscript's own supplementary material is in `manuscript_supplements`, + and answers for no citation label. A claim pointing at *this paper's* + "Table S3" is judged against those; with none supplied the verdict is + `not_retrieved`, never a guess from the main text. + + **Check `verified` on each supplement.** True means its own title or DOI + named the work it accompanies. False means it was attached because its + filename carried the reference's tokens and nothing read it — say so when + a verdict rests on one of those. + 2. **Ingest on demand.** If `case/ingest//` doesn't exist yet: `papertrace ingest -o case/ingest/` + Supplements use their own slug, so they get their own folder. 3. **Read for the claim.** Search the source's `clean.md` for the claim's subject (numbers, named methods, populations). Read the matching blocks in diff --git a/.claude/skills/review/SKILL.md b/.claude/skills/review/SKILL.md index 181e263..b97150e 100644 --- a/.claude/skills/review/SKILL.md +++ b/.claude/skills/review/SKILL.md @@ -31,17 +31,26 @@ Print this banner (verbatim, in a fenced code block), then the one-liner: ## 1 · Intake (one batched exchange, not an interrogation) -Ask for all four at once, as a short checklist the user can answer in one message: +Ask for all five at once, as a short checklist the user can answer in one message: 1. **The paper** — path to the PDF of interest (a paper you build on, one you're evaluating, your own). 2. **Sources you already have** — a folder of reference PDFs, if any. Optional: open-access copies of the rest are fetched automatically. -3. **Your questions** — what you want answered about this paper: free text, + **Tell them not to bother renaming**: each PDF is identified by its own + DOI or title, so publisher downloads work as they came. Supplementary + material for a cited paper goes in the same folder too; several per + reference is fine, and each is judged as its own document. One whose + article is missing is set aside, so ask for the article too if they + have it. +3. **Supplementary material for the paper itself** — the paper's own SI, + appendices or eTables, if any. Optional, but a claim pointing at + "Table S3" cannot be checked without it. +4. **Your questions** — what you want answered about this paper: free text, a list, or screenshots of form fields (a journal's reviewer form works too). Optional — without it, the standard audit runs: citation accuracy, coverage, uncited assertions, newer & overlooked literature. -4. **Prior critique** — what's already been said about this paper, to be +5. **Prior critique** — what's already been said about this paper, to be weighed as source material: published comments or letters, PubPeer threads, earlier reviewer reports and author responses, your own notes. Optional. @@ -99,15 +108,26 @@ literature, methods and results consistency — and say you did. ## 4 · Retrieval — live ticker ```bash -papertrace refs --provided -c +papertrace refs --provided -c --supplement ``` +Drop `--supplement` when the paper has none; repeat it when it has several. + Stream the per-reference ticker as it runs (✓ retrieved via unpaywall · ✓ provided by you · ⚠ paywalled · ⚠ no DOI). Close with the honest summary line, e.g. **“19/42 sources available — 23 not obtainable (paywall / no DOI / unpublished)”**, and remind the user they can drop more PDFs into the sources folder at any point; you'll pick them up on request. +Three lines in that ticker deserve saying out loud rather than scrolling past. +`identified … by its own DOI/title` means a file the filename rule could not +place was recognised from its contents. `+ N supplements` means extra documents +will be judged, and extra model calls spent. `⚠ … set aside` means a file the +user supplied did **nothing** — its article was never obtained, its title +matched two references equally, or nothing in it could be read — and each has a +different fix. Never let that last one pass silently: the user believes that +file was used. + For a published paper, also run the literature scout (`papertrace scout -c `, `--doi` if the title lookup misses) and show its two registers: published since, and existed-but-uncited. Candidates for @@ -121,8 +141,9 @@ Never bypass a paywall. Never pretend a source was read that wasn't. from `source_map.json` and say it in your first message after ingest. Not only when it is bad news: a reader who is told nothing cannot judge the fidelity of what follows in either direction, and the CLI now states it at the start and the -end of every run. Say also that cited sources are always read as flat text, so a -"docling" line is not mistaken for a claim about them. +end of every run. Cited sources and supplements are read with the **same** +backend as the paper (since 0.5.0), each recorded under its own slug — so name +any that came back `pymupdf` while the paper did not. If the backend is `docling`, table blocks are real GFM tables and figure blocks carry captions with page bboxes. While working through Results/Discussion, diff --git a/CHANGELOG.md b/CHANGELOG.md index 3155842..e919725 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,870 @@ All notable changes to PaperTrace are documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow [SemVer](https://semver.org/). +## [0.6.0] — unreleased + +### Added — a provided PDF is identified by what is in it + +`--provided` matched on the **filename** and nothing said so. It needs the +reference's surname and year in the name, which a reference-manager export has +and a publisher download never does: + +``` +pyrros-2023.pdf matches +Pyrros et al. - 2023 - Opportunistic....pdf matches +s41467-023-39631-x.pdf no +1-s2.0-S0140673623001234-main.pdf no +41467_2023_39631_MOESM1_ESM.pdf no (the standard Nature supplement name) +``` + +So a user who dragged in a folder of downloads got an audit that looked +entirely normal and used none of it — and the failure was **asymmetric**: an +unmatched supplement was reported, an unmatched *article* was skipped in +silence. + +Each unrecognised PDF is now identified from **its own DOI**, else from **its +own title** compared against the reference list. Filename matching still runs +first and still wins: that is the user's own assertion about the file, and +content only fills the gap it leaves. Supplements are identified the same way, +which matters more than it sounds — the publisher forms carry no filename +marker at all (`\besm\b` cannot match inside `MOESM1_ESM`, and `mmc1` and +`media-1` say nothing) while their first page states plainly what they are. + +**Nothing in the folder goes unremarked.** `unused_provided` lists every PDF +that ended up attached to nothing, with the reason kept apart: unrecognisable, +ambiguous, a spare copy of a paper already matched, or a supplement whose +article is missing. + +⚠️ **Two refusals, both deliberate.** A title matching **two** references is +used for neither — a corrigendum shares nearly every distinctive word with its +original, and picking the better score would judge a claim against the wrong +paper with nothing downstream able to notice. And a title with too few +distinctive words to tell papers apart is not a match: a filename match may be +accepted as `unverifiable` because the user named the file, but nobody asserted +anything about a file identified by content. + +Not reused for this: `_title_check_text`, the rule that already vets a +filename-matched file. Measured on the demo's real sources it verifies +`pyrros-2023.pdf` against an unrelated NEJM review as well, because it counts a +reference's words anywhere on a whole page and both are about AI in medical +imaging. It is a forgiving veto for a file already chosen, and it stays that. + +`Supplement.verified` and `SourceJudgement.verified` record which supplements +were established to belong to their work. The 0.6.0 disclosure said +*"supplements carry no identity check"*; that was true of all of them then and +is true of only some now, so the report states the split per file instead of +warning about both equally. + +Also fixed: a surname under four characters is dropped by the filename token +filter, so `liu-2019` matched on the **year alone** and `smith-2019-appendix.pdf` +would attach to Liu 2019 — with no title check to catch it, since supplements +had none. Such a match now requires the slug itself in the filename. + +The guided wizard asks *whether* you have cited PDFs before asking *where*, +defaulting to yes when `/sources` already holds some, and asks the same +about the paper's own supplementary material. A user with neither now answers +two questions instead of reading two explainers and two path prompts. + +### Added — supplementary material, read as its own document + +A subgroup table in Supplementary Table S2, a sensitivity analysis in Appendix +B, a protocol in an ESM: real papers put the decisive evidence outside the +article, and a user holding that file had no way to hand it over. Worse, +`refs.py` recognised supplement filenames **only in order to discard them**, +because judging a claim against an appendix while calling it the cited source +is the laundering this tool exists to prevent. + +The organising idea is that **a judgement target is a document, not a +reference**. Multi-source checking already judged one claim against N documents +— one model call each, per-document verdicts and crops, most-adverse headline, +`not_addressed` unranked so a silent document taints nothing. A supplement +enters as one more document, which is why this needs no new verdict, no new +headline rule and no change to `coverage/3`. + +- **A cited work's supplements need no flag.** Drop + `pyrros-2023-supplement.pdf` beside `pyrros-2023.pdf` in the sources folder. + Several per reference is fine. Each is judged separately, and a claim citing + `[14]` is read against every document `[14]` has. +- **`--supplement` (repeatable) for the audited paper**, which has no reference + slug for a filename to key on. A claim pointing at its own `Table S3` is read + against those; with none supplied it is `not retrieved` and names the flag, + rather than sitting in the uncited register — the paper said where its + evidence was and nobody opened it. +- **A supplement never stands in for the article.** It attaches only to a + reference that was actually obtained; an orphan is named in the ticker with + the reason, because a file the user supplied that then did nothing is the + quietest possible failure. +- **The wizard now asks for the sources folder**, which it never did: + `run_wizard` hardcoded `provided=None`, so the guided path could not reach a + flag the CLI has had all along. + +⚠️ **Two disclosures you should expect to see.** Supplements carry **no +identity check** — a supplement's own title is not its parent's, so the check +that guards every cited source cannot apply, and it is not faked. And a +citation appearing *only* inside a supplement is **not counted** by the +coverage audit, which reads the manuscript alone. Both are stated in all three +reports whenever supplements were read, and a claim whose headline came from a +supplement rather than the article body says so on the claim. + +Slugs come from the file stem, never an ordinal: `-suppl1`/`-suppl2` numbered +in folder order is the shifting-id defect this codebase already rejects for +citation occurrences, where deleting one file re-points another document's +stored verdicts and crops. + +Wire format: `RefEntry.supplements`, `RefManifest.manuscript_supplements`, +`SourceJudgement.kind`, `ClaimResult.own_supplement`, all schema-declared and +absent-safe, so 0.5.x files still load. `RefManifest.from_json` also stops +raising `TypeError` on a key it does not know — a manifest from a newer +papertrace used to kill an older one outright. + +Both prompts changed, so `evals/provenance.prompt_fingerprint()` moves and +`agreement.py` will refuse to compare a 0.6.0 run against an earlier one. That +is the guard working, not a regression. + +Judgement quality here is **unmeasured**, like everything since ADR 0001. + +## [0.5.0] — unreleased + +0.4.1 was never released, so its entries below ship together with these. + +### Added — `papertrace --version` + +The first thing anyone types after installing, and it answered *"No such +option: --version"*. Found by installing this branch from GitHub into a clean +virtualenv and typing it. The only way to check was +`python -c "import papertrace; print(papertrace.__version__)"`, which nobody +guesses — so a user who had just installed from a branch had no way to confirm +which one they were running. + +`--version` / `-V`, eager so it answers before the callback body runs: a bare +`papertrace` on a terminal opens the guided wizard, and a version flag resolved +after that would have interviewed the user about their manuscript before +telling them the number. It reads `papertrace.__version__`, the one home +`docs/RELEASING.md` names, rather than restating it where it could drift. + +### Changed — extraction is told where the citations are ⚠️ **`coverage/3`** + +The old flow discarded the location and then worked to reconstruct it. The +model returned a paraphrase plus a free-text `location` ("Methods ¶2"), and +Python guessed which of several `[3]` markers that paraphrase had come from: +normalise both sides, score with `SequenceMatcher`, accept only on +`ratio ≥ 0.45` **and** `margin ≥ 0.10`, assign globally best-first, and report +everything it could not decide as `uncertain`. The counts were right and the +*pointer* could be wrong. + +The inventory it was matching against had been there all along — built +deterministically from `source_map.json`, just *after* the model call instead +of before it. + +- **The inventory goes into the prompt.** `_render_inventory()` renders each + citation occurrence as `ctx_NNNN` with its page, section, labels and + sentence; `EXTRACT_PROMPT` asks the model to work through that list and + return, per claim, the ids it was taken from. One sentence citing [2] and [3] + is **one** claim carrying **both** ids. +- **Attribution becomes a set lookup.** An occurrence is covered when some + claim's `ctx_ids` names it. `results.json` gains `ctx_ids` per claim and the + audit is `"schema": "coverage/3"`; `coverage/2` files still validate, and + `labels_in_text`/`covered`/`missing` keep their label-level meaning byte for + byte, because `evals/align.py` reads `missing` to apportion blame. +- **Six symbols deleted** — `_attribute_label`, `_normalize_for_match`, + `_ratio`, `_location_matches`, `OCCURRENCE_MIN_RATIO`, + `OCCURRENCE_MIN_MARGIN` — and the `unicodedata`/`SequenceMatcher` imports + with them. **This is not a net line saving and should not be sold as one:** + `check.py` loses 110 lines and gains 115, roughly a third of the new ones + being prompt text and comments. What goes is a *mechanism* — a scoring + function, two tuned thresholds and a global assignment pass — replaced by a + dictionary lookup. The audit no longer + publishes `min_ratio`/`min_margin` because there is nothing to tune. The + deliberate ~10-line duplication with `evals/align.py` is gone too — the + reason it existed (papertrace cannot import `evals`, `evals` must not import + a matcher from the thing it grades) no longer applies, since there is no + matcher on this side. +- **A `ctx` the inventory does not contain is dropped, never repaired.** + A hallucinated `ctx_9999` and an honest `"ctx": []` carry the same amount of + information about which sentence was meant, and both are treated as such. + Falling back to "the first occurrence of that label" would manufacture + exactly the confident wrong pointer this removes. +- **`uncertain` survives, with one cause instead of several.** A claim cites a + label and names none of that label's contexts ⇒ a claim reached one of those + places and nothing can say which, so they are `uncertain` and counted as + **not** covered. Previously it also absorbed close calls the matcher refused; + that category no longer exists. +- The report's attribution self-caveat is correspondingly shorter, and its + token changes: attribution is no longer "a text match that can be wrong" but + "the context the extractor named" — still a model step, so still capable of + naming the wrong place, and the report keeps saying so. + +**Also unmeasured**, per ADR 0001. The argument for it is structural — it +deletes a guess and a whole class of silent wrong pointer — not a score. + +### Changed — the committed demo report is regenerated, and its judge is pinned + +`examples/demo/output/` is the only committed output and the artefact the README +links as *"See a completed report"*. It was produced on 2026-08-30 by 0.4.1, so +it showed none of what this release changed. + +- **Regenerated under 0.5.0**, and the demo command now pins + `--model claude-opus-5`. Without it `claude -p` takes the account default, + which had silently moved from opus to haiku between two regenerations — so + the committed showcase's judge depended on the day it was rebuilt. +- **The pinned expectation moves to `1 supported · 2 contradicted · 1 not + retrieved · 1 uncited assertion`, over 4 claims rather than 5.** All four + planted defects are still found; what changed is that the sentence citing + both [2] and [3] now arrives as **one multi-source claim** instead of two + single-source ones, because extraction is asked for the verbatim sentence. + **Reproduced on `claude-opus-5` and `claude-haiku-4-5` alike**, so it is the + prompt and not the model — which is worth stating, because the first + regeneration changed both at once and the cause was ambiguous until the + second run isolated it. +- Two README claims corrected as a consequence: the counts, and the line + asserting that no claim in the demo cites more than one reference. That is + now false, and the demo consequently exercises the per-source breakdown and + the new `most adverse of 2 cited sources` qualifier — which the old one + never did. + +### Changed — cited sources are read with the layout backend ⚠️ **breaking** + +`check.py` hard-coded `backend="pymupdf"` for every cited source, and said why: +*"Layout fidelity (tables/figures) is spent on the audited paper, not its +sources."* That had the asymmetry backwards. The manuscript's claim is the +question; the **source** is the evidence — and the evidence for a subgroup +claim is usually a table row. Read flat, the row is gone. + +- **Sources now get the same backend as the paper.** `check` gains + `--backend`, `run` forwards its own, and `check_claims` takes it as a + **required** keyword — no default, like `_clip`'s truncation accumulator in + the same module and for the same reason. Neither possible default is honest: + `auto` drags docling into an offline test run, `pymupdf` silently downgrades + a caller who asked for layout. +- **`docling` moves from an extra to a base dependency.** It cannot be optional + once the sources depend on it. The `[docling]` and `[full]` extras are kept as + aliases so 0.4.x install commands still resolve. **Measured in a clean + virtualenv: 1.4 GB installed** (torch 591 MB, then opencv, transformers, + scipy), plus the ~500 MB layout-model download on first *use*. That number is + in the README install table rather than left as "pulls torch", because it is + the kind of cost a user should meet before typing the command and not after. + `--backend pymupdf` remains the escape hatch. +- **CI installs it and never runs it.** The models download on use, not on + install, and every test pins `backend="pymupdf"` — which the required + argument now makes impossible to forget. The suite stays offline and no + slower: measured back to back on one machine, 632 tests in 22.6 s before this + change and 645 tests in 17.1 s after. `import docling` is itself only ~0.2 s, + because it does not pull torch until something converts a PDF. + +**Two defects this would otherwise have introduced, both found by looking:** + +- **`_stale_ingest` compared only the PDF hash**, so re-running an existing + case folder would have reused its 0.4.x **pymupdf** source maps while the run + reported layout-aware source ingest — a silent wrong-fidelity judgement, + which is the exact failure class this project exists to refuse. It now + compares the recorded `converter` too, resolving `auto` and ignoring + docling's version suffix through a shared `ingest.resolve_backend()`. +- **The reports never said how the sources were read.** `RunResults.converter` + is the *manuscript's*, and the only mention of the sources was one dim line + in the terminal — the markdown and both HTML looks said nothing. Each + source's converter now travels in `RunResults.source_converters`, and any + source read as flat text is **named by slug** in all three formats. An empty + dict means the run never recorded it (every 0.4.x file) and is deliberately + not read as "all of them were flat". + +**Measured cost**, since this is a real slowdown and not an unpriced one: on +this machine the first docling ingest in a process costs ~41 s (loading the +layout models) and each subsequent source ~3 s. Under `papertrace run` the +models are already loaded from the manuscript, so a 20-source paper pays +roughly a minute more in total; `papertrace check` on its own pays the load +once. `--backend pymupdf` remains a deliberate choice for a constrained +machine, and now says so per source in the report instead of being the +unstated default. + +A source-ingest failure — docling can run out of memory or fail to fetch its +models, which flat text never could — unchecks that one source with the reason +in its note, and is never laundered into `not_retrieved`. + +### Changed — the judge reads the paper's own sentence, not a summary of it + +`EXTRACT_PROMPT` asked for each claim "tightly paraphrased, ≤160 chars", and +`CHECK_PROMPT` was handed `{id, claim, location}`. So the population, the +effect size, the confidence interval and the hedging — the things that actually +decide whether a citation supports a statement — had to survive a compression +the judge could not undo. *"Mortality fell by 12% in the subgroup over 65 (HR +0.88, 95% CI 0.79-0.98)"* and *"mortality fell by 12%"* are different claims, +and only one of them is checkable. + +- **Extraction returns a verbatim `quote`** — the manuscript's own sentence, + ≤500 chars — alongside the paraphrase, whose cap rises to 300. The paraphrase + stays because it is what a report headline reads well; the quote is what gets + judged, and `CHECK_PROMPT` says so explicitly. +- **The quote appears in the report** above each verdict, in all three formats, + so what was judged is visible rather than taken on trust. +- **A claim judged without one says so.** An empty quote means the model did + not return a sentence, so the verdict rests on the paraphrase — weaker + evidence, and now a warn-level disclosure in every format rather than + something a reader has to infer from a missing blockquote. It fires only + where a judgement actually happened: nothing read an unretrieved source, so + the notice would otherwise land on every row of the gap register. +- **The quote is never back-filled from the paraphrase.** That would reinstate + the exact compression this change removes while looking like it had been + fixed. +- Coverage attribution briefly took its similarity ratio on the quote rather + than the paraphrase, which was a free improvement to the matcher — and then + the matcher was deleted outright by the change below. Nothing of it remains; + the note is kept only so the two entries do not appear to contradict each + other. +- `results.json` gains `quote` on both cited claims and the uncited register; + `schemas/results.schema.json` is updated and `from_json` still loads a 0.4.x + file, where the field is simply absent. + +**One consequence for `evals/`:** `prompt_fingerprint()` is a content hash of +the prompts, so this invalidates comparison against any pre-0.5.0 run. +`agreement.py` already refuses to compare runs that do not share the +`(set_id, prompt fingerprint, converter)` triple — that is the correct +behaviour, not a regression. And per ADR 0001 there is no benchmark to +compare against anyway: **this change is unmeasured.** It removes a known +information loss; that is not the same as evidence that verdicts improved. + +### Changed — `report.md` by default; the HTML looks on request ⚠️ **breaking** + +Every run wrote three report files and a ~1 MB font bundle, whether or not +anyone wanted three. `report.md` is what almost every run is read through; the +editor and terminal looks exist for sharing and for screenshots. + +- **`papertrace run` and `papertrace report` now write `report.md` alone.** + Add `--format editor`, `--format terminal`, or both — `-f` for short, and + repeatable. The fonts are copied only when an HTML look is actually written. +- **`report.md` is always written**, whatever `--format` says. It is the record + of the audit, not one presentation of it among three; a request for only a + screenshot look must not leave the case folder without the report itself. +- **`--png` pulls in the HTML it screenshots.** `--png --format md` cannot mean + "photograph a file I told you not to write", so the HTML looks are rendered + regardless. Honouring it literally would have produced no PNG and said + nothing about why. +- **A mistyped format is refused** — `unknown --format pdf — expected any of + md, editor, terminal`, exit 2, checked before `results.json` is even loaded + so a bad flag cannot half-write a report folder. Silently ignoring it would + answer `--format pdf` with a folder containing no PDF and no complaint, which + is the same shape as the unknown-backend bug `ingest_pdf` already refuses. +- `write_reports()` itself still defaults to every format. It is the seam the + disclosure-parity suite drives, and that suite has to render all three or it + stops comparing anything; the narrower default belongs to the CLI, where the + user's intent actually is. + +**To restore the old behaviour:** `papertrace run paper.pdf -f editor -f +terminal`. + +### Fixed — a wizard-driven audit would have crashed at the report stage + +Found while adding `--format`, and the third appearance of a bug class this +codebase has now met three times. `run_wizard()` calls `cli.run` as a plain +Python function, and Typer's declared defaults are `OptionInfo` sentinels +rather than the values `--help` displays — so the new parameter the wizard did +not name would have arrived as a sentinel, reached `write_reports`, and raised +on not being iterable. After every paid model call had already been made. + +- `report` is now split into the Typer command and `_report_pipeline()`, which + is keyword-only with ordinary Python defaults — the same treatment `ingest` + and `refs` already had, and for the same reason. `run()` calls the pipeline + function. +- The wizard now names **every** parameter `run` declares, and a new test + asserts that against `inspect.signature(cli.run)` rather than against a list + of names — so the next parameter added to `run` is caught without anyone + remembering to come back and update the test. + +### Changed — the headline no longer reads as a verdict on the whole claim + +`❌ CONTRADICTED` is one source's verdict. On a claim citing four references it +reads as a statement about the claim, and a compound sentence may legitimately +draw different parts from different references — so one dissenting source of +four overstates by exactly the amount the status line cannot show. + +- **A multi-source headline now names what it ranged over**: *"❌ CONTRADICTED + — most adverse of 4 cited sources"*, in all three report formats. The rule + itself is unchanged and deliberately so: the most adverse verdict is the + right triage signal, and one dissenter must never be averaged away. What + changes is that it stops being stated unqualified. +- **Single-source claims are not qualified**, and neither is a claim with no + judgements. With one source the headline *is* the claim's verdict, and + "most adverse of 1" would be noise that teaches readers to skip the line; a + `not_retrieved` claim ranked nothing at all, so naming a comparison that + never happened would be its own small invention. +- No new verdict value, no schema change. A `disputed`/`mixed` state was + considered and declined: it would have meant a `VERDICTS` entry, a schema + update, a gold-verdict enum change and six render sites, to express something + the existing per-source breakdown already shows. + +**The limitation this leaves, stated rather than glossed:** the run's summary +counts and `results.json` still tally each claim once, under its headline. A +claim splitting 2 support / 1 partial / 1 contradict appears in the +`contradicted` total and nowhere else. That total means *"claims with at least +one contradicting source"*, not *"claims that are wrong"*, and the README's +does-not list now says so. Fixing the totals properly needs the per-source +population counted separately, which is a larger change than this one. + +### Decided against — two proposals declined in writing, with reasons on file + +A full-stack review raised seven items. Five became changes; two are declined, +and `docs/adr/` now exists to record why so that a future review does not +re-derive them. Choosing not to build something is user-visible scope, which +is why it is here and not only in a commit message. + +- **No gold benchmark, and therefore still no accuracy figure** + ([ADR 0001](docs/adr/0001-no-gold-benchmark.md)). The evaluation harness is + not the thing that was missing: `evals/` already holds ten modules, 22 metric + functions, a JSON-schema'd gold contract and twelve CI-green test modules, + and `evals/PROPOSAL.md` already specifies the ≥40-case paired set down to its + acceptance criteria. What is missing is data, and one precondition for it — + `evals/DESIGN.md` requires ≥ 2 labellers who did not write the prompts. + There is one maintainer, who wrote them. Building the set self-labelled would + produce a number the harness itself prints a conflict-of-interest caveat + against, and a number nobody may cite is worse than no number, because the + number gets cited. `evals/PROPOSAL.md` is kept, with its status updated: it + is the plan if that precondition ever changes. + + The consequence is stated rather than glossed: the other changes in this + release **ship unmeasured**. They remove mechanisms that could only degrade + judgment quality; that is not the same as evidence it improved, and the two + are not blurred anywhere in this file or the README. + +- **No GROBID** ([ADR 0002](docs/adr/0002-no-grobid.md)). The reference + parsing and reconciliation really is ~707 contiguous lines of `refs.py`, but + only ~261 of those are *parsing* a specialist parser would displace. The + other ~470 — the Crossref deposit, corroboration and `reconcile` — exist + because any reading of a reference list can be wrong and the tool must be + able to say so, and they survive a parser swap: a parser cannot certify + itself. Against that, GROBID wants Java, Docker and 2–4 GB of memory, and + its own citation-context linking is 0.76–0.91 F1 — a probabilistic gain for + a disqualifying deployment cost in a `pip install` tool. Not benchmarking it + is part of the decision: a benchmark is only worth running if a favourable + result would change the outcome. The roadmap item is removed rather than left + implying a plan that does not exist. + + Superscript-citation support, which shares this surface and would fix three + of seven papers with unconfirmed numbering, is unaffected and remains the + higher-value work here. + +## [0.4.1] — unreleased + +### Added — the reference list is now checked against what the paper cites + +`parse_references` was the only stage in the pipeline with no way to report its +own failure. Every other stage has one — `not_retrieved`, `unchecked`, the +anchor tri-state, `unverifiable`, coverage `uncertain` — but the reference +parser always returned a confident list, and nothing ever compared it to +anything. The citation label is the **join key** between a claim and the source +it is judged against, so a list off by one does not produce a worse audit; it +produces a confident audit of the wrong papers. One live run misnumbered 27 of +41 references and said so nowhere. + +- **Three-way reconciliation.** Two independent readings of the reference list + are taken — the tool's parse of the printed text, and the list the publisher + deposited with Crossref — and the manuscript's own `[N]` markers arbitrate + between them. A reading is used only if it accounts for exactly the labels + the body cites, which under citation-order numbering is a structural test + rather than a heuristic: reference *N* is by definition the *N*th first-cited + work. `refs` gains `--doi`, defaulting to the DOI printed on page 1. +- **Crossref is a candidate, not an oracle.** A short deposit is more dangerous + than a bad parse because it looks authoritative: mapped onto `[1]`, `[2]` it + would silently discard the rest. One record in the test spread carries 2 + references for a paper citing about 40, and the payload cannot reveal it — + Crossref's `references-count` counts what was *deposited*, so it always equals + the array length. The body's labels are the only thing that catches it. +- **A reference deposited as a bare DOI is kept, and a shortfall is named as + this tool's.** Some publishers deposit references as a DOI and nothing else; + those rendered to an empty string and were dropped, and the run then reported + that the publisher had deposited a fraction of its own list — a false + accusation, and a plausible-looking number in place of an admission. They are + now kept and named after the DOI, which is the best case for retrieval: the + DOI is already resolved, so the title search is skipped entirely. Where this + tool still cannot render part of a deposit, the deposit is set aside rather + than used to renumber, and the disclosure says whose limitation it is. + Reference numbering is read from **array order**, never from the `key` field — keys are + publisher-specific (`_b0005`, `_bib1`, `3400_CR1`, `bibr1-…`, + `R10-45-20210317`), and two schemes turned up inside a single deposit. +- **The DOI is checked against the paper before its record is trusted.** The + DOI is typed by hand or read off page 1, and the deposit is the one retrieval + route that can replace the *entire* reference list — a companion paper, an + erratum or an earlier version can carry exactly as many references as the body + cites, so the count test passes and the run would print "numbering confirmed" + over another paper's bibliography. The record's title is now compared with the + paper's own, tri-state like every other title check here: a mismatch sets the + deposit aside, and too little title to compare leaves the list in use with the + identity disclosed as unconfirmed rather than assumed either way. The DOI used + and where it came from are printed and recorded. +- **The paper's title comes from the paper, not from its layout.** Source maps + record `declared_title`, the title the PDF states in its own metadata. + Measured on the seven-paper spread, the first heading is the article-type + banner whenever the layout heuristic was wrong — `CLINICAL GUIDELINE`, + `RESEARCH ARTICLE`, `Journal Pre-proofs`, `Editorial` — while the metadata + carried the exact title for six of the seven. Docling does not help here: on + the seventh it emits no `title` item at all. A declaration that is not + title-shaped (too few words, a producer's filename, a `Microsoft Word -` + prefix) is passed over for the layout, because an author's PDF declares the + name of the file it was exported from, and this tool's main case is an + author's PDF. `scout` uses the same title to identify the paper, so its + Europe PMC lookup stops searching for "RESEARCH ARTICLE". +- **A paper's bibliography identifies it when its title cannot.** Where the + title comparison is unverifiable, the deposit is checked against the reference + list printed in the paper: 38 of 41 deposited works appear in the printed list + for the audited paper, against 0 of 41 for a different paper's list. Compared + as a set, never positionally — the same pair scores 34% in order, because that + paper's parse is the misnumbered one this feature exists to catch, so the + numbering cannot be an input to the identity test. The asymmetry is + deliberate: agreement is evidence of identity, disagreement is not evidence of + difference, since two lists that disagree may be one paper read badly. Across + the spread this settles all seven papers — six by title, one by bibliography, + where before it settled three. +- **Failure is disclosed, not fatal.** When neither reading can be confirmed the + audit continues, a run-level disclosure states that the numbering is + unconfirmed, and every claim citing a doubtful label carries the caveat beside + its verdict — in all three report formats. Where the two readings corroborate + each other the doubt starts at their first divergence, so a list that is right + for its first 30 entries is not tainted wholesale. +- **Three absences read differently.** No DOI, no deposit, and Crossref + unreachable are three different facts asking the reader for three different + things, and are never collapsed into one message. +- `RefManifest` gains `reference_source`, `numbering_verified`, + `numbering_note` and `unverified_from`; all additive, and an older manifest + still loads — as a parse whose numbering was never checked, which is what it + is. `scripts/reference_audit.py` reports the three counts per PDF, offline of + the model and free. + +Measured on seven papers across four publishers: all seven deposit a reference +list, and the check catches both known parse failures (43 parsed vs 41 real; +106 parsed vs 101 real). Three of the seven cite by **superscript numeral**, +which flattens to indistinguishable prose when the PDF is converted to text — +those papers have no arbiter, and are reported as unconfirmed rather than +presented as checked. + +### Fixed — eleven cited sources were downloading to one file + +Found by a live run on a JAMA editorial while verifying the above, and worse +than the `TypeError` that revealed it. `_slug` took the *first* token of the +reference, stripped non-letters, and fell back to the literal `ref` when nothing +survived. `_parse_bulleted` leaves the printed list numeral at the front of the +reference text, so the first token was `1`, `2`, `3`… and **23 of 28 references +slugged `ref-2024`**. The slug is also the download's filename, so all eleven +retrieved sources wrote to one path, each overwriting the last — every claim +citing any of them would have been judged against whichever paper downloaded +last, with no error. + +- `_slug` now takes the first token that actually contains letters. +- `_unique_slugs` guarantees no two entries in a manifest share a slug, applied + to both producers. A genuine collision needs no parser bug — the same first + author and year cited twice does it — so uniqueness is enforced rather than + assumed to follow from a better slug. The first entry keeps the natural slug, + so a `--provided` file named `-.pdf` still matches. +- `_parse_bulleted` strips the leading numeral, which also kept it out of the + Crossref bibliographic search and the title check. + +### Fixed — `--parse-only` and the offline test suite reached the network + +`refs` is also called as a plain Python function, by `run` and by the tests, and +Typer's declared default for an option is an `OptionInfo` object rather than the +value the help screen shows. `OptionInfo` is truthy, so the new `doi or +detect_doi(...)` took it for a real DOI and built a request URL out of its repr. +The offline test suite began making live Crossref calls — and passed, because +the machine running it had network. Same shape as the bug that made `ingest`'s +backend an `OptionInfo` and read every paper as flat text while reporting +layout-aware ingest. + +### Fixed — the scout's wrong-paper warning stopped firing when the DOI became a guess + +`_resolve_paper` records `via: doi` whenever a DOI is supplied, and the console +warned "wrong paper? pass --doi" only on `via: title` — so when `run` began +reading the DOI off page 1 and handing it down, a funder, data-availability or +erratum DOI could anchor the whole literature scan to somebody else's paper +*and* suppress the only signal that it had. The provenance is not recoverable +inside `scout`, and it is the wrong question: the record's own title is +comparable with the paper's. + +`ScoutResults` gains `identity` — `confirmed` / `unverified` / `mismatch`, +additive, and `""` on an older `scout.json` means not recorded rather than +confirmed. A mismatch stops the scan and says so instead of filling both +registers from another paper, since the registers *are* the finding. Too little +title to compare leaves the scan in place and discloses the unknown, the same +tri-state used for a deposit and for a downloaded source. The title comparison +itself moved to `models.titles_match`: three readers now need it, and a copy in +`scout` is the defect the other shared rules in that module exist to prevent. + +### Fixed — a `--provided` file could be judged as two different references + +`_unique_slugs` renames the second of two colliding entries to +`smith-2019-r7`, and `_provided_candidates` drops slug tokens of three +characters or fewer — so `r7`, the only thing telling the two apart, was +invisible and `sources/smith-2019.pdf` matched both. Measured: entry [7] came +back `status=provided`, `title_check=mismatch`, pointing at entry [2]'s paper, +and its claims would have been judged against it. Worse than before slugs were +made unique, when both entries shared a slug and were grouped into one source. + +"Disclosed, not fatal" still holds for a file the user *named* for a reference — +they chose it, there is nothing to fall back to, and a scanned PDF yields no +text to check. It does not hold for a file a token match found: nobody chose it +for that reference, so a title check that says "different paper" is now a reason +to keep looking. Candidates are read in rank order until one is usable, and +where the retrieval chain then finds nothing, the reason names the file that was +set aside and why — a gap that withholds what the tool already knows is the +failure this project exists to avoid. + +### Fixed — the coverage audit had its own idea of where the bibliography begins + +`coverage_audit` cut the body at `^##\s+(references|bibliography|literature)`, +a second boundary rule beside `models.is_references_heading` — which carries a +comment saying two readers need one rule because two is a defect this project +already shipped. The regex needs ingest to have *typed* the block as a heading, +and flat-text ingest guesses headings from font size, so a `References` line at +body size reaches `clean.md` with no `##`. Reproduced on a generated paper: +`labels_in_text` came back `['1','2','3']` where `[3]` appears only inside the +reference list, so the audit reported a gap that does not exist — in the one +figure it computes mechanically so that it cannot. Both the label reading and +the `clean.md` occurrence fallback now cut on the shared rule. + +### Added — `init --for ` names the case folder the way `run` would + +`init` then `run paper.pdf` used to orphan `case/sources/`: `run`/`refs` name +their own folder after the paper, so a hand-made `./case/` is only reused if +`-c case` is remembered every time. `init --for paper.pdf` now names the +folder exactly as `default_case` would, so a plain follow-up +`papertrace run paper.pdf` finds it automatically. An explicit folder name +still wins over `--for`; omitting `--for` keeps the previous `./case/` +default and its `-c` reminder. + +### Fixed — the judging call ran with the wrong repo's rules and a full toolset + +`_ask`, the only seam that calls a model, passed no `cwd` to `claude -p` and no +tool restriction. Running an audit from inside a repo silently fed that repo's +own `CLAUDE.md` into every verdict, undisclosed anywhere in the report, and the +judge held the CLI's default toolset — Bash, Edit, WebFetch — while it is only +ever supposed to read the prompt it is given and answer. `_ask` now runs with +`--safe-mode`, `--tools ""` and `cwd` set to a private, per-process scratch +directory — not the shared, world-writable system temp root, which another +local user could otherwise plant config into. + +### Fixed — the "no case folder" hint implied a search it never ran + +`check`, `highlight`, `report` and `scout` take no manuscript path, so when +`-c` is omitted and no case folder is found, the hint had nothing to look +beside and only ever checked the current working directory — but it said "no +case folder found here," which reads as an exhaustive search. Reworded to "no +case folder found in the current directory," naming the one thing that was +actually checked. + +### Fixed — a table's own numbers were read as citations + +`_LABEL_GROUP` matches `[N]` and `[N, M]` alike, and a results table's 95% CI +column is written exactly that way — `[100, 100]`, `[51, 85]`. Reproduced on a +real radiology paper: two table blocks holding CI columns supplied every +square-bracket match in the manuscript, none from prose, and pushed the highest +cited label the reconciler saw from the paper's real count to 100 — a confident, +wrong numbering read for a paper whose actual in-text citation style +(round-bracket numeric) this tool does not yet recognise at all, so the honest +answer was "unconfirmable," not "[1]-[100]." `_body_citation_labels`, +`citation_occurrences`, and `citation_labels_in_text` now skip table content — +by block type where a source map is available, by each row's own GFM `| ... |` +shape in the `clean.md` fallback, since flat text carries no block type. + +### Fixed — the numbering banner and the per-claim caveat contradicted each other + +When the doubt could not be narrowed, the run-level disclosure rendered "every +entry is affected" while `label_is_doubtful` returned False for every label for +the same reason — `unverified_from is None`. The report asserted that every +entry was suspect and marked no claim suspect, so a reader acting on a single +verdict was told nothing. An unconfirmed numbering with no recorded scope now +puts every label in doubt. Two shapes reach that state: a manifest written +before the list was reconciled at all, and two readings that agree entry for +entry with no arbiter to confirm either — the superscript-citation case, which +is about half of real papers, so those reports now carry the caveat on every +claim rather than on none. + +### Fixed — a reused source directory could hold a different paper + +`check` re-ingests a cited source only when `annotated.md` is missing, and the +directory it reuses is named after the reference's slug. A slug is not an +identity that holds still: fixing a slug collision renames one of the two +colliding entries, and the reconciler can hand `refs` the publisher's list on +one run and the parsed list on the next. Re-running an existing case could +therefore hand the model the directory's previous occupant and judge a claim, +confidently, against a different paper. `SourceMap.doc` could not catch it — +every cited source is stored as `.pdf`, so it reads the same either way. + +Source maps now record `source_sha256`, the hash of the bytes they were built +from, and a directory whose hash does not match the file now at `pdf_path` is +re-ingested. An unhashed map — written before this — counts as stale: +re-ingesting is local, free and quick, while trusting it is a guess about which +paper is in a file. The field is additive and older maps still load, where +absent means unknown and never "matches". + +### Fixed — the tool could invent a reference + +Found by the first real audit: a 43-reference Elsevier paper was reported as +having 46, and the three extra "references" were the paper's own table +captions, published in the retrieval manifest as `paywalled` works with real +DOIs attached. + +- **A resumed reference list must look like references.** `references_span` + scanned to the end of the document for any run of blocks sharing the + bibliography's block *type*, with no test on the text — so three `list` + blocks under a `TABLE TITLES` heading became references 44–46. A candidate + run now has to be at least half reference-shaped. Half rather than all, + because a genuine continuation can carry a bare-URL entry with no year. The + docstring claimed this was already the case; it was not. + +- **A non-reference is never title-searched, and a component DOI is never + accepted.** Crossref answered a title search for "Table 1. Dataset + characteristics" with `10.7717/peerj.7892/table-1` — a *table* belonging to + an unrelated paper — and nothing caught it, because the title sanity check + only runs on the download path and no copy was ever downloaded. Entries that + read as nothing citable are refused before the search, mirroring the existing + web-page gate, and any DOI naming a table, figure or supplement is rejected + wherever it came from. + + `looks_like_reference` accepts a year, a DOI, an arXiv id **or an author + list**. The author clause is not decoration: two real references in the same + paper reached the resolver truncated mid-title with no year at all, and + Crossref found both correct DOIs from the author string. A year-only test + turned them into gaps. + +- **The retrieval manifest keeps the evidence for a title check that passed.** + `title_check: verified` and `title_check: unverifiable` both arrived as bare + assurances; the detail was recorded only on mismatch. Accepted downloads now + carry it too — `title check: 18/19 reference tokens on its first page`. + +- **The scout says which failure it was.** With `--doi` supplied and no record + found, it reported "paper not identified in Europe PMC — pass `--doi` to pin + it", advising the operator to do what they had just done, and wrote + `"doi": ""` into `scout.json` so the artifact could not show what was tried. + A DOI that returns nothing means the paper is not indexed — usual for an + in-press pre-proof, and a stronger fact than a failed title heuristic. Both + registers being empty is absence of data, not a clean literature search. + +### Fixed — the literature scout, and the escaping hole it uncovered + +Found by a second live audit, of a pancreatic-cancer paper. + +- **The keyword query is about the subject now.** `_keywords` took the first + four content words of the title, so *"Image registration improves inter-reader + agreement of objective response in CT assessment of pancreas adenocarcinoma"* + searched for `image AND registration AND improves AND inter-reader` — a method + phrase containing a verb, never reaching the disease. It matched a stroke + conference abstract on the word IMPROVES. Words are now ranked by length as a + proxy for topical specificity rather than by position, and a short list of + words that state what a paper *claims* rather than what it is *about* + (`improves`, `reduces`, `assessment`, …) joins the stop list. The same title + now yields `adenocarcinoma AND registration AND inter-reader AND agreement`. + +- **A paper from the manuscript's own year is no longer "existed but uncited".** + That register invites the reader to ask what the authors missed, and a + same-year paper may have appeared after submission — on the audited paper all + fifteen candidates were from its own year. `same_year` is a third register, + rendered apart and labelled, because folding it into either neighbour states + something false and dropping it would lose a finding a reviewer might + legitimately raise. Additive in `schemas/scout.schema.json`; an older + `scout.json` still loads. + +- **Europe PMC's escaped markup is decoded.** Titles arrived as + `CTV<sub>boost</sub>` and were rendered verbatim. + +- **The HTML reports actually escape their interpolations.** `report.py` passed + `select_autoescape(["html"])`, which matches a name ending in `.html` — the + templates are `report_editor.html.j2` and `report_terminal.html.j2`, so + nothing ever matched and **autoescape was off for all three formats**. It + stayed invisible because the one field carrying angle brackets, a Europe PMC + title, arrived pre-escaped from the API; decoding those entities above is what + made it reachable. Cited source PDFs are downloaded from third parties and + their text reaches the report, so this was not hypothetical. Matched on + `.html.j2` now. Markdown is not HTML and is left verbatim. + +### Changed + +- **A substantive verdict must now name a page and a block the source actually + has, and must be showable.** `check` validates every `supported`, `partial` + and `contradicted` judgement against the cited source's own + `source_map.json`: the page must exist, `source_block` is now **required**, + and it must sit on the page the verdict names. Anything else is + `unchecked` with a note, never a verdict. `highlight` enforces the same rule + against reality — a substantive judgement that produced no evidence image is + downgraded there too, because the PDF can be missing from + `sources_resolved/` and a source map can disagree with the PDF it came from. + + The block requirement is what makes the picture unconditional: the crop + region is the block's bbox, so a valid block always yields an image and the + anchor phrases only decide whether a red box is drawn on it. `CHECK_PROMPT` + already asked for `source_block` and already told the model to omit it only + for `not_addressed`, so no prompt text changed and eval runs stay comparable + across this release. + + **This changes counts.** A run that previously reported a verdict resting on + page-only provenance, an impossible page or a nonexistent block now reports a + gap. `not_addressed` is unaffected — it never claimed a passage. + +- **A source with no `source_map.json` can no longer produce a verdict.** Its + judgements are `unchecked`, with a note naming the re-ingest that fixes it. + Previously the location it named could not be checked against anything. + +### Fixed + +- **Two ways around the one-case-one-paper guard.** `papertrace ingest` never + consulted `_guard_case`, so a different paper could overwrite + `/ingest/manuscript` — the slot `refs` fills and the coverage audit + reads — while the manifest still described the first paper. The guard now + runs whenever the output *is* that slot, recognised by shape so `--out` + cannot walk in behind `-c`'s back; a cited source ingested into + `/ingest/` is untouched. And `refs --parse-only` on a pre-hash + case re-ingested the manuscript slot and then returned before writing the + manifest; an inspection command now reads the paper into a temporary + directory and mutates nothing. + +- **Claims whose headline is `not_retrieved` or `unchecked` now show their full + per-source state.** The gap sections printed the claim text alone, so a claim + citing [1,2] where source 1's check failed and source 2 was never obtainable + said neither thing, and a `not_addressed` from a source that *was* read + vanished behind the `unchecked` headline that outranks it. All three formats + now render the co-citation breakdown, the unretrieved co-citations and one + row per judgement with its note. The editor look also labelled a whole + section row with `items[0].verdict`, calling a mixed section whichever + verdict came first; it is one row per claim now. + +- **The anchor tri-state is no longer flattened.** `anchor_located` is `True` + (searched and located), `False` (searched, not located) or `None` (never + searched) — three facts. The disclosure was gated on `evidence_image`, so a + verdict with a page and no crop disclosed nothing; it is gated on provenance + now, with wording that does not describe a picture that was not written. The + `highlight` console branched on truthiness and described `None` as "no anchor + phrase found on the page", asserting a search that never happened. + +### Evaluation harness + +Developer tooling; none of this affects an ordinary audit. + +- Gold-case eligibility is decided **before** alignment, not after. An + unresolved or drift-invalidated case used to compete for predictions and + consume the one an eligible case needed — which then reported as the tool's + extraction gap, moving blame off the tool silently. +- Cases that were never eligible no longer vote in repeated-run agreement. +- Duplicate prediction ids are refused with an error naming them, instead of a + dict comprehension keeping whichever came last — the one place alignment's + documented order-independence did not hold. +- Repeated-run agreement enforces the whole **(`set_id`, prompt fingerprint, + ingest converter)** triple. The error message already claimed the triple + while only `set_id` was checked. +- The two agreement figures are renamed for what they are: **penalized** + (a genuine lower bound) and **complete-case** (a different population, not a + bound in either direction). `intersection` was labelled the upper bound, + which is false — dropping a case whose true agreement is high pulls the mean + down. +- `not_addressed` is a rendered confusion-matrix **column**, not only a row. + The arithmetic always had four classes; the table printed three, so a + mistake was counted and then hidden. +- `evals/DESIGN.md` describes all four judgement classes. + +### Documentation + +`README.md` corrections, each a statement that did not match the code: page +provenance is not universal (`not_addressed` has none by design) and is now +page *and* block; an unboxed crop needs a valid block to exist at all; +`not_addressed` is deliberately unranked in the headline rule; the default case +folder is the paper's stem, not `case/`; text drawn inside a raster figure has +no text layer to box; and both Quick Starts need `git clone` because PaperTrace +is not on PyPI. + ## [0.4.0] — 2026-08-30 (beta) ### Added diff --git a/CITATION.cff b/CITATION.cff index 9f28f82..ee13374 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -10,7 +10,7 @@ authors: repository-code: "https://github.com/defraction0/PaperTrace" url: "https://github.com/defraction0/PaperTrace" license: MIT -version: 0.4.0 +version: 0.6.0 date-released: 2026-08-30 keywords: - fact-checking diff --git a/CLAUDE.md b/CLAUDE.md index fb43ec5..df67878 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -53,13 +53,26 @@ End-to-end smoke test — needs network and a logged-in `claude` CLI, ~5 min: ```bash python examples/demo/make_manuscript.py -papertrace run examples/demo/demo_manuscript.pdf -c demo_case -# expect: 2 supported · 2 contradicted · 1 not retrieved · 1 uncited assertion +# --model is pinned so the committed showcase is reproducible: without it +# `claude -p` takes the account default, which silently changed the judge +# from opus to haiku between two regenerations of examples/demo/output/ +papertrace run examples/demo/demo_manuscript.pdf -c demo_case \ + --model claude-opus-5 --format terminal --png +# expect: 1 supported · 2 contradicted · 1 not retrieved · 1 uncited assertion +# 4 claims, not 5: the sentence citing [2] and [3] comes back as ONE +# multi-source claim, because 0.5.0 asks extraction for the verbatim sentence. +# Reproduced on both claude-opus-5 and claude-haiku-4-5, so it is the prompt +# and not the model. Both planted contradictions, the paywalled [4] and the +# uncited assertion are what actually matter and are unaffected. ``` -Install: `pip install -e ".[dev]"` for development (this is also exactly what CI -installs — `full`/`docling`/`png` pull docling, torch and playwright, which must -stay out of CI). `playwright install chromium` once, only for `--png`. +Install: `pip install -e ".[dev]"` for development (this is also exactly what +CI installs). **`docling` is a base dependency as of 0.5.0**, so CI installs it +and torch with it — but never *runs* it: the ~500 MB layout models download on +first use, not on install, and every test pins `backend="pymupdf"` +(`check_claims` makes `backend` a required argument so none can forget). +`playwright` must still stay out of CI; it is the `png` extra, needed only for +`--png`, with `playwright install chromium` once. ## Architecture @@ -71,19 +84,38 @@ there is no in-memory pipeline object: ingest → refs → scout → check → highlight → report ``` -- **`ingest/`** — two backends behind one contract: `pymupdf_.py` (always - available, flat text, tables linearized) and `docling_.py` (optional, - layout-aware, ~500 MB model download on first run). `backend="auto"` prefers - docling and falls back loudly. Everything downstream reads only - `source_map.json` and does not know which backend ran. +- **`ingest/`** — two backends behind one contract: `pymupdf_.py` (flat text, + tables linearized) and `docling_.py` (layout-aware, ~500 MB model download on + first run). Both are installed; `--backend pymupdf` is a deliberate choice, + not a fallback for a missing package. `backend="auto"` resolves through the + shared `resolve_backend()` and falls back loudly. Everything downstream reads + only `source_map.json` and does not know which backend ran — except + `check._stale_ingest`, which compares the recorded `converter` so a source + map from an earlier run with the other backend is rebuilt rather than reused. + **Cited sources are ingested with the same backend as the paper** (0.5.0); + each source's converter travels in `RunResults.source_converters` and a + flat-read source is named in all three reports. - **`refs.py`** — resolves citations through legal open-access routes only (Crossref → Unpaywall → Europe PMC → arXiv), with a title sanity check that rejects a mismatched download rather than judging against the wrong paper. - Per-ref status from `REF_STATUSES`. + Per-ref status from `REF_STATUSES`. Also attaches **supplements** (0.6.0): + `_named_for` is the one token-match rule, `_provided_candidates` and + `_supplement_candidates` are that rule with `_SUPPLEMENT_RE` inverted, and a + supplement attaches only to an already-available reference — the orphan is + reported by `unused_provided`, never silently dropped. + **`identify_by_content` is the second pass**, for files the filename rule + cannot place: DOI first, then `titles_match` against a *short* title string. + Deliberately **not** `_title_check_text` — that counts a reference's words + across a whole page, which is right for vetoing a file the user already named + and measurably wrong for discovery (it verified one demo source against two + unrelated references). A non-unique match is refused, never ranked, and + `titles_match` returning `None` is not an accept: a filename carries the + user's assertion, content carries none. - **`check.py`** — the **only** module that calls a model, and only through the `_ask()` seam (`claude -p` subprocess; inherits the user's Claude Code login, no API key). Two prompts: `EXTRACT_PROMPT` then `CHECK_PROMPT`, one call per - source so context stays small. Also holds `coverage_audit()`, which is + **document** so context stays small — an article, each of its supplements, + and each of the audited paper's own are separate calls with separate verdicts. Also holds `coverage_audit()`, which is deliberately **mechanical and prompt-independent** — a regex (`_LABEL_GROUP`) over bracketed numeric labels, so a citation the extractor missed still surfaces. The module global `_LAST_MODEL` carries the judging @@ -93,9 +125,14 @@ ingest → refs → scout → check → highlight → report them with PyMuPDF `page.search_for` and draws the boxes. Boxes are never model-placed or hand-placed. - **`models.py`** — the dataclasses *are* the wire format. `VERDICTS`, - `REF_STATUSES` and `BLOCK_TYPES` are the vocabularies; `to_json`/`from_json` - pairs must stay symmetric, and `from_json` uses `.get(...)` defaults so older - `results.json` files still load. + `REF_STATUSES`, `BLOCK_TYPES` and `DOCUMENT_KINDS` are the vocabularies; + `to_json`/`from_json` pairs must stay symmetric, and `from_json` uses + `.get(...)` defaults so older `results.json` files still load. **A judgement + target is a document, not a reference**: `RefManifest.document(slug)` / + `.documents()` resolve an article, a cited work's supplement or the audited + paper's own behind one interface, so no consumer hand-rolls + `next(e for e in entries if e.slug == slug)` — that shape can only ever find + an article, and every supplement would be invisible to it. - **`report.py`** — Jinja2 over `src/papertrace/templates/` (three templates: markdown, editor HTML, terminal HTML). Templates are **package data** loaded via `importlib.resources`, not a repo-relative path — an installed wheel has @@ -119,24 +156,35 @@ Three decisions not to re-litigate: byte for byte.** `evals/align.py` reads `missing` as a list of label strings to decide whether an unmatched gold case is the tool's failure or the evaluator's; reshaping it would move that blame silently, with no test going - red. Everything occurrence-level is additive under `"schema": "coverage/2"`. - In particular `covered` is *not* "labels with ≥1 covered occurrence" — that - would push an all-uncertain label into `missing`. + red. Everything occurrence-level is additive; `"schema"` is `coverage/3` + since 0.5.0 and `coverage/2` files still validate. In particular `covered` is + *not* "labels with ≥1 covered occurrence" — that would push an all-uncertain + label into `missing`. - **`uncertain` is a third status, never folded into either.** An attribution the tool cannot make counts as *not covered*, and the uncertain count is always printed beside the ratio: when it is large the ratio is close to - meaningless, and a percentage alone hides that. -- **Reading-order zipping is rejected.** `EXTRACT_PROMPT` asks for reading - order, so pairing claim *n* with occurrence *n* is tempting. The order is - unverified and degrades silently — one skipped claim shifts every later - pairing and manufactures confident, wrong attributions. Attribution is - location narrowing plus text similarity assigned globally best-first, - accepted only on `ratio ≥ 0.45` **and** `margin ≥ 0.10`; the margin is the - decisive test, since the question is only *which* occurrence. - -The attributor duplicates ~10 lines of normalize-and-ratio with -`evals/align.py` **on purpose**: `papertrace` cannot import `evals` (not in the -wheel), and `evals` must not import a matcher from the thing it grades. + meaningless, and a percentage alone hides that. Since 0.5.0 it has exactly + one cause: a claim cites a label and names none of that label's contexts, so + a claim reached one of them and nothing can say which. +- **Attribution is a lookup, not a match** (`coverage/3`). `citation_occurrences()` + builds the inventory **before** the model call, `_render_inventory()` renders + it as `ctx_NNNN` into `EXTRACT_PROMPT`, and each claim comes back carrying + the ids it was taken from — resolved through the map built in that same pass, + in `extract_claims`, and stored in `ClaimResult.ctx_ids`. A `ctx` not in the + inventory is **dropped**, never repaired into "the first occurrence of that + label". + + This replaced ~130 lines of similarity matching (`_attribute_label`, + `_normalize_for_match`, `_ratio`, `_location_matches`, `OCCURRENCE_MIN_RATIO`, + `OCCURRENCE_MIN_MARGIN`). Do not reintroduce a text-similarity fallback for + an unresolvable `ctx`: that is the confident-wrong-pointer failure the + redesign removed, and `uncertain` is the honest answer instead. +- **Reading-order zipping is still rejected**, and `ctx_NNNN` is not a licence + to reintroduce it. The labels are *assigned* in reading order, but they are + resolved through the mapping built with them — never by re-deriving position + later. Any consumer that pairs the *n*th ctx with the *n*th occurrence of a + freshly recomputed list has rebuilt the bug: one dropped occurrence shifts + every id after it, silently. ## Non-negotiable gates @@ -208,7 +256,10 @@ honest scope — when behaviour changes, that list changes with it. Specific current constraints documented there, worth not re-breaking: the coverage audit reads bracketed numeric labels only; batch mode judges a co-cited claim against every retrievable source and reports the most adverse verdict as the claim's -headline; the model reads extracted text with page +headline — where `not_addressed` is deliberately unranked and becomes the +headline only when no source addressed the claim at all; a substantive verdict +must name a page and a block that exist in the source's own map, so a verdict +nobody can be shown is `unchecked`; the model reads extracted text with page markers, not page images. Update `CHANGELOG.md` for any user-visible change, and `README.md` when flags, diff --git a/README.md b/README.md index bec6b79..fae2622 100644 --- a/README.md +++ b/README.md @@ -36,10 +36,12 @@ A missed citation is reported, not silently skipped.

> **See the result first — no install needed.** The demo report committed at > [`examples/demo/output/report.md`](examples/demo/output/report.md) audits a > fictional mini-review with planted citation errors and real, published -> references: **2 supported · 2 contradicted · 1 not retrieved · 1 uncited +> references: **1 supported · 2 contradicted · 1 not retrieved · 1 uncited > assertion** — the planted errors, and exactly them, *in that run*. Extraction > and judgement are model steps, so the committed report is an inspectable -> artefact, not a guaranteed re-run. +> artefact, not a guaranteed re-run: the same demo audited under 0.4.1 returned +> *2* supported, because extraction split one sentence citing two references +> into two claims instead of keeping it whole. Same defects found either way. Pick a paper that matters to you — the landmark your project builds on, the method paper you are about to adopt, your own published work. PaperTrace @@ -58,45 +60,129 @@ went uncited?** distinctive words on the retrieved first page; a first page that is **empty or unreadable** (scanned, image-only) **passes** — unverifiable is not the same as wrong, so a scanned source is checked rather than silently discarded. +- Judge the manuscript's **own sentence**, not a summary of it. Extraction + returns the verbatim sentence carrying each citation alongside a short + paraphrase, and the verbatim sentence is what the source is checked against — + because the population, the effect size, the confidence interval and the + hedging are exactly what decide whether a citation supports a statement, and + they are the first things a summary loses. The quote appears in the report + above each verdict, so you can see what was actually judged. Where extraction + returns no quote the paraphrase is judged instead, and the report shows that + is what happened. - Attempt to extract **every** citation-backed claim, then judge each against - the text of its cited source, with page-level provenance for every verdict. + the text of its cited source. Every `supported`, `partial` or `contradicted` + verdict carries a page **and** the source block it rests on, both checked + against that source's own ingest — a verdict naming a page or block the + source does not have is reported `⚠ not checked`, not published. `◌ does not + address the claim` carries no page by design: the source was read and says + nothing, so there is no passage to point at. Extraction is a model step, so it is an attempt, not a guarantee — which is why the coverage audit below exists. - Show the evidence: real page crops with the matched text boxed in red. Claude proposes the page, the block and verbatim anchor phrases; Python then finds those phrases in the PDF and draws the boxes — placed by text search, - never by hand, and never by the model. A crop whose anchor matched nothing - is shown unboxed and labelled as such. + never by hand, and never by the model. The crop region comes from the source + block the verdict names, so a crop whose anchor phrase matched nothing is + still shown — unboxed, and captioned as unboxed. Where no anchor phrase was + offered at all, the caption says that instead: "searched and not found" and + "never searched for" are different facts and are never merged. - Preserve unavailable sources as explicit gaps: a claim whose source couldn't be retrieved is `⊘ not retrieved` — recorded, never guessed. +- **Check its own reference numbering before trusting it.** The citation label + is the join key between a claim and the source it is judged against, so a + list off by one produces a confident audit of the *wrong papers*. Two + independent readings are taken — the tool's parse of the printed list, and + the reference list the publisher deposited with Crossref (`refs --doi`, + defaulting to the DOI printed on page 1) — and the manuscript's own `[N]` + markers arbitrate between them. A reading is used only if it accounts for + exactly the labels the body cites. When neither does, the audit continues, + the report says the numbering is unconfirmed, and every verdict on a claim + citing a doubtful label carries that caveat beside it. Crossref is a second + reading, **not** an oracle. A deposit this tool can only partly read is set + aside rather than used to renumber a longer list, and the shortfall is + reported as the tool's own, not the publisher's. A deposit can also be + genuinely short — one record in this project's spread carries 2 references + for a paper citing about 40 — and nothing in the payload gives that away, + because Crossref's own count field counts what was deposited. The + manuscript's labels are what catch it. And the DOI is checked against the + paper before its record is trusted: a deposit whose Crossref record is titled + as some other paper is set aside. The paper's title is taken from the PDF's + own metadata where it states one, since the largest heading on a first page + is often the article-type banner rather than the title. Where the titles + cannot be compared, the paper's own bibliography settles it — the works the + publisher deposited are looked for in the reference list printed in the + paper — and where neither can, the list is used and the manifest says the + identity behind it was never confirmed. - Report every citation **occurrence** — each bracketed marker at its own place in the text — that no extracted claim reached, so a second sentence citing an already-checked reference is not silently counted as covered. It also registers assertions carrying no citation at all. **Detection** is mechanical and prompt-independent (a regex over bracketed numeric labels): if extraction - skipped a citation, it shows up here. **Attribution** of a claim to a specific - occurrence is a text match the tool can get wrong; an attribution it cannot - make is reported as *uncertain* and counted as **not** covered, never as - covered. + skipped a citation, it shows up here. **Attribution** of a claim to a + specific occurrence is a *lookup*, not a guess: extraction is handed that + same mechanical inventory of citation places and returns which ones each + claim came from. It is still a model step and can name the wrong place, but + it is no longer a text comparison between a paraphrase and a sentence. A + claim that names no place is reported as *uncertain* and counted as **not** + covered, never as covered. - Judge a co-cited claim against **every** cited source it could retrieve, one model call each, and show the passage behind each verdict. Co-citation is an offer of support, so each source is checked on its own text: a claim citing four references gets four verdicts, four notes and four evidence crops, with a count beside it (*"4 cited sources checked: 2 fully support it; 1 partially supports it; 1 contradicts it"*). The claim's headline is the **most adverse** - verdict any of them gave, so one dissenting source is never averaged away. - A source that turns out to say nothing about the claim is `◌ does not address + verdict any of them gave, so one dissenting source is never averaged away — + and on a multi-source claim the headline says so on its own line + (*"❌ contradicted — most adverse of 4 cited sources"*), because a compound + sentence may legitimately draw different parts from different references, and + one citation conflicting is not the same finding as the statement being + wrong. A source that turns out to say nothing about the claim is `◌ does not address the claim` — an inapt citation, distinct from a contradiction and from a - retrieval gap. + retrieval gap. It is deliberately **not ranked** among the three: while any + source actually spoke to the claim, that source decides the headline, and + `◌` becomes the headline only when no available source addressed the claim + at all. The per-source breakdown beside the headline is where an inapt + citation stays visible. - Disclose its ingest fidelity: every report — markdown, editor and terminal — - names the converter that read the **audited paper**, and a flat-text fallback - says so loudly. Cited sources are ingested separately (see *Tables and - figures are evidence too*). + names the converter that read the audited paper, and a flat-text fallback + says so loudly. The cited sources get the **same** backend, and any source + that was nonetheless read as flat text is named by slug in all three reports + — a verdict resting on a linearized table is weaker than one resting on the + table (see *Tables and figures are evidence too*). +- **Identify a reference PDF from the file itself**, not from what it is + called. A folder of publisher-named downloads (`s41467-023-39631-x.pdf`, + `mmc1.pdf`) is matched by each file's own DOI, else by its own title against + the reference list. A file named for its reference is still taken at your + word first. Anything that cannot be placed — unrecognisable, or matching two + references equally — is listed with the reason rather than skipped. +- Read **supplementary material** you supply, as its own document. A cited + reference may carry several — dropped in the sources folder named after the + reference — and the audited paper's own are named with `--supplement`. Each + gets its own model call, verdict, page anchor and evidence crop, and a claim's + headline is the most adverse across all of them, so a contradiction that lives + only in an appendix is still reported. A supplement never stands in for the + article: one whose article could not be obtained is named and set aside. - Keep the human responsible for interpretation — it prepares evidence and drafts; the conclusions are yours. **PaperTrace does not** +- **Always** establish that a supplement belongs to the work it was attached + to. Where its own title or DOI names that work, it does; where only its + filename matched, nothing read it, and the audited paper's own are whatever + you passed to `--supplement`. That last case is the thinnest provenance + anything here carries, and all three reports name which files it applies to + rather than warning about every supplement equally. +- **Guarantee** it can place every PDF you supply. Identification needs a + readable title or a DOI on the first page, so a scanned copy with no text + layer is unplaceable — and a title matching two references is refused rather + than guessed. Either way the file is listed with the reason, never skipped in + silence. +- Count a citation that appears **only inside a supplement**. The coverage audit + reads the manuscript, so a reference cited nowhere but in supplementary + material is absent from the labels rather than reported as uncovered. Stated + in the report whenever supplements were read. + - Bypass paywalls — what it can't get legally, it reports as not obtainable. - Treat model memory as evidence — verdicts come only from retrieved or user-provided pages. @@ -109,6 +195,18 @@ went uncited?** labels only — `[12]`, `[7,8]`, `[9-11]`. Author-year, parenthetical and bare-superscript styles are not audited, and the report says *"coverage not audited"* rather than quietly reporting zero gaps. + This is not a rare corner: superscript numerals lose their superscript when a + PDF is flattened to text, so `burnout.1` arrives as `burnout. 1` + and is indistinguishable from prose. Three of the seven papers in this + project's test spread — Wiley, AMA and one Elsevier journal — cite that way. + For those papers the numbering has no arbiter either, so the reference list + is reported as unconfirmed rather than presented as checked. +- Count a mixed claim as mixed in the **totals**. The per-claim headline is + qualified and the per-source breakdown sits beside it, but the run's summary + counts (and `results.json`) tally each claim once, under its headline — so a + claim whose four sources split 2 support / 1 partial / 1 contradict appears in + the `contradicted` total and nowhere else. Read the totals as *"claims with at + least one contradicting source"*, not as *"claims that are wrong"*. - Read the source pages as images. In batch mode the model receives the cited source as extracted text with `page / block` provenance markers — the page picture is for you, in the evidence crop, not for the judge. @@ -138,10 +236,14 @@ went uncited?** ### Guided — `papertrace`, and answer the questions ```bash +git clone https://github.com/defraction0/PaperTrace && cd PaperTrace pip install -e ".[full]" # standard install — layout-aware ingest papertrace # asks for the paper, the DOI and your email ``` +*(PaperTrace is not on PyPI yet, so the clone is not optional — `pip install -e .` +installs the checkout you are standing in.)* + Nothing to memorise. It checks your setup first — so a missing `claude` CLI is a sentence before you type anything, not a traceback twenty minutes in — then asks one question at a time: the paper (drag the file in; quotes and escaped @@ -156,7 +258,7 @@ terminal — a pipe, a CI job — bare `papertrace` prints help instead of waiti on stdin.

- The guided audit in a terminal: a setup check listing the claude CLI and layout-aware ingest as present and PNG export as missing with its one-line fix, then the questions one at a time — the paper's path, the case folder, a DOI found on the first page offered for confirmation, a contact email it offers to remember — and finally the cost stated as up to 25 model calls before asking permission to start. + The guided audit in a terminal: a setup check listing the claude CLI and layout-aware ingest as present and PNG export as missing with its one-line fix, then the questions one at a time — the paper's path, the case folder, whether any cited PDFs are already to hand, whether the paper has supplementary material of its own, a DOI found on the first page offered for confirmation, a contact email it offers to remember — and finally the cost stated as a number of model calls before asking permission to start.

### Interactive — the `/review` skill (deepest mode) @@ -169,7 +271,7 @@ claude # start Claude Code here ``` *(First run of the layout backend downloads docling's models — ~500 MB, once. -On a constrained machine, `pip install -e .` gives the light flat-text core.)* +On a constrained machine, `--backend pymupdf` skips it and takes flat text.)* The interactive audit interviews you: the paper's PDF, any reference PDFs you already have — and, if you are using it for peer review, screenshots of your @@ -180,6 +282,7 @@ and batching its questions. ### Batch — one command, scriptable ```bash +git clone https://github.com/defraction0/PaperTrace && cd PaperTrace pip install -e ".[full]" # standard install (see matrix below) export PAPERTRACE_EMAIL="you@example.org" # Unpaywall asks for a contact papertrace run paper.pdf --provided ./my_pdfs # case folder: ./paper/ beside the PDF @@ -189,58 +292,145 @@ Install options: | Command | What you get | |---|---| -| `pip install -e ".[full]"` | ⭐ **standard install** — layout-aware ingest (real tables, figures, lists) + PNG rendering. Pulls torch; first run downloads docling's layout models (~500 MB, once) | -| `pip install -e ".[docling]"` | layout-aware ingest only | -| `pip install -e ".[png]"` | PNG report rendering only | +| `pip install -e .` | ⭐ **standard install** — layout-aware ingest of the paper **and its cited sources** (real tables, figures, lists). **~1.4 GB installed** (torch 591 MB, opencv, transformers, scipy), plus a ~500 MB one-time download of docling's layout models on first run | +| `pip install -e ".[png]"` | the above plus PNG rendering of the report looks | | `pip install -e ".[dev]"` | the test and lint tooling — `pytest`, `ruff`, `jsonschema`. This is what CI installs | -| `pip install -e ".[dev,full]"` | everything: run audits **and** run the suite | -| `pip install -e .` | minimal core — flat-text ingest. For CI and constrained machines; every report will carry a "tables linearized" warning | - -> **`[full]` does not include the test tooling.** The extras are independent: -> `full` is user features, `dev` is `pytest` + `ruff`. Installing `[full]` and -> then running `pytest` finds whatever `pytest` happens to be on your `PATH` — -> usually a system one, with none of this project's dependencies — and fails -> with `ModuleNotFoundError: No module named 'pymupdf'`. If you intend to run -> the suite, install `".[dev,full]"` and invoke it as `python -m pytest`, which -> fails loudly instead of silently using the wrong interpreter. - -`--backend auto` (default) uses docling when installed and falls back to flat -text otherwise — and the report always says which one ran, because a -linearized table is a degradation worth disclosing. - -**`--provided` matches by filename**, so the name decides which file stands for -a reference. Files must contain the reference's author and year (`pyrros-2023` -matches `pyrros-2023.pdf` and `pyrros-et-al-2023-chest-radiographs.pdf`), and -where several match, an exact `-.pdf` wins, else the shortest -name. A filename that reads as supplemental material — `supplement`, `appendix`, -`supporting information`, `ESM`, `online only` — is **not** used as the source, -and if it is the only match the reference is left to the online resolver -instead: a supplement is not the paper it accompanies. Rename it to the plain -`-.pdf` if you do mean it to stand in. Provided files are -title-checked like downloaded ones, but a mismatch is recorded in the manifest -rather than refused — you named the file, so it is used and the doubt is -disclosed. - -Output in `case/out/`: `report.md` with inline evidence images, the same -report as a dark **editor-window** page and as a **terminal-run** page -(`report_editor.html`, `report_terminal.html`), plus machine-readable +| `pip install -e ".[dev,png]"` | everything: run audits, export PNGs **and** run the suite | + +Confirm what you got with `papertrace --version`. + +**As of 0.5.0 the layout backend is not optional.** It was an extra through +0.4.x, and a plain `pip install papertrace` got flat-text ingest. It is now a +base dependency, because the cited sources are read with it too: the evidence +for a subgroup claim is usually a table row, and a linearized table has lost +the row. Spending layout fidelity on the paper but not on the papers it is +judged against had the asymmetry backwards. + +That is a real cost and it is stated rather than buried: a clean virtualenv +measures **1.4 GB** after `pip install papertrace`, almost all of it torch and +its dependencies. If that is unacceptable on your machine, `--backend pymupdf` +still takes the flat path for both the paper and its sources, and every report +names each source it read that way. + +`[docling]` and `[full]` still resolve — `[docling]` is now empty and `[full]` +is just `png` — so install commands written against 0.4.x do not break. + +> **`[png]` does not include the test tooling.** The extras are independent: +> `dev` is `pytest` + `ruff`. Installing without it and then running `pytest` +> finds whatever `pytest` happens to be on your `PATH` — usually a system one, +> with none of this project's dependencies — and fails with +> `ModuleNotFoundError: No module named 'pymupdf'`. If you intend to run the +> suite, install `".[dev]"` and invoke it as `python -m pytest`, which fails +> loudly instead of silently using the wrong interpreter. + +`--backend auto` (default) uses docling; `--backend pymupdf` chooses flat text +deliberately, for speed or on a constrained machine. The report always says +which one ran — for the paper, and by name for any cited source that was read +flat — because a linearized table is a degradation worth disclosing. + +**`--provided` reads the PDFs, so their names do not have to be tidy.** Drop a +folder in as it came off the publisher's site. Each unrecognised file is +identified from **its own DOI**, and failing that from **its own title** +compared against the reference list — so `s41467-023-39631-x.pdf`, +`1-s2.0-S0140673623001234-main.pdf` and `mmc1.pdf` all find their reference +without being renamed. + +Two rules keep that from guessing. A file whose title matches **more than one** +reference is used for neither, and is named so you can rename it to choose — a +corrigendum shares nearly every word with its original, and picking the better +score there would judge a claim against the wrong paper with nothing able to +notice. And a title too thin to tell papers apart is not a match at all. + +**A filename that names its reference still wins**, because that is your own +assertion about the file: `pyrros-2023.pdf`, or anything containing the +author and year (`pyrros-et-al-2023-chest-radiographs.pdf`); where several +match, an exact `-.pdf` wins, else the shortest name. Such a file +is title-checked like a downloaded one, but a mismatch is recorded rather than +refused — you named it, so it is used and the doubt is disclosed. Content +identification only fills the gap that leaves. + +**Nothing in the folder goes unremarked.** Every PDF that ends up attached to +no reference is listed with the reason — unrecognisable, ambiguous, a spare +copy of a paper already matched, or a supplement whose article is missing. +Before this, an unmatched article PDF was skipped in silence, so a folder of +publisher-named downloads produced an audit that looked entirely normal and +used none of it. + +**Supplementary material is read, as its own document.** Drop +`pyrros-2023-supplement.pdf` beside `pyrros-2023.pdf` in the same folder and it +is judged separately: its own model call, its own verdict, its own page anchor +and evidence crop. Several per reference is fine. A claim citing `[14]` is read +against every document `[14]` has, and the claim's headline is the most adverse +of them — so a contradiction that lives only in Table S2 is still reported. + +Publisher names work here too, and they are the common case: `MOESM1_ESM.pdf`, +`mmc1.pdf` and `media-1.pdf` carry no hint of being supplements in their names +at all, but their first page says so plainly, so that is where it is read from. + +Two rules hold this together. A supplement **only attaches to a reference that +was actually obtained**; one whose article is missing is named and set aside, +because there is nothing to judge it as part of. And the report says **how each +supplement was attached**: by its own title or DOI naming that work — which +establishes it belongs there — or by its filename alone, which nothing checked. +The second is the thinnest provenance anything here carries, and it is named +per file rather than as a blanket warning over both. + +```bash +papertrace run paper.pdf --provided ./my_pdfs \ + --supplement paper_si.pdf --supplement paper_appendix.pdf +``` + +A claim that points at the paper's own `Table S3` or `eFigure 2` is then read +against those. With nothing supplied, such a claim is `not retrieved` and names +the flag — the paper said where its evidence was and nobody opened it, which is +a gap rather than an assertion made without a citation. + +Output in `/out/` — where `` defaults to a folder named after the +paper, beside the paper (`paper.pdf` → `./paper/`), and `-c` chooses another. +It holds `report.md` with inline evidence images, plus machine-readable `results.json` and `scout.json`. The retrieval manifest is written one level -up, at `case/refs_manifest.json`. Want shareable PNG images of the report -looks? Add `--png` (one-time setup: `playwright install chromium`). +up, at `/refs_manifest.json`. + +The same report also renders as a dark **editor-window** page and as a +**terminal-run** page, on request: `--format editor`, `--format terminal`, or +both (`-f` for short, repeatable). They are for sharing and for screenshots, so +they are not written unless asked for — `report.md` is the record and is always +written. Want shareable PNG images of those looks? Add `--png`, which renders +the HTML it needs whether or not you asked for it (one-time setup: +`playwright install chromium`). **`--doi` is the DOI of the paper you are auditing** — not of anything it -cites. It is optional, and it feeds only the literature scout, which has to -identify your paper in Europe PMC before it can look for work published since -or work in the field you did not cite. Nothing else in the audit uses it: the -verdicts, evidence crops and coverage figures are identical with or without. - -- **Published paper** → pass it. Without it the scout falls back to matching by - title, and a *wrong* match is silent: the scan anchors to somebody else's - paper and the two registers describe that one instead. The report flags - `resolved_via: title`, but it does not error. -- **Unpublished manuscript** → there is no DOI to pass, and the scout can never - identify it. Use `--no-scout` to skip the step rather than reading an empty - result as "nothing to find". The guided flow does this for you. +cites. It is optional, it defaults to the DOI printed on the paper's own first +page, and it feeds two steps: + +- **The reference-numbering check** (`refs`, and so `run`). It fetches the + reference list the publisher deposited with Crossref, as a second reading to + measure the tool's own parse against. The record's own title is compared with + the paper's first, so a mistyped or mis-scraped DOI cannot substitute another + paper's bibliography; a comparison too thin to settle it leaves the list in + use and the identity disclosed as unconfirmed. Without a DOI there is only + one reading, and the manifest says the numbering is unconfirmed rather than + implying it was checked. +- **The literature scout**, which has to identify your paper in Europe PMC + before it can look for work published since, or work in the field you did + not cite. + +Verdicts, evidence crops and coverage figures still come only from the +retrieved sources — but *which* source a claim is judged against depends on the +reference numbering, so a `--doi` that confirms the numbering can change the +audit's answers. + +- **Published paper** → pass it, or let it be detected. Either way the record + the scout finds is checked against the paper's own title: a record that is + some other paper stops the scan and is reported, rather than anchoring both + registers to it, and a comparison too thin to settle leaves the scan in place + with the identity disclosed as unverified. `resolved_via` says which query + answered — `doi` or `title` — and, since a detected DOI answers `doi` too, it + is not the thing to read for reliability; `identity` is. +- **Unpublished manuscript** → there is no DOI to pass. The scout can never + identify it, so use `--no-scout` to skip that step rather than reading an + empty result as "nothing to find"; the guided flow does this for you. The + numbering check has nothing to compare against either, and says so. **One case folder per paper.** `case` is only the default name — give each paper its own (`papertrace run zhang2025.pdf -c zhang2025`). Re-running the @@ -248,7 +438,11 @@ same paper into its case is fine; pointing a *different* paper at a used case is refused, so two audits can never mix. Batch checking runs on headless Claude Code (`claude -p`) — it inherits your -existing login, **no API key to configure**. It is the **only step that calls a +existing login, **no API key to configure**. Each call runs with `--safe-mode` +and no tool access, from a neutral working directory: the judge only ever +reads the prompt it is given and returns a verdict, regardless of which +project's `CLAUDE.md` or `.claude/` config happens to sit above the directory +you ran `papertrace` from. It is the **only step that calls a model**; every other step is plain Python. Ingest, crops and reports are also **deterministic** — same input, same output. Retrieval and the scout are **not**: they query Crossref, Unpaywall, Europe PMC and arXiv live, so their @@ -284,17 +478,19 @@ accusations. ## Tables and figures are evidence too -A number in a table cell, or drawn inside a figure, is still in the PDF's text -layer — so the red box lands on it whichever backend read the document. -`highlight` searches the real page, never the extracted text: +A number in a table cell is in the PDF's text layer, and so is text drawn +inside a figure **when the figure carries a text layer at all** — a vector +chart usually does, a scanned or raster-exported one does not, and nothing can +box text that is only pixels. Where the text is there, the red box lands on it +whichever backend read the document, because `highlight` searches the real +page, never the extracted text:

Two evidence crops: a table cell (N = 8382, 84.3%) and a number inside a flow-chart figure (97%), each boxed in red

-Both crops above come from **cited sources** whose block types (`table block`, -`picture block`) come from ingesting those sources with the layout backend by -hand — in batch mode `check` reads a cited source as flat text. +Both crops above come from **cited sources** ingested with the layout backend, +which as of 0.5.0 is what `check` does for every cited source by default. Whether such a number can be *claimed and checked* in the first place is a different question, decided by what the backend hands the model: @@ -302,7 +498,7 @@ different question, decided by what the backend hands the model: | | a table cell | text drawn inside a figure | |---|---|---| | **flat text** (`pymupdf`) | reaches the model linearised — the row and column it belongs to are lost | reaches the model as loose words, with no figure to belong to | -| **layout-aware** (`docling`; standard install, audited paper only) | reaches the model as a GFM table | the figure arrives as `[FIGURE: ]`; in-figure text arrives only where docling's layout model found a text region inside the figure | +| **layout-aware** (`docling`; the default, for the paper **and** its cited sources) | reaches the model as a GFM table | the figure arrives as `[FIGURE: ]`; in-figure text arrives only where docling's layout model found a text region inside the figure | On the one paper measured for this, it found none: of 9 figures, 5 carried text in the PDF's text layer, and docling emitted no text block anywhere inside a @@ -316,14 +512,18 @@ is the weakest evidence this tool produces — under the layout backend the judg may never have seen the number, and under flat text it saw the number without the figure that gives it meaning. -That layout fidelity is spent on the **audited paper**. In batch mode a cited -source that has **not yet been ingested** is ingested with the fast flat-text -backend, so its tables reach the judge linearised and its figures only as -whatever loose words sat inside them. -`check` reuses an existing `case/ingest//annotated.md` if one is already -there — so a source you ingested yourself with `papertrace ingest --backend -docling` keeps its layout, and the report does **not** currently distinguish -the two cases. +**Cited sources get the same backend as the paper** (0.5.0). Through 0.4.x they +were always read as flat text, on the theory that text anchors are all a +verdict needs — but the evidence for a subgroup claim is usually a table row, +and a linearized table has lost the row, so the asymmetry was backwards. + +`check` still reuses an existing `case/ingest//annotated.md` rather than +re-reading a source every run, and it now rebuilds one that a *different* +backend wrote: reusing a flat map under `--backend docling` would hand the +judge the linearized table while the run reported layout-aware ingest. A source +can still end up flat — you asked for `--backend pymupdf`, or its map survives +from an earlier run whose PDF is no longer on disk — and every such source is +**named** in all three reports rather than left to be assumed. ## Try the demo yourself @@ -345,13 +545,17 @@ claim checker runs on `claude -p`. pip install -e ".[full]" && playwright install chromium # 1 · install export PAPERTRACE_EMAIL="you@example.org" # 2 · Unpaywall contact python examples/demo/make_manuscript.py # 3 · build the demo paper -papertrace run examples/demo/demo_manuscript.pdf -c demo_case # 4 · audit it +papertrace run examples/demo/demo_manuscript.pdf -c demo_case \ + --model claude-opus-5 # 4 · audit it ``` When it finishes, open `demo_case/out/report.md`. Expected result: -**2 supported · 2 contradicted · 1 not retrieved**, one uncited assertion +**1 supported · 2 contradicted · 1 not retrieved**, one uncited assertion flagged, and all 5 citation occurrences — spread across the 4 labels — reached -by an extracted claim, 0 uncertain. (The scout step reports the fictional paper +by an extracted claim, 0 uncertain. That is **4** claims for 5 occurrences, +because the sentence citing both [2] and [3] arrives as one multi-source claim; +0.4.1 split it and reported 2 supported across 5 claims. What matters is the +same in both: the two planted contradictions found, [4] declined. (The scout step reports the fictional paper as *not identified* in Europe PMC — the tool would rather say so than invent neighbours. Verdict wording varies run to run, and extraction and judgement are live model behaviour that nothing in the code @@ -366,8 +570,11 @@ Details per plant: > stays an inspectable artefact, not a byte-exact expected output: judgement > wording differs between runs, and so can the page an anchor is found on — the > crop for claim 4 moved from page 1 to page 2 across two runs that reached the -> same verdict. No claim in the demo cites more than one reference, so the -> per-source breakdown and its summary count do not appear in it. +> same verdict. Whether a claim cites more than one reference varies too: in +> the committed 0.5.0 run the sentence citing [2] and [3] is a single +> multi-source claim, so the per-source breakdown and the `most adverse of 2 +> cited sources` qualifier both appear; under 0.4.1 the same sentence became +> two single-source claims and neither did. ## How it works @@ -385,7 +592,9 @@ paper.pdf ─────ingest──▶ clean.md + source_map.json (page │ └─highlight─▶ out/evidence/claim_NN.png (red box on the matched text) │ - └─report──▶ report.md · report_editor.html/png · report_terminal.html/png + └─report──▶ report.md (always) + · report_editor.html/png · report_terminal.html/png + (--format / --png) ``` The JSON contracts are versioned in [`schemas/`](schemas/). The two skills in @@ -489,7 +698,6 @@ generated: `python scripts/make_logo.py`. Changes are tracked in - [ ] MCP server — drive PaperTrace as a tool from any MCP-capable client - [ ] DOCX ingest - [ ] Revision (R1) mode polish -- [ ] GROBID-grade reference parsing - [ ] Figure-vs-text consistency pass (batch) - [ ] PyPI release - [ ] Journal review packs — may be added in the future diff --git a/docs/adr/0001-no-gold-benchmark.md b/docs/adr/0001-no-gold-benchmark.md new file mode 100644 index 0000000..ee1c662 --- /dev/null +++ b/docs/adr/0001-no-gold-benchmark.md @@ -0,0 +1,127 @@ +# ADR 0001 — No gold benchmark, and therefore no accuracy figure + +- **Status:** accepted +- **Date:** 2026-09-05 +- **Supersedes:** nothing +- **Related:** [`evals/DESIGN.md`](../../evals/DESIGN.md), + [`evals/PROPOSAL.md`](../../evals/PROPOSAL.md), ADR + [0002](0002-no-grobid.md) + +## Context + +A full-stack review of PaperTrace raised the point that judgment quality has +never been established: the test suite demonstrates that the plumbing works, +but says nothing about how often a `contradicted` verdict is real or an +important discrepancy is missed. The review proposed a modest, independently +reviewed evaluation set as more valuable than another layer of defensive +logic. + +That diagnosis is correct, and the project already agrees with it in writing. +What the review assumed was missing — a design — is not what is missing. + +**The harness exists.** `evals/` holds ten modules (`align.py`, +`agreement.py`, `metrics.py` with 22 metric functions, `eligibility.py`, +`scoring.py`, `provenance.py`, `tool_coverage.py`, `eval_report.py` and two +runners) plus twelve deterministic test modules that run in CI. +`schemas/eval_gold.schema.json` is a published contract with conditional shape +enforcement per verdict. `evals/DESIGN.md` specifies the evaluation unit, +declared denominators, the population algebra, the alignment procedure and its +failure modes, run provenance, and a gold-set freeze policy that answers three +ways rather than two. + +**The benchmark is already specified too.** `evals/PROPOSAL.md` is a written +proposal for exactly the set the review asked for: ≥40 cases, ≥15 of them in +faithful/altered pairs, all four gold verdicts represented including `partial`, +`not_retrieved` cases resting on genuinely paywalled DOIs, and +`pair_collapse_rate` as the signal that separates a checker which reads the +source from one which pattern-matches plausibility. + +**What is missing is data, and one specific precondition for it.** +`evals/DESIGN.md` requires that any set supporting a validity claim be +labelled by **≥ 2 people who did not write the prompts**, on manuscripts this +project's authors did not construct, with two independent blind labels per +case, third-party adjudication, both labels recorded, and inter-labeller +agreement reported as a *ceiling* on the achievable score. That precondition +cannot currently be met: there is one maintainer, who wrote the prompts. + +The only gold set that exists is `evals/gold/demo_v1.gold.json` — five cases, +four sources, zero pairs, `labellers: ["maintainer"]`, `independent: false`, +`authored_the_prompts: true`. `evals/scoring.py` already emits its own caveat +for that flag: *"conflict of interest: the same party authored the prompts and +assigned the gold verdicts — no validity claim can rest on this set."* + +Three options were considered: + +1. **Build the set anyway, self-labelled.** Rejected. It would satisfy the + letter of "a gold set exists" while failing the condition that makes a gold + set mean anything, and the harness would print a conflict-of-interest + caveat on every report it produced. A number nobody may cite is worse than + no number, because the number gets cited. +2. **Treat the benchmark as blocked rather than declined,** parking it until a + second labeller appears. Rejected as a status: an indefinite block that + nothing is scheduled to unblock is a decision wearing a delay's clothes, + and it would leave every future architecture review re-proposing it. +3. **Decline it, and say what the cost is.** Accepted. + +## Decision + +**No gold benchmark will be built, and no accuracy figure will be claimed.** + +- `evals/` stays as it is — the harness is not deleted. It is the artefact that + would make the benchmark cheap if the labelling precondition ever changes, + and its twelve test modules are conventional software tests of deterministic + arithmetic, which belong in CI regardless. +- `evals/gold/demo_v1.gold.json` stays a `demonstration`, never a `benchmark`. + Its `kind` enum already forbids the promotion without the labelling policy + being satisfied. +- `evals/PROPOSAL.md` stays in the repository, with its status updated to + record that it was considered and declined here. It is the design that would + be executed if the precondition changes; deleting it would mean re-deriving + it. +- The standing README rule is unchanged and now has a reason on file: **no + accuracy figure is claimed anywhere, because none has been measured.** + +## Consequences + +**The cost, stated plainly.** The changes made in v0.5.0 in response to the +same review — passing the judge a verbatim manuscript quotation instead of a +compressed paraphrase, reading cited sources with the layout-aware backend, +and qualifying the most-adverse headline — **ship unmeasured**. Each is +justified structurally rather than by a score: + +- the judge reading the author's actual sentence rather than a ≤160-character + compression of it removes a known information loss, whether or not the loss + was changing verdicts; +- a layout-aware source read cannot be worse than a flat one for evidence that + lives in a table; +- and the headline change alters wording only, so it cannot move a verdict at + all. + +None of that is evidence that judgment quality improved. It is evidence that +three specific mechanisms which could only degrade it were removed. The +difference matters and must not be blurred in the CHANGELOG or the README. + +**What stays unanswerable.** "How often is a `contradicted` verdict real?" and +"how often is a real discrepancy missed?" have no answer for this tool, will +not acquire one under this decision, and must not be answered by inference, +anecdote, or a live run that happened to look good. `missed_gap_rate` — the +safety metric `evals/DESIGN.md` names as the one that matters most — is +uncomputable without gold data. + +**What still catches regressions.** The deterministic suite, the honest- +degradation states (`not_retrieved`, `unchecked`, `not_addressed`, the anchor +tri-state, coverage `uncertain`, the numbering disclosures), and the demo +end-to-end run with its pinned result. These catch *breakage*. They do not +measure *quality*, and no combination of them may be presented as if it did. + +**Prompt changes are no longer comparable across versions, and that is +correct.** `evals/provenance.prompt_fingerprint()` is a content hash, so the +v0.5.0 prompt change invalidates comparison with any earlier run. +`evals/agreement.py` refuses outright to compare runs that do not share the +`(set_id, prompt fingerprint, converter)` triple. With no benchmark this costs +nothing, and the refusal remains the right behaviour. + +**Reopening this.** The single fact that would reverse it is the availability +of two labellers who did not write the prompts. If that changes, +`evals/PROPOSAL.md` is the plan, and nothing in this decision needs +re-litigating first. diff --git a/docs/adr/0002-no-grobid.md b/docs/adr/0002-no-grobid.md new file mode 100644 index 0000000..543e608 --- /dev/null +++ b/docs/adr/0002-no-grobid.md @@ -0,0 +1,113 @@ +# ADR 0002 — No GROBID; Crossref stays the independent second reading + +- **Status:** accepted +- **Date:** 2026-09-05 +- **Related:** ADR [0001](0001-no-gold-benchmark.md), `src/papertrace/refs.py`, + README §"Two independent readings" + +## Context + +A full-stack review observed that the custom bibliography parsing and +reference-number reconciliation carry substantial complexity, and suggested +benchmarking a specialist parser before extending those rules further. GROBID +was named specifically: it supports citation-context recognition, links +citation markers to bibliography entries, and reports PDF coordinates. The +review was explicit that it is a candidate replacement, not a guaranteed +improvement. + +The complexity is real. Measured on the current tree: + +| Region of `refs.py` | Lines | +|---|---| +| printed-bibliography parsing (`parse_references`, `_parse_bulleted`, numeral handling) | ~161 | +| Crossref deposit as a second reading | ~217 | +| corroboration (identity when the title is unreadable) | ~45 | +| reconciliation (`_covers`, `_same_work`, `_first_divergence`, `reconcile`) | ~208 | +| entry construction, slugs, slug uniqueness | ~50 | +| **contiguous total** | **~707** | + +Plus ~100 lines in `ingest/pymupdf_.py` locating and resuming the reference +span, ~145 lines of shared rules in `models.py`, and ~133 lines of +orchestration in `cli.py` — roughly 1,140 lines of source, pinned by a +1,205-line test module. + +Two facts decided this. + +**GROBID would displace far less of that than the total suggests.** Only the +~261 lines of *parsing* (bibliography text → numbered entries, including +locating the reference span) are work a parser does. The other ~470 lines — +the Crossref deposit, `deposit_corroborates`, and `reconcile` — are not parser +code. They exist because **any** reading of a reference list can be wrong and +the tool must be able to say so. `reconcile` takes candidate readings and lets +the manuscript's own `[N]` markers arbitrate between them; that machinery is +needed whether the candidate came from a regex or from a CRF. A better parser +would reduce how often `contested` and `unverified_from` fire. It would not +remove the need for them, because a parser cannot certify itself. + +**GROBID is not an oracle, and it is a heavy dependency.** Its own published +figures put reference parsing at ~0.87–0.90 F1 and citation-context linking at +0.76–0.91 F1 depending on the collection. It is written in Java with JNI calls +to native CRF and/or deep-learning libraries, distributed as a Docker image or +a Java service, and wants 2–4 GB of memory for full PDF processing. PaperTrace +is a `pip install` command-line tool whose central promise is that it degrades +honestly rather than guessing; requiring users to stand up a JVM service to +audit a PDF is disproportionate to a parsing gain that is itself probabilistic. + +Set against that, the thing GROBID would replace is not the weak link. The +three-way reconciliation was built for, and catches, exactly the failures that +motivated this concern: a 43-vs-41 parse inflation and a 106-vs-101, both +caught on real papers; 7 of 7 test-spread papers had a usable Crossref +deposit. Crossref supplies the *publisher's own* reference list at zero +deployment cost — an independent reading authored by someone other than this +tool, which is the property that makes reconciliation possible at all. + +Two arguments in GROBID's favour were weighed and found insufficient: + +- **Superscript citations.** Three of seven papers in the measured spread have + their numbering unconfirmed because the coverage audit reads bracketed + numeric labels only, and GROBID reads superscripts. This is a genuine gap — + but it is a gap in *citation-marker detection*, addressable directly + (`pymupdf` span flags already expose superscript formatting; see the + `papertrace-superscript-citations` note) without adopting a JVM service. +- **PDF coordinates.** GROBID reports them. So does the existing pipeline: + `highlight.py` locates anchor phrases with PyMuPDF `page.search_for` and + draws every box in Python. Boxes are never model-placed, and that division + of labour is not improved by a second coordinate source. + +## Decision + +**GROBID will not be adopted, and will not be benchmarked.** + +- The roadmap item `- [ ] GROBID-grade reference parsing` is removed from + `README.md`. Leaving it implies a plan that does not exist. +- `refs.py`'s parsing stays. Where it needs to improve, it improves in place — + and the honest-degradation states it already carries (`boundary_ambiguous` + with the DOI nulled, `reference_source`, `numbering_verified`, + `numbering_note`, `unverified_from`, `label_is_doubtful`, + `references_resumed`, `Reconciliation.contested`) are what keep a wrong + parse visible rather than confident. +- Crossref remains a **candidate, not an oracle**. That distinction is + load-bearing and is not softened by this decision: one publisher in the + measured spread deposited 3 of its own declared 52 references, and that + deposit is discarded by `_covers`. + +Not benchmarking is part of the decision, not an omission from it. A benchmark +is only worth running if a favourable result would change the outcome, and it +would not: the deployment cost is disqualifying independent of the F1. + +## Consequences + +- The ~707 lines stay, and stay tested. Anyone proposing to extend them should + read `_covers`, `_same_work` and `_first_divergence` first — several past + attempts to simplify these have been reverted for cause, and + `looks_like_reference`'s author-list clause in particular must keep it (a + year-only test passes every unit test and drops real references that arrive + truncated mid-title). +- If reference parsing ever becomes the dominant source of wrong audits, the + cheap escalation is **a third candidate reading** wired into the existing + `reconcile` arbitration behind an opt-in flag — not a replacement of the + parser. That path is strictly additive and leaves the two-reading behaviour + unchanged when the third is absent. This ADR does not authorise it; it + records it as the shape a future proposal should take. +- Superscript-citation support remains open and is the higher-value work on + this surface. It is unaffected by this decision. diff --git a/evals/DESIGN.md b/evals/DESIGN.md index 48c35fb..5a2a832 100644 --- a/evals/DESIGN.md +++ b/evals/DESIGN.md @@ -129,11 +129,17 @@ record and checks each `n` against its declared population, so the *next* denominator that drifts away from what it claims fails a test rather than printing a plausible number. -Judgement classes are `supported`, `partial`, `contradicted`. `not_retrieved` -is a retrieval fact; `unchecked` is a harness error. **Neither is scored as a -wrong verdict** — both get their own rates. This is the review requirement to -keep retrieval failures separate from model-judgement failures, and the -existing verdict enum already encodes the distinction. +Judgement classes are `supported`, `partial`, `contradicted` and +`not_addressed` — **four**, since `not_addressed` was added to the verdict +vocabulary. It is a judgement like the others: the model read the source and +found it silent on the claim, which is a real finding about the citation, not a +failure. It therefore takes a row *and a column* in the confusion matrix, and +counts in `macro_f1` on the same terms as the rest. + +`not_retrieved` is a retrieval fact; `unchecked` is a harness error. **Neither +is scored as a wrong verdict** — both get their own rates. This is the review +requirement to keep retrieval failures separate from model-judgement failures, +and the existing verdict enum already encodes the distinction. | Metric | Numerator / denominator | Population | |---|---|---| @@ -209,20 +215,33 @@ reason whenever anything is `__absent__`, because κ assumes every item is rated by every rater. **The run count is passed explicitly, never inferred from the first vector.** -Inference was safe only while the caller filtered to the intersection first — -the very filter that introduced the bias above. Removing the filter without -passing the count would have swapped a disclosed upward bias for an +Inference was safe only while the caller filtered to the complete-case set +first — the very filter that introduced the bias above. Removing the filter +without passing the count would have swapped a disclosed upward bias for an undisclosed arithmetic error. Ragged input raises. -**Both bounds are printed, side by side.** The *intersection* (only cases -present in every run) is the **upper** bound: it excludes the harness's own -gaps. The *union* (`__absent__`-padded) is the **lower** bound: it charges -those gaps to the model. Neither is the answer alone, so neither is printed -alone, and the omitted cases are named per run. - -**Two different `set_id`s are refused outright.** Averaging agreement across -gold sets produces a number describing no set, and no caveat repairs it. That -is a category error, not a partial comparison. +**Two populations are printed side by side, and only one of them is a bound.** +The *penalized* figure (`__absent__`-padded, every case seen in any run) is a +genuine **lower** bound: filling in any real vote where the harness never asked +can only raise the modal count. The *complete-case* figure (only cases present +in every run) was previously labelled the **upper** bound, and that was wrong — +it drops cases rather than penalising them, and a dropped case whose true +agreement is high pulls the reported mean *down*. With three or more runs the +omitted set can sit either side of the kept set, so complete-case is reported +as a different population ("how stable was the model where we actually asked +it") and explicitly not as a ceiling. The omitted cases are named per run. + +**Only eligible cases vote.** `per_case` keeps excluded rows so they can be +rendered in their own section; they are filtered out before the agreement +vectors are built. A case that was never scoreable cannot be evidence of the +model disagreeing with itself. + +**Runs that are not comparable are refused outright.** Agreement is defined +within one **(`set_id`, prompt fingerprint, ingest converter)** triple, and all +three are checked. Averaging across gold sets produces a number describing no +set; averaging across prompts or across ingest backends compares two different +systems and calls the difference instability. No caveat repairs either — a +category error, not a partial comparison. - **`modal_agreement`** (headline) — mean over cases of (modal verdict count) / k. - `unanimous_rate` — cases where all runs agree. diff --git a/evals/PROPOSAL.md b/evals/PROPOSAL.md index 758e8cf..3a15220 100644 --- a/evals/PROPOSAL.md +++ b/evals/PROPOSAL.md @@ -1,8 +1,12 @@ # Draft issue: a paired evaluation benchmark for the claim checker -**Status: draft text, not posted.** Open it as a GitHub issue only after review. -It is deliberately written to invite disagreement about the design rather than -to announce a decision. +**Status: considered and declined, 2026-09-05 — see +[`docs/adr/0001-no-gold-benchmark.md`](../docs/adr/0001-no-gold-benchmark.md).** +Not posted as an issue. The blocker is not the design below but its labelling +precondition: `DESIGN.md` requires ≥ 2 labellers who did not write the prompts, +and there is one maintainer, who wrote them. Kept in the repository unchanged +because it is the plan that would be executed if that ever changes — the text +from here down is the original draft and still reflects what should be built. --- diff --git a/evals/README.md b/evals/README.md index 7d67f1b..5ef9148 100644 --- a/evals/README.md +++ b/evals/README.md @@ -42,10 +42,13 @@ Compare repeated runs: python evals/runners/score_only.py --agreement evals/runs/ evals/runs/ evals/runs/ ``` -Reports the **intersection** (upper bound — only cases every run produced) and -the **union** (lower bound — every case seen in any run, with the gaps charged -to the model) side by side, and names which cases each run omitted. Runs from -two different gold sets are refused outright rather than averaged. +Reports the **complete-case** figure (only cases every run produced — a +different population, not a bound in either direction) and the **penalized** +figure (every case seen in any run, with the gaps charged to the model, which +is a genuine lower bound) side by side, and names which cases each run omitted. +Cases that were never eligible for scoring do not vote. Runs that differ in +gold set, prompt fingerprint or ingest converter are refused outright rather +than averaged. ## Run a live evaluation — costs money diff --git a/evals/agreement.py b/evals/agreement.py index fc1137c..5fdc991 100644 --- a/evals/agreement.py +++ b/evals/agreement.py @@ -18,6 +18,12 @@ not describe. `fleiss_kappa` itself is correct and is not touched; the guard lives in `agreement()`. +**Two figures, and only one of them is a bound.** `agreement_report` reports +a *penalized* figure over every case seen in any run and a *complete-case* +figure over the cases every run answered. The penalized one is a true lower +bound; the complete-case one is a different population and is labelled as such, +because a dropped case whose true agreement is high pulls the mean down. + **The run count is passed, never inferred.** Deriving `k` from the first vector is only safe when something upstream has already guaranteed equal lengths — which used to be the caller's intersection filter, the very thing that @@ -85,60 +91,99 @@ def agreement(vectors: dict[str, list[str]], runs: int) -> dict: } -def require_one_set_id(set_ids: list[str | None]) -> str | None: - """Refuse to aggregate two gold sets. A category error, not a partial view. +def _distinct(values: list) -> list: + """Stable, sortable distinct — values may be dicts, which are unhashable.""" + out: list = [] + for v in values: + if v not in out: + out.append(v) + return out + + +def require_one_provenance(set_ids: list[str | None], + provenances: list[dict] | None = None) -> str | None: + """Refuse to aggregate runs that are not comparable. Returns the set id. - Averaging agreement across different sets produces a number that describes - no set, and there is no caveat that repairs it — so it is refused outright - rather than reported with a warning. + Agreement is only defined within one **(set_id, prompt fingerprint, + converter)** triple. That sentence was already in the error message while + only the first third was checked: two runs of different prompts, or of + different ingest backends, were averaged into a single stability figure + that describes neither. A disagreement between them is not the model being + unstable — it is two different systems being compared. + + A category error, not a partial view, so it is refused outright rather than + reported with a caveat. """ - distinct = sorted({s for s in set_ids}, key=lambda x: (x is None, x)) - if len(distinct) > 1: + sets = _distinct(sorted(set_ids, key=lambda x: (x is None, x))) + if len(sets) > 1: raise ValueError( "refusing to aggregate runs from different gold sets: " - f"{', '.join(repr(d) for d in distinct)}. Agreement is only " + f"{', '.join(repr(d) for d in sets)}. Agreement is only " "defined within one (set_id, prompt fingerprint, converter) triple." ) - return distinct[0] if distinct else None + for field, label in (("prompt_fingerprint", "prompt fingerprint"), + ("converter", "ingest converter")): + values = [(p or {}).get(field) for p in (provenances or [])] + if len(_distinct(values)) > 1: + raise ValueError( + f"refusing to aggregate runs with a different {label}: " + f"{'; '.join(repr(v) for v in _distinct(values))}. A " + f"disagreement between two runs that read different text, or " + f"answered different prompts, is not the model being unstable." + ) -def agreement_report(vectors: dict[str, list[str]], runs: int, - run_labels: list[str], set_ids: list[str | None]) -> dict: - """Both bounds, side by side, with the omissions named. + return sets[0] if sets else None - Reporting only the intersection silently drops the cases one run never - produced; reporting only the union charges the harness's own gaps to the - model. Neither number is the answer on its own, so both are printed and - labelled as what they are. + +def agreement_report(vectors: dict[str, list[str]], runs: int, + run_labels: list[str], set_ids: list[str | None], + provenances: list[dict] | None = None) -> dict: + """Two populations, side by side, with the omissions named. + + **Neither is called a bound except the one that is.** The penalized figure + counts every case seen in any run and scores an ABSENT vote as + disagreement; replacing an ABSENT with any real vote can only raise the + modal count, so it genuinely understates stability and is a lower bound. + + The complete-case figure is *not* an upper bound, and calling it one was + wrong. It drops cases rather than penalising them, and a dropped case whose + true agreement is high pulls the reported mean **down**, not up. With three + or more runs the dropped set can sit anywhere relative to the kept set, so + the complete-case figure is simply a different population — reported + because it answers "how stable was the model where we actually asked it", + and labelled as that rather than as a bound in either direction. """ - set_id = require_one_set_id(set_ids) + set_id = require_one_provenance(set_ids, provenances) if len(run_labels) != runs: raise ValueError( f"{len(run_labels)} run label(s) for {runs} run(s)") - intersection_vectors = {c: v for c, v in vectors.items() if ABSENT not in v} + complete_vectors = {c: v for c, v in vectors.items() if ABSENT not in v} omissions = { label: sorted(c for c, v in vectors.items() if v[i] == ABSENT) for i, label in enumerate(run_labels) } - union = agreement(vectors, runs) - union["bound"] = "lower" - union["bound_note"] = ( - "includes every case seen in any run; a case the harness never asked a " - "run about counts as disagreement, so this understates stability") - inter = agreement(intersection_vectors, runs) - inter["bound"] = "upper" - inter["bound_note"] = ( - "only cases present in every run; excludes the harness's own gaps, so " - "this overstates stability") + penalized = agreement(vectors, runs) + penalized["bound"] = "lower" + penalized["bound_note"] = ( + "every case seen in any run; a case the harness never asked a run about " + "counts as disagreement. Filling in any real vote could only raise this, " + "so it is a genuine lower bound on stability") + complete = agreement(complete_vectors, runs) + complete["bound"] = None + complete["bound_note"] = ( + "only cases present in every run — a different population, not a bound. " + "The omitted cases could have agreed more or less than the kept ones, so " + "this can sit either side of the true figure") return { "set_id": set_id, "runs": runs, "run_labels": list(run_labels), - "union": union, - "intersection": inter, + "complete_case": complete, + "penalized": penalized, "omissions": omissions, "n_omitted": sum(len(v) for v in omissions.values()), } diff --git a/evals/align.py b/evals/align.py index ca0d375..9429dc2 100644 --- a/evals/align.py +++ b/evals/align.py @@ -41,6 +41,7 @@ import re import unicodedata +from collections import Counter from dataclasses import dataclass, field from difflib import SequenceMatcher @@ -124,6 +125,18 @@ def align(gold: dict, results, min_ratio: float = MATCH_MIN_RATIO, min_margin: float = MATCH_MIN_MARGIN) -> Alignment: cases = gold["cases"] by_case = {c["case_id"]: c for c in cases} + # a dict comprehension over claim ids silently keeps the LAST duplicate, so + # permuting the prediction list changed which one was graded — the one place + # this module's order-independence contract did not hold. Refused, not + # repaired: the harness cannot know which of two same-id claims was meant. + counts = Counter(c.id for c in results.claims) + if clashes := sorted(i for i, n in counts.items() if n > 1): + raise ValueError( + f"duplicate prediction id(s) in results.json: " + f"{', '.join(str(i) for i in clashes)}. Claim ids must be unique — " + f"alignment consumes each prediction once, and with a duplicate the " + f"input order would decide which one is graded." + ) pool = {c.id: c for c in results.claims} a = Alignment() diff --git a/evals/runners/score_only.py b/evals/runners/score_only.py index 916cde3..03683f6 100755 --- a/evals/runners/score_only.py +++ b/evals/runners/score_only.py @@ -73,25 +73,34 @@ def score_one(gold_path: Path, results_path: Path, out_root: Path, def score_agreement(run_dirs: list[Path], out_root: Path) -> Path: """Compare repeated runs of the SAME gold set. - The version this replaces filtered to the intersection with a bare + The version this replaces filtered to the complete-case set with a bare `if len(v) == n`, silently dropping every case one run never produced — defeating `agreement.py`'s own documented contract, in the direction that - flatters the model. Both bounds are now reported and the omissions named. + flatters the model. Both populations are now reported and the omissions + named, with only the penalized one called a bound. """ - labels, set_ids, per_run = [], [], [] + labels, set_ids, provenances, per_run = [], [], [], [] for d in run_dirs: record = json.loads((Path(d) / "eval.json").read_text()) labels.append(Path(d).name) set_ids.append((record.get("gold") or {}).get("set_id")) + provenances.append(record.get("provenance") or {}) + # `per_case` carries excluded rows on purpose — they are rendered in + # their own section. They must not therefore vote here: a case that was + # never scoreable cannot be evidence of the model disagreeing with + # itself, and an unresolved gold label is the harness's gap, not the + # model's instability. `.get("eligible", True)` so a record written + # before the flag existed still counts every row, as it used to. per_run.append({row["case_id"]: row.get("predicted") or UNMATCHED - for row in record["per_case"]}) + for row in record["per_case"] + if row.get("eligible", True)}) n = len(run_dirs) all_cases = sorted({c for run in per_run for c in run}) # a case missing from a run is ABSENT — the harness never asked — which is # a different fact from UNMATCHED, where it asked and the aligner failed vectors = {c: [run.get(c, ABSENT) for run in per_run] for c in all_cases} - result = agreement_report(vectors, n, labels, set_ids) + result = agreement_report(vectors, n, labels, set_ids, provenances) out = out_root / f"agg__{_stamp()}" out.mkdir(parents=True, exist_ok=True) @@ -105,26 +114,29 @@ def _fmt(value: float | None, spec: str) -> str: def _agreement_md(r: dict) -> str: + names = {"complete_case": "complete-case", "penalized": "penalized"} lines = [ f"# Repeated-run agreement — {r['set_id'] or 'unknown set'}", "", f"{r['runs']} runs: {', '.join(f'`{x}`' for x in r['run_labels'])}.", "", - "Two bounds, because neither is the answer alone. The intersection", - "drops cases the harness never asked some run about; the union charges", - "those gaps to the model.", "", + "Two populations, because neither answers the question alone. The", + "complete-case figure covers only the cases every run answered; the", + "penalized figure covers every case seen in any run and scores the gaps", + "as disagreement. Only the penalized figure is a bound.", "", "| | Cases | Modal agreement | Unanimous | Fleiss' kappa |", "|---|---|---|---|---|", ] - for key in ("intersection", "union"): + for key, label in names.items(): b = r[key] + qualifier = f" ({b['bound']} bound)" if b["bound"] else " (not a bound)" lines.append( - f"| **{key} ({b['bound']} bound)** | {b['cases']} | " + f"| **{label}{qualifier}** | {b['cases']} | " f"{_fmt(b['modal_agreement'], '.2f')} | " f"{_fmt(b['unanimous_rate'], '.0%')} | " f"{_fmt(b['fleiss'].get('value'), '.2f')}" f" ({b['fleiss'].get('reason', b['fleiss'].get('note', ''))}) |" ) - lines += ["", f"- *intersection* — {r['intersection']['bound_note']}", - f"- *union* — {r['union']['bound_note']}", ""] + lines += ["", f"- *complete-case* — {r['complete_case']['bound_note']}", + f"- *penalized* — {r['penalized']['bound_note']}", ""] if r["n_omitted"]: lines += ["## Cases the harness never asked about", "", "Not model disagreement — an operator gap, named so it is not", diff --git a/evals/scoring.py b/evals/scoring.py index 2c36895..0107f22 100644 --- a/evals/scoring.py +++ b/evals/scoring.py @@ -23,13 +23,19 @@ def score(gold: dict, results, gold_path: Path | None = None, if refs_drift is None: refs_drift = (provenance or {}).get("refs_status_drift") - alignment = align_mod.align(gold, results) - pairs = align_mod.matched_pairs(gold, results, alignment) - matched = {g["case_id"]: p for g, p in pairs} - - # eligibility is decided once, for both reasons, before any metric runs + # eligibility is decided once, for both reasons, BEFORE alignment — not + # merely before the metrics. A case that can never be scored used to + # compete for predictions anyway, and a prediction is consumed once: an + # unresolved gold case sitting on the same citation label as an eligible + # one took its match, and the eligible case was then reported as the tool's + # extraction gap. Blame moved off the harness and onto the tool, silently. scoreable, excluded = elig_mod.eligibility(gold, refs_drift) eligible_ids = {c["case_id"] for c in scoreable} + + eligible_gold = {**gold, "cases": scoreable} + alignment = align_mod.align(eligible_gold, results) + pairs = align_mod.matched_pairs(eligible_gold, results, alignment) + matched = {g["case_id"]: p for g, p in pairs} unresolved = [e for e in excluded if e.reason == elig_mod.GOLD_VERDICT_UNRESOLVED] drifted = [e for e in excluded if e.reason == elig_mod.REFS_STATUS_DRIFT] unverified = elig_mod.not_verified(refs_drift) diff --git a/evals/templates/eval.md.j2 b/evals/templates/eval.md.j2 index c403a98..cada7ad 100644 --- a/evals/templates/eval.md.j2 +++ b/evals/templates/eval.md.j2 @@ -102,10 +102,13 @@ asserted: ### Confusion (gold ↓ / predicted →) -| | supported | partial | contradicted | -|---|---|---|---| +{# every judgement class gets a column. `not_addressed` had a row and no + column, so a run that answered it against a `supported` gold had the + mistake counted in the arithmetic and then hidden in the picture. #} +| gold \ predicted | supported | partial | contradicted | not_addressed | +|---|---|---|---|---| {% for g, row in r.confusion.items() %} -| **{{ g }}** | {{ row.supported }} | {{ row.partial }} | {{ row.contradicted }} | +| **{{ g }}** | {{ row.supported }} | {{ row.partial }} | {{ row.contradicted }} | {{ row.not_addressed }} | {% endfor %} ## Retrieval, evidence and coverage diff --git a/evals/tests/test_agreement.py b/evals/tests/test_agreement.py index 75c42a1..604f017 100644 --- a/evals/tests/test_agreement.py +++ b/evals/tests/test_agreement.py @@ -10,7 +10,7 @@ agreement, agreement_report, fleiss_kappa, - require_one_set_id, + require_one_provenance, ) @@ -102,18 +102,22 @@ def test_fleiss_still_runs_when_nothing_is_absent(): assert "reason" in a["fleiss"] or a["fleiss"]["value"] is not None -# --- both bounds, and the omissions by name --------------------------------- +# --- two populations, and the omissions by name ----------------------------- -def test_intersection_is_the_upper_bound_and_union_the_lower(): +def test_the_penalized_figure_is_a_lower_bound_and_complete_case_is_not_a_bound(): + """Renamed from intersection/union. Only the penalized figure is a bound: + filling in an ABSENT vote can only raise the modal count. Dropping a case + can move the mean either way, so complete-case is a population, not a + ceiling — see `agreement_report`.""" vectors = {"c1": ["supported", "supported"], "c2": ["partial", ABSENT]} r = agreement_report(vectors, runs=2, run_labels=["runA", "runB"], set_ids=["demo-v1", "demo-v1"]) - assert r["intersection"]["bound"] == "upper" - assert r["union"]["bound"] == "lower" - assert r["intersection"]["cases"] == 1 - assert r["union"]["cases"] == 2 - assert r["union"]["modal_agreement"] <= r["intersection"]["modal_agreement"] + assert r["penalized"]["bound"] == "lower" + assert r["complete_case"]["bound"] is None + assert r["complete_case"]["cases"] == 1 + assert r["penalized"]["cases"] == 2 + assert r["penalized"]["modal_agreement"] <= r["complete_case"]["modal_agreement"] def test_per_run_omissions_are_named(): @@ -128,9 +132,9 @@ def test_per_run_omissions_are_named(): def test_two_different_set_ids_are_refused(): """A category error, not a partial comparison.""" with pytest.raises(ValueError) as e: - require_one_set_id(["demo-v1", "other-v2"]) + require_one_provenance(["demo-v1", "other-v2"]) assert "demo-v1" in str(e.value) and "other-v2" in str(e.value) - assert require_one_set_id(["demo-v1", "demo-v1"]) == "demo-v1" + assert require_one_provenance(["demo-v1", "demo-v1"]) == "demo-v1" with pytest.raises(ValueError): agreement_report({"c1": ["supported", "supported"]}, runs=2, @@ -154,8 +158,8 @@ def _run(name, rows): out = score_only.score_agreement([a, b], tmp_path / "out") result = json.loads((out / "agreement.json").read_text()) - assert result["union"]["cases"] == 2 - assert result["intersection"]["cases"] == 1 + assert result["penalized"]["cases"] == 2 + assert result["complete_case"]["cases"] == 1 assert result["omissions"]["b"] == ["c2"] md = (out / "AGREEMENT.md").read_text() - assert "c2" in md and "upper" in md and "lower" in md + assert "c2" in md and "not a bound" in md and "lower bound" in md diff --git a/evals/tests/test_eval_report.py b/evals/tests/test_eval_report.py index 4f464ea..7ad9859 100644 --- a/evals/tests/test_eval_report.py +++ b/evals/tests/test_eval_report.py @@ -64,10 +64,20 @@ def test_rendered_report_carries_the_denial(record): def test_every_percentage_carries_its_denominator(record): md = render(record) + # the fuzzy-alignment caveats are the one exemption: they are a property of + # the matcher, not a rate over a population. The window looks BOTH ways — + # "33% of matches were fuzzy" puts the word after the number, and a + # lookbehind-only guard missed it the moment the fraction crossed its + # threshold. + # the fuzzy-alignment caveats are the one exemption: they describe the + # matcher, not a rate over a population. The window looks BOTH ways — + # "33% of matches were fuzzy" puts the word after the number, and the + # lookbehind-only guard missed it as soon as the fraction crossed its + # reporting threshold. bare = [ m.group(0) for m in re.finditer(r"\d+%(?! \(\d+/\d+\))", md) - if "fuzzy" not in md[max(0, m.start() - 120):m.start()] + if "fuzzy" not in md[max(0, m.start() - 120):m.end() + 120] ] assert not bare, f"percentages without (k/n): {bare}" @@ -238,6 +248,6 @@ def test_no_percentage_from_the_new_sections_is_bare(gold_mini, results_mini): bare = [ m.group(0) for m in re.finditer(r"\d+%(?! \(\d+/\d+\))", md) - if "fuzzy" not in md[max(0, m.start() - 120):m.start()] + if "fuzzy" not in md[max(0, m.start() - 120):m.end() + 120] ] assert not bare, f"percentages without (k/n): {bare}" diff --git a/evals/tests/test_v041_eval_corrections.py b/evals/tests/test_v041_eval_corrections.py new file mode 100644 index 0000000..85432cc --- /dev/null +++ b/evals/tests/test_v041_eval_corrections.py @@ -0,0 +1,220 @@ +"""Six evaluation defects that let an ineligible or incomparable case count. + +Grouped in one file because they share a theme: the harness was measuring a +population it had not established. A case excluded from scoring still competed +for predictions and still voted in agreement; two runs of different prompts +were averaged together; a duplicate prediction id let input order pick a +winner; and the two agreement figures were labelled as bounds when only one of +them is one. +""" + +import copy +import json + +import pytest + +from evals import scoring +from evals.agreement import ABSENT, agreement_report, require_one_provenance +from evals.align import align + +# --- 1. eligibility is decided before alignment, not after ------------------ + + +def _gold_with_an_ineligible_rival(gold_mini: dict) -> dict: + """An unresolved case that shadows an eligible one on the same label. + + Both cases sit on label [1] and read almost alike, so they compete for the + same prediction. The unresolved one carries no gold verdict, so it can + never be scored — but it used to consume the prediction anyway, and the + eligible case was then reported as the tool's extraction failure. + """ + gold = copy.deepcopy(gold_mini) + eligible = next(c for c in gold["cases"] if c["case_id"] == "m-c01") + rival = copy.deepcopy(eligible) + rival.update({ + "case_id": "m-c00-unresolved", + "claim_text": eligible["claim_text"] + " overall", + "gold_verdict": None, + "ambiguity": "labellers split on whether this is one claim or two", + }) + rival.pop("pair", None) + gold["cases"].insert(0, rival) + return gold + + +def test_an_ineligible_case_cannot_consume_an_eligible_cases_prediction( + gold_mini, results_mini +): + gold = _gold_with_an_ineligible_rival(gold_mini) + rec = scoring.score(gold, results_mini) + + rows = {r["case_id"]: r for r in rec["per_case"]} + assert rows["m-c00-unresolved"]["eligible"] is False + assert rows["m-c01"]["predicted"] is not None, ( + "an unscoreable case took the prediction the eligible case needed" + ) + + +def test_the_ineligible_case_still_gets_a_row_and_a_reason(gold_mini, results_mini): + """Excluding it from alignment must not delete it from the record.""" + rec = scoring.score(_gold_with_an_ineligible_rival(gold_mini), results_mini) + row = next(r for r in rec["per_case"] if r["case_id"] == "m-c00-unresolved") + assert row["excluded_reason"] == "gold_verdict_unresolved" + assert "labellers" in row["excluded_detail"] + + +# --- 2. duplicate prediction ids --------------------------------------------- + + +def test_duplicate_prediction_ids_are_refused_not_silently_collapsed( + gold_mini, results_mini +): + """`{c.id: c for c in results.claims}` kept whichever came last, so input + order decided which of two same-id claims was graded. Alignment is + documented as order-independent; this was the one place it was not.""" + doubled = copy.deepcopy(results_mini) + clash = copy.deepcopy(doubled.claims[1]) + clash.id = doubled.claims[0].id + doubled.claims.append(clash) + + with pytest.raises(ValueError, match="duplicate prediction id"): + align(gold_mini, doubled) + + +def test_the_duplicate_error_names_the_offending_ids(gold_mini, results_mini): + doubled = copy.deepcopy(results_mini) + clash = copy.deepcopy(doubled.claims[1]) + clash.id = doubled.claims[0].id + doubled.claims.append(clash) + + with pytest.raises(ValueError) as exc: + align(gold_mini, doubled) + assert str(doubled.claims[0].id) in str(exc.value) + + +# --- 3. agreement compares like with like ------------------------------------- + + +def _prov(prompt="sha256:aaaa", converter="pymupdf"): + return {"prompt_fingerprint": {"scheme": "sha256-content", + "EXTRACT_PROMPT": prompt, "CHECK_PROMPT": prompt}, + "converter": converter} + + +def test_one_provenance_triple_accepts_matching_runs(): + assert require_one_provenance(["demo-v1", "demo-v1"], [_prov(), _prov()]) == "demo-v1" + + +def test_a_different_prompt_fingerprint_is_refused(): + """Two runs of different prompts describe different systems. Averaging + their agreement produces a number about neither.""" + with pytest.raises(ValueError, match="prompt"): + require_one_provenance(["demo-v1", "demo-v1"], + [_prov(), _prov(prompt="sha256:bbbb")]) + + +def test_a_different_converter_is_refused(): + """The judge read different text, so a disagreement is not the model's.""" + with pytest.raises(ValueError, match="converter"): + require_one_provenance(["demo-v1", "demo-v1"], + [_prov(), _prov(converter="docling 2.118.1")]) + + +def test_a_different_set_id_is_still_refused(): + with pytest.raises(ValueError, match="gold sets"): + require_one_provenance(["demo-v1", "other-v2"], [_prov(), _prov()]) + + +# --- 4. the two agreement figures are named for what they are ----------------- + + +def test_the_two_figures_are_complete_case_and_penalized_not_bounds(): + """`intersection` was labelled the upper bound. It is not one: dropping a + case whose true agreement is high pulls the mean DOWN, so the complete-case + figure can sit below the true value as easily as above it. Only the + penalized figure is a genuine bound, and only downward.""" + vectors = {"c1": ["supported", "supported", "supported"], + "c2": ["partial", "partial", ABSENT]} + r = agreement_report(vectors, runs=3, run_labels=["a", "b", "c"], + set_ids=["s", "s", "s"], provenances=[_prov()] * 3) + + assert set(r) >= {"complete_case", "penalized"} + assert "intersection" not in r and "union" not in r + assert r["penalized"]["bound"] == "lower" + assert r["complete_case"]["bound"] is None, ( + "the complete-case figure is not a bound in either direction" + ) + assert "not a bound" in r["complete_case"]["bound_note"] + + +def test_three_runs_with_a_missing_case_keep_both_populations_visible(): + vectors = {"c1": ["supported", "supported", "supported"], + "c2": ["partial", "partial", ABSENT]} + r = agreement_report(vectors, runs=3, run_labels=["a", "b", "c"], + set_ids=["s", "s", "s"], provenances=[_prov()] * 3) + + assert r["complete_case"]["cases"] == 1 + assert r["penalized"]["cases"] == 2 + assert r["omissions"]["c"] == ["c2"] + assert r["n_omitted"] == 1 + + +# --- 5. not_addressed is a rendered column, not just a row -------------------- + + +def test_the_confusion_matrix_renders_the_not_addressed_column(gold_mini, results_mini): + """The matrix has always computed four classes. The template printed three, + so a run that answered `not_addressed` when the gold said `supported` had + the mistake counted and then hidden.""" + from evals.eval_report import render + + rec = scoring.score(gold_mini, results_mini) + md = render(rec) + header = next(line for line in md.splitlines() if line.startswith("| gold \\")) + assert "not_addressed" in header, header + + +def test_a_not_addressed_prediction_against_not_addressed_gold_is_a_hit( + gold_mini, results_mini +): + """The diagonal cell for the fourth class must be reachable at all.""" + from evals import metrics + + gold = copy.deepcopy(gold_mini) + case = next(c for c in gold["cases"] if c["case_id"] == "m-c01") + case["gold_verdict"] = "not_addressed" + preds = copy.deepcopy(results_mini) + pred = preds.claims[0] + pred.verdict = "not_addressed" + + m = metrics.confusion([(case, pred)]) + assert m["not_addressed"]["not_addressed"] == 1 + + +# --- 6. an ineligible case does not vote in agreement ------------------------- + + +def test_score_agreement_ignores_cases_that_were_never_scoreable(tmp_path): + """An excluded case is in `per_case` by design — it must not therefore be + counted as the model disagreeing with itself.""" + from evals.runners.score_only import score_agreement + + def _run(name: str, predicted: str) -> None: + d = tmp_path / name + d.mkdir() + (d / "eval.json").write_text(json.dumps({ + "gold": {"set_id": "s"}, + "provenance": _prov(), + "per_case": [ + {"case_id": "ok", "eligible": True, "predicted": "supported"}, + {"case_id": "dropped", "eligible": False, "predicted": predicted}, + ], + })) + + _run("runA", "supported") + _run("runB", "contradicted") + + out = score_agreement([tmp_path / "runA", tmp_path / "runB"], tmp_path / "out") + result = json.loads((out / "agreement.json").read_text()) + assert result["penalized"]["cases"] == 1 + assert "dropped" not in result["penalized"]["per_case"] diff --git a/examples/demo/README.md b/examples/demo/README.md index 4a6ff1b..5bbacc8 100644 --- a/examples/demo/README.md +++ b/examples/demo/README.md @@ -16,10 +16,14 @@ planted on purpose: | [4] | a headline claim resting on a **paywalled** reference | ⊘ `not_retrieved` | | — | "Routine imaging archives are among the largest untapped screening resources in medicine." — assertive, **no citation** | flagged in the uncited register | -The remaining claims citing [2] and [3] state published cohort facts +The remaining material citing [2] and [3] states published cohort facts faithfully and should come back ✅ `supported`. (Judgement calls like supported-vs-partial can vary a little between runs — the checker is a -model. The planted contradictions are stable.) +model. The *shape* changed with 0.5.0 rather than with the model: asking +extraction for the verbatim sentence makes the sentence citing both [2] and +[3] one multi-source claim, where 0.4.1 split it into two — so the total is 4 +claims for 5 citation occurrences. Reproduced on both `claude-opus-5` and +`claude-haiku-4-5`. The planted contradictions are stable throughout.) ## Run it @@ -39,10 +43,16 @@ export PAPERTRACE_EMAIL="you@example.org" # use your real address python examples/demo/make_manuscript.py # 4) run the audit — the resolver fetches the open-access references live -papertrace run examples/demo/demo_manuscript.pdf -c demo_case +# --model pinned so this matches the committed output/ — otherwise the +# account default decides the judge, and it changes +papertrace run examples/demo/demo_manuscript.pdf -c demo_case \ + --model claude-opus-5 # then read the results: -open demo_case/out/report.md # or report_editor.html / report_terminal.html +open demo_case/out/report.md + +# want the shareable looks too? ask for them: +# papertrace report -c demo_case -f editor -f terminal ``` No reference PDFs ship with this repo — retrieving them **is** the demo. diff --git a/examples/demo/output/evidence/claim_01_pyrros-2023_p4.png b/examples/demo/output/evidence/claim_01_pyrros-2023_p4.png index 3927514..f73bff1 100644 Binary files a/examples/demo/output/evidence/claim_01_pyrros-2023_p4.png and b/examples/demo/output/evidence/claim_01_pyrros-2023_p4.png differ diff --git a/examples/demo/output/evidence/claim_03_littlejohns-2020_p2.png b/examples/demo/output/evidence/claim_03_littlejohns-2020_p2.png new file mode 100644 index 0000000..680888c Binary files /dev/null and b/examples/demo/output/evidence/claim_03_littlejohns-2020_p2.png differ diff --git a/examples/demo/output/evidence/claim_03_sudlow-2015_p1.png b/examples/demo/output/evidence/claim_03_sudlow-2015_p1.png deleted file mode 100644 index 35871b6..0000000 Binary files a/examples/demo/output/evidence/claim_03_sudlow-2015_p1.png and /dev/null differ diff --git a/examples/demo/output/evidence/claim_03_sudlow-2015_p3.png b/examples/demo/output/evidence/claim_03_sudlow-2015_p3.png new file mode 100644 index 0000000..384cfec Binary files /dev/null and b/examples/demo/output/evidence/claim_03_sudlow-2015_p3.png differ diff --git a/examples/demo/output/evidence/claim_04_littlejohns-2020_p2.png b/examples/demo/output/evidence/claim_04_littlejohns-2020_p2.png deleted file mode 100644 index 028b78a..0000000 Binary files a/examples/demo/output/evidence/claim_04_littlejohns-2020_p2.png and /dev/null differ diff --git a/examples/demo/output/evidence/claim_04_littlejohns-2020_p3.png b/examples/demo/output/evidence/claim_04_littlejohns-2020_p3.png new file mode 100644 index 0000000..b4ad869 Binary files /dev/null and b/examples/demo/output/evidence/claim_04_littlejohns-2020_p3.png differ diff --git a/examples/demo/output/evidence/claim_05_littlejohns-2020_p3.png b/examples/demo/output/evidence/claim_05_littlejohns-2020_p3.png deleted file mode 100644 index d3b5493..0000000 Binary files a/examples/demo/output/evidence/claim_05_littlejohns-2020_p3.png and /dev/null differ diff --git a/examples/demo/output/report.md b/examples/demo/output/report.md index 82a4a1a..dc82268 100644 --- a/examples/demo/output/report.md +++ b/examples/demo/output/report.md @@ -1,85 +1,93 @@ # Fact-Check Report -Checker: `claude -p · claude-opus-5[1m]` · PaperTrace · 2026-08-30 +Checker: `claude -p · claude-opus-5` · PaperTrace · 2026-09-06 Manuscript: `demo_manuscript.pdf` · Sources: `3 / 4` cited references available -**Claims:** 5 | ✅ **Supported:** 2 | ⚠️ **Partial:** 0 | ❌ **Contradicted:** 2 | ⊘ **Not retrieved:** 1 +**Claims:** 4 | ✅ **Supported:** 1 | ⚠️ **Partial:** 0 | ❌ **Contradicted:** 2 | ⊘ **Not retrieved:** 1 **Citation coverage:** 5/5 citation occurrences reached by an extracted claim, across 4 labels — 0 unaddressed, 0 uncertain. Coverage counts places an extracted claim *reached*, not sources that were read. > Ingest `converter: docling 2.118.1` — layout-aware. -> ⚠️ How to read that figure: attribution is a text match that can be wrong — deciding which citation a claim came from is a text comparison, so the counts can be right while a pointer is wrong. An attribution the tool cannot make counts as NOT covered, never as covered — and it refuses close calls, so two similar sentences citing one reference can both read as unaddressed where a reader would pair them at a glance. This figure understates coverage there. A sentence citing the same reference twice needs two extracted claims, so the ratio is not comparable between papers. And detection still reads bracketed numeric markers only — a citation style it cannot see contributes no occurrences at all, which makes this ratio look better than reality, not worse. +> ⚠️ How to read that figure: attribution is the context the extractor named. Extraction is shown every place this paper cites something and returns which of them each claim came from, so the pointer is no longer a text comparison — but naming it is still a model step, and a claim can be placed on the wrong sentence. A claim that names no place at all leaves that reference's remaining places counted as NOT covered, never as covered, so the figure understates coverage there. And detection still reads bracketed numeric markers only — a citation style it cannot see contributes no occurrences at all, which makes this ratio look better than reality, not worse. --- ## Claim 1: "Deep learning on frontal chest radiographs detected type 2 diabetes with an external validation AUC of 0.94." **Status:** ❌ CONTRADICTED -**Location:** Background ¶1 · cites [1] +**Location:** Background · cites [1] + +> Deep learning applied to frontal chest radiographs detected type 2 diabetes with an external validation AUC of 0.94 [1]. +
### ❌ CONTRADICTED — `pyrros-2023` (cited as [1]) -- **Source:** Page 4 `(block_0094)` -> The source does detect T2D from frontal CXRs with deep learning, but its external validation at a separate institution gave a ROC AUC of 0.77, not 0.94 (internal prospective AUC was 0.84). +- **Source:** Page 4 `(block_0050)` +> The source reports external validation ROC AUC of 0.77 (internal prospective AUC 0.84); no AUC of 0.94 appears anywhere. ![evidence](evidence/claim_01_pyrros-2023_p4.png) *red box = matched text* -## Claim 3: "The UK Biobank cohort profile describes recruitment of approximately 500,000 adults aged 40-69 years." +## Claim 3: "UK Biobank recruited approximately 500,000 adults aged 40-69 years, and its imaging enhancement targets 100,000 participants." + +**Status:** ✅ SUPPORTED — *most adverse of 2 cited sources* +**Location:** Population imaging · cites [2, 3] + +> Dedicated cohorts complement such opportunistic reuse: the UK Biobank cohort profile describes recruitment of approximately 500,000 adults aged 40-69 years [2], and its imaging enhancement targets 100,000 participants [3]. + +> **2 cited sources checked for this claim: 2 fully support it.** -**Status:** ✅ SUPPORTED -**Location:** Population imaging ¶1 · cites [2]
### ✅ SUPPORTED — `sudlow-2015` (cited as [2]) -- **Source:** Page 1 `(block_0006)` -> The source states UK Biobank has over 500,000 participants aged 40–69 years recruited in 2006–2010, matching both the size and age range claimed. +- **Source:** Page 3 `(block_0038)` +> Source states over 500,000 participants aged 40-69 recruited 2006-2010 (block_0018/block_0027) and that multimodal imaging is planned in subsets of 100,000 participants (block_0038, Table 3). -![evidence](evidence/claim_03_sudlow-2015_p1.png) +![evidence](evidence/claim_03_sudlow-2015_p3.png) *red box = matched text* -## Claim 4: "The UK Biobank imaging enhancement targets 100,000 participants." - -**Status:** ✅ SUPPORTED -**Location:** Population imaging ¶1 · cites [3]
### ✅ SUPPORTED — `littlejohns-2020` (cited as [3]) - **Source:** Page 2 `(block_0013)` -> The source states the imaging enhancement aims to scan 100,000 of the 500,000 existing UK Biobank participants. +> The source states the imaging enhancement aims to image "100,000 of the existing 500,000 UKB participants", and describes the cohort as half a million participants aged 40-69 recruited 2006-2010. -![evidence](evidence/claim_04_littlejohns-2020_p2.png) +![evidence](evidence/claim_03_littlejohns-2020_p2.png) *red box = matched text* -## Claim 5: "Nearly one in five confirmed participants had not attended an imaging assessment centre." +## Claim 4: "Nearly one in five confirmed UK Biobank participants had not attended an imaging assessment centre." **Status:** ❌ CONTRADICTED -**Location:** Population imaging ¶1 · cites [3] +**Location:** Population imaging · cites [3] + +> Attendance logistics remain a bottleneck, however - nearly one in five confirmed participants had not attended an imaging assessment centre [3]. +
### ❌ CONTRADICTED — `littlejohns-2020` (cited as [3]) -- **Source:** Page 3 `(block_0027)` -> Among confirmed participants the source reports 97% attended and only 3% have not yet attended, not nearly one in five; the 17% figure applies to invited participants who did not wish to attend, not to confirmed ones. +- **Source:** Page 3 `(block_0023)` +> The source reports that of those eligible who booked an appointment, "97% have attended an imaging assessment centre" — i.e., about 3%, not nearly one in five, had not attended. -![evidence](evidence/claim_05_littlejohns-2020_p3.png) -*red box = matched text* +![evidence](evidence/claim_04_littlejohns-2020_p3.png) +*⚠️ no anchor phrase was found on this page — the crop is shown for context and nothing is boxed.* --- -## Not verified — source not retrieved, or check failed (1 of 5) +## Not verified — source not retrieved, or check failed (1 of 4) Either the cited PDF could not be obtained, or the source was available but the check step failed (see each note). Reported as such — never filled in from memory. - **Background** (1): - - [4] Regulators are clearing AI systems for clinical imaging use at an accelerating pace. — *cited source not available (paywalled)* + - ⊘ NOT RETRIEVED · [4] Regulatory clearances of AI systems for clinical imaging are accelerating. — *cited source not available (paywalled)* + ## Assertions without citation (1) — your judgement required Statements that would normally carry a reference but don't. Not verified — flagged for you to weigh. -- **[U1]** Routine imaging archives are among the largest untapped screening resources in medicine. *(Background ¶1)* +- **[U1]** Routine imaging archives are among the largest untapped screening resources in medicine. *(Background)* ## Literature scout — what the reference list doesn't know @@ -92,9 +100,9 @@ proves nothing, and presence is a candidate for your judgement, not an accusatio | # | Reference | Status | Via | Note | |---|-----------|--------|-----|------| -| 1 | Pyrros A, Borstelmann SM, Mantravadi R, et al (2023) Opportunistic detection of … | retrieved | unpaywall | open-access copy via Unpaywall | -| 2 | Sudlow C, Gallacher J, Allen N, et al (2015) UK Biobank: An Open Access Resource… | retrieved | unpaywall | open-access copy via Unpaywall | -| 3 | Littlejohns TJ, Holliday J, Gibson LM, et al (2020) The UK Biobank imaging enhan… | retrieved | unpaywall | open-access copy via Unpaywall | +| 1 | Pyrros A, Borstelmann SM, Mantravadi R, et al (2023) Opportunistic detection of … | retrieved | unpaywall | open-access copy via Unpaywall · title check: 11/11 reference tokens on its first page | +| 2 | Sudlow C, Gallacher J, Allen N, et al (2015) UK Biobank: An Open Access Resource… | retrieved | unpaywall | open-access copy via Unpaywall · title check: 12/12 reference tokens on its first page | +| 3 | Littlejohns TJ, Holliday J, Gibson LM, et al (2020) The UK Biobank imaging enhan… | retrieved | unpaywall | open-access copy via Unpaywall · title check: 12/12 reference tokens on its first page | | 4 | Rajpurkar P, Lungren MP (2023) The Current and Future State of AI Interpretation… | paywalled | — | DOI resolved but no legal open-access copy found | --- diff --git a/examples/demo/output/report_terminal.png b/examples/demo/output/report_terminal.png index 25039a3..91b3974 100644 Binary files a/examples/demo/output/report_terminal.png and b/examples/demo/output/report_terminal.png differ diff --git a/pyproject.toml b/pyproject.toml index 17cfe36..a987dde 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -22,6 +22,14 @@ classifiers = [ dependencies = [ "pymupdf>=1.24", + # NOT an extra any more. Cited sources are judged on their own text, and the + # evidence for a subgroup claim is usually a table row — read flat, the row + # is gone. Spending layout fidelity on the paper but not on the papers it is + # judged against had the asymmetry backwards, so the layout backend is part + # of the tool rather than an upgrade to it. Pulls torch; the ~500 MB layout + # models download on first *use*, not on install, so CI installs this and + # never runs it (every test pins backend="pymupdf"). + "docling>=2.0", "pillow>=10.0", "httpx>=0.27", "rich>=13.0", @@ -31,8 +39,10 @@ dependencies = [ [project.optional-dependencies] png = ["playwright>=1.45"] -docling = ["docling>=2.0"] -full = ["playwright>=1.45", "docling>=2.0"] +# kept as a no-op alias so `pip install papertrace[docling]` and the `[full]` +# extra named all over the 0.4.x docs keep resolving instead of erroring +docling = [] +full = ["playwright>=1.45"] dev = ["pytest>=8.0", "ruff>=0.5", "jsonschema>=4.0", # 3.10 has no tomllib, and without a parser the whole packaging # suite skipped on the oldest version requires-python allows diff --git a/schemas/refs_manifest.schema.json b/schemas/refs_manifest.schema.json index fabbafb..294cecd 100644 --- a/schemas/refs_manifest.schema.json +++ b/schemas/refs_manifest.schema.json @@ -102,6 +102,31 @@ null ], "description": "Whether anyone established that the retrieved or provided file is the paper the reference names. 'unverifiable' (e.g. a scanned first page) is not 'verified' and must never be reported as a match." + }, + "supplements": { + "type": "array", + "description": "Supplementary files the user supplied for this reference, each judged as its own document. Only ever non-empty when the reference itself is available: a supplement with no article behind it is set aside rather than allowed to stand in for the cited source. Absent on manifests written before 0.6.0, where it means none were supplied. Note that no supplement carries a title_check — a supplement's own title does not match its parent's, so the identity check that guards every article cannot apply, and these are attached on a filename match alone.", + "items": { + "type": "object", + "required": [ + "slug", + "pdf_path" + ], + "properties": { + "slug": { + "type": "string", + "description": "The id this document is read, ingested and cropped under: ingest// and sources_resolved/.pdf. Derived from the file stem and never from an ordinal position in the folder, so removing one file cannot silently re-point another document's stored verdicts." + }, + "pdf_path": { + "type": "string", + "description": "Local path to the supplementary PDF, as supplied by the user." + }, + "verified": { + "type": "boolean", + "description": "Did anything establish that this file belongs to the reference it is attached to? True when the file's own title or DOI named that work. False when it was attached because its FILENAME carried the reference's tokens — a guess nobody checked. Absent on manifests written before 0.6.0, meaning false. The report distinguishes the two rather than warning about both equally." + } + } + } } } } @@ -109,6 +134,50 @@ "references_resumed": { "type": "boolean", "description": "True when the reference list was picked up again after an intervening section, so the entry numbering spans a boundary the parser chose to cross. Gated on block type and run length, but still a judgement — the reader should be able to check it." + }, + "reference_source": { + "enum": [ + "crossref", + "parsed" + ], + "description": "Which reading of the reference list these entries came from: the publisher's list deposited with Crossref, or the tool's own parse of the PDF text. Absent on manifests written before reconciliation, which are all parses." + }, + "numbering_verified": { + "type": "boolean", + "description": "True when the chosen list accounts for exactly the labels the manuscript's body cites. The label is the join key between a claim and its source, so an unverified numbering means a verdict may be about a different paper than the label names. Absent on older manifests, which never checked." + }, + "numbering_note": { + "type": "string", + "description": "Why the numbering was or was not confirmed, in the reader's terms — the counts each reading produced and what the body cites." + }, + "unverified_from": { + "type": [ + "integer", + "null" + ], + "minimum": 1, + "description": "The first citation label from which the numbering is in doubt, or null when it is not in doubt. 1 means 'from the very start', which is what a single unchecked reading warrants: one candidate gives no evidence about where it went wrong." + }, + "manuscript_supplements": { + "type": "array", + "description": "Supplementary material belonging to the AUDITED paper, supplied with --supplement. Deliberately not an entry in `entries`: it answers for no citation label, so putting it there would inflate the reference total and let a claim citing a number be handed the paper's own appendix. Absent on manifests written before 0.6.0.", + "items": { + "type": "object", + "required": [ + "slug", + "pdf_path" + ], + "properties": { + "slug": { + "type": "string", + "description": "The id this document is read, ingested and cropped under, derived from the file stem." + }, + "pdf_path": { + "type": "string", + "description": "Local path to the supplementary PDF, as supplied by the user." + } + } + } } } } diff --git a/schemas/results.schema.json b/schemas/results.schema.json index c65a67d..96ef229 100644 --- a/schemas/results.schema.json +++ b/schemas/results.schema.json @@ -37,7 +37,17 @@ "type": "integer" }, "claim": { - "type": "string" + "type": "string", + "description": "The claim tightly paraphrased \u2014 what a report headline reads well." + }, + "ctx_ids": { + "type": "array", + "description": "Ids of the citation occurrences this claim was extracted from, resolved from the `ctx_NNNN` labels the inventory offered the extractor. Empty when the model named none or named one not in the inventory \u2014 never back-filled by guessing which occurrence of the label was meant. Absent before 0.5.0.", + "items": { "type": "string" } + }, + "quote": { + "type": "string", + "description": "The manuscript's own sentence carrying the claim, verbatim. This is what the judge rules on: a paraphrase can drop the population, the interval or the hedging that decides whether a source supports the statement. Empty string when extraction returned no quote \u2014 never back-filled from `claim`, and absent entirely in files written before 0.5.0." }, "location": { "type": "string" @@ -48,6 +58,10 @@ "type": "string" } }, + "own_supplement": { + "type": "boolean", + "description": "True when the claim points at the AUDITED paper's own supplementary material — \"Table S3\", \"eFigure 2\", \"Supplementary Methods\". A pointer, not a citation, so it never appears in `refs`; a claim can carry both. Such a claim is judged against every supplement supplied with --supplement, and is `not_retrieved` when none was — the paper said where its evidence is and nobody opened it, which is a retrieval gap rather than an assertion made without a citation. Absent on files written before 0.6.0, meaning false." + }, "verdict": { "enum": [ "supported", @@ -102,7 +116,15 @@ "source_slug": { "type": "string" }, "ref": { "type": "string", - "description": "The citation label this source answers for, e.g. \"3\"." + "description": "The citation label this source answers for, e.g. \"3\". A supplement answers for the label its parent article carries; the audited paper's own supplement answers for none and carries \"\"." + }, + "verified": { + "type": "boolean", + "description": "For a supplement: did anything establish that it belongs to the work it was attached to? True when the file's own title or DOI named that work; false when it was attached because its filename carried the reference's tokens and nothing read it. Always false for an article, whose identity is recorded in the manifest's title_check instead. Absent on files written before 0.6.0, meaning false." + }, + "kind": { + "enum": ["article", "supplement", "own_supplement"], + "description": "Which document this verdict was read from: the cited article, supplementary material accompanying it, or supplementary material belonging to the audited paper itself. Absent on files written before 0.6.0, where every judgement was an article. A reader holding only results.json needs this to tell a verdict resting on an appendix from one resting on the paper of record — and note that a supplement is attached by filename with NO identity check, so its provenance is weaker than an article's." }, "verdict": { "enum": [ @@ -141,7 +163,15 @@ } }, "converter": { - "type": "string" + "type": "string", + "description": "The ingest backend that read the AUDITED PAPER. Says nothing about the cited sources \u2014 see source_converters." + }, + "source_converters": { + "type": "object", + "description": "Document slug -> the converter that read it. Keyed by document and not by reference, so a supplement appears under its own slug beside the article it accompanies. Sources are ingested with the same backend as the paper, but a case folder can hold a map from an earlier run whose PDF is now gone, so the two can differ. An empty object or an absent key means the run did not record this (every file written before 0.5.0) \u2014 which is NOT the same as every source having been read flat.", + "additionalProperties": { + "type": "string" + } }, "uncited": { "type": "array", @@ -158,6 +188,10 @@ "claim": { "type": "string" }, + "quote": { + "type": "string", + "description": "The manuscript's own sentence, verbatim \u2014 the wording the reviewer has to judge. Empty when extraction returned no quote; absent before 0.5.0." + }, "location": { "type": "string" } @@ -190,8 +224,8 @@ }, "schema": { "type": "string", - "description": "'coverage/2' when occurrence-level keys are present. Absent means the label-level audit that shipped before them.", - "enum": ["coverage/2"] + "description": "'coverage/3' since 0.5.0: attribution is a lookup of the context ids extraction returned. 'coverage/2' was the same occurrence keys with similarity-based attribution and its two thresholds. Absent means the label-level audit that shipped before either. Both occurrence versions are accepted so an older results.json still validates.", + "enum": ["coverage/2", "coverage/3"] }, "unit": { "type": "string", @@ -248,14 +282,12 @@ }, "attribution": { "type": "object", - "description": "How claims were matched to occurrences. Reading-order zipping is deliberately not used: the order is unverified and degrades silently on one skipped claim.", + "description": "How claims were placed on occurrences. Since coverage/3 this is a lookup, not a match: extraction is shown the occurrence inventory as `ctx_NNNN` and returns the ids it used, so there are no similarity thresholds to report. Reading-order zipping remains deliberately unused \u2014 the order is unverified and degrades silently on one skipped claim.", "properties": { "method": { "type": "string" }, - "min_ratio": { "type": "number" }, - "min_margin": { "type": "number" }, "claims_unattributed": { "type": "array", - "description": "Claims citing a label that reached no specific occurrence — surplus claims, and claims the attributor refused to place.", + "description": "Claims citing a label but naming none of that label's own contexts \u2014 a surplus claim, an empty `ctx`, or a `ctx` that was not in the inventory. Their label's unreached occurrences are `uncertain`, never `uncovered`.", "items": { "type": "object", "properties": { diff --git a/schemas/scout.schema.json b/schemas/scout.schema.json index cbea0db..8b919b1 100644 --- a/schemas/scout.schema.json +++ b/schemas/scout.schema.json @@ -3,43 +3,114 @@ "title": "scout", "description": "Post-publication literature scan around one paper. Search-based and incomplete by construction: absence from the registers proves nothing; a non-empty error means the scan soft-failed.", "type": "object", - "required": ["paper", "newer", "overlooked"], + "required": [ + "paper", + "newer", + "overlooked" + ], "properties": { "paper": { "type": "object", "properties": { - "title": { "type": "string" }, - "doi": { "type": "string" }, - "year": { "type": ["integer", "null"] }, - "resolved_via": { "type": "string", "enum": ["doi", "title", ""] } + "title": { + "type": "string" + }, + "doi": { + "type": "string" + }, + "year": { + "type": [ + "integer", + "null" + ] + }, + "identity": { + "type": "string", + "description": "did anyone establish that the record found is this paper? Compared on the record's title and the paper's own. `resolved_via` cannot answer it: a DOI read off page 1 also resolves `via: doi`. Absent on scout.json written before the check; absent and \"\" mean not recorded, never confirmed.", + "enum": [ + "confirmed", + "unverified", + "mismatch", + "" + ] + }, + "resolved_via": { + "type": "string", + "enum": [ + "doi", + "title", + "" + ] + } } }, - "query": { "type": "string" }, - "date": { "type": "string" }, + "query": { + "type": "string" + }, + "date": { + "type": "string" + }, "counts": { "type": "object", "properties": { - "newer": { "type": "integer" }, - "overlooked": { "type": "integer" } + "newer": { + "type": "integer" + }, + "overlooked": { + "type": "integer" + }, + "same_year": { + "type": "integer" + } } }, - "newer": { "$ref": "#/$defs/hits" }, - "overlooked": { "$ref": "#/$defs/hits" }, - "error": { "type": "string" } + "newer": { + "$ref": "#/$defs/hits" + }, + "overlooked": { + "$ref": "#/$defs/hits" + }, + "error": { + "type": "string" + }, + "same_year": { + "$ref": "#/$defs/hits" + } }, "$defs": { "hits": { "type": "array", "items": { "type": "object", - "required": ["title"], + "required": [ + "title" + ], "properties": { - "title": { "type": "string" }, - "year": { "type": ["integer", "null"] }, - "doi": { "type": "string" }, - "via": { "type": "string", "enum": ["citing", "search"] }, - "journal": { "type": "string" }, - "authors": { "type": "string" } + "title": { + "type": "string" + }, + "year": { + "type": [ + "integer", + "null" + ] + }, + "doi": { + "type": "string" + }, + "via": { + "type": "string", + "enum": [ + "citing", + "search" + ] + }, + "journal": { + "type": "string" + }, + "authors": { + "type": "string" + } } } } diff --git a/schemas/source_map.schema.json b/schemas/source_map.schema.json index b263a6e..601b356 100644 --- a/schemas/source_map.schema.json +++ b/schemas/source_map.schema.json @@ -68,6 +68,17 @@ "converter": { "type": "string", "description": "ingest backend that produced this map, e.g. 'pymupdf' or 'docling 2.x'" + }, + "declared_title": { + "type": "string", + "description": "the title the PDF declares in its own metadata (XMP / Info dictionary), verbatim and unjudged. Absent on maps written before it was recorded; empty means the file declares none, never that the paper has none. Whether it is title-shaped is decided by models.paper_title, not here." + }, + "source_sha256": { + "type": [ + "string", + "null" + ], + "description": "sha256 of the PDF bytes this map was built from. `doc` cannot serve as an identity: a cited source is stored as .pdf, so every source map in a case names a different paper the same way. Absent on maps written before content hashing; absent means unknown, never 'matches'." } } -} \ No newline at end of file +} diff --git a/scripts/reference_audit.py b/scripts/reference_audit.py new file mode 100644 index 0000000..380540d --- /dev/null +++ b/scripts/reference_audit.py @@ -0,0 +1,159 @@ +#!/usr/bin/env python3 +"""Measure the three readings of a paper's reference list, side by side. + +Read-only and free: it ingests each PDF, counts what the body cites, parses the +printed list, asks Crossref what the publisher deposited, and prints the three +numbers with the verdict the reconciler would reach. No model calls, no +downloads, no case folder written. + +This exists because the reconciliation rules are a design guess until they meet +a spread of journals. Two audited papers agreeing is not evidence that the rule +generalises — it is two data points from one publisher. + + python scripts/reference_audit.py ~/papers/*.pdf + python scripts/reference_audit.py --email you@example.org paper.pdf + +A row reading `parse 43 · crossref 41 · body [1]-[41] → crossref` is the failure +this was built for: the parse invented two references and the deposit caught it. +""" + +from __future__ import annotations + +import argparse +import os +import sys +import tempfile +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from papertrace.ingest import ingest_pdf, references_span # noqa: E402 +from papertrace.models import ( # noqa: E402 + citation_labels, + is_references_heading, + paper_title, +) +from papertrace.refs import ( # noqa: E402 + _client, + crossref_deposit, + deposit_corroborates, + deposit_is_this_paper, + parse_references, + reconcile, +) +from papertrace.wizard import detect_doi # noqa: E402 + + +def _body_labels(smap) -> set[str]: + body: list[str] = [] + for b in smap.blocks: + if is_references_heading(b.type, b.text): + break + body.append(b.text) + return citation_labels("\n".join(body)) + + +def _span(labels: set[str]) -> str: + if not labels: + return "none" + nums = sorted(int(x) for x in labels) + gaps = [n for n in range(1, nums[-1] + 1) if n not in set(nums)] + return f"[1]-[{nums[-1]}]" + (f" ({len(gaps)} gaps)" if gaps else "") + + +def audit(pdf: Path, email: str, backend: str) -> dict: + row: dict = {"pdf": pdf.name} + with tempfile.TemporaryDirectory() as scratch: + smap = ingest_pdf(pdf, Path(scratch), backend=backend) + row["converter"] = smap.converter + body = _body_labels(smap) + row["body"] = _span(body) + + refs_text, resumed = references_span(smap) + parsed = parse_references(refs_text) + row["parsed"] = len(parsed) + row["resumed"] = resumed + + doi = detect_doi(pdf) + row["doi"] = doi or "—" + with _client() as client: + deposit = crossref_deposit(client, doi, email) + row["publisher"] = deposit.publisher or "—" + row["crossref"] = len(deposit.entries) if deposit.entries else None + row["deposited"] = deposit.deposited + row["unrenderable"] = deposit.unrenderable + row["absent"] = deposit.absent + + # the same gate the `refs` command applies — a diagnostic that measures a + # different pipeline than the tool is measuring nothing + identity = ( + deposit_is_this_paper(paper_title(smap), deposit.title) if deposit.entries else None + ) + corroboration = ( + deposit_corroborates(deposit.entries, parsed) + if identity is None and deposit.entries else None + ) + row["identity"] = ( + "confirmed" if identity + else "WRONG PAPER" if identity is False + else f"confirmed by bibliography ({corroboration.found}/{corroboration.total})" + if corroboration and corroboration.confirms + else "unverified" + ) + usable = not deposit.unrenderable and identity is not False + candidate = (deposit.entries or None) if usable else None + absent = deposit.absent or ( + f"the DOI used ({doi}) belongs to a record titled \u201c{deposit.title}\u201d, " + "which is not this paper" if identity is False else "" + ) + _entries, rec = reconcile(body, candidate, parsed, crossref_absent=absent) + row["chosen"] = rec.source + row["verified"] = rec.verified + row["note"] = rec.note + return row + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("pdfs", nargs="+", type=Path) + ap.add_argument("--email", default=os.environ.get("PAPERTRACE_EMAIL", ""), + help="contact address for Crossref's polite pool") + ap.add_argument("--backend", default="auto", help="auto | docling | pymupdf") + args = ap.parse_args() + + rows = [] + for pdf in args.pdfs: + if not pdf.exists(): + print(f" ? {pdf} — not found") + continue + try: + row = audit(pdf, args.email, args.backend) + except Exception as e: # noqa: BLE001 — a diagnostic reports, it does not stop + print(f" ! {pdf.name} — {type(e).__name__}: {e}") + continue + rows.append(row) + mark = "OK " if row["verified"] else " ⚠" + cr = row["crossref"] if row["crossref"] is not None else "—" + partial = f" ({row['unrenderable']} unreadable of {row['deposited']})" if row["unrenderable"] else "" + print( + f"{mark} {row['pdf'][:44]:<44} " + f"parse {row['parsed']:>3} · crossref {str(cr):>3}{partial} · " + f"body {row['body']:<16} → {row['chosen']}" + ) + print(f" {row['publisher'][:40]:<40} {row['converter']}" + f" · doi identity {row['identity']}" + f"{' · resumed' if row['resumed'] else ''}") + if not row["verified"]: + print(f" {row['note']}") + + if rows: + ok = sum(1 for r in rows if r["verified"]) + deposits = sum(1 for r in rows if r["crossref"] is not None) + print(f"\n{ok}/{len(rows)} numberings confirmed · " + f"{deposits}/{len(rows)} publishers deposited a reference list") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/papertrace/__init__.py b/src/papertrace/__init__.py index e4cdfd1..9510225 100644 --- a/src/papertrace/__init__.py +++ b/src/papertrace/__init__.py @@ -1,3 +1,3 @@ """PaperTrace — trace a paper’s claims to their sources, then scout what came after.""" -__version__ = "0.4.0" +__version__ = "0.6.0" diff --git a/src/papertrace/check.py b/src/papertrace/check.py index a1df054..1da6ba8 100644 --- a/src/papertrace/check.py +++ b/src/papertrace/check.py @@ -13,18 +13,20 @@ import re import shutil import subprocess -import unicodedata +import tempfile from dataclasses import dataclass, field -from difflib import SequenceMatcher from pathlib import Path from .models import ( + _LABEL_GROUP, JUDGMENT_VERDICTS, PIPELINE_STATES, ClaimResult, RefManifest, SourceJudgement, UncitedClaim, + _expand_label_group, + citation_labels, is_references_heading, ) @@ -48,6 +50,11 @@ def last_model() -> str | None: # A cut is recorded and disclosed in the report rather than passing silently. MANUSCRIPT_CHAR_LIMIT = 180_000 SOURCE_CHAR_LIMIT = 150_000 +# the citation inventory sent with the extraction prompt. A cut here is not +# cosmetic: contexts past it are never offered to the model, so nothing can be +# attributed to them and they can only ever come back `uncovered`. Disclosed +# like every other cut rather than passing quietly. +CONTEXT_CHAR_LIMIT = 60_000 @dataclass class Truncations: @@ -81,11 +88,21 @@ def _clip(text: str, limit: int, what: str, truncations: Truncations | None) -> Below is a manuscript converted to markdown with provenance markers (``). -Task 1 — CITED claims: extract EVERY claim that carries a citation marker. -Completeness over selectivity: each bracketed label like [3] or [7,8] or [9-11] -that supports a statement must appear in at least one extracted claim. This -includes numerical results, "X showed Y", methodological attributions, -guideline statements, prevalence claims — and claims made inside TABLES. +CITATION CONTEXTS below is the complete list of places this manuscript makes a +citation, found mechanically. Each line is `ctx_NNNN`, its page, its section, +the label(s) cited there, and the sentence. + +Task 1 — CITED claims: extract EVERY claim that carries a citation marker, +working through the CITATION CONTEXTS list. Completeness over selectivity: +each context must appear in at least one extracted claim. This includes +numerical results, "X showed Y", methodological attributions, guideline +statements, prevalence claims — and claims made inside TABLES. + +Every cited claim carries `ctx`: the ids of the contexts it was taken from, +copied exactly from the list. One sentence citing [2] and [3] is ONE claim +carrying BOTH ids, not two claims. Never invent an id and never guess: if you +cannot tell which context a claim came from, return `"ctx": []` and it will be +reported as unplaced rather than attributed to the wrong sentence. Task 2 — UNCITED assertions: list assertive factual statements that carry NO citation but would normally need one (numbers, prevalence, mechanisms, @@ -93,14 +110,29 @@ def _clip(text: str, limit: int, what: str, truncations: Truncations | None) -> methods descriptions of what the authors themselves did. Rules for both: -- claim: the statement, tightly paraphrased, ≤160 chars. +- quote: the manuscript's own sentence carrying the claim, copied VERBATIM, + ≤500 chars. Copy it exactly as written — do not tidy, shorten or rephrase it, + and keep the numbers, units, intervals and hedging words as they appear. If + the claim spans two sentences, quote both. Strip nothing except the citation + marker itself. This is the text that will be checked against the source. +- claim: the same statement tightly paraphrased for a headline, ≤300 chars. - location: manuscript section (e.g. "Introduction ¶2", "Methods", "Table 2"). - cited claims also carry refs: citation labels as strings, e.g. ["3"] or ["7","8"]. +- own_supplement: true when the claim points at THIS paper's own supplementary + material — "Table S3", "eFigure 2", "Supplementary Methods", "Appendix A". + That is a pointer, not a citation, so it does NOT go in refs. A claim can + carry both: "as in [4] and Table S2" cites [4] and sets own_supplement. + Reserve it for the paper's own numbering; "the supplement of [4]" is just [4]. + A claim whose ONLY support is such a pointer still belongs in `cited`, with + an empty refs list — it is not an assertion made without evidence. - Number each list from 1 in reading order. Answer with ONLY a JSON object, no prose, no code fences: -{"cited":[{"id":1,"claim":"...","location":"...","refs":["1"]}], - "uncited":[{"id":1,"claim":"...","location":"..."}]} +{"cited":[{"id":1,"ctx":["ctx_0001"],"quote":"...","claim":"...","location":"...","refs":["1"],"own_supplement":false}], + "uncited":[{"id":1,"quote":"...","claim":"...","location":"..."}]} + +CITATION CONTEXTS: +<> MANUSCRIPT: """ @@ -115,6 +147,16 @@ def _clip(text: str, limit: int, what: str, truncations: Truncations | None) -> THIS source does or does not say, and do not speculate about the others: each is judged in its own call and the results are combined afterwards. +WHAT YOU ARE HOLDING: <> + +Each claim carries `quote`, the manuscript's own sentence, and `claim`, a short +paraphrase of it. **Judge the quote.** It holds the population, the effect +size, the interval and the hedging that decide whether the source supports the +statement; the paraphrase is a label and may have dropped any of them. Where +the two differ, the quote is the claim. A claim with an empty `quote` is all +there is for it — judge the paraphrase, and let the missing scope count against +"supported" rather than for it. + For each claim output: - verdict: "supported" (source states it), "partial" (kernel true but scope, strength or object differs — say what differs), "contradicted" (source says @@ -153,13 +195,33 @@ def claude_available() -> bool: return shutil.which("claude") is not None +_SCRATCH_CWD: str | None = None + + +def _scratch_cwd() -> str: + # /tmp itself is shared and world-writable; a private 0700 directory (one per + # process, reused across calls) keeps another local user from planting + # anything the judging call would walk into + global _SCRATCH_CWD + if _SCRATCH_CWD is None: + _SCRATCH_CWD = tempfile.mkdtemp(prefix="papertrace-ask-") + return _SCRATCH_CWD + + def _ask(prompt: str, model: str | None = None) -> str: - cmd = ["claude", "-p", "--output-format", "json"] + # judging happens wherever the user ran papertrace from — never that repo's own + # CLAUDE.md, and never with more than the ability to read the prompt and answer + cmd = ["claude", "-p", "--output-format", "json", "--safe-mode", "--tools", ""] if model: cmd += ["--model", model] try: proc = subprocess.run( - cmd, input=prompt, capture_output=True, text=True, timeout=CLAUDE_TIMEOUT + cmd, + input=prompt, + capture_output=True, + text=True, + timeout=CLAUDE_TIMEOUT, + cwd=_scratch_cwd(), ) except subprocess.TimeoutExpired: raise RuntimeError(f"claude -p timed out after {CLAUDE_TIMEOUT}s") from None @@ -192,6 +254,46 @@ def _parse_json_object(text: str) -> dict: return json.loads(text[start : end + 1]) +def _ctx_labels(claim: dict) -> list[str]: + """The `ctx` values one returned claim carries, however it phrased them. + + A list is the contract, but a model that has exactly one context sometimes + sends the bare string. Accepting both costs one branch; rejecting the + string form would discard a correct answer over its punctuation. + """ + raw = claim.get("ctx") + if isinstance(raw, str): + return [raw] + if isinstance(raw, list): + return [str(x) for x in raw] + return [] + + +def _render_inventory(occurrences: list[dict]) -> tuple[str, dict[str, str]]: + """The citation inventory as the model sees it, and the map back. + + Returns the `ctx_NNNN` block for the prompt and `{ctx label -> occurrence + id}`. Both come out of **one pass** over the same list, on purpose: the + caller resolves the model's answer through this map rather than re-deriving + the pairing from position later. Re-deriving it would be reading-order + zipping wearing a different hat — one dropped occurrence and every id after + it points at the wrong sentence, silently. + + Internal occurrence ids (`block_0012:345:7`) are deliberately not shown. + They are long, punctuated, and a model asked to copy one exactly will + sometimes not; `ctx_0007` it copies. + """ + lines: list[str] = [] + mapping: dict[str, str] = {} + for n, o in enumerate(occurrences, start=1): + ctx = f"ctx_{n:04d}" + mapping[ctx] = o["id"] + where = f"p{o['page']}" if o.get("page") else "p?" + section = f" §{o['section']}" if o.get("section") else "" + lines.append(f"{ctx} {where}{section} [{o.get('group') or o['label']}] {o['sentence']}") + return "\n".join(lines), mapping + + def extract_claims( case_dir: Path, model: str | None = None, @@ -200,14 +302,33 @@ def extract_claims( ) -> tuple[list[ClaimResult], list[UncitedClaim]]: annotated = case_dir / "ingest" / "manuscript" / "annotated.md" text = _clip(annotated.read_text(), MANUSCRIPT_CHAR_LIMIT, "manuscript", truncations) - raw = _ask(EXTRACT_PROMPT + text, model) + # the inventory is built BEFORE the call, from source_map.json, and handed + # to the model — rather than built afterwards and matched back against a + # paraphrase. This is the whole redesign. + occurrences, _source = citation_occurrences(case_dir) + inventory, ctx_map = _render_inventory(occurrences) + prompt = EXTRACT_PROMPT.replace( + "<>", + _clip(inventory or "(none found)", CONTEXT_CHAR_LIMIT, "citation contexts", truncations), + ) + raw = _ask(prompt + text, model) data = _parse_json_object(raw) cited = [ ClaimResult( id=int(c["id"]), claim=str(c["claim"]), + # `.get`, and NOT falling back to `claim`: a missing quote means the + # model did not give one, and copying the paraphrase in would put + # the compression back while looking like it had been removed + quote=str(c.get("quote", "")), location=str(c.get("location", "")), + # resolved here, where the map is in hand. An id that is not in the + # inventory is DROPPED, never repaired into "the first occurrence of + # that label" — coverage reports the claim as unplaced instead, and + # the label's occurrences as uncertain. + ctx_ids=[ctx_map[k] for k in _ctx_labels(c) if k in ctx_map], refs=[str(r) for r in c.get("refs", [])], + own_supplement=bool(c.get("own_supplement", False)), ) for c in data.get("cited", []) ] @@ -215,6 +336,7 @@ def extract_claims( UncitedClaim( id=int(u["id"]), claim=str(u["claim"]), + quote=str(u.get("quote", "")), location=str(u.get("location", "")), ) for u in data.get("uncited", []) @@ -226,31 +348,60 @@ def extract_claims( # deterministic citation-label coverage audit # --------------------------------------------------------------------------- -_LABEL_GROUP = re.compile(r"\[(\d{1,3}(?:\s*[,\u2013\u2014-]\s*\d{1,3})*)\]") -_REFS_HEADING = re.compile(r"^##\s+(references|bibliography|literature)\b", re.I | re.M) +def _body_before_references(clean_md: str) -> str: + """`clean_md` up to the line where the bibliography begins. + + Uses `models.is_references_heading` — the rule `refs` and the occurrence + walk already share — instead of a second regex of its own. That regex + required a markdown `##`, which needs ingest to have *typed* the block as a + heading; flat-text ingest guesses headings from font size, so a `References` + line at body size reaches `clean.md` as plain text. The shared rule is built + for exactly that case and says True where this cut said False, and the two + disagreeing is how every `[N]` printed in the reference list came to be + counted as a body citation — reporting gaps that do not exist, in the one + figure the audit computes mechanically so that it cannot. + + A markdown-marked line is read as a heading, so `## References and further + reading` cuts. A plain line must be the word and nothing else, which is what + keeps `References were checked by hand [1].` from swallowing the paper. + """ + out: list[str] = [] + for line in clean_md.splitlines(keepends=True): + marked = line.lstrip().startswith("#") + if is_references_heading("sectionheader" if marked else "text", line): + break + out.append(line) + return "".join(out) + + +_TABLE_ROW = re.compile(r"^\s*\|.*\|\s*$") -def _expand_label_group(group: str) -> set[str]: - labels: set[str] = set() - for part in re.split(r"\s*,\s*", group): - m = re.match(r"^(\d{1,3})\s*[\u2013\u2014-]\s*(\d{1,3})$", part.strip()) - if m: - lo, hi = int(m.group(1)), int(m.group(2)) - if lo <= hi and hi - lo <= 50: - labels.update(str(n) for n in range(lo, hi + 1)) - elif part.strip().isdigit(): - labels.add(part.strip()) - return labels +def _strip_table_rows(text: str) -> str: + """Blank out GitHub-flavoured-markdown table rows. + + `clean.md` is flat text with no block-type information, so a table can only + be recognised by its own linearized shape (`| ... |`, one row per line \u2014 see + `Block` in `models.py`). A results table's own numbers are not citations: a + 95% CI column like `[51, 77]` matches the same bracket-and-comma syntax as a + citation group `[7,8]`, and a live audit read a table's CI columns as + citations to references that did not exist at that number. + """ + return "\n".join("" if _TABLE_ROW.match(line) else line for line in text.splitlines()) def citation_labels_in_text(clean_md: str) -> set[str]: - """Every citation label appearing in the body text (References section excluded).""" - cut = _REFS_HEADING.search(clean_md) - body = clean_md[: cut.start()] if cut else clean_md - labels: set[str] = set() - for m in _LABEL_GROUP.finditer(body): - labels.update(_expand_label_group(m.group(1))) - return labels + """Every citation label appearing in the body text (References section excluded). + + The label rule itself lives in `models.citation_labels` \u2014 `refs.py` needs the + same reading to reconcile the reference list against what the manuscript + cites, and two independent readings of one fact are only evidence when they + come from one rule. This wrapper owns the one thing that is local to + coverage: stopping at the bibliography, so its own `[N]` markers are not + counted as body citations, and skipping table rows, whose own numbers are not + citations either. + """ + return citation_labels(_strip_table_rows(_body_before_references(clean_md))) # --------------------------------------------------------------------------- @@ -258,14 +409,20 @@ def citation_labels_in_text(clean_md: str) -> set[str]: # # Label-level coverage was pure set arithmetic — mechanical and incapable of a # false positive, but blind: two sentences citing [3] with one extracted claim -# reported [3] as covered and left the other sentence invisible. Occurrences -# fix the blindness and buy a new failure mode with it (the *counts* stay -# right; the *pointer* can be wrong), which is why `uncertain` is a third -# status and why the report carries a self-caveat. +# reported [3] as covered and left the other sentence invisible. Occurrences fix +# the blindness. +# +# Occurrences first bought a new failure mode with it: the inventory was built +# AFTER the model call, so a paraphrase had to be matched back to a sentence by +# text similarity, and the *pointer* could be wrong while the counts were right. +# That is gone. The inventory is now built first and handed to the extractor as +# `ctx_NNNN`, and attribution is a lookup of the ids it returned. +# +# `uncertain` stays, for the one case that can still produce doubt: a claim +# cites a label and names none of that label's contexts, so a claim did reach +# one of them and nothing can say which. # --------------------------------------------------------------------------- -OCCURRENCE_MIN_RATIO = 0.45 -OCCURRENCE_MIN_MARGIN = 0.10 _EXCERPT_RADIUS = 120 # structural, not textual: the source map says a block IS a section header, so @@ -273,37 +430,14 @@ def citation_labels_in_text(clean_md: str) -> set[str]: _SENTENCE_END = re.compile(r"(?<=[.!?])\s+") -def _normalize_for_match(text: str) -> str: - """Fold the differences that are never semantic, for attribution only. - - Deliberately *similar* to `evals/align.py::normalize_claim`, not identical — - do not collapse the two. This one removes quotes outright and strips - bracketed citation markers, because it compares a claim against manuscript - sentences that carry `[3]` the claim text never had; `normalize_claim` - converts quotes to ASCII and keeps the markers, because it compares two - claim texts where a marker is signal. - - They stay separate for a second reason: papertrace cannot import `evals` - (it is not in the wheel), and `evals` must not import a matcher from the - very thing it grades. Ten lines is the right price. - """ - t = unicodedata.normalize("NFKC", text or "") - t = t.translate(dict.fromkeys(map(ord, "‐‑‒–—―"), "-")) - t = re.sub(r"[*_`\"'‘’“”]", "", t).casefold() - t = re.sub(r"\[\d[\d\s,–—-]*\]", " ", t) # the markers themselves carry no meaning - return re.sub(r"\s+", " ", t).strip(" .") - - -def _ratio(a: str, b: str) -> float: - return SequenceMatcher(None, a, b).ratio() - - def _excerpt(text: str, start: int, end: int) -> str: """The sentence carrying the marker, clipped to ±120 chars around it. The excerpt is what turns an uncovered occurrence from a bare number into - something a reader can act on, and it is also what the attributor matches - against — so it is the sentence, not an arbitrary window. + something a reader can act on, and it is also what the extractor is shown + in the `ctx_NNNN` inventory — so it is the sentence, not an arbitrary + window. A window that cut mid-clause would ask the model to place a claim + against half of the sentence it came from. """ begin = 0 for m in _SENTENCE_END.finditer(text, 0, start): @@ -362,6 +496,9 @@ def citation_occurrences(case_dir: Path) -> tuple[list[dict], str]: # the SAME rule `references_section` uses — see models.is_references_heading if is_references_heading(b.type, b.text or ""): break + if b.type == "table": + # a table's own numbers are not citations — see _strip_table_rows + continue out += _occurrences_in( b.text or "", block=b.id, @@ -372,88 +509,40 @@ def citation_occurrences(case_dir: Path) -> tuple[list[dict], str]: clean = manuscript / "clean.md" if clean.exists(): - text = clean.read_text() - cut = _REFS_HEADING.search(text) - body = text[: cut.start()] if cut else text + # the same cut as the label reading above: a fallback that counted + # reference-list markers as occurrences would inflate the denominator of + # the coverage ratio, not just the label set + body = _strip_table_rows(_body_before_references(clean.read_text())) return _occurrences_in(body, block=None, page=None, section=""), "clean.md" return [], "none" -def _location_matches(location: str, section: str) -> bool: - """Does a claim's free-text `location` ("Methods §2") name this section?""" - loc, sec = _normalize_for_match(location), _normalize_for_match(section) - if not loc or not sec: - return False - return sec in loc or loc in sec - - -def _attribute_label(occs: list[dict], claims: list[ClaimResult]) -> tuple[dict, list[int], bool]: - """Which occurrence of ONE label each claim citing it reached. - - Returns (occurrence id -> claim id, claim ids attributed to nothing, - whether any attribution was refused on the margin). - - One occurrence is the whole answer: a claim citing the label reached the - only place the label appears. With several, the claim's `location` narrows - the field and text similarity decides, assigned globally best-first. The - extraction prompt returns a tight ≤160-char paraphrase, so the absolute - ratio is weak evidence — **the margin is the decisive test**, since the - question is only *which* occurrence. - - Reading-order zipping (claim 1 → occurrence 1, and so on) is deliberately - NOT used. `EXTRACT_PROMPT` does ask for reading order, which makes it - tempting, but the order is unverified and degrades silently on a single - skipped claim: every later pairing shifts by one and the audit manufactures - confident, wrong attributions. Refusing to answer is the honest failure. - """ - if not occs: - return {}, [], False - if len(occs) == 1: - if claims: - return {occs[0]["id"]: claims[0].id}, [c.id for c in claims[1:]], False - return {}, [], False - - edges: list[tuple[float, int, str]] = [] - for c in claims: - narrowed = [o for o in occs if _location_matches(c.location, o["section"])] or occs - want = _normalize_for_match(c.claim) - for o in narrowed: - edges.append((_ratio(want, _normalize_for_match(o["sentence"])), c.id, o["id"])) - edges.sort(key=lambda t: (-t[0], t[1], t[2])) - - assigned: dict[str, int] = {} - taken: set[int] = set() - refused = False - for score, cid, oid in edges: - if cid in taken or oid in assigned or score < OCCURRENCE_MIN_RATIO: - continue - # the competitor is the best LIVE candidate sharing either endpoint — - # a row rival (this claim, another occurrence) or a column rival - # (another claim, this occurrence). Both are coin flips. - rivals = [ - t for t in edges - if (t[1] == cid) != (t[2] == oid) and t[1] not in taken and t[2] not in assigned - ] - if score - max((t[0] for t in rivals), default=0.0) >= OCCURRENCE_MIN_MARGIN: - assigned[oid] = cid - taken.add(cid) - else: - refused = True - return assigned, [c.id for c in claims if c.id not in taken], refused - - def attribute_occurrences( occurrences: list[dict], claims: list[ClaimResult] ) -> tuple[list[dict], list[dict]]: """Stamp a `status` and a `claim_id` on every occurrence. - `uncertain` is a third status and is NEVER counted as covered: it means a - claim did reach this label and the tool cannot say which sentence it came - from. Calling that covered restores the overstatement; calling it - uncovered cries wolf. Surplus claims (more claims cite the label than there - are places citing it) are recorded and cast no doubt on anything — every - occurrence is already attributed, so nothing is left to be uncertain about. + Bookkeeping, not matching: an occurrence is `covered` when some claim's + `ctx_ids` names it. The extractor was shown this exact inventory and told + which ids to copy, so nothing here has to work out where a paraphrase came + from. + + `uncertain` survives, for the one case that can still produce doubt: a + claim cites the label but named no usable context for it, so a claim *did* + reach one of these places and nothing can say which. Calling those covered + restores the overstatement occurrences exist to remove; calling them + uncovered cries wolf about a citation that was in fact read. It is never + counted as covered. + + A claim naming a context that is not in the inventory lands here too, and + on purpose — a hallucinated `ctx_9999` and an honest `"ctx": []` are the + same amount of information about which sentence was meant. """ + reached: dict[str, int] = {} + for c in claims: + for cid in c.ctx_ids: + reached.setdefault(cid, c.id) + by_label: dict[str, list[dict]] = {} for o in occurrences: by_label.setdefault(o["label"], []).append(o) @@ -461,13 +550,16 @@ def attribute_occurrences( status: dict[str, tuple[str, int | None]] = {} unattributed: list[dict] = [] for label, occs in by_label.items(): - citing = [c for c in claims if label in c.refs] - assigned, orphans, refused = _attribute_label(occs, citing) - rest = "uncertain" if (refused or orphans) else "uncovered" + here = {o["id"] for o in occs} + # a claim is unplaced *for this label* when it cites the label and + # named none of the label's own contexts — including when it named a + # context belonging to some other label + unplaced = [c.id for c in claims if label in c.refs and not (set(c.ctx_ids) & here)] + rest = "uncertain" if unplaced else "uncovered" for o in occs: - claim_id = assigned.get(o["id"]) + claim_id = reached.get(o["id"]) status[o["id"]] = ("covered", claim_id) if claim_id is not None else (rest, None) - unattributed += [{"claim_id": cid, "label": label} for cid in orphans] + unattributed += [{"claim_id": cid, "label": label} for cid in unplaced] items = [{**o, "status": status[o["id"]][0], "claim_id": status[o["id"]][1]} for o in occurrences] @@ -510,21 +602,68 @@ def coverage_audit(case_dir: Path, claims: list[ClaimResult]) -> dict: "labels_in_text": sorted(in_text, key=int), "covered": sorted(covered & in_text, key=int), "missing": missing, - "schema": "coverage/2", + "schema": "coverage/3", "unit": "occurrence", "source": source, "labels_partially_covered": partial, "labels_uncertain_only": uncertain_only, "occurrences": {"total": len(items), **counts, "items": items}, "attribution": { - "method": "location narrowing, then text similarity assigned globally best-first", - "min_ratio": OCCURRENCE_MIN_RATIO, - "min_margin": OCCURRENCE_MIN_MARGIN, + # no thresholds to report any more: the extractor was shown this + # inventory and returned the ids it used, so there is nothing to + # tune and no close call to refuse + "method": "context id returned by extraction, resolved against the inventory", "claims_unattributed": unattributed, }, } +def _stale_ingest(ingest_dir: Path, pdf_path: str | None, *, backend: str) -> bool: + """Must `ingest_dir` be rebuilt — wrong PDF, or read by the wrong backend? + + The directory is named after the reference's slug, and a slug is not an + identity that holds still. Fixing a slug collision renames one of the two + colliding entries, and the reconciler can hand `refs` the publisher's list + on one run and the parsed list on the next — so re-running an existing case + could hand the model the directory's previous occupant and judge a claim, + confidently, against a different paper. `SourceMap.doc` cannot catch it: + every cited source is stored as `.pdf`, so it reads the same either + way. + + An unhashed map — written before source maps recorded what they read — is + treated as stale. Re-ingesting is local, free and quick; trusting it is a + guess about which paper is in a file, and that guess is the whole thing this + module refuses to make. + + The **converter** is checked for the same reason, and it is not the same + question as the hash: a case folder built before sources were read + layout-aware holds `pymupdf` maps of exactly the right PDFs. Reusing one + under `--backend docling` would hand the judge a linearized table while the + run reports layout-aware source ingest — the fidelity claim would be true + of the paper and false of the papers it is judged against. + """ + if not pdf_path or not Path(pdf_path).exists(): + return False # nothing better to ingest; SourceProvenance reports the gap + smap_path = ingest_dir / "source_map.json" + if not smap_path.exists(): + return True + try: + from .models import SourceMap + + smap = SourceMap.from_json(smap_path) + recorded, converter = smap.source_sha256, smap.converter + except (OSError, ValueError, KeyError, TypeError): + return True + from .ingest import resolve_backend + from .models import manuscript_fingerprint + + if recorded != manuscript_fingerprint(Path(pdf_path)): + return True + # "auto" is not a converter name, and a docling map records its version + # ("docling 2.53.0"), so compare the resolved backend against the first word + return resolve_backend(backend) != converter.split()[0] + + def _slug_for_ref(manifest: RefManifest, label: str): return next((e for e in manifest.entries if e.num == label), None) @@ -532,15 +671,54 @@ def _slug_for_ref(manifest: RefManifest, label: str): _MAX_PAGE_DIGITS = 5 # a page number, not an integer literal -def _judgement_from(entry) -> tuple[dict | None, str]: +@dataclass(frozen=True) +class SourceProvenance: + """What one source actually contains, read from its own source map. + + The yardstick a judgement is held to. Without it a verdict's page and block + are the model's unchecked word for it, which is how `page 99999` and + `block_nope` survived into a report as provenance. + """ + + pages: int + block_pages: dict[str, int] # block id -> the page it is on + + @classmethod + def from_map(cls, smap) -> SourceProvenance: + return cls(pages=smap.pages, block_pages={b.id: b.page for b in smap.blocks}) + + @classmethod + def read(cls, source_map: Path) -> SourceProvenance | None: + """None when the map is missing or unreadable — never a permissive default. + + A guessed yardstick measures nothing. The caller turns None into + `unchecked`, so an unverifiable location is refused rather than trusted. + """ + if not source_map.exists(): + return None + try: + from .models import SourceMap + + return cls.from_map(SourceMap.from_json(source_map)) + except (OSError, ValueError, KeyError, TypeError): + return None + + +def _judgement_from(entry, provenance: SourceProvenance | None) -> tuple[dict | None, str]: """Validate one model response object into claim fields, or say why not. - Total by construction: every branch is an isinstance test, so this cannot - raise. A validator that throws would turn a bug in OUR code into a note - blaming the model — the same laundering `unchecked` exists to prevent. + Total by construction: every branch is an isinstance test or a lookup, so + this cannot raise. A validator that throws would turn a bug in OUR code into + a note blaming the model — the same laundering `unchecked` exists to prevent. Rejection is all-or-nothing. The caller writes no field unless every field validated, so a bad response never leaves half-applied provenance behind. + + `provenance` is the source's own source map. A substantive verdict must + name a page that exists and a block that exists **on that page**, because + the block is what guarantees the reader an evidence image: `crop_for_anchor` + takes its region from the block's bbox, so a valid block always produces a + crop and the anchor phrases only decide whether a red box is drawn on it. """ if not isinstance(entry, dict): return None, f"model returned an unusable verdict (not an object: {type(entry).__name__})" @@ -598,12 +776,44 @@ def _judgement_from(entry) -> tuple[dict | None, str]: if page < 1: return None, unusable_page # highlight does doc[page - 1] - # optional — but a non-string block id is never coerced into one + # the source map is the only thing that can contradict the model here. With + # no map nothing can, so nothing does — and a location nobody can check is + # refused rather than trusted. + if provenance is None: + return None, ( + f"model returned {verdict!r} but the source map could not be read, so " + f"the page and block it names cannot be checked against the source — " + f"re-run `papertrace ingest` for this source, then `papertrace check`" + ) + if page > provenance.pages: + return None, ( + f"model returned {verdict!r} for a passage on page {page}, but the " + f"source has {provenance.pages} page{'s' if provenance.pages != 1 else ''} " + f"— there is no such page to show" + ) + + # REQUIRED, not optional: the block's bbox is what `crop_for_anchor` uses as + # the crop region, so a judgement without one can leave the reader with no + # evidence image at all — a verdict nobody can look at. block = entry.get("source_block") - if block is not None and not isinstance(block, str): + if not isinstance(block, str) or not block.strip(): + return None, ( + f"model returned {verdict!r} with no source_block ({block!r}) — without " + f"one there is no region to crop, so the verdict would carry no evidence " + f"image a reader could check" + ) + block = block.strip() + block_page = provenance.block_pages.get(block) + if block_page is None: return None, ( - f"model returned an unusable verdict for this claim: {verdict!r} with a " - f"source_block that is not a block id ({block!r})" + f"model returned {verdict!r} citing {block}, which is not a block of " + f"this source — nothing to crop, nothing to check" + ) + if block_page != page: + return None, ( + f"model returned {verdict!r} citing {block}, which is on page " + f"{block_page}, not the page {page} it named — a crop of page {page} " + f"would show the reader a different passage" ) # absent means "none offered" and is allowed, as is an empty list. null, a @@ -626,6 +836,29 @@ def _judgement_from(entry) -> tuple[dict | None, str]: }, "" +# What the judge is actually reading. A supplement handed over unannounced gets +# treated as the article: `not_addressed` is the ordinary answer for an appendix +# that covers a different part of the work, and a judge with no reason to expect +# it reaches for `partial` instead and invents a true kernel. +_DOCKIND = { + "article": "the cited article itself.", + "supplement": ( + "supplementary material accompanying the cited article — an appendix, " + "supporting information, or an online-only data supplement. It is part of " + "the cited work, so what it states counts. But it covers only part of that " + "work, so a claim it simply does not speak to is `not_addressed`, and that " + "is the expected answer here far more often than for an article." + ), + "own_supplement": ( + "supplementary material belonging to the manuscript UNDER REVIEW, not to a " + "cited work. The claim points at it — a table, figure or section number the " + "paper names. Judge whether this document actually states what the paper " + "says it does. `not_addressed` means the paper pointed here and the thing it " + "pointed at is not here." + ), +} + + def check_claims( claims: list[ClaimResult], manifest: RefManifest, @@ -635,6 +868,11 @@ def check_claims( on_error=None, *, truncations: Truncations | None = None, + # REQUIRED, like `_clip`'s accumulator above: this decides whether a table + # in a cited source is readable at all, and neither possible default is + # honest. "auto" drags docling into an offline test run; "pymupdf" silently + # downgrades a caller who asked for layout. So there is no default. + backend: str, ) -> list[ClaimResult]: """Fill verdicts in place. One model call per source that carries claims. @@ -643,18 +881,33 @@ def check_claims( the claims get verdict `unchecked`, the reason lands in the note, and `on_error(slug, message)` fires so the CLI can say so loudly. - Sources are ingested with the flat backend on purpose — fast and - dependable, and text anchors are what verdicts and crops need. Layout - fidelity (tables/figures) is spent on the audited paper, not its sources. + Sources are ingested with the SAME backend as the audited paper. They used + to be read flat on the theory that text anchors are all a verdict needs, + but the decisive evidence for a claim is often a table — a subgroup row, a + confidence interval in a column — and a linearized table loses the + relationships that make those readable. Spending layout fidelity on the + paper and not on the papers it is judged against had the asymmetry + backwards. """ by_slug: dict[str, list[ClaimResult]] = {} for c in claims: pairs = [(r, _slug_for_ref(manifest, r)) for r in c.refs] avail = [(r, e) for r, e in pairs if e and e.status in ("retrieved", "provided") and e.slug] - if not avail: + own = manifest.manuscript_supplements if c.own_supplement else [] + if not avail and not own: c.verdict = "not_retrieved" - reasons = {e.status for _, e in pairs if e} - c.note = f"cited source not available ({', '.join(sorted(reasons)) or 'unknown ref'})" + if c.own_supplement: + # the paper said exactly where its evidence was and nobody + # opened it. That is a retrieval gap, not an uncited assertion. + c.note = ( + "points at this paper's own supplementary material, which was not " + "provided — pass it with --supplement" + ) + else: + reasons = {e.status for _, e in pairs if e} + c.note = ( + f"cited source not available ({', '.join(sorted(reasons)) or 'unknown ref'})" + ) continue # Co-citation is an offer of support: every source cited for this claim # was put forward as backing it, so every one that could be obtained is @@ -665,8 +918,35 @@ def check_claims( if e.slug in seen_slugs: # the same paper cited under two labels continue seen_slugs.add(e.slug) - c.judgements.append(SourceJudgement(source_slug=e.slug, ref=r)) + c.judgements.append(SourceJudgement(source_slug=e.slug, ref=r, kind="article")) by_slug.setdefault(e.slug, []).append(c) + # A supplement is part of the work that was cited, so it is read for + # every claim citing that label rather than only when the article + # turns out to be silent — a supplement contradicting a claim the + # article supports is exactly the finding that would be missed. + # Costs one extra call per supplement, not per claim: the loop below + # groups every claim for a document into a single call. + for s in e.supplements: + if s.slug in seen_slugs: + continue + seen_slugs.add(s.slug) + c.judgements.append( + SourceJudgement(source_slug=s.slug, ref=r, kind="supplement", + verified=s.verified) + ) + by_slug.setdefault(s.slug, []).append(c) + # the paper's own supplements answer for no citation label, so `ref` is + # empty: filling in a number would say the claim cited something it did + # not. Every one provided is read, matching the cited side and sparing + # the extractor a guess about which file "S3" lives in. + for s in own: + if s.slug in seen_slugs: + continue + seen_slugs.add(s.slug) + c.judgements.append( + SourceJudgement(source_slug=s.slug, ref="", kind="own_supplement") + ) + by_slug.setdefault(s.slug, []).append(c) # what is left here could NOT be obtained — the only remaining reason a # cited source goes unopened avail_refs = {r for r, _ in avail} @@ -676,16 +956,32 @@ def check_claims( try: ingest_dir = case_dir / "ingest" / slug annotated = ingest_dir / "annotated.md" - if not annotated.exists(): - entry = next(e for e in manifest.entries if e.slug == slug) + # a missing source_map.json is NOT re-ingested here: `entry.pdf_path` + # may be gone, and turning one absent artifact into a group-wide + # FileNotFoundError buries the real problem. It degrades per + # judgement instead, with a note naming the fix — see + # SourceProvenance.read. + doc = manifest.document(slug) + if doc is None: # pragma: no cover - every judged slug names a document + raise KeyError(f"no document named {slug!r} in the manifest") + if not annotated.exists() or _stale_ingest( + ingest_dir, doc.pdf_path, backend=backend + ): from .ingest import ingest_pdf - ingest_pdf(Path(entry.pdf_path), ingest_dir, backend="pymupdf") + ingest_pdf(Path(doc.pdf_path), ingest_dir, backend=backend) + # the quote goes with the paraphrase, not instead of it: the judge + # is told to rule on the quote, and the paraphrase stays so a claim + # whose extraction returned no quote is still judgeable claims_json = json.dumps( - [{"id": c.id, "claim": c.claim, "location": c.location} for c in group] + [ + {"id": c.id, "quote": c.quote, "claim": c.claim, "location": c.location} + for c in group + ] ) prompt = ( CHECK_PROMPT.replace("<>", claims_json) + .replace("<>", _DOCKIND[doc.kind]) .replace("<>", slug) .replace( "<>", @@ -716,6 +1012,7 @@ def check_claims( # deliberately NO per-claim `except Exception`: a blanket catch would # relabel our own bugs as the model's fault. The per-group except above # stays as scoped — ingest/prompt/_ask failures really are group-wide. + provenance = SourceProvenance.read(case_dir / "ingest" / slug / "source_map.json") for c in group: j = next((x for x in c.judgements if x.source_slug == slug), None) if j is None: # pragma: no cover - group membership implies one @@ -724,7 +1021,7 @@ def check_claims( if v is None: j.verdict, j.note = "unchecked", "model returned no verdict for this claim" continue - fields, why = _judgement_from(v) + fields, why = _judgement_from(v, provenance) if fields is None: j.verdict, j.note = "unchecked", why continue diff --git a/src/papertrace/cli.py b/src/papertrace/cli.py index f77e26e..e121e14 100644 --- a/src/papertrace/cli.py +++ b/src/papertrace/cli.py @@ -12,12 +12,14 @@ import os import re import sys +import tempfile from pathlib import Path import typer from rich.console import Console from rich.prompt import Prompt +from . import __version__ from .models import ClaimResult, RefManifest, RunResults, manuscript_fingerprint app = typer.Typer(add_completion=False, rich_markup_mode="rich", invoke_without_command=True) @@ -112,7 +114,8 @@ def _stage_case(case: Path | None) -> Path: hint = ( " audits in this folder: " + ", ".join(f"[cyan]-c {n}[/cyan]" for n in found[:8]) if found - else " no case folder found here — `papertrace run ` makes one." + else " no case folder found in the current directory — " + "`papertrace run ` makes one." ) console.print(f"[red]which audit? this step needs [bold]-c [/bold].[/red]\n{hint}") raise typer.Exit(2) @@ -136,6 +139,19 @@ def _case_conflict(case: Path, manuscript: Path) -> tuple[str | None, str]: return (None if same else previous.manuscript), "name" +def _manuscript_slot_owner(out: Path) -> Path | None: + """The case folder whose manuscript slot `out` is, or None. + + `/ingest/manuscript` is the one output path that stands for the + audited paper itself. Recognised by shape rather than by flag, so `--out` + cannot walk in behind `-c`'s back. + """ + out = Path(out) + if out.name != "manuscript" or out.parent.name != "ingest": + return None + return out.parent.parent + + def _guard_case(case: Path, manuscript: Path) -> str: """Refuse a case that holds another paper; return what that rested on. @@ -161,8 +177,9 @@ def _guard_case(case: Path, manuscript: Path) -> str: # uninformed and less usable, and `refs` re-ingests to make it true console.print( "[yellow]⚠ this case folder predates content hashing, so its identity is " - "unverified — only the file name was compared. Re-reading the paper from " - "scratch so the manifest and its hash describe the same file.[/yellow]" + "unverified — only the file name was compared, and two different papers " + "are routinely both called the same thing. The paper is re-read from " + "scratch rather than trusted from cache.[/yellow]" ) return basis @@ -260,9 +277,10 @@ def _provenance_line(converter: str) -> str: """Which backend read the manuscript, and how the sources were read. Both halves matter. The manuscript's backend decides whether tables and - figures exist at all. The sources are ingested flat-text *always* and on - purpose (`check.py` passes `backend="pymupdf"`), which no reader can infer - from a line that names docling — so it is said rather than assumed. + figures exist at all, and the cited sources are now read with the *same* + backend — so the one name covers both, which is exactly why it has to say + so. This line used to promise the opposite ("sources are always read as + flat text"), and a stale reassurance is worse than none. """ flat = converter.startswith("pymupdf") manuscript = ( @@ -272,8 +290,8 @@ def _provenance_line(converter: str) -> str: ) return ( f" read with: {manuscript}\n" - f" [dim]cited sources are always read as flat text — text anchors are what " - f"verdicts and crops need[/dim]" + f" [dim]cited sources are read with the same backend — the report names any " + f"that fell back to flat text[/dim]" ) @@ -299,8 +317,27 @@ def _email(cli_value: str | None) -> str: return email +def _version(value: bool) -> None: + """Print the installed version and stop. + + Read from `papertrace.__version__`, which `docs/RELEASING.md` names as the + version's one home — a literal here would drift at the next release and + answer confidently wrong, which is the failure this codebase exists to + refuse. + """ + if value: + console.print(f"papertrace {__version__}") + raise typer.Exit(0) + + @app.callback() -def _root(ctx: typer.Context) -> None: +def _root( + ctx: typer.Context, + version: bool = typer.Option( + None, "--version", "-V", callback=_version, is_eager=True, + help="Print the installed version and exit", + ), +) -> None: """Fact-check a paper's citations against the actual cited sources. New here? Run [bold]papertrace[/bold] with no arguments and answer the @@ -330,8 +367,19 @@ def start() -> None: @app.command(rich_help_panel="Utilities") -def init(case: Path = typer.Argument(Path("case"), help="Case folder to create")) -> None: +def init( + case: Path = typer.Argument(None, help="Case folder to create (default: ./case)"), + manuscript: Path = typer.Option( + None, "--for", exists=True, + help="Name the folder the way `run`/`refs` would for this paper, so a plain " + "follow-up run finds it on its own instead of leaving ./case/ orphaned", + ), +) -> None: """Create a case folder skeleton (gitignored by design — keep manuscripts local).""" + # an explicit folder name always wins; --for only fills in what an + # unnamed default would otherwise have to guess + if case is None: + case = default_case(manuscript) if manuscript else Path("case") _open_case(case) for sub in ("sources", "form", "ingest", "out/evidence"): (case / sub).mkdir(parents=True, exist_ok=True) @@ -339,28 +387,46 @@ def init(case: Path = typer.Argument(Path("case"), help="Case folder to create") console.print(f"case folder ready: [cyan]{case}/[/cyan]") console.print(" put reference PDFs you already have into [cyan]sources/[/cyan]") console.print(" put your questions or form-field screenshots into [cyan]form/[/cyan]") - # `run` and `refs` name their own folder after the paper, so a hand-made one - # is only used if it is passed - saying so here beats orphaned sources/ - console.print(f" [dim]hand this folder to every step: [cyan]-c {case}[/cyan][/dim]") + if manuscript: + console.print( + f" [dim]papertrace run {manuscript} will find this folder automatically[/dim]" + ) + else: + # `run` and `refs` name their own folder after the paper, so a hand-made + # one is only used if it is passed - saying so here beats orphaned sources/ + console.print(f" [dim]hand this folder to every step: [cyan]-c {case}[/cyan][/dim]") -@app.command(rich_help_panel="Pipeline stages — `run` calls these in order") -def ingest( - pdf: Path = typer.Argument(..., exists=True, help="PDF to convert"), - out: Path = typer.Option(None, "--out", "-o", help="Output dir (default /ingest/)"), - case: Path = typer.Option( - None, "--case", "-c", - help="Case folder; writes /ingest/. Ignored when --out is given", - ), - backend: str = typer.Option("auto", "--backend", help="auto | docling | pymupdf"), +def _ingest_pipeline( + *, + pdf: Path, + out: Path | None = None, + case: Path | None = None, + backend: str = "auto", ) -> None: - """PDF → clean.md + annotated.md + source_map.json (page + bbox provenance).""" + """PDF → clean.md + annotated.md + source_map.json (page + bbox provenance). + + Keyword-only and plain-default on purpose: `ingest()` below is a Typer + command, and Typer's declared defaults are `OptionInfo` objects rather than + the values the help screen shows — calling it directly (as `run()` and the + tests do) with a shifted or missing argument used to take that sentinel as + the value. This function is what they actually call; `ingest()` is a thin + CLI adapter over it. + """ from .ingest import ingest_pdf # -c means the same thing here as in every other subcommand; `papertrace # ingest -c foo` used to fail with "No such option: -c" while its # neighbours all took it. --out stays authoritative and unchanged. out = out or (case or default_case(pdf)) / "ingest" / pdf.stem + # the guard is about the manuscript SLOT, not the folder. A cited source + # ingested into /ingest/ is not the audited paper and must stay + # ingestable — `check` does exactly that. But /ingest/manuscript is + # what `refs` filled and `coverage_audit` reads, so a different paper + # landing there is the mixing `_guard_case` exists to prevent, reached by a + # command that never asked it. + if (owner := _manuscript_slot_owner(out)) is not None: + _guard_case(owner, pdf) smap = ingest_pdf(pdf, out, backend=backend) by_type = {t: sum(1 for b in smap.blocks if b.type == t) for t in ("sectionheader", "text", "table", "picture", "list")} @@ -389,25 +455,84 @@ def ingest( @app.command(rich_help_panel="Pipeline stages — `run` calls these in order") -def refs( - manuscript: Path = typer.Argument(..., exists=True), +def ingest( + pdf: Path = typer.Argument(..., exists=True, help="PDF to convert"), + out: Path = typer.Option(None, "--out", "-o", help="Output dir (default /ingest/)"), case: Path = typer.Option( None, "--case", "-c", - help="Case folder (default: a folder named after the paper, beside the paper)", - ), - provided: Path = typer.Option( - None, "--provided", - help="Folder of reference PDFs you already have; files match by name " - "-.pdf (e.g. pyrros-2023.pdf)", + help="Case folder; writes /ingest/. Ignored when --out is given", ), - email: str = typer.Option(None, "--email", envvar=["PAPERTRACE_EMAIL", "MANUSCRIPTAGENT_EMAIL"]), - parse_only: bool = typer.Option(False, "--parse-only", help="List references, no network"), backend: str = typer.Option("auto", "--backend", help="auto | docling | pymupdf"), ) -> None: - """Parse the References section, then retrieve open-access copies with an honest manifest.""" + """PDF → clean.md + annotated.md + source_map.json (page + bbox provenance).""" + _ingest_pipeline(pdf=pdf, out=out, case=case, backend=backend) + + +def _body_citation_labels(smap, citation_labels, is_references_heading) -> set[str]: + """The `[N]` markers the manuscript's body actually cites. + + Read from the source map and stopped at the bibliography, matching what + `coverage_audit` counts — the two readings are only worth comparing because + they come from one rule in `models.py`. Passed its two functions rather than + importing them, so this stays a pure function of the map. + + Table blocks are skipped: a 95% CI column like `[51, 77]` matches the same + bracket-and-comma syntax as a citation group `[7,8]`, and a table's own + numbers are never citations. + """ + body: list[str] = [] + for b in smap.blocks: + if is_references_heading(b.type, b.text): + break + if b.type != "table": + body.append(b.text) + return citation_labels("\n".join(body)) + + +def _detected_doi(manuscript: Path) -> str | None: + """The DOI printed on the paper's own front matter, or None. + + Imported lazily: `wizard` pulls in pymupdf and the interactive stack, and + `refs` should not pay for that to look up one string. + """ + from .wizard import detect_doi + + return detect_doi(manuscript) + + +def _refs_pipeline( + *, + manuscript: Path, + case: Path | None = None, + provided: Path | None = None, + email: str | None = None, + parse_only: bool = False, + backend: str = "auto", + doi: str | None = None, + supplement: list[Path] | None = None, +) -> None: + """Parse the References section, then retrieve open-access copies with an honest manifest. + + Keyword-only and plain-default on purpose: `refs()` below is a Typer + command, and Typer's declared defaults are `OptionInfo` objects rather than + the values the help screen shows — calling it directly (as `run()` and the + tests do) with a shifted or omitted argument used to take that sentinel as + the value. This function is what they actually call; `refs()` is a thin CLI + adapter over it. + """ from .ingest import ingest_pdf, references_span - from .models import SourceMap - from .refs import parse_references, resolve_all + from .models import SourceMap, citation_labels, is_references_heading, paper_title + from .refs import ( + _client, + crossref_deposit, + deposit_corroborates, + deposit_is_this_paper, + manuscript_supplements, + parse_references, + reconcile, + resolve_all, + unused_provided, + ) case = _resolve_case(case, manuscript) # named after the paper unless -c said otherwise # identity first — the cached source map below is a manuscript-derived @@ -422,6 +547,13 @@ def refs( # to share this one's name, so re-read the paper we were actually given if cached.exists() and basis != "name": smap = SourceMap.from_json(cached) + elif parse_only: + # --parse-only is an inspection: "List references, no network". It must + # not rewrite the case's manuscript slot and then return before the + # manifest catches up, which left the source map describing one paper + # and the manifest another. Read the paper somewhere disposable instead. + with tempfile.TemporaryDirectory() as scratch: + smap = ingest_pdf(manuscript, Path(scratch), backend=backend) else: smap = ingest_pdf(manuscript, ingest_dir, backend=backend) @@ -431,6 +563,95 @@ def refs( console.print("[red]No numbered references found — is there a References section?[/red]") raise typer.Exit(1) console.print(f"parsed [bold]{len(entries)}[/bold] numbered references") + + # The manuscript's own [N] markers arbitrate. Free, offline, and the only + # one of the three readings that is definitionally right about what the + # paper cites — the parse and the deposit are both candidates measured + # against it. --parse-only stays offline, so it gets no second candidate. + body_labels = _body_citation_labels(smap, citation_labels, is_references_heading) + crossref_entries, absent, identity_note = None, "", "" + if not parse_only: + doi = doi or _detected_doi(manuscript) + with _client() as client: + deposit = crossref_deposit(client, doi, _email(email)) + absent = deposit.absent + # Is the record behind that DOI this paper at all? The DOI is typed by + # hand or scraped off page 1, and this is the one retrieval route in + # `refs` that can replace the *entire* reference list — every other one + # has been title-checked since a wrong download was judged as a source. + identity = ( + deposit_is_this_paper(paper_title(smap), deposit.title) + if deposit.entries else None + ) + if deposit.unrenderable: + # the tool's shortfall, named as the tool's. A list this one could + # only half read must not be mapped onto [1]..[n] — that would drop + # the rest silently — but the reader is told whose limitation it is + absent = ( + f"{deposit.publisher or 'the publisher'} deposited {deposit.deposited} " + f"references and this tool could only read {len(deposit.entries)} of " + "them, so the deposit was set aside rather than used to renumber the " + "list. The gap is this tool's, not the publisher's" + ) + elif identity is False: + absent = ( + f"the DOI used ({doi}) belongs to a Crossref record titled " + f"\u201c{deposit.title}\u201d, which is not this paper, so the " + f"{len(deposit.entries)} references it deposited were not used to " + "renumber this list" + ) + console.print( + f"[yellow]⚠ the DOI {doi} resolves to a different paper[/yellow] — " + f"“{deposit.title[:70]}”. Its reference list was not used." + ) + elif deposit.entries: + crossref_entries = deposit.entries + # A title this tool cannot read is common — an article-type banner + # where the title should be, and no metadata behind it. The paper's + # own bibliography settles it instead: two readings of one reference + # list agree about the works, and no other paper's list does. + corroboration = ( + deposit_corroborates(deposit.entries, entries) if identity is None else None + ) + # a verified identity is worth as much as the count match it licenses, + # and an unverifiable one must not be read as either + if identity: + identity_note = f". The DOI {doi} was confirmed as this paper by title" + elif corroboration and corroboration.confirms: + identity_note = ( + f". The paper's title could not be compared with the record's, but " + f"{corroboration.found} of the {corroboration.total} references the " + "DOI's record deposited appear in the list printed in this paper, " + "which another paper's bibliography would not" + ) + else: + identity_note = ( + f". The DOI {doi} could not be confirmed as this paper — too little " + "title to compare, and " + + ("too few references to compare either" + if corroboration and corroboration.too_few + else f"only {corroboration.found} of the {corroboration.total} " + "references it deposited appear in this paper's own list" + if corroboration else "no deposit to compare") + + ", so the identity behind this list is unverified" + ) + console.print( + f"crossref: [bold]{len(deposit.entries)}[/bold] references deposited by " + f"{deposit.publisher or 'the publisher'} " + f"[dim](DOI {doi}; identity " + f"{'confirmed by title' if identity else 'confirmed by bibliography' if corroboration and corroboration.confirms else 'unverified'})[/dim]" + ) + + entries, rec = reconcile(body_labels, crossref_entries, entries, crossref_absent=absent) + if rec.source == "crossref" and identity_note: + # the note is what a reader of `refs_manifest.json` gets, so a list taken + # from a publisher's record says on whose authority it was adopted + rec.note += identity_note + if rec.verified: + console.print(f"[green]✓ numbering confirmed[/green] — {rec.note}") + else: + console.print(f"[yellow]⚠ numbering unconfirmed[/yellow] — {rec.note}") + if references_resumed: # a list interrupted by another section used to end at the interruption: # 9 of 15 references parsed, and the last 6 never retrieved or checked @@ -462,14 +683,39 @@ def tick(e): mark = STATUS_MARK.get(e.status, "?") via = f" via {e.resolver}" if e.resolver else "" console.print(f" {mark} [{e.num:>3}] {e.status:<10}{via:<16} {e.reason}") - - resolve_all(entries, dest, _email(email), provided_dir=provided, progress=tick) + if e.supplements: + n = len(e.supplements) + names = ", ".join(s.slug for s in e.supplements) + console.print( + f" [cyan]+ {n} supplement{'' if n == 1 else 's'}[/cyan] " + f"[dim]{names} — judged as separate documents[/dim]" + ) + + # the paper's own supplements claim their slugs FIRST, then `resolve_all` + # works around them: one namespace, because both end up as `ingest//` + # and `sources_resolved/.pdf` + taken: set[str] = {e.slug for e in entries if e.slug} + own = manuscript_supplements(list(supplement or []), taken) + for s in own: + console.print(f" [cyan]+[/cyan] {Path(s.pdf_path).name} → this paper's own supplement") + + resolve_all(entries, dest, _email(email), provided_dir=provided, progress=tick, taken=taken) + + # a file the user deliberately put in the folder that then did nothing is the + # quietest possible failure — they would go on believing it had been read + for pdf, why in unused_provided(entries, provided): + console.print(f" [yellow]⚠ {pdf.name} set aside — {why}[/yellow]") manifest = RefManifest( manuscript=manuscript.name, entries=entries, + manuscript_supplements=own, manuscript_sha256=manuscript_fingerprint(manuscript), # identity, not the name references_resumed=references_resumed, + reference_source=rec.source, + numbering_verified=rec.verified, + numbering_note=rec.note, + unverified_from=rec.unverified_from, ) manifest.to_json(case / "refs_manifest.json") ok = len(manifest.retrieved) @@ -482,6 +728,38 @@ def tick(e): " unverifiable, never guessed.[/dim]") +@app.command(rich_help_panel="Pipeline stages — `run` calls these in order") +def refs( + manuscript: Path = typer.Argument(..., exists=True), + case: Path = typer.Option( + None, "--case", "-c", + help="Case folder (default: a folder named after the paper, beside the paper)", + ), + provided: Path = typer.Option( + None, "--provided", + help="Folder of reference PDFs you already have; files match by name " + "-.pdf (e.g. pyrros-2023.pdf)", + ), + email: str = typer.Option(None, "--email", envvar=["PAPERTRACE_EMAIL", "MANUSCRIPTAGENT_EMAIL"]), + parse_only: bool = typer.Option(False, "--parse-only", help="List references, no network"), + backend: str = typer.Option("auto", "--backend", help="auto | docling | pymupdf"), + doi: str = typer.Option( + None, "--doi", + help="DOI of the paper itself — fetches the publisher's own reference list to " + "check the parsed numbering against (default: the DOI printed on page 1)", + ), + supplement: list[Path] = typer.Option( + None, "--supplement", exists=True, + help="Supplementary material for THIS paper (repeatable). A cited work's " + "supplement needs no flag — drop it in the sources folder named after " + "the reference, e.g. pyrros-2023-supplement.pdf", + ), +) -> None: + """Parse the References section, then retrieve open-access copies with an honest manifest.""" + _refs_pipeline(manuscript=manuscript, case=case, provided=provided, email=email, + parse_only=parse_only, backend=backend, doi=doi, supplement=supplement) + + @app.command(rich_help_panel="Pipeline stages — `run` calls these in order") def scout( case: Path = typer.Option( @@ -507,10 +785,18 @@ def scout( if res.error: console.print(f"[yellow]⚠ scout incomplete: {res.error}[/yellow]") if res.paper_title: + # `via doi` stopped meaning "identified reliably" when `run` began + # reading the DOI off page 1, so the warning turns on the identity check + # rather than on which query happened to answer + caveat = { + "confirmed": "", + "unverified": " — identity unverified, check this is your paper", + "mismatch": " — NOT this paper", + }.get(res.paper_identity, " — wrong paper? pass --doi") console.print( f"paper: [bold]{res.paper_title[:80]}[/bold] ({res.paper_year or '?'})" - f" · [dim]identified via {res.resolved_via}" - f"{' — wrong paper? pass --doi' if res.resolved_via == 'title' else ''}[/dim]" + f" · [dim]identified via {res.resolved_via}, identity " + f"{res.paper_identity or 'not recorded'}{caveat}[/dim]" ) console.print(f"[green]▸[/green] published since: [bold]{len(res.newer)}[/bold] candidates") for h in res.newer[:5]: @@ -524,21 +810,36 @@ def scout( console.print(f" [cyan]{h.year or '?'}[/cyan] {h.title[:76]}") if len(res.overlooked) > 5: console.print(f" [dim]… {len(res.overlooked) - 5} more in scout.json[/dim]") + if res.same_year: + console.print( + f"[yellow]▸[/yellow] same year as the paper: [bold]{len(res.same_year)}[/bold]" + " candidates [dim]— may postdate submission, so neither newer nor owed[/dim]" + ) + for h in res.same_year[:5]: + console.print(f" [cyan]{h.year or '?'}[/cyan] {h.title[:76]}") + if len(res.same_year) > 5: + console.print(f" [dim]… {len(res.same_year) - 5} more in scout.json[/dim]") console.print( "[dim]search-based — absence from these lists proves nothing; presence is a" " candidate for your judgement, not an accusation.[/dim]" ) -@app.command(rich_help_panel="Pipeline stages — `run` calls these in order") -def check( - case: Path = typer.Option( - None, "--case", "-c", - help="Case folder holding the audit (required unless ./case exists)", - ), - model: str = typer.Option(None, "--model", help="Model override for claude -p"), +def _check_pipeline( + *, + case: Path | None = None, + model: str | None = None, + backend: str = "auto", ) -> None: - """Extract citation-backed claims and judge each against its cited source (claude -p).""" + """`check`'s work, with ordinary Python defaults. + + Keyword-only for the reason the other pipeline functions are: an omitted + argument to the Typer command is an `OptionInfo`, not the default `--help` + shows. `backend` is what made this split necessary — it reaches + `ingest_pdf`, which refuses an unrecognised value loudly, so a sentinel + arriving here would fail an audit at the judging step after the retrieval + work was already done. + """ from .check import Truncations, check_claims, claude_available, extract_claims case = _stage_case(case) @@ -572,7 +873,7 @@ def fail(slug, msg): with console.status("reading claims against their cited pages…"): check_claims( claims, manifest, case, model, progress=tick, on_error=fail, - truncations=truncations, + truncations=truncations, backend=backend, ) from .check import coverage_audit @@ -581,6 +882,14 @@ def fail(slug, msg): coverage = coverage_audit(case, claims) smap_path = case / "ingest" / "manuscript" / "source_map.json" converter = SourceMap.from_json(smap_path).converter if smap_path.exists() else "pymupdf" + # how each cited source was read, recorded per slug. The manuscript's + # converter above says nothing about them, and until this was carried the + # markdown and HTML reports said nothing about them either. + source_converters: dict[str, str] = {} + for doc in manifest.documents(): + sp = case / "ingest" / doc.slug / "source_map.json" + if sp.exists() and doc.slug not in source_converters: + source_converters[doc.slug] = SourceMap.from_json(sp).converter from .check import last_model @@ -591,6 +900,7 @@ def fail(slug, msg): refs_total=len(manifest.entries), refs_available=len(manifest.retrieved), converter=converter, + source_converters=source_converters, claims=claims, uncited=uncited, coverage=coverage, @@ -622,6 +932,45 @@ def fail(slug, msg): console.print(f"[cyan]{len(uncited)} uncited assertions[/cyan] — see report section") +@app.command(rich_help_panel="Pipeline stages — `run` calls these in order") +def check( + case: Path = typer.Option( + None, "--case", "-c", + help="Case folder holding the audit (required unless ./case exists)", + ), + model: str = typer.Option(None, "--model", help="Model override for claude -p"), + backend: str = typer.Option("auto", "--backend", help="auto | docling | pymupdf"), +) -> None: + """Extract citation-backed claims and judge each against its cited source (claude -p).""" + _check_pipeline(case=case, model=model, backend=backend) + + +def _downgrade_unshowable(anchor) -> bool: + """A substantive verdict with no evidence image stops being a verdict. + + `check` validates page and block against the source map, which is what + normally guarantees a crop. This is the same rule enforced against reality: + the PDF can be absent from `sources_resolved/`, and a source map can + disagree with the PDF it was built from. `not_addressed` is exempt — it + never claimed a passage, so it owes no picture. + + Returns True when it downgraded, so the caller can say so on the console. + """ + substantive = ("supported", "partial", "contradicted") + if anchor.verdict not in substantive or anchor.evidence_image: + return False + anchor.verdict = "unchecked" + anchor.note = ( + "no evidence image could be produced for the passage this verdict rests on " + f"(page {anchor.source_page}" + + (f", {anchor.source_block}" if anchor.source_block else "") + + ") — the source PDF is missing from sources_resolved/, or its pages no " + "longer match the source map it was ingested from. Re-run " + "`papertrace refs` and `papertrace check` for this source." + ) + return True + + @app.command(rich_help_panel="Pipeline stages — `run` calls these in order") def highlight( case: Path = typer.Option( @@ -646,11 +995,14 @@ def highlight( for a in c.judgements or [c]: img = crop_for_anchor(a, c.id, case / "sources_resolved", case / "ingest", out_dir) if img is None and a.source_slug: - # sources provided by the user live elsewhere — try the manifest path + # sources provided by the user live elsewhere — try the manifest + # path. `document()` and not a scan of `entries`: a supplement is + # never in `entries`, so scanning them left every supplement + # verdict with no crop and no reason given. manifest = RefManifest.from_json(case / "refs_manifest.json") - entry = next((e for e in manifest.entries if e.slug == a.source_slug), None) - if entry and entry.pdf_path: - src = Path(entry.pdf_path) + doc = manifest.document(a.source_slug) + if doc and doc.pdf_path: + src = Path(doc.pdf_path) tmp = case / "sources_resolved" / f"{a.source_slug}.pdf" if src.exists() and not tmp.exists(): tmp.parent.mkdir(parents=True, exist_ok=True) @@ -662,12 +1014,22 @@ def highlight( if img: a.evidence_image = str(Path(img).relative_to(case / "out")) done += 1 - if a.anchor_located: + # `is True` / `is False` / `is None` — never truthiness. None + # means nothing was ever searched for, and calling that "not + # found on the page" asserts a search that did not happen. + if a.anchor_located is True: console.print(f" [green]✓[/green] {tag}: {a.evidence_image}") + elif a.anchor_located is False: + console.print( + f" [yellow]○ {tag}: {a.evidence_image} — the anchor phrase " + f"was searched for and not found on the page; crop written " + f"unboxed[/yellow]" + ) else: console.print( f" [yellow]○ {tag}: {a.evidence_image} — no anchor phrase " - f"found on the page; crop written unboxed[/yellow]" + f"was offered, so none was searched for; crop written " + f"unboxed[/yellow]" ) elif a.source_slug and a.source_page: # a page the source does not have is not the same as a page that @@ -679,6 +1041,10 @@ def highlight( f"but {a.source_slug} has {n} — no page to read, so no crop " f"and no anchor claim[/yellow]" ) + if _downgrade_unshowable(a): + console.print( + f" [yellow]⚠ {tag}: {a.note}[/yellow]" + ) # the claim-level evidence_image must follow the deciding judgement, or # the crop shown beside the headline belongs to a different source c.apply_headline() @@ -686,20 +1052,32 @@ def highlight( console.print(f"[bold]{done}[/bold] evidence crops written") -@app.command(rich_help_panel="Pipeline stages — `run` calls these in order") -def report( - case: Path = typer.Option( - None, "--case", "-c", - help="Case folder holding the audit (required unless ./case exists)", - ), - png: bool = typer.Option( - False, "--png/--no-png", - help="Also export PNG images of the report looks (one-time: playwright install chromium)", - ), +def _report_pipeline( + *, + case: Path | None = None, + png: bool = False, + formats: list[str] | None = None, ) -> None: - """Render report.md + the editor/terminal looks from results.json.""" + """`report`'s work, with ordinary Python defaults. + + Keyword-only so `run()` and the tests calling it directly cannot silently + receive a Typer `OptionInfo` in place of a value — the flaw that has shipped + twice here already. `formats` is the parameter that made this split + necessary: `run()` used to call `report(case=..., png=...)`, so a new + option would have arrived as a truthy sentinel and rendered whatever that + happened to mean. + """ from .models import ScoutResults - from .report import write_reports + from .report import FORMATS, write_reports + + # a mistyped flag is user error, answered before the results are loaded so + # it cannot half-write a report folder — and with a line, not a traceback + if bad := [f for f in (formats or []) if f not in FORMATS]: + console.print( + f"[red]unknown --format {', '.join(bad)}[/red] — " + f"expected any of {', '.join(f'[cyan]{f}[/cyan]' for f in FORMATS)}" + ) + raise typer.Exit(2) case = _stage_case(case) results = RunResults.from_json(case / "out" / "results.json") @@ -711,11 +1089,31 @@ def report( # once, minutes earlier and above a wall of model-loading logs; a standalone # `papertrace report` never said it at all. console.print(_provenance_line(results.converter)) - paths = write_reports(results, manifest, case / "out", png=png, scout=scout_res) + paths = write_reports(results, manifest, case / "out", png=png, scout=scout_res, + formats=formats or ["md"]) for p in paths: console.print(f" [green]✓[/green] {p.relative_to(case)}") +@app.command(rich_help_panel="Pipeline stages — `run` calls these in order") +def report( + case: Path = typer.Option( + None, "--case", "-c", + help="Case folder holding the audit (required unless ./case exists)", + ), + png: bool = typer.Option( + False, "--png/--no-png", + help="Also export PNG images of the report looks (one-time: playwright install chromium)", + ), + formats: list[str] = typer.Option( + None, "--format", "-f", + help="Extra looks to render beside report.md: editor | terminal (repeatable)", + ), +) -> None: + """Render report.md — and the editor/terminal looks on request — from results.json.""" + _report_pipeline(case=case, png=png, formats=formats) + + @app.command(rich_help_panel="Start here") def run( manuscript: Path = typer.Argument(..., exists=True), @@ -739,7 +1137,21 @@ def run( True, "--scout/--no-scout", help="Also scan Europe PMC for newer + uncited literature", ), - doi: str = typer.Option(None, "--doi", help="DOI of the paper itself, for the scout step"), + doi: str = typer.Option( + None, "--doi", + help="DOI of the paper itself — checks the reference numbering against the " + "publisher's deposited list, and pins the scout's literature search", + ), + formats: list[str] = typer.Option( + None, "--format", "-f", + help="Extra looks to render beside report.md: editor | terminal (repeatable)", + ), + supplement: list[Path] = typer.Option( + None, "--supplement", exists=True, + help="Supplementary material for THIS paper (repeatable). A cited work's " + "supplement needs no flag — drop it in the sources folder named after " + "the reference, e.g. pyrros-2023-supplement.pdf", + ), ) -> None: """Full pipeline: ingest → refs → scout → check → highlight → report.""" console.print(BANNER) @@ -749,22 +1161,33 @@ def run( case = _resolve_case(case, manuscript) _guard_case(case, manuscript) # one case folder per paper — never mix two audits _open_case(case) - # KEYWORDS ONLY, deliberately. These stages are Typer commands called as - # plain functions, and Typer's declared defaults are OptionInfo objects - # rather than the values they display. A positional call therefore breaks - # silently the moment any stage gains a parameter: the arguments shift, the - # shifted-in default is an OptionInfo that equals none of the expected - # strings, and the stage takes a fallback branch. Adding `--case` to - # `ingest` did exactly that — the backend became an OptionInfo and every - # audit ingested as flat text while claiming layout-aware ingest. - ingest(pdf=manuscript, out=case / "ingest" / "manuscript", case=case, backend=backend) - refs(manuscript=manuscript, case=case, provided=provided, email=email, - parse_only=False, backend=backend) + # KEYWORDS ONLY, deliberately, for every stage below still called through its + # Typer command. Typer's declared defaults are OptionInfo objects rather than + # the values they display, so a positional call breaks silently the moment a + # stage gains a parameter: the arguments shift, the shifted-in default is an + # OptionInfo that equals none of the expected strings, and the stage takes a + # fallback branch. Adding `--case` to `ingest` did exactly that — the backend + # became an OptionInfo and every audit ingested as flat text while claiming + # layout-aware ingest. `_ingest_pipeline`, `_refs_pipeline`, + # `_check_pipeline` and `_report_pipeline` below are split out of their + # Typer commands specifically to make that mistake impossible rather than + # just avoided by convention — `scout` and `highlight` are still + # convention-only, and each should be split the next time it gains a + # parameter. + _ingest_pipeline(pdf=manuscript, out=case / "ingest" / "manuscript", case=case, backend=backend) + # detected once, here, and handed to both consumers. `refs` detects for + # itself when called alone, so forwarding the raw option left the scout + # guessing by title on the very runs where the paper's DOI was sitting on + # page 1 — and a wrong title match anchors the whole scan to another paper + # without erroring. + doi = doi or _detected_doi(manuscript) + _refs_pipeline(manuscript=manuscript, case=case, provided=provided, email=email, + parse_only=False, backend=backend, doi=doi, supplement=supplement) if with_scout: scout(case=case, doi=doi, email=email) - check(case=case, model=model) + _check_pipeline(case=case, model=model, backend=backend) highlight(case=case, claim=None) - report(case=case, png=png) + _report_pipeline(case=case, png=png, formats=formats) # a four-minute run should not need scrolling to learn how the paper was # read, so the backend rides on the last line too smap_path = case / "ingest" / "manuscript" / "source_map.json" diff --git a/src/papertrace/disclosures.py b/src/papertrace/disclosures.py index 04cb2b8..cc6e59b 100644 --- a/src/papertrace/disclosures.py +++ b/src/papertrace/disclosures.py @@ -1,11 +1,11 @@ """What a run must disclose — decided once in Python, rendered three times in Jinja. Deliberately not a shared Jinja macro. The three report formats need genuinely -different markup, so one macro would carry format switches; `report.py` sets -`autoescape=select_autoescape(["html"])`, so a macro shared between `.md.j2` -and `.html.j2` is escaped differently depending on which template imported it — -a correctness hazard on the one surface whose job is not lying; and a macro is -testable only by rendering. +different markup, so one macro would carry format switches; `report.py` escapes +`.html.j2` templates and not `.md.j2` ones, so a macro shared between them is +escaped differently depending on which template imported it — a correctness +hazard on the one surface whose job is not lying; and a macro is testable only +by rendering. Every `Disclosure` carries a `token`: a short literal that must appear verbatim in **all three** formats. Each format phrases at its own length around it, and @@ -17,20 +17,41 @@ from dataclasses import dataclass +# the one place the judgement vocabulary is defined — a local copy of the four +# names here would be a second vocabulary to keep in step +from .models import JUDGMENT_VERDICTS + # Tokens are the contract. Changing one is a change to all three templates, and # tests/test_disclosure_parity.py is what says so out loud. TRUNCATION_TOKEN = "text past the cut was never read" COVERAGE_TOKEN = "reached by an extracted claim" # published contract — tests/test_coverage.py asserts this literal COVERAGE_CAVEAT_TOKEN = "coverage not audited" -COVERAGE_ATTRIBUTION_TOKEN = "attribution is a text match that can be wrong" +COVERAGE_ATTRIBUTION_TOKEN = "attribution is the context the extractor named" UNJUDGED_TOKEN = "could not be obtained, so was never opened" MULTISOURCE_TOKEN = "cited sources checked" +# same falseness `headline_qualifier` avoids: one cited work read as its article +# plus a supplement is not two cited works, and this count sits under the headline +MULTISOURCE_DOCUMENTS_TOKEN = "documents checked" ANCHOR_LOCATED_TOKEN = "red box = matched text" ANCHOR_NOT_LOCATED_TOKEN = "no anchor phrase was found on this page" ANCHOR_UNKNOWN_TOKEN = "anchor match not recorded" SOURCE_IDENTITY_TOKEN = "identity was never confirmed" REFERENCES_RESUMED_TOKEN = "reference list continued past a section break" +NUMBERING_TOKEN = "reference numbering could not be confirmed" +CLAIM_NUMBERING_TOKEN = "cites a reference whose numbering was never confirmed" +# no apostrophe, and no `&`, `<` or `>`: a token is asserted as a literal in the +# HTML formats too, and autoescape would rewrite it there but not in markdown — +# so the parity test would fail on a difference the reader never sees +NO_QUOTE_TOKEN = "judged on a paraphrase, not the sentence in the paper" +SOURCE_FIDELITY_TOKEN = "cited sources read as flat text" +# Neutral on purpose. The token is asserted verbatim in all three formats, so it +# must stay true whether every supplement was checked, none was, or some were — +# "carry no identity check" was true when nothing could be verified and became a +# falsehood about the checked ones the moment some could. +SUPPLEMENT_IDENTITY_TOKEN = "how each supplement was attached" +SUPPLEMENT_COVERAGE_TOKEN = "citations inside a supplement are not counted" +SUPPLEMENT_HEADLINE_TOKEN = "this verdict rests on supplementary material" @dataclass(frozen=True) @@ -39,6 +60,8 @@ class Disclosure: key: str # truncation | converter | coverage | coverage_caveat # | coverage_attribution | sources | unjudged_refs | anchor + # | no_quote | claim_numbering | numbering | references_resumed + # | source_identity | source_fidelity level: str # info | warn token: str # SHORT literal that must appear verbatim in ALL THREE formats text: str # full sentence for markdown / editor @@ -98,6 +121,62 @@ def anchor_state(claim) -> str: ), } +# The same three facts when no crop was written. Same tokens on purpose — the +# parity contract is the token, so a format cannot drop one by taking this +# branch — but the sentence must not describe a picture that does not exist. +ANCHOR_NO_IMAGE: dict[str, Disclosure] = { + "located": Disclosure( + key="anchor", + level="info", + token=ANCHOR_LOCATED_TOKEN, + text=( + f"{ANCHOR_LOCATED_TOKEN} — the anchor phrase was located on this page " + "by text search, though no evidence image was written for it." + ), + short=f"{ANCHOR_LOCATED_TOKEN} — no evidence image", + ), + "not_located": Disclosure( + key="anchor", + level="warn", + token=ANCHOR_NOT_LOCATED_TOKEN, + text=( + f"{ANCHOR_NOT_LOCATED_TOKEN}, and no evidence image was produced — so " + "the page named above is the only provenance this verdict carries." + ), + short=f"{ANCHOR_NOT_LOCATED_TOKEN} — and no evidence image", + ), + "unknown": Disclosure( + key="anchor", + level="warn", + token=ANCHOR_UNKNOWN_TOKEN, + text=( + f"{ANCHOR_UNKNOWN_TOKEN} — no anchor phrase was searched for, or the " + "highlight step did not run, and no evidence image was produced. " + "Nothing here claims a match." + ), + short=f"{ANCHOR_UNKNOWN_TOKEN} — and no evidence image", + ), +} + + +def anchor_disclosure(anchor) -> Disclosure | None: + """The anchor caption for one claim or judgement, or None if it owes none. + + Gated on *provenance*, not on the picture. A judgement that names a page has + made a claim about where the evidence is, and owes the reader a statement + about whether anything was found there — whether or not a crop was written. + Gating on `evidence_image` was how a verdict with a page number and no crop + came to disclose nothing at all. + + A claim with no page (`not_retrieved`, or a check that failed before any + location was offered) gets None: silence about nothing is not a dropped + disclosure. + """ + if getattr(anchor, "source_page", None) is None: + return None + table = ANCHOR if getattr(anchor, "evidence_image", None) else ANCHOR_NO_IMAGE + return table[anchor_state(anchor)] + # -------------------------------------------------------------------------- # run-level and claim-level rules @@ -225,28 +304,33 @@ def _coverage_attribution() -> Disclosure: Label-level coverage was set arithmetic and could not produce a false positive. This can, so the ways it is weaker are printed on the report's face rather than merely known. + + Shorter than it was, because one of the weaknesses is gone: attribution + used to be a similarity match between a paraphrase and a sentence, which + could place a claim on the wrong one of two look-alike sentences and + refused close calls outright. Extraction is now shown the occurrence list + and returns the id it used. What remains is that naming the id is still a + model step. """ return Disclosure( key="coverage_attribution", level="warn", token=COVERAGE_ATTRIBUTION_TOKEN, text=( - f"How to read that figure: {COVERAGE_ATTRIBUTION_TOKEN} — deciding which " - "citation a claim came from is a text comparison, so the counts can be " - "right while a pointer is wrong. An attribution the tool cannot make " - "counts as NOT covered, never as covered — and it refuses close " - "calls, so two similar sentences citing one reference can both read " - "as unaddressed where a reader would pair them at a glance. This " - "figure understates coverage there. A sentence citing the same " - "reference twice needs two extracted claims, so the ratio is not " - "comparable between papers. And detection still reads bracketed numeric " - "markers only — a citation style it cannot see contributes no " - "occurrences at all, which makes this ratio look better than reality, " - "not worse." + f"How to read that figure: {COVERAGE_ATTRIBUTION_TOKEN}. Extraction is " + "shown every place this paper cites something and returns which of them " + "each claim came from, so the pointer is no longer a text comparison — " + "but naming it is still a model step, and a claim can be placed on the " + "wrong sentence. A claim that names no place at all leaves that " + "reference's remaining places counted as NOT covered, never as covered, " + "so the figure understates coverage there. And detection still reads " + "bracketed numeric markers only — a citation style it cannot see " + "contributes no occurrences at all, which makes this ratio look better " + "than reality, not worse." ), short=( - f"{COVERAGE_ATTRIBUTION_TOKEN}; unattributable = not covered, " - "including close calls it refuses to decide; unseen citation styles " + f"{COVERAGE_ATTRIBUTION_TOKEN}, which is still a model step; a claim " + "placed nowhere counts as not covered; unseen citation styles " "contribute no occurrences, so the ratio flatters the run" ), ) @@ -321,6 +405,176 @@ def _references_resumed(total: int) -> Disclosure: ) +def _numbering(manifest) -> Disclosure: + """Nobody established that entry [N] is the work the manuscript's [N] means. + + The one disclosure that can invalidate every other finding on the page. The + citation label is the join key between a claim and the source it is judged + against, so a list off by one does not produce a *worse* audit — it produces + a confident audit of the wrong papers. One live run misnumbered 27 of 41 + references and said so nowhere, because `parse_references` was the only + stage in the pipeline that could not report its own failure. + """ + start = manifest.unverified_from + scope = ( + f"Entries from [{start}] onward are affected" + if start and start > 1 + else "Every entry is affected" + ) + detail = manifest.numbering_note or ( + "this manifest was written before the reference list was reconciled against " + "the manuscript's own citation labels, so nothing ever checked it" + ) + return Disclosure( + key="numbering", + level="warn", + token=NUMBERING_TOKEN, + text=( + f"The {NUMBERING_TOKEN} — {detail}. {scope}. The citation label is what " + "joins a claim to the source it is judged against, so where the numbering " + "is wrong the verdict is about a different paper than the one named. " + "Check the retrieval manifest against the paper's own reference list." + ), + short=f"{NUMBERING_TOKEN} — {scope.lower()}", + ) + + +def _claim_numbering(claim, manifest) -> Disclosure: + """The run-level warning, said again where the verdict is read. + + A banner at the top of a report is not where someone acting on a single + verdict is looking. The labels are named, because the reader's next move is + to check those specific references by hand. + """ + doubtful = sorted( + (r for r in claim.refs if manifest.label_is_doubtful(r)), + key=lambda r: int(r), + ) + labels = f"[{'], ['.join(doubtful)}]" + return Disclosure( + key="claim_numbering", + level="warn", + token=CLAIM_NUMBERING_TOKEN, + text=( + f"This claim {CLAIM_NUMBERING_TOKEN}: {labels}. The source judged here was " + "chosen by that label, so if the reference list is misnumbered this verdict " + "is about a different paper. Verify the reference before relying on it." + ), + short=f"{CLAIM_NUMBERING_TOKEN}: {labels}", + ) + + +def _source_fidelity(flat: list[str], total: int) -> Disclosure: + """Which cited sources were read as flat text, and what that costs. + + `_converter` above says how the *audited paper* was read. This says how the + papers it was judged **against** were read, which the report never stated: + the terminal line said it, once, and the markdown and HTML said nothing. + A subgroup claim usually turns on a table row, and a linearized table has + lost the row. + """ + named = ", ".join(f"`{s}`" for s in flat) + return Disclosure( + key="source_fidelity", + level="warn", + token=SOURCE_FIDELITY_TOKEN, + text=( + f"{len(flat)} of {total} {SOURCE_FIDELITY_TOKEN} — {named}. Tables in " + "those sources were linearized and their figures were invisible to " + "the judge, so a verdict resting on one is weaker than a verdict " + "resting on a table that was read as a table." + ), + short=f"{len(flat)} of {total} {SOURCE_FIDELITY_TOKEN}", + ) + + +def _supplement_verification(results) -> tuple[list[str], list[str]]: + """Supplementary documents read, split into (checked, taken on the filename).""" + seen: dict[str, bool] = {} + for c in results.claims: + for j in c.judgements: + if j.kind in ("supplement", "own_supplement"): + seen[j.source_slug] = seen.get(j.source_slug, False) or j.verified + return sorted(s for s, v in seen.items() if v), sorted(s for s, v in seen.items() if not v) + + +def _supplement_identity(checked: list[str], named: list[str]) -> Disclosure: + """Which supplements were established to belong to their work, and which were not. + + An article is always checked against the reference that names it. A + supplement can be checked only when its own title or DOI names the work it + accompanies — often it does, and the publisher forms usually say + "Supplementary Information for " outright. When it does not, the file + was attached because its NAME carried the reference's tokens, and nothing + read it. Those are different provenances and the report states which. + """ + n = len(checked) + len(named) + parts = [] + if checked: + parts.append( + f"{len(checked)} by {'its' if len(checked) == 1 else 'their'} own title or DOI " + f"naming that work ({', '.join(f'`{s}`' for s in checked)})" + ) + if named: + parts.append( + f"{len(named)} by filename alone, which nothing checked " + f"({', '.join(f'`{s}`' for s in named)}) — a supplement carries its own " + "title and not the title of the article it accompanies, so the identity " + "check that guards every cited source cannot be applied to one" + ) + return Disclosure( + key="supplement_identity", + # only a guess warrants a warning; a checked attachment is information + level="warn" if named else "info", + token=SUPPLEMENT_IDENTITY_TOKEN, + text=( + f"{n} supplementary {'document was' if n == 1 else 'documents were'} read. " + f"This is {SUPPLEMENT_IDENTITY_TOKEN}: " + "; ".join(parts) + "." + ), + short=f"{n} supplementary read · {SUPPLEMENT_IDENTITY_TOKEN}: " + f"{len(checked)} checked, {len(named)} by filename", + rows=tuple(checked + named), + ) + + +def _supplement_coverage() -> Disclosure: + """The coverage audit reads the manuscript, and only the manuscript.""" + return Disclosure( + key="supplement_coverage", + level="info", + token=SUPPLEMENT_COVERAGE_TOKEN, + text=( + f"The coverage audit reads the manuscript alone, so {SUPPLEMENT_COVERAGE_TOKEN}. " + "A reference cited only inside supplementary material is absent from the " + "labels below rather than reported as uncovered, and the ratio is over the " + "main text only." + ), + short=SUPPLEMENT_COVERAGE_TOKEN, + ) + + +def _supplement_headline(claim) -> Disclosure: + """The claim's headline came from an appendix, not the article of record.""" + d = claim.deciding_judgement() + if d.kind == "own_supplement": + where = "this paper's own supplementary material, not its main text" + caveat = "which was supplied by hand and whose contents nobody checked against the claim" + else: + where = f"supplementary material accompanying [{d.ref}], not the article body" + caveat = "which was attached by filename and whose identity nobody confirmed" + return Disclosure( + key="supplement_headline", + level="warn", + token=SUPPLEMENT_HEADLINE_TOKEN, + text=( + f"The headline above is the verdict of `{d.source_slug}` — {where}. So " + f"{SUPPLEMENT_HEADLINE_TOKEN}, {caveat}. Read the per-document breakdown " + "before relying on it." + ), + short=SUPPLEMENT_HEADLINE_TOKEN, + ) + + def run_disclosures(results, manifest=None) -> list[Disclosure]: """Every run-level disclosure this RunResults owes its reader. @@ -331,6 +585,17 @@ def run_disclosures(results, manifest=None) -> list[Disclosure]: if results.truncated: out.append(_truncation(results.truncated)) out.append(_converter(results.converter)) + # an empty dict means the run never recorded this, which is not the same as + # "all of them were read flat" — a 0.4.x file must not grow a warning it + # has no evidence for + if recorded := (getattr(results, "source_converters", None) or {}): + flat = sorted(s for s, c in recorded.items() if c.split()[0] == "pymupdf") + if flat: + out.append(_source_fidelity(flat, len(recorded))) + checked_sup, named_sup = _supplement_verification(results) + supplements = checked_sup + named_sup + if supplements: + out.append(_supplement_identity(checked_sup, named_sup)) coverage = results.coverage or {} if coverage: # occurrences without labels means clean.md was missing while the source @@ -341,6 +606,10 @@ def run_disclosures(results, manifest=None) -> list[Disclosure]: # label-level audit it replaces could not make that mistake if coverage.get("occurrences"): out.append(_coverage_attribution()) + # gated on the audit having produced labels: on a run with no + # citations at all there is no ratio for the blind spot to qualify + if supplements: + out.append(_supplement_coverage()) else: out.append(_coverage_caveat()) # a source whose identity nobody established is a run-level fact: it is not @@ -353,6 +622,10 @@ def run_disclosures(results, manifest=None) -> list[Disclosure]: out.append(_source_identity(unverified, mismatched)) if getattr(manifest, "references_resumed", False): out.append(_references_resumed(len(manifest.entries))) + # the numbering is the join key, so an unconfirmed one outranks + # everything above it — a reader who stops reading should have read this + if not getattr(manifest, "numbering_verified", False): + out.append(_numbering(manifest)) return out @@ -403,18 +676,21 @@ def _sources(claim) -> Disclosure: # a lone dissenter is the whole reason this box exists, so name the split # rather than leaving the reader to compare numbers split = s["supported"] and (s["contradicted"] or s["partial"]) + token = (MULTISOURCE_DOCUMENTS_TOKEN + if any(j.kind != "article" for j in claim.judgements) + else MULTISOURCE_TOKEN) return Disclosure( key="sources", level="warn" if s["contradicted"] else "info", - token=MULTISOURCE_TOKEN, + token=token, text=( - f"{s['total']} {MULTISOURCE_TOKEN} for this claim: {breakdown}." + f"{s['total']} {token} for this claim: {breakdown}." + (" The sources disagree — each verdict below rests only on that " "source's own text." if split else "") ), - short=f"{s['total']} {MULTISOURCE_TOKEN}: {breakdown}", + short=f"{s['total']} {token}: {breakdown}", rows=tuple( - f"[{j.ref}] {j.source_slug} — {j.verdict}" + f"{j.origin} {j.source_slug} — {j.verdict}" + (f" (p{j.source_page})" if j.source_page else "") for j in claim.judgements ), @@ -428,16 +704,58 @@ def judgement_disclosures(j) -> list[Disclosure]: has no co-citations and no breakdown of its own, only the anchor state of the single page it points at. """ - return [ANCHOR[anchor_state(j)]] if j.evidence_image else [] + d = anchor_disclosure(j) + return [d] if d else [] + + +def _no_quote(claim) -> Disclosure: + """This verdict was reached without the manuscript's own sentence. + + Extraction is asked for a verbatim quote every time, so an empty one means + the model did not return it — and the judgement then rests on a paraphrase + that may already have dropped the population, the interval or the hedging + the verdict turns on. Weaker evidence, said so rather than left to be + inferred from a missing blockquote: "no quote" and "quote identical to the + paraphrase" look the same on the page otherwise. + """ + return Disclosure( + key="no_quote", + level="warn", + token=NO_QUOTE_TOKEN, + text=( + f"{NO_QUOTE_TOKEN} — extraction returned no verbatim sentence for this " + "claim, so the source was checked against the short paraphrase above. " + "Any scope, interval or hedging the paraphrase dropped was not judged." + ), + short=NO_QUOTE_TOKEN, + ) + +def claim_disclosures(claim, manifest=None) -> list[Disclosure]: + """Every claim-level disclosure this ClaimResult owes its reader. -def claim_disclosures(claim) -> list[Disclosure]: - """Every claim-level disclosure this ClaimResult owes its reader.""" + `manifest` is optional because most callers have no reason to hold one, and + every disclosure that does not depend on it must keep firing without it. + `report.py` binds it once so the templates keep their one-argument call. + """ out: list[Disclosure] = [] if claim.is_multi_source(): out.append(_sources(claim)) if claim.unjudged_refs: out.append(_unjudged(claim)) - if claim.evidence_image: - out.append(ANCHOR[anchor_state(claim)]) + # only where a judgement actually happened: nothing read an unretrieved + # source, so the quote changed nothing there and the notice would land on + # every row of the gap register until readers stopped seeing it + if not claim.quote and claim.verdict in JUDGMENT_VERDICTS: + out.append(_no_quote(claim)) + # the headline is an appendix's word, not the article's. A run-level note + # that supplements were read does not tell a reader that THIS verdict is + # one of them, and the headline is what most readers act on + _d = claim.deciding_judgement() + if _d is not None and _d.kind != "article" and claim.verdict in JUDGMENT_VERDICTS: + out.append(_supplement_headline(claim)) + if manifest is not None and any(manifest.label_is_doubtful(r) for r in claim.refs): + out.append(_claim_numbering(claim, manifest)) + if (d := anchor_disclosure(claim)) is not None: + out.append(d) return out diff --git a/src/papertrace/ingest/__init__.py b/src/papertrace/ingest/__init__.py index 9ba12c7..058e863 100644 --- a/src/papertrace/ingest/__init__.py +++ b/src/papertrace/ingest/__init__.py @@ -18,10 +18,16 @@ from pathlib import Path -from ..models import Block, SourceMap -from .pymupdf_ import ingest_blocks_pymupdf, references_section, references_span +from ..models import Block, SourceMap, manuscript_fingerprint +from .pymupdf_ import ( + declared_title, + ingest_blocks_pymupdf, + references_section, + references_span, +) -__all__ = ["ingest_pdf", "references_section", "references_span", "available_backends"] +__all__ = ["ingest_pdf", "references_section", "references_span", + "available_backends", "resolve_backend"] def _docling_available() -> bool: @@ -37,18 +43,32 @@ def available_backends() -> list[str]: return ["docling", "pymupdf"] if _docling_available() else ["pymupdf"] -def ingest_pdf(pdf_path: Path, out_dir: Path, backend: str = "auto") -> SourceMap: - """Convert one PDF with the chosen backend and write the three outputs.""" +def resolve_backend(backend: str) -> str: + """Turn a backend *request* into the backend that will actually run. + + Shared because two readers need it and a second copy of this rule is how + they would drift: `ingest_pdf` dispatches on it, and `check._stale_ingest` + compares it against the converter a source map records — where `"auto"` is + not a converter name, so comparing the request literally would report every + existing map as stale and re-ingest the whole reference list every run. + + NOT a silent downgrade to pymupdf for an unrecognised value. Treating one + as flat text is what hid a caller passing a Typer OptionInfo instead of a + backend name: the run ingested as flat text and then told the user to + install a layout backend they already had. + """ if backend == "auto": - backend = "docling" if _docling_available() else "pymupdf" + return "docling" if _docling_available() else "pymupdf" if backend not in ("docling", "pymupdf"): - # NOT a silent downgrade to pymupdf. Treating every unrecognised value - # as flat text is what hid a caller passing a Typer OptionInfo instead - # of a backend name: the run ingested as flat text and then told the - # user to install a layout backend they already had. raise ValueError( f"unknown ingest backend {backend!r} — expected 'auto', 'docling' or 'pymupdf'" ) + return backend + + +def ingest_pdf(pdf_path: Path, out_dir: Path, backend: str = "auto") -> SourceMap: + """Convert one PDF with the chosen backend and write the three outputs.""" + backend = resolve_backend(backend) if backend == "docling": if not _docling_available(): raise RuntimeError( @@ -64,7 +84,20 @@ def ingest_pdf(pdf_path: Path, out_dir: Path, backend: str = "auto") -> SourceMa pages, blocks = ingest_blocks_pymupdf(pdf_path) converter = "pymupdf" - smap = SourceMap(doc=pdf_path.name, pages=pages, converter=converter, blocks=blocks) + # content identity, because `doc` is not one: a cited source is stored as + # `<slug>.pdf`, so every source map in a case names a different paper the + # same way, and a directory named after a slug was trusted to hold whatever + # it held + smap = SourceMap( + doc=pdf_path.name, + pages=pages, + converter=converter, + blocks=blocks, + source_sha256=manuscript_fingerprint(pdf_path), + # what the file says its title is — the layout's first heading is the + # article-type banner often enough that it cannot be the first choice + declared_title=declared_title(pdf_path), + ) write_outputs(smap, out_dir) return smap diff --git a/src/papertrace/ingest/pymupdf_.py b/src/papertrace/ingest/pymupdf_.py index 1c475ef..9753b8b 100644 --- a/src/papertrace/ingest/pymupdf_.py +++ b/src/papertrace/ingest/pymupdf_.py @@ -16,11 +16,29 @@ except ImportError: # pragma: no cover - older PyMuPDF exposes only `fitz` import fitz -from ..models import Block, SourceMap, is_references_heading +from ..models import Block, SourceMap, is_references_heading, looks_like_reference _HEADING_MAX_LEN = 120 +def declared_title(pdf_path: Path) -> str: + """The title the PDF declares in its own metadata, verbatim, or "". + + Read here because this module owns the pymupdf import, and read for *both* + backends because it is a property of the file rather than of the converter: + docling does not expose it, and on the one paper of seven whose metadata is + empty docling emitted no `title` item either. + + Never raises: a metadata dictionary this tool cannot read is one more + unknown, and the caller already has a fallback for it. + """ + try: + with fitz.open(pdf_path) as doc: + return " ".join(((doc.metadata or {}).get("title") or "").split()) + except Exception: # noqa: BLE001 — an unreadable title is not a failed ingest + return "" + + def _block_text(raw: dict) -> tuple[str, float]: """Join a PyMuPDF text block; return (text, max font size).""" parts: list[str] = [] @@ -88,6 +106,17 @@ def ingest_blocks_pymupdf(pdf_path: Path) -> tuple[int, list[Block]]: _MIN_RESUME_RUN = 2 +def _mostly_references(run) -> bool: + """Does this run of same-typed blocks read as a bibliography? + + Half, not all. Requiring every entry would drop a real continuation over one + bare-URL entry; requiring one would let a single dated line drag a whole + section of back matter in behind it. + """ + hits = sum(1 for b in run if looks_like_reference(b.text)) + return hits * 2 >= len(run) + + def references_span(smap: SourceMap) -> tuple[str, bool]: """Reference-list text, and whether it was resumed across a section break. @@ -104,10 +133,19 @@ def references_span(smap: SourceMap) -> tuple[str, bool]: intervening section. That is worth surfacing rather than hiding: a real pre-proof put refs 1-9 on page 7, `Declaration of interests` next, then refs 10-15 on page 8, and stopping at the first header lost six sources without - saying so. Only reference-shaped runs are collected, so the prose of the - intervening section never enters the list — which matters because - `_parse_bulleted` appends a non-bullet line to the *previous* entry, so a - stray paragraph corrupts a reference rather than merely adding noise. + saying so. The prose of the intervening section never enters the list, which + matters because `_parse_bulleted` appends a non-bullet line to the *previous* + entry, so a stray paragraph corrupts a reference rather than merely adding + noise. + + A run has to look like references, not merely share their block type. This + docstring used to claim that and it was false — the only test was the type, + so three `list` blocks under a `TABLE TITLES` heading became references + 44-46 of a 43-reference paper, and the resolver title-searched the paper's + own table captions into table-component DOIs belonging to other papers. + The test is applied to the run rather than to each entry: a genuine + continuation can hold a bare URL entry with no year, and rejecting the whole + run over it would undo the fix above. """ blocks = smap.blocks start = next( @@ -143,7 +181,7 @@ def references_span(smap: SourceMap) -> tuple[str, bool]: j = k while j < len(rest) and rest[j].type == entry_type: j += 1 - if j - k >= _MIN_RESUME_RUN: + if j - k >= _MIN_RESUME_RUN and _mostly_references(rest[k:j]): out.extend(b.text for b in rest[k:j]) resumed = True k = j diff --git a/src/papertrace/models.py b/src/papertrace/models.py index 6ca7d52..906f80a 100644 --- a/src/papertrace/models.py +++ b/src/papertrace/models.py @@ -11,7 +11,7 @@ import hashlib import json import re -from dataclasses import asdict, dataclass, field +from dataclasses import asdict, dataclass, field, fields from pathlib import Path # --------------------------------------------------------------------------- @@ -46,6 +46,191 @@ def is_references_heading(block_type: str, text: str) -> bool: return bool(_REFS_HEADING_EXACT.match(text)) +# What a PDF declares as its title but is not one. Three bounded rules, each +# from an observed shape, not a list that grows with every journal: +# a banner or placeholder (too few words), a producer's filename, and a +# producer's prefix. `Microsoft Word - Manuscript revised final clean.docx` is +# the shape that matters — a Word-produced manuscript is this tool's main case, +# and four confident words describing no paper would let the identity check +# report a mismatch and discard a good deposit. +_TITLE_FILE_SUFFIX = re.compile(r"\.(docx?|tex|dvi|indd|pdf|rtf|odt|pages)$", re.I) +_TITLE_PRODUCER = re.compile(r"^\s*microsoft\s+(word|powerpoint)\s*-", re.I) + + +def _declared_title_is_usable(title: str) -> bool: + title = (title or "").strip() + return ( + len(title.split()) >= 3 + and not _TITLE_FILE_SUFFIX.search(title) + and not _TITLE_PRODUCER.match(title) + ) + + +_TITLE_STOPWORDS = frozenset( + {"commun", "nature", "science", "journal", "lancet", "article", + "elsevier", "springer", "wiley", "volume", "press", "https"} +) + +_URL_RE = re.compile(r"(?:https?://|www\.)\S+", re.I) + + +def _title_tokens(raw: str) -> set[str]: + """The reference's own distinctive words — URLs removed first. + + A URL is not part of a title, and a *tracking parameter* least of all: + `?utm_source=chatgpt.com` on a cited news page contributed `chatgpt` and + `source` to this set, and the wrong paper Crossref returned was an + editorial about ChatGPT. Path segments do the same from the other side, + inflating the denominator with `firstmedical`, `assuranceprogram` and + `publications` — words no first page will carry, so they dilute the ratio + the check is measured on. + """ + return set(re.findall(r"[a-z]{5,}", _URL_RE.sub(" ", raw).lower())) - _TITLE_STOPWORDS + + +# Four distinct words, not three. The observed false positive cleared the 0.35 +# ratio on `artificial`, `intelligence` and `medical` — three words that are the +# subject of most papers in this field, so no stopword list can retire them +# without rejecting correct matches. Falling below the floor yields +# `unverifiable`, never `mismatch`: too few words to tell is not evidence of a +# different paper, and a `mismatch` would discard a possibly-correct download. +_TITLE_MIN_MATCHES = 4 + + +def titles_match(a: str, b: str) -> bool | None: + """Do these two title strings name the same work? True, False, or None. + + **None means "cannot tell"**, and it is a third answer rather than a + collapsed False for a measured reason: `paper_title` is a heuristic over the + first blocks of a page, and on a seven-paper spread the block it offers was + an article-type banner four times — `CLINICAL GUIDELINE`, `RESEARCH + ARTICLE`. Two comparable words are not evidence of a different paper, and a + confident False there discards a good deposit or a correct Europe PMC record. + + Lives here because three readers need it and none may import another: + `refs` asks whether a Crossref deposit belongs to this paper, `scout` asks + the same of a Europe PMC record, and `refs._title_check_text` asks it of a + downloaded first page. The rule was `refs`-private until the second reader + appeared; a copy in `scout` is the defect this module's other shared rules + exist to prevent. + """ + ta, tb = _title_tokens(a), _title_tokens(b) + if min(len(ta), len(tb)) < _TITLE_MIN_MATCHES: + return None + return len(ta & tb) / min(len(ta), len(tb)) >= 0.5 + + +def paper_title(smap) -> str: + """Best-effort title of the paper a source map describes. + + What the PDF declares about itself first, then the layout: the first + substantial section header, else the first substantial text block. Third + rule to live here for the reason the two above it do — `scout` needs it to + identify the paper in Europe PMC and `refs` needs it to check that the + Crossref record behind a DOI is this paper, and neither module may import + the other. + + The declaration comes first because the layout is measurably worse at this: + on a seven-paper spread the first heading was the article-type banner every + time it was wrong, and a banner identifies nothing. It is not trusted + blindly either — an author's PDF declares its Word filename — so a + declaration that is not title-shaped is passed over for the layout. + + Still best-effort, and treated as such by both callers: this is why the + identity check it feeds has an "unverifiable" answer and never reads a thin + title as a mismatch. + """ + if _declared_title_is_usable(getattr(smap, "declared_title", "")): + return " ".join(smap.declared_title.split())[:220] + for b in smap.blocks: + if b.type == "sectionheader" and len(b.text.strip()) >= 15: + return " ".join(b.text.split())[:220] + for b in smap.blocks: + if b.type == "text" and len(b.text.strip()) >= 25: + return " ".join(b.text.split())[:220] + return "" + + +# What marks a line as a bibliographic reference rather than back matter. +# Deliberately three cheap structural marks and nothing else — the question is +# only "is this a citable work at all", not "is this a good reference". +_REF_YEAR = re.compile(r"\b(?:19|20)\d{2}\b") +_REF_DOI = re.compile(r"10\.\d{4,9}/\S", re.I) +_REF_ARXIV = re.compile(r"arxiv[:\s]*\d{4}\.\d{4,5}", re.I) +# An author list, in the two styles that actually turn up: `M.A. Slabaugh` and +# `Slabaugh MA`. Two names, not one — a single match is easy to hit by accident. +_REF_AUTHORS = re.compile( + r"\b[A-Z]\.(?:\s*[A-Z]\.)*\s*[A-Z][a-z]+" # M.A. Slabaugh + r"|\b[A-Z][a-z]+\s+[A-Z]{1,3}\b" # Slabaugh MA +) + + +def looks_like_reference(text: str) -> bool: + """Could this line be a cited work? A year, a DOI or an arXiv id. + + Lives here rather than in `refs.py` or `ingest/` for the same reason + `is_references_heading` does: two readers need the rule, they cannot import + each other, and each keeping its own copy is a defect this codebase has + already shipped once. + + The bar is deliberately low. This is not a quality test on a reference — it + is the difference between a cited work and the paper's own back matter. + `Table 1. Dataset characteristics` carries none of the three, and three of + those became references 44-46 of a 43-reference paper, were title-searched + against Crossref, and came back as table-component DOIs belonging to other + papers. + + Being wrong in the permissive direction is the cheap error: a stray line + that sneaks through is one bad entry in a manifest. Being wrong in the + strict direction drops a real reference from the audit entirely, and that + failure is silent. + """ + text = text or "" + if _REF_YEAR.search(text) or _REF_DOI.search(text) or _REF_ARXIV.search(text): + return True + # An author list, for the references that arrive truncated. Two real + # references in one audit reached the resolver as authors plus half a title + # and nothing else — no journal, no year — and Crossref found both correct + # DOIs from exactly that. A year-only test threw them away. + return len(_REF_AUTHORS.findall(text)) >= 2 + + +# How a citation marker is written. The THIRD rule to live here for the reason +# `is_references_heading` and `looks_like_reference` do: two modules need it and +# neither may import the other. `check.py` reads these to audit coverage and +# `refs.py` reads them to learn which references the manuscript actually cites — +# and those two readings are only worth comparing if they are the same rule. +_LABEL_GROUP = re.compile(r"\[(\d{1,3}(?:\s*[,–—-]\s*\d{1,3})*)\]") + + +def _expand_label_group(group: str) -> set[str]: + labels: set[str] = set() + for part in re.split(r"\s*,\s*", group): + m = re.match(r"^(\d{1,3})\s*[–—-]\s*(\d{1,3})$", part.strip()) + if m: + lo, hi = int(m.group(1)), int(m.group(2)) + if lo <= hi and hi - lo <= 50: + labels.update(str(n) for n in range(lo, hi + 1)) + elif part.strip().isdigit(): + labels.add(part.strip()) + return labels + + +def citation_labels(text: str) -> set[str]: + """Every bracketed numeric citation label in this text: `[3]`, `[7,8]`, `[11-13]`. + + Bracketed numeric styles only — author-year and bare superscripts are not + read, here or anywhere else in the tool, and an empty set from a paper that + plainly cites things means the style was not recognised rather than that + nothing was cited. `check.citation_labels_in_text` wraps this to exclude the + reference list; callers that want the whole document use this directly. + """ + labels: set[str] = set() + for m in _LABEL_GROUP.finditer(text or ""): + labels.update(_expand_label_group(m.group(1))) + return labels + + @dataclass class Block: """One layout block of a source document, with page-level provenance. @@ -73,12 +258,27 @@ class SourceMap: pages: int converter: str = "pymupdf" # which ingest backend produced this map blocks: list[Block] = field(default_factory=list) + # sha256 of the PDF this map was built from. `doc` cannot serve: a cited + # source is stored as `<slug>.pdf`, so every source map in a case says the + # same thing about a different paper. Without a content identity, a + # directory named after a slug is trusted to hold whatever it holds — and + # slugs are not eternal, so a re-run can read the previous occupant. + source_sha256: str | None = None + # the title the PDF declares about itself (XMP / Info dictionary), verbatim + # and unjudged. Publishers populate it and the layout does not: measured on + # seven papers, the first heading is the article-type banner — `CLINICAL + # GUIDELINE`, `RESEARCH ARTICLE`, `Journal Pre-proofs`, `Editorial` — while + # the metadata carried the exact title for six of the seven. Recording it + # raw is provenance; deciding whether it is usable is `paper_title`'s job. + declared_title: str = "" def to_json(self, path: Path) -> None: payload = { "doc": self.doc, "pages": self.pages, "converter": self.converter, + "source_sha256": self.source_sha256, + "declared_title": self.declared_title, "blocks": [ {**asdict(b), "bbox": list(b.bbox), "text_preview": b.preview} for b in self.blocks ], @@ -104,6 +304,10 @@ def from_json(cls, path: Path) -> SourceMap: pages=data["pages"], converter=data.get("converter", "pymupdf"), blocks=blocks, + # absent on maps written before content hashing — None means + # "unknown", never "matches", so a reader must re-establish it + source_sha256=data.get("source_sha256"), + declared_title=data.get("declared_title", ""), ) def find(self, block_id: str) -> Block | None: @@ -130,6 +334,60 @@ def manuscript_fingerprint(path: Path) -> str: return h.hexdigest() +# What a judgeable file *is*, as distinct from which reference it answers for. +# `article` is the cited work itself; `supplement` accompanies one; the audited +# paper's own supplementary material is neither — it answers for no citation +# label at all, which is why it cannot just be a `supplement` with an empty ref. +DOCUMENT_KINDS = ("article", "supplement", "own_supplement") + + +@dataclass +class Supplement: + """One supplementary file the user handed over, and the slug it is read as. + + The slug is derived from the FILE STEM, never from an ordinal position in + the folder. `-suppl1`/`-suppl2` assigned in sorted order is the same defect + CLAUDE.md rejects for citation occurrences: remove one file and every id + after it silently shifts, so a re-run points last run's verdicts and + evidence crops at a different PDF. + + There is no `title_check`. Every article this tool accepts is checked + against the reference that names it; a supplement's own title does not + match its parent's, so that check cannot apply and is not faked. A + supplement is attached on a filename match alone — the weakest provenance + anything here carries — and the report says so rather than letting it pass + as an equal of a verified source. + """ + + slug: str + pdf_path: str + # did anything establish that this file belongs to the work it is attached + # to? True when its own title or DOI named that work; False when it was + # attached because its FILENAME carried the reference's tokens, which is a + # guess nobody checked. The report tells the two apart rather than warning + # about both equally. + verified: bool = False + + +@dataclass(frozen=True) +class Document: + """One file a claim can be judged against, with the reference it answers for. + + The join key everything downstream already uses is the *slug*: + `ingest/<slug>/`, `sources_resolved/<slug>.pdf`, `SourceJudgement.source_slug` + and `RunResults.source_converters` all key off it. Four call sites used to + hand-roll `next(e for e in manifest.entries if e.slug == slug)`, which can + only ever find an article. Resolving through here instead means none of them + has to learn that supplements exist. + """ + + slug: str + pdf_path: str | None + ref_num: str # the citation label this document answers for; "" for the paper's own + kind: str # one of DOCUMENT_KINDS + parent_slug: str | None # the article this accompanies, or None + + @dataclass class RefEntry: num: str # citation label as used in the manuscript, e.g. "14" @@ -151,6 +409,30 @@ class RefEntry: # A single nullable "did the check fail" flag conflated the first two, so a # scanned PDF read as a successful match. title_check: str | None = None + # supplementary files the user supplied for THIS reference. Only ever + # non-empty when the reference itself is available: a supplement with no + # article behind it is set aside, because judging a claim against an + # appendix while calling it the cited source is the laundering this + # codebase exists to prevent. + supplements: list[Supplement] = field(default_factory=list) + + +def _ref_entry_from(d: dict) -> RefEntry: + """One manifest entry, hydrated — the counterpart of `_claim_from` below. + + Two things a bare `RefEntry(**d)` got wrong. It handed `supplements` back as + a list of plain dicts, because nothing in this manifest was a nested + dataclass until now and `asdict` flattens on the way out. And it raised + `TypeError` on any key it did not declare, so a manifest written by a NEWER + papertrace killed an older one outright instead of ignoring what it could + not use — the opposite of how every other reader here defaults forward. + """ + known = {f.name for f in fields(RefEntry)} + kwargs = {k: v for k, v in d.items() if k in known and k != "supplements"} + return RefEntry( + **kwargs, + supplements=[Supplement(**s) for s in d.get("supplements", [])], + ) @dataclass @@ -167,6 +449,81 @@ class RefManifest: # reader has to be able to check it, because the alternative failure is # silent: a list parsed short simply reports fewer references. references_resumed: bool = False + # Which reading of the reference list this manifest holds, and whether + # anything checked it. `parse_references` was the only stage that could not + # report its own failure, and the label is the join key — a numbering off by + # one judges every later claim against the wrong paper, silently. The + # defaults are the honest reading of an older manifest: the parser's list, + # never checked. + reference_source: str = "parsed" # crossref | parsed + numbering_verified: bool = False + numbering_note: str = "" + # the first label from which the numbering is in doubt, or None when it is + # not in doubt. 1 means "from the very start" — used when there was only one + # candidate, because a single unchecked reading gives no evidence about + # *where* it went wrong + unverified_from: int | None = None + # the AUDITED paper's own supplementary material. Not a RefEntry: it answers + # for no citation label, and putting it in `entries` would inflate + # `refs_total` and let `_slug_for_ref` hand it to a claim citing a number. + manuscript_supplements: list[Supplement] = field(default_factory=list) + + def document(self, slug: str) -> Document | None: + """The judgeable file this slug names, article or supplement, or None.""" + for e in self.entries: + if e.slug == slug: + return Document(slug, e.pdf_path, e.num, "article", None) + for s in e.supplements: + if s.slug == slug: + return Document(slug, s.pdf_path, e.num, "supplement", e.slug) + for s in self.manuscript_supplements: + if s.slug == slug: + return Document(slug, s.pdf_path, "", "own_supplement", None) + return None + + def documents(self) -> list[Document]: + """Every judgeable file, once, in reading order. + + A caller walking `entries` sees only articles — which is how the reports + came to disclose how each *source* was read while saying nothing at all + about the supplements judged beside them. + """ + out: list[Document] = [] + for e in self.entries: + if e.slug: + out.append(Document(e.slug, e.pdf_path, e.num, "article", None)) + out += [ + Document(s.slug, s.pdf_path, e.num, "supplement", e.slug) + for s in e.supplements + ] + out += [ + Document(s.slug, s.pdf_path, "", "own_supplement", None) + for s in self.manuscript_supplements + ] + return out + + def label_is_doubtful(self, label: str) -> bool: + """Does a claim citing this label rest on a numbering nobody confirmed? + + An unconfirmed numbering with **no recorded scope** puts every label in + doubt, rather than none. `unverified_from is None` used to answer False + for every label while the run-level disclosure rendered "every entry is + affected" for the same reason — the report asserted that every entry was + suspect and marked no claim suspect, so a reader acting on a single + verdict was told nothing. Two shapes reach that state: a manifest + written before the list was reconciled at all, and two readings that + agree entry for entry with no arbiter to confirm either. Neither + establishes *which* entries are wrong, and unknown scope has to read the + same way in both places. + """ + if self.numbering_verified: + return False + if self.unverified_from is None: + return True + try: + return int(label) >= self.unverified_from + except (TypeError, ValueError): + return False @property def retrieved(self) -> list[RefEntry]: @@ -181,6 +538,10 @@ def to_json(self, path: Path) -> None: "manuscript": self.manuscript, "manuscript_sha256": self.manuscript_sha256, "references_resumed": self.references_resumed, + "reference_source": self.reference_source, + "numbering_verified": self.numbering_verified, + "numbering_note": self.numbering_note, + "unverified_from": self.unverified_from, "summary": { "total": len(self.entries), "available": len(self.retrieved), @@ -189,6 +550,7 @@ def to_json(self, path: Path) -> None: }, }, "entries": [asdict(e) for e in self.entries], + "manuscript_supplements": [asdict(s) for s in self.manuscript_supplements], } path.write_text(json.dumps(payload, indent=2, ensure_ascii=False)) @@ -197,10 +559,19 @@ def from_json(cls, path: Path) -> RefManifest: data = json.loads(path.read_text()) return cls( manuscript=data["manuscript"], - entries=[RefEntry(**e) for e in data["entries"]], + entries=[_ref_entry_from(e) for e in data["entries"]], + manuscript_supplements=[ + Supplement(**s) for s in data.get("manuscript_supplements", []) + ], manuscript_sha256=data.get("manuscript_sha256"), # .get: a manifest written before this field must still load references_resumed=bool(data.get("references_resumed", False)), + # an older manifest carries the parser's list and never checked it, + # which is exactly what these defaults say + reference_source=data.get("reference_source", "parsed"), + numbering_verified=bool(data.get("numbering_verified", False)), + numbering_note=data.get("numbering_note", ""), + unverified_from=data.get("unverified_from"), ) @@ -244,6 +615,15 @@ class SourceJudgement: source_slug: str ref: str # the citation label this source answers for, e.g. "3" + # which of DOCUMENT_KINDS this document is. Stored rather than looked up in + # the manifest: the templates are handed `results` alone and + # `run_disclosures` takes the manifest optionally, so a reader with only + # `results.json` must still be able to tell an appendix from an article. + kind: str = "article" + # for a supplement: did anything establish it belongs to the work it was + # attached to? Carried here for the reason `kind` is — the reader of a + # results.json alone has no manifest to consult. + verified: bool = False verdict: str = "unchecked" # one of VERDICTS note: str = "" source_page: int | None = None @@ -256,6 +636,20 @@ class SourceJudgement: def label(self) -> str: return VERDICT_LABEL.get(self.verdict, self.verdict.upper()) + @property + def origin(self) -> str: + """Where this verdict came from, in the reader's terms. + + One property rather than `cited as [{{ j.ref }}]` written out in three + templates: the paper's own supplement answers for no label at all, and + every one of them would otherwise have rendered `cited as []`. + """ + if self.kind == "own_supplement": + return "this paper's own supplement" + if self.kind == "supplement": + return f"supplement to [{self.ref}]" + return f"cited as [{self.ref}]" + # how adverse each judgement is, for picking a claim's headline. A single cited # source contradicting the claim is the finding a reviewer needs, so it wins @@ -267,8 +661,22 @@ def label(self) -> str: @dataclass class ClaimResult: id: int - claim: str # the claim, quoted or tightly paraphrased - location: str # where in the manuscript, e.g. "Methods §2" + claim: str # the claim, tightly paraphrased — what a headline reads well + # the manuscript's own sentence, verbatim. Empty when the model did not + # return one: never back-filled from `claim`, which would reinstate exactly + # the compression the quote exists to remove + quote: str = "" + # the claim points at the AUDITED paper's own supplementary material — + # "Table S3", "eFigure 2", "Supplementary Methods". Not a citation: there is + # no label, so it cannot travel in `refs`, and a statement whose evidence the + # paper located precisely is not an assertion made without one. + own_supplement: bool = False + location: str = "" # where in the manuscript, e.g. "Methods §2" + # ids of the citation occurrences this claim was extracted from, as resolved + # from the `ctx_NNNN` labels the inventory offered the extractor. Empty when + # the model named none, or named one that was not in the inventory: never + # back-filled by guessing which occurrence of the label it must have meant. + ctx_ids: list[str] = field(default_factory=list) refs: list[str] = field(default_factory=list) # citation labels, e.g. ["14"] verdict: str = "not_retrieved" # one of VERDICTS note: str = "" # one/two-sentence finding @@ -321,6 +729,27 @@ def headline_verdict(self) -> str: return "not_addressed" return "unchecked" + def headline_qualifier(self) -> str: + """What the headline actually ranged over, for rendering beside it. + + The headline is one source's verdict. On a multi-source claim it reads + as a statement about the claim, and a compound sentence may legitimately + draw different parts from different references — so `❌ CONTRADICTED` + with one dissenter of four overstates by exactly the amount a reader + cannot see from the status line alone. + + Empty when there is nothing to qualify: one source means the headline + *is* the claim's verdict, and no judgements means nothing was ranked, so + naming a comparison that never happened would be its own invention. + """ + if not self.is_multi_source(): + return "" + # "cited sources" is false the moment a supplement is among them: one + # cited work read as two documents is not two cited works, and the + # count would overstate how many independent papers were consulted. + noun = "documents" if any(j.kind != "article" for j in self.judgements) else "cited sources" + return f"most adverse of {len(self.judgements)} {noun}" + def deciding_judgement(self) -> SourceJudgement | None: """The judgement the headline came from — whose page the crop shows.""" want = self.headline_verdict() @@ -364,6 +793,7 @@ class UncitedClaim: id: int claim: str + quote: str = "" # the manuscript's own sentence — what the reviewer judges location: str = "" @@ -386,6 +816,12 @@ class RunResults: refs_total: int = 0 refs_available: int = 0 converter: str = "pymupdf" # ingest backend used for the manuscript + # slug -> the converter that read THAT cited source. Separate from + # `converter` above, which is the manuscript's: the two can differ, and a + # verdict resting on a linearized table is weaker than one resting on the + # table. An EMPTY dict means the run never recorded this (every 0.4.x + # file), which is not the same as "all of them were read flat". + source_converters: dict[str, str] = field(default_factory=dict) claims: list[ClaimResult] = field(default_factory=list) uncited: list[UncitedClaim] = field(default_factory=list) # deterministic citation-label audit: which [N] labels appear in the text, @@ -415,6 +851,7 @@ def to_json(self, path: Path) -> None: "date": self.date, "refs": {"total": self.refs_total, "available": self.refs_available}, "converter": self.converter, + "source_converters": self.source_converters, "counts": self.counts(), "claims": [asdict(c) for c in self.claims], "uncited": [asdict(u) for u in self.uncited], @@ -433,6 +870,7 @@ def from_json(cls, path: Path) -> RunResults: refs_total=data.get("refs", {}).get("total", 0), refs_available=data.get("refs", {}).get("available", 0), converter=data.get("converter", "pymupdf"), + source_converters=data.get("source_converters", {}), claims=[_claim_from(c) for c in data["claims"]], uncited=[UncitedClaim(**u) for u in data.get("uncited", [])], coverage=data.get("coverage", {}), @@ -462,20 +900,38 @@ class ScoutResults: """Post-publication scan around one paper. `newer` holds what appeared after the paper (citing articles + later - keyword hits); `overlooked` holds what existed by the paper's year but is - absent from its reference list. Both are candidates for the user's - judgement — search-based, so absence from these lists proves nothing. + keyword hits); `overlooked` holds what was in print *before* the paper's + year and is absent from its reference list. Both are candidates for the + user's judgement — search-based, so absence from these lists proves nothing. A non-empty `error` means the scan soft-failed and may be incomplete. + + `same_year` is the third register, and it is deliberately not folded into + either neighbour. A paper from the manuscript's own year may have appeared + after submission, so "existed but uncited" holds it to a standard no author + can meet — on one real 2026 manuscript all fifteen overlooked candidates + were from 2026. It is not `newer` either, since it did not appear after. + Dropping it would lose a real finding: a paper published early in the same + year is exactly what a reviewer might legitimately raise. """ paper_title: str = "" paper_doi: str = "" paper_year: int | None = None resolved_via: str = "" # "doi" | "title" | "" + # Did anyone establish that the record found is this paper? + # "confirmed" | "unverified" | "mismatch" | "" (nothing resolved). + # `resolved_via` cannot answer it: `_resolve_paper` records "doi" whenever a + # DOI is supplied, and `run` reads the DOI off page 1, so the "wrong paper?" + # warning stopped firing exactly when the DOI became a guess. The provenance + # is not recoverable here and is the wrong question anyway — the record's own + # title is comparable with the paper's. + paper_identity: str = "" query: str = "" # the keyword query used for the related search date: str = "" newer: list[ScoutHit] = field(default_factory=list) overlooked: list[ScoutHit] = field(default_factory=list) + # the paper's own year — neither "since" nor "should have known" + same_year: list[ScoutHit] = field(default_factory=list) error: str = "" def to_json(self, path: Path) -> None: @@ -485,12 +941,18 @@ def to_json(self, path: Path) -> None: "doi": self.paper_doi, "year": self.paper_year, "resolved_via": self.resolved_via, + "identity": self.paper_identity, }, "query": self.query, "date": self.date, - "counts": {"newer": len(self.newer), "overlooked": len(self.overlooked)}, + "counts": { + "newer": len(self.newer), + "overlooked": len(self.overlooked), + "same_year": len(self.same_year), + }, "newer": [asdict(h) for h in self.newer], "overlooked": [asdict(h) for h in self.overlooked], + "same_year": [asdict(h) for h in self.same_year], "error": self.error, } path.write_text(json.dumps(payload, indent=2, ensure_ascii=False)) @@ -504,9 +966,14 @@ def from_json(cls, path: Path) -> ScoutResults: paper_doi=paper.get("doi", ""), paper_year=paper.get("year"), resolved_via=paper.get("resolved_via", ""), + # absent on scout.json written before the check existed: "" reads as + # not recorded, never as confirmed + paper_identity=paper.get("identity", ""), query=data.get("query", ""), date=data.get("date", ""), newer=[ScoutHit(**h) for h in data.get("newer", [])], overlooked=[ScoutHit(**h) for h in data.get("overlooked", [])], + # .get: a scout.json written before the third register still loads + same_year=[ScoutHit(**h) for h in data.get("same_year", [])], error=data.get("error", ""), ) diff --git a/src/papertrace/refs.py b/src/papertrace/refs.py index fb02177..b1234c5 100644 --- a/src/papertrace/refs.py +++ b/src/papertrace/refs.py @@ -10,12 +10,22 @@ import re from collections.abc import Callable +from dataclasses import dataclass, field from pathlib import Path +from typing import NamedTuple import httpx from . import __version__ -from .models import RefEntry +from .models import ( + _TITLE_MIN_MATCHES, + _URL_RE, + RefEntry, + Supplement, + _title_tokens, + looks_like_reference, + titles_match, +) # Two user agents on purpose. The contact address is sent ONLY to the services # that ask for one — Unpaywall requires it, Crossref's polite pool uses it. One @@ -106,15 +116,31 @@ def parse_references(text: str) -> list[RefEntry]: entries.append(e) if not entries: entries = _parse_bulleted(text) - return entries + # two entries sharing a slug share a download path — see _unique_slugs + return _unique_slugs(entries) def _parse_bulleted(text: str) -> list[RefEntry]: - """Fallback for lists whose numerals the converter stripped. - - docling flattens some journals' numbered hanging-indent reference lists - (e.g. Nature-family layouts) into plain bullets — number the bullets - sequentially by document order instead of giving up with zero entries. + """Fallback for lists the converter flattened into bullets. + + **The printed numeral is the label when there is one.** Numbering the + bullets `1..N` by document order was the single worst bug this module has + had: a running header interrupting reference [14] at a page break made + docling emit two bullets, every later label shifted by one, and 27 of 41 + references on a real paper were judged against the wrong papers. The numeral + was sitting at the front of the text the whole time — `_strip_printed_numeral` + captured it and threw it away one line before the label was invented. + + A bullet with no numeral, following one that has it, is the tail of an entry + the converter split; it is joined back on rather than becoming a reference of + its own. So a page-break split now heals, and a *merge* — two references in + one bullet — leaves a gap in the labels, which `_covers` reports as + unconfirmed instead of silently shifting everything after it. + + Sequential numbering survives only for lists that genuinely carry no + numerals (Nature-family layouts, where the converter really did strip them): + a guess, but the only reading available, and the reconciler marks it + unverified. """ items: list[str] = [] for line in text.splitlines(): @@ -123,11 +149,551 @@ def _parse_bulleted(text: str) -> list[RefEntry]: items.append(s[2:].strip()) elif items and s: items[-1] += " " + s # wrapped continuation of the previous entry - return [ - _entry(str(i), re.sub(r"\s+", " ", raw).strip()) - for i, raw in enumerate(items, 1) - if raw.strip() + items = [re.sub(r"\s+", " ", raw).strip() for raw in items if raw.strip()] + + numerals = [_leading_numeral(raw) for raw in items] + if _usable_printed_numerals(numerals): + entries: list[RefEntry] = [] + for num, raw in zip(numerals, items, strict=True): + if num is None: + if entries: # the tail of an entry split across a page break + entries[-1].raw = f"{entries[-1].raw} {raw}".strip() + continue + entries.append(_entry(str(num), _strip_printed_numeral(raw))) + # rebuilding re-reads the DOI and year out of the joined text: a split + # entry's identifiers often live in the half that was cut off + return [_entry(e.num, e.raw) for e in entries] + + return [_entry(str(i), _strip_printed_numeral(raw)) for i, raw in enumerate(items, 1)] + + +def _leading_numeral(raw: str) -> int | None: + m = _PRINTED_NUMERAL_RE.match(raw) + return int(m.group(1)) if m else None + + +def _usable_printed_numerals(numerals: list[int | None]) -> bool: + """Do these bullets carry a reference numbering, or just happen to start with digits? + + Required: a first entry labelled [1], a strictly ascending run, and most + bullets carrying one. Gaps are allowed and are *informative* — a gap is a + reference the parser could not isolate, and reporting the gap is the honest + result where renumbering around it is the silent one. + """ + seen = [n for n in numerals if n is not None] + if len(seen) < 2 or seen[0] != 1: + return False + if any(b <= a for a, b in zip(seen, seen[1:], strict=False)): + return False + return len(seen) * 2 >= len(numerals) + + +# `1 . Rivara FP` and `1. Rivara FP` — the list numeral the converter turned +# into a bullet without removing. Anchored and bounded: a reference genuinely +# starting with a number ("2019 WHO classification of tumours") keeps it, +# because the separator is required. +_PRINTED_NUMERAL_RE = re.compile(r"^\(?(\d{1,3})\)?\s*[.):\]]\s+") + + +def _strip_printed_numeral(raw: str) -> str: + """Drop a leading list numeral the converter left in the reference text. + + Not cosmetic. `_slug` reads the first token carrying letters, so a numeral + in front is harmless there now — but the numeral also reaches `_title_check` + and the Crossref bibliographic search as part of the reference string, and + it is not part of the reference. + """ + return _PRINTED_NUMERAL_RE.sub("", raw, count=1).strip() or raw + + +# --------------------------------------------------------------------------- +# the publisher's own reference list +# +# `parse_references` was the only stage in the pipeline with no way to say it +# had failed. It always returned a confident list, and nothing compared that +# list to anything. One live audit misnumbered 27 of 41 references and the +# report said so nowhere — the label is the join key, so claims citing [15] and +# up were judged against the wrong papers. +# +# Crossref carries the list the publisher deposited. It is a second, independent +# reading — NOT an oracle. A deposit can be partial (one publisher returned 2 +# references for a paper with about 40), and a partial deposit is more dangerous +# than a bad parse because it looks authoritative. So it is a candidate, and the +# manuscript's own `[N]` markers arbitrate between the candidates. +# --------------------------------------------------------------------------- + +CROSSREF_NO_DOI = ( + "no DOI for the manuscript itself, so the publisher's deposited reference list " + "could not be looked up — usual for a paper under review, which is this tool's " + "main case. Pass --doi if the paper does have one" +) +CROSSREF_NO_DEPOSIT = ( + "the publisher deposited no reference list for this DOI, so there was nothing to " + "check the parsed list against. This is a property of the publisher, not of the paper" +) +CROSSREF_UNREACHABLE = ( + "Crossref could not be reached, so the publisher's deposited reference list was " + "never seen — this run had one reading of the list where it normally has two" +) + + +@dataclass(frozen=True) +class CrossrefDeposit: + """What the publisher deposited, and how much of it this tool could read.""" + + entries: list[RefEntry] = field(default_factory=list) + deposited: int = 0 # references in the record, before this tool read them + publisher: str = "" + title: str = "" # the record's own title — is this DOI even this paper? + absent: str = "" # one of the three CROSSREF_* notes when there is no list + + @property + def unrenderable(self) -> int: + """Deposited references this tool could not turn into an entry. + + Named for whose limitation it is. An earlier version compared + `len(entries)` against the record's `references-count` and called the + shortfall a *partial deposit* — but that field counts the references + **deposited**, so it always equals the array length, and the only way + the comparison could fire was this tool dropping entries it failed to + render. Wiley deposits references as a bare DOI and nothing else; 49 of + its 52 were discarded and the report blamed Wiley for depositing 3. + A shortfall here is the tool's, and says so. + """ + return max(0, self.deposited - len(self.entries)) + + +def _surname(author: str | None) -> str: + """Crossref's `author` is usually `Initials Surname` — keep the surname first. + + Springer deposits `"author": "C Huang"`, so the first letter-bearing token + was the initial and the entry slugged `c-2020`. The slug is the download + filename, the report's source id, and the `--provided` match key documented + as `<firstauthor>-<year>.pdf`, so an initial there quietly stops user files + matching. Elsevier's bare `"Foy"` and any multi-word surname are untouched: + only a short all-caps leading token is dropped. + """ + author = (author or "").strip() + head, _, rest = author.partition(" ") + if rest and head.isupper() and len(head) <= 3: + return rest.strip() + return author + + +def _doi_slug(doi: str) -> str: + """A readable id for a reference known only by its DOI. + + `10.1056/NEJMoa1911793` → `nejmoa1911793`. The suffix is the publisher's own + article id, so it is both stable and recognisable — and it is honest about + what is known, which is the DOI and not an author. + """ + tail = doi.rsplit("/", 1)[-1].lower() + return re.sub(r"[^a-z0-9]+", "-", tail).strip("-")[:32] or "doi" + + +def _reference_raw(ref: dict) -> str: + """One deposited reference as a printed reference string. + + `raw` is what `_slug`, `_title_check` and the Crossref title search all + consume, so a structured deposit has to be assembled back into the shape + those readers expect rather than left as a dict. + + A DOI-only deposit falls back to the DOI itself, so the reference survives. + Dropping those was a silent data loss dressed up as a publisher's fault, and + they are the *best* references in a deposit, not the worst: the DOI is + already resolved, so retrieval skips the bibliographic title search that has + been this module's richest source of wrong-paper bugs. What is lost is the + title check, which then reports `unverifiable` — a disclosed gap, not a + guess. + """ + if unstructured := (ref.get("unstructured") or "").strip(): + return re.sub(r"\s+", " ", unstructured) + parts = [ + _surname(ref.get("author")), + ref.get("article-title") or ref.get("volume-title") or ref.get("series-title"), + ref.get("journal-title"), + ref.get("volume"), + ref.get("first-page"), + ref.get("year"), ] + assembled = re.sub(r"\s+", " ", " ".join(str(p) for p in parts if p)).strip() + return assembled or (ref.get("DOI") or "").strip() + + +def crossref_deposit(client: httpx.Client, doi: str | None, email: str) -> CrossrefDeposit: + """The publisher's deposited reference list for `doi`, numbered by array order. + + **Array order is the only portable numbering signal.** The `key` field looks + like it carries the number and does not: `_b0005`/`_b0010` and `_bib1` turn up + on two Elsevier papers — and both schemes inside a single deposit — beside + `3400_CR1` (Springer), `bibr1-…` (SAGE) and `R10-45-20210317` (Ovid). Parsing + a number out of any of those renumbers every reference of every publisher + that spells it differently. + + Never raises. Three different absences are recorded as three different + notes, because "you have no DOI", "your publisher deposits nothing" and + "Crossref is down" ask the reader for three different things. + """ + if not doi: + return CrossrefDeposit(absent=CROSSREF_NO_DOI) + try: + r = client.get( + f"https://api.crossref.org/works/{doi}", + headers=_contact(email), + ) + if r.status_code != 200: + return CrossrefDeposit(absent=CROSSREF_NO_DEPOSIT if r.status_code == 404 + else CROSSREF_UNREACHABLE) + message = r.json().get("message") + except (httpx.HTTPError, ValueError): + # ValueError covers a 200 that is not JSON — a captive portal or an + # error page, which is Crossref not answering, not Crossref answering no + return CrossrefDeposit(absent=CROSSREF_UNREACHABLE) + # a 200 whose body is `null`, a list, or anything but the documented object + # is also Crossref not answering — and `.get` on it is an AttributeError + # that would take down a run this function promises never to break + if not isinstance(message, dict): + return CrossrefDeposit(absent=CROSSREF_UNREACHABLE) + + refs = message.get("reference") + refs = refs if isinstance(refs, list) else [] + if not refs: + return CrossrefDeposit(absent=CROSSREF_NO_DEPOSIT, + publisher=message.get("publisher", "")) + + entries: list[RefEntry] = [] + for i, ref in enumerate(refs, 1): + if not isinstance(ref, dict): + continue + raw = _reference_raw(ref) + if not raw: + continue + e = _entry(str(i), raw) + # `_entry` scrapes a DOI out of the text; the deposit states one, and a + # stated DOI is better evidence than a scraped one + if doi_field := (ref.get("DOI") or "").strip(): + e.doi = doi_field + # a reference known only by its DOI has no author to be named + # after, so name it after what IS known + if raw == doi_field: + e.slug = _doi_slug(doi_field) + # a publisher can deposit a reference to its own table, and a part of a + # work is never the work a reference cites — wherever the DOI came from + if _is_component_doi(e.doi): + e.doi = None + entries.append(e) + + titles = message.get("title") or [] + return CrossrefDeposit( + entries=_unique_slugs(entries), + deposited=len(refs), + publisher=message.get("publisher", ""), + title=titles[0] if isinstance(titles, list) and titles else "", + ) + + +def deposit_is_this_paper(manuscript_title: str, record_title: str) -> bool | None: + """Is the Crossref record behind the DOI the paper being audited? + + True, False, or **None for "cannot tell"** — the same tri-state the source + title check uses, for the same reason: an unknown is not a match and it is + not a mismatch either, and collapsing it would either discard good deposits + or wave wrong ones through. + + This is the gate every other retrieval route in this module already has, on + the one route that can replace the *entire* reference list. The DOI is + scraped off page 1 or typed by hand; a data-availability DOI, an erratum or + a preprint version can easily carry the same number of references as the + paper, so the count test would pass and the report would print "numbering + confirmed" over another paper's bibliography. + """ + # the rule itself is `models.titles_match` — `scout` asks the same question + # of a Europe PMC record, and neither module may import the other + return titles_match(manuscript_title, record_title) + + +def crossref_reference_list( + client: httpx.Client, doi: str | None, email: str +) -> list[RefEntry] | None: + """The deposited list, or None when there is none. + + None rather than `[]`: an empty list reads as "this paper cites nothing", + and the reconciler has to be able to tell that apart from "nobody deposited + anything to read". + """ + deposit = crossref_deposit(client, doi, email) + return deposit.entries or None + + +# Two readings of one bibliography are a fingerprint of the paper they belong +# to. Measured on the 41-reference audit: 38 of 41 deposited works appear +# somewhere in the printed list (93%), against 0 of 41 for a different paper — +# a separation wide enough that the threshold is not a tuning parameter. +_CORROBORATION_RATIO = 0.5 +# Below this, agreement is a coincidence a short comment piece can produce. +_CORROBORATION_MIN = 5 + + +@dataclass(frozen=True) +class Corroboration: + """Whether two readings of a reference list describe the same paper's work. + + `refutes` is always False, and that asymmetry is the point: agreement is + evidence of identity, disagreement is *not* evidence of difference. Two + lists that disagree may be one paper read badly — which is the case this + whole module exists for — so a low overlap leaves the identity unconfirmed + rather than calling the record another paper. + """ + + found: int = 0 + total: int = 0 + confirms: bool = False + too_few: bool = False + refutes: bool = False # never true; named so the asymmetry is readable + + +def deposit_corroborates(deposit: list[RefEntry], parsed: list[RefEntry]) -> Corroboration: + """Do these two readings of a reference list name the same works? + + Set membership, not position. Positionally the audited paper scores 34% + against its own deposit, because its parse is misnumbered from [15] on — + and the numbering is exactly the thing in question, so it cannot be an input + to the identity test. + + This is the identity check for a paper whose title cannot be read: an + article-type banner where the title should be, no metadata, and a converter + that offers nothing better. + """ + if len(deposit) < _CORROBORATION_MIN or len(parsed) < _CORROBORATION_MIN: + return Corroboration(total=len(deposit), too_few=True) + # Counted whole, with no early exit once the threshold is settled: `found` + # is printed to the reader as "N of M", and a comparison that stopped + # counting would report a lower bound as if it were the number. The cost is + # quadratic and measured: 6 ms at 41 references, 39 ms at 100, 0.9 s at 500 + # — against a run that makes paid model calls. + found = sum(1 for d in deposit if any(_same_work(d, p) for p in parsed)) + return Corroboration(found=found, total=len(deposit), + confirms=found / len(deposit) >= _CORROBORATION_RATIO) + + +# --------------------------------------------------------------------------- +# reconciliation — the body's labels arbitrate between two candidate readings +# --------------------------------------------------------------------------- + + +@dataclass +class Reconciliation: + """Which reading of the reference list was used, and whether it was checked.""" + + source: str = "parsed" # crossref | parsed + verified: bool = False + # the chosen reading matched the body's labels, but the OTHER reading + # disagreed. `_covers` is a test of extent, not of content, so a second + # independent reading calling the list wrong is worth the reader's eye even + # when the count checks out — burying it in a field no template renders was + # how a compensating parse error could pass unmentioned. + contested: bool = False + note: str = "" + unverified_from: int | None = None # first label whose numbering is in doubt + body_labels: int = 0 + crossref_count: int | None = None + parsed_count: int = 0 + + +def _covers(body: set[str], entries: list[RefEntry]) -> bool: + """Does this candidate account for exactly the references the body cites? + + Two conditions. Every cited label must exist in the list, and the list must + be exactly as long as the highest label cited — which under citation-order + numbering is not a heuristic: in a numeric-citation journal reference N *is* + the Nth first-cited work, so the body's labels run 1..N by construction. + + **What this cannot see.** Both conditions are about extent, not content. A + parse that merges one pair of references and splits another keeps the count + and passes here, with every label between the two errors pointing one paper + off. The subset test only bites since `_parse_bulleted` began reading the + *printed* numerals, which can leave gaps; while numbering was positional, + `nums` was always `{1..len(entries)}` and the subset test was implied by the + count. So this is a strong test of "is the list the right length" and a weak + one of "is entry N the right paper" — which is why a matching count is not + the end of it, and why a second reading that disagrees is still reported. + + The cost is a paper whose reference list holds a work the body never cites: + it earns a warning it did not deserve. That is the right way round — the + alternative silently accepts the numbering that judged 27 references against + the wrong papers. + """ + if not body or not entries: + return False + nums = {e.num for e in entries} + return body <= nums and len(entries) == max(int(x) for x in body) + + +def _same_work(x: RefEntry, y: RefEntry) -> bool: + """Do these two readings name the same paper? + + A DOI settles it when both carry one. Otherwise it is a token overlap, and + it has to be, because the two readings describe a paper in different + dialects: Elsevier prints `F.P. Rivara, D.C. Grossman, …` while the deposit + carries `author: "Rivara"`. Comparing slugs made those two `fp-2019` and + `rivara-2019` — a reported divergence at entry [1] for a whole class of + journals, which tainted every claim in the report and made the corroboration + path unreachable. + """ + if x.doi and y.doi: + return x.doi.lower() == y.doi.lower() + if x.year and y.year and x.year != y.year: + return False + tx, ty = _title_tokens(x.raw), _title_tokens(y.raw) + if not tx or not ty: + return True # nothing to compare is not evidence of disagreement + return len(tx & ty) / min(len(tx), len(ty)) >= 0.34 + + +def _first_divergence(a: list[RefEntry], b: list[RefEntry]) -> int | None: + """The first 1-based position where two readings stop describing one paper. + + None when they agree the whole way down the shorter list — *not* the index + past the end. Returning `min(len)+1` produced "entries from [42] onward are + affected" on a 41-entry list, a warning naming an entry that does not exist + while `label_is_doubtful` quietly returned False for every real label. The + banner and the per-claim layer then said opposite things. + + Compared with `_same_work`, not by string or slug: the two legs are a PDF + parse and a publisher deposit, so they never agree character-for-character + even when they name the same paper. + """ + # strict=False on purpose: the two readings having different lengths is the + # normal case here, and it is the caller's finding, not an error to raise + for i, (x, y) in enumerate(zip(a, b, strict=False), 1): + if not _same_work(x, y): + return i + if len(a) != len(b): + # they agree as far as the shorter one goes, and then one simply stops: + # the first entry the two readings disagree about existing is in doubt + return min(len(a), len(b)) + 1 + return None + + +def reconcile( + body_labels: set[str], + crossref: list[RefEntry] | None, + parsed: list[RefEntry], + crossref_absent: str = "", +) -> tuple[list[RefEntry], Reconciliation]: + """Choose between two readings of the reference list, and say how sure it is. + + The manuscript's own `[N]` markers are the arbiter — free, needing no DOI and + no network, and the only one of the three that is definitionally right about + what the paper cites. Crossref and the PDF parse are candidates measured + against it. + + On unresolvable disagreement the audit continues: the numbering is marked + unverified, the report discloses it, and every verdict resting on a doubtful + label carries the caveat. Refusing to run would throw away a useful audit + over a numbering the reader can check by hand. + """ + rec = Reconciliation( + body_labels=len(body_labels), + crossref_count=len(crossref) if crossref is not None else None, + parsed_count=len(parsed), + ) + cr_ok = _covers(body_labels, crossref or []) + parse_ok = _covers(body_labels, parsed) + cited = max((int(x) for x in body_labels), default=0) + + if cr_ok: + # both matching is not a tie to break: prefer the deposit, whose DOIs + # are already resolved, which skips the title search that has been this + # module's richest source of wrong-paper bugs + rec.source, rec.verified = "crossref", True + rec.contested = not parse_ok and bool(parsed) + rec.note = ( + f"the publisher's deposited list has {len(crossref)} references and the " + f"manuscript cites [1]-[{cited}] — they agree" + + (f"; the parsed list has {len(parsed)}, which does not, so it was not used" + if rec.contested else "") + ) + return list(crossref), rec + + if parse_ok: + rec.source, rec.verified = "parsed", True + rec.contested = crossref is not None + rec.note = ( + f"the parsed list has {len(parsed)} references and the manuscript cites " + f"[1]-[{cited}] — they agree" + ) + if crossref is not None: + rec.note += ( + f"; the publisher deposited {len(crossref)}, which does not. The count " + "checks out, but a second independent reading calls this list wrong — " + "and a count cannot tell a right list from one that merged two " + "references and split another" + ) + elif crossref_absent: + rec.note += f"; {crossref_absent}" + return list(parsed), rec + + # Nothing matched. Use the parse — it is at least a reading of the paper in + # hand, where a deposit describes the published version, which a manuscript + # under review is not. + chosen = parsed or list(crossref or []) + rec.source = "parsed" if parsed else "crossref" + rec.verified = False + # Two independent readings agreeing about entry N is evidence about entry N + # even with no arbiter to confirm either — a PDF parse and a publisher + # deposit have no common failure mode. Only where they diverge is the + # numbering actually in doubt. With one reading there is no such evidence, + # and claiming a divergence point would present unchecked entries as checked. + rec.unverified_from = ( + _first_divergence(crossref, parsed) if crossref is not None and parsed else 1 + ) + if not body_labels: + # A third fact, not a failure of either candidate: the arbiter does not + # exist. Superscript-numeric styles are the common case and the numbering + # is still the join key there — the markers are simply invisible once the + # PDF is flattened to text, so half a spread of journals lands here. The + # coverage audit is blind to exactly the same papers. + both = crossref is not None and parsed + if both and rec.unverified_from is None: + corroborated = ( + " The parsed list and the publisher's deposit agree throughout, which " + "is evidence for this numbering but not confirmation of it — they can " + "still be wrong together." + ) + elif both and rec.unverified_from > 1: + corroborated = ( + f" The parsed list and the publisher's deposit agree as far as entry " + f"[{rec.unverified_from - 1}], which is evidence about those entries " + "but not confirmation." + ) + else: + corroborated = "" + rec.note = ( + "no bracketed numeric citation markers were found in the body, so there is " + "nothing to check the reference numbering against — only styles like [12], " + "[7,8] and [9-11] can be read. The numbering below is unconfirmed." + + corroborated + ) + return chosen, rec + + detail = f"the manuscript cites [1]-[{cited}], the parsed list has {len(parsed)} references" + if crossref is not None: + detail += f" and the publisher deposited {len(crossref)}" + elif crossref_absent: + detail += f", and {crossref_absent}" + scope = ( + f"Entries from [{rec.unverified_from}] on may name a different paper than the " + "label they carry, and verdicts on claims citing them are marked accordingly" + if rec.unverified_from + else "The two readings agree with each other entry for entry, so both are " + "wrong in the same way or the body's labels were read incompletely" + ) + rec.note = ( + f"{detail} — that does not add up, so the numbering could not be confirmed. " + + scope + ) + return chosen, rec def _entry(num: str, raw: str) -> RefEntry: @@ -144,9 +710,46 @@ def _entry(num: str, raw: str) -> RefEntry: def _slug(raw: str, year: str | None) -> str: - first = re.split(r"[,\s]", raw.strip(), maxsplit=1)[0] - first = re.sub(r"[^A-Za-z\-]", "", first).lower() or "ref" - return f"{first}-{year}" if year else first + """A short id for a reference: first author's surname plus year. + + Takes the first token that actually contains letters, rather than the first + token. A leading numeral the converter failed to strip left nothing after + the non-letter filter, so the entry fell back to the literal `ref` — and + since the slug is also the download's filename, every such reference + resolved to the same path. + """ + for token in re.split(r"[,\s]+", raw.strip()): + if cleaned := re.sub(r"[^A-Za-z\-]", "", token).lower().strip("-"): + return f"{cleaned}-{year}" if year else cleaned + return f"ref-{year}" if year else "ref" + + +def _unique_slugs(entries: list[RefEntry]) -> list[RefEntry]: + """Guarantee no two entries share a slug, in place. + + `resolve_all` writes each download to `<slug>.pdf`, so two entries sharing a + slug share a *file*: the second download overwrites the first, and every + claim citing the first is then judged against the second's paper. A live run + put 11 retrieved sources at one path this way. + + A genuine collision is possible without any parser bug — the same first + author and year cited twice — so uniqueness is enforced here rather than + assumed to fall out of a better slug. The first entry keeps the natural + slug, so a `--provided` file named `<author>-<year>.pdf` still matches. + """ + seen: set[str] = set() + for e in entries: + base = e.slug or "ref" + slug = base + if slug in seen: + # the reference number is the one thing guaranteed distinct, and it + # keeps the name legible in a report where the slug is shown + slug = f"{base}-r{e.num}" + while slug in seen: + slug += "x" + e.slug = slug + seen.add(slug) + return entries # --------------------------------------------------------------------------- @@ -213,35 +816,6 @@ def _download_pdf(client: httpx.Client, url: str, dest: Path) -> bool: # journal names and boilerplate that appear on almost any first page — # they must not let a wrong paper pass the title check -_TITLE_STOPWORDS = frozenset( - {"commun", "nature", "science", "journal", "lancet", "article", - "elsevier", "springer", "wiley", "volume", "press", "https"} -) - -_URL_RE = re.compile(r"(?:https?://|www\.)\S+", re.I) - - -def _title_tokens(raw: str) -> set[str]: - """The reference's own distinctive words — URLs removed first. - - A URL is not part of a title, and a *tracking parameter* least of all: - `?utm_source=chatgpt.com` on a cited news page contributed `chatgpt` and - `source` to this set, and the wrong paper Crossref returned was an - editorial about ChatGPT. Path segments do the same from the other side, - inflating the denominator with `firstmedical`, `assuranceprogram` and - `publications` — words no first page will carry, so they dilute the ratio - the check is measured on. - """ - return set(re.findall(r"[a-z]{5,}", _URL_RE.sub(" ", raw).lower())) - _TITLE_STOPWORDS - - -# Four distinct words, not three. The observed false positive cleared the 0.35 -# ratio on `artificial`, `intelligence` and `medical` — three words that are the -# subject of most papers in this field, so no stopword list can retire them -# without rejecting correct matches. Falling below the floor yields -# `unverifiable`, never `mismatch`: too few words to tell is not evidence of a -# different paper, and a `mismatch` would discard a possibly-correct download. -_TITLE_MIN_MATCHES = 4 # the three answers the check can give. "unverifiable" used to share `None` @@ -315,7 +889,11 @@ def _accept( ) return False entry.status, entry.resolver, entry.pdf_path = "retrieved", resolver, str(dest) - entry.reason = why + # Carry the check's own evidence. The mismatch branch above already states + # its detail; the accepting branch discarded it, so `title_check: verified` + # and `title_check: unverifiable` reached the manifest as bare assurances + # with nothing behind them — and those two mean very different things. + entry.reason = f"{why} · title check: {detail}" if detail else why return True @@ -327,12 +905,33 @@ def _accept( ) +def _named_for(entry: RefEntry, provided_dir: Path | None) -> list[Path]: + """Every file in the folder whose name contains all of the reference's tokens. + + The one place the match rule lives. Token containment stays loose on purpose + — real filenames carry author lists and titles, and `tests/test_refs.py` + pins that. It is shared because the article scan and the supplement scan are + the *same* question asked with `_SUPPLEMENT_RE` inverted, and a second copy + of the rule is how they would drift into disagreeing about which reference a + file belongs to — which would attach an appendix to the wrong paper. + """ + if not provided_dir or not provided_dir.is_dir(): + return [] + tokens = [t for t in (entry.slug or "").lower().split("-") if len(t) > 3] + if not tokens: + return [] + return [ + pdf + for pdf in sorted(provided_dir.glob("*.pdf")) + if all(t in pdf.name.lower() for t in tokens) + ] + + def _provided_candidates(entry: RefEntry, provided_dir: Path | None) -> list[Path]: """Every file in the folder that could be this reference, best first. - Token containment stays loose on purpose — real filenames carry author lists - and titles, and `tests/test_refs.py` pins that. What is tightened is the - choice among the matches: + What is tightened, relative to the shared match above, is the choice among + the matches: * an exact `<slug>.pdf` wins outright; * otherwise the shortest stem, tie-broken by name. Shortest means fewest @@ -343,23 +942,295 @@ def _provided_candidates(entry: RefEntry, provided_dir: Path | None) -> list[Pat Supplements are excluded rather than ranked last. Judging a claim against an appendix while calling it the cited source is the laundering this codebase exists to prevent, and returning nothing lets the online chain try for the - real article instead. + real article instead. They are not discarded any more, though — + `_supplement_candidates` picks up exactly what this drops. """ - if not provided_dir or not provided_dir.is_dir(): - return [] slug = (entry.slug or "").lower() - tokens = [t for t in slug.split("-") if len(t) > 3] - if not tokens: - return [] - matches = [ - pdf - for pdf in sorted(provided_dir.glob("*.pdf")) - if all(t in pdf.name.lower() for t in tokens) - and not _SUPPLEMENT_RE.search(pdf.stem) - ] + matches = [p for p in _named_for(entry, provided_dir) if not _SUPPLEMENT_RE.search(p.stem)] return sorted(matches, key=lambda p: (p.stem.lower() != slug, len(p.stem), p.name)) +def _supplement_candidates(entry: RefEntry, provided_dir: Path | None) -> list[Path]: + """The inverse of `_provided_candidates`: this reference's supplements. + + No ranking and no best-of-one. Every supplement a user supplies is a + document they are asking to have read, and choosing between them would put + one of them silently out of the audit. + """ + hits = [p for p in _named_for(entry, provided_dir) if _SUPPLEMENT_RE.search(p.stem)] + # A surname under four characters is dropped by `_named_for`'s length + # filter, so `liu-2019` matches on the YEAR alone and every 2019 appendix in + # the folder looked like Liu's. An article surviving that is caught by its + # title check; a supplement gets none, so a wrong one would be judged as + # part of the cited work with nothing able to notice. + # + # Requiring the slug itself in the name, rather than refusing outright, is + # what keeps the case this rule serves: `liu-2019-appendix.pdf` still + # attaches, `smith-2019-appendix.pdf` no longer does. A supplement whose own + # title named the work never reaches here — it is verified, not guessed. + tokens = [t for t in (entry.slug or "").lower().split("-") if len(t) > 3] + if all(t.isdigit() for t in tokens): + slug = (entry.slug or "").lower() + hits = [p for p in hits if slug and slug in p.name.lower()] + return hits + + +_SLUG_UNSAFE = re.compile(r"[^a-z0-9]+") + + +def _stem_slug(path: Path) -> str: + """The document id a supplementary file is read under. + + Derived from the file STEM, never from an ordinal position in the folder. + `-suppl1`/`-suppl2` numbered in sorted order is the defect CLAUDE.md rejects + for citation occurrences — remove one file and every id after it shifts, so + a re-run points the previous run's stored verdicts and evidence crops at a + different PDF, with nothing to notice. + """ + return _SLUG_UNSAFE.sub("-", path.stem.lower()).strip("-") or "supplement" + + +def _free_slug(base: str, taken: set[str]) -> str: + """`base`, or the first `-N` variant nobody has claimed. + + Slugs are the identity of a document everywhere downstream — `ingest/<slug>/`, + `sources_resolved/<slug>.pdf`, every judgement and every crop — so two + documents sharing one means a verdict shown against the wrong paper. + `_unique_slugs` cannot do this job: it runs at parse time, and supplements + are not discovered until resolution. + """ + slug, n = base, 2 + while slug in taken: + slug, n = f"{base}-{n}", n + 1 + return slug + + +def attach_supplements( + entry: RefEntry, + provided_dir: Path | None, + taken: set[str], + content: list[Identified] = (), +) -> None: + """Attach this reference's supplementary files, in place. + + **Only to an available reference.** A supplement whose article could not be + obtained is left for `orphaned_supplements` to report: judging a claim + against an appendix while nothing establishes what the article itself says + is the laundering `_provided_candidates` already refuses, and attaching here + would reintroduce it through the back door. + + `taken` is read and written — the caller owns one set for the whole run, so + a supplement cannot collide with an article slug or with another + supplement's. + """ + if entry.status not in ("retrieved", "provided"): + return + # a supplement whose own title or DOI named this work HAS been checked, and + # the report must stop warning about it in the same breath as one that only + # matched a filename + named = {(pdf, False) for pdf in _supplement_candidates(entry, provided_dir)} + inferred = {(found.path, True) for found in content} + for pdf, verified in sorted(named | inferred): + slug = _free_slug(_stem_slug(pdf), taken) + taken.add(slug) + entry.supplements.append(Supplement(slug=slug, pdf_path=str(pdf), verified=verified)) + + +# How much of a first page stands in for a title when the PDF declares none. +# Short on purpose: the comparison below is title-against-title, and letting a +# whole page in is what makes it imprecise — see `identify_by_content`. +_TITLE_FALLBACK_CHARS = 300 + + +def _self_declared_title(pdf: Path) -> str: + """The best short string this PDF offers about what it is. + + Metadata first, because measured on real papers it carries the exact title + while the top of page 1 carries an article-type banner — `REVIEW ARTICLE`, + `Article`, `HEALTH IN ACTION`. `_declared_title_is_usable` rejects the + `Microsoft Word - draft.docx` shapes, and then the head of page 1 is the + fallback for the roughly one paper in seven whose metadata is empty. + """ + from .ingest import declared_title + from .models import _declared_title_is_usable + + title = declared_title(pdf) + if _declared_title_is_usable(title): + return title + return " ".join(_first_page_text(pdf).split())[:_TITLE_FALLBACK_CHARS] + + +def _doi_in(pdf: Path) -> str | None: + """The DOI this PDF prints about itself, or None. + + Truncated at the references heading for the reason `wizard.detect_doi` is: + a short paper's first page reaches its own bibliography, and every DOI + printed there belongs to somebody else. + """ + from .wizard import _before_references + + if m := DOI_RE.search(_before_references(_first_page_text(pdf))): + return m.group(0).rstrip(".,);]").lower() + return None + + +class Identified(NamedTuple): + """What one file turned out to be, and what said so. + + `signal` is provenance rather than decoration: a DOI is exact and a title is + a judgement over token overlap, and a reader weighing a verdict is owed the + difference. It reaches the manifest `reason`. + """ + + path: Path + entry: RefEntry + kind: str # "article" | "supplement" + signal: str # "DOI" | "title" + + +def identify_by_content( + entries: list[RefEntry], provided_dir: Path | None, claimed: set[Path] +) -> tuple[dict[Path, Identified], list[tuple[Path, str]]]: + """Work out which reference each unclaimed PDF is, by reading it. + + Filename matching answers "which files could be this entry?". This asks the + opposite — "which entry is this file?" — because a publisher download + (`s41467-023-39631-x.pdf`, `mmc1.pdf`) names nothing, and before this it was + ignored without a word while the audit looked entirely normal. + + Returns `{path: Identified(entry, kind, signal)}` and the files it could + not place, each with the reason. + + Two signals, and deliberately NOT the one already in this module. + `_title_check_text` counts a reference's words anywhere on a whole page, + which is right for vetoing a file the user already named and wrong here: + measured on a real pair, a chest-radiograph paper "verified" against an NEJM + review as well, on `current`, `future`, `interpretation`, `medical`, + `images`. Comparing title against title instead keeps the denominator small + and the answer unique. + + A non-unique match is REFUSED, never ranked. A corrigendum shares nearly + every distinctive word with its original, so a best-score pick would judge a + claim against the wrong paper with nothing downstream able to notice. + + `None` from `titles_match` — too few distinctive words to tell — is not an + accept either. A filename match may be taken as `unverifiable` because the + user asserted it by naming the file; nobody asserted anything here. + """ + from .models import titles_match + + if not provided_dir or not provided_dir.is_dir(): + return {}, [] + assigned: dict[Path, Identified] = {} + unclaimed: list[tuple[Path, str]] = [] + by_doi = {e.doi.lower(): e for e in entries if e.doi} + + for pdf in sorted(provided_dir.glob("*.pdf")): + if pdf in claimed: + continue + title = _self_declared_title(pdf) + # the marker list is right about prose and wrong about publisher + # filenames — `\besm\b` cannot match inside `MOESM1_ESM` — so both are + # consulted and the file's own words are what usually decide + kind = "supplement" if ( + _SUPPLEMENT_RE.search(pdf.stem) or _SUPPLEMENT_RE.search(title) + ) else "article" + + if (doi := _doi_in(pdf)) and doi in by_doi: + assigned[pdf] = Identified(pdf, by_doi[doi], kind, "DOI") + continue + hits = [e for e in entries if titles_match(e.raw, title) is True] + if len(hits) == 1: + assigned[pdf] = Identified(pdf, hits[0], kind, "title") + elif hits: + labels = ", ".join(f"[{e.num}]" for e in hits) + unclaimed.append(( + pdf, + f"its title matches {labels} equally well, and guessing between them " + f"would judge a claim against the wrong paper — rename it to " + f"<reference>.pdf to choose", + )) + else: + unclaimed.append(( + pdf, + "could not tell which reference this is from its title or its DOI", + )) + return assigned, unclaimed + + +def manuscript_supplements(paths: list[Path], taken: set[str]) -> list[Supplement]: + """The audited paper's own supplementary files, as documents. + + Named explicitly with `--supplement` rather than discovered by filename: + the sources folder is matched against *reference* slugs, and the paper under + audit has none to match, so there is nothing for the convention to key on. + + Shares `taken` with the reference supplements for the reason they share it + with each other — a slug is a folder under `ingest/` and a file under + `sources_resolved/`, so the paper's appendix colliding with a cited source + means one silently reading the other's pages. + """ + out = [] + for p in paths: + slug = _free_slug(_stem_slug(p), taken) + taken.add(slug) + out.append(Supplement(slug=slug, pdf_path=str(p))) + return out + + +def unused_provided( + entries: list[RefEntry], + provided_dir: Path | None, + reasons: dict[Path, str] | None = None, +) -> list[tuple[Path, str]]: + """Every PDF in the folder that ended up attached to nothing, each with why. + + A file the user deliberately put in the folder and that then did nothing is + the quietest possible failure: they go on believing it was read. This used + to cover supplements only, so an unmatched *article* PDF was dropped in + silence while a supplement-named one was named — the asymmetry meant a + folder of publisher-named downloads produced an audit that looked entirely + normal and used none of it. + + Reasons are kept apart because the fixes differ: `reasons` carries what + `identify_by_content` already worked out (ambiguous title, unrecognisable), + an unavailable article is named as such because supplying the article is + the fix, and anything left could not be placed at all. + """ + if not provided_dir or not provided_dir.is_dir(): + return [] + used = {Path(e.pdf_path) for e in entries if e.pdf_path} + used |= {Path(s.pdf_path) for e in entries for s in e.supplements} + unavailable: dict[Path, str] = {} + for e in entries: + if e.status in ("retrieved", "provided"): + continue + for pdf in _supplement_candidates(e, provided_dir): + unavailable.setdefault( + pdf, f"[{e.num}] is not available ({e.status}), so nothing can be judged against it" + ) + # asked here rather than carried down from `resolve_all`, so there is ONE + # place that explains why a file went unused. Re-running inference over the + # leftovers is a reporting pass, not a second decision — and at ~9 ms a file + # it costs nothing worth arranging around. + identified, unclaimed = identify_by_content(entries, provided_dir, used) + inferred = dict(unclaimed) + out = [] + for pdf in sorted(provided_dir.glob("*.pdf")): + if pdf in used: + continue + why = (reasons or {}).get(pdf) or unavailable.get(pdf) or inferred.get(pdf) + if why is None and (found := identified.get(pdf)) is not None: + # it WAS recognised — another file got there first. A spare copy of + # a paper already matched is not a mystery and must not read as one. + why = ( + f"it is [{found.entry.num}], recognised by its {found.signal}, but " + f"[{found.entry.num}] already has a file — this one was not needed" + ) + out.append((pdf, why or "could not tell which reference this is")) + return out + + def _match_provided(entry: RefEntry, provided_dir: Path | None) -> Path | None: candidates = _provided_candidates(entry, provided_dir) return candidates[0] if candidates else None @@ -387,6 +1258,38 @@ def _match_provided(entry: RefEntry, provided_dir: Path | None) -> Path | None: ) +# A DOI naming a PART of a work: Crossref mints these for tables, figures and +# supplements, and a title search will happily return one. `/table-1` came back +# for the caption "Table 1. Dataset characteristics" and was reported as a +# paywalled cited work. +_COMPONENT_DOI_RE = re.compile( + r"/(?:table|figure|fig|scheme|supp(?:l|lement(?:al|ary)?)?)[-_.]?\d+/?$" + r"|\.s\d{3,}$", + re.I, +) + + +def _component_doi_reason(doi: str | None) -> str: + return ( + f"the only DOI available ({doi}) names a table, figure or supplement, not a " + "paper — a part of a work is never the work a reference cites. Recorded as " + "no DOI rather than resolved, because fetching it would judge claims against " + "someone else's table" + ) + + +def _is_component_doi(doi: str | None) -> bool: + """Does this DOI name a table, figure or supplement rather than a work? + + A part of a paper is never the thing a reference cites, so accepting one is + always wrong — whether it arrived from a Crossref title search or was + printed in the reference itself. Anchored at the end of the DOI so an + ordinary suffix that merely contains the word (`.../figures-in-radiology`) + is untouched. + """ + return bool(doi and _COMPONENT_DOI_RE.search(doi)) + + def _is_webpage_reference(raw: str) -> bool: """Is this reference a web page rather than an article? @@ -421,8 +1324,15 @@ def resolve_entry( email: str, client: httpx.Client, provided_dir: Path | None = None, + content_match: Identified | None = None, ) -> RefEntry: - """Resolve one reference in place. Never raises — failures land in status/reason.""" + """Resolve one reference in place. Never raises — failures land in status/reason. + + `content_match` is a file `identify_by_content` recognised as this reference + from its own DOI or title. Consulted only after the filename rule has had + its say: the filename is the user's own assertion about the file, and + content fills the gap it leaves rather than overruling it. + """ # before anything else: an ambiguous boundary makes `raw` two references # spliced together, so the slug, the title and any Crossref lookup derived # from it can all name the wrong paper. A recorded gap is the honest result. @@ -432,26 +1342,88 @@ def resolve_entry( dest = dest_dir / f"{entry.slug}.pdf" + refused: Path | None = None if candidates := _provided_candidates(entry, provided_dir): - provided = candidates[0] + # Ranked, and now read in order rather than by taking the first: a + # reference whose slug carries a uniqueness suffix can match a file named + # after the base slug, and that file is another reference's paper. + chosen: tuple[Path, str, str] | None = None + for cand in candidates: + state, detail = _title_check(entry, cand) + # `named` is the user's own act: they wrote this reference's slug on + # the file. Then a failed check is DISCLOSED, not fatal — they chose + # it, there is nothing to fall back to, and a scanned PDF yields no + # text at all. A token match is not their act, so a check that says + # "different paper" is a reason to keep looking. + named = cand.stem.lower() == (entry.slug or "").lower() + if named or state != TITLE_MISMATCH: + chosen = (cand, state, detail) + break + refused = refused or cand + if chosen is not None: + provided, state, detail = chosen + entry.status, entry.resolver = "provided", "user" + entry.pdf_path = str(provided) + if len(candidates) == 1: + others = "" + elif provided is candidates[0]: + others = f" ({len(candidates)} candidates matched; picked the closest name)" + else: + # the closest-named file is positively a different paper, so the + # reader is told the pick was not the obvious one + others = ( + f" ({len(candidates)} candidates matched; the closest-named " + "ones are other papers)" + ) + entry.title_check = state + if state == TITLE_VERIFIED: + note = f" — identity confirmed: {detail}" + else: + # "unverified" for both remaining states, because both mean the + # same thing to a reader: nobody established that this file is + # the paper the reference names + note = f" — identity unverified: {detail}" + entry.reason = f"matched {provided.name} in your sources folder{others}{note}" + return entry + + if content_match is not None: + # `verified` by construction: only a positive DOI or title match reaches + # here. `titles_match` returning None — too few distinctive words to + # tell — was already refused upstream, because nobody named this file + # and there is no user assertion to fall back on. + pdf = content_match.path entry.status, entry.resolver = "provided", "user" - entry.pdf_path = str(provided) - others = f" ({len(candidates)} candidates matched; picked the closest name)" \ - if len(candidates) > 1 else "" - # a provided file is title-checked like a downloaded one, but a failure - # is DISCLOSED, not fatal: the user named this file, there is nothing to - # fall back to, and a scanned PDF yields no text at all - state, detail = _title_check(entry, provided) - entry.title_check = state - if state == TITLE_VERIFIED: - note = f" — identity confirmed: {detail}" - else: - # "unverified" for both remaining states, because both mean the same - # thing to a reader: nobody established that this file is the paper - note = f" — identity unverified: {detail}" - entry.reason = f"matched {provided.name} in your sources folder{others}{note}" + entry.pdf_path = str(pdf) + entry.title_check = TITLE_VERIFIED + entry.reason = ( + f"identified {pdf.name} in your sources folder by its " + f"{'own DOI' if content_match.signal == 'DOI' else 'title'} — its filename " + "names no reference, so nothing but the file itself chose it" + ) return entry + _resolve_by_retrieval(entry, dest, email, client) + if refused is not None and not entry.pdf_path: + # the chain may still have found the real paper; where it did not, the + # file this tool looked at and declined is named, because a gap that + # withholds what the tool already knows is the failure this codebase + # exists to avoid + entry.reason = ( + f"{entry.reason}. {refused.name} in your sources folder was set aside: its " + "first page is a different paper, and the file is not named for this " + "reference, so nobody chose it for this one" + ) + return entry + + +def _resolve_by_retrieval( + entry: RefEntry, dest: Path, email: str, client: httpx.Client +) -> RefEntry: + """The online chain: arXiv, then a DOI, then Unpaywall and Europe PMC. + + Split out of `resolve_entry` so a provided file that was set aside can be + reported after the chain has run, rather than at each of its nine exits. + """ try: if arxiv := ARXIV_RE.search(entry.raw): url = f"https://arxiv.org/pdf/{arxiv.group(1)}" @@ -474,9 +1446,38 @@ def resolve_entry( ) return entry + # The parser is fallible, so this is the second line of defence. An + # entry with no year, no DOI and no arXiv id is not a citable work, and + # a bibliographic search always answers with *something*: three of one + # paper's own table captions were searched by title and came back as + # table-component DOIs belonging to unrelated papers, then published as + # paywalled references. + if not entry.doi and not looks_like_reference(entry.raw): + entry.status = "no_doi" + entry.reason = ( + "this entry carries no year, DOI or arXiv id, so nothing here reads as " + "a cited work — it is more likely a caption or a heading the reference " + "parser swept in. Not searched by title: Crossref would answer with the " + "closest-looking record, and inventing a reference is worse than " + "reporting one the parser got wrong" + ) + return entry + + # a DOI printed in the reference can name a part of a paper too + if _is_component_doi(entry.doi): + entry.status, entry.reason = "no_doi", _component_doi_reason(entry.doi) + entry.doi = None + return entry + if not entry.doi: try: - entry.doi = _crossref_doi(client, entry.raw, email) + found = _crossref_doi(client, entry.raw, email) + if _is_component_doi(found): + # a title match is not a work match — a table's title is the + # table's, and this one belonged to a different paper + entry.status, entry.reason = "no_doi", _component_doi_reason(found) + return entry + entry.doi = found if entry.doi: entry.resolver = "crossref" except httpx.HTTPError: @@ -508,11 +1509,50 @@ def resolve_all( email: str, provided_dir: Path | None = None, progress: ProgressCb | None = None, + taken: set[str] | None = None, ) -> list[RefEntry]: dest_dir.mkdir(parents=True, exist_ok=True) + # one registry for the whole run, seeded with the article slugs `_unique_slugs` + # already fixed at parse time — supplements are only discovered here, so they + # cannot go through it and must not be allowed to shadow an article's folder. + # The caller may pass its own so the AUDITED paper's supplements, which are + # named on the command line rather than found here, share the same namespace. + taken = set() if taken is None else taken + taken |= {e.slug for e in entries if e.slug} + # Content inference runs once, over the whole folder, BEFORE the per-entry + # loop: the filename rule asks "which files could be this entry?" and this + # asks the opposite. Everything the filename rule could claim — as an + # article or as a supplement — is withheld from it, so the user's own naming + # always decides first and content only fills the gap it leaves. + claimed = { + p + for e in entries + for p in _provided_candidates(e, provided_dir) + _supplement_candidates(e, provided_dir) + } + identified, _unidentified = identify_by_content(entries, provided_dir, claimed) + articles: dict[str, Identified] = {} + supplements: dict[str, list[Identified]] = {} + # Two files can honestly identify as the same reference — a duplicate copy, + # or a full text beside a truncated one. `setdefault` over a sorted glob + # settled that by ALPHABETICAL ORDER, so a title match could beat a DOI + # match and the audit depended on what the files happened to be called: the + # filesystem-order defect `_provided_candidates` already fixed once. A DOI + # is exact identity and a title is a token-overlap judgement, so the exact + # signal wins; name order breaks the remaining tie so the answer is the same + # on every machine. + for found in sorted(identified.values(), key=lambda f: (f.signal != "DOI", f.path.name)): + if found.kind == "article": + articles.setdefault(found.entry.num, found) + else: + supplements.setdefault(found.entry.num, []).append(found) with _client() as client: for entry in entries: - resolve_entry(entry, dest_dir, email, client, provided_dir) + resolve_entry(entry, dest_dir, email, client, provided_dir, + content_match=articles.get(entry.num)) + # after resolution, never before: whether a supplement may attach at + # all depends on the status `resolve_entry` just decided + attach_supplements(entry, provided_dir, taken, + content=supplements.get(entry.num, [])) if progress: progress(entry) return entries diff --git a/src/papertrace/report.py b/src/papertrace/report.py index e38f162..9c041b7 100644 --- a/src/papertrace/report.py +++ b/src/papertrace/report.py @@ -9,10 +9,12 @@ from __future__ import annotations import shutil +from collections.abc import Sequence +from functools import partial from importlib import resources from pathlib import Path -from jinja2 import Environment, FileSystemLoader, select_autoescape +from jinja2 import Environment, FileSystemLoader from . import __version__ from .disclosures import ( @@ -27,23 +29,72 @@ TEMPLATES = Path(str(resources.files("papertrace") / "templates")) +def _autoescape(name: str | None) -> bool: + """Escape interpolations in the HTML looks, never in the markdown one. + + Matched on `.html.j2`, not by `select_autoescape(["html"])`, which tests for + a name ending in `.html` — these templates end in `.j2`, so nothing ever + matched and every format rendered unescaped. It stayed invisible because the + one field that carries angle brackets, a Europe PMC title, arrives + pre-escaped from the API; decoding those entities is what made it reachable. + + Cited source PDFs are downloaded from third parties and their text reaches + the report, so this is not hypothetical. No template interpolation is meant + to emit markup — there is no `|safe` anywhere — so escaping every one of + them is the whole fix. Markdown is not HTML and is left alone. + """ + return bool(name) and name.endswith((".html.j2", ".htm.j2")) + + def _env() -> Environment: return Environment( loader=FileSystemLoader(TEMPLATES), - autoescape=select_autoescape(["html"]), + autoescape=_autoescape, trim_blocks=True, lstrip_blocks=True, ) +# the looks a caller may ask for. "md" is the audit's record; the other two are +# for sharing and for the screenshots. Published as a tuple so the CLI's help +# text and the validation below cannot drift apart. +FORMATS = ("md", "editor", "terminal") +_HTML_FORMATS = ("editor", "terminal") + + def write_reports( results: RunResults, manifest: RefManifest | None, out_dir: Path, - png: bool = True, + png: bool = False, scout: ScoutResults | None = None, + formats: Sequence[str] = FORMATS, ) -> list[Path]: - """Write report.md + both HTML looks (+ PNGs if possible). Returns paths.""" + """Write report.md and any requested HTML looks (+ PNGs if possible). + + `formats` defaults to every look because this is the seam the disclosure- + parity suite drives, and that suite has to render all three or it stops + comparing anything. The narrower default belongs to the CLI, where the + user's intent is. `report.md` is written regardless of what was asked for: + it is the record of the audit, not one presentation of it among three. + + An unrecognised name raises. Ignoring it would answer `--format pdf` with a + folder containing no PDF and no complaint — the same silent-downgrade shape + `ingest_pdf` refuses for an unknown backend. + """ + if unknown := [f for f in formats if f not in FORMATS]: + raise ValueError( + f"unknown report format(s) {', '.join(map(repr, unknown))} — " + f"expected any of {', '.join(FORMATS)}" + ) + if not formats: + # `report.md` is written either way, so an empty request contradicts + # itself. Say `("md",)` and mean it. + raise ValueError(f"no report format requested — expected any of {', '.join(FORMATS)}") + # PNG is a screenshot OF the HTML, so asking for one without the other + # cannot be honoured literally: it would render nothing and say nothing. + html = [f for f in _HTML_FORMATS if f in formats] or (list(_HTML_FORMATS) if png else []) + out_dir.mkdir(parents=True, exist_ok=True) env = _env() @@ -64,7 +115,10 @@ def write_reports( # disclosures are decided here, once, and only styled by the templates — # a format cannot silently drop one without failing the parity test "disclosures": run_disclosures(results, manifest), - "claim_disclosures": claim_disclosures, + # bound here, not in the templates: the numbering taint is the only + # claim-level disclosure that needs the manifest, and three templates + # each threading a second argument is three chances to drop it + "claim_disclosures": partial(claim_disclosures, manifest=manifest), "anchor_state": anchor_state, "judgement_disclosures": judgement_disclosures, } @@ -75,16 +129,18 @@ def write_reports( (out_dir / "report.md").write_text(md) written.append(out_dir / "report.md") - # bundle fonts next to the HTML so the pages are self-contained + # bundle fonts next to the HTML so the pages are self-contained — and only + # then: ~1 MB of typefaces beside a markdown file is litter assets_src = TEMPLATES / "assets" assets_dst = out_dir / "assets" - if assets_src.exists(): + if html and assets_src.exists(): shutil.copytree(assets_src, assets_dst, dirs_exist_ok=True) - for name in ("report_editor", "report_terminal"): - html = env.get_template(f"{name}.html.j2").render(**ctx) + for look in html: + name = f"report_{look}" + rendered = env.get_template(f"{name}.html.j2").render(**ctx) html_path = out_dir / f"{name}.html" - html_path.write_text(html) + html_path.write_text(rendered) written.append(html_path) if png: from .render import html_to_png diff --git a/src/papertrace/scout.py b/src/papertrace/scout.py index 6276a5b..d5d1fda 100644 --- a/src/papertrace/scout.py +++ b/src/papertrace/scout.py @@ -1,11 +1,15 @@ """Scout the literature around a paper — what its reference list doesn't know. -Two registers, both candidates for the user's judgement, never accusations: +Three registers, all candidates for the user's judgement, never accusations: - ``newer`` — appeared after the paper: articles that cite it, plus later keyword hits. What the paper could not have known. -- ``overlooked`` — existed by the paper's year but is absent from its +- ``overlooked`` — in print *before* the paper's year and absent from its reference list. What it could have cited. +- ``same_year`` — the paper's own year. Split out because it answers neither + question: it may have appeared after submission, so it is + not a citation the authors owed, and it did not come after, + so it is not literature published since. Search-based (Europe PMC) and therefore incomplete by construction — absence from these lists proves nothing. Network failures soft-fail: the error is @@ -15,12 +19,20 @@ from __future__ import annotations import datetime +import html import re from pathlib import Path import httpx -from .models import RefManifest, ScoutHit, ScoutResults, SourceMap +from .models import ( + RefManifest, + ScoutHit, + ScoutResults, + SourceMap, + paper_title, + titles_match, +) from .refs import UA EPMC = "https://www.ebi.ac.uk/europepmc/webservices/rest" @@ -31,6 +43,13 @@ "the", "and", "for", "with", "from", "into", "using", "based", "toward", "towards", "study", "analysis", "review", "novel", "between", "among", "their", "this", "that", "after", "before", "during", "versus", + # verbs and framing nouns that state what a paper CLAIMS, not what it is + # about. `improves` matched a stroke abstract shouting "IMPROVES" at a + # pancreatic-cancer paper, which is how this list grew. + "improve", "improves", "improved", "improving", "improvement", + "increase", "increases", "increased", "reduce", "reduces", "reduced", + "enhance", "enhances", "enhanced", "enables", "enabling", + "assessment", "evaluation", "comparison", "investigation", } @@ -64,13 +83,35 @@ def _norm_title(title: str) -> str: def _keywords(title: str, n: int = 4) -> list[str]: + """The n most specific-looking words of a title, for the neighbour search. + + Ranked by length, not by position. Taking the first n searched the opening + of the title and never reached its subject: "Image registration improves + inter-reader agreement ... in CT assessment of pancreas adenocarcinoma" + produced `image AND registration AND improves AND inter-reader`, so the + query described a method and omitted the disease entirely. + + Length is a proxy for topical specificity and nothing more — `adenocarcinoma` + over `image`. It is a heuristic, but it is one rule rather than a word list + that has to grow with every title style. The stop list only holds words that + carry no topic in any paper; guessing at more is how a filter starts + dropping real subject terms. + """ words = re.findall(r"[A-Za-z][A-Za-z\-]{3,}", title.lower()) - return [w for w in words if w not in _STOPWORDS][:n] + seen: dict[str, int] = {} + for i, w in enumerate(words): + if w not in _STOPWORDS and w not in seen: + seen[w] = i + ranked = sorted(seen, key=lambda w: (-len(w), seen[w])) + return ranked[:n] def _hit(d: dict, via: str) -> ScoutHit: + # Europe PMC escapes the markup its titles carry, so `CTV<sub>boost</sub>` + # arrives as `CTV<sub>boost</sub>` and was rendered verbatim + # into the report. Decoded once, here, where every hit is built. return ScoutHit( - title=" ".join((d.get("title") or "").split()).rstrip("."), + title=" ".join(html.unescape(d.get("title") or "").split()).rstrip("."), year=_year(d.get("pubYear")), doi=(d.get("doi") or "").lower(), via=via, @@ -150,19 +191,15 @@ def _resolve_paper(client: httpx.Client, doi: str | None, title: str) -> dict | def _title_from_case(case: Path) -> str: - """Best-effort paper title from the ingest output — the first substantial - section header, else the first substantial text block. `--doi` overrides.""" + """The paper's title from the ingest output on disk. `--doi` overrides. + + The rule itself lives in `models.paper_title`, because `refs` needs the same + title to ask whether a Crossref record is this paper. + """ smap_path = case / "ingest" / "manuscript" / "source_map.json" if not smap_path.exists(): return "" - smap = SourceMap.from_json(smap_path) - for b in smap.blocks: - if b.type == "sectionheader" and len(b.text.strip()) >= 15: - return " ".join(b.text.split())[:220] - for b in smap.blocks: - if b.type == "text" and len(b.text.strip()) >= 25: - return " ".join(b.text.split())[:220] - return "" + return paper_title(SourceMap.from_json(smap_path)) # --------------------------------------------------------------------------- @@ -189,16 +226,51 @@ def scout_case( with _client(email, transport) as client: paper = _resolve_paper(client, doi, _title_from_case(case)) if paper is None: - res.error = ( - "paper not identified in Europe PMC — pass --doi to pin it " - "(title heuristics can miss)" - ) + # Which failure this was decides what the reader should do, and + # the two are not the same fact. Telling an operator who just + # passed --doi to pass --doi sent them to verify by hand what + # the tool already knew. + if doi: + res.paper_doi = doi # so the artifact shows what was tried + res.error = ( + f"Europe PMC returned no record for DOI {doi}, so this paper is " + "not indexed there — usual for an in-press or pre-proof article. " + "Both registers below are empty for want of a starting point, " + "which is absence of data, not a clean literature search" + ) + else: + res.error = ( + "paper not identified in Europe PMC — pass --doi to pin it " + "(title heuristics can miss)" + ) return res res.paper_title = paper["title"] res.paper_doi = paper["doi"] res.paper_year = paper["year"] res.resolved_via = paper["via"] + # Is the record this paper? `resolved_via == "doi"` used to stand in + # for "identified reliably", and it stopped meaning that when `run` + # began reading the DOI off page 1 — a funder, data-availability or + # erratum DOI resolves to somebody else's paper, and both registers + # would then describe that paper while the artifact said `doi`. + own_title = _title_from_case(case) + identity = titles_match(own_title, paper["title"]) + res.paper_identity = ( + "confirmed" if identity else "mismatch" if identity is False else "unverified" + ) + if identity is False: + # The registers ARE the finding, so they are not built from a + # record this tool can see is not the paper. Empty-and-disclosed, + # like every other unreadable source here. + res.error = ( + f"the DOI {doi} resolves to \u201c{paper['title']}\u201d, which is " + "not this paper — nothing was scanned, because both registers would " + "have described that paper instead. Check the DOI on the paper's " + "first page, or pass the right one with --doi" + ) + return res + self_keys = {k for k in (paper["doi"], _norm_title(paper["title"])) if k} seen: set[str] = set() @@ -225,13 +297,20 @@ def scout_case( continue # undatable → can't be placed honestly if res.paper_year and h.year > res.paper_year: res.newer.append(h) - elif not _probably_cited(h, cited_dois, cited_slugs): + elif _probably_cited(h, cited_dois, cited_slugs): + continue + elif res.paper_year and h.year == res.paper_year: + # its own year is neither "since" nor "should have + # known" — see ScoutResults for why it gets a register + res.same_year.append(h) + else: res.overlooked.append(h) - res.newer.sort(key=lambda h: (-(h.year or 0), h.title)) - res.overlooked.sort(key=lambda h: (-(h.year or 0), h.title)) + for reg in (res.newer, res.overlooked, res.same_year): + reg.sort(key=lambda h: (-(h.year or 0), h.title)) res.newer = res.newer[:NEWER_CAP] res.overlooked = res.overlooked[:OVERLOOKED_CAP] + res.same_year = res.same_year[:OVERLOOKED_CAP] except httpx.HTTPError as e: res.error = f"network: {type(e).__name__} — scan incomplete" return res diff --git a/src/papertrace/templates/report.md.j2 b/src/papertrace/templates/report.md.j2 index c296400..98aa1fe 100644 --- a/src/papertrace/templates/report.md.j2 +++ b/src/papertrace/templates/report.md.j2 @@ -26,14 +26,37 @@ Manuscript: `{{ r.manuscript }}` · Sources: `{{ r.refs_available }} / {{ r.refs ## Claim {{ c.id }}: "{{ c.claim }}" -**Status:** {{ c.label }} +{# an inline expression, NOT {% if %}: trim_blocks eats the newline after a + block tag, which merged this line into the Location line below it #} +**Status:** {{ c.label }}{{ " — *%s*" % c.headline_qualifier() if c.headline_qualifier() else "" }} **Location:** {{ c.location }}{% if c.refs %} · cites [{{ c.refs|join(', ') }}]{% endif %} +{% if c.quote %} +{# two blank lines, not one: trim_blocks eats the newline after each block tag, + and a blockquote wants a blank line ahead of it rather than relying on a + renderer letting it interrupt the paragraph above. trim_blocks eats the + newline after this comment too, so there are two blanks below, not one #} + + +> {{ c.quote }} +{% endif %} {% for d in claim_disclosures(c) if d.key == "sources" %} > **{{ d.text }}** {% endfor %} +{% for d in claim_disclosures(c) if d.key == "no_quote" %} + +> ⚠️ {{ d.text }} +{% endfor %} +{% for d in claim_disclosures(c) if d.key == "supplement_headline" %} + +> ⚠️ {{ d.text }} +{% endfor %} {% for d in claim_disclosures(c) if d.key == "unjudged_refs" %} +> ⚠️ {{ d.text }} +{% endfor %} +{% for d in claim_disclosures(c) if d.key == "claim_numbering" %} + > ⚠️ {{ d.text }} {% endfor %} {% if c.judgements %} @@ -41,7 +64,7 @@ Manuscript: `{{ r.manuscript }}` · Sources: `{{ r.refs_available }} / {{ r.refs <br> -### {{ j.label }} — `{{ j.source_slug }}` (cited as [{{ j.ref }}]) +### {{ j.label }} — `{{ j.source_slug }}` ({{ j.origin }}) {% if j.source_page %} - **Source:** Page {{ j.source_page }}{% if j.source_block %} `({{ j.source_block }})` {% endif %} @@ -53,8 +76,13 @@ Manuscript: `{{ r.manuscript }}` · Sources: `{{ r.refs_available }} / {{ r.refs {% if j.evidence_image %} ![evidence]({{ j.evidence_image }}) -{% for d in judgement_disclosures(j) %}{% if d.level == "warn" %}*⚠️ {{ d.text }}*{% else %}*{{ d.token }}*{% endif %}{% endfor %} {% endif %} +{% for d in judgement_disclosures(j) %} +{% if not j.evidence_image %} + +{% endif %} +{% if d.level == "warn" %}*⚠️ {{ d.text }}*{% else %}*{{ d.token }}*{% endif %} +{% endfor %} {% endfor %} {% else %} {% if c.source_slug %} @@ -65,8 +93,13 @@ Manuscript: `{{ r.manuscript }}` · Sources: `{{ r.refs_available }} / {{ r.refs {% if c.evidence_image %} ![evidence]({{ c.evidence_image }}) -{% for d in claim_disclosures(c) if d.key == "anchor" %}{% if d.level == "warn" %}*⚠️ {{ d.text }}*{% else %}*{{ d.token }}*{% endif %}{% endfor %} {% endif %} +{% for d in claim_disclosures(c) if d.key == "anchor" %} +{% if not c.evidence_image %} + +{% endif %} +{% if d.level == "warn" %}*⚠️ {{ d.text }}*{% else %}*{{ d.token }}*{% endif %} +{% endfor %} {% endif %} {% if c.note %} @@ -86,7 +119,16 @@ Reported as such — never filled in from memory. {% for section, items in gaps.items() %} - **{{ section }}** ({{ items|length }}): {% for c in items %} - - [{{ c.refs|join(', ') }}] {{ c.claim }}{% if c.note %} — *{{ c.note }}*{% endif %} + - {{ c.label }} · [{{ c.refs|join(', ') }}] {{ c.claim }}{% if c.note %} — *{{ c.note }}*{% endif +%} +{% for d in claim_disclosures(c) if d.key in ("sources", "unjudged_refs", "claim_numbering") %} + - {{ '⚠️ ' if d.level == 'warn' else '' }}{{ d.text }} +{% endfor %} +{% for j in c.judgements %} + - `{{ j.source_slug }}` {{ j.origin }} — **{{ j.verdict }}**{% if j.source_page %} · p.{{ j.source_page }}{% endif %}{% if j.note %} — *{{ j.note }}*{% endif +%} +{% for d in judgement_disclosures(j) %} + - {{ '⚠️ ' if d.level == 'warn' else '' }}{{ d.text }} +{% endfor %} +{% endfor %} {% endfor %} {% endfor %} {% if r.uncited %} @@ -97,15 +139,15 @@ Statements that would normally carry a reference but don't. Not verified — flagged for you to weigh. {% for u in r.uncited %} -- **[U{{ u.id }}]** {{ u.claim }}{% if u.location %} *({{ u.location }})*{% endif %} +- **[U{{ u.id }}]** {{ u.quote or u.claim }}{% if u.location %} *({{ u.location }})*{% endif %} {% endfor %} {% endif %} -{% if scout and (scout.newer or scout.overlooked or scout.error) %} +{% if scout and (scout.newer or scout.overlooked or scout.same_year or scout.error) %} ## Literature scout — what the reference list doesn't know {% if scout.paper_title %} -Paper identified as *{{ scout.paper_title }}* ({{ scout.paper_year or "year?" }}{% if scout.paper_doi %} · `{{ scout.paper_doi }}`{% endif %}) via {{ scout.resolved_via }} lookup on Europe PMC. +Paper identified as *{{ scout.paper_title }}* ({{ scout.paper_year or "year?" }}{% if scout.paper_doi %} · `{{ scout.paper_doi }}`{% endif %}) via {{ scout.resolved_via }} lookup on Europe PMC{% if scout.paper_identity == 'confirmed' %}, and its title matches this paper{% elif scout.paper_identity == 'unverified' %} — **nobody established that this record is your paper**: there was too little title to compare, so read the two registers below with that in mind{% elif scout.paper_identity == 'mismatch' %} — **this record is not your paper**{% endif %}. {% endif %} {% if scout.error %} @@ -127,8 +169,8 @@ What the paper could not have known — articles citing it, plus later keyword h ### Existed but uncited ({{ scout.overlooked|length }} candidates) -In print by the paper's year and absent from its reference list (matched by -DOI and first-author heuristics — verify against the reference list yourself). +In print **before** the paper's year and absent from its reference list (matched +by DOI and first-author heuristics — verify against the reference list yourself). | Year | Title | Journal | DOI | |------|-------|---------|-----| @@ -136,6 +178,22 @@ DOI and first-author heuristics — verify against the reference list yourself). | {{ h.year or "?" }} | {{ h.title[:90] }} | {{ h.journal[:30] }} | {{ h.doi or "—" }} | {% endfor %} {% endif %} +{% if scout.same_year %} + +### Same year as the paper ({{ scout.same_year|length }} candidates) + +Published in {{ scout.paper_year or "the paper's own year" }} and not cited. +Held apart from the list above on purpose: a paper from the manuscript's own +year may have appeared after it was submitted, so it is **not** evidence of +something the authors should have known. It is also not literature published +since. Judge each on its date. + +| Year | Title | Journal | DOI | +|------|-------|---------|-----| +{% for h in scout.same_year %} +| {{ h.year or "?" }} | {{ h.title[:90] }} | {{ h.journal[:30] }} | {{ h.doi or "—" }} | +{% endfor %} +{% endif %} *Search-based (Europe PMC{% if scout.query %}, query `{{ scout.query }}`{% endif %}) — absence from these lists proves nothing, and presence is a candidate for your judgement, not an accusation.* diff --git a/src/papertrace/templates/report_editor.html.j2 b/src/papertrace/templates/report_editor.html.j2 index 5e42f7c..bc74c08 100644 --- a/src/papertrace/templates/report_editor.html.j2 +++ b/src/papertrace/templates/report_editor.html.j2 @@ -58,6 +58,7 @@ figure{margin:2px 0 0;} .crop{display:block;max-width:600px;max-height:230px;border:1px solid var(--edge);border-radius:3px;background:#fff;} figcaption{font-family:"Mono",monospace;font-size:11px;color:var(--dim);margin-top:5px;} + .mquote{font-size:13.5px;color:#c8c8c8;margin:0 0 11px;padding:7px 0 7px 14px;border-left:3px solid #4a4a4a;line-height:1.55;max-width:850px;font-style:italic;} .cap{font-size:13px;color:var(--muted);margin:9px 0 0;padding-left:14px;border-left:3px solid var(--edge);line-height:1.5;max-width:850px;} .cap b{color:#d0d0d0;} .gap-intro{font-size:13.5px;color:var(--muted);margin:5px 0 12px;line-height:1.5;} @@ -124,16 +125,28 @@ {% for c in checked %} <h2><span class="hx">##</span> Claim {{ c.id }}: <span class="q">“{{ c.claim }}”</span></h2> - <div class="status {{ c.verdict }}"><span class="lbl">Status:</span> <span class="val">{{ c.label }}</span></div> + <div class="status {{ c.verdict }}"><span class="lbl">Status:</span> <span class="val">{{ c.label }}</span>{% if c.headline_qualifier() %} <span class="cap">— {{ c.headline_qualifier() }}</span>{% endif %}</div> +{% if c.quote %} + <blockquote class="mquote">{{ c.quote }}</blockquote> +{% endif %} {% for d in claim_disclosures(c) if d.key == "sources" %} <p class="src-summary"><b>{{ d.text }}</b></p> {% endfor %} +{% for d in claim_disclosures(c) if d.key == "no_quote" %} + <p class="cap">⚠️ {{ d.text }}</p> +{% endfor %} +{% for d in claim_disclosures(c) if d.key == "supplement_headline" %} + <p class="cap">⚠️ {{ d.text }}</p> +{% endfor %} {% for d in claim_disclosures(c) if d.key == "unjudged_refs" %} <p class="cap"><span style="color:var(--amber)">⚠ {{ d.text }}</span></p> {% endfor %} +{% for d in claim_disclosures(c) if d.key == "claim_numbering" %} + <p class="cap"><span style="color:var(--amber)">⚠ {{ d.text }}</span></p> +{% endfor %} {% if c.judgements %} {% for j in c.judgements %} - <h3><span class="hx">###</span> <span class="status {{ j.verdict }}">{{ j.label }}</span> — <span class="slug">{{ j.source_slug }}</span> <span class="cap">cited as [{{ j.ref }}]</span></h3> + <h3><span class="hx">###</span> <span class="status {{ j.verdict }}">{{ j.label }}</span> — <span class="slug">{{ j.source_slug }}</span> <span class="cap">{{ j.origin }}</span></h3> {% if j.source_page %} <ul class="src"><li><b>Source:</b> Page {{ j.source_page }}{% if j.source_block %} <code>({{ j.source_block }})</code>{% endif %}</li></ul> {% endif %} @@ -145,6 +158,10 @@ <img class="crop" src="{{ j.evidence_image }}"> <figcaption>![evidence] {{ j.source_slug }} · page {{ j.source_page }} · {% for d in judgement_disclosures(j) %}{% if d.level == "warn" %}<span style="color:var(--amber)">⚠ {{ d.text }}</span>{% else %}{{ d.token }}{% endif %}{% endfor %}</figcaption> </figure> +{% else %} +{% for d in judgement_disclosures(j) %} + <p class="cap"><span style="color:var(--{{ 'amber' if d.level == 'warn' else 'fg' }})">{{ '⚠ ' if d.level == 'warn' else '' }}{{ d.text }}</span></p> +{% endfor %} {% endif %} {% endfor %} {% else %} @@ -157,6 +174,10 @@ <img class="crop" src="{{ c.evidence_image }}"> <figcaption>![evidence] {{ c.source_slug }} · page {{ c.source_page }} · {% for d in claim_disclosures(c) if d.key == "anchor" %}{% if d.level == "warn" %}<span style="color:var(--amber)">⚠ {{ d.text }}</span>{% else %}{{ d.token }}{% endif %}{% endfor %}</figcaption> </figure> +{% else %} +{% for d in claim_disclosures(c) if d.key == "anchor" %} + <p class="cap"><span style="color:var(--{{ 'amber' if d.level == 'warn' else 'fg' }})">{{ '⚠ ' if d.level == 'warn' else '' }}{{ d.text }}</span></p> +{% endfor %} {% endif %} {% if c.note %} <p class="cap">{{ c.note }}</p> @@ -169,7 +190,7 @@ <p class="gap-intro">Statements that would normally carry a reference but don't. Not verified — flagged for you to weigh.</p> <ul class="src"> {% for u in r.uncited %} - <li><b>[U{{ u.id }}]</b> {{ u.claim }}{% if u.location %} <code>{{ u.location }}</code>{% endif %}</li> + <li><b>[U{{ u.id }}]</b> {{ u.quote or u.claim }}{% if u.location %} <code>{{ u.location }}</code>{% endif %}</li> {% endfor %} </ul> {% endif %} @@ -178,12 +199,29 @@ <p class="gap-intro">Either the cited PDF could not be obtained, or the check step failed on an available source (see notes). Reported as such — never filled in from memory.</p> <table class="gaptbl"> +{# one row per CLAIM, not per section: `items[0].verdict` labelled a section + holding one not_retrieved and one unchecked as whichever came first #} {% for section, items in gaps.items() %} - <tr><td class="sec">{{ section }}</td><td class="st">{{ items[0].verdict|replace("_", " ") }} · {{ items|length }}</td> - <td class="it">{% for c in items %}{{ c.claim }}{% if not loop.last %}<span class="s">·</span>{% endif %}{% endfor %}</td></tr> +{% for c in items %} + <tr><td class="sec">{{ section if loop.first else "" }}</td><td class="st">{{ c.verdict|replace("_", " ") }}</td> + <td class="it">{{ c.claim }} +{% if c.note %} + <div class="cap">{{ c.note }}</div> +{% endif %} +{% for d in claim_disclosures(c) if d.key in ("sources", "unjudged_refs", "claim_numbering") %} + <div class="cap"><span style="color:var(--{{ 'amber' if d.level == 'warn' else 'fg' }})">{{ '⚠ ' if d.level == 'warn' else '' }}{{ d.text }}</span></div> +{% endfor %} +{% for j in c.judgements %} + <div class="cap"><span class="slug">{{ j.source_slug }}</span> {{ j.origin }} — <span class="status {{ j.verdict }}">{{ j.verdict }}</span>{% if j.source_page %} · p.{{ j.source_page }}{% endif %}{% if j.note %} — {{ j.note }}{% endif %}</div> +{% for d in judgement_disclosures(j) %} + <div class="cap"><span style="color:var(--{{ 'amber' if d.level == 'warn' else 'fg' }})">{{ '⚠ ' if d.level == 'warn' else '' }}{{ d.text }}</span></div> +{% endfor %} +{% endfor %} + </td></tr> +{% endfor %} {% endfor %} </table> -{% if scout and (scout.newer or scout.overlooked or scout.error) %} +{% if scout and (scout.newer or scout.overlooked or scout.same_year or scout.error) %} <h2><span class="hx">##</span> Literature scout <span class="q">— what the reference list doesn't know</span></h2> <p class="gap-intro">Candidates for your judgement, not accusations — search-based (Europe PMC{% if scout.paper_year %}, paper year {{ scout.paper_year }}{% endif %}), @@ -194,11 +232,19 @@ <td class="it">{{ h.title }}{% if h.journal %} <span class="s">·</span> {{ h.journal }}{% endif %}{% if h.doi %} <span class="s">·</span> <code>{{ h.doi }}</code>{% endif %}</td></tr> {% endfor %} {% for h in scout.overlooked[:8] %} - <tr><td class="sec">uncited · {{ h.year or "?" }}</td><td class="st">existed at pub. time</td> + <tr><td class="sec">uncited · {{ h.year or "?" }}</td><td class="st">in print before pub.</td> + <td class="it">{{ h.title }}{% if h.journal %} <span class="s">·</span> {{ h.journal }}{% endif %}{% if h.doi %} <span class="s">·</span> <code>{{ h.doi }}</code>{% endif %}</td></tr> +{% endfor %} +{% for h in scout.same_year[:8] %} + <tr><td class="sec">same year · {{ h.year or "?" }}</td><td class="st">may postdate submission</td> <td class="it">{{ h.title }}{% if h.journal %} <span class="s">·</span> {{ h.journal }}{% endif %}{% if h.doi %} <span class="s">·</span> <code>{{ h.doi }}</code>{% endif %}</td></tr> {% endfor %} </table> -{% if scout.newer|length > 10 or scout.overlooked|length > 8 %} +{% if scout.same_year %} + <p class="gap-intro" style="margin-top:8px">Same-year candidates are listed apart on purpose: one from the paper's own year may + have appeared after submission, so it is neither literature published since nor something the authors should have known.</p> +{% endif %} +{% if scout.newer|length > 10 or scout.overlooked|length > 8 or scout.same_year|length > 8 %} <p class="gap-intro" style="margin-top:8px">Full lists in <code>scout.json</code>.</p> {% endif %} {% endif %} diff --git a/src/papertrace/templates/report_terminal.html.j2 b/src/papertrace/templates/report_terminal.html.j2 index 62d8c85..e938b58 100644 --- a/src/papertrace/templates/report_terminal.html.j2 +++ b/src/papertrace/templates/report_terminal.html.j2 @@ -37,6 +37,7 @@ .cl-meta .st{font-weight:700;} .cl-meta .from{color:var(--blue);} .cl-meta .loc{color:var(--dim);} + .cl-meta .it{color:#adbac7;line-height:1.5;} .st.supported{color:var(--green);} .st.partial{color:var(--amber);} .st.contradicted{color:var(--red);} .dot{font-size:12px;vertical-align:1px;} .imgwrap{border:1px solid var(--edge);border-radius:4px;overflow:hidden;width:600px;background:#fff;position:relative;margin:2px 0 2px;} @@ -75,6 +76,15 @@ {% for d in disclosures if d.key == "converter" %} <div class="log"><span class="step">▸ ingest</span><span class="cyan">{{ d.token }}</span>{% if d.level == "warn" %}<span class="k"> · </span><span class="amber">⚠ {{ d.short }}</span>{% endif %}</div> {% endfor %} +{% for d in disclosures if d.key == "source_fidelity" %} + <div class="log"><span class="step">▸ sources</span><span class="amber">⚠ {{ d.short }}</span></div> +{% endfor %} +{% for d in disclosures if d.key == "supplement_identity" %} + <div class="log"><span class="step">▸ sources</span><span class="amber">⚠ {{ d.short }}</span></div> +{% endfor %} +{% for d in disclosures if d.key == "supplement_coverage" %} + <div class="log"><span class="step">▸ coverage</span><span class="dim">{{ d.short }}</span></div> +{% endfor %} {% for d in disclosures if d.key == "truncation" %} <div class="log"><span class="step">▸ limits</span><span class="amber">⚠ {{ d.short }}</span></div> {% endfor %} @@ -83,6 +93,9 @@ {% endfor %} {% for d in disclosures if d.key == "references_resumed" %} <div class="log"><span class="step">▸ resolve</span><span class="amber">⚠ {{ d.short }}</span></div> +{% endfor %} +{% for d in disclosures if d.key == "numbering" %} + <div class="log"><span class="step">▸ resolve</span><span class="amber">⚠ {{ d.short }}</span></div> {% endfor %} <div class="log"><span class="step">▸ check</span><span class="k">{{ r.claims|length }} citation-backed claims · reading each against its cited page{% if r.uncited %} · </span><span class="amber">{{ r.uncited|length }} uncited assertions flagged</span><span class="k">{% endif %}</span></div> {% if scout and not scout.error %} @@ -97,35 +110,43 @@ {% for d in claim_disclosures(c) if d.key == "sources" %} <div class="cl-meta"><span class="{{ 'amber' if d.level == 'warn' else 'dim' }}">{{ d.short }}</span></div> {% endfor %} {% for d in claim_disclosures(c) if d.key == "unjudged_refs" %} <div class="cl-meta"><span class="amber">⚠ {{ d.short }}</span></div> +{% endfor %} +{% for d in claim_disclosures(c) if d.key == "no_quote" %} <div class="cl-meta"><span class="amber">⚠ {{ d.short }}</span></div> +{% endfor %} +{% for d in claim_disclosures(c) if d.key == "supplement_headline" %} <div class="cl-meta"><span class="amber">⚠ {{ d.short }}</span></div> +{% endfor %} +{% if c.quote %} <div class="cl-meta"><span class="lab">quote </span><span class="it">{{ c.quote }}</span></div> +{% endif %} +{% for d in claim_disclosures(c) if d.key == "claim_numbering" %} <div class="cl-meta"><span class="amber">⚠ {{ d.short }}</span></div> {% endfor %} <div class="cl-meta"> <span style="display:inline-block;width:190px"><span class="lab">status </span><span class="st {{ c.verdict }}"><span class="dot">●</span> {{ c.verdict|upper }}</span></span> - {% if c.source_slug %}<span class="lab">from </span><span class="from">{{ c.source_slug }}</span> + {% if c.headline_qualifier() %}<span class="loc">{{ c.headline_qualifier() }} · </span>{% endif %}{% if c.source_slug %}<span class="lab">from </span><span class="from">{{ c.source_slug }}</span> <span class="loc"> · p.{{ c.source_page }}{% if c.source_block %} · {{ c.source_block }}{% endif %}</span>{% endif %} </div> {% if c.judgements %} {% for j in c.judgements %} <div class="cl-meta"> - <span style="display:inline-block;width:190px"><span class="lab">[{{ j.ref }}] </span><span class="st {{ j.verdict }}"><span class="dot">●</span> {{ j.verdict|upper }}</span></span> - <span class="from">{{ j.source_slug }}</span>{% if j.source_page %}<span class="loc"> · p.{{ j.source_page }}</span>{% endif %} + <span style="display:inline-block;width:190px"><span class="lab">{% if j.kind == "article" %}[{{ j.ref }}] {% else %}[+] {% endif %}</span><span class="st {{ j.verdict }}"><span class="dot">●</span> {{ j.verdict|upper }}</span></span> + <span class="from">{{ j.source_slug }}</span>{% if j.kind != "article" %}<span class="loc"> · {{ j.origin }}</span>{% endif %}{% if j.source_page %}<span class="loc"> · p.{{ j.source_page }}</span>{% endif %} </div> {% if j.note %} <div class="note"><span class="a">└─</span> {{ j.note }}</div> {% endif %} {% if j.evidence_image %} <div class="imgwrap" data-src="{{ j.source_slug }} · page {{ j.source_page }}"><img src="{{ j.evidence_image }}"></div> +{% endif %} {% for d in judgement_disclosures(j) %} <div class="cl-meta"><span class="{{ 'amber' if d.level == 'warn' else 'dim' }}">{{ '⚠ ' if d.level == 'warn' else '' }}{{ d.short }}</span></div> {% endfor %} -{% endif %} {% endfor %} {% else %} {% if c.evidence_image %} <div class="imgwrap" data-src="{{ c.source_slug }} · page {{ c.source_page }}"><img src="{{ c.evidence_image }}"></div> +{% endif %} {% for d in claim_disclosures(c) if d.key == "anchor" %} <div class="cl-meta"><span class="{{ 'amber' if d.level == 'warn' else 'dim' }}">{{ '⚠ ' if d.level == 'warn' else '' }}{{ d.short }}</span></div> {% endfor %} -{% endif %} {% if c.note %} <div class="note"><span class="a">└─</span> {{ c.note }}</div> {% endif %} @@ -161,16 +182,27 @@ <div class="nr-head"><span class="amber">▲</span> assertions without citation <span class="c">({{ r.uncited|length }}) — your judgement required</span></div> <div class="nr-sub">would normally carry a reference but don't — not verified, flagged for you to weigh.</div> {% for u in r.uncited %} - <div class="nrrow"><span class="sec">[U{{ u.id }}]</span><span class="n"></span><span class="it">{{ u.claim }}{% if u.location %} <span class="s">·</span> {{ u.location }}{% endif %}</span></div> + <div class="nrrow"><span class="sec">[U{{ u.id }}]</span><span class="n"></span><span class="it">{{ u.quote or u.claim }}{% if u.location %} <span class="s">·</span> {{ u.location }}{% endif %}</span></div> {% endfor %} <div style="height:14px"></div> {% endif %} <div class="nr-head"><span class="dim">○</span> not verified — source not retrieved or check failed <span class="c">({{ gap_total }} / {{ r.claims|length }})</span></div> <div class="nr-sub">the cited PDF couldn’t be fetched — or the check step failed on an available source — so the claim is reported unverified, never guessed.</div> {% for section, items in gaps.items() %} - <div class="nrrow"><span class="sec">{{ section|lower }}</span><span class="n">[{{ items|length }}]</span><span class="it">{% for c in items %}{{ c.claim }}{% if not loop.last %}<span class="s">·</span>{% endif %}{% endfor %}</span></div> +{% for c in items %} + <div class="nrrow"><span class="sec">{{ section|lower if loop.first else "" }}</span><span class="n">{{ c.verdict|replace("_", " ") }}</span><span class="it">{{ c.claim }}{% if c.note %} <span class="s">·</span> {{ c.note }}{% endif %}</span></div> +{% for d in claim_disclosures(c) if d.key in ("sources", "unjudged_refs", "claim_numbering") %} + <div class="nrrow"><span class="sec"></span><span class="n"></span><span class="it {{ 'amber' if d.level == 'warn' else 'dim' }}">{{ '⚠ ' if d.level == 'warn' else '' }}{{ d.short }}</span></div> +{% endfor %} +{% for j in c.judgements %} + <div class="nrrow"><span class="sec"></span><span class="n">{% if j.kind == "article" %}[{{ j.ref }}]{% else %}[+]{% endif %}</span><span class="it dim">{{ j.source_slug }} {% if j.kind != "article" %}<span class="s">·</span> {{ j.origin }} {% endif %}<span class="s">·</span> {{ j.verdict }}{% if j.source_page %} <span class="s">·</span> p.{{ j.source_page }}{% endif %}{% if j.note %} <span class="s">·</span> {{ j.note }}{% endif %}</span></div> +{% for d in judgement_disclosures(j) %} + <div class="nrrow"><span class="sec"></span><span class="n"></span><span class="it {{ 'amber' if d.level == 'warn' else 'dim' }}">{{ '⚠ ' if d.level == 'warn' else '' }}{{ d.short }}</span></div> {% endfor %} -{% if scout and (scout.newer or scout.overlooked or scout.error) %} +{% endfor %} +{% endfor %} +{% endfor %} +{% if scout and (scout.newer or scout.overlooked or scout.same_year or scout.error) %} <div style="height:14px"></div> <div class="nr-head"><span class="cyan">◆</span> literature the reference list doesn't know <span class="c">(scout · europe pmc{% if scout.paper_year %} · paper year {{ scout.paper_year }}{% endif %})</span></div> @@ -187,6 +219,12 @@ {% if scout.overlooked|length > 6 %} <div class="nr-sub" style="margin-top:7px">… {{ scout.overlooked|length - 6 }} more uncited candidates in scout.json</div> {% endif %} +{% for h in scout.same_year[:6] %} + <div class="nrrow"><span class="sec">same year · {{ h.year or "?" }}</span><span class="n">[{{ h.via[:6] }}]</span><span class="it">{{ h.title }}{% if h.journal %} <span class="s">·</span> {{ h.journal }}{% endif %}{% if h.doi %} <span class="s">·</span> {{ h.doi }}{% endif %}</span></div> +{% endfor %} +{% if scout.same_year %} + <div class="nr-sub" style="margin-top:7px">same-year candidates listed apart — one from the paper's own year may postdate submission, so it is neither newer literature nor a citation the authors owed{% if scout.same_year|length > 6 %} · … {{ scout.same_year|length - 6 }} more in scout.json{% endif %}</div> +{% endif %} {% endif %} <div class="foot"># evidence crops are real pages of the cited sources · each crop states whether its anchor phrase was located<br> diff --git a/src/papertrace/wizard.py b/src/papertrace/wizard.py index 912cb42..54cfe6f 100644 --- a/src/papertrace/wizard.py +++ b/src/papertrace/wizard.py @@ -25,9 +25,9 @@ from rich.prompt import Confirm, Prompt from . import config -from .check import _LABEL_GROUP, ASK_ATTEMPTS, _expand_label_group, claude_available +from .check import ASK_ATTEMPTS, claude_available from .ingest import _docling_available as docling_available -from .models import is_references_heading +from .models import _LABEL_GROUP, _expand_label_group, is_references_heading from .refs import DOI_RE console = Console() @@ -107,12 +107,13 @@ def preflight() -> list[Check]: fatal=False, label="layout-aware ingest", detail=( - "docling found — tables and figures are read as structure" + "docling found — the paper and its cited sources are read as structure" if docling_available() - else "flat-text ingest only: tables are linearized and figures " - "are invisible, and every report says so" + else "docling is a required dependency but did not import, so this " + "install is broken: flat-text ingest only, tables linearized and " + "figures invisible, in the sources as well as the paper" ), - fix="pip install 'papertrace[docling]'", + fix="reinstall: pip install --force-reinstall papertrace", ), Check( key="png", @@ -224,6 +225,27 @@ def workload(pdf: Path) -> dict: } +def supplement_workload(provided_dir: Path | None, supplement: list[Path]) -> int: + """How many extra judging calls the supplements on hand will cost. + + One per supplementary *document*, not per claim: `check_claims` groups every + claim for a document into a single call. An upper bound like the rest of + this estimate — a supplement whose reference no claim cites is never opened, + and one whose article turns out to be unavailable is set aside entirely. + + Counted from the folder rather than from the manifest because this runs + before `refs` does. The alternative is to state a price that leaves the + supplements out, and check.py's own comment on ASK_ATTEMPTS is the rule + here: a cost ceiling that gets exceeded is a false promise about money. + """ + from .refs import _SUPPLEMENT_RE + + n = len(supplement or []) + if provided_dir and provided_dir.is_dir(): + n += sum(1 for p in provided_dir.glob("*.pdf") if _SUPPLEMENT_RE.search(p.stem)) + return n + + def equivalent_command( *, manuscript: Path, @@ -233,6 +255,7 @@ def equivalent_command( with_scout: bool, provided: Path | None, email: str | None = None, + supplement: list[Path] | None = None, ) -> str: """The `papertrace run` line this session amounts to. @@ -246,6 +269,8 @@ def equivalent_command( argv = ["papertrace", "run", str(manuscript), "-c", str(case)] if provided: argv += ["--provided", str(provided)] + for s in supplement or []: + argv += ["--supplement", str(s)] if doi: argv += ["--doi", doi] if email: @@ -282,6 +307,74 @@ def _ask_paper() -> Path: return path +def _ask_sources(case: Path) -> Path | None: + """The folder of reference PDFs the user already holds. + + Never asked before this: `run_wizard` passed `provided=None`, so the guided + path could not reach a flag the CLI has had all along, and a first-time user + following the wizard silently got open-access retrieval only. + """ + default = case / "sources" + # the folder's contents answer this better than any fixed default can: a + # user whose PDFs are already sitting there should not skip them by pressing + # return, and a user with none should not be handed a path prompt at all + waiting = default.is_dir() and any(default.glob("*.pdf")) + if not Confirm.ask( + "\n[bold]Do you have any of the cited PDFs already?[/bold]" + + (f"\n [dim]{default} looks like it holds some.[/dim]" if waiting else ""), + default=waiting, + ): + return None + console.print( + " [dim]Point me at a folder. Names do not have to be tidy — each PDF is " + "identified\n by its own title or DOI, so a publisher download works as is. A " + "file named for\n its reference ([cyan]pyrros-2023.pdf[/cyan]) is taken at your " + "word instead.\n Supplementary material for a cited paper goes in the same " + "folder; each is\n judged as its own document, and anything I cannot place I " + "will name.[/dim]" + ) + raw = Prompt.ask(" folder", default=str(default)).strip() + if not raw: + return None + path = clean_path(raw) + if not path.is_dir(): + console.print(f" [dim]No folder at {path} — continuing without one.[/dim]") + return None + n = len(list(path.glob("*.pdf"))) + console.print(f" [green]✓[/green] {n} PDF{'' if n == 1 else 's'} in {path}") + return path + + +def _ask_supplements() -> list[Path]: + """Supplementary material belonging to the paper under audit. + + Asked separately because the sources folder is matched against *reference* + slugs, and the audited paper has none for a filename to key on. + """ + if not Confirm.ask( + "\n[bold]Does this paper have supplementary material of its own?[/bold]", + default=False, + ): + return [] + console.print( + " [dim]A claim that points at Table S3 or eFigure 2 is read against these; " + "with\n nothing supplied it is reported as not retrieved, never guessed.[/dim]" + ) + out: list[Path] = [] + while True: + raw = Prompt.ask( + " path (blank when done)" if out else " path (blank to skip)", default="" + ).strip() + if not raw: + return out + path = clean_path(raw) + if not path.is_file(): + console.print(f" [red]No file at {path}[/red] — try again, or drag it in.") + continue + out.append(path) + console.print(f" [green]✓[/green] {path.name}") + + def _ask_email() -> str: if existing := (config.load().get("email") or "").strip(): console.print(f" Using saved contact email: [cyan]{existing}[/cyan]") @@ -376,6 +469,8 @@ def _n(count: int, noun: str) -> str: Prompt.ask("\n[bold]Where should I keep this audit?[/bold]\n folder", default=_suggest_case(paper)) ) + provided = _ask_sources(case) + supplement = _ask_supplements() doi, with_scout = _ask_doi(paper) email = _ask_email() @@ -383,14 +478,26 @@ def _n(count: int, noun: str) -> str: if png_available: png = Confirm.ask("\n Also export PNG pictures of the reports?", default=False) + # each supplement is one more document, so one more judging call. Folded in + # here rather than in `workload()` because it is not known until the sources + # folder has been named, which happens after the paper is measured. + extra = supplement_workload(provided, supplement) + calls = w["model_calls"] + extra + calls_max = w["model_calls_max"] + ASK_ATTEMPTS * extra + console.print("\n[bold]Ready.[/bold]") console.print( " This makes live requests to Crossref, Unpaywall" + (" and Europe PMC" if with_scout else "") - + f", and about [bold]{w['model_calls']}[/bold] model calls through `claude -p`" - + (f" — up to [bold]{w['model_calls_max']}[/bold] if calls have to be retried." - if w["model_calls_max"] != w["model_calls"] else ".") + + f", and about [bold]{calls}[/bold] model calls through `claude -p`" + + (f" — up to [bold]{calls_max}[/bold] if calls have to be retried." + if calls_max != calls else ".") ) + if extra: + console.print( + f" [dim]{_n(extra, 'supplementary document')} included — each is judged " + "separately from the article it accompanies.[/dim]" + ) if w["multi"]: console.print( f" [dim]{w['multi']} of {w['places']} citation places cite several sources, " @@ -403,15 +510,21 @@ def _n(count: int, noun: str) -> str: cmd = equivalent_command( manuscript=paper, case=case, doi=doi, png=png, - with_scout=with_scout, provided=None, email=email, + with_scout=with_scout, provided=provided, email=email, supplement=supplement, ) console.print(f"\n[dim]Same thing as one command, for next time:[/dim]\n [cyan]{cmd}[/cyan]\n") from .cli import run as run_cmd + # every parameter `run` declares is named here, including the ones taking + # their default: `run` is a Typer command, so an omitted argument arrives as + # an OptionInfo sentinel rather than the default the help screen shows. + # `formats=None` means report.md alone — the PNG answer above already pulls + # in the HTML looks when it needs them, since a PNG is a shot of one. run_cmd( - manuscript=paper, case=case, provided=None, email=email, model=None, - png=png, backend="auto", with_scout=with_scout, doi=doi, + manuscript=paper, case=case, provided=provided, email=email, model=None, + png=png, backend="auto", with_scout=with_scout, doi=doi, formats=None, + supplement=supplement, ) @@ -422,5 +535,6 @@ def _n(count: int, noun: str) -> str: "equivalent_command", "preflight", "run_wizard", + "supplement_workload", "workload", ] diff --git a/tests/test_anchor_state.py b/tests/test_anchor_state.py new file mode 100644 index 0000000..df27d7d --- /dev/null +++ b/tests/test_anchor_state.py @@ -0,0 +1,91 @@ +"""True, False and None are three different facts, and stay three. + +- `True` — an anchor phrase was searched for and located. +- `False` — searched for and NOT located. +- `None` — no search was possible or attempted (no phrases offered, or the + highlight step never ran). + +Two ways the tri-state was being flattened. The CLI branched on truthiness, so +`None` printed "no anchor phrase found on the page" — asserting a search that +never happened. And both disclosure helpers gated on `evidence_image`, so a +judgement with a page but no crop disclosed nothing at all: the reader saw a +verdict with a page number and no statement about whether anything backed it. + +The gate is *provenance*, not the picture. A claim with no page says nothing, +because there is nothing to say. +""" + +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from papertrace.disclosures import ( # noqa: E402 + ANCHOR_LOCATED_TOKEN, + ANCHOR_NOT_LOCATED_TOKEN, + ANCHOR_UNKNOWN_TOKEN, + anchor_state, + claim_disclosures, + judgement_disclosures, +) +from papertrace.models import ClaimResult, SourceJudgement # noqa: E402 + + +def _j(**kw) -> SourceJudgement: + base = dict(source_slug="a-2020", ref="1", verdict="supported", source_page=3, + source_block="block_0007", anchor_phrases=["84.3%"]) + base.update(kw) + return SourceJudgement(**base) + + +def test_anchor_state_keeps_three_names(): + assert anchor_state(_j(anchor_located=True)) == "located" + assert anchor_state(_j(anchor_located=False)) == "not_located" + assert anchor_state(_j(anchor_located=None)) == "unknown" + + +def test_a_judgement_with_no_crop_still_discloses_its_anchor_state(): + """The old gate. No `evidence_image`, so the reader was told nothing — + while the report still printed "Page 3" as if it were provenance.""" + ds = judgement_disclosures(_j(anchor_located=False, evidence_image=None)) + assert [d.token for d in ds] == [ANCHOR_NOT_LOCATED_TOKEN] + + +def test_an_unknown_anchor_with_no_crop_is_disclosed_too(): + ds = judgement_disclosures(_j(anchor_phrases=[], anchor_located=None, evidence_image=None)) + assert [d.token for d in ds] == [ANCHOR_UNKNOWN_TOKEN] + + +def test_a_judgement_with_no_page_discloses_nothing(): + """Silence about nothing is not a dropped disclosure. An unretrieved source + has no page, so there is no anchor claim to qualify.""" + assert judgement_disclosures(_j(source_page=None, verdict="unchecked")) == [] + + +def test_the_no_crop_wording_never_mentions_a_crop(): + """Same token — the parity contract holds — but the sentence must not + describe a picture that was not written.""" + with_crop = judgement_disclosures(_j(anchor_located=False, evidence_image="e/x.png"))[0] + without = judgement_disclosures(_j(anchor_located=False, evidence_image=None))[0] + + assert with_crop.token == without.token == ANCHOR_NOT_LOCATED_TOKEN + assert "crop" in with_crop.text + assert "crop" not in without.text + assert "no evidence image" in without.text + + +def test_located_needs_no_no_crop_variant_but_still_only_fires_with_provenance(): + d = judgement_disclosures(_j(anchor_located=True, evidence_image="e/x.png"))[0] + assert d.token == ANCHOR_LOCATED_TOKEN + + +def test_claim_level_disclosure_follows_the_same_rule(): + claim = ClaimResult(id=1, claim="c", location="Results", refs=["1"], + verdict="supported", source_slug="a-2020", source_page=3, + anchor_phrases=["84.3%"], anchor_located=False) + keys = [d.key for d in claim_disclosures(claim)] + assert "anchor" in keys + + ungrounded = ClaimResult(id=2, claim="c", location="Results", refs=["1"], + verdict="not_retrieved") + assert "anchor" not in [d.key for d in claim_disclosures(ungrounded)] diff --git a/tests/test_ask_boot_hygiene.py b/tests/test_ask_boot_hygiene.py new file mode 100644 index 0000000..1622ac6 --- /dev/null +++ b/tests/test_ask_boot_hygiene.py @@ -0,0 +1,78 @@ +"""The only seam that calls a model must not call it from wherever the user happened +to be standing. + +`_ask` inherited the caller's `cwd` and the CLI's full default toolset. Judging +from inside a repo silently fed that repo's own CLAUDE.md into every verdict, and +the judge held Bash/Edit/WebFetch while it is only ever supposed to read +third-party PDF text and return one. Neither was disclosed anywhere in the +report — a silent scope creep this project otherwise refuses to have. +""" + +import json +import sys +import tempfile +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from papertrace import check as check_mod # noqa: E402 + + +class _FakeCompleted: + def __init__(self): + self.returncode = 0 + self.stdout = json.dumps({"result": "ok", "model": "claude-sonnet-5"}) + self.stderr = "" + + +def test_ask_does_not_run_from_the_caller_s_working_directory(monkeypatch): + captured = {} + + def fake_run(cmd, **kwargs): + captured["cmd"] = cmd + captured["kwargs"] = kwargs + return _FakeCompleted() + + monkeypatch.setattr(check_mod.subprocess, "run", fake_run) + + check_mod._ask("judge this") + + assert captured["kwargs"].get("cwd") is not None + assert captured["kwargs"]["cwd"] != "." + + +def test_ask_disables_customizations_and_tool_access(monkeypatch): + captured = {} + + def fake_run(cmd, **kwargs): + captured["cmd"] = cmd + return _FakeCompleted() + + monkeypatch.setattr(check_mod.subprocess, "run", fake_run) + + check_mod._ask("judge this") + + cmd = captured["cmd"] + assert "--safe-mode" in cmd + assert "--tools" in cmd + assert cmd[cmd.index("--tools") + 1] == "" + + +def test_ask_does_not_run_from_the_shared_temp_root(monkeypatch): + """The system temp root (`/tmp`) is world-writable and predictable — a + private, per-process directory is required so another local user cannot + plant config the judging call would then walk into.""" + captured = [] + + def fake_run(cmd, **kwargs): + captured.append(kwargs.get("cwd")) + return _FakeCompleted() + + monkeypatch.setattr(check_mod.subprocess, "run", fake_run) + + check_mod._ask("judge this") + check_mod._ask("judge this again") + + assert all(cwd != tempfile.gettempdir() for cwd in captured) + assert all(Path(cwd).is_dir() for cwd in captured) + assert len(set(captured)) == 1, "the scratch directory should be reused, not recreated per call" diff --git a/tests/test_case_identity.py b/tests/test_case_identity.py index 70c43cd..919615b 100644 --- a/tests/test_case_identity.py +++ b/tests/test_case_identity.py @@ -51,7 +51,7 @@ def offline(monkeypatch): import papertrace.refs as refs_mod monkeypatch.setattr(refs_mod, "resolve_all", - lambda entries, dest, email, provided_dir=None, progress=None: entries) + lambda entries, dest, email, provided_dir=None, progress=None, taken=None: entries) monkeypatch.setenv("PAPERTRACE_EMAIL", "test@example.org") @@ -66,7 +66,7 @@ def test_a_legacy_case_reingests_rather_than_certifying_the_old_paper(tmp_path, new_pdf = _paper(tmp_path / "new" / "paper.pdf", "NEW", "10.1000/new") # build the legacy case: ingest OLD, then strip the hash from its manifest - cli.refs(manuscript=old_pdf, case=case, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=old_pdf, case=case, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") payload = json.loads((case / "refs_manifest.json").read_text()) del payload["manuscript_sha256"] @@ -74,7 +74,7 @@ def test_a_legacy_case_reingests_rather_than_certifying_the_old_paper(tmp_path, assert RefManifest.from_json(case / "refs_manifest.json").manuscript_sha256 is None # now the same filename, different content - cli.refs(manuscript=new_pdf, case=case, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=new_pdf, case=case, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") written = RefManifest.from_json(case / "refs_manifest.json") @@ -92,7 +92,7 @@ def test_an_ordinary_case_still_reuses_its_cached_ingest(tmp_path, offline, monk case = tmp_path / "case" pdf = _paper(tmp_path / "a" / "paper.pdf", "SAME", "10.1000/same") - cli.refs(manuscript=pdf, case=case, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=pdf, case=case, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") assert RefManifest.from_json(case / "refs_manifest.json").manuscript_sha256 is not None @@ -103,7 +103,7 @@ def test_an_ordinary_case_still_reuses_its_cached_ingest(tmp_path, offline, monk monkeypatch.setattr(ing, "ingest_pdf", lambda p, o, **kw: calls.append(p) or real(p, o, **kw)) - cli.refs(manuscript=pdf, case=case, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=pdf, case=case, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") assert calls == [], "a verified case re-ingested when it did not need to" @@ -114,10 +114,10 @@ def test_a_different_paper_in_a_hashed_case_is_still_refused(tmp_path, offline): one = _paper(tmp_path / "one" / "alpha.pdf", "ALPHA", "10.1000/alpha") two = _paper(tmp_path / "two" / "beta.pdf", "BETA", "10.1000/beta") - cli.refs(manuscript=one, case=case, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=one, case=case, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") with pytest.raises(typer.Exit) as e: - cli.refs(manuscript=two, case=case, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=two, case=case, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") assert e.value.exit_code == 2 @@ -156,7 +156,7 @@ def test_the_case_folder_defaults_to_the_papers_own_name(tmp_path, offline, monk monkeypatch.chdir(elsewhere) # the audit must not follow the user's cwd pdf = _paper(tmp_path / "papers" / "PIIS0720048X2600522X.pdf", "ONE", "10.1000/one") - cli.refs(manuscript=pdf, case=None, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=pdf, case=None, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") derived = tmp_path / "papers" / "PIIS0720048X2600522X" @@ -174,7 +174,7 @@ def test_an_explicit_case_flag_still_wins(tmp_path, offline, monkeypatch, termin chosen = tmp_path / "mycase" for _ in range(2): # twice: an explicit re-run is not interrogated either - cli.refs(manuscript=pdf, case=chosen, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=pdf, case=chosen, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") assert (chosen / "refs_manifest.json").exists() @@ -187,10 +187,10 @@ def test_a_rerun_of_the_same_paper_can_amend_its_case(tmp_path, offline, monkeyp pdf = _paper(tmp_path / "papers" / "beta.pdf", "BETA", "10.1000/beta") derived = tmp_path / "papers" / "beta" - cli.refs(manuscript=pdf, case=None, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=pdf, case=None, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") terminal.append("amend") - cli.refs(manuscript=pdf, case=None, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=pdf, case=None, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") assert terminal == [], "the collision was not put to the user" @@ -203,11 +203,11 @@ def test_a_rerun_can_start_a_fresh_numbered_case(tmp_path, offline, monkeypatch, monkeypatch.chdir(tmp_path) pdf = _paper(tmp_path / "papers" / "gamma.pdf", "GAMMA", "10.1000/gamma") - cli.refs(manuscript=pdf, case=None, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=pdf, case=None, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") first = (tmp_path / "papers" / "gamma" / "refs_manifest.json").read_bytes() terminal.append("fresh") - cli.refs(manuscript=pdf, case=None, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=pdf, case=None, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") assert (tmp_path / "papers" / "gamma-2" / "refs_manifest.json").exists() @@ -224,10 +224,10 @@ def test_a_rerun_without_a_terminal_amends_and_says_so(tmp_path, offline, monkey monkeypatch.chdir(tmp_path) pdf = _paper(tmp_path / "papers" / "delta.pdf", "DELTA", "10.1000/delta") - cli.refs(manuscript=pdf, case=None, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=pdf, case=None, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") capsys.readouterr() - cli.refs(manuscript=pdf, case=None, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=pdf, case=None, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") out = capsys.readouterr().out @@ -245,12 +245,12 @@ def test_a_replaced_paper_of_the_same_name_is_still_refused(tmp_path, offline, m monkeypatch.chdir(tmp_path) path = tmp_path / "papers" / "epsilon.pdf" _paper(path, "FIRST", "10.1000/first") - cli.refs(manuscript=path, case=None, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=path, case=None, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") _paper(path, "SECOND", "10.1000/second") # same name, different paper with pytest.raises(typer.Exit) as e: - cli.refs(manuscript=path, case=None, provided=None, email="test@example.org", + cli._refs_pipeline(manuscript=path, case=None, provided=None, email="test@example.org", parse_only=False, backend="pymupdf") assert e.value.exit_code == 2 @@ -262,7 +262,14 @@ def test_run_derives_one_case_folder_and_hands_it_to_every_stage(tmp_path, offli monkeypatch.chdir(tmp_path) pdf = _paper(tmp_path / "papers" / "zeta.pdf", "ZETA", "10.1000/zeta") seen: dict[str, dict] = {} - for name in ("ingest", "refs", "scout", "check", "highlight", "report"): + # `ingest` and `refs` are split into a Typer command plus a `_..._pipeline` + # function, the plain function `run` actually calls — see cli.py's comment + # on `run()`. + monkeypatch.setattr(cli, "_ingest_pipeline", lambda **kw: seen.__setitem__("ingest", kw)) + monkeypatch.setattr(cli, "_refs_pipeline", lambda **kw: seen.__setitem__("refs", kw)) + monkeypatch.setattr(cli, "_report_pipeline", lambda **kw: seen.__setitem__("report", kw)) + monkeypatch.setattr(cli, "_check_pipeline", lambda **kw: seen.__setitem__("check", kw)) + for name in ("scout", "highlight"): monkeypatch.setattr(cli, name, (lambda n: lambda **kw: seen.__setitem__(n, kw))(name)) cli.run(manuscript=pdf, case=None, provided=None, email="test@example.org", model=None, @@ -281,3 +288,95 @@ def test_the_wizard_suggests_the_folder_batch_mode_would_use(tmp_path): pdf.parent.mkdir(parents=True) pdf.write_bytes(b"%PDF-1.4\n") assert wizard._suggest_case(pdf) == str(cli.default_case(pdf)) + + +# --- the two ways around the guard ----------------------------------------- +# +# `_guard_case` only ever ran inside `refs`. Two other paths could write into a +# case's manuscript slot: `ingest` never consulted the guard at all, and +# `refs --parse-only` re-ingested a legacy case and then returned before the +# manifest caught up. Both leave one case folder describing two papers, which +# is precisely the state the guard exists to make impossible. + + +def test_ingest_refuses_to_overwrite_another_papers_manuscript_slot(tmp_path, offline): + """`papertrace ingest manuscript.pdf -c CASE` writes <case>/ingest/manuscript + — the same slot `refs` filled and `coverage_audit` reads. A different paper + landing there leaves the source map describing NEW and the manifest OLD. + """ + case = tmp_path / "case" + old_pdf = _paper(tmp_path / "old" / "manuscript.pdf", "OLD", "10.1000/old") + new_pdf = _paper(tmp_path / "new" / "manuscript.pdf", "NEW", "10.1000/new") + + cli._refs_pipeline(manuscript=old_pdf, case=case, provided=None, email="test@example.org", + parse_only=False, backend="pymupdf") + + with pytest.raises(typer.Exit): + cli._ingest_pipeline(pdf=new_pdf, out=None, case=case, backend="pymupdf") + + smap = json.loads((case / "ingest" / "manuscript" / "source_map.json").read_text()) + body = " ".join(b.get("text", "") for b in smap["blocks"]) + assert "NEW" not in body, "a different paper overwrote the case's manuscript" + assert "OLD" in body + + +def test_ingest_of_a_cited_source_into_the_same_case_is_untouched(tmp_path, offline): + """The guard is about the manuscript slot, not the folder. A cited source + ingested into `<case>/ingest/<slug>` is not the audited paper and must stay + ingestable — guarding it would break `check`'s own source ingest.""" + case = tmp_path / "case" + paper = _paper(tmp_path / "a" / "paper.pdf", "PAPER", "10.1000/paper") + source = _paper(tmp_path / "b" / "smith-2020.pdf", "SOURCE", "10.1000/src") + + cli._refs_pipeline(manuscript=paper, case=case, provided=None, email="test@example.org", + parse_only=False, backend="pymupdf") + cli._ingest_pipeline(pdf=source, out=None, case=case, backend="pymupdf") + + assert (case / "ingest" / "smith-2020" / "source_map.json").exists() + + +def test_parse_only_on_a_legacy_case_leaves_the_case_coherent(tmp_path, offline): + """`--parse-only` says "List references, no network" — an inspection. On a + legacy case it re-ingested into the manuscript slot and then returned before + writing the manifest, so the source map described NEW while the manifest and + its (absent) hash still described OLD. + """ + case = tmp_path / "case" + old_pdf = _paper(tmp_path / "old" / "paper.pdf", "OLD", "10.1000/old") + new_pdf = _paper(tmp_path / "new" / "paper.pdf", "NEW", "10.1000/new") + + cli._refs_pipeline(manuscript=old_pdf, case=case, provided=None, email="test@example.org", + parse_only=False, backend="pymupdf") + payload = json.loads((case / "refs_manifest.json").read_text()) + del payload["manuscript_sha256"] + (case / "refs_manifest.json").write_text(json.dumps(payload)) + + cli._refs_pipeline(manuscript=new_pdf, case=case, provided=None, email="test@example.org", + parse_only=True, backend="pymupdf") + + smap = json.loads((case / "ingest" / "manuscript" / "source_map.json").read_text()) + body = " ".join(b.get("text", "") for b in smap["blocks"]) + manifest = RefManifest.from_json(case / "refs_manifest.json") + raws = " ".join(e.raw for e in manifest.entries) + assert ("NEW" in body) == ("NEW PAPER" in raws), ( + "the source map and the manifest describe different papers" + ) + + +def test_parse_only_still_lists_the_new_papers_references(tmp_path, offline, capsys): + """Not mutating the case must not mean reading the wrong paper: the listing + is of the file that was passed, whatever the case folder holds.""" + case = tmp_path / "case" + old_pdf = _paper(tmp_path / "old" / "paper.pdf", "OLD", "10.1000/old") + new_pdf = _paper(tmp_path / "new" / "paper.pdf", "NEW", "10.1000/new") + + cli._refs_pipeline(manuscript=old_pdf, case=case, provided=None, email="test@example.org", + parse_only=False, backend="pymupdf") + payload = json.loads((case / "refs_manifest.json").read_text()) + del payload["manuscript_sha256"] + (case / "refs_manifest.json").write_text(json.dumps(payload)) + capsys.readouterr() + + cli._refs_pipeline(manuscript=new_pdf, case=case, provided=None, email="test@example.org", + parse_only=True, backend="pymupdf") + assert "NEW PAPER" in capsys.readouterr().out diff --git a/tests/test_check_provenance.py b/tests/test_check_provenance.py new file mode 100644 index 0000000..76cef39 --- /dev/null +++ b/tests/test_check_provenance.py @@ -0,0 +1,450 @@ +"""A substantive verdict must point at a passage the reader can be shown. + +`supported`, `partial` and `contradicted` all assert that a specific piece of +the source settles the claim. Before this, the assertion was unchecked: the +model could name page 99999 and `block_nope`, and the verdict stood. The +report then printed "Page 99999" as provenance and `crop_for_anchor` quietly +produced nothing, so the one claim the reader most wanted to verify was the +one with no picture. + +The bar is the picture. `crop_for_anchor` writes a crop when the region comes +from a valid block, or from an anchor phrase that actually matched the page — +so requiring a **valid block** is what makes the image unconditional. With one, +`crop_evidence` always writes the block; the red box is drawn on top if a +phrase matches inside it. Anchor phrases stay optional because they decide +whether there is a box, not whether there is a picture. + +`not_addressed` is exempt and must stay exempt: the source was read and says +nothing, so there is no passage to point at, and demanding one would force the +model to invent a citation for an absence. +""" + +import json +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from papertrace import check as check_mod # noqa: E402 +from papertrace.check import SourceProvenance, _judgement_from, check_claims # noqa: E402 +from papertrace.models import ClaimResult, RefEntry, RefManifest, SourceMap # noqa: E402 + +# one page, one block on it — the smallest source a verdict can point into +ONE_PAGE = SourceProvenance(pages=1, block_pages={"block_0001": 1}) +TWO_PAGE = SourceProvenance(pages=2, block_pages={"block_0001": 1, "block_0007": 2}) + + +def _entry(**kw) -> dict: + base = { + "id": 1, + "verdict": "supported", + "note": "the source states it", + "source_page": 1, + "source_block": "block_0001", + "anchor_phrases": ["84.3%"], + } + base.update(kw) + return base + + +# --- what a valid judgement still looks like ------------------------------- + + +def test_a_valid_block_on_the_named_page_is_accepted(): + fields, why = _judgement_from(_entry(), ONE_PAGE) + assert why == "" + assert fields["verdict"] == "supported" + assert fields["source_block"] == "block_0001" + + +def test_a_valid_block_with_no_anchor_phrases_is_still_accepted(): + """The block is what guarantees the crop; the phrases only add the red box. + + Rejecting this would discard a reading the model got right and merely + under-quoted — and the reader still gets an image of the exact block, shown + unboxed and captioned as such. + """ + fields, why = _judgement_from(_entry(anchor_phrases=[]), ONE_PAGE) + assert why == "" + assert fields["anchor_phrases"] == [] + + +def test_not_addressed_still_needs_no_page_or_block(): + """Absence of relevant content has no decisive passage by construction.""" + fields, why = _judgement_from( + {"id": 1, "verdict": "not_addressed", "note": "silent on mortality"}, ONE_PAGE + ) + assert why == "" + assert fields["source_page"] is None and fields["source_block"] is None + + +def test_not_addressed_survives_an_unreadable_source_map(): + """It asserts nothing about a location, so there is nothing to validate.""" + fields, why = _judgement_from({"id": 1, "verdict": "not_addressed", "note": "n"}, None) + assert why == "" and fields["verdict"] == "not_addressed" + + +# --- provenance that cannot be true ---------------------------------------- + + +@pytest.mark.parametrize( + ("entry", "expect_in_note"), + [ + (_entry(source_page=99999, source_block=None), "1 page"), + (_entry(source_page=2, source_block=None), "1 page"), + (_entry(source_block="block_9999"), "block_9999"), + (_entry(source_block=None), "no source_block"), + ], +) +def test_impossible_provenance_is_unchecked_not_a_verdict(entry, expect_in_note): + fields, why = _judgement_from(entry, ONE_PAGE) + assert fields is None + assert expect_in_note in why + + +def test_a_block_on_another_page_is_refused(): + """`block_0007` is real, but it is on page 2 — so page 1 is not where it is, + and a crop of page 1 would show the reader something else entirely.""" + fields, why = _judgement_from( + _entry(source_page=1, source_block="block_0007"), TWO_PAGE + ) + assert fields is None + assert "block_0007" in why and "page 2" in why + + +def test_an_unreadable_source_map_refuses_substantive_verdicts(): + """Unverifiable provenance is not verified provenance. The note names the + fix, because the cause is our own artifact, not the model.""" + fields, why = _judgement_from(_entry(), None) + assert fields is None + assert "source map" in why and "papertrace ingest" in why + + +# --- through check_claims, where it actually matters ------------------------ + + +def _write_source(case: Path, slug: str, pages: int = 1, + bbox: tuple = (0.0, 0.0, 10.0, 10.0)) -> None: + """An ingested source: the text and the map that says where its blocks are. + + `bbox` matters only where a crop is actually drawn — the highlight tests + need a region that covers the text on the generated page. + """ + from papertrace.models import Block + + d = case / "ingest" / slug + d.mkdir(parents=True, exist_ok=True) + (d / "annotated.md").write_text("<!-- block_0001, page 1 -->\nThe rate was 84.3%.\n") + SourceMap( + doc=f"{slug}.pdf", + pages=pages, + blocks=[Block("block_0001", "text", 1, bbox, [], "The rate was 84.3%.")], + ).to_json(d / "source_map.json") + + +def _manifest(slug: str) -> RefManifest: + return RefManifest( + manuscript="paper.pdf", + entries=[RefEntry(num="1", raw="ref", status="retrieved", slug=slug, + pdf_path=f"/nonexistent/{slug}.pdf")], + ) + + +def test_check_claims_downgrades_an_impossible_page_to_unchecked(tmp_path, monkeypatch): + _write_source(tmp_path, "smith-2020") + monkeypatch.setattr( + check_mod, "_ask", + lambda prompt, model=None: json.dumps( + [{"id": 1, "verdict": "supported", "note": "yes", + "source_page": 99999, "source_block": "nope", "anchor_phrases": []}] + ), + ) + claims = [ClaimResult(id=1, claim="the rate was 84.3%", location="Results", refs=["1"])] + check_claims(claims, _manifest("smith-2020"), tmp_path, backend="pymupdf") + + j = claims[0].judgements[0] + assert j.verdict == "unchecked" + assert claims[0].verdict == "unchecked" + # the source WAS retrieved — this must never be laundered into a gap + assert claims[0].verdict != "not_retrieved" + + +def test_check_claims_keeps_a_verdict_whose_block_is_real(tmp_path, monkeypatch): + _write_source(tmp_path, "smith-2020") + monkeypatch.setattr( + check_mod, "_ask", + lambda prompt, model=None: json.dumps( + [{"id": 1, "verdict": "contradicted", "note": "says 48%", + "source_page": 1, "source_block": "block_0001", + "anchor_phrases": ["84.3%"]}] + ), + ) + claims = [ClaimResult(id=1, claim="the rate was 84.3%", location="Results", refs=["1"])] + check_claims(claims, _manifest("smith-2020"), tmp_path, backend="pymupdf") + assert claims[0].judgements[0].verdict == "contradicted" + assert claims[0].verdict == "contradicted" + + +# --- the backstop: a verdict with no picture is not a verdict --------------- +# +# Check-time validation makes this nearly unreachable, and "nearly" is not the +# standard. The PDF can be missing from sources_resolved/, and a source map can +# disagree with the PDF it was built from. Without a backstop at the point the +# image is actually produced, the rule holds by inference rather than by +# construction — so `highlight` enforces it again, against reality this time. + + +def _case_with_judgement(tmp_path: Path, *, verdict: str = "supported") -> Path: + from papertrace.models import RunResults, SourceJudgement + + case = tmp_path / "case" + (case / "out").mkdir(parents=True) + _write_source(case, "a-2020", bbox=(72.0, 90.0, 300.0, 110.0)) + + claim = ClaimResult( + id=1, claim="the rate was 84.3%", location="Results", refs=["1"], + judgements=[SourceJudgement( + source_slug="a-2020", ref="1", verdict=verdict, note="the source states it", + source_page=1, source_block="block_0001", anchor_phrases=["84.3%"], + )], + ) + claim.apply_headline() + RunResults(manuscript="m.pdf", claims=[claim]).to_json(case / "out" / "results.json") + _manifest("a-2020").to_json(case / "refs_manifest.json") + return case + + +def test_a_verdict_that_produced_no_evidence_image_is_downgraded(tmp_path): + """The block validated at check time, but the PDF is not in the case folder, + so no crop exists. A `supported` a reader cannot look at is not `supported`.""" + from papertrace import cli + from papertrace.models import RunResults + + case = _case_with_judgement(tmp_path) + cli.highlight(case=case, claim=None) + + after = RunResults.from_json(case / "out" / "results.json") + j = after.claims[0].judgements[0] + assert j.verdict == "unchecked" + assert "evidence image" in j.note + assert after.claims[0].verdict == "unchecked" + + +def test_not_addressed_is_not_downgraded_for_having_no_image(tmp_path): + """It never claimed a passage, so there is no picture it owes anyone.""" + from papertrace import cli + from papertrace.models import RunResults + + case = _case_with_judgement(tmp_path, verdict="not_addressed") + cli.highlight(case=case, claim=None) + + after = RunResults.from_json(case / "out" / "results.json") + assert after.claims[0].judgements[0].verdict == "not_addressed" + + +def test_a_verdict_whose_crop_was_written_survives(tmp_path): + """The positive control: a real PDF, a real block, a crop on disk.""" + import pymupdf + + from papertrace import cli + from papertrace.models import RunResults + + case = _case_with_judgement(tmp_path) + pdf = case / "sources_resolved" / "a-2020.pdf" + pdf.parent.mkdir(parents=True, exist_ok=True) + doc = pymupdf.open() + doc.new_page().insert_text((72, 100), "The rate was 84.3% overall.", fontsize=11) + doc.save(pdf) + doc.close() + + cli.highlight(case=case, claim=None) + + after = RunResults.from_json(case / "out" / "results.json") + j = after.claims[0].judgements[0] + assert j.verdict == "supported" + assert j.evidence_image and (case / "out" / j.evidence_image).exists() + assert j.anchor_located is True + + +# --- a source directory must hold the source it is named after ---------------- +# +# `check_claims` reused `case/ingest/<slug>/annotated.md` whenever it existed, +# keyed on the slug alone. `SourceMap.doc` is `<slug>.pdf` whichever paper's +# bytes are inside, so nothing could tell one from another. Slugs are not +# eternal — a collision fix renames one of two colliding entries, and the +# reconciler can hand `refs` the Crossref list on one run and the parsed list on +# the next — so a re-run of an existing case could judge a claim against the +# previous occupant of that directory: a confident verdict on the wrong paper, +# which is the one thing this codebase refuses to do. + + +def _pdf_saying(path: Path, text: str) -> Path: + import pymupdf + + doc = pymupdf.open() + page = doc.new_page() + page.insert_text((72, 100), text, fontsize=12) + path.parent.mkdir(parents=True, exist_ok=True) + doc.save(path) + doc.close() + return path + + +def test_the_ingest_of_a_source_records_which_file_it_read(tmp_path): + """Additive field, round-tripped, and absent-safe for older maps.""" + import json as _json + + import jsonschema + + from papertrace.ingest import ingest_pdf + from papertrace.models import SourceMap, manuscript_fingerprint + + pdf = _pdf_saying(tmp_path / "a.pdf", "Paper A says alpha.") + smap = ingest_pdf(pdf, tmp_path / "out", backend="pymupdf") + assert smap.source_sha256 == manuscript_fingerprint(pdf) + + path = tmp_path / "out" / "source_map.json" + schema_path = Path(__file__).resolve().parent.parent / "schemas" / "source_map.schema.json" + schema = _json.loads(schema_path.read_text()) + payload = _json.loads(path.read_text()) + jsonschema.validate(payload, schema) + assert SourceMap.from_json(path).source_sha256 == smap.source_sha256 + + del payload["source_sha256"] # a map written before this field + path.write_text(_json.dumps(payload)) + jsonschema.validate(_json.loads(path.read_text()), schema) + assert SourceMap.from_json(path).source_sha256 is None + + +def test_a_stale_source_directory_is_re_ingested_not_reused(tmp_path, monkeypatch): + """The wrong-paper route, stated as the thing that must not happen: the + directory holds paper B, the manifest entry points at paper A, and the + model must be shown A.""" + from papertrace.check import check_claims + from papertrace.ingest import ingest_pdf + from papertrace.models import ClaimResult, RefEntry, RefManifest + + case = tmp_path / "case" + sources = case / "sources_resolved" + a = _pdf_saying(sources / "smith-2020.pdf", "Paper A reports alpha in twelve patients.") + b = _pdf_saying(tmp_path / "b.pdf", "Paper B reports beta in nine patients.") + + # the stale state: ingest/smith-2020 built from B, while the manifest's + # smith-2020 now points at A + ingest_pdf(b, case / "ingest" / "smith-2020", backend="pymupdf") + assert "beta" in (case / "ingest" / "smith-2020" / "annotated.md").read_text() + + manifest = RefManifest( + manuscript="m.pdf", + entries=[RefEntry(num="5", raw="Smith J. Paper A. 2020.", status="retrieved", + slug="smith-2020", pdf_path=str(a))], + ) + (case / "refs_manifest.json").parent.mkdir(parents=True, exist_ok=True) + manifest.to_json(case / "refs_manifest.json") + + claims = [ClaimResult(id=1, claim="alpha was reported", location="Results", refs=["5"])] + prompts: list[str] = [] + + def _fake_ask(prompt, model=None): + prompts.append(prompt) + return '[{"id":1,"verdict":"not_addressed","note":"n/a"}]' + + monkeypatch.setattr("papertrace.check._ask", _fake_ask) + check_claims(claims, manifest, case, backend="pymupdf") + + assert prompts, "no model call was made" + body = prompts[0] + assert "alpha" in body, "the model was not shown the paper the manifest names" + assert "beta" not in body, "the model was shown the previous occupant of the directory" + + +def test_a_source_read_by_the_other_backend_is_re_ingested(tmp_path): + """A case folder built before layout-aware source ingest holds pymupdf + source maps. Re-running `check` on it must not reuse them while the run + claims docling: the whole point of reading sources with the layout backend + is that a table in a source is readable, and a stale flat map silently + gives the judge the linearized version instead. + + `source_sha256` cannot catch this — it is the same PDF. The converter is + the fact that changed. + """ + from papertrace.check import _stale_ingest + from papertrace.models import manuscript_fingerprint + + pdf = tmp_path / "smith-2020.pdf" + _pdf_saying(pdf, "HR 0.88 (0.79-0.98).") + ingest_dir = tmp_path / "ingest" / "smith-2020" + ingest_dir.mkdir(parents=True) + smap = { + "doc": "smith-2020.pdf", "pages": 1, "converter": "pymupdf", + "source_sha256": manuscript_fingerprint(pdf), + "blocks": [{"id": "block_0001", "page": 1, "type": "text", + "bbox": [0, 0, 10, 10], "text": "HR 0.88 (0.79-0.98)."}], + } + (ingest_dir / "source_map.json").write_text(json.dumps(smap)) + + # same backend that wrote it: nothing to redo + assert _stale_ingest(ingest_dir, str(pdf), backend="pymupdf") is False + # a different backend: the map does not answer the question being asked now + assert _stale_ingest(ingest_dir, str(pdf), backend="docling") is True + + +def test_auto_is_resolved_before_the_converter_is_compared(tmp_path): + """`--backend auto` is not a converter name, so comparing it literally + would report every existing map as stale and re-ingest the whole reference + list on every run.""" + from papertrace.check import _stale_ingest + from papertrace.ingest import _docling_available + from papertrace.models import manuscript_fingerprint + + pdf = tmp_path / "smith-2020.pdf" + _pdf_saying(pdf, "text") + ingest_dir = tmp_path / "ingest" / "smith-2020" + ingest_dir.mkdir(parents=True) + resolved = "docling" if _docling_available() else "pymupdf" + (ingest_dir / "source_map.json").write_text(json.dumps({ + "doc": "smith-2020.pdf", "pages": 1, + # a docling map records its version too — "docling 2.x", not "docling" + "converter": f"{resolved} 9.9.9" if resolved == "docling" else resolved, + "source_sha256": manuscript_fingerprint(pdf), + "blocks": [], + })) + + assert _stale_ingest(ingest_dir, str(pdf), backend="auto") is False + + +def test_a_source_ingest_that_fails_unchecks_only_that_source(tmp_path, monkeypatch): + """Reading sources with the layout backend introduces a failure mode flat + text never had: docling can run out of memory, fail to fetch its models, or + choke on a malformed PDF. That must uncheck the one source and name the + reason — not abort a run that has already paid for retrieval, and never be + laundered into `not_retrieved`, which would blame the publisher for a local + failure. + """ + pdf = tmp_path / "sources_resolved" / "smith-2020.pdf" + _pdf_saying(pdf, "HR 0.88 (0.79-0.98).") + manifest = RefManifest(manuscript="m.pdf", entries=[ + RefEntry(num="1", raw="Smith J (2020)", status="retrieved", slug="smith-2020", + pdf_path=str(pdf))]) + claim = ClaimResult(id=1, claim="Mortality fell.", location="Results", refs=["1"]) + + import papertrace.ingest as ingest_mod + + def boom(*a, **kw): + raise RuntimeError("docling layout model could not be loaded") + + monkeypatch.setattr(ingest_mod, "ingest_pdf", boom) + called = [] + monkeypatch.setattr(check_mod, "_ask", lambda *a, **k: called.append(1) or "[]") + + errors: list[tuple[str, str]] = [] + check_claims([claim], manifest, tmp_path, on_error=lambda s, m: errors.append((s, m)), + backend="docling") + + assert claim.verdict == "unchecked", "a local ingest failure is not a retrieval gap" + assert "docling layout model" in claim.note + assert "re-run" in claim.note, "the note must name the fix" + assert called == [], "no model call once there is no source text to send" + assert errors and errors[0][0] == "smith-2020" diff --git a/tests/test_claim_quote.py b/tests/test_claim_quote.py new file mode 100644 index 0000000..daddb6d --- /dev/null +++ b/tests/test_claim_quote.py @@ -0,0 +1,305 @@ +"""The judge used to read a ≤160-character paraphrase of the sentence it was +checking, never the sentence. + +`EXTRACT_PROMPT` asked for the statement "tightly paraphrased, ≤160 chars", and +`CHECK_PROMPT` was handed `{id, claim, location}` — so population, conditions, +effect direction, uncertainty and every other qualification that decides whether +a citation supports a statement had to survive a compression the judge could not +undo. "Mortality fell by 12% in the subgroup over 65 (HR 0.88, 95% CI +0.79-0.98)" and "mortality fell by 12%" are different claims, and only one of +them is checkable. + +The claim now carries the manuscript's own sentence alongside the paraphrase. +The paraphrase stays because it is what a report headline reads well; the quote +is what gets judged. +""" + +import json +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from papertrace import check as check_mod # noqa: E402 +from papertrace.models import ClaimResult, RunResults # noqa: E402 +from papertrace.report import write_reports # noqa: E402 + +SCHEMA = json.loads((Path(__file__).parent.parent / "schemas" / "results.schema.json").read_text()) + + +# --- the prompts ask for it and pass it on --------------------------------- + + +def test_the_extraction_prompt_asks_for_a_verbatim_quote(): + p = check_mod.EXTRACT_PROMPT + assert "quote" in p + assert "verbatim" in p.lower(), "a quote that is not required to be verbatim is a paraphrase" + + +def test_the_paraphrase_cap_is_no_longer_160_chars(): + """160 characters cannot hold a population, an interval and a direction at + once. The cap stays — an unbounded 'paraphrase' is just the quote again — + but at a width that fits a real clinical sentence.""" + assert "≤160 chars" not in check_mod.EXTRACT_PROMPT + + +def test_the_judge_is_shown_the_quote_not_only_the_paraphrase(tmp_path, monkeypatch): + """The whole point. A quote stored but never sent leaves the judgement + exactly as compressed as it was.""" + seen = {} + + def fake_ask(prompt, model=None): + seen["prompt"] = prompt + return json.dumps([{"id": 1, "verdict": "supported", "note": "Yes.", + "source_page": 1, "source_block": "block_0001", + "anchor_phrases": ["0.88"]}]) + + _stub_source(tmp_path, monkeypatch, fake_ask) + claim = ClaimResult(id=1, claim="Mortality fell 12%.", location="Results", + refs=["1"], quote="Mortality fell by 12% in the subgroup " + "over 65 (HR 0.88, 95% CI 0.79-0.98).") + check_mod.check_claims([claim], _manifest(), tmp_path, backend="pymupdf") + + assert "95% CI 0.79-0.98" in seen["prompt"], "the judge never saw the real sentence" + + +# --- parsing degrades honestly --------------------------------------------- + + +def test_a_response_with_no_quote_is_not_invented(tmp_path, monkeypatch): + """An older or sloppier model answer omits the field. The claim keeps an + empty quote — it must never be back-filled from the paraphrase, which would + silently reinstate exactly the compression this change removes.""" + monkeypatch.setattr(check_mod, "_ask", lambda p, m=None: json.dumps( + {"cited": [{"id": 1, "claim": "X causes Y.", "location": "Intro", "refs": ["1"]}], + "uncited": []})) + _annotated(tmp_path) + cited, _ = check_mod.extract_claims(tmp_path) + assert cited[0].quote == "" + assert cited[0].claim == "X causes Y." + + +def test_a_quote_that_is_present_is_kept_verbatim(tmp_path, monkeypatch): + quote = "Mortality fell by 12% in the subgroup over 65 (HR 0.88, 95% CI 0.79-0.98)." + monkeypatch.setattr(check_mod, "_ask", lambda p, m=None: json.dumps( + {"cited": [{"id": 1, "claim": "Mortality fell 12%.", "location": "Results", + "refs": ["1"], "quote": quote}], "uncited": []})) + _annotated(tmp_path) + cited, _ = check_mod.extract_claims(tmp_path) + assert cited[0].quote == quote + + +# --- the wire format ------------------------------------------------------ + + +def test_the_quote_round_trips_and_validates(tmp_path): + import jsonschema + + quote = 'He said "12%" — <em>not</em> 21%.' + r = RunResults(manuscript="m.pdf", claims=[ + ClaimResult(id=1, claim="X.", location="Intro", refs=["1"], + verdict="supported", quote=quote)]) + path = tmp_path / "results.json" + r.to_json(path) + jsonschema.validate(json.loads(path.read_text()), SCHEMA) + assert RunResults.from_json(path).claims[0].quote == quote + + +def test_a_0_4_x_results_file_without_a_quote_still_loads(tmp_path): + """`from_json` uses `.get` defaults so an older case folder keeps opening — + the quote is absent, not empty-because-the-manuscript-was-silent.""" + legacy = { + "manuscript": "m.pdf", "date": "2026-01-01", + "claims": [{"id": 1, "claim": "X.", "location": "Intro", "refs": ["1"], + "verdict": "supported", "note": "n"}], + } + path = tmp_path / "results.json" + path.write_text(json.dumps(legacy)) + assert RunResults.from_json(path).claims[0].quote == "" + + +# --- it reaches the reader ------------------------------------------------ + + +def test_the_quote_reaches_all_three_formats(tmp_path): + from papertrace.report import write_reports + + quote = "Mortality fell by 12% in the subgroup over 65 (HR 0.88)." + claim = ClaimResult(id=1, claim="Mortality fell 12%.", location="Results", + refs=["1"], verdict="supported", note="Stated.", quote=quote) + write_reports(RunResults(manuscript="m.pdf", claims=[claim]), None, tmp_path, png=False) + for name in ("report.md", "report_editor.html", "report_terminal.html"): + text = " ".join((tmp_path / name).read_text().split()) + assert "Mortality fell by 12% in the subgroup over 65" in text, f"{name} drops the quote" + + +# --- helpers -------------------------------------------------------------- + + +def _annotated(case: Path) -> None: + d = case / "ingest" / "manuscript" + d.mkdir(parents=True, exist_ok=True) + (d / "annotated.md").write_text("<!-- block_0001, page 1 -->\nX causes Y [1].\n") + + +def _manifest(): + from papertrace.models import RefEntry, RefManifest + + return RefManifest(manuscript="m.pdf", entries=[ + RefEntry(num="1", raw="Smith J (2020)", status="retrieved", slug="smith-2020", + pdf_path="/nonexistent/smith-2020.pdf")]) + + +def _stub_source(case: Path, monkeypatch, fake_ask) -> None: + """A cited source already ingested, so `check_claims` reaches the model.""" + d = case / "ingest" / "smith-2020" + d.mkdir(parents=True, exist_ok=True) + (d / "annotated.md").write_text("<!-- block_0001, page 1 -->\nHR 0.88 (0.79-0.98).\n") + json.dump({"doc": "smith-2020.pdf", "pages": 1, "converter": "pymupdf", + "blocks": [{"id": "block_0001", "page": 1, "type": "text", + "bbox": [0, 0, 10, 10], "text": "HR 0.88 (0.79-0.98)."}]}, + (d / "source_map.json").open("w")) + monkeypatch.setattr(check_mod, "_ask", fake_ask) + monkeypatch.setattr(check_mod, "_stale_ingest", lambda *a, **k: False) + + +# --- gate 4: the failure paths still degrade honestly --------------------- + + +def test_a_malformed_extraction_response_still_raises_rather_than_guessing(tmp_path, monkeypatch): + """Adding a field must not turn a broken response into a partial success. + Extraction has no honest half-answer — the manuscript was never read — so + it raises and `check` reports the run as failed.""" + import pytest + + monkeypatch.setattr(check_mod, "_ask", lambda p, m=None: "not json at all") + _annotated(tmp_path) + with pytest.raises((ValueError, RuntimeError)): + check_mod.extract_claims(tmp_path) + + +def test_a_quoted_claim_whose_source_was_never_retrieved_is_still_not_retrieved(tmp_path): + """A quote is manuscript-side evidence and says nothing about the source. + A claim with a perfect quote and no obtainable source must still be + `not_retrieved`, with no model call — the rule this whole codebase is + built on.""" + from papertrace.models import RefEntry, RefManifest + + calls = [] + manifest = RefManifest(manuscript="m.pdf", entries=[ + RefEntry(num="1", raw="Smith J (2020)", status="paywalled", slug="smith-2020")]) + claim = ClaimResult(id=1, claim="Mortality fell 12%.", location="Results", + refs=["1"], quote="Mortality fell by 12% (HR 0.88).") + + check_mod.check_claims([claim], manifest, tmp_path, backend="pymupdf") + + assert claim.verdict == "not_retrieved" + assert calls == [], "an unretrieved source must cost no model call" + assert claim.quote == "Mortality fell by 12% (HR 0.88).", "the quote survives the gap" + + +def test_an_unchecked_claim_keeps_its_quote(tmp_path, monkeypatch): + """The source was available and the call failed. The verdict is `unchecked` + and the quote is still the manuscript's sentence — a failed check does not + retract what the manuscript says.""" + def boom(prompt, model=None): + raise RuntimeError("claude -p timed out") + + _stub_source(tmp_path, monkeypatch, boom) + claim = ClaimResult(id=1, claim="Mortality fell 12%.", location="Results", + refs=["1"], quote="Mortality fell by 12% (HR 0.88).") + + check_mod.check_claims([claim], _manifest(), tmp_path, backend="pymupdf") + + assert claim.verdict == "unchecked" + assert "timed out" in claim.note or "claude" in claim.note.lower() + assert claim.quote == "Mortality fell by 12% (HR 0.88)." + + +# --- a claim judged without its own sentence says so ---------------------- + + +def test_a_claim_judged_on_its_paraphrase_discloses_that(): + """The verbatim quote is what makes a verdict trustworthy, so a verdict + reached without one is weaker evidence and the reader is told. Otherwise + "no quote" is indistinguishable from "quote identical to the paraphrase", + and the README's promise that the report shows what was judged is false.""" + from papertrace.disclosures import NO_QUOTE_TOKEN, claim_disclosures + + judged = ClaimResult(id=1, claim="Mortality fell.", location="Results", + refs=["3"], verdict="supported", source_page=2) + keys = [d.key for d in claim_disclosures(judged)] + assert "no_quote" in keys + assert any(NO_QUOTE_TOKEN in d.text for d in claim_disclosures(judged)) + + +def test_a_claim_with_a_quote_says_nothing_extra(): + from papertrace.disclosures import claim_disclosures + + quoted = ClaimResult(id=1, claim="Mortality fell.", location="Results", refs=["3"], + verdict="supported", source_page=2, quote="Mortality fell by 12%.") + assert "no_quote" not in [d.key for d in claim_disclosures(quoted)] + + +def test_an_unretrieved_claim_is_not_nagged_about_a_missing_quote(): + """Nothing judged it, so the quote changed nothing. Firing here would put + the notice on every gap in the register and train readers to ignore it.""" + from papertrace.disclosures import claim_disclosures + + gap = ClaimResult(id=1, claim="Mortality fell.", location="Results", + refs=["3"], verdict="not_retrieved") + assert "no_quote" not in [d.key for d in claim_disclosures(gap)] + + +# --- how the SOURCES were read, disclosed where verdicts are read --------- + + +def test_a_flat_read_source_is_named_in_all_three_formats(tmp_path): + """`RunResults.converter` is the *manuscript's* backend, and the terminal + line was the only place that said anything about the sources. Now that a + source can be read either way, a verdict resting on a linearized table has + to say so in the report — a table's rows are exactly the evidence a + subgroup claim turns on.""" + from papertrace.disclosures import SOURCE_FIDELITY_TOKEN, run_disclosures + + r = RunResults( + manuscript="m.pdf", converter="docling 2.1.0", + claims=[ClaimResult(id=1, claim="x", location="Intro", refs=["4"], + verdict="supported", source_slug="flat-2020", source_page=1)], + source_converters={"flat-2020": "pymupdf", "rich-2021": "docling 2.1.0"}, + ) + fired = [d for d in run_disclosures(r, None) if d.key == "source_fidelity"] + assert fired, "a flat-read source disclosed nothing" + assert "flat-2020" in fired[0].text + + write_reports(r, None, tmp_path, png=False) + for name in ("report.md", "report_editor.html", "report_terminal.html"): + text = " ".join((tmp_path / name).read_text().split()) + assert SOURCE_FIDELITY_TOKEN in text, f"{name} drops the source-fidelity notice" + + +def test_sources_all_read_layout_aware_disclose_nothing(): + """No loss, nothing to warn about. A notice that fires on the good path is + a notice readers learn to skip.""" + from papertrace.disclosures import run_disclosures + + r = RunResults(manuscript="m.pdf", converter="docling 2.1.0", + source_converters={"a-2020": "docling 2.1.0", "b-2021": "docling 2.1.0"}) + assert [d for d in run_disclosures(r, None) if d.key == "source_fidelity"] == [] + + +def test_a_run_that_recorded_no_source_converters_claims_nothing(): + """A 0.4.x results.json has no such field. Absent is not "all flat" — the + run simply never recorded it, and inventing either answer is the failure + this codebase is built to avoid.""" + from papertrace.disclosures import run_disclosures + + r = RunResults(manuscript="m.pdf", converter="pymupdf") + assert [d for d in run_disclosures(r, None) if d.key == "source_fidelity"] == [] + + +def test_source_converters_round_trip(tmp_path): + r = RunResults(manuscript="m.pdf", source_converters={"a-2020": "pymupdf"}) + p = tmp_path / "results.json" + r.to_json(p) + assert RunResults.from_json(p).source_converters == {"a-2020": "pymupdf"} diff --git a/tests/test_console.py b/tests/test_console.py index 96cff70..713ecb0 100644 --- a/tests/test_console.py +++ b/tests/test_console.py @@ -109,6 +109,22 @@ def test_report_says_cited_sources_are_read_as_flat_text(tmp_path): assert "flat" in out.lower() +def test_no_case_hint_does_not_claim_a_search_it_never_ran(tmp_path, monkeypatch): + """A staged command takes no manuscript path, so `_stage_case`'s hint has + nothing to look beside — it only ever checks the current working + directory. "no case folder found here" reads as if a broader search came + up empty; it must say what was actually checked instead.""" + from typer.testing import CliRunner + + from papertrace.cli import app + + monkeypatch.chdir(tmp_path) + res = CliRunner().invoke(app, ["report"]) + assert res.exit_code == 2 + out = " ".join(res.output.split()) + assert "current directory" in out.lower() + + # --- the noise, silenced at the only seam that works ---------------------- diff --git a/tests/test_context_extraction.py b/tests/test_context_extraction.py new file mode 100644 index 0000000..02223e8 --- /dev/null +++ b/tests/test_context_extraction.py @@ -0,0 +1,329 @@ +"""Extraction is told where the citations are, instead of being asked to +describe where it looked. + +The old flow threw the location away and then spent 130 lines getting it back: +the model returned a paraphrase plus a free-text `location` ("Methods ¶2"), and +Python guessed which of several `[3]` markers that paraphrase had come from — +normalising both sides, scoring them with `SequenceMatcher`, and accepting a +pairing only on `ratio ≥ 0.45` **and** `margin ≥ 0.10`. Everything it could not +decide became `uncertain`, which is honest but was avoidable. + +The citation inventory was already built, deterministically, from +`source_map.json` — just *after* the model call rather than before it. Now it +goes into the prompt with a stable `ctx_NNNN` per occurrence, claims come back +carrying the ids they were extracted from, and coverage is a set lookup. + +The uncertainty does not vanish; it moves to one place. A claim that cites a +label and names no usable context for it leaves that label's unreached +occurrences `uncertain` — because a claim *did* reach one of them and nobody +can say which. That is the same third status, kept for the one case that can +still produce it. +""" + +import json +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from papertrace import check as check_mod # noqa: E402 +from papertrace.models import ClaimResult, RunResults # noqa: E402 + +SCHEMA = json.loads((Path(__file__).parent.parent / "schemas" / "results.schema.json").read_text()) + + +# --- fixtures -------------------------------------------------------------- + + +def _case(tmp_path: Path, blocks: list[dict]) -> Path: + """A case folder whose manuscript source map holds `blocks`.""" + d = tmp_path / "ingest" / "manuscript" + d.mkdir(parents=True, exist_ok=True) + smap = {"doc": "m.pdf", "pages": 1, "converter": "pymupdf", "blocks": blocks} + (d / "source_map.json").write_text(json.dumps(smap)) + (d / "annotated.md").write_text("\n".join(b["text"] for b in blocks)) + (d / "clean.md").write_text("\n".join(b["text"] for b in blocks)) + return tmp_path + + +def _block(bid: str, text: str, page: int = 1, heading: str = "Results") -> dict: + return {"id": bid, "page": page, "type": "text", "bbox": [0, 0, 10, 10], + "text": text, "heading_path": [heading]} + + +TWO_PLACES = [ + _block("block_0001", "Mortality fell by 12% in the over-65 subgroup [3]."), + _block("block_0002", "Readmission was unchanged across all strata [3]."), +] + + +# --- the inventory reaches the model -------------------------------------- + + +def test_the_extraction_prompt_carries_the_citation_inventory(tmp_path, monkeypatch): + """The whole point: the model is shown the places, so it never has to be + asked to describe one.""" + seen = {} + monkeypatch.setattr(check_mod, "_ask", lambda p, m=None: seen.setdefault("p", p) and "" or + json.dumps({"cited": [], "uncited": []})) + check_mod.extract_claims(_case(tmp_path, TWO_PLACES)) + + p = seen["p"] + assert "ctx_0001" in p and "ctx_0002" in p, "the inventory is not in the prompt" + assert "over-65 subgroup" in p, "an inventory without the sentences names nothing" + assert "block_0001" not in p, "internal ids are not the model's to copy" + + +def test_the_inventory_is_ordered_and_stable(): + """`ctx_NNNN` is assigned in reading order over the occurrence list, and the + map back to real occurrence ids is built in the same pass — so no consumer + ever re-derives the pairing by position. Re-deriving it is the + reading-order-zipping mistake in a new costume.""" + occs = [ + {"id": "block_0001:46:3", "label": "3", "page": 1, "section": "Results", + "sentence": "A [3]."}, + {"id": "block_0002:44:3", "label": "3", "page": 2, "section": "Discussion", + "sentence": "B [3]."}, + ] + text, mapping = check_mod._render_inventory(occs) + assert mapping == {"ctx_0001": "block_0001:46:3", "ctx_0002": "block_0002:44:3"} + assert "ctx_0001" in text and "ctx_0002" in text + assert text.index("ctx_0001") < text.index("ctx_0002") + + +# --- a claim keeps the context it came from ------------------------------- + + +def _extract(tmp_path, blocks, cited, monkeypatch): + monkeypatch.setattr(check_mod, "_ask", lambda p, m=None: json.dumps( + {"cited": cited, "uncited": []})) + return check_mod.extract_claims(_case(tmp_path, blocks))[0] + + +def test_a_claim_records_the_occurrence_it_was_extracted_from(tmp_path, monkeypatch): + claims = _extract(tmp_path, TWO_PLACES, [ + {"id": 1, "ctx": ["ctx_0002"], "quote": "Readmission was unchanged across all strata.", + "claim": "Readmission unchanged.", "location": "Results", "refs": ["3"]}, + ], monkeypatch) + assert claims[0].ctx_ids == ["block_0002:44:3"], "the ctx label was not resolved to an id" + + +def test_one_claim_can_span_two_occurrences(tmp_path, monkeypatch): + """A sentence citing [2] and [3] is one claim over two occurrences, and a + group `[7,8]` is one marker over two. Neither can be expressed by a single + id, which is why this is a list.""" + blocks = [_block("block_0001", "Both cohorts agree [2], and the imaging arm too [3].")] + claims = _extract(tmp_path, blocks, [ + {"id": 1, "ctx": ["ctx_0001", "ctx_0002"], "quote": "Both cohorts agree, and the " + "imaging arm too.", "claim": "Cohorts agree.", "location": "Results", + "refs": ["2", "3"]}, + ], monkeypatch) + assert len(claims[0].ctx_ids) == 2 + + +def test_an_unknown_context_id_is_dropped_not_guessed(tmp_path, monkeypatch): + """A model returning `ctx_9999` has told us nothing. Falling back to "the + first occurrence of that label" would manufacture exactly the confident, + wrong attribution this redesign removes.""" + claims = _extract(tmp_path, TWO_PLACES, [ + {"id": 1, "ctx": ["ctx_9999"], "quote": "q", "claim": "c", + "location": "Results", "refs": ["3"]}, + ], monkeypatch) + assert claims[0].ctx_ids == [] + + +def test_a_missing_ctx_field_is_not_an_error(tmp_path, monkeypatch): + """Older prompts and sloppier answers omit it. The claim is still a claim — + it simply carries no context, and coverage says so.""" + claims = _extract(tmp_path, TWO_PLACES, [ + {"id": 1, "quote": "q", "claim": "c", "location": "Results", "refs": ["3"]}, + ], monkeypatch) + assert claims[0].ctx_ids == [] + assert claims[0].claim == "c" + + +# --- coverage becomes bookkeeping ----------------------------------------- + + +def _cov(tmp_path, blocks, claims): + return check_mod.coverage_audit(_case(tmp_path, blocks), claims) + + +def test_an_occurrence_is_covered_only_if_a_claim_named_it(tmp_path): + c = ClaimResult(id=1, claim="c", location="Results", refs=["3"], + ctx_ids=["block_0002:44:3"]) + cov = _cov(tmp_path, TWO_PLACES, [c]) + by_id = {o["id"]: o for o in cov["occurrences"]["items"]} + assert by_id["block_0002:44:3"]["status"] == "covered" + assert by_id["block_0002:44:3"]["claim_id"] == 1 + assert by_id["block_0001:46:3"]["status"] == "uncovered" + assert cov["occurrences"]["covered"] == 1 + assert cov["occurrences"]["uncertain"] == 0 + + +def test_a_claim_citing_a_label_with_no_usable_context_leaves_it_uncertain(tmp_path): + """The one case that can still produce doubt, and the reason `uncertain` + survives the redesign: a claim reached one of these two sentences and + nothing can say which. Calling both uncovered would cry wolf; calling + either covered would be the overstatement.""" + c = ClaimResult(id=1, claim="c", location="Results", refs=["3"], ctx_ids=[]) + cov = _cov(tmp_path, TWO_PLACES, [c]) + assert {o["status"] for o in cov["occurrences"]["items"]} == {"uncertain"} + assert cov["occurrences"]["uncertain"] == 2 + assert cov["occurrences"]["covered"] == 0 + assert cov["labels_uncertain_only"] == ["3"] + + +def test_an_uncited_label_is_uncovered_not_uncertain(tmp_path): + """No claim mentioned [3] at all, so nothing is in doubt — it was simply + missed, which is the finding the audit exists to report.""" + cov = _cov(tmp_path, TWO_PLACES, []) + assert {o["status"] for o in cov["occurrences"]["items"]} == {"uncovered"} + assert cov["missing"] == ["3"] + + +def test_a_partly_placed_label_reports_both_states(tmp_path): + """One claim names one of the two places; a second claim cites the label + and names nothing. The named place is covered, the other is uncertain — + not uncovered, because a claim did reach it.""" + named = ClaimResult(id=1, claim="a", location="Results", refs=["3"], + ctx_ids=["block_0001:46:3"]) + vague = ClaimResult(id=2, claim="b", location="Results", refs=["3"], ctx_ids=[]) + cov = _cov(tmp_path, TWO_PLACES, [named, vague]) + by_id = {o["id"]: o["status"] for o in cov["occurrences"]["items"]} + assert by_id["block_0001:46:3"] == "covered" + assert by_id["block_0002:44:3"] == "uncertain" + assert cov["labels_partially_covered"] == ["3"] + + +def test_a_claim_that_named_nothing_is_reported_as_unattributed(tmp_path): + c = ClaimResult(id=7, claim="c", location="Results", refs=["3"], ctx_ids=[]) + cov = _cov(tmp_path, TWO_PLACES, [c]) + assert cov["attribution"]["claims_unattributed"] == [{"claim_id": 7, "label": "3"}] + + +# --- the label-level contract is byte-for-byte unchanged ------------------ + + +def test_the_label_level_keys_keep_their_meaning(tmp_path): + """CLAUDE.md pins these three: `evals/align.py` reads `missing` as a list of + label strings to decide whether an unmatched gold case is the tool's fault + or the harness's. Reshaping them would move that blame with no test going + red.""" + c = ClaimResult(id=1, claim="c", location="Results", refs=["3"], + ctx_ids=["block_0001:46:3"]) + cov = _cov(tmp_path, TWO_PLACES, [c]) + assert cov["labels_in_text"] == ["3"] + assert cov["covered"] == ["3"] + assert cov["missing"] == [] + assert all(isinstance(x, str) for x in cov["missing"] + cov["covered"]) + + +# --- what is gone --------------------------------------------------------- + + +def test_the_similarity_matcher_and_its_thresholds_are_gone(): + """The deletion this change is for. Leaving them behind as dead code would + keep two answers to "which occurrence" in the tree, and the report's + attribution caveat would still be describing one of them.""" + for name in ("_attribute_label", "_normalize_for_match", "_ratio", + "_location_matches", "OCCURRENCE_MIN_RATIO", "OCCURRENCE_MIN_MARGIN"): + assert not hasattr(check_mod, name), f"{name} survived the redesign" + + +def test_the_audit_no_longer_advertises_a_similarity_threshold(tmp_path): + cov = _cov(tmp_path, TWO_PLACES, []) + assert "min_ratio" not in cov["attribution"] + assert "min_margin" not in cov["attribution"] + assert "similarity" not in cov["attribution"]["method"] + assert cov["schema"] == "coverage/3" + + +# --- the wire format ------------------------------------------------------ + + +def test_ctx_ids_round_trip_and_validate(tmp_path): + import jsonschema + + r = RunResults(manuscript="m.pdf", claims=[ + ClaimResult(id=1, claim="c", location="Results", refs=["3"], + ctx_ids=["block_0001:46:3"], verdict="supported")]) + p = tmp_path / "results.json" + r.to_json(p) + jsonschema.validate(json.loads(p.read_text()), SCHEMA) + assert RunResults.from_json(p).claims[0].ctx_ids == ["block_0001:46:3"] + + +def test_a_0_4_x_claim_without_ctx_ids_still_loads(tmp_path): + p = tmp_path / "results.json" + p.write_text(json.dumps({ + "manuscript": "m.pdf", + "claims": [{"id": 1, "claim": "c", "location": "Results", "refs": ["3"], + "verdict": "supported"}], + })) + assert RunResults.from_json(p).claims[0].ctx_ids == [] + + +# --- gate 4: the new failure paths degrade honestly ----------------------- + + +def test_a_manuscript_with_no_bracketed_citations_still_extracts(tmp_path, monkeypatch): + """The inventory can legitimately be empty — an author-year paper, or a + style the detector cannot see. Extraction must still run (Task 2, the + uncited register, does not depend on citations existing) and the prompt must + say the list is empty rather than trailing off after a heading.""" + seen = {} + + def fake(p, m=None): + seen["p"] = p + return json.dumps({"cited": [], "uncited": [{"id": 1, "claim": "x", "location": "Intro"}]}) + + monkeypatch.setattr(check_mod, "_ask", fake) + blocks = [_block("block_0001", "Prevalence is high (Smith 2020).")] + cited, uncited = check_mod.extract_claims(_case(tmp_path, blocks)) + + assert cited == [] + assert len(uncited) == 1 + assert "(none found)" in seen["p"], "an empty inventory must say so" + + +def test_a_case_with_no_ingest_artifacts_does_not_crash(tmp_path, monkeypatch): + """`citation_occurrences` returns `("none")` when there is neither a source + map nor clean.md. Extraction still has a manuscript to read.""" + d = tmp_path / "ingest" / "manuscript" + d.mkdir(parents=True) + (d / "annotated.md").write_text("Some prose with no citations.") + monkeypatch.setattr(check_mod, "_ask", lambda p, m=None: json.dumps( + {"cited": [], "uncited": []})) + assert check_mod.extract_claims(tmp_path) == ([], []) + + +def test_a_truncated_inventory_is_recorded_and_disclosed(tmp_path, monkeypatch): + """Contexts past the cut are never offered to the model, so nothing can be + attributed to them and they can only come back `uncovered` — a coverage + figure depressed by a limit rather than by the paper. That is exactly the + class of thing this codebase discloses rather than absorbing.""" + from papertrace.check import Truncations + + monkeypatch.setattr(check_mod, "CONTEXT_CHAR_LIMIT", 80) + monkeypatch.setattr(check_mod, "_ask", lambda p, m=None: json.dumps( + {"cited": [], "uncited": []})) + blocks = [_block(f"block_{i:04d}", f"Finding number {i} was reported [{i + 1}].") + for i in range(1, 8)] + t = Truncations() + check_mod.extract_claims(_case(tmp_path, blocks), truncations=t) + + assert "citation contexts" in t.report(), "the inventory cut was not recorded" + assert t.report()["citation contexts"]["limit"] == 80 + + +def test_the_truncated_inventory_cut_reaches_the_report(tmp_path): + """A recorded cut nobody renders is not a disclosure. `_truncation` names + every key it was given, so this only has to prove the key travels.""" + from papertrace.disclosures import TRUNCATION_TOKEN, run_disclosures + + r = RunResults(manuscript="m.pdf", + truncated={"citation contexts": {"chars": 900, "limit": 80}}) + d = next(x for x in run_disclosures(r, None) if x.key == "truncation") + assert TRUNCATION_TOKEN in d.text + assert "citation contexts" in d.text diff --git a/tests/test_coverage.py b/tests/test_coverage.py index 2b11d0a..96f85b4 100644 --- a/tests/test_coverage.py +++ b/tests/test_coverage.py @@ -140,7 +140,8 @@ def boom(prompt, model=None): monkeypatch.setattr(check_mod, "_ask", boom) errors = [] - check_claims(claims, manifest, tmp_path, on_error=lambda s, m: errors.append((s, m))) + check_claims(claims, manifest, tmp_path, on_error=lambda s, m: errors.append((s, m)), + backend="pymupdf") assert len(calls) == 2 # one retry before giving up assert claims[0].verdict == "unchecked" @@ -149,19 +150,40 @@ def boom(prompt, model=None): # and a genuinely missing source still reads not_retrieved claims2 = [ClaimResult(id=2, claim="d", location="Intro", refs=["9"])] - check_claims(claims2, manifest, tmp_path) + check_claims(claims2, manifest, tmp_path, backend="pymupdf") assert claims2[0].verdict == "not_retrieved" +def _ingested(dirpath, slug: str, pages: int = 2) -> None: + """An ingested source: the text AND the map that proves where its blocks are. + + Both, always — a source map is no longer optional. `check` validates every + substantive verdict's page and block against it, so a source without one + can produce no verdict at all. + """ + from papertrace.models import Block, SourceMap + + dirpath.mkdir(parents=True, exist_ok=True) + (dirpath / "annotated.md").write_text( + f"<!-- block_0001, page 1 -->\nText of {slug}.\n" + f"<!-- block_0002, page 2 -->\nMore of {slug}.\n" + ) + SourceMap( + doc=f"{slug}.pdf", pages=pages, + blocks=[ + Block("block_0001", "text", 1, (0.0, 0.0, 100.0, 20.0), [], f"Text of {slug}."), + Block("block_0002", "text", 2, (0.0, 0.0, 100.0, 20.0), [], f"More of {slug}."), + ], + ).to_json(dirpath / "source_map.json") + + def _one_source_manifest(tmp_path, *entries): - """Manifest + an ingested annotated.md for every retrieved entry.""" + """Manifest + a fully ingested source for every retrieved entry.""" from papertrace.models import RefManifest for e in entries: if e.status in ("retrieved", "provided"): - d = tmp_path / "ingest" / e.slug - d.mkdir(parents=True, exist_ok=True) - (d / "annotated.md").write_text(f"<!-- block_0001, page 1 -->\nText of {e.slug}.") + _ingested(tmp_path / "ingest" / e.slug, e.slug) return RefManifest(manuscript="m.pdf", entries=list(entries)) @@ -186,7 +208,7 @@ def test_malformed_verdict_becomes_unchecked_never_partial(tmp_path, monkeypatch lambda prompt, model=None: '[{"id":1,"note":"no verdict key"},' ' {"id":2,"verdict":"probably fine","note":"junk"}]', ) - check_claims(claims, manifest, tmp_path) + check_claims(claims, manifest, tmp_path, backend="pymupdf") assert claims[0].verdict == "unchecked" assert claims[1].verdict == "unchecked" @@ -219,7 +241,7 @@ def test_multiref_claim_records_only_the_sources_it_could_not_obtain(tmp_path, m ' "source_page":1,"source_block":"block_0001",' ' "anchor_phrases":["Text"]}]', ) - check_claims(claims, manifest, tmp_path) + check_claims(claims, manifest, tmp_path, backend="pymupdf") c = claims[0] assert c.verdict == "supported" @@ -350,7 +372,7 @@ def test_model_may_not_assign_a_pipeline_state(tmp_path, monkeypatch): check_mod, "_ask", lambda prompt, model=None: '[{"id":1,"verdict":"not_retrieved","note":"could not find"}]', ) - check_claims(claims, manifest, tmp_path) + check_claims(claims, manifest, tmp_path, backend="pymupdf") c = claims[0] assert c.verdict == "unchecked" @@ -394,7 +416,7 @@ def test_verdict_without_a_source_page_is_unchecked_not_page_none(tmp_path, monk lambda prompt, model=None: '[{"id":1,"verdict":"supported","note":"ok",' ' "anchor_phrases":["Text"]}]', ) - check_claims(claims, manifest, tmp_path) + check_claims(claims, manifest, tmp_path, backend="pymupdf") assert claims[0].verdict == "unchecked" assert "source_page" in claims[0].note @@ -422,7 +444,7 @@ def test_a_rejected_response_writes_nothing_to_the_claim(tmp_path, monkeypatch): lambda prompt, model=None: '[{"id":1,"verdict":"supported","note":"ok",' ' "source_block":"block_0001","anchor_phrases":["Text"]}]', ) - check_claims(claims, manifest, tmp_path) + check_claims(claims, manifest, tmp_path, backend="pymupdf") c = claims[0] assert c.verdict == "unchecked" @@ -455,7 +477,7 @@ def test_null_anchor_phrases_is_rejected_without_poisoning_siblings(tmp_path, mo ' {"id":2,"verdict":"partial","note":"fine","source_page":2,' ' "source_block":"block_0002","anchor_phrases":["Text"]}]', ) - check_claims(claims, manifest, tmp_path) # must not raise + check_claims(claims, manifest, tmp_path, backend="pymupdf") # must not raise assert claims[0].verdict == "unchecked" assert "anchor_phrases" in claims[0].note @@ -483,12 +505,12 @@ def test_a_bug_in_our_validator_is_not_relabelled_as_the_models_fault(tmp_path, ' "source_page":1,"anchor_phrases":[]}]', ) - def our_bug(entry): + def our_bug(entry, provenance): raise AttributeError("a bug in PaperTrace, not in the model's answer") monkeypatch.setattr(check_mod, "_judgement_from", our_bug) with pytest.raises(AttributeError): - check_claims(claims, manifest, tmp_path) + check_claims(claims, manifest, tmp_path, backend="pymupdf") def test_truncations_cannot_leak_between_runs(tmp_path, monkeypatch): @@ -729,11 +751,37 @@ def test_references_section_occurrences_are_excluded_structurally(tmp_path): cov = coverage_audit(case, []) assert [o["label"] for o in cov["occurrences"]["items"]] == ["4"] - # and the clean.md regex genuinely cannot see it — the headings carry no `##`, - # which is exactly why the structural test is not a restatement of the regex - assert "99" in citation_labels_in_text( + # The two readings now agree, and that is the point of the fix rather than a + # restatement of it. This line used to assert the opposite — that the label + # reading DID see [99] — as proof that the occurrence walk's structural test + # was doing independent work. It was proof of a defect: the headings here + # carry no `##`, the label reading cut on `^##\s+references`, and so a label + # printed only inside the reference list was counted as a body citation and + # reported as an uncovered gap. Both readings cut on + # `models.is_references_heading` now. + assert "99" not in citation_labels_in_text( (case / "ingest" / "manuscript" / "clean.md").read_text() ) + assert cov["labels_in_text"] == ["4"] + + +def test_table_ci_brackets_are_not_read_as_citation_occurrences(tmp_path): + """A results table's own numbers are not citations. `[54, 100]` in a 95% CI + column matches the same bracket-and-comma syntax as a citation group + `[7,8]`, and a live audit read a table's CI columns as citations to + references #54 and #100 — in both the occurrence walk (block-type aware, + reading `source_map.json`) and the label-level count (`labels_in_text`, + which reads flat `clean.md` and has no block-type information, so it must + recognise the table by its own GFM `| ... |` row shape instead).""" + case = _case_with_source_map(tmp_path, [ + {"id": "block_0001", "page": 1, "text": "Uptake was low [4]."}, + {"id": "block_0002", "type": "table", "page": 1, + "text": "| PPV (%) | 83 (5/6) [54, 100] |\n|---|---|\n"}, + ]) + + cov = coverage_audit(case, []) + assert [o["label"] for o in cov["occurrences"]["items"]] == ["4"] + assert cov["labels_in_text"] == ["4"] def test_surplus_claims_are_recorded_without_making_anything_uncertain(tmp_path): @@ -743,11 +791,15 @@ def test_surplus_claims_are_recorded_without_making_anything_uncertain(tmp_path) {"id": "block_0001", "page": 1, "text": "Recruitment reached 500,000 adults [2]."}, ]) claims = [ - ClaimResult(id=1, claim="recruitment reached 500,000 adults", location="", refs=["2"]), + ClaimResult(id=1, claim="recruitment reached 500,000 adults", location="", refs=["2"], + ctx_ids=["block_0001:35:2"]), ClaimResult(id=2, claim="the cohort is large", location="", refs=["2"]), ] cov = coverage_audit(case, claims) + # claim 1 named the only place [2] is cited, so nothing about that place is + # in doubt. Claim 2 named nothing and is reported unattributed — it cannot + # make an occurrence uncertain that another claim has already placed. assert cov["occurrences"]["covered"] == 1 assert cov["occurrences"]["uncertain"] == 0 assert [c["claim_id"] for c in cov["attribution"]["claims_unattributed"]] == [2] @@ -868,6 +920,16 @@ def test_the_occurrence_list_is_capped_with_a_pointer_to_results_json(tmp_path): # --- _judgement_from is total by construction, and stays that way ----------- +def _prov(): + """A three-page source with one block per page — enough that the page-shape + guards below fail on the shape, not on a location that doesn't exist.""" + from papertrace.check import SourceProvenance + + return SourceProvenance( + pages=3, block_pages={"block_0001": 1, "block_0002": 2, "block_0003": 3} + ) + + @pytest.mark.parametrize("page", [ "9" * 5000, # passes isascii() and isdigit(), then int() raises "9" * 4301, # one past CPython's default limit @@ -882,7 +944,10 @@ def test_an_absurdly_long_page_number_degrades_instead_of_raising(page): """ import papertrace.check as check_mod - j, note = check_mod._judgement_from({"id": 1, "verdict": "supported", "source_page": page}) + j, note = check_mod._judgement_from( + {"id": 1, "verdict": "supported", "source_page": page, + "source_block": "block_0003"}, _prov() + ) assert j is None assert "unusable" in note @@ -896,7 +961,10 @@ def test_other_page_shapes_still_degrade_rather_than_raise(page): by breaking another. `" 3 "` is deliberately accepted after stripping.""" import papertrace.check as check_mod - j, note = check_mod._judgement_from({"id": 1, "verdict": "supported", "source_page": page}) + j, note = check_mod._judgement_from( + {"id": 1, "verdict": "supported", "source_page": page, + "source_block": "block_0003"}, _prov() + ) if isinstance(page, str) and page.strip() == "3": assert j is not None and j["source_page"] == 3 # a dict, not a dataclass else: @@ -965,3 +1033,36 @@ def test_both_readers_agree_on_where_the_bibliography_starts(tmp_path): occ, _ = citation_occurrences(tmp_path) assert [o["label"] for o in occ] == ["1"] + + +def test_a_bibliography_heading_the_ingest_did_not_mark_still_ends_the_body(): + """Two boundary rules, one claiming to be the other. `coverage_audit` cut the + body at `^##\\s+references`, which needs ingest to have emitted a markdown + heading — but flat-text ingest guesses headings from font size, and a + `References` line at body size stays body text and is written to `clean.md` + without `##`. `models.is_references_heading` is built for exactly that case + and says True; this cut said False, so every `[N]` printed in the reference + list was counted as a body citation and the audit reported gaps that do not + exist. Reproduced on a generated flat-ingest paper: labels_in_text held [3], + a label the body never cites. + """ + body = ( + "## A Study\n\nBody text citing [1] and [2] here.\n\n" + "References\n\n" + "[1] Alpha A. First paper. 2020. [2] Bravo B. Second. 2021. " + "[3] Gamma G. Never cited in body. 2022.\n" + ) + assert citation_labels_in_text(body) == {"1", "2"} + + +def test_a_sentence_about_references_does_not_end_the_body(): + """The other half of the same rule, and why the plain-line test has to be + exact: a body sentence starting with the word must not swallow the paper.""" + body = "References were checked by hand [1].\n\nMore body citing [2].\n" + assert citation_labels_in_text(body) == {"1", "2"} + + +def test_a_marked_heading_with_a_suffix_still_ends_the_body(): + """Unchanged behaviour for a heading ingest did mark: the prefix rule.""" + body = "Body cites [1].\n\n## References and further reading\n\n[1] A. 2020. [9] B. 2021.\n" + assert citation_labels_in_text(body) == {"1"} diff --git a/tests/test_disclosure_parity.py b/tests/test_disclosure_parity.py index 31af3df..204ab9f 100644 --- a/tests/test_disclosure_parity.py +++ b/tests/test_disclosure_parity.py @@ -100,7 +100,11 @@ def test_every_claim_disclosure_appears_in_all_three_formats(tmp_path, anchor_lo rendered = _render(results, tmp_path) fired = claim_disclosures(claim) - assert {d.key for d in fired} == {"unjudged_refs", "anchor"} + # `no_quote` belongs here: the fixture claim carries no verbatim quote, so + # a verdict on it rests on the paraphrase and the reader is owed that in + # every format. Pinning the set is what makes a newly added disclosure + # arrive in this loop instead of quietly missing one template. + assert {d.key for d in fired} == {"unjudged_refs", "anchor", "no_quote"} for d in fired: for name, body in rendered.items(): assert d.token in body, f"{d.key}: token {d.token!r} missing from {name}" @@ -292,3 +296,127 @@ def test_references_resumed_round_trips_and_older_manifests_still_load(tmp_path) path.write_text(_json.dumps(payload)) jsonschema.validate(_json.loads(path.read_text()), _json.loads(schema_path.read_text())) assert RefManifest.from_json(path).references_resumed is False + + +# --- claims whose HEADLINE is a pipeline state ------------------------------ +# +# `gaps_by_location()` routes any claim whose headline is `not_retrieved` or +# `unchecked` out of the main loop and into the gap section — which printed the +# claim text and nothing else. So a claim citing [1,2] where source 1's check +# failed and source 2 was never obtainable said neither thing, and a +# `not_addressed` from a source that WAS successfully read vanished behind the +# `unchecked` headline that outranks it. + + +def _gap_claim() -> ClaimResult: + """One source read and silent, one source's check failed, one never obtained. + + The headline is `unchecked`: a failed check makes "every available source + was read and none addressed it" an assertion the run cannot make. + """ + from papertrace.models import SourceJudgement + + claim = ClaimResult( + id=4, + claim="the intervention halved readmissions", + location="Discussion", + refs=["1", "2", "3"], + judgements=[ + SourceJudgement(source_slug="read-2019", ref="1", verdict="not_addressed", + note="reports incidence only; silent on readmission"), + SourceJudgement(source_slug="failed-2021", ref="2", verdict="unchecked", + note="check failed (TimeoutError) — the source WAS retrieved"), + ], + unjudged_refs=["3"], + ) + claim.apply_headline() + assert claim.verdict == "unchecked" + return claim + + +def test_a_gap_claims_disclosures_reach_all_three_formats(tmp_path): + claim = _gap_claim() + results = RunResults(manuscript="m.pdf", claims=[claim]) + rendered = _render(results, tmp_path) + + fired = claim_disclosures(claim) + assert {d.key for d in fired} == {"sources", "unjudged_refs"} + for d in fired: + for name, body in rendered.items(): + assert d.token in body, f"{d.key}: token {d.token!r} missing from {name}" + + +def test_a_gap_claim_names_each_source_and_its_verdict(tmp_path): + """The per-source rows themselves, not just the summary. A reader has to be + able to see that [1] was read and said nothing while [2] was never read.""" + rendered = _render(RunResults(manuscript="m.pdf", claims=[_gap_claim()]), tmp_path) + + for name, body in rendered.items(): + assert "read-2019" in body, f"the source that WAS read is missing from {name}" + assert "failed-2021" in body, f"the source whose check failed is missing from {name}" + assert "not_addressed" in body or "DOES NOT ADDRESS" in body, ( + f"a successfully-checked not_addressed verdict is invisible in {name}" + ) + + +def test_a_gap_claim_keeps_its_note_in_every_format(tmp_path): + """The two HTML looks printed no note at all for gap claims.""" + rendered = _render(RunResults(manuscript="m.pdf", claims=[_gap_claim()]), tmp_path) + for name, body in rendered.items(): + assert "silent on readmission" in body, f"per-source note missing from {name}" + + +def test_a_gap_section_row_does_not_label_a_mixed_section_with_one_verdict(tmp_path): + """The editor look printed `items[0].verdict` for the whole row, so a + section holding one `not_retrieved` and one `unchecked` claimed both were + whichever came first.""" + gap = _gap_claim() + other = ClaimResult(id=5, claim="a second claim", location="Discussion", refs=["9"], + verdict="not_retrieved", note="cited source not available (paywalled)") + rendered = _render(RunResults(manuscript="m.pdf", claims=[gap, other]), tmp_path) + + # scoped to the gap table: both verdicts appear elsewhere on the page (the + # summary counts them), so an unscoped assertion passes even unfixed + editor = rendered["report_editor.html"] + table = editor.split('<table class="gaptbl">')[1].split("</table>")[0] + assert "not retrieved" in table, table + assert "unchecked" in table, table + + +# --- provenance without a picture ------------------------------------------- + + +@pytest.mark.parametrize("anchor_located", [False, None]) +def test_an_anchor_state_without_a_crop_still_reaches_every_format(tmp_path, anchor_located): + """The disclosure used to be gated on `evidence_image`, so a verdict with a + page and no crop told the reader nothing about what backed it.""" + claim = _claim(evidence_image=None, anchor_located=anchor_located, source_page=3) + rendered = _render(RunResults(manuscript="m.pdf", claims=[claim]), tmp_path) + + anchor = next(d for d in claim_disclosures(claim) if d.key == "anchor") + for name, body in rendered.items(): + assert anchor.token in body, f"anchor token missing from {name}" + + +# --- the HTML reports actually escape what they interpolate ------------------ + + +def test_markup_in_source_text_cannot_reach_the_html_reports_unescaped(tmp_path): + """`select_autoescape(["html"])` matches names ending `.html`. The templates + are named `report_editor.html.j2`, so nothing ever matched and autoescape + was off for all three formats — including the two that emit HTML. + + It went unnoticed because Europe PMC pre-escapes the markup in its titles, + so the one field carrying angle brackets arrived already safe. Cited source + PDFs are downloaded from third parties, and their text reaches the report. + """ + hostile = '<script>alert("xss")</script>' + claim = _claim(claim=f"a claim containing {hostile}", verdict="supported") + rendered = _render(RunResults(manuscript="m.pdf", claims=[claim]), tmp_path) + + for name in ("report_editor.html", "report_terminal.html"): + assert "<script>" not in rendered[name], f"{name} interpolated raw markup" + assert "<script>" in rendered[name], f"{name} did not escape it" + + # markdown is not HTML and must not grow entities — it stays verbatim + assert hostile in rendered["report.md"] diff --git a/tests/test_init.py b/tests/test_init.py new file mode 100644 index 0000000..ac50454 --- /dev/null +++ b/tests/test_init.py @@ -0,0 +1,56 @@ +"""`papertrace init` then `papertrace run paper.pdf` used to leave an orphaned +`case/sources/` — `run` and `refs` name their own folder after the paper, so a +hand-made `./case/` is only ever used again if the user remembers `-c`. + +`init --for <paper>` closes that gap by naming the folder the same way `run` +would, so a plain follow-up `papertrace run paper.pdf` finds it automatically. +""" + +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from typer.testing import CliRunner # noqa: E402 + +from papertrace.cli import app, default_case # noqa: E402 + + +def _pdf(tmp_path: Path, name: str = "zhang2025.pdf") -> Path: + pdf = tmp_path / name + pdf.write_bytes(b"%PDF-1.4\n%%EOF") + return pdf + + +def test_init_with_no_arguments_still_makes_the_legacy_case_folder(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + res = CliRunner().invoke(app, ["init"]) + assert res.exit_code == 0, res.output + assert (tmp_path / "case" / "sources").is_dir() + + +def test_init_for_a_paper_names_the_folder_the_way_run_would(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + pdf = _pdf(tmp_path) + res = CliRunner().invoke(app, ["init", "--for", str(pdf)]) + assert res.exit_code == 0, res.output + expected = default_case(pdf) + assert (expected / "sources").is_dir() + assert not (tmp_path / "case").exists() + + +def test_init_for_a_paper_says_run_will_find_it_automatically(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + pdf = _pdf(tmp_path) + out = " ".join(CliRunner().invoke(app, ["init", "--for", str(pdf)]).output.split()) + assert "automatically" in out.lower() + assert "-c" not in out + + +def test_an_explicit_case_folder_still_wins_over_for(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + pdf = _pdf(tmp_path) + res = CliRunner().invoke(app, ["init", "chosen-name", "--for", str(pdf)]) + assert res.exit_code == 0, res.output + assert (tmp_path / "chosen-name" / "sources").is_dir() + assert not default_case(pdf).exists() diff --git a/tests/test_multisource.py b/tests/test_multisource.py index 78a719d..ffed9bd 100644 --- a/tests/test_multisource.py +++ b/tests/test_multisource.py @@ -304,14 +304,35 @@ def test_a_hand_built_legacy_results_json_loads_and_still_renders(tmp_path): # --- the fan-out ---------------------------------------------------------- +def _ingested(dirpath, slug: str, pages: int = 2) -> None: + """An ingested source: the text AND the map that proves where its blocks are. + + Both, always — a source map is no longer optional. `check` validates every + substantive verdict's page and block against it, so a source without one + can produce no verdict at all. + """ + from papertrace.models import Block, SourceMap + + dirpath.mkdir(parents=True, exist_ok=True) + (dirpath / "annotated.md").write_text( + f"<!-- block_0001, page 1 -->\nText of {slug}.\n" + f"<!-- block_0002, page 2 -->\nMore of {slug}.\n" + ) + SourceMap( + doc=f"{slug}.pdf", pages=pages, + blocks=[ + Block("block_0001", "text", 1, (0.0, 0.0, 100.0, 20.0), [], f"Text of {slug}."), + Block("block_0002", "text", 2, (0.0, 0.0, 100.0, 20.0), [], f"More of {slug}."), + ], + ).to_json(dirpath / "source_map.json") + + def _case(tmp_path: Path, slugs: list[str]): - """A case folder with an ingested annotated.md per source.""" + """A case folder with a fully ingested source per slug.""" from papertrace.models import RefEntry, RefManifest for slug in slugs: - d = tmp_path / "ingest" / slug - d.mkdir(parents=True) - (d / "annotated.md").write_text(f"<!-- block_0001, page 1 -->\nText of {slug}.") + _ingested(tmp_path / "ingest" / slug, slug) manifest = RefManifest( manuscript="m.pdf", entries=[ @@ -346,7 +367,7 @@ def fake_ask(prompt, model=None): }]) monkeypatch.setattr(check_mod, "_ask", fake_ask) - check_claims([claim], manifest, tmp_path) + check_claims([claim], manifest, tmp_path, backend="pymupdf") assert sorted(seen) == ["a-2020", "b-2021", "c-2022"], "one call per source" assert {j.source_slug: j.verdict for j in claim.judgements} == verdict_for @@ -378,7 +399,7 @@ def test_unjudged_refs_now_means_could_not_be_obtained(tmp_path, monkeypatch): "id": 1, "verdict": "supported", "note": "n", "source_page": 1, "source_block": "block_0001", "anchor_phrases": ["Text of a-2020"], }])) - check_claims([claim], manifest, tmp_path) + check_claims([claim], manifest, tmp_path, backend="pymupdf") assert [j.source_slug for j in claim.judgements] == ["a-2020"] assert claim.unjudged_refs == ["2"], "the paywalled co-citation, and only that" @@ -402,7 +423,7 @@ def fake_ask(prompt, model=None): }]) monkeypatch.setattr(check_mod, "_ask", fake_ask) - check_claims([claim], manifest, tmp_path) + check_claims([claim], manifest, tmp_path, backend="pymupdf") by_slug = {j.source_slug: j.verdict for j in claim.judgements} assert by_slug["good-2020"] == "supported" @@ -501,3 +522,51 @@ def test_a_claim_no_cited_source_addresses_says_so(tmp_path): # Only the new bucket is asserted verbatim; the markdown counts line is the # one with the number inline, so it is where that is checkable. assert "◌ **Does not address:** 1" in out["report.md"] + + +# --- the headline is one source's verdict, never the claim's ---------------- + + +def test_a_multi_source_headline_names_how_many_sources_it_ranked(): + """`❌ CONTRADICTED` on a four-source claim is one source's verdict, but it + reads as a statement about the claim. One dissenter of four is a finding + worth surfacing and worth *qualifying* — a compound sentence may draw + different parts from different references legitimately.""" + claim = _four_source_claim() + assert claim.verdict == "contradicted" + assert claim.headline_qualifier() == "most adverse of 4 cited sources" + + +def test_a_single_source_headline_carries_no_qualifier(): + """With one source the headline *is* a statement about the claim, and + "most adverse of 1" would be noise that trains readers to skip the line.""" + claim = ClaimResult(id=1, claim="x", location="Intro", refs=["3"], + judgements=[_j("a-2022", "3", "supported", note="Yes.")]) + claim.apply_headline() + assert claim.headline_qualifier() == "" + + +def test_a_claim_with_no_judgements_carries_no_qualifier(): + """`not_retrieved` ranked nothing. Claiming it was the most adverse of some + number of sources would invent a comparison that never happened.""" + claim = ClaimResult(id=1, claim="x", location="Intro", refs=["3"]) + assert claim.verdict == "not_retrieved" + assert claim.headline_qualifier() == "" + + +def test_the_headline_qualifier_reaches_all_three_formats(tmp_path): + """The parity contract again: a status line qualified in the markdown but + not the HTML would leave the overstatement exactly where it is most often + read.""" + out = _render(_four_source_claim(), tmp_path) + for name, text in out.items(): + assert "most adverse of 4 cited sources" in text, f"{name} drops the qualifier" + + +def test_a_single_source_claim_is_not_qualified_in_any_format(tmp_path): + claim = ClaimResult(id=1, claim="x", location="Intro", refs=["3"], + judgements=[_j("a-2022", "3", "supported", note="Yes.")]) + claim.apply_headline() + out = _render(claim, tmp_path) + for name, text in out.items(): + assert "most adverse of" not in text, f"{name} qualifies a single-source claim" diff --git a/tests/test_packaging.py b/tests/test_packaging.py index 2d14e1a..5fc2e4c 100644 --- a/tests/test_packaging.py +++ b/tests/test_packaging.py @@ -297,3 +297,29 @@ def _mentions_evals(value) -> bool: return False assert not _mentions_evals(WHEEL), f"wheel target mentions evals: {WHEEL}" + + +# --------------------------------------------------------------------------- +# the layout backend is part of the tool, not an upgrade to it +# --------------------------------------------------------------------------- + + +def test_docling_is_a_base_dependency(): + """Cited sources are judged on their own text, and the evidence for a + subgroup claim is usually a table row — read flat, the row is gone. So the + layout backend cannot be optional: `pip install papertrace` must be able to + read a table in a cited source.""" + names = [d.split(">")[0].split("[")[0].strip() for d in PYPROJECT["project"]["dependencies"]] + assert "docling" in names, "docling must be a base dependency, not an extra" + + +def test_the_docling_and_full_extras_still_resolve(): + """`pip install 'papertrace[docling]'` and `[full]` appear throughout the + 0.4.x docs and in other people's notes. Deleting the extras would turn + those into an install error, so they stay as aliases — `[docling]` empty + because it is now redundant, `[full]` keeping only playwright.""" + extras = PYPROJECT["project"]["optional-dependencies"] + assert "docling" in extras and extras["docling"] == [] + full = [d.split(">")[0].strip() for d in extras["full"]] + assert "playwright" in full + assert "docling" not in full, "docling in [full] would install it twice over" diff --git a/tests/test_pipeline_split.py b/tests/test_pipeline_split.py new file mode 100644 index 0000000..bf383a5 --- /dev/null +++ b/tests/test_pipeline_split.py @@ -0,0 +1,168 @@ +"""`ingest()` the Typer command and `_ingest_pipeline()` the logic it delegates to +are two different things now, on purpose. + +A positional call to a Typer-decorated function silently receives an `OptionInfo` +sentinel instead of the value the help screen shows, because Typer only resolves +its defaults when it dispatches the call itself. That flaw already shipped twice +(`backend`, then `doi`) — see `run()`'s own comment in cli.py. `_ingest_pipeline` +has ordinary Python defaults and is keyword-only, so the same mistake now raises +immediately instead of silently taking a wrong default. +""" + +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from papertrace import cli # noqa: E402 + + +def test_ingest_pipeline_rejects_a_positional_call(): + with pytest.raises(TypeError): + cli._ingest_pipeline(Path("whatever.pdf")) + + +def test_ingest_command_delegates_to_the_pipeline_function(tmp_path, monkeypatch): + seen = {} + + def fake_pipeline(**kw): + seen.update(kw) + + monkeypatch.setattr(cli, "_ingest_pipeline", fake_pipeline) + pdf = tmp_path / "p.pdf" + pdf.write_bytes(b"%PDF-1.4\n") + + cli.ingest(pdf=pdf, out=None, case=None, backend="pymupdf") + + assert seen == {"pdf": pdf, "out": None, "case": None, "backend": "pymupdf"} + + +def test_refs_pipeline_rejects_a_positional_call(): + with pytest.raises(TypeError): + cli._refs_pipeline(Path("whatever.pdf")) + + +def test_refs_command_delegates_to_the_pipeline_function(tmp_path, monkeypatch): + seen = {} + + def fake_pipeline(**kw): + seen.update(kw) + + monkeypatch.setattr(cli, "_refs_pipeline", fake_pipeline) + pdf = tmp_path / "p.pdf" + pdf.write_bytes(b"%PDF-1.4\n") + + cli.refs(manuscript=pdf, case=None, provided=None, email=None, + parse_only=False, backend="auto", doi=None, supplement=None) + + assert seen == { + "manuscript": pdf, "case": None, "provided": None, "email": None, + "parse_only": False, "backend": "auto", "doi": None, "supplement": None, + } + + +def test_report_pipeline_rejects_a_positional_call(): + with pytest.raises(TypeError): + cli._report_pipeline(Path("some-case")) + + +def test_report_command_delegates_to_the_pipeline_function(monkeypatch, tmp_path): + seen = {} + + def fake_pipeline(**kw): + seen.update(kw) + + monkeypatch.setattr(cli, "_report_pipeline", fake_pipeline) + + cli.report(case=tmp_path, png=False, formats=["md"]) + + assert seen == {"case": tmp_path, "png": False, "formats": ["md"]} + + +def test_run_hands_report_its_formats_rather_than_an_option_info(monkeypatch, tmp_path): + """`run()` calls the pipeline function, not the Typer command. Were it still + calling `report(...)` and omitting the new parameter, `formats` would arrive + as an `OptionInfo` and every run would render whatever that truthy sentinel + happened to mean — the failure this split exists to make impossible.""" + seen = {} + monkeypatch.setattr(cli, "_report_pipeline", lambda **kw: seen.update(kw)) + for name in ("_ingest_pipeline", "_refs_pipeline", "_check_pipeline", + "scout", "highlight"): + monkeypatch.setattr(cli, name, lambda **kw: None) + monkeypatch.setattr(cli, "_resolve_case", lambda case, manuscript: tmp_path) + monkeypatch.setattr(cli, "_guard_case", lambda *a, **k: None) + monkeypatch.setattr(cli, "_open_case", lambda *a, **k: None) + monkeypatch.setattr(cli, "_detected_doi", lambda m: None) + # not what is under test, and it must not be read from the machine: `_email` + # falls back to a SAVED CONFIG in the developer's home, so this test passed + # locally and failed on every CI python at `raise typer.Exit(2)` + monkeypatch.setattr(cli, "_email", lambda v: "t@example.org") + pdf = tmp_path / "p.pdf" + pdf.write_bytes(b"%PDF-1.4\n") + + cli.run(manuscript=pdf, case=tmp_path, provided=None, email=None, model=None, + png=False, backend="pymupdf", with_scout=False, doi=None, formats=["md"]) + + assert seen["formats"] == ["md"], "run() must pass formats through explicitly" + + +def test_check_pipeline_rejects_a_positional_call(): + with pytest.raises(TypeError): + cli._check_pipeline(Path("some-case")) + + +def test_check_command_delegates_to_the_pipeline_function(monkeypatch, tmp_path): + seen = {} + + def fake_pipeline(**kw): + seen.update(kw) + + monkeypatch.setattr(cli, "_check_pipeline", fake_pipeline) + + cli.check(case=tmp_path, model=None, backend="pymupdf") + + assert seen == {"case": tmp_path, "model": None, "backend": "pymupdf"} + + +def test_run_hands_check_its_backend_rather_than_an_option_info(monkeypatch, tmp_path): + """`check` gains `--backend` because the cited sources are now read with the + same backend as the paper. `run()` must pass it, or the sources would be + ingested against an `OptionInfo` — and `ingest_pdf` refuses an unknown + backend loudly, so a whole audit would fail at the judging step.""" + seen = {} + monkeypatch.setattr(cli, "_check_pipeline", lambda **kw: seen.update(kw)) + for name in ("_ingest_pipeline", "_refs_pipeline", "_report_pipeline", "scout", "highlight"): + monkeypatch.setattr(cli, name, lambda **kw: None) + monkeypatch.setattr(cli, "_resolve_case", lambda case, manuscript: tmp_path) + monkeypatch.setattr(cli, "_guard_case", lambda *a, **k: None) + monkeypatch.setattr(cli, "_open_case", lambda *a, **k: None) + monkeypatch.setattr(cli, "_detected_doi", lambda m: None) + # not what is under test, and it must not be read from the machine: `_email` + # falls back to a SAVED CONFIG in the developer's home, so this test passed + # locally and failed on every CI python at `raise typer.Exit(2)` + monkeypatch.setattr(cli, "_email", lambda v: "t@example.org") + pdf = tmp_path / "p.pdf" + pdf.write_bytes(b"%PDF-1.4\n") + + cli.run(manuscript=pdf, case=tmp_path, provided=None, email=None, model=None, + png=False, backend="docling", with_scout=False, doi=None, formats=["md"]) + + assert seen["backend"] == "docling", "run() must forward the backend to check" + + +def test_check_claims_will_not_default_its_backend(): + """`check_claims` reads every cited source, so its backend decides whether + a table in a source is readable at all. There is no honest default: `auto` + silently pulls docling into an offline test run, and `pymupdf` silently + downgrades a caller who asked for layout. So it is required, like `_clip`'s + truncation accumulator in the same module and for the same reason — no + future call site can omit it and quietly get the wrong one.""" + import inspect + + from papertrace.check import check_claims + + p = inspect.signature(check_claims).parameters["backend"] + assert p.default is inspect.Parameter.empty, "backend must not have a default" + assert p.kind is inspect.Parameter.KEYWORD_ONLY diff --git a/tests/test_provided_identity.py b/tests/test_provided_identity.py new file mode 100644 index 0000000..68e3091 --- /dev/null +++ b/tests/test_provided_identity.py @@ -0,0 +1,634 @@ +"""A provided PDF is identified by what is in it, not by what it is called. + +Filename matching stays the first pass — it carries the user's own assertion +that this file is that reference. This module is about the gap it leaves: a +publisher download is called `s41467-023-39631-x.pdf` and matches nothing, so +before this the audit ran, looked entirely normal, and used none of it. + +Offline like the rest of the suite. PDFs are built in-test and never committed. +""" + +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from papertrace.models import RefEntry # noqa: E402 + +PYRROS = ("Pyrros A, Borstelmann SM, et al (2023) Opportunistic detection of type 2 " + "diabetes using deep learning from frontal chest radiographs. " + "Nat Commun 14:4039. doi:10.1038/s41467-023-39631-x") +SUDLOW = ("Sudlow C, Gallacher J, Allen N, et al (2015) UK Biobank: An Open Access " + "Resource for Identifying the Causes of a Wide Range of Complex Diseases " + "of Middle and Old Age. PLoS Med 12:e1001779") +CORRIGENDUM = ("Sudlow C, et al (2015) Correction: UK Biobank: An Open Access Resource " + "for Identifying the Causes of a Wide Range of Complex Diseases of " + "Middle and Old Age. PLoS Med 12:e1001800") + + +def _pdf(path: Path, *, title: str = "", body: str = "", doi: str | None = None) -> Path: + """A one-page PDF with an optional metadata title — how real papers arrive.""" + import pymupdf + + doc = pymupdf.open() + page = doc.new_page() + y = 80 + for line in ([f"https://doi.org/{doi}"] if doi else []) + body.split("\n"): + page.insert_text((55, y), line, fontsize=10) + y += 15 + if title: + doc.set_metadata({"title": title}) + path.parent.mkdir(parents=True, exist_ok=True) + doc.save(path) + doc.close() + return path + + +def _entries() -> list[RefEntry]: + return [ + RefEntry(num="1", raw=PYRROS, slug="pyrros-2023", doi="10.1038/s41467-023-39631-x"), + RefEntry(num="2", raw=SUDLOW, slug="sudlow-2015", doi="10.1371/journal.pmed.1001779"), + ] + + +def _identify(entries, folder, claimed=frozenset()): + from papertrace.refs import identify_by_content + + return identify_by_content(entries, folder, set(claimed)) + + +# --- the two signals ------------------------------------------------------- + + +def test_a_publisher_named_download_is_identified_by_its_doi(tmp_path): + """The motivating case: nothing in `s41467-023-39631-x.pdf` names Pyrros.""" + _pdf(tmp_path / "src" / "s41467-023-39631-x.pdf", + title="Opportunistic detection of type 2 diabetes using deep learning " + "from frontal chest radiographs", + doi="10.1038/s41467-023-39631-x", body="Article") + assigned, unclaimed = _identify(_entries(), tmp_path / "src") + + assert {p.name: (v.entry.num, v.kind, v.signal) for p, v in assigned.items()} == { + "s41467-023-39631-x.pdf": ("1", "article", "DOI") + } + assert unclaimed == [] + + +def test_a_paper_whose_reference_printed_no_doi_is_identified_by_its_title(tmp_path): + """Bibliographies often omit the DOI, so title has to carry the rest.""" + entries = _entries() + entries[1].doi = None # the reference printed no DOI + f = _pdf(tmp_path / "src" / "journal.pmed.1001779.pdf", + title="UK Biobank: An Open Access Resource for Identifying the Causes of " + "a Wide Range of Complex Diseases of Middle and Old Age", + body="HEALTH IN ACTION") + assigned, unclaimed = _identify(entries, tmp_path / "src") + + assert [(v.entry.num, v.kind, v.signal) for v in assigned.values()] == [ + ("2", "article", "title") + ] + assert f in assigned + + +def test_a_pdf_with_no_metadata_title_falls_back_to_its_first_page(tmp_path): + """Roughly one paper in seven carries no usable metadata title.""" + entries = _entries() + entries[1].doi = None + _pdf(tmp_path / "src" / "download.pdf", + body="UK Biobank: An Open Access Resource for Identifying the Causes of a\n" + "Wide Range of Complex Diseases of Middle and Old Age") + assigned, _ = _identify(entries, tmp_path / "src") + assert [(v.entry.num, v.kind) for v in assigned.values()] == [("2", "article")] + + +# --- refusing to guess ----------------------------------------------------- + + +def test_a_title_matching_two_references_is_refused_not_ranked(tmp_path): + """A corrigendum shares nearly every distinctive word with its original. + Picking the better score here judges a claim against the wrong paper, and + nothing downstream could ever notice.""" + entries = _entries() + [RefEntry(num="9", raw=CORRIGENDUM, slug="sudlow-2015-r9")] + for e in entries: + e.doi = None + f = _pdf(tmp_path / "src" / "download.pdf", + title="UK Biobank: An Open Access Resource for Identifying the Causes of " + "a Wide Range of Complex Diseases of Middle and Old Age") + assigned, unclaimed = _identify(entries, tmp_path / "src") + + assert assigned == {} + assert len(unclaimed) == 1 + path, why = unclaimed[0] + assert path == f + assert "[2]" in why and "[9]" in why, why + + +def test_a_pdf_that_matches_nothing_is_reported_not_dropped(tmp_path): + """Before this, an unmatched article PDF was silently ignored while a + supplement-named one was reported — the user learned nothing either way.""" + f = _pdf(tmp_path / "src" / "some-other-paper.pdf", + title="Cardiac magnetic resonance in hypertrophic cardiomyopathy") + assigned, unclaimed = _identify(_entries(), tmp_path / "src") + + assert assigned == {} + assert [p for p, _ in unclaimed] == [f] + assert "could not tell" in unclaimed[0][1].lower() + + +def test_too_few_distinctive_words_is_not_an_accept(tmp_path): + """`titles_match` returns None for a title with under four distinctive + words. A filename match may be accepted as unverifiable because the user + named it; a content match has no such assertion behind it.""" + entries = [RefEntry(num="1", raw="Smith J (2020) Brief note. BMJ 1:1", slug="smith-2020")] + _pdf(tmp_path / "src" / "download.pdf", title="Brief note") + assigned, unclaimed = _identify(entries, tmp_path / "src") + assert assigned == {} + assert len(unclaimed) == 1 + + +# --- supplements ----------------------------------------------------------- + + +def test_a_publisher_named_supplement_is_recognised_from_its_own_text(tmp_path): + """`41467_2023_39631_MOESM1_ESM.pdf` carries no filename marker at all — + `\\besm\\b` cannot match after an underscore — but its first page says what + it is.""" + f = _pdf(tmp_path / "src" / "41467_2023_39631_MOESM1_ESM.pdf", + title="Supplementary Information for Opportunistic detection of type 2 " + "diabetes using deep learning from frontal chest radiographs") + assigned, unclaimed = _identify(_entries(), tmp_path / "src") + + assert [(v.entry.num, v.kind) for v in assigned.values()] == [("1", "supplement")] + assert f in assigned + + +def test_the_article_and_its_supplement_both_land_on_the_same_reference(tmp_path): + """Both titles match [1]; the supplementary marker is what separates them.""" + art = _pdf(tmp_path / "src" / "s41467-023-39631-x.pdf", + title="Opportunistic detection of type 2 diabetes using deep learning " + "from frontal chest radiographs") + sup = _pdf(tmp_path / "src" / "mmc1.pdf", + title="Supplementary Information for Opportunistic detection of type 2 " + "diabetes using deep learning from frontal chest radiographs") + entries = _entries() + entries[0].doi = None + assigned, unclaimed = _identify(entries, tmp_path / "src") + + assert assigned[art].kind == "article" + assert assigned[sup].kind == "supplement" + assert assigned[art].entry.num == assigned[sup].entry.num == "1" + assert unclaimed == [] + + +# --- filename matching keeps priority -------------------------------------- + + +def test_a_file_already_claimed_by_filename_is_left_alone(tmp_path): + """Content inference fills gaps; it never re-decides what the user named.""" + named = _pdf(tmp_path / "src" / "pyrros-2023.pdf", title="Something else entirely") + assigned, unclaimed = _identify(_entries(), tmp_path / "src", claimed={named}) + assert assigned == {} + assert unclaimed == [] + + +def test_no_folder_is_not_an_error(tmp_path): + assert _identify(_entries(), None) == ({}, []) + assert _identify(_entries(), tmp_path / "nope") == ({}, []) + + +# --- nothing in the folder goes unremarked --------------------------------- + + +def test_an_unused_article_pdf_is_reported_like_an_orphan_supplement(tmp_path): + """The asymmetry this removes: a supplement-named orphan was reported, an + unmatched article PDF was ignored without a word, and the user could not + tell the difference between "used" and "silently skipped".""" + from papertrace.models import Supplement + from papertrace.refs import unused_provided + + d = tmp_path / "src" + used = _pdf(d / "pyrros-2023.pdf", title="Opportunistic detection") + used_sup = _pdf(d / "pyrros-2023-supplement.pdf", title="Supplementary Information") + stray = _pdf(d / "some-other-paper.pdf", title="Cardiac magnetic resonance") + + entries = _entries() + entries[0].pdf_path = str(used) + entries[0].supplements = [Supplement("pyrros-2023-supplement", str(used_sup))] + + out = dict(unused_provided(entries, d)) + assert list(out) == [stray], out + assert "could not tell" in out[stray].lower() + + +def test_a_reason_from_the_inference_pass_is_carried_through(tmp_path): + """`identify_by_content` already knows *why* it refused a file — an + ambiguous title is a different problem from an unrecognisable one, and the + fixes differ.""" + from papertrace.refs import unused_provided + + d = tmp_path / "src" + f = _pdf(d / "download.pdf", title="UK Biobank: An Open Access Resource") + out = dict(unused_provided(_entries(), d, reasons={f: "its title matches [2] and [9]"})) + assert out[f] == "its title matches [2] and [9]" + + +def test_a_supplement_with_no_article_keeps_its_own_reason(tmp_path): + """Carried over from 0.6.0: this reason names the fix, which is to supply + the article, and it must not be flattened into the generic one.""" + from papertrace.refs import unused_provided + + d = tmp_path / "src" + orphan = _pdf(d / "littlejohns-2020-appendix.pdf", title="Appendix") + entries = [RefEntry(num="3", raw="Littlejohns TJ (2020) UK Biobank imaging", + slug="littlejohns-2020", status="paywalled")] + out = dict(unused_provided(entries, d)) + assert "[3]" in out[orphan] and "not available" in out[orphan] + + +# --- wired into resolution ------------------------------------------------- + + +def _no_network(monkeypatch): + """The online chain must not be reached for a file we already identified.""" + from papertrace import refs as refs_mod + + def _boom(*a, **k): + raise AssertionError("the online chain was reached for an identified file") + + monkeypatch.setattr(refs_mod, "_resolve_by_retrieval", _boom) + + +def test_resolve_all_uses_a_content_identified_file(tmp_path, monkeypatch): + from papertrace.refs import resolve_all + + _no_network(monkeypatch) + d = tmp_path / "src" + f = _pdf(d / "s41467-023-39631-x.pdf", + title="Opportunistic detection of type 2 diabetes using deep learning " + "from frontal chest radiographs", + doi="10.1038/s41467-023-39631-x") + entries = [_entries()[0]] + resolve_all(entries, tmp_path / "dest", "t@example.org", provided_dir=d) + + e = entries[0] + assert (e.status, e.resolver) == ("provided", "user") + assert e.pdf_path == str(f) + # verified by construction: a positive title or DOI match is what chose it + assert e.title_check == "verified" + + +def test_the_reason_says_which_signal_identified_it(tmp_path, monkeypatch): + """Provenance, not decoration: a DOI is exact and a title is a judgement, + and a reader deciding how much to trust the verdict needs to know which.""" + from papertrace.refs import resolve_all + + _no_network(monkeypatch) + d = tmp_path / "src" + _pdf(d / "by-doi.pdf", title="Opportunistic detection of type 2 diabetes using " + "deep learning from frontal chest radiographs", + doi="10.1038/s41467-023-39631-x") + _pdf(d / "by-title.pdf", + title="UK Biobank: An Open Access Resource for Identifying the Causes of a " + "Wide Range of Complex Diseases of Middle and Old Age") + entries = _entries() + entries[1].doi = None + resolve_all(entries, tmp_path / "dest", "t@example.org", provided_dir=d) + + assert "DOI" in entries[0].reason, entries[0].reason + assert "title" in entries[1].reason, entries[1].reason + assert "by-doi.pdf" in entries[0].reason and "by-title.pdf" in entries[1].reason + + +def test_a_filename_match_still_wins_over_content(tmp_path, monkeypatch): + """The filename is the user's own assertion about this file. Content fills + the gap it leaves; it never overrules it.""" + from papertrace.refs import resolve_all + + _no_network(monkeypatch) + d = tmp_path / "src" + named = _pdf(d / "pyrros-2023.pdf", + title="Opportunistic detection of type 2 diabetes using deep learning " + "from frontal chest radiographs") + _pdf(d / "s41467-023-39631-x.pdf", + title="Opportunistic detection of type 2 diabetes using deep learning " + "from frontal chest radiographs", + doi="10.1038/s41467-023-39631-x") + entries = [_entries()[0]] + resolve_all(entries, tmp_path / "dest", "t@example.org", provided_dir=d) + + assert entries[0].pdf_path == str(named) + assert "sources folder" in entries[0].reason + + +def test_a_second_copy_of_an_already_matched_paper_says_so(tmp_path, monkeypatch): + """Realistic: the user has both the reference-manager export and the + publisher download of the same paper. The spare is not a mystery file and + must not be reported as one.""" + from papertrace.refs import resolve_all, unused_provided + + _no_network(monkeypatch) + d = tmp_path / "src" + _pdf(d / "pyrros-2023.pdf", title="Opportunistic detection of type 2 diabetes " + "using deep learning from frontal chest radiographs") + spare = _pdf(d / "s41467-023-39631-x.pdf", + title="Opportunistic detection of type 2 diabetes using deep learning " + "from frontal chest radiographs", + doi="10.1038/s41467-023-39631-x") + entries = [_entries()[0]] + resolve_all(entries, tmp_path / "dest", "t@example.org", provided_dir=d) + + out = dict(unused_provided(entries, d)) + assert list(out) == [spare] + assert "[1]" in out[spare] and "already" in out[spare], out[spare] + + +# --- supplements get the identity check they lacked ------------------------ + + +def test_a_content_identified_supplement_attaches_and_is_marked_verified(tmp_path, + monkeypatch): + """0.6.0 attached supplements on a filename alone and disclosed that nothing + verified them. A supplement whose own title names its parent HAS been + verified, and the report should stop saying otherwise about it.""" + from papertrace.refs import resolve_all + + _no_network(monkeypatch) + d = tmp_path / "src" + art = _pdf(d / "s41467-023-39631-x.pdf", + title="Opportunistic detection of type 2 diabetes using deep learning " + "from frontal chest radiographs", + doi="10.1038/s41467-023-39631-x") + sup = _pdf(d / "41467_2023_39631_MOESM1_ESM.pdf", + title="Supplementary Information for Opportunistic detection of type 2 " + "diabetes using deep learning from frontal chest radiographs") + entries = [_entries()[0]] + resolve_all(entries, tmp_path / "dest", "t@example.org", provided_dir=d) + + assert entries[0].pdf_path == str(art) + assert [(s.pdf_path, s.verified) for s in entries[0].supplements] == [(str(sup), True)] + + +def test_a_filename_attached_supplement_is_not_marked_verified(tmp_path, monkeypatch): + """Nothing read it; it matched a name. The distinction is the whole point.""" + from papertrace.refs import resolve_all + + _no_network(monkeypatch) + d = tmp_path / "src" + _pdf(d / "pyrros-2023.pdf", title="Opportunistic detection of type 2 diabetes " + "using deep learning from frontal chest radiographs") + _pdf(d / "pyrros-2023-supplement.pdf", title="") # no title to verify against + entries = [_entries()[0]] + resolve_all(entries, tmp_path / "dest", "t@example.org", provided_dir=d) + + assert [s.verified for s in entries[0].supplements] == [False] + + +def test_a_content_identified_supplement_still_needs_its_article(tmp_path, monkeypatch): + """The 0.6.0 rule is unchanged by inference: with no article there is + nothing for the appendix to be part of.""" + from papertrace import refs as refs_mod + from papertrace.refs import resolve_all, unused_provided + + # the article is genuinely absent, so the chain IS reached — and finds + # nothing. That is the situation under test. + def _paywalled(entry, *a, **k): + entry.status, entry.reason = "paywalled", "no open-access copy" + + monkeypatch.setattr(refs_mod, "_resolve_by_retrieval", _paywalled) + d = tmp_path / "src" + sup = _pdf(d / "41467_2023_39631_MOESM1_ESM.pdf", + title="Supplementary Information for Opportunistic detection of type 2 " + "diabetes using deep learning from frontal chest radiographs") + entries = [_entries()[0]] + resolve_all(entries, tmp_path / "dest", "t@example.org", provided_dir=d) + + assert entries[0].supplements == [] + assert [p for p, _ in unused_provided(entries, d)] == [sup] + + +def test_verified_round_trips_and_older_manifests_still_load(tmp_path): + import json + + import jsonschema + + from papertrace.models import RefManifest, Supplement + + def _root(): + for x in [Path(__file__).resolve(), *Path(__file__).resolve().parents]: + if (x / "pyproject.toml").exists(): + return x + raise RuntimeError("no pyproject.toml") + + m = RefManifest(manuscript="p.pdf", entries=[ + RefEntry(num="1", raw=PYRROS, slug="pyrros-2023", status="provided", + pdf_path="/tmp/a.pdf", + supplements=[Supplement("a-suppl", "/tmp/a-suppl.pdf", verified=True), + Supplement("b-suppl", "/tmp/b-suppl.pdf")])]) + path = tmp_path / "refs_manifest.json" + m.to_json(path) + schema = json.loads((_root() / "schemas" / "refs_manifest.schema.json").read_text()) + jsonschema.validate(json.loads(path.read_text()), schema) + + sup = schema["properties"]["entries"]["items"]["properties"]["supplements"] + assert "verified" in sup["items"]["properties"], "the identity flag is undeclared" + assert [s.verified for s in RefManifest.from_json(path).entries[0].supplements] == [True, False] + + payload = json.loads(path.read_text()) + for s in payload["entries"][0]["supplements"]: + del s["verified"] + path.write_text(json.dumps(payload)) + jsonschema.validate(json.loads(path.read_text()), schema) + assert [s.verified for s in RefManifest.from_json(path).entries[0].supplements] == [False, False] + + +# --- the short-surname hole ------------------------------------------------ + + +def _liu() -> RefEntry: + """`liu-2019` yields tokens ['2019']: the len>3 filter drops the surname.""" + return RefEntry(num="7", raw="Liu B, Xu J (2019) A study of something. BMJ 1:1", + slug="liu-2019", status="provided", pdf_path="/tmp/liu-2019.pdf") + + +def test_another_authors_appendix_does_not_attach_on_the_year_alone(tmp_path): + """With a surname under four characters only the year survives, so every + 2019 supplement in the folder matched — and a supplement gets no title + check to catch it. That is a wrong-paper-as-evidence path.""" + from papertrace.refs import attach_supplements + + d = tmp_path / "src" + _pdf(d / "smith-2019-appendix.pdf", title="Appendix") + e = _liu() + attach_supplements(e, d, taken={e.slug}) + assert e.supplements == [] + + +def test_the_correctly_named_appendix_still_attaches(tmp_path): + """The fix must not cost the case it is meant to serve: a file named for + this reference carries the user's assertion and is still honoured.""" + from papertrace.refs import attach_supplements + + d = tmp_path / "src" + good = _pdf(d / "liu-2019-appendix.pdf", title="Appendix") + e = _liu() + attach_supplements(e, d, taken={e.slug}) + assert [s.pdf_path for s in e.supplements] == [str(good)] + + +def test_a_content_verified_supplement_is_unaffected_by_the_rule(tmp_path): + """The rule exists because nothing checked the file. When something did, + it does not apply.""" + from papertrace.refs import Identified, attach_supplements + + d = tmp_path / "src" + sup = _pdf(d / "smith-2019-appendix.pdf", title="Appendix") + e = _liu() + attach_supplements(e, d, taken={e.slug}, + content=[Identified(sup, e, "supplement", "title")]) + assert [(s.pdf_path, s.verified) for s in e.supplements] == [(str(sup), True)] + + +def test_a_normal_surname_is_untouched_by_the_rule(tmp_path): + from papertrace.refs import attach_supplements + + d = tmp_path / "src" + sup = _pdf(d / "littlejohns-2020-appendix.pdf", title="Appendix") + e = RefEntry(num="3", raw="Littlejohns TJ (2020) UK Biobank imaging", + slug="littlejohns-2020", status="provided", pdf_path="/tmp/x.pdf") + attach_supplements(e, d, taken={e.slug}) + assert [s.pdf_path for s in e.supplements] == [str(sup)] + + +# --- the report tells the two apart ---------------------------------------- + + +def test_check_carries_the_verification_onto_the_judgement(tmp_path, monkeypatch): + """The templates get `results` alone and run_disclosures takes the manifest + optionally, so a reader holding only results.json must still be able to see + which supplements were checked.""" + import json as _json + + from papertrace import check as check_mod + from papertrace.models import Block, ClaimResult, RefManifest, SourceMap, Supplement + + for slug in ("pyrros-2023", "sup-verified", "sup-named"): + d = tmp_path / "ingest" / slug + d.mkdir(parents=True) + (d / "annotated.md").write_text(f"<!-- block_0001, page 1 -->\nText of {slug}.\n") + SourceMap(doc=f"{slug}.pdf", pages=1, blocks=[ + Block("block_0001", "text", 1, (0.0, 0.0, 9.0, 9.0), [], f"Text of {slug}.") + ]).to_json(d / "source_map.json") + + manifest = RefManifest(manuscript="m.pdf", entries=[ + RefEntry(num="1", raw=PYRROS, status="retrieved", slug="pyrros-2023", + pdf_path="pyrros-2023.pdf", supplements=[ + Supplement("sup-verified", "a.pdf", verified=True), + Supplement("sup-named", "b.pdf")])]) + claim = ClaimResult(id=1, claim="c", location="M", refs=["1"]) + monkeypatch.setattr(check_mod, "_ask", lambda p, m=None: _json.dumps( + [{"id": 1, "verdict": "not_addressed", "note": "n"}])) + check_mod.check_claims([claim], manifest, tmp_path, backend="pymupdf") + + assert {j.source_slug: j.verified for j in claim.judgements} == { + "pyrros-2023": False, "sup-verified": True, "sup-named": False, + } + + +def _supplement_results(verified_flags): + from papertrace.models import ClaimResult, RunResults, SourceJudgement + + js = [SourceJudgement("pyrros-2023", "1", kind="article", verdict="supported", + source_page=1, source_block="block_0001")] + js += [SourceJudgement(f"sup-{i}", "1", kind="supplement", verdict="not_addressed", + verified=v) for i, v in enumerate(verified_flags)] + c = ClaimResult(id=1, claim="c", location="M", refs=["1"], judgements=js) + c.apply_headline() + return RunResults(manuscript="m.pdf", claims=[c]) + + +def test_the_disclosure_splits_checked_from_unchecked(tmp_path): + from papertrace.disclosures import run_disclosures + from papertrace.report import write_reports + + results = _supplement_results([True, True, False]) + d = next(x for x in run_disclosures(results) if x.key == "supplement_identity") + assert "2" in d.text and "1" in d.text + assert d.level == "warn" # one of them is still a guess + + write_reports(results, None, tmp_path, png=False) + for name in ("report.md", "report_editor.html", "report_terminal.html"): + assert d.token in (tmp_path / name).read_text(), name + + +def test_all_verified_stops_the_report_claiming_nothing_was_checked(tmp_path): + """0.6.0 said supplements carry no identity check. That was true of every + supplement then and is now true only of some — saying it of a file whose + own title named its parent is simply false.""" + from papertrace.disclosures import run_disclosures + + d = next(x for x in run_disclosures(_supplement_results([True, True])) + if x.key == "supplement_identity") + assert d.level == "info" + assert "no identity check" not in d.text.lower() + + +def test_the_results_schema_declares_the_supplement_verification(tmp_path): + import json + + import jsonschema + + def _root(): + for x in [Path(__file__).resolve(), *Path(__file__).resolve().parents]: + if (x / "pyproject.toml").exists(): + return x + raise RuntimeError("no pyproject.toml") + + from papertrace.models import RunResults + + results = _supplement_results([True, False]) + path = tmp_path / "results.json" + results.to_json(path) + schema = json.loads((_root() / "schemas" / "results.schema.json").read_text()) + jsonschema.validate(json.loads(path.read_text()), schema) + + j = schema["properties"]["claims"]["items"]["properties"]["judgements"]["items"] + assert "verified" in j["properties"], "the supplement verification is undeclared" + + payload = json.loads(path.read_text()) + for x in payload["claims"][0]["judgements"]: + del x["verified"] + path.write_text(json.dumps(payload)) + assert [x.verified for x in RunResults.from_json(path).claims[0].judgements] == [ + False, False, False + ] + + +def test_a_doi_match_beats_a_title_match_for_the_same_reference(tmp_path, monkeypatch): + """Two files can honestly identify as the same reference — a duplicate copy, + or a full text beside a truncated one. Which is used was decided by + alphabetical order, so a DOI match could lose to a title match, and the + audit depended on what the files happened to be called. That is the + filesystem-order defect `_provided_candidates` already fixed once. + + A DOI is exact identity; a title is a token-overlap judgement. The stronger + signal wins, and the rest is name order so the answer is the same on every + machine.""" + from papertrace.refs import resolve_all, unused_provided + + _no_network(monkeypatch) + d = tmp_path / "src" + title_only = _pdf(d / "aaa-first-alphabetically.pdf", + title="Opportunistic detection of type 2 diabetes using deep " + "learning from frontal chest radiographs") + by_doi = _pdf(d / "zzz-last-alphabetically.pdf", + title="Opportunistic detection of type 2 diabetes using deep " + "learning from frontal chest radiographs", + doi="10.1038/s41467-023-39631-x") + entries = [_entries()[0]] + resolve_all(entries, tmp_path / "dest", "t@example.org", provided_dir=d) + + assert entries[0].pdf_path == str(by_doi), "the exact signal must win" + assert "own DOI" in entries[0].reason + spare = dict(unused_provided(entries, d)) + assert list(spare) == [title_only] + assert "already has a file" in spare[title_only] diff --git a/tests/test_reference_list.py b/tests/test_reference_list.py index d2ae453..78acbd7 100644 --- a/tests/test_reference_list.py +++ b/tests/test_reference_list.py @@ -129,3 +129,57 @@ def test_a_differently_typed_run_does_not_resume_the_list(): text, resumed = references_span(m) assert "supplementary consideration" not in text, text assert resumed is False + + +# --- back matter is not a continued bibliography ---------------------------- +# +# The first real audit of an Elsevier paper reported 46 references on a paper +# citing 43. Blocks 120-162 were the references; block 163 was a `TABLE TITLES` +# heading; blocks 164-166 were three `list` blocks holding the paper's own table +# captions. The resume scan runs to the END of the document and accepted them, +# because the only test it applied was the block *type*. Refs 44-46 were then +# title-searched against Crossref, which answered with table-component DOIs from +# unrelated papers, and the report published three works that do not exist. + +_TABLE_TITLES = _map( + ("sectionheader", 22, "REFERENCES"), + ("list", 22, "- G.C. Feuerriegel, R.P. Marcus, S. Sommer, Rotator cuff. Eur Radiol 2023."), + ("list", 22, "- D.A. Lansdown, S. Lee, C. Sam, A prospective quantitative study. 2017."), + ("list", 22, "- W.T. Dixon, Simple proton spectroscopic imaging, Radiology 153 (1984) 189-194."), + ("sectionheader", 26, "TABLE TITLES"), + ("list", 26, "- Table 1. Dataset characteristics"), + ("list", 26, "- Table 2. Accuracy and reliability of automated thresholding models"), + ("list", 26, "- Table 3. Diagnostic accuracy for clinical cutoffs of Goutallier"), +) + + +def test_table_captions_after_the_references_are_not_references(): + """Three list blocks under `TABLE TITLES` share the reference list's block + type and nothing else. None carries a year, a DOI or an arXiv id.""" + text, resumed = references_span(_TABLE_TITLES) + + assert "Table 1." not in text, "the paper's own table captions became references" + assert "Table 2." not in text + assert "Table 3." not in text + assert resumed is False, "nothing was resumed, so nothing should be reported as resumed" + assert text.count("- ") == 3, text + + +def test_the_real_references_survive_the_shape_test(): + """The other half of the same assertion: rejecting back matter must not + reject the bibliography it follows.""" + text, _ = references_span(_TABLE_TITLES) + for surname in ("Feuerriegel", "Lansdown", "Dixon"): + assert surname in text, f"{surname} was lost to the shape test" + + +def test_a_genuine_continuation_still_resumes_when_one_entry_lacks_a_year(): + """The shape test is applied to the RUN, not to each entry. `_SPLIT`'s + resumed run holds two dated references and one URL-only entry; requiring + every entry to be reference-shaped would undo the 0.4.0 fix over the one + entry that is a bare link.""" + text, resumed = references_span(_SPLIT) + + assert resumed is True + assert "Dean" in text and "Kitamura" in text + assert "Assess-AI" in text, "the year-less entry in a real run was dropped" diff --git a/tests/test_reference_reconciliation.py b/tests/test_reference_reconciliation.py new file mode 100644 index 0000000..3071291 --- /dev/null +++ b/tests/test_reference_reconciliation.py @@ -0,0 +1,1207 @@ +"""The reference list is reconciled against what the manuscript actually cites. + +`parse_references` was the one stage with no honest-degradation state: it always +returned a confident list, and nothing ever checked the count. A live audit +misnumbered 27 of 41 references — docling stripped the `[N]` numerals, a running +header split reference [14] across a page break, and every label from [15] on +shifted by one. The label is the join key, so claims citing >= [15] were judged +against the wrong papers. + +Offline: Crossref is faked with `httpx.MockTransport`, no model calls. +""" + +import json +import sys +from pathlib import Path + +import httpx +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from papertrace.models import RefEntry, citation_labels, is_references_heading # noqa: E402 +from papertrace.refs import _entry # noqa: E402 + +# --- the label rule, now shared --------------------------------------------- + + +def test_citation_labels_reads_singles_lists_and_ranges(): + """The rule `coverage_audit` has always used, now callable by `refs` too.""" + assert citation_labels("as shown [3] and [7,8] and [11-13]") == { + "3", "7", "8", "11", "12", "13" + } + + +def test_citation_labels_is_the_same_rule_the_coverage_audit_uses(): + """Moved, not reimplemented. A second copy of this rule is exactly the + defect the codebase already shipped for `is_references_heading`.""" + from papertrace.check import citation_labels_in_text + + body = "one [1] two [2,3] three [5-7]\n\n## References\n[9] Never cited in body.\n" + assert citation_labels_in_text(body) == {"1", "2", "3", "5", "6", "7"} + # the wrapper's only job is cutting the reference list off first + assert citation_labels(body) == {"1", "2", "3", "5", "6", "7", "9"} + + +def test_table_ci_brackets_are_not_read_as_body_citations(): + """A table's own numbers are not citations. A radiology results table + reports 95% CIs as `[51, 77]` — the same bracket-and-comma syntax as a + citation group `[7,8]` — and a live audit read a table's `[100, 100]` and + `[54, 100]` CI columns as citations to references #100 and #54, inflating + the highest cited label from 3 (the real count) to 100 and breaking + `reconcile`'s numbering check on a paper with a perfectly ordinary + bibliography.""" + from papertrace.cli import _body_citation_labels + from papertrace.models import Block, SourceMap + + smap = SourceMap(doc="m.pdf", pages=1, converter="docling 2.118.1", blocks=[ + Block("block_0001", "text", 1, (0.0, 0.0, 1.0, 1.0), [], + "Uptake was low [1] in prior work [2,3]."), + Block("block_0002", "table", 1, (0.0, 0.0, 1.0, 1.0), [], + "| Sensitivity (%) | 65 [51, 77] |\n|---|---|\n"), + ]) + + assert _body_citation_labels(smap, citation_labels, is_references_heading) == { + "1", "2", "3" + } + + +# --- the Crossref leg -------------------------------------------------------- + + +def _deposit(refs: list[dict], total: int | None = None) -> httpx.MockTransport: + """A Crossref `/works/{doi}` response carrying a deposited reference list.""" + body = { + "message": { + "publisher": "Fixture Publishing", + "references-count": total if total is not None else len(refs), + "reference": refs, + } + } + + def handler(request: httpx.Request) -> httpx.Response: + assert "api.crossref.org/works/" in str(request.url), request.url + return httpx.Response(200, json=body) + + return httpx.MockTransport(handler) + + +def _structured(n: int) -> list[dict]: + return [ + { + "key": f"10.1/x_b{i * 5:04d}", # publisher-specific: never a numbering signal + "DOI": f"10.1234/fixture.{i}", + "author": f"Author{i} A", + "article-title": f"A paper about topic {i}", + "journal-title": "J Fixture", + "volume": "12", + "first-page": str(100 + i), + "year": "2020", + } + for i in range(1, n + 1) + ] + + +def test_a_deposited_list_is_numbered_by_array_order_not_by_key(): + """Keys are publisher-specific — `_b0005`, `_bib1`, `3400_CR1`, + `R10-45-20210317`, and two schemes inside one Elsevier deposit. Array order + is the only portable signal, and reading a number out of a key would + renumber every reference of every publisher that does not use `_bibN`.""" + from papertrace.refs import crossref_reference_list + + with httpx.Client(transport=_deposit(_structured(3))) as client: + entries = crossref_reference_list(client, "10.1/paper", "t@example.org") + + assert [e.num for e in entries] == ["1", "2", "3"] + assert entries[0].doi == "10.1234/fixture.1" + assert "A paper about topic 1" in entries[0].raw + + +def test_an_unstructured_deposit_keeps_the_citation_string_verbatim(): + """`raw` is what `_title_check` and `_slug` consume, so it has to read like + a printed reference whichever shape the deposit arrives in.""" + from papertrace.refs import crossref_reference_list + + refs = [{"key": "ref1", "unstructured": "Smith J. A paper. J Fixture 2019;12:100-9."}] + with httpx.Client(transport=_deposit(refs)) as client: + entries = crossref_reference_list(client, "10.1/paper", "t@example.org") + + assert entries[0].raw == "Smith J. A paper. J Fixture 2019;12:100-9." + assert entries[0].year == "2019" + assert entries[0].slug == "smith-2019" + + +def test_a_component_doi_in_a_deposit_is_refused_like_any_other(): + """A publisher can deposit a reference to its own table. A part of a work is + never the work a reference cites, wherever the DOI came from.""" + from papertrace.refs import crossref_reference_list + + refs = [{"key": "ref1", "DOI": "10.7717/peerj.7892/table-1", + "article-title": "Table 1. Dataset characteristics", "year": "2019"}] + with httpx.Client(transport=_deposit(refs)) as client: + entries = crossref_reference_list(client, "10.1/paper", "t@example.org") + + assert entries[0].doi is None, "a table-component DOI was accepted from Crossref" + + +def test_no_deposit_is_none_not_an_empty_list(): + """None means "this publisher deposits nothing"; `[]` would read as "this + paper cites nothing", and the reconciler must be able to tell them apart.""" + from papertrace.refs import crossref_reference_list + + def handler(request): + return httpx.Response(200, json={"message": {"publisher": "X", "reference": []}}) + + with httpx.Client(transport=httpx.MockTransport(handler)) as client: + assert crossref_reference_list(client, "10.1/paper", "t@example.org") is None + + +@pytest.mark.parametrize("status", [404, 500]) +def test_crossref_unreachable_is_none_and_never_raises(status): + """A missing DOI and a Crossref outage are both "no candidate from this + leg" — neither may take the audit down with it.""" + from papertrace.refs import crossref_reference_list + + with httpx.Client(transport=httpx.MockTransport(lambda r: httpx.Response(status))) as client: + assert crossref_reference_list(client, "10.1/paper", "t@example.org") is None + + +def test_a_doi_only_deposit_is_kept_not_discarded(): + """Wiley deposits references as a bare DOI and nothing else. An earlier + version rendered those to an empty string, dropped them, and reported + "the publisher deposited 3 of its own declared 52" — blaming Wiley for a + complete 52-reference deposit this tool could not read. They are the *best* + references in a deposit: the DOI is already resolved, so retrieval skips the + title search entirely.""" + from papertrace.refs import crossref_deposit + + refs = [{"key": f"e_1_2_{i}", "doi-asserted-by": "publisher", + "DOI": f"10.1056/NEJMoa19117{i:02d}"} for i in range(1, 6)] + with httpx.Client(transport=_deposit(refs)) as client: + deposit = crossref_deposit(client, "10.1/paper", "t@example.org") + + assert len(deposit.entries) == 5, "DOI-only references were discarded" + assert deposit.unrenderable == 0 + assert deposit.entries[0].doi == "10.1056/NEJMoa1911701" + # named for what is actually known about it, which is the DOI, not an author + assert deposit.entries[0].slug == "nejmoa1911701" + assert len({e.slug for e in deposit.entries}) == 5 + + +def test_a_shortfall_is_reported_as_this_tools_limitation(): + """`references-count` counts the references *deposited*, so it always equals + the array length — verified live on Elsevier, Springer, Wiley, PLOS and + RSNA records. A shortfall can therefore only mean this tool failed to render + an entry, and must be counted and named as such rather than described as the + publisher depositing less than it claimed.""" + from papertrace.refs import crossref_deposit + + refs = [{"key": "r1", "author": "Smith J", "article-title": "One", "year": "2020"}, + {"key": "r2"}] # nothing renderable: no author, no title, no DOI + with httpx.Client(transport=_deposit(refs)) as client: + deposit = crossref_deposit(client, "10.1/paper", "t@example.org") + + assert deposit.deposited == 2 + assert len(deposit.entries) == 1 + assert deposit.unrenderable == 1 + + +def test_a_short_deposit_is_caught_by_the_body_labels_not_by_a_count_field(): + """The real protection against a genuinely short deposit. One publisher + deposited 2 references for a ~40-reference paper; the record reports + `references-count: 2`, so nothing in the payload gives it away. The + manuscript's own labels do.""" + from papertrace.refs import reconcile + + body = {str(n) for n in range(1, 41)} + _entries, rec = reconcile(body, crossref=_parsed([1, 2]), parsed=_parsed(list(range(1, 41)))) + assert rec.source == "parsed" and rec.verified is True + + +# --- the reconciler ---------------------------------------------------------- + + +# Distinct, reference-shaped content for every entry, sharing no word with any +# other. `_same_work` compares *content* — a fixture whose entries all read +# alike cannot tell a right list from a shuffled one, and the one that read +# `f"Parsed reference {n}. 2020."` could not: every entry shared the token +# `reference`, so a divergent entry compared equal and the two divergence tests +# went red the moment slug comparison was replaced. +_SYL = ("mar", "hol", "dahl", "vik", "berg", "rud", "gren", "stad", "lund", "sund") + + +def _ref_text(n: int) -> str: + """Every word of length >= 5 carries the entry's own stem. + + `_title_tokens` keeps words of five letters or more, and `_same_work` + accepts a 0.34 overlap — so two shared boilerplate words ("after", + "Fixture") in a six-word reference were enough to make every entry the same + work as every other. Real reference strings are long, and the ratio means + something there; a fixture has to earn that the hard way. + """ + stem = _SYL[n % 10] + _SYL[(n // 10) % 10] + return (f"{(stem + 'sen').capitalize()} A, {(stem + 'strom').capitalize()} B. " + f"{(stem + 'osis').capitalize()} and {stem}ectomy in {stem}opathy. " + f"{(stem + 'ology').capitalize()} 2020;1:1-9.") + + +def _parsed(nums: list[int]) -> list[RefEntry]: + """Built through `_entry` — the production path, which parses the year. + + Constructing `RefEntry` directly left `year` empty, so `_same_work` fell + through to its token comparison with nothing to contradict it. + """ + return [_entry(str(n), _ref_text(n)) for n in nums] + + +def test_the_real_failure_crossref_matches_the_body_and_the_parse_does_not(): + """The audit that started this: body cites [1]-[41], the PDF parse yields + 43 entries, the deposit yields 41.""" + from papertrace.refs import reconcile + + body = {str(n) for n in range(1, 42)} + entries, rec = reconcile(body, crossref=_parsed(list(range(1, 42))), parsed=_parsed( + list(range(1, 44)))) + + assert rec.source == "crossref" + assert rec.verified is True + assert len(entries) == 41 + assert rec.unverified_from is None + + +def test_ovids_partial_deposit_loses_to_the_parse(): + """Two references deposited, forty in the paper. The body's labels decide, + and they say the deposit is wrong — so the parse is used and the deposit is + not silently blended in.""" + from papertrace.refs import reconcile + + body = {str(n) for n in range(1, 41)} + entries, rec = reconcile(body, crossref=_parsed([1, 2]), parsed=_parsed(list(range(1, 41)))) + + assert rec.source == "parsed" + assert rec.verified is True + assert len(entries) == 40 + + +def test_when_neither_candidate_matches_the_body_nothing_is_verified(): + """Continue, disclose loudly, taint the affected verdicts — decided with the + user. Refusing to run would be the wrong trade: the audit is still useful, + it just must not present a numbering it cannot stand behind.""" + from papertrace.refs import reconcile + + body = {str(n) for n in range(1, 42)} + entries, rec = reconcile(body, crossref=_parsed(list(range(1, 40))), parsed=_parsed( + list(range(1, 44)))) + + assert rec.verified is False + assert rec.source == "parsed", "the parse is the fallback — it is at least the paper's own text" + assert len(entries) == 43 + assert rec.note, "an unverified numbering owes the reader a reason" + + +def test_two_disagreeing_candidates_locate_the_first_divergence(): + """`[1]`-`[14]` agreed in the real failure, and saying so is worth more than + a blanket warning: a reader can still trust the head of the list.""" + from papertrace.refs import reconcile + + crossref = _parsed(list(range(1, 20))) + parsed = _parsed(list(range(1, 20))) + parsed[14] = _entry("15", "Quite B. Another work entirely. Nature 1999;1:1.") + + body = {str(n) for n in range(1, 30)} # matches neither + _entries, rec = reconcile(body, crossref=crossref, parsed=parsed) + + assert rec.verified is False + assert rec.unverified_from == 15 + + +def test_a_single_unchecked_candidate_never_claims_to_know_where_it_went_wrong(): + """With nothing to compare against, the first divergence is unknowable. + Claiming `unverified_from = 40` on a 41-entry list would present 39 entries + as verified on no evidence at all.""" + from papertrace.refs import reconcile + + body = {str(n) for n in range(1, 42)} + _entries, rec = reconcile(body, crossref=None, parsed=_parsed(list(range(1, 44)))) + + assert rec.verified is False + assert rec.unverified_from == 1 + + +def test_a_clean_paper_gains_no_warning_it_has_not_earned(): + """`main.pdf`: 35 references, both readings agree with the body.""" + from papertrace.refs import reconcile + + body = {str(n) for n in range(1, 36)} + entries, rec = reconcile(body, crossref=_parsed(list(range(1, 36))), parsed=_parsed( + list(range(1, 36)))) + + assert rec.verified is True + assert rec.source == "crossref", "both matched — prefer the leg whose DOIs are already resolved" + assert rec.unverified_from is None + assert len(entries) == 35 + + +def test_no_body_labels_means_nothing_to_arbitrate_with(): + """An author-year paper cites nothing this tool can read, so the arbiter + does not exist. That is not the same as "the parse is correct", and it is + not the same as "the parse is wrong" — it is a third fact.""" + from papertrace.refs import reconcile + + entries, rec = reconcile(set(), crossref=None, parsed=_parsed(list(range(1, 20)))) + + assert rec.verified is False + assert rec.source == "parsed" + assert "no bracketed numeric citation" in rec.note.lower() + assert len(entries) == 19 + assert rec.unverified_from == 1, "one unchecked reading is evidence about nothing" + + +def test_a_shuffled_reading_is_not_corroborated(): + """The assertion the fixtures could not make until they carried distinct + content. With every entry reading `Parsed reference N. 2020.`, a *fully + shuffled* list reported `_first_divergence -> None`, so + `test_two_disagreeing_candidates_locate_the_first_divergence` would have + passed on a list where no entry was the paper its label names. This pins the + fixture's discriminating power, not just the reconciler's.""" + from papertrace.refs import reconcile + + crossref = _parsed(list(range(1, 11))) + # the same ten works, in reverse, renumbered [1]..[10] — the shape of the + # bug this whole feature exists to catch, at maximum severity + shuffled = [_entry(str(i), e.raw) for i, e in enumerate(reversed(crossref), 1)] + + _entries, rec = reconcile(set(), crossref=crossref, parsed=shuffled) + + assert rec.verified is False + assert rec.unverified_from == 1, "a shuffled list was reported as corroborated" + assert "agree as far as" not in rec.note + + +def test_two_agreeing_candidates_narrow_the_doubt_even_with_no_arbiter(): + """Measured on a real spread: three of six journals use superscript-numeric + citations, whose markers vanish when the PDF is flattened to text, so there + is no arbiter for half the papers this tool will meet. A PDF parse and a + publisher deposit have no common failure mode, so where they agree is + evidence about those entries — not confirmation, but not nothing either.""" + from papertrace.refs import reconcile + + crossref = _parsed(list(range(1, 31))) + parsed = _parsed(list(range(1, 31))) + parsed[19] = _entry("20", "Quite B. Another work entirely. Nature 1999;1:1.") + + _entries, rec = reconcile(set(), crossref=crossref, parsed=parsed) + + assert rec.verified is False, "corroboration is not confirmation" + assert rec.unverified_from == 20 + assert "agree as far as entry [19]" in rec.note + + +def test_the_three_ways_a_crossref_candidate_can_be_absent_read_differently(): + """No DOI, no deposit, and Crossref unreachable are three different facts. + Collapsing them is the scout's "pass --doi" bug, which told an operator who + had just passed --doi to pass --doi.""" + from papertrace.refs import CROSSREF_NO_DEPOSIT, CROSSREF_NO_DOI, CROSSREF_UNREACHABLE + + notes = {CROSSREF_NO_DOI, CROSSREF_NO_DEPOSIT, CROSSREF_UNREACHABLE} + assert len(notes) == 3, "two of the three absences share wording" + assert all(n.strip() for n in notes) + + +# --- carrying it through the manifest ---------------------------------------- + + +def test_the_new_manifest_fields_round_trip_and_older_manifests_still_load(tmp_path): + """New JSON field ⇒ schema update plus a round-trip test (gate 2), and + absent-safe: a manifest written before these fields has no such keys.""" + import jsonschema + + from papertrace.models import RefManifest + + path = tmp_path / "refs_manifest.json" + RefManifest( + manuscript="m.pdf", + entries=_parsed([1, 2]), + reference_source="crossref", + numbering_verified=False, + numbering_note="the body cites [1]-[41]; this list has 43 entries", + unverified_from=15, + ).to_json(path) + + schema = json.loads( + (Path(__file__).resolve().parent.parent / "schemas" / "refs_manifest.schema.json").read_text() + ) + payload = json.loads(path.read_text()) + jsonschema.validate(payload, schema) + + back = RefManifest.from_json(path) + assert back.reference_source == "crossref" + assert back.numbering_verified is False + assert back.unverified_from == 15 + + for key in ("reference_source", "numbering_verified", "numbering_note", "unverified_from"): + del payload[key] + path.write_text(json.dumps(payload)) + jsonschema.validate(json.loads(path.read_text()), schema) + old = RefManifest.from_json(path) + assert old.reference_source == "parsed" + # an old manifest never checked its numbering, and must not claim it did + assert old.numbering_verified is False + assert old.unverified_from is None + + +# --- the reader has to be told ------------------------------------------------ + + +def _results(): + from papertrace.models import ClaimResult, RunResults + + return RunResults( + manuscript="m.pdf", + converter="pymupdf", + claims=[ClaimResult(id=1, claim="a claim", location="Methods", refs=["20"], + verdict="supported", source_slug="x-2020", source_page=1)], + ) + + +def _unverified_manifest(): + from papertrace.models import RefManifest + + return RefManifest( + manuscript="m.pdf", + entries=_parsed(list(range(1, 44))), + reference_source="parsed", + numbering_verified=False, + numbering_note="the manuscript cites [1]-[41], the parsed list has 43 references", + unverified_from=15, + ) + + +def test_an_unconfirmed_numbering_is_disclosed_in_all_three_formats(tmp_path): + """The whole point of reconciling: the failure has to be visible where the + report is read, not only in `refs_manifest.json`.""" + from papertrace.disclosures import run_disclosures + from papertrace.report import write_reports + + manifest = _unverified_manifest() + results = _results() + write_reports(results, manifest, tmp_path, png=False) + + d = next((x for x in run_disclosures(results, manifest) if x.key == "numbering"), None) + assert d is not None, [x.key for x in run_disclosures(results, manifest)] + for name in ("report.md", "report_editor.html", "report_terminal.html"): + assert d.token in (tmp_path / name).read_text(), f"token missing from {name}" + + +def test_a_confirmed_numbering_adds_no_warning(): + """An ordinary paper must not grow a caveat it has not earned.""" + from papertrace.disclosures import run_disclosures + from papertrace.models import RefManifest + + manifest = RefManifest( + manuscript="m.pdf", entries=_parsed([1, 2]), + reference_source="crossref", numbering_verified=True, + ) + assert not any(d.key == "numbering" for d in run_disclosures(_results(), manifest)) + + +def test_a_claim_citing_a_doubtful_label_is_tainted_in_all_three_formats(tmp_path): + """A run-level banner is not enough. The label is the join key, so the + verdict itself may be about a different paper — and that has to be said + beside the verdict, where someone acting on it will read it.""" + from papertrace.disclosures import claim_disclosures + from papertrace.report import write_reports + + manifest = _unverified_manifest() + results = _results() # the claim cites [20], and doubt starts at [15] + write_reports(results, manifest, tmp_path, png=False) + + d = next((x for x in claim_disclosures(results.claims[0], manifest) + if x.key == "claim_numbering"), None) + assert d is not None + for name in ("report.md", "report_editor.html", "report_terminal.html"): + assert d.token in (tmp_path / name).read_text(), f"token missing from {name}" + + +def test_a_claim_below_the_divergence_keeps_its_verdict_clean(): + """`[1]`-`[14]` agreed in the real failure. Tainting them too would spend + the warning's credibility on claims that are fine.""" + from papertrace.disclosures import claim_disclosures + from papertrace.models import ClaimResult + + claim = ClaimResult(id=2, claim="an early claim", location="Intro", refs=["3"], + verdict="supported", source_slug="y-2019", source_page=1) + fired = claim_disclosures(claim, _unverified_manifest()) + assert not any(d.key == "claim_numbering" for d in fired) + + +def test_claim_disclosures_without_a_manifest_is_unchanged(): + """Every existing caller passes one argument, and must keep working.""" + from papertrace.disclosures import claim_disclosures + from papertrace.models import ClaimResult + + claim = ClaimResult(id=3, claim="a claim", location="Methods", refs=["20"], + verdict="supported", source_slug="x-2020", source_page=1) + assert not any(d.key == "claim_numbering" for d in claim_disclosures(claim)) + + +# --- the CLI seam ------------------------------------------------------------- + + +def test_refs_pipeline_doi_defaults_to_none_not_an_option_info(): + """`refs()` used to double as the plain function `run` and the tests called + directly, so omitting `doi` fell through to Typer's own declared default — + an `OptionInfo`, not `None` — and `doi or detect_doi(...)` took it for a + real DOI, building a Crossref URL out of its repr. `_refs_pipeline` is what + `run` and the tests call now, with an ordinary Python default, so omitting + `doi` here is just `None` — no `_text_opt`-style guard needed. + """ + import inspect + + from papertrace.cli import _refs_pipeline + + assert inspect.signature(_refs_pipeline).parameters["doi"].default is None + + +def _one_page_paper(path: Path): + import pymupdf + + doc = pymupdf.open() + page = doc.new_page() + page.insert_text((72, 100), "A Study", fontsize=16) + page.insert_text((72, 140), "Body text citing [1] and [2] here.", fontsize=11) + page.insert_text((72, 200), "References", fontsize=14) + page.insert_text((72, 230), "[1] Alpha A. First paper. 2020.", fontsize=11) + page.insert_text((72, 250), "[2] Beta B. Second paper. 2021.", fontsize=11) + path.parent.mkdir(parents=True, exist_ok=True) + doc.save(path) + doc.close() + return path + + +def test_refs_records_the_reconciliation_it_performed(tmp_path, monkeypatch): + """End to end through the CLI stage, with both network legs stubbed: the + body cites [1] and [2], the parse finds two references, so the numbering is + confirmed and the manifest says which reading it holds.""" + import papertrace.cli as cli_mod + import papertrace.refs as refs_mod + from papertrace.models import RefManifest + from papertrace.refs import CROSSREF_NO_DOI, CrossrefDeposit + + monkeypatch.setattr(refs_mod, "resolve_all", + lambda entries, dest, email, provided_dir=None, progress=None, taken=None: entries) + seen = {} + + def _no_deposit(client, doi, email): + seen["doi"] = doi + return CrossrefDeposit(absent=CROSSREF_NO_DOI) + + monkeypatch.setattr(refs_mod, "crossref_deposit", _no_deposit) + + pdf = _one_page_paper(tmp_path / "paper.pdf") + cli_mod.refs(manuscript=pdf, case=tmp_path / "case", provided=None, + email="t@example.org", parse_only=False, backend="pymupdf", doi=None, + supplement=None) + + assert seen["doi"] is None, "an unpublished manuscript has no DOI to look up" + manifest = RefManifest.from_json(tmp_path / "case" / "refs_manifest.json") + assert manifest.numbering_verified is True + assert manifest.reference_source == "parsed" + assert manifest.unverified_from is None + # the reader is told the deposit was never seen, and why + assert "no DOI for the manuscript itself" in manifest.numbering_note + + +def _titled_paper(path: Path, title: str): + """Like `_one_page_paper`, but the first block is a real-looking title. + + `paper_title` takes the first substantial block, so a fixture whose opening + line is `A Study` gives the identity check almost nothing to work with — + which is a real condition, tested separately, not the one under test here. + """ + import pymupdf + + doc = pymupdf.open() + page = doc.new_page() + page.insert_text((72, 100), title, fontsize=16) + page.insert_text((72, 140), "Body text citing [1] and [2] here.", fontsize=11) + page.insert_text((72, 200), "References", fontsize=14) + page.insert_text((72, 230), "[1] Alpha A. First paper. 2020.", fontsize=11) + page.insert_text((72, 250), "[2] Beta B. Second paper. 2021.", fontsize=11) + path.parent.mkdir(parents=True, exist_ok=True) + doc.save(path) + doc.close() + return path + + +def _refs_with_deposit(tmp_path, monkeypatch, deposit, title="Image registration improves " + "inter-reader agreement in CT of the pancreas"): + """Run the `refs` stage with a stubbed deposit, and return the manifest.""" + import papertrace.cli as cli_mod + import papertrace.refs as refs_mod + from papertrace.models import RefManifest + + monkeypatch.setattr(refs_mod, "resolve_all", + lambda entries, dest, email, provided_dir=None, progress=None, taken=None: entries) + monkeypatch.setattr(refs_mod, "crossref_deposit", lambda client, doi, email: deposit) + + pdf = _titled_paper(tmp_path / "paper.pdf", title) + cli_mod.refs(manuscript=pdf, case=tmp_path / "case", provided=None, + email="t@example.org", parse_only=False, backend="pymupdf", + doi="10.1234/asserted", supplement=None) + return RefManifest.from_json(tmp_path / "case" / "refs_manifest.json") + + +def test_a_deposit_from_another_paper_is_not_used_to_renumber(tmp_path, monkeypatch): + """`deposit_is_this_paper` existed, was documented and was unit-tested — and + nothing called it. The DOI is scraped off page 1 or typed by hand, and a + companion paper, an erratum or an earlier version can carry exactly as many + references as the body cites, so `_covers` passes and the run prints + "numbering confirmed" over another paper's bibliography. Every other + retrieval route in this module is title-checked; the one route that can + replace the whole list was not.""" + from papertrace.refs import CrossrefDeposit + + manifest = _refs_with_deposit( + tmp_path, monkeypatch, + CrossrefDeposit(entries=_parsed([1, 2]), deposited=2, + publisher="Fixture Publishing", + title="Maternal urinary fluoride and child neurobehavior at age three"), + ) + + assert manifest.reference_source == "parsed", "another paper's list was adopted" + assert [e.num for e in manifest.entries] == ["1", "2"] + assert "Alpha A" in manifest.entries[0].raw, manifest.entries[0].raw + assert "10.1234/asserted" in manifest.numbering_note + assert "not this paper" in manifest.numbering_note + + +def test_a_deposit_confirmed_as_this_paper_is_still_used(tmp_path, monkeypatch): + """The gate must not cost the feature its point: a matching record is used, + and the note says the identity was checked rather than assumed.""" + from papertrace.refs import CrossrefDeposit + + manifest = _refs_with_deposit( + tmp_path, monkeypatch, + CrossrefDeposit(entries=_parsed([1, 2]), deposited=2, + publisher="Fixture Publishing", + title="Image registration improves inter-reader agreement in CT " + "of the pancreas"), + ) + + assert manifest.reference_source == "crossref" + assert manifest.numbering_verified is True + # "confirmed", not merely "not refused" — and distinct from the unverifiable + # wording, which contains the same phrase negated + assert "was confirmed as this paper by title" in manifest.numbering_note + assert "could not be confirmed" not in manifest.numbering_note + + +def test_an_identity_that_cannot_be_checked_is_disclosed_not_assumed(tmp_path, monkeypatch): + """Tri-state, like every other title check here. A record with no title, or + a paper whose title the ingest could not find, is an unknown — and an + unknown is not a match and not a mismatch. Discarding it would throw away + good deposits for a thin first page; using it silently would stamp + "confirmed" on an identity nobody established.""" + from papertrace.refs import CrossrefDeposit + + manifest = _refs_with_deposit( + tmp_path, monkeypatch, + CrossrefDeposit(entries=_parsed([1, 2]), deposited=2, + publisher="Fixture Publishing", title=""), + ) + + assert manifest.reference_source == "crossref" + assert manifest.numbering_verified is True + assert "could not be confirmed as this paper" in manifest.numbering_note + assert "too little title to compare" in manifest.numbering_note + + +def test_parse_only_reaches_no_network_leg_at_all(tmp_path, monkeypatch): + """`--parse-only` is documented as "List references, no network". The + reconciler's Crossref leg must not quietly break that promise.""" + import papertrace.cli as cli_mod + import papertrace.refs as refs_mod + + def _boom(*a, **k): + raise AssertionError("--parse-only reached the network") + + monkeypatch.setattr(refs_mod, "crossref_deposit", _boom) + monkeypatch.setattr(refs_mod, "resolve_all", _boom) + + pdf = _one_page_paper(tmp_path / "paper.pdf") + cli_mod.refs(manuscript=pdf, case=tmp_path / "case", provided=None, + email="t@example.org", parse_only=True, backend="pymupdf", doi=None, + supplement=None) + + +# --- one slug, one source file ------------------------------------------------ +# +# Found by a live run on a JAMA editorial, and far worse than the TypeError that +# revealed it. `_slug` takes the first token of `raw`, strips non-letters, and +# falls back to the literal "ref" when nothing is left. `_parse_bulleted` leaves +# the printed numeral at the front of `raw`, so every entry slugged `ref-<year>` +# — 23 of 28 references on that paper — and `resolve_all` writes each download +# to `<slug>.pdf`. Eleven retrieved sources wrote ONE file, each overwriting the +# last, and every claim citing them would be judged against whichever paper +# happened to download last. That is the wrong-paper failure this whole project +# exists to prevent, arriving silently. + + +def test_a_bulleted_reference_does_not_keep_its_printed_numeral(): + """The numeral is the list marker the converter failed to strip, not part of + the reference. Leaving it in makes the first token non-alphabetic, which is + what collapsed every slug onto the fallback.""" + from papertrace.refs import parse_references + + text = "\n".join([ + "- 1 . Rivara FP. The privilege of being Editor-in-Chief. JAMA Netw Open. 2024;7(6):e2421821.", + "- 2 . Perencevich EN. Adapting to open access publishing. JAMA Netw Open. 2024;7(7):e2425000.", + "- 3 . Loftfield E, Abnet CC, et al. Multivitamin use and mortality risk. JAMA Netw Open. 2024.", + ]) + entries = parse_references(text) + + assert [e.num for e in entries] == ["1", "2", "3"] + assert entries[0].raw.startswith("Rivara FP"), entries[0].raw + assert entries[0].slug == "rivara-2024" + assert entries[1].slug == "perencevich-2024" + + +def test_two_references_can_never_share_a_slug(): + """`resolve_all` writes each download to `<slug>.pdf`, so a shared slug is a + shared file. Even a genuine collision — the same first author and year cited + twice — must not put two different papers at one path.""" + from papertrace.refs import parse_references + + text = "\n".join([ + "1. Smith J. First paper by Smith. J Fixture. 2020;1:1-9.", + "2. Smith J. A different paper by the same Smith. J Fixture. 2020;1:10-19.", + "3. Smith J. A third one, same year again. J Fixture. 2020;1:20-29.", + ]) + entries = parse_references(text) + + slugs = [e.slug for e in entries] + assert len(set(slugs)) == len(slugs), slugs + assert slugs[0] == "smith-2020", "the first keeps the natural slug" + + +def test_a_deposited_list_also_gets_unique_slugs(): + """The Crossref leg is the other producer of entries and needs the same + guarantee — a deposit routinely carries the same author twice.""" + from papertrace.refs import crossref_reference_list + + refs = [ + {"key": "r1", "author": "Smith J", "article-title": "One", "year": "2020"}, + {"key": "r2", "author": "Smith J", "article-title": "Two", "year": "2020"}, + ] + with httpx.Client(transport=_deposit(refs)) as client: + entries = crossref_reference_list(client, "10.1/paper", "t@example.org") + + assert len({e.slug for e in entries}) == 2, [e.slug for e in entries] + + +def test_every_retrieved_source_gets_its_own_file(tmp_path): + """The damage, stated as the thing that must not happen.""" + from papertrace.refs import parse_references + + text = "\n".join([ + "- 1 . Alpha A. A paper. J Fixture. 2020;1:1-9.", + "- 2 . Beta B. Another paper. J Fixture. 2020;1:10-19.", + "- 3 . Gamma G. A third paper. J Fixture. 2020;1:20-29.", + ]) + entries = parse_references(text) + dests = [tmp_path / f"{e.slug}.pdf" for e in entries] + assert len(set(dests)) == len(dests), dests + + +def test_run_detects_the_doi_once_and_gives_it_to_both_stages(monkeypatch, tmp_path): + """`refs` detects the printed DOI for the numbering check, but `run` used to + forward its own raw `--doi` to the scout — so on `papertrace run paper.pdf` + the numbering check got the detected DOI and the scout silently fell back to + title matching, which the README warns can anchor to somebody else's paper. + One detection, both consumers.""" + import papertrace.cli as cli_mod + + seen = {} + monkeypatch.setattr(cli_mod, "_detected_doi", lambda m: "10.1234/detected") + # `ingest` and `refs` are split into a Typer command plus a `_..._pipeline` + # function, the plain function `run` actually calls — see cli.py's comment + # on `run()`. + monkeypatch.setattr(cli_mod, "_ingest_pipeline", lambda **kw: None) + monkeypatch.setattr(cli_mod, "_refs_pipeline", lambda **kw: seen.__setitem__("refs", kw.get("doi"))) + monkeypatch.setattr(cli_mod, "scout", lambda **kw: seen.__setitem__("scout", kw.get("doi"))) + monkeypatch.setattr(cli_mod, "_check_pipeline", lambda **kw: None) + monkeypatch.setattr(cli_mod, "highlight", lambda **kw: None) + monkeypatch.setattr(cli_mod, "_report_pipeline", lambda **kw: None) + monkeypatch.setattr(cli_mod, "_guard_case", lambda *a, **k: "hash") + monkeypatch.setattr(cli_mod, "_open_case", lambda *a, **k: None) + + pdf = _one_page_paper(tmp_path / "paper.pdf") + cli_mod.run(manuscript=pdf, case=tmp_path / "case", provided=None, + email="t@example.org", model=None, png=False, backend="pymupdf", + with_scout=True, doi=None) + + assert seen["refs"] == "10.1234/detected" + assert seen["scout"] == "10.1234/detected", "the scout was left to guess by title" + + +def test_a_deposit_from_the_wrong_paper_is_refused(tmp_path): + """The DOI is scraped off page 1, or typed by hand. Nothing checked that the + record behind it IS this manuscript — and an erratum, a preprint version or + a companion paper can easily carry the same number of references, so + `_covers` would pass and the report would print "numbering confirmed" over + another paper's reference list. Every other retrieval path in this module is + title-checked; the one path that can replace the entire list was not.""" + from papertrace.refs import deposit_is_this_paper + + assert deposit_is_this_paper( + "Image registration improves inter-reader agreement in CT of pancreas", + "Image registration improves inter-reader agreement in CT of pancreas") is True + # a different paper entirely + assert deposit_is_this_paper( + "Image registration improves inter-reader agreement in CT of pancreas", + "Maternal urinary fluoride and child neurobehavior at age three") is False + # nothing to compare with is not a mismatch — it is an unknown, and an + # unknown must not silently discard a good deposit + assert deposit_is_this_paper("", "Some title") is None + assert deposit_is_this_paper("Some title", "") is None + + +def test_a_confirmed_numbering_still_says_when_the_other_reading_disagreed(): + """`_covers` is a cardinality test. If the parse matches the body's labels + the numbering is confirmed — but a deposit that disagreed is a second + independent reading saying the list is wrong, and burying it in a field no + template renders tells the reader nothing.""" + from papertrace.refs import reconcile + + body = {str(n) for n in range(1, 42)} + _entries, rec = reconcile(body, crossref=_parsed(list(range(1, 40))), + parsed=_parsed(list(range(1, 42)))) + + assert rec.verified is True and rec.source == "parsed" + assert rec.contested is True, "the deposit disagreed and nobody is told" + + +def test_both_readings_agreeing_leaves_no_phantom_divergence(): + """`_first_divergence` returned `min(len)+1` when the readings agreed, so a + 41-entry list warned about "entries from [42] onward" while tainting no + claim at all — the banner and the per-claim layer contradicting each other.""" + from papertrace.refs import reconcile + + same = lambda: _parsed(list(range(1, 42))) # noqa: E731 + _entries, rec = reconcile(set(), crossref=same(), parsed=same()) + + assert rec.verified is False, "no arbiter, so nothing is confirmed" + assert rec.unverified_from is None, "agreement is not a divergence" + assert "agree throughout" in rec.note + + +def test_a_deposits_author_initials_do_not_become_the_slug(): + """Springer deposits `"author": "C Huang"`, so the first letter-bearing + token is the initial and the slug became `c-2020`. The slug is the download + filename, the report's source id, and the `--provided` match key that + `refs --provided`'s own help documents as `<firstauthor>-<year>.pdf`.""" + from papertrace.refs import crossref_reference_list + + refs = [ + {"key": "r1", "author": "C Huang", "article-title": "A paper", "year": "2020"}, + {"key": "r2", "author": "BP Abbott", "article-title": "Another", "year": "2016"}, + {"key": "r3", "author": "Foy", "article-title": "A third", "year": "2019"}, + ] + with httpx.Client(transport=_deposit(refs)) as client: + entries = crossref_reference_list(client, "10.1/paper", "t@example.org") + + assert [e.slug for e in entries] == ["huang-2020", "abbott-2016", "foy-2019"] + + +# --- the paper's own title, as the PDF declares it --------------------------- +# +# Measured on seven papers, four publishers: the block heuristic returns the +# article-type banner, not the title — `CLINICAL GUIDELINE` (Wiley), +# `RESEARCH ARTICLE` (Springer), `Journal Pre-proofs` (Elsevier), `Editorial` +# (AMA). The PDF's own metadata carries the exact title for six of the seven. +# Docling does not help: on the seventh it emits no `title` item at all and +# labels the real title `section_header`, behind the banner. + + +def _pdf_with_metadata_title(path: Path, title: str, first_block: str = "CLINICAL GUIDELINE"): + import pymupdf + + doc = pymupdf.open() + doc.set_metadata({"title": title}) + page = doc.new_page() + page.insert_text((72, 100), first_block, fontsize=16) + page.insert_text((72, 140), "Body text long enough to be a candidate title block.", fontsize=11) + path.parent.mkdir(parents=True, exist_ok=True) + doc.save(path) + doc.close() + return path + + +def test_ingest_records_the_title_the_pdf_declares(tmp_path): + """Recorded verbatim at ingest, for both backends, because it is provenance: + what the document says about itself. Whether it is *usable* is a separate + question, answered by `paper_title`.""" + from papertrace.ingest import ingest_pdf + + pdf = _pdf_with_metadata_title(tmp_path / "p.pdf", "A Real Article Title About Pancreas CT") + smap = ingest_pdf(pdf, tmp_path / "out", backend="pymupdf") + assert smap.declared_title == "A Real Article Title About Pancreas CT" + + from papertrace.models import SourceMap + + assert SourceMap.from_json(tmp_path / "out" / "source_map.json").declared_title == \ + "A Real Article Title About Pancreas CT" + + +def test_a_source_map_without_a_declared_title_still_loads(tmp_path): + """Additive, like every other field added here: an older map opens, and the + absence reads as "not recorded" rather than as an empty title.""" + import json + + from papertrace.models import SourceMap + + p = tmp_path / "old.json" + p.write_text(json.dumps({"doc": "p.pdf", "pages": 1, "converter": "pymupdf", "blocks": []})) + assert SourceMap.from_json(p).declared_title == "" + + +def test_paper_title_prefers_what_the_pdf_declares_over_the_first_block(tmp_path): + """The whole point: `CLINICAL GUIDELINE` is what the layout offers and it + identifies nothing.""" + from papertrace.ingest import ingest_pdf + from papertrace.models import paper_title + + pdf = _pdf_with_metadata_title(tmp_path / "p.pdf", + "Gaussian mixture modelling of intramuscular fat") + smap = ingest_pdf(pdf, tmp_path / "out", backend="pymupdf") + assert paper_title(smap) == "Gaussian mixture modelling of intramuscular fat" + + +def test_a_producer_artifact_is_not_a_title(tmp_path): + """What a Word-produced manuscript declares — this tool's main case. Reading + `Microsoft Word - Manuscript revised clean.docx` as the paper's title would + hand the identity check four confident words that describe no paper, and a + confident mismatch discards a good deposit.""" + from papertrace.ingest import ingest_pdf + from papertrace.models import paper_title + + for junk in ("Microsoft Word - Manuscript revised final clean.docx", + "manuscript_revised_final_clean.docx", "untitled", ""): + pdf = _pdf_with_metadata_title(tmp_path / "p.pdf", junk, + first_block="Body block that is long enough to serve") + smap = ingest_pdf(pdf, tmp_path / "out", backend="pymupdf") + assert paper_title(smap) == "Body block that is long enough to serve", junk + + +# --- the bibliography as a fingerprint --------------------------------------- +# +# Measured on the 41-reference audit: 38 of 41 deposited works appear somewhere +# in the printed list (93%), and 0 of 41 appear in a different paper's list. +# Compared as a set, not positionally — the same pair scores 34% in order, +# because that paper's parse is the misnumbered one this feature exists to +# catch, and order is precisely what is in question. + + +def test_two_readings_of_one_bibliography_confirm_the_paper(tmp_path): + """Identity without a title at all. This has to survive misnumbering, or it + would only work on the papers that never needed it.""" + from papertrace.refs import deposit_corroborates + + deposit = _parsed(list(range(1, 21))) + shuffled = [_entry(str(i), e.raw) for i, e in enumerate(reversed(deposit), 1)] + + c = deposit_corroborates(deposit, shuffled) + assert c.confirms is True + assert (c.found, c.total) == (20, 20) + + +def test_another_papers_bibliography_does_not_confirm_but_does_not_refute(tmp_path): + """Asymmetric on purpose: agreement is evidence of identity, disagreement is + not evidence of difference — two lists that disagree may be one paper read + badly, which is this module's whole subject. So a low overlap says "no + evidence", and the caller leaves the identity unconfirmed rather than + calling the record another paper.""" + from papertrace.refs import deposit_corroborates + + c = deposit_corroborates(_parsed(list(range(1, 21))), _parsed(list(range(40, 60)))) + assert c.confirms is False + assert c.found == 0 + assert c.refutes is False, "a bad parse must not be reported as a wrong paper" + + +def test_a_short_list_agreeing_proves_nothing(): + """Three references matching is a coincidence a two-page comment can produce.""" + from papertrace.refs import deposit_corroborates + + c = deposit_corroborates(_parsed([1, 2, 3]), _parsed([1, 2, 3])) + assert c.confirms is False + assert c.too_few is True + + +def test_an_unverifiable_title_falls_back_to_the_bibliography(tmp_path, monkeypatch): + """The AMA-shaped paper: no metadata title, and docling offers only + `Editorial`. The deposit is still checkable — against the list printed in + the paper itself.""" + from papertrace.refs import CrossrefDeposit + + manifest = _refs_with_deposit( + tmp_path, monkeypatch, + CrossrefDeposit(entries=_parsed([1, 2]), deposited=2, + publisher="Fixture Publishing", title=""), + title="Editorial", + ) + # two entries is below the floor, so the bibliography cannot settle it either + assert "could not be confirmed as this paper" in manifest.numbering_note + + +def _paper_with_n_refs(path: Path, title: str, n: int): + """A paper citing [1]..[n], whose printed list holds the same works + `_parsed` builds — so a deposit of those works corroborates it.""" + import pymupdf + + doc = pymupdf.open() + doc.set_metadata({"title": title}) + page = doc.new_page() + page.insert_text((72, 60), title, fontsize=16) + cites = " ".join(f"[{i}]" for i in range(1, n + 1)) + page.insert_text((72, 90), f"Body text citing {cites} here.", fontsize=9) + page.insert_text((72, 120), "References", fontsize=14) + for i in range(1, n + 1): + page.insert_text((72, 140 + i * 14), f"[{i}] {_ref_text(i)}", fontsize=7) + path.parent.mkdir(parents=True, exist_ok=True) + doc.save(path) + doc.close() + return path + + +def test_the_bibliography_confirms_the_deposit_when_the_title_cannot(tmp_path, monkeypatch): + """The AMA shape with a real reference list: `Editorial` where the title + should be, no usable metadata — and the deposit still checked, against the + list printed in the paper itself. Without this the gate is inert on the + papers whose titles are unreadable, which measurement put at four of seven.""" + import papertrace.cli as cli_mod + import papertrace.refs as refs_mod + from papertrace.models import RefManifest + from papertrace.refs import CrossrefDeposit + + monkeypatch.setattr(refs_mod, "resolve_all", + lambda entries, dest, email, provided_dir=None, progress=None, taken=None: entries) + monkeypatch.setattr(refs_mod, "crossref_deposit", lambda client, doi, email: CrossrefDeposit( + entries=_parsed(list(range(1, 9))), deposited=8, + publisher="Fixture Publishing", title="", # the record offers no title either + )) + + pdf = _paper_with_n_refs(tmp_path / "paper.pdf", "Editorial", 8) + cli_mod.refs(manuscript=pdf, case=tmp_path / "case", provided=None, + email="t@example.org", parse_only=False, backend="pymupdf", + doi="10.1234/asserted", supplement=None) + + manifest = RefManifest.from_json(tmp_path / "case" / "refs_manifest.json") + assert manifest.reference_source == "crossref" + assert "8 of the 8 references" in manifest.numbering_note + assert "another paper's bibliography would not" in manifest.numbering_note + assert "unverified" not in manifest.numbering_note + + +# --- the banner and the per-claim layer must say the same thing -------------- + + +def test_an_unnarrowed_doubt_taints_every_label_not_none(tmp_path): + """`_numbering` renders "every entry is affected" whenever the doubt could + not be narrowed, and `label_is_doubtful` returned False for *every* label + for exactly the same reason — `unverified_from is None`. So the report's + most severe warn-level disclosure asserted every entry was suspect while + marking no claim suspect, and a reader acting on one verdict saw nothing. + + Two shapes land here: a manifest written before the reference list was + reconciled at all, and a run where the two readings agree entry for entry + with no arbiter to confirm either — the superscript-citation case. In both, + nobody established which entries are wrong, and "unknown scope" has to read + as "all of them" in both places or in neither.""" + from papertrace.disclosures import _numbering + from papertrace.models import RefManifest + + old = RefManifest(manuscript="p.pdf", entries=_parsed([1, 2, 7])) + assert old.numbering_verified is False and old.unverified_from is None + + assert "every entry is affected" in _numbering(old).short + assert [x for x in ("1", "2", "7", "99") if old.label_is_doubtful(x)] == \ + ["1", "2", "7", "99"], "the banner claims every entry and the labels claim none" + + +def test_a_confirmed_numbering_still_taints_nothing(): + """The other direction, so the fix cannot be "taint everything always".""" + from papertrace.models import RefManifest + + ok = RefManifest(manuscript="p.pdf", entries=_parsed([1, 2]), + numbering_verified=True, unverified_from=None) + assert [x for x in ("1", "2", "99") if ok.label_is_doubtful(x)] == [] + + +def test_a_narrowed_doubt_still_taints_only_the_tail(): + """And a located divergence keeps its scope: [1]-[14] stay trustworthy.""" + from papertrace.models import RefManifest + + m = RefManifest(manuscript="p.pdf", entries=_parsed(list(range(1, 20))), + numbering_verified=False, unverified_from=15) + assert [x for x in ("1", "14", "15", "19") if m.label_is_doubtful(x)] == ["15", "19"] + + +# --- a file nobody named for this reference ---------------------------------- + + +def _paper_pdf(path: Path, title: str, byline: str): + import pymupdf + + doc = pymupdf.open() + page = doc.new_page() + page.insert_text((72, 100), title, fontsize=14) + page.insert_text((72, 140), byline, fontsize=10) + path.parent.mkdir(parents=True, exist_ok=True) + doc.save(path) + doc.close() + return path + + +def test_a_token_matched_provided_file_that_is_another_paper_is_not_used(tmp_path, monkeypatch): + """`_unique_slugs` renames the second of two colliding entries to + `smith-2019-r7`, and `_provided_candidates` drops slug tokens of three + characters or fewer — so `r7`, the only thing distinguishing them, is + invisible and `sources/smith-2019.pdf` matches *both*. Measured before this + change: entry [7] came back `status=provided`, `title_check=mismatch`, + `pdf_path=smith-2019.pdf` — judged against entry [2]'s paper. + + "Disclosed, not fatal" is right for a file the user *named* for a reference: + they chose it, there is nothing to fall back to, and a scanned PDF yields no + text to check. It is wrong for a file a token match found, because nobody + chose it for this reference and the check says it is a different paper. Then + the honest move is to keep looking, and to say the file was set aside.""" + import papertrace.refs as refs_mod + from papertrace.refs import _entry, _unique_slugs, resolve_all + + monkeypatch.setattr(refs_mod, "_crossref_doi", lambda client, raw, email: None) + prov, dest = tmp_path / "sources", tmp_path / "resolved" + dest.mkdir() + _paper_pdf(prov / "smith-2019.pdf", + "Ultrasound elastography of the thyroid gland in children", + "Smith J, Jones B. Journal of Paediatric Radiology 2019;12:100-9.") + + ents = _unique_slugs([ + _entry("2", "Smith J, Jones B. Ultrasound elastography of the thyroid gland in " + "children. J Paediatr Radiol 2019;12:100-9."), + _entry("7", "Smith J, Patel R. Deep learning segmentation of renal cysts on CT. " + "Eur J Radiol 2019;44:220-8."), + ]) + two, seven = resolve_all(ents, dest, email="t@example.org", provided_dir=prov) + + # the reference the file really is: unchanged, and confirmed + assert (two.status, two.title_check) == ("provided", "verified") + assert Path(two.pdf_path).name == "smith-2019.pdf" + + # the other one is not judged against it + assert seven.pdf_path is None, "a claim would be judged against another paper" + assert seven.status == "no_doi" + # and the file it declined is named, because the tool knows it considered one + assert "smith-2019.pdf" in seven.reason + assert "different paper" in seven.reason diff --git a/tests/test_refs.py b/tests/test_refs.py index 8c1a60b..5a58e1d 100644 --- a/tests/test_refs.py +++ b/tests/test_refs.py @@ -790,3 +790,147 @@ def test_three_generic_domain_words_are_not_an_identity_check(): assert state == "unverifiable", detail # a page with none of the reference's words is still called wrong outright assert _title_check_text(ACR_WEBPAGE_REF, RIGHT_PAGE)[0] == "mismatch" + + +# --- the resolver must not mint bibliographic facts ------------------------- +# +# A first real audit reported 46 references on a paper citing 43. Three of the +# paper's own table captions reached the resolver, and Crossref answered a title +# search for "Table 1. Dataset characteristics" with 10.7717/peerj.7892/table-1 +# — a table-component DOI belonging to an unrelated paper. The report published +# all three as `paywalled`, i.e. as real works held behind a paywall. The title +# sanity check never fired, because it only runs on the download path and +# nothing was ever downloaded. + + +def test_a_non_reference_is_never_title_searched(tmp_path): + """The parser is fallible, so the resolver is the second line. An entry with + no year, no DOI and no arXiv id is not a citable work, and Crossref always + answers a title search with *something*.""" + asked: list[str] = [] + + def handler(request: httpx.Request) -> httpx.Response: + asked.append(str(request.url)) + return httpx.Response( + 200, json={"message": {"items": [{"DOI": "10.7717/peerj.7892/table-1"}]}} + ) + + e = RefEntry(num="44", raw="Table 1. Dataset characteristics") + resolve_entry(e, tmp_path, "t@example.org", httpx.Client(transport=httpx.MockTransport(handler))) + + assert asked == [], f"a table caption was sent to a bibliographic search: {asked}" + assert e.status == "no_doi" + assert e.doi is None + + +def test_the_refusal_says_why_rather_than_reading_as_a_lookup_failure(tmp_path): + """`no_doi` alone would read as "we looked and found nothing".""" + e = RefEntry(num="44", raw="Table 2. Accuracy and reliability of thresholding models") + resolve_entry(e, tmp_path, "t@example.org", _client({})) + assert "not searched by title" in e.reason.lower() + + +def test_a_component_doi_from_a_title_search_is_refused(tmp_path): + """Belt and braces: even a reference-shaped entry must not accept a DOI that + names a table, a figure or a supplement. Those are parts of a work, never a + work, whatever the title matched.""" + e = RefEntry(num="7", raw="Someone S. A real-looking reference. J Imaging 2020;5:1-9.") + client = _client({ + "api.crossref.org": httpx.Response( + 200, json={"message": {"items": [{"DOI": "10.7717/peerj-cs.847/table-10"}]}} + ), + }) + resolve_entry(e, tmp_path, "t@example.org", client) + + assert e.doi is None, f"accepted a component DOI: {e.doi}" + assert e.status == "no_doi" + assert "table" in e.reason.lower() or "component" in e.reason.lower() + + +def test_a_component_doi_printed_in_the_reference_is_also_refused(tmp_path): + """The same rule wherever the DOI came from — a supplement DOI printed in + the reference itself is still not the paper. + + Parsed through `parse_references` on purpose: building a RefEntry by hand + leaves `doi` unset, so the test would pass without exercising the guard. + """ + (e,) = parse_references( + "References\n1. Someone S. A paper. J Imaging 2020. doi:10.1234/abcd.2020.s001\n" + ) + assert e.doi == "10.1234/abcd.2020.s001", "fixture did not parse the DOI it is testing" + + resolve_entry(e, tmp_path, "t@example.org", _client({})) + assert e.status == "no_doi" + assert e.doi is None + + +def test_an_ordinary_reference_still_reaches_crossref(tmp_path): + """The guard must not gate real references — the failure that matters most + here is the strict one, because a dropped reference is silent.""" + e = RefEntry(num="1", raw="Fixture F, Example E (2023) A method. J Synth Methods 5:e230024") + client = _client({ + "api.crossref.org": httpx.Response( + 200, json={"message": {"items": [{"DOI": "10.1148/ryai.230024"}]}} + ), + "api.unpaywall.org": httpx.Response(200, json={}), + }) + resolve_entry(e, tmp_path, "t@example.org", client) + assert e.doi == "10.1148/ryai.230024" + assert e.status == "paywalled" + + +def test_an_accepted_download_records_what_its_title_check_rested_on(tmp_path): + """`title_check: verified` with no evidence beside it is a bare assurance. + + The mismatch branch always stated its detail; the accepting branch threw it + away, so a real audit's manifest showed six sources marked verified with + nothing a reader could weigh — and `unverifiable` looked the same. + """ + pdf = _real_pdf_bytes("Preoperative deltoid size and fatty infiltration of the deltoid") + (e,) = parse_references( + "References\n1. B.P. Wiater et al. Preoperative deltoid size and fatty " + "infiltration of the deltoid. Clin Orthop 2015. doi:10.1007/s11999-014-4047-2\n" + ) + client = _client({ + "api.unpaywall.org": httpx.Response( + 200, json={"best_oa_location": {"url_for_pdf": "https://x/oa.pdf"}} + ), + "https://x/oa.pdf": httpx.Response(200, content=pdf), + }) + resolve_entry(e, tmp_path, "t@example.org", client) + + assert e.status == "retrieved" and e.title_check == "verified" + assert "title check:" in e.reason, e.reason + assert "tokens on its first page" in e.reason, e.reason + + +def test_a_truncated_but_real_reference_is_still_searched(tmp_path): + """The strict-direction error, caught on real data before it shipped. + + Two references in one audit reached the resolver truncated mid-title — + authors plus half a title, no journal, no volume, no year — because the + converter cut them short. Crossref found both correct DOIs from exactly + that string. A shape test keyed on the year alone refused them, turning two + resolvable references into recorded gaps: the silent failure, and the one + that matters more than letting a stray caption through. + """ + asked: list[str] = [] + + def handler(request: httpx.Request) -> httpx.Response: + asked.append(str(request.url)) + if "api.crossref.org" in str(request.url): + return httpx.Response( + 200, json={"message": {"items": [{"DOI": "10.1177/0363546512452714"}]}} + ) + return httpx.Response(404) + + e = RefEntry( + num="14", + raw="M.A. Slabaugh, N.A. Friel, V. Karas, A.A. Romeo, N.N. Verma, B.J. Cole, " + "Interobserver and intraobserver reliability of the Goutallier Classification using", + ) + resolve_entry(e, tmp_path, "t@example.org", + httpx.Client(transport=httpx.MockTransport(handler))) + + assert any("crossref" in u for u in asked), "a real reference was never looked up" + assert e.doi == "10.1177/0363546512452714" diff --git a/tests/test_report_formats.py b/tests/test_report_formats.py new file mode 100644 index 0000000..c0965b2 --- /dev/null +++ b/tests/test_report_formats.py @@ -0,0 +1,155 @@ +"""Three report files were written whether or not anyone wanted three. + +`report.md` is the artefact almost every run is read through; the editor and +terminal looks are for sharing and for screenshots. Writing all three plus a +bundled font directory on every run was output nobody asked for, so the CLI now +writes markdown alone and takes the HTML looks on request. + +`write_reports` itself still defaults to every format: it is the seam the +disclosure-parity suite drives, and that suite must keep rendering all three or +it stops testing parity. The *policy* moved to the CLI, which is where the +user's intent actually is. +""" + +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from papertrace import cli # noqa: E402 +from papertrace.models import ClaimResult, RunResults # noqa: E402 +from papertrace.report import FORMATS, write_reports # noqa: E402 + + +def _results() -> RunResults: + claim = ClaimResult(id=1, claim="X causes Y.", location="Intro", refs=["1"], + verdict="supported", note="Stated on page 2.", + source_slug="a-2020", source_page=2, source_block="block_0002") + return RunResults(manuscript="m.pdf", date="2026-01-01", claims=[claim]) + + +# --- the library seam: unchanged by default -------------------------------- + + +def test_write_reports_still_writes_every_format_by_default(tmp_path): + """The parity suite calls this with no `formats` and asserts a disclosure + reached all three looks. Narrowing this default would leave that suite + green while it silently stopped comparing anything.""" + paths = write_reports(_results(), None, tmp_path, png=False) + assert {p.name for p in paths} == { + "report.md", "report_editor.html", "report_terminal.html" + } + + +def test_the_format_vocabulary_is_published(tmp_path): + assert FORMATS == ("md", "editor", "terminal") + + +# --- asking for less ------------------------------------------------------- + + +def test_markdown_alone_writes_no_html_and_no_font_bundle(tmp_path): + paths = write_reports(_results(), None, tmp_path, png=False, formats=("md",)) + assert {p.name for p in paths} == {"report.md"} + assert not (tmp_path / "report_editor.html").exists() + assert not (tmp_path / "report_terminal.html").exists() + # the fonts exist only to make the HTML self-contained; copying ~1 MB of + # them beside a markdown file nobody will open in a browser is litter + assert not (tmp_path / "assets").exists() + + +def test_one_html_look_does_not_drag_in_the_other(tmp_path): + paths = write_reports(_results(), None, tmp_path, png=False, + formats=("md", "editor")) + assert {p.name for p in paths} == {"report.md", "report_editor.html"} + assert not (tmp_path / "report_terminal.html").exists() + assert (tmp_path / "assets").exists(), "an HTML look still needs its fonts" + + +def test_markdown_is_always_written_even_if_not_asked_for(tmp_path): + """`report.md` is the machine-and-human readable record of the audit. A + request for only a screenshot look must not leave the case folder without + the report itself.""" + paths = write_reports(_results(), None, tmp_path, png=False, + formats=("terminal",)) + assert (tmp_path / "report.md") in paths + + +# --- refusing what it cannot do ------------------------------------------- + + +def test_an_unknown_format_is_refused_rather_than_ignored(tmp_path): + """Silently dropping an unrecognised name would answer "--format pdf" with + a folder that has no PDF in it and no complaint — the same shape as the + unknown-backend bug, which is why `ingest_pdf` raises there too.""" + with pytest.raises(ValueError, match="unknown report format"): + write_reports(_results(), None, tmp_path, png=False, formats=("pdf",)) + + +def test_no_formats_at_all_is_refused(tmp_path): + with pytest.raises(ValueError, match="unknown report format|no report format"): + write_reports(_results(), None, tmp_path, png=False, formats=()) + + +# --- PNG is rendered FROM the HTML, so it implies it ----------------------- + + +def test_png_pulls_in_the_html_it_is_rendered_from(tmp_path, monkeypatch): + """`--png --format md` cannot mean "screenshot a file I told you not to + write". Honouring it literally would produce no PNG and say nothing.""" + shot = [] + import papertrace.render as render_mod + + monkeypatch.setattr(render_mod, "html_to_png", + lambda src, dst: shot.append(src.name) or False) + write_reports(_results(), None, tmp_path, png=True, formats=("md",)) + assert (tmp_path / "report_editor.html").exists() + assert (tmp_path / "report_terminal.html").exists() + assert shot == ["report_editor.html", "report_terminal.html"] + + +# --- the CLI is where the default changed ---------------------------------- + + +def _case(tmp_path: Path) -> Path: + case = tmp_path / "case" + (case / "out").mkdir(parents=True) + _results().to_json(case / "out" / "results.json") + return case + + +def test_the_report_stage_writes_markdown_alone_by_default(tmp_path): + case = _case(tmp_path) + cli._report_pipeline(case=case) + assert (case / "out" / "report.md").exists() + assert not (case / "out" / "report_editor.html").exists() + assert not (case / "out" / "report_terminal.html").exists() + + +def test_a_format_flag_adds_exactly_that_look(tmp_path): + case = _case(tmp_path) + cli._report_pipeline(case=case, formats=["editor"]) + assert (case / "out" / "report_editor.html").exists() + assert not (case / "out" / "report_terminal.html").exists() + + +def test_repeated_format_flags_accumulate(tmp_path): + case = _case(tmp_path) + cli._report_pipeline(case=case, formats=["editor", "terminal"]) + assert (case / "out" / "report_editor.html").exists() + assert (case / "out" / "report_terminal.html").exists() + + +def test_a_mistyped_format_is_refused_before_anything_is_written(tmp_path): + """A typo in a flag is user error, and this CLI answers user error with a + red line and exit 2 — not a ValueError traceback. Checked before the + results are loaded, so a bad flag cannot half-write a report folder.""" + import typer + + case = _case(tmp_path) + with pytest.raises(typer.Exit) as exc: + cli._report_pipeline(case=case, formats=["pdf"]) + assert exc.value.exit_code == 2 + assert not (case / "out" / "report.md").exists() diff --git a/tests/test_scout.py b/tests/test_scout.py index 135ed21..1e0c7b7 100644 --- a/tests/test_scout.py +++ b/tests/test_scout.py @@ -109,10 +109,11 @@ def test_scout_registers_and_dedup(tmp_path): assert newer == {"A citing follow-up", "Newer keyword hit"} assert {h.via for h in res.newer} == {"citing", "search"} - # cited-by-DOI, cited-by-slug and the paper itself never reach overlooked; - # a same-year hit does (year ≤ paper year, plausibly knowable) + # cited-by-DOI, cited-by-slug and the paper itself never reach overlooked — + # and neither does a same-year hit, which has its own register overlooked = {h.title for h in res.overlooked} - assert overlooked == {"Old uncited candidate", "Same-year neighbour"} + assert overlooked == {"Old uncited candidate"} + assert {h.title for h in res.same_year} == {"Same-year neighbour"} # newest first assert [h.year for h in res.newer] == [2023, 2022] @@ -154,7 +155,7 @@ def test_scout_json_roundtrip(tmp_path): res.to_json(out) data = json.loads(out.read_text()) - assert data["counts"] == {"newer": 2, "overlooked": 2} + assert data["counts"] == {"newer": 2, "overlooked": 1, "same_year": 1} again = ScoutResults.from_json(out) assert again.paper_year == 2020 assert {h.title for h in again.newer} == {h.title for h in res.newer} @@ -164,7 +165,9 @@ def test_scout_json_roundtrip(tmp_path): def test_keywords_drop_stopwords(): kws = _keywords("Towards a novel deep learning analysis of chest radiographs") assert "towards" not in kws and "novel" not in kws and "analysis" not in kws - assert kws[:3] == ["deep", "learning", "chest"] + # ranked by length rather than by position, so the specific words win + # wherever they sit in the title — `radiographs` over `deep` + assert kws[:2] == ["radiographs", "learning"], kws def test_email_fallback_old_env_var(monkeypatch): @@ -175,3 +178,261 @@ def test_email_fallback_old_env_var(monkeypatch): assert _email(None) == "old@example.org" monkeypatch.setenv("PAPERTRACE_EMAIL", "new@example.org") assert _email(None) == "new@example.org" # new name wins + + +# --- a failure has to say which failure it was ------------------------------ +# +# On a real audit the operator was told "paper not identified in Europe PMC — +# pass --doi to pin it", so they found the DOI and re-ran with it. Same message. +# They then curled Europe PMC by hand to establish what the tool already knew: +# the paper is a Journal Pre-proof and simply is not indexed. `scout.json` also +# recorded `"doi": ""`, so the artifact could not show what had been tried. + + +def _no_hits(request): + return httpx.Response(200, json={"resultList": {"result": []}}) + + +def test_a_pinned_doi_that_finds_nothing_does_not_ask_for_a_doi(tmp_path): + case = _case(tmp_path) + res = scout_case(case, doi="10.1016/j.ejrad.2026.113206", + transport=httpx.MockTransport(_no_hits)) + + assert "--doi" not in res.error, res.error + assert "10.1016/j.ejrad.2026.113206" in res.error, res.error + + +def test_a_pinned_doi_that_finds_nothing_says_the_paper_is_not_indexed(tmp_path): + """A DOI lookup returning nothing is a stronger fact than a failed title + heuristic, and a different one: absence of indexing, not absence of skill. + Zero candidates must not read as a clean literature search.""" + case = _case(tmp_path) + res = scout_case(case, doi="10.1016/j.ejrad.2026.113206", + transport=httpx.MockTransport(_no_hits)) + + assert "not indexed" in res.error.lower(), res.error + assert res.newer == [] and res.overlooked == [] + + +def test_the_doi_that_was_tried_survives_into_the_artifact(tmp_path): + """`scout.json` carried an empty doi, so the null was uninterpretable from + the file alone.""" + case = _case(tmp_path) + res = scout_case(case, doi="10.1016/j.ejrad.2026.113206", + transport=httpx.MockTransport(_no_hits)) + + assert res.paper_doi == "10.1016/j.ejrad.2026.113206" + + +def test_without_a_doi_the_advice_to_pin_one_still_stands(tmp_path): + """The original message is right when no DOI was given — keep it.""" + case = _case(tmp_path) + res = scout_case(case, transport=httpx.MockTransport(_no_hits)) + assert "--doi" in res.error + + +# --- the keyword query has to be about the subject --------------------------- +# +# A live audit of "Image registration improves inter-reader agreement of +# objective response in CT assessment of pancreas adenocarcinoma" searched for +# `image AND registration AND improves AND inter-reader`. Two faults in one +# line: `improves` is a verb carrying no topic, and taking the FIRST four +# content words never reaches the subject, which in this title sits at the end. +# The register came back with a stroke conference abstract whose shouted title +# contained "IMPROVES". + +_REAL_TITLE = ( + "Image registration improves inter-reader agreement of objective response " + "in CT assessment of pancreas adenocarcinoma" +) + + +def test_the_keyword_query_reaches_the_subject_of_the_paper(): + kws = _keywords(_REAL_TITLE) + assert "adenocarcinoma" in kws, kws + assert "registration" in kws, kws + + +def test_a_title_verb_is_not_a_keyword(): + """`improves` matched an unrelated abstract on the same verb.""" + assert "improves" not in _keywords(_REAL_TITLE) + + +def test_keyword_selection_does_not_depend_on_position_in_the_title(): + """The subject is as often at the end of a title as the start.""" + front = _keywords("Pancreas adenocarcinoma assessed by registration of CT") + back = _keywords("Registration of CT for assessment of pancreas adenocarcinoma") + assert "adenocarcinoma" in front and "adenocarcinoma" in back + + +# --- a same-year paper is not an overlooked one ------------------------------ + + +def test_a_same_year_hit_is_not_filed_as_overlooked(tmp_path): + """"Existed but uncited" invites the reader to ask what the authors missed. + A paper from the manuscript's own year may have appeared after submission, + so holding it to that standard is unfair — and on a real 2026 paper every + one of the fifteen candidates was from 2026.""" + case = _case(tmp_path) + res = scout_case(case, transport=_mock_transport()) + + assert "Same-year neighbour" not in {h.title for h in res.overlooked} + + +def test_a_same_year_hit_is_kept_in_its_own_register(tmp_path): + """Not dropped either: a paper published early in the same year is exactly + the kind of thing a reviewer might legitimately raise. It is a third + status, and folding it into either neighbour states something false.""" + case = _case(tmp_path) + res = scout_case(case, transport=_mock_transport()) + + assert "Same-year neighbour" in {h.title for h in res.same_year} + assert "Same-year neighbour" not in {h.title for h in res.newer} + + +def test_the_same_year_register_round_trips(tmp_path): + """Gate 2 — a new field is a schema change and a round-trip test.""" + from papertrace.models import ScoutResults + + case = _case(tmp_path) + res = scout_case(case, transport=_mock_transport()) + path = tmp_path / "scout.json" + res.to_json(path) + + back = ScoutResults.from_json(path) + assert [h.title for h in back.same_year] == [h.title for h in res.same_year] + assert json.loads(path.read_text())["counts"]["same_year"] == len(res.same_year) + + +def test_an_older_uncited_hit_is_still_overlooked(tmp_path): + """The register keeps its job — this is a narrowing, not a removal.""" + case = _case(tmp_path) + res = scout_case(case, transport=_mock_transport()) + assert "Old uncited candidate" in {h.title for h in res.overlooked} + + +# --- Europe PMC returns escaped markup --------------------------------------- + + +def test_markup_entities_in_a_title_are_decoded(tmp_path): + """Real hits arrived as `CTV<sub>boost</sub>` and were rendered + verbatim into the report.""" + def handler(request: httpx.Request) -> httpx.Response: + url = str(request.url) + if "/citations" in url: + return httpx.Response(200, json={"citationList": {"citation": []}}) + q = request.url.params.get("query", "") + if q.startswith('DOI:"') or q.startswith('TITLE:"'): + return httpx.Response(200, json={"resultList": {"result": [{ + "id": "33333333", "source": "MED", "doi": "10.1000/PAPER", + "title": PAPER_TITLE, "pubYear": "2020", + }]}}) + return httpx.Response(200, json={"resultList": {"result": [ + _epmc_result("Improving CTV<sub>boost</sub> delineation", + 2018, doi="10.1000/esc"), + ]}}) + + res = scout_case(_case(tmp_path), transport=httpx.MockTransport(handler)) + titles = " ".join(h.title for h in res.overlooked) + assert "<" not in titles, titles + assert "CTV<sub>boost</sub>" in titles + + +def test_the_same_year_register_reaches_all_three_report_formats(tmp_path): + """A register the reader of one format cannot see is a register that does + not exist for them — the same rule the disclosure parity test enforces.""" + from papertrace.models import RunResults, ScoutHit, ScoutResults + from papertrace.report import write_reports + + scout = ScoutResults( + paper_title="A paper", paper_year=2026, date="2026-09-04", + # no apostrophe: the HTML looks autoescape their interpolations, so a + # literal assertion on the rendered page must not straddle an escape + same_year=[ScoutHit(title="A neighbour from the same publication year", year=2026, + doi="10.1000/sy", via="search", journal="Eur J Radiol")], + ) + out = tmp_path / "out" + write_reports(RunResults(manuscript="m.pdf"), None, out, png=False, scout=scout) + + for name in ("report.md", "report_editor.html", "report_terminal.html"): + body = (out / name).read_text() + assert "A neighbour from the same publication year" in body, f"missing from {name}" + assert "same year" in body.lower(), f"unlabelled in {name}" + + +# --- is the record behind the DOI this paper? -------------------------------- +# +# `run` reads the DOI off page 1 and hands it to `scout`, and `_resolve_paper` +# records `via="doi"` whenever a DOI is supplied — so the "wrong paper? pass +# --doi" warning, which fires only on `via="title"`, stopped firing exactly when +# the DOI became a guess. The provenance is not recoverable inside `scout`, and +# it is the wrong question anyway: the answer is to check the record. + + +def _other_paper_transport(): + """Europe PMC answers the DOI query with a different paper entirely.""" + + def handler(request: httpx.Request) -> httpx.Response: + q = request.url.params.get("query", "") + if q.startswith('DOI:"'): + return httpx.Response(200, json={"resultList": {"result": [{ + "id": "77777777", "source": "MED", "doi": "10.1000/OTHER", + "title": "Maternal urinary fluoride and child neurobehavior at age three", + "pubYear": "2017", + }]}}) + return httpx.Response(200, json={"resultList": {"result": []}}) + + return httpx.MockTransport(handler) + + +def test_a_doi_resolving_to_another_paper_is_refused(tmp_path): + """A funder, data-availability or erratum DOI on page 1 anchors the whole + scan to somebody else's paper, and both registers then describe that paper. + The registers are the finding, so they must not be built from a record this + tool can see is not the paper.""" + case = _case(tmp_path) + res = scout_case(case, doi="10.1000/other", transport=_other_paper_transport()) + + assert res.paper_identity == "mismatch" + assert res.newer == [] and res.overlooked == [] and res.same_year == [] + assert "not this paper" in res.error + assert "Maternal urinary fluoride" in res.error + + +def test_a_confirmed_paper_scans_as_before(tmp_path): + """The check must not cost the scan its point.""" + case = _case(tmp_path) + res = scout_case(case, doi="10.1000/paper", transport=_mock_transport()) + + assert res.paper_identity == "confirmed" + assert res.resolved_via == "doi" + assert res.newer, "the registers still get built" + + +def test_an_identity_too_thin_to_check_is_disclosed_not_assumed(tmp_path): + """Same tri-state as everywhere else: a first page whose opening block is a + journal banner leaves too little to compare, and that is not a mismatch.""" + case = _case(tmp_path) + smap_path = case / "ingest" / "manuscript" / "source_map.json" + smap = SourceMap.from_json(smap_path) + smap.blocks[0].text = "RESEARCH ARTICLE" # the measured real shape + smap.blocks[1].text = "Short prose." + smap.to_json(smap_path) + + res = scout_case(case, doi="10.1000/paper", transport=_mock_transport()) + assert res.paper_identity == "unverified" + assert res.newer, "an unknown identity does not discard a usable scan" + + +def test_scout_results_round_trip_the_identity(tmp_path): + """New field, so: schema vocabulary plus a round trip. An older scout.json + loads with `""` — not recorded, never "confirmed".""" + import json + + res = ScoutResults(paper_title="X", paper_identity="mismatch") + p = tmp_path / "scout.json" + res.to_json(p) + assert ScoutResults.from_json(p).paper_identity == "mismatch" + + p.write_text(json.dumps({"paper": {"title": "X"}, "newer": [], "overlooked": []})) + assert ScoutResults.from_json(p).paper_identity == "" diff --git a/tests/test_supplements.py b/tests/test_supplements.py new file mode 100644 index 0000000..6b15c15 --- /dev/null +++ b/tests/test_supplements.py @@ -0,0 +1,1061 @@ +"""Supplemental material: a supplement is a document, never the article. + +Offline like the rest of the suite — no network, no model calls. +""" + +import json +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from papertrace.models import RefEntry, RefManifest, Supplement # noqa: E402 + + +def _repo_root() -> Path: + for d in [Path(__file__).resolve(), *Path(__file__).resolve().parents]: + if (d / "pyproject.toml").exists(): + return d + raise RuntimeError("no pyproject.toml above this test file") + + +def _manifest() -> RefManifest: + return RefManifest( + manuscript="paper.pdf", + entries=[ + RefEntry( + num="14", raw="Pyrros A (2023) Something.", status="provided", + slug="pyrros-2023", pdf_path="/tmp/mine/pyrros-2023.pdf", + supplements=[ + Supplement(slug="pyrros-2023-supplement", + pdf_path="/tmp/mine/pyrros-2023-supplement.pdf"), + Supplement(slug="pyrros-2023-appendix-b", + pdf_path="/tmp/mine/pyrros-2023-appendix-b.pdf"), + ], + ), + RefEntry(num="15", raw="Chen B (2021) Other.", status="retrieved", + slug="chen-2021", pdf_path="/tmp/case/sources_resolved/chen-2021.pdf"), + ], + manuscript_supplements=[ + Supplement(slug="paper-si", pdf_path="/tmp/paper-si.pdf"), + ], + ) + + +# --- the wire format ------------------------------------------------------- + + +def test_supplements_round_trip_as_objects_not_dicts(tmp_path): + """`RefEntry(**e)` is a bare splat, and the manifest has carried no nested + dataclass until now — so a `list[Supplement]` serialises correctly through + `asdict` and reads back as plain dicts unless the reader hydrates it.""" + path = tmp_path / "refs_manifest.json" + _manifest().to_json(path) + + back = RefManifest.from_json(path) + got = back.entries[0].supplements + assert [type(s) for s in got] == [Supplement, Supplement], got + assert [s.slug for s in got] == ["pyrros-2023-supplement", "pyrros-2023-appendix-b"] + assert [type(s) for s in back.manuscript_supplements] == [Supplement] + assert back.manuscript_supplements[0].pdf_path == "/tmp/paper-si.pdf" + + +def test_the_manifest_schema_declares_supplements(tmp_path): + """`schemas/` is the published contract, not documentation. + + Validating alone proves nothing here: this schema sets no + `additionalProperties: false`, so an undeclared key passes silently. The + contract is only kept if the properties are actually written down. + """ + import jsonschema + + path = tmp_path / "refs_manifest.json" + _manifest().to_json(path) + schema = json.loads((_repo_root() / "schemas" / "refs_manifest.schema.json").read_text()) + jsonschema.validate(json.loads(path.read_text()), schema) + + entry = schema["properties"]["entries"]["items"]["properties"] + assert "supplements" in entry, "an entry's supplements are undeclared" + assert entry["supplements"]["type"] == "array" + sup = entry["supplements"]["items"]["properties"] + assert set(sup) == {"slug", "pdf_path", "verified"}, sup + assert "manuscript_supplements" in schema["properties"] + + +def test_a_manifest_written_before_supplements_still_loads(tmp_path): + """Absent means none, never "unknown" — an 0.5.x manifest has no such key.""" + import jsonschema + + path = tmp_path / "refs_manifest.json" + _manifest().to_json(path) + payload = json.loads(path.read_text()) + payload.pop("manuscript_supplements", None) + for e in payload["entries"]: + e.pop("supplements", None) + path.write_text(json.dumps(payload)) + + schema = json.loads((_repo_root() / "schemas" / "refs_manifest.schema.json").read_text()) + jsonschema.validate(json.loads(path.read_text()), schema) + + legacy = RefManifest.from_json(path) + assert [e.supplements for e in legacy.entries] == [[], []] + assert legacy.manuscript_supplements == [] + + +def test_an_unknown_manifest_key_does_not_crash_the_reader(tmp_path): + """`RefEntry(**e)` raises TypeError on a key it does not declare, so a + manifest from a NEWER papertrace killed an older one outright. Every other + reader in this codebase defaults forward; this one refused to.""" + path = tmp_path / "refs_manifest.json" + _manifest().to_json(path) + payload = json.loads(path.read_text()) + payload["entries"][0]["some_field_from_the_future"] = "hello" + path.write_text(json.dumps(payload)) + + back = RefManifest.from_json(path) + assert back.entries[0].slug == "pyrros-2023" + + +# --- the lookup that hides the distinction --------------------------------- + + +def test_document_resolves_an_article_a_supplement_and_the_papers_own(tmp_path): + """Four call sites hand-rolled `next(e for e in entries if e.slug == slug)`. + None of them may need to learn what a supplement is.""" + m = _manifest() + + art = m.document("chen-2021") + assert (art.kind, art.ref_num, art.parent_slug) == ("article", "15", None) + assert art.pdf_path == "/tmp/case/sources_resolved/chen-2021.pdf" + + sup = m.document("pyrros-2023-appendix-b") + assert (sup.kind, sup.ref_num, sup.parent_slug) == ("supplement", "14", "pyrros-2023") + assert sup.pdf_path == "/tmp/mine/pyrros-2023-appendix-b.pdf" + + own = m.document("paper-si") + assert (own.kind, own.ref_num, own.parent_slug) == ("own_supplement", "", None) + + assert m.document("nobody-2099") is None + + +def test_documents_lists_every_judgeable_file_once(tmp_path): + """`_check_pipeline` walks the manifest to record how each source was read. + Walking `entries` alone would leave every supplement undisclosed.""" + m = _manifest() + assert [d.slug for d in m.documents()] == [ + "pyrros-2023", + "pyrros-2023-supplement", + "pyrros-2023-appendix-b", + "chen-2021", + "paper-si", + ] + + +def test_a_slug_is_never_shared_between_a_supplement_and_an_article(): + """The whole identity story rests on the slug: `ingest/<slug>/`, + `sources_resolved/<slug>.pdf` and every judgement key off it, so two + documents sharing one means a verdict rendered against the wrong paper.""" + m = _manifest() + slugs = [d.slug for d in m.documents()] + assert len(slugs) == len(set(slugs)), slugs + + +# --- refs: attaching, and refusing to attach ------------------------------- + +PDF = b"%PDF-1.4 fake" + + +def _folder(tmp_path: Path, *names: str) -> Path: + d = tmp_path / "mine" + d.mkdir(parents=True, exist_ok=True) + for n in names: + (d / n).write_bytes(PDF) + return d + + +def _available(slug: str = "littlejohns-2020", num: str = "3") -> RefEntry: + return RefEntry(num=num, raw=f"{slug} et al.", slug=slug, status="provided", + pdf_path=f"/tmp/mine/{slug}.pdf") + + +def test_several_supplements_attach_to_one_available_reference(tmp_path): + from papertrace.refs import attach_supplements + + d = _folder(tmp_path, "littlejohns-2020.pdf", "littlejohns-2020-supplement.pdf", + "littlejohns-2020-appendix-b.pdf") + e = _available() + attach_supplements(e, d, taken={e.slug}) + + assert [s.slug for s in e.supplements] == [ + "littlejohns-2020-appendix-b", "littlejohns-2020-supplement", + ] + assert all(Path(s.pdf_path).exists() for s in e.supplements) + + +def test_the_article_itself_is_never_attached_as_its_own_supplement(tmp_path): + from papertrace.refs import attach_supplements + + d = _folder(tmp_path, "littlejohns-2020.pdf", "littlejohns-2020-supplement.pdf") + e = _available() + attach_supplements(e, d, taken={e.slug}) + assert [Path(s.pdf_path).name for s in e.supplements] == [ + "littlejohns-2020-supplement.pdf" + ] + + +def test_a_supplement_does_not_attach_to_a_reference_nobody_could_obtain(tmp_path): + """The user's rule, and the one that keeps + `test_a_supplement_alone_is_not_the_article` true: supplementary material + with no article behind it is judged against nothing at all.""" + from papertrace.refs import attach_supplements + + d = _folder(tmp_path, "littlejohns-2020-appendix.pdf") + e = RefEntry(num="3", raw="Littlejohns", slug="littlejohns-2020", status="paywalled") + attach_supplements(e, d, taken={e.slug}) + assert e.supplements == [] + + +def test_an_orphan_supplement_is_named_with_the_reason_it_was_set_aside(tmp_path): + """Silently ignoring a file the user deliberately supplied is the failure + mode this codebase exists to avoid — they would never learn it did nothing.""" + from papertrace.refs import attach_supplements, unused_provided + + d = _folder(tmp_path, "littlejohns-2020-appendix.pdf", "unrelated-supplement.pdf") + paywalled = RefEntry(num="3", raw="Littlejohns", slug="littlejohns-2020", + status="paywalled") + attach_supplements(paywalled, d, taken={paywalled.slug}) + + orphans = dict(unused_provided([paywalled], d)) + assert set(orphans) == {d / "littlejohns-2020-appendix.pdf", d / "unrelated-supplement.pdf"} + assert "[3]" in orphans[d / "littlejohns-2020-appendix.pdf"] + assert "could not tell" in orphans[d / "unrelated-supplement.pdf"] + + +def test_an_attached_supplement_is_not_also_reported_as_an_orphan(tmp_path): + from papertrace.refs import attach_supplements, unused_provided + + d = _folder(tmp_path, "littlejohns-2020.pdf", "littlejohns-2020-supplement.pdf") + e = _available() + e.pdf_path = str(d / "littlejohns-2020.pdf") # the article really is this file + attach_supplements(e, d, taken={e.slug}) + assert unused_provided([e], d) == [] + + +def test_a_supplement_slug_survives_a_sibling_being_removed(tmp_path): + """Ordinal slugs (-suppl1/-suppl2) would renumber here, and a stored verdict + would then point at a different PDF. Stem slugs are content-addressed.""" + from papertrace.refs import attach_supplements + + d = _folder(tmp_path, "littlejohns-2020-appendix-a.pdf", + "littlejohns-2020-appendix-b.pdf") + first = _available() + attach_supplements(first, d, taken={first.slug}) + before = {s.slug for s in first.supplements} + + (d / "littlejohns-2020-appendix-a.pdf").unlink() + second = _available() + attach_supplements(second, d, taken={second.slug}) + + assert [s.slug for s in second.supplements] == ["littlejohns-2020-appendix-b"] + assert "littlejohns-2020-appendix-b" in before + + +def test_a_supplement_never_takes_a_slug_an_article_already_has(tmp_path): + from papertrace.refs import attach_supplements + + d = _folder(tmp_path, "littlejohns-2020-supplement.pdf") + e = _available() + # a different reference already resolved to exactly this stem + attach_supplements(e, d, taken={e.slug, "littlejohns-2020-supplement"}) + assert [s.slug for s in e.supplements] == ["littlejohns-2020-supplement-2"] + + +def test_the_supplement_markers_still_do_not_eat_a_real_author(tmp_path): + """`si-mohamed-2021` is a real slug from a real audit. The attach path uses + the same marker list as the exclude path, so a mistake there would now go + the other way: the article itself judged as its own supplement.""" + from papertrace.refs import attach_supplements + + d = _folder(tmp_path, "si-mohamed-2021.pdf") + e = _available(slug="si-mohamed-2021") + attach_supplements(e, d, taken={e.slug}) + assert e.supplements == [] + + +# --- --supplement, for the audited paper's own ----------------------------- + + +def test_manuscript_supplements_take_stem_slugs_in_the_order_given(tmp_path): + from papertrace.refs import manuscript_supplements + + a, b = tmp_path / "paper SI.pdf", tmp_path / "paper-appendix.pdf" + for p in (a, b): + p.write_bytes(PDF) + + got = manuscript_supplements([a, b], taken=set()) + assert [s.slug for s in got] == ["paper-si", "paper-appendix"] + assert [s.pdf_path for s in got] == [str(a), str(b)] + + +def test_a_manuscript_supplement_cannot_shadow_a_reference(tmp_path): + """Both are read as `ingest/<slug>/` and cropped from + `sources_resolved/<slug>.pdf`, so a shared slug is a shared folder — the + paper's own appendix overwriting a cited source's ingest.""" + from papertrace.refs import manuscript_supplements + + p = tmp_path / "chen-2021.pdf" + p.write_bytes(PDF) + got = manuscript_supplements([p], taken={"chen-2021"}) + assert [s.slug for s in got] == ["chen-2021-2"] + + +def test_the_refs_command_passes_supplements_to_the_pipeline(tmp_path, monkeypatch): + from papertrace import cli + + seen = {} + monkeypatch.setattr(cli, "_refs_pipeline", lambda **kw: seen.update(kw)) + pdf = tmp_path / "p.pdf" + pdf.write_bytes(PDF) + si = tmp_path / "p-si.pdf" + si.write_bytes(PDF) + + cli.refs(manuscript=pdf, case=None, provided=None, email=None, parse_only=False, + backend="pymupdf", doi=None, supplement=[si]) + + assert seen["supplement"] == [si] + + +def test_run_forwards_supplements_to_refs(tmp_path, monkeypatch): + """`run` calls the pipeline functions directly, so a parameter it forgets to + name is simply dropped — the audit would run without the supplement and say + nothing about it.""" + import inspect + + from papertrace import cli + + seen = {} + monkeypatch.setattr(cli, "_ingest_pipeline", lambda **kw: None) + monkeypatch.setattr(cli, "_refs_pipeline", lambda **kw: seen.update(kw)) + monkeypatch.setattr(cli, "scout", lambda **kw: None) + monkeypatch.setattr(cli, "_check_pipeline", lambda **kw: None) + monkeypatch.setattr(cli, "highlight", lambda **kw: None) + monkeypatch.setattr(cli, "_report_pipeline", lambda **kw: None) + monkeypatch.setattr(cli, "_email", lambda v: "e@example.com") + monkeypatch.setattr(cli, "_detected_doi", lambda m: None) + + pdf = tmp_path / "p.pdf" + pdf.write_bytes(PDF) + si = tmp_path / "p-si.pdf" + si.write_bytes(PDF) + + cli.run(manuscript=pdf, case=tmp_path / "c", provided=None, email="e@example.com", + model=None, png=False, backend="pymupdf", with_scout=False, doi=None, + formats=None, supplement=[si]) + + assert seen["supplement"] == [si] + # and the flag really is declared on `run`, not silently swallowed by **kw + assert "supplement" in inspect.signature(cli.run).parameters + + +# --- check: a cited work's supplements are judged too ---------------------- + + +def _ingested(dirpath: Path, slug: str) -> None: + from papertrace.models import Block, SourceMap + + dirpath.mkdir(parents=True, exist_ok=True) + (dirpath / "annotated.md").write_text( + f"<!-- block_0001, page 1 -->\nText of {slug}.\n" + ) + SourceMap( + doc=f"{slug}.pdf", pages=1, + blocks=[Block("block_0001", "text", 1, (0.0, 0.0, 100.0, 20.0), [], + f"Text of {slug}.")], + ).to_json(dirpath / "source_map.json") + + +def _case_with_supplement(tmp_path: Path) -> RefManifest: + """One cited reference [1] carrying two supplements, all already ingested.""" + for slug in ("pyrros-2023", "pyrros-2023-supplement", "pyrros-2023-appendix-b"): + _ingested(tmp_path / "ingest" / slug, slug) + return RefManifest( + manuscript="m.pdf", + entries=[ + RefEntry(num="1", raw="Pyrros", status="retrieved", slug="pyrros-2023", + pdf_path="pyrros-2023.pdf", + supplements=[ + Supplement("pyrros-2023-supplement", "pyrros-2023-supplement.pdf"), + Supplement("pyrros-2023-appendix-b", "pyrros-2023-appendix-b.pdf"), + ]), + ], + ) + + +def _verdicts(mapping: dict[str, str]): + """A fake `_ask` answering per document, and the record of what it was asked.""" + seen: list[str] = [] + + def fake_ask(prompt, model=None): + slug = next(s for s in mapping if f"SOURCE ({s})" in prompt) + seen.append(slug) + return json.dumps([{ + "id": 1, "verdict": mapping[slug], "note": f"per {slug}", + "source_page": 1, "source_block": "block_0001", + "anchor_phrases": [f"Text of {slug}"], + }]) + + return fake_ask, seen + + +def test_a_supplement_is_judged_as_its_own_document(tmp_path, monkeypatch): + from papertrace import check as check_mod + from papertrace.models import ClaimResult + + manifest = _case_with_supplement(tmp_path) + claim = ClaimResult(id=1, claim="the cohort was imaged twice", location="Methods", + refs=["1"]) + fake_ask, seen = _verdicts({ + "pyrros-2023": "not_addressed", + "pyrros-2023-supplement": "contradicted", + "pyrros-2023-appendix-b": "not_addressed", + }) + monkeypatch.setattr(check_mod, "_ask", fake_ask) + check_mod.check_claims([claim], manifest, tmp_path, backend="pymupdf") + + assert sorted(seen) == [ + "pyrros-2023", "pyrros-2023-appendix-b", "pyrros-2023-supplement", + ], "one call per document" + assert {j.source_slug: j.verdict for j in claim.judgements} == { + "pyrros-2023": "not_addressed", + "pyrros-2023-supplement": "contradicted", + "pyrros-2023-appendix-b": "not_addressed", + } + # a supplement answers for the label its parent carries + assert {j.ref for j in claim.judgements} == {"1"} + assert {j.kind for j in claim.judgements} == {"article", "supplement"} + + +def test_a_supplement_contradicting_decides_the_headline(tmp_path, monkeypatch): + """The supplement is part of the cited work. A contradiction found only in + the appendix is still a contradiction the reviewer needs.""" + from papertrace import check as check_mod + from papertrace.models import ClaimResult + + manifest = _case_with_supplement(tmp_path) + claim = ClaimResult(id=1, claim="c", location="Methods", refs=["1"]) + fake_ask, _ = _verdicts({ + "pyrros-2023": "not_addressed", + "pyrros-2023-supplement": "contradicted", + "pyrros-2023-appendix-b": "not_addressed", + }) + monkeypatch.setattr(check_mod, "_ask", fake_ask) + check_mod.check_claims([claim], manifest, tmp_path, backend="pymupdf") + + assert claim.verdict == "contradicted" + assert claim.source_slug == "pyrros-2023-supplement" + + +def test_the_judge_is_told_which_kind_of_document_it_is_holding(tmp_path, monkeypatch): + """Handed an appendix with no warning, a judge has no reason to expect + `not_addressed` to be the ordinary answer.""" + from papertrace import check as check_mod + from papertrace.models import ClaimResult + + manifest = _case_with_supplement(tmp_path) + claim = ClaimResult(id=1, claim="c", location="Methods", refs=["1"]) + prompts: dict[str, str] = {} + + def fake_ask(prompt, model=None): + slug = next(s for s in ("pyrros-2023-supplement", "pyrros-2023-appendix-b", + "pyrros-2023") if f"SOURCE ({s})" in prompt) + prompts[slug] = prompt + return json.dumps([{"id": 1, "verdict": "not_addressed", "note": "n"}]) + + monkeypatch.setattr(check_mod, "_ask", fake_ask) + check_mod.check_claims([claim], manifest, tmp_path, backend="pymupdf") + + assert "<<DOCKIND>>" not in prompts["pyrros-2023"], "placeholder left unsubstituted" + assert "supplementary material" in prompts["pyrros-2023-supplement"].lower() + assert "supplementary material" not in prompts["pyrros-2023"].lower() + + +def test_a_supplement_is_not_judged_when_its_reference_is_not_cited(tmp_path, monkeypatch): + """A claim citing [2] must not pick up [1]'s appendix.""" + from papertrace import check as check_mod + from papertrace.models import ClaimResult + + manifest = _case_with_supplement(tmp_path) + manifest.entries.append( + RefEntry(num="2", raw="Chen", status="retrieved", slug="chen-2021", + pdf_path="chen-2021.pdf") + ) + _ingested(tmp_path / "ingest" / "chen-2021", "chen-2021") + claim = ClaimResult(id=1, claim="c", location="Methods", refs=["2"]) + fake_ask, seen = _verdicts({"chen-2021": "supported"}) + monkeypatch.setattr(check_mod, "_ask", fake_ask) + check_mod.check_claims([claim], manifest, tmp_path, backend="pymupdf") + + assert seen == ["chen-2021"] + assert [j.source_slug for j in claim.judgements] == ["chen-2021"] + + +def test_the_results_schema_declares_the_document_kind(tmp_path): + """A judgement's `kind` is what tells a reader of results.json alone that a + verdict came from an appendix. Declaring it is the contract; validating is + not enough, since nothing here forbids an undeclared key.""" + import jsonschema + + from papertrace.models import ClaimResult, RunResults, SourceJudgement + + results = RunResults( + manuscript="m.pdf", checker="claude -p", date="2026-09-06", + refs_total=1, refs_available=1, + claims=[ClaimResult( + id=1, claim="c", location="Methods", refs=["1"], + judgements=[ + SourceJudgement("pyrros-2023", "1", kind="article", verdict="not_addressed"), + SourceJudgement("pyrros-2023-supplement", "1", kind="supplement", + verdict="contradicted", source_page=1, + source_block="block_0001"), + ], + )], + ) + path = tmp_path / "results.json" + results.to_json(path) + schema = json.loads((_repo_root() / "schemas" / "results.schema.json").read_text()) + jsonschema.validate(json.loads(path.read_text()), schema) + + j = schema["properties"]["claims"]["items"]["properties"]["judgements"]["items"] + assert "kind" in j["properties"], "a judgement's document kind is undeclared" + assert set(j["properties"]["kind"]["enum"]) == {"article", "supplement", "own_supplement"} + + back = RunResults.from_json(path) + assert [x.kind for x in back.claims[0].judgements] == ["article", "supplement"] + + +def test_a_results_file_written_before_supplements_still_loads(tmp_path): + """Absent `kind` means the article — every judgement before 0.6.0 was one.""" + from papertrace.models import ClaimResult, RunResults, SourceJudgement + + results = RunResults( + manuscript="m.pdf", checker="claude -p", date="2026-09-06", + refs_total=1, refs_available=1, + claims=[ClaimResult(id=1, claim="c", location="M", refs=["1"], + judgements=[SourceJudgement("a-2020", "1", verdict="supported")])], + ) + path = tmp_path / "results.json" + results.to_json(path) + payload = json.loads(path.read_text()) + for j in payload["claims"][0]["judgements"]: + del j["kind"] + path.write_text(json.dumps(payload)) + + back = RunResults.from_json(path) + assert [x.kind for x in back.claims[0].judgements] == ["article"] + assert back.claims[0].judgements[0].origin == "cited as [1]" + + +def test_a_supplement_that_cannot_be_read_never_taints_the_articles_verdict(tmp_path, + monkeypatch): + """Gate 4. The supplement's own PDF is gone and it was never ingested, so + nobody read it — that is `unchecked`, on that document alone. The article + was read and its verdict stands; discarding it would report a gap that does + not exist, and letting the supplement default to anything would be a verdict + on a document nobody opened.""" + from papertrace import check as check_mod + from papertrace.models import ClaimResult + + manifest = RefManifest( + manuscript="m.pdf", + entries=[RefEntry(num="1", raw="Pyrros", status="retrieved", slug="pyrros-2023", + pdf_path="pyrros-2023.pdf", + supplements=[Supplement("pyrros-2023-supplement", + "/nonexistent/pyrros-2023-supplement.pdf")])], + ) + _ingested(tmp_path / "ingest" / "pyrros-2023", "pyrros-2023") # the supplement: not ingested + claim = ClaimResult(id=1, claim="c", location="Methods", refs=["1"]) + + monkeypatch.setattr(check_mod, "_ask", lambda p, model=None: json.dumps([{ + "id": 1, "verdict": "supported", "note": "per the article", + "source_page": 1, "source_block": "block_0001", + "anchor_phrases": ["Text of pyrros-2023"], + }])) + check_mod.check_claims([claim], manifest, tmp_path, backend="pymupdf") + + by_slug = {j.source_slug: j for j in claim.judgements} + assert by_slug["pyrros-2023"].verdict == "supported" + assert by_slug["pyrros-2023-supplement"].verdict == "unchecked" + assert "re-run" in by_slug["pyrros-2023-supplement"].note + # the readable document still decides the headline + assert claim.verdict == "supported" + # and the gap is NOT laundered into "the source could not be retrieved" + assert claim.unjudged_refs == [] + + +# --- check: claims that point at this paper's own supplement --------------- + + +def _manuscript_ingest(case: Path) -> None: + d = case / "ingest" / "manuscript" + d.mkdir(parents=True, exist_ok=True) + (d / "annotated.md").write_text( + "<!-- block_0001, page 1 -->\nModel AUC was 0.91 (Table S3).\n" + ) + + +def test_extraction_records_a_pointer_at_this_papers_own_supplement(tmp_path, monkeypatch): + from papertrace import check as check_mod + + monkeypatch.setattr(check_mod, "_ask", lambda p, m=None: json.dumps( + {"cited": [{"id": 1, "claim": "AUC 0.91.", "location": "Results", "refs": [], + "own_supplement": True}], + "uncited": []})) + _manuscript_ingest(tmp_path) + cited, _ = check_mod.extract_claims(tmp_path) + assert cited[0].own_supplement is True + + +def test_a_claim_with_no_such_pointer_defaults_to_false(tmp_path, monkeypatch): + from papertrace import check as check_mod + + monkeypatch.setattr(check_mod, "_ask", lambda p, m=None: json.dumps( + {"cited": [{"id": 1, "claim": "X causes Y.", "location": "Intro", "refs": ["1"]}], + "uncited": []})) + _manuscript_ingest(tmp_path) + cited, _ = check_mod.extract_claims(tmp_path) + assert cited[0].own_supplement is False + + +def test_the_extraction_prompt_asks_for_the_pointer(tmp_path, monkeypatch): + from papertrace import check as check_mod + + seen = {} + + def fake_ask(prompt, m=None): + seen["prompt"] = prompt + return json.dumps({"cited": [], "uncited": []}) + + monkeypatch.setattr(check_mod, "_ask", fake_ask) + _manuscript_ingest(tmp_path) + check_mod.extract_claims(tmp_path) + assert "own_supplement" in seen["prompt"] + + +def test_such_a_claim_is_judged_against_every_supplement_of_this_paper(tmp_path, monkeypatch): + """Judged against all of them, matching the rule on the cited side — and it + avoids asking the extractor to guess which file `S3` lives in.""" + from papertrace import check as check_mod + from papertrace.models import ClaimResult + + for slug in ("paper-si", "paper-appendix"): + _ingested(tmp_path / "ingest" / slug, slug) + manifest = RefManifest( + manuscript="m.pdf", entries=[], + manuscript_supplements=[Supplement("paper-si", "paper-si.pdf"), + Supplement("paper-appendix", "paper-appendix.pdf")], + ) + claim = ClaimResult(id=1, claim="AUC 0.91", location="Results", refs=[], + own_supplement=True) + fake_ask, seen = _verdicts({"paper-si": "supported", "paper-appendix": "not_addressed"}) + monkeypatch.setattr(check_mod, "_ask", fake_ask) + check_mod.check_claims([claim], manifest, tmp_path, backend="pymupdf") + + assert sorted(seen) == ["paper-appendix", "paper-si"] + assert {j.kind for j in claim.judgements} == {"own_supplement"} + assert {j.ref for j in claim.judgements} == {""} + assert claim.verdict == "supported" + assert claim.judgements[0].origin == "this paper's own supplement" + + +def test_a_pointer_with_nothing_supplied_is_not_retrieved_and_says_how_to_fix_it( + tmp_path, monkeypatch +): + """The paper named where its evidence was and nobody opened it. Leaving + that in the uncited register would call it an assertion with no citation, + which understates it.""" + from papertrace import check as check_mod + from papertrace.models import ClaimResult + + manifest = RefManifest(manuscript="m.pdf", entries=[]) + claim = ClaimResult(id=1, claim="AUC 0.91", location="Results", refs=[], + own_supplement=True) + monkeypatch.setattr(check_mod, "_ask", lambda p, m=None: pytest.fail("no call is due")) + check_mod.check_claims([claim], manifest, tmp_path, backend="pymupdf") + + assert claim.verdict == "not_retrieved" + assert "--supplement" in claim.note + assert claim.judgements == [] + + +def test_a_claim_pointing_at_both_a_citation_and_the_supplement_gets_both(tmp_path, + monkeypatch): + from papertrace import check as check_mod + from papertrace.models import ClaimResult + + for slug in ("chen-2021", "paper-si"): + _ingested(tmp_path / "ingest" / slug, slug) + manifest = RefManifest( + manuscript="m.pdf", + entries=[RefEntry(num="1", raw="Chen", status="retrieved", slug="chen-2021", + pdf_path="chen-2021.pdf")], + manuscript_supplements=[Supplement("paper-si", "paper-si.pdf")], + ) + claim = ClaimResult(id=1, claim="c", location="Results", refs=["1"], + own_supplement=True) + fake_ask, seen = _verdicts({"chen-2021": "supported", "paper-si": "supported"}) + monkeypatch.setattr(check_mod, "_ask", fake_ask) + check_mod.check_claims([claim], manifest, tmp_path, backend="pymupdf") + + assert sorted(seen) == ["chen-2021", "paper-si"] + assert {j.source_slug: j.kind for j in claim.judgements} == { + "chen-2021": "article", "paper-si": "own_supplement", + } + + +def test_the_results_schema_declares_the_own_supplement_pointer(tmp_path): + import jsonschema + + from papertrace.models import ClaimResult, RunResults + + results = RunResults( + manuscript="m.pdf", checker="claude -p", date="2026-09-06", + refs_total=0, refs_available=0, + claims=[ClaimResult(id=1, claim="AUC 0.91", location="Results", refs=[], + own_supplement=True, verdict="not_retrieved")], + ) + path = tmp_path / "results.json" + results.to_json(path) + schema = json.loads((_repo_root() / "schemas" / "results.schema.json").read_text()) + jsonschema.validate(json.loads(path.read_text()), schema) + assert "own_supplement" in schema["properties"]["claims"]["items"]["properties"] + + payload = json.loads(path.read_text()) + del payload["claims"][0]["own_supplement"] + path.write_text(json.dumps(payload)) + assert RunResults.from_json(path).claims[0].own_supplement is False + + +# --- reports: a headline ranges over documents ----------------------------- + + +def _results_with_supplement(**claim_kw): + from papertrace.models import ClaimResult, RunResults, SourceJudgement + + base = dict( + id=1, claim="the cohort was imaged twice", location="Methods", refs=["14"], + judgements=[ + SourceJudgement("pyrros-2023", "14", kind="article", verdict="not_addressed", + note="silent on this"), + SourceJudgement("pyrros-2023-supplement", "14", kind="supplement", + verdict="contradicted", note="Table S2 reports 0.71", + source_page=1, source_block="block_0001"), + ], + ) + base.update(claim_kw) + claim = ClaimResult(**base) + claim.apply_headline() + return RunResults(manuscript="m.pdf", converter="docling", claims=[claim]), claim + + +def test_the_qualifier_says_documents_when_a_supplement_is_among_them(): + """"most adverse of 2 cited sources" would be false — there is one cited + source here, read as two documents.""" + results, claim = _results_with_supplement() + assert claim.verdict == "contradicted" + assert claim.headline_qualifier() == "most adverse of 2 documents" + + +def test_the_qualifier_still_says_cited_sources_when_they_all_are(): + from papertrace.models import ClaimResult, SourceJudgement + + c = ClaimResult(id=1, claim="c", location="M", refs=["1", "2"], judgements=[ + SourceJudgement("a-2020", "1", verdict="supported"), + SourceJudgement("b-2021", "2", verdict="partial"), + ]) + c.apply_headline() + assert c.headline_qualifier() == "most adverse of 2 cited sources" + + +def test_every_format_names_the_document_a_verdict_came_from(tmp_path): + from papertrace.report import write_reports + + results, _ = _results_with_supplement() + write_reports(results, None, tmp_path, png=False) + for name in ("report.md", "report_editor.html", "report_terminal.html"): + body = (tmp_path / name).read_text() + assert "supplement to [14]" in body, f"{name} does not say the verdict is from a supplement" + + +def test_no_format_renders_an_empty_citation_label(tmp_path): + """The paper's own supplement answers for no label. Three templates used to + build `cited as [{{ j.ref }}]` by hand, which renders `cited as []`.""" + from papertrace.models import ClaimResult, RunResults, SourceJudgement + from papertrace.report import write_reports + + c = ClaimResult(id=1, claim="AUC 0.91", location="Results", refs=[], + own_supplement=True, judgements=[ + SourceJudgement("paper-si", "", kind="own_supplement", + verdict="supported", note="Table S3 gives 0.91", + source_page=1, source_block="block_0001")]) + c.apply_headline() + write_reports(RunResults(manuscript="m.pdf", claims=[c]), None, tmp_path, png=False) + for name in ("report.md", "report_editor.html", "report_terminal.html"): + body = (tmp_path / name).read_text() + assert "cited as []" not in body, name + assert "this paper" in body and "own supplement" in body, name + + +# --- reports: the disclosures supplements owe the reader ------------------- + + +def test_a_supplement_is_disclosed_as_unverified_in_all_three_formats(tmp_path): + """Every article is checked against the reference that names it. A + supplement's title does not match its parent's, so that check cannot apply + and is not faked — which makes this the weakest provenance in the tool, and + a reader has to be told.""" + from papertrace.disclosures import run_disclosures + from papertrace.report import write_reports + + results, _ = _results_with_supplement() + fired = {d.key: d for d in run_disclosures(results)} + assert "supplement_identity" in fired + + write_reports(results, None, tmp_path, png=False) + for name in ("report.md", "report_editor.html", "report_terminal.html"): + body = (tmp_path / name).read_text() + assert fired["supplement_identity"].token in body, name + + +def test_the_coverage_blind_spot_is_disclosed_when_supplements_were_read(tmp_path): + """A [N] occurring only inside a supplement is not counted by the audit, + which reads the manuscript alone. Stated, not hidden.""" + from papertrace.disclosures import run_disclosures + from papertrace.report import write_reports + + results, _ = _results_with_supplement() + results.coverage = {"labels_in_text": ["14"], "covered": ["14"], "missing": []} + fired = {d.key: d for d in run_disclosures(results)} + assert "supplement_coverage" in fired + + write_reports(results, None, tmp_path, png=False) + for name in ("report.md", "report_editor.html", "report_terminal.html"): + assert fired["supplement_coverage"].token in (tmp_path / name).read_text(), name + + +def test_neither_disclosure_fires_when_no_supplement_was_read(): + from papertrace.disclosures import run_disclosures + from papertrace.models import ClaimResult, RunResults, SourceJudgement + + c = ClaimResult(id=1, claim="c", location="M", refs=["1"], judgements=[ + SourceJudgement("a-2020", "1", verdict="supported")]) + results = RunResults(manuscript="m.pdf", claims=[c], + coverage={"labels_in_text": ["1"], "covered": ["1"], "missing": []}) + keys = {d.key for d in run_disclosures(results)} + assert "supplement_identity" not in keys + assert "supplement_coverage" not in keys + + +def test_a_headline_decided_by_a_supplement_says_so_in_all_three_formats(tmp_path): + """The claim reads `contradicted` on the strength of an appendix while the + article of record is silent. A reader acting on the headline alone needs + that on the claim, not only in a run-level footnote.""" + from papertrace.disclosures import claim_disclosures + from papertrace.report import write_reports + + results, claim = _results_with_supplement() + fired = {d.key: d for d in claim_disclosures(claim)} + assert "supplement_headline" in fired + + write_reports(results, None, tmp_path, png=False) + for name in ("report.md", "report_editor.html", "report_terminal.html"): + assert fired["supplement_headline"].token in (tmp_path / name).read_text(), name + + +def test_no_such_claim_disclosure_when_the_article_itself_decided(): + from papertrace.disclosures import claim_disclosures + from papertrace.models import ClaimResult, SourceJudgement + + c = ClaimResult(id=1, claim="c", location="M", refs=["14"], judgements=[ + SourceJudgement("pyrros-2023", "14", kind="article", verdict="contradicted", + source_page=1, source_block="block_0001"), + SourceJudgement("pyrros-2023-supplement", "14", kind="supplement", + verdict="not_addressed"), + ]) + c.apply_headline() + assert "supplement_headline" not in {d.key for d in claim_disclosures(c)} + + +def test_the_per_claim_count_does_not_call_a_supplement_a_cited_source(): + """One cited work read as two documents is not two cited works. The count + sits directly under the headline and would overstate how many independent + papers were consulted.""" + from papertrace.disclosures import claim_disclosures + + _, claim = _results_with_supplement() + d = next(x for x in claim_disclosures(claim) if x.key == "sources") + assert "2 cited sources checked" not in d.text + assert "2 documents checked" in d.text + assert d.token in d.text and d.token in d.short + + +def test_a_plain_multi_source_claim_still_says_cited_sources(): + from papertrace.disclosures import claim_disclosures + from papertrace.models import ClaimResult, SourceJudgement + + c = ClaimResult(id=1, claim="c", location="M", refs=["1", "2"], judgements=[ + SourceJudgement("a-2020", "1", verdict="supported"), + SourceJudgement("b-2021", "2", verdict="contradicted"), + ]) + c.apply_headline() + d = next(x for x in claim_disclosures(c) if x.key == "sources") + assert "2 cited sources checked" in d.text + + +def test_the_papers_own_supplement_is_not_described_as_matched_by_filename(): + """It was named on the command line. Saying a file the user pointed at + directly was guessed from its name misstates which part is uncertain — the + identity gap here is that nothing checks the contents, not the file.""" + from papertrace.disclosures import claim_disclosures, run_disclosures + from papertrace.models import ClaimResult, RunResults, SourceJudgement + + c = ClaimResult(id=1, claim="AUC 0.91", location="Results", refs=[], + own_supplement=True, judgements=[ + SourceJudgement("paper-si", "", kind="own_supplement", + verdict="supported", source_page=1, + source_block="block_0001")]) + c.apply_headline() + head = next(x for x in claim_disclosures(c) if x.key == "supplement_headline") + assert "filename" not in head.text + + run = next(x for x in run_disclosures(RunResults(manuscript="m.pdf", claims=[c])) + if x.key == "supplement_identity") + assert "matched by filename" not in run.text + + +# --- the wizard asks for the sources folder it never asked for ------------- + + +def test_the_equivalent_command_replays_the_sources_folder_and_supplements(tmp_path): + """The wizard prints the one-line command its session amounts to. A line + that omits the flags the session used does not reproduce the audit.""" + from papertrace.wizard import equivalent_command + + cmd = equivalent_command( + manuscript=tmp_path / "paper.pdf", case=tmp_path / "case", doi=None, png=False, + with_scout=False, provided=tmp_path / "sources", email="e@example.com", + supplement=[tmp_path / "si.pdf", tmp_path / "appendix.pdf"], + ) + assert "--provided" in cmd + assert cmd.count("--supplement") == 2 + assert "si.pdf" in cmd and "appendix.pdf" in cmd + + +def test_the_cost_estimate_counts_the_supplements_it_can_see(tmp_path): + """Each supplement is one more document, so one more judging call. The + wizard states the cost before the user agrees to pay it, and an estimate + that ignores supplements understates what they are agreeing to.""" + from papertrace.wizard import supplement_workload + + d = tmp_path / "sources" + d.mkdir() + for n in ("pyrros-2023.pdf", "pyrros-2023-supplement.pdf", "chen-2021-appendix.pdf"): + (d / n).write_bytes(PDF) + + assert supplement_workload(d, [tmp_path / "own-si.pdf"]) == 3 + assert supplement_workload(d, []) == 2 + assert supplement_workload(None, []) == 0 + assert supplement_workload(tmp_path / "nope", []) == 0 + + +def test_the_wizard_hands_run_the_sources_folder_and_supplements(monkeypatch, tmp_path): + """`provided=None` was hardcoded, so a wizard user could not use a sources + folder at all — the flag existed and the guided path could not reach it.""" + import inspect + + from papertrace import wizard as wiz + + src = inspect.getsource(wiz.run_wizard) + assert "provided=None" not in src, "the sources folder is still hardcoded away" + assert "supplement=None" not in src, "supplements are still hardcoded away" + + +# --- the wizard asks before it interrogates -------------------------------- + + +class _Answers: + """Stands in for rich's Prompt/Confirm, replaying scripted answers.""" + + def __init__(self, confirms, prompts=()): + self.confirms, self.prompts = list(confirms), list(prompts) + self.asked: list[str] = [] + + def confirm(self, text, **kw): + self.asked.append(text) + return self.confirms.pop(0) + + def prompt(self, text, **kw): + self.asked.append(text) + return self.prompts.pop(0) if self.prompts else kw.get("default", "") + + +def _script(monkeypatch, answers): + from papertrace import wizard as wiz + + monkeypatch.setattr(wiz.Confirm, "ask", staticmethod(answers.confirm)) + monkeypatch.setattr(wiz.Prompt, "ask", staticmethod(answers.prompt)) + return answers + + +def test_saying_no_to_sources_asks_for_no_path(tmp_path, monkeypatch): + """A user with nothing was made to read two paragraphs and answer a path + prompt to say so.""" + from papertrace import wizard as wiz + + a = _script(monkeypatch, _Answers(confirms=[False])) + assert wiz._ask_sources(tmp_path / "case") is None + assert not a.prompts and len([q for q in a.asked if "folder" in q.lower()]) == 0 + + +def test_saying_no_to_supplements_asks_for_no_path(tmp_path, monkeypatch): + from papertrace import wizard as wiz + + a = _script(monkeypatch, _Answers(confirms=[False])) + assert wiz._ask_supplements() == [] + assert len(a.asked) == 1 + + +def test_the_sources_question_defaults_to_yes_when_the_folder_has_pdfs(tmp_path, + monkeypatch): + """The folder's contents are better evidence of the answer than a fixed + default — and a user whose PDFs are already in place pressing return should + not silently skip them.""" + from papertrace import wizard as wiz + + seen = {} + + def confirm(text, **kw): + seen["default"] = kw.get("default") + return False + + monkeypatch.setattr(wiz.Confirm, "ask", staticmethod(confirm)) + case = tmp_path / "case" + (case / "sources").mkdir(parents=True) + + wiz._ask_sources(case) + assert seen["default"] is False, "an empty folder must not suggest yes" + + (case / "sources" / "pyrros-2023.pdf").write_bytes(b"%PDF") + wiz._ask_sources(case) + assert seen["default"] is True + + +def test_saying_yes_still_reaches_the_path_prompt(tmp_path, monkeypatch): + from papertrace import wizard as wiz + + case = tmp_path / "case" + (case / "sources").mkdir(parents=True) + (case / "sources" / "a.pdf").write_bytes(b"%PDF") + _script(monkeypatch, _Answers(confirms=[True], prompts=[str(case / "sources")])) + assert wiz._ask_sources(case) == case / "sources" diff --git a/tests/test_version_flag.py b/tests/test_version_flag.py new file mode 100644 index 0000000..25bb6a9 --- /dev/null +++ b/tests/test_version_flag.py @@ -0,0 +1,101 @@ +"""`papertrace --version` — the first thing anyone types after installing. + +It used to answer `No such option: --version`, so a user who had just installed +from a branch had no way to confirm what they got. The only route was +`python -c "import papertrace; print(papertrace.__version__)"`, which nobody +guesses. + +It is eager on purpose: a bare `papertrace` on a TTY starts the guided wizard, +so a `--version` resolved after the callback body would answer the question by +interrogating the user about their manuscript. +""" + +import re +import sys +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from papertrace import __version__ # noqa: E402 +from papertrace.cli import app # noqa: E402 + + +def _plain(output: str) -> str: + """`output` with the styling removed — what a reader actually sees. + + Every assertion on rendered CLI text goes through this. Rich styles pieces + of a token independently when colour is on, and colour depends on the + environment: `--version` comes out as `-` + `-version`, and `0.6.0` as + `0.6` + `.0`, so a substring assertion on the raw bytes is really a test of + whoever ran it. One such assertion passed on a laptop and failed on all + five CI pythons. + """ + return re.sub(r"\x1b\[[0-9;]*m", "", output) + + +@pytest.mark.parametrize("flag", ["--version", "-V"]) +def test_it_prints_the_installed_version_and_exits_cleanly(flag): + res = CliRunner().invoke(app, [flag]) + assert res.exit_code == 0, res.output + assert __version__ in _plain(res.output) + + +def test_the_version_it_prints_is_the_package_version_not_a_literal(): + """A hard-coded string here would drift from `__init__.py` at the next + release and answer confidently wrong — which is the whole failure mode this + codebase is built to refuse. `docs/RELEASING.md` names exactly one home for + the version; this must read from it.""" + from papertrace import cli + + src = Path(cli.__file__).read_text() + assert "__version__" in src, "cli.py must read the version, not restate it" + + +def test_it_does_not_start_the_wizard(monkeypatch): + """Eagerness is the point. `--version` on a terminal must not walk the user + through a setup interview before answering.""" + from papertrace import wizard + + started = [] + monkeypatch.setattr(wizard, "run_wizard", lambda: started.append(1)) + res = CliRunner().invoke(app, ["--version"]) + + assert res.exit_code == 0, res.output + assert started == [], "--version started the wizard" + + +def test_it_is_advertised_on_the_help_screen(): + """Asserted on the text a READER sees, with the styling stripped. + + Not a nicety. Where colour is enabled — every CI runner sets `FORCE_COLOR` + — rich's help highlighter emits the option as + `\x1b[1;36m-\x1b[0m\x1b[1;36m-version\x1b[0m`: the two dashes are styled + apart, so the literal `--version` is nowhere in the bytes. Asserting on the + raw output tested the colour support of whoever ran it, and passed on a + laptop while failing on all five CI pythons. + """ + plain = _plain(CliRunner().invoke(app, ["--help"]).output) + assert "--version" in plain, plain + + +def test_a_bare_invocation_still_reaches_the_wizard(monkeypatch): + """The guard: adding an eager option must not break the first-run path it + was added beside.""" + import types + + from papertrace import cli, wizard + + # patch cli's own `sys` reference, not `sys.stdin`: CliRunner replaces the + # real stdin inside `invoke`, so a patch on the module attribute is undone + # before the callback ever reads it + monkeypatch.setattr(cli, "sys", types.SimpleNamespace( + stdin=types.SimpleNamespace(isatty=lambda: True))) + started = [] + monkeypatch.setattr(wizard, "run_wizard", lambda: started.append(1)) + res = CliRunner().invoke(app, []) + + assert res.exit_code == 0, res.output + assert started == [1], "a bare papertrace no longer opens the wizard" diff --git a/tests/test_wizard.py b/tests/test_wizard.py index 464a3c1..4fc8a48 100644 --- a/tests/test_wizard.py +++ b/tests/test_wizard.py @@ -367,7 +367,15 @@ def f(*a, **kw): seen[name] = (a, kw) return f - for stage in ("ingest", "refs", "scout", "check", "highlight", "report"): + # `ingest` and `refs` are split into a Typer command plus a `_..._pipeline` + # function, the plain function `run` actually calls — see cli.py's comment + # on `run()`. The rest are still Typer commands called directly, + # keyword-only by convention. + monkeypatch.setattr(cli, "_ingest_pipeline", spy("ingest")) + monkeypatch.setattr(cli, "_refs_pipeline", spy("refs")) + monkeypatch.setattr(cli, "_report_pipeline", spy("report")) + monkeypatch.setattr(cli, "_check_pipeline", spy("check")) + for stage in ("scout", "highlight"): monkeypatch.setattr(cli, stage, spy(stage)) monkeypatch.setattr(cli, "_guard_case", lambda case, manuscript: None) @@ -619,3 +627,38 @@ def test_the_printed_estimate_does_not_promise_a_ceiling_it_can_exceed(): assert "up to [bold]{w['model_calls']}" not in src, ( "the base estimate is still presented as a ceiling" ) + + +def test_the_wizard_passes_run_every_parameter_run_declares(): + """`run_wizard` calls `cli.run` — a Typer command — as a plain function, so + every parameter it omits arrives as a truthy `OptionInfo` rather than the + default the help screen shows. + + This is the third appearance of that bug class and the first at this edge. + Adding `--format` to `run` left the wizard's call short by one, and an + `OptionInfo` reaching `write_reports` is not iterable — so a wizard-driven + audit would have crashed at the report stage, after every paid model call + had already been made. + + Asserted against the signature rather than against a list of names, so the + next parameter added to `run` is caught without anyone remembering to come + back here. + """ + import inspect + + from papertrace import cli, wizard + + declared = { + n for n, p in inspect.signature(cli.run).parameters.items() + if p.kind is not inspect.Parameter.VAR_KEYWORD + } + src = inspect.getsource(wizard.run_wizard) + call = src[src.index("run_cmd("):] + call = call[: call.index(")\n")] + passed = set(re.findall(r"(\w+)\s*=", call)) + + missing = declared - passed + assert not missing, ( + f"run_wizard's call to cli.run omits {sorted(missing)} — Typer will " + "supply an OptionInfo sentinel for each, not the documented default" + )