0.6.0 — supplemental material, and a provided PDF identified by what is in it - #3
Merged
Conversation
Every supported, partial and contradicted judgement is now validated against the cited source's own source_map.json: the page must exist, source_block is required rather than optional, and it must sit on the page the verdict names. Anything else is unchecked with a note. The block is what makes the evidence image unconditional — crop_for_anchor takes its crop region from the block's bbox, so a valid block always produces a picture and the anchor phrases only decide whether a red box is drawn on it. highlight enforces the same rule against reality, since a source PDF can be missing from sources_resolved/ and a source map can disagree with the PDF it was built from. A substantive judgement that produced no image is downgraded there too, so the rule holds by construction rather than by inference. The console stops calling an unsearched anchor "not found on the page". CHECK_PROMPT already asked for source_block, and already told the model to omit it only for not_addressed, so no prompt text changed and eval runs stay comparable across this release. Counts do change: a verdict that rested on page-only provenance now reports a gap. The coverage and multisource fixtures grow a source map because a source without one can no longer be judged at all.
_guard_case only ever ran inside refs, so two routes reached a case's manuscript ingest without it. papertrace ingest never consulted the guard at all: a paper whose stem is literally "manuscript", or any --out naming that path, overwrote <case>/ingest/manuscript — the slot refs fills and coverage_audit reads — while refs_manifest.json still described the first paper. The guard now runs whenever the output is that slot, recognised by shape so --out cannot walk in behind -c. A cited source ingested into <case>/ingest/<slug> is untouched, because check does exactly that. refs --parse-only on a pre-hash case re-ingested the manuscript slot and then returned before writing the manifest, leaving the source map describing one paper and the manifest another. "List references, no network" is an inspection, so it reads the paper into a temporary directory and mutates nothing. The legacy-case warning promised a manifest rewrite that ingest does not do; it now says only what both callers actually do.
Claims whose headline is not_retrieved or unchecked are routed out of the main loop into the gap sections, which printed the claim text and nothing else. So a claim citing [1,2] where source 1's check failed and source 2 was never obtainable said neither thing, and a not_addressed from a source that WAS read disappeared behind the unchecked headline that outranks it. All three formats now render the co-citation breakdown, the unretrieved co-citations and one row per judgement with its note. The editor look also labelled a whole section row with items[0].verdict, calling a mixed section whichever verdict came first; it is one row per claim now. anchor_located is True, False or None — searched and located, searched and not located, never searched. Both disclosure helpers gated on evidence_image, so a verdict carrying a page but no crop disclosed nothing at all while the report still printed the page as provenance. They gate on provenance now, and ANCHOR_NO_IMAGE carries the same three tokens with wording that does not describe a picture nobody wrote. A claim with no page still says nothing: silence about nothing is not a dropped disclosure. The parity test gains a gap claim and an image-less anchor, so the token loop covers both branches rather than only the one a crop reaches.
Eligibility was decided after alignment, so a case that could never be scored still competed for predictions — and a prediction is consumed once. An unresolved gold case sitting on the same citation label as an eligible one took its match, and the eligible case was then reported as the tool's extraction gap. Blame moved off the tool and onto the harness grading it, silently. score() now aligns against the eligible cases alone, and excluded cases keep their row and their reason. They also stop voting in repeated-run agreement: a case nobody could score is not evidence of the model disagreeing with itself. Duplicate prediction ids were collapsed by a dict comprehension that kept whichever came last, so input order decided which of two same-id claims was graded — the one place this module's documented order-independence did not hold. Refused with an error naming them, because the harness cannot know which was meant. require_one_set_id checked a third of what its own error message claimed. Agreement is defined within one (set_id, prompt fingerprint, ingest converter) triple, and all three are checked now: two runs that read different text, or answered different prompts, are two systems, and their difference is not instability. The intersection figure was labelled the upper bound. It is not one — dropping a case whose true agreement is high pulls the mean down, so with three or more runs it can sit either side of the truth. The two figures are complete-case (a population) and penalized (a genuine lower bound). not_addressed also gains its confusion-matrix column; the arithmetic always had four classes and the table printed three, so a mistake was counted and then hidden.
The README is a correctness surface, so these are defects like any other. Page-level provenance was claimed "for every verdict" — false for not_addressed, which has no decisive passage by design, and understated for the rest, which now carry a block as well as a page. An unmatched anchor was said to be "shown unboxed", which only holds when a valid block supplied the crop region. not_addressed was folded into "the most adverse verdict any of them gave"; it is deliberately unranked and becomes the headline only when no source addressed the claim at all. Output was still documented under case/, which stopped being the default when audits moved beside the paper. Text drawn inside a figure was said to be in the PDF's text layer, which a raster figure is not — contradicted four paragraphs later by this section's own measurement. And both Quick Starts ran pip install -e . with no clone, which cannot work while PaperTrace is not on PyPI. CLAUDE.md repeats the headline rule, so it gains the same correction plus the new provenance requirement. The changelog opens a [0.4.1] section rather than folding into [0.4.0]: that one is released history now. It leads with the behaviour change, because runs that previously reported a verdict on page-only provenance will report a gap.
The first real audit reported 46 references on a paper citing 43. The three extras were the paper's own table captions, and by the time they reached the report they had real DOIs attached and were labelled paywalled — the manifest asserted three works that do not exist. references_span scanned to the end of the document for any run of blocks sharing the bibliography's block type, testing nothing about the text, so three list blocks under a TABLE TITLES heading joined the list. A candidate run must now be at least half reference-shaped. Half rather than all, because a genuine continuation can carry a bare-URL entry with no year — the case the resume feature exists for. The resolver is the second line, because the parser will be wrong again. Crossref answered a title search for "Table 1. Dataset characteristics" with 10.7717/peerj.7892/table-1, a table belonging to an unrelated paper, and nothing caught it: the title sanity check only runs on the download path and no copy was ever downloaded. Entries that read as nothing citable are refused before the search, mirroring the web-page gate directly above, and a DOI naming a table, figure or supplement is rejected wherever it came from. looks_like_reference accepts an author list as well as a year, DOI or arXiv id. That clause is load-bearing: re-running the failing paper showed two real references arriving truncated mid-title with no year, whose correct DOIs Crossref had been finding from the author string alone. A year-only test made them gaps — the silent failure, and the worse one. Accepted downloads also keep the evidence for their title check, which until now was recorded only when the check failed.
With --doi supplied and no Europe PMC record found, the scout answered "paper not identified in Europe PMC — pass --doi to pin it". The operator had just passed --doi, so they went and curled Europe PMC by hand to establish what the tool already knew: the paper is a Journal Pre-proof and is not indexed. A DOI lookup that returns nothing is a different and stronger fact than a title heuristic that missed, and it changes what the reader should conclude — both registers are empty for want of a starting point, which is absence of data rather than a clean literature search. The message now branches, and the DOI that was tried is recorded in scout.json, which had been writing an empty string and leaving the null uninterpretable from the artifact alone.
Folded into the existing [0.4.1] section — that version is still unreleased, so these belong with it rather than opening another heading.
A live audit of "Image registration improves inter-reader agreement of objective response in CT assessment of pancreas adenocarcinoma" searched Europe PMC for `image AND registration AND improves AND inter-reader`. Two faults in one line: `improves` is a verb carrying no topic, and taking the first four content words never reaches a subject that sits at the end of the title. The register came back with a stroke conference abstract matching on IMPROVES. Words are ranked by length now — one rule, a proxy for specificity — rather than by position, and words that state what a paper claims rather than what it is about join the stop list. The same title yields `adenocarcinoma AND registration AND inter-reader AND agreement`. All fifteen "existed but uncited" candidates were from the paper's own year. That register asks the reader what the authors missed, and a same-year paper may have appeared after submission, so it cannot answer that. It is not newer literature either. same_year is a third register, following the rule coverage already uses for `uncertain`: a third status is never folded into either neighbour. Dropping them instead would lose a real finding, since a paper published early in the same year is exactly what a reviewer might raise. Additive in the schema; an older scout.json still loads. Europe PMC escapes the markup in its titles, so `CTV<sub>boost</sub>` arrived as `CTV<sub>boost</sub>` and was rendered verbatim. Decoded where every hit is built.
select_autoescape(["html"]) matches a template name ending in `.html`. These are `report_editor.html.j2` and `report_terminal.html.j2`, so nothing matched and autoescape was off for all three formats — including the two that emit HTML. disclosures.py reasons about the setting in its own docstring, which is how long it went unread. It stayed invisible because the one field carrying angle brackets is a Europe PMC title, and the API pre-escapes those. Decoding them in the commit before this one is what made it reachable. Cited source PDFs are downloaded from third parties and their text reaches the report, so this is not hypothetical. Matched on `.html.j2` now, via an explicit predicate rather than a helper whose matching rule has to be remembered. No interpolation is meant to emit markup — there is no `|safe` in any template — so escaping all of them is the whole fix. Markdown is not HTML and stays verbatim.
Still [0.4.1] and still unreleased, so they belong in that section.
`parse_references` was the one stage with no honest-degradation state: it always returned a confident list and nothing ever checked it. A live audit misnumbered 27 of 41 references — docling stripped the `[N]` numerals and a running header split reference [14] across a page break, so every label from [15] shifted by one. The label is the join key, so those claims were judged against the wrong papers, confidently. Two independent readings are now taken — the parse of the printed list and the list the publisher deposited with Crossref (`refs --doi`, defaulting to the DOI printed on page 1) — and the manuscript's own `[N]` markers arbitrate. A reading is used only if it accounts for exactly the labels the body cites. Failure is disclosed, never fatal: the manifest records which reading was used and from which entry the numbering is in doubt, and every claim citing a doubtful label carries the caveat beside its verdict in all three formats. Crossref is a candidate, not an oracle. A deposit this tool can only partly read is set aside rather than used to renumber, and the shortfall is named as this tool's own. No DOI, no deposit and Crossref unreachable stay three distinct facts. Two pre-existing bugs the work exposed are fixed with it: `_slug` fell back to a leading list numeral, so 23 of 28 references slugged `ref-2024` and eleven retrieved sources overwrote one file; and Typer's unpassed-option `OptionInfo` is truthy, which had the offline suite making live Crossref calls and passing. Measured on seven papers across four publishers: 7/7 deposit a list, and the check catches both known parse inflations (43 vs 41, 106 vs 101). Three of the seven cite by superscript numeral, which flattens to prose — those have no arbiter and are reported unconfirmed rather than presented as checked.
`check` reuses an existing `ingest/<slug>/` when `annotated.md` is there, and the slug naming that directory is not a stable identity — `_unique_slugs` renames one of two colliding entries, and the reconciler can choose the publisher's list on one run and the parse on the next. A re-run could hand the model the directory's previous occupant and judge a claim against a different paper, with no error anywhere. `SourceMap.doc` cannot see it: every cited source is stored as `<slug>.pdf`. Source maps now carry `source_sha256`. A directory whose recorded hash does not match the file at `pdf_path` is re-ingested, and an unhashed map counts as stale — re-ingesting is free, while trusting it is a guess about which paper is in a file. Additive: older maps load, absent meaning unknown, never "matches".
`deposit_is_this_paper` was written, documented and unit-tested, and nothing called it — so the check passed while the pipeline it was meant to guard ran without it. The DOI is typed by hand or read off page 1, and the deposit is the one retrieval route in this module that can replace the *entire* reference list: a companion paper, an erratum or an earlier version can carry exactly as many references as the body cites, `_covers` passes, and the run prints "✓ numbering confirmed" over another paper's bibliography. Every other retrieval route here has been title-checked since a wrong download was judged as a source. Wired into `refs`, tri-state, and pinned at the CLI level rather than on the function — the unwired guard is exactly what a unit test cannot see: - mismatch: the deposit is set aside, and the note names the DOI and the title it actually resolves to; - confirmed: the list is used, and the note says so on whose authority; - too little title to compare: the list is used and the identity is disclosed as unconfirmed. `deposit_is_this_paper` now applies the same `_TITLE_MIN_MATCHES` floor `_title_check` uses, because `paper_title` is a heuristic over the first blocks of a page and a journal banner must not read as a mismatch. The title rule moves to `models.paper_title` — third rule to live there for the reason the other two do: `scout` and `refs` both need it and neither may import the other. `scripts/reference_audit.py` applies the same gate and reports the identity, since a diagnostic measuring a different pipeline measures nothing.
… its bibliography Wiring the deposit identity check exposed that the title it rests on is guesswork. Measured on seven papers, four publishers: the layout heuristic returns the article-type banner every time it is wrong — `CLINICAL GUIDELINE` (Wiley), `RESEARCH ARTICLE` (Springer), `Journal Pre-proofs` (Elsevier), `Editorial` (AMA) — so the gate settled 3 of 7 identities and reported the rest unverified. Docling offers nothing here: on the AMA paper it emits no `title` item at all and labels the real title `section_header`, behind the banner. Two signals, both already in hand: - Source maps record `declared_title`, the title the PDF states in its own metadata. It is exact for six of the seven, costs one pymupdf read, and is read for both backends because it is a property of the file. Not trusted blindly: a declaration that is too short, ends in a document extension or carries a `Microsoft Word -` prefix is passed over for the layout, because an author's PDF declares the file it was exported from and an author's PDF is this tool's main case. - Where the titles cannot be compared, the paper's own bibliography identifies it. 38 of 41 deposited works appear in the printed list of the audited paper; 0 of 41 appear in a different paper's list. Set membership, never position — the same pair scores 34% in order, because that paper's parse is the misnumbered one this feature exists to catch, so the numbering cannot be an input to the identity test. Agreement is evidence of identity; disagreement is not evidence of difference, since two lists that disagree may be one paper read badly, so a low overlap leaves the identity unconfirmed and never calls the record another paper. Result across the spread: 7 of 7 identities settled, six by title and one by bibliography, where three were settled before. `scout` shares the title rule, so its Europe PMC lookup stops searching for "RESEARCH ARTICLE"; verified on the audited paper, which it now identifies by name rather than as "Journal Pre-proofs". `declared_title` is additive and older maps load without it, where absent means not recorded rather than absent from the paper. The DOI provenance claim in the note is dropped rather than left wrong: `run` detects the DOI and hands it to `refs`, which cannot tell that from a DOI the user typed, and the note said "(given)" for a DOI nobody gave.
`coverage_audit` cut the body at `^##\s+(references|bibliography|literature)`, a second boundary rule beside `models.is_references_heading` — while the comment above that shared rule says two readers need one rule because having two is a defect this codebase already shipped. It shipped again. The regex needs ingest to have *typed* the block as a heading, and flat-text ingest guesses headings from font size: a `References` line at body size stays a text block and reaches `clean.md` with no `##`. The shared rule exists for exactly that case and returns True where this cut returned False. Reproduced on a generated flat-ingest paper — `labels_in_text` came back `['1','2','3']` where `[3]` appears only inside the reference list, so the audit reported a gap that does not exist, in the one figure it computes mechanically so that it cannot. Both the label reading and the `clean.md` occurrence fallback now cut on the shared rule, applied per line: a markdown-marked line is read as a heading, so `## References and further reading` still cuts, and a plain line must be the word and nothing else, which is what keeps `References were checked by hand [1].` from swallowing the paper. One existing test asserted the defect as evidence that the occurrence walk's structural test did independent work. It does, and the two readings agreeing is the point of the fix, so that assertion is inverted with the reason written down.
… claimed `_numbering` renders "every entry is affected" whenever `unverified_from` is None, and `label_is_doubtful` returned False for every label for exactly the same reason. The report's most severe warn-level disclosure asserted that every entry was suspect while marking no claim suspect — and the per-claim layer exists precisely because a banner is not where someone acting on one verdict is looking. An unconfirmed numbering with no recorded scope now puts every label in doubt. Two shapes reach that state: a manifest written before the list was reconciled at all, and two readings that agree entry for entry with no arbiter to confirm either. Neither establishes *which* entries are wrong, and unknown scope has to read the same way in both places or in neither. A located divergence keeps its scope — [1]-[14] stay untainted when the readings part at [15] — and a confirmed numbering still taints nothing; both pinned, so the fix cannot decay into "taint everything always". Consequence worth naming: the superscript-citation papers, about half of a real spread, reach the no-arbiter state, so their reports now carry the caveat on every claim.
Two entries colliding on `smith-2019` leave the second slugged `smith-2019-r7`, and `_provided_candidates` drops tokens of three characters or fewer — so the only thing telling them apart was invisible and one `sources/smith-2019.pdf` matched both. Measured before the fix: entry [7] came back `status=provided`, `title_check=mismatch`, `pdf_path=smith-2019.pdf`, and every claim citing [7] would have been judged against entry [2]'s paper. The title check caught the wrong paper and the run used it anyway. The rule now turns on who chose the file. A file named for the reference's own slug keeps the documented behaviour — a failed check is disclosed, never fatal, because the user chose it, there is nothing to fall back to, and a scanned PDF yields no text at all. A file a token match found was chosen by nobody, so a check that says "different paper" is a reason to keep looking: candidates are read in rank order until one is usable, and the retrieval chain runs as if the folder held nothing. Where the chain then finds nothing, the reason names the file that was set aside and why, rather than reporting a bare gap while knowing more. The chain moved into `_resolve_by_retrieval` so that could be said once instead of at each of its nine exits.
…y found it `_resolve_paper` records `via: doi` whenever a DOI is supplied, and the console warned "wrong paper? pass --doi" only on `via: title`. That was a fair proxy while a DOI meant the user had typed one, and stopped being one when `run` began reading the DOI off page 1: a funder, data-availability or erratum DOI now anchors the whole scan to another paper *and* suppresses the only signal that it did. The provenance is not recoverable inside `scout`, and it is the wrong question — the record's own title is comparable with the paper's. `ScoutResults.identity` is `confirmed` / `unverified` / `mismatch`, additive, and `""` on an older artifact means not recorded rather than confirmed. A mismatch stops the scan: the registers ARE the finding, so they are not built from a record this tool can see is not the paper, and the reason names the title it actually resolved to. Too little title to compare leaves the scan in place and discloses the unknown — the same tri-state as a deposit and a downloaded source. The comparison moved to `models.titles_match` with `_title_tokens` and the `_TITLE_MIN_MATCHES` floor. Three readers need it now — a Crossref deposit, a Europe PMC record, a downloaded first page — and a copy in `scout` is the defect that module's other shared rules exist to prevent. `refs.deposit_is_this_paper` delegates to it and keeps its own docstring, so its tests still pin the domain question rather than the arithmetic.
A results table's 95% CI column is written [100, 100], [51, 85] — the same bracket-and-comma syntax as a citation group [7,8]. Reproduced on a real radiology paper: two table blocks supplied every square-bracket match in the manuscript, none from prose, and pushed the reconciler's read of the highest cited label from the real count to 100. _body_citation_labels, citation_occurrences and citation_labels_in_text now skip table content — by block type where a source map is available, by each row's own GFM `| ... |` shape in the clean.md fallback, since flat text carries no block type.
check/highlight/report/scout take no manuscript path, so the hint only ever checks the current working directory. Saying "no case folder found here" reads as an exhaustive search; reworded to name the one directory actually checked.
…olset _ask passed no cwd to claude -p and no tool restriction. Running an audit from inside a repo silently fed that repo's own CLAUDE.md into every verdict, and the judge held Bash/Edit/WebFetch while it only ever needs to read a prompt and answer. Now runs with --safe-mode, --tools "" and cwd set to a neutral directory.
init then run paper.pdf used to orphan case/sources/, since run/refs name their own folder after the paper and a hand-made ./case/ is only reused if -c is remembered every time. init --for paper.pdf now names the folder exactly as default_case would, so a plain follow-up run finds it automatically. An explicit folder name still wins over --for.
Flagged by automated security review of the prior commit: cwd was set to tempfile.gettempdir() directly, a predictable, shared location another local user could plant config into. Now a private 0700 directory, created once per process and reused across calls.
…ed as ingest() was both a Typer command (whose declared defaults are OptionInfo sentinels) and the plain function run() and tests called directly by keyword. That shape already shipped one bug (backend became an OptionInfo, every audit silently ingested as flat text). _ingest_pipeline now holds the logic, keyword-only with ordinary Python defaults; ingest() is a thin Typer wrapper over it, and run() calls _ingest_pipeline directly. A positional call now raises TypeError immediately instead of taking a wrong default. First of six stages; refs, scout, check, highlight and report still go through their Typer command by convention only.
Same shape as the ingest split, and the more load-bearing one: refs() doubled as a Typer command and the plain function run() and 22 direct test calls used, all of which omit --doi and so silently received a raw OptionInfo instead of None, caught only by _text_opt. _refs_pipeline now holds the ~185-line body, keyword-only with ordinary Python defaults; refs() is a thin wrapper over it, run() and every test call it directly. _text_opt is deleted — it existed solely to guard doi against exactly this shape. Second of six stages; scout, check, highlight and report still go through their Typer command by convention only.
A gold benchmark and a GROBID swap were both put up by a full-stack review and both are declined. docs/adr/ now exists so neither gets re-derived from scratch by the next review. The benchmark's blocker is not its design — evals/PROPOSAL.md already specifies it down to the acceptance criteria, and the harness has been waiting for data, not for a plan. It is the labelling precondition: DESIGN.md wants two labellers who did not write the prompts. Building it self-labelled would produce a figure the scorer itself prints a conflict-of-interest caveat against. GROBID would displace ~261 of the ~707 lines it appears to target; the deposit, corroboration and reconciliation code is there because any parse can be wrong, and a parser cannot certify itself. Java, Docker and 2-4 GB against that is not a trade, so the benchmark is skipped too — it could not change the answer. The proposal text stays. It is the plan if a second labeller ever exists.
A claim citing four references gets four verdicts, and the most adverse of them becomes the headline. That is the right triage signal and it stays. But printed bare, "CONTRADICTED" reads as a statement about the claim, when a compound sentence may draw different parts from different references quite legitimately. So a multi-source headline now says what it ranged over: most adverse of 4 cited sources. Single-source claims are not qualified, because there the headline is the claim's verdict; nor is a claim with no judgements, which ranked nothing at all. No new verdict value. A `disputed` state would have cost a VERDICTS entry, a schema change, a gold enum change and six render sites to say what the per-source breakdown beside it already says. The totals still count each claim once, under its headline, so the contradicted count means "claims with at least one contradicting source". The does-not list now says that rather than leaving it to be discovered.
Three report files and a megabyte of fonts were written whether or not anyone wanted three. report.md is the record and is still always written; the editor and terminal looks are for sharing and screenshots, so they now take --format (-f, repeatable). --png pulls in the HTML it photographs, since honouring "--png --format md" literally would render nothing and explain nothing. A mistyped format exits 2 before results.json is even opened. write_reports keeps defaulting to all three looks: it is the seam the parity suite drives, and narrowing it there would have left that suite green while it quietly stopped comparing formats. The policy belongs to the CLI. Adding the option exposed a live bug one level up. run_wizard calls cli.run as a plain function, so the parameter it did not name would have arrived as an OptionInfo, reached write_reports and raised on not being iterable — after every paid model call had been made. report is therefore split into its Typer command and _report_pipeline, and the wizard now names every parameter run declares. The new test asserts that against inspect.signature rather than a list of names, so the next one is caught without anyone remembering.
Extraction was asked for each claim "tightly paraphrased, <=160 chars", and the judging prompt got that paraphrase and nothing else. The population, the effect size, the interval and the hedging are what decide whether a citation supports a statement, and they are the first things 160 characters lose. So extraction now returns the manuscript's own sentence verbatim alongside the paraphrase, and the judging prompt says to rule on the sentence. The paraphrase stays because a headline reads better for it. The quote is never back-filled from the paraphrase, which would put the compression back while looking fixed. A claim judged without a quote discloses that in all three formats. Weaker evidence, said out loud rather than inferred from a missing blockquote, and fired only where a judgement happened so it stays off the gap register. Coverage attribution improves for free: the ratio is now taken on the quote, comparing a sentence with a sentence. Both thresholds stay put, because the evidence under them changed and moving them too would confound the two. Unmeasured, per ADR 0001. It removes a known information loss, which is not evidence that verdicts got better.
…optional check.py hard-coded backend="pymupdf" for every cited source and explained itself: layout fidelity is spent on the audited paper, not its sources. That had the asymmetry backwards. The manuscript's claim is the question; the source is the evidence, and the evidence for a subgroup claim is usually a table row. Read flat, the row is gone. So sources now take the same backend as the paper. check gains --backend, run forwards its own, and check_claims takes it as a required keyword with no default, like _clip's accumulator: "auto" would drag docling into an offline test run and "pymupdf" would silently downgrade a caller who asked for layout. docling therefore moves from an extra into the base dependencies; [docling] and [full] stay as aliases so 0.4.x install lines still resolve. CI installs it and never runs it, because the layout models download on use and every test pins the flat backend. Two things this would have broken, both found by looking rather than by a test going red. _stale_ingest compared the PDF hash only, so an existing case folder would have reused its 0.4.x pymupdf source maps while the run claimed layout-aware ingest; it now compares the recorded converter through a shared resolve_backend. And the reports never said how the sources were read at all -- one dim terminal line, nothing in markdown or either HTML. Each source's converter now travels in RunResults.source_converters and a flat-read source is named by slug in all three formats. Measured, since it is a real slowdown: ~41s for the first docling ingest in a process and ~3s per source after, so under `run` the models are already warm from the manuscript. check is also split into its Typer command and _check_pipeline, for the OptionInfo reason ingest, refs and report already were -- the new --backend is exactly the kind of added parameter that has silently shifted arguments here twice. scout and highlight are the last two left on convention. Version bumped to 0.5.0: the default report output and the dependency matrix both change. CITATION.cff was still on 0.4.0 and is now consistent.
examples/demo/output/ is the artefact the README links as "See a completed report" and it was produced by 0.4.1, so it showed none of what this release changed -- no verbatim quote above any verdict, which is the visible headline of 0.5.0. The demo command now pins --model claude-opus-5. Without it `claude -p` takes the account default, and that default had quietly moved from opus to haiku between two regenerations: the showcase's judge depended on which day it was rebuilt, which is not a property a committed artefact should have. The pinned expectation moves to 1 supported / 2 contradicted / 1 not retrieved / 1 uncited, over 4 claims rather than 5. Every planted defect is still found. What changed is that the sentence citing both [2] and [3] now arrives as one multi-source claim instead of two single-source ones, because extraction is asked for the verbatim sentence. Reproduced on opus-5 and haiku-4.5 alike, so it is the prompt and not the model -- the first regeneration changed both at once and the cause stayed ambiguous until a second run separated them. Two README claims fall out of that: the counts, and the sentence asserting no demo claim cites more than one reference. The demo now does, so it exercises the per-source breakdown and the new most-adverse-of-N qualifier, which the old one never reached.
"Pulls torch" is not a number. A clean virtualenv measures 1.4 GB after pip install papertrace -- torch 591 MB, then opencv, transformers and scipy -- on top of the ~500 MB layout-model download on first use. That belongs in the install table, where someone reads it before typing the command, rather than being discovered afterwards. --backend pymupdf is named there as the escape hatch for a machine where it is unacceptable.
…after Extraction discarded the location and then worked to get it back. The model returned a paraphrase plus a free-text location, and Python guessed which of several [3] markers it had come from: normalise both sides, score with SequenceMatcher, accept only on ratio >= 0.45 and margin >= 0.10, assign globally best-first, and call everything it could not decide uncertain. The counts came out right and the pointer could be wrong. The inventory it was matching against had been there the whole time, built deterministically from source_map.json -- just after the model call instead of before it. Now _render_inventory puts every citation occurrence in the prompt as ctx_NNNN with its page, section, labels and sentence, each claim comes back carrying the ids it was taken from, and coverage is a dictionary lookup. One sentence citing [2] and [3] is one claim holding both ids. A ctx the inventory does not contain is dropped, never repaired into "the first occurrence of that label": a hallucinated ctx_9999 and an honest empty list say the same amount about which sentence was meant. uncertain therefore survives with exactly one cause left -- a claim cites a label and names none of that label's contexts, so a claim reached one of them and nothing can say which. Gone: _attribute_label, _normalize_for_match, _ratio, _location_matches and both thresholds, plus the unicodedata and SequenceMatcher imports. check.py is not smaller for it (110 lines out, 115 in, a third of those prompt text) -- what goes is a mechanism, not a line count, and the deliberate duplication with evals/align.py goes with it because there is no matcher on this side to duplicate. results.json gains ctx_ids per claim; the audit is coverage/3 and coverage/2 files still validate. The label-level keys are untouched, because evals/align.py reads missing to decide whether an unmatched gold case is our failure or the harness's. The report's attribution caveat is shorter and its token changed: naming the id is still a model step, so it can still name the wrong place. Unmeasured, per ADR 0001.
The first thing anyone types after installing, and it answered "No such option: --version". Found by installing this branch from GitHub into a clean virtualenv and typing it -- the only way to check was `python -c "import papertrace; print(papertrace.__version__)"`, which nobody guesses. A user who had just installed from a branch had no way to confirm which one they were running. Eager, so it answers before the callback body: a bare `papertrace` on a terminal opens the guided wizard, and a version flag resolved after that would have interviewed the user about their manuscript before telling them the number. Reads papertrace.__version__ rather than restating it, since docs/RELEASING.md names exactly one home for the version and a literal here would drift at the next release.
The committed report still carried the pre-item-5 attribution caveat -- "attribution is a text match that can be wrong", describing the similarity matcher that no longer exists. A stale disclosure in the repo's one committed artefact is the same class of defect as a stale README claim, so it is regenerated rather than annotated. The pin holds: 1 supported / 2 contradicted / 1 not retrieved / 1 uncited over 4 claims, opus-5, and every planted defect found. Coverage is now coverage/3 with 5 of 5 occurrences placed and none uncertain, and claim 3 carries two ctx ids -- the two labels in one sentence, returned by the extractor rather than guessed from a paraphrase. Note wording and the decisive passage differ from the previous run, as they do between any two runs of a model step. sudlow-2015's crop moved from page 1 to page 3, where the judge cited Table 3 by name.
Everything downstream already keys a judgement target off its slug — ingest/<slug>/, sources_resolved/<slug>.pdf, SourceJudgement.source_slug, RunResults.source_converters. Four call sites hand-rolled the lookup as `next(e for e in manifest.entries if e.slug == slug)`, which can only ever find an article. RefManifest.document() and documents() resolve any of the three kinds, so none of those callers has to learn supplements exist. _ref_entry_from replaces the bare RefEntry(**e) splat. It hydrates nested Supplements — the manifest has carried no nested dataclass until now, so asdict flattened them on the way out and nothing put them back — and it drops unknown keys instead of raising, which is how every other reader here already defaults forward.
_SUPPLEMENT_RE recognised these filenames only in order to discard them, so a user holding an appendix had no way to have it read. It still cannot stand in for the article — attach_supplements refuses unless the reference already resolved, which is the same rule _provided_candidates enforces and is what keeps a lone appendix reported as not_retrieved. The match rule now lives once, in _named_for; the article scan and the supplement scan are that question asked with the marker inverted. A second copy is how they would drift into attaching an appendix to the wrong paper. Slugs come from the file stem, never an ordinal. -suppl1/-suppl2 numbered in folder order is the shifting-id defect CLAUDE.md rejects for citation occurrences: delete one file and a stored verdict points at another PDF. orphaned_supplements names every supplement that attached to nothing, with the two reasons kept apart — the article is missing, or the name matches no reference. The fixes differ, and silence here is invisible to the user.
Repeatable, following --format. The cited side needs no flag: the sources folder is matched against reference slugs and the audited paper has none for a filename to key on, so it has to be named. Manuscript supplements claim their slugs before resolve_all runs, and the registry is handed to it, because both kinds land in ingest/<slug>/ and sources_resolved/<slug>.pdf — the paper's appendix sharing a slug with a cited source means one reading the other's pages. test_the_wizard_passes_run_every_parameter_run_declares went red on this commit, which is what it is for: run_wizard would have handed the new parameter an OptionInfo. It passes None here; the wizard learns to ask for supplements separately.
A supplement is part of the work that was cited, so it is read for every claim citing that label — not only when the article turns out to be silent. A supplement contradicting a claim the article supports is exactly the finding the cheaper rule would miss. The cost is one call per supplement, not per claim: the loop already groups every claim for a document. CHECK_PROMPT now says what the judge is holding. Handed an appendix unannounced it has no reason to expect not_addressed to be the ordinary answer, and reaches for partial instead — inventing a true kernel. SourceJudgement.kind is stored rather than looked up, because the templates get results alone and run_disclosures takes the manifest optionally: a reader holding only results.json still has to be able to tell a verdict resting on an appendix from one resting on the paper of record.
"Model AUC was 0.91 (Table S3)" named exactly where its evidence lives, and the tool had no way to look. Extraction now marks such a claim; check reads it against every supplement supplied with --supplement, the same judge-them-all rule the cited side uses, which also spares the extractor a guess about which file S3 is in. The pointer is not a citation and never enters refs — there is no label. When no supplement was supplied the claim is not_retrieved with the flag named, rather than being left in the uncited register: a statement whose evidence the paper located precisely is not an assertion made without one, and calling it that understates the gap.
One cited work read as its article plus a supplement is not two cited
works. Both places that counted them said it was: the headline qualifier
and the per-claim breakdown that sits right under it. Each now says
"documents" when a supplement is among them and keeps the old wording
when they really are all cited sources.
SourceJudgement.origin replaces `cited as [{{ j.ref }}]` written out by
hand in three templates. The paper's own supplement answers for no label,
so every one of them would have rendered `cited as []`.
Three disclosures. Two run-level: supplements carry no identity check —
a supplement's own title is not its parent's, so the check that guards
every cited source cannot apply and is not faked — and the coverage audit
reads the manuscript alone, so a label appearing only in a supplement is
not counted. One claim-level, where the headline is an appendix's word
while the article of record is silent: a run-level footnote does not tell
a reader that THIS verdict is one of them.
The wording of both identity notes is split by how the file was actually
obtained. Calling a file the user pointed --supplement straight at
"matched by filename" misstates which part is uncertain, and only the
cited side is a guess. Found by rendering a report and reading it; every
assertion passed while both sentences were false.
run_wizard hardcoded provided=None, so the guided path could not reach a flag the CLI has had all along: a first-time user following the wizard got open-access retrieval only, with nothing saying so. It now asks, defaulting to <case>/sources, and asks separately for the audited paper's own supplements — the sources folder matches against reference slugs and the paper under audit has none for a filename to key on. The cost line counts them. Each supplement is one more document and so one more judging call, and an estimate that leaves them out understates what the user is agreeing to pay before they agree to it. equivalent_command emits both, or the line it prints as "the same thing as one command" would not reproduce the audit it just ran.
README does/does-not gains both sides: supplements are read as their own documents, and neither their identity nor a citation appearing only inside one is checked. The --provided paragraph said a supplement-named file is left to the online resolver, which stopped being true two commits ago. /review SKILL.md line 125 claimed cited sources are always read as flat text. That has been false since 0.5.0 made docling a base dependency and gave the sources the paper's own backend — the skill was telling users their tables had been linearized when they had not been. Version 0.6.0 in __init__ and CITATION.cff; 0.5.0 was never released so its entries ship with these.
The label column already prints [1], so appending "· cited as [1]" beside it was noise on every row of an article-only run — and it would have made the committed showcase PNG stale for a change no reader benefits from. Supplements still say what they are, which is the case the line exists for. Verified: for the demo, which has no supplements, the rendered terminal HTML carries no supplement markup at all and the markdown structure is unchanged.
A publisher download is called s41467-023-39631-x.pdf and matches no reference, so the audit ran, looked entirely normal and used none of it. identify_by_content asks the question the filename matcher cannot: not "which files could be this entry" but "which entry is this file". Two signals, and deliberately NOT _title_check_text. That rule counts a reference's words anywhere on a whole page, which is right for vetoing a file the user already named and wrong for discovery: measured on the demo's real sources, a chest-radiograph paper verified against an NEJM review too, on current/future/interpretation/medical/images. Comparing title against title keeps the denominator small. Measured on the same three papers, renamed to publisher style, both DOI and title give a unique correct answer. A non-unique match is refused, never ranked. A corrigendum shares nearly every distinctive word with its original, and a best-score pick there judges a claim against the wrong paper with nothing able to notice. None from titles_match is not an accept either: a filename match carries the user's assertion, a content match has nothing behind it. orphaned_supplements widens to unused_provided. It covered supplements only, so an unmatched article PDF was dropped in silence while a supplement-named one was named — and a folder of publisher-named downloads produced an audit that mentioned none of it.
resolve_all runs inference once over the folder before the per-entry loop, withholding everything the filename rule could take — as an article or as a supplement — so the user's own naming always decides first. The reason names the signal. A DOI is exact and a title is a judgement over token overlap, and a reader weighing a verdict is owed the difference, so Identified carries it to the manifest rather than both arriving as "matched in your sources folder". unused_provided explains the leftovers itself instead of having them handed down, so one place answers why a file went unused. That also lets it name the realistic case where a user has both the reference-manager export and the publisher download of one paper: the spare says which reference it is and that another file got there first, rather than reading as unrecognised.
0.6.0 attached supplements on a filename alone, and the report said nothing verified them — true of every supplement then, and now true only of some. Supplement.verified records which: the file's own title or DOI named the work, or its filename carried the reference's tokens and nobody checked. This is what makes inference matter for supplements at all. The publisher forms carry no filename marker — \besm\b cannot match inside MOESM1_ESM, and mmc1 and media-1 say nothing — while their first page states plainly what they are. Same marker list, read from the file instead of its name. The 0.6.0 rule is unchanged: no article, no attachment. A supplement whose parent could not be obtained is still set aside and named.
`liu-2019` yields tokens ['2019'] — the len>3 filter drops the surname — so every 2019 appendix in the folder matched it. An article surviving that is caught by its title check. A supplement gets none, so smith-2019-appendix would have been judged as part of Liu 2019 with nothing able to notice. The rule requires the slug itself in the filename rather than refusing outright, which keeps the case it exists to serve: liu-2019-appendix.pdf still attaches. A supplement whose own title named the work never reaches the rule — it was verified, not guessed. Found while answering a question about how filename matching works, not by a test going red.
0.6.0's disclosure read "supplements carry no identity check". That was true of every supplement then and is true of only some now, so it became a falsehood about the ones whose own title named their parent. The token is now neutral and the split is in the text; the level drops to info when nothing was guessed. SourceJudgement.verified carries it, for the reason kind does: the templates get results alone and run_disclosures takes the manifest optionally, so a reader holding only results.json must be able to see which supplements were checked. _supplement_slugs went with it — the cited/own split it computed is no longer what the disclosure turns on.
A user with none was made to read two paragraphs and answer a path prompt to say so. Both questions now open with a yes/no, and the explainer moves behind it. The sources gate defaults to yes when <case>/sources already holds PDFs and no otherwise: the folder's contents answer this better than a fixed default, and a user whose files are already in place should not skip them by pressing return. The explainer also stops implying <firstauthor>-<year>.pdf is a required format. It is a fallback now — a publisher download is identified by its own title or DOI, and a file named for its reference is taken at the user's word instead. Both gates sit inside the helpers rather than in run_wizard, where an early-out would reintroduce the literal `provided=None` that test_the_wizard_hands_run_the_sources_folder_and_supplements forbids.
The README said --provided matches by filename and that files must contain the author and year. Both stopped being the whole truth two commits ago, and the sentence that mattered most to a user with a folder of publisher downloads was the one that was wrong. The does-not list gains the honest limits of the new path: identification needs a readable title or a first-page DOI, so a scanned copy is unplaceable, and a title matching two references is refused. Both are listed with the reason rather than skipped. The wizard alt text at README:250 enumerated the questions and had already missed two of them. docs/wizard.png itself is a hand-taken screenshot and still shows the old flow — it needs a manual re-shoot.
Two files can honestly identify as the same reference — a duplicate copy, or a full text beside a truncated one. setdefault over a sorted glob settled that by filename order, so a title match beat a DOI match and which file the audit used depended on what it happened to be called. That is the filesystem-order defect _provided_candidates already fixed once. A DOI is exact identity and a title is a token-overlap judgement, so the exact signal wins; name order breaks the remaining tie so the answer is the same on every machine. Found by running a folder of publisher-named downloads through the CLI: a 3 kB truncated copy of one source won [1] over the complete file, purely because "mystery-download" sorts before "s41467". Every test passed.
CI has never run on this branch — the workflow triggers on push to main and on pull_request only — so PR #3 was the first time these 751 tests met anything but one local 3.13. Three failed on all five pythons, both causes being the test reading its environment rather than the code. `_email` falls back to a saved config in the developer's home directory, so the two `run` delegation tests passed here and exited 2 on CI. They now monkeypatch `_email`: the address is not what they are testing, and it must not come from the machine. The help-screen assertion tested colour support. Where colour is on — every CI runner sets FORCE_COLOR — rich's highlighter styles the two dashes of `--version` separately, so the literal string is nowhere in the output. It now asserts on the text with styling stripped, which is what a reader sees. `_plain` is shared, because `0.6.0` splits the same way at `0.6` + `.0`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fast-forward of 53 commits.
feat/v0.6.0-supplementsfully containsmain,fix/v0.4.1-provenanceandfeat/v0.5.0-review— verified withgit merge-base --is-ancestor— andmainhas no commits the branch lacks,so those two branches need no separate merge and can be deleted after this.
This is the first CI run on any of these commits. The workflow triggers on
pushto main and onpull_requestonly, so 53 commits across three pushedbranches have never been tested outside one local Python 3.13.
compileallis clean on 3.10 and 3.14; the 751 tests have not run on either.
Ships 0.4.1, 0.5.0 and 0.6.0 together, as none was released.
CHANGELOG.mdsays so at the top of each section.
0.5.0 — a seven-item review, answered
Five items shipped, two declined with ADRs in
docs/adr/.doclingis a base dependencynow (~1.4 GB installed, stated in the README install table rather than
buried). A table row is usually what a subgroup claim turns on.
report.mdby default; the HTML looks on--format.coverage/3). The old flowdiscarded the location and then reconstructed it with
SequenceMatcher;the inventory it was matching against had been there all along, just built
after the model call. ~130 lines of similarity matching deleted — not a net
line saving, and the CHANGELOG says so.
not) and GROBID (ADR 0002).
0.6.0 — supplementary material as documents
A judgement target is a document, not a reference. Multi-source checking
already judged one claim against N documents, so a supplement enters as one
more: no new verdict, no new headline rule, no change to
coverage/3.--supplement(repeatable) for the audited paper's own.be obtained is named and set aside.
Table S3are judged against it, and arenot_retrievednaming the flag when none was supplied.0.6.0 — a provided PDF is identified by what is in it
--providedmatched on the filename, needing the surname and year in it.A reference-manager export has that; a publisher download never does — so a
folder of downloads produced an audit that looked normal and used none of it,
and an unmatched article was skipped in silence while an unmatched
supplement was reported.
Each unrecognised file is now identified from its own DOI, else its own
title against the reference list. Filename matching still runs first and
still wins: that is the user's assertion about the file.
Deliberately not reusing
_title_check_textfor this. Measured on thedemo's real sources it verifies
pyrros-2023.pdfagainst an unrelated NEJMreview too, because it counts a reference's words anywhere on a whole page.
It is a forgiving veto for a file already chosen, and it stays that.
Two refusals, both measured: a title matching two references is used for
neither (a corrigendum shares nearly every distinctive word with its original),
and a title too thin to tell papers apart is not a match at all.
unused_providedlists every PDF that ended up attached to nothing, with thereason kept apart — unrecognisable, ambiguous, a spare copy, or a supplement
whose article is missing.
Verification
uv buildproduces both 0.6.0 artifacts — all onlocal 3.13 only, which is what this PR is for.
1 supported · 0 partial · 2 contradicted · 1 not retrieved · 1 uncited,4 claims, coverage 5/5, 0 uncertain.
Two defects found by running it, not by a test
[1]over the complete file,because
mystery-downloadsorts befores41467andsetdefaultkept thefirst — alphabetical order was deciding the audit, and a title match was
beating a DOI match. Every test passed. Fixed: exact signal first.
"2 cited sources checked" for one work read as two documents, and "matched by
filename" for a file passed to
--supplementby hand.Known-stale, not fixed here
docs/wizard.pngis a hand-taken screenshot showing the pre-gate wizard flow.The alt text is corrected; the image needs a manual re-shoot.
Not claimed
No accuracy figure, here or in the README. Judgement quality remains
unmeasured — see ADR 0001.