v0.4.0 — honest degradation everywhere, a guided CLI, and every co-cited source read - #1
Merged
Merged
Conversation
…self The pipeline could emit a verdict it had no basis for. Five ways, all closed: - VERDICTS splits into JUDGMENT_VERDICTS + PIPELINE_STATES. A model answering "not_retrieved" for a source that was retrieved now yields `unchecked` with a note. Concatenated in the original order, so the wire format is byte-identical and no schema enum moves. - `_judgement_from` is atomic and total by construction: every branch is an isinstance test, so it cannot raise, and fields reach the claim only after all of them validate. A substantive verdict with no usable `source_page` becomes `unchecked` — the report used to print a literal "Page None" that no reader could falsify. Deliberately no per-claim blanket except: that would blame the model for our own AttributeError. - Truncation travels in a per-run `Truncations` object. The `_TRUNCATED` global is deleted outright, not shimmed, and `_clip` takes the accumulator as a required positional so no future call site can silently omit it. - Case identity is the manuscript's sha256, not its basename. Legacy manifests carry no hash, fall back to name comparison, and say the identity is unverified rather than asserting a match. - highlight bounds both `doc[page - 1]` sites. A cited page the source does not have used to raise IndexError out of the whole stage, losing every crop already written. The claim keeps `anchor_located = null`, never `false`: nothing was searched, so "looked and missed" would be a fabrication. Citation coverage is now per occurrence, not per label. Two sentences citing [3] with one extracted claim reported [3] as covered — set arithmetic, and the tool's single largest overstatement. `labels_in_text`, `covered` and `missing` keep their label-level meaning byte for byte, because evals/align.py reads `missing` to decide whether an unmatched case is the tool's failure or the evaluator's; everything occurrence-level is additive under "coverage/2". Attribution is location narrowing plus text similarity assigned globally best-first, accepted only on ratio >= 0.45 and margin >= 0.10, and an attribution the tool cannot make counts as uncertain — never as covered.
The three templates had drifted apart, and each gap let one format tell a reader something the others contradicted: - the editor report never mentioned `unjudged_refs` — the co-cited sources a batch-mode claim was never checked against. Zero occurrences, while markdown had two. - the terminal report captioned every crop "red box = matched text" unconditionally, including crops with no box at all. - `anchor_located` is a tri-state and the templates tested only `== false`, so `null` rendered as a located match. In Jinja `None == false` is false. - the editor's truncation warning was nested inside its coverage block, so a run with truncation and no coverage disclosed neither. `src/papertrace/disclosures.py` now defines each rule once, with a `token`: a short literal that must appear verbatim in markdown, editor HTML and terminal HTML. Each format still phrases at its own length; the parity test asserts the token. That turns "no format silently drops a disclosure" from a checklist into a loop — which is what was missing when the editor shipped with none of them. Not a shared Jinja macro: report.py sets autoescape for html only, so a macro file imported by both .md.j2 and .html.j2 escapes differently depending on the importer. A correctness hazard in the one surface whose job is not lying. Templates move to src/papertrace/templates/ and load through importlib.resources. They were read from a repo-relative path, so an installed wheel could not render a report at all — the repo was a hidden dependency of the shipped package.
v0.3.1's sdist carried exactly one file under evals/ — evals/README.md — while that very README instructed the reader to run `python evals/runners/score_only.py`. The runners, gold sets, metrics and the harness's own tests were all absent, so the only instruction the packaged file gave could not be followed from the archive. The cause was not an oversight in the file list. Hatchling's include patterns are gitignore-style, so the bare entry "README.md" matched at any depth: evals/README.md and examples/demo/README.md travelled and nothing else from either tree did. Root entries are anchored with "/" now, directories are named, and an exclude table keeps run artefacts, __pycache__, .serena, case folders and dist out rather than trusting the backend to honour VCS ignores. evals/tests ships too: testpaths lists it, so an archive carrying that config without the directory makes a bare `pytest` fail out of the box. Shipping a harness without the tests that prove its arithmetic ships an unverifiable harness. The wheel does not change. evals/__init__.py makes it a top-level package named `evals`; installing that into every user's site-packages is generic-name squatting, and it would not make the documented command work anyway, since the runners resolve paths relative to a source tree. tests/test_packaging.py reads the pyproject tables directly and asserts what they would match, so the list cannot silently narrow again. It skips itself below 3.11 — tomllib is stdlib from 3.11 and requires-python is >=3.10, so a module-level import made *collection* the failure mode on the 3.10 CI cell, which interrupts the whole session rather than failing one file. docs/RELEASING.md records the verification, including extracting the archive and running the packaged README's own command from inside it. A one-off check is how this regressed.
Software tests and model evaluation are different claims and stay separate: tests/ checks deterministic behaviour offline, evals/ measures judgement quality against human gold labels. Live runs cost money and never run under pytest or CI — run_eval.py refuses when CI=true. The harness built to measure honesty was its own worst offender. It reported a macro F1 of 1.00 for a run that got one class of three right, and printed "no gold and/or no predicted instances" to explain the exclusion when both existed. F1 is now derived from what it is — 2TP/(predicted + gold) — so it is undefined only when a class is in neither column. A class present and always wrong scores 0.0 and stays in the mean. Exclusion reasons are derived from the counts, so they cannot go stale the way a constant did. The aligner was order-dependent: the same data with the gold list permuted scored 1/2 or 0/2, while its own docstring promised order independence. It now scores every candidate pair, sorts globally with a deterministic tie-break and assigns best-first. The margin gained a second direction — under global assignment the winner is a *pair*, so it can be a coin flip against a rival in its row or its column, and a deterministic guess is not an honest answer. Three population leaks are closed by one eligibility module: null-gold cases inflated retrieval 4/4 to 5/5 by matching `None != "not_retrieved"`; a pair whose sibling was excluded scored 100% pair discrimination as a group of one; and an unnamed denominator sat beside named ones. Every rate now names its population, and one test walks the record asserting each rate's n equals the population it declares — findings of this shape catch the next one too. Drift was warned about and scored anyway, and in practice never even ran: the check required --case-dir, which the documented invocation omits. A missing manifest returned [] — the same value as "checked, nothing changed". Both now report not_verified, which does not invalidate a case but is visible. agreement() takes the run count explicitly rather than inferring it from vector zero, and raises on ragged input. ABSENT stays distinct from UNMATCHED: UNMATCHED means the run was asked and the aligner failed, ABSENT means the harness never asked. Conflating them charges the operator's gap to the model.
CI ran 3.11 alone while requires-python said >=3.10. The matrix is now 3.10-3.13 with a badge, and the suite genuinely passes on a real 3.10 interpreter — verified locally, not assumed from the config. .serena/ is gitignored: its project.yml, memories and pickled symbol cache are machine-local, regenerated by onboarding, and drift from the code silently. The review skill's description said it fact-checks "high-value" claims and reports labels the checks "did not cover" — the tool attempts every citation-backed claim it extracts, and reports the labels no checked claim reached. The bug template no longer pins a version that is already stale.
The README is a correctness surface, not marketing. Eight statements did not match the implementation and now do: only one of three formats stamped the converter; the retrieval manifest is written to the case root, not case/out/; CI runs on pull requests and pushes to main, not "every push"; the title sanity check lets an empty or unreadable first page pass, which the README omitted; retrieval and the scout were called deterministic when they query live services; the demo's numbers describe one run; coverage is per occurrence. CLAUDE.md is new and leads with the rule that overrides convenience — an unread source never receives a verdict — because most past bugs have been breaches of it. It records the decisions worth not relitigating, including the two the occurrence audit rejected: reading-order zipping, and folding `uncertain` into covered or missing. CONTRIBUTING gains the disclosures workflow: a new report disclosure goes in one module with its own token, then all three templates, with the parity test asserting it reaches every format. Adding one straight to a template is the drift this codebase already suffered once. No accuracy figure is claimed anywhere, because none has been measured, and the changelog no longer carries hard-coded counts — both of its numbers were false on the day they were written.
examples/demo/output/ is what the README sends a first-time reader to, and it predated every disclosure the last two rounds added. It named the checker only as "Claude", reported "all 4 citation labels covered", and carried neither the converter stamp nor the attribution caveat. Regenerated from a live run. Same result as documented — 2 supported, 2 contradicted, 1 not retrieved, 1 uncited assertion — and now: "5/5 citation occurrences reached by an extracted claim, across 4 labels". Claims 4 and 5 both cite [3], which is the duplicate-label case that motivated occurrence coverage in the first place, so the artefact now demonstrates it. make_manuscript.py crashed with a bare ModuleNotFoundError when playwright was absent, which is the case after `pip install -e ".[dev]"`. It names the extra to install, the way render.py already did for the same import.
Co-citation is an offer of support. Batch mode used to take `avail[0]`, judge
the claim against that one source, and file every other cited source as never
opened — so a claim citing [3], [5], [7] and [9] was decided by whichever
happened to come first in reference order, and the reader could not tell
whether the rest agreed.
Every retrievable cited source is now judged in its own model call, so each
verdict rests on that source's own text and a long source cannot crowd out a
short one. Each carries its own note, page anchor and evidence crop; the report
renders one commented crop per source under the claim, with a count above them
("4 cited sources checked: 2 fully support it; 1 partially supports it; 1
contradicts it").
The claim's headline is the MOST ADVERSE verdict any source gave. One cited
reference contradicting the claim is precisely the finding a reviewer needs,
and averaging it away against two that agree would bury it. That risks
overstating the split, so the per-source breakdown is always rendered beside the
headline and says outright when the sources disagree. The claim-level
slug/page/crop are copied from the deciding judgement, or the evidence shown
next to the headline would belong to a different paper.
`unjudged_refs` narrows to one meaning as a result: the source could not be
obtained. It used to mix "available but not chosen" with "unobtainable" under
one wording, which left a reader unable to tell whether a co-citation was
unexamined or unavailable.
New verdict `not_addressed`, and the fan-out is what forces it. A source cited
for another part of a compound claim is not `contradicted` (it does not say
otherwise) and not `partial` (there is no true kernel); with only those three
available the model must mislabel, and the report would gain false findings
indistinguishable from real ones. An inapt citation is a real result and now
has a name. Appended to JUDGMENT_VERDICTS so every existing verdict keeps its
position — counts() gains a key and never reorders one — and it is the one
verdict exempt from the source_page requirement, because demanding a page for
an absence would force the model to cite something that isn't there.
A failed call now unchecks only that source: another co-cited source may
already have produced a real verdict, and discarding it would report a gap that
does not exist.
Cost: a claim citing four retrievable sources takes four model calls, not one.
Closes the roadmap item; README's does/does-not list moves with the behaviour.
Simulating a first-time, non-technical user installing from GitHub and auditing
one paper surfaced nine bumps. None of them were bugs — every flag was correct
and documented — but a newcomer had to assemble six decisions from a `--help`
screen before anything happened, and three of the ways a run can fail only
surfaced minutes in.
Bare `papertrace` now walks one audit. It checks the environment FIRST, which is
the largest of the wins: a missing `claude` CLI is a sentence before the user
types a path, not a traceback after ingest has run. A missing chromium only
removes the `--png` question rather than blocking anything, because the markdown
and HTML reports do not depend on it — `fatal` separates those two cases so
neither is treated as the other.
Then one question at a time: the paper, the case folder (defaulted from the
filename), whether it is published, the email. Before spending anything it
states the workload counted from the paper itself and asks permission; on yes it
prints the equivalent `papertrace run` line, because a wizard that hides the CLI
leaves its user unable to repeat or script what they just did.
Four specifics worth recording:
- `--doi` is a *definition* problem, not a documentation one. It means the DOI of
the paper being audited, and saying so did not stop it being misread — this
project's own README illustrated it with a cited reference's DOI. The wizard
reads the DOI off the front matter and asks "is that this paper's own DOI?",
which replaces the definition with a yes/no. Only page 1 is read: a reference
list is full of other papers' DOIs, and picking one up would anchor the
literature scout to somebody else's work with nothing to notice.
- The workload counts citation *places* and how many cite several sources.
Places alone now understate the bill, since each cited source is judged
separately. Zero places is reported as an unrecognised citation style, not as
free.
- The email persists to ~/.config/papertrace/config.json, read after --email and
the env vars so an explicit value always wins. JSON, not TOML: requires-python
is >=3.10 and tomllib is 3.11+. Missing, empty or corrupt reads as {} — a
convenience may not become a hard failure.
- chromium is detected by looking for the browsers directory, not by asking
playwright. Asking starts its driver subprocess, which prints "Task was
destroyed but it is pending!" and a TargetClosedError to stderr — on the one
screen whose job is reassuring a nervous first-time user that their setup is
fine.
Not a TTY means print help and exit, never block: a wizard that waits on stdin
in CI is the same class of failure as a test that needs a local case folder.
`papertrace ingest -c` also works now. `-c` meant the case folder in every
subcommand except that one, which failed with "No such option: -c". `-o/--out`
is unchanged.
The help screen is grouped into Start here / Utilities / Pipeline stages. Nine
commands as one flat list gave no clue that six of them are internals the
seventh calls for you.
README: Quick start leads with the guided flow and a rendered screenshot, and
the install matrix gains the `[dev]` and `[dev,full]` rows it never had. That
omission is what made a real user's `pytest` resolve to a system interpreter and
die with ModuleNotFoundError, so it is the most load-bearing edit here. Image
URLs are absolute now — repo-relative paths 404 on PyPI, where README is the
long description.
`requires-python` said `>=3.10` while the matrix tested 3.10–3.13, so anyone installing on 3.14 ran on a version nothing verified. That is not hypothetical: Homebrew's `python3` is now 3.14, and a real first-time install landed there. The suite passes on it — 316 passed, 1 skipped, verified locally on all five cells before advertising any of them — so the matrix says so rather than the metadata over-promising. The alternative, capping `requires-python`, would drop support that demonstrably works. The README and CHANGELOG lines naming the range rode in the previous commit, which touched the same files.
Testing the guided flow on a published European Journal of Radiology paper failed twice: the preflight reported layout-aware ingest as available and the run then ingested as flat text and advised installing a backend that was already installed, and the audit stopped at "No numbered references found" on a paper with 34 of them. Both symptoms had causes underneath them. 1. `run` passed Typer option objects to the stages. The stage commands are Typer commands called as plain functions, where a declared default is an `OptionInfo`, not the string it displays — and `run` called them positionally. Adding `--case` to `ingest` shifted every later argument, so `backend` became an `OptionInfo`, equalled neither "auto" nor "docling", and every audit silently ingested as flat text. My regression, introduced in the commit that added `--case`. All six calls are keyword arguments now, so the next inserted parameter cannot repeat it. 2. An unrecognised ingest backend silently meant pymupdf, and that is what kept (1) invisible: `ingest_pdf` treated every value that was not "docling" as flat text. Had it raised, the mistake would have been loud and immediate instead of a contradiction the user had to notice. It raises now — this is the "falls back to something reasonable" the project forbids. 3. The flat-ingest warning did not say why it was flat. It advised `pip install 'papertrace[docling]'` unconditionally, including to somebody who had docling and had simply chosen `--backend pymupdf`. Advice that cannot be followed sends a reader to fix the wrong thing. 4. A References heading the ingest did not classify as one was invisible. `references_section` only started collecting at a `sectionheader`. Flat-text ingest guesses headings from font size and on this paper made three author lines headings while leaving `References` as body text, so a 34-reference list read as absent. A block whose entire text is the word now counts, whatever the backend called it; a sentence merely starting with it does not, or "References were checked by hand" would swallow the paper. 5. Reference entries running together were parsed as one, and the one was WRONG. The marker regex required `[N]` at a line start; Elsevier PDFs extract with entries mid-line. All 34 references collapsed into entry [1], which then took its DOI from reference [2] — a mis-attribution, not a shortfall. The resolver would have fetched the wrong paper and judged [1]'s claim against it, with only the title check between that and a confident wrong verdict. The bracketed form is recognised anywhere in a line now; the bare `12.` form still needs a line start, because mid-sentence it is prose. Also: `Br. J. Radiol. 89 (1061) (2016)` slugged an entry `a-1061` — YEAR_RE's parenthesised branch accepted any four digits and now requires a plausible century. Verified on the paper that failed: 34 entries parsed, 26 with DOIs, every DOI with the reference it belongs to, no implausible years. Known and left: entry [1] still resolves no DOI because the PDF wrapped it as `10.1259/ bjr.20150914`, with a space inside. Joining DOIs across whitespace risks corrupting valid ones, and Crossref-by-citation-string is the documented route that resolves it regardless. One commit rather than five: the tests for the plumbing bug and the parsing bugs share a file, so splitting would leave an intermediate commit red.
The verdict tally omitted `not_addressed`, announcing 31 of 34 claims. Built by `_verdict_line` now; the test asserts the buckets sum to the claim count. The backend was printed once, buried under 13 lines of noise. Half was ours — setting TORCH_LOGS *and* calling set_logs makes torch warn about it. RapidOCR adds its own handler inside `convert()`, so a pre-set level cannot work; the conversion runs inside a scoped `logging.disable(INFO)`. Only third-party chatter is affected — our disclosures go through `rich`, not `logging`. `report` and `run` now restate the backend, and both say cited sources are always read as flat text. `/review` names it unconditionally, not only when it is bad news. Also merges the duplicate [Unreleased] changelog headings two earlier commits left behind (52 bullets in, 52 out).
`_match_provided` returned the first filename containing every slug token over an unsorted `Path.glob`, so a sources folder holding an article and its supplement produced an undefined choice, and a supplement-only folder supplied one as the source. Candidates are ranked now — exact `<slug>.pdf`, else shortest name — and supplement-looking names are excluded rather than ranked last, so a supplement-only match falls through to the online resolver. No marker is shorter than five characters: `si` would reject the real slug `si-mohamed-2021`. Provided files also went entirely unverified — the title check lives in `_accept`, which only sees downloads. They are checked now, and a mismatch is recorded rather than refused: the user named the file and there is nothing to fall back to. README gains the `--provided` naming rule and, in the does-not list, that supplemental material is read for neither the audited paper nor its sources. The first of those is the tool's one silent gap, so it is stated rather than left to be found.
`tick` read `ClaimResult.verdict`, which multi-source checking only assigns in `apply_headline()` — after every source group. So each group printed the field's default, `not_retrieved`, whose glyph `○` is what the final tally uses for a source that was never obtained: the demo judged 2 supported and 2 contradicted while the console showed `○ ○ ○ ○`. Marks now come from that source's own `SourceJudgement`, and `partial`, `not_addressed` and `unchecked` stop sharing the fallback glyph.
…aims about it Live run on 2026-08-30 with docling 2.118.1. Verdicts match the documented expectation; the report now renders through the per-source judgement path and carries the occurrence-level coverage line, so the README's "all 4 citation labels" becomes 5 occurrences across 4 labels. Two things the README now states rather than leaving to assumption: no demo claim is co-cited, so nothing in the committed output exercises the per-source summary count; and an anchor's page can move between runs that reach the same verdict — claim 4's crop went from page 1 to page 2. The manuscript is one page, not two.
rich styles single words inside a sentence, so with colour on the help screen contains `Run \x1b[1;2mpapertrace\x1b[0m\x1b[2m with no arguments` and a plain substring test cannot match. CI has colour on, a developer terminal usually does not — which is how this passed locally and failed on all five Python versions at once, the first time CI ever ran against this branch. The render now forces colour on and the assertion strips it, so the styled path is exercised everywhere rather than depending on the terminal.
…evidence `is_references_heading` exists because two readers needed the same answer and each had its own: `references_section` accepts a body-typed `References` block (flat ingest guesses headings from font size and gets it wrong), while occurrence scanning stopped only at a `sectionheader`. One rule now, in the module both import. `headline_verdict` ranked `not_addressed` above `unchecked`. `not_addressed` asserts every available source was read and none spoke to the claim — a finding about the paper. A failed check was not read, so that assertion is unavailable. New optional fields `RefEntry.boundary_ambiguous` and `title_check`, both in the schema, neither required.
A `[2]` printed inside reference 1's title was accepted as the start of reference 2, which then carried reference 1's DOI. Neither first nor last occurrence is safe — the mirror case breaks the opposite guess — so a duplicated label is disclosed and `resolve_entry` refuses it before the provided-file match, the Crossref title lookup and every download, all of which derive from a `raw` string that is two references spliced together. The title check was two-state, so "no readable first page" and "the title matches" shared one `None` and a scanned PDF read as `matched ...`. Tri-state now; the file is still used, but the reason says which of the three happened. One client carried a `mailto:` User-Agent to every host, including arbitrary PDF servers, while the wizard promised Unpaywall and Crossref. Now a per-request header on those two. The User-Agent also said 0.3 at version 0.3.1; it derives.
…otal Occurrence scanning ran past a body-typed `References` heading, so on the very Elsevier paper that motivated teaching `references_section` about it, every `[N]` in the reference list counted as a manuscript citation and coverage reported gaps that do not exist. Both readers call the shared rule; the second, competing regex in this module is deleted rather than left to be reused. A 5,000-digit ASCII `source_page` passed `isascii()` and `isdigit()`, then hit CPython's 4300-digit conversion limit — a `ValueError` out of a function documented as unable to raise. Bounded before conversion, with a narrow catch; still no blanket `except Exception`, which would relabel our bugs as the model's. `ASK_ATTEMPTS` names the retry so the wizard can quote a cost that matches it. The pipeline tests called `ingest_pdf` with no backend, so `auto` chose docling and fetched layout models from Hugging Face. CI installs `[dev]` only and local runs had a warm cache, so the suite's offline property was never actually tested.
…cript `refs` parsed the cached `source_map.json` first and guarded second, so a different PDF with the same file name produced a manifest describing the previous paper while carrying the new one's hash — and the warning promising that "re-running `papertrace refs` fixes it" laundered the identity instead of fixing it. The guard runs first now and a name-only case re-ingests what it was given, so the manifest and its hash describe one file. `_guard_case` returns the basis its answer rested on, which is what lets the caller act on it.
The retrieval manifest carries the per-source reason but renders in `report.md.j2` alone, so a source nobody could identify was invisible to editor and terminal readers. It is a run-level disclosure now, computed from the manifest. Adding it also exposed why that gap was possible: markdown and editor render run-level disclosures through a catch-all, while the terminal template filters by explicit key, so a new disclosure silently vanishes there. The parity loop caught this one only because a test happened to construct the input that fires it, so there is now a mechanical check that every key in `disclosures.py` has a branch.
`detect_doi` read the whole first page, which on a short paper reaches the reference list — so it offered a cited paper's DOI as the manuscript's own, and the confirmation defaulted to yes. It stops at the references heading, using the same rule as ingest, and the question must now be answered. The printed "equivalent command" interpolated paths unquoted, so one with a space split into two arguments, and it omitted the email entirely. Built as argv and rendered with `shlex.join`. "Up to N model calls" was a ceiling `_ask`'s retry could exceed. The estimate shows a base and a worst case derived from `check.ASK_ATTEMPTS`.
`cli.ingest(manuscript, out, backend)` bound `backend` to `case` and left the real `backend` as Typer's `OptionInfo`, which the dispatcher rejects — so a paid evaluation died before judging anything. The keyword-only fix was applied in `cli.py` and missed here. A forwarding test now fails if an argument shifts.
Python 3.10 skipped all 26 packaging tests — a module-level
`importorskip("tomllib")`, which is 3.11+ — so the oldest version
`requires-python` allows was the one never checking packaging, on the branch that
moved package data into the wheel. `[dev]` supplies `tomli` below 3.11.
`README.md` pointed at `assets/logo.png` relatively and `assets/` is not
shipped, so the first image on the PyPI page was the broken one; the four
`docs/*.png` were already absolute and the release note named those three
instead. It now names the rule and gives the grep that would have caught it.
"The run takes about five minutes" was unmeasured and roughly triple the one run
we timed (95s with models cached), so it states the measurement and its
conditions.
Two changelog statements were false: appending `not_addressed` to
`JUDGMENT_VERDICTS` does shift `VERDICTS` (`not_retrieved` 3→4), which is
breaking for anyone indexing it; and one entry still described judging a
multi-reference claim against the first available source, which this same
release replaced.
defraction0
changed the title
Every image URL is pinned to `/main/`, so an image added on a feature branch is a 404 until that branch merges — correct URL, correct file, no bytes behind it. The existing grep only catches repo-relative paths and cannot see this at all. `docs/wizard.png` sat in exactly that state for the life of this branch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
24 commits, 90 files, +12918/-325. Version is
0.4.0;CHANGELOG.mdhas a[0.4.0] — 2026-08-30section with 68 bullets.Status: three external review rounds, all findings fixed. Round 3 was a
pre-merge review against
e972623that returned 12 correctness findings plus arelease-readiness finding, every one with a working offline reproducer. All 13
are fixed here, each with a failing test written first. Its verdict on that
commit was do not merge, do not release, and it was right.
Verification
Beyond the default environment:
HF_HUB_OFFLINE=1,tests/254 passed— the offline gate holds without a warm cacheuv415 passed, 1 skipped(playwright only) — packaging suite now runs there, was 26 skippedWhat round 3 found, and what it cost
Three of the worst were introduced by earlier commits on this branch — the
price of fixing the adjacent bug and not looking for the mirror image.
Two wrong-paper routes. A
[2]printed inside reference 1's title wasaccepted as the start of reference 2, which then carried reference 1's DOI —
the same hazard the mid-line marker rule was added to fix, from the other
direction. Neither first-nor-last occurrence is safe, so an ambiguous boundary is
now disclosed and never resolved. Separately,
refsparsed a legacy case'scached source map before the identity guard and then stamped the new
manuscript's hash onto it, so the warning promising that re-running
refsfixesidentity was laundering it instead.
The offline gate was never actually tested. Pipeline tests called
ingest_pdfwith no backend, soautochose docling and fetched layout modelsfrom Hugging Face. CI installs
[dev]only and local caches were warm, so thesuite proved offline-with-warm-cache and nothing more.
The coverage audit counted the bibliography.
references_sectionlearned toaccept a body-typed
Referencesblock; occurrence scanning did not, so on theexact paper that motivated the fix every
[N]in the reference list became amanuscript citation and coverage reported gaps that do not exist. One shared rule
now, and the competing regex is deleted rather than left to be reused.
Two headlines that outran their evidence.
not_addressed+uncheckedproduced "does not address the claim", turning a run failure into a finding about
the paper. And a scanned provided PDF was reported as
matched ...because"unverifiable" and "verified" shared one
None— with the detail rendered inmarkdown only, so two of three readers never saw it.
Also fixed: the paid evaluation runner crashed on a positional Typer call before
judging anything;
_judgement_fromwas not total (a 5,000-digit page hitCPython's conversion limit); the contact email went to every host including
arbitrary PDF servers while the wizard promised two;
detect_doioffered a citedpaper's DOI with the confirmation defaulting to yes; the printed "equivalent
command" did not survive a path with a space and omitted the email; and "up to N
model calls" was a ceiling the retry could exceed.
Release readiness
importorskip("tomllib"), 3.11+) —the oldest supported version was the one never checking packaging, on the
branch that moved package data into the wheel.
[dev]now suppliestomli.README.mdreferencedassets/logo.pngrelatively andassets/is notshipped, so the first image on a PyPI page was the broken one. The four
docs/*.pngwere already absolute; the release note named those three andmissed the logo. It now carries the grep that would have caught it.
timed. It states the measurement (95s, models cached) and its conditions.
described behaviour this same release replaced.
Breaking, for anyone indexing
VERDICTSAppending
not_addressedtoJUDGMENT_VERDICTSshifts the concatenated tuple:not_retrieved3→4,unchecked4→5. Nothing in this repository indexes itpositionally and
counts()gains a key without reordering one, but an externalconsumer that does index it will read the wrong name. The changelog previously
claimed positions were preserved; that claim is gone.
Still true, and deliberate
examples/demo/is a controlled demonstration with planted errors.claim shifts every later pairing and manufactures confident wrong attributions.
labels_in_text,coveredandmissingkeep label-level meaning byte forbyte, because
evals/align.pyreadsmissingto decide whether an unmatchedgold case is the tool's failure or the evaluator's.
evals/align.pyon purpose —papertracecannot importevals, andevalsmust not import a matcher fromthe thing it grades.
directions. The audited paper's own supplement is the one gap that is silent
rather than disclosed, which is why it is documented.
Known, logged, not fixed
_askretries once with no backoff, so a systematic environment failure costsdouble. The wizard now quotes a worst case that includes it.
check: recovering N unchecked claims re-judges every group.committed example. It is unit-tested and the README says so.