Four defects from a real first-run, and the two that were mine - #2
Merged
Conversation
Reference [8] of a real audit was an ACR news page with no DOI. With nothing to look up, `resolve_entry` fell through to a Crossref bibliographic title search — which always returns something — and got `10.1002/acr2.11538`: ACR Open Rheumatology, American College of Rheumatology, not Radiology. Two claims were then reported against a rheumatology editorial. The title check passed it 6/15, and two of those matches were `chatgpt` and `source`, taken from the `?utm_source=chatgpt.com` tracking parameter in the reference's own URL — the wrong paper being an editorial about ChatGPT. A reference whose identity is carried by a URL now ends at `no_doi` with no request sent. The gate keys on the absence of article structure, not the presence of a link, because publishers print URLs beside volumes. Also: the title check no longer takes tokens from a URL, and `verified` needs four matched words rather than a ratio a three-word reference clears on generic vocabulary. Falling short reads `unverifiable`, never `mismatch` — too few words to tell is not evidence of a different paper, and `mismatch` discards the file.
A real pre-proof put refs 1-9 on page 7, a `Declaration of interests` section next, then refs 10-15 on page 8. `references_section` stops at the following header — the guard that keeps the list from swallowing the paper — so six references were never parsed, never retrieved and never mentioned. Resuming takes two independent signals, because neither alone separates a split bibliography from an appendix: the entries must be `list`-typed, and the resumed run at least two blocks. `list` and never `text` on purpose — under the flat backend entries are `text`, the same type as every paragraph, so resuming there would swallow the Discussion of any paper whose references are not last. The cost is that a flat-ingested split list is still parsed short. Only reference-shaped runs are collected: `_parse_bulleted` appends a non-bullet line to the previous entry, so stray prose corrupts a reference, not just noise.
docling drops the hyphen when joining a word split across two lines. That is right far more often than wrong — `approxi-`/`mately` is one word, and 87 of 94 breaks on the measured paper were that kind — and wrong when the hyphen is the word's own: `Non-`/`Hispanic` arrived as `NonHispanic`. Repairing all of them rewrote 46 of 181 blocks and turned `approximately` into `approxi- mately`, so the adapter now repairs a join only where the paper writes that compound out unbroken somewhere else. The document's own evidence, not a lower→upper junction, which proves nothing: `HbA1c` has one and is correct, `Timedependent` has one and is broken. Four blocks rewritten on that paper. Separately, and this one was observed: `search_for` reads a hyphenated line break as a space, so a page printing `Non-`/`Hispanic` carries only `Non- Hispanic` — neither the compound nor docling's join. In a real audit `sohn-2022` p2 matched nothing and kept its box only because a second phrase matched. `highlight` retries in forms the page dictates. Still exact search: a phrase the page lacks returns no box and `anchor_located = False`.
Crossing a section boundary to finish a bibliography is a judgement, and the numbering of the later entries rests on it. `RefManifest.references_resumed` records it — optional in the schema, so older manifests still load — and a run-level disclosure carries it into markdown, editor and terminal. The wording points both ways deliberately: the guess can be wrong, but not making it was the previous behaviour and that failed silently. The terminal template needed an explicit branch because it filters disclosures by key while the other two use a catch-all. The mechanical guard added in 0.4.0 caught the omission before any parity test did — its first live catch.
A first-time batch run put its output in `PaperTrace/case/` — the root of a git clone, unnamed for the paper, and the same folder every later paper would use. `case` is one name for all papers, and the cwd is wherever the shell happened to be. `run` and `refs` now default to the paper's own folder, named after it: the one location stable across invocations, so a re-run finds its case without a flag. An explicit `-c` still wins. `check`, `highlight`, `report` and `scout` have no paper to take a name from, so they get no default — `./case` still works when it exists, otherwise they refuse and list the folders that look like audits rather than picking one. Because a case folder is no longer called `case`, `.gitignore`'s name-based guardrail no longer covers it, so the folder now carries its own. A derived folder already holding this paper asks: amend, or a numbered sibling. Only when the tool chose the name. With no terminal it amends and says so — never blocking on stdin, and `fresh` would move a scripted caller's output somewhere it never named. A different paper is still exit 2.
Four documented commands did not exist: `refs … -o case/`, `report case/`, `highlight case/ --claim <id>`, and a prose `refs --parse-only` with no manuscript. A skill is documentation an agent runs verbatim, so each was a usage error waiting to happen. Every `papertrace` line in every skill is now parsed against the real Typer commands. Two things the check had to get right: appending `--help` would have masked `report case/`, because `--help` is eager and fires before click reports an unexpected extra argument — so each line is parsed into a context instead, which validates without invoking. And scanning only `review/` is how the fourth command survived while the other three were fixed, so it walks every skill. `group.context_class` rather than `import click`: typer 0.27 vendors click and declares no dependency on it, so the import is a collection error on 3.10 — and a collection error interrupts the whole session, losing every test in it.
The README said a claim inside a figure "is found, checked, and shown like any other". True of the red box — a figure's numbers are in the text layer, so text search finds them on the real page — and unsupported for the judging half: under the layout backend a figure arrives as `[FIGURE: <caption>]`, and in-figure text only where docling found a text region inside it. On the one paper measured it found none: of 9 figures, 5 carried text in the text layer and no docling block landed inside any figure region, while the flat backend did carry it. The section now names which backend each half holds for, states that as one paper rather than a rate, and says the two illustrating crops come from cited sources — which batch mode always reads as flat text. Its opposite error is fixed too: flat text does not deliver figures "not at all", but as loose words with no figure to belong to. Also drops the `-c case` example, which now reads as a default it is not.
0.4.0 is untagged and unpublished, so these nine bullets belong in its section instead of a 0.4.1 that existed for one afternoon. One heading of each kind, as before — this file has grown duplicate Added/Fixed headings twice.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eight commits off
main. Four items, all found by running the tool on a realpaper rather than by reading the code. Folded into the
[0.4.0]changelogsection, since 0.4.0 is untagged and unpublished — no version that existed for
one afternoon.
Verification
CI has not run on this branch —
ci.ymlfires onpull_requestand pushes tomain, so this PR is the first time 3.11, 3.12 and 3.14 see it. That mattershere: see "Why local green isn't enough" below.
What changed
A URL-only reference was resolved to an unrelated paper. Ref [8] of a real
audit is an ACR news page with no DOI. With nothing to look up,
resolve_entryfell through to a Crossref title search — which always returns something — and
got
10.1002/acr2.11538: ACR Open Rheumatology, American College ofRheumatology, not Radiology. The title check passed it 6/15, two of those
matches being
chatgptandsource, harvested from the?utm_source=chatgpt.comtracking parameter in the reference's own URL, against an editorial about
ChatGPT. Two claims were reported against a rheumatology editorial. A reference
whose identity is carried by a URL now ends at
no_doiwith no request sent.A reference list split by a section was parsed short. Refs 1-9 on page 7, a
Declaration of interestssection, then refs 10-15 on page 8 — andreferences_sectionstops at the next header, so six references were neverparsed, retrieved or mentioned. Now parsed whole (verified: 15 of 15 on the real
paper), gated on two signals, and the crossing is disclosed in all three report
formats plus the console.
Every audit defaulted into one folder called
case. A first run from a gitclone root put its output in
PaperTrace/case/— the same folder every laterpaper would use.
runandrefsnow default to a folder named after the paper,beside the paper; a collision on a derived name asks amend-or-fresh, and never
blocks without a terminal.
Four documented skill commands did not exist.
refs … -o case/,report case/,highlight case/ --claim <id>, and a proserefs --parse-onlywith no manuscript. Every
papertraceline in every skill is now parsed againstthe real Typer commands.
Plus two ingest/highlight fixes — docling deleting a hyphen that belongs to
the word, and an anchor phrase being unboxable for typesetting reasons alone
(observed: one real anchor matched nothing and kept its box only because a second
phrase did).
And a README claim withdrawn. "A claim that lives in a table cell or inside a
figure is found, checked, and shown like any other" was true of the red box and
unsupported for the judging half.
Why local green isn't enough — the argument for this PR
Two of the four items were introduced by me, and two failures were invisible to
a local run:
mainfailed CI on all five versions (rich styles wordsinside a sentence, so a plain substring assertion cannot match with colour on).
Local was green.
clickat module level. Typer 0.27vendors click and declares no dependency on it, so on 3.10 that is a
collection error — which interrupts the whole pytest session and loses all
448 tests, not one. Caught only by hand-building a real 3.10 venv; 3.13 has
click from elsewhere.
Both were the same shape: an environment difference no single local interpreter
shows.
Corrections to my own instructions, recorded
the page." Of 94 breaks on a real paper only 7 are lexical hyphens; the
other 87 are syllabic, where docling is right. That fix rewrote 46 of 181
blocks and produced
approxi- mately. The shipped version repairs only wherethe document writes the compound out unbroken elsewhere: 4 blocks.
--help. That masksbugs:
--helpis eager and fires before click reports an unexpected extraargument, so
report case/ --helpexits 0.225–232en-dash case. docling's jointriggers on ASCII
-only. Recorded as unverified rather than asserted.Known and deliberate — not oversights
restricted to
listblocks because under the flat backend entries aretext,the same type as every paragraph, so resuming there would swallow the
Discussion of any paper whose references are not last.
_askretries once with no backoff; no resume-onlycheck.papertrace initstill defaults to a folder namedcase, soinitthenrun paper.pdforphanscase/sources/. Disclosed by a hint rather than fixed,because changing it breaks bare
papertrace init.Left undone, ordered
report. Same class as everything fixed here: a silent gap rather than a
disclosed one.
de fi nitionsin the ingest text vsdefinitionsin the page glyphs. Measured, reproducible, pymupdf path.editorial. Nothing here retracts them; that case needs a re-run.