A PDF URL cached by URLSource gets the URL as its title

[cmungall]

When URLSource caches a PDF URL, the resulting entry's title is the URL itself. There is no title to extract from a PDF byte stream, so the reference ends up identified by its own address.

This matters for any consumer that treats title as bibliographic metadata. dismech compares reference_title in its KB against the cached title, so a URL-as-title either blocks that check or gets copied into the knowledge base as though it were the paper's name.

J-STAGE is the case that surfaced it — 53 such references in the dismech cache. A J-STAGE PDF URL has a sibling _article page carrying a citation_title meta tag:

https://www.jstage.jst.go.jp/article/<journal>/<vol>/<issue>/<article>/_pdf
https://www.jstage.jst.go.jp/article/<journal>/<vol>/<issue>/<article>/_article   <- citation_title here

The general shape is not J-STAGE-specific. Publishers routinely serve a PDF at one URL and its metadata at a predictable sibling, and citation_title is a near-universal convention (Google Scholar requires it).

Suggested fix

When a url fetch yields full_text_pdf and no title distinct from the identifier, try to recover one — either from the PDF's own metadata where present, or from a landing page discovered by a documented rule. A conservative version that only fires when title == identifier would fix the whole class without changing any entry that already has a real title.

If you would rather not chase landing pages, extracting the embedded PDF /Title where it exists would still cover part of it, and a consumer could then tell "no title available" from "title is the URL".

Context

dismech currently does this as a runtime patch over URLSource.fetch. That makes two patches it still carries — this and #92 — after retiring ten whose fixes landed here. Filing rather than keeping the workaround quiet, per the rule that a patch needs an upstream issue and an exit.

[github-actions]

Root cause, confirmed in current main

URLSource.fetch sets title=url unconditionally on the PDF branch, with no attempt at recovery:

# src/linkml_reference_validator/etl/sources/url.py:82-92
if is_pdf:
    text = PDFExtractor(backend=config.pdf_backend).extract(
        data, content_type="application/pdf"
    )
    return ReferenceContent(
        reference_id=f"url:{url}",
        title=url,
        content=text,
        content_type="full_text_pdf" if text else "unavailable",
        full_text_url=url,
    )

Two things worth noting that make the two suggested fixes cheaper than they might look:

1. The embedded-PDF-title path is nearly free. PDFExtractor already builds a pypdf.PdfReader (etl/extract/pdf.py:35) but only calls .extract_text() on it and throws the reader away — reader.metadata.title is sitting right there unused. No new dependency, no new parse pass, just a second read off an object already in hand. This wouldn't catch J-STAGE (scanned/generated PDFs from Japanese aggregators are notorious for empty or garbage /Title metadata), but it's a same-file, low-risk win for whatever fraction of the wider is_pdf class does carry real embedded metadata.
2. The landing-page path can reuse existing code. URLSource._extract_title (url.py:123-148) already regex-matches <title>; a citation_title meta tag is just <meta name="citation_title" content="...">, one more regex alternative in the same function. So "fetch sibling landing page, run it through (an extended) _extract_title" is a small addition, not new machinery.

A more general discovery rule than URL pattern-matching

The issue frames the fix as "documented rule" for finding the sibling landing page, with J-STAGE's _pdf → _article as the example. That's inherently publisher-specific and will need a new rule per publisher shape. Two alternatives worth considering that don't require enumerating URL patterns:

• citation_pdf_url in reverse. This is the same Highwire/Google-Scholar tag family as citation_title, and it's what most link resolvers already use to go from a landing page to a PDF. There's no standard reverse index, but if the PDF response's Content-Disposition or the PDF's own metadata carries a DOI, that sidesteps guessing entirely.
• DOI-first. Many publisher PDFs embed a DOI in /Subject or /Keywords PDF metadata, or it's derivable from the URL path. If a DOI is recoverable, a single Crossref lookup (https://api.crossref.org/works/{doi}) gives a real title without touching the publisher's site shape at all — one code path instead of one per publisher. Given linkml-reference-validator likely already resolves DOIs elsewhere (worth checking sources/doi.py if it exists) this could share logic rather than adding a second title-recovery mechanism.

Either way, the issue's own fallback — "at least distinguish 'no title available' from 'title is the URL'" via the title == identifier conservative check — is worth keeping as the baseline regardless of which recovery mechanism is picked, since it's what makes the fix safe to land incrementally (recover what you can now, mark the rest honestly, extend later).

Pattern across #92/#93

Worth noting for tracking: this is the second of two remaining dismech runtime patches against URLSource/adjacent code, after ten were retired once fixed upstream here (#66–#74, #85, #87, #88). Both remaining ones — this and #92 (raw HTML/script caching) — sit in the same small file area (etl/sources/url.py) and share a theme: URLSource is the one source class that does no extraction/sanitization at all, unlike the HTML full-text path (HTMLExtractor) or XML path (XMLExtractor). If both land, they'd touch overlapping code (the non-PDF branch for #92, the PDF branch here), so doing them in the same pass — or at least the same PR series — would avoid two separate re-fetch/re-cache campaigns over the same files, similar to the note on #67/#68.