The PMC HTML fallback cannot succeed from a Python HTTP client: keep it, gate it, or drop it?

[cmungall]

The pmc full-text provider's HTML fallback cannot succeed from a Python HTTP client, and I do not think that is fixable within the provider. This is a question about whether to keep advertising it, not a bug report.

PMC answers requests with a bot-check interstitial carried on an HTTP 200, so no status check sees it. Measured against PMC6451728:

client / headers	bytes	result
requests, default	21,214	interstitial
requests, User-Agent: curl/8.7.1	21,210	interstitial
requests, honest tool UA + contact address	21,316	interstitial
requests, Accept-Encoding: identity	21,214	interstitial
curl, default	236,615	article

Giving requests curl's own User-Agent still returns the interstitial, so the discrimination is below the header layer — TLS or HTTP fingerprint. I did not pin down which, because the only way past it is impersonating a browser's fingerprint, and that is not something this library should do.

So _fetch_pmc_html returns None for every article, every time. #94 fixed its container selector, which was independently wrong — the container moved from div.article-body to <section class="body main-article-body"> — but that only means the extraction would work if the fetch ever returned an article page. It does not.

Why it matters beyond a dead code path

Two things downstream depend on the route existing:

1. pmc sits in the default full_text_providers chain. Every reference that reaches it spends a rate_limit_delay plus a request, and the only possible outcomes are None or a TransientFullTextError on a 429. In a batch validation that is real time spent on a call that cannot succeed.
2. The stale-HTML warning used to tell users to retry. Let a reviewed cache serve stale HTML, and find PMC's current article container #94 changed that text, because "retry when the source serves full text again" is advice a curator cannot act on for a PMC-only article. The underlying situation is still that a cached PMC HTML entry can never be repaired by this library.

Options

• Drop the HTML fallback and let pmc be XML-only. Honest, removes a guaranteed-failing request from the default chain. Costs nothing that currently works.
• Keep it but mark it non-functional — leave the code for a future where PMC serves plain clients again, and take it out of the default full_text_providers so it is opt-in.
• Keep it as is and accept that one provider in the default chain is a no-op, now that its selector is at least correct.

I lean toward the second: the code is right, the access is not, and nothing distinguishes those two states in a log line today.

Worth noting the one legitimate route still works — Europe PMC's fullTextXML for OA-subset articles. This is only about the HTML page fallback for articles outside that subset, which is also the set where the licence would not permit redistribution anyway.

Found while investigating monarch-initiative/dismech#12672.

[github-actions]

Summary

Direct follow-on to #94: that PR fixed _find_pmc_article_body's selector (div.article-body/div.tsec → the current class="main-article-body" on a <section>), but confirmed as a side effect that the fetch it operates on never returns an article page from a plain Python client — PMC serves a bot-check interstitial on an HTTP 200 regardless of User-Agent or Accept-Encoding, and only bypassing it via TLS/HTTP fingerprinting (browser impersonation) would work, which you've correctly ruled out as something this library shouldn't do. So _fetch_pmc_html is a dead path, not a buggy one.

Confirmed against current code

• pmc is first in the default full_text_providers chain (models.py:527: ["pmc", "epmc_preprint", "unpaywall", "openalex"]).
• The HTML fallback isn't a separately-addressable provider — PMCFullTextProvider.fetch() tries XML first, then calls self._fetch_pmc_html() inline (etl/fulltext/pmc.py:116). So "take it out of the default chain" can't be done by editing full_text_providers, since that list only has a "pmc" entry, and removing it would also drop the XML route (the one that does work via Entrez/efetch). Gating the HTML half specifically needs a new, more granular knob — e.g. a pmc_html_fallback: bool on ReferenceValidationConfig, threaded into _fetch_pmc_html as an early return, separate from serve_stale_html_full_text (Let a reviewed cache serve stale HTML, and find PMC's current article container #94's flag governs serving a cached stale entry, not whether a live fetch is attempted at all).

On the three options

Agree with leaning toward option 2. Option 1 (delete it outright) throws away code that's correct and might start working again if PMC's bot-detection posture changes — and per your own numbers it costs nothing today since it never succeeds. Option 3 (leave it in the default chain) has a real cost you've already quantified: every PMC reference pays a rate_limit_delay plus a wasted round-trip, and can raise TransientFullTextError on a 429 for a call that was never going to succeed anyway. A quiet log line distinguishing "no article body found" from "PMC declined to serve us one at all" would also help anyone debugging a similar report later, since right now both look identical from the caller's side (None, or a transient error).

Worth linking to the same root cause

monarch-initiative/dismech#12747 (open) is the same NCBI bot-detection behavior showing up on a different route: the Bookshelf fetch also gets a Checking your browser - reCAPTCHA interstitial on occasion, and because that response is currently treated as a fetched document rather than a failed fetch, its <title> gets diff'd against the curated reference_title and produces false-positive validation errors. That issue's suggested fix — recognize known bot-challenge titles/interstitial fingerprints and raise "could not fetch" rather than returning/caching the page — is close enough to what "gate it" would need for PMC HTML that the two might share a small helper (a set of known interstitial title/byte-size signatures, checked wherever a raw HTML response is about to be trusted as content). Might be worth solving once for both NCBI surfaces rather than twice.

Also the source of both: monarch-initiative/dismech#12672, where the stale-cache-serving guard from #94 originated, and which is what surfaced this whole PMC-fetch investigation in the first place. That issue's "suggested work, in priority order" item 1 was literally "report the PMC selector drift upstream" — which is now done (#94) — with this issue being the natural next item once the fix landed and the fetch was still observed to fail.

No PR from me here per instructions — this is a design decision for a maintainer to make, and you've already scoped the implementation (add a config gate, keep the code, default it off from the live-chain but leave it callable) tightly enough that it should be a small, well-contained change whenever you're ready.