add statpearls scraper - #25
Open
nahatav wants to merge 4 commits into
Open
Conversation
Discovers chapters through NCBI's E-utilities books search, since there is no public listing endpoint, deduplicating section-level hits by their parent chapter, then scrapes each chapter's Bookshelf HTML page. Sends the `tool` parameter E-utilities usage policy asks for, and respects the 5 second crawl delay robots.txt sets for /books/NBK*. Records each chapter's top-level section count, rolling nested subsections into their parent. StatPearls is listed as an encyclopedic-fallback source in the AMFV project doc.
Applied the same bars @zndr27's review on MedARC-AI#23 established for MedlinePlus, proactively rather than waiting for a second review pass: - Documented StatPearls' actual license: CC BY-NC-ND 4.0, confirmed from NCBI's own copyright dialog on the book page. Unlike MedlinePlus this is not public domain: NonCommercial and NoDerivatives, the latter in direct tension with claim decomposition and training-data use. Every document now carries `license`/`license_url` in its metadata so this travels downstream, and the module docstring spells out the conflict explicitly. - scrape_chapter now returns None (logged) on a fetch failure or missing content instead of raising, so one bad chapter in a multi-hour, ~11k chapter crawl costs a document instead of the run. Verified against a real nonexistent accession. scrape_chapter_by_url keeps raising, since a single --url request has no other document to fall back on. - search_section_uids and summarize_chapters now raise StatpearlsFetchError on request failure instead of a raw httpx error escaping. - Added the coverage that was missing before any review flagged it: scrape_statpearls end to end (including a chapter fetch failure mid-run not aborting the rest), the --url branch, and a CLI dispatch test. - Documented typical chapter size next to the licensing paragraph.
zndr27
suggested changes
Sep 9, 2026
| continue | ||
| seen.add(ref.accession) | ||
| new_refs.append(ref) | ||
| return new_refs |
Contributor
There was a problem hiding this comment.
The list_chapters helper in statpearls.py, the website collector, removes previously seen chapter identifiers and returns the remainder. The shared empty-list branch reads an empty remainder as end of the entire source.
An accepted fix continues over nonempty search batches containing no new chapters and stops only when the underlying search is exhausted.
TLDR A batch full of already-seen chapters can make the collector stop before later chapters are found.
| body = bodies[0] | ||
| sections = [child for child in body.xpath("./div[@id]") if _TOP_LEVEL_SECTION_RE.fullmatch(child.get("id") or "")] | ||
| body_html = lxml_html.tostring(body, encoding="unicode") | ||
| markdown = html_to_markdown(body_html, link_mode=link_mode, base_url=BOOKS_BASE_URL) |
Contributor
There was a problem hiding this comment.
The text converter _chapter_content in statpearls.py, the chapter collector, passes the Bookshelf root as the address used to resolve relative links. A relative link is interpreted from the page containing it; using its parent site changes the destination.
An accepted fix passes that URL through both chapter-reading paths into the content helper and tests relative links. The closing step gathers both required changes.
TLDR Some links inside a chapter point to the wrong address because the converter starts from the site root.
…ter page Two fixes from @zndr27's review. list_chapters filtered out chapters it had already yielded and returned the remainder, and scrape_listing_documents stops at the first page that comes back empty. Searching by section means one chapter occupies as many hits as it has sections, so a batch can hold nothing new without the search being anywhere near finished; that batch ended the crawl and left the rest of the corpus unscraped. The docstring asserted NCBI clusters a chapter's sections together, which is behaviour rather than a contract, and it was carrying the correctness of the whole crawl. Discovery now consumes batches until one contributes a chapter or the search itself runs out, which is the only case that returns empty. That loop can issue several request pairs where there used to be one per listing page, and those are no longer spaced apart by the document fetches between pages, so back-to-back batches are paced by SEARCH_DELAY_SECONDS to stay inside the 3-per-second E-utilities guidance the module documents. The cursor moved into ChapterSearch because the offset can no longer be derived from the page index the shared contract passes. _chapter_content resolved relative links against the Bookshelf root, so a chapter link written as `figure/A1/` came out as /figure/A1/ instead of /books/NBK111/figure/A1/. Both chapter-reading paths now pass the chapter URL. Tests cover a batch of only-seen chapters, exhaustion as the one empty case, and relative links on both readers. Each fails against the previous code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…stops ChapterSearch used an Attributes: block; every other dataclass in the package takes a one-line summary and a prose paragraph, so it does too, and the prose now says why the offset lives here rather than coming from the page index. The exhaustion test claimed in a comment that an exhausted cursor issues no further request but never checked it. It counts searches now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Licensing — read this first
StatPearls chapters are not public domain, unlike the MedlinePlus source in #23. Confirmed straight from NCBI's own copyright dialog on the book page (
https://www.ncbi.nlm.nih.gov/books/NBK430685/):NonCommercial and NoDerivatives. The ND term is the one that matters here: claim decomposition and training-data construction are both derivative uses of the source text, which is what this project intends to do with scraped sources. I'm not the right person to make that call, so I haven't tried to. Every document carries
metadata["license"]andmetadata["license_url"]so it's not silent in the corpus, and the module docstring spells the conflict out. Whoever owns the decomposition/training pipeline should decide whether StatPearls-sourced documents get included, held out, or something in between, before this feeds anything downstream. Happy to gate it behind a flag or exclude it from default runs if that's the safer default — just say so.Given IDSA is already flagged in the project doc as not redistributable, this project clearly already tracks this distinction; StatPearls belongs in that same bucket.
Summary
Adds a StatPearls source scraper. StatPearls has no public listing of chapter accessions, so chapters are discovered through NCBI's E-utilities search API rather than a crawlable index page: searching the
booksdatabase forstatpearls[book]returns one hit per chapter section (Introduction, Treatment, Review Questions, etc.), so discovery deduplicates by each section's parent chapter accession to build the chapter list. Chapter content itself is scraped from its NCBI Bookshelf HTML page. StatPearls is listed as an encyclopedic-fallback source in the AMFV project doc, alongside MedlinePlus.Compliance
/books/NBK*is explicitlyAllowed in NCBI's robots.txt with a 5 second crawl delay, which is applied between chapter fetches.NBK25497) asks for at most 3 requests/second without an API key and for callers to identify themselves via thetoolparameter — sent asEUTILS_TOOLon every request. Discovery issues two sequential requests per listing page, well inside that limit.eutils.ncbi.nlm.nih.gov/robots.txtitself sets a blanketDisallow: /for all agents, which reads as a generic crawler-exclusion default rather than a statement about the documented, versioned API this scraper actually calls (the same host that publishes the usage policy above). I want to flag that reasoning explicitly rather than let it pass silently — a maintainer might read that robots.txt differently than I did.Handling a long crawl
A full run is on the order of 10,000+ chapters at a mandatory 5s delay, roughly 15 hours. Two things followed from that scale once I thought it through:
scrape_chapternow returnsNone(logged) on a fetch failure or missing content instead of raising, so one bad chapter costs a document instead of the rest of the crawl. Verified against a real nonexistent NCBI accession, both that this path logs and skips, and thatscrape_chapter_by_url(the--urlsingle-document path) still raises clearly, since there's no other document to fall back on there.search_section_uidsandsummarize_chapters(the E-utilities discovery calls) raiseStatpearlsFetchErrorrather than a rawhttpxerror, so callers catching this module's own error type don't miss the most likely failure.Components affected
datasets/amfv_datasets/scraping/statpearls.py(new)datasets/amfv_datasets/scraping/cli.py— one import, oneSCRAPERSentrydatasets/test/test_scraping_statpearls.py(new)datasets/test/test_scraping_cli.py— the unknown-source test asserts the registered source listTesting
uv run ruff check/uv run ruff format --checkcleanuv run pytest— 47 passing, run on Python 3.13 to match CI--urlmode, and both error-handling paths against a real nonexistent chapter (one skips and logs, the other raises)