Add CCO scraper

[NisargOza]

What this does

Adds a Cancer Care Ontario (CCO) scraper to the datasets scraping pipeline. Same shape as the NICE scraper — discovery and extraction are separate, and everything outputs a normalized ScrapedDocument.

Unlike NICE, CCO sits behind an Azure WAF JS challenge that rejects plain HTTP requests regardless of headers, so this scraper drives a real headless browser (Playwright) instead of httpx. Every function that touches the network takes a plain fetch(url) -> html callable rather than a browser handle directly, so discovery and parsing stay unit-testable without a browser.

The scraper extracts each guideline's short summary sections (objective, patient population, intended users, etc.) and bibliographic metadata (ID, version, type of content, document status, authors). It does not fetch the linked PDF, matching this repo's existing HTML-over-PDF stance for NICE — the PDF is recorded as a markdown link and as metadata["pdf_url"] for a later PDF-aware pass. It also handles several distinct page templates spanning roughly two decades of CCO's CMS history, including guidelines with no narrative sections at all, and guidelines archived 5+ years where CCO itself hides the abstract and PDF behind an HTML comment.

scrape_listing_documents in base.py is generalized to accept any client type, not just httpx.Client (NICE is unaffected — its type is inferred the same as before).

How discovery works

Discovery starts from CCO's types-of-cancer index page, which is itself the full master listing of every guideline — the per-cancer-type category buttons on it are just filtered subsets of the same list, so they aren't crawled separately. The listing is paged through a Drupal "Load more" pager rather than a rel=next link, so pagination follows that pager's markup directly.

Discovery retries itself up to 3 times if it comes back with a suspiciously small number of guidelines (fewer than 50, when CCO reports several hundred) — a thin first result turned out to be a load-timing issue rather than a real "no guidelines" state. If it's still thin after retries, it raises loudly rather than silently reporting a false "0 documents scraped" success.

How extraction works

Each guideline page is fetched through Playwright (resolving CCO's WAF challenge the same way a real browser would) and parsed with lxml. Bibliographic fields (Version, ID, Type of Content, Document Status, Authors) and the PDF link are matched against the whole page — they're precise, unambiguous matches, and at least one CCO template puts this metadata block outside the page's main-content region entirely. Narrative sections (Guideline Objective, Patient Population, Intended Guideline Users, Research Question(s), etc.) are matched preferentially within the page's role="main"/#content region, falling back to the whole page only if that comes up empty — this stops global nav/footer boilerplate from being absorbed into whichever section happens to be last on the page.

The extraction preserves:

• bibliographic metadata (version, ID, type of content, document status, authors), both as structured metadata and as inline content when no narrative sections exist
• narrative sections (objective, patient population, intended users, research questions, etc.)
• the linked full-report PDF as a markdown link and as metadata["pdf_url"]
• the page title, falling back to the page's <h1> when CCO's own <title> tag comes back empty (observed on some archived pages)

Guidelines that fail to parse (an unrecognized template, a fetch error, a persistent WAF-challenge failure) don't abort the run — they're yielded as a placeholder document with metadata["scrape_error"] set, so a run across ~400 guidelines spanning many CMS eras can complete even if a handful of pages don't match anything recognized.

CLI

uv run playwright install chromium # one-time browser binary download
uv run amfv-scrape --source cco --documents 1
uv run amfv-scrape --source cco --url https://www.cancercareontario.ca/en/guidelines-advice/types-of-cancer/64736
uv run amfv-scrape --source cco --documents 5 -f markdown -o ./cco-out/
uv run amfv-scrape --source cco --documents all -o cco.jsonl

--source all now includes CCO alongside NICE.

Licensing

CCO / Ontario Health guideline content appears on Meditron's list of redistributable sources for this kind of pipeline, but we haven't independently confirmed a redistribution license directly from CCO/Ontario Health. Treat bulk reuse as needing the same confirmation step as any other source until that's verified.

QA

• CCO's own listing page reports ~400 guidelines under the "Guidelines & Advice" content type; a full run discovered 379 unique detail pages, with real (non-placeholder) content for the large majority. The remaining gap is mostly accounted for by archived-guideline pages where CCO itself has hidden the body content, plus a handful of flagged placeholders visible via metadata["scrape_error"].
• Confirmed at least three distinct page templates in production: the standard "Advice" template (inline objective/population/users sections), a thin PEBC-style template (bibliographic metadata only, no narrative sections), and an archived (5+ years) template where the abstract and PDF link are HTML-commented-out and the metadata block sits outside the main-content region.

Test plan

• uv run pytest datasets/test/test_scraping_cco.py — URL validation, field/section parsing across all three known templates, Load-more pagination, WAF-challenge and transient-error retries, discovery sanity-check retry, per-item failure resilience
• uv run pytest — no regressions (39 passed)
• uv run ruff check . / uv run ruff format --check .
• Reviewer runs markdown output: uv run amfv-scrape --source cco --documents 5 -f markdown -o /tmp/cco-out/
• Reviewer runs a fresh full scrape (--documents all) to confirm discovered/scraped counts and spot-check any placeholder records

[CLAassistant]

All committers have signed the CLA.

[zndr27]

The Playwright imports in cco.py, the website collector, run as soon as the module loads. The package also imports CCO at initialization, so the dependency reaches callers before a browser is requested.

An accepted fix declares the Python dependency, updates the lockfile and documents browser installation, or makes browser support an explicit optional feature with an import boundary. Then run the normal tests without the review adapter. Next we inspect all-source dispatch.

TLDR A normal environment cannot even import the collector because a required package is missing from its dependencies.

[zndr27]

Now cli.py, the terminal dispatcher, and scrape_documents. _expand_source supplies CCO followed by NICE, but the CCO branch returns immediately, ending the function before NICE is called.

An accepted fix combines the selected runs and their totals, with an offline assertion that both sources execute. Reconcile this branch with the current shared dispatcher before merging. Next we follow a failed page into output.

TLDR Asking for every source returns only CCO and leaves NICE out.

[zndr27]

_scrape_guideline_or_placeholder in cco.py, the collector’s per-page adapter. Its exception branch returns a normal document with empty content, zero sections and an error flag.

An accepted fix separates error records from corpus records, skips failures when counting successful documents, and tests a bad page followed by a good one. Preserve visibility of the error. The closing step gathers the decision.

TLDR A failed page can be counted as a collected document even though it contains no source text.