|
Markdownee is a web scraper that crawls websites and saves their content as Markdown, HTML or plain text for LLMs, retrieval pipelines, and research datasets.
Choose which links to follow, set page and depth limits, and select how much page content to keep. Control tables, links, images, and user comments separately.
-
Boilerplate removal uses Trafilatura Core, our open-source extraction engine based on Trafilatura, available in TypeScript and Python. Markdownee's TypeScript and Python versions each use the matching Core implementation.
The Core in its name means it is reduced to one task: extracting main content by removing boilerplate. Other packages handle output conversion, including Markdown.
Core's TypeScript engine ports the original Python Trafilatura, with go-trafilatura as a DOM translation aid. Its native Python engine translates that port.
-
Crawlee handles crawling and uses Playwright for browser rendering.
-
Two language versions — TypeScript and Python: self-host with the npm CLI, npm library, or Python library, or use the hosted Apify Actor.
-
Save image files with the optional image downloading mode.
The playground previews a page and generates commands
for your settings. Use fetch for a single page or crawl to collect records with
limits and link filters. Library calls return data to your application; the
library guide explains their options
and storage behavior. Browser installation is needed only for browser crawling.
Fetch one page into memory or a file, or collect a crawl for later export. The CLI can export stored results or remove its selected storage. Start with a bounded crawl, inspect the first result, and then expand the collection. Python uses Crawlee Python and Python Trafilatura Core, with its capability differences documented separately in Python library help.
Use Node.js 22.22.2+ on 22.x, 24.15.0+ on 24.x, or 26+:
npx --package=@markdownee/markdownee markdownee fetch \
https://en.wikipedia.org/wiki/Web_scraping \
--crawler-type cheerio --save markdown-file -o page.mdOpen page.md for the extracted article. Cheerio fetches over HTTP and needs no
browser installation; npx downloads the package when needed. For JavaScript-rendered
pages, follow browser setup.
Continue with the CLI, Node.js library, Python library, or Apify guide. The examples contain complete programs.
Optional image downloading keeps local assets with extracted content.
Trafilatura Core removes boilerplate. Markdownee's TypeScript and Python versions each use the matching Core implementation. Core's TypeScript engine ports original Python Trafilatura, with go-trafilatura as a DOM translation aid. Its native Python engine translates that port. Crawlee drives Playwright for browser rendering.
Use the npm CLI, npm library, native Python library, or hosted Apify Actor.
Report reproducible problems through the issue tracker.