Reddit Scraper

Scrape reddit subreddit posts, comments and subreddit details into clean JSON or CSV. Point it at any subreddit and get the title, author, score, upvote ratio, comment count, flair, post type, linked URL, thumbnail, self-text and timestamp of every post in the hot, new, top, rising or controversial feed. Add --comments to also get each post's comment threads (author, score, depth, parent, OP flag, body), or ask for a subreddit's details (description, creation date, weekly visitors and contributions).

Powered by ScrapeUnblocker: the getPageSource API loads each reddit page in a real browser and returns the HTML, and this package turns it into typed records. No reddit API app, OAuth tokens or rate-limit bookkeeping, and you don't need to run proxies, headless browsers or retry logic yourself.

Features

Every subreddit feed. hot, new, top, rising and controversial, with the hour / day / week / month / year / all time window for top and controversial.
Real pagination. Follows the same "load more posts" requests reddit makes while you scroll, page after page, until your --limit, a --max-pages cap or the end of the feed.
Rich, typed posts. Post ID and fullname, title, author and author ID, subreddit, score, upvote ratio, comment count, award count, post type (text, link, image, multi_media), domain, linked URL (the article for link posts, the image for image posts), thumbnail, flair, full self-text, language, stickied flag, ISO 8601 UTC timestamp and permalink.
Comments. post (or --comments) fetches the post page and returns the comment threads reddit renders there, flattened in thread order with depth and parent_id, plus author, OP flag, score, awards, body and timestamp. Pick the comment order: best, top, new, controversial, old or Q&A.
Filters. Server-side flair filter (--flair Discussion) and client-side keyword (--keyword), minimum score (--min-score) and pinned-post (--no-stickied) filters.
Several subreddits in one run. Results are merged and de-duplicated by post ID. A subreddit that does not exist is reported and skipped, and the others still run.
Subreddit details. Title, description, creation date, weekly visitors and weekly contributions.
Clean text. Paragraphs, list items and code blocks keep their structure. Promoted posts are left out.
Resilient. Timeouts, transient API errors and half-loaded pages are retried with doubling backoff.
JSON or CSV output. One CSV row per post, plus an optional comments CSV with one row per comment, written as UTF-8 with a BOM so Excel reads emoji and accents cleanly.
CLI + Python library. Fully typed, and tested offline with a mocked client.

Install

git clone https://github.com/ScrapeUnblocker/reddit-scraper.git
cd reddit-scraper
pip install .

Get an API key at scrapeunblocker.com and expose it as SCRAPEUNBLOCKER_KEY (the official SDK reads it from there):

export SCRAPEUNBLOCKER_KEY=your_key_here        # Windows PowerShell: $env:SCRAPEUNBLOCKER_KEY="your_key_here"

You can also copy .env.example to .env and load it with your tool of choice.

CLI usage

# The 25 hottest posts in r/Python
reddit-scraper posts Python --pretty

# Top posts of the week from two subreddits, 50 each, to CSV
reddit-scraper posts webscraping datascience --sort top --time week --limit 50 \
    --format csv --output top_week.csv

# Newest posts that mention your keywords (social listening), pinned posts left out
reddit-scraper posts webscraping Python --sort new --keyword scraper --keyword captcha \
    --no-stickied --max-pages 5 -f csv -o mentions.csv

# Only one flair, with every post's comments (one extra request per post)
reddit-scraper posts Python --flair Discussion --limit 10 --comments \
    --comments-csv comments.csv -o posts.json

# One or more posts with their comments, top comments first
reddit-scraper post "https://www.reddit.com/r/Python/comments/1wv4f0d/" --comment-sort top --pretty

# Subreddit details
reddit-scraper subreddit Python webscraping --pretty

Option	Meaning
`SUBREDDIT ...`	One or more subreddits: `Python`, `r/Python` or a subreddit URL
`-s, --sort`	`hot` (default), `new`, `top`, `rising` or `controversial`
`-t, --time`	`hour`, `day`, `week`, `month`, `year` or `all` for top/controversial (default `day`)
`-n, --limit`	Maximum posts per subreddit (default 25)
`--max-pages`	Fetch at most N feed pages per subreddit (caps requests)
`--flair`	Only posts with this flair, exactly as shown on reddit
`-k, --keyword`	Keep posts whose title or text contains the word; repeat for several
`--min-score`	Keep posts with at least this score
`--no-stickied`	Leave out posts pinned by the moderators
`--comments`	Also fetch every post page for the full text and comments
`--comment-sort`	`best` (default), `top`, `new`, `controversial`, `old` or `qa`
`--comments-csv`	Also write the comments to this CSV, one row per comment
`-f, --format json\|csv`, `-o, --output`	Output format and file (CSV needs `--output`)
`--proxy-country`	Country to route requests through (default `US`)
`--retries`	Retries per page on transient errors (default 3)

Run reddit-scraper <command> --help for every option. python -m reddit_scraper works too.

Library usage

from reddit_scraper import RedditScraper, comments_to_csv, to_csv

reddit = RedditScraper()  # reads SCRAPEUNBLOCKER_KEY; or RedditScraper(api_key="...")

# Top posts of the month
posts = reddit.posts("Python", sort="top", time_filter="month", limit=100)
for post in posts[:5]:
    print(post.score, post.num_comments, post.title, post.permalink)

# Several subreddits, filtered, de-duplicated
mentions = reddit.posts_many(
    ["webscraping", "Python", "dataengineering"],
    sort="new",
    limit=20,
    max_pages=3,
    keywords=["scraper", "captcha"],
    skip_stickied=True,
    on_error=lambda sub, exc: print(f"skipping r/{sub}: {exc}"),
)

# Comments: for posts you already have, or for a single post URL
posts = reddit.with_comments(posts[:10], comment_sort="top")
post = reddit.get_post("https://www.reddit.com/r/Python/comments/1wv4f0d/")
for comment in post.comments:
    print("  " * comment.depth, comment.score, comment.body[:80])

# Subreddit details
info = reddit.subreddit_info("webscraping")
print(info.title, info.created, info.weekly_visitors)

to_csv(posts, "posts.csv")
comments_to_csv(posts, "comments.csv")

To de-duplicate across your own calls, pass the same seen=set() to each posts().

Examples

Script	What it does
`examples/subreddit_to_json.py`	Top posts of the week from a subreddit, saved as JSON
`examples/keyword_monitor_csv.py`	New posts mentioning your keywords across several subreddits, to CSV
`examples/comments_dataset.py`	The month's top posts plus their comment threads, as posts.csv + comments.csv

Example output

reddit-scraper post https://www.reddit.com/r/Python/comments/1wv4f0d/ --comment-sort top --pretty (usernames redacted, comment list shortened):

[
  {
    "id": "1wv4f0d",
    "fullname": "t3_1wv4f0d",
    "title": "New Python releases are out, 3.10.x reaches EOL, (and 3.15.0 is scheduled for today)!",
    "author": "<username>",
    "author_id": "t2_<id>",
    "subreddit": "Python",
    "subreddit_id": "t5_2qh0y",
    "score": 75,
    "upvote_ratio": 0.963,
    "num_comments": 22,
    "award_count": 0,
    "post_type": "multi_media",
    "domain": "self.Python",
    "is_self": true,
    "url": "https://www.reddit.com/r/Python/comments/1wv4f0d/new_python_releases_are_out_310x_reaches_eol_and/",
    "permalink": "https://www.reddit.com/r/Python/comments/1wv4f0d/new_python_releases_are_out_310x_reaches_eol_and/",
    "flair": "News",
    "text": "The new releases announced today, including EOL for 3.10.x: https://blog.python.org/2026/10/python-31022-31117/\n\nNote: The post above doesn't mention 3.15 - assuming they will do a separate post later today when that releases.",
    "thumbnail": null,
    "language": "en",
    "stickied": false,
    "created_utc": "2026-10-01T16:42:44+00:00",
    "rank": null,
    "sort": null,
    "time_filter": null,
    "comments": [
      {
        "id": "pdavx44",
        "fullname": "t1_pdavx44",
        "post_id": "1wv4f0d",
        "parent_id": null,
        "depth": 0,
        "author": "<username>",
        "is_op": false,
        "score": 29,
        "award_count": 0,
        "body": "3.15 has been postponed by a week: https://discuss.python.org/t/python-3-15-following-tradition-lets-have-a-surprise-rc3/109313",
        "created_utc": "2026-10-01T22:48:43+00:00",
        "permalink": "https://www.reddit.com/r/Python/comments/1wv4f0d/comment/pdavx44/"
      },
      {
        "id": "pdblnbl",
        "fullname": "t1_pdblnbl",
        "post_id": "1wv4f0d",
        "parent_id": "t1_pdavx44",
        "depth": 1,
        "author": "<username>",
        "is_op": true,
        "score": 3,
        "award_count": 0,
        "body": "Thank you for linking this.",
        "created_utc": "2026-10-02T01:04:17+00:00",
        "permalink": "https://www.reddit.com/r/Python/comments/1wv4f0d/comment/pdblnbl/"
      }
    ],
    "has_comments": true
  }
]

reddit-scraper subreddit webscraping --pretty:

[
  {
    "name": "webscraping",
    "id": "t5_318ly",
    "title": "webscraping",
    "description": "The first rule of web scraping is... do not talk about web scraping. But if you must, you've come to the right place ••• read the sub rules before posting ••• check the community bookmarks below for a guide to getting started.",
    "created": "2014-04-06",
    "weekly_visitors": 19902,
    "weekly_contributions": 459,
    "url": "https://www.reddit.com/r/webscraping/"
  }
]

Field notes:

Feed posts carry rank (position in the feed, counted before your filters), sort and time_filter. Posts fetched with post / get_post() have them as null.
url is what the post points at: the linked article for link posts, the image for image posts, the post itself for text posts. is_self is true for text posts.
post_type is reddit's own label: text, link, image or multi_media (galleries, videos and text posts with inline media).
text is the post's self-text as plain text: paragraphs and list items are separated by blank lines, and code blocks keep their line breaks. Link and image posts have null.
created_utc is ISO 8601 in UTC. upvote_ratio is a fraction (0.963 = 96.3% upvoted).
comments holds the threads reddit renders on the post page: roughly the first 25 comments and replies, in the chosen order. Deeper "more replies" branches are only loaded when you click them on reddit, so they are not included. num_comments is the post's total. has_comments tells you whether comments were fetched at all.
Comments come in thread order. depth is 0 for top-level comments, and parent_id is the t1_ fullname of the comment being replied to. is_op marks replies by the post's author.
weekly_visitors and weekly_contributions are the counts reddit shows in the subreddit header.
Only what reddit shows to signed-out visitors is available, so private and age-restricted communities are out of reach. A subreddit or post that does not exist is reported as not found. r/all and r/popular are front-page feeds, not communities, so name the subreddits you want instead.

Project layout

reddit-scraper/
├── src/reddit_scraper/
│   ├── __init__.py      # public API + __version__
│   ├── scraper.py       # RedditScraper: fetching, retries, pagination, filters, dedupe, comments
│   ├── parsing.py       # <shreddit-post> / <shreddit-comment> / subreddit header parsing
│   ├── urls.py          # subreddit / post URL builders, sorts, time windows, comment sorts
│   ├── models.py        # Post, Comment, Subreddit and ListingPage dataclasses
│   ├── export.py        # JSON and CSV export (posts, comments, subreddits)
│   ├── cli.py           # argparse CLI (reddit-scraper)
│   └── __main__.py      # python -m reddit_scraper
├── examples/            # runnable scripts
├── tests/               # offline tests with a mocked client (no API calls)
├── pyproject.toml
├── Makefile
└── .github/workflows/ci.yml

Development

python -m venv .venv && source .venv/bin/activate
make install      # pip install -e ".[dev]"
make lint         # ruff check
make format       # ruff format
make test         # pytest (offline, no API key needed)
pre-commit install

The tests replay synthetic reddit pages through a fake client. They spend no API credit and run in CI on Python 3.9 and 3.12.

Responsible use

Scrape at a reasonable pace and respect reddit's terms of use and its users. Posts and comments are written by real people: handle usernames and personal details with care, honour deletions, and follow the privacy laws that apply to you (such as the GDPR / CCPA) if you store or analyse them. This project is not affiliated with or endorsed by reddit.

Links

ScrapeUnblocker - the web scraping API behind this project
ScrapeUnblocker documentation - getPageSource, SDKs, parsing options
Python SDK on PyPI - pip install scrapeunblocker

Navigation Menu

Repository navigation

About

Repository files navigation

Reddit Scraper

Features

Install

CLI usage

Library usage

Examples

Example output

Project layout

Development

Responsible use

Links

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
examples		examples
src/reddit_scraper		src/reddit_scraper
tests		tests
.env.example		.env.example
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

Reddit Scraper

Features

Install

CLI usage

Library usage

Examples

Example output

Project layout

Development

Responsible use

Links

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages