Skip to content

Add Kodexis Reddit API provider as fallback with exam/cert filter - #22

Merged
Devathmaj merged 6 commits into
mainfrom
feat/add-kodexis-collector
Sep 23, 2026
Merged

Devathmaj merged 6 commits into
mainfrom
feat/add-kodexis-collector

Conversation

@Devathmaj

Copy link
Copy Markdown
Owner

Description

Adds Kodexis Reddit API provider as a fallback when REDDIT_INGESTION_ENABLED=false. The Kodexis API fetches the latest posts (using sort=new filter) from configured subreddits, with built-in filtering to exclude exam/certification passed posts while preserving voucher-related terms like free exam and exam voucher.

Also removes the deprecated mcsa subreddit from the catalog.

Type of Change

  • New feature
  • New source
  • Configuration / settings change
  • Documentation update

Affected Components

  • Scheduler / Dispatcher
  • Reddit Integration
  • Configuration / Settings

Testing

  • Ran pytest - all tests pass (52 passed, 15 skipped)
  • Ran ruff check . && ruff format . - no lint errors
  • Verified Kodexis API integration works with live API calls (tested AWSCertifications, AzureCertification, CompTIA)
  • Verified exam/certification + pass filter logic with unit tests
  • Verified fallback chain: OAuth -> Kodexis -> RSS

Migration

  • This PR includes an Alembic migration (not applicable)

Policy Checklist

  • This PR touches one or more policy-sensitive files (scheduler.py, reddit/collector.py)

If checked, confirm all of the following:

  • robots.txt compliance is preserved - Kodexis API is a third-party service, no direct scraping
  • No default crawl delays have been reduced below 2.0 seconds
  • Reddit rate limits and Responsible Builder Policy are respected (Kodexis handles this server-side)
  • REDDIT_INGESTION_ENABLED=false still collects Reddit - now via Kodexis API instead of RSS, with RSS as final fallback
  • No new direct httpx or aiohttp calls exist outside the policy layer - Kodexis client uses the standard policy layer via scheduler

Explanation:
The Kodexis API is a third-party Reddit data provider that respects Reddits API terms. When REDDIT_INGESTION_ENABLED=false, the system now uses Kodexis (if API key configured) as a more reliable alternative to RSS feeds, with RSS as the final fallback. The exam/certification + pass filter reduces noise from self-promotional I passed! posts while preserving voucher signals.

Additional Notes

Files Added

  • voucherbot/providers/Kodexis/client.py - KodexisRedditClient
  • voucherbot/providers/Kodexis/init.py - Module export

Files Modified

  • .env.example - Added KODEXIS_API_KEY config
  • voucherbot/config/settings.py - Added kodexis_api_key setting
  • voucherbot/providers/reddit/collector.py - Kodexis fallback + exam/cert + pass filter
  • voucherbot/services/scheduler.py - Initialize Kodexis client and pass to collector
  • voucherbot/database/bootstrap.py - Removed mcsa subreddit
  • sources/Subreddit.txt - Removed mcsa from documentation

Fallback Chain

  1. OAuth API (REDDIT_INGESTION_ENABLED=true + credentials) -> asyncpraw
  2. Kodexis API (REDDIT_INGESTION_ENABLED=false + KODEXIS_API_KEY) -> sort=new filter
  3. RSS (no Kodexis key or Kodexis fails) -> public Reddit RSS feeds

Exam/Certification + Pass Filter

Blocks posts containing standalone exam or certification + pass/passed/passing, with exclusions for:

  • free exam, exam voucher, exam credit, beta exam, retake
  • free certification, certification voucher

This preserves voucher signals while filtering out I passed my exam! self-reports.

@Devathmaj
Devathmaj merged commit 4be7c0e into main Sep 23, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant