A retrieval-augmented chatbot over Python's official documentation for asyncio
and typing, with an admin dashboard that tracks every question asked, which
documentation chunks were retrieved, and where retrieval struggles.
Built as a focused demo of production RAG patterns: structured ingestion, citation-grounded answers, observability, and input hardening — deployable end-to-end on free-tier hosting.
Live demo: https://rag-python-doc.vercel.app
Try a question like:
What is the difference between Optional and Union in typing?
You'll get a streamed answer with clickable citations back to
docs.python.org. The admin dashboard at /admin is password-gated and
shows every question, the chunks retrieved, and any inputs flagged by the
moderation + injection layers.
The suggested query above is one the retrieval eval covers and passes. Two queries in the golden set do not pass, and both are documented with their diagnosis in EVAL_RESULTS.md. Finding out which questions a retriever gets wrong is the point of this project, so they are published rather than hidden.
Hosted on Render's free tier — first request after a cold start may take ~30 seconds while the service wakes up.
| Chat with citations | Admin dashboard |
|---|---|
 |
 |
┌──────────────────────────────┐
│ React + TS + Vite (Vercel) │
│ Chat UI Admin Dashboard │
└──────────────┬───────────────┘
│ JSON / SSE
▼
┌─────────────────────────────────────────────────────────┐
│ FastAPI backend (Render) │
│ ┌───────────────┐ ┌────────────────┐ ┌────────────┐ │
│ │ /api/chat │ │ /api/admin/* │ │ session │ │
│ │ (SSE stream) │ │ (cookie auth) │ │ cookie │ │
│ └───────┬───────┘ └────────┬───────┘ └────────────┘ │
│ │ │ │
│ ┌───────▼──────────┐ ┌─────▼───────────────────────┐ │
│ │ RAG pipeline │ │ SQLite (aiosqlite) │ │
│ │ • moderation │ │ • questions │ │
│ │ • injection │ │ • retrieved_sources │ │
│ │ detection │ │ • flagged_inputs │ │
│ │ • retrieval │ └─────────────────────────────┘ │
│ │ • OpenAI stream │ │
│ └───────┬──────────┘ │
└──────────┼──────────────────────────────────────────────┘
▼
ChromaDB (on-disk)
python_docs collection
• api-entry chunks (one per dl.py.function/class/method)
• prose chunks at section boundaries
• metadata: source_url, heading_path, module, anchor
- Backend: Python 3.11, FastAPI, SQLAlchemy 2 (async), aiosqlite, LlamaIndex, OpenAI SDK
- Vector store: ChromaDB, on-disk, persisted at
backend/data/chroma/ - Frontend: React + TypeScript (strict) + Vite + Tailwind
- Docs source: official Python 3.12 HTML archive, pinned to a patch version
# 1. Clone and enter
git clone <repo> && cd python-doc-chat
# 2. Backend (requires Python 3.11+ — install via pyenv/asdf if needed)
cd backend
python3.11 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp ../.env.example ../.env # then fill in OPENAI_API_KEY and ADMIN_PASSWORD
# 3. Ingest Python docs (one-time, ~2 min; embeds ~400 chunks)
python scripts/ingest.py
# 4. Seed the admin dashboard from the captured fixture
python scripts/seed_db.py
# 5. Run the API
uvicorn app.main:app --reload
# 6. Frontend (in a second terminal)
cd ../frontend
npm install && npm run devInteractive API docs auto-generated by FastAPI at http://localhost:8000/docs.
This project ships with a first-class retrieval eval harness rather than a
scratch script. backend/scripts/eval.py runs a golden set of five queries
against the live Chroma index and asserts that expected documentation anchors
appear within a target top-k rank.
cd backend && python scripts/eval.py # print results to stdout
cd backend && python scripts/eval.py --write-md # regenerate EVAL_RESULTS.mdExit code is non-zero on unexpected regressions. Failures that match a
documented weakness are marked KNOWN FAIL and don't fail the suite — if a
tuning change flips one to IMPROVED, that's the signal to update the file.
Current baseline: EVAL_RESULTS.md — 3/5 pass, 2 known failures documented honestly. The known failures inform the What I'd do differently at scale section.
Three layers, in order of enforcement strength:
- OpenAI Moderation API on every user input. Flagged inputs return a canned refusal and are logged with their flagged categories. This does block.
- Regex pattern detection for known prompt-injection shapes (role-swap
attempts, system-prompt leaks, "ignore previous instructions" variants).
Matches are logged to
flagged_inputsbut do not block — they exist as observability signal for the admin dashboard. - Scoped system prompt + Pydantic length cap. The model is told explicitly to answer only from retrieved context and to refuse off-topic questions. Input is capped at 1000 characters at the schema layer.
What this is, honestly: observability-grade, not security-grade. A determined adversary will get past the regex list. The point of this layer is to give the admin dashboard a meaningful signal about what people try, not to make the system bulletproof.
What a production version would add: a trained classifier for injection scoring (not regex), output moderation on the generated response (not just input), a secondary LLM pass to verify the response stays on-topic, and rate limits tied to session identity rather than IP.
Password-protected via a single ADMIN_PASSWORD env var. Login POSTs the
password, backend issues an HTTP-only signed cookie (itsdangerous), subsequent
requests gated by a FastAPI Depends().
Dashboard shows:
- All questions with timestamp, session ID, retrieved sources, generated answer
- Questions-per-day chart (last 14 days)
- Most-retrieved doc sections
- Mean retrieval similarity per question
- Flagged inputs table (moderation + regex) with filters
scripts/capture_seed.py runs each demo prompt once through the actual chat
pipeline (moderation → retrieval → OpenAI → flag detection) and saves the
result to data/seed_fixture.json. scripts/seed_db.py loads that fixture
on startup or on demand. The dashboard shows actual model output, real
retrieval scores, and a real moderation block — captured once, committed to
the repo, replayed deterministically. Costs ~$0.03 in OpenAI credit to
recapture; the fixture is offline + free to seed thereafter.
- Backend → Render (free web service). ChromaDB index is committed to the
repo because Render's free tier has no persistent disk. Rebuilds happen by
running
scripts/ingest.py --rebuildlocally and committing the result. - Frontend → Vercel (free static hosting).
The eval harness (EVAL_RESULTS.md) surfaces two specific weaknesses in the
baseline cosine-similarity retriever:
- Q1 — prose chunks outrank terse API signatures on natural-language
queries. "How do I run multiple coroutines concurrently?" returns generic
coroutine prose instead of
asyncio.gather/asyncio.TaskGroup, even though those chunks exist and rank first on literal-name queries. Fix: enrich each API chunk's embedding text with a natural-language intro derived from the first paragraph of its<dd>description, so a how-to question matches the API entry's semantic content rather than just its signature. - Q5 — term ambiguity + module imbalance. "What is a Protocol?" returns
only
asyncio.*Protocolchunks because asyncio contributes 8+ Protocol- related entries while typing contributes exactly one. Cosine similarity has no mechanism to enforce module diversity. Fix: apply MMR (Maximal Marginal Relevance) re-ranking to diversify top-k across modules — retrieve top-20, then greedily pick 5 that balance relevance with inter-result dissimilarity. - Broader mitigation for both: hybrid BM25 + vector retrieval with
reciprocal rank fusion. Lexical matching gives literal terms (
Protocol,gather) a floor when the embedding model under-weights them, and the fusion score tends to smooth over the prose-vs-API mismatch from Q1.
- ChromaDB → managed vector DB (Pinecone, pgvector, or Turbopuffer). A single-node embedded store is fine up to ~10⁵ chunks; beyond that it bottlenecks on ingest, replication, and cross-region latency.
- SQLite → managed Postgres with proper migrations (Alembic). SQLite is here to keep the demo free-tier and reviewable; it's not the right pick above a handful of concurrent writers.
- Ingest as a pipeline, not a script. Version each chunk by
(doc_version, source_url, content_hash), support incremental re-embedding on upstream docs changes, and run it on a schedule with dbt-style freshness checks. - Expand the eval. Grow the golden set from 5 to ~50 queries drawn from real admin-dashboard traffic; add Recall@k and NDCG metrics; gate CI on it; track scores over time so tuning changes have a visible regression signal.
- Output-side safety. Moderate the generated response, not just input; add a secondary model to verify factual grounding in the retrieved context.
- Auth. Swap the single-password admin for SSO (Clerk, Auth0, WorkOS) and add per-user rate limits on the chat endpoint.
- Observability. Ship structured logs + traces to Honeycomb/Datadog; track retrieval latency, LLM latency, and token cost as first-class metrics.
backend/
app/
main.py FastAPI entry, router mounts, CORS, lifespan
core/ config (pydantic-settings), shared dependencies
routers/ chat.py, admin.py — one APIRouter per feature
rag/ parser, chunker, indexer, retriever, prompts
db/ SQLAlchemy async models + session
security/ injection patterns, moderation, admin auth dep
schemas/ Pydantic request/response models
data/
raw/ downloaded HTML (gitignored)
chroma/ vector store (committed — see Deployment above)
app.db SQLite runtime DB (gitignored)
scripts/
ingest.py one-shot: download → chunk → embed → index
eval.py golden-query retrieval eval → EVAL_RESULTS.md
capture_seed.py one-time: run prompts through real pipeline, snapshot to JSON
seed_db.py load data/seed_fixture.json into the admin DB
frontend/
src/
pages/ Chat.tsx, AdminLogin.tsx, AdminDashboard.tsx
components/ reusable UI
lib/ typed API client
MIT.