one requirement txt file

AGENT.md

@@ -14,7 +14,7 @@
 
 **Local dev:** `./local-run` (Postgres in Docker `wp-pg`, Next.js on host). See `scripts/local-run.sh`. **Local tests (CI parity):** `./local-test` runs **three** Python coverage gates (core 100%, reporting 100%, tools 100%); `./local-test browser` for `@pytest.mark.browser` integration tests — see `scripts/local-test.sh`. Mocked browser unit tests: `tests/test_browser_fetcher_unit.py`.
 
-**JavaScript crawl (optional):** Config keys `crawl_render_mode` (`static` | `javascript` | `auto`) and `crawl_js_*` in pipeline config / `pipelineConfigSchema.ts`. JS/auto crawls can capture browser console errors and uncaught exceptions (`crawl_js_capture_console`, stored under `page_analysis.browser`). **Auto mode** uses static-first fetch, pre-parse SPA heuristics (`needs_js_render`), then post-parse low-outlink fallback (`needs_js_render_after_parse`) in `crawler.py`. **Preflight:** `GET /api/crawl/browser-status` (localhost) spawns Python `browser_status()`; Run audit settings/run validation calls it when render mode is `javascript` or `auto`. Browser deps: `requirements-browser.txt` (installed by `./local-run setup` and `./local-test`). Runtime needs Chromium on `PATH` or `CHROME_PATH` (Docker sets `CHROME_PATH=/usr/bin/chromium`). Integration tests: `@pytest.mark.browser` — excluded by default in `pytest.ini`; Docker CI runs `tests/test_crawl_fetchers.py` and `tests/test_crawler_browser_e2e.py -m browser`; locally `./local-test browser`.
+**JavaScript crawl (optional):** Config keys `crawl_render_mode` (`static` | `javascript` | `auto`) and `crawl_js_*` in pipeline config / `pipelineConfigSchema.ts`. JS/auto crawls can capture browser console errors and uncaught exceptions (`crawl_js_capture_console`, stored under `page_analysis.browser`). **Auto mode** uses static-first fetch, pre-parse SPA heuristics (`needs_js_render`), then post-parse low-outlink fallback (`needs_js_render_after_parse`) in `crawler.py`. **Preflight:** `GET /api/crawl/browser-status` (localhost) spawns Python `browser_status()`; Run audit settings/run validation calls it when render mode is `javascript` or `auto`. Browser deps: Playwright from `requirements.txt` (installed by `./local-run setup` and `./local-test`). Runtime needs Chromium on `PATH` or `CHROME_PATH` (Docker sets `CHROME_PATH=/usr/bin/chromium`). Integration tests: `@pytest.mark.browser` — excluded by default in `pytest.ini`; Docker CI runs `tests/test_crawl_fetchers.py` and `tests/test_crawler_browser_e2e.py -m browser`; locally `./local-test browser`.
 
 **Run / APIs**
 
@@ -25,7 +25,7 @@
 - **Pipeline data** (crawl, edges, nodes, report payload, Lighthouse, keywords, warnings) is stored in **PostgreSQL only** — no JSON/CSV/HTML exports from the main pipeline.
 - **Pool tuning:** `DB_POOL_MIN` / `DB_POOL_MAX` (Python), `PGPOOL_MAX` (Node). Bulk crawl writes via `executemany`; optional **`crawl_stream_to_db`** streams rows during fetch.
 - **`web/`:** `/api/report/*` (PostgreSQL); `/api/run` spawns Python (localhost only); `/api/crawl/browser-status` GET (localhost, Playwright/Chromium preflight); `/api/pipeline-config` GET/PUT; `/api/llm-config` GET/PUT (AI only); `/api/chat` POST (SSE agent); `/api/chat/sessions` GET/POST; `/api/properties/{id}/google/links/import` POST (GSC Links CSV); `PipelineRunnerFab` saves pipeline + LLM state before each run
-- **MCP:** `python -m website_profiling.mcp` (stdio, **221 read-only audit tools** + MCP resources). See `docs/MCP.md`. Requires `pip install -r requirements-mcp.txt`.
+- **MCP:** `python -m website_profiling.mcp` (stdio, **221 read-only audit tools** + MCP resources). See `docs/MCP.md`. Requires `pip install -r requirements.txt`.
 - **AI Chat UI:** `/chat` — property-scoped chat with saved sessions (`chat_sessions`, `chat_messages` tables, migration `012_chat_sessions`).
 - **Job store:** in-memory on `globalThis` in `web/src/server/pipelineJobs.ts` — job status/log is lost on server restart (single-process dev/Docker only).
 - **Docker:** `Dockerfile` + `docker-compose.yml` (postgres + web); **`docker-compose.pull.yml`** for pre-built images (`WEB_IMAGE`); **`LIGHTHOUSE_CHROME_FLAGS`**
@@ -38,7 +38,7 @@
 | Report | `reporting/builder.py`, `reporting/categories.py` |
 | DB schema | `alembic/versions/` |
 | Local analysis | `analysis/local.py`, `requirements.txt` |
-| AI insights (LLM) | `llm/enrich.py`, `llm/agent.py`, `llm_config.py`, `requirements-llm.txt` |
+| AI insights (LLM) | `llm/enrich.py`, `llm/agent.py`, `llm_config.py`, `requirements.txt` |
 | Audit query tools (MCP + chat) | `tools/audit_tools/`, `mcp/server.py`, `commands/chat_cmd.py` |
 | Config / CLI | `config.py` (`load_config`, `load_config_from_db`), `cli.py`, `input.txt.example` |
 | UI pipeline schema | `web/src/lib/pipelineConfigSchema.ts` |

CONTRIBUTING.md

@@ -19,7 +19,7 @@ Thank you for helping improve this project. All contributions are welcome under
 
 Details: [README.md](README.md), [AGENT.md](AGENT.md).
 
-JavaScript/auto crawl needs Playwright (`requirements-browser.txt`, installed by `./local-run setup`) and Chromium on `PATH` or `CHROME_PATH`. Unit tests mock the browser fetcher; integration tests use `@pytest.mark.browser` and run in the Docker CI job (`tests/test_crawl_fetchers.py`, `tests/test_crawler_browser_e2e.py`). Locally: `./local-test browser` (skips gracefully if Chromium is missing).
+JavaScript/auto crawl needs Playwright (from `requirements.txt`, installed by `./local-run setup`) and Chromium on `PATH` or `CHROME_PATH`. Unit tests mock the browser fetcher; integration tests use `@pytest.mark.browser` and run in the Docker CI job (`tests/test_crawl_fetchers.py`, `tests/test_crawler_browser_e2e.py`). Locally: `./local-test browser` (skips gracefully if Chromium is missing).
 
 ## Running tests
 

Dockerfile

@@ -39,18 +39,14 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
     CHROME_PATH=/usr/bin/chromium \
     LIGHTHOUSE_PATH=/usr/local/bin/lighthouse
 
-# Python: base requirements + optional LLM API clients
+# Python dependencies
 COPY requirements.txt /app/requirements.txt
-COPY requirements-llm.txt /app/requirements-llm.txt
-COPY requirements-browser.txt /app/requirements-browser.txt
 COPY alembic.ini /app/alembic.ini
 COPY alembic /app/alembic
 RUN --mount=type=cache,target=/root/.cache/pip \
     python3 -m venv /opt/venv \
   && /opt/venv/bin/pip install --upgrade pip \
   && /opt/venv/bin/pip install -r /app/requirements.txt \
-  && /opt/venv/bin/pip install -r /app/requirements-llm.txt \
-  && /opt/venv/bin/pip install -r /app/requirements-browser.txt \
   && ln -sf /opt/venv/bin/python /usr/local/bin/python \
   && ln -sf /opt/venv/bin/python /usr/local/bin/python3
 

README.md

@@ -111,7 +111,7 @@ WebsiteProfiling/
 ├── Dockerfile                 # Production image
 ├── local-run                  # Dev setup & start script
 ├── local-test                 # Full test suite (CI parity)
-├── requirements*.txt          # Python deps (core, browser, LLM, MCP)
+├── requirements.txt           # Python dependencies
 └── pipeline-config.example.txt
 ```
 
@@ -173,9 +173,9 @@ Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for setup and
 
 Google Search Console / Analytics: connect via **Integrations** (gear icon) in the app.
 
-**JavaScript crawl (optional):** In Audit settings, set **Crawl rendering** to `javascript` (always headless Chromium) or `auto` (static first, browser when SPA heuristics match). Install locally: `pip install -r requirements-browser.txt` and Chromium on `PATH` or `CHROME_PATH` (included in Docker). The UI preflights via `GET /api/crawl/browser-status` before runs when JS/auto is selected.
+**JavaScript crawl (optional):** In Audit settings, set **Crawl rendering** to `javascript` (always headless Chromium) or `auto` (static first, browser when SPA heuristics match). Requires Playwright from `requirements.txt` and Chromium on `PATH` or `CHROME_PATH` (included in Docker). The UI preflights via `GET /api/crawl/browser-status` before runs when JS/auto is selected.
 
-**AI Chat (optional):** Ask questions about your audit data at [http://localhost:3000/chat](http://localhost:3000/chat). Enable a provider under **Run audit → AI settings** (`llm_enabled`, provider, model). `./local-run setup` installs `requirements-llm.txt` (`httpx`, OpenAI, Anthropic SDKs).
+**AI Chat (optional):** Ask questions about your audit data at [http://localhost:3000/chat](http://localhost:3000/chat). Enable a provider under **Run audit → AI settings** (`llm_enabled`, provider, model). `./local-run setup` installs all Python deps from `requirements.txt` (including `httpx`, OpenAI, and Anthropic SDKs).
 
 | Provider | Notes |
 |----------|--------|

docs/MCP.md

@@ -5,7 +5,7 @@ Read-only [Model Context Protocol](https://modelcontextprotocol.io) tools for qu
 ## Install
 
 ```bash
-pip install -r requirements-mcp.txt
+pip install -r requirements.txt
 export DATABASE_URL=postgres://profiling:profiling@localhost:5432/website_profiling
 export PYTHONPATH=src
 ```

requirements.txt

@@ -30,6 +30,21 @@ alembic>=1.13
 # Audit export (PDF)
 reportlab>=4.0.0
 
+# JavaScript rendering crawl (headless Chromium via Playwright)
+playwright>=1.49.0
+
+# LLM providers for AI enrichment (configure via web UI AI tab)
+httpx>=0.27.0
+openai>=1.0.0
+anthropic>=0.25.0
+
+# Spell-check / HTML validation extras
+pyspellchecker>=0.8.1
+html5lib>=1.1
+
+# MCP server for Cursor / Claude Desktop
+mcp>=1.0.0
+
 # Dev / test
 pytest>=7.0.0
 pytest-cov>=5.0.0

scripts/local-run.ps1

@@ -181,10 +181,6 @@ function Invoke-Venv {
     Write-Log "Installing Python dependencies"
     & $VENV_PIP install -q -r (Join-Path $ROOT "requirements.txt")
     Assert-LastExitCode "Failed to install requirements.txt"
-    & $VENV_PIP install -q -r (Join-Path $ROOT "requirements-browser.txt")
-    Assert-LastExitCode "Failed to install requirements-browser.txt"
-    & $VENV_PIP install -q -r (Join-Path $ROOT "requirements-llm.txt")
-    Assert-LastExitCode "Failed to install requirements-llm.txt"
 }
 
 function Invoke-Migrate {

scripts/local-run.sh

@@ -83,8 +83,6 @@ cmd_venv() {
   fi
   log "Installing Python dependencies"
   "$VENV/bin/pip" install -q -r "$ROOT/requirements.txt"
-  "$VENV/bin/pip" install -q -r "$ROOT/requirements-browser.txt"
-  "$VENV/bin/pip" install -q -r "$ROOT/requirements-llm.txt"
 }
 
 cmd_migrate() {

scripts/local-test.ps1

@@ -184,8 +184,6 @@ function Invoke-Venv {
         Write-Log "Installing Python dependencies"
         & $VENV_PIP install -q -r (Join-Path $ROOT "requirements.txt")
         Assert-LastExitCode "Failed to install requirements.txt"
-        & $VENV_PIP install -q -r (Join-Path $ROOT "requirements-browser.txt")
-        Assert-LastExitCode "Failed to install requirements-browser.txt"
     }
 }
 

scripts/local-test.sh

@@ -84,7 +84,6 @@ cmd_venv() {
   if [[ ! -x "$VENV/bin/pytest" ]]; then
     log "Installing Python dependencies"
     "$VENV/bin/pip" install -q -r "$ROOT/requirements.txt"
-    "$VENV/bin/pip" install -q -r "$ROOT/requirements-browser.txt"
   fi
 }
 

src/website_profiling/crawl/fetchers/browser.py

@@ -15,7 +15,7 @@
 
 _BROWSER_INSTALL_MSG = (
     "JavaScript crawl requires Playwright and Chromium. Install: "
-    "pip install -r requirements-browser.txt. "
+    "pip install -r requirements.txt. "
     "Chrome or Chromium must be available (set CHROME_PATH if needed)."
 )
 

src/website_profiling/crawl/fetchers/browser_deps.py

@@ -67,11 +67,8 @@ def browser_status() -> dict[str, str | bool]:
 
 
 def _pip_install_browser_requirements() -> None:
-    req = _repo_root() / "requirements-browser.txt"
-    if not req.is_file():
-        raise RuntimeError(f"Missing {req.name}; cannot auto-install Playwright.")
     subprocess.run(
-        [sys.executable, "-m", "pip", "install", "-q", "-r", str(req)],
+        [sys.executable, "-m", "pip", "install", "-q", "playwright>=1.49.0"],
         check=True,
         cwd=_repo_root(),
     )

src/website_profiling/llm/agent.py

@@ -185,9 +185,9 @@ def on_token(text: str) -> None:
                 result = _react_step(client, llm_messages, _tools_description(compact=True), on_token)
         except Exception as e:
             msg = str(e).strip() or type(e).__name__
-            if "httpx" in msg.lower() or "requirements-llm" in msg.lower():
+            if "httpx" in msg.lower() or "requirements.txt" in msg.lower():
                 msg = (
-                    "LLM dependencies are missing. Run: pip install -r requirements-llm.txt "
+                    "LLM dependencies are missing. Run: pip install -r requirements.txt "
                     f"(or restart with ./local-run setup). Details: {msg}"
                 )
             _emit(on_event, {"type": "error", "message": msg})

src/website_profiling/llm/enrich.py

@@ -22,7 +22,7 @@
     SIMILAR_SYSTEM,
 )
 
-LLM_INSTALL_HINT = "Install LLM dependencies: pip install -r requirements-llm.txt"
+LLM_INSTALL_HINT = "Install LLM dependencies: pip install -r requirements.txt"
 
 
 def _cfg_bool(cfg: dict[str, str] | None, key: str, default: bool = False) -> bool:

src/website_profiling/llm/providers/anthropic.py

@@ -20,7 +20,7 @@ def complete_json(self, system: str, user: str) -> dict[str, Any]:
         try:
             import anthropic
         except ImportError as e:
-            raise ImportError("pip install anthropic (or requirements-llm.txt)") from e
+            raise ImportError("pip install -r requirements.txt") from e
 
         client = anthropic.Anthropic(api_key=self._api_key, timeout=self._timeout)
         msg = client.messages.create(
@@ -47,7 +47,7 @@ def chat_with_tools(
         try:
             import anthropic
         except ImportError as e:
-            raise ImportError("pip install anthropic (or requirements-llm.txt)") from e
+            raise ImportError("pip install -r requirements.txt") from e
 
         system_parts: list[str] = []
         anthropic_messages: list[dict[str, Any]] = []

src/website_profiling/llm/providers/gemini.py

@@ -18,7 +18,7 @@ def complete_json(self, system: str, user: str) -> dict[str, Any]:
         try:
             import httpx
         except ImportError as e:
-            raise ImportError("pip install httpx (or requirements-llm.txt)") from e
+            raise ImportError("pip install -r requirements.txt") from e
 
         url = f"https://generativelanguage.googleapis.com/v1beta/models/{self._model}:generateContent"
         payload = {

src/website_profiling/llm/providers/ollama.py

@@ -112,7 +112,7 @@ def _client(self):
         try:
             import httpx
         except ImportError as e:
-            raise ImportError("pip install httpx (or requirements-llm.txt)") from e
+            raise ImportError("pip install -r requirements.txt") from e
         return httpx.Client(timeout=self._timeout)
 
     def _raise_for_status(self, response: Any) -> None:

src/website_profiling/llm/providers/openai.py

@@ -21,7 +21,7 @@ def complete_json(self, system: str, user: str) -> dict[str, Any]:
         try:
             import httpx
         except ImportError as e:
-            raise ImportError("pip install httpx (or requirements-llm.txt)") from e
+            raise ImportError("pip install -r requirements.txt") from e
 
         payload = {
             "model": self._model,
@@ -53,7 +53,7 @@ def chat_with_tools(
         try:
             import httpx
         except ImportError as e:
-            raise ImportError("pip install httpx (or requirements-llm.txt)") from e
+            raise ImportError("pip install -r requirements.txt") from e
 
         payload: dict[str, Any] = {
             "model": self._model,

src/website_profiling/mcp/server.py

@@ -199,7 +199,7 @@ def main() -> None:
         from mcp.types import Resource, TextContent, Tool
     except ImportError as e:
         raise SystemExit(
-            "MCP SDK not installed. Run: pip install -r requirements-mcp.txt",
+            "MCP SDK not installed. Run: pip install -r requirements.txt",
         ) from e
 
     server = Server("site-audit")

src/website_profiling/reporting/optional_audits.py

@@ -100,7 +100,7 @@ def spell_check_issues(df: pd.DataFrame, *, max_pages: int = 50) -> tuple[list[d
     try:
         from spellchecker import SpellChecker  # type: ignore[import-untyped]
     except ImportError:
-        return issues, "pyspellchecker not installed (pip install -r requirements-optional.txt)"
+        return issues, "pyspellchecker not installed (pip install -r requirements.txt)"
     spell = SpellChecker()
     checked = 0
     for _, row in df.iterrows():

tests/test_crawl_fetchers.py

@@ -254,7 +254,7 @@ def test_repo_root_defaults_to_project_root(monkeypatch):
 
     monkeypatch.delenv("WEBSITE_PROFILING_ROOT", raising=False)
     root = browser_deps._repo_root()
-    assert (root / "requirements-browser.txt").is_file()
+    assert (root / "requirements.txt").is_file()
 
 
 def test_playwright_chromium_unavailable_without_playwright(monkeypatch):
@@ -345,12 +345,18 @@ def test_ensure_browser_deps_returns_ok_without_install(monkeypatch):
     assert status["ok"] is True
 
 
-def test_pip_install_browser_requirements_missing_file(monkeypatch, tmp_path):
+def test_pip_install_browser_requirements_runs_playwright_install(monkeypatch, tmp_path):
     from website_profiling.crawl.fetchers import browser_deps
 
     monkeypatch.setenv("WEBSITE_PROFILING_ROOT", str(tmp_path))
-    with pytest.raises(RuntimeError, match="Missing requirements-browser.txt"):
-        browser_deps._pip_install_browser_requirements()
+    called: list = []
+
+    def fake_run(cmd, **kwargs):
+        called.append(cmd)
+
+    monkeypatch.setattr(browser_deps.subprocess, "run", fake_run)
+    browser_deps._pip_install_browser_requirements()
+    assert called and "playwright>=1.49.0" in called[0]
 
 
 def test_ensure_browser_deps_reports_auto_install_failure(monkeypatch):

tests/test_fetchers_sitemap_config_unit.py

@@ -298,7 +298,6 @@ def close(self):
 def test_pip_install_browser_requirements_runs_subprocess(monkeypatch, tmp_path):
     from website_profiling.crawl.fetchers import browser_deps
 
-    (tmp_path / "requirements-browser.txt").write_text("playwright\n", encoding="utf-8")
     monkeypatch.setenv("WEBSITE_PROFILING_ROOT", str(tmp_path))
     called: list = []
 
@@ -307,7 +306,7 @@ def fake_run(cmd, **kwargs):
 
     monkeypatch.setattr(browser_deps.subprocess, "run", fake_run)
     browser_deps._pip_install_browser_requirements()
-    assert called and "requirements-browser.txt" in called[0][-1]
+    assert called and "playwright>=1.49.0" in called[0]
 
 
 def test_playwright_install_chromium_runs_subprocess(monkeypatch, tmp_path):

tests/test_mcp_server_helpers.py

@@ -190,9 +190,10 @@ def test_mcp_package_main(monkeypatch) -> None:
 def test_mcp_server_main_guard() -> None:
     # run_module executes __main__ in a fresh import; drop any prior import from this file.
     sys.modules.pop("website_profiling.mcp.server", None)
-    with pytest.raises(SystemExit, match="MCP SDK"):
-        runpy.run_module(
-            "website_profiling.mcp.server",
-            run_name="__main__",
-            alter_sys=False,
-        )
+    with patch.dict(sys.modules, {"mcp.server": None, "mcp.server.stdio": None, "mcp.types": None}):
+        with pytest.raises(SystemExit, match="MCP SDK"):
+            runpy.run_module(
+                "website_profiling.mcp.server",
+                run_name="__main__",
+                alter_sys=False,
+            )