No more static crawl

.github/workflows/ci.yml

@@ -44,6 +44,12 @@ jobs:
       - uses: actions/checkout@v4
       - name: Build image
         run: docker build -t website-profiling:ci .
+      - name: Browser crawl tests in image
+        run: |
+          docker run --rm \
+            -e DATABASE_URL=postgres://profiling:profiling@localhost:5432/website_profiling \
+            website-profiling:ci \
+            /opt/venv/bin/pytest tests/test_crawl_fetchers.py tests/test_crawler_browser_e2e.py -m browser -q -o addopts=
 
   web:
     runs-on: ubuntu-latest

AGENT.md

@@ -12,7 +12,9 @@
 - `web/app/` -- routes; `web/src/` -- React; pipeline: `PipelineRunnerFab`, `server/pipelineJobs.ts`, `server/pipelineConfig.ts`, `server/llmConfig.ts`, `server/db.ts`
 - `alembic/` -- schema migrations
 
-**Local dev:** `./local-run` (Postgres in Docker `wp-pg`, Next.js on host). See `scripts/local-run.sh`. **Local tests (CI parity):** `./local-test` — see `scripts/local-test.sh`.
+**Local dev:** `./local-run` (Postgres in Docker `wp-pg`, Next.js on host). See `scripts/local-run.sh`. **Local tests (CI parity):** `./local-test` (100% in-scope coverage gate); `./local-test browser` for `@pytest.mark.browser` integration tests — see `scripts/local-test.sh`. Mocked browser unit tests: `tests/test_browser_fetcher_unit.py`.
+
+**JavaScript crawl (optional):** Config keys `crawl_render_mode` (`static` | `javascript` | `auto`) and `crawl_js_*` in pipeline config / `pipelineConfigSchema.ts`. JS/auto crawls can capture browser console errors and uncaught exceptions (`crawl_js_capture_console`, stored under `page_analysis.browser`). **Auto mode** uses static-first fetch, pre-parse SPA heuristics (`needs_js_render`), then post-parse low-outlink fallback (`needs_js_render_after_parse`) in `crawler.py`. **Preflight:** `GET /api/crawl/browser-status` (localhost) spawns Python `browser_status()`; Run audit settings/run validation calls it when render mode is `javascript` or `auto`. Browser deps: `requirements-browser.txt` (installed by `./local-run setup` and `./local-test`). Runtime needs Chromium on `PATH` or `CHROME_PATH` (Docker sets `CHROME_PATH=/usr/bin/chromium`). Integration tests: `@pytest.mark.browser` — excluded by default in `pytest.ini`; Docker CI runs `tests/test_crawl_fetchers.py` and `tests/test_crawler_browser_e2e.py -m browser`; locally `./local-test browser`.
 
 **Run / APIs**
 
@@ -22,15 +24,15 @@
 - **`DATABASE_URL`** env: PostgreSQL connection string (required). **`DATA_DIR`**: secrets + shadow config (Docker: `/data`).
 - **Pipeline data** (crawl, edges, nodes, report payload, Lighthouse, keywords, warnings) is stored in **PostgreSQL only** — no JSON/CSV/HTML exports from the main pipeline.
 - **Pool tuning:** `DB_POOL_MIN` / `DB_POOL_MAX` (Python), `PGPOOL_MAX` (Node). Bulk crawl writes via `executemany`; optional **`crawl_stream_to_db`** streams rows during fetch.
-- **`web/`:** `/api/report/*` (PostgreSQL); `/api/run` spawns Python (localhost only); `/api/pipeline-config` GET/PUT; `/api/llm-config` GET/PUT (AI only); `PipelineRunnerFab` saves pipeline + LLM state before each run
+- **`web/`:** `/api/report/*` (PostgreSQL); `/api/run` spawns Python (localhost only); `/api/crawl/browser-status` GET (localhost, Playwright/Chromium preflight); `/api/pipeline-config` GET/PUT; `/api/llm-config` GET/PUT (AI only); `/api/properties/{id}/google/links/import` POST (GSC Links CSV); `PipelineRunnerFab` saves pipeline + LLM state before each run
 - **Job store:** in-memory on `globalThis` in `web/src/server/pipelineJobs.ts` — job status/log is lost on server restart (single-process dev/Docker only).
 - **Docker:** `Dockerfile` + `docker-compose.yml` (postgres + web); **`LIGHTHOUSE_CHROME_FLAGS`**
 
 **Where to edit**
 
 | Task | Where |
 |------|--------|
-| Crawl | `crawl/crawler.py` |
+| Crawl | `crawl/crawler.py`, `crawl/fetchers/` |
 | Report | `reporting/builder.py`, `reporting/categories.py` |
 | DB schema | `alembic/versions/` |
 | Local analysis | `analysis/local.py`, `requirements.txt` |

CONTRIBUTING.md

@@ -19,18 +19,21 @@ Thank you for helping improve this project. All contributions are welcome under
 
 Details: [README.md](README.md), [AGENT.md](AGENT.md).
 
+JavaScript/auto crawl needs Playwright (`requirements-browser.txt`, installed by `./local-run setup`) and Chromium on `PATH` or `CHROME_PATH`. Unit tests mock the browser fetcher; integration tests use `@pytest.mark.browser` and run in the Docker CI job (`tests/test_crawl_fetchers.py`, `tests/test_crawler_browser_e2e.py`). Locally: `./local-test browser` (skips gracefully if Chromium is missing).
+
 ## Running tests
 
 Match CI before opening a pull request:
 
 ```bash
 ./local-test              # full check (recommended)
 ./local-test python       # backend only
+./local-test browser      # JS crawl integration tests (skips if Chromium unavailable)
 ./local-test web            # frontend only
 ./local-test quick          # faster; DB must already be running
 ```
 
-CI runs Python tests (PostgreSQL + Alembic), web typecheck/lint/vitest, CLI smoke, and a Docker build (see [.github/workflows/ci.yml](.github/workflows/ci.yml)).
+CI runs Python tests (PostgreSQL + Alembic, 80% coverage gate), web typecheck/lint/vitest, CLI smoke, and a Docker build that also runs browser-marked pytest inside the image (see [.github/workflows/ci.yml](.github/workflows/ci.yml)).
 
 ## How to contribute
 

Dockerfile

@@ -43,13 +43,15 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
 # Python: base requirements + optional LLM API clients
 COPY requirements.txt /app/requirements.txt
 COPY requirements-llm.txt /app/requirements-llm.txt
+COPY requirements-browser.txt /app/requirements-browser.txt
 COPY alembic.ini /app/alembic.ini
 COPY alembic /app/alembic
 RUN --mount=type=cache,target=/root/.cache/pip \
     python3 -m venv /opt/venv \
   && /opt/venv/bin/pip install --upgrade pip \
   && /opt/venv/bin/pip install -r /app/requirements.txt \
   && /opt/venv/bin/pip install -r /app/requirements-llm.txt \
+  && /opt/venv/bin/pip install -r /app/requirements-browser.txt \
   && ln -sf /opt/venv/bin/python /usr/local/bin/python \
   && ln -sf /opt/venv/bin/python /usr/local/bin/python3
 
@@ -66,7 +68,9 @@ RUN --mount=type=cache,target=/root/.npm \
     cd /app/web && npm ci
 
 # Application source
+COPY pytest.ini /app/pytest.ini
 COPY src /app/src
+COPY tests /app/tests
 COPY web /app/web
 COPY alembic /app/alembic
 COPY alembic.ini /app/alembic.ini

README.md

@@ -32,10 +32,11 @@ Open [http://localhost:3000/home](http://localhost:3000/home).
 
 ```bash
 ./local-test              # before push: full CI parity (DB + pytest + web)
-./local-test python         # backend only: pytest + CLI smoke
-./local-test web            # frontend only: typecheck, lint, vitest
-./local-test quick          # fast loop: skip Docker start; needs DB already up
-./local-test all --no-cov   # full run without pytest coverage gate
+./local-test python       # backend: pytest (80% coverage) + browser pytest + CLI smoke
+./local-test browser      # JS crawl integration tests (skips if Chromium unavailable)
+./local-test web          # frontend: typecheck, lint, vitest
+./local-test quick        # fast loop; needs DB already up (no coverage gate)
+./local-test all --no-cov # full run without pytest coverage gate
 ```
 
 ## Contributing
@@ -53,6 +54,8 @@ Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for setup and
 
 Google Search Console / Analytics: connect via **Integrations** (gear icon) in the app.
 
+**JavaScript crawl (optional):** In Audit settings, set **Crawl rendering** to `javascript` (always headless Chromium) or `auto` (static first, browser when SPA heuristics match). Install locally: `pip install -r requirements-browser.txt` and Chromium on `PATH` or `CHROME_PATH` (included in Docker). The UI preflights via `GET /api/crawl/browser-status` before runs when JS/auto is selected.
+
 Production: `docker-compose.prod.yml` (set `POSTGRES_PASSWORD`, `AUTH_SECRET`).
 
 ## License

alembic/versions/008_crawl_render_mode.py

@@ -0,0 +1,24 @@
+"""Add render_mode to crawl_runs for audit provenance."""
+
+from alembic import op
+
+revision = "008_crawl_render_mode"
+down_revision = "007_keyword_property_id"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.execute(
+        """
+        ALTER TABLE crawl_runs ADD COLUMN IF NOT EXISTS render_mode TEXT DEFAULT 'static';
+        """
+    )
+
+
+def downgrade() -> None:
+    op.execute(
+        """
+        ALTER TABLE crawl_runs DROP COLUMN IF EXISTS render_mode;
+        """
+    )

alembic/versions/009_crawl_results_fetch_method.py

@@ -0,0 +1,30 @@
+"""Add fetch_method column to crawl_results for SQL-level filtering."""
+
+from alembic import op
+
+revision = "009_crawl_results_fetch_method"
+down_revision = "008_crawl_render_mode"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.execute(
+        """
+        ALTER TABLE crawl_results ADD COLUMN IF NOT EXISTS fetch_method TEXT DEFAULT 'static';
+        CREATE INDEX IF NOT EXISTS idx_crawl_results_run_fetch_method
+            ON crawl_results (crawl_run_id, fetch_method);
+        UPDATE crawl_results
+        SET fetch_method = COALESCE(data->>'fetch_method', 'static')
+        WHERE fetch_method IS NULL OR fetch_method = 'static';
+        """
+    )
+
+
+def downgrade() -> None:
+    op.execute(
+        """
+        DROP INDEX IF EXISTS idx_crawl_results_run_fetch_method;
+        ALTER TABLE crawl_results DROP COLUMN IF EXISTS fetch_method;
+        """
+    )

alembic/versions/010_gsc_links_data.py

@@ -0,0 +1,33 @@
+"""Property-scoped GSC Links CSV import snapshots."""
+
+from alembic import op
+
+revision = "010_gsc_links_data"
+down_revision = "009_crawl_results_fetch_method"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.execute(
+        """
+        CREATE TABLE IF NOT EXISTS gsc_links_data (
+            id BIGSERIAL PRIMARY KEY,
+            fetched_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+            property_id BIGINT NOT NULL REFERENCES properties(id) ON DELETE CASCADE,
+            data JSONB NOT NULL
+        );
+
+        CREATE INDEX IF NOT EXISTS idx_gsc_links_data_property_fetched
+            ON gsc_links_data (property_id, fetched_at DESC);
+        """
+    )
+
+
+def downgrade() -> None:
+    op.execute(
+        """
+        DROP INDEX IF EXISTS idx_gsc_links_data_property_fetched;
+        DROP TABLE IF EXISTS gsc_links_data CASCADE;
+        """
+    )

docs/COMPANY_STANDARDS.md

@@ -16,8 +16,11 @@ Audit category scores (0–100) are **internal audit scores**, not Google rankin
 
 ## Crawl limitations
 
-- Crawl uses **HTTP GET + static HTML parsing** only (no JavaScript execution). See [Docs.md](../Docs.md).
-- Client-rendered links and SPAs may be under-represented; reports must show crawl scope (pages crawled vs limit, robots blocks).
+- Default crawl uses **HTTP GET + static HTML parsing** (no JavaScript execution). `crawl_render_mode = static` (default).
+- Optional **JavaScript rendering** (`crawl_render_mode = javascript`) loads every page in headless Chromium before parsing — slower (~10–20×) and heavier on memory, but required for many React, Vue, Next.js, Angular, Svelte, and Shopify themes.
+- **Auto rendering** (`crawl_render_mode = auto`) fetches static HTML first, then uses browser fallback when SPA shell heuristics or low outlink counts suggest client-rendered content. Per-page `fetch_method` (`static` vs `rendered`) is stored on crawl rows for provenance.
+- Client-rendered links and SPAs may be under-represented in static-only mode; reports show crawl scope (pages crawled vs limit, robots blocks, render mode, browser diagnostic counts when applicable).
+- JS and auto modes require Playwright + Chromium; the Run audit UI checks availability via `GET /api/crawl/browser-status` before starting a job.
 - Only crawl sites you are **authorized** to test. Respect `robots.txt` unless an admin explicitly overrides for owned properties.
 
 ## Security scanning

docs/GLOSSARY.md

@@ -18,6 +18,7 @@ UI terms agencies recognize, mapped to internal keys and data sources.
 | Technologies | `tech-stack` | Wappalyzer-style detection | BuiltWith |
 | Crawl summary | `charts` | Crawl aggregates | SF overview |
 | Internal links | `network` | Link graph | Ahrefs Internal Links |
+| Backlinks | `backlinks`, `gsc_links`, `gsc_links_data` | GSC Links CSV import (Google sample) | GSC Links report |
 | Page previews | `gallery` | Crawl excerpts | Visual QA |
 | Search Console | `search-performance`, `google_data` (scoped by `property_id`) | GSC API per property | Google Search Console |
 | Analytics (GA4) | `traffic`, `google_data` (scoped by `property_id`) | GA4 API per property | Google Analytics |
@@ -32,7 +33,10 @@ UI terms agencies recognize, mapped to internal keys and data sources.
 | Inlinks | `inlinks` | Crawl graph |
 | Outlinks | `outlinks` | Crawl graph |
 | Status code | `status` | HTTP |
+| Crawl rendering | `crawl_render_mode` on run; `fetch_method` per URL | `static`, `javascript`, or `auto` crawl config; `static` vs `rendered` per page |
 | Impressions | `gsc_impressions` | Search Console |
+| Referring domains | `top_linking_sites` | GSC Links CSV import |
+| External links to site | `sample_links`, `latest_links` | GSC Links CSV import |
 | Clicks | `gsc_clicks` | Search Console |
 | CTR | `gsc_ctr` | Search Console |
 | Average position | `gsc_position` | Search Console |

input.txt.example

@@ -18,6 +18,17 @@ content_excerpt_max_chars = 4096
 preserve_crawl_history = true
 crawl_stream_to_db = false
 crawl_exclude_urls = 
+# crawl_render_mode: static | javascript | auto (auto = static first, browser when SPA heuristics match)
+crawl_render_mode = static
+crawl_js_concurrency = 3
+crawl_js_timeout = 30
+crawl_js_wait_until = domcontentloaded
+crawl_js_extra_wait_ms = 1500
+crawl_js_block_resources = true
+crawl_js_capture_console = true
+crawl_js_console_levels = error,warning
+crawl_js_capture_failed_requests = false
+crawl_js_console_max_per_page = 20
 
 # --- Report ---
 outbound_domain_max_rows = 200

pipeline-config.example.txt

@@ -19,6 +19,17 @@ content_excerpt_max_chars = 4096
 preserve_crawl_history = true
 crawl_stream_to_db = false
 crawl_exclude_urls = 
+# crawl_render_mode: static | javascript | auto
+crawl_render_mode = static
+crawl_js_concurrency = 3
+crawl_js_timeout = 30
+crawl_js_wait_until = domcontentloaded
+crawl_js_extra_wait_ms = 1500
+crawl_js_block_resources = true
+crawl_js_capture_console = true
+crawl_js_console_levels = error,warning
+crawl_js_capture_failed_requests = false
+crawl_js_console_max_per_page = 20
 
 # --- Report ---
 outbound_domain_max_rows = 200

pytest.ini

@@ -1,8 +1,11 @@
 [pytest]
 pythonpath = src
 testpaths = tests
+markers =
+    browser: integration tests requiring Chromium/Playwright (deselect with '-m "not browser"')
 addopts =
     --cov=website_profiling
     --cov-config=.coveragerc
     --cov-report=term-missing
-    --cov-fail-under=80
+    --cov-fail-under=100
+    -m "not browser"

requirements-browser.txt

@@ -0,0 +1,2 @@
+# Optional: JavaScript rendering crawl (headless Chromium via Playwright)
+playwright>=1.49.0

scripts/local-run.sh

@@ -23,6 +23,7 @@ export DATABASE_URL="${DATABASE_URL:-postgres://${PG_USER}:${PG_PASSWORD}@127.0.
 export DATA_DIR="${DATA_DIR:-$ROOT/data}"
 export PYTHON="${PYTHON:-$ROOT/.venv/bin/python}"
 export WEBSITE_PROFILING_ROOT="$ROOT"
+export PYTHONPATH="${PYTHONPATH:+$PYTHONPATH:}$ROOT/src"
 
 VENV="$ROOT/.venv"
 WEB="$ROOT/web"
@@ -82,6 +83,7 @@ cmd_venv() {
   fi
   log "Installing Python dependencies"
   "$VENV/bin/pip" install -q -r "$ROOT/requirements.txt"
+  "$VENV/bin/pip" install -q -r "$ROOT/requirements-browser.txt"
 }
 
 cmd_migrate() {
@@ -99,10 +101,25 @@ cmd_web_deps() {
   fi
 }
 
+cmd_browser_deps() {
+  [[ -x "$VENV/bin/python" ]] || cmd_venv
+  log "Ensuring Playwright + Chromium for JS crawl"
+  if ! "$VENV/bin/python" -c "
+from website_profiling.crawl.fetchers import ensure_browser_deps
+import json, sys
+status = ensure_browser_deps()
+print(json.dumps(status))
+sys.exit(0 if status.get('ok') else 1)
+"; then
+    warn "Browser deps unavailable — JS/auto crawl disabled until Playwright + Chromium install successfully"
+  fi
+}
+
 cmd_setup() {
   mkdir -p "$DATA_DIR"
   cmd_db
   cmd_venv
+  cmd_browser_deps
   cmd_migrate
   cmd_web_deps
   log "Setup complete."
@@ -114,6 +131,7 @@ cmd_start() {
   mkdir -p "$DATA_DIR"
   cmd_db
   [[ -x "$VENV/bin/alembic" ]] || cmd_venv
+  cmd_browser_deps
   log "Ensuring migrations are up to date"
   "$VENV/bin/alembic" upgrade head
   cmd_web_deps
@@ -122,7 +140,7 @@ cmd_start() {
   log "DATA_DIR=$DATA_DIR"
   log "PYTHON=$PYTHON"
   cd "$WEB"
-  export DATABASE_URL DATA_DIR PYTHON WEBSITE_PROFILING_ROOT
+  export DATABASE_URL DATA_DIR PYTHON WEBSITE_PROFILING_ROOT PYTHONPATH
   exec npm run dev
 }
 
@@ -155,7 +173,7 @@ Environment overrides (optional):
 After start, open: http://localhost:3000/home
 Run audits via sidebar "Run audit" (bottom-right FAB).
 
-Run CI-style tests: ./local-test (see ./local-test help).
+Run CI-style tests: ./local-test (see ./local-test help). JS crawl integration: ./local-test browser.
 EOF
 }