diff --git a/.coverage b/.coverage index fea11124..fb5527cf 100644 Binary files a/.coverage and b/.coverage differ diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index da82f0fa..82a03f55 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -44,6 +44,12 @@ jobs: - uses: actions/checkout@v4 - name: Build image run: docker build -t website-profiling:ci . + - name: Browser crawl tests in image + run: | + docker run --rm \ + -e DATABASE_URL=postgres://profiling:profiling@localhost:5432/website_profiling \ + website-profiling:ci \ + /opt/venv/bin/pytest tests/test_crawl_fetchers.py tests/test_crawler_browser_e2e.py -m browser -q -o addopts= web: runs-on: ubuntu-latest diff --git a/AGENT.md b/AGENT.md index e0677ed5..7c0b7a61 100644 --- a/AGENT.md +++ b/AGENT.md @@ -12,7 +12,9 @@ - `web/app/` -- routes; `web/src/` -- React; pipeline: `PipelineRunnerFab`, `server/pipelineJobs.ts`, `server/pipelineConfig.ts`, `server/llmConfig.ts`, `server/db.ts` - `alembic/` -- schema migrations -**Local dev:** `./local-run` (Postgres in Docker `wp-pg`, Next.js on host). See `scripts/local-run.sh`. **Local tests (CI parity):** `./local-test` — see `scripts/local-test.sh`. +**Local dev:** `./local-run` (Postgres in Docker `wp-pg`, Next.js on host). See `scripts/local-run.sh`. **Local tests (CI parity):** `./local-test` (100% in-scope coverage gate); `./local-test browser` for `@pytest.mark.browser` integration tests — see `scripts/local-test.sh`. Mocked browser unit tests: `tests/test_browser_fetcher_unit.py`. + +**JavaScript crawl (optional):** Config keys `crawl_render_mode` (`static` | `javascript` | `auto`) and `crawl_js_*` in pipeline config / `pipelineConfigSchema.ts`. JS/auto crawls can capture browser console errors and uncaught exceptions (`crawl_js_capture_console`, stored under `page_analysis.browser`). **Auto mode** uses static-first fetch, pre-parse SPA heuristics (`needs_js_render`), then post-parse low-outlink fallback (`needs_js_render_after_parse`) in `crawler.py`. **Preflight:** `GET /api/crawl/browser-status` (localhost) spawns Python `browser_status()`; Run audit settings/run validation calls it when render mode is `javascript` or `auto`. Browser deps: `requirements-browser.txt` (installed by `./local-run setup` and `./local-test`). Runtime needs Chromium on `PATH` or `CHROME_PATH` (Docker sets `CHROME_PATH=/usr/bin/chromium`). Integration tests: `@pytest.mark.browser` — excluded by default in `pytest.ini`; Docker CI runs `tests/test_crawl_fetchers.py` and `tests/test_crawler_browser_e2e.py -m browser`; locally `./local-test browser`. **Run / APIs** @@ -22,7 +24,7 @@ - **`DATABASE_URL`** env: PostgreSQL connection string (required). **`DATA_DIR`**: secrets + shadow config (Docker: `/data`). - **Pipeline data** (crawl, edges, nodes, report payload, Lighthouse, keywords, warnings) is stored in **PostgreSQL only** — no JSON/CSV/HTML exports from the main pipeline. - **Pool tuning:** `DB_POOL_MIN` / `DB_POOL_MAX` (Python), `PGPOOL_MAX` (Node). Bulk crawl writes via `executemany`; optional **`crawl_stream_to_db`** streams rows during fetch. -- **`web/`:** `/api/report/*` (PostgreSQL); `/api/run` spawns Python (localhost only); `/api/pipeline-config` GET/PUT; `/api/llm-config` GET/PUT (AI only); `PipelineRunnerFab` saves pipeline + LLM state before each run +- **`web/`:** `/api/report/*` (PostgreSQL); `/api/run` spawns Python (localhost only); `/api/crawl/browser-status` GET (localhost, Playwright/Chromium preflight); `/api/pipeline-config` GET/PUT; `/api/llm-config` GET/PUT (AI only); `/api/properties/{id}/google/links/import` POST (GSC Links CSV); `PipelineRunnerFab` saves pipeline + LLM state before each run - **Job store:** in-memory on `globalThis` in `web/src/server/pipelineJobs.ts` — job status/log is lost on server restart (single-process dev/Docker only). - **Docker:** `Dockerfile` + `docker-compose.yml` (postgres + web); **`LIGHTHOUSE_CHROME_FLAGS`** @@ -30,7 +32,7 @@ | Task | Where | |------|--------| -| Crawl | `crawl/crawler.py` | +| Crawl | `crawl/crawler.py`, `crawl/fetchers/` | | Report | `reporting/builder.py`, `reporting/categories.py` | | DB schema | `alembic/versions/` | | Local analysis | `analysis/local.py`, `requirements.txt` | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8ac51803..4c858768 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -19,6 +19,8 @@ Thank you for helping improve this project. All contributions are welcome under Details: [README.md](README.md), [AGENT.md](AGENT.md). +JavaScript/auto crawl needs Playwright (`requirements-browser.txt`, installed by `./local-run setup`) and Chromium on `PATH` or `CHROME_PATH`. Unit tests mock the browser fetcher; integration tests use `@pytest.mark.browser` and run in the Docker CI job (`tests/test_crawl_fetchers.py`, `tests/test_crawler_browser_e2e.py`). Locally: `./local-test browser` (skips gracefully if Chromium is missing). + ## Running tests Match CI before opening a pull request: @@ -26,11 +28,12 @@ Match CI before opening a pull request: ```bash ./local-test # full check (recommended) ./local-test python # backend only +./local-test browser # JS crawl integration tests (skips if Chromium unavailable) ./local-test web # frontend only ./local-test quick # faster; DB must already be running ``` -CI runs Python tests (PostgreSQL + Alembic), web typecheck/lint/vitest, CLI smoke, and a Docker build (see [.github/workflows/ci.yml](.github/workflows/ci.yml)). +CI runs Python tests (PostgreSQL + Alembic, 80% coverage gate), web typecheck/lint/vitest, CLI smoke, and a Docker build that also runs browser-marked pytest inside the image (see [.github/workflows/ci.yml](.github/workflows/ci.yml)). ## How to contribute diff --git a/Dockerfile b/Dockerfile index b6f0dd9c..3557010a 100644 --- a/Dockerfile +++ b/Dockerfile @@ -43,6 +43,7 @@ ENV PYTHONDONTWRITEBYTECODE=1 \ # Python: base requirements + optional LLM API clients COPY requirements.txt /app/requirements.txt COPY requirements-llm.txt /app/requirements-llm.txt +COPY requirements-browser.txt /app/requirements-browser.txt COPY alembic.ini /app/alembic.ini COPY alembic /app/alembic RUN --mount=type=cache,target=/root/.cache/pip \ @@ -50,6 +51,7 @@ RUN --mount=type=cache,target=/root/.cache/pip \ && /opt/venv/bin/pip install --upgrade pip \ && /opt/venv/bin/pip install -r /app/requirements.txt \ && /opt/venv/bin/pip install -r /app/requirements-llm.txt \ + && /opt/venv/bin/pip install -r /app/requirements-browser.txt \ && ln -sf /opt/venv/bin/python /usr/local/bin/python \ && ln -sf /opt/venv/bin/python /usr/local/bin/python3 @@ -66,7 +68,9 @@ RUN --mount=type=cache,target=/root/.npm \ cd /app/web && npm ci # Application source +COPY pytest.ini /app/pytest.ini COPY src /app/src +COPY tests /app/tests COPY web /app/web COPY alembic /app/alembic COPY alembic.ini /app/alembic.ini diff --git a/README.md b/README.md index dd7a6268..d1da21fa 100644 --- a/README.md +++ b/README.md @@ -32,10 +32,11 @@ Open [http://localhost:3000/home](http://localhost:3000/home). ```bash ./local-test # before push: full CI parity (DB + pytest + web) -./local-test python # backend only: pytest + CLI smoke -./local-test web # frontend only: typecheck, lint, vitest -./local-test quick # fast loop: skip Docker start; needs DB already up -./local-test all --no-cov # full run without pytest coverage gate +./local-test python # backend: pytest (80% coverage) + browser pytest + CLI smoke +./local-test browser # JS crawl integration tests (skips if Chromium unavailable) +./local-test web # frontend: typecheck, lint, vitest +./local-test quick # fast loop; needs DB already up (no coverage gate) +./local-test all --no-cov # full run without pytest coverage gate ``` ## Contributing @@ -53,6 +54,8 @@ Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for setup and Google Search Console / Analytics: connect via **Integrations** (gear icon) in the app. +**JavaScript crawl (optional):** In Audit settings, set **Crawl rendering** to `javascript` (always headless Chromium) or `auto` (static first, browser when SPA heuristics match). Install locally: `pip install -r requirements-browser.txt` and Chromium on `PATH` or `CHROME_PATH` (included in Docker). The UI preflights via `GET /api/crawl/browser-status` before runs when JS/auto is selected. + Production: `docker-compose.prod.yml` (set `POSTGRES_PASSWORD`, `AUTH_SECRET`). ## License diff --git a/alembic/versions/008_crawl_render_mode.py b/alembic/versions/008_crawl_render_mode.py new file mode 100644 index 00000000..5c9e0f4a --- /dev/null +++ b/alembic/versions/008_crawl_render_mode.py @@ -0,0 +1,24 @@ +"""Add render_mode to crawl_runs for audit provenance.""" + +from alembic import op + +revision = "008_crawl_render_mode" +down_revision = "007_keyword_property_id" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + op.execute( + """ + ALTER TABLE crawl_runs ADD COLUMN IF NOT EXISTS render_mode TEXT DEFAULT 'static'; + """ + ) + + +def downgrade() -> None: + op.execute( + """ + ALTER TABLE crawl_runs DROP COLUMN IF EXISTS render_mode; + """ + ) diff --git a/alembic/versions/009_crawl_results_fetch_method.py b/alembic/versions/009_crawl_results_fetch_method.py new file mode 100644 index 00000000..1ce0b19d --- /dev/null +++ b/alembic/versions/009_crawl_results_fetch_method.py @@ -0,0 +1,30 @@ +"""Add fetch_method column to crawl_results for SQL-level filtering.""" + +from alembic import op + +revision = "009_crawl_results_fetch_method" +down_revision = "008_crawl_render_mode" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + op.execute( + """ + ALTER TABLE crawl_results ADD COLUMN IF NOT EXISTS fetch_method TEXT DEFAULT 'static'; + CREATE INDEX IF NOT EXISTS idx_crawl_results_run_fetch_method + ON crawl_results (crawl_run_id, fetch_method); + UPDATE crawl_results + SET fetch_method = COALESCE(data->>'fetch_method', 'static') + WHERE fetch_method IS NULL OR fetch_method = 'static'; + """ + ) + + +def downgrade() -> None: + op.execute( + """ + DROP INDEX IF EXISTS idx_crawl_results_run_fetch_method; + ALTER TABLE crawl_results DROP COLUMN IF EXISTS fetch_method; + """ + ) diff --git a/alembic/versions/010_gsc_links_data.py b/alembic/versions/010_gsc_links_data.py new file mode 100644 index 00000000..8471aa8c --- /dev/null +++ b/alembic/versions/010_gsc_links_data.py @@ -0,0 +1,33 @@ +"""Property-scoped GSC Links CSV import snapshots.""" + +from alembic import op + +revision = "010_gsc_links_data" +down_revision = "009_crawl_results_fetch_method" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + op.execute( + """ + CREATE TABLE IF NOT EXISTS gsc_links_data ( + id BIGSERIAL PRIMARY KEY, + fetched_at TIMESTAMPTZ NOT NULL DEFAULT now(), + property_id BIGINT NOT NULL REFERENCES properties(id) ON DELETE CASCADE, + data JSONB NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_gsc_links_data_property_fetched + ON gsc_links_data (property_id, fetched_at DESC); + """ + ) + + +def downgrade() -> None: + op.execute( + """ + DROP INDEX IF EXISTS idx_gsc_links_data_property_fetched; + DROP TABLE IF EXISTS gsc_links_data CASCADE; + """ + ) diff --git a/docs/COMPANY_STANDARDS.md b/docs/COMPANY_STANDARDS.md index a220027a..6ca07285 100644 --- a/docs/COMPANY_STANDARDS.md +++ b/docs/COMPANY_STANDARDS.md @@ -16,8 +16,11 @@ Audit category scores (0–100) are **internal audit scores**, not Google rankin ## Crawl limitations -- Crawl uses **HTTP GET + static HTML parsing** only (no JavaScript execution). See [Docs.md](../Docs.md). -- Client-rendered links and SPAs may be under-represented; reports must show crawl scope (pages crawled vs limit, robots blocks). +- Default crawl uses **HTTP GET + static HTML parsing** (no JavaScript execution). `crawl_render_mode = static` (default). +- Optional **JavaScript rendering** (`crawl_render_mode = javascript`) loads every page in headless Chromium before parsing — slower (~10–20×) and heavier on memory, but required for many React, Vue, Next.js, Angular, Svelte, and Shopify themes. +- **Auto rendering** (`crawl_render_mode = auto`) fetches static HTML first, then uses browser fallback when SPA shell heuristics or low outlink counts suggest client-rendered content. Per-page `fetch_method` (`static` vs `rendered`) is stored on crawl rows for provenance. +- Client-rendered links and SPAs may be under-represented in static-only mode; reports show crawl scope (pages crawled vs limit, robots blocks, render mode, browser diagnostic counts when applicable). +- JS and auto modes require Playwright + Chromium; the Run audit UI checks availability via `GET /api/crawl/browser-status` before starting a job. - Only crawl sites you are **authorized** to test. Respect `robots.txt` unless an admin explicitly overrides for owned properties. ## Security scanning diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index 142440ea..663fb7a1 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -18,6 +18,7 @@ UI terms agencies recognize, mapped to internal keys and data sources. | Technologies | `tech-stack` | Wappalyzer-style detection | BuiltWith | | Crawl summary | `charts` | Crawl aggregates | SF overview | | Internal links | `network` | Link graph | Ahrefs Internal Links | +| Backlinks | `backlinks`, `gsc_links`, `gsc_links_data` | GSC Links CSV import (Google sample) | GSC Links report | | Page previews | `gallery` | Crawl excerpts | Visual QA | | Search Console | `search-performance`, `google_data` (scoped by `property_id`) | GSC API per property | Google Search Console | | Analytics (GA4) | `traffic`, `google_data` (scoped by `property_id`) | GA4 API per property | Google Analytics | @@ -32,7 +33,10 @@ UI terms agencies recognize, mapped to internal keys and data sources. | Inlinks | `inlinks` | Crawl graph | | Outlinks | `outlinks` | Crawl graph | | Status code | `status` | HTTP | +| Crawl rendering | `crawl_render_mode` on run; `fetch_method` per URL | `static`, `javascript`, or `auto` crawl config; `static` vs `rendered` per page | | Impressions | `gsc_impressions` | Search Console | +| Referring domains | `top_linking_sites` | GSC Links CSV import | +| External links to site | `sample_links`, `latest_links` | GSC Links CSV import | | Clicks | `gsc_clicks` | Search Console | | CTR | `gsc_ctr` | Search Console | | Average position | `gsc_position` | Search Console | diff --git a/input.txt.example b/input.txt.example index e5259a3f..c829d07f 100644 --- a/input.txt.example +++ b/input.txt.example @@ -18,6 +18,17 @@ content_excerpt_max_chars = 4096 preserve_crawl_history = true crawl_stream_to_db = false crawl_exclude_urls = +# crawl_render_mode: static | javascript | auto (auto = static first, browser when SPA heuristics match) +crawl_render_mode = static +crawl_js_concurrency = 3 +crawl_js_timeout = 30 +crawl_js_wait_until = domcontentloaded +crawl_js_extra_wait_ms = 1500 +crawl_js_block_resources = true +crawl_js_capture_console = true +crawl_js_console_levels = error,warning +crawl_js_capture_failed_requests = false +crawl_js_console_max_per_page = 20 # --- Report --- outbound_domain_max_rows = 200 diff --git a/pipeline-config.example.txt b/pipeline-config.example.txt index 9ce7acf2..b7e5b7f2 100644 --- a/pipeline-config.example.txt +++ b/pipeline-config.example.txt @@ -19,6 +19,17 @@ content_excerpt_max_chars = 4096 preserve_crawl_history = true crawl_stream_to_db = false crawl_exclude_urls = +# crawl_render_mode: static | javascript | auto +crawl_render_mode = static +crawl_js_concurrency = 3 +crawl_js_timeout = 30 +crawl_js_wait_until = domcontentloaded +crawl_js_extra_wait_ms = 1500 +crawl_js_block_resources = true +crawl_js_capture_console = true +crawl_js_console_levels = error,warning +crawl_js_capture_failed_requests = false +crawl_js_console_max_per_page = 20 # --- Report --- outbound_domain_max_rows = 200 diff --git a/pytest.ini b/pytest.ini index ccbe3d7f..3514d814 100644 --- a/pytest.ini +++ b/pytest.ini @@ -1,8 +1,11 @@ [pytest] pythonpath = src testpaths = tests +markers = + browser: integration tests requiring Chromium/Playwright (deselect with '-m "not browser"') addopts = --cov=website_profiling --cov-config=.coveragerc --cov-report=term-missing - --cov-fail-under=80 + --cov-fail-under=100 + -m "not browser" diff --git a/requirements-browser.txt b/requirements-browser.txt new file mode 100644 index 00000000..4415efe0 --- /dev/null +++ b/requirements-browser.txt @@ -0,0 +1,2 @@ +# Optional: JavaScript rendering crawl (headless Chromium via Playwright) +playwright>=1.49.0 diff --git a/scripts/local-run.sh b/scripts/local-run.sh index 6b1fb55c..b32b7410 100755 --- a/scripts/local-run.sh +++ b/scripts/local-run.sh @@ -23,6 +23,7 @@ export DATABASE_URL="${DATABASE_URL:-postgres://${PG_USER}:${PG_PASSWORD}@127.0. export DATA_DIR="${DATA_DIR:-$ROOT/data}" export PYTHON="${PYTHON:-$ROOT/.venv/bin/python}" export WEBSITE_PROFILING_ROOT="$ROOT" +export PYTHONPATH="${PYTHONPATH:+$PYTHONPATH:}$ROOT/src" VENV="$ROOT/.venv" WEB="$ROOT/web" @@ -82,6 +83,7 @@ cmd_venv() { fi log "Installing Python dependencies" "$VENV/bin/pip" install -q -r "$ROOT/requirements.txt" + "$VENV/bin/pip" install -q -r "$ROOT/requirements-browser.txt" } cmd_migrate() { @@ -99,10 +101,25 @@ cmd_web_deps() { fi } +cmd_browser_deps() { + [[ -x "$VENV/bin/python" ]] || cmd_venv + log "Ensuring Playwright + Chromium for JS crawl" + if ! "$VENV/bin/python" -c " +from website_profiling.crawl.fetchers import ensure_browser_deps +import json, sys +status = ensure_browser_deps() +print(json.dumps(status)) +sys.exit(0 if status.get('ok') else 1) +"; then + warn "Browser deps unavailable — JS/auto crawl disabled until Playwright + Chromium install successfully" + fi +} + cmd_setup() { mkdir -p "$DATA_DIR" cmd_db cmd_venv + cmd_browser_deps cmd_migrate cmd_web_deps log "Setup complete." @@ -114,6 +131,7 @@ cmd_start() { mkdir -p "$DATA_DIR" cmd_db [[ -x "$VENV/bin/alembic" ]] || cmd_venv + cmd_browser_deps log "Ensuring migrations are up to date" "$VENV/bin/alembic" upgrade head cmd_web_deps @@ -122,7 +140,7 @@ cmd_start() { log "DATA_DIR=$DATA_DIR" log "PYTHON=$PYTHON" cd "$WEB" - export DATABASE_URL DATA_DIR PYTHON WEBSITE_PROFILING_ROOT + export DATABASE_URL DATA_DIR PYTHON WEBSITE_PROFILING_ROOT PYTHONPATH exec npm run dev } @@ -155,7 +173,7 @@ Environment overrides (optional): After start, open: http://localhost:3000/home Run audits via sidebar "Run audit" (bottom-right FAB). -Run CI-style tests: ./local-test (see ./local-test help). +Run CI-style tests: ./local-test (see ./local-test help). JS crawl integration: ./local-test browser. EOF } diff --git a/scripts/local-test.sh b/scripts/local-test.sh index 324b539a..8d0172d4 100755 --- a/scripts/local-test.sh +++ b/scripts/local-test.sh @@ -21,6 +21,7 @@ PG_DB="${WP_PG_DB:-website_profiling}" export DATABASE_URL="${DATABASE_URL:-postgres://${PG_USER}:${PG_PASSWORD}@127.0.0.1:${PG_PORT}/${PG_DB}}" export DATA_DIR="${DATA_DIR:-$ROOT/data}" export WEBSITE_PROFILING_ROOT="$ROOT" +export PYTHONPATH="${PYTHONPATH:+$PYTHONPATH:}$ROOT/src" VENV="$ROOT/.venv" WEB="$ROOT/web" @@ -83,6 +84,7 @@ cmd_venv() { if [[ ! -x "$VENV/bin/pytest" ]]; then log "Installing Python dependencies" "$VENV/bin/pip" install -q -r "$ROOT/requirements.txt" + "$VENV/bin/pip" install -q -r "$ROOT/requirements-browser.txt" fi } @@ -105,21 +107,37 @@ run_pytest() { log "Pytest (tests/ -q --no-cov)" "$VENV/bin/pytest" tests/ -q --no-cov else - log "Pytest (tests/ -q, 80% coverage gate — same as CI)" + log "Pytest (tests/ -q, 100% in-scope coverage gate — same as CI)" "$VENV/bin/pytest" tests/ -q fi } +run_browser_pytest() { + if "$VENV/bin/python" -c "from website_profiling.crawl.fetchers import browser_status; import sys; sys.exit(0 if browser_status().get('ok') else 1)" 2>/dev/null; then + log "Browser pytest (tests/test_crawl_fetchers.py tests/test_crawler_browser_e2e.py -m browser)" + "$VENV/bin/pytest" tests/test_crawl_fetchers.py tests/test_crawler_browser_e2e.py -m browser -q --no-cov + else + warn "Chromium unavailable — skipping browser integration tests" + fi +} + cmd_python() { cmd_db cmd_venv cmd_migrate run_pytest + run_browser_pytest log "CLI smoke (python -m src --help)" "$VENV/bin/python" -m src --help >/dev/null ok "Python checks passed" } +cmd_browser() { + cmd_venv + run_browser_pytest + ok "Browser pytest finished" +} + cmd_web() { cmd_web_deps log "Web typecheck" @@ -162,7 +180,8 @@ Local test runner — mirrors CI (python + web jobs) ./local-test Same as: all ./local-test all Postgres + migrations + pytest + CLI + web checks - ./local-test python DB + pytest + python -m src --help + ./local-test python DB + pytest + browser pytest + python -m src --help + ./local-test browser Browser integration pytest only (skips if no Chromium) ./local-test web typecheck, lint, vitest (no Docker) ./local-test quick pytest + web without starting Docker (DB must be ready) @@ -191,6 +210,7 @@ main() { case "$raw_cmd" in all|"") cmd_all ;; python) cmd_python ;; + browser) cmd_browser ;; web) cmd_web ;; quick) PYTEST_NO_COV=1 diff --git a/src/website_profiling/analysis/local.py b/src/website_profiling/analysis/local.py index 47a82d2d..c55adf13 100644 --- a/src/website_profiling/analysis/local.py +++ b/src/website_profiling/analysis/local.py @@ -130,8 +130,6 @@ def compute_duplicate_groups( parent: dict[str, str] = {} def find(x: str) -> str: - if x not in parent: - parent[x] = x if parent[x] != x: parent[x] = find(parent[x]) return parent[x] diff --git a/src/website_profiling/cli.py b/src/website_profiling/cli.py index 1438756c..f492f16d 100644 --- a/src/website_profiling/cli.py +++ b/src/website_profiling/cli.py @@ -7,6 +7,7 @@ config_resolve, enrich_cmd, google_cmd, + gsc_links_cmd, keywords_cmd, lighthouse_cmd, page_coach_cmd, @@ -33,6 +34,8 @@ def main() -> None: enrich_cmd.run(cfg, args) elif args.command == "google": google_cmd.run(cfg, cwd, path, args) + elif args.command == "gsc-links-import": + gsc_links_cmd.run(cfg, args) elif args.command == "page-live": page_live_cmd.run(cfg, cwd, args) elif args.command == "page-coach": diff --git a/src/website_profiling/commands/config_resolve.py b/src/website_profiling/commands/config_resolve.py index 6064a4a5..90aab5be 100644 --- a/src/website_profiling/commands/config_resolve.py +++ b/src/website_profiling/commands/config_resolve.py @@ -181,13 +181,21 @@ def resolve_config(args: argparse.Namespace) -> tuple[dict[str, str], str]: cwd = get_data_dir() if cfg: - print("[Config] Loaded from pipeline_config table (PostgreSQL)", flush=True) + print( + "[Config] Loaded from pipeline_config table (PostgreSQL)", + file=sys.stderr, + flush=True, + ) else: shadow = shadow_config_path() if os.path.isfile(shadow): cfg = load_config(shadow) cwd = os.path.dirname(shadow) or os.getcwd() - print(f"[Config] Loaded from shadow file ({shadow})", flush=True) + print( + f"[Config] Loaded from shadow file ({shadow})", + file=sys.stderr, + flush=True, + ) else: print( "No audit settings found. Open Run audit in the web app, " @@ -221,6 +229,7 @@ def build_parser() -> argparse.ArgumentParser: "warnings", "enrich", "google", + "gsc-links-import", "page-live", "page-coach", ], @@ -266,6 +275,29 @@ def build_parser() -> argparse.ArgumentParser: dest="property_id", help="WebsiteProfiling property id for per-site Google credentials.", ) + parser.add_argument( + "--csv-stdin", + action="store_true", + dest="csv_stdin", + help="For gsc-links-import: read CSV from stdin.", + ) + parser.add_argument( + "--csv-file", + default=None, + dest="csv_file", + help="For gsc-links-import: path to CSV file.", + ) + parser.add_argument( + "--file-name", + default=None, + dest="file_name", + help="For gsc-links-import: original upload file name.", + ) + parser.add_argument( + "--status", + action="store_true", + help="For gsc-links-import: print import status JSON and exit.", + ) parser.add_argument( "--enrich-google", action="store_true", diff --git a/src/website_profiling/commands/gsc_links_cmd.py b/src/website_profiling/commands/gsc_links_cmd.py new file mode 100644 index 00000000..ce83ba24 --- /dev/null +++ b/src/website_profiling/commands/gsc_links_cmd.py @@ -0,0 +1,66 @@ +"""CLI: gsc-links-import command.""" +from __future__ import annotations + +import argparse +import json +import sys + + +def run(cfg: dict, args: argparse.Namespace) -> None: + from ..db import db_session, get_latest_crawl_run_id, read_crawl + from ..integrations.google.gsc_links_store import import_gsc_links_csv, read_gsc_links_status + from .config_resolve import resolve_property_id_from_cfg + + property_id = getattr(args, "property_id", None) + if not property_id: + property_id = resolve_property_id_from_cfg(cfg) + if not property_id: + print("Error: --property-id is required.", file=sys.stderr) + sys.exit(1) + + if getattr(args, "status", False): + with db_session() as conn: + status = read_gsc_links_status(conn, int(property_id)) + print(json.dumps(status), flush=True) + sys.exit(0) + + csv_text = "" + if getattr(args, "csv_stdin", False): + csv_text = sys.stdin.read() + elif getattr(args, "csv_file", None): + with open(args.csv_file, encoding="utf-8-sig") as f: + csv_text = f.read() + else: + print("Error: provide --csv-stdin or --csv-file.", file=sys.stderr) + sys.exit(1) + + file_name = getattr(args, "file_name", None) or "" + + crawl_urls: list[str] = [] + try: + with db_session() as conn: + run_id = get_latest_crawl_run_id(conn) + if run_id is not None: + df = read_crawl(conn, run_id) + if "url" in df.columns: + crawl_urls = df["url"].dropna().astype(str).str.strip().tolist() + except Exception: + pass + + try: + with db_session() as conn: + result = import_gsc_links_csv( + conn, + int(property_id), + csv_text, + crawl_urls=crawl_urls, + file_name=file_name, + ) + print(json.dumps(result), flush=True) + sys.exit(0) + except ValueError as e: + print(json.dumps({"ok": False, "error": str(e)}), flush=True) + sys.exit(1) + except Exception as e: + print(json.dumps({"ok": False, "error": str(e)}), flush=True) + sys.exit(1) diff --git a/src/website_profiling/commands/pipeline_cmd.py b/src/website_profiling/commands/pipeline_cmd.py index 8e7592e2..9906015d 100644 --- a/src/website_profiling/commands/pipeline_cmd.py +++ b/src/website_profiling/commands/pipeline_cmd.py @@ -17,6 +17,19 @@ should_enrich_keywords_after_report, ) +_ALLOWED_RENDER_MODES = frozenset({"static", "javascript", "auto"}) + + +def _normalize_render_mode(cfg: dict) -> str: + mode = (cfg.get("crawl_render_mode") or "static").strip().lower() + if mode not in _ALLOWED_RENDER_MODES: + print( + f"Warning: invalid crawl_render_mode {mode!r}; using static.", + file=sys.stderr, + ) + return "static" + return mode + def select_lighthouse_urls_from_crawl(df: pd.DataFrame, max_pages: int) -> list[str]: if df.empty or "url" not in df.columns: @@ -98,6 +111,18 @@ def _run_crawl(cfg: dict, use_database: bool) -> None: content_excerpt_max_chars = get_int(cfg, "content_excerpt_max_chars", 4096) or 4096 crawl_stream_to_db = get_bool(cfg, "crawl_stream_to_db", False) property_id = active_property_id_from_cfg(cfg) + render_mode = _normalize_render_mode(cfg) + js_concurrency = get_int(cfg, "crawl_js_concurrency", 3) or 3 + js_timeout = get_int(cfg, "crawl_js_timeout", 30) or 30 + js_wait_until = (cfg.get("crawl_js_wait_until") or "domcontentloaded").strip() + js_extra_wait_ms = get_int(cfg, "crawl_js_extra_wait_ms", 1500) + if js_extra_wait_ms is None: + js_extra_wait_ms = 1500 + js_block_resources = get_bool(cfg, "crawl_js_block_resources", True) + capture_console = get_bool(cfg, "crawl_js_capture_console", True) + js_console_levels = (cfg.get("crawl_js_console_levels") or "error,warning").strip() + capture_failed_requests = get_bool(cfg, "crawl_js_capture_failed_requests", False) + console_max_per_page = get_int(cfg, "crawl_js_console_max_per_page", 20) or 20 print("Crawling...") run_crawler( start_url=start_url, @@ -118,6 +143,16 @@ def _run_crawl(cfg: dict, use_database: bool) -> None: content_excerpt_max_chars=content_excerpt_max_chars, crawl_stream_to_db=crawl_stream_to_db, property_id=property_id, + render_mode=render_mode, + js_concurrency=js_concurrency, + js_timeout=js_timeout, + js_wait_until=js_wait_until, + js_extra_wait_ms=js_extra_wait_ms, + js_block_resources=js_block_resources, + capture_console=capture_console, + js_console_levels=js_console_levels, + capture_failed_requests=capture_failed_requests, + console_max_per_page=console_max_per_page, ) print("[Crawl] Done.", flush=True) print("Crawl results: PostgreSQL") @@ -233,6 +268,20 @@ def _run_plot(cfg: dict, use_database: bool) -> None: from ..tools.plot import run_plot as do_plot print("[Plot] Starting...", flush=True) + render_mode = (cfg.get("crawl_render_mode") or "").strip().lower() or None + if render_mode is not None and render_mode not in _ALLOWED_RENDER_MODES: + print( + f"Warning: invalid crawl_render_mode {render_mode!r}; using crawl run default.", + file=sys.stderr, + ) + render_mode = None + js_concurrency = get_int(cfg, "crawl_js_concurrency", 3) or 3 + js_timeout = get_int(cfg, "crawl_js_timeout", 30) or 30 + js_wait_until = (cfg.get("crawl_js_wait_until") or "domcontentloaded").strip() + js_extra_wait_ms = get_int(cfg, "crawl_js_extra_wait_ms", 1500) + if js_extra_wait_ms is None: + js_extra_wait_ms = 1500 + js_block_resources = get_bool(cfg, "crawl_js_block_resources", True) e = do_plot( same_domain_only=get_bool(cfg, "same_domain_only", True), max_fetch_for_edges=get_int(cfg, "max_fetch_for_edges", 500), @@ -240,6 +289,12 @@ def _run_plot(cfg: dict, use_database: bool) -> None: timeout=10, polite_delay=0.15, use_database=use_database, + render_mode=render_mode, + js_timeout=js_timeout, + js_concurrency=js_concurrency, + js_wait_until=js_wait_until, + js_extra_wait_ms=js_extra_wait_ms, + js_block_resources=js_block_resources, ) print("[Plot] Done.", flush=True) print(f"Plot data: {e}") diff --git a/src/website_profiling/crawl/crawler.py b/src/website_profiling/crawl/crawler.py index 83ed3e17..2adc2f7d 100644 --- a/src/website_profiling/crawl/crawler.py +++ b/src/website_profiling/crawl/crawler.py @@ -40,20 +40,15 @@ def _url_matches_exclude(url: str, exclude_urls: list[str]) -> bool: parse_tech_stack, ) from ..analysis.page import analyze_html +from .fetchers import build_fetcher +from .fetchers.base import FetchResult +from .fetchers.browser_diagnostics import merge_browser_into_page_analysis +from .fetchers.hybrid import HybridFetcher +from .fetchers.spa_heuristics import needs_js_render_after_parse +from .sitemap import discover_sitemap_urls DEFAULT_USER_AGENT = "WebsiteProfilingCrawler/1.0" -# Headers we store for caching and security -HEADER_KEYS = ( - "Cache-Control", - "ETag", - "X-Robots-Tag", - "Strict-Transport-Security", - "X-Content-Type-Options", - "X-Frame-Options", - "Content-Security-Policy", -) - class Crawler: def __init__( @@ -72,13 +67,30 @@ def __init__( use_wappalyzer: bool = True, store_content_excerpt: bool = False, content_excerpt_max_chars: int = 4096, + render_mode: str = "static", + js_concurrency: int = 3, + js_timeout: int = 30, + js_wait_until: str = "domcontentloaded", + js_extra_wait_ms: int = 1500, + js_block_resources: bool = True, + capture_console: bool = True, + js_console_levels: str = "error,warning", + capture_failed_requests: bool = False, + console_max_per_page: int = 20, ): self.start_url = start_url.rstrip("/") self.start_netloc = urlparse(self.start_url).netloc + self.render_mode = (render_mode or "static").strip().lower() + self.js_concurrency = max(1, int(js_concurrency)) + effective_concurrency = ( + self.js_concurrency + if self.render_mode == "javascript" + else max(1, int(concurrency)) + ) self.max_pages = ( max_pages if (max_pages is not None and max_pages > 0) else float("inf") ) - self.concurrency = max(1, int(concurrency)) + self.concurrency = effective_concurrency self.timeout = timeout self.ignore_robots = ignore_robots self.allow_external = allow_external @@ -102,6 +114,25 @@ def __init__( self.session = requests.Session() self.session.headers.update({"User-Agent": self.user_agent}) self.rp = None if self.ignore_robots else load_robots(self.start_url) + self.fetcher = build_fetcher( + render_mode="javascript" if self.render_mode == "javascript" else ("auto" if self.render_mode == "auto" else "static"), + timeout=timeout, + user_agent=self.user_agent, + session=self.session, + js_concurrency=self.js_concurrency, + js_timeout=js_timeout, + js_wait_until=js_wait_until, + js_extra_wait_ms=js_extra_wait_ms, + js_block_resources=js_block_resources, + capture_console=capture_console, + js_console_levels=js_console_levels, + capture_failed_requests=capture_failed_requests, + console_max_per_page=console_max_per_page, + ) + self._hybrid_fetcher = ( + self.fetcher if isinstance(self.fetcher, HybridFetcher) else None + ) + self._seed_sitemap_urls(timeout) def same_domain(self, url): return urlparse(url).netloc == self.start_netloc @@ -114,36 +145,27 @@ def allowed_by_robots(self, url): except Exception: return True - def fetch(self, url): + def _seed_sitemap_urls(self, timeout: int) -> None: try: - t0 = time.perf_counter() - resp = self.session.get( - url, timeout=self.timeout, allow_redirects=True - ) - response_time_ms = int((time.perf_counter() - t0) * 1000) - ct = resp.headers.get("Content-Type", "") - is_html = resp.status_code == 200 and ( - "text/html" in ct or "application/xhtml+xml" in ct - ) - text = resp.text if is_html else None - content_length = len(resp.content) if resp.content is not None else 0 - final_url = resp.url or url - redirect_chain_length = len(resp.history) - headers_dict = { - k: (resp.headers.get(k) or "") for k in HEADER_KEYS - } - return ( - resp.status_code, - ct, - text, - response_time_ms, - content_length, - final_url, - headers_dict, - redirect_chain_length, + seeds = discover_sitemap_urls( + self.start_url, + timeout=timeout, + session=self.session, ) except Exception: - return None, None, None, None, None, None, {}, 0 + return + for url in seeds: + if _url_matches_exclude(url, self.exclude_urls): + continue + if not self.allow_external and not self.same_domain(url): + continue + if url == self.start_url or url in self.depths: + continue + self.queue.put(url) + self.depths[url] = 0 + + def fetch(self, url) -> FetchResult: + return self.fetcher.fetch(url) def _empty_seo(self, url: str, headers_dict: Optional[dict] = None, redirect_chain_length: int = 0) -> dict: """Default SEO/performance fields when no HTML or error.""" @@ -197,6 +219,127 @@ def _empty_seo(self, url: str, headers_dict: Optional[dict] = None, redirect_cha "page_analysis": "{}", } + def _parse_page_content( + self, + url: str, + text: str, + final_url: str, + headers_dict: dict, + redirect_chain_length: int, + ) -> dict: + """Extract title, links, and SEO/content fields from HTML.""" + ext = self._empty_seo(url, headers_dict, redirect_chain_length) + title, links = parse_links(url, text) + meta_description, meta_description_len, h1_text, h1_count, canonical_url = ( + parse_seo(url, text) + ) + seo_ext = parse_seo_extended(text, final_url or url) + ext["viewport_present"] = seo_ext.get("viewport_present", False) + ext["viewport_content"] = seo_ext.get("viewport_content", "") + ext["noindex"] = seo_ext.get("noindex", False) + if (headers_dict.get("X-Robots-Tag") or "").lower().find("noindex") >= 0: + ext["noindex"] = True + ext["has_schema"] = seo_ext.get("has_schema", False) + ext["heading_sequence"] = ",".join(seo_ext.get("heading_sequence") or []) + ext["images_without_alt"] = seo_ext.get("images_without_alt", 0) + ext["images_total"] = seo_ext.get("images_total", 0) + ext["img_without_lazy"] = seo_ext.get("img_without_lazy", 0) + ext["img_without_dimensions"] = seo_ext.get("img_without_dimensions", 0) + ext["aria_count"] = seo_ext.get("aria_count", 0) + ext["mixed_content_count"] = seo_ext.get("mixed_content_count", 0) + res_res = parse_resources(text, final_url or url) + ext["script_count"] = res_res.get("script_count", 0) + ext["link_stylesheet_count"] = res_res.get("link_stylesheet_count", 0) + from bs4 import BeautifulSoup as _BS + + _soup = _BS(text, "lxml") + excerpt_max = self.content_excerpt_max_chars if self.store_content_excerpt else 0 + ct_data = parse_content_text(_soup, text, excerpt_max_chars=excerpt_max) + ext["word_count"] = ct_data.get("word_count", 0) + ext["reading_level"] = ct_data.get("reading_level", 0.0) + ext["content_html_ratio"] = ct_data.get("content_html_ratio", 0.0) + ext["top_keywords"] = ct_data.get("top_keywords", "[]") + ext["content_excerpt"] = ct_data.get("content_excerpt") or "" + social = parse_social_meta(_soup) + ext["og_title"] = social.get("og_title", "") + ext["og_description"] = social.get("og_description", "") + ext["og_image"] = social.get("og_image", "") + ext["og_type"] = social.get("og_type", "") + ext["twitter_card"] = social.get("twitter_card", "") + ext["twitter_title"] = social.get("twitter_title", "") + ext["twitter_image"] = social.get("twitter_image", "") + if self.use_wappalyzer: + ext["tech_stack"] = detect_tech_wappalyzer( + final_url or url, text, headers_dict, _soup, self._wappalyzer_instance + ) + else: + ext["tech_stack"] = parse_tech_stack(_soup, headers_dict, final_url or url) + ext["page_analysis"] = json.dumps( + analyze_html(text, final_url or url, final_url or url, canonical_url) + ) + return { + "title": title, + "links": links, + "meta_description": meta_description, + "meta_description_len": meta_description_len, + "h1_text": h1_text, + "h1_count": h1_count, + "canonical_url": canonical_url, + "ext": ext, + } + + def _maybe_refetch_after_parse( + self, + url: str, + result: FetchResult, + *, + link_count: int, + same_domain_link_count: int, + ) -> FetchResult: + """Post-parse auto-mode fallback when static HTML has too few links.""" + if self.render_mode != "auto" or self._hybrid_fetcher is None: + return result + if result.fetch_method != "static": + return result + if not needs_js_render_after_parse( + result, + link_count=link_count, + same_domain_link_count=same_domain_link_count, + ): + return result + rendered = self._hybrid_fetcher.refetch_rendered(url) + if rendered.status == 200 and rendered.text: + return rendered + return result + + @staticmethod + def _sync_from_fetch_result( + result: FetchResult, + url: str, + *, + text: Optional[str], + fetch_method: str, + final_url: str, + content_length: int, + response_time_ms: Optional[int], + headers_dict: dict, + redirect_chain_length: int, + status: Optional[int], + ct: Optional[str], + ) -> dict: + """Copy all FetchResult fields after a post-parse browser refetch.""" + return { + "text": result.text, + "fetch_method": result.fetch_method, + "final_url": result.final_url or url, + "content_length": result.content_length or content_length, + "response_time_ms": result.response_time_ms, + "headers_dict": result.headers_dict or headers_dict, + "redirect_chain_length": result.redirect_chain_length, + "status": result.status, + "ct": result.content_type, + } + def worker(self, url): if not self.allowed_by_robots(url): out = { @@ -205,6 +348,7 @@ def worker(self, url): "content_type": "", "title": "", "outlinks": 0, + "fetch_method": "static", **self._empty_seo(url), } if self.store_outlinks: @@ -212,14 +356,15 @@ def worker(self, url): return out result = self.fetch(url) - status = result[0] - ct = result[1] - text = result[2] - response_time_ms = result[3] if len(result) > 3 else None - content_length = result[4] if len(result) > 4 else 0 - final_url = result[5] if len(result) > 5 else url - headers_dict = result[6] if len(result) > 6 else {} - redirect_chain_length = result[7] if len(result) > 7 else 0 + status = result.status + ct = result.content_type + text = result.text + response_time_ms = result.response_time_ms + content_length = result.content_length or 0 + final_url = result.final_url or url + headers_dict = result.headers_dict or {} + redirect_chain_length = result.redirect_chain_length + fetch_method = result.fetch_method if status is None: out = { @@ -228,10 +373,15 @@ def worker(self, url): "content_type": "", "title": "", "outlinks": 0, + "fetch_method": fetch_method, **self._empty_seo(url, headers_dict, redirect_chain_length), } if self.store_outlinks: out["outlink_targets"] = "[]" + if result.browser_diagnostics: + out["page_analysis"] = merge_browser_into_page_analysis( + None, result.browser_diagnostics + ) return out title = "" @@ -245,54 +395,54 @@ def worker(self, url): ext = self._empty_seo(url, headers_dict, redirect_chain_length) if text: - title, links = parse_links(url, text) - outlinks_count = len(links) - meta_description, meta_description_len, h1_text, h1_count, canonical_url = ( - parse_seo(url, text) + parsed = self._parse_page_content( + url, text, final_url or url, headers_dict, redirect_chain_length ) - seo_ext = parse_seo_extended(text, final_url or url) - ext["viewport_present"] = seo_ext.get("viewport_present", False) - ext["viewport_content"] = seo_ext.get("viewport_content", "") - ext["noindex"] = seo_ext.get("noindex", False) - if (headers_dict.get("X-Robots-Tag") or "").lower().find("noindex") >= 0: - ext["noindex"] = True - ext["has_schema"] = seo_ext.get("has_schema", False) - ext["heading_sequence"] = ",".join(seo_ext.get("heading_sequence") or []) - ext["images_without_alt"] = seo_ext.get("images_without_alt", 0) - ext["images_total"] = seo_ext.get("images_total", 0) - ext["img_without_lazy"] = seo_ext.get("img_without_lazy", 0) - ext["img_without_dimensions"] = seo_ext.get("img_without_dimensions", 0) - ext["aria_count"] = seo_ext.get("aria_count", 0) - ext["mixed_content_count"] = seo_ext.get("mixed_content_count", 0) - res_res = parse_resources(text, final_url or url) - ext["script_count"] = res_res.get("script_count", 0) - ext["link_stylesheet_count"] = res_res.get("link_stylesheet_count", 0) - from bs4 import BeautifulSoup as _BS - _soup = _BS(text, "lxml") - excerpt_max = self.content_excerpt_max_chars if self.store_content_excerpt else 0 - ct_data = parse_content_text(_soup, text, excerpt_max_chars=excerpt_max) - ext["word_count"] = ct_data.get("word_count", 0) - ext["reading_level"] = ct_data.get("reading_level", 0.0) - ext["content_html_ratio"] = ct_data.get("content_html_ratio", 0.0) - ext["top_keywords"] = ct_data.get("top_keywords", "[]") - ext["content_excerpt"] = ct_data.get("content_excerpt") or "" - social = parse_social_meta(_soup) - ext["og_title"] = social.get("og_title", "") - ext["og_description"] = social.get("og_description", "") - ext["og_image"] = social.get("og_image", "") - ext["og_type"] = social.get("og_type", "") - ext["twitter_card"] = social.get("twitter_card", "") - ext["twitter_title"] = social.get("twitter_title", "") - ext["twitter_image"] = social.get("twitter_image", "") - if self.use_wappalyzer: - ext["tech_stack"] = detect_tech_wappalyzer( - final_url or url, text, headers_dict, _soup, self._wappalyzer_instance - ) - else: - ext["tech_stack"] = parse_tech_stack(_soup, headers_dict, final_url or url) - ext["page_analysis"] = json.dumps( - analyze_html(text, final_url or url, final_url or url, canonical_url) + links = parsed["links"] + same_domain_link_count = sum(1 for link in links if self.same_domain(link)) + result = self._maybe_refetch_after_parse( + url, + result, + link_count=len(links), + same_domain_link_count=same_domain_link_count, ) + if result.text and result.text != text: + synced = self._sync_from_fetch_result( + result, + url, + text=text, + fetch_method=fetch_method, + final_url=final_url, + content_length=content_length, + response_time_ms=response_time_ms, + headers_dict=headers_dict, + redirect_chain_length=redirect_chain_length, + status=status, + ct=ct, + ) + text = synced["text"] + fetch_method = synced["fetch_method"] + final_url = synced["final_url"] + content_length = synced["content_length"] + response_time_ms = synced["response_time_ms"] + headers_dict = synced["headers_dict"] + redirect_chain_length = synced["redirect_chain_length"] + status = synced["status"] + ct = synced["ct"] + parsed = self._parse_page_content( + url, text, final_url, headers_dict, redirect_chain_length + ) + links = parsed["links"] + + title = parsed["title"] + outlinks_count = len(links) + meta_description = parsed["meta_description"] + meta_description_len = parsed["meta_description_len"] + h1_text = parsed["h1_text"] + h1_count = parsed["h1_count"] + canonical_url = parsed["canonical_url"] + ext = parsed["ext"] + for link in links: if _url_matches_exclude(link, self.exclude_urls): continue @@ -333,12 +483,18 @@ def worker(self, url): if self.polite_delay: time.sleep(self.polite_delay) + if result.browser_diagnostics: + ext["page_analysis"] = merge_browser_into_page_analysis( + ext.get("page_analysis"), result.browser_diagnostics + ) + res = { "url": url, "status": status, "content_type": ct or "", "title": title, "outlinks": outlinks_count, + "fetch_method": fetch_method, **ext, } if self.store_outlinks: @@ -368,98 +524,100 @@ def crawl( desc="Pages", disable=not show_progress, ) - with ThreadPoolExecutor(max_workers=self.concurrency) as ex: - while (len(self.results) < self.max_pages) and ( - not self.queue.empty() or futures - ): - while ( - not self.queue.empty() - and len(futures) < self.concurrency - and len(self.results) + len(futures) < self.max_pages + try: + with ThreadPoolExecutor(max_workers=self.concurrency) as ex: + while (len(self.results) < self.max_pages) and ( + not self.queue.empty() or futures ): - url = self.queue.get() - if _url_matches_exclude(url, self.exclude_urls): - continue - with self.lock: - if url in self.visited: + while ( + not self.queue.empty() + and len(futures) < self.concurrency + and len(self.results) + len(futures) < self.max_pages + ): + url = self.queue.get() + if _url_matches_exclude(url, self.exclude_urls): continue - self.visited.add(url) - futures.append(ex.submit(self.worker, url)) - - remaining = [] - for f in futures: - if f.done(): - try: - res = f.result() - except Exception: - res = { - "url": None, - "status": "error", - "content_type": "", - "title": "", - "outlinks": 0, - "response_time_ms": "", - "content_length": 0, - "final_url": "", - "meta_description": "", - "meta_description_len": 0, - "h1": "", - "h1_count": 0, - "canonical_url": "", - "viewport_present": False, - "viewport_content": "", - "noindex": False, - "has_schema": False, - "heading_sequence": "", - "images_without_alt": 0, - "images_total": 0, - "img_without_lazy": 0, - "img_without_dimensions": 0, - "aria_count": 0, - "mixed_content_count": 0, - "redirect_chain_length": 0, - "cache_control": "", - "etag": "", - "x_robots_tag": "", - "strict_transport_security": "", - "x_content_type_options": "", - "x_frame_options": "", - "content_security_policy": "", - "script_count": 0, - "link_stylesheet_count": 0, - "total_js_bytes": 0, - "total_css_bytes": 0, - "word_count": 0, - "reading_level": 0.0, - "content_html_ratio": 0.0, - "top_keywords": "[]", - "content_excerpt": "", - "og_title": "", - "og_description": "", - "og_image": "", - "og_type": "", - "twitter_card": "", - "twitter_title": "", - "twitter_image": "", - "tech_stack": "[]", - "depth": None, - "page_analysis": "{}", - } - if self.store_outlinks: - res["outlink_targets"] = "[]" - self.results.append(res) - if db_writer is not None and res.get("url"): - db_writer.enqueue(res) - pbar.update(1) - else: - remaining.append(f) - futures = remaining - time.sleep(0.01) - - if self.queue.empty() and not futures: - break - - pbar.close() + with self.lock: + if url in self.visited: + continue + self.visited.add(url) + futures.append(ex.submit(self.worker, url)) + + remaining = [] + for f in futures: + if f.done(): + try: + res = f.result() + except Exception: + res = { + "url": None, + "status": "error", + "content_type": "", + "title": "", + "outlinks": 0, + "response_time_ms": "", + "content_length": 0, + "final_url": "", + "meta_description": "", + "meta_description_len": 0, + "h1": "", + "h1_count": 0, + "canonical_url": "", + "viewport_present": False, + "viewport_content": "", + "noindex": False, + "has_schema": False, + "heading_sequence": "", + "images_without_alt": 0, + "images_total": 0, + "img_without_lazy": 0, + "img_without_dimensions": 0, + "aria_count": 0, + "mixed_content_count": 0, + "redirect_chain_length": 0, + "cache_control": "", + "etag": "", + "x_robots_tag": "", + "strict_transport_security": "", + "x_content_type_options": "", + "x_frame_options": "", + "content_security_policy": "", + "script_count": 0, + "link_stylesheet_count": 0, + "total_js_bytes": 0, + "total_css_bytes": 0, + "word_count": 0, + "reading_level": 0.0, + "content_html_ratio": 0.0, + "top_keywords": "[]", + "content_excerpt": "", + "og_title": "", + "og_description": "", + "og_image": "", + "og_type": "", + "twitter_card": "", + "twitter_title": "", + "twitter_image": "", + "tech_stack": "[]", + "depth": None, + "page_analysis": "{}", + } + if self.store_outlinks: + res["outlink_targets"] = "[]" + self.results.append(res) + if db_writer is not None and res.get("url"): + db_writer.enqueue(res) + pbar.update(1) + else: + remaining.append(f) + futures = remaining + time.sleep(0.01) + + if self.queue.empty() and not futures: + break + finally: + self.fetcher.close() + pbar.close() if db_writer is not None: db_writer.finish() db_writer.join() @@ -519,6 +677,7 @@ def crawl( "tech_stack", "depth", "page_analysis", + "fetch_method", ] if self.store_outlinks: cols.append("outlink_targets") @@ -592,11 +751,27 @@ def run_crawler( content_excerpt_max_chars: int = 4096, crawl_stream_to_db: bool = False, property_id: Optional[int] = None, + render_mode: str = "static", + js_concurrency: int = 3, + js_timeout: int = 30, + js_wait_until: str = "domcontentloaded", + js_extra_wait_ms: int = 1500, + js_block_resources: bool = True, + capture_console: bool = True, + js_console_levels: str = "error,warning", + capture_failed_requests: bool = False, + console_max_per_page: int = 20, ) -> pd.DataFrame: """Run crawler and optionally save to CSV/JSON or PostgreSQL. Returns DataFrame.""" import sys max_p = max_pages if max_pages is not None else 0 - print(f" Crawling {start_url} (max_pages={max_p or 'unlimited'}, concurrency={concurrency})...", flush=True) + mode_label = (render_mode or "static").strip().lower() + conc_label = js_concurrency if mode_label == "javascript" else concurrency + print( + f" Crawling {start_url} (max_pages={max_p or 'unlimited'}, " + f"render_mode={mode_label}, concurrency={conc_label})...", + flush=True, + ) crawler = Crawler( start_url=start_url, max_pages=max_pages, @@ -610,6 +785,16 @@ def run_crawler( exclude_urls=exclude_urls, store_content_excerpt=store_content_excerpt, content_excerpt_max_chars=content_excerpt_max_chars, + render_mode=render_mode, + js_concurrency=js_concurrency, + js_timeout=js_timeout, + js_wait_until=js_wait_until, + js_extra_wait_ms=js_extra_wait_ms, + js_block_resources=js_block_resources, + capture_console=capture_console, + js_console_levels=js_console_levels, + capture_failed_requests=capture_failed_requests, + console_max_per_page=console_max_per_page, ) stream_run_id: Optional[int] = None if output_db: @@ -629,7 +814,9 @@ def run_crawler( ensure_crawl_tables_cleared(conn) if historical: restore_historical_data(conn, historical) - stream_run_id = create_crawl_run(conn, start_url, property_id=property_id) + stream_run_id = create_crawl_run( + conn, start_url, property_id=property_id, render_mode=render_mode + ) print(f" Streaming crawl results to DB (run_id={stream_run_id})...", flush=True) df = crawler.crawl( @@ -656,7 +843,9 @@ def run_crawler( ensure_crawl_tables_cleared(conn) if historical: restore_historical_data(conn, historical) - run_id = create_crawl_run(conn, start_url, property_id=property_id) + run_id = create_crawl_run( + conn, start_url, property_id=property_id, render_mode=render_mode + ) write_crawl(conn, df, crawl_run_id=run_id) print(" Crawl DB write complete.", flush=True) elif output_db and stream_run_id is not None: diff --git a/src/website_profiling/crawl/fetchers/__init__.py b/src/website_profiling/crawl/fetchers/__init__.py new file mode 100644 index 00000000..f66f493b --- /dev/null +++ b/src/website_profiling/crawl/fetchers/__init__.py @@ -0,0 +1,14 @@ +"""HTTP and browser fetchers for the website crawler.""" + +from .base import FetchResult, HEADER_KEYS +from .browser_deps import browser_status, ensure_browser_deps +from .factory import build_fetcher, validate_browser_available + +__all__ = [ + "FetchResult", + "HEADER_KEYS", + "browser_status", + "build_fetcher", + "ensure_browser_deps", + "validate_browser_available", +] diff --git a/src/website_profiling/crawl/fetchers/base.py b/src/website_profiling/crawl/fetchers/base.py new file mode 100644 index 00000000..6681de32 --- /dev/null +++ b/src/website_profiling/crawl/fetchers/base.py @@ -0,0 +1,49 @@ +"""Shared fetch result types for static and browser crawlers.""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any, Literal, Optional, Protocol + +HEADER_KEYS = ( + "Cache-Control", + "ETag", + "X-Robots-Tag", + "Strict-Transport-Security", + "X-Content-Type-Options", + "X-Frame-Options", + "Content-Security-Policy", +) + + +@dataclass +class FetchResult: + status: Optional[int] + content_type: Optional[str] + text: Optional[str] + response_time_ms: Optional[int] + content_length: Optional[int] + final_url: Optional[str] + headers_dict: dict[str, str] + redirect_chain_length: int + fetch_method: Literal["static", "rendered"] = "static" + browser_diagnostics: Optional[dict[str, Any]] = None + + def as_tuple(self) -> tuple: + """Legacy tuple shape used by Crawler.worker.""" + return ( + self.status, + self.content_type, + self.text, + self.response_time_ms, + self.content_length, + self.final_url, + self.headers_dict, + self.redirect_chain_length, + ) + + +class PageFetcher(Protocol): + def fetch(self, url: str) -> FetchResult: ... + + def close(self) -> None: ... diff --git a/src/website_profiling/crawl/fetchers/browser.py b/src/website_profiling/crawl/fetchers/browser.py new file mode 100644 index 00000000..c91a764f --- /dev/null +++ b/src/website_profiling/crawl/fetchers/browser.py @@ -0,0 +1,403 @@ +"""Headless browser fetcher: async Playwright on a dedicated event-loop thread.""" + +from __future__ import annotations + +import asyncio +import os +import threading +import time +from concurrent.futures import Future +from dataclasses import dataclass +from typing import Any, Callable, Optional + +from .base import HEADER_KEYS, FetchResult +from .browser_diagnostics import finalize_browser_diagnostics, truncate_diag_text + +_BROWSER_INSTALL_MSG = ( + "JavaScript crawl requires Playwright and Chromium. Install: " + "pip install -r requirements-browser.txt. " + "Chrome or Chromium must be available (set CHROME_PATH if needed)." +) + +_DEFAULT_CHROME_ARGS = [ + "--no-sandbox", + "--disable-dev-shm-usage", + "--disable-gpu", + "--headless=new", +] + +_BLOCKED_RESOURCE_TYPES = frozenset({"image", "media", "font"}) + + +@dataclass +class _FetchJob: + url: str + future: Future[FetchResult] + + +class _PageDiagnosticsCollector: + def __init__( + self, + *, + capture_console: bool, + console_levels: frozenset[str], + capture_failed_requests: bool, + max_per_page: int, + ) -> None: + self.capture_console = capture_console + self.console_levels = console_levels + self.capture_failed_requests = capture_failed_requests + self.max_per_page = max(1, int(max_per_page)) + self.console: list[dict[str, Any]] = [] + self.page_errors: list[dict[str, Any]] = [] + self.failed_requests: list[dict[str, Any]] = [] + self._handlers: list[tuple[str, Callable[..., Any]]] = [] + + def attach(self, page: Any) -> None: + if self.capture_console: + + def on_console(msg: Any) -> None: + level = str(getattr(msg, "type", "") or "").lower() + if level not in self.console_levels or len(self.console) >= self.max_per_page: + return + entry: dict[str, Any] = { + "level": level, + "text": truncate_diag_text(getattr(msg, "text", "")), + } + loc = getattr(msg, "location", None) + if isinstance(loc, dict): + if loc.get("url"): + entry["source_url"] = str(loc["url"]) + if loc.get("lineNumber") is not None: + entry["line"] = int(loc["lineNumber"]) + self.console.append(entry) + + page.on("console", on_console) + self._handlers.append(("console", on_console)) + + def on_page_error(err: Any) -> None: + if len(self.page_errors) >= self.max_per_page: + return + self.page_errors.append( + { + "message": truncate_diag_text(str(err)), + "stack": truncate_diag_text(getattr(err, "stack", "") or ""), + } + ) + + page.on("pageerror", on_page_error) + self._handlers.append(("pageerror", on_page_error)) + + if self.capture_failed_requests: + + def on_request_failed(request: Any) -> None: + if len(self.failed_requests) >= self.max_per_page: + return + failure = getattr(request, "failure", None) + if isinstance(failure, str): + fail_text = failure + elif failure is not None: + fail_text = getattr(failure, "error_text", None) or str(failure) + else: + fail_text = "" + self.failed_requests.append( + { + "url": str(getattr(request, "url", "") or ""), + "method": str(getattr(request, "method", "") or ""), + "failure": truncate_diag_text(fail_text), + } + ) + + page.on("requestfailed", on_request_failed) + self._handlers.append(("requestfailed", on_request_failed)) + + def detach(self, page: Any) -> None: + for event, handler in self._handlers: + try: + page.remove_listener(event, handler) + except Exception: + pass + self._handlers.clear() + + def build(self) -> dict[str, Any]: + return finalize_browser_diagnostics(self.console, self.page_errors, self.failed_requests) + + +class BrowserFetcher: + """Sync API bridging crawler threads to async Playwright page pool.""" + + def __init__( + self, + *, + timeout: int = 30, + user_agent: str = "WebsiteProfilingCrawler/1.0", + js_concurrency: int = 3, + wait_until: str = "domcontentloaded", + extra_wait_ms: int = 1500, + block_resources: bool = True, + capture_console: bool = True, + console_levels: frozenset[str] | None = None, + capture_failed_requests: bool = False, + console_max_per_page: int = 20, + ) -> None: + self.timeout = max(1, int(timeout)) + self.user_agent = user_agent + self.js_concurrency = max(1, int(js_concurrency)) + self.wait_until = wait_until if wait_until in ("domcontentloaded", "load", "commit") else "domcontentloaded" + self.extra_wait_ms = max(0, int(extra_wait_ms)) + self.block_resources = bool(block_resources) + self.capture_console = bool(capture_console) + self.console_levels = console_levels or frozenset({"error", "warning"}) + self.capture_failed_requests = bool(capture_failed_requests) + self.console_max_per_page = max(1, int(console_max_per_page)) + + self._loop: Optional[asyncio.AbstractEventLoop] = None + self._thread: Optional[threading.Thread] = None + self._ready = threading.Event() + self._startup_error: Optional[BaseException] = None + self._closed = False + self._jobs: asyncio.Queue[_FetchJob | None] | None = None + + self._thread = threading.Thread(target=self._run_loop_thread, name="browser-fetcher", daemon=True) + self._thread.start() + if not self._ready.wait(timeout=60): + raise RuntimeError("Browser fetcher failed to start within 60 seconds") + if self._startup_error is not None: + raise RuntimeError(_BROWSER_INSTALL_MSG) from self._startup_error + + def _run_loop_thread(self) -> None: + loop = asyncio.new_event_loop() + asyncio.set_event_loop(loop) + self._loop = loop + try: + loop.run_until_complete(self._async_main()) + except BaseException as e: + self._startup_error = e + self._ready.set() + finally: + try: + pending = asyncio.all_tasks(loop) + for task in pending: + task.cancel() + if pending: + loop.run_until_complete(asyncio.gather(*pending, return_exceptions=True)) + except Exception: + pass + loop.close() + + async def _async_main(self) -> None: + from playwright.async_api import async_playwright + + self._jobs = asyncio.Queue() + playwright = await async_playwright().start() + chrome_path = (os.environ.get("CHROME_PATH") or "").strip() or None + launch_kwargs: dict[str, Any] = { + "headless": True, + "args": list(_DEFAULT_CHROME_ARGS), + } + if chrome_path: + launch_kwargs["executable_path"] = chrome_path + + browser = await playwright.chromium.launch(**launch_kwargs) + context = await browser.new_context(user_agent=self.user_agent) + semaphore = asyncio.Semaphore(self.js_concurrency) + pages: list[Any] = [] + for _ in range(self.js_concurrency): + page = await context.new_page() + if self.block_resources: + + async def _route_handler(route: Any, request: Any) -> None: + if request.resource_type in _BLOCKED_RESOURCE_TYPES: + await route.abort() + else: + await route.continue_() + + await page.route("**/*", _route_handler) + pages.append(page) + page_queue: asyncio.Queue[Any] = asyncio.Queue() + for page in pages: + await page_queue.put(page) + + async def worker() -> None: + assert self._jobs is not None + while True: + job = await self._jobs.get() + if job is None: + self._jobs.task_done() + break + page = await page_queue.get() + try: + async with semaphore: + result = await self._fetch_page(page, job.url) + if not job.future.done(): + job.future.set_result(result) + except Exception: + if not job.future.done(): + job.future.set_result( + FetchResult( + status=None, + content_type=None, + text=None, + response_time_ms=None, + content_length=None, + final_url=job.url, + headers_dict={}, + redirect_chain_length=0, + fetch_method="rendered", + ) + ) + finally: + await page_queue.put(page) + self._jobs.task_done() + + workers = [asyncio.create_task(worker()) for _ in range(self.js_concurrency)] + self._ready.set() + await asyncio.gather(*workers) + + for page in pages: + try: + await page.close() + except Exception: + pass + try: + await context.close() + except Exception: + pass + try: + await browser.close() + except Exception: + pass + try: + await playwright.stop() + except Exception: + pass + + def _diagnostics_enabled(self) -> bool: + return self.capture_console or self.capture_failed_requests + + async def _fetch_page(self, page: Any, url: str) -> FetchResult: + t0 = time.perf_counter() + response = None + collector: Optional[_PageDiagnosticsCollector] = None + if self._diagnostics_enabled(): + collector = _PageDiagnosticsCollector( + capture_console=self.capture_console, + console_levels=self.console_levels, + capture_failed_requests=self.capture_failed_requests, + max_per_page=self.console_max_per_page, + ) + collector.attach(page) + try: + try: + response = await page.goto( + url, + wait_until=self.wait_until, + timeout=self.timeout * 1000, + ) + except Exception: + response = None + + if self.extra_wait_ms and response is not None: + await asyncio.sleep(self.extra_wait_ms / 1000.0) + finally: + if collector is not None: + collector.detach(page) + + response_time_ms = int((time.perf_counter() - t0) * 1000) + final_url = page.url or url + redirect_chain_length = 1 if final_url.rstrip("/") != url.rstrip("/") else 0 + browser_diagnostics = collector.build() if collector is not None else None + + if response is None: + return FetchResult( + status=None, + content_type=None, + text=None, + response_time_ms=response_time_ms, + content_length=0, + final_url=final_url, + headers_dict={}, + redirect_chain_length=redirect_chain_length, + fetch_method="rendered", + browser_diagnostics=browser_diagnostics, + ) + + status = response.status + headers = response.headers or {} + lower_headers = {str(k).lower(): v for k, v in headers.items()} + ct = lower_headers.get("content-type", "") + headers_dict = { + k: (headers.get(k) or lower_headers.get(k.lower(), "")) for k in HEADER_KEYS + } + + is_html = status == 200 and ("text/html" in ct or "application/xhtml+xml" in ct) + text: Optional[str] = None + content_length = 0 + if is_html: + try: + text = await page.content() + content_length = len(text.encode("utf-8")) if text else 0 + except Exception: + text = None + + return FetchResult( + status=status, + content_type=ct, + text=text, + response_time_ms=response_time_ms, + content_length=content_length, + final_url=final_url, + headers_dict=headers_dict, + redirect_chain_length=redirect_chain_length, + fetch_method="rendered", + browser_diagnostics=browser_diagnostics, + ) + + def fetch(self, url: str) -> FetchResult: + if self._closed: + return FetchResult( + status=None, + content_type=None, + text=None, + response_time_ms=None, + content_length=None, + final_url=url, + headers_dict={}, + redirect_chain_length=0, + fetch_method="rendered", + ) + assert self._loop is not None and self._jobs is not None + fut: Future[FetchResult] = Future() + job = _FetchJob(url=url, future=fut) + + def _submit() -> None: + assert self._jobs is not None + self._jobs.put_nowait(job) + + self._loop.call_soon_threadsafe(_submit) + total_timeout = self.timeout + (self.extra_wait_ms / 1000.0) + 15 + try: + return fut.result(timeout=total_timeout) + except Exception: + return FetchResult( + status=None, + content_type=None, + text=None, + response_time_ms=None, + content_length=None, + final_url=url, + headers_dict={}, + redirect_chain_length=0, + fetch_method="rendered", + ) + + def close(self) -> None: + if self._closed: + return + self._closed = True + if self._loop is None or self._jobs is None: + return + for _ in range(self.js_concurrency): + self._loop.call_soon_threadsafe(self._jobs.put_nowait, None) + if self._thread is not None and self._thread.is_alive(): + self._thread.join(timeout=30) diff --git a/src/website_profiling/crawl/fetchers/browser_deps.py b/src/website_profiling/crawl/fetchers/browser_deps.py new file mode 100644 index 00000000..055a5ea0 --- /dev/null +++ b/src/website_profiling/crawl/fetchers/browser_deps.py @@ -0,0 +1,109 @@ +"""Install and verify Playwright + Chromium for JavaScript crawls.""" + +from __future__ import annotations + +import os +import shutil +import subprocess +import sys +from pathlib import Path + +from .browser import _BROWSER_INSTALL_MSG + +_CHROME_NAMES = ("chromium", "chromium-browser", "google-chrome", "google-chrome-stable") + + +def _repo_root() -> Path: + root = (os.environ.get("WEBSITE_PROFILING_ROOT") or "").strip() + if root: + return Path(root) + return Path(__file__).resolve().parents[4] + + +def _auto_install_enabled() -> bool: + flag = os.environ.get("WP_SKIP_BROWSER_AUTO_INSTALL", "").strip().lower() + return flag not in ("1", "true", "yes") + + +def _playwright_importable() -> bool: + try: + import playwright # noqa: F401 + except ImportError: + return False + return True + + +def _system_chromium_available() -> bool: + chrome_path = (os.environ.get("CHROME_PATH") or "").strip() + if chrome_path and os.path.isfile(chrome_path): + return True + return any(shutil.which(name) for name in _CHROME_NAMES) + + +def _playwright_chromium_available() -> bool: + if not _playwright_importable(): + return False + try: + from playwright.sync_api import sync_playwright + + with sync_playwright() as playwright: + executable = (playwright.chromium.executable_path or "").strip() + return bool(executable and os.path.isfile(executable)) + except Exception: + return False + + +def chromium_available() -> bool: + return _system_chromium_available() or _playwright_chromium_available() + + +def browser_status() -> dict[str, str | bool]: + """Non-raising check for JS crawl prerequisites.""" + if not _playwright_importable(): + return {"ok": False, "message": _BROWSER_INSTALL_MSG} + if chromium_available(): + return {"ok": True} + return {"ok": False, "message": _BROWSER_INSTALL_MSG} + + +def _pip_install_browser_requirements() -> None: + req = _repo_root() / "requirements-browser.txt" + if not req.is_file(): + raise RuntimeError(f"Missing {req.name}; cannot auto-install Playwright.") + subprocess.run( + [sys.executable, "-m", "pip", "install", "-q", "-r", str(req)], + check=True, + cwd=_repo_root(), + ) + + +def _playwright_install_chromium() -> None: + subprocess.run( + [sys.executable, "-m", "playwright", "install", "chromium"], + check=True, + cwd=_repo_root(), + ) + + +def ensure_browser_deps(*, install: bool | None = None) -> dict[str, str | bool]: + """Install Playwright and Chromium when missing, then return browser_status().""" + status = browser_status() + if status["ok"]: + return status + + should_install = _auto_install_enabled() if install is None else install + if not should_install: + return status + + try: + if not _playwright_importable(): + _pip_install_browser_requirements() + if not chromium_available(): + _playwright_install_chromium() + except (OSError, subprocess.CalledProcessError) as exc: + return { + "ok": False, + "message": f"{_BROWSER_INSTALL_MSG} Auto-install failed: {exc}", + } + + return browser_status() diff --git a/src/website_profiling/crawl/fetchers/browser_diagnostics.py b/src/website_profiling/crawl/fetchers/browser_diagnostics.py new file mode 100644 index 00000000..e0d8671e --- /dev/null +++ b/src/website_profiling/crawl/fetchers/browser_diagnostics.py @@ -0,0 +1,149 @@ +"""Browser runtime diagnostics: console messages, page errors, failed requests.""" + +from __future__ import annotations + +import json +from typing import Any, Optional + +_TEXT_MAX = 500 + + +def parse_console_levels(raw: str) -> frozenset[str]: + parts = [p.strip().lower() for p in (raw or "error,warning").split(",") if p.strip()] + return frozenset(parts) if parts else frozenset({"error", "warning"}) + + +def truncate_diag_text(value: Any, max_len: int = _TEXT_MAX) -> str: + s = str(value or "") + if len(s) <= max_len: + return s + return s[: max_len - 3] + "..." + + +def finalize_browser_diagnostics( + console: list[dict[str, Any]], + page_errors: list[dict[str, Any]], + failed_requests: list[dict[str, Any]], +) -> dict[str, Any]: + console_error_count = sum(1 for c in console if c.get("level") == "error") + console_warning_count = sum(1 for c in console if c.get("level") == "warning") + return { + "console": console, + "page_errors": page_errors, + "failed_requests": failed_requests, + "summary": { + "console_error_count": console_error_count, + "console_warning_count": console_warning_count, + "page_error_count": len(page_errors), + "failed_request_count": len(failed_requests), + }, + } + + +def merge_browser_into_page_analysis( + page_analysis_json: Optional[str], + browser_diagnostics: Optional[dict[str, Any]], +) -> str: + if not browser_diagnostics: + return page_analysis_json or "{}" + pa: dict[str, Any] = {} + if page_analysis_json: + try: + parsed = json.loads(page_analysis_json) + if isinstance(parsed, dict): + pa = parsed + except json.JSONDecodeError: + pa = {} + pa["browser"] = browser_diagnostics + return json.dumps(pa) + + +def browser_summary_from_page_analysis(pa: dict[str, Any]) -> dict[str, int]: + browser = pa.get("browser") if isinstance(pa.get("browser"), dict) else {} + summary = browser.get("summary") if isinstance(browser.get("summary"), dict) else {} + return { + "console_error_count": int(summary.get("console_error_count") or 0), + "console_warning_count": int(summary.get("console_warning_count") or 0), + "page_error_count": int(summary.get("page_error_count") or 0), + "failed_request_count": int(summary.get("failed_request_count") or 0), + } + + +def _parse_page_analysis_cell(raw: object) -> dict[str, Any]: + if raw is None: + return {} + try: + import pandas as pd + + if isinstance(raw, float) and pd.isna(raw): + return {} + except Exception: + pass + s = str(raw).strip() + if not s or s == "{}": + return {} + try: + o = json.loads(s) + return o if isinstance(o, dict) else {} + except json.JSONDecodeError: + return {} + + +def aggregate_browser_diagnostics_df(df) -> dict[str, Any]: + """Site-level browser diagnostic counts from crawl DataFrame page_analysis cells.""" + pages_with_console_errors = 0 + pages_with_page_errors = 0 + total_console_errors = 0 + total_page_errors = 0 + message_counts: dict[str, dict[str, Any]] = {} + + if df is None or getattr(df, "empty", True) or "page_analysis" not in df.columns: + return {} + + for _, row in df.iterrows(): + pa = _parse_page_analysis_cell(row.get("page_analysis")) + if not pa: + continue + counts = browser_summary_from_page_analysis(pa) + url = str(row.get("url") or "").strip() + ce = counts["console_error_count"] + pe = counts["page_error_count"] + if ce > 0: + pages_with_console_errors += 1 + total_console_errors += ce + if pe > 0: + pages_with_page_errors += 1 + total_page_errors += pe + browser = pa.get("browser") if isinstance(pa.get("browser"), dict) else {} + for msg in browser.get("console") or []: + if not isinstance(msg, dict) or msg.get("level") != "error": + continue + text = str(msg.get("text") or "").strip() + if not text: + continue + bucket = message_counts.setdefault(text, {"text": text, "count": 0, "sample_urls": []}) + bucket["count"] += 1 + if url and url not in bucket["sample_urls"] and len(bucket["sample_urls"]) < 3: + bucket["sample_urls"].append(url) + + if ( + pages_with_console_errors == 0 + and pages_with_page_errors == 0 + and total_console_errors == 0 + and total_page_errors == 0 + ): + return {} + + top_console_messages = sorted( + message_counts.values(), + key=lambda x: int(x.get("count") or 0), + reverse=True, + )[:5] + + return { + "pages_with_console_errors": pages_with_console_errors, + "pages_with_page_errors": pages_with_page_errors, + "total_console_errors": total_console_errors, + "total_page_errors": total_page_errors, + "top_console_messages": top_console_messages, + } diff --git a/src/website_profiling/crawl/fetchers/factory.py b/src/website_profiling/crawl/fetchers/factory.py new file mode 100644 index 00000000..d4086dd2 --- /dev/null +++ b/src/website_profiling/crawl/fetchers/factory.py @@ -0,0 +1,98 @@ +"""Build page fetchers from crawl configuration.""" + +from __future__ import annotations + +from typing import Callable, Literal, Optional + +import requests + +from .base import PageFetcher +from .browser import BrowserFetcher, _BROWSER_INSTALL_MSG +from .browser_deps import browser_status, ensure_browser_deps +from .browser_diagnostics import parse_console_levels +from .hybrid import HybridFetcher +from .static import StaticFetcher + +RenderMode = Literal["static", "javascript", "auto"] + + +def validate_browser_available() -> None: + """Raise RuntimeError if JS crawl prerequisites are missing.""" + status = ensure_browser_deps() + if not status["ok"]: + raise RuntimeError(str(status.get("message") or _BROWSER_INSTALL_MSG)) + + +def _browser_factory( + *, + js_timeout: int, + user_agent: str, + js_concurrency: int, + js_wait_until: str, + js_extra_wait_ms: int, + js_block_resources: bool, + capture_console: bool = True, + console_levels: frozenset[str] | None = None, + capture_failed_requests: bool = False, + console_max_per_page: int = 20, +) -> Callable[[], PageFetcher]: + def _make() -> PageFetcher: + return BrowserFetcher( + timeout=js_timeout, + user_agent=user_agent, + js_concurrency=js_concurrency, + wait_until=js_wait_until, + extra_wait_ms=js_extra_wait_ms, + block_resources=js_block_resources, + capture_console=capture_console, + console_levels=console_levels, + capture_failed_requests=capture_failed_requests, + console_max_per_page=console_max_per_page, + ) + + return _make + + +def build_fetcher( + *, + render_mode: RenderMode = "static", + timeout: int = 12, + user_agent: str = "WebsiteProfilingCrawler/1.0", + session: Optional[requests.Session] = None, + js_concurrency: int = 3, + js_timeout: int = 30, + js_wait_until: str = "domcontentloaded", + js_extra_wait_ms: int = 1500, + js_block_resources: bool = True, + capture_console: bool = True, + js_console_levels: str = "error,warning", + capture_failed_requests: bool = False, + console_max_per_page: int = 20, +) -> PageFetcher: + mode = (render_mode or "static").strip().lower() + levels = parse_console_levels(js_console_levels) + browser_kwargs = dict( + js_timeout=js_timeout, + user_agent=user_agent, + js_concurrency=js_concurrency, + js_wait_until=js_wait_until, + js_extra_wait_ms=js_extra_wait_ms, + js_block_resources=js_block_resources, + capture_console=capture_console, + console_levels=levels, + capture_failed_requests=capture_failed_requests, + console_max_per_page=console_max_per_page, + ) + if mode == "javascript": + validate_browser_available() + return _browser_factory(**browser_kwargs)() + static = StaticFetcher(timeout=timeout, user_agent=user_agent, session=session) + if mode == "static": + return static + if mode == "auto": + validate_browser_available() + return HybridFetcher( + static, + _browser_factory(**browser_kwargs), + ) + return static diff --git a/src/website_profiling/crawl/fetchers/hybrid.py b/src/website_profiling/crawl/fetchers/hybrid.py new file mode 100644 index 00000000..df4d6df2 --- /dev/null +++ b/src/website_profiling/crawl/fetchers/hybrid.py @@ -0,0 +1,50 @@ +"""Static-first fetcher with optional JavaScript fallback for SPA shells.""" + +from __future__ import annotations + +from typing import Callable, Optional + +from .base import FetchResult, PageFetcher +from .spa_heuristics import needs_js_render + + +class HybridFetcher: + """Try static HTTP first; re-fetch with browser when SPA heuristics match.""" + + def __init__( + self, + static: PageFetcher, + browser_factory: Callable[[], PageFetcher], + ) -> None: + self._static = static + self._browser_factory = browser_factory + self._browser_instance: Optional[PageFetcher] = None + + def _get_browser(self) -> PageFetcher: + if self._browser_instance is None: + self._browser_instance = self._browser_factory() + return self._browser_instance + + def fetch(self, url: str) -> FetchResult: + static_result = self._static.fetch(url) + if not needs_js_render(static_result): + return static_result + rendered = self._get_browser().fetch(url) + if rendered.status is None and static_result.status is not None: + return static_result + return rendered + + def refetch_rendered(self, url: str) -> FetchResult: + """Re-fetch with browser (post-parse auto-mode fallback).""" + rendered = self._get_browser().fetch(url) + if rendered.status is None: + static_result = self._static.fetch(url) + if static_result.status is not None: + return static_result + return rendered + + def close(self) -> None: + self._static.close() + if self._browser_instance is not None: + self._browser_instance.close() + self._browser_instance = None diff --git a/src/website_profiling/crawl/fetchers/spa_heuristics.py b/src/website_profiling/crawl/fetchers/spa_heuristics.py new file mode 100644 index 00000000..c4ad055f --- /dev/null +++ b/src/website_profiling/crawl/fetchers/spa_heuristics.py @@ -0,0 +1,95 @@ +"""Detect SPA shells that benefit from JavaScript re-fetch.""" + +from __future__ import annotations + +from .base import FetchResult + +_SPA_MARKERS = ( + "__NEXT_DATA__", + 'id="root"', + "id='root'", + 'id="app"', + "id='app'", + "data-reactroot", + "cdn.shopify.com", + "__NUXT__", + "window.__INITIAL_STATE__", + "_next/static", + "__REACT_DEVTOOLS", + "react.production.min", + "__vue", + "vue.min.js", + "ng-version", + "ng-app", + "svelte", +) + + +def _has_spa_markers(html: str) -> bool: + lower = html.lower() + return any(marker.lower() in lower for marker in _SPA_MARKERS) + + +def _html_word_count(html: str) -> int: + from bs4 import BeautifulSoup + + try: + text = BeautifulSoup(html, "lxml").get_text(separator=" ", strip=True) + return len(text.split()) + except Exception: + return 0 + + +def needs_js_render(result: FetchResult) -> bool: + """True when static HTML looks like a client-rendered shell.""" + if result.fetch_method == "rendered": + return False + if result.status != 200 or not result.text: + return False + html = result.text + html_len = len(html) + if html_len == 0: + return False + + lower = html.lower() + if _has_spa_markers(html): + return True + + script_count = lower.count("= 8 and html_len < 8000: + return True + + word_count = _html_word_count(html) + if word_count < 40 and script_count >= 3 and html_len > 1500: + return True + + return False + + +def needs_js_render_after_parse( + result: FetchResult, + *, + link_count: int, + same_domain_link_count: int, +) -> bool: + """True when parsed static HTML has too few links for a likely SPA shell.""" + if result.fetch_method == "rendered": + return False + if result.status != 200 or not result.text: + return False + html = result.text + html_len = len(html) + if html_len == 0: + return False + if same_domain_link_count > 1: + return False + + lower = html.lower() + script_count = lower.count("= 3 and word_count < 40) + or (html_len > 1500 and link_count == 0) + ) + return has_signal and same_domain_link_count <= 1 diff --git a/src/website_profiling/crawl/fetchers/static.py b/src/website_profiling/crawl/fetchers/static.py new file mode 100644 index 00000000..8059e56d --- /dev/null +++ b/src/website_profiling/crawl/fetchers/static.py @@ -0,0 +1,67 @@ +"""Static HTTP fetcher using requests.""" + +from __future__ import annotations + +import time +from typing import Optional + +import requests + +from .base import HEADER_KEYS, FetchResult + + +class StaticFetcher: + def __init__( + self, + *, + timeout: int = 12, + user_agent: str = "WebsiteProfilingCrawler/1.0", + session: Optional[requests.Session] = None, + ) -> None: + self.timeout = timeout + self.session = session or requests.Session() + if session is None: + self.session.headers.update({"User-Agent": user_agent}) + self._owns_session = session is None + + def fetch(self, url: str) -> FetchResult: + try: + t0 = time.perf_counter() + resp = self.session.get(url, timeout=self.timeout, allow_redirects=True) + response_time_ms = int((time.perf_counter() - t0) * 1000) + ct = resp.headers.get("Content-Type", "") + is_html = resp.status_code == 200 and ( + "text/html" in ct or "application/xhtml+xml" in ct + ) + text = resp.text if is_html else None + content_length = len(resp.content) if resp.content is not None else 0 + final_url = resp.url or url + redirect_chain_length = len(resp.history) + headers_dict = {k: (resp.headers.get(k) or "") for k in HEADER_KEYS} + return FetchResult( + status=resp.status_code, + content_type=ct, + text=text, + response_time_ms=response_time_ms, + content_length=content_length, + final_url=final_url, + headers_dict=headers_dict, + redirect_chain_length=redirect_chain_length, + fetch_method="static", + ) + except Exception: + return FetchResult( + status=None, + content_type=None, + text=None, + response_time_ms=None, + content_length=None, + final_url=None, + headers_dict={}, + redirect_chain_length=0, + fetch_method="static", + ) + + def close(self) -> None: + if self._owns_session: + self.session.close() diff --git a/src/website_profiling/crawl/sitemap.py b/src/website_profiling/crawl/sitemap.py new file mode 100644 index 00000000..39bab8bc --- /dev/null +++ b/src/website_profiling/crawl/sitemap.py @@ -0,0 +1,121 @@ +"""Discover URLs from robots.txt and sitemap.xml for crawl seeding.""" + +from __future__ import annotations + +import xml.etree.ElementTree as ET +from urllib.parse import urlparse + +import requests + +from ..common import normalize_link + +_USER_AGENT = "WebsiteProfilingCrawler/1.0" +_MAX_SITEMAP_URLS = 5000 + + +def _origin(start_url: str) -> str: + parsed = urlparse(start_url) + if not parsed.scheme or not parsed.netloc: + return "" + return f"{parsed.scheme}://{parsed.netloc}" + + +def _sitemap_urls_from_robots(text: str) -> list[str]: + urls: list[str] = [] + for line in text.splitlines(): + line = line.strip() + if line.lower().startswith("sitemap:"): + part = line.split(":", 1)[1].strip() + if part: + urls.append(part) + return urls + + +def _parse_sitemap_xml(content: str, base_url: str) -> tuple[list[str], list[str]]: + """Return (page_urls, nested_sitemap_urls).""" + page_urls: list[str] = [] + nested: list[str] = [] + try: + root = ET.fromstring(content) + except ET.ParseError: + return page_urls, nested + + tag = root.tag.lower() + if tag.endswith("sitemapindex"): + for loc in root.iter(): + if loc.tag.lower().endswith("loc") and loc.text: + nested.append(loc.text.strip()) + elif tag.endswith("urlset"): + for url_el in root.iter(): + if not url_el.tag.lower().endswith("url"): + continue + for loc in url_el: + if loc.tag.lower().endswith("loc") and loc.text: + normalized = normalize_link(base_url, loc.text.strip()) + if normalized: + page_urls.append(normalized) + break + return page_urls, nested + + +def discover_sitemap_urls( + start_url: str, + *, + timeout: int = 12, + max_urls: int = _MAX_SITEMAP_URLS, + session: requests.Session | None = None, +) -> list[str]: + """Collect same-origin page URLs from robots.txt and sitemap chain.""" + origin = _origin(start_url) + if not origin: + return [] + + sess = session or requests.Session() + owns_session = session is None + if owns_session: + sess.headers.update({"User-Agent": _USER_AGENT}) + + sitemap_queue: list[str] = [] + seen_sitemaps: set[str] = set() + found: list[str] = [] + seen_pages: set[str] = set() + + try: + try: + r = sess.get(f"{origin}/robots.txt", timeout=timeout) + if r.status_code == 200 and r.text: + sitemap_queue.extend(_sitemap_urls_from_robots(r.text)) + except Exception: + pass + + if not sitemap_queue: + sitemap_queue.append(f"{origin}/sitemap.xml") + + while sitemap_queue and len(found) < max_urls: + sm_url = sitemap_queue.pop(0).strip() + if not sm_url or sm_url in seen_sitemaps: + continue + seen_sitemaps.add(sm_url) + try: + r = sess.get(sm_url, timeout=timeout) + if r.status_code != 200 or not r.text or "<" not in r.text: + continue + pages, nested = _parse_sitemap_xml(r.text, sm_url) + for n in nested: + if n not in seen_sitemaps: + sitemap_queue.append(n) + for page in pages: + if urlparse(page).netloc != urlparse(origin).netloc: + continue + if page not in seen_pages: + seen_pages.add(page) + found.append(page) + if len(found) >= max_urls: + break + except Exception: + continue + finally: + if owns_session: + sess.close() + + return found diff --git a/src/website_profiling/db/crawl_store.py b/src/website_profiling/db/crawl_store.py index 24d1dac1..81438225 100644 --- a/src/website_profiling/db/crawl_store.py +++ b/src/website_profiling/db/crawl_store.py @@ -29,11 +29,19 @@ def create_crawl_run( conn: Connection, start_url: Optional[str] = None, property_id: Optional[int] = None, + render_mode: Optional[str] = None, ) -> int: - cur = conn.execute( - "INSERT INTO crawl_runs (created_at, start_url, property_id) VALUES (%s, %s, %s) RETURNING id", - (_now_iso(), start_url, property_id), - ) + mode = (render_mode or "static").strip().lower() + try: + cur = conn.execute( + "INSERT INTO crawl_runs (created_at, start_url, property_id, render_mode) VALUES (%s, %s, %s, %s) RETURNING id", + (_now_iso(), start_url, property_id, mode), + ) + except Exception: + cur = conn.execute( + "INSERT INTO crawl_runs (created_at, start_url, property_id) VALUES (%s, %s, %s) RETURNING id", + (_now_iso(), start_url, property_id), + ) row = cur.fetchone() conn.commit() return int(row["id"]) @@ -50,13 +58,32 @@ def get_latest_crawl_run_id(conn: Connection) -> Optional[int]: def get_crawl_run_info(conn: Connection, run_id: int) -> Optional[dict[str, Any]]: try: - cur = conn.execute("SELECT created_at, start_url FROM crawl_runs WHERE id = %s", (run_id,)) + cur = conn.execute( + "SELECT created_at, start_url, render_mode FROM crawl_runs WHERE id = %s", + (run_id,), + ) row = cur.fetchone() if row is None: return None - return {"created_at": row["created_at"], "start_url": row["start_url"]} + out: dict[str, Any] = { + "created_at": row["created_at"], + "start_url": row["start_url"], + } + if "render_mode" in row.keys(): + out["render_mode"] = row["render_mode"] + return out except Exception: - return None + try: + cur = conn.execute( + "SELECT created_at, start_url FROM crawl_runs WHERE id = %s", + (run_id,), + ) + row = cur.fetchone() + if row is None: + return None + return {"created_at": row["created_at"], "start_url": row["start_url"]} + except Exception: + return None def _df_row_to_crawl_json(row: pd.Series) -> dict[str, Any]: @@ -100,7 +127,15 @@ def _canonical_domain_from_report(conn: Connection, report_data: dict[str, Any]) return _extract_hostname(start_url) or _extract_hostname(fallback_url) -_CRAWL_INSERT_SQL = """INSERT INTO crawl_results (crawl_run_id, url, status, title, data) +_CRAWL_INSERT_SQL = """INSERT INTO crawl_results (crawl_run_id, url, status, title, fetch_method, data) +VALUES (%s, %s, %s, %s, %s, %s) +ON CONFLICT (crawl_run_id, url) DO UPDATE SET + status = EXCLUDED.status, + title = EXCLUDED.title, + fetch_method = EXCLUDED.fetch_method, + data = EXCLUDED.data""" + +_CRAWL_INSERT_SQL_LEGACY = """INSERT INTO crawl_results (crawl_run_id, url, status, title, data) VALUES (%s, %s, %s, %s, %s) ON CONFLICT (crawl_run_id, url) DO UPDATE SET status = EXCLUDED.status, @@ -112,7 +147,9 @@ def _crawl_rows_from_df(df: pd.DataFrame, crawl_run_id: int) -> list[tuple]: rows: list[tuple] = [] if df.empty or "url" not in df.columns: return rows - data_cols = [c for c in df.columns if c not in ("url", "crawl_run_id")] + data_cols = [ + c for c in df.columns if c not in ("url", "crawl_run_id", "fetch_method") + ] for rec in df.to_dict(orient="records"): url = str(rec.get("url", "")).rstrip("/") if not url: @@ -120,10 +157,32 @@ def _crawl_rows_from_df(df: pd.DataFrame, crawl_run_id: int) -> list[tuple]: payload = {c: _sanitize_for_json(rec[c]) if not pd.isna(rec.get(c)) else None for c in data_cols} status = str(rec.get("status") or "") if "status" in rec else None title = str(rec.get("title") or "") if "title" in rec else None - rows.append((crawl_run_id, url, status, title, _json_val(payload))) + raw_fm = rec.get("fetch_method") + fetch_method = ( + "static" + if pd.isna(raw_fm) + else (str(raw_fm).strip() or "static") + ) + rows.append((crawl_run_id, url, status, title, fetch_method, _json_val(payload))) return rows +def _write_crawl_rows(conn: Connection, rows: list[tuple]) -> None: + if not rows: + return + normalized: list[tuple] = [] + for row in rows: + if len(row) == 5: + normalized.append((row[0], row[1], row[2], row[3], "static", row[4])) + else: + normalized.append(row) + try: + _executemany(conn, _CRAWL_INSERT_SQL, normalized, page_size=_CRAWL_BATCH_SIZE) + except Exception: + legacy = [(r[0], r[1], r[2], r[3], r[5]) for r in normalized] + _executemany(conn, _CRAWL_INSERT_SQL_LEGACY, legacy, page_size=_CRAWL_BATCH_SIZE) + + def write_crawl_batch( conn: Connection, rows: list[tuple], @@ -131,10 +190,10 @@ def write_crawl_batch( *, commit: bool = True, ) -> None: - """Insert a batch of crawl rows (each tuple: run_id, url, status, title, data Json).""" + """Insert a batch of crawl rows (each tuple: run_id, url, status, title, fetch_method, data Json).""" if not rows: return - _executemany(conn, _CRAWL_INSERT_SQL, rows, page_size=_CRAWL_BATCH_SIZE) + _write_crawl_rows(conn, rows) if commit: conn.commit() @@ -170,37 +229,68 @@ def write_crawl(conn: Connection, df: pd.DataFrame, crawl_run_id: Optional[int] rows = _crawl_rows_from_df(df, target_run_id) if rows: - _executemany(conn, _CRAWL_INSERT_SQL, rows, page_size=_CRAWL_BATCH_SIZE) + _write_crawl_rows(conn, rows) def read_crawl(conn: Connection, run_id: Optional[int] = None) -> pd.DataFrame: try: + return _read_crawl_rows(conn, run_id, include_fetch_method=True) + except Exception: + try: + return _read_crawl_rows(conn, run_id, include_fetch_method=False) + except Exception: + return pd.DataFrame() + + +def _read_crawl_rows( + conn: Connection, + run_id: Optional[int], + *, + include_fetch_method: bool, +) -> pd.DataFrame: + if run_id is None: + run_id = get_latest_crawl_run_id(conn) + if include_fetch_method: if run_id is None: - run_id = get_latest_crawl_run_id(conn) - if run_id is None: - cur = conn.execute("SELECT url, data FROM crawl_results") + cur = conn.execute("SELECT url, fetch_method, data FROM crawl_results") else: cur = conn.execute( - "SELECT url, data FROM crawl_results WHERE crawl_run_id = %s", + "SELECT url, fetch_method, data FROM crawl_results WHERE crawl_run_id = %s", (run_id,), ) - rows = cur.fetchall() - if not rows: - return pd.DataFrame() - records = [] - for row in rows: - rec = {"url": row["url"]} - data = _parse_row_json(row) or {} - if isinstance(data, dict): - rec.update(data) - records.append(rec) - df = pd.DataFrame(records) - for c in _BOOL_COLS: - if c in df.columns: - df[c] = df[c].astype(bool) - return df - except Exception: + elif run_id is None: + cur = conn.execute("SELECT url, data FROM crawl_results") + else: + cur = conn.execute( + "SELECT url, data FROM crawl_results WHERE crawl_run_id = %s", + (run_id,), + ) + rows = cur.fetchall() + if not rows: return pd.DataFrame() + records = [] + for row in rows: + rec: dict[str, Any] = {"url": row["url"]} + fm_col: Optional[str] = None + if include_fetch_method and "fetch_method" in row.keys(): + fm_col = str(row["fetch_method"] or "static").strip() or "static" + data = _parse_row_json(row) or {} + if isinstance(data, dict): + rec.update(data) + if fm_col is not None: + rec["fetch_method"] = fm_col + elif not include_fetch_method: + rec["fetch_method"] = str( + (data.get("fetch_method") if isinstance(data, dict) else None) or "static" + ).strip() or "static" + elif "fetch_method" not in rec: + rec["fetch_method"] = "static" + records.append(rec) + df = pd.DataFrame(records) + for c in _BOOL_COLS: + if c in df.columns: + df[c] = df[c].astype(bool) + return df def write_edges(conn: Connection, edges: list[tuple[str, str]], crawl_run_id: Optional[int] = None) -> None: diff --git a/src/website_profiling/db/historical.py b/src/website_profiling/db/historical.py index 58abe487..0879c28e 100644 --- a/src/website_profiling/db/historical.py +++ b/src/website_profiling/db/historical.py @@ -60,6 +60,7 @@ def read_historical_data() -> dict[str, list]: "lh_audits", "lh_audit_items", "google_data", + "gsc_links_data", "keyword_data", "keyword_history", "keyword_suggest_cache", @@ -166,6 +167,13 @@ def _bulk( ["id", "fetched_at", "data"], {"data": json_t}, ) + _bulk( + """INSERT INTO gsc_links_data (id, fetched_at, property_id, data) + VALUES (%s, %s, %s, %s) ON CONFLICT (id) DO NOTHING""", + data.get("gsc_links_data", []), + ["id", "fetched_at", "property_id", "data"], + {"data": json_t}, + ) _bulk( """INSERT INTO keyword_data (id, fetched_at, data) VALUES (%s, %s, %s) ON CONFLICT (id) DO NOTHING""", diff --git a/src/website_profiling/db/property_store.py b/src/website_profiling/db/property_store.py index 3e3e7f95..434a2a76 100644 --- a/src/website_profiling/db/property_store.py +++ b/src/website_profiling/db/property_store.py @@ -56,7 +56,7 @@ def upsert_property_by_domain( ) row = cur.fetchone() conn.commit() - return int(row[0]) + return int(_row_field(row, "id", index=0)) def resolve_property_id_from_start_url(conn: Connection, start_url: str) -> int | None: @@ -189,20 +189,20 @@ def list_properties_public(conn: Connection) -> list[dict[str, Any]]: ) out: list[dict[str, Any]] = [] for row in cur.fetchall(): - connected_at = row[7] - crawl_auth = row[10] + connected_at = _row_field(row, "google_connected_at", index=7) + crawl_auth = _row_field(row, "crawl_authorized_at", index=10) out.append({ - "id": int(row[0]), - "name": row[1], - "canonical_domain": row[2], - "site_url": row[3], - "gsc_site_url": row[4], - "ga4_property_id": row[5], - "google_auth_mode": row[6], + "id": int(_row_field(row, "id", index=0)), + "name": _row_field(row, "name", index=1), + "canonical_domain": _row_field(row, "canonical_domain", index=2), + "site_url": _row_field(row, "site_url", index=3), + "gsc_site_url": _row_field(row, "gsc_site_url", index=4), + "ga4_property_id": _row_field(row, "ga4_property_id", index=5), + "google_auth_mode": _row_field(row, "google_auth_mode", index=6), "google_connected": connected_at is not None, "google_connected_at": connected_at.isoformat() if connected_at else None, - "google_connected_email": row[8], - "google_date_range_days": row[9], + "google_connected_email": _row_field(row, "google_connected_email", index=8), + "google_date_range_days": _row_field(row, "google_date_range_days", index=9), "crawl_authorized_at": crawl_auth.isoformat() if crawl_auth else None, }) return out diff --git a/src/website_profiling/integrations/google/gsc_links_csv.py b/src/website_profiling/integrations/google/gsc_links_csv.py new file mode 100644 index 00000000..78568e38 --- /dev/null +++ b/src/website_profiling/integrations/google/gsc_links_csv.py @@ -0,0 +1,304 @@ +""" +Parse Google Search Console Links report CSV exports. + +GSC does not expose Links via the Search Console API; users export CSV from the UI. +Auto-detects export type from header row. +""" +from __future__ import annotations + +import csv +import io +from datetime import datetime, timezone +from typing import Any + +from .normalize import build_crawl_norm_map, normalize_url + +_SECTION_KEYS = ( + "top_linking_sites", + "top_linked_pages", + "top_linking_text", + "sample_links", + "latest_links", +) + + +def _norm_header(h: str) -> str: + return (h or "").strip().lower().replace("\ufeff", "") + + +def _find_col(headers: list[str], *needles: str) -> str | None: + for raw in headers: + n = _norm_header(raw) + if all(needle in n for needle in needles): + return raw + for raw in headers: + n = _norm_header(raw) + if any(needle in n for needle in needles): + return raw + return None + + +def _parse_int(val: str) -> int: + s = (val or "").strip().replace(",", "") + if not s or s in ("~", "-"): + return 0 + try: + return int(float(s)) + except ValueError: + return 0 + + +def detect_export_type(headers: list[str]) -> str | None: + """Return export type key or None if unrecognized.""" + norm = [_norm_header(h) for h in headers] + joined = " ".join(norm) + + if "source page" in joined or ("source" in joined and "target page" in joined): + if any("discover" in n or "first" in n for n in norm): + return "latest_links" + return "sample_links" + + if "link text" in joined or ("linking text" in joined): + if "target page" not in joined and "source" not in joined: + return "top_linking_text" + + if "target page" in joined and ("linking sites" in joined or "linking site" in joined): + return "top_linked_pages" + + if ("site" in joined or "domain" in joined) and "target page" in joined: + return "top_linking_sites" + + site_col = _find_col(headers, "site") + target_pages_col = _find_col(headers, "target", "page") + if site_col and target_pages_col: + return "top_linking_sites" + + target_col = _find_col(headers, "target", "page") + linking_sites_col = _find_col(headers, "linking", "site") + if target_col and linking_sites_col: + return "top_linked_pages" + + text_col = _find_col(headers, "link", "text") + if text_col and not target_col: + return "top_linking_text" + + source_col = _find_col(headers, "source") + if source_col and target_col: + return "sample_links" + + return None + + +def parse_gsc_links_csv(csv_text: str) -> tuple[str, list[dict[str, Any]]]: + """ + Parse CSV text. Returns (export_type, rows). + Raises ValueError on empty or unrecognized format. + """ + text = (csv_text or "").strip() + if not text: + raise ValueError("CSV content is empty") + + # GSC exports may use UTF-8 BOM + if text.startswith("\ufeff"): + text = text[1:] + + reader = csv.DictReader(io.StringIO(text)) + if not reader.fieldnames: + raise ValueError("CSV has no header row") + + headers = [h for h in reader.fieldnames if h] + export_type = detect_export_type(headers) + if not export_type: + raise ValueError( + "Unrecognized GSC Links export format. " + "Export from Search Console → Links (Top linking sites, Top linked pages, " + "Top linking text, or Latest/More sample links)." + ) + + rows: list[dict[str, Any]] = [] + for raw_row in reader: + if not raw_row: + continue + row = parse_row(export_type, headers, raw_row) + if row: + rows.append(row) + + if not rows: + raise ValueError("CSV contains no data rows") + + return export_type, rows + + +def parse_row( + export_type: str, + headers: list[str], + raw_row: dict[str, str | None], +) -> dict[str, Any] | None: + """Parse one CSV row into a normalized dict for the given export type.""" + + def get(*needles: str) -> str: + col = _find_col(headers, *needles) + if col and col in raw_row: + return str(raw_row.get(col) or "").strip() + return "" + + if export_type == "top_linking_sites": + site = get("site") or get("domain") + if not site: + return None + return { + "site": site, + "link_count": _parse_int(get("link")), + "target_page_count": _parse_int(get("target", "page")), + } + + if export_type == "top_linked_pages": + target = get("target", "page") + if not target: + return None + return { + "target_page": target, + "link_count": _parse_int(get("link")), + "linking_site_count": _parse_int(get("linking", "site")), + } + + if export_type == "top_linking_text": + text = get("link", "text") or get("linking", "text") + if text == "(empty)": + text = "" + return { + "anchor_text": text, + "link_count": _parse_int(get("link")), + } + + if export_type in ("sample_links", "latest_links"): + source = get("source", "page") or get("source") + target = get("target", "page") or get("target") + if not source and not target: + return None + target_alt = get("target", "url") + discovered = get("discover") or get("first") + anchor = get("link", "text") or get("anchor") + row: dict[str, Any] = { + "source_page": source, + "target_page": target or target_alt, + } + if target_alt and target_alt != row["target_page"]: + row["target_url_on_linking_page"] = target_alt + if anchor: + row["anchor_text"] = anchor if anchor != "(empty)" else "" + if discovered: + row["discovered_at"] = discovered + # Extract domain from source URL + try: + from urllib.parse import urlparse + + host = urlparse(source).netloc.lower().lstrip("www.") + if host: + row["linking_site"] = host + except Exception: + pass + return row + + return None + + +def _empty_snapshot() -> dict[str, Any]: + return { + "imported_at": datetime.now(timezone.utc).isoformat(), + "source": "gsc_links_csv", + "export_types": [], + "row_counts": {}, + "top_linking_sites": [], + "top_linked_pages": [], + "top_linking_text": [], + "sample_links": [], + "latest_links": [], + "sample_links_full_count": 0, + "latest_links_full_count": 0, + "errors": [], + } + + +def merge_parsed_into_snapshot( + base: dict[str, Any] | None, + export_type: str, + rows: list[dict[str, Any]], + *, + crawl_norm_map: dict[str, str] | None = None, +) -> dict[str, Any]: + """Merge parsed rows into snapshot, replacing the section for export_type.""" + out = dict(base) if base else _empty_snapshot() + out["imported_at"] = datetime.now(timezone.utc).isoformat() + out["source"] = "gsc_links_csv" + + export_types: list[str] = list(out.get("export_types") or []) + if export_type not in export_types: + export_types.append(export_type) + out["export_types"] = export_types + + row_counts: dict[str, int] = dict(out.get("row_counts") or {}) + row_counts[export_type] = len(rows) + out["row_counts"] = row_counts + + if crawl_norm_map: + rows = _annotate_crawl_match(rows, export_type, crawl_norm_map) + + out[export_type] = rows + if export_type == "sample_links": + out["sample_links_full_count"] = len(rows) + elif export_type == "latest_links": + out["latest_links_full_count"] = len(rows) + + return out + + +def _annotate_crawl_match( + rows: list[dict[str, Any]], + export_type: str, + crawl_norm_map: dict[str, str], +) -> list[dict[str, Any]]: + """Add target_in_crawl and crawl_url when target matches a crawled URL.""" + if export_type not in ("top_linked_pages", "sample_links", "latest_links"): + return rows + annotated: list[dict[str, Any]] = [] + for row in rows: + r = dict(row) + target = str(r.get("target_page") or "").strip() + if target: + key = normalize_url(target) + if key in crawl_norm_map: + r["target_in_crawl"] = True + r["crawl_url"] = crawl_norm_map[key] + else: + r["target_in_crawl"] = False + annotated.append(r) + return annotated + + +def build_crawl_norm_from_urls(crawl_urls: list[str]) -> dict[str, str]: + links = [{"url": u} for u in crawl_urls if u] + return build_crawl_norm_map(links) + + +def parse_and_merge( + csv_text: str, + existing: dict[str, Any] | None = None, + *, + crawl_urls: list[str] | None = None, + file_name: str = "", +) -> dict[str, Any]: + """Parse one CSV file and merge into snapshot dict (does not persist).""" + export_type, rows = parse_gsc_links_csv(csv_text) + crawl_norm = build_crawl_norm_from_urls(crawl_urls or []) if crawl_urls else None + merged = merge_parsed_into_snapshot( + existing, + export_type, + rows, + crawl_norm_map=crawl_norm, + ) + if file_name: + imports: list[str] = list(merged.get("import_file_names") or []) + imports.append(file_name) + merged["import_file_names"] = imports[-20:] + return merged diff --git a/src/website_profiling/integrations/google/gsc_links_store.py b/src/website_profiling/integrations/google/gsc_links_store.py new file mode 100644 index 00000000..a8929ed2 --- /dev/null +++ b/src/website_profiling/integrations/google/gsc_links_store.py @@ -0,0 +1,140 @@ +""" +Read/write gsc_links_data table (GSC Links CSV import snapshots). +""" +from __future__ import annotations + +from datetime import datetime, timezone +from typing import Any + +from psycopg import Connection +from psycopg.types.json import Json + +from ...db.storage import _parse_row_json, _sanitize_for_json + +_PAYLOAD_SAMPLE_CAP = 2000 + + +def write_gsc_links_data( + conn: Connection, + data: dict[str, Any], + *, + property_id: int | None = None, +) -> None: + """Insert a new gsc_links_data snapshot scoped to property_id.""" + if property_id is None: + raise RuntimeError( + "property_id is required to store GSC links data. Set active_property_id." + ) + fetched_at = data.get("imported_at") or datetime.now(timezone.utc).isoformat() + payload = {**data, "property_id": property_id} + conn.execute( + "INSERT INTO gsc_links_data (fetched_at, data, property_id) VALUES (%s, %s, %s)", + (fetched_at, Json(_sanitize_for_json(payload)), property_id), + ) + conn.commit() + + +def read_latest_gsc_links_data( + conn: Connection, + property_id: int | None = None, + *, + for_report: bool = True, +) -> dict[str, Any] | None: + """Return latest gsc_links_data row for property_id.""" + if property_id is None: + return None + try: + cur = conn.execute( + """ + SELECT data FROM gsc_links_data + WHERE property_id = %s + ORDER BY id DESC LIMIT 1 + """, + (property_id,), + ) + row = cur.fetchone() + if row is None: + return None + data = _parse_row_json(row) + if not isinstance(data, dict): + return None + if for_report: + return _cap_for_payload(data) + return data + except Exception: + return None + + +def _cap_for_payload(data: dict[str, Any]) -> dict[str, Any]: + """Truncate large link lists for report/API payload; full data stays in DB.""" + out = dict(data) + sample = list(out.get("sample_links") or []) + latest = list(out.get("latest_links") or []) + full_sample = len(sample) + full_latest = len(latest) + out["sample_links_full_count"] = full_sample + out["latest_links_full_count"] = full_latest + + combined_cap = _PAYLOAD_SAMPLE_CAP + if len(sample) > combined_cap: + out["sample_links"] = sample[:combined_cap] + if len(latest) > max(0, combined_cap - len(out.get("sample_links") or [])): + latest_cap = max(0, combined_cap - len(out.get("sample_links") or [])) + out["latest_links"] = latest[:latest_cap] + + return out + + +def import_gsc_links_csv( + conn: Connection, + property_id: int, + csv_text: str, + *, + crawl_urls: list[str] | None = None, + file_name: str = "", +) -> dict[str, Any]: + """ + Parse CSV, merge with latest snapshot for property, persist, return summary. + """ + from .gsc_links_csv import parse_and_merge + + existing = read_latest_gsc_links_data(conn, property_id, for_report=False) + merged = parse_and_merge( + csv_text, + existing, + crawl_urls=crawl_urls, + file_name=file_name, + ) + write_gsc_links_data(conn, merged, property_id=property_id) + return { + "ok": True, + "imported_at": merged.get("imported_at"), + "export_types": merged.get("export_types"), + "row_counts": merged.get("row_counts"), + "last_export_type": _last_export_type(merged), + } + + +def _last_export_type(data: dict[str, Any]) -> str | None: + types = data.get("export_types") or [] + return types[-1] if types else None + + +def read_gsc_links_status( + conn: Connection, + property_id: int, +) -> dict[str, Any]: + """Lightweight status for Integrations UI.""" + data = read_latest_gsc_links_data(conn, property_id, for_report=False) + if not data: + return {"hasData": False} + return { + "hasData": True, + "lastImportedAt": data.get("imported_at"), + "exportTypes": data.get("export_types") or [], + "rowCounts": data.get("row_counts") or {}, + "referringDomainCount": len(data.get("top_linking_sites") or []), + "topLinkedPageCount": len(data.get("top_linked_pages") or []), + "sampleLinkCount": len(data.get("sample_links") or []), + "latestLinkCount": len(data.get("latest_links") or []), + } diff --git a/src/website_profiling/reporting/builder.py b/src/website_profiling/reporting/builder.py index cfad1c38..addfc996 100644 --- a/src/website_profiling/reporting/builder.py +++ b/src/website_profiling/reporting/builder.py @@ -232,6 +232,12 @@ def build_edges_from_df( concurrency: int, timeout: int, polite_delay: float, + render_mode: str = "static", + js_timeout: int = 30, + js_concurrency: int = 3, + js_wait_until: str = "domcontentloaded", + js_extra_wait_ms: int = 1500, + js_block_resources: bool = True, ) -> list[tuple[str, str]]: """Build or load edges; return list of (from, to) tuples.""" edges = load_edges(edges_csv) if (edges_csv or "").strip() else [] @@ -260,13 +266,37 @@ def build_edges_from_df( session = requests.Session() session.headers.update({"User-Agent": "WebsiteProfiling/1.0"}) urls = df["url"].tolist()[:max_fetch_for_edges] + mode = (render_mode or "static").strip().lower() + use_js = mode in ("javascript", "auto") + fetcher = None + if use_js: + from ..crawl.fetchers import build_fetcher + + fetcher = build_fetcher( + render_mode="javascript" if mode == "javascript" else "auto", + timeout=timeout, + user_agent="WebsiteProfiling/1.0", + session=session, + js_timeout=js_timeout, + js_concurrency=js_concurrency, + js_wait_until=js_wait_until, + js_extra_wait_ms=js_extra_wait_ms, + js_block_resources=js_block_resources, + ) def fetch(src): try: - r = session.get(src, timeout=timeout, allow_redirects=True) - if r.status_code != 200 or not r.headers.get("Content-Type", "").lower().startswith("text/html"): - return [] - soup = BeautifulSoup(r.text, "lxml") + if fetcher is not None: + r = fetcher.fetch(src) + if r.status != 200 or not r.text: + return [] + html = r.text + else: + resp = session.get(src, timeout=timeout, allow_redirects=True) + if resp.status_code != 200 or not resp.headers.get("Content-Type", "").lower().startswith("text/html"): + return [] + html = resp.text + soup = BeautifulSoup(html, "lxml") out = set() for a in soup.find_all("a", href=True): ln = normalize_link(src, a["href"]) @@ -279,16 +309,20 @@ def fetch(src): except Exception: return [] - with ThreadPoolExecutor(max_workers=concurrency) as ex: - futures = {ex.submit(fetch, u): u for u in urls} - for f in tqdm(as_completed(futures), total=len(futures), desc="Extracting links"): - src = futures[f] - try: - outs = f.result() - except Exception: - outs = [] - for t in outs: - edges.append((src, t)) + try: + with ThreadPoolExecutor(max_workers=concurrency) as ex: + futures = {ex.submit(fetch, u): u for u in urls} + for f in tqdm(as_completed(futures), total=len(futures), desc="Extracting links"): + src = futures[f] + try: + outs = f.result() + except Exception: + outs = [] + for t in outs: + edges.append((src, t)) + finally: + if fetcher is not None: + fetcher.close() return edges @@ -830,6 +864,7 @@ def _build_report_metadata( ml_bundle: dict[str, Any], run_id: Optional[int], crawl_run_created_at: Optional[str], + gsc_links_data: Optional[dict[str, Any]] = None, ) -> dict[str, Any]: """Provenance and crawl scope for agency-facing audits.""" sources: list[str] = ["crawl"] @@ -840,6 +875,8 @@ def _build_report_metadata( sources.append("search_console") if google_data.get("ga4") or google_data.get("ga4_summary"): sources.append("analytics") + if gsc_links_data and "search_console" not in sources: + sources.append("search_console") llm_meta = ml_bundle.get("llm_meta") if isinstance(llm_meta, dict) and llm_meta.get("model"): sources.append("ai") @@ -858,16 +895,37 @@ def _build_report_metadata( if not df.empty and "status" in df.columns: blocked = int((df["status"].astype(str) == "blocked_by_robots").sum()) + render_mode = (str((config or {}).get("crawl_render_mode") or "static")).strip().lower() + js_concurrency = get_int(config or {}, "crawl_js_concurrency", 3) or 3 + static_html_only = render_mode == "static" + + crawl_scope: dict[str, Any] = { + "pages_crawled": pages_crawled, + "max_pages_configured": max_pages_cfg or pages_crawled, + "robots_blocked_count": blocked, + "static_html_only": static_html_only, + "render_mode": render_mode, + "js_concurrency": js_concurrency if not static_html_only else None, + "crawl_limited": bool(max_pages_cfg and pages_crawled >= max_pages_cfg), + } + if not df.empty and "fetch_method" in df.columns: + fm = df["fetch_method"].astype(str).str.strip().str.lower() + pages_static = int((fm == "static").sum()) + pages_rendered = int((fm == "rendered").sum()) + if render_mode == "auto" or pages_rendered > 0: + crawl_scope["pages_static"] = pages_static + crawl_scope["pages_rendered"] = pages_rendered + + from ..crawl.fetchers.browser_diagnostics import aggregate_browser_diagnostics_df + + browser_agg = aggregate_browser_diagnostics_df(df) + if browser_agg and (render_mode != "static" or browser_agg.get("total_console_errors", 0) > 0): + crawl_scope["browser_diagnostics"] = browser_agg + meta: dict[str, Any] = { "data_sources": sources, "generated_at": datetime.now(timezone.utc).isoformat(), - "crawl_scope": { - "pages_crawled": pages_crawled, - "max_pages_configured": max_pages_cfg or pages_crawled, - "robots_blocked_count": blocked, - "static_html_only": True, - "crawl_limited": bool(max_pages_cfg and pages_crawled >= max_pages_cfg), - }, + "crawl_scope": crawl_scope, } if run_id is not None: meta["crawl_run_id"] = run_id @@ -881,6 +939,12 @@ def _build_report_metadata( meta["gsc_row_count"] = gsc.get("row_count") if keywords_data: meta["keywords_enriched_at"] = keywords_data.get("enriched_at") or keywords_data.get("fetched_at") + if gsc_links_data: + meta["gsc_links_imported_at"] = gsc_links_data.get("imported_at") + meta["gsc_links_referring_domains"] = len(gsc_links_data.get("top_linking_sites") or []) + sample_n = len(gsc_links_data.get("sample_links") or []) + latest_n = len(gsc_links_data.get("latest_links") or []) + meta["gsc_links_sample_count"] = sample_n + latest_n if isinstance(llm_meta, dict): meta["llm"] = llm_meta return meta @@ -979,8 +1043,28 @@ def run_simple_report( if not edges and not df.empty: print(" Building edges from crawl data...", flush=True) + render_mode = (str((config or {}).get("crawl_render_mode") or "static")).strip().lower() + js_concurrency_cfg = get_int(config or {}, "crawl_js_concurrency", 3) or 3 + js_timeout_cfg = get_int(config or {}, "crawl_js_timeout", 30) or 30 + js_wait_until_cfg = (str((config or {}).get("crawl_js_wait_until") or "domcontentloaded")).strip() + js_extra_wait_ms_cfg = get_int(config or {}, "crawl_js_extra_wait_ms", 1500) + if js_extra_wait_ms_cfg is None: + js_extra_wait_ms_cfg = 1500 + js_block_resources_cfg = get_bool(config or {}, "crawl_js_block_resources", True) edges = build_edges_from_df( - df, "", same_domain_only, max_fetch_for_edges, concurrency, timeout, 0.12 + df, + "", + same_domain_only, + max_fetch_for_edges, + concurrency, + timeout, + 0.12, + render_mode=render_mode, + js_timeout=js_timeout_cfg, + js_concurrency=js_concurrency_cfg, + js_wait_until=js_wait_until_cfg, + js_extra_wait_ms=js_extra_wait_ms_cfg, + js_block_resources=js_block_resources_cfg, ) print(f" Edges: {len(edges)}.", flush=True) if edges: @@ -1264,6 +1348,14 @@ def _bool_col(col): rec["page_analysis"] = pa_obj rec["internal_link_count"] = int(pa_obj.get("internal_link_count") or 0) rec["external_link_count"] = int(pa_obj.get("external_link_count") or 0) + from ..crawl.fetchers.browser_diagnostics import browser_summary_from_page_analysis + + browser_counts = browser_summary_from_page_analysis(pa_obj) + rec["console_error_count"] = browser_counts["console_error_count"] + rec["page_error_count"] = browser_counts["page_error_count"] + rec["has_browser_errors"] = ( + browser_counts["console_error_count"] > 0 or browser_counts["page_error_count"] > 0 + ) rec["lighthouse"] = lighthouse_for_url(lighthouse_by_url or {}, u) @@ -1433,11 +1525,17 @@ def _bool_col(col): with _db() as conn: google_data: Optional[dict[str, Any]] = None kw_data: Optional[dict[str, Any]] = None + gsc_links: Optional[dict[str, Any]] = None + property_id: Optional[int] = None try: from ..commands.config_resolve import resolve_property_id_from_cfg - from ..integrations.google.store import read_latest_google_data property_id = resolve_property_id_from_cfg(config, conn) + except Exception: + property_id = None + try: + from ..integrations.google.store import read_latest_google_data + google_data = read_latest_google_data(conn, property_id=property_id) if google_data: report_data["google"] = google_data @@ -1445,6 +1543,8 @@ def _bool_col(col): pass try: from ..integrations.google.keyword_store import read_latest_keyword_data + from ..integrations.google.gsc_links_store import read_latest_gsc_links_data + kw_data = read_latest_keyword_data(conn, property_id) if kw_data: rows = kw_data.get("rows") or [] @@ -1452,6 +1552,9 @@ def _bool_col(col): rows = rows[:500] kw_data = {**kw_data, "rows": rows} report_data["keywords"] = kw_data + gsc_links = read_latest_gsc_links_data(conn, property_id) + if gsc_links: + report_data["gsc_links"] = gsc_links except Exception: pass report_data["report_meta"] = _build_report_metadata( @@ -1463,6 +1566,7 @@ def _bool_col(col): ml_bundle, run_id, crawl_run_created_at, + gsc_links, ) db_write_report_payload(conn, report_data) return "postgresql" diff --git a/src/website_profiling/reporting/categories.py b/src/website_profiling/reporting/categories.py index a06e5c2e..3d3ed6ef 100644 --- a/src/website_profiling/reporting/categories.py +++ b/src/website_profiling/reporting/categories.py @@ -198,6 +198,32 @@ def category_technical_seo( )) deductions.append((min(10, max(2, missing_lang // 5)), True)) + if "page_analysis" in df.columns and len(success_df) > 0: + from ..crawl.fetchers.browser_diagnostics import browser_summary_from_page_analysis + + pages_with_console = 0 + for _, row in success_df.iterrows(): + pa = _page_analysis_dict(row) + counts = browser_summary_from_page_analysis(pa) + url = str(row.get("url") or "").strip() + if counts["console_error_count"] > 0: + pages_with_console += 1 + if counts["page_error_count"] > 0 and url: + issues.append(_issue( + "Uncaught JavaScript error during browser render.", + url=url, + priority="High", + recommendation="Fix runtime JS errors that may break page functionality or SEO signals.", + )) + deductions.append((5, True)) + if pages_with_console > 0: + issues.append(_issue( + f"{pages_with_console} page(s) logged console errors during JavaScript rendering.", + priority="High" if pages_with_console > 3 else "Medium", + recommendation="Inspect browser console errors on affected URLs; fix broken scripts or API calls.", + )) + deductions.append((min(15, pages_with_console * 2), True)) + score = _score_deductions(100, deductions) return { "id": "technical_seo", diff --git a/src/website_profiling/tools/export_audit.py b/src/website_profiling/tools/export_audit.py index 06ac5dd9..3aa2b86c 100644 --- a/src/website_profiling/tools/export_audit.py +++ b/src/website_profiling/tools/export_audit.py @@ -78,7 +78,33 @@ def _summary_lines(payload: dict[str, Any]) -> list[tuple[str, str]]: scope_txt += f" (limit {max_p})" if scope.get("crawl_limited"): scope_txt += " — crawl limit reached" + render_mode = scope.get("render_mode") + if render_mode == "javascript": + js_c = scope.get("js_concurrency") + scope_txt += " — JavaScript rendering" + if js_c: + scope_txt += f" ({js_c} parallel pages)" + elif render_mode == "auto": + scope_txt += " — auto rendering (static + JS fallback)" + ps = scope.get("pages_static") + pr = scope.get("pages_rendered") + if ps is not None and pr is not None: + scope_txt += f" ({ps} static, {pr} JavaScript-rendered)" + elif scope.get("static_html_only"): + scope_txt += " — static HTML only" lines.append(("Crawl scope", scope_txt)) + browser_diag = scope.get("browser_diagnostics") + if isinstance(browser_diag, dict): + pce = browser_diag.get("pages_with_console_errors") + tce = browser_diag.get("total_console_errors") + ppe = browser_diag.get("pages_with_page_errors") + if pce or ppe: + parts = [] + if pce: + parts.append(f"{pce} page(s) with console errors ({tce or 0} total)") + if ppe: + parts.append(f"{ppe} page(s) with uncaught JS errors") + lines.append(("Browser diagnostics", "; ".join(parts))) if meta.get("google_fetched_at"): lines.append(("Google data fetched", str(meta["google_fetched_at"]))) summary = payload.get("summary") or {} diff --git a/src/website_profiling/tools/plot.py b/src/website_profiling/tools/plot.py index 1f397e41..efdd30a0 100644 --- a/src/website_profiling/tools/plot.py +++ b/src/website_profiling/tools/plot.py @@ -15,6 +15,12 @@ def run_plot( timeout: int = 10, polite_delay: float = 0.15, use_database: bool = True, + render_mode: Optional[str] = None, + js_timeout: int = 30, + js_concurrency: int = 3, + js_wait_until: str = "domcontentloaded", + js_extra_wait_ms: int = 1500, + js_block_resources: bool = True, ) -> str: """ Load crawl data, build edges (and nodes), write to PostgreSQL. @@ -25,11 +31,15 @@ def run_plot( run_id = None print(" Loading crawl and edges from DB...", flush=True) - from ..db import db_session, get_latest_crawl_run_id, read_crawl, read_edges + from ..db import db_session, get_crawl_run_info, get_latest_crawl_run_id, read_crawl, read_edges with db_session() as conn: run_id = get_latest_crawl_run_id(conn) df = read_crawl(conn, run_id) edges = read_edges(conn, run_id) + if render_mode is None and run_id is not None: + info = get_crawl_run_info(conn, run_id) + if info and info.get("render_mode"): + render_mode = str(info["render_mode"]) print(f" Loaded {len(df)} URLs, {len(edges)} edges.", flush=True) if df.empty and not edges: raise FileNotFoundError("No crawl or edges data in database.") @@ -41,10 +51,24 @@ def run_plot( df = df.copy() df["url"] = df["url"].astype(str).str.rstrip("/") + mode = (render_mode or "static").strip().lower() + if not edges and not df.empty: print(" Building edges from crawl data...", flush=True) edges = build_edges_from_df( - df, "", same_domain_only, max_fetch_for_edges, concurrency, timeout, polite_delay + df, + "", + same_domain_only, + max_fetch_for_edges, + concurrency, + timeout, + polite_delay, + render_mode=mode, + js_timeout=js_timeout, + js_concurrency=js_concurrency, + js_wait_until=js_wait_until, + js_extra_wait_ms=js_extra_wait_ms, + js_block_resources=js_block_resources, ) print(f" Edges: {len(edges)}.", flush=True) diff --git a/tests/conftest.py b/tests/conftest.py index bf909fc2..372ecf86 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -2,9 +2,34 @@ from __future__ import annotations import sys +import threading +from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path +import pytest + ROOT = Path(__file__).resolve().parents[1] SRC = ROOT / "src" if str(SRC) not in sys.path: sys.path.insert(0, str(SRC)) + +FIXTURES = Path(__file__).resolve().parent / "fixtures" + + +class _FixtureHandler(SimpleHTTPRequestHandler): + def __init__(self, *args, **kwargs): + super().__init__(*args, directory=str(FIXTURES), **kwargs) + + def log_message(self, format, *args): + return + + +@pytest.fixture(scope="module") +def spa_server(): + server = ThreadingHTTPServer(("127.0.0.1", 0), _FixtureHandler) + port = server.server_address[1] + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + base = f"http://127.0.0.1:{port}/spa_shell.html" + yield base + server.shutdown() diff --git a/tests/db_test_fakes.py b/tests/db_test_fakes.py index 4e1e9283..0831251b 100644 --- a/tests/db_test_fakes.py +++ b/tests/db_test_fakes.py @@ -59,3 +59,28 @@ def commit(self) -> None: def transaction(self) -> Iterator[None]: yield None + +class CrawlConn(FakeConn): + """FakeConn with fetchone/fetchall routing for crawl_store SQL patterns.""" + + def __init__(self, *, fetchone=None, fetchall=None, boom_execute: bool = False) -> None: + super().__init__() + self._fetchone = fetchone + self._fetchall = fetchall or [] + self.boom_execute = boom_execute + self._cursor = FakeCursor(fetchone_value=fetchone, fetchall_value=fetchall) + + def execute(self, sql: str, params: tuple[Any, ...] | None = None) -> FakeCursor: + self.executed.append((sql, params)) + if self.boom_execute: + raise RuntimeError("boom") + if "RETURNING id" in sql: + self._cursor = FakeCursor(fetchone_value=self._fetchone or {"id": 1}) + elif "SELECT id FROM crawl_runs" in sql: + self._cursor = FakeCursor(fetchone_value=self._fetchone) + elif "FROM crawl_results" in sql or "FROM edges" in sql or "FROM nodes" in sql: + self._cursor = FakeCursor(fetchall_value=self._fetchall) + elif "FROM crawl_runs WHERE" in sql: + self._cursor = FakeCursor(fetchone_value=self._fetchone) + return self._cursor + diff --git a/tests/fixtures/angular_shell.html b/tests/fixtures/angular_shell.html new file mode 100644 index 00000000..07bdf805 --- /dev/null +++ b/tests/fixtures/angular_shell.html @@ -0,0 +1,10 @@ + + +Angular shell + + + + + + + diff --git a/tests/fixtures/console_error.html b/tests/fixtures/console_error.html new file mode 100644 index 00000000..30ec9c19 --- /dev/null +++ b/tests/fixtures/console_error.html @@ -0,0 +1,16 @@ + + + + + Console Error Fixture + + +

Console error fixture

+ + + diff --git a/tests/fixtures/gsc_links/latest_links.csv b/tests/fixtures/gsc_links/latest_links.csv new file mode 100644 index 00000000..cf0e06b3 --- /dev/null +++ b/tests/fixtures/gsc_links/latest_links.csv @@ -0,0 +1,2 @@ +Source page,Target page,Link text,First discovered +https://partner.org/new,https://example.com/about,about us,2024-01-15 diff --git a/tests/fixtures/gsc_links/sample_links.csv b/tests/fixtures/gsc_links/sample_links.csv new file mode 100644 index 00000000..6bc23b2e --- /dev/null +++ b/tests/fixtures/gsc_links/sample_links.csv @@ -0,0 +1,3 @@ +Source page,Target page,Link text +https://partner.org/page,https://example.com/,company name +https://news.site/article,https://example.com/blog/post,read more diff --git a/tests/fixtures/gsc_links/top_linked_pages.csv b/tests/fixtures/gsc_links/top_linked_pages.csv new file mode 100644 index 00000000..87c77fc4 --- /dev/null +++ b/tests/fixtures/gsc_links/top_linked_pages.csv @@ -0,0 +1,3 @@ +Target page,Links,Linking sites +https://example.com/,100,25 +https://example.com/blog/post,12,4 diff --git a/tests/fixtures/gsc_links/top_linking_sites.csv b/tests/fixtures/gsc_links/top_linking_sites.csv new file mode 100644 index 00000000..96b5b1c6 --- /dev/null +++ b/tests/fixtures/gsc_links/top_linking_sites.csv @@ -0,0 +1,4 @@ +Site,Links,Target pages +example.com,42,15 +partner.org,10,3 +news.site,5,2 diff --git a/tests/fixtures/gsc_links/top_linking_text.csv b/tests/fixtures/gsc_links/top_linking_text.csv new file mode 100644 index 00000000..28d189e0 --- /dev/null +++ b/tests/fixtures/gsc_links/top_linking_text.csv @@ -0,0 +1,4 @@ +Link text,Links +company name,30 +read more,8 +(empty),2 diff --git a/tests/fixtures/post_parse_shell.html b/tests/fixtures/post_parse_shell.html new file mode 100644 index 00000000..03ab51e2 --- /dev/null +++ b/tests/fixtures/post_parse_shell.html @@ -0,0 +1,21 @@ + + +Post-parse shell + +
+ + + + + + + diff --git a/tests/fixtures/spa_shell.html b/tests/fixtures/spa_shell.html new file mode 100644 index 00000000..9250372c --- /dev/null +++ b/tests/fixtures/spa_shell.html @@ -0,0 +1,17 @@ + + +SPA shell + +
+ + + diff --git a/tests/fixtures/svelte_shell.html b/tests/fixtures/svelte_shell.html new file mode 100644 index 00000000..b6a0ec74 --- /dev/null +++ b/tests/fixtures/svelte_shell.html @@ -0,0 +1,9 @@ + + +Svelte shell + +
+ + + + diff --git a/tests/fixtures/vue_shell.html b/tests/fixtures/vue_shell.html new file mode 100644 index 00000000..35b0e330 --- /dev/null +++ b/tests/fixtures/vue_shell.html @@ -0,0 +1,9 @@ + + +Vue shell + +
+ + + + diff --git a/tests/test_analysis_crawl_stores_edge_unit.py b/tests/test_analysis_crawl_stores_edge_unit.py new file mode 100644 index 00000000..3bfc891a --- /dev/null +++ b/tests/test_analysis_crawl_stores_edge_unit.py @@ -0,0 +1,599 @@ +"""Edge-case unit tests for analysis, fetchers, crawl_store, and db stores.""" +from __future__ import annotations + +import json +import sys +import types + +import numpy as np +import pandas as pd +import pytest + +from tests.db_test_fakes import CrawlConn, FakeConn, FakeCursor + + +def test_common_mixed_content_srcset_and_links_serialized_fallback() -> None: + from website_profiling.common import parse_links_serialized, parse_seo_extended + + html = """ + + + + """ + ext = parse_seo_extended(html, "https://secure.com/page") + assert ext["mixed_content_count"] >= 1 + + assert parse_links_serialized("[unclosed") == ["[unclosed"] + + +def test_analysis_page_hreflang_preload_and_duplicates() -> None: + from website_profiling.analysis.page import analyze_html + + html = """ + + + + + + + + One + Two + Ext + + + """ + out = analyze_html(html, "https://site.com/page", "https://site.com/page") + assert out["external_link_count"] >= 1 + assert out["preload_count"] >= 1 + assert out["preconnect_count"] >= 1 + + +def test_analysis_page_json_ld_and_table_warnings() -> None: + from website_profiling.analysis.page import analyze_html + + html = """ + + +
HC
+

Only H1

+ + """ + out = analyze_html(html, "https://site.com/p", "https://site.com/p") + assert any(w["id"] == "json_ld_missing_type" for w in out["warnings"]) + + +def test_analysis_local_duplicate_and_language_paths(monkeypatch) -> None: + from website_profiling.analysis import local + + fuzz = types.SimpleNamespace( + token_set_ratio=lambda a, b: 95 if "duplicate phrase" in a and "duplicate phrase" in b else 0 + ) + monkeypatch.setattr(local, "_import_rapidfuzz", lambda: fuzz) + monkeypatch.setattr( + local, + "_import_langdetect", + lambda: (lambda text: "en", type("LDE", (Exception,), {})), + ) + + long_text = "duplicate phrase " * 10 + df = pd.DataFrame( + [ + {"url": "https://a.com/1", "status": "200", "content_type": "text/html", "title": long_text}, + {"url": "https://a.com/2", "status": "200", "content_type": "text/html", "title": long_text}, + {"url": "", "status": "200", "content_type": "text/html", "title": "short"}, + {"url": "https://a.com/3", "status": "404", "content_type": "text/html", "title": long_text}, + ] + ) + groups, mapping = local.compute_duplicate_groups( + df, + { + "enable_duplicate_detection": "true", + "analysis_simhash_hamming": "64", + "analysis_fuzzy_threshold": "90", + }, + ) + assert groups or mapping + + by_url, summary = local.compute_language_signals( + df, + {"enable_language_detection": "true"}, + ) + assert summary["detected_pages"] >= 1 + + merged = local.merge_bundles( + {"language_by_url": {"a": "en"}, "url_duplicate_group_id": {"u": "d1"}}, + {"language_by_url": {"b": "fr"}, "url_duplicate_group_id": {"v": "d2"}}, + ) + assert merged["language_by_url"]["b"] == "fr" + assert merged["url_duplicate_group_id"]["v"] == "d2" + + payload = {"links": [{"url": "https://a.com/x", "page_analysis": {}}]} + local.merge_analysis_into_payload( + payload, + { + "content_duplicates": [], + "url_duplicate_group_id": {}, + "language_by_url": {}, + "language_summary": {}, + "ner_site_summary": {"Person": 2}, + "ml_errors": ["x"], + "similar_internal_by_url": {}, + "spacy_by_url": {}, + "keyphrases_by_url": {}, + }, + ) + assert payload["ner_site_summary"]["Person"] == 2 + assert payload["ml_errors"] == ["x"] + + +def test_browser_diagnostics_pandas_and_aggregate_paths() -> None: + from website_profiling.crawl.fetchers.browser_diagnostics import ( + _parse_page_analysis_cell, + aggregate_browser_diagnostics_df, + ) + + assert _parse_page_analysis_cell(pd.NA) == {} + + df = pd.DataFrame( + [ + {"url": "https://a.com", "page_analysis": "{}"}, + { + "url": "https://b.com", + "page_analysis": json.dumps( + { + "browser": { + "console": [{"level": "info", "text": "ok"}], + "summary": {"console_error_count": 0, "page_error_count": 0}, + } + } + ), + }, + ] + ) + assert aggregate_browser_diagnostics_df(df) == {} + + +def test_spa_and_sitemap_last_lines(monkeypatch) -> None: + from website_profiling.crawl.fetchers.base import FetchResult + from website_profiling.crawl.fetchers.spa_heuristics import needs_js_render_after_parse + from website_profiling.crawl.sitemap import discover_sitemap_urls + + rendered = FetchResult( + status=200, + content_type="text/html", + text="
", + response_time_ms=1, + content_length=20, + final_url="https://x.com", + headers_dict={}, + redirect_chain_length=0, + fetch_method="rendered", + ) + assert needs_js_render_after_parse(rendered, link_count=0, same_domain_link_count=0) is False + + class FakeResp: + def __init__(self, code, text): + self.status_code = code + self.text = text + + class FakeSession: + headers = {} + n = 0 + + def get(self, url, timeout=0): + if url.endswith("/robots.txt"): + return FakeResp(200, "Sitemap: https://example.com/sitemap.xml\n") + self.n += 1 + raise OSError("network") + + def close(self): + pass + + monkeypatch.setattr("website_profiling.crawl.sitemap.requests.Session", lambda: FakeSession()) + assert discover_sitemap_urls("https://example.com") == [] + + +def test_crawl_store_last_branches(monkeypatch) -> None: + from website_profiling.db import crawl_store as cs + from website_profiling.db._common import _json_val + + class SecondNone(CrawlConn): + def execute(self, sql, params=None): + self.executed.append((sql, params)) + if "render_mode" in sql: + raise RuntimeError("no col") + if "FROM crawl_runs WHERE" in sql: + return FakeCursor(fetchone_value=None) + return super().execute(sql, params) + + assert cs.get_crawl_run_info(SecondNone(), 1) is None # type: ignore[arg-type] + + class ItemSeries(pd.Series): + def __getitem__(self, key): + val = super().__getitem__(key) + return val + + row = pd.Series({"url": "u", "n": np.int64(5)}) + out = cs._df_row_to_crawl_json(row) + assert out["n"] == 5 + + monkeypatch.setattr( + "website_profiling.db.crawl_store.urlparse", + lambda *_a, **_k: (_ for _ in ()).throw(ValueError()), + ) + assert cs._extract_hostname("bad") == "" + + batch = CrawlConn() + cs.write_crawl_batch(batch, [(1, "u", "200", "t", "static", _json_val({}))], 1) # type: ignore[arg-type] + + rconn = CrawlConn(fetchall=[{"url": "u", "data": {}}]) + df = cs._read_crawl_rows(rconn, 1, include_fetch_method=True) # type: ignore[arg-type] + assert df.iloc[0]["fetch_method"] == "static" + + rconn2 = CrawlConn(fetchall=[{"url": "u", "data": {"fetch_method": "rendered"}}]) + df2 = cs._read_crawl_rows(rconn2, 1, include_fetch_method=False) # type: ignore[arg-type] + assert df2.iloc[0]["fetch_method"] == "rendered" + + nconn = CrawlConn() + monkeypatch.setattr(cs, "get_latest_crawl_run_id", lambda _c: None) + cs.write_nodes(nconn, pd.DataFrame([{"url": "https://a.com", "count": 1}]), crawl_run_id=None) # type: ignore[arg-type] + + nr = CrawlConn(fetchall=[]) + monkeypatch.setattr(cs, "get_latest_crawl_run_id", lambda _c: None) + empty = cs.read_nodes(nr, run_id=None) # type: ignore[arg-type] + assert list(empty.columns) == ["url", "count"] + assert empty.empty + + +def test_db_stores_last_lines(monkeypatch) -> None: + from website_profiling.db import _common, config_store, llm_cache_store, property_store, report_store + + assert _common._row_field(("only",), "x", index=0) == "only" + assert _common._sanitize_for_json(None) is None + + conn = FakeConn() + config_store.write_pipeline_config(conn, {"k": "v"}, unknown_keys=None) # type: ignore[arg-type] + + wconn = FakeConn() + config_store.write_llm_config(wconn, {"model": "gpt"}, secret_keys={"api_key"}) # type: ignore[arg-type] + assert wconn.executed + + lconn = FakeConn() + lconn.set_next_cursor(FakeCursor(fetchone_value=None)) + assert llm_cache_store.read_llm_cache(lconn, "missing") is None # type: ignore[arg-type] + + lconn2 = FakeConn() + lconn2.set_next_cursor(FakeCursor(fetchone_value={"response_json": {"ok": True}})) + assert json.loads(llm_cache_store.read_llm_cache(lconn2, "k") or "{}")["ok"] is True # type: ignore[arg-type] + + monkeypatch.setattr( + "website_profiling.db.property_store.urlparse", + lambda *_a, **_k: (_ for _ in ()).throw(ValueError()), + ) + monkeypatch.setattr( + "website_profiling.db.report_store.urlparse", + lambda *_a, **_k: (_ for _ in ()).throw(ValueError()), + ) + assert report_store._extract_hostname("x") == "" + assert property_store._extract_hostname("x") == "" + + rconn = FakeConn() + rconn.set_next_cursor(FakeCursor(fetchone_value=None)) + assert report_store.read_report_payload(rconn) is None # type: ignore[arg-type] + + +def test_lighthouse_store_audit_paths(monkeypatch) -> None: + from website_profiling.db import lighthouse_store as ls + + bad_summary = FakeConn() + bad_summary.set_next_cursor(FakeCursor(fetchone_value={"data": [1]})) + assert ls.read_lighthouse_summary(bad_summary) is None # type: ignore[arg-type] + + monkeypatch.setitem( + sys.modules, + "website_profiling.lighthouse.schema", + types.SimpleNamespace( + lhr_to_audit_rows=lambda _d: ( + [ + { + "audit_id": "a", + "category_id": "c", + "score": 1, + "score_display_mode": "numeric", + "title": "t", + "description": "d", + "display_value": "v", + "numeric_value": 1, + "help_text": "h", + "details_type": "table", + "details_headings": '["h"]', + "details_meta": '{"k":1}', + } + ], + [], + ) + ), + ) + audit_conn = FakeConn() + audit_conn.set_next_cursor(FakeCursor(fetchall_value=[])) + ls.write_lh_audits_from_run(audit_conn, 1, {"audits": {}}) # type: ignore[arg-type] + + run_conn = FakeConn() + run_conn.set_next_cursor(FakeCursor(fetchone_value={"data": "not-a-dict"})) + assert ls.read_latest_lighthouse_run_json(run_conn) is None # type: ignore[arg-type] + + class AuditConn(FakeConn): + def execute(self, sql, params=None): + self.executed.append((sql, params)) + if "lh_audit_items" in sql: + return FakeCursor(fetchall_value=[{"row_data": "bad"}]) + return FakeCursor( + fetchall_value=[ + { + "id": 1, + "audit_id": "a", + "category_id": "c", + "title": "t", + "description": "d", + "score": 1, + "score_display_mode": "numeric", + "display_value": "v", + "numeric_value": 1, + "help_text": "h", + "details_type": "table", + "details_headings": None, + "details_meta": "not-json", + } + ] + ) + + audits = ls.read_lh_audits_with_items(AuditConn(), 1) # type: ignore[arg-type] + assert audits[0]["id"] == "a" + + +def test_pipeline_cmd_js_extra_wait_branches(monkeypatch) -> None: + from website_profiling.commands import pipeline_cmd + + real_get_int = pipeline_cmd.get_int + + def fake_get_int(cfg, key, default=None): + if key == "crawl_js_extra_wait_ms": + return None + return real_get_int(cfg, key, default) + + monkeypatch.setattr(pipeline_cmd, "get_int", fake_get_int) + monkeypatch.setattr(pipeline_cmd, "require_start_url", lambda *_a, **_k: "https://a.com") + monkeypatch.setitem( + sys.modules, + "website_profiling.crawl.crawler", + types.SimpleNamespace(run_crawler=lambda **_k: None), + ) + pipeline_cmd._run_crawl({"crawl_render_mode": "static"}, True) + + monkeypatch.setitem( + sys.modules, + "website_profiling.tools.plot", + types.SimpleNamespace(run_plot=lambda **_k: 0), + ) + pipeline_cmd._run_plot({"crawl_render_mode": "static"}, True) + + +def test_remaining_in_scope_edge_cases(monkeypatch) -> None: + from website_profiling.analysis import local + from website_profiling.analysis.page import _json_ld_missing_type, analyze_html + from website_profiling.common import parse_links_serialized, parse_seo_extended + from website_profiling.crawl.fetchers import spa_heuristics + from website_profiling.crawl.fetchers.base import FetchResult + from website_profiling.crawl.fetchers.browser_diagnostics import _parse_page_analysis_cell + from website_profiling.db import _common, config_store, crawl_store as cs + from website_profiling.db import lighthouse_store as ls + + # common: srcset part-level mixed content + literal_eval except + srcset_html = '' + assert parse_seo_extended(srcset_html, "https://secure.com")["mixed_content_count"] >= 1 + assert parse_links_serialized("[not valid python]") == ["[not valid python]"] + + # spa: rendered skip + truthy zero-length html + class TruthyEmpty: + def __bool__(self) -> bool: + return True + + def __len__(self) -> int: + return 0 + + def lower(self) -> str: + return "" + + def count(self, _sub: str) -> int: + return 0 + + rendered = FetchResult(200, "text/html", "", 1, 0, "https://x.com", {}, 0, "rendered") + assert spa_heuristics.needs_js_render(rendered) is False + empty_body = FetchResult(200, "text/html", TruthyEmpty(), 1, 0, "https://x.com", {}, 0, "static") + assert spa_heuristics.needs_js_render(empty_body) is False + assert spa_heuristics.needs_js_render_after_parse(empty_body, link_count=0, same_domain_link_count=0) is False + + # browser diagnostics: None + pandas import failure + assert _parse_page_analysis_cell(None) == {} + real_import = __import__("builtins").__import__ + + def block_pandas(name, *args, **kwargs): + if name == "pandas": + raise ImportError("no pandas") + return real_import(name, *args, **kwargs) + + monkeypatch.setattr("builtins.__import__", block_pandas) + assert _parse_page_analysis_cell(float("nan")) == {} + monkeypatch.setattr("builtins.__import__", real_import) + + # _common: non-dict row without index + assert _common._row_field("plain", "col") is None + assert _common._row_field(("a",), "missing", index=3) is None + + # config_store read_llm_config success + ok_conn = FakeConn() + ok_conn.set_next_cursor(FakeCursor(fetchall_value=[{"key": "model", "value": "gpt"}])) + assert config_store.read_llm_config(ok_conn)["model"] == "gpt" # type: ignore[arg-type] + + # crawl_store: delete-only when no run, or resolve latest run id + nconn = CrawlConn() + monkeypatch.setattr(cs, "get_latest_crawl_run_id", lambda _c: None) + cs.write_nodes(nconn, pd.DataFrame([{"url": "https://a.com", "count": 1}]), crawl_run_id=None) # type: ignore[arg-type] + assert any("DELETE FROM nodes" in sql for sql, _ in nconn.executed) + + rid_conn = CrawlConn() + monkeypatch.setattr(cs, "get_latest_crawl_run_id", lambda _c: 9) + cs.write_nodes(rid_conn, pd.DataFrame([{"url": "https://a.com", "count": 1}]), crawl_run_id=None) # type: ignore[arg-type] + assert any("DELETE FROM nodes WHERE crawl_run_id" in sql for sql, _ in rid_conn.executed) + + # analysis page: no head, stylesheet without href, duplicate links, nested json-ld + assert _json_ld_missing_type({"items": [{"name": "entity without type"}]}) is True + assert _json_ld_missing_type({"@graph": [], "foo": "bar"}) is False + assert _json_ld_missing_type({"outer": {"inner": {"name": "no type"}}}) is True + assert _json_ld_missing_type({"@context": {"name": "no type"}}) is True + page_html = """ + + 12 + bad + ext + + + + + + """ + out = analyze_html(page_html, "https://site.com/p", "https://site.com/p") + assert out["external_link_count"] >= 1 + assert out["internal_link_count"] == 1 + + class MockLink: + def __init__(self, **attrs: str) -> None: + self._attrs = attrs + + def get(self, key: str, default=None): + return self._attrs.get(key, default) + + from website_profiling.analysis import page as page_mod + + class MockSoup: + def find(self, name: str): + if name == "html": + return types.SimpleNamespace(get=lambda _k, default="": default) + if name == "head": + return None + return None + + def find_all(self, name=None, **kwargs): + if name == "link" and kwargs.get("href") is True: + return [ + MockLink(rel="alternate", hreflang="en", href="/en"), + MockLink(rel="preload", href="/a.woff2"), + ] + if name == "link" and kwargs.get("rel"): + return [] + if name in ("a", "script", "img"): + return [] + if name in ("h1", "h2", "h3", "h4", "h5", "h6"): + return [] + return [] + + monkeypatch.setattr(page_mod, "BeautifulSoup", lambda *_a, **_k: MockSoup()) + rel_out = analyze_html("", "https://site.com/p", "https://site.com/p") + assert rel_out["preload_count"] == 1 + assert rel_out["hreflang_alternates"] + + # local: successful langdetect import, singleton buckets, path compression, max groups + detect, _exc = local._import_langdetect() + assert callable(detect) + + singleton_df = pd.DataFrame( + [{"url": "https://a.com/only", "status": "200", "content_type": "text/html", "title": "solo page content here"}] + ) + assert local.compute_duplicate_groups(singleton_df, {"enable_duplicate_detection": "true"})[0] == [] + + monkeypatch.setattr( + local, + "_import_langdetect", + lambda: (lambda _t: "en", type("LDE", (Exception,), {})), + ) + short_df = pd.DataFrame([{"url": "https://a.com", "status": "200", "title": "short"}]) + assert local.compute_language_signals(short_df, {"enable_language_detection": "true"})[0] == {} + + fuzz = types.SimpleNamespace(token_set_ratio=lambda a, b: 100 if a == b else 0) + monkeypatch.setattr(local, "_import_rapidfuzz", lambda: fuzz) + dup_df = pd.DataFrame( + [ + {"url": "https://a.com/1", "status": "200", "content_type": "text/html", "title": "word " * 20}, + {"url": "https://a.com/2", "status": "200", "content_type": "text/html", "title": "word " * 20}, + {"url": "https://a.com/3", "status": "200", "content_type": "text/html", "title": "word " * 20}, + ] + ) + groups, _ = local.compute_duplicate_groups( + dup_df, {"enable_duplicate_detection": "true", "analysis_simhash_hamming": "64"} + ) + assert groups + + import itertools + + sim_seq = itertools.count() + monkeypatch.setattr(local, "simhash_64", lambda _fp: next(sim_seq)) + many_rows = [] + for i in range(200): + title = f"unique duplicate group title number {i} " * 4 + many_rows.append({"url": f"https://a.com/{i}a", "status": "200", "content_type": "text/html", "title": title}) + many_rows.append({"url": f"https://a.com/{i}b", "status": "200", "content_type": "text/html", "title": title}) + many_groups, _ = local.compute_duplicate_groups( + pd.DataFrame(many_rows), + {"enable_duplicate_detection": "true", "analysis_fuzzy_threshold": "90"}, + ) + assert len(many_groups) == 200 + + # lighthouse: summary except, headings/meta json strings, audit read except + class SummaryBoom(FakeConn): + def execute(self, *_a, **_k): + raise RuntimeError("x") + + assert ls.read_lighthouse_summary(SummaryBoom()) is None # type: ignore[arg-type] + + import website_profiling.lighthouse.schema as lh_schema + + monkeypatch.setattr( + lh_schema, + "lhr_to_audit_rows", + lambda _d: ( + [ + { + "audit_id": "a", + "category_id": "c", + "score": 1, + "score_display_mode": "numeric", + "title": "t", + "description": "d", + "display_value": "v", + "numeric_value": 1, + "help_text": "h", + "details_type": "table", + "details_headings": ["Col"], + "details_meta": {"k": 1}, + } + ], + [], + ), + ) + audit_write_conn = FakeConn() + audit_write_conn.set_next_cursor(FakeCursor(fetchall_value=[{"id": 1}])) + ls.write_lh_audits_from_run(audit_write_conn, 1, {"audits": {}}) # type: ignore[arg-type] + + class RunJsonBoom(FakeConn): + def execute(self, *_a, **_k): + raise RuntimeError("db") + + assert ls.read_lighthouse_run_json(RunJsonBoom(), 1) is None # type: ignore[arg-type] + assert ls.read_latest_lighthouse_run_json(RunJsonBoom()) is None # type: ignore[arg-type] + + class FailAudit(FakeConn): + def execute(self, sql, params=None): + raise RuntimeError("boom") + + assert ls.read_lh_audits_with_items(FailAudit(), 1) == [] # type: ignore[arg-type] diff --git a/tests/test_analysis_text.py b/tests/test_analysis_text.py new file mode 100644 index 00000000..bb7e76fc --- /dev/null +++ b/tests/test_analysis_text.py @@ -0,0 +1,88 @@ +from __future__ import annotations + +import math + +import pandas as pd +import pytest + + +def test_top_keywords_as_text_missing_column() -> None: + from website_profiling.analysis.text import top_keywords_as_text + + row = pd.Series({"title": "x"}) + assert top_keywords_as_text(row) == "" + + +def test_top_keywords_as_text_none_nan_empty() -> None: + from website_profiling.analysis.text import top_keywords_as_text + + assert top_keywords_as_text(pd.Series({"top_keywords": None})) == "" + assert top_keywords_as_text(pd.Series({"top_keywords": float("nan")})) == "" + assert top_keywords_as_text(pd.Series({"top_keywords": "[]"})) == "" + assert top_keywords_as_text(pd.Series({"top_keywords": " "})) == "" + + +def test_top_keywords_as_text_invalid_json() -> None: + from website_profiling.analysis.text import top_keywords_as_text + + assert top_keywords_as_text(pd.Series({"top_keywords": "not-json"})) == "" + + +def test_top_keywords_as_text_non_list() -> None: + from website_profiling.analysis.text import top_keywords_as_text + + assert top_keywords_as_text(pd.Series({"top_keywords": '{"word": "a"}'})) == "" + + +def test_top_keywords_as_text_skips_items_without_word() -> None: + from website_profiling.analysis.text import top_keywords_as_text + + row = pd.Series({"top_keywords": '[{"word": "seo"}, {"nope": 1}, "x"]'}) + assert top_keywords_as_text(row) == "seo" + + +def test_top_keywords_as_text_respects_max_terms() -> None: + from website_profiling.analysis.text import top_keywords_as_text + + items = [{"word": f"w{i}"} for i in range(20)] + import json + + row = pd.Series({"top_keywords": json.dumps(items)}) + assert top_keywords_as_text(row, max_terms=3) == "w0 w1 w2" + + +def test_normalize_fingerprint_text_concatenates_columns() -> None: + from website_profiling.analysis.text import normalize_fingerprint_text + + row = pd.Series( + { + "title": " Hello World ", + "h1": "H1", + "meta_description": None, + "heading_sequence": float("nan"), + "og_title": "", + "og_description": "OG", + "twitter_title": "Tw", + "content_excerpt": "Body text", + "top_keywords": '[{"word": "kw1"}]', + } + ) + out = normalize_fingerprint_text(row) + assert "hello world" in out + assert "h1" in out + assert "og" in out + assert "kw1" in out + assert " " not in out + + +def test_normalize_fingerprint_text_truncates() -> None: + from website_profiling.analysis.text import normalize_fingerprint_text + + row = pd.Series({"title": "x" * 15000}) + assert len(normalize_fingerprint_text(row)) == 12000 + + +def test_normalize_fingerprint_text_skips_missing_columns() -> None: + from website_profiling.analysis.text import normalize_fingerprint_text + + assert normalize_fingerprint_text(pd.Series(dtype=object)) == "" diff --git a/tests/test_browser_fetcher_unit.py b/tests/test_browser_fetcher_unit.py new file mode 100644 index 00000000..a1a75382 --- /dev/null +++ b/tests/test_browser_fetcher_unit.py @@ -0,0 +1,848 @@ +from __future__ import annotations + +import asyncio +from typing import Any +from unittest.mock import MagicMock + +import pytest + + +class _FakeResponse: + status = 200 + headers = { + "content-type": "text/html; charset=utf-8", + "Cache-Control": "private", + "ETag": '"abc"', + } + + +class _FakePage: + def __init__(self) -> None: + self.url = "https://example.com/page" + self._handlers: dict[str, list] = {} + + async def goto(self, _url: str, **_kwargs: Any) -> _FakeResponse: + return _FakeResponse() + + async def content(self) -> str: + return "Renderedok" + + async def route(self, _pattern: str, _handler: Any) -> None: + return None + + async def close(self) -> None: + return None + + def on(self, event: str, handler: Any) -> None: + self._handlers.setdefault(event, []).append(handler) + + def remove_listener(self, event: str, handler: Any) -> None: + if event in self._handlers: + self._handlers[event] = [h for h in self._handlers[event] if h is not handler] + + +class _FakeContext: + async def new_page(self) -> _FakePage: + return _FakePage() + + async def close(self) -> None: + return None + + +class _FakeBrowser: + async def new_context(self, **_kwargs: Any) -> _FakeContext: + return _FakeContext() + + async def close(self) -> None: + return None + + +class _FakeChromium: + async def launch(self, **_kwargs: Any) -> _FakeBrowser: + return _FakeBrowser() + + +class _FakePlaywright: + chromium = _FakeChromium() + + async def stop(self) -> None: + return None + + +class _FakePlaywrightContext: + async def start(self) -> _FakePlaywright: + return _FakePlaywright() + + +def _install_fake_playwright(monkeypatch: pytest.MonkeyPatch) -> None: + fake_api = MagicMock() + fake_api.async_playwright = lambda: _FakePlaywrightContext() + monkeypatch.setitem( + __import__("sys").modules, + "playwright", + MagicMock(async_api=fake_api), + ) + monkeypatch.setitem( + __import__("sys").modules, + "playwright.async_api", + fake_api, + ) + + +@pytest.fixture +def fake_playwright(monkeypatch: pytest.MonkeyPatch): + _install_fake_playwright(monkeypatch) + yield + + +def test_browser_fetcher_fetch_applies_extra_wait(fake_playwright): + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + sleeps: list[float] = [] + + async def _sleep(seconds: float) -> None: + sleeps.append(seconds) + + import website_profiling.crawl.fetchers.browser as browser_mod + + original_sleep = browser_mod.asyncio.sleep + browser_mod.asyncio.sleep = _sleep + try: + fetcher = BrowserFetcher( + timeout=5, + js_concurrency=1, + extra_wait_ms=100, + block_resources=False, + capture_console=False, + capture_failed_requests=False, + ) + try: + result = fetcher.fetch("https://example.com/") + assert result.status == 200 + assert sleeps == [0.1] + finally: + fetcher.close() + finally: + browser_mod.asyncio.sleep = original_sleep + + +def test_browser_fetcher_fetch_returns_rendered_html(fake_playwright): + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + fetcher = BrowserFetcher( + timeout=5, + js_concurrency=1, + extra_wait_ms=0, + block_resources=False, + capture_console=False, + capture_failed_requests=False, + ) + try: + result = fetcher.fetch("https://example.com/") + assert result.status == 200 + assert result.fetch_method == "rendered" + assert result.text is not None + assert "Rendered" in result.text + assert result.response_time_ms is not None + assert result.headers_dict.get("Cache-Control") == "private" + finally: + fetcher.close() + + +def test_browser_fetcher_closed_fetch_returns_error(fake_playwright): + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + fetcher = BrowserFetcher(timeout=5, js_concurrency=1, extra_wait_ms=0, block_resources=False) + fetcher.close() + result = fetcher.fetch("https://example.com/") + assert result.status is None + assert result.fetch_method == "rendered" + + +def test_browser_fetcher_captures_console_with_diagnostics(fake_playwright): + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + fetcher = BrowserFetcher( + timeout=5, + js_concurrency=1, + extra_wait_ms=0, + block_resources=True, + capture_console=True, + console_levels=frozenset({"error"}), + ) + try: + result = fetcher.fetch("https://example.com/") + assert result.status == 200 + assert result.browser_diagnostics is not None + assert "summary" in result.browser_diagnostics + finally: + fetcher.close() + + +def test_hybrid_fetch_uses_browser_when_spa_detected(monkeypatch): + from website_profiling.crawl.fetchers.base import FetchResult + from website_profiling.crawl.fetchers.hybrid import HybridFetcher + + static_html = '
' + rendered = FetchResult( + status=200, + content_type="text/html", + text="rendered", + response_time_ms=10, + content_length=30, + final_url="https://example.com/", + headers_dict={}, + redirect_chain_length=0, + fetch_method="rendered", + ) + + class FakeStatic: + def fetch(self, _url): + return FetchResult( + status=200, + content_type="text/html", + text=static_html, + response_time_ms=1, + content_length=len(static_html), + final_url="https://example.com/", + headers_dict={}, + redirect_chain_length=0, + fetch_method="static", + ) + + def close(self): + pass + + class FakeBrowser: + def fetch(self, _url): + return rendered + + def close(self): + pass + + hybrid = HybridFetcher(FakeStatic(), lambda: FakeBrowser()) + try: + out = hybrid.fetch("https://example.com/") + assert out.fetch_method == "rendered" + assert "rendered" in (out.text or "") + finally: + hybrid.close() + + +def test_build_fetcher_javascript_mode(monkeypatch): + from website_profiling.crawl.fetchers.browser import BrowserFetcher + from website_profiling.crawl.fetchers.factory import build_fetcher + + monkeypatch.setattr( + "website_profiling.crawl.fetchers.factory.validate_browser_available", + lambda: None, + ) + _install_fake_playwright(monkeypatch) + + fetcher = build_fetcher(render_mode="javascript", js_timeout=5, js_concurrency=1, js_extra_wait_ms=0) + try: + assert isinstance(fetcher, BrowserFetcher) + finally: + fetcher.close() + + +def test_build_fetcher_auto_mode(monkeypatch): + from website_profiling.crawl.fetchers.factory import build_fetcher + from website_profiling.crawl.fetchers.hybrid import HybridFetcher + + monkeypatch.setattr( + "website_profiling.crawl.fetchers.factory.validate_browser_available", + lambda: None, + ) + + fetcher = build_fetcher(render_mode="auto", timeout=5) + try: + assert isinstance(fetcher, HybridFetcher) + finally: + fetcher.close() + + +def test_merge_browser_into_page_analysis_invalid_json(): + from website_profiling.crawl.fetchers.browser_diagnostics import merge_browser_into_page_analysis + + diag = {"summary": {"console_error_count": 1}} + out = merge_browser_into_page_analysis("not-json", diag) + import json + + parsed = json.loads(out) + assert parsed["browser"]["summary"]["console_error_count"] == 1 + + +def test_browser_summary_from_page_analysis(): + from website_profiling.crawl.fetchers.browser_diagnostics import browser_summary_from_page_analysis + + summary = browser_summary_from_page_analysis( + {"browser": {"summary": {"console_error_count": 2, "page_error_count": 1}}} + ) + assert summary["console_error_count"] == 2 + assert summary["page_error_count"] == 1 + + +def test_parse_console_levels_defaults(): + from website_profiling.crawl.fetchers.browser_diagnostics import parse_console_levels + + assert parse_console_levels("") == frozenset({"error", "warning"}) + assert parse_console_levels("info, error") == frozenset({"info", "error"}) + + +def test_truncate_diag_text(): + from website_profiling.crawl.fetchers.browser_diagnostics import truncate_diag_text + + assert truncate_diag_text("short") == "short" + long = "x" * 600 + assert truncate_diag_text(long).endswith("...") + assert len(truncate_diag_text(long)) == 500 + + +def test_page_diagnostics_collector_handlers_and_detach() -> None: + from website_profiling.crawl.fetchers.browser import _PageDiagnosticsCollector + + collector = _PageDiagnosticsCollector( + capture_console=True, + console_levels=frozenset({"error"}), + capture_failed_requests=True, + max_per_page=3, + ) + + class _FakePage: + def __init__(self) -> None: + self.handlers: dict[str, Any] = {} + + def on(self, event: str, handler: Any) -> None: + self.handlers[event] = handler + + def remove_listener(self, event: str, handler: Any) -> None: + raise RuntimeError("listener missing") + + page = _FakePage() + collector.attach(page) + + class _ConsoleMsg: + type = "error" + text = "console boom" + location = {"url": "https://example.com/app.js", "lineNumber": 7} + + page.handlers["console"](_ConsoleMsg()) + assert collector.console[0]["source_url"] == "https://example.com/app.js" + assert collector.console[0]["line"] == 7 + + class _InfoMsg: + type = "info" + text = "ignored" + location = None + + page.handlers["console"](_InfoMsg()) + assert len(collector.console) == 1 + + page.handlers["console"](_ConsoleMsg()) + page.handlers["console"](_ConsoleMsg()) + assert len(collector.console) == 3 + page.handlers["console"](_ConsoleMsg()) + assert len(collector.console) == 3 + + class _PageErr: + def __str__(self) -> str: + return "page err" + + stack = "stack trace" + + page.handlers["pageerror"](_PageErr()) + page.handlers["pageerror"](_PageErr()) + page.handlers["pageerror"](_PageErr()) + page.handlers["pageerror"](_PageErr()) + assert len(collector.page_errors) == 3 + + class _ReqStr: + url = "https://example.com/a" + method = "GET" + failure = "net::ERR_FAILED" + + class _ReqObj: + url = "https://example.com/b" + method = "POST" + + class failure: + error_text = "timeout" + + class _ReqNone: + url = "https://example.com/c" + method = "HEAD" + failure = None + + page.handlers["requestfailed"](_ReqStr()) + page.handlers["requestfailed"](_ReqObj()) + page.handlers["requestfailed"](_ReqNone()) + page.handlers["requestfailed"](_ReqStr()) + assert len(collector.failed_requests) == 3 + assert collector.failed_requests[0]["failure"] == "net::ERR_FAILED" + assert collector.failed_requests[1]["failure"] == "timeout" + assert collector.failed_requests[2]["failure"] == "" + + collector.detach(page) + + +def test_browser_fetcher_startup_timeout(monkeypatch: pytest.MonkeyPatch, fake_playwright) -> None: + import threading + + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + _orig_wait = threading.Event.wait + + def _wait_false_on_ready(self, timeout=None): + if timeout == 60: + return False + return _orig_wait(self, timeout=timeout) + + monkeypatch.setattr(threading.Event, "wait", _wait_false_on_ready) + + with pytest.raises(RuntimeError, match="60 seconds"): + BrowserFetcher(timeout=5, js_concurrency=1, extra_wait_ms=0, block_resources=False) + + +def test_browser_fetcher_startup_error(monkeypatch: pytest.MonkeyPatch) -> None: + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + class _FailPlaywrightContext: + async def start(self) -> None: + raise RuntimeError("playwright init failed") + + fake_api = MagicMock() + fake_api.async_playwright = lambda: _FailPlaywrightContext() + monkeypatch.setitem(__import__("sys").modules, "playwright", MagicMock(async_api=fake_api)) + monkeypatch.setitem(__import__("sys").modules, "playwright.async_api", fake_api) + + with pytest.raises(RuntimeError, match="JavaScript crawl requires"): + BrowserFetcher(timeout=5, js_concurrency=1, extra_wait_ms=0, block_resources=False) + + +def test_browser_fetcher_uses_chrome_path(monkeypatch: pytest.MonkeyPatch) -> None: + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + launch_kwargs: dict[str, Any] = {} + + class _CapturingChromium: + async def launch(self, **kwargs: Any) -> _FakeBrowser: + launch_kwargs.update(kwargs) + return _FakeBrowser() + + class _CapturingPlaywright: + chromium = _CapturingChromium() + + async def stop(self) -> None: + return None + + class _CapturingPlaywrightContext: + async def start(self) -> _CapturingPlaywright: + return _CapturingPlaywright() + + fake_api = MagicMock() + fake_api.async_playwright = lambda: _CapturingPlaywrightContext() + monkeypatch.setitem(__import__("sys").modules, "playwright", MagicMock(async_api=fake_api)) + monkeypatch.setitem(__import__("sys").modules, "playwright.async_api", fake_api) + monkeypatch.setenv("CHROME_PATH", "/opt/chrome/chrome") + + fetcher = BrowserFetcher(timeout=5, js_concurrency=1, extra_wait_ms=0, block_resources=False) + try: + assert launch_kwargs.get("executable_path") == "/opt/chrome/chrome" + finally: + fetcher.close() + + +class _RouteTestingPage(_FakePage): + async def route(self, _pattern: str, handler: Any) -> None: + class _FakeRoute: + def __init__(self) -> None: + self.aborted = False + self.continued = False + + async def abort(self) -> None: + self.aborted = True + + async def continue_(self) -> None: + self.continued = True + + for resource_type in ("image", "document"): + route = _FakeRoute() + + class _FakeReq: + pass + + _FakeReq.resource_type = resource_type + await handler(route, _FakeReq()) + if resource_type == "image": + assert route.aborted + else: + assert route.continued + + +class _RouteTestingContext(_FakeContext): + async def new_page(self) -> _RouteTestingPage: + return _RouteTestingPage() + + +class _RouteTestingBrowser(_FakeBrowser): + async def new_context(self, **_kwargs: Any) -> _RouteTestingContext: + return _RouteTestingContext() + + +class _RouteTestingChromium: + async def launch(self, **_kwargs: Any) -> _RouteTestingBrowser: + return _RouteTestingBrowser() + + +class _RouteTestingPlaywright: + chromium = _RouteTestingChromium() + + async def stop(self) -> None: + return None + + +class _RouteTestingPlaywrightContext: + async def start(self) -> _RouteTestingPlaywright: + return _RouteTestingPlaywright() + + +def test_browser_fetcher_block_resources_route_handler(monkeypatch: pytest.MonkeyPatch) -> None: + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + fake_api = MagicMock() + fake_api.async_playwright = lambda: _RouteTestingPlaywrightContext() + monkeypatch.setitem(__import__("sys").modules, "playwright", MagicMock(async_api=fake_api)) + monkeypatch.setitem(__import__("sys").modules, "playwright.async_api", fake_api) + + fetcher = BrowserFetcher(timeout=5, js_concurrency=1, extra_wait_ms=0, block_resources=True) + try: + result = fetcher.fetch("https://example.com/") + assert result.status == 200 + finally: + fetcher.close() + + +class _WorkerErrorPage(_FakePage): + def __init__(self) -> None: + self._handlers: dict[str, list] = {} + + async def goto(self, _url: str, **_kwargs: Any) -> _FakeResponse: + return _FakeResponse() + + @property + def url(self) -> str: + raise RuntimeError("url access failed in worker") + + +class _WorkerErrorContext(_FakeContext): + async def new_page(self) -> _WorkerErrorPage: + return _WorkerErrorPage() + + +class _WorkerErrorBrowser(_FakeBrowser): + async def new_context(self, **_kwargs: Any) -> _WorkerErrorContext: + return _WorkerErrorContext() + + +class _WorkerErrorChromium: + async def launch(self, **_kwargs: Any) -> _WorkerErrorBrowser: + return _WorkerErrorBrowser() + + +class _WorkerErrorPlaywright: + chromium = _WorkerErrorChromium() + + async def stop(self) -> None: + return None + + +class _WorkerErrorPlaywrightContext: + async def start(self) -> _WorkerErrorPlaywright: + return _WorkerErrorPlaywright() + + +def test_browser_fetcher_worker_exception_returns_empty_result(monkeypatch: pytest.MonkeyPatch) -> None: + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + fake_api = MagicMock() + fake_api.async_playwright = lambda: _WorkerErrorPlaywrightContext() + monkeypatch.setitem(__import__("sys").modules, "playwright", MagicMock(async_api=fake_api)) + monkeypatch.setitem(__import__("sys").modules, "playwright.async_api", fake_api) + + fetcher = BrowserFetcher(timeout=5, js_concurrency=1, extra_wait_ms=0, block_resources=False) + try: + result = fetcher.fetch("https://example.com/boom") + assert result.status is None + assert result.final_url == "https://example.com/boom" + finally: + fetcher.close() + + +class _CloseFailPage(_FakePage): + async def close(self) -> None: + raise RuntimeError("page close failed") + + +class _CloseFailContext(_FakeContext): + async def new_page(self) -> _CloseFailPage: + return _CloseFailPage() + + async def close(self) -> None: + raise RuntimeError("context close failed") + + +class _CloseFailBrowser(_FakeBrowser): + async def new_context(self, **_kwargs: Any) -> _CloseFailContext: + return _CloseFailContext() + + async def close(self) -> None: + raise RuntimeError("browser close failed") + + +class _CloseFailChromium: + async def launch(self, **_kwargs: Any) -> _CloseFailBrowser: + return _CloseFailBrowser() + + +class _CloseFailPlaywright: + chromium = _CloseFailChromium() + + async def stop(self) -> None: + raise RuntimeError("playwright stop failed") + + +class _CloseFailPlaywrightContext: + async def start(self) -> _CloseFailPlaywright: + return _CloseFailPlaywright() + + +def test_browser_fetcher_close_swallows_cleanup_errors(monkeypatch: pytest.MonkeyPatch) -> None: + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + fake_api = MagicMock() + fake_api.async_playwright = lambda: _CloseFailPlaywrightContext() + monkeypatch.setitem(__import__("sys").modules, "playwright", MagicMock(async_api=fake_api)) + monkeypatch.setitem(__import__("sys").modules, "playwright.async_api", fake_api) + + fetcher = BrowserFetcher(timeout=5, js_concurrency=1, extra_wait_ms=0, block_resources=False) + fetcher.close() + fetcher.close() + + +class _GotoFailPage(_FakePage): + async def goto(self, _url: str, **_kwargs: Any) -> None: + raise RuntimeError("navigation failed") + + +class _GotoFailContext(_FakeContext): + async def new_page(self) -> _GotoFailPage: + return _GotoFailPage() + + +class _GotoFailBrowser(_FakeBrowser): + async def new_context(self, **_kwargs: Any) -> _GotoFailContext: + return _GotoFailContext() + + +class _GotoFailChromium: + async def launch(self, **_kwargs: Any) -> _GotoFailBrowser: + return _GotoFailBrowser() + + +class _GotoFailPlaywright: + chromium = _GotoFailChromium() + + async def stop(self) -> None: + return None + + +class _GotoFailPlaywrightContext: + async def start(self) -> _GotoFailPlaywright: + return _GotoFailPlaywright() + + +def test_browser_fetcher_goto_exception_returns_none_status(monkeypatch: pytest.MonkeyPatch) -> None: + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + fake_api = MagicMock() + fake_api.async_playwright = lambda: _GotoFailPlaywrightContext() + monkeypatch.setitem(__import__("sys").modules, "playwright", MagicMock(async_api=fake_api)) + monkeypatch.setitem(__import__("sys").modules, "playwright.async_api", fake_api) + + fetcher = BrowserFetcher(timeout=5, js_concurrency=1, extra_wait_ms=50, block_resources=False) + try: + result = fetcher.fetch("https://example.com/missing") + assert result.status is None + assert result.content_length == 0 + finally: + fetcher.close() + + +class _NullResponsePage(_FakePage): + async def goto(self, _url: str, **_kwargs: Any) -> None: + return None + + +class _NullResponseContext(_FakeContext): + async def new_page(self) -> _NullResponsePage: + return _NullResponsePage() + + +class _NullResponseBrowser(_FakeBrowser): + async def new_context(self, **_kwargs: Any) -> _NullResponseContext: + return _NullResponseContext() + + +class _NullResponseChromium: + async def launch(self, **_kwargs: Any) -> _NullResponseBrowser: + return _NullResponseBrowser() + + +class _NullResponsePlaywright: + chromium = _NullResponseChromium() + + async def stop(self) -> None: + return None + + +class _NullResponsePlaywrightContext: + async def start(self) -> _NullResponsePlaywright: + return _NullResponsePlaywright() + + +def test_browser_fetcher_null_response_skips_extra_wait(monkeypatch: pytest.MonkeyPatch) -> None: + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + fake_api = MagicMock() + fake_api.async_playwright = lambda: _NullResponsePlaywrightContext() + monkeypatch.setitem(__import__("sys").modules, "playwright", MagicMock(async_api=fake_api)) + monkeypatch.setitem(__import__("sys").modules, "playwright.async_api", fake_api) + + fetcher = BrowserFetcher(timeout=5, js_concurrency=1, extra_wait_ms=500, block_resources=False) + try: + result = fetcher.fetch("https://example.com/null") + assert result.status is None + finally: + fetcher.close() + + +class _ContentFailPage(_FakePage): + async def content(self) -> str: + raise RuntimeError("content extraction failed") + + +class _ContentFailContext(_FakeContext): + async def new_page(self) -> _ContentFailPage: + return _ContentFailPage() + + +class _ContentFailBrowser(_FakeBrowser): + async def new_context(self, **_kwargs: Any) -> _ContentFailContext: + return _ContentFailContext() + + +class _ContentFailChromium: + async def launch(self, **_kwargs: Any) -> _ContentFailBrowser: + return _ContentFailBrowser() + + +class _ContentFailPlaywright: + chromium = _ContentFailChromium() + + async def stop(self) -> None: + return None + + +class _ContentFailPlaywrightContext: + async def start(self) -> _ContentFailPlaywright: + return _ContentFailPlaywright() + + +def test_browser_fetcher_content_exception_returns_none_text(monkeypatch: pytest.MonkeyPatch) -> None: + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + fake_api = MagicMock() + fake_api.async_playwright = lambda: _ContentFailPlaywrightContext() + monkeypatch.setitem(__import__("sys").modules, "playwright", MagicMock(async_api=fake_api)) + monkeypatch.setitem(__import__("sys").modules, "playwright.async_api", fake_api) + + fetcher = BrowserFetcher(timeout=5, js_concurrency=1, extra_wait_ms=0, block_resources=False) + try: + result = fetcher.fetch("https://example.com/") + assert result.status == 200 + assert result.text is None + finally: + fetcher.close() + + +def test_browser_fetcher_fetch_timeout_returns_empty_result( + monkeypatch: pytest.MonkeyPatch, fake_playwright +) -> None: + import concurrent.futures + + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + real_result = concurrent.futures.Future.result + + def _timeout_on_fetch(self, timeout=None): + if timeout is not None: + raise concurrent.futures.TimeoutError() + return real_result(self, timeout=timeout) + + monkeypatch.setattr(concurrent.futures.Future, "result", _timeout_on_fetch) + + fetcher = BrowserFetcher(timeout=5, js_concurrency=1, extra_wait_ms=0, block_resources=False) + try: + result = fetcher.fetch("https://example.com/slow") + assert result.status is None + assert result.final_url == "https://example.com/slow" + finally: + fetcher.close() + + +def test_run_loop_thread_records_startup_error_and_cancels_tasks( + monkeypatch: pytest.MonkeyPatch, +) -> None: + import asyncio + import threading + + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + cancelled: list[bool] = [] + + class _FakeTask: + def cancel(self) -> None: + cancelled.append(True) + + async def _boom_main(self) -> None: + raise RuntimeError("async main failed") + + async def _gather_fail(*_tasks, return_exceptions=False): + raise RuntimeError("gather failed") + + monkeypatch.setattr(BrowserFetcher, "_async_main", _boom_main) + monkeypatch.setattr(asyncio, "all_tasks", lambda _loop: [_FakeTask()]) + monkeypatch.setattr(asyncio, "gather", _gather_fail) + + fetcher = BrowserFetcher.__new__(BrowserFetcher) + fetcher._ready = threading.Event() + fetcher._startup_error = None + fetcher._loop = None + fetcher._thread = None + fetcher._closed = False + fetcher._jobs = None + + fetcher._run_loop_thread() + + assert fetcher._startup_error is not None + assert cancelled + + +def test_browser_fetcher_close_when_loop_unavailable(fake_playwright) -> None: + from website_profiling.crawl.fetchers.browser import BrowserFetcher + + fetcher = BrowserFetcher(timeout=5, js_concurrency=1, extra_wait_ms=0, block_resources=False) + fetcher._loop = None + fetcher._closed = False + fetcher.close() diff --git a/tests/test_cli_dispatch.py b/tests/test_cli_dispatch.py index 646b1c14..ebec6d01 100644 --- a/tests/test_cli_dispatch.py +++ b/tests/test_cli_dispatch.py @@ -17,5 +17,5 @@ def test_cli_help_lists_commands(): timeout=15, ) assert proc.returncode == 0 - for cmd in ("crawl", "report", "plot", "lighthouse", "keywords", "warnings", "enrich", "google"): + for cmd in ("crawl", "report", "plot", "lighthouse", "keywords", "warnings", "enrich", "google", "gsc-links-import"): assert cmd in proc.stdout diff --git a/tests/test_commands_config_stores_edge_unit.py b/tests/test_commands_config_stores_edge_unit.py new file mode 100644 index 00000000..dbaa148f --- /dev/null +++ b/tests/test_commands_config_stores_edge_unit.py @@ -0,0 +1,1167 @@ +"""Edge-case unit tests for commands, config resolution, and db stores.""" +from __future__ import annotations + +import argparse +import json +import math +import sys +import types +from contextlib import contextmanager +from unittest.mock import MagicMock + +import pandas as pd +import pytest + +from tests.db_test_fakes import FakeConn, FakeCursor +from tests.db_test_fakes import CrawlConn + + +# --------------------------------------------------------------------------- +# analysis/local.py +# --------------------------------------------------------------------------- + + +def test_local_import_rapidfuzz_and_langdetect_errors(monkeypatch) -> None: + from website_profiling.analysis import local + + real_import = __import__("builtins").__import__ + + def mock_import(name, *args, **kwargs): + if name == "rapidfuzz": + raise ImportError("no rapidfuzz") + if name == "langdetect": + raise ImportError("no langdetect") + return real_import(name, *args, **kwargs) + + monkeypatch.setattr("builtins.__import__", mock_import) + with pytest.raises(ImportError, match="pip install"): + local._import_rapidfuzz() + with pytest.raises(ImportError, match="pip install"): + local._import_langdetect() + + +def test_hamming_distance() -> None: + from website_profiling.analysis.local import _hamming + + assert _hamming(0b1010, 0b1100) == 2 + + +def test_compute_duplicate_groups_full_paths(monkeypatch) -> None: + from website_profiling.analysis import local + + fuzz = types.SimpleNamespace(token_set_ratio=lambda a, b: 100 if a == b else 50) + monkeypatch.setattr(local, "_import_rapidfuzz", lambda: fuzz) + + df = pd.DataFrame( + [ + { + "url": "https://a.com/1", + "status": "200", + "content_type": "text/html", + "title": "same content here for duplicate", + }, + { + "url": "https://a.com/2", + "status": "200", + "content_type": "text/html", + "title": "same content here for duplicate", + }, + ] + ) + groups, mapping = local.compute_duplicate_groups( + df, {"enable_duplicate_detection": "true", "analysis_simhash_hamming": "3", "analysis_fuzzy_threshold": "90"} + ) + assert len(groups) >= 1 + assert mapping + + # short fingerprint skipped + df_short = pd.DataFrame([{"url": "https://a.com/x", "status": "200", "content_type": "text/html", "title": "hi"}]) + g2, m2 = local.compute_duplicate_groups(df_short, {"enable_duplicate_detection": "true"}) + assert g2 == [] + + +def test_compute_language_signals_langdetect_exception(monkeypatch) -> None: + from website_profiling.analysis import local + + class LangDetectException(Exception): + pass + + def boom(_text): + raise LangDetectException("unknown") + + monkeypatch.setattr(local, "_import_langdetect", lambda: (boom, LangDetectException)) + df = pd.DataFrame( + [{"url": "https://a.com", "status": "200", "title": "Enough text here for language detection test case"}] + ) + by_url, summary = local.compute_language_signals(df, {"enable_language_detection": "true"}) + assert by_url == {} + assert summary["mixed_site"] is False + + +def test_run_local_enrichment_empty_df() -> None: + from website_profiling.analysis.local import run_local_enrichment + + out = run_local_enrichment(pd.DataFrame(), {}) + assert out["content_duplicates"] == [] + + +# --------------------------------------------------------------------------- +# analysis/page.py +# --------------------------------------------------------------------------- + + +def test_visible_anchor_text_and_input_label_paths() -> None: + from website_profiling.analysis.page import _input_has_label, _visible_anchor_text + from bs4 import BeautifulSoup + + soup = BeautifulSoup( + 'child tail', + "lxml", + ) + a = soup.find("a") + assert "x" in _visible_anchor_text(a) + + soup2 = BeautifulSoup( + """ +
+ + + + + +
+ """, + "lxml", + ) + assert _input_has_label(soup2, soup2.find("input", {"name": "h"})) is True + assert _input_has_label(soup2, soup2.find("input", {"id": "q"})) is True + assert _input_has_label(soup2, soup2.find("input", {"id": "e"})) is True + assert _input_has_label(soup2, soup2.find("input", {"name": "w"})) is True + assert _input_has_label(soup2, soup2.find("input", {"name": "bare"})) is False + + +def test_analyze_html_skipped_headings_and_hreflang() -> None: + from website_profiling.analysis.page import analyze_html + + html = """ + + + + + + +

One

+

Skipped

+ + + + + """ + out = analyze_html(html, "https://site.com/p", "https://site.com/p") + assert out["html_lang"] == "en" + assert len(out["hreflang_alternates"]) >= 1 + assert isinstance(out["warnings"], list) + + +def test_json_ld_walk_nested_graph() -> None: + from website_profiling.analysis.page import _json_ld_missing_type + + assert _json_ld_missing_type({"@graph": [{"@type": "Thing"}, {"name": "no type"}]}) is True + assert _json_ld_missing_type([{"@context": "x"}, {"brand": "Acme"}]) is True + + +# --------------------------------------------------------------------------- +# common.py, config.py +# --------------------------------------------------------------------------- + + +def test_load_edges_non_dict_list(tmp_path) -> None: + from website_profiling.common import load_edges + + p = tmp_path / "edges.json" + p.write_text(json.dumps(["not", "dicts"]), encoding="utf-8") + assert load_edges(str(p)) == [] + + +def test_parse_content_text_reading_level_branch() -> None: + from bs4 import BeautifulSoup + from website_profiling.common import parse_content_text + + words = " ".join(["word"] * 50) + html = f"

{words}

" + soup = BeautifulSoup(html, "lxml") + out = parse_content_text(soup, html, excerpt_max_chars=100) + assert out["reading_level"] > 0 + + +def test_detect_tech_wappalyzer_regex_warning_disables(monkeypatch) -> None: + from website_profiling import common + + common._wappalyzer_disabled = False + common._wappalyzer_instance = None + + class FakeWappalyzer: + @staticmethod + def latest(): + return FakeWappalyzer() + + def analyze(self, _page): + return {"X"} + + class FakeWebPage: + def __init__(self, *args, **kwargs): + pass + + import warnings + + def warn(message, *args, **kwargs): + w = warnings.WarningMessage("Compiling regex with unbalanced parenthesis", UserWarning, "") + return [w] + + monkeypatch.setitem( + sys.modules, + "Wappalyzer", + types.SimpleNamespace(Wappalyzer=FakeWappalyzer, WebPage=FakeWebPage), + ) + monkeypatch.setattr(warnings, "catch_warnings", lambda *a, **k: types.SimpleNamespace(__enter__=lambda s: s, __exit__=lambda *x: None, append=lambda m: None)) + monkeypatch.setattr( + warnings, + "simplefilter", + lambda *a, **k: None, + ) + + # Force regex warning path + class Caught: + def __enter__(self): + return self + + def __exit__(self, *_): + return False + + def __iter__(self): + class W: + message = "Compiling regex with unbalanced parenthesis" + + return iter([W()]) + + monkeypatch.setattr(warnings, "catch_warnings", lambda *a, **k: Caught()) + + from bs4 import BeautifulSoup + + soup = BeautifulSoup("", "lxml") + result = common.detect_tech_wappalyzer("https://x.com", "", {}, soup) + assert isinstance(result, str) + + +def test_parse_links_serialized_ast_fallback() -> None: + from website_profiling.common import parse_links_serialized + + assert parse_links_serialized("[bad") == ["[bad"] + assert parse_links_serialized(float("nan")) == [] + + +def test_load_config_from_db_success(monkeypatch) -> None: + from website_profiling.config import load_config_from_db + + monkeypatch.setenv("DATABASE_URL", "postgres://x") + + class Ctx: + def __enter__(self): + return object() + + def __exit__(self, *_): + return False + + monkeypatch.setattr("website_profiling.db.storage.get_database_url", lambda: "postgres://x") + monkeypatch.setattr("website_profiling.db.db_session", lambda: Ctx()) + monkeypatch.setattr( + "website_profiling.db.storage.read_pipeline_config", + lambda _c: ({"start_url": "https://a.com"}, []), + ) + assert load_config_from_db()["start_url"] == "https://a.com" + + +# --------------------------------------------------------------------------- +# spa_heuristics, sitemap, browser_diagnostics +# --------------------------------------------------------------------------- + + +def test_spa_heuristics_non_200_and_empty() -> None: + from website_profiling.crawl.fetchers.base import FetchResult + from website_profiling.crawl.fetchers.spa_heuristics import needs_js_render, needs_js_render_after_parse + + bad = FetchResult( + status=404, + content_type="text/html", + text="", + response_time_ms=1, + content_length=10, + final_url="https://x.com", + headers_dict={}, + redirect_chain_length=0, + fetch_method="static", + ) + assert needs_js_render(bad) is False + empty = FetchResult( + status=200, + content_type="text/html", + text="", + response_time_ms=1, + content_length=0, + final_url="https://x.com", + headers_dict={}, + redirect_chain_length=0, + fetch_method="static", + ) + assert needs_js_render(empty) is False + assert needs_js_render_after_parse(empty, link_count=0, same_domain_link_count=0) is False + + +def test_discover_sitemap_duplicate_and_exception_in_loop(monkeypatch) -> None: + from website_profiling.crawl.sitemap import discover_sitemap_urls + + class FakeResp: + def __init__(self, code, text): + self.status_code = code + self.text = text + + seen = {"n": 0} + + class FakeSession: + headers = {} + + def get(self, url, timeout=0): + if url.endswith("/robots.txt"): + return FakeResp(200, "Sitemap: https://example.com/sitemap.xml\nSitemap: https://example.com/sitemap.xml\n") + if url.endswith("/sitemap.xml"): + seen["n"] += 1 + if seen["n"] > 1: + raise ConnectionError("fail") + return FakeResp( + 200, + """ + https://example.com/p""", + ) + return FakeResp(404, "") + + def close(self): + pass + + monkeypatch.setattr("website_profiling.crawl.sitemap.requests.Session", lambda: FakeSession()) + urls = discover_sitemap_urls("https://example.com", max_urls=5) + assert "https://example.com/p" in urls + + +def test_browser_diagnostics_aggregate_page_errors_and_empty() -> None: + from website_profiling.crawl.fetchers.browser_diagnostics import aggregate_browser_diagnostics_df + + pa = json.dumps( + { + "browser": { + "console": [], + "page_errors": [{"message": "err"}], + "summary": {"console_error_count": 0, "page_error_count": 1}, + } + } + ) + df = pd.DataFrame([{"url": "https://a.com", "page_analysis": pa}]) + agg = aggregate_browser_diagnostics_df(df) + assert agg["pages_with_page_errors"] == 1 + + pa2 = json.dumps({"browser": {"summary": {}}}) + df2 = pd.DataFrame([{"url": "https://a.com", "page_analysis": pa2}]) + assert aggregate_browser_diagnostics_df(df2) == {} + + +# --------------------------------------------------------------------------- +# google_cmd.py +# --------------------------------------------------------------------------- + + +def test_google_cmd_crawl_read_warning(monkeypatch, capsys) -> None: + from website_profiling.commands import google_cmd + import website_profiling.db as db + + fetch_mod = types.SimpleNamespace( + fetch_google_data=lambda **_k: {"errors": []}, + list_properties=lambda **_k: {}, + ) + monkeypatch.setitem(sys.modules, "website_profiling.integrations.google.fetch", fetch_mod) + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.store", + types.SimpleNamespace(write_google_data=lambda *_a, **_k: None), + ) + monkeypatch.setattr(google_cmd, "resolve_property_id_from_cfg", lambda _c: 1) + + class Ctx: + def __enter__(self): + return object() + + def __exit__(self, *_): + return False + + monkeypatch.setattr(db, "db_session", lambda: Ctx()) + monkeypatch.setattr( + db, + "get_latest_crawl_run_id", + lambda _c: (_ for _ in ()).throw(RuntimeError("no crawl")), + ) + + with pytest.raises(SystemExit) as e: + google_cmd.run({}, "/tmp", lambda _k, d: d, argparse.Namespace(list_properties=False, test=False, property_id=None)) + assert e.value.code == 0 + + +def test_google_cmd_test_gsc_ga4_branches(monkeypatch, capsys) -> None: + from website_profiling.commands import google_cmd + + gsc = types.SimpleNamespace( + list_gsc_sites=lambda _c: ["https://example.com/"], + resolve_gsc_site_url=lambda configured, sites: ("https://example.com/", None), + probe_gsc_site=lambda _c, _u: (False, "probe failed"), + describe_gsc_site_mismatch=lambda *_a: "mismatch", + ) + ga4 = types.SimpleNamespace( + list_ga4_properties=lambda _c: ([{"id": "123", "displayName": "P"}], None), + probe_ga4_property=lambda _c, _id: (False, "ga4 probe failed"), + ) + monkeypatch.setitem(sys.modules, "website_profiling.integrations.google.gsc", gsc) + monkeypatch.setitem(sys.modules, "website_profiling.integrations.google.ga4", ga4) + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.auth", + types.SimpleNamespace( + build_credentials=lambda **_k: object(), + resolve_google_targets=lambda **_k: ("https://other.com/", "999", 28), + ), + ) + google_mod = types.ModuleType("google") + auth_mod = types.ModuleType("google.auth") + exc_mod = types.ModuleType("google.auth.exceptions") + exc_mod.RefreshError = type("RefreshError", (Exception,), {}) + auth_mod.exceptions = exc_mod + google_mod.auth = auth_mod + monkeypatch.setitem(sys.modules, "google", google_mod) + monkeypatch.setitem(sys.modules, "google.auth", auth_mod) + monkeypatch.setitem(sys.modules, "google.auth.exceptions", exc_mod) + + with pytest.raises(SystemExit) as e: + google_cmd._run_google_test(1) + assert e.value.code == 1 # warnings from site mismatch note + + +def _install_google_auth(monkeypatch) -> type[Exception]: + google_mod = types.ModuleType("google") + auth_mod = types.ModuleType("google.auth") + exc_mod = types.ModuleType("google.auth.exceptions") + refresh_err = type("RefreshError", (Exception,), {}) + exc_mod.RefreshError = refresh_err + auth_mod.exceptions = exc_mod + google_mod.auth = auth_mod + monkeypatch.setitem(sys.modules, "google", google_mod) + monkeypatch.setitem(sys.modules, "google.auth", auth_mod) + monkeypatch.setitem(sys.modules, "google.auth.exceptions", exc_mod) + return refresh_err + + +def test_google_cmd_refresh_and_test_exception(monkeypatch) -> None: + from website_profiling.commands import google_cmd + + refresh_err = _install_google_auth(monkeypatch) + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.fetch", + types.SimpleNamespace( + fetch_google_data=lambda **_k: (_ for _ in ()).throw(refresh_err()), + list_properties=lambda **_k: {}, + ), + ) + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.store", + types.SimpleNamespace(write_google_data=lambda *_a, **_k: None), + ) + monkeypatch.setattr(google_cmd, "resolve_property_id_from_cfg", lambda _c: 1) + + class Ctx: + def __enter__(self): + return object() + + def __exit__(self, *_): + return False + + monkeypatch.setattr("website_profiling.db.db_session", lambda: Ctx()) + with pytest.raises(SystemExit) as e: + google_cmd.run({}, "/tmp", lambda _k, d: d, argparse.Namespace(list_properties=False, test=False, property_id=None)) + assert e.value.code == 1 + + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.auth", + types.SimpleNamespace( + build_credentials=lambda **_k: (_ for _ in ()).throw(RuntimeError("fail")), + resolve_google_targets=lambda **_k: ("", "", 28), + ), + ) + _install_google_auth(monkeypatch) + with pytest.raises(SystemExit) as e2: + google_cmd._run_google_test(1) + assert e2.value.code == 1 + + +# --------------------------------------------------------------------------- +# pipeline_cmd.py +# --------------------------------------------------------------------------- + + +def test_pipeline_select_lighthouse_urls_missing_status() -> None: + from website_profiling.commands.pipeline_cmd import select_lighthouse_urls_from_crawl + + assert select_lighthouse_urls_from_crawl(pd.DataFrame([{"url": "https://a.com"}]), 5) == [] + + +def test_pipeline_lighthouse_on_pages_and_enrich_failure(monkeypatch) -> None: + from website_profiling.commands import pipeline_cmd + + monkeypatch.setattr( + "website_profiling.db.get_latest_crawl_run_id", + lambda _c: 1, + ) + monkeypatch.setattr( + "website_profiling.db.read_crawl", + lambda _c, _r: pd.DataFrame([{"url": "https://a.com", "status": "200"}]), + ) + + class Ctx: + def __enter__(self): + return object() + + def __exit__(self, *_): + return False + + monkeypatch.setattr("website_profiling.db.db_session", lambda: Ctx()) + monkeypatch.setitem( + sys.modules, + "website_profiling.lighthouse.runner", + types.SimpleNamespace(run_lighthouse_on_pages=lambda **_k: None), + ) + monkeypatch.setattr(pipeline_cmd, "lighthouse_work_dir", lambda: "/tmp/lh") + monkeypatch.setattr(pipeline_cmd, "cleanup_lighthouse_work_dir", lambda _p: None) + + pipeline_cmd._run_lighthouse_on_pages( + {"lighthouse_strategy": "tablet", "lighthouse_mode": "navigation"}, + 5, + ) + + monkeypatch.setattr(pipeline_cmd, "should_enrich_keywords_after_report", lambda _c: True) + monkeypatch.setattr(pipeline_cmd, "google_db_has_gsc", lambda _c: True) + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.keyword_enrich", + types.SimpleNamespace(run_enrichment=lambda _c: (_ for _ in ()).throw(RuntimeError("kw fail"))), + ) + monkeypatch.setitem( + sys.modules, + "website_profiling.reporting.builder", + types.SimpleNamespace(run_simple_report=lambda **_k: "/tmp/report.html"), + ) + monkeypatch.setattr(pipeline_cmd, "require_start_url", lambda _c, for_step="": "https://a.com") + pipeline_cmd._run_report({}, True) + + monkeypatch.setitem( + sys.modules, + "website_profiling.tools.plot", + types.SimpleNamespace(run_plot=lambda **_k: 0), + ) + pipeline_cmd._run_plot({"crawl_render_mode": "bogus", "crawl_js_extra_wait_ms": ""}, True) + + +def test_pipeline_run_prints_steps(monkeypatch, capsys) -> None: + from website_profiling.commands import pipeline_cmd + + monkeypatch.setattr(pipeline_cmd, "_run_crawl", lambda *_a, **_k: None) + monkeypatch.setattr(pipeline_cmd, "_run_report", lambda *_a, **_k: None) + monkeypatch.setattr(pipeline_cmd, "_run_plot", lambda *_a, **_k: None) + monkeypatch.setattr(pipeline_cmd, "_run_lighthouse_on_pages", lambda *_a, **_k: None) + pipeline_cmd.run( + {"run_crawl": "true", "run_report": "true", "run_plot": "true"}, + argparse.Namespace(command=None), + ) + assert "Site Audit" in capsys.readouterr().out + + +# --------------------------------------------------------------------------- +# crawl_store and other db gaps +# --------------------------------------------------------------------------- + + +def test_crawl_store_remaining_branches(monkeypatch) -> None: + from website_profiling.db import crawl_store as cs + from website_profiling.db._common import _json_val + + assert cs.get_crawl_run_info(CrawlConn(fetchone=None), 1) is None # type: ignore[arg-type] + + class DoubleBoom(CrawlConn): + def execute(self, sql, params=None): + raise RuntimeError("always") + + assert cs.get_crawl_run_info(DoubleBoom(), 1) is None # type: ignore[arg-type] + + row = pd.Series({"url": "u", "n": pd.NA, "flag": True}) + row["flag"] = True + out = cs._df_row_to_crawl_json(row) + assert "url" not in out + + monkeypatch.setattr(cs, "get_crawl_run_info", lambda _c, _r: None) + assert cs._canonical_domain_from_report( + FakeConn(), # type: ignore[arg-type] + {"crawl_run_id": 1, "top_pages": [], "links": [{"url": "https://b.com/x"}]}, + ) == "b.com" + + monkeypatch.setattr(cs, "get_crawl_run_info", lambda _c, _r: {"start_url": "https://c.com"}) + assert cs._canonical_domain_from_report(FakeConn(), {"crawl_run_id": 1}) == "c.com" # type: ignore[arg-type] + + assert cs._crawl_rows_from_df(pd.DataFrame(), 1) == [] + cs._write_crawl_rows(CrawlConn(), []) # type: ignore[arg-type] + cs._write_crawl_rows(CrawlConn(), [(1, "u", "200", "t", _json_val({}))]) # type: ignore[arg-type] + + batch_conn = CrawlConn() + cs.write_crawl_batch(batch_conn, [], 1) # type: ignore[arg-type] + cs.write_crawl_batch(batch_conn, [(1, "u", "200", "t", "static", _json_val({}))], 1, commit=False) # type: ignore[arg-type] + + del_conn = CrawlConn(boom_execute=True) + cs.write_crawl(del_conn, pd.DataFrame(), crawl_run_id=None) # type: ignore[arg-type] + + class AlwaysBoom(CrawlConn): + def execute(self, sql, params=None): + raise RuntimeError("x") + + assert cs.read_crawl(AlwaysBoom()).empty # type: ignore[arg-type] + + rconn = CrawlConn(fetchall=[{"url": "u", "fetch_method": "rendered", "data": {}}]) + monkeypatch.setattr(cs, "get_latest_crawl_run_id", lambda _c: None) + df = cs.read_crawl(rconn, run_id=None) # type: ignore[arg-type] + assert df.iloc[0]["fetch_method"] == "rendered" + + rconn2 = CrawlConn(fetchall=[{"url": "u", "data": {"fetch_method": "static"}}]) + monkeypatch.setattr(cs, "get_latest_crawl_run_id", lambda _c: 2) + df2 = cs._read_crawl_rows(rconn2, 2, include_fetch_method=False) # type: ignore[arg-type] + assert df2.iloc[0]["fetch_method"] == "static" + + nconn = CrawlConn() + monkeypatch.setattr(cs, "get_latest_crawl_run_id", lambda _c: None) + cs.write_nodes(nconn, pd.DataFrame([{"url": "https://a.com", "count": 1}]), crawl_run_id=None) # type: ignore[arg-type] + + nread = CrawlConn(fetchall=[]) + monkeypatch.setattr(cs, "get_latest_crawl_run_id", lambda _c: 3) + assert cs.read_nodes(nread, run_id=None).empty # type: ignore[arg-type] + + nread2 = CrawlConn(fetchall=[{"url": "u", "count": 2}]) + df_nodes = cs.read_nodes(nread2, run_id=3) # type: ignore[arg-type] + assert df_nodes.iloc[0]["count"] == 2 + + +def test_db_common_remaining() -> None: + from website_profiling.db import _common + + assert _common._parse_row_json({"data": "not-json"}) == "not-json" + assert _common._row_field(("a", "b"), "missing", index=99) is None + assert _common._sanitize_for_json(3.14) == 3.14 + assert _common._sanitize_for_json(float("nan")) is None + + +def test_config_store_write_empty_unknown() -> None: + from website_profiling.db.config_store import write_pipeline_config + + conn = FakeConn() + write_pipeline_config(conn, {}, unknown_keys=[]) # type: ignore[arg-type] + assert conn.executed + + +def test_historical_read_outer_exception(monkeypatch) -> None: + from website_profiling.db import historical as h + + monkeypatch.setattr(h, "db_session", lambda: (_ for _ in ()).throw(RuntimeError("no session"))) + assert h.read_historical_data()["report_payload"] == [] + + +def test_historical_restore_row_execute_failure(monkeypatch) -> None: + from website_profiling.db import historical as h + + class RowFailConn(FakeConn): + def execute(self, sql, params=None): + self.executed.append((sql, params)) + raise RuntimeError("row fail") + + monkeypatch.setattr( + h, + "_executemany", + lambda *_a, **_k: (_ for _ in ()).throw(RuntimeError("bulk fail")), + ) + h.restore_historical_data( + RowFailConn(), # type: ignore[arg-type] + {"report_payload": [{"id": 1, "generated_at": "x", "site_name": "s", "canonical_domain": "d", "data": {}}]}, + ) + + +def test_lighthouse_store_remaining(monkeypatch) -> None: + from website_profiling.db import lighthouse_store as ls + + conn = FakeConn() + conn.set_next_cursor(FakeCursor(fetchone_value={"data": {"audits": {}}})) + assert ls.read_lighthouse_summary(conn) == {"audits": {}} # type: ignore[arg-type] + + page_conn = FakeConn() + page_conn.set_next_cursor( + FakeCursor(fetchall_value=[{"url": "https://a.com", "data": {"score": 90}}]) + ) + summaries = ls.read_lighthouse_page_summaries(page_conn) # type: ignore[arg-type] + assert summaries["https://a.com"]["score"] == 90 + + runs_conn = FakeConn() + runs_conn.set_next_cursor( + FakeCursor(fetchall_value=[{"id": 5, "url": "https://a.com"}]) + ) + by_url = ls.read_lh_runs_by_url(runs_conn) # type: ignore[arg-type] + assert by_url["https://a.com"] == [5] + + assert ls.read_lighthouse_run_json(FakeConn(), 1) is None # type: ignore[arg-type] + none_conn = FakeConn() + none_conn.set_next_cursor(FakeCursor(fetchone_value=None)) + assert ls.read_latest_lighthouse_run_json(none_conn) is None # type: ignore[arg-type] + + bad_conn = FakeConn() + bad_conn.set_next_cursor(FakeCursor(fetchone_value={"data": [1, 2, 3]})) + assert ls.read_lighthouse_run_json(bad_conn, 2) is None # type: ignore[arg-type] + + monkeypatch.setitem( + sys.modules, + "website_profiling.lighthouse.schema", + types.SimpleNamespace( + lhr_to_audit_rows=lambda _d: ( + [ + { + "audit_id": "a", + "category_id": "c", + "score": 1, + "score_display_mode": "numeric", + "title": "t", + "description": "d", + "display_value": "v", + "numeric_value": 1, + "help_text": "h", + "details_type": "table", + "details_headings": '["h"]', + "details_meta": '{"k":1}', + } + ], + [(0, 0, {"cell": 1})], + ) + ), + ) + audit_conn = FakeConn() + audit_conn.set_next_cursor(FakeCursor(fetchall_value=[{"id": 1}])) + ls.write_lh_audits_from_run(audit_conn, 1, {"audits": {}}) # type: ignore[arg-type] + + item_conn = FakeConn() + item_conn.set_next_cursor( + FakeCursor( + fetchall_value=[ + { + "id": 10, + "audit_id": "a", + "category_id": "c", + "title": "t", + "description": "d", + "score": 1, + "score_display_mode": "numeric", + "display_value": "v", + "numeric_value": 1, + "help_text": "h", + "details_type": "table", + "details_headings": "[]", + "details_meta": "{}", + } + ] + ) + ) + + + +def test_llm_cache_read_exception() -> None: + from website_profiling.db.llm_cache_store import read_llm_cache + + class BoomConn(FakeConn): + def execute(self, *_a, **_k): + raise RuntimeError("x") + + assert read_llm_cache(BoomConn(), "k") is None # type: ignore[arg-type] + + +def test_report_store_read_latest_and_exception() -> None: + from website_profiling.db.report_store import read_report_payload + + conn = FakeConn() + conn.set_next_cursor(FakeCursor(fetchone_value={"data": {"ok": True}})) + assert read_report_payload(conn)["ok"] is True # type: ignore[arg-type] + + class BoomConn(FakeConn): + def execute(self, *_a, **_k): + raise RuntimeError("x") + + assert read_report_payload(BoomConn()) is None # type: ignore[arg-type] + + +def test_google_app_store_sa_dict(monkeypatch) -> None: + from website_profiling.db import google_app_store as gas + + monkeypatch.setitem( + sys.modules, + "google.oauth2.service_account", + types.SimpleNamespace( + Credentials=types.SimpleNamespace( + from_service_account_info=lambda info, scopes=None: {"ok": True, "info": info} + ) + ), + ) + out = gas.build_service_account_credentials( + { + "service_account_json": { + "type": "service_account", + "client_email": "x@y.iam.gserviceaccount.com", + "private_key": "k", + } + } + ) + assert out["ok"] is True + + +def test_config_resolve_shadow_and_missing(monkeypatch, tmp_path, capsys) -> None: + from website_profiling.commands import config_resolve + + monkeypatch.setattr("website_profiling.db.storage.get_data_dir", lambda: str(tmp_path)) + assert config_resolve.shadow_config_path().endswith("pipeline-config.txt") + + monkeypatch.setattr("website_profiling.db.storage.get_database_url", lambda: "postgres://x") + config_resolve.require_database_url() + + config_resolve.cleanup_lighthouse_work_dir("") + config_resolve.cleanup_lighthouse_work_dir("/outside/tmp/not-under-temp") + + assert config_resolve.resolve_property_id_from_cfg(None) is None + + args = argparse.Namespace(config=str(tmp_path / "missing.cfg")) + with pytest.raises(SystemExit): + config_resolve.resolve_config(args) + + monkeypatch.setattr(config_resolve, "require_database_url", lambda: (_ for _ in ()).throw(RuntimeError("no db"))) + args2 = argparse.Namespace(config=None) + with pytest.raises(SystemExit): + config_resolve.resolve_config(args2) + + shadow = tmp_path / "pipeline-config.txt" + shadow.write_text("start_url = https://shadow.com\n", encoding="utf-8") + monkeypatch.setattr(config_resolve, "require_database_url", lambda: None) + monkeypatch.setattr(config_resolve, "load_config_from_db", lambda: {}) + monkeypatch.setattr("website_profiling.db.storage.get_data_dir", lambda: str(tmp_path)) + cfg, _cwd = config_resolve.resolve_config(argparse.Namespace(config=None)) + assert cfg["start_url"] == "https://shadow.com" + + monkeypatch.setattr(config_resolve, "load_config_from_db", lambda: {"start_url": "https://db.com"}) + cfg2, _ = config_resolve.resolve_config(argparse.Namespace(config=None)) + assert cfg2["start_url"] == "https://db.com" + + monkeypatch.setattr(config_resolve, "load_config_from_db", lambda: {}) + (tmp_path / "pipeline-config.txt").unlink() + with pytest.raises(SystemExit): + config_resolve.resolve_config(argparse.Namespace(config=None)) + + +def test_remaining_gaps_misc(monkeypatch) -> None: + """Cover scattered one-line branches across modules.""" + from bs4 import BeautifulSoup + from website_profiling.analysis import local + from website_profiling.analysis.page import _input_has_label, analyze_html + from website_profiling.commands.pipeline_cmd import select_lighthouse_urls_from_crawl + from website_profiling.common import parse_links_serialized, parse_seo_extended + from website_profiling.crawl.fetchers import spa_heuristics + from website_profiling.crawl.fetchers.base import FetchResult + from website_profiling.crawl.fetchers.browser_diagnostics import _parse_page_analysis_cell + from website_profiling.db import crawl_store as cs, config_store, llm_cache_store, report_store + from website_profiling.db import property_store + + # pipeline select - non-matching status + assert select_lighthouse_urls_from_crawl( + pd.DataFrame([{"url": "https://a.com", "status": "404"}]), 5 + ) == [] + + # common parse_links_serialized fallback split + assert parse_links_serialized("https://a.com, https://b.com") == ["https://a.com", "https://b.com"] + + # common parse_seo_extended microdata branch + html = 'X' + ext = parse_seo_extended(html, "https://s.com") + assert ext["has_schema"] is True + + # spa empty status / text + bad = FetchResult( + status=None, + content_type="text/html", + text=None, + response_time_ms=1, + content_length=0, + final_url="https://x.com", + headers_dict={}, + redirect_chain_length=0, + fetch_method="static", + ) + assert spa_heuristics.needs_js_render(bad) is False + from website_profiling.crawl.fetchers.base import FetchResult + + shell = FetchResult( + status=200, + content_type="text/html", + text="x" * 1600, + response_time_ms=1, + content_length=1600, + final_url="https://x.com", + headers_dict={}, + redirect_chain_length=0, + fetch_method="static", + ) + assert spa_heuristics.needs_js_render_after_parse( + shell, link_count=0, same_domain_link_count=0 + ) is True + + # browser diagnostics pandas path + class FakePd: + @staticmethod + def isna(v): + return v != v + + import website_profiling.crawl.fetchers.browser_diagnostics as bd + + assert _parse_page_analysis_cell(float("nan")) == {} + + # crawl_store branches + class NoRow(CrawlConn): + def execute(self, sql, params=None): + if "FROM crawl_runs WHERE" in sql and "render_mode" not in sql: + return FakeCursor(fetchone_value=None) + return super().execute(sql, params) + + assert cs.get_crawl_run_info(NoRow(), 1) is None # type: ignore[arg-type] + + row = pd.Series({"url": "u", "count": 1}) + row["count"] = 1 + out = cs._df_row_to_crawl_json(row) + assert "count" in out + + from website_profiling.db._common import _json_val + + batch_conn = CrawlConn() + cs.write_crawl_batch( + batch_conn, + [(1, "u", "200", "t", "static", _json_val({}))], + 1, + commit=True, + ) + + rconn = CrawlConn(fetchall=[{"url": "u", "fetch_method": None, "data": {}}]) + df = cs._read_crawl_rows(rconn, 1, include_fetch_method=True) # type: ignore[arg-type] + assert df.iloc[0]["fetch_method"] == "static" + + nconn = CrawlConn() + cs.write_nodes(nconn, pd.DataFrame([{"url": "https://a.com", "count": 1}]), crawl_run_id=5) # type: ignore[arg-type] + + empty_nodes = CrawlConn(fetchall=[]) + assert cs.read_nodes(empty_nodes, run_id=5).empty # type: ignore[arg-type] + + # config_store read_llm_config exception + class BoomConn(FakeConn): + def execute(self, *_a, **_k): + raise RuntimeError("x") + + assert config_store.read_llm_config(BoomConn()) == {} # type: ignore[arg-type] + + # llm_cache empty batch early return + assert llm_cache_store.read_llm_cache_batch(FakeConn(), []) == {} # type: ignore[arg-type] + + # report_store without report_id uses latest + rconn = FakeConn() + rconn.set_next_cursor(FakeCursor(fetchone_value={"data": {"latest": True}})) + assert report_store.read_report_payload(rconn, report_id=None)["latest"] is True # type: ignore[arg-type] + + # property_store exception in extract + assert property_store._extract_hostname(object()) == "" # type: ignore[arg-type] + + # analysis page labelledby and nested walk + soup = BeautifulSoup( + 'Name', + "lxml", + ) + inp = soup.find("input") + assert _input_has_label(soup, inp) is True + + out = analyze_html( + '
HeaderCell
', + "https://site.com/t", + "https://site.com/t", + ) + assert isinstance(out["warnings"], list) + + # local merge paths + merged = local.merge_bundles( + {"url_duplicate_group_id": {"a": "d1"}, "ml_errors": []}, + {"url_duplicate_group_id": {"b": "d2"}, "ner_site_summary": {"org": 1}, "ml_errors": ["e"]}, + ) + assert merged["url_duplicate_group_id"]["b"] == "d2" + assert merged["ner_site_summary"]["org"] == 1 + + payload = {"links": [{"url": "https://a.com", "page_analysis": {"signals": {}}}]} + bundle = { + "content_duplicates": [], + "url_duplicate_group_id": {"https://a.com": "dup_0"}, + "language_by_url": {"https://a.com": "en"}, + "similar_internal_by_url": {"https://a.com": ["https://a.com/b"]}, + "spacy_by_url": {"https://a.com": [{"text": "Acme"}]}, + "keyphrases_by_url": {"https://a.com": ["kw"]}, + "language_summary": {}, + "ner_site_summary": {}, + "ml_errors": [], + } + local.merge_analysis_into_payload(payload, bundle) + assert payload["links"][0]["duplicate_group_id"] == "dup_0" + + +def test_google_cmd_ga4_note_and_refresh_in_test(monkeypatch) -> None: + from website_profiling.commands import google_cmd + + _install_google_auth(monkeypatch) + gsc = types.SimpleNamespace( + list_gsc_sites=lambda _c: [], + resolve_gsc_site_url=lambda *_a: (None, "bad"), + probe_gsc_site=lambda *_a: (False, "x"), + describe_gsc_site_mismatch=lambda *_a: "mismatch", + ) + ga4 = types.SimpleNamespace( + list_ga4_properties=lambda _c: ([{"id": "123", "displayName": "P"}], None), + probe_ga4_property=lambda _c, _id: (True, "ok"), + ) + monkeypatch.setitem(sys.modules, "website_profiling.integrations.google.gsc", gsc) + monkeypatch.setitem(sys.modules, "website_profiling.integrations.google.ga4", ga4) + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.auth", + types.SimpleNamespace( + build_credentials=lambda **_k: object(), + resolve_google_targets=lambda **_k: ("https://x.com/", "999", 28), + ), + ) + with pytest.raises(SystemExit) as e: + google_cmd._run_google_test(1) + assert e.value.code == 1 + + refresh_err = _install_google_auth(monkeypatch) + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.auth", + types.SimpleNamespace( + build_credentials=lambda **_k: (_ for _ in ()).throw(refresh_err()), + resolve_google_targets=lambda **_k: ("", "", 28), + ), + ) + with pytest.raises(SystemExit): + google_cmd._run_google_test(1) + + +def test_pipeline_cmd_js_extra_wait_none_branch(monkeypatch) -> None: + from website_profiling.commands import pipeline_cmd + + real_get_int = pipeline_cmd.get_int + + def fake_get_int(cfg, key, default=None): + if key == "crawl_js_extra_wait_ms": + return None + return real_get_int(cfg, key, default) + + monkeypatch.setattr(pipeline_cmd, "get_int", fake_get_int) + monkeypatch.setitem( + sys.modules, + "website_profiling.crawl.crawler", + types.SimpleNamespace(run_crawler=lambda **_k: None), + ) + monkeypatch.setattr(pipeline_cmd, "require_start_url", lambda *_a, **_k: "https://a.com") + pipeline_cmd._run_crawl( + { + "crawl_render_mode": "static", + "preserve_crawl_history": "true", + }, + True, + ) + + monkeypatch.setattr(pipeline_cmd, "require_lighthouse_url", lambda _c: "https://a.com") + monkeypatch.setattr(pipeline_cmd, "lighthouse_work_dir", lambda: "/tmp/lh") + monkeypatch.setattr(pipeline_cmd, "cleanup_lighthouse_work_dir", lambda _p: None) + monkeypatch.setitem( + sys.modules, + "website_profiling.lighthouse.runner", + types.SimpleNamespace(main=lambda **_k: 3), + ) + with pytest.raises(SystemExit) as e: + pipeline_cmd._run_single_lighthouse({"lighthouse_strategy": "tablet"}, True) + assert e.value.code == 3 + + monkeypatch.setitem( + sys.modules, + "website_profiling.lighthouse.runner", + types.SimpleNamespace(main=lambda **_k: 0), + ) + pipeline_cmd._run_single_lighthouse({"lighthouse_strategy": "mobile", "lighthouse_categories": "perf"}, True) + + monkeypatch.setitem( + sys.modules, + "website_profiling.tools.plot", + types.SimpleNamespace(run_plot=lambda **_k: 0), + ) + pipeline_cmd._run_plot({"crawl_render_mode": "static"}, True) + + +def test_sitemap_max_urls_break(monkeypatch) -> None: + from website_profiling.crawl.sitemap import discover_sitemap_urls + + class FakeResp: + def __init__(self, code, text): + self.status_code = code + self.text = text + + class FakeSession: + headers = {} + + def get(self, url, timeout=0): + if url.endswith("/robots.txt"): + return FakeResp(200, "") + if url.endswith("/sitemap.xml"): + urls = "".join( + f"https://example.com/p{i}" for i in range(10) + ) + return FakeResp( + 200, + f'{urls}', + ) + return FakeResp(404, "") + + def close(self): + pass + + monkeypatch.setattr("website_profiling.crawl.sitemap.requests.Session", lambda: FakeSession()) + urls = discover_sitemap_urls("https://example.com", max_urls=3) + assert len(urls) == 3 + + +def test_db_common_row_field_dict_missing_key() -> None: + from website_profiling.db import _common + + assert _common._row_field({"other": 1}, "data") is None + assert _common._sanitize_for_json(True) is True diff --git a/tests/test_commands_page_gsc_unit.py b/tests/test_commands_page_gsc_unit.py new file mode 100644 index 00000000..56acfd5f --- /dev/null +++ b/tests/test_commands_page_gsc_unit.py @@ -0,0 +1,449 @@ +from __future__ import annotations + +import argparse +import json +import sys +import types +from contextlib import contextmanager +from pathlib import Path + +import pandas as pd +import pytest + + +@contextmanager +def _fake_db_session(conn=None): + yield conn or object() + + +def test_gsc_links_cmd_missing_property_id(monkeypatch, capsys) -> None: + from website_profiling.commands import gsc_links_cmd + + monkeypatch.setattr( + "website_profiling.commands.config_resolve.resolve_property_id_from_cfg", + lambda _cfg: None, + ) + args = argparse.Namespace(property_id=None, status=False) + with pytest.raises(SystemExit) as exc: + gsc_links_cmd.run({}, args) + assert exc.value.code == 1 + assert "property-id" in capsys.readouterr().err.lower() + + +def test_gsc_links_cmd_status(monkeypatch, capsys) -> None: + from website_profiling.commands import gsc_links_cmd + + fake_store = types.SimpleNamespace( + read_gsc_links_status=lambda _conn, pid: {"property_id": pid, "has_data": True}, + import_gsc_links_csv=None, + ) + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.gsc_links_store", + fake_store, + ) + monkeypatch.setitem( + sys.modules, + "website_profiling.db", + types.SimpleNamespace( + db_session=_fake_db_session, + get_latest_crawl_run_id=None, + read_crawl=None, + ), + ) + + args = argparse.Namespace(property_id="42", status=True) + with pytest.raises(SystemExit) as exc: + gsc_links_cmd.run({}, args) + assert exc.value.code == 0 + out = json.loads(capsys.readouterr().out) + assert out["property_id"] == 42 + + +def test_gsc_links_cmd_csv_file_import(monkeypatch, capsys, tmp_path: Path) -> None: + from website_profiling.commands import gsc_links_cmd + + csv_path = tmp_path / "links.csv" + csv_path.write_text("Source,Target\nhttps://a.com,https://b.com\n", encoding="utf-8") + + captured: dict = {} + + def fake_import(_conn, pid, csv_text, *, crawl_urls=None, file_name=""): + captured["pid"] = pid + captured["csv"] = csv_text + captured["crawl_urls"] = crawl_urls + captured["file_name"] = file_name + return {"ok": True, "rows": 1} + + fake_store = types.SimpleNamespace( + import_gsc_links_csv=fake_import, + read_gsc_links_status=None, + ) + df = pd.DataFrame({"url": ["https://c.com/", ""]}) + + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.gsc_links_store", + fake_store, + ) + monkeypatch.setitem( + sys.modules, + "website_profiling.db", + types.SimpleNamespace( + db_session=_fake_db_session, + get_latest_crawl_run_id=lambda _c: 9, + read_crawl=lambda _c, _rid: df, + ), + ) + + args = argparse.Namespace( + property_id="7", + status=False, + csv_stdin=False, + csv_file=str(csv_path), + file_name="links.csv", + ) + with pytest.raises(SystemExit) as exc: + gsc_links_cmd.run({}, args) + assert exc.value.code == 0 + assert captured["pid"] == 7 + assert "https://a.com" in captured["csv"] + assert captured["crawl_urls"] == ["https://c.com/", ""] + out = json.loads(capsys.readouterr().out) + assert out["ok"] is True + + +def test_gsc_links_cmd_csv_stdin(monkeypatch, capsys) -> None: + from website_profiling.commands import gsc_links_cmd + + fake_store = types.SimpleNamespace( + import_gsc_links_csv=lambda *_a, **_k: {"ok": True}, + read_gsc_links_status=None, + ) + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.gsc_links_store", + fake_store, + ) + monkeypatch.setitem( + sys.modules, + "website_profiling.db", + types.SimpleNamespace( + db_session=_fake_db_session, + get_latest_crawl_run_id=lambda _c: None, + read_crawl=None, + ), + ) + monkeypatch.setattr("sys.stdin", types.SimpleNamespace(read=lambda: "a,b\n")) + + args = argparse.Namespace( + property_id="1", + status=False, + csv_stdin=True, + csv_file=None, + file_name="", + ) + with pytest.raises(SystemExit) as exc: + gsc_links_cmd.run({}, args) + assert exc.value.code == 0 + + +def test_gsc_links_cmd_missing_csv_source(monkeypatch, capsys) -> None: + from website_profiling.commands import gsc_links_cmd + + args = argparse.Namespace( + property_id="1", + status=False, + csv_stdin=False, + csv_file=None, + file_name="", + ) + with pytest.raises(SystemExit) as exc: + gsc_links_cmd.run({}, args) + assert exc.value.code == 1 + assert "csv" in capsys.readouterr().err.lower() + + +def test_gsc_links_cmd_crawl_enrichment_exception_swallowed(monkeypatch, capsys) -> None: + from website_profiling.commands import gsc_links_cmd + + session_calls = {"n": 0} + + @contextmanager + def flaky_session(): + session_calls["n"] += 1 + if session_calls["n"] == 1: + raise RuntimeError("db down") + yield object() + + fake_store = types.SimpleNamespace( + import_gsc_links_csv=lambda *_a, **_k: {"ok": True, "crawl_urls": []}, + read_gsc_links_status=None, + ) + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.gsc_links_store", + fake_store, + ) + monkeypatch.setitem( + sys.modules, + "website_profiling.db", + types.SimpleNamespace( + db_session=flaky_session, + get_latest_crawl_run_id=lambda _c: 1, + read_crawl=lambda _c, _rid: pd.DataFrame(), + ), + ) + + args = argparse.Namespace( + property_id="3", + status=False, + csv_stdin=True, + csv_file=None, + file_name="", + ) + monkeypatch.setattr("sys.stdin", types.SimpleNamespace(read=lambda: "x")) + with pytest.raises(SystemExit) as exc: + gsc_links_cmd.run({}, args) + assert exc.value.code == 0 + + +def test_gsc_links_cmd_value_error(monkeypatch, capsys) -> None: + from website_profiling.commands import gsc_links_cmd + + def raise_value(*_a, **_k): + raise ValueError("bad csv") + + fake_store = types.SimpleNamespace( + import_gsc_links_csv=raise_value, + read_gsc_links_status=None, + ) + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.gsc_links_store", + fake_store, + ) + monkeypatch.setitem( + sys.modules, + "website_profiling.db", + types.SimpleNamespace( + db_session=_fake_db_session, + get_latest_crawl_run_id=lambda _c: None, + read_crawl=None, + ), + ) + monkeypatch.setattr("sys.stdin", types.SimpleNamespace(read=lambda: "bad")) + + args = argparse.Namespace( + property_id="1", + status=False, + csv_stdin=True, + csv_file=None, + file_name="", + ) + with pytest.raises(SystemExit) as exc: + gsc_links_cmd.run({}, args) + assert exc.value.code == 1 + out = json.loads(capsys.readouterr().out) + assert out["ok"] is False + assert "bad csv" in out["error"] + + +def test_gsc_links_cmd_generic_exception(monkeypatch, capsys) -> None: + from website_profiling.commands import gsc_links_cmd + + def raise_other(*_a, **_k): + raise RuntimeError("boom") + + fake_store = types.SimpleNamespace( + import_gsc_links_csv=raise_other, + read_gsc_links_status=None, + ) + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.gsc_links_store", + fake_store, + ) + monkeypatch.setitem( + sys.modules, + "website_profiling.db", + types.SimpleNamespace( + db_session=_fake_db_session, + get_latest_crawl_run_id=lambda _c: None, + read_crawl=None, + ), + ) + monkeypatch.setattr("sys.stdin", types.SimpleNamespace(read=lambda: "x")) + + args = argparse.Namespace( + property_id="1", + status=False, + csv_stdin=True, + csv_file=None, + file_name="", + ) + with pytest.raises(SystemExit) as exc: + gsc_links_cmd.run({}, args) + assert exc.value.code == 1 + out = json.loads(capsys.readouterr().out) + assert "boom" in out["error"] + + +def test_page_coach_cmd_missing_url(capsys) -> None: + from website_profiling.commands import page_coach_cmd + + args = argparse.Namespace(url="", refresh=False) + with pytest.raises(SystemExit) as exc: + page_coach_cmd.run({}, "/tmp", args) + assert exc.value.code == 1 + assert "url" in capsys.readouterr().err.lower() + + +def test_page_coach_cmd_success_and_env(monkeypatch, capsys) -> None: + from website_profiling.commands import page_coach_cmd + + captured: dict = {} + + def fake_run(url, cfg, **kwargs): + captured["url"] = url + captured["kwargs"] = kwargs + return {"ok": True, "suggestions": []} + + monkeypatch.setitem( + sys.modules, + "website_profiling.llm.page_coach", + types.SimpleNamespace(run_page_coach=fake_run), + ) + monkeypatch.setenv("WP_PAGE_COACH_CURRENT", "crawl:5") + monkeypatch.setenv("WP_PAGE_COACH_BASELINE", "crawl:2") + + args = argparse.Namespace(url="https://example.com/page", refresh=True) + with pytest.raises(SystemExit) as exc: + page_coach_cmd.run({"start_url": "https://example.com"}, "/tmp", args) + assert exc.value.code == 0 + assert captured["url"] == "https://example.com/page" + assert captured["kwargs"]["current_type"] == "crawl" + assert captured["kwargs"]["current_id"] == 5 + assert captured["kwargs"]["baseline_type"] == "crawl" + assert captured["kwargs"]["baseline_id"] == 2 + out = json.loads(capsys.readouterr().out) + assert out["ok"] is True + + +def test_page_coach_cmd_failure_exit(monkeypatch, capsys) -> None: + from website_profiling.commands import page_coach_cmd + + monkeypatch.setitem( + sys.modules, + "website_profiling.llm.page_coach", + types.SimpleNamespace(run_page_coach=lambda *_a, **_k: {"ok": False}), + ) + monkeypatch.delenv("WP_PAGE_COACH_CURRENT", raising=False) + monkeypatch.delenv("WP_PAGE_COACH_BASELINE", raising=False) + + args = argparse.Namespace(url="https://x.com", refresh=False) + with pytest.raises(SystemExit) as exc: + page_coach_cmd.run({}, "/tmp", args) + assert exc.value.code == 1 + + +def test_page_live_cmd_missing_url(capsys) -> None: + from website_profiling.commands import page_live_cmd + + args = argparse.Namespace(url=" ", no_persist=False) + with pytest.raises(SystemExit) as exc: + page_live_cmd.run({}, "/tmp", args) + assert exc.value.code == 1 + + +def test_page_live_cmd_success(monkeypatch, capsys) -> None: + from website_profiling.commands import page_live_cmd + + captured: dict = {} + + def fake_fetch(url, cfg, *, persist=True, property_id=None): + captured["url"] = url + captured["persist"] = persist + captured["property_id"] = property_id + return {"ok": True, "gsc": {"clicks": 1}} + + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.page_live", + types.SimpleNamespace(fetch_page_live=fake_fetch), + ) + monkeypatch.setattr( + "website_profiling.commands.config_resolve.resolve_property_id_from_cfg", + lambda _cfg: 99, + ) + + args = argparse.Namespace(url="https://example.com/p", no_persist=False) + with pytest.raises(SystemExit) as exc: + page_live_cmd.run({"property_id": "99"}, "/tmp", args) + assert exc.value.code == 0 + assert captured["persist"] is True + assert captured["property_id"] == 99 + + +def test_page_live_cmd_partial_ga4_exit_zero(monkeypatch, capsys) -> None: + from website_profiling.commands import page_live_cmd + + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.page_live", + types.SimpleNamespace( + fetch_page_live=lambda *_a, **_k: {"ok": False, "ga4": {"sessions": 3}} + ), + ) + monkeypatch.setattr( + "website_profiling.commands.config_resolve.resolve_property_id_from_cfg", + lambda _cfg: None, + ) + + args = argparse.Namespace(url="https://x.com", no_persist=True) + with pytest.raises(SystemExit) as exc: + page_live_cmd.run({}, "/tmp", args) + assert exc.value.code == 0 + + +def test_page_live_cmd_failure_exit(monkeypatch) -> None: + from website_profiling.commands import page_live_cmd + + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.page_live", + types.SimpleNamespace(fetch_page_live=lambda *_a, **_k: {"ok": False}), + ) + monkeypatch.setattr( + "website_profiling.commands.config_resolve.resolve_property_id_from_cfg", + lambda _cfg: None, + ) + + args = argparse.Namespace(url="https://x.com", no_persist=False) + with pytest.raises(SystemExit) as exc: + page_live_cmd.run({}, "/tmp", args) + assert exc.value.code == 1 + + +def test_page_live_cmd_exception(monkeypatch, capsys) -> None: + from website_profiling.commands import page_live_cmd + + def boom(*_a, **_k): + raise RuntimeError("api down") + + monkeypatch.setitem( + sys.modules, + "website_profiling.integrations.google.page_live", + types.SimpleNamespace(fetch_page_live=boom), + ) + monkeypatch.setattr( + "website_profiling.commands.config_resolve.resolve_property_id_from_cfg", + lambda _cfg: 1, + ) + + args = argparse.Namespace(url="https://x.com", no_persist=False) + with pytest.raises(SystemExit) as exc: + page_live_cmd.run({}, "/tmp", args) + assert exc.value.code == 1 + out = json.loads(capsys.readouterr().out) + assert "api down" in out["error"] diff --git a/tests/test_common_analysis_commands_db_unit.py b/tests/test_common_analysis_commands_db_unit.py new file mode 100644 index 00000000..9eb589bc --- /dev/null +++ b/tests/test_common_analysis_commands_db_unit.py @@ -0,0 +1,1052 @@ +"""Unit tests for common, analysis, commands, and db store modules.""" +from __future__ import annotations + +import argparse +import json +import math +import types +import warnings +from contextlib import contextmanager +from datetime import datetime, timezone +from unittest.mock import MagicMock + +import pandas as pd +import pytest + +from tests.db_test_fakes import CrawlConn, FakeConn, FakeCursor + + +# --------------------------------------------------------------------------- +# common.py +# --------------------------------------------------------------------------- + + +def test_load_dataframe_file_not_found() -> None: + from website_profiling.common import load_dataframe + + with pytest.raises(FileNotFoundError): + load_dataframe("/no/such/file.csv") + + +def test_load_save_dataframe_json(tmp_path) -> None: + from website_profiling.common import load_dataframe, save_dataframe + + df = pd.DataFrame([{"a": 1, "b": "x"}]) + path = tmp_path / "data.json" + save_dataframe(df, str(path)) + loaded = load_dataframe(str(path)) + assert loaded.shape[0] == 1 + assert "a" in loaded.columns + + +def test_load_edges_json_and_missing_file(tmp_path) -> None: + from website_profiling.common import load_edges, save_edges + + assert load_edges(str(tmp_path / "missing.json")) == [] + + edges = [("https://a.com", "https://b.com")] + jp = tmp_path / "edges.json" + save_edges(edges, str(jp)) + assert load_edges(str(jp)) == edges + + jp.write_text(json.dumps([{"from": "https://x.com", "to": "https://y.com"}]), encoding="utf-8") + assert load_edges(str(jp)) == [("https://x.com", "https://y.com")] + + bad = tmp_path / "bad.json" + bad.write_text("{not json", encoding="utf-8") + assert load_edges(str(bad)) == [] + + +def test_normalize_link_empty_href() -> None: + from website_profiling.common import normalize_link + + assert normalize_link("https://x.com", "") is None + + +def test_parse_seo_og_description_fallback() -> None: + from website_profiling.common import parse_seo + + html = """ + + +

H

+ """ + meta_desc, *_ = parse_seo("https://s.com", html) + assert meta_desc == "OG desc" + + +def test_parse_seo_extended_microdata_and_srcset() -> None: + from website_profiling.common import parse_seo_extended + + html = """ + +
+ + + """ + ext = parse_seo_extended(html, "https://secure.com") + assert ext["has_schema"] is True + assert ext["mixed_content_count"] >= 1 + + +def test_parse_content_text_reading_level() -> None: + from bs4 import BeautifulSoup + + from website_profiling.common import parse_content_text + + words = " ".join(f"word{i}" for i in range(50)) + sentences = ". ".join(f"This is sentence number {i} with enough words here" for i in range(8)) + body = f"{words}. {sentences}." + html = f"

{body}

" + soup = BeautifulSoup(html, "lxml") + out = parse_content_text(soup, raw_html=html) + assert out["word_count"] > 30 + assert out["reading_level"] > 0 + + +def test_is_wappalyzer_regex_warning() -> None: + from website_profiling.common import _is_wappalyzer_regex_warning + + assert _is_wappalyzer_regex_warning("Error compiling regex: unbalanced parenthesis") is True + assert _is_wappalyzer_regex_warning("other warning") is False + + +def test_detect_tech_wappalyzer_paths(monkeypatch) -> None: + from bs4 import BeautifulSoup + + from website_profiling import common + + common._wappalyzer_instance = None + common._wappalyzer_disabled = False + + html = "test" + soup = BeautifulSoup(html, "lxml") + + # Disabled flag falls back immediately + common._wappalyzer_disabled = True + out = common.detect_tech_wappalyzer("https://a.com", html, {}, soup) + assert out.startswith("[") + common._wappalyzer_disabled = False + + # ImportError path + import builtins + + real_import = builtins.__import__ + + def fake_import(name, *args, **kwargs): + if name == "Wappalyzer": + raise ImportError("no wappalyzer") + return real_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", fake_import) + assert common.detect_tech_wappalyzer("https://a.com", html, {}, soup).startswith("[") + + # Success + regex-warning disable path + monkeypatch.undo() + common._wappalyzer_instance = None + common._wappalyzer_disabled = False + + class FakeWebPage: + def __init__(self, url, html, headers): + self.url = url + + class FakeWapp: + @staticmethod + def latest(): + return FakeWapp() + + def analyze(self, _page): + warnings.warn("Error compiling regex: unbalanced parenthesis") + return {"React", "jQuery"} + + wapp_mod = types.ModuleType("Wappalyzer") + wapp_mod.Wappalyzer = FakeWapp + wapp_mod.WebPage = FakeWebPage + monkeypatch.setitem(__import__("sys").modules, "Wappalyzer", wapp_mod) + + out2 = common.detect_tech_wappalyzer("https://a.com", html, {}, soup) + assert common._wappalyzer_disabled is True + assert out2.startswith("[") + + # Exception during analyze + common._wappalyzer_disabled = False + common._wappalyzer_instance = None + + class BoomWapp(FakeWapp): + def analyze(self, _page): + raise RuntimeError("boom") + + wapp_mod.Wappalyzer = BoomWapp + assert common.detect_tech_wappalyzer("https://a.com", html, {}, soup, wappalyzer=BoomWapp()).startswith("[") + + # Cached instance path (no warnings) + common._wappalyzer_disabled = False + common._wappalyzer_instance = None + + class CleanWapp: + def analyze(self, _page): + return {"Vue.js"} + + out3 = common.detect_tech_wappalyzer("https://a.com", html, {}, soup, wappalyzer=CleanWapp()) + assert "Vue.js" in out3 + + +def test_parse_tech_stack_header_match() -> None: + from bs4 import BeautifulSoup + + from website_profiling.common import parse_tech_stack + + soup = BeautifulSoup("", "lxml") + stack = json.loads(parse_tech_stack(soup, {"X-Custom": "contains cf-ray value"}, "https://a.com")) + assert "Cloudflare" in stack + + +def test_parse_links_serialized_branches() -> None: + from website_profiling.common import _is_empty, parse_links_serialized + + assert _is_empty(float("nan")) is True + assert parse_links_serialized(["https://a.com/", "https://b.com"]) == ["https://a.com", "https://b.com"] + assert parse_links_serialized(" ") == [] + assert parse_links_serialized('["a"') == ["[\"a\""] + assert parse_links_serialized("a.com, b.com") == ["a.com", "b.com"] + + +def test_load_robots(monkeypatch) -> None: + from website_profiling.common import load_robots + + class RP: + def set_url(self, _url): + pass + + def read(self): + pass + + monkeypatch.setattr("urllib.robotparser.RobotFileParser", lambda: RP()) + assert load_robots("https://example.com/page") is not None + + class BadRP: + def set_url(self, _url): + pass + + def read(self): + raise OSError("fail") + + monkeypatch.setattr("urllib.robotparser.RobotFileParser", lambda: BadRP()) + assert load_robots("https://example.com") is None + + +# --------------------------------------------------------------------------- +# analysis/page.py +# --------------------------------------------------------------------------- + + +def test_analyze_html_comprehensive_warnings() -> None: + from website_profiling.analysis.page import _input_has_label, _visible_anchor_text, analyze_html + from bs4 import BeautifulSoup + + big_script = "x" * 9000 + html = f""" + + + + + + + + + + + + + + +

One

+

Skipped

+ + dup + + + + + + + + """ + out = analyze_html( + html=html, + page_url="https://site.com/Path/", + base_url="https://site.com/Path/", + canonical_url="", + ) + ids = {w["id"] for w in out["warnings"]} + assert "missing_canonical" in ids + assert "missing_html_lang" in ids + assert "hreflang_multiple_x_default" in ids + assert "trailing_slash_path" in ids + assert "uppercase_path" in ids + assert "skipped_heading_level" in ids + assert "large_inline_script" in ids + assert "render_blocking_script" in ids + assert "stylesheet_blocking_hint" in ids + assert "json_ld_missing_type" in ids + assert "json_ld_parse" in ids + assert "empty_anchor" in ids + assert "form_missing_label" in ids + assert out["preload_count"] >= 1 + assert out["preconnect_count"] >= 1 + + soup = BeautifulSoup('child', "lxml") + a = soup.find("a") + assert _visible_anchor_text(a) == "child" + main_soup = BeautifulSoup(html, "lxml") + assert _input_has_label(main_soup, main_soup.find("input", {"type": "hidden"})) is True + + +def test_analyze_html_empty_returns_early() -> None: + from website_profiling.analysis.page import analyze_html + + out = analyze_html(html="", page_url="", base_url="") + assert out["internal_link_count"] == 0 + + +def test_json_ld_walk_nested_list() -> None: + from website_profiling.analysis.page import _json_ld_missing_type + + assert _json_ld_missing_type([{"name": "X"}]) is True + assert _json_ld_missing_type({"outer": {"name": "nested"}}) is True + + +# --------------------------------------------------------------------------- +# analysis/local.py +# --------------------------------------------------------------------------- + + +def test_local_cfg_helpers() -> None: + from website_profiling.analysis.local import _cfg_bool, _cfg_int + + assert _cfg_bool(None, "x", True) is True + assert _cfg_int(None, "analysis_fuzzy_threshold", 92) == 92 + assert _cfg_int({"analysis_fuzzy_threshold": "bad"}, "analysis_fuzzy_threshold", 92) == 92 + assert _cfg_int({"ml_simhash_hamming": "3"}, "analysis_simhash_hamming", 0) == 3 + + +def test_simhash_and_hamming() -> None: + from website_profiling.analysis.local import _hamming, simhash_64 + + assert simhash_64("") == 0 + h1 = simhash_64("hello world test content here") + h2 = simhash_64("hello world test content here") + assert h1 == h2 + assert _hamming(h1, h2) == 0 + assert _hamming(h1, h1 ^ 1) >= 1 + + +def test_compute_duplicate_groups_hamming_and_fuzzy(monkeypatch) -> None: + from website_profiling.analysis import local + + text = "this is enough textual content for duplicate check and more words" + monkeypatch.setattr(local, "_import_rapidfuzz", lambda: types.SimpleNamespace(token_set_ratio=lambda a, b: 95)) + monkeypatch.setattr(local, "normalize_fingerprint_text", lambda _row: text) + + df = pd.DataFrame( + [ + {"url": "https://a.com/1", "status": "200", "content_type": "text/html"}, + {"url": "https://a.com/2", "status": "200", "content_type": "text/html"}, + {"url": "https://a.com/3", "status": "404", "content_type": "text/html"}, + ] + ) + cfg = { + "enable_duplicate_detection": "true", + "analysis_simhash_hamming": "64", + "analysis_fuzzy_threshold": "90", + "analysis_dup_max_pages": "10", + } + groups, mapping = local.compute_duplicate_groups(df, cfg) + assert len(groups) >= 1 + assert any(k.startswith("dup_") for k in mapping.values()) + + +def test_compute_language_signals_enabled(monkeypatch) -> None: + from website_profiling.analysis import local + + monkeypatch.setattr( + local, + "_import_langdetect", + lambda: (lambda _t: "en", type("E", (Exception,), {})), + ) + monkeypatch.setattr(local, "normalize_fingerprint_text", lambda _row: "x" * 40) + df = pd.DataFrame([{"url": "https://a.com", "status": "200"}]) + by_url, summary = local.compute_language_signals(df, {"enable_language_detection": "true"}) + assert by_url.get("https://a.com") == "en" + assert summary["detected_pages"] == 1 + + +def test_run_local_enrichment_success(monkeypatch) -> None: + from website_profiling.analysis import local + + monkeypatch.setattr(local, "compute_duplicate_groups", lambda *_a, **_k: ([{"id": "dup_0"}], {"https://a.com": "dup_0"})) + monkeypatch.setattr( + local, + "compute_language_signals", + lambda *_a, **_k: ({"https://a.com": "en"}, {"counts": {"en": 1}, "mixed_site": False}), + ) + out = local.run_local_enrichment(pd.DataFrame([{"url": "https://a.com"}]), {"enable_duplicate_detection": "true"}) + assert out["content_duplicates"] + assert out["language_by_url"]["https://a.com"] == "en" + + +def test_merge_bundles_and_payload_edges() -> None: + from website_profiling.analysis.local import merge_analysis_into_payload, merge_bundles + + merged = merge_bundles( + {"url_duplicate_group_id": {"a": "d0"}}, + {"content_duplicates": [{"id": "d0"}], "ner_site_summary": {"ORG": 1}, "ml_errors": ["e2"]}, + ) + assert merged["content_duplicates"] + assert merged["ner_site_summary"]["ORG"] == 1 + + payload = {"links": ["not-a-dict", {"url": "https://z.com", "page_analysis": {}}]} + bundle = { + "ner_site_summary": {}, + "ml_errors": [], + "language_by_url": {"https://z.com": "fr"}, + "spacy_by_url": {"https://z.com": []}, + } + merge_analysis_into_payload(payload, bundle) + assert payload["links"][1]["detected_language"] == "fr" + + +# --------------------------------------------------------------------------- +# config_resolve.py +# --------------------------------------------------------------------------- + + +def test_shadow_config_path_and_require_database_url(monkeypatch, tmp_path) -> None: + from website_profiling.commands import config_resolve + + monkeypatch.setattr("website_profiling.db.storage.get_data_dir", lambda: str(tmp_path)) + assert config_resolve.shadow_config_path().endswith("pipeline-config.txt") + + monkeypatch.setattr("website_profiling.db.storage.get_database_url", lambda: "postgres://localhost/db") + config_resolve.require_database_url() + + +def test_cleanup_lighthouse_work_dir_branches() -> None: + from website_profiling.commands.config_resolve import cleanup_lighthouse_work_dir + + cleanup_lighthouse_work_dir("") + cleanup_lighthouse_work_dir("/etc/passwd") + + +def test_resolve_property_id_no_cfg() -> None: + from website_profiling.commands.config_resolve import resolve_property_id_from_cfg + + assert resolve_property_id_from_cfg(None) is None + + +def test_google_db_has_gsc_non_dict_data(monkeypatch) -> None: + from website_profiling.commands import config_resolve + + conn = FakeConn() + conn.set_next_cursor(FakeCursor(fetchone_value={"data": "not-a-dict"})) + + class Ctx: + def __enter__(self): + return conn + + def __exit__(self, _t, _v, _tb): + return False + + monkeypatch.setattr("website_profiling.db.db_session", lambda: Ctx()) + assert config_resolve.google_db_has_gsc({}) is False + + +def test_should_enrich_keywords_fallback_to_gsc_flag() -> None: + from website_profiling.commands.config_resolve import should_enrich_keywords_after_report + + assert should_enrich_keywords_after_report({"enable_google_search_console": "true"}) is True + + +def test_resolve_config_db_and_shadow_paths(monkeypatch, tmp_path) -> None: + from website_profiling.commands import config_resolve + + # Missing config file + args = argparse.Namespace(config=str(tmp_path / "missing.txt")) + with pytest.raises(SystemExit) as e: + config_resolve.resolve_config(args) + assert e.value.code == 1 + + # DB path with shadow fallback + shadow = tmp_path / "pipeline-config.txt" + shadow.write_text("start_url = https://shadow.com\n", encoding="utf-8") + monkeypatch.setattr(config_resolve, "require_database_url", lambda: None) + monkeypatch.setattr(config_resolve, "load_config_from_db", lambda: {}) + monkeypatch.setattr(config_resolve, "shadow_config_path", lambda: str(shadow)) + monkeypatch.setattr("website_profiling.db.storage.get_data_dir", lambda: str(tmp_path)) + + cfg, cwd = config_resolve.resolve_config(argparse.Namespace(config=None)) + assert cfg["start_url"] == "https://shadow.com" + + # DB path with config loaded + monkeypatch.setattr(config_resolve, "load_config_from_db", lambda: {"start_url": "https://db.com"}) + cfg2, _ = config_resolve.resolve_config(argparse.Namespace(config=None)) + assert cfg2["start_url"] == "https://db.com" + + # No config anywhere + monkeypatch.setattr(config_resolve, "load_config_from_db", lambda: {}) + monkeypatch.setattr(config_resolve, "shadow_config_path", lambda: str(tmp_path / "nope.txt")) + with pytest.raises(SystemExit) as e2: + config_resolve.resolve_config(argparse.Namespace(config=None)) + assert e2.value.code == 1 + + # DB URL missing + def _boom(): + raise RuntimeError("no DATABASE_URL") + + monkeypatch.setattr(config_resolve, "require_database_url", _boom) + with pytest.raises(SystemExit) as e3: + config_resolve.resolve_config(argparse.Namespace(config=None)) + assert e3.value.code == 1 + + +# --------------------------------------------------------------------------- +# commands +# --------------------------------------------------------------------------- + + +def test_google_cmd_branches(monkeypatch) -> None: + from website_profiling.commands import google_cmd + + # list_properties success + monkeypatch.setitem( + __import__("sys").modules, + "website_profiling.integrations.google.fetch", + types.SimpleNamespace(list_properties=lambda **_k: {"sites": []}, fetch_google_data=lambda **_k: {}), + ) + with pytest.raises(SystemExit) as e: + google_cmd.run({}, "/tmp", lambda _k, d: d, argparse.Namespace(list_properties=True, test=False, property_id=None)) + assert e.value.code == 0 + + # list_properties error + monkeypatch.setitem( + __import__("sys").modules, + "website_profiling.integrations.google.fetch", + types.SimpleNamespace(list_properties=lambda **_k: (_ for _ in ()).throw(RuntimeError("x")), fetch_google_data=lambda **_k: {}), + ) + with pytest.raises(SystemExit) as e2: + google_cmd.run({}, "/tmp", lambda _k, d: d, argparse.Namespace(list_properties=True, test=False, property_id=None)) + assert e2.value.code == 1 + + # property_id from args + assert google_cmd._resolved_property_id({}, argparse.Namespace(property_id=5)) == 5 + + +def test_google_cmd_fetch_with_crawl_and_errors(monkeypatch) -> None: + from website_profiling.commands import google_cmd + import sys as _sys + + g = types.ModuleType("google") + ga = types.ModuleType("google.auth") + ge = types.ModuleType("google.auth.exceptions") + ge.RefreshError = RuntimeError + ga.exceptions = ge + g.auth = ga + monkeypatch.setitem(_sys.modules, "google", g) + monkeypatch.setitem(_sys.modules, "google.auth", ga) + monkeypatch.setitem(_sys.modules, "google.auth.exceptions", ge) + + class Ctx: + def __enter__(self): + return object() + + def __exit__(self, _t, _v, _tb): + return False + + captured: dict = {} + + def fake_fetch(**kwargs): + captured.update(kwargs) + return {"errors": ["partial"]} + + monkeypatch.setitem( + _sys.modules, + "website_profiling.integrations.google.fetch", + types.SimpleNamespace(fetch_google_data=fake_fetch, list_properties=lambda *_a, **_k: {}), + ) + monkeypatch.setitem( + _sys.modules, + "website_profiling.integrations.google.store", + types.SimpleNamespace(write_google_data=lambda *_a, **_k: None), + ) + import website_profiling.db as db + + monkeypatch.setattr(db, "db_session", lambda: Ctx()) + monkeypatch.setattr(db, "get_latest_crawl_run_id", lambda _c: 1) + monkeypatch.setattr(db, "read_crawl", lambda _c, _rid: pd.DataFrame([{"url": "https://a.com"}])) + monkeypatch.setattr(google_cmd, "resolve_property_id_from_cfg", lambda _cfg: None) + + with pytest.raises(SystemExit) as e: + google_cmd.run( + {"start_url": "https://a.com"}, + "/tmp", + lambda _k, d: d, + argparse.Namespace(list_properties=False, test=False, property_id=9), + ) + assert e.value.code == 0 + assert captured.get("property_id") == 9 + assert captured.get("crawl_urls") == ["https://a.com"] + + # RefreshError + monkeypatch.setitem( + _sys.modules, + "website_profiling.integrations.google.fetch", + types.SimpleNamespace( + fetch_google_data=lambda **_k: (_ for _ in ()).throw(ge.RefreshError("expired")), + list_properties=lambda *_a, **_k: {}, + ), + ) + with pytest.raises(SystemExit) as e2: + google_cmd.run( + {"start_url": "https://a.com"}, + "/tmp", + lambda _k, d: d, + argparse.Namespace(list_properties=False, test=False, property_id=None), + ) + assert e2.value.code == 1 + + +def test_pipeline_cmd_remaining_branches(monkeypatch) -> None: + from website_profiling.commands import pipeline_cmd + + assert pipeline_cmd._normalize_render_mode({"crawl_render_mode": "bogus"}) == "static" + assert pipeline_cmd.select_lighthouse_urls_from_crawl(pd.DataFrame(), 5) == [] + assert pipeline_cmd.select_lighthouse_urls_from_crawl(pd.DataFrame([{"url": "x"}]), 5) == [] + + # js_extra_wait_ms None branch in _run_crawl + monkeypatch.setattr(pipeline_cmd, "require_start_url", lambda *_a, **_k: "https://a.com") + monkeypatch.setitem( + __import__("sys").modules, + "website_profiling.crawl.crawler", + types.SimpleNamespace(run_crawler=lambda **_k: None), + ) + pipeline_cmd._run_crawl({"crawl_js_extra_wait_ms": "", "crawl_render_mode": "auto"}, True) + + # lighthouse on pages skip + class Ctx: + def __enter__(self): + return object() + + def __exit__(self, _t, _v, _tb): + return False + + monkeypatch.setattr("website_profiling.db.db_session", lambda: Ctx()) + monkeypatch.setattr("website_profiling.db.get_latest_crawl_run_id", lambda _c: 1) + monkeypatch.setattr("website_profiling.db.read_crawl", lambda _c, _rid: pd.DataFrame()) + monkeypatch.setattr(pipeline_cmd, "lighthouse_work_dir", lambda: "/tmp/lh") + monkeypatch.setattr(pipeline_cmd, "cleanup_lighthouse_work_dir", lambda _p: None) + pipeline_cmd._run_lighthouse_on_pages({"lighthouse_strategy": "bogus"}, 5) + + # report keyword enrich path + monkeypatch.setattr(pipeline_cmd, "require_start_url", lambda *_a, **_k: "https://a.com") + monkeypatch.setattr(pipeline_cmd, "should_enrich_keywords_after_report", lambda _cfg: True) + monkeypatch.setattr(pipeline_cmd, "google_db_has_gsc", lambda _cfg=None: True) + monkeypatch.setitem( + __import__("sys").modules, + "website_profiling.reporting.builder", + types.SimpleNamespace(run_simple_report=lambda **_k: "out.json"), + ) + monkeypatch.setitem( + __import__("sys").modules, + "website_profiling.integrations.google.keyword_enrich", + types.SimpleNamespace(run_enrichment=lambda _cfg: None), + ) + pipeline_cmd._run_report({}, True) + + # plot invalid render mode warning + captured: dict = {} + + def fake_plot(**kwargs): + captured.update(kwargs) + + monkeypatch.setitem(__import__("sys").modules, "website_profiling.tools.plot", types.SimpleNamespace(run_plot=fake_plot)) + pipeline_cmd._run_plot({"crawl_render_mode": "invalid-mode"}, True) + assert captured.get("render_mode") is None + + +def test_keywords_cmd_enrich_warning(monkeypatch) -> None: + from website_profiling.commands import keywords_cmd + + monkeypatch.setattr(keywords_cmd, "require_start_url", lambda *_a, **_k: "https://a.com") + monkeypatch.setattr(keywords_cmd, "google_db_has_gsc", lambda _cfg=None: True) + monkeypatch.setitem( + __import__("sys").modules, + "website_profiling.tools.keywords", + types.SimpleNamespace(main=lambda **_k: 0), + ) + monkeypatch.setitem( + __import__("sys").modules, + "website_profiling.integrations.google.keyword_enrich", + types.SimpleNamespace(run_enrichment=lambda _cfg: (_ for _ in ()).throw(RuntimeError("kw fail"))), + ) + with pytest.raises(SystemExit) as e: + keywords_cmd.run({"enable_google_suggest": "false"}, argparse.Namespace(expand_only=False, enrich_google=False)) + assert e.value.code == 0 + + +def test_lighthouse_cmd_invalid_strategy(monkeypatch) -> None: + from website_profiling.commands import lighthouse_cmd + + monkeypatch.setattr(lighthouse_cmd, "require_lighthouse_url", lambda _cfg: "https://a.com") + monkeypatch.setattr(lighthouse_cmd, "lighthouse_work_dir", lambda: "/tmp/lh") + monkeypatch.setattr(lighthouse_cmd, "cleanup_lighthouse_work_dir", lambda _p: None) + captured: dict = {} + + def fake_main(**kwargs): + captured.update(kwargs) + return 0 + + monkeypatch.setitem(__import__("sys").modules, "website_profiling.lighthouse.runner", types.SimpleNamespace(main=fake_main)) + with pytest.raises(SystemExit): + lighthouse_cmd.run({"lighthouse_url": "https://a.com", "lighthouse_strategy": "tablet"}, argparse.Namespace()) + assert captured.get("strategy") == "mobile" + + +def test_warnings_cmd_relative_input(monkeypatch, tmp_path) -> None: + from website_profiling.commands import warnings_cmd + + captured: dict = {} + + def fake_main(**kwargs): + captured.update(kwargs) + return 0 + + monkeypatch.setitem(__import__("sys").modules, "website_profiling.tools.warnings", types.SimpleNamespace(main=fake_main)) + rel = "input.json" + (tmp_path / rel).write_text("{}", encoding="utf-8") + with pytest.raises(SystemExit): + warnings_cmd.run( + {"warning_mapper_input": rel, "warning_mapper_input_type": "lighthouse"}, + str(tmp_path), + lambda _k, d: d, + argparse.Namespace(), + ) + assert str(captured.get("input_path", "")).endswith("input.json") + + +# --------------------------------------------------------------------------- +# db/_common.py and stores +# --------------------------------------------------------------------------- + + +def test_db_common_json_val_executemany_and_sanitize() -> None: + from website_profiling.db import _common + + assert _common._json_val({"a": 1}) is not None + assert _common._parse_json_field(42) == 42 + assert _common._row_field(("only",), "data", index=5) is None + + class BadItem: + def item(self): + raise ValueError("nope") + + assert _common._sanitize_for_json(BadItem()) is None + + class Dt: + def isoformat(self): + return "2020-01-01" + + assert _common._sanitize_for_json(Dt()) == "2020-01-01" + assert _common._sanitize_for_json(object()) is not None + + class CursorConn(FakeConn): + def __init__(self) -> None: + super().__init__() + self.cursors: list[FakeCursor] = [] + + @contextmanager + def cursor(self): + cur = FakeCursor() + self.cursors.append(cur) + yield cur + + conn = CursorConn() + _common._executemany(conn, "INSERT INTO t VALUES (%s)", [], page_size=10) + _common._executemany(conn, "INSERT INTO t VALUES (%s)", [(1,), (2,)], page_size=1) + assert sum(len(c.executemany_calls) for c in conn.cursors) >= 2 + + +def test_config_store_read_write_pipeline(monkeypatch) -> None: + from website_profiling.db.config_store import read_pipeline_config, write_pipeline_config + + conn = FakeConn() + conn.set_next_cursor( + FakeCursor( + fetchall_value=[ + {"key": "start_url", "value": "https://a.com", "is_unknown": False}, + {"key": "legacy", "value": "v", "is_unknown": True}, + ] + ) + ) + known, unknown = read_pipeline_config(conn) # type: ignore[arg-type] + assert known["start_url"] == "https://a.com" + assert unknown[0]["key"] == "legacy" + + wconn = FakeConn() + write_pipeline_config(wconn, {"k": "v"}, unknown_keys=[{"key": "u", "value": "1"}]) # type: ignore[arg-type] + assert any("INSERT INTO pipeline_config" in sql for sql, _ in wconn.executed) + + +def test_crawl_store_branches(monkeypatch) -> None: + from website_profiling.db import crawl_store as cs + + # create_crawl_run fallback without render_mode + conn = CrawlConn(fetchone={"id": 3}, boom_execute=True) + conn.boom_execute = False + + class BoomFirst(CrawlConn): + def execute(self, sql, params=None): + self.executed.append((sql, params)) + if "render_mode" in sql: + raise RuntimeError("no column") + return super().execute(sql, params) + + conn2 = BoomFirst(fetchone={"id": 4}) + assert cs.create_crawl_run(conn2, start_url="https://a.com", render_mode="js") == 4 # type: ignore[arg-type] + + assert cs.get_latest_crawl_run_id(CrawlConn(boom_execute=True)) is None # type: ignore[arg-type] + + info_conn = CrawlConn(fetchone={"created_at": "t", "start_url": "u", "render_mode": "static"}) + assert cs.get_crawl_run_info(info_conn, 1)["render_mode"] == "static" # type: ignore[arg-type] + + # fallback query without render_mode + class RenderBoom(CrawlConn): + def execute(self, sql, params=None): + self.executed.append((sql, params)) + if "render_mode" in sql: + raise RuntimeError("no render_mode") + if "FROM crawl_runs WHERE" in sql: + return FakeCursor(fetchone_value={"created_at": "t", "start_url": "u"}) + return super().execute(sql, params) + + assert cs.get_crawl_run_info(RenderBoom(), 1)["start_url"] == "u" # type: ignore[arg-type] + + row = pd.Series({"url": "https://a.com", "status": float("nan"), "n": 1}) + out = cs._df_row_to_crawl_json(row) + assert out["status"] is None + + assert cs._extract_hostname("not-a-url") == "" + + # write_crawl empty with no run id + empty_conn = CrawlConn() + cs.write_crawl(empty_conn, pd.DataFrame(), crawl_run_id=None) # type: ignore[arg-type] + + # write_crawl creates run when missing + wconn = CrawlConn(fetchone={"id": 9}) + monkeypatch.setattr(cs, "get_latest_crawl_run_id", lambda _c: None) + df = pd.DataFrame([{"url": "https://a.com/", "status": 200}]) + cs.write_crawl(wconn, df, crawl_run_id=None) # type: ignore[arg-type] + + # legacy insert fallback + def boom_executemany(conn, sql, params, **kwargs): + if "fetch_method" in sql: + raise RuntimeError("legacy") + from website_profiling.db._common import _executemany as real + + return real(conn, sql, params, page_size=kwargs.get("page_size", 500)) + + monkeypatch.setattr(cs, "_executemany", boom_executemany) + from website_profiling.db._common import _json_val + + cs._write_crawl_rows(wconn, [(1, "u", "200", "t", "static", _json_val({}))]) # type: ignore[arg-type] + + # read_crawl fallback without fetch_method + rconn = CrawlConn(fetchall=[{"url": "u", "data": {"viewport_present": "true"}}]) + + class FailFirst(CrawlConn): + def execute(self, sql, params=None): + if "fetch_method" in sql: + raise RuntimeError("no fm") + return super().execute(sql, params) + + monkeypatch.setattr(cs, "get_latest_crawl_run_id", lambda _c: 1) + df_read = cs.read_crawl(FailFirst(fetchall=[{"url": "u", "data": {}}]), run_id=1) # type: ignore[arg-type] + assert "fetch_method" in df_read.columns + + # write_edges no run id, no latest + econn = CrawlConn() + monkeypatch.setattr(cs, "get_latest_crawl_run_id", lambda _c: None) + cs.write_edges(econn, [("a", "b")], crawl_run_id=None) # type: ignore[arg-type] + + # write_nodes empty / missing columns / no run + cs.write_nodes(CrawlConn(), pd.DataFrame(), crawl_run_id=None) # type: ignore[arg-type] + cs.write_nodes(CrawlConn(), pd.DataFrame([{"x": 1}]), crawl_run_id=None) # type: ignore[arg-type] + nconn = CrawlConn() + cs.write_nodes(nconn, pd.DataFrame([{"index": "https://a.com", "count": 2}]), crawl_run_id=None) # type: ignore[arg-type] + + assert cs.read_edges(CrawlConn(boom_execute=True), run_id=1) == [] # type: ignore[arg-type] + assert cs.read_nodes(CrawlConn(boom_execute=True), run_id=1).empty # type: ignore[arg-type] + + +def test_historical_backup_success_and_restore_fallback(monkeypatch, tmp_path) -> None: + from website_profiling.db import historical as h + + monkeypatch.setattr(h, "get_data_dir", lambda: str(tmp_path)) + monkeypatch.setattr(h, "get_database_url", lambda: "postgres://u:p@h/db") + + dump_path = tmp_path / "backups" / "out.dump" + + def fake_run(cmd, **kwargs): + dump_path.parent.mkdir(parents=True, exist_ok=True) + dump_path.write_bytes(b"dump") + return types.SimpleNamespace(returncode=0) + + monkeypatch.setattr(h.subprocess, "run", fake_run) + result = h.backup_db_if_exists(skip_in_ci=False) + assert result is not None + + # read_historical_data table exception + class BadConn: + def cursor(self): + raise RuntimeError("cursor fail") + + class Ctx: + def __enter__(self): + return BadConn() + + def __exit__(self, _t, _v, _tb): + return False + + monkeypatch.setattr(h, "db_session", lambda: Ctx()) + data = h.read_historical_data() + assert data["report_payload"] == [] + + # restore _bulk row-by-row fallback + conn = FakeConn() + + def boom_bulk(_conn, _sql, _params, **kwargs): + raise RuntimeError("bulk fail") + + monkeypatch.setattr(h, "_executemany", boom_bulk) + h.restore_historical_data( + conn, # type: ignore[arg-type] + { + "report_payload": [{"id": 1, "generated_at": "x", "site_name": "s", "canonical_domain": "d", "data": {}}], + "gsc_links_data": [{"id": 1, "fetched_at": "x", "property_id": 1, "data": {}}], + }, + ) + assert conn.commits == 1 + + +def test_lighthouse_store_branches(monkeypatch) -> None: + from website_profiling.db import lighthouse_store as ls + + class BoomConn(FakeConn): + def execute(self, sql, params=None): + raise RuntimeError("boom") + + assert ls.read_lighthouse_summary(FakeConn()) is None # type: ignore[arg-type] + assert ls.read_lh_runs_by_url(BoomConn()) == {} # type: ignore[arg-type] + + conn = FakeConn() + conn.set_next_cursor(FakeCursor(fetchone_value={"data": {"score": 1}})) + assert ls.read_lighthouse_run_json(conn, 1) == {"score": 1} # type: ignore[arg-type] + + conn2 = FakeConn() + conn2.set_next_cursor(FakeCursor(fetchone_value={"data": [1, 2]})) + assert ls.read_latest_lighthouse_run_json(conn2) is None # type: ignore[arg-type] + + # write_lh_audits_from_run empty + ls.write_lh_audits_from_run(FakeConn(), 1, {}) # type: ignore[arg-type] + + monkeypatch.setitem( + __import__("sys").modules, + "website_profiling.lighthouse.schema", + types.SimpleNamespace( + lhr_to_audit_rows=lambda _d: ( + [ + { + "audit_id": "a", + "category_id": "c", + "score": 1, + "score_display_mode": "numeric", + "title": "t", + "description": "d", + "display_value": "v", + "numeric_value": 1, + "help_text": "h", + "details_type": "table", + "details_headings": "[]", + "details_meta": "{}", + } + ], + [], + ) + ), + ) + audit_conn = FakeConn() + audit_conn.set_next_cursor(FakeCursor(fetchall_value=[{"id": 99}])) + ls.write_lh_audits_from_run(audit_conn, 1, {"audits": {}}) # type: ignore[arg-type] + + # read_lh_audits_with_items + item_conn = FakeConn() + item_conn.set_next_cursor( + FakeCursor( + fetchall_value=[ + { + "id": 1, + "audit_id": "a", + "category_id": "c", + "title": "t", + "description": "d", + "score": 1, + "score_display_mode": "numeric", + "display_value": "v", + "numeric_value": 1, + "help_text": "h", + "details_type": "table", + "details_headings": [], + "details_meta": {}, + } + ] + ) + ) + audits = ls.read_lh_audits_with_items(item_conn, 1) # type: ignore[arg-type] + assert audits[0]["id"] == "a" + + assert ls.read_lighthouse_page_summaries(BoomConn()) == {} # type: ignore[arg-type] + + +def test_llm_cache_and_report_store_branches() -> None: + from website_profiling.db.llm_cache_store import read_llm_cache, read_llm_cache_batch + from website_profiling.db.report_store import read_report_payload + + class BoomConn(FakeConn): + def execute(self, sql, params=None): + raise RuntimeError("boom") + + assert read_llm_cache(BoomConn(), "k") is None # type: ignore[arg-type] + assert read_llm_cache_batch(FakeConn(), []) == {} # type: ignore[arg-type] + + conn = FakeConn() + conn.set_next_cursor( + FakeCursor( + fetchall_value=[ + {"cache_key": "k1", "response_json": {"a": 1}}, + {"cache_key": "k2", "response_json": "not-json"}, + ] + ) + ) + out = read_llm_cache_batch(conn, ["k1", "k2"]) # type: ignore[arg-type] + assert out["k1"]["a"] == 1 + assert "k2" not in out + + rconn = FakeConn() + rconn.set_next_cursor(FakeCursor(fetchone_value={"data": {"site": 1}})) + assert read_report_payload(rconn, report_id=5)["site"] == 1 # type: ignore[arg-type] + + +def test_google_app_store_build_sa_credentials() -> None: + from website_profiling.db.google_app_store import build_service_account_credentials + + with pytest.raises(RuntimeError, match="No service account"): + build_service_account_credentials({"service_account_json": None}) diff --git a/tests/test_config_schema_keys.py b/tests/test_config_schema_keys.py index eb9fc46d..90cf7434 100644 --- a/tests/test_config_schema_keys.py +++ b/tests/test_config_schema_keys.py @@ -21,6 +21,16 @@ "preserve_crawl_history", "crawl_stream_to_db", "crawl_exclude_urls", + "crawl_render_mode", + "crawl_js_concurrency", + "crawl_js_timeout", + "crawl_js_wait_until", + "crawl_js_extra_wait_ms", + "crawl_js_block_resources", + "crawl_js_capture_console", + "crawl_js_console_levels", + "crawl_js_capture_failed_requests", + "crawl_js_console_max_per_page", "outbound_domain_max_rows", "include_keyword_opportunities", "site_name", diff --git a/tests/test_coverage_edge_win.py b/tests/test_coverage_edge_win.py deleted file mode 100644 index e9035652..00000000 --- a/tests/test_coverage_edge_win.py +++ /dev/null @@ -1,26 +0,0 @@ -from __future__ import annotations - -import pytest - - -def test_pool_get_database_url_raises_when_missing(monkeypatch): - from website_profiling.db.pool import get_database_url - - monkeypatch.delenv("DATABASE_URL", raising=False) - with pytest.raises(RuntimeError): - get_database_url() - - -def test_report_store_fallback_links_and_exception_path(monkeypatch): - from website_profiling.db import report_store - - monkeypatch.setattr(report_store, "get_crawl_run_info", lambda _c, _rid: None) - domain = report_store._canonical_domain_from_report(object(), {"links": [{"url": "https://fallback.com/p"}]}) # type: ignore[arg-type] - assert domain == "fallback.com" - - class BoomConn: - def execute(self, *_a, **_k): - raise RuntimeError("x") - - assert report_store.read_report_payload(BoomConn()) is None # type: ignore[arg-type] - diff --git a/tests/test_crawl_db_writer_imports.py b/tests/test_crawl_db_writer_imports.py index ebf8014f..c5b845a8 100644 --- a/tests/test_crawl_db_writer_imports.py +++ b/tests/test_crawl_db_writer_imports.py @@ -1,6 +1,62 @@ import pytest +def test_crawl_db_writer_enqueue_and_batch_flush(monkeypatch: pytest.MonkeyPatch) -> None: + from website_profiling.crawl.crawler import _CrawlDbWriter + + written: list[tuple[int, int, bool]] = [] + + class _FakeConn: + pass + + class _FakeCtx: + def __enter__(self): + return _FakeConn() + + def __exit__(self, _t, _v, _tb): + return False + + monkeypatch.setattr("website_profiling.db.db_session", lambda: _FakeCtx()) + monkeypatch.setattr( + "website_profiling.db.crawl_store._crawl_rows_from_df", + lambda df, run_id: [{"url": row["url"], "run_id": run_id} for row in df.to_dict("records")], + ) + monkeypatch.setattr( + "website_profiling.db.crawl_store.write_crawl_batch", + lambda _conn, rows, run_id, commit=True: written.append((len(rows), run_id, commit)), + ) + + writer = _CrawlDbWriter(crawl_run_id=5, batch_size=50) + for i in range(51): + writer.enqueue({"url": f"https://a.com/{i}"}) + writer.finish() + writer.run() + writer.raise_if_failed() + + assert written == [(50, 5, True), (1, 5, True)] + + +def test_crawl_db_writer_records_run_errors(monkeypatch: pytest.MonkeyPatch) -> None: + from website_profiling.crawl.crawler import _CrawlDbWriter + + class _BrokenCtx: + def __enter__(self): + raise RuntimeError("db unavailable") + + def __exit__(self, _t, _v, _tb): + return False + + monkeypatch.setattr("website_profiling.db.db_session", lambda: _BrokenCtx()) + + writer = _CrawlDbWriter(crawl_run_id=1, batch_size=50) + writer.enqueue({"url": "https://a.com"}) + writer.finish() + writer.run() + + with pytest.raises(RuntimeError, match="db unavailable"): + writer.raise_if_failed() + + def test_crawl_db_writer_run_does_not_import_error() -> None: """ Regression: during the db/ split, _CrawlDbWriter.run() imported helpers from diff --git a/tests/test_crawl_fetchers.py b/tests/test_crawl_fetchers.py new file mode 100644 index 00000000..15837e85 --- /dev/null +++ b/tests/test_crawl_fetchers.py @@ -0,0 +1,527 @@ +from __future__ import annotations + +from pathlib import Path + +import pytest + +from website_profiling.crawl.fetchers.base import FetchResult +from website_profiling.crawl.fetchers.browser_deps import ensure_browser_deps +from website_profiling.crawl.fetchers.factory import browser_status, build_fetcher, validate_browser_available +from website_profiling.crawl.fetchers.spa_heuristics import needs_js_render, needs_js_render_after_parse +from website_profiling.crawl.fetchers.static import StaticFetcher +from website_profiling.crawl.sitemap import discover_sitemap_urls, _parse_sitemap_xml + + +FIXTURES = Path(__file__).resolve().parent / "fixtures" + + +def _static_result(html: str, *, fetch_method: str = "static") -> FetchResult: + return FetchResult( + status=200, + content_type="text/html", + text=html, + response_time_ms=1, + content_length=len(html), + final_url="https://example.com/", + headers_dict={}, + redirect_chain_length=0, + fetch_method=fetch_method, # type: ignore[arg-type] + ) + + +def test_static_fetcher_parses_html(): + fetcher = StaticFetcher(timeout=5) + try: + result = fetcher.fetch("https://example.com") + finally: + fetcher.close() + assert isinstance(result, FetchResult) + assert result.fetch_method == "static" + assert result.browser_diagnostics is None + + +def test_page_diagnostics_collector_builds_summary(): + from website_profiling.crawl.fetchers.browser import _PageDiagnosticsCollector + + collector = _PageDiagnosticsCollector( + capture_console=True, + console_levels=frozenset({"error", "warning"}), + capture_failed_requests=False, + max_per_page=20, + ) + + class FakeLoc: + url = "https://example.com/app.js" + lineNumber = 12 + + class FakeMsg: + type = "error" + text = "Something broke" + location = FakeLoc() + + class FakeErr: + def __str__(self): + return "Uncaught TypeError" + + stack = "Error: Uncaught TypeError\n at main.js:1:1" + + collector.console.append( + { + "level": "error", + "text": FakeMsg.text, + "source_url": FakeLoc.url, + "line": FakeLoc.lineNumber, + } + ) + collector.page_errors.append( + {"message": str(FakeErr()), "stack": FakeErr.stack} + ) + diag = collector.build() + assert diag["summary"]["console_error_count"] == 1 + assert diag["summary"]["page_error_count"] == 1 + assert len(diag["console"]) == 1 + assert len(diag["page_errors"]) == 1 + + +def test_finalize_browser_diagnostics_empty(): + from website_profiling.crawl.fetchers.browser_diagnostics import finalize_browser_diagnostics + + diag = finalize_browser_diagnostics([], [], []) + assert diag["summary"]["console_error_count"] == 0 + assert diag["summary"]["page_error_count"] == 0 + + +def test_build_fetcher_static_mode(): + fetcher = build_fetcher(render_mode="static", timeout=5) + try: + assert isinstance(fetcher, StaticFetcher) + finally: + fetcher.close() + + +def test_validate_browser_available_raises_without_playwright(monkeypatch): + import builtins + + monkeypatch.setenv("WP_SKIP_BROWSER_AUTO_INSTALL", "1") + real_import = builtins.__import__ + + def mock_import(name, globals=None, locals=None, fromlist=(), level=0): + if name == "playwright": + raise ImportError("no playwright") + return real_import(name, globals, locals, fromlist, level) + + monkeypatch.setattr(builtins, "__import__", mock_import) + with pytest.raises(RuntimeError, match="JavaScript crawl requires"): + validate_browser_available() + + +def test_needs_js_render_detects_next_shell(): + html = '
' + assert needs_js_render(_static_result(html)) is True + + +@pytest.mark.parametrize( + "fixture_name", + ["angular_shell.html", "svelte_shell.html", "vue_shell.html"], +) +def test_needs_js_render_detects_framework_shell(fixture_name): + html = (FIXTURES / fixture_name).read_text(encoding="utf-8") + assert needs_js_render(_static_result(html)) is True + + +def test_needs_js_render_after_parse_low_links_with_scripts(): + html = (FIXTURES / "post_parse_shell.html").read_text(encoding="utf-8") + result = _static_result(html) + assert needs_js_render(result) is False + assert needs_js_render_after_parse( + result, link_count=0, same_domain_link_count=0 + ) is True + + +def test_needs_js_render_after_parse_skips_rendered(): + html = (FIXTURES / "post_parse_shell.html").read_text(encoding="utf-8") + result = _static_result(html, fetch_method="rendered") + assert needs_js_render_after_parse( + result, link_count=0, same_domain_link_count=0 + ) is False + + +def test_needs_js_render_after_parse_normal_page(): + links = "".join( + f'p{i}' for i in range(12) + ) + html = f"

Blog

" + result = _static_result(html) + assert needs_js_render_after_parse( + result, link_count=12, same_domain_link_count=12 + ) is False + + +def test_hybrid_refetch_rendered_uses_browser(monkeypatch): + from website_profiling.crawl.fetchers.hybrid import HybridFetcher + + rendered = _static_result("rendered", fetch_method="rendered") + + class FakeBrowser: + def fetch(self, _url): + return rendered + + def close(self): + pass + + class FakeStatic: + def fetch(self, _url): + return _static_result("static") + + def close(self): + pass + + hybrid = HybridFetcher(FakeStatic(), lambda: FakeBrowser()) + try: + out = hybrid.refetch_rendered("https://example.com/") + assert out.fetch_method == "rendered" + assert out.text == "rendered" + finally: + hybrid.close() + + +def test_browser_status_reports_missing_playwright(monkeypatch): + import builtins + + real_import = builtins.__import__ + + def mock_import(name, globals=None, locals=None, fromlist=(), level=0): + if name == "playwright": + raise ImportError("no playwright") + return real_import(name, globals, locals, fromlist, level) + + monkeypatch.setattr(builtins, "__import__", mock_import) + status = browser_status() + assert status["ok"] is False + assert "JavaScript crawl requires" in str(status.get("message", "")) + + +@pytest.mark.parametrize( + "env_chrome_path,isfile_return,which_return", + [ + ("/usr/bin/chromium", True, None), + ("", False, "/usr/bin/chromium"), + ], +) +def test_browser_status_ok_when_chromium_available( + monkeypatch, env_chrome_path, isfile_return, which_return +): + import sys + import types + + monkeypatch.setitem(sys.modules, "playwright", types.ModuleType("playwright")) + monkeypatch.setenv("CHROME_PATH", env_chrome_path) + monkeypatch.setattr( + "website_profiling.crawl.fetchers.browser_deps.os.path.isfile", + lambda path: isfile_return and path == env_chrome_path, + ) + monkeypatch.setattr( + "website_profiling.crawl.fetchers.browser_deps.shutil.which", + lambda _name: which_return, + ) + status = browser_status() + assert status["ok"] is True + + +def test_ensure_browser_deps_skips_install_when_disabled(monkeypatch): + monkeypatch.setenv("WP_SKIP_BROWSER_AUTO_INSTALL", "1") + monkeypatch.setattr( + "website_profiling.crawl.fetchers.browser_deps.browser_status", + lambda: {"ok": False, "message": "missing"}, + ) + monkeypatch.setattr( + "website_profiling.crawl.fetchers.browser_deps._pip_install_browser_requirements", + lambda: (_ for _ in ()).throw(AssertionError("should not pip install")), + ) + status = ensure_browser_deps() + assert status["ok"] is False + + +def test_repo_root_uses_website_profiling_root_env(monkeypatch, tmp_path): + from website_profiling.crawl.fetchers import browser_deps + + monkeypatch.setenv("WEBSITE_PROFILING_ROOT", str(tmp_path)) + assert browser_deps._repo_root() == tmp_path + + +def test_repo_root_defaults_to_project_root(monkeypatch): + from website_profiling.crawl.fetchers import browser_deps + + monkeypatch.delenv("WEBSITE_PROFILING_ROOT", raising=False) + root = browser_deps._repo_root() + assert (root / "requirements-browser.txt").is_file() + + +def test_playwright_chromium_unavailable_without_playwright(monkeypatch): + from website_profiling.crawl.fetchers import browser_deps + + monkeypatch.setattr(browser_deps, "_playwright_importable", lambda: False) + assert browser_deps._playwright_chromium_available() is False + + +def test_chromium_available_via_playwright_executable(monkeypatch): + import sys + import types + + from website_profiling.crawl.fetchers import browser_deps + + class FakeChromium: + executable_path = "/tmp/fake-chromium" + + class FakePlaywright: + chromium = FakeChromium() + + class FakeContext: + def __enter__(self): + return FakePlaywright() + + def __exit__(self, *_args): + return False + + sync_api = types.ModuleType("playwright.sync_api") + sync_api.sync_playwright = lambda: FakeContext() + playwright_mod = types.ModuleType("playwright") + playwright_mod.sync_api = sync_api + monkeypatch.setitem(sys.modules, "playwright", playwright_mod) + monkeypatch.setitem(sys.modules, "playwright.sync_api", sync_api) + + monkeypatch.setattr(browser_deps, "_system_chromium_available", lambda: False) + monkeypatch.setattr(browser_deps, "_playwright_importable", lambda: True) + monkeypatch.setattr( + browser_deps.os.path, + "isfile", + lambda path: path == "/tmp/fake-chromium", + ) + assert browser_deps.chromium_available() is True + + +def test_playwright_chromium_available_returns_false_on_error(monkeypatch): + from website_profiling.crawl.fetchers import browser_deps + + monkeypatch.setattr(browser_deps, "_playwright_importable", lambda: True) + + def boom(): + raise RuntimeError("playwright broken") + + import sys + import types + + sync_api = types.ModuleType("playwright.sync_api") + sync_api.sync_playwright = boom + monkeypatch.setitem(sys.modules, "playwright", types.ModuleType("playwright")) + monkeypatch.setitem(sys.modules, "playwright.sync_api", sync_api) + assert browser_deps._playwright_chromium_available() is False + + +def test_browser_status_missing_chromium_with_playwright(monkeypatch): + import sys + import types + + monkeypatch.setitem(sys.modules, "playwright", types.ModuleType("playwright")) + monkeypatch.setattr( + "website_profiling.crawl.fetchers.browser_deps.chromium_available", + lambda: False, + ) + status = browser_status() + assert status["ok"] is False + assert "JavaScript crawl requires" in str(status.get("message", "")) + + +def test_ensure_browser_deps_returns_ok_without_install(monkeypatch): + monkeypatch.setattr( + "website_profiling.crawl.fetchers.browser_deps.browser_status", + lambda: {"ok": True}, + ) + monkeypatch.setattr( + "website_profiling.crawl.fetchers.browser_deps._pip_install_browser_requirements", + lambda: (_ for _ in ()).throw(AssertionError("should not pip install")), + ) + status = ensure_browser_deps() + assert status["ok"] is True + + +def test_pip_install_browser_requirements_missing_file(monkeypatch, tmp_path): + from website_profiling.crawl.fetchers import browser_deps + + monkeypatch.setenv("WEBSITE_PROFILING_ROOT", str(tmp_path)) + with pytest.raises(RuntimeError, match="Missing requirements-browser.txt"): + browser_deps._pip_install_browser_requirements() + + +def test_ensure_browser_deps_reports_auto_install_failure(monkeypatch): + import subprocess + + monkeypatch.setattr( + "website_profiling.crawl.fetchers.browser_deps.browser_status", + lambda: {"ok": False, "message": "missing"}, + ) + monkeypatch.setattr( + "website_profiling.crawl.fetchers.browser_deps._playwright_importable", + lambda: False, + ) + monkeypatch.setattr( + "website_profiling.crawl.fetchers.browser_deps._pip_install_browser_requirements", + lambda: (_ for _ in ()).throw(subprocess.CalledProcessError(1, "pip")), + ) + status = ensure_browser_deps(install=True) + assert status["ok"] is False + assert "Auto-install failed" in str(status.get("message", "")) + + +def test_build_fetcher_unknown_mode_falls_back_to_static(): + fetcher = build_fetcher(render_mode="unknown-mode", timeout=5) # type: ignore[arg-type] + try: + assert isinstance(fetcher, StaticFetcher) + finally: + fetcher.close() + + +def test_ensure_browser_deps_installs_when_missing(monkeypatch): + pip_called: list[str] = [] + pw_called: list[str] = [] + statuses = [{"ok": False, "message": "missing"}, {"ok": True}] + + monkeypatch.setattr( + "website_profiling.crawl.fetchers.browser_deps.browser_status", + lambda: statuses.pop(0) if statuses else {"ok": True}, + ) + monkeypatch.setattr( + "website_profiling.crawl.fetchers.browser_deps._playwright_importable", + lambda: bool(pip_called), + ) + monkeypatch.setattr( + "website_profiling.crawl.fetchers.browser_deps.chromium_available", + lambda: bool(pw_called), + ) + monkeypatch.setattr( + "website_profiling.crawl.fetchers.browser_deps._pip_install_browser_requirements", + lambda: pip_called.append("pip"), + ) + monkeypatch.setattr( + "website_profiling.crawl.fetchers.browser_deps._playwright_install_chromium", + lambda: pw_called.append("playwright"), + ) + status = ensure_browser_deps() + assert status["ok"] is True + assert pip_called == ["pip"] + assert pw_called == ["playwright"] + + +def test_parse_sitemap_xml_urlset(): + xml = """ + + https://example.com/a + https://example.com/b + """ + pages, nested = _parse_sitemap_xml(xml, "https://example.com/sitemap.xml") + assert nested == [] + assert "https://example.com/a" in pages + assert "https://example.com/b" in pages + + +def test_discover_sitemap_urls_from_local_robots(monkeypatch): + class FakeResp: + def __init__(self, status_code, text): + self.status_code = status_code + self.text = text + + class FakeSession: + headers = {} + + def get(self, url, timeout=0): + if url.endswith("/robots.txt"): + return FakeResp(200, "Sitemap: https://example.com/sitemap.xml\n") + if url.endswith("/sitemap.xml"): + return FakeResp( + 200, + """ + https://example.com/page-1""", + ) + return FakeResp(404, "") + + def close(self): + pass + + monkeypatch.setattr( + "website_profiling.crawl.sitemap.requests.Session", + lambda: FakeSession(), + ) + urls = discover_sitemap_urls("https://example.com") + assert "https://example.com/page-1" in urls + + +@pytest.mark.browser +def test_browser_fetcher_discovers_js_link(spa_server): + validate_browser_available() + fetcher = build_fetcher( + render_mode="javascript", + js_timeout=15, + js_extra_wait_ms=500, + js_concurrency=1, + ) + try: + static = StaticFetcher(timeout=5) + try: + static_result = static.fetch(spa_server) + finally: + static.close() + assert static_result.text is not None + assert 'href="/discovered-by-js"' not in static_result.text + + rendered = fetcher.fetch(spa_server) + assert rendered.status == 200 + assert rendered.text is not None + assert 'href="/discovered-by-js"' in rendered.text + assert rendered.fetch_method == "rendered" + finally: + fetcher.close() + + +@pytest.mark.browser +def test_browser_fetcher_captures_console_errors(spa_server): + validate_browser_available() + base = spa_server.rsplit("/", 1)[0] + url = f"{base}/console_error.html" + fetcher = build_fetcher( + render_mode="javascript", + js_timeout=15, + js_extra_wait_ms=800, + js_concurrency=1, + capture_console=True, + js_console_levels="error,warning", + ) + try: + rendered = fetcher.fetch(url) + assert rendered.status == 200 + assert rendered.browser_diagnostics is not None + summary = rendered.browser_diagnostics.get("summary") or {} + assert summary.get("console_error_count", 0) >= 1 + assert summary.get("page_error_count", 0) >= 1 + console = rendered.browser_diagnostics.get("console") or [] + assert any("fixture console error" in str(c.get("text", "")).lower() for c in console) + finally: + fetcher.close() + + +@pytest.mark.browser +def test_auto_fetcher_falls_back_for_spa_shell(spa_server): + validate_browser_available() + fetcher = build_fetcher( + render_mode="auto", + js_timeout=15, + js_extra_wait_ms=500, + js_concurrency=1, + ) + try: + result = fetcher.fetch(spa_server) + assert result.status == 200 + assert result.text is not None + assert "/discovered-by-js" in result.text + assert result.fetch_method == "rendered" + finally: + fetcher.close() diff --git a/tests/test_final_push80.py b/tests/test_crawl_lighthouse_google_cmd_unit.py similarity index 95% rename from tests/test_final_push80.py rename to tests/test_crawl_lighthouse_google_cmd_unit.py index afd03e3a..ef770ad4 100644 --- a/tests/test_final_push80.py +++ b/tests/test_crawl_lighthouse_google_cmd_unit.py @@ -1,3 +1,4 @@ +"""Unit tests for crawl_store, lighthouse_store, and google_cmd branches.""" from __future__ import annotations import argparse @@ -22,6 +23,16 @@ def execute(self, sql, params=None): return C(row={"id": 7}) if "ORDER BY id DESC LIMIT 1" in sql: return C(row={"id": 2}) + if "SELECT url, fetch_method, data FROM crawl_results" in sql: + return C( + rows=[ + { + "url": "u", + "fetch_method": "static", + "data": {"viewport_present": True, "noindex": False, "has_schema": True}, + } + ] + ) if "SELECT url, data FROM crawl_results" in sql: return C(rows=[{"url": "u", "data": {"viewport_present": True, "noindex": False, "has_schema": True}}]) if "SELECT from_url, to_url FROM edges" in sql: diff --git a/tests/test_crawler_browser_e2e.py b/tests/test_crawler_browser_e2e.py new file mode 100644 index 00000000..d46ec9e9 --- /dev/null +++ b/tests/test_crawler_browser_e2e.py @@ -0,0 +1,47 @@ +from __future__ import annotations + +import json + +import pytest + +from website_profiling.crawl.crawler import run_crawler +from website_profiling.crawl.fetchers.factory import validate_browser_available + + +def _urls_from_df(df) -> set[str]: + urls: set[str] = set() + if "url" in df.columns: + urls.update(str(u) for u in df["url"].dropna()) + if "outlink_targets" in df.columns: + for raw in df["outlink_targets"].fillna(""): + try: + for link in json.loads(raw or "[]"): + urls.add(str(link)) + except (json.JSONDecodeError, TypeError): + continue + return urls + + +@pytest.mark.browser +def test_run_crawler_auto_discovers_js_links(spa_server): + validate_browser_available() + base = spa_server.rsplit("/", 1)[0] + start_url = f"{base}/post_parse_shell.html" + df = run_crawler( + start_url=start_url, + render_mode="auto", + max_pages=10, + ignore_robots=True, + show_progress=False, + output_csv=None, + output_db=False, + js_timeout=15, + js_extra_wait_ms=500, + js_concurrency=1, + concurrency=2, + ) + assert not df.empty + all_urls = _urls_from_df(df) + assert any("discovered-by-js" in u for u in all_urls) + if "fetch_method" in df.columns: + assert (df["fetch_method"] == "rendered").any() diff --git a/tests/test_crawler_deep.py b/tests/test_crawler_deep.py index 381c355f..fd096d43 100644 --- a/tests/test_crawler_deep.py +++ b/tests/test_crawler_deep.py @@ -8,7 +8,12 @@ def test_worker_success_path_populates_many_fields(monkeypatch): import website_profiling.crawl.crawler as mod + from website_profiling.crawl.fetchers.base import FetchResult + monkeypatch.setattr( + "website_profiling.crawl.sitemap.discover_sitemap_urls", + lambda *_a, **_k: [], + ) c = mod.Crawler( start_url="https://site.com", ignore_robots=True, @@ -17,15 +22,16 @@ def test_worker_success_path_populates_many_fields(monkeypatch): store_outlinks=True, max_depth=2, ) - c.fetch = lambda _url: ( # type: ignore[method-assign] - 200, - "text/html", - "ok", - 12, - 100, - "https://site.com/page", - {"X-Robots-Tag": ""}, - 0, + c.fetch = lambda _url: FetchResult( # type: ignore[method-assign] + status=200, + content_type="text/html", + text="ok", + response_time_ms=12, + content_length=100, + final_url="https://site.com/page", + headers_dict={"X-Robots-Tag": ""}, + redirect_chain_length=0, + fetch_method="static", ) monkeypatch.setattr(mod, "parse_links", lambda _u, _t: ("T", {"https://site.com/a", "https://ext.com/x"})) monkeypatch.setattr(mod, "parse_seo", lambda *_a, **_k: ("desc", 4, "h1", 1, "https://site.com/canon")) @@ -83,9 +89,93 @@ def test_worker_success_path_populates_many_fields(monkeypatch): assert "https://site.com/a" in json.loads(out["outlink_targets"]) +def test_crawl_keeps_pending_futures_between_iterations(monkeypatch): + import time + import website_profiling.crawl.crawler as mod + + monkeypatch.setattr( + "website_profiling.crawl.sitemap.discover_sitemap_urls", + lambda *_a, **_k: [], + ) + c = mod.Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + concurrency=2, + max_pages=2, + ) + c.queue.put("https://site.com/slow") + + def _slow_worker(url): + if url.endswith("/slow"): + time.sleep(0.05) + return { + "url": url, + "status": 200, + "content_type": "text/html", + "title": "ok", + "outlinks": 0, + "fetch_method": "static", + "response_time_ms": "", + "content_length": 0, + "final_url": url, + "meta_description": "", + "meta_description_len": 0, + "h1": "", + "h1_count": 0, + "canonical_url": "", + "viewport_present": False, + "viewport_content": "", + "noindex": False, + "has_schema": False, + "heading_sequence": "", + "images_without_alt": 0, + "images_total": 0, + "img_without_lazy": 0, + "img_without_dimensions": 0, + "aria_count": 0, + "mixed_content_count": 0, + "redirect_chain_length": 0, + "cache_control": "", + "etag": "", + "x_robots_tag": "", + "strict_transport_security": "", + "x_content_type_options": "", + "x_frame_options": "", + "content_security_policy": "", + "script_count": 0, + "link_stylesheet_count": 0, + "total_js_bytes": 0, + "total_css_bytes": 0, + "word_count": 0, + "reading_level": 0.0, + "content_html_ratio": 0.0, + "top_keywords": "[]", + "content_excerpt": "", + "og_title": "", + "og_description": "", + "og_image": "", + "og_type": "", + "twitter_card": "", + "twitter_title": "", + "twitter_image": "", + "tech_stack": "[]", + "depth": None, + "page_analysis": "{}", + } + + monkeypatch.setattr(c, "worker", _slow_worker) + df = c.crawl(show_progress=False) + assert len(df) == 2 + + def test_crawl_runs_and_handles_done_futures(monkeypatch): import website_profiling.crawl.crawler as mod + monkeypatch.setattr( + "website_profiling.crawl.sitemap.discover_sitemap_urls", + lambda *_a, **_k: [], + ) c = mod.Crawler(start_url="https://site.com", ignore_robots=True, use_wappalyzer=False, concurrency=1, max_pages=1) monkeypatch.setattr( c, @@ -114,6 +204,299 @@ def crawl(self, **_kwargs): assert out_file.exists() +def test_crawl_empty_results_builds_full_column_schema(monkeypatch): + import website_profiling.crawl.crawler as mod + + monkeypatch.setattr( + "website_profiling.crawl.sitemap.discover_sitemap_urls", + lambda *_a, **_k: [], + ) + c = mod.Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + exclude_urls=["https://site.com"], + store_outlinks=True, + ) + df = c.crawl(show_progress=False) + assert df.empty + assert "fetch_method" in df.columns + assert "outlink_targets" in df.columns + assert "crawl_time_s" in df.columns + + +def test_crawl_skips_excluded_and_visited_urls(monkeypatch): + import website_profiling.crawl.crawler as mod + + monkeypatch.setattr( + "website_profiling.crawl.sitemap.discover_sitemap_urls", + lambda *_a, **_k: [], + ) + c = mod.Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + concurrency=1, + max_pages=2, + exclude_urls=["https://site.com/skip"], + ) + c.queue.put("https://site.com/skip") + c.queue.put("https://site.com") + c.visited.add("https://site.com") + c.queue.put("https://site.com/second") + + calls: list[str] = [] + + def _worker(url): + calls.append(url) + return { + "url": url, + "status": 200, + "content_type": "text/html", + "title": "ok", + "outlinks": 0, + "fetch_method": "static", + "response_time_ms": "", + "content_length": 0, + "final_url": url, + "meta_description": "", + "meta_description_len": 0, + "h1": "", + "h1_count": 0, + "canonical_url": "", + "viewport_present": False, + "viewport_content": "", + "noindex": False, + "has_schema": False, + "heading_sequence": "", + "images_without_alt": 0, + "images_total": 0, + "img_without_lazy": 0, + "img_without_dimensions": 0, + "aria_count": 0, + "mixed_content_count": 0, + "redirect_chain_length": 0, + "cache_control": "", + "etag": "", + "x_robots_tag": "", + "strict_transport_security": "", + "x_content_type_options": "", + "x_frame_options": "", + "content_security_policy": "", + "script_count": 0, + "link_stylesheet_count": 0, + "total_js_bytes": 0, + "total_css_bytes": 0, + "word_count": 0, + "reading_level": 0.0, + "content_html_ratio": 0.0, + "top_keywords": "[]", + "content_excerpt": "", + "og_title": "", + "og_description": "", + "og_image": "", + "og_type": "", + "twitter_card": "", + "twitter_title": "", + "twitter_image": "", + "tech_stack": "[]", + "depth": None, + "page_analysis": "{}", + } + + monkeypatch.setattr(c, "worker", _worker) + df = c.crawl(show_progress=False) + assert calls == ["https://site.com/second"] + assert len(df) == 1 + + +def test_crawl_handles_worker_exception(monkeypatch): + import website_profiling.crawl.crawler as mod + + monkeypatch.setattr( + "website_profiling.crawl.sitemap.discover_sitemap_urls", + lambda *_a, **_k: [], + ) + c = mod.Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + concurrency=1, + max_pages=1, + store_outlinks=True, + ) + + def _boom(_url): + raise RuntimeError("worker exploded") + + monkeypatch.setattr(c, "worker", _boom) + df = c.crawl(show_progress=False) + assert len(df) == 1 + assert df.iloc[0]["status"] == "error" + assert df.iloc[0]["url"] is None + assert "outlink_targets" in df.columns + + +def test_crawl_streams_rows_to_db_writer(monkeypatch): + import website_profiling.crawl.crawler as mod + + monkeypatch.setattr( + "website_profiling.crawl.sitemap.discover_sitemap_urls", + lambda *_a, **_k: [], + ) + + class FakeDbWriter: + instances: list["FakeDbWriter"] = [] + + def __init__(self, crawl_run_id: int, batch_size: int) -> None: + self.crawl_run_id = crawl_run_id + self.batch_size = batch_size + self.enqueued: list[dict] = [] + self.started = False + self.finished = False + FakeDbWriter.instances.append(self) + + def start(self) -> None: + self.started = True + + def enqueue(self, record: dict) -> None: + self.enqueued.append(record) + + def finish(self) -> None: + self.finished = True + + def join(self) -> None: + return None + + def raise_if_failed(self) -> None: + return None + + monkeypatch.setattr(mod, "_CrawlDbWriter", FakeDbWriter) + c = mod.Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + concurrency=1, + max_pages=1, + ) + monkeypatch.setattr( + c, + "worker", + lambda url: {"url": url, "status": 200, "content_type": "", "title": "", "outlinks": 0}, + ) + c.crawl(show_progress=False, stream_crawl_run_id=42, stream_batch_size=100) + writer = FakeDbWriter.instances[-1] + assert writer.started is True + assert writer.finished is True + assert writer.enqueued and writer.enqueued[0]["url"] == "https://site.com" + + +def test_run_crawler_writes_json(monkeypatch, tmp_path): + import website_profiling.crawl.crawler as mod + + class FakeCrawler: + def __init__(self, **_kwargs): + pass + + def crawl(self, **_kwargs): + return pd.DataFrame([{"url": "https://a.com", "status": 200}]) + + monkeypatch.setattr(mod, "Crawler", FakeCrawler) + out_file = tmp_path / "out.json" + mod.run_crawler("https://a.com", output_db=False, output_csv=str(out_file), show_progress=False) + assert out_file.exists() + + +def test_run_crawler_non_streaming_db_write(monkeypatch): + import website_profiling.crawl.crawler as mod + + class FakeCrawler: + def __init__(self, **_kwargs): + pass + + def crawl(self, **_kwargs): + return pd.DataFrame([{"url": "https://a.com", "status": 200, "title": "ok"}]) + + class _Ctx: + def __enter__(self): + return object() + + def __exit__(self, _t, _v, _tb): + return False + + writes: list[tuple] = [] + + fake_db = types.SimpleNamespace( + backup_db_if_exists=lambda: "/tmp/backup.sql", + create_crawl_run=lambda _c, _u, property_id=None, render_mode=None: 7, + db_session=lambda: _Ctx(), + read_historical_data=lambda: {"report_payload": [{"id": 1}]}, + restore_historical_data=lambda *_a, **_k: None, + write_crawl=lambda conn, df, crawl_run_id=None: writes.append((conn, len(df), crawl_run_id)), + ) + fake_storage = types.SimpleNamespace(ensure_crawl_tables_cleared=lambda *_a, **_k: None) + monkeypatch.setattr(mod, "Crawler", FakeCrawler) + monkeypatch.setitem(__import__("sys").modules, "website_profiling.db", fake_db) + monkeypatch.setitem(__import__("sys").modules, "website_profiling.db.storage", fake_storage) + + df = mod.run_crawler( + "https://a.com", + output_db=True, + crawl_stream_to_db=False, + max_pages=5, + preserve_crawl_history=False, + show_progress=False, + ) + assert not df.empty + assert writes and writes[0][2] == 7 + + +def test_run_crawler_streaming_db_with_history_backup(monkeypatch): + import website_profiling.crawl.crawler as mod + + class FakeCrawler: + def __init__(self, **_kwargs): + pass + + def crawl(self, **_kwargs): + return pd.DataFrame([{"url": "https://a.com", "status": 200}]) + + class _Ctx: + def __enter__(self): + return object() + + def __exit__(self, _t, _v, _tb): + return False + + cleared: list[bool] = [] + restored: list[bool] = [] + + fake_db = types.SimpleNamespace( + backup_db_if_exists=lambda: "/tmp/backup.sql", + create_crawl_run=lambda _c, _u, property_id=None, render_mode=None: 11, + db_session=lambda: _Ctx(), + read_historical_data=lambda: {"report_payload": [{"id": 1}]}, + restore_historical_data=lambda *_a, **_k: restored.append(True), + ) + fake_storage = types.SimpleNamespace( + ensure_crawl_tables_cleared=lambda *_a, **_k: cleared.append(True), + ) + monkeypatch.setattr(mod, "Crawler", FakeCrawler) + monkeypatch.setitem(__import__("sys").modules, "website_profiling.db", fake_db) + monkeypatch.setitem(__import__("sys").modules, "website_profiling.db.storage", fake_storage) + + df = mod.run_crawler( + "https://a.com", + output_db=True, + crawl_stream_to_db=True, + preserve_crawl_history=False, + show_progress=False, + ) + assert not df.empty + assert cleared + assert restored + + def test_run_crawler_streaming_db_path(monkeypatch): import website_profiling.crawl.crawler as mod @@ -134,7 +517,7 @@ def __exit__(self, _t, _v, _tb): monkeypatch.setattr(mod, "Crawler", FakeCrawler) fake_db = types.SimpleNamespace( backup_db_if_exists=lambda: None, - create_crawl_run=lambda _c, _u, property_id=None: 10, + create_crawl_run=lambda _c, _u, property_id=None, render_mode=None: 10, db_session=lambda: _Ctx(), read_historical_data=lambda: {}, restore_historical_data=lambda *_a, **_k: None, diff --git a/tests/test_crawler_unit.py b/tests/test_crawler_unit.py index 2b350338..4d02f995 100644 --- a/tests/test_crawler_unit.py +++ b/tests/test_crawler_unit.py @@ -1,3 +1,16 @@ +import pytest + + +@pytest.fixture(autouse=True) +def _mock_sitemap_unless_seeding_test(monkeypatch, request): + if request.node.name == "test_crawler_seeds_sitemap_urls": + return + monkeypatch.setattr( + "website_profiling.crawl.crawler.discover_sitemap_urls", + lambda *_a, **_k: [], + ) + + def test_url_matches_exclude_prefix_and_exact() -> None: from website_profiling.crawl.crawler import _url_matches_exclude @@ -5,6 +18,80 @@ def test_url_matches_exclude_prefix_and_exact() -> None: assert _url_matches_exclude("https://a.com/x/", ["https://a.com/x"]) is True assert _url_matches_exclude("https://a.com/x/y", ["https://a.com/x/"]) is True assert _url_matches_exclude("https://a.com/other", ["https://a.com/x"]) is False + assert _url_matches_exclude("https://a.com/x", [""]) is False + + +def test_crawler_allowed_by_robots_swallows_rp_errors() -> None: + from website_profiling.crawl.crawler import Crawler + + class _BrokenRp: + def can_fetch(self, _ua: str, _url: str) -> bool: + raise ValueError("robots parser failed") + + c = Crawler( + start_url="https://site.com", + ignore_robots=False, + use_wappalyzer=False, + ) + c.rp = _BrokenRp() + assert c.allowed_by_robots("https://site.com/page") is True + + +def test_crawler_sitemap_seed_exception_is_ignored(monkeypatch) -> None: + from website_profiling.crawl.crawler import Crawler + + def _boom(*_a, **_k): + raise RuntimeError("sitemap unavailable") + + monkeypatch.setattr("website_profiling.crawl.crawler.discover_sitemap_urls", _boom) + c = Crawler(start_url="https://site.com", ignore_robots=True, use_wappalyzer=False) + assert c.queue.qsize() == 1 + + +def test_crawler_sitemap_seed_filters_exclude_external_and_duplicates(monkeypatch) -> None: + from website_profiling.crawl.crawler import Crawler + + monkeypatch.setattr( + "website_profiling.crawl.crawler.discover_sitemap_urls", + lambda *_a, **_k: [ + "https://site.com/skip-me", + "https://external.com/page", + "https://site.com", + "https://site.com/new-page", + ], + ) + c = Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + exclude_urls=["https://site.com/skip-me"], + ) + queued = [] + while not c.queue.empty(): + queued.append(c.queue.get()) + assert "https://site.com/new-page" in queued + assert "https://site.com/skip-me" not in queued + assert "https://external.com/page" not in queued + assert queued.count("https://site.com") == 1 + + +def test_crawler_seeds_sitemap_urls(monkeypatch) -> None: + from website_profiling.crawl.crawler import Crawler + + monkeypatch.setattr( + "website_profiling.crawl.crawler.discover_sitemap_urls", + lambda *_a, **_k: ["https://site.com/from-sitemap"], + ) + c = Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + ) + queued = [] + while not c.queue.empty(): + queued.append(c.queue.get()) + assert "https://site.com/from-sitemap" in queued + assert c.depths.get("https://site.com/from-sitemap") == 0 def test_crawler_init_respects_exclude_and_same_domain() -> None: @@ -45,15 +132,477 @@ def test_worker_blocked_by_robots_returns_stub_fields() -> None: assert "outlink_targets" in out -def test_worker_fetch_error_returns_error_status() -> None: +def test_worker_fetch_error_returns_error_status(monkeypatch) -> None: + from website_profiling.crawl.crawler import Crawler + from website_profiling.crawl.fetchers.base import FetchResult + + monkeypatch.setattr( + "website_profiling.crawl.sitemap.discover_sitemap_urls", + lambda *_a, **_k: [], + ) + c = Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + ) + c.fetch = lambda _url: FetchResult( # type: ignore[method-assign] + status=None, + content_type=None, + text=None, + response_time_ms=None, + content_length=None, + final_url=None, + headers_dict={}, + redirect_chain_length=0, + fetch_method="static", + ) + out = c.worker("https://site.com/a") + assert out["status"] == "error" + + +def test_worker_merges_browser_diagnostics_into_page_analysis(monkeypatch) -> None: + import json + + from website_profiling.crawl.crawler import Crawler + from website_profiling.crawl.fetchers.base import FetchResult + + diagnostics = { + "console": [{"level": "error", "text": "boom"}], + "page_errors": [], + "failed_requests": [], + "summary": {"console_error_count": 1, "console_warning_count": 0, "page_error_count": 0}, + } + html = "Tlink" + monkeypatch.setattr( + "website_profiling.crawl.sitemap.discover_sitemap_urls", + lambda *_a, **_k: [], + ) + c = Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + ) + c.fetch = lambda _url: FetchResult( # type: ignore[method-assign] + status=200, + content_type="text/html", + text=html, + response_time_ms=10, + content_length=len(html), + final_url="https://site.com/a", + headers_dict={}, + redirect_chain_length=0, + fetch_method="rendered", + browser_diagnostics=diagnostics, + ) + out = c.worker("https://site.com/a") + pa = json.loads(out["page_analysis"]) + assert pa["browser"]["summary"]["console_error_count"] == 1 + assert pa.get("internal_link_count") is not None + + +def test_worker_auto_post_parse_refetches_when_few_links(monkeypatch) -> None: + from pathlib import Path + + from website_profiling.crawl.crawler import Crawler + from website_profiling.crawl.fetchers.base import FetchResult + + fixtures = Path(__file__).resolve().parent / "fixtures" + html = (fixtures / "post_parse_shell.html").read_text(encoding="utf-8") + rendered_html = ( + html.replace("", 'js') + ) + static_result = FetchResult( + status=200, + content_type="text/html", + text=html, + response_time_ms=5, + content_length=len(html), + final_url="https://site.com/", + headers_dict={}, + redirect_chain_length=0, + fetch_method="static", + ) + rendered_result = FetchResult( + status=200, + content_type="text/html; charset=utf-8", + text=rendered_html, + response_time_ms=50, + content_length=len(rendered_html), + final_url="https://site.com/rendered", + headers_dict={"Cache-Control": "rendered"}, + redirect_chain_length=1, + fetch_method="rendered", + ) + + class DummyFetcher: + def fetch(self, _url): + return static_result + + def close(self): + pass + + class FakeHybrid: + def refetch_rendered(self, _url): + return rendered_result + + monkeypatch.setattr( + "website_profiling.crawl.crawler.build_fetcher", + lambda **_kwargs: DummyFetcher(), + ) + c = Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + render_mode="auto", + store_outlinks=True, + ) + c._hybrid_fetcher = FakeHybrid() + out = c.worker("https://site.com/") + assert out["fetch_method"] == "rendered" + assert out["outlinks"] >= 1 + assert "discovered-by-js" in str(out.get("outlink_targets", "")) + assert out["response_time_ms"] == 50 + assert out["cache_control"] == "rendered" + assert out["final_url"] == "https://site.com/rendered" + assert out["redirect_chain_length"] == 1 + + +def test_worker_sets_noindex_from_x_robots_tag(monkeypatch) -> None: + from website_profiling.crawl.crawler import Crawler + from website_profiling.crawl.fetchers.base import FetchResult + + html = "Tok" + c = Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + ) + c.fetch = lambda _url: FetchResult( # type: ignore[method-assign] + status=200, + content_type="text/html", + text=html, + response_time_ms=10, + content_length=len(html), + final_url="https://site.com/a", + headers_dict={"X-Robots-Tag": "noindex"}, + redirect_chain_length=0, + fetch_method="static", + ) + out = c.worker("https://site.com/a") + assert out["noindex"] is True + + +def test_worker_uses_wappalyzer_when_enabled(monkeypatch) -> None: + from website_profiling.crawl.crawler import Crawler + from website_profiling.crawl.fetchers.base import FetchResult + + html = "Tok" + monkeypatch.setattr( + "website_profiling.crawl.crawler.detect_tech_wappalyzer", + lambda *_a, **_k: '["React"]', + ) + c = Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=True, + ) + c.fetch = lambda _url: FetchResult( # type: ignore[method-assign] + status=200, + content_type="text/html", + text=html, + response_time_ms=10, + content_length=len(html), + final_url="https://site.com/a", + headers_dict={}, + redirect_chain_length=0, + fetch_method="static", + ) + out = c.worker("https://site.com/a") + assert out["tech_stack"] == '["React"]' + + +def test_worker_auto_refetch_skips_when_already_rendered(monkeypatch) -> None: + from website_profiling.crawl.crawler import Crawler + from website_profiling.crawl.fetchers.base import FetchResult + + html = "Tok" + + class DummyFetcher: + def fetch(self, _url): + return FetchResult( + status=200, + content_type="text/html", + text=html, + response_time_ms=5, + content_length=len(html), + final_url="https://site.com/", + headers_dict={}, + redirect_chain_length=0, + fetch_method="rendered", + ) + + def close(self): + pass + + class FakeHybrid: + def refetch_rendered(self, _url): + raise AssertionError("refetch should not run") + + monkeypatch.setattr( + "website_profiling.crawl.crawler.build_fetcher", + lambda **_kwargs: DummyFetcher(), + ) + c = Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + render_mode="auto", + ) + c._hybrid_fetcher = FakeHybrid() + out = c.worker("https://site.com/") + assert out["fetch_method"] == "rendered" + + +def test_worker_auto_refetch_skips_when_not_needed(monkeypatch) -> None: + from website_profiling.crawl.crawler import Crawler + from website_profiling.crawl.fetchers.base import FetchResult + + html = ( + "T" + 'one' + 'two' + "" + ) + static_result = FetchResult( + status=200, + content_type="text/html", + text=html, + response_time_ms=5, + content_length=len(html), + final_url="https://site.com/", + headers_dict={}, + redirect_chain_length=0, + fetch_method="static", + ) + + class DummyFetcher: + def fetch(self, _url): + return static_result + + def close(self): + pass + + class FakeHybrid: + def refetch_rendered(self, _url): + raise AssertionError("refetch should not run") + + monkeypatch.setattr( + "website_profiling.crawl.crawler.build_fetcher", + lambda **_kwargs: DummyFetcher(), + ) + c = Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + render_mode="auto", + ) + c._hybrid_fetcher = FakeHybrid() + out = c.worker("https://site.com/") + assert out["fetch_method"] == "static" + + +def test_worker_auto_refetch_keeps_static_when_render_fails(monkeypatch) -> None: + from pathlib import Path + + from website_profiling.crawl.crawler import Crawler + from website_profiling.crawl.fetchers.base import FetchResult + + fixtures = Path(__file__).resolve().parent / "fixtures" + html = (fixtures / "post_parse_shell.html").read_text(encoding="utf-8") + static_result = FetchResult( + status=200, + content_type="text/html", + text=html, + response_time_ms=5, + content_length=len(html), + final_url="https://site.com/", + headers_dict={}, + redirect_chain_length=0, + fetch_method="static", + ) + + class DummyFetcher: + def fetch(self, _url): + return static_result + + def close(self): + pass + + class FakeHybrid: + def refetch_rendered(self, _url): + return FetchResult( + status=500, + content_type="text/html", + text=None, + response_time_ms=50, + content_length=0, + final_url="https://site.com/", + headers_dict={}, + redirect_chain_length=0, + fetch_method="rendered", + ) + + monkeypatch.setattr( + "website_profiling.crawl.crawler.build_fetcher", + lambda **_kwargs: DummyFetcher(), + ) + c = Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + render_mode="auto", + ) + c._hybrid_fetcher = FakeHybrid() + out = c.worker("https://site.com/") + assert out["fetch_method"] == "static" + + +def test_worker_error_includes_outlinks_when_enabled(monkeypatch) -> None: from website_profiling.crawl.crawler import Crawler + from website_profiling.crawl.fetchers.base import FetchResult c = Crawler( start_url="https://site.com", ignore_robots=True, use_wappalyzer=False, + store_outlinks=True, + ) + c.fetch = lambda _url: FetchResult( # type: ignore[method-assign] + status=None, + content_type=None, + text=None, + response_time_ms=None, + content_length=None, + final_url="https://site.com/a", + headers_dict={}, + redirect_chain_length=0, + fetch_method="static", + ) + out = c.worker("https://site.com/a") + assert out["status"] == "error" + assert out["outlink_targets"] == "[]" + + +def test_worker_respects_exclude_and_max_depth_for_links(monkeypatch) -> None: + from website_profiling.crawl.crawler import Crawler + from website_profiling.crawl.fetchers.base import FetchResult + + html = ( + "" + 'child' + 'skip' + "" + ) + c = Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + exclude_urls=["https://site.com/skip"], + max_depth=0, + store_outlinks=True, + ) + c.fetch = lambda _url: FetchResult( # type: ignore[method-assign] + status=200, + content_type="text/html", + text=html, + response_time_ms=10, + content_length=len(html), + final_url="https://site.com/", + headers_dict={}, + redirect_chain_length=0, + fetch_method="static", + ) + out = c.worker("https://site.com/") + assert "https://site.com/child" not in c.depths + assert out.get("outlink_targets") == "[]" + + +def test_worker_applies_polite_delay(monkeypatch) -> None: + from website_profiling.crawl.crawler import Crawler + from website_profiling.crawl.fetchers.base import FetchResult + + sleeps: list[float] = [] + monkeypatch.setattr("website_profiling.crawl.crawler.time.sleep", lambda s: sleeps.append(s)) + html = "ok" + c = Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + polite_delay=0.05, + ) + c.fetch = lambda _url: FetchResult( # type: ignore[method-assign] + status=200, + content_type="text/html", + text=html, + response_time_ms=10, + content_length=len(html), + final_url="https://site.com/", + headers_dict={}, + redirect_chain_length=0, + fetch_method="static", + ) + c.worker("https://site.com/") + assert sleeps == [0.05] + + +def test_queue_contains_swallows_queue_errors() -> None: + from website_profiling.crawl.crawler import Crawler + + class _BrokenQueue: + @property + def queue(self): + raise RuntimeError("no peek") + + c = Crawler(start_url="https://site.com", ignore_robots=True, use_wappalyzer=False) + c.queue = _BrokenQueue() + assert c._queue_contains("https://site.com/x") is False + + +def test_worker_error_path_stores_browser_diagnostics_only(monkeypatch) -> None: + import json + + from website_profiling.crawl.crawler import Crawler + from website_profiling.crawl.fetchers.base import FetchResult + + diagnostics = { + "console": [{"level": "error", "text": "fetch failed"}], + "page_errors": [], + "failed_requests": [], + "summary": {"console_error_count": 1, "console_warning_count": 0, "page_error_count": 0}, + } + monkeypatch.setattr( + "website_profiling.crawl.sitemap.discover_sitemap_urls", + lambda *_a, **_k: [], + ) + c = Crawler( + start_url="https://site.com", + ignore_robots=True, + use_wappalyzer=False, + ) + c.fetch = lambda _url: FetchResult( # type: ignore[method-assign] + status=None, + content_type=None, + text=None, + response_time_ms=None, + content_length=None, + final_url="https://site.com/a", + headers_dict={}, + redirect_chain_length=0, + fetch_method="rendered", + browser_diagnostics=diagnostics, ) - c.fetch = lambda _url: (None, None, None, None, None, None, {}, 0) # type: ignore[method-assign] out = c.worker("https://site.com/a") assert out["status"] == "error" + pa = json.loads(out["page_analysis"]) + assert pa["browser"]["summary"]["console_error_count"] == 1 diff --git a/tests/test_db_stores_unit.py b/tests/test_db_stores_unit.py index 34762e41..3771c173 100644 --- a/tests/test_db_stores_unit.py +++ b/tests/test_db_stores_unit.py @@ -126,3 +126,77 @@ def test_lighthouse_store_summary_and_run_id() -> None: rid = write_lighthouse_run(conn, url="u", strategy="mobile", run_index=0, data={"x": 1}) # type: ignore[arg-type] assert rid == 9 + +class _LegacyConn: + """Minimal conn used by config/llm/report store error-path tests.""" + + def __init__(self, row=None, rows=None, boom=False): + self.row = row + self.rows = rows or [] + self.boom = boom + self.commits = 0 + self.executed = [] + + def execute(self, sql, params=None): + self.executed.append((sql, params)) + if self.boom: + raise RuntimeError("boom") + return self + + def fetchone(self): + return self.row + + def fetchall(self): + return list(self.rows) + + def commit(self): + self.commits += 1 + + def transaction(self): + class _CM: + def __enter__(self_non): + return None + + def __exit__(self_non, _t, _v, _tb): + return False + + return _CM() + + +def test_config_store_read_error_fallbacks() -> None: + from website_profiling.db.config_store import read_llm_config, read_pipeline_config + + known, unknown = read_pipeline_config(_LegacyConn(boom=True)) # type: ignore[arg-type] + assert known == {} + assert unknown == [] + assert read_llm_config(_LegacyConn(boom=True)) == {} # type: ignore[arg-type] + + +def test_config_store_write_llm_config() -> None: + from website_profiling.db.config_store import write_llm_config + + conn = _LegacyConn() + write_llm_config(conn, {"k": "v"}, secret_keys={"k"}) # type: ignore[arg-type] + assert any("INSERT INTO llm_config" in s for s, _ in conn.executed) + + +def test_llm_cache_write_and_batch_read_error() -> None: + from website_profiling.db.llm_cache_store import read_llm_cache_batch, write_llm_cache + + conn = _LegacyConn() + write_llm_cache(conn, "k", '{"a":1}') # type: ignore[arg-type] + write_llm_cache(conn, "k", "not-json") # type: ignore[arg-type] + assert conn.commits == 2 + assert read_llm_cache_batch(_LegacyConn(boom=True), ["k"]) == {} # type: ignore[arg-type] + + +def test_report_store_write_and_read_none() -> None: + from website_profiling.db.report_store import _extract_hostname, read_report_payload, write_report_payload + + conn = _LegacyConn(row=None) + assert read_report_payload(conn) is None # type: ignore[arg-type] + conn2 = _LegacyConn() + write_report_payload(conn2, {"site_name": "S", "top_pages": [{"url": "https://x.com"}]}) # type: ignore[arg-type] + assert conn2.commits == 1 + assert _extract_hostname("https://X.com/a") == "x.com" + diff --git a/tests/test_fetchers_sitemap_config_unit.py b/tests/test_fetchers_sitemap_config_unit.py new file mode 100644 index 00000000..9cb09d1f --- /dev/null +++ b/tests/test_fetchers_sitemap_config_unit.py @@ -0,0 +1,443 @@ +"""Unit tests for crawl fetchers, sitemap discovery, browser deps, and config loading.""" +from __future__ import annotations + +import json +import subprocess +import sys +import types +from pathlib import Path +from unittest.mock import MagicMock + +import pandas as pd +import pytest + +from website_profiling.crawl.fetchers.base import FetchResult, HEADER_KEYS +from website_profiling.crawl.fetchers.static import StaticFetcher + + +def _static_result(html: str, **kwargs) -> FetchResult: + defaults = dict( + status=200, + content_type="text/html", + text=html, + response_time_ms=1, + content_length=len(html), + final_url="https://example.com/", + headers_dict={}, + redirect_chain_length=0, + fetch_method="static", + ) + defaults.update(kwargs) + return FetchResult(**defaults) # type: ignore[arg-type] + + +def test_fetch_result_as_tuple(): + r = _static_result("") + t = r.as_tuple() + assert t[0] == 200 + assert t[6] == r.headers_dict + + +def test_static_fetcher_network_error_returns_empty_result(): + class BoomSession: + headers = {} + + def get(self, *_a, **_k): + raise ConnectionError("offline") + + def close(self): + pass + + f = StaticFetcher(session=BoomSession()) # type: ignore[arg-type] + try: + out = f.fetch("https://example.com") + assert out.status is None + assert out.text is None + finally: + f.close() + + +def test_hybrid_fetch_returns_static_when_no_spa(): + from website_profiling.crawl.fetchers.hybrid import HybridFetcher + + static_html = "

Normal page with enough text content here

" + + class FakeStatic: + def fetch(self, _url): + return _static_result(static_html) + + def close(self): + pass + + class FakeBrowser: + def fetch(self, _url): + raise AssertionError("browser should not run") + + def close(self): + pass + + h = HybridFetcher(FakeStatic(), lambda: FakeBrowser()) + try: + out = h.fetch("https://example.com/") + assert out.text == static_html + finally: + h.close() + + +def test_hybrid_fetch_falls_back_when_browser_fails(): + from website_profiling.crawl.fetchers.hybrid import HybridFetcher + + static = _static_result('
') + fail = FetchResult( + status=None, + content_type=None, + text=None, + response_time_ms=None, + content_length=None, + final_url=None, + headers_dict={}, + redirect_chain_length=0, + fetch_method="rendered", + ) + + class FakeStatic: + def fetch(self, _url): + return static + + def close(self): + pass + + class FakeBrowser: + def fetch(self, _url): + return fail + + def close(self): + pass + + h = HybridFetcher(FakeStatic(), lambda: FakeBrowser()) + try: + out = h.fetch("https://example.com/") + assert out is static + finally: + h.close() + + +def test_hybrid_refetch_falls_back_to_static_on_browser_failure(): + from website_profiling.crawl.fetchers.hybrid import HybridFetcher + + static = _static_result("static ok") + fail = FetchResult( + status=None, + content_type=None, + text=None, + response_time_ms=None, + content_length=None, + final_url=None, + headers_dict={}, + redirect_chain_length=0, + fetch_method="rendered", + ) + + class FakeStatic: + def fetch(self, _url): + return static + + def close(self): + pass + + class FakeBrowser: + def fetch(self, _url): + return fail + + def close(self): + pass + + h = HybridFetcher(FakeStatic(), lambda: FakeBrowser()) + try: + out = h.refetch_rendered("https://example.com/") + assert "static ok" in (out.text or "") + finally: + h.close() + + +def test_spa_heuristics_script_heavy_shell(): + from website_profiling.crawl.fetchers.spa_heuristics import needs_js_render + + scripts = "".join("" for _ in range(10)) + html = f"{scripts}

tiny

" + assert needs_js_render(_static_result(html)) is True + + +def test_spa_heuristics_low_word_count_with_scripts(): + from website_profiling.crawl.fetchers.spa_heuristics import needs_js_render + + scripts = "".join("" for _ in range(4)) + html = "x" * 2000 + scripts + assert needs_js_render(_static_result(html)) is True + + +def test_spa_heuristics_word_count_parse_error(monkeypatch): + from website_profiling.crawl.fetchers import spa_heuristics + + def boom(_html): + raise RuntimeError("parse fail") + + import bs4 + + monkeypatch.setattr(bs4, "BeautifulSoup", boom) + assert spa_heuristics._html_word_count("x") == 0 + + +def test_spa_heuristics_after_parse_empty_html(): + from website_profiling.crawl.fetchers.spa_heuristics import needs_js_render_after_parse + + assert needs_js_render_after_parse(_static_result(""), link_count=0, same_domain_link_count=0) is False + + +def test_spa_heuristics_after_parse_many_same_domain_links(): + from website_profiling.crawl.fetchers.spa_heuristics import needs_js_render_after_parse + + html = '
' + assert needs_js_render_after_parse( + _static_result(html), link_count=0, same_domain_link_count=2 + ) is False + + +def test_sitemap_origin_invalid(): + from website_profiling.crawl.sitemap import _origin, discover_sitemap_urls + + assert _origin("not-a-url") == "" + assert discover_sitemap_urls("not-a-url") == [] + + +def test_sitemap_parse_invalid_xml(): + from website_profiling.crawl.sitemap import _parse_sitemap_xml + + pages, nested = _parse_sitemap_xml("not xml", "https://example.com/sm.xml") + assert pages == [] + assert nested == [] + + +def test_sitemap_parse_sitemap_index(): + from website_profiling.crawl.sitemap import _parse_sitemap_xml + + xml = """ + + https://example.com/sitemap-2.xml + """ + pages, nested = _parse_sitemap_xml(xml, "https://example.com/sitemap.xml") + assert pages == [] + assert "https://example.com/sitemap-2.xml" in nested + + +def test_discover_sitemap_skips_bad_responses(monkeypatch): + from website_profiling.crawl.sitemap import discover_sitemap_urls + + class FakeResp: + def __init__(self, code, text=""): + self.status_code = code + self.text = text + + class FakeSession: + headers = {} + + def get(self, url, timeout=0): + if url.endswith("/robots.txt"): + raise ConnectionError("robots down") + if url.endswith("/sitemap.xml"): + return FakeResp(404, "") + return FakeResp(404, "") + + def close(self): + pass + + monkeypatch.setattr("website_profiling.crawl.sitemap.requests.Session", lambda: FakeSession()) + urls = discover_sitemap_urls("https://example.com") + assert urls == [] + + +def test_discover_sitemap_nested_and_external_filter(monkeypatch): + from website_profiling.crawl.sitemap import discover_sitemap_urls + + class FakeResp: + def __init__(self, code, text): + self.status_code = code + self.text = text + + class FakeSession: + headers = {} + calls = 0 + + def get(self, url, timeout=0): + if url.endswith("/robots.txt"): + return FakeResp(200, "Sitemap: https://example.com/index.xml\n") + if url.endswith("/index.xml"): + return FakeResp( + 200, + """ + https://example.com/pages.xml""", + ) + if url.endswith("/pages.xml"): + return FakeResp( + 200, + """ + https://example.com/p1 + https://other.com/p2""", + ) + return FakeResp(404, "") + + def close(self): + pass + + monkeypatch.setattr("website_profiling.crawl.sitemap.requests.Session", lambda: FakeSession()) + urls = discover_sitemap_urls("https://example.com", max_urls=10) + assert "https://example.com/p1" in urls + assert all("other.com" not in u for u in urls) + + +def test_pip_install_browser_requirements_runs_subprocess(monkeypatch, tmp_path): + from website_profiling.crawl.fetchers import browser_deps + + (tmp_path / "requirements-browser.txt").write_text("playwright\n", encoding="utf-8") + monkeypatch.setenv("WEBSITE_PROFILING_ROOT", str(tmp_path)) + called: list = [] + + def fake_run(cmd, **kwargs): + called.append(cmd) + + monkeypatch.setattr(browser_deps.subprocess, "run", fake_run) + browser_deps._pip_install_browser_requirements() + assert called and "requirements-browser.txt" in called[0][-1] + + +def test_playwright_install_chromium_runs_subprocess(monkeypatch, tmp_path): + from website_profiling.crawl.fetchers import browser_deps + + monkeypatch.setenv("WEBSITE_PROFILING_ROOT", str(tmp_path)) + called: list = [] + + def fake_run(cmd, **kwargs): + called.append(cmd) + + monkeypatch.setattr(browser_deps.subprocess, "run", fake_run) + browser_deps._playwright_install_chromium() + assert called and "playwright" in called[0] + + +def test_config_get_int_float_list_invalid(): + from website_profiling.config import get_float, get_int, get_list + + assert get_int({"n": "bad"}, "n", 5) == 5 + assert get_float({"f": "bad"}, "f", 1.5) == 1.5 + assert get_list({"l": ""}, "l") == [] + assert get_list({"l": "a,b"}, "l") == ["a", "b"] + + +def test_load_config_missing_file(tmp_path): + from website_profiling.config import load_config + + assert load_config(str(tmp_path / "missing.txt")) == {} + + +def test_load_config_from_db_no_database_url(monkeypatch): + from website_profiling.config import load_config_from_db + + monkeypatch.delenv("DATABASE_URL", raising=False) + assert load_config_from_db() == {} + + +def test_load_config_from_db_runtime_error(monkeypatch, capsys): + from website_profiling.config import load_config_from_db + + monkeypatch.setenv("DATABASE_URL", "postgres://x") + monkeypatch.setattr( + "website_profiling.db.storage.get_database_url", + lambda: (_ for _ in ()).throw(RuntimeError("no url")), + ) + assert load_config_from_db() == {} + assert "no url" in capsys.readouterr().err + + +def test_load_config_from_db_query_error(monkeypatch, capsys): + from website_profiling.config import load_config_from_db + + monkeypatch.setenv("DATABASE_URL", "postgres://x") + monkeypatch.setattr("website_profiling.db.storage.get_database_url", lambda: "postgres://x") + + def boom(): + raise OSError("db") + + monkeypatch.setattr("website_profiling.db.db_session", boom) + assert load_config_from_db() == {} + assert "PostgreSQL" in capsys.readouterr().err + + +def test_property_store_extract_hostname_invalid(): + from website_profiling.db import property_store + + assert property_store._extract_hostname("://bad") == "" + + +def test_browser_diagnostics_merge_and_aggregate(): + from website_profiling.crawl.fetchers.browser_diagnostics import ( + aggregate_browser_diagnostics_df, + browser_summary_from_page_analysis, + merge_browser_into_page_analysis, + truncate_diag_text, + ) + + assert merge_browser_into_page_analysis(None, None) == "{}" + diag = {"summary": {"console_error_count": 1, "page_error_count": 0}} + merged = json.loads(merge_browser_into_page_analysis('{"x":1}', diag)) + assert merged["browser"]["summary"]["console_error_count"] == 1 + + assert browser_summary_from_page_analysis({})["console_error_count"] == 0 + + long = "z" * 600 + assert len(truncate_diag_text(long)) == 500 + + pa = json.dumps( + { + "browser": { + "console": [{"level": "error", "text": "err"}], + "page_errors": [], + "summary": {"console_error_count": 1, "page_error_count": 0}, + } + } + ) + df = pd.DataFrame([{"url": "https://a.com", "page_analysis": pa}]) + agg = aggregate_browser_diagnostics_df(df) + assert agg["pages_with_console_errors"] == 1 + assert agg["top_console_messages"][0]["text"] == "err" + + +def test_browser_diagnostics_aggregate_skips_empty_messages(): + from website_profiling.crawl.fetchers.browser_diagnostics import aggregate_browser_diagnostics_df + + pa = json.dumps( + { + "browser": { + "console": [{"level": "error", "text": " "}], + "summary": {"console_error_count": 1}, + } + } + ) + df = pd.DataFrame([{"url": "https://a.com", "page_analysis": pa}]) + agg = aggregate_browser_diagnostics_df(df) + assert agg.get("top_console_messages", []) == [] + + +def test_browser_diagnostics_parse_cell_nan(): + from website_profiling.crawl.fetchers.browser_diagnostics import _parse_page_analysis_cell + + assert _parse_page_analysis_cell(float("nan")) == {} + assert _parse_page_analysis_cell("{}") == {} + assert _parse_page_analysis_cell("not-json") == {} + assert _parse_page_analysis_cell('["list"]') == {} + + +def test_browser_diagnostics_aggregate_empty_df(): + from website_profiling.crawl.fetchers.browser_diagnostics import aggregate_browser_diagnostics_df + + assert aggregate_browser_diagnostics_df(None) == {} + assert aggregate_browser_diagnostics_df(pd.DataFrame()) == {} diff --git a/tests/test_gsc_links_csv.py b/tests/test_gsc_links_csv.py new file mode 100644 index 00000000..5b2fc9b2 --- /dev/null +++ b/tests/test_gsc_links_csv.py @@ -0,0 +1,71 @@ +"""Tests for GSC Links CSV parser.""" +from __future__ import annotations + +from pathlib import Path + +import pytest + +from website_profiling.integrations.google.gsc_links_csv import ( + detect_export_type, + merge_parsed_into_snapshot, + parse_and_merge, + parse_gsc_links_csv, +) + +FIXTURES = Path(__file__).resolve().parent / "fixtures" / "gsc_links" + + +@pytest.mark.parametrize( + "filename,expected_type,min_rows", + [ + ("top_linking_sites.csv", "top_linking_sites", 3), + ("top_linked_pages.csv", "top_linked_pages", 2), + ("top_linking_text.csv", "top_linking_text", 3), + ("sample_links.csv", "sample_links", 2), + ("latest_links.csv", "latest_links", 1), + ], +) +def test_parse_gsc_links_fixture_files(filename, expected_type, min_rows): + text = (FIXTURES / filename).read_text(encoding="utf-8") + export_type, rows = parse_gsc_links_csv(text) + assert export_type == expected_type + assert len(rows) >= min_rows + + +def test_detect_export_type_from_headers(): + assert detect_export_type(["Site", "Links", "Target pages"]) == "top_linking_sites" + assert detect_export_type(["Target page", "Links", "Linking sites"]) == "top_linked_pages" + assert detect_export_type(["Link text", "Links"]) == "top_linking_text" + assert detect_export_type(["Source page", "Target page"]) == "sample_links" + assert detect_export_type(["Source page", "Target page", "First discovered"]) == "latest_links" + + +def test_parse_empty_raises(): + with pytest.raises(ValueError, match="empty"): + parse_gsc_links_csv("") + + +def test_parse_unknown_raises(): + with pytest.raises(ValueError, match="Unrecognized"): + parse_gsc_links_csv("foo,bar\n1,2") + + +def test_merge_replaces_same_section(): + base = merge_parsed_into_snapshot(None, "top_linking_sites", [{"site": "a.com", "link_count": 1, "target_page_count": 1}]) + merged = merge_parsed_into_snapshot( + base, + "top_linking_sites", + [{"site": "b.com", "link_count": 2, "target_page_count": 3}], + ) + assert merged["top_linking_sites"][0]["site"] == "b.com" + assert "top_linking_sites" in merged["export_types"] + + +def test_parse_and_merge_accumulates_sections(): + sites = (FIXTURES / "top_linking_sites.csv").read_text(encoding="utf-8") + pages = (FIXTURES / "top_linked_pages.csv").read_text(encoding="utf-8") + snap = parse_and_merge(sites, None, crawl_urls=["https://example.com/"]) + snap = parse_and_merge(pages, snap, crawl_urls=["https://example.com/"]) + assert "top_linking_sites" in snap["export_types"] + assert "top_linked_pages" in snap["export_types"] + assert snap["top_linked_pages"][0].get("target_in_crawl") is True diff --git a/tests/test_gsc_links_store.py b/tests/test_gsc_links_store.py new file mode 100644 index 00000000..b1d5f026 --- /dev/null +++ b/tests/test_gsc_links_store.py @@ -0,0 +1,57 @@ +"""Tests for gsc_links_data store (requires PostgreSQL).""" +from __future__ import annotations + +import os + +import pytest + +from website_profiling.db import db_session +from website_profiling.db.property_store import upsert_property_by_domain +from website_profiling.integrations.google.gsc_links_store import ( + import_gsc_links_csv, + read_gsc_links_status, + read_latest_gsc_links_data, +) + + +@pytest.fixture +def property_id(): + if not (os.environ.get("DATABASE_URL") or "").strip(): + pytest.skip("DATABASE_URL not set") + with db_session() as conn: + pid = upsert_property_by_domain(conn, "GSC Links Test", "gsc-links-test.example") + yield pid + + +def test_import_and_read_roundtrip(property_id): + csv_text = "Site,Links,Target pages\nexample.com,5,2\n" + with db_session() as conn: + result = import_gsc_links_csv(conn, property_id, csv_text, file_name="sites.csv") + assert result["ok"] is True + assert "top_linking_sites" in result["export_types"] + + latest = read_latest_gsc_links_data(conn, property_id, for_report=False) + assert latest is not None + assert len(latest.get("top_linking_sites") or []) == 1 + + status = read_gsc_links_status(conn, property_id) + assert status["hasData"] is True + assert status["referringDomainCount"] == 1 + + +def test_merge_second_import(property_id): + with db_session() as conn: + import_gsc_links_csv( + conn, + property_id, + "Site,Links,Target pages\na.com,1,1\n", + ) + import_gsc_links_csv( + conn, + property_id, + "Target page,Links,Linking sites\nhttps://a.com/,3,2\n", + ) + latest = read_latest_gsc_links_data(conn, property_id, for_report=False) + assert latest is not None + assert len(latest.get("top_linking_sites") or []) == 1 + assert len(latest.get("top_linked_pages") or []) == 1 diff --git a/tests/test_more_coverage_push.py b/tests/test_historical_keywords_crawl_store_unit.py similarity index 98% rename from tests/test_more_coverage_push.py rename to tests/test_historical_keywords_crawl_store_unit.py index 8802bc74..48b7fb44 100644 --- a/tests/test_more_coverage_push.py +++ b/tests/test_historical_keywords_crawl_store_unit.py @@ -1,3 +1,4 @@ +"""Unit tests for historical db, keywords_cmd, crawl_store, and lighthouse_store.""" from __future__ import annotations import argparse @@ -370,8 +371,8 @@ def test_lighthouse_store_write_audits_from_run(monkeypatch): "numeric_value": 1, "help_text": "", "details_type": "table", - "details_headings": "[]", - "details_meta": "{}", + "details_headings": [], + "details_meta": {}, } ], [(0, 0, {"x": 1})], diff --git a/tests/test_pipeline_cmd_run_unit.py b/tests/test_pipeline_cmd_run_unit.py index 314fe2c6..136f5711 100644 --- a/tests/test_pipeline_cmd_run_unit.py +++ b/tests/test_pipeline_cmd_run_unit.py @@ -31,6 +31,38 @@ def fake_run_crawler(**_kwargs): assert called["crawl"] == 1 +def test_run_crawl_passes_render_mode_to_run_crawler(monkeypatch) -> None: + from website_profiling.commands import pipeline_cmd + + captured: dict = {} + + def fake_run_crawler(**kwargs): + captured.update(kwargs) + return pd.DataFrame([{"url": "https://site.com", "status": 200}]) + + import website_profiling.crawl.crawler as crawler_mod + + monkeypatch.setattr(crawler_mod, "run_crawler", fake_run_crawler) + + cfg = { + "start_url": "https://site.com", + "run_crawl": "true", + "run_report": "false", + "run_plot": "false", + "run_lighthouse": "false", + "run_lighthouse_on_pages": "false", + "crawl_render_mode": "javascript", + "crawl_js_concurrency": "2", + "crawl_js_timeout": "25", + } + args = argparse.Namespace(command=None) + pipeline_cmd.run(cfg, args) + + assert captured.get("render_mode") == "javascript" + assert captured.get("js_concurrency") == 2 + assert captured.get("js_timeout") == 25 + + def test_pipeline_lighthouse_on_pages_uses_selected_urls(monkeypatch) -> None: from website_profiling.commands import pipeline_cmd diff --git a/tests/test_high_coverage_push2.py b/tests/test_pipeline_report_pool_unit.py similarity index 58% rename from tests/test_high_coverage_push2.py rename to tests/test_pipeline_report_pool_unit.py index 3bd71ff3..cfeecc43 100644 --- a/tests/test_high_coverage_push2.py +++ b/tests/test_pipeline_report_pool_unit.py @@ -1,3 +1,4 @@ +"""Unit tests for pipeline_cmd, report metadata, and db pool lifecycle.""" from __future__ import annotations import argparse @@ -54,6 +55,109 @@ def test_run_single_lighthouse_exits_on_nonzero(monkeypatch): assert e.value.code == 2 +def test_run_plot_passes_render_mode_to_run_plot(monkeypatch): + from website_profiling.commands import pipeline_cmd + + captured: dict = {} + + def fake_run_plot(**kwargs): + captured.update(kwargs) + + monkeypatch.setitem( + __import__("sys").modules, + "website_profiling.tools.plot", + types.SimpleNamespace(run_plot=fake_run_plot), + ) + + pipeline_cmd._run_plot( + { + "crawl_render_mode": "javascript", + "crawl_js_concurrency": "2", + "crawl_js_timeout": "25", + "crawl_js_wait_until": "load", + "crawl_js_extra_wait_ms": "2000", + "crawl_js_block_resources": "false", + }, + True, + ) + + assert captured.get("render_mode") == "javascript" + assert captured.get("js_concurrency") == 2 + assert captured.get("js_timeout") == 25 + assert captured.get("js_wait_until") == "load" + assert captured.get("js_extra_wait_ms") == 2000 + assert captured.get("js_block_resources") is False + + +def test_build_report_metadata_includes_fetch_method_counts() -> None: + import pandas as pd + from website_profiling.reporting.builder import _build_report_metadata + + df = pd.DataFrame( + [ + {"url": "https://a.com/1", "status": "200", "fetch_method": "static"}, + {"url": "https://a.com/2", "status": "200", "fetch_method": "rendered"}, + {"url": "https://a.com/3", "status": "200", "fetch_method": "static"}, + ] + ) + meta = _build_report_metadata( + df, + {"crawl_render_mode": "auto", "max_pages": "10"}, + None, + None, + None, + {}, + 1, + None, + ) + scope = meta["crawl_scope"] + assert scope["pages_static"] == 2 + assert scope["pages_rendered"] == 1 + assert scope["render_mode"] == "auto" + + +def test_build_report_metadata_aggregates_browser_diagnostics(): + import json + + import pandas as pd + + from website_profiling.reporting.builder import _build_report_metadata + + pa_with_error = json.dumps( + { + "browser": { + "console": [{"level": "error", "text": "Same error"}], + "page_errors": [{"message": "Uncaught"}], + "summary": {"console_error_count": 1, "page_error_count": 1}, + } + } + ) + pa_clean = json.dumps({"browser": {"console": [], "page_errors": [], "summary": {"console_error_count": 0}}}) + df = pd.DataFrame( + [ + {"url": "https://a.com/1", "status": "200", "fetch_method": "rendered", "page_analysis": pa_with_error}, + {"url": "https://a.com/2", "status": "200", "fetch_method": "rendered", "page_analysis": pa_with_error}, + {"url": "https://a.com/3", "status": "200", "fetch_method": "static", "page_analysis": pa_clean}, + ] + ) + meta = _build_report_metadata( + df, + {"crawl_render_mode": "javascript", "max_pages": "10"}, + None, + None, + None, + {}, + 1, + None, + ) + bd = meta["crawl_scope"]["browser_diagnostics"] + assert bd["pages_with_console_errors"] == 2 + assert bd["total_console_errors"] == 2 + assert bd["pages_with_page_errors"] == 2 + assert len(bd["top_console_messages"]) >= 1 + assert bd["top_console_messages"][0]["text"] == "Same error" + + def test_run_report_and_plot_paths(monkeypatch): from website_profiling.commands import pipeline_cmd diff --git a/tests/test_pool_and_report_store_unit.py b/tests/test_pool_and_report_store_unit.py index f83f2494..458cd9f8 100644 --- a/tests/test_pool_and_report_store_unit.py +++ b/tests/test_pool_and_report_store_unit.py @@ -32,3 +32,27 @@ def test_canonical_domain_falls_back_to_top_pages(monkeypatch) -> None: domain = report_store._canonical_domain_from_report(conn, {"top_pages": [{"url": "https://Top.Example/a"}]}) # type: ignore[arg-type] assert domain == "top.example" + +def test_get_database_url_raises_when_missing(monkeypatch) -> None: + from website_profiling.db.pool import get_database_url + + monkeypatch.delenv("DATABASE_URL", raising=False) + import pytest + + with pytest.raises(RuntimeError): + get_database_url() + + +def test_report_store_links_fallback_and_read_exception(monkeypatch) -> None: + from website_profiling.db import report_store + + monkeypatch.setattr(report_store, "get_crawl_run_info", lambda _c, _rid: None) + domain = report_store._canonical_domain_from_report(object(), {"links": [{"url": "https://fallback.com/p"}]}) # type: ignore[arg-type] + assert domain == "fallback.com" + + class BoomConn: + def execute(self, *_a, **_k): + raise RuntimeError("x") + + assert report_store.read_report_payload(BoomConn()) is None # type: ignore[arg-type] + diff --git a/tests/test_small_coverage_win.py b/tests/test_small_coverage_win.py deleted file mode 100644 index 094b77b2..00000000 --- a/tests/test_small_coverage_win.py +++ /dev/null @@ -1,79 +0,0 @@ -from __future__ import annotations - -import types - - -class Conn: - def __init__(self, row=None, rows=None, boom=False): - self.row = row - self.rows = rows or [] - self.boom = boom - self.commits = 0 - self.executed = [] - - def execute(self, sql, params=None): - self.executed.append((sql, params)) - if self.boom: - raise RuntimeError("boom") - return self - - def fetchone(self): - return self.row - - def fetchall(self): - return list(self.rows) - - def commit(self): - self.commits += 1 - - def transaction(self): - class CM: - def __enter__(self_non): - return None - - def __exit__(self_non, _t, _v, _tb): - return False - - return CM() - - -def test_config_store_error_fallbacks(): - from website_profiling.db.config_store import read_llm_config, read_pipeline_config - - known, unknown = read_pipeline_config(Conn(boom=True)) # type: ignore[arg-type] - assert known == {} - assert unknown == [] - assert read_llm_config(Conn(boom=True)) == {} # type: ignore[arg-type] - - -def test_config_store_write_llm_config_commits(): - from website_profiling.db.config_store import write_llm_config - - conn = Conn() - write_llm_config(conn, {"k": "v"}, secret_keys={"k"}) # type: ignore[arg-type] - assert conn.commits == 0 # transaction style; no explicit commit in function - assert any("INSERT INTO llm_config" in s for s, _ in conn.executed) - - -def test_llm_cache_write_and_error_paths(): - from website_profiling.db.llm_cache_store import read_llm_cache_batch, write_llm_cache - - conn = Conn() - write_llm_cache(conn, "k", '{"a":1}') # type: ignore[arg-type] - write_llm_cache(conn, "k", "not-json") # type: ignore[arg-type] - assert conn.commits == 2 - - # exception branch - assert read_llm_cache_batch(Conn(boom=True), ["k"]) == {} # type: ignore[arg-type] - - -def test_report_store_write_and_read_none(): - from website_profiling.db.report_store import _extract_hostname, read_report_payload, write_report_payload - - conn = Conn(row=None) - assert read_report_payload(conn) is None # type: ignore[arg-type] - conn2 = Conn() - write_report_payload(conn2, {"site_name": "S", "top_pages": [{"url": "https://x.com"}]}) # type: ignore[arg-type] - assert conn2.commits == 1 - assert _extract_hostname("https://X.com/a") == "x.com" - diff --git a/tests/test_storage_bulk.py b/tests/test_storage_bulk.py index f4c928cb..776976d8 100644 --- a/tests/test_storage_bulk.py +++ b/tests/test_storage_bulk.py @@ -51,3 +51,39 @@ def test_write_crawl_dataframe_executemany(pg_conn): write_crawl(pg_conn, df, crawl_run_id=run_id) out = read_crawl(pg_conn, run_id) assert len(out) == 50 + + +def test_write_crawl_persists_fetch_method(pg_conn): + run_id = create_crawl_run(pg_conn, "https://fetch.example.com") + df = pd.DataFrame( + [ + { + "url": "https://fetch.example.com/rendered", + "status": "200", + "title": "Rendered", + "fetch_method": "rendered", + }, + { + "url": "https://fetch.example.com/static", + "status": "200", + "title": "Static", + }, + ] + ) + write_crawl(pg_conn, df, crawl_run_id=run_id) + + cur = pg_conn.execute( + "SELECT url, fetch_method FROM crawl_results WHERE crawl_run_id = %s ORDER BY url", + (run_id,), + ) + rows = cur.fetchall() + assert len(rows) == 2 + by_url = {r["url"]: r["fetch_method"] for r in rows} + assert by_url["https://fetch.example.com/rendered"] == "rendered" + assert by_url["https://fetch.example.com/static"] == "static" + + df = read_crawl(pg_conn, run_id) + assert len(df) == 2 + by_url_df = df.set_index("url")["fetch_method"].to_dict() + assert by_url_df["https://fetch.example.com/rendered"] == "rendered" + assert by_url_df["https://fetch.example.com/static"] == "static" diff --git a/web/app/api/crawl/browser-status/route.ts b/web/app/api/crawl/browser-status/route.ts new file mode 100644 index 00000000..53860c7a --- /dev/null +++ b/web/app/api/crawl/browser-status/route.ts @@ -0,0 +1,98 @@ +import { spawn } from 'child_process'; +import { NextResponse } from 'next/server'; +import { forbiddenIfNotLocal } from '@/server/localOnly'; +import { getPipelineSpawnEnv, getRepoRoot } from '@/server/pipelineSpawnEnv'; +import { formatPythonSpawnError, resolvePythonExecutable } from '@/server/resolvePython'; +import type { ApiRouteHandler } from '@/types/api'; + +export const runtime = 'nodejs'; +export const dynamic = 'force-dynamic'; + +const CHECK_SCRIPT = + 'from website_profiling.crawl.fetchers import ensure_browser_deps; import json; print(json.dumps(ensure_browser_deps()))'; + +/** First-time Playwright/Chromium install can take a few minutes. */ +const CHECK_TIMEOUT_MS = 180_000; + +/** + * GET /api/crawl/browser-status + * Returns whether Playwright and Chromium are available for JS/auto crawls. + */ +export const GET: ApiRouteHandler = async (request): Promise => { + const denied = forbiddenIfNotLocal(request); + if (denied) return denied; + + const repoRoot = getRepoRoot(); + const pythonExe = resolvePythonExecutable(null, repoRoot); + + return new Promise((resolve) => { + let stdout = ''; + let stderr = ''; + const proc = spawn(pythonExe, ['-c', CHECK_SCRIPT], { + cwd: repoRoot, + env: getPipelineSpawnEnv(), + shell: false, + }); + + const appendStdout = (chunk: Buffer | string): void => { + stdout += chunk.toString(); + }; + const appendStderr = (chunk: Buffer | string): void => { + stderr += chunk.toString(); + }; + proc.stdout?.on('data', appendStdout); + proc.stderr?.on('data', appendStderr); + + const finish = (payload: { ok: boolean; message?: string; error?: string }, status = 200) => { + resolve(NextResponse.json(payload, { status })); + }; + + proc.on('error', (err: Error) => { + finish({ + ok: false, + message: formatPythonSpawnError(err, pythonExe, repoRoot), + error: err.message, + }); + }); + + proc.on('close', (code: number | null) => { + if (code !== 0) { + finish({ + ok: false, + message: + stderr.trim() || + 'JavaScript crawl requires Playwright and Chromium. Install: pip install -r requirements-browser.txt.', + error: stderr.trim() || `exit ${code}`, + }); + return; + } + try { + const line = stdout.trim().split('\n').filter(Boolean).pop() || '{}'; + const parsed = JSON.parse(line) as { ok?: boolean; message?: string }; + finish({ + ok: Boolean(parsed.ok), + message: parsed.message, + }); + } catch { + finish({ + ok: false, + message: 'Could not parse browser status from Python.', + error: stdout.slice(-500) || stderr.slice(-500), + }); + } + }); + + setTimeout(() => { + try { + proc.kill(); + } catch { + /* ignore */ + } + finish({ + ok: false, + message: 'Browser status check timed out.', + error: 'timeout', + }); + }, CHECK_TIMEOUT_MS); + }); +}; diff --git a/web/app/api/properties/[id]/google/links/import/route.ts b/web/app/api/properties/[id]/google/links/import/route.ts new file mode 100644 index 00000000..e1ccf240 --- /dev/null +++ b/web/app/api/properties/[id]/google/links/import/route.ts @@ -0,0 +1,119 @@ +import { NextResponse, type NextRequest } from 'next/server'; +import { spawn } from 'child_process'; +import { forbiddenIfNotLocal } from '@/server/localOnly'; +import { getPropertyById } from '@/server/propertiesDb'; +import { getPipelineSpawnEnv, getRepoRoot } from '@/server/pipelineSpawnEnv'; +import { + formatPythonSpawnError, + parsePythonJsonStdout, + resolvePythonExecutable, +} from '@/server/resolvePython'; +import type { ApiRouteHandlerWithParams } from '@/types/api'; + +export const runtime = 'nodejs'; + +interface ImportBody { + fileContent?: string; + fileName?: string; +} + +export const POST: ApiRouteHandlerWithParams<{ id: string }> = async ( + request: NextRequest, + { params }: { params: Promise<{ id: string }> }, +): Promise => { + const denied = forbiddenIfNotLocal(request); + if (denied) return denied; + + const { id } = await params; + const propertyId = parseInt(id, 10); + if (!Number.isFinite(propertyId)) { + return NextResponse.json({ error: 'Invalid property id' }, { status: 400 }); + } + const row = await getPropertyById(propertyId); + if (!row) { + return NextResponse.json({ error: 'Property not found' }, { status: 404 }); + } + + const body = (await request.json().catch(() => ({}))) as ImportBody; + const fileContent = body.fileContent; + if (!fileContent || typeof fileContent !== 'string' || !fileContent.trim()) { + return NextResponse.json({ error: 'fileContent is required' }, { status: 400 }); + } + + const fileName = typeof body.fileName === 'string' ? body.fileName : ''; + const repoRoot = getRepoRoot(); + const pythonExe = resolvePythonExecutable(null, repoRoot); + + return new Promise((resolve) => { + let stdout = ''; + let stderr = ''; + const args = [ + '-m', + 'src', + 'gsc-links-import', + '--property-id', + String(propertyId), + '--csv-stdin', + ]; + if (fileName) { + args.push('--file-name', fileName); + } + + const proc = spawn(pythonExe, args, { + cwd: repoRoot, + env: getPipelineSpawnEnv(repoRoot, propertyId), + shell: false, + }); + + proc.stdin?.write(fileContent); + proc.stdin?.end(); + + proc.stdout?.on('data', (c: Buffer | string) => { + stdout += c.toString(); + }); + proc.stderr?.on('data', (c: Buffer | string) => { + stderr += c.toString(); + }); + + proc.on('error', (err: Error) => { + resolve( + NextResponse.json( + { error: formatPythonSpawnError(err, pythonExe, repoRoot) }, + { status: 500 }, + ), + ); + }); + + proc.on('close', (code: number | null) => { + const parsed = parsePythonJsonStdout(stdout); + if (parsed && code === 0 && parsed.ok) { + resolve(NextResponse.json(parsed)); + return; + } + if (parsed) { + const errMsg = + typeof parsed.error === 'string' + ? parsed.error + : stdout.trim() || stderr.trim() || 'Import failed'; + resolve(NextResponse.json({ error: errMsg, detail: parsed }, { status: 400 })); + return; + } + const raw = stdout.trim() || stderr.trim(); + resolve( + NextResponse.json( + { error: raw || 'Import failed', exitCode: code }, + { status: code === 0 ? 500 : 400 }, + ), + ); + }); + + setTimeout(() => { + try { + proc.kill(); + } catch { + /* ignore */ + } + resolve(NextResponse.json({ error: 'Timed out' }, { status: 504 })); + }, 120_000); + }); +}; diff --git a/web/app/api/properties/[id]/google/links/status/route.ts b/web/app/api/properties/[id]/google/links/status/route.ts new file mode 100644 index 00000000..bfd6222e --- /dev/null +++ b/web/app/api/properties/[id]/google/links/status/route.ts @@ -0,0 +1,84 @@ +import { NextResponse, type NextRequest } from 'next/server'; +import { spawn } from 'child_process'; +import { forbiddenIfNotLocal } from '@/server/localOnly'; +import { getPropertyById } from '@/server/propertiesDb'; +import { getPipelineSpawnEnv, getRepoRoot } from '@/server/pipelineSpawnEnv'; +import { + formatPythonSpawnError, + parsePythonJsonStdout, + resolvePythonExecutable, +} from '@/server/resolvePython'; +import type { ApiRouteHandlerWithParams } from '@/types/api'; + +export const runtime = 'nodejs'; + +export const GET: ApiRouteHandlerWithParams<{ id: string }> = async ( + request: NextRequest, + { params }: { params: Promise<{ id: string }> }, +): Promise => { + const denied = forbiddenIfNotLocal(request); + if (denied) return denied; + + const { id } = await params; + const propertyId = parseInt(id, 10); + if (!Number.isFinite(propertyId)) { + return NextResponse.json({ error: 'Invalid property id' }, { status: 400 }); + } + const row = await getPropertyById(propertyId); + if (!row) { + return NextResponse.json({ error: 'Property not found' }, { status: 404 }); + } + + const repoRoot = getRepoRoot(); + const pythonExe = resolvePythonExecutable(null, repoRoot); + + return new Promise((resolve) => { + let stdout = ''; + let stderr = ''; + const proc = spawn( + pythonExe, + ['-m', 'src', 'gsc-links-import', '--status', '--property-id', String(propertyId)], + { cwd: repoRoot, env: getPipelineSpawnEnv(repoRoot, propertyId), shell: false }, + ); + + proc.stdout?.on('data', (c: Buffer | string) => { + stdout += c.toString(); + }); + proc.stderr?.on('data', (c: Buffer | string) => { + stderr += c.toString(); + }); + + proc.on('error', (err: Error) => { + resolve( + NextResponse.json( + { error: formatPythonSpawnError(err, pythonExe, repoRoot) }, + { status: 500 }, + ), + ); + }); + + proc.on('close', (code: number | null) => { + const parsed = parsePythonJsonStdout(stdout); + if (parsed && code === 0) { + resolve(NextResponse.json(parsed)); + return; + } + const raw = stdout.trim() || stderr.trim(); + resolve( + NextResponse.json( + { error: raw || 'Status check failed', exitCode: code }, + { status: code === 0 ? 500 : 400 }, + ), + ); + }); + + setTimeout(() => { + try { + proc.kill(); + } catch { + /* ignore */ + } + resolve(NextResponse.json({ error: 'Timed out' }, { status: 504 })); + }, 30_000); + }); +}; diff --git a/web/src/ReportShell.tsx b/web/src/ReportShell.tsx index dc39e901..15d305fb 100644 --- a/web/src/ReportShell.tsx +++ b/web/src/ReportShell.tsx @@ -20,6 +20,7 @@ import { Images, FolderTree, TrendingUp, + Link2, Key, ArrowLeftRight, FileDown, @@ -62,6 +63,7 @@ const ContentAnalytics = dynamic(() => import('./views/ContentAnalytics'), { loa const TechStack = dynamic(() => import('./views/TechStack'), { loading: () => viewLoading() }); const Gallery = dynamic(() => import('./views/Gallery'), { loading: () => viewLoading() }); const SearchPerformance = dynamic(() => import('./views/SearchPerformance'), { loading: () => viewLoading() }); +const Backlinks = dynamic(() => import('./views/Backlinks'), { loading: () => viewLoading() }); const Traffic = dynamic(() => import('./views/Traffic'), { loading: () => viewLoading() }); const KeywordsExplorer = dynamic(() => import('./views/KeywordsExplorer'), { loading: () => viewLoading() }); const ExportReport = dynamic(() => import('./views/ExportReport'), { loading: () => viewLoading() }); @@ -113,6 +115,7 @@ const VIEW_CONFIG: ViewConfigEntry[] = [ { id: 'network', component: Network as ComponentType, icon: Share2 }, { id: 'gallery', component: Gallery as ComponentType, icon: Images }, { id: 'search-performance', component: SearchPerformance as ComponentType, icon: TrendingUp }, + { id: 'backlinks', component: Backlinks as ComponentType, icon: Link2 }, { id: 'traffic', component: Traffic as ComponentType, icon: BarChart2 }, { id: 'keywords-explorer', component: KeywordsExplorer as ComponentType, icon: Key }, ]; diff --git a/web/src/components/CrawlScopeBanner.tsx b/web/src/components/CrawlScopeBanner.tsx index 5b027674..c0ece50b 100644 --- a/web/src/components/CrawlScopeBanner.tsx +++ b/web/src/components/CrawlScopeBanner.tsx @@ -11,6 +11,20 @@ export default function CrawlScopeBanner({ data }: { data: ReportPayload | null const max = Number(scope.max_pages_configured ?? 0); const blocked = Number(scope.robots_blocked_count ?? 0); const limited = Boolean(scope.crawl_limited); + const renderMode = String(scope.render_mode ?? 'static'); + const jsConcurrency = scope.js_concurrency != null ? Number(scope.js_concurrency) : null; + const pagesStatic = scope.pages_static != null ? Number(scope.pages_static) : null; + const pagesRendered = scope.pages_rendered != null ? Number(scope.pages_rendered) : null; + const browserDiag = scope.browser_diagnostics as + | { + pages_with_console_errors?: number; + pages_with_page_errors?: number; + total_console_errors?: number; + } + | undefined; + const pagesWithConsoleErrors = Number(browserDiag?.pages_with_console_errors ?? 0); + const totalConsoleErrors = Number(browserDiag?.total_console_errors ?? 0); + const pagesWithPageErrors = Number(browserDiag?.pages_with_page_errors ?? 0); return (
0 ? (

{format(cs.robotsLine, { count: blocked.toLocaleString() })}

) : null} -

{cs.staticHtmlNote}

+

+ {renderMode === 'javascript' + ? cs.javascriptNote + : renderMode === 'auto' + ? cs.autoNote + : cs.staticHtmlNote} +

+ {renderMode !== 'static' && jsConcurrency != null && jsConcurrency > 0 ? ( +

+ {format(cs.jsConcurrencyLine, { count: jsConcurrency.toLocaleString() })} +

+ ) : null} + {renderMode === 'auto' && + pagesStatic != null && + pagesRendered != null && + (pagesStatic > 0 || pagesRendered > 0) ? ( +

+ {format(cs.fetchMethodMixLine, { + staticCount: pagesStatic.toLocaleString(), + renderedCount: pagesRendered.toLocaleString(), + })} +

+ ) : null} + {pagesWithConsoleErrors > 0 ? ( +

+ {format(cs.browserConsoleErrorsLine, { + pages: pagesWithConsoleErrors.toLocaleString(), + errors: totalConsoleErrors.toLocaleString(), + })} +

+ ) : null} + {pagesWithPageErrors > 0 ? ( +

+ {format(cs.browserPageErrorsLine, { + pages: pagesWithPageErrors.toLocaleString(), + })} +

+ ) : null}
diff --git a/web/src/components/GoogleIntegrationsPanel.tsx b/web/src/components/GoogleIntegrationsPanel.tsx index 6384e480..bf9760f7 100644 --- a/web/src/components/GoogleIntegrationsPanel.tsx +++ b/web/src/components/GoogleIntegrationsPanel.tsx @@ -360,6 +360,21 @@ export default function GoogleIntegrationsPanel({ const [fetchJobStatus, setFetchJobStatus] = useState(''); const fetchPollStopRef = useRef<(() => void) | null>(null); + const [linksStatus, setLinksStatus] = useState<{ + hasData?: boolean; + lastImportedAt?: string; + exportTypes?: string[]; + rowCounts?: Record; + referringDomainCount?: number; + topLinkedPageCount?: number; + sampleLinkCount?: number; + latestLinkCount?: number; + } | null>(null); + const [loadingLinksStatus, setLoadingLinksStatus] = useState(false); + const [uploadingLinks, setUploadingLinks] = useState(false); + const [linksUploadMessage, setLinksUploadMessage] = useState(''); + const linksFileInputRef = useRef(null); + // Advanced accordion (paste refresh token) const [showAdvanced, setShowAdvanced] = useState(false); const [refreshToken, setRefreshToken] = useState(''); @@ -426,6 +441,63 @@ export default function GoogleIntegrationsPanel({ void fetchStatus(); }, [fetchStatus]); + const fetchLinksStatus = useCallback(async () => { + if (effectivePropertyId == null || !endpoints.linksStatus) { + setLinksStatus(null); + return; + } + setLoadingLinksStatus(true); + try { + const res = await fetch(endpoints.linksStatus); + if (res.ok) { + setLinksStatus((await res.json()) as typeof linksStatus); + } + } catch { + setLinksStatus(null); + } finally { + setLoadingLinksStatus(false); + } + }, [effectivePropertyId, endpoints.linksStatus]); + + useEffect(() => { + void fetchLinksStatus(); + }, [fetchLinksStatus]); + + const handleLinksFile = useCallback( + async (file: File) => { + if (effectivePropertyId == null || !endpoints.linksImport) return; + setUploadingLinks(true); + setLinksUploadMessage(''); + try { + const fileContent = await file.text(); + const res = await fetch(endpoints.linksImport, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ fileContent, fileName: file.name }), + }); + const data = (await res.json()) as { ok?: boolean; error?: string }; + if (!res.ok || !data.ok) { + setLinksUploadMessage( + format(s.gscLinksUploadFailed, { message: data.error || res.statusText }), + ); + return; + } + setLinksUploadMessage(s.gscLinksUploadSuccess); + await fetchLinksStatus(); + } catch (e) { + setLinksUploadMessage( + format(s.gscLinksUploadFailed, { + message: e instanceof Error ? e.message : String(e), + }), + ); + } finally { + setUploadingLinks(false); + if (linksFileInputRef.current) linksFileInputRef.current.value = ''; + } + }, + [effectivePropertyId, endpoints.linksImport, fetchLinksStatus, s.gscLinksUploadFailed, s.gscLinksUploadSuccess], + ); + useEffect(() => { if (initialToast) setToast(initialToast); }, [initialToast]); @@ -1114,6 +1186,85 @@ export default function GoogleIntegrationsPanel({ ) : null} + {effectivePropertyId != null && endpoints.linksImport ? ( + +

+ + {s.gscLinksHelpLabel} + + +

+

{s.gscLinksUploadHint}

+ + {loadingLinksStatus ? ( +

+ + Loading import status… +

+ ) : linksStatus?.hasData && linksStatus.lastImportedAt ? ( +

+ {format(s.gscLinksLastImport, { + date: new Date(String(linksStatus.lastImportedAt)).toLocaleString(), + })} + {' · '} + {format(s.gscLinksRowCounts, { + domains: linksStatus.referringDomainCount ?? 0, + pages: linksStatus.topLinkedPageCount ?? 0, + sample: + (linksStatus.sampleLinkCount ?? 0) + (linksStatus.latestLinkCount ?? 0), + })} +

+ ) : ( +

{s.gscLinksNoData}

+ )} + +
+ { + const file = e.target.files?.[0]; + if (file) void handleLinksFile(file); + }} + /> + +
+ {linksUploadMessage ? ( +

+ {linksUploadMessage} +

+ ) : null} + + ) : null} + {effectivePropertyId != null ? (
+ ); +} diff --git a/web/src/components/backlinks/backlinksTableUtils.ts b/web/src/components/backlinks/backlinksTableUtils.ts new file mode 100644 index 00000000..bbbb4d6f --- /dev/null +++ b/web/src/components/backlinks/backlinksTableUtils.ts @@ -0,0 +1,63 @@ +import type { ExportColumn } from '@/types/components'; +import type { GscLinksReportData } from '@/types/report'; + +export { + PAGE_SIZE, + paginateSlice, + filterBySearch, + exportCsv, +} from '../google/tableUtils'; + +export function buildDomainExportColumns( + table: Record, +): ExportColumn[] { + return [ + { key: 'site', label: table.site }, + { key: 'link_count', label: table.links }, + { key: 'target_page_count', label: table.targetPages }, + ]; +} + +export function buildLinkedPageExportColumns( + table: Record, +): ExportColumn[] { + return [ + { key: 'target_page', label: table.targetPage }, + { key: 'link_count', label: table.links }, + { key: 'linking_site_count', label: table.linkingSites }, + ]; +} + +export function buildAnchorExportColumns(table: Record): ExportColumn[] { + return [ + { key: 'anchor_text', label: table.anchorText }, + { key: 'link_count', label: table.links }, + ]; +} + +export function buildSampleLinkExportColumns( + table: Record, +): ExportColumn[] { + return [ + { key: 'source_page', label: table.sourcePage }, + { key: 'target_page', label: table.targetPage }, + { key: 'anchor_text', label: table.anchorText }, + { key: 'linking_site', label: table.site }, + { key: 'discovered_at', label: table.discovered }, + ]; +} + +export function combinedSampleLinks(data: GscLinksReportData | undefined) { + const sample = (data?.sample_links ?? []).map((r) => ({ ...r, link_kind: 'sample' })); + const latest = (data?.latest_links ?? []).map((r) => ({ ...r, link_kind: 'latest' })); + return [...sample, ...latest]; +} + +export function summaryCounts(data: GscLinksReportData | undefined) { + return { + referringDomains: data?.top_linking_sites?.length ?? 0, + linkedPages: data?.top_linked_pages?.length ?? 0, + sampleLinks: data?.sample_links?.length ?? 0, + latestLinks: data?.latest_links?.length ?? 0, + }; +} diff --git a/web/src/components/links/tabs/PageAnalysisTab.tsx b/web/src/components/links/tabs/PageAnalysisTab.tsx index a9aa4b24..667abaa6 100644 --- a/web/src/components/links/tabs/PageAnalysisTab.tsx +++ b/web/src/components/links/tabs/PageAnalysisTab.tsx @@ -2,7 +2,7 @@ import { useMemo, useState } from 'react'; import { Bar } from 'react-chartjs-2'; import type { TooltipItem } from 'chart.js'; import { Gauge, ChevronDown, ChevronRight } from 'lucide-react'; -import type { LinkDetail, LinkLighthouseData, LighthouseAuditRef, NlpSignals, PageAnalysis, SimilarInternalRow } from '@/types/report'; +import type { LinkDetail, LinkLighthouseData, LighthouseAuditRef, NlpSignals, PageAnalysis, SimilarInternalRow, BrowserDiagnostics } from '@/types/report'; import { useReport } from '../../../context/useReport'; import { formatLhMetric, parseKeywords, normaliseKw, severityBg } from '../../../utils/linkUtils'; import { palette, scoreBandColor } from '../../../utils/chartPalette'; @@ -31,6 +31,116 @@ function normalizeSimilarInternal(raw: unknown): SimilarInternalRow[] { .filter((row): row is SimilarInternalRow => row != null); } +function BrowserDiagnosticsBlock({ browser }: { browser: BrowserDiagnostics | undefined }) { + const p = strings.components.linkTabs.pageAnalysis; + const [expandedStacks, setExpandedStacks] = useState>({}); + + if (!browser || typeof browser !== 'object') return null; + + const consoleMsgs = Array.isArray(browser.console) ? browser.console : []; + const pageErrors = Array.isArray(browser.page_errors) ? browser.page_errors : []; + const failedRequests = Array.isArray(browser.failed_requests) ? browser.failed_requests : []; + const hasAny = consoleMsgs.length > 0 || pageErrors.length > 0 || failedRequests.length > 0; + + return ( +
+

+ {p.browserConsoleTitle} +

+ {!hasAny ? ( +

{p.browserConsoleClean}

+ ) : ( +
+ {consoleMsgs.length > 0 ? ( +
+
+ {p.browserConsoleMessages} +
+ + + + + + + + + + {consoleMsgs.map((msg, i) => ( + + + + + + ))} + +
{p.browserThLevel}{p.browserThMessage}{p.browserThLocation}
{msg.level || '—'}{msg.text || '—'} + {msg.source_url + ? `${msg.source_url}${msg.line != null ? `:${msg.line}` : ''}` + : '—'} +
+
+ ) : null} + + {pageErrors.length > 0 ? ( +
+
+ {p.browserUncaughtExceptions} +
+
    + {pageErrors.map((err, i) => ( +
  • +
    {err.message || '—'}
    + {err.stack ? ( + + ) : null} + {err.stack && expandedStacks[i] ? ( + 
    +                        {err.stack}
+
    + ) : null} +
    • + ))} +
+
+ ) : null} + + {failedRequests.length > 0 ? ( +
+
+ {p.browserFailedRequests} +
+ + + + + + + + + + {failedRequests.map((req, i) => ( + + + + + + ))} + +
{p.browserThMethod}{p.browserThUrl}{p.browserThFailure}
{req.method || '—'}{req.url || '—'}{req.failure || '—'}
+
+ ) : null} +
+ )} +
+ ); +} + interface NerBlockProps { nlp: NlpSignals | undefined; } @@ -517,6 +627,8 @@ export default function PageAnalysisTab({ link }: PageAnalysisTabProps) { )} + + {/* Warnings */}
diff --git a/web/src/components/pipeline/PipelineSettingsPanel.tsx b/web/src/components/pipeline/PipelineSettingsPanel.tsx index cca668a5..02e1734c 100644 --- a/web/src/components/pipeline/PipelineSettingsPanel.tsx +++ b/web/src/components/pipeline/PipelineSettingsPanel.tsx @@ -4,7 +4,8 @@ import { useEffect, useMemo, useState, type ReactNode } from 'react'; import { Loader2, Save, X } from 'lucide-react'; import { strings, format } from '@/lib/strings'; import type { IntegrationToast } from '@/types/api'; -import { PIPELINE_CONFIG_SECTIONS } from '@/lib/pipelineConfigSchema'; +import { crawlRenderModeUsesBrowser } from '@/lib/browserCrawlStatus'; +import { PIPELINE_CONFIG_SECTIONS, isPipelineFieldVisible } from '@/lib/pipelineConfigSchema'; import { LLM_CONFIG_SECTIONS } from '@/lib/llmConfigSchema'; import { usePipeline } from '@/context/PipelineContext'; import Button from '@/components/Button'; @@ -40,7 +41,7 @@ function ConfigSectionFields({ }) { return (
- {section.fields.map((f) => ( + {section.fields.filter((f) => isPipelineFieldVisible(f, values)).map((f) => ( g.id === activeGroup); const showLegacyBanner = configSource === 'legacy' && !legacyBannerDismissed && activeGroup === 'crawl-report'; @@ -300,6 +307,18 @@ export default function PipelineSettingsPanel({ return (
+ {showBrowserCrawlBanner ? ( +
+

+ {s.browserCrawlBannerTitle} +

+

+ {browserCrawlChecking + ? s.browserCrawlChecking + : browserCrawlStatus?.message?.trim() || s.browserCrawlBannerHint} +

+
+ ) : null} {showLegacyBanner ? (

{s.legacyBanner}

diff --git a/web/src/context/PipelineContext.tsx b/web/src/context/PipelineContext.tsx index e07aaff7..e343d3af 100644 --- a/web/src/context/PipelineContext.tsx +++ b/web/src/context/PipelineContext.tsx @@ -20,6 +20,11 @@ import { currentPathForReturn, readPipelineReturnPath, storePipelineReturnPath, import { deriveSiteNameFromStartUrl } from '@/lib/domainSlug'; import { useOptionalReport } from '@/context/useReport'; import { strings, format } from '@/lib/strings'; +import { + type BrowserCrawlStatus, + crawlRenderModeUsesBrowser, + fetchBrowserCrawlStatus, +} from '@/lib/browserCrawlStatus'; import { buildInitialPipelineConfigState, validatePipelineRun, @@ -58,6 +63,9 @@ export interface PipelineContextValue { status: PipelineJobStatus | ''; backgroundMode: boolean; startUrl: string; + browserCrawlStatus: BrowserCrawlStatus | null; + browserCrawlChecking: boolean; + refreshBrowserCrawlStatus: () => Promise; setPresetId: (id: PipelinePresetId) => void; setCustomCommand: (value: string) => void; setField: (key: string, value: string | boolean) => void; @@ -112,8 +120,29 @@ export function PipelineProvider({ children }: { children: ReactNode }) { const [status, setStatus] = useState(''); const [backgroundMode, setBackgroundMode] = useState(false); const [configLoaded, setConfigLoaded] = useState(false); + const [browserCrawlStatus, setBrowserCrawlStatus] = useState(null); + const [browserCrawlChecking, setBrowserCrawlChecking] = useState(false); const pollStopRef = useRef<(() => void) | null>(null); + const refreshBrowserCrawlStatus = useCallback(async () => { + setBrowserCrawlChecking(true); + try { + const status = await fetchBrowserCrawlStatus(); + setBrowserCrawlStatus(status); + } finally { + setBrowserCrawlChecking(false); + } + }, []); + + const crawlRenderMode = String(configState.crawl_render_mode ?? 'static'); + useEffect(() => { + if (!crawlRenderModeUsesBrowser({ crawl_render_mode: crawlRenderMode })) { + setBrowserCrawlStatus(null); + return; + } + void refreshBrowserCrawlStatus(); + }, [crawlRenderMode, refreshBrowserCrawlStatus]); + useEffect(() => { savePipelineRunnerPrefs({ pythonExe, repoRoot }); }, [pythonExe, repoRoot]); @@ -351,7 +380,16 @@ export function PipelineProvider({ children }: { children: ReactNode }) { const run = useCallback(async () => { const command = effectiveCommand || null; - const validationErrors = validatePipelineRun({ state: configState, command }); + let browserStatus = browserCrawlStatus; + if (crawlRenderModeUsesBrowser(configState)) { + browserStatus = await fetchBrowserCrawlStatus(); + setBrowserCrawlStatus(browserStatus); + } + const validationErrors = validatePipelineRun({ + state: configState, + command, + browserStatus, + }); if (validationErrors.length > 0) { const message = validationErrors.join(' '); logPipelineFailure('Run validation failed', { command, errors: validationErrors }); @@ -403,6 +441,7 @@ export function PipelineProvider({ children }: { children: ReactNode }) { buildLlmPayload, pythonExe, repoRoot, + browserCrawlStatus, stopPoll, watchJob, ]); @@ -432,6 +471,9 @@ export function PipelineProvider({ children }: { children: ReactNode }) { status, backgroundMode, startUrl: String(configState.start_url ?? ''), + browserCrawlStatus, + browserCrawlChecking, + refreshBrowserCrawlStatus, setPresetId, setCustomCommand, setField, @@ -466,6 +508,9 @@ export function PipelineProvider({ children }: { children: ReactNode }) { log, status, backgroundMode, + browserCrawlStatus, + browserCrawlChecking, + refreshBrowserCrawlStatus, setLlmField, handleStartUrlChange, handlePresetChange, diff --git a/web/src/lib/appNav.ts b/web/src/lib/appNav.ts index 5aeb983f..66f22c24 100644 --- a/web/src/lib/appNav.ts +++ b/web/src/lib/appNav.ts @@ -2,6 +2,7 @@ import type { LucideIcon } from 'lucide-react'; import { AlertOctagon, ArrowLeftRight, + Link2, BarChart2, Cpu, FileDown, @@ -51,6 +52,7 @@ const VIEW_NAV: { id: ViewId; icon: LucideIcon }[] = [ { id: 'network', icon: Share2 }, { id: 'gallery', icon: Images }, { id: 'search-performance', icon: TrendingUp }, + { id: 'backlinks', icon: Link2 }, { id: 'traffic', icon: BarChart2 }, { id: 'keywords-explorer', icon: Key }, ]; diff --git a/web/src/lib/browserCrawlStatus.ts b/web/src/lib/browserCrawlStatus.ts new file mode 100644 index 00000000..d8173e76 --- /dev/null +++ b/web/src/lib/browserCrawlStatus.ts @@ -0,0 +1,35 @@ +import { apiUrl } from '@/lib/publicBase'; +import type { PipelineConfigState } from '@/types/api'; + +export interface BrowserCrawlStatus { + ok: boolean; + message?: string; +} + +export function crawlRenderModeUsesBrowser(state: PipelineConfigState): boolean { + const mode = String(state.crawl_render_mode ?? 'static').trim().toLowerCase(); + return mode === 'javascript' || mode === 'auto'; +} + +export async function fetchBrowserCrawlStatus(): Promise { + try { + const res = await fetch(apiUrl('/crawl/browser-status')); + const data = (await res.json().catch(() => ({}))) as BrowserCrawlStatus & { error?: string }; + if (!res.ok) { + return { + ok: false, + message: + data.message || + data.error || + 'JavaScript crawl requires Playwright and Chromium on this machine.', + }; + } + return { ok: Boolean(data.ok), message: data.message }; + } catch (e) { + const message = e instanceof Error ? e.message : String(e); + return { + ok: false, + message: `Could not check browser availability: ${message}`, + }; + } +} diff --git a/web/src/lib/googlePropertyEndpoints.ts b/web/src/lib/googlePropertyEndpoints.ts index 86ecebe4..3caea3ac 100644 --- a/web/src/lib/googlePropertyEndpoints.ts +++ b/web/src/lib/googlePropertyEndpoints.ts @@ -5,6 +5,8 @@ export interface GooglePropertyEndpoints { credentials: string; listProperties: string; test: string; + linksImport: string; + linksStatus: string; auth: (returnTo: string) => string; disconnect: string; isPerProperty: boolean; @@ -18,6 +20,8 @@ export function googlePropertyEndpoints(propertyId: number | null): GoogleProper credentials: apiUrl(`${base}/credentials`), listProperties: apiUrl(`${base}/properties`), test: apiUrl(`${base}/test`), + linksImport: apiUrl(`${base}/links/import`), + linksStatus: apiUrl(`${base}/links/status`), auth: (returnTo: string) => apiUrl( `/integrations/google/auth?propertyId=${propertyId}&returnTo=${encodeURIComponent(returnTo)}`, @@ -31,6 +35,8 @@ export function googlePropertyEndpoints(propertyId: number | null): GoogleProper credentials: apiUrl('/integrations/google/credentials'), listProperties: apiUrl('/integrations/google/properties'), test: apiUrl('/integrations/google/test'), + linksImport: '', + linksStatus: '', auth: (_returnTo: string) => apiUrl('/integrations/google/auth'), disconnect: apiUrl('/integrations/google/disconnect'), isPerProperty: false, diff --git a/web/src/lib/loadReportDb.ts b/web/src/lib/loadReportDb.ts index ce92957f..1a18c7d7 100644 --- a/web/src/lib/loadReportDb.ts +++ b/web/src/lib/loadReportDb.ts @@ -162,6 +162,42 @@ export async function readLatestKeywordPayload( } } +/** Latest gsc_links_data row (GSC Links CSV import). */ +export async function readLatestGscLinksPayload( + client: PoolClient, + propertyId: number | null = null, +): Promise | null> { + if (propertyId == null) return null; + try { + const { rows } = await client.query( + `SELECT data FROM gsc_links_data + WHERE property_id = $1 ORDER BY id DESC LIMIT 1`, + [propertyId], + ); + if (!rows.length) return null; + const raw = parseJsonField(rows[0].data); + if (!raw || typeof raw !== 'object') return null; + const sample = Array.isArray(raw.sample_links) ? raw.sample_links : []; + const latest = Array.isArray(raw.latest_links) ? raw.latest_links : []; + const cap = 2000; + let out = raw as Record; + if (sample.length + latest.length > cap) { + const sampleCap = Math.min(sample.length, cap); + const latestCap = Math.max(0, cap - sampleCap); + out = { + ...raw, + sample_links: sample.slice(0, sampleCap), + latest_links: latest.slice(0, latestCap), + sample_links_full_count: sample.length, + latest_links_full_count: latest.length, + }; + } + return out; + } catch { + return null; + } +} + async function lookupPropertyIdByDomain( client: PoolClient, domainRaw: string, @@ -226,6 +262,9 @@ export async function mergeSidecarPayloadData( const keywords = await readLatestKeywordPayload(client, propertyId); if (keywords) merged.keywords = keywords as ReportPayload['keywords']; + const gscLinks = await readLatestGscLinksPayload(client, propertyId); + if (gscLinks) merged.gsc_links = gscLinks as ReportPayload['gsc_links']; + if (scopedDomain) { merged = stripGoogleIfDomainMismatch(merged, scopedDomain); } diff --git a/web/src/lib/pipelineConfigSchema.test.ts b/web/src/lib/pipelineConfigSchema.test.ts index dc3246ab..b21dee03 100644 --- a/web/src/lib/pipelineConfigSchema.test.ts +++ b/web/src/lib/pipelineConfigSchema.test.ts @@ -1,7 +1,14 @@ import { describe, expect, it } from 'vitest'; import { readFileSync } from 'fs'; import { join } from 'path'; -import { ALL_SCHEMA_KEYS, INTERNAL_PIPELINE_KEYS } from '@/lib/pipelineConfigSchema'; +import { + ALL_SCHEMA_KEYS, + BROWSER_CRAWL_UNAVAILABLE_MSG, + INTERNAL_PIPELINE_KEYS, + getFieldByKey, + isPipelineFieldVisible, + validatePipelineRun, +} from '@/lib/pipelineConfigSchema'; function parseConfigKeys(raw: string): Set { const keys = new Set(); @@ -35,3 +42,69 @@ describe('pipelineConfigSchema', () => { } }); }); + +describe('isPipelineFieldVisible', () => { + const jsConcurrency = getFieldByKey('crawl_js_concurrency'); + const jsTimeout = getFieldByKey('crawl_js_timeout'); + const captureConsole = getFieldByKey('crawl_js_capture_console'); + + it('hides JS fields when crawl_render_mode is static', () => { + expect(jsConcurrency).toBeDefined(); + expect(isPipelineFieldVisible(jsConcurrency!, { crawl_render_mode: 'static' })).toBe(false); + expect(isPipelineFieldVisible(jsTimeout!, { crawl_render_mode: 'static' })).toBe(false); + expect(isPipelineFieldVisible(captureConsole!, { crawl_render_mode: 'static' })).toBe(false); + }); + + it('shows JS fields for javascript and auto modes', () => { + expect(isPipelineFieldVisible(jsConcurrency!, { crawl_render_mode: 'javascript' })).toBe(true); + expect(isPipelineFieldVisible(jsConcurrency!, { crawl_render_mode: 'auto' })).toBe(true); + expect(isPipelineFieldVisible(jsTimeout!, { crawl_render_mode: 'javascript' })).toBe(true); + expect(isPipelineFieldVisible(captureConsole!, { crawl_render_mode: 'javascript' })).toBe(true); + expect(isPipelineFieldVisible(captureConsole!, { crawl_render_mode: 'auto' })).toBe(true); + }); +}); + +describe('validatePipelineRun browser preflight', () => { + const baseState = { + start_url: 'https://example.com', + site_name: 'Example', + crawl_render_mode: 'javascript', + run_crawl: true, + }; + + it('blocks crawl when JS mode is selected and browser is unavailable', () => { + const errors = validatePipelineRun({ + state: baseState, + command: 'crawl', + browserStatus: { ok: false, message: 'missing chromium' }, + }); + expect(errors.some((e) => e.includes('missing chromium'))).toBe(true); + }); + + it('uses default message when browser status has no detail', () => { + const errors = validatePipelineRun({ + state: { ...baseState, crawl_render_mode: 'auto' }, + command: null, + browserStatus: { ok: false }, + }); + expect(errors).toContain(BROWSER_CRAWL_UNAVAILABLE_MSG); + }); + + it('does not require browser for static crawl mode', () => { + const errors = validatePipelineRun({ + state: { ...baseState, crawl_render_mode: 'static' }, + command: 'crawl', + browserStatus: { ok: false, message: 'missing chromium' }, + }); + expect(errors.some((e) => e.includes('missing chromium'))).toBe(false); + }); + + it('skips browser check when crawl is not part of the run', () => { + const errors = validatePipelineRun({ + state: { ...baseState, run_crawl: false }, + command: 'report', + browserStatus: { ok: false, message: 'missing chromium' }, + }); + expect(errors.some((e) => e.includes('missing chromium'))).toBe(false); + }); +}); diff --git a/web/src/lib/pipelineConfigSchema.ts b/web/src/lib/pipelineConfigSchema.ts index 3d2a642f..529c1589 100644 --- a/web/src/lib/pipelineConfigSchema.ts +++ b/web/src/lib/pipelineConfigSchema.ts @@ -14,8 +14,13 @@ * multiselect – checkbox group (stored as comma-separated values) * textarea – multi-line text input */ +import type { BrowserCrawlStatus } from '@/lib/browserCrawlStatus'; +import { crawlRenderModeUsesBrowser } from '@/lib/browserCrawlStatus'; import type { PipelineConfigState } from '@/types/api'; +export const BROWSER_CRAWL_UNAVAILABLE_MSG = + 'JavaScript crawl requires Playwright and Chromium. Install: pip install -r requirements-browser.txt. Chrome or Chromium must be on PATH or set CHROME_PATH.'; + export interface PipelineConfigField { key: string; label: string; @@ -27,8 +32,11 @@ export interface PipelineConfigField { span?: 1 | 2; unit?: string; required?: boolean; + visibleWhen?: { key: string; not?: readonly string[] }; } +const JS_FIELD_VISIBLE_WHEN = { key: 'crawl_render_mode', not: ['static'] as const }; + export interface PipelineConfigSection { id: string; label: string; @@ -79,6 +87,93 @@ export const PIPELINE_CONFIG_SECTIONS: PipelineConfigSection[] = [ defaultValue: '', help: 'Comma-separated URL substrings or patterns to exclude from crawling.', }, + { + key: 'crawl_render_mode', + label: 'Crawl rendering', + type: 'singleselect', + defaultValue: 'static', + options: [ + { value: 'static', label: 'Static HTML (fast)' }, + { value: 'javascript', label: 'JavaScript rendering (slow)' }, + { value: 'auto', label: 'Auto (static, JS when needed)' }, + ], + help: 'JavaScript mode uses headless Chromium for React, Vue, Next.js, and Shopify themes. Roughly 10–20× slower than static.', + }, + { + key: 'crawl_js_concurrency', + label: 'JS parallel pages', + type: 'number', + defaultValue: '3', + help: 'Parallel browser page slots when JavaScript rendering is enabled. HTTP concurrency is ignored in JS mode.', + visibleWhen: JS_FIELD_VISIBLE_WHEN, + }, + { + key: 'crawl_js_timeout', + label: 'JS navigation timeout (s)', + type: 'number', + defaultValue: '30', + visibleWhen: JS_FIELD_VISIBLE_WHEN, + }, + { + key: 'crawl_js_wait_until', + label: 'JS wait until', + type: 'select', + defaultValue: 'domcontentloaded', + visibleWhen: JS_FIELD_VISIBLE_WHEN, + options: [ + { value: 'domcontentloaded', label: 'DOM content loaded' }, + { value: 'load', label: 'Full load' }, + { value: 'commit', label: 'First commit' }, + ], + }, + { + key: 'crawl_js_extra_wait_ms', + label: 'JS hydration wait (ms)', + type: 'number', + defaultValue: '1500', + help: 'Extra wait after load for client-side hydration before capturing HTML.', + visibleWhen: JS_FIELD_VISIBLE_WHEN, + }, + { + key: 'crawl_js_block_resources', + label: 'Block images/fonts in JS crawl', + type: 'bool', + defaultValue: true, + help: 'Blocks images, fonts, and media during JS crawl for faster rendering.', + visibleWhen: JS_FIELD_VISIBLE_WHEN, + }, + { + key: 'crawl_js_capture_console', + label: 'Capture browser console during JS crawl', + type: 'bool', + defaultValue: true, + help: 'Records console.error and console.warning messages while pages load in headless Chromium.', + visibleWhen: JS_FIELD_VISIBLE_WHEN, + }, + { + key: 'crawl_js_console_levels', + label: 'Console levels to capture', + type: 'text', + defaultValue: 'error,warning', + help: 'Comma-separated console levels (e.g. error, warning, info).', + visibleWhen: JS_FIELD_VISIBLE_WHEN, + }, + { + key: 'crawl_js_capture_failed_requests', + label: 'Capture failed network requests', + type: 'bool', + defaultValue: false, + help: 'Records failed XHR/fetch during JS crawl (can be noisy).', + visibleWhen: JS_FIELD_VISIBLE_WHEN, + }, + { + key: 'crawl_js_console_max_per_page', + label: 'Max console entries per page', + type: 'number', + defaultValue: '20', + help: 'Cap on console messages, page errors, and failed requests stored per URL.', + visibleWhen: JS_FIELD_VISIBLE_WHEN, + }, ], }, { @@ -370,6 +465,21 @@ export function getFieldByKey(key: string): PipelineConfigField | undefined { return undefined; } +/** Whether a field should be shown given current config state (visibleWhen rules). */ +export function isPipelineFieldVisible( + field: PipelineConfigField, + state: Record, +): boolean { + const rule = field.visibleWhen; + if (!rule) return true; + const raw = state[rule.key]; + const value = raw === undefined || raw === null ? '' : String(raw).trim().toLowerCase(); + if (rule.not?.length) { + return !rule.not.some((excluded) => excluded.toLowerCase() === value); + } + return true; +} + function isTruthyPipelineBool( value: string | boolean | undefined, defaultWhenUnset = false, @@ -382,6 +492,14 @@ function isTruthyPipelineBool( export interface ValidatePipelineRunInput { state: PipelineConfigState; command?: string | null; + browserStatus?: BrowserCrawlStatus | null; +} + +function runIncludesCrawl(state: PipelineConfigState, command: string | null | undefined): boolean { + if (command === 'crawl') return true; + if (command === 'report' || command === 'keywords' || command === 'lighthouse') return false; + if (!command) return isTruthyPipelineBool(state?.run_crawl, true); + return false; } /** Validate schema fields marked `required`. */ @@ -404,7 +522,11 @@ export function validateRequiredPipelineFields(state: PipelineConfigState): stri * Validate config before starting a pipeline job. * @returns error messages (empty if ok) */ -export function validatePipelineRun({ state, command = null }: ValidatePipelineRunInput): string[] { +export function validatePipelineRun({ + state, + command = null, + browserStatus = null, +}: ValidatePipelineRunInput): string[] { const startUrl = String(state?.start_url ?? '').trim(); const lighthouseUrl = String(state?.lighthouse_url ?? '').trim(); const errors: string[] = []; @@ -433,6 +555,14 @@ export function validatePipelineRun({ state, command = null }: ValidatePipelineR if (needsLighthouseUrl && !lighthouseUrl && !startUrl) { errors.push('Lighthouse URL or Start URL is required for single-URL Lighthouse.'); } + if ( + runIncludesCrawl(state, command) && + crawlRenderModeUsesBrowser(state) && + browserStatus != null && + !browserStatus.ok + ) { + errors.push(browserStatus.message?.trim() || BROWSER_CRAWL_UNAVAILABLE_MSG); + } return errors; } diff --git a/web/src/routes.ts b/web/src/routes.ts index 07f10119..ffb5e6c4 100644 --- a/web/src/routes.ts +++ b/web/src/routes.ts @@ -15,6 +15,7 @@ export type ViewId = | 'network' | 'gallery' | 'search-performance' + | 'backlinks' | 'traffic' | 'keywords-explorer' | 'compare' @@ -36,6 +37,7 @@ const VIEW_IDS = new Set([ 'network', 'gallery', 'search-performance', + 'backlinks', 'traffic', 'keywords-explorer', 'compare', diff --git a/web/src/server/browserStatusRoute.test.ts b/web/src/server/browserStatusRoute.test.ts new file mode 100644 index 00000000..0466dd19 --- /dev/null +++ b/web/src/server/browserStatusRoute.test.ts @@ -0,0 +1,61 @@ +import { describe, expect, it, vi, beforeEach } from 'vitest'; +import { NextRequest } from 'next/server'; +import { EventEmitter } from 'events'; + +const spawnMock = vi.fn(); + +vi.mock('child_process', () => ({ + spawn: (...args: unknown[]) => spawnMock(...args), +})); + +function makeChildProcess(stdout: string, exitCode: number) { + const proc = new EventEmitter() as EventEmitter & { + stdout: EventEmitter; + stderr: EventEmitter; + kill: ReturnType; + }; + proc.stdout = new EventEmitter(); + proc.stderr = new EventEmitter(); + proc.kill = vi.fn(); + setTimeout(() => { + if (stdout) proc.stdout.emit('data', Buffer.from(stdout)); + proc.emit('close', exitCode); + }, 10); + return proc; +} + +function localRequest(path: string): NextRequest { + return new NextRequest(`http://localhost:3000${path}`, { headers: { host: 'localhost:3000' } }); +} + +describe('GET /api/crawl/browser-status', () => { + beforeEach(() => { + spawnMock.mockReset(); + vi.resetModules(); + }); + + it('returns ok when Python reports browser available', async () => { + spawnMock.mockImplementation(() => makeChildProcess('{"ok": true}\n', 0)); + const { GET } = await import('../../app/api/crawl/browser-status/route'); + const res = await GET(localRequest('/api/crawl/browser-status')); + expect(res.status).toBe(200); + const body = await res.json(); + expect(body.ok).toBe(true); + }); + + it('returns not ok when Python exits non-zero', async () => { + spawnMock.mockImplementation(() => makeChildProcess('playwright missing\n', 1)); + const { GET } = await import('../../app/api/crawl/browser-status/route'); + const res = await GET(localRequest('/api/crawl/browser-status')); + expect(res.status).toBe(200); + const body = await res.json(); + expect(body.ok).toBe(false); + expect(String(body.message || '')).toMatch(/playwright|JavaScript crawl/i); + }); + + it('rejects non-local hosts', async () => { + const { GET } = await import('../../app/api/crawl/browser-status/route'); + const res = await GET(new NextRequest('http://192.168.1.5:3000/api/crawl/browser-status')); + expect(res.status).toBe(403); + }); +}); diff --git a/web/src/server/gscLinksImportRoute.test.ts b/web/src/server/gscLinksImportRoute.test.ts new file mode 100644 index 00000000..45f88561 --- /dev/null +++ b/web/src/server/gscLinksImportRoute.test.ts @@ -0,0 +1,13 @@ +import { describe, expect, it } from 'vitest'; + +describe('GSC links import validation', () => { + it('rejects empty fileContent', () => { + const fileContent: string = ''; + expect(!fileContent || !fileContent.trim()).toBe(true); + }); + + it('accepts non-empty CSV content', () => { + const fileContent = 'Site,Links,Target pages\nexample.com,1,1\n'; + expect(fileContent.trim().length).toBeGreaterThan(0); + }); +}); diff --git a/web/src/server/parsePythonJsonStdout.test.ts b/web/src/server/parsePythonJsonStdout.test.ts new file mode 100644 index 00000000..552a6024 --- /dev/null +++ b/web/src/server/parsePythonJsonStdout.test.ts @@ -0,0 +1,33 @@ +import { describe, expect, it } from 'vitest'; +import { parsePythonJsonStdout } from '@/server/resolvePython'; + +describe('parsePythonJsonStdout', () => { + it('parses a single JSON line', () => { + expect(parsePythonJsonStdout('{"ok": true, "count": 3}\n')).toEqual({ ok: true, count: 3 }); + }); + + it('parses the last JSON line when config logs precede output', () => { + const stdout = [ + '[Config] Loaded from pipeline_config table (PostgreSQL)', + '{"ok": true, "imported_at": "2026-06-05T21:21:03Z", "row_counts": {"top_linking_text": 104}}', + ].join('\n'); + expect(parsePythonJsonStdout(stdout)).toEqual({ + ok: true, + imported_at: '2026-06-05T21:21:03Z', + row_counts: { top_linking_text: 104 }, + }); + }); + + it('parses JSON suffix on the same line as log text', () => { + const stdout = + '[Config] Loaded from pipeline_config table (PostgreSQL) {"ok": true, "last_export_type": "top_linking_text"}'; + expect(parsePythonJsonStdout(stdout)).toEqual({ + ok: true, + last_export_type: 'top_linking_text', + }); + }); + + it('returns null when stdout has no JSON object', () => { + expect(parsePythonJsonStdout('[Config] Loaded from PostgreSQL')).toBeNull(); + }); +}); diff --git a/web/src/server/pipelineSpawnEnv.ts b/web/src/server/pipelineSpawnEnv.ts index 5fbf9b6a..61740acc 100644 --- a/web/src/server/pipelineSpawnEnv.ts +++ b/web/src/server/pipelineSpawnEnv.ts @@ -15,6 +15,8 @@ export function getPipelineSpawnEnv( ...process.env, WEBSITE_PROFILING_ROOT: repoRoot, DATA_DIR: getDataDir(), + // Required for `python -c "from website_profiling ..."` (browser-status, exports, etc.). + PYTHONPATH: path.join(repoRoot, 'src'), }; if (propertyId != null && Number.isFinite(propertyId)) { env.WP_PROPERTY_ID = String(propertyId); diff --git a/web/src/server/resolvePython.ts b/web/src/server/resolvePython.ts index 38bdd9e9..173a3132 100644 --- a/web/src/server/resolvePython.ts +++ b/web/src/server/resolvePython.ts @@ -74,6 +74,34 @@ export function resolvePythonExecutable( return process.platform === 'win32' ? 'python' : 'python3'; } +/** + * Parse JSON from Python CLI stdout when log lines may precede the payload. + * Tries each non-empty line from the bottom; also handles `{...}` suffix on a line. + */ +export function parsePythonJsonStdout(stdout: string): Record | null { + const lines = stdout.trim().split('\n').filter(Boolean); + for (let i = lines.length - 1; i >= 0; i--) { + const line = lines[i].trim(); + const candidates = [line]; + const jsonStart = line.lastIndexOf('{'); + if (jsonStart > 0) { + candidates.push(line.slice(jsonStart)); + } + for (const candidate of candidates) { + if (!candidate.startsWith('{') && !candidate.startsWith('[')) continue; + try { + const parsed = JSON.parse(candidate) as unknown; + if (parsed !== null && typeof parsed === 'object' && !Array.isArray(parsed)) { + return parsed as Record; + } + } catch { + /* try next candidate */ + } + } + } + return null; +} + export function formatPythonSpawnError(err: Error, resolvedPython: string, repoRoot: string): string { if (!/ENOENT/i.test(err.message)) { return err.message; diff --git a/web/src/strings.json b/web/src/strings.json index 0589f3c0..92411376 100644 --- a/web/src/strings.json +++ b/web/src/strings.json @@ -94,6 +94,10 @@ "label": "Search Console", "section": "Search & analytics" }, + "backlinks": { + "label": "Backlinks", + "section": "Search & analytics" + }, "traffic": { "label": "Analytics (GA4)", "section": "Search & analytics" @@ -160,6 +164,9 @@ "loadingSettings": "Loading settings…", "settingsTitle": "Audit settings", "settingsSubtitle": "Fine-tune crawl, audit report, Lighthouse, keywords, and AI options.", + "browserCrawlBannerTitle": "Headless browser not available", + "browserCrawlBannerHint": "JavaScript and Auto crawl modes need Playwright Python packages and Chromium. Run: pip install -r requirements-browser.txt. Ensure Chrome or Chromium is on PATH or set CHROME_PATH.", + "browserCrawlChecking": "Checking browser availability…", "saveSettings": "Save settings", "saveAndClose": "Save & close", "saving": "Saving…", @@ -192,6 +199,18 @@ "googlePropertyContextEmail": "Signed in as {email}", "googlePropertyGscGa4": "GSC: {gsc} · GA4: {ga4}", "googlePropertySyncSaving": "Updating Site URL…", + "gscLinksTitle": "Search Console Links (backlinks)", + "gscLinksDescription": "Import CSV exports from Google Search Console → Links. This is Google's sample of external sites linking to you — not crawl inlinks or a full web index.", + "gscLinksHelpUrl": "https://support.google.com/webmasters/answer/9049606", + "gscLinksHelpLabel": "GSC Links report help", + "gscLinksUploadLabel": "Upload GSC Links CSV", + "gscLinksUploadHint": "Export Top linking sites, Top linked pages, Top linking text, or Latest/More sample links from Search Console, then upload each file here.", + "gscLinksUploading": "Importing…", + "gscLinksUploadSuccess": "Import complete.", + "gscLinksUploadFailed": "Import failed: {message}", + "gscLinksLastImport": "Last import: {date}", + "gscLinksNoData": "No GSC Links import yet for this property.", + "gscLinksRowCounts": "{domains} referring domains · {pages} linked pages · {sample} sample links", "contentAiHint": "AI insights run when the audit report is built (optional).", "settingsSectionTabsLabel": "Settings sections", "settingsTabIntegrations": "Integrations", @@ -1026,6 +1045,57 @@ }, "connectIntegrations": "Connect Google" }, + "backlinks": { + "title": "Backlinks", + "subtitle": "External links to your site from Google Search Console Links import", + "disclaimer": "Google's curated sample of backlinks it knows about — not Ahrefs-style full index, not crawl inlinks, and nofollow is not split out. Data may lag indexing by weeks.", + "fetchedAt": "Imported {date}", + "emptyTitle": "No backlink data yet", + "emptyBody": "Export CSV files from Google Search Console → Links and import them in Integrations. OAuth is optional for CSV import; data is scoped to the selected property.", + "emptyIntegrationsHint": "Open Integrations from the top-right header → Search Console Links (backlinks).", + "tabs": { + "overview": "Overview", + "domains": "Referring domains", + "pages": "Linked pages", + "anchors": "Anchor text", + "sample": "Sample links" + }, + "kpi": { + "referringDomains": "Referring domains", + "linkedPages": "Top linked pages", + "sampleLinks": "Sample links", + "latestLinks": "Latest links" + }, + "table": { + "site": "Site", + "targetPage": "Target page", + "sourcePage": "Source page", + "anchorText": "Anchor text", + "links": "Links", + "targetPages": "Target pages", + "linkingSites": "Linking sites", + "discovered": "Discovered", + "inCrawl": "In crawl", + "yes": "Yes", + "no": "No", + "searchDomains": "Search domains…", + "searchPages": "Search target pages…", + "searchAnchors": "Search anchor text…", + "searchLinks": "Search URLs…", + "exportCsv": "Export CSV", + "noData": "No data for this tab", + "showingSlice": "Showing {from}–{to} of {total}", + "pageOf": "Page", + "of": "of", + "previous": "Previous", + "next": "Next", + "rowsPerPage": "{n} per page" + }, + "overview": { + "topDomainsTitle": "Top referring domains", + "topPagesTitle": "Most linked pages on your site" + } + }, "traffic": { "title": "Traffic & Engagement", "subtitle": "Google Analytics 4 sessions and engagement for {range}", @@ -1427,7 +1497,13 @@ "pagesLine": "{pages} URLs crawled (limit {max}).", "limitedNote": "The crawl limit was reached; this is not a full-site audit unless the limit covers your site.", "robotsLine": "{count} URLs blocked by robots.txt.", - "staticHtmlNote": "Static HTML only — links rendered only by JavaScript may be missing." + "staticHtmlNote": "Static HTML only — links rendered only by JavaScript may be missing.", + "javascriptNote": "JavaScript rendering enabled — pages were loaded in headless Chromium before analysis.", + "autoNote": "Auto rendering — static HTML first; headless Chromium used when a page looks like a JavaScript app shell.", + "fetchMethodMixLine": "{staticCount} static, {renderedCount} JavaScript-rendered.", + "jsConcurrencyLine": "JS parallel pages: {count}.", + "browserConsoleErrorsLine": "{pages} page(s) logged {errors} console error(s) during JavaScript rendering.", + "browserPageErrorsLine": "{pages} page(s) had uncaught JavaScript errors during rendering." }, "googleStaleWarning": "Search Console & Analytics data is older than 7 days. Refresh from Integrations.", "googlePartialWarning": "Search Console or Analytics data is missing from the last sync.", @@ -1869,6 +1945,19 @@ "severityLow": "Low", "noMatchingWarnings": "No matching warnings for this page.", "resources": "Resources", + "browserConsoleTitle": "Browser console (JS crawl)", + "browserConsoleClean": "No console errors or uncaught exceptions were captured during JavaScript rendering.", + "browserConsoleMessages": "Console messages", + "browserUncaughtExceptions": "Uncaught exceptions", + "browserFailedRequests": "Failed network requests", + "browserThLevel": "Level", + "browserThMessage": "Message", + "browserThLocation": "Location", + "browserThUrl": "URL", + "browserThMethod": "Method", + "browserThFailure": "Failure", + "browserShowStack": "Show stack", + "browserHideStack": "Hide stack", "resourceChartLabels": [ "Internal links", "External links", diff --git a/web/src/types/components.ts b/web/src/types/components.ts index f89d63c6..d599a828 100644 --- a/web/src/types/components.ts +++ b/web/src/types/components.ts @@ -78,6 +78,40 @@ export interface GscDailyRow { [key: string]: unknown; } +export interface GscTopLinkingSiteRow { + site?: string; + link_count?: number; + target_page_count?: number; + [key: string]: unknown; +} + +export interface GscTopLinkedPageRow { + target_page?: string; + link_count?: number; + linking_site_count?: number; + target_in_crawl?: boolean; + crawl_url?: string; + [key: string]: unknown; +} + +export interface GscTopLinkingTextRow { + anchor_text?: string; + link_count?: number; + [key: string]: unknown; +} + +export interface GscSampleLinkRow { + source_page?: string; + target_page?: string; + target_url_on_linking_page?: string; + anchor_text?: string; + linking_site?: string; + discovered_at?: string; + target_in_crawl?: boolean; + crawl_url?: string; + [key: string]: unknown; +} + export interface Ga4PageRow { path: string; sessions?: number; diff --git a/web/src/types/index.ts b/web/src/types/index.ts index f3c8f0aa..8123c300 100644 --- a/web/src/types/index.ts +++ b/web/src/types/index.ts @@ -128,6 +128,10 @@ export type { GscQueryRow, GscPageRow, GscDailyRow, + GscTopLinkingSiteRow, + GscTopLinkedPageRow, + GscTopLinkingTextRow, + GscSampleLinkRow, Ga4PageRow, Ga4DailyRow, Ga4ChannelRow, diff --git a/web/src/types/report.ts b/web/src/types/report.ts index 9619efa4..ae878fa8 100644 --- a/web/src/types/report.ts +++ b/web/src/types/report.ts @@ -5,6 +5,10 @@ import type { GscDailyRow, GscPageRow, GscQueryRow, + GscSampleLinkRow, + GscTopLinkedPageRow, + GscTopLinkingSiteRow, + GscTopLinkingTextRow, KeywordRow, UrlJoinData, } from '@/types/components'; @@ -291,6 +295,22 @@ export interface KeywordReportData { [key: string]: unknown; } +export interface GscLinksReportData { + imported_at?: string; + source?: 'gsc_links_csv'; + export_types?: string[]; + row_counts?: Record; + top_linking_sites?: GscTopLinkingSiteRow[]; + top_linked_pages?: GscTopLinkedPageRow[]; + top_linking_text?: GscTopLinkingTextRow[]; + sample_links?: GscSampleLinkRow[]; + latest_links?: GscSampleLinkRow[]; + sample_links_full_count?: number; + latest_links_full_count?: number; + errors?: string[]; + [key: string]: unknown; +} + export interface SiteLevelChecks { robots_present?: boolean; sitemap_present?: boolean; @@ -357,10 +377,18 @@ export interface ReportPayload { max_pages_configured?: number; robots_blocked_count?: number; static_html_only?: boolean; + render_mode?: string; + js_concurrency?: number | null; + pages_static?: number; + pages_rendered?: number; crawl_limited?: boolean; + browser_diagnostics?: BrowserDiagnosticsAggregate; }; google_fetched_at?: string; google_date_range_days?: number; + gsc_links_imported_at?: string; + gsc_links_referring_domains?: number; + gsc_links_sample_count?: number; llm?: { model?: string; prompt_version?: string; generated_at?: string }; }; links?: ReportLink[]; @@ -370,6 +398,7 @@ export interface ReportPayload { url_fingerprints?: UrlFingerprint[]; google?: GoogleReportData; keywords?: KeywordReportData; + gsc_links?: GscLinksReportData; lighthouse_summary?: LighthousePageSummary; lighthouse_diagnostics?: LighthouseDiagnostic[]; lighthouse_human_summary?: string; @@ -428,6 +457,9 @@ export interface ReportLink { seo_score?: number; }; }; + console_error_count?: number; + page_error_count?: number; + has_browser_errors?: boolean; } export interface ReportTopPage { @@ -634,6 +666,52 @@ export interface PageWarning { [key: string]: unknown; } +export interface BrowserConsoleMessage { + level?: string; + text?: string; + source_url?: string; + line?: number; +} + +export interface BrowserPageError { + message?: string; + stack?: string; +} + +export interface BrowserFailedRequest { + url?: string; + method?: string; + failure?: string; +} + +export interface BrowserDiagnosticsSummary { + console_error_count?: number; + console_warning_count?: number; + page_error_count?: number; + failed_request_count?: number; +} + +export interface BrowserDiagnostics { + console?: BrowserConsoleMessage[]; + page_errors?: BrowserPageError[]; + failed_requests?: BrowserFailedRequest[]; + summary?: BrowserDiagnosticsSummary; +} + +export interface TopConsoleMessage { + text?: string; + count?: number; + sample_urls?: string[]; +} + +export interface BrowserDiagnosticsAggregate { + pages_with_console_errors?: number; + pages_with_page_errors?: number; + total_console_errors?: number; + total_page_errors?: number; + top_console_messages?: TopConsoleMessage[]; +} + export interface PageAnalysis { internal_link_count?: number; external_link_count?: number; @@ -645,6 +723,7 @@ export interface PageAnalysis { stylesheet_urls?: string[]; script_urls?: string[]; image_urls?: string[]; + browser?: BrowserDiagnostics; signals?: { nlp_entities?: NlpSignals; language?: string }; [key: string]: unknown; } diff --git a/web/src/views/Backlinks.tsx b/web/src/views/Backlinks.tsx new file mode 100644 index 00000000..f4326965 --- /dev/null +++ b/web/src/views/Backlinks.tsx @@ -0,0 +1,381 @@ +'use client'; + +import type { ReactNode } from 'react'; +import { useMemo, useState } from 'react'; +import { Link2, Settings2 } from 'lucide-react'; +import { useSearchParams } from 'next/navigation'; +import { useReport } from '../context/useReport'; +import { strings, format } from '../lib/strings'; +import { PageLayout } from '../components'; +import SortablePaginatedTable from '../components/google/SortablePaginatedTable'; +import GoogleTableToolbar from '../components/google/GoogleTableToolbar'; +import GscLinksSummaryCards from '../components/backlinks/GscLinksSummaryCards'; +import { + buildAnchorExportColumns, + buildDomainExportColumns, + buildLinkedPageExportColumns, + buildSampleLinkExportColumns, + combinedSampleLinks, + exportCsv, + filterBySearch, +} from '../components/backlinks/backlinksTableUtils'; +import { buildLinksInspectHref } from '../lib/reportNav'; +import type { TableColumn } from '@/types/components'; +import type { ViewProps } from '@/types'; + +const TABS = ['overview', 'domains', 'pages', 'anchors', 'sample'] as const; +type BacklinksTabId = (typeof TABS)[number]; + +export default function Backlinks(_props: ViewProps) { + const vb = strings.views.backlinks; + const searchParams = useSearchParams(); + const { data } = useReport(); + const gscLinks = data?.gsc_links; + + const paginationLabels = { + showingSlice: vb.table.showingSlice, + pageOf: vb.table.pageOf, + of: vb.table.of, + previous: vb.table.previous, + next: vb.table.next, + rowsPerPage: vb.table.rowsPerPage, + }; + + const [activeTab, setActiveTab] = useState('overview'); + const [domainSearch, setDomainSearch] = useState(''); + const [pageSearch, setPageSearch] = useState(''); + const [anchorSearch, setAnchorSearch] = useState(''); + const [sampleSearch, setSampleSearch] = useState(''); + + const filteredDomains = useMemo( + () => filterBySearch(gscLinks?.top_linking_sites ?? [], domainSearch, 'site'), + [gscLinks?.top_linking_sites, domainSearch], + ); + const filteredPages = useMemo( + () => filterBySearch(gscLinks?.top_linked_pages ?? [], pageSearch, 'target_page'), + [gscLinks?.top_linked_pages, pageSearch], + ); + const filteredAnchors = useMemo( + () => filterBySearch(gscLinks?.top_linking_text ?? [], anchorSearch, 'anchor_text'), + [gscLinks?.top_linking_text, anchorSearch], + ); + const allSample = useMemo(() => combinedSampleLinks(gscLinks), [gscLinks]); + const filteredSample = useMemo(() => { + const q = sampleSearch.trim().toLowerCase(); + if (!q) return allSample; + return allSample.filter((row) => { + const hay = [ + row.source_page, + row.target_page, + row.anchor_text, + row.linking_site, + ] + .filter(Boolean) + .join(' ') + .toLowerCase(); + return hay.includes(q); + }); + }, [allSample, sampleSearch]); + + const headerMeta: ReactNode = gscLinks?.imported_at ? ( + + {' '} + · {format(vb.fetchedAt, { date: new Date(String(gscLinks.imported_at)).toLocaleDateString() })} + + ) : null; + + const domainColumns = useMemo( + (): TableColumn[] => [ + { key: 'site', label: vb.table.site, render: (v) => {String(v ?? '')} }, + { + key: 'link_count', + label: vb.table.links, + render: (v) => {Number(v ?? 0).toLocaleString()}, + }, + { + key: 'target_page_count', + label: vb.table.targetPages, + render: (v) => {Number(v ?? 0).toLocaleString()}, + }, + ], + [vb.table], + ); + + const pageColumns = useMemo( + (): TableColumn[] => [ + { + key: 'target_page', + label: vb.table.targetPage, + render: (v, row) => { + const url = String(v ?? ''); + const inCrawl = row?.target_in_crawl === true; + const inspectHref = inCrawl + ? buildLinksInspectHref(String(row?.crawl_url || url), searchParams) + : null; + return inspectHref ? ( + + {url} + + ) : ( + {url} + ); + }, + }, + { + key: 'link_count', + label: vb.table.links, + render: (v) => {Number(v ?? 0).toLocaleString()}, + }, + { + key: 'linking_site_count', + label: vb.table.linkingSites, + render: (v) => {Number(v ?? 0).toLocaleString()}, + }, + ], + [vb.table, searchParams], + ); + + const anchorColumns = useMemo( + (): TableColumn[] => [ + { + key: 'anchor_text', + label: vb.table.anchorText, + render: (v) => ( + {String(v ?? '').trim() || '—'} + ), + }, + { + key: 'link_count', + label: vb.table.links, + render: (v) => {Number(v ?? 0).toLocaleString()}, + }, + ], + [vb.table], + ); + + const sampleColumns = useMemo( + (): TableColumn[] => [ + { + key: 'source_page', + label: vb.table.sourcePage, + render: (v) => ( + + {String(v ?? '')} + + ), + }, + { + key: 'target_page', + label: vb.table.targetPage, + render: (v, row) => { + const url = String(v ?? ''); + const inspectHref = + row?.target_in_crawl === true + ? buildLinksInspectHref(String(row?.crawl_url || url), searchParams) + : null; + if (inspectHref) { + return ( + + {url} + + ); + } + return {url}; + }, + }, + { + key: 'anchor_text', + label: vb.table.anchorText, + render: (v) => {String(v ?? '').trim() || '—'}, + }, + { + key: 'discovered_at', + label: vb.table.discovered, + render: (v) => {String(v ?? '—')}, + }, + ], + [vb.table, searchParams], + ); + + if (!gscLinks?.export_types?.length) { + return ( + +
+ +

{vb.emptyTitle}

+

{vb.emptyBody}

+

+ + {vb.emptyIntegrationsHint} +

+
+ + ); + } + + const tabLabels = vb.tabs as Record; + + return ( + +
+

+ + {vb.title} +

+

+ {vb.subtitle} + {headerMeta} +

+

{vb.disclaimer}

+
+ + + +
+
+ {TABS.map((id) => ( + + ))} +
+
+ + {activeTab === 'overview' && ( +
+
+

{vb.overview.topDomainsTitle}

+ []} + columns={domainColumns} + emptyMessage={vb.table.noData} + paginationLabels={paginationLabels} + /> +
+
+

{vb.overview.topPagesTitle}

+ []} + columns={pageColumns} + emptyMessage={vb.table.noData} + paginationLabels={paginationLabels} + /> +
+
+ )} + + {activeTab === 'domains' && ( + <> + + exportCsv( + filteredDomains as Record[], + buildDomainExportColumns(vb.table), + 'gsc-referring-domains.csv', + ) + } + exportLabel={vb.table.exportCsv} + /> + []} + columns={domainColumns} + emptyMessage={vb.table.noData} + paginationLabels={paginationLabels} + /> + + )} + + {activeTab === 'pages' && ( + <> + + exportCsv( + filteredPages as Record[], + buildLinkedPageExportColumns(vb.table), + 'gsc-linked-pages.csv', + ) + } + exportLabel={vb.table.exportCsv} + /> + []} + columns={pageColumns} + emptyMessage={vb.table.noData} + paginationLabels={paginationLabels} + /> + + )} + + {activeTab === 'anchors' && ( + <> + + exportCsv( + filteredAnchors as Record[], + buildAnchorExportColumns(vb.table), + 'gsc-linking-text.csv', + ) + } + exportLabel={vb.table.exportCsv} + /> + []} + columns={anchorColumns} + emptyMessage={vb.table.noData} + paginationLabels={paginationLabels} + /> + + )} + + {activeTab === 'sample' && ( + <> + + exportCsv( + filteredSample as Record[], + buildSampleLinkExportColumns(vb.table), + 'gsc-sample-links.csv', + ) + } + exportLabel={vb.table.exportCsv} + /> + []} + columns={sampleColumns} + emptyMessage={vb.table.noData} + paginationLabels={paginationLabels} + /> + + )} + + ); +}