diff --git a/AGENT.md b/AGENT.md index ec06cbe5..a1375318 100644 --- a/AGENT.md +++ b/AGENT.md @@ -1,8 +1,10 @@ # Agent instructions — Site Audit (WebsiteProfiling) +Developer reference for agents and contributors. User-facing overview: [README.md](README.md). Full doc index: [docs/README.md](docs/README.md). + **What it is:** `python -m src` from repo root (`src/__main__.py` -> package **`website_profiling`**). Config: stored in **PostgreSQL** (`pipeline_config` table, `key/value/is_unknown/updated_at`). A shadow **`pipeline-config.txt`** is auto-written to `DATA_DIR` on every Save/Run. CLI loads DB first (`DATABASE_URL`), then shadow file; `--config` overrides with a file. Reference keys: `input.txt.example` and `pipeline-config.example.txt` (not auto-loaded). -**LLM / AI:** Settings live in **`llm_config`** table in PostgreSQL. Configure only via web UI **AI** tab (`GET/PUT /api/llm-config`, localhost). Never in `pipeline-config.txt` or `--config`. +**LLM / AI:** Settings live in **`llm_config`** table in PostgreSQL. Providers: OpenAI, Google Gemini, Anthropic, Ollama (`web/src/lib/llmConfigSchema.ts`). Configure only via web UI **AI** tab (`GET/PUT /api/llm-config`, localhost). Never in `pipeline-config.txt` or `--config`. **Frontend:** **`web/`** (Next.js) -- server reads PostgreSQL via `/api/report/*`. @@ -12,7 +14,7 @@ - `web/app/` -- routes; `web/src/` -- React; pipeline: `PipelineRunnerFab`, `server/pipelineJobs.ts`, `server/pipelineConfig.ts`, `server/llmConfig.ts`, `server/db.ts` - `alembic/` -- schema migrations -**Local dev:** `./local-run` (Postgres in Docker `wp-pg`, Next.js on host). See `scripts/local-run.sh`. **Local tests (CI parity):** `./local-test` runs **three** Python coverage gates (core 100%, reporting 100%, tools 100%); `./local-test browser` for `@pytest.mark.browser` integration tests — see `scripts/local-test.sh`. Mocked browser unit tests: `tests/test_browser_fetcher_unit.py`. +**Local dev:** `./local-run` (Postgres in Docker `wp-pg`, Next.js on host; default `DATABASE_URL`: `postgres://postgres:dev@127.0.0.1:5432/website_profiling`). See `scripts/local-run.sh`. **Local tests:** `./local-test` runs **three** Python coverage gates (core 100%, reporting 100%, tools 100%) plus web checks — mirrors CI **python** and **web** jobs; Docker CI is separate (see `.github/workflows/ci.yml`). `./local-test browser` for `@pytest.mark.browser` integration tests — see `scripts/local-test.sh`. Mocked browser unit tests: `tests/test_browser_fetcher_unit.py`. **JavaScript crawl (optional):** Config keys `crawl_render_mode` (`static` | `javascript` | `auto`) and `crawl_js_*` in pipeline config / `pipelineConfigSchema.ts`. JS/auto crawls can capture browser console errors and uncaught exceptions (`crawl_js_capture_console`, stored under `page_analysis.browser`). **Auto mode** uses static-first fetch, pre-parse SPA heuristics (`needs_js_render`), then post-parse low-outlink fallback (`needs_js_render_after_parse`) in `crawler.py`. **Preflight:** `GET /api/crawl/browser-status` (localhost) spawns Python `browser_status()`; Run audit settings/run validation calls it when render mode is `javascript` or `auto`. Browser deps: Playwright from `requirements.txt` (installed by `./local-run setup` and `./local-test`). Runtime needs Chromium on `PATH` or `CHROME_PATH` (Docker sets `CHROME_PATH=/usr/bin/chromium`). Integration tests: `@pytest.mark.browser` — excluded by default in `pytest.ini`; Docker CI runs `tests/test_crawl_fetchers.py` and `tests/test_crawler_browser_e2e.py -m browser`; locally `./local-test browser`. @@ -22,13 +24,14 @@ - Optional step: `crawl` | `report` | `plot` | `lighthouse` | `keywords` | `warnings` | `enrich` | `google` | `chat` - **`preserve_crawl_history`** (default true): append crawls; `false` truncates crawl tables but restores `report_payload`, Lighthouse, `google_data`, `keyword_data`, `keyword_history`, `keyword_suggest_cache`, and `crawl_runs` - **`DATABASE_URL`** env: PostgreSQL connection string (required). **`DATA_DIR`**: secrets + shadow config (Docker: `/data`). -- **Pipeline data** (crawl, edges, nodes, report payload, Lighthouse, keywords, warnings) is stored in **PostgreSQL only** — no JSON/CSV/HTML exports from the main pipeline. -- **Pool tuning:** `DB_POOL_MIN` / `DB_POOL_MAX` (Python), `PGPOOL_MAX` (Node). Bulk crawl writes via `executemany`; optional **`crawl_stream_to_db`** streams rows during fetch. -- **`web/`:** `/api/report/*` (PostgreSQL); `/api/run` spawns Python (localhost only); `/api/crawl/browser-status` GET (localhost, Playwright/Chromium preflight); `/api/pipeline-config` GET/PUT; `/api/llm-config` GET/PUT (AI only); `/api/chat` POST (SSE agent); `/api/chat/sessions` GET/POST; `/api/properties/{id}/google/links/import` POST (GSC Links CSV); `PipelineRunnerFab` saves pipeline + LLM state before each run +- **Pipeline storage** (crawl, edges, nodes, report payload, Lighthouse, keywords, warnings) lives in **PostgreSQL only**. Deliverables use the Export view, `GET /api/report/export`, or MCP `export_*` tools — not files written by the main pipeline step. +- **Pool tuning:** `DB_POOL_MIN` / `DB_POOL_MAX` (Python), `PGPOOL_MAX` (Node). Bulk crawl writes via `executemany`; optional **`crawl_stream_to_db`** streams rows during fetch. Per-URL raw HTML: `crawl_page_html` table (migration `015`); API `GET/POST /api/crawl/page-html` (localhost). +- **`web/` APIs:** `/api/report/*` read routes (payload, meta, history — not localhost-guarded; protect with `AUTH_*` when exposed); `/api/run` spawns Python (localhost); `/api/jobs`, `/api/jobs/[id]`, `/api/jobs/[id]/cancel` (localhost); `/api/crawl/browser-status`, `/api/crawl/page-html` (localhost); `/api/pipeline-config` GET/PUT; `/api/llm-config` GET/PUT; `/api/chat` POST (SSE); `/api/chat/sessions` GET/POST; `/api/ollama/status` (localhost); `/api/properties/{id}/google/links/import` POST; `PipelineRunnerFab` saves pipeline + LLM state before each run. Full route list: `web/app/api/**/route.ts`. - **MCP:** `python -m website_profiling.mcp` (stdio, **340 read-only audit tools**, domain-scoped via `WP_MCP_DOMAIN`). See `docs/MCP.md`. Requires `pip install -r requirements.txt`. -- **AI Chat UI:** `/chat` — property-scoped chat with saved sessions (`chat_sessions`, `chat_messages` tables, migration `012_chat_sessions`). -- **Job store:** in-memory on `globalThis` in `web/src/server/pipelineJobs.ts` — job status/log is lost on server restart (single-process dev/Docker only). -- **Docker:** `Dockerfile` + `docker-compose.yml` (postgres + web); **`docker-compose.pull.yml`** for pre-built images (`WEB_IMAGE`); **`LIGHTHOUSE_CHROME_FLAGS`** +- **AI Chat UI:** `/chat` — property-scoped chat with saved sessions (`chat_sessions`, `chat_messages`; migration `012_chat_sessions`). +- **Job store:** PostgreSQL `pipeline_jobs` when `DATABASE_URL` is set (`pipelineJobsDb.ts` — status, timestamps, truncated logs). In-memory map in `pipelineJobs.ts` holds live log tail and child process handles; stale rows reconciled via `PIPELINE_JOB_STALE_HOURS`. +- **Schema head:** `015_crawl_page_html` (recent: `013` link_edges/discovery, `014` job log truncation, `015` per-URL HTML storage). +- **Docker:** `Dockerfile` + `docker-compose.yml` (postgres + web); **`docker-compose.prod.yml`** (production); **`docker-compose.pull.yml`** for pre-built images (`WEB_IMAGE`); **`LIGHTHOUSE_CHROME_FLAGS`** **Where to edit** diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 02962479..3f78ab2a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -17,7 +17,7 @@ Thank you for helping improve this project. All contributions are welcome under ./local-run # dev server → http://localhost:3000/home ``` -Details: [README.md](README.md), [AGENT.md](AGENT.md). +Details: [README.md](README.md), [AGENT.md](AGENT.md), [docs/README.md](docs/README.md). JavaScript/auto crawl needs Playwright (from `requirements.txt`, installed by `./local-run setup`) and Chromium on `PATH` or `CHROME_PATH`. Unit tests mock the browser fetcher; integration tests use `@pytest.mark.browser` and run in the Docker CI job (`tests/test_crawl_fetchers.py`, `tests/test_crawler_browser_e2e.py`). Locally: `./local-test browser` (skips gracefully if Chromium is missing). @@ -26,14 +26,16 @@ JavaScript/auto crawl needs Playwright (from `requirements.txt`, installed by `. Match CI before opening a pull request: ```bash -./local-test # full check (recommended) -./local-test python # backend only +./local-test # full check — three Python 100% gates + web (recommended) +./local-test python # backend: core, reporting, and tools coverage gates ./local-test browser # JS crawl integration tests (skips if Chromium unavailable) -./local-test web # frontend only -./local-test quick # faster; DB must already be running +./local-test web # frontend only +./local-test quick # faster; DB must already be running ``` -CI runs Python tests (PostgreSQL + Alembic, 80% coverage gate), web typecheck/lint/vitest, CLI smoke, and a Docker build that also runs browser-marked pytest inside the image (see [.github/workflows/ci.yml](.github/workflows/ci.yml)). +CI also runs a **Docker** job (image build, browser pytest in container, compose smoke). `./local-test` does not run that job — see [.github/workflows/ci.yml](.github/workflows/ci.yml). + +When adding tools coverage tests, register new files in `scripts/local-test.sh`, `scripts/local-test.ps1`, and `.github/workflows/ci.yml` (see [AGENT.md](AGENT.md)). ## How to contribute @@ -58,13 +60,13 @@ Per-URL GSC/GA4 in Link Explorer uses two histories: | Store | How it is created | Compare | |-------|-------------------|---------| | `google_data` | Site-wide **Fetch data now** (scoped to the `properties` row for the audit Site URL) | Pick an older site snapshot in the tab | - -**Google OAuth:** App Client ID/Secret and optional service account JSON live in PostgreSQL (`google_app_settings`, migration `006_google_app_settings`). Each **property** (domain) has its own refresh token and GSC/GA4 IDs on the `properties` table. Set a Site URL, then connect Google from Pipeline → Integrations. Configure via the UI or `GOOGLE_CLIENT_ID` / `GOOGLE_CLIENT_SECRET` env vars for bootstrap only. | `page_google_snapshots` | **Fetch live data** on the Search & retention tab | Defaults to previous live fetch for the same URL | -**Prerequisites:** Google connected with GSC site URL and GA4 property saved; at least one site-wide fetch for snapshot defaults; two or more live fetches on the same URL to compare live periods. AI page coach requires **Enable AI insights** and **Link Explorer page coach** in Pipeline → Content & AI. +**Google OAuth:** App Client ID/Secret and optional service account JSON live in PostgreSQL (`google_app_settings`, migration `006_google_app_settings`). Each **property** (domain) has its own refresh token and GSC/GA4 IDs on the `properties` table. Set a Site URL, then connect Google from **Integrations**. Configure via the UI or `GOOGLE_CLIENT_ID` / `GOOGLE_CLIENT_SECRET` env vars for bootstrap only. + +**Prerequisites:** Google connected with GSC site URL and GA4 property saved; at least one site-wide fetch for snapshot defaults; two or more live fetches on the same URL to compare live periods. AI page coach requires **Enable AI insights** and **Link Explorer page coach** under Run audit → **Content quality & AI insights**. -Apply migration `004_page_google_snapshots` (`alembic upgrade head`) before using live fetch. +Apply migrations before using live fetch: `./local-run migrate` or `alembic upgrade head`. ### PostgreSQL JSONB rows diff --git a/README.md b/README.md index 9059ecc6..ca1ad734 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@

Site Audit — Open Source SEO Crawl & Audit
- Free, self-hosted — no vendor paywalls. + Self-hosted technical SEO — your infrastructure, your data.

@@ -24,11 +24,12 @@

- Quick start · + Quick start · Features · - Structure · + Limitations · + Structure · Contributing · - Docs · + Docs · License

@@ -36,15 +37,35 @@ # Site Audit -**Open Source SEO Crawl & Audit** — self-hosted UI built with **Next.js + Python + PostgreSQL**. +**Open-source SEO crawl and technical audit platform** — self-hosted UI built with **Next.js, Python, and PostgreSQL**. Repository: [codefrydev/WebsiteProfiling](https://github.com/codefrydev/WebsiteProfiling) ## Overview -**Why this project** — Most site-audit and SEO tools are paid, limited, or built to upsell: paywalls, capped crawls, teaser scores, and “subscribe to see how to fix this.” Many free options give shallow or unreliable reports that push you toward a paid plan instead of real answers. +Site Audit is a self-hosted alternative to commercial SEO audit tools. It runs on your infrastructure, stores data in your PostgreSQL database, and produces transparent technical reports without subscription tiers or gated exports. -**Goal** — A free, self-hosted audit you control: crawl your sites, see honest technical SEO issues, connect Search Console and Analytics when you want, and export reports for clients — without a vendor sitting between you and the data. +**Use cases** + +- Technical SEO audits for owned or client properties +- Crawl analysis with static and JavaScript rendering +- Search Console, GA4, and Bing Webmaster integration +- Agency portfolio management and run comparison +- Optional AI-assisted analysis over audit data via MCP-compatible tools + +## Scope and limitations + +Site Audit focuses on **honest, self-hosted technical SEO**. It is not a drop-in replacement for every paid SaaS data product. + +- **No live backlink index** — Backlink tools read **Google Search Console Links CSV imports** (and optional third-party CSV overlays). There is no Ahrefs, Semrush, Moz, or Majestic API integration. +- **No daily rank tracking** — Keyword positions come from **GSC snapshots** on your connected property, not a proprietary SERP tracker or rank-history database. +- **No live AI citation checks** — GEO/AEO tools use **on-site heuristics**; they do not query ChatGPT, Perplexity, or other AI search engines in real time. +- **No third-party keyword volume APIs** — Keyword explorer uses **on-site frequency plus Search Console**; difficulty and SERP feature overlays are estimated unless you supply your own data. +- **No managed cloud** — You run it (Docker or local dev). This repo is not a hosted multi-tenant SaaS. +- **No substitute for Google access** — Search Console, Analytics, and Bing Webmaster require **your credentials**; missing or stale integrations show empty states with provenance labels, not fabricated metrics. +- **Not a ranking guarantee** — Category scores (0–100) are **internal audit scores**, not Google rankings or predicted traffic impact. + +**Planned extensions** (not yet shipped): full backlink index beyond GSC import, SERP rank tracking beyond GSC snapshots, and live AI citation APIs. See [docs/MCP.md](docs/MCP.md#future-pipeline-items). ## Features @@ -73,13 +94,13 @@ Repository: [codefrydev/WebsiteProfiling](https://github.com/codefrydev/WebsiteP -Also included: **AI chat** over audit data (optional), **340 MCP tools** (domain-scoped servers), keyword explorer, backlinks, compare runs, and portfolio management for agencies. +Also included: **AI chat** over audit data (optional), **340 MCP tools** (domain-scoped servers), keyword explorer (GSC + on-site), backlinks (GSC Links import), compare runs, and portfolio management for agencies.

Site Audit preview

-## Project structure +## Architecture ``` WebsiteProfiling/ @@ -108,6 +129,8 @@ WebsiteProfiling/ ├── scripts/ # local-run.sh, local-test.sh helpers ├── .github/workflows/ # CI (Python + web + browser crawl) ├── docker-compose.yml # Dev stack (Postgres + web) +├── docker-compose.prod.yml # Production stack (requires AUTH_SECRET) +├── docker-compose.pull.yml # Pre-built WEB_IMAGE ├── Dockerfile # Production image ├── local-run # Dev setup & start script ├── local-test # Full test suite (CI parity) @@ -122,14 +145,18 @@ WebsiteProfiling/ | `web/src/lib/pipelineConfigSchema.ts` | Audit settings schema (UI ↔ PostgreSQL) | | `alembic/versions/` | Database migrations — run `./local-run migrate` | | `tests/` | Backend tests; `./local-test browser` for Playwright crawl integration | -| `docs/MCP.md` | MCP server setup for IDE / agent integrations | -| `data/` | Local secrets + shadow `pipeline-config.txt` (gitignored) | +| `docs/MCP.md` | MCP server setup for IDE and agent integrations | +| `data/` | Local secrets and shadow `pipeline-config.txt` (gitignored) | +| `docker-compose.prod.yml` | Production stack (`POSTGRES_USER`, `POSTGRES_PASSWORD`, `AUTH_SECRET`) | +| `docker-compose.pull.yml` | Pre-built `WEB_IMAGE` deployment | + +For layout details and common development patterns, see [AGENT.md](AGENT.md). -For deeper layout notes and edit targets, see [AGENT.md](AGENT.md). +## Getting started -## Quick start +### Docker -**Docker (build from source)** +Build and run from source: ```bash docker compose up --build @@ -137,60 +164,83 @@ docker compose up --build Open [http://localhost:3000/home](http://localhost:3000/home). -**Local dev** +Production deployment: `docker-compose.prod.yml` — set `POSTGRES_USER`, `POSTGRES_PASSWORD`, and `AUTH_SECRET`. Pre-built images: `docker-compose.pull.yml` (`WEB_IMAGE`). + +### Local development ```bash -./local-run setup # first time: Postgres, Python venv, migrations, npm deps -./local-run # daily: start DB + Next.js dev server → http://localhost:3000/home +./local-run setup # First time: Postgres, Python venv, migrations, npm deps +./local-run # Start DB + Next.js dev server → http://localhost:3000/home ./local-run db # Postgres only (no app) -./local-run migrate # apply Alembic migrations only -./local-run stop # stop Postgres container +./local-run migrate # Apply Alembic migrations only +./local-run stop # Stop Postgres container ``` +Default local `DATABASE_URL`: `postgres://postgres:dev@127.0.0.1:5432/website_profiling` (Docker Compose dev stack uses `profiling:profiling`). + `requirements.txt` pins direct Python dependencies to versions verified by `./local-test python`. Re-run the full test suite after intentional upgrades. -Pipeline jobs: stuck `running` rows are reconciled after **1 hour** by default (`PIPELINE_JOB_STALE_HOURS`). Orphan jobs with no live server process are cleared after **5 minutes** (`PIPELINE_JOB_ORPHAN_MINUTES`). Increase `PIPELINE_JOB_STALE_HOURS` for crawls that routinely run longer than an hour. +### Pipeline job timeouts + +| Setting | Default | Description | +|---------|---------|-------------| +| `PIPELINE_JOB_STALE_HOURS` | 1 hour | Reconciles stuck `running` rows | +| `PIPELINE_JOB_ORPHAN_MINUTES` | 5 minutes | Clears orphan jobs with no live server process | -**Tests** +Increase `PIPELINE_JOB_STALE_HOURS` for crawls that routinely exceed one hour. + +### Testing ```bash -./local-test # before push: full CI parity (DB + pytest + web) -./local-test python # backend: pytest (80% coverage) + browser pytest + CLI smoke +./local-test # Python + web (matches CI python and web jobs) +./local-test python # Backend: three 100% coverage gates + browser pytest + CLI smoke ./local-test browser # JS crawl integration tests (skips if Chromium unavailable) -./local-test web # frontend: typecheck, lint, vitest -./local-test quick # fast loop; needs DB already up (no coverage gate) -./local-test all --no-cov # full run without pytest coverage gate +./local-test web # Frontend: typecheck, lint, vitest +./local-test quick # Fast loop; requires DB already running (no coverage gate) +./local-test all --no-cov # Full run without pytest coverage gate ``` -## Contributing +CI also runs a **Docker** job (image build, browser pytest in container, compose smoke). See [.github/workflows/ci.yml](.github/workflows/ci.yml). -Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for setup and pull request guidelines. +## Configuration -- [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) — community standards -- [SECURITY.md](SECURITY.md) — report vulnerabilities privately +### Integrations -## Docs +Connect Google Search Console and Analytics via **Integrations** (gear icon) in the application UI. -- [AGENT.md](AGENT.md) — repo layout and dev commands -- [docs/GLOSSARY.md](docs/GLOSSARY.md) — UI terminology -- [docs/COMPANY_STANDARDS.md](docs/COMPANY_STANDARDS.md) — data and security policy +### JavaScript crawl (optional) -Google Search Console / Analytics: connect via **Integrations** (gear icon) in the app. +In Audit settings, set **Crawl rendering** to `javascript` (always headless Chromium) or `auto` (static first, browser when SPA heuristics match). Requires Playwright from `requirements.txt` and Chromium on `PATH` or `CHROME_PATH` (included in Docker). The UI preflights via `GET /api/crawl/browser-status` before runs when JS or auto mode is selected. -**JavaScript crawl (optional):** In Audit settings, set **Crawl rendering** to `javascript` (always headless Chromium) or `auto` (static first, browser when SPA heuristics match). Requires Playwright from `requirements.txt` and Chromium on `PATH` or `CHROME_PATH` (included in Docker). The UI preflights via `GET /api/crawl/browser-status` before runs when JS/auto is selected. +### AI chat (optional) -**AI Chat (optional):** Ask questions about your audit data at [http://localhost:3000/chat](http://localhost:3000/chat). Enable a provider under **Run audit → AI settings** (`llm_enabled`, provider, model). `./local-run setup` installs all Python deps from `requirements.txt` (including `httpx`, OpenAI, and Anthropic SDKs). +Ask questions about audit data at [http://localhost:3000/chat](http://localhost:3000/chat). Enable a provider under **Run audit → AI settings** (`llm_enabled`, provider, model). `./local-run setup` installs Python deps from `requirements.txt` (including `httpx`, OpenAI, and Anthropic SDKs; Gemini uses `httpx` via REST). | Provider | Notes | -|----------|--------| -| **Ollama** | Local daemon at `http://127.0.0.1:11434`. Chat UI lists installed models plus the live Ollama cloud catalog (billing: free local, account free tier, Pro). Native tool calling when supported; otherwise ReAct fallback. Pick the model in-chat without leaving the page. | -| **OpenAI** / **Anthropic** | API key in AI settings; native tool calling with streaming. | +|----------|-------| +| **Ollama** | Local daemon at `http://127.0.0.1:11434`. Chat UI lists installed models plus the live Ollama cloud catalog. Native tool calling when supported; ReAct fallback otherwise. | +| **OpenAI** / **Anthropic** | API key in AI settings or env (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`); native tool calling with streaming. | +| **Google Gemini** | API key in AI settings or `GEMINI_API_KEY`; REST via `httpx`. | + +The agent uses the same **340 read-only audit tools** as the MCP server ([docs/MCP.md](docs/MCP.md)), with **dynamic routing** (~45 tools per turn). Responses stream over SSE (`POST /api/chat`). Sessions persist per property (`chat_sessions` / `chat_messages`). + +## Contributing -The agent uses the same **340 read-only audit tools** as the MCP server (`docs/MCP.md`), with **dynamic routing** (~45 tools per turn plus router meta-tools). Responses stream over SSE (`POST /api/chat`) with status, tool activity, and tokens. Sessions are saved per property (`chat_sessions` / `chat_messages`). +Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for setup and pull request guidelines. -Production: `docker-compose.prod.yml` (set `POSTGRES_PASSWORD`, `AUTH_SECRET`). +- [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) — community standards +- [SECURITY.md](SECURITY.md) — report vulnerabilities privately +## Documentation +| Document | Description | +|----------|-------------| +| [docs/README.md](docs/README.md) | Documentation index and brand assets | +| [AGENT.md](AGENT.md) | Repository layout and development commands | +| [docs/GLOSSARY.md](docs/GLOSSARY.md) | UI terminology | +| [docs/COMPANY_STANDARDS.md](docs/COMPANY_STANDARDS.md) | Data and security policy | +| [docs/MCP.md](docs/MCP.md) | MCP server setup | +| [docs/OPS.md](docs/OPS.md) | Scheduled audits, alerts, production ops | ## Star History @@ -198,4 +248,6 @@ Production: `docker-compose.prod.yml` (set `POSTGRES_PASSWORD`, `AUTH_SECRET`). ## License -Copyright (c) 2026 [codefrydev](https://github.com/codefrydev). Released under the **MIT License** — see [LICENSE](LICENSE). Issues and pull requests: [codefrydev/WebsiteProfiling](https://github.com/codefrydev/WebsiteProfiling). \ No newline at end of file +Copyright © 2026 [codefrydev](https://github.com/codefrydev). Released under the [MIT License](LICENSE). + +Issues and pull requests: [codefrydev/WebsiteProfiling](https://github.com/codefrydev/WebsiteProfiling) diff --git a/SECURITY.md b/SECURITY.md index 49b3936e..fb009999 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -25,5 +25,5 @@ If you find a vulnerability **in Site Audit itself** (e.g. remote code execution ## Safe defaults - Run production deployments with strong `POSTGRES_PASSWORD` and `AUTH_SECRET` (see `docker-compose.prod.yml`). -- For client-facing dashboards, set `AUTH_DEFAULT_ROLE=client-readonly` so logins cannot run audits or mutate settings (API enforces 403; UI hides Run audit). +- For client-facing dashboards, set `AUTH_DEFAULT_ROLE=client-readonly` (view reports and use chat) or `viewer` (view reports only). API enforces 403 on mutations; UI hides **Run audit** for read-only roles. - Do not commit `.env`, `.secrets/`, or OAuth client secrets. Google credentials are stored in PostgreSQL (`google_app_settings` and per-property columns on `properties`). diff --git a/docs/COMPANY_STANDARDS.md b/docs/COMPANY_STANDARDS.md index 6ca07285..da25ca84 100644 --- a/docs/COMPANY_STANDARDS.md +++ b/docs/COMPANY_STANDARDS.md @@ -1,52 +1,100 @@ -# Company standards +# Company Standards -WebsiteProfiling (UI: **Site Audit**) is an agency-grade technical SEO and site analysis tool. This document defines how data must be presented and how the product may be used in a professional context. +This document defines data presentation requirements, crawl scope, and acceptable use for **Site Audit** (repository: WebsiteProfiling) in professional and agency contexts. + +**Related documentation:** [README.md](../README.md) · [GLOSSARY.md](GLOSSARY.md) · [Documentation index](README.md) + +--- + +## Purpose + +Site Audit is an agency-grade technical SEO and site analysis platform. Reports must clearly distinguish measured data from estimates and third-party integrations. Category scores and heuristics must never be presented as guaranteed ranking or traffic outcomes. + +--- ## Data classification -| Class | Meaning | Examples | -|-------|---------|----------| -| **Measured** | Direct observation from HTTP crawl or Lighthouse | Status codes, title tags, LCP, header presence | -| **Search Console** | Google Search Console API | Impressions, clicks, average position | -| **Analytics** | Google Analytics 4 API | Sessions, users, engagement rate | -| **Estimated** | Heuristic when external APIs are unavailable | On-site keyword frequency, fixed difficulty placeholder | -| **AI insight** | LLM enrichment (optional) | Summaries, semantic clusters — always labeled | +All metrics displayed in the UI or exports must align with one of the following classes: + +| Class | Definition | Examples | +|-------|------------|----------| +| **Measured** | Direct observation from HTTP crawl or Lighthouse | Status codes, title tags, LCP, response headers | +| **Search Console** | Data retrieved via Google Search Console API | Impressions, clicks, average position | +| **Analytics** | Data retrieved via Google Analytics 4 API | Sessions, users, engagement rate | +| **Estimated** | Heuristic derived when external APIs are unavailable | On-site keyword frequency, difficulty placeholders | +| **AI insight** | Optional LLM-generated content | Summaries, semantic clusters — always labeled | + +**Audit category scores (0–100)** are internal prioritization scores. They are not Google rankings, PageRank, or predicted traffic impact. + +--- + +## Crawl scope and rendering -Audit category scores (0–100) are **internal audit scores**, not Google rankings or guaranteed traffic impact. +| Mode | Config value | Behavior | +|------|--------------|----------| +| Static (default) | `crawl_render_mode = static` | HTTP GET with HTML parsing; no JavaScript execution | +| JavaScript | `crawl_render_mode = javascript` | Every page loaded in headless Chromium before parsing | +| Auto | `crawl_render_mode = auto` | Static fetch first; browser fallback when SPA heuristics or low outlink counts indicate client-rendered content | -## Crawl limitations +**Rendering notes:** -- Default crawl uses **HTTP GET + static HTML parsing** (no JavaScript execution). `crawl_render_mode = static` (default). -- Optional **JavaScript rendering** (`crawl_render_mode = javascript`) loads every page in headless Chromium before parsing — slower (~10–20×) and heavier on memory, but required for many React, Vue, Next.js, Angular, Svelte, and Shopify themes. -- **Auto rendering** (`crawl_render_mode = auto`) fetches static HTML first, then uses browser fallback when SPA shell heuristics or low outlink counts suggest client-rendered content. Per-page `fetch_method` (`static` vs `rendered`) is stored on crawl rows for provenance. -- Client-rendered links and SPAs may be under-represented in static-only mode; reports show crawl scope (pages crawled vs limit, robots blocks, render mode, browser diagnostic counts when applicable). -- JS and auto modes require Playwright + Chromium; the Run audit UI checks availability via `GET /api/crawl/browser-status` before starting a job. -- Only crawl sites you are **authorized** to test. Respect `robots.txt` unless an admin explicitly overrides for owned properties. +- JavaScript mode is approximately 10–20× slower and more memory-intensive than static mode. It is required for many React, Vue, Next.js, Angular, Svelte, and Shopify implementations. +- Auto mode stores per-page `fetch_method` (`static` or `rendered`) on crawl rows for provenance. +- Static-only crawls may under-represent client-rendered links and single-page applications. Reports include crawl scope metadata: pages crawled versus limit, robots blocks, render mode, and browser diagnostic counts when applicable. +- JavaScript and auto modes require Playwright and Chromium. The Run audit UI validates availability via `GET /api/crawl/browser-status` before starting a job. + +**Authorization:** Crawl only properties you own or have written permission to test. Respect `robots.txt` unless an administrator explicitly overrides for owned properties. + +--- ## Security scanning -- **Passive** checks use crawl response headers (default). -- **Active** probes (`security_scan_active`) send controlled requests — enable only with written authorization for the target property. +| Mode | Config | Requirements | +|------|--------|--------------| +| Passive | Default | Analysis of response headers from crawl requests | +| Active | `security_scan_active` | Sends controlled probe requests — enable only with written authorization for the target property | + +--- ## Google integrations -- Use official names in client-facing copy: **Google Search Console**, **Google Analytics 4**. -- Snapshots include fetch time and date range; stale or partial data must not appear as current without a warning. +- Use official product names in client-facing copy: **Google Search Console**, **Google Analytics 4**. +- Snapshots must include fetch time and date range. +- Stale or partial integration data must not appear as current without an explicit warning. + +--- ## Agency workflow -- **Properties** group client sites (canonical domain, optional GSC/GA4 binding). -- An **audit run** is a stored report snapshot (crawl + analysis + optional Lighthouse/Google). -- Exports (PDF/CSV) include a data source legend. -- Category titles and issue copy in Python use agency vocabulary (`src/website_profiling/reporting/terminology.py`); see [GLOSSARY.md](GLOSSARY.md). +| Concept | Definition | +|---------|------------| +| **Property** | A client site grouped by canonical domain, with optional GSC/GA4 binding | +| **Audit run** | A stored report snapshot comprising crawl, analysis, and optional Lighthouse or Google data | +| **Export** | PDF, CSV, or HTML deliverable including a data source legend | + +Category titles and issue copy in Python use agency vocabulary (`src/website_profiling/reporting/terminology.py`). See [GLOSSARY.md](GLOSSARY.md) for UI term mappings. + +--- + +## Production requirements + +| Requirement | Guidance | +|-------------|----------| +| Database credentials | Do not use default passwords in production (`POSTGRES_USER`, `POSTGRES_PASSWORD`) | +| Session auth | Set `AUTH_SECRET`; optionally `AUTH_USER`, `AUTH_PASSWORD`, `AUTH_DEFAULT_ROLE` | +| API access | Protect pipeline and integration endpoints when not bound to localhost | +| Backups | Back up PostgreSQL regularly — e.g. `pg_dump -Fc "$DATABASE_URL" > site-audit-$(date +%F).dump` | +| Client dashboards | `AUTH_DEFAULT_ROLE=client-readonly` (view + chat) or `viewer` (view only, no chat) | + +--- -## Production expectations +## Continuous integration -- Do not use default database passwords in production. -- Protect pipeline and integration APIs with authentication when not on localhost. -- Back up PostgreSQL regularly (`pg_dump` — see [README.md](../README.md)). +Pull requests should pass: -## CI and releases +- Python tests (three 100% coverage gates: core, reporting, tools) +- Web typecheck, lint, and Vitest +- Alembic migrations on an empty database +- Docker image build -- All PRs should pass Python tests, web typecheck/lint/test, migrations on empty DB, and Docker image build (see `.github/workflows/ci.yml`). -- Prefer branch protection on `master`: require CI checks before merge. +Configuration: [.github/workflows/ci.yml](../.github/workflows/ci.yml). Branch protection on `master` with required CI checks is recommended. diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index 436850f4..0bb21d7c 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -1,62 +1,70 @@ # Glossary -UI terms agencies recognize, mapped to internal keys and data sources. +This glossary maps agency-facing UI terms to internal keys, database tables, and data sources. Use it when writing UI copy, client reports, or integration documentation. -| UI term | Internal key / table | Data source | Similar tools | -|---------|----------------------|-------------|---------------| +**Related documentation:** [COMPANY_STANDARDS.md](COMPANY_STANDARDS.md) · [MCP.md](MCP.md) · [Documentation index](README.md) + +--- + +## Report views and features + +| UI term | Internal key / table | Data source | Comparable tools | +|---------|------------------------|-------------|------------------| | Properties | Home portfolio, `properties` | User-defined; one Google OAuth + GSC/GA4 mapping per row | GA4 Property, GSC property | | Audit run | `report_payload`, report list | Crawl + report build | Semrush Site Audit run | | Overview | `overview` view | Report payload | Site Audit dashboard | -| Issues | `issues` view, `categories[].issues` | Rule engine on crawl + Lighthouse | Semrush Issues, SF errors | +| Issues | `issues` view, `categories[].issues` | Rule engine on crawl + Lighthouse | Semrush Issues, Screaming Frog errors | | All URLs | `links` view, crawl rows | HTTP crawl | Screaming Frog Internal URLs | -| URL structure | `site-structure` | Crawl graph | SF directory tree | -| Redirects & chains | `redirects` | Crawl status + chains | SF redirect chains | +| URL structure | `site-structure` | Crawl graph | Screaming Frog directory tree | +| Redirects & chains | `redirects` | Crawl status + chains | Screaming Frog redirect chains | | On-page SEO | `content` | Crawl meta/titles | Semrush On Page | | Performance (Core Web Vitals) | `lighthouse`, `lighthouse_summary` | Lighthouse | PageSpeed Insights | | Security | `security`, `security_findings` | Headers + optional probes | Sitebulb Security | | Content quality | Category `content quality` (id: intelligence) | Crawl + analysis + optional AI insights | Duplicate/thin content audits | | Technologies | `tech-stack` | Wappalyzer-style detection | BuiltWith | -| Crawl summary | `charts` | Crawl aggregates | SF overview | +| Crawl summary | `charts` | Crawl aggregates | Screaming Frog overview | | Internal links | `network` | Link graph | Ahrefs Internal Links | -| Backlinks | `backlinks`, `gsc_links`, `gsc_links_data` | GSC Links CSV import (Google sample) | GSC Links report | -| Page previews | `gallery`, `list_site_image_urls`, `image_inventory` | Crawl excerpts + optional HTTP probe | Visual QA; size/format when probed | +| Backlinks | `backlinks`, `gsc_links`, `gsc_links_data` | GSC Links CSV import | Google Search Console Links report | +| Page previews | `gallery`, `list_site_image_urls`, `image_inventory` | Crawl excerpts + optional HTTP probe | Visual QA | | Search Console | `search-performance`, `google_data` (scoped by `property_id`) | GSC API per property | Google Search Console | | Analytics (GA4) | `traffic`, `google_data` (scoped by `property_id`) | GA4 API per property | Google Analytics | | Keywords | `keywords-explorer`, `keyword_data` | Crawl + Search Console + research | Keyword tools (site-scoped) | | Compare audits | `compare` | Two report payloads | Historical comparison | -| Indexation & coverage | `indexation`, `indexation_coverage` | Crawl + sitemap + GSC URL join | SEMrush indexability, GSC coverage | +| Indexation & coverage | `indexation`, `indexation_coverage` | Crawl + sitemap + GSC URL join | Semrush indexability, GSC coverage | | CrUX field CWV | `crux_summary` | Chrome UX Report API | PageSpeed field data | | Executive summary | `executive_summary` | Issues + GSC + optional AI | Agency audit cover page | | Run audit | Pipeline / `python -m src` | User-triggered job | Start site audit | -| Issue task board | `issues` view (board tab), `issue_status` | Workflow persistence per property | Jira-style triage | +| Issue task board | `issues` view (board tab), `issue_status` | Workflow persistence per property | Issue triage board | | Query–page alignment | `keywords-explorer` alignment tab, `query_page_misalignment` | Search Console heuristics | Landing-page targeting | | Crawl segments | `site-structure` overview, `crawl_segments` | `crawl_path_segments` config + crawl | Section health rollups | | Log analyzer | `log-analyzer` view | Uploaded access log vs crawl | Log file insights | | Competitor link gap | `backlinks` overview, `competitor_link_gap` | GSC Links import + `competitor_domains` | Link gap analysis | -| Moz / Majestic overlay | `third_party_overlays` on `gsc_links`, `/api/backlinks/third-party-import` | CSV export upload | Estimated referring-domain comparison vs GSC sample | +| Moz / Majestic overlay | `third_party_overlays` on `gsc_links`, `/api/backlinks/third-party-import` | CSV export upload | Referring-domain comparison vs GSC sample | | Bing backlinks | `bing_backlinks`, Integrations sync | Bing Webmaster API (optional) | Secondary link source | | SERP competition overlay | `serp_estimated_competition` on keywords | SerpAPI (optional) | Estimated SERP difficulty | | Scheduled audits | `properties.schedule_cron`, `/api/schedule/check` | Cron + pipeline spawn | Recurring site audit — see [OPS.md](OPS.md) | -| Property alerts | `alert_webhook_url`, `/api/alerts/check` | Health snapshot rules | Ops notifications | +| Property alerts | `alert_webhook_url`, `/api/alerts/check` | Health snapshot rules | Operations notifications | | Content brief | Keywords Brief button, `/api/keywords/content-brief` | LLM or deterministic | Content planning | -| AI fix suggestions | `llm_recommendation`, `/api/ai/fix-suggestion`, `/api/issues/fix-suggestion` (legacy) | LLM on demand + report build | Actionable remediation across Issues, Lighthouse, Security, and other surfaces | -| AI Chat | `/chat`, `/api/chat`, `chat_sessions` | LLM + read-only audit tools | Conversational site audit queries | -| MCP tools | `python -m website_profiling.mcp` | Same `audit_tools` as chat | Cursor / Claude Desktop integration — see [MCP.md](MCP.md) | -| Read-only session | `AUTH_DEFAULT_ROLE=client-readonly`, `/api/auth/session` | Session cookie | Client view-only access | -| Export executive summary | `export_audit_html/pdf/csv`, `export_audit_report` (chat/MCP), Export view | Report payload + optional AI | Client deliverable | +| AI fix suggestions | `llm_recommendation`, `/api/ai/fix-suggestion` | LLM on demand + report build | Actionable remediation | +| AI Chat | `/chat`, `/api/chat`, `chat_sessions` | LLM + read-only audit tools | Conversational audit queries | +| MCP tools | `python -m website_profiling.mcp` | Same `audit_tools` as chat | IDE integration — see [MCP.md](MCP.md) | +| Read-only session | `AUTH_DEFAULT_ROLE=client-readonly` or `viewer`; `/api/auth/session` returns role and mutation flags | Session cookie | `client-readonly`: view + chat; `viewer`: view only (no chat) | +| Export executive summary | `export_audit_html/pdf/csv`, Export view | Report payload + optional AI | Client deliverable | | ads.txt / security.txt | `site_level`, `get_ads_txt_status`, `get_security_txt_status` | Root file fetch at report build | Publisher / contact file hygiene | | Subdomain inventory | `subdomains`, `list_subdomains`, `/subdomains` view | Crawl + GSC + optional crt.sh | Host footprint vs crawl scope | | Contact intelligence | `contact_intelligence`, `get_contact_intelligence`, `/contacts` view | Crawl schema/mailto + security.txt + RDAP org | Business identity consistency | -## Metric names +--- -| UI | Field | Source | -|----|-------|--------| -| Impact score | `impact_score` on issues | GSC clicks + GA4 sessions + priority weight (see below) | +## Metrics + +| UI label | Field | Source | +|----------|-------|--------| +| Impact score | `impact_score` on issues | GSC clicks + GA4 sessions + priority weight | | Link edges | `link_edges`, `link_rel_summary` | Crawl anchor/rel attributes | | Outlinks | `outlinks` | Crawl graph | -| Status code | `status` | HTTP | -| Crawl rendering | `crawl_render_mode` on run; `fetch_method` per URL | `static`, `javascript`, or `auto` crawl config; `static` vs `rendered` per page | +| Status code | `status` | HTTP response | +| Crawl rendering | `crawl_render_mode` on run; `fetch_method` per URL | `static`, `javascript`, or `auto`; per-page `static` vs `rendered` | | Impressions | `gsc_impressions` | Search Console | | Referring domains | `top_linking_sites` | GSC Links CSV import | | External links to site | `sample_links`, `latest_links` | GSC Links CSV import | @@ -66,22 +74,34 @@ UI terms agencies recognize, mapped to internal keys and data sources. | On-site frequency | `volume` (heuristic) | Estimated from crawl | | Sessions | GA4 metrics | Analytics | -**Impact score:** `priority_weight + (gsc_clicks × 10) + (ga4_sessions × 5)` with Critical=1000, High=100, Medium=10, Low=1. +### Impact score formula + +``` +impact_score = priority_weight + (gsc_clicks × 10) + (ga4_sessions × 5) +``` + +Priority weights: Critical = 1000, High = 100, Medium = 10, Low = 1. + +--- ## Provenance badges +Every data point in the UI should display one of the following provenance labels where applicable: + | Badge | Meaning | |-------|---------| -| Crawl | Spider/fetch data | +| Crawl | Spider or HTTP fetch data | | Lighthouse | Lab performance audit | | Search Console | Google Search Console API | | Analytics | Google Analytics 4 API | -| Estimated | Heuristic, not from Google | +| Estimated | Heuristic; not sourced from Google | | AI insights | LLM-generated content (optional) | +--- + ## Client-facing wording -Prefer industry-standard terms in UI copy: +Use industry-standard terms in UI copy and exports: | Avoid | Prefer | |-------|--------| @@ -94,4 +114,4 @@ Prefer industry-standard terms in UI copy: | Inspector | URL details | | Heuristic (alone) | Estimated from crawl | -Python stores canonical category titles in `src/website_profiling/reporting/terminology.py`. New audits use those names in `categories[].name`; older audits may still have legacy names (mapped in export and the web UI). +**Implementation:** Canonical category titles are defined in `src/website_profiling/reporting/terminology.py`. New audits use those names in `categories[].name`. Legacy report names are mapped in export and the web UI. diff --git a/docs/MCP.md b/docs/MCP.md index b39b8bab..25e9f7b5 100644 --- a/docs/MCP.md +++ b/docs/MCP.md @@ -1,28 +1,67 @@ -# Site Audit MCP server +# MCP Server Reference -Read-only [Model Context Protocol](https://modelcontextprotocol.io) tools for querying audit data from Cursor, Claude Desktop, or other MCP clients. +Site Audit exposes **340 read-only tools** via the [Model Context Protocol (MCP)](https://modelcontextprotocol.io). Connect from Cursor, Claude Desktop, or any MCP-compatible client to query audit data programmatically. -## Install +The same tool catalog powers in-app **AI Chat** at `/chat`. + +**Related documentation:** [GLOSSARY.md](GLOSSARY.md) · [Documentation index](README.md) + +--- + +## Table of contents + +- [Prerequisites](#prerequisites) +- [Domain-scoped servers](#domain-scoped-servers) +- [Configuration](#configuration) +- [MCP resources](#mcp-resources) +- [Tool reference](#tool-reference) +- [In-app chat](#in-app-chat) +- [Provider notes](#provider-notes) +- [Roadmap](#roadmap) +- [Example prompts](#example-prompts) + +--- + +## Prerequisites ```bash pip install -r requirements.txt -export DATABASE_URL=postgres://profiling:profiling@localhost:5432/website_profiling +export DATABASE_URL=postgres://profiling:profiling@localhost:5432/website_profiling # Docker default +# ./local-run default: postgres://postgres:dev@127.0.0.1:5432/website_profiling export PYTHONPATH=src ``` -## Cursor configuration +Start the server: + +```bash +python -m website_profiling.mcp +``` + +--- + +## Domain-scoped servers + +Rather than loading all 340 tools in a single server, Site Audit supports **domain-scoped bundles**. Connect only the domains relevant to your workflow. + +| `WP_MCP_DOMAIN` | Tool count | Scope | Recommended use | +|-----------------|------------|-------|-----------------| +| `core` (default) | Tier 0 + `core`/`insight` domains | Router, workflows, insight | General queries, tool search, coverage reports | +| `crawl` | Domain subset | Crawl, on-page, schema, accessibility | Technical crawl audits | +| `google` | Domain subset | Google, insight, CTR, keywords | GSC/GA4 analysis | +| `links` | Domain subset | Links, backlinks, indexation | Link architecture | +| `full` | 340 | All tools | Debugging, legacy single-server setup | + +Tier 0 alone includes 16 router/insight tools (`TIER_0_TOOLS` in `tool_domains.py`). Use the `audit://tools` resource or `WP_MCP_DOMAIN=full` for the complete catalog. + +Set `WP_PROPERTY_ID` to the default property when tools omit an explicit `property_id` argument. -Site Audit exposes **domain-scoped MCP servers** (like Cursor optional plugins). Connect only the bundles you need instead of loading all 340 tools in one server. +--- -| `WP_MCP_DOMAIN` | Typical tools | Use when | -|-----------------|---------------|----------| -| `core` (default) | Router, workflows, insight (~12) | General chat, tool search, coverage | -| `crawl` | Crawl, on-page, schema, accessibility | Technical crawl audits | -| `google` | Google, insight, CTR, keywords | GSC/GA4 analysis | -| `links` | Links, backlinks, indexation | Link architecture | -| `full` | All 340 tools | Debugging / legacy single-server setup | +## Configuration -Add to `.cursor/mcp.json` (or Cursor MCP settings): +### Multi-domain setup (recommended) + +Add to `.cursor/mcp.json` or your MCP client settings: ```json { @@ -51,7 +90,7 @@ Add to `.cursor/mcp.json` (or Cursor MCP settings): } ``` -Single-server legacy setup (all tools): +### Single-server setup (all tools) ```json { @@ -70,37 +109,43 @@ Single-server legacy setup (all tools): } ``` -`WP_PROPERTY_ID` sets the default property when tools omit `property_id`. +--- ## MCP resources | URI | Content | |-----|---------| | `audit://properties` | JSON list of properties | -| `audit://property/{id}` | Property details + latest report summary | +| `audit://property/{id}` | Property details and latest report summary | | `audit://property/{id}/report/latest` | Payload key index (counts, not full blob) | | `audit://property/{id}/report/{report_id}` | Payload key index for a specific report | -| `audit://glossary` | Excerpt from `docs/GLOSSARY.md` | -| `audit://tools` | Tool catalog for the connected `WP_MCP_DOMAIN` server | +| `audit://glossary` | Excerpt from [GLOSSARY.md](GLOSSARY.md) | +| `audit://tools` | Tool catalog for the connected `WP_MCP_DOMAIN` | | `audit://domains` | Available MCP domain bundles and tool groupings | -## Tools (340 read-only + export) +--- + +## Tool reference + +All tools are read-only. This section is a **curated subset** of the 340-tool registry. For the complete catalog, connect with `WP_MCP_DOMAIN=full` or read the `audit://tools` MCP resource. + +Export tools write artifact files with a 24-hour TTL; in-app chat renders download buttons via `/api/chat/artifacts/{id}`. -### Router and insight (Tier 0 — `WP_MCP_DOMAIN=core`) +### Router and insight (Tier 0 — included in every chat turn) -`search_audit_tools`, `list_tool_domains`, `get_data_coverage_report`, `run_insight_workflow`, `run_technical_workflow`, `run_keyword_workflow`, `run_domain_agent`, `get_landing_page_blended_table`, `get_opportunity_matrix`, `get_traffic_health_check`, `get_landing_page_full_diagnosis`, `get_issue_to_traffic_map` +`search_audit_tools`, `list_tool_domains`, `get_data_coverage_report`, `run_insight_workflow`, `run_technical_workflow`, `run_keyword_workflow`, `run_domain_agent`, `get_report_summary`, `list_top_impact_issues`, `prioritize_fix_roadmap`, `get_landing_page_blended_table`, `get_opportunity_matrix`, `get_traffic_health_check`, `get_landing_page_full_diagnosis`, `get_issue_to_traffic_map`, `get_google_summary` ### Export and deliverables `export_audit_report`, `export_compare_csv`, `export_list_as_csv`, `export_sitemap_xml`, `validate_rich_results`, `compose_custom_report`, `export_custom_report`, `list_export_formats` -Full audit exports reuse the same generators as the Export view (PDF requires `reportlab`). Export tools store files as artifacts (24h TTL); in-app chat renders download buttons via `/api/chat/artifacts/{id}`. +Full audit exports use the same generators as the Export view. PDF export requires `reportlab`. ### Image audit `get_image_audit_summary`, `list_pages_without_lazy_images`, `list_pages_with_images_missing_dimensions`, `list_site_image_urls`, `list_lighthouse_image_opportunities`, `list_largest_images`, `list_unoptimized_images`, `list_images_needing_attention` -Size-based tools require `probe_image_inventory=true` in pipeline config when building the report. Keys: `max_image_probe_urls` (default 500), `image_probe_concurrency`, `image_probe_timeout`, `image_unoptimized_min_kb` (default 200). +Size-based tools require `probe_image_inventory=true` in pipeline config. Related keys: `max_image_probe_urls` (default 500), `image_probe_concurrency`, `image_probe_timeout`, `image_unoptimized_min_kb` (default 200). ### Portfolio and report @@ -170,46 +215,66 @@ Size-based tools require `probe_image_inventory=true` in pipeline config when bu `get_bing_index_status` (requires `bing_webmaster_api_key` in audit settings) -### Ops and logs +### Operations and logs `get_integration_alerts`, `get_property_ops`, `list_crawl_runs`, `list_log_uploads`, `get_latest_log_analysis`, `get_log_top_paths`, `list_log_only_paths`, `list_crawl_only_paths`, `get_log_googlebot_stats`, `get_log_analysis_by_id`, `get_page_coach` -## Future pipeline items +--- -- Full backlink index and anchor-text analytics (beyond GSC Links import) -- SERP rank tracking beyond GSC position snapshots -- Live AI citation checks across ChatGPT/Perplexity (current `check_ai_citation_presence` uses on-site heuristics) +## In-app chat -Already available: `validate_rich_results`, `get_gsc_url_inspection`, `export_sitemap_xml`, workbook export, axe audits via `enable_axe` on browser crawls. +The same tools power **AI Chat** at [http://localhost:3000/chat](http://localhost:3000/chat). Enable a provider under **Run audit → AI settings**. -## Example prompts +In-app chat uses **dynamic tool routing**: each turn loads Tier 0 router tools plus a domain-scoped subset (default ~45 tools via `CHAT_TOOL_MAX`). Set `CHAT_TOOL_MODE=full` to load all tools for debugging. Optional: `CHAT_TOOL_MAX` (default 45, max 120). -- "What indexation gaps exist between crawl and GSC?" -- "List pages missing canonical tags or with canonical mismatches" -- "Which paths appear in access logs but were not crawled?" -- "Compare GSC clicks vs the previous audit" -- "List pages failing Core Web Vitals thresholds" -- "Show security finding changes since report 38" -- "Which pages link to broken URLs?" -- "Generate a content brief for keyword X" -- "Download the audit as PDF" -- "Export broken links as CSV" -- "Compare report 38 to the current audit and give me a CSV diff" -- "Build a client report with executive summary, category scores, and top critical issues as PDF" -- "Which images are largest and unoptimized?" -- "What should we fix first on high-traffic pages?" (use `list_top_impact_issues` or `prioritize_fix_roadmap`) -- "What's our GEO readiness score?" -- "Inspect GSC indexing for https://example.com/page" -- "Which pages are soft 404s or dead ends?" -- "Suggest internal links for our top blog post" -- "List pages with images missing alt or lazy loading" +Responses stream over SSE via `POST /api/chat`. Sessions persist per property in `chat_sessions` and `chat_messages`. -## In-app chat +--- -The same tools power **AI Chat** at [http://localhost:3000/chat](http://localhost:3000/chat). Enable AI in Run audit → AI settings. +## Provider notes -In-app chat uses **dynamic tool routing**: each turn loads Tier 0 router tools plus a domain-scoped subset (~45 tools), not the full catalog. Set `CHAT_TOOL_MODE=full` to load all tools for debugging. +| Provider | Tool calling | Notes | +|----------|--------------|-------| +| **Ollama** | Native when supported; ReAct fallback otherwise | Local daemon at `http://127.0.0.1:11434` | +| **OpenAI** | Native with streaming | API key in AI settings or `OPENAI_API_KEY` | +| **Anthropic** | Native with streaming | API key in AI settings or `ANTHROPIC_API_KEY` | +| **Google Gemini** | Native with streaming | API key in AI settings or `GEMINI_API_KEY`; REST via `httpx` | -## Ollama note +--- + +## Roadmap + +The following capabilities are planned but not yet available: + +| Capability | Current state | +|------------|---------------| +| Full backlink index and anchor-text analytics | GSC Links CSV import only | +| SERP rank tracking | GSC position snapshots only | +| Live AI citation checks | On-site heuristics via `check_ai_citation_presence` | + +**Already available:** `validate_rich_results`, `get_gsc_url_inspection`, `export_sitemap_xml`, workbook export, axe audits via `enable_axe` on browser crawls. + +--- + +## Example prompts -When the local Ollama daemon supports native tools (most current models, including Ollama cloud refs like `minimax-m3:cloud`), chat uses Ollama’s `/api/chat` tool format. Older or tool-less models fall back to JSON ReAct parsing. OpenAI and Anthropic always use native tool calling with streaming in the chat UI. +| Goal | Example prompt | +|------|----------------| +| Indexation | "What indexation gaps exist between crawl and GSC?" | +| On-page | "List pages missing canonical tags or with canonical mismatches" | +| Log analysis | "Which paths appear in access logs but were not crawled?" | +| Google data | "Compare GSC clicks vs the previous audit" | +| Performance | "List pages failing Core Web Vitals thresholds" | +| Security | "Show security finding changes since report 38" | +| Links | "Which pages link to broken URLs?" | +| Content | "Generate a content brief for keyword X" | +| Export | "Download the audit as PDF" | +| Compare | "Compare report 38 to the current audit and give me a CSV diff" | +| Client report | "Build a client report with executive summary, category scores, and top critical issues as PDF" | +| Images | "Which images are largest and unoptimized?" | +| Prioritization | "What should we fix first on high-traffic pages?" | +| GEO | "What's our GEO readiness score?" | +| GSC inspection | "Inspect GSC indexing for https://example.com/page" | +| Crawl quality | "Which pages are soft 404s or dead ends?" | +| Internal links | "Suggest internal links for our top blog post" | +| Accessibility | "List pages with images missing alt or lazy loading" | diff --git a/docs/OPS.md b/docs/OPS.md index ed1a1069..c283ed9d 100644 --- a/docs/OPS.md +++ b/docs/OPS.md @@ -1,58 +1,152 @@ -# Site Audit — operations +# Operations Guide -Cron-friendly HTTP endpoints for scheduled audits and property alerts. All routes require local access (same host) unless you proxy them behind your own auth. +This guide covers production operations for Site Audit: scheduled audits, property alerts, access control, database migrations, and test execution. + +**Related documentation:** [README.md](../README.md) · [Documentation index](README.md) + +--- + +## Overview + +Site Audit exposes HTTP endpoints suitable for cron and monitoring systems. By default, these routes accept requests from localhost only. When exposing the application beyond a single host, place the endpoints behind your own authentication and network controls. + +| Capability | Endpoint | Typical schedule | +|------------|----------|------------------| +| Scheduled audits | `POST /api/schedule/check` | Weekly or daily | +| Property alerts | `POST /api/alerts/check?propertyId={id}` | Daily | + +Configure per-property schedules and webhooks under **Integrations → Scheduled audits & alerts**. + +--- ## Scheduled audits -**Endpoint:** `POST /api/schedule/check` +### Endpoint -Runs `schedule_runner.py`, which: +``` +POST /api/schedule/check +``` + +### Behavior -1. Matches each property’s `schedule_cron` (UTC, five-field cron) against the current minute. +The endpoint invokes `schedule_runner.py`, which: + +1. Evaluates each property's `schedule_cron` expression (UTC, five-field cron syntax) against the current minute. 2. Spawns a full audit (`python -m src`) with `WP_PROPERTY_ID` and `WP_SCHEDULED_SPAWN=1`. -3. The child process **reads** `pipeline_config` only for shared integration keys (Google, etc.). Crawl settings come from the property’s `site_url` and `default_crawl_preset` (starter / spa / ecommerce / performance). **`pipeline_config` is never written or overwritten** — not by cron, not in memory on top of saved workspace keys. +3. Reads `pipeline_config` for shared integration keys (Google, and similar) only. Crawl settings are derived from the property's `site_url` and `default_crawl_preset` (`starter`, `spa`, `ecommerce`, or `performance`). + +**Important:** Scheduled runs never write to or overwrite `pipeline_config`. Manual **Run audit** actions from the web UI also use saved `pipeline_config` without modification. -Manual **Run audit** from the web UI uses saved `pipeline_config` unchanged (no overlay, no env property id on the pipeline spawn). +### Example -**Example (every Monday 06:00 UTC):** +Run scheduled audits every Monday at 06:00 UTC: ```bash # crontab -e 0 6 * * 1 curl -fsS -X POST http://127.0.0.1:3000/api/schedule/check ``` -Response includes `output` (runner log) and `gscLinksStale` (properties needing a GSC Links CSV re-import). +### Response + +The response includes: + +- `output` — runner log +- `gscLinksStale` — properties that require a Google Search Console Links CSV re-import + +--- ## Property alerts -**Endpoint:** `POST /api/alerts/check?propertyId={id}` +### Endpoint + +``` +POST /api/alerts/check?propertyId={id} +``` + +### Behavior + +Evaluates health-score changes and stale GSC Links imports for the specified property. When `alert_webhook_url` is configured on the property, sends a POST notification to that URL. -Checks health-score drops and stale GSC Links imports; POSTs to `alert_webhook_url` when configured on the property. +### Example + +Check alerts daily at 07:00 UTC for property ID 1: ```bash +# crontab -e 0 7 * * * curl -fsS -X POST "http://127.0.0.1:3000/api/alerts/check?propertyId=1" ``` -Configure webhook, email, and cron per property under **Integrations → Scheduled audits & alerts**. +--- + +## Access control + +### Session roles + +When `AUTH_SECRET` (or `SESSION_SECRET`) is set, the application requires login. Roles (`web/src/server/auth.ts`): + +| Role | Mutations | AI Chat | +|------|-----------|---------| +| `analyst` (default) | Allowed | Allowed | +| `editor` | Allowed | Allowed | +| `admin` | Allowed | Allowed | +| `client-readonly` | Blocked (403) | Allowed | +| `viewer` | Blocked (403) | Blocked (403) | + +Set the default role for new sessions: + +``` +AUTH_DEFAULT_ROLE=client-readonly +``` + +Production also requires `AUTH_SECRET` and optionally `AUTH_USER` / `AUTH_PASSWORD` (see `docker-compose.prod.yml`). + +### Read-only client dashboards -## Read-only client access +Set `AUTH_DEFAULT_ROLE=client-readonly` so session logins cannot run audits or save settings. The API returns 403 on mutations; the UI hides **Run audit** and disables save controls. Use `viewer` instead if chat access should also be blocked. -Set `AUTH_DEFAULT_ROLE=client-readonly` so session logins cannot run audits or save settings. The API returns 403 on mutations; the UI hides **Run audit** and disables save controls. +--- ## Database migrations -After pulling roadmap changes, apply Alembic revision `011` (included in the full local/CI test run): +Apply schema changes after pulling updates. Current Alembic head: **`015_crawl_page_html`** (per-URL HTML storage). Recent migrations: `013` (link edges, discovery mode), `014` (pipeline job log truncation). + +```bash +./local-run migrate +``` + +If PostgreSQL is already running: + +```bash +alembic upgrade head +``` + +### Docker deployments + +Migrations run automatically at container start. Use one of the following so Postgres and the application share a network: ```bash -./local-test all -# or, if Postgres is already up: ./local-test quick +docker compose up # build from source +docker compose -f docker-compose.pull.yml up # pre-built WEB_IMAGE ``` -**Docker:** run migrations automatically at container start. Use `docker compose up` (build) or `docker compose -f docker-compose.pull.yml up` (pre-built `WEB_IMAGE`) so Postgres and the app share a network — not standalone `docker run`. +Do not run the application container in isolation with `docker run` unless you provide a reachable `DATABASE_URL`. + +--- ## Running tests -**Python (core, 100% coverage on non-omitted modules):** +For CI parity, run from the repository root: + +```bash +./local-test # Python + web (matches CI python and web jobs) +./local-test python # Backend gates + browser pytest + CLI smoke +``` + +CI also runs a **Docker** job (image build, browser pytest in container, compose smoke). See [.github/workflows/ci.yml](../.github/workflows/ci.yml). + +### Individual test targets + +**Python (core coverage gate — 100%):** ```bash export DATABASE_URL=postgres://profiling:profiling@localhost:5432/website_profiling @@ -60,26 +154,23 @@ alembic upgrade head pytest tests/ -m "not browser" ``` -Integration tests (`@pytest.mark.integration`) skip when `DATABASE_URL` is unset. Browser crawl E2E: +Integration tests marked `@pytest.mark.integration` skip when `DATABASE_URL` is unset. + +**Browser crawl end-to-end:** ```bash pytest tests/test_crawler_browser_e2e.py -m browser ``` -**Reporting and tools** (separate 100% coverage gates, same as CI): +**Reporting and tools coverage gates:** ```bash -pytest tests/test_categories_roadmap.py tests/test_report_categories_golden.py \ - tests/test_categories_coverage.py tests/test_indexation_coverage.py tests/test_crawl_segments.py \ - tests/test_terminology.py \ - --cov=website_profiling.reporting --cov-config=.coveragerc.reporting --cov-fail-under=100 -o addopts= - -pytest tests/test_alert_checker.py tests/test_schedule_runner.py tests/test_export_audit.py \ - tests/test_export_audit_coverage.py \ - --cov=website_profiling.tools --cov-config=.coveragerc.tools --cov-fail-under=100 -o addopts= +./local-test python ``` -**Web (Vitest route and lib tests):** +Test file lists for reporting and tools gates are maintained in [scripts/local-test.sh](../scripts/local-test.sh) and [.github/workflows/ci.yml](../.github/workflows/ci.yml). Update all three locations when adding coverage tests. + +**Web (Vitest):** ```bash cd web && npm test diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..b28f6d15 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,41 @@ +# Site Audit — Documentation + +This directory contains product, integration, and operations documentation for **Site Audit** (repository: WebsiteProfiling), a self-hosted SEO crawl and technical audit platform. + +--- + +## Document index + +| Document | Audience | Description | +|----------|----------|-------------| +| [README.md](../README.md) | All users | Product overview, installation, configuration | +| [CONTRIBUTING.md](../CONTRIBUTING.md) | Contributors | Development setup, testing, pull request guidelines | +| [AGENT.md](../AGENT.md) | Developers | Repository layout, APIs, edit targets | +| [GLOSSARY.md](GLOSSARY.md) | Product / UX | UI terminology mapped to internal keys and data sources | +| [COMPANY_STANDARDS.md](COMPANY_STANDARDS.md) | Agencies / operators | Data classification, crawl scope, security policy | +| [MCP.md](MCP.md) | Integrators | Model Context Protocol server configuration and tool reference | +| [OPS.md](OPS.md) | Operators | Scheduled audits, alerts, migrations, production notes | + +--- + +## Brand assets + +Marketing and README assets are stored in [assets/](assets/): + +| Asset | Purpose | +|-------|---------| +| `readme-banner.png` | README header banner | +| `social-preview.png` | Application screenshot for README and social previews | +| `banner.svg` | Source artwork for the banner | +| `logo.svg`, `logo-icon.svg` | Product logo and icon | +| `icon-crawl.svg`, `icon-audit.svg`, `icon-integrations.svg`, `icon-self-hosted.svg` | Feature icons for README | + +--- + +## Related repository files + +| File | Description | +|------|-------------| +| [SECURITY.md](../SECURITY.md) | Vulnerability reporting policy | +| [CODE_OF_CONDUCT.md](../CODE_OF_CONDUCT.md) | Community standards | +| [pipeline-config.example.txt](../pipeline-config.example.txt) | Pipeline configuration key reference | diff --git a/web/README.md b/web/README.md index 66bb426f..0cc2f8e5 100644 --- a/web/README.md +++ b/web/README.md @@ -1,36 +1,41 @@ -This is a [Next.js](https://nextjs.org) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app). +# Site Audit — Web UI -## Getting Started +Next.js frontend for [Site Audit](../README.md). The app reads audit data from PostgreSQL and spawns the Python pipeline for crawl and report jobs. -First, run the development server: +## Development + +Use the repo root scripts — do not run `npm run dev` in isolation unless Postgres is already up: ```bash -npm run dev -# or -yarn dev -# or -pnpm dev -# or -bun dev +./local-run setup # first time +./local-run # http://localhost:3000/home ``` -Open [http://localhost:3000](http://localhost:3000) with your browser to see the result. - -You can start editing the page by modifying `app/page.js`. The page auto-updates as you edit the file. - -This project uses [`next/font`](https://nextjs.org/docs/app/building-your-application/optimizing/fonts) to automatically optimize and load [Geist](https://vercel.com/font), a new font family for Vercel. +## Structure -## Learn More +| Path | Purpose | +|------|---------| +| `app/` | App Router pages and `/api` route handlers | +| `src/components/` | Shared React components | +| `src/views/` | Report views (overview, issues, links, …) | +| `src/server/` | DB access, pipeline jobs, config I/O | +| `src/lib/` | Schemas (`pipelineConfigSchema.ts`, `llmConfigSchema.ts`) | +| `public/` | Static assets (logo, favicon) | -To learn more about Next.js, take a look at the following resources: +## Commands -- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API. -- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial. +Run from `web/`: -You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js) - your feedback and contributions are welcome! +```bash +npm run typecheck +npm run lint +npm test +``` -## Deploy on Vercel +Full CI parity from repo root: `./local-test web`. -The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js. +## Further reading -Check out our [Next.js deployment documentation](https://nextjs.org/docs/app/building-your-application/deploying) for more details. +- [README.md](../README.md) — setup and configuration +- [AGENT.md](../AGENT.md) — API routes, React footguns, where to edit +- [docs/GLOSSARY.md](../docs/GLOSSARY.md) — UI terminology (`web/src/strings.json`)