Removing ml and adding llm instead

.coveragerc

@@ -0,0 +1,21 @@
+[run]
+source = website_profiling
+
+# We are NOT hiding core code. We only omit modules that require external
+# services/binaries (Google APIs, Lighthouse) or are impractical to unit-test here.
+omit =
+    */website_profiling/integrations/google/*
+    */website_profiling/lighthouse/*
+    */website_profiling/reporting/*
+    */website_profiling/tools/*
+    */website_profiling/security_scanner.py
+    */website_profiling/llm/providers/*
+    */website_profiling/llm/*
+    */website_profiling/llm_config.py
+    */website_profiling/cli.py
+    */website_profiling/commands/enrich_cmd.py
+
+[report]
+show_missing = True
+skip_empty = True
+

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,56 @@
+name: Bug report
+description: Something is broken or incorrect
+title: "[Bug]: "
+labels: ["bug"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for reporting. Please search existing issues first.
+        Do **not** use this form for security vulnerabilities — see [SECURITY.md](../../SECURITY.md).
+  - type: textarea
+    id: description
+    attributes:
+      label: What happened?
+      description: Clear description of the bug
+    validations:
+      required: true
+  - type: textarea
+    id: expected
+    attributes:
+      label: What did you expect?
+    validations:
+      required: true
+  - type: textarea
+    id: reproduce
+    attributes:
+      label: Steps to reproduce
+      placeholder: |
+        1. Run ./local-run
+        2. Open Overview for example.com
+        3. See error ...
+    validations:
+      required: true
+  - type: dropdown
+    id: area
+    attributes:
+      label: Area
+      options:
+        - Web UI
+        - Python / crawl / report
+        - Database / migrations
+        - Google integrations
+        - Docker / deploy
+        - Other
+    validations:
+      required: true
+  - type: input
+    id: version
+    attributes:
+      label: Commit or version (if known)
+      placeholder: git SHA or tag
+  - type: textarea
+    id: logs
+    attributes:
+      label: Logs or screenshots
+      description: Paste relevant errors or attach screenshots

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,8 @@
+blank_issues_enabled: false
+contact_links:
+  - name: Contributing guide
+    url: https://github.com/codefrydev/WebsiteProfiling/blob/master/CONTRIBUTING.md
+    about: Setup, tests, and pull request expectations
+  - name: Code of Conduct
+    url: https://github.com/codefrydev/WebsiteProfiling/blob/master/CODE_OF_CONDUCT.md
+    about: Community standards

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,33 @@
+name: Feature request
+description: Suggest an improvement or new capability
+title: "[Feature]: "
+labels: ["enhancement"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Describe the problem you are trying to solve. Feature requests that fit the project goal (free, honest technical SEO audits) are most likely to be accepted.
+  - type: textarea
+    id: problem
+    attributes:
+      label: Problem
+      description: What limitation or pain point does this address?
+    validations:
+      required: true
+  - type: textarea
+    id: solution
+    attributes:
+      label: Proposed solution
+    validations:
+      required: true
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: Alternatives considered
+  - type: checkboxes
+    id: terms
+    attributes:
+      label: Contribution
+      options:
+        - label: I am willing to help implement this via a pull request
+          required: false

.github/pull_request_template.md

@@ -0,0 +1,27 @@
+## Summary
+
+<!-- What does this PR change and why? -->
+
+## Type of change
+
+- [ ] Bug fix
+- [ ] New feature
+- [ ] Documentation
+- [ ] Refactor / chore
+- [ ] Database migration
+
+## How to test
+
+<!-- Steps for reviewers, e.g. ./local-run, run audit on X, expect Y -->
+
+## Checklist
+
+- [ ] I read [CONTRIBUTING.md](../CONTRIBUTING.md) and followed it
+- [ ] `./local-test` or an equivalent subset passes locally
+- [ ] UI copy updates are in `web/src/strings.json` (if applicable)
+- [ ] No secrets or credentials in the diff
+- [ ] Alembic migration included if the schema changed
+
+## Screenshots (UI only)
+
+<!-- If applicable -->

.github/workflows/ci.yml

@@ -0,0 +1,67 @@
+name: CI
+
+on:
+  push:
+    branches: [ master]
+  pull_request:
+
+jobs:
+  python:
+    runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres:16-alpine
+        env:
+          POSTGRES_USER: profiling
+          POSTGRES_PASSWORD: profiling
+          POSTGRES_DB: website_profiling
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U profiling -d website_profiling"
+          --health-interval 5s
+          --health-timeout 3s
+          --health-retries 5
+    env:
+      DATABASE_URL: postgres://profiling:profiling@localhost:5432/website_profiling
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - name: Install dependencies
+        run: pip install -r requirements.txt
+      - name: Apply migrations
+        run: alembic upgrade head
+      - name: Pytest
+        run: pytest tests/ -q
+      - name: CLI smoke
+        run: python -m src --help
+
+  docker:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Build image
+        run: docker build -t website-profiling:ci .
+
+  web:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: web
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+          cache: npm
+          cache-dependency-path: web/package-lock.json
+      - name: Install
+        run: npm ci
+      - name: Typecheck
+        run: npm run typecheck
+      - name: Lint
+        run: npm run lint
+      - name: Test
+        run: npm test

.gitignore

@@ -2,19 +2,29 @@
 
 # Next.js UI: generated pipeline configs from the runner modal (repo root; must match Python cwd for paths)
 .website-profiling-ui-*.txt
-# Legacy: configs under subfolder broke sqlite_db = report.db vs Next (kept for ignore if present)
+# Legacy UI pipeline configs under subfolder (kept for ignore if present)
 .web-pipeline/
 
-# WebsiteProfiling generated outputs
+# Local runtime data (shadow config, secrets volume mount)
+data/*
+!data/.gitkeep
+
+# Python virtual environments (README recommends .venv; ignore both names at repo root)
+.venv/
+venv/
+
+# Legacy SQLite outputs (if present locally)
 report.db
 report.db.*
-
 report.db-journal
 nodes.json
 
 # Google integration secrets (never commit)
 .secrets/
 web/.env.local
 
-# Legacy local config file (use report.db / web UI instead)
+# Legacy local config file (use PostgreSQL pipeline_config / web UI instead)
 input.txt
+# Runtime shadow written to DATA_DIR by the web UI (not the committed example)
+pipeline-config.txt
+*__pycache__*

AGENT.md

@@ -1,34 +1,45 @@
-# Agent instructions -- WebsiteProfiling
+# Agent instructions — Site Audit (WebsiteProfiling)
 
-**What it is:** `python -m src` from repo root (`src/__main__.py` -> package **`website_profiling`**). Config: stored in **`report.db`** (`pipeline_config` table, `key/value/is_unknown/updated_at`). A shadow **`pipeline-config.txt`** is auto-written next to `report.db` on every Save/Run. CLI loads DB first (`REPORT_DB_PATH` or `cwd/report.db`), then shadow file; `--config` overrides with a file. Reference keys: `input.txt.example` (not auto-loaded).
+**What it is:** `python -m src` from repo root (`src/__main__.py` -> package **`website_profiling`**). Config: stored in **PostgreSQL** (`pipeline_config` table, `key/value/is_unknown/updated_at`). A shadow **`pipeline-config.txt`** is auto-written to `DATA_DIR` on every Save/Run. CLI loads DB first (`DATABASE_URL`), then shadow file; `--config` overrides with a file. Reference keys: `input.txt.example` and `pipeline-config.example.txt` (not auto-loaded).
 
-**Frontend:** **`web/`** (Next.js) -- server reads `report.db` via `/api/report/*`.
+**LLM / AI:** Settings live in **`llm_config`** table in PostgreSQL. Configure only via web UI **AI** tab (`GET/PUT /api/llm-config`, localhost). Never in `pipeline-config.txt` or `--config`.
+
+**Frontend:** **`web/`** (Next.js) -- server reads PostgreSQL via `/api/report/*`.
 
 **Key paths**
 
-- `src/website_profiling/` -- `cli.py`, `config.py`, `crawl/`, `db/storage.py`, `lighthouse/`, `reporting/`, `ml/enrich.py`, `tools/`
-- `web/app/` -- routes; `web/src/` -- React; pipeline: `PipelineRunnerFab`, `server/pipelineJobs.js`, `server/pipelineConfig.js`
+- `src/website_profiling/` -- `cli.py`, `config.py`, `crawl/`, `db/storage.py`, `lighthouse/`, `reporting/`, `analysis/`, `llm/`, `tools/`
+- `web/app/` -- routes; `web/src/` -- React; pipeline: `PipelineRunnerFab`, `server/pipelineJobs.ts`, `server/pipelineConfig.ts`, `server/llmConfig.ts`, `server/db.ts`
+- `alembic/` -- schema migrations
+
+**Local dev:** `./local-run` (Postgres in Docker `wp-pg`, Next.js on host). See `scripts/local-run.sh`. **Local tests (CI parity):** `./local-test` — see `scripts/local-test.sh`.
 
 **Run / APIs**
 
-- Pipeline: `python -m src` — reads config from `report.db` (`pipeline_config`); shadow `pipeline-config.txt` if table empty. CLI override: `python -m src --config path`
-- Optional step: `crawl` | `report` | `plot` | `lighthouse` | `keywords` | `warnings` | `enrich`
-- **`preserve_crawl_history`** (default true): append crawls; `false` recreates crawl tables but restores `report_payload`, Lighthouse, `google_data`, `keyword_data`, `keyword_history`, `keyword_suggest_cache`, and `crawl_runs`
-- **`enrich_keywords_after_report`**: when omitted or `auto` (UI: Auto), follows `enable_google_search_console`; when set to Yes/No, explicit override
-- **`REPORT_DB_PATH`** env: DB path used by both Python and Next.js (Docker: `/data/report.db`; local default: `report.db` at repo root). Pipeline config lives in this DB.
-- **`web/`:** `/api/report/*` (SQLite); `/api/run` spawns Python (localhost only); `/api/pipeline-config` GET/PUT for persistent settings; `PipelineRunnerFab` saves state to `report.db` (`pipeline_config` table) + shadow `pipeline-config.txt` before each run
-- **Docker:** `Dockerfile` + `docker-compose.yml`; **`LIGHTHOUSE_CHROME_FLAGS`**; ML caches under `/data/cache/*` in compose
+- Run audit (CLI): `python -m src` — reads config from PostgreSQL (`pipeline_config`); shadow `DATA_DIR/pipeline-config.txt` if table empty. CLI override: `python -m src --config path`
+- Optional step: `crawl` | `report` | `plot` | `lighthouse` | `keywords` | `warnings` | `enrich` | `google`
+- **`preserve_crawl_history`** (default true): append crawls; `false` truncates crawl tables but restores `report_payload`, Lighthouse, `google_data`, `keyword_data`, `keyword_history`, `keyword_suggest_cache`, and `crawl_runs`
+- **`DATABASE_URL`** env: PostgreSQL connection string (required). **`DATA_DIR`**: secrets + shadow config (Docker: `/data`).
+- **Pipeline data** (crawl, edges, nodes, report payload, Lighthouse, keywords, warnings) is stored in **PostgreSQL only** — no JSON/CSV/HTML exports from the main pipeline.
+- **Pool tuning:** `DB_POOL_MIN` / `DB_POOL_MAX` (Python), `PGPOOL_MAX` (Node). Bulk crawl writes via `executemany`; optional **`crawl_stream_to_db`** streams rows during fetch.
+- **`web/`:** `/api/report/*` (PostgreSQL); `/api/run` spawns Python (localhost only); `/api/pipeline-config` GET/PUT; `/api/llm-config` GET/PUT (AI only); `PipelineRunnerFab` saves pipeline + LLM state before each run
+- **Job store:** in-memory on `globalThis` in `web/src/server/pipelineJobs.ts` — job status/log is lost on server restart (single-process dev/Docker only).
+- **Docker:** `Dockerfile` + `docker-compose.yml` (postgres + web); **`LIGHTHOUSE_CHROME_FLAGS`**
 
 **Where to edit**
 
 | Task | Where |
 |------|--------|
 | Crawl | `crawl/crawler.py` |
 | Report | `reporting/builder.py`, `reporting/categories.py` |
-| DB schema | `db/storage.py` `init_schema` |
-| ML | `ml/enrich.py`, `requirements-ml.txt` |
+| DB schema | `alembic/versions/` |
+| Local analysis | `analysis/local.py`, `requirements.txt` |
+| AI insights (LLM) | `llm/enrich.py`, `llm_config.py`, `requirements-llm.txt` |
 | Config / CLI | `config.py` (`load_config`, `load_config_from_db`), `cli.py`, `input.txt.example` |
-| UI config schema | `web/src/lib/pipelineConfigSchema.js` |
-| UI config I/O | `web/src/server/pipelineConfig.js` |
+| UI pipeline schema | `web/src/lib/pipelineConfigSchema.ts` |
+| UI LLM schema | `web/src/lib/llmConfigSchema.ts` |
+| UI config I/O | `web/src/server/pipelineConfig.ts`, `web/src/server/llmConfig.ts` |
+
+Schema changes: add Alembic migration (`alembic revision`).
 
-Schema changes: edit `init_schema` only (no migration layer). ML stack: prefer Python **3.12** for spaCy/blis; **3.13** may fail pip builds.
+**Company standards:** UI copy in `web/src/strings.json` (Site Audit, Properties, Run audit). Data provenance on `report_meta` in report payload. Docs: `docs/COMPANY_STANDARDS.md`, `docs/GLOSSARY.md`. Migration `003_company_standards` (properties, pipeline_jobs, audit_log). Durable jobs in `web/src/server/pipelineJobsDb.ts`. Export: `GET /api/report/export`, `src/website_profiling/tools/export_audit.py`.