better tools calling

.github/workflows/ci.yml

@@ -43,6 +43,8 @@ jobs:
             tests/test_indexation_coverage.py tests/test_crawl_segments.py \
             tests/test_terminology.py tests/test_compare_payload.py \
             tests/test_optional_audits.py tests/test_property_profile.py tests/test_reporting_gaps.py \
+            tests/test_text_content_analysis.py tests/test_builder_image_buckets.py \
+            tests/test_pipeline_report_pool_unit.py tests/test_reporting_builder_modules.py \
             --cov=website_profiling.reporting --cov-config=.coveragerc.reporting \
             --cov-report=term-missing --cov-fail-under=100 -q -o addopts=
       - name: Pytest (tools coverage gate)
@@ -51,11 +53,13 @@ jobs:
             tests/test_export_audit_coverage.py tests/test_audit_tools.py tests/test_audit_tools_expanded.py \
             tests/test_audit_tools_coverage.py tests/test_audit_tools_dispatch_coverage.py \
             tests/test_audit_tools_links_extras.py tests/test_audit_tools_expansion.py \
-            tests/test_audit_tools_expansion_coverage.py tests/test_export_custom_coverage.py \
+            tests/test_audit_tools_expansion_coverage.py tests/test_audit_tools_batch100_coverage.py tests/test_export_custom_coverage.py \
             tests/test_export_artifacts_coverage.py tests/test_export_compare_coverage.py \
             tests/test_export_tools_coverage.py tests/test_image_tools.py tests/test_export_custom.py \
             tests/test_export_artifacts.py tests/test_export_compare.py tests/test_export_workbook.py \
-            tests/test_export_sitemap.py tests/test_mcp_registry.py tests/test_mcp_resources.py \
+            tests/test_export_sitemap.py             tests/test_mcp_registry.py tests/test_mcp_resources.py \
+            tests/test_router_tools.py tests/test_tool_selector.py \
+            tests/test_tools_gate100_coverage.py \
             tests/test_tools_branch_coverage.py \
             --cov=website_profiling.tools --cov-config=.coveragerc.tools \
             --cov-report=term-missing --cov-fail-under=100 -q -o addopts=

AGENT.md

@@ -25,7 +25,7 @@
 - **Pipeline data** (crawl, edges, nodes, report payload, Lighthouse, keywords, warnings) is stored in **PostgreSQL only** — no JSON/CSV/HTML exports from the main pipeline.
 - **Pool tuning:** `DB_POOL_MIN` / `DB_POOL_MAX` (Python), `PGPOOL_MAX` (Node). Bulk crawl writes via `executemany`; optional **`crawl_stream_to_db`** streams rows during fetch.
 - **`web/`:** `/api/report/*` (PostgreSQL); `/api/run` spawns Python (localhost only); `/api/crawl/browser-status` GET (localhost, Playwright/Chromium preflight); `/api/pipeline-config` GET/PUT; `/api/llm-config` GET/PUT (AI only); `/api/chat` POST (SSE agent); `/api/chat/sessions` GET/POST; `/api/properties/{id}/google/links/import` POST (GSC Links CSV); `PipelineRunnerFab` saves pipeline + LLM state before each run
-- **MCP:** `python -m website_profiling.mcp` (stdio, **221 read-only audit tools** + MCP resources). See `docs/MCP.md`. Requires `pip install -r requirements.txt`.
+- **MCP:** `python -m website_profiling.mcp` (stdio, **340 read-only audit tools**, domain-scoped via `WP_MCP_DOMAIN`). See `docs/MCP.md`. Requires `pip install -r requirements.txt`.
 - **AI Chat UI:** `/chat` — property-scoped chat with saved sessions (`chat_sessions`, `chat_messages` tables, migration `012_chat_sessions`).
 - **Job store:** in-memory on `globalThis` in `web/src/server/pipelineJobs.ts` — job status/log is lost on server restart (single-process dev/Docker only).
 - **Docker:** `Dockerfile` + `docker-compose.yml` (postgres + web); **`docker-compose.pull.yml`** for pre-built images (`WEB_IMAGE`); **`LIGHTHOUSE_CHROME_FLAGS`**

README.md

@@ -73,7 +73,7 @@ Repository: [codefrydev/WebsiteProfiling](https://github.com/codefrydev/WebsiteP
   </tr>
 </table>
 
-Also included: **AI chat** over audit data (optional), **221 MCP tools**, keyword explorer, backlinks, compare runs, and portfolio management for agencies.
+Also included: **AI chat** over audit data (optional), **340 MCP tools** (domain-scoped servers), keyword explorer, backlinks, compare runs, and portfolio management for agencies.
 
 <p align="center">
   <img src="docs/assets/social-preview.png" alt="Site Audit preview" width="640">
@@ -91,7 +91,7 @@ WebsiteProfiling/
 │   ├── integrations/          # Google Search Console, GA4, Bing, CrUX
 │   ├── llm/                   # AI enrich + chat agent
 │   ├── tools/                 # Exports, audit query tools, MCP helpers
-│   ├── mcp/                   # MCP server (221 read-only tools)
+│   ├── mcp/                   # MCP server (340 read-only tools, domain bundles)
 │   ├── db/                    # PostgreSQL storage layer
 │   ├── commands/              # CLI subcommands
 │   ├── cli.py                 # Pipeline entrypoint
@@ -147,6 +147,10 @@ Open [http://localhost:3000/home](http://localhost:3000/home).
 ./local-run stop    # stop Postgres container
 ```
 
+`requirements.txt` pins direct Python dependencies to versions verified by `./local-test python`. Re-run the full test suite after intentional upgrades.
+
+Pipeline jobs: stuck `running` rows are reconciled after **1 hour** by default (`PIPELINE_JOB_STALE_HOURS`). Orphan jobs with no live server process are cleared after **5 minutes** (`PIPELINE_JOB_ORPHAN_MINUTES`). Increase `PIPELINE_JOB_STALE_HOURS` for crawls that routinely run longer than an hour.
+
 **Tests**
 
 ```bash
@@ -182,7 +186,7 @@ Google Search Console / Analytics: connect via **Integrations** (gear icon) in t
 | **Ollama** | Local daemon at `http://127.0.0.1:11434`. Chat UI lists installed models plus the live Ollama cloud catalog (billing: free local, account free tier, Pro). Native tool calling when supported; otherwise ReAct fallback. Pick the model in-chat without leaving the page. |
 | **OpenAI** / **Anthropic** | API key in AI settings; native tool calling with streaming. |
 
-The agent uses the same **221 read-only audit tools** as the MCP server (`docs/MCP.md`). Responses stream over SSE (`POST /api/chat`) with status, tool activity, and tokens. Sessions are saved per property (`chat_sessions` / `chat_messages`).
+The agent uses the same **340 read-only audit tools** as the MCP server (`docs/MCP.md`), with **dynamic routing** (~45 tools per turn plus router meta-tools). Responses stream over SSE (`POST /api/chat`) with status, tool activity, and tokens. Sessions are saved per property (`chat_sessions` / `chat_messages`).
 
 Production: `docker-compose.prod.yml` (set `POSTGRES_PASSWORD`, `AUTH_SECRET`).
 

alembic/versions/014_pipeline_job_log_truncated.py

@@ -0,0 +1,27 @@
+"""Add log_truncated flag to pipeline_jobs.
+
+Revision ID: 014_pipeline_log_truncated
+Revises: 013_crawl_discovery_edges
+"""
+from __future__ import annotations
+
+from alembic import op
+
+revision = "014_pipeline_log_truncated"
+down_revision = "013_crawl_discovery_edges"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.execute("""
+        ALTER TABLE pipeline_jobs
+            ADD COLUMN IF NOT EXISTS log_truncated BOOLEAN NOT NULL DEFAULT false;
+    """)
+
+
+def downgrade() -> None:
+    op.execute("""
+        ALTER TABLE pipeline_jobs
+            DROP COLUMN IF EXISTS log_truncated;
+    """)

alembic/versions/015_crawl_page_html.py

@@ -0,0 +1,38 @@
+"""Add crawl_page_html table for per-URL raw HTML storage.
+
+Revision ID: 015_crawl_page_html
+Revises: 014_pipeline_log_truncated
+"""
+from __future__ import annotations
+
+from alembic import op
+
+revision = "015_crawl_page_html"
+down_revision = "014_pipeline_log_truncated"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.execute("""
+        CREATE TABLE crawl_page_html (
+            crawl_run_id BIGINT NOT NULL REFERENCES crawl_runs(id) ON DELETE CASCADE,
+            url TEXT NOT NULL,
+            html TEXT NOT NULL,
+            status TEXT,
+            content_type TEXT,
+            fetch_method TEXT NOT NULL DEFAULT 'static',
+            byte_length INTEGER NOT NULL DEFAULT 0,
+            captured_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+            PRIMARY KEY (crawl_run_id, url)
+        );
+        CREATE INDEX IF NOT EXISTS idx_crawl_page_html_run
+            ON crawl_page_html (crawl_run_id);
+    """)
+
+
+def downgrade() -> None:
+    op.execute("""
+        DROP INDEX IF EXISTS idx_crawl_page_html_run;
+        DROP TABLE IF EXISTS crawl_page_html;
+    """)

docs/MCP.md

@@ -12,8 +12,47 @@ export PYTHONPATH=src
 
 ## Cursor configuration
 
+Site Audit exposes **domain-scoped MCP servers** (like Cursor optional plugins). Connect only the bundles you need instead of loading all 340 tools in one server.
+
+| `WP_MCP_DOMAIN` | Typical tools | Use when |
+|-----------------|---------------|----------|
+| `core` (default) | Router, workflows, insight (~12) | General chat, tool search, coverage |
+| `crawl` | Crawl, on-page, schema, accessibility | Technical crawl audits |
+| `google` | Google, insight, CTR, keywords | GSC/GA4 analysis |
+| `links` | Links, backlinks, indexation | Link architecture |
+| `full` | All 340 tools | Debugging / legacy single-server setup |
+
 Add to `.cursor/mcp.json` (or Cursor MCP settings):
 
+```json
+{
+  "mcpServers": {
+    "site-audit-core": {
+      "command": "python",
+      "args": ["-m", "website_profiling.mcp"],
+      "env": {
+        "DATABASE_URL": "postgres://profiling:profiling@localhost:5432/website_profiling",
+        "PYTHONPATH": "src",
+        "WP_MCP_DOMAIN": "core",
+        "WP_PROPERTY_ID": "1"
+      }
+    },
+    "site-audit-google": {
+      "command": "python",
+      "args": ["-m", "website_profiling.mcp"],
+      "env": {
+        "DATABASE_URL": "postgres://profiling:profiling@localhost:5432/website_profiling",
+        "PYTHONPATH": "src",
+        "WP_MCP_DOMAIN": "google",
+        "WP_PROPERTY_ID": "1"
+      }
+    }
+  }
+}
+```
+
+Single-server legacy setup (all tools):
+
 ```json
 {
   "mcpServers": {
@@ -23,6 +62,7 @@ Add to `.cursor/mcp.json` (or Cursor MCP settings):
       "env": {
         "DATABASE_URL": "postgres://profiling:profiling@localhost:5432/website_profiling",
         "PYTHONPATH": "src",
+        "WP_MCP_DOMAIN": "full",
         "WP_PROPERTY_ID": "1"
       }
     }
@@ -41,9 +81,14 @@ Add to `.cursor/mcp.json` (or Cursor MCP settings):
 | `audit://property/{id}/report/latest` | Payload key index (counts, not full blob) |
 | `audit://property/{id}/report/{report_id}` | Payload key index for a specific report |
 | `audit://glossary` | Excerpt from `docs/GLOSSARY.md` |
-| `audit://tools` | Tool catalog grouped by SEO domain |
+| `audit://tools` | Tool catalog for the connected `WP_MCP_DOMAIN` server |
+| `audit://domains` | Available MCP domain bundles and tool groupings |
 
-## Tools (221 read-only + export)
+## Tools (340 read-only + export)
+
+### Router and insight (Tier 0 — `WP_MCP_DOMAIN=core`)
+
+`search_audit_tools`, `list_tool_domains`, `get_data_coverage_report`, `run_insight_workflow`, `run_technical_workflow`, `run_keyword_workflow`, `run_domain_agent`, `get_landing_page_blended_table`, `get_opportunity_matrix`, `get_traffic_health_check`, `get_landing_page_full_diagnosis`, `get_issue_to_traffic_map`
 
 ### Export and deliverables
 
@@ -99,11 +144,11 @@ Size-based tools require `probe_image_inventory=true` in pipeline config when bu
 
 ### Keywords
 
-`get_keyword_summary`, `search_keywords`, `get_striking_distance_keywords`, `get_keyword_cannibalisation`, `get_query_page_misalignment`, `get_semantic_keyword_clusters`, `get_keyword_history`, `get_keyword_serp_overlay`, `get_serp_feature_overlay`, `list_keywords_by_action`, `list_keywords_by_position`, `list_keywords_by_impressions`, `list_keywords_ctr_opportunity`, `expand_keywords`, `generate_content_brief`
+`get_keyword_summary`, `search_keywords`, `get_striking_distance_keywords`, `get_keyword_cannibalisation`, `get_query_page_misalignment`, `get_semantic_keyword_clusters`, `get_keyword_history`, `get_keyword_serp_overlay`, `get_serp_feature_overlay`, `list_keywords_by_action`, `list_keywords_by_position`, `list_keywords_by_impressions`, `list_keywords_ctr_opportunity`, `expand_keywords`, `generate_content_brief`, `get_brand_keyword_split`, `list_keywords_by_intent`
 
 ### Google and CTR
 
-`get_google_summary`, `get_google_integration_status`, `get_gsc_top_queries`, `get_gsc_top_pages`, `get_gsc_ctr_opportunity_pages`, `get_ga4_summary`, `get_ga4_page_metrics`, `get_gsc_page_query_slice`, `get_gsc_url_inspection`, `get_gsc_index_coverage`, `analyze_serp_snippet_for_url`
+`get_google_summary`, `get_google_integration_status`, `get_gsc_top_queries`, `get_gsc_top_pages`, `get_gsc_ctr_opportunity_pages`, `get_ga4_summary`, `get_ga4_page_metrics`, `get_gsc_page_query_slice`, `get_gsc_url_inspection`, `get_gsc_index_coverage`, `analyze_serp_snippet_for_url`, `get_gsc_daily_trend`, `get_ga4_daily_trend`, `get_ga4_by_device`, `get_ga4_by_channel`, `get_gsc_page_queries`
 
 ### Backlinks
 
@@ -163,6 +208,8 @@ Already available: `validate_rich_results`, `get_gsc_url_inspection`, `export_si
 
 The same tools power **AI Chat** at [http://localhost:3000/chat](http://localhost:3000/chat). Enable AI in Run audit → AI settings.
 
+In-app chat uses **dynamic tool routing**: each turn loads Tier 0 router tools plus a domain-scoped subset (~45 tools), not the full catalog. Set `CHAT_TOOL_MODE=full` to load all tools for debugging.
+
 ## Ollama note
 
 When the local Ollama daemon supports native tools (most current models, including Ollama cloud refs like `minimax-m3:cloud`), chat uses Ollama’s `/api/chat` tool format. Older or tool-less models fall back to JSON ReAct parsing. OpenAI and Anthropic always use native tool calling with streaming in the chat UI.

input.txt.example

@@ -15,6 +15,11 @@ allow_external = false
 store_outlinks = true
 store_content_excerpt = true
 content_excerpt_max_chars = 4096
+store_page_html = false
+max_stored_html_bytes = 2097152
+run_content_analysis = false
+content_analysis_strategy = main_only
+content_analysis_workers = 4
 preserve_crawl_history = true
 crawl_stream_to_db = false
 crawl_exclude_urls = 

pipeline-config.example.txt

@@ -16,6 +16,11 @@ allow_external = false
 store_outlinks = true
 store_content_excerpt = true
 content_excerpt_max_chars = 4096
+store_page_html = false
+max_stored_html_bytes = 2097152
+run_content_analysis = false
+content_analysis_strategy = main_only
+content_analysis_workers = 4
 preserve_crawl_history = true
 crawl_stream_to_db = false
 crawl_exclude_urls = 

requirements.txt

@@ -1,50 +1,50 @@
-requests>=2.28.0
-beautifulsoup4>=4.11.0
-lxml>=4.9.0
-pandas>=1.5.0
-tqdm>=4.64.0
-networkx>=2.8.0
-python-Wappalyzer>=0.3.1
+requests==2.34.2
+beautifulsoup4==4.14.3
+lxml==6.1.1
+pandas==3.0.3
+tqdm==4.67.3
+networkx==3.6.1
+python-Wappalyzer==0.3.1
 
 # Local content analysis (duplicates, language)
-rapidfuzz>=3.0.0
-langdetect>=1.0.9
+rapidfuzz==3.14.5
+langdetect==1.0.9
 
 # Google Search Console + GA4 integration (optional; required for `python -m src google`)
-google-auth>=2.0.0
-google-auth-oauthlib>=1.0.0
-google-api-python-client>=2.0.0
-google-analytics-data>=0.18.0
-google-analytics-admin>=0.22.0
+google-auth==2.53.0
+google-auth-oauthlib==1.4.0
+google-api-python-client==2.197.0
+google-analytics-data==0.23.0
+google-analytics-admin==0.30.0
 
 # Keywords Explorer — Google Suggest + Wikipedia + Datamuse (all free, no auth needed)
 # requests already listed above
 # pytrends is OPTIONAL and frequently rate-limited — uncomment only if you need trend direction
 # pytrends>=4.9,<5
 
 # PostgreSQL
-psycopg[binary,pool]>=3.2
-sqlalchemy>=2.0.0
-alembic>=1.13
+psycopg[binary,pool]==3.3.4
+sqlalchemy==2.0.50
+alembic==1.18.4
 
 # Audit export (PDF)
-reportlab>=4.0.0
+reportlab==4.5.1
 
 # JavaScript rendering crawl (headless Chromium via Playwright)
-playwright>=1.49.0
+playwright==1.60.0
 
 # LLM providers for AI enrichment (configure via web UI AI tab)
-httpx>=0.27.0
-openai>=1.0.0
-anthropic>=0.25.0
+httpx==0.28.1
+openai==2.41.0
+anthropic==0.107.0
 
 # Spell-check / HTML validation extras
-pyspellchecker>=0.8.1
-html5lib>=1.1
+pyspellchecker==0.9.0
+html5lib==1.1
 
 # MCP server for Cursor / Claude Desktop
-mcp>=1.0.0
+mcp~=1.0.0
 
 # Dev / test
-pytest>=7.0.0
-pytest-cov>=5.0.0
+pytest==9.0.3
+pytest-cov==7.1.0

scripts/local-test.ps1

@@ -237,6 +237,10 @@ function Invoke-PytestReporting {
         tests/test_optional_audits.py `
         tests/test_property_profile.py `
         tests/test_reporting_gaps.py `
+        tests/test_text_content_analysis.py `
+        tests/test_builder_image_buckets.py `
+        tests/test_pipeline_report_pool_unit.py `
+        tests/test_reporting_builder_modules.py `
         --cov=website_profiling.reporting `
         --cov-config=.coveragerc.reporting `
         --cov-report=term-missing `
@@ -272,6 +276,7 @@ function Invoke-PytestTools {
         tests/test_export_sitemap.py `
         tests/test_mcp_registry.py `
         tests/test_mcp_resources.py `
+        tests/test_tools_gate100_coverage.py `
         tests/test_tools_branch_coverage.py `
         --cov=website_profiling.tools `
         --cov-config=.coveragerc.tools `

scripts/local-test.sh

@@ -126,6 +126,10 @@ run_pytest_reporting() {
     tests/test_optional_audits.py \
     tests/test_property_profile.py \
     tests/test_reporting_gaps.py \
+    tests/test_text_content_analysis.py \
+    tests/test_builder_image_buckets.py \
+    tests/test_pipeline_report_pool_unit.py \
+    tests/test_reporting_builder_modules.py \
     --cov=website_profiling.reporting \
     --cov-config=.coveragerc.reporting \
     --cov-report=term-missing \
@@ -149,6 +153,7 @@ run_pytest_tools() {
     tests/test_audit_tools_links_extras.py \
     tests/test_audit_tools_expansion.py \
     tests/test_audit_tools_expansion_coverage.py \
+    tests/test_audit_tools_batch100_coverage.py \
     tests/test_export_custom_coverage.py \
     tests/test_export_artifacts_coverage.py \
     tests/test_export_compare_coverage.py \
@@ -161,6 +166,9 @@ run_pytest_tools() {
     tests/test_export_sitemap.py \
     tests/test_mcp_registry.py \
     tests/test_mcp_resources.py \
+    tests/test_router_tools.py \
+    tests/test_tool_selector.py \
+    tests/test_tools_gate100_coverage.py \
     tests/test_tools_branch_coverage.py \
     --cov=website_profiling.tools \
     --cov-config=.coveragerc.tools \