Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,10 @@ jobs:
- name: Pytest (reporting coverage gate)
run: |
pytest tests/test_categories_roadmap.py tests/test_report_categories_golden.py \
tests/test_categories_coverage.py tests/test_indexation_coverage.py tests/test_crawl_segments.py \
tests/test_categories_coverage.py tests/test_contrast_issues.py \
tests/test_indexation_coverage.py tests/test_crawl_segments.py \
tests/test_terminology.py tests/test_compare_payload.py \
tests/test_optional_audits.py tests/test_property_profile.py tests/test_reporting_gaps.py \
--cov=website_profiling.reporting --cov-config=.coveragerc.reporting \
--cov-report=term-missing --cov-fail-under=100 -q -o addopts=
- name: Pytest (tools coverage gate)
Expand Down
129 changes: 127 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,132 @@
<p align="center">
<a href="https://github.com/codefrydev/WebsiteProfiling">
<img src="docs/assets/readme-banner.png" alt="Site Audit — Open Source SEO Crawl &amp; Audit" width="920">
</a>
</p>

<p align="center">
<strong>Site Audit — Open Source SEO Crawl &amp; Audit</strong><br>
<sub>Free, self-hosted — no vendor paywalls.</sub>
</p>

<p align="center">
<a href="https://github.com/codefrydev/WebsiteProfiling/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/codefrydev/WebsiteProfiling/ci.yml?branch=master&label=CI&logo=github" alt="CI status"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License"></a>
<a href="https://github.com/codefrydev/WebsiteProfiling"><img src="https://img.shields.io/badge/open%20source-yes-brightgreen.svg" alt="Open source"></a>
<a href="https://github.com/codefrydev/WebsiteProfiling/stargazers"><img src="https://img.shields.io/github/stars/codefrydev/WebsiteProfiling?style=social" alt="GitHub stars"></a>
</p>

<p align="center">
<img src="https://img.shields.io/badge/Next.js-000?logo=next.js&logoColor=white" alt="Next.js">
<img src="https://img.shields.io/badge/Python-3776AB?logo=python&logoColor=white" alt="Python">
<img src="https://img.shields.io/badge/PostgreSQL-4169E1?logo=postgresql&logoColor=white" alt="PostgreSQL">
<img src="https://img.shields.io/badge/Docker-2496ED?logo=docker&logoColor=white" alt="Docker">
</p>

<p align="center">
<a href="#quick-start">Quick start</a> ·
<a href="#features">Features</a> ·
<a href="#project-structure">Structure</a> ·
<a href="#contributing">Contributing</a> ·
<a href="#docs">Docs</a> ·
<a href="#license">License</a>
</p>

---

# Site Audit

Open-source technical SEO crawl and audit UI (Next.js + Python + PostgreSQL).
**Open Source SEO Crawl & Audit** — self-hosted UI built with **Next.js + Python + PostgreSQL**.

Repository: [codefrydev/WebsiteProfiling](https://github.com/codefrydev/WebsiteProfiling)

## Overview

**Why this project** — Most site-audit and SEO tools are paid, limited, or built to upsell: paywalls, capped crawls, teaser scores, and “subscribe to see how to fix this.” Many free options give shallow or unreliable reports that push you toward a paid plan instead of real answers.

**Goal** — A free, self-hosted audit you control: crawl your sites, see honest technical SEO issues, connect Search Console and Analytics when you want, and export reports for clients — without a vendor sitting between you and the data.

## Features

<table>
<tr>
<td align="center" width="25%">
<img src="docs/assets/icon-crawl.svg" width="48" alt=""><br>
<strong>Site crawl</strong><br>
<sub>Static &amp; JS rendering, sitemap export, crawl maps</sub>
</td>
<td align="center" width="25%">
<img src="docs/assets/icon-audit.svg" width="48" alt=""><br>
<strong>Technical audit</strong><br>
<sub>Issues, Lighthouse, on-page checks, workbooks</sub>
</td>
<td align="center" width="25%">
<img src="docs/assets/icon-integrations.svg" width="48" alt=""><br>
<strong>Integrations</strong><br>
<sub>Google Search Console, GA4, Bing Webmaster</sub>
</td>
<td align="center" width="25%">
<img src="docs/assets/icon-self-hosted.svg" width="48" alt=""><br>
<strong>Self-hosted</strong><br>
<sub>Docker or local dev — your data stays yours</sub>
</td>
</tr>
</table>

Also included: **AI chat** over audit data (optional), **121 MCP tools**, keyword explorer, backlinks, compare runs, and portfolio management for agencies.

<p align="center">
<img src="docs/assets/social-preview.png" alt="Site Audit preview" width="640">
</p>

## Project structure

```
WebsiteProfiling/
├── src/website_profiling/ # Python audit engine (CLI: python -m src)
│ ├── crawl/ # Crawler, fetchers, JS rendering
│ ├── reporting/ # Report builder, issue categories
│ ├── analysis/ # On-page / local analysis
│ ├── lighthouse/ # Lighthouse runner
│ ├── integrations/ # Google Search Console, GA4, Bing, CrUX
│ ├── llm/ # AI enrich + chat agent
│ ├── tools/ # Exports, audit query tools, MCP helpers
│ ├── mcp/ # MCP server (121 read-only tools)
│ ├── db/ # PostgreSQL storage layer
│ ├── commands/ # CLI subcommands
│ ├── cli.py # Pipeline entrypoint
│ └── config.py # Config load (DB + shadow file)
├── web/ # Next.js UI
│ ├── app/ # App Router pages + /api routes
│ ├── src/components/ # React UI components
│ ├── src/views/ # Report views (overview, links, issues, …)
│ ├── src/server/ # Server-side DB, pipeline jobs, config I/O
│ └── public/ # Static assets (logo, favicon)
├── alembic/versions/ # PostgreSQL schema migrations
├── tests/ # pytest suite + fixtures
├── docs/ # Glossary, MCP, ops, brand assets
├── scripts/ # local-run.sh, local-test.sh helpers
├── .github/workflows/ # CI (Python + web + browser crawl)
├── docker-compose.yml # Dev stack (Postgres + web)
├── Dockerfile # Production image
├── local-run # Dev setup & start script
├── local-test # Full test suite (CI parity)
├── requirements*.txt # Python deps (core, browser, LLM, MCP)
└── pipeline-config.example.txt
```

| Path | Purpose |
|------|---------|
| `src/website_profiling/` | Crawl, analyze, report, Lighthouse, integrations, AI — run via `python -m src` |
| `web/app/api/` | REST APIs: report data, pipeline runs, chat (SSE), Google/Bing sync |
| `web/src/lib/pipelineConfigSchema.ts` | Audit settings schema (UI ↔ PostgreSQL) |
| `alembic/versions/` | Database migrations — run `./local-run migrate` |
| `tests/` | Backend tests; `./local-test browser` for Playwright crawl integration |
| `docs/MCP.md` | MCP server setup for IDE / agent integrations |
| `data/` | Local secrets + shadow `pipeline-config.txt` (gitignored) |

For deeper layout notes and edit targets, see [AGENT.md](AGENT.md).

## Quick start

**Docker (build from source)**
Expand Down Expand Up @@ -67,6 +186,12 @@ The agent uses the same **121 read-only audit tools** as the MCP server (`docs/M

Production: `docker-compose.prod.yml` (set `POSTGRES_PASSWORD`, `AUTH_SECRET`).



## Star History

[![Star History Chart](https://api.star-history.com/svg?repos=codefrydev/WebsiteProfiling&type=Date)](https://star-history.com/#codefrydev/WebsiteProfiling&Date)

## License

Copyright (c) 2026 [codefrydev](https://github.com/codefrydev). Released under the **MIT License** — see [LICENSE](LICENSE). Issues and pull requests: [codefrydev/WebsiteProfiling](https://github.com/codefrydev/WebsiteProfiling).
Copyright (c) 2026 [codefrydev](https://github.com/codefrydev). Released under the **MIT License** — see [LICENSE](LICENSE). Issues and pull requests: [codefrydev/WebsiteProfiling](https://github.com/codefrydev/WebsiteProfiling).
55 changes: 55 additions & 0 deletions docs/assets/banner.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 6 additions & 0 deletions docs/assets/icon-audit.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
7 changes: 7 additions & 0 deletions docs/assets/icon-crawl.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 6 additions & 0 deletions docs/assets/icon-integrations.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 6 additions & 0 deletions docs/assets/icon-self-hosted.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 8 additions & 0 deletions docs/assets/logo-icon.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading