A multi-user, all-in-one proxy manager

---

Magpie is a self-hosted proxy manager that scrapes public proxy sources, continuously checks their health, filters bad entries, calculates reputation scores, and creates rotating proxy endpoints from the healthy pool.

This is the Magpie distribution repository. It connects the independently versioned frontend and backend images with PostgreSQL and Redis, and owns the install, update, and performance-validation tooling. Application source code lives in the component repositories below.

Repository map

Repository	Responsibility
Magpie-Tools/magpie	Distribution: Docker Compose, installers, update helpers, release configuration, and performance gates
Magpie-Tools/magpie-frontend	Angular application and frontend container image
Magpie-Tools/magpie-backend	Go API, background jobs, proxy workers, and backend container image
Magpie-Tools/magpie-website	magpie.tools marketing website
Magpie-Tools/magpie-docs	Docusaurus documentation published at /docs

The repositories are siblings, not Git submodules. A production installation only needs this distribution repository or the one-command installer. It pulls the published component images.

Backend source builds require Go 1.27.2. Backend CI reads the version from magpie-backend/go.mod, and its Docker builder uses the matching golang:1.27.2-alpine image. Toolchain and dependency upgrades must also pass the backend validation and checker benchmarks described in scripts/perf.

Features

• Multi-workspace dashboard and API with owner, admin, operator, and viewer roles
• Account-bound workspace invitations with an in-app inbox and optional email notifications
• Workspace and rotating-pool alerts with incident history and email, Slack, Discord, and webhook delivery
• Workspace-owned capacity, operational settings, managed proxies, tags, sources, judges, and rotators
• Automatic proxy scraping and health checks
• Default and ordered tag checker rules with Replace, Add, Remove, and shared transport, timeout, retries, and inheritance choices per tag
• Provider hostname, IPv4, and IPv6 proxy import, checking, search, export, and rotation. IP blacklists apply to literal addresses, and automatic scraping remains IPv4-only.
• Workspace-owned, color-coded proxy tags with multi-tag assignment, import tagging, automatic source tagging, search, and filtering
• Active, paused, and archived managed-proxy lifecycle with multi-select list, export, and delete filters; capacity overflow is retained rather than deleted
• Workspace-wide Pause or Delete action after consecutive proxy check failures. Pause remains the default; deletion affects future failed checks, preserves other workspaces, and allows later rediscovery.
• Reputation scoring and filters
• User-defined rotating proxy endpoints
• HTTP, HTTPS, SOCKS4, and SOCKS5 application protocols
• TCP and QUIC/HTTP3 transport support

More screenshots

Quick start

Prerequisite: Docker Desktop or Docker Engine with Docker Compose.

One-command install

macOS/Linux:

curl -fsSL https://raw.githubusercontent.com/Magpie-Tools/magpie/refs/heads/master/scripts/install.sh | bash

Windows PowerShell:

iwr -useb https://raw.githubusercontent.com/Magpie-Tools/magpie/refs/heads/master/scripts/install.ps1 | iex

The installer creates a magpie/ deployment directory, generates .env, pulls the published images, and starts the complete stack.

If Docker reports a socket permission error on Linux, add your user to the Docker group and start a new login session:

sudo usermod -aG docker "$USER"

Clone and run

git clone https://github.com/Magpie-Tools/magpie.git
cd magpie
cp .env.example .env
# Edit .env and replace PROXY_ENCRYPTION_KEY and the example credentials.
docker compose run --rm backend --migrate-only
docker compose up -d

Open:

• UI: http://localhost:5050
• API: http://localhost:5656/api
• Documentation: https://magpie.tools/docs/

The first registered user becomes the administrator in the default local configuration. Optional geolocation and reputation integrations are available under Admin → Plugins.

Warning

Keep PROXY_ENCRYPTION_KEY stable across restarts and updates. It encrypts proxy usernames and passwords in PostgreSQL and keys route fingerprints. Starting the backend with a different key prevents existing secrets from being decrypted. For checker throughput, Redis queue payloads contain proxy addresses and credentials in plaintext by default. Keep Redis private and protect its access, volumes, and backups. Set PROXY_QUEUE_ENCRYPT_CREDENTIALS=true only if you accept its per-check cost.

For production updates, take coordinated PostgreSQL and Redis backups, stop the backend, and run the new image's database migration before starting it again:

docker compose stop backend
docker compose run --rm backend --migrate-only
docker compose up -d

Do not skip the migration when updating an existing installation. Workspace-capable releases create one personal workspace and owner membership for every existing account, move operational ownership from user_id to workspace_id, and repair PostgreSQL foreign keys. Existing resources remain available through the new default workspace. The migration is not compatible with older backend images; rollback requires the coordinated PostgreSQL and Redis backups.

Longevity maintenance adds replay-floor storage, a sortable statistics-ledger key, and lifetime check totals. Run --migrate-only with old backend instances stopped; moving large DEFAULT-partition history can extend the migration window. Keep PostgreSQL and Redis backups coordinated. Daily compaction preserves all-time counts, and partition precreation continues with DB_AUTO_MIGRATE=false.

Each backend checks the queue marker every ten seconds and reconstructs lost queues from PostgreSQL. /readyz returns 503 after detection until reconstruction completes. Orphan cleanup drains repeated batches with a 45-second budget and a one-minute retry after interruption. .env.example lists retention, cleanup, tunnel-limit, and certificate controls. Monitor maintenance success/failure metrics and the statistics replay floor; an old pending event delays retirement.

API clients may select a workspace with X-Workspace-ID. When omitted, the backend uses the authenticated account's default workspace membership.

The notification bell shows the five latest invitations, workspace alerts, and releases. Its badge counts unread items in that list. Opening an item saves a per-account read receipt in PostgreSQL, shared across browsers. The normal --migrate-only update creates notification_reads; deploy the updated backend and frontend together. No new environment setting or volume is required.

Workspace access is offered through expiring invitations for existing Magpie accounts. Accepting an invitation does not change the account's default workspace. SMTP is optional; pending invitations remain available in the app when email delivery is disabled.

The legacy proxy_limits.max_per_user setting seeds active-route capacity for new workspaces when limits are enabled. Scrape sources and manual imports retain overflow as paused proxies, so stored totals can exceed the active limit. Editing this global default or its administrator exclusion does not change existing workspace allowances. The workspace page shows the current active limit and stored count.

The installers generate a unique JWT signing secret and save it in .env. Manual Compose starts with an empty JWT_SECRET generate and persist a key in backend_data at /app/data/jwt_secret. Custom secrets remain unchanged. The backend replaces the old published Compose JWT default with a generated key, so existing sessions using that default must sign in again. Keep PROXY_ENCRYPTION_KEY unchanged across upgrades.

Compose binds the API to 127.0.0.1 by default. Set BACKEND_BIND_ADDRESS to change it. The frontend remains available on port 5050. The volume initializer repairs settings/cache ownership for UID 65532 before the backend starts.

For production, use the production overlay for migration and startup:

docker compose -f docker-compose.yml -f docker-compose.production.yml run --rm backend --migrate-only
docker compose -f docker-compose.yml -f docker-compose.production.yml up -d

This enables production mode, strict secret validation and protected observability endpoints. Public registration and first-admin bootstrap are disabled. For a fresh installation, enable ENABLE_PUBLIC_FIRST_ADMIN_BOOTSTRAP only while you create the first account through a restricted frontend, then set it back to false and recreate the backend. Configure TLS termination, Redis authentication, database access and backups for your deployment.

The container waits up to 90 seconds on stop. The API drains for up to 20 seconds, then background jobs stop and usage/statistics consumers finish their bounded flushes. After statistics and reputation writers stop, read models refresh proxy filters, then drain all queued scrape-source statistics batches before Redis closes. Increase BACKEND_STOP_GRACE_PERIOD if you raise either shutdown timeout.

Before releasing a backend image, run the fresh/repaired-volume production startup check:

scripts/validate-container-startup.sh kuuchen/magpie-backend:YOUR_TAG

It uses disposable services and volumes and also verifies that the generated JWT key survives restart. Run the release performance gate on representative hardware before making capacity claims.

Alerts setup

Run the normal --migrate-only upgrade step with all backend instances stopped before starting an alerts-capable backend. It creates workspace-owned alert rules, destinations, incidents, and delivery records. Update the frontend to expose the new /alerts page. Existing workspaces start with no alert rules.

Workspace admins and owners configure shared destinations; operators configure rules, and viewers can read alert history. Email uses the existing MAIL_FROM_* and SMTP_* configuration. Slack and Discord use incoming webhook URLs, and generic webhooks support optional HMAC-SHA256 signatures. Destination targets and signing secrets use the stable PROXY_ENCRYPTION_KEY and are write-only through the API. PostgreSQL backups contain incident history and encrypted destinations, so retain the key with your normal secret backup process.

Discord and Slack destinations can optionally mention a selected audience, including a Discord role ID or a Slack user group ID. Mentions default to off for new and existing destinations. Run migrations before starting a backend with mention settings. Chat alerts use native timestamps with an ISO timestamp below; email includes readable UTC time and the ISO timestamp.

The evaluator also requires the migration that adds internal attempt timestamps to alert rules. Those timestamps preserve scheduling priority across timeouts and leadership changes. Existing measurements and incidents remain intact. Evaluation uses four PostgreSQL workers with separate query budgets. Each worker evaluates a workspace's rules together, batching state, incident, and outbox writes. Whole-workspace route counts share grouped reads of current evidence; pool rules reuse counts for identical eligibility filters. Each backend instance drains notifications through ten workers, polling every five seconds when idle. These limits require no new environment variables or additional migrations beyond the attempt-timestamp migration above.

The cadence regression covers 2,000 workspaces with 100 rules each on two-CPU PostgreSQL. All 200,000 rules completed each of six consecutive passes in 19–30 seconds with one destination per rule, including opening and recovering 200,000 incidents and queuing 400,000 notifications. See the backend's docs/adr/0012-evaluate-alerts-from-existing-workspace-measurements.md for the workload, exact measurements, and reproducible test command. Database history size, pool filters, notification fanout, and other database traffic affect capacity; measure those workloads on the deployment's hardware.

Rules monitor usable-route minima, checker success-rate minima, or average successful-check latency maxima. Counts follow the current rotating policy; rotator success/latency measurements cover workspace checks for its upstream protocol over TCP. Two-minute breach and recovery periods limit noise, with 15-minute windows and minimum samples for checker metrics. Recent check history survives managed-proxy deletion for the observation window. Evaluation and delivery run outside the checker loop and add no operations per proxy check.

Webhook requests follow the existing outbound network restrictions and reject redirects. ALLOW_PRIVATE_NETWORK_EGRESS=false is the Compose default. Set it explicitly to true only when internal webhook targets or HTTP generic webhooks are required; it also affects other application outbound requests.

See the alerts guide for incident, retry, and retention behavior.

Rotator creation and checker settings

Rotators can be created with any supported upstream protocol even when all Default checker protocols are disabled or no healthy proxies are available. The frontend shows a warning when Default has no enabled protocols. To serve traffic, the rotator still needs active upstream proxies with current successful TCP checks for its selected protocol. Enable checks in Default settings or matching tag rules.

Deploy the updated frontend and backend together. Older backends reject rotator creation for protocols disabled in Default settings. This change needs no schema migration or new environment setting.

Tag checker settings upgrade

Update the frontend and all backend replicas together. Run the migration with every backend stopped. This release adds protocol defaults, ordered workspace tag rules, a sparse projection for tagged checks, and workspace/configuration keys on latest statistics. The latest-statistics primary key changes, so older backend images must not write to the migrated database.

Existing checker values become shared Default settings with no tag rules. Legacy checks remain in history. Current health and TCP rotator eligibility wait for new matching checks because old results lack reliable transport and settings attribution. Failure streaks remain saved. No new environment variables are required, and queue encryption stays opt-in. Restore coordinated PostgreSQL and Redis backups to roll back.

In Checker Settings, select Default or a workspace tag, choose Replace, Add, or Remove, and arrange matching tags in the Tag priority popup with drag handles or up/down buttons. The top rule has highest priority and applies last. Transport, timeout, and retries inherit independently and apply to every enabled protocol. Judges, HTTPS-for-SOCKS, and automatic failure actions remain workspace-wide. Changes apply at the next scheduled check; empty selections skip checking without pausing or deleting the proxy.

REST exposes checker_settings and GraphQL exposes checkerSettings. Clients that omit these fields preserve stored rules and shared profile values. Column and judge saves send only their edited fields. Checker refreshes reconcile the persisted projection across replicas and rewrite only affected routes and source summaries. Unchanged reconciliation skips route reads, and column-only saves skip checker refresh. Dashboard caches verify the committed checker generation before serving health; recent checks use indexed candidates before fetching evidence. Run the updated migration even when upgrading an earlier tag-settings build: it adds workspace revision counters, a coalesced route-change journal, and the active recent-checks index. No new environment settings are required. The checker guide and performance harness explain behavior and validation.

Queue interval synchronization now starts after instance settings load. Older builds could publish a one-second placeholder when a test, migration, or replica started against the shared Redis, causing rapid proxy rechecks. Update and restart every backend replica to restore the configured interval. Existing due entries converge through normal requeue. This fix needs no database migration or new environment setting; tag rules continue to use the global checker interval.

API client upgrade note

Automatic source tags require the updated frontend and backend images. Run the backend's --migrate-only command before starting the updated workers to create the workspace source-tag rule table. Existing sources start with no automatic tags, and configuring them affects future scrape results, including rediscovered proxies. Rules add missing tags without replacing existing assignments. Update all backend workers together because older workers do not apply these rules. Source rules add no work to the ordinary proxy checker loop.

GraphQL clients must omit scrapingSources from UpdateUserSettingsInput. That input previously reported success without saving sources and now returns a validation error. Sources remain readable through GraphQL; use the REST scrape-source endpoints to manage them. Settings mutations now reject negative or oversized checker integers and refresh the checker judge cache after saving. No additional environment variables or database migration are needed for these API corrections.

Component image versions

Frontend and backend releases can be selected independently in .env:

MAGPIE_BACKEND_IMAGE=kuuchen/magpie-backend
MAGPIE_BACKEND_TAG=latest
MAGPIE_FRONTEND_IMAGE=kuuchen/magpie-frontend
MAGPIE_FRONTEND_TAG=latest

Pin immutable release tags for reproducible deployments. The legacy MAGPIE_IMAGE_TAG variable remains supported as a shared fallback; an explicit component tag takes precedence.

Updating

For an installer-created deployment, refresh the Compose definition, pull images, and restart the stack with:

macOS/Linux:

curl -fsSL https://raw.githubusercontent.com/Magpie-Tools/magpie/refs/heads/master/scripts/update.sh | bash

Windows PowerShell:

iwr -useb https://raw.githubusercontent.com/Magpie-Tools/magpie/refs/heads/master/scripts/update.ps1 | iex

For a cloned distribution repository:

./scripts/update-stack.sh

Or from Windows Command Prompt:

scripts\update-stack.bat

These helpers pull distribution changes and published images; they no longer build frontend or backend source from this repository.

All update helpers discover this installation's backend through Docker labels and preserve its project name, working directory and ordered Compose file list. This includes names selected by an overlay's name:, -p or COMPOSE_PROJECT_NAME, and stopped backends after a failed migration. Production mode and the existing project's volumes and networks remain selected even when downloaded files change the name. Unrelated projects and one-off migration containers are ignored. Missing files, unreadable metadata or multiple matching configurations abort the update before the backend is stopped.

If you remove the containers with docker compose down, keep the selection in .env so Compose and the helpers can recover it without container metadata:

COMPOSE_FILE=docker-compose.yml:docker-compose.production.yml
# If you use a custom project name:
# COMPOSE_PROJECT_NAME=magpie-production

Use ; instead of : on Windows. Include any custom overlays in the same order and persist a custom project name with COMPOSE_PROJECT_NAME. With no existing backend, helpers use Compose's settings from the shell or .env, or its normal default files. Setting MAGPIE_ENV=production in .env also enables backend production defaults.

Proxy export timeouts

Proxy exports allow up to 300 seconds of database and formatting work by default. Other API responses retain their normal write timeout. Export batches flush as ready; disconnecting cancels the export queries. A failed transfer must be retried and is not a complete export file.

Compose passes PROXY_EXPORT_TIMEOUT_SECONDS=300 to the backend and EXPORT_PROXY_READ_TIMEOUT_SECONDS=310 to the frontend nginx container. Set both in .env to tune exports, keeping nginx's value at least 10 seconds higher. The backend allows an extra 5 seconds to write a timeout error before any data has been sent. Any additional reverse proxy must allow the same wait for export responses. Apply changed values by recreating the frontend and backend services. Both updated images are needed for this behavior; updating .env alone does not add it to older releases.

An export that repeatedly takes over 30 seconds warrants checking database query times and resource contention with the checker, even when a higher limit lets it finish.

Local development

Clone all five repositories as siblings:

workspace/
├── magpie/
├── magpie-backend/
├── magpie-frontend/
├── magpie-website/
└── magpie-docs/

Use this repository for shared infrastructure:

cd magpie
cp .env.example .env
# Edit .env first.
docker compose up -d postgres redis

Then run the component you are developing from its own repository:

• Backend: cd ../magpie-backend && go run ./cmd/magpie
• Frontend: cd ../magpie-frontend && npm ci && npm run start
• Website: cd ../magpie-website && npm ci && npm run dev
• Docs: cd ../magpie-docs && npm ci && npm run start

The backend must be configured to use PostgreSQL at localhost:5434 and Redis at localhost:8946 when those services are started from this Compose file. Each component repository contains its own build, test, and development details.

To populate three development accounts, stop the backend and run:

./scripts/seed-dev-data.sh --dry-run
./scripts/seed-dev-data.sh --reset
cd ../magpie-backend
DEVELOPMENT_FIXTURES=true go run ./cmd/magpie

The reset preserves existing logins and account roles, replaces their workspaces with five fixture workspaces, and adds 5,000 routes plus 30 days of check history. It also populates sources, tags, rotating proxies, alerts, invitations, and usage. Run it again to restore the fixtures with current dates. A workspace shared with an unselected account blocks the reset.

The script uses the sibling backend repository and its .env; override the path with MAGPIE_BACKEND_REPO. Use --accounts to select three accounts, --proxies-per-workspace to change the route count, and --history-days or --seed to change the generated history. New accounts require MAGPIE_DEV_PASSWORD; existing passwords remain unchanged. Run the normal backend migration command first on a fresh database.

DEVELOPMENT_FIXTURES=true disables background jobs and rotating listeners so simulated data stays intact. Rotators created through the API are saved without opening TCP or UDP listeners. Restart with fixture mode disabled to activate them. Set it in the backend's local .env, or in this repository's .env when using a backend container built from the updated source. Fixture mode and seeding are rejected in production. The local development guide describes the fixture accounts and reset options.

Run backend tests and benchmarks against disposable PostgreSQL and Redis instances. Keep their Redis databases separate from the running development stack and from each other; queue concurrency fixtures flush their test database.

The performance release gate remains in scripts/perf.

Publishing the website and documentation

The public site is assembled on this repository's gh-pages branch. With all five repositories cloned as siblings, publish the website and documentation in sequence:

cd ../magpie-website
npm run deploy

cd ../magpie-docs
npm run deploy

Both component scripts build their source, then call scripts/publish-pages-artifact.sh in this repository. The website replaces the branch root while preserving /docs; the documentation deployment replaces only /docs.

Set MAGPIE_DISTRIBUTION_REPO if this repository is not at ../magpie. Set MAGPIE_DEPLOY_DRY_RUN=1 to build and validate without changing gh-pages, or MAGPIE_DEPLOY_PUSH=0 to create the local gh-pages commit without pushing it. Run the two deployments sequentially so each starts from the latest branch.

Attributions and community

• The AbuseIPDB logo is used with permission when linking to its site
• Website: https://magpie.tools
• Docs: https://magpie.tools/docs/
• Discord: https://discord.gg/7FWAGXzhkC

License

Magpie is distributed under the GNU Affero General Public License v3.0. See LICENSE for the complete license.

Scraper fetching modes

New sources use HTTP fetching. Enable Requires JavaScript when adding a source or in its detail page to render it in Chromium. This setting belongs to the active workspace. Existing sources retain browser rendering after migration.

Run the normal --migrate-only upgrade step before starting this backend version. The migration adds fetching mode and last-scrape status fields to workspace source associations. Upgrade all backend replicas together so older workers do not keep scraping HTTP sources through Chromium.

Chromium starts only for browser sources. SCRAPER_PAGE_POOL_MAX_CAPACITY limits concurrent browser scrapes per instance, defaults to 4, and accepts 1–64. The former minimum page-pool setting no longer preallocates idle pages. SCRAPER_POST_PROCESS_QUEUE_CAPACITY defaults to 16 to limit queued HTML memory. Network failures for optional resources such as images leave valid proxy HTML and concurrent browser scrapes usable. Main-document failures and outbound policy violations still fail the scrape.

New sources are eligible immediately and workers drain them at configured concurrency. Browser capacity failures, outages, timeouts, HTTP 429 and HTTP 5xx retry after 30 seconds or the scrape interval, whichever is shorter. Other failures use the normal interval. The source list and detail page show the last scrape outcome separately from proxy health.

If a scrape logs extended protocol limited to 65535 parameters, upgrade to a backend build with the proxy-ingestion batching fix and retry the source. Older builds could exceed PostgreSQL's per-statement limit when saving workspace associations or looking up large proxy lists. Inserts now use a separate batch size for each table, and large hash lookups are split. This fix requires no additional schema migration or environment setting.

Account profile compatibility

The sidebar displays the signed-in account email using GET /api/user/profile. Deploy matching frontend and backend releases together so session restoration can access this endpoint.

Table action columns

The proxy and scrape source lists default to an ellipsis Actions menu. Select Actions (buttons) in the column picker for the previous inline buttons. Update the frontend and backend together to save this preference; older backends discard the actions_buttons column identifier. The scrape source list hides Robots Check by default; existing saved column selections are preserved.

Scrape source table sorting

Deploy matching frontend and backend releases for scrape source sorting. The frontend sends sortField and sortOrder to GET /api/getScrapingSourcesPage/{page}; the backend sorts all matching sources before pagination. Older backends ignore these parameters. No schema migration or new environment setting is required.

When validating a release, use more than one page of sources and check URL, proxy count, alive count, and health in both directions. Changing the sort should return to page one; moving to the next page should preserve the sort and filters.