Skip to content
Merged
38 changes: 38 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: Release

# Builds and publishes dbdeployer binaries when a version tag is pushed.
# Uses GoReleaser with the config in .goreleaser.yaml.
# Produces: linux/darwin x amd64/arm64 tarballs + checksums.txt
#
# Security note: no user-controlled inputs are used. Triggers only on
# version tags pushed by maintainers.

on:
push:
tags:
- 'v*'

permissions:
contents: write

jobs:
release:
name: Build and Release
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- uses: actions/setup-go@v5
with:
go-version: '1.22'

- name: Run GoReleaser
uses: goreleaser/goreleaser-action@v6
with:
distribution: goreleaser
version: latest
args: release --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
145 changes: 145 additions & 0 deletions .github/workflows/villagesql_flavor_test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
name: VillageSQL Flavor Test

# Tests VillageSQL flavor detection, sandbox deployment, and capability
# inheritance using the official VillageSQL 0.0.3 release tarball from
# https://github.com/villagesql/villagesql-server/releases/tag/0.0.3
#
# The workflow verifies:
# 1. dbdeployer unpack detects the flavor as "villagesql" via the
# share/villagesql_schema.sql marker file
# 2. Single sandbox deployment works with --unpack-version mapping
# 3. Replication deployment works with data verification
# 4. VillageSQL capabilities correctly inherit MySQL capabilities
#
# Note: The 0.0.3 tarball contains two symlinks in
# mysql-test/suite/villagesql/examples/ that point outside the extraction
# directory. These are stripped before unpacking. See
# https://github.com/villagesql/villagesql-server/issues/237
#
# Security note: this workflow uses no user-controlled inputs (issue
# bodies, PR titles, commit messages, etc.). All values are hardcoded.

on:
push:
branches: [master]
pull_request:
branches: [master]

env:
VILLAGESQL_VERSION: "0.0.3"
VILLAGESQL_SHA256: "8b15522a973b17b430ed9e64e8bdbf97bf858cef028bfbc7f9b9608002406393"
VILLAGESQL_TARBALL: "villagesql-dev-server-0.0.3-dev-linux-x86_64.tar.gz"
VILLAGESQL_URL: "https://github.com/villagesql/villagesql-server/releases/download/0.0.3/villagesql-dev-server-0.0.3-dev-linux-x86_64.tar.gz"
# VillageSQL uses its own version scheme (0.0.3). Map to MySQL 8.0.40 for
# capability lookups (mysqld --initialize, CREATE USER, GTID, etc.)
MYSQL_MAPPED_VERSION: "8.0.40"
GO111MODULE: on
SANDBOX_BINARY: ${{ github.workspace }}/opt/mysql

jobs:
villagesql-deploy:
name: Deploy (VillageSQL ${{ env.VILLAGESQL_VERSION }})
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🌐 Web query:

GitHub Actions context availability for jobs.<job_id>.name env

💡 Result:

No, the jobs.<job_id>.name (the display name of a specific job) is not directly available as an environment variable or in the job context. The job context provides information about the current running job only, with properties like job.status, job.check_run_id, but does not include a name property listing the display name (official docs list explicit properties without name). The jobs context is available only in reusable workflows and contains outputs and results of previous jobs (e.g., jobs.example_job.result), not names. However, within steps of the current job (after it starts running on the runner): - The job ID (YAML key, e.g., my-job) is available as the environment variable $GITHUB_JOB or ${{ github.job }}. - The display name (from jobs.<job_id>.name) can be accessed via ${{ jobs[github.job].name }} or $GITHUB_JOB_NAME if using certain third-party actions, but this relies on the jobs context being populated with name info during step execution (community-reported working as of 2023-2025). jobs.&lt;job_id&gt;.name cannot be used in job-level keys like jobs.&lt;job_id&gt;.env (only github, needs, strategy, matrix, vars, secrets, inputs available there per context availability table). To use the current job's display name in env, set it explicitly in job env using allowed contexts (e.g., MY_JOB_NAME: static or ${{ strategy.job }} if matrix), or compute it in a step and export via $GITHUB_ENV.

Citations:


🏁 Script executed:

cat -n .github/workflows/villagesql_flavor_test.yml | sed -n '35,50p'

Repository: ProxySQL/dbdeployer

Length of output: 554


🏁 Script executed:

rg "VILLAGESQL_VERSION" .github/workflows/villagesql_flavor_test.yml -n -B 2 -A 2

Repository: ProxySQL/dbdeployer

Length of output: 1599


env context is not available in jobs.<job_id>.name.

Per GitHub Actions context availability, jobs.<job_id>.name only supports github, needs, vars, inputs, matrix, and strategy — not env. The interpolation ${{ env.VILLAGESQL_VERSION }} will silently expand to an empty string, producing the job name Deploy (VillageSQL ).

Hardcode the version or move it to vars/inputs.

Proposed fix
   villagesql-deploy:
-    name: Deploy (VillageSQL ${{ env.VILLAGESQL_VERSION }})
+    name: Deploy (VillageSQL 0.0.3)
     runs-on: ubuntu-22.04
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
name: Deploy (VillageSQL ${{ env.VILLAGESQL_VERSION }})
name: Deploy (VillageSQL 0.0.3)
🧰 Tools
🪛 actionlint (1.7.12)

[error] 41-41: context "env" is not allowed here. available contexts are "github", "inputs", "matrix", "needs", "strategy", "vars". see https://docs.github.com/en/actions/learn-github-actions/contexts#context-availability for more details

(expression)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/villagesql_flavor_test.yml at line 41, The job name uses
`${{ env.VILLAGESQL_VERSION }}` which is not available in `jobs.<job_id>.name`
so it will expand to empty; update the workflow to either hardcode the version
in the `name` line or move the value to a supported context (e.g., define
VILLAGESQL_VERSION under `vars:` or as a workflow `input`) and reference it as
`${{ vars.VILLAGESQL_VERSION }}` or `${{ inputs.VILLAGESQL_VERSION }}` in the
`name` field; look for the `name: Deploy (VillageSQL ${{ env.VILLAGESQL_VERSION
}})` line and change the source of VILLAGESQL_VERSION accordingly.

runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4

- uses: actions/setup-go@v5
with:
go-version: '1.22'

- name: Install system libraries
run: |
sudo apt-get update
sudo apt-get install -y libaio1 libnuma1 libncurses5

- name: Build dbdeployer
run: go build -o dbdeployer .

- name: Cache VillageSQL tarball
uses: actions/cache@v4
with:
path: /tmp/villagesql-tarball
key: villagesql-${{ env.VILLAGESQL_VERSION }}-linux-x86_64-v1

- name: Download and verify VillageSQL tarball
run: |
mkdir -p /tmp/villagesql-tarball
if [ ! -f "/tmp/villagesql-tarball/$VILLAGESQL_TARBALL" ]; then
echo "Downloading VillageSQL $VILLAGESQL_VERSION..."
curl -L -f -o "/tmp/villagesql-tarball/$VILLAGESQL_TARBALL" "$VILLAGESQL_URL"
fi
echo "Verifying checksum..."
echo "$VILLAGESQL_SHA256 /tmp/villagesql-tarball/$VILLAGESQL_TARBALL" | sha256sum -c
ls -lh "/tmp/villagesql-tarball/$VILLAGESQL_TARBALL"

- name: Repack tarball without broken symlinks
run: |
cd /tmp
mkdir -p villagesql-staging villagesql-clean
tar xzf "villagesql-tarball/$VILLAGESQL_TARBALL" -C villagesql-staging
INNER="villagesql-staging/villagesql-dev-server-$VILLAGESQL_VERSION-dev-linux-x86_64"
# Remove symlinks that point outside the extraction directory
# (https://github.com/villagesql/villagesql-server/issues/237)
rm -f "$INNER/mysql-test/suite/villagesql/examples/vsql-complex"
rm -f "$INNER/mysql-test/suite/villagesql/examples/vsql-tvector"
tar czf "villagesql-clean/$VILLAGESQL_TARBALL" -C villagesql-staging \
"villagesql-dev-server-$VILLAGESQL_VERSION-dev-linux-x86_64"
rm -rf villagesql-staging

- name: Test unpack with --unpack-version
run: |
mkdir -p "$SANDBOX_BINARY"
./dbdeployer unpack "/tmp/villagesql-clean/$VILLAGESQL_TARBALL" \
--sandbox-binary="$SANDBOX_BINARY" \
--unpack-version="$MYSQL_MAPPED_VERSION"

# Verify flavor detected as villagesql
FLAVOR_FILE="$SANDBOX_BINARY/$MYSQL_MAPPED_VERSION/FLAVOR"
[ -f "$FLAVOR_FILE" ] || { echo "FAIL: No FLAVOR file"; exit 1; }
FLAVOR=$(cat "$FLAVOR_FILE")
echo "Detected flavor: $FLAVOR"
[ "$FLAVOR" = "villagesql" ] || { echo "FAIL: expected 'villagesql', got '$FLAVOR'"; exit 1; }

# Verify marker file
MARKER="$SANDBOX_BINARY/$MYSQL_MAPPED_VERSION/share/villagesql_schema.sql"
[ -f "$MARKER" ] || { echo "FAIL: marker file not found"; exit 1; }
echo "OK: flavor=villagesql, marker file present"

- name: Test deploy single sandbox
run: |
./dbdeployer deploy single "$MYSQL_MAPPED_VERSION" --sandbox-binary="$SANDBOX_BINARY"
VERSION=$(~/sandboxes/msb_*/use -BN -e "SELECT VERSION();")
echo "Server version: $VERSION"
echo "$VERSION" | grep -qi villagesql || { echo "FAIL: expected villagesql in VERSION()"; exit 1; }
echo "OK: VillageSQL single sandbox running"
./dbdeployer delete all --skip-confirm

- name: Test deploy replication sandbox
run: |
./dbdeployer deploy replication "$MYSQL_MAPPED_VERSION" --sandbox-binary="$SANDBOX_BINARY"
~/sandboxes/rsandbox_*/check_slaves
~/sandboxes/rsandbox_*/test_replication
echo "OK: VillageSQL replication sandbox works"
./dbdeployer delete all --skip-confirm

- name: Cleanup
if: always()
run: |
./dbdeployer delete all --skip-confirm 2>/dev/null || true
pkill -9 -u "$USER" mysqld 2>/dev/null || true

villagesql-capabilities:
name: Capability Inheritance Tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-go@v5
with:
go-version: '1.22'

- name: Run capability tests
run: go test ./common/... -v -run TestHasCapability -count=1

- name: Run copy capabilities tests
run: go test ./common/... -v -run TestCopyCapabilities -count=1
130 changes: 130 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,133 @@
## 2.2.2 18-Apr-2026

## NEW FEATURES

* Add VillageSQL flavor support. VillageSQL is a MySQL drop-in replacement
with extensions. Its tarballs are detected via the unique marker file
`share/villagesql_schema.sql` and reuse MySQL's sandbox lifecycle (init,
start, stop, grants, replication) unchanged. Because VillageSQL uses its
own version scheme (e.g. 0.0.3), unpacking requires `--unpack-version`
mapped to the MySQL base version (e.g. `--unpack-version=8.0.40`).
* Add VillageSQL to CI: real tarball download, SHA256 checksum verification,
single and replication deployment tests, capability inheritance tests.

## 2.2.1 09-Apr-2026

## BUGS FIXED

* Fix InnoDB Cluster deployment: let mysqlsh manage Group Replication from
scratch instead of conflicting with dbdeployer's GR setup
* Fix InnoDB Cluster Basedir template pointing to wrong directory
* Fix `--with-proxysql` failing for InnoDB Cluster (wrong sandbox path in
ProxySQL config)
* Fix Router port extraction including config file path in the result
* Fix Router start hanging forever when mysqlrouter process forks
* Fix ProxySQL grep -v exiting under `set -e` in monitoring scripts
* Fix ProxySQL GR monitor seeing all nodes as offline (hostgroup 3)
* Fix fan-in CREATE DATABASE on node2 conflicting with node1's database
* Fix copy of mysqlsh lib/mysqlsh/ directory (only .so files were copied)
* Fix PostgreSQL multiple sandbox directory naming
* Fix symlink opt/mysql for ts tests in CI (HOME path mismatch)
* Remove MariaDB 11.4 from CI (authentication bug #82)

## CI

* Add install script test workflow (downloads and verifies dbdeployer install)
* Drop macos-13 from install test (unsupported runner)
* Replace sleep+check with retry loops for replication verification in CI

## 2.2.0 08-Apr-2026

## NEW FEATURES

* Add MariaDB 10.6/10.11/11.4/11.7/11.8 and Percona Server 5.7/8.0/8.4
to tarball registry
* Add ts replication test suite (MySQL 5.7, 8.0, 8.4, 9.5)
* Add PostgreSQL ts testscript tests (single + replication)
* Add MySQL 9.5 support for semisync and ts replication tests

## BUGS FIXED

* Use Slave|Replica pattern for IO/SQL thread status in multi-source
ts tests (compatibility with MySQL 8.x terminology)
* Resolve CI failures for MariaDB, Percona, ts replication, and fan-in

## CI

* Add Percona Server and MariaDB integration tests
* Add PostgreSQL ts testscript tests to CI
* Add ts replication test suite to CI (5.7, 8.0, 8.4, 9.5)

## 2.1.1 04-Apr-2026

## BUGS FIXED

* Fix macOS --minimal fallback and `--guess` using real URL patterns
* Fix install script to download checksums.txt instead of per-file .sha256

## 2.1.0 04-Apr-2026

## NEW FEATURES

* Add InnoDB Cluster topology (`--topology=innodb-cluster`) with MySQL Shell
and MySQL Router support
* Add ProxySQL GR-aware hostgroups for InnoDB Cluster and Group Replication
(`--with-proxysql` configures reader/writer hostgroups automatically)
* Add `--topology=group` single-primary and multi-primary Group Replication
with full CI coverage
* Add fan-in and all-masters replication topologies with data verification
* Add `downloads add-url` command for custom tarball URLs
* Add MySQL 8.4-specific replication and group replication templates
* Add ProxySQL PostgreSQL backend wiring (`pgsql_servers/pgsql_users`)
* Add `--provider` flag and PostgreSQL routing to all deploy commands
* Add `dbdeployer deploy postgresql` standalone command
* Add `dbdeployer init --provider=postgresql` for one-command setup
* Add macOS PostgreSQL support via Postgres.app binary detection
* Add cross-database topology constraint validation
* Add comprehensive topology/provider/proxy reference documentation
* Add group replication, fan-in, all-masters, PostgreSQL multiple tests to CI
* Add InnoDB Cluster integration tests (MySQL 8.4.8 + 9.5.0)
* Add functional verification (write/read) to all integration tests
* Add ProxySQL `--bootstrap` mode test script
* Add admin web UI proof of concept

## BUGS FIXED

* Replace `\G` with `--vertical` in all replication templates (MySQL 9.5 compat)
* Fix semisync template variable scoping and version detection
* Fix PostgreSQL deb extraction version detection and binary setup
* Fix PostgreSQL initdb requiring empty data dir (create log dir after initdb)
* Fix PostgreSQL share files for deb-extracted binaries (timezonesets path)
* Remove dead commented-out `semisync_master_enabled` from template
* Fix `gosec` and `staticcheck` lint warnings in PostgreSQL and ProxySQL code

## 2.0.0 24-Mar-2026

Initial release under the ProxySQL organization. Forked from
[datacharmer/dbdeployer](https://github.com/datacharmer/dbdeployer) v1.73.0
with Giuseppe Maxia's blessing.

## NEW FEATURES

* PostgreSQL provider: full provider architecture with `initdb`, config
generation (`postgresql.conf`, `pg_hba.conf`), single sandbox deployment,
streaming replication via `pg_basebackup`, and monitoring scripts
* PostgreSQL deb extraction for binary management (`unpack --provider=postgresql`)
* ProxySQL provider: standalone and topology-integrated deployment
(`--with-proxysql` wires read/write split into any MySQL/PostgreSQL topology)
* Provider interface with `SupportedTopologies` and `CreateReplica`
* Add MySQL 8.4.0–8.4.8, 9.0.1, 9.1.0, 9.2.0, 9.3.0–9.5.0 to tarball registry
* Add `dbdeployer init` with curl-based install script
* Add admin web UI proof of concept
* Add comprehensive website documentation at proxysql.github.io/dbdeployer

## CI

* Full GitHub Actions CI pipeline: lint, unit tests, build verification
* Integration tests: MySQL, Percona Server, MariaDB, PostgreSQL, InnoDB Cluster,
Group Replication, fan-in, all-masters, ProxySQL wiring
* Install script test workflow across multiple OS versions

## 1.73.0 09-Jul-2023

## NEW FEATURES
Expand Down
18 changes: 18 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,23 @@ dbdeployer deploy replication 16.13 --provider=postgresql

> **Note:** The `apt-get download` command downloads `.deb` files to the current directory without installing anything. Your system is untouched. See the [PostgreSQL provider guide](https://proxysql.github.io/dbdeployer/providers/postgresql/) for details and alternative installation methods.

### VillageSQL

[VillageSQL](https://github.com/villagesql/villagesql-server) is a MySQL drop-in replacement with extensions (custom types, VDFs). Since it uses its own version scheme, unpack with `--unpack-version` mapped to the MySQL base version:

```bash
# Download from GitHub Releases
curl -L -o villagesql-dev-server-0.0.3-dev-linux-x86_64.tar.gz \
https://github.com/villagesql/villagesql-server/releases/download/0.0.3/villagesql-dev-server-0.0.3-dev-linux-x86_64.tar.gz

# Unpack with MySQL 8.0 version mapping (required for capabilities)
dbdeployer unpack villagesql-dev-server-0.0.3-dev-linux-x86_64.tar.gz --unpack-version=8.0.40
Comment on lines +67 to +73
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Make the quick-start use the cleaned tarball path.

The wiki notes VillageSQL 0.0.3 has out-of-tree symlinks that dbdeployer rejects, so this direct dbdeployer unpack example can fail for users following the README. Either add the symlink cleanup step here or link to the workaround before the unpack command.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@README.md` around lines 67 - 73, The README quick-start uses dbdeployer
unpack directly on villagesql-dev-server-0.0.3-dev-linux-x86_64.tar.gz which can
fail due to out-of-tree symlinks; update the instructions to either (a) use the
cleaned tarball path (the cleaned filename or URL for the symlink-free release)
in the curl/unpack example or (b) insert a pre-unpack symlink cleanup step
(extract the tar to a temp dir, remove or recreate offending symlinks, repack or
point dbdeployer at the cleaned directory) and/or add a link to the wiki
workaround; reference the tarball name
villagesql-dev-server-0.0.3-dev-linux-x86_64.tar.gz and the dbdeployer unpack
command so readers can find and replace the failing command.


# Deploy
dbdeployer deploy single 8.0.40
~/sandboxes/msb_8_0_40/use -e "SELECT VERSION();"
```

## Supported Databases

| Provider | Single | Replication | Group Replication | ProxySQL Wiring |
Expand All @@ -71,6 +88,7 @@ dbdeployer deploy replication 16.13 --provider=postgresql
| MariaDB | ✓ | ✓ | — | ✓ |
| NDB Cluster | ✓ | ✓ | — | — |
| Percona XtraDB Cluster | ✓ | ✓ | — | — |
| VillageSQL | ✓ | ✓ | — | — |

## Key Features

Expand Down
15 changes: 15 additions & 0 deletions common/capabilities.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ const (
NdbFlavor = "ndb"
PxcFlavor = "pxc"
TiDbFlavor = "tidb"
VillageSQLFlavor = "villagesql"

// Feature names
InstallDb = "installdb"
Expand Down Expand Up @@ -255,6 +256,13 @@ var FlavorCompositionList = []flavorIndicator{
},
flavor: PerconaServerFlavor,
},
{
AllNeeded: false,
elements: []elementPath{
{"share", globals.FnVillagesqlSchema},
},
flavor: VillageSQLFlavor,
},
{
AllNeeded: false,
elements: []elementPath{
Expand Down Expand Up @@ -287,6 +295,12 @@ var PerconaCapabilities = Capabilities{
Features: MySQLCapabilities.Features,
}

var VillageSQLCapabilities = Capabilities{
Flavor: VillageSQLFlavor,
Description: "VillageSQL server",
Features: MySQLCapabilities.Features,
}
Comment on lines +298 to +302
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

VillageSQL is described as having extensions (custom types, VDFs). Assigning MySQLCapabilities.Features directly to VillageSQLCapabilities.Features results in both flavors sharing the same map instance in memory. Any future changes to VillageSQL's capabilities would inadvertently affect MySQL's capabilities as well. It is safer to create a copy of the map using addCapabilities with an empty list, similar to how PxcCapabilities is defined.

Suggested change
var VillageSQLCapabilities = Capabilities{
Flavor: VillageSQLFlavor,
Description: "VillageSQL server",
Features: MySQLCapabilities.Features,
}
var VillageSQLCapabilities = Capabilities{
Flavor: VillageSQLFlavor,
Description: "VillageSQL server",
Features: addCapabilities(MySQLCapabilities.Features, FeatureList{}),
}


var TiDBCapabilities = Capabilities{
Flavor: TiDbFlavor,
Description: "TiDB isolated server",
Expand Down Expand Up @@ -382,6 +396,7 @@ var MySQLShellCapabilities = Capabilities{
var AllCapabilities = map[string]Capabilities{
MySQLFlavor: MySQLCapabilities,
PerconaServerFlavor: PerconaCapabilities,
VillageSQLFlavor: VillageSQLCapabilities,
MariaDbFlavor: MariadbCapabilities,
TiDbFlavor: TiDBCapabilities,
NdbFlavor: NdbCapabilities,
Expand Down
Loading
Loading