diff --git a/.gitignore b/.gitignore index d37e4d8..3d332c7 100644 --- a/.gitignore +++ b/.gitignore @@ -51,3 +51,9 @@ test/stack/tmp/ # dist/ (already ignored above); this is a separate directory so the two # builds never delete each other's output, and nothing reads it. packages/ui/dist-ts/ + +# The lock backup.sh and restore.sh hold while they run. +.backup-restore-*.lock/ +# The mark restore.sh leaves while the stores may be half restored, one per +# install, and the file it is written to first. +.restore-incomplete-* diff --git a/CLAUDE.md b/CLAUDE.md index 18d8bf3..3b2b3ba 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -57,8 +57,9 @@ built on the admin API and nothing it can reach that a script cannot), retention periods on the settings row, enforced hourly by the worker, dropping raw clicks a partition at a time and blanking the address on a click in place), the CLI (`migrate`, `domain add|list|verify`, `link add`, `settings show|set`, `ipdata status|update`, -`admin create|passwd`, `admin totp disable`, `apikey create|list|revoke`), `install.sh`, a -Compose stack, and the restart durability and stack test suites. +`admin create|passwd`, `admin totp disable`, `apikey create|list|revoke`, `version`), +`install.sh`, `backup.sh` and `restore.sh`, a Compose stack, and the restart durability and +stack test suites. **A retention period is a floor, not a deadline, and it is never defaulted by a reader that deletes.** `clicks` is partitioned by month and dropped whole, so 90 days keeps 90 to @@ -106,7 +107,9 @@ What does not exist yet, and must not be implied by any documentation: recognised by ASN only, since no cloud provider's range file states a licence. The region and city columns exist on a click and are always empty, and a breakdown by either would be a new materialized view and a backfill of it. -- **Backup and restore.** +- **Backups that encrypt, rotate or leave the server**, and **a restore of one part of a + backup**. `backup.sh` writes one directory; `restore.sh` replaces everything and is an + outage while it runs. - Segments ClickHouse rejects are set aside as `.bad` files, and nothing reports them. - One redirect process per spool directory. - A full spool stops recording without stopping redirects; the drop count is on @@ -165,7 +168,8 @@ packages/ui/ the web interface: React 19, Tailwind 4, a Vite build the ad Radix overlay primitive, no runtime import from a service package, one

per screen. Every time shown is the browser's own zone, except a click's raw fields, which are labelled UTC. -packages/cli/ `clickmonk migrate | domain | link add | settings | ipdata | admin | apikey`. +packages/cli/ `clickmonk migrate | domain | link add | settings | ipdata | admin | apikey | + version`. Two commands here are deliberately not API routes: `admin create`, because nothing can authenticate before the account exists, and `admin totp disable`, because it is the way back in when the second @@ -176,6 +180,13 @@ caddy/ the Caddyfile, and the tls.d/ and proxy.d/ drop-in directori admin service and every other name to the redirect, with a CEL expression rather than a host matcher so an unset value starts. install.sh writes .env once, with fresh secrets, and starts the stack. +backup.sh copies Postgres, ClickHouse, the spool, caddy-data and .env into one + directory, stopping only the worker, and only while the spool and + ClickHouse are copied. +restore.sh checks a backup, then replaces all four stores with it; the only + script here that deletes data. Both source backup-lib.sh, which + holds the lock they share: one run at a time per install. +clickhouse/ backup-disk.xml, the disk ClickHouse's own BACKUP writes to. test/stack/ brings the whole stack up against a local certificate authority and DNS server; proves TLS issuance, domain verification, real client addresses over IPv4 and IPv6, and the admin host — its @@ -267,7 +278,9 @@ up, sign in to it over HTTPS on the admin host, and drive a password-protected l the form to the redirect. One of them clicks a link through Caddy and then reads that click back through the admin host as a summary, a chart, a breakdown, a log page and a CSV, and lowers a retention period and watches the worker enforce it — the one path where -the admin service's own ClickHouse configuration has to be right. They bind **80 and 443**; the +the admin service's own ClickHouse configuration has to be right. One of them takes backups +of a running stack, refuses damaged and newer ones without touching anything, then destroys +every volume and restores into an empty stack. They bind **80 and 443**; the durability suite binds 8080 and 8123 and the test databases 8123 and 5433. Run one at a time. @@ -329,7 +342,9 @@ That rule is easy to satisfy badly, so: Anything that runs on an operator's host rather than in a container — an installer, backup or restore — targets **bash 3.2**, the version macOS still ships: no associative -arrays, no `mapfile`. `install.sh` is the first. `shellcheck` is pinned as a Docker image +arrays, no `mapfile`. `install.sh` is the first. `backup.sh`, `backup-lib.sh` and +`restore.sh` follow the same rules, and the backup suite parses all three under a real +bash 3.2. `shellcheck` is pinned as a Docker image and run by `pnpm lint:sh` over every `*.sh` in the repository, so a new script is linted without anything being added to the script, and by CI with the same command, so a failure there is reproducible here. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 59e5b94..d2e5a4e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -21,7 +21,7 @@ Thanks for your interest. ClickMonk is early: the redirect and the click pipelin ## Development setup -You need Node 22, Docker with Compose v2, and `corepack` (it ships with Node). Then: +You need Node 22, Docker with Compose 2.23 or later, and `corepack` (it ships with Node). Then: corepack enable pnpm install diff --git a/README.md b/README.md index 4fb36a6..37c4ab3 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,8 @@ on your own infrastructure, and your click data stays yours. verified, and for the admin host name if you set one, so a host name somebody else points at your server never makes your install ask a certificate authority for anything. - **Domain verification.** `clickmonk domain add` prints a TXT record to publish. The - worker looks for it every five minutes, and `clickmonk domain verify` looks now. Until + worker looks for it every five minutes, up to 50 domains a pass with the longest unchecked + first, and `clickmonk domain verify` looks now. Until it is found, links on that domain answer 404 and it gets no certificate. A check that fails later is recorded and shown, and never takes a verified domain back down. - **`./install.sh`**, which writes `.env` with fresh secrets the first time, builds the @@ -47,8 +48,9 @@ on your own infrastructure, and your click data stays yours. - **Traffic classification.** Every click is classed as human, bot, abuser, anonymous (a Tor exit), datacenter, or unknown when the IP checks could not run, from its user-agent, the number of requests from the same client in the current one-minute window, - and IP data held on your server. Each class other than human has an action: flag (the - default), nothing, block, or send to a safe URL. A flagged click never uses up a click + and IP data held on your server. Bot, abuser, anonymous and datacenter each have an + action: flag (the default), nothing, block, or send to a safe URL. An unknown click is sent + on as a human one is. A flagged click never uses up a click cap, but a cap that is used up closes the link to it like any other. A HEAD request, which link checkers and preview bots send, gets the same answer as a GET, is recorded as a bot unless another check gives it a class, and never uses up a click cap. Each @@ -66,10 +68,11 @@ on your own infrastructure, and your click data stays yours. a chart by day tells you it gave you the whole day. - **The click log, and a CSV of it.** `GET /api/clicks` lists the clicks themselves, newest first, filterable by link, traffic class, outcome, country and time, and exact to the - millisecond. `GET /api/clicks.csv` is the same rows as a file: it streams, so the window - can be as wide as you like, and when it reaches its row cap it says so in a header rather - than handing you a prefix that looks complete. The file identifies visitors, which - "Reading the clicks back out" spells out before you send one to anybody. + millisecond. `GET /api/clicks.csv` is the same rows as a file: it streams, so the file is + never built in memory, and its window, like every other, is at most 400 days. When it + reaches its row cap it says so in a header rather than handing you a prefix that looks + complete. The file identifies visitors, which "Reading the clicks back out" spells out + before you send one to anybody. - **Addresses are shown as networks, always.** The log and the export show `198.51.100.0/24` and `2001:db8:1234:5678::/64`, never the address itself — enough to see a pattern or a bot, and not a file of addresses. The whole address stays in the database @@ -83,7 +86,8 @@ on your own infrastructure, and your click data stays yours. 90 days keeps 90 to 121 days of clicks and 30 days of addresses keeps them 30 to 61. - **The CLI**: `clickmonk migrate`, `clickmonk domain add|list|verify`, `clickmonk link add`, `clickmonk settings show|set`, `clickmonk ipdata status|update`, `clickmonk admin create`, - `clickmonk admin passwd`, `clickmonk admin totp disable`, `clickmonk apikey create|list|revoke`. + `clickmonk admin passwd`, `clickmonk admin totp disable`, `clickmonk apikey create|list|revoke`, + `clickmonk version`. `settings set` covers the traffic actions, the safe URL, the abuser threshold and how long this install keeps clicks and the addresses on them (`--keep-clicks`, `--keep-addresses`, each taking a number of days or `never`). @@ -92,6 +96,11 @@ on your own infrastructure, and your click data stays yours. - **A Docker Compose stack** that runs all of it, and a test that restarts each service under continuous traffic and requires every redirect the client received to arrive in ClickHouse as a click. +- **Backup and restore.** `./backup.sh` copies everything an install holds — the clicks + and rollups, the configuration, clicks still waiting in the spool, the certificates and + `.env` — while links keep answering, and `./restore.sh` puts all of it back, into the + same release or a newer one. A test takes a backup, destroys every volume, restores into + an empty stack and counts every click once. See ["Backup and restore"](#backup-and-restore). What does not work yet: @@ -131,7 +140,7 @@ What does not work yet: check did not find its token, and every domain no check has reached, which is what an operator has instead. A domain verified by hand (`domain add --verified`) is the one exception: it is not listed until a check has passed for it at least once, because - nothing changed for it — a check that then fails, having once passed, is listed again. + nothing changed for it — a check that then fails, having once passed, is listed. - **Most link settings in the CLI.** `clickmonk link add` sets targets, a backup URL, a click cap, an expiry, passthrough and traffic action overrides only, and no command changes a link once it is added. Per-device destinations, a returning-visitor @@ -146,7 +155,11 @@ What does not work yet: network (ASN) only. The providers' range files state no licence, so they are not used. - **Region and city.** A click records its country only. The two columns exist and are always empty, and a breakdown by either would need a rollup of its own. -- **Backup and restore.** +- **Backups that look after themselves.** `backup.sh` writes one directory and stops: + it does not encrypt it, rotate old ones, or copy anything off the server. Those are yours, + and the section on backups says what to use. +- **Restoring one part of a backup.** A restore replaces everything, and your links answer + nothing while it runs. - **Rejected clicks are not reported.** A batch of clicks ClickHouse refuses is set aside as a `.bad` file in the spool, and nothing tells you it is there. - **More than one redirect process per spool directory.** @@ -156,26 +169,30 @@ What does not work yet: - **The redirect and the admin API are reachable from the whole compose network.** None of their ports is published on the host, but any other container on the stack's own Docker network can reach all of them, not only Caddy. On `redirect:9091` that means reading the - `ask` check, which lists this install's verified domains. On `redirect:8080` it means - more: the stack trusts a forwarded address from that network - (`CLICKMONK_TRUSTED_PROXIES` is `uniquelocal,loopback`), so a container inside the - install can set `X-Forwarded-For` and choose the address recorded, counted and looked up - for every click it sends. On `admin:9100` a request still has to carry the admin host - name in `Host` — a forwarded header will not do, and every route but `/health` is - refused without it — and then it still needs a credential. Every container on that - network is one you put there, which is what keeps this a limit rather than a way in. + `ask` check, which answers, for any name, whether it is a verified domain here. On + `redirect:8080` it means more: the stack trusts a forwarded address from that network + (`CLICKMONK_TRUSTED_PROXIES` is `uniquelocal,loopback`), so a container inside the install + can set `X-Forwarded-For` and choose the address recorded, counted and looked up for every + click it sends. On `admin:9100` a request still has to carry the admin host name in `Host` + — a forwarded header will not do, and every route but `/health` is refused without it — + and then it still needs a credential. Every container on that network is one you put + there, which is what keeps this a limit rather than a way in. - **IPv6 coverage depends on the host the tests run on.** The published-port IPv6 test is skipped when that host has no IPv6 address of its own — most CI runners — and is meant to be run by hand, on a host that has one, before a release. A second, always-run test proves the address Caddy passes on is the IPv6 client's own, but only on the stack's unique-local subnet. +How fast a redirect is, how to back an install up, how to upgrade it and what it exposes +each have a section below. + If link tracking is a problem you have today, [open an issue](../../issues) describing it. That is the most useful contribution at this stage. ## Installing -You need a Linux host with Docker and Docker Compose v2, and ports 80 and 443 free. +You need a Linux host with Docker and Docker Compose 2.23 or later (the backup scripts need +it), and ports 80 and 443 free. ```sh git clone https://github.com/ClickMonk/clickmonk.git @@ -219,11 +236,12 @@ Add a domain: docker compose exec -T worker node packages/cli/dist/index.js domain add links.example.com ``` -It prints the TXT record to publish, at `_clickmonk.links.example.com`, and reminds you -to point `links.example.com` at this server yourself — an A or AAAA record, or a CNAME, -whichever your DNS provider gives you. Publish both, then wait: the worker checks every -five minutes, `domain verify links.example.com` checks at once, and `domain list` shows -where each domain stands. +It prints the TXT record to publish, at `_clickmonk.links.example.com`, and reminds you to +point `links.example.com` at this server yourself — an A or AAAA record, or a CNAME, +whichever your DNS provider gives you. Publish both, then wait: the worker checks every five +minutes, up to 50 domains a pass with the longest unchecked first, +`domain verify links.example.com` checks at once, and `domain list` shows where each domain +stands. **Until the TXT record is found the domain serves nothing.** Over plain HTTP, links on it answer 404. Over HTTPS there is no certificate to present, so the connection never gets @@ -233,7 +251,7 @@ out of your install. `domain add --verified` is the way round it, for a trial or domain you proved some other way; it says on screen that no DNS check was made, and such a domain is not shown in `GET /api/alerts` or counted in `GET /api/status` until a check has passed for it once — until then nothing has changed for it, so there is nothing to -warn about. A check that later fails, having once passed, is an alert again. +warn about. A check that later fails, having once passed, is an alert. ClickMonk does not know its own public address, so it records what a domain resolves to rather than judging it — a host behind NAT, a load balancer or a CDN is normal. A domain @@ -262,7 +280,40 @@ Origin certificate. Nothing mounts one into Caddy by default: add the mount in and Caddy exits at boot, unable to find a file that was never there. Flexible mode sends traffic to your server in clear and is not an option. -Certificates live in the `caddy-data` volume. Back it up with the rest. +Certificates live in the `caddy-data` volume, and `backup.sh` copies it with everything +else, so a restored install presents the certificates it had rather than asking the +certificate authority for every domain again at once. + +## How fast a redirect is + +A redirect is answered from memory. The redirect holds every link, domain and setting in an +in-memory copy it refreshes when something changes, looks the visitor's address up in IP data +it also holds in memory, writes the click to a file on local disk, and answers. It makes no +network call for a link without a click cap. A capped link asks Postgres to count the click, +waits at most 150 milliseconds for the answer, and sends the visitor on anyway if none comes; +either database can be down and links keep answering. + +Measured with the stack this repository ships, on an 8-core AMD EPYC virtual machine with +6 GB of memory, over plain HTTP from a container on the stack's own network — so the figure +is Caddy and the redirect, not a network — on a link with no click cap: a median of 1.1 to +1.4 ms and a 99th percentile of 2.3 to 4.4 ms, over three runs of 2,000 requests one after +another. Measure your own server: + +```sh +for i in $(seq 200); do + curl -so /dev/null -w '%{time_starttransfer}\n' -H 'Host: links.example.com' http://localhost/promo +done | sort -n | awk '{a[NR]=$1} END {print "median", a[int(NR/2)], "p99", a[int(NR*0.99)]}' +``` + +**What no self-hosted tool can change is the distance.** A visitor in Sydney clicking a link +served from Frankfurt waits for the round trips between them — a TCP connection, a TLS +handshake and the request — which is hundreds of milliseconds whatever the server does. A +hosted service with servers on every continent answers from somewhere near the visitor; +ClickMonk answers from wherever you run it. Put it near the people who click. + +**Each click is written before the visitor is answered**, and the disk is synced in batches +every 200 milliseconds: a click survives the redirect crashing the moment after it answered, +and a power cut loses at most the last 200 milliseconds of clicks. ## The admin API @@ -476,7 +527,8 @@ do, a `curl` command or the CLI can do too. **What it does not do yet:** there is no time zone setting for the install itself, only the browser's own zone; no bulk operations — one link, one domain, one setting at a time, the same as the CLI; and no local development mode, so trying it needs a real admin host set up, the -same as the API does. See [#23](../../issues/23). +same as the API does. See [#43](../../issues/43), [#39](../../issues/39) and +[#23](../../issues/23). ## Reading the clicks back out @@ -540,7 +592,8 @@ IP list's version and when it was fetched, `null` for a source never fetched and the whole field on an install with no IP data yet — `CLICKMONK_IPDATA_UPDATE=off`, or a fresh one. `ipDataProblem` is set, with `ipData` then `null` too, only when the manifest is there and could not be read; the response never says why, so check the admin service's own -log for the reason. `alerts` is the count `GET /api/alerts` would list, capped the same way. +log for the reason. `alerts` is the count `GET /api/alerts` would list, capped at 501, which +means more than 500. **A report counts whole buckets; the log counts milliseconds.** Before a report is counted, `from` is floored and `to` raised to the next boundary, and **the boundary is the grain that @@ -551,9 +604,9 @@ instants exactly as they were sent. So the same window can answer differently through these surfaces and none of them is broken: ask `from=2026-09-24T10:30:00Z&to=2026-09-24T10:50:00Z` and the summary, the hourly chart and the breakdown all count 10:00 to 11:00, a chart by day counts the whole of the 24th, and the log -counts the twenty minutes you named — three different numbers from one request. Every one of -these six answers repeats the window it actually counted, in its own `window` field. Read -that before comparing two numbers. +counts the twenty minutes you named — three different numbers from one request. Every JSON +answer among these repeats the window it actually counted, in its own `window` field, and the +export puts it in its file name. Read that before comparing two numbers. **What the numbers mean.** `clicks` counts distinct clicks and `visitors` distinct visitor cookies, and both are counted as sets rather than added up: a click the worker shipped twice @@ -610,7 +663,7 @@ running. The window can gain clicks between this answer and the export; the expo header stays the authority on the file it actually wrote. **A cell that would otherwise look like a formula is prefixed with an apostrophe.** A -spreadsheet runs a cell beginning `=`, `+`, `-` or `@`, or with whitespace in front of one, +spreadsheet runs a cell beginning `=`, `+`, `-` or `@`, or with a tab, carriage return or newline, whether the cell is quoted or not. The two columns a visitor's own browser chooses the contents of — `referrer` and `userAgent` — are where that turns up, so those are the ones that can come back with an apostrophe in front of them. @@ -723,8 +776,9 @@ you find out. The worker downloads four lists to your server, and `clickmonk ipdata update` fetches them on demand. The redirect looks each visitor's address up in memory: no lookup -leaves your server, and no request waits for a download. The lists take about 25 MB of -the redirect's memory, and each has a fixed ceiling. The admin service also mounts this +leaves your server, and no request waits for a download. With the editions of September +2026 the lists take about 25 MB of the redirect's memory, measured, and each has a fixed +ceiling. The admin service also mounts this volume, read-only, and reads it for nothing but `GET /api/status` — to say how old each list is. @@ -762,14 +816,14 @@ own lookup format. The same credit is in the web interface's own footer, on ever screen. A download replaces the list in use only when it parses whole and holds at least a -minimum number of entries (about a fifth of a current edition; for country, half the -IPv4 space); otherwise the previous one stays. `clickmonk ipdata status` shows each +minimum number of entries (a small fraction of a current edition, about 5 to 15%; for +country, half the IPv4 space); otherwise the previous one stays. `clickmonk ipdata status` shows each list's version and when it was fetched, and `clickmonk ipdata update` fetches them now. On a server without internet access, set `CLICKMONK_IPDATA_UPDATE=off`. The redirect then -runs without IP data: countries are unknown, so a link limited to a list of countries -sends every visitor to its backup URL, and clicks that no other check marks are classed -unknown rather than human. +runs without IP data: countries are unknown, so a link limited to a list of countries sends +every visitor to its backup URL, or answers 403 when it has none, and clicks that no other +check marks are classed unknown rather than human. bad-asn-list's licence: @@ -795,40 +849,348 @@ bad-asn-list's licence: OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +## Security + +What an install exposes, and what stands behind each part. + +- **Two ports.** Caddy publishes 80 and 443 and no other container publishes anything: + Postgres, ClickHouse, the redirect and the admin service are reachable only on the stack's + own Docker network. A test reads `docker compose config` and fails if that changes. +- **Links are public by nature.** Anyone can request any path on a link domain, so everything + a request can reach is bounded: the path's length, the header block's size, the number of + clients the per-client request count remembers, the password attempts per client per link, + and the spool's size on disk. +- **A certificate only for a name you verified**, or for the admin host you named. A host name + someone else points at your server gets no certificate, and your install asks the + certificate authority for nothing on its behalf. +- **The admin interface and API answer on one host name, over HTTPS**, and nowhere else; + plain HTTP on that name is sent to HTTPS. Signing in takes the one account's password and, + once you turn it on, a code from an authenticator app. After five wrong passwords the + account refuses sign-ins for five minutes, and each further wrong one adds five more, up to + an hour. A session ends after 12 hours unused, and after 30 days whatever happens. Every + request that changes anything, the sign-in included, must carry an `Origin` naming the + admin host unless it carries an API key, which is what a script uses. The interface runs under a content security + policy that allows no inline script and no inline style. +- **Nothing that signs in is stored in a form that can be replayed.** Passwords — the + account's and every link's — and recovery codes are scrypt hashes; session tokens and API + keys are stored as SHA-256 digests. The exception is the two-factor secret, which the + server must be able to read to check a code; it is one reason a backup is a credential. +- **Visitors' addresses** are shown as networks, never as addresses, and blanked in the + database once the address retention period has passed, 30 days on a new install + (["How long ClickMonk keeps things"](#how-long-clickmonk-keeps-things)). +- **What your server contacts:** the certificate authority, for the names you verified (Caddy's + defaults, or the authority you name in `caddy/tls.d/`); the three hosts the four IP data + lists under ["IP data"](#ip-data) come from (`CLICKMONK_IPDATA_UPDATE=off` stops them); DNS + resolvers, to check each domain's verification record (`CLICKMONK_DNS_CHECK=off` stops the + worker's scheduled checks, and `domain verify` and the admin API's check still ask when you + run them); and Docker's registries when you pull or build. ClickMonk sends nothing else + anywhere: no telemetry, no licence check, no update check. A redirect makes no network call + except the click-cap count on a capped link, which goes to your own Postgres. +- **What is not a boundary:** anyone with a shell on the server has everything, `.env` and the + databases included; and the containers on the stack's own network can reach each other, as + the limits under ["Status"](#status) spell out. + +To report a vulnerability, see [SECURITY.md](SECURITY.md) — not a public issue. + +## Backup and restore + +Two scripts sit beside `install.sh`. `backup.sh` is the one you schedule. `restore.sh` is +the one you run once, under pressure, and it is the only script here that deletes data. + +### Taking a backup + +```sh +./backup.sh /var/backups/clickmonk +``` + +Each run writes one new directory, named for the moment it started, in UTC: + +``` +/var/backups/clickmonk/2026-10-01T041500Z/ + clickhouse.zip the clicks and the hourly rollups + postgres.dump domains, links, click-cap counters, settings, the admin account, + its sessions and API keys + spool.tar clicks the redirect has accepted and the worker has not shipped yet + caddy-data.tar certificates, their private keys, and the certificate authority account + env a copy of .env + MANIFEST versions, row counts, and a SHA-256 of each file +``` + +**Your links keep answering while it runs.** The one container it stops is the worker, while +it copies the spool and ClickHouse, and it starts the worker again on every path it can, a +failed step included. A Ctrl-C that arrives while the worker is still stopping does not leave +it stopped: the script lets the stop finish, then starts it again. It tries the start three +times, so a second Ctrl-C that cuts one attempt short does not by itself leave the worker +stopped; if all three fail it says so and prints the command to run, +`docker compose start worker` with any `COMPOSE_*` variables in effect in front of it. +Meanwhile the redirect keeps sending visitors on +and writing their clicks to the spool, and the worker ships them when it is back: the +reports pause, and nothing is lost. A worker that was already stopped when you ran the +script is left stopped. + +**The pause and the disk space grow with your data.** ClickHouse writes its own archive +beside its data before the script copies it out, so a backup needs free space about the size +of ClickHouse's data twice: once on the disk Docker keeps its volumes on, and once at the +destination — both on the same disk if the destination is on this server. `backup.sh` +measures the first before it stops anything, and refuses when there is not room for it plus +a tenth. It does not measure the destination: a copy that runs out of room there fails, and +the half-written directory is deleted. For scale, measured on an 8-core AMD EPYC virtual +machine with 6 GB of memory and ClickHouse 24.8 holding 2.2 GiB of ClickMonk's clicks and +rollups: the archive was 1.77 GB, and ClickHouse took 49 seconds to write it, all of which +the worker was stopped for. Take your own figures from the first backup you run, and +schedule it when a pause of the reports matters least. + +**What the backup is a picture of.** The worker is the only thing that moves clicks from +the spool into ClickHouse, so with it stopped the two are copied at one instant: every click +the redirect had written before that instant is in the backup — in ClickHouse, in the spool, +or in both. Both is normal, and it is not counted twice. The worker deletes a batch of clicks +from the spool only after ClickHouse has accepted it, so a batch caught in between is in both +places, and after a restore it is shipped again; every count ClickMonk makes is of distinct +clicks, so the second copy adds nothing. Postgres is dumped after that, as one consistent +snapshot of its own: a link created in between is in the backup, and a click on it after the +spool was copied is not. + +**One backup or restore at a time.** The two scripts share a lock, the directory +`.backup-restore-.lock` in the checkout, where `` is the install's Compose +project name (`clickmonk` unless you set `COMPOSE_PROJECT_NAME`), and a run that finds it +taken is refused. Two installs run from one checkout under different project names have a +lock each and do not block each other. A run that +is killed outright leaves the lock behind, and every run after it is refused, with the +command that removes it, until you remove it. Nothing removes a lock for being old: two runs +that both judged it stale would both go ahead. + +**It refuses, before it stops anything,** while ClickHouse is still running a backup or a +restore that a killed run started (the server finishes one even when its client is gone), +and while a restore has not finished — see "If a restore is interrupted" below. A backup of +half-restored stores would be complete, checksummed, and a picture of a moment that never +existed. + +A nightly entry for cron: + +``` +17 4 * * * { /srv/clickmonk/backup.sh /var/backups/clickmonk || docker compose --project-directory /srv/clickmonk ps --all; } >>/var/log/clickmonk-backup.log 2>&1 +``` + +The script changes into its own directory, so cron does not need to. The `|| … ps --all` is +not decoration: the script starts the worker again on every exit it can see, but a `SIGKILL` +— an out-of-memory kill, `docker kill`, a hard `systemctl stop` — runs nothing, and can +leave the worker stopped and the lock in place. The `ps --all` puts the first in your log, +as an exited worker, and the next night's refusal says the second. If you pipe the script +anywhere, test `${PIPESTATUS[0]}`, not `$?`. + +**Encryption, rotation and copies off the server are yours to choose.** `find -mtime` for +old directories, `age` to encrypt one, `restic` or `rclone` to send it somewhere else all do +it better than a script here would, and `backup.sh` does none of them. + +### The backup is a credential + +A backup holds everything needed to be this install: + +- `env` has the database passwords and `CLICKMONK_SECRET`, which signs visitor cookies and + the proof a visitor holds after answering a link's password. With it and `postgres.dump`, + which is beside it, anyone can make that proof for any password-protected link. +- `postgres.dump` has the admin password's hash, **the two-factor secret itself** — a server + has to be able to read it to check a code — the hashes of the recovery codes, and the + digests of every API key and session. +- `caddy-data.tar` has every certificate's private key, and the key of the account Caddy + uses with the certificate authority. + +`backup.sh` creates the directory readable by you alone (`0700`) and every file in it +`0600`, from the moment each exists. Treat the directory as you would the server itself, and +encrypt it before it leaves the server. + +### Restoring + +```sh +./restore.sh /var/backups/clickmonk/2026-10-01T041500Z +``` + +You are asked to type the backup's timestamp. There is no `--force`, and no answer within +two minutes is a refusal. + +**Before anything is stopped or changed** it checks every file against its checksum, the +backup against the image this checkout builds, and that ClickHouse's disk has room for it; +then it asks. A refusal at any of those, or any answer but the timestamp, leaves the running +install as it was, not even stopped. + +Then it stops Caddy, the redirect, the admin service and the worker — **your links answer +nothing until it finishes**, which is seconds on a small install and longer on a large one +— replaces ClickHouse, Postgres, the spool and Caddy's certificates with the backup's, +checks the restored schema version and row counts against the manifest, and starts +everything. **Everything recorded since the backup is gone**: clicks, links, domains, +settings. + +**It restores more than data.** The admin account comes back as it was when the backup was +taken: its password, its two-factor setting, the sessions signed in then, and its API keys. +A key you revoked since, or a password you changed since, is back as it was. When a restore +finishes, check the keys and change the password if either has moved on. The script prints +this command with any `COMPOSE_*` variables in effect: + +```sh +docker compose exec -T worker node packages/cli/dist/index.js apikey list +``` + +Click caps count from where the backup left them, and a restored backup older than your +retention periods loses what is past them as soon as the restore starts the worker, whose +first retention pass runs at once. To keep older clicks, lengthen the retention periods +before you restore. + +**Into a newer release.** A backup from an older release restores into a newer one: the +worker applies the migrations the backup is missing when the restore starts it. A backup +from a newer release than your checkout is refused before anything is touched — check out +that release or a later one, and restore again. + +**On a new server.** Clone the repository and check out the release the backup came from or +a later one. Copy the backup's `env` to `.env` **before** the first start, so that visitors +keep their cookies and nobody has to answer a link's password again. Then: + +```sh +./install.sh +./restore.sh /path/to/the/backup/2026-10-01T041500Z +``` + +`install.sh` keeps a `.env` that is already there. `restore.sh` never writes `.env` itself — +the database passwords in it have to match the volumes already on this server — and if its +`CLICKMONK_SECRET` or `CLICKMONK_ADMIN_HOST` differs from the backup's it says so before it +asks you to confirm. + +**Certificates come back too**, so the same names present the same certificates without the +certificate authority being asked again. One that has expired since the backup is renewed +the first time someone asks for it, as any expired certificate is. + +**If a restore is interrupted part way,** it leaves Caddy, the redirect, the admin service +and the worker stopped and says which stores are in which state. It does not start them, +because a redirect running on a half-restored database serves whatever happens to be there +and looks healthy doing it. If it is interrupted while a step is still writing — ClickHouse's +restore, Postgres's, or the spool or Caddy's data — it waits for that step to finish before +it exits, and then says what state it left. Run it again with the same backup; it starts +from the beginning, and the command it prints for that carries any `COMPOSE_*` variables +in effect. A restore killed outright waits for nothing: ClickHouse finishes a restore it has +started even with the script gone, and running the script again is refused, changing +nothing, until it has. Until a restore has finished, `backup.sh` refuses to run; the file +`.restore-incomplete-` in the checkout is what tells it. + +**If the restored stores do not match the manifest** — a schema version or a row count that +differs from what the backup recorded — it lists the differences and leaves the same four +services stopped and `.restore-incomplete-` in place. Restore a different backup, +or start on what was restored with `docker compose up -d`, which the script prints with any +`COMPOSE_*` variables in effect, and then delete `.restore-incomplete-` yourself. + +**A restore needs room too.** It copies the ClickHouse archive into ClickHouse's volume and +restores the database beside it there, so before anything is stopped it refuses unless that +disk has about two and a half times the archive's size free, counting the space the current +ClickHouse database gives up when it is replaced. + +**Not in a backup:** the IP data, which a restore leaves where it is, and which the worker +downloads as soon as it starts on a server that has none (until then countries are unknown, +as on a new install); the redirect's own copy of its configuration, which a restore deletes +and the redirect rebuilds from Postgres; and Caddy's autosaved configuration. + ## Upgrading -Each release that needs anything of you has its own heading here. Read the one you are -coming from, and every one after it. +**Take a backup first.** Migrations run when the worker starts, and a database a newer +release has migrated is refused by an older one: -### Upgrading to the admin API +```sh +./backup.sh /var/backups/clickmonk +``` + +Then move the checkout to the release, build it, and start the worker before anything else: -**`docker compose up -d --build` adds one container and restarts most of the others.** The -new `admin` service starts beside them. The redirect and the worker are rebuilt from the -same image, so they are recreated too, and Caddy is recreated because it reads the new -setting — which makes this a brief outage for your links, as any rebuild is. Postgres and -ClickHouse are left alone. Caddy still binds 80 and 443 and is still the only service with -a published port. +```sh +git fetch --tags +git checkout vX.Y.Z +docker compose pull caddy postgres clickhouse +docker compose build +docker compose up -d worker +docker compose exec -T worker node packages/cli/dist/index.js migrate +docker compose up -d --wait +``` -**There is nothing to run by hand:** the worker applies the schema change when it starts, -as it does for every release, and `up -d --wait` waits for the new container, which becomes -healthy whether or not you have configured it. Clicks are written at record version 3 from -here on, which "Every release: upgrade the worker before the redirect" below covers. +`migrate` is the wait: it takes the lock the worker's own migration holds, so it returns once +the migrations are done, and prints `up to date` or the versions it applied. `up -d worker` +also recreates Postgres or ClickHouse if the pull brought a newer image for either; links +keep answering meanwhile. The last line recreates the redirect and the admin service from +the new image, and any other container whose image or compose definition changed, which is a few +seconds in which links do not answer, as with any rebuild. A change to a file a container +mounts, such as the Caddyfile, is not one of those: the release notes say when to run +`docker compose restart caddy`. Read the release's notes before +you start: a release that needs anything more of you says so there, and under its own +heading below. + +**Why the worker goes first.** Each release's redirect writes clicks in the record version it +knows, and a worker older than the redirect does not read a newer one: it leaves those spool +segments where they are, counting toward the spool's size bound, until a worker that reads +them is running. Nothing is lost if the order slips — the clicks wait — but the reports stop +moving until the worker catches up. Support for reading a record version always ships no +later than writing it, so the worker-first order is always enough. + +**There is no published image.** A release is a tag, and the image is built from the checkout, +as `install.sh` builds it. If you have changed files in your checkout that the release also +changes, `git checkout` refuses and names them; commit or stash the change first. + +**If the new version will not start,** read `docker compose logs worker`. `Database schema +version N is newer than this build understands (M)` means the checkout went backwards, to a +release older than the database. Check out the release you were on, or a later one — or +restore the backup you took before upgrading. It is refused on purpose: an older build would +write what the newer schema no longer expects. + +**Nothing updates itself.** No part of ClickMonk checks for a release or installs one. To hear +about one, watch this repository's releases on GitHub (Watch, then Custom, then Releases). +Security fixes ship in the next release, and only the latest release is supported: see +[SECURITY.md](SECURITY.md). + +### Upgrading a checkout from before 0.1.0 + +0.1.0 is the first release. Before it, this repository could be cloned and run from `main`. +Such a checkout has no `backup.sh`, so its backup is taken from 0.1.0's scripts against the +volumes you already have, before anything is built. Which commands depends on whether the +checkout already serves HTTPS. + +**If it serves HTTPS:** -**The admin API is off until you name a host for it, and your existing `.env` names -none.** `CLICKMONK_ADMIN_HOST` is optional and defaults to empty, so an `.env` written by -an earlier `install.sh` needs no edit, Compose warns about nothing, and the upgraded -install behaves exactly as it did: every name goes to the redirect, and the admin -container answers 503 to everything but its own healthcheck. Naming a host is what turns -it on, and that has one consequence to decide before you do it rather than after: the -first HTTPS request for that name makes Caddy ask a certificate authority for a -certificate in it. ["The admin API"](#the-admin-api) above is how to set it up, and -[Password-protected links](#password-protected-links) is the other half of what this -release adds. +```sh +git fetch --tags +git checkout v0.1.0 +docker compose up -d clickhouse +./backup.sh /var/backups/clickmonk +``` -### Upgrading from before TLS +The third line recreates ClickHouse alone, with the disk 0.1.0's backups are written to; the +clicks in it are untouched, links keep answering, and the worker waits for it. -Four things change, and two of them destroy something. **Nothing in this heading applies to -an install that was already serving HTTPS.** +**If it is from before TLS** (nothing of yours listens on 443; see "If nothing of yours +listens on 443" below), that third line fails, because Compose cannot change the stack's +network in place while the other containers are on it. On Compose 5.1.4 it stops ClickHouse, +leaves it stopped and says the network `has active endpoints`. Take the backup with the stack +down instead, which means your links answer nothing from here until the upgrade's last step: + +```sh +git fetch --tags +git checkout v0.1.0 +docker compose down +docker compose up -d postgres clickhouse +./backup.sh /var/backups/clickmonk +``` + +`down` without `-v` removes the containers and the network and keeps every volume. + +Either way the backup records the release it came from as `unknown`, because the image you +are running predates `clickmonk version`, or is not running at all; its schema version is +exact, and that is what a restore checks. A checkout from before the hourly rollups has none +to back up: the backup counts the clicks alone, and the worker creates the rollups when it +applies the migrations. They start empty, so reports read zeroes for the time before, as +"What does not work yet" says. Then continue from `docker compose pull` in the steps above. +Two older changes need more of you, depending on how old the checkout is. + +**If `docker compose ps` shows no `admin` service,** the upgrade adds it, and it is off until +you name a host for it: `CLICKMONK_ADMIN_HOST` is optional and empty by default, so the `.env` +you have needs no edit and every name keeps going to the redirect. Naming a host is what turns +it on, and the first HTTPS request for that name makes Caddy ask a certificate authority for a +certificate in it. ["The admin API"](#the-admin-api) is how to set it up. + +**If nothing of yours listens on 443** — the redirect answered on `127.0.0.1:8080` — four +things change, and two of them destroy something. **Ports 80 and 443 on the host have to be free, and the redirect publishes nothing.** Caddy binds both, so the stack does not start while something else holds either one — @@ -841,7 +1203,7 @@ itself breaks issuance rather than passing it on — that case needs a certifica already hold, which `caddy/tls.d/00-defaults.caddy` explains. **Remove `CLICKMONK_TRUSTED_PROXIES` from `.env`.** -The previous release told you to set it to the Docker network's gateway or subnet. A +An earlier checkout's README told you to set it to the Docker network's gateway or subnet. A value in `.env` overrides the `uniquelocal,loopback` the stack now sets. The old value named the proxy you ran then — a gateway address, usually — and Caddy's container is not that, so unless what you set covers the whole of the stack's own network the redirect @@ -852,7 +1214,7 @@ own in front of Caddy is not this variable's business any more: name its ranges `caddy/proxy.d/` instead. The one case left for a value here is publishing the redirect's port yourself, which the "IP data" section above describes. -**Domains you added before this release stay verified.** The migration deliberately +**Domains you added before TLS stay verified.** The migration deliberately leaves the `verified` column alone — clearing it would 404 every live link on your install until each domain published a TXT record. But verified is also what makes a domain eligible for a certificate, so each of those domains gets one on its first HTTPS @@ -861,9 +1223,9 @@ as verified with no check recorded. **The stack's own Docker network now has IPv6 on it**, with the unique-local subnet `CLICKMONK_IPV6_SUBNET` names, so that IPv6 visitors arrive as themselves. Your existing -network does not have it, and Compose cannot change a network in place: `docker compose -up -d --build` stops the containers, removes `clickmonk_default`, creates it again and -starts them, so that step is a short outage and the addresses on that network can come +network does not have it, and Compose cannot change a network in place, which is why the +backup above is taken with the stack down: `docker compose down` removes `clickmonk_default` +and the next `docker compose up` creates it again, so the addresses on that network can come back different. Two things can go wrong. If the range collides with a network this host already has, set `CLICKMONK_IPV6_SUBNET` in `.env` to one that does not. If your Docker daemon has no IPv6 support turned on, creating the network fails outright and nothing @@ -872,14 +1234,6 @@ and it is the same setting the "Domains and TLS" section above asks for — or, cannot, delete the `networks:` block at the end of `docker-compose.yml` and accept that every IPv6 visitor is recorded as your Docker bridge's address rather than their own. -### Every release: upgrade the worker before the redirect - -Each release's redirect writes clicks in the record version it knows, and a worker older -than the redirect does not read a newer version: it leaves those spool segments where they -are, and they count toward the spool's size bound until a worker that reads them is -running. The Compose stack builds both from one image, so `docker compose up -d --build` -upgrades them together and this takes care of itself. - ## License Fair-code, under the [Sustainable Use License](LICENSE.md): free to use, self-host and diff --git a/backup-lib.sh b/backup-lib.sh new file mode 100644 index 0000000..fa40b58 --- /dev/null +++ b/backup-lib.sh @@ -0,0 +1,379 @@ +# shellcheck shell=bash +# shellcheck disable=SC2034 # the variables below are read by the scripts that source this file +# +# Shared by backup.sh and restore.sh, which source it. Never executed, so it +# has no shebang and is not executable. +# +# Written for bash 3.2, which is what macOS ships: no associative arrays, no +# `mapfile`, no `${var,,}`. Every docker call is plain `docker compose` with no +# `-f`, so COMPOSE_FILE, COMPOSE_PROJECT_NAME and COMPOSE_ENV_FILES select the +# install exactly as they do for any other compose command. + +# The database and the role in each store, as docker-compose.yml creates them. +CH_DATABASE=clickmonk +PG_DATABASE=clickmonk +PG_USER=clickmonk + +# Paths inside the containers. The first is the disk clickhouse/backup-disk.xml +# declares; the others are where docker-compose.yml mounts the volumes. +CH_BACKUP_DIR=/var/lib/clickhouse/backups +SPOOL_DIR=/var/lib/clickmonk/spool +STATE_DIR=/var/lib/clickmonk/state +CADDY_DATA_DIR=/data + +# The layout of a backup directory. restore.sh refuses any other number. +BACKUP_FORMAT=1 +# Every file a backup holds besides MANIFEST. +ARTEFACTS="spool.tar clickhouse.zip postgres.dump caddy-data.tar env" +# The ClickHouse tables whose rows a restore is checked against, each one +# where it exists. The raw table is always there; the two rollups arrive with +# a migration, so an install from before it has only clicks, and its backup +# records only that. The manifest's rows.clickhouse.* lines say what was counted. +CH_COUNTED="clicks clicks_hourly clicks_hourly_dim" +CH_REQUIRED=clicks + +# The archive's name on the backups disk, and whether this run has put one +# there, so the EXIT trap knows to delete it. +CH_FILE='' +CH_ARTEFACT_CREATED=0 + +# Output that cannot fail. A reader that went away (`| head`) turns a write +# into an error, and a message nobody reads must not end the run. +say() { echo "$@" 2>/dev/null || true; } +note() { echo "$@" >&2 2>/dev/null || true; } + +# ch_query [database] +# +# The password is the container's own CLICKHOUSE_PASSWORD, referenced inside +# the single-quoted program, so the host shell never expands it and it is in +# no host process's arguments. `default` is the database to name while +# `clickmonk` does not exist, between restore.sh's DROP and its RESTORE: +# naming a missing database is itself an error. stdin is closed because +# `exec -T` would otherwise hand the container the caller's. +ch_query() { + docker compose exec -T clickhouse sh -c \ + 'clickhouse-client --user "$CLICKHOUSE_USER" --password "$CLICKHOUSE_PASSWORD" --database "$1" --query "$0"' \ + "$1" "${2:-$CH_DATABASE}" -- one value per line, unaligned. +pg_query() { + docker compose exec -T postgres sh -c \ + 'PGPASSWORD="$POSTGRES_PASSWORD" psql -U "$1" -d "$2" -At -v ON_ERROR_STOP=1 -c "$0"' \ + "$1" "$PG_USER" "$PG_DATABASE" -- the one place either script creates a file by +# writing to it. Under the umask both scripts set first, every file is 0600 +# from the moment it exists; nothing here ever widens it. +artefact_write() { + cat >"$1" +} + +service_running() { + docker compose ps --status running --services 2>/dev/null | grep -qx "$1" +} + +# wait_until_stopped +wait_until_stopped() { + local i=0 + while [ "$i" -lt "$2" ]; do + service_running "$1" || return 0 + i=$((i + 1)) + sleep 1 + done + return 1 +} + +have_sha256() { + command -v sha256sum >/dev/null 2>&1 || + command -v shasum >/dev/null 2>&1 || + command -v openssl >/dev/null 2>&1 +} + +# sha256_stdin -- the hex digest of standard input, with whichever tool this +# host has: sha256sum on Linux, shasum on macOS, openssl as the last resort. +sha256_stdin() { + if command -v sha256sum >/dev/null 2>&1; then + sha256sum | cut -d' ' -f1 + elif command -v shasum >/dev/null 2>&1; then + shasum -a 256 | cut -d' ' -f1 + else + openssl dgst -sha256 -r | cut -d' ' -f1 + fi +} + +sha256_of() { + sha256_stdin <"$1" +} + +# env_file_path -- the file Compose reads this install's variables from: +# COMPOSE_ENV_FILES when it names one file, .env beside docker-compose.yml +# otherwise. More than one is refused by the caller rather than guessed at. +env_file_path() { + case "${COMPOSE_ENV_FILES:-}" in + '') echo ".env" ;; + *,*) return 1 ;; + *) echo "$COMPOSE_ENV_FILES" ;; + esac +} + +# env_value -- the text after the first `=` on the last line that +# sets key, with one pair of matching surrounding quotes removed, as Compose +# reads it. install.sh writes no quotes; an operator may have added them. +env_value() { + local line value='' + while IFS= read -r line || [ -n "$line" ]; do + case "$line" in + "$2="*) value="${line#*=}" ;; + esac + done <"$1" + case "$value" in + \"*\" | \'*\') + if [ "${#value}" -ge 2 ]; then + value="${value#?}" + value="${value%?}" + fi + ;; + esac + printf '%s' "$value" +} + +# read_dir_of -- a tar of on standard output, from the +# service's running container, or from a one-off container of it when it is +# not running. Either way the volume is the one that service mounts. +read_dir_of() { + if service_running "$1"; then + docker compose exec -T "$1" tar -cf - -C "$2" . -- 16 hex characters that tell whether two +# installs share a CLICKMONK_SECRET without being it. The prefix keeps it from +# being the plain digest of the secret, which some other tool might also print. +secret_fingerprint() { + printf 'clickmonk-secret:%s' "$1" | sha256_stdin | cut -c1-16 +} + +# The line `clickmonk version` prints, from a running container of , +# or from a one-off container of the image `docker compose up` would start now. +version_via_exec() { + docker compose exec -T "$1" node packages/cli/dist/index.js version =` for each counted table that +# exists, and a failure when the raw table does not. FINAL, because both +# engines merge rows that share a key when ClickHouse chooses to: counted +# FINAL, the same parts give the same number whenever they are read. +ch_counts() { + local t n present + present="$(ch_query "SELECT name FROM system.tables WHERE database = '$CH_DATABASE' FORMAT TSVRaw")" || return 1 + for t in $CH_COUNTED; do + if ! printf '%s\n' "$present" | grep -qx "$t"; then + [ "$t" = "$CH_REQUIRED" ] && return 1 + continue + fi + n="$(ch_query "SELECT count() FROM $t FINAL")" || return 1 + case "$n" in + '' | *[!0-9]*) return 1 ;; + esac + echo "rows.clickhouse.$t=$n" + done +} + +# file_get -- the text after the FIRST `=` on the line for key. +# Keys never contain `=`; a value might. +file_get() { + local line + while IFS= read -r line || [ -n "$line" ]; do + case "$line" in + "$2="*) + printf '%s' "${line#*=}" + return 0 + ;; + esac + done <"$1" + return 1 +} + +# manifest_get +manifest_get() { + [ -f "$1/MANIFEST" ] || return 1 + file_get "$1/MANIFEST" "$2" +} + +# remove_in_container_artefact -- deletes this run's archive from the +# backups disk. The disk is inside the clickhouse-data volume, so an archive +# left there is a whole backup's worth of that volume, every night. +remove_in_container_artefact() { + [ "$CH_ARTEFACT_CREATED" = 1 ] || return 0 + docker compose exec -T clickhouse rm -f "$CH_BACKUP_DIR/$CH_FILE" "$CH_BACKUP_DIR/$CH_FILE.lock" \ + /dev/null 2>&1 || return 1 + CH_ARTEFACT_CREATED=0 +} + +# remove_orphan_archives -- deletes every archive either script leaves on the +# backups disk (clickmonk-*.zip) and their lock files. Called only while the +# lock below is held, so no run of either script is using one. A client that +# was killed can still reach the server after its trap's rm, and that BACKUP +# then finishes and leaves a whole archive; the next run removes it here. An +# orphan still being written fails once its lock file is gone, and ClickHouse +# removes what it had written. +remove_orphan_archives() { + docker compose exec -T clickhouse sh -c 'rm -f "$0"/clickmonk-*.zip "$0"/clickmonk-*.zip.lock' \ + "$CH_BACKUP_DIR" /dev/null 2>&1 +} + +# The variables that select an install, in the order advice prints them. +COMPOSE_SELECTORS="COMPOSE_PROJECT_NAME COMPOSE_FILE COMPOSE_ENV_FILES COMPOSE_PATH_SEPARATOR COMPOSE_PROFILES" + +# compose_env_prefix -- the COMPOSE_* variables set now, as `NAME=value ` +# words to put in front of a command: advice that names a command must reach +# the same install when it is typed into another shell. Empty on an install +# selected by nothing but its directory. +compose_env_prefix() { + local name value out='' + for name in $COMPOSE_SELECTORS; do + eval "value=\${$name:-}" + [ -n "$value" ] || continue + out="$out$(printf '%s=%q ' "$name" "$value")" + done + printf '%s' "$out" +} + +# compose_project -- the Compose project this install is, worked out the way +# Compose does without asking it (asking fails when the env file is missing, +# which is a refusal these scripts make themselves): COMPOSE_PROJECT_NAME from +# the environment, then from the env file, then the last top-level `name:` in +# the compose files, then the directory's name. Lower case, and only the +# characters Compose keeps. +compose_project() { + local name='' file files envf line + name="${COMPOSE_PROJECT_NAME:-}" + if [ -z "$name" ]; then + envf="$(env_file_path 2>/dev/null)" || envf='' + if [ -n "$envf" ] && [ -r "$envf" ]; then + name="$(env_value "$envf" COMPOSE_PROJECT_NAME)" + fi + fi + if [ -z "$name" ]; then + files="${COMPOSE_FILE:-docker-compose.yml}" + local IFS="${COMPOSE_PATH_SEPARATOR:-:}" + for file in $files; do + [ -r "$file" ] || continue + while IFS= read -r line || [ -n "$line" ]; do + case "$line" in + name:*) + line="${line#name:}" + line="$(printf '%s' "$line" | sed 's/#.*//; s/^[[:space:]]*//; s/[[:space:]]*$//; s/^["'\'']//; s/["'\'']$//')" + [ -n "$line" ] && name="$line" + ;; + esac + done <"$file" + done + fi + [ -n "$name" ] || name="$(basename "$PWD")" + printf '%s' "$name" | tr '[:upper:]' '[:lower:]' | tr -cd 'a-z0-9_-' +} + +# set_install_paths -- called by both scripts once they are in their own +# directory. One lock and one restore marker per install: two installs run +# from one checkout under different project names never block each other. +# +# LOCK_DIR is the one lock backup.sh and restore.sh share, beside this file, +# so every run against this install takes the same one whatever its +# destination. mkdir, because it is atomic everywhere and macOS has no flock. +# +# RESTORE_MARKER is written by restore.sh before it changes any store and +# removed once every store is restored. It records the backup and the COMPOSE_* +# variables in effect. While it exists the stores may be half restored: +# backup.sh refuses, and restore.sh, run again, finishes the job. +# +# DC is `docker compose` as advice prints it, with those variables in front. +set_install_paths() { + local here project + here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + project="$(compose_project)" + LOCK_DIR="$here/.backup-restore-$project.lock" + RESTORE_MARKER="$here/.restore-incomplete-$project" + DC="$(compose_env_prefix)docker compose" +} +LOCK_DIR='' +RESTORE_MARKER='' +DC='docker compose' +LOCK_HELD=0 + +# acquire_lock -- creates LOCK_DIR and writes this shell's pid into it, or +# returns 1 and leaves an existing lock exactly as it is. A lock is never +# removed for being stale: two runs that both judged it stale would both go +# on to hold it. The caller refuses, naming LOCK_DIR and lock_pid. +acquire_lock() { + mkdir "$LOCK_DIR" 2>/dev/null || return 1 + LOCK_HELD=1 + echo "$$" | artefact_write "$LOCK_DIR/pid" || true +} + +# lock_pid -- the pid recorded in the lock, or `unknown`. +lock_pid() { + local pid + pid="$(cat "$LOCK_DIR/pid" 2>/dev/null)" || pid='' + printf '%s' "${pid:-unknown}" +} + +# release_lock -- removes the lock only if this run took it. The last thing +# either script's EXIT trap does, so another run can start only once this +# one has finished with the stores. +release_lock() { + [ "$LOCK_HELD" = 1 ] || return 0 + rm -rf "$LOCK_DIR" 2>/dev/null || return 1 + LOCK_HELD=0 +} + +# run_to_end -- a step that writes to a store, run as a child this +# shell waits for. A signal ends the wait and runs the EXIT trap at once, but +# not the child: an asynchronous command ignores the terminal's Ctrl-C, and a +# TERM sent to this script alone never reaches it. The trap then waits for the +# child (wait_for_step) before it reports or releases the lock, so no second +# run starts while the first is still writing. `<&0` hands the child this +# function's standard input, which a bare `&` would replace with /dev/null. +STEP_PID='' +STEP_STATUS='' +run_to_end() { + local rc=0 + "$@" <&0 & + STEP_PID=$! + wait "$STEP_PID" || rc=$? + STEP_PID='' + return "$rc" +} + +# wait_for_step -- in the EXIT trap: waits for a step a signal interrupted, and +# records how it ended in STEP_STATUS. Nothing when no step was running. +wait_for_step() { + [ -n "$STEP_PID" ] || return 0 + note "Waiting for the step that was running to finish..." + STEP_STATUS=0 + wait "$STEP_PID" 2>/dev/null || STEP_STATUS=$? + STEP_PID='' +} diff --git a/backup.sh b/backup.sh new file mode 100755 index 0000000..5d9070d --- /dev/null +++ b/backup.sh @@ -0,0 +1,398 @@ +#!/usr/bin/env bash +# Backs up a ClickMonk install -- ClickHouse, Postgres, the spool, Caddy's +# certificates and .env -- into one new directory. +# +# Links keep answering throughout. The one container stopped is the worker, +# and only while the spool and ClickHouse are copied: it is the only thing +# that moves clicks from the one into the other, so with it stopped the two +# copies describe one instant. A click in both is shipped again after a +# restore and counted once, because every count ClickMonk makes is of distinct +# click ids. Clicks the redirect accepts meanwhile wait in the spool. +# +# Every file this writes is a credential. The umask comes before anything is +# created, rather than a chmod afterwards, so no file ever exists with a +# looser mode, and every file is created by artefact_write in backup-lib.sh. +# +# bash for `set -o pipefail` -- a dump that fails half way through a pipe must +# fail the run -- and for the EXIT trap that starts the worker again. Written +# for bash 3.2, which macOS ships. +set -euo pipefail +umask 077 + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=backup-lib.sh +. "$SCRIPT_DIR/backup-lib.sh" + +# A reader that goes away turns a write into an error instead of ending the +# run half way through, with the worker stopped. +trap "" PIPE + +usage() { + cat >&2 <<'EOF' +usage: ./backup.sh + +Writes // holding clickhouse.zip, postgres.dump, +spool.tar, caddy-data.tar, env and MANIFEST, readable by you alone. + +Links keep answering while it runs. The worker is stopped while the spool and +ClickHouse are copied -- a pause that grows with the data -- and started again +afterwards; a worker that was already stopped is left stopped. ClickHouse +writes its archive beside its data first, so the disk Docker keeps its volumes +on needs about ClickHouse's size free; a backup that would not fit is refused. + +Every file it writes is a credential: the database passwords, the secret that +signs visitor cookies, the admin account's two-factor secret and every +certificate's private key. Keep the destination private, and encrypt it before +it leaves this host. + +If you pipe this anywhere, test ${PIPESTATUS[0]}, not $?. +EOF +} + +DEST="${1:-}" +case "$DEST" in + -h | --help) + usage + exit 0 + ;; + '') + usage + exit 2 + ;; +esac +# Absolute before the cd below, so a relative destination is relative to +# where it was typed. +case "$DEST" in + /*) ;; + *) DEST="$PWD/$DEST" ;; +esac +# docker-compose.yml and .env are found from here, so cron need not cd first. +cd "$SCRIPT_DIR" +set_install_paths + +STAMP="$(date -u +%Y-%m-%dT%H%M%SZ)" +OUT="$DEST/$STAMP" +# The process id too: two runs started in the same second into different +# destinations share a timestamp, and the archive name is what the EXIT trap +# deletes. +CH_FILE="clickmonk-$STAMP-$$.zip" +OUT_CREATED=0 +BACKUP_COMPLETE=0 +WORKER_STOPPED_BY_US=0 + +# fail -- every failure of this script, before or after the +# worker was stopped. The last sentence is true on every path: nothing here +# writes to either store. +fail() { + local step="$1" line + shift + note "" + note "ERROR: the backup failed during the ${step} step." + for line in "$@"; do + note " ${line}" + done + note "" + note "No data was changed: this script only reads from Postgres, ClickHouse," + note "the spool and Caddy's data." + exit 1 +} + +# worker_up -- running, or restarting under its restart policy: either way a +# worker that can ship clicks between the BACKUP and the counts. +worker_up() { + docker compose ps --status running --status restarting --services 2>/dev/null | grep -qx worker +} + +start_worker_if_we_stopped_it() { + [ "$WORKER_STOPPED_BY_US" = 1 ] || return 0 + local attempt + for attempt in 1 2 3; do + if docker compose start worker >/dev/null 2>&1 && worker_up; then + WORKER_STOPPED_BY_US=0 + return 0 + fi + [ "$attempt" = 3 ] && break + say "Could not start the worker (attempt $attempt of 3); trying again..." + sleep 2 + done + return 1 +} + +remove_incomplete_output() { + [ "$OUT_CREATED" = 1 ] || return 0 + [ "$BACKUP_COMPLETE" = 1 ] && return 0 + rm -rf "$OUT" 2>/dev/null || note "Could not remove the incomplete backup at $OUT; delete it by hand." + return 0 +} + +# THE RESTART COMES FIRST, once the signals that could end the trap are +# masked: an operator who presses Ctrl-C again while the worker is being +# started would otherwise kill the trap before the restart, leaving the +# reports frozen with nothing saying why. The mask protects this shell, not +# the restart: `docker compose` installs its own handlers for INT and TERM, so +# a second Ctrl-C can cut one attempt short. That is why the restart is tried +# three times, and why the run fails saying so if the worker is still stopped. +# Each tidy-up after the restart cannot fail, and the lock goes last, so no +# other run starts until this one is done with the stores. +cleanup() { + trap '' INT TERM HUP + local restart_ok=1 + # A signal that arrives while `docker compose stop worker` is still waiting + # for the worker to drain runs this at once, and that stop goes on without + # this script. A start now would find the worker still running and do + # nothing, and the stop would then finish: stopped, and nothing saying so. + # So the stop is finished first, and then the worker started. + if [ "$WORKER_STOPPED_BY_US" = 1 ]; then + docker compose stop worker >/dev/null 2>&1 || true + fi + start_worker_if_we_stopped_it || restart_ok=0 + remove_in_container_artefact || + note "WARNING: could not delete $CH_BACKUP_DIR/$CH_FILE inside the clickhouse container; delete it by hand." + remove_incomplete_output || true + if [ "$restart_ok" = 0 ]; then + note "" + note "ERROR: the worker is stopped and this script could not start it again." + note " Run: $DC start worker" + note "Links are still answering and their clicks are waiting in the spool," + note "but nothing reaches the reports until the worker is running." + release_lock || true + exit 1 + fi + release_lock || note "Could not remove $LOCK_DIR; delete it by hand before the next backup or restore." +} +trap cleanup EXIT + +# One backup or restore at a time against this install: a second run would +# see the worker the first one stopped as already stopped, and copy while the +# first starts it again. +acquire_lock || + fail "validation" \ + "Another backup or restore holds $LOCK_DIR (pid $(lock_pid))." \ + "If none is running, one was killed before it could clean up. Remove the lock," \ + "then run this again: rm -rf $LOCK_DIR" + +# --- Every check first, before anything is stopped --------------------- + +# A restore that was interrupted leaves the stores from two moments. A backup +# of that would be complete, checksummed and of a state that never existed. +if [ -e "$RESTORE_MARKER" ]; then + RESTORING_FROM="$(file_get "$RESTORE_MARKER" backup 2>/dev/null)" || RESTORING_FROM='' + RESTORING_WITH="$(file_get "$RESTORE_MARKER" environment 2>/dev/null)" || RESTORING_WITH='' + fail "validation" \ + "A restore of ${RESTORING_FROM:-an unknown backup} did not finish, so the stores may be half restored." \ + "Finish it first: $RESTORING_WITH$SCRIPT_DIR/restore.sh ${RESTORING_FROM:-}" \ + "If the stack is running on what that restore left, on purpose, remove $RESTORE_MARKER." +fi + +# The env file before the services: a missing file that COMPOSE_ENV_FILES +# names makes every compose command fail, which would read as a stopped store. +ENV_FILE="$(env_file_path)" || + fail "validation" \ + "COMPOSE_ENV_FILES names more than one file, and a backup copies exactly one." \ + "Name the one file that holds this install's passwords." +[ -r "$ENV_FILE" ] || + fail "validation" \ + "Cannot read $ENV_FILE, which holds this install's passwords and is part of every backup." + +# Only the two stores have to be running. The spool and Caddy's data are read +# through a running container when there is one and a one-off container +# otherwise, so a crash-looping Caddy -- the likeliest broken service -- does +# not make the backup impossible at the moment it is most wanted. +for service in postgres clickhouse; do + service_running "$service" || + fail "validation" \ + "The '$service' service is not running." \ + "Start it first: $DC up -d $service" +done + +have_sha256 || + fail "validation" \ + "No SHA-256 tool found (looked for sha256sum, shasum and openssl)." \ + "The manifest records one per file so that a damaged copy is refused." + +# The release is informational: restore.sh checks the schema version, which +# comes from the ledger below. An image from before `clickmonk version` existed +# prints its usage instead -- every install upgrading to the first release +# runs one -- and its backup must still be taken, before the upgrade. +# Asked of the redirect, or of the worker when the redirect is down, before +# the worker is stopped: either runs the image that wrote the data. +VERSION_LINE='' +VERSION_ASKED=0 +for service in redirect worker; do + if service_running "$service"; then + VERSION_ASKED=1 + VERSION_LINE="$(version_via_exec "$service" 2>/dev/null)" || VERSION_LINE='' + break + fi +done +RELEASE="$(printf '%s\n' "$VERSION_LINE" | release_of)" +if [ -z "$RELEASE" ]; then + RELEASE=unknown + if [ "$VERSION_ASKED" = 1 ]; then + note "The running image predates 'clickmonk version'; the manifest records the release as unknown." + else + note "Neither the redirect nor the worker is running; the manifest records the release as unknown." + fi +fi + +SCHEMA="$(ledger_version)" || SCHEMA='' +case "$SCHEMA" in + '' | *[!0-9]*) + fail "validation" "Could not read the schema version from Postgres: '${SCHEMA:-nothing}'." + ;; +esac + +# A BACKUP or RESTORE the server is still running outlives the client that +# started it: a backup or restore killed part way, whose lock is gone. Its +# archive is what the sweep below would delete, and a RESTORE still running +# means ClickHouse holds half a database. This run has started neither yet. +IN_PROGRESS="$(ch_query "SELECT concat(status, ' ', name) FROM system.backups WHERE status IN ('CREATING_BACKUP', 'RESTORING') LIMIT 1 FORMAT TSVRaw")" || + fail "validation" "Could not ask ClickHouse whether a backup or restore is still running." +[ -z "$IN_PROGRESS" ] || + fail "validation" \ + "ClickHouse is still running an earlier backup or restore: $IN_PROGRESS." \ + "Its client has gone, but the server finishes it. Wait for it to finish, then run this again." + +remove_orphan_archives || + note "WARNING: could not clear old archives from $CH_BACKUP_DIR inside the clickhouse container." + +# ClickHouse writes its archive onto the disk its data is on, and keeps taking +# inserts beside it once the worker is back. A disk that fills stops inserts +# and merges, so a backup that would not fit is refused before anything stops. +CH_BYTES="$(ch_query "SELECT sum(bytes_on_disk) FROM system.parts WHERE active AND database = '$CH_DATABASE'")" || CH_BYTES='' +CH_FREE="$(ch_query "SELECT free_space FROM system.disks WHERE name = 'backups'")" || CH_FREE='' +# No row at all: ClickHouse was started without clickhouse/backup-disk.xml, +# which is every install from before this script existed until ClickHouse is +# recreated with the new configuration. Nothing else here would say why. +[ -n "$CH_FREE" ] || + fail "validation" \ + "ClickHouse has no disk named 'backups': it was started before this checkout's" \ + "clickhouse/backup-disk.xml existed. Recreate it with the new configuration" \ + "(links keep answering; the worker waits for it): $DC up -d clickhouse" +# Each on its own: an empty one inside a concatenation would still read as a +# number, and as zero bytes needed below. +for value in "$CH_BYTES" "$CH_FREE"; do + case "$value" in + '' | *[!0-9]*) + fail "validation" "Could not read ClickHouse's size and free space: '${CH_BYTES:-nothing}', '${CH_FREE:-nothing}'." + ;; + esac +done +# bash's arithmetic is 64-bit, so byte counts of any real disk fit. +CH_NEED=$((CH_BYTES + CH_BYTES / 10)) +[ "$CH_FREE" -ge "$CH_NEED" ] || + fail "validation" \ + "The ClickHouse volume has $((CH_FREE / 1048576)) MiB free and a backup needs about $((CH_NEED / 1048576)) MiB." \ + "ClickHouse writes its archive beside its data before this script copies it out." \ + "Free space on the disk Docker keeps its volumes on, then run this again." + +SECRET_FP="$(secret_fingerprint "$(env_value "$ENV_FILE" CLICKMONK_SECRET)")" +ADMIN_HOST="$(env_value "$ENV_FILE" CLICKMONK_ADMIN_HOST)" + +mkdir -p "$DEST" 2>/dev/null || + fail "destination" "Could not create $DEST." "Check that the path is a directory you can write to." +# Not -p: this mkdir fails if the directory exists, so of two runs started in +# the same second exactly one owns it, and the other cannot delete it on its +# way out. +mkdir "$OUT" 2>/dev/null || + fail "destination" \ + "Could not create $OUT." \ + "Another backup started in the same second, or the destination is not writable." +OUT_CREATED=1 + +# --- The spool and ClickHouse, at one instant ---------------------------- + +# Restarting counts as up: Docker's restart policy could start it between the +# BACKUP and the counts, and the manifest would then count clicks the archive +# does not hold. +if worker_up; then + say "Stopping the worker while the spool and ClickHouse are copied..." + # Set before the stop, not after: `stop` can stop the container and still + # exit non-zero (Ctrl-C during its drain), and the trap must then know to + # start it again. + WORKER_STOPPED_BY_US=1 + docker compose stop worker >/dev/null 2>&1 || + fail "quiesce" "docker compose stop worker exited non-zero." "It is being started again either way." + wait_until_stopped worker 60 || + fail "quiesce" "The worker did not stop within 60 seconds." +else + say "The worker is not running; it is left that way." +fi + +say "Copying the spool..." +read_dir_of redirect "$SPOOL_DIR" | artefact_write "$OUT/spool.tar" || + fail "spool" "Could not copy the spool out of the redirect's volume." + +say "Backing up ClickHouse..." +# Before the BACKUP, not after: a client that is killed does not stop a BACKUP, +# which the server finishes and leaves behind. The trap deletes the name +# (and its lock file) whether or not the archive has appeared yet; a BACKUP +# still running then writes into an unlinked file whose space is freed when +# it ends. Removing a name that never appeared is harmless. +CH_ARTEFACT_CREATED=1 +ch_query "BACKUP DATABASE $CH_DATABASE TO Disk('backups', '$CH_FILE')" >/dev/null || + fail "ClickHouse" "BACKUP DATABASE failed or was interrupted; any error is above." +CH_COUNTS="$(ch_counts)" || + fail "ClickHouse" "Counting the rows the manifest records failed or was interrupted; any error is above." + +# The spool and ClickHouse are copied: the worker may ship again. A failure +# here is retried by the trap, which also makes the run fail if it cannot. +start_worker_if_we_stopped_it || note "WARNING: the worker did not start yet; trying again at the end." + +docker compose exec -T clickhouse cat "$CH_BACKUP_DIR/$CH_FILE" /dev/null || note "WARNING: sync failed; the files may not all be on disk yet." +BACKUP_COMPLETE=1 + +start_worker_if_we_stopped_it || true +say "Backup written to $OUT" diff --git a/clickhouse/backup-disk.xml b/clickhouse/backup-disk.xml new file mode 100644 index 0000000..d226ab9 --- /dev/null +++ b/clickhouse/backup-disk.xml @@ -0,0 +1,25 @@ + + + + + + local + /var/lib/clickhouse/backups/ + + + + + backups + + diff --git a/docker-compose.yml b/docker-compose.yml index 588dbaf..df72f2b 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -158,7 +158,11 @@ services: CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1 ulimits: nofile: { soft: 262144, hard: 262144 } - volumes: ["clickhouse-data:/var/lib/clickhouse"] + volumes: + - clickhouse-data:/var/lib/clickhouse + # The disk backup.sh writes ClickHouse's own backup archive to. Without + # it, BACKUP refuses to run at all. + - ./clickhouse/backup-disk.xml:/etc/clickhouse-server/config.d/backup-disk.xml:ro healthcheck: test: ["CMD-SHELL", "wget -qO- \"http://$$(hostname):8123/ping\""] interval: 5s diff --git a/install.sh b/install.sh index 0a082aa..1c910c3 100755 --- a/install.sh +++ b/install.sh @@ -48,7 +48,7 @@ if [ "$START" = 1 ]; then exit 1 fi if ! docker compose version >/dev/null 2>&1; then - echo "This Docker has no 'compose' command. Install Docker Compose v2." >&2 + echo "This Docker has no 'compose' command. Install Docker Compose 2.23 or later." >&2 exit 1 fi # Fail before writing anything if the ports Caddy needs are taken. Skipped @@ -160,8 +160,8 @@ Then add a link: docker compose exec -T worker node packages/cli/dist/index.js \ link add links.example.com spring --target https://example.com/offer -There is no web interface yet, but there is an admin API. Create its one account -first -- nothing can authenticate until it exists. The password is piped in and +The admin account comes next: the web interface and the admin API both sign in +with it, and nothing can sign in until it exists. The password is piped in and never typed as an argument. Keep the -T: without it, Compose asks for a terminal inside the container and then refuses to attach the pipe to one, and the command never runs. @@ -170,10 +170,12 @@ never runs. node packages/cli/dist/index.js admin create you@example.com Then set CLICKMONK_ADMIN_HOST in .env to a host name that is not one of your link -domains, point that name at this server, and run "docker compose up -d". Until -you do, the admin service answers 503 to everything but its own healthcheck, and -the CLI is the only way in. The admin API needs a real name over HTTPS; unlike a -link domain, there is no way to try it over plain HTTP on this host. - -README.md, under "The admin API", has the rest. +domains, point that name at this server, and run "docker compose up -d". Visit +https://that-name in a browser to sign in; the API answers on the same name. +Until you set it, the admin service answers 503 to everything but its own +healthcheck, and the CLI is the only way in. Both need a real name over HTTPS: +there is no way to use them over plain HTTP on this host. + +Back the install up with ./backup.sh . README.md has the rest, under +"The web interface", "The admin API" and "Backup and restore". NEXT diff --git a/package.json b/package.json index 0a12940..0451547 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,7 @@ { "name": "clickmonk", "private": true, + "version": "0.0.0", "type": "module", "engines": { "node": ">=22 <23" }, "packageManager": "pnpm@9.15.9", diff --git a/packages/cli/src/commands.test.ts b/packages/cli/src/commands.test.ts index 7b3af6b..8073075 100644 --- a/packages/cli/src/commands.test.ts +++ b/packages/cli/src/commands.test.ts @@ -8,12 +8,15 @@ import { ADMIN_SCRYPT, LINK_SCRYPT, MAX_PASSWORD_LENGTH, + SCHEMA_VERSION, SCRYPT_PREFIX, + VERSION, hashPassword, hashToken, parseApiKey, verifyPassword, } from '@clickmonk/core' +import type { Pool } from '@clickmonk/db' import { resetDatabases, testCh, testPg } from '@clickmonk/db/testing' import { type Fetcher, SOURCES, SOURCE_IDS } from '@clickmonk/ipdata' import type { DomainResolver } from '@clickmonk/worker/domains' @@ -37,6 +40,42 @@ afterAll(async () => { }) describe('clickmonk cli', () => { + // The backup scripts run this in a one-off container to learn which schema + // an image understands, before any database is known to be reachable, so + // it must answer without one: a Postgres pool that throws on any use, and a + // ClickHouse factory that throws if called, prove it never asked. + it('prints the release and the schema version, and touches no database', async () => { + const out: string[] = [] + const untouchable = new Proxy( + {}, + { + get() { + throw new Error('version must not touch Postgres') + }, + }, + ) as unknown as Pool + const code = await runCli(['version'], { + pg: untouchable, + ch: () => { + throw new Error('version must not touch ClickHouse') + }, + out: (s) => out.push(s), + ipdata: { dir: ipdataDir }, + }) + expect(code).toBe(0) + // The shape is what the scripts parse; the numbers are the constants the + // command exists to print, so reading them from core here is the claim + // itself rather than a comparison of a value with its own source. + expect(out).toEqual([`clickmonk ${VERSION} (schema version ${SCHEMA_VERSION})`]) + expect(out[0]).toMatch(/^clickmonk \d+\.\d+\.\d+ \(schema version \d+\)$/) + }) + + it('takes no arguments after version', async () => { + lines.length = 0 + expect(await run('version', 'now')).toBe(1) + expect(lines.join('\n')).toContain('clickmonk version') + }) + it('migrate is idempotent', async () => { expect(await run('migrate')).toBe(0) expect(await run('migrate')).toBe(0) diff --git a/packages/cli/src/commands.ts b/packages/cli/src/commands.ts index d093dae..dda4643 100644 --- a/packages/cli/src/commands.ts +++ b/packages/cli/src/commands.ts @@ -19,7 +19,9 @@ import { MAX_PASSWORD_LENGTH, MIN_ADMIN_PASSWORD_LENGTH, NON_HUMAN_CLASSES, + SCHEMA_VERSION, type TrafficSettings, + VERSION, isDomainUrl, normaliseHost, parseLinkInput, @@ -108,7 +110,8 @@ const USAGE = `usage: clickmonk admin totp disable # the way back in when the authenticator is gone clickmonk apikey create [--expires-days ] clickmonk apikey list - clickmonk apikey revoke ` + clickmonk apikey revoke + clickmonk version # the release, and the schema version it understands` class Rejected extends Error {} @@ -799,6 +802,12 @@ function ipdataStatus(d: CliDeps): void { export async function runCli(argv: string[], d: CliDeps): Promise { const [cmd, sub, ...rest] = argv try { + // First, and before anything that could open a connection: the backup + // scripts ask an image this in a container with no database it can reach. + if (cmd === 'version' && sub === undefined) { + d.out(`clickmonk ${VERSION} (schema version ${SCHEMA_VERSION})`) + return 0 + } if (cmd === 'migrate' && sub === undefined) { const { applied } = await migrateToLatest(d.pg, d.ch()) d.out(applied.length ? `applied ${applied.join(', ')}` : 'up to date') diff --git a/packages/cli/src/index.test.ts b/packages/cli/src/index.test.ts new file mode 100644 index 0000000..309b01f --- /dev/null +++ b/packages/cli/src/index.test.ts @@ -0,0 +1,40 @@ +import { spawnSync } from 'node:child_process' +import { join } from 'node:path' +import { SCHEMA_VERSION, VERSION } from '@clickmonk/core' +import { describe, expect, it } from 'vitest' + +// Runs the built entry point, not commands.ts directly: the guard this pins +// is in index.ts, before runCli is ever called, so importing commands.ts +// would not exercise it. +const DIST = join(import.meta.dirname, '..', 'dist', 'index.js') + +function runVersion(env: NodeJS.ProcessEnv): { + status: number | null + stdout: string + stderr: string +} { + const r = spawnSync('node', [DIST, 'version'], { encoding: 'utf8', env }) + return { status: r.status, stdout: r.stdout, stderr: r.stderr } +} + +describe('the entry point', () => { + // A backup script runs `version` in a one-off container to learn an + // image's schema version before any database in the install is known to + // be reachable — the variable every other command requires up front must + // not be asked for here. + it('answers version with no CLICKMONK_POSTGRES_URL set', () => { + const r = runVersion({ PATH: process.env.PATH }) + expect(r.stderr).toBe('') + expect(r.stdout).toBe(`clickmonk ${VERSION} (schema version ${SCHEMA_VERSION})\n`) + expect(r.status).toBe(0) + }) + + // The same container's own CLICKMONK_DNS_SERVERS can be malformed; that + // must not stop `version` from answering either. + it('answers version even when CLICKMONK_DNS_SERVERS is malformed', () => { + const r = runVersion({ PATH: process.env.PATH, CLICKMONK_DNS_SERVERS: 'not-a-resolver' }) + expect(r.stderr).toBe('') + expect(r.stdout).toBe(`clickmonk ${VERSION} (schema version ${SCHEMA_VERSION})\n`) + expect(r.status).toBe(0) + }) +}) diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts index 18fcb63..dcc8c90 100644 --- a/packages/cli/src/index.ts +++ b/packages/cli/src/index.ts @@ -1,6 +1,6 @@ #!/usr/bin/env node import { formatConfigError, normaliseHost } from '@clickmonk/core' -import { type ClickHouseClient, createChClient, createPgPool } from '@clickmonk/db' +import { type ClickHouseClient, type Pool, createChClient, createPgPool } from '@clickmonk/db' import { DEFAULT_IPDATA_DIR } from '@clickmonk/ipdata' import { isResolverAddress } from '@clickmonk/worker/domains' import { z } from 'zod' @@ -21,6 +21,32 @@ const ChEnv = z.object({ class ConfigError extends Error {} async function main(): Promise { + const argv = process.argv.slice(2) + // Answered before either environment is parsed: a backup script runs this + // in a one-off container to learn an image's schema version before any + // database is known to be reachable, and this checkout's own + // CLICKMONK_DNS_SERVERS may be malformed there too. Neither may stop it + // from answering, so `commands.ts`'s `version` branch — the only one that + // touches neither dependency — runs against stand-ins that throw if it + // ever did. + if (argv.length === 1 && argv[0] === 'version') { + const untouchable = new Proxy( + {}, + { + get() { + throw new Error('version must not touch Postgres') + }, + }, + ) as unknown as Pool + return runCli(argv, { + pg: untouchable, + ch: () => { + throw new Error('version must not touch ClickHouse') + }, + out: (s) => console.log(s), + ipdata: { dir: process.env.CLICKMONK_IPDATA_DIR || DEFAULT_IPDATA_DIR }, + }) + } const env = PgEnv.safeParse(process.env) if (!env.success) throw new ConfigError(formatConfigError(env.error)) const pg = createPgPool(env.data.CLICKMONK_POSTGRES_URL, { max: 1 }) @@ -49,7 +75,7 @@ async function main(): Promise { return opened.ch } try { - return await runCli(process.argv.slice(2), { + return await runCli(argv, { pg, ch, out: (s) => console.log(s), diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 07e731e..9bfdd32 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -24,3 +24,4 @@ export * from './tokens.js' export * from './totp.js' export * from './traffic.js' export * from './user-agent.js' +export * from './version.js' diff --git a/packages/core/src/version.test.ts b/packages/core/src/version.test.ts index 8dac57d..7ac43ba 100644 --- a/packages/core/src/version.test.ts +++ b/packages/core/src/version.test.ts @@ -1,5 +1,8 @@ +import { readFileSync, readdirSync } from 'node:fs' +import { join } from 'node:path' import { describe, expect, it } from 'vitest' import { SCHEMA_VERSION } from './index.js' +import { VERSION } from './version.js' describe('SCHEMA_VERSION', () => { it('is the highest migration version this build knows about', () => { @@ -8,3 +11,36 @@ describe('SCHEMA_VERSION', () => { expect(SCHEMA_VERSION).toBe(10) }) }) + +// packages/core/src -> the repository root. +const ROOT = join(import.meta.dirname, '..', '..', '..') + +/** Every manifest a release bumps: the root's, and one per package. */ +function manifests(): { file: string; version: unknown }[] { + const files = [ + 'package.json', + ...readdirSync(join(ROOT, 'packages')).map((p) => join('packages', p, 'package.json')), + ] + return files.map((file) => ({ + file, + version: (JSON.parse(readFileSync(join(ROOT, file), 'utf8')) as { version?: unknown }).version, + })) +} + +describe('VERSION', () => { + it('is a plain release number, with no prefix and no pre-release tag', () => { + expect(VERSION).toMatch(/^\d+\.\d+\.\d+$/) + }) + + // A release bumps every manifest and this constant together. A bump that + // misses one leaves a package claiming a version nobody released, and the + // constant is what `clickmonk version` prints, so it has to be the same + // number everywhere or the answer an operator gives in a bug report is wrong. + it('is the version every package.json in the repository carries, the root included', () => { + const found = manifests() + // Eight packages and the root. Written out, so a ninth package that is + // never given a version fails here rather than being skipped. + expect(found).toHaveLength(9) + for (const { file, version } of found) expect(version, file).toBe(VERSION) + }) +}) diff --git a/packages/core/src/version.ts b/packages/core/src/version.ts new file mode 100644 index 0000000..4aab571 --- /dev/null +++ b/packages/core/src/version.ts @@ -0,0 +1,6 @@ +/** + * The release this build is. Every package.json in the repository carries the + * same number, and version.test.ts holds them to it: a release bumps all of + * them and this together. `clickmonk version` prints it. + */ +export const VERSION = '0.0.0' diff --git a/restore.sh b/restore.sh new file mode 100755 index 0000000..27d6b07 --- /dev/null +++ b/restore.sh @@ -0,0 +1,579 @@ +#!/usr/bin/env bash +# Puts a ClickMonk install back to a backup taken by ./backup.sh. +# +# The only script in this repository that destroys data, and it destroys all +# of it: the ClickHouse database, the Postgres schema, the spool and Caddy's +# certificates are each emptied and refilled from the backup. Everything +# recorded since the backup was taken is gone. Every check runs before any of +# that, and a refusal leaves the install as it was -- not even stopped. +# +# Caddy, the redirect, the admin service and the worker are stopped for the +# whole restore, so links answer nothing until it finishes. The redirect +# cannot keep serving: it writes the spool this replaces, and a spool has one +# writer. +# +# ONE STORE IS NEVER PUT BACK ON ITS OWN, and no flag does it. Postgres holds +# the migration ledger for both stores, and the spool's clicks belong to links +# in Postgres: a Postgres from one moment beside a ClickHouse from another has +# a ledger describing tables that are not there, and the worker trusts it. +# +# AN INTERRUPTED RESTORE LEAVES THOSE FOUR STOPPED, deliberately. A redirect +# started on a half-restored Postgres serves whatever links happen to be there +# and looks healthy; a stopped one is loud. Running this again with the same +# backup starts from the beginning and finishes the job. +# +# bash for pipefail and the EXIT trap; written for bash 3.2, which macOS ships. +set -euo pipefail +umask 077 + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=backup-lib.sh +. "$SCRIPT_DIR/backup-lib.sh" + +trap "" PIPE + +# How long the confirmation waits: long enough to read the warning, short +# enough that an unattended caller that will never answer is refused rather +# than hung. +CONFIRM_TIMEOUT=120 +APP_SERVICES="caddy redirect admin worker" + +usage() { + cat >&2 <<'EOF' +usage: ./restore.sh + +Replaces everything this install holds -- clicks, links, domains, settings, +the admin account, API keys and certificates -- with the backup in +, one of the timestamped directories ./backup.sh writes. + +It checks every file against its checksum and the backup against this +checkout's image, then asks you to type the backup's timestamp. Nothing is +stopped or changed before you do. + +Links answer nothing while it runs. If it is interrupted it leaves the stack +stopped and says what state each store is in; run it again with the same +backup to finish. +EOF +} + +SRC="${1:-}" +case "$SRC" in + -h | --help) + usage + exit 0 + ;; + '') + usage + exit 2 + ;; +esac +case "$SRC" in + /*) ;; + *) SRC="$PWD/$SRC" ;; +esac +cd "$SCRIPT_DIR" +set_install_paths + +STAMP='' +BACKUP_SCHEMA='' +RUNNING_BEFORE='' +APPS_STOPPED=0 +DESTRUCTION_BEGUN=0 +STORES_RESTORED=0 +# Set when every store was restored and does not match the manifest: running +# the restore again would only reproduce the mismatch, so cleanup says so. +VERIFY_FAILED=0 +# Set while this run's RESTORE statement may be running in ClickHouse. The +# server carries on with it after the client is gone, so an interrupted run +# says so only when it was interrupted there. +CH_RESTORE_SENT=0 +# Set by refuse and abort, which say why the run ended; any other early end +# gets one line from cleanup. +EXPLAINED=0 +# Set once `up` has brought the stack up, the last thing the restore does. +STACK_STARTED=0 +# What has happened to the data, in words, for the message an interrupted run +# prints. Assigned BEFORE each destructive command, never after: a signal +# lands while a command runs and bash acts on it before the next statement, so +# an assignment after the command would describe the step before. +DATA_STATE="nothing has been changed" +# What the data is once the running step has finished, for a trap that waited +# for it and saw it succeed. +DATA_STATE_AFTER='' +# The backup an earlier run was restoring when it was interrupted, read from +# its marker: this run's "nothing was changed" is then true of this run only. +PRIOR_RESTORE='' + +# prior_restore_note -- after "nothing was changed", what an earlier, +# interrupted restore left, which this run has not changed either. +prior_restore_note() { + [ -n "$PRIOR_RESTORE" ] || return 0 + note "The stores are still as the interrupted restore of $PRIOR_RESTORE left them," + note "and the stack is as it was. To finish that restore:" + note " $(compose_env_prefix)$SCRIPT_DIR/restore.sh $PRIOR_RESTORE" +} + +# refuse -- a guard declining. Only ever called before +# anything is stopped, so what it says at the end is true. +refuse() { + local line + EXPLAINED=1 + note "" + note "Refused: $1" + shift + for line in "$@"; do + note " $line" + done + note "" + note "Nothing was changed, and nothing was stopped." + prior_restore_note + exit 1 +} + +# abort -- a failure after the stack was stopped. cleanup +# says what state that left the data in. +abort() { + local step="$1" line + EXPLAINED=1 + shift + note "" + note "ERROR: the restore failed during the ${step} step." + for line in "$@"; do + note " ${line}" + done + [ "$DESTRUCTION_BEGUN" = 1 ] || prior_restore_note + exit 1 +} + +# THE SIGNALS ARE MASKED FIRST: an operator who presses Ctrl-C again while +# the services are being started would otherwise end the trap before the +# start, and before the archive is deleted and the lock released. The mask +# protects this shell, not the start: `docker compose` installs its own +# handlers for INT and TERM, so a second Ctrl-C can cut one attempt short, +# which is why start_services tries three times. +# +# A STEP A SIGNAL INTERRUPTED IS WAITED FOR FIRST: it is still writing to its +# store, and its archive or the lock gone from under it would let a second run +# start beside it. Then, before anything was changed, the services: a stop the +# signal interrupted is finished before they are started, or the start would +# find them still running and the stop would then leave them stopped. Then the +# archive, and the lock last: no backup may start, and sweep the backups disk, +# while this run's archive is still there. +# start_services -- `docker compose start`, tried three times, and +# a failure unless every one of them is running afterwards. +start_services() { + local attempt service all + for attempt in 1 2 3; do + if docker compose start "$@" >/dev/null 2>&1; then + all=1 + for service in "$@"; do + service_running "$service" || all=0 + done + [ "$all" = 1 ] && return 0 + fi + [ "$attempt" = 3 ] && break + note "Could not start them (attempt $attempt of 3); trying again..." + sleep 2 + done + return 1 +} + +cleanup() { + trap '' INT TERM HUP + local status="$1" start_failed=0 + wait_for_step + if [ "$STEP_STATUS" = 0 ] && [ -n "$DATA_STATE_AFTER" ]; then + DATA_STATE="$DATA_STATE_AFTER" + fi + # The RESTORE's client returned, so ClickHouse is no longer running it -- + # unless the client itself was killed by a signal. + if [ -n "$STEP_STATUS" ] && [ "$STEP_STATUS" -le 128 ]; then + CH_RESTORE_SENT=0 + fi + if [ "$APPS_STOPPED" = 1 ] && [ "$DESTRUCTION_BEGUN" = 0 ] && [ -n "$RUNNING_BEFORE" ]; then + note "Nothing was changed. Starting$RUNNING_BEFORE again..." + # shellcheck disable=SC2086 # a list of service names, split on purpose + docker compose stop $RUNNING_BEFORE >/dev/null 2>&1 || true + # shellcheck disable=SC2086 # a list of service names, split on purpose + start_services $RUNNING_BEFORE || start_failed=1 + fi + remove_in_container_artefact || + note "WARNING: could not delete $CH_BACKUP_DIR/$CH_FILE inside the clickhouse container; delete it by hand." + if [ "$VERIFY_FAILED" = 1 ]; then + note "" + note "Every store was restored and does not match the manifest; the differences" + note "are listed above. Caddy, the redirect, the admin service and the worker have" + note "been left stopped. Restore a different backup, or start on what was restored:" + note " $DC up -d" + note "and then remove $RESTORE_MARKER, which makes backup.sh refuse until you do." + status=1 + elif [ "$DESTRUCTION_BEGUN" = 1 ] && [ "$STORES_RESTORED" = 0 ]; then + note "" + note "The restore stopped part way: $DATA_STATE." + note "Caddy, the redirect, the admin service and the worker have been left" + note "stopped, so nothing serves your links. Run the restore again with the same" + note "backup; it starts from the beginning:" + note " $(compose_env_prefix)$SCRIPT_DIR/restore.sh $SRC" + note "Until it has finished, backup.sh refuses to run." + if [ "$CH_RESTORE_SENT" = 1 ]; then + note "ClickHouse may still be finishing the restore this run started. Until it" + note "has, running this again is refused and changes nothing; wait a minute and" + note "run it again." + fi + status=1 + elif [ "$start_failed" = 1 ]; then + note "Could not start them. Run: $DC start$RUNNING_BEFORE" + status=1 + elif [ "$STORES_RESTORED" = 1 ] && [ "$STACK_STARTED" = 0 ] && [ "$EXPLAINED" = 0 ]; then + # Interrupted while the stack was being started: the data is done. + note "" + note "Every store is restored and matches the manifest; the stack was being" + note "started when this stopped, and may be partly up. Start the rest with:" + note " $DC up -d" + elif [ "$APPS_STOPPED" = 0 ] && [ "$EXPLAINED" = 0 ]; then + # Ended before anything was stopped, and not by a refusal: a signal, most + # likely Ctrl-C at the prompt. Say so, or the prompt is the last word. + note "" + note "Stopped before anything was changed." + prior_restore_note + fi + release_lock || note "Could not remove $LOCK_DIR; delete it by hand before the next backup or restore." + exit "$status" +} +trap 'cleanup "$?"' EXIT + +# One backup or restore at a time against this install. A backup's sweep of +# the backups disk would delete the archive this restores from, and a backup +# taken while this runs would copy half-restored stores. +acquire_lock || + refuse "Another backup or restore holds $LOCK_DIR (pid $(lock_pid))." \ + "If none is running, one was killed before it could clean up. Remove the lock," \ + "then run this again: rm -rf $LOCK_DIR" + +# Read under the lock, so no other run is writing it. +if [ -e "$RESTORE_MARKER" ]; then + PRIOR_RESTORE="$(file_get "$RESTORE_MARKER" backup 2>/dev/null)" || PRIOR_RESTORE='an unknown backup' +fi + +# --- Checks. Nothing below this line and above the confirmation changes or +# --- stops anything. + +[ -d "$SRC" ] || refuse "$SRC is not a directory." +[ -f "$SRC/MANIFEST" ] || + refuse "There is no MANIFEST in $SRC." \ + "Point this at one of the timestamped directories backup.sh writes, not at" \ + "the destination you gave it." + +FORMAT="$(manifest_get "$SRC" clickmonk_backup_version)" || FORMAT='' +[ "$FORMAT" = "$BACKUP_FORMAT" ] || + refuse "This is not a backup this script can read." \ + "Its format is '${FORMAT:-missing}'; this script reads $BACKUP_FORMAT." + +STAMP="$(manifest_get "$SRC" timestamp)" || STAMP='' +[ -n "$STAMP" ] || refuse "The manifest has no timestamp." +# It becomes part of a file name and of a ClickHouse statement, so it is only +# ever the shape backup.sh writes. +case "$STAMP" in + [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]T[0-9][0-9][0-9][0-9][0-9][0-9]Z) ;; + *) refuse "The manifest's timestamp is '$STAMP', which is not one backup.sh writes." ;; +esac +BACKUP_RELEASE="$(manifest_get "$SRC" clickmonk_version)" || BACKUP_RELEASE='unknown' +BACKUP_SCHEMA="$(manifest_get "$SRC" schema_version)" || BACKUP_SCHEMA='' +case "$BACKUP_SCHEMA" in + '' | *[!0-9]*) refuse "The manifest's schema_version is '${BACKUP_SCHEMA:-missing}', which is not a number." ;; +esac +# The raw table's count is always recorded; a rollup's only when the install +# backed up had it, and the check after the restore covers what was recorded. +manifest_get "$SRC" "rows.clickhouse.$CH_REQUIRED" >/dev/null || + refuse "The manifest records no row count for $CH_REQUIRED, so the restore could not be checked." + +have_sha256 || + refuse "No SHA-256 tool found (looked for sha256sum, shasum and openssl)." \ + "This script does not restore a file it cannot check." + +say "Checking every file against the manifest..." +for artefact in $ARTEFACTS; do + [ -s "$SRC/$artefact" ] || refuse "$SRC/$artefact is missing or empty." + want="$(manifest_get "$SRC" "sha256.$artefact")" || + refuse "The manifest records no checksum for $artefact." + got="$(sha256_of "$SRC/$artefact")" || got='' + [ "$got" = "$want" ] || + refuse "$artefact does not match the checksum in the manifest." \ + "manifest: $want" \ + "on disk: ${got:-could not be computed}" \ + "This backup is damaged. Do not restore it; find another copy." +done + +# The env file before the services: a missing file that COMPOSE_ENV_FILES +# names makes every compose command fail, which would read as a stopped store. +ENV_FILE="$(env_file_path)" || + refuse "COMPOSE_ENV_FILES names more than one file." \ + "Name the one file that holds this install's passwords." +[ -r "$ENV_FILE" ] || refuse "Cannot read $ENV_FILE." + +for service in postgres clickhouse; do + service_running "$service" || + refuse "The '$service' service is not running." \ + "Both stores must be up to restore into: $DC up -d postgres clickhouse" +done + +# Asked of the `default` database, which always exists: after an interrupted +# restore, clickmonk may not. A BACKUP or RESTORE outlives the client that +# started it -- a restore killed part way, whose lock is gone -- and this +# restore's DROP would pull the database out from under it. +IN_PROGRESS="$(ch_query "SELECT concat(status, ' ', name) FROM system.backups WHERE status IN ('CREATING_BACKUP', 'RESTORING') LIMIT 1 FORMAT TSVRaw" default)" || + refuse "Could not ask ClickHouse whether a backup or restore is still running." +[ -z "$IN_PROGRESS" ] || + refuse "ClickHouse is still running an earlier backup or restore: $IN_PROGRESS." \ + "Its client has gone, but the server finishes it. Wait for it to finish, then run this again." + +# The RESTORE reads its archive from this disk and runs after the DROP, so a +# ClickHouse without it would be emptied and not refilled. That is every +# install from before clickhouse/backup-disk.xml until ClickHouse is recreated. +BACKUPS_DISK="$(ch_query "SELECT name FROM system.disks WHERE name = 'backups'" default)" || BACKUPS_DISK='' +[ "$BACKUPS_DISK" = backups ] || + refuse "ClickHouse has no disk named 'backups': it was started before this checkout's" \ + "clickhouse/backup-disk.xml existed. Recreate it with the new configuration" \ + "(links keep answering): $DC up -d clickhouse" + +# The archive is copied onto that disk, and after the DROP the RESTORE writes +# the restored database beside it: about four thirds of the archive's size, +# measured. The DROP frees what ClickHouse holds now. A disk that fills there +# leaves ClickHouse partial with the stack stopped, so it is refused here. +ZIP_BYTES="$(wc -c <"$SRC/clickhouse.zip" | tr -d ' ')" || ZIP_BYTES='' +CH_FREE="$(ch_query "SELECT free_space FROM system.disks WHERE name = 'backups'" default)" || CH_FREE='' +CH_BYTES="$(ch_query "SELECT sum(bytes_on_disk) FROM system.parts WHERE active AND database = '$CH_DATABASE'" default)" || CH_BYTES='' +# Each on its own: an empty one inside a concatenation would still read as a +# number, and as zero in the sum below. +for value in "$ZIP_BYTES" "$CH_FREE" "$CH_BYTES"; do + case "$value" in + '' | *[!0-9]*) + refuse "Could not read the archive's size and ClickHouse's size and free space: '${ZIP_BYTES:-nothing}', '${CH_BYTES:-nothing}', '${CH_FREE:-nothing}'." + ;; + esac +done +CH_NEED=$((ZIP_BYTES * 5 / 2)) +[ $((CH_FREE + CH_BYTES)) -ge "$CH_NEED" ] || + refuse "The ClickHouse volume has $((CH_FREE / 1048576)) MiB free, and $((CH_BYTES / 1048576)) MiB more once its database is dropped; this restore needs about $((CH_NEED / 1048576)) MiB." \ + "The archive is copied beside ClickHouse's data and the database is restored from it there." \ + "Free space on the disk Docker keeps its volumes on, then run this again." + +say "Checking the backup against this checkout's image..." +# The image `docker compose up` would start now, which is the one this script +# starts at the end -- not whatever container happens to be running, which +# can be older while an upgrade is half done. +VERSION_LINE="$(version_via_run)" || VERSION_LINE='' +IMAGE_SCHEMA="$(printf '%s\n' "$VERSION_LINE" | schema_of)" +[ -n "$IMAGE_SCHEMA" ] || + refuse "Could not read the schema version this checkout's image understands." \ + "It printed: '${VERSION_LINE:-nothing}'." \ + "Build it first: $DC build" +# A backup of an install from before `clickmonk version` records its release +# as unknown; its schema version is still exact, and is what the advice names. +if [ "$BACKUP_RELEASE" = unknown ]; then + NEWER_ADVICE="Check out a release whose schema version is at least $BACKUP_SCHEMA, build it, and run this again." +else + NEWER_ADVICE="Check out ClickMonk $BACKUP_RELEASE or later, build it, and run this again." +fi +[ "$BACKUP_SCHEMA" -le "$IMAGE_SCHEMA" ] || + refuse "This backup is newer than this image." \ + "The backup is schema version $BACKUP_SCHEMA, from ClickMonk $BACKUP_RELEASE; this image understands $IMAGE_SCHEMA." \ + "$NEWER_ADVICE" + +WARNINGS='' +BACKUP_FP="$(manifest_get "$SRC" secret_fingerprint)" || BACKUP_FP='' +if [ "$(secret_fingerprint "$(env_value "$ENV_FILE" CLICKMONK_SECRET)")" != "$BACKUP_FP" ]; then + WARNINGS="${WARNINGS} +WARNING: CLICKMONK_SECRET in $ENV_FILE is not the one this backup was taken with. + After the restore every returning visitor counts as new, and anyone who had + answered a link's password is asked again. To keep them: answer anything + but the timestamp, copy the CLICKMONK_SECRET line from $SRC/env into + $ENV_FILE, run $DC up -d, and restore again." +fi +BACKUP_ADMIN_HOST="$(manifest_get "$SRC" admin_host)" || BACKUP_ADMIN_HOST='' +LIVE_ADMIN_HOST="$(env_value "$ENV_FILE" CLICKMONK_ADMIN_HOST)" +if [ "$LIVE_ADMIN_HOST" != "$BACKUP_ADMIN_HOST" ]; then + WARNINGS="${WARNINGS} +WARNING: CLICKMONK_ADMIN_HOST is '$LIVE_ADMIN_HOST' here and was '$BACKUP_ADMIN_HOST' + when this backup was taken. After the restore the admin interface answers + on '$LIVE_ADMIN_HOST'." +fi + +note "" +note "About to REPLACE everything this install holds with the backup in" +note " $SRC" +note "taken at $STAMP by ClickMonk $BACKUP_RELEASE, schema version $BACKUP_SCHEMA." +note "This image understands schema version $IMAGE_SCHEMA; the worker applies any" +note "migrations in between when it starts." +if [ -n "$WARNINGS" ]; then + note "$WARNINGS" +fi +note "" +note "Caddy, the redirect, the admin service and the worker are stopped until it" +note "finishes: your links answer nothing meanwhile. Everything recorded since" +note "$STAMP is lost -- clicks, links, domains, settings and keys." +note "" +note "Type the backup’s timestamp to go ahead, or anything else to stop:" +CONFIRMATION='' +IFS= read -r -t "$CONFIRM_TIMEOUT" CONFIRMATION || CONFIRMATION='' +[ "$CONFIRMATION" = "$STAMP" ] || + refuse "That is not the backup’s timestamp." "Expected: $STAMP" + +# --- Stop the four application services ---------------------------------- + +for service in $APP_SERVICES; do + if service_running "$service"; then RUNNING_BEFORE="$RUNNING_BEFORE $service"; fi +done +say "Stopping Caddy, the redirect, the admin service and the worker..." +# Before the stop: `stop` can exit non-zero having stopped them. +APPS_STOPPED=1 +# shellcheck disable=SC2086 # a list of service names, split on purpose +docker compose stop $APP_SERVICES >/dev/null 2>&1 || + abort "stop" "docker compose stop exited non-zero. Nothing has been changed yet." +for service in $APP_SERVICES; do + wait_until_stopped "$service" 60 || + abort "stop" "$service did not stop within 60 seconds. Nothing has been changed yet." +done + +# --- ClickHouse ---------------------------------------------------------- + +# Archives a killed run left behind. Under the lock, and after the check above +# that ClickHouse is running no backup or restore, so none is in use. +remove_orphan_archives || + note "WARNING: could not clear old archives from $CH_BACKUP_DIR inside the clickhouse container." + +say "Restoring ClickHouse..." +# The process id too, so that a run started while an earlier run's RESTORE is +# still reading its archive never writes over it. +CH_FILE="clickmonk-restore-$STAMP-$$.zip" +# Before the copy, so a copy that dies half way is still deleted. +CH_ARTEFACT_CREATED=1 +# As root inside the container, handing the directory to the server's own +# user: on an install that has never taken a backup it may not exist yet, and +# a directory root created would refuse the server's next BACKUP. +docker compose exec -T clickhouse sh -c \ + 'mkdir -p "$0" && chown clickhouse:clickhouse "$0" && cat >"$0/$1"' \ + "$CH_BACKUP_DIR" "$CH_FILE" <"$SRC/clickhouse.zip" || + abort "ClickHouse" "Could not copy the archive into the clickhouse container. Nothing has been changed yet." + +# Before the first change to any store: until every store is restored, the +# marker makes backup.sh refuse rather than copy a mix of two moments. +# Written beside it and renamed, so that a failed write never touches a marker +# an earlier, interrupted run left: that one still describes the stores. +if ! printf 'backup=%s\nenvironment=%s\n' "$SRC" "$(compose_env_prefix)" | artefact_write "$RESTORE_MARKER.partial" || + ! mv -f "$RESTORE_MARKER.partial" "$RESTORE_MARKER"; then + rm -f "$RESTORE_MARKER.partial" 2>/dev/null || true + abort "ClickHouse" "Could not write $RESTORE_MARKER. Nothing has been changed yet." +fi +DESTRUCTION_BEGUN=1 +DATA_STATE="ClickHouse was being dropped and may be gone; Postgres, the spool and the certificates are as they were" +DATA_STATE_AFTER="ClickHouse was dropped and is empty; Postgres, the spool and the certificates are as they were" +run_to_end ch_query "DROP DATABASE IF EXISTS $CH_DATABASE SYNC" default >/dev/null || + abort "ClickHouse" "DROP DATABASE failed; the ClickHouse error is above." +DATA_STATE="ClickHouse was being restored and may be missing or partial; Postgres, the spool and the certificates are as they were" +DATA_STATE_AFTER="ClickHouse is restored; Postgres, the spool and the certificates are as they were" +CH_RESTORE_SENT=1 +if ! run_to_end ch_query "RESTORE DATABASE $CH_DATABASE FROM Disk('backups', '$CH_FILE')" default >/dev/null; then + # The server answered with an error: it is not running this RESTORE. + CH_RESTORE_SENT=0 + abort "ClickHouse" "RESTORE DATABASE failed; the ClickHouse error is above." +fi +CH_RESTORE_SENT=0 +remove_in_container_artefact || + note "WARNING: could not delete $CH_BACKUP_DIR/$CH_FILE inside the clickhouse container; delete it by hand." + +# --- Postgres ------------------------------------------------------------ + +say "Restoring Postgres..." +DATA_STATE="ClickHouse is restored; Postgres was being emptied and may be empty; the spool and the certificates are as they were" +DATA_STATE_AFTER="ClickHouse is restored; Postgres is empty; the spool and the certificates are as they were" +# shellcheck disable=SC2016 # expanded by the container's shell, not this one +run_to_end docker compose exec -T postgres sh -c \ + 'PGPASSWORD="$POSTGRES_PASSWORD" psql -U "$0" -d "$1" -v ON_ERROR_STOP=1 -q -c "SET client_min_messages = warning; DROP SCHEMA IF EXISTS public CASCADE; CREATE SCHEMA public"' \ + "$PG_USER" "$PG_DATABASE" /dev/null || + abort "Postgres" "Could not empty the public schema; the Postgres error is above." +DATA_STATE="ClickHouse is restored; Postgres was being refilled and is either empty or complete; the spool and the certificates are as they were" +DATA_STATE_AFTER="ClickHouse and Postgres are restored; the spool and the certificates are as they were" +# One transaction: the dump is applied whole or not at all. +# shellcheck disable=SC2016 # expanded by the container's shell, not this one +run_to_end docker compose exec -T postgres sh -c \ + 'PGPASSWORD="$POSTGRES_PASSWORD" pg_restore -U "$0" -d "$1" --clean --if-exists --no-owner --single-transaction' \ + "$PG_USER" "$PG_DATABASE" <"$SRC/postgres.dump" || + abort "Postgres" "pg_restore failed; its error is above. Nothing of the dump was applied." + +# --- The spool, the redirect's state, Caddy's data ----------------------- + +say "Restoring the spool..." +DATA_STATE="ClickHouse and Postgres are restored; the spool was being replaced and may be partial; the certificates are as they were" +DATA_STATE_AFTER="ClickHouse, Postgres and the spool are restored; the certificates are as they were" +# shellcheck disable=SC2016 # expanded by the container's shell, not this one +run_to_end docker compose --progress quiet run --rm --no-deps -T worker sh -c \ + 'find "$0" -mindepth 1 -delete && tar -xf - -C "$0"' "$SPOOL_DIR" <"$SRC/spool.tar" || + abort "spool" "Could not replace the spool." +# The redirect's copy of its configuration describes the install before the +# restore. It reads Postgres at start and writes this again; without the file, +# a stale configuration can never be served from it. +# shellcheck disable=SC2016 # expanded by the container's shell, not this one +run_to_end docker compose --progress quiet run --rm --no-deps -T redirect sh -c \ + 'find "$0" -mindepth 1 -delete' "$STATE_DIR" `backup=${dir}\nenvironment=${ADVICE_ENV}\n` +const SCRIPT_ENV: Record = { + ...ENV, + COMPOSE_FILE: 'docker-compose.yml:test/stack/docker-compose.tls.yml', + COMPOSE_ENV_FILES: ENV_FILE, +} + +let root = '' +let cookie = '' +let setUp = false +let failures = 0 +/** The first backup the backup tests take. The refusal tests restore copies of it. */ +let firstBackup = '' + +interface ScriptResult { + status: number + out: string +} + +function runScript( + script: 'backup.sh' | 'restore.sh', + args: string[], + o: { input?: string; env?: Record } = {}, +): ScriptResult { + const r = spawnSync(join(ROOT, script), args, { + cwd: ROOT, + env: { ...SCRIPT_ENV, ...o.env }, + input: o.input ?? '', + encoding: 'utf8', + stdio: 'pipe', + timeout: 600_000, + }) + if (r.error) throw new Error(`could not run ${script}: ${r.error.message}`) + // A run ended by a signal has no status; a shell reports it as 128 + its number. + const status = r.status ?? (r.signal ? 128 + constants.signals[r.signal] : -1) + return { status, out: `${r.stdout ?? ''}${r.stderr ?? ''}` } +} + +/** The services running now, sorted. */ +const running = (): string[] => + compose('ps', '--status', 'running', '--services') + .split('\n') + .map((s) => s.trim()) + .filter(Boolean) + .sort() + +/** When a service's container last started: a restart changes it, and nothing else does. */ +function startedAt(service: string): string { + const id = compose('ps', '-aq', service).trim() + if (id === '') throw new Error(`no container for ${service}`) + return execFileSync('docker', ['inspect', '--format', '{{.State.StartedAt}}', id], { + encoding: 'utf8', + }).trim() +} + +const pg = (sql: string): string => + compose( + 'exec', + '-T', + 'postgres', + 'psql', + '-U', + 'clickmonk', + '-d', + 'clickmonk', + '-At', + '-c', + sql, + ).trim() + +/** + * One ClickHouse statement, through the container's own credentials: the + * password is read inside the container, so it is in no argument list here. + */ +const ch = (sql: string): string => + compose( + 'exec', + '-T', + 'clickhouse', + 'sh', + '-c', + 'clickhouse-client --user "$CLICKHOUSE_USER" --password "$CLICKHOUSE_PASSWORD" --database clickmonk --query "$0"', + sql, + ).trim() + +/** A number a store answered, never a silent zero from an empty answer. */ +function num(out: string): number { + const n = Number(out) + if (out === '' || !Number.isInteger(n)) throw new Error(`the store answered "${out}"`) + return n +} + +/** The sealed segments in the spool now. */ +const segments = (): string[] => + compose('exec', '-T', 'redirect', 'ls', '-1', SPOOL) + .split('\n') + .map((s) => s.trim()) + .filter((s) => SEGMENT.test(s)) + +/** Lines across every sealed segment: one line is one click. */ +const spoolLines = (): number => + num( + compose( + 'exec', + '-T', + 'redirect', + 'sh', + '-c', + 'cat "$0"/seg-*.ndjson 2>/dev/null | wc -l', + SPOOL, + ).trim(), + ) + +/** A backup's MANIFEST as a map, split on the first `=` of each line as restore.sh splits it. */ +function manifest(dir: string): Record { + const out: Record = {} + for (const line of readFileSync(join(dir, 'MANIFEST'), 'utf8').split('\n')) { + if (line === '') continue + const eq = line.indexOf('=') + out[line.slice(0, eq)] = line.slice(eq + 1) + } + return out +} + +const sha256 = (path: string): string => + createHash('sha256').update(readFileSync(path)).digest('hex') + +/** The one directory a backup run wrote under `dest`. */ +function onlyBackupIn(dest: string): string { + const entries = readdirSync(dest) + expect(entries, `${dest} should hold exactly one backup`).toHaveLength(1) + return join(dest, entries[0] as string) +} + +/** The schema version the stack's image understands, as the CLI prints it. */ +function imageSchema(): number { + const line = cli('version').trim() + const m = /^clickmonk \d+\.\d+\.\d+ \(schema version (\d+)\)$/.exec(line) + if (!m) throw new Error(`clickmonk version printed "${line}"`) + return Number(m[1]) +} + +function api( + method: string, + path: string, + o: { body?: string; cookie?: string; origin?: string; cacert?: string } = {}, +): CurlResult { + const args = ['--cacert', o.cacert ?? root, '--max-time', '60', '-X', method, '-D', '-'] + if (o.body !== undefined) { + args.push('-H', 'content-type: application/json', '--data-binary', o.body) + } + if (o.cookie) args.push('-H', `cookie: ${o.cookie}`) + if (o.origin !== undefined) args.push('-H', `origin: ${o.origin}`) + args.push(`https://${ADMIN_HOST}${path}`) + return curl(args, CA_MOUNT, { body: true }) +} + +const cookieFrom = (headers: string): string => { + const line = headers.split('\n').find((l) => /^set-cookie:/i.test(l)) ?? '' + return (line.slice(line.indexOf(':') + 1).split(';')[0] ?? '').trim() +} + +function signIn(cacert?: string): CurlResult { + return api('POST', '/api/session', { + body: JSON.stringify({ email: EMAIL, password: PASSWORD }), + origin: `https://${ADMIN_HOST}`, + ...(cacert === undefined ? {} : { cacert }), + }) +} + +/** A day either side of now, on the hour, so the summary counts every click this suite made. */ +function window(): string { + const hour = Math.floor(Date.now() / HOUR_MS) * HOUR_MS + const from = new Date(hour - 24 * HOUR_MS).toISOString() + const to = new Date(hour + 24 * HOUR_MS).toISOString() + return `from=${encodeURIComponent(from)}&to=${encodeURIComponent(to)}` +} + +/** Clicks the summary report counts; -1 while the admin service cannot answer. */ +function reportedClicks(o: { cookie: string; cacert?: string }): number { + const r = api('GET', `/api/reports/summary?${window()}`, o) + if (r.status !== 200) return -1 + return (JSON.parse(r.body) as { clicks: number }).clicks +} + +function click(): void { + const r = curl(['-4', '--max-time', '30', `http://${LINK_HOST}/${SLUG}`]) + if (r.status !== 302) throw new Error(`the link answered ${r.status}`) +} + +afterEach((ctx) => { + if (ctx.task.result?.state === 'fail') failures++ +}) + +beforeAll(async () => { + compose('down', '-v') + rmSync(BACKUPS, { recursive: true, force: true }) + // A lock or marker a failed earlier run of this suite left would refuse every backup. + rmSync(LOCK, { recursive: true, force: true }) + rmSync(MARKER, { force: true }) + rmSync(`${MARKER}.partial`, { recursive: true, force: true }) + mkdirSync(BACKUPS, { recursive: true }) + // The stack's own values, in the file the scripts copy as this install's + // .env. Compose reads it too, through COMPOSE_ENV_FILES, and agrees with + // the environment every other call here passes. + writeFileSync( + ENV_FILE, + ( + [ + 'POSTGRES_PASSWORD', + 'CLICKHOUSE_PASSWORD', + 'CLICKMONK_SECRET', + 'CLICKMONK_ADMIN_HOST', + ] as const + ) + .map((k) => `${k}=${ENV[k]}\n`) + .join(''), + { mode: 0o600 }, + ) + writeAcmeRoot() + publishZone() + compose('up', '-d', '--build', '--wait', ...WAIT_TIMEOUT) + root = writeIssuingRoot() + cliWithInput(PASSWORD, 'admin', 'create', EMAIL) + cli('domain', 'add', LINK_HOST, '--verified') + cli('link', 'add', LINK_HOST, SLUG, '--target', TARGET) + await until('a certificate for the admin host', 120_000, () => api('GET', '/api/me').exit === 0) + const signedIn = signIn() + if (signedIn.status !== 200) + throw new Error(`sign-in failed: ${signedIn.status} ${signedIn.body}`) + cookie = cookieFrom(signedIn.headers) + // Two clicks shipped before any backup, so the counts a manifest records are + // not all zero, where a count that was never taken would look the same. + click() + click() + await until( + 'the first clicks to reach the reports', + 120_000, + () => reportedClicks({ cookie }) === 2, + ) + setUp = true +}, 900_000) + +afterAll(() => { + try { + if (failures > 0 || !setUp) console.error(compose('logs', '--no-color', '--tail', '200')) + } finally { + compose('down', '-v') + rmSync(BACKUPS, { recursive: true, force: true }) + rmSync(LOCK, { recursive: true, force: true }) + rmSync(MARKER, { force: true }) + rmSync(`${MARKER}.partial`, { recursive: true, force: true }) + rmSync(ENV_FILE, { force: true }) + } +}, 300_000) + +/** A long step's budget: a script run, or a wait on the stack, inside one test. */ +const LONG = 300_000 + +/** What the ClickHouse backups disk holds now; empty after every run. */ +const backupsDisk = (): string => + compose( + 'exec', + '-T', + 'clickhouse', + 'sh', + '-c', + 'ls -A /var/lib/clickhouse/backups 2>/dev/null || true', + ).trim() + +/** + * A `docker` first on PATH that answers the calls one `case` names and hands + * every other call to the real one. `cases` is the body of a shell `case "$a"` + * over each argument. Returns the PATH to run a script with. + */ +function stubDocker(cases: string): { path: string; remove: () => void } { + const bin = join(TMP, 'stub-docker') + rmSync(bin, { recursive: true, force: true }) + mkdirSync(bin) + const real = execFileSync('sh', ['-c', 'command -v docker'], { encoding: 'utf8' }).trim() + writeFileSync( + join(bin, 'docker'), + [ + '#!/bin/sh', + 'for a in "$@"; do', + ' case "$a" in', + cases, + ' esac', + 'done', + `exec ${real} "$@"`, + '', + ].join('\n'), + { mode: 0o755 }, + ) + return { + path: `${bin}:${process.env.PATH ?? ''}`, + remove: () => rmSync(bin, { recursive: true, force: true }), + } +} + +describe('backup.sh', () => { + it( + 'writes six files, readable by their owner alone, and a manifest that describes them', + () => { + const workerBefore = startedAt('worker') + const redirectBefore = startedAt('redirect') + const r = runScript('backup.sh', [join(BACKUPS, 'first')]) + expect(r.status, r.out).toBe(0) + firstBackup = onlyBackupIn(join(BACKUPS, 'first')) + + expect(readdirSync(firstBackup).sort()).toEqual(['MANIFEST', ...ARTEFACTS]) + expect(statSync(firstBackup).mode & 0o777).toBe(0o700) + for (const f of readdirSync(firstBackup)) { + expect(statSync(join(firstBackup, f)).mode & 0o777, f).toBe(0o600) + } + + const m = manifest(firstBackup) + expect(Object.keys(m).sort()).toEqual( + [ + 'admin_host', + 'clickmonk_backup_version', + 'clickmonk_version', + 'rows.clickhouse.clicks', + 'rows.clickhouse.clicks_hourly', + 'rows.clickhouse.clicks_hourly_dim', + 'schema_version', + 'secret_fingerprint', + ...ARTEFACTS.map((a) => `sha256.${a}`), + 'timestamp', + ].sort(), + ) + expect(m.clickmonk_backup_version).toBe('1') + expect(m.timestamp).toBe(firstBackup.slice(firstBackup.lastIndexOf('/') + 1)) + expect(m.timestamp).toMatch(/^\d{4}-\d{2}-\d{2}T\d{6}Z$/) + expect(m.admin_host).toBe(ADMIN_HOST) + expect(m.schema_version).toBe(pg('SELECT max(version) FROM schema_migrations')) + // The release against the one the CLI prints; the schema version above is + // the ledger's, which is the number a restore checks. + const printed = /^clickmonk (\d+\.\d+\.\d+) \(schema version \d+\)$/.exec( + cli('version').trim(), + ) + expect(m.clickmonk_version).toBe(printed?.[1]) + for (const a of ARTEFACTS) expect(m[`sha256.${a}`], a).toBe(sha256(join(firstBackup, a))) + // Counted by this test from the store, not read back from the manifest. + expect(m['rows.clickhouse.clicks']).toBe(ch('SELECT count() FROM clicks FINAL')) + expect(m['rows.clickhouse.clicks']).toBe('2') + expect(m['rows.clickhouse.clicks_hourly']).toBe(ch('SELECT count() FROM clicks_hourly FINAL')) + expect(m['rows.clickhouse.clicks_hourly_dim']).toBe( + ch('SELECT count() FROM clicks_hourly_dim FINAL'), + ) + // The secret itself is in `env`, which is the point of `env`; the manifest + // carries only a fingerprint of it, computed here independently. + expect(readFileSync(join(firstBackup, 'env'), 'utf8')).toBe(readFileSync(ENV_FILE, 'utf8')) + const secret = ENV.CLICKMONK_SECRET ?? '' + expect(m.secret_fingerprint).toBe( + createHash('sha256').update(`clickmonk-secret:${secret}`).digest('hex').slice(0, 16), + ) + expect(readFileSync(join(firstBackup, 'MANIFEST'), 'utf8')).not.toContain(secret) + + // The worker was stopped and started again; the redirect never stopped. + expect(startedAt('worker')).not.toBe(workerBefore) + expect(startedAt('redirect')).toBe(redirectBefore) + expect(running()).toEqual(ALL_SERVICES) + }, + LONG, + ) + + it('leaves nothing on the ClickHouse backups disk', () => { + expect(backupsDisk()).toBe('') + }) + + // An archive a killed run's BACKUP finished after that run's trap had gone: + // nothing else would ever remove it, and it is a whole backup's size. + it( + 'removes an archive an earlier run left on the ClickHouse backups disk', + () => { + compose( + 'exec', + '-T', + 'clickhouse', + 'touch', + '/var/lib/clickhouse/backups/clickmonk-2000-01-01T000000Z-1.zip', + ) + expect(backupsDisk()).toBe('clickmonk-2000-01-01T000000Z-1.zip') + const r = runScript('backup.sh', [join(BACKUPS, 'orphan')]) + expect(r.status, r.out).toBe(0) + onlyBackupIn(join(BACKUPS, 'orphan')) + expect(backupsDisk()).toBe('') + }, + LONG, + ) + + // Rows that share a sort key and have not been merged yet: the shipper's + // normal case when a segment is shipped twice. A plain count() includes them, + // and a restore that merged them would then disagree with the manifest. + it( + 'records the counts ClickHouse gives with FINAL, not the unmerged rows', + () => { + const tables = ['clicks', 'clicks_hourly', 'clicks_hourly_dim'] + for (const t of tables) ch(`SYSTEM STOP MERGES ${t}`) + try { + ch('INSERT INTO clicks SELECT * FROM clicks LIMIT 1') + expect(ch('SELECT count() FROM clicks')).toBe('3') + expect(ch('SELECT count() FROM clicks FINAL')).toBe('2') + const r = runScript('backup.sh', [join(BACKUPS, 'unmerged')]) + expect(r.status, r.out).toBe(0) + const m = manifest(onlyBackupIn(join(BACKUPS, 'unmerged'))) + expect(m['rows.clickhouse.clicks']).toBe('2') + for (const t of tables) { + expect(m[`rows.clickhouse.${t}`], t).toBe(ch(`SELECT count() FROM ${t} FINAL`)) + } + } finally { + for (const t of tables) ch(`SYSTEM START MERGES ${t}`) + } + }, + LONG, + ) + + // An operator's wrapper reading destinations from a file: a docker call + // that inherited the loop's stdin would swallow the rest of the list. + it( + 'reads nothing from its standard input, so a while-read loop runs it for every line', + () => { + const a = join(BACKUPS, 'loop-a') + const b = join(BACKUPS, 'loop-b') + const r = spawnSync( + 'bash', + ['-c', 'while read -r d; do "$0" "$d" || exit 1; done', join(ROOT, 'backup.sh')], + { + cwd: ROOT, + env: SCRIPT_ENV, + input: `${a}\n${b}\n`, + encoding: 'utf8', + stdio: 'pipe', + timeout: 600_000, + }, + ) + expect(r.status, `${r.stdout}${r.stderr}`).toBe(0) + onlyBackupIn(a) + onlyBackupIn(b) + }, + LONG, + ) + + // A reader that goes away: `./backup.sh dest | head -1` in a cron line. + it( + 'finishes the backup when whatever reads its output stops reading', + () => { + const dest = join(BACKUPS, 'head') + const r = spawnSync( + 'bash', + ['-c', '"$0" "$1" | head -1; exit ${PIPESTATUS[0]}', join(ROOT, 'backup.sh'), dest], + { + cwd: ROOT, + env: SCRIPT_ENV, + encoding: 'utf8', + stdio: 'pipe', + timeout: 600_000, + }, + ) + expect(r.status, `${r.stdout}${r.stderr}`).toBe(0) + expect(readdirSync(onlyBackupIn(dest))).toContain('MANIFEST') + expect(running()).toEqual(ALL_SERVICES) + }, + LONG, + ) + + it( + 'asks the worker for the release when the redirect is stopped', + () => { + compose('stop', 'redirect') + try { + const r = runScript('backup.sh', [join(BACKUPS, 'redirect-stopped')]) + expect(r.status, r.out).toBe(0) + expect(r.out).not.toContain('predates') + const m = manifest(onlyBackupIn(join(BACKUPS, 'redirect-stopped'))) + const printed = /^clickmonk (\d+\.\d+\.\d+) \(schema version \d+\)$/.exec( + cli('version').trim(), + ) + expect(printed).not.toBeNull() + expect(m.clickmonk_version).toBe(printed?.[1]) + } finally { + compose('start', 'redirect') + } + }, + LONG, + ) + + it( + 'leaves a worker it did not stop stopped, and still takes the backup', + () => { + // The redirect too, so that no container can say which release this is. + compose('stop', 'worker', 'redirect') + try { + const r = runScript('backup.sh', [join(BACKUPS, 'worker-stopped')]) + expect(r.status, r.out).toBe(0) + expect(r.out).toContain('The worker is not running; it is left that way.') + expect(r.out).toContain('Neither the redirect nor the worker is running') + expect(running()).not.toContain('worker') + const m = manifest(onlyBackupIn(join(BACKUPS, 'worker-stopped'))) + expect(m.clickmonk_version).toBe('unknown') + } finally { + compose('start', 'redirect', 'worker') + } + }, + LONG, + ) + + // Caddy crash-looping on a bad Caddyfile is the likeliest broken service, and + // the moment a backup is most wanted. Its volume is read through a one-off + // container instead, and Caddy is left as it was. + it( + 'takes the backup with Caddy stopped, and leaves it stopped', + () => { + compose('stop', 'caddy') + try { + const r = runScript('backup.sh', [join(BACKUPS, 'caddy-stopped')]) + expect(r.status, r.out).toBe(0) + const dir = onlyBackupIn(join(BACKUPS, 'caddy-stopped')) + // The certificate authority's account and the admin host's certificate + // are in Caddy's data, so the archive lists more than its root entry. + const listed = execFileSync('tar', ['-tf', join(dir, 'caddy-data.tar')], { + encoding: 'utf8', + }) + expect(listed).toMatch(/certificates\//) + expect(running()).not.toContain('caddy') + } finally { + compose('start', 'caddy') + } + }, + LONG, + ) + + // Every install upgrading to the first release runs an image whose CLI has + // no `version`: it prints its usage and exits 1. Its backup is the one the + // upgrade instructions ask for first, so it must still be taken. + it( + 'takes the backup from an image that predates `clickmonk version`, recording the release as unknown', + () => { + const stub = stubDocker(' version) echo "usage:"; exit 1 ;;') + let r: ScriptResult + try { + r = runScript('backup.sh', [join(BACKUPS, 'old-image')], { env: { PATH: stub.path } }) + } finally { + stub.remove() + } + expect(r.status, r.out).toBe(0) + expect(r.out).toContain("The running image predates 'clickmonk version'") + const m = manifest(onlyBackupIn(join(BACKUPS, 'old-image'))) + expect(m.clickmonk_version).toBe('unknown') + expect(m.schema_version).toBe(pg('SELECT max(version) FROM schema_migrations')) + expect(Object.keys(m)).toHaveLength(14) + }, + LONG, + ) + + // The one step that runs while the worker is stopped is the one that fails + // here, so "running again afterwards" is something the trap had to do. + it( + 'starts the worker again, and leaves nothing behind, when a step fails while it is stopped', + () => { + // The lock is still held while the trap starts the worker: released + // first, another run could start and find the worker stopped. + const held = join(TMP, 'lock-held-at-start') + rmSync(held, { force: true }) + const stub = stubDocker( + [ + ' *"BACKUP DATABASE"*) echo "simulated failure of BACKUP" >&2; exit 1 ;;', + ` start) [ -d ${LOCK} ] && touch ${held} ;;`, + ].join('\n'), + ) + const dest = join(BACKUPS, 'failed') + const workerBefore = startedAt('worker') + let r: ScriptResult + try { + r = runScript('backup.sh', [dest], { env: { PATH: stub.path } }) + } finally { + stub.remove() + } + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('the backup failed during the ClickHouse step') + expect(r.out).toContain('No data was changed') + expect(existsSync(held), 'the lock was gone when the worker was started').toBe(true) + rmSync(held, { force: true }) + expect(existsSync(LOCK)).toBe(false) + expect(readdirSync(dest)).toEqual([]) + expect(backupsDisk()).toBe('') + expect(running()).toEqual(ALL_SERVICES) + expect(startedAt('worker')).not.toBe(workerBefore) + }, + LONG, + ) + + // An operator who sends TERM, sees the prompt wait while the trap starts the + // worker, and sends it again. The stub sends both: the first during the + // BACKUP, the second as the trap's `docker compose start` begins. + it( + 'starts the worker again when a second TERM arrives while it is being started', + () => { + const stub = stubDocker( + [ + ' *"BACKUP DATABASE"*) kill -TERM $PPID; sleep 1; exit 1 ;;', + ' start) kill -TERM $PPID ;;', + ].join('\n'), + ) + const dest = join(BACKUPS, 'twice') + let r: ScriptResult + try { + r = runScript('backup.sh', [dest], { env: { PATH: stub.path } }) + } finally { + stub.remove() + } + try { + expect(r.status, r.out).not.toBe(0) + expect(running()).toEqual(ALL_SERVICES) + expect(readdirSync(dest)).toEqual([]) + expect(backupsDisk()).toBe('') + expect(existsSync(LOCK)).toBe(false) + } finally { + // A trap killed before its end leaves the lock; every later test + // would then fail on it rather than on what it tests. + rmSync(LOCK, { recursive: true, force: true }) + } + }, + LONG, + ) + + // A dump that fails half way still wrote something, and without pipefail + // the pipe's status is the write's: a truncated dump with a valid checksum. + it( + 'fails, and keeps nothing, when pg_dump fails after writing part of the dump', + () => { + const stub = stubDocker(' *pg_dump*) printf PGDMP-partial; exit 1 ;;') + const dest = join(BACKUPS, 'partial-dump') + let r: ScriptResult + try { + r = runScript('backup.sh', [dest], { env: { PATH: stub.path } }) + } finally { + stub.remove() + } + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('the backup failed during the Postgres step') + expect(readdirSync(dest)).toEqual([]) + expect(running()).toEqual(ALL_SERVICES) + }, + LONG, + ) + + // ClickHouse writes its archive beside its data. A backup that would not fit + // is refused before the worker stops and before anything is written. + it( + 'refuses a backup the ClickHouse volume has no room for, before stopping anything', + () => { + const stub = stubDocker(' *"free_space FROM system.disks"*) echo 1; exit 0 ;;') + const dest = join(BACKUPS, 'no-room') + const workerBefore = startedAt('worker') + let r: ScriptResult + try { + r = runScript('backup.sh', [dest], { env: { PATH: stub.path } }) + } finally { + stub.remove() + } + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('The ClickHouse volume has 0 MiB free and a backup needs about') + expect(r.out).toContain('the backup failed during the validation step') + expect(readdirSync(BACKUPS)).not.toContain('no-room') + expect(startedAt('worker')).toBe(workerBefore) + }, + LONG, + ) + + // A size ClickHouse cannot give must not count as zero bytes needed. + it( + 'refuses when ClickHouse cannot say how much its database holds, before stopping anything', + () => { + const stub = stubDocker(' *"FROM system.parts"*) exit 1 ;;') + const workerBefore = startedAt('worker') + let r: ScriptResult + try { + r = runScript('backup.sh', [join(BACKUPS, 'no-size')], { env: { PATH: stub.path } }) + } finally { + stub.remove() + } + expect(r.status, r.out).toBe(1) + expect(r.out).toContain("Could not read ClickHouse's size and free space: 'nothing'") + expect(r.out).toContain('the backup failed during the validation step') + expect(readdirSync(BACKUPS)).not.toContain('no-size') + expect(startedAt('worker')).toBe(workerBefore) + }, + LONG, + ) + + // Every install from before the backups disk existed, until ClickHouse is + // recreated: system.disks has no such row, and nothing else would say why. + it( + 'refuses a ClickHouse started without the backups disk, before stopping anything', + () => { + const stub = stubDocker(' *"FROM system.disks"*) exit 0 ;;') + const dest = join(BACKUPS, 'no-disk') + const workerBefore = startedAt('worker') + let r: ScriptResult + try { + r = runScript('backup.sh', [dest], { env: { PATH: stub.path } }) + } finally { + stub.remove() + } + expect(r.status, r.out).toBe(1) + expect(r.out).toContain("ClickHouse has no disk named 'backups'") + expect(r.out).toContain('the backup failed during the validation step') + expect(readdirSync(BACKUPS)).not.toContain('no-disk') + expect(startedAt('worker')).toBe(workerBefore) + }, + LONG, + ) + + // Two runs started in the same second name the same directory. The second + // must fail without touching it, since the first is still writing there. + it( + 'refuses a backup directory that already exists, and leaves it as it was', + () => { + const bin = join(TMP, 'stub-date') + rmSync(bin, { recursive: true, force: true }) + mkdirSync(bin) + writeFileSync(join(bin, 'date'), '#!/bin/sh\necho 2026-10-01T041500Z\n', { mode: 0o755 }) + const theirs = join(BACKUPS, 'same-second', '2026-10-01T041500Z') + mkdirSync(theirs, { recursive: true }) + writeFileSync(join(theirs, 'marker'), 'written by the other run\n') + const workerBefore = startedAt('worker') + let r: ScriptResult + try { + r = runScript('backup.sh', [join(BACKUPS, 'same-second')], { + env: { PATH: `${bin}:${process.env.PATH ?? ''}` }, + }) + } finally { + rmSync(bin, { recursive: true, force: true }) + } + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('Another backup started in the same second') + expect(r.out).toContain('the backup failed during the destination step') + expect(readdirSync(theirs)).toEqual(['marker']) + expect(readFileSync(join(theirs, 'marker'), 'utf8')).toBe('written by the other run\n') + expect(startedAt('worker')).toBe(workerBefore) + }, + LONG, + ) + + it( + 'refuses a destination it cannot create, before stopping anything', + () => { + const workerBefore = startedAt('worker') + // A path under a regular file: no mkdir can make it. + const r = runScript('backup.sh', [join(ENV_FILE, 'nested')]) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('the backup failed during the destination step') + expect(startedAt('worker')).toBe(workerBefore) + }, + LONG, + ) + + it( + 'refuses when COMPOSE_ENV_FILES names more than one file, before stopping anything', + () => { + const workerBefore = startedAt('worker') + const r = runScript('backup.sh', [join(BACKUPS, 'two-env-files')], { + env: { COMPOSE_ENV_FILES: `${ENV_FILE},${ENV_FILE}` }, + }) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('COMPOSE_ENV_FILES names more than one file') + expect(startedAt('worker')).toBe(workerBefore) + }, + LONG, + ) + + // A worker that will not start again: the backup is kept and the run fails + // saying so, and the lock is released, or every later run would refuse. + it( + 'releases the lock when it cannot start the worker again', + () => { + const stub = stubDocker(' start) exit 1 ;;') + let r: ScriptResult + try { + r = runScript('backup.sh', [join(BACKUPS, 'no-restart')], { env: { PATH: stub.path } }) + } finally { + stub.remove() + compose('start', 'worker') + } + try { + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('the worker is stopped and this script could not start it again') + onlyBackupIn(join(BACKUPS, 'no-restart')) + expect(existsSync(LOCK)).toBe(false) + } finally { + // A lock left here would fail every later test on the lock instead. + rmSync(LOCK, { recursive: true, force: true }) + } + }, + LONG, + ) + + // A BACKUP or RESTORE whose client was killed runs on in the server. Its + // archive is what the sweep would delete, so the backup waits for it. + it( + 'refuses while ClickHouse is still running an earlier backup or restore, before stopping anything', + () => { + const stub = stubDocker( + ` *"FROM system.backups"*) echo "RESTORING Disk('backups', 'clickmonk-restore-2026-10-01T041500Z.zip')"; exit 0 ;;`, + ) + const dest = join(BACKUPS, 'in-progress') + const workerBefore = startedAt('worker') + compose( + 'exec', + '-T', + 'clickhouse', + 'touch', + '/var/lib/clickhouse/backups/clickmonk-restore-2026-10-01T041500Z.zip', + ) + let r: ScriptResult + try { + r = runScript('backup.sh', [dest], { env: { PATH: stub.path } }) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain( + "ClickHouse is still running an earlier backup or restore: RESTORING Disk('backups', 'clickmonk-restore-2026-10-01T041500Z.zip').", + ) + expect(r.out).toContain('Wait for it to finish') + expect(r.out).toContain('the backup failed during the validation step') + // Its archive is left for it. + expect(backupsDisk()).toBe('clickmonk-restore-2026-10-01T041500Z.zip') + expect(readdirSync(BACKUPS)).not.toContain('in-progress') + expect(startedAt('worker')).toBe(workerBefore) + } finally { + stub.remove() + compose( + 'exec', + '-T', + 'clickhouse', + 'rm', + '-f', + '/var/lib/clickhouse/backups/clickmonk-restore-2026-10-01T041500Z.zip', + ) + } + }, + LONG, + ) + + // Not knowing is not the same as nothing running: the sweep waits for an answer. + it( + 'refuses when ClickHouse cannot say whether a backup or restore is running', + () => { + const stub = stubDocker(' *"FROM system.backups"*) exit 1 ;;') + const workerBefore = startedAt('worker') + let r: ScriptResult + try { + r = runScript('backup.sh', [join(BACKUPS, 'unknown-progress')], { + env: { PATH: stub.path }, + }) + } finally { + stub.remove() + } + expect(r.status, r.out).toBe(1) + expect(r.out).toContain( + 'Could not ask ClickHouse whether a backup or restore is still running.', + ) + expect(readdirSync(BACKUPS)).not.toContain('unknown-progress') + expect(startedAt('worker')).toBe(workerBefore) + }, + LONG, + ) + + // Compose itself fails on an env file it cannot find, so a check of the + // services first would blame a store that is running. + it( + 'names a missing env file, rather than a store, before stopping anything', + () => { + const missing = join(TMP, 'no-such.env') + const workerBefore = startedAt('worker') + const r = runScript('backup.sh', [join(BACKUPS, 'no-env')], { + env: { COMPOSE_ENV_FILES: missing }, + }) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain(`Cannot read ${missing}`) + expect(r.out).not.toContain('service is not running') + expect(startedAt('worker')).toBe(workerBefore) + }, + LONG, + ) + + // Two runs at once: the second would take the worker the first stopped for + // one it may leave alone, and copy while the first starts it again. + it( + 'refuses while another run holds the lock, and leaves that run to finish', + async () => { + const reached = join(TMP, 'lock-reached') + const release = join(TMP, 'lock-release') + rmSync(reached, { force: true }) + rmSync(release, { force: true }) + // The holder's BACKUP runs for real and it pauses after it, with its + // archive on the backups disk: a refused run must not sweep it away. + const real = execFileSync('sh', ['-c', 'command -v docker'], { encoding: 'utf8' }).trim() + const stub = stubDocker( + ` *"BACKUP DATABASE"*) ${real} "$@"; rc=$?; touch ${reached}; while [ ! -e ${release} ]; do sleep 0.2; done; exit $rc ;;`, + ) + const first = join(BACKUPS, 'holder') + let out = '' + const holder = spawn(join(ROOT, 'backup.sh'), [first], { + cwd: ROOT, + env: { ...SCRIPT_ENV, PATH: stub.path }, + stdio: ['ignore', 'pipe', 'pipe'], + }) + holder.stdout.on('data', (d: Buffer) => { + out += d.toString() + }) + holder.stderr.on('data', (d: Buffer) => { + out += d.toString() + }) + const done = new Promise((resolve) => + holder.on('close', (code) => resolve(code ?? -1)), + ) + try { + await until('the first run to reach its BACKUP', 120_000, () => existsSync(reached)) + const holderPid = readFileSync(join(LOCK, 'pid'), 'utf8').trim() + expect(holderPid).toBe(String(holder.pid)) + + const r = runScript('backup.sh', [join(BACKUPS, 'second')]) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain(`Another backup or restore holds ${LOCK} (pid ${holderPid})`) + expect(r.out).toContain('the backup failed during the validation step') + expect(readdirSync(BACKUPS)).not.toContain('second') + // The first run's lock, and its stopped worker, are as it left them. + expect(readFileSync(join(LOCK, 'pid'), 'utf8').trim()).toBe(holderPid) + expect(running()).not.toContain('worker') + expect(backupsDisk()).toMatch(/^clickmonk-.*\.zip$/m) + } finally { + // Let the holder finish even when an assertion above failed, so the + // next test does not start beside it. + writeFileSync(release, '') + await done + stub.remove() + rmSync(reached, { force: true }) + rmSync(release, { force: true }) + } + const status = await done + expect(status, out).toBe(0) + onlyBackupIn(first) + expect(existsSync(LOCK)).toBe(false) + expect(running()).toEqual(ALL_SERVICES) + }, + LONG, + ) + + // A run killed with SIGKILL leaves its lock. It is never taken over: two + // runs that both judged it stale would both hold it. + it( + 'refuses a lock left by a run that was killed, and says how to remove it', + () => { + mkdirSync(LOCK) + writeFileSync(join(LOCK, 'pid'), '4242\n') + const workerBefore = startedAt('worker') + try { + const r = runScript('backup.sh', [join(BACKUPS, 'stale-lock')]) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain(`Another backup or restore holds ${LOCK} (pid 4242)`) + expect(r.out).toContain(`rm -rf ${LOCK}`) + expect(readFileSync(join(LOCK, 'pid'), 'utf8')).toBe('4242\n') + expect(readdirSync(BACKUPS)).not.toContain('stale-lock') + expect(startedAt('worker')).toBe(workerBefore) + } finally { + rmSync(LOCK, { recursive: true, force: true }) + } + }, + LONG, + ) + + // A restore interrupted part way leaves stores from two moments; a backup + // of them would be complete, checksummed and of a state that never existed. + it( + 'refuses while a restore has not finished, and names the backup to finish it with', + () => { + writeFileSync( + MARKER, + 'backup=/srv/backups/2026-10-01T041500Z\nenvironment=COMPOSE_PROJECT_NAME=elsewhere \n', + ) + const workerBefore = startedAt('worker') + try { + const r = runScript('backup.sh', [join(BACKUPS, 'half-restored')]) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain( + 'A restore of /srv/backups/2026-10-01T041500Z did not finish, so the stores may be half restored.', + ) + // The variables that restore ran with, so the advice reaches that install. + expect(r.out).toContain( + `Finish it first: COMPOSE_PROJECT_NAME=elsewhere ${ROOT}/restore.sh /srv/backups/2026-10-01T041500Z`, + ) + expect(r.out).toContain('the backup failed during the validation step') + expect(readdirSync(BACKUPS)).not.toContain('half-restored') + expect(startedAt('worker')).toBe(workerBefore) + expect(existsSync(MARKER)).toBe(true) + } finally { + rmSync(MARKER, { force: true }) + } + }, + LONG, + ) + + // The worker stopped, clicks written, and the worker started again: a spool + // copied after the start would find their segments shipped and deleted, and + // a ClickHouse backup taken before it would not have them either. The stub + // holds the stop open until the clicks are sealed, and holds the start until + // the worker has drained the spool, so that ordering is seen every time. + it( + 'every click written before the worker stopped is in the backup: in ClickHouse or in spool.tar', + async () => { + const stopped = join(TMP, 'worker-stopped') + const go = join(TMP, 'worker-may-start') + rmSync(stopped, { force: true }) + rmSync(go, { force: true }) + const real = execFileSync('sh', ['-c', 'command -v docker'], { encoding: 'utf8' }).trim() + const stub = stubDocker( + [ + ' stop)', + ` case " $* " in *" worker "*) ${real} "$@" || exit 1; touch ${stopped}`, + ` i=0; while [ ! -e ${go} ] && [ $i -lt 240 ]; do sleep 0.5; i=$((i + 1)); done; exit 0 ;; esac ;;`, + ' start)', + ` case " $* " in *" worker "*) ${real} "$@" || exit 1`, + ' i=0; while [ $i -lt 240 ]; do', + ` n=$(${real} compose exec -T redirect sh -c 'ls "$0" | grep -c "^seg-"' ${SPOOL} { + out += d.toString() + }) + child.stderr.on('data', (d: Buffer) => { + out += d.toString() + }) + const exited = new Promise((resolve) => child.on('close', (c) => resolve(c ?? -1))) + await until('the backup to stop the worker', 60_000, () => existsSync(stopped)) + // With the worker stopped, nothing moves between the two. + const inStore = num(ch('SELECT count() FROM clicks FINAL')) + const waiting = spoolLines() + const before = inStore + waiting + click() + click() + await until('the two clicks sealed', 60_000, () => spoolLines() === waiting + 2) + writeFileSync(go, '') + const status = await exited + expect(status, out).toBe(0) + const dir = onlyBackupIn(dest) + const inSpool = execFileSync('tar', ['-xOf', join(dir, 'spool.tar')], { encoding: 'utf8' }) + .split('\n') + .filter((l) => l !== '').length + expect(inSpool).toBeGreaterThanOrEqual(2) + expect(num(manifest(dir)['rows.clickhouse.clicks'] ?? '') + inSpool).toBe(before + 2) + } finally { + stub.remove() + rmSync(stopped, { force: true }) + rmSync(go, { force: true }) + } + }, + LONG, + ) + + // A signal while `docker compose stop worker` still waits for the worker to + // drain: that stop goes on without the script. The stub models the drain. + it( + 'starts the worker again when a signal arrives while it is still stopping', + async () => { + const real = execFileSync('sh', ['-c', 'command -v docker'], { encoding: 'utf8' }).trim() + const stub = stubDocker(` stop) kill -TERM $PPID; sleep 3; exec ${real} "$@" ;;`) + const dest = join(BACKUPS, 'during-stop') + let r: ScriptResult + try { + r = runScript('backup.sh', [dest], { env: { PATH: stub.path } }) + } finally { + stub.remove() + } + expect(r.status, r.out).toBe(143) + // Long enough for a stop still in flight to have finished. + await sleep(8_000) + expect(running()).toEqual(ALL_SERVICES) + expect(existsSync(LOCK)).toBe(false) + expect(readdirSync(dest)).toEqual([]) + }, + LONG, + ) + + // `stop` can stop the worker and still exit non-zero; the trap must know it did. + it( + 'starts the worker again when its stop failed after stopping it', + () => { + const real = execFileSync('sh', ['-c', 'command -v docker'], { encoding: 'utf8' }).trim() + const stub = stubDocker(` stop) ${real} "$@"; exit 1 ;;`) + let r: ScriptResult + try { + r = runScript('backup.sh', [join(BACKUPS, 'stop-failed')], { env: { PATH: stub.path } }) + } finally { + stub.remove() + } + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('the backup failed during the quiesce step') + expect(running()).toEqual(ALL_SERVICES) + expect(existsSync(LOCK)).toBe(false) + }, + LONG, + ) + + it( + 'parses, and prints its usage, under bash 3.2', + () => { + for (const script of ['backup.sh', 'backup-lib.sh', 'restore.sh']) { + const parsed = spawnSync( + 'docker', + ['run', '--rm', '-v', `${ROOT}:/src:ro`, 'bash:3.2.57', 'bash', '-n', `/src/${script}`], + { encoding: 'utf8', stdio: 'pipe', timeout: 120_000 }, + ) + expect(parsed.status, `${script}: ${parsed.stderr}`).toBe(0) + } + for (const script of ['backup.sh', 'restore.sh']) { + const help = spawnSync( + 'docker', + [ + 'run', + '--rm', + '-v', + `${ROOT}:/src:ro`, + 'bash:3.2.57', + 'bash', + `/src/${script}`, + '--help', + ], + { encoding: 'utf8', stdio: 'pipe', timeout: 120_000 }, + ) + expect(help.status, `${script}: ${help.stderr}`).toBe(0) + expect(help.stderr).toContain(`usage: ./${script}`) + } + }, + LONG, + ) +}) + +describe('restore.sh refuses, before it stops or changes anything', () => { + interface World { + running: string[] + redirectStarted: string + links: string + clicks: string + /** A refused run releases the lock it took, and leaves one it did not take. */ + locked: boolean + } + /** What a refusal must leave exactly as it found it. */ + const world = (): World => ({ + running: running(), + redirectStarted: startedAt('redirect'), + links: pg('SELECT count(*) FROM links'), + clicks: ch('SELECT count() FROM clicks FINAL'), + locked: existsSync(LOCK), + }) + + /** A copy of the first backup, to damage without damaging the original. */ + function copyOfFirst(name: string): string { + const dir = join(BACKUPS, 'copies', name) + rmSync(dir, { recursive: true, force: true }) + cpSync(firstBackup, dir, { recursive: true }) + return dir + } + + function expectRefusal(r: ScriptResult, reason: string, before: World): void { + expect(r.status, r.out).toBe(1) + expect(r.out).toContain(reason) + expect(r.out).toContain('Nothing was changed, and nothing was stopped.') + expect(r.out).not.toContain('Stopped before anything was changed.') + expect(world()).toEqual(before) + } + + it( + 'refuses a directory that is not one backup', + () => { + const before = world() + // The destination backup.sh was given, rather than the directory it wrote. + expectRefusal( + runScript('restore.sh', [join(BACKUPS, 'first')], { input: 'irrelevant\n' }), + 'There is no MANIFEST in', + before, + ) + }, + LONG, + ) + + it( + 'refuses a backup missing one of its files', + () => { + const dir = copyOfFirst('missing') + rmSync(join(dir, 'env')) + const before = world() + expectRefusal(runScript('restore.sh', [dir]), 'env is missing or empty', before) + }, + LONG, + ) + + it( + 'refuses a damaged file, by its checksum', + () => { + const dir = copyOfFirst('damaged') + appendFileSync(join(dir, 'postgres.dump'), 'x') + const before = world() + expectRefusal( + runScript('restore.sh', [dir]), + 'postgres.dump does not match the checksum in the manifest', + before, + ) + }, + LONG, + ) + + it( + 'refuses a backup newer than the image that would run it', + () => { + const dir = copyOfFirst('newer') + const path = join(dir, 'MANIFEST') + const newer = imageSchema() + 1 + writeFileSync( + path, + readFileSync(path, 'utf8').replace(/^schema_version=\d+$/m, `schema_version=${newer}`), + ) + const before = world() + expectRefusal(runScript('restore.sh', [dir]), 'This backup is newer than this image', before) + }, + LONG, + ) + + it( + 'refuses when the answer is not the backup’s timestamp', + () => { + const before = world() + const r = runScript('restore.sh', [firstBackup], { input: 'yes\n' }) + expect(r.out).toContain('Type the backup’s timestamp to go ahead') + // The same secret and admin host as the backup's, so neither warning. + expect(r.out).not.toContain('is not the one this backup was taken with') + expect(r.out).not.toContain('CLICKMONK_ADMIN_HOST is') + expectRefusal(r, 'That is not the backup’s timestamp', before) + }, + LONG, + ) + + it( + 'refuses when nobody answers', + () => { + const before = world() + expectRefusal( + runScript('restore.sh', [firstBackup], { input: '' }), + 'That is not the backup’s timestamp', + before, + ) + }, + LONG, + ) + + it( + 'warns, before asking, when this install signs cookies with a different secret', + () => { + const other = join(TMP, 'backup-other-secret.env') + writeFileSync( + other, + readFileSync(ENV_FILE, 'utf8').replace( + /^CLICKMONK_SECRET=.*$/m, + `CLICKMONK_SECRET=${'o'.repeat(40)}`, + ), + { mode: 0o600 }, + ) + const before = world() + try { + const r = runScript('restore.sh', [firstBackup], { + input: 'no\n', + env: { COMPOSE_ENV_FILES: other }, + }) + expect(r.out).toContain('is not the one this backup was taken with') + expect(r.out.indexOf('is not the one this backup was taken with')).toBeLessThan( + r.out.indexOf('Type the backup’s timestamp to go ahead'), + ) + expectRefusal(r, 'That is not the backup’s timestamp', before) + } finally { + rmSync(other, { force: true }) + } + }, + LONG, + ) + + it( + 'refuses a path that is not a directory', + () => { + const before = world() + const missing = join(BACKUPS, 'no-such-backup') + expectRefusal(runScript('restore.sh', [missing]), `${missing} is not a directory`, before) + }, + LONG, + ) + + // One manifest line changed or taken away at a time. Each is refused by its + // own guard, before anything is asked of the stack. + it.each([ + { + what: 'a format this script does not read', + edit: (m: string) => + m.replace(/^clickmonk_backup_version=.*$/m, 'clickmonk_backup_version=2'), + reason: "Its format is '2'; this script reads 1.", + }, + { + what: 'no timestamp', + edit: (m: string) => m.replace(/^timestamp=.*\n/m, ''), + reason: 'The manifest has no timestamp.', + }, + { + // The timestamp is part of a file name and of a ClickHouse statement. + what: 'a timestamp backup.sh would not write', + edit: (m: string) => m.replace(/^timestamp=.*$/m, "timestamp=2026-10-01T041500Z')"), + reason: + "The manifest's timestamp is '2026-10-01T041500Z')', which is not one backup.sh writes.", + }, + { + what: 'a schema version that is not a number', + edit: (m: string) => m.replace(/^schema_version=.*$/m, 'schema_version=ten'), + reason: "The manifest's schema_version is 'ten', which is not a number.", + }, + { + // The rollups' counts are absent from a backup of an install without + // them; the raw table's never is. + what: 'no row count for the raw clicks table', + edit: (m: string) => m.replace(/^rows\.clickhouse\.clicks=.*\n/m, ''), + reason: 'The manifest records no row count for clicks,', + }, + { + what: 'no checksum for one file', + edit: (m: string) => m.replace(/^sha256\.env=.*\n/m, ''), + reason: 'The manifest records no checksum for env.', + }, + ])( + 'refuses a manifest with $what', + ({ what, edit, reason }) => { + const dir = copyOfFirst(`manifest-${what.replaceAll(' ', '-')}`) + const path = join(dir, 'MANIFEST') + const edited = edit(readFileSync(path, 'utf8')) + expect(edited).not.toBe(readFileSync(path, 'utf8')) + writeFileSync(path, edited) + const before = world() + expectRefusal(runScript('restore.sh', [dir]), reason, before) + }, + LONG, + ) + + // A file it cannot check is not restored. Every tool on PATH but those three. + it( + 'refuses on a host with no SHA-256 tool', + () => { + const bin = join(TMP, 'no-sha256') + rmSync(bin, { recursive: true, force: true }) + mkdirSync(bin) + const linked = new Set(['sha256sum', 'shasum', 'openssl']) + for (const dir of (process.env.PATH ?? '').split(':')) { + if (dir === '' || !existsSync(dir) || !statSync(dir).isDirectory()) continue + for (const name of readdirSync(dir)) { + if (linked.has(name)) continue + linked.add(name) + symlinkSync(join(dir, name), join(bin, name)) + } + } + const before = world() + try { + expectRefusal( + runScript('restore.sh', [firstBackup], { env: { PATH: bin } }), + 'No SHA-256 tool found', + before, + ) + } finally { + rmSync(bin, { recursive: true, force: true }) + } + }, + LONG, + ) + + // Compose itself fails on an env file it cannot find, so a check of the + // services first would blame a store that is running. + it( + 'names a missing env file, rather than a store', + () => { + const missing = join(TMP, 'no-such.env') + const before = world() + const r = runScript('restore.sh', [firstBackup], { env: { COMPOSE_ENV_FILES: missing } }) + expect(r.out).not.toContain('service is not running') + expectRefusal(r, `Cannot read ${missing}`, before) + }, + LONG, + ) + + it( + 'refuses when COMPOSE_ENV_FILES names more than one file', + () => { + const before = world() + expectRefusal( + runScript('restore.sh', [firstBackup], { + env: { COMPOSE_ENV_FILES: `${ENV_FILE},${ENV_FILE}` }, + }), + 'COMPOSE_ENV_FILES names more than one file', + before, + ) + }, + LONG, + ) + + it( + 'refuses when a store is not running', + () => { + // Compose answers that only Postgres is running; the stack itself is untouched. + const stub = stubDocker(' --services) echo postgres; exit 0 ;;') + const before = world() + try { + expectRefusal( + runScript('restore.sh', [firstBackup], { env: { PATH: stub.path } }), + "The 'clickhouse' service is not running.", + before, + ) + } finally { + stub.remove() + } + }, + LONG, + ) + + // A second run while a backup or restore holds the lock: a backup's sweep + // would delete the archive a restore is reading. + it( + 'refuses while another run holds the lock, and leaves the lock alone', + () => { + mkdirSync(LOCK) + writeFileSync(join(LOCK, 'pid'), '4242\n') + const before = world() + try { + const r = runScript('restore.sh', [firstBackup]) + expect(r.out).toContain(`rm -rf ${LOCK}`) + expectRefusal(r, `Another backup or restore holds ${LOCK} (pid 4242)`, before) + expect(readFileSync(join(LOCK, 'pid'), 'utf8')).toBe('4242\n') + } finally { + rmSync(LOCK, { recursive: true, force: true }) + } + }, + LONG, + ) + + // A restore killed part way whose RESTORE the server is still running: a + // second one would drop the database under it. + it( + 'refuses while ClickHouse is still running an earlier backup or restore', + () => { + const stub = stubDocker( + ` *"FROM system.backups"*) echo "RESTORING Disk('backups', 'clickmonk-restore-2026-10-01T041500Z-1.zip')"; exit 0 ;;`, + ) + const before = world() + try { + const r = runScript('restore.sh', [firstBackup], { env: { PATH: stub.path } }) + expect(r.out).toContain('Wait for it to finish') + expectRefusal( + r, + "ClickHouse is still running an earlier backup or restore: RESTORING Disk('backups', 'clickmonk-restore-2026-10-01T041500Z-1.zip').", + before, + ) + } finally { + stub.remove() + } + }, + LONG, + ) + + it( + 'refuses when ClickHouse cannot say whether a backup or restore is running', + () => { + const stub = stubDocker(' *"FROM system.backups"*) exit 1 ;;') + const before = world() + try { + expectRefusal( + runScript('restore.sh', [firstBackup], { env: { PATH: stub.path } }), + 'Could not ask ClickHouse whether a backup or restore is still running.', + before, + ) + } finally { + stub.remove() + } + }, + LONG, + ) + + // Without the disk the RESTORE fails, and it runs after the DROP: the one + // failure that would leave ClickHouse empty is refused before it. + it( + 'refuses a ClickHouse started without the backups disk', + () => { + const stub = stubDocker(' *"FROM system.disks"*) exit 0 ;;') + const before = world() + try { + expectRefusal( + runScript('restore.sh', [firstBackup], { env: { PATH: stub.path } }), + "ClickHouse has no disk named 'backups'", + before, + ) + } finally { + stub.remove() + } + }, + LONG, + ) + + it( + 'refuses when the image cannot say which schema version it understands', + () => { + const stub = stubDocker(' version) echo "usage:"; exit 1 ;;') + const before = world() + try { + expectRefusal( + runScript('restore.sh', [firstBackup], { env: { PATH: stub.path } }), + "Could not read the schema version this checkout's image understands.", + before, + ) + } finally { + stub.remove() + } + }, + LONG, + ) + + // Half way through an upgrade the running containers are older than the + // image `up` starts. Here a running container claims schema version 1, below + // the backup's: asked instead of the image, it would refuse the backup as newer. + it( + 'reads the schema version from the image `up` would start, not a running container', + () => { + const stub = stubDocker( + ' version) case " $* " in *" exec "*) echo "clickmonk 0.0.0 (schema version 1)"; exit 0 ;; esac ;;', + ) + const before = world() + try { + const r = runScript('restore.sh', [firstBackup], { env: { PATH: stub.path } }) + expect(r.out).not.toContain('This backup is newer than this image') + expect(r.out).toContain(`This image understands schema version ${imageSchema()}`) + expectRefusal(r, 'That is not the backup’s timestamp', before) + } finally { + stub.remove() + } + }, + LONG, + ) + + it( + 'warns, before asking, when the admin host differs from the backup’s', + () => { + const other = join(TMP, 'backup-other-host.env') + writeFileSync( + other, + readFileSync(ENV_FILE, 'utf8').replace( + /^CLICKMONK_ADMIN_HOST=.*$/m, + 'CLICKMONK_ADMIN_HOST=elsewhere.example.test', + ), + { mode: 0o600 }, + ) + const before = world() + try { + const r = runScript('restore.sh', [firstBackup], { + input: 'no\n', + env: { COMPOSE_ENV_FILES: other }, + }) + const warning = `CLICKMONK_ADMIN_HOST is 'elsewhere.example.test' here and was '${ADMIN_HOST}'` + expect(r.out).toContain(warning) + expect(r.out).not.toContain('is not the one this backup was taken with') + expect(r.out.indexOf(warning)).toBeLessThan( + r.out.indexOf('Type the backup’s timestamp to go ahead'), + ) + expectRefusal(r, 'That is not the backup’s timestamp', before) + } finally { + rmSync(other, { force: true }) + } + }, + LONG, + ) +}) + +describe('restore.sh refuses, after an interrupted restore or on a full disk', () => { + const unchanged = (): { running: string[]; redirect: string; locked: boolean } => ({ + running: running(), + redirect: startedAt('redirect'), + locked: existsSync(LOCK), + }) + + // Between a DROP and its RESTORE the clickmonk database does not exist, and + // naming it is itself an error. A rerun must still get as far as asking. + it( + 'asks nothing of the clickmonk database before the prompt', + () => { + const stub = stubDocker( + ' clickmonk) echo "Database clickmonk does not exist" >&2; exit 1 ;;', + ) + const b = unchanged() + try { + const r = runScript('restore.sh', [firstBackup], { env: { PATH: stub.path } }) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('That is not the backup’s timestamp') + expect(r.out).toContain('Nothing was changed, and nothing was stopped.') + } finally { + stub.remove() + } + expect(unchanged()).toEqual(b) + }, + LONG, + ) + + // The RESTORE runs after the DROP; a disk that fills there leaves ClickHouse + // partial with the stack stopped. + it( + 'refuses a restore the ClickHouse volume has no room for', + () => { + const stub = stubDocker( + [ + ' *"free_space FROM system.disks"*) echo 1; exit 0 ;;', + ' *"FROM system.parts"*) echo 0; exit 0 ;;', + ].join('\n'), + ) + const b = unchanged() + try { + const r = runScript('restore.sh', [firstBackup], { env: { PATH: stub.path } }) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain( + 'The ClickHouse volume has 0 MiB free, and 0 MiB more once its database is dropped', + ) + expect(r.out).toContain('Nothing was changed, and nothing was stopped.') + expect(r.out).not.toContain('Type the backup’s timestamp') + } finally { + stub.remove() + } + expect(unchanged()).toEqual(b) + }, + LONG, + ) + + // The restore needs the archive and the restored database side by side: + // two and a half times the archive, less what the DROP frees. Twice is short. + it( + 'refuses a volume with room for exactly twice the archive', + () => { + const twice = 2 * statSync(join(firstBackup, 'clickhouse.zip')).size + const stub = stubDocker( + [ + ` *"free_space FROM system.disks"*) echo ${twice}; exit 0 ;;`, + ' *"FROM system.parts"*) echo 0; exit 0 ;;', + ].join('\n'), + ) + const b = unchanged() + try { + const r = runScript('restore.sh', [firstBackup], { env: { PATH: stub.path } }) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('this restore needs about') + expect(r.out).toContain('Nothing was changed, and nothing was stopped.') + } finally { + stub.remove() + } + expect(unchanged()).toEqual(b) + }, + LONG, + ) + + // One size ClickHouse cannot give: it must not count as zero bytes to free. + it( + 'refuses when ClickHouse cannot say how much its database holds', + () => { + const stub = stubDocker(' *"FROM system.parts"*) exit 1 ;;') + const b = unchanged() + try { + const r = runScript('restore.sh', [firstBackup], { env: { PATH: stub.path } }) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain( + "Could not read the archive's size and ClickHouse's size and free space", + ) + expect(r.out).toContain('Nothing was changed, and nothing was stopped.') + } finally { + stub.remove() + } + expect(unchanged()).toEqual(b) + }, + LONG, + ) + + // Ctrl-C during the checks: without a word the prompt would be the last thing said. + it( + 'says it stopped before anything was changed when a signal arrives during the checks', + () => { + const stub = stubDocker(` --services) kill -TERM "$(cat ${LOCK}/pid)"; sleep 1; exit 1 ;;`) + const b = unchanged() + try { + const r = runScript('restore.sh', [firstBackup], { env: { PATH: stub.path } }) + expect(r.status, r.out).toBe(143) + expect(r.out).toContain('Stopped before anything was changed.') + expect(r.out).not.toContain('stopped part way') + } finally { + stub.remove() + } + expect(unchanged()).toEqual(b) + }, + LONG, + ) +}) + +describe('restore.sh and backup.sh, per install', () => { + // A refusal is true of its own run; after an interrupted restore the stores + // are not as they were, and the refusal says so. + it( + 'says what an earlier, interrupted restore left when it refuses', + () => { + writeFileSync(MARKER, 'backup=/srv/backups/2026-10-01T041500Z\nenvironment=\n') + try { + const r = runScript('restore.sh', [firstBackup], { input: 'no\n' }) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('Nothing was changed, and nothing was stopped.') + expect(r.out).toContain( + 'The stores are still as the interrupted restore of /srv/backups/2026-10-01T041500Z left them,', + ) + expect(r.out).toContain(`${ADVICE_ENV}${ROOT}/restore.sh /srv/backups/2026-10-01T041500Z`) + } finally { + rmSync(MARKER, { force: true }) + } + }, + LONG, + ) + + // Two installs from one checkout, told apart by their project names: one's + // lock and unfinished restore are nothing to the other. + it( + 'keeps one install’s lock and unfinished restore from blocking another', + () => { + mkdirSync(LOCK) + writeFileSync(join(LOCK, 'pid'), '4242\n') + writeFileSync(MARKER, 'backup=/srv/backups/2026-10-01T041500Z\nenvironment=\n') + // Named in the other install's env file, as an operator names it, not in + // the environment: the scripts must find it where Compose does. + const otherEnv = join(TMP, 'backup-elsewhere.env') + writeFileSync( + otherEnv, + `${readFileSync(ENV_FILE, 'utf8')}COMPOSE_PROJECT_NAME=clickmonk-elsewhere\n`, + { mode: 0o600 }, + ) + const other = { COMPOSE_ENV_FILES: otherEnv } + try { + // The other install has no containers, so each gets as far as its stores. + const b = runScript('backup.sh', [join(BACKUPS, 'elsewhere')], { env: other }) + expect(b.status, b.out).toBe(1) + expect(b.out).not.toContain('Another backup or restore holds') + expect(b.out).not.toContain('did not finish') + expect(b.out).toContain("The 'postgres' service is not running.") + expect(b.out).toContain(`COMPOSE_ENV_FILES=${otherEnv} docker compose up -d postgres`) + const r = runScript('restore.sh', [firstBackup], { env: other }) + expect(r.status, r.out).toBe(1) + expect(r.out).not.toContain('Another backup or restore holds') + expect(r.out).not.toContain('The stores are still as the interrupted restore') + expect(r.out).toContain("The 'postgres' service is not running.") + // This install's own are as they were, and the other's were released. + expect(readFileSync(join(LOCK, 'pid'), 'utf8')).toBe('4242\n') + expect(existsSync(MARKER)).toBe(true) + expect(existsSync(join(ROOT, '.backup-restore-clickmonk-elsewhere.lock'))).toBe(false) + } finally { + rmSync(LOCK, { recursive: true, force: true }) + rmSync(MARKER, { force: true }) + rmSync(otherEnv, { force: true }) + } + }, + LONG, + ) +}) + +describe('restore.sh, stopped before it changes anything', () => { + // A signal while `docker compose stop` still drains the four services: the + // stop goes on without the script, and a start before it finished would find + // them running, do nothing, and leave every link down once it had. + it( + 'starts again what it stopped when a signal arrives while they are still stopping', + async () => { + const real = execFileSync('sh', ['-c', 'command -v docker'], { encoding: 'utf8' }).trim() + const stub = stubDocker(` stop) kill -TERM $PPID; sleep 3; exec ${real} "$@" ;;`) + const runningBefore = running() + const links = pg('SELECT count(*) FROM links') + let r: ScriptResult + try { + r = runScript('restore.sh', [firstBackup], { + input: `${manifest(firstBackup).timestamp}\n`, + env: { PATH: stub.path }, + }) + } finally { + stub.remove() + } + expect(r.status, r.out).toBe(143) + expect(r.out).toContain('Nothing was changed. Starting caddy redirect admin worker again...') + await sleep(8_000) + expect(running()).toEqual(runningBefore) + expect(pg('SELECT count(*) FROM links')).toBe(links) + expect(existsSync(LOCK)).toBe(false) + }, + LONG, + ) + + // `stop` can stop the services and still exit non-zero. + it( + 'starts again what it stopped when the stop failed after stopping them', + () => { + const real = execFileSync('sh', ['-c', 'command -v docker'], { encoding: 'utf8' }).trim() + const stub = stubDocker(` stop) ${real} "$@"; exit 1 ;;`) + const runningBefore = running() + let r: ScriptResult + try { + r = runScript('restore.sh', [firstBackup], { + input: `${manifest(firstBackup).timestamp}\n`, + env: { PATH: stub.path }, + }) + } finally { + stub.remove() + } + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('the restore failed during the stop step') + expect(running()).toEqual(runningBefore) + expect(existsSync(LOCK)).toBe(false) + }, + LONG, + ) + + // `docker compose start` can answer 0 and leave a container stopped: success + // is every service running afterwards, and failing that the run says so. + it( + 'says so when the services are still stopped after it has started them', + () => { + const stub = stubDocker( + [ + ' *"cat >"*) echo "simulated copy failure" >&2; exit 1 ;;', + ' start) exit 0 ;;', + ].join('\n'), + ) + const runningBefore = running() + let r: ScriptResult + try { + r = runScript('restore.sh', [firstBackup], { + input: `${manifest(firstBackup).timestamp}\n`, + env: { PATH: stub.path }, + }) + } finally { + stub.remove() + } + try { + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('Could not start them (attempt 2 of 3); trying again...') + expect(r.out).toContain( + `Could not start them. Run: ${ADVICE_ENV}docker compose start caddy redirect admin worker`, + ) + expect(existsSync(LOCK)).toBe(false) + } finally { + rmSync(LOCK, { recursive: true, force: true }) + compose('start', 'caddy', 'redirect', 'admin', 'worker') + } + expect(running()).toEqual(runningBefore) + }, + LONG, + ) + + // docker compose handles INT and TERM itself, so one attempt at starting + // the services can be cut short or fail; the next one must follow. + it( + 'tries starting the services again when the first attempt fails', + () => { + const once = join(TMP, 'start-failed-once') + rmSync(once, { force: true }) + const stub = stubDocker( + [ + ' *"cat >"*) echo "simulated copy failure" >&2; exit 1 ;;', + ` start) [ -e ${once} ] || { touch ${once}; exit 1; } ;;`, + ].join('\n'), + ) + const runningBefore = running() + let r: ScriptResult + try { + r = runScript('restore.sh', [firstBackup], { + input: `${manifest(firstBackup).timestamp}\n`, + env: { PATH: stub.path }, + }) + } finally { + stub.remove() + } + try { + expect(existsSync(once), 'the first start was not attempted').toBe(true) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('Could not copy the archive into the clickhouse container') + expect(r.out).toContain('Could not start them (attempt 1 of 3); trying again...') + expect(r.out).not.toContain('Could not start them. Run:') + expect(running()).toEqual(runningBefore) + expect(existsSync(LOCK)).toBe(false) + } finally { + rmSync(once, { force: true }) + rmSync(LOCK, { recursive: true, force: true }) + } + }, + LONG, + ) + + // An operator who sends TERM while the archive is copied in, and again while + // the services are being started: the second must not abandon the trap. + it( + 'starts again what it stopped, and leaves nothing behind, when a second TERM arrives while it does', + () => { + const real = execFileSync('sh', ['-c', 'command -v docker'], { encoding: 'utf8' }).trim() + const stub = stubDocker( + [ + ` *"cat >"*) ${real} "$@"; kill -TERM $PPID; sleep 1; exit 1 ;;`, + ' start) kill -TERM $PPID ;;', + ].join('\n'), + ) + const runningBefore = running() + const links = pg('SELECT count(*) FROM links') + const clicks = ch('SELECT count() FROM clicks FINAL') + const redirectBefore = startedAt('redirect') + let r: ScriptResult + try { + r = runScript('restore.sh', [firstBackup], { + input: `${manifest(firstBackup).timestamp}\n`, + env: { PATH: stub.path }, + }) + } finally { + stub.remove() + } + try { + expect(r.status, r.out).not.toBe(0) + expect(r.out).toContain( + 'Nothing was changed. Starting caddy redirect admin worker again...', + ) + expect(r.out).not.toContain('stopped part way') + // Stopped and started again: nothing else about the install moved. + expect(startedAt('redirect')).not.toBe(redirectBefore) + expect(running()).toEqual(runningBefore) + expect(pg('SELECT count(*) FROM links')).toBe(links) + expect(ch('SELECT count() FROM clicks FINAL')).toBe(clicks) + expect(backupsDisk()).toBe('') + expect(existsSync(LOCK)).toBe(false) + } finally { + // A trap killed before its end leaves the lock. + rmSync(LOCK, { recursive: true, force: true }) + } + }, + LONG, + ) +}) + +// These change the stores and restore them again, so they come last. +describe('restore.sh, interrupted after it has changed a store', () => { + const APPS = ['admin', 'caddy', 'redirect', 'worker'] + const stamp = (): string => manifest(firstBackup).timestamp ?? '' + + /** Written by the interrupted step when it ends, if the lock is still held then. */ + const HELD = join(TMP, 'lock-held-when-the-step-ended') + + /** + * TERM to restore.sh, from inside the docker call `pattern` names, once that + * call has run; the call then goes on for two seconds, as a step still + * writing does, and records whether the lock outlived it. + */ + function interruptAt(pattern: string): { path: string; remove: () => void } { + rmSync(HELD, { force: true }) + const real = execFileSync('sh', ['-c', 'command -v docker'], { encoding: 'utf8' }).trim() + return stubDocker( + ` ${pattern}) ${real} "$@"; kill -TERM "$(cat ${LOCK}/pid)"; sleep 2; [ -d ${LOCK} ] && touch ${HELD}; exit 1 ;;`, + ) + } + + function expectLeftStopped(r: ScriptResult, state: string): void { + expect(r.status, r.out).toBe(143) + expect(r.out).toContain(`The restore stopped part way: ${state}`) + expect(r.out).not.toContain('Nothing was changed') + expect(r.out).not.toContain('Stopped before anything was changed') + const up = running() + for (const s of APPS) expect(up, s).not.toContain(s) + expect(readFileSync(MARKER, 'utf8')).toBe(markerFor(firstBackup)) + expect(existsSync(LOCK)).toBe(false) + expect(backupsDisk()).toBe('') + // The interrupted step was waited for before the lock went. + expect(existsSync(HELD), 'the lock was released while the step was still running').toBe(true) + rmSync(HELD, { force: true }) + } + + function expectFinishedByRerun(): void { + const r = runScript('restore.sh', [firstBackup], { input: `${stamp()}\n` }) + expect(r.status, r.out).toBe(0) + expect(existsSync(MARKER)).toBe(false) + expect(existsSync(LOCK)).toBe(false) + expect(running()).toEqual(ALL_SERVICES) + expect(ch('SELECT count() FROM clicks FINAL')).toBe( + manifest(firstBackup)['rows.clickhouse.clicks'], + ) + } + + it( + 'leaves the stack stopped at the spool step, refuses a backup, and finishes when run again', + () => { + const stub = interruptAt('/var/lib/clickmonk/spool') + let r: ScriptResult + try { + r = runScript('restore.sh', [firstBackup], { + input: `${stamp()}\n`, + env: { PATH: stub.path }, + }) + } finally { + stub.remove() + } + try { + expectLeftStopped(r, 'ClickHouse and Postgres are restored; the spool was being replaced') + expect(r.out).not.toContain('may still be finishing') + + const b = runScript('backup.sh', [join(BACKUPS, 'half-restored-for-real')]) + expect(b.status, b.out).toBe(1) + expect(b.out).toContain(`A restore of ${firstBackup} did not finish`) + expect(b.out).toContain(`restore.sh ${firstBackup}`) + expect(readdirSync(BACKUPS)).not.toContain('half-restored-for-real') + + // An archive a killed run left, which the rerun sweeps away, and a file + // in the redirect's state, which describes the install before the + // restore and which the rerun empties. + compose( + 'exec', + '-T', + 'clickhouse', + 'touch', + '/var/lib/clickhouse/backups/clickmonk-2000-01-01T000000Z-1.zip', + ) + compose( + '--progress', + 'quiet', + 'run', + '--rm', + '--no-deps', + '-T', + 'redirect', + 'sh', + '-c', + 'echo stale > /var/lib/clickmonk/state/probe-before-restore', + ) + expectFinishedByRerun() + expect(backupsDisk()).toBe('') + expect( + compose('exec', '-T', 'redirect', 'ls', '-A', '/var/lib/clickmonk/state'), + ).not.toContain('probe-before-restore') + } finally { + rmSync(LOCK, { recursive: true, force: true }) + } + }, + LONG, + ) + + // The first destructive statement: from here on nothing may be started again. + it( + 'leaves the stack stopped when interrupted right after the DROP, and finishes when run again', + () => { + const stub = interruptAt('*"DROP DATABASE"*') + let r: ScriptResult + try { + r = runScript('restore.sh', [firstBackup], { + input: `${stamp()}\n`, + env: { PATH: stub.path }, + }) + } finally { + stub.remove() + } + try { + expectLeftStopped(r, 'ClickHouse was being dropped and may be gone') + expectFinishedByRerun() + } finally { + rmSync(LOCK, { recursive: true, force: true }) + } + }, + LONG, + ) + + // Postgres is refilled in one transaction, by a child the trap waits for: + // the lock is held until pg_restore has ended, and the state says so. + it( + 'leaves the stack stopped at the pg_restore step, waits for it, and finishes when run again', + () => { + const stub = interruptAt('*pg_restore*') + let r: ScriptResult + try { + r = runScript('restore.sh', [firstBackup], { + input: `${stamp()}\n`, + env: { PATH: stub.path }, + }) + } finally { + stub.remove() + } + try { + expectLeftStopped( + r, + 'ClickHouse is restored; Postgres was being refilled and is either empty or complete', + ) + expectFinishedByRerun() + } finally { + rmSync(LOCK, { recursive: true, force: true }) + } + }, + LONG, + ) + + // A signal while ClickHouse runs the RESTORE: the statement is in the + // server, and the script waits for it before it deletes the archive the + // RESTORE reads or releases the lock, then says what it left. + it( + 'waits for a RESTORE a signal interrupted, and says ClickHouse is restored', + () => { + const real = execFileSync('sh', ['-c', 'command -v docker'], { encoding: 'utf8' }).trim() + rmSync(HELD, { force: true }) + const stub = stubDocker( + ` *"RESTORE DATABASE"*) kill -TERM "$(cat ${LOCK}/pid)"; ${real} "$@"; rc=$?; sleep 1; [ -d ${LOCK} ] && touch ${HELD}; exit $rc ;;`, + ) + let r: ScriptResult + try { + r = runScript('restore.sh', [firstBackup], { + input: `${stamp()}\n`, + env: { PATH: stub.path }, + }) + } finally { + stub.remove() + } + try { + expectLeftStopped( + r, + 'ClickHouse is restored; Postgres, the spool and the certificates are as they were.', + ) + expect(r.out).toContain('Waiting for the step that was running to finish...') + expect(r.out).not.toContain('may still be finishing') + expect( + ch( + "SELECT status FROM system.backups WHERE name LIKE '%clickmonk-restore-%' ORDER BY start_time DESC LIMIT 1", + ), + ).toBe('RESTORED') + expect(ch('SELECT count() FROM clicks FINAL')).toBe( + manifest(firstBackup)['rows.clickhouse.clicks'], + ) + expectFinishedByRerun() + } finally { + rmSync(LOCK, { recursive: true, force: true }) + } + }, + LONG, + ) + + // Interrupted while the stack starts: every store is done, and the run says so. + it( + 'says every store is restored when interrupted while the stack starts', + async () => { + const stub = stubDocker(` up) kill -TERM "$(cat ${LOCK}/pid)"; sleep 1; exit 1 ;;`) + let r: ScriptResult + try { + r = runScript('restore.sh', [firstBackup], { + input: `${stamp()}\n`, + env: { PATH: stub.path }, + }) + } finally { + stub.remove() + } + try { + expect(r.status, r.out).toBe(143) + expect(r.out).toContain( + 'Every store is restored and matches the manifest; the stack was being', + ) + expect(r.out).toContain(` ${ADVICE_ENV}docker compose up -d`) + expect(r.out).not.toContain('stopped part way') + expect(existsSync(MARKER)).toBe(false) + expect(existsSync(LOCK)).toBe(false) + } finally { + rmSync(LOCK, { recursive: true, force: true }) + compose('up', '-d', '--wait', ...WAIT_TIMEOUT) + } + expect(running()).toEqual(ALL_SERVICES) + }, + LONG, + ) + + // A rerun after an interruption finds the earlier run's marker. If it cannot + // write its own, the earlier one still describes the stores and must stay. + it( + 'leaves an earlier marker as it was when it cannot write its own', + () => { + const earlier = 'backup=/srv/backups/2026-10-01T041500Z\nenvironment=\n' + writeFileSync(MARKER, earlier) + // A directory where the new marker is written first: the write fails for real. + mkdirSync(`${MARKER}.partial`) + try { + const r = runScript('restore.sh', [firstBackup], { input: `${stamp()}\n` }) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain(`Could not write ${MARKER}. Nothing has been changed yet.`) + expect(r.out).not.toContain('stopped part way') + // True of this run; the stores are still as the earlier one left them. + expect(r.out).toContain( + 'The stores are still as the interrupted restore of /srv/backups/2026-10-01T041500Z left them,', + ) + expect(readFileSync(MARKER, 'utf8')).toBe(earlier) + expect(running()).toEqual(ALL_SERVICES) + expect(existsSync(LOCK)).toBe(false) + } finally { + rmSync(`${MARKER}.partial`, { recursive: true, force: true }) + rmSync(MARKER, { force: true }) + rmSync(LOCK, { recursive: true, force: true }) + } + }, + LONG, + ) + + // Every store was restored and does not match the manifest: the stores are + // not the backup's, so the marker stays and backups stay refused. + it( + 'keeps the marker, and backups refused, when the restored stores do not match the manifest', + () => { + const dir = join(BACKUPS, 'copies', 'wrong-count') + rmSync(dir, { recursive: true, force: true }) + cpSync(firstBackup, dir, { recursive: true }) + const path = join(dir, 'MANIFEST') + const edited = readFileSync(path, 'utf8').replace( + /^rows\.clickhouse\.clicks=\d+$/m, + 'rows.clickhouse.clicks=99', + ) + expect(edited).not.toBe(readFileSync(path, 'utf8')) + writeFileSync(path, edited) + try { + const r = runScript('restore.sh', [dir], { input: `${manifest(dir).timestamp}\n` }) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('the restore failed during the verification step') + expect(r.out).toContain('clicks: the backup recorded 99 rows') + expect(readFileSync(MARKER, 'utf8')).toBe(markerFor(dir)) + for (const s of APPS) expect(running(), s).not.toContain(s) + + const b = runScript('backup.sh', [join(BACKUPS, 'after-mismatch')]) + expect(b.status, b.out).toBe(1) + expect(b.out).toContain(`A restore of ${dir} did not finish`) + expect(readdirSync(BACKUPS)).not.toContain('after-mismatch') + + rmSync(MARKER, { force: true }) + expectFinishedByRerun() + } finally { + rmSync(MARKER, { force: true }) + rmSync(LOCK, { recursive: true, force: true }) + } + }, + LONG, + ) +}) + +describe('a backup restored into an empty stack', () => { + const NEW_ORDER = '/order-plz' + const orders = (): number => compose('logs', '--no-color', 'pebble').split(NEW_ORDER).length - 1 + const OLD_ROOT = join(TMP, 'issuing-root-before-restore.pem') + /** The same file as seen from a client container, which mounts TMP at /ca. */ + const OLD_ROOT_IN_CLIENT = '/ca/issuing-root-before-restore.pem' + + /** + * The two newest migrations, undone by hand before the backup, so that the + * backup is of an older schema than the image: the case of a backup taken + * before an upgrade and restored after it. Newest first. 009 is one index; + * 010 records when a domain's DNS check last passed. + */ + const UNDONE = [9, 10] + const UNDO = [ + // 010 adds one column and backfills it; dropping the column undoes both. + 'ALTER TABLE domain_dns_checks DROP COLUMN passed_at', + // 009 + 'DROP INDEX links_created_idx', + ] + /** One row each when 009's and 010's objects exist. */ + const EXISTS = [ + "SELECT count(*) FROM pg_indexes WHERE indexname = 'links_created_idx'", + "SELECT count(*) FROM information_schema.columns WHERE table_name = 'domain_dns_checks' AND column_name = 'passed_at'", + ] + + let base = 0 + let oldCookie = '' + let restoreOut = '' + let imageVersion = 0 + /** Postgres's clock just before every volume is destroyed. */ + let downAt = '' + + beforeAll(async () => { + base = reportedClicks({ cookie }) + expect(base).toBe(2) + // Read while the worker runs; the CLI is executed in its container. + imageVersion = imageSchema() + + compose('stop', 'worker') + for (let i = 0; i < 3; i++) click() + await until('three clicks in sealed segments', 60_000, () => spoolLines() === 3) + const shipped = segments().map((name) => ({ + name, + body: compose('exec', '-T', 'redirect', 'cat', `${SPOOL}/${name}`), + })) + + compose('start', 'worker') + await until( + 'the three clicks in the reports', + 120_000, + () => reportedClicks({ cookie }) === base + 3, + ) + await until('the shipped segments deleted', 60_000, () => segments().length === 0) + + // The state a crash between ClickHouse accepting a segment and the worker + // deleting it leaves: the same clicks in both places. + compose('stop', 'worker') + for (const s of shipped) { + compose( + 'exec', + '-T', + '-e', + `CM_BODY=${s.body}`, + 'redirect', + 'sh', + '-c', + 'printf "%s" "$CM_BODY" > "$0"', + `${SPOOL}/${s.name}`, + ) + } + click() + click() + await until('five click lines in sealed segments', 60_000, () => spoolLines() === 5) + + // Two migrations the backup will not have, and the image does. + for (const sql of UNDO) pg(sql) + pg(`DELETE FROM schema_migrations WHERE version IN (${UNDONE.join(', ')})`) + for (const sql of EXISTS) expect(pg(sql), sql).toBe('0') + + const backup = runScript('backup.sh', [join(BACKUPS, 'round-trip')]) + if (backup.status !== 0) throw new Error(`backup failed:\n${backup.out}`) + const dir = onlyBackupIn(join(BACKUPS, 'round-trip')) + expect(running()).not.toContain('worker') + expect(reportedClicks({ cookie })).toBe(base + 3) + // Older than the image: the guard's `<` case, not only its `=` one. + expect(Number(manifest(dir).schema_version)).toBe(imageVersion - UNDONE.length) + + copyFileSync(join(TMP, 'issuing-root.pem'), OLD_ROOT) + oldCookie = cookie + + downAt = pg('SELECT now()') + compose('down', '-v') + writeAcmeRoot() + publishZone() + compose('up', '-d', '--wait', ...WAIT_TIMEOUT) + root = writeIssuingRoot() + // Empty: the worker has migrated fresh stores to the image's version, and + // there is no link. Waited on the ledger reaching the image's version, not + // on `links` existing: that table is migration 1, and ClickHouse's tables + // come later in the same sequence. + const target = String(imageVersion) + await until('the empty stack migrated', 120_000, () => { + try { + return pg('SELECT max(version) FROM schema_migrations') === target + } catch { + return false + } + }) + expect(pg('SELECT count(*) FROM links')).toBe('0') + expect(ch('SELECT count() FROM clicks')).toBe('0') + // What the empty stack holds that the backup does not: a restore replaces + // the spool and Caddy's data, it does not add to them. + compose('exec', '-T', 'redirect', 'sh', '-c', 'echo stale > "$0/stale.bad"', SPOOL) + compose('exec', '-T', 'caddy', 'sh', '-c', 'echo stale > /data/stale-before-restore') + + const restored = runScript('restore.sh', [dir], { input: `${manifest(dir).timestamp}\n` }) + restoreOut = restored.out + if (restored.status !== 0) throw new Error(`restore failed:\n${restored.out}`) + }, 1_500_000) + + // The store and the spool first, the report last: a certificate that did not + // come back fails the report, and must not read as clicks that were lost. + it( + 'counts every click once: those ClickHouse held, those only in the spool, and those in both', + async () => { + await until( + 'the restored spool shipped', + 120_000, + () => num(ch('SELECT count() FROM clicks FINAL')) === base + 5, + ) + await until('the spool emptied', 60_000, () => segments().length === 0) + expect(num(ch('SELECT uniqExact(click_id) FROM clicks'))).toBe(base + 5) + expect(num(ch('SELECT uniqExactMerge(clicks_state) FROM clicks_hourly'))).toBe(base + 5) + expect(reportedClicks({ cookie: oldCookie, cacert: OLD_ROOT_IN_CLIENT })).toBe(base + 5) + }, + LONG, + ) + + it( + 'serves the admin host the certificate it had, and asks the new authority for none', + () => { + const r = api('GET', '/api/me', { cookie: oldCookie, cacert: OLD_ROOT_IN_CLIENT }) + // The order first: a regression that orders and serves a new certificate + // then names both symptoms, not only the failed handshake. + expect(orders()).toBe(0) + expect(r.exit, r.stderr).toBe(0) + }, + LONG, + ) + + it( + 'brings back the account, the session signed in before the backup, and the link', + () => { + expect(api('GET', '/api/me', { cookie: oldCookie, cacert: OLD_ROOT_IN_CLIENT }).status).toBe( + 200, + ) + expect(signIn(OLD_ROOT_IN_CLIENT).status).toBe(200) + const r = curl(['-4', '--max-time', '30', '-D', '-', `http://${LINK_HOST}/${SLUG}`]) + expect(r.status).toBe(302) + expect(r.headers).toMatch(new RegExp(`^location: ${TARGET}\\r?$`, 'im')) + }, + LONG, + ) + + it( + 'applies, when the worker starts, the migrations the backup was missing', + async () => { + await until( + 'migrations 9 and 10 applied again', + 60_000, + () => + pg(`SELECT count(*) FROM schema_migrations WHERE version IN (${UNDONE.join(', ')})`) === + String(UNDONE.length), + ) + for (const sql of EXISTS) expect(pg(sql), sql).toBe('1') + // The rest of the ledger is the backup's, written before the old stack + // went down. The empty stack's worker wrote a whole ledger of its own + // after that, and a restore that left it in place would list them all. + expect( + pg( + `SELECT string_agg(version::text, ',' ORDER BY version) FROM schema_migrations WHERE applied_at > '${downAt}'`, + ), + ).toBe('9,10') + }, + LONG, + ) + + it( + 'leaves the whole stack running, and nothing on the ClickHouse backups disk', + () => { + expect(running()).toEqual(ALL_SERVICES) + expect(backupsDisk()).toBe('') + }, + LONG, + ) + + it( + 'empties the spool and Caddy’s data before putting the backup’s back', + () => { + expect(compose('exec', '-T', 'redirect', 'ls', '-A', SPOOL)).not.toContain('stale.bad') + expect(compose('exec', '-T', 'caddy', 'ls', '-A', '/data')).not.toContain( + 'stale-before-restore', + ) + }, + LONG, + ) + + it( + 'tells the operator what a restore brought back that they may not want', + () => { + expect(restoreOut).toContain('A key revoked or a password changed') + expect(restoreOut).toContain('apikey list') + }, + LONG, + ) +}) + +// Every install from before the rollups existed: its ClickHouse has only the +// raw table, which is the install the upgrade instructions tell to back up +// first. Built here by undoing the rollups' migration on the running stack. +describe('a backup of an install from before the ClickHouse rollups', () => { + const ROLLUP_VERSION = 8 + const VIEWS = [ + 'clicks_hourly_mv', + 'clicks_hourly_country_mv', + 'clicks_hourly_device_mv', + 'clicks_hourly_os_mv', + 'clicks_hourly_browser_mv', + 'clicks_hourly_referrer_mv', + 'clicks_hourly_target_mv', + ] + const rollups = (): string => + ch( + "SELECT count() FROM system.tables WHERE database = 'clickmonk' AND name IN ('clicks_hourly', 'clicks_hourly_dim')", + ) + + it( + 'is taken with the raw table counted alone, and restored into the image that has them', + async () => { + // The worker migrates at start; stopped, it leaves the older schema be. + compose('stop', 'worker') + for (const v of VIEWS) ch(`DROP VIEW IF EXISTS ${v} SYNC`) + ch('DROP TABLE IF EXISTS clicks_hourly SYNC') + ch('DROP TABLE IF EXISTS clicks_hourly_dim SYNC') + pg(`DELETE FROM schema_migrations WHERE version = ${ROLLUP_VERSION}`) + expect(rollups()).toBe('0') + const clicks = ch('SELECT count() FROM clicks FINAL') + + const r = runScript('backup.sh', [join(BACKUPS, 'pre-rollups')]) + expect(r.status, r.out).toBe(0) + const dir = onlyBackupIn(join(BACKUPS, 'pre-rollups')) + const m = manifest(dir) + expect(m['rows.clickhouse.clicks']).toBe(clicks) + expect(Object.keys(m).filter((k) => k.startsWith('rows.'))).toEqual([ + 'rows.clickhouse.clicks', + ]) + + const restored = runScript('restore.sh', [dir], { input: `${m.timestamp}\n` }) + expect(restored.status, restored.out).toBe(0) + // Its verification compared the restored clicks with the manifest before + // the worker started and shipped the spool's. + expect(restored.out).toContain('Checking what was restored against the manifest...') + expect(existsSync(MARKER)).toBe(false) + // Started by the restore, the worker applies the rollups' migration. + await until('the worker to create the rollups again', 120_000, () => rollups() === '2') + await until( + 'the rollups migration in the ledger', + 60_000, + () => + pg(`SELECT count(*) FROM schema_migrations WHERE version = ${ROLLUP_VERSION}`) === '1', + ) + expect(running()).toEqual(ALL_SERVICES) + }, + LONG, + ) +}) + +// Last, because it leaves ClickHouse without its raw table. +describe('a ClickHouse without the raw clicks table', () => { + it( + 'fails the backup rather than record a manifest that could not check a restore', + () => { + ch('DROP TABLE clicks SYNC') + const dest = join(BACKUPS, 'no-clicks') + const r = runScript('backup.sh', [dest]) + expect(r.status, r.out).toBe(1) + expect(r.out).toContain('the backup failed during the ClickHouse step') + expect(r.out).toContain('Counting the rows the manifest records failed') + expect(readdirSync(dest)).toEqual([]) + expect(existsSync(LOCK)).toBe(false) + }, + LONG, + ) +}) diff --git a/test/stack/compose-config.test.ts b/test/stack/compose-config.test.ts index bfc8ae2..c67a60b 100644 --- a/test/stack/compose-config.test.ts +++ b/test/stack/compose-config.test.ts @@ -181,6 +181,28 @@ describe('what the stack publishes', () => { ) }) + // backup.sh runs `BACKUP DATABASE … TO Disk('backups', …)`, which ClickHouse + // refuses outright until a disk is allowed for it. The mount is what makes + // the backup work at all, so it is pinned here, where a two-second check + // finds it missing, rather than twenty minutes into the backup suite. + it('gives ClickHouse the disk backups are written to, read-only', () => { + const block = serviceBlock(cfg, 'clickhouse') + expect(block).toMatch( + /\n {6}- type: bind\n {8}source: [^\n]*\/clickhouse\/backup-disk\.xml\n {8}target: \/etc\/clickhouse-server\/config\.d\/backup-disk\.xml\n {8}read_only: true\n/, + ) + // Comments stripped first: a commented-out line is still + // text in the file, and a free-text match on it would stay green while + // the real server refuses every BACKUP (Code 318). + const xml = readFileSync(join(ROOT, 'clickhouse', 'backup-disk.xml'), 'utf8').replace( + //g, + '', + ) + expect(xml).toMatch(/\s*backups<\/allowed_disk>\s*<\/backups>/) + expect(xml).toMatch( + /\s*local<\/type>\s*\/var\/lib\/clickhouse\/backups\/<\/path>/, + ) + }) + it('tells the redirect to believe a forwarded address only from the stack’s own network', () => { expect(cfg).toContain('CLICKMONK_TRUSTED_PROXIES: uniquelocal,loopback') }) diff --git a/test/stack/docker-compose.tls.yml b/test/stack/docker-compose.tls.yml index 543d023..4bfd9af 100644 --- a/test/stack/docker-compose.tls.yml +++ b/test/stack/docker-compose.tls.yml @@ -35,6 +35,9 @@ services: # The browser suite's link domain: the browser clicks a link on it # and watches the click arrive in the interface. - ui.example.test + # The backup suite's link domain: clicked before a backup, and + # answering again after the restore. + - bak.example.test worker: environment: