diff --git a/docs/_static/xtrabackup-process.png b/docs/_static/xtrabackup-process.png new file mode 100644 index 000000000..c08530e78 Binary files /dev/null and b/docs/_static/xtrabackup-process.png differ diff --git a/docs/differences-logical-backup.md b/docs/differences-logical-backup.md index 4eaf720d3..3bd2afdda 100644 --- a/docs/differences-logical-backup.md +++ b/docs/differences-logical-backup.md @@ -42,7 +42,7 @@ Use physical backups with XtraBackup when you need fast, non-blocking backups an !!! admonition "See also" - [How Percona XtraBackup works](how-xtrabackup-works.md) + [How Percona XtraBackup works: what happens underneath](how-xtrabackup-works-explanation.md) [Backup process overview](backup-overview.md) diff --git a/docs/features.md b/docs/features.md index a41cfadeb..ef5110b48 100644 --- a/docs/features.md +++ b/docs/features.md @@ -22,4 +22,4 @@ Percona XtraBackup automatically uses backup locks, a lightweight alternative to !!! admonition "See also" - [How Percona XtraBackup works](how-xtrabackup-works.md) + [How Percona XtraBackup works: what happens underneath](how-xtrabackup-works-explanation.md) diff --git a/docs/how-xtrabackup-works-explanation.md b/docs/how-xtrabackup-works-explanation.md new file mode 100644 index 000000000..c1d9cee91 --- /dev/null +++ b/docs/how-xtrabackup-works-explanation.md @@ -0,0 +1,269 @@ +# Percona XtraBackup internals + +Percona XtraBackup (PXB) runs three sequential phases: backup, prepare, and restore. Hot physical backup depends on log sequence numbers (LSNs), redo capture, prepare recovery, and locking. + +Commands, operational tuning, and step-by-step procedures appear in separate documents. For tasks and flags, see [How to run a backup lifecycle](how-xtrabackup-works-how-to.md). For privileges, defaults, and quick tables, see [Backup lifecycle reference](how-xtrabackup-works-reference.md). + +## What work does each phase perform? + +The following table maps each operator phase to the InnoDB-internal work in that phase: + +| Phase | Internal work | +|-------|---------------| +| Backup (hot copy) | Copy data files and capture redo for the full backup window | +| Prepare | Replay captured redo, then roll back transactions open at backup end | +| Restore | Place the prepared files in the destination `datadir` | + +Prepare uses the same pipeline as InnoDB crash recovery. Prepare runs offline on the backup tree. + +## How do data copy and redo capture interact? + +PXB copies files and redo in parallel during the backup phase. Prepare runs later on the backup directory while the server stays offline. + +![XtraBackup backup, prepare, and restore process](_static/xtrabackup-process.png) + +The following table maps the timeline to operator-visible work: + +| Stage | Server state | PXB internal work | +|-------|--------------|-------------------| +| Backup start | Online | Record checkpoint LSN. Start data copy and redo thread. | +| Backup window | Online | Copy redo while pages copy at mixed LSNs. | +| Final sync | Online | Pin sync point. Stop redo thread. Write checkpoints. | +| Prepare | Offline | Apply redo, then undo, on the backup tree. | +| Restore | Target `datadir` offline | Copy prepared files into server paths. | + +For backup artifacts, see [Index of files created by Percona XtraBackup](xtrabackup-files.md) and [XtraBackup backup files](generated-files.md). + +## What happens during the backup phase? + +The backup phase produces a directory with everything `mysqld` needs to start a server. The output includes data files for every storage engine, a captured slice of the InnoDB redo log, and metadata files. + +PXB reads the latest InnoDB redo checkpoint to record a start LSN. PXB then runs two parallel streams of work: + +* A data-file stream copies on-disk files for every storage engine in use. InnoDB tablespaces (`.ibd` files), the shared tablespace (`ibdata1`), and undo tablespaces make up most of the volume. PXB also copies files for MyISAM, CSV, MyRocks, and other recognized engines. + +* A background redo-log thread copies the InnoDB redo log from that checkpoint LSN through the end of the backup window. The thread writes the capture into `xtrabackup_logfile` in the backup directory. + +During final sync, PXB runs `FLUSH NO_WRITE_TO_BINLOG BINARY LOGS`. PXB queries `performance_schema.log_status`. When backup locks are in use, final sync runs before locks are released. The query records the checkpoint LSN, the current LSN, binary log file and position, and replica coordinates. The redo thread copies redo through the current LSN, then stops. PXB records both values in `xtrabackup_checkpoints`. The `to_lsn` field holds the checkpoint LSN from the sync point. The `last_lsn` field holds the current LSN where the redo thread stopped. + +MyRocks copies are always full copies. The MyRocks storage layout has no per-page change tracking. PXB copies every MyRocks file on each backup, including incremental backups. Plan backup storage and network capacity as repeated full RocksDB copies, not as InnoDB-style incrementals. On mixed-engine servers, InnoDB incrementals can still save space while MyRocks data is re-copied every run. For details, see [About Percona XtraBackup](about-xtrabackup.md) and [Create an incremental backup](create-incremental-backup.md). + +Dual capture lets prepare advance the copy to a single LSN. PXB also writes metadata files for prepare and restore. Files include `xtrabackup_checkpoints`, `xtrabackup_info`, `xtrabackup_binlog_info`, and `backup-my.cnf`. + +Two flags shape locking and redo retention during the backup: + +| Flag | Default | Effect | +|------|---------|--------| +| `--lock-ddl` | `ON` | Controls when the backup lock takes effect | +| `--register-redo-log-consumer` | `OFF` | Retains redo files on the server until PXB finishes reading them | + +### Why is the backup copy not yet consistent? + +InnoDB keeps writing while PXB reads files. Copied pages reflect mixed points in time. Pages copied early carry older LSNs than pages copied later. + +PXB records the redo checkpoint LSN at backup start. PXB copies InnoDB data files. A background thread copies the InnoDB redo log in parallel for the entire backup window. + +The redo capture matters for two reasons: + +* The `.ibd` page reads span real clock time. Without a continuous redo log, prepare cannot advance the copy to a single LSN. + +* InnoDB redo files wrap and recycle. The redo thread must read every record before the server overwrites the underlying file. + +Without redo capture, the directory holds physically mixed page images. The copy cannot run InnoDB recovery against a single LSN. The dataset becomes consistent during prepare, not during the file copy. + +### What lock postures does PXB use during backup? + +PXB stages locking so InnoDB data manipulation language (DML) keeps running through the bulk file copy. The lock posture during the InnoDB copy depends on `--lock-ddl`. Final sync always runs under the backup lock when locks are in use. + +With `--lock-ddl=ON` (default), PXB takes `LOCK INSTANCE FOR BACKUP` or `LOCK TABLES FOR BACKUP` before copying InnoDB files. Data definition language (DDL) stays blocked for the full run. + +| # | Lock posture | Internal work | +|---|--------------|---------------| +| 1 | Backup lock | Copy InnoDB data files and copy redo while DML continues | +| 2 | Backup lock | Copy non-InnoDB files such as `.MRG`, `.MYD`, `.MYI`, `.CSM`, `.CSV`, `.sdi`, and `.par` | +| 3 | Backup lock (short) | Run `FLUSH NO_WRITE_TO_BINLOG BINARY LOGS`, read `performance_schema.log_status`, stop the redo thread, then release locks | + +With `--lock-ddl=REDUCED`, PXB copies InnoDB files before taking the backup lock. DDL can run during the InnoDB phase. PXB tracks tablespace changes from redo and recopies affected files after the lock is taken. + +| # | Lock posture | Internal work | +|---|--------------|---------------| +| 1 | No backup lock | Copy InnoDB data files and copy redo while DML continues | +| 2 | Backup lock | Copy non-InnoDB files. Recopy tablespaces changed by concurrent DDL. | +| 3 | Backup lock (short) | Run `FLUSH NO_WRITE_TO_BINLOG BINARY LOGS`, read `performance_schema.log_status`, stop the redo thread, then release locks | + +For lock primitives, see [Reduced backup lock time](reduction-in-locks.md) and the MySQL documentation for [`LOCK INSTANCE FOR BACKUP`](https://dev.mysql.com/doc/refman/{{vers}}/en/lock-instance-for-backup.html). + +### Why does PXB prefer backup locks over FTWRL? + +Backup locks block DDL but allow InnoDB DML to continue. PXB uses redo capture and prepare for InnoDB consistency. PXB does not freeze InnoDB writes during the copy. + +`FLUSH TABLES WITH READ LOCK` (FTWRL) closes all open tables and blocks writes broadly. FTWRL waits for long-running queries before the lock can succeed. + +PXB uses backup locks when the server supports them and non-InnoDB tables need protection. PXB falls back to FTWRL when backup locks are unavailable. PXB can require FTWRL when `--slave-info` needs relay coordinates that stock MySQL does not expose through `log_status`. + +For FTWRL behavior and wait options, see [FLUSH TABLES WITH READ LOCK option](flush-tables-with-read-lock.md). + +### How do `--lock-ddl=ON` and `--lock-ddl=REDUCED` compare? + +The `--lock-ddl` flag controls when PXB takes the backup lock that blocks DDL. + +The following table compares the two modes: + +| Value | Backup lock taken | DDL impact | +|-------|-------------------|------------| +| `ON` (default) | Backup start | DDL is restricted for the full backup window | +| `REDUCED` | After the InnoDB copy finishes | DDL is restricted for a shorter window. DDL runs concurrently with the InnoDB copy. PXB tracks tablespace changes from redo and recopies affected tablespaces when the lock is taken. | + +InnoDB DML continues through the main copy under both modes. + +### How does `--register-redo-log-consumer` retain redo? + +On a high-traffic host, the server can recycle redo files before PXB finishes reading them. A purged redo file makes the backup unusable. + +The `--register-redo-log-consumer` flag registers PXB as a redo consumer. The server retains a redo file until PXB has copied that file. + +Review the following trade-offs before you enable the flag: + +* Redo files stay on disk longer. Peak redo size rises during the run. + +* Free space can drop sharply on a long backup of a write-heavy server. + +* The server can pause writes briefly when the consumer advances. + +!!! warning + + Monitor free disk during the backup. Abort with Ctrl+C or `SIGTERM` to `xtrabackup` if disk space reaches a critical threshold. The consumer releases on abort. The server can then purge redo. + +The default value is `OFF`. Enable the flag only when spare disk space is available. + +### When does PXB skip backup locks? + +PXB skips backup locks only when every table in every schema uses InnoDB. The check includes the `mysql` system schema. Most installations still hold CSV or MyISAM tables in `mysql`, such as `general_log`. PXB takes backup locks in practice. + +Replication coordinate handling depends on the server build: + +* Percona Server for MySQL can fold relay coordinates into `log_status`. The fold reduces the locks needed for `--slave-info`. + +* Stock MySQL can require `FLUSH TABLES WITH READ LOCK` for some relay-position scenarios. + +### How do incremental backups fit? + +InnoDB incrementals rely on LSNs stored in `xtrabackup_checkpoints`. Each InnoDB page carries an LSN. PXB copies pages whose LSN is newer than the previous backup `to_lsn`. Prepare merges `.delta` files from each incremental into the base backup tree before the final redo and undo passes. + +Long incremental chains increase prepare time and failure risk. A broken link in the chain invalidates downstream incrementals. Periodic full backups limit chain depth. + +The `--page-tracking` flag can skip full page scans when the `mysqlbackup` component tracks changed pages. MyRocks has no InnoDB-style incrementals. See the MyRocks paragraph under the backup phase section. + +For procedures and chain management, see [Create an incremental backup](create-incremental-backup.md), [Prepare an incremental backup](prepare-incremental-backup.md), [Backup example](example-full-incr.md), and [Take an incremental backup using page tracking](page-tracking.md). + +### How are binlog and replication coordinates captured? + +Final sync reads `performance_schema.log_status`. PXB records binary log file, position, and Global Transaction Identifier (GTID) data in backup metadata. + +The following artifacts carry replication-related coordinates: + +* `xtrabackup_binlog_info`: binary log position at backup end. Used for point-in-time recovery. + +* Standard error (STDERR) output: binlog coordinates printed when the backup completes. + +* `xtrabackup_slave_info`: written when `--slave-info` is set. Used to configure a replica. + +PXB captures InnoDB consistency through redo and prepare. Binary log metadata extends recovery beyond the physical backup to point-in-time and replication workflows. + +For replication backup workflows and point-in-time recovery, see [Make backups in replication environments](make-backup-in-replication-env.md), [How to set up a replica for replication in 6 simple steps with Percona XtraBackup](set-up-replication.md), [Point-in-time recovery](point-in-time-recovery.md), and [Working with binary logs](working-with-binary-logs.md). + +## What happens during the prepare phase? + +`xtrabackup --prepare` runs two coordinated passes against the backup directory. + +Roll forward applies the captured redo to the copied tablespaces. The pass is physical. Each redo record describes a page-level change. InnoDB reapplies that change to the page image on disk. + +Redo replay alone does not produce a transactionally consistent backup. InnoDB can log and flush page changes before a transaction commits. Those changes can appear in the captured redo at backup end. + +Roll back removes transactions that had not committed at the backup end LSN. Redo replay can have written those changes to pages. Prepare must reverse the changes at the logical level. Both passes are required. Redo converges mixed-time page copies to one LSN. Undo strips uncommitted work that redo restored to disk. + +Roll back relies on two structures inside the tablespaces: + +* Undo records describe how to reverse the row-level effects of each open transaction. + +* Serialized Dictionary Information (SDI) carries table and index definitions inside each tablespace. SDI supplies the schema context that older `.frm` files once provided to rollback. + +### Why are InnoDB and MyISAM consistent after prepare? + +InnoDB tablespaces are copied over a long window. Each page can reflect a different LSN when the copy finishes. Prepare uses redo to roll all InnoDB pages forward to the backup end LSN. Prepare uses undo to remove work from transactions that had not committed at that point. + +MyISAM and other non-InnoDB engines do not use redo for recovery. PXB copies their files under a backup lock near final sync. The lock pins a stable file snapshot that aligns with the backup sync point. + +After prepare completes, InnoDB data matches the sync-point LSN. Non-InnoDB data matches the same logical moment. Open transactions are stripped. A restored server sees a coherent dataset across storage engines. + +## What happens during the restore phase? + +Restore places a prepared backup tree into a target `datadir`. PXB reads destination paths from the server configuration. Paths include `datadir`, `innodb_data_home_dir`, `innodb_data_file_path`, and `innodb_log_group_home_dir`. + +`--copy-back` and `--move-back` place files in a fixed order so `mysqld` can start cleanly: + +* Undo tablespaces and the InnoDB system tablespace (`ibdata`, and related files from `innodb_data_file_path`) first. + +* Binary log files, when present in the backup. + +* Remaining files in parallel, including InnoDB tablespaces (`.ibd`), non-InnoDB tables (`.MRG`, `.MYD`, `.MYI`, `.CSM`, `.CSV`, `.sdi`, `.par`), and InnoDB redo files under `#innodb_redo`. + +* MyRocks data and write-ahead log (WAL) files last, when the backup includes MyRocks. + +File attributes survive the copy. A backup must reach a prepared state before restore. For validation and restore commands, see [How to run a backup lifecycle](how-xtrabackup-works-how-to.md) and [Restore full, incremental, and compressed backups](restore-a-backup.md). + +## How do streaming and cloud backups differ? + +Streaming and `xbcloud` workflows use the same backup and prepare mechanics as a local backup. Compression, encryption, and transport change how bytes are written and stored. LSN capture and prepare recovery stay the same. Slow networks extend the final-sync phase. The extension keeps the backup lock held through the binary log flush and `log_status` read. + +## Related documentation + +Lifecycle and overview: + +* [How to run a backup lifecycle](how-xtrabackup-works-how-to.md) + +* [Backup lifecycle reference](how-xtrabackup-works-reference.md) + +* [Backup process overview](backup-overview.md) + +Artifacts and generated files: + +* [Index of files created by Percona XtraBackup](xtrabackup-files.md) + +* [XtraBackup backup files](generated-files.md) + +Incrementals and examples: + +* [Create an incremental backup](create-incremental-backup.md) + +* [Prepare an incremental backup](prepare-incremental-backup.md) + +* [Backup example](example-full-incr.md) + +Replication and point-in-time recovery: + +* [Make backups in replication environments](make-backup-in-replication-env.md) + +* [Point-in-time recovery](point-in-time-recovery.md) + +Performance and resource use: + +* [Throttling backups](throttling-backups.md) + +* [Smart memory estimation](smart-memory-estimation.md) + +Streaming, compression, and encryption: + +* [Take a streaming backup](take-streaming-backup.md) + +* [xbstream binary overview](xbstream-binary-overview.md) + +* [Encrypt backups](encrypt-backups.md) + +* [xbcloud binary overview](xbcloud-binary-overview.md) + +Verification and errors: + +* [Verify backups with replication and pt-checksum](verify-backup.md) + +* [xtrabackup exit codes](xtrabackup-exit-codes.md) diff --git a/docs/how-xtrabackup-works-how-to.md b/docs/how-xtrabackup-works-how-to.md new file mode 100644 index 000000000..52f719919 --- /dev/null +++ b/docs/how-xtrabackup-works-how-to.md @@ -0,0 +1,142 @@ +# How to run a backup lifecycle + +Percona XtraBackup (PXB) runs three sequential phases: backup, prepare, and restore. Each section covers one phase and the operational flags that shape locking, redo retention, and recovery output. + +Apply the following principles across every lifecycle phase: + +* Grant only the privileges that the workflow requires. + +* Run backup, prepare, and restore in sequence. + +* Capture binary log coordinates for recovery and replication. + +* Choose operational flags such as `--lock-ddl` and `--register-redo-log-consumer` to match the workload and lock tolerance. + +## How do you grant the minimum privileges? + +PXB needs database privileges that match the locking operations and metadata reads in the workflow. + +Complete the following privilege steps: +{ .power-number } + +1. Grant `BACKUP_ADMIN` so PXB can read `performance_schema.log_status`. + +2. Confirm `BACKUP_ADMIN` allows PXB to take `LOCK INSTANCE FOR BACKUP` or `LOCK TABLES FOR BACKUP`. + +3. Add `RELOAD`, `LOCK TABLES`, or `REPLICATION CLIENT` when the workflow needs extra capabilities. These include `FLUSH NO_WRITE_TO_BINLOG BINARY LOGS` at final sync, `FLUSH TABLES WITH READ LOCK`, and the `--slave-info` option. + +For privilege lists and worked examples, see [Connection and privileges needed](privileges.md). + +## How do you run a backup and capture binary log coordinates? + +Capture binary log coordinates during the backup phase to support recovery and replica bootstrap. + +Complete the following backup steps: +{ .power-number } + +1. Run `xtrabackup` against the target directory with the standard options. + +2. Redirect standard error (STDERR) to a log file. PXB writes binary log coordinates to STDERR. Example: `xtrabackup ... 2> backupout.log`. + +3. Confirm the command returns exit code `0`. + +For the artifact list a backup produces, see [Index of files created by Percona XtraBackup](xtrabackup-files.md). + +## How do you choose a `--lock-ddl` mode? + +PXB uses backup locks to copy non-InnoDB files without blocking InnoDB data manipulation language (DML). The server must support backup locks. + +The following table compares the two `--lock-ddl` modes: + +| Option | Effect | +|--------|--------| +| `--lock-ddl=ON` (default) | Takes the backup lock at start. DDL stays blocked for the full run unless the workflow changes. | +| `--lock-ddl=REDUCED` | Takes the lock after InnoDB is copied. DDL blocks for a shorter window. The mode accepts DDL contention during the InnoDB phase in exchange for a shorter DDL block. | + +!!! warning + + Both modes block schema changes (DDL) for part of the backup window. Schedule DDL outside the backup, or accept the contention. + +For a compact comparison, see [Backup lifecycle reference](how-xtrabackup-works-reference.md). + +## When should you enable `--register-redo-log-consumer`? + +Enable `--register-redo-log-consumer` when heavy write throughput risks purging redo before PXB finishes reading the redo stream. The flag prevents failures caused by missing redo. + +Review the following checks before you enable the flag: + +* Plan for redo bloat and possible disk exhaustion. The server retains redo longer. + +* Monitor free space and I/O throughout the run. + +* Abort the backup with Ctrl+C or `SIGTERM` to the `xtrabackup` process if disk space reaches a critical threshold. The consumer releases. The server purges redo. The operator can free space and retry. + +The default value is `OFF`. Enable the flag only when spare disk space is available. + +## How do you prepare the backup with `--prepare`? + +The prepare phase applies redo and rolls back uncommitted transactions. The result is a consistent snapshot. + +Complete the following prepare steps: +{ .power-number } + +1. Run `xtrabackup --prepare` with `--target-dir` set to the backup directory. + +2. For large or incremental backups, tune prepare performance with the following options: + + * Raise `--use-memory` to between 1 GB and 2 GB when random-access memory (RAM) allows. The default value is small. + + * Pass `--parallel` to apply `.delta` files in parallel on incremental backups. Parallel apply is available from 8.4.0-3. The flag does not parallelize first-pass redo on a full backup. + +The following command shows a tuned prepare run: + +```bash +xtrabackup --prepare --use-memory=2G --parallel=4 --target-dir=/data/backups/ +``` + +!!! warning + + Leave RAM headroom for the OS and `mysqld` when prepare runs on the same host as production. Insufficient headroom triggers out-of-memory (OOM) kills. + +For flag references, see [`--use-memory`](xtrabackup-option-reference.md#use-memory) and [`--parallel`](xtrabackup-option-reference.md#parallel). + +## How do you restore with `--copy-back` or `--move-back`? + +The restore phase places prepared backup files into the destination `datadir` for `mysqld` startup. + +Complete the following restore steps: +{ .power-number } + +1. Stop `mysqld` if the process owns the destination `datadir`. + +2. Run `xtrabackup --copy-back` or `xtrabackup --move-back` with `--target-dir` set to the prepared backup. PXB reads destination paths from the configuration. Paths include `datadir`, InnoDB paths, and log paths. + +3. Set ownership and permissions before starting `mysqld`. Example: `chown -R mysql:mysql /var/lib/mysql`. Backup files belong to the user that ran `xtrabackup`. The server expects the `mysql` user. + +4. Start `mysqld`. + +!!! warning + + `--move-back` destroys the original backup directory. The command moves files instead of copying them. Use `--move-back` only when disk space is constrained. Confirm a separate backup copy is available. + +For full restore flows covering full, incremental, and compressed backups, see [Restore full, incremental, and compressed backups](restore-a-backup.md). + +## How do you stream backups or upload to object storage? + +Streaming backups and cloud uploads follow the same lifecycle as a local backup. Slow networks extend the final-sync phase. The extension keeps the backup lock held through the binary log flush and `log_status` read. + +Plan streaming and upload runs with the following guidance: + +* Run streaming backups inside a maintenance window. + +* Size network bandwidth and destination throughput to keep the final-sync phase short. + +* Confirm the destination object store accepts the backup payload size. + +For workflow details, see the following references: + +* [Take a streaming backup](take-streaming-backup.md) + +* [xbcloud binary overview](xbcloud-binary-overview.md) + +* [Percona XtraBackup internals](how-xtrabackup-works-explanation.md) diff --git a/docs/how-xtrabackup-works-reference.md b/docs/how-xtrabackup-works-reference.md new file mode 100644 index 000000000..2e9684f09 --- /dev/null +++ b/docs/how-xtrabackup-works-reference.md @@ -0,0 +1,97 @@ +# Backup lifecycle reference + +Quick-scan facts for privileges, phases, defaults, and behavior. For internals, see [Percona XtraBackup internals](how-xtrabackup-works-explanation.md). For commands and flags, see [How to run a backup lifecycle](how-xtrabackup-works-how-to.md). + +## What privileges does PXB require? + +Percona XtraBackup (PXB) typically needs the privileges in the following table: + +| Privilege | Typical use | +|-----------|-------------| +| `BACKUP_ADMIN` | Read `performance_schema.log_status`. Use `LOCK INSTANCE FOR BACKUP` or `LOCK TABLES FOR BACKUP`. | +| `RELOAD` | Run `FLUSH NO_WRITE_TO_BINLOG BINARY LOGS` at final sync. Run `FLUSH TABLES WITH READ LOCK` when backup locks are unavailable. | +| `LOCK TABLES` | Issue table-level locks in some workflows. | +| `REPLICATION CLIENT` | Use `--slave-info` and related replication metadata. | + +[Connection and privileges needed](privileges.md) has the full list. + +## What are the three operator-facing phases? + +| Step | Phase | What happens | +|------|-------|----------------| +| 1 | Backup (hot copy) | Copy data files. Copy redo for the whole run. | +| 2 | Prepare | Replay redo. Roll back uncommitted work. | +| 3 | Restore | Run `--copy-back` or `--move-back` into `datadir`. | + +## What are the in-backup sub-phases? + +Final sync is the same in both modes. PXB runs `FLUSH NO_WRITE_TO_BINLOG BINARY LOGS`. PXB reads `performance_schema.log_status`. PXB stops the redo thread and releases locks. With `--lock-ddl=ON` (default), the backup lock is held from backup start through final sync. + +| # | `--lock-ddl=ON` | `--lock-ddl=REDUCED` | +|---|-----------------|----------------------| +| 1 | Backup lock: copy InnoDB files and redo | No backup lock: copy InnoDB files and redo | +| 2 | Backup lock: copy non-InnoDB files | Backup lock: copy non-InnoDB files. Recopy DDL-affected tablespaces. | +| 3 | Backup lock: final sync, then release | Backup lock: final sync, then release | + +Compare backup locks with `FLUSH TABLES WITH READ LOCK` (FTWRL): [Percona Server backup locks](https://docs.percona.com/percona-server/innovation-release/backup-locks.html). +MySQL {{vers}}: [`LOCK INSTANCE FOR BACKUP`](https://dev.mysql.com/doc/refman/{{vers}}/en/lock-instance-for-backup.html). + +## What are the `--lock-ddl` values? + +| Value | When the backup lock is taken (typical) | +|-------|----------------------------------------| +| `ON` (default) | Backup start | +| `REDUCED` | After InnoDB data copy finishes | + +`ON` blocks data definition language (DDL) for the whole window from the beginning. `REDUCED` delays the block until InnoDB is copied. InnoDB data manipulation language (DML) keeps running through the main copy under both modes. + +## What does `--register-redo-log-consumer` do? + +| Field | Detail | +|-------|--------| +| Default | Off | +| Does | Registers PXB as a redo consumer. The server retains a redo file until PXB copies that file. | +| Cost | Redo can pile up. Disk use may spike. | +| Writes | Server may pause writes briefly while the consumer advances. | + +## What prepare options affect performance? + +| Option | Notes | +|--------|--------| +| `--use-memory` | Random-access memory (RAM) for prepare only (default 100 MB). Larger values often help. | +| `--parallel` | 8.4.0-3 and later: parallel `.delta` apply for incrementals. Not parallel full-backup redo replay. | + +Full flags: [xtrabackup option reference](xtrabackup-option-reference.md). + +## What does restore copy back? + +Restore reads paths from `my.cnf`. Paths include `datadir`, `innodb_data_home_dir`, `innodb_data_file_path`, and `innodb_log_group_home_dir`. + +Restore places files in the following order: + +* Undo and system tablespaces, then binary logs, then remaining files in parallel (`.ibd`, non-InnoDB tables, `#innodb_redo`), then MyRocks when present. + +* File attributes are preserved. + +* A successful backup prints binlog coordinates to standard error (STDERR). Redirect STDERR when the workflow must capture coordinates. + +## When are backup locks skipped? + +Locks stay off only if every table in every schema uses InnoDB. The check includes `mysql`. Commonly `mysql` still holds CSV or MyISAM tables, such as `general_log`. PXB takes backup locks in most installations. + +| Server | Notes | +|--------|--------| +| Percona Server for MySQL {{vers}} | `log_status` may carry relay coordinates. `--slave-info` can skip extra locks. | +| Standard MySQL {{vers}} | May need `FLUSH TABLES WITH READ LOCK` with `--slave-info` when relay position is required. | + +## How do cloud and streaming backups differ? + +Cloud and streaming backups use the same phases. Slow networks extend final sync and can lengthen the backup lock window. See [Take a streaming backup](take-streaming-backup.md) and [xbcloud binary overview](xbcloud-binary-overview.md). + +## See also + +* [Index of files created by Percona XtraBackup](xtrabackup-files.md) + +* [Restore full, incremental, and compressed backups](restore-a-backup.md) + +* [Create a full backup](create-full-backup.md) diff --git a/docs/how-xtrabackup-works.md b/docs/how-xtrabackup-works.md deleted file mode 100644 index aa2250fd5..000000000 --- a/docs/how-xtrabackup-works.md +++ /dev/null @@ -1,114 +0,0 @@ -# How Percona XtraBackup works - -Percona XtraBackup is based on InnoDB’s crash-recovery functionality. -It copies your InnoDB data files, which results in data that is internally -inconsistent; but then it performs crash recovery on the files to make them a -consistent, usable database again. - -This works because InnoDB maintains a redo log, also called the transaction -log. This contains a record of every change to InnoDB data. When InnoDB -starts, it inspects the data files and the transaction log, and performs two -steps. It applies committed transaction log entries to the data files, and it -performs an undo operation on any transactions that modified data but did not -commit. - -The `--register-redo-log-consumer` parameter is disabled by default. When enabled, this parameter lets Percona XtraBackup register as a redo log consumer at the start of the backup. The server does not remove a redo log that Percona XtraBackup (the consumer) has not yet copied. The consumer reads the redo log and manually advances the log sequence number (LSN). The server blocks the writes during the process. Based on the redo log consumption, the server determines when it can purge the log. - -Percona XtraBackup remembers the LSN when it starts, and then copies the data files. The operation takes time, and the files may change, then LSN reflects the state of the database at different points in time. Percona XtraBackup also runs a background process that watches the transaction log files, and copies any changes. Percona XtraBackup does this continually. The transaction logs are written in a round-robin fashion, and can be reused. - -Percona XtraBackup uses [Backup locks :octicons-link-external-16:](https://docs.percona.com/percona-server/innovation-release/backup-locks.html) where available as a lightweight alternative to `FLUSH TABLES WITH READ LOCK`. MySQL {{vers}} allows acquiring an instance level backup lock via the `LOCK INSTANCE FOR BACKUP` statement. - -Locking is only done for MyISAM and other non-InnoDB tables -after Percona XtraBackup finishes backing up all InnoDB/XtraDB data and -logs. Percona XtraBackup uses this automatically to copy non-InnoDB data to -avoid blocking DML queries that modify InnoDB tables. - -!!! important - - The `BACKUP_ADMIN` privilege is required to query the - `performance_schema_log_status` for either `LOCK - INSTANCE FOR BACKUP` or `LOCK TABLES FOR BACKUP`. - -xtrabackup tries to avoid backup locks and `FLUSH TABLES WITH READ LOCK` -when the instance contains only InnoDB tables. In this case, xtrabackup -obtains binary log coordinates from `performance_schema.log_status`. `FLUSH -TABLES WITH READ LOCK` is still required in MySQL {{vers}} when xtrabackup is -started with the `--slave-info`. The `log_status` table in Percona -Server for MySQL {{vers}} is extended to include the relay log coordinates, so no locks are -needed even with the `--slave-info` option. - -!!! admonition "See also" - - [MySQL Documentation: LOCK INSTANCE FOR BACKUP :octicons-link-external-16:](https://dev.mysql.com/doc/refman/{{vers}}/en/lock-instance-for-backup.html) - -When backup locks are supported by the server, xtrabackup first copies -InnoDB data, runs the `LOCK TABLES FOR BACKUP` and then copies the MyISAM -tables. Once this is done, the backup of the files will -begin. It will backup .frm, .MRG, .MYD, .MYI, .CSM, -.CSV, `.sdi` and `.par` files. - -After that xtrabackup will use `LOCK BINLOG FOR BACKUP` to block all -operations that might change either binary log position or -`Exec_Source_Log_Pos` or `Exec_Gtid_Set` (i.e. source binary log coordinates -corresponding to the current SQL thread state on a replication replica) as -reported by `SHOW BINARY LOG STATUS` or `SHOW REPLICA STATUS`. xtrabackup will then finish copying -the REDO log files and fetch the binary log coordinates. After this is completed -xtrabackup will unlock the binary log and tables. - -Finally, the binary log position will be printed to `STDERR` and xtrabackup -will exit returning 0 if all went OK. - -Note that the `STDERR` of xtrabackup is not written in any file. You will -have to redirect it to a file, for example, `xtrabackup OPTIONS 2> backupout.log`. - -It will also create the following files in the -directory of the backup. - -## Prepare phase - -This phase involves two primary operations: applying the redo log and the undo log. - -### Redo Log Application (Physical Operation) - -XtraBackup directly applies changes recorded in the redo log to specific page offsets within the tablespace (IBD file). This is a physical operation, meaning it works at the page level, without regard for rows or transactions. - -It's important to understand that the redo log might contain uncommitted transactions, as the server can flush or write these to the log. Therefore, the redo log application doesn't inherently guarantee transactional consistency. - -### Undo Log Application (Logical Operation) - -Following the redo log application, XtraBackup uses the undo log to logically roll back changes from any uncommitted transactions present in the redo log. -Undo log records are of two types: `INSERT` and `UPDATE`. Each record contains a `table_id`, which XtraBackup uses to locate the table definition. - -To perform the rollback, XtraBackup initializes the InnoDB engine and data dictionary, then uses Serialized Dictionary Information (SDI) from the tablespace (a JSON representation of the table) to parse index pages and apply undo operations. - -Tables are loaded as evictable, and XtraBackup scans data dictionary indexes to relate `table_id` to tablespace, which is used during rollback. User tables are loaded only when needed for rollback. This design significantly reduces memory and I/O usage, speeds up the `--prepare` phase, and improves Percona XtraDB Cluster SST performance. - -### Achieving Consistency: Redo, Undo, and MyISAM - -The `--prepare` phase ensures that InnoDB tables are rolled forward to the point where the backup completed, not rolled back to where it began. This point aligns with the time a `FLUSH TABLES WITH READ LOCK` was taken, which is crucial for maintaining consistency with MyISAM tables. - -Therefore, after the `--prepare` phase, both InnoDB and MyISAM tables are eventually consistent with each other. - -## Restore a backup - -To restore a backup with xtrabackup you can use the `--copy-back` or -`--move-back` options. - -xtrabackup will read from the `my.cnf` the variables datadir, -innodb_data_home_dir, innodb_data_file_path, -innodb_log_group_home_dir and check that the directories exist. - -It will copy the MyISAM tables, indexes, etc. (.MRG, .MYD, -.MYI, .CSM, .CSV, `.sdi`, -and `par` files) first, InnoDB tables and indexes next and the log files at -last. It will preserve file’s attributes when copying them, you may have to -change the files’ ownership to `mysql` before starting the database server, as -they will be owned by the user who created the backup. - -Alternatively, the `--move-back` option may be used to -restore a backup. This option is similar to `--copy-back` -with the only difference that instead of copying files it moves them to their -target locations. As this option removes backup files, it must be used with -caution. It is useful in cases when there is not enough free disk space to hold -both data files and their backup copies. - diff --git a/docs/index-contents.md b/docs/index-contents.md index 9199be759..6e754523c 100644 --- a/docs/index-contents.md +++ b/docs/index-contents.md @@ -21,6 +21,7 @@ - [Dump schema for all tables](dump-schema.md) - [Encrypt backups](encrypt-backups.md) - [Encrypted InnoDB tablespace backups](encrypted-innodb-tablespace-backups.md) + - [How Percona XtraBackup works: what happens underneath](how-xtrabackup-works-explanation.md) - [Exponential backoff](xbcloud-exbackoff.md) - [FIFO data sink](xbcloud-binary-fifo-datasink.md) - [Files in the DEB package](apt-files.md) @@ -29,7 +30,7 @@ - [Frequently asked questions](faq.md) - [Get help from Percona](get-help.md) - [Glossary](glossary.md) - - [How Percona XtraBackup works](how-xtrabackup-works.md) + - [How-to: backup lifecycle operations](how-xtrabackup-works-how-to.md) - [How to create a new (or repair a broken) GTID-based replica](create-gtid-replica.md) - [How to set up a replica for replication in 6 simple steps with Percona XtraBackup](set-up-replication.md) - [Improved log statements](log-enhancements.md) @@ -55,6 +56,7 @@ - [Prepare an individual partitions backup](prepare-individual-partitions-backup.md) - [Quickstart Overview](quickstart-overview.md) - [Reduced backup lock time](reduction-in-locks.md) + - [Reference: backup lifecycle](how-xtrabackup-works-reference.md) - [Restore a partial backup](restore-partial-backup.md) - [Restore full, incremental, and compressed backups](restore-a-backup.md) - [Restore single tables between databases](restore-individual-tables.md) diff --git a/docs/reduction-in-locks.md b/docs/reduction-in-locks.md index 2a7b08d12..458c72c5c 100644 --- a/docs/reduction-in-locks.md +++ b/docs/reduction-in-locks.md @@ -115,5 +115,5 @@ The following operations are performed to ensure data remains intact and consist 5. Replace the file that matches the name without the `.new` extension using the `.new` extension. 6. Replace the `meta` and `delta` files that match the name without the `.new` in the name using `.new.meta` and `.new.delta`. -After `Phase 3` is completed, the regular [crash recovery](./how-xtrabackup-works.md) starts. +After `Phase 3` is completed, the regular [crash recovery](./how-xtrabackup-works-explanation.md) starts. diff --git a/docs/server-backup-version-comparison.md b/docs/server-backup-version-comparison.md index 2bc07617b..3edceac40 100644 --- a/docs/server-backup-version-comparison.md +++ b/docs/server-backup-version-comparison.md @@ -7,7 +7,7 @@ XtraBackup version that is equal to your source server major version. This means !!! admonition "See also" - [How XtraBackup works](how-xtrabackup-works.md) + [How Percona XtraBackup works: what happens underneath](how-xtrabackup-works-explanation.md) The `--no-server-version-check` option performs the following test. Before the backup/prepare starts, XtraBackup compares the source server version to diff --git a/mkdocs-base.yml b/mkdocs-base.yml index 22338f531..9b6b20feb 100644 --- a/mkdocs-base.yml +++ b/mkdocs-base.yml @@ -184,7 +184,9 @@ nav: - How Percona XtraBackup works: - about-xtrabackup.md - features.md - - how-xtrabackup-works.md + - 'What happens underneath': how-xtrabackup-works-explanation.md + - 'How-to: backup lifecycle operations': how-xtrabackup-works-how-to.md + - 'Reference: backup lifecycle': how-xtrabackup-works-reference.md - differences-logical-backup.md - backup-overview.md - backup-strategy.md