Add continuous SQLite replica via Online Backup API - #1105
Open
vincenzopalazzo wants to merge 1 commit into
Open
vincenzopalazzo wants to merge 1 commit into
vincenzopalazzo wants to merge 1 commit into
Conversation
Mirror `{storage_dir}/ldk_node_data.sqlite` to a second file after open
and after every successful KV write/remove, Core Lightning-style.
Replica failures fail the persist. Restore by placing the replica at
`{storage_dir}/ldk_node_data.sqlite`. Seed/entropy is not included.
AI assistance: Goose (AAIF).
|
I've assigned @tnull as a reviewer! |
vincenzopalazzo
force-pushed
the
feat/sqlite-continuous-backup
branch
from
September 18, 2026 02:11
865aecc to
2f0a93e
Compare
tnull
reviewed
Sep 18, 2026
tnull
left a comment
Collaborator
There was a problem hiding this comment.
Yes, this is a feature we want and have explored quite a bit already.
We should however generally offer the option for a local SQLite-based backup, not only when SQLite is set as the storage backend. It's particular important for any kind of remote storage where you don't fully trust that the remote won't go away (e.g. VSS).
But, please see the parallel ongoing work at #692 etc.
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Operators who run a self-custodial LDK Node need a second copy of
{storage_dir}/ldk_node_data.sqlitethat stays current as the node persists. Channel monitors, payments, and the BDK wallet changeset all live in that file; losing it (disk failure, bad deploy, accidental wipe) is unrecoverable without a replica.Core Lightning already solves this for its wallet DB with
--wallet=sqlite3://primary:backup. Every wallet write is mirrored to a second SQLite file via SQLite's Online Backup API. A replica error is a wallet error, so the backup cannot silently lag.This PR brings the same model to LDK Node's SQLite store.
Builder::set_storage_dir_pathis a directory, not a SQLite URI, and:is a valid Unix path character, so we do not parseprimary:backupout of the storage path. The replica is a separate setter:Behaviour
KVStore::write/KVStore::remove, the store copies the whole primary database tobackup_pathwithConnection::backup.{storage_dir}/ldk_node_data.sqliteandbuild()as usual.This is the same two-step as CLN (commit, then backup). A crash between the primary commit and the replica copy can leave the backup one persist behind until the next successful start/persist.
The replica is a consistent full copy of the primary (schema, indexes,
user_version), not a second connection replaying SQL. Cost is O(database size) per persist.API
Builder::set_sqlite_backup_pathstorage-sqliteSqliteStore::new_with_backupfor direct store useBenchmarks
Measured locally against the KV hot path from #915 (
cargo bench --bench database --features bench -- payment_store_single), comparing stock SQLite to the same store with a continuous replica. Apple Silicon, Criterion 20 samples / 5s measurement.The 17× is
15 µsvs15 µs + copy every page. Absolute overhead here is ~250 µs on an empty store; it grows with database size. Reads are free.The #915 bench overlay used for this measurement is not in this PR.
Testing
cargo test --lib io::sqlite_store— 17 passed, including write + remove replication and rejecting a backup path that coincides with the primary file.AI assistance: Goose (AAIF).