Scheduled database backup verification

README.md

@@ -15,6 +15,7 @@ Soroban smart contracts for a decentralized utility metering and streaming proto
 - **Emergency Response** — Circuit breakers, legal freezes, velocity limits, protocol pauses
 - **Dust Sweeper** — Prunes fractional remainders from depleted streams
 - **Grant Stream** — Conservation goals trigger automatic grant matching
+- **Scheduled Backup Verification** — Restore-tested database backups with metrics, alerts, and canary rollout guidance
 
 ## Project Structure
 

docs/SCHEDULED_BACKUP_VERIFICATION.md

@@ -0,0 +1,104 @@
+# Scheduled Database Backup Verification with Restore Testing
+
+This runbook defines the system-wide backup verification pattern for services in the Utility Protocol stack. It is designed for scheduled jobs that prove backups can be restored, queried, monitored, and promoted safely without touching production state.
+
+## Architecture
+
+1. **Scheduler** starts a verification job after each database backup completes.
+2. **Artifact validation** confirms that the backup exists, is fresh, and optionally matches a SHA-256 manifest.
+3. **Isolated restore** restores the backup into an ephemeral directory, disposable database, or managed-database clone.
+4. **Integrity probes** run service-owned checks, such as schema migration status, row-count thresholds, foreign-key validation, contract indexer checkpoint validation, or application health queries.
+5. **Metrics export** writes Prometheus textfile metrics for success, duration, and last completion timestamp.
+6. **Alerting** pages the on-call owner when restore verification fails, runs longer than the service objective, or has not completed within the expected backup interval.
+7. **Blue-green/canary rollout** enables the job for one low-risk service first, then expands to 10%, 50%, and 100% of backup jobs after successful verification windows.
+
+The implementation entry point is `scripts/verify_backup_restore.sh`. The script is database-agnostic: services provide their own restore and probe commands while the shared wrapper handles freshness checks, cleanup, canary gating, and metrics.
+
+## Performance, Availability, and Security Bounds
+
+- **Critical path budget:** Backup verification must run out of band and must not add latency to request paths. Any control-plane status endpoint that reads verification results must serve cached status in less than 100ms P99.
+- **Availability target:** Verification jobs must never mutate production databases. Restore targets must be isolated and disposable to preserve the 99.99% uptime target.
+- **Security controls:** Store backup credentials in the deployment secret manager, grant restore-only access where supported, encrypt artifacts at rest and in transit, and prevent production write credentials from being mounted into verification containers.
+- **Data handling:** Use masked, least-privilege probes for regulated data. Do not print row contents or secrets in logs.
+- **Retention:** Keep verification logs and metrics for at least the longest backup retention period so auditors can correlate every retained backup with a restore test.
+
+## Example Scheduled Job
+
+```bash
+SERVICE_NAME=contract-indexer \
+ENVIRONMENT=production \
+scripts/verify_backup_restore.sh \
+  --backup /backups/contract-indexer/latest.dump \
+  --manifest /backups/contract-indexer/latest.dump.sha256 \
+  --max-age-seconds 90000 \
+  --metric-file /var/lib/node_exporter/textfile_collector/backup_restore.prom \
+  --restore-command 'pg_restore --clean --if-exists --dbname "$VERIFY_DATABASE_URL" {backup}' \
+  --probe-command 'psql "$VERIFY_DATABASE_URL" -v ON_ERROR_STOP=1 -c "select count(*) from ledger_checkpoints"' \
+  --probe-command 'psql "$VERIFY_DATABASE_URL" -v ON_ERROR_STOP=1 -c "select max(sequence) from ledger_checkpoints"'
+```
+
+For SQLite-style local artifacts, restore into the provided temporary directory:
+
+```bash
+scripts/verify_backup_restore.sh \
+  --backup ./backup.sqlite \
+  --restore-command 'cp {backup} {restore_dir}/restore.sqlite' \
+  --probe-command 'sqlite3 {restore_dir}/restore.sqlite "pragma integrity_check"'
+```
+
+## Monitoring and Alerting
+
+The script emits these Prometheus metrics when `--metric-file` is provided:
+
+| Metric | Type | Meaning |
+| --- | --- | --- |
+| `utility_backup_restore_verification_success` | gauge | `1` for the latest successful verification, `0` for failure. |
+| `utility_backup_restore_verification_duration_seconds` | gauge | Runtime of the latest verification. |
+| `utility_backup_restore_verification_last_timestamp_seconds` | gauge | Unix timestamp when the latest verification completed. |
+
+Recommended alerts:
+
+```promql
+utility_backup_restore_verification_success{environment="production"} == 0
+```
+
+```promql
+time() - utility_backup_restore_verification_last_timestamp_seconds{environment="production"} > 90000
+```
+
+```promql
+histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{route="/backup-verification/status"}[5m])) > 0.1
+```
+
+Dashboard panels should show the latest result by service, verification age, duration trend, artifact age, and failed probe logs.
+
+## Deployment Plan
+
+1. **Design review:** Confirm each service owner has documented restore commands, probe commands, secrets, and isolation boundaries.
+2. **Blue environment:** Deploy verification jobs disabled by default with `--canary-percent 0` and validate scheduling, secrets, and metrics wiring.
+3. **Canary:** Enable one non-critical service at 10% of scheduled runs. Require three consecutive successful restore tests before expansion.
+4. **Green rollout:** Increase to 50%, then 100% of services after alert noise and runtime are acceptable.
+5. **Promotion gate:** Do not mark backup automation production-ready until every critical database has at least one successful restore test in the dashboard.
+6. **Rollback:** Set the job canary percentage back to `0`, revoke temporary restore credentials, and keep existing backup creation unchanged.
+
+## Runbook
+
+### Routine Verification
+
+1. Confirm the latest backup completed.
+2. Run `scripts/verify_backup_restore.sh` with the service restore and probe commands.
+3. Confirm `utility_backup_restore_verification_success` is `1`.
+4. Attach the job log and dashboard link to the backup audit record.
+
+### Failure Response
+
+1. Treat a failed restore test as a backup-severity incident, not as a production outage unless production data is also affected.
+2. Freeze backup retention deletion for the affected service.
+3. Re-run verification with `--dry-run` to confirm command rendering and artifact selection.
+4. Restore the previous known-good artifact and compare failure mode.
+5. Escalate to the database owner when two consecutive artifacts fail or the latest valid restore point is older than the recovery-point objective.
+6. Update this runbook with the root cause, failed probe, and remediation.
+
+## Test Coverage
+
+At minimum, CI must parse the verification script with `bash -n`. Service repositories should also run a dry-run invocation and one disposable restore test for any database image they own.

scripts/verify_backup_restore.sh

@@ -0,0 +1,164 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+usage() {
+  cat <<'USAGE'
+Usage: verify_backup_restore.sh --backup PATH --restore-command COMMAND [options]
+
+Verifies a scheduled database backup by restoring it into an isolated target and
+running integrity probes. The script is database-agnostic: pass the restore and
+probe commands used by your PostgreSQL, MySQL, SQLite, or managed-database tool.
+
+Required:
+  --backup PATH              Backup artifact to verify.
+  --restore-command COMMAND  Command used to restore the artifact. The token
+                             {backup} is replaced with the backup path and
+                             {restore_dir} with a temporary restore directory.
+
+Optional:
+  --probe-command COMMAND    Command run after restore; may use {backup} and
+                             {restore_dir}. Repeat for multiple probes.
+  --metric-file PATH         Prometheus textfile output path.
+  --manifest PATH            Expected SHA-256 manifest in "<sha>  <file>" form.
+  --max-age-seconds N        Fail when the backup mtime is older than N seconds.
+  --canary-percent N         Canary gate percentage (0-100). Defaults to 100.
+  --dry-run                  Print planned actions without executing commands.
+  --help                     Show this help.
+
+Environment labels included in metrics:
+  SERVICE_NAME (default: utility_contracts)
+  ENVIRONMENT  (default: local)
+USAGE
+}
+
+backup=""
+restore_command=""
+metric_file=""
+manifest=""
+max_age_seconds=""
+canary_percent="100"
+dry_run="false"
+probe_commands=()
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --backup) backup="${2:?missing value for --backup}"; shift 2 ;;
+    --restore-command) restore_command="${2:?missing value for --restore-command}"; shift 2 ;;
+    --probe-command) probe_commands+=("${2:?missing value for --probe-command}"); shift 2 ;;
+    --metric-file) metric_file="${2:?missing value for --metric-file}"; shift 2 ;;
+    --manifest) manifest="${2:?missing value for --manifest}"; shift 2 ;;
+    --max-age-seconds) max_age_seconds="${2:?missing value for --max-age-seconds}"; shift 2 ;;
+    --canary-percent) canary_percent="${2:?missing value for --canary-percent}"; shift 2 ;;
+    --dry-run) dry_run="true"; shift ;;
+    --help) usage; exit 0 ;;
+    *) echo "Unknown argument: $1" >&2; usage >&2; exit 2 ;;
+  esac
+done
+
+if [[ -z "$backup" || -z "$restore_command" ]]; then
+  usage >&2
+  exit 2
+fi
+
+if [[ ! "$canary_percent" =~ ^[0-9]+$ ]] || (( canary_percent < 0 || canary_percent > 100 )); then
+  echo "--canary-percent must be an integer from 0 to 100" >&2
+  exit 2
+fi
+
+if [[ ! -f "$backup" ]]; then
+  echo "Backup artifact not found: $backup" >&2
+  exit 1
+fi
+
+started_at=$(date +%s)
+restore_dir=""
+
+emit_metrics() {
+  local exit_code="$1"
+  local completed_at duration labels
+  completed_at=$(date +%s)
+  duration=$((completed_at - started_at))
+  labels="service=\"${SERVICE_NAME:-utility_contracts}\",environment=\"${ENVIRONMENT:-local}\""
+
+  if [[ -n "$metric_file" ]]; then
+    mkdir -p "$(dirname "$metric_file")"
+    cat > "$metric_file" <<METRICS
+# HELP utility_backup_restore_verification_success Backup restore verification result (1 success, 0 failure).
+# TYPE utility_backup_restore_verification_success gauge
+utility_backup_restore_verification_success{$labels} $([[ "$exit_code" == "0" ]] && echo 1 || echo 0)
+# HELP utility_backup_restore_verification_duration_seconds Backup restore verification runtime.
+# TYPE utility_backup_restore_verification_duration_seconds gauge
+utility_backup_restore_verification_duration_seconds{$labels} $duration
+# HELP utility_backup_restore_verification_last_timestamp_seconds Last verification completion timestamp.
+# TYPE utility_backup_restore_verification_last_timestamp_seconds gauge
+utility_backup_restore_verification_last_timestamp_seconds{$labels} $completed_at
+METRICS
+  fi
+}
+
+cleanup() {
+  local exit_code=$?
+  [[ -n "$restore_dir" && -d "$restore_dir" ]] && rm -rf "$restore_dir"
+  emit_metrics "$exit_code"
+  exit "$exit_code"
+}
+trap cleanup EXIT
+
+render_command() {
+  local template="$1"
+  template="${template//\{backup\}/$backup}"
+  template="${template//\{restore_dir\}/$restore_dir}"
+  printf '%s' "$template"
+}
+
+run_step() {
+  local description="$1"
+  local command_template="$2"
+  local rendered
+  rendered=$(render_command "$command_template")
+  echo "==> $description"
+  echo "    $rendered"
+  if [[ "$dry_run" != "true" ]]; then
+    bash -euo pipefail -c "$rendered"
+  fi
+}
+
+if [[ "$canary_percent" == "0" ]]; then
+  echo "Canary percentage is 0; skipping restore verification by policy."
+  exit 0
+fi
+
+if [[ -n "$max_age_seconds" ]]; then
+  if [[ ! "$max_age_seconds" =~ ^[0-9]+$ ]]; then
+    echo "--max-age-seconds must be a non-negative integer" >&2
+    exit 2
+  fi
+  backup_mtime=$(stat -c %Y "$backup")
+  backup_age=$((started_at - backup_mtime))
+  if (( backup_age > max_age_seconds )); then
+    echo "Backup age ${backup_age}s exceeds maximum ${max_age_seconds}s" >&2
+    exit 1
+  fi
+fi
+
+if [[ -n "$manifest" ]]; then
+  if [[ ! -f "$manifest" ]]; then
+    echo "Manifest not found: $manifest" >&2
+    exit 1
+  fi
+  echo "==> Verifying SHA-256 manifest"
+  sha256sum --check "$manifest"
+fi
+
+restore_dir=$(mktemp -d)
+run_step "Restoring backup into isolated target" "$restore_command"
+
+if [[ ${#probe_commands[@]} -eq 0 ]]; then
+  echo "==> No probe commands supplied; restore command completion is the integrity check."
+else
+  for probe in "${probe_commands[@]}"; do
+    run_step "Running integrity probe" "$probe"
+  done
+fi
+
+echo "Backup restore verification completed successfully."