Skip to content

Backups: survive a day-long first upload over a home line - #39

Merged
davior merged 1 commit into
mainfrom
claude/wonderful-goldberg-ifbtdt
Oct 22, 2026
Merged

davior merged 1 commit into
mainfrom
claude/wonderful-goldberg-ifbtdt

Conversation

@davior

@davior davior commented Oct 22, 2026

Copy link
Copy Markdown
Owner

Why

The first real backup run died partway through uploading media:

subprocess ssh: Timeout, server geckopico.synology.me not responding.
...
Fatal: unable to save snapshot: ssh command exited: exit status 255

ssh gave up after 90 s of silence (ServerAliveInterval 30 × ServerAliveCountMax 3). Five parallel restic connections were saturating the home NAS and its line for an upload with a ~13 h ETA, so a stall that long was close to certain. restic can't reconnect once its ssh subprocess exits, so the whole run was lost. Notes' small hourly backups over the same link run cleanly.

Changes

  • entrypoint.sh: ssh now waits through 5 minutes of real silence (ServerAliveInterval 15 × ServerAliveCountMax 20) instead of 90 seconds, and sets TCPKeepAlive yes.
  • backup.sh:
    • One run at a time (flock in the container's tmpfs). An hourly tick during a long run prints previous run still going, skipping instead of failing on restic's repository lock.
    • restic unlock first. Without --remove-all it only removes stale locks, so a crashed run can't block the next one.
    • Two SFTP connections by default (BACKUP_SFTP_CONNECTIONS; restic's default is 5). The mirror's rclone uses the same number of --transfers.
    • Optional upload cap in KiB/s for restic and the mirror (BACKUP_LIMIT_UPLOAD_KBPS; empty means no cap).
  • docker-compose.yml / .env.example: pass the two new settings through.
  • README:
    • Step 11 starts the first backup with exec -d inside the scheduled container, so the run lock covers it.
    • New "A large first upload" section: if the run dies, rerun it, because restic reuses what already landed.
    • New troubleshooting entry for this exact error.

Checked: Alpine 3.20 ships restic 0.16.4, which has the sftp connections option, and rclone 1.66.

Testing

  • sh -n passes on both scripts. docker compose config -q passes.
  • Ran backup.sh locally with restic and sqlite3 shims and rclone's local remote:
    • unlock runs first, and -o sftp.connections=2 is on every restic call.
    • --limit-upload and --bwlimit appear only when the cap is set.
    • An overlapping run skips with exit 0.
    • The mirror tree is unchanged.
  • pytest -q: 1272 passed. npm test: 395 passed. npm run build is clean. Neither suite is touched by this change.
  • Not tested: the real NAS link. The next run on the server is that test.

Generated by Claude Code

The first real run died 90 seconds into a stall: ServerAliveInterval 30
x CountMax 3, five parallel restic connections saturating a home NAS,
and restic cannot reconnect once its ssh exits, so the run was lost.

- ssh waits five minutes of real silence (15s x 20), TCPKeepAlive on
- two SFTP connections by default (BACKUP_SFTP_CONNECTIONS), and an
  optional upload cap for restic and the mirror (BACKUP_LIMIT_UPLOAD_KBPS)
- one run at a time: an hourly tick during a long run skips itself
  instead of failing on the repository lock
- restic unlock first, so a crashed run's stale lock clears itself

README: start the first backup inside the scheduled container, and what
to do when a large first upload dies (rerun; restic reuses what landed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XWoa1a1upTria7ntq56x3B
@davior
davior marked this pull request as ready for review October 22, 2026 03:29
@davior
davior merged commit b07da47 into main Oct 22, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants