Skip to content

Retention policies are not atomic across a backup's snapshots #168

Description

@fcanovai

Retention policies do not see a backup atomically. Each backup is composed of at least three Kopia snapshots, with additional snapshots if there are additional tablespaces. This has the following problems:

  1. If a backup fails between snapshots, the previous ones are valid, and will be used to evaluate retention policies. They can trigger deletion of snapshots that are still required by backups that must be retained.
  2. They can get different retention labels if they are in different time boundaries (hourly, daily, ...), and this will be extremely hard to detect.

I think we should ditch Kopia retention policies, and have our own retention policy manager, that given a set of retention policies and a catalog of completed backups can identify which ones are outside of retention and then remove them and the WALs that fall outside of retention. Orphan snapshots left by failed backups could be removed if a newer completed backup exists.

Retention policies configuration should be defined alongside the metadata. We can probably keep using the current klio retention commands to set them, adding a klio retention apply to force the evaluation of the retention policy.

Ditching Kopia retention policies could allow us to choose a simpler retention policy model, like the one in https://cloudnative-pg.io/plugin-barman-cloud/docs/retention/, or even support multiple models.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions