Retention policies do not see a backup atomically. Each backup is composed of at least three Kopia snapshots, with additional snapshots if there are additional tablespaces. This has the following problems:
- If a backup fails between snapshots, the previous ones are valid, and will be used to evaluate retention policies. They can trigger deletion of snapshots that are still required by backups that must be retained.
- They can get different retention labels if they are in different time boundaries (hourly, daily, ...), and this will be extremely hard to detect.
I think we should ditch Kopia retention policies, and have our own retention policy manager, that given a set of retention policies and a catalog of completed backups can identify which ones are outside of retention and then remove them and the WALs that fall outside of retention. Orphan snapshots left by failed backups could be removed if a newer completed backup exists.
Retention policies configuration should be defined alongside the metadata. We can probably keep using the current klio retention commands to set them, adding a klio retention apply to force the evaluation of the retention policy.
Ditching Kopia retention policies could allow us to choose a simpler retention policy model, like the one in https://cloudnative-pg.io/plugin-barman-cloud/docs/retention/, or even support multiple models.
Retention policies do not see a backup atomically. Each backup is composed of at least three Kopia snapshots, with additional snapshots if there are additional tablespaces. This has the following problems:
I think we should ditch Kopia retention policies, and have our own retention policy manager, that given a set of retention policies and a catalog of completed backups can identify which ones are outside of retention and then remove them and the WALs that fall outside of retention. Orphan snapshots left by failed backups could be removed if a newer completed backup exists.
Retention policies configuration should be defined alongside the metadata. We can probably keep using the current
klio retentioncommands to set them, adding aklio retention applyto force the evaluation of the retention policy.Ditching Kopia retention policies could allow us to choose a simpler retention policy model, like the one in https://cloudnative-pg.io/plugin-barman-cloud/docs/retention/, or even support multiple models.