Add offline backup for foremanctl

[sjha4]

Why are you introducing these changes? (Problem description, related links)

This implements offline backup with foremanctl.

What are the changes introduced in this pull request?

• Offline backup

  • Adds foremanctl backup command for offline backups
  • Implements comprehensive backup of core deployment components:
    • All databases (foreman, candlepin, pulp, IOP databases)
    • Foremanctl state directory
    • Pulp content directory (optional via --skip-pulp-content)
    • Container image metadata (digests for restore compatibility)
  • Preflight checks to ensure backup safety:
    • Verifies no running Foreman/Pulp tasks (with --wait-for-tasks option)
    • Database integrity verification using PostgreSQL amcheck extension
  • Automatic service restoration on backup failure
  • Adds comprehensive user documentation at docs/user/backup.md
  • New roles: backup, check_database_index

How to test this pull request

I got a foremanctl box with normal deploy. On this box, clone foremanctl repo and checkout this branch.
cd /root/foremanctl
source .venv/bin/activate
export OBSAH_STATE=/var/lib/foremanctl

Then try ./foremanctl --help

Also,

(.venv) [root@ip-10-0-167-40 foremanctl]# ./foremanctl backup  --help
usage: foremanctl backup [-h] [-v] [--skip-pulp-content][--wait-for-tasks] backup_dir

Create offline backup of Foreman databases and configuration

positional arguments:
  backup_dir            Directory where backup files will be stored

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         verbose output
  --skip-pulp-content   Skip Pulp content directory backup
  --wait-for-tasks      Wait for running tasks to complete instead of failing immediately

You can run :
Command:

cd /root/foremanctl
source .venv/bin/activate
export OBSAH_STATE=/var/lib/foremanctl
./foremanctl backup /var/tmp/foreman-backup-test --wait-for-tasks

Steps to reproduce:

• On a foremanctl box, run foremanctl backup BACKUP_DIR

Checklist

• Tests added/updated (if applicable)
• Documentation updated (if applicable)

[ianballou]

I did an automated test run, take these with a grain of salt, but I wanted to post early in case they're real for extra time to test and fix.

Firstly, the DBs did get backed up, so awesome, but there were some hiccups that stopped the full process from running:

Bug #1: podman_network.yaml fails when no custom networks exist

File: src/playbooks/backup/tasks/podman_network.yaml
Severity: Blocker — prevents backup from completing
Description: The shell command podman network ls --format '{{.Name}}' | grep -v '^podman$' | while read net; do ... returns exit code 1 when there are no custom networks, because grep -v finds no matching lines.
Fix: Add || true after the grep, or use a different approach:

# Option A: tolerate empty result
  failed_when: networks_json.rc not in [0, 1]

# Option B: check first, skip if no custom networks

Bug #2: Wrong Foreman tasks API endpoint

File: src/playbooks/backup/tasks/preflight.yaml
Severity: High — preflight silently skips running task detection
Description: Uses https://{{ fqdn }}/api/v2/tasks?state=running which returns 404. The correct endpoint is https://{{ fqdn }}/foreman_tasks/api/tasks?state=running&search=state%3Drunning. Because failed_when: false is set, the error is silently ignored.
Impact: Backups will proceed even with running Foreman tasks, risking data inconsistency.

Bug #3: pg_isready and pg_dump not available on host

... I cut the output here, I'm not sure why these commands weren't on my box. It's not related to this PR I don't think.

Bug #4: Hardcoded parameters.yaml path in metadata task

File: src/playbooks/backup/tasks/metadata.yaml
Severity: Low — affects metadata accuracy only
Description: ansible.builtin.slurp reads from /var/lib/foremanctl/parameters.yaml but foremanctl's state directory is configurable via OBSAH_STATE. In dev/vagrant setups, the actual path is different (e.g., /vagrant/.var/lib/foremanctl/parameters.yaml).
Impact: enabled_features: [] in metadata despite features being configured. Doesn't affect DB dump correctness.
Fix: Use the state_dir variable instead of hardcoding the path.

[sjha4]

Will update the tasks endpoint and parameters.yaml path..

About the podman networks, those are created for IOP.. Like https://github.com/theforeman/foremanctl/blob/master/src/roles/iop_network/tasks/main.yaml so I'd assume we have that present in production deployments. We can add some handling for when it's not.

[aidenfine]

Maybe nitpick but does it make sense to include the not yet implemented flags when doing backup --help? Would you be opposed to just leaving them out in this PR?

[sjha4]

Maybe nitpick but does it make sense to include the not yet implemented flags when doing backup --help? Would you be opposed to just leaving them out in this PR?

I am fine either way but it's helpful guidance for future PRs and documentation to look at.

[evgeni]

not a full review (I stopped somewhere around the secrets backup), but overall this feels a lot like "let's write a huge bash script and then wrap it in YAML" and not like Ansible :(

[sjha4]

@ehelms @evgeni @ianballou Does this 🍞 look baked enough? We have some follow up PRs in this area we are starting to focus on based on this PR if we think this looks good.

[ianballou]

A few small comments, overall this is working well. Overview of my testing:

- All three databases dumped and valid
- State directory archived (passwords, OAuth keys, parameters.yaml, and certs when `OBSAH_STATE=/var/lib/foremanctl`)
- Pulp content + encryption keys captured
- `--skip-pulp-content` works
- Preflight checks (tasks, amcheck) work
- Services always come back up after backup or failure
- No dead code
- Restore works (with some bugs caused by the restore code, not backup)

The comments are minor and non-blocking from my standpoint, so I'm going to ack this. If we discover anything else during restore (which I don't expect since I tested restore), I think we can address it there.

[ianballou]

This use of database.yml causes password files to be created even for databases that don't exist. In my case, it was for the IoP databases on my vanilla Katello installation.

I'm not sure this should be fixed by backup/restore. Without being an Ansible expert, I wonder if backup/restore should not be responsible for filtering out databases that are outside of the scope of features used in the installation. Otherwise we'll have having feature filtering logic spread all over.

This is not a critical issue I don't think. Side effect is that my machine ended up with:

-rw------- 1 root root 21 Jun 16 14:46 iop-advisor-db-password
-rw------- 1 root root 21 Jun 16 14:46 iop-inventory-db-password
-rw------- 1 root root 21 Jun 16 14:46 iop-remediation-db-password
-rw------- 1 root root 21 Jun 16 14:46 iop-vmaas-db-password
-rw------- 1 root root 21 Jun 16 14:46 iop-vulnerability-db-password

even though I had no IoP DBs. @ehelms do you think this is worth a broader issue?

[ehelms]

Yes - please file a github issue.

[ehelms]

I think this should be type: offline

[sjha4]

Updated to type: offline ✅

[ehelms]

iop will be in the features list and is not needed here

[sjha4]

Updated. 👍🏼

[ehelms]

Two minor nitpicks. I am ok with adding tests as a follow up PR.