Fedora Infrastructure Team
backup03 and others
backups of critical data
We are now running a rdiff-backup of all our critical data on a daily basis. This allows us to keep incremental changes over time as well has have a recent copy in case of disaster recovery.
The backups are run from backup03 every day at 22:10UTC as root. All config is in ansible.
The cron job checks out the ansible repo from git, then runs ansible-playbook with the rdiff-backup playbook. This playbook looks at variables to decide which machines and partitions to backup.
First, machines in the backup_clients group in inventory are operated on. If a host is not in that group it is not backed up via rdiff-backup.
Next, any machines in the backup_clients group will have their
/homedirectories backed up by the server running rdiff-backup and using the rdiff-backup ssh key to access the client.
Next, if any of the hosts in backup_clients have a variable set for host_backup_targets, those directories will also be backed up in the same manner as above with the rdiff-backup ssh key.
For each backup an email will be sent to sysadmin-backup-members with a summary.
Backups are stored on a netapp volume, so in addition to the
incrementals that rdiff-backup provides there are netapp snapshots. This
netapp volume is mounted on
/fedora_backups and is running dedup on the
When backup03 is rebooted, you must restart the ssh-agent and reload the rdiff-backup ssh key into that agent so backups can take place.
sudo -i ssh-agent -s > sshagent source sshgent ssh-add .ssh/rdiff-backup-key
add the host to the backup_clients inventory group in ansible.
If you wish to backup more than
/home, add a variable to
host_backup_targets: ['/srv']See https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/pagure#_81 for example.
On the client to be backed up, install rdiff-backup.
On the client to be backed up, install the rdiff-backup ssh public key to
/root/.ssh/authorized_keysIt should be restricted from:
and command can be restricted to:
command="rdiff-backup --server --restrict-update-only"
rdiff backup keeps a copy of the most recent version of files on disk, so if you wish to restore the last backup copy, simply rsync from backup03. If you wish an older incremental, see rdiff-backup man page for how to specify the exact time.
Backups are currently kept forever, but likely down the road we will look at pruning them some to match available space.
ssh-dss AAAAB3NzaC1kc3MAAACBAJr3xqn/hHIXeth+NuXPu9P91FG9jozF3Q1JaGmg6szo770rrmhiSsxso/Ibm2mObqQLCyfm/qSOQRynv6tL3tQVHA6EEx0PNacnBcOV7UowR5kd4AYv82K1vQhof3YTxOMmNIOrdy6deDqIf4sLz1TDHvEDwjrxtFf8ugyZWNbTAAAAFQCS5puRZF4gpNbaWxe6gLzm3rBeewAAAIBcEd6pRatE2Qc/dW0YwwudTEaOCUnHmtYs2PHKbOPds0+Woe1aWH38NiE+CmklcUpyRsGEf3O0l5vm3VrVlnfuHpgt/a/pbzxm0U6DGm2AebtqEmaCX3CIuYzKhG5wmXqJ/z+Hc5MDj2mn2TchHqsk1O8VZM+1Ml6zX3Hl4vvBsQAAAIALDt5NFv6GLuid8eik/nn8NORd9FJPDBJxgVqHNIm08RMC6aI++fqwkBhVPFKBra5utrMKQmnKs/sOWycLYTqqcSMPdWSkdWYjBCSJ/QNpyN4laCmPWLgb3I+2zORgR0EjeV2e/46geS0MWLmeEsFwztpSj4Tv4e18L8Dsp2uB2Q== root@backup03-rdiff-backup