We use Borg to handle backups.

Note: on some machines we use Borgmatic instead of the following documentation. Much simpler, much cleaner. Check skwotcustom:/etc/borgmatic/config.yaml

To backup a machine, we can use the backup script:

  • root@server:~# cd /opt
  • root@server:/opt# git clone https://0xacab.org/squatnet/scripts
  • root@server:/opt# ln -s /opt/scripts/do-backup.sh /usr/local/bin/do-backup.sh

You will need theses instructions to prepare both the server to backup and the backup server:

For skwotmail, skip step 4 (mysql is backed up offline, by copying its files directly)

  • Generate an ssh key on the client machine (the one to backup):
    • # ssh-keygen -t rsa -b 4096 -f ~/.ssh/for_spica
  • Add the public part of it to the backup server (through ansible)
  • Also add a key for admins to recover backups (same procedure, in ansible)
  • Edit .ssh/config file and, at the beggining, add: Host spica HostName spica.tachanka.org User borgy IdentityFile ~/.ssh/for_spica
  • If needed, create a MySQL readonly user:
    • # mysql
    • > GRANT SELECT, SHOW VIEW, LOCK TABLES, RELOAD, REPLICATION CLIENT ON *.* TO 'backupuser'@'localhost' IDENTIFIED BY 'pa$$w0RD';
    • > FLUSH PRIVILEGES;

And for each repo you want (by default we use one per server, but who knows in the future!), do these once:

  • Init some vars:
    • # BKSERVER=<backup server like in ~/.ssh/config (probably 'spica')>
    • # REPO=<name of the repo to create (eg. 'rootfs')>
    • # export BORG_PASSCOMMAND="cat /root/borg-stuff/${REPO}.passphrase"
  • Add a passphrase for the repo (might need to create /root/borg-stuff/):
    • # tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 4000 > /root/borg-stuff/${REPO}.passphrase
  • You may want an excludes-file, if so place it in /root/borg-stuff/${REPO}.excludes
  • Initialise the repo, or borg will just fail and do nothing:
    • # borg init --encryption=keyfile --append-only "${BKSERVER}:./${REPO}"
  • Backup the repo key and passphrase:
    • # borg key export $BKSERVER:$REPO /root/borg-stuff/${REPO}.key
    • Copy the key just exported and the passphrase generated earlier to your local computer.
    • Tell other admins to do it as well
  • Launch the script once, in a screen or tmux first:
    • # tmux
    • # DBPASS=<DB password for backupuser> do-backup.sh -h "$BKSERVER" -u backupuser -r "$REPO"
  • You can check the backup mounting the repo:
    • # borg mount $BKSERVER:./$REPO /mnt/
  • If everything went fine, cron it:
    • # crontab -e
    • Add: MAILTO=<email address, eg techâ’¶squat.net> 10 6 * * * DBPASS=<DB password for backupuser> /usr/local/bin/do-backup.sh -h "$BKSERVER" -u backupuser -r "$REPO"
    • (but do not use variables here!)
  • Add a line in /root/Changelog

--

borg list --json yourrepo.borg | jq -r .archives[].archive | xargs -d '\n' -I "{}" borg info "yourrepo.borg::{}" | sed '/Archive fingerprint\|Comment\|Utilization\|Command\|Number\|Duration\|Hostname\|Username\|Time\|Chunk\|Unique\|---/d'\