I use small VMs on my local network to run things like DNS and Home Assistant. Some of them have persistent data, but I haven't had a reliable backup strategy for them yet. This weekend I consolidated the backups to run from Synology DSM via their Active Backup for Business (ABB) feature. Working backwards from what finally worked...
- Read-only SSH+rsync Container with an ABB File Server connection
- abb-cli uses a custom CA bundle ignoring self-signed CAs already trusted by the host.
- Full ISO image backups were too large when using an ABB Virtual Machine connection.
- Incompatible
rrsyncusage because ABB wants to use the-soption.
Read-only SSH+rsync Container
Ultimately, I ended up using a dedicated, read-only containerized SSH server which mounts the persistent data that I want backed up. Most services are already provisioned as containers, so this was a quick addition. This approach also helped avoid extra changes to the host's users and SSH configuration. A few other principles I ended up applying here:
rsyncbackupuser - runs asuid=0for access to multi-user service directories, but seeingrootin the backup manager just felt... security-wrong.- read-only container and mount - avoids any unexpected modifications of persistent or container image files.
ForceCommandoption - forces the use of thersynccommand. Not quiterrsynccontainment, but close enough in this setup and it keeps Synology happy.
It's worth noting that this approach does not try to pause the services or disks for the snapshot. For now, at least, that's less of a concern for these kinds of VMs. For more technical details, review the small rsync-sender.sh, sshd_config, or Dockerfile files.
Basic Usage
As part of my deployment script, the following steps are effectively run on each VM.
$# generate a host-specific ssh server host key$ssh-keygen -t ed25519 -N '' -f /srv/rsync-sender/ssh_host_ed25519_key$# add an externally-generated ssh public key (ABB uses the private key)$cat > /srv/rsyncbackup/authorized_keys$# run the container, one way or another$podman run -d --name rsyncbackup --read-only \ -p 20261:22 \ -v /srv/rsyncbackup/ssh_host_ed25519_key:/etc/ssh/ssh_host_ed25519_key:ro \ -v /srv/rsyncbackup/authorized_keys:/etc/ssh/authorized_keys:ro \ -v /mnt/persistent:/mnt/persistent:ro \ rsyncbackup
Sanity Checks
After the container is up, you can test it out with a few commands...
$# read operations should succeed$rsync -a -s -e 'ssh -p 20261 -i ./id_ed25519' rsyncbackup@192.0.2.42:/mnt/persistent/ ./restore-test/$# write operations should be rejected$rsync -a -e 'ssh -p 20261 -i ./id_ed25519' /dev/urandom rsyncbackup@192.0.2.42:/mnt/persistent/unwritableerror: only read-only rsync is permitted rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(232) [sender=3.2.7]$# read-then-sneaky-writes should fail due to the read-only filesystem$rsync -a -s --remove-source-files -e 'ssh -p 20261 -i ./id_ed25519' rsyncbackup@192.0.2.42:/mnt/persistent/ ./restore-test/rsync: [sender] sender failed to remove example.txt: Read-only file system (30) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1887) [generator=3.2.7]
Configure Active Backup for Business
Once provisioned, set up for ABB is pretty straight-forward.
- Open Active Backup for Business.
- Navigate to File Server.
- Click Add Server.
- Choose rsync server and click Next.
- For Connection Method, be sure that rsync shell mode via SSH is selected.
- Fill out the rest of the connection parameters (including the container's non-standard SSH port).
- Click Done, and a quick verification test should confirm the configuration.
- Configure the backup strategy (e.g. mirror, incremental) and target storage.
- Optionally, initiate a backup immediately.
From there, Synology DSM will provide notifications on success/failure, and you might want to configure other offsite backups for these backups, depending on your strategy.
Of course, a final solution is rarely the first, so now I need to document what I should avoid trying again...
Failed Attempt #1: Native Linux Agent
I had first started by installing the native client software for ABB. And the first hiccup came before any backups - connecting to my NAS which uses a private CA's certificate.
$sudo abb-cli -cEnter server address: nas.example.com Enter username: abb-client Enter password: Connecting... The SSL certificate of the Synology NAS is not trusted. To learn how to obtain a valid certificate, please have a registered domain by setting up DDNS (http://sy.to/ddns) and applying for Let's Encrypt (http://sy.to/letsencrypt) certificate. Proceed anyway? [y/n](default: y):
The private CA was already installed in the host's trusted certificates, so that was weird. A quick Google search and I found a workaround for manually installing my private CA to Synology's trust store.
$sudo systemctl stop synology-active-backup-business-linux-service$sudo rm -f /opt/Synology/ActiveBackupforBusiness/data/certs.pem$curl https://pki.example.com/roots.pem | sudo tee /opt/Synology/ActiveBackupforBusiness/data/certs.pem$sudo systemctl start synology-active-backup-business-linux-service
After that, sudo abb-cli -c connected without complaint.
Failed Attempt #1b: Full VM ISO Images
Once my VM was connected to ABB, I started an initial backup. I quickly learned it creates a full disk image as the backup - no way to point it at just /mnt/persistent. Considering that some of the persistent data is under 100 MB and the VM/OS/containers are disposable and recreatable... it seemed like a poor use of disk space. Especially considering further bloat for multiple backup versions and for some servers that end up in additional, offsite backups.
Failed Attempt #2: incompatible rrsync options
Avoiding full VM backups, I next went with the rsync approach. More specifically, rrsync, which ships with rsync, and restricts access to a single directory. I created a backup user on the VM with a ForceCommand of /usr/bin/sudo -n /usr/bin/rrsync -ro /mnt/persistent.
After verifying it manually, I switched over to ABB, created a new File Server with rsync, clicked next, and then... error.
Confusingly, 43 is not an error code according to rsync man pages. Finding nothing in Synology's Log Center, I connected to the NAS via SSH and took a look at journalctl. In the logs, I found the following...
Oct 24 11:11:22 nas synoscgi_SYNO.ActiveBackup.Agentless_1_test_connection[5610]: synoabk_synoscgi::[ERROR] rsync-tool.cpp:225(5610,140305762057856) Failed to test connection [43] Warning: Permanently added '192.0.2.42' (ECDSA) to the list of known hosts.
Oct 24 11:11:22 nas synoscgi_SYNO.ActiveBackup.Agentless_1_test_connection[5610]: synoabk_synoscgi::[ERROR] rsync-tool.cpp:225(5610,140305762057856) Failed to test connection [43] /usr/bin/rrsync error: option -s has been disabled on this server.
Oct 24 11:11:22 nas synoscgi_SYNO.ActiveBackup.Agentless_1_test_connection[5610]: synoabk_synoscgi::[ERROR] rsync-tool.cpp:225(5610,140305762057856) Failed to test connection [43] rsync error: rsync service is no running (code 43) at io.c(254) [Receiver=3.1.2]
Oct 24 11:11:22 nas synoscgi_SYNO.ActiveBackup.Agentless_1_test_connection[5610]: synoabk_synoscgi::[ERROR] rsync-tool.cpp:227(5610,140305762057856) Failed to test connection [43] result
Oct 24 11:11:22 nas synoscgi_SYNO.ActiveBackup.Agentless_1_test_connection[5610]: synoabk_synoscgi::[ERROR] protocol-wrapper.cpp:248(5610,140305762057856) ProtocolWrapper: Failed to TestConnection '-17', '0', ' [rsync code 43]'
Notably, /usr/bin/rrsync error: option -s has been disabled on this server which indicated it had indeed connected to the port and rrsync process. Apparently, ABB uses the following flag in its connection parameters.
-s, --protect-args no space-splitting; wildcard chars only
The flag is used to prevent interpreting of special characters (which is a convenience for some setups), but, sadly, I could not find a way to disable it in ABB. I also took a quick look at the rrsync wrapper source code, but it seemed a little silly to be disabling the singular option the command disabled by default – as if it were there for good reason.
Comments