Do you even backup ?

Anarch157a — Thu, 05 Nov 2026 17:37:26 GMT

A tale of a near disaster

A couple of weeks ago I had a scare that shook me to my core. It happened when I, for some random reason, logged into my homelab server from work, an for some even more random reason, typed zpool status. For the two of you who don't know what the zpool command does, it's the tool to manage ZFS, a most amazing filesystem, created by the much missed Sun Microsystems.

By now you should be starting to have an idea of where this is going, but first a quick description of my homelab:

Hardware
An El-cheapo desktop motherboard with a Ryzen 3600 CPU and 32 GB of RAM
Two add-in network cards for the firewall
A M.2 SSD for boot, a small-ish SATA SSH for local files
Two 8 TB spinning rust SATA disks in a ZFS pool
Software
Proxmox on bare metal as Hypervisor
OPNSense firewall in a virtual machine, using the two ad-in NICs
Home Assistant on a VM
Assorted LCX containers (SearxNG, Pihole, Forgejo, Jellyfin, Arr suite, etc.)

Now, anyone with a Jellyfin instance running at home undertands why 8TB storage is necessary. Imagine how awfull it would be if all that was gonne in a puff of smoke...

And that's what almost happened. Or could have happened. Because on that random day, the status of the pool was DEGRADED!!! One of the disks had failed and there lots and lots of scarry messages in syslog, like this:

2026-10-17T23:58:59.780603-03:00 proxymoxy kernel: [360017.678925] sd 4:0:0:0: [sdb] tag#10 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRI
VER_OK cmd_age=0s
2026-10-17T23:58:59.780626-03:00 proxymoxy kernel: [360017.678944] sd 4:0:0:0: [sdb] tag#10 CDB: ATA command pass through(16) 85 06 2c 00 00 00 0
0 00 00 00 00 00 00 00 e5 00
2026-10-17T23:58:59.809459-03:00 proxymoxy pve-firewall[1747]: status update error: iptables_restore_cmdlist: Try `ip6tables-restore -h' or 'ip6t
ables-restore --help' for more information.

Fortunatelly, those are non-fatal errors. Most likelly a temporary issue whith a SATA cable. After getting home, I turned the server off, shuffled and reseated the cables and the disk came back online, the pool automatically resilvered the affected disk and everything ended up fine, right ? Right ?

No, it was not fine

What if the faillure had been catastrophic ? I can´t fathom losing terabytes of ~~hentai~~, ~~porn~~, films, series, music and other suff that I hoarded over the ~~years~~decades...

What to do then ? Well, get off my ass and eat my own dog food. Time to set up automated, regular backups!

How, and most important, where

Backing up on a local disk, no matter if it's inside the "server" itself or a USB enclosure, is not enough by itself. The same event that destroys the computer can make a USB drive unavailable too (floods, fire, theft, stupidity, etc.). A good backup strategy puts a copy outside the premisses. It's the famous "3-2-1 rule": 3 copies, 2 media types, 1 of them off-site.

Ok, then, so how do we do it in a home lab ? Well, my solution was involved first buying a new HDD, the same size as the two on my ZFS pool. The idea is that two drives of the exact same age have a high probability of failling within a short timeframe. Mine are almost 4 years old, so they should be considered a moderate risk. A new drive on the pool will make it less likelly to be completely destroyed on a single event, and releases one of the old drives to be used as backup media. But, but, but... use and internal drive for backups ? Yes. I configured the SATA ports to hot-swap mode, so the extra HDD can be considered removable media. No problem here, it's job is to help with the recovery of accidentally deleted data, not recovery from massive hardware loss or other types of catastrophic events.

That's the job of Hetzner.

I moved my personal VPS to Hetzner about a year ago, and they've been pretty good so far. One of the products they offer fits my backup needs like a glove, it's called Storage Box and besides being reasonably priced, it supports an impressive list of acces methods, including native to suport for Borg Backup, which means no need for hacks or kludges or sync scripts. Just point Borg to the Storage Box and that's it. Plus, I can backup this very website and the other stuff hosted in the same VPS (Nextcloud, Mailcow and Vaultwarden being the the more important) at no extra cost.

As for the details of how, well, Borg Backup, of course. But the nitty-gritty deets will be on a post of their own. They're boring, they're technical and they're even longer than this post already is.

So, if you're really insterest in all that, stay tunned.

Anarch157a's Blog (Posts about homelab)

Do you even backup ?

A tale of a near disaster

No, it was not fine

How, and most important, where