A tale of a near disaster

A couple of weeks ago I had a scare that shook me to my core. It happened when I, for some random reason, logged into my homelab server from work, an for some even more random reason, typed zpool status. For the two of you who don't know what the zpool command does, it's the tool to manage ZFS, a most amazing filesystem, created by the much missed Sun Microsystems.

By now you should be starting to have an idea of where this is going, but first a quick description of my homelab:

• Hardware
• An El-cheapo desktop motherboard with a Ryzen 3600 CPU and 32 GB of RAM
• Two add-in network cards for the firewall
• A M.2 SSD for boot, a small-ish SATA SSH for local files
• Two 8 TB spinning rust SATA disks in a ZFS pool
• Software
• Proxmox on bare metal as Hypervisor
• OPNSense firewall in a virtual machine, using the two ad-in NICs
• Home Assistant on a VM
• Assorted LCX containers (SearxNG, Pihole, Forgejo, Jellyfin, Arr suite, etc.)

Now, anyone with a Jellyfin instance running at home undertands why 8TB storage is necessary. Imagine how awfull it would be if all that was gonne in a puff of smoke...

And that's what almost happened. Or could have happened. Because on that random day, the status of the pool was DEGRADED!!! One of the disks had failed and there lots and lots of scarry messages in syslog, like this:

2026-10-17T23:58:59.780603-03:00 proxymoxy kernel: [360017.678925] sd 4:0:0:0: [sdb] tag#10 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRI
VER_OK cmd_age=0s
2026-10-17T23:58:59.780626-03:00 proxymoxy kernel: [360017.678944] sd 4:0:0:0: [sdb] tag#10 CDB: ATA command pass through(16) 85 06 2c 00 00 00 0
0 00 00 00 00 00 00 00 e5 00
2026-10-17T23:58:59.809459-03:00 proxymoxy pve-firewall[1747]: status update error: iptables_restore_cmdlist: Try `ip6tables-restore -h' or 'ip6t
ables-restore --help' for more information.

Fortunatelly, those are non-fatal errors. Most likelly a temporary issue whith a SATA cable. After getting home, I turned the server off, shuffled and reseated the cables and the disk came back online, the pool automatically resilvered the affected disk and everything ended up fine, right ? Right ?

No, it was not fine

What if the faillure had been catastrophic ? I can´t fathom losing terabytes of hentai, porn, films, series, music and other suff that I hoarded over the yearsdecades...

What to do then ? Well, get off my ass and eat my own dog food. Time to set up automated, regular backups!

How, and most important, where

Backing up on a local disk, no matter if it's inside the "server" itself or a USB enclosure, is not enough by itself. The same event that destroys the computer can make a USB drive unavailable too (floods, fire, theft, stupidity, etc.). A good backup strategy puts a copy outside the premisses. It's the famous "3-2-1 rule": 3 copies, 2 media types, 1 of them off-site.

Ok, then, so how do we do it in a home lab ? Well, my solution was involved first buying a new HDD, the same size as the two on my ZFS pool. The idea is that two drives of the exact same age have a high probability of failling within a short timeframe. Mine are almost 4 years old, so they should be considered a moderate risk. A new drive on the pool will make it less likelly to be completely destroyed on a single event, and releases one of the old drives to be used as backup media. But, but, but... use and internal drive for backups ? Yes. I configured the SATA ports to hot-swap mode, so the extra HDD can be considered removable media. No problem here, it's job is to help with the recovery of accidentally deleted data, not recovery from massive hardware loss or other types of catastrophic events.

That's the job of Hetzner.

I moved my personal VPS to Hetzner about a year ago, and they've been pretty good so far. One of the products they offer fits my backup needs like a glove, it's called Storage Box and besides being reasonably priced, it supports an impressive list of acces methods, including native to suport for Borg Backup, which means no need for hacks or kludges or sync scripts. Just point Borg to the Storage Box and that's it. Plus, I can backup this very website and the other stuff hosted in the same VPS (Nextcloud, Mailcow and Vaultwarden being the the more important) at no extra cost.

As for the details of how, well, Borg Backup, of course. But the nitty-gritty deets will be on a post of their own. They're boring, they're technical and they're even longer than this post already is.

So, if you're really insterest in all that, stay tunned.

New static site generator: Nikola

After a long while, I decided to start writing in this blog again, but I was not happy with Pelican, the static site generator I was using before. So I started shopping arround for something more to my liking. In the end I decided to give Nikola a try.

The best feature I've seen is the much better command line tool to manage the site. Both Pelican and Nikola provide a command to generate the site, but the diferences are stiking. In Pelican, the CLI tool generates the static site and provides a basic HTTP server so you can test the site while writing, not to mention the configuration, which requires different config files for development and publishing.

Nikola on the other hand, adopts the command <verb> format that became so popular this century (it's the same paradigm used by other popular tools like git, docker, openssl, etc.). This tool has verbs to help set up a website, start new posts/pages with metadata pre-filled, publish the site, manage plugins and themes, and other goodies.

Well, this is it. Hope you enjoy.

Listing non-root users

This is a simple Bash script to list all non-root and non-system users in Linux. Those users are determined by two settings, UID_MIN and UID_MAX, found on the /etc/login.defs file.

#!/bin/bash -
#===============================================================================
#
#          FILE: lista-users.sh
#
#   DESCRIPTION:
#
#        AUTHOR: Nerdeiro da Silva <a n a r c h 1 5 7 a  at  n i n j a z u m b i . c o m>
#       CREATED: 01/08/2023 14:14:18
#      REVISION:  0.01
#       LICENSE: GNU GPL 2.X (https://www.gnu.org/licenses/old-licenses/gpl-2.0.html)
#===============================================================================

set -o nounset                                  # Treat unset variables as an error

# Get lowest and highest UID from login.defs
export min_uid=$( awk '/^UID_MIN/{print $2}' /etc/login.defs )
export max_uid=$( awk '/^UID_MAX/{print $2}' /etc/login.defs )

# Use Awk expresisons to list only users between those limits
USERS=( $(awk -v min=${min_uid} -v max=${max_uid} -F: '$3 >= min && $3 <= max { print $1}' /etc/passwd ) )

echo "Users :: ${USERS[@]}"
echo
# Example for loop to do something for each user
for U in ${USERS[@]}; do
    USR_DIR=$(awk -F: -v usr=${U} '$1 == usr {print $6}' /etc/passwd)
    if [[ -d ${USR_DIR} ]]; then
        echo "User: $U; Directory: $USR_DIR"
    fi
done

Edit: New version of the script

Check the repository on https://gitlab.com/Nerdeiro/game-cloud-sync and follow the README.md there.

Sync your games to (almost) any cloud service

This is a simple Bash script to sync your game-saves with pretty much any cloud storage service out there. It uses Rclone, which supports dozens (not hyperbole) of providers.

The motivation to create this script was to sync my copy of Stardew Valley between my desktop and my Steam Deck. But wait, doesn't Steam do this ? Yes, it does, for copies bought on Steam. But my copy is from GOG... It runs just fine on the Deck, regardless if it was installed from a manual download, with Lutris or Heroic, it only lacks cloud saves.

This is where this script and Lutris come in. The role Lutris will play here, is to allow us to run a script both before and after running the game, all without having to change the default start.sh lancher script that the GOG installer creates.

What you need to use this script

In no particular order:

• Lutris (optional but recommended)
• Rclone
• A cloud service supported by Rclone
• A working remote configuration in Rclone

If you're syncing between two computers, getting Lutris and Rcloneis as easy as installing them using your distro's package manager. On the Steam Deck, Lutris have to be installed using Flatpack, after changing to Desktop mode. Rclone is a little bit more challenging, since it's not on Flathub and the Deck ships with a read-only Root (/) file-system, so using Pacman to get it is not an option.

Thankfully, the creators of Rclone have a statically compiled binary on the official website, which means you can run it on the Deck without any issues, only you'll have to put it somewhere on the $HOME directory. To avoid having to fiddle with the $PATH, I recommend that after unziping the downloaded file, you move the rclone executable to a convenient place, then edit the script to point the variable RC to the full, absolute path of the binary.

After installing Rclone, follow the official documentation to create a configuration for your remote cloud service. It's important that you can successfuly run a copy or sync between the two sides. Lets say that you run rclone copy mycloud:somedir/ ~/tmp/someotherdir, if the command finishes and the files show up in ~/tmp/someotherdir, then you're good to go.

Getting the script

Just clone it from my Gitlab Repo at https://gitlab.com/Nerdeiro/game-cloud-sync. You can do it with the command

git clone https://gitlab.com/Nerdeiro/game-cloud-sync.git

Once it's cloned, you can use it right away, on a normal Linux distro. To put use it on a Steam Deck, before transfering a copy to the console, open it in your favorite flavor of Vim and find the line:

RC=$(which rclone)

and change everything after the equals sign for the full path to the Rclone binary that you downloaded before. It should look like:

RC=/home/deck/bin/rclone

or something like that.

Using the script on the command line

Just run it with the following options:

• -s : Source. It can be a local directory or an Rclone remote
• -d : Destination. It can be a local directory or an Rclone remote
• -v : Verbose. Shows the copy progress on the terminal

One of the parameters should be the local directory where your save games are, eg. $HOME/.config/StardewValley, the other should be a remote configuration and path, like mycloud:/StardewValley. Which one you'll use with -s or -d will depend on which direction you want to sync, to or from the cloud.

Using the script with Lutris

Open Lutris and click with the right mouse button on your game, selecting "Configure" to show the screen bellow:

If your screen is not like that, tick the option "Advanced" on the top of the dialog, then scroll all the way to the botton of the "System Options" tab. There you'll the boxes to configure a Pre-launch and a Post-exit scripts.

To download the save files from the cloud to your computer (or Deck) fill the "Pre-launch" line with:

sync-game.sh -s my_cloud:saved_games/ -d $HOME/game_dir/

To upload the save files from the computer to the cloud, fill the "Post-exit" line with:

sync-game.sh -s $HOME/game_dir/ -d my_cloud:saved_games/

If everything works as they should, your game should be saved on your cloud storage, ready to sync to other devices. Do the same on the Steam Deck if necessary.

And happy gaming.

Intro

All right, you automated lots of stuff arround your place with Home Assistant, but what about your PC ?

Wouldn't it be cool if you could turn it on or off from anywhere, just tapping on a button in your phone screen ?

Well, if your PC supports Sleep/Resume, you can do it with a couple of scripts. I'll demonstate here how to do it for a Linux desktop/notebook. You can addapt this to MacOS, maybe even Windows. You you do it, let me know on Mastodon, just Toot me at @anarch157a@cyberpunk.lol.

Checking system

First of all, let's check if your system supports Sleep/Resume. On a laptop, this is pretty much a given, but on a desktop, it may not be enable on the BIOS settings. Run the command bellow to check it:

$ cat /sys/power/state

freeze mem disk

If you get this response, your system supports S3 sleep. Good. If not, check your BIOS settings. Since each mother board maker have different menus, I can't help you that, sorry.

Second, to wake up the system, it'll be necessary to enable Wake on Lan (WoL) too. This is another BIOS setting the you need to check, keep in mind that some board makers (Like Asus), might not have an explicit "Wol" or "Wake on Lan" setting, in Asus boards you need to activate a setting that says "Power On By PCI-E".

Now, to make WoL work, you're have to find what is your actual MAC Address. It is possible to use a custom MAC when the system is up and running, but the system is on S3 (Sleep) state, it's the BIOS that will answer to the "Magic Packets", using the factory assigned MAC. Fortunatelly, using the ip command in Linux, we can figure this easily:

$ ip link show

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1420 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether xx:yy:de:ad:be:ef brd ff:ff:ff:ff:ff:ff permaddr xx:yy:zz:aa:bb:cc

The address after 'permaddr' is your factory assigned MAC. Write it for later.

Remote scripts

Now, let's prepare the scripts that will do the magic. SSH to your Home Assistant box, if HA is running on a container instead of a as stand-alone OS, make sure you're comfortable with Docker's or Podman's commands, because you're gonna need to do some restarts.

Now, let's see what our scripts look like, before deploying them to HA.

The suspend script

This is the script that should put the machine to sleep:

#!/bin/bash - 

ssh -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    -i /config/keypair nerdeiro@192.168.1.2 \
       'logger "Sent to sleep by Home Assistant"
        sudo systemctl start suspend.target'

A little bit of explanation about the script. Since it's supposed to run a command using sudo (see bellow why), we need to run SSH with -t option, so it creates a pseudo-TTY on the other side, or it will fail, then since we don't want to deal with SSH known hosts stuff, We tell it to ignore strict host checking and use /dev/null instead of $HOME/.ssh/known_hosts.

After that, we have the private key used to authenticate in the target box, the user and host and the script itself. This is where you should be asking "Hey, but what if the IP address changes ?". Well, make sure it doesn't. Either configure a static IP on your machine, or set up your DHCP server/WiFi router/whatever to always assign the same IP to you, based on the MAC Address. In my case, the DHCP server that distributes addresses on my LAN is a Pi-hole, which you should be using anyway to block ads and trackers.

Then the script that will run on the other side. In this example, it's a two-liner because I added a line to generate a syslog message with logger. If you just want the suspend command, there's no need to put it between single quotes. For multi-line scripts, that's necessary.

Now, you might be wondering, why use sudo with systemctl to start the suspend target ? I don't need it locally ! The answer for that is polkit, which has a local agent that authenticates you and lets you suspend the machine without explicitly using pkexec or sudo, but since polkit agents don't work over SSH (maybe they do, but my DuckDuckGo skills failed me this time), we'll have to use good old sudo instead.

Now, let's deal with authentication on the machine we want to suspend. Putting your password on a script is a bad idea and it's cumbersome to use, so a key-pair is the way to go. So, SSH to the machine running Home Assistant and create a new pair of keys there. If you're running HA on a container, you should have a directory somewhere that is mounted as a volume under /config inside the container. You can even do this from inside the container itself. Run docker ps, get the name of the container (let's say it's called "homeassistant") and run docker exec -ti homeassistant bash. This should put you right at the /config directory inside the container. If your HA is a stand-alone OS, once logged into it, change to the root user with sudo su - and cd /config. Creating a key-pair is as simple as running ssh-keygen -t ed25519 -f /config/keypair and pressing Return a couple of time to leave the password empty. If you prefer RSA keys, intead of ED25519, just remove the -t ed25519.

Once you have the key-pair, copy the contents of the file ending in .pub to the target machine and add it as a line in $HOME/.ssh/authorized_keys. Try SSHing to your PC from inside HA-OS or container using this key, just to confirm everything is OK before proceeding.

The resume script

This is the one that wakes uo your machine:

#!/bin/bash - 

wakeonlan -i 192.168.1.255 xx:yy:zz:aa:bb:cc

This is a very simple one. We don't even need to worry about installing the wakeonlan command, as it's already provided on the Home Assistant container/OS. The only thing you should change here is the network broadcast address and the MAC Address your your PC. To find what the broadcast address is, run the command:

$ ip address show

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1420 qdisc mq state UP group default qlen 1000
    link/ether xx:xx:de:ad:be:ef brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.2/24 brd 192.168.1.255 scope global enp4s0
       valid_lft forever preferred_lft forever

The broadcast address is the one right after brd on the 3rd line.

You can test if its working, as long as you have some other device that can SSH to Home Assistant. Put your PC to sleep, then from HA, run the wakeonlan command like in the script. If the PC turns on, we're set. If not, make sure that the machine supports this feature, same for the network card, if it's an add-on, and your BIOS is properly configured.

Home Assistant setup

All right, lets get to the good part. Setting up Home Assistant itself.

Before we start pushing the mouse on the GUI, there are a few configurations we need to do, to tell Home Assistant the scripts exist. Locate whre the file configuration.yaml is. If you're docker execing into the container or SSHd to the HA-OS, it'll be located in /config, if you're outside the container, check in docker which directory is used as the /config volume.

Create both scripts on the /config directory, making sure they're executable, run chmod 755 script if neccessary. Run each of them from the CLI, just to be sure they work.

Now, open up configuration.yaml on your favorite editor, such as Vim or Nano, and add this lines:

shell_command:
  sleep_script: /config/sleep.sh
  wake_script: /config/wake.sh

Listing them as shell commands in configuration.yaml will create the services that can be run, but to add them as buttons on a dashboard we need to create them as script entities. If you look at the configuration, you should see a line like:

script: !include scripts.yaml

So, save the configuration file and open scripts.yaml on your editor. Don't worry if it's empty, just add the following lines to it:

pc_sleep:
  alias: pc_sleep
  sequence:
  - service: shell_command.sleep_script
    data: {}
  mode: single
pc_wake:
  alias: pc_wake
  sequence:
  - service: shell_command.wake_script
    data: {}
  mode: single

Those will tell Home Assistant that they exist, so we can set them up on the GUI, which is the next step. So, save the files and restart Home Assistant, either by rebooting the server if it's an OS install or by restarting the container. After that, log on to the GUI and lets create some buttons.

Creating buttons

On the Home Assistant GUI, chose the dashboard where you want the suspend and resume buttons to be and click on the "hamburger" menu, it should be on the top-right of the page, and chose Edit Dashboard. Then click Add Card and chose "Button". Fill in the following:

• Entity: Select one of the scripts we created (e.g. script.pc_sleep)
• Name, Icon, etc.: It's up to you how it would look like.
• Tap Action: Select Call Service
• Service: Again, select the script you want to run.

It should look like this:

Testing the setup

I made a small video showing it working:

By now, you too should have the capability to sleep/resume your PC from anywhere, even your bed, so no more worries if the computer was left on or not. Just reach to your phone and turn it off.

Well, this was a long one, so happy automations and see you on the next one.