Short notes on keeping small servers boring.
Things worth writing down after they broke once. Mostly Linux, networking and backups, kept brief on purpose.
Recent
Networking
Queueing discipline does not survive a reboot
Setting a queueing discipline with tc applies immediately but is forgotten on restart. The interface quietly falls back to the distribution default, and throughput drops without any obvious error.
The fix is dull: a one-shot unit that reapplies it after the network is up. Worth checking with tc qdisc show after every reboot rather than trusting that it stuck.
Linux
Package locks during first boot
A fresh image often runs unattended upgrades in the background. Any install started in the first few minutes fails on a lock held by another process, and the error text points at the lock file rather than the cause.
Waiting for the lock to clear is more reliable than forcing it. Removing the lock by hand leaves the package database half-written often enough to regret it.
Certificates
Renewal fails quietly more often than issuance
Issuing a certificate is visible and loud. Renewal happens on a timer, and when the webroot path moved six weeks earlier nobody notices until the expiry warning arrives.
Checking the renewal path on the day the certificate is issued costs a minute. Finding out at expiry costs an evening.
Backups
An untested restore is not a backup
Snapshots that run nightly for a year prove only that the job exits zero. The first real restore is where missing permissions, absent database roles and an unmounted volume all surface at once.
Restoring to a spare host once a quarter turns the question from whether it works into how long it takes.