Backups and Health Checks

The plan

Three copies, on two machines, one of them a separate drive:

1. The laptop, where you work.
2. A mirror on the server: ~/laptop on the server’s own disk, an exact copy of the laptop’s Documents and Pictures as of the last push.
3. Snapshots on the backup drive: every night, the server’s snapshot script copies the mirror to dated folders on the backup drive, keeping two weeks of history.

The mirror follows the laptop, deletions included. The snapshots are what let you go back: a file deleted or overwritten by mistake on Monday is still in Sunday’s snapshot.

The backup drive

Plug the ext4 drive labelled backup (module 4) into the server and mount it from fstab, exactly as in module 4 ¹:

LABEL=backup  /mnt/backup  ext4  nofail,x-systemd.device-timeout=5s  0  2

Then sudo mkdir -p /mnt/backup, sudo systemctl daemon-reload, sudo findmnt --verify, sudo mount /mnt/backup, and sudo chown $USER: /mnt/backup.

The weekly push

On the laptop, add a function to your dotfiles’ bash_aliases:

# Mirror Documents and Pictures to the home server's ~/laptop folder
push-to-server() {
    rsync -av --delete ~/Documents ~/Pictures server:laptop/
}

server:laptop/ means the laptop folder in your home on the host called server, reached over SSH using your ~/.ssh/config entry ². With no trailing slash on ~/Documents, rsync copies the folder itself, giving laptop/Documents ². -a keeps permissions and times, -v lists what it sends, and --delete removes files from the mirror that are gone from the laptop ². Only changed files travel, so after the first push it’s quick.

Run it once a week, as part of update-all day. The first run copies everything and takes a while; start it in tmux if you’ll walk away.

Why by hand? Your SSH key has a passphrase, and a timer can’t type it. A fully automatic push is possible with a second key without a passphrase, limited on the server by options in authorized_keys, such as restrict, which disables forwarding and terminal access for that key ³. It’s a reasonable next step once the manual version has run for a while; a key with no passphrase needs that care.

The nightly snapshots

On the server, copy the Shell course’s snapshot-backup script to ~/.local/bin/ (scp it from the laptop), and change its SOURCES line to back up the mirror:

readonly SOURCES=("$HOME/laptop")

Then give it the user service and timer from module 3, pointed at /mnt/backup ⁴. In ~/.config/systemd/user/snapshot-backup.service:

[Unit]
Description=Snapshot backup to the backup drive

[Service]
Type=oneshot
ExecStart=%h/.local/bin/snapshot-backup /mnt/backup

and in ~/.config/systemd/user/snapshot-backup.timer:

[Unit]
Description=Nightly snapshot backup

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true

[Install]
WantedBy=timers.target

Since nobody logs in to the server, turn on lingering so your user manager, and its timer, run from boot ⁵:

me@garden-server:~$ loginctl enable-linger
me@garden-server:~$ systemctl --user daemon-reload
me@garden-server:~$ systemctl --user enable --now snapshot-backup.timer
me@garden-server:~$ systemctl --user start snapshot-backup.service
me@garden-server:~$ journalctl --user -u snapshot-backup.service

The manual start makes the first snapshot now; the journal shows how it went ⁶. After that, it runs every night at 2:30, and catches up after any night the server was off.

A health check

A server you never look at needs a way to tell you how it’s doing. This script gathers the checks from modules 3 and 4 into one report. Save it as health-check:

#!/usr/bin/env bash
#
# health-check: a quick report on this server's health.
#
# Usage: sudo health-check
#
# Shows uptime, disk space, failed services, recent errors, the newest
# backup snapshot, and each disk's SMART health. Changes nothing.

set -uo pipefail

readonly BACKUP=/mnt/backup

section() {
  printf '\n== %s ==\n' "$1"
}

main() {
  if [[ $EUID -ne 0 ]]; then
    echo "Run it with sudo: sudo health-check" >&2
    exit 1
  fi

  section "Uptime and load"
  uptime

  section "Disk space"
  df -h / "$BACKUP"

  section "Failed services"
  local failed
  failed="$(systemctl --failed --no-legend)"
  echo "${failed:-none}"

  section "Errors in the last 7 days (newest 20)"
  journalctl -p err --since "7d ago" --no-pager | tail -n 20

  section "Newest backup snapshot"
  if [[ -L "$BACKUP/latest" ]]; then
    readlink "$BACKUP/latest"
  else
    echo "none found in $BACKUP"
  fi

  section "Drive health (SMART)"
  local disk
  for disk in $(lsblk -dno NAME,TYPE | awk '$2 == "disk" { print $1 }'); do
    printf '/dev/%s: ' "$disk"
    smartctl -H "/dev/$disk" | grep -i 'result' || echo "no SMART report"
  done
}

main "$@"

It passes ShellCheck ⁷. Notes:

• It needs root for smartctl and the full journal, so it checks $EUID first and says how to run it.
• It uses set -uo pipefail but not -e: one failing check (say, a disk without SMART) shouldn’t stop the rest of the report.
• failed="$(...)" then ${failed:-none} prints “none” instead of an empty section.
• lsblk -dno NAME,TYPE lists whole disks only, and awk keeps those of type disk, skipping snaps’ loop devices.
• smartctl -H needs the smartmontools package (module 4) ⁸.

Install it for all users, owned by root:

me@garden-server:~$ sudo apt install smartmontools
me@garden-server:~$ sudo install -m 755 health-check /usr/local/bin/

Then, from the laptop:

me@garden-laptop:~$ ssh -t server sudo health-check

-t gives the remote command a terminal, which sudo needs to ask for your password ⁹. Make it part of the weekly routine, with the push.

Prove a restore works

A backup is only as good as the last time you restored from it. Every few months, pick a file and bring it back from a snapshot, from the laptop:

me@garden-laptop:~$ ssh server ls /mnt/backup
me@garden-laptop:~$ scp server:/mnt/backup/latest/laptop/Documents/budget.ods /tmp/

Open the copy and check it’s the version you expected. Try an older snapshot’s folder too.

Where to go next

The server is now a dependable Linux machine you manage entirely from the command line, which was the point of this course. Ideas to build on it, each a matter of installing a package and applying what you know (a service, a firewall rule, logs, backups):

• File sharing with your other computers.
• A second backup drive, rotated and kept away from the house.
• The automatic push with a restricted key, as above.
• Monitoring that runs health-check on a timer and saves the report.

Your turn

So

Three copies: the laptop, a mirror on the server pushed weekly with rsync -av --delete over SSH, and 14 nightly snapshots of that mirror on a separate ext4 drive, made by the Shell course’s script on a lingering user timer. A health-check script, run with ssh -t server sudo health-check, reports space, failures, errors, the newest snapshot, and SMART status. Restore a file now and then to prove it all works. That completes the course: a Linux desktop you can install, maintain, and fix, and a server you run entirely from the command line.