Bonham Lab Wiki

thoreau Backup and Restore

Properties2
tagscomputing, data
typeinformation

4 min read

How backups work on thoreau and how to restore.

See also:

The scheme in one picture

daily 03:00 (btrbk.timer)
  │
  ├─ services-predump.service   pg_dump each DB → /srv/services/postgres/dumps/
  │        │  (must finish first — btrbk waits on it, not on a timer)
  ▼        ▼
  btrbk.service
     ├─ snapshot @services (NVMe) → send to @backup (HDD raid1)   ← disaster copy
     └─ snapshot @data     (HDD)  → local only                    ← undo button
  • @services → @backup: the disaster copy. Survives loss of the NVMe. Each snapshot contains the raw service files and the SQL dumps, so it’s restore-ready even if the raw Postgres files won’t cleanly recover.
  • @data local snapshots: the “undo button” for accidental deletion or overwrite. Fast, local, no network. Does not protect against losing the box.

Retention: @services keeps 7 daily + 5 weekly + 3 monthly (~90 days). @data keeps 7 daily + 5 weekly (~35 days — covers a month-long absence).

Not yet covered

  • Off-site: everything currently lives on thoreau. A fire / theft / dual-disk failure loses all of it. Off-site replication of @backup, @data, and @media is still to be built — the biggest open gap, especially given month-long unattended periods.
  • @media (Paperless / Immich) has no backup in this scheme — excluded as high-churn bulk. Its protection is meant to be off-site + app-level trash. Until off-site exists, @media is unprotected.
  • Restore has not been tested end-to-end. Do a throwaway restore (below) before trusting these backups with real stakes.

Important Files

PathWhat
/etc/btrbk/btrbk.confbtrbk config (sources, targets, retention)
/usr/local/bin/services-predump.shDB dump script
/etc/systemd/system/services-predump.servicedump unit
/etc/systemd/system/btrbk.servicebtrbk run (chained after dump)
/etc/systemd/system/btrbk.timerdaily 03:00 trigger
/srv/services/postgres/dumps/latest SQL dumps
/store/backup/@backup/received @services snapshots (disaster copy)
/mnt/btrbk/{srv,store}tool-only fs-root mounts btrbk operates through

Routine operations

Run a backup now (dump → snapshot → send):

sudo systemctl start btrbk.service
journalctl -u services-predump.service -u btrbk.service --since "5 min ago"

Dry run (plan only, no changes):

sudo btrbk -c /etc/btrbk/btrbk.conf -n run

List snapshots / backups:

sudo btrbk -c /etc/btrbk/btrbk.conf list snapshots
sudo btrfs subvolume list /mnt/btrbk/store | grep @backup

Check the timer:

systemctl list-timers btrbk.timer

Restore — undo an accidental deletion in @data

Local snapshots are directly readable. Copy the file back out:

ls /mnt/btrbk/store/btrbk-snapshots/          # find the dated snapshot
cp /mnt/btrbk/store/btrbk-snapshots/@data.<DATE>/path/to/file /store/data/path/to/file

No btrbk needed — snapshots are just read-only subvolumes.

Restore — a single database

From the dump inside the latest snapshot (or the live dumps dir):

# forgejo example — stop the app first so nothing writes mid-restore
cd /srv/docker && docker compose stop forgejo
docker exec -i selfhosted-postgres-1 psql -U postgres forgejo \
    < /srv/services/postgres/dumps/forgejo.sql
docker compose start forgejo

The dumps use --clean --if-exists, so they drop and recreate objects — safe to re-run into an existing database.

Restore — full services after losing the NVMe

  1. Reinstall/replace the system disk; recreate the @services subvolume and its mount (see thoreau architecture).

  2. Send the latest @services snapshot back from the HDD copy:

    btrfs send /mnt/btrbk/store/@backup/@services.<DATE> \  | btrfs receive /mnt/btrbk/srv/# then position/rename it as the live @services mount

  3. If the raw Postgres files won’t start cleanly, restore roles then each DB from the dumps carried inside that snapshot:

    docker exec -i selfhosted-postgres-1 psql -U postgres < .../globals.sqldocker exec -i selfhosted-postgres-1 psql -U postgres forgejo     < .../forgejo.sqldocker exec -i selfhosted-postgres-1 psql -U postgres paperless   < .../paperless.sqldocker exec -i selfhosted-postgres-1 psql -U postgres vaultwarden < .../vaultwarden.sql

  4. docker compose up -d, then verify per thoreau Maintenance health checks.

Test a restore

Spin up a throwaway Postgres and load a dump. This proves the dumps are valid without touching production:

docker run --rm -d --name pgtest -e POSTGRES_PASSWORD=test postgres:16-alpine
docker exec -i pgtest psql -U postgres < /srv/services/postgres/dumps/globals.sql
docker exec -i pgtest psql -U postgres -c 'CREATE DATABASE forgejo;'
docker exec -i pgtest psql -U postgres forgejo < /srv/services/postgres/dumps/forgejo.sql
docker exec pgtest psql -U postgres forgejo -c 'SELECT count(*) FROM repository;'  # sanity
docker stop pgtest

A sensible row count means the dump can restore. This is the only way to know the backups work for real

Graph View

Backlinks