Bonham Lab Wiki

thoreau architecture

Properties2
tagscomputing, data, drylab
typeinformation

3 min read

Backend host for the Bonham Lab self-hosted services. Replaced Dogen lab server as the services backend in Sept 2026. OpenSUSE Tumbleweed, BTRFS throughout.

See also:

Role

thoreau runs the containerised service stack (Forgejo, Vaultwarden, Paperless, Postgres, Redis) and holds the lab’s data and backups. It sits behind the Hetzner VPS, reachable only through the WireGuard tunnel — the VPS is the internet-facing edge; thoreau has no public exposure.

Internet → Hetzner VPS (167.233.140.224)
              │  SSLH :443 ─ HTTPS ─→ nginx ─┐
              │             ─ SSH  ─→ tunnel ─┤
              │  WireGuard :51820/udp         │
              └──────────────────────────────┤
                                    WireGuard tunnel
                                              │
                              thoreau (10.100.0.2)
                                ├─ Docker Compose stack
                                └─ reverse SSH tunnel → VPS :2222 (Forgejo SSH)

Hardware & storage

2× 2 TB NVMe, 4× 4 TB HDD. BTRFS, with redundancy only where it’s needed: data is protected, the OS is disposable-and-reinstallable.

Pool / diskFilesystemRedundancyHolds
NVMe nvme0n1 (25153K...)systemnone (single)OS root, @home, @services
NVMe nvme1n1 (26035Z...)worknone (single)@scratch, 32 GB swap
4× HDD (raid1)storeraid1 (~8 TB usable)@data, @backup, @registry, @media

The NVMe/HDD split is deliberate: services and homes want speed and don’t need mirroring (services are backed up, homes are small/replaceable); the bulk data pool gets raid1. See thoreau Backup and Restore for why services aren’t mirrored.

Subvolumes

SubvolumeMountPoolNotes
@/system NVMeOS root; snapper+GRUB rollback
@home/homesystem NVMesmall fast homes
@services/srv/servicessystem NVMeservice data + DB dumps; backed up to @backup
@scratch/scratchwork NVMedisposable analysis workspace
@data/store/dataHDD raid1lab data; local snapshots (undo)
@backup/store/backupHDD raid1receives @services snapshots
@registry/store/registryHDD raid1container registry (reproducible)
@media/store/mediaHDD raid1Paperless docs / Immich (irreplaceable)

/mnt/btrbk/srv and /mnt/btrbk/store are backup-tool-only mounts of the filesystem roots (subvolid=5); ignore them in normal use.

Services

Docker Compose stack at /srv/docker (compose.yml + .env). Managed by the selfhosted.service systemd unit. All service data bind-mounts live under /srv/services/*, except Paperless’s document store which is on /store/media/paperless (bulk, on the raid1 pool).

ServiceDomainData location
Forgejocode.bonhamlab.bio/srv/services/forgejo
Vaultwardenpw.bonhamlab.bio/srv/services/vaultwarden (Postgres)
Paperlessdocuments.bonhamlab.bio/srv/services/paperless + /store/media/paperless
Postgres(internal)/srv/services/postgres
Redis(internal)/srv/services/redis
Forgejo runner—/srv/services/forgejo-runner

Each service manages its own auth (local accounts + 2FA); there is no shared SSO. Credentials live in Vaultwarden. Docker data-root is /srv/services/docker, storage driver btrfs.

Networking

  • WireGuard (wg0, 10.100.0.2): the tunnel to the VPS. Config at /etc/wireguard/wg0.conf. thoreau reuses Dogen lab server’s key/identity, so the VPS needed no change at cutover.
  • Reverse SSH tunnel (forgejo-tunnel.service): forwards the VPS’s localhost:2222 to Forgejo’s SSH, so git over ssh://…:443 works via SSLH. Uses /etc/wireguard/tunnel_key.
  • Services bind to 10.100.0.2 only. Nothing is reachable without the tunnel.

Boot dependencies

selfhosted.service requires docker.service and wg-quick@wg0.service, and waits for both before starting the stack (with a short settle delay to avoid an iptables/firewalld startup race). Restart=on-failure self-heals the occasional lock race. The stack survives reboot unattended (verified).

Migration lineage

Migrated from Dogen lab server (also OpenSUSE/BTRFS) via btrfs-preserving tar of the service data + raw Postgres data dir, verified by row/file counts. dogen is retained intact as a rollback option until decommissioned.

Design decisions (why)

  • BTRFS, not ZFS: capacity difference (raid1 50% vs raidz 75%) was ~4 TB at this scale — not worth leaving the in-tree filesystem and taking on a DKMS module. BTRFS raid1 also expands one drive-pair at a time, which fits “add drives as we grow” better than raidz.
  • Tumbleweed, not Leap/Rocky: staying on the in-tree BTRFS distro keeps snapper+GRUB rollback (best on OpenSUSE) and parity with Dogen lab server. The rolling-kernel risk is covered by that rollback.
  • OS not mirrored: reproducible and documented → reinstall is cheap; mirror would spend a 2 TB NVMe to save an hour. Data redundancy is what matters.

Graph View

Backlinks