Self-hosted lab services

The lab runs a set of self-hosted services including:

Forgejo for hosting git repos (see also Coding and version control)
Paperless for storing and characterizing documents
Vaultwarden for password management

For new users, see Lab services onboarding.

Architecture Overview

Two-machine setup connected via an encrypted WireGuard tunnel:

Internet → Hetzner VPS (167.233.140.224)
               │
               ├── SSLH (port 443) → SSH-over-443 → WireGuard → Backend:2222 (Forgejo SSH)
               │                  ↘ HTTPS → Nginx (port 8443)
               │                               │
               │                               ├── pw.bonhamlab.bio        → Vaultwarden
               │                               ├── documents.bonhamlab.bio → Paperless
               │                               └── code.bonhamlab.bio      → Forgejo
               │
               └── WireGuard (port 51820/udp) ──── Backend (10.100.0.2)
                                                        │
                                                        └── Docker Compose
                                                            ├── Vaultwarden    :8080
                                                            ├── Paperless      :8000
                                                            ├── Forgejo        :3000/:2222
                                                            ├── Forgejo runner (CI)
                                                            ├── Postgres       (internal)
                                                            └── Redis          (internal)

Key Design Decisions

The VPS is a dumb proxy — it holds no application data. All services run on the backend.
Services bind to 10.100.0.2 (the WireGuard interface) only — not reachable from the public internet directly.
SSLH multiplexes port 443: SSH handshakes go to Forgejo’s built-in SSH server via the reverse tunnel; TLS goes to Nginx.
The backend is portable — migrating to new hardware means copying /srv/docker and /etc/wireguard/wg0.conf. The VPS config requires zero changes.
This machine is currently dogen, but will eventually be its own dedicated system.

WireGuard Addressing

Host	WireGuard IP
Hetzner VPS	`10.100.0.1`
Backend	`10.100.0.2`

Domain Structure

Subdomain	Service	Auth
`pw.bonhamlab.bio`	Vaultwarden	Vaultwarden native (local accounts)
`documents.bonhamlab.bio`	Paperless-ngx	Local accounts + 2FA (TOTP)
`code.bonhamlab.bio`	Forgejo	Local accounts + 2FA enforced instance-wide
`bonhamlab.bio`	Astro website	Public (static, no auth)
`wiki.bonhamlab.bio`	Quartz wiki	Public (static, no auth)

Authentication

Each service manages its own authentication independently:

Vaultwarden: local accounts with master password + optional 2FA (TOTP). Admin panel at /admin requires the admin token stored in Vaultwarden.
Forgejo: local accounts with password + 2FA (TOTP, enforced for all users via GLOBAL_TWO_FACTOR_REQUIREMENT = all in app.ini). GitHub and GitLab OAuth2 also available for collaborators who prefer it.
Paperless: local accounts with password + 2FA (TOTP, required by policy but not enforced at the app level — enforced during onboarding).

There is no shared SSO layer. Credentials for each service are stored in Vaultwarden.

Service Management

Backend

All Docker Compose commands run from /srv/docker.

Action	Command
Start all services	`cd /srv/docker && docker compose up -d`
Stop all services	`cd /srv/docker && docker compose down`
Restart a service	`cd /srv/docker && docker compose restart <service>`
Force recreate (picks up env changes)	`cd /srv/docker && docker compose down && docker compose up -d`
View logs	`cd /srv/docker && docker compose logs -f <service>`
Check status	`cd /srv/docker && docker compose ps`
Update images	`cd /srv/docker && docker compose pull && docker compose up -d`

Service names: postgres, redis, vaultwarden, paperless, forgejo, forgejo-runner

Action	Command
WireGuard status	`sudo wg show`
Start WireGuard	`sudo systemctl start wg-quick@wg0`
Stop WireGuard	`sudo systemctl stop wg-quick@wg0`
Forgejo SSH tunnel status	`sudo systemctl status forgejo-tunnel.service`
Restart Forgejo SSH tunnel	`sudo systemctl restart forgejo-tunnel.service`
Tunnel logs	`sudo journalctl -u forgejo-tunnel.service --no-pager -n 50`

VPS

Action	Command
WireGuard status	`sudo wg show`
Nginx reload (no downtime)	`sudo systemctl reload nginx`
Nginx restart	`sudo systemctl restart nginx`
Nginx config test	`sudo nginx -t`
Nginx error logs	`sudo tail -50 /var/log/nginx/error.log`
SSLH restart	`sudo systemctl restart sslh`
Check port 443 listener	`sudo ss -tlnp \| grep 443`
Check tunnel port	`sudo ss -tlnp \| grep 2222`

Admin Operations

Reset a user’s Forgejo password (when locked out):

docker exec -it --user 1000 selfhosted-forgejo-1 forgejo admin user change-password --username <username> --password '<newpassword>'

Reset a user’s Paperless password:

docker exec -it selfhosted-paperless-1 python3 manage.py changepassword <username>

Reset Forgejo 2FA (if user loses their device): Admin panel → Site Administration → User Accounts → edit user → Reset two-factor authentication

Vaultwarden admin panel: https://pw.bonhamlab.bio/admin — requires the admin token stored in Vaultwarden.

Adding a New User

See Lab services onboarding for the full user-facing procedure.

Admin steps before sending the user anything:

Invite their email in the Vaultwarden admin panel (/admin → Users → Invite)
Create a Forgejo account: Site Administration → User Accounts → Create User; set a temporary password
Create a Paperless account: https://documents.bonhamlab.bio/admin/auth/user/add/
Share all temporary credentials via Vaultwarden Send (https://pw.bonhamlab.bio/#/send) — never via email or chat

Adding a New Service

Add DNS A record → 167.233.140.224 in Porkbun
On VPS: sudo certbot certonly --nginx -d newservice.bonhamlab.bio

On VPS: create /etc/nginx/sites-available/newservice.bonhamlab.bio, symlink to sites-enabled

Minimal template (no auth gate — each service handles its own auth):

server {
    listen 8443 ssl;
    http2 on;
    server_name newservice.bonhamlab.bio;
 
    ssl_certificate     /etc/letsencrypt/live/newservice.bonhamlab.bio/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/newservice.bonhamlab.bio/privkey.pem;
    include /etc/nginx/snippets/ssl-params.conf;
 
    client_max_body_size 100M;
 
    location / {
        proxy_pass http://10.100.0.2:<PORT>;
        include /etc/nginx/snippets/proxy-params.conf;
    }
}

On VPS: sudo nginx -t && sudo systemctl reload nginx
On backend: add service block to /srv/docker/compose.yml

On backend: if Postgres needed, create DB manually (init script only runs on first Postgres boot):

docker exec -it selfhosted-postgres-1 psql -U postgres -c "CREATE USER newservice WITH PASSWORD 'password';"
docker exec -it selfhosted-postgres-1 psql -U postgres -c "CREATE DATABASE newservice OWNER newservice;"

Also add to /srv/docker/postgres/initdb/01-create-dbs.sh for future fresh installs.

On backend: create data directory with correct ownership for the container’s uid:

sudo mkdir -p /srv/docker/newservice/data
sudo chown -R <uid>:<gid> /srv/docker/newservice/data

On backend: cd /srv/docker && docker compose up -d newservice
Verify binding: sudo ss -tlnp | grep <PORT> — should show 10.100.0.2:<PORT>

Static Sites

Two static sites are hosted directly by Nginx on the VPS (no backend container needed):

Site	Path on VPS	Repo	Build trigger
`bonhamlab.bio`	`/var/www/bonhamlab.bio`	`Web/bonhamlab.bio`	Push to `main` → Forgejo Actions → rsync
`wiki.bonhamlab.bio`	`/var/www/wiki.bonhamlab.bio`	`Internal/ObsidianVault`	Push to `main` → Forgejo Actions → clone `Web/wiki-backend` → build Quartz → rsync

Deploy SSH key is stored at /srv/docker/forgejo-runner/deploy_key on the backend (used by the runner to rsync to the VPS). The corresponding public key is in /home/kevin/.ssh/authorized_keys on the VPS, restricted via rrsync -wo /var/www/.

Files & Paths

Hetzner VPS

Path	Purpose
`/etc/wireguard/wg0.conf`	WireGuard interface config
`/etc/wireguard/private.key`	WireGuard private key (root, 600)
`/etc/wireguard/public.key`	WireGuard public key
`/etc/nginx/sites-available/pw.bonhamlab.bio`	Vaultwarden Nginx vhost
`/etc/nginx/sites-available/documents.bonhamlab.bio`	Paperless Nginx vhost
`/etc/nginx/sites-available/code.bonhamlab.bio`	Forgejo Nginx vhost
`/etc/nginx/sites-available/bonhamlab.bio`	Main website Nginx vhost
`/etc/nginx/sites-available/wiki.bonhamlab.bio`	Wiki Nginx vhost
`/etc/nginx/snippets/ssl-params.conf`	Shared TLS settings
`/etc/nginx/snippets/proxy-params.conf`	Shared proxy headers
`/etc/letsencrypt/live/pw.bonhamlab.bio/`	TLS cert (shared by code, auth subdomains)
`/etc/letsencrypt/live/documents.bonhamlab.bio/`	TLS cert for documents subdomain
`/etc/letsencrypt/live/bonhamlab.bio/`	TLS cert for apex + www
`/etc/letsencrypt/live/wiki.bonhamlab.bio/`	TLS cert for wiki
`/etc/default/sslh`	SSLH daemon config
`/var/www/bonhamlab.bio/`	Built Astro site files
`/var/www/wiki.bonhamlab.bio/`	Built Quartz wiki files

Backend (OpenSUSE Tumbleweed)

Path	Purpose
`/etc/wireguard/wg0.conf`	WireGuard interface config
`/etc/wireguard/private.key`	WireGuard private key (root, 600)
`/etc/wireguard/public.key`	WireGuard public key
`/etc/wireguard/tunnel_key`	SSH private key for Forgejo reverse tunnel (root, 600)
`/etc/wireguard/tunnel_key.pub`	SSH public key for Forgejo reverse tunnel
`/etc/systemd/system/selfhosted.service`	Docker Compose systemd unit
`/etc/systemd/system/forgejo-tunnel.service`	Forgejo SSH reverse tunnel systemd unit
`/srv/docker/compose.yml`	Docker Compose stack definition
`/srv/docker/.env`	Secrets and environment variables (selfhosted:docker, 640)
`/srv/docker/postgres/data/`	Postgres data volume (root, 700)
`/srv/docker/postgres/initdb/01-create-dbs.sh`	One-time DB/user init script (runs on fresh Postgres only)
`/srv/docker/redis/data/`	Redis data volume
`/srv/docker/vaultwarden/data/`	Vaultwarden data volume
`/srv/docker/paperless/data/`	Paperless data volume
`/srv/docker/paperless/media/`	Paperless media volume
`/srv/docker/paperless/consume/`	Paperless document ingestion drop folder (777)
`/srv/docker/forgejo/data/`	Forgejo data volume
`/srv/docker/forgejo/data/gitea/conf/app.ini`	Forgejo runtime config (managed by container; env vars take precedence)
`/srv/docker/forgejo-runner/config.yml`	Forgejo Actions runner connection config
`/srv/docker/forgejo-runner/data/`	Forgejo runner working data
`/srv/docker/forgejo-runner/deploy_key`	SSH private key for rsync deployment to VPS (root, 600)

Secrets & Keys

WireGuard Keys

Generated on each machine independently:

sudo sh -c 'umask 077; wg genkey | tee /etc/wireguard/private.key | wg pubkey > /etc/wireguard/public.key'

VPS private key → PrivateKey in /etc/wireguard/wg0.conf [Interface]
VPS public key → PublicKey in backend /etc/wireguard/wg0.conf [Peer]
Backend private key → PrivateKey in backend /etc/wireguard/wg0.conf [Interface]
Backend public key → PublicKey in VPS /etc/wireguard/wg0.conf [Peer]

Forgejo Tunnel SSH Key

Generated on the backend:

sudo ssh-keygen -t ed25519 -f /etc/wireguard/tunnel_key -N "" -C "forgejo-tunnel"

Private key stays at /etc/wireguard/tunnel_key. Public key goes into /home/kevin/.ssh/authorized_keys on the VPS, restricted to port-forwarding only:

command="echo 'no shell'",no-pty,no-agent-forwarding,permitopen="localhost:2222" ssh-ed25519 ...

Deploy SSH Key

Generated on the backend for static site deployment:

sudo ssh-keygen -t ed25519 -f /srv/docker/forgejo-runner/deploy_key -N "" -C "site-deploy"

Public key in /home/kevin/.ssh/authorized_keys on VPS, restricted to rsync write-only under /var/www/:

command="rrsync -wo /var/www/",restrict ssh-ed25519 ...

Used in Forgejo Actions workflows via the DEPLOY_SSH_KEY secret (set per-repo, not org-wide).

Docker Compose Secrets (in `/srv/docker/.env`)

Variable	Generated with	Purpose
`POSTGRES_PASSWORD`	`openssl rand -base64 32`	Postgres superuser password
`VAULTWARDEN_DB_PASS`	`openssl rand -base64 32`	Vaultwarden DB user password
`PAPERLESS_DB_PASS`	`openssl rand -base64 32`	Paperless DB user password
`FORGEJO_DB_PASS`	`openssl rand -base64 32`	Forgejo DB user password
`PAPERLESS_SECRET_KEY`	`openssl rand -base64 48`	Paperless Django secret key
`VAULTWARDEN_ADMIN_TOKEN`	`docker run --rm -it vaultwarden/server /vaultwarden hash --profile owasp`	Vaultwarden admin panel token (argon2 hash)

The raw Vaultwarden admin token (plaintext, pre-hash) is stored in Vaultwarden itself.

Migration to New Hardware

The VPS requires zero changes. On the new backend machine:

Install Debian/OpenSUSE, Docker, WireGuard tools
rsync -aAX /srv/docker/ newhost:/srv/docker/
Copy /etc/wireguard/wg0.conf — same private key, same IP 10.100.0.2
Copy /etc/systemd/system/selfhosted.service and forgejo-tunnel.service
Copy /etc/wireguard/tunnel_key and /etc/wireguard/tunnel_key.pub
sudo systemctl enable --now wg-quick@wg0
sudo systemctl enable --now selfhosted.service
sudo systemctl enable --now forgejo-tunnel.service
Verify: sudo wg show and cd /srv/docker && docker compose ps

tags	computing, web, admin, infrastructure
type	resource

Bonham Lab Wiki

Explorer

Self-hosted lab services

Architecture Overview

Key Design Decisions

WireGuard Addressing

Domain Structure

Authentication

Service Management

Backend

VPS

Admin Operations

Adding a New User

Adding a New Service

Static Sites

Files & Paths

Hetzner VPS

Backend (OpenSUSE Tumbleweed)

Secrets & Keys

WireGuard Keys

Forgejo Tunnel SSH Key

Deploy SSH Key

Docker Compose Secrets (in `/srv/docker/.env`)

Migration to New Hardware

Graph View

Table of Contents

Backlinks

Bonham Lab Wiki

Explorer

Self-hosted lab services

Architecture Overview

Key Design Decisions

WireGuard Addressing

Domain Structure

Authentication

Service Management

Backend

VPS

Admin Operations

Adding a New User

Adding a New Service

Static Sites

Files & Paths

Hetzner VPS

Backend (OpenSUSE Tumbleweed)

Secrets & Keys

WireGuard Keys

Forgejo Tunnel SSH Key

Deploy SSH Key

Docker Compose Secrets (in /srv/docker/.env)

Migration to New Hardware

Graph View

Table of Contents

Backlinks

Docker Compose Secrets (in `/srv/docker/.env`)