No description
Find a file
Josh Bairstow 892601f65b feat(backup): add forgejo dump systemd timer + wire into runbook
Daily 'forgejo dump' (tar.zst) to /opt/backups/forgejo with 14-day retention,
via a oneshot service + timer. Maintenance section now documents both backup
layers (Codeberg push-mirror for git, dump timer for full state).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 00:55:03 +10:00
backup feat(backup): add forgejo dump systemd timer + wire into runbook 2026-06-03 00:55:03 +10:00
runner fix(runner): point unit at /usr/local/sbin, gitignore bare .runner, fix step 5 2026-06-02 23:05:07 +10:00
scripts docs: record live VPS details + DNS-at-Cloudflare caveat; add pre-flight script 2026-06-02 16:13:45 +10:00
.gitignore fix(runner): point unit at /usr/local/sbin, gitignore bare .runner, fix step 5 2026-06-02 23:05:07 +10:00
compose.yml chore: initial VPS edge stack (caddy-docker-proxy + Forgejo) 2026-06-02 15:31:25 +10:00
README.md feat(backup): add forgejo dump systemd timer + wire into runbook 2026-06-03 00:55:03 +10:00

jb-infra

VPS edge stack for joshbairstow.com: a caddy-docker-proxy edge reverse proxy (TLS + label-based routing) and a self-hosted Forgejo git server with CI and a container registry. Everything else — jb-website and future side projects — is deployed by CI and just attaches to the edge network and declares routing labels.

[home desktop]                                  [Vultr VPS]
 Forgejo runner (host exec)                      docker network: edge (external)
  └ docker build  ───push image──►   ┌────────── edge-caddy   :80/:443  (TLS, routing)
  └ ssh deploy    ───pull+up─────►   ├────────── forgejo      git.joshbairstow.com (+registry)
                                     ├────────── jb-website   internal :80
                                     └────────── <future apps> e.g. shop.joshbairstow.com

Why this shape

  • Edge proxy decouples projects. Each project ships its own container with caddy labels; the edge picks them up automatically. Adding a project needs zero edits here — no central config to touch.
  • Build off-box. The RAM-hungry Docker build runs on the always-on desktop runner; the VPS only pulls and runs prebuilt images, so it stays small.
  • Self-hosted git/CI (Forgejo) — independent of GitHub/GitLab, with a cloud push-mirror kept only as backup.

The VPS (provisioned 2026-06-02)

Vultr shared-CPU 2 GB, Debian 13 (trixie), kernel 6.12, ~5.4 GB swap already present (no need to add any). Desktop SSH key installed; Vultr firewall groups applied on both IPv4 and IPv6.

IPv4 45.32.242.27
IPv6 2001:19f0:5801:1313:5400:6ff:fe35:c0a7
SSH root@45.32.242.27 (port 22)

Pre-flight check

Before and after each bootstrap step, run the read-only checker from your desktop to confirm reachability, ports, the box's facts, and DNS:

./scripts/check-vps.sh --ip6 2001:19f0:5801:1313:5400:6ff:fe35:c0a7

It exits 0 when SSH + port 22 are good; closed app ports and unset DNS show as warnings (expected until DNS is updated and the stack is up).

Bootstrap (one-time, in order)

Order matters — TLS issuance needs DNS + open ports first, and the edge network must exist before any stack references it.

1. DNS — managed at Cloudflare, not VentraIP

VentraIP is only the registrar; the domain's nameservers are delegated to Cloudflare (grace/keenan.ns.cloudflare.com), so create the records in the Cloudflare dashboard. Point them at the VPS and wait for propagation:

  • A joshbairstow.com → 45.32.242.27
  • A *.joshbairstow.com → 45.32.242.27 (wildcard: covers blog/art/code/git + future)
  • AAAA joshbairstow.com → 2001:19f0:5801:1313:5400:6ff:fe35:c0a7
  • AAAA *.joshbairstow.com → 2001:19f0:5801:1313:5400:6ff:fe35:c0a7

Set these records to DNS-only (grey cloud), not proxied (orange cloud). The current records point at Cloudflare's proxy. With the proxy ON: Cloudflare terminates TLS with its own edge cert (so Caddy's automatic HTTPS is bypassed), and it only proxies HTTP(S) ports — git-over-SSH on 222 and the registry would break. DNS-only makes clients hit the VPS directly, so Caddy issues real Let's Encrypt certs via HTTP-01 (no DNS-challenge plugin needed) and all ports work. If you later want Cloudflare's proxy/CDN, switch Caddy to the DNS-01 challenge with the Cloudflare plugin + an API token — a separate change.

2. VPS base

Firewall is already handled by the Vultr firewall group (80, 443, 222, 22), so host ufw is optional. Swap already exists. The remaining task is Docker:

# Install Docker Engine + compose plugin
curl -fsSL https://get.docker.com | sh
docker --version && docker compose version   # confirm

(Re-run ./scripts/check-vps.shDOCKER should now report a version.)

3. Get the files onto the VPS + bring up the stack

Forgejo doesn't exist yet, so this repo has no server to clone from during bootstrap. The VPS can't pull from your (NAT'd) desktop either — but your desktop can push to the VPS, and docker compose only needs the files, not git. So copy them up with rsync (run from your desktop, in ~/git/jb-infra):

rsync -av --exclude '.git' --exclude 'runner' --exclude 'scripts' \
  ./ root@45.32.242.27:/opt/jb-infra/

(runner/ and scripts/ are desktop-side tools — they don't belong on the VPS.) Then on the VPS:

cd /opt/jb-infra
docker network create edge          # MUST exist before `up`
docker compose up -d
docker compose logs -f caddy        # watch the first cert issuance

Once Forgejo is live (step 4+), create a jb-infra repo inside it and push this there for its permanent, versioned home; future infra edits then deploy like any other project. If you don't want git history on the box, a one-shot git bundle is an alternative to rsync (see project notes).

4. Configure Forgejo

Browse to https://git.joshbairstow.com (cert should be live). The install page is mostly prepopulated from the env vars in compose.yml — confirm these, then complete it:

  • Run As Username stays git (the container's OS user, not your login).
  • Application URL must be https://git.joshbairstow.com/ (not http://localhost:3000/) — wrong value breaks docker login/push + clone URLs.
  • SSH Server Domain git.joshbairstow.com, SSH Server Port 222, Database SQLite3.
  • Under server settings: enable Disable Self-Registration (personal instance) and Hide Email Addresses by Default; leave "Require Sign-In to View Pages" off (it would block anonymous clones/pulls).
  • Expand Administrator Account Settings and create your admin user with a lowercase username (e.g. josh) — this becomes your registry path git.joshbairstow.com/<owner>/…, which must be lowercase.

Then:

  • Confirm Actions is enabled (Site Administration → Actions).
  • Create a Personal Access Token with write:package scope (Settings → Applications). The automatic Actions token CANNOT push packages.

5. Register the desktop runner (always-on)

On the home desktop (already has Docker + is in the docker group). Put the binary in place, register in a home-dir working folder (snap Docker needs the build context under $HOME), then install the systemd unit so it survives reboots and auto-claims jobs: Run these from the cloned jb-infra repo:

# 1. Binary — skip if already present (check: command -v forgejo-runner).
#    The unit's ExecStart points at /usr/local/sbin/forgejo-runner.
sudo install -m 0755 forgejo-runner /usr/local/sbin/forgejo-runner

# 2. Working dir + config (under $HOME so snap Docker can read build contexts).
mkdir -p ~/forgejo-runner && cp runner/config.yml ~/forgejo-runner/

# 3. Register IN the working dir so .runner lands there (token: Forgejo →
#    Site Admin → Actions → Runners → Create). The subshell keeps cwd in the repo.
( cd ~/forgejo-runner && forgejo-runner register --no-interactive \
    --instance https://git.joshbairstow.com \
    --token <REGISTRATION_TOKEN> \
    --name desktop --labels desktop:host )     # ':host' => run on host, use host Docker

# 4. Install + enable the always-on service.
sudo cp runner/forgejo-runner.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now forgejo-runner
systemctl status forgejo-runner                # expect active (running); runner shows green in Forgejo

Already registered in the wrong directory? Don't re-register — just mv .runner ~/forgejo-runner/ (it's location-independent credentials). The :host label is what lets the docker build step work (the job runs on the host using its Docker, not in a socket-less container). See runner/config.yml and runner/forgejo-runner.service.

The desktop's Docker daemon must be running and enabled at boot — the runner builds against it. docker login works without the daemon but docker build won't, so a stopped daemon shows up as "Cannot connect to the Docker daemon" mid-pipeline. Ensure one daemon owns /var/run/docker.sock:

sudo snap stop --disable docker.dockerd      # if a crashing snap docker is present
sudo systemctl enable --now docker.service   # classic engine, on at boot
docker info | head                           # verify as your user, no sudo

Fedora gotcha: if dockerd fails with ZONE_CONFLICT: 'docker0' already bound to 'FedoraWorkstation', the cause is a leftover NetworkManager bridge profile for docker0 (no zone set → lands in the default zone), which firewalld then re-applies. Delete it, stop NM managing Docker's bridges (so it can't recur at boot), and start Docker:

sudo nmcli connection delete docker0
sudo tee /etc/NetworkManager/conf.d/docker-unmanaged.conf >/dev/null <<'EOF'
[keyfile]
unmanaged-devices=interface-name:docker0;interface-name:br-*
EOF
sudo systemctl reload NetworkManager
sudo ip link delete docker0 2>/dev/null
sudo systemctl reset-failed docker.service && sudo systemctl start docker.service

6. Wire the jb-website repo

In the jb-website repo settings on Forgejo (Settings → Actions):

Variable:

  • REGISTRY_OWNER — your lowercase Forgejo username/org; the image namespace (git.joshbairstow.com/<owner>/jb-website).

Secrets:

  • REGISTRY_USER — your Forgejo login username (the account that owns the PAT); the username for docker login. Usually equals REGISTRY_OWNER.
  • REGISTRY_TOKEN — the PAT with write:package scope; the docker login password.
  • DEPLOY_HOST45.32.242.27.
  • DEPLOY_USER — the VPS account the runner SSHes in as: root to start, or a dedicated deploy user (in the docker group) for least privilege.
  • DEPLOY_SSH_KEY — the private key of a dedicated deploy keypair (full file incl. BEGIN/END lines; paste multi-line — OpenSSH needs the newlines). NOT your personal key. Generate + install its public half on the VPS:
    ssh-keygen -t ed25519 -f ~/.ssh/jb-deploy -C forgejo-ci-deploy -N ""
    ssh-copy-id -i ~/.ssh/jb-deploy.pub <DEPLOY_USER>@45.32.242.27
    cat ~/.ssh/jb-deploy        # <- paste this into DEPLOY_SSH_KEY
    

On the VPS, put the production compose file in place — rsync it from your desktop (creates the dir + copies; only the prod compose.yml, not the dev one):

rsync -av --rsync-path="mkdir -p /opt/jb-website && rsync" \
  ~/git/jb-website/compose.yml root@45.32.242.27:/opt/jb-website/

For docker compose pull to read the image, the package must be pullable. Note: package visibility ≠ repository visibility — packages are owned by the user/org, so making the repo public does NOT necessarily make the image anonymously pullable, and the package only exists after the first push. After the first pipeline run, verify from the VPS:

docker pull git.joshbairstow.com/<owner>/jb-website:latest
# unauthorized? -> set the PACKAGE public (profile → Packages → jb-website →
# settings), or run `docker login git.joshbairstow.com` once as DEPLOY_USER.

If DEPLOY_USER is not root, it must be in the docker group; for least privilege, restrict its authorized_keys entry with a forced command=.

7. First deploy

# in the jb-website repo (default branch is already `main`).
# Custom SSH port 222 needs the ssh:// URL form (scp-style host:port doesn't work):
git remote add origin ssh://git@git.joshbairstow.com:222/<owner>/jb-website.git
git push -u origin main

The push triggers .forgejo/workflows/deploy.yml: the desktop builds + pushes the image, then SSHes to the VPS to pull && up -d. The edge issues certs for the apex + subdomains on first request.


Adding another project later

No changes to this repo. In the new project:

  1. Ship a compose.yml on the VPS that joins the external edge network and carries labels, e.g.:
    services:
      app:
        image: git.joshbairstow.com/<owner>/<app>:latest
        networks: [edge]
        labels:
          caddy: shop.joshbairstow.com
          caddy.reverse_proxy: "{{upstreams 3000}}"
    networks:
      edge:
        external: true
    
  2. Give it its own .forgejo/workflows/deploy.yml (copy jb-website's).
  3. *.joshbairstow.com already resolves, so the edge issues its cert automatically on first hit.

Maintenance / hardening

  • Backups: the forgejo_data volume is now the source of truth (repos, DB, registry blobs). Two layers:
    • Off-box git: a push-mirror of each repo to Codeberg (repo → Settings → Mirror Settings → Add Push Mirror; HTTPS URL + a write:repository token).
    • Full dump: the systemd timer in backup/ runs forgejo dump daily and keeps 14 days under /opt/backups/forgejo. Install on the VPS:
      sudo install -m 0755 backup/forgejo-backup.sh /usr/local/bin/forgejo-backup.sh
      sudo cp backup/forgejo-backup.{service,timer} /etc/systemd/system/
      sudo systemctl daemon-reload && sudo systemctl enable --now forgejo-backup.timer
      sudo systemctl start forgejo-backup.service   # test once
      ls -lh /opt/backups/forgejo                   # confirm an archive landed
      systemctl list-timers forgejo-backup.timer    # confirm next run scheduled
      
      Copy the dumps off-box too (they're useless if the VPS dies). Add --skip-package in the script to exclude registry blobs if dumps get large.
  • Registry GC: per-commit :sha tags accumulate. Enable Forgejo's package cleanup rules / run periodic GC so the VPS disk doesn't fill.
  • Certs: never delete the caddy_data volume — losing it re-issues every cert and risks Let's Encrypt rate limits.
  • Least privilege: restrict the deploy key in authorized_keys with a forced command= that only runs the compose pull/up script.
  • Reset the admin password (no email/old password needed): as an admin via Site Administration → User Accounts → (user) → Password, or from the VPS:
    docker compose exec -u git forgejo \
      forgejo admin user change-password --username <user> \
      --password 'NEW' --must-change-password=false