jb-infra/README.md
Josh Bairstow d02844366a feat(runner): add always-on systemd unit + wire into runbook step 5
System service running as the desktop user (already in docker group); starts
at boot, auto-restarts, auto-claims queued jobs. WorkingDirectory under $HOME
because snap Docker can't read build contexts outside home. README step 5 now
covers binary install, registration, and enabling the service.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 22:42:52 +10:00

10 KiB

jb-infra

VPS edge stack for joshbairstow.com: a caddy-docker-proxy edge reverse proxy (TLS + label-based routing) and a self-hosted Forgejo git server with CI and a container registry. Everything else — jb-website and future side projects — is deployed by CI and just attaches to the edge network and declares routing labels.

[home desktop]                                  [Vultr VPS]
 Forgejo runner (host exec)                      docker network: edge (external)
  └ docker build  ───push image──►   ┌────────── edge-caddy   :80/:443  (TLS, routing)
  └ ssh deploy    ───pull+up─────►   ├────────── forgejo      git.joshbairstow.com (+registry)
                                     ├────────── jb-website   internal :80
                                     └────────── <future apps> e.g. shop.joshbairstow.com

Why this shape

  • Edge proxy decouples projects. Each project ships its own container with caddy labels; the edge picks them up automatically. Adding a project needs zero edits here — no central config to touch.
  • Build off-box. The RAM-hungry Docker build runs on the always-on desktop runner; the VPS only pulls and runs prebuilt images, so it stays small.
  • Self-hosted git/CI (Forgejo) — independent of GitHub/GitLab, with a cloud push-mirror kept only as backup.

The VPS (provisioned 2026-06-02)

Vultr shared-CPU 2 GB, Debian 13 (trixie), kernel 6.12, ~5.4 GB swap already present (no need to add any). Desktop SSH key installed; Vultr firewall groups applied on both IPv4 and IPv6.

IPv4 45.32.242.27
IPv6 2001:19f0:5801:1313:5400:6ff:fe35:c0a7
SSH root@45.32.242.27 (port 22)

Pre-flight check

Before and after each bootstrap step, run the read-only checker from your desktop to confirm reachability, ports, the box's facts, and DNS:

./scripts/check-vps.sh --ip6 2001:19f0:5801:1313:5400:6ff:fe35:c0a7

It exits 0 when SSH + port 22 are good; closed app ports and unset DNS show as warnings (expected until DNS is updated and the stack is up).

Bootstrap (one-time, in order)

Order matters — TLS issuance needs DNS + open ports first, and the edge network must exist before any stack references it.

1. DNS — managed at Cloudflare, not VentraIP

VentraIP is only the registrar; the domain's nameservers are delegated to Cloudflare (grace/keenan.ns.cloudflare.com), so create the records in the Cloudflare dashboard. Point them at the VPS and wait for propagation:

  • A joshbairstow.com → 45.32.242.27
  • A *.joshbairstow.com → 45.32.242.27 (wildcard: covers blog/art/code/git + future)
  • AAAA joshbairstow.com → 2001:19f0:5801:1313:5400:6ff:fe35:c0a7
  • AAAA *.joshbairstow.com → 2001:19f0:5801:1313:5400:6ff:fe35:c0a7

Set these records to DNS-only (grey cloud), not proxied (orange cloud). The current records point at Cloudflare's proxy. With the proxy ON: Cloudflare terminates TLS with its own edge cert (so Caddy's automatic HTTPS is bypassed), and it only proxies HTTP(S) ports — git-over-SSH on 222 and the registry would break. DNS-only makes clients hit the VPS directly, so Caddy issues real Let's Encrypt certs via HTTP-01 (no DNS-challenge plugin needed) and all ports work. If you later want Cloudflare's proxy/CDN, switch Caddy to the DNS-01 challenge with the Cloudflare plugin + an API token — a separate change.

2. VPS base

Firewall is already handled by the Vultr firewall group (80, 443, 222, 22), so host ufw is optional. Swap already exists. The remaining task is Docker:

# Install Docker Engine + compose plugin
curl -fsSL https://get.docker.com | sh
docker --version && docker compose version   # confirm

(Re-run ./scripts/check-vps.shDOCKER should now report a version.)

3. Get the files onto the VPS + bring up the stack

Forgejo doesn't exist yet, so this repo has no server to clone from during bootstrap. The VPS can't pull from your (NAT'd) desktop either — but your desktop can push to the VPS, and docker compose only needs the files, not git. So copy them up with rsync (run from your desktop, in ~/git/jb-infra):

rsync -av --exclude '.git' --exclude 'runner' --exclude 'scripts' \
  ./ root@45.32.242.27:/opt/jb-infra/

(runner/ and scripts/ are desktop-side tools — they don't belong on the VPS.) Then on the VPS:

cd /opt/jb-infra
docker network create edge          # MUST exist before `up`
docker compose up -d
docker compose logs -f caddy        # watch the first cert issuance

Once Forgejo is live (step 4+), create a jb-infra repo inside it and push this there for its permanent, versioned home; future infra edits then deploy like any other project. If you don't want git history on the box, a one-shot git bundle is an alternative to rsync (see project notes).

4. Configure Forgejo

Browse to https://git.joshbairstow.com (cert should be live). The install page is mostly prepopulated from the env vars in compose.yml — confirm these, then complete it:

  • Run As Username stays git (the container's OS user, not your login).
  • Application URL must be https://git.joshbairstow.com/ (not http://localhost:3000/) — wrong value breaks docker login/push + clone URLs.
  • SSH Server Domain git.joshbairstow.com, SSH Server Port 222, Database SQLite3.
  • Under server settings: enable Disable Self-Registration (personal instance) and Hide Email Addresses by Default; leave "Require Sign-In to View Pages" off (it would block anonymous clones/pulls).
  • Expand Administrator Account Settings and create your admin user with a lowercase username (e.g. josh) — this becomes your registry path git.joshbairstow.com/<owner>/…, which must be lowercase.

Then:

  • Confirm Actions is enabled (Site Administration → Actions).
  • Create a Personal Access Token with write:package scope (Settings → Applications). The automatic Actions token CANNOT push packages.

5. Register the desktop runner (always-on)

On the home desktop (already has Docker + is in the docker group). Put the binary in place, register in a home-dir working folder (snap Docker needs the build context under $HOME), then install the systemd unit so it survives reboots and auto-claims jobs:

sudo install -m 0755 forgejo-runner /usr/local/bin/forgejo-runner
mkdir -p ~/forgejo-runner && cp runner/config.yml ~/forgejo-runner/
cd ~/forgejo-runner

# Registration token: Forgejo → Site Admin → Actions → Runners → Create
forgejo-runner register --no-interactive \
  --instance https://git.joshbairstow.com \
  --token <REGISTRATION_TOKEN> \
  --name desktop --labels desktop:host        # ':host' => run jobs on the host, use host Docker

sudo cp runner/forgejo-runner.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now forgejo-runner
systemctl status forgejo-runner               # expect active (running); runner shows green in Forgejo

The :host label is what lets the docker build step work (the job runs on the host using its Docker, not in a socket-less container). See runner/config.yml and runner/forgejo-runner.service.

6. Wire the jb-website repo

In the jb-website repo settings on Forgejo (Settings → Actions):

  • Variables: REGISTRY_OWNER = your lowercase owner.
  • Secrets: REGISTRY_USER, REGISTRY_TOKEN (the PAT), DEPLOY_HOST (45.32.242.27), DEPLOY_USER, DEPLOY_SSH_KEY.

On the VPS, prepare the deploy target:

mkdir -p /opt/jb-website
# copy jb-website/compose.yml here (the production one — pulls, no build)
docker login git.joshbairstow.com      # one-time, so `compose pull` can read the package
# (or mark the jb-website package public in Forgejo to skip VPS login)

Create a restricted deploy user whose authorized_keys holds the public half of DEPLOY_SSH_KEY.

7. First deploy

# in the jb-website repo (default branch is already `main`).
# Custom SSH port 222 needs the ssh:// URL form (scp-style host:port doesn't work):
git remote add origin ssh://git@git.joshbairstow.com:222/<owner>/jb-website.git
git push -u origin main

The push triggers .forgejo/workflows/deploy.yml: the desktop builds + pushes the image, then SSHes to the VPS to pull && up -d. The edge issues certs for the apex + subdomains on first request.


Adding another project later

No changes to this repo. In the new project:

  1. Ship a compose.yml on the VPS that joins the external edge network and carries labels, e.g.:
    services:
      app:
        image: git.joshbairstow.com/<owner>/<app>:latest
        networks: [edge]
        labels:
          caddy: shop.joshbairstow.com
          caddy.reverse_proxy: "{{upstreams 3000}}"
    networks:
      edge:
        external: true
    
  2. Give it its own .forgejo/workflows/deploy.yml (copy jb-website's).
  3. *.joshbairstow.com already resolves, so the edge issues its cert automatically on first hit.

Maintenance / hardening

  • Backups: the forgejo_data volume is now the source of truth (repos, DB, registry blobs). Run docker compose exec forgejo forgejo dump on a schedule, and set a push-mirror on each repo to a cloud host (GitLab/GitHub) for off-box git backup.
  • Registry GC: per-commit :sha tags accumulate. Enable Forgejo's package cleanup rules / run periodic GC so the VPS disk doesn't fill.
  • Certs: never delete the caddy_data volume — losing it re-issues every cert and risks Let's Encrypt rate limits.
  • Least privilege: restrict the deploy key in authorized_keys with a forced command= that only runs the compose pull/up script.
  • Reset the admin password (no email/old password needed): as an admin via Site Administration → User Accounts → (user) → Password, or from the VPS:
    docker compose exec -u git forgejo \
      forgejo admin user change-password --username <user> \
      --password 'NEW' --must-change-password=false