jb-infra/README.md

# jb-infra

VPS edge stack for `joshbairstow.com`: a **caddy-docker-proxy** edge reverse
proxy (TLS + label-based routing) and a self-hosted **Forgejo** git server with
CI and a container registry. Everything else — `jb-website` and future side
projects — is deployed by CI and just attaches to the `edge` network and
declares routing labels.

```
[home desktop]                                  [Vultr VPS]
 Forgejo runner (host exec)                      docker network: edge (external)
  └ docker build  ───push image──►   ┌────────── edge-caddy   :80/:443  (TLS, routing)
  └ ssh deploy    ───pull+up─────►   ├────────── forgejo      git.joshbairstow.com (+registry)
                                     ├────────── jb-website   internal :80
                                     └────────── <future apps> e.g. shop.joshbairstow.com
```

## Why this shape

- **Edge proxy decouples projects.** Each project ships its own container with
  `caddy` labels; the edge picks them up automatically. Adding a project needs
  **zero edits here** — no central config to touch.
- **Build off-box.** The RAM-hungry Docker build runs on the always-on desktop
  runner; the VPS only pulls and runs prebuilt images, so it stays small.
- **Self-hosted git/CI** (Forgejo) — independent of GitHub/GitLab, with a cloud
  push-mirror kept only as backup.

---

## The VPS (provisioned 2026-06-02)

Vultr shared-CPU **2 GB**, **Debian 13 (trixie)**, kernel 6.12, ~5.4 GB swap
already present (no need to add any). Desktop SSH key installed; Vultr firewall
groups applied on both IPv4 and IPv6.

| | |
|---|---|
| IPv4 | `45.32.242.27` |
| IPv6 | `2001:19f0:5801:1313:5400:6ff:fe35:c0a7` |
| SSH  | `root@45.32.242.27` (port 22) |

## Pre-flight check

Before and after each bootstrap step, run the read-only checker from your
desktop to confirm reachability, ports, the box's facts, and DNS:

```sh
./scripts/check-vps.sh --ip6 2001:19f0:5801:1313:5400:6ff:fe35:c0a7
```

It exits 0 when SSH + port 22 are good; closed app ports and unset DNS show as
warnings (expected until DNS is updated and the stack is up).

## Bootstrap (one-time, in order)

Order matters — TLS issuance needs DNS + open ports first, and the `edge`
network must exist before any stack references it.

### 1. DNS — managed at **Cloudflare**, not VentraIP
VentraIP is only the registrar; the domain's nameservers are delegated to
Cloudflare (`grace/keenan.ns.cloudflare.com`), so create the records **in the
Cloudflare dashboard**. Point them at the VPS and **wait for propagation**:
- `A     joshbairstow.com     → 45.32.242.27`
- `A     *.joshbairstow.com   → 45.32.242.27`   (wildcard: covers blog/art/code/git + future)
- `AAAA  joshbairstow.com     → 2001:19f0:5801:1313:5400:6ff:fe35:c0a7`
- `AAAA  *.joshbairstow.com   → 2001:19f0:5801:1313:5400:6ff:fe35:c0a7`

> **Set these records to DNS-only (grey cloud), not proxied (orange cloud).**
> The current records point at Cloudflare's proxy. With the proxy ON: Cloudflare
> terminates TLS with its own edge cert (so Caddy's automatic HTTPS is bypassed),
> and it only proxies HTTP(S) ports — **git-over-SSH on 222 and the registry
> would break**. DNS-only makes clients hit the VPS directly, so Caddy issues
> real Let's Encrypt certs via HTTP-01 (no DNS-challenge plugin needed) and all
> ports work. If you later want Cloudflare's proxy/CDN, switch Caddy to the
> DNS-01 challenge with the Cloudflare plugin + an API token — a separate change.

### 2. VPS base
Firewall is already handled by the Vultr firewall group (80, 443, 222, 22), so
host `ufw` is optional. Swap already exists. The remaining task is Docker:
```sh
# Install Docker Engine + compose plugin
curl -fsSL https://get.docker.com | sh
docker --version && docker compose version   # confirm
```
(Re-run `./scripts/check-vps.sh` — `DOCKER` should now report a version.)

### 3. Get the files onto the VPS + bring up the stack
Forgejo doesn't exist yet, so this repo has no server to clone from during
bootstrap. The VPS can't pull from your (NAT'd) desktop either — but your
desktop can push to the VPS, and `docker compose` only needs the files, not
git. So copy them up with rsync (run from your desktop, in `~/git/jb-infra`):
```sh
rsync -av --exclude '.git' --exclude 'runner' --exclude 'scripts' \
  ./ root@45.32.242.27:/opt/jb-infra/
```
(`runner/` and `scripts/` are desktop-side tools — they don't belong on the
VPS.) Then on the VPS:
```sh
cd /opt/jb-infra
docker network create edge          # MUST exist before `up`
docker compose up -d
docker compose logs -f caddy        # watch the first cert issuance
```
Once Forgejo is live (step 4+), create a `jb-infra` repo inside it and push
this there for its permanent, versioned home; future infra edits then deploy
like any other project. If you don't want git history on the box, a one-shot
`git bundle` is an alternative to rsync (see project notes).

### 4. Configure Forgejo
Browse to `https://git.joshbairstow.com` (cert should be live). The install page
is mostly prepopulated from the env vars in `compose.yml` — confirm these, then
complete it:
- **Run As Username** stays `git` (the container's OS user, *not* your login).
- **Application URL** must be `https://git.joshbairstow.com/` (not
  `http://localhost:3000/`) — wrong value breaks `docker login`/push + clone URLs.
- **SSH Server Domain** `git.joshbairstow.com`, **SSH Server Port** `222`,
  **Database** `SQLite3`.
- Under server settings: enable **Disable Self-Registration** (personal instance)
  and **Hide Email Addresses by Default**; leave "Require Sign-In to View Pages"
  off (it would block anonymous clones/pulls).
- Expand **Administrator Account Settings** and create your admin user with a
  **lowercase** username (e.g. `josh`) — this becomes your registry path
  `git.joshbairstow.com/<owner>/…`, which must be lowercase.

Then:
- Confirm Actions is enabled (`Site Administration → Actions`).
- Create a **Personal Access Token** with **`write:package`** scope
  (`Settings → Applications`). The automatic Actions token CANNOT push packages.

### 5. Register the desktop runner (always-on)
On the **home desktop** (already has Docker + is in the `docker` group). Put the
binary in place, register in a home-dir working folder (snap Docker needs the
build context under `$HOME`), then install the systemd unit so it survives
reboots and auto-claims jobs:
```sh
sudo install -m 0755 forgejo-runner /usr/local/bin/forgejo-runner
mkdir -p ~/forgejo-runner && cp runner/config.yml ~/forgejo-runner/
cd ~/forgejo-runner

# Registration token: Forgejo → Site Admin → Actions → Runners → Create
forgejo-runner register --no-interactive \
  --instance https://git.joshbairstow.com \
  --token <REGISTRATION_TOKEN> \
  --name desktop --labels desktop:host        # ':host' => run jobs on the host, use host Docker

sudo cp runner/forgejo-runner.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now forgejo-runner
systemctl status forgejo-runner               # expect active (running); runner shows green in Forgejo
```
The `:host` label is what lets the `docker build` step work (the job runs on the
host using its Docker, not in a socket-less container). See
[`runner/config.yml`](runner/config.yml) and
[`runner/forgejo-runner.service`](runner/forgejo-runner.service).

### 6. Wire the jb-website repo
In the `jb-website` repo settings on Forgejo (`Settings → Actions`):
- **Variables:** `REGISTRY_OWNER` = your lowercase owner.
- **Secrets:** `REGISTRY_USER`, `REGISTRY_TOKEN` (the PAT), `DEPLOY_HOST`
  (`45.32.242.27`), `DEPLOY_USER`, `DEPLOY_SSH_KEY`.

On the **VPS**, prepare the deploy target:
```sh
mkdir -p /opt/jb-website
# copy jb-website/compose.yml here (the production one — pulls, no build)
docker login git.joshbairstow.com      # one-time, so `compose pull` can read the package
# (or mark the jb-website package public in Forgejo to skip VPS login)
```
Create a restricted deploy user whose `authorized_keys` holds the public half
of `DEPLOY_SSH_KEY`.

### 7. First deploy
```sh
# in the jb-website repo (default branch is already `main`).
# Custom SSH port 222 needs the ssh:// URL form (scp-style host:port doesn't work):
git remote add origin ssh://git@git.joshbairstow.com:222/<owner>/jb-website.git
git push -u origin main
```
The push triggers `.forgejo/workflows/deploy.yml`: the desktop builds + pushes
the image, then SSHes to the VPS to `pull && up -d`. The edge issues certs for
the apex + subdomains on first request.

---

## Adding another project later

No changes to this repo. In the new project:
1. Ship a `compose.yml` on the VPS that joins the external `edge` network and
   carries labels, e.g.:
   ```yaml
   services:
     app:
       image: git.joshbairstow.com/<owner>/<app>:latest
       networks: [edge]
       labels:
         caddy: shop.joshbairstow.com
         caddy.reverse_proxy: "{{upstreams 3000}}"
   networks:
     edge:
       external: true
   ```
2. Give it its own `.forgejo/workflows/deploy.yml` (copy jb-website's).
3. `*.joshbairstow.com` already resolves, so the edge issues its cert
   automatically on first hit.

---

## Maintenance / hardening

- **Backups:** the `forgejo_data` volume is now the source of truth (repos, DB,
  registry blobs). Run `docker compose exec forgejo forgejo dump` on a schedule,
  and set a **push-mirror** on each repo to a cloud host (GitLab/GitHub) for
  off-box git backup.
- **Registry GC:** per-commit `:sha` tags accumulate. Enable Forgejo's package
  cleanup rules / run periodic GC so the VPS disk doesn't fill.
- **Certs:** never delete the `caddy_data` volume — losing it re-issues every
  cert and risks Let's Encrypt rate limits.
- **Least privilege:** restrict the deploy key in `authorized_keys` with a
  forced `command=` that only runs the compose pull/up script.
- **Reset the admin password** (no email/old password needed): as an admin via
  *Site Administration → User Accounts → (user) → Password*, or from the VPS:
  ```sh
  docker compose exec -u git forgejo \
    forgejo admin user change-password --username <user> \
    --password 'NEW' --must-change-password=false
  ```
```