jb-infra/README.md

# jb-infra

VPS edge stack for `joshbairstow.com`: a **caddy-docker-proxy** edge reverse
proxy (TLS + label-based routing) and a self-hosted **Forgejo** git server with
CI and a container registry. Everything else — `jb-website` and future side
projects — is deployed by CI and just attaches to the `edge` network and
declares routing labels.

```
[home desktop]                                  [Vultr VPS]
 Forgejo runner (host exec)                      docker network: edge (external)
  └ docker build  ───push image──►   ┌────────── edge-caddy   :80/:443  (TLS, routing)
  └ ssh deploy    ───pull+up─────►   ├────────── forgejo      git.joshbairstow.com (+registry)
                                     ├────────── jb-website   internal :80
                                     └────────── <future apps> e.g. shop.joshbairstow.com
```

## Why this shape

- **Edge proxy decouples projects.** Each project ships its own container with
  `caddy` labels; the edge picks them up automatically. Adding a project needs
  **zero edits here** — no central config to touch.
- **Build off-box.** The RAM-hungry Docker build runs on the always-on desktop
  runner; the VPS only pulls and runs prebuilt images, so it stays small.
- **Self-hosted git/CI** (Forgejo) — independent of GitHub/GitLab, with a cloud
  push-mirror kept only as backup.

---

## The VPS (provisioned 2026-06-02)

Vultr shared-CPU **2 GB**, **Debian 13 (trixie)**, kernel 6.12, ~5.4 GB swap
already present (no need to add any). Desktop SSH key installed; Vultr firewall
groups applied on both IPv4 and IPv6.

| | |
|---|---|
| IPv4 | `45.32.242.27` |
| IPv6 | `2001:19f0:5801:1313:5400:6ff:fe35:c0a7` |
| SSH  | `root@45.32.242.27` (port 22) |

## Pre-flight check

Before and after each bootstrap step, run the read-only checker from your
desktop to confirm reachability, ports, the box's facts, and DNS:

```sh
./scripts/check-vps.sh --ip6 2001:19f0:5801:1313:5400:6ff:fe35:c0a7
```

It exits 0 when SSH + port 22 are good; closed app ports and unset DNS show as
warnings (expected until DNS is updated and the stack is up).

## Bootstrap (one-time, in order)

Order matters — TLS issuance needs DNS + open ports first, and the `edge`
network must exist before any stack references it.

### 1. DNS — managed at **Cloudflare**, not VentraIP
VentraIP is only the registrar; the domain's nameservers are delegated to
Cloudflare (`grace/keenan.ns.cloudflare.com`), so create the records **in the
Cloudflare dashboard**. Point them at the VPS and **wait for propagation**:
- `A     joshbairstow.com     → 45.32.242.27`
- `A     *.joshbairstow.com   → 45.32.242.27`   (wildcard: covers blog/art/code/git + future)
- `AAAA  joshbairstow.com     → 2001:19f0:5801:1313:5400:6ff:fe35:c0a7`
- `AAAA  *.joshbairstow.com   → 2001:19f0:5801:1313:5400:6ff:fe35:c0a7`

> **Set these records to DNS-only (grey cloud), not proxied (orange cloud).**
> The current records point at Cloudflare's proxy. With the proxy ON: Cloudflare
> terminates TLS with its own edge cert (so Caddy's automatic HTTPS is bypassed),
> and it only proxies HTTP(S) ports — **git-over-SSH on 222 and the registry
> would break**. DNS-only makes clients hit the VPS directly, so Caddy issues
> real Let's Encrypt certs via HTTP-01 (no DNS-challenge plugin needed) and all
> ports work. If you later want Cloudflare's proxy/CDN, switch Caddy to the
> DNS-01 challenge with the Cloudflare plugin + an API token — a separate change.

### 2. VPS base
Firewall is already handled by the Vultr firewall group (80, 443, 222, 22), so
host `ufw` is optional. Swap already exists. The remaining task is Docker:
```sh
# Install Docker Engine + compose plugin
curl -fsSL https://get.docker.com | sh
docker --version && docker compose version   # confirm
```
(Re-run `./scripts/check-vps.sh` — `DOCKER` should now report a version.)

### 3. Get the files onto the VPS + bring up the stack
Forgejo doesn't exist yet, so this repo has no server to clone from during
bootstrap. The VPS can't pull from your (NAT'd) desktop either — but your
desktop can push to the VPS, and `docker compose` only needs the files, not
git. So copy them up with rsync (run from your desktop, in `~/git/jb-infra`):
```sh
rsync -av --exclude '.git' --exclude 'runner' --exclude 'scripts' \
  ./ root@45.32.242.27:/opt/jb-infra/
```
(`runner/` and `scripts/` are desktop-side tools — they don't belong on the
VPS.) Then on the VPS:
```sh
cd /opt/jb-infra
docker network create edge          # MUST exist before `up`
docker compose up -d
docker compose logs -f caddy        # watch the first cert issuance
```
Once Forgejo is live (step 4+), create a `jb-infra` repo inside it and push
this there for its permanent, versioned home; future infra edits then deploy
like any other project. If you don't want git history on the box, a one-shot
`git bundle` is an alternative to rsync (see project notes).

### 4. Configure Forgejo
Browse to `https://git.joshbairstow.com` (cert should be live). The install page
is mostly prepopulated from the env vars in `compose.yml` — confirm these, then
complete it:
- **Run As Username** stays `git` (the container's OS user, *not* your login).
- **Application URL** must be `https://git.joshbairstow.com/` (not
  `http://localhost:3000/`) — wrong value breaks `docker login`/push + clone URLs.
- **SSH Server Domain** `git.joshbairstow.com`, **SSH Server Port** `222`,
  **Database** `SQLite3`.
- Under server settings: enable **Disable Self-Registration** (personal instance)
  and **Hide Email Addresses by Default**; leave "Require Sign-In to View Pages"
  off (it would block anonymous clones/pulls).
- Expand **Administrator Account Settings** and create your admin user with a
  **lowercase** username (e.g. `josh`) — this becomes your registry path
  `git.joshbairstow.com/<owner>/…`, which must be lowercase.

Then:
- Confirm Actions is enabled (`Site Administration → Actions`).
- Create a **Personal Access Token** with **`write:package`** scope
  (`Settings → Applications`). The automatic Actions token CANNOT push packages.

### 5. Register the desktop runner (always-on)
On the **home desktop** (already has Docker + is in the `docker` group). Put the
binary in place, register in a home-dir working folder (snap Docker needs the
build context under `$HOME`), then install the systemd unit so it survives
reboots and auto-claims jobs:
Run these from the cloned `jb-infra` repo:
```sh
# 1. Binary — skip if already present (check: command -v forgejo-runner).
#    The unit's ExecStart points at /usr/local/sbin/forgejo-runner.
sudo install -m 0755 forgejo-runner /usr/local/sbin/forgejo-runner

# 2. Working dir + config (under $HOME so snap Docker can read build contexts).
mkdir -p ~/forgejo-runner && cp runner/config.yml ~/forgejo-runner/

# 3. Register IN the working dir so .runner lands there (token: Forgejo →
#    Site Admin → Actions → Runners → Create). The subshell keeps cwd in the repo.
( cd ~/forgejo-runner && forgejo-runner register --no-interactive \
    --instance https://git.joshbairstow.com \
    --token <REGISTRATION_TOKEN> \
    --name desktop --labels desktop:host )     # ':host' => run on host, use host Docker

# 4. Install + enable the always-on service.
sudo cp runner/forgejo-runner.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now forgejo-runner
systemctl status forgejo-runner                # expect active (running); runner shows green in Forgejo
```
Already registered in the wrong directory? Don't re-register — just
`mv .runner ~/forgejo-runner/` (it's location-independent credentials).
The `:host` label is what lets the `docker build` step work (the job runs on the
host using its Docker, not in a socket-less container). See
[`runner/config.yml`](runner/config.yml) and
[`runner/forgejo-runner.service`](runner/forgejo-runner.service).

The desktop's **Docker daemon must be running and enabled at boot** — the runner
builds against it. `docker login` works without the daemon but `docker build`
won't, so a stopped daemon shows up as *"Cannot connect to the Docker daemon"*
mid-pipeline. Ensure one daemon owns `/var/run/docker.sock`:
```sh
sudo snap stop --disable docker.dockerd      # if a crashing snap docker is present
sudo systemctl enable --now docker.service   # classic engine, on at boot
docker info | head                           # verify as your user, no sudo
```
Fedora gotcha: if `dockerd` fails with
`ZONE_CONFLICT: 'docker0' already bound to 'FedoraWorkstation'`, the cause is a
leftover **NetworkManager** bridge profile for `docker0` (no zone set → lands in
the default zone), which firewalld then re-applies. Delete it, stop NM managing
Docker's bridges (so it can't recur at boot), and start Docker:
```sh
sudo nmcli connection delete docker0
sudo tee /etc/NetworkManager/conf.d/docker-unmanaged.conf >/dev/null <<'EOF'
[keyfile]
unmanaged-devices=interface-name:docker0;interface-name:br-*
EOF
sudo systemctl reload NetworkManager
sudo ip link delete docker0 2>/dev/null
sudo systemctl reset-failed docker.service && sudo systemctl start docker.service
```

### 6. Wire the jb-website repo
In the `jb-website` repo settings on Forgejo (`Settings → Actions`):

**Variable:**
- `REGISTRY_OWNER` — your **lowercase** Forgejo username/org; the image namespace
  (`git.joshbairstow.com/<owner>/jb-website`).

**Secrets:**
- `REGISTRY_USER` — your Forgejo login username (the account that owns the PAT);
  the username for `docker login`. Usually equals `REGISTRY_OWNER`.
- `REGISTRY_TOKEN` — the PAT with `write:package` scope; the `docker login` password.
- `DEPLOY_HOST` — `45.32.242.27`.
- `DEPLOY_USER` — the VPS account the runner SSHes in as: `root` to start, or a
  dedicated `deploy` user (in the `docker` group) for least privilege.
- `DEPLOY_SSH_KEY` — the **private** key of a *dedicated deploy keypair* (full
  file incl. `BEGIN/END` lines; paste multi-line — OpenSSH needs the newlines).
  NOT your personal key. Generate + install its public half on the VPS:
  ```sh
  ssh-keygen -t ed25519 -f ~/.ssh/jb-deploy -C forgejo-ci-deploy -N ""
  ssh-copy-id -i ~/.ssh/jb-deploy.pub <DEPLOY_USER>@45.32.242.27
  cat ~/.ssh/jb-deploy        # <- paste this into DEPLOY_SSH_KEY
  ```

On the **VPS**, put the production compose file in place — rsync it from your
desktop (creates the dir + copies; only the prod `compose.yml`, not the dev one):
```sh
rsync -av --rsync-path="mkdir -p /opt/jb-website && rsync" \
  ~/git/jb-website/compose.yml root@45.32.242.27:/opt/jb-website/
```
For `docker compose pull` to read the image, the **package** must be pullable.
Note: **package visibility ≠ repository visibility** — packages are owned by the
user/org, so making the *repo* public does NOT necessarily make the *image*
anonymously pullable, and the package only exists after the first push. After
the first pipeline run, verify from the VPS:
```sh
docker pull git.joshbairstow.com/<owner>/jb-website:latest
# unauthorized? -> set the PACKAGE public (profile → Packages → jb-website →
# settings), or run `docker login git.joshbairstow.com` once as DEPLOY_USER.
```
If `DEPLOY_USER` is not root, it must be in the `docker` group; for least
privilege, restrict its `authorized_keys` entry with a forced `command=`.

### 7. First deploy
```sh
# in the jb-website repo (default branch is already `main`).
# Custom SSH port 222 needs the ssh:// URL form (scp-style host:port doesn't work):
git remote add origin ssh://git@git.joshbairstow.com:222/<owner>/jb-website.git
git push -u origin main
```
The push triggers `.forgejo/workflows/deploy.yml`: the desktop builds + pushes
the image, then SSHes to the VPS to `pull && up -d`. The edge issues certs for
the apex + subdomains on first request.

---

## Adding another project later

No changes to this repo. In the new project:
1. Ship a `compose.yml` on the VPS that joins the external `edge` network and
   carries labels, e.g.:
   ```yaml
   services:
     app:
       image: git.joshbairstow.com/<owner>/<app>:latest
       networks: [edge]
       labels:
         caddy: shop.joshbairstow.com
         caddy.reverse_proxy: "{{upstreams 3000}}"
   networks:
     edge:
       external: true
   ```
2. Give it its own `.forgejo/workflows/deploy.yml` (copy jb-website's).
3. `*.joshbairstow.com` already resolves, so the edge issues its cert
   automatically on first hit.

---

## Maintenance / hardening

- **Backups:** the `forgejo_data` volume is now the source of truth (repos, DB,
  registry blobs). Run `docker compose exec forgejo forgejo dump` on a schedule,
  and set a **push-mirror** on each repo to a cloud host (GitLab/GitHub) for
  off-box git backup.
- **Registry GC:** per-commit `:sha` tags accumulate. Enable Forgejo's package
  cleanup rules / run periodic GC so the VPS disk doesn't fill.
- **Certs:** never delete the `caddy_data` volume — losing it re-issues every
  cert and risks Let's Encrypt rate limits.
- **Least privilege:** restrict the deploy key in `authorized_keys` with a
  forced `command=` that only runs the compose pull/up script.
- **Reset the admin password** (no email/old password needed): as an admin via
  *Site Administration → User Accounts → (user) → Password*, or from the VPS:
  ```sh
  docker compose exec -u git forgejo \
    forgejo admin user change-password --username <user> \
    --password 'NEW' --must-change-password=false
  ```
```