From 75fabc864e44e647d292a8497242a1286670b717 Mon Sep 17 00:00:00 2001 From: Josh Bairstow Date: Tue, 2 Jun 2026 16:13:45 +1000 Subject: [PATCH 01/10] docs: record live VPS details + DNS-at-Cloudflare caveat; add pre-flight script MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - scripts/check-vps.sh: read-only reachability/ports/SSH-facts/DNS checker - README: real IPv4/IPv6 + Debian 13 + existing swap; firewall already on Vultr - DNS is managed at Cloudflare (NS delegated), not VentraIP — records must be DNS-only (grey cloud) so Caddy issues real certs and port 222 works - fix git remote to ssh:// form for the custom 222 port Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 73 ++++++++++++------ scripts/check-vps.sh | 174 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 224 insertions(+), 23 deletions(-) create mode 100755 scripts/check-vps.sh diff --git a/README.md b/README.md index 1eb7c1e..96f24ec 100644 --- a/README.md +++ b/README.md @@ -27,36 +27,62 @@ declares routing labels. --- +## The VPS (provisioned 2026-06-02) + +Vultr shared-CPU **2 GB**, **Debian 13 (trixie)**, kernel 6.12, ~5.4 GB swap +already present (no need to add any). Desktop SSH key installed; Vultr firewall +groups applied on both IPv4 and IPv6. + +| | | +|---|---| +| IPv4 | `45.32.242.27` | +| IPv6 | `2001:19f0:5801:1313:5400:6ff:fe35:c0a7` | +| SSH | `root@45.32.242.27` (port 22) | + +## Pre-flight check + +Before and after each bootstrap step, run the read-only checker from your +desktop to confirm reachability, ports, the box's facts, and DNS: + +```sh +./scripts/check-vps.sh --ip6 2001:19f0:5801:1313:5400:6ff:fe35:c0a7 +``` + +It exits 0 when SSH + port 22 are good; closed app ports and unset DNS show as +warnings (expected until DNS is updated and the stack is up). + ## Bootstrap (one-time, in order) Order matters — TLS issuance needs DNS + open ports first, and the `edge` network must exist before any stack references it. -### 1. DNS -Point at the VPS public IP, and **wait for propagation** before step 4: -- `A joshbairstow.com → ` -- `A *.joshbairstow.com → ` (wildcard covers blog/art/code/git + future) -- `AAAA` equivalents if the VPS has IPv6. +### 1. DNS — managed at **Cloudflare**, not VentraIP +VentraIP is only the registrar; the domain's nameservers are delegated to +Cloudflare (`grace/keenan.ns.cloudflare.com`), so create the records **in the +Cloudflare dashboard**. Point them at the VPS and **wait for propagation**: +- `A joshbairstow.com → 45.32.242.27` +- `A *.joshbairstow.com → 45.32.242.27` (wildcard: covers blog/art/code/git + future) +- `AAAA joshbairstow.com → 2001:19f0:5801:1313:5400:6ff:fe35:c0a7` +- `AAAA *.joshbairstow.com → 2001:19f0:5801:1313:5400:6ff:fe35:c0a7` -A wildcard record means new subdomains resolve with no DNS change; certs are -still issued per-host via HTTP-01, so no DNS-challenge plugin is required. +> **Set these records to DNS-only (grey cloud), not proxied (orange cloud).** +> The current records point at Cloudflare's proxy. With the proxy ON: Cloudflare +> terminates TLS with its own edge cert (so Caddy's automatic HTTPS is bypassed), +> and it only proxies HTTP(S) ports — **git-over-SSH on 222 and the registry +> would break**. DNS-only makes clients hit the VPS directly, so Caddy issues +> real Let's Encrypt certs via HTTP-01 (no DNS-challenge plugin needed) and all +> ports work. If you later want Cloudflare's proxy/CDN, switch Caddy to the +> DNS-01 challenge with the Cloudflare plugin + an API token — a separate change. ### 2. VPS base +Firewall is already handled by the Vultr firewall group (80, 443, 222, 22), so +host `ufw` is optional. Swap already exists. The remaining task is Docker: ```sh -# Install Docker Engine + compose plugin (official convenience script or distro pkg) +# Install Docker Engine + compose plugin curl -fsSL https://get.docker.com | sh - -# Firewall: allow only what we serve. -# 80, 443 -> edge proxy (HTTP/HTTPS, incl. ACME challenges) -# 222 -> Forgejo git-over-SSH -# 22 -> your normal host SSH (keep it!) -ufw allow 80,443/tcp && ufw allow 222/tcp && ufw allow OpenSSH && ufw enable -``` -RAM: prefer the **2 GB** Vultr tier. On 1 GB, add swap first: -```sh -fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile -echo '/swapfile none swap sw 0 0' >> /etc/fstab +docker --version && docker compose version # confirm ``` +(Re-run `./scripts/check-vps.sh` — `DOCKER` should now report a version.) ### 3. Create the shared network + bring up the stack ```sh @@ -90,8 +116,8 @@ forgejo-runner daemon --config /path/to/jb-infra/runner/config.yml ### 6. Wire the jb-website repo In the `jb-website` repo settings on Forgejo (`Settings → Actions`): - **Variables:** `REGISTRY_OWNER` = your lowercase owner. -- **Secrets:** `REGISTRY_USER`, `REGISTRY_TOKEN` (the PAT), `DEPLOY_HOST`, - `DEPLOY_USER`, `DEPLOY_SSH_KEY`. +- **Secrets:** `REGISTRY_USER`, `REGISTRY_TOKEN` (the PAT), `DEPLOY_HOST` + (`45.32.242.27`), `DEPLOY_USER`, `DEPLOY_SSH_KEY`. On the **VPS**, prepare the deploy target: ```sh @@ -105,8 +131,9 @@ of `DEPLOY_SSH_KEY`. ### 7. First deploy ```sh -# in the jb-website repo (rename master→main first; see that repo's README) -git remote add origin git@git.joshbairstow.com:222//jb-website.git +# in the jb-website repo (default branch is already `main`). +# Custom SSH port 222 needs the ssh:// URL form (scp-style host:port doesn't work): +git remote add origin ssh://git@git.joshbairstow.com:222//jb-website.git git push -u origin main ``` The push triggers `.forgejo/workflows/deploy.yml`: the desktop builds + pushes diff --git a/scripts/check-vps.sh b/scripts/check-vps.sh new file mode 100755 index 0000000..f8e6e01 --- /dev/null +++ b/scripts/check-vps.sh @@ -0,0 +1,174 @@ +#!/usr/bin/env bash +# check-vps.sh — verify the joshbairstow.com VPS is reachable and correctly +# configured, before and after the jb-infra bootstrap. READ-ONLY: it changes +# nothing on the box or in DNS. +# +# What it checks: +# - ICMP reachability (IPv4, and IPv6 if you pass --ip6) +# - TCP ports 22/80/443/222 (firewall + whether a service is listening yet) +# - SSH login with your key, then reports the box's OS / kernel / RAM / swap / +# Docker / compose / the `edge` network / its own public IPs +# - DNS for the apex, the wildcard, and each subdomain vs the expected IP(s) +# +# Usage: +# ./check-vps.sh +# ./check-vps.sh --ip6 2001:19f0:... --key ~/.ssh/id_ed25519 +# VPS_IP=1.2.3.4 DOMAIN=example.com ./check-vps.sh +# +# Exit status: 0 if the CRITICAL checks pass (SSH login + port 22), else 1. +# Warnings (!) — closed app ports, unset DNS — are EXPECTED before you update +# DNS and bring up the edge/Forgejo stack, so they don't fail the run. + +set -u + +# ---- defaults (override via env or flags) ---------------------------------- +VPS_IP="${VPS_IP:-45.32.242.27}" +VPS_IP6="${VPS_IP6:-}" +DOMAIN="${DOMAIN:-joshbairstow.com}" +SSH_USER="${SSH_USER:-root}" +SSH_PORT="${SSH_PORT:-22}" +SSH_KEY="${SSH_KEY:-}" +SUBDOMAINS=(www blog art code git) +TCP_PORTS=(22 80 443 222) + +while [ $# -gt 0 ]; do + case "$1" in + --ip) VPS_IP="$2"; shift 2;; + --ip6) VPS_IP6="$2"; shift 2;; + --domain) DOMAIN="$2"; shift 2;; + --user) SSH_USER="$2"; shift 2;; + --ssh-port) SSH_PORT="$2"; shift 2;; + --key) SSH_KEY="$2"; shift 2;; + -h|--help) sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'; exit 0;; + *) echo "unknown arg: $1" >&2; exit 2;; + esac +done + +# ---- output helpers -------------------------------------------------------- +if [ -t 1 ]; then R=$'\e[31m'; G=$'\e[32m'; Y=$'\e[33m'; B=$'\e[1;34m'; D=$'\e[2m'; Z=$'\e[0m' +else R=; G=; Y=; B=; D=; Z=; fi +pass=0; warn=0; fail=0 +ok() { printf ' %s✓%s %s\n' "$G" "$Z" "$*"; pass=$((pass+1)); } +nok() { printf ' %s✗%s %s\n' "$R" "$Z" "$*"; fail=$((fail+1)); } +note() { printf ' %s!%s %s\n' "$Y" "$Z" "$*"; warn=$((warn+1)); } +info() { printf ' %s·%s %s\n' "$D" "$Z" "$*"; } +hdr() { printf '\n%s== %s ==%s\n' "$B" "$*" "$Z"; } +have() { command -v "$1" >/dev/null 2>&1; } + +printf '%sVPS pre-flight%s ip=%s ip6=%s domain=%s ssh=%s@%s:%s\n' \ + "$B" "$Z" "$VPS_IP" "${VPS_IP6:-none}" "$DOMAIN" "$SSH_USER" "$VPS_IP" "$SSH_PORT" + +# ---- DNS resolvers (dig → host → getent) ----------------------------------- +resolve_a() { + if have dig; then dig +short A "$1" | grep -E '^[0-9.]+$' + elif have host; then host -t A "$1" 2>/dev/null | sed -n 's/.*has address //p' + else getent ahostsv4 "$1" 2>/dev/null | awk '{print $1}' | sort -u; fi +} +resolve_aaaa() { + if have dig; then dig +short AAAA "$1" | grep -E ':' + elif have host; then host -t AAAA "$1" 2>/dev/null | sed -n 's/.*IPv6 address //p' + else getent ahostsv6 "$1" 2>/dev/null | awk '{print $1}' | sort -u; fi +} + +# ---- TCP port probe (bash /dev/tcp — variant-independent) ------------------- +check_port() { # host port -> "open" | "closed" + if timeout 4 bash -c ">/dev/tcp/$1/$2" 2>/dev/null; then echo open; else echo closed; fi +} + +# ---- 1. ICMP --------------------------------------------------------------- +hdr "ICMP reachability" +if ping -c1 -W3 "$VPS_IP" >/dev/null 2>&1; then + ok "ping $VPS_IP (IPv4)" +else + note "ping $VPS_IP (IPv4) no reply — fine if ICMP isn't allowed in the firewall" +fi +if [ -n "$VPS_IP6" ]; then + if ping -6 -c1 -W3 "$VPS_IP6" >/dev/null 2>&1; then ok "ping $VPS_IP6 (IPv6)" + else note "ping $VPS_IP6 (IPv6) no reply — your local link may lack IPv6"; fi +else + info "no --ip6 given; skipping IPv6 ping (SSH will report the box's v6 address)" +fi + +# ---- 2. TCP ports ---------------------------------------------------------- +hdr "TCP ports (firewall + listeners)" +for p in "${TCP_PORTS[@]}"; do + st=$(check_port "$VPS_IP" "$p") + case "$p:$st" in + 22:open) ok "22 SSH — open" ;; + 22:*) nok "22 SSH — $st (you can't manage the box; check firewall 22 + sshd)" ;; + *:open) ok "$p open (service listening)" ;; + *:*) note "$p $st — expected until the edge/Forgejo stack is up" ;; + esac +done + +# ---- 3. SSH login + remote facts ------------------------------------------- +hdr "SSH login + remote facts" +ssh_opts=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new -p "$SSH_PORT") +[ -n "$SSH_KEY" ] && ssh_opts+=(-i "$SSH_KEY") +err=$(mktemp) +remote=$(ssh "${ssh_opts[@]}" "$SSH_USER@$VPS_IP" ' + . /etc/os-release 2>/dev/null + echo "OS=${PRETTY_NAME:-unknown}" + echo "KERNEL=$(uname -r)" + echo "ARCH=$(uname -m)" + mem_kb=$(grep MemTotal /proc/meminfo | tr -dc 0-9); echo "MEM_MB=$((mem_kb/1024))" + swap_kb=$(grep SwapTotal /proc/meminfo | tr -dc 0-9); echo "SWAP_MB=$((swap_kb/1024))" + echo "PUBIP4=$(ip -4 -o addr show scope global 2>/dev/null | sed -n "s@.*inet \([0-9.]*\)/.*@\1@p" | head -1)" + echo "PUBIP6=$(ip -6 -o addr show scope global 2>/dev/null | sed -n "s@.*inet6 \([0-9a-f:]*\)/.*@\1@p" | head -1)" + if command -v docker >/dev/null 2>&1; then echo "DOCKER=$(docker --version 2>/dev/null)"; else echo "DOCKER=none"; fi + echo "COMPOSE=$(docker compose version --short 2>/dev/null || echo none)" + if docker network ls --format "{{.Name}}" 2>/dev/null | grep -qx edge; then echo "EDGE_NET=present"; else echo "EDGE_NET=absent"; fi +' 2>"$err") +rc=$? +if [ $rc -eq 0 ]; then + ok "SSH login as $SSH_USER@$VPS_IP:$SSH_PORT" + # surface every fact, then sanity-check the important ones + while IFS='=' read -r k v; do [ -n "$k" ] && info "$k = $v"; done <<<"$remote" + get() { printf '%s\n' "$remote" | sed -n "s/^$1=//p"; } + case "$(get OS)" in + *[Dd]ebian*13*|*trixie*) ok "OS is Debian 13 (trixie)";; + *[Dd]ebian*) note "Debian, but not 13: $(get OS)";; + *) note "unexpected OS: $(get OS)";; + esac + [ "$(get MEM_MB)" -ge 1900 ] 2>/dev/null && ok "RAM ~2 GB ($(get MEM_MB) MB)" \ + || note "RAM is $(get MEM_MB) MB — expected ~2 GB" + [ "$(get PUBIP4)" = "$VPS_IP" ] && ok "box's public IPv4 matches ($VPS_IP)" \ + || note "box reports IPv4 $(get PUBIP4), expected $VPS_IP" + if [ -n "$(get PUBIP6)" ]; then + if [ -n "$VPS_IP6" ] && [ "$(get PUBIP6)" = "$VPS_IP6" ]; then ok "box's public IPv6 matches" + else info "box's public IPv6 is $(get PUBIP6) — set this as your AAAA records"; fi + fi + [ "$(get DOCKER)" = none ] && note "Docker not installed yet (runbook step 2)" || ok "Docker present: $(get DOCKER)" + [ "$(get EDGE_NET)" = present ] && ok "'edge' network exists" || info "'edge' network not created yet (runbook step 3)" +else + nok "SSH login failed (rc=$rc): $(tr -d '\n' <"$err")" + info "checklist: key uploaded to Vultr? user is '$SSH_USER'? firewall allows 22 from your IP? try --key " +fi +rm -f "$err" + +# ---- 4. DNS ---------------------------------------------------------------- +hdr "DNS records (expecting → $VPS_IP${VPS_IP6:+ / $VPS_IP6})" +check_name() { + local fqdn=$1 a aaaa + a=$(resolve_a "$fqdn" | paste -sd, -) + aaaa=$(resolve_aaaa "$fqdn" | paste -sd, -) + if [ -z "$a$aaaa" ]; then note "$fqdn — no record yet"; return; fi + if printf '%s' "$a" | grep -qw "$VPS_IP"; then ok "$fqdn A → $a" + else note "$fqdn A → ${a:-none} (expected $VPS_IP)"; fi + if [ -n "$VPS_IP6" ]; then + if printf '%s' "$aaaa" | grep -qiw "$VPS_IP6"; then ok "$fqdn AAAA → $aaaa" + else note "$fqdn AAAA → ${aaaa:-none} (expected $VPS_IP6)"; fi + elif [ -n "$aaaa" ]; then info "$fqdn AAAA → $aaaa"; fi +} +check_name "$DOMAIN" +check_name "wildcard-probe.$DOMAIN" # resolves only if *.DOMAIN is set +for s in "${SUBDOMAINS[@]}"; do check_name "$s.$DOMAIN"; done + +# ---- summary --------------------------------------------------------------- +hdr "Summary" +printf ' %s%d passed%s · %s%d warnings%s · %s%d failed%s\n' \ + "$G" "$pass" "$Z" "$Y" "$warn" "$Z" "$R" "$fail" "$Z" +if [ "$fail" -gt 0 ]; then + printf ' %sCritical checks failed%s — see ✗ above.\n' "$R" "$Z"; exit 1 +fi +printf ' No critical failures. Warnings (!) are expected before DNS update / stack bootstrap.\n' From 7a3a8fb4002d0cb53aac1adaa369f9c9af7262b2 Mon Sep 17 00:00:00 2001 From: Josh Bairstow Date: Tue, 2 Jun 2026 17:55:44 +1000 Subject: [PATCH 02/10] docs: replace clone placeholder with rsync bootstrap + flesh out Forgejo first-run - step 3: rsync files from desktop (Forgejo can't host its own bootstrap repo; VPS can't pull from a NAT'd desktop); note permanent Forgejo home + git bundle - step 4: confirm Application URL / SSH domain+port / DB / RunAs=git, disable self-registration, create lowercase admin under Administrator Account Settings Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 37 ++++++++++++++++++++++++++++++++----- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 96f24ec..a0584d2 100644 --- a/README.md +++ b/README.md @@ -84,18 +84,45 @@ docker --version && docker compose version # confirm ``` (Re-run `./scripts/check-vps.sh` — `DOCKER` should now report a version.) -### 3. Create the shared network + bring up the stack +### 3. Get the files onto the VPS + bring up the stack +Forgejo doesn't exist yet, so this repo has no server to clone from during +bootstrap. The VPS can't pull from your (NAT'd) desktop either — but your +desktop can push to the VPS, and `docker compose` only needs the files, not +git. So copy them up with rsync (run from your desktop, in `~/git/jb-infra`): ```sh +rsync -av --exclude '.git' --exclude 'runner' --exclude 'scripts' \ + ./ root@45.32.242.27:/opt/jb-infra/ +``` +(`runner/` and `scripts/` are desktop-side tools — they don't belong on the +VPS.) Then on the VPS: +```sh +cd /opt/jb-infra docker network create edge # MUST exist before `up` -git clone /opt/jb-infra && cd /opt/jb-infra docker compose up -d docker compose logs -f caddy # watch the first cert issuance ``` +Once Forgejo is live (step 4+), create a `jb-infra` repo inside it and push +this there for its permanent, versioned home; future infra edits then deploy +like any other project. If you don't want git history on the box, a one-shot +`git bundle` is an alternative to rsync (see project notes). ### 4. Configure Forgejo -Browse to `https://git.joshbairstow.com` (cert should be live): -- Complete first-run setup; create your admin user with a **lowercase** - username (the registry path is case-sensitive and must be lowercase). +Browse to `https://git.joshbairstow.com` (cert should be live). The install page +is mostly prepopulated from the env vars in `compose.yml` — confirm these, then +complete it: +- **Run As Username** stays `git` (the container's OS user, *not* your login). +- **Application URL** must be `https://git.joshbairstow.com/` (not + `http://localhost:3000/`) — wrong value breaks `docker login`/push + clone URLs. +- **SSH Server Domain** `git.joshbairstow.com`, **SSH Server Port** `222`, + **Database** `SQLite3`. +- Under server settings: enable **Disable Self-Registration** (personal instance) + and **Hide Email Addresses by Default**; leave "Require Sign-In to View Pages" + off (it would block anonymous clones/pulls). +- Expand **Administrator Account Settings** and create your admin user with a + **lowercase** username (e.g. `josh`) — this becomes your registry path + `git.joshbairstow.com//…`, which must be lowercase. + +Then: - Confirm Actions is enabled (`Site Administration → Actions`). - Create a **Personal Access Token** with **`write:package`** scope (`Settings → Applications`). The automatic Actions token CANNOT push packages. From 74b1c3fcbbdf2d5e2ccc80dd0d2aaa6e3ed4dbb8 Mon Sep 17 00:00:00 2001 From: Josh Bairstow Date: Tue, 2 Jun 2026 22:12:11 +1000 Subject: [PATCH 03/10] docs: add admin password-reset recipe; gitignore local secret notes - README maintenance: reset admin password via UI or 'forgejo admin user change-password' (no email/old password needed) - .gitignore: secrets.local.md + *.local.secret so off-git credential backups can never be committed Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 4 ++++ README.md | 7 +++++++ 2 files changed, 11 insertions(+) diff --git a/.gitignore b/.gitignore index 7cdd3c1..1688b7a 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,7 @@ # Forgejo runner registration file — contains a secret, never commit. runner/.runner *.env + +# Local secret notes — backup of credentials kept off git. NEVER commit. +secrets.local.md +*.local.secret diff --git a/README.md b/README.md index a0584d2..20c7058 100644 --- a/README.md +++ b/README.md @@ -204,4 +204,11 @@ No changes to this repo. In the new project: cert and risks Let's Encrypt rate limits. - **Least privilege:** restrict the deploy key in `authorized_keys` with a forced `command=` that only runs the compose pull/up script. +- **Reset the admin password** (no email/old password needed): as an admin via + *Site Administration → User Accounts → (user) → Password*, or from the VPS: + ```sh + docker compose exec -u git forgejo \ + forgejo admin user change-password --username \ + --password 'NEW' --must-change-password=false + ``` ``` From d02844366a6822b29f489dd40ee57e23e248a8cb Mon Sep 17 00:00:00 2001 From: Josh Bairstow Date: Tue, 2 Jun 2026 22:42:52 +1000 Subject: [PATCH 04/10] feat(runner): add always-on systemd unit + wire into runbook step 5 System service running as the desktop user (already in docker group); starts at boot, auto-restarts, auto-claims queued jobs. WorkingDirectory under $HOME because snap Docker can't read build contexts outside home. README step 5 now covers binary install, registration, and enabling the service. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 27 ++++++++++++++------ runner/forgejo-runner.service | 47 +++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+), 7 deletions(-) create mode 100644 runner/forgejo-runner.service diff --git a/README.md b/README.md index 20c7058..86a3408 100644 --- a/README.md +++ b/README.md @@ -127,18 +127,31 @@ Then: - Create a **Personal Access Token** with **`write:package`** scope (`Settings → Applications`). The automatic Actions token CANNOT push packages. -### 5. Register the desktop runner -On the **home desktop** (has Docker + ssh): +### 5. Register the desktop runner (always-on) +On the **home desktop** (already has Docker + is in the `docker` group). Put the +binary in place, register in a home-dir working folder (snap Docker needs the +build context under `$HOME`), then install the systemd unit so it survives +reboots and auto-claims jobs: ```sh -# Get a registration token: Forgejo → Site Admin → Actions → Runners → Create +sudo install -m 0755 forgejo-runner /usr/local/bin/forgejo-runner +mkdir -p ~/forgejo-runner && cp runner/config.yml ~/forgejo-runner/ +cd ~/forgejo-runner + +# Registration token: Forgejo → Site Admin → Actions → Runners → Create forgejo-runner register --no-interactive \ --instance https://git.joshbairstow.com \ --token \ - --name desktop --labels desktop:host -forgejo-runner daemon --config /path/to/jb-infra/runner/config.yml + --name desktop --labels desktop:host # ':host' => run jobs on the host, use host Docker + +sudo cp runner/forgejo-runner.service /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable --now forgejo-runner +systemctl status forgejo-runner # expect active (running); runner shows green in Forgejo ``` -(For always-on, wrap the daemon in a systemd user service.) See -[`runner/config.yml`](runner/config.yml). +The `:host` label is what lets the `docker build` step work (the job runs on the +host using its Docker, not in a socket-less container). See +[`runner/config.yml`](runner/config.yml) and +[`runner/forgejo-runner.service`](runner/forgejo-runner.service). ### 6. Wire the jb-website repo In the `jb-website` repo settings on Forgejo (`Settings → Actions`): diff --git a/runner/forgejo-runner.service b/runner/forgejo-runner.service new file mode 100644 index 0000000..92bc95c --- /dev/null +++ b/runner/forgejo-runner.service @@ -0,0 +1,47 @@ +# ---------------------------------------------------------------------------- +# forgejo-runner.service — always-on Forgejo Actions runner on the home desktop. +# +# System service: starts at boot, no login required. Runs as your user (already +# in the `docker` group) so host-execution jobs can run `docker build`. +# +# Install: +# sudo install -m 0755 forgejo-runner /usr/local/bin/forgejo-runner # the binary +# mkdir -p ~/forgejo-runner && cp config.yml ~/forgejo-runner/ +# cd ~/forgejo-runner +# forgejo-runner register --no-interactive \ +# --instance https://git.joshbairstow.com \ +# --token --name desktop --labels desktop:host +# # ^ writes ./.runner (credentials) into ~/forgejo-runner +# sudo cp forgejo-runner.service /etc/systemd/system/ +# # (edit User=/paths below if your username differs) +# sudo systemctl daemon-reload +# sudo systemctl enable --now forgejo-runner +# systemctl status forgejo-runner # expect: active (running) +# journalctl -u forgejo-runner -f # watch it claim jobs +# +# Why WorkingDirectory is under $HOME: this machine's Docker is snap-based, and +# snap confinement blocks the daemon from reading build contexts outside the +# user's home — so the checkout/build must happen there, not in /opt. +# ---------------------------------------------------------------------------- + +[Unit] +Description=Forgejo Actions runner (jb CI) +Documentation=https://forgejo.org/docs/latest/admin/actions/runner-installation/ +After=network-online.target snap.docker.dockerd.service docker.service +Wants=network-online.target +# Disable the start-rate limiter so it keeps retrying until Docker is ready at boot. +StartLimitIntervalSec=0 + +[Service] +Type=simple +User=joshbairstow +SupplementaryGroups=docker +WorkingDirectory=/home/joshbairstow/forgejo-runner +Environment=HOME=/home/joshbairstow +Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin +ExecStart=/usr/local/bin/forgejo-runner daemon --config /home/joshbairstow/forgejo-runner/config.yml +Restart=on-failure +RestartSec=5 + +[Install] +WantedBy=multi-user.target From 2979816ac73cb0ff3e61894480443f637df60a4f Mon Sep 17 00:00:00 2001 From: Josh Bairstow Date: Tue, 2 Jun 2026 23:05:07 +1000 Subject: [PATCH 05/10] fix(runner): point unit at /usr/local/sbin, gitignore bare .runner, fix step 5 - ExecStart path matches the actual install location (/usr/local/sbin) - .gitignore: bare '.runner' so the secret is ignored in any dir (was only runner/.runner) - step 5: register inside ~/forgejo-runner (subshell), fix stray cd that broke the later 'cp runner/...' paths; note how to relocate a misplaced .runner Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 2 ++ README.md | 24 ++++++++++++++++-------- runner/forgejo-runner.service | 4 ++-- 3 files changed, 20 insertions(+), 10 deletions(-) diff --git a/.gitignore b/.gitignore index 1688b7a..615a741 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,6 @@ # Forgejo runner registration file — contains a secret, never commit. +# Bare `.runner` (no slash) matches it in any directory, incl. the repo root. +.runner runner/.runner *.env diff --git a/README.md b/README.md index 86a3408..3e09098 100644 --- a/README.md +++ b/README.md @@ -132,22 +132,30 @@ On the **home desktop** (already has Docker + is in the `docker` group). Put the binary in place, register in a home-dir working folder (snap Docker needs the build context under `$HOME`), then install the systemd unit so it survives reboots and auto-claims jobs: +Run these from the cloned `jb-infra` repo: ```sh -sudo install -m 0755 forgejo-runner /usr/local/bin/forgejo-runner +# 1. Binary — skip if already present (check: command -v forgejo-runner). +# The unit's ExecStart points at /usr/local/sbin/forgejo-runner. +sudo install -m 0755 forgejo-runner /usr/local/sbin/forgejo-runner + +# 2. Working dir + config (under $HOME so snap Docker can read build contexts). mkdir -p ~/forgejo-runner && cp runner/config.yml ~/forgejo-runner/ -cd ~/forgejo-runner -# Registration token: Forgejo → Site Admin → Actions → Runners → Create -forgejo-runner register --no-interactive \ - --instance https://git.joshbairstow.com \ - --token \ - --name desktop --labels desktop:host # ':host' => run jobs on the host, use host Docker +# 3. Register IN the working dir so .runner lands there (token: Forgejo → +# Site Admin → Actions → Runners → Create). The subshell keeps cwd in the repo. +( cd ~/forgejo-runner && forgejo-runner register --no-interactive \ + --instance https://git.joshbairstow.com \ + --token \ + --name desktop --labels desktop:host ) # ':host' => run on host, use host Docker +# 4. Install + enable the always-on service. sudo cp runner/forgejo-runner.service /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl enable --now forgejo-runner -systemctl status forgejo-runner # expect active (running); runner shows green in Forgejo +systemctl status forgejo-runner # expect active (running); runner shows green in Forgejo ``` +Already registered in the wrong directory? Don't re-register — just +`mv .runner ~/forgejo-runner/` (it's location-independent credentials). The `:host` label is what lets the `docker build` step work (the job runs on the host using its Docker, not in a socket-less container). See [`runner/config.yml`](runner/config.yml) and diff --git a/runner/forgejo-runner.service b/runner/forgejo-runner.service index 92bc95c..341ed21 100644 --- a/runner/forgejo-runner.service +++ b/runner/forgejo-runner.service @@ -5,7 +5,7 @@ # in the `docker` group) so host-execution jobs can run `docker build`. # # Install: -# sudo install -m 0755 forgejo-runner /usr/local/bin/forgejo-runner # the binary +# sudo install -m 0755 forgejo-runner /usr/local/sbin/forgejo-runner # the binary # mkdir -p ~/forgejo-runner && cp config.yml ~/forgejo-runner/ # cd ~/forgejo-runner # forgejo-runner register --no-interactive \ @@ -39,7 +39,7 @@ SupplementaryGroups=docker WorkingDirectory=/home/joshbairstow/forgejo-runner Environment=HOME=/home/joshbairstow Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin -ExecStart=/usr/local/bin/forgejo-runner daemon --config /home/joshbairstow/forgejo-runner/config.yml +ExecStart=/usr/local/sbin/forgejo-runner daemon --config /home/joshbairstow/forgejo-runner/config.yml Restart=on-failure RestartSec=5 From ef24f96b14ec7c2abba973c331163ab7df7eb365 Mon Sep 17 00:00:00 2001 From: Josh Bairstow Date: Tue, 2 Jun 2026 23:16:54 +1000 Subject: [PATCH 06/10] docs: define every Actions variable/secret in step 6 REGISTRY_USER/OWNER, REGISTRY_TOKEN, DEPLOY_HOST/USER/SSH_KEY each spelled out with what to store and why, plus deploy-keypair generation. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 3e09098..be5115c 100644 --- a/README.md +++ b/README.md @@ -163,9 +163,26 @@ host using its Docker, not in a socket-less container). See ### 6. Wire the jb-website repo In the `jb-website` repo settings on Forgejo (`Settings → Actions`): -- **Variables:** `REGISTRY_OWNER` = your lowercase owner. -- **Secrets:** `REGISTRY_USER`, `REGISTRY_TOKEN` (the PAT), `DEPLOY_HOST` - (`45.32.242.27`), `DEPLOY_USER`, `DEPLOY_SSH_KEY`. + +**Variable:** +- `REGISTRY_OWNER` — your **lowercase** Forgejo username/org; the image namespace + (`git.joshbairstow.com//jb-website`). + +**Secrets:** +- `REGISTRY_USER` — your Forgejo login username (the account that owns the PAT); + the username for `docker login`. Usually equals `REGISTRY_OWNER`. +- `REGISTRY_TOKEN` — the PAT with `write:package` scope; the `docker login` password. +- `DEPLOY_HOST` — `45.32.242.27`. +- `DEPLOY_USER` — the VPS account the runner SSHes in as: `root` to start, or a + dedicated `deploy` user (in the `docker` group) for least privilege. +- `DEPLOY_SSH_KEY` — the **private** key of a *dedicated deploy keypair* (full + file incl. `BEGIN/END` lines; paste multi-line — OpenSSH needs the newlines). + NOT your personal key. Generate + install its public half on the VPS: + ```sh + ssh-keygen -t ed25519 -f ~/.ssh/jb-deploy -C forgejo-ci-deploy -N "" + ssh-copy-id -i ~/.ssh/jb-deploy.pub @45.32.242.27 + cat ~/.ssh/jb-deploy # <- paste this into DEPLOY_SSH_KEY + ``` On the **VPS**, prepare the deploy target: ```sh From 5bb124acfd2051ddac1124d29b0939a7fa678341 Mon Sep 17 00:00:00 2001 From: Josh Bairstow Date: Tue, 2 Jun 2026 23:33:20 +1000 Subject: [PATCH 07/10] docs: rsync prod compose to VPS; clarify package vs repo visibility Repo-public does not imply package-public in Forgejo (packages are owner-scoped, created on first push). Add verify-then-fix guidance. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index be5115c..d785006 100644 --- a/README.md +++ b/README.md @@ -184,15 +184,24 @@ In the `jb-website` repo settings on Forgejo (`Settings → Actions`): cat ~/.ssh/jb-deploy # <- paste this into DEPLOY_SSH_KEY ``` -On the **VPS**, prepare the deploy target: +On the **VPS**, put the production compose file in place — rsync it from your +desktop (creates the dir + copies; only the prod `compose.yml`, not the dev one): ```sh -mkdir -p /opt/jb-website -# copy jb-website/compose.yml here (the production one — pulls, no build) -docker login git.joshbairstow.com # one-time, so `compose pull` can read the package -# (or mark the jb-website package public in Forgejo to skip VPS login) +rsync -av --rsync-path="mkdir -p /opt/jb-website && rsync" \ + ~/git/jb-website/compose.yml root@45.32.242.27:/opt/jb-website/ ``` -Create a restricted deploy user whose `authorized_keys` holds the public half -of `DEPLOY_SSH_KEY`. +For `docker compose pull` to read the image, the **package** must be pullable. +Note: **package visibility ≠ repository visibility** — packages are owned by the +user/org, so making the *repo* public does NOT necessarily make the *image* +anonymously pullable, and the package only exists after the first push. After +the first pipeline run, verify from the VPS: +```sh +docker pull git.joshbairstow.com//jb-website:latest +# unauthorized? -> set the PACKAGE public (profile → Packages → jb-website → +# settings), or run `docker login git.joshbairstow.com` once as DEPLOY_USER. +``` +If `DEPLOY_USER` is not root, it must be in the `docker` group; for least +privilege, restrict its `authorized_keys` entry with a forced `command=`. ### 7. First deploy ```sh From a1d4d82382bbd54764e04b4e05cc730b56f1179d Mon Sep 17 00:00:00 2001 From: Josh Bairstow Date: Wed, 3 Jun 2026 00:01:31 +1000 Subject: [PATCH 08/10] docs: note the desktop Docker daemon must be running/enabled for builds Stopped daemon surfaces as 'Cannot connect to the Docker daemon' at the build step (login works without it). Add the enable-at-boot recipe. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/README.md b/README.md index d785006..a47837f 100644 --- a/README.md +++ b/README.md @@ -161,6 +161,16 @@ host using its Docker, not in a socket-less container). See [`runner/config.yml`](runner/config.yml) and [`runner/forgejo-runner.service`](runner/forgejo-runner.service). +The desktop's **Docker daemon must be running and enabled at boot** — the runner +builds against it. `docker login` works without the daemon but `docker build` +won't, so a stopped daemon shows up as *"Cannot connect to the Docker daemon"* +mid-pipeline. Ensure one daemon owns `/var/run/docker.sock`: +```sh +sudo snap stop --disable docker.dockerd # if a crashing snap docker is present +sudo systemctl enable --now docker.service # classic engine, on at boot +docker info | head # verify as your user, no sudo +``` + ### 6. Wire the jb-website repo In the `jb-website` repo settings on Forgejo (`Settings → Actions`): From 75bfd3a5168e27115e3ad07e112f29c91dd053fb Mon Sep 17 00:00:00 2001 From: Josh Bairstow Date: Wed, 3 Jun 2026 00:08:11 +1000 Subject: [PATCH 09/10] docs: note Fedora docker0 ZONE_CONFLICT fix for the runner host Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/README.md b/README.md index a47837f..5bc0c18 100644 --- a/README.md +++ b/README.md @@ -170,6 +170,14 @@ sudo snap stop --disable docker.dockerd # if a crashing snap docker is pres sudo systemctl enable --now docker.service # classic engine, on at boot docker info | head # verify as your user, no sudo ``` +Fedora gotcha: if `dockerd` then fails with +`ZONE_CONFLICT: 'docker0' already bound to 'FedoraWorkstation'` (a stale binding, +often left by a prior snap-docker), free the interface and retry: +```sh +sudo firewall-cmd --permanent --zone=FedoraWorkstation --remove-interface=docker0 +sudo firewall-cmd --reload && sudo ip link delete docker0 2>/dev/null +sudo systemctl reset-failed docker.service && sudo systemctl start docker.service +``` ### 6. Wire the jb-website repo In the `jb-website` repo settings on Forgejo (`Settings → Actions`): From d326670b1fbd8632134f1da8441c3ed6a12c5ba6 Mon Sep 17 00:00:00 2001 From: Josh Bairstow Date: Wed, 3 Jun 2026 00:17:15 +1000 Subject: [PATCH 10/10] =?UTF-8?q?docs:=20correct=20ZONE=5FCONFLICT=20fix?= =?UTF-8?q?=20=E2=80=94=20delete=20NM=20docker0=20profile,=20set=20unmanag?= =?UTF-8?q?ed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause was a leftover NetworkManager docker0 bridge connection binding the interface to the default zone (firewalld removal alone didn't hold). Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 5bc0c18..9fb9ac2 100644 --- a/README.md +++ b/README.md @@ -170,12 +170,19 @@ sudo snap stop --disable docker.dockerd # if a crashing snap docker is pres sudo systemctl enable --now docker.service # classic engine, on at boot docker info | head # verify as your user, no sudo ``` -Fedora gotcha: if `dockerd` then fails with -`ZONE_CONFLICT: 'docker0' already bound to 'FedoraWorkstation'` (a stale binding, -often left by a prior snap-docker), free the interface and retry: +Fedora gotcha: if `dockerd` fails with +`ZONE_CONFLICT: 'docker0' already bound to 'FedoraWorkstation'`, the cause is a +leftover **NetworkManager** bridge profile for `docker0` (no zone set → lands in +the default zone), which firewalld then re-applies. Delete it, stop NM managing +Docker's bridges (so it can't recur at boot), and start Docker: ```sh -sudo firewall-cmd --permanent --zone=FedoraWorkstation --remove-interface=docker0 -sudo firewall-cmd --reload && sudo ip link delete docker0 2>/dev/null +sudo nmcli connection delete docker0 +sudo tee /etc/NetworkManager/conf.d/docker-unmanaged.conf >/dev/null <<'EOF' +[keyfile] +unmanaged-devices=interface-name:docker0;interface-name:br-* +EOF +sudo systemctl reload NetworkManager +sudo ip link delete docker0 2>/dev/null sudo systemctl reset-failed docker.service && sudo systemctl start docker.service ```