— the heading for one group of questions. # # The question half of a run reads as one long page: four topics, each a few # lines of context and a prompt, separated by a single blank line. Everything in # it carries the same weight, so the boundary between "which domain" and "shall # I restart your Docker daemon" is invisible — an operator scrolling back to # check what they just agreed to has nothing to aim at. A rule and a title cost # one line and give the run a visible structure. # # Distinct from the two rules already on screen, deliberately: `====` is a # whole-run banner (introduction, plan, closing summary) and `----` underlines a # heading inside a `?` explanation. This one is neither. # # Interactive only, like the introduction itself: unattended output is read by # grep and log collectors, where these would be padding. section() { have_tty && [ "$ASSUME_YES" != true ] || return 0 local title="$*" rule="" i i=$(( 54 - ${#title} )) [ "$i" -lt 3 ] && i=3 while [ "$i" -gt 0 ]; do rule="$rule─"; i=$((i - 1)); done printf '\n\033[1;36m ── %s %s\033[0m\n' "$title" "$rule" >&2 } # docker-compose.yml is product-owned: it is re-fetched and rewritten on every # run, so an operator edit to it is lost at the next upgrade — silently, which is # the part that makes it a trap. docker-compose.override.yml is the supported # place for local changes; this installer never writes or reads it beyond adding # it to the file list, and compose merges it over the base file. # # It has to be passed explicitly: compose only auto-discovers an override file # when no -f is given at all, and -f is what pins this to $DEPLOY_DIR. compose() { if [ -f "$DEPLOY_DIR/docker-compose.override.yml" ]; then docker compose --project-directory "$DEPLOY_DIR" \ -f "$DEPLOY_DIR/docker-compose.yml" \ -f "$DEPLOY_DIR/docker-compose.override.yml" "$@" else docker compose --project-directory "$DEPLOY_DIR" -f "$DEPLOY_DIR/docker-compose.yml" "$@" fi } usage() { cat <<'USAGE' DevGrail installer. curl -fsSL https://<devgrail-web>/install.sh | sudo bash curl -fsSL https://<devgrail-web>/install.sh | sudo bash -s -- [options] Options: --version=vX.Y.Z install that published release instead of the current one --rollback restore the previous images and the newest pre-upgrade snapshot, then wait for the server to become ready --auto-rollback if the server does not become ready after an upgrade, do the above automatically instead of asking --dry-run report what this run would change — release, image and asset versions, download size, free disk, settings — and exit without touching anything -y, --yes never prompt; take the prior value (or the default) for every question, and do not ask to confirm the plan (which is still printed). For unattended runs. It is deliberately not consent for the two Docker daemon settings, which restart the daemon — use --userns-remap= / --address-pools= for those. --skip-dns-check do not verify that the domains resolve to this host --userns-remap=yes|no answer the userns-remap prompt without asking. yes edits /etc/docker/daemon.json and restarts Docker (only offered on a host with no Docker state); no skips the question --address-pools=yes|no answer the address-pool prompt without asking. yes widens Docker's default pools (raising the ~27-workspace ceiling) and restarts Docker; no skips the question --workspace-image=yes|no whether to download the ~750 MiB workspace image (default yes). no installs the server stack only — most of the release in about 5% of the bytes — but no workspace can be created until a later run fetches it. That later run is this installer again with --workspace-image=yes, which re-downloads nothing else. The answer is remembered, so an upgrade keeps skipping until you say otherwise --dashboard-upgrades=yes|no whether admins may upgrade DevGrail from the dashboard (default yes, needs systemd). The dashboard only files a request; a root-owned helper on this host verifies the release and runs this installer. The answer is remembered --acme-ca=production|staging|<url> which ACME CA to order certificates from (default production). staging issues browser-untrusted certificates but is not meaningfully rate limited, so repeated installs on one hostname do not exhaust Let's Encrypt's 5-per-week limit. Each CA keeps its own certificate store, so switching back to production restores the real certs --acme-bypass=yes|no whether a TLS-passthrough admin route may carry Let's Encrypt's TLS-ALPN-01 challenges to the server behind it (default no). Turn it on to run a second DevGrail inside a workspace with certificates of its own; see "A DevGrail inside a workspace" in docs/RUNBOOK.md. This host's own certificates are unaffected. The answer is remembered -h, --help show this and exit A fresh interactive install asks for the admin username and password. Anything unattended (--yes, or no terminal) creates `admin` with a generated password, printed at the end and saved to /etc/devgrail/initial-admin-password (root only). Both are first-boot seeds; afterwards, change the password in the dashboard under Settings, or with `sudo devgrail passwd`. Inputs may also be passed as environment variables; see the header of this file. USAGE } DO_ROLLBACK=false AUTO_ROLLBACK=false DRY_RUN=false ASSUME_YES=false # The env var exists so an unattended install (cloud-init, a marketplace image) # can opt out without rewriting the command line it was handed. case "$(printf '%s' "${DEVGRAIL_SKIP_DNS_CHECK:-}" | tr '[:upper:]' '[:lower:]')" in 1|y|yes|true) SKIP_DNS_CHECK=true ;; *) SKIP_DNS_CHECK=false ;; esac # Answers to the two daemon-hardening questions (see the "Docker daemon # preflight" section). Empty means "ask"; "no" means the operator has already # decided and does not want to be asked again on the next upgrade. Anything # unrecognised is normalised to empty rather than guessed at — both settings # restart dockerd, so a typo must not be read as consent. norm_yes_no() { case "$(printf '%s' "${1:-}" | tr '[:upper:]' '[:lower:]')" in 1|y|yes|true) echo yes ;; 0|n|no|false) echo no ;; *) echo "" ;; esac } USERNS_REMAP_ANSWER="$(norm_yes_no "${DEVGRAIL_USERNS_REMAP:-}")" ADDRESS_POOLS_ANSWER="$(norm_yes_no "${DEVGRAIL_ADDRESS_POOLS:-}")" # Whether to fetch the workspace image. Unlike the two answers above, empty here # does not mean "ask" — nothing prompts for this — so a value that was set but # unrecognised is refused rather than normalised away: DEVGRAIL_WORKSPACE_IMAGE=nope # silently downloading 750 MiB is the opposite of what was asked for. WORKSPACE_IMAGE_ANSWER="$(norm_yes_no "${DEVGRAIL_WORKSPACE_IMAGE:-}")" if [ -n "${DEVGRAIL_WORKSPACE_IMAGE:-}" ] && [ -z "$WORKSPACE_IMAGE_ANSWER" ]; then die "DEVGRAIL_WORKSPACE_IMAGE takes yes or no — got '$DEVGRAIL_WORKSPACE_IMAGE'." fi # Whether admins may request upgrades from the dashboard (see the upgrade helper # below). Refused rather than normalised when unrecognised, like the workspace # image: a typo must not quietly leave a root-run path switched on. DASHBOARD_UPGRADES_ANSWER="$(norm_yes_no "${DEVGRAIL_DASHBOARD_UPGRADES:-}")" if [ -n "${DEVGRAIL_DASHBOARD_UPGRADES:-}" ] && [ -z "$DASHBOARD_UPGRADES_ANSWER" ]; then die "DEVGRAIL_DASHBOARD_UPGRADES takes yes or no — got '$DEVGRAIL_DASHBOARD_UPGRADES'." fi # Whether outer Traefik hands ACME challenges to passthrough routes (see "ACME # bypass" below). Refused when unrecognised, like the two above. ACME_BYPASS_ANSWER="$(norm_yes_no "${DEVGRAIL_ACME_BYPASS:-}")" if [ -n "${DEVGRAIL_ACME_BYPASS:-}" ] && [ -z "$ACME_BYPASS_ANSWER" ]; then die "DEVGRAIL_ACME_BYPASS takes yes or no — got '$DEVGRAIL_ACME_BYPASS'." fi while [ $# -gt 0 ]; do case "$1" in --version=*) DEVGRAIL_VERSION="${1#*=}" ;; --version) shift; DEVGRAIL_VERSION="${1:-}" [ -n "$DEVGRAIL_VERSION" ] || die "--version needs a value, e.g. --version=v0.3.0" ;; --rollback) DO_ROLLBACK=true ;; --auto-rollback) AUTO_ROLLBACK=true ;; --dry-run) DRY_RUN=true ;; -y|--yes) ASSUME_YES=true ;; --skip-dns-check) SKIP_DNS_CHECK=true ;; --userns-remap=*) USERNS_REMAP_ANSWER="$(norm_yes_no "${1#*=}")" [ -n "$USERNS_REMAP_ANSWER" ] || die "--userns-remap takes yes or no" ;; --address-pools=*) ADDRESS_POOLS_ANSWER="$(norm_yes_no "${1#*=}")" [ -n "$ADDRESS_POOLS_ANSWER" ] || die "--address-pools takes yes or no" ;; --workspace-image=*) WORKSPACE_IMAGE_ANSWER="$(norm_yes_no "${1#*=}")" [ -n "$WORKSPACE_IMAGE_ANSWER" ] || die "--workspace-image takes yes or no" ;; --dashboard-upgrades=*) DASHBOARD_UPGRADES_ANSWER="$(norm_yes_no "${1#*=}")" [ -n "$DASHBOARD_UPGRADES_ANSWER" ] || die "--dashboard-upgrades takes yes or no" ;; --acme-ca=*) DEVGRAIL_ACME_CA="${1#*=}" [ -n "$DEVGRAIL_ACME_CA" ] || die "--acme-ca takes production, staging or a directory URL" ;; --acme-bypass=*) ACME_BYPASS_ANSWER="$(norm_yes_no "${1#*=}")" [ -n "$ACME_BYPASS_ANSWER" ] || die "--acme-bypass takes yes or no" ;; -h|--help) usage; exit 0 ;; *) usage >&2; die "unknown option: $1" ;; esac shift done # --rollback restores images and a snapshot; there is nothing about it a dry run # could report without doing it. Refuse the combination rather than pick one. if [ "$DO_ROLLBACK" = true ] && [ "$DRY_RUN" = true ]; then die "--dry-run cannot be combined with --rollback." fi [ "$(id -u)" = 0 ] || die "run as root: curl -fsSL $WEB/install.sh | sudo bash" case "$(uname -m)" in x86_64|amd64) ARCH=amd64 ;; aarch64|arm64) ARCH=arm64 ;; *) die "unsupported architecture: $(uname -m)" ;; esac # --- concurrency lock -------------------------------------------------------- # Two runs at once — an impatient operator re-running while the first is still # downloading, or a scheduled upgrade landing on top of a manual one — race on # .env, versions.env, config.yaml and `docker compose up`. The loser can leave a # half-written .env (which holds the admin password) or a compose project # pointing at an image the other run has already retagged. One flock closes it. # # Held on fd 9 for the lifetime of the process, so the kernel drops it on any # exit, including a crash. That is not the whole story: fd 9 is not # close-on-exec, so anything the installer forks inherits it, and a *daemon* that # inherits it holds the lock for as long as it runs. `systemctl enable --now # docker` below is exactly that case on a host whose systemctl starts dockerd as # a plain child (containers, the local sandbox) — the installer exits, dockerd # keeps fd 9, and every later run waits out the timeout for a run that ended # minutes ago. Two defences, because neither alone is enough: # # 1. The commands that can fork a surviving daemon are run with 9>&- (see the # Docker install and the backup timer). Prevention, but only for the forks # this script knows about. # 2. The holder records its PID in $LOCK_OWNER_FILE, removed on exit. A waiter # that cannot take the lock and finds no live owner knows the lock leaked # into some other process and breaks it. Recovery, for everything else. LOCK_WAIT="${DEVGRAIL_LOCK_WAIT:-300}" # Consecutive owner-less polls before a held lock is declared leaked. There is a # sub-second window where a legitimate peer holds the flock but has not yet # written its owner file; requiring several *consecutive* misses closes it, # since a real installer writes the record microseconds after acquiring. LOCK_STALE_GRACE=5 # True when the lock is held by a process that is still running. Anything else — # no file, junk in the file, a PID that has gone — is "no live owner", which is # what makes a leaked lock recoverable. lock_owner_alive() { local pid [ -f "$LOCK_OWNER_FILE" ] || return 1 IFS= read -r pid < "$LOCK_OWNER_FILE" 2>/dev/null || return 1 case "$pid" in ''|*[!0-9]*) return 1 ;; esac kill -0 "$pid" 2>/dev/null } acquire_lock() { if ! command -v flock >/dev/null 2>&1; then warn "flock is not available on this host — running without the install lock." echo " Do not run two installers at the same time." >&2 return 0 fi # A dry run reads the same files, so it takes the lock shared: it must not # read a half-written .env, and it must not block a real install either. # On a host with no install there is nothing to race over and nothing to read, # so it does not create $INSTALL_DIR just to lock it. if [ "$DRY_RUN" = true ] && [ ! -d "$INSTALL_DIR" ]; then return 0 fi mkdir -p "$INSTALL_DIR" || die "cannot create $INSTALL_DIR" # Append, not truncate. The lock file itself carries no content, but `>` also # truncates, and keeping every open of this path non-destructive is what lets # the file be reasoned about at all while several runs hold it open. exec 9>>"$LOCK_FILE" || die "cannot open $LOCK_FILE" local mode=-x what=exclusive waited=0 stale_for=0 broke_once=false if [ "$DRY_RUN" = true ]; then mode=-s; what=shared; fi # Polled with -n rather than blocking with -w: busybox's flock (Alpine and # most minimal images) has no -w at all, and rejects it with a usage error # whose exit status is indistinguishable from "the lock is held" — which made # this refuse to run at all on those hosts. -n and -s are the forms busybox # and util-linux agree on. while ! flock "$mode" -n 9; do if lock_owner_alive; then stale_for=0 else stale_for=$((stale_for + 1)) fi # Break a leaked lock, at most once. Unlinking and reopening puts us on a # fresh inode that the leaking process does not hold, while it keeps its # lock on the orphaned one; a third run opening the path lands on the new # inode too, so mutual exclusion is restored rather than abandoned. Once # only: if something re-takes the lock afterwards, that is a real peer, and # the right answer is to wait for it and time out, not to keep unlinking. if [ "$broke_once" = false ] && [ "$stale_for" -ge "$LOCK_STALE_GRACE" ]; then warn "Breaking a lock left behind by an installer that has already exited." echo " $LOCK_FILE is held, but no running process claims it — usually a" >&2 echo " daemon (dockerd) that inherited the lock from an interrupted run." >&2 echo " Continuing." >&2 rm -f "$LOCK_FILE" exec 9>>"$LOCK_FILE" || die "cannot reopen $LOCK_FILE" broke_once=true continue fi if [ "$waited" -ge "$LOCK_WAIT" ]; then die "another install.sh is already running on this host. Waited ${LOCK_WAIT}s for the $what lock on $LOCK_FILE. Wait for that run to finish and try again. Nothing has been changed. If you are sure no installer is running, see who holds it: fuser -v $LOCK_FILE # or: lsof $LOCK_FILE and break the lock by hand with: rm -f $LOCK_FILE" fi if [ "$waited" -eq 0 ]; then log "Another install.sh holds the lock — waiting up to ${LOCK_WAIT}s..." fi sleep 1 waited=$((waited + 1)) done # Under the lock now, so this cannot race a peer. Written in both modes: a # leaked *shared* lock blocks a real install just as effectively as a leaked # exclusive one, and is just as invisible without the record. printf '%s\n' "$$" > "$LOCK_OWNER_FILE" 2>/dev/null || true } acquire_lock # Installed here, as soon as there is something to clean up, and never replaced: # a second `trap ... EXIT` would silently drop this one. Everything it removes is # guarded with :+ because it runs on every exit path, including ones taken before # TMP or the snapshot staging dir have been chosen — `rm -rf` with no operands # fails, and a failing EXIT trap is a confusing thing to leave behind after an # otherwise clean run. cleanup() { rm -rf ${TMP:+"$TMP"} ${SNAPSHOT_STAGE:+"$SNAPSHOT_STAGE"} ${SIG_DIR:+"$SIG_DIR"} 2>/dev/null || true # Only if it is still ours. Two dry runs can hold the shared lock at once, and # the loser of that race must not delete the winner's record — doing so would # make a live peer look like a leak to the next run. if [ -f "$LOCK_OWNER_FILE" ]; then local owner="" IFS= read -r owner < "$LOCK_OWNER_FILE" 2>/dev/null || owner="" [ "$owner" = "$$" ] && rm -f "$LOCK_OWNER_FILE" 2>/dev/null fi return 0 } trap cleanup EXIT # --- detect a prior install (upgrade) -------------------------------------- # Read only now that the lock is held: this is the file a concurrent run would # be rewriting, and adopting a half-written copy of it is precisely what the # lock exists to prevent. # # On re-run the previous answers live in the deploy .env (rewritten every run). # The WHOLE file is read into a PRIOR_* namespace, not a hand-maintained list of # four keys: the rule is that an upgrade preserves everything unless the operator # changes it. Reading only some keys meant HTTP_PORT/HTTPS_PORT were silently # reset to 80/443 on every upgrade — best case Traefik then failed to bind, worst # case it bound and hijacked another service's ports. # # Precedence, everywhere: explicit env var > prompt answer > prior value > default. IS_UPGRADE=false PRIOR_KEYS="" if [ -f "$DEPLOY_DIR/.env" ]; then IS_UPGRADE=true while IFS= read -r _line || [ -n "$_line" ]; do # Skip blanks, comments, and anything without a '='. case "$_line" in ''|'#'*) continue ;; *=*) : ;; *) continue ;; esac _key="${_line%%=*}" _val="${_line#*=}" # Ignore anything that is not a plain shell identifier — this file is # written by us, but it is also the file an operator hand-edits. case "$_key" in ''|*[!A-Za-z0-9_]*) continue ;; esac # Strip one layer of surrounding quotes, matching how compose reads it. case "$_val" in \"*\") _val="${_val#\"}"; _val="${_val%\"}" ;; \'*\') _val="${_val#\'}"; _val="${_val%\'}" ;; esac eval "PRIOR_$_key=\$_val" PRIOR_KEYS="${PRIOR_KEYS:+$PRIOR_KEYS }$_key" done < "$DEPLOY_DIR/.env" fi # prior <KEY> — the prior install's value for KEY, or empty. prior() { eval "printf '%s' \"\${PRIOR_$1:-}\""; } # resolve <KEY> [default] — settle KEY per the precedence above. Prompted keys # call prompt() instead, passing `prior KEY` as the offered default. resolve() { local key="$1" def="${2:-}" cur eval "cur=\${$key:-}" [ -n "$cur" ] && return 0 cur="$(prior "$key")" [ -n "$cur" ] || cur="$def" eval "$key=\$cur" } # --- TLS switch (default on) ------------------------------------------------ # DEVGRAIL_TLS=off serves plain HTTP on :80 — only for a private network or # behind an external TLS terminator. Default on = HTTPS via Traefik + Let's # Encrypt. The choice selects which Traefik router set to install and the scheme # advertised in every server-generated URL (see internal/config Scheme()). resolve DEVGRAIL_TLS on case "$(printf '%s' "$DEVGRAIL_TLS" | tr '[:upper:]' '[:lower:]')" in off|false|0|no) TLS_ENABLED=false; SCHEME=http; TRAEFIK_ASSET=traefik-dynamic-http.yml ;; *) TLS_ENABLED=true; SCHEME=https; TRAEFIK_ASSET=traefik-dynamic.yml ;; esac # --- workspace image (default on) ------------------------------------------- # The server stack is the install; the workspace image is the only genuinely # optional artifact, and at ~750 MiB against the server's ~45 it is very nearly # the whole download. `no` skips it: the stack comes up and the dashboard works, # but creating a workspace needs the image, so this is a deferral rather than a # mode. Re-running this installer with --workspace-image=yes fetches it and # re-downloads nothing else, which is why there is no separate fetch command. # # Extracted by test/installer/workspace_image_test.sh from this marker to the # closing brace below — keep them adjacent. WORKSPACE_IMAGE_DEFAULT=yes # settle_workspace_image <answer> <prior> — yes or no, by the precedence this # file uses everywhere: explicit flag/env > prior .env value > default. The # prior value is re-normalised rather than trusted, because .env is a file # operators hand-edit. settle_workspace_image() { local chosen="$1" prior_val="$2" [ -n "$chosen" ] || chosen="$(norm_yes_no "$prior_val")" [ -n "$chosen" ] || chosen="$WORKSPACE_IMAGE_DEFAULT" printf '%s' "$chosen" } # Settled back into the DEVGRAIL_ name, the way resolve() does: it is the key # written to deploy/.env, read back as the prior answer on the next run, and # reported by the dry run's Settings diff. DEVGRAIL_WORKSPACE_IMAGE="$(settle_workspace_image "$WORKSPACE_IMAGE_ANSWER" "$(prior DEVGRAIL_WORKSPACE_IMAGE)")" # Flag/env > prior .env > yes. The prior value is re-normalised, not trusted, # because .env is a file operators hand-edit. DEVGRAIL_DASHBOARD_UPGRADES="$DASHBOARD_UPGRADES_ANSWER" [ -n "$DEVGRAIL_DASHBOARD_UPGRADES" ] || DEVGRAIL_DASHBOARD_UPGRADES="$(norm_yes_no "$(prior DEVGRAIL_DASHBOARD_UPGRADES)")" [ -n "$DEVGRAIL_DASHBOARD_UPGRADES" ] || DEVGRAIL_DASHBOARD_UPGRADES=yes sha256_of() { if command -v sha256sum >/dev/null 2>&1; then sha256sum "$1" | awk '{print $1}' else shasum -a 256 "$1" | awk '{print $1}'; fi } # Same, over stdin rather than a file. sha256_str() { if command -v sha256sum >/dev/null 2>&1; then sha256sum | awk '{print $1}' else shasum -a 256 | awk '{print $1}'; fi } gen_secret() { openssl rand -hex 32 2>/dev/null || head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n'; } gen_password() { (openssl rand -base64 18 2>/dev/null || head -c 15 /dev/urandom | base64) | tr -dc 'A-Za-z0-9' | cut -c1-20; } # --- admin credential validation -------------------------------------------- # The password ends up in deploy/.env as a bare, unquoted value, read back by # docker compose, by this script's own PRIOR_* reader and by the test harnesses. # Rather than teach all of them to unquote, the charset an operator may type is # restricted to what survives that file verbatim: no whitespace, no quotes, and # none of # $ \ ` & | < > ( ) [ ] { } / which either end the value or mean # something to compose. A generated password (gen_password) is alphanumeric, so # it is inside this set by construction. Passwords set later through the app or # `devgrail passwd` never touch .env and are not restricted this way. ADMIN_PASSWORD_MIN=12 valid_admin_password() { local pw="$1" [ "${#pw}" -ge "$ADMIN_PASSWORD_MIN" ] || return 1 # bcrypt truncates at 72 bytes, so anything longer is only half-checked. [ "${#pw}" -le 72 ] || return 1 case "$pw" in *[!A-Za-z0-9!%*+,.:\;=?@^_~-]*) return 1 ;; esac return 0 } # 3-32 characters, starting alphanumeric. Matches what a login form and the # users table accept, and leaves no room for a leading dash or an empty name. valid_admin_username() { case "$1" in ''|[!A-Za-z0-9]*) return 1 ;; *[!A-Za-z0-9._-]*) return 1 ;; esac [ "${#1}" -ge 3 ] && [ "${#1}" -le 32 ] } # read_secret VAR PROMPT — read a line from the terminal without echoing it. # Same /dev/tty convention as prompt(): stdin is the piped script. read_secret() { local var="$1" msg="$2" val printf '%s: ' "$msg" > /dev/tty read -rs val < /dev/tty || true printf '\n' > /dev/tty eval "$var=\$val" } # --- readiness --------------------------------------------------------------- # `docker compose up -d` returning 0 only means containers were *created*. The # failure that matters most — the server crash-looping on a migration — passes # that check and hands the customer a URL that does not work. /api/v1/healthz # answers the real question (database reachable, Docker reachable, schema # matches the binary) and returns 503 until all three hold. # # Two probe targets: the public URL is what the customer will actually use, but # it depends on DNS already pointing here and on Let's Encrypt having issued, # neither of which is true on a first install. The container-local address # verifies the server itself, so a not-yet-pointed DNS record does not masquerade # as a broken server. HEALTH_TIMEOUT="${DEVGRAIL_HEALTH_TIMEOUT:-180}" # The URL the operator actually opens, port included. HTTP_PORT/HTTPS_PORT are a # supported setting and are carried across upgrades on purpose (see the # prior-install block), so an install on :8443 is reached only on :8443 — a # banner or a probe that names the default port instead points at nothing. # A function rather than a variable because --rollback prints it before the # ports have been resolved; the prior .env is the source there. dashboard_url() { local port if [ "$SCHEME" = https ]; then port="${HTTPS_PORT:-$(prior HTTPS_PORT)}"; port="${port:-443}" [ "$port" = 443 ] && { printf '%s://%s' "$SCHEME" "$DEVGRAIL_DOMAIN"; return; } else port="${HTTP_PORT:-$(prior HTTP_PORT)}"; port="${port:-80}" [ "$port" = 80 ] && { printf '%s://%s' "$SCHEME" "$DEVGRAIL_DOMAIN"; return; } fi printf '%s://%s:%s' "$SCHEME" "$DEVGRAIL_DOMAIN" "$port" } # The two A records this install needs, with the arrows aligned whatever length # the domains are — they were hand-spaced for one example domain and lined up # for nobody else. $1 is the indent. dns_records() { local indent="${1:- }" wild="*.$DEVGRAIL_BASE_DOMAIN" w w=${#DEVGRAIL_DOMAIN} [ "${#wild}" -gt "$w" ] && w="${#wild}" printf '%sA %-*s -> this VPS ' "$indent" "$w" "$DEVGRAIL_DOMAIN" printf '%sA %-*s -> this VPS ' "$indent" "$w" "$wild" } probe_health() { # Public route through Traefik. -k because the certificate may still be the # ACME self-signed placeholder; this checks readiness, not trust. curl -fsSk --max-time 5 "$(dashboard_url)/api/v1/healthz" >/dev/null 2>&1 && return 0 # The server directly, from inside its own container. Compose exec keeps stdin # attached even with -T; detach it explicitly because, under `curl | bash`, # stdin is the unread remainder of this script. Letting Compose consume it # makes bash reach EOF after this block and silently skip the closing summary. compose exec -T devgrail-server /usr/local/bin/devgrail-server -healthcheck </dev/null >/dev/null 2>&1 } wait_for_health() { local waited=0 # step() rather than log(): this is a phase an operator waits through, and on # a first install it is the longest one after the download. --rollback calls # wait_for_health too, outside any counted sequence — step() prints an # unnumbered heading there, which is what STEP_TOTAL=0 is for. step "Waiting for DevGrail to become ready (up to ${HEALTH_TIMEOUT}s)..." while [ "$waited" -lt "$HEALTH_TIMEOUT" ]; do if probe_health; then log "DevGrail is ready." return 0 fi sleep 3 waited=$((waited + 3)) done return 1 } # --- rollback ---------------------------------------------------------------- # data_volume prints the Docker volume backing /var/lib/devgrail. It is resolved # from the running container rather than assumed from the Compose project name, # which is derived from the deploy directory and so differs between installs. data_volume() { local vol vol="$(docker inspect devgrail-server \ --format '{{range .Mounts}}{{if eq .Destination "/var/lib/devgrail"}}{{.Name}}{{end}}{{end}}' 2>/dev/null)" || vol="" if [ -n "$vol" ]; then printf '%s' "$vol"; return 0; fi docker volume ls --format '{{.Name}}' 2>/dev/null | grep -E '(^|_)devgrail-data$' | head -n1 || true } newest_snapshot() { ls -1t "$BACKUP_DIR"/pre-upgrade-*.tgz 2>/dev/null | head -n1 || true } # Point :latest back at the image the previous run displaced. This is a tag move, # not a copy — which is the whole reason the retag is worth doing on every # upgrade: it turns rollback into a retag plus a compose recreate. rollback_images() { local slug moved=0 for slug in devgrail-server devgrail-container; do if docker image inspect "$slug:previous" >/dev/null 2>&1; then docker tag "$slug:previous" "$slug:latest" moved=$((moved + 1)) else warn "no $slug:previous image on this host — leaving $slug:latest where it is" fi done [ "$moved" -gt 0 ] } # restore_snapshot <archive> — put the database and config.yaml back as they were # before the upgrade that took this archive. restore_snapshot() { local archive="$1" vol stage vol="$(data_volume)" [ -n "$vol" ] || die "cannot find the DevGrail data volume — refusing to guess which volume to overwrite." stage="$(mktemp -d)" tar xzf "$archive" -C "$stage" || { rm -rf "$stage"; die "could not read $archive"; } if [ ! -f "$stage/devgrail.db" ] || [ ! -f "$stage/config.yaml" ]; then rm -rf "$stage" die "$archive is missing devgrail.db or config.yaml — a database without the config.yaml that keys its encryption is not a usable restore." fi compose stop devgrail-server >/dev/null 2>&1 || true # config.yaml first, deliberately: the JWT secret in it decrypts everything in # the database, so a half-completed restore that stopped here is recoverable, # whereas one that restored only the database is not. install -m 600 "$stage/config.yaml" "$CONFIG_DIR/config.yaml" \ || { rm -rf "$stage"; die "could not restore $CONFIG_DIR/config.yaml"; } # The -wal/-shm sidecars belong to the database being replaced. Leaving them # next to a restored file is how a "successful" restore produces a corrupt # database, so they go before the copy lands. docker run --rm --userns=host \ -v "$vol":/var/lib/devgrail \ -v "$stage":/restore:ro \ --entrypoint /bin/sh devgrail-server:latest -c ' rm -f /var/lib/devgrail/devgrail.db-wal /var/lib/devgrail/devgrail.db-shm && cp /restore/devgrail.db /var/lib/devgrail/devgrail.db' \ || { rm -rf "$stage"; die "could not restore the database into volume '$vol'"; } rm -rf "$stage" log "Restored database + config.yaml from $archive" } # do_rollback returns 0 when the rolled-back stack answers its readiness probe. do_rollback() { [ -f "$DEPLOY_DIR/docker-compose.yml" ] \ || die "no DevGrail install found at $INSTALL_DIR — there is nothing to roll back to." # Read from the prior .env rather than prompting: a rollback happens when # something is already wrong, and it should not stop to ask questions it can # answer itself. DEVGRAIL_DOMAIN="${DEVGRAIL_DOMAIN:-$(prior DEVGRAIL_DOMAIN)}" local archive archive="$(newest_snapshot)" log "Rolling back to the previous images and snapshot..." rollback_images || warn "no previous images were available to restore" if [ -n "$archive" ]; then restore_snapshot "$archive" else warn "No pre-upgrade snapshot in $BACKUP_DIR — restoring images only." echo " The database keeps whatever schema the newer server migrated it to," >&2 echo " and the older binary will refuse to start against it (by design)." >&2 fi compose up -d wait_for_health } # Whether this run has a terminal to talk to. # # Not a test that the device node exists, which is what every one of these # checks used to be: the node exists inside a container started without a TTY, # but opening it fails with ENXIO. Existence answered "yes, ask the operator" in # precisely the environment where there is nobody to answer — the installer # stopped on the first prompt with "No such device or address" instead of taking # the default it already had. Opening it is the only test that distinguishes the # two, and it errs the safe way: an unopenable terminal is no terminal, and no # terminal means take the default. have_tty() { { : < /dev/tty; } 2>/dev/null; } # Prompt for a required input on /dev/tty (stdin is the piped script). An # optional default ($3, e.g. the prior install's answer) is shown in brackets; # pressing Enter keeps it. Without a terminal a default is used, else we die. # # $4, when given, names a function printing a short explanation of what the # value is for. Printed rather than offered behind `?` the way ask_yn does it: # these are the first questions of a fresh install, and an operator who has # never seen DevGrail cannot be expected to know that "base domain" means a # wildcard record. It is only reached on the interactive path, so an upgrade # taking prior answers and an unattended run stay exactly as quiet as before. prompt() { local var="$1" msg="$2" def="${3:-}" explain="${4:-}" val why; eval "val=\${$var:-}" [ -n "$val" ] && return # --yes is the same contract as having no terminal: take the offered default, # which on an upgrade is the prior install's answer. if [ "$ASSUME_YES" = true ] || ! have_tty; then [ -n "$def" ] && { eval "$var=\$def"; return; } if [ "$ASSUME_YES" = true ]; then why="--yes was given, so nothing is prompted for" else why="no terminal is available"; fi die "$var is not set and there is no prior value to fall back to ($why). Pass it as an environment variable." fi [ -n "$explain" ] && "$explain" >&2 if [ -n "$def" ]; then printf '%s [%s]: ' "$msg" "$def" > /dev/tty else printf '%s: ' "$msg" > /dev/tty; fi read -r val < /dev/tty || true val="${val:-$def}" [ -n "$val" ] || die "$var is required" eval "$var=\$val" } # Ask a yes/no question on /dev/tty (stdin is the piped script). $2 is the answer # a bare Enter takes. Callers decide what --yes and "no terminal" mean before # getting here; this only runs when there is a terminal and a question to ask. # # $3, when given, names a function printing the long explanation, and adds `?` to # the accepted answers. The two daemon questions below have a page of reasoning # behind them that is genuinely worth reading once and skipped forever after — # printed before the prompt it is a wall of text arriving at the least patient # moment of the install, so it is offered instead of imposed. Asking `?` re-asks # the question afterwards, so it costs nothing to look. # # Anything that is not an answer is re-asked rather than read as a no. A typo'd # key used to be silently taken as the negative answer — which for "Proceed?" is # an aborted install and for the daemon questions is a decision the operator # never made. Only a bare Enter takes the default, because that is the one # "invalid" answer that was actually meant. ask_yn() { local msg="$1" def="${2:-n}" explain="${3:-}" ans hint case "$def" in y|Y) hint='[Y/n]' ;; *) hint='[y/N]' ;; esac [ -n "$explain" ] && hint="${hint%]}/?]" while :; do printf '%s %s: ' "$msg" "$hint" > /dev/tty read -r ans < /dev/tty || ans="" ans="${ans:-$def}" case "$ans" in y|Y|yes|YES|Yes) return 0 ;; n|N|no|NO|No) return 1 ;; '?'|h|H|help|HELP) # With nothing to explain, `?` is just another answer that is not one. [ -n "$explain" ] && { "$explain" >&2; continue; } ;; esac if [ -n "$explain" ]; then printf ' Please answer y or n, or ? for the details.\n' > /dev/tty else printf ' Please answer y or n.\n' > /dev/tty fi done } # --rollback is a mode of its own: it touches no registry, asks nothing, and must # not fall through into an install. Handled here, once the helpers above exist # and before any prompt. if [ "$DO_ROLLBACK" = true ]; then if do_rollback; then cat <<EOF ============================================================ Rolled back. DevGrail is answering its readiness probe. Dashboard: $(dashboard_url)/ Images: devgrail-server:latest now points at the previous build. Versions: $VERSIONS_FILE still names the release you rolled off — it is written before the health gate, so this cannot un-write it. /healthz is the authority on what is running. Forward: re-running install.sh upgrades again, and moves the tag back without re-downloading anything. Logs: docker compose --project-directory $DEPLOY_DIR logs -f ============================================================ EOF exit 0 fi echo >&2 printf '\033[1;31merror:\033[0m %s\n' "The rolled-back stack did not become ready either." >&2 compose logs --tail=50 devgrail-server >&2 || true echo " Snapshots available: $BACKUP_DIR" >&2 exit 1 fi # --- introduction ------------------------------------------------------------ # The first thing a fresh install used to print was "Hostname for the DevGrail # dashboard:" — a question that presumes the reader already knows what DevGrail # will do with a hostname, and that DNS has to exist before a certificate can be # issued for it. Both are cheap to say once, here, while nothing has happened # yet and the answers can still be gathered. # # Interactive only: an unattended run has nobody to read it, and the log lines # it would push off the screen are the ones that matter there. intro_install() { cat >&2 <<EOF ============================================================ DevGrail installer DevGrail turns this VPS into a home for AI coding agents: isolated Docker workspaces, each with a browser terminal and its own subdomains, all managed from one dashboard. How this run goes: 1. Questions — all of them, up front. Each has a suggested answer in [brackets]; press Enter to take it. 2. Prerequisites — Docker and a few small tools, installed if this host does not already have them. 3. A plan, printed for you to confirm. The images, the Docker settings you chose and the stack itself all come after you say yes. Usually 5-10 minutes, most of it download. Worth having ready: - a hostname for the dashboard, e.g. devgrail.example.com - a base domain for workspace apps, e.g. example.com - an email address for Let's Encrypt notices - DNS pointing both names at this VPS: A devgrail.example.com -> this VPS A *.example.com -> this VPS They are checked later in this run; if they are not there yet you can add them and re-check without losing answers. DNS is not a prerequisite for installing — press d below for what an install without it does and does not get you. ============================================================ EOF } # The long version: what this host has to be, what ends up on it, and what DNS # is actually for. Offered rather than printed, the way ask_yn offers `?`: an # operator installing their second VPS should not have to scroll past a page # they have read, and one installing their first should not have to guess what # "prerequisites" a curl | bash line assumed about their machine. # # The figures come from README.md's Requirements section; keep them in step. intro_details() { cat >&2 <<EOF ============================================================ What DevGrail is, and what this run does to this host The product A dashboard, a REST API, a CLI and an MCP endpoint in front of Docker workspaces. A workspace is a container with a browser terminal, an SSH login, its own Docker network and its own app subdomains. Coding agents run inside those containers — never on this host directly. What this host needs - Root (you are running this with sudo) on x86-64 or ARM64 Linux. Ubuntu 22.04/24.04 is the primary target; Debian 12 and Rocky/Alma 9 use the same paths. Alpine works but has no systemd, so no scheduled backups there. - 2 GB RAM and 20 GB disk to run the stack at all; 8 GB and 60 GB+ if you mean to run several workspaces at once. A workspace is a full Ubuntu container — size for the ones you will run concurrently, not for DevGrail itself. - Ports 80 and 443 reachable from the internet. Let's Encrypt answers its challenge on the real 443; HTTP_PORT/HTTPS_PORT exist for running behind another TLS terminator, not for avoiding that. - Outbound HTTPS to get.docker.com if Docker is missing, to Let's Encrypt, and to the registry serving this release: $WEB - A domain you control — see DNS below. What gets installed - Docker Engine and the compose plugin from get.docker.com, only if this host has none. jq and openssl through the package manager, only if missing: one reads the release manifest, the other verifies its signature. - Two images, fetched over HTTPS, checked against a SHA-256 in a signed manifest and loaded with 'docker load' — no registry login, no 'docker pull': devgrail-server the product, one static binary devgrail-container the workspace base image, ~750 MiB (skip with --workspace-image=no) - Three long-running containers: Traefik at the edge, the DevGrail server, and a Docker socket proxy that is what keeps the server off the raw socket. - Files: $INSTALL_DIR (compose file, assets, backups), $CONFIG_DIR/config.yaml (root-only — it holds the key that encrypts every secret in the database) and /usr/local/bin/devgrail (backup, restore, list, passwd). - fs.inotify.max_user_instances raised to 1024 when it is lower, kept in /etc/sysctl.d/60-devgrail-inotify.conf: workspace file watchers share root's limit with systemd. - Docker volumes for the database and the certificates. Workspaces get their own as you create them. - Only if you say yes, each asked separately: two /etc/docker/daemon.json settings (both restart Docker) and a weekly systemd backup timer. Nothing is written outside those paths, and uninstall.sh removes what this script created. DNS, and installing before it exists Two records at your registrar or DNS host — the dashboard's own name, and a wildcard so <label>.<base domain> reaches a workspace app. They are not a prerequisite for installing. This run checks them and lets you continue anyway: the stack comes up, the dashboard is served and everything on this host is finished. What you do not get is a trusted certificate — Let's Encrypt cannot validate a name that does not reach this VPS, so Traefik serves its own placeholder and the browser calls the site untrusted until the records resolve. Once they do, Traefik orders the certificate itself; you can force the attempt with docker compose --project-directory $DEPLOY_DIR restart traefik No re-install and no re-run is needed. The one cost of that order is Let's Encrypt's rate limits, which count failures — so DNS first is cheaper when it is a choice at all. Time, and undoing it 5-10 minutes, most of it the workspace image download. Every question comes first, then one plan to confirm. Re-running this script upgrades in place after snapshotting the database; --rollback puts the previous release back. ============================================================ EOF } # One keypress in front of the page above. The loop is what makes it free to # look: reading the details returns here, so 'd' costs nothing and Enter still # starts the run. Anything that is neither is asked again rather than taken as # "begin": starting the run is not what a mistyped key should mean. intro_details_gate() { local ans while :; do printf ' Press Enter to begin, or d for the details (prerequisites, what\n gets installed, and installing before DNS exists): ' > /dev/tty read -r ans < /dev/tty || ans="" case "$ans" in d|D|details|'?'|h|H|help|HELP) intro_details ;; ''|y|Y|yes|YES|Yes) return 0 ;; *) printf ' Please press Enter to begin, or d for the details.\n' > /dev/tty ;; esac done } intro_upgrade() { cat >&2 <<EOF ============================================================ DevGrail upgrade An existing install was found in $INSTALL_DIR. Your current answers are offered as [defaults] — press Enter to keep each one, or type a new value to change it. A plan follows, and nothing is downloaded or restarted until you have confirmed it. The database and config.yaml are snapshotted before the new server starts, so this run can be undone with: install.sh --rollback ============================================================ EOF } if have_tty && [ "$ASSUME_YES" != true ]; then if [ "$IS_UPGRADE" = true ]; then intro_upgrade else intro_install # Only on a fresh install: an upgrade is being run by somebody who already # has all of this, and its own introduction is four lines by design. intro_details_gate fi elif [ "$IS_UPGRADE" = true ]; then log "Existing DevGrail install detected — current answers shown as [defaults]; press Enter to keep, or type a new value to change." fi explain_domain() { cat >&2 <<EXPLAIN The dashboard's own hostname. You log in here, and the API, the CLI and the MCP endpoint all live behind this one name. Its TLS certificate is issued for exactly this name, so an A record for it has to point at this VPS. A subdomain of a domain you own is the usual choice, and better than the bare domain: browsers that open the dashboard refuse plain HTTP for every name under it while they keep using it. EXPLAIN } explain_base_domain() { cat >&2 <<EXPLAIN Anything you run inside a workspace — a dev server on :3000, a preview build — is published as <label>.<base domain>, so this name needs a WILDCARD record: *.<base> -> this VPS. Usually the bare domain the dashboard sits on: example.com when the dashboard is devgrail.example.com. This only seeds the first entry — more base domains can be added later under Settings -> App subdomains, and re-running this installer does not overwrite a list you have edited there. EXPLAIN } explain_acme_email() { cat >&2 <<EXPLAIN Where Let's Encrypt sends certificate expiry and problem notices. It is not published anywhere and is used for nothing else — any mailbox you actually read will do. EXPLAIN } # bare_domain_hsts_note <domain> — an explanation, or nothing, when the # dashboard is about to move onto a bare domain such as example.com. The server # sends Strict-Transport-Security with includeSubDomains on the dashboard's # name (cmd/server/securityheaders.go), so every browser that opens it refuses # plain HTTP for that name and every name under it, and offers no way past a # certificate error there, until the max-age runs out. On devgrail.example.com # that is DevGrail's own names. On example.com it is every host the owner runs, # a NAS or a router page among them, and no server-side change takes it back. # # Silent without TLS (no header is sent), when max-age is 0 (nothing is pinned), # and when the name is the one already installed: its browsers are pinned # already, and asking again on every upgrade would only teach the operator to # press Enter. Two labels is the test. It misses a bare domain under a two-part # suffix (example.co.uk) and flags a name bought for DevGrail alone. Telling # those apart needs the public suffix list, and both mistakes are cheap: this # explains and asks, and never refuses. bare_domain_hsts_note() { local d d="$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')" d="${d%.}" [ "$TLS_ENABLED" = true ] || return 0 [ "$(prior DEVGRAIL_HSTS_MAX_AGE)" = 0 ] && return 0 [ "$d" = "$(prior DEVGRAIL_DOMAIN | tr '[:upper:]' '[:lower:]')" ] && return 0 case "$d" in .*|*..*|*.*.*) return 0 ;; ?*.?*) ;; *) return 0 ;; esac cat <<NOTE The dashboard tells browsers to use only HTTPS for its name and every name under it (HSTS with includeSubDomains), renewed on every visit and kept for a day after the last one by default. On $d that covers every host under $d, not only DevGrail's: a browser that uses the dashboard refuses plain HTTP to any of them and gives no way past a certificate error on any of them. Nothing on the server can undo that in a browser that has seen it. If $d serves nothing else, keep it. Otherwise a subdomain such as devgrail.$d keeps this to DevGrail's own names. NOTE } section "Names" # Asked again on a no rather than left to the plan's Proceed?, which would throw # away every answer given so far. A name passed in, or taken unattended, is # kept, with the note left in the log. while :; do prompt DEVGRAIL_DOMAIN " Hostname for the DevGrail dashboard, e.g. devgrail.example.com" "$(prior DEVGRAIL_DOMAIN)" explain_domain _note="$(bare_domain_hsts_note "$DEVGRAIL_DOMAIN")" [ -n "$_note" ] || break warn "$DEVGRAIL_DOMAIN is a bare domain." printf '%s\n' "$_note" >&2 if [ "$ASSUME_YES" = true ] || ! have_tty || ask_yn " Use $DEVGRAIL_DOMAIN anyway?" n; then break fi DEVGRAIL_DOMAIN= done unset _note # Seeds the deployment's first base domain on first boot only; after that the # database owns the list (internal/db.SeedBaseDomain), and more can be added, # renamed or re-defaulted under Settings -> App subdomains. Re-running this # installer therefore does NOT change a list that has already been edited there. prompt DEVGRAIL_BASE_DOMAIN " Base domain for workspace app subdomains, e.g. example.com" "$(prior DEVGRAIL_BASE_DOMAIN)" explain_base_domain # Only needed for Let's Encrypt; plain-HTTP installs skip it (ACME is never run). if [ "$TLS_ENABLED" = true ]; then prompt ACME_EMAIL " Email for Let's Encrypt certificates" "$(prior ACME_EMAIL)" explain_acme_email fi # --- ACME CA (production / staging / custom) -------------------------------- # Let's Encrypt production issues at most 5 certificates per exact set of # identifiers per 168h. Any test cycle that starts from an empty certificate # store re-orders on every run — `uninstall.sh --purge` removes the traefik-acme # volume, and the first install after enabling userns-remap moves Docker's data # root to /var/lib/docker/<uid>.<gid>, orphaning the old volume just the same — # so five reinstalls on one hostname exhaust the quota. Traefik then falls back # to its self-signed "TRAEFIK DEFAULT CERT", which is what an invalid-certificate # browser warning on a freshly installed host actually means. Ordering from the # staging CA avoids it: the certificates are untrusted, but nothing else about # the issuance path differs, so it still exercises DNS, :443 and TLS-ALPN-01. # # Not prompted: production is right for every real install, and this only exists # for repeated testing against one hostname. ACME_CA_PRODUCTION_URL=https://acme-v02.api.letsencrypt.org/directory ACME_CA_STAGING_URL=https://acme-staging-v02.api.letsencrypt.org/directory # set_acme_ca <production|staging|url> — settle DEVGRAIL_ACME_CASERVER and the # store file that goes with it. The full URLs are accepted as well as the # aliases so a prior .env, which records the URL, round-trips unchanged. # # The store is per-CA on purpose: acme.json holds the registered ACME account, # and an account issued by one CA is unknown to another, so a shared file makes # every order fail with "account does not exist". Separate files also mean a # staging run leaves the production certificates already on this host intact, # and switching back picks them up again instead of re-ordering. set_acme_ca() { case "$(printf '%s' "${1:-}" | tr '[:upper:]' '[:lower:]')" in ''|production|prod|live|"$ACME_CA_PRODUCTION_URL") DEVGRAIL_ACME_CASERVER="$ACME_CA_PRODUCTION_URL" DEVGRAIL_ACME_STORAGE=acme.json ;; staging|test|"$ACME_CA_STAGING_URL") DEVGRAIL_ACME_CASERVER="$ACME_CA_STAGING_URL" DEVGRAIL_ACME_STORAGE=acme-staging.json ;; http://*|https://*) # A custom directory (pebble, an internal CA). The store name is derived # from the URL so two custom CAs cannot share a store or clobber acme.json. DEVGRAIL_ACME_CASERVER="$1" DEVGRAIL_ACME_STORAGE="acme-$(printf '%s' "$1" | sha256_str | cut -c1-8).json" ;; *) die "the ACME CA must be production, staging, or a directory URL starting with https:// — got '$1'. Set it with --acme-ca=, DEVGRAIL_ACME_CA, or DEVGRAIL_ACME_CASERVER in $DEPLOY_DIR/.env." ;; esac } if [ -n "${DEVGRAIL_ACME_CA:-}" ]; then set_acme_ca "$DEVGRAIL_ACME_CA" else # No explicit choice: keep what the prior install used. An upgrade must not # silently move a staging test back onto production and spend its quota. set_acme_ca "$(prior DEVGRAIL_ACME_CASERVER)" fi ACME_CA_IS_PRODUCTION=true [ "$DEVGRAIL_ACME_CASERVER" = "$ACME_CA_PRODUCTION_URL" ] || ACME_CA_IS_PRODUCTION=false # --- ACME bypass (a DevGrail inside a workspace) ----------------------------- # Traefik answers every TLS-ALPN-01 handshake on :443 itself, before routing, so # a second DevGrail behind a passthrough admin route (*.inner.example.com -> # a workspace) never sees its own challenges and cannot get a certificate. # DEVGRAIL_ACME_BYPASS=true sets the websecure entrypoint's allowACMEByPass, which # lets a TCP router whose HostSNI matches take the challenge instead; names no # TCP router claims, this install's own included, are still answered here. # deploy/docker-compose.yml has the flag, docs/RUNBOOK.md the whole setup. # # Off unless asked for, and written as true/false because that is what Traefik # parses. Flag/env > prior .env > false; the prior value is re-normalised rather # than trusted, because .env is a file operators hand-edit. # # Extracted by test/installer/acme_bypass_test.sh from this marker to the # closing brace below — keep them adjacent. ACME_BYPASS_DEFAULT=false # settle_acme_bypass <answer> <prior> — true or false. settle_acme_bypass() { local chosen="$1" [ -n "$chosen" ] || chosen="$(norm_yes_no "$2")" case "$chosen" in yes) printf true ;; no) printf false ;; *) printf '%s' "$ACME_BYPASS_DEFAULT" ;; esac } DEVGRAIL_ACME_BYPASS="$(settle_acme_bypass "$ACME_BYPASS_ANSWER" "$(prior DEVGRAIL_ACME_BYPASS)")" # --- admin credentials ------------------------------------------------------ # Both are first-boot seeds: the server creates the admin account from them the # first time it starts and ignores them ever after (internal/db/seed.go), so an # upgrade asks nothing and simply carries the prior .env values over. Changing # them later is `devgrail passwd` or Settings -> Password in the dashboard. # # Unattended (--yes, or no terminal) keeps what every install did before this # was askable: the user `admin`, and a generated password printed in the closing # banner — the only place it is ever shown. ADMIN_PW_GENERATED=false ADMIN_USER="${DEVGRAIL_ADMIN_USERNAME:-}" ADMIN_PW="${DEVGRAIL_ADMIN_PASSWORD:-}" # Validate what the environment supplied before it reaches deploy/.env: a value # this script cannot round-trip would produce a stack nobody can log into, and # the failure would surface as "wrong password", not as a bad install. if [ -n "$ADMIN_USER" ] && ! valid_admin_username "$ADMIN_USER"; then die "DEVGRAIL_ADMIN_USERNAME must be 3-32 characters — letters, digits, dot, dash or underscore — starting with a letter or digit." fi if [ -n "$ADMIN_PW" ] && ! valid_admin_password "$ADMIN_PW"; then die "DEVGRAIL_ADMIN_PASSWORD must be $ADMIN_PASSWORD_MIN-72 characters of letters, digits or the punctuation ! % * + , - . : ; = ? @ ^ _ ~ (it is stored verbatim in deploy/.env, which quotes nothing)." fi [ -n "$ADMIN_USER" ] || ADMIN_USER="$(prior DEVGRAIL_ADMIN_USERNAME)" [ -n "$ADMIN_PW" ] || ADMIN_PW="$(prior DEVGRAIL_ADMIN_PASSWORD)" if [ "$IS_UPGRADE" = false ] && have_tty && [ "$ASSUME_YES" != true ]; then section "Admin account" echo " The first login for the dashboard, and the account that can see and" >&2 echo " manage every workspace on this host. Both parts can be changed later" >&2 echo " — Settings -> Password in the dashboard, or 'devgrail passwd'." >&2 while [ -z "$ADMIN_USER" ]; do prompt ADMIN_USER " Admin username" admin valid_admin_username "$ADMIN_USER" && break warn "3-32 characters: letters, digits, dot, dash or underscore, starting with a letter or digit." ADMIN_USER="" done if [ -z "$ADMIN_PW" ]; then echo >&2 echo " A generated password is 20 random characters. It is printed once, in" >&2 echo " the summary at the end of this run — copy it then. (It is also left" >&2 echo " in $DEPLOY_DIR/.env, readable by root only.)" >&2 echo " Recommended: yes, then change it in the dashboard if you want" >&2 echo " something memorable." >&2 if ask_yn " Generate a secure admin password automatically?" y; then ADMIN_PW="$(gen_password)" ADMIN_PW_GENERATED=true else _tries=0 _pw1="" _pw2="" while [ "$_tries" -lt 3 ]; do _tries=$((_tries + 1)) read_secret _pw1 " Admin password (at least $ADMIN_PASSWORD_MIN characters)" read_secret _pw2 " Repeat password" if [ "$_pw1" != "$_pw2" ]; then warn "The passwords do not match." elif ! valid_admin_password "$_pw1"; then warn "$ADMIN_PASSWORD_MIN-72 characters of letters, digits or ! % * + , - . : ; = ? @ ^ _ ~" else ADMIN_PW="$_pw1" break fi done unset _pw1 _pw2 _tries if [ -z "$ADMIN_PW" ]; then ADMIN_PW="$(gen_password)" ADMIN_PW_GENERATED=true warn "Using a generated password instead; it is printed at the end of this run." fi fi fi fi # Unattended, or an upgrade whose .env predates these keys. [ -n "$ADMIN_USER" ] || ADMIN_USER="admin" if [ -z "$ADMIN_PW" ]; then ADMIN_PW="$(gen_password)" # Only announce it on a fresh install. On an upgrade the admin row already # exists, so this value is never applied — an operator who blanked the # password in .env after logging in (which the compose file suggests) must not # be handed a banner naming a password that does not work. [ "$IS_UPGRADE" = true ] || ADMIN_PW_GENERATED=true fi # Never prompted, but just as destructive to reset: an operator running on # non-standard ports had them silently rewritten to 80/443 by every upgrade. resolve HTTP_PORT 80 resolve HTTPS_PORT 443 resolve DEVGRAIL_SERVER_IMAGE devgrail-server # --- DNS preflight ---------------------------------------------------------- # "DNS is the only manual step" (spec §9) — and it was the only unverified one. # The installer prompted for the domain, brought the stack up, and printed the # required A records *after* it had finished. When they were not in place, # Traefik's ACME TLS-ALPN-01 challenge failed and the customer got a browser TLS # error with nothing connecting it to the cause. Let's Encrypt's failure rate # limits then make retrying worse rather than better. # # So the names are resolved and compared against this host's addresses here, # before Docker is installed and long before the stack is started. This is a # warning, not a verdict: split-horizon DNS, a pre-cutover install and a record # that has not propagated yet all look identical from inside the host. It asks # before continuing when there is a terminal; --skip-dns-check turns it off. # host_ips <name> — the IPv4 addresses <name> resolves to, one per line, empty # when it does not resolve. Four resolvers because no single one is present # everywhere: getent is glibc (absent on busybox), dig and host come from # bind-utils which most VPS images do not install, nslookup is busybox's. host_ips() { local name="$1" out="" if command -v getent >/dev/null 2>&1; then out="$(getent ahostsv4 "$name" 2>/dev/null | awk '{print $1}')" || out="" fi if [ -z "$out" ] && command -v dig >/dev/null 2>&1; then out="$(dig +short +time=3 +tries=1 A "$name" 2>/dev/null)" || out="" fi if [ -z "$out" ] && command -v host >/dev/null 2>&1; then out="$(host -W 3 -t A "$name" 2>/dev/null | sed -n 's/.*has address //p')" || out="" fi if [ -z "$out" ] && command -v nslookup >/dev/null 2>&1; then out="$(nslookup "$name" 2>/dev/null | sed -n '/^Name:/,$p' | sed -n 's/^Address: *//p')" || out="" fi # Whatever the tool printed, reduced to one de-duplicated IPv4 per line. The # grep also drops the resolver's own address, which nslookup prints first. # shellcheck disable=SC2086 # deliberate word splitting: one address per line printf '%s\n' $out | grep -E '^([0-9]{1,3}\.){3}[0-9]{1,3}$' | sort -u || true } have_resolver() { command -v getent >/dev/null 2>&1 || command -v dig >/dev/null 2>&1 || command -v host >/dev/null 2>&1 || command -v nslookup >/dev/null 2>&1 } # this_host_ips — addresses that legitimately point at this machine. Both halves # matter: a marketplace VPS carries its public address on an interface, but a # NAT'd or floating-IP host does not, and there an echo service is the only way # to learn what the world sees. Either matching is enough. this_host_ips() { local locals="" public="" url if command -v ip >/dev/null 2>&1; then locals="$(ip -4 -o addr show scope global 2>/dev/null | awk '{print $4}' | cut -d/ -f1)" || locals="" fi if [ -z "$locals" ] && command -v hostname >/dev/null 2>&1; then locals="$(hostname -I 2>/dev/null)" || locals="" fi for url in https://api.ipify.org https://checkip.amazonaws.com https://ifconfig.me/ip; do public="$(curl -fsS --max-time 5 "$url" 2>/dev/null | tr -cd '0-9.')" || public="" [ -n "$public" ] && break done # shellcheck disable=SC2086 # deliberate word splitting: one address per line printf '%s\n' $locals $public | grep -E '^([0-9]{1,3}\.){3}[0-9]{1,3}$' | sort -u || true } # dns_problem <name> <this-host's-ips> <what> — one problem line, or nothing when # <name> resolves to an address this host answers on. dns_problem() { local name="$1" mine="$2" what="$3" got ip got="$(host_ips "$name")" if [ -z "$got" ]; then printf ' %s does not resolve — no %s found.\n' "$name" "$what" return 0 fi for ip in $got; do case " $(printf '%s' "$mine" | tr '\n' ' ') " in *" $ip "*) return 0 ;; esac done printf ' %s resolves to %s, which is not this host.\n' \ "$name" "$(printf '%s' "$got" | tr '\n' ' ')" } dns_preflight() { if [ "$SKIP_DNS_CHECK" = true ]; then log "Skipping the DNS preflight (--skip-dns-check)." return 0 fi if ! have_resolver; then warn "no DNS lookup tool here (getent/dig/host/nslookup) — skipping the DNS preflight." return 0 fi section "DNS" local mine problems probe attempt=0 answer mine="$(this_host_ips)" if [ -z "$mine" ]; then warn "could not determine this host's IP address — skipping the DNS preflight." return 0 fi # The whole check loops, because the natural response to it failing is to go # and add the record. It used to be a one-shot "Continue anyway? [y/N]": the # only way back from a typo'd record was to abort and re-run the installer, # which re-asks every question — so the cheap answer was to press on with DNS # that does not work and meet it later as a browser TLS error. while :; do attempt=$((attempt + 1)) if [ "$attempt" -eq 1 ]; then log "Checking DNS for $DEVGRAIL_DOMAIN and *.$DEVGRAIL_BASE_DOMAIN..." else log "Re-checking DNS (attempt $attempt)..." fi # A random label under the base domain can only resolve through the wildcard # record. Checking the base domain itself would pass on a bare A record and # then fail for every workspace subdomain, which is the case that matters. # The label carries the attempt number so a resolver that negatively cached # the previous one cannot keep a record that now exists looking missing. probe="devgrail-dns-probe-$$-$attempt.$DEVGRAIL_BASE_DOMAIN" problems="$(dns_problem "$DEVGRAIL_DOMAIN" "$mine" "A record")" problems="$problems$(dns_problem "$probe" "$mine" "wildcard A record for *.$DEVGRAIL_BASE_DOMAIN")" if [ -z "$problems" ]; then log "DNS resolves to this host." return 0 fi warn "DNS does not point at this host yet." printf '%s' "$problems" >&2 echo " This host answers on: $(printf '%s' "$mine" | tr '\n' ' ')" >&2 echo >&2 echo " Required records:" >&2 dns_records " " >&2 if [ "$TLS_ENABLED" = true ]; then echo >&2 echo " Continuing now means Let's Encrypt will attempt its TLS-ALPN-01" >&2 echo " challenge against a name that does not reach this host. It will fail," >&2 echo " and repeated failures hit rate limits that make retrying slower rather" >&2 echo " than faster — so it is usually cheaper to fix DNS first." >&2 echo >&2 echo " The install itself does not need DNS: continuing finishes the whole" >&2 echo " run, and the only thing missing is a trusted certificate. Traefik" >&2 echo " serves its own placeholder — which is what an 'untrusted site'" >&2 echo " warning on a fresh install means — and orders the real one once the" >&2 echo " records resolve. Nothing has to be re-installed." >&2 fi echo >&2 echo " Legitimate reasons to continue anyway: split-horizon DNS, a pre-cutover" >&2 echo " install, or a record that has not propagated yet. --skip-dns-check" >&2 echo " suppresses this check entirely." >&2 echo >&2 # Unattended behaviour is unchanged, and deliberately so: an install driven # by cloud-init frequently runs before the records exist, and refusing would # break that legitimate case. Both paths say so loudly and proceed; neither # can reach the prompt below, so neither can loop. if [ "$ASSUME_YES" = true ]; then warn "--yes given — continuing with unverified DNS." return 0 fi if ! have_tty; then warn "no terminal to ask — continuing with unverified DNS." return 0 fi echo " Add the records now in another window, then re-check — nothing on this" >&2 echo " host has been changed yet, and re-checking costs nothing." >&2 # Aborting is what 'a' means, not what an unrecognised key means: this is # asked with the records still missing, which is exactly when a mistyped # answer is most expensive. The inner loop re-asks it without reprinting the # page above; 'r' leaves it, to re-check. while :; do printf '[R]e-check, [c]ontinue anyway, or [a]bort? [R/c/a]: ' > /dev/tty read -r answer < /dev/tty || answer="" case "${answer:-r}" in r|R|re|retry|RETRY) echo >&2; continue 2 ;; c|C|y|Y|yes|YES|continue) log "Continuing with unverified DNS." return 0 ;; a|A|abort|ABORT|q|Q|quit|QUIT) die "aborted before anything on this host was changed. Fix the records and re-run, or re-run with --skip-dns-check." ;; *) printf ' Please answer r, c or a.\n' > /dev/tty ;; esac done done } dns_preflight # --- release verification policy -------------------------------------------- # Whether a release that cannot be verified may be installed anyway. It may not, # by default. Every artifact is also checked against a digest, but that digest # comes from the same origin as the bytes, so it detects a corrupted download and # nothing else — the signature is the only thing between a compromised registry # and root on this host. Warning and continuing left the sole authenticity check # switched off unless someone had already thought to switch it on, which is # exactly backwards for a check whose whole purpose is the case nobody expected. # # DEVGRAIL_ALLOW_UNSIGNED=1 is the way out, and deliberately not a flag: opting # out of the only authenticity check this install has should look like a decision # recorded in someone's provisioning script, not a keystroke at a prompt. It is # for a white-label operator running an unsigned registry of their own, and for # recovering a host onto a release published before signing existed. # # There is no cost here to hosts that predate signing: an installer old enough to # carry no key is a copy already on disk, running its own old rules. This one # refuses. # # DEVGRAIL_REQUIRE_SIGNATURE is what this used to be called when the default was # the other way round. =1 now asks for what already happens; =0 still means what # it always meant, so a provisioning script carrying it does not silently get the # opposite of what it asked for. ALLOW_UNSIGNED=0 case "${DEVGRAIL_ALLOW_UNSIGNED:-0}" in 1|yes|true) ALLOW_UNSIGNED=1 ;; esac if [ "${DEVGRAIL_REQUIRE_SIGNATURE:-}" = 0 ]; then ALLOW_UNSIGNED=1; fi # --- tooling: the fast half ------------------------------------------------ # jq, openssl and the coreutils go in here, ahead of the remaining questions; # Docker itself goes in further down, after the last of them. The split is not # cosmetic: the address-pool question below asks jq to read this daemon's # configured pools, and a jq that is not installed yet answers "no pools # configured" — which is the one answer that makes the installer offer to write # pools over an operator's deliberate ones. These three take seconds. Docker # takes minutes, and nothing that is asked depends on it: a host with no Docker # has no Docker state, which is exactly what the probes below need to know. # # A dry run must not install anything, and it cannot read the registry without # jq — so on a host missing either, the honest report is that they would be # installed, and it stops there. if [ "$DRY_RUN" = true ]; then MISSING_TOOLS="" command -v docker >/dev/null 2>&1 || MISSING_TOOLS="Docker" command -v jq >/dev/null 2>&1 || MISSING_TOOLS="${MISSING_TOOLS:+$MISSING_TOOLS and }jq" if [ -n "$MISSING_TOOLS" ]; then echo echo "============================================================" echo " Dry run — nothing on this host has been changed." echo echo " $MISSING_TOOLS would be installed first; this host does not have it yet." echo " Nothing further can be reported without it: the release manifest," echo " the version diff and the disk figures all come from the registry." echo "============================================================" exit 0 fi fi if ! command -v jq >/dev/null 2>&1; then log "Installing jq..." if command -v apt-get >/dev/null 2>&1; then apt-get update -qq && apt-get install -y -qq jq elif command -v dnf >/dev/null 2>&1; then dnf install -y jq elif command -v yum >/dev/null 2>&1; then yum install -y jq elif command -v apk >/dev/null 2>&1; then apk add --no-cache jq else die "please install 'jq' and re-run"; fi fi # openssl verifies the release signature (and generates secrets when present). # The install attempt is best-effort, but the outcome is not: verification is # required, and a host that cannot check a signature cannot be told the bytes it # is about to run as root are genuine. Refusing here rather than at the # signature check saves the operator the remaining questions. if ! command -v openssl >/dev/null 2>&1; then log "Installing openssl..." if command -v apt-get >/dev/null 2>&1; then apt-get install -y -qq openssl || true elif command -v dnf >/dev/null 2>&1; then dnf install -y openssl || true elif command -v yum >/dev/null 2>&1; then yum install -y openssl || true elif command -v apk >/dev/null 2>&1; then apk add --no-cache openssl || true fi fi if ! command -v openssl >/dev/null 2>&1 && [ "$ALLOW_UNSIGNED" != 1 ]; then die "'openssl' is required to verify the release signature, and it could not be installed here. Install it and re-run, or set DEVGRAIL_ALLOW_UNSIGNED=1 to install without verifying anything (secrets then come from /dev/urandom). Nothing has been changed." fi # tar builds the pre-upgrade snapshot archive; df/awk size the free-space check. for tool in tar df awk; do command -v "$tool" >/dev/null 2>&1 || die "'$tool' is required (used to take the pre-upgrade backup)" done # --- Docker daemon preflight (userns-remap + default address pools) -------- # # Two daemon-level settings materially change how DevGrail runs, and neither can # be expressed in a container spec: both live in /etc/docker/daemon.json and take # effect only when dockerd restarts. The installer is the one moment an operator # is reliably at a terminal, as root, on a host where these are still cheap to # change — so this section detects, explains, offers and applies, rather than # printing a wall of manual steps and hoping. # # Nothing here is applied without an explicit yes, and --yes is deliberately not # that yes: restarting dockerd stops every container on the host, and enabling # remap moves the daemon's data root. Those are decisions, not defaults. # DEVGRAIL_USERNS_REMAP / DEVGRAIL_ADDRESS_POOLS (and the matching flags) are how # an unattended run — a marketplace image build, cloud-init — opts in on purpose. DAEMON_JSON="${DEVGRAIL_DAEMON_JSON:-/etc/docker/daemon.json}" # Restart dockerd across the init systems this installer supports: systemd on the # Debian/RHEL images, OpenRC on Alpine. Non-zero means no known mechanism worked, # which callers treat as "put the config back". restart_docker() { if command -v systemctl >/dev/null 2>&1; then systemctl restart docker >/dev/null 2>&1 && return 0 fi if command -v rc-service >/dev/null 2>&1; then rc-service docker restart >/dev/null 2>&1 && return 0 fi if command -v service >/dev/null 2>&1; then service docker restart >/dev/null 2>&1 && return 0 fi return 1 } # Wait for the daemon to answer again. dockerd is slow to come back when it has # images to re-index, and the first start under userns-remap is slower still — it # lays down a fresh data root — so this window is generous on purpose. wait_for_docker() { local i=0 while [ "$i" -lt 90 ]; do docker info >/dev/null 2>&1 && return 0 sleep 1 i=$((i + 1)) done return 1 } # Merge a jq expression into $DAEMON_JSON and restart the daemon. # $1 human label for the change, used in messages # $2 jq expression applied to the existing object (or {} when there is none) # # The prior file is kept as <file>.devgrail-bak-<timestamp>. If dockerd does not # come back, the backup is restored and the daemon restarted again: leaving a # host without Docker is a worse outcome than either setting is worth. A # daemon.json that does not parse is never touched — that is an operator edit in # progress, and clobbering it would destroy work we cannot see. apply_daemon_json() { local label="$1" filter="$2" current='{}' backup="" merged stamp if [ -s "$DAEMON_JSON" ]; then if ! jq -e . "$DAEMON_JSON" >/dev/null 2>&1; then warn "$DAEMON_JSON exists but is not valid JSON — leaving it alone." echo " Fix the file (or move it aside) and re-run to apply $label." >&2 return 1 fi current="$(cat "$DAEMON_JSON")" stamp="$(date +%Y%m%d%H%M%S)" backup="$DAEMON_JSON.devgrail-bak-$stamp" cp -p "$DAEMON_JSON" "$backup" || { warn "could not back up $DAEMON_JSON — not changing it." return 1 } fi if ! merged="$(printf '%s' "$current" | jq "$filter" 2>/dev/null)" || [ -z "$merged" ]; then warn "could not compute the new $DAEMON_JSON — leaving it alone." [ -n "$backup" ] && rm -f "$backup" return 1 fi mkdir -p "$(dirname "$DAEMON_JSON")" if ! printf '%s\n' "$merged" > "$DAEMON_JSON.devgrail-new"; then warn "could not write $DAEMON_JSON — leaving it alone." [ -n "$backup" ] && rm -f "$backup" return 1 fi chmod 0644 "$DAEMON_JSON.devgrail-new" mv -f "$DAEMON_JSON.devgrail-new" "$DAEMON_JSON" log "Restarting Docker to apply $label (this stops running containers)..." if restart_docker && wait_for_docker; then log "Docker restarted; $label is in effect." [ -n "$backup" ] && log "Previous daemon config kept at $backup" return 0 fi warn "Docker did not come back after applying $label — rolling the change back." if [ -n "$backup" ]; then mv -f "$backup" "$DAEMON_JSON"; else rm -f "$DAEMON_JSON"; fi if restart_docker && wait_for_docker; then warn "Docker is back on its previous configuration; $label was NOT applied." return 1 fi die "Docker is not responding and restoring the previous configuration did not revive it. $DAEMON_JSON holds what it held before this run. Investigate with: journalctl -u docker --no-pager -n 50 Nothing else on this host has been changed." } # True when the daemon reports userns-remap as active. userns_remap_enabled() { docker info --format '{{.SecurityOptions}}' 2>/dev/null | grep -q 'name=userns' } # True when this host already has Docker state that enabling userns-remap would # orphan. Remap moves the daemon to a per-UID data root, so every image, # container and volume created before the switch becomes invisible — including, # on an upgrade, DevGrail's own data volume. This check is what keeps the offer # limited to "fresh host, nothing to lose". docker_has_state() { local what n for what in "ps -aq" "volume ls -q" "image ls -q"; do # shellcheck disable=SC2086 # grep -c '.' rather than wc -l: it counts a final line that has no trailing # newline, and skips blank ones. wc -l would report 0 for a one-line answer # that ended without a newline — i.e. read "this host is empty" off a host # that has exactly one image. n="$({ docker $what 2>/dev/null || true; } | grep -c '.' || true)" [ "${n:-0}" -gt 0 ] && return 0 done return 1 } # How many networks the daemon's configured pools can hand out. Docker carves # each pool into /<size> blocks, so a pool contributes 2^(size - mask) networks. # Zero means no pools are configured, i.e. the stock defaults, which top out # near 31 — the number this whole section exists to raise. address_pool_capacity() { local pools cap pools="$(docker info --format '{{json .DefaultAddressPools}}' 2>/dev/null || true)" case "$pools" in ''|null) echo 0; return 0 ;; esac cap="$(printf '%s' "$pools" | jq -r ' [ .[]? | (.Size - (.Base | split("/")[1] | tonumber)) | select(. >= 0) | pow(2; .) ] | add // 0 | floor' 2>/dev/null || true)" case "$cap" in ''|null) echo 0 ;; *) echo "$cap" ;; esac } # True when no route on this host already lives inside the candidate range. # POSIX sh has no CIDR arithmetic and the candidates are fixed, so each carries a # regex matching an address inside it. Routes on docker0 / br-* / veth are # excluded: Docker's own bridges already sit in 172.17-172.31 and would otherwise # veto the very range we want to hand it. pool_base_free() { local re="$1" routes command -v ip >/dev/null 2>&1 || return 0 routes="$({ ip -4 route show 2>/dev/null || true; } \ | grep -Ev 'dev (docker[0-9]*|br-[0-9a-f]+|veth[0-9a-z]*)' || true)" if printf '%s\n' "$routes" | grep -Eq "$re"; then return 1; fi return 0 } running_container_count() { # See docker_has_state for why this counts with grep rather than wc. { docker ps -q 2>/dev/null || true; } | grep -c '.' || true } # --- userns-remap ---------------------------------------------------------- # The decision is taken here, among the other questions. Acting on it — which # rewrites daemon.json and restarts dockerd — is apply_daemon_decisions(), # called once the plan has been shown and confirmed. Keeping the two apart is # what lets every question in this installer be asked before any of them is # acted on, and it is why this block only ever sets USERNS_DECISION. # # The reasoning below is worth reading once and never again, so it is offered # behind `?` rather than printed at a prompt nobody has patience for. explain_userns() { cat >&2 <<EXPLAIN Why userns-remap matters ---------------------------------------------------------- Workspaces run AI agents and whatever code those agents write. Inside the container the agent is the unprivileged 'dev' user, but it has passwordless sudo — so it can become container root at will, and without userns-remap container root IS host root. A bug in the kernel or the container runtime would then be a full host takeover. userns-remap maps container root onto an unprivileged host UID, so the same escape lands as a nobody. DevGrail's own infra containers (traefik, the two docker socket proxies, devgrail-server) opt out via userns_mode: host, so only workspaces are remapped. What it costs: one line in $DAEMON_JSON and a Docker restart. The restart moves the daemon's data root, which is why it is only offered on a host with no images, containers or volumes to orphan — after that, migrating is a deliberate, backed-up operation. EXPLAIN } # Printed whenever the answer ends up "no", by any route. It names the risk in # one line: an unattended run never sees the prompt, and "userns-remap was # skipped" means nothing to an operator reading the log three weeks later. userns_declined_note() { warn "Continuing without userns-remap — a workspace escape would be host root." echo " To enable it later (one-time, needs root):" >&2 echo " 1. Add to $DAEMON_JSON: { \"userns-remap\": \"default\" }" >&2 echo " 2. systemctl restart docker" >&2 echo " Do it before workspaces exist — afterwards the data-root change" >&2 echo " orphans them. Set DEVGRAIL_USERNS_REMAP=no to stop being asked." >&2 echo >&2 } USERNS_DECISION=skip if userns_remap_enabled; then USERNS_DECISION=active log "Docker userns-remap is enabled — workspace root maps to an unprivileged host UID." elif [ "$USERNS_REMAP_ANSWER" = no ]; then : # the operator has decided; do not re-ask on every upgrade elif docker_has_state; then # Not offered, and not a warning the operator can act on in place: switching # now would hide everything already on this host behind a new data root. warn "userns-remap is NOT enabled, and cannot be turned on safely from here." echo " This host already has Docker images, containers or volumes. Enabling" >&2 echo " remap moves the daemon to a new data root, so all of them — including" >&2 echo " DevGrail's data volume — would become invisible. Migrating that is a" >&2 echo " deliberate, backed-up operation, not something an installer should do" >&2 echo " behind a prompt." >&2 echo >&2 echo " To do it by hand, with the stack down and a backup taken:" >&2 echo " 1. devgrail backup" >&2 echo " 2. Add to $DAEMON_JSON: { \"userns-remap\": \"default\" }" >&2 echo " 3. systemctl restart docker" >&2 echo " 4. Re-run this installer, then devgrail restore <snapshot>" >&2 echo " Continuing without remap..." >&2 echo >&2 elif [ "$DRY_RUN" = true ]; then echo " Dry run: this host has no Docker state yet, so a real run would offer" >&2 echo " to enable userns-remap and restart Docker. Nothing has been changed." >&2 echo >&2 elif [ "$USERNS_REMAP_ANSWER" = yes ]; then USERNS_DECISION=enable log "DEVGRAIL_USERNS_REMAP=yes — enabling userns-remap." elif have_tty && [ "$ASSUME_YES" != true ]; then section "Harden Docker" echo " Map workspace root to an unprivileged host UID." >&2 echo " Workspaces run AI agents and whatever code those agents write, and the" >&2 echo " agent can become container root at will. Without this, container root" >&2 echo " IS host root, so a container escape owns the whole VPS." >&2 echo " Applying it rewrites $DAEMON_JSON and restarts Docker." >&2 echo >&2 echo " Recommended: YES. This host has no Docker images, containers or" >&2 echo " volumes yet, which is the only moment it is free — afterwards the" >&2 echo " data-root change would orphan everything, and this installer will" >&2 echo " refuse to offer it. Say no only if you knowingly need unmapped" >&2 echo " container UIDs on this host (host-root-owned bind mounts, say)." >&2 echo " Answer ? for the full reasoning." >&2 if ask_yn " Enable userns-remap?" y explain_userns; then USERNS_DECISION=enable else userns_declined_note fi else # Unattended with no answer given. --yes is deliberately not consent here: # this restarts dockerd and moves its data root, which is a decision rather # than a default. DEVGRAIL_USERNS_REMAP=yes is how an unattended run opts in. userns_declined_note fi # --- default address pools ------------------------------------------------- # Each workspace gets its own bridge network so untrusted agent code cannot reach # a peer workspace directly. Docker's stock pools only yield ~31 networks in # total, so a busy host runs out and container creation starts failing with # "could not find an available, non-overlapping IPv4 address pool". # # Same decide-now / apply-later split as userns-remap above. explain_pools() { cat >&2 <<EXPLAIN Why the address pools matter ---------------------------------------------------------- Every DevGrail workspace gets its own Docker network so one workspace cannot reach another. Docker's stock address pools yield only ~31 networks in total, and past that container creation fails with 'could not find an available, non-overlapping IPv4 address pool' — an error that shows up when you add a workspace, not now. Widening to $POOL_BASE (a /24 per network) raises the ceiling to ~$POOL_NEW_CAP. Networks that already exist keep their current subnets; this only affects networks created from now on. What it costs: one line in $DAEMON_JSON and a Docker restart. The range offered is checked against this host's routing table first, so it cannot shadow a route you already use. EXPLAIN } # The restart is the whole cost, and what it stops depends on what is running. # Printed on every path that leads to actually applying the change — including # the env-var one, so an unattended run still records what it stopped. pools_cost_note() { if [ "${RUNNING:-0}" -gt 0 ]; then echo " Applying it restarts the Docker daemon, stopping the $RUNNING container(s)" >&2 echo " running here. The DevGrail stack comes back on its own (restart:" >&2 echo " unless-stopped). Each workspace comes back or not according to" >&2 echo " its own restart policy — the default keeps whatever state it was" >&2 echo " in, and any workspace set to \"Never start\" you start again from" >&2 echo " the dashboard." >&2 else echo " Applying it restarts the Docker daemon. Nothing is running on this host" >&2 echo " yet, so that costs nothing right now." >&2 fi } pools_declined_note() { warn "Continuing with the default address pools — workspace ceiling stays at ~27." echo " To raise it later (one-time, needs root):" >&2 echo " 1. Add to $DAEMON_JSON:" >&2 echo " { \"default-address-pools\": [{\"base\": \"$POOL_BASE\", \"size\": 24}] }" >&2 echo " 2. systemctl restart docker" >&2 echo " Safe to defer until you approach the limit. Set" >&2 echo " DEVGRAIL_ADDRESS_POOLS=no to stop being asked." >&2 echo >&2 } POOL_CAP="$(address_pool_capacity)" POOLS_DECISION=skip POOL_BASE=""; POOL_NEW_CAP=0; RUNNING=0 if [ "${POOL_CAP:-0}" -ge 256 ]; then log "Docker address pools allow ~$POOL_CAP networks — no workspace ceiling to worry about." elif [ "$ADDRESS_POOLS_ANSWER" = no ]; then : # already decided elif [ "${POOL_CAP:-0}" -gt 0 ]; then # Pools are configured, just narrow. That is an operator decision about this # host's addressing, and it is not one to overwrite from behind a prompt. warn "Docker address pools are configured but allow only ~$POOL_CAP networks." echo " They were set deliberately, so this installer will not rewrite them." >&2 echo " To widen them, edit default-address-pools in $DAEMON_JSON and" >&2 echo " restart Docker. Continuing..." >&2 echo >&2 else if pool_base_free '(^|[[:space:]])172\.(1[6-9]|2[0-9]|3[01])\.'; then POOL_BASE=172.16.0.0/12; POOL_NEW_CAP=4096 elif pool_base_free '(^|[[:space:]])10\.201\.'; then POOL_BASE=10.201.0.0/16; POOL_NEW_CAP=256 fi RUNNING="$(running_container_count)" if [ -z "$POOL_BASE" ]; then warn "No safe range to offer — this host already routes both 172.16/12 and 10.201/16." echo " Handing Docker a pool that shadows one of your own routes is worse" >&2 echo " than the ceiling it would lift. Pick a free range yourself, add it" >&2 echo " as default-address-pools in $DAEMON_JSON, and restart Docker." >&2 echo " Continuing with the defaults..." >&2 echo >&2 elif [ "$DRY_RUN" = true ]; then echo " Dry run: a real run would offer to widen the pools to $POOL_BASE" >&2 echo " (a /24 per network, ~$POOL_NEW_CAP workspaces) and restart Docker." >&2 echo " Nothing has been changed." >&2 echo >&2 elif [ "$ADDRESS_POOLS_ANSWER" = yes ]; then POOLS_DECISION=widen log "DEVGRAIL_ADDRESS_POOLS=yes — widening the default address pools to $POOL_BASE." pools_cost_note elif have_tty && [ "$ASSUME_YES" != true ]; then section "Workspace ceiling" echo " Raise the ceiling on concurrent workspaces, currently about 27." >&2 echo " Each workspace gets its own Docker network so one workspace cannot" >&2 echo " reach another, and the stock pools run out at ~31 networks. Past that," >&2 echo " creating a workspace just fails. Widening to $POOL_BASE — checked" >&2 echo " free against this host's routes — raises the ceiling to ~$POOL_NEW_CAP." >&2 echo " Networks that already exist keep the subnets they have." >&2 pools_cost_note POOL_DEFAULT=y [ "${RUNNING:-0}" -gt 0 ] && POOL_DEFAULT=n echo >&2 if [ "${RUNNING:-0}" -gt 0 ]; then echo " Recommended: defer unless you are heading for more than ~27" >&2 echo " workspaces. Unlike userns-remap this can be applied at any later" >&2 echo " time with the same effect, and right now it costs a restart of" >&2 echo " the $RUNNING running container(s). Re-run this installer when you" >&2 echo " want it, or make the change by hand." >&2 else echo " Recommended: YES. Nothing is running on this host yet, so it is" >&2 echo " free right now, and the alternative is discovering the limit as a" >&2 echo " failed workspace creation months from now. Say no if you expect" >&2 echo " only a handful of workspaces, or if you know $POOL_BASE is used" >&2 echo " by a network this VPS reaches over a VPN — the check above only sees" >&2 echo " routes that exist right now." >&2 fi echo " Answer ? for the full reasoning." >&2 if ask_yn " Widen the address pools?" "$POOL_DEFAULT" explain_pools; then POOLS_DECISION=widen else pools_declined_note fi else # Unattended with no answer. As above, --yes is not consent for a daemon # restart; DEVGRAIL_ADDRESS_POOLS=yes is. pools_declined_note fi fi # --- apply the daemon decisions --------------------------------------------- # Called from the main flow after the plan summary, and before the first image is # downloaded. That order is not incidental: enabling userns-remap moves the # daemon's data root, so an image loaded before the restart would be invisible # after it. apply_daemon_decisions() { if [ "$USERNS_DECISION" = enable ]; then # dockerd creates the "dockremap" user and its subordinate ranges itself, # but only if these files exist for it to append to. Creating them empty is # a no-op where they already work, and removes a known first-start failure # on minimal images. [ -f /etc/subuid ] || : > /etc/subuid [ -f /etc/subgid ] || : > /etc/subgid if apply_daemon_json "userns-remap" '. + {"userns-remap": "default"}'; then if userns_remap_enabled; then log "userns-remap is active — workspace root is now an unprivileged host UID." else warn "Docker restarted but does not report userns-remap as active." echo " Check 'docker info' and journalctl -u docker. Continuing." >&2 fi fi fi if [ "$POOLS_DECISION" = widen ]; then apply_daemon_json "the wider address pools" \ ". + {\"default-address-pools\": [{\"base\": \"$POOL_BASE\", \"size\": 24}]}" || true fi return 0 } # --- inotify limit ----------------------------------------------------------- # Extracted by test/installer/inotify_limit_test.sh between this marker and the # matching "end inotify limit" one. Keep both comments intact, and keep # everything between them free of top-level side effects. # # fs.inotify.max_user_instances caps how many inotify instances one user may # hold, and Ubuntu ships it at 128. Without userns-remap a workspace's processes # are host root, so every file watcher in every workspace — dev servers, editors, # language servers — draws on the same budget as systemd. When it runs out, # devgrail-upgrade.path fails with "Too many open files" and stays failed (a path # unit has no Restart=), and the dashboard's Upgrade button waits forever for a # host that will never pick the request up. Raised, never lowered: a host whose # operator already set it higher keeps their value. # # The file records the value it replaced, so uninstall.sh --nuke can put it back. INOTIFY_KEY=fs.inotify.max_user_instances INOTIFY_PROC="${DEVGRAIL_INOTIFY_PROC:-/proc/sys/fs/inotify/max_user_instances}" INOTIFY_CONF="${DEVGRAIL_INOTIFY_CONF:-/etc/sysctl.d/60-devgrail-inotify.conf}" INOTIFY_TARGET=1024 # raise | keep (already at or above the target) | readonly (/proc/sys cannot be # written, as inside a container) | unknown (the value could not be read). INOTIFY_DECISION=keep INOTIFY_NOW="" # decide_inotify — sets INOTIFY_NOW and INOTIFY_DECISION. Reads only. decide_inotify() { INOTIFY_NOW="$(cat "$INOTIFY_PROC" 2>/dev/null || true)" case "$INOTIFY_NOW" in ''|*[!0-9]*) INOTIFY_DECISION=unknown ;; *) if [ "$INOTIFY_NOW" -ge "$INOTIFY_TARGET" ]; then INOTIFY_DECISION=keep elif [ -w "$INOTIFY_PROC" ]; then INOTIFY_DECISION=raise else INOTIFY_DECISION="readonly"; fi ;; esac return 0 } # inotify_recorded_previous [file] — the value an earlier run replaced, or empty. inotify_recorded_previous() { sed -n '/^# devgrail-previous: [0-9][0-9]*$/{s/^# devgrail-previous: //p;q;}' \ "${1:-$INOTIFY_CONF}" 2>/dev/null || true } # apply_inotify — raise the running value, then persist it for the next boot. # The running value first: a file that promises 1024 on a kernel still at 128 # would leave the host broken until a reboot nobody knows to do. Never fatal — # a missed limit is worth a warning, not an aborted install. apply_inotify() { local prev case "$INOTIFY_DECISION" in raise) ;; readonly) log "Cannot raise $INOTIFY_KEY here ($INOTIFY_PROC is read-only) — leaving it at $INOTIFY_NOW." return 0 ;; *) return 0 ;; esac # A re-raise (something lowered it after an earlier run) must not overwrite # the value that run recorded with one it set itself. prev="$(inotify_recorded_previous)" [ -n "$prev" ] || prev="$INOTIFY_NOW" if ! { printf '%s\n' "$INOTIFY_TARGET" > "$INOTIFY_PROC"; } 2>/dev/null; then warn "could not raise $INOTIFY_KEY from $INOTIFY_NOW to $INOTIFY_TARGET — continuing." return 0 fi if mkdir -p "$(dirname "$INOTIFY_CONF")" && cat > "$INOTIFY_CONF" <<EOF # Written by DevGrail's install.sh. Workspace file watchers share root's inotify # budget with systemd; at the default of 128 the dashboard upgrade watcher # (devgrail-upgrade.path) is the first thing to fail. uninstall.sh --nuke # removes this file and restores the value below. # devgrail-previous: $prev $INOTIFY_KEY = $INOTIFY_TARGET EOF then log "Raised $INOTIFY_KEY from $INOTIFY_NOW to $INOTIFY_TARGET (kept across reboots by $INOTIFY_CONF)." else warn "raised $INOTIFY_KEY to $INOTIFY_TARGET, but could not write $INOTIFY_CONF — it reverts on reboot." fi return 0 } # --- end inotify limit ------------------------------------------------------- # --- AppArmor workspace profile ---------------------------------------------- # Extracted by test/installer/apparmor_profile_test.sh between this marker and # the matching "end AppArmor workspace profile" one. Keep both comments intact, # and keep everything between them free of top-level side effects. # # userns-remap gives every workspace the same host UID range. A process that # gets out of a workspace with its credentials intact (a leaked descriptor, as # in runc's CVE-2024-21626) can then read and write every other workspace's # layers and volumes under Docker's data root, and swap the stdio FIFOs of # their `docker exec`s in the remapped exec root, which the range owns. # AppArmor stays attached to a process that gets out that way, so workspaces # run under devgrail-workspace: Docker's docker-default plus deny rules over # those two roots. A path reached through a leaked descriptor resolves to its # host path, so the rules hold there too; the spike in # docs/plans/WORKSPACE_UID_ISOLATION_PLAN.md §5.2 checked both cases on a real # host. The rules deny the same paths inside a workspace as well. That only # matters to Docker run inside one, which docker-default does not allow either. # # The file lives in /etc/apparmor.d, so the profile is loaded at boot before # dockerd restarts the workspaces. Every run rewrites and reloads it, and a # reload replaces the profile under running workspaces too. The server learns # the name (DEVGRAIL_WORKSPACE_APPARMOR_PROFILE in .env) only once the kernel # holds the profile: a name the kernel does not hold fails every create. # # A Docker-enabled workspace runs its own dockerd, which docker-default's mount # deny and its write denies on /proc/sys and /sys stop. Those workspaces run # under devgrail-workspace-nested instead: the same profile with mounts and # the network sysctls allowed, and both DevGrail deny rules kept. The inner # daemon's data and exec roots are moved off the denied paths by the # workspace image, not here. docs/plans/NESTED_DEVGRAIL_PLAN.md §3.6 is the # spike that found each change, and §3.7 the escape probe run under it. APPARMOR_PROFILE=devgrail-workspace APPARMOR_NESTED_PROFILE=devgrail-workspace-nested APPARMOR_DIR="${DEVGRAIL_APPARMOR_DIR:-/etc/apparmor.d}" APPARMOR_ENABLED="${DEVGRAIL_APPARMOR_ENABLED:-/sys/module/apparmor/parameters/enabled}" APPARMOR_PARSER="${DEVGRAIL_APPARMOR_PARSER:-apparmor_parser}" # The inner daemon's storage driver is fuse-overlayfs, which needs this device. FUSE_DEVICE="${DEVGRAIL_FUSE_DEVICE:-/dev/fuse}" # load | unavailable (the kernel has no AppArmor, or has it off) | noparser # (AppArmor is on, but apparmor_parser is not installed). APPARMOR_DECISION=unavailable # What .env tells the server: the profile name once it is loaded, else empty. WORKSPACE_APPARMOR_PROFILE="" # The nested profile's name once it is loaded and the host can run # Docker-enabled workspaces safely, else empty. The server offers the mode only # when this is set, so it is the host's whole readiness signal. WORKSPACE_NESTED_APPARMOR_PROFILE="" # decide_apparmor — sets APPARMOR_DECISION. Reads only. decide_apparmor() { if [ "$(cat "$APPARMOR_ENABLED" 2>/dev/null || true)" != Y ]; then APPARMOR_DECISION=unavailable elif command -v "$APPARMOR_PARSER" >/dev/null 2>&1; then APPARMOR_DECISION=load else APPARMOR_DECISION=noparser fi return 0 } # apparmor_root_ok <path> — whether <path> may go in a deny rule. Plain # characters only, because AppArmor reads globs in a path. And never a # directory a workspace needs for itself, because the rule denies that path # inside the workspace as well. apparmor_root_ok() { case "$1" in /?*) ;; *) return 1 ;; esac case "$1" in *[!A-Za-z0-9._/-]*|*//*|*/|*/.|*/..|*/./*|*/../*) return 1 ;; esac case "$1" in /bin|/boot|/dev|/etc|/home|/lib|/lib64|/media|/mnt|/opt|/proc|/root|/run|/sbin|/srv|/sys|/tmp|/usr|/usr/*|/var|/var/cache|/var/lib|/var/log|/var/run|/var/tmp|/workspace) return 1 ;; esac return 0 } # apparmor_canonical <path> — <path> with symlinks resolved, as AppArmor # matches it, when it exists. /var/run is a symlink to /run on every systemd host. apparmor_canonical() { local dir="${1%/*}" base="${1##*/}" if [ -d "$1" ]; then (cd "$1" 2>/dev/null && pwd -P) && return 0; fi if [ -n "$dir" ] && [ -d "$dir" ]; then printf '%s/%s\n' "$(cd "$dir" 2>/dev/null && pwd -P)" "$base"; return 0 fi printf '%s\n' "$1" } # apparmor_data_root <DockerRootDir> — the daemon's data root. Under # userns-remap `docker info` reports the remapped root inside it # (/var/lib/docker/100000.100000); the rule covers all of it. apparmor_data_root() { local root="${1:-/var/lib/docker}" leaf leaf="${root##*/}" case "$leaf" in *[!0-9.]*|.*|*.) ;; *.*) root="${root%/*}" ;; esac apparmor_canonical "$root" } # apparmor_exec_root <daemon.json exec-root> — the daemon's exec root. The # remapped one sits beside it with the range appended (/run/docker.100000.100000). apparmor_exec_root() { apparmor_canonical "${1:-/var/run/docker}" } # render_apparmor_profile <data root> <exec root> [dockerd's AppArmor label] [nested] # The body is Docker's docker-default (github.com/moby/profiles/apparmor, # template.go), including the ABI pin that keeps `network,` covering unix # sockets, and the includes Docker adds only where the host has them. Keep it in # step with that template. With "nested" as the fourth argument it renders # devgrail-workspace-nested, which differs only where the comments below say. render_apparmor_profile() { local data="$1" exec_root="$2" daemon="${3:-unconfined}" variant="${4:-}" name="$APPARMOR_PROFILE" [ "$variant" = nested ] && name="$APPARMOR_NESTED_PROFILE" echo "# Written by DevGrail's install.sh, and rewritten by every run: edits here" echo "# are lost. Docker's docker-default plus the deny rules at the end." [ "$variant" = nested ] \ && echo "# For Docker-enabled workspaces: mounts and the network sysctls allowed." [ -f "$APPARMOR_DIR/abi/3.0" ] && echo 'abi <abi/3.0>,' if [ -f "$APPARMOR_DIR/tunables/global" ]; then echo '#include <tunables/global>' else echo '@{PROC}=/proc/' fi echo echo "profile $name flags=(attach_disconnected,mediate_deleted) {" [ -f "$APPARMOR_DIR/abstractions/base" ] && echo ' #include <abstractions/base>' cat <<'EOF' network, deny network ax25, deny network ipx, deny network appletalk, deny network netrom, deny network rose, deny network econet, deny network atmsvc, deny network irda, deny network wanpipe, deny network isdn, deny network caif, deny network alg, deny network vsock, capability, file, umount, signal (receive) peer=unconfined, signal (receive) peer=runc, signal (receive) peer=crun, EOF # docker kill comes from dockerd. Unconfined is already covered above. case "$daemon" in unconfined|'') ;; *[!A-Za-z0-9._/-]*) ;; *) echo " signal (receive) peer=$daemon," ;; esac cat <<EOF signal (send,receive) peer=$name, deny @{PROC}/* w, deny @{PROC}/{[^1-9/],[^1-9/][^0-9/],[^1-9s/][^0-9y/][^0-9s/],[^1-9/][^0-9/][^0-9/][^0-9/]*}/** w, EOF if [ "$variant" = nested ]; then # docker-default's "deny @{PROC}/sys/[^k]** w," minus /proc/sys/net, which # belongs to the workspace's own network namespace: dockerd and the inner # runc set sysctls there. cat <<'EOF' deny @{PROC}/sys/[^kn]** w, deny @{PROC}/sys/n[^e]** w, deny @{PROC}/sys/ne[^t]** w, deny @{PROC}/sys/net[^/]** w, EOF else echo ' deny @{PROC}/sys/[^k]** w,' fi cat <<'EOF' deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w, deny @{PROC}/sysrq-trigger rwklx, deny @{PROC}/kcore rwklx, EOF if [ "$variant" = nested ]; then # The inner dockerd mounts, and its containers pivot_root. The kernel's own # user-namespace rules still apply to every mount. cat <<'EOF' mount, remount, pivot_root, deny /sys/[^fn]*/** wklx, deny /sys/n[^e]*/** wklx, deny /sys/ne[^t]*/** wklx, deny /sys/net?*/** wklx, EOF # runc writes sysctls through a detached procfs, which AppArmor names # /sys/net/... Sysfs has no top-level net directory, so this opens nothing # of the host's. else cat <<'EOF' deny mount, deny /sys/[^f]*/** wklx, EOF fi cat <<EOF deny /sys/f[^s]*/** wklx, deny /sys/fs/[^c]*/** wklx, deny /sys/fs/c[^g]*/** wklx, deny /sys/fs/cg[^r]*/** wklx, deny /sys/firmware/** rwklx, deny /sys/devices/virtual/powercap/** rwklx, deny /sys/kernel/security/** rwklx, ptrace (trace,tracedby,read,readby) peer=$name, # DevGrail: every workspace's layers and volumes, and the control plane's # volumes, all in the one tree the shared UID range can traverse. deny $data{,/**} rwlkmx, # DevGrail: the exec roots, where the remapped one is owned by the range and # holds the stdio FIFOs of every workspace's docker exec. deny $exec_root{,.[0-9]*}{,/**} rwlkmx, } EOF } # load_apparmor_profile <name> <fallback> <data root> <exec root> <dockerd's label> [nested] # — write one profile, load it and keep it for the boot-time load. Succeeds # only once the kernel holds it and its file is in place; otherwise warns, # saying <fallback> is what the affected workspaces get instead. load_apparmor_profile() { local name="$1" fallback="$2" file tmp file="$APPARMOR_DIR/$name" # A dot file: if this run dies before the rename, the boot-time load of # $APPARMOR_DIR skips it. tmp="$APPARMOR_DIR/.$name.new" if ! { mkdir -p "$APPARMOR_DIR" && render_apparmor_profile "$3" "$4" "$5" "${6:-}" > "$tmp"; } 2>/dev/null; then rm -f "$tmp" warn "could not write $file — $fallback." return 1 fi if ! "$APPARMOR_PARSER" -r "$tmp" > "$tmp.err" 2>&1; then warn "could not load the AppArmor profile $name — $fallback." sed 's/^/ /' "$tmp.err" >&2 rm -f "$tmp" "$tmp.err" return 1 fi rm -f "$tmp.err" # Loaded but not persisted would be worse than not loaded: the workspaces # created with it would fail to start after the next reboot. if ! { chmod 644 "$tmp" && mv -f "$tmp" "$file"; }; then rm -f "$tmp" warn "loaded $name but could not write $file — $fallback." return 1 fi return 0 } # nested_docker_blocker <userns-remap active: yes|no> — why this host cannot # run Docker-enabled workspaces, or nothing when it can. Without userns-remap # a workspace's root is host root, and the relaxations the mode needs would # hand it the host. nested_docker_blocker() { if [ "$1" != yes ]; then echo "Docker does not remap user namespaces" elif [ ! -c "$FUSE_DEVICE" ]; then echo "the kernel offers no $FUSE_DEVICE" fi return 0 } # apply_apparmor <DockerRootDir> <daemon.json exec-root> [dockerd's label] [userns-remap active: yes|no] # — write both profiles and load them. WORKSPACE_APPARMOR_PROFILE is set once # the kernel holds devgrail-workspace, and WORKSPACE_NESTED_APPARMOR_PROFILE # once it holds the nested one too and the host passes nested_docker_blocker. # Never fatal: without the first, workspaces keep docker-default, as before; # without the second, Docker-enabled workspaces are not offered. apply_apparmor() { local data exec_root blocker WORKSPACE_APPARMOR_PROFILE="" WORKSPACE_NESTED_APPARMOR_PROFILE="" case "$APPARMOR_DECISION" in load) ;; noparser) warn "AppArmor is on, but apparmor_parser is not installed — workspaces keep Docker's docker-default profile." echo " Install the apparmor package and re-run to confine them further." >&2 return 0 ;; *) return 0 ;; esac data="$(apparmor_data_root "$1")" exec_root="$(apparmor_exec_root "$2")" if ! apparmor_root_ok "$data" || ! apparmor_root_ok "$exec_root"; then warn "Docker's data root ($data) or exec root ($exec_root) cannot go in an AppArmor rule — workspaces keep docker-default." return 0 fi load_apparmor_profile "$APPARMOR_PROFILE" "workspaces keep docker-default" \ "$data" "$exec_root" "${3:-}" || return 0 WORKSPACE_APPARMOR_PROFILE="$APPARMOR_PROFILE" log "Loaded the AppArmor profile $APPARMOR_PROFILE for workspaces (kept across reboots by $APPARMOR_DIR/$APPARMOR_PROFILE)." # Loaded whether or not the host is ready, so its file and the kernel's copy # never lag a re-run that makes it ready, and a reload reaches the # Docker-enabled workspaces already running. load_apparmor_profile "$APPARMOR_NESTED_PROFILE" "Docker-enabled workspaces are not offered" \ "$data" "$exec_root" "${3:-}" nested || return 0 blocker="$(nested_docker_blocker "${4:-no}")" if [ -n "$blocker" ]; then log "Docker-enabled workspaces are not offered: $blocker." return 0 fi WORKSPACE_NESTED_APPARMOR_PROFILE="$APPARMOR_NESTED_PROFILE" log "Loaded the AppArmor profile $APPARMOR_NESTED_PROFILE for Docker-enabled workspaces." return 0 } # --- end AppArmor workspace profile ------------------------------------------ # --- download + load images ------------------------------------------------ # TMP is chosen after the release is resolved, once the download size is known: # /tmp is a tmpfs on many VPS images (sized from RAM, often well under 1 GiB) # and a multi-hundred-megabyte image tarball does not fit. It is declared (and # the EXIT trap that removes it installed) up with the lock, so an abort at any # point between here and there still cleans up. # free_bytes <dir> — free space on the filesystem holding <dir>, in bytes. Prints # 0 when it cannot be determined (df on an unwritable path, an exotic # filesystem) — every caller treats 0 as "unknown, do not block on it", so a # preflight that cannot measure never becomes a preflight that refuses. free_bytes() { local v v="$(df -Pk "$1" 2>/dev/null | awk 'NR==2 {print $4 * 1024}')" || v="" case "$v" in ''|*[!0-9]*) v=0 ;; esac printf '%s' "$v" } # human <bytes> — MiB/GiB for a message an operator can act on. Integer shell # arithmetic only: this runs before jq is guaranteed and awk is already required. human() { local b="${1:-0}" if [ "$b" -ge 1073741824 ]; then printf '%s.%s GiB' "$((b / 1073741824))" "$(((b % 1073741824) * 10 / 1073741824))" elif [ "$b" -ge 1048576 ]; then printf '%s MiB' "$((b / 1048576))" else printf '%s bytes' "$b"; fi } # Prior install versions (for skip-if-unchanged upgrades). SERVER_RELEASE, # CONTAINER_RELEASE and ASSET_VERSIONS were added later, so an install that # predates them simply leaves these empty — reported as "unknown" or treated as # "not yet fetched", never as a failure. SERVER_VERSION=""; CONTAINER_VERSION="" SERVER_RELEASE=""; CONTAINER_RELEASE="" ASSET_VERSIONS="" RELEASE_VERSION="" # shellcheck source=/dev/null # written by a previous run of this script [ -f "$VERSIONS_FILE" ] && . "$VERSIONS_FILE" # RELEASE_VERSION is reused below for the release this run resolves, so the one # the file just supplied — what is installed *now* — is kept under its own name. PRIOR_RELEASE_VERSION="$RELEASE_VERSION" # asset_version_of <slug> — the version recorded for this slug at the last run, # or empty. ASSET_VERSIONS is a space-separated "slug=version" list, which keeps # it to one shell variable regardless of how many assets a release carries (slugs # are filenames, so they are not usable as variable names). asset_version_of() { # shellcheck disable=SC2086 # deliberate word splitting into one entry per line printf '%s\n' ${ASSET_VERSIONS:-} | awk -F= -v s="$1" '$1==s {print $2; exit}' } # --- weekly backup (decide) -------------------------------------------------- # The last question, asked here with the others rather than where the timer is # actually installed — which is after the images have come down, several minutes # into a run the operator would otherwise have been free to walk away from. # BACKUP_TIMER_DECISION is what install_backup_timer acts on later. # # Offered, never imposed: it writes to the customer's disk on a timer, and an # operator who already has their own backup story should not get a second one # they did not ask for. DEVGRAIL_SCHEDULED_BACKUPS=yes|no answers it unattended; # an existing timer is left exactly as configured. BACKUP_TIMER_DECISION=skip BACKUP_TIMER_NOTE="" if ! command -v systemctl >/dev/null 2>&1; then : # no systemd, no timer to offer elif [ -n "${DEVGRAIL_SCHEDULED_BACKUPS:-}" ]; then case "$(printf '%s' "$DEVGRAIL_SCHEDULED_BACKUPS" | tr '[:upper:]' '[:lower:]')" in yes|y|true|1) BACKUP_TIMER_DECISION=install ;; no|n|false|0) BACKUP_TIMER_DECISION=disable ;; *) die "DEVGRAIL_SCHEDULED_BACKUPS must be yes or no" ;; esac elif [ -f "$INSTALL_DIR/backups-host/policy.json" ] && [ -z "${DEVGRAIL_SCHEDULED_BACKUPS:-}" ]; then BACKUP_TIMER_DECISION=already BACKUP_TIMER_NOTE="Backups: existing dashboard schedule preserved. Manage in Administration -> Backups." elif systemctl list-unit-files devgrail-backup.timer >/dev/null 2>&1 && systemctl is-enabled devgrail-backup.timer >/dev/null 2>&1; then BACKUP_TIMER_DECISION=already BACKUP_TIMER_NOTE="Weekly backups: already scheduled (systemctl status devgrail-backup.timer)" else _want_timer="$(printf '%s' "${DEVGRAIL_SCHEDULED_BACKUPS:-}" | tr '[:upper:]' '[:lower:]')" # --yes takes the prompt's own default, which is yes: a machine nobody is # watching is exactly the one that benefits from a scheduled backup. # DEVGRAIL_SCHEDULED_BACKUPS still wins, so it stays possible to say no. if [ -z "$_want_timer" ] && [ "$ASSUME_YES" = true ]; then _want_timer=yes; fi if [ -z "$_want_timer" ] && [ "$DRY_RUN" != true ] && have_tty; then section "Weekly backups" echo " A systemd timer runs 'devgrail backup' once a week: the database plus" >&2 echo " the config.yaml whose key decrypts it, as one archive in $BACKUP_DIR" >&2 echo " on this same host. Usually a few MB, and taken without downtime." >&2 echo >&2 echo " What it does NOT cover: the files inside your workspaces — your" >&2 echo " source code lives in Docker volumes and is far too large to copy on a" >&2 echo " timer. Keep that in git. And a copy on this host does not survive" >&2 echo " losing this host, so for real disaster recovery add VPS snapshots or" >&2 echo " copy the archives off the box." >&2 echo >&2 echo " What it is for: getting the accounts, API keys, secrets and workspace" >&2 echo " records back after a bad change or a corrupted database." >&2 echo " devgrail list what archives exist here" >&2 echo " devgrail backup take one right now" >&2 echo " devgrail restore ARCHIVE put one back" >&2 echo " Upgrades snapshot themselves regardless of this answer; the timer is" >&2 echo " for the weeks in between. Keep-all is the default; manage archives and" >&2 echo " optional retention under Administration -> Backups." >&2 echo >&2 echo " Recommended: yes, unless you already back this host up yourself." >&2 if ask_yn " Schedule a weekly backup to $BACKUP_DIR?" y; then _want_timer=yes; else _want_timer=no; fi fi case "$_want_timer" in yes|y|true|1) BACKUP_TIMER_DECISION=install ;; *) BACKUP_TIMER_DECISION=skip ;; esac unset _want_timer fi # --- Docker ------------------------------------------------------------------ # Last thing before the run stops asking and starts doing. Deliberately after # every question above: installing Docker is the slow step, and an operator who # has just answered the last prompt should be able to leave. Nothing above it # needs Docker — a host without Docker has no Docker state, which is precisely # what the daemon preflight wanted to know. if ! command -v docker >/dev/null 2>&1; then log "Installing Docker..." # 9>&- keeps the install lock out of dockerd. Where systemd starts the daemon # it never sees our fds anyway, but on a host whose systemctl forks dockerd # directly (containers, the local sandbox) the daemon would inherit fd 9 and # hold the lock forever. acquire_lock can recover from that; not causing it is # cheaper. Plain `docker`/`docker compose` calls need no such guard — they are # short-lived clients, and workspace containers are children of dockerd. curl -fsSL https://get.docker.com | sh 9>&- systemctl enable --now docker 2>/dev/null 9>&- || true fi docker compose version >/dev/null 2>&1 || die "the 'docker compose' plugin is required" # --- resolve the release ---------------------------------------------------- # A DevGrail release is seven artifacts, not one. Resolving them all from a # single manifest is what stops an upgrade from mixing a new compose file with # an old server image: before this, the compose and Traefik config were # re-fetched unconditionally on every run because there was no asset version to # compare against, so even a "nothing changed" re-run silently adopted whatever # had been published since. DEVGRAIL_VERSION="${DEVGRAIL_VERSION:-current}" RELEASE_MANIFEST="" RELEASE_VERSION="" resolve_release() { local body body="$(curl -fsSL "$WEB/api/registry/releases/$DEVGRAIL_VERSION" 2>/dev/null)" || body="" if [ -z "$body" ]; then # An explicit version that does not resolve is an error — silently # installing something else is the one thing --version must never do. [ "$DEVGRAIL_VERSION" = current ] \ || die "release '$DEVGRAIL_VERSION' is not published at $WEB. Nothing has been changed." # No release manifest at all means a registry older than this installer. # Fall back to resolving each artifact by slug, exactly as before. log "Registry publishes no release manifest — resolving each artifact individually." return 0 fi RELEASE_MANIFEST="$body" RELEASE_VERSION="$(printf '%s' "$body" | jq -r '.release_version // empty')" log "Installing release ${RELEASE_VERSION:-unknown}" } resolve_release # --- release signature ------------------------------------------------------- # Every artifact is verified against a SHA-256 the registry publishes — which # catches a corrupted download but not a compromised registry, because whoever # can serve the bytes can serve the matching digest. The same origin also serves # this script, the compose file and the Traefik config, all executed as root. # # A detached signature over the release closes that, and only if the signing key # never touches the web host. One signature covers all seven artifacts: that is # the payoff of resolving a release from a single manifest. # # What is signed is a payload listing every artifact's kind, slug, version and # digest — not the manifest itself, whose download URLs are rewritten per origin # and so cannot be signed offline. Verification therefore proves the digests, # and the digests prove the bytes. See docs/SECURITY.md. # Public keys this installer trusts to sign a release. Generated by # scripts/release-key.sh; the private half is never in this repo. # # Rotation: add the new key here and ship an installer carrying both, then start # signing with the new one, then drop the old key a release later. Hosts that # never upgraded the installer keep working throughout. release_public_keys() { release_embedded_public_keys release_extra_public_keys } # The keys compiled into this installer. Split from the extras below because # the dashboard upgrade helper stores the two apart: the embedded set is # replaced by each installer that runs, so a rotated-out key actually goes. release_embedded_public_keys() { # key id f1f1eaf642454d65 — first embedded in v0.4.0 cat <<'PUBKEYS' -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEK1NbYrxBM8Ia1iM4BcR0m/8DelN5 ZDvOwKmEOJA24dci3ftqHWwsxzbw8bACRps0rIOTqa45wblm6260fQVdIw== -----END PUBLIC KEY----- PUBKEYS } # An operator running their own registry (or the test harness) can add keys # without editing this file. This grants trust rather than removing it, and # anyone able to set it already runs this script as root. release_extra_public_keys() { if [ -n "${DEVGRAIL_RELEASE_PUBKEY_FILE:-}" ] && [ -f "$DEVGRAIL_RELEASE_PUBKEY_FILE" ]; then cat "$DEVGRAIL_RELEASE_PUBKEY_FILE" fi } # The verified payload, once there is one. Empty means nothing is enforced # beyond the registry's own digests. SIGNED_PAYLOAD_FILE="" SIG_DIR="" # unverified <why> — how a release that cannot be checked is treated. See # ALLOW_UNSIGNED, resolved up with the tooling because openssl's absence is # the one form of "cannot be checked" that is knowable before any of this. unverified() { if [ "$ALLOW_UNSIGNED" != 1 ]; then die "refusing to install a release whose signature could not be verified: $1 Artifacts are checked against digests this registry publishes, which catches a corrupted download but not a compromised registry — whoever can serve the bytes can serve the matching digest. Only the signature tells those apart. Set DEVGRAIL_ALLOW_UNSIGNED=1 to install anyway, accepting that this host is trusting $WEB and nothing else. See docs/SECURITY.md. Nothing has been changed." fi warn "release signature not verified — $1" echo " DEVGRAIL_ALLOW_UNSIGNED=1 is set, so this run continues. Artifacts are" >&2 echo " still checked against the digests this registry publishes, which detects" >&2 echo " a corrupted download but not a compromised registry. See docs/SECURITY.md." >&2 } verify_release_signature() { local sig alg kid keys dir pem verified=false if [ -z "$RELEASE_MANIFEST" ]; then unverified "this registry publishes no release manifest" return 0 fi sig="$(printf '%s' "$RELEASE_MANIFEST" | jq -c '.signature // empty' 2>/dev/null || true)" if [ -z "$sig" ] || [ "$sig" = null ]; then unverified "release ${RELEASE_VERSION:-current} was published without one" return 0 fi keys="$(release_public_keys)" if [ -z "$keys" ]; then unverified "this installer carries no release signing key" return 0 fi if ! command -v openssl >/dev/null 2>&1; then unverified "openssl is not installed here, so the signature cannot be checked" return 0 fi alg="$(printf '%s' "$sig" | jq -r '.algorithm // empty')" kid="$(printf '%s' "$sig" | jq -r '.key_id // empty')" # An unknown algorithm is not the same as an unsigned release: the publisher # signed it with something this installer cannot check, and treating that as # "unsigned" would let an attacker downgrade verification by naming a scheme # we do not implement. [ "$alg" = "ecdsa-p256-sha256" ] \ || die "release ${RELEASE_VERSION:-current} is signed with '$alg', which this installer cannot verify. Upgrade the installer. Nothing has been changed." dir="$(mktemp -d)" SIG_DIR="$dir" # -j, not -r: the signature is over exactly the published bytes, and -r adds a # trailing newline the signer never saw. That difference verifies as tampering. printf '%s' "$sig" | jq -j '.payload' > "$dir/payload" printf '%s' "$sig" | jq -r '.signature' | base64 -d > "$dir/sig" 2>/dev/null \ || die "the signature on release ${RELEASE_VERSION:-current} is not valid base64. Nothing has been changed." # Each embedded key gets its own file; any one of them verifying is enough, # which is what makes rotation a matter of shipping two keys for a while. printf '%s\n' "$keys" > "$dir/keys.pem" awk -v d="$dir" '/-----BEGIN PUBLIC KEY-----/{n++} n>0 {print > (d "/key" n ".pem")}' "$dir/keys.pem" for pem in "$dir"/key*.pem; do [ -f "$pem" ] || continue if openssl dgst -sha256 -verify "$pem" -signature "$dir/sig" "$dir/payload" >/dev/null 2>&1; then verified=true break fi done if [ "$verified" != true ]; then die "the signature on release ${RELEASE_VERSION:-current} does not verify against any key this installer carries (signing key id: ${kid:-unknown}). That is either a corrupted download or a tampered release, and there is no way to tell which from here. Nothing has been changed." fi # A valid signature over somebody else's payload proves nothing about this # release, so the payload has to name it — otherwise an old signed release # could be replayed against a host asking for a newer one. if ! grep -qxF "devgrail-release-signature-v1" "$dir/payload"; then die "the release signature verifies but its payload is not in a format this installer understands. Nothing has been changed." fi if ! grep -qxF "release ${RELEASE_VERSION}" "$dir/payload"; then die "the signature verifies, but it was made over a different release than ${RELEASE_VERSION:-the one requested}. Nothing has been changed." fi SIGNED_PAYLOAD_FILE="$dir/payload" log "Release ${RELEASE_VERSION} signature verified (key ${kid:-unknown})." } # signed_digest_check <payload line> <what> — refuse anything the signed payload # does not name. Called after the registry's own digest matched, so this is the # check that survives a compromised registry: matching bytes are not enough if # they are not the bytes the release was signed for. signed_digest_check() { [ -n "$SIGNED_PAYLOAD_FILE" ] || return 0 if grep -qxF "$1" "$SIGNED_PAYLOAD_FILE"; then return 0 fi die "'$2' is not what release $RELEASE_VERSION was signed for. Its bytes match the digest this registry served, but that digest does not appear in the signed release payload: $1 Refusing to install it. Nothing has been changed." } verify_release_signature # release_field <slug> <jq filter> — a field of one artifact in the release # manifest, or empty when there is no manifest or no such artifact. release_field() { [ -n "$RELEASE_MANIFEST" ] || return 0 printf '%s' "$RELEASE_MANIFEST" | jq -r --arg s "$1" --arg a "$ARCH" "$2" 2>/dev/null || true } # same_image <ref> <ref> — true when both names resolve to one image. Either # name missing is "not the same", which is the answer that makes callers do the # safe thing. same_image() { local a b a="$(docker image inspect -f '{{.Id}}' "$1" 2>/dev/null)" || return 1 b="$(docker image inspect -f '{{.Id}}' "$2" 2>/dev/null)" || return 1 [ -n "$a" ] && [ "$a" = "$b" ] } # resolve_image <slug> <target_tag> <current_version> — settle which build of # <slug> this run needs, and whether it has to be downloaded at all, without # fetching a byte. Sets R_VER R_REL R_URL R_SHA R_SIZE R_TAG R_SKIP R_RETAG. # # Split out from download_image so the disk preflight below can total up what is # about to be written *before* the first download starts. Running out of space # halfway through is precisely the condition that causes a mid-migration # failure, so discovering it after one image has already landed is not good # enough. R_VER=""; R_REL=""; R_URL=""; R_SHA=""; R_SIZE=0; R_TAG=""; R_SKIP=false; R_RETAG=false resolve_image() { local slug="$1" target="$2" cur="$3" manifest # Prefer the release manifest: it pins the exact version this release is, and # its download URLs carry that pin so they keep resolving to these bytes even # after a newer release is published. R_VER="$(release_field "$slug" '.images[]|select(.slug==$s)|.image_version')" R_URL="$(release_field "$slug" '.images[]|select(.slug==$s)|.artifacts[]|select(.arch==$a)|.download_url')" R_SHA="$(release_field "$slug" '.images[]|select(.slug==$s)|.artifacts[]|select(.arch==$a)|.sha256')" R_SIZE="$(release_field "$slug" '.images[]|select(.slug==$s)|.artifacts[]|select(.arch==$a)|.size')" # The release this *image* was built under, which is not always the release # being installed. An image whose build context did not change is carried into # later releases at the version — and under the tarball tag — it originally # shipped with, so the tag to look for on this host is the image's own, not # this run's. Absent from a registry predating the field, where the two were # always the same thing anyway. R_REL="$(release_field "$slug" '.images[]|select(.slug==$s)|.image_release // empty')" [ -n "$R_REL" ] && [ "$R_REL" != "null" ] || R_REL="$RELEASE_VERSION" if [ -z "$R_VER" ]; then manifest="$(curl -fsSL "$WEB/api/registry/images/$slug")" \ || die "cannot fetch image manifest for '$slug' from $WEB" R_VER="$(printf '%s' "$manifest" | jq -r '.image_version')" # Additive field: a registry that predates it, or a row uploaded before it # existed, yields null — which becomes an empty string, not an error. R_REL="$(printf '%s' "$manifest" | jq -r '.release_version // empty')" R_URL="$(printf '%s' "$manifest" | jq -r --arg a "$ARCH" '.artifacts[]|select(.arch==$a)|.download_url')" R_SHA="$(printf '%s' "$manifest" | jq -r --arg a "$ARCH" '.artifacts[]|select(.arch==$a)|.sha256')" R_SIZE="$(printf '%s' "$manifest" | jq -r --arg a "$ARCH" '.artifacts[]|select(.arch==$a)|.size')" fi [ -n "$R_URL" ] && [ "$R_URL" != "null" ] || die "no $ARCH build published for '$slug'" # A registry that does not declare sizes yields null; treat that as "unknown" # (0) rather than failing — the preflight then simply has less to check. case "$R_SIZE" in ''|null|*[!0-9]*) R_SIZE=0 ;; esac # The tag `docker load` gave these exact bytes on the run that fetched them: # publish.sh builds every image as <slug>:<release>. Empty for a registry old # enough to publish no release version, which leaves the checks below on their # pre-release behaviour rather than guessing at a tag. R_TAG="" case "$R_REL" in ''|null) ;; *) R_TAG="$slug:$R_REL" ;; esac # Skipping requires that $target *is* this version — not merely that something # is tagged $target. `--rollback` points :latest back at the displaced image # and leaves versions.env naming the new one (it is written before the health # gate, so the rollback cannot un-write it). Testing existence alone made every # later run report "already at vX — skipping" for a version the host had rolled # off: it stayed on the old binary while versions.env and this script's own # closing banner both claimed the new one, /healthz was the only thing telling # the truth, and there was no way forward short of deleting versions.env. R_SKIP=false R_RETAG=false if [ -n "$cur" ] && [ "$cur" = "$R_VER" ] && docker image inspect "$target" >/dev/null 2>&1; then if [ -z "$R_TAG" ] || same_image "$target" "$R_TAG"; then R_SKIP=true elif docker image inspect "$R_TAG" >/dev/null 2>&1; then # The bytes are still on this host under their release tag — downloaded, # checksum- and signature-checked by the run that loaded them. Moving the # tag back is the whole recovery; re-fetching 750 MiB to arrive at the # image already sitting there is not. R_RETAG=true fi fi } LOADED_VERSION=""; LOADED_RELEASE="" # retag_target <target> <source_ref> — point <target> at <source_ref>, naming # whatever it displaced :previous first. # # Retagging moves :latest off the image currently running, which then survives # only as an untagged layer set until someone prunes it. Naming it :previous is # a tag, not a copy — so it costs nothing, and rollback becomes a retag plus a # compose recreate rather than a re-download. retag_target() { local target="$1" src="$2" prev_tag old_id new_id prev_tag="${target%:*}:previous" old_id="$(docker image inspect -f '{{.Id}}' "$target" 2>/dev/null || true)" new_id="$(docker image inspect -f '{{.Id}}' "$src" 2>/dev/null || true)" if [ -n "$old_id" ] && [ "$old_id" != "$new_id" ]; then docker tag "$old_id" "$prev_tag" log "Previous ${target%:*} kept as $prev_tag" fi docker tag "$src" "$target" } # download_image <slug> <target_tag> <current_version> [current_release] # Uses the R_* values resolve_image already settled for this slug. download_image() { local slug="$1" target="$2" cur="$3" cur_rel="${4:-}" loaded got tar local url="$R_URL" sha="$R_SHA" ver="$R_VER" rel="$R_REL" if [ "$R_SKIP" = true ]; then # Two different situations, and an operator has to be able to tell them # apart. Re-running the same release is "already at it". A *newer* release # that ships this image unchanged keeps an image whose release name is # older than the one being installed — which reads like a failed upgrade # unless it says why. if [ -n "$rel" ] && [ -n "$RELEASE_VERSION" ] && [ "$rel" != "$RELEASE_VERSION" ]; then log "$slug is unchanged in $RELEASE_VERSION — keeping $rel, nothing to download" else log "$slug already at ${rel:-v$ver} — skipping" fi LOADED_VERSION="$ver" # Prefer what the registry says now: it fills in the release for an install # that was made before the field existed, without re-downloading anything. LOADED_RELEASE="${rel:-$cur_rel}" return fi if [ "$R_RETAG" = true ]; then log "$slug ${rel:-v$ver} is loaded but $target points elsewhere — restoring it" retag_target "$target" "$R_TAG" LOADED_VERSION="$ver"; LOADED_RELEASE="$rel" return fi tar="$TMP/$slug.tar" log "Downloading $slug ${rel:-v$ver} ($ARCH)..." curl -fL --progress-bar -o "$tar" "$url" || die "download failed for '$slug'" log "Verifying checksum..." got="$(sha256_of "$tar")" [ "$got" = "$sha" ] || die "checksum mismatch for '$slug' (expected $sha, got $got)" signed_digest_check "image $slug $ver $ARCH $got" "$slug ($ARCH)" log "Loading image..." # Not `| head -n1`: head exits after one line, sed dies of SIGPIPE, and # `set -o pipefail` turns that into a fatal exit 141 — mid-upgrade, right after # the image was loaded, with no message. sed reads all of docker load's output # here; the first line is taken with parameter expansion afterwards. loaded="$(docker load -i "$tar" | sed -n 's/^Loaded image: //p')" loaded="${loaded%%$'\n'*}" [ -n "$loaded" ] || die "docker load produced no image for '$slug'" retag_target "$target" "$loaded" rm -f "$tar" LOADED_VERSION="$ver"; LOADED_RELEASE="$rel" } # Resolve both images first, so the preflight sees the whole run. resolve_image devgrail-server "devgrail-server:latest" "$SERVER_VERSION" SERVER_SKIP="$R_SKIP"; SERVER_RETAG="$R_RETAG"; SERVER_BYTES="$R_SIZE" SERVER_R_VER="$R_VER"; SERVER_R_REL="$R_REL"; SERVER_R_URL="$R_URL"; SERVER_R_SHA="$R_SHA"; SERVER_R_TAG="$R_TAG" # --workspace-image=no does not resolve it at all, rather than resolving it and # then ignoring the answer: resolve_image talks to the registry and dies when the # release publishes no build for this architecture, which is not a reason to # refuse an install that was never going to download it. CONTAINER_SKIP=false; CONTAINER_RETAG=false; CONTAINER_BYTES=0 CONTAINER_R_VER=""; CONTAINER_R_REL=""; CONTAINER_R_URL=""; CONTAINER_R_SHA=""; CONTAINER_R_TAG="" if [ "$DEVGRAIL_WORKSPACE_IMAGE" = yes ]; then resolve_image devgrail-container "devgrail-container:latest" "$CONTAINER_VERSION" CONTAINER_SKIP="$R_SKIP"; CONTAINER_RETAG="$R_RETAG"; CONTAINER_BYTES="$R_SIZE" CONTAINER_R_VER="$R_VER"; CONTAINER_R_REL="$R_REL"; CONTAINER_R_URL="$R_URL"; CONTAINER_R_SHA="$R_SHA"; CONTAINER_R_TAG="$R_TAG" fi # --- disk preflight --------------------------------------------------------- # Every upgrade downloads a tarball and then needs the space again in the Docker # data root when `docker load` unpacks it, and nothing prunes. Disk-full is not # merely untidy here: it is the condition that produces a failure mid-migration, # which is the worst state this product can be in. So the numbers are checked # before the first byte is fetched, and reported when they do not add up. # # 2.5x the declared download size: 1x for the tarball, ~1x again for the # unpacked layers, plus half for the compressed layers Docker keeps while # loading and for headroom. A registry that declares no sizes yields 0, and the # check for that filesystem is skipped rather than guessed at. DOWNLOAD_BYTES=0 # A retag fetches nothing either — the bytes are already in the Docker data root. [ "$SERVER_SKIP" = true ] || [ "$SERVER_RETAG" = true ] || DOWNLOAD_BYTES=$((DOWNLOAD_BYTES + SERVER_BYTES)) # So does a workspace image this run is not fetching. CONTAINER_BYTES is already # 0 in that case; the test is here so the intent survives a later edit. if [ "$DEVGRAIL_WORKSPACE_IMAGE" = yes ]; then [ "$CONTAINER_SKIP" = true ] || [ "$CONTAINER_RETAG" = true ] || DOWNLOAD_BYTES=$((DOWNLOAD_BYTES + CONTAINER_BYTES)) fi NEED_BYTES=$((DOWNLOAD_BYTES * 5 / 2)) # --- dry run ----------------------------------------------------------------- # "Re-run the same script that did the first install" is a fine upgrade # mechanism right up until the operator wants to know what it will do before it # does it. Everything above this line is resolution — reading the registry and # the prior install's own files — so the report can be produced with nothing # downloaded, nothing written and nothing started. # # It sits here rather than earlier because the release has to be resolved for # any of it to be true, and earlier than the next line because that is where the # first byte gets staged on disk. # dry_change <from> <to> — "a -> b", or "b (unchanged)" when they match. dry_change() { local from="${1:-}" to="${2:-}" [ -n "$to" ] || to="unknown" if [ -z "$from" ]; then printf '%s (new)' "$to" elif [ "$from" = "$to" ]; then printf '%s (unchanged)' "$to" else printf '%s -> %s' "$from" "$to"; fi } dry_image_line() { local slug="$1" cur="$2" ver="$3" rel="$4" skip="$5" bytes="$6" retag="${7:-false}" if [ "$skip" = true ]; then # The release is named here because it need not be the one being installed: # an image carried unchanged into a newer release keeps its own. printf ' %-20s v%s%s (unchanged)\n' "$slug" "$ver" "${rel:+ $rel}" elif [ "$retag" = true ]; then # Not "(unchanged)": the version is the one recorded, but :latest is not on # it — which is exactly the state an operator needs told, since it is what # a rollback leaves behind and what this run would undo. printf ' %-20s v%s%s already loaded — would point %s:latest back at it\n' \ "$slug" "$ver" "${rel:+ $rel}" "$slug" else printf ' %-20s %s%s — would download %s\n' "$slug" \ "$(dry_change "${cur:+v$cur}" "v$ver")" "${rel:+ $rel}" "$(human "$bytes")" fi } dry_asset_line() { local slug="$1" ver have ver="$(release_field "$slug" '.assets[]|select(.slug==$s)|.asset_version')" have="$(asset_version_of "$slug")" if [ -z "$ver" ]; then printf ' %-24s would be re-fetched (the registry publishes no version for it)\n' "$slug" elif [ "$ver" = "$have" ] && [ -f "$ASSET_DIR/$slug" ]; then printf ' %-24s v%s (unchanged)\n' "$slug" "$ver" else printf ' %-24s %s — would download\n' "$slug" "$(dry_change "${have:+v$have}" "v$ver")" fi } dry_run_report() { local tmpdir="${TMPDIR:-/tmp}" docker_root key was now changed=false health schema echo echo "============================================================" echo " Dry run — nothing on this host has been changed." echo if [ "$IS_UPGRADE" = true ]; then echo " Mode: upgrade of the install at $INSTALL_DIR" else echo " Mode: first install into $INSTALL_DIR" fi echo " Registry: $WEB" echo " Release: $(dry_change "$PRIOR_RELEASE_VERSION" "${RELEASE_VERSION:-not published as a release}")" if [ -n "$SIGNED_PAYLOAD_FILE" ]; then echo " Signature: verified against a key embedded in this installer" else echo " Signature: not verified (see the warning above)" fi echo echo " Images" dry_image_line devgrail-server "$SERVER_VERSION" "$SERVER_R_VER" \ "$SERVER_R_REL" "$SERVER_SKIP" "$SERVER_BYTES" "$SERVER_RETAG" if [ "$DEVGRAIL_WORKSPACE_IMAGE" = yes ]; then dry_image_line devgrail-container "$CONTAINER_VERSION" "$CONTAINER_R_VER" \ "$CONTAINER_R_REL" "$CONTAINER_SKIP" "$CONTAINER_BYTES" "$CONTAINER_RETAG" elif docker image inspect devgrail-container:latest >/dev/null 2>&1; then # Worth distinguishing: "skipped" on a host that already has the image reads # like something is about to break, and nothing is. printf ' %-20s %s\n' devgrail-container \ "not fetched (--workspace-image=no) — the copy already on this host is kept" else printf ' %-20s %s\n' devgrail-container \ "skipped (--workspace-image=no) — no workspace can be created until it is fetched" fi echo echo " Assets" # Only the two this run would actually fetch. install.sh and uninstall.sh are # published as assets too, but they are served to `curl`, never downloaded # here, and listing them would suggest otherwise. dry_asset_line docker-compose.yml dry_asset_line "$TRAEFIK_ASSET" echo echo " Disk" if [ "$NEED_BYTES" -gt 0 ]; then printf ' would need %s (2.5x the %s to download)\n' \ "$(human "$NEED_BYTES")" "$(human "$DOWNLOAD_BYTES")" else echo " nothing to download" fi printf ' free in %s: %s\n' "$tmpdir" "$(human "$(free_bytes "$tmpdir")")" docker_root="$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || true)" [ -n "$docker_root" ] && [ -d "$docker_root" ] || docker_root=/var/lib/docker if [ -d "$docker_root" ]; then printf ' free in %s: %s\n' "$docker_root" "$(human "$(free_bytes "$docker_root")")" fi if [ "$INOTIFY_DECISION" = raise ]; then echo echo " Kernel" printf ' %s would be raised %s -> %s, and kept in %s\n' \ "$INOTIFY_KEY" "$INOTIFY_NOW" "$INOTIFY_TARGET" "$INOTIFY_CONF" fi case "$APPARMOR_DECISION" in load) echo echo " AppArmor" echo " workspaces created from now on would run under $APPARMOR_PROFILE," echo " written to $APPARMOR_DIR/$APPARMOR_PROFILE and loaded" echo " Docker-enabled ones under $APPARMOR_NESTED_PROFILE, where userns-remap" echo " is on and $FUSE_DEVICE exists" ;; noparser) echo echo " AppArmor" echo " apparmor_parser is not installed, so workspaces would keep docker-default" ;; esac if [ "$IS_UPGRADE" = true ]; then echo echo " Settings" # DEVGRAIL_ADMIN_USERNAME is listed; the password never is. for key in DEVGRAIL_DOMAIN DEVGRAIL_BASE_DOMAIN ACME_EMAIL DEVGRAIL_ACME_CASERVER DEVGRAIL_ACME_BYPASS DEVGRAIL_ADMIN_USERNAME DEVGRAIL_TLS DEVGRAIL_WORKSPACE_IMAGE DEVGRAIL_DASHBOARD_UPGRADES HTTP_PORT HTTPS_PORT; do was="$(prior "$key")" eval "now=\${$key:-}" if [ "$was" != "$now" ]; then printf ' %-22s %s -> %s\n' "$key" "${was:-(unset)}" "${now:-(unset)}" changed=true fi done if [ "$changed" = false ]; then echo " unchanged — every value in the prior .env would be preserved" fi echo echo " Data" echo " a snapshot of the database and config.yaml would be written to" echo " $BACKUP_DIR before the new server starts (newest ${DEVGRAIL_SNAPSHOT_RETAIN:-5} kept)" health="$(curl -fsSk --max-time 5 "$(dashboard_url)/api/v1/healthz" 2>/dev/null || true)" schema="$(printf '%s' "$health" | jq -r '.schema_version.applied // empty' 2>/dev/null || true)" if [ -n "$schema" ]; then echo " this database is at schema version $schema"; fi echo " any pending migrations are applied by the new server when it starts;" echo " which ones cannot be listed without loading that image, and a dry" echo " run does not download it" fi echo echo " Re-run without --dry-run to apply this." echo "============================================================" echo return 0 } decide_inotify decide_apparmor if [ "$DRY_RUN" = true ]; then dry_run_report exit 0 fi # --- the plan ---------------------------------------------------------------- # Everything above this line asked questions or read something. Everything below # it changes this host. So this is where the run says what it is about to do, in # one screen, and — when there is somebody to ask — gets one confirmation for the # lot. That is the point the operator can walk away from. # # Unattended runs (--yes, or no terminal) print the same summary and proceed: # the plan is as useful in a cloud-init log as it is on a terminal, and a build # that cannot answer must not be stopped by a question. # 13 wide because "Certificates:" is the longest label; a narrower column # would push that one row out of line with the rest. plan_line() { printf ' %-13s %s\n' "$1" "$2"; } # The phases the apply half will walk through, counted now that every condition # that adds or removes one has been settled. Five are unconditional: fetch the # images, write the deployment, install the CLI, start the stack, wait for it to # answer. The rest depend on this particular run. STEP_TOTAL=5 if [ "$USERNS_DECISION" = enable ] || [ "$POOLS_DECISION" = widen ]; then STEP_TOTAL=$((STEP_TOTAL + 1)) fi [ "$IS_UPGRADE" = true ] && STEP_TOTAL=$((STEP_TOTAL + 1)) [ "$BACKUP_TIMER_DECISION" = install ] && STEP_TOTAL=$((STEP_TOTAL + 1)) echo echo "============================================================" if [ "$IS_UPGRADE" = true ]; then echo " Ready to upgrade DevGrail at $INSTALL_DIR" else echo " Ready to install DevGrail into $INSTALL_DIR" fi echo plan_line "Release:" "$(dry_change "$PRIOR_RELEASE_VERSION" "${RELEASE_VERSION:-unversioned}") from $WEB" plan_line "Dashboard:" "$(dashboard_url)/" if [ "$TLS_ENABLED" = true ]; then if [ "$ACME_CA_IS_PRODUCTION" = true ]; then plan_line "Certificates:" "Let's Encrypt, contact ${ACME_EMAIL:-unset}" else plan_line "Certificates:" "NON-PRODUCTION ACME CA — browsers will not trust them" fi if [ "$DEVGRAIL_ACME_BYPASS" = true ]; then plan_line "ACME bypass:" "on — passthrough routes receive their own challenges" fi else plan_line "Certificates:" "none — plain HTTP (DEVGRAIL_TLS=off)" fi plan_line "Subdomains:" "*.$DEVGRAIL_BASE_DOMAIN" if [ "$IS_UPGRADE" = true ]; then plan_line "Admin:" "unchanged — the account already exists" elif [ "$ADMIN_PW_GENERATED" = true ]; then plan_line "Admin:" "$ADMIN_USER, with a generated password shown at the end" else plan_line "Admin:" "$ADMIN_USER, with the password you entered" fi echo if [ "$DOWNLOAD_BYTES" -gt 0 ]; then plan_line "Download:" "$(human "$DOWNLOAD_BYTES") ($(human "$NEED_BYTES") of disk needed to unpack it)" else plan_line "Download:" "nothing — every artifact this release names is already here" fi if [ "$DEVGRAIL_WORKSPACE_IMAGE" != yes ]; then plan_line "" "workspace image skipped (--workspace-image=no)" fi if [ "$IS_UPGRADE" = true ]; then plan_line "Snapshot:" "database + config.yaml into $BACKUP_DIR before the new server starts" fi # Only mentioned when something is going to happen to the daemon. A run that # changes nothing there should not spend two lines saying so. if [ "$USERNS_DECISION" = enable ] || [ "$POOLS_DECISION" = widen ]; then echo [ "$USERNS_DECISION" = enable ] && plan_line "Docker:" "enable userns-remap" [ "$POOLS_DECISION" = widen ] && plan_line "Docker:" "widen the default address pools to $POOL_BASE" plan_line "" "both edit $DAEMON_JSON and restart Docker" fi if [ "$INOTIFY_DECISION" = raise ]; then echo plan_line "Kernel:" "raise $INOTIFY_KEY $INOTIFY_NOW -> $INOTIFY_TARGET ($INOTIFY_CONF)" fi # Only on the run that first loads it; every later run reloads it quietly. _aa_label="AppArmor:" if [ "$APPARMOR_DECISION" = load ] && [ ! -f "$APPARMOR_DIR/$APPARMOR_PROFILE" ]; then echo plan_line "$_aa_label" "load the $APPARMOR_PROFILE profile for new workspaces ($APPARMOR_DIR/$APPARMOR_PROFILE)" _aa_label="" fi if [ "$APPARMOR_DECISION" = load ] && [ ! -f "$APPARMOR_DIR/$APPARMOR_NESTED_PROFILE" ]; then [ -n "$_aa_label" ] && echo plan_line "$_aa_label" "load the $APPARMOR_NESTED_PROFILE profile for Docker-enabled workspaces ($APPARMOR_DIR/$APPARMOR_NESTED_PROFILE)" fi unset _aa_label case "$BACKUP_TIMER_DECISION" in install) plan_line "Backups:" "weekly systemd timer into $BACKUP_DIR" ;; already) plan_line "Backups:" "weekly timer already scheduled — left as configured" ;; esac echo "============================================================" echo if have_tty && [ "$ASSUME_YES" != true ]; then if ! ask_yn "Proceed?" y; then die "aborted at the plan. Nothing on this host has been changed." fi echo fi # Before the first download, for the reason apply_daemon_decisions documents: # enabling userns-remap moves the daemon's data root out from under any image # loaded ahead of it. if [ "$USERNS_DECISION" = enable ] || [ "$POOLS_DECISION" = widen ]; then step "Applying the Docker daemon settings..." fi apply_daemon_decisions apply_inotify # After the daemon decisions: enabling userns-remap moves the data root, and # whether it is now on decides if Docker-enabled workspaces are offered. The # label is dockerd's own confinement, which docker kill signals from. apply_apparmor "$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || true)" \ "$(jq -r '."exec-root" // empty' "$DAEMON_JSON" 2>/dev/null || true)" \ "$(sed 's/ (.*//' "/proc/$(pgrep -xo dockerd 2>/dev/null || echo 0)/attr/current" 2>/dev/null || true)" \ "$(if userns_remap_enabled; then echo yes; else echo no; fi)" # Where the tarballs are staged. /tmp is a tmpfs on many VPS images — sized from # RAM and often well under a gigabyte — so falling back to $INSTALL_DIR/tmp when # it does not fit turns "no space left on device" partway through a download # into a directory that has the room. choose_tmp() { local candidate="${TMPDIR:-/tmp}" fallback="$INSTALL_DIR/tmp" have have_fb have="$(free_bytes "$candidate")" # 0 means "could not measure", which is not the same as "no room". Nothing # here is worth refusing an upgrade over on a guess. if [ "$NEED_BYTES" -gt 0 ] && [ "$have" -gt 0 ] && [ "$have" -lt "$NEED_BYTES" ]; then mkdir -p "$fallback" || die "cannot create $fallback for the download" have_fb="$(free_bytes "$fallback")" if [ "$have_fb" -gt 0 ] && [ "$have_fb" -lt "$NEED_BYTES" ]; then die "not enough free space to download this release. Needed: $(human "$NEED_BYTES") (2.5x the $(human "$DOWNLOAD_BYTES") of artifacts) Free in $candidate: $(human "$have") Free in $fallback: $(human "$have_fb") Free some space and re-run. Nothing has been changed." fi log "$candidate is too small for this download — staging in $fallback instead" TMP="$(mktemp -d "$fallback/install.XXXXXX")" else TMP="$(mktemp -d)" fi } choose_tmp if [ "$NEED_BYTES" -gt 0 ]; then DOCKER_ROOT="$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || true)" [ -n "$DOCKER_ROOT" ] && [ -d "$DOCKER_ROOT" ] || DOCKER_ROOT=/var/lib/docker if [ -d "$DOCKER_ROOT" ]; then DOCKER_FREE="$(free_bytes "$DOCKER_ROOT")" if [ "$DOCKER_FREE" -gt 0 ] && [ "$DOCKER_FREE" -lt "$NEED_BYTES" ]; then die "not enough free space in the Docker data root to load this release. Needed: $(human "$NEED_BYTES") (2.5x the $(human "$DOWNLOAD_BYTES") of artifacts) Free in $DOCKER_ROOT: $(human "$DOCKER_FREE") Reclaim space (\`docker image prune\`, \`docker system df\`) and re-run. Nothing has been changed." fi fi fi if [ "$DOWNLOAD_BYTES" -gt 0 ]; then step "Fetching the DevGrail images ($(human "$DOWNLOAD_BYTES"))..." else step "Checking the DevGrail images..." fi R_SKIP="$SERVER_SKIP"; R_RETAG="$SERVER_RETAG"; R_TAG="$SERVER_R_TAG" R_VER="$SERVER_R_VER"; R_REL="$SERVER_R_REL"; R_URL="$SERVER_R_URL"; R_SHA="$SERVER_R_SHA" download_image devgrail-server "devgrail-server:latest" "$SERVER_VERSION" "$SERVER_RELEASE" NEW_SERVER_VERSION="$LOADED_VERSION"; NEW_SERVER_RELEASE="$LOADED_RELEASE" # The workspace image tag is hard-coded in internal/docker (DefaultImage = # devgrail-container:latest), so retag the loaded release to :latest. # # Skipped by --workspace-image=no, which carries the prior versions forward # verbatim instead of recording anything. That is right in both directions: a # host that never fetched the image keeps two empty values, so the run that does # fetch it sees no current version and downloads; a host that fetched it on an # earlier run keeps versions.env describing the image still sitting on its disk. # Note it does NOT go through download_image with R_SKIP=true — that path logs # "already at ..." and would be a lie on the first case. if [ "$DEVGRAIL_WORKSPACE_IMAGE" = yes ]; then R_SKIP="$CONTAINER_SKIP"; R_RETAG="$CONTAINER_RETAG"; R_TAG="$CONTAINER_R_TAG" R_VER="$CONTAINER_R_VER"; R_REL="$CONTAINER_R_REL"; R_URL="$CONTAINER_R_URL"; R_SHA="$CONTAINER_R_SHA" download_image devgrail-container "devgrail-container:latest" "$CONTAINER_VERSION" "$CONTAINER_RELEASE" NEW_CONTAINER_VERSION="$LOADED_VERSION"; NEW_CONTAINER_RELEASE="$LOADED_RELEASE" else log "Skipping the workspace image (--workspace-image=no)" NEW_CONTAINER_VERSION="$CONTAINER_VERSION"; NEW_CONTAINER_RELEASE="$CONTAINER_RELEASE" fi step "Writing the deployment to $DEPLOY_DIR..." # --- deploy assets --------------------------------------------------------- log "Writing deploy config to $INSTALL_DIR..." mkdir -p "$DEPLOY_DIR/traefik" "$DEPLOY_DIR/dynamic" "$CONFIG_DIR" "$ASSET_DIR" mkdir -p "$BACKUP_DIR" "$DEPLOY_DIR/backups-request" "$DEPLOY_DIR/backups-status" chmod 700 "$BACKUP_DIR" "$DEPLOY_DIR/backups-request" "$DEPLOY_DIR/backups-status" # Traefik mounts $DEPLOY_DIR/dynamic read-only (it reads routes; devgrail-server # writes them), and the compose file then bind-mounts traefik/dynamic.yml at # dynamic/system.yml — nested inside that read-only mount. Docker creates a # missing mountpoint in the target directory, which it cannot do on a read-only # one: the container dies at start with EROFS and the whole site 404s. So create # the empty stub here. Harmless on an upgrade, where a previous rw mount already # left one behind, and harmless on a release whose compose still mounts rw. [ -e "$DEPLOY_DIR/dynamic/system.yml" ] || : > "$DEPLOY_DIR/dynamic/system.yml" NEW_ASSET_VERSIONS="" # fetch_asset <slug> — leave the release's copy of <slug> at $ASSET_PATH, # downloading and verifying it only if this run needs a version we do not # already hold. ASSET_PATH="" fetch_asset() { local slug="$1" url ver sha cached got have cached="$ASSET_DIR/$slug" ver="$(release_field "$slug" '.assets[]|select(.slug==$s)|.asset_version')" url="$(release_field "$slug" '.assets[]|select(.slug==$s)|.download_url')" sha="$(release_field "$slug" '.assets[]|select(.slug==$s)|.sha256')" if [ -z "$url" ]; then # No release manifest (or an asset it does not carry): fetch whatever is # published, as installers did before releases existed. Unversioned, so it # cannot be skipped — that is the behaviour releases exist to fix. url="$WEB/install/$slug"; ver=""; sha="" fi have="$(asset_version_of "$slug")" if [ -n "$ver" ] && [ "$ver" = "$have" ] && [ -f "$cached" ]; then log "$slug already at v$ver — skipping" else curl -fsSL "$url" -o "$cached.tmp" || die "cannot fetch '$slug' from $WEB" got="$(sha256_of "$cached.tmp")" if [ -n "$sha" ]; then [ "$got" = "$sha" ] || die "checksum mismatch for '$slug' (expected $sha, got $got)" fi # Verified before the templating below rewrites it: what the release was # signed for is the published file, not what this host makes of it. if [ -n "$ver" ]; then signed_digest_check "asset $slug $ver $got" "$slug" elif [ -n "$SIGNED_PAYLOAD_FILE" ]; then # The release verified, but does not name this asset — so it came from the # unversioned fallback path and nothing signed it. That is the same hole as # an unsigned release, opened one file at a time, and this file is either # the compose that defines the whole stack or the Traefik config that # decides who may reach it. A verified release does not get to contain an # unverified artifact. if [ "$ALLOW_UNSIGNED" != 1 ]; then die "'$slug' is not part of signed release $RELEASE_VERSION, so nothing vouches for the bytes just fetched from $WEB. The rest of this release verified, which makes this the odd one out rather than an old registry. Set DEVGRAIL_ALLOW_UNSIGNED=1 to accept it anyway." fi warn "'$slug' is not part of signed release $RELEASE_VERSION — fetched unverified." fi mv -f "$cached.tmp" "$cached" fi if [ -n "$ver" ]; then NEW_ASSET_VERSIONS="${NEW_ASSET_VERSIONS:+$NEW_ASSET_VERSIONS }$slug=$ver" fi ASSET_PATH="$cached" } # The rendered outputs are always rewritten from the cached copies, even when # nothing was downloaded: DEVGRAIL_DOMAIN or DEVGRAIL_TLS may have changed at # the prompt, and a config that silently kept a stale domain would be worse than # a redundant write. fetch_asset docker-compose.yml cp -f "$ASSET_PATH" "$DEPLOY_DIR/docker-compose.yml" # This script is always served at latest, but --version can install an older # release whose compose file predates DEVGRAIL_ADMIN_USERNAME. That file would # simply drop the variable, the server would create `admin`, and the banner below # would name a user that does not exist. Say so and fall back instead. if [ "$ADMIN_USER" != admin ] && ! grep -q DEVGRAIL_ADMIN_USERNAME "$DEPLOY_DIR/docker-compose.yml"; then warn "release ${NEW_SERVER_RELEASE:-this one} cannot take a custom admin username; creating \`admin\` instead." ADMIN_USER="admin" fi # The TLS or plain-HTTP router set per DEVGRAIL_TLS. Either is written to the # same dynamic.yml the compose file mounts by default, so no compose var is needed. fetch_asset "$TRAEFIK_ASSET" sed "s|__DEVGRAIL_DOMAIN__|$DEVGRAIL_DOMAIN|g" "$ASSET_PATH" > "$DEPLOY_DIR/traefik/dynamic.yml" # config.yaml holds the JWT secret (not passed via env). It used to be written # only when absent, so registry_url/tls_enabled went stale as soon as the # operator changed them at the prompt. The .env does override both at runtime # (internal/config/config.go), so that was latent rather than active — but two # files disagreeing about the domain is a trap to walk into during an incident. # The derived keys are rewritten every run; the secret never is. CONFIG_FILE="$CONFIG_DIR/config.yaml" if [ -f "$CONFIG_FILE" ]; then # Same SIGPIPE-under-pipefail hazard as the docker load above. Aborting here # would be particularly bad: the operator would see a bare exit 141 while the # installer was reading the secret that decrypts their entire database. JWT_SECRET="$(sed -n 's/^jwt_secret:[[:space:]]*//p' "$CONFIG_FILE")" JWT_SECRET="$(printf '%s' "${JWT_SECRET%%$'\n'*}" | tr -d "\"' ")" # Generating a fresh secret here would orphan every encrypted value in the # database — every API key secret, user secret and workspace SSH password. # Refusing loudly is the only safe answer. [ -n "$JWT_SECRET" ] || die "$CONFIG_FILE exists but has no jwt_secret. That secret keys every encrypted value in the database; generating a new one would make all of them undecryptable. Restore the file from a backup ($BACKUP_DIR holds config.yaml alongside the database) and re-run. Nothing has been changed." else JWT_SECRET="$(gen_secret)" fi # Anything the operator added by hand is carried over — this installer owns the # derived keys, not the whole file. CONFIG_EXTRA="" if [ -f "$CONFIG_FILE" ]; then CONFIG_EXTRA="$(grep -vE '^(jwt_secret|domain|base_domain|registry_url|traefik_dynamic_path|tls_enabled|backup_dir):' "$CONFIG_FILE" || true)" fi cat > "$CONFIG_FILE.tmp" <<EOF jwt_secret: "$JWT_SECRET" domain: "$DEVGRAIL_DOMAIN" base_domain: "$DEVGRAIL_BASE_DOMAIN" registry_url: "$WEB" traefik_dynamic_path: /dynamic/managed.yml tls_enabled: $TLS_ENABLED backup_dir: "$BACKUP_DIR" EOF if [ -n "$CONFIG_EXTRA" ]; then printf '%s\n' "$CONFIG_EXTRA" >> "$CONFIG_FILE.tmp"; fi # Same reasoning as .env, more so: a truncated config.yaml loses the JWT secret, # which is unrecoverable without a backup. chmod 600 "$CONFIG_FILE.tmp" mv -f "$CONFIG_FILE.tmp" "$CONFIG_FILE" # ADMIN_USER / ADMIN_PW were settled with the other answers above, prior values # included: they seed the admin account on first boot only (internal/db/seed.go: # SeedAdmin ignores them thereafter unless DEVGRAIL_FORCE_ADMIN_PASSWORD is set), # so dropping the password here would leave the operator with an account whose # password nobody knows. # Keys this installer owns. Everything else found in the prior .env is carried # over verbatim below — an operator who added DEVGRAIL_FORCE_ADMIN_PASSWORD or # any other compose variable must not lose it to an upgrade. MANAGED_ENV_KEYS=" ACME_EMAIL DEVGRAIL_ACME_CASERVER DEVGRAIL_ACME_STORAGE DEVGRAIL_ACME_BYPASS DEVGRAIL_DOMAIN DEVGRAIL_BASE_DOMAIN DEVGRAIL_ADMIN_USERNAME DEVGRAIL_ADMIN_PASSWORD DEVGRAIL_SERVER_IMAGE DEVGRAIL_TLS DEVGRAIL_WORKSPACE_IMAGE DEVGRAIL_WORKSPACE_APPARMOR_PROFILE DEVGRAIL_WORKSPACE_NESTED_APPARMOR_PROFILE DEVGRAIL_DASHBOARD_UPGRADES TAG HTTP_PORT HTTPS_PORT " cat > "$DEPLOY_DIR/.env.tmp" <<EOF ACME_EMAIL=${ACME_EMAIL:-} DEVGRAIL_ACME_CASERVER=$DEVGRAIL_ACME_CASERVER DEVGRAIL_ACME_STORAGE=$DEVGRAIL_ACME_STORAGE DEVGRAIL_ACME_BYPASS=$DEVGRAIL_ACME_BYPASS DEVGRAIL_DOMAIN=$DEVGRAIL_DOMAIN DEVGRAIL_BASE_DOMAIN=$DEVGRAIL_BASE_DOMAIN DEVGRAIL_ADMIN_USERNAME=$ADMIN_USER DEVGRAIL_ADMIN_PASSWORD=$ADMIN_PW DEVGRAIL_SERVER_IMAGE=$DEVGRAIL_SERVER_IMAGE DEVGRAIL_TLS=$DEVGRAIL_TLS DEVGRAIL_WORKSPACE_IMAGE=$DEVGRAIL_WORKSPACE_IMAGE DEVGRAIL_WORKSPACE_APPARMOR_PROFILE=$WORKSPACE_APPARMOR_PROFILE DEVGRAIL_WORKSPACE_NESTED_APPARMOR_PROFILE=$WORKSPACE_NESTED_APPARMOR_PROFILE DEVGRAIL_DASHBOARD_UPGRADES=$DEVGRAIL_DASHBOARD_UPGRADES TAG=latest HTTP_PORT=$HTTP_PORT HTTPS_PORT=$HTTPS_PORT EOF for _key in $PRIOR_KEYS; do case "$MANAGED_ENV_KEYS" in *" $_key "*) continue ;; esac # Copied verbatim from the prior file rather than re-serialised from the # parsed value: quoting we did not write is not ours to normalise, and this # runs before the temp file replaces it. Keys are [A-Za-z0-9_], so the pattern # carries no regex metacharacters. grep -m1 "^$_key=" "$DEPLOY_DIR/.env" >> "$DEPLOY_DIR/.env.tmp" || true done # Written via a temp file: this holds the admin password, and a truncated write # on a full disk would hand the operator a stack that cannot be logged into. chmod 600 "$DEPLOY_DIR/.env.tmp" mv -f "$DEPLOY_DIR/.env.tmp" "$DEPLOY_DIR/.env" # The integers are the registry's ordering key (what the skip-if-unchanged check # compares); the releases are the human-facing identity reported in the banner. # ASSET_VERSIONS is quoted because it holds a space-separated list, and this file # is sourced by the next run. cat > "$VERSIONS_FILE" <<EOF SERVER_VERSION=$NEW_SERVER_VERSION CONTAINER_VERSION=$NEW_CONTAINER_VERSION SERVER_RELEASE=$NEW_SERVER_RELEASE CONTAINER_RELEASE=$NEW_CONTAINER_RELEASE RELEASE_VERSION=$RELEASE_VERSION ASSET_VERSIONS="$NEW_ASSET_VERSIONS" EOF # --- pre-upgrade snapshot --------------------------------------------------- # Everything above rewrote config in place; the next line starts the new server, # which migrates the database on boot. Until this ran there was no point in the # upgrade where the pre-upgrade state was recoverable. # # The archive holds devgrail.db AND config.yaml. A database-only backup is # worthless: the JWT secret in config.yaml keys every at-rest cipher, so without # it every API key secret, user secret and workspace SSH password decrypts to # garbage. SNAPSHOT_RETAIN="${DEVGRAIL_SNAPSHOT_RETAIN:-5}" snapshot_before_upgrade() { local vol db_bytes free_bytes need ts stage archive summary schema vol="$(data_volume)" if [ -z "$vol" ]; then log "No existing DevGrail data volume found — nothing to snapshot." return 0 fi db_bytes="$(docker run --rm --userns=host -v "$vol":/var/lib/devgrail:ro \ --entrypoint /bin/sh devgrail-server:latest \ -c 'stat -c %s /var/lib/devgrail/devgrail.db 2>/dev/null || echo 0' 2>/dev/null)" || db_bytes=0 case "$db_bytes" in ''|*[!0-9]*) db_bytes=0 ;; esac if [ "$db_bytes" -eq 0 ]; then log "Data volume '$vol' holds no database yet — nothing to snapshot." return 0 fi exec 7>"$INSTALL_DIR/.backups.lock" flock -x 7 mkdir -p "$BACKUP_DIR" chmod 700 "$BACKUP_DIR" # Room is needed twice: once for the vacuumed copy, once for the tarball built # from it. Refusing here is not just hygiene — a full disk is precisely the # condition that causes the mid-migration failure this snapshot exists for. need=$((db_bytes * 2)) free_bytes="$(df -Pk "$BACKUP_DIR" | awk 'NR==2 {print $4 * 1024}')" || free_bytes=0 if [ "${free_bytes:-0}" -lt "$need" ]; then die "not enough free space for the pre-upgrade snapshot in $BACKUP_DIR: need $need bytes, have ${free_bytes:-0}. Free some space, or move $INSTALL_DIR to a larger filesystem, then re-run. Nothing has been changed." fi ts="$(date -u +%Y%m%dT%H%M%SZ)" stage="$BACKUP_DIR/.stage-$ts" rm -rf "$stage" mkdir -p "$stage" SNAPSHOT_STAGE="$stage" log "Taking a pre-upgrade snapshot (data volume: $vol)..." # Stop only the server; Traefik stays up so the host keeps answering. compose stop devgrail-server >/dev/null 2>&1 || true # VACUUM INTO, never `cp`: in WAL mode the newest committed data lives in the # -wal sidecar, so copying devgrail.db alone yields a corrupt backup. It runs # in a one-shot container off the image just loaded, so the host needs no # sqlite3; /etc/devgrail is mounted so it reads the deployment's real db_path. # --userns=host mirrors the compose services, which opt out of userns-remap. summary="$(docker run --rm --userns=host \ -v "$vol":/var/lib/devgrail \ -v "$stage":/backup \ -v "$CONFIG_DIR":/etc/devgrail:ro \ -e DEVGRAIL_CONFIG=/etc/devgrail/config.yaml \ --entrypoint /usr/local/bin/devgrail-server \ devgrail-server:latest -vacuum-into /backup/devgrail.db)" \ || die "pre-upgrade snapshot failed — refusing to start the new version. Your data is untouched and the previous version is still installed; re-run this installer once the cause is fixed." schema="$(printf '%s' "$summary" | jq -r '.schema_version // "unknown"')" cp -p "$CONFIG_DIR/config.yaml" "$stage/config.yaml" \ || die "pre-upgrade snapshot: could not copy $CONFIG_DIR/config.yaml. A database-only backup cannot decrypt anything it contains, so this is not a usable snapshot — refusing to continue." cat > "$stage/manifest.json" <<EOF { "created_at": "$(date -u +%Y-%m-%dT%H:%M:%SZ)", "domain": "$DEVGRAIL_DOMAIN", "schema_version": "$schema", "data_volume": "$vol", "prior_server_version": "${SERVER_VERSION:-unknown}", "prior_container_version": "${CONTAINER_VERSION:-unknown}", "prior_release_version": "${SERVER_RELEASE:-unknown}", "new_server_version": "${NEW_SERVER_VERSION:-unknown}", "new_container_version": "${NEW_CONTAINER_VERSION:-unknown}", "contents": ["devgrail.db", "config.yaml", "manifest.json"] } EOF archive="$BACKUP_DIR/pre-upgrade-$ts.tgz" tar czf "$stage/archive.tgz" -C "$stage" devgrail.db config.yaml manifest.json \ || die "pre-upgrade snapshot: could not write $archive" chmod 600 "$stage/archive.tgz" ln -- "$stage/archive.tgz" "$archive" \ || die "pre-upgrade snapshot: could not publish $archive without overwriting another archive" # The archive contains the JWT secret and every encrypted secret it protects. chmod 600 "$archive" rm -rf "$stage" SNAPSHOT_STAGE="" SNAPSHOT_ARCHIVE="$archive" log "Pre-upgrade snapshot written to $archive" # Keep the newest N. Unbounded snapshots eventually fill the disk, which is # the condition that causes the failure they protect against. # `|| true` because pipefail would otherwise abort the whole install on the # nothing-to-prune case, where ls has no matches. ls -1t "$BACKUP_DIR"/pre-upgrade-*.tgz 2>/dev/null | tail -n +$((SNAPSHOT_RETAIN + 1)) \ | while read -r old; do rm -f "$old"; done || true } if [ "$IS_UPGRADE" = true ]; then step "Snapshotting the database before the new server starts..." snapshot_before_upgrade fi exec 7>&- # --- devgrail maintenance command ------------------------------------------- # The pre-upgrade snapshot only exists when an upgrade happens to run. An # operator also needs a backup they can take now — before editing something, or # on a schedule — and a restore that does the steps in the one order that works. # Both live in the server binary (`-backup` / `-restore`); this wrapper is what # runs them as a one-shot container off the installed image, so the host needs # neither sqlite3 nor a copy of the Go toolchain. # # Written from here rather than shipped as an eighth release artifact: it is # entirely derived from paths this installer already owns, so it can never # disagree with the install it was written for. CLI_PATH=/usr/local/bin/devgrail write_devgrail_cli() { cat > "$CLI_PATH.tmp" <<CLIEOF #!/usr/bin/env bash # DevGrail maintenance CLI — written by install.sh. Do not edit; re-running the # installer overwrites it. set -euo pipefail INSTALL_DIR="$INSTALL_DIR" DEPLOY_DIR="$DEPLOY_DIR" CONFIG_DIR="$CONFIG_DIR" BACKUP_DIR="$BACKUP_DIR" IMAGE=devgrail-server:latest log() { printf '\033[1;34m==>\033[0m %s\n' "\$*"; } die() { printf '\033[1;31merror:\033[0m %s\n' "\$*" >&2; exit 1; } # Mirrors install.sh: an operator's docker-compose.override.yml applies to the # stop/start this does around a restore, or the restored server comes back # without it. compose() { if [ -f "\$DEPLOY_DIR/docker-compose.override.yml" ]; then docker compose --project-directory "\$DEPLOY_DIR" \\ -f "\$DEPLOY_DIR/docker-compose.yml" \\ -f "\$DEPLOY_DIR/docker-compose.override.yml" "\$@" else docker compose --project-directory "\$DEPLOY_DIR" -f "\$DEPLOY_DIR/docker-compose.yml" "\$@" fi } usage() { cat <<'USAGE' DevGrail maintenance commands. Run as root. devgrail backup [--out PATH] write a backup archive (database + config.yaml + manifest). Default: the backups directory, named backup-<timestamp>.tgz devgrail restore ARCHIVE stop the server, restore the archive, start it again, and wait for the readiness probe devgrail list list the archives on this host devgrail summary print the install summary and reference card again — dashboard URL, DNS records, upgrade, rollback and backup commands. Rewritten by every install.sh run. devgrail passwd [USERNAME] set a user's password (default: the admin this install created) and revoke their sessions. The way back in after a forgotten password. devgrail list-users list the accounts on this deployment with their role and whether each is disabled devgrail add-user NAME [--admin] create an account, prompting for its password. --admin makes it an admin. Accounts are normally created under Users in the dashboard; this is what works when the dashboard does not, and what a provisioning script can call. devgrail certificates production|staging switch the installation certificate environment. Staging is browser-untrusted; keep SSH open. devgrail help show this passwd and add-user prompt twice at a terminal. When stdin is not a terminal they read the password from its first line instead, for scripts: printf '%s\n' "\$PASSWORD" | devgrail add-user alice A backup archive holds BOTH the database and config.yaml. The jwt_secret in config.yaml keys every encrypted value in the database, so a database-only backup restores rows that decrypt to garbage. Never separate them. Workspace volumes are NOT included — they hold the customer's source code and are far larger than the database. See docs/RUNBOOK.md for the export recipe. USAGE } # data_volume prints the Docker volume backing /var/lib/devgrail, resolved from # the running container rather than guessed from the Compose project name. data_volume() { local vol vol="\$(docker inspect devgrail-server \\ --format '{{range .Mounts}}{{if eq .Destination "/var/lib/devgrail"}}{{.Name}}{{end}}{{end}}' 2>/dev/null)" || vol="" if [ -n "\$vol" ]; then printf '%s' "\$vol"; return 0; fi docker volume ls --format '{{.Name}}' 2>/dev/null | grep -E '(^|_)devgrail-data\$' | head -n1 || true } # read_new_password PROMPT sets NEW_PASSWORD for cmd_passwd and cmd_add_user. At # a terminal it prompts twice with echo off. Otherwise it takes the first line of # stdin, so a provisioning script can run # \`printf '%s\n' "\$pw" | devgrail add-user alice\` without a pseudo-terminal. # Any later lines are ignored. # # IFS= is what keeps leading and trailing spaces. A bare \`read -r\` strips them, # which stored a password other than the one typed; the server keeps whitespace, # so the shell has to as well. read_new_password() { local prompt="\$1" pw2="" NEW_PASSWORD="" if [ -t 0 ]; then printf '%s: ' "\$prompt" > /dev/tty stty -echo 2>/dev/null || true IFS= read -r NEW_PASSWORD < /dev/tty || true stty echo 2>/dev/null || true printf '\n' > /dev/tty printf 'Repeat password: ' > /dev/tty stty -echo 2>/dev/null || true IFS= read -r pw2 < /dev/tty || true stty echo 2>/dev/null || true printf '\n' > /dev/tty [ -n "\$NEW_PASSWORD" ] || die "no password entered" [ "\$NEW_PASSWORD" = "\$pw2" ] || die "the passwords do not match" else # No confirmation here: a script has nobody to mistype, and asking it for # the same line twice would only make it harder to call. IFS= read -r NEW_PASSWORD || true [ -n "\$NEW_PASSWORD" ] || die "no password on stdin" fi } # cmd_backup writes one archive via a one-shot container off the installed # server image, with the data volume, /etc/devgrail and the output directory # mounted. --userns=host mirrors the compose services, which opt out of # userns-remap. cmd_backup() { if [ "\$#" -eq 0 ] && [ -x "\$INSTALL_DIR/backups-host/devgrail-backups-helper" ]; then "\$INSTALL_DIR/backups-host/devgrail-backups-helper" --cli return fi local out="" ts vol dir while [ \$# -gt 0 ]; do case "\$1" in --out=*) out="\${1#*=}" ;; --out) shift; out="\${1:-}" ;; *) usage >&2; die "unknown option: \$1" ;; esac shift done ts="\$(date -u +%Y%m%dT%H%M%SZ)" [ -n "\$out" ] || out="\$BACKUP_DIR/backup-\$ts.tgz" dir="\$(cd "\$(dirname "\$out")" 2>/dev/null && pwd)" || die "no such directory: \$(dirname "\$out")" mkdir -p "\$BACKUP_DIR"; chmod 700 "\$BACKUP_DIR" vol="\$(data_volume)" [ -n "\$vol" ] || die "cannot find the DevGrail data volume — is DevGrail installed here?" # Taken against a running server on purpose: VACUUM INTO reads through a single # read transaction, so the copy is consistent without downtime. The volume is # mounted read-write even though nothing here modifies the data: opening a # WAL-mode SQLite database requires writing its -shm sidecar, so a read-only # mount fails outright with "attempt to write a readonly database". docker run --rm --userns=host \\ -v "\$vol":/var/lib/devgrail \\ -v "\$CONFIG_DIR":/etc/devgrail:ro \\ -v "\$dir":/backup \\ -e DEVGRAIL_CONFIG=/etc/devgrail/config.yaml \\ --entrypoint /usr/local/bin/devgrail-server \\ "\$IMAGE" -backup "/backup/\$(basename "\$out")" \\ || die "backup failed — nothing was written" chmod 600 "\$out" 2>/dev/null || true log "Backup written to \$out" } cmd_restore() { local archive="\${1:-}" vol dir [ -n "\$archive" ] || { usage >&2; die "restore needs an archive path"; } [ -f "\$archive" ] || die "no such archive: \$archive" dir="\$(cd "\$(dirname "\$archive")" && pwd)" vol="\$(data_volume)" [ -n "\$vol" ] || die "cannot find the DevGrail data volume — refusing to guess which volume to overwrite." # The documented order, enforced: stop -> config.yaml -> database -> start -> # verify. Restoring under a running server would swap the database out from # under open connections. log "Stopping devgrail-server..." compose stop devgrail-server >/dev/null 2>&1 || true docker run --rm --userns=host \\ -v "\$vol":/var/lib/devgrail \\ -v "\$CONFIG_DIR":/etc/devgrail \\ -v "\$dir":/restore:ro \\ -e DEVGRAIL_CONFIG=/etc/devgrail/config.yaml \\ --entrypoint /usr/local/bin/devgrail-server \\ "\$IMAGE" -restore "/restore/\$(basename "\$archive")" \\ || { compose up -d >/dev/null 2>&1 || true; die "restore failed — the previous state is unchanged and the server has been restarted"; } log "Starting devgrail-server..." compose up -d local waited=0 while [ "\$waited" -lt 180 ]; do if compose exec -T devgrail-server /usr/local/bin/devgrail-server -healthcheck >/dev/null 2>&1; then log "DevGrail is ready. Restore complete." return 0 fi sleep 3 waited=\$((waited + 3)) done compose logs --tail=50 devgrail-server >&2 || true die "the restored server did not become ready within 180s (logs above)" } # cmd_passwd resets a password without a login — the way back in when nobody # knows the admin's. It runs the same one-shot container cmd_backup does; the new # password goes in on stdin, never as an argument or an environment variable, # because both are readable via \`ps\` and \`docker inspect\` while the process runs. # # No stop/start: SQLite is in WAL mode, so the write lands under the running # server, and the sessions it revokes are re-checked on the next request. cmd_passwd() { local user="\${1:-}" vol if [ -z "\$user" ]; then user="\$(sed -n 's/^DEVGRAIL_ADMIN_USERNAME=//p' "\$DEPLOY_DIR/.env" 2>/dev/null | tail -n1)" user="\${user:-admin}" fi read_new_password "New password for \$user" vol="\$(data_volume)" [ -n "\$vol" ] || die "cannot find the DevGrail data volume — is DevGrail installed here?" printf '%s' "\$NEW_PASSWORD" | docker run --rm -i --userns=host \\ -v "\$vol":/var/lib/devgrail \\ -v "\$CONFIG_DIR":/etc/devgrail:ro \\ -e DEVGRAIL_CONFIG=/etc/devgrail/config.yaml \\ --entrypoint /usr/local/bin/devgrail-server \\ "\$IMAGE" -set-password "\$user" \\ || die "password not changed" } # cmd_list_users and cmd_add_user are the accounts half of the same escape # hatch. Accounts are normally managed under Users in the dashboard; these exist # for when it cannot be reached, and for unattended provisioning. Reading and # creating only: disabling and purging an account destroy another person's work, # and those belong where they are audited and state their consequences first. cmd_list_users() { local vol [ \$# -eq 0 ] || { usage >&2; die "list-users takes no arguments"; } vol="\$(data_volume)" [ -n "\$vol" ] || die "cannot find the DevGrail data volume — is DevGrail installed here?" # Mounted read-write for the reason cmd_backup gives: opening a WAL-mode # SQLite database writes its -shm sidecar even when nothing is being changed. docker run --rm --userns=host \ -v "\$vol":/var/lib/devgrail \ -v "\$CONFIG_DIR":/etc/devgrail:ro \ -e DEVGRAIL_CONFIG=/etc/devgrail/config.yaml \ --entrypoint /usr/local/bin/devgrail-server \ "\$IMAGE" -list-users } # The password goes in on stdin, never as an argument or an environment # variable, for the reason cmd_passwd states. cmd_add_user() { local user="" admin=0 vol while [ \$# -gt 0 ]; do case "\$1" in --admin) admin=1 ;; -*) usage >&2; die "unknown option: \$1" ;; *) [ -z "\$user" ] || { usage >&2; die "add-user takes one username"; } user="\$1" ;; esac shift done [ -n "\$user" ] || { usage >&2; die "add-user needs a username"; } read_new_password "Password for \$user" vol="\$(data_volume)" [ -n "\$vol" ] || die "cannot find the DevGrail data volume — is DevGrail installed here?" # --admin is folded into the positional list rather than interpolated as a # possibly-empty word, which the server would otherwise read as a username. set -- -add-user "\$user" if [ "\$admin" -eq 1 ]; then set -- "\$@" -admin; fi printf '%s' "\$NEW_PASSWORD" | docker run --rm -i --userns=host \ -v "\$vol":/var/lib/devgrail \ -v "\$CONFIG_DIR":/etc/devgrail:ro \ -e DEVGRAIL_CONFIG=/etc/devgrail/config.yaml \ --entrypoint /usr/local/bin/devgrail-server \ "\$IMAGE" "\$@" \ || die "account not created" } case "\${1:-help}" in certificates) shift [ "\$(id -u)" -eq 0 ] || die "Run as root: sudo devgrail certificates production" [ \$# -eq 1 ] || die "Use: devgrail certificates production|staging" case "\$1" in production|staging) ;; *) die "Choose production or staging" ;; esac exec "\$INSTALL_DIR/certificates/devgrail-certificates-helper" --apply "\$1" ;; backup) shift; cmd_backup "\$@" ;; restore) shift; cmd_restore "\$@" ;; passwd) shift; cmd_passwd "\$@" ;; list-users) shift; cmd_list_users "\$@" ;; add-user) shift; cmd_add_user "\$@" ;; list) ls -lh "\$BACKUP_DIR" 2>/dev/null || echo "no archives in \$BACKUP_DIR" ;; summary) # The answer to "the install scrolled away and I lost the URL". install.sh # rewrites this file on every run, so it always describes the install that # is actually here. if [ -f "\$INSTALL_DIR/README-install.txt" ]; then cat "\$INSTALL_DIR/README-install.txt" else die "no \$INSTALL_DIR/README-install.txt on this host — re-run install.sh to write one." fi ;; help|-h|--help) usage ;; *) usage >&2; die "unknown command: \$1" ;; esac CLIEOF chmod 755 "$CLI_PATH.tmp" mv -f "$CLI_PATH.tmp" "$CLI_PATH" } step "Installing the devgrail command..." if write_devgrail_cli; then log "Maintenance CLI installed: $CLI_PATH (backup / restore / passwd / users / certificates)" else warn "could not write $CLI_PATH — backup and restore will need the docker run commands from docs/RUNBOOK.md" fi # --- backup management host helper ------------------------------------------ # Fixed, self-contained script: requests contain data, never executable shell. backups_helper_body() { cat <<'BACKUPSHELPER' set -euo pipefail umask 077 WORK="$HELPER_DIR/work" POLICY="$HELPER_DIR/policy.json" mkdir -p "$WORK" "$STATUS_DIR/operations" "$ARCHIVES" chmod 700 "$HELPER_DIR" "$WORK" "$STATUS_DIR" "$ARCHIVES" now() { date -u +%Y-%m-%dT%H:%M:%SZ; } volume() { local id id="$(docker compose --project-directory "$DEPLOY_DIR" -f "$DEPLOY_DIR/docker-compose.yml" ps -a -q devgrail-server 2>/dev/null || true)" if [ -n "$id" ]; then docker inspect -f '{{range .Mounts}}{{if eq .Destination "/var/lib/devgrail"}}{{.Name}}{{end}}{{end}}' "$id" else docker volume ls --filter label=com.docker.compose.project=deploy --filter label=com.docker.compose.volume=devgrail-data -q | head -n1 fi } run_mode() { local mode="$1"; shift local vol vol="$(volume)"; [ -n "$vol" ] || { echo 'DevGrail data volume unavailable' >&2; return 1; } docker run --rm -i --userns=host \ -v "$vol":/var/lib/devgrail -v "$CONFIG_DIR":/etc/devgrail:ro \ -v "$ARCHIVES":/backups/archives -v "$STATUS_DIR":/backups/status \ -e DEVGRAIL_CONFIG=/etc/devgrail/config.yaml \ -e DEVGRAIL_BACKUP_ARCHIVE_DIR=/backups/archives -e DEVGRAIL_BACKUP_STATUS_DIR=/backups/status \ --entrypoint /usr/local/bin/devgrail-server devgrail-server:latest -backup-manage="$mode" "$@" } atomic_jq() { local target="$1"; shift; jq "$@" > "$target.tmp" && mv -f "$target.tmp" "$target"; } operation() { local state="$1" message="${2:-}" atomic_jq "$STATUS_DIR/operations/$OP_ID.json" --arg state "$state" --arg message "$message" --arg finished "$(now)" \ '.operation | .state=$state | .message=$message | .finished_at=(if $state=="running" then "" else $finished end)' "$REQ" cp "$STATUS_DIR/operations/$OP_ID.json" "$STATUS_DIR/latest.json.tmp" mv -f "$STATUS_DIR/latest.json.tmp" "$STATUS_DIR/latest.json" jq -c '{actor:.actor,action:.action,operation:.id,state:.state}' "$STATUS_DIR/latest.json" >&2 if [ "$(jq -r .operation.action "$REQ")" = create ]; then cp "$STATUS_DIR/latest.json" "$STATUS_DIR/last-backup.json.tmp" mv -f "$STATUS_DIR/last-backup.json.tmp" "$STATUS_DIR/last-backup.json" fi } import_policy() { if [ ! -f "$POLICY" ]; then local enabled=false systemctl is-enabled devgrail-backup.timer >/dev/null 2>&1 && enabled=true printf '{"enabled":%s,"frequency":"weekly","time":"00:00","weekday":1,"month_day":1,"keep":0}\n' "$enabled" > "$POLICY.tmp" mv -f "$POLICY.tmp" "$POLICY" fi } report() ( exec 6>"$HELPER_DIR/.report.lock" flock -x 6 import_policy local enabled=false available=false reason="Backup request watcher is inactive. Check devgrail-backups.path on the host." calendar delay next timezone latest custom=false systemctl is-enabled devgrail-backup.timer >/dev/null 2>&1 && enabled=true if systemctl is-active --quiet devgrail-backups.path; then available=true; reason=""; fi calendar="$(systemctl cat devgrail-backup.timer 2>/dev/null | awk -F= '/^OnCalendar=/{if($2==""){result=""}else{result=result (result==""?"":";") $2}} END{print result}' || true)" delay="$(systemctl show devgrail-backup.timer -p RandomizedDelayUSec --value 2>/dev/null || true)" next="$(systemctl show devgrail-backup.timer -p NextElapseUSecRealtime --value 2>/dev/null || true)" timezone="$(timedatectl show -p Timezone --value 2>/dev/null || date +%Z)" latest=null; [ ! -f "$STATUS_DIR/latest.json" ] || latest="$(cat "$STATUS_DIR/latest.json")" local expected expected="$(jq -r 'if .frequency=="daily" then "*-*-* "+.time+":00" elif .frequency=="weekly" then (["Sun","Mon","Tue","Wed","Thu","Fri","Sat"][.weekday])+" *-*-* "+.time+":00" else "*-*-"+(if .month_day<10 then "0" else "" end)+(.month_day|tostring)+" "+.time+":00" end' "$POLICY")" if [ -n "$calendar" ] && [ "$calendar" != "$expected" ] && { [ "$calendar" != weekly ] || [ "$expected" != "Mon *-*-* 00:00:00" ]; }; then custom=true; fi jq -n --slurpfile policy "$POLICY" --argjson enabled "$enabled" --argjson available "$available" --arg reason "$reason" --arg calendar "$calendar" --arg delay "$delay" \ --arg next "$next" --arg timezone "$timezone" --arg observed "$(now)" --argjson custom "$custom" --argjson latest "$latest" \ --argjson snapshots "$PRE_UPGRADE_KEEP" \ '{protocol:1,settings:{available:$available,schedule_available:$available,unavailable_reason:$reason,policy:($policy[0]|.enabled=$enabled),calendar:$calendar,random_delay:$delay,next_run:$next,timezone:$timezone,observed_at:$observed,custom_schedule:$custom,pre_upgrade_keep:$snapshots,latest_operation:$latest}}' > "$STATUS_DIR/report.json.tmp" mv -f "$STATUS_DIR/report.json.tmp" "$STATUS_DIR/report.json" ) # Reporting never claims a request and does not need the installer lock. if [ "${1:-}" = --report ]; then report # Mark abandoned work only after the service is no longer active. if (exec 7>"$INSTALL_DIR/.backups.lock"; flock -xn 7) && [ -f "$STATUS_DIR/latest.json" ] && jq -e '.state=="running"' "$STATUS_DIR/latest.json" >/dev/null; then jq '.state="interrupted" | .message="The host backup worker stopped. Check its journal and retry."' "$STATUS_DIR/latest.json" > "$STATUS_DIR/latest.json.tmp" mv -f "$STATUS_DIR/latest.json.tmp" "$STATUS_DIR/latest.json" id="$(jq -r .id "$STATUS_DIR/latest.json")" [[ "$id" =~ ^[a-f0-9]{32}$ ]] && cp "$STATUS_DIR/latest.json" "$STATUS_DIR/operations/$id.json" report fi exit 0 fi # Install/upgrade takes this lock exclusively. Shared backup operations never # overwrite its owner record; lock order is always install, then backup. exec 9>"$INSTALL_DIR/.install.lock" flock -s -w 120 9 || { echo 'Installer busy; backup deferred' >&2; exit 1; } exec 8>"$INSTALL_DIR/.backups.lock" flock -x -w 120 8 || { echo 'Another backup operation is running' >&2; exit 1; } REQ="$WORK/request" rm -f "$REQ" case "${1:-}" in --scheduled|--cli) id="$(cat /proc/sys/kernel/random/uuid | tr -d '-')" jq -n --arg id "$id" --arg created "$(now)" '{protocol:1,operation:{id:$id,action:"create",actor:"system",state:"queued",created_at:$created,finished_at:"",message:"",archive_id:""}}' > "$REQ" origin=scheduled; [ "$1" != --cli ] || origin=cli ;; --index) run_mode index; exit $? ;; *) # Rename a symlink/FIFO as itself, then reject it before any read. mv -fT "$REQUEST_DIR/request" "$REQ" 2>/dev/null || exit 0 origin=manual ;; esac if [ -L "$REQ" ] || [ ! -f "$REQ" ] || [ "$(stat -c %s "$REQ")" -gt 8192 ]; then rm -f "$REQ"; echo 'Rejected unsafe backup request' >&2; exit 1 fi # Read at most one bounded snapshot into private storage; an open descriptor # retained by a hostile request writer cannot change the parsed request later. head -c 8193 "$REQ" > "$WORK/captured.json" [ "$(stat -c %s "$WORK/captured.json")" -le 8192 ] || exit 1 run_mode validate < "$WORK/captured.json" > "$WORK/validated.json" || { rm -f "$REQ"; exit 1; } mv -f "$WORK/validated.json" "$REQ" OP_ID="$(jq -r .operation.id "$REQ")" action="$(jq -r .operation.action "$REQ")" # Always record failure if a worker exits or is killed after claiming work. completed=false finish() { if [ "$completed" != true ]; then operation failed 'Backup operation interrupted or failed; check the host backup journal.' || true; fi rm -f "$REQ" report || true } trap finish EXIT trap 'exit 143' TERM trap 'exit 130' INT operation running report import_policy if [ "$action" = settings ]; then calendar="$(run_mode calendar < "$REQ")" for unit in devgrail-backup.timer devgrail-backup.service; do rm -f "$WORK/$unit.before"; [ ! -f "$UNIT_DIR/$unit" ] || cp -p "$UNIT_DIR/$unit" "$WORK/$unit.before" done dropin="$UNIT_DIR/devgrail-backup.timer.d/zzzz-devgrail-backups.conf" mkdir -p "$(dirname "$dropin")" rm -f "$WORK/timer-dropin.before" [ ! -f "$dropin" ] || cp -p "$dropin" "$WORK/timer-dropin.before" cp -p "$POLICY" "$WORK/policy.before" was_enabled=false; systemctl is-enabled devgrail-backup.timer >/dev/null 2>&1 && was_enabled=true cat > $UNIT_DIR/devgrail-backup.service <<UNIT [Unit] Description=DevGrail scheduled backup After=docker.service Requires=docker.service [Service] Type=oneshot ExecStart=$HELPER_DIR/devgrail-backups-helper --scheduled TimeoutStartSec=2h UNIT cat > $UNIT_DIR/devgrail-backup.timer <<UNIT [Unit] Description=Scheduled DevGrail backup [Timer] OnCalendar=$calendar Persistent=true [Install] WantedBy=timers.target UNIT # Reset inherited OnCalendar entries without deleting operator-owned drop-ins. cat > "$dropin" <<UNIT [Timer] OnCalendar= OnCalendar=$calendar RandomizedDelaySec=0 Persistent=true UNIT apply_timer() { systemctl daemon-reload && systemctl disable --now devgrail-backup.timer && { if jq -e '.policy.enabled' "$REQ" >/dev/null; then systemctl enable --now devgrail-backup.timer; fi; } } if apply_timer; then jq .policy "$REQ" > "$POLICY.tmp"; mv -f "$POLICY.tmp" "$POLICY" operation succeeded; completed=true else for unit in devgrail-backup.timer devgrail-backup.service; do if [ -f "$WORK/$unit.before" ]; then cp -p "$WORK/$unit.before" "$UNIT_DIR/$unit"; else rm -f "$UNIT_DIR/$unit"; fi done if [ -f "$WORK/timer-dropin.before" ]; then cp -p "$WORK/timer-dropin.before" "$dropin"; else rm -f "$dropin"; fi cp -p "$WORK/policy.before" "$POLICY" systemctl daemon-reload || true if [ "$was_enabled" = true ]; then systemctl enable --now devgrail-backup.timer || true; else systemctl disable --now devgrail-backup.timer || true; fi operation failed 'Schedule update failed; previous settings restored.'; completed=true fi else keep="$(jq -r .keep "$POLICY")" if run_mode execute -backup-keep="$keep" -backup-origin="$origin" < "$REQ"; then completed=true else if jq -e --arg id "$OP_ID" '.id==$id and (.state=="failed" or .state=="succeeded")' "$STATUS_DIR/latest.json" >/dev/null; then completed=true; fi exit 1 fi fi # Operation history is bounded; archives are pruned only by the Go retention code. find "$STATUS_DIR/operations" -maxdepth 1 -type f -name '*.json' -printf '%T@ %p\n' | sort -rn | tail -n +51 | cut -d' ' -f2- | while IFS= read -r file; do rm -f -- "$file"; done BACKUPSHELPER } install_backups_helper() { local helper_dir="$INSTALL_DIR/backups-host" mkdir -p "$helper_dir" "$DEPLOY_DIR/backups-request" "$DEPLOY_DIR/backups-status/operations" "$BACKUP_DIR" chmod 700 "$helper_dir" "$DEPLOY_DIR/backups-request" "$DEPLOY_DIR/backups-status" "$BACKUP_DIR" { printf '#!/usr/bin/env bash\nUNIT_DIR=/etc/systemd/system\n' printf 'INSTALL_DIR=%q\nDEPLOY_DIR=%q\nCONFIG_DIR=%q\nARCHIVES=%q\nHELPER_DIR=%q\nREQUEST_DIR=%q\nSTATUS_DIR=%q\nPRE_UPGRADE_KEEP=%q\n' \ "$INSTALL_DIR" "$DEPLOY_DIR" "$CONFIG_DIR" "$BACKUP_DIR" "$helper_dir" "$DEPLOY_DIR/backups-request" "$DEPLOY_DIR/backups-status" "${SNAPSHOT_RETAIN:-5}" backups_helper_body } > "$helper_dir/devgrail-backups-helper.tmp" chmod 700 "$helper_dir/devgrail-backups-helper.tmp" mv -f "$helper_dir/devgrail-backups-helper.tmp" "$helper_dir/devgrail-backups-helper" command -v systemctl >/dev/null 2>&1 && [ -d /run/systemd/system ] || return 0 cat > /etc/systemd/system/devgrail-backups.service <<EOF [Unit] Description=Manage DevGrail backup requests After=docker.service Requires=docker.service [Service] Type=oneshot ExecStart=$helper_dir/devgrail-backups-helper TimeoutStartSec=2h EOF cat > /etc/systemd/system/devgrail-backups.path <<EOF [Path] PathExists=$DEPLOY_DIR/backups-request/request [Install] WantedBy=paths.target EOF cat > /etc/systemd/system/devgrail-backups-report.service <<EOF [Service] Type=oneshot ExecStart=$helper_dir/devgrail-backups-helper --report EOF cat > /etc/systemd/system/devgrail-backups-report.timer <<EOF [Timer] OnBootSec=30s OnUnitActiveSec=30s [Install] WantedBy=timers.target EOF # One-time legacy indexing is off the HTTP path and keeps the page responsive. cat > /etc/systemd/system/devgrail-backups-index.service <<EOF [Unit] After=docker.service Requires=docker.service [Service] Type=oneshot ExecStart=$helper_dir/devgrail-backups-helper --index TimeoutStartSec=2h EOF systemctl daemon-reload 9>&- systemctl enable --now devgrail-backups.path devgrail-backups-report.timer 9>&- "$helper_dir/devgrail-backups-helper" --report 9>&- } install_backups_helper || warn "Could not enable backup management; check the host systemd journal." # --- scheduled backups ------------------------------------------------------- # Offered, never imposed: it writes to the customer's disk on a timer, and an # operator who already has their own backup story should not get a second one # they did not ask for. DEVGRAIL_SCHEDULED_BACKUPS=yes|no answers it unattended; # an existing timer is left exactly as configured. install_backup_timer() { command -v systemctl >/dev/null 2>&1 || return 1 cat > /etc/systemd/system/devgrail-backup.service <<EOF [Unit] Description=DevGrail weekly backup After=docker.service Requires=docker.service [Service] Type=oneshot ExecStart=$INSTALL_DIR/backups-host/devgrail-backups-helper --scheduled EOF # Randomised delay so a fleet of VPSes does not all wake at once; Persistent # so a machine that was off at the scheduled time still runs it on next boot. cat > /etc/systemd/system/devgrail-backup.timer <<EOF [Unit] Description=Weekly DevGrail backup [Timer] OnCalendar=weekly RandomizedDelaySec=1h Persistent=true [Install] WantedBy=timers.target EOF # 9>&- for the same reason as the Docker install above: whatever these start # must not inherit the install lock and outlive this run holding it. systemctl daemon-reload >/dev/null 2>&1 9>&- || true systemctl enable --now devgrail-backup.timer >/dev/null 2>&1 9>&- || return 1 return 0 } # --- weekly backup (apply) --------------------------------------------------- # Decided up in the question block; only the doing is left. The CLI has to exist # by now — the timer's ExecStart is that file. if [ "$BACKUP_TIMER_DECISION" = install ] && [ -f "$CLI_PATH" ]; then step "Scheduling the weekly backup..." if [ -f "$INSTALL_DIR/backups-host/policy.json" ] && [ -f /etc/systemd/system/devgrail-backup.timer ]; then systemctl enable --now devgrail-backup.timer 9>&- || warn "Could not enable saved backup timer" BACKUP_TIMER_NOTE="Backups: saved schedule enabled. Manage in Administration -> Backups." elif install_backup_timer; then BACKUP_TIMER_NOTE="Weekly backups: enabled (systemctl status devgrail-backup.timer)" else warn "could not enable the weekly backup timer — run '$CLI_PATH backup' yourself, or from your own scheduler" BACKUP_TIMER_NOTE="Weekly backups: could not be scheduled — run '$CLI_PATH backup' from your own scheduler." fi elif [ "$BACKUP_TIMER_DECISION" = disable ]; then systemctl disable --now devgrail-backup.timer 9>&- || true BACKUP_TIMER_NOTE="Scheduled backups: disabled. Manage in Administration -> Backups." elif [ "$BACKUP_TIMER_DECISION" = skip ] && command -v systemctl >/dev/null 2>&1; then BACKUP_TIMER_NOTE="Weekly backups: not scheduled. Enable later with: DEVGRAIL_SCHEDULED_BACKUPS=yes re-run, or run '$CLI_PATH backup' from your own scheduler." fi # --- certificate environment helper ----------------------------------------- certificates_helper_body() { cat <<'CERTHELPER' set -euo pipefail umask 077 DEPLOY_DIR="$INSTALL_DIR/deploy" WORK="$HELPER_DIR/work" mkdir -p "$WORK" chmod 700 "$HELPER_DIR" "$WORK" PRODUCTION=https://acme-v02.api.letsencrypt.org/directory STAGING=https://acme-staging-v02.api.letsencrypt.org/directory ENV_FILE="$DEPLOY_DIR/.env" LOG="$STATUS_DIR/certificates.log" TARGET=""; STARTED=""; STATE=""; CHANGED=false; INSTALL_LOCKED=false; CLI=false # Compose must read the saved CA rather than a CLI caller's exported value. unset DEVGRAIL_ACME_CASERVER DEVGRAIL_ACME_STORAGE now() { date -u +%Y-%m-%dT%H:%M:%SZ; } lock_wait() { local deadline=$((SECONDS + $2)) until flock -n "$1"; do [ "$SECONDS" -lt "$deadline" ] || return 1 sleep 1 done } compose() { local -a files=(-f "$DEPLOY_DIR/docker-compose.yml") [ ! -f "$DEPLOY_DIR/docker-compose.override.yml" ] || files+=(-f "$DEPLOY_DIR/docker-compose.override.yml") docker compose --project-directory "$DEPLOY_DIR" --env-file "$ENV_FILE" "${files[@]}" "$@" } # Parse only named values, never source .env (which also contains passwords). value() { local v v="$(sed -n "s/^$1=//p" "$ENV_FILE" | tail -n1)" v="${v%$'\r'}" case "$v" in \"*\") v="${v:1:${#v}-2}" ;; \'*\') v="${v:1:${#v}-2}" ;; esac printf '%s' "$v" } environment() { case "$1" in "$PRODUCTION") echo production ;; "$STAGING") echo staging ;; '') echo unknown ;; *) echo custom ;; esac } traefik_id() { compose ps -q traefik; } inspect_config() { local id id="$(traefik_id)"; [ -n "$id" ] || return 1 docker inspect "$id" | jq -e '.[0] | {running: .State.Running, command: .Config.Cmd, networks: .NetworkSettings.Networks}' } flag() { jq -r --arg prefix "--certificatesresolvers.letsencrypt.acme.$1=" '.command[] | select(startswith($prefix)) | ltrimstr($prefix)' | tail -n1; } report() ( exec 7>"$WORK/.report.lock"; lock_wait 7 5 || return 0 local saved active ca storage info allowed=false reason tls saved="$(value DEVGRAIL_ACME_CASERVER)"; saved="${saved:-$PRODUCTION}" active=unknown; ca=""; storage="" if info="$(inspect_config 2>/dev/null)" && printf '%s' "$info" | jq -e '.running' >/dev/null; then ca="$(printf '%s' "$info" | flag caserver)" storage="$(printf '%s' "$info" | flag storage)" active="$(environment "$ca")" fi reason="Traefik is not running with a supported ACME resolver. Check the host configuration." tls="$(value DEVGRAIL_TLS)" case "${tls,,}" in off|false|0|no) reason="TLS is off; certificate switching is unavailable." ;; *) if [ -n "$ca" ] && [ -n "$storage" ]; then allowed=true; reason=""; fi ;; esac jq -n --arg saved "$(environment "$saved")" --arg active "$active" --arg ca "$ca" \ --arg at "$(now)" --argjson allowed "$allowed" --arg reason "$reason" \ '{saved_environment:$saved, active_environment:$active, ca_server:$ca, observed_at:$at, change_allowed:$allowed, reason:$reason}' \ > "$STATUS_DIR/.configuration.json.tmp" chmod 644 "$STATUS_DIR/.configuration.json.tmp" mv -f "$STATUS_DIR/.configuration.json.tmp" "$STATUS_DIR/configuration.json" ) write_status() { local finished="" STATE="$1"; [ "$STATE" = running ] || finished="$(now)" jq -n --arg state "$STATE" --arg target "$TARGET" --arg started "$STARTED" \ --arg finished "$finished" --arg message "$2" \ '{protocol:1, state:$state, target_environment:$target, started_at:$started, finished_at:$finished, message:$message}' \ > "$STATUS_DIR/.status.json.tmp" chmod 644 "$STATUS_DIR/.status.json.tmp" mv -f "$STATUS_DIR/.status.json.tmp" "$STATUS_DIR/status.json" if $CLI; then printf '%s\n' "$2"; fi } log() { printf '[%s] %s\n' "$(now)" "$*" >> "$LOG"; } reject() { write_status rejected "$1"; exit 0; } # Every dynamically attached network is lost on recreate. Keep its aliases in # root-only work, so both apply and rollback restore in-container MCP routing. restore_networks() { local id net alias workspace_ids ws local -a args id="$(traefik_id)"; [ -n "$id" ] || return 1 # Include workspaces created during recreation and recover when an earlier # failure removed Traefik's original network endpoints entirely. workspace_ids="$(docker ps -aq --filter label=devgrail.managed=true)" || return 1 for ws in $workspace_ids; do net="$(docker inspect "$ws" | jq -r '.[0].Config.Labels["traefik.docker.network"] // empty')" || continue case "$net" in devgrail-ws-*) ;; *) continue ;; esac docker network inspect "$net" | jq -e '.[0].Labels["devgrail.managed"] == "true"' >/dev/null || continue jq --arg n "$net" 'if has($n) then . else . + {($n):{Aliases:["traefik"]}} end' "$WORK/networks.json" > "$WORK/.networks.tmp" || return 1 mv -f "$WORK/.networks.tmp" "$WORK/networks.json" done while IFS= read -r net; do docker network inspect "$net" >/dev/null 2>&1 || continue if docker inspect "$id" | jq -e --arg n "$net" '.[0].NetworkSettings.Networks | has($n)' >/dev/null; then continue; fi args=() while IFS= read -r alias; do args+=(--alias "$alias"); done < <(jq -r --arg n "$net" '.[$n].Aliases[]? // empty' "$WORK/networks.json") docker network connect "${args[@]}" "$net" "$id" || return 1 done < <(jq -r 'keys[]' "$WORK/networks.json") } recreate() { compose up -d --no-deps --force-recreate traefik >> "$LOG" 2>&1 || return 1 restore_networks >> "$LOG" 2>&1 } rollback() { # Copy to the same directory and rename, preserving the original file mode. cp -p "$WORK/env.before" "$DEPLOY_DIR/.env.certificates.tmp" && mv -f "$DEPLOY_DIR/.env.certificates.tmp" "$ENV_FILE" && recreate } on_exit() { local rc=$? trap - EXIT TERM INT if $CHANGED; then if rollback; then CHANGED=false rm -f "$WORK/env.before" write_status failed "The change failed. The previous configuration and workspace network attachments were restored. Check the log before retrying." || true else write_status failed "The change failed and recovery did not finish. Through SSH run sudo devgrail certificates production, then check sudo journalctl -u devgrail-certificates.service." || true fi elif [ "$STATE" = running ]; then write_status failed "The operation stopped before applying the change. Check sudo journalctl -u devgrail-certificates.service." || true fi report 2>/dev/null || true if $INSTALL_LOCKED; then rm -f "$INSTALL_DIR/.install.owner"; fi exit "$rc" } if [ "${1:-}" = --report ] && [ "$#" -eq 1 ]; then report; exit 0 fi exec 8>"$WORK/.lock" lock_wait 8 5 || exit 1 trap on_exit EXIT trap 'exit 1' TERM INT STARTED="$(now)" if [ "${1:-}" = --apply ] && [ "$#" -eq 2 ]; then CLI=true # Invoked by the root maintenance CLI, also works without systemd. TARGET="$2" # SSH recovery supersedes a queued staging request. Otherwise the watcher # could undo recovery immediately after this command releases its lock. rm -rf -- "$REQUEST_DIR/request" elif [ "$#" -eq 0 ]; then # Move untrusted input into root-only work before examining it. Never follow # symlinks or read a FIFO planted in the server-writable request directory. rm -rf -- "$WORK/request" mv -fT -- "$REQUEST_DIR/request" "$WORK/request" 2>/dev/null || exit 0 [ ! -L "$WORK/request" ] && [ -f "$WORK/request" ] || reject "The request was not a plain file." [ "$(stat -c %s "$WORK/request")" -le 32 ] || reject "The request was too large." TARGET="$(cat "$WORK/request")" rm -f "$WORK/request" else reject "Use devgrail certificates production or devgrail certificates staging." fi case "$TARGET" in production) CA="$PRODUCTION"; STORE=acme.json ;; staging) CA="$STAGING"; STORE=acme-staging.json ;; *) TARGET=""; reject "The environment must be production or staging." ;; esac write_status running "Waiting for any installation or upgrade to finish." # Participate in the installer's lock AND owner record. Its leaked-lock # recovery must see this helper as a live owner, or it could unlink our lock. exec 9>>"$INSTALL_DIR/.install.lock" lock_wait 9 300 || { write_status failed "An installation or upgrade still holds the host lock. Try again when it finishes."; exit 1; } INSTALL_LOCKED=true printf '%s\n' "$$" > "$INSTALL_DIR/.install.owner" : > "$LOG"; chmod 640 "$LOG" case "$(value DEVGRAIL_TLS | tr '[:upper:]' '[:lower:]')" in off|false|0|no) reject "TLS is off; the configuration was left unchanged." ;; esac # A killed helper leaves its journal. Recover it before another operation, # including after a host reboot, instead of overwriting the only backup. if [ -f "$WORK/env.before" ]; then log "Recovering an interrupted certificate change." rollback || { write_status failed "Interrupted change recovery failed. Check the host logs and retained backup."; exit 1; } rm -f "$WORK/env.before" fi if info="$(inspect_config 2>/dev/null)"; then printf '%s' "$info" | jq '.networks' > "$WORK/networks.json" elif [ ! -f "$WORK/networks.json" ]; then printf '{}\n' > "$WORK/networks.json" fi cp -p "$ENV_FILE" "$WORK/env.before" # Preserve all unrelated entries (including passwords), comments and permissions. cp -p "$ENV_FILE" "$DEPLOY_DIR/.env.certificates.tmp" awk '!/^(DEVGRAIL_ACME_CASERVER|DEVGRAIL_ACME_STORAGE)=/' "$ENV_FILE" > "$DEPLOY_DIR/.env.certificates.tmp" printf '\nDEVGRAIL_ACME_CASERVER=%s\nDEVGRAIL_ACME_STORAGE=%s\n' "$CA" "$STORE" >> "$DEPLOY_DIR/.env.certificates.tmp" mv -f "$DEPLOY_DIR/.env.certificates.tmp" "$ENV_FILE" CHANGED=true # Inspect the effective Compose command before causing downtime. Overrides # that hard-code a resolver must not make this form appear to work silently. effective="$(compose config --format json)" printf '%s' "$effective" | jq -e --arg ca "--certificatesresolvers.letsencrypt.acme.caserver=$CA" \ --arg store "--certificatesresolvers.letsencrypt.acme.storage=/acme/$STORE" \ '.services.traefik.command | index($ca) != null and index($store) != null' >/dev/null || { # No container has changed yet; restore .env without recreating Traefik. cp -p "$WORK/env.before" "$ENV_FILE"; CHANGED=false; rm -f "$WORK/env.before" reject "A Compose override prevents changing the ACME resolver. The previous setting was preserved." } write_status running "Applying the certificate environment and restoring workspace connections." log "Switching to $TARGET, preserving both ACME stores." recreate || exit 1 info="$(inspect_config)" [ "$(printf '%s' "$info" | flag caserver)" = "$CA" ] && [ "$(printf '%s' "$info" | flag storage)" = "/acme/$STORE" ] || exit 1 DOMAIN="$(value DEVGRAIL_DOMAIN)" case "$DOMAIN" in ''|*[!a-zA-Z0-9.-]*) exit 1 ;; esac PORT="$(value HTTPS_PORT)"; PORT="${PORT:-443}" case "$PORT" in *[!0-9]*|'') exit 1 ;; esac write_status running "Configuration applied. Checking dashboard connectivity and its certificate (up to 90 seconds)." verified=false deadline=$((SECONDS + 90)) while [ "$SECONDS" -lt "$deadline" ]; do # Trust staging only in this narrow connectivity probe; the production probe # requires a publicly trusted certificate. Domain/SNI still match the dashboard. curl_args=() [ "$TARGET" != staging ] || curl_args+=(-k) if curl "${curl_args[@]}" -fsS --max-time 2 --resolve "$DOMAIN:$PORT:127.0.0.1" \ "https://$DOMAIN:$PORT/api/v1/healthz" >/dev/null 2>&1; then if [ "$TARGET" = production ] || timeout 3 openssl s_client -connect "127.0.0.1:$PORT" -servername "$DOMAIN" </dev/null 2>/dev/null \ | openssl x509 -noout -issuer 2>/dev/null | grep -qi '(STAGING)'; then verified=true; break; fi fi sleep 1 done $verified || exit 1 CHANGED=false rm -f "$WORK/env.before" write_status succeeded "The $TARGET configuration is active and the dashboard is reachable. Other domains may still be waiting for certificates; check the Traefik log if they do not serve." log "Configuration applied and dashboard connectivity verified." CERTHELPER } # Independent of dashboard upgrades. CLI recovery is available without systemd; # only the web request watcher and fresh configuration reports need systemd. CERTIFICATES_HELPER_DIR="$INSTALL_DIR/certificates" CERTIFICATES_HELPER="$CERTIFICATES_HELPER_DIR/devgrail-certificates-helper" CERTIFICATES_REQUEST_DIR="$DEPLOY_DIR/certificates-request" CERTIFICATES_STATUS_DIR="$DEPLOY_DIR/certificates-status" mkdir -p "$CERTIFICATES_HELPER_DIR" "$CERTIFICATES_REQUEST_DIR" "$CERTIFICATES_STATUS_DIR" chmod 700 "$CERTIFICATES_HELPER_DIR" "$CERTIFICATES_REQUEST_DIR" chmod 755 "$CERTIFICATES_STATUS_DIR" # A manual installer run supersedes pending requests and interrupted journals: # its freshly written .env is now the operator's authoritative choice. rm -rf -- "$CERTIFICATES_REQUEST_DIR/request" rm -f "$CERTIFICATES_HELPER_DIR/work/env.before" "$CERTIFICATES_HELPER_DIR/work/networks.json" rm -f "$CERTIFICATES_STATUS_DIR/status.json" { printf '#!/usr/bin/env bash\nINSTALL_DIR=%q\nHELPER_DIR=%q\nREQUEST_DIR=%q\nSTATUS_DIR=%q\n' \ "$INSTALL_DIR" "$CERTIFICATES_HELPER_DIR" "$CERTIFICATES_REQUEST_DIR" "$CERTIFICATES_STATUS_DIR" certificates_helper_body } > "$CERTIFICATES_HELPER.tmp" chmod 700 "$CERTIFICATES_HELPER.tmp" mv -f "$CERTIFICATES_HELPER.tmp" "$CERTIFICATES_HELPER" rm -f "$CERTIFICATES_STATUS_DIR/helper.json" if command -v systemctl >/dev/null 2>&1 && [ -d /run/systemd/system ]; then cat > /etc/systemd/system/devgrail-certificates.service <<EOF [Unit] Description=Apply DevGrail certificate environment After=docker.service [Service] Type=oneshot ExecStart=$CERTIFICATES_HELPER Environment=HOME=/root TimeoutStartSec=10min EOF cat > /etc/systemd/system/devgrail-certificates.path <<EOF [Unit] Description=Watch DevGrail certificate requests [Path] PathExists=$CERTIFICATES_REQUEST_DIR/request [Install] WantedBy=paths.target EOF cat > /etc/systemd/system/devgrail-certificates-report.service <<EOF [Unit] Description=Report DevGrail certificate configuration After=docker.service [Service] Type=oneshot ExecStart=$CERTIFICATES_HELPER --report Environment=HOME=/root TimeoutStartSec=20s EOF cat > /etc/systemd/system/devgrail-certificates-report.timer <<EOF [Unit] Description=Refresh DevGrail certificate configuration [Timer] OnBootSec=10s OnUnitActiveSec=30s AccuracySec=1s [Install] WantedBy=timers.target EOF if systemctl daemon-reload 9>&- && systemctl enable --now devgrail-certificates.path devgrail-certificates-report.timer 9>&-; then printf '{"protocol":1}\n' > "$CERTIFICATES_STATUS_DIR/helper.json.tmp" chmod 644 "$CERTIFICATES_STATUS_DIR/helper.json.tmp" mv -f "$CERTIFICATES_STATUS_DIR/helper.json.tmp" "$CERTIFICATES_STATUS_DIR/helper.json" else warn "Could not enable certificate changes from Administration. SSH recovery remains available: sudo devgrail certificates production" fi fi # --- upgrade helper ---------------------------------------------------------- # What lets an admin upgrade DevGrail from the dashboard — without the # dashboard running anything on this host. # # devgrail-server can only leave a one-line request in $UPGRADE_REQUEST_DIR. A # systemd path unit notices it and runs the helper written below, as root. The # helper decides everything the request does not say: it accepts only "upgrade # <newer release>", verifies the release signature and the new installer's # digest against keys this installer wrote, and runs that installer unattended # with --auto-rollback. It reports back through # $UPGRADE_STATUS_DIR, which the server mounts read-only. # # The whole design assumes devgrail-server may be compromised, so the rules that # make it safe are properties of the files, not of the code that writes them — # docs/SECURITY.md, "Upgrading from the dashboard", lists them. In short: the # helper and its keys live outside every container mount; the status directory # is read-only to the server by mount flag (the server is host uid 0, so file # ownership would not stop it); and the request directory is the only place the # server can write, and holds nothing the installer reads. UPGRADE_HELPER_DIR="$INSTALL_DIR/upgrade" UPGRADE_HELPER="$UPGRADE_HELPER_DIR/devgrail-upgrade-helper" UPGRADE_REQUEST_DIR="$DEPLOY_DIR/upgrade-request" UPGRADE_STATUS_DIR="$DEPLOY_DIR/upgrade-status" UPGRADE_HELPER_NOTE="" # upgrade_helper_body — the fixed part of the helper. Quoted heredoc: nothing in # it is expanded here, so what is on this page is exactly what runs. The # install-specific constants are printed above it by write_upgrade_helper. # # Extracted by test/installer/upgrade_helper_test.sh from this marker to the # HELPER terminator — keep the function a single heredoc. upgrade_helper_body() { cat <<'HELPER' # Everything above this line describes the install it was written for; # everything below is fixed. It runs as root under devgrail-upgrade.service # whenever devgrail-server leaves a request, and it treats that request as # hostile: devgrail-server is the process this design assumes may be compromised. # # The request says exactly one thing, and nothing else is read from it: # upgrade vX.Y.Z install that release, which must be newer than this one # Where to download from, which keys to trust and which flags to pass are all # decided here, from files only root can write. set -euo pipefail umask 077 WORK="$HELPER_DIR/work" KEYS="$HELPER_DIR/release-keys.pem" EXTRA_KEYS="$HELPER_DIR/extra-release-keys.pem" VERSIONS_FILE="$INSTALL_DIR/versions.env" STATUS="$STATUS_DIR/status.json" LOG="$STATUS_DIR/upgrade.log" REQ="$WORK/request" RUN="$WORK/run" MAX_REQUEST_BYTES=64 RELEASE_RE='^v[0-9]{1,4}\.[0-9]{1,4}\.[0-9]{1,6}$' ACTION=""; TARGET=""; FROM=""; STARTED=""; STATE="" now() { date -u +%Y-%m-%dT%H:%M:%SZ; } log() { printf '[%s] %s\n' "$(now)" "$*" >> "$LOG"; } # write_status <state> <message> — replace status.json in one rename. Writing # here as root is safe because the server mounts this directory read-only: # nothing in it can have been planted by the process this helper distrusts. write_status() { local finished="" STATE="$1" [ "$1" = running ] || finished="$(now)" jq -n --arg state "$1" --arg message "$2" --arg action "$ACTION" \ --arg target "$TARGET" --arg from "$FROM" --arg started "$STARTED" \ --arg finished "$finished" \ '{protocol: 1, state: $state, action: $action, target_version: $target, from_version: $from, started_at: $started, finished_at: $finished, message: $message}' > "$STATUS_DIR/.status.json.tmp" chmod 644 "$STATUS_DIR/.status.json.tmp" mv -f "$STATUS_DIR/.status.json.tmp" "$STATUS" } # reject <message> — refuse the request. Exits 0: refusing is this helper # working, and must not count towards the unit's start limit. reject() { write_status rejected "$1"; exit 0; } # fail <message> — a valid request that could not be carried out. fail() { log "$1"; write_status failed "$1"; exit 1; } # An exit nobody planned for (set -e) must not leave "running" on screen. on_exit() { local rc=$? if [ "$STATE" = running ]; then write_status failed "The upgrade helper stopped unexpectedly (exit $rc). See the log." || true fi rm -rf -- "$RUN" "$REQ" 2>/dev/null || true } # release_gt <a> <b> — true when release a is newer than b. Both have already # matched RELEASE_RE; 10# stops a component like "08" being read as octal. release_gt() { local -a a b local i IFS=. read -r -a a <<< "${1#v}" IFS=. read -r -a b <<< "${2#v}" for i in 0 1 2; do if (( 10#${a[i]} > 10#${b[i]} )); then return 0; fi if (( 10#${a[i]} < 10#${b[i]} )); then return 1; fi done return 1 } mkdir -p "$WORK" "$STATUS_DIR" chmod 700 "$HELPER_DIR" "$WORK" # One run at a time. A trigger that finds the lock held leaves the request where # it is; the path unit fires again once this run ends and the file still exists. exec 8>"$WORK/.lock" flock -n 8 || exit 0 trap on_exit EXIT # Take the request out of devgrail-server's reach before looking at it. rename(2) # moves a symlink or a FIFO as itself, never what it points at, so nothing the # checks below examine can be swapped underneath them. rm -rf -- "$REQ" mv -fT -- "$REQUEST_DIR/request" "$REQ" 2>/dev/null || exit 0 STARTED="$(now)" if [ -L "$REQ" ] || [ ! -f "$REQ" ]; then reject "The request was not a plain file." fi if [ "$(stat -c %s -- "$REQ")" -gt "$MAX_REQUEST_BYTES" ]; then reject "The request was larger than $MAX_REQUEST_BYTES bytes." fi # Command substitution drops trailing newlines, so a request with or without a # final newline reads alike; a line break or carriage return left inside it # after that means more than one line, which is refused rather than truncated. line="$(cat -- "$REQ" 2>/dev/null)" || true case "$line" in *$'\n'*|*$'\r'*) reject "The request had more than one line." ;; esac # The requested version is not echoed back anywhere until it has matched. case "$line" in "upgrade "*) ACTION=upgrade; TARGET="${line#upgrade }" ;; *) reject "The request was not one this helper understands." ;; esac if ! [[ $TARGET =~ $RELEASE_RE ]]; then TARGET="" reject "The requested version is not a release version (vMAJOR.MINOR.PATCH)." fi # The installed release, from the file install.sh writes. Parsed, not sourced. if [ -f "$VERSIONS_FILE" ]; then FROM="$(sed -n 's/^RELEASE_VERSION=//p' "$VERSIONS_FILE")" FROM="${FROM%%$'\n'*}" fi if ! [[ $FROM =~ $RELEASE_RE ]]; then FROM="" reject "Cannot tell which release is installed here. Upgrade once from the command line." fi # Newer only. An older release is signed too, and a compromised server asking # for one is asking to reopen whatever that release had wrong. release_gt "$TARGET" "$FROM" \ || reject "$TARGET is not newer than the installed $FROM. Only upgrades can be requested from the dashboard; --version on the command line does the rest." : > "$LOG" chmod 640 "$LOG" write_status running "Verifying release $TARGET." log "Request: $ACTION (installed $FROM, target $TARGET)" rm -rf -- "$RUN" mkdir -m 700 "$RUN" # --- verify the release, then the installer it ships --------------------------- # The same checks install.sh makes of every artifact, made here of install.sh # itself — which a `curl | sudo bash` upgrade never checks at all. curl -fsSL --max-time 60 --max-filesize 5242880 \ "$WEB/api/registry/releases/$TARGET" -o "$RUN/manifest.json" \ || fail "Could not fetch release $TARGET from $WEB." [ "$(jq -r '.release_version // empty' "$RUN/manifest.json")" = "$TARGET" ] \ || fail "The registry answered with a different release than $TARGET." [ "$(jq -r '.signature.algorithm // empty' "$RUN/manifest.json")" = "ecdsa-p256-sha256" ] \ || fail "Release $TARGET is unsigned, or signed with a scheme this helper cannot verify." jq -j '.signature.payload' "$RUN/manifest.json" > "$RUN/payload" \ || fail "Release $TARGET has no signed payload." jq -r '.signature.signature' "$RUN/manifest.json" | base64 -d > "$RUN/sig" 2>/dev/null \ || fail "The signature on release $TARGET is not valid base64." awk -v d="$RUN" '/-----BEGIN PUBLIC KEY-----/{n++} n>0 {print > (d "/key" n ".pem")}' "$KEYS" verified=false for pem in "$RUN"/key*.pem; do [ -f "$pem" ] || continue if openssl dgst -sha256 -verify "$pem" -signature "$RUN/sig" "$RUN/payload" >/dev/null 2>&1; then verified=true break fi done [ "$verified" = true ] \ || fail "The signature on release $TARGET does not verify against any key this host trusts." grep -qxF "devgrail-release-signature-v1" "$RUN/payload" \ || fail "The signed payload of release $TARGET is not in a format this helper understands." grep -qxF "release $TARGET" "$RUN/payload" \ || fail "The signature was made over a different release than $TARGET." asset() { jq -r --arg f "$1" '[.assets[]? | select(.slug == "install.sh")][0][$f] // empty' "$RUN/manifest.json"; } ver="$(asset asset_version)" url="$(asset download_url)" sha="$(asset sha256)" if ! [[ $ver =~ ^[0-9]+$ ]] || ! [[ $sha =~ ^[0-9a-f]{64}$ ]]; then fail "Release $TARGET does not publish an installer this helper can verify." fi case "$url" in /*) url="$WEB$url" ;; https://*|http://*) ;; *) fail "Release $TARGET names an installer URL this helper will not fetch." ;; esac curl -fsSL --max-time 600 --max-filesize 10485760 "$url" -o "$RUN/install.sh" \ || fail "Could not download the $TARGET installer." got="$(sha256sum "$RUN/install.sh")" got="${got%% *}" [ "$got" = "$sha" ] \ || fail "The $TARGET installer does not match its published checksum." grep -qxF "asset install.sh $ver $got" "$RUN/payload" \ || fail "The $TARGET installer is not the one release $TARGET was signed for." log "Release $TARGET and its installer verified." # --- run it ------------------------------------------------------------------ # Unattended, so every question takes its prior answer. The three daemon and # timer questions are answered no outright: --yes would otherwise take the # backup prompt's default of yes, and an upgrade nobody watched must not change # more than the release. --auto-rollback because nobody is at a shell to answer. args=(--version="$TARGET" --yes --auto-rollback) write_status running "Upgrading from $FROM to $TARGET." envs=(DEVGRAIL_WEB_URL="$WEB" DEVGRAIL_INSTALL_DIR="$INSTALL_DIR" DEVGRAIL_USERNS_REMAP=no DEVGRAIL_ADDRESS_POOLS=no DEVGRAIL_SCHEDULED_BACKUPS=no) if [ -s "$EXTRA_KEYS" ]; then envs+=(DEVGRAIL_RELEASE_PUBKEY_FILE="$EXTRA_KEYS") fi log "Running the $TARGET installer: ${args[*]}" set +e env "${envs[@]}" bash "$RUN/install.sh" "${args[@]}" < /dev/null >> "$LOG" 2>&1 8>&- rc=$? set -e if [ "$rc" -ne 0 ]; then fail "The installer exited with status $rc. An upgrade that does not come up healthy is rolled back automatically; the log has the details." fi write_status succeeded "Upgraded from $FROM to $TARGET." HELPER } # write_upgrade_helper — the helper, its trusted keys, and the two units. write_upgrade_helper() { mkdir -p "$UPGRADE_HELPER_DIR" || return 1 chmod 700 "$UPGRADE_HELPER_DIR" || return 1 # Through a temp file and a rename: this may be the run the helper itself # started, and bash reads a script as it goes — overwriting it in place would # change the lines it has yet to reach. A rename leaves it the old inode. { printf '#!/usr/bin/env bash\n' printf '# DevGrail upgrade helper, written by install.sh. Do not edit: re-running the\n' printf '# installer overwrites it. See docs/SECURITY.md, "Upgrading from the dashboard".\n' printf 'INSTALL_DIR=%q\n' "$INSTALL_DIR" printf 'WEB=%q\n' "$WEB" printf 'HELPER_DIR=%q\n' "$UPGRADE_HELPER_DIR" printf 'REQUEST_DIR=%q\n' "$UPGRADE_REQUEST_DIR" printf 'STATUS_DIR=%q\n' "$UPGRADE_STATUS_DIR" upgrade_helper_body } > "$UPGRADE_HELPER.tmp" || return 1 chmod 700 "$UPGRADE_HELPER.tmp" && mv -f "$UPGRADE_HELPER.tmp" "$UPGRADE_HELPER" || return 1 # Two files, not one: the helper verifies with both, but passes on only the # extras. Handing the full set to the next installer would make it write # every key it was given back out, and a key rotated out of the embedded set # would never actually leave. release_public_keys > "$UPGRADE_HELPER_DIR/release-keys.pem.tmp" \ && mv -f "$UPGRADE_HELPER_DIR/release-keys.pem.tmp" "$UPGRADE_HELPER_DIR/release-keys.pem" || return 1 release_extra_public_keys > "$UPGRADE_HELPER_DIR/extra-release-keys.pem.tmp" \ && mv -f "$UPGRADE_HELPER_DIR/extra-release-keys.pem.tmp" "$UPGRADE_HELPER_DIR/extra-release-keys.pem" || return 1 chmod 600 "$UPGRADE_HELPER_DIR/release-keys.pem" "$UPGRADE_HELPER_DIR/extra-release-keys.pem" # HOME because systemd sets none for a root service and the docker CLI wants # one. Two hours bounds a run that hangs; the server reports a run silent for # longer than that as interrupted. The start limit stops a request that the # helper somehow cannot consume from re-triggering it in a tight loop. cat > /etc/systemd/system/devgrail-upgrade.service <<EOF [Unit] Description=DevGrail upgrade requested from the dashboard After=docker.service network-online.target Wants=network-online.target StartLimitIntervalSec=300 StartLimitBurst=5 [Service] Type=oneshot ExecStart=$UPGRADE_HELPER Environment=HOME=/root TimeoutStartSec=2h EOF cat > /etc/systemd/system/devgrail-upgrade.path <<EOF [Unit] Description=Watch for DevGrail upgrade requests from the dashboard [Path] PathExists=$UPGRADE_REQUEST_DIR/request [Install] WantedBy=paths.target EOF return 0 } # remove_upgrade_helper — switch dashboard upgrades off and say so to the # server, which reads helper.json to decide whether to offer them. remove_upgrade_helper() { rm -f "$UPGRADE_STATUS_DIR/helper.json" if command -v systemctl >/dev/null 2>&1; then systemctl disable --now devgrail-upgrade.path >/dev/null 2>&1 9>&- || true fi rm -f /etc/systemd/system/devgrail-upgrade.path /etc/systemd/system/devgrail-upgrade.service rm -rf "$UPGRADE_HELPER_DIR" } # Both directories are created whatever the answer below: the compose file # mounts them, and a mount Docker has to create itself is a root-owned 0755 # directory, not the modes chosen here. 0700 on the request directory is for # everyone else on the host; the server is uid 0 and writes it regardless. mkdir -p "$UPGRADE_REQUEST_DIR" "$UPGRADE_STATUS_DIR" chmod 700 "$UPGRADE_REQUEST_DIR" chmod 755 "$UPGRADE_STATUS_DIR" # A request left over from before this run is superseded by it: whatever it # asked for, the operator has just done something else by hand. rm -rf -- "$UPGRADE_REQUEST_DIR/request" if [ "$DEVGRAIL_DASHBOARD_UPGRADES" != yes ]; then remove_upgrade_helper UPGRADE_HELPER_NOTE="Dashboard upgrades: off (DEVGRAIL_DASHBOARD_UPGRADES=no)." elif ! command -v systemctl >/dev/null 2>&1 || [ ! -d /run/systemd/system ]; then rm -f "$UPGRADE_STATUS_DIR/helper.json" UPGRADE_HELPER_NOTE="Dashboard upgrades: unavailable without systemd — upgrade with the command above." else log "Enabling upgrades from the dashboard..." if write_upgrade_helper && systemctl daemon-reload >/dev/null 2>&1 9>&- && systemctl enable --now devgrail-upgrade.path >/dev/null 2>&1 9>&-; then printf '{"protocol":1}\n' > "$UPGRADE_STATUS_DIR/helper.json.tmp" && chmod 644 "$UPGRADE_STATUS_DIR/helper.json.tmp" && mv -f "$UPGRADE_STATUS_DIR/helper.json.tmp" "$UPGRADE_STATUS_DIR/helper.json" UPGRADE_HELPER_NOTE="Dashboard upgrades: on (systemctl status devgrail-upgrade.path). Turn off with --dashboard-upgrades=no." else rm -f "$UPGRADE_STATUS_DIR/helper.json" warn "could not enable upgrades from the dashboard — upgrade by re-running this installer" UPGRADE_HELPER_NOTE="Dashboard upgrades: could not be enabled — upgrade with the command above." fi fi # --- bring up -------------------------------------------------------------- docker network inspect devgrail >/dev/null 2>&1 || docker network create devgrail >/dev/null step "Starting DevGrail..." # Via compose(), so a docker-compose.override.yml is honoured here too — not # just by the stop/logs calls above. An override that only applied to some of # the installer's compose invocations would be worse than none. compose up -d # --- wait for readiness ----------------------------------------------------- # probe_health / wait_for_health are defined near the top, because --rollback # needs them too. if ! wait_for_health; then echo >&2 printf '\033[1;31merror:\033[0m %s\n' "DevGrail did not become ready within ${HEALTH_TIMEOUT}s." >&2 echo " Last 50 lines from devgrail-server:" >&2 compose logs --tail=50 devgrail-server >&2 || true echo >&2 echo " Health detail:" >&2 curl -sSk --max-time 5 "$(dashboard_url)/api/v1/healthz" >&2 || true echo >&2 # Automatic rollback, on an upgrade only. A first install has nothing to roll # back to: there are no :previous images and no snapshot, and "rolling back" to # an empty host would replace a diagnosable failure with a confusing one. ROLLED_BACK=false if [ "$IS_UPGRADE" = true ]; then # --yes deliberately does not imply yes here. Rolling back is a decision # about which of two states the host ends in, not a question in the way of # an unattended run; --auto-rollback is how that one is answered up front. DO_IT="$AUTO_ROLLBACK" if [ "$DO_IT" != true ] && [ "$ASSUME_YES" != true ] && have_tty; then # ask_yn, so an answer that is neither yes nor no is asked again here too. if ask_yn "Roll back to the previous version and snapshot?" n; then DO_IT=true fi fi if [ "$DO_IT" = true ]; then echo >&2 if do_rollback; then ROLLED_BACK=true else printf '\033[1;31merror:\033[0m %s\n' "The rolled-back stack did not become ready either." >&2 fi fi fi # Say plainly which state the host ended in. After a failed upgrade that is the # only thing the operator needs to know before deciding what to do next. if [ "$ROLLED_BACK" = true ]; then cat >&2 <<EOF ============================================================ The upgrade failed and was ROLLED BACK. This host is running the previous version again and is answering its readiness probe at $(dashboard_url)/ Nothing was lost. Report the log above before retrying the upgrade. ============================================================ EOF exit 1 fi if [ -n "$SNAPSHOT_ARCHIVE" ]; then echo " A pre-upgrade snapshot of the database and config.yaml was taken first:" >&2 echo " $SNAPSHOT_ARCHIVE" >&2 echo " Nothing was deleted, but the new server may already have migrated the" >&2 echo " database — that archive is the way back to the previous state." >&2 echo " To go back: curl -fsSL $WEB/install.sh | sudo bash -s -- --rollback" >&2 fi echo " Logs: docker compose --project-directory $DEPLOY_DIR logs -f" >&2 exit 1 fi # Publish a fresh observation after recreate, so Administration and smoke checks # do not temporarily show the previous Traefik container's environment. "$CERTIFICATES_HELPER" --report 9>&- || warn "Could not refresh the certificate report; check the host report timer." "$INSTALL_DIR/backups-host/devgrail-backups-helper" --report 9>&- || true if command -v systemctl >/dev/null 2>&1 && [ -d /run/systemd/system ]; then systemctl start --no-block devgrail-backups-index.service 9>&- || true fi # --- reclaim superseded images ---------------------------------------------- # Only now, and never earlier: :previous is the rollback target, and until the # new stack has answered its readiness probe it is the thing standing between a # bad upgrade and a broken host. Once health is green, every DevGrail image older # than :previous is dead weight — an upgrade loads a new one each time and # nothing else removes them, so a host that upgrades for a year accumulates a # year of unreferenced layers on the disk whose exhaustion causes the failure # this whole sequence exists to survive. reclaim_old_images() { local slug keep_latest keep_prev id removed=0 for slug in devgrail-server devgrail-container; do keep_latest="$(docker image inspect -f '{{.Id}}' "$slug:latest" 2>/dev/null || true)" keep_prev="$(docker image inspect -f '{{.Id}}' "$slug:previous" 2>/dev/null || true)" [ -n "$keep_latest" ] || continue # Every image ID this repository still has, tagged or dangling. Untagged # ones are the displaced builds: `docker load` + retag leaves them with no # reference at all, which is exactly why nothing ever cleaned them up. for id in $(docker images --filter "reference=$slug" --format '{{.ID}}' 2>/dev/null | sort -u); do id="$(docker image inspect -f '{{.Id}}' "$id" 2>/dev/null || true)" [ -n "$id" ] || continue [ "$id" = "$keep_latest" ] && continue [ -n "$keep_prev" ] && [ "$id" = "$keep_prev" ] && continue # Not forced: an image a container is still using must stay, and that is # a refusal to respect rather than override. A workspace created before an # upgrade is running the old devgrail-container image until it is # recreated — removing it out from under the container is not reclamation, # it is breakage. if docker image rm "$id" >/dev/null 2>&1; then removed=$((removed + 1)) fi done done [ "$removed" -gt 0 ] && log "Reclaimed $removed superseded image(s); :latest and :previous kept." return 0 } # Called at the very bottom, after the summary has been printed — see there. # --workspace-image=no means the banner has to be able to say "there is no # workspace image" — and then say what to do about it, because a dashboard that # cannot create a workspace is not an obviously deliberate state three weeks # later. Asked of Docker rather than of the flag: an operator who passed `no` on # a host that already had the image is not missing anything. if docker image inspect devgrail-container:latest >/dev/null 2>&1; then WORKSPACE_VERSION_NOTE="workspace ${NEW_CONTAINER_RELEASE:-unknown}" WORKSPACE_NOTE="" else WORKSPACE_VERSION_NOTE="workspace image: NOT INSTALLED" WORKSPACE_NOTE="Workspace: no workspace image on this host, so no workspace can be created yet (~750 MiB). Everything else works. Fetch it with: curl -fsSL $WEB/install.sh | sudo bash -s -- --workspace-image=yes" fi if [ "$TLS_ENABLED" = true ] && [ "$ACME_CA_IS_PRODUCTION" = false ]; then TLS_NOTE="Certificates come from a NON-PRODUCTION ACME CA ($DEVGRAIL_ACME_CASERVER), so browsers will show this site as untrusted — that is expected, and this install is for testing. Re-run with --acme-ca=production for real certificates. Ensure DNS is pointed here:" elif [ "$TLS_ENABLED" = true ]; then TLS_NOTE="It may take a minute for Let's Encrypt to issue the TLS certificate on first start. Ensure DNS is pointed here:" else TLS_NOTE="Serving plain HTTP (DEVGRAIL_TLS=off) — do not expose this directly to the public internet; run it on a private network or behind a TLS terminator. Ensure DNS is pointed here:" fi # The reference card. It used to be eleven items in the closing banner, mixed in # with the two an operator needs in the next sixty seconds — so the URL and the # password, the only things that cannot be looked up again, were the hardest to # find. This is the half that keeps: it goes to a file, which is also where it # is actually wanted, three weeks later, by somebody who has long since closed # that terminal. Written on every run, so it never describes a previous install. REFERENCE_FILE="$INSTALL_DIR/README-install.txt" write_reference_file() { cat > "$REFERENCE_FILE.tmp" <<EOF DevGrail — installed reference ============================== Written by install.sh on $(date -u '+%Y-%m-%d %H:%M:%S UTC'). Rewritten every run. Dashboard $(dashboard_url)/ Version ${NEW_SERVER_RELEASE:-unknown} ($WORKSPACE_VERSION_NOTE) Admin user $ADMIN_USER Registry $WEB DNS $(dns_records " ") Additional base domains are managed under Settings -> App subdomains in the dashboard, and each needs its own wildcard A record. Day to day Logs docker compose --project-directory $DEPLOY_DIR logs -f Backups UI $(dashboard_url)/admin/backups Backup sudo devgrail backup Restore sudo devgrail restore <archive> List sudo devgrail list Password change it under Settings in the dashboard, or: sudo devgrail passwd Accounts Users in the dashboard, or: sudo devgrail list-users sudo devgrail add-user <name> [--admin] ${BACKUP_TIMER_NOTE:-Weekly backups: not configured.} Upgrading Upgrade curl -fsSL $WEB/install.sh | sudo bash Preview curl -fsSL $WEB/install.sh | sudo bash -s -- --dry-run Pin curl -fsSL $WEB/install.sh | sudo bash -s -- --version=vX.Y.Z Roll back curl -fsSL $WEB/install.sh | sudo bash -s -- --rollback An upgrade snapshots the database and config.yaml into $BACKUP_DIR before the new server starts, and keeps the newest ${SNAPSHOT_RETAIN:-5}. ${UPGRADE_HELPER_NOTE:-Dashboard upgrades: not configured.} Local changes docker-compose.yml and traefik/dynamic.yml under $DEPLOY_DIR are product-owned: they are rewritten on every run, so an edit to them is lost at the next upgrade. Put local changes in $DEPLOY_DIR/docker-compose.override.yml which this installer never reads or writes. Compose merges it over the base. EOF if [ -n "${WORKSPACE_NOTE:-}" ]; then cat >> "$REFERENCE_FILE.tmp" <<EOF Workspace image Not installed on this host, so no workspace can be created yet (~750 MiB). Everything else works. Fetch it with: curl -fsSL $WEB/install.sh | sudo bash -s -- --workspace-image=yes EOF fi mv -f "$REFERENCE_FILE.tmp" "$REFERENCE_FILE" || return 1 chmod 0644 "$REFERENCE_FILE" 2>/dev/null || true return 0 } write_reference_file || { warn "could not write $REFERENCE_FILE"; REFERENCE_FILE=""; } # A generated password exists in exactly one place otherwise: this terminal's # scrollback. A window that closes, a session that scrolls, a `| tee` nobody # ran — and the only credential for the install is gone, with no path back # except editing the database by hand. So it also goes to a root-only file, and # the banner says where. Only on a fresh install: on an upgrade the admin row # already exists and this value was never applied, so writing it would create a # file naming a password that does not work. PASSWORD_FILE="" if [ "$ADMIN_PW_GENERATED" = true ] && [ "$IS_UPGRADE" = false ]; then PASSWORD_FILE="$CONFIG_DIR/initial-admin-password" # Created empty and locked down before the secret goes in, so it is never # briefly world-readable on a host where umask is loose. if : > "$PASSWORD_FILE.tmp" 2>/dev/null && chmod 0600 "$PASSWORD_FILE.tmp" 2>/dev/null; then printf '%s\n' "$ADMIN_PW" >> "$PASSWORD_FILE.tmp" if mv -f "$PASSWORD_FILE.tmp" "$PASSWORD_FILE" 2>/dev/null; then chmod 0600 "$PASSWORD_FILE" 2>/dev/null || true else rm -f "$PASSWORD_FILE.tmp" 2>/dev/null || true PASSWORD_FILE="" fi else rm -f "$PASSWORD_FILE.tmp" 2>/dev/null || true PASSWORD_FILE="" fi [ -n "$PASSWORD_FILE" ] || warn "could not save the generated password to $CONFIG_DIR — it is shown below, and only below." fi # What an operator needs in the next sixty seconds, and nothing else. Anything # that can be looked up later lives in $REFERENCE_FILE, named at the bottom. # # One function, rendered once into a variable and then written out, rather than # the dozen bare `cat <<EOF`s this used to be. Three things depend on it being # a single string: # # - It is written to the terminal as well as to stdout when stdout has been # redirected. `install.sh > install.log` is a normal thing to do, and it # used to mean the one screen an operator actually needs — dashboard URL and # the generated password — went to the file and never to them. # - It is appended to $REFERENCE_FILE, so the summary survives the terminal # it was printed in. `devgrail summary` prints that file back. # - It is emitted before the image reclamation below, which is silent, slow # (docker image rm over ~800 MiB of superseded layers) and pure # housekeeping. Anything that kills the run in that window — a dropped ssh # session, an impatient ^C — used to take the whole summary with it. closing_summary() { cat <<EOF ============================================================ DevGrail is up and answering its readiness probe. Visit the dashboard to get started. Dashboard: $(dashboard_url)/ Login: $ADMIN_USER EOF if [ "$ADMIN_PW_GENERATED" = true ]; then cat <<EOF Password: $ADMIN_PW EOF if [ -n "$PASSWORD_FILE" ]; then cat <<EOF Saved to $PASSWORD_FILE (root only). Delete it once you have signed in and changed the password. EOF else cat <<EOF This is the only place it is shown — copy it now. EOF fi else cat <<EOF Password: the one you set during the install EOF fi cat <<EOF $TLS_NOTE $(dns_records " ") EOF if [ -n "${WORKSPACE_NOTE:-}" ]; then cat <<EOF ${WORKSPACE_NOTE} EOF fi cat <<EOF Version: ${NEW_SERVER_RELEASE:-unknown} ($WORKSPACE_VERSION_NOTE) Logs: docker compose --project-directory $DEPLOY_DIR logs -f EOF if [ -n "$REFERENCE_FILE" ]; then cat <<EOF Reference: upgrading, rollback, backups and local changes are written to $REFERENCE_FILE Print this screen again any time with: sudo devgrail summary EOF fi cat <<EOF ============================================================ EOF if [ -n "$SNAPSHOT_ARCHIVE" ]; then cat <<EOF Pre-upgrade snapshot (database + config.yaml + manifest): $SNAPSHOT_ARCHIVE Keep it until you have confirmed this upgrade. The newest $SNAPSHOT_RETAIN are retained. EOF fi } SUMMARY="$(closing_summary)" printf '%s\n' "$SUMMARY" # Also to the terminal, when stdout is not one. Guarded on both halves: there is # nothing to duplicate to when stdout is already the terminal, and a run with no # terminal at all (cloud-init, CI) must not fail trying to open /dev/tty. if [ ! -t 1 ] && { : > /dev/tty; } 2>/dev/null; then printf '%s\n' "$SUMMARY" > /dev/tty 2>/dev/null || true fi # And to disk, under the reference card written just above — the copy that is # still there tomorrow. Appended rather than folded into write_reference_file # because the password lines are deliberately not in that file: it is 0644, and # the generated password already has a 0600 home of its own. if [ -n "$REFERENCE_FILE" ] && [ -f "$REFERENCE_FILE" ]; then { printf '\nInstall summary (printed at the end of the run)\n' # Minus two blocks: the password, which does not belong in a 0644 file, and # the "Reference:" pointer, which in this file points at itself. Each is a # labelled line plus however many continuation lines follow it at the # deeper indent, so the filter has to carry that state rather than match # line by line. printf '%s\n' "$SUMMARY" | awk ' /^ (Password|Reference):/ { skip = 1; next } skip && /^ / { next } { skip = 0; print } ' } >> "$REFERENCE_FILE" 2>/dev/null || true fi # Housekeeping, deliberately last: the operator has what they need on screen and # on disk before this starts, so nothing that happens here can cost them the # summary. Its one log line lands below the banner, which is the right way round # — it is a note about disk space, not part of the result. reclaim_old

#!/usr/bin/env bash # DevGrail one-command installer. # # curl -fsSL https:///install.sh | sudo bash # # Detects arch, installs Docker if missing, downloads the DevGrail image tarballs # from devgrail-web (verified by SHA-256, loaded with `docker load` — no GitHub, # no `docker pull`, no registry auth), lays down the compose + Traefik config, # writes config + secrets, and brings the stack up behind Traefik + Let's Encrypt. # # Everything is resolved from one release manifest, so an upgrade cannot mix a # new compose file with an old server image. A registry that publishes no release # manifest falls back to resolving each artifact by slug, exactly as before. # # Re-running upgrades in place: only artifacts whose version changed are # re-downloaded — images *and* assets; existing secrets and the data volume are # preserved. Free space is checked against the declared artifact sizes before # anything is fetched. An upgrade takes a snapshot of the database + config.yaml # into /opt/devgrail/backups before the new server starts, then waits for the # server's readiness probe and exits non-zero (with logs and the snapshot path) # if it never becomes ready. Images older than :previous are reclaimed only once # that probe is green — until then :previous is the rollback target. # # It also installs /usr/local/bin/devgrail (backup / restore / list / passwd / # summary — the last re-prints the closing screen from $INSTALL_DIR) # and offers to schedule a weekly backup. # # Shape of a run: every question is asked first — domains, admin account, DNS, # the two Docker daemon settings, the backup timer — then the plan is printed # and confirmed once, and nothing stops after that. Installing Docker is the # boundary; see the "# --- Docker ---" and "# --- the plan ---" sections, and # test/installer/flow_test.sh, which fails if a question moves below it. # Unattended runs (--yes, or no terminal) answer from env vars and defaults and # skip only the confirmation. An interactive fresh install opens with an # introduction (intro_install) and offers the long version behind one keypress # (intro_details: host requirements, everything this run puts on the box, and # what installing before DNS exists costs). Each group of questions carries a # heading — see section(). # # Prerequisites: DNS A records `DEVGRAIL_DOMAIN` and `*.DEVGRAIL_BASE_DOMAIN` # pointing at this VPS. Inputs may be passed as env vars or answered at the # prompt. DNS is required for a *trusted certificate*, not for installing: the # preflight warns and offers to continue, the stack comes up either way, and # Traefik orders the certificate once the records resolve. # # Vars: # DEVGRAIL_WEB_URL public site serving images + assets # (default: the site this script was downloaded from) # DEVGRAIL_DOMAIN dashboard/API/MCP hostname (prompted) # DEVGRAIL_BASE_DOMAIN base domain for app subdomains (prompted). First-boot # seed only: the database owns the list afterwards, and # it is managed under Settings -> App subdomains. # ACME_EMAIL Let's Encrypt contact email (prompted) # DEVGRAIL_ACME_CA which ACME CA to order certificates from: # production (default), staging, or a directory URL. # staging issues untrusted certs but is not meaningfully # rate limited — use it for repeated install testing on # one hostname. Same as --acme-ca=. # DEVGRAIL_ACME_BYPASS yes|no — let a passthrough admin route carry ACME # TLS-ALPN-01 challenges to the DevGrail behind it, for # a DevGrail running inside a workspace (default no). # Same as --acme-bypass=. Remembered. # DEVGRAIL_ADMIN_USERNAME admin account name (prompted on a fresh install; # default admin). First boot only — it renames nobody # afterwards. # DEVGRAIL_ADMIN_PASSWORD admin password (prompted on a fresh install, or # generated and printed at the end). First boot only; # use `devgrail passwd` to change it later. # HTTP_PORT/HTTPS_PORT host ports (default 80/443; real 80/443 needed for ACME) # DEVGRAIL_HEALTH_TIMEOUT seconds to wait for readiness after start (default 180) # DEVGRAIL_SNAPSHOT_RETAIN pre-upgrade snapshots to keep (default 5) # DEVGRAIL_VERSION release to install, e.g. v0.3.0 (default: current) # DEVGRAIL_WORKSPACE_IMAGE yes|no — whether to download the ~750 MiB workspace # image (default yes). `no` installs the server stack # only, which is most of the release in ~5% of the # bytes; no workspace can be created until a later run # fetches it. Same as --workspace-image=. The answer is # remembered, so an upgrade keeps skipping until you say # otherwise. # DEVGRAIL_DASHBOARD_UPGRADES yes|no — let admins upgrade from the dashboard # (default yes; needs systemd). Remembered. # DEVGRAIL_SCHEDULED_BACKUPS yes|no — answer the weekly-backup prompt unattended # DEVGRAIL_SKIP_DNS_CHECK 1 — skip the DNS preflight (same as --skip-dns-check) # DEVGRAIL_USERNS_REMAP yes|no — answer the userns-remap prompt unattended # DEVGRAIL_ADDRESS_POOLS yes|no — answer the address-pool prompt unattended # Both edit /etc/docker/daemon.json and restart Docker; # `no` also silences the explanation. See the "Docker # daemon preflight" section. # DEVGRAIL_LOCK_WAIT seconds to wait for a concurrent run to finish (default 300) # DEVGRAIL_ALLOW_UNSIGNED 1 — install a release whose signature cannot be # verified, instead of refusing. Verification is # required by default; this is the way out for a # private registry that does not sign. # DEVGRAIL_RELEASE_PUBKEY_FILE extra trusted release signing keys (PEM) # # Flags (pass with `| sudo bash -s -- `): --version=vX.Y.Z, --rollback, # --auto-rollback, --dry-run, --yes, --skip-dns-check, --userns-remap=yes|no, # --address-pools=yes|no, --workspace-image=yes|no, --dashboard-upgrades=yes|no, # --acme-ca=, --acme-bypass=yes|no, --help. See usage() below. set -euo pipefail # https://get.devgrail.com is substituted with the serving origin by devgrail-web's # /install.sh route handler (app/install.sh/route.ts) so the default always points # at the site this script was actually downloaded from. An explicit env var still # wins. This file is the single source of truth: `make publish` uploads it to the # registry as the install.sh asset; devgrail-web serves it. Do not keep a second copy. DEVGRAIL_WEB_URL="${DEVGRAIL_WEB_URL:-https://get.devgrail.com}" INSTALL_DIR="${DEVGRAIL_INSTALL_DIR:-/opt/devgrail}" DEPLOY_DIR="$INSTALL_DIR/deploy" CONFIG_DIR=/etc/devgrail VERSIONS_FILE="$INSTALL_DIR/versions.env" BACKUP_DIR="$INSTALL_DIR/backups" # Verified copies of the installer assets, kept so an unchanged re-run # re-downloads nothing. The rendered outputs under $DEPLOY_DIR are always # regenerated from these — substitution is cheap, a download is not. ASSET_DIR="$INSTALL_DIR/assets" WEB="${DEVGRAIL_WEB_URL%/}" # The concurrency lock (see acquire_lock) and the PID of whoever holds it. LOCK_FILE="$INSTALL_DIR/.install.lock" LOCK_OWNER_FILE="$INSTALL_DIR/.install.owner" # Set by snapshot_before_upgrade; reported in the success banner and on failure. SNAPSHOT_ARCHIVE="" # Staging directory for a snapshot in progress; removed by the EXIT trap so an # aborted run leaves no half-written copy of the database lying around. SNAPSHOT_STAGE="" # Scratch directories, declared here rather than where they are chosen so the # EXIT trap — installed as soon as the lock is taken, long before either has a # value — names only variables that exist. TMP="" SIG_DIR="" log() { printf '\033[1;34m==>\033[0m %s\n' "$*"; } warn() { printf '\033[1;33mWARNING:\033[0m %s\n' "$*" >&2; } die() { printf '\033[1;31merror:\033[0m %s\n' "$*" >&2; exit 1; } # step — a phase heading, numbered out of STEP_TOTAL. # # The install is long, the slowest part of it is in the middle, and until this # existed every line of it looked the same: an operator watching a fresh VPS had # no way to tell "two minutes left" from "ten". Phases are the coarse units a # person actually waits through, so they are counted, and log() stays what it # always was — detail within the phase. # # STEP_TOTAL is set once the run knows which phases it will have; a step() before # that, or after more phases than expected, degrades to an unnumbered heading # rather than lying about the denominator. STEP_NO=0 STEP_TOTAL=0 step() { STEP_NO=$((STEP_NO + 1)) if [ "$STEP_TOTAL" -gt 0 ] && [ "$STEP_NO" -le "$STEP_TOTAL" ]; then printf '\033[1;34m==> [%d/%d]\033[0m %s\n' "$STEP_NO" "$STEP_TOTAL" "$*" else printf '\033[1;34m==>\033[0m %s\n' "$*" fi } # section — the heading for one group of questions. # # The question half of a run reads as one long page: four topics, each a few # lines of context and a prompt, separated by a single blank line. Everything in # it carries the same weight, so the boundary between "which domain" and "shall # I restart your Docker daemon" is invisible — an operator scrolling back to # check what they just agreed to has nothing to aim at. A rule and a title cost # one line and give the run a visible structure. # # Distinct from the two rules already on screen, deliberately: `====` is a # whole-run banner (introduction, plan, closing summary) and `----` underlines a # heading inside a `?` explanation. This one is neither. # # Interactive only, like the introduction itself: unattended output is read by # grep and log collectors, where these would be padding. section() { have_tty && [ "$ASSUME_YES" != true ] || return 0 local title="$*" rule="" i i=$(( 54 - ${#title} )) [ "$i" -lt 3 ] && i=3 while [ "$i" -gt 0 ]; do rule="$rule─"; i=$((i - 1)); done printf '\n\033[1;36m ── %s %s\033[0m\n' "$title" "$rule" >&2 } # docker-compose.yml is product-owned: it is re-fetched and rewritten on every # run, so an operator edit to it is lost at the next upgrade — silently, which is # the part that makes it a trap. docker-compose.override.yml is the supported # place for local changes; this installer never writes or reads it beyond adding # it to the file list, and compose merges it over the base file. # # It has to be passed explicitly: compose only auto-discovers an override file # when no -f is given at all, and -f is what pins this to $DEPLOY_DIR. compose() { if [ -f "$DEPLOY_DIR/docker-compose.override.yml" ]; then docker compose --project-directory "$DEPLOY_DIR" \ -f "$DEPLOY_DIR/docker-compose.yml" \ -f "$DEPLOY_DIR/docker-compose.override.yml" "$@" else docker compose --project-directory "$DEPLOY_DIR" -f "$DEPLOY_DIR/docker-compose.yml" "$@" fi } usage() { cat <<'USAGE' DevGrail installer. curl -fsSL https://<devgrail-web>/install.sh | sudo bash curl -fsSL https://<devgrail-web>/install.sh | sudo bash -s -- [options] Options: --version=vX.Y.Z install that published release instead of the current one --rollback restore the previous images and the newest pre-upgrade snapshot, then wait for the server to become ready --auto-rollback if the server does not become ready after an upgrade, do the above automatically instead of asking --dry-run report what this run would change — release, image and asset versions, download size, free disk, settings — and exit without touching anything -y, --yes never prompt; take the prior value (or the default) for every question, and do not ask to confirm the plan (which is still printed). For unattended runs. It is deliberately not consent for the two Docker daemon settings, which restart the daemon — use --userns-remap= / --address-pools= for those. --skip-dns-check do not verify that the domains resolve to this host --userns-remap=yes|no answer the userns-remap prompt without asking. yes edits /etc/docker/daemon.json and restarts Docker (only offered on a host with no Docker state); no skips the question --address-pools=yes|no answer the address-pool prompt without asking. yes widens Docker's default pools (raising the ~27-workspace ceiling) and restarts Docker; no skips the question --workspace-image=yes|no whether to download the ~750 MiB workspace image (default yes). no installs the server stack only — most of the release in about 5% of the bytes — but no workspace can be created until a later run fetches it. That later run is this installer again with --workspace-image=yes, which re-downloads nothing else. The answer is remembered, so an upgrade keeps skipping until you say otherwise --dashboard-upgrades=yes|no whether admins may upgrade DevGrail from the dashboard (default yes, needs systemd). The dashboard only files a request; a root-owned helper on this host verifies the release and runs this installer. The answer is remembered --acme-ca=production|staging|<url> which ACME CA to order certificates from (default production). staging issues browser-untrusted certificates but is not meaningfully rate limited, so repeated installs on one hostname do not exhaust Let's Encrypt's 5-per-week limit. Each CA keeps its own certificate store, so switching back to production restores the real certs --acme-bypass=yes|no whether a TLS-passthrough admin route may carry Let's Encrypt's TLS-ALPN-01 challenges to the server behind it (default no). Turn it on to run a second DevGrail inside a workspace with certificates of its own; see "A DevGrail inside a workspace" in docs/RUNBOOK.md. This host's own certificates are unaffected. The answer is remembered -h, --help show this and exit A fresh interactive install asks for the admin username and password. Anything unattended (--yes, or no terminal) creates `admin` with a generated password, printed at the end and saved to /etc/devgrail/initial-admin-password (root only). Both are first-boot seeds; afterwards, change the password in the dashboard under Settings, or with `sudo devgrail passwd`. Inputs may also be passed as environment variables; see the header of this file. USAGE } DO_ROLLBACK=false AUTO_ROLLBACK=false DRY_RUN=false ASSUME_YES=false # The env var exists so an unattended install (cloud-init, a marketplace image) # can opt out without rewriting the command line it was handed. case "$(printf '%s' "${DEVGRAIL_SKIP_DNS_CHECK:-}" | tr '[:upper:]' '[:lower:]')" in 1|y|yes|true) SKIP_DNS_CHECK=true ;; *) SKIP_DNS_CHECK=false ;; esac # Answers to the two daemon-hardening questions (see the "Docker daemon # preflight" section). Empty means "ask"; "no" means the operator has already # decided and does not want to be asked again on the next upgrade. Anything # unrecognised is normalised to empty rather than guessed at — both settings # restart dockerd, so a typo must not be read as consent. norm_yes_no() { case "$(printf '%s' "${1:-}" | tr '[:upper:]' '[:lower:]')" in 1|y|yes|true) echo yes ;; 0|n|no|false) echo no ;; *) echo "" ;; esac } USERNS_REMAP_ANSWER="$(norm_yes_no "${DEVGRAIL_USERNS_REMAP:-}")" ADDRESS_POOLS_ANSWER="$(norm_yes_no "${DEVGRAIL_ADDRESS_POOLS:-}")" # Whether to fetch the workspace image. Unlike the two answers above, empty here # does not mean "ask" — nothing prompts for this — so a value that was set but # unrecognised is refused rather than normalised away: DEVGRAIL_WORKSPACE_IMAGE=nope # silently downloading 750 MiB is the opposite of what was asked for. WORKSPACE_IMAGE_ANSWER="$(norm_yes_no "${DEVGRAIL_WORKSPACE_IMAGE:-}")" if [ -n "${DEVGRAIL_WORKSPACE_IMAGE:-}" ] && [ -z "$WORKSPACE_IMAGE_ANSWER" ]; then die "DEVGRAIL_WORKSPACE_IMAGE takes yes or no — got '$DEVGRAIL_WORKSPACE_IMAGE'." fi # Whether admins may request upgrades from the dashboard (see the upgrade helper # below). Refused rather than normalised when unrecognised, like the workspace # image: a typo must not quietly leave a root-run path switched on. DASHBOARD_UPGRADES_ANSWER="$(norm_yes_no "${DEVGRAIL_DASHBOARD_UPGRADES:-}")" if [ -n "${DEVGRAIL_DASHBOARD_UPGRADES:-}" ] && [ -z "$DASHBOARD_UPGRADES_ANSWER" ]; then die "DEVGRAIL_DASHBOARD_UPGRADES takes yes or no — got '$DEVGRAIL_DASHBOARD_UPGRADES'." fi # Whether outer Traefik hands ACME challenges to passthrough routes (see "ACME # bypass" below). Refused when unrecognised, like the two above. ACME_BYPASS_ANSWER="$(norm_yes_no "${DEVGRAIL_ACME_BYPASS:-}")" if [ -n "${DEVGRAIL_ACME_BYPASS:-}" ] && [ -z "$ACME_BYPASS_ANSWER" ]; then die "DEVGRAIL_ACME_BYPASS takes yes or no — got '$DEVGRAIL_ACME_BYPASS'." fi while [ $# -gt 0 ]; do case "$1" in --version=*) DEVGRAIL_VERSION="${1#*=}" ;; --version) shift; DEVGRAIL_VERSION="${1:-}" [ -n "$DEVGRAIL_VERSION" ] || die "--version needs a value, e.g. --version=v0.3.0" ;; --rollback) DO_ROLLBACK=true ;; --auto-rollback) AUTO_ROLLBACK=true ;; --dry-run) DRY_RUN=true ;; -y|--yes) ASSUME_YES=true ;; --skip-dns-check) SKIP_DNS_CHECK=true ;; --userns-remap=*) USERNS_REMAP_ANSWER="$(norm_yes_no "${1#*=}")" [ -n "$USERNS_REMAP_ANSWER" ] || die "--userns-remap takes yes or no" ;; --address-pools=*) ADDRESS_POOLS_ANSWER="$(norm_yes_no "${1#*=}")" [ -n "$ADDRESS_POOLS_ANSWER" ] || die "--address-pools takes yes or no" ;; --workspace-image=*) WORKSPACE_IMAGE_ANSWER="$(norm_yes_no "${1#*=}")" [ -n "$WORKSPACE_IMAGE_ANSWER" ] || die "--workspace-image takes yes or no" ;; --dashboard-upgrades=*) DASHBOARD_UPGRADES_ANSWER="$(norm_yes_no "${1#*=}")" [ -n "$DASHBOARD_UPGRADES_ANSWER" ] || die "--dashboard-upgrades takes yes or no" ;; --acme-ca=*) DEVGRAIL_ACME_CA="${1#*=}" [ -n "$DEVGRAIL_ACME_CA" ] || die "--acme-ca takes production, staging or a directory URL" ;; --acme-bypass=*) ACME_BYPASS_ANSWER="$(norm_yes_no "${1#*=}")" [ -n "$ACME_BYPASS_ANSWER" ] || die "--acme-bypass takes yes or no" ;; -h|--help) usage; exit 0 ;; *) usage >&2; die "unknown option: $1" ;; esac shift done # --rollback restores images and a snapshot; there is nothing about it a dry run # could report without doing it. Refuse the combination rather than pick one. if [ "$DO_ROLLBACK" = true ] && [ "$DRY_RUN" = true ]; then die "--dry-run cannot be combined with --rollback." fi [ "$(id -u)" = 0 ] || die "run as root: curl -fsSL $WEB/install.sh | sudo bash" case "$(uname -m)" in x86_64|amd64) ARCH=amd64 ;; aarch64|arm64) ARCH=arm64 ;; *) die "unsupported architecture: $(uname -m)" ;; esac # --- concurrency lock -------------------------------------------------------- # Two runs at once — an impatient operator re-running while the first is still # downloading, or a scheduled upgrade landing on top of a manual one — race on # .env, versions.env, config.yaml and `docker compose up`. The loser can leave a # half-written .env (which holds the admin password) or a compose project # pointing at an image the other run has already retagged. One flock closes it. # # Held on fd 9 for the lifetime of the process, so the kernel drops it on any # exit, including a crash. That is not the whole story: fd 9 is not # close-on-exec, so anything the installer forks inherits it, and a *daemon* that # inherits it holds the lock for as long as it runs. `systemctl enable --now # docker` below is exactly that case on a host whose systemctl starts dockerd as # a plain child (containers, the local sandbox) — the installer exits, dockerd # keeps fd 9, and every later run waits out the timeout for a run that ended # minutes ago. Two defences, because neither alone is enough: # # 1. The commands that can fork a surviving daemon are run with 9>&- (see the # Docker install and the backup timer). Prevention, but only for the forks # this script knows about. # 2. The holder records its PID in $LOCK_OWNER_FILE, removed on exit. A waiter # that cannot take the lock and finds no live owner knows the lock leaked # into some other process and breaks it. Recovery, for everything else. LOCK_WAIT="${DEVGRAIL_LOCK_WAIT:-300}" # Consecutive owner-less polls before a held lock is declared leaked. There is a # sub-second window where a legitimate peer holds the flock but has not yet # written its owner file; requiring several *consecutive* misses closes it, # since a real installer writes the record microseconds after acquiring. LOCK_STALE_GRACE=5 # True when the lock is held by a process that is still running. Anything else — # no file, junk in the file, a PID that has gone — is "no live owner", which is # what makes a leaked lock recoverable. lock_owner_alive() { local pid [ -f "$LOCK_OWNER_FILE" ] || return 1 IFS= read -r pid < "$LOCK_OWNER_FILE" 2>/dev/null || return 1 case "$pid" in ''|*[!0-9]*) return 1 ;; esac kill -0 "$pid" 2>/dev/null } acquire_lock() { if ! command -v flock >/dev/null 2>&1; then warn "flock is not available on this host — running without the install lock." echo " Do not run two installers at the same time." >&2 return 0 fi # A dry run reads the same files, so it takes the lock shared: it must not # read a half-written .env, and it must not block a real install either. # On a host with no install there is nothing to race over and nothing to read, # so it does not create $INSTALL_DIR just to lock it. if [ "$DRY_RUN" = true ] && [ ! -d "$INSTALL_DIR" ]; then return 0 fi mkdir -p "$INSTALL_DIR" || die "cannot create $INSTALL_DIR" # Append, not truncate. The lock file itself carries no content, but `>` also # truncates, and keeping every open of this path non-destructive is what lets # the file be reasoned about at all while several runs hold it open. exec 9>>"$LOCK_FILE" || die "cannot open $LOCK_FILE" local mode=-x what=exclusive waited=0 stale_for=0 broke_once=false if [ "$DRY_RUN" = true ]; then mode=-s; what=shared; fi # Polled with -n rather than blocking with -w: busybox's flock (Alpine and # most minimal images) has no -w at all, and rejects it with a usage error # whose exit status is indistinguishable from "the lock is held" — which made # this refuse to run at all on those hosts. -n and -s are the forms busybox # and util-linux agree on. while ! flock "$mode" -n 9; do if lock_owner_alive; then stale_for=0 else stale_for=$((stale_for + 1)) fi # Break a leaked lock, at most once. Unlinking and reopening puts us on a # fresh inode that the leaking process does not hold, while it keeps its # lock on the orphaned one; a third run opening the path lands on the new # inode too, so mutual exclusion is restored rather than abandoned. Once # only: if something re-takes the lock afterwards, that is a real peer, and # the right answer is to wait for it and time out, not to keep unlinking. if [ "$broke_once" = false ] && [ "$stale_for" -ge "$LOCK_STALE_GRACE" ]; then warn "Breaking a lock left behind by an installer that has already exited." echo " $LOCK_FILE is held, but no running process claims it — usually a" >&2 echo " daemon (dockerd) that inherited the lock from an interrupted run." >&2 echo " Continuing." >&2 rm -f "$LOCK_FILE" exec 9>>"$LOCK_FILE" || die "cannot reopen $LOCK_FILE" broke_once=true continue fi if [ "$waited" -ge "$LOCK_WAIT" ]; then die "another install.sh is already running on this host. Waited ${LOCK_WAIT}s for the $what lock on $LOCK_FILE. Wait for that run to finish and try again. Nothing has been changed. If you are sure no installer is running, see who holds it: fuser -v $LOCK_FILE # or: lsof $LOCK_FILE and break the lock by hand with: rm -f $LOCK_FILE" fi if [ "$waited" -eq 0 ]; then log "Another install.sh holds the lock — waiting up to ${LOCK_WAIT}s..." fi sleep 1 waited=$((waited + 1)) done # Under the lock now, so this cannot race a peer. Written in both modes: a # leaked *shared* lock blocks a real install just as effectively as a leaked # exclusive one, and is just as invisible without the record. printf '%s\n' "$$" > "$LOCK_OWNER_FILE" 2>/dev/null || true } acquire_lock # Installed here, as soon as there is something to clean up, and never replaced: # a second `trap ... EXIT` would silently drop this one. Everything it removes is # guarded with :+ because it runs on every exit path, including ones taken before # TMP or the snapshot staging dir have been chosen — `rm -rf` with no operands # fails, and a failing EXIT trap is a confusing thing to leave behind after an # otherwise clean run. cleanup() { rm -rf ${TMP:+"$TMP"} ${SNAPSHOT_STAGE:+"$SNAPSHOT_STAGE"} ${SIG_DIR:+"$SIG_DIR"} 2>/dev/null || true # Only if it is still ours. Two dry runs can hold the shared lock at once, and # the loser of that race must not delete the winner's record — doing so would # make a live peer look like a leak to the next run. if [ -f "$LOCK_OWNER_FILE" ]; then local owner="" IFS= read -r owner < "$LOCK_OWNER_FILE" 2>/dev/null || owner="" [ "$owner" = "$$" ] && rm -f "$LOCK_OWNER_FILE" 2>/dev/null fi return 0 } trap cleanup EXIT # --- detect a prior install (upgrade) -------------------------------------- # Read only now that the lock is held: this is the file a concurrent run would # be rewriting, and adopting a half-written copy of it is precisely what the # lock exists to prevent. # # On re-run the previous answers live in the deploy .env (rewritten every run). # The WHOLE file is read into a PRIOR_* namespace, not a hand-maintained list of # four keys: the rule is that an upgrade preserves everything unless the operator # changes it. Reading only some keys meant HTTP_PORT/HTTPS_PORT were silently # reset to 80/443 on every upgrade — best case Traefik then failed to bind, worst # case it bound and hijacked another service's ports. # # Precedence, everywhere: explicit env var > prompt answer > prior value > default. IS_UPGRADE=false PRIOR_KEYS="" if [ -f "$DEPLOY_DIR/.env" ]; then IS_UPGRADE=true while IFS= read -r _line || [ -n "$_line" ]; do # Skip blanks, comments, and anything without a '='. case "$_line" in ''|'#'*) continue ;; *=*) : ;; *) continue ;; esac _key="${_line%%=*}" _val="${_line#*=}" # Ignore anything that is not a plain shell identifier — this file is # written by us, but it is also the file an operator hand-edits. case "$_key" in ''|*[!A-Za-z0-9_]*) continue ;; esac # Strip one layer of surrounding quotes, matching how compose reads it. case "$_val" in \"*\") _val="${_val#\"}"; _val="${_val%\"}" ;; \'*\') _val="${_val#\'}"; _val="${_val%\'}" ;; esac eval "PRIOR_$_key=\$_val" PRIOR_KEYS="${PRIOR_KEYS:+$PRIOR_KEYS }$_key" done < "$DEPLOY_DIR/.env" fi # prior <KEY> — the prior install's value for KEY, or empty. prior() { eval "printf '%s' \"\${PRIOR_$1:-}\""; } # resolve <KEY> [default] — settle KEY per the precedence above. Prompted keys # call prompt() instead, passing `prior KEY` as the offered default. resolve() { local key="$1" def="${2:-}" cur eval "cur=\${$key:-}" [ -n "$cur" ] && return 0 cur="$(prior "$key")" [ -n "$cur" ] || cur="$def" eval "$key=\$cur" } # --- TLS switch (default on) ------------------------------------------------ # DEVGRAIL_TLS=off serves plain HTTP on :80 — only for a private network or # behind an external TLS terminator. Default on = HTTPS via Traefik + Let's # Encrypt. The choice selects which Traefik router set to install and the scheme # advertised in every server-generated URL (see internal/config Scheme()). resolve DEVGRAIL_TLS on case "$(printf '%s' "$DEVGRAIL_TLS" | tr '[:upper:]' '[:lower:]')" in off|false|0|no) TLS_ENABLED=false; SCHEME=http; TRAEFIK_ASSET=traefik-dynamic-http.yml ;; *) TLS_ENABLED=true; SCHEME=https; TRAEFIK_ASSET=traefik-dynamic.yml ;; esac # --- workspace image (default on) ------------------------------------------- # The server stack is the install; the workspace image is the only genuinely # optional artifact, and at ~750 MiB against the server's ~45 it is very nearly # the whole download. `no` skips it: the stack comes up and the dashboard works, # but creating a workspace needs the image, so this is a deferral rather than a # mode. Re-running this installer with --workspace-image=yes fetches it and # re-downloads nothing else, which is why there is no separate fetch command. # # Extracted by test/installer/workspace_image_test.sh from this marker to the # closing brace below — keep them adjacent. WORKSPACE_IMAGE_DEFAULT=yes # settle_workspace_image <answer> <prior> — yes or no, by the precedence this # file uses everywhere: explicit flag/env > prior .env value > default. The # prior value is re-normalised rather than trusted, because .env is a file # operators hand-edit. settle_workspace_image() { local chosen="$1" prior_val="$2" [ -n "$chosen" ] || chosen="$(norm_yes_no "$prior_val")" [ -n "$chosen" ] || chosen="$WORKSPACE_IMAGE_DEFAULT" printf '%s' "$chosen" } # Settled back into the DEVGRAIL_ name, the way resolve() does: it is the key # written to deploy/.env, read back as the prior answer on the next run, and # reported by the dry run's Settings diff. DEVGRAIL_WORKSPACE_IMAGE="$(settle_workspace_image "$WORKSPACE_IMAGE_ANSWER" "$(prior DEVGRAIL_WORKSPACE_IMAGE)")" # Flag/env > prior .env > yes. The prior value is re-normalised, not trusted, # because .env is a file operators hand-edit. DEVGRAIL_DASHBOARD_UPGRADES="$DASHBOARD_UPGRADES_ANSWER" [ -n "$DEVGRAIL_DASHBOARD_UPGRADES" ] || DEVGRAIL_DASHBOARD_UPGRADES="$(norm_yes_no "$(prior DEVGRAIL_DASHBOARD_UPGRADES)")" [ -n "$DEVGRAIL_DASHBOARD_UPGRADES" ] || DEVGRAIL_DASHBOARD_UPGRADES=yes sha256_of() { if command -v sha256sum >/dev/null 2>&1; then sha256sum "$1" | awk '{print $1}' else shasum -a 256 "$1" | awk '{print $1}'; fi } # Same, over stdin rather than a file. sha256_str() { if command -v sha256sum >/dev/null 2>&1; then sha256sum | awk '{print $1}' else shasum -a 256 | awk '{print $1}'; fi } gen_secret() { openssl rand -hex 32 2>/dev/null || head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n'; } gen_password() { (openssl rand -base64 18 2>/dev/null || head -c 15 /dev/urandom | base64) | tr -dc 'A-Za-z0-9' | cut -c1-20; } # --- admin credential validation -------------------------------------------- # The password ends up in deploy/.env as a bare, unquoted value, read back by # docker compose, by this script's own PRIOR_* reader and by the test harnesses. # Rather than teach all of them to unquote, the charset an operator may type is # restricted to what survives that file verbatim: no whitespace, no quotes, and # none of # $ \ ` & | < > ( ) [ ] { } / which either end the value or mean # something to compose. A generated password (gen_password) is alphanumeric, so # it is inside this set by construction. Passwords set later through the app or # `devgrail passwd` never touch .env and are not restricted this way. ADMIN_PASSWORD_MIN=12 valid_admin_password() { local pw="$1" [ "${#pw}" -ge "$ADMIN_PASSWORD_MIN" ] || return 1 # bcrypt truncates at 72 bytes, so anything longer is only half-checked. [ "${#pw}" -le 72 ] || return 1 case "$pw" in *[!A-Za-z0-9!%*+,.:\;=?@^_~-]*) return 1 ;; esac return 0 } # 3-32 characters, starting alphanumeric. Matches what a login form and the # users table accept, and leaves no room for a leading dash or an empty name. valid_admin_username() { case "$1" in ''|[!A-Za-z0-9]*) return 1 ;; *[!A-Za-z0-9._-]*) return 1 ;; esac [ "${#1}" -ge 3 ] && [ "${#1}" -le 32 ] } # read_secret VAR PROMPT — read a line from the terminal without echoing it. # Same /dev/tty convention as prompt(): stdin is the piped script. read_secret() { local var="$1" msg="$2" val printf '%s: ' "$msg" > /dev/tty read -rs val < /dev/tty || true printf '\n' > /dev/tty eval "$var=\$val" } # --- readiness --------------------------------------------------------------- # `docker compose up -d` returning 0 only means containers were *created*. The # failure that matters most — the server crash-looping on a migration — passes # that check and hands the customer a URL that does not work. /api/v1/healthz # answers the real question (database reachable, Docker reachable, schema # matches the binary) and returns 503 until all three hold. # # Two probe targets: the public URL is what the customer will actually use, but # it depends on DNS already pointing here and on Let's Encrypt having issued, # neither of which is true on a first install. The container-local address # verifies the server itself, so a not-yet-pointed DNS record does not masquerade # as a broken server. HEALTH_TIMEOUT="${DEVGRAIL_HEALTH_TIMEOUT:-180}" # The URL the operator actually opens, port included. HTTP_PORT/HTTPS_PORT are a # supported setting and are carried across upgrades on purpose (see the # prior-install block), so an install on :8443 is reached only on :8443 — a # banner or a probe that names the default port instead points at nothing. # A function rather than a variable because --rollback prints it before the # ports have been resolved; the prior .env is the source there. dashboard_url() { local port if [ "$SCHEME" = https ]; then port="${HTTPS_PORT:-$(prior HTTPS_PORT)}"; port="${port:-443}" [ "$port" = 443 ] && { printf '%s://%s' "$SCHEME" "$DEVGRAIL_DOMAIN"; return; } else port="${HTTP_PORT:-$(prior HTTP_PORT)}"; port="${port:-80}" [ "$port" = 80 ] && { printf '%s://%s' "$SCHEME" "$DEVGRAIL_DOMAIN"; return; } fi printf '%s://%s:%s' "$SCHEME" "$DEVGRAIL_DOMAIN" "$port" } # The two A records this install needs, with the arrows aligned whatever length # the domains are — they were hand-spaced for one example domain and lined up # for nobody else. $1 is the indent. dns_records() { local indent="${1:- }" wild="*.$DEVGRAIL_BASE_DOMAIN" w w=${#DEVGRAIL_DOMAIN} [ "${#wild}" -gt "$w" ] && w="${#wild}" printf '%sA %-*s -> this VPS ' "$indent" "$w" "$DEVGRAIL_DOMAIN" printf '%sA %-*s -> this VPS ' "$indent" "$w" "$wild" } probe_health() { # Public route through Traefik. -k because the certificate may still be the # ACME self-signed placeholder; this checks readiness, not trust. curl -fsSk --max-time 5 "$(dashboard_url)/api/v1/healthz" >/dev/null 2>&1 && return 0 # The server directly, from inside its own container. Compose exec keeps stdin # attached even with -T; detach it explicitly because, under `curl | bash`, # stdin is the unread remainder of this script. Letting Compose consume it # makes bash reach EOF after this block and silently skip the closing summary. compose exec -T devgrail-server /usr/local/bin/devgrail-server -healthcheck </dev/null >/dev/null 2>&1 } wait_for_health() { local waited=0 # step() rather than log(): this is a phase an operator waits through, and on # a first install it is the longest one after the download. --rollback calls # wait_for_health too, outside any counted sequence — step() prints an # unnumbered heading there, which is what STEP_TOTAL=0 is for. step "Waiting for DevGrail to become ready (up to ${HEALTH_TIMEOUT}s)..." while [ "$waited" -lt "$HEALTH_TIMEOUT" ]; do if probe_health; then log "DevGrail is ready." return 0 fi sleep 3 waited=$((waited + 3)) done return 1 } # --- rollback ---------------------------------------------------------------- # data_volume prints the Docker volume backing /var/lib/devgrail. It is resolved # from the running container rather than assumed from the Compose project name, # which is derived from the deploy directory and so differs between installs. data_volume() { local vol vol="$(docker inspect devgrail-server \ --format '{{range .Mounts}}{{if eq .Destination "/var/lib/devgrail"}}{{.Name}}{{end}}{{end}}' 2>/dev/null)" || vol="" if [ -n "$vol" ]; then printf '%s' "$vol"; return 0; fi docker volume ls --format '{{.Name}}' 2>/dev/null | grep -E '(^|_)devgrail-data$' | head -n1 || true } newest_snapshot() { ls -1t "$BACKUP_DIR"/pre-upgrade-*.tgz 2>/dev/null | head -n1 || true } # Point :latest back at the image the previous run displaced. This is a tag move, # not a copy — which is the whole reason the retag is worth doing on every # upgrade: it turns rollback into a retag plus a compose recreate. rollback_images() { local slug moved=0 for slug in devgrail-server devgrail-container; do if docker image inspect "$slug:previous" >/dev/null 2>&1; then docker tag "$slug:previous" "$slug:latest" moved=$((moved + 1)) else warn "no $slug:previous image on this host — leaving $slug:latest where it is" fi done [ "$moved" -gt 0 ] } # restore_snapshot <archive> — put the database and config.yaml back as they were # before the upgrade that took this archive. restore_snapshot() { local archive="$1" vol stage vol="$(data_volume)" [ -n "$vol" ] || die "cannot find the DevGrail data volume — refusing to guess which volume to overwrite." stage="$(mktemp -d)" tar xzf "$archive" -C "$stage" || { rm -rf "$stage"; die "could not read $archive"; } if [ ! -f "$stage/devgrail.db" ] || [ ! -f "$stage/config.yaml" ]; then rm -rf "$stage" die "$archive is missing devgrail.db or config.yaml — a database without the config.yaml that keys its encryption is not a usable restore." fi compose stop devgrail-server >/dev/null 2>&1 || true # config.yaml first, deliberately: the JWT secret in it decrypts everything in # the database, so a half-completed restore that stopped here is recoverable, # whereas one that restored only the database is not. install -m 600 "$stage/config.yaml" "$CONFIG_DIR/config.yaml" \ || { rm -rf "$stage"; die "could not restore $CONFIG_DIR/config.yaml"; } # The -wal/-shm sidecars belong to the database being replaced. Leaving them # next to a restored file is how a "successful" restore produces a corrupt # database, so they go before the copy lands. docker run --rm --userns=host \ -v "$vol":/var/lib/devgrail \ -v "$stage":/restore:ro \ --entrypoint /bin/sh devgrail-server:latest -c ' rm -f /var/lib/devgrail/devgrail.db-wal /var/lib/devgrail/devgrail.db-shm && cp /restore/devgrail.db /var/lib/devgrail/devgrail.db' \ || { rm -rf "$stage"; die "could not restore the database into volume '$vol'"; } rm -rf "$stage" log "Restored database + config.yaml from $archive" } # do_rollback returns 0 when the rolled-back stack answers its readiness probe. do_rollback() { [ -f "$DEPLOY_DIR/docker-compose.yml" ] \ || die "no DevGrail install found at $INSTALL_DIR — there is nothing to roll back to." # Read from the prior .env rather than prompting: a rollback happens when # something is already wrong, and it should not stop to ask questions it can # answer itself. DEVGRAIL_DOMAIN="${DEVGRAIL_DOMAIN:-$(prior DEVGRAIL_DOMAIN)}" local archive archive="$(newest_snapshot)" log "Rolling back to the previous images and snapshot..." rollback_images || warn "no previous images were available to restore" if [ -n "$archive" ]; then restore_snapshot "$archive" else warn "No pre-upgrade snapshot in $BACKUP_DIR — restoring images only." echo " The database keeps whatever schema the newer server migrated it to," >&2 echo " and the older binary will refuse to start against it (by design)." >&2 fi compose up -d wait_for_health } # Whether this run has a terminal to talk to. # # Not a test that the device node exists, which is what every one of these # checks used to be: the node exists inside a container started without a TTY, # but opening it fails with ENXIO. Existence answered "yes, ask the operator" in # precisely the environment where there is nobody to answer — the installer # stopped on the first prompt with "No such device or address" instead of taking # the default it already had. Opening it is the only test that distinguishes the # two, and it errs the safe way: an unopenable terminal is no terminal, and no # terminal means take the default. have_tty() { { : < /dev/tty; } 2>/dev/null; } # Prompt for a required input on /dev/tty (stdin is the piped script). An # optional default ($3, e.g. the prior install's answer) is shown in brackets; # pressing Enter keeps it. Without a terminal a default is used, else we die. # # $4, when given, names a function printing a short explanation of what the # value is for. Printed rather than offered behind `?` the way ask_yn does it: # these are the first questions of a fresh install, and an operator who has # never seen DevGrail cannot be expected to know that "base domain" means a # wildcard record. It is only reached on the interactive path, so an upgrade # taking prior answers and an unattended run stay exactly as quiet as before. prompt() { local var="$1" msg="$2" def="${3:-}" explain="${4:-}" val why; eval "val=\${$var:-}" [ -n "$val" ] && return # --yes is the same contract as having no terminal: take the offered default, # which on an upgrade is the prior install's answer. if [ "$ASSUME_YES" = true ] || ! have_tty; then [ -n "$def" ] && { eval "$var=\$def"; return; } if [ "$ASSUME_YES" = true ]; then why="--yes was given, so nothing is prompted for" else why="no terminal is available"; fi die "$var is not set and there is no prior value to fall back to ($why). Pass it as an environment variable." fi [ -n "$explain" ] && "$explain" >&2 if [ -n "$def" ]; then printf '%s [%s]: ' "$msg" "$def" > /dev/tty else printf '%s: ' "$msg" > /dev/tty; fi read -r val < /dev/tty || true val="${val:-$def}" [ -n "$val" ] || die "$var is required" eval "$var=\$val" } # Ask a yes/no question on /dev/tty (stdin is the piped script). $2 is the answer # a bare Enter takes. Callers decide what --yes and "no terminal" mean before # getting here; this only runs when there is a terminal and a question to ask. # # $3, when given, names a function printing the long explanation, and adds `?` to # the accepted answers. The two daemon questions below have a page of reasoning # behind them that is genuinely worth reading once and skipped forever after — # printed before the prompt it is a wall of text arriving at the least patient # moment of the install, so it is offered instead of imposed. Asking `?` re-asks # the question afterwards, so it costs nothing to look. # # Anything that is not an answer is re-asked rather than read as a no. A typo'd # key used to be silently taken as the negative answer — which for "Proceed?" is # an aborted install and for the daemon questions is a decision the operator # never made. Only a bare Enter takes the default, because that is the one # "invalid" answer that was actually meant. ask_yn() { local msg="$1" def="${2:-n}" explain="${3:-}" ans hint case "$def" in y|Y) hint='[Y/n]' ;; *) hint='[y/N]' ;; esac [ -n "$explain" ] && hint="${hint%]}/?]" while :; do printf '%s %s: ' "$msg" "$hint" > /dev/tty read -r ans < /dev/tty || ans="" ans="${ans:-$def}" case "$ans" in y|Y|yes|YES|Yes) return 0 ;; n|N|no|NO|No) return 1 ;; '?'|h|H|help|HELP) # With nothing to explain, `?` is just another answer that is not one. [ -n "$explain" ] && { "$explain" >&2; continue; } ;; esac if [ -n "$explain" ]; then printf ' Please answer y or n, or ? for the details.\n' > /dev/tty else printf ' Please answer y or n.\n' > /dev/tty fi done } # --rollback is a mode of its own: it touches no registry, asks nothing, and must # not fall through into an install. Handled here, once the helpers above exist # and before any prompt. if [ "$DO_ROLLBACK" = true ]; then if do_rollback; then cat <<EOF ============================================================ Rolled back. DevGrail is answering its readiness probe. Dashboard: $(dashboard_url)/ Images: devgrail-server:latest now points at the previous build. Versions: $VERSIONS_FILE still names the release you rolled off — it is written before the health gate, so this cannot un-write it. /healthz is the authority on what is running. Forward: re-running install.sh upgrades again, and moves the tag back without re-downloading anything. Logs: docker compose --project-directory $DEPLOY_DIR logs -f ============================================================ EOF exit 0 fi echo >&2 printf '\033[1;31merror:\033[0m %s\n' "The rolled-back stack did not become ready either." >&2 compose logs --tail=50 devgrail-server >&2 || true echo " Snapshots available: $BACKUP_DIR" >&2 exit 1 fi # --- introduction ------------------------------------------------------------ # The first thing a fresh install used to print was "Hostname for the DevGrail # dashboard:" — a question that presumes the reader already knows what DevGrail # will do with a hostname, and that DNS has to exist before a certificate can be # issued for it. Both are cheap to say once, here, while nothing has happened # yet and the answers can still be gathered. # # Interactive only: an unattended run has nobody to read it, and the log lines # it would push off the screen are the ones that matter there. intro_install() { cat >&2 <<EOF ============================================================ DevGrail installer DevGrail turns this VPS into a home for AI coding agents: isolated Docker workspaces, each with a browser terminal and its own subdomains, all managed from one dashboard. How this run goes: 1. Questions — all of them, up front. Each has a suggested answer in [brackets]; press Enter to take it. 2. Prerequisites — Docker and a few small tools, installed if this host does not already have them. 3. A plan, printed for you to confirm. The images, the Docker settings you chose and the stack itself all come after you say yes. Usually 5-10 minutes, most of it download. Worth having ready: - a hostname for the dashboard, e.g. devgrail.example.com - a base domain for workspace apps, e.g. example.com - an email address for Let's Encrypt notices - DNS pointing both names at this VPS: A devgrail.example.com -> this VPS A *.example.com -> this VPS They are checked later in this run; if they are not there yet you can add them and re-check without losing answers. DNS is not a prerequisite for installing — press d below for what an install without it does and does not get you. ============================================================ EOF } # The long version: what this host has to be, what ends up on it, and what DNS # is actually for. Offered rather than printed, the way ask_yn offers `?`: an # operator installing their second VPS should not have to scroll past a page # they have read, and one installing their first should not have to guess what # "prerequisites" a curl | bash line assumed about their machine. # # The figures come from README.md's Requirements section; keep them in step. intro_details() { cat >&2 <<EOF ============================================================ What DevGrail is, and what this run does to this host The product A dashboard, a REST API, a CLI and an MCP endpoint in front of Docker workspaces. A workspace is a container with a browser terminal, an SSH login, its own Docker network and its own app subdomains. Coding agents run inside those containers — never on this host directly. What this host needs - Root (you are running this with sudo) on x86-64 or ARM64 Linux. Ubuntu 22.04/24.04 is the primary target; Debian 12 and Rocky/Alma 9 use the same paths. Alpine works but has no systemd, so no scheduled backups there. - 2 GB RAM and 20 GB disk to run the stack at all; 8 GB and 60 GB+ if you mean to run several workspaces at once. A workspace is a full Ubuntu container — size for the ones you will run concurrently, not for DevGrail itself. - Ports 80 and 443 reachable from the internet. Let's Encrypt answers its challenge on the real 443; HTTP_PORT/HTTPS_PORT exist for running behind another TLS terminator, not for avoiding that. - Outbound HTTPS to get.docker.com if Docker is missing, to Let's Encrypt, and to the registry serving this release: $WEB - A domain you control — see DNS below. What gets installed - Docker Engine and the compose plugin from get.docker.com, only if this host has none. jq and openssl through the package manager, only if missing: one reads the release manifest, the other verifies its signature. - Two images, fetched over HTTPS, checked against a SHA-256 in a signed manifest and loaded with 'docker load' — no registry login, no 'docker pull': devgrail-server the product, one static binary devgrail-container the workspace base image, ~750 MiB (skip with --workspace-image=no) - Three long-running containers: Traefik at the edge, the DevGrail server, and a Docker socket proxy that is what keeps the server off the raw socket. - Files: $INSTALL_DIR (compose file, assets, backups), $CONFIG_DIR/config.yaml (root-only — it holds the key that encrypts every secret in the database) and /usr/local/bin/devgrail (backup, restore, list, passwd). - fs.inotify.max_user_instances raised to 1024 when it is lower, kept in /etc/sysctl.d/60-devgrail-inotify.conf: workspace file watchers share root's limit with systemd. - Docker volumes for the database and the certificates. Workspaces get their own as you create them. - Only if you say yes, each asked separately: two /etc/docker/daemon.json settings (both restart Docker) and a weekly systemd backup timer. Nothing is written outside those paths, and uninstall.sh removes what this script created. DNS, and installing before it exists Two records at your registrar or DNS host — the dashboard's own name, and a wildcard so <label>.<base domain> reaches a workspace app. They are not a prerequisite for installing. This run checks them and lets you continue anyway: the stack comes up, the dashboard is served and everything on this host is finished. What you do not get is a trusted certificate — Let's Encrypt cannot validate a name that does not reach this VPS, so Traefik serves its own placeholder and the browser calls the site untrusted until the records resolve. Once they do, Traefik orders the certificate itself; you can force the attempt with docker compose --project-directory $DEPLOY_DIR restart traefik No re-install and no re-run is needed. The one cost of that order is Let's Encrypt's rate limits, which count failures — so DNS first is cheaper when it is a choice at all. Time, and undoing it 5-10 minutes, most of it the workspace image download. Every question comes first, then one plan to confirm. Re-running this script upgrades in place after snapshotting the database; --rollback puts the previous release back. ============================================================ EOF } # One keypress in front of the page above. The loop is what makes it free to # look: reading the details returns here, so 'd' costs nothing and Enter still # starts the run. Anything that is neither is asked again rather than taken as # "begin": starting the run is not what a mistyped key should mean. intro_details_gate() { local ans while :; do printf ' Press Enter to begin, or d for the details (prerequisites, what\n gets installed, and installing before DNS exists): ' > /dev/tty read -r ans < /dev/tty || ans="" case "$ans" in d|D|details|'?'|h|H|help|HELP) intro_details ;; ''|y|Y|yes|YES|Yes) return 0 ;; *) printf ' Please press Enter to begin, or d for the details.\n' > /dev/tty ;; esac done } intro_upgrade() { cat >&2 <<EOF ============================================================ DevGrail upgrade An existing install was found in $INSTALL_DIR. Your current answers are offered as [defaults] — press Enter to keep each one, or type a new value to change it. A plan follows, and nothing is downloaded or restarted until you have confirmed it. The database and config.yaml are snapshotted before the new server starts, so this run can be undone with: install.sh --rollback ============================================================ EOF } if have_tty && [ "$ASSUME_YES" != true ]; then if [ "$IS_UPGRADE" = true ]; then intro_upgrade else intro_install # Only on a fresh install: an upgrade is being run by somebody who already # has all of this, and its own introduction is four lines by design. intro_details_gate fi elif [ "$IS_UPGRADE" = true ]; then log "Existing DevGrail install detected — current answers shown as [defaults]; press Enter to keep, or type a new value to change." fi explain_domain() { cat >&2 <<EXPLAIN The dashboard's own hostname. You log in here, and the API, the CLI and the MCP endpoint all live behind this one name. Its TLS certificate is issued for exactly this name, so an A record for it has to point at this VPS. A subdomain of a domain you own is the usual choice, and better than the bare domain: browsers that open the dashboard refuse plain HTTP for every name under it while they keep using it. EXPLAIN } explain_base_domain() { cat >&2 <<EXPLAIN Anything you run inside a workspace — a dev server on :3000, a preview build — is published as <label>.<base domain>, so this name needs a WILDCARD record: *.<base> -> this VPS. Usually the bare domain the dashboard sits on: example.com when the dashboard is devgrail.example.com. This only seeds the first entry — more base domains can be added later under Settings -> App subdomains, and re-running this installer does not overwrite a list you have edited there. EXPLAIN } explain_acme_email() { cat >&2 <<EXPLAIN Where Let's Encrypt sends certificate expiry and problem notices. It is not published anywhere and is used for nothing else — any mailbox you actually read will do. EXPLAIN } # bare_domain_hsts_note <domain> — an explanation, or nothing, when the # dashboard is about to move onto a bare domain such as example.com. The server # sends Strict-Transport-Security with includeSubDomains on the dashboard's # name (cmd/server/securityheaders.go), so every browser that opens it refuses # plain HTTP for that name and every name under it, and offers no way past a # certificate error there, until the max-age runs out. On devgrail.example.com # that is DevGrail's own names. On example.com it is every host the owner runs, # a NAS or a router page among them, and no server-side change takes it back. # # Silent without TLS (no header is sent), when max-age is 0 (nothing is pinned), # and when the name is the one already installed: its browsers are pinned # already, and asking again on every upgrade would only teach the operator to # press Enter. Two labels is the test. It misses a bare domain under a two-part # suffix (example.co.uk) and flags a name bought for DevGrail alone. Telling # those apart needs the public suffix list, and both mistakes are cheap: this # explains and asks, and never refuses. bare_domain_hsts_note() { local d d="$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')" d="${d%.}" [ "$TLS_ENABLED" = true ] || return 0 [ "$(prior DEVGRAIL_HSTS_MAX_AGE)" = 0 ] && return 0 [ "$d" = "$(prior DEVGRAIL_DOMAIN | tr '[:upper:]' '[:lower:]')" ] && return 0 case "$d" in .*|*..*|*.*.*) return 0 ;; ?*.?*) ;; *) return 0 ;; esac cat <<NOTE The dashboard tells browsers to use only HTTPS for its name and every name under it (HSTS with includeSubDomains), renewed on every visit and kept for a day after the last one by default. On $d that covers every host under $d, not only DevGrail's: a browser that uses the dashboard refuses plain HTTP to any of them and gives no way past a certificate error on any of them. Nothing on the server can undo that in a browser that has seen it. If $d serves nothing else, keep it. Otherwise a subdomain such as devgrail.$d keeps this to DevGrail's own names. NOTE } section "Names" # Asked again on a no rather than left to the plan's Proceed?, which would throw # away every answer given so far. A name passed in, or taken unattended, is # kept, with the note left in the log. while :; do prompt DEVGRAIL_DOMAIN " Hostname for the DevGrail dashboard, e.g. devgrail.example.com" "$(prior DEVGRAIL_DOMAIN)" explain_domain _note="$(bare_domain_hsts_note "$DEVGRAIL_DOMAIN")" [ -n "$_note" ] || break warn "$DEVGRAIL_DOMAIN is a bare domain." printf '%s\n' "$_note" >&2 if [ "$ASSUME_YES" = true ] || ! have_tty || ask_yn " Use $DEVGRAIL_DOMAIN anyway?" n; then break fi DEVGRAIL_DOMAIN= done unset _note # Seeds the deployment's first base domain on first boot only; after that the # database owns the list (internal/db.SeedBaseDomain), and more can be added, # renamed or re-defaulted under Settings -> App subdomains. Re-running this # installer therefore does NOT change a list that has already been edited there. prompt DEVGRAIL_BASE_DOMAIN " Base domain for workspace app subdomains, e.g. example.com" "$(prior DEVGRAIL_BASE_DOMAIN)" explain_base_domain # Only needed for Let's Encrypt; plain-HTTP installs skip it (ACME is never run). if [ "$TLS_ENABLED" = true ]; then prompt ACME_EMAIL " Email for Let's Encrypt certificates" "$(prior ACME_EMAIL)" explain_acme_email fi # --- ACME CA (production / staging / custom) -------------------------------- # Let's Encrypt production issues at most 5 certificates per exact set of # identifiers per 168h. Any test cycle that starts from an empty certificate # store re-orders on every run — `uninstall.sh --purge` removes the traefik-acme # volume, and the first install after enabling userns-remap moves Docker's data # root to /var/lib/docker/<uid>.<gid>, orphaning the old volume just the same — # so five reinstalls on one hostname exhaust the quota. Traefik then falls back # to its self-signed "TRAEFIK DEFAULT CERT", which is what an invalid-certificate # browser warning on a freshly installed host actually means. Ordering from the # staging CA avoids it: the certificates are untrusted, but nothing else about # the issuance path differs, so it still exercises DNS, :443 and TLS-ALPN-01. # # Not prompted: production is right for every real install, and this only exists # for repeated testing against one hostname. ACME_CA_PRODUCTION_URL=https://acme-v02.api.letsencrypt.org/directory ACME_CA_STAGING_URL=https://acme-staging-v02.api.letsencrypt.org/directory # set_acme_ca <production|staging|url> — settle DEVGRAIL_ACME_CASERVER and the # store file that goes with it. The full URLs are accepted as well as the # aliases so a prior .env, which records the URL, round-trips unchanged. # # The store is per-CA on purpose: acme.json holds the registered ACME account, # and an account issued by one CA is unknown to another, so a shared file makes # every order fail with "account does not exist". Separate files also mean a # staging run leaves the production certificates already on this host intact, # and switching back picks them up again instead of re-ordering. set_acme_ca() { case "$(printf '%s' "${1:-}" | tr '[:upper:]' '[:lower:]')" in ''|production|prod|live|"$ACME_CA_PRODUCTION_URL") DEVGRAIL_ACME_CASERVER="$ACME_CA_PRODUCTION_URL" DEVGRAIL_ACME_STORAGE=acme.json ;; staging|test|"$ACME_CA_STAGING_URL") DEVGRAIL_ACME_CASERVER="$ACME_CA_STAGING_URL" DEVGRAIL_ACME_STORAGE=acme-staging.json ;; http://*|https://*) # A custom directory (pebble, an internal CA). The store name is derived # from the URL so two custom CAs cannot share a store or clobber acme.json. DEVGRAIL_ACME_CASERVER="$1" DEVGRAIL_ACME_STORAGE="acme-$(printf '%s' "$1" | sha256_str | cut -c1-8).json" ;; *) die "the ACME CA must be production, staging, or a directory URL starting with https:// — got '$1'. Set it with --acme-ca=, DEVGRAIL_ACME_CA, or DEVGRAIL_ACME_CASERVER in $DEPLOY_DIR/.env." ;; esac } if [ -n "${DEVGRAIL_ACME_CA:-}" ]; then set_acme_ca "$DEVGRAIL_ACME_CA" else # No explicit choice: keep what the prior install used. An upgrade must not # silently move a staging test back onto production and spend its quota. set_acme_ca "$(prior DEVGRAIL_ACME_CASERVER)" fi ACME_CA_IS_PRODUCTION=true [ "$DEVGRAIL_ACME_CASERVER" = "$ACME_CA_PRODUCTION_URL" ] || ACME_CA_IS_PRODUCTION=false # --- ACME bypass (a DevGrail inside a workspace) ----------------------------- # Traefik answers every TLS-ALPN-01 handshake on :443 itself, before routing, so # a second DevGrail behind a passthrough admin route (*.inner.example.com -> # a workspace) never sees its own challenges and cannot get a certificate. # DEVGRAIL_ACME_BYPASS=true sets the websecure entrypoint's allowACMEByPass, which # lets a TCP router whose HostSNI matches take the challenge instead; names no # TCP router claims, this install's own included, are still answered here. # deploy/docker-compose.yml has the flag, docs/RUNBOOK.md the whole setup. # # Off unless asked for, and written as true/false because that is what Traefik # parses. Flag/env > prior .env > false; the prior value is re-normalised rather # than trusted, because .env is a file operators hand-edit. # # Extracted by test/installer/acme_bypass_test.sh from this marker to the # closing brace below — keep them adjacent. ACME_BYPASS_DEFAULT=false # settle_acme_bypass <answer> <prior> — true or false. settle_acme_bypass() { local chosen="$1" [ -n "$chosen" ] || chosen="$(norm_yes_no "$2")" case "$chosen" in yes) printf true ;; no) printf false ;; *) printf '%s' "$ACME_BYPASS_DEFAULT" ;; esac } DEVGRAIL_ACME_BYPASS="$(settle_acme_bypass "$ACME_BYPASS_ANSWER" "$(prior DEVGRAIL_ACME_BYPASS)")" # --- admin credentials ------------------------------------------------------ # Both are first-boot seeds: the server creates the admin account from them the # first time it starts and ignores them ever after (internal/db/seed.go), so an # upgrade asks nothing and simply carries the prior .env values over. Changing # them later is `devgrail passwd` or Settings -> Password in the dashboard. # # Unattended (--yes, or no terminal) keeps what every install did before this # was askable: the user `admin`, and a generated password printed in the closing # banner — the only place it is ever shown. ADMIN_PW_GENERATED=false ADMIN_USER="${DEVGRAIL_ADMIN_USERNAME:-}" ADMIN_PW="${DEVGRAIL_ADMIN_PASSWORD:-}" # Validate what the environment supplied before it reaches deploy/.env: a value # this script cannot round-trip would produce a stack nobody can log into, and # the failure would surface as "wrong password", not as a bad install. if [ -n "$ADMIN_USER" ] && ! valid_admin_username "$ADMIN_USER"; then die "DEVGRAIL_ADMIN_USERNAME must be 3-32 characters — letters, digits, dot, dash or underscore — starting with a letter or digit." fi if [ -n "$ADMIN_PW" ] && ! valid_admin_password "$ADMIN_PW"; then die "DEVGRAIL_ADMIN_PASSWORD must be $ADMIN_PASSWORD_MIN-72 characters of letters, digits or the punctuation ! % * + , - . : ; = ? @ ^ _ ~ (it is stored verbatim in deploy/.env, which quotes nothing)." fi [ -n "$ADMIN_USER" ] || ADMIN_USER="$(prior DEVGRAIL_ADMIN_USERNAME)" [ -n "$ADMIN_PW" ] || ADMIN_PW="$(prior DEVGRAIL_ADMIN_PASSWORD)" if [ "$IS_UPGRADE" = false ] && have_tty && [ "$ASSUME_YES" != true ]; then section "Admin account" echo " The first login for the dashboard, and the account that can see and" >&2 echo " manage every workspace on this host. Both parts can be changed later" >&2 echo " — Settings -> Password in the dashboard, or 'devgrail passwd'." >&2 while [ -z "$ADMIN_USER" ]; do prompt ADMIN_USER " Admin username" admin valid_admin_username "$ADMIN_USER" && break warn "3-32 characters: letters, digits, dot, dash or underscore, starting with a letter or digit." ADMIN_USER="" done if [ -z "$ADMIN_PW" ]; then echo >&2 echo " A generated password is 20 random characters. It is printed once, in" >&2 echo " the summary at the end of this run — copy it then. (It is also left" >&2 echo " in $DEPLOY_DIR/.env, readable by root only.)" >&2 echo " Recommended: yes, then change it in the dashboard if you want" >&2 echo " something memorable." >&2 if ask_yn " Generate a secure admin password automatically?" y; then ADMIN_PW="$(gen_password)" ADMIN_PW_GENERATED=true else _tries=0 _pw1="" _pw2="" while [ "$_tries" -lt 3 ]; do _tries=$((_tries + 1)) read_secret _pw1 " Admin password (at least $ADMIN_PASSWORD_MIN characters)" read_secret _pw2 " Repeat password" if [ "$_pw1" != "$_pw2" ]; then warn "The passwords do not match." elif ! valid_admin_password "$_pw1"; then warn "$ADMIN_PASSWORD_MIN-72 characters of letters, digits or ! % * + , - . : ; = ? @ ^ _ ~" else ADMIN_PW="$_pw1" break fi done unset _pw1 _pw2 _tries if [ -z "$ADMIN_PW" ]; then ADMIN_PW="$(gen_password)" ADMIN_PW_GENERATED=true warn "Using a generated password instead; it is printed at the end of this run." fi fi fi fi # Unattended, or an upgrade whose .env predates these keys. [ -n "$ADMIN_USER" ] || ADMIN_USER="admin" if [ -z "$ADMIN_PW" ]; then ADMIN_PW="$(gen_password)" # Only announce it on a fresh install. On an upgrade the admin row already # exists, so this value is never applied — an operator who blanked the # password in .env after logging in (which the compose file suggests) must not # be handed a banner naming a password that does not work. [ "$IS_UPGRADE" = true ] || ADMIN_PW_GENERATED=true fi # Never prompted, but just as destructive to reset: an operator running on # non-standard ports had them silently rewritten to 80/443 by every upgrade. resolve HTTP_PORT 80 resolve HTTPS_PORT 443 resolve DEVGRAIL_SERVER_IMAGE devgrail-server # --- DNS preflight ---------------------------------------------------------- # "DNS is the only manual step" (spec §9) — and it was the only unverified one. # The installer prompted for the domain, brought the stack up, and printed the # required A records *after* it had finished. When they were not in place, # Traefik's ACME TLS-ALPN-01 challenge failed and the customer got a browser TLS # error with nothing connecting it to the cause. Let's Encrypt's failure rate # limits then make retrying worse rather than better. # # So the names are resolved and compared against this host's addresses here, # before Docker is installed and long before the stack is started. This is a # warning, not a verdict: split-horizon DNS, a pre-cutover install and a record # that has not propagated yet all look identical from inside the host. It asks # before continuing when there is a terminal; --skip-dns-check turns it off. # host_ips <name> — the IPv4 addresses <name> resolves to, one per line, empty # when it does not resolve. Four resolvers because no single one is present # everywhere: getent is glibc (absent on busybox), dig and host come from # bind-utils which most VPS images do not install, nslookup is busybox's. host_ips() { local name="$1" out="" if command -v getent >/dev/null 2>&1; then out="$(getent ahostsv4 "$name" 2>/dev/null | awk '{print $1}')" || out="" fi if [ -z "$out" ] && command -v dig >/dev/null 2>&1; then out="$(dig +short +time=3 +tries=1 A "$name" 2>/dev/null)" || out="" fi if [ -z "$out" ] && command -v host >/dev/null 2>&1; then out="$(host -W 3 -t A "$name" 2>/dev/null | sed -n 's/.*has address //p')" || out="" fi if [ -z "$out" ] && command -v nslookup >/dev/null 2>&1; then out="$(nslookup "$name" 2>/dev/null | sed -n '/^Name:/,$p' | sed -n 's/^Address: *//p')" || out="" fi # Whatever the tool printed, reduced to one de-duplicated IPv4 per line. The # grep also drops the resolver's own address, which nslookup prints first. # shellcheck disable=SC2086 # deliberate word splitting: one address per line printf '%s\n' $out | grep -E '^([0-9]{1,3}\.){3}[0-9]{1,3}$' | sort -u || true } have_resolver() { command -v getent >/dev/null 2>&1 || command -v dig >/dev/null 2>&1 || command -v host >/dev/null 2>&1 || command -v nslookup >/dev/null 2>&1 } # this_host_ips — addresses that legitimately point at this machine. Both halves # matter: a marketplace VPS carries its public address on an interface, but a # NAT'd or floating-IP host does not, and there an echo service is the only way # to learn what the world sees. Either matching is enough. this_host_ips() { local locals="" public="" url if command -v ip >/dev/null 2>&1; then locals="$(ip -4 -o addr show scope global 2>/dev/null | awk '{print $4}' | cut -d/ -f1)" || locals="" fi if [ -z "$locals" ] && command -v hostname >/dev/null 2>&1; then locals="$(hostname -I 2>/dev/null)" || locals="" fi for url in https://api.ipify.org https://checkip.amazonaws.com https://ifconfig.me/ip; do public="$(curl -fsS --max-time 5 "$url" 2>/dev/null | tr -cd '0-9.')" || public="" [ -n "$public" ] && break done # shellcheck disable=SC2086 # deliberate word splitting: one address per line printf '%s\n' $locals $public | grep -E '^([0-9]{1,3}\.){3}[0-9]{1,3}$' | sort -u || true } # dns_problem <name> <this-host's-ips> <what> — one problem line, or nothing when # <name> resolves to an address this host answers on. dns_problem() { local name="$1" mine="$2" what="$3" got ip got="$(host_ips "$name")" if [ -z "$got" ]; then printf ' %s does not resolve — no %s found.\n' "$name" "$what" return 0 fi for ip in $got; do case " $(printf '%s' "$mine" | tr '\n' ' ') " in *" $ip "*) return 0 ;; esac done printf ' %s resolves to %s, which is not this host.\n' \ "$name" "$(printf '%s' "$got" | tr '\n' ' ')" } dns_preflight() { if [ "$SKIP_DNS_CHECK" = true ]; then log "Skipping the DNS preflight (--skip-dns-check)." return 0 fi if ! have_resolver; then warn "no DNS lookup tool here (getent/dig/host/nslookup) — skipping the DNS preflight." return 0 fi section "DNS" local mine problems probe attempt=0 answer mine="$(this_host_ips)" if [ -z "$mine" ]; then warn "could not determine this host's IP address — skipping the DNS preflight." return 0 fi # The whole check loops, because the natural response to it failing is to go # and add the record. It used to be a one-shot "Continue anyway? [y/N]": the # only way back from a typo'd record was to abort and re-run the installer, # which re-asks every question — so the cheap answer was to press on with DNS # that does not work and meet it later as a browser TLS error. while :; do attempt=$((attempt + 1)) if [ "$attempt" -eq 1 ]; then log "Checking DNS for $DEVGRAIL_DOMAIN and *.$DEVGRAIL_BASE_DOMAIN..." else log "Re-checking DNS (attempt $attempt)..." fi # A random label under the base domain can only resolve through the wildcard # record. Checking the base domain itself would pass on a bare A record and # then fail for every workspace subdomain, which is the case that matters. # The label carries the attempt number so a resolver that negatively cached # the previous one cannot keep a record that now exists looking missing. probe="devgrail-dns-probe-$$-$attempt.$DEVGRAIL_BASE_DOMAIN" problems="$(dns_problem "$DEVGRAIL_DOMAIN" "$mine" "A record")" problems="$problems$(dns_problem "$probe" "$mine" "wildcard A record for *.$DEVGRAIL_BASE_DOMAIN")" if [ -z "$problems" ]; then log "DNS resolves to this host." return 0 fi warn "DNS does not point at this host yet." printf '%s' "$problems" >&2 echo " This host answers on: $(printf '%s' "$mine" | tr '\n' ' ')" >&2 echo >&2 echo " Required records:" >&2 dns_records " " >&2 if [ "$TLS_ENABLED" = true ]; then echo >&2 echo " Continuing now means Let's Encrypt will attempt its TLS-ALPN-01" >&2 echo " challenge against a name that does not reach this host. It will fail," >&2 echo " and repeated failures hit rate limits that make retrying slower rather" >&2 echo " than faster — so it is usually cheaper to fix DNS first." >&2 echo >&2 echo " The install itself does not need DNS: continuing finishes the whole" >&2 echo " run, and the only thing missing is a trusted certificate. Traefik" >&2 echo " serves its own placeholder — which is what an 'untrusted site'" >&2 echo " warning on a fresh install means — and orders the real one once the" >&2 echo " records resolve. Nothing has to be re-installed." >&2 fi echo >&2 echo " Legitimate reasons to continue anyway: split-horizon DNS, a pre-cutover" >&2 echo " install, or a record that has not propagated yet. --skip-dns-check" >&2 echo " suppresses this check entirely." >&2 echo >&2 # Unattended behaviour is unchanged, and deliberately so: an install driven # by cloud-init frequently runs before the records exist, and refusing would # break that legitimate case. Both paths say so loudly and proceed; neither # can reach the prompt below, so neither can loop. if [ "$ASSUME_YES" = true ]; then warn "--yes given — continuing with unverified DNS." return 0 fi if ! have_tty; then warn "no terminal to ask — continuing with unverified DNS." return 0 fi echo " Add the records now in another window, then re-check — nothing on this" >&2 echo " host has been changed yet, and re-checking costs nothing." >&2 # Aborting is what 'a' means, not what an unrecognised key means: this is # asked with the records still missing, which is exactly when a mistyped # answer is most expensive. The inner loop re-asks it without reprinting the # page above; 'r' leaves it, to re-check. while :; do printf '[R]e-check, [c]ontinue anyway, or [a]bort? [R/c/a]: ' > /dev/tty read -r answer < /dev/tty || answer="" case "${answer:-r}" in r|R|re|retry|RETRY) echo >&2; continue 2 ;; c|C|y|Y|yes|YES|continue) log "Continuing with unverified DNS." return 0 ;; a|A|abort|ABORT|q|Q|quit|QUIT) die "aborted before anything on this host was changed. Fix the records and re-run, or re-run with --skip-dns-check." ;; *) printf ' Please answer r, c or a.\n' > /dev/tty ;; esac done done } dns_preflight # --- release verification policy -------------------------------------------- # Whether a release that cannot be verified may be installed anyway. It may not, # by default. Every artifact is also checked against a digest, but that digest # comes from the same origin as the bytes, so it detects a corrupted download and # nothing else — the signature is the only thing between a compromised registry # and root on this host. Warning and continuing left the sole authenticity check # switched off unless someone had already thought to switch it on, which is # exactly backwards for a check whose whole purpose is the case nobody expected. # # DEVGRAIL_ALLOW_UNSIGNED=1 is the way out, and deliberately not a flag: opting # out of the only authenticity check this install has should look like a decision # recorded in someone's provisioning script, not a keystroke at a prompt. It is # for a white-label operator running an unsigned registry of their own, and for # recovering a host onto a release published before signing existed. # # There is no cost here to hosts that predate signing: an installer old enough to # carry no key is a copy already on disk, running its own old rules. This one # refuses. # # DEVGRAIL_REQUIRE_SIGNATURE is what this used to be called when the default was # the other way round. =1 now asks for what already happens; =0 still means what # it always meant, so a provisioning script carrying it does not silently get the # opposite of what it asked for. ALLOW_UNSIGNED=0 case "${DEVGRAIL_ALLOW_UNSIGNED:-0}" in 1|yes|true) ALLOW_UNSIGNED=1 ;; esac if [ "${DEVGRAIL_REQUIRE_SIGNATURE:-}" = 0 ]; then ALLOW_UNSIGNED=1; fi # --- tooling: the fast half ------------------------------------------------ # jq, openssl and the coreutils go in here, ahead of the remaining questions; # Docker itself goes in further down, after the last of them. The split is not # cosmetic: the address-pool question below asks jq to read this daemon's # configured pools, and a jq that is not installed yet answers "no pools # configured" — which is the one answer that makes the installer offer to write # pools over an operator's deliberate ones. These three take seconds. Docker # takes minutes, and nothing that is asked depends on it: a host with no Docker # has no Docker state, which is exactly what the probes below need to know. # # A dry run must not install anything, and it cannot read the registry without # jq — so on a host missing either, the honest report is that they would be # installed, and it stops there. if [ "$DRY_RUN" = true ]; then MISSING_TOOLS="" command -v docker >/dev/null 2>&1 || MISSING_TOOLS="Docker" command -v jq >/dev/null 2>&1 || MISSING_TOOLS="${MISSING_TOOLS:+$MISSING_TOOLS and }jq" if [ -n "$MISSING_TOOLS" ]; then echo echo "============================================================" echo " Dry run — nothing on this host has been changed." echo echo " $MISSING_TOOLS would be installed first; this host does not have it yet." echo " Nothing further can be reported without it: the release manifest," echo " the version diff and the disk figures all come from the registry." echo "============================================================" exit 0 fi fi if ! command -v jq >/dev/null 2>&1; then log "Installing jq..." if command -v apt-get >/dev/null 2>&1; then apt-get update -qq && apt-get install -y -qq jq elif command -v dnf >/dev/null 2>&1; then dnf install -y jq elif command -v yum >/dev/null 2>&1; then yum install -y jq elif command -v apk >/dev/null 2>&1; then apk add --no-cache jq else die "please install 'jq' and re-run"; fi fi # openssl verifies the release signature (and generates secrets when present). # The install attempt is best-effort, but the outcome is not: verification is # required, and a host that cannot check a signature cannot be told the bytes it # is about to run as root are genuine. Refusing here rather than at the # signature check saves the operator the remaining questions. if ! command -v openssl >/dev/null 2>&1; then log "Installing openssl..." if command -v apt-get >/dev/null 2>&1; then apt-get install -y -qq openssl || true elif command -v dnf >/dev/null 2>&1; then dnf install -y openssl || true elif command -v yum >/dev/null 2>&1; then yum install -y openssl || true elif command -v apk >/dev/null 2>&1; then apk add --no-cache openssl || true fi fi if ! command -v openssl >/dev/null 2>&1 && [ "$ALLOW_UNSIGNED" != 1 ]; then die "'openssl' is required to verify the release signature, and it could not be installed here. Install it and re-run, or set DEVGRAIL_ALLOW_UNSIGNED=1 to install without verifying anything (secrets then come from /dev/urandom). Nothing has been changed." fi # tar builds the pre-upgrade snapshot archive; df/awk size the free-space check. for tool in tar df awk; do command -v "$tool" >/dev/null 2>&1 || die "'$tool' is required (used to take the pre-upgrade backup)" done # --- Docker daemon preflight (userns-remap + default address pools) -------- # # Two daemon-level settings materially change how DevGrail runs, and neither can # be expressed in a container spec: both live in /etc/docker/daemon.json and take # effect only when dockerd restarts. The installer is the one moment an operator # is reliably at a terminal, as root, on a host where these are still cheap to # change — so this section detects, explains, offers and applies, rather than # printing a wall of manual steps and hoping. # # Nothing here is applied without an explicit yes, and --yes is deliberately not # that yes: restarting dockerd stops every container on the host, and enabling # remap moves the daemon's data root. Those are decisions, not defaults. # DEVGRAIL_USERNS_REMAP / DEVGRAIL_ADDRESS_POOLS (and the matching flags) are how # an unattended run — a marketplace image build, cloud-init — opts in on purpose. DAEMON_JSON="${DEVGRAIL_DAEMON_JSON:-/etc/docker/daemon.json}" # Restart dockerd across the init systems this installer supports: systemd on the # Debian/RHEL images, OpenRC on Alpine. Non-zero means no known mechanism worked, # which callers treat as "put the config back". restart_docker() { if command -v systemctl >/dev/null 2>&1; then systemctl restart docker >/dev/null 2>&1 && return 0 fi if command -v rc-service >/dev/null 2>&1; then rc-service docker restart >/dev/null 2>&1 && return 0 fi if command -v service >/dev/null 2>&1; then service docker restart >/dev/null 2>&1 && return 0 fi return 1 } # Wait for the daemon to answer again. dockerd is slow to come back when it has # images to re-index, and the first start under userns-remap is slower still — it # lays down a fresh data root — so this window is generous on purpose. wait_for_docker() { local i=0 while [ "$i" -lt 90 ]; do docker info >/dev/null 2>&1 && return 0 sleep 1 i=$((i + 1)) done return 1 } # Merge a jq expression into $DAEMON_JSON and restart the daemon. # $1 human label for the change, used in messages # $2 jq expression applied to the existing object (or {} when there is none) # # The prior file is kept as <file>.devgrail-bak-<timestamp>. If dockerd does not # come back, the backup is restored and the daemon restarted again: leaving a # host without Docker is a worse outcome than either setting is worth. A # daemon.json that does not parse is never touched — that is an operator edit in # progress, and clobbering it would destroy work we cannot see. apply_daemon_json() { local label="$1" filter="$2" current='{}' backup="" merged stamp if [ -s "$DAEMON_JSON" ]; then if ! jq -e . "$DAEMON_JSON" >/dev/null 2>&1; then warn "$DAEMON_JSON exists but is not valid JSON — leaving it alone." echo " Fix the file (or move it aside) and re-run to apply $label." >&2 return 1 fi current="$(cat "$DAEMON_JSON")" stamp="$(date +%Y%m%d%H%M%S)" backup="$DAEMON_JSON.devgrail-bak-$stamp" cp -p "$DAEMON_JSON" "$backup" || { warn "could not back up $DAEMON_JSON — not changing it." return 1 } fi if ! merged="$(printf '%s' "$current" | jq "$filter" 2>/dev/null)" || [ -z "$merged" ]; then warn "could not compute the new $DAEMON_JSON — leaving it alone." [ -n "$backup" ] && rm -f "$backup" return 1 fi mkdir -p "$(dirname "$DAEMON_JSON")" if ! printf '%s\n' "$merged" > "$DAEMON_JSON.devgrail-new"; then warn "could not write $DAEMON_JSON — leaving it alone." [ -n "$backup" ] && rm -f "$backup" return 1 fi chmod 0644 "$DAEMON_JSON.devgrail-new" mv -f "$DAEMON_JSON.devgrail-new" "$DAEMON_JSON" log "Restarting Docker to apply $label (this stops running containers)..." if restart_docker && wait_for_docker; then log "Docker restarted; $label is in effect." [ -n "$backup" ] && log "Previous daemon config kept at $backup" return 0 fi warn "Docker did not come back after applying $label — rolling the change back." if [ -n "$backup" ]; then mv -f "$backup" "$DAEMON_JSON"; else rm -f "$DAEMON_JSON"; fi if restart_docker && wait_for_docker; then warn "Docker is back on its previous configuration; $label was NOT applied." return 1 fi die "Docker is not responding and restoring the previous configuration did not revive it. $DAEMON_JSON holds what it held before this run. Investigate with: journalctl -u docker --no-pager -n 50 Nothing else on this host has been changed." } # True when the daemon reports userns-remap as active. userns_remap_enabled() { docker info --format '{{.SecurityOptions}}' 2>/dev/null | grep -q 'name=userns' } # True when this host already has Docker state that enabling userns-remap would # orphan. Remap moves the daemon to a per-UID data root, so every image, # container and volume created before the switch becomes invisible — including, # on an upgrade, DevGrail's own data volume. This check is what keeps the offer # limited to "fresh host, nothing to lose". docker_has_state() { local what n for what in "ps -aq" "volume ls -q" "image ls -q"; do # shellcheck disable=SC2086 # grep -c '.' rather than wc -l: it counts a final line that has no trailing # newline, and skips blank ones. wc -l would report 0 for a one-line answer # that ended without a newline — i.e. read "this host is empty" off a host # that has exactly one image. n="$({ docker $what 2>/dev/null || true; } | grep -c '.' || true)" [ "${n:-0}" -gt 0 ] && return 0 done return 1 } # How many networks the daemon's configured pools can hand out. Docker carves # each pool into /<size> blocks, so a pool contributes 2^(size - mask) networks. # Zero means no pools are configured, i.e. the stock defaults, which top out # near 31 — the number this whole section exists to raise. address_pool_capacity() { local pools cap pools="$(docker info --format '{{json .DefaultAddressPools}}' 2>/dev/null || true)" case "$pools" in ''|null) echo 0; return 0 ;; esac cap="$(printf '%s' "$pools" | jq -r ' [ .[]? | (.Size - (.Base | split("/")[1] | tonumber)) | select(. >= 0) | pow(2; .) ] | add // 0 | floor' 2>/dev/null || true)" case "$cap" in ''|null) echo 0 ;; *) echo "$cap" ;; esac } # True when no route on this host already lives inside the candidate range. # POSIX sh has no CIDR arithmetic and the candidates are fixed, so each carries a # regex matching an address inside it. Routes on docker0 / br-* / veth are # excluded: Docker's own bridges already sit in 172.17-172.31 and would otherwise # veto the very range we want to hand it. pool_base_free() { local re="$1" routes command -v ip >/dev/null 2>&1 || return 0 routes="$({ ip -4 route show 2>/dev/null || true; } \ | grep -Ev 'dev (docker[0-9]*|br-[0-9a-f]+|veth[0-9a-z]*)' || true)" if printf '%s\n' "$routes" | grep -Eq "$re"; then return 1; fi return 0 } running_container_count() { # See docker_has_state for why this counts with grep rather than wc. { docker ps -q 2>/dev/null || true; } | grep -c '.' || true } # --- userns-remap ---------------------------------------------------------- # The decision is taken here, among the other questions. Acting on it — which # rewrites daemon.json and restarts dockerd — is apply_daemon_decisions(), # called once the plan has been shown and confirmed. Keeping the two apart is # what lets every question in this installer be asked before any of them is # acted on, and it is why this block only ever sets USERNS_DECISION. # # The reasoning below is worth reading once and never again, so it is offered # behind `?` rather than printed at a prompt nobody has patience for. explain_userns() { cat >&2 <<EXPLAIN Why userns-remap matters ---------------------------------------------------------- Workspaces run AI agents and whatever code those agents write. Inside the container the agent is the unprivileged 'dev' user, but it has passwordless sudo — so it can become container root at will, and without userns-remap container root IS host root. A bug in the kernel or the container runtime would then be a full host takeover. userns-remap maps container root onto an unprivileged host UID, so the same escape lands as a nobody. DevGrail's own infra containers (traefik, the two docker socket proxies, devgrail-server) opt out via userns_mode: host, so only workspaces are remapped. What it costs: one line in $DAEMON_JSON and a Docker restart. The restart moves the daemon's data root, which is why it is only offered on a host with no images, containers or volumes to orphan — after that, migrating is a deliberate, backed-up operation. EXPLAIN } # Printed whenever the answer ends up "no", by any route. It names the risk in # one line: an unattended run never sees the prompt, and "userns-remap was # skipped" means nothing to an operator reading the log three weeks later. userns_declined_note() { warn "Continuing without userns-remap — a workspace escape would be host root." echo " To enable it later (one-time, needs root):" >&2 echo " 1. Add to $DAEMON_JSON: { \"userns-remap\": \"default\" }" >&2 echo " 2. systemctl restart docker" >&2 echo " Do it before workspaces exist — afterwards the data-root change" >&2 echo " orphans them. Set DEVGRAIL_USERNS_REMAP=no to stop being asked." >&2 echo >&2 } USERNS_DECISION=skip if userns_remap_enabled; then USERNS_DECISION=active log "Docker userns-remap is enabled — workspace root maps to an unprivileged host UID." elif [ "$USERNS_REMAP_ANSWER" = no ]; then : # the operator has decided; do not re-ask on every upgrade elif docker_has_state; then # Not offered, and not a warning the operator can act on in place: switching # now would hide everything already on this host behind a new data root. warn "userns-remap is NOT enabled, and cannot be turned on safely from here." echo " This host already has Docker images, containers or volumes. Enabling" >&2 echo " remap moves the daemon to a new data root, so all of them — including" >&2 echo " DevGrail's data volume — would become invisible. Migrating that is a" >&2 echo " deliberate, backed-up operation, not something an installer should do" >&2 echo " behind a prompt." >&2 echo >&2 echo " To do it by hand, with the stack down and a backup taken:" >&2 echo " 1. devgrail backup" >&2 echo " 2. Add to $DAEMON_JSON: { \"userns-remap\": \"default\" }" >&2 echo " 3. systemctl restart docker" >&2 echo " 4. Re-run this installer, then devgrail restore <snapshot>" >&2 echo " Continuing without remap..." >&2 echo >&2 elif [ "$DRY_RUN" = true ]; then echo " Dry run: this host has no Docker state yet, so a real run would offer" >&2 echo " to enable userns-remap and restart Docker. Nothing has been changed." >&2 echo >&2 elif [ "$USERNS_REMAP_ANSWER" = yes ]; then USERNS_DECISION=enable log "DEVGRAIL_USERNS_REMAP=yes — enabling userns-remap." elif have_tty && [ "$ASSUME_YES" != true ]; then section "Harden Docker" echo " Map workspace root to an unprivileged host UID." >&2 echo " Workspaces run AI agents and whatever code those agents write, and the" >&2 echo " agent can become container root at will. Without this, container root" >&2 echo " IS host root, so a container escape owns the whole VPS." >&2 echo " Applying it rewrites $DAEMON_JSON and restarts Docker." >&2 echo >&2 echo " Recommended: YES. This host has no Docker images, containers or" >&2 echo " volumes yet, which is the only moment it is free — afterwards the" >&2 echo " data-root change would orphan everything, and this installer will" >&2 echo " refuse to offer it. Say no only if you knowingly need unmapped" >&2 echo " container UIDs on this host (host-root-owned bind mounts, say)." >&2 echo " Answer ? for the full reasoning." >&2 if ask_yn " Enable userns-remap?" y explain_userns; then USERNS_DECISION=enable else userns_declined_note fi else # Unattended with no answer given. --yes is deliberately not consent here: # this restarts dockerd and moves its data root, which is a decision rather # than a default. DEVGRAIL_USERNS_REMAP=yes is how an unattended run opts in. userns_declined_note fi # --- default address pools ------------------------------------------------- # Each workspace gets its own bridge network so untrusted agent code cannot reach # a peer workspace directly. Docker's stock pools only yield ~31 networks in # total, so a busy host runs out and container creation starts failing with # "could not find an available, non-overlapping IPv4 address pool". # # Same decide-now / apply-later split as userns-remap above. explain_pools() { cat >&2 <<EXPLAIN Why the address pools matter ---------------------------------------------------------- Every DevGrail workspace gets its own Docker network so one workspace cannot reach another. Docker's stock address pools yield only ~31 networks in total, and past that container creation fails with 'could not find an available, non-overlapping IPv4 address pool' — an error that shows up when you add a workspace, not now. Widening to $POOL_BASE (a /24 per network) raises the ceiling to ~$POOL_NEW_CAP. Networks that already exist keep their current subnets; this only affects networks created from now on. What it costs: one line in $DAEMON_JSON and a Docker restart. The range offered is checked against this host's routing table first, so it cannot shadow a route you already use. EXPLAIN } # The restart is the whole cost, and what it stops depends on what is running. # Printed on every path that leads to actually applying the change — including # the env-var one, so an unattended run still records what it stopped. pools_cost_note() { if [ "${RUNNING:-0}" -gt 0 ]; then echo " Applying it restarts the Docker daemon, stopping the $RUNNING container(s)" >&2 echo " running here. The DevGrail stack comes back on its own (restart:" >&2 echo " unless-stopped). Each workspace comes back or not according to" >&2 echo " its own restart policy — the default keeps whatever state it was" >&2 echo " in, and any workspace set to \"Never start\" you start again from" >&2 echo " the dashboard." >&2 else echo " Applying it restarts the Docker daemon. Nothing is running on this host" >&2 echo " yet, so that costs nothing right now." >&2 fi } pools_declined_note() { warn "Continuing with the default address pools — workspace ceiling stays at ~27." echo " To raise it later (one-time, needs root):" >&2 echo " 1. Add to $DAEMON_JSON:" >&2 echo " { \"default-address-pools\": [{\"base\": \"$POOL_BASE\", \"size\": 24}] }" >&2 echo " 2. systemctl restart docker" >&2 echo " Safe to defer until you approach the limit. Set" >&2 echo " DEVGRAIL_ADDRESS_POOLS=no to stop being asked." >&2 echo >&2 } POOL_CAP="$(address_pool_capacity)" POOLS_DECISION=skip POOL_BASE=""; POOL_NEW_CAP=0; RUNNING=0 if [ "${POOL_CAP:-0}" -ge 256 ]; then log "Docker address pools allow ~$POOL_CAP networks — no workspace ceiling to worry about." elif [ "$ADDRESS_POOLS_ANSWER" = no ]; then : # already decided elif [ "${POOL_CAP:-0}" -gt 0 ]; then # Pools are configured, just narrow. That is an operator decision about this # host's addressing, and it is not one to overwrite from behind a prompt. warn "Docker address pools are configured but allow only ~$POOL_CAP networks." echo " They were set deliberately, so this installer will not rewrite them." >&2 echo " To widen them, edit default-address-pools in $DAEMON_JSON and" >&2 echo " restart Docker. Continuing..." >&2 echo >&2 else if pool_base_free '(^|[[:space:]])172\.(1[6-9]|2[0-9]|3[01])\.'; then POOL_BASE=172.16.0.0/12; POOL_NEW_CAP=4096 elif pool_base_free '(^|[[:space:]])10\.201\.'; then POOL_BASE=10.201.0.0/16; POOL_NEW_CAP=256 fi RUNNING="$(running_container_count)" if [ -z "$POOL_BASE" ]; then warn "No safe range to offer — this host already routes both 172.16/12 and 10.201/16." echo " Handing Docker a pool that shadows one of your own routes is worse" >&2 echo " than the ceiling it would lift. Pick a free range yourself, add it" >&2 echo " as default-address-pools in $DAEMON_JSON, and restart Docker." >&2 echo " Continuing with the defaults..." >&2 echo >&2 elif [ "$DRY_RUN" = true ]; then echo " Dry run: a real run would offer to widen the pools to $POOL_BASE" >&2 echo " (a /24 per network, ~$POOL_NEW_CAP workspaces) and restart Docker." >&2 echo " Nothing has been changed." >&2 echo >&2 elif [ "$ADDRESS_POOLS_ANSWER" = yes ]; then POOLS_DECISION=widen log "DEVGRAIL_ADDRESS_POOLS=yes — widening the default address pools to $POOL_BASE." pools_cost_note elif have_tty && [ "$ASSUME_YES" != true ]; then section "Workspace ceiling" echo " Raise the ceiling on concurrent workspaces, currently about 27." >&2 echo " Each workspace gets its own Docker network so one workspace cannot" >&2 echo " reach another, and the stock pools run out at ~31 networks. Past that," >&2 echo " creating a workspace just fails. Widening to $POOL_BASE — checked" >&2 echo " free against this host's routes — raises the ceiling to ~$POOL_NEW_CAP." >&2 echo " Networks that already exist keep the subnets they have." >&2 pools_cost_note POOL_DEFAULT=y [ "${RUNNING:-0}" -gt 0 ] && POOL_DEFAULT=n echo >&2 if [ "${RUNNING:-0}" -gt 0 ]; then echo " Recommended: defer unless you are heading for more than ~27" >&2 echo " workspaces. Unlike userns-remap this can be applied at any later" >&2 echo " time with the same effect, and right now it costs a restart of" >&2 echo " the $RUNNING running container(s). Re-run this installer when you" >&2 echo " want it, or make the change by hand." >&2 else echo " Recommended: YES. Nothing is running on this host yet, so it is" >&2 echo " free right now, and the alternative is discovering the limit as a" >&2 echo " failed workspace creation months from now. Say no if you expect" >&2 echo " only a handful of workspaces, or if you know $POOL_BASE is used" >&2 echo " by a network this VPS reaches over a VPN — the check above only sees" >&2 echo " routes that exist right now." >&2 fi echo " Answer ? for the full reasoning." >&2 if ask_yn " Widen the address pools?" "$POOL_DEFAULT" explain_pools; then POOLS_DECISION=widen else pools_declined_note fi else # Unattended with no answer. As above, --yes is not consent for a daemon # restart; DEVGRAIL_ADDRESS_POOLS=yes is. pools_declined_note fi fi # --- apply the daemon decisions --------------------------------------------- # Called from the main flow after the plan summary, and before the first image is # downloaded. That order is not incidental: enabling userns-remap moves the # daemon's data root, so an image loaded before the restart would be invisible # after it. apply_daemon_decisions() { if [ "$USERNS_DECISION" = enable ]; then # dockerd creates the "dockremap" user and its subordinate ranges itself, # but only if these files exist for it to append to. Creating them empty is # a no-op where they already work, and removes a known first-start failure # on minimal images. [ -f /etc/subuid ] || : > /etc/subuid [ -f /etc/subgid ] || : > /etc/subgid if apply_daemon_json "userns-remap" '. + {"userns-remap": "default"}'; then if userns_remap_enabled; then log "userns-remap is active — workspace root is now an unprivileged host UID." else warn "Docker restarted but does not report userns-remap as active." echo " Check 'docker info' and journalctl -u docker. Continuing." >&2 fi fi fi if [ "$POOLS_DECISION" = widen ]; then apply_daemon_json "the wider address pools" \ ". + {\"default-address-pools\": [{\"base\": \"$POOL_BASE\", \"size\": 24}]}" || true fi return 0 } # --- inotify limit ----------------------------------------------------------- # Extracted by test/installer/inotify_limit_test.sh between this marker and the # matching "end inotify limit" one. Keep both comments intact, and keep # everything between them free of top-level side effects. # # fs.inotify.max_user_instances caps how many inotify instances one user may # hold, and Ubuntu ships it at 128. Without userns-remap a workspace's processes # are host root, so every file watcher in every workspace — dev servers, editors, # language servers — draws on the same budget as systemd. When it runs out, # devgrail-upgrade.path fails with "Too many open files" and stays failed (a path # unit has no Restart=), and the dashboard's Upgrade button waits forever for a # host that will never pick the request up. Raised, never lowered: a host whose # operator already set it higher keeps their value. # # The file records the value it replaced, so uninstall.sh --nuke can put it back. INOTIFY_KEY=fs.inotify.max_user_instances INOTIFY_PROC="${DEVGRAIL_INOTIFY_PROC:-/proc/sys/fs/inotify/max_user_instances}" INOTIFY_CONF="${DEVGRAIL_INOTIFY_CONF:-/etc/sysctl.d/60-devgrail-inotify.conf}" INOTIFY_TARGET=1024 # raise | keep (already at or above the target) | readonly (/proc/sys cannot be # written, as inside a container) | unknown (the value could not be read). INOTIFY_DECISION=keep INOTIFY_NOW="" # decide_inotify — sets INOTIFY_NOW and INOTIFY_DECISION. Reads only. decide_inotify() { INOTIFY_NOW="$(cat "$INOTIFY_PROC" 2>/dev/null || true)" case "$INOTIFY_NOW" in ''|*[!0-9]*) INOTIFY_DECISION=unknown ;; *) if [ "$INOTIFY_NOW" -ge "$INOTIFY_TARGET" ]; then INOTIFY_DECISION=keep elif [ -w "$INOTIFY_PROC" ]; then INOTIFY_DECISION=raise else INOTIFY_DECISION="readonly"; fi ;; esac return 0 } # inotify_recorded_previous [file] — the value an earlier run replaced, or empty. inotify_recorded_previous() { sed -n '/^# devgrail-previous: [0-9][0-9]*$/{s/^# devgrail-previous: //p;q;}' \ "${1:-$INOTIFY_CONF}" 2>/dev/null || true } # apply_inotify — raise the running value, then persist it for the next boot. # The running value first: a file that promises 1024 on a kernel still at 128 # would leave the host broken until a reboot nobody knows to do. Never fatal — # a missed limit is worth a warning, not an aborted install. apply_inotify() { local prev case "$INOTIFY_DECISION" in raise) ;; readonly) log "Cannot raise $INOTIFY_KEY here ($INOTIFY_PROC is read-only) — leaving it at $INOTIFY_NOW." return 0 ;; *) return 0 ;; esac # A re-raise (something lowered it after an earlier run) must not overwrite # the value that run recorded with one it set itself. prev="$(inotify_recorded_previous)" [ -n "$prev" ] || prev="$INOTIFY_NOW" if ! { printf '%s\n' "$INOTIFY_TARGET" > "$INOTIFY_PROC"; } 2>/dev/null; then warn "could not raise $INOTIFY_KEY from $INOTIFY_NOW to $INOTIFY_TARGET — continuing." return 0 fi if mkdir -p "$(dirname "$INOTIFY_CONF")" && cat > "$INOTIFY_CONF" <<EOF # Written by DevGrail's install.sh. Workspace file watchers share root's inotify # budget with systemd; at the default of 128 the dashboard upgrade watcher # (devgrail-upgrade.path) is the first thing to fail. uninstall.sh --nuke # removes this file and restores the value below. # devgrail-previous: $prev $INOTIFY_KEY = $INOTIFY_TARGET EOF then log "Raised $INOTIFY_KEY from $INOTIFY_NOW to $INOTIFY_TARGET (kept across reboots by $INOTIFY_CONF)." else warn "raised $INOTIFY_KEY to $INOTIFY_TARGET, but could not write $INOTIFY_CONF — it reverts on reboot." fi return 0 } # --- end inotify limit ------------------------------------------------------- # --- AppArmor workspace profile ---------------------------------------------- # Extracted by test/installer/apparmor_profile_test.sh between this marker and # the matching "end AppArmor workspace profile" one. Keep both comments intact, # and keep everything between them free of top-level side effects. # # userns-remap gives every workspace the same host UID range. A process that # gets out of a workspace with its credentials intact (a leaked descriptor, as # in runc's CVE-2024-21626) can then read and write every other workspace's # layers and volumes under Docker's data root, and swap the stdio FIFOs of # their `docker exec`s in the remapped exec root, which the range owns. # AppArmor stays attached to a process that gets out that way, so workspaces # run under devgrail-workspace: Docker's docker-default plus deny rules over # those two roots. A path reached through a leaked descriptor resolves to its # host path, so the rules hold there too; the spike in # docs/plans/WORKSPACE_UID_ISOLATION_PLAN.md §5.2 checked both cases on a real # host. The rules deny the same paths inside a workspace as well. That only # matters to Docker run inside one, which docker-default does not allow either. # # The file lives in /etc/apparmor.d, so the profile is loaded at boot before # dockerd restarts the workspaces. Every run rewrites and reloads it, and a # reload replaces the profile under running workspaces too. The server learns # the name (DEVGRAIL_WORKSPACE_APPARMOR_PROFILE in .env) only once the kernel # holds the profile: a name the kernel does not hold fails every create. # # A Docker-enabled workspace runs its own dockerd, which docker-default's mount # deny and its write denies on /proc/sys and /sys stop. Those workspaces run # under devgrail-workspace-nested instead: the same profile with mounts and # the network sysctls allowed, and both DevGrail deny rules kept. The inner # daemon's data and exec roots are moved off the denied paths by the # workspace image, not here. docs/plans/NESTED_DEVGRAIL_PLAN.md §3.6 is the # spike that found each change, and §3.7 the escape probe run under it. APPARMOR_PROFILE=devgrail-workspace APPARMOR_NESTED_PROFILE=devgrail-workspace-nested APPARMOR_DIR="${DEVGRAIL_APPARMOR_DIR:-/etc/apparmor.d}" APPARMOR_ENABLED="${DEVGRAIL_APPARMOR_ENABLED:-/sys/module/apparmor/parameters/enabled}" APPARMOR_PARSER="${DEVGRAIL_APPARMOR_PARSER:-apparmor_parser}" # The inner daemon's storage driver is fuse-overlayfs, which needs this device. FUSE_DEVICE="${DEVGRAIL_FUSE_DEVICE:-/dev/fuse}" # load | unavailable (the kernel has no AppArmor, or has it off) | noparser # (AppArmor is on, but apparmor_parser is not installed). APPARMOR_DECISION=unavailable # What .env tells the server: the profile name once it is loaded, else empty. WORKSPACE_APPARMOR_PROFILE="" # The nested profile's name once it is loaded and the host can run # Docker-enabled workspaces safely, else empty. The server offers the mode only # when this is set, so it is the host's whole readiness signal. WORKSPACE_NESTED_APPARMOR_PROFILE="" # decide_apparmor — sets APPARMOR_DECISION. Reads only. decide_apparmor() { if [ "$(cat "$APPARMOR_ENABLED" 2>/dev/null || true)" != Y ]; then APPARMOR_DECISION=unavailable elif command -v "$APPARMOR_PARSER" >/dev/null 2>&1; then APPARMOR_DECISION=load else APPARMOR_DECISION=noparser fi return 0 } # apparmor_root_ok <path> — whether <path> may go in a deny rule. Plain # characters only, because AppArmor reads globs in a path. And never a # directory a workspace needs for itself, because the rule denies that path # inside the workspace as well. apparmor_root_ok() { case "$1" in /?*) ;; *) return 1 ;; esac case "$1" in *[!A-Za-z0-9._/-]*|*//*|*/|*/.|*/..|*/./*|*/../*) return 1 ;; esac case "$1" in /bin|/boot|/dev|/etc|/home|/lib|/lib64|/media|/mnt|/opt|/proc|/root|/run|/sbin|/srv|/sys|/tmp|/usr|/usr/*|/var|/var/cache|/var/lib|/var/log|/var/run|/var/tmp|/workspace) return 1 ;; esac return 0 } # apparmor_canonical <path> — <path> with symlinks resolved, as AppArmor # matches it, when it exists. /var/run is a symlink to /run on every systemd host. apparmor_canonical() { local dir="${1%/*}" base="${1##*/}" if [ -d "$1" ]; then (cd "$1" 2>/dev/null && pwd -P) && return 0; fi if [ -n "$dir" ] && [ -d "$dir" ]; then printf '%s/%s\n' "$(cd "$dir" 2>/dev/null && pwd -P)" "$base"; return 0 fi printf '%s\n' "$1" } # apparmor_data_root <DockerRootDir> — the daemon's data root. Under # userns-remap `docker info` reports the remapped root inside it # (/var/lib/docker/100000.100000); the rule covers all of it. apparmor_data_root() { local root="${1:-/var/lib/docker}" leaf leaf="${root##*/}" case "$leaf" in *[!0-9.]*|.*|*.) ;; *.*) root="${root%/*}" ;; esac apparmor_canonical "$root" } # apparmor_exec_root <daemon.json exec-root> — the daemon's exec root. The # remapped one sits beside it with the range appended (/run/docker.100000.100000). apparmor_exec_root() { apparmor_canonical "${1:-/var/run/docker}" } # render_apparmor_profile <data root> <exec root> [dockerd's AppArmor label] [nested] # The body is Docker's docker-default (github.com/moby/profiles/apparmor, # template.go), including the ABI pin that keeps `network,` covering unix # sockets, and the includes Docker adds only where the host has them. Keep it in # step with that template. With "nested" as the fourth argument it renders # devgrail-workspace-nested, which differs only where the comments below say. render_apparmor_profile() { local data="$1" exec_root="$2" daemon="${3:-unconfined}" variant="${4:-}" name="$APPARMOR_PROFILE" [ "$variant" = nested ] && name="$APPARMOR_NESTED_PROFILE" echo "# Written by DevGrail's install.sh, and rewritten by every run: edits here" echo "# are lost. Docker's docker-default plus the deny rules at the end." [ "$variant" = nested ] \ && echo "# For Docker-enabled workspaces: mounts and the network sysctls allowed." [ -f "$APPARMOR_DIR/abi/3.0" ] && echo 'abi <abi/3.0>,' if [ -f "$APPARMOR_DIR/tunables/global" ]; then echo '#include <tunables/global>' else echo '@{PROC}=/proc/' fi echo echo "profile $name flags=(attach_disconnected,mediate_deleted) {" [ -f "$APPARMOR_DIR/abstractions/base" ] && echo ' #include <abstractions/base>' cat <<'EOF' network, deny network ax25, deny network ipx, deny network appletalk, deny network netrom, deny network rose, deny network econet, deny network atmsvc, deny network irda, deny network wanpipe, deny network isdn, deny network caif, deny network alg, deny network vsock, capability, file, umount, signal (receive) peer=unconfined, signal (receive) peer=runc, signal (receive) peer=crun, EOF # docker kill comes from dockerd. Unconfined is already covered above. case "$daemon" in unconfined|'') ;; *[!A-Za-z0-9._/-]*) ;; *) echo " signal (receive) peer=$daemon," ;; esac cat <<EOF signal (send,receive) peer=$name, deny @{PROC}/* w, deny @{PROC}/{[^1-9/],[^1-9/][^0-9/],[^1-9s/][^0-9y/][^0-9s/],[^1-9/][^0-9/][^0-9/][^0-9/]*}/** w, EOF if [ "$variant" = nested ]; then # docker-default's "deny @{PROC}/sys/[^k]** w," minus /proc/sys/net, which # belongs to the workspace's own network namespace: dockerd and the inner # runc set sysctls there. cat <<'EOF' deny @{PROC}/sys/[^kn]** w, deny @{PROC}/sys/n[^e]** w, deny @{PROC}/sys/ne[^t]** w, deny @{PROC}/sys/net[^/]** w, EOF else echo ' deny @{PROC}/sys/[^k]** w,' fi cat <<'EOF' deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w, deny @{PROC}/sysrq-trigger rwklx, deny @{PROC}/kcore rwklx, EOF if [ "$variant" = nested ]; then # The inner dockerd mounts, and its containers pivot_root. The kernel's own # user-namespace rules still apply to every mount. cat <<'EOF' mount, remount, pivot_root, deny /sys/[^fn]*/** wklx, deny /sys/n[^e]*/** wklx, deny /sys/ne[^t]*/** wklx, deny /sys/net?*/** wklx, EOF # runc writes sysctls through a detached procfs, which AppArmor names # /sys/net/... Sysfs has no top-level net directory, so this opens nothing # of the host's. else cat <<'EOF' deny mount, deny /sys/[^f]*/** wklx, EOF fi cat <<EOF deny /sys/f[^s]*/** wklx, deny /sys/fs/[^c]*/** wklx, deny /sys/fs/c[^g]*/** wklx, deny /sys/fs/cg[^r]*/** wklx, deny /sys/firmware/** rwklx, deny /sys/devices/virtual/powercap/** rwklx, deny /sys/kernel/security/** rwklx, ptrace (trace,tracedby,read,readby) peer=$name, # DevGrail: every workspace's layers and volumes, and the control plane's # volumes, all in the one tree the shared UID range can traverse. deny $data{,/**} rwlkmx, # DevGrail: the exec roots, where the remapped one is owned by the range and # holds the stdio FIFOs of every workspace's docker exec. deny $exec_root{,.[0-9]*}{,/**} rwlkmx, } EOF } # load_apparmor_profile <name> <fallback> <data root> <exec root> <dockerd's label> [nested] # — write one profile, load it and keep it for the boot-time load. Succeeds # only once the kernel holds it and its file is in place; otherwise warns, # saying <fallback> is what the affected workspaces get instead. load_apparmor_profile() { local name="$1" fallback="$2" file tmp file="$APPARMOR_DIR/$name" # A dot file: if this run dies before the rename, the boot-time load of # $APPARMOR_DIR skips it. tmp="$APPARMOR_DIR/.$name.new" if ! { mkdir -p "$APPARMOR_DIR" && render_apparmor_profile "$3" "$4" "$5" "${6:-}" > "$tmp"; } 2>/dev/null; then rm -f "$tmp" warn "could not write $file — $fallback." return 1 fi if ! "$APPARMOR_PARSER" -r "$tmp" > "$tmp.err" 2>&1; then warn "could not load the AppArmor profile $name — $fallback." sed 's/^/ /' "$tmp.err" >&2 rm -f "$tmp" "$tmp.err" return 1 fi rm -f "$tmp.err" # Loaded but not persisted would be worse than not loaded: the workspaces # created with it would fail to start after the next reboot. if ! { chmod 644 "$tmp" && mv -f "$tmp" "$file"; }; then rm -f "$tmp" warn "loaded $name but could not write $file — $fallback." return 1 fi return 0 } # nested_docker_blocker <userns-remap active: yes|no> — why this host cannot # run Docker-enabled workspaces, or nothing when it can. Without userns-remap # a workspace's root is host root, and the relaxations the mode needs would # hand it the host. nested_docker_blocker() { if [ "$1" != yes ]; then echo "Docker does not remap user namespaces" elif [ ! -c "$FUSE_DEVICE" ]; then echo "the kernel offers no $FUSE_DEVICE" fi return 0 } # apply_apparmor <DockerRootDir> <daemon.json exec-root> [dockerd's label] [userns-remap active: yes|no] # — write both profiles and load them. WORKSPACE_APPARMOR_PROFILE is set once # the kernel holds devgrail-workspace, and WORKSPACE_NESTED_APPARMOR_PROFILE # once it holds the nested one too and the host passes nested_docker_blocker. # Never fatal: without the first, workspaces keep docker-default, as before; # without the second, Docker-enabled workspaces are not offered. apply_apparmor() { local data exec_root blocker WORKSPACE_APPARMOR_PROFILE="" WORKSPACE_NESTED_APPARMOR_PROFILE="" case "$APPARMOR_DECISION" in load) ;; noparser) warn "AppArmor is on, but apparmor_parser is not installed — workspaces keep Docker's docker-default profile." echo " Install the apparmor package and re-run to confine them further." >&2 return 0 ;; *) return 0 ;; esac data="$(apparmor_data_root "$1")" exec_root="$(apparmor_exec_root "$2")" if ! apparmor_root_ok "$data" || ! apparmor_root_ok "$exec_root"; then warn "Docker's data root ($data) or exec root ($exec_root) cannot go in an AppArmor rule — workspaces keep docker-default." return 0 fi load_apparmor_profile "$APPARMOR_PROFILE" "workspaces keep docker-default" \ "$data" "$exec_root" "${3:-}" || return 0 WORKSPACE_APPARMOR_PROFILE="$APPARMOR_PROFILE" log "Loaded the AppArmor profile $APPARMOR_PROFILE for workspaces (kept across reboots by $APPARMOR_DIR/$APPARMOR_PROFILE)." # Loaded whether or not the host is ready, so its file and the kernel's copy # never lag a re-run that makes it ready, and a reload reaches the # Docker-enabled workspaces already running. load_apparmor_profile "$APPARMOR_NESTED_PROFILE" "Docker-enabled workspaces are not offered" \ "$data" "$exec_root" "${3:-}" nested || return 0 blocker="$(nested_docker_blocker "${4:-no}")" if [ -n "$blocker" ]; then log "Docker-enabled workspaces are not offered: $blocker." return 0 fi WORKSPACE_NESTED_APPARMOR_PROFILE="$APPARMOR_NESTED_PROFILE" log "Loaded the AppArmor profile $APPARMOR_NESTED_PROFILE for Docker-enabled workspaces." return 0 } # --- end AppArmor workspace profile ------------------------------------------ # --- download + load images ------------------------------------------------ # TMP is chosen after the release is resolved, once the download size is known: # /tmp is a tmpfs on many VPS images (sized from RAM, often well under 1 GiB) # and a multi-hundred-megabyte image tarball does not fit. It is declared (and # the EXIT trap that removes it installed) up with the lock, so an abort at any # point between here and there still cleans up. # free_bytes <dir> — free space on the filesystem holding <dir>, in bytes. Prints # 0 when it cannot be determined (df on an unwritable path, an exotic # filesystem) — every caller treats 0 as "unknown, do not block on it", so a # preflight that cannot measure never becomes a preflight that refuses. free_bytes() { local v v="$(df -Pk "$1" 2>/dev/null | awk 'NR==2 {print $4 * 1024}')" || v="" case "$v" in ''|*[!0-9]*) v=0 ;; esac printf '%s' "$v" } # human <bytes> — MiB/GiB for a message an operator can act on. Integer shell # arithmetic only: this runs before jq is guaranteed and awk is already required. human() { local b="${1:-0}" if [ "$b" -ge 1073741824 ]; then printf '%s.%s GiB' "$((b / 1073741824))" "$(((b % 1073741824) * 10 / 1073741824))" elif [ "$b" -ge 1048576 ]; then printf '%s MiB' "$((b / 1048576))" else printf '%s bytes' "$b"; fi } # Prior install versions (for skip-if-unchanged upgrades). SERVER_RELEASE, # CONTAINER_RELEASE and ASSET_VERSIONS were added later, so an install that # predates them simply leaves these empty — reported as "unknown" or treated as # "not yet fetched", never as a failure. SERVER_VERSION=""; CONTAINER_VERSION="" SERVER_RELEASE=""; CONTAINER_RELEASE="" ASSET_VERSIONS="" RELEASE_VERSION="" # shellcheck source=/dev/null # written by a previous run of this script [ -f "$VERSIONS_FILE" ] && . "$VERSIONS_FILE" # RELEASE_VERSION is reused below for the release this run resolves, so the one # the file just supplied — what is installed *now* — is kept under its own name. PRIOR_RELEASE_VERSION="$RELEASE_VERSION" # asset_version_of <slug> — the version recorded for this slug at the last run, # or empty. ASSET_VERSIONS is a space-separated "slug=version" list, which keeps # it to one shell variable regardless of how many assets a release carries (slugs # are filenames, so they are not usable as variable names). asset_version_of() { # shellcheck disable=SC2086 # deliberate word splitting into one entry per line printf '%s\n' ${ASSET_VERSIONS:-} | awk -F= -v s="$1" '$1==s {print $2; exit}' } # --- weekly backup (decide) -------------------------------------------------- # The last question, asked here with the others rather than where the timer is # actually installed — which is after the images have come down, several minutes # into a run the operator would otherwise have been free to walk away from. # BACKUP_TIMER_DECISION is what install_backup_timer acts on later. # # Offered, never imposed: it writes to the customer's disk on a timer, and an # operator who already has their own backup story should not get a second one # they did not ask for. DEVGRAIL_SCHEDULED_BACKUPS=yes|no answers it unattended; # an existing timer is left exactly as configured. BACKUP_TIMER_DECISION=skip BACKUP_TIMER_NOTE="" if ! command -v systemctl >/dev/null 2>&1; then : # no systemd, no timer to offer elif [ -n "${DEVGRAIL_SCHEDULED_BACKUPS:-}" ]; then case "$(printf '%s' "$DEVGRAIL_SCHEDULED_BACKUPS" | tr '[:upper:]' '[:lower:]')" in yes|y|true|1) BACKUP_TIMER_DECISION=install ;; no|n|false|0) BACKUP_TIMER_DECISION=disable ;; *) die "DEVGRAIL_SCHEDULED_BACKUPS must be yes or no" ;; esac elif [ -f "$INSTALL_DIR/backups-host/policy.json" ] && [ -z "${DEVGRAIL_SCHEDULED_BACKUPS:-}" ]; then BACKUP_TIMER_DECISION=already BACKUP_TIMER_NOTE="Backups: existing dashboard schedule preserved. Manage in Administration -> Backups." elif systemctl list-unit-files devgrail-backup.timer >/dev/null 2>&1 && systemctl is-enabled devgrail-backup.timer >/dev/null 2>&1; then BACKUP_TIMER_DECISION=already BACKUP_TIMER_NOTE="Weekly backups: already scheduled (systemctl status devgrail-backup.timer)" else _want_timer="$(printf '%s' "${DEVGRAIL_SCHEDULED_BACKUPS:-}" | tr '[:upper:]' '[:lower:]')" # --yes takes the prompt's own default, which is yes: a machine nobody is # watching is exactly the one that benefits from a scheduled backup. # DEVGRAIL_SCHEDULED_BACKUPS still wins, so it stays possible to say no. if [ -z "$_want_timer" ] && [ "$ASSUME_YES" = true ]; then _want_timer=yes; fi if [ -z "$_want_timer" ] && [ "$DRY_RUN" != true ] && have_tty; then section "Weekly backups" echo " A systemd timer runs 'devgrail backup' once a week: the database plus" >&2 echo " the config.yaml whose key decrypts it, as one archive in $BACKUP_DIR" >&2 echo " on this same host. Usually a few MB, and taken without downtime." >&2 echo >&2 echo " What it does NOT cover: the files inside your workspaces — your" >&2 echo " source code lives in Docker volumes and is far too large to copy on a" >&2 echo " timer. Keep that in git. And a copy on this host does not survive" >&2 echo " losing this host, so for real disaster recovery add VPS snapshots or" >&2 echo " copy the archives off the box." >&2 echo >&2 echo " What it is for: getting the accounts, API keys, secrets and workspace" >&2 echo " records back after a bad change or a corrupted database." >&2 echo " devgrail list what archives exist here" >&2 echo " devgrail backup take one right now" >&2 echo " devgrail restore ARCHIVE put one back" >&2 echo " Upgrades snapshot themselves regardless of this answer; the timer is" >&2 echo " for the weeks in between. Keep-all is the default; manage archives and" >&2 echo " optional retention under Administration -> Backups." >&2 echo >&2 echo " Recommended: yes, unless you already back this host up yourself." >&2 if ask_yn " Schedule a weekly backup to $BACKUP_DIR?" y; then _want_timer=yes; else _want_timer=no; fi fi case "$_want_timer" in yes|y|true|1) BACKUP_TIMER_DECISION=install ;; *) BACKUP_TIMER_DECISION=skip ;; esac unset _want_timer fi # --- Docker ------------------------------------------------------------------ # Last thing before the run stops asking and starts doing. Deliberately after # every question above: installing Docker is the slow step, and an operator who # has just answered the last prompt should be able to leave. Nothing above it # needs Docker — a host without Docker has no Docker state, which is precisely # what the daemon preflight wanted to know. if ! command -v docker >/dev/null 2>&1; then log "Installing Docker..." # 9>&- keeps the install lock out of dockerd. Where systemd starts the daemon # it never sees our fds anyway, but on a host whose systemctl forks dockerd # directly (containers, the local sandbox) the daemon would inherit fd 9 and # hold the lock forever. acquire_lock can recover from that; not causing it is # cheaper. Plain `docker`/`docker compose` calls need no such guard — they are # short-lived clients, and workspace containers are children of dockerd. curl -fsSL https://get.docker.com | sh 9>&- systemctl enable --now docker 2>/dev/null 9>&- || true fi docker compose version >/dev/null 2>&1 || die "the 'docker compose' plugin is required" # --- resolve the release ---------------------------------------------------- # A DevGrail release is seven artifacts, not one. Resolving them all from a # single manifest is what stops an upgrade from mixing a new compose file with # an old server image: before this, the compose and Traefik config were # re-fetched unconditionally on every run because there was no asset version to # compare against, so even a "nothing changed" re-run silently adopted whatever # had been published since. DEVGRAIL_VERSION="${DEVGRAIL_VERSION:-current}" RELEASE_MANIFEST="" RELEASE_VERSION="" resolve_release() { local body body="$(curl -fsSL "$WEB/api/registry/releases/$DEVGRAIL_VERSION" 2>/dev/null)" || body="" if [ -z "$body" ]; then # An explicit version that does not resolve is an error — silently # installing something else is the one thing --version must never do. [ "$DEVGRAIL_VERSION" = current ] \ || die "release '$DEVGRAIL_VERSION' is not published at $WEB. Nothing has been changed." # No release manifest at all means a registry older than this installer. # Fall back to resolving each artifact by slug, exactly as before. log "Registry publishes no release manifest — resolving each artifact individually." return 0 fi RELEASE_MANIFEST="$body" RELEASE_VERSION="$(printf '%s' "$body" | jq -r '.release_version // empty')" log "Installing release ${RELEASE_VERSION:-unknown}" } resolve_release # --- release signature ------------------------------------------------------- # Every artifact is verified against a SHA-256 the registry publishes — which # catches a corrupted download but not a compromised registry, because whoever # can serve the bytes can serve the matching digest. The same origin also serves # this script, the compose file and the Traefik config, all executed as root. # # A detached signature over the release closes that, and only if the signing key # never touches the web host. One signature covers all seven artifacts: that is # the payoff of resolving a release from a single manifest. # # What is signed is a payload listing every artifact's kind, slug, version and # digest — not the manifest itself, whose download URLs are rewritten per origin # and so cannot be signed offline. Verification therefore proves the digests, # and the digests prove the bytes. See docs/SECURITY.md. # Public keys this installer trusts to sign a release. Generated by # scripts/release-key.sh; the private half is never in this repo. # # Rotation: add the new key here and ship an installer carrying both, then start # signing with the new one, then drop the old key a release later. Hosts that # never upgraded the installer keep working throughout. release_public_keys() { release_embedded_public_keys release_extra_public_keys } # The keys compiled into this installer. Split from the extras below because # the dashboard upgrade helper stores the two apart: the embedded set is # replaced by each installer that runs, so a rotated-out key actually goes. release_embedded_public_keys() { # key id f1f1eaf642454d65 — first embedded in v0.4.0 cat <<'PUBKEYS' -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEK1NbYrxBM8Ia1iM4BcR0m/8DelN5 ZDvOwKmEOJA24dci3ftqHWwsxzbw8bACRps0rIOTqa45wblm6260fQVdIw== -----END PUBLIC KEY----- PUBKEYS } # An operator running their own registry (or the test harness) can add keys # without editing this file. This grants trust rather than removing it, and # anyone able to set it already runs this script as root. release_extra_public_keys() { if [ -n "${DEVGRAIL_RELEASE_PUBKEY_FILE:-}" ] && [ -f "$DEVGRAIL_RELEASE_PUBKEY_FILE" ]; then cat "$DEVGRAIL_RELEASE_PUBKEY_FILE" fi } # The verified payload, once there is one. Empty means nothing is enforced # beyond the registry's own digests. SIGNED_PAYLOAD_FILE="" SIG_DIR="" # unverified <why> — how a release that cannot be checked is treated. See # ALLOW_UNSIGNED, resolved up with the tooling because openssl's absence is # the one form of "cannot be checked" that is knowable before any of this. unverified() { if [ "$ALLOW_UNSIGNED" != 1 ]; then die "refusing to install a release whose signature could not be verified: $1 Artifacts are checked against digests this registry publishes, which catches a corrupted download but not a compromised registry — whoever can serve the bytes can serve the matching digest. Only the signature tells those apart. Set DEVGRAIL_ALLOW_UNSIGNED=1 to install anyway, accepting that this host is trusting $WEB and nothing else. See docs/SECURITY.md. Nothing has been changed." fi warn "release signature not verified — $1" echo " DEVGRAIL_ALLOW_UNSIGNED=1 is set, so this run continues. Artifacts are" >&2 echo " still checked against the digests this registry publishes, which detects" >&2 echo " a corrupted download but not a compromised registry. See docs/SECURITY.md." >&2 } verify_release_signature() { local sig alg kid keys dir pem verified=false if [ -z "$RELEASE_MANIFEST" ]; then unverified "this registry publishes no release manifest" return 0 fi sig="$(printf '%s' "$RELEASE_MANIFEST" | jq -c '.signature // empty' 2>/dev/null || true)" if [ -z "$sig" ] || [ "$sig" = null ]; then unverified "release ${RELEASE_VERSION:-current} was published without one" return 0 fi keys="$(release_public_keys)" if [ -z "$keys" ]; then unverified "this installer carries no release signing key" return 0 fi if ! command -v openssl >/dev/null 2>&1; then unverified "openssl is not installed here, so the signature cannot be checked" return 0 fi alg="$(printf '%s' "$sig" | jq -r '.algorithm // empty')" kid="$(printf '%s' "$sig" | jq -r '.key_id // empty')" # An unknown algorithm is not the same as an unsigned release: the publisher # signed it with something this installer cannot check, and treating that as # "unsigned" would let an attacker downgrade verification by naming a scheme # we do not implement. [ "$alg" = "ecdsa-p256-sha256" ] \ || die "release ${RELEASE_VERSION:-current} is signed with '$alg', which this installer cannot verify. Upgrade the installer. Nothing has been changed." dir="$(mktemp -d)" SIG_DIR="$dir" # -j, not -r: the signature is over exactly the published bytes, and -r adds a # trailing newline the signer never saw. That difference verifies as tampering. printf '%s' "$sig" | jq -j '.payload' > "$dir/payload" printf '%s' "$sig" | jq -r '.signature' | base64 -d > "$dir/sig" 2>/dev/null \ || die "the signature on release ${RELEASE_VERSION:-current} is not valid base64. Nothing has been changed." # Each embedded key gets its own file; any one of them verifying is enough, # which is what makes rotation a matter of shipping two keys for a while. printf '%s\n' "$keys" > "$dir/keys.pem" awk -v d="$dir" '/-----BEGIN PUBLIC KEY-----/{n++} n>0 {print > (d "/key" n ".pem")}' "$dir/keys.pem" for pem in "$dir"/key*.pem; do [ -f "$pem" ] || continue if openssl dgst -sha256 -verify "$pem" -signature "$dir/sig" "$dir/payload" >/dev/null 2>&1; then verified=true break fi done if [ "$verified" != true ]; then die "the signature on release ${RELEASE_VERSION:-current} does not verify against any key this installer carries (signing key id: ${kid:-unknown}). That is either a corrupted download or a tampered release, and there is no way to tell which from here. Nothing has been changed." fi # A valid signature over somebody else's payload proves nothing about this # release, so the payload has to name it — otherwise an old signed release # could be replayed against a host asking for a newer one. if ! grep -qxF "devgrail-release-signature-v1" "$dir/payload"; then die "the release signature verifies but its payload is not in a format this installer understands. Nothing has been changed." fi if ! grep -qxF "release ${RELEASE_VERSION}" "$dir/payload"; then die "the signature verifies, but it was made over a different release than ${RELEASE_VERSION:-the one requested}. Nothing has been changed." fi SIGNED_PAYLOAD_FILE="$dir/payload" log "Release ${RELEASE_VERSION} signature verified (key ${kid:-unknown})." } # signed_digest_check <payload line> <what> — refuse anything the signed payload # does not name. Called after the registry's own digest matched, so this is the # check that survives a compromised registry: matching bytes are not enough if # they are not the bytes the release was signed for. signed_digest_check() { [ -n "$SIGNED_PAYLOAD_FILE" ] || return 0 if grep -qxF "$1" "$SIGNED_PAYLOAD_FILE"; then return 0 fi die "'$2' is not what release $RELEASE_VERSION was signed for. Its bytes match the digest this registry served, but that digest does not appear in the signed release payload: $1 Refusing to install it. Nothing has been changed." } verify_release_signature # release_field <slug> <jq filter> — a field of one artifact in the release # manifest, or empty when there is no manifest or no such artifact. release_field() { [ -n "$RELEASE_MANIFEST" ] || return 0 printf '%s' "$RELEASE_MANIFEST" | jq -r --arg s "$1" --arg a "$ARCH" "$2" 2>/dev/null || true } # same_image <ref> <ref> — true when both names resolve to one image. Either # name missing is "not the same", which is the answer that makes callers do the # safe thing. same_image() { local a b a="$(docker image inspect -f '{{.Id}}' "$1" 2>/dev/null)" || return 1 b="$(docker image inspect -f '{{.Id}}' "$2" 2>/dev/null)" || return 1 [ -n "$a" ] && [ "$a" = "$b" ] } # resolve_image <slug> <target_tag> <current_version> — settle which build of # <slug> this run needs, and whether it has to be downloaded at all, without # fetching a byte. Sets R_VER R_REL R_URL R_SHA R_SIZE R_TAG R_SKIP R_RETAG. # # Split out from download_image so the disk preflight below can total up what is # about to be written *before* the first download starts. Running out of space # halfway through is precisely the condition that causes a mid-migration # failure, so discovering it after one image has already landed is not good # enough. R_VER=""; R_REL=""; R_URL=""; R_SHA=""; R_SIZE=0; R_TAG=""; R_SKIP=false; R_RETAG=false resolve_image() { local slug="$1" target="$2" cur="$3" manifest # Prefer the release manifest: it pins the exact version this release is, and # its download URLs carry that pin so they keep resolving to these bytes even # after a newer release is published. R_VER="$(release_field "$slug" '.images[]|select(.slug==$s)|.image_version')" R_URL="$(release_field "$slug" '.images[]|select(.slug==$s)|.artifacts[]|select(.arch==$a)|.download_url')" R_SHA="$(release_field "$slug" '.images[]|select(.slug==$s)|.artifacts[]|select(.arch==$a)|.sha256')" R_SIZE="$(release_field "$slug" '.images[]|select(.slug==$s)|.artifacts[]|select(.arch==$a)|.size')" # The release this *image* was built under, which is not always the release # being installed. An image whose build context did not change is carried into # later releases at the version — and under the tarball tag — it originally # shipped with, so the tag to look for on this host is the image's own, not # this run's. Absent from a registry predating the field, where the two were # always the same thing anyway. R_REL="$(release_field "$slug" '.images[]|select(.slug==$s)|.image_release // empty')" [ -n "$R_REL" ] && [ "$R_REL" != "null" ] || R_REL="$RELEASE_VERSION" if [ -z "$R_VER" ]; then manifest="$(curl -fsSL "$WEB/api/registry/images/$slug")" \ || die "cannot fetch image manifest for '$slug' from $WEB" R_VER="$(printf '%s' "$manifest" | jq -r '.image_version')" # Additive field: a registry that predates it, or a row uploaded before it # existed, yields null — which becomes an empty string, not an error. R_REL="$(printf '%s' "$manifest" | jq -r '.release_version // empty')" R_URL="$(printf '%s' "$manifest" | jq -r --arg a "$ARCH" '.artifacts[]|select(.arch==$a)|.download_url')" R_SHA="$(printf '%s' "$manifest" | jq -r --arg a "$ARCH" '.artifacts[]|select(.arch==$a)|.sha256')" R_SIZE="$(printf '%s' "$manifest" | jq -r --arg a "$ARCH" '.artifacts[]|select(.arch==$a)|.size')" fi [ -n "$R_URL" ] && [ "$R_URL" != "null" ] || die "no $ARCH build published for '$slug'" # A registry that does not declare sizes yields null; treat that as "unknown" # (0) rather than failing — the preflight then simply has less to check. case "$R_SIZE" in ''|null|*[!0-9]*) R_SIZE=0 ;; esac # The tag `docker load` gave these exact bytes on the run that fetched them: # publish.sh builds every image as <slug>:<release>. Empty for a registry old # enough to publish no release version, which leaves the checks below on their # pre-release behaviour rather than guessing at a tag. R_TAG="" case "$R_REL" in ''|null) ;; *) R_TAG="$slug:$R_REL" ;; esac # Skipping requires that $target *is* this version — not merely that something # is tagged $target. `--rollback` points :latest back at the displaced image # and leaves versions.env naming the new one (it is written before the health # gate, so the rollback cannot un-write it). Testing existence alone made every # later run report "already at vX — skipping" for a version the host had rolled # off: it stayed on the old binary while versions.env and this script's own # closing banner both claimed the new one, /healthz was the only thing telling # the truth, and there was no way forward short of deleting versions.env. R_SKIP=false R_RETAG=false if [ -n "$cur" ] && [ "$cur" = "$R_VER" ] && docker image inspect "$target" >/dev/null 2>&1; then if [ -z "$R_TAG" ] || same_image "$target" "$R_TAG"; then R_SKIP=true elif docker image inspect "$R_TAG" >/dev/null 2>&1; then # The bytes are still on this host under their release tag — downloaded, # checksum- and signature-checked by the run that loaded them. Moving the # tag back is the whole recovery; re-fetching 750 MiB to arrive at the # image already sitting there is not. R_RETAG=true fi fi } LOADED_VERSION=""; LOADED_RELEASE="" # retag_target <target> <source_ref> — point <target> at <source_ref>, naming # whatever it displaced :previous first. # # Retagging moves :latest off the image currently running, which then survives # only as an untagged layer set until someone prunes it. Naming it :previous is # a tag, not a copy — so it costs nothing, and rollback becomes a retag plus a # compose recreate rather than a re-download. retag_target() { local target="$1" src="$2" prev_tag old_id new_id prev_tag="${target%:*}:previous" old_id="$(docker image inspect -f '{{.Id}}' "$target" 2>/dev/null || true)" new_id="$(docker image inspect -f '{{.Id}}' "$src" 2>/dev/null || true)" if [ -n "$old_id" ] && [ "$old_id" != "$new_id" ]; then docker tag "$old_id" "$prev_tag" log "Previous ${target%:*} kept as $prev_tag" fi docker tag "$src" "$target" } # download_image <slug> <target_tag> <current_version> [current_release] # Uses the R_* values resolve_image already settled for this slug. download_image() { local slug="$1" target="$2" cur="$3" cur_rel="${4:-}" loaded got tar local url="$R_URL" sha="$R_SHA" ver="$R_VER" rel="$R_REL" if [ "$R_SKIP" = true ]; then # Two different situations, and an operator has to be able to tell them # apart. Re-running the same release is "already at it". A *newer* release # that ships this image unchanged keeps an image whose release name is # older than the one being installed — which reads like a failed upgrade # unless it says why. if [ -n "$rel" ] && [ -n "$RELEASE_VERSION" ] && [ "$rel" != "$RELEASE_VERSION" ]; then log "$slug is unchanged in $RELEASE_VERSION — keeping $rel, nothing to download" else log "$slug already at ${rel:-v$ver} — skipping" fi LOADED_VERSION="$ver" # Prefer what the registry says now: it fills in the release for an install # that was made before the field existed, without re-downloading anything. LOADED_RELEASE="${rel:-$cur_rel}" return fi if [ "$R_RETAG" = true ]; then log "$slug ${rel:-v$ver} is loaded but $target points elsewhere — restoring it" retag_target "$target" "$R_TAG" LOADED_VERSION="$ver"; LOADED_RELEASE="$rel" return fi tar="$TMP/$slug.tar" log "Downloading $slug ${rel:-v$ver} ($ARCH)..." curl -fL --progress-bar -o "$tar" "$url" || die "download failed for '$slug'" log "Verifying checksum..." got="$(sha256_of "$tar")" [ "$got" = "$sha" ] || die "checksum mismatch for '$slug' (expected $sha, got $got)" signed_digest_check "image $slug $ver $ARCH $got" "$slug ($ARCH)" log "Loading image..." # Not `| head -n1`: head exits after one line, sed dies of SIGPIPE, and # `set -o pipefail` turns that into a fatal exit 141 — mid-upgrade, right after # the image was loaded, with no message. sed reads all of docker load's output # here; the first line is taken with parameter expansion afterwards. loaded="$(docker load -i "$tar" | sed -n 's/^Loaded image: //p')" loaded="${loaded%%$'\n'*}" [ -n "$loaded" ] || die "docker load produced no image for '$slug'" retag_target "$target" "$loaded" rm -f "$tar" LOADED_VERSION="$ver"; LOADED_RELEASE="$rel" } # Resolve both images first, so the preflight sees the whole run. resolve_image devgrail-server "devgrail-server:latest" "$SERVER_VERSION" SERVER_SKIP="$R_SKIP"; SERVER_RETAG="$R_RETAG"; SERVER_BYTES="$R_SIZE" SERVER_R_VER="$R_VER"; SERVER_R_REL="$R_REL"; SERVER_R_URL="$R_URL"; SERVER_R_SHA="$R_SHA"; SERVER_R_TAG="$R_TAG" # --workspace-image=no does not resolve it at all, rather than resolving it and # then ignoring the answer: resolve_image talks to the registry and dies when the # release publishes no build for this architecture, which is not a reason to # refuse an install that was never going to download it. CONTAINER_SKIP=false; CONTAINER_RETAG=false; CONTAINER_BYTES=0 CONTAINER_R_VER=""; CONTAINER_R_REL=""; CONTAINER_R_URL=""; CONTAINER_R_SHA=""; CONTAINER_R_TAG="" if [ "$DEVGRAIL_WORKSPACE_IMAGE" = yes ]; then resolve_image devgrail-container "devgrail-container:latest" "$CONTAINER_VERSION" CONTAINER_SKIP="$R_SKIP"; CONTAINER_RETAG="$R_RETAG"; CONTAINER_BYTES="$R_SIZE" CONTAINER_R_VER="$R_VER"; CONTAINER_R_REL="$R_REL"; CONTAINER_R_URL="$R_URL"; CONTAINER_R_SHA="$R_SHA"; CONTAINER_R_TAG="$R_TAG" fi # --- disk preflight --------------------------------------------------------- # Every upgrade downloads a tarball and then needs the space again in the Docker # data root when `docker load` unpacks it, and nothing prunes. Disk-full is not # merely untidy here: it is the condition that produces a failure mid-migration, # which is the worst state this product can be in. So the numbers are checked # before the first byte is fetched, and reported when they do not add up. # # 2.5x the declared download size: 1x for the tarball, ~1x again for the # unpacked layers, plus half for the compressed layers Docker keeps while # loading and for headroom. A registry that declares no sizes yields 0, and the # check for that filesystem is skipped rather than guessed at. DOWNLOAD_BYTES=0 # A retag fetches nothing either — the bytes are already in the Docker data root. [ "$SERVER_SKIP" = true ] || [ "$SERVER_RETAG" = true ] || DOWNLOAD_BYTES=$((DOWNLOAD_BYTES + SERVER_BYTES)) # So does a workspace image this run is not fetching. CONTAINER_BYTES is already # 0 in that case; the test is here so the intent survives a later edit. if [ "$DEVGRAIL_WORKSPACE_IMAGE" = yes ]; then [ "$CONTAINER_SKIP" = true ] || [ "$CONTAINER_RETAG" = true ] || DOWNLOAD_BYTES=$((DOWNLOAD_BYTES + CONTAINER_BYTES)) fi NEED_BYTES=$((DOWNLOAD_BYTES * 5 / 2)) # --- dry run ----------------------------------------------------------------- # "Re-run the same script that did the first install" is a fine upgrade # mechanism right up until the operator wants to know what it will do before it # does it. Everything above this line is resolution — reading the registry and # the prior install's own files — so the report can be produced with nothing # downloaded, nothing written and nothing started. # # It sits here rather than earlier because the release has to be resolved for # any of it to be true, and earlier than the next line because that is where the # first byte gets staged on disk. # dry_change <from> <to> — "a -> b", or "b (unchanged)" when they match. dry_change() { local from="${1:-}" to="${2:-}" [ -n "$to" ] || to="unknown" if [ -z "$from" ]; then printf '%s (new)' "$to" elif [ "$from" = "$to" ]; then printf '%s (unchanged)' "$to" else printf '%s -> %s' "$from" "$to"; fi } dry_image_line() { local slug="$1" cur="$2" ver="$3" rel="$4" skip="$5" bytes="$6" retag="${7:-false}" if [ "$skip" = true ]; then # The release is named here because it need not be the one being installed: # an image carried unchanged into a newer release keeps its own. printf ' %-20s v%s%s (unchanged)\n' "$slug" "$ver" "${rel:+ $rel}" elif [ "$retag" = true ]; then # Not "(unchanged)": the version is the one recorded, but :latest is not on # it — which is exactly the state an operator needs told, since it is what # a rollback leaves behind and what this run would undo. printf ' %-20s v%s%s already loaded — would point %s:latest back at it\n' \ "$slug" "$ver" "${rel:+ $rel}" "$slug" else printf ' %-20s %s%s — would download %s\n' "$slug" \ "$(dry_change "${cur:+v$cur}" "v$ver")" "${rel:+ $rel}" "$(human "$bytes")" fi } dry_asset_line() { local slug="$1" ver have ver="$(release_field "$slug" '.assets[]|select(.slug==$s)|.asset_version')" have="$(asset_version_of "$slug")" if [ -z "$ver" ]; then printf ' %-24s would be re-fetched (the registry publishes no version for it)\n' "$slug" elif [ "$ver" = "$have" ] && [ -f "$ASSET_DIR/$slug" ]; then printf ' %-24s v%s (unchanged)\n' "$slug" "$ver" else printf ' %-24s %s — would download\n' "$slug" "$(dry_change "${have:+v$have}" "v$ver")" fi } dry_run_report() { local tmpdir="${TMPDIR:-/tmp}" docker_root key was now changed=false health schema echo echo "============================================================" echo " Dry run — nothing on this host has been changed." echo if [ "$IS_UPGRADE" = true ]; then echo " Mode: upgrade of the install at $INSTALL_DIR" else echo " Mode: first install into $INSTALL_DIR" fi echo " Registry: $WEB" echo " Release: $(dry_change "$PRIOR_RELEASE_VERSION" "${RELEASE_VERSION:-not published as a release}")" if [ -n "$SIGNED_PAYLOAD_FILE" ]; then echo " Signature: verified against a key embedded in this installer" else echo " Signature: not verified (see the warning above)" fi echo echo " Images" dry_image_line devgrail-server "$SERVER_VERSION" "$SERVER_R_VER" \ "$SERVER_R_REL" "$SERVER_SKIP" "$SERVER_BYTES" "$SERVER_RETAG" if [ "$DEVGRAIL_WORKSPACE_IMAGE" = yes ]; then dry_image_line devgrail-container "$CONTAINER_VERSION" "$CONTAINER_R_VER" \ "$CONTAINER_R_REL" "$CONTAINER_SKIP" "$CONTAINER_BYTES" "$CONTAINER_RETAG" elif docker image inspect devgrail-container:latest >/dev/null 2>&1; then # Worth distinguishing: "skipped" on a host that already has the image reads # like something is about to break, and nothing is. printf ' %-20s %s\n' devgrail-container \ "not fetched (--workspace-image=no) — the copy already on this host is kept" else printf ' %-20s %s\n' devgrail-container \ "skipped (--workspace-image=no) — no workspace can be created until it is fetched" fi echo echo " Assets" # Only the two this run would actually fetch. install.sh and uninstall.sh are # published as assets too, but they are served to `curl`, never downloaded # here, and listing them would suggest otherwise. dry_asset_line docker-compose.yml dry_asset_line "$TRAEFIK_ASSET" echo echo " Disk" if [ "$NEED_BYTES" -gt 0 ]; then printf ' would need %s (2.5x the %s to download)\n' \ "$(human "$NEED_BYTES")" "$(human "$DOWNLOAD_BYTES")" else echo " nothing to download" fi printf ' free in %s: %s\n' "$tmpdir" "$(human "$(free_bytes "$tmpdir")")" docker_root="$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || true)" [ -n "$docker_root" ] && [ -d "$docker_root" ] || docker_root=/var/lib/docker if [ -d "$docker_root" ]; then printf ' free in %s: %s\n' "$docker_root" "$(human "$(free_bytes "$docker_root")")" fi if [ "$INOTIFY_DECISION" = raise ]; then echo echo " Kernel" printf ' %s would be raised %s -> %s, and kept in %s\n' \ "$INOTIFY_KEY" "$INOTIFY_NOW" "$INOTIFY_TARGET" "$INOTIFY_CONF" fi case "$APPARMOR_DECISION" in load) echo echo " AppArmor" echo " workspaces created from now on would run under $APPARMOR_PROFILE," echo " written to $APPARMOR_DIR/$APPARMOR_PROFILE and loaded" echo " Docker-enabled ones under $APPARMOR_NESTED_PROFILE, where userns-remap" echo " is on and $FUSE_DEVICE exists" ;; noparser) echo echo " AppArmor" echo " apparmor_parser is not installed, so workspaces would keep docker-default" ;; esac if [ "$IS_UPGRADE" = true ]; then echo echo " Settings" # DEVGRAIL_ADMIN_USERNAME is listed; the password never is. for key in DEVGRAIL_DOMAIN DEVGRAIL_BASE_DOMAIN ACME_EMAIL DEVGRAIL_ACME_CASERVER DEVGRAIL_ACME_BYPASS DEVGRAIL_ADMIN_USERNAME DEVGRAIL_TLS DEVGRAIL_WORKSPACE_IMAGE DEVGRAIL_DASHBOARD_UPGRADES HTTP_PORT HTTPS_PORT; do was="$(prior "$key")" eval "now=\${$key:-}" if [ "$was" != "$now" ]; then printf ' %-22s %s -> %s\n' "$key" "${was:-(unset)}" "${now:-(unset)}" changed=true fi done if [ "$changed" = false ]; then echo " unchanged — every value in the prior .env would be preserved" fi echo echo " Data" echo " a snapshot of the database and config.yaml would be written to" echo " $BACKUP_DIR before the new server starts (newest ${DEVGRAIL_SNAPSHOT_RETAIN:-5} kept)" health="$(curl -fsSk --max-time 5 "$(dashboard_url)/api/v1/healthz" 2>/dev/null || true)" schema="$(printf '%s' "$health" | jq -r '.schema_version.applied // empty' 2>/dev/null || true)" if [ -n "$schema" ]; then echo " this database is at schema version $schema"; fi echo " any pending migrations are applied by the new server when it starts;" echo " which ones cannot be listed without loading that image, and a dry" echo " run does not download it" fi echo echo " Re-run without --dry-run to apply this." echo "============================================================" echo return 0 } decide_inotify decide_apparmor if [ "$DRY_RUN" = true ]; then dry_run_report exit 0 fi # --- the plan ---------------------------------------------------------------- # Everything above this line asked questions or read something. Everything below # it changes this host. So this is where the run says what it is about to do, in # one screen, and — when there is somebody to ask — gets one confirmation for the # lot. That is the point the operator can walk away from. # # Unattended runs (--yes, or no terminal) print the same summary and proceed: # the plan is as useful in a cloud-init log as it is on a terminal, and a build # that cannot answer must not be stopped by a question. # 13 wide because "Certificates:" is the longest label; a narrower column # would push that one row out of line with the rest. plan_line() { printf ' %-13s %s\n' "$1" "$2"; } # The phases the apply half will walk through, counted now that every condition # that adds or removes one has been settled. Five are unconditional: fetch the # images, write the deployment, install the CLI, start the stack, wait for it to # answer. The rest depend on this particular run. STEP_TOTAL=5 if [ "$USERNS_DECISION" = enable ] || [ "$POOLS_DECISION" = widen ]; then STEP_TOTAL=$((STEP_TOTAL + 1)) fi [ "$IS_UPGRADE" = true ] && STEP_TOTAL=$((STEP_TOTAL + 1)) [ "$BACKUP_TIMER_DECISION" = install ] && STEP_TOTAL=$((STEP_TOTAL + 1)) echo echo "============================================================" if [ "$IS_UPGRADE" = true ]; then echo " Ready to upgrade DevGrail at $INSTALL_DIR" else echo " Ready to install DevGrail into $INSTALL_DIR" fi echo plan_line "Release:" "$(dry_change "$PRIOR_RELEASE_VERSION" "${RELEASE_VERSION:-unversioned}") from $WEB" plan_line "Dashboard:" "$(dashboard_url)/" if [ "$TLS_ENABLED" = true ]; then if [ "$ACME_CA_IS_PRODUCTION" = true ]; then plan_line "Certificates:" "Let's Encrypt, contact ${ACME_EMAIL:-unset}" else plan_line "Certificates:" "NON-PRODUCTION ACME CA — browsers will not trust them" fi if [ "$DEVGRAIL_ACME_BYPASS" = true ]; then plan_line "ACME bypass:" "on — passthrough routes receive their own challenges" fi else plan_line "Certificates:" "none — plain HTTP (DEVGRAIL_TLS=off)" fi plan_line "Subdomains:" "*.$DEVGRAIL_BASE_DOMAIN" if [ "$IS_UPGRADE" = true ]; then plan_line "Admin:" "unchanged — the account already exists" elif [ "$ADMIN_PW_GENERATED" = true ]; then plan_line "Admin:" "$ADMIN_USER, with a generated password shown at the end" else plan_line "Admin:" "$ADMIN_USER, with the password you entered" fi echo if [ "$DOWNLOAD_BYTES" -gt 0 ]; then plan_line "Download:" "$(human "$DOWNLOAD_BYTES") ($(human "$NEED_BYTES") of disk needed to unpack it)" else plan_line "Download:" "nothing — every artifact this release names is already here" fi if [ "$DEVGRAIL_WORKSPACE_IMAGE" != yes ]; then plan_line "" "workspace image skipped (--workspace-image=no)" fi if [ "$IS_UPGRADE" = true ]; then plan_line "Snapshot:" "database + config.yaml into $BACKUP_DIR before the new server starts" fi # Only mentioned when something is going to happen to the daemon. A run that # changes nothing there should not spend two lines saying so. if [ "$USERNS_DECISION" = enable ] || [ "$POOLS_DECISION" = widen ]; then echo [ "$USERNS_DECISION" = enable ] && plan_line "Docker:" "enable userns-remap" [ "$POOLS_DECISION" = widen ] && plan_line "Docker:" "widen the default address pools to $POOL_BASE" plan_line "" "both edit $DAEMON_JSON and restart Docker" fi if [ "$INOTIFY_DECISION" = raise ]; then echo plan_line "Kernel:" "raise $INOTIFY_KEY $INOTIFY_NOW -> $INOTIFY_TARGET ($INOTIFY_CONF)" fi # Only on the run that first loads it; every later run reloads it quietly. _aa_label="AppArmor:" if [ "$APPARMOR_DECISION" = load ] && [ ! -f "$APPARMOR_DIR/$APPARMOR_PROFILE" ]; then echo plan_line "$_aa_label" "load the $APPARMOR_PROFILE profile for new workspaces ($APPARMOR_DIR/$APPARMOR_PROFILE)" _aa_label="" fi if [ "$APPARMOR_DECISION" = load ] && [ ! -f "$APPARMOR_DIR/$APPARMOR_NESTED_PROFILE" ]; then [ -n "$_aa_label" ] && echo plan_line "$_aa_label" "load the $APPARMOR_NESTED_PROFILE profile for Docker-enabled workspaces ($APPARMOR_DIR/$APPARMOR_NESTED_PROFILE)" fi unset _aa_label case "$BACKUP_TIMER_DECISION" in install) plan_line "Backups:" "weekly systemd timer into $BACKUP_DIR" ;; already) plan_line "Backups:" "weekly timer already scheduled — left as configured" ;; esac echo "============================================================" echo if have_tty && [ "$ASSUME_YES" != true ]; then if ! ask_yn "Proceed?" y; then die "aborted at the plan. Nothing on this host has been changed." fi echo fi # Before the first download, for the reason apply_daemon_decisions documents: # enabling userns-remap moves the daemon's data root out from under any image # loaded ahead of it. if [ "$USERNS_DECISION" = enable ] || [ "$POOLS_DECISION" = widen ]; then step "Applying the Docker daemon settings..." fi apply_daemon_decisions apply_inotify # After the daemon decisions: enabling userns-remap moves the data root, and # whether it is now on decides if Docker-enabled workspaces are offered. The # label is dockerd's own confinement, which docker kill signals from. apply_apparmor "$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || true)" \ "$(jq -r '."exec-root" // empty' "$DAEMON_JSON" 2>/dev/null || true)" \ "$(sed 's/ (.*//' "/proc/$(pgrep -xo dockerd 2>/dev/null || echo 0)/attr/current" 2>/dev/null || true)" \ "$(if userns_remap_enabled; then echo yes; else echo no; fi)" # Where the tarballs are staged. /tmp is a tmpfs on many VPS images — sized from # RAM and often well under a gigabyte — so falling back to $INSTALL_DIR/tmp when # it does not fit turns "no space left on device" partway through a download # into a directory that has the room. choose_tmp() { local candidate="${TMPDIR:-/tmp}" fallback="$INSTALL_DIR/tmp" have have_fb have="$(free_bytes "$candidate")" # 0 means "could not measure", which is not the same as "no room". Nothing # here is worth refusing an upgrade over on a guess. if [ "$NEED_BYTES" -gt 0 ] && [ "$have" -gt 0 ] && [ "$have" -lt "$NEED_BYTES" ]; then mkdir -p "$fallback" || die "cannot create $fallback for the download" have_fb="$(free_bytes "$fallback")" if [ "$have_fb" -gt 0 ] && [ "$have_fb" -lt "$NEED_BYTES" ]; then die "not enough free space to download this release. Needed: $(human "$NEED_BYTES") (2.5x the $(human "$DOWNLOAD_BYTES") of artifacts) Free in $candidate: $(human "$have") Free in $fallback: $(human "$have_fb") Free some space and re-run. Nothing has been changed." fi log "$candidate is too small for this download — staging in $fallback instead" TMP="$(mktemp -d "$fallback/install.XXXXXX")" else TMP="$(mktemp -d)" fi } choose_tmp if [ "$NEED_BYTES" -gt 0 ]; then DOCKER_ROOT="$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || true)" [ -n "$DOCKER_ROOT" ] && [ -d "$DOCKER_ROOT" ] || DOCKER_ROOT=/var/lib/docker if [ -d "$DOCKER_ROOT" ]; then DOCKER_FREE="$(free_bytes "$DOCKER_ROOT")" if [ "$DOCKER_FREE" -gt 0 ] && [ "$DOCKER_FREE" -lt "$NEED_BYTES" ]; then die "not enough free space in the Docker data root to load this release. Needed: $(human "$NEED_BYTES") (2.5x the $(human "$DOWNLOAD_BYTES") of artifacts) Free in $DOCKER_ROOT: $(human "$DOCKER_FREE") Reclaim space (\`docker image prune\`, \`docker system df\`) and re-run. Nothing has been changed." fi fi fi if [ "$DOWNLOAD_BYTES" -gt 0 ]; then step "Fetching the DevGrail images ($(human "$DOWNLOAD_BYTES"))..." else step "Checking the DevGrail images..." fi R_SKIP="$SERVER_SKIP"; R_RETAG="$SERVER_RETAG"; R_TAG="$SERVER_R_TAG" R_VER="$SERVER_R_VER"; R_REL="$SERVER_R_REL"; R_URL="$SERVER_R_URL"; R_SHA="$SERVER_R_SHA" download_image devgrail-server "devgrail-server:latest" "$SERVER_VERSION" "$SERVER_RELEASE" NEW_SERVER_VERSION="$LOADED_VERSION"; NEW_SERVER_RELEASE="$LOADED_RELEASE" # The workspace image tag is hard-coded in internal/docker (DefaultImage = # devgrail-container:latest), so retag the loaded release to :latest. # # Skipped by --workspace-image=no, which carries the prior versions forward # verbatim instead of recording anything. That is right in both directions: a # host that never fetched the image keeps two empty values, so the run that does # fetch it sees no current version and downloads; a host that fetched it on an # earlier run keeps versions.env describing the image still sitting on its disk. # Note it does NOT go through download_image with R_SKIP=true — that path logs # "already at ..." and would be a lie on the first case. if [ "$DEVGRAIL_WORKSPACE_IMAGE" = yes ]; then R_SKIP="$CONTAINER_SKIP"; R_RETAG="$CONTAINER_RETAG"; R_TAG="$CONTAINER_R_TAG" R_VER="$CONTAINER_R_VER"; R_REL="$CONTAINER_R_REL"; R_URL="$CONTAINER_R_URL"; R_SHA="$CONTAINER_R_SHA" download_image devgrail-container "devgrail-container:latest" "$CONTAINER_VERSION" "$CONTAINER_RELEASE" NEW_CONTAINER_VERSION="$LOADED_VERSION"; NEW_CONTAINER_RELEASE="$LOADED_RELEASE" else log "Skipping the workspace image (--workspace-image=no)" NEW_CONTAINER_VERSION="$CONTAINER_VERSION"; NEW_CONTAINER_RELEASE="$CONTAINER_RELEASE" fi step "Writing the deployment to $DEPLOY_DIR..." # --- deploy assets --------------------------------------------------------- log "Writing deploy config to $INSTALL_DIR..." mkdir -p "$DEPLOY_DIR/traefik" "$DEPLOY_DIR/dynamic" "$CONFIG_DIR" "$ASSET_DIR" mkdir -p "$BACKUP_DIR" "$DEPLOY_DIR/backups-request" "$DEPLOY_DIR/backups-status" chmod 700 "$BACKUP_DIR" "$DEPLOY_DIR/backups-request" "$DEPLOY_DIR/backups-status" # Traefik mounts $DEPLOY_DIR/dynamic read-only (it reads routes; devgrail-server # writes them), and the compose file then bind-mounts traefik/dynamic.yml at # dynamic/system.yml — nested inside that read-only mount. Docker creates a # missing mountpoint in the target directory, which it cannot do on a read-only # one: the container dies at start with EROFS and the whole site 404s. So create # the empty stub here. Harmless on an upgrade, where a previous rw mount already # left one behind, and harmless on a release whose compose still mounts rw. [ -e "$DEPLOY_DIR/dynamic/system.yml" ] || : > "$DEPLOY_DIR/dynamic/system.yml" NEW_ASSET_VERSIONS="" # fetch_asset <slug> — leave the release's copy of <slug> at $ASSET_PATH, # downloading and verifying it only if this run needs a version we do not # already hold. ASSET_PATH="" fetch_asset() { local slug="$1" url ver sha cached got have cached="$ASSET_DIR/$slug" ver="$(release_field "$slug" '.assets[]|select(.slug==$s)|.asset_version')" url="$(release_field "$slug" '.assets[]|select(.slug==$s)|.download_url')" sha="$(release_field "$slug" '.assets[]|select(.slug==$s)|.sha256')" if [ -z "$url" ]; then # No release manifest (or an asset it does not carry): fetch whatever is # published, as installers did before releases existed. Unversioned, so it # cannot be skipped — that is the behaviour releases exist to fix. url="$WEB/install/$slug"; ver=""; sha="" fi have="$(asset_version_of "$slug")" if [ -n "$ver" ] && [ "$ver" = "$have" ] && [ -f "$cached" ]; then log "$slug already at v$ver — skipping" else curl -fsSL "$url" -o "$cached.tmp" || die "cannot fetch '$slug' from $WEB" got="$(sha256_of "$cached.tmp")" if [ -n "$sha" ]; then [ "$got" = "$sha" ] || die "checksum mismatch for '$slug' (expected $sha, got $got)" fi # Verified before the templating below rewrites it: what the release was # signed for is the published file, not what this host makes of it. if [ -n "$ver" ]; then signed_digest_check "asset $slug $ver $got" "$slug" elif [ -n "$SIGNED_PAYLOAD_FILE" ]; then # The release verified, but does not name this asset — so it came from the # unversioned fallback path and nothing signed it. That is the same hole as # an unsigned release, opened one file at a time, and this file is either # the compose that defines the whole stack or the Traefik config that # decides who may reach it. A verified release does not get to contain an # unverified artifact. if [ "$ALLOW_UNSIGNED" != 1 ]; then die "'$slug' is not part of signed release $RELEASE_VERSION, so nothing vouches for the bytes just fetched from $WEB. The rest of this release verified, which makes this the odd one out rather than an old registry. Set DEVGRAIL_ALLOW_UNSIGNED=1 to accept it anyway." fi warn "'$slug' is not part of signed release $RELEASE_VERSION — fetched unverified." fi mv -f "$cached.tmp" "$cached" fi if [ -n "$ver" ]; then NEW_ASSET_VERSIONS="${NEW_ASSET_VERSIONS:+$NEW_ASSET_VERSIONS }$slug=$ver" fi ASSET_PATH="$cached" } # The rendered outputs are always rewritten from the cached copies, even when # nothing was downloaded: DEVGRAIL_DOMAIN or DEVGRAIL_TLS may have changed at # the prompt, and a config that silently kept a stale domain would be worse than # a redundant write. fetch_asset docker-compose.yml cp -f "$ASSET_PATH" "$DEPLOY_DIR/docker-compose.yml" # This script is always served at latest, but --version can install an older # release whose compose file predates DEVGRAIL_ADMIN_USERNAME. That file would # simply drop the variable, the server would create `admin`, and the banner below # would name a user that does not exist. Say so and fall back instead. if [ "$ADMIN_USER" != admin ] && ! grep -q DEVGRAIL_ADMIN_USERNAME "$DEPLOY_DIR/docker-compose.yml"; then warn "release ${NEW_SERVER_RELEASE:-this one} cannot take a custom admin username; creating \`admin\` instead." ADMIN_USER="admin" fi # The TLS or plain-HTTP router set per DEVGRAIL_TLS. Either is written to the # same dynamic.yml the compose file mounts by default, so no compose var is needed. fetch_asset "$TRAEFIK_ASSET" sed "s|__DEVGRAIL_DOMAIN__|$DEVGRAIL_DOMAIN|g" "$ASSET_PATH" > "$DEPLOY_DIR/traefik/dynamic.yml" # config.yaml holds the JWT secret (not passed via env). It used to be written # only when absent, so registry_url/tls_enabled went stale as soon as the # operator changed them at the prompt. The .env does override both at runtime # (internal/config/config.go), so that was latent rather than active — but two # files disagreeing about the domain is a trap to walk into during an incident. # The derived keys are rewritten every run; the secret never is. CONFIG_FILE="$CONFIG_DIR/config.yaml" if [ -f "$CONFIG_FILE" ]; then # Same SIGPIPE-under-pipefail hazard as the docker load above. Aborting here # would be particularly bad: the operator would see a bare exit 141 while the # installer was reading the secret that decrypts their entire database. JWT_SECRET="$(sed -n 's/^jwt_secret:[[:space:]]*//p' "$CONFIG_FILE")" JWT_SECRET="$(printf '%s' "${JWT_SECRET%%$'\n'*}" | tr -d "\"' ")" # Generating a fresh secret here would orphan every encrypted value in the # database — every API key secret, user secret and workspace SSH password. # Refusing loudly is the only safe answer. [ -n "$JWT_SECRET" ] || die "$CONFIG_FILE exists but has no jwt_secret. That secret keys every encrypted value in the database; generating a new one would make all of them undecryptable. Restore the file from a backup ($BACKUP_DIR holds config.yaml alongside the database) and re-run. Nothing has been changed." else JWT_SECRET="$(gen_secret)" fi # Anything the operator added by hand is carried over — this installer owns the # derived keys, not the whole file. CONFIG_EXTRA="" if [ -f "$CONFIG_FILE" ]; then CONFIG_EXTRA="$(grep -vE '^(jwt_secret|domain|base_domain|registry_url|traefik_dynamic_path|tls_enabled|backup_dir):' "$CONFIG_FILE" || true)" fi cat > "$CONFIG_FILE.tmp" <<EOF jwt_secret: "$JWT_SECRET" domain: "$DEVGRAIL_DOMAIN" base_domain: "$DEVGRAIL_BASE_DOMAIN" registry_url: "$WEB" traefik_dynamic_path: /dynamic/managed.yml tls_enabled: $TLS_ENABLED backup_dir: "$BACKUP_DIR" EOF if [ -n "$CONFIG_EXTRA" ]; then printf '%s\n' "$CONFIG_EXTRA" >> "$CONFIG_FILE.tmp"; fi # Same reasoning as .env, more so: a truncated config.yaml loses the JWT secret, # which is unrecoverable without a backup. chmod 600 "$CONFIG_FILE.tmp" mv -f "$CONFIG_FILE.tmp" "$CONFIG_FILE" # ADMIN_USER / ADMIN_PW were settled with the other answers above, prior values # included: they seed the admin account on first boot only (internal/db/seed.go: # SeedAdmin ignores them thereafter unless DEVGRAIL_FORCE_ADMIN_PASSWORD is set), # so dropping the password here would leave the operator with an account whose # password nobody knows. # Keys this installer owns. Everything else found in the prior .env is carried # over verbatim below — an operator who added DEVGRAIL_FORCE_ADMIN_PASSWORD or # any other compose variable must not lose it to an upgrade. MANAGED_ENV_KEYS=" ACME_EMAIL DEVGRAIL_ACME_CASERVER DEVGRAIL_ACME_STORAGE DEVGRAIL_ACME_BYPASS DEVGRAIL_DOMAIN DEVGRAIL_BASE_DOMAIN DEVGRAIL_ADMIN_USERNAME DEVGRAIL_ADMIN_PASSWORD DEVGRAIL_SERVER_IMAGE DEVGRAIL_TLS DEVGRAIL_WORKSPACE_IMAGE DEVGRAIL_WORKSPACE_APPARMOR_PROFILE DEVGRAIL_WORKSPACE_NESTED_APPARMOR_PROFILE DEVGRAIL_DASHBOARD_UPGRADES TAG HTTP_PORT HTTPS_PORT " cat > "$DEPLOY_DIR/.env.tmp" <<EOF ACME_EMAIL=${ACME_EMAIL:-} DEVGRAIL_ACME_CASERVER=$DEVGRAIL_ACME_CASERVER DEVGRAIL_ACME_STORAGE=$DEVGRAIL_ACME_STORAGE DEVGRAIL_ACME_BYPASS=$DEVGRAIL_ACME_BYPASS DEVGRAIL_DOMAIN=$DEVGRAIL_DOMAIN DEVGRAIL_BASE_DOMAIN=$DEVGRAIL_BASE_DOMAIN DEVGRAIL_ADMIN_USERNAME=$ADMIN_USER DEVGRAIL_ADMIN_PASSWORD=$ADMIN_PW DEVGRAIL_SERVER_IMAGE=$DEVGRAIL_SERVER_IMAGE DEVGRAIL_TLS=$DEVGRAIL_TLS DEVGRAIL_WORKSPACE_IMAGE=$DEVGRAIL_WORKSPACE_IMAGE DEVGRAIL_WORKSPACE_APPARMOR_PROFILE=$WORKSPACE_APPARMOR_PROFILE DEVGRAIL_WORKSPACE_NESTED_APPARMOR_PROFILE=$WORKSPACE_NESTED_APPARMOR_PROFILE DEVGRAIL_DASHBOARD_UPGRADES=$DEVGRAIL_DASHBOARD_UPGRADES TAG=latest HTTP_PORT=$HTTP_PORT HTTPS_PORT=$HTTPS_PORT EOF for _key in $PRIOR_KEYS; do case "$MANAGED_ENV_KEYS" in *" $_key "*) continue ;; esac # Copied verbatim from the prior file rather than re-serialised from the # parsed value: quoting we did not write is not ours to normalise, and this # runs before the temp file replaces it. Keys are [A-Za-z0-9_], so the pattern # carries no regex metacharacters. grep -m1 "^$_key=" "$DEPLOY_DIR/.env" >> "$DEPLOY_DIR/.env.tmp" || true done # Written via a temp file: this holds the admin password, and a truncated write # on a full disk would hand the operator a stack that cannot be logged into. chmod 600 "$DEPLOY_DIR/.env.tmp" mv -f "$DEPLOY_DIR/.env.tmp" "$DEPLOY_DIR/.env" # The integers are the registry's ordering key (what the skip-if-unchanged check # compares); the releases are the human-facing identity reported in the banner. # ASSET_VERSIONS is quoted because it holds a space-separated list, and this file # is sourced by the next run. cat > "$VERSIONS_FILE" <<EOF SERVER_VERSION=$NEW_SERVER_VERSION CONTAINER_VERSION=$NEW_CONTAINER_VERSION SERVER_RELEASE=$NEW_SERVER_RELEASE CONTAINER_RELEASE=$NEW_CONTAINER_RELEASE RELEASE_VERSION=$RELEASE_VERSION ASSET_VERSIONS="$NEW_ASSET_VERSIONS" EOF # --- pre-upgrade snapshot --------------------------------------------------- # Everything above rewrote config in place; the next line starts the new server, # which migrates the database on boot. Until this ran there was no point in the # upgrade where the pre-upgrade state was recoverable. # # The archive holds devgrail.db AND config.yaml. A database-only backup is # worthless: the JWT secret in config.yaml keys every at-rest cipher, so without # it every API key secret, user secret and workspace SSH password decrypts to # garbage. SNAPSHOT_RETAIN="${DEVGRAIL_SNAPSHOT_RETAIN:-5}" snapshot_before_upgrade() { local vol db_bytes free_bytes need ts stage archive summary schema vol="$(data_volume)" if [ -z "$vol" ]; then log "No existing DevGrail data volume found — nothing to snapshot." return 0 fi db_bytes="$(docker run --rm --userns=host -v "$vol":/var/lib/devgrail:ro \ --entrypoint /bin/sh devgrail-server:latest \ -c 'stat -c %s /var/lib/devgrail/devgrail.db 2>/dev/null || echo 0' 2>/dev/null)" || db_bytes=0 case "$db_bytes" in ''|*[!0-9]*) db_bytes=0 ;; esac if [ "$db_bytes" -eq 0 ]; then log "Data volume '$vol' holds no database yet — nothing to snapshot." return 0 fi exec 7>"$INSTALL_DIR/.backups.lock" flock -x 7 mkdir -p "$BACKUP_DIR" chmod 700 "$BACKUP_DIR" # Room is needed twice: once for the vacuumed copy, once for the tarball built # from it. Refusing here is not just hygiene — a full disk is precisely the # condition that causes the mid-migration failure this snapshot exists for. need=$((db_bytes * 2)) free_bytes="$(df -Pk "$BACKUP_DIR" | awk 'NR==2 {print $4 * 1024}')" || free_bytes=0 if [ "${free_bytes:-0}" -lt "$need" ]; then die "not enough free space for the pre-upgrade snapshot in $BACKUP_DIR: need $need bytes, have ${free_bytes:-0}. Free some space, or move $INSTALL_DIR to a larger filesystem, then re-run. Nothing has been changed." fi ts="$(date -u +%Y%m%dT%H%M%SZ)" stage="$BACKUP_DIR/.stage-$ts" rm -rf "$stage" mkdir -p "$stage" SNAPSHOT_STAGE="$stage" log "Taking a pre-upgrade snapshot (data volume: $vol)..." # Stop only the server; Traefik stays up so the host keeps answering. compose stop devgrail-server >/dev/null 2>&1 || true # VACUUM INTO, never `cp`: in WAL mode the newest committed data lives in the # -wal sidecar, so copying devgrail.db alone yields a corrupt backup. It runs # in a one-shot container off the image just loaded, so the host needs no # sqlite3; /etc/devgrail is mounted so it reads the deployment's real db_path. # --userns=host mirrors the compose services, which opt out of userns-remap. summary="$(docker run --rm --userns=host \ -v "$vol":/var/lib/devgrail \ -v "$stage":/backup \ -v "$CONFIG_DIR":/etc/devgrail:ro \ -e DEVGRAIL_CONFIG=/etc/devgrail/config.yaml \ --entrypoint /usr/local/bin/devgrail-server \ devgrail-server:latest -vacuum-into /backup/devgrail.db)" \ || die "pre-upgrade snapshot failed — refusing to start the new version. Your data is untouched and the previous version is still installed; re-run this installer once the cause is fixed." schema="$(printf '%s' "$summary" | jq -r '.schema_version // "unknown"')" cp -p "$CONFIG_DIR/config.yaml" "$stage/config.yaml" \ || die "pre-upgrade snapshot: could not copy $CONFIG_DIR/config.yaml. A database-only backup cannot decrypt anything it contains, so this is not a usable snapshot — refusing to continue." cat > "$stage/manifest.json" <<EOF { "created_at": "$(date -u +%Y-%m-%dT%H:%M:%SZ)", "domain": "$DEVGRAIL_DOMAIN", "schema_version": "$schema", "data_volume": "$vol", "prior_server_version": "${SERVER_VERSION:-unknown}", "prior_container_version": "${CONTAINER_VERSION:-unknown}", "prior_release_version": "${SERVER_RELEASE:-unknown}", "new_server_version": "${NEW_SERVER_VERSION:-unknown}", "new_container_version": "${NEW_CONTAINER_VERSION:-unknown}", "contents": ["devgrail.db", "config.yaml", "manifest.json"] } EOF archive="$BACKUP_DIR/pre-upgrade-$ts.tgz" tar czf "$stage/archive.tgz" -C "$stage" devgrail.db config.yaml manifest.json \ || die "pre-upgrade snapshot: could not write $archive" chmod 600 "$stage/archive.tgz" ln -- "$stage/archive.tgz" "$archive" \ || die "pre-upgrade snapshot: could not publish $archive without overwriting another archive" # The archive contains the JWT secret and every encrypted secret it protects. chmod 600 "$archive" rm -rf "$stage" SNAPSHOT_STAGE="" SNAPSHOT_ARCHIVE="$archive" log "Pre-upgrade snapshot written to $archive" # Keep the newest N. Unbounded snapshots eventually fill the disk, which is # the condition that causes the failure they protect against. # `|| true` because pipefail would otherwise abort the whole install on the # nothing-to-prune case, where ls has no matches. ls -1t "$BACKUP_DIR"/pre-upgrade-*.tgz 2>/dev/null | tail -n +$((SNAPSHOT_RETAIN + 1)) \ | while read -r old; do rm -f "$old"; done || true } if [ "$IS_UPGRADE" = true ]; then step "Snapshotting the database before the new server starts..." snapshot_before_upgrade fi exec 7>&- # --- devgrail maintenance command ------------------------------------------- # The pre-upgrade snapshot only exists when an upgrade happens to run. An # operator also needs a backup they can take now — before editing something, or # on a schedule — and a restore that does the steps in the one order that works. # Both live in the server binary (`-backup` / `-restore`); this wrapper is what # runs them as a one-shot container off the installed image, so the host needs # neither sqlite3 nor a copy of the Go toolchain. # # Written from here rather than shipped as an eighth release artifact: it is # entirely derived from paths this installer already owns, so it can never # disagree with the install it was written for. CLI_PATH=/usr/local/bin/devgrail write_devgrail_cli() { cat > "$CLI_PATH.tmp" <<CLIEOF #!/usr/bin/env bash # DevGrail maintenance CLI — written by install.sh. Do not edit; re-running the # installer overwrites it. set -euo pipefail INSTALL_DIR="$INSTALL_DIR" DEPLOY_DIR="$DEPLOY_DIR" CONFIG_DIR="$CONFIG_DIR" BACKUP_DIR="$BACKUP_DIR" IMAGE=devgrail-server:latest log() { printf '\033[1;34m==>\033[0m %s\n' "\$*"; } die() { printf '\033[1;31merror:\033[0m %s\n' "\$*" >&2; exit 1; } # Mirrors install.sh: an operator's docker-compose.override.yml applies to the # stop/start this does around a restore, or the restored server comes back # without it. compose() { if [ -f "\$DEPLOY_DIR/docker-compose.override.yml" ]; then docker compose --project-directory "\$DEPLOY_DIR" \\ -f "\$DEPLOY_DIR/docker-compose.yml" \\ -f "\$DEPLOY_DIR/docker-compose.override.yml" "\$@" else docker compose --project-directory "\$DEPLOY_DIR" -f "\$DEPLOY_DIR/docker-compose.yml" "\$@" fi } usage() { cat <<'USAGE' DevGrail maintenance commands. Run as root. devgrail backup [--out PATH] write a backup archive (database + config.yaml + manifest). Default: the backups directory, named backup-<timestamp>.tgz devgrail restore ARCHIVE stop the server, restore the archive, start it again, and wait for the readiness probe devgrail list list the archives on this host devgrail summary print the install summary and reference card again — dashboard URL, DNS records, upgrade, rollback and backup commands. Rewritten by every install.sh run. devgrail passwd [USERNAME] set a user's password (default: the admin this install created) and revoke their sessions. The way back in after a forgotten password. devgrail list-users list the accounts on this deployment with their role and whether each is disabled devgrail add-user NAME [--admin] create an account, prompting for its password. --admin makes it an admin. Accounts are normally created under Users in the dashboard; this is what works when the dashboard does not, and what a provisioning script can call. devgrail certificates production|staging switch the installation certificate environment. Staging is browser-untrusted; keep SSH open. devgrail help show this passwd and add-user prompt twice at a terminal. When stdin is not a terminal they read the password from its first line instead, for scripts: printf '%s\n' "\$PASSWORD" | devgrail add-user alice A backup archive holds BOTH the database and config.yaml. The jwt_secret in config.yaml keys every encrypted value in the database, so a database-only backup restores rows that decrypt to garbage. Never separate them. Workspace volumes are NOT included — they hold the customer's source code and are far larger than the database. See docs/RUNBOOK.md for the export recipe. USAGE } # data_volume prints the Docker volume backing /var/lib/devgrail, resolved from # the running container rather than guessed from the Compose project name. data_volume() { local vol vol="\$(docker inspect devgrail-server \\ --format '{{range .Mounts}}{{if eq .Destination "/var/lib/devgrail"}}{{.Name}}{{end}}{{end}}' 2>/dev/null)" || vol="" if [ -n "\$vol" ]; then printf '%s' "\$vol"; return 0; fi docker volume ls --format '{{.Name}}' 2>/dev/null | grep -E '(^|_)devgrail-data\$' | head -n1 || true } # read_new_password PROMPT sets NEW_PASSWORD for cmd_passwd and cmd_add_user. At # a terminal it prompts twice with echo off. Otherwise it takes the first line of # stdin, so a provisioning script can run # \`printf '%s\n' "\$pw" | devgrail add-user alice\` without a pseudo-terminal. # Any later lines are ignored. # # IFS= is what keeps leading and trailing spaces. A bare \`read -r\` strips them, # which stored a password other than the one typed; the server keeps whitespace, # so the shell has to as well. read_new_password() { local prompt="\$1" pw2="" NEW_PASSWORD="" if [ -t 0 ]; then printf '%s: ' "\$prompt" > /dev/tty stty -echo 2>/dev/null || true IFS= read -r NEW_PASSWORD < /dev/tty || true stty echo 2>/dev/null || true printf '\n' > /dev/tty printf 'Repeat password: ' > /dev/tty stty -echo 2>/dev/null || true IFS= read -r pw2 < /dev/tty || true stty echo 2>/dev/null || true printf '\n' > /dev/tty [ -n "\$NEW_PASSWORD" ] || die "no password entered" [ "\$NEW_PASSWORD" = "\$pw2" ] || die "the passwords do not match" else # No confirmation here: a script has nobody to mistype, and asking it for # the same line twice would only make it harder to call. IFS= read -r NEW_PASSWORD || true [ -n "\$NEW_PASSWORD" ] || die "no password on stdin" fi } # cmd_backup writes one archive via a one-shot container off the installed # server image, with the data volume, /etc/devgrail and the output directory # mounted. --userns=host mirrors the compose services, which opt out of # userns-remap. cmd_backup() { if [ "\$#" -eq 0 ] && [ -x "\$INSTALL_DIR/backups-host/devgrail-backups-helper" ]; then "\$INSTALL_DIR/backups-host/devgrail-backups-helper" --cli return fi local out="" ts vol dir while [ \$# -gt 0 ]; do case "\$1" in --out=*) out="\${1#*=}" ;; --out) shift; out="\${1:-}" ;; *) usage >&2; die "unknown option: \$1" ;; esac shift done ts="\$(date -u +%Y%m%dT%H%M%SZ)" [ -n "\$out" ] || out="\$BACKUP_DIR/backup-\$ts.tgz" dir="\$(cd "\$(dirname "\$out")" 2>/dev/null && pwd)" || die "no such directory: \$(dirname "\$out")" mkdir -p "\$BACKUP_DIR"; chmod 700 "\$BACKUP_DIR" vol="\$(data_volume)" [ -n "\$vol" ] || die "cannot find the DevGrail data volume — is DevGrail installed here?" # Taken against a running server on purpose: VACUUM INTO reads through a single # read transaction, so the copy is consistent without downtime. The volume is # mounted read-write even though nothing here modifies the data: opening a # WAL-mode SQLite database requires writing its -shm sidecar, so a read-only # mount fails outright with "attempt to write a readonly database". docker run --rm --userns=host \\ -v "\$vol":/var/lib/devgrail \\ -v "\$CONFIG_DIR":/etc/devgrail:ro \\ -v "\$dir":/backup \\ -e DEVGRAIL_CONFIG=/etc/devgrail/config.yaml \\ --entrypoint /usr/local/bin/devgrail-server \\ "\$IMAGE" -backup "/backup/\$(basename "\$out")" \\ || die "backup failed — nothing was written" chmod 600 "\$out" 2>/dev/null || true log "Backup written to \$out" } cmd_restore() { local archive="\${1:-}" vol dir [ -n "\$archive" ] || { usage >&2; die "restore needs an archive path"; } [ -f "\$archive" ] || die "no such archive: \$archive" dir="\$(cd "\$(dirname "\$archive")" && pwd)" vol="\$(data_volume)" [ -n "\$vol" ] || die "cannot find the DevGrail data volume — refusing to guess which volume to overwrite." # The documented order, enforced: stop -> config.yaml -> database -> start -> # verify. Restoring under a running server would swap the database out from # under open connections. log "Stopping devgrail-server..." compose stop devgrail-server >/dev/null 2>&1 || true docker run --rm --userns=host \\ -v "\$vol":/var/lib/devgrail \\ -v "\$CONFIG_DIR":/etc/devgrail \\ -v "\$dir":/restore:ro \\ -e DEVGRAIL_CONFIG=/etc/devgrail/config.yaml \\ --entrypoint /usr/local/bin/devgrail-server \\ "\$IMAGE" -restore "/restore/\$(basename "\$archive")" \\ || { compose up -d >/dev/null 2>&1 || true; die "restore failed — the previous state is unchanged and the server has been restarted"; } log "Starting devgrail-server..." compose up -d local waited=0 while [ "\$waited" -lt 180 ]; do if compose exec -T devgrail-server /usr/local/bin/devgrail-server -healthcheck >/dev/null 2>&1; then log "DevGrail is ready. Restore complete." return 0 fi sleep 3 waited=\$((waited + 3)) done compose logs --tail=50 devgrail-server >&2 || true die "the restored server did not become ready within 180s (logs above)" } # cmd_passwd resets a password without a login — the way back in when nobody # knows the admin's. It runs the same one-shot container cmd_backup does; the new # password goes in on stdin, never as an argument or an environment variable, # because both are readable via \`ps\` and \`docker inspect\` while the process runs. # # No stop/start: SQLite is in WAL mode, so the write lands under the running # server, and the sessions it revokes are re-checked on the next request. cmd_passwd() { local user="\${1:-}" vol if [ -z "\$user" ]; then user="\$(sed -n 's/^DEVGRAIL_ADMIN_USERNAME=//p' "\$DEPLOY_DIR/.env" 2>/dev/null | tail -n1)" user="\${user:-admin}" fi read_new_password "New password for \$user" vol="\$(data_volume)" [ -n "\$vol" ] || die "cannot find the DevGrail data volume — is DevGrail installed here?" printf '%s' "\$NEW_PASSWORD" | docker run --rm -i --userns=host \\ -v "\$vol":/var/lib/devgrail \\ -v "\$CONFIG_DIR":/etc/devgrail:ro \\ -e DEVGRAIL_CONFIG=/etc/devgrail/config.yaml \\ --entrypoint /usr/local/bin/devgrail-server \\ "\$IMAGE" -set-password "\$user" \\ || die "password not changed" } # cmd_list_users and cmd_add_user are the accounts half of the same escape # hatch. Accounts are normally managed under Users in the dashboard; these exist # for when it cannot be reached, and for unattended provisioning. Reading and # creating only: disabling and purging an account destroy another person's work, # and those belong where they are audited and state their consequences first. cmd_list_users() { local vol [ \$# -eq 0 ] || { usage >&2; die "list-users takes no arguments"; } vol="\$(data_volume)" [ -n "\$vol" ] || die "cannot find the DevGrail data volume — is DevGrail installed here?" # Mounted read-write for the reason cmd_backup gives: opening a WAL-mode # SQLite database writes its -shm sidecar even when nothing is being changed. docker run --rm --userns=host \ -v "\$vol":/var/lib/devgrail \ -v "\$CONFIG_DIR":/etc/devgrail:ro \ -e DEVGRAIL_CONFIG=/etc/devgrail/config.yaml \ --entrypoint /usr/local/bin/devgrail-server \ "\$IMAGE" -list-users } # The password goes in on stdin, never as an argument or an environment # variable, for the reason cmd_passwd states. cmd_add_user() { local user="" admin=0 vol while [ \$# -gt 0 ]; do case "\$1" in --admin) admin=1 ;; -*) usage >&2; die "unknown option: \$1" ;; *) [ -z "\$user" ] || { usage >&2; die "add-user takes one username"; } user="\$1" ;; esac shift done [ -n "\$user" ] || { usage >&2; die "add-user needs a username"; } read_new_password "Password for \$user" vol="\$(data_volume)" [ -n "\$vol" ] || die "cannot find the DevGrail data volume — is DevGrail installed here?" # --admin is folded into the positional list rather than interpolated as a # possibly-empty word, which the server would otherwise read as a username. set -- -add-user "\$user" if [ "\$admin" -eq 1 ]; then set -- "\$@" -admin; fi printf '%s' "\$NEW_PASSWORD" | docker run --rm -i --userns=host \ -v "\$vol":/var/lib/devgrail \ -v "\$CONFIG_DIR":/etc/devgrail:ro \ -e DEVGRAIL_CONFIG=/etc/devgrail/config.yaml \ --entrypoint /usr/local/bin/devgrail-server \ "\$IMAGE" "\$@" \ || die "account not created" } case "\${1:-help}" in certificates) shift [ "\$(id -u)" -eq 0 ] || die "Run as root: sudo devgrail certificates production" [ \$# -eq 1 ] || die "Use: devgrail certificates production|staging" case "\$1" in production|staging) ;; *) die "Choose production or staging" ;; esac exec "\$INSTALL_DIR/certificates/devgrail-certificates-helper" --apply "\$1" ;; backup) shift; cmd_backup "\$@" ;; restore) shift; cmd_restore "\$@" ;; passwd) shift; cmd_passwd "\$@" ;; list-users) shift; cmd_list_users "\$@" ;; add-user) shift; cmd_add_user "\$@" ;; list) ls -lh "\$BACKUP_DIR" 2>/dev/null || echo "no archives in \$BACKUP_DIR" ;; summary) # The answer to "the install scrolled away and I lost the URL". install.sh # rewrites this file on every run, so it always describes the install that # is actually here. if [ -f "\$INSTALL_DIR/README-install.txt" ]; then cat "\$INSTALL_DIR/README-install.txt" else die "no \$INSTALL_DIR/README-install.txt on this host — re-run install.sh to write one." fi ;; help|-h|--help) usage ;; *) usage >&2; die "unknown command: \$1" ;; esac CLIEOF chmod 755 "$CLI_PATH.tmp" mv -f "$CLI_PATH.tmp" "$CLI_PATH" } step "Installing the devgrail command..." if write_devgrail_cli; then log "Maintenance CLI installed: $CLI_PATH (backup / restore / passwd / users / certificates)" else warn "could not write $CLI_PATH — backup and restore will need the docker run commands from docs/RUNBOOK.md" fi # --- backup management host helper ------------------------------------------ # Fixed, self-contained script: requests contain data, never executable shell. backups_helper_body() { cat <<'BACKUPSHELPER' set -euo pipefail umask 077 WORK="$HELPER_DIR/work" POLICY="$HELPER_DIR/policy.json" mkdir -p "$WORK" "$STATUS_DIR/operations" "$ARCHIVES" chmod 700 "$HELPER_DIR" "$WORK" "$STATUS_DIR" "$ARCHIVES" now() { date -u +%Y-%m-%dT%H:%M:%SZ; } volume() { local id id="$(docker compose --project-directory "$DEPLOY_DIR" -f "$DEPLOY_DIR/docker-compose.yml" ps -a -q devgrail-server 2>/dev/null || true)" if [ -n "$id" ]; then docker inspect -f '{{range .Mounts}}{{if eq .Destination "/var/lib/devgrail"}}{{.Name}}{{end}}{{end}}' "$id" else docker volume ls --filter label=com.docker.compose.project=deploy --filter label=com.docker.compose.volume=devgrail-data -q | head -n1 fi } run_mode() { local mode="$1"; shift local vol vol="$(volume)"; [ -n "$vol" ] || { echo 'DevGrail data volume unavailable' >&2; return 1; } docker run --rm -i --userns=host \ -v "$vol":/var/lib/devgrail -v "$CONFIG_DIR":/etc/devgrail:ro \ -v "$ARCHIVES":/backups/archives -v "$STATUS_DIR":/backups/status \ -e DEVGRAIL_CONFIG=/etc/devgrail/config.yaml \ -e DEVGRAIL_BACKUP_ARCHIVE_DIR=/backups/archives -e DEVGRAIL_BACKUP_STATUS_DIR=/backups/status \ --entrypoint /usr/local/bin/devgrail-server devgrail-server:latest -backup-manage="$mode" "$@" } atomic_jq() { local target="$1"; shift; jq "$@" > "$target.tmp" && mv -f "$target.tmp" "$target"; } operation() { local state="$1" message="${2:-}" atomic_jq "$STATUS_DIR/operations/$OP_ID.json" --arg state "$state" --arg message "$message" --arg finished "$(now)" \ '.operation | .state=$state | .message=$message | .finished_at=(if $state=="running" then "" else $finished end)' "$REQ" cp "$STATUS_DIR/operations/$OP_ID.json" "$STATUS_DIR/latest.json.tmp" mv -f "$STATUS_DIR/latest.json.tmp" "$STATUS_DIR/latest.json" jq -c '{actor:.actor,action:.action,operation:.id,state:.state}' "$STATUS_DIR/latest.json" >&2 if [ "$(jq -r .operation.action "$REQ")" = create ]; then cp "$STATUS_DIR/latest.json" "$STATUS_DIR/last-backup.json.tmp" mv -f "$STATUS_DIR/last-backup.json.tmp" "$STATUS_DIR/last-backup.json" fi } import_policy() { if [ ! -f "$POLICY" ]; then local enabled=false systemctl is-enabled devgrail-backup.timer >/dev/null 2>&1 && enabled=true printf '{"enabled":%s,"frequency":"weekly","time":"00:00","weekday":1,"month_day":1,"keep":0}\n' "$enabled" > "$POLICY.tmp" mv -f "$POLICY.tmp" "$POLICY" fi } report() ( exec 6>"$HELPER_DIR/.report.lock" flock -x 6 import_policy local enabled=false available=false reason="Backup request watcher is inactive. Check devgrail-backups.path on the host." calendar delay next timezone latest custom=false systemctl is-enabled devgrail-backup.timer >/dev/null 2>&1 && enabled=true if systemctl is-active --quiet devgrail-backups.path; then available=true; reason=""; fi calendar="$(systemctl cat devgrail-backup.timer 2>/dev/null | awk -F= '/^OnCalendar=/{if($2==""){result=""}else{result=result (result==""?"":";") $2}} END{print result}' || true)" delay="$(systemctl show devgrail-backup.timer -p RandomizedDelayUSec --value 2>/dev/null || true)" next="$(systemctl show devgrail-backup.timer -p NextElapseUSecRealtime --value 2>/dev/null || true)" timezone="$(timedatectl show -p Timezone --value 2>/dev/null || date +%Z)" latest=null; [ ! -f "$STATUS_DIR/latest.json" ] || latest="$(cat "$STATUS_DIR/latest.json")" local expected expected="$(jq -r 'if .frequency=="daily" then "*-*-* "+.time+":00" elif .frequency=="weekly" then (["Sun","Mon","Tue","Wed","Thu","Fri","Sat"][.weekday])+" *-*-* "+.time+":00" else "*-*-"+(if .month_day<10 then "0" else "" end)+(.month_day|tostring)+" "+.time+":00" end' "$POLICY")" if [ -n "$calendar" ] && [ "$calendar" != "$expected" ] && { [ "$calendar" != weekly ] || [ "$expected" != "Mon *-*-* 00:00:00" ]; }; then custom=true; fi jq -n --slurpfile policy "$POLICY" --argjson enabled "$enabled" --argjson available "$available" --arg reason "$reason" --arg calendar "$calendar" --arg delay "$delay" \ --arg next "$next" --arg timezone "$timezone" --arg observed "$(now)" --argjson custom "$custom" --argjson latest "$latest" \ --argjson snapshots "$PRE_UPGRADE_KEEP" \ '{protocol:1,settings:{available:$available,schedule_available:$available,unavailable_reason:$reason,policy:($policy[0]|.enabled=$enabled),calendar:$calendar,random_delay:$delay,next_run:$next,timezone:$timezone,observed_at:$observed,custom_schedule:$custom,pre_upgrade_keep:$snapshots,latest_operation:$latest}}' > "$STATUS_DIR/report.json.tmp" mv -f "$STATUS_DIR/report.json.tmp" "$STATUS_DIR/report.json" ) # Reporting never claims a request and does not need the installer lock. if [ "${1:-}" = --report ]; then report # Mark abandoned work only after the service is no longer active. if (exec 7>"$INSTALL_DIR/.backups.lock"; flock -xn 7) && [ -f "$STATUS_DIR/latest.json" ] && jq -e '.state=="running"' "$STATUS_DIR/latest.json" >/dev/null; then jq '.state="interrupted" | .message="The host backup worker stopped. Check its journal and retry."' "$STATUS_DIR/latest.json" > "$STATUS_DIR/latest.json.tmp" mv -f "$STATUS_DIR/latest.json.tmp" "$STATUS_DIR/latest.json" id="$(jq -r .id "$STATUS_DIR/latest.json")" [[ "$id" =~ ^[a-f0-9]{32}$ ]] && cp "$STATUS_DIR/latest.json" "$STATUS_DIR/operations/$id.json" report fi exit 0 fi # Install/upgrade takes this lock exclusively. Shared backup operations never # overwrite its owner record; lock order is always install, then backup. exec 9>"$INSTALL_DIR/.install.lock" flock -s -w 120 9 || { echo 'Installer busy; backup deferred' >&2; exit 1; } exec 8>"$INSTALL_DIR/.backups.lock" flock -x -w 120 8 || { echo 'Another backup operation is running' >&2; exit 1; } REQ="$WORK/request" rm -f "$REQ" case "${1:-}" in --scheduled|--cli) id="$(cat /proc/sys/kernel/random/uuid | tr -d '-')" jq -n --arg id "$id" --arg created "$(now)" '{protocol:1,operation:{id:$id,action:"create",actor:"system",state:"queued",created_at:$created,finished_at:"",message:"",archive_id:""}}' > "$REQ" origin=scheduled; [ "$1" != --cli ] || origin=cli ;; --index) run_mode index; exit $? ;; *) # Rename a symlink/FIFO as itself, then reject it before any read. mv -fT "$REQUEST_DIR/request" "$REQ" 2>/dev/null || exit 0 origin=manual ;; esac if [ -L "$REQ" ] || [ ! -f "$REQ" ] || [ "$(stat -c %s "$REQ")" -gt 8192 ]; then rm -f "$REQ"; echo 'Rejected unsafe backup request' >&2; exit 1 fi # Read at most one bounded snapshot into private storage; an open descriptor # retained by a hostile request writer cannot change the parsed request later. head -c 8193 "$REQ" > "$WORK/captured.json" [ "$(stat -c %s "$WORK/captured.json")" -le 8192 ] || exit 1 run_mode validate < "$WORK/captured.json" > "$WORK/validated.json" || { rm -f "$REQ"; exit 1; } mv -f "$WORK/validated.json" "$REQ" OP_ID="$(jq -r .operation.id "$REQ")" action="$(jq -r .operation.action "$REQ")" # Always record failure if a worker exits or is killed after claiming work. completed=false finish() { if [ "$completed" != true ]; then operation failed 'Backup operation interrupted or failed; check the host backup journal.' || true; fi rm -f "$REQ" report || true } trap finish EXIT trap 'exit 143' TERM trap 'exit 130' INT operation running report import_policy if [ "$action" = settings ]; then calendar="$(run_mode calendar < "$REQ")" for unit in devgrail-backup.timer devgrail-backup.service; do rm -f "$WORK/$unit.before"; [ ! -f "$UNIT_DIR/$unit" ] || cp -p "$UNIT_DIR/$unit" "$WORK/$unit.before" done dropin="$UNIT_DIR/devgrail-backup.timer.d/zzzz-devgrail-backups.conf" mkdir -p "$(dirname "$dropin")" rm -f "$WORK/timer-dropin.before" [ ! -f "$dropin" ] || cp -p "$dropin" "$WORK/timer-dropin.before" cp -p "$POLICY" "$WORK/policy.before" was_enabled=false; systemctl is-enabled devgrail-backup.timer >/dev/null 2>&1 && was_enabled=true cat > $UNIT_DIR/devgrail-backup.service <<UNIT [Unit] Description=DevGrail scheduled backup After=docker.service Requires=docker.service [Service] Type=oneshot ExecStart=$HELPER_DIR/devgrail-backups-helper --scheduled TimeoutStartSec=2h UNIT cat > $UNIT_DIR/devgrail-backup.timer <<UNIT [Unit] Description=Scheduled DevGrail backup [Timer] OnCalendar=$calendar Persistent=true [Install] WantedBy=timers.target UNIT # Reset inherited OnCalendar entries without deleting operator-owned drop-ins. cat > "$dropin" <<UNIT [Timer] OnCalendar= OnCalendar=$calendar RandomizedDelaySec=0 Persistent=true UNIT apply_timer() { systemctl daemon-reload && systemctl disable --now devgrail-backup.timer && { if jq -e '.policy.enabled' "$REQ" >/dev/null; then systemctl enable --now devgrail-backup.timer; fi; } } if apply_timer; then jq .policy "$REQ" > "$POLICY.tmp"; mv -f "$POLICY.tmp" "$POLICY" operation succeeded; completed=true else for unit in devgrail-backup.timer devgrail-backup.service; do if [ -f "$WORK/$unit.before" ]; then cp -p "$WORK/$unit.before" "$UNIT_DIR/$unit"; else rm -f "$UNIT_DIR/$unit"; fi done if [ -f "$WORK/timer-dropin.before" ]; then cp -p "$WORK/timer-dropin.before" "$dropin"; else rm -f "$dropin"; fi cp -p "$WORK/policy.before" "$POLICY" systemctl daemon-reload || true if [ "$was_enabled" = true ]; then systemctl enable --now devgrail-backup.timer || true; else systemctl disable --now devgrail-backup.timer || true; fi operation failed 'Schedule update failed; previous settings restored.'; completed=true fi else keep="$(jq -r .keep "$POLICY")" if run_mode execute -backup-keep="$keep" -backup-origin="$origin" < "$REQ"; then completed=true else if jq -e --arg id "$OP_ID" '.id==$id and (.state=="failed" or .state=="succeeded")' "$STATUS_DIR/latest.json" >/dev/null; then completed=true; fi exit 1 fi fi # Operation history is bounded; archives are pruned only by the Go retention code. find "$STATUS_DIR/operations" -maxdepth 1 -type f -name '*.json' -printf '%T@ %p\n' | sort -rn | tail -n +51 | cut -d' ' -f2- | while IFS= read -r file; do rm -f -- "$file"; done BACKUPSHELPER } install_backups_helper() { local helper_dir="$INSTALL_DIR/backups-host" mkdir -p "$helper_dir" "$DEPLOY_DIR/backups-request" "$DEPLOY_DIR/backups-status/operations" "$BACKUP_DIR" chmod 700 "$helper_dir" "$DEPLOY_DIR/backups-request" "$DEPLOY_DIR/backups-status" "$BACKUP_DIR" { printf '#!/usr/bin/env bash\nUNIT_DIR=/etc/systemd/system\n' printf 'INSTALL_DIR=%q\nDEPLOY_DIR=%q\nCONFIG_DIR=%q\nARCHIVES=%q\nHELPER_DIR=%q\nREQUEST_DIR=%q\nSTATUS_DIR=%q\nPRE_UPGRADE_KEEP=%q\n' \ "$INSTALL_DIR" "$DEPLOY_DIR" "$CONFIG_DIR" "$BACKUP_DIR" "$helper_dir" "$DEPLOY_DIR/backups-request" "$DEPLOY_DIR/backups-status" "${SNAPSHOT_RETAIN:-5}" backups_helper_body } > "$helper_dir/devgrail-backups-helper.tmp" chmod 700 "$helper_dir/devgrail-backups-helper.tmp" mv -f "$helper_dir/devgrail-backups-helper.tmp" "$helper_dir/devgrail-backups-helper" command -v systemctl >/dev/null 2>&1 && [ -d /run/systemd/system ] || return 0 cat > /etc/systemd/system/devgrail-backups.service <<EOF [Unit] Description=Manage DevGrail backup requests After=docker.service Requires=docker.service [Service] Type=oneshot ExecStart=$helper_dir/devgrail-backups-helper TimeoutStartSec=2h EOF cat > /etc/systemd/system/devgrail-backups.path <<EOF [Path] PathExists=$DEPLOY_DIR/backups-request/request [Install] WantedBy=paths.target EOF cat > /etc/systemd/system/devgrail-backups-report.service <<EOF [Service] Type=oneshot ExecStart=$helper_dir/devgrail-backups-helper --report EOF cat > /etc/systemd/system/devgrail-backups-report.timer <<EOF [Timer] OnBootSec=30s OnUnitActiveSec=30s [Install] WantedBy=timers.target EOF # One-time legacy indexing is off the HTTP path and keeps the page responsive. cat > /etc/systemd/system/devgrail-backups-index.service <<EOF [Unit] After=docker.service Requires=docker.service [Service] Type=oneshot ExecStart=$helper_dir/devgrail-backups-helper --index TimeoutStartSec=2h EOF systemctl daemon-reload 9>&- systemctl enable --now devgrail-backups.path devgrail-backups-report.timer 9>&- "$helper_dir/devgrail-backups-helper" --report 9>&- } install_backups_helper || warn "Could not enable backup management; check the host systemd journal." # --- scheduled backups ------------------------------------------------------- # Offered, never imposed: it writes to the customer's disk on a timer, and an # operator who already has their own backup story should not get a second one # they did not ask for. DEVGRAIL_SCHEDULED_BACKUPS=yes|no answers it unattended; # an existing timer is left exactly as configured. install_backup_timer() { command -v systemctl >/dev/null 2>&1 || return 1 cat > /etc/systemd/system/devgrail-backup.service <<EOF [Unit] Description=DevGrail weekly backup After=docker.service Requires=docker.service [Service] Type=oneshot ExecStart=$INSTALL_DIR/backups-host/devgrail-backups-helper --scheduled EOF # Randomised delay so a fleet of VPSes does not all wake at once; Persistent # so a machine that was off at the scheduled time still runs it on next boot. cat > /etc/systemd/system/devgrail-backup.timer <<EOF [Unit] Description=Weekly DevGrail backup [Timer] OnCalendar=weekly RandomizedDelaySec=1h Persistent=true [Install] WantedBy=timers.target EOF # 9>&- for the same reason as the Docker install above: whatever these start # must not inherit the install lock and outlive this run holding it. systemctl daemon-reload >/dev/null 2>&1 9>&- || true systemctl enable --now devgrail-backup.timer >/dev/null 2>&1 9>&- || return 1 return 0 } # --- weekly backup (apply) --------------------------------------------------- # Decided up in the question block; only the doing is left. The CLI has to exist # by now — the timer's ExecStart is that file. if [ "$BACKUP_TIMER_DECISION" = install ] && [ -f "$CLI_PATH" ]; then step "Scheduling the weekly backup..." if [ -f "$INSTALL_DIR/backups-host/policy.json" ] && [ -f /etc/systemd/system/devgrail-backup.timer ]; then systemctl enable --now devgrail-backup.timer 9>&- || warn "Could not enable saved backup timer" BACKUP_TIMER_NOTE="Backups: saved schedule enabled. Manage in Administration -> Backups." elif install_backup_timer; then BACKUP_TIMER_NOTE="Weekly backups: enabled (systemctl status devgrail-backup.timer)" else warn "could not enable the weekly backup timer — run '$CLI_PATH backup' yourself, or from your own scheduler" BACKUP_TIMER_NOTE="Weekly backups: could not be scheduled — run '$CLI_PATH backup' from your own scheduler." fi elif [ "$BACKUP_TIMER_DECISION" = disable ]; then systemctl disable --now devgrail-backup.timer 9>&- || true BACKUP_TIMER_NOTE="Scheduled backups: disabled. Manage in Administration -> Backups." elif [ "$BACKUP_TIMER_DECISION" = skip ] && command -v systemctl >/dev/null 2>&1; then BACKUP_TIMER_NOTE="Weekly backups: not scheduled. Enable later with: DEVGRAIL_SCHEDULED_BACKUPS=yes re-run, or run '$CLI_PATH backup' from your own scheduler." fi # --- certificate environment helper ----------------------------------------- certificates_helper_body() { cat <<'CERTHELPER' set -euo pipefail umask 077 DEPLOY_DIR="$INSTALL_DIR/deploy" WORK="$HELPER_DIR/work" mkdir -p "$WORK" chmod 700 "$HELPER_DIR" "$WORK" PRODUCTION=https://acme-v02.api.letsencrypt.org/directory STAGING=https://acme-staging-v02.api.letsencrypt.org/directory ENV_FILE="$DEPLOY_DIR/.env" LOG="$STATUS_DIR/certificates.log" TARGET=""; STARTED=""; STATE=""; CHANGED=false; INSTALL_LOCKED=false; CLI=false # Compose must read the saved CA rather than a CLI caller's exported value. unset DEVGRAIL_ACME_CASERVER DEVGRAIL_ACME_STORAGE now() { date -u +%Y-%m-%dT%H:%M:%SZ; } lock_wait() { local deadline=$((SECONDS + $2)) until flock -n "$1"; do [ "$SECONDS" -lt "$deadline" ] || return 1 sleep 1 done } compose() { local -a files=(-f "$DEPLOY_DIR/docker-compose.yml") [ ! -f "$DEPLOY_DIR/docker-compose.override.yml" ] || files+=(-f "$DEPLOY_DIR/docker-compose.override.yml") docker compose --project-directory "$DEPLOY_DIR" --env-file "$ENV_FILE" "${files[@]}" "$@" } # Parse only named values, never source .env (which also contains passwords). value() { local v v="$(sed -n "s/^$1=//p" "$ENV_FILE" | tail -n1)" v="${v%$'\r'}" case "$v" in \"*\") v="${v:1:${#v}-2}" ;; \'*\') v="${v:1:${#v}-2}" ;; esac printf '%s' "$v" } environment() { case "$1" in "$PRODUCTION") echo production ;; "$STAGING") echo staging ;; '') echo unknown ;; *) echo custom ;; esac } traefik_id() { compose ps -q traefik; } inspect_config() { local id id="$(traefik_id)"; [ -n "$id" ] || return 1 docker inspect "$id" | jq -e '.[0] | {running: .State.Running, command: .Config.Cmd, networks: .NetworkSettings.Networks}' } flag() { jq -r --arg prefix "--certificatesresolvers.letsencrypt.acme.$1=" '.command[] | select(startswith($prefix)) | ltrimstr($prefix)' | tail -n1; } report() ( exec 7>"$WORK/.report.lock"; lock_wait 7 5 || return 0 local saved active ca storage info allowed=false reason tls saved="$(value DEVGRAIL_ACME_CASERVER)"; saved="${saved:-$PRODUCTION}" active=unknown; ca=""; storage="" if info="$(inspect_config 2>/dev/null)" && printf '%s' "$info" | jq -e '.running' >/dev/null; then ca="$(printf '%s' "$info" | flag caserver)" storage="$(printf '%s' "$info" | flag storage)" active="$(environment "$ca")" fi reason="Traefik is not running with a supported ACME resolver. Check the host configuration." tls="$(value DEVGRAIL_TLS)" case "${tls,,}" in off|false|0|no) reason="TLS is off; certificate switching is unavailable." ;; *) if [ -n "$ca" ] && [ -n "$storage" ]; then allowed=true; reason=""; fi ;; esac jq -n --arg saved "$(environment "$saved")" --arg active "$active" --arg ca "$ca" \ --arg at "$(now)" --argjson allowed "$allowed" --arg reason "$reason" \ '{saved_environment:$saved, active_environment:$active, ca_server:$ca, observed_at:$at, change_allowed:$allowed, reason:$reason}' \ > "$STATUS_DIR/.configuration.json.tmp" chmod 644 "$STATUS_DIR/.configuration.json.tmp" mv -f "$STATUS_DIR/.configuration.json.tmp" "$STATUS_DIR/configuration.json" ) write_status() { local finished="" STATE="$1"; [ "$STATE" = running ] || finished="$(now)" jq -n --arg state "$STATE" --arg target "$TARGET" --arg started "$STARTED" \ --arg finished "$finished" --arg message "$2" \ '{protocol:1, state:$state, target_environment:$target, started_at:$started, finished_at:$finished, message:$message}' \ > "$STATUS_DIR/.status.json.tmp" chmod 644 "$STATUS_DIR/.status.json.tmp" mv -f "$STATUS_DIR/.status.json.tmp" "$STATUS_DIR/status.json" if $CLI; then printf '%s\n' "$2"; fi } log() { printf '[%s] %s\n' "$(now)" "$*" >> "$LOG"; } reject() { write_status rejected "$1"; exit 0; } # Every dynamically attached network is lost on recreate. Keep its aliases in # root-only work, so both apply and rollback restore in-container MCP routing. restore_networks() { local id net alias workspace_ids ws local -a args id="$(traefik_id)"; [ -n "$id" ] || return 1 # Include workspaces created during recreation and recover when an earlier # failure removed Traefik's original network endpoints entirely. workspace_ids="$(docker ps -aq --filter label=devgrail.managed=true)" || return 1 for ws in $workspace_ids; do net="$(docker inspect "$ws" | jq -r '.[0].Config.Labels["traefik.docker.network"] // empty')" || continue case "$net" in devgrail-ws-*) ;; *) continue ;; esac docker network inspect "$net" | jq -e '.[0].Labels["devgrail.managed"] == "true"' >/dev/null || continue jq --arg n "$net" 'if has($n) then . else . + {($n):{Aliases:["traefik"]}} end' "$WORK/networks.json" > "$WORK/.networks.tmp" || return 1 mv -f "$WORK/.networks.tmp" "$WORK/networks.json" done while IFS= read -r net; do docker network inspect "$net" >/dev/null 2>&1 || continue if docker inspect "$id" | jq -e --arg n "$net" '.[0].NetworkSettings.Networks | has($n)' >/dev/null; then continue; fi args=() while IFS= read -r alias; do args+=(--alias "$alias"); done < <(jq -r --arg n "$net" '.[$n].Aliases[]? // empty' "$WORK/networks.json") docker network connect "${args[@]}" "$net" "$id" || return 1 done < <(jq -r 'keys[]' "$WORK/networks.json") } recreate() { compose up -d --no-deps --force-recreate traefik >> "$LOG" 2>&1 || return 1 restore_networks >> "$LOG" 2>&1 } rollback() { # Copy to the same directory and rename, preserving the original file mode. cp -p "$WORK/env.before" "$DEPLOY_DIR/.env.certificates.tmp" && mv -f "$DEPLOY_DIR/.env.certificates.tmp" "$ENV_FILE" && recreate } on_exit() { local rc=$? trap - EXIT TERM INT if $CHANGED; then if rollback; then CHANGED=false rm -f "$WORK/env.before" write_status failed "The change failed. The previous configuration and workspace network attachments were restored. Check the log before retrying." || true else write_status failed "The change failed and recovery did not finish. Through SSH run sudo devgrail certificates production, then check sudo journalctl -u devgrail-certificates.service." || true fi elif [ "$STATE" = running ]; then write_status failed "The operation stopped before applying the change. Check sudo journalctl -u devgrail-certificates.service." || true fi report 2>/dev/null || true if $INSTALL_LOCKED; then rm -f "$INSTALL_DIR/.install.owner"; fi exit "$rc" } if [ "${1:-}" = --report ] && [ "$#" -eq 1 ]; then report; exit 0 fi exec 8>"$WORK/.lock" lock_wait 8 5 || exit 1 trap on_exit EXIT trap 'exit 1' TERM INT STARTED="$(now)" if [ "${1:-}" = --apply ] && [ "$#" -eq 2 ]; then CLI=true # Invoked by the root maintenance CLI, also works without systemd. TARGET="$2" # SSH recovery supersedes a queued staging request. Otherwise the watcher # could undo recovery immediately after this command releases its lock. rm -rf -- "$REQUEST_DIR/request" elif [ "$#" -eq 0 ]; then # Move untrusted input into root-only work before examining it. Never follow # symlinks or read a FIFO planted in the server-writable request directory. rm -rf -- "$WORK/request" mv -fT -- "$REQUEST_DIR/request" "$WORK/request" 2>/dev/null || exit 0 [ ! -L "$WORK/request" ] && [ -f "$WORK/request" ] || reject "The request was not a plain file." [ "$(stat -c %s "$WORK/request")" -le 32 ] || reject "The request was too large." TARGET="$(cat "$WORK/request")" rm -f "$WORK/request" else reject "Use devgrail certificates production or devgrail certificates staging." fi case "$TARGET" in production) CA="$PRODUCTION"; STORE=acme.json ;; staging) CA="$STAGING"; STORE=acme-staging.json ;; *) TARGET=""; reject "The environment must be production or staging." ;; esac write_status running "Waiting for any installation or upgrade to finish." # Participate in the installer's lock AND owner record. Its leaked-lock # recovery must see this helper as a live owner, or it could unlink our lock. exec 9>>"$INSTALL_DIR/.install.lock" lock_wait 9 300 || { write_status failed "An installation or upgrade still holds the host lock. Try again when it finishes."; exit 1; } INSTALL_LOCKED=true printf '%s\n' "$$" > "$INSTALL_DIR/.install.owner" : > "$LOG"; chmod 640 "$LOG" case "$(value DEVGRAIL_TLS | tr '[:upper:]' '[:lower:]')" in off|false|0|no) reject "TLS is off; the configuration was left unchanged." ;; esac # A killed helper leaves its journal. Recover it before another operation, # including after a host reboot, instead of overwriting the only backup. if [ -f "$WORK/env.before" ]; then log "Recovering an interrupted certificate change." rollback || { write_status failed "Interrupted change recovery failed. Check the host logs and retained backup."; exit 1; } rm -f "$WORK/env.before" fi if info="$(inspect_config 2>/dev/null)"; then printf '%s' "$info" | jq '.networks' > "$WORK/networks.json" elif [ ! -f "$WORK/networks.json" ]; then printf '{}\n' > "$WORK/networks.json" fi cp -p "$ENV_FILE" "$WORK/env.before" # Preserve all unrelated entries (including passwords), comments and permissions. cp -p "$ENV_FILE" "$DEPLOY_DIR/.env.certificates.tmp" awk '!/^(DEVGRAIL_ACME_CASERVER|DEVGRAIL_ACME_STORAGE)=/' "$ENV_FILE" > "$DEPLOY_DIR/.env.certificates.tmp" printf '\nDEVGRAIL_ACME_CASERVER=%s\nDEVGRAIL_ACME_STORAGE=%s\n' "$CA" "$STORE" >> "$DEPLOY_DIR/.env.certificates.tmp" mv -f "$DEPLOY_DIR/.env.certificates.tmp" "$ENV_FILE" CHANGED=true # Inspect the effective Compose command before causing downtime. Overrides # that hard-code a resolver must not make this form appear to work silently. effective="$(compose config --format json)" printf '%s' "$effective" | jq -e --arg ca "--certificatesresolvers.letsencrypt.acme.caserver=$CA" \ --arg store "--certificatesresolvers.letsencrypt.acme.storage=/acme/$STORE" \ '.services.traefik.command | index($ca) != null and index($store) != null' >/dev/null || { # No container has changed yet; restore .env without recreating Traefik. cp -p "$WORK/env.before" "$ENV_FILE"; CHANGED=false; rm -f "$WORK/env.before" reject "A Compose override prevents changing the ACME resolver. The previous setting was preserved." } write_status running "Applying the certificate environment and restoring workspace connections." log "Switching to $TARGET, preserving both ACME stores." recreate || exit 1 info="$(inspect_config)" [ "$(printf '%s' "$info" | flag caserver)" = "$CA" ] && [ "$(printf '%s' "$info" | flag storage)" = "/acme/$STORE" ] || exit 1 DOMAIN="$(value DEVGRAIL_DOMAIN)" case "$DOMAIN" in ''|*[!a-zA-Z0-9.-]*) exit 1 ;; esac PORT="$(value HTTPS_PORT)"; PORT="${PORT:-443}" case "$PORT" in *[!0-9]*|'') exit 1 ;; esac write_status running "Configuration applied. Checking dashboard connectivity and its certificate (up to 90 seconds)." verified=false deadline=$((SECONDS + 90)) while [ "$SECONDS" -lt "$deadline" ]; do # Trust staging only in this narrow connectivity probe; the production probe # requires a publicly trusted certificate. Domain/SNI still match the dashboard. curl_args=() [ "$TARGET" != staging ] || curl_args+=(-k) if curl "${curl_args[@]}" -fsS --max-time 2 --resolve "$DOMAIN:$PORT:127.0.0.1" \ "https://$DOMAIN:$PORT/api/v1/healthz" >/dev/null 2>&1; then if [ "$TARGET" = production ] || timeout 3 openssl s_client -connect "127.0.0.1:$PORT" -servername "$DOMAIN" </dev/null 2>/dev/null \ | openssl x509 -noout -issuer 2>/dev/null | grep -qi '(STAGING)'; then verified=true; break; fi fi sleep 1 done $verified || exit 1 CHANGED=false rm -f "$WORK/env.before" write_status succeeded "The $TARGET configuration is active and the dashboard is reachable. Other domains may still be waiting for certificates; check the Traefik log if they do not serve." log "Configuration applied and dashboard connectivity verified." CERTHELPER } # Independent of dashboard upgrades. CLI recovery is available without systemd; # only the web request watcher and fresh configuration reports need systemd. CERTIFICATES_HELPER_DIR="$INSTALL_DIR/certificates" CERTIFICATES_HELPER="$CERTIFICATES_HELPER_DIR/devgrail-certificates-helper" CERTIFICATES_REQUEST_DIR="$DEPLOY_DIR/certificates-request" CERTIFICATES_STATUS_DIR="$DEPLOY_DIR/certificates-status" mkdir -p "$CERTIFICATES_HELPER_DIR" "$CERTIFICATES_REQUEST_DIR" "$CERTIFICATES_STATUS_DIR" chmod 700 "$CERTIFICATES_HELPER_DIR" "$CERTIFICATES_REQUEST_DIR" chmod 755 "$CERTIFICATES_STATUS_DIR" # A manual installer run supersedes pending requests and interrupted journals: # its freshly written .env is now the operator's authoritative choice. rm -rf -- "$CERTIFICATES_REQUEST_DIR/request" rm -f "$CERTIFICATES_HELPER_DIR/work/env.before" "$CERTIFICATES_HELPER_DIR/work/networks.json" rm -f "$CERTIFICATES_STATUS_DIR/status.json" { printf '#!/usr/bin/env bash\nINSTALL_DIR=%q\nHELPER_DIR=%q\nREQUEST_DIR=%q\nSTATUS_DIR=%q\n' \ "$INSTALL_DIR" "$CERTIFICATES_HELPER_DIR" "$CERTIFICATES_REQUEST_DIR" "$CERTIFICATES_STATUS_DIR" certificates_helper_body } > "$CERTIFICATES_HELPER.tmp" chmod 700 "$CERTIFICATES_HELPER.tmp" mv -f "$CERTIFICATES_HELPER.tmp" "$CERTIFICATES_HELPER" rm -f "$CERTIFICATES_STATUS_DIR/helper.json" if command -v systemctl >/dev/null 2>&1 && [ -d /run/systemd/system ]; then cat > /etc/systemd/system/devgrail-certificates.service <<EOF [Unit] Description=Apply DevGrail certificate environment After=docker.service [Service] Type=oneshot ExecStart=$CERTIFICATES_HELPER Environment=HOME=/root TimeoutStartSec=10min EOF cat > /etc/systemd/system/devgrail-certificates.path <<EOF [Unit] Description=Watch DevGrail certificate requests [Path] PathExists=$CERTIFICATES_REQUEST_DIR/request [Install] WantedBy=paths.target EOF cat > /etc/systemd/system/devgrail-certificates-report.service <<EOF [Unit] Description=Report DevGrail certificate configuration After=docker.service [Service] Type=oneshot ExecStart=$CERTIFICATES_HELPER --report Environment=HOME=/root TimeoutStartSec=20s EOF cat > /etc/systemd/system/devgrail-certificates-report.timer <<EOF [Unit] Description=Refresh DevGrail certificate configuration [Timer] OnBootSec=10s OnUnitActiveSec=30s AccuracySec=1s [Install] WantedBy=timers.target EOF if systemctl daemon-reload 9>&- && systemctl enable --now devgrail-certificates.path devgrail-certificates-report.timer 9>&-; then printf '{"protocol":1}\n' > "$CERTIFICATES_STATUS_DIR/helper.json.tmp" chmod 644 "$CERTIFICATES_STATUS_DIR/helper.json.tmp" mv -f "$CERTIFICATES_STATUS_DIR/helper.json.tmp" "$CERTIFICATES_STATUS_DIR/helper.json" else warn "Could not enable certificate changes from Administration. SSH recovery remains available: sudo devgrail certificates production" fi fi # --- upgrade helper ---------------------------------------------------------- # What lets an admin upgrade DevGrail from the dashboard — without the # dashboard running anything on this host. # # devgrail-server can only leave a one-line request in $UPGRADE_REQUEST_DIR. A # systemd path unit notices it and runs the helper written below, as root. The # helper decides everything the request does not say: it accepts only "upgrade # <newer release>", verifies the release signature and the new installer's # digest against keys this installer wrote, and runs that installer unattended # with --auto-rollback. It reports back through # $UPGRADE_STATUS_DIR, which the server mounts read-only. # # The whole design assumes devgrail-server may be compromised, so the rules that # make it safe are properties of the files, not of the code that writes them — # docs/SECURITY.md, "Upgrading from the dashboard", lists them. In short: the # helper and its keys live outside every container mount; the status directory # is read-only to the server by mount flag (the server is host uid 0, so file # ownership would not stop it); and the request directory is the only place the # server can write, and holds nothing the installer reads. UPGRADE_HELPER_DIR="$INSTALL_DIR/upgrade" UPGRADE_HELPER="$UPGRADE_HELPER_DIR/devgrail-upgrade-helper" UPGRADE_REQUEST_DIR="$DEPLOY_DIR/upgrade-request" UPGRADE_STATUS_DIR="$DEPLOY_DIR/upgrade-status" UPGRADE_HELPER_NOTE="" # upgrade_helper_body — the fixed part of the helper. Quoted heredoc: nothing in # it is expanded here, so what is on this page is exactly what runs. The # install-specific constants are printed above it by write_upgrade_helper. # # Extracted by test/installer/upgrade_helper_test.sh from this marker to the # HELPER terminator — keep the function a single heredoc. upgrade_helper_body() { cat <<'HELPER' # Everything above this line describes the install it was written for; # everything below is fixed. It runs as root under devgrail-upgrade.service # whenever devgrail-server leaves a request, and it treats that request as # hostile: devgrail-server is the process this design assumes may be compromised. # # The request says exactly one thing, and nothing else is read from it: # upgrade vX.Y.Z install that release, which must be newer than this one # Where to download from, which keys to trust and which flags to pass are all # decided here, from files only root can write. set -euo pipefail umask 077 WORK="$HELPER_DIR/work" KEYS="$HELPER_DIR/release-keys.pem" EXTRA_KEYS="$HELPER_DIR/extra-release-keys.pem" VERSIONS_FILE="$INSTALL_DIR/versions.env" STATUS="$STATUS_DIR/status.json" LOG="$STATUS_DIR/upgrade.log" REQ="$WORK/request" RUN="$WORK/run" MAX_REQUEST_BYTES=64 RELEASE_RE='^v[0-9]{1,4}\.[0-9]{1,4}\.[0-9]{1,6}$' ACTION=""; TARGET=""; FROM=""; STARTED=""; STATE="" now() { date -u +%Y-%m-%dT%H:%M:%SZ; } log() { printf '[%s] %s\n' "$(now)" "$*" >> "$LOG"; } # write_status <state> <message> — replace status.json in one rename. Writing # here as root is safe because the server mounts this directory read-only: # nothing in it can have been planted by the process this helper distrusts. write_status() { local finished="" STATE="$1" [ "$1" = running ] || finished="$(now)" jq -n --arg state "$1" --arg message "$2" --arg action "$ACTION" \ --arg target "$TARGET" --arg from "$FROM" --arg started "$STARTED" \ --arg finished "$finished" \ '{protocol: 1, state: $state, action: $action, target_version: $target, from_version: $from, started_at: $started, finished_at: $finished, message: $message}' > "$STATUS_DIR/.status.json.tmp" chmod 644 "$STATUS_DIR/.status.json.tmp" mv -f "$STATUS_DIR/.status.json.tmp" "$STATUS" } # reject <message> — refuse the request. Exits 0: refusing is this helper # working, and must not count towards the unit's start limit. reject() { write_status rejected "$1"; exit 0; } # fail <message> — a valid request that could not be carried out. fail() { log "$1"; write_status failed "$1"; exit 1; } # An exit nobody planned for (set -e) must not leave "running" on screen. on_exit() { local rc=$? if [ "$STATE" = running ]; then write_status failed "The upgrade helper stopped unexpectedly (exit $rc). See the log." || true fi rm -rf -- "$RUN" "$REQ" 2>/dev/null || true } # release_gt <a> <b> — true when release a is newer than b. Both have already # matched RELEASE_RE; 10# stops a component like "08" being read as octal. release_gt() { local -a a b local i IFS=. read -r -a a <<< "${1#v}" IFS=. read -r -a b <<< "${2#v}" for i in 0 1 2; do if (( 10#${a[i]} > 10#${b[i]} )); then return 0; fi if (( 10#${a[i]} < 10#${b[i]} )); then return 1; fi done return 1 } mkdir -p "$WORK" "$STATUS_DIR" chmod 700 "$HELPER_DIR" "$WORK" # One run at a time. A trigger that finds the lock held leaves the request where # it is; the path unit fires again once this run ends and the file still exists. exec 8>"$WORK/.lock" flock -n 8 || exit 0 trap on_exit EXIT # Take the request out of devgrail-server's reach before looking at it. rename(2) # moves a symlink or a FIFO as itself, never what it points at, so nothing the # checks below examine can be swapped underneath them. rm -rf -- "$REQ" mv -fT -- "$REQUEST_DIR/request" "$REQ" 2>/dev/null || exit 0 STARTED="$(now)" if [ -L "$REQ" ] || [ ! -f "$REQ" ]; then reject "The request was not a plain file." fi if [ "$(stat -c %s -- "$REQ")" -gt "$MAX_REQUEST_BYTES" ]; then reject "The request was larger than $MAX_REQUEST_BYTES bytes." fi # Command substitution drops trailing newlines, so a request with or without a # final newline reads alike; a line break or carriage return left inside it # after that means more than one line, which is refused rather than truncated. line="$(cat -- "$REQ" 2>/dev/null)" || true case "$line" in *$'\n'*|*$'\r'*) reject "The request had more than one line." ;; esac # The requested version is not echoed back anywhere until it has matched. case "$line" in "upgrade "*) ACTION=upgrade; TARGET="${line#upgrade }" ;; *) reject "The request was not one this helper understands." ;; esac if ! [[ $TARGET =~ $RELEASE_RE ]]; then TARGET="" reject "The requested version is not a release version (vMAJOR.MINOR.PATCH)." fi # The installed release, from the file install.sh writes. Parsed, not sourced. if [ -f "$VERSIONS_FILE" ]; then FROM="$(sed -n 's/^RELEASE_VERSION=//p' "$VERSIONS_FILE")" FROM="${FROM%%$'\n'*}" fi if ! [[ $FROM =~ $RELEASE_RE ]]; then FROM="" reject "Cannot tell which release is installed here. Upgrade once from the command line." fi # Newer only. An older release is signed too, and a compromised server asking # for one is asking to reopen whatever that release had wrong. release_gt "$TARGET" "$FROM" \ || reject "$TARGET is not newer than the installed $FROM. Only upgrades can be requested from the dashboard; --version on the command line does the rest." : > "$LOG" chmod 640 "$LOG" write_status running "Verifying release $TARGET." log "Request: $ACTION (installed $FROM, target $TARGET)" rm -rf -- "$RUN" mkdir -m 700 "$RUN" # --- verify the release, then the installer it ships --------------------------- # The same checks install.sh makes of every artifact, made here of install.sh # itself — which a `curl | sudo bash` upgrade never checks at all. curl -fsSL --max-time 60 --max-filesize 5242880 \ "$WEB/api/registry/releases/$TARGET" -o "$RUN/manifest.json" \ || fail "Could not fetch release $TARGET from $WEB." [ "$(jq -r '.release_version // empty' "$RUN/manifest.json")" = "$TARGET" ] \ || fail "The registry answered with a different release than $TARGET." [ "$(jq -r '.signature.algorithm // empty' "$RUN/manifest.json")" = "ecdsa-p256-sha256" ] \ || fail "Release $TARGET is unsigned, or signed with a scheme this helper cannot verify." jq -j '.signature.payload' "$RUN/manifest.json" > "$RUN/payload" \ || fail "Release $TARGET has no signed payload." jq -r '.signature.signature' "$RUN/manifest.json" | base64 -d > "$RUN/sig" 2>/dev/null \ || fail "The signature on release $TARGET is not valid base64." awk -v d="$RUN" '/-----BEGIN PUBLIC KEY-----/{n++} n>0 {print > (d "/key" n ".pem")}' "$KEYS" verified=false for pem in "$RUN"/key*.pem; do [ -f "$pem" ] || continue if openssl dgst -sha256 -verify "$pem" -signature "$RUN/sig" "$RUN/payload" >/dev/null 2>&1; then verified=true break fi done [ "$verified" = true ] \ || fail "The signature on release $TARGET does not verify against any key this host trusts." grep -qxF "devgrail-release-signature-v1" "$RUN/payload" \ || fail "The signed payload of release $TARGET is not in a format this helper understands." grep -qxF "release $TARGET" "$RUN/payload" \ || fail "The signature was made over a different release than $TARGET." asset() { jq -r --arg f "$1" '[.assets[]? | select(.slug == "install.sh")][0][$f] // empty' "$RUN/manifest.json"; } ver="$(asset asset_version)" url="$(asset download_url)" sha="$(asset sha256)" if ! [[ $ver =~ ^[0-9]+$ ]] || ! [[ $sha =~ ^[0-9a-f]{64}$ ]]; then fail "Release $TARGET does not publish an installer this helper can verify." fi case "$url" in /*) url="$WEB$url" ;; https://*|http://*) ;; *) fail "Release $TARGET names an installer URL this helper will not fetch." ;; esac curl -fsSL --max-time 600 --max-filesize 10485760 "$url" -o "$RUN/install.sh" \ || fail "Could not download the $TARGET installer." got="$(sha256sum "$RUN/install.sh")" got="${got%% *}" [ "$got" = "$sha" ] \ || fail "The $TARGET installer does not match its published checksum." grep -qxF "asset install.sh $ver $got" "$RUN/payload" \ || fail "The $TARGET installer is not the one release $TARGET was signed for." log "Release $TARGET and its installer verified." # --- run it ------------------------------------------------------------------ # Unattended, so every question takes its prior answer. The three daemon and # timer questions are answered no outright: --yes would otherwise take the # backup prompt's default of yes, and an upgrade nobody watched must not change # more than the release. --auto-rollback because nobody is at a shell to answer. args=(--version="$TARGET" --yes --auto-rollback) write_status running "Upgrading from $FROM to $TARGET." envs=(DEVGRAIL_WEB_URL="$WEB" DEVGRAIL_INSTALL_DIR="$INSTALL_DIR" DEVGRAIL_USERNS_REMAP=no DEVGRAIL_ADDRESS_POOLS=no DEVGRAIL_SCHEDULED_BACKUPS=no) if [ -s "$EXTRA_KEYS" ]; then envs+=(DEVGRAIL_RELEASE_PUBKEY_FILE="$EXTRA_KEYS") fi log "Running the $TARGET installer: ${args[*]}" set +e env "${envs[@]}" bash "$RUN/install.sh" "${args[@]}" < /dev/null >> "$LOG" 2>&1 8>&- rc=$? set -e if [ "$rc" -ne 0 ]; then fail "The installer exited with status $rc. An upgrade that does not come up healthy is rolled back automatically; the log has the details." fi write_status succeeded "Upgraded from $FROM to $TARGET." HELPER } # write_upgrade_helper — the helper, its trusted keys, and the two units. write_upgrade_helper() { mkdir -p "$UPGRADE_HELPER_DIR" || return 1 chmod 700 "$UPGRADE_HELPER_DIR" || return 1 # Through a temp file and a rename: this may be the run the helper itself # started, and bash reads a script as it goes — overwriting it in place would # change the lines it has yet to reach. A rename leaves it the old inode. { printf '#!/usr/bin/env bash\n' printf '# DevGrail upgrade helper, written by install.sh. Do not edit: re-running the\n' printf '# installer overwrites it. See docs/SECURITY.md, "Upgrading from the dashboard".\n' printf 'INSTALL_DIR=%q\n' "$INSTALL_DIR" printf 'WEB=%q\n' "$WEB" printf 'HELPER_DIR=%q\n' "$UPGRADE_HELPER_DIR" printf 'REQUEST_DIR=%q\n' "$UPGRADE_REQUEST_DIR" printf 'STATUS_DIR=%q\n' "$UPGRADE_STATUS_DIR" upgrade_helper_body } > "$UPGRADE_HELPER.tmp" || return 1 chmod 700 "$UPGRADE_HELPER.tmp" && mv -f "$UPGRADE_HELPER.tmp" "$UPGRADE_HELPER" || return 1 # Two files, not one: the helper verifies with both, but passes on only the # extras. Handing the full set to the next installer would make it write # every key it was given back out, and a key rotated out of the embedded set # would never actually leave. release_public_keys > "$UPGRADE_HELPER_DIR/release-keys.pem.tmp" \ && mv -f "$UPGRADE_HELPER_DIR/release-keys.pem.tmp" "$UPGRADE_HELPER_DIR/release-keys.pem" || return 1 release_extra_public_keys > "$UPGRADE_HELPER_DIR/extra-release-keys.pem.tmp" \ && mv -f "$UPGRADE_HELPER_DIR/extra-release-keys.pem.tmp" "$UPGRADE_HELPER_DIR/extra-release-keys.pem" || return 1 chmod 600 "$UPGRADE_HELPER_DIR/release-keys.pem" "$UPGRADE_HELPER_DIR/extra-release-keys.pem" # HOME because systemd sets none for a root service and the docker CLI wants # one. Two hours bounds a run that hangs; the server reports a run silent for # longer than that as interrupted. The start limit stops a request that the # helper somehow cannot consume from re-triggering it in a tight loop. cat > /etc/systemd/system/devgrail-upgrade.service <<EOF [Unit] Description=DevGrail upgrade requested from the dashboard After=docker.service network-online.target Wants=network-online.target StartLimitIntervalSec=300 StartLimitBurst=5 [Service] Type=oneshot ExecStart=$UPGRADE_HELPER Environment=HOME=/root TimeoutStartSec=2h EOF cat > /etc/systemd/system/devgrail-upgrade.path <<EOF [Unit] Description=Watch for DevGrail upgrade requests from the dashboard [Path] PathExists=$UPGRADE_REQUEST_DIR/request [Install] WantedBy=paths.target EOF return 0 } # remove_upgrade_helper — switch dashboard upgrades off and say so to the # server, which reads helper.json to decide whether to offer them. remove_upgrade_helper() { rm -f "$UPGRADE_STATUS_DIR/helper.json" if command -v systemctl >/dev/null 2>&1; then systemctl disable --now devgrail-upgrade.path >/dev/null 2>&1 9>&- || true fi rm -f /etc/systemd/system/devgrail-upgrade.path /etc/systemd/system/devgrail-upgrade.service rm -rf "$UPGRADE_HELPER_DIR" } # Both directories are created whatever the answer below: the compose file # mounts them, and a mount Docker has to create itself is a root-owned 0755 # directory, not the modes chosen here. 0700 on the request directory is for # everyone else on the host; the server is uid 0 and writes it regardless. mkdir -p "$UPGRADE_REQUEST_DIR" "$UPGRADE_STATUS_DIR" chmod 700 "$UPGRADE_REQUEST_DIR" chmod 755 "$UPGRADE_STATUS_DIR" # A request left over from before this run is superseded by it: whatever it # asked for, the operator has just done something else by hand. rm -rf -- "$UPGRADE_REQUEST_DIR/request" if [ "$DEVGRAIL_DASHBOARD_UPGRADES" != yes ]; then remove_upgrade_helper UPGRADE_HELPER_NOTE="Dashboard upgrades: off (DEVGRAIL_DASHBOARD_UPGRADES=no)." elif ! command -v systemctl >/dev/null 2>&1 || [ ! -d /run/systemd/system ]; then rm -f "$UPGRADE_STATUS_DIR/helper.json" UPGRADE_HELPER_NOTE="Dashboard upgrades: unavailable without systemd — upgrade with the command above." else log "Enabling upgrades from the dashboard..." if write_upgrade_helper && systemctl daemon-reload >/dev/null 2>&1 9>&- && systemctl enable --now devgrail-upgrade.path >/dev/null 2>&1 9>&-; then printf '{"protocol":1}\n' > "$UPGRADE_STATUS_DIR/helper.json.tmp" && chmod 644 "$UPGRADE_STATUS_DIR/helper.json.tmp" && mv -f "$UPGRADE_STATUS_DIR/helper.json.tmp" "$UPGRADE_STATUS_DIR/helper.json" UPGRADE_HELPER_NOTE="Dashboard upgrades: on (systemctl status devgrail-upgrade.path). Turn off with --dashboard-upgrades=no." else rm -f "$UPGRADE_STATUS_DIR/helper.json" warn "could not enable upgrades from the dashboard — upgrade by re-running this installer" UPGRADE_HELPER_NOTE="Dashboard upgrades: could not be enabled — upgrade with the command above." fi fi # --- bring up -------------------------------------------------------------- docker network inspect devgrail >/dev/null 2>&1 || docker network create devgrail >/dev/null step "Starting DevGrail..." # Via compose(), so a docker-compose.override.yml is honoured here too — not # just by the stop/logs calls above. An override that only applied to some of # the installer's compose invocations would be worse than none. compose up -d # --- wait for readiness ----------------------------------------------------- # probe_health / wait_for_health are defined near the top, because --rollback # needs them too. if ! wait_for_health; then echo >&2 printf '\033[1;31merror:\033[0m %s\n' "DevGrail did not become ready within ${HEALTH_TIMEOUT}s." >&2 echo " Last 50 lines from devgrail-server:" >&2 compose logs --tail=50 devgrail-server >&2 || true echo >&2 echo " Health detail:" >&2 curl -sSk --max-time 5 "$(dashboard_url)/api/v1/healthz" >&2 || true echo >&2 # Automatic rollback, on an upgrade only. A first install has nothing to roll # back to: there are no :previous images and no snapshot, and "rolling back" to # an empty host would replace a diagnosable failure with a confusing one. ROLLED_BACK=false if [ "$IS_UPGRADE" = true ]; then # --yes deliberately does not imply yes here. Rolling back is a decision # about which of two states the host ends in, not a question in the way of # an unattended run; --auto-rollback is how that one is answered up front. DO_IT="$AUTO_ROLLBACK" if [ "$DO_IT" != true ] && [ "$ASSUME_YES" != true ] && have_tty; then # ask_yn, so an answer that is neither yes nor no is asked again here too. if ask_yn "Roll back to the previous version and snapshot?" n; then DO_IT=true fi fi if [ "$DO_IT" = true ]; then echo >&2 if do_rollback; then ROLLED_BACK=true else printf '\033[1;31merror:\033[0m %s\n' "The rolled-back stack did not become ready either." >&2 fi fi fi # Say plainly which state the host ended in. After a failed upgrade that is the # only thing the operator needs to know before deciding what to do next. if [ "$ROLLED_BACK" = true ]; then cat >&2 <<EOF ============================================================ The upgrade failed and was ROLLED BACK. This host is running the previous version again and is answering its readiness probe at $(dashboard_url)/ Nothing was lost. Report the log above before retrying the upgrade. ============================================================ EOF exit 1 fi if [ -n "$SNAPSHOT_ARCHIVE" ]; then echo " A pre-upgrade snapshot of the database and config.yaml was taken first:" >&2 echo " $SNAPSHOT_ARCHIVE" >&2 echo " Nothing was deleted, but the new server may already have migrated the" >&2 echo " database — that archive is the way back to the previous state." >&2 echo " To go back: curl -fsSL $WEB/install.sh | sudo bash -s -- --rollback" >&2 fi echo " Logs: docker compose --project-directory $DEPLOY_DIR logs -f" >&2 exit 1 fi # Publish a fresh observation after recreate, so Administration and smoke checks # do not temporarily show the previous Traefik container's environment. "$CERTIFICATES_HELPER" --report 9>&- || warn "Could not refresh the certificate report; check the host report timer." "$INSTALL_DIR/backups-host/devgrail-backups-helper" --report 9>&- || true if command -v systemctl >/dev/null 2>&1 && [ -d /run/systemd/system ]; then systemctl start --no-block devgrail-backups-index.service 9>&- || true fi # --- reclaim superseded images ---------------------------------------------- # Only now, and never earlier: :previous is the rollback target, and until the # new stack has answered its readiness probe it is the thing standing between a # bad upgrade and a broken host. Once health is green, every DevGrail image older # than :previous is dead weight — an upgrade loads a new one each time and # nothing else removes them, so a host that upgrades for a year accumulates a # year of unreferenced layers on the disk whose exhaustion causes the failure # this whole sequence exists to survive. reclaim_old_images() { local slug keep_latest keep_prev id removed=0 for slug in devgrail-server devgrail-container; do keep_latest="$(docker image inspect -f '{{.Id}}' "$slug:latest" 2>/dev/null || true)" keep_prev="$(docker image inspect -f '{{.Id}}' "$slug:previous" 2>/dev/null || true)" [ -n "$keep_latest" ] || continue # Every image ID this repository still has, tagged or dangling. Untagged # ones are the displaced builds: `docker load` + retag leaves them with no # reference at all, which is exactly why nothing ever cleaned them up. for id in $(docker images --filter "reference=$slug" --format '{{.ID}}' 2>/dev/null | sort -u); do id="$(docker image inspect -f '{{.Id}}' "$id" 2>/dev/null || true)" [ -n "$id" ] || continue [ "$id" = "$keep_latest" ] && continue [ -n "$keep_prev" ] && [ "$id" = "$keep_prev" ] && continue # Not forced: an image a container is still using must stay, and that is # a refusal to respect rather than override. A workspace created before an # upgrade is running the old devgrail-container image until it is # recreated — removing it out from under the container is not reclamation, # it is breakage. if docker image rm "$id" >/dev/null 2>&1; then removed=$((removed + 1)) fi done done [ "$removed" -gt 0 ] && log "Reclaimed $removed superseded image(s); :latest and :previous kept." return 0 } # Called at the very bottom, after the summary has been printed — see there. # --workspace-image=no means the banner has to be able to say "there is no # workspace image" — and then say what to do about it, because a dashboard that # cannot create a workspace is not an obviously deliberate state three weeks # later. Asked of Docker rather than of the flag: an operator who passed `no` on # a host that already had the image is not missing anything. if docker image inspect devgrail-container:latest >/dev/null 2>&1; then WORKSPACE_VERSION_NOTE="workspace ${NEW_CONTAINER_RELEASE:-unknown}" WORKSPACE_NOTE="" else WORKSPACE_VERSION_NOTE="workspace image: NOT INSTALLED" WORKSPACE_NOTE="Workspace: no workspace image on this host, so no workspace can be created yet (~750 MiB). Everything else works. Fetch it with: curl -fsSL $WEB/install.sh | sudo bash -s -- --workspace-image=yes" fi if [ "$TLS_ENABLED" = true ] && [ "$ACME_CA_IS_PRODUCTION" = false ]; then TLS_NOTE="Certificates come from a NON-PRODUCTION ACME CA ($DEVGRAIL_ACME_CASERVER), so browsers will show this site as untrusted — that is expected, and this install is for testing. Re-run with --acme-ca=production for real certificates. Ensure DNS is pointed here:" elif [ "$TLS_ENABLED" = true ]; then TLS_NOTE="It may take a minute for Let's Encrypt to issue the TLS certificate on first start. Ensure DNS is pointed here:" else TLS_NOTE="Serving plain HTTP (DEVGRAIL_TLS=off) — do not expose this directly to the public internet; run it on a private network or behind a TLS terminator. Ensure DNS is pointed here:" fi # The reference card. It used to be eleven items in the closing banner, mixed in # with the two an operator needs in the next sixty seconds — so the URL and the # password, the only things that cannot be looked up again, were the hardest to # find. This is the half that keeps: it goes to a file, which is also where it # is actually wanted, three weeks later, by somebody who has long since closed # that terminal. Written on every run, so it never describes a previous install. REFERENCE_FILE="$INSTALL_DIR/README-install.txt" write_reference_file() { cat > "$REFERENCE_FILE.tmp" <<EOF DevGrail — installed reference ============================== Written by install.sh on $(date -u '+%Y-%m-%d %H:%M:%S UTC'). Rewritten every run. Dashboard $(dashboard_url)/ Version ${NEW_SERVER_RELEASE:-unknown} ($WORKSPACE_VERSION_NOTE) Admin user $ADMIN_USER Registry $WEB DNS $(dns_records " ") Additional base domains are managed under Settings -> App subdomains in the dashboard, and each needs its own wildcard A record. Day to day Logs docker compose --project-directory $DEPLOY_DIR logs -f Backups UI $(dashboard_url)/admin/backups Backup sudo devgrail backup Restore sudo devgrail restore <archive> List sudo devgrail list Password change it under Settings in the dashboard, or: sudo devgrail passwd Accounts Users in the dashboard, or: sudo devgrail list-users sudo devgrail add-user <name> [--admin] ${BACKUP_TIMER_NOTE:-Weekly backups: not configured.} Upgrading Upgrade curl -fsSL $WEB/install.sh | sudo bash Preview curl -fsSL $WEB/install.sh | sudo bash -s -- --dry-run Pin curl -fsSL $WEB/install.sh | sudo bash -s -- --version=vX.Y.Z Roll back curl -fsSL $WEB/install.sh | sudo bash -s -- --rollback An upgrade snapshots the database and config.yaml into $BACKUP_DIR before the new server starts, and keeps the newest ${SNAPSHOT_RETAIN:-5}. ${UPGRADE_HELPER_NOTE:-Dashboard upgrades: not configured.} Local changes docker-compose.yml and traefik/dynamic.yml under $DEPLOY_DIR are product-owned: they are rewritten on every run, so an edit to them is lost at the next upgrade. Put local changes in $DEPLOY_DIR/docker-compose.override.yml which this installer never reads or writes. Compose merges it over the base. EOF if [ -n "${WORKSPACE_NOTE:-}" ]; then cat >> "$REFERENCE_FILE.tmp" <<EOF Workspace image Not installed on this host, so no workspace can be created yet (~750 MiB). Everything else works. Fetch it with: curl -fsSL $WEB/install.sh | sudo bash -s -- --workspace-image=yes EOF fi mv -f "$REFERENCE_FILE.tmp" "$REFERENCE_FILE" || return 1 chmod 0644 "$REFERENCE_FILE" 2>/dev/null || true return 0 } write_reference_file || { warn "could not write $REFERENCE_FILE"; REFERENCE_FILE=""; } # A generated password exists in exactly one place otherwise: this terminal's # scrollback. A window that closes, a session that scrolls, a `| tee` nobody # ran — and the only credential for the install is gone, with no path back # except editing the database by hand. So it also goes to a root-only file, and # the banner says where. Only on a fresh install: on an upgrade the admin row # already exists and this value was never applied, so writing it would create a # file naming a password that does not work. PASSWORD_FILE="" if [ "$ADMIN_PW_GENERATED" = true ] && [ "$IS_UPGRADE" = false ]; then PASSWORD_FILE="$CONFIG_DIR/initial-admin-password" # Created empty and locked down before the secret goes in, so it is never # briefly world-readable on a host where umask is loose. if : > "$PASSWORD_FILE.tmp" 2>/dev/null && chmod 0600 "$PASSWORD_FILE.tmp" 2>/dev/null; then printf '%s\n' "$ADMIN_PW" >> "$PASSWORD_FILE.tmp" if mv -f "$PASSWORD_FILE.tmp" "$PASSWORD_FILE" 2>/dev/null; then chmod 0600 "$PASSWORD_FILE" 2>/dev/null || true else rm -f "$PASSWORD_FILE.tmp" 2>/dev/null || true PASSWORD_FILE="" fi else rm -f "$PASSWORD_FILE.tmp" 2>/dev/null || true PASSWORD_FILE="" fi [ -n "$PASSWORD_FILE" ] || warn "could not save the generated password to $CONFIG_DIR — it is shown below, and only below." fi # What an operator needs in the next sixty seconds, and nothing else. Anything # that can be looked up later lives in $REFERENCE_FILE, named at the bottom. # # One function, rendered once into a variable and then written out, rather than # the dozen bare `cat <<EOF`s this used to be. Three things depend on it being # a single string: # # - It is written to the terminal as well as to stdout when stdout has been # redirected. `install.sh > install.log` is a normal thing to do, and it # used to mean the one screen an operator actually needs — dashboard URL and # the generated password — went to the file and never to them. # - It is appended to $REFERENCE_FILE, so the summary survives the terminal # it was printed in. `devgrail summary` prints that file back. # - It is emitted before the image reclamation below, which is silent, slow # (docker image rm over ~800 MiB of superseded layers) and pure # housekeeping. Anything that kills the run in that window — a dropped ssh # session, an impatient ^C — used to take the whole summary with it. closing_summary() { cat <<EOF ============================================================ DevGrail is up and answering its readiness probe. Visit the dashboard to get started. Dashboard: $(dashboard_url)/ Login: $ADMIN_USER EOF if [ "$ADMIN_PW_GENERATED" = true ]; then cat <<EOF Password: $ADMIN_PW EOF if [ -n "$PASSWORD_FILE" ]; then cat <<EOF Saved to $PASSWORD_FILE (root only). Delete it once you have signed in and changed the password. EOF else cat <<EOF This is the only place it is shown — copy it now. EOF fi else cat <<EOF Password: the one you set during the install EOF fi cat <<EOF $TLS_NOTE $(dns_records " ") EOF if [ -n "${WORKSPACE_NOTE:-}" ]; then cat <<EOF ${WORKSPACE_NOTE} EOF fi cat <<EOF Version: ${NEW_SERVER_RELEASE:-unknown} ($WORKSPACE_VERSION_NOTE) Logs: docker compose --project-directory $DEPLOY_DIR logs -f EOF if [ -n "$REFERENCE_FILE" ]; then cat <<EOF Reference: upgrading, rollback, backups and local changes are written to $REFERENCE_FILE Print this screen again any time with: sudo devgrail summary EOF fi cat <<EOF ============================================================ EOF if [ -n "$SNAPSHOT_ARCHIVE" ]; then cat <<EOF Pre-upgrade snapshot (database + config.yaml + manifest): $SNAPSHOT_ARCHIVE Keep it until you have confirmed this upgrade. The newest $SNAPSHOT_RETAIN are retained. EOF fi } SUMMARY="$(closing_summary)" printf '%s\n' "$SUMMARY" # Also to the terminal, when stdout is not one. Guarded on both halves: there is # nothing to duplicate to when stdout is already the terminal, and a run with no # terminal at all (cloud-init, CI) must not fail trying to open /dev/tty. if [ ! -t 1 ] && { : > /dev/tty; } 2>/dev/null; then printf '%s\n' "$SUMMARY" > /dev/tty 2>/dev/null || true fi # And to disk, under the reference card written just above — the copy that is # still there tomorrow. Appended rather than folded into write_reference_file # because the password lines are deliberately not in that file: it is 0644, and # the generated password already has a 0600 home of its own. if [ -n "$REFERENCE_FILE" ] && [ -f "$REFERENCE_FILE" ]; then { printf '\nInstall summary (printed at the end of the run)\n' # Minus two blocks: the password, which does not belong in a 0644 file, and # the "Reference:" pointer, which in this file points at itself. Each is a # labelled line plus however many continuation lines follow it at the # deeper indent, so the filter has to carry that state rather than match # line by line. printf '%s\n' "$SUMMARY" | awk ' /^ (Password|Reference):/ { skip = 1; next } skip && /^ / { next } { skip = 0; print } ' } >> "$REFERENCE_FILE" 2>/dev/null || true fi # Housekeeping, deliberately last: the operator has what they need on screen and # on disk before this starts, so nothing that happens here can cost them the # summary. Its one log line lands below the banner, which is the right way round # — it is a note about disk space, not part of the result. reclaim_old_images || true