fix(installer): make the record and rollback paths trustworthy (#2552)

fluxer_docs/src/content/docs/operator/configuration.mdx

@@ -647,6 +647,22 @@ Default empty. The LiveKit secret. Compose fills it from `LIVEKIT_API_SECRET`.
 
 Default empty. The client-facing LiveKit URL. When empty the API derives it from the public API origin as `wss://host/livekit`, or `ws://` under `http`, and keeps a non-default port. Set it only when LiveKit is served from another host. Compose forwards the `.env` value, empty by default.
 
+#### `FLUXER_LIVEKIT_USE_EXTERNAL_IP`
+
+Default `true`. Whether LiveKit discovers the address browsers dial by asking a STUN server. A host that cannot reach one over UDP fails to start with `could not resolve external IP: context deadline exceeded`. Set it to `false` and give `FLUXER_LIVEKIT_NODE_IP` the address instead. Read only by the `livekit` service.
+
+#### `FLUXER_LIVEKIT_NODE_IP`
+
+Default empty. The address LiveKit publishes in its ICE candidates. Empty leaves the choice to the discovery above, and with that discovery off LiveKit falls back to the container's own address, which no browser can reach. Set both together. `docker compose logs livekit` names the address in use as `nodeIP` on the starting line.
+
+#### `FLUXER_LIVEKIT_STUN_PRIMARY`
+
+Default `stun.l.google.com:19302`. The first STUN server the discovery asks. Point it at another server to keep the discovery without reaching Google.
+
+#### `FLUXER_LIVEKIT_STUN_SECONDARY`
+
+Default `stun1.l.google.com:19302`. The second STUN server, used the same way.
+
 #### `FLUXER_LIVEKIT_INTERNAL_URL`
 
 Default empty. The server-side LiveKit control API. Compose sets `http://livekit:7880`.

fluxer_docs/src/content/docs/operator/get-started.mdx

@@ -148,7 +148,7 @@ The run prints one line per phase and ends with the URL to open. It takes severa
 | --- | --- |
 | `--domain <host>` | Hostname the instance answers on. Prompted when absent |
 | `--email <address>` | Contact address written as `FLUXER_VAPID_EMAIL`. Prompted when absent |
-| `--dir <path>` | Working directory. Default `~/fluxer` |
+| `--dir <path>` | Working directory. Default `~/fluxer`, or the current directory when it holds an instance |
 | `--ref <git ref>` | Ref the stack files come from. Defaults to the image tag, and to `main` when that tag is `v1` or `latest` |
 | `--image-tag <tag>` | Image tag the stack runs. Default `v1` |
 | `--tls <mode>` | `bundled` or `proxy`. Default `bundled` |
@@ -255,14 +255,16 @@ A dump and a tarball in `backups`, and every service back to `running`, is the s
 
 ### Remove the instance
 
+:::danger[This deletes every account, message and upload]
+`-v` deletes all seven named volumes. Every account, message, upload, search index and certificate the instance holds is gone, and no `docker compose up -d` brings any of it back. Take the copies above first. To upgrade rather than delete, the command is `sh install.sh --update`, below.
+:::
+
 Take the instance down and delete its data:
 
 ```bash
 docker compose down -v
 ```
 
-`-v` deletes all seven named volumes, so every account, message, upload, search index and certificate the instance holds is gone, and no `docker compose up -d` brings them back. Take the copies above first.
-
 ## Upgrading
 
 The default tag `v1` tracks the latest compatible release. The same script upgrades the instance:
@@ -272,7 +274,7 @@ cd ~/fluxer
 sh install.sh --update
 ```
 
-`--update` records the running images, backs up the database and the uploads, refreshes the four stack files, pulls, recreates, and verifies. It leaves every secret in `.env` untouched. On Windows it is `.\install.ps1 -Update`. Run `sh install.sh --update --dry-run` first to see the plan.
+`--update` records the running images, backs up the database and the uploads, refreshes the five stack files, pulls, recreates, and verifies. It leaves every secret in `.env` untouched. On Windows it is `.\install.ps1 -Update`. Run `sh install.sh --update --dry-run` first to see the plan.
 
 [Upgrading](/operator/upgrading/) covers what the backup holds, rolling back, pinning a release and reclaiming disk.
 

fluxer_docs/src/content/docs/operator/upgrading.mdx

@@ -31,6 +31,7 @@ The repository holds no ref by the name of a pinned image tag. A release tags ea
 
 | Step | What it does |
 | --- | --- |
+| Mint | Writes any key the refreshed stack requires that `.env` does not hold, listed under [Run the upgrade](#run-the-upgrade) |
 | Record | Writes the image references, the image ID each container has, and the tag `.env` names, into a new record directory |
 | Save | Copies `.env` and the five stack files into that record |
 | Dump | Dumps the database with the stack still serving, and checks the custom-format header on the result |
@@ -46,6 +47,8 @@ A failed run leaves the instance running on the images it already had.
 
 `docker compose up -d` does not notice a changed `Caddyfile`. The script restarts `edge` by name to pick it up.
 
+An instance older than the `/livekit` routing needs one edit no upgrade can make for it. [Voice signalling moved to /livekit](#voice-signalling-moved-to-livekit) has it, and voice stays silent until it is made.
+
 The refreshed `docker-compose.yml` renames the `caddy` service to `edge`, and the two publish the same host ports. `--remove-orphans` takes the old container down in the same call, so the new one can bind them. Without it the step stops with `Bind for 0.0.0.0:443 failed`. It also removes any container in the project whose service the loaded Compose files no longer define, so keep `COMPOSE_FILE` the same across an upgrade. [Keep a local compose change](#keep-a-local-compose-change) has the file layout that survives one, and [When the compose file list names a missing file](#when-the-compose-file-list-names-a-missing-file) has the failure a line naming an absent file produces.
 
 The rename also moves the `caddy-data` and `caddy-config` volumes to `edge-data` and `edge-config`, so the old two are left unused and the edge requests its certificate again on the first start.
@@ -86,7 +89,7 @@ COMPOSE_FILE=docker-compose.yml:docker-compose.proxy.yml:local.compose.yml
 
 Compose merges the files left to right, so `local.compose.yml` wins over the ones before it. An upgrade refreshes the five stack files and nothing else, so a file outside that set survives every upgrade untouched. A record copies `.env` and those five files, and no other file from the working directory, so an override file is never backed up with them and belongs wherever the rest of the configuration lives.
 
-`.env` needs no such file. `FLUXER_IMAGE_TAG` is the only line either upgrade mode writes.
+`.env` needs no such file. The only value either upgrade mode replaces there is `FLUXER_IMAGE_TAG`, and the Mint step adds a required key the file does not hold at all.
 
 [Enable the overlay](/operator/reverse-proxy/#enable-the-overlay) has the overlay this is most often used for, and [Docker labels](/operator/reverse-proxy/#docker-labels) has a worked third file.
 

fluxer_docs/src/installer/install.ps1

@@ -194,7 +194,9 @@ function Show-FluxerUsage {
 	Write-FluxerLine 'Options:'
 	Write-FluxerLine '  -Domain <host>          Hostname the instance answers on. Prompted when absent.'
 	Write-FluxerLine '  -Email <address>        Address written as FLUXER_VAPID_EMAIL. Prompted when absent.'
-	Write-FluxerLine '  -Dir <path>             Working directory. Default: the fluxer folder in the home directory.'
+	Write-FluxerLine '  -Dir <path>             Working directory. Default: the fluxer folder in the home'
+	Write-FluxerLine '                          directory, or the working directory itself when -Update or'
+	Write-FluxerLine '                          -Rollback is given and it holds an instance.'
 	Write-FluxerLine '  -Ref <git ref>          Ref the stack files come from. Default: the image tag, and main when that tag is v1 or latest.'
 	Write-FluxerLine '  -ImageTag <tag>         Value written as FLUXER_IMAGE_TAG. Default: v1.'
 	Write-FluxerLine '  -Tls <bundled|proxy>    Certificate mode. Default: bundled.'
@@ -773,6 +775,7 @@ function Get-FluxerComposeProject([string]$TargetDir) {
 }
 
 $script:FluxerComposeOut = ''
+$script:FluxerComposeSelector = ''
 $script:FluxerComposeErr = ''
 
 # One read-only Compose query, with what Compose printed on stderr kept.
@@ -785,9 +788,19 @@ function Invoke-FluxerComposeQuery([string]$Selector) {
 	$result = Invoke-FluxerCapture @('compose', 'config', $Selector)
 	$script:FluxerComposeOut = $result.Text
 	$script:FluxerComposeErr = $result.Error
+	$script:FluxerComposeSelector = $Selector
 	return $result.Code
 }
 
+# The two readers below parse whatever the last query returned, so each one names
+# the selector it belongs to. Reading the wrong one would otherwise hand back the
+# other kind of list with nothing to say it was wrong.
+function Assert-FluxerComposeSelector([string]$Selector) {
+	if ($script:FluxerComposeSelector -ne $Selector) {
+		Stop-Fluxer "Internal error: read of $Selector after a $($script:FluxerComposeSelector) query." $FluxerExitSecret
+	}
+}
+
 # What Compose printed on the last query, indented one level for the caller.
 function Get-FluxerComposeError([string]$Indent) {
 	if ($script:FluxerComposeErr.Length -eq 0) {
@@ -803,6 +816,7 @@ function Get-FluxerComposeError([string]$Indent) {
 # By hand:
 #   docker compose config --images
 function Get-FluxerComposeImages {
+	Assert-FluxerComposeSelector '--images'
 	$refs = @()
 	foreach ($line in $script:FluxerComposeOut.Split("`n")) {
 		$candidate = $line.Trim()
@@ -833,10 +847,22 @@ function Get-FluxerImageId([string]$Reference) {
 #
 # By hand:
 #   docker compose ps -aq | xargs docker inspect --format '{{.Config.Image}} {{.Image}}'
+# The text of a captured stream, indented one level for the caller, or a word
+# saying there was none.
+function Get-FluxerIndented([string]$Text, [string]$Indent) {
+	if ([string]::IsNullOrWhiteSpace($Text)) {
+		return "${Indent}nothing"
+	}
+	return (($Text -split "`n") | ForEach-Object {"$Indent$_"}) -join "`n"
+}
+
 function Get-FluxerRunningImageIds {
 	$ids = Invoke-FluxerCapture @('compose', 'ps', '-aq')
 	$map = @{}
-	if ($ids.Code -ne 0 -or $ids.Text.Length -eq 0) {
+	if ($ids.Code -ne 0) {
+		Stop-Fluxer "docker compose ps failed, so what is running cannot be read and the record would name no image ID at all. Compose printed:`n$(Get-FluxerIndented $ids.Error '  ')" $FluxerExitPrerequisite
+	}
+	if ($ids.Text.Length -eq 0) {
 		return $map
 	}
 	foreach ($container in $ids.Text.Split("`n")) {
@@ -846,7 +872,7 @@ function Get-FluxerRunningImageIds {
 		}
 		$row = Invoke-FluxerCapture @('inspect', '--format', '{{.Config.Image}} {{.Image}}', $candidate)
 		if ($row.Code -ne 0) {
-			continue
+			Stop-Fluxer "docker inspect failed for $candidate, so the image ID under its reference cannot be read and a rollback would have nothing to go back to. Docker printed:`n$(Get-FluxerIndented $row.Error '  ')" $FluxerExitPrerequisite
 		}
 		$parts = $row.Text.Trim().Split(' ')
 		if ($parts.Count -lt 2) {
@@ -925,6 +951,7 @@ function Get-FluxerChangedMounts([string]$TargetDir, [string]$StagingDir) {
 # By hand:
 #   docker compose config --services
 function Get-FluxerComposeServices {
+	Assert-FluxerComposeSelector '--services'
 	$services = @()
 	foreach ($line in $script:FluxerComposeOut.Split("`n")) {
 		$candidate = $line.Trim()
@@ -1011,6 +1038,30 @@ function Set-FluxerEnvValue([string]$EnvPath, [string]$Name, [string]$Value) {
 # every command this script runs before it reaches the step that would have reported it. Writing
 # the value first is what makes the rest of the run possible. This runs before the record, so the
 # .env the record saves is the one that works.
+# The keys the run would write, without writing any of them. The plan and the run
+# read the same list, so a plan cannot describe a run that does something else.
+function Get-FluxerMissingRequiredSecrets([string]$EnvPath) {
+	$missing = @()
+	foreach ($entry in $FluxerUpgradeSecretKeys) {
+		$current = Get-FluxerEnvValue $EnvPath $entry.Name
+		if ($current.Length -eq 0 -or $current -eq 'CHANGE_ME') {
+			$missing += $entry.Name
+		}
+	}
+	return @($missing)
+}
+
+# A plan writes nothing itself. The run it describes writes .env before it reads
+# anything when a required key is absent, and saying otherwise would describe a
+# different run.
+function Write-FluxerPlanFooter($Missing) {
+	if ($Missing.Count -gt 0) {
+		Write-FluxerLine 'This plan wrote nothing. The run it describes writes the keys above into .env before anything else.'
+	} else {
+		Write-FluxerLine 'Nothing outside a temporary directory was written.'
+	}
+}
+
 function Add-FluxerRequiredSecrets([string]$EnvPath) {
 	foreach ($entry in $FluxerUpgradeSecretKeys) {
 		$current = Get-FluxerEnvValue $EnvPath $entry.Name
@@ -1101,8 +1152,11 @@ function Save-FluxerCurrentFiles([string]$Record, [string]$TargetDir, [string]$E
 	Set-FluxerPrivateFile $envCopy
 	foreach ($name in $FluxerStackFiles) {
 		$source = Join-Path $TargetDir $name
-		if (Test-Path -LiteralPath $source) {
+		$length = Get-FluxerFileLength $source
+		if ($length -gt 0) {
 			Copy-Item -LiteralPath $source -Destination (Join-Path $Record $name) -Force
+		} elseif (Test-Path -LiteralPath $source) {
+			Stop-Fluxer "$source is empty, so the record would hold a file a rollback could not use. Put the file back before upgrading." $FluxerExitBackup
 		}
 	}
 }
@@ -1169,7 +1223,7 @@ function Backup-FluxerDatabase([string]$Record, [string]$TargetDir) {
 	}
 	if (-not (Test-FluxerDumpHeader $dump)) {
 		Remove-FluxerTemporary $dump
-		Stop-Fluxer 'The dump does not carry the custom-format header. The instance is untouched.' $FluxerExitBackup
+		Stop-Fluxer 'The dump does not begin with the custom-format header. The instance is untouched.' $FluxerExitBackup
 	}
 	Write-FluxerLine "Dumped the database to $dump."
 }
@@ -1317,20 +1371,31 @@ function Show-FluxerUpdatePlan([string]$TargetDir, [string]$EnvPath, [string]$Ba
 	Write-FluxerLine "  Ref:        $Ref"
 	Write-FluxerLine "  Image tag:  $(Get-FluxerEnvValue $EnvPath 'FLUXER_IMAGE_TAG') from .env"
 	Write-FluxerLine "  Backup dir: $BackupRoot"
+	$missing = Get-FluxerMissingRequiredSecrets $EnvPath
+	if ($missing.Count -gt 0) {
+		Write-FluxerLine '  Writes .env: the run mints these before it reads anything, because the refreshed stack requires them'
+		foreach ($name in $missing) {
+			Write-FluxerLine "              $name"
+		}
+	}
 	$running = Get-FluxerRunningImageIds
 	if ((Invoke-FluxerComposeQuery '--images') -ne 0) {
 		Write-FluxerLine '  Refusal:    docker compose config --images fails here, and step 1 of the upgrade reads that list'
 		Write-FluxerLine '  Compose said:'
 		Write-FluxerLine (Get-FluxerComposeError '    ')
-		Write-FluxerLine '  Outcome:    the run stops on step 1 and changes nothing'
-		Write-FluxerLine 'Nothing outside a temporary directory was written. Fix what Compose reports, then run this again.'
+		if ($missing.Count -gt 0) {
+			Write-FluxerLine '  Outcome:    the run writes the keys above first, which may be what Compose is missing, so this refusal may not stand'
+		} else {
+			Write-FluxerLine '  Outcome:    the run stops on step 1 and changes nothing'
+		}
+		Write-FluxerPlanFooter $missing
 		return
 	}
 	Write-FluxerLine '  Running now:'
 	foreach ($reference in Get-FluxerComposeImages) {
 		$id = Get-FluxerRunningImageId $running $reference
 		if ($id.Length -eq 0) {
-			$id = 'no container carries this image'
+			$id = 'no container runs this image'
 		}
 		Write-FluxerLine "    $reference $id"
 	}
@@ -1370,7 +1435,7 @@ function Show-FluxerUpdatePlan([string]$TargetDir, [string]$EnvPath, [string]$Ba
 		if ($old.Length -gt 0 -and $new.Length -gt 0 -and $old -ne $new) {
 			Write-FluxerLine "  Refusal:    postgres moves from $old to $new, which this script does not do"
 			Write-FluxerLine '  Outcome:    the run stops at that refusal and changes nothing'
-			Write-FluxerLine 'Nothing outside a temporary directory was written.'
+			Write-FluxerPlanFooter $missing
 			return
 		}
 		foreach ($entry in Get-FluxerChangedMounts $TargetDir $staging) {
@@ -1380,7 +1445,8 @@ function Show-FluxerUpdatePlan([string]$TargetDir, [string]$EnvPath, [string]$Ba
 		Remove-FluxerStagingDirectory $staging
 	}
 	Write-FluxerLine '  Commands:   docker compose pull, docker compose up -d'
-	Write-FluxerLine 'Nothing outside a temporary directory was written. Drop -DryRun to run this.'
+	Write-FluxerPlanFooter $missing
+	Write-FluxerLine 'Drop -DryRun to run this.'
 }
 
 function Show-FluxerRollbackPlan([string]$TargetDir, [string]$EnvPath, [string]$BackupRoot) {
@@ -1539,8 +1605,11 @@ function Invoke-FluxerRollback([string]$TargetDir, [string]$EnvPath, [string]$Ba
 
 	foreach ($name in $FluxerStackFiles) {
 		$source = Join-Path $record $name
-		if (Test-Path -LiteralPath $source) {
+		$length = Get-FluxerFileLength $source
+		if ($length -gt 0) {
 			Copy-Item -LiteralPath $source -Destination (Join-Path $TargetDir $name) -Force
+		} elseif (Test-Path -LiteralPath $source) {
+			Stop-Fluxer "$source is empty, so restoring it would replace a working file with nothing. Nothing was restored. Take the file from another record or from the ref the record names." $FluxerExitRefused
 		}
 	}
 	Write-FluxerLine "Stack files in $TargetDir are the ones the record holds."

fluxer_docs/src/installer/install.sh

@@ -54,6 +54,10 @@ FLUXER_RAW_BASE='https://raw.githubusercontent.com/fluxerapp/fluxer'
 FLUXER_STACK_PATH='deploy/self-hosting'
 FLUXER_MIN_ENGINE='24.0.0'
 FLUXER_MIN_COMPOSE='2.20.2'
+# Both overlays this script downloads use the !override tag, which Compose learned
+# in 2.24.4. A stack that loads neither runs on the lower minimum, so the higher
+# one is required only once COMPOSE_FILE names more than one file.
+FLUXER_MIN_COMPOSE_OVERLAY='2.24.4'
 FLUXER_READY_TIMEOUT=600
 FLUXER_READY_INTERVAL=5
 FLUXER_VAPID_ATTEMPTS=8
@@ -170,7 +174,9 @@ Usage: sh install.sh --domain <host> --email <address> [options]
 Options:
   --domain <host>          Hostname the instance answers on. Prompted when absent.
   --email <address>        Address the operator reads. Prompted when absent.
-  --dir <path>             Working directory. Default ~/fluxer.
+  --dir <path>             Working directory. Default ~/fluxer, or the working
+                           directory itself when --update or --rollback is given
+                           and it holds an instance.
   --ref <git ref>          Ref the stack files come from. Default: the image tag,
                            and main when that tag is v1 or latest.
   --image-tag <tag>        Image tag the stack runs. Default v1.
@@ -679,7 +685,7 @@ fluxer_fetch_stack() {
 			fluxer_fail 4 "Downloaded $fluxer_file is empty."
 		fi
 		if [ "$fluxer_file" = 'docker-compose.yml' ] && ! grep -q '^services:' "$fluxer_part"; then
-			fluxer_fail 4 'The downloaded docker-compose.yml carries no services block.'
+			fluxer_fail 4 "The docker-compose.yml downloaded from $opt_ref holds no services block."
 		fi
 	done < "$fluxer_scratch/files"
 }
@@ -731,7 +737,7 @@ fluxer_check_volumes() {
 	fi
 	docker volume ls -q --filter "label=com.docker.compose.project=$fluxer_project" > "$fluxer_scratch/volumes" 2>/dev/null || return 0
 	if [ -s "$fluxer_scratch/volumes" ]; then
-		fluxer_fail 3 "Docker already holds volumes for the $fluxer_project project. They carry the secrets of an earlier install, and this .env does not open them. Run install.sh --update in the directory that holds that instance, or remove the volumes with docker volume rm before you install again."
+		fluxer_fail 3 "Docker already holds volumes for the $fluxer_project project. They hold the secrets of an earlier install, and this .env does not open them. Run install.sh --update in the directory that holds that instance, or remove the volumes with docker volume rm before you install again."
 	fi
 }
 
@@ -961,6 +967,20 @@ fluxer_env_set() {
 #
 # This runs before the record, so the .env the record saves is the one that
 # works and a rollback does not reintroduce the gap.
+# The keys the run would write, one per line, without writing any of them. The
+# plan and the run read the same list, so a plan cannot describe a run that does
+# something else.
+fluxer_missing_required_secrets() {
+	fluxer_upgrade_secret_keys > "$fluxer_scratch/upgrade-keys"
+	while read -r fluxer_key fluxer_kind; do
+		[ -n "$fluxer_key" ] || continue
+		fluxer_current=$(fluxer_env_value "$fluxer_key")
+		case ${fluxer_current:-} in
+			''|CHANGE_ME) printf '%s\n' "$fluxer_key" ;;
+		esac
+	done < "$fluxer_scratch/upgrade-keys"
+}
+
 fluxer_fill_required_secrets() {
 	fluxer_upgrade_secret_keys > "$fluxer_scratch/upgrade-keys"
 	while read -r fluxer_key fluxer_kind; do
@@ -1037,6 +1057,7 @@ fluxer_require_compose_files() {
 	if [ -z "$fluxer_path_sep" ]; then
 		fluxer_path_sep=':'
 	fi
+	fluxer_compose_count=0
 	fluxer_rest=$fluxer_compose_file
 	while [ -n "$fluxer_rest" ]; do
 		case $fluxer_rest in
@@ -1054,6 +1075,7 @@ fluxer_require_compose_files() {
 			/*) fluxer_path=$fluxer_name ;;
 			*) fluxer_path="$opt_dir/$fluxer_name" ;;
 		esac
+		fluxer_compose_count=$((${fluxer_compose_count:-0} + 1))
 		[ ! -e "$fluxer_path" ] || continue
 		if fluxer_stack_files | grep -qxF "$fluxer_name"; then
 			fluxer_fail 2 "COMPOSE_FILE from $fluxer_compose_from names $fluxer_name and $fluxer_path is not there, so every docker compose command in $opt_dir fails and this run stops before it changes anything. This script downloads $fluxer_name, and an instance set up before it existed does not hold that file yet. Put it in place and run this again:
@@ -1062,6 +1084,10 @@ Leave the COMPOSE_FILE line as it is. Without $fluxer_name the edge container bi
 		fi
 		fluxer_fail 2 "COMPOSE_FILE from $fluxer_compose_from names $fluxer_name and $fluxer_path is not there, so every docker compose command in $opt_dir fails. This script does not download $fluxer_name. Put that file back, or take it out of the COMPOSE_FILE line."
 	done
+	if [ "$fluxer_compose_count" -gt 1 ] &&
+		! fluxer_version_ge "$fluxer_compose_version" "$FLUXER_MIN_COMPOSE_OVERLAY"; then
+		fluxer_fail 2 "COMPOSE_FILE from $fluxer_compose_from loads $fluxer_compose_count files and this host runs Compose $fluxer_compose_version. Every overlay this script downloads uses the !override tag, which needs Compose $FLUXER_MIN_COMPOSE_OVERLAY or newer. Upgrade Compose, or load only docker-compose.yml."
+	fi
 }
 
 # One read-only Compose query, with what Compose printed on stderr kept.
@@ -1113,10 +1139,29 @@ fluxer_compose_images() {
 #
 # By hand:
 #   docker compose ps -aq | xargs docker inspect --format '{{.Config.Image}} {{.Image}}'
+# The text of a captured stream, indented one level for the caller, or a word
+# saying there was none. A command that fails without printing is rarer than one
+# that prints the reason, and both read the same way here.
+fluxer_indent_file() {
+	if [ -s "$1" ]; then
+		sed "s/^/$2/" "$1"
+	else
+		printf '%snothing\n' "$2"
+	fi
+}
+
 fluxer_running_image_ids() {
-	docker compose ps -aq > "$fluxer_scratch/containers" 2>/dev/null || return 0
+	if ! docker compose ps -aq > "$fluxer_scratch/containers" 2> "$fluxer_scratch/ps-err"; then
+		fluxer_fail 2 "docker compose ps failed in $opt_dir, so what is running cannot be read and the record would name no image ID at all. Compose printed:
+$(fluxer_indent_file "$fluxer_scratch/ps-err" '  ')"
+	fi
 	[ -s "$fluxer_scratch/containers" ] || return 0
-	xargs docker inspect --format '{{.Config.Image}} {{.Image}}' < "$fluxer_scratch/containers" 2>/dev/null | sort -u
+	if ! xargs docker inspect --format '{{.Config.Image}} {{.Image}}' \
+		< "$fluxer_scratch/containers" > "$fluxer_scratch/inspected" 2> "$fluxer_scratch/inspect-err"; then
+		fluxer_fail 2 "docker inspect failed in $opt_dir, so the image ID under each reference cannot be read and a rollback would have nothing to go back to. Docker printed:
+$(fluxer_indent_file "$fluxer_scratch/inspect-err" '  ')"
+	fi
+	sort -u "$fluxer_scratch/inspected"
 }
 
 # The recorded ID for one reference, or nothing when no container carries it.
@@ -1171,8 +1216,10 @@ fluxer_save_current_files() {
 	chmod 600 "$fluxer_record/.env"
 	while read -r fluxer_file; do
 		[ -n "$fluxer_file" ] || continue
-		if [ -e "$opt_dir/$fluxer_file" ]; then
+		if [ -s "$opt_dir/$fluxer_file" ]; then
 			cp -p "$opt_dir/$fluxer_file" "$fluxer_record/$fluxer_file"
+		elif [ -e "$opt_dir/$fluxer_file" ]; then
+			fluxer_fail 7 "$opt_dir/$fluxer_file is empty, so the record would hold a file a rollback could not use. Put the file back before upgrading."
 		fi
 	done < "$fluxer_scratch/files"
 }
@@ -1220,7 +1267,7 @@ fluxer_dump_postgres() {
 	fi
 	if [ "$(head -c 5 "$fluxer_dump_path")" != 'PGDMP' ]; then
 		rm -f "$fluxer_dump_path"
-		fluxer_fail 7 'The dump does not carry the custom-format header. The instance is untouched.'
+		fluxer_fail 7 'The dump does not begin with the custom-format header. The instance is untouched.'
 	fi
 	fluxer_say "Dumped the database to $fluxer_dump_path."
 }
@@ -1425,19 +1472,42 @@ fluxer_verify_stack() {
 	fi
 }
 
+# A plan writes nothing itself. The run it describes writes .env before it reads
+# anything when a required key is absent, and saying otherwise would describe a
+# different run.
+fluxer_plan_footer() {
+	if [ -s "$fluxer_scratch/plan-secrets" ]; then
+		fluxer_say 'This plan wrote nothing. The run it describes writes the keys above into .env before anything else.'
+	else
+		fluxer_say 'Nothing outside a temporary directory was written.'
+	fi
+}
+
 fluxer_plan_update() {
 	fluxer_say 'Plan: upgrade'
 	fluxer_say "  directory     $opt_dir"
 	fluxer_say "  ref           $opt_ref"
 	fluxer_say "  image tag     $(fluxer_env_value FLUXER_IMAGE_TAG) from .env"
 	fluxer_say "  backup dir    $opt_backup_dir"
+	fluxer_missing_required_secrets > "$fluxer_scratch/plan-secrets"
+	if [ -s "$fluxer_scratch/plan-secrets" ]; then
+		fluxer_say '  writes .env   the run mints these before it reads anything, because the refreshed stack requires them'
+		while read -r fluxer_key; do
+			[ -n "$fluxer_key" ] || continue
+			fluxer_say "                $fluxer_key"
+		done < "$fluxer_scratch/plan-secrets"
+	fi
 	fluxer_running_image_ids > "$fluxer_scratch/running"
 	if ! fluxer_compose_images > "$fluxer_scratch/refs"; then
 		fluxer_say '  refusal       docker compose config --images fails here, and step 1 of the upgrade reads that list'
 		fluxer_say '  compose said'
 		fluxer_compose_error '    '
-		fluxer_say '  outcome       the run stops on step 1 and changes nothing'
-		fluxer_say 'Nothing outside a temporary directory was written. Fix what Compose reports, then run this again.'
+		if [ -s "$fluxer_scratch/plan-secrets" ]; then
+			fluxer_say '  outcome       the run writes the keys above first, which may be what Compose is missing, so this refusal may not stand'
+		else
+			fluxer_say '  outcome       the run stops on step 1 and changes nothing'
+		fi
+		fluxer_plan_footer
 		return 0
 	fi
 	fluxer_say '  running now'
@@ -1445,7 +1515,7 @@ fluxer_plan_update() {
 		[ -n "$fluxer_ref" ] || continue
 		fluxer_id=$(fluxer_recorded_id_for "$fluxer_ref")
 		if [ -z "$fluxer_id" ]; then
-			fluxer_id='no container carries this image'
+			fluxer_id='no container runs this image'
 		fi
 		fluxer_say "    $fluxer_ref $fluxer_id"
 	done < "$fluxer_scratch/refs"
@@ -1480,7 +1550,7 @@ fluxer_plan_update() {
 	if [ -n "$fluxer_old_major" ] && [ -n "$fluxer_new_major" ] && [ "$fluxer_old_major" != "$fluxer_new_major" ]; then
 		fluxer_say "  refusal       postgres moves from $fluxer_old_major to $fluxer_new_major, which this script does not do"
 		fluxer_say '  outcome       the run stops at that refusal and changes nothing'
-		fluxer_say 'Nothing outside a temporary directory was written.'
+		fluxer_plan_footer
 		return 0
 	fi
 	fluxer_note_mounted_changes
@@ -1491,7 +1561,8 @@ fluxer_plan_update() {
 		done < "$fluxer_scratch/restart"
 	fi
 	fluxer_say '  commands      docker compose pull, docker compose up -d'
-	fluxer_say 'Nothing outside a temporary directory was written. Drop --dry-run to run this.'
+	fluxer_plan_footer
+	fluxer_say 'Drop --dry-run to run this.'
 }
 
 fluxer_plan_rollback() {
@@ -1666,8 +1737,10 @@ fluxer_run_rollback() {
 
 	while read -r fluxer_file; do
 		[ -n "$fluxer_file" ] || continue
-		if [ -e "$fluxer_rollback_dir/$fluxer_file" ]; then
+		if [ -s "$fluxer_rollback_dir/$fluxer_file" ]; then
 			cp -p "$fluxer_rollback_dir/$fluxer_file" "$opt_dir/$fluxer_file"
+		elif [ -e "$fluxer_rollback_dir/$fluxer_file" ]; then
+			fluxer_fail 3 "$fluxer_rollback_dir/$fluxer_file is empty, so restoring it would replace a working file with nothing. Nothing was restored. Take the file from another record or from the ref the record names."
 		fi
 	done < "$fluxer_scratch/files"
 	fluxer_say "Stack files in $opt_dir are the ones the record holds."