fix: repair production recovery lock ownership
Independent Staging Quality Gate / validate (push) Successful in 12m23s
Independent Staging Quality Gate / publish (push) Successful in 2m6s

This commit is contained in:
Jesse_Chen
2026-08-16 01:50:50 +08:00
parent 934c4175d3
commit 2b1deff0e9
5 changed files with 43 additions and 7 deletions
+1 -1
View File
@@ -254,7 +254,7 @@ The deploy and migration workflows share the `staging-mutation` Actions concurre
Use Gitea Actions → **Create Production Recovery Point** from the exact current `main` release SHA before every production schema migration. The manual workflow requires `main`, `staging`, public staging health, and the successful release gate to identify the same SHA. It uses the pinned production SSH identity, shares the `production-mutation` lock, and runs `deploy/run-production-recovery.sh` with shell tracing disabled.
The host script writes an AES-256-CBC/PBKDF2 encrypted custom-format PostgreSQL dump under `/opt/jyotisha-production/backups`, restores it into a uniquely named disposable database, validates non-sensitive row/table counts, removes only that disposable database, and writes a mode-`0600` verification manifest. The workflow retrieves only the encrypted dump and verification metadata, verifies the SHA-256 digest, and uploads them as a 30-day Gitea Actions artifact with compression disabled. The artifact-backed `recovery_reference`, `recovery_created_at`, and `restore_verified=true` output are the inputs for **Migrate Production Database**. Do not use the attestation if backup, restore, retrieval, digest validation, or artifact upload fails.
The host script writes an AES-256-CBC/PBKDF2 encrypted custom-format PostgreSQL dump under `/opt/jyotisha-production/backups`, restores it into a uniquely named disposable database, validates non-sensitive row/table counts, removes only that disposable database, and writes a mode-`0600` verification manifest. Before acquiring the shared host lock it validates that `.state` and `backups` are real directories, then uses the constrained deployment-ownership sudo boundary to restore only those directories and an existing regular `mutation.lock` to the current `deploy` UID/GID; it never removes or replaces a lock inode. The workflow retrieves only the encrypted dump and verification metadata, verifies the SHA-256 digest, and uploads them as a 30-day Gitea Actions artifact with compression disabled. The artifact-backed `recovery_reference`, `recovery_created_at`, and `restore_verified=true` output are the inputs for **Migrate Production Database**. Do not use the attestation if backup, restore, retrieval, digest validation, ownership normalization, or artifact upload fails.
Never restore over `jyotisha`, delete the PostgreSQL volume, print `.env.production.database`, expose `PRODUCTION_BACKUP_ENCRYPTION_KEY`, or substitute a staging recovery artifact. The encrypted local archive is preserved for repair; the Actions artifact supplies the required off-host copy.
+22 -5
View File
@@ -18,18 +18,35 @@ umask 077
state_directory="$DEPLOY_PATH/.state"
backup_directory="$DEPLOY_PATH/backups"
environment_file="$DEPLOY_PATH/.env.production.database"
lock_file="$state_directory/mutation.lock"
[ -f "$environment_file" ] && [ ! -L "$environment_file" ] || {
echo "production database environment file is missing or unsafe" >&2
exit 1
}
for directory in "$state_directory" "$backup_directory"; do
if [ -e "$directory" ]; then
[ -d "$directory" ] && [ ! -L "$directory" ] || {
echo "production state or backup directory is unsafe" >&2
exit 1
}
fi
done
install -d -m 700 "$state_directory" "$backup_directory"
[ ! -L "$state_directory" ] && [ ! -L "$backup_directory" ] || {
echo "production state or backup directory is unsafe" >&2
exit 1
}
deployment_uid="$(id -u)"
deployment_gid="$(id -g)"
sudo -n chown "$deployment_uid:$deployment_gid" "$state_directory" "$backup_directory"
chmod 700 "$state_directory" "$backup_directory"
if [ -e "$lock_file" ]; then
[ -f "$lock_file" ] && [ ! -L "$lock_file" ] || {
echo "production mutation lock is unsafe" >&2
exit 1
}
sudo -n chown "$deployment_uid:$deployment_gid" "$lock_file"
chmod 600 "$lock_file"
fi
exec 9>"$state_directory/mutation.lock"
exec 9>"$lock_file"
flock -n 9 || {
echo "another production mutation holds the host lock" >&2
exit 75