fix: pin and serialize staging releases
This commit is contained in:
Jesse_Chen
+7
-7
@@ -149,7 +149,7 @@ The GitHub `staging` Environment contains the secret `STAGING_SSH_PRIVATE_KEY` a
|
||||
|
||||
The repository-level public build inputs are configured at GitHub **Settings -> Secrets and variables -> Actions -> Variables** (the UI is also shown as **Settings → Secrets and variables → Actions → Variables**): `STAGING_SUPABASE_URL` and `STAGING_SUPABASE_ANON_KEY`. They are public build inputs, required for publish, and exposed to the browser; keep them staging-only and never print their values in workflow output, summaries, or support messages. The workflow passes them only as the `NEXT_PUBLIC_*` build arguments after non-empty/HTTPS validation.
|
||||
|
||||
`Staging Backend Quality Gate` runs for `pull_request`, pushes to `staging`, and `workflow_dispatch`. It validates the Python/database/frontend contract; only a successful push to `staging` can publish immutable full-SHA GHCR images. `.github/workflows/deploy-staging.yml` consumes the successful gate's exact SHA, and its manual `deploy_sha` input must identify a full 40-character commit with a successful `staging` gate run.
|
||||
`Staging Backend Quality Gate` runs for relevant `pull_request` paths, pushes to `staging`, and `workflow_dispatch`. It validates the Python/database/frontend contract; only a successful push to `staging` publishes the API/web images and a run-bound manifest containing their `sha256` digests. `.github/workflows/deploy-staging.yml` consumes that exact successful run, validates its manifest against the full 40-character commit, and deploys digest references rather than trusting the discoverability tags.
|
||||
|
||||
The staging env file must include these non-secret selectors so Compose cannot fall back to production paths:
|
||||
|
||||
@@ -165,13 +165,13 @@ After source sync and before `up`, the workflow validates `.env.staging` mode/se
|
||||
|
||||
1. Complete the server and GitHub bootstrap: create both mode-`0600` env files, configure the staging Environment variables/secrets, and configure the repository staging build variables.
|
||||
2. Merge the reviewed change, then push the reviewed SHA to `staging`; do not rely on a `main` workflow dispatch to publish images.
|
||||
3. The `Staging Backend Quality Gate` runs for that push and, when successful, publishes the SHA-tagged API/web images for that exact 40-character commit SHA.
|
||||
4. The automatic `Deploy staging` workflow starts from that successful gate, syncs the exact SHA, and validates both `.env.staging` and `.env.staging.database` before any app change.
|
||||
3. The `Staging Backend Quality Gate` runs for that push and, when successful, publishes API/web images plus an artifact binding the exact SHA to both immutable image digests.
|
||||
4. The automatic `Deploy staging` workflow downloads that gate-run artifact, syncs the exact revision under the shared staging host lock, and validates both `.env.staging` and `.env.staging.database` before any app change.
|
||||
5. If environment validation fails, fix the server-side env files without committing or copying secrets, then manually rerun `Deploy staging` from `main` with the same successful SHA in `deploy_sha`; the workflow rechecks a successful staging gate for that exact SHA.
|
||||
6. If the read-only checker reports a pending migration, stop app deployment and run `Migrate Staging Database` manually with the same full SHA; a successful migration re-dispatches `Deploy staging` with that same SHA.
|
||||
7. Confirm `https://staging.jyotisha.chat/api/health` reports the exact SHA and private API health.
|
||||
|
||||
Application rollback uses the same workflow: manually dispatch `Deploy staging` from `main` with a previous known-good full SHA that has a successful `Staging Backend Quality Gate` run. Database migrations are separate and are not rolled back by an application deployment. Restore a staging database backup before running any destructive migration rehearsal.
|
||||
Application rollback uses the same workflow: manually dispatch `Deploy staging` from the `main` controller with a previous known-good full SHA that has a successful `Staging Backend Quality Gate` run, and explicitly set `allow_rollback=true`. Normal and migration-triggered deployments reject stale, divergent, or backward revisions. Rollback still consumes the selected gate run's digest manifest; database migrations are separate and are not rolled back by an application deployment. Restore a staging database backup before running any destructive migration rehearsal.
|
||||
|
||||
Inspect staging without printing secrets:
|
||||
|
||||
@@ -229,14 +229,14 @@ PostgreSQL is private: `deploy/docker-compose.postgres.yml` has no `ports` mappi
|
||||
Use this order for every staging revision:
|
||||
|
||||
1. Merge to `staging` after reviewing the change.
|
||||
2. Wait for `Staging Backend Quality Gate` to pass and for that exact full SHA's API/web images to be published.
|
||||
2. Wait for `Staging Backend Quality Gate` to pass and publish that exact full SHA's API/web digest manifest.
|
||||
3. The automatic `Deploy staging` workflow checks the exact SHA in read-only migration-check mode before changing API, web, or Caddy. If it reports pending or drifted migrations, stop; do not retry the application deployment as if it were a migration.
|
||||
4. Open **Migrate Staging Database -> Run workflow** and enter the reported full lowercase 40-character SHA in `deploy_sha`. The workflow validates that exact SHA against a successful `staging` gate, checks it out, starts only PostgreSQL, and runs the reviewed migrator.
|
||||
5. A successful migration prints the ordered migration ledger and re-dispatches `Deploy staging` automatically with the same exact SHA. Do not substitute a branch name, a short SHA, or a newer commit.
|
||||
5. A successful migration rechecks that `staging` still points at the same exact SHA, prints the ordered migration ledger, and dispatches the `main` controller for digest-pinned deployment with `allow_rollback=false`. If `staging` advanced during migration, it refuses the stale dispatch. Do not substitute a branch name, a short SHA, or a newer commit.
|
||||
6. Confirm `https://staging.jyotisha.chat/api/health` and verify that its deployment SHA is the SHA from step 2.
|
||||
7. After health verification, create the local encrypted backup described below.
|
||||
|
||||
The read-only checker exits before app changes when a migration is pending. Its message includes the exact SHA and the `Migrate Staging Database` workflow name. A failed migration does not re-dispatch deployment. Application rollback restores a previously verified image/SHA only; it does not roll back database state.
|
||||
The deploy and migration workflows share the `staging-mutation` Actions concurrency group, and their live-tree sync plus Compose work runs under `/opt/jyotisha-staging/.state/mutation.lock`. The synchronized tree explicitly preserves `/backups/`, `.env*`, `.state`, and `.incoming`. The read-only checker exits before app changes when a migration is pending. Its message includes the exact SHA and the `Migrate Staging Database` workflow name. A failed migration does not re-dispatch deployment. Application rollback restores previously recorded digest references and SHA only; it does not roll back database state.
|
||||
|
||||
### Local encrypted staging backups (three-copy limit)
|
||||
|
||||
|
||||
Executable
+203
@@ -0,0 +1,203 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
set +x
|
||||
|
||||
required=(
|
||||
INCOMING_PATH DEPLOY_PATH API_IMAGE WEB_IMAGE DEPLOY_SHA
|
||||
EXPECTED_PREVIOUS_SHA ALLOW_ROLLBACK DOCKER_CONFIG STAGING_URL
|
||||
)
|
||||
for key in "${required[@]}"; do
|
||||
if [ -z "${!key:-}" ]; then
|
||||
echo "required staging deployment input is missing: $key" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
sha_pattern='^[0-9a-f]{40}$'
|
||||
digest_pattern='^ghcr\.io/jesse-ux/jyotisha-(api|web)@sha256:[0-9a-f]{64}$'
|
||||
if [[ ! "$DEPLOY_SHA" =~ $sha_pattern ]] ||
|
||||
[[ ! "$API_IMAGE" =~ $digest_pattern ]] ||
|
||||
[[ ! "$WEB_IMAGE" =~ $digest_pattern ]]; then
|
||||
echo "unsafe staging image identity" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ "$ALLOW_ROLLBACK" != "true" ] && [ "$ALLOW_ROLLBACK" != "false" ]; then
|
||||
echo "invalid rollback authorization" >&2
|
||||
exit 1
|
||||
fi
|
||||
case "$INCOMING_PATH" in
|
||||
"$DEPLOY_PATH"/.incoming/*) ;;
|
||||
*) echo "unsafe incoming staging path" >&2; exit 1 ;;
|
||||
esac
|
||||
|
||||
state_directory="$DEPLOY_PATH/.state"
|
||||
install -d -m 700 "$state_directory"
|
||||
exec 9>"$state_directory/mutation.lock"
|
||||
flock -n 9 || {
|
||||
echo "another staging mutation holds the host lock" >&2
|
||||
exit 75
|
||||
}
|
||||
|
||||
current_sha="not-deployed"
|
||||
if [ -f "$state_directory/deployed-revision" ]; then
|
||||
current_sha="$(<"$state_directory/deployed-revision")"
|
||||
else
|
||||
existing_web="$(docker ps -aq \
|
||||
--filter 'label=com.docker.compose.project=jyotisha-staging' \
|
||||
--filter 'label=com.docker.compose.service=web' | head -n 1)"
|
||||
if [ -n "$existing_web" ]; then
|
||||
discovered_sha="$(docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' \
|
||||
"$existing_web" | sed -n 's/^GITHUB_SHA=//p' | head -n 1)"
|
||||
if [ -n "$discovered_sha" ]; then current_sha="$discovered_sha"; fi
|
||||
fi
|
||||
fi
|
||||
if [ "$current_sha" != "not-deployed" ] && [[ ! "$current_sha" =~ $sha_pattern ]]; then
|
||||
echo "invalid deployed staging revision state" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ "$current_sha" != "$EXPECTED_PREVIOUS_SHA" ]; then
|
||||
echo "staging revision changed while this deployment was waiting" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ "$ALLOW_ROLLBACK" = "false" ] &&
|
||||
[ "$current_sha" != "not-deployed" ] &&
|
||||
[ "$current_sha" != "$DEPLOY_SHA" ] &&
|
||||
[ "${FORWARD_REVISION_VERIFIED:-false}" != "true" ]; then
|
||||
echo "forward staging revision was not verified" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
container_id() {
|
||||
docker ps -aq \
|
||||
--filter 'label=com.docker.compose.project=jyotisha-staging' \
|
||||
--filter "label=com.docker.compose.service=$1" | head -n 1
|
||||
}
|
||||
|
||||
repo_digest_for_container() {
|
||||
local service="$1"
|
||||
local repository="$2"
|
||||
local id image_id
|
||||
id="$(container_id "$service")"
|
||||
[ -n "$id" ] || return 0
|
||||
image_id="$(docker inspect --format '{{.Image}}' "$id")"
|
||||
docker image inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' "$image_id" |
|
||||
awk -v prefix="$repository@sha256:" 'index($0, prefix) == 1 { print; exit }'
|
||||
}
|
||||
|
||||
previous_api_image="$(repo_digest_for_container api ghcr.io/jesse-ux/jyotisha-api)"
|
||||
previous_web_image="$(repo_digest_for_container web ghcr.io/jesse-ux/jyotisha-web)"
|
||||
previous_api_id=""
|
||||
previous_web_id=""
|
||||
if [ -n "$(container_id api)" ]; then
|
||||
previous_api_id="$(docker inspect --format '{{.Image}}' "$(container_id api)")"
|
||||
fi
|
||||
if [ -n "$(container_id web)" ]; then
|
||||
previous_web_id="$(docker inspect --format '{{.Image}}' "$(container_id web)")"
|
||||
fi
|
||||
|
||||
bash "$INCOMING_PATH/deploy/sync-staging-tree.sh" \
|
||||
"$INCOMING_PATH" "$DEPLOY_PATH"
|
||||
|
||||
cd "$DEPLOY_PATH"
|
||||
bash deploy/validate-staging-env.sh \
|
||||
.env.staging staging.jyotisha.chat deploy/Caddyfile.staging
|
||||
bash deploy/validate-staging-database-env.sh .env.staging.database
|
||||
|
||||
compose=(
|
||||
docker compose -p jyotisha-staging --env-file .env.staging
|
||||
-f deploy/docker-compose.server.yml -f deploy/docker-compose.postgres.yml
|
||||
)
|
||||
export APP_ENV_FILE='../.env.staging'
|
||||
export DATABASE_ENV_FILE='../.env.staging.database'
|
||||
export CADDYFILE_PATH='./Caddyfile.staging'
|
||||
export SITE_ADDRESS='https://staging.jyotisha.chat'
|
||||
export GITHUB_SHA="$DEPLOY_SHA"
|
||||
|
||||
"${compose[@]}" config --quiet
|
||||
"${compose[@]}" pull api web postgres
|
||||
"${compose[@]}" up -d --no-build --wait postgres
|
||||
|
||||
set +e
|
||||
"${compose[@]}" --profile migration-check run --rm migration-checker
|
||||
check_status=$?
|
||||
set -e
|
||||
if [ "$check_status" -eq 3 ]; then
|
||||
echo "pending migrations: run Migrate Staging Database for $DEPLOY_SHA" >&2
|
||||
exit 3
|
||||
fi
|
||||
if [ "$check_status" -ne 0 ]; then
|
||||
echo "staging migration check failed safely" >&2
|
||||
exit "$check_status"
|
||||
fi
|
||||
|
||||
switched=false
|
||||
rollback() {
|
||||
local status=$?
|
||||
if [ "$switched" = "true" ] &&
|
||||
[[ "$previous_api_image" =~ $digest_pattern ]] &&
|
||||
[[ "$previous_web_image" =~ $digest_pattern ]] &&
|
||||
[[ "$current_sha" =~ $sha_pattern ]]; then
|
||||
echo "staging verification failed; restoring prior image digests" >&2
|
||||
API_IMAGE="$previous_api_image" WEB_IMAGE="$previous_web_image" \
|
||||
GITHUB_SHA="$current_sha" \
|
||||
"${compose[@]}" up -d --no-build --remove-orphans api web caddy || true
|
||||
fi
|
||||
exit "$status"
|
||||
}
|
||||
trap rollback ERR
|
||||
|
||||
"${compose[@]}" up -d --no-build --remove-orphans
|
||||
switched=true
|
||||
|
||||
verify_container_image() {
|
||||
local service="$1"
|
||||
local expected_ref="$2"
|
||||
local id expected_id running_id repo_digests
|
||||
id="$(container_id "$service")"
|
||||
[ -n "$id" ]
|
||||
expected_id="$(docker image inspect --format '{{.Id}}' "$expected_ref")"
|
||||
running_id="$(docker inspect --format '{{.Image}}' "$id")"
|
||||
[ "$running_id" = "$expected_id" ]
|
||||
repo_digests="$(docker image inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' "$expected_id")"
|
||||
grep -Fqx "$expected_ref" <<<"$repo_digests"
|
||||
}
|
||||
verify_container_image api "$API_IMAGE"
|
||||
verify_container_image web "$WEB_IMAGE"
|
||||
|
||||
"${compose[@]}" exec -T \
|
||||
-e EXPECTED_SHA="$DEPLOY_SHA" -e STAGING_URL="$STAGING_URL" \
|
||||
web node --input-type=module <<'NODE'
|
||||
const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
|
||||
let login;
|
||||
for (let attempt = 0; attempt < 12; attempt += 1) {
|
||||
try {
|
||||
login = await fetch(`${process.env.STAGING_URL}/login`);
|
||||
if (login.ok) break;
|
||||
} catch {}
|
||||
await delay(5_000);
|
||||
}
|
||||
if (!login?.ok) process.exit(1);
|
||||
const account = await fetch(`${process.env.STAGING_URL}/api/account`);
|
||||
if (account.status !== 401) process.exit(1);
|
||||
const publicHealth = await fetch(`${process.env.STAGING_URL}/api/health`);
|
||||
const publicBody = await publicHealth.json();
|
||||
if (!publicHealth.ok || publicBody.deployment?.gitCommit !== process.env.EXPECTED_SHA) {
|
||||
process.exit(1);
|
||||
}
|
||||
const privateHealth = await fetch("http://api:5200/api/health");
|
||||
const privateBody = await privateHealth.json();
|
||||
if (!privateHealth.ok || privateBody.status !== "ok" || privateBody.swisseph_available !== true) {
|
||||
process.exit(1);
|
||||
}
|
||||
NODE
|
||||
|
||||
revision_file="$state_directory/deployed-revision.tmp.$$"
|
||||
printf '%s\n' "$DEPLOY_SHA" >"$revision_file"
|
||||
chmod 600 "$revision_file"
|
||||
mv -f "$revision_file" "$state_directory/deployed-revision"
|
||||
trap - ERR
|
||||
|
||||
printf 'previous_sha=%s\nprevious_api_image=%s\nprevious_api_id=%s\n' \
|
||||
"$current_sha" "${previous_api_image:-not-deployed}" "${previous_api_id:-not-deployed}"
|
||||
printf 'previous_web_image=%s\nprevious_web_id=%s\nverified_sha=%s\n' \
|
||||
"${previous_web_image:-not-deployed}" "${previous_web_id:-not-deployed}" "$DEPLOY_SHA"
|
||||
Executable
+79
@@ -0,0 +1,79 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
set +x
|
||||
|
||||
required=(
|
||||
INCOMING_PATH DEPLOY_PATH WEB_IMAGE DEPLOY_SHA EXPECTED_PREVIOUS_SHA
|
||||
DOCKER_CONFIG
|
||||
)
|
||||
for key in "${required[@]}"; do
|
||||
if [ -z "${!key:-}" ]; then
|
||||
echo "required staging migration input is missing: $key" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
[[ "$DEPLOY_SHA" =~ ^[0-9a-f]{40}$ ]] || {
|
||||
echo "unsafe staging migration revision" >&2
|
||||
exit 1
|
||||
}
|
||||
[[ "$WEB_IMAGE" =~ ^ghcr\.io/jesse-ux/jyotisha-web@sha256:[0-9a-f]{64}$ ]] || {
|
||||
echo "unsafe staging migration image" >&2
|
||||
exit 1
|
||||
}
|
||||
case "$INCOMING_PATH" in
|
||||
"$DEPLOY_PATH"/.incoming/*) ;;
|
||||
*) echo "unsafe incoming staging path" >&2; exit 1 ;;
|
||||
esac
|
||||
|
||||
state_directory="$DEPLOY_PATH/.state"
|
||||
install -d -m 700 "$state_directory"
|
||||
exec 9>"$state_directory/mutation.lock"
|
||||
flock -n 9 || {
|
||||
echo "another staging mutation holds the host lock" >&2
|
||||
exit 75
|
||||
}
|
||||
|
||||
current_sha="not-deployed"
|
||||
if [ -f "$state_directory/deployed-revision" ]; then
|
||||
current_sha="$(<"$state_directory/deployed-revision")"
|
||||
else
|
||||
existing_web="$(docker ps -aq \
|
||||
--filter 'label=com.docker.compose.project=jyotisha-staging' \
|
||||
--filter 'label=com.docker.compose.service=web' | head -n 1)"
|
||||
if [ -n "$existing_web" ]; then
|
||||
discovered_sha="$(docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' \
|
||||
"$existing_web" | sed -n 's/^GITHUB_SHA=//p' | head -n 1)"
|
||||
if [ -n "$discovered_sha" ]; then current_sha="$discovered_sha"; fi
|
||||
fi
|
||||
fi
|
||||
if [ "$current_sha" != "not-deployed" ] && [[ ! "$current_sha" =~ ^[0-9a-f]{40}$ ]]; then
|
||||
echo "invalid deployed staging revision state" >&2
|
||||
exit 1
|
||||
fi
|
||||
[ "$current_sha" = "$EXPECTED_PREVIOUS_SHA" ] || {
|
||||
echo "staging revision changed while this migration was waiting" >&2
|
||||
exit 1
|
||||
}
|
||||
[ "$current_sha" = "not-deployed" ] ||
|
||||
[ "$current_sha" = "$DEPLOY_SHA" ] ||
|
||||
[ "${FORWARD_REVISION_VERIFIED:-false}" = "true" ] || {
|
||||
echo "forward staging revision was not verified" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
bash "$INCOMING_PATH/deploy/sync-staging-tree.sh" \
|
||||
"$INCOMING_PATH" "$DEPLOY_PATH"
|
||||
|
||||
cd "$DEPLOY_PATH"
|
||||
bash deploy/validate-staging-env.sh \
|
||||
.env.staging staging.jyotisha.chat deploy/Caddyfile.staging
|
||||
bash deploy/validate-staging-database-env.sh .env.staging.database
|
||||
|
||||
export DATABASE_ENV_FILE='../.env.staging.database'
|
||||
compose=(docker compose -p jyotisha-staging -f deploy/docker-compose.postgres.yml)
|
||||
docker pull "$WEB_IMAGE"
|
||||
"${compose[@]}" up -d --wait postgres
|
||||
"${compose[@]}" --profile migration run --rm migrator
|
||||
"${compose[@]}" exec -T postgres psql -U postgres -d jyotisha -Atc \
|
||||
'select filename from migration.schema_migrations order by filename'
|
||||
Executable
+17
@@ -0,0 +1,17 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
if [ "$#" -ne 2 ] || [ ! -d "$1" ] || [ ! -d "$2" ]; then
|
||||
echo "usage: sync-staging-tree.sh SOURCE_DIRECTORY DESTINATION_DIRECTORY" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
rsync -az --delete \
|
||||
--exclude='/.git/' \
|
||||
--exclude='/.env*' \
|
||||
--exclude='/backups/' \
|
||||
--exclude='/.state/' \
|
||||
--exclude='/.incoming/' \
|
||||
--exclude='/frontend/node_modules/' \
|
||||
--exclude='/frontend/.next/' \
|
||||
"$1/" "$2/"
|
||||
Reference in New Issue
Block a user