ops: prepare self-hosted production migration

2026-08-08 21:12:55 -04:00
parent 61673d3a8b
commit 9b07e12b0c
13 changed files with 1342 additions and 404 deletions
@@ -0,0 +1,19 @@
+{$SITE_ADDRESS:https://jyotisha.chat} {
+    encode zstd gzip
+
+    @adminPaths path /admin /admin/* /api/admin/*
+    respond @adminPaths "Not found" 404
+
+    reverse_proxy web:3000
+}
+
+https://admin.jyotisha.chat {
+    encode zstd gzip
+    @root path /
+    redir @root /admin 308
+    reverse_proxy web:3000
+}
+
+https://www.jyotisha.chat {
+    redir https://jyotisha.chat{uri} 308
+}
@@ -1,77 +1,78 @@
 # Production deployment and maintenance

-This file is the operational source of truth for the current Jyotisha demo deployment.
+This file is the operational source of truth for Jyotisha deployment. The production migration is governed by `docs/operations/production-server-migration-2026-08.md`; do not change DNS or retire the old environment outside that runbook.

-## Current production
+## Production migration state

 | Item | Value |
 | --- | --- |
 | Public domain | `https://jyotisha.chat` |
 | DNS | Spaceship nameservers (`launch1.spaceship.net`, `launch2.spaceship.net`) |
-| Server | Hong Kong VPS, Ubuntu 22.04 x86_64 |
-| Public host | `103.117.123.53` |
-| SSH | port `22000`, public-key authentication only |
-| Capacity | 1 vCPU / 2 GB RAM / 40 GB disk / 5 Mbps |
-| App directory | `/opt/jyotisha-app` |
-| Environment file | `/opt/jyotisha-app/.env.production` (`0600`) |
+| Current public host | Old VPS; keep as a rollback asset until reconciliation completes |
+| Target host | `118.194.235.34`, Ubuntu x86_64 |
+| Target SSH | dedicated `deploy` user, confirmed variable port, public-key authentication only |
+| Target capacity | 2 vCPU / 4 GB RAM; no application builds on host |
+| Target app directory | `/opt/jyotisha-production` |
+| Target environment files | `.env.production` and `.env.production.database` (`0600`) |
 | Primary source repository | `https://git.copse.top/root/Jyotisha.git` |
 | GitHub upstream/mirror | `https://github.com/jesse-ux/Jyotisha.git` |
-| Supabase project | `vtvnfqmonbfuxmqkqdlc` |
+| Migration source | Supabase project `vtvnfqmonbfuxmqkqdlc` + Supabase Auth |
+| Migration target | private PostgreSQL 17 + Better Auth |

-This machine is suitable for a client demo and low concurrency. Supabase and the model provider stay managed externally; do not self-host them on this VPS.
+This migration changes both infrastructure and persistence. It is a controlled Supabase-to-private-PostgreSQL ETL, not a database-volume copy or full-dump restore. Keep the old VPS for 7–14 days and Supabase for 14–30 days after cutover.

 ## Architecture

 ```text
-Spaceship DNS
-  -> Caddy :80/:443
-     -> web:3000 (Next.js + Mastra, Docker-private)
-        -> api:5200 (Python Jyotish API, Docker-private)
-           -> Swiss Ephemeris / local engine
-           -> VedAstro gateway with local fallback
-     -> Supabase Cloud
-     -> external OpenAI-compatible model API
+Spaceship DNS -> Caddy :80/:443 -> web:3000 -> api:5200
+                                  |           -> local astrology engines
+                                  -> private PostgreSQL 17 + Better Auth
+                                  -> external model and mail providers
 ```

 Only Caddy publishes host ports. Ports `3000` and `5200` must remain private.

 ## DNS and Supabase Auth

-Spaceship resource records:
+Final Spaceship resource records (apply only during the approved cutover window):

 ```text
-A      @      103.117.123.53
+A      @      118.194.235.34
+A      admin  118.194.235.34
 CNAME  www    jyotisha.chat
 ```

-Supabase Authentication URL Configuration:
-
-```text
-Site URL:      https://jyotisha.chat
-Redirect URLs: https://jyotisha.chat/**
-               https://www.jyotisha.chat/**
-```
-
-Before changing Caddy to the domain, verify the authoritative DNS result:
+After cutover, verify both authoritative nameservers:

 ```bash
 dig +short @launch1.spaceship.net A jyotisha.chat
+dig +short @launch2.spaceship.net A admin.jyotisha.chat
 ```

-It must return `103.117.123.53`. Caddy provisions and renews HTTPS automatically after DNS resolves.
+Both user and admin hosts must return `118.194.235.34`. Caddy provisions and renews HTTPS automatically after DNS resolves.

 ## Production environment

-`.env.production` combines the backend and frontend server variables. Required groups:
+`.env.production` contains runtime-only application settings. Database bootstrap, migration, and backup credentials belong only in `.env.production.database`.

 ```dotenv
 SITE_ADDRESS=https://jyotisha.chat
+APP_ENV_FILE=../.env.production
+CADDYFILE_PATH=./Caddyfile.production.selfhosted
 JYOTISH_API_BASE=http://api:5200
 GEOAPIFY_API_KEY=<server-side Geoapify geocoding key>

-NEXT_PUBLIC_SUPABASE_URL=...
-NEXT_PUBLIC_SUPABASE_ANON_KEY=...
-SUPABASE_SERVICE_ROLE_KEY=...
+AUTH_PROVIDER=self-hosted
+SELF_HOSTED_IDENTITY_ENABLED=true
+AUTH_USER_ORIGIN=https://jyotisha.chat
+ADMIN_USER_ORIGIN=https://admin.jyotisha.chat
+IDENTITY_DATABASE_URL=postgresql://identity_runtime:<encoded-secret>@postgres:5432/jyotisha
+APP_DATABASE_URL=postgresql://app_runtime:<encoded-secret>@postgres:5432/jyotisha
+SERVICE_DATABASE_URL=postgresql://service_runtime:<encoded-secret>@postgres:5432/jyotisha
+ADMIN_DATABASE_URL=postgresql://admin_runtime:<encoded-secret>@postgres:5432/jyotisha
+BETTER_AUTH_USER_SECRET=<production-only-secret>
+RESEND_API_KEY=<production-only-secret>
+RESEND_FROM_EMAIL=<verified-sender>
 ADMIN_EMAILS=...

 # Required to save/read database-backed 易支付 settings. Base64 decoding must
@@ -87,10 +88,7 @@ RECTIFICATION_PRICE_CREDITS=3

 # Required to encrypt/decrypt model-provider API keys stored in the admin database.
 # Base64 decoding must produce exactly 32 random bytes; do not reuse other keys.
-# Staging migration/deploy creates this once when absent and removes legacy model API-key settings.
 MODEL_PROVIDER_CONFIG_ENCRYPTION_KEY=<independent-base64-encoded-32-byte-key>
-# OpenAI-compatible origins must be explicitly server-allowlisted.
-# MODEL_PROVIDER_BASE_URL_ALLOWLIST=https://api.deepseek.com

 # The admin database model catalog is the only runtime model/provider source.
 # Provider API keys are entered in the admin UI and are never read from provider env vars.
@@ -105,47 +103,37 @@ VEDASTRO_TIMEOUT_SECONDS=20
 VEDASTRO_API_KEY=<server-secret>
 ```

-Never commit `.env.production`, `SUPABASE_SERVICE_ROLE_KEY`, model keys, user JWTs, SSH private keys or passwords. `NEXT_PUBLIC_SUPABASE_ANON_KEY` is intentionally public; authorization is enforced by Supabase RLS and server-side checks.
+Never commit either production env file, model/payment keys, user tokens, SSH private keys, database URLs, or passwords. Do not retain Supabase runtime selectors in the target env; Supabase is migration source/rollback storage only.

-After changing VedAstro variables, restart the API and verify the configuration without printing credentials:
-```bash
-docker compose --env-file .env.production -f deploy/docker-compose.server.yml up -d --build api
-docker compose --env-file .env.production -f deploy/docker-compose.server.yml exec api python3 scripts/diagnose_vedastro_mode.py
-```
-The report must show `mode: official_extended` and `network_enabled: true`. A missing raw response remains an upstream response boundary, not a successful external verification.
+After changing runtime variables, use the exact-SHA Gitea workflow to recreate services; do not rebuild or rsync a local tree on the host.

 ## Connect and inspect

 ```bash
-ssh -p 22000 root@103.117.123.53
-cd /opt/jyotisha-app
-COMPOSE='docker compose --env-file .env.production -f deploy/docker-compose.server.yml'
+ssh -p <confirmed-port> deploy@118.194.235.34
+cd /opt/jyotisha-production
+COMPOSE='docker compose -p jyotisha-production --env-file .env.production -f deploy/docker-compose.server.yml -f deploy/docker-compose.postgres.yml -f deploy/docker-compose.production.yml'
 $COMPOSE ps
-$COMPOSE logs --tail=100 api web caddy
+$COMPOSE logs --tail=100 postgres api web caddy
 free -h
 docker stats --no-stream
 ```

-The server has a persistent 2 GB `/swapfile`. UFW permits only SSH `22000/tcp`, HTTP `80/tcp`, HTTPS `443/tcp`, and the pre-existing WireGuard `51820/udp` rule.
+UFW permits only the confirmed SSH port, HTTP, and HTTPS. PostgreSQL, Web, API, and the Docker API remain private.

-## Manual deployment with GitHub Actions
+## Manual production deployment with Gitea Actions

-Production pushes and pull requests do not start GitHub Actions automatically. Run the required validation workflows from the Actions page, then manually start `.github/workflows/deploy-production.yml` for the tested branch. The deployment workflow syncs that revision with `rsync`, preserves `/opt/jyotisha-app/.env.production`, rebuilds both Docker services, and verifies the public login route, logged-out account response, and private Python health endpoint.
+GitHub production deployment is retired. Production changes are released only by manually dispatching `.gitea/workflows/deploy-production.yml`. The workflow requires an exact SHA shared by `main` and `staging`, an exact-SHA staging push gate and image manifest, the manual release gate, and matching public staging health. It deploys immutable registry digests and never builds application images on the production host.

-For the reviewed conversational rectification and global birthplace schema set,
-run `.github/workflows/apply-production-rectification-migrations.yml` with
-`operation=check` first. If the ledger and checksums are clean, rerun the same
-current-`main` revision with `operation=apply`. The workflow only accepts the
-four allowlisted forward migrations, applies each migration and its ledger row
-in one transaction, and refuses stale revisions or checksum drift.
+The `internal` verification mode is for pre-DNS checks on the new host. The `public` mode is for the same SHA after authoritative DNS and Caddy TLS converge. Neither mode imports data, applies pending migrations, or changes DNS.

-Required GitHub Actions secret:
+Required Gitea Actions secret:

 ```text
-PRODUCTION_SSH_PRIVATE_KEY = dedicated production deploy private key
+PRODUCTION_SSH_PRIVATE_KEY = one-line base64 of the dedicated deploy private-key file
 ```

-The workflow pins the VPS Ed25519 host key and serializes deployments with the `production` concurrency group.
+The independently verified host key is stored in `PRODUCTION_KNOWN_HOSTS`. See the production migration runbook for all variables, data gates, and rollback boundaries.

 ## Staging deployment

@@ -178,7 +166,7 @@ SITE_ADDRESS=https://staging.jyotisha.chat

 模型供应商的 `base_url` 不再依赖域名白名单，任意公网 HTTPS origin 均可由管理员配置；部署环境不需要 `MODEL_PROVIDER_BASE_URL_ALLOWLIST`。服务端仍强制 HTTPS、禁止凭据、localhost/内网/保留地址，并在 DNS 解析、请求地址 pinning 和重定向处理上执行 SSRF 防护。

-Staging is fully self-hosted: set `AUTH_PROVIDER=self-hosted` and `SELF_HOSTED_IDENTITY_ENABLED=true`. Add the four role-specific server-only database URLs, the exact `AUTH_USER_ORIGIN=https://staging.jyotisha.chat` and `ADMIN_USER_ORIGIN=https://admin.staging.jyotisha.chat`, the single `BETTER_AUTH_USER_SECRET`, and staging-only Resend settings listed in `deploy/.env.staging.identity.example`. Both hosts run the same application and Better Auth service, but cookies remain host-only; the admin host `/` redirects to `/admin`, and unauthenticated admin requests continue to `/login` on that host. Better Auth trusts only those two origins, while unknown identity hosts fail closed. Persisted `identity.users.role=admin` is the only self-hosted backend role, while `viewer` and ordinary users are denied. Browser code uses same-origin APIs; it receives neither database credentials nor Supabase keys. Production remains on Supabase and is not changed by the staging workflow. See `docs/operations/self-hosted-identity.md` for validation and rollback commands.
+Staging is fully self-hosted: set `AUTH_PROVIDER=self-hosted` and `SELF_HOSTED_IDENTITY_ENABLED=true`. Add the four role-specific server-only database URLs, the exact `AUTH_USER_ORIGIN=https://staging.jyotisha.chat` and `ADMIN_USER_ORIGIN=https://admin.staging.jyotisha.chat`, the single `BETTER_AUTH_USER_SECRET`, and staging-only Resend settings listed in `deploy/.env.staging.identity.example`. Both hosts run the same application and Better Auth service, but cookies remain host-only; the admin host `/` redirects to `/admin`, and unauthenticated admin requests continue to `/login` on that host. Better Auth trusts only those two origins, while unknown identity hosts fail closed. Persisted `identity.users.role=admin` is the only self-hosted backend role, while `viewer` and ordinary users are denied. Browser code uses same-origin APIs; it receives neither database credentials nor Supabase keys. Production uses the same architecture only after the reviewed migration and cutover. See `docs/operations/self-hosted-identity.md` for identity validation.

 After source sync and before `up`, the workflow validates `.env.staging` mode/selectors, explicitly pins the three staging selectors against ambient shell overrides, and runs `docker compose --env-file .env.staging -f deploy/docker-compose.server.yml config --quiet`. For later manual inspections, run the same checks only after the tracked deployment files exist on the server. Do not use a manual gate run from `main` as the first publishing path: publishing requires a successful push to `staging`, while manual `Deploy staging` requires a successful gate run for the exact SHA.

@@ -321,23 +309,7 @@ This disposable staging procedure does not authorize a production migration, pro

 ## Manual deployment fallback

-If GitHub Actions is unavailable, deploy the tracked tree without copying local secrets:
-
-```bash
-cd /Users/jesse/Downloads/Copse/astrology/yinduzhanxing
-git status --short --branch
-rsync -az --delete \
-  --exclude='.git/' \
-  --exclude='.env.production' \
-  --exclude='frontend/node_modules/' \
-  --exclude='frontend/.next/' \
-  -e 'ssh -p 22000' \
-  ./ root@103.117.123.53:/opt/jyotisha-app/
-ssh -p 22000 root@103.117.123.53 \
-  'cd /opt/jyotisha-app && docker compose --env-file .env.production -f deploy/docker-compose.server.yml up -d --build --remove-orphans'
-```
-
-The excluded `.env.production` remains only on the VPS.
+There is no unreviewed rsync/build fallback for the new production. If Gitea Actions or the immutable artifact is unavailable, stop the release and restore the control plane; do not substitute a mutable image tag or copy a local working tree to production.

 ## Verification

@@ -349,29 +321,13 @@ curl -fsS -o /dev/null -w '%{http_code}\n' https://jyotisha.chat/api/account
 The second command should return `401` while logged out. Verify the private Python API from inside the web container:

 ```bash
-ssh -p 22000 root@103.117.123.53 \
-  'cd /opt/jyotisha-app && docker compose --env-file .env.production -f deploy/docker-compose.server.yml exec -T web node -e "fetch(\"http://api:5200/api/health\").then(async r=>{console.log(r.status); console.log(await r.text())})"'
+ssh -p <confirmed-port> deploy@118.194.235.34 \
+  'cd /opt/jyotisha-production && docker compose -p jyotisha-production --env-file .env.production -f deploy/docker-compose.server.yml -f deploy/docker-compose.postgres.yml -f deploy/docker-compose.production.yml exec -T web node -e "fetch(\"http://api:5200/api/health\").then(async r=>{console.log(r.status); console.log(await r.text())})"'
 ```

-Expected: HTTP `200`, `"status": "ok"`, and `"swisseph_available": true`. Public access to `103.117.123.53:5200` must fail.
+Expected: HTTP `200`, `"status": "ok"`, and `"swisseph_available": true`. Public access to `118.194.235.34:5200` and `:5432` must fail. The deployment workflow also executes `SELECT 1` through all four runtime database roles.

-Before deploying application code that depends on any new Supabase migration (columns, tables, grants, policies, or RPCs), run `cd frontend && npx supabase db push --linked`; the GitHub deployment workflow does not apply database migrations. Multi-model chat specifically requires `20260717010000_chat_session_model.sql` before the new web image is deployed. Then manually verify: OTP login, onboarding/profile persistence, per-session `model_id` persistence, code redemption, admin code generation, authenticated `/api/models` returns only sanitized public metadata, invalid model IDs are rejected before charging, each configured model can answer, the 2.5-second free undo window, streaming response, one-credit charge, refund before the first output chunk, and charged stop with partial output preserved after streaming starts.
-
-For the July 2026 new-user profile save fix, either run the manual GitHub Action
-`Apply Supabase profile migrations` after adding `SUPABASE_DB_URL` or `DATABASE_URL`
-to `/opt/jyotisha-app/.env.production`, or execute these five SQL migrations in
-the Supabase SQL Editor with a project member account:
-
- `20260718010000_recover_missing_profile_rows.sql`
- `20260718020000_profiles_service_role_upsert_grants.sql`
- `20260718050000_profiles_service_role_upsert_grants.sql`
- `20260718070000_profiles_service_role_upsert_id.sql`
- `20260718080000_profiles_service_role_account_upsert_selects.sql`
-
-Do not treat a green app deployment as proof this database step ran. If the SQL
-Editor shows `You do not have access to this project`, use the correct Supabase
-organization account or invite the current GitHub user to project
-`vtvnfqmonbfuxmqkqdlc` before retrying.
+Do not treat a green app deployment as proof of database migration or data reconciliation. The production deployment refuses pending target migrations but does not apply them. Follow the schema-first ETL and verification gates in the production migration runbook. Supabase remains the read-only rollback source until the retention and reconciliation window closes.

 ## Agentic birth-time rectification

@@ -392,16 +348,16 @@ evidence.

 ```bash
 # Restart without rebuilding
-docker compose --env-file .env.production -f deploy/docker-compose.server.yml up -d
+docker compose -p jyotisha-production --env-file .env.production \
+  -f deploy/docker-compose.server.yml -f deploy/docker-compose.postgres.yml \
+  -f deploy/docker-compose.production.yml up -d

-# Rebuild only the web container
-docker compose --env-file .env.production -f deploy/docker-compose.server.yml up -d --build web caddy
-
-# Rebuild only the Python API
-docker compose --env-file .env.production -f deploy/docker-compose.server.yml up -d --build api
+# Pull/deploy application images only through the exact-SHA Gitea workflow.

 # Follow logs
-docker compose --env-file .env.production -f deploy/docker-compose.server.yml logs -f --tail=100 api web caddy
+docker compose -p jyotisha-production --env-file .env.production \
+  -f deploy/docker-compose.server.yml -f deploy/docker-compose.postgres.yml \
+  -f deploy/docker-compose.production.yml logs -f --tail=100 postgres api web caddy
 ```

 ## Optional Railway deployment
@@ -0,0 +1,8 @@
+services:
+  web:
+    networks:
+      - default
+      - app
+
+networks:
+  app:
@@ -0,0 +1,314 @@
+#!/usr/bin/env bash
+set -euo pipefail
+set +x
+
+required=(
+  INCOMING_PATH DEPLOY_PATH API_IMAGE WEB_IMAGE DEPLOY_SHA
+  EXPECTED_PREVIOUS_SHA ALLOW_ROLLBACK DOCKER_CONFIG PRODUCTION_URL
+  PRODUCTION_ADMIN_URL VERIFICATION_MODE
+)
+case "${DOCKER_BIN:-docker}" in
+  docker) docker_command=(docker) ;;
+  "sudo -n docker") docker_command=(sudo -n docker --config "$DOCKER_CONFIG") ;;
+  *) echo "unsafe production Docker command" >&2; exit 1 ;;
+esac
+for key in "${required[@]}"; do
+  if [ -z "${!key:-}" ]; then
+    echo "required production deployment input is missing: $key" >&2
+    exit 1
+  fi
+done
+
+sha_pattern='^[0-9a-f]{40}$'
+digest_pattern='^[a-z0-9]([a-z0-9.-]*[a-z0-9])?(:[1-9][0-9]{0,4})?(/[a-z0-9]+([._-][a-z0-9]+)*)+@sha256:[0-9a-f]{64}$'
+image_id_pattern='^sha256:[0-9a-f]{64}$'
+if [[ ! "$DEPLOY_SHA" =~ $sha_pattern ]] ||
+  [[ ! "$API_IMAGE" =~ $digest_pattern ]] ||
+  [[ ! "$WEB_IMAGE" =~ $digest_pattern ]]; then
+  echo "unsafe production image identity" >&2
+  exit 1
+fi
+api_repository="${API_IMAGE%@sha256:*}"
+web_repository="${WEB_IMAGE%@sha256:*}"
+if [ "$ALLOW_ROLLBACK" != "true" ] && [ "$ALLOW_ROLLBACK" != "false" ]; then
+  echo "invalid rollback authorization" >&2
+  exit 1
+fi
+if [ "$VERIFICATION_MODE" != "internal" ] && [ "$VERIFICATION_MODE" != "public" ]; then
+  echo "invalid production verification mode" >&2
+  exit 1
+fi
+case "$INCOMING_PATH" in
+  /tmp/jyotisha-production.*) ;;
+  *) echo "unsafe incoming production path" >&2; exit 1 ;;
+esac
+
+state_directory="$DEPLOY_PATH/.state"
+install -d -m 700 "$state_directory"
+exec 9>"$state_directory/mutation.lock"
+flock -n 9 || {
+  echo "another production mutation holds the host lock" >&2
+  exit 75
+}
+
+current_sha="not-deployed"
+if [ -f "$state_directory/deployed-revision" ]; then
+  current_sha="$(<"$state_directory/deployed-revision")"
+else
+  existing_web="$("${docker_command[@]}" ps -aq \
+    --filter 'label=com.docker.compose.project=jyotisha-production' \
+    --filter 'label=com.docker.compose.service=web' | head -n 1)"
+  if [ -n "$existing_web" ]; then
+    discovered_sha="$("${docker_command[@]}" inspect --format '{{range .Config.Env}}{{println .}}{{end}}' \
+      "$existing_web" | sed -n 's/^GITHUB_SHA=//p' | head -n 1)"
+    if [ -n "$discovered_sha" ]; then current_sha="$discovered_sha"; fi
+  fi
+fi
+if [ "$current_sha" != "not-deployed" ] && [[ ! "$current_sha" =~ $sha_pattern ]]; then
+  echo "invalid deployed production revision state" >&2
+  exit 1
+fi
+if [ "$current_sha" != "$EXPECTED_PREVIOUS_SHA" ]; then
+  echo "production revision changed while this deployment was waiting" >&2
+  exit 1
+fi
+if [ "$ALLOW_ROLLBACK" = "false" ] &&
+  [ "$current_sha" != "not-deployed" ] &&
+  [ "$current_sha" != "$DEPLOY_SHA" ] &&
+  [ "${FORWARD_REVISION_VERIFIED:-false}" != "true" ]; then
+  echo "forward production revision was not verified" >&2
+  exit 1
+fi
+
+container_id() {
+  "${docker_command[@]}" ps -aq \
+    --filter 'label=com.docker.compose.project=jyotisha-production' \
+    --filter "label=com.docker.compose.service=$1" | head -n 1
+}
+
+repo_digest_for_container() {
+  local service="$1"
+  local repository="$2"
+  local id image_id
+  id="$(container_id "$service")"
+  [ -n "$id" ] || return 0
+  image_id="$("${docker_command[@]}" inspect --format '{{.Image}}' "$id")"
+  "${docker_command[@]}" image inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' "$image_id" |
+    awk -v prefix="$repository@sha256:" 'index($0, prefix) == 1 { print; exit }'
+}
+
+previous_api_image="$(repo_digest_for_container api "$api_repository")"
+previous_web_image="$(repo_digest_for_container web "$web_repository")"
+previous_api_id=""
+previous_web_id=""
+if [ -n "$(container_id api)" ]; then
+  previous_api_id="$("${docker_command[@]}" inspect --format '{{.Image}}' "$(container_id api)")"
+fi
+if [ -n "$(container_id web)" ]; then
+  previous_web_id="$("${docker_command[@]}" inspect --format '{{.Image}}' "$(container_id web)")"
+fi
+
+rollback_image() {
+  local digest_ref="$1"
+  local image_id="$2"
+  if [[ "$digest_ref" =~ $digest_pattern ]]; then
+    printf '%s' "$digest_ref"
+  elif [[ "$image_id" =~ $image_id_pattern ]]; then
+    printf '%s' "$image_id"
+  fi
+}
+
+previous_api_target="$(rollback_image "$previous_api_image" "$previous_api_id")"
+previous_web_target="$(rollback_image "$previous_web_image" "$previous_web_id")"
+
+bash "$INCOMING_PATH/deploy/sync-production-tree.sh" \
+  "$INCOMING_PATH" "$DEPLOY_PATH"
+
+cd "$DEPLOY_PATH"
+EXPECTED_PRODUCTION_ENV_OWNER_UID="$(stat -c '%u' "$DEPLOY_PATH" 2>/dev/null || stat -f '%u' "$DEPLOY_PATH")"
+[[ "$EXPECTED_PRODUCTION_ENV_OWNER_UID" =~ ^[0-9]+$ ]] || {
+  echo "production deployment owner is invalid" >&2
+  exit 1
+}
+export EXPECTED_PRODUCTION_ENV_OWNER_UID
+bash deploy/validate-production-env.sh \
+  .env.production
+bash deploy/validate-production-database-env.sh .env.production.database
+
+compose=(
+  "${docker_command[@]}" compose -p jyotisha-production --env-file .env.production
+  -f deploy/docker-compose.server.yml -f deploy/docker-compose.postgres.yml
+  -f deploy/docker-compose.production.yml
+)
+export APP_ENV_FILE='../.env.production'
+export DATABASE_ENV_FILE='../.env.production.database'
+export CADDYFILE_PATH='./Caddyfile.production.selfhosted'
+export SITE_ADDRESS='https://jyotisha.chat'
+export GITHUB_SHA="$DEPLOY_SHA"
+
+"${compose[@]}" config --quiet
+"${compose[@]}" pull api web
+"${compose[@]}" up -d --no-build --pull never --wait postgres
+
+set +e
+"${compose[@]}" --profile migration-check run --rm migration-checker
+check_status=$?
+set -e
+if [ "$check_status" -eq 3 ]; then
+  echo "pending migrations: run Migrate Production Database for $DEPLOY_SHA" >&2
+  exit 3
+fi
+if [ "$check_status" -ne 0 ]; then
+  echo "production migration check failed safely" >&2
+  exit "$check_status"
+fi
+
+switched=false
+rollback() {
+  local status=$?
+  if [ "$switched" = "true" ] &&
+    [ -n "$previous_api_target" ] &&
+    [ -n "$previous_web_target" ] &&
+    [[ "$current_sha" =~ $sha_pattern ]]; then
+    echo "production verification failed; restoring prior application images" >&2
+    rollback_services=(api web)
+    if [ "$VERIFICATION_MODE" = "public" ]; then rollback_services+=(caddy); fi
+    API_IMAGE="$previous_api_target" WEB_IMAGE="$previous_web_target" \
+      GITHUB_SHA="$current_sha" \
+      "${compose[@]}" up -d --no-build --remove-orphans \
+        "${rollback_services[@]}" || true
+  fi
+  exit "$status"
+}
+trap rollback ERR
+
+switched=true
+if [ "$VERIFICATION_MODE" = "public" ]; then
+  "${compose[@]}" up -d --no-build --remove-orphans
+  "${compose[@]}" up -d --no-build --force-recreate --no-deps caddy
+else
+  # Before DNS cutover, do not trigger public certificate issuance for domains
+  # that still resolve to the old production host.
+  "${compose[@]}" up -d --no-build api web
+fi
+
+verify_container_image() {
+  local service="$1"
+  local expected_ref="$2"
+  local id expected_id running_id repo_digests
+  id="$(container_id "$service")"
+  [ -n "$id" ]
+  expected_id="$("${docker_command[@]}" image inspect --format '{{.Id}}' "$expected_ref")"
+  running_id="$("${docker_command[@]}" inspect --format '{{.Image}}' "$id")"
+  [ "$running_id" = "$expected_id" ]
+  repo_digests="$("${docker_command[@]}" image inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' "$expected_id")"
+  grep -Fqx "$expected_ref" <<<"$repo_digests"
+}
+verify_container_image api "$API_IMAGE"
+verify_container_image web "$WEB_IMAGE"
+
+"${compose[@]}" exec -T \
+  -e EXPECTED_SHA="$DEPLOY_SHA" \
+  -e PRODUCTION_URL="$PRODUCTION_URL" \
+  -e PRODUCTION_ADMIN_URL="$PRODUCTION_ADMIN_URL" \
+  -e VERIFICATION_MODE="$VERIFICATION_MODE" \
+  web node --input-type=module <<'NODE'
+import { Pool } from "pg";
+
+const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
+for (const [role, key] of [
+  ["identity", "IDENTITY_DATABASE_URL"],
+  ["app", "APP_DATABASE_URL"],
+  ["service", "SERVICE_DATABASE_URL"],
+  ["admin", "ADMIN_DATABASE_URL"],
+]) {
+  const connectionString = process.env[key];
+  if (!connectionString) {
+    console.error(`database readiness missing for ${role}`);
+    process.exit(1);
+  }
+  const pool = new Pool({ connectionString, max: 1, connectionTimeoutMillis: 5_000 });
+  try {
+    const result = await pool.query("select 1 as ready");
+    if (result.rows[0]?.ready !== 1) throw new Error("unexpected readiness result");
+  } catch (error) {
+    console.error(`database readiness failed for ${role}`, error instanceof Error ? error.name : "query_error");
+    process.exit(1);
+  } finally {
+    await pool.end();
+  }
+}
+
+const internal = process.env.VERIFICATION_MODE === "internal";
+const request = (origin, path, options = {}) => {
+  const target = internal ? `http://127.0.0.1:3000${path}` : `${origin}${path}`;
+  const headers = internal ? { ...options.headers, host: new URL(origin).host } : options.headers;
+  return fetch(target, { ...options, headers });
+};
+let observed = {};
+for (let attempt = 1; attempt <= 12; attempt += 1) {
+  try {
+    const login = await request(process.env.PRODUCTION_URL, "/login");
+    const userAdminPage = await request(process.env.PRODUCTION_URL, "/admin", { redirect: "manual" });
+    const userAdminApi = await request(process.env.PRODUCTION_URL, "/api/admin/session");
+    const adminPage = await request(process.env.PRODUCTION_ADMIN_URL, "/admin", { redirect: "manual" });
+    const adminApi = await request(process.env.PRODUCTION_ADMIN_URL, "/api/admin/session");
+    const account = await request(process.env.PRODUCTION_URL, "/api/account");
+    const publicHealth = await request(process.env.PRODUCTION_URL, "/api/health");
+    const publicBody = await publicHealth.json();
+    const privateHealth = await fetch("http://api:5200/api/health");
+    const privateBody = await privateHealth.json();
+    observed = {
+      attempt,
+      login: login.status,
+      userAdminPage: userAdminPage.status,
+      userAdminApi: userAdminApi.status,
+      adminPage: adminPage.status,
+      adminLocation: adminPage.headers.get("location"),
+      adminApi: adminApi.status,
+      account: account.status,
+      publicHealth: publicHealth.status,
+      publicSha: publicBody.deployment?.gitCommit ?? "missing",
+      privateHealth: privateHealth.status,
+      privateStatus: privateBody.status ?? "missing",
+      swissephAvailable: privateBody.swisseph_available === true,
+      verificationMode: process.env.VERIFICATION_MODE,
+    };
+    if (
+      login.ok
+      && userAdminPage.status === 404
+      && userAdminApi.status === 404
+      && adminPage.status === 307
+      && adminPage.headers.get("location") === "/login"
+      && adminApi.status === 401
+      && account.status === 401
+      && publicHealth.ok
+      && publicBody.deployment?.gitCommit === process.env.EXPECTED_SHA
+      && privateHealth.ok
+      && privateBody.status === "ok"
+      && privateBody.swisseph_available === true
+    ) {
+      process.exit(0);
+    }
+  } catch (error) {
+    observed = {
+      attempt,
+      error: error instanceof Error ? error.name : "verification_error",
+    };
+  }
+  if (attempt < 12) await delay(5_000);
+}
+console.error("production verification predicates did not converge", JSON.stringify(observed));
+process.exit(1);
+NODE
+
+revision_file="$state_directory/deployed-revision.tmp.$$"
+printf '%s\n' "$DEPLOY_SHA" >"$revision_file"
+chmod 600 "$revision_file"
+mv -f "$revision_file" "$state_directory/deployed-revision"
+trap - ERR
+
+printf 'previous_sha=%s\nprevious_api_image=%s\nprevious_api_id=%s\n' \
+  "$current_sha" "${previous_api_image:-not-deployed}" "${previous_api_id:-not-deployed}"
+printf 'previous_web_image=%s\nprevious_web_id=%s\nverified_sha=%s\n' \
+  "${previous_web_image:-not-deployed}" "${previous_web_id:-not-deployed}" "$DEPLOY_SHA"
@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+if [ "$#" -ne 2 ] || [ ! -d "$1" ] || [ ! -d "$2" ]; then
+  echo "usage: sync-production-tree.sh SOURCE_DIRECTORY DESTINATION_DIRECTORY" >&2
+  exit 1
+fi
+
+destination_deploy="$2/deploy"
+if [ -d "$destination_deploy" ]; then
+  docker run --rm --pull never --network none --read-only --user 0:0 \
+    --cap-drop ALL --cap-add CHOWN --security-opt no-new-privileges \
+    -v "$destination_deploy:/destination" postgres:17-alpine \
+    chown -R "$(id -u):$(id -g)" /destination
+  chmod -R u+rwX "$destination_deploy"
+fi
+
+rsync -az --delete --no-owner --no-group \
+  --exclude='/.git/' \
+  --exclude='/.env*' \
+  --exclude='/.docker/' \
+  --exclude='/backups/' \
+  --exclude='/.state/' \
+  --exclude='/.incoming/' \
+  --exclude='/frontend/node_modules/' \
+  --exclude='/frontend/.next/' \
+  "$1/" "$2/"
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+set -euo pipefail
+set +x
+
+ENV_FILE="${1:-.env.production.database}"
+
+if [ ! -e "$ENV_FILE" ]; then
+  echo "production database environment file is missing" >&2
+  exit 1
+fi
+
+if [ -L "$ENV_FILE" ]; then
+  echo "production database environment file must not be a symlink" >&2
+  exit 1
+fi
+
+if [ ! -f "$ENV_FILE" ]; then
+  echo "production database environment path must be a regular file" >&2
+  exit 1
+fi
+
+if MODE="$(stat -c '%a' "$ENV_FILE" 2>/dev/null)"; then
+  :
+else
+  MODE="$(stat -f '%Lp' "$ENV_FILE")"
+fi
+
+if [ "$MODE" != "600" ]; then
+  echo "production database environment file must have mode 0600" >&2
+  exit 1
+fi
+
+if OWNER="$(stat -c '%u' "$ENV_FILE" 2>/dev/null)"; then
+  :
+else
+  OWNER="$(stat -f '%u' "$ENV_FILE")"
+fi
+EXPECTED_OWNER_UID="${EXPECTED_PRODUCTION_ENV_OWNER_UID:-$(id -u)}"
+if [[ ! "$EXPECTED_OWNER_UID" =~ ^[0-9]+$ ]] || [ "$OWNER" != "$EXPECTED_OWNER_UID" ]; then
+  echo "production database environment file has an invalid owner" >&2
+  exit 1
+fi
+
+definition_count() {
+  local key="$1"
+  grep -Ec "^[[:space:]]*(export[[:space:]]+)?${key}([[:space:]]*=|[[:space:]]*$)" "$ENV_FILE" || true
+}
+
+environment_value() {
+  local key="$1"
+  sed -n -E "s/^[[:space:]]*(export[[:space:]]+)?${key}[[:space:]]*=[[:space:]]*(.*)$/\\2/p" "$ENV_FILE"
+}
+
+is_safe_literal() {
+  local value="$1"
+  local inner
+
+  # Required values are literal single-line values: use an unquoted token or
+  # matching non-empty quotes. Dotenv interpolation, comments, and malformed
+  # quoting are rejected rather than evaluated, so generate secrets without $.
+  if [ -z "$value" ] || [[ "$value" == *'$'* ]]; then
+    return 1
+  fi
+
+  case "$value" in
+    \"*\")
+      inner="${value:1}"
+      inner="${inner%?}"
+      [ -n "$inner" ] && [[ "$inner" != *'"'* ]]
+      ;;
+    \'*\')
+      inner="${value:1}"
+      inner="${inner%?}"
+      [ -n "$inner" ] && [[ "$inner" != *"'"* ]]
+      ;;
+    *\"*|*\'*)
+      return 1
+      ;;
+    *[[:space:]]*|*\#*)
+      return 1
+      ;;
+  esac
+}
+
+require_once_non_empty() {
+  local key="$1"
+  local count
+  local value
+  count="$(definition_count "$key")"
+  value="$(environment_value "$key")"
+  if [ "$count" -ne 1 ] || ! is_safe_literal "$value"; then
+    echo "required production database literal is missing, duplicated, or ambiguous: $key" >&2
+    exit 1
+  fi
+}
+
+required=(
+  POSTGRES_DB POSTGRES_USER POSTGRES_PASSWORD
+  SCHEMA_OWNER_PASSWORD IDENTITY_RUNTIME_PASSWORD APP_RUNTIME_PASSWORD
+  SERVICE_RUNTIME_PASSWORD ADMIN_RUNTIME_PASSWORD MIGRATION_RUNNER_PASSWORD
+  BACKUP_READER_PASSWORD
+  PRODUCTION_BACKUP_ENCRYPTION_KEY SCHEMA_DATABASE_URL
+)
+for key in "${required[@]}"; do
+  require_once_non_empty "$key"
+done
+
+if [ "$(environment_value POSTGRES_DB)" != "jyotisha" ]; then
+  echo "invalid production database selector: POSTGRES_DB" >&2
+  exit 1
+fi
+
+if [ "$(environment_value POSTGRES_USER)" != "postgres" ]; then
+  echo "invalid production database selector: POSTGRES_USER" >&2
+  exit 1
+fi
+
+if ! [[ "$(environment_value SCHEMA_DATABASE_URL)" =~ ^postgresql://schema_owner:([A-Za-z0-9._~-]|%[0-9A-Fa-f]{2})+@postgres:5432/jyotisha$ ]]; then
+  echo "invalid production database selector: SCHEMA_DATABASE_URL" >&2
+  exit 1
+fi
+
+echo "production database environment validated"
@@ -0,0 +1,158 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ENV_FILE="${1:-.env.production}"
+
+if [ ! -f "$ENV_FILE" ]; then
+  echo "production environment file is missing: $ENV_FILE" >&2
+  exit 1
+fi
+
+if [ -L "$ENV_FILE" ]; then
+  echo "production environment file must not be a symlink" >&2
+  exit 1
+fi
+
+if MODE="$(stat -c '%a' "$ENV_FILE" 2>/dev/null)"; then
+  :
+else
+  MODE="$(stat -f '%Lp' "$ENV_FILE")"
+fi
+
+if [ "$MODE" != "600" ]; then
+  echo "production environment file must have mode 0600" >&2
+  exit 1
+fi
+
+if OWNER="$(stat -c '%u' "$ENV_FILE" 2>/dev/null)"; then
+  :
+else
+  OWNER="$(stat -f '%u' "$ENV_FILE")"
+fi
+EXPECTED_OWNER_UID="${EXPECTED_PRODUCTION_ENV_OWNER_UID:-$(id -u)}"
+if [[ ! "$EXPECTED_OWNER_UID" =~ ^[0-9]+$ ]] || [ "$OWNER" != "$EXPECTED_OWNER_UID" ]; then
+  echo "production environment file has an invalid owner" >&2
+  exit 1
+fi
+
+require_selector() {
+  local key="$1"
+  local expected="$2"
+  local count
+  local definition_pattern
+
+  definition_pattern="^[[:space:]]*(export[[:space:]]+)?${key}([[:space:]]*=|[[:space:]]*$)"
+  count="$(grep -Ec "$definition_pattern" "$ENV_FILE" || true)"
+  if [ "$count" -ne 1 ] || ! grep -Fqx "${key}=${expected}" "$ENV_FILE"; then
+    echo "invalid production selector: $key" >&2
+    exit 1
+  fi
+}
+
+require_selector APP_ENV_FILE ../.env.production
+require_selector CADDYFILE_PATH ./Caddyfile.production.selfhosted
+require_selector SITE_ADDRESS https://jyotisha.chat
+require_selector AUTH_PROVIDER self-hosted
+require_selector SELF_HOSTED_IDENTITY_ENABLED true
+require_selector AUTH_USER_ORIGIN https://jyotisha.chat
+require_selector ADMIN_USER_ORIGIN https://admin.jyotisha.chat
+
+require_literal() {
+  local key="$1"
+  local minimum_length="$2"
+  local count value
+  count="$(grep -Ec "^${key}=" "$ENV_FILE" || true)"
+  if [ "$count" -ne 1 ]; then
+    echo "invalid production identity setting: $key" >&2
+    exit 1
+  fi
+  value="$(grep -E "^${key}=" "$ENV_FILE")"
+  value="${value#*=}"
+  if [ "${#value}" -lt "$minimum_length" ] ||
+    [[ "$value" == *'$'* || "$value" == *'"'* || "$value" == *"'"* ]]; then
+    echo "invalid production identity setting: $key" >&2
+    exit 1
+  fi
+  LITERAL_VALUE="$value"
+}
+
+require_literal IDENTITY_DATABASE_URL 50
+identity_database_url="$LITERAL_VALUE"
+if ! [[ "$identity_database_url" =~ ^postgresql://identity_runtime:([A-Za-z0-9._~-]|%[0-9A-Fa-f]{2})+@postgres:5432/jyotisha$ ]]; then
+  echo "invalid production identity setting: IDENTITY_DATABASE_URL" >&2
+  exit 1
+fi
+
+require_literal APP_DATABASE_URL 45
+app_database_url="$LITERAL_VALUE"
+if ! [[ "$app_database_url" =~ ^postgresql://app_runtime:([A-Za-z0-9._~-]|%[0-9A-Fa-f]{2})+@postgres:5432/jyotisha$ ]]; then
+  echo "invalid production database setting: APP_DATABASE_URL" >&2
+  exit 1
+fi
+
+require_literal SERVICE_DATABASE_URL 49
+service_database_url="$LITERAL_VALUE"
+if ! [[ "$service_database_url" =~ ^postgresql://service_runtime:([A-Za-z0-9._~-]|%[0-9A-Fa-f]{2})+@postgres:5432/jyotisha$ ]]; then
+  echo "invalid production database setting: SERVICE_DATABASE_URL" >&2
+  exit 1
+fi
+
+require_literal ADMIN_DATABASE_URL 45
+admin_database_url="$LITERAL_VALUE"
+if ! [[ "$admin_database_url" =~ ^postgresql://admin_runtime:([A-Za-z0-9._~-]|%[0-9A-Fa-f]{2})+@postgres:5432/jyotisha$ ]]; then
+  echo "invalid production database setting: ADMIN_DATABASE_URL" >&2
+  exit 1
+fi
+
+require_literal BETTER_AUTH_USER_SECRET 32
+require_literal RESEND_API_KEY 10
+require_literal RESEND_FROM_EMAIL 5
+if [[ "$LITERAL_VALUE" != *@* ]]; then
+  echo "invalid production identity setting: RESEND_FROM_EMAIL" >&2
+  exit 1
+fi
+require_literal ADMIN_EMAILS 3
+if [[ "$LITERAL_VALUE" != *@* ]]; then
+  echo "invalid production identity setting: ADMIN_EMAILS" >&2
+  exit 1
+fi
+require_literal EPAY_CONFIG_ENCRYPTION_KEY 44
+if [ "${#LITERAL_VALUE}" -ne 44 ] ||
+  [[ ! "$LITERAL_VALUE" =~ ^[A-Za-z0-9+/]{43}=$ ]]; then
+  echo "invalid production identity setting: EPAY_CONFIG_ENCRYPTION_KEY" >&2
+  exit 1
+fi
+require_literal MODEL_PROVIDER_CONFIG_ENCRYPTION_KEY 44
+if [ "${#LITERAL_VALUE}" -ne 44 ] ||
+  [[ ! "$LITERAL_VALUE" =~ ^[A-Za-z0-9+/]{43}=$ ]]; then
+  echo "invalid production model provider setting: MODEL_PROVIDER_CONFIG_ENCRYPTION_KEY" >&2
+  exit 1
+fi
+legacy_model_setting_pattern='^(OPENAI_API_KEY|ANTHROPIC_API_KEY|DEEPSEEK_API_KEY|LLM_API_KEY|LLM_MODELS_JSON|LLM_BASE_URL|LLM_MODEL|LLM_DEFAULT_MODEL_ID|LLM_PROVIDER_ID|MASTRA_MODEL|MODEL_PROVIDER_[A-Z0-9_]+_API_KEY)='
+if grep -Eq "$legacy_model_setting_pattern" "$ENV_FILE"; then
+  echo "legacy model environment settings are forbidden" >&2
+  exit 1
+fi
+
+require_selector EPAY_CHAT_ENABLED false
+require_literal JYOTISH_DYNAMIC_RECTIFICATION_TOKEN 32
+
+personal_report_enabled_count="$(grep -Ec '^PERSONAL_REPORT_ENABLED=' "$ENV_FILE" || true)"
+personal_report_enabled="$(grep -E '^PERSONAL_REPORT_ENABLED=' "$ENV_FILE" || true)"
+personal_report_enabled="${personal_report_enabled#*=}"
+if [ "$personal_report_enabled_count" -ne 1 ] ||
+  [[ "$personal_report_enabled" != "true" && "$personal_report_enabled" != "false" ]]; then
+  echo "invalid production personal report setting: PERSONAL_REPORT_ENABLED" >&2
+  exit 1
+fi
+
+personal_report_daily_limit_count="$(grep -Ec '^PERSONAL_REPORT_DAILY_LIMIT=' "$ENV_FILE" || true)"
+personal_report_daily_limit="$(grep -E '^PERSONAL_REPORT_DAILY_LIMIT=' "$ENV_FILE" || true)"
+personal_report_daily_limit="${personal_report_daily_limit#*=}"
+if [ "$personal_report_daily_limit_count" -ne 1 ] ||
+  [[ ! "$personal_report_daily_limit" =~ ^[1-9][0-9]*$ ]]; then
+  echo "invalid production personal report setting: PERSONAL_REPORT_DAILY_LIMIT" >&2
+  exit 1
+fi
+
+echo "production environment selectors: valid"