Jyotisha/docs/operations/production-server-migration-2026-08.md

# Production migration to 118.194.235.34

Status: **planned; do not change DNS or retire the old production yet**.

This runbook moves production to the current reviewed `staging` release while also changing the persistence and identity layers:

- old runtime: VPS + Supabase PostgreSQL + Supabase Auth;
- target runtime: `118.194.235.34` + private PostgreSQL 17 + Better Auth;
- user site: `https://jyotisha.chat`;
- admin site: `https://admin.jyotisha.chat`.

This is not a volume copy. A full Supabase dump must not be restored over the target database.

## Release invariants

The production workflow is manual-only and accepts a full lowercase 40-character `deploy_sha`. A normal deployment proceeds only when all of the following identify that exact SHA:

1. current `main`;
2. current `staging`;
3. a successful push-triggered `Staging Backend Quality Gate`;
4. a successful manually triggered `Jyotish Release Quality Gate`;
5. the public staging `/api/health` deployment identity.

The workflow consumes the exact API and Web image digests recorded by the staging gate. It does not build on the 2-core/4-GB production host, import user data, run schema migrations, or change DNS.

## Required Gitea configuration

Repository variables:

| Name | Required value |
| --- | --- |
| `PRODUCTION_HOST` | `118.194.235.34` |
| `PRODUCTION_PORT` | Confirmed SSH port; do not assume `22` |
| `PRODUCTION_USER` | `deploy` |
| `PRODUCTION_PATH` | `/opt/jyotisha-production` |
| `PRODUCTION_URL` | `https://jyotisha.chat` |
| `PRODUCTION_ADMIN_URL` | `https://admin.jyotisha.chat` |
| `PRODUCTION_KNOWN_HOSTS` | Independently verified pinned host-key line |
| `STAGING_URL` | `https://staging.jyotisha.chat` |

Repository secrets:

- `PRODUCTION_SSH_PRIVATE_KEY`: the dedicated deploy private-key file encoded as one unwrapped base64 line;
- `REGISTRY_USERNAME` and `REGISTRY_PASSWORD`.

Do not put the Ubuntu password, database URLs, Resend key, payment key, model-provider key, or encryption master keys in Gitea. The supplied bootstrap password must be rotated after an SSH key has been verified; it must never be committed or printed in a workflow.

## New-server bootstrap

Perform this interactively before any workflow dispatch:

1. Patch Ubuntu and install Docker Engine, Compose v2, `rsync`, `curl`, `jq`, `flock`, and UFW.
2. Create a non-root `deploy` user, install a dedicated Ed25519 public key, and grant only the reviewed passwordless commands needed for Docker and deployment-tree ownership.
3. Verify a second key-only session, then disable root login and password authentication and rotate the bootstrap password.
4. Permit only the confirmed SSH port plus `80/tcp`, `443/tcp`, and `443/udp`. Do not publish `3000`, `5200`, `5432`, or the Docker API.
5. Create a 2–4 GB swap file and enable Docker log rotation. Keep at least 15 GB free before the first image pull and database import.
6. Create `/opt/jyotisha-production`, owned by `deploy`, and preload the reviewed `postgres:17-alpine` and Caddy images. PostgreSQL image upgrades are separate maintenance operations.

Create these host-only files with owner `deploy` and mode `0600`:

```text
/opt/jyotisha-production/.env.production
/opt/jyotisha-production/.env.production.database
```

The application selectors must be exact:

```dotenv
APP_ENV_FILE=../.env.production
CADDYFILE_PATH=./Caddyfile.production.selfhosted
SITE_ADDRESS=https://jyotisha.chat
AUTH_PROVIDER=self-hosted
SELF_HOSTED_IDENTITY_ENABLED=true
AUTH_USER_ORIGIN=https://jyotisha.chat
ADMIN_USER_ORIGIN=https://admin.jyotisha.chat
EPAY_CHAT_ENABLED=false
```

Use distinct production credentials for PostgreSQL roles, Better Auth, Resend, backup encryption, and dynamic rectification. Preserve the existing `EPAY_CONFIG_ENCRYPTION_KEY` and `MODEL_PROVIDER_CONFIG_ENCRYPTION_KEY` only through an approved secret-to-secret transfer. If either key cannot be transferred safely, exclude its ciphertext rows from import and re-enter those settings in the new admin UI.

## Database migration engineering gate

Before production cutover, implement and review `frontend/scripts/migrate-supabase-production.mjs` with `--preflight`, `--apply`, and `--verify` modes. Until that tool and its fixtures pass, production data cutover is blocked.

The tool must:

- connect to the source using `REPEATABLE READ READ ONLY`;
- refuse a non-empty target business database;
- use explicit columns and dependency order, never `SELECT *`;
- preserve all user UUIDs and transactional primary keys;
- import Supabase `auth.users` into Better Auth without passwords, sessions, JWTs, provider tokens, or MFA secrets;
- map `banned_until`, or emit an explicit blocked-user reconciliation manifest;
- merge seed/configuration records by natural key rather than copying target-generated IDs;
- map active administrators to canonical target role codes and require at least one explicit Owner;
- run post-import reconciliation for legacy billing and retired birth-time rectification rows;
- emit only counts, state aggregates, and normalized SHA-256 manifests—not email addresses, birth data, tokens, or connection strings;
- run all target writes in a transaction and roll back on failure.

Do not import platform schemas, source roles/grants, Supabase migration ledgers, sessions, refresh tokens, or provider tokens. Do not use a full-database `pg_restore` against the target.

Because migrated users have no portable password/session, all sessions are invalidated and users sign in again through email OTP. Administrators re-enrol MFA.

## Rehearsal

Complete at least one isolated full-data rehearsal before scheduling the final window:

1. Apply all target schema migrations to an empty rehearsal database.
2. Run migration preflight, apply, post-import reconciliation, and verify.
3. Verify source/target row counts, primary-key set hashes, normalized row hashes, credit totals, payment state totals, subscriptions, reports, consultations, and rectification records.
4. Verify one Owner exists, every active admin has a target role, and database roles remain isolated.
5. Test OTP login, historical balance and history reads, admin login/RBAC, report generation/export, payment callback rejection/idempotency, and model provider access.
6. Create an encrypted backup, restore it into a separate database, and repeat smoke checks.
7. Record the observed export/import/verification duration and use it to set the maintenance window.

For the 2-core/4-GB host, keep database pools bounded (recommended starting maxima: identity 5, app 5, admin 3) rather than allowing three pools of 10 to consume all 30 PostgreSQL connections.

## DNS preparation

At least one current TTL period before cutover, reduce relevant Spaceship TTLs to `300`. Do not change record targets yet. Check both authoritative nameservers and remove any legacy `AAAA` record that points elsewhere.

Final records are:

| Type | Host | Value |
| --- | --- | --- |
| `A` | `@` | `118.194.235.34` |
| `A` | `admin` | `118.194.235.34` |
| `CNAME` | `www` | `jyotisha.chat` |

Both `jyotisha.chat` and `admin.jyotisha.chat` are required. The application rejects unknown identity hosts, and the user domain intentionally hides `/admin` and `/api/admin/*`.

## Cutover sequence

### T-24 hours

- Confirm the exact release SHA is deployed and accepted on staging.
- Run the manual release quality gate for that SHA.
- Confirm final backup capacity, restore rehearsal, SMTP/OTP delivery, and rollback contacts.
- Record pending payment orders and long-running jobs; choose an explicit disposition for each.
- Dispatch `Deploy production` with `verification_mode=internal` only after target schema/data preparation. This verifies the new host without depending on public DNS.

### Maintenance freeze

1. Set payment/package creation off and keep `EPAY_CHAT_ENABLED=false`.
2. Put old production in maintenance mode and stop Web/Agent/background writes.
3. Disable new Supabase registrations for the window.
4. Confirm source row counts stop changing.
5. Take the final encrypted source backup and one consistent source snapshot.
6. Run the production ETL once, then post-import reconciliation and verification.
7. Verify role-specific `SELECT 1` connectivity for identity, app, service, and admin roles; verify permission isolation separately.
8. Keep the new site in maintenance mode while running user/admin/report/payment smoke checks.

Do not attempt an ad-hoc full-plus-incremental migration. Several tables lack a common `updated_at` or soft-delete contract, so an improvised delta can lose deletes, refunds, or accounting changes.

### DNS and public verification

1. Change the three Spaceship records only after all final data assertions pass.
2. Verify both authoritative nameservers, then public recursive resolvers.
3. Wait for Caddy certificates for both user and admin hosts.
4. Dispatch the same exact SHA with `verification_mode=public`.
5. Verify OTP login, logged-out account `401`, user-host admin paths `404`, admin-host unauthenticated behavior, health SHA, report generation/export, and one controlled payment callback test.
6. Re-enable public writes. Re-enable payment only after DNS convergence and callback verification.

## Go/no-go assertions

Cutover is **no-go** if any of these conditions is true:

- `main`, `staging`, staging health, gate artifact, or requested SHA differs;
- any migration or manifest checksum is unresolved;
- source/target identity, balance, order, subscription, or report reconciliation differs;
- there is no active Owner or an active admin has no role;
- migrated banned users are not accounted for;
- a production encryption key/ciphertext decision is unresolved;
- backup restore has not been demonstrated;
- role-specific database readiness or isolation fails;
- either user/admin TLS host is unavailable;
- pending payment writes or callbacks can still reach the old writable database.

## Rollback boundary

Before the new database accepts real writes, rollback is: restore old DNS targets, keep the old site/Supabase authoritative, and investigate the isolated target.

After the new database accepts real writes, a DNS-only rollback is unsafe. First stop new writes, reconcile the new PostgreSQL delta back to the chosen authority, and obtain an explicit operator decision. Otherwise post-cutover users, orders, credits, and reports can be lost.

Keep the old VPS in maintenance/read-only mode for at least 7–14 days and retain Supabase for 14–30 days. Do not destroy either immediately after DNS cutover.

## Workflow dispatch

Normal release:

1. Merge the reviewed `staging` release into `main` so both heads are the same SHA.
2. Confirm the staging push gate, public staging SHA, and manual release gate all succeeded for that SHA.
3. Open Gitea Actions → `Deploy production`.
4. Enter the exact 40-character SHA, leave `allow_rollback=false`, and choose `internal` or `public` for the current cutover phase.

Application rollback accepts only an explicitly authorized, previously gate-attested SHA in reviewed `main` history. Database migrations and imported data are not rolled back by the application workflow.