Jyotisha/docs/operations/production-server-migration-2026-08.md

# Production migration to 118.194.235.34

Status: **planned; do not change DNS or retire the old production yet**.

This runbook moves production to the current reviewed `staging` release while also changing the persistence and identity layers:

- old runtime: VPS + Supabase PostgreSQL + Supabase Auth;
- target runtime: `118.194.235.34` + private PostgreSQL 17 + Better Auth;
- user site: `https://jyotisha.chat`;
- admin site: `https://admin.jyotisha.chat`.

This is not a volume copy. A full Supabase dump must not be restored over the target database.

## Confirmed migration decisions

- production Owner: `luna@copse.life` / `b8907d0c-6ed0-4270-b866-7e83bb4a1b26`;
- source Supabase project: `vtvnfqmonbfuxmqkqdlc`;
- the legacy production `ADMIN_EMAILS` allowlist contains only the designated Owner, and the legacy database has no `public.admin_users`; the ETL seeds that attested Owner into the target admin model;
- payment and model-provider ciphertext mode: `exclude`; re-enter both configurations after cutover with newly generated encryption keys;
- release policy: validate on `staging`, then promote the same accepted SHA to production;
- DNS remains unchanged until the final ETL and reconciliation pass;
- proposed maintenance window: `2026-08-12 02:00–04:00 UTC+8`, pending operator confirmation after rehearsal timing.

## Release invariants

The production deployment and schema-migration workflows are manual-only and accept a full lowercase 40-character `deploy_sha`. A normal production mutation proceeds only when all of the following identify that exact SHA:

1. current `main`;
2. current `staging`;
3. a successful push-triggered `Staging Backend Quality Gate`;
4. a successful manually triggered `Jyotish Release Quality Gate`;
5. the public staging `/api/health` deployment identity.

The deployment workflow consumes the exact API and Web image digests recorded by the staging gate. It does not build on the 2-core/4-GB production host, import user data, run schema migrations, or change DNS. The separate `Migrate Production Database` workflow uses the same gate-attested Web image only to run the schema checker/migrator/checker sequence; it does not run ETL, deploy the application, or change DNS. Both workflows share the `production-mutation` lock.

## Required Gitea configuration

Repository variables:

| Name | Required value |
| --- | --- |
| `PRODUCTION_HOST` | `118.194.235.34` |
| `PRODUCTION_PORT` | `22` (confirmed on 2026-08-09) |
| `PRODUCTION_USER` | `deploy` |
| `PRODUCTION_PATH` | `/opt/jyotisha-production` |
| `PRODUCTION_URL` | `https://jyotisha.chat` |
| `PRODUCTION_ADMIN_URL` | `https://admin.jyotisha.chat` |
| `PRODUCTION_KNOWN_HOSTS` | Independently verified pinned host-key line |
| `STAGING_URL` | `https://staging.jyotisha.chat` |

Repository secrets:

- `PRODUCTION_SSH_PRIVATE_KEY`: the dedicated deploy private-key file encoded as one unwrapped base64 line;
- `REGISTRY_USERNAME` and `REGISTRY_PASSWORD`.

Do not put the Ubuntu password, database URLs, Resend key, payment key, model-provider key, or encryption master keys in Gitea. The supplied bootstrap password must be rotated after an SSH key has been verified; it must never be committed or printed in a workflow.

## New-server bootstrap

Perform this interactively before any workflow dispatch:

1. Patch Ubuntu and install Docker Engine, Compose v2, `rsync`, `curl`, `jq`, `flock`, and UFW.
2. Create a non-root `deploy` user, install a dedicated Ed25519 public key, and grant only the reviewed passwordless Docker command used by the deployment workflows. Ownership normalization must reuse the capability-constrained helper container described below rather than adding a separate passwordless host `chown` rule.
3. Verify a second key-only session and rotate the exposed bootstrap password. For this host, the production owner explicitly requires password authentication to remain enabled for other operators; do not change `PasswordAuthentication`. Workflows must still use the dedicated deploy key.
4. Permit only the confirmed SSH port plus `80/tcp`, `443/tcp`, and `443/udp`. Do not publish `3000`, `5200`, `5432`, or the Docker API.
5. Create a 2–4 GB swap file and enable Docker log rotation. Keep at least 15 GB free before the first image pull and database import.
6. Create `/opt/jyotisha-production`, owned by `deploy`, and preload the reviewed `postgres:17-alpine` and Caddy images. PostgreSQL image upgrades are separate maintenance operations.

Create these host-only files with owner `deploy` and mode `0600`:

```text
/opt/jyotisha-production/.env.production
/opt/jyotisha-production/.env.production.database
```

The application selectors must be exact:

```dotenv
APP_ENV_FILE=../.env.production
CADDYFILE_PATH=./Caddyfile.production.selfhosted
SITE_ADDRESS=https://jyotisha.chat
AUTH_PROVIDER=self-hosted
SELF_HOSTED_IDENTITY_ENABLED=true
AUTH_USER_ORIGIN=https://jyotisha.chat
ADMIN_USER_ORIGIN=https://admin.jyotisha.chat
EPAY_CHAT_ENABLED=false
```

Use distinct production credentials for PostgreSQL roles, Better Auth, Resend, backup encryption, and dynamic rectification. Preserve the existing `EPAY_CONFIG_ENCRYPTION_KEY` and `MODEL_PROVIDER_CONFIG_ENCRYPTION_KEY` only through an approved secret-to-secret transfer. If either key cannot be transferred safely, exclude its ciphertext rows from import and re-enter those settings in the new admin UI.

## Database migration engineering gate

Before importing data, dispatch Gitea Actions → `Create Production Recovery Point` for the exact accepted release SHA. It validates the private production `.state` and `backups` paths, obtains the immutable image ID of the already-running PostgreSQL container, and reuses the existing passwordless Docker boundary to run that local image with `--pull never`, no network, a read-only root filesystem, `no-new-privileges`, and only `CHOWN` retained. With `.state` and `backups` bind-mounted and an existing regular shared lock additionally mounted directly at `/mutation.lock`, it restores that inode before changing its mode-`0700` parent directory, then restores the two directory mount points to documented `deploy` ownership when bootstrap ownership has drifted. The direct file mount handles a parent left half-repaired by an earlier failed run without adding `DAC_OVERRIDE` or depending on traversal through that parent. It then creates an AES-256-CBC/PBKDF2 encrypted custom-format dump on the production host, restores it into a disposable database, verifies non-sensitive table counts, removes only that disposable database, and uploads the encrypted dump plus verification manifest as a 30-day Gitea Actions artifact. It must preserve the existing lock inode, avoid recursively changing historical backups, and fail closed on symlinks, non-regular lock paths, or an unexpected image identity. The workflow prints the non-sensitive `recovery_reference` and exact UTC `recovery_created_at`; use those values only after the artifact upload succeeds.

Then dispatch Gitea Actions → `Migrate Production Database` for the same exact accepted release SHA. The workflow requires `main == staging == deploy_sha`, the same successful staging backend gate, the same manual release gate, and the public staging `/api/health` identity for that SHA. It also requires the recovery workflow's non-sensitive reference, exact UTC creation time, and `restore_verified=true`; the recovery point must be no more than 24 hours old and must already have passed the restore verification. It verifies the current production revision, obtains the gate-attested immutable Web image, runs the schema checker, applies only pending application schema migrations, and requires the checker to converge afterward.

Schema migration files are committed sequentially and are not one atomic transaction as a set. If a later file or post-check fails, earlier files may remain applied; stop, preserve evidence, and restore from the attested recovery point when repair-in-place is not explicitly reviewed. Do not assume a failed workflow means the database is unchanged.

The production database must already grant `migration_runner` membership in `schema_owner`; `deploy/postgres/001-bootstrap-roles.sh` grants only that migration role the ability to `SET ROLE schema_owner`. Identity, app, service, admin, and backup runtime roles must not receive this membership. The workflow checks the membership and refuses to add it itself.

The reviewed data-transfer entry point is `frontend/scripts/migrate-supabase-production.mjs`. It has separate `--preflight`, `--apply`, and `--verify` modes; the application deployment workflow never runs it automatically. Production cutover remains blocked until the exact production snapshot has completed an isolated rehearsal and final verification.

The tool must:

- connect to the source using `REPEATABLE READ READ ONLY`;
- refuse a non-empty target business database;
- use explicit columns and dependency order, never `SELECT *`;
- preserve all user UUIDs and transactional primary keys;
- import Supabase `auth.users` into Better Auth without passwords, sessions, JWTs, provider tokens, or MFA secrets;
- map `banned_until`, or emit an explicit blocked-user reconciliation manifest;
- merge seed/configuration records by natural key rather than copying target-generated IDs;
- map active administrators to canonical target role codes and require at least one explicit Owner;
- run post-import reconciliation for legacy billing and retired birth-time rectification rows;
- emit only counts, state aggregates, and normalized SHA-256 manifests—not email addresses, birth data, tokens, or connection strings;
- run all target writes in a transaction and roll back on failure.

The operator supplies these values only on the trusted migration host; do not store the database URLs or encryption keys in Gitea:

- `SUPABASE_SOURCE_DATABASE_URL`: the consistent read-only Supabase snapshot/source URL;
- `PRODUCTION_TARGET_DATABASE_URL`: the PostgreSQL 17 target URL using the migration role;
- `PRODUCTION_OWNER_USER_ID`: the UUID of the designated active source administrator;
- `PRODUCTION_OWNER_EMAIL`: the canonical email that must match that UUID in source `auth.users`;
- `PRODUCTION_CIPHERTEXT_MODE=preserve|exclude`; preserve additionally requires `PRODUCTION_CIPHERTEXT_KEYS_CONFIRMED=true`.

Run each phase separately and retain its redacted JSON manifest:

```bash
cd frontend
node scripts/migrate-supabase-production.mjs --preflight
node scripts/migrate-supabase-production.mjs --apply
node scripts/migrate-supabase-production.mjs --verify
```

`--apply` is intentionally one-shot: it refuses a populated target. If apply fails, discard or restore the isolated target, correct the cause, and rerun from an empty migrated schema rather than improvising a partial resume.

Do not import platform schemas, source roles/grants, Supabase migration ledgers, sessions, refresh tokens, or provider tokens. Do not use a full-database `pg_restore` against the target.

Because migrated users have no portable password/session, all sessions are invalidated and users sign in again through email OTP. Administrators re-enrol MFA.

## Rehearsal

On 2026-08-09, an isolated PostgreSQL 17 rehearsal completed `--preflight`, `--apply`, and `--verify` against a read-only production Supabase transaction. It reconciled 85 identities and all 18 selected non-empty/seeded legacy public tables by count, primary-key hash, normalized row hash, credit totals, consultation states, and rectification counts. This is rehearsal evidence only; it does not authorize the final write freeze, production import, or DNS change.

Complete the remaining runtime and restore rehearsal before scheduling the final window:

1. Apply all target schema migrations to an empty rehearsal database using the same schema migrator path as `Migrate Production Database`.
2. Run migration preflight, apply, post-import reconciliation, and verify.
3. Verify source/target row counts, primary-key set hashes, normalized row hashes, credit totals, payment state totals, subscriptions, reports, consultations, and rectification records.
4. Verify one Owner exists, every active admin has a target role, and database roles remain isolated.
5. Test OTP login, historical balance and history reads, admin login/RBAC, report generation/export, payment callback rejection/idempotency, and model provider access.
6. Create an encrypted backup, restore it into a separate database, and repeat smoke checks.
7. Record the observed export/import/verification duration and use it to set the maintenance window.

For the 2-core/4-GB host, keep database pools bounded (recommended starting maxima: identity 5, app 5, admin 3) rather than allowing three pools of 10 to consume all 30 PostgreSQL connections.

## DNS preparation

At least one current TTL period before cutover, reduce relevant Spaceship TTLs to `300`. Do not change record targets yet. Check both authoritative nameservers and remove any legacy `AAAA` record that points elsewhere.

Final records are:

| Type | Host | Value |
| --- | --- | --- |
| `A` | `@` | `118.194.235.34` |
| `A` | `admin` | `118.194.235.34` |
| `CNAME` | `www` | `jyotisha.chat` |

Both `jyotisha.chat` and `admin.jyotisha.chat` are required. The application rejects unknown identity hosts, and the user domain intentionally hides `/admin` and `/api/admin/*`.

## Cutover sequence

### T-24 hours

- Confirm the exact release SHA is deployed and accepted on staging.
- Run the manual release quality gate for that SHA.
- Confirm final backup capacity, restore rehearsal, SMTP/OTP delivery, and rollback contacts.
- Run `Create Production Recovery Point` no more than 24 hours before the schema migration; retain its encrypted off-site artifact and record the printed non-sensitive reference and UTC creation time.
- Dispatch `Migrate Production Database` for the accepted SHA with that recovery attestation and confirm its post-check reports no pending schema migrations.
- Record pending payment orders and long-running jobs; choose an explicit disposition for each.
- Dispatch `Deploy production` with `verification_mode=internal` only after target schema/data preparation. This verifies the new host without depending on public DNS.

### Maintenance freeze

1. Set payment/package creation off and keep `EPAY_CHAT_ENABLED=false`.
2. Put old production in maintenance mode and stop Web/Agent/background writes.
3. Disable new Supabase registrations for the window.
4. Confirm source row counts stop changing.
5. Take the final encrypted source backup and one consistent source snapshot.
6. Run the production ETL once, then post-import reconciliation and verification.
7. Verify role-specific `SELECT 1` connectivity for identity, app, service, and admin roles; verify permission isolation separately.
8. Keep the new site in maintenance mode while running user/admin/report/payment smoke checks.

Do not attempt an ad-hoc full-plus-incremental migration. Several tables lack a common `updated_at` or soft-delete contract, so an improvised delta can lose deletes, refunds, or accounting changes.

### DNS and public verification

1. Change the three Spaceship records only after all final data assertions pass.
2. Verify both authoritative nameservers, then public recursive resolvers.
3. Wait for Caddy certificates for both user and admin hosts.
4. Dispatch the same exact SHA with `verification_mode=public`.
5. Verify OTP login, logged-out account `401`, user-host admin paths `404`, admin-host unauthenticated behavior, health SHA, report generation/export, and one controlled payment callback test.
6. Re-enable public writes. Re-enable payment only after DNS convergence and callback verification.

## Go/no-go assertions

Cutover is **no-go** if any of these conditions is true:

- `main`, `staging`, staging health, gate artifact, or requested SHA differs;
- any migration or manifest checksum is unresolved;
- source/target identity, balance, order, subscription, or report reconciliation differs;
- there is no active Owner or an active admin has no role;
- migrated banned users are not accounted for;
- a production encryption key/ciphertext decision is unresolved;
- backup restore has not been demonstrated;
- role-specific database readiness or isolation fails;
- either user/admin TLS host is unavailable;
- pending payment writes or callbacks can still reach the old writable database.

## Rollback boundary

Before the new database accepts real writes, rollback is: restore old DNS targets, keep the old site/Supabase authoritative, and investigate the isolated target.

After the new database accepts real writes, a DNS-only rollback is unsafe. First stop new writes, reconcile the new PostgreSQL delta back to the chosen authority, and obtain an explicit operator decision. Otherwise post-cutover users, orders, credits, and reports can be lost.

Keep the old VPS in maintenance/read-only mode for at least 7–14 days and retain Supabase for 14–30 days. Do not destroy either immediately after DNS cutover.

## Workflow dispatch

Normal release:

1. Merge the reviewed `staging` release into `main` so both heads are the same SHA.
2. Confirm the staging push gate, public staging SHA, and manual release gate all succeeded for that SHA.
3. Open Gitea Actions → `Create Production Recovery Point`; enter the exact 40-character SHA and wait for the encrypted artifact upload plus restore verification to succeed. Record the printed `recovery_reference` and `recovery_created_at`.
4. Open Gitea Actions → `Migrate Production Database`; enter the same SHA and the no-more-than-24-hour-old recovery attestation, then confirm `restore_verified=true`. Wait for the post-migration checker to converge. Do not use this workflow for Supabase ETL.
5. Run the trusted-host ETL phases and retain the redacted reconciliation manifests when a data cutover is actually required. Ordinary forward schema releases do not rerun the one-shot Supabase ETL.
6. Open Gitea Actions → `Deploy production`.
7. Enter the same exact SHA, leave `allow_rollback=false`, and choose `internal` or `public` for the current cutover phase.

Application rollback accepts only an explicitly authorized, previously gate-attested SHA in reviewed `main` history. Database migrations and imported data are not rolled back by the application workflow.