Jyotisha/docs/operations/production-server-migration-2026-08.md

Production migration to 118.194.235.34

Status: planned; do not change DNS or retire the old production yet.

This runbook moves production to the current reviewed staging release while also changing the persistence and identity layers:

• old runtime: VPS + Supabase PostgreSQL + Supabase Auth;
• target runtime: 118.194.235.34 + private PostgreSQL 17 + Better Auth;
• user site: https://jyotisha.chat;
• admin site: https://admin.jyotisha.chat.

This is not a volume copy. A full Supabase dump must not be restored over the target database.

Confirmed migration decisions

• production Owner: luna@copse.life / b8907d0c-6ed0-4270-b866-7e83bb4a1b26;
• source Supabase project: vtvnfqmonbfuxmqkqdlc;
• the legacy production ADMIN_EMAILS allowlist contains only the designated Owner, and the legacy database has no public.admin_users; the ETL seeds that attested Owner into the target admin model;
• payment and model-provider ciphertext mode: exclude; re-enter both configurations after cutover with newly generated encryption keys;
• release policy: validate on staging, then promote the same accepted SHA to production;
• DNS remains unchanged until the final ETL and reconciliation pass;
• proposed maintenance window: 2026-08-12 02:00–04:00 UTC+8, pending operator confirmation after rehearsal timing.

Release invariants

The production deployment and schema-migration workflows are manual-only and accept a full lowercase 40-character deploy_sha. A normal production mutation proceeds only when all of the following identify that exact SHA:

1. current main;
2. current staging;
3. a successful push-triggered Staging Backend Quality Gate;
4. a successful manually triggered Jyotish Release Quality Gate;
5. the public staging /api/health deployment identity.

The deployment workflow consumes the exact API and Web image digests recorded by the staging gate. It does not build on the 2-core/4-GB production host, import user data, run schema migrations, or change DNS. The separate Migrate Production Database workflow uses the same gate-attested Web image only to run the schema checker/migrator/checker sequence; it does not run ETL, deploy the application, or change DNS. Both workflows share the production-mutation lock.

Required Gitea configuration

Repository variables:

Name	Required value
PRODUCTION_HOST	118.194.235.34
PRODUCTION_PORT	22 (confirmed on 2026-08-09)
PRODUCTION_USER	deploy
PRODUCTION_PATH	/opt/jyotisha-production
PRODUCTION_URL	https://jyotisha.chat
PRODUCTION_ADMIN_URL	https://admin.jyotisha.chat
PRODUCTION_KNOWN_HOSTS	Independently verified pinned host-key line
STAGING_URL	https://staging.jyotisha.chat

Repository secrets:

• PRODUCTION_SSH_PRIVATE_KEY: the dedicated deploy private-key file encoded as one unwrapped base64 line;
• REGISTRY_USERNAME and REGISTRY_PASSWORD.

Do not put the Ubuntu password, database URLs, Resend key, payment key, model-provider key, or encryption master keys in Gitea. The supplied bootstrap password must be rotated after an SSH key has been verified; it must never be committed or printed in a workflow.

New-server bootstrap

Perform this interactively before any workflow dispatch:

1. Patch Ubuntu and install Docker Engine, Compose v2, rsync, curl, jq, flock, and UFW.
2. Create a non-root deploy user, install a dedicated Ed25519 public key, and grant only the reviewed passwordless commands needed for Docker and deployment-tree ownership.
3. Verify a second key-only session and rotate the exposed bootstrap password. For this host, the production owner explicitly requires password authentication to remain enabled for other operators; do not change PasswordAuthentication. Workflows must still use the dedicated deploy key.
4. Permit only the confirmed SSH port plus 80/tcp, 443/tcp, and 443/udp. Do not publish 3000, 5200, 5432, or the Docker API.
5. Create a 2–4 GB swap file and enable Docker log rotation. Keep at least 15 GB free before the first image pull and database import.
6. Create /opt/jyotisha-production, owned by deploy, and preload the reviewed postgres:17-alpine and Caddy images. PostgreSQL image upgrades are separate maintenance operations.

Create these host-only files with owner deploy and mode 0600:

/opt/jyotisha-production/.env.production
/opt/jyotisha-production/.env.production.database

The application selectors must be exact:

APP_ENV_FILE=../.env.production
CADDYFILE_PATH=./Caddyfile.production.selfhosted
SITE_ADDRESS=https://jyotisha.chat
AUTH_PROVIDER=self-hosted
SELF_HOSTED_IDENTITY_ENABLED=true
AUTH_USER_ORIGIN=https://jyotisha.chat
ADMIN_USER_ORIGIN=https://admin.jyotisha.chat
EPAY_CHAT_ENABLED=false

Use distinct production credentials for PostgreSQL roles, Better Auth, Resend, backup encryption, and dynamic rectification. Preserve the existing EPAY_CONFIG_ENCRYPTION_KEY and MODEL_PROVIDER_CONFIG_ENCRYPTION_KEY only through an approved secret-to-secret transfer. If either key cannot be transferred safely, exclude its ciphertext rows from import and re-enter those settings in the new admin UI.

Database migration engineering gate

Before importing data, dispatch Gitea Actions → Create Production Recovery Point for the exact accepted release SHA. It creates an AES-256-CBC/PBKDF2 encrypted custom-format dump on the production host, restores it into a disposable database, verifies non-sensitive table counts, removes only the disposable database, and uploads the encrypted dump plus verification manifest as a 30-day Gitea Actions artifact. The workflow prints the non-sensitive recovery_reference and exact UTC recovery_created_at; use those values only after the artifact upload succeeds.

Then dispatch Gitea Actions → Migrate Production Database for the same exact accepted release SHA. The workflow requires main == staging == deploy_sha, the same successful staging backend gate, the same manual release gate, and the public staging /api/health identity for that SHA. It also requires the recovery workflow's non-sensitive reference, exact UTC creation time, and restore_verified=true; the recovery point must be no more than 24 hours old and must already have passed the restore verification. It verifies the current production revision, obtains the gate-attested immutable Web image, runs the schema checker, applies only pending application schema migrations, and requires the checker to converge afterward.

Schema migration files are committed sequentially and are not one atomic transaction as a set. If a later file or post-check fails, earlier files may remain applied; stop, preserve evidence, and restore from the attested recovery point when repair-in-place is not explicitly reviewed. Do not assume a failed workflow means the database is unchanged.

The production database must already grant migration_runner membership in schema_owner; deploy/postgres/001-bootstrap-roles.sh grants only that migration role the ability to SET ROLE schema_owner. Identity, app, service, admin, and backup runtime roles must not receive this membership. The workflow checks the membership and refuses to add it itself.

The reviewed data-transfer entry point is frontend/scripts/migrate-supabase-production.mjs. It has separate --preflight, --apply, and --verify modes; the application deployment workflow never runs it automatically. Production cutover remains blocked until the exact production snapshot has completed an isolated rehearsal and final verification.

The tool must:

• connect to the source using REPEATABLE READ READ ONLY;
• refuse a non-empty target business database;
• use explicit columns and dependency order, never SELECT *;
• preserve all user UUIDs and transactional primary keys;
• import Supabase auth.users into Better Auth without passwords, sessions, JWTs, provider tokens, or MFA secrets;
• map banned_until, or emit an explicit blocked-user reconciliation manifest;
• merge seed/configuration records by natural key rather than copying target-generated IDs;
• map active administrators to canonical target role codes and require at least one explicit Owner;
• run post-import reconciliation for legacy billing and retired birth-time rectification rows;
• emit only counts, state aggregates, and normalized SHA-256 manifests—not email addresses, birth data, tokens, or connection strings;
• run all target writes in a transaction and roll back on failure.

The operator supplies these values only on the trusted migration host; do not store the database URLs or encryption keys in Gitea:

• SUPABASE_SOURCE_DATABASE_URL: the consistent read-only Supabase snapshot/source URL;
• PRODUCTION_TARGET_DATABASE_URL: the PostgreSQL 17 target URL using the migration role;
• PRODUCTION_OWNER_USER_ID: the UUID of the designated active source administrator;
• PRODUCTION_OWNER_EMAIL: the canonical email that must match that UUID in source auth.users;
• PRODUCTION_CIPHERTEXT_MODE=preserve|exclude; preserve additionally requires PRODUCTION_CIPHERTEXT_KEYS_CONFIRMED=true.

Run each phase separately and retain its redacted JSON manifest:

cd frontend
node scripts/migrate-supabase-production.mjs --preflight
node scripts/migrate-supabase-production.mjs --apply
node scripts/migrate-supabase-production.mjs --verify

--apply is intentionally one-shot: it refuses a populated target. If apply fails, discard or restore the isolated target, correct the cause, and rerun from an empty migrated schema rather than improvising a partial resume.

Do not import platform schemas, source roles/grants, Supabase migration ledgers, sessions, refresh tokens, or provider tokens. Do not use a full-database pg_restore against the target.

Because migrated users have no portable password/session, all sessions are invalidated and users sign in again through email OTP. Administrators re-enrol MFA.

Rehearsal

On 2026-08-09, an isolated PostgreSQL 17 rehearsal completed --preflight, --apply, and --verify against a read-only production Supabase transaction. It reconciled 85 identities and all 18 selected non-empty/seeded legacy public tables by count, primary-key hash, normalized row hash, credit totals, consultation states, and rectification counts. This is rehearsal evidence only; it does not authorize the final write freeze, production import, or DNS change.

Complete the remaining runtime and restore rehearsal before scheduling the final window:

1. Apply all target schema migrations to an empty rehearsal database using the same schema migrator path as Migrate Production Database.
2. Run migration preflight, apply, post-import reconciliation, and verify.
3. Verify source/target row counts, primary-key set hashes, normalized row hashes, credit totals, payment state totals, subscriptions, reports, consultations, and rectification records.
4. Verify one Owner exists, every active admin has a target role, and database roles remain isolated.
5. Test OTP login, historical balance and history reads, admin login/RBAC, report generation/export, payment callback rejection/idempotency, and model provider access.
6. Create an encrypted backup, restore it into a separate database, and repeat smoke checks.
7. Record the observed export/import/verification duration and use it to set the maintenance window.

For the 2-core/4-GB host, keep database pools bounded (recommended starting maxima: identity 5, app 5, admin 3) rather than allowing three pools of 10 to consume all 30 PostgreSQL connections.

DNS preparation

At least one current TTL period before cutover, reduce relevant Spaceship TTLs to 300. Do not change record targets yet. Check both authoritative nameservers and remove any legacy AAAA record that points elsewhere.

Final records are:

Type	Host	Value
A	@	118.194.235.34
A	admin	118.194.235.34
CNAME	www	jyotisha.chat

Both jyotisha.chat and admin.jyotisha.chat are required. The application rejects unknown identity hosts, and the user domain intentionally hides /admin and /api/admin/*.

Cutover sequence

T-24 hours

• Confirm the exact release SHA is deployed and accepted on staging.
• Run the manual release quality gate for that SHA.
• Confirm final backup capacity, restore rehearsal, SMTP/OTP delivery, and rollback contacts.
• Run Create Production Recovery Point no more than 24 hours before the schema migration; retain its encrypted off-site artifact and record the printed non-sensitive reference and UTC creation time.
• Dispatch Migrate Production Database for the accepted SHA with that recovery attestation and confirm its post-check reports no pending schema migrations.
• Record pending payment orders and long-running jobs; choose an explicit disposition for each.
• Dispatch Deploy production with verification_mode=internal only after target schema/data preparation. This verifies the new host without depending on public DNS.

Maintenance freeze

1. Set payment/package creation off and keep EPAY_CHAT_ENABLED=false.
2. Put old production in maintenance mode and stop Web/Agent/background writes.
3. Disable new Supabase registrations for the window.
4. Confirm source row counts stop changing.
5. Take the final encrypted source backup and one consistent source snapshot.
6. Run the production ETL once, then post-import reconciliation and verification.
7. Verify role-specific SELECT 1 connectivity for identity, app, service, and admin roles; verify permission isolation separately.
8. Keep the new site in maintenance mode while running user/admin/report/payment smoke checks.

Do not attempt an ad-hoc full-plus-incremental migration. Several tables lack a common updated_at or soft-delete contract, so an improvised delta can lose deletes, refunds, or accounting changes.

DNS and public verification

1. Change the three Spaceship records only after all final data assertions pass.
2. Verify both authoritative nameservers, then public recursive resolvers.
3. Wait for Caddy certificates for both user and admin hosts.
4. Dispatch the same exact SHA with verification_mode=public.
5. Verify OTP login, logged-out account 401, user-host admin paths 404, admin-host unauthenticated behavior, health SHA, report generation/export, and one controlled payment callback test.
6. Re-enable public writes. Re-enable payment only after DNS convergence and callback verification.

Go/no-go assertions

Cutover is no-go if any of these conditions is true:

• main, staging, staging health, gate artifact, or requested SHA differs;
• any migration or manifest checksum is unresolved;
• source/target identity, balance, order, subscription, or report reconciliation differs;
• there is no active Owner or an active admin has no role;
• migrated banned users are not accounted for;
• a production encryption key/ciphertext decision is unresolved;
• backup restore has not been demonstrated;
• role-specific database readiness or isolation fails;
• either user/admin TLS host is unavailable;
• pending payment writes or callbacks can still reach the old writable database.

Rollback boundary

Before the new database accepts real writes, rollback is: restore old DNS targets, keep the old site/Supabase authoritative, and investigate the isolated target.

After the new database accepts real writes, a DNS-only rollback is unsafe. First stop new writes, reconcile the new PostgreSQL delta back to the chosen authority, and obtain an explicit operator decision. Otherwise post-cutover users, orders, credits, and reports can be lost.

Keep the old VPS in maintenance/read-only mode for at least 7–14 days and retain Supabase for 14–30 days. Do not destroy either immediately after DNS cutover.

Workflow dispatch

Normal release:

1. Merge the reviewed staging release into main so both heads are the same SHA.
2. Confirm the staging push gate, public staging SHA, and manual release gate all succeeded for that SHA.
3. Open Gitea Actions → Create Production Recovery Point; enter the exact 40-character SHA and wait for the encrypted artifact upload plus restore verification to succeed. Record the printed recovery_reference and recovery_created_at.
4. Open Gitea Actions → Migrate Production Database; enter the same SHA and the no-more-than-24-hour-old recovery attestation, then confirm restore_verified=true. Wait for the post-migration checker to converge. Do not use this workflow for Supabase ETL.
5. Run the trusted-host ETL phases and retain the redacted reconciliation manifests when a data cutover is actually required. Ordinary forward schema releases do not rerun the one-shot Supabase ETL.
6. Open Gitea Actions → Deploy production.
7. Enter the same exact SHA, leave allow_rollback=false, and choose internal or public for the current cutover phase.

Application rollback accepts only an explicitly authorized, previously gate-attested SHA in reviewed main history. Database migrations and imported data are not rolled back by the application workflow.