Jyotisha/docs/operations/production-server-migration-2026-08.md

Production migration to 118.194.235.34

Status: planned; do not change DNS or retire the old production yet.

This runbook moves production to the current reviewed staging release while also changing the persistence and identity layers:

• old runtime: VPS + Supabase PostgreSQL + Supabase Auth;
• target runtime: 118.194.235.34 + private PostgreSQL 17 + Better Auth;
• user site: https://jyotisha.chat;
• admin site: https://admin.jyotisha.chat.

This is not a volume copy. A full Supabase dump must not be restored over the target database.

Release invariants

The production workflow is manual-only and accepts a full lowercase 40-character deploy_sha. A normal deployment proceeds only when all of the following identify that exact SHA:

1. current main;
2. current staging;
3. a successful push-triggered Staging Backend Quality Gate;
4. a successful manually triggered Jyotish Release Quality Gate;
5. the public staging /api/health deployment identity.

The workflow consumes the exact API and Web image digests recorded by the staging gate. It does not build on the 2-core/4-GB production host, import user data, run schema migrations, or change DNS.

Required Gitea configuration

Repository variables:

Name	Required value
PRODUCTION_HOST	118.194.235.34
PRODUCTION_PORT	Confirmed SSH port; do not assume 22
PRODUCTION_USER	deploy
PRODUCTION_PATH	/opt/jyotisha-production
PRODUCTION_URL	https://jyotisha.chat
PRODUCTION_ADMIN_URL	https://admin.jyotisha.chat
PRODUCTION_KNOWN_HOSTS	Independently verified pinned host-key line
STAGING_URL	https://staging.jyotisha.chat

Repository secrets:

• PRODUCTION_SSH_PRIVATE_KEY: the dedicated deploy private-key file encoded as one unwrapped base64 line;
• REGISTRY_USERNAME and REGISTRY_PASSWORD.

Do not put the Ubuntu password, database URLs, Resend key, payment key, model-provider key, or encryption master keys in Gitea. The supplied bootstrap password must be rotated after an SSH key has been verified; it must never be committed or printed in a workflow.

New-server bootstrap

Perform this interactively before any workflow dispatch:

1. Patch Ubuntu and install Docker Engine, Compose v2, rsync, curl, jq, flock, and UFW.
2. Create a non-root deploy user, install a dedicated Ed25519 public key, and grant only the reviewed passwordless commands needed for Docker and deployment-tree ownership.
3. Verify a second key-only session, then disable root login and password authentication and rotate the bootstrap password.
4. Permit only the confirmed SSH port plus 80/tcp, 443/tcp, and 443/udp. Do not publish 3000, 5200, 5432, or the Docker API.
5. Create a 2–4 GB swap file and enable Docker log rotation. Keep at least 15 GB free before the first image pull and database import.
6. Create /opt/jyotisha-production, owned by deploy, and preload the reviewed postgres:17-alpine and Caddy images. PostgreSQL image upgrades are separate maintenance operations.

Create these host-only files with owner deploy and mode 0600:

/opt/jyotisha-production/.env.production
/opt/jyotisha-production/.env.production.database

The application selectors must be exact:

APP_ENV_FILE=../.env.production
CADDYFILE_PATH=./Caddyfile.production.selfhosted
SITE_ADDRESS=https://jyotisha.chat
AUTH_PROVIDER=self-hosted
SELF_HOSTED_IDENTITY_ENABLED=true
AUTH_USER_ORIGIN=https://jyotisha.chat
ADMIN_USER_ORIGIN=https://admin.jyotisha.chat
EPAY_CHAT_ENABLED=false

Use distinct production credentials for PostgreSQL roles, Better Auth, Resend, backup encryption, and dynamic rectification. Preserve the existing EPAY_CONFIG_ENCRYPTION_KEY and MODEL_PROVIDER_CONFIG_ENCRYPTION_KEY only through an approved secret-to-secret transfer. If either key cannot be transferred safely, exclude its ciphertext rows from import and re-enter those settings in the new admin UI.

Database migration engineering gate

Before production cutover, implement and review frontend/scripts/migrate-supabase-production.mjs with --preflight, --apply, and --verify modes. Until that tool and its fixtures pass, production data cutover is blocked.

The tool must:

• connect to the source using REPEATABLE READ READ ONLY;
• refuse a non-empty target business database;
• use explicit columns and dependency order, never SELECT *;
• preserve all user UUIDs and transactional primary keys;
• import Supabase auth.users into Better Auth without passwords, sessions, JWTs, provider tokens, or MFA secrets;
• map banned_until, or emit an explicit blocked-user reconciliation manifest;
• merge seed/configuration records by natural key rather than copying target-generated IDs;
• map active administrators to canonical target role codes and require at least one explicit Owner;
• run post-import reconciliation for legacy billing and retired birth-time rectification rows;
• emit only counts, state aggregates, and normalized SHA-256 manifests—not email addresses, birth data, tokens, or connection strings;
• run all target writes in a transaction and roll back on failure.

Do not import platform schemas, source roles/grants, Supabase migration ledgers, sessions, refresh tokens, or provider tokens. Do not use a full-database pg_restore against the target.

Because migrated users have no portable password/session, all sessions are invalidated and users sign in again through email OTP. Administrators re-enrol MFA.

Rehearsal

Complete at least one isolated full-data rehearsal before scheduling the final window:

1. Apply all target schema migrations to an empty rehearsal database.
2. Run migration preflight, apply, post-import reconciliation, and verify.
3. Verify source/target row counts, primary-key set hashes, normalized row hashes, credit totals, payment state totals, subscriptions, reports, consultations, and rectification records.
4. Verify one Owner exists, every active admin has a target role, and database roles remain isolated.
5. Test OTP login, historical balance and history reads, admin login/RBAC, report generation/export, payment callback rejection/idempotency, and model provider access.
6. Create an encrypted backup, restore it into a separate database, and repeat smoke checks.
7. Record the observed export/import/verification duration and use it to set the maintenance window.

For the 2-core/4-GB host, keep database pools bounded (recommended starting maxima: identity 5, app 5, admin 3) rather than allowing three pools of 10 to consume all 30 PostgreSQL connections.

DNS preparation

At least one current TTL period before cutover, reduce relevant Spaceship TTLs to 300. Do not change record targets yet. Check both authoritative nameservers and remove any legacy AAAA record that points elsewhere.

Final records are:

Type	Host	Value
A	@	118.194.235.34
A	admin	118.194.235.34
CNAME	www	jyotisha.chat

Both jyotisha.chat and admin.jyotisha.chat are required. The application rejects unknown identity hosts, and the user domain intentionally hides /admin and /api/admin/*.

Cutover sequence

T-24 hours

• Confirm the exact release SHA is deployed and accepted on staging.
• Run the manual release quality gate for that SHA.
• Confirm final backup capacity, restore rehearsal, SMTP/OTP delivery, and rollback contacts.
• Record pending payment orders and long-running jobs; choose an explicit disposition for each.
• Dispatch Deploy production with verification_mode=internal only after target schema/data preparation. This verifies the new host without depending on public DNS.

Maintenance freeze

1. Set payment/package creation off and keep EPAY_CHAT_ENABLED=false.
2. Put old production in maintenance mode and stop Web/Agent/background writes.
3. Disable new Supabase registrations for the window.
4. Confirm source row counts stop changing.
5. Take the final encrypted source backup and one consistent source snapshot.
6. Run the production ETL once, then post-import reconciliation and verification.
7. Verify role-specific SELECT 1 connectivity for identity, app, service, and admin roles; verify permission isolation separately.
8. Keep the new site in maintenance mode while running user/admin/report/payment smoke checks.

Do not attempt an ad-hoc full-plus-incremental migration. Several tables lack a common updated_at or soft-delete contract, so an improvised delta can lose deletes, refunds, or accounting changes.

DNS and public verification

1. Change the three Spaceship records only after all final data assertions pass.
2. Verify both authoritative nameservers, then public recursive resolvers.
3. Wait for Caddy certificates for both user and admin hosts.
4. Dispatch the same exact SHA with verification_mode=public.
5. Verify OTP login, logged-out account 401, user-host admin paths 404, admin-host unauthenticated behavior, health SHA, report generation/export, and one controlled payment callback test.
6. Re-enable public writes. Re-enable payment only after DNS convergence and callback verification.

Go/no-go assertions

Cutover is no-go if any of these conditions is true:

• main, staging, staging health, gate artifact, or requested SHA differs;
• any migration or manifest checksum is unresolved;
• source/target identity, balance, order, subscription, or report reconciliation differs;
• there is no active Owner or an active admin has no role;
• migrated banned users are not accounted for;
• a production encryption key/ciphertext decision is unresolved;
• backup restore has not been demonstrated;
• role-specific database readiness or isolation fails;
• either user/admin TLS host is unavailable;
• pending payment writes or callbacks can still reach the old writable database.

Rollback boundary

Before the new database accepts real writes, rollback is: restore old DNS targets, keep the old site/Supabase authoritative, and investigate the isolated target.

After the new database accepts real writes, a DNS-only rollback is unsafe. First stop new writes, reconcile the new PostgreSQL delta back to the chosen authority, and obtain an explicit operator decision. Otherwise post-cutover users, orders, credits, and reports can be lost.

Keep the old VPS in maintenance/read-only mode for at least 7–14 days and retain Supabase for 14–30 days. Do not destroy either immediately after DNS cutover.

Workflow dispatch

Normal release:

1. Merge the reviewed staging release into main so both heads are the same SHA.
2. Confirm the staging push gate, public staging SHA, and manual release gate all succeeded for that SHA.
3. Open Gitea Actions → Deploy production.
4. Enter the exact 40-character SHA, leave allow_rollback=false, and choose internal or public for the current cutover phase.

Application rollback accepts only an explicitly authorized, previously gate-attested SHA in reviewed main history. Database migrations and imported data are not rolled back by the application workflow.