ops: add production PostgreSQL migration tooling

docs/operations/production-server-migration-2026-08.md

@@ -13,7 +13,7 @@ This is not a volume copy. A full Supabase dump must not be restored over the ta
 
 ## Release invariants
 
-The production workflow is manual-only and accepts a full lowercase 40-character `deploy_sha`. A normal deployment proceeds only when all of the following identify that exact SHA:
+The production deployment and schema-migration workflows are manual-only and accept a full lowercase 40-character `deploy_sha`. A normal production mutation proceeds only when all of the following identify that exact SHA:
 
 1. current `main`;
 2. current `staging`;
@@ -21,7 +21,7 @@ The production workflow is manual-only and accepts a full lowercase 40-character
 4. a successful manually triggered `Jyotish Release Quality Gate`;
 5. the public staging `/api/health` deployment identity.
 
-The workflow consumes the exact API and Web image digests recorded by the staging gate. It does not build on the 2-core/4-GB production host, import user data, run schema migrations, or change DNS.
+The deployment workflow consumes the exact API and Web image digests recorded by the staging gate. It does not build on the 2-core/4-GB production host, import user data, run schema migrations, or change DNS. The separate `Migrate Production Database` workflow uses the same gate-attested Web image only to run the schema checker/migrator/checker sequence; it does not run ETL, deploy the application, or change DNS. Both workflows share the `production-mutation` lock.
 
 ## Required Gitea configuration
 
@@ -80,7 +80,13 @@ Use distinct production credentials for PostgreSQL roles, Better Auth, Resend, b
 
 ## Database migration engineering gate
 
-Before production cutover, implement and review `frontend/scripts/migrate-supabase-production.mjs` with `--preflight`, `--apply`, and `--verify` modes. Until that tool and its fixtures pass, production data cutover is blocked.
+Before importing data, dispatch Gitea Actions → `Migrate Production Database` for the exact accepted release SHA. The workflow requires `main == staging == deploy_sha`, the same successful staging backend gate, the same manual release gate, and the public staging `/api/health` identity for that SHA. It also requires a non-sensitive recovery reference, its exact UTC creation time, and `restore_verified=true`; the recovery point must be no more than 24 hours old and must already have passed a restore verification. It verifies the current production revision, obtains the gate-attested immutable Web image, runs the schema checker, applies only pending application schema migrations, and requires the checker to converge afterward.
+
+Schema migration files are committed sequentially and are not one atomic transaction as a set. If a later file or post-check fails, earlier files may remain applied; stop, preserve evidence, and restore from the attested recovery point when repair-in-place is not explicitly reviewed. Do not assume a failed workflow means the database is unchanged.
+
+The production database must already grant `migration_runner` membership in `schema_owner`; `deploy/postgres/001-bootstrap-roles.sh` grants only that migration role the ability to `SET ROLE schema_owner`. Identity, app, service, admin, and backup runtime roles must not receive this membership. The workflow checks the membership and refuses to add it itself.
+
+The reviewed data-transfer entry point is `frontend/scripts/migrate-supabase-production.mjs`. It has separate `--preflight`, `--apply`, and `--verify` modes; the application deployment workflow never runs it automatically. Production cutover remains blocked until the exact production snapshot has completed an isolated rehearsal and final verification.
 
 The tool must:
 
@@ -96,6 +102,24 @@ The tool must:
 - emit only counts, state aggregates, and normalized SHA-256 manifests—not email addresses, birth data, tokens, or connection strings;
 - run all target writes in a transaction and roll back on failure.
 
+The operator supplies these values only on the trusted migration host; do not store the database URLs or encryption keys in Gitea:
+
+- `SUPABASE_SOURCE_DATABASE_URL`: the consistent read-only Supabase snapshot/source URL;
+- `PRODUCTION_TARGET_DATABASE_URL`: the PostgreSQL 17 target URL using the migration role;
+- `PRODUCTION_OWNER_USER_ID`: the UUID of the designated active source administrator;
+- `PRODUCTION_CIPHERTEXT_MODE=preserve|exclude`; preserve additionally requires `PRODUCTION_CIPHERTEXT_KEYS_CONFIRMED=true`.
+
+Run each phase separately and retain its redacted JSON manifest:
+
+```bash
+cd frontend
+node scripts/migrate-supabase-production.mjs --preflight
+node scripts/migrate-supabase-production.mjs --apply
+node scripts/migrate-supabase-production.mjs --verify
+```
+
+`--apply` is intentionally one-shot: it refuses a populated target. If apply fails, discard or restore the isolated target, correct the cause, and rerun from an empty migrated schema rather than improvising a partial resume.
+
 Do not import platform schemas, source roles/grants, Supabase migration ledgers, sessions, refresh tokens, or provider tokens. Do not use a full-database `pg_restore` against the target.
 
 Because migrated users have no portable password/session, all sessions are invalidated and users sign in again through email OTP. Administrators re-enrol MFA.
@@ -104,7 +128,7 @@ Because migrated users have no portable password/session, all sessions are inval
 
 Complete at least one isolated full-data rehearsal before scheduling the final window:
 
-1. Apply all target schema migrations to an empty rehearsal database.
+1. Apply all target schema migrations to an empty rehearsal database using the same schema migrator path as `Migrate Production Database`.
 2. Run migration preflight, apply, post-import reconciliation, and verify.
 3. Verify source/target row counts, primary-key set hashes, normalized row hashes, credit totals, payment state totals, subscriptions, reports, consultations, and rectification records.
 4. Verify one Owner exists, every active admin has a target role, and database roles remain isolated.
@@ -135,6 +159,8 @@ Both `jyotisha.chat` and `admin.jyotisha.chat` are required. The application rej
 - Confirm the exact release SHA is deployed and accepted on staging.
 - Run the manual release quality gate for that SHA.
 - Confirm final backup capacity, restore rehearsal, SMTP/OTP delivery, and rollback contacts.
+- Create and restore-verify a production recovery point no more than 24 hours before the schema migration; record its non-sensitive reference and UTC creation time.
+- Dispatch `Migrate Production Database` for the accepted SHA with that recovery attestation and confirm its post-check reports no pending schema migrations.
 - Record pending payment orders and long-running jobs; choose an explicit disposition for each.
 - Dispatch `Deploy production` with `verification_mode=internal` only after target schema/data preparation. This verifies the new host without depending on public DNS.
 
@@ -189,7 +215,9 @@ Normal release:
 
 1. Merge the reviewed `staging` release into `main` so both heads are the same SHA.
 2. Confirm the staging push gate, public staging SHA, and manual release gate all succeeded for that SHA.
-3. Open Gitea Actions → `Deploy production`.
-4. Enter the exact 40-character SHA, leave `allow_rollback=false`, and choose `internal` or `public` for the current cutover phase.
+3. Open Gitea Actions → `Migrate Production Database`; enter the exact 40-character SHA, the no-more-than-24-hour-old recovery reference and UTC creation time, and confirm `restore_verified=true`. Wait for the post-migration checker to converge. Do not use this workflow for Supabase ETL.
+4. Run the trusted-host ETL phases and retain the redacted reconciliation manifests.
+5. Open Gitea Actions → `Deploy production`.
+6. Enter the same exact SHA, leave `allow_rollback=false`, and choose `internal` or `public` for the current cutover phase.
 
 Application rollback accepts only an explicitly authorized, previously gate-attested SHA in reviewed `main` history. Database migrations and imported data are not rolled back by the application workflow.

docs/research/pre_work_error_ledger.md

@@ -251,3 +251,9 @@ Prevention: keep evidence-request, life-event, private-candidate, public-recap,
 After accumulated historical evidence produced a very narrow winning segment, that segment could contain fewer than two linked samples or discriminating divisional themes. Packet construction treated this valid “not enough distinction yet” state as a dependency failure, so a later answer returned 503 even though scoring and the astrology service were healthy.
 
 Prevention: classify insufficient candidate-range discrimination explicitly; when a newly narrowed segment cannot support the technical evidence contract, retain the prior candidate range, preserve scored evidence, clear the unconfirmed result, and continue conversational collection.
+
+## ERR-102 | Gitea returned HTTP 502 during final remote synchronization check | active 2026-08-09
+
+Two final `git fetch origin --prune` attempts against the configured primary Gitea remote failed before ref exchange with HTTP `502`. The last locally verified refs remain available, but this run cannot prove that they are still current and must not claim a completed remote synchronization or push.
+
+Prevention: retry fetch and `git ls-remote` before any push or release action, compare the full `main`, `staging`, and migration-branch SHAs, and stop if Gitea remains unavailable. Do not substitute cached refs, the GitHub mirror, or a successful local commit for current Gitea synchronization evidence.