root/Jyotisha
2
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

ops: automate production recovery attestation
Independent Staging Quality Gate / validate (push) Successful in 13m3s
Details
Independent Staging Quality Gate / publish (push) Successful in 2m18s
Details

Browse Source
This commit is contained in:
Jesse_Chen
2026-08-16 01:03:10 +08:00
parent 0d59d51814
commit 934c4175d3
6 changed files with 410 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/operations/production-server-migration-2026-08.md
+9 -6
View File
@@ -90,7 +90,9 @@ Use distinct production credentials for PostgreSQL roles, Better Auth, Resend, b
## Database migration engineering gate
Before importing data, dispatch Gitea Actions → `Migrate Production Database` for the exact accepted release SHA. The workflow requires `main == staging == deploy_sha`, the same successful staging backend gate, the same manual release gate, and the public staging `/api/health` identity for that SHA. It also requires a non-sensitive recovery reference, its exact UTC creation time, and `restore_verified=true`; the recovery point must be no more than 24 hours old and must already have passed a restore verification. It verifies the current production revision, obtains the gate-attested immutable Web image, runs the schema checker, applies only pending application schema migrations, and requires the checker to converge afterward.
Before importing data, dispatch Gitea Actions → `Create Production Recovery Point` for the exact accepted release SHA. It creates an AES-256-CBC/PBKDF2 encrypted custom-format dump on the production host, restores it into a disposable database, verifies non-sensitive table counts, removes only the disposable database, and uploads the encrypted dump plus verification manifest as a 30-day Gitea Actions artifact. The workflow prints the non-sensitive `recovery_reference` and exact UTC `recovery_created_at`; use those values only after the artifact upload succeeds.
Then dispatch Gitea Actions → `Migrate Production Database` for the same exact accepted release SHA. The workflow requires `main == staging == deploy_sha`, the same successful staging backend gate, the same manual release gate, and the public staging `/api/health` identity for that SHA. It also requires the recovery workflow's non-sensitive reference, exact UTC creation time, and `restore_verified=true`; the recovery point must be no more than 24 hours old and must already have passed the restore verification. It verifies the current production revision, obtains the gate-attested immutable Web image, runs the schema checker, applies only pending application schema migrations, and requires the checker to converge afterward.
Schema migration files are committed sequentially and are not one atomic transaction as a set. If a later file or post-check fails, earlier files may remain applied; stop, preserve evidence, and restore from the attested recovery point when repair-in-place is not explicitly reviewed. Do not assume a failed workflow means the database is unchanged.
@@ -172,7 +174,7 @@ Both `jyotisha.chat` and `admin.jyotisha.chat` are required. The application rej
- Confirm the exact release SHA is deployed and accepted on staging.
- Run the manual release quality gate for that SHA.
- Confirm final backup capacity, restore rehearsal, SMTP/OTP delivery, and rollback contacts.
- Create and restore-verify a production recovery point no more than 24 hours before the schema migration; record its non-sensitive reference and UTC creation time.
- Run `Create Production Recovery Point` no more than 24 hours before the schema migration; retain its encrypted off-site artifact and record the printed non-sensitive reference and UTC creation time.
- Dispatch `Migrate Production Database` for the accepted SHA with that recovery attestation and confirm its post-check reports no pending schema migrations.
- Record pending payment orders and long-running jobs; choose an explicit disposition for each.
- Dispatch `Deploy production` with `verification_mode=internal` only after target schema/data preparation. This verifies the new host without depending on public DNS.
@@ -228,9 +230,10 @@ Normal release:
1. Merge the reviewed `staging` release into `main` so both heads are the same SHA.
2. Confirm the staging push gate, public staging SHA, and manual release gate all succeeded for that SHA.
3. Open Gitea Actions → `Migrate Production Database`; enter the exact 40-character SHA, the no-more-than-24-hour-old recovery reference and UTC creation time, and confirm `restore_verified=true`. Wait for the post-migration checker to converge. Do not use this workflow for Supabase ETL.
4. Run the trusted-host ETL phases and retain the redacted reconciliation manifests.
5. Open Gitea Actions → `Deploy production`.
6. Enter the same exact SHA, leave `allow_rollback=false`, and choose `internal` or `public` for the current cutover phase.
3. Open Gitea Actions → `Create Production Recovery Point`; enter the exact 40-character SHA and wait for the encrypted artifact upload plus restore verification to succeed. Record the printed `recovery_reference` and `recovery_created_at`.
4. Open Gitea Actions → `Migrate Production Database`; enter the same SHA and the no-more-than-24-hour-old recovery attestation, then confirm `restore_verified=true`. Wait for the post-migration checker to converge. Do not use this workflow for Supabase ETL.
5. Run the trusted-host ETL phases and retain the redacted reconciliation manifests when a data cutover is actually required. Ordinary forward schema releases do not rerun the one-shot Supabase ETL.
6. Open Gitea Actions → `Deploy production`.
7. Enter the same exact SHA, leave `allow_rollback=false`, and choose `internal` or `public` for the current cutover phase.
Application rollback accepts only an explicitly authorized, previously gate-attested SHA in reviewed `main` history. Database migrations and imported data are not rolled back by the application workflow.