diff --git a/deploy/README.md b/deploy/README.md index 3d729fc5..ce3d7cd5 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -139,13 +139,17 @@ Staging is isolated from production: | URL | `https://staging.jyotisha.chat` | | Host | `118.26.111.127` | | Path | `/opt/jyotisha-staging` | -| Runtime env | `/opt/jyotisha-staging/.env.staging` (`0600`) | +| Runtime app env | `/opt/jyotisha-staging/.env.staging` (`0600`) | +| Runtime database env | `/opt/jyotisha-staging/.env.staging.database` (`0600`) | +| PostgreSQL | private Compose network; no published host port | | Supabase | separate `Jyotisha Staging` project | | GitHub Environment | `staging` | -The GitHub Environment contains `STAGING_SSH_PRIVATE_KEY` and the variables `STAGING_HOST`, `STAGING_PORT`, `STAGING_USER`, `STAGING_PATH`, `STAGING_URL`, and `STAGING_KNOWN_HOSTS`. Its deployment branch policy allows the `main` controller branch: GitHub's `workflow_run` event executes from the default branch while the workflow separately requires the successfully tested upstream branch to be `staging`. The staging key, database, Supabase keys, and model-provider keys must not be shared with production. +The GitHub `staging` Environment contains the secret `STAGING_SSH_PRIVATE_KEY` and the variables `STAGING_HOST`, `STAGING_PORT`, `STAGING_USER`, `STAGING_PATH`, `STAGING_URL`, and `STAGING_KNOWN_HOSTS`. Its deployment branch policy allows the `main` controller branch: GitHub's `workflow_run` event executes from the default branch while the workflow separately requires the successfully tested upstream branch to be `staging`. The staging key, database, Supabase keys, and model-provider keys must not be shared with production. -A push to branch `staging` runs `Jyotish Skill CI`. A successful push run triggers `.github/workflows/deploy-staging.yml`, which deploys the tested SHA and verifies the login route, logged-out account response, deployment SHA, and private Python health endpoint. +The repository-level public build inputs are configured at GitHub **Settings -> Secrets and variables -> Actions -> Variables** (the UI is also shown as **Settings → Secrets and variables → Actions → Variables**): `STAGING_SUPABASE_URL` and `STAGING_SUPABASE_ANON_KEY`. They are public build inputs, required for publish, and exposed to the browser; keep them staging-only and never print their values in workflow output, summaries, or support messages. The workflow passes them only as the `NEXT_PUBLIC_*` build arguments after non-empty/HTTPS validation. + +`Staging Backend Quality Gate` runs for `pull_request`, pushes to `staging`, and `workflow_dispatch`. It validates the Python/database/frontend contract; only a successful push to `staging` can publish immutable full-SHA GHCR images. `.github/workflows/deploy-staging.yml` consumes the successful gate's exact SHA, and its manual `deploy_sha` input must identify a full 40-character commit with a successful `staging` gate run. The staging env file must include these non-secret selectors so Compose cannot fall back to production paths: @@ -158,13 +162,13 @@ SITE_ADDRESS=https://staging.jyotisha.chat After source sync and before `up`, the workflow validates `.env.staging` mode/selectors, explicitly pins the three staging selectors against ambient shell overrides, and runs `docker compose --env-file .env.staging -f deploy/docker-compose.server.yml config --quiet`. For later manual inspections, run the same checks only after the tracked deployment files exist on the server. The first deployment should be manual: 1. Confirm `/opt/jyotisha-staging/.env.staging` exists, has mode `0600`, and contains the three selectors above. -2. Open GitHub Actions -> Jyotish Skill CI -> Run workflow, using workflow from `main`. +2. Open GitHub Actions -> Staging Backend Quality Gate -> Run workflow, using workflow from `main`. 3. Wait for success and copy that run's exact 40-character commit SHA. -4. Open GitHub Actions -> Deploy staging -> Run workflow, using workflow from `main`, and enter the SHA in `git_sha`. +4. Open GitHub Actions -> Deploy staging -> Run workflow, using workflow from `main`, and enter the SHA in `deploy_sha`. 5. Confirm `https://staging.jyotisha.chat/api/health` reports that SHA. 6. Only after the manual deployment passes, push a reviewed revision to branch `staging` to validate automatic deployment. -Application rollback uses the same workflow: manually dispatch `Deploy staging` from `main` with a previous known-good full SHA that has a successful CI run. Database migrations are separate and are not rolled back by an application deployment. Restore a staging database backup before running any destructive migration rehearsal. +Application rollback uses the same workflow: manually dispatch `Deploy staging` from `main` with a previous known-good full SHA that has a successful `Staging Backend Quality Gate` run. Database migrations are separate and are not rolled back by an application deployment. Restore a staging database backup before running any destructive migration rehearsal. Inspect staging without printing secrets: @@ -178,6 +182,111 @@ curl -fsS https://staging.jyotisha.chat/api/health The normal application deployment workflow never runs database migrations. Apply migrations to the separate staging project first, verify them, and only then deploy application code that depends on them. +## Staging PostgreSQL operations + +This section is the server-side runbook for the disposable staging PostgreSQL volume. It does not replace the production instructions above. + +### Bootstrap and environment-file boundary + +SSH to the staging host as the deployment user and create both environment files with a restrictive umask. The application file and the database file are separate, both are mode `0600`, the database file is owned by the deployment user, and neither is committed or copied through `rsync`: + +```bash +cd /opt/jyotisha-staging +umask 077 +touch .env.staging +chmod 600 .env.staging +touch .env.staging.database +chmod 600 .env.staging.database +``` + +`.env.staging` contains application selectors and server-only application credentials. `SCHEMA_DATABASE_URL` must not appear in `.env.staging`; neither may any database bootstrap password, `STAGING_BACKUP_ENCRYPTION_KEY`, or migration-runner credential. In particular, there is no `SCHEMA_DATABASE_URL` in `.env.staging`; the schema URL exists only in `.env.staging.database`, which is read by PostgreSQL and the opt-in migrator. + +Generate every `` value from independently generated 32 random bytes (for example, run `openssl rand -base64 32` separately for each value and place it directly into the mode-`0600` file or an approved secret store). Do not reuse a password between roles, paste values into chat, commit either file, or print them in workflow logs. The schema-owner password in `SCHEMA_DATABASE_URL` is the same secret as `SCHEMA_OWNER_PASSWORD`; use a percent-encoded URL password component only, and do not encode the scheme, host, port, or database name. + +The exact database keys are: + +```dotenv +POSTGRES_DB=jyotisha +POSTGRES_USER=postgres +POSTGRES_PASSWORD= +SCHEMA_OWNER_PASSWORD= +IDENTITY_RUNTIME_PASSWORD= +APP_RUNTIME_PASSWORD= +ADMIN_RUNTIME_PASSWORD= +MIGRATION_RUNNER_PASSWORD= +BACKUP_READER_PASSWORD= +STAGING_BACKUP_ENCRYPTION_KEY= +SCHEMA_DATABASE_URL=postgresql://schema_owner:@postgres:5432/jyotisha +``` + +PostgreSQL is private: `deploy/docker-compose.postgres.yml` has no `ports` mapping, so the staging database is reachable only on the Docker `app` network. The CI overlay is the only host binding and is loopback-only (`127.0.0.1:${POSTGRES_HOST_PORT:-55432}:5432`); do not add a public database port, firewall exception, or browser-facing SQL tool. Normal web/API containers never receive `SCHEMA_DATABASE_URL`. + +### Exact deployment and migration order + +Use this order for every staging revision: + +1. Merge to `staging` after reviewing the change. +2. Wait for `Staging Backend Quality Gate` to pass and for that exact full SHA's API/web images to be published. +3. The automatic `Deploy staging` workflow checks the exact SHA in read-only migration-check mode before changing API, web, or Caddy. If it reports pending or drifted migrations, stop; do not retry the application deployment as if it were a migration. +4. Open **Migrate Staging Database -> Run workflow** and enter the reported full lowercase 40-character SHA in `deploy_sha`. The workflow validates that exact SHA against a successful `staging` gate, checks it out, starts only PostgreSQL, and runs the reviewed migrator. +5. A successful migration prints the ordered migration ledger and re-dispatches `Deploy staging` automatically with the same exact SHA. Do not substitute a branch name, a short SHA, or a newer commit. +6. Confirm `https://staging.jyotisha.chat/api/health` and verify that its deployment SHA is the SHA from step 2. +7. After health verification, create the local encrypted backup described below. + +The read-only checker exits before app changes when a migration is pending. Its message includes the exact SHA and the `Migrate Staging Database` workflow name. A failed migration does not re-dispatch deployment. Application rollback restores a previously verified image/SHA only; it does not roll back database state. + +### Local encrypted staging backups (three-copy limit) + +After the health check, run the repository backup helper from the synchronized staging checkout: + +```bash +cd /opt/jyotisha-staging +./deploy/backup-staging-postgres.sh \ + .env.staging.database \ + /opt/jyotisha-staging/backups/staging-db +``` + +The helper invokes `pg_dump --format=custom --no-owner` in the PostgreSQL container and encrypts the stream with `openssl enc -aes-256-cbc -salt -pbkdf2 -pass env:STAGING_BACKUP_ENCRYPTION_KEY`. It creates mode-`0600` `.dump.enc` files in a mode-`0700` directory, refuses disk usage at or above 70%, publishes atomically, and retains only the newest three encrypted local backups. The encryption passphrase is supplied through the environment, never as a command-line argument or printed value. Keep the archive directory on this staging VPS only; there is no off-site staging recovery and no off-site staging backup. These three local encrypted copies are rehearsal/rollback aids, not disaster-recovery backups. + +### Restore drill into a disposable database + +Run a restore drill only against the disposable `jyotisha_restore_check` database. Choose one archive and use a temporary decrypted custom-format dump; the commands below match the backup helper's AES-256-CBC/PBKDF2 and `pg_dump --format=custom` interfaces: + +```bash +cd /opt/jyotisha-staging +export DATABASE_ENV_FILE=../.env.staging.database +BACKUP_FILE=/opt/jyotisha-staging/backups/staging-db/.dump.enc +RESTORE_DUMP="$(mktemp /tmp/jyotisha-staging-restore.XXXXXX.dump)" +chmod 600 "$RESTORE_DUMP" +trap 'rm -f -- "$RESTORE_DUMP"' EXIT +read -r -s -p 'Backup passphrase: ' STAGING_BACKUP_ENCRYPTION_KEY +printf '\n' >&2 +export STAGING_BACKUP_ENCRYPTION_KEY + +openssl enc -d -aes-256-cbc -pbkdf2 \ + -pass env:STAGING_BACKUP_ENCRYPTION_KEY \ + -in "$BACKUP_FILE" -out "$RESTORE_DUMP" + +docker compose -p jyotisha-staging -f deploy/docker-compose.postgres.yml \ + exec -T postgres createdb -U postgres jyotisha_restore_check +docker compose -p jyotisha-staging -f deploy/docker-compose.postgres.yml \ + exec -T postgres pg_restore -U postgres --no-owner --exit-on-error \ + --dbname=jyotisha_restore_check < "$RESTORE_DUMP" + +# Inspect the restored disposable database, then remove only the drill target. +docker compose -p jyotisha-staging -f deploy/docker-compose.postgres.yml \ + exec -T postgres dropdb -U postgres --if-exists jyotisha_restore_check +rm -f -- "$RESTORE_DUMP" +trap - EXIT +unset STAGING_BACKUP_ENCRYPTION_KEY +``` + +The passphrase is read silently into an environment variable; do not put it in argv, shell history, logs, or support messages. The cleanup scope is deliberately narrow: delete only `jyotisha_restore_check` and the temporary decrypted dump. Do not run `docker compose down`, `down -v`, `dropdb jyotisha`, volume deletion, or archive deletion as part of this drill. If restore fails, preserve the encrypted archive and PostgreSQL volume for inspection, remove only the temporary dump, and investigate before retrying. + +### Staging/production boundary + +This disposable staging procedure does not authorize a production migration, production backup policy, production database replacement, domain switch, Supabase deletion, or production cutover. Production deployment and migration remain manual-only and require a separate reviewed approval, off-site encrypted backups, and a successful production restore drill. Keep the production `.env.production` and all production credentials on the production host; never copy them into staging. + ## Manual deployment fallback If GitHub Actions is unavailable, deploy the tracked tree without copying local secrets: diff --git a/frontend/tests/staging-backend-workflows.test.ts b/frontend/tests/staging-backend-workflows.test.ts index a11bcca5..23fc937b 100644 --- a/frontend/tests/staging-backend-workflows.test.ts +++ b/frontend/tests/staging-backend-workflows.test.ts @@ -42,6 +42,10 @@ const migrationWorkflowUrl = new URL( "../../.github/workflows/migrate-staging-database.yml", import.meta.url, ); +const deploymentReadmeUrl = new URL( + "../../deploy/README.md", + import.meta.url, +); function indentation(line: string): number { return line.match(/^ */)?.[0].length ?? 0; @@ -202,6 +206,114 @@ function logicalShellLines(script: string): string[] { .filter(Boolean); } +test("operations runbook documents the staging database boundary and deployment order", () => { + const readme = readFileSync(deploymentReadmeUrl, "utf8"); + + assert.match(readme, /Staging PostgreSQL operations/); + assert.match(readme, /\/opt\/jyotisha-staging\/\.env\.staging` \(`?0600`?\)/); + assert.match(readme, /\/opt\/jyotisha-staging\/\.env\.staging\.database` \(`?0600`?\)/); + assert.match(readme, /umask 077[\s\S]*touch \.env\.staging\.database[\s\S]*chmod 600 \.env\.staging\.database/); + + for (const key of [ + "POSTGRES_DB=jyotisha", + "POSTGRES_USER=postgres", + "POSTGRES_PASSWORD=", + "SCHEMA_OWNER_PASSWORD=", + "IDENTITY_RUNTIME_PASSWORD=", + "APP_RUNTIME_PASSWORD=", + "ADMIN_RUNTIME_PASSWORD=", + "MIGRATION_RUNNER_PASSWORD=", + "BACKUP_READER_PASSWORD=", + "STAGING_BACKUP_ENCRYPTION_KEY=", + "SCHEMA_DATABASE_URL=postgresql://schema_owner:@postgres:5432/jyotisha", + ]) { + assert.match(readme, new RegExp(key.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"))); + } + assert.match(readme, /independently generated 32 random bytes/); + assert.match(readme, /openssl rand -base64 32/); + assert.match(readme, /percent-encod(?:e|ed)[^\n]*URL password/i); + assert.match(readme, /SCHEMA_DATABASE_URL[^\n]*must not[^\n]*\.env\.staging/i); + assert.match( + readme, + /there is no `?SCHEMA_DATABASE_URL`? in `?\.env\.staging`?/i, + ); + + assert.match(readme, /private[^\n]*PostgreSQL|PostgreSQL[^\n]*private/i); + assert.match(readme, /no published host port|no host port/i); + assert.match(readme, /127\.0\.0\.1:\$\{POSTGRES_HOST_PORT:-55432\}:5432/); + + for (const workflow of [ + "Staging Backend Quality Gate", + "Migrate Staging Database", + "Deploy staging", + ]) { + assert.match(readme, new RegExp(workflow)); + } + assert.match(readme, /pull_request/); + assert.match(readme, /push[^\n]*staging|staging[^\n]*push/i); + assert.match(readme, /workflow_dispatch/); + assert.match(readme, /deploy_sha/); + assert.match(readme, /exact[^\n]*40-character[^\n]*SHA/i); + assert.match(readme, /re-dispatch|redispatch/i); + assert.match(readme, /same[^\n]*SHA/i); + + for (const variable of [ + "STAGING_SUPABASE_URL", + "STAGING_SUPABASE_ANON_KEY", + "STAGING_HOST", + "STAGING_PORT", + "STAGING_USER", + "STAGING_PATH", + "STAGING_URL", + "STAGING_KNOWN_HOSTS", + ]) { + assert.match(readme, new RegExp(variable)); + } + assert.match(readme, /Settings[ ]*[→>-][ ]*Secrets and variables[ ]*[→>-][ ]*Actions[ ]*[→>-][ ]*Variables/); + assert.match(readme, /public build inputs[^\n]*required for publish/i); + assert.match(readme, /never print[^\n]*(?:values|keys)/i); + assert.match(readme, /STAGING_SSH_PRIVATE_KEY/); + + const mergeIndex = readme.indexOf("Merge to `staging`"); + const gateIndex = readme.indexOf("Staging Backend Quality Gate", mergeIndex); + const migrationIndex = readme.indexOf("Migrate Staging Database", gateIndex); + const redispatchIndex = readme.search(/re-dispatch|redispatch/i); + const healthIndex = readme.indexOf( + "https://staging.jyotisha.chat/api/health", + redispatchIndex, + ); + const backupIndex = readme.indexOf("backup-staging-postgres.sh", healthIndex); + assert.ok(mergeIndex >= 0, "runbook must state the staging merge step"); + assert.ok(gateIndex > mergeIndex, "quality gate must follow the staging merge"); + assert.ok(migrationIndex > gateIndex, "manual migration must follow the gate"); + assert.ok(redispatchIndex > migrationIndex, "migration must redispatch the same SHA"); + assert.ok(healthIndex > redispatchIndex, "health check must follow redispatch"); + assert.ok(backupIndex > healthIndex, "backup must follow health verification"); +}); + +test("operations runbook documents three encrypted local backups and a safe restore drill", () => { + const readme = readFileSync(deploymentReadmeUrl, "utf8"); + + assert.match(readme, /newest three|three[^\n]*encrypted local backups/i); + assert.match(readme, /AES-256-CBC/i); + assert.match(readme, /PBKDF2|pbkdf2/); + assert.match(readme, /custom[^\n]*format|format:[ ]*custom/i); + assert.match(readme, /no off[- ]site[^\n]*staging[^\n]*(?:recovery|backup)/i); + assert.match( + readme, + /\.\/deploy\/backup-staging-postgres\.sh[\s\S]*\.env\.staging\.database[\s\S]*\/opt\/jyotisha-staging\/backups\/staging-db/, + ); + assert.match(readme, /-pass env:STAGING_BACKUP_ENCRYPTION_KEY/); + assert.match(readme, /passphrase[^\n]*(?:argv|command line|output|history)/i); + assert.match(readme, /jyotisha_restore_check/); + assert.match(readme, /pg_restore/); + assert.match(readme, /dropdb[^\n]*jyotisha_restore_check|DROP DATABASE[^\n]*jyotisha_restore_check/i); + assert.match(readme, /temporary decrypted dump/); + assert.match(readme, /delete[^\n]*(?:only|just)[^\n]*(?:disposable database|jyotisha_restore_check)[^\n]*(?:and|,)[^\n]*temporary decrypted dump/i); + assert.match(readme, /do not[^\n]*(?:docker compose[^\n]*down|down -v)[^\n]*(?:restore|drill|staging)/i); + assert.match(readme, /does not authorize[^\n]*(?:production|cutover)|no production[^\n]*cutover[^\n]*authoriz/i); +}); + test("backend quality gate has structured staging triggers and concurrency", () => { const document = parseWorkflow();