root/Jyotisha
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(staging): switch to local postgres

Browse Source
This commit is contained in:
Jesse_Chen
2026-07-22 11:04:32 +08:00
parent f5c57efeba
commit a1b8eaec36
30 changed files with 1011 additions and 141 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy/.env.staging.identity.example
+7 -3
View File
@@ -5,14 +5,18 @@ CADDYFILE_PATH=./Caddyfile.staging
SITE_ADDRESS=https://staging.jyotisha.chat
ADMIN_SITE_ADDRESS=https://admin.staging.jyotisha.chat
# Coexistence mode: the public site still uses Supabase-backed business routes,
# while the self-hosted identity API and the admin-host login can be tested.
AUTH_PROVIDER=supabase
# Staging-only cutover: identity and business data both use the private local
# PostgreSQL service. Production remains on Supabase until a separate cutover.
AUTH_PROVIDER=self-hosted
SELF_HOSTED_IDENTITY_ENABLED=true
AUTH_USER_ORIGIN=https://staging.jyotisha.chat
AUTH_ADMIN_ORIGIN=https://admin.staging.jyotisha.chat
IDENTITY_DATABASE_URL=postgresql://identity_runtime:<percent-encoded-identity-runtime-password>@postgres:5432/jyotisha
APP_DATABASE_URL=postgresql://app_runtime:<percent-encoded-app-runtime-password>@postgres:5432/jyotisha
ADMIN_DATABASE_URL=postgresql://admin_runtime:<percent-encoded-admin-runtime-password>@postgres:5432/jyotisha
BETTER_AUTH_USER_SECRET=<independent-openssl-rand-base64-32-output>
BETTER_AUTH_ADMIN_SECRET=<different-openssl-rand-base64-32-output>
RESEND_API_KEY=<staging-only-resend-api-key>
RESEND_FROM_EMAIL=Jyotisha Staging <login@staging.jyotisha.chat>
ADMIN_EMAILS=<comma-separated-staging-admin-emails>
JYOTISH_DYNAMIC_RECTIFICATION_TOKEN=<independent-openssl-rand-base64-32-output>
deploy/Caddyfile.staging
+9 -2
View File
@@ -1,13 +1,20 @@
{$SITE_ADDRESS:https://staging.jyotisha.chat} {
encode zstd gzip
@adminPaths path /admin /admin/* /api/admin/*
respond @adminPaths "Not found" 404
reverse_proxy web:3000
}
{$ADMIN_SITE_ADDRESS:https://admin.staging.jyotisha.chat} {
encode zstd gzip
@identity path /login /api/auth/* /_next/* /jyotish-logo.png /favicon.ico
handle @identity {
@adminRoot path /
redir @adminRoot /admin/codes 302
@adminSurface path /login /admin /admin/* /api/admin/* /api/auth/* /_next/* /jyotish-logo.png /favicon.ico
handle @adminSurface {
reverse_proxy web:3000
}
respond "Not found" 404
deploy/README.md
+6 -7
View File
@@ -152,12 +152,11 @@ Staging is isolated from production:
| Runtime app env | `/opt/jyotisha-staging/.env.staging` (`0600`) |
| Runtime database env | `/opt/jyotisha-staging/.env.staging.database` (`0600`) |
| PostgreSQL | private Compose network; no published host port |
| Supabase | separate `Jyotisha Staging` project |
| Business database | local private PostgreSQL (`jyotisha-staging` Compose project) |
| Identity | Better Auth + Resend OTP on the same private PostgreSQL cluster |
| GitHub Environment | `staging` |
The GitHub `staging` Environment contains the secret `STAGING_SSH_PRIVATE_KEY` and the variables `STAGING_HOST`, `STAGING_PORT`, `STAGING_USER`, `STAGING_PATH`, `STAGING_URL`, and `STAGING_KNOWN_HOSTS`. Its deployment branch policy allows the `main` controller branch: GitHub's `workflow_run` event executes from the default branch while the workflow separately requires the successfully tested upstream branch to be `staging`. The controller checks out only `main` with full history, requires the requested staging SHA to be an ancestor of that reviewed history, and uploads only the allowlisted `deploy/` control files. It never executes deployment validators or remote orchestration scripts from the target/rollback revision. The staging key, database, Supabase keys, and model-provider keys must not be shared with production.
The repository-level public build inputs are configured at GitHub **Settings -> Secrets and variables -> Actions -> Variables** (the UI is also shown as **Settings → Secrets and variables → Actions → Variables**): `STAGING_SUPABASE_URL` and `STAGING_SUPABASE_ANON_KEY`. They are public build inputs, required for publish, and exposed to the browser; keep them staging-only and never print their values in workflow output, summaries, or support messages. The workflow passes them only as the `NEXT_PUBLIC_*` build arguments after non-empty/HTTPS validation.
The GitHub `staging` Environment contains the secret `STAGING_SSH_PRIVATE_KEY` and the variables `STAGING_HOST`, `STAGING_PORT`, `STAGING_USER`, `STAGING_PATH`, `STAGING_URL`, and `STAGING_KNOWN_HOSTS`. Its deployment branch policy allows the `main` controller branch: GitHub's `workflow_run` event executes from the default branch while the workflow separately requires the successfully tested upstream branch to be `staging`. The controller checks out only `main` with full history, requires the requested staging SHA to be an ancestor of that reviewed history, and uploads only the allowlisted `deploy/` control files. It never executes deployment validators or remote orchestration scripts from the target/rollback revision. The staging key, database, Resend key, and model-provider keys must not be shared with production. Staging image publishing has no Supabase build variables.
`Staging Backend Quality Gate` runs for relevant `pull_request` paths, pushes to `staging`, and `workflow_dispatch`. It validates the Python/database/frontend contract; only a successful push to `staging` publishes the API/web images and a run-bound manifest containing their `sha256` digests. `.github/workflows/deploy-staging.yml` consumes that exact successful run, validates its manifest against the full 40-character commit, and deploys digest references rather than trusting the discoverability tags.
@@ -170,13 +169,13 @@ SITE_ADDRESS=https://staging.jyotisha.chat
ADMIN_SITE_ADDRESS=https://admin.staging.jyotisha.chat
```
The self-hosted identity milestone runs in coexistence mode: keep `AUTH_PROVIDER=supabase` and set `SELF_HOSTED_IDENTITY_ENABLED=true`. Add the server-only identity database, separate user/admin Better Auth secrets, origins, and staging-only Resend settings listed in `deploy/.env.staging.identity.example`. The public login remains on Supabase while the admin host and identity API are exercised. Do not set `AUTH_PROVIDER=self-hosted` until the Supabase-backed business modules have migrated. See `docs/operations/self-hosted-identity.md` for validation, import, smoke, and rollback commands.
Staging is fully self-hosted: set `AUTH_PROVIDER=self-hosted` and `SELF_HOSTED_IDENTITY_ENABLED=true`. Add the three role-specific server-only database URLs, separate user/admin Better Auth secrets, origins, and staging-only Resend settings listed in `deploy/.env.staging.identity.example`. Browser code uses same-origin APIs; it receives neither database credentials nor Supabase keys. Production remains on Supabase and is not changed by the staging workflow. See `docs/operations/self-hosted-identity.md` for validation and rollback commands.
After source sync and before `up`, the workflow validates `.env.staging` mode/selectors, explicitly pins the three staging selectors against ambient shell overrides, and runs `docker compose --env-file .env.staging -f deploy/docker-compose.server.yml config --quiet`. For later manual inspections, run the same checks only after the tracked deployment files exist on the server. Do not use a manual gate run from `main` as the first publishing path: publishing requires a successful push to `staging`, while manual `Deploy staging` requires a successful gate run for the exact SHA.
### First-deploy sequence
1. Complete the server and GitHub bootstrap: create both mode-`0600` env files, preload the reviewed `postgres:17-alpine` image, configure the staging Environment variables/secrets, and configure the repository staging build variables. Deployment and migration workflows use `--pull never` for PostgreSQL, so database image upgrades remain an explicit operator-controlled maintenance action rather than an application-deploy side effect.
1. Complete the server and GitHub bootstrap: create both mode-`0600` env files, preload the reviewed `postgres:17-alpine` image, and configure the staging Environment variables/secrets. No repository-level Supabase variables are required. Deployment and migration workflows use `--pull never` for PostgreSQL, so database image upgrades remain an explicit operator-controlled maintenance action rather than an application-deploy side effect.
2. Merge the reviewed change to `main`, then fast-forward/push that exact reviewed SHA to `staging`; do not create a staging-only target or rely on a `main` workflow dispatch to publish images.
3. The `Staging Backend Quality Gate` runs for that push and, when successful, publishes API/web images plus an artifact binding the exact SHA to both immutable image digests.
4. The automatic `Deploy staging` workflow downloads that gate-run artifact, syncs only the trusted `main` controller's allowlisted `deploy/` files under the shared staging host lock, and validates both `.env.staging` and `.env.staging.database` before any app change. The target application's code is carried only by the digest-pinned images.
@@ -196,7 +195,7 @@ docker compose --env-file .env.staging -f deploy/docker-compose.server.yml logs
curl -fsS https://staging.jyotisha.chat/api/health
```
The normal application deployment workflow never runs database migrations. Apply migrations to the separate staging project first, verify them, and only then deploy application code that depends on them.
The normal application deployment workflow never runs database migrations. Apply migrations to the private staging PostgreSQL cluster first, verify them, and only then deploy application code that depends on them.
## Staging PostgreSQL operations
deploy/docker-compose.server.yml
+2 -2
View File
@@ -26,8 +26,8 @@ services:
context: ..
dockerfile: deploy/railway-web.Dockerfile
args:
NEXT_PUBLIC_SUPABASE_URL: ${NEXT_PUBLIC_SUPABASE_URL}
NEXT_PUBLIC_SUPABASE_ANON_KEY: ${NEXT_PUBLIC_SUPABASE_ANON_KEY}
NEXT_PUBLIC_SUPABASE_URL: ${NEXT_PUBLIC_SUPABASE_URL:-}
NEXT_PUBLIC_SUPABASE_ANON_KEY: ${NEXT_PUBLIC_SUPABASE_ANON_KEY:-}
restart: unless-stopped
env_file:
- ${APP_ENV_FILE:-../.env.production}
deploy/postgres/001-bootstrap-roles.sh
+11
View File
@@ -48,6 +48,17 @@ SELECT format(
) WHERE NOT EXISTS (
SELECT 1 FROM pg_roles WHERE rolname = 'admin_runtime'
) \gexec
SELECT 'CREATE ROLE anon NOLOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT'
WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'anon') \gexec
SELECT 'CREATE ROLE authenticated NOLOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT'
WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'authenticated') \gexec
SELECT 'CREATE ROLE service_role NOLOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT BYPASSRLS'
WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'service_role') \gexec
ALTER ROLE service_role BYPASSRLS;
GRANT authenticated TO app_runtime;
GRANT service_role TO admin_runtime;
SELECT format(
'CREATE ROLE migration_runner WITH LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT PASSWORD %L',
:'migration_runner_password'
deploy/postgres/002-ensure-business-compatibility-roles.sql
+11
View File
@@ -0,0 +1,11 @@
select 'create role anon nologin nosuperuser nocreatedb nocreaterole noinherit'
where not exists (select 1 from pg_roles where rolname = 'anon') \gexec
select 'create role authenticated nologin nosuperuser nocreatedb nocreaterole noinherit'
where not exists (select 1 from pg_roles where rolname = 'authenticated') \gexec
select 'create role service_role nologin nosuperuser nocreatedb nocreaterole noinherit bypassrls'
where not exists (select 1 from pg_roles where rolname = 'service_role') \gexec
alter role service_role bypassrls;
grant authenticated to app_runtime;
grant service_role to admin_runtime;
deploy/railway-web.Dockerfile
+1
View File
@@ -9,6 +9,7 @@ COPY frontend/public ./public
COPY frontend/next.config.ts frontend/postcss.config.mjs frontend/tsconfig.json ./
COPY frontend/scripts ./scripts
COPY frontend/db ./db
COPY frontend/supabase/migrations ./supabase/migrations
ARG NEXT_PUBLIC_SUPABASE_URL
ARG NEXT_PUBLIC_SUPABASE_ANON_KEY
deploy/run-staging-migration.sh
+2
View File
@@ -74,6 +74,8 @@ export DATABASE_ENV_FILE='../.env.staging.database'
compose=(docker compose -p jyotisha-staging -f deploy/docker-compose.postgres.yml)
docker pull "$WEB_IMAGE"
"${compose[@]}" up -d --no-build --pull never --wait postgres
"${compose[@]}" exec -T postgres psql -v ON_ERROR_STOP=1 -U postgres -d jyotisha \
-f /dev/stdin < deploy/postgres/002-ensure-business-compatibility-roles.sql
"${compose[@]}" --profile migration run --rm migrator
"${compose[@]}" exec -T postgres psql -U postgres -d jyotisha -Atc \
'select filename from migration.schema_migrations order by filename'
deploy/validate-staging-env.sh
+21 -1
View File
@@ -42,7 +42,7 @@ require_selector APP_ENV_FILE ../.env.staging
require_selector CADDYFILE_PATH ./Caddyfile.staging
require_selector SITE_ADDRESS https://staging.jyotisha.chat
require_selector ADMIN_SITE_ADDRESS https://admin.staging.jyotisha.chat
require_selector AUTH_PROVIDER supabase
require_selector AUTH_PROVIDER self-hosted
require_selector SELF_HOSTED_IDENTITY_ENABLED true
require_selector AUTH_USER_ORIGIN https://staging.jyotisha.chat
require_selector AUTH_ADMIN_ORIGIN https://admin.staging.jyotisha.chat
@@ -73,6 +73,20 @@ if ! [[ "$identity_database_url" =~ ^postgresql://identity_runtime:([A-Za-z0-9._
exit 1
fi
require_literal APP_DATABASE_URL 45
app_database_url="$LITERAL_VALUE"
if ! [[ "$app_database_url" =~ ^postgresql://app_runtime:([A-Za-z0-9._~-]|%[0-9A-Fa-f]{2})+@postgres:5432/jyotisha$ ]]; then
echo "invalid staging database setting: APP_DATABASE_URL" >&2
exit 1
fi
require_literal ADMIN_DATABASE_URL 45
admin_database_url="$LITERAL_VALUE"
if ! [[ "$admin_database_url" =~ ^postgresql://admin_runtime:([A-Za-z0-9._~-]|%[0-9A-Fa-f]{2})+@postgres:5432/jyotisha$ ]]; then
echo "invalid staging database setting: ADMIN_DATABASE_URL" >&2
exit 1
fi
require_literal BETTER_AUTH_USER_SECRET 32
user_secret="$LITERAL_VALUE"
require_literal BETTER_AUTH_ADMIN_SECRET 32
@@ -87,5 +101,11 @@ if [[ "$LITERAL_VALUE" != *@* ]]; then
echo "invalid staging identity setting: RESEND_FROM_EMAIL" >&2
exit 1
fi
require_literal ADMIN_EMAILS 3
if [[ "$LITERAL_VALUE" != *@* ]]; then
echo "invalid staging identity setting: ADMIN_EMAILS" >&2
exit 1
fi
require_literal JYOTISH_DYNAMIC_RECTIFICATION_TOKEN 32
echo "staging environment selectors: valid"