fix(api): persist scratch/local and bound heavy compute concurrency

Keep async job and chart-cache files across API recreates, freeze jyotish_api_server.py growth, and fail fast with 429 when rectification or high-rigor compute is saturated.

Co-authored-by: Cursor <cursoragent@cursor.com>

deploy/README.md

@@ -26,12 +26,19 @@ This migration changes both infrastructure and persistence. It is a controlled S
 ```text
 Spaceship DNS -> Caddy :80/:443 -> web:3000 -> api:5200
                                   |           -> local astrology engines
+                                  |           -> named volume api_scratch -> /app/scratch/local
                                   -> private PostgreSQL 17 + Better Auth
                                   -> external model and mail providers
 ```
 
 Only Caddy publishes host ports. Ports `3000` and `5200` must remain private.
 
+The API image `WORKDIR` is `/app` (`deploy/railway-api.Dockerfile`). Runtime files written by `scripts/jyotish_api_server.py` resolve to `/app/scratch/local` (chart cache `api_chart_cache`, async job files or `async_jobs.sqlite3`, high-rigor job records). Compose mounts the named volume `api_scratch` at that path so those files survive container recreate. Docker creates the volume on first `up`; there is no data migration because the previous container overlay was ephemeral. Deploy scripts (`run-staging-deploy.sh`, `run-production-deploy.sh`) do not need extra volume flags: `docker compose up` creates and reattaches named volumes automatically.
+
+The volume is not a backup domain. Chart cache is a TTL'd compute cache (default 900s) and is cheaper to rebuild than to archive. Async / high-rigor job state is short-lived (default TTL 3600s) and is not a restore unit. Restart persistence is the only reason the volume exists. Staging encrypted backups remain PostgreSQL-only via `deploy/backup-staging-postgres.sh`; do not add `api_scratch` to that helper. Project-prefixed volume names are `jyotisha-staging_api_scratch` and `jyotisha-production_api_scratch`.
+
+Heavy rectification scans and high-rigor workflows share a process-wide fail-fast concurrency gate (`JYOTISH_HEAVY_COMPUTE_CONCURRENCY`, default `2` to match the 2 vCPU host). Saturated requests return HTTP 429 with `Retry-After` and are not queued. Health checks and other light routes do not take a slot.
+
 ## DNS and Supabase Auth
 
 Final Spaceship resource records (apply only during the approved cutover window):
@@ -280,6 +287,8 @@ cd /opt/jyotisha-staging
 
 The helper invokes `pg_dump --format=custom --no-owner` in the PostgreSQL container and encrypts the stream with `openssl enc -aes-256-cbc -salt -pbkdf2 -pass env:STAGING_BACKUP_ENCRYPTION_KEY`. It creates mode-`0600` `.dump.enc` files in a mode-`0700` directory, refuses disk usage at or above 70%, publishes atomically, and retains only the newest three encrypted local backups. The encryption passphrase is supplied through the environment, never as a command-line argument or printed value. Keep the archive directory on this staging VPS only; there is no off-site staging recovery and no off-site staging backup. These three local encrypted copies are rehearsal/rollback aids, not disaster-recovery backups.
 
+Do not back up the API `api_scratch` volume. Chart cache (`/app/scratch/local/api_chart_cache`) is a disposable TTL cache; restoring it has no user-visible correctness value. Async job records (`async_jobs` / `async_jobs.sqlite3` and `high_rigor_jobs`) expire within an hour by default and represent in-flight work, not durable product state. A staging or production restore is a PostgreSQL restore. Recreating the API container without the named volume only drops cache and in-flight jobs, which is the same loss the overlay filesystem already had before this volume existed.
+
 ### Restore drill into a disposable database
 
 Run a restore drill only against the disposable `jyotisha_restore_check` database. Choose one archive and use a temporary decrypted custom-format dump; the commands below match the backup helper's AES-256-CBC/PBKDF2 and `pg_dump --format=custom` interfaces:

deploy/docker-compose.server.yml

@@ -19,6 +19,11 @@ services:
       retries: 5
       start_period: 30s
       start_interval: 1s
+    volumes:
+      # WORKDIR is /app (deploy/railway-api.Dockerfile). Chart cache, async job
+      # files, and high-rigor job records live under scratch/local. Named volume
+      # is created on first use; previous overlay data was ephemeral.
+      - api_scratch:/app/scratch/local
 
   web:
     image: ${WEB_IMAGE:-jyotisha-web:local}
@@ -68,3 +73,4 @@ services:
 volumes:
   caddy_data:
   caddy_config:
+  api_scratch: