fix(api): persist scratch/local and bound heavy compute concurrency
Keep async job and chart-cache files across API recreates, freeze jyotish_api_server.py growth, and fail fast with 429 when rectification or high-rigor compute is saturated. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -170,3 +170,7 @@ Triage uses the canonical `needs-triage`, `needs-info`, `ready-for-agent`, `read
|
||||
### Domain docs
|
||||
|
||||
Domain documentation uses the single-context layout. See `docs/agents/domain.md`.
|
||||
|
||||
## 8. API Server Growth Freeze
|
||||
|
||||
`scripts/jyotish_api_server.py` must not grow. New endpoints and features go in dedicated modules under `scripts/` and are thinly registered from the main file. Do not add new handler bodies, workflows, or feature branches to this file. The line-count cap is enforced by `tests/test_api_server_growth_contract.py` (live count at freeze plus 300 lines of bugfix slack).
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
# PROGRESS · Python 引擎运行时治理(2026-09-01)
|
||||
|
||||
工作树:`.worktrees/engine-runtime-hygiene-20260901`
|
||||
分支:`codex/engine-runtime-hygiene-20260901`
|
||||
HEAD:`80e7736163be684119c03013a07980031468c634`(仍等于 `origin/staging`)
|
||||
|
||||
未提交、未推送。未改 `frontend/src/app/page.tsx`。未改 `.gitea/workflows/**`。未切分支、未 reset。迁移 squash 按任务书明确延后。
|
||||
|
||||
## 任务 1 · api 命名卷
|
||||
|
||||
- 容器 `WORKDIR`(`deploy/railway-api.Dockerfile`):`/app`
|
||||
- 代码路径:`Path(REPO_ROOT) / 'scratch' / 'local' / ...` → 容器内 **`/app/scratch/local`**
|
||||
- Compose 卷名:`api_scratch`
|
||||
- 挂载:`api_scratch:/app/scratch/local`(`deploy/docker-compose.server.yml` 的 api 服务)
|
||||
- 项目前缀后的 Docker 卷名:`jyotisha-staging_api_scratch` / `jyotisha-production_api_scratch`
|
||||
- 向后兼容:首次 `docker compose up` 创建命名卷;当前 overlay 数据本就是易失的,无迁移
|
||||
|
||||
**备份结论(已写入 `deploy/README.md`)**
|
||||
|
||||
- Chart cache(`/app/scratch/local/api_chart_cache`,默认 TTL 900s)不值得备份;重建比归档便宜。
|
||||
- Async / high-rigor job 态(`async_jobs` / `async_jobs.sqlite3` / `high_rigor_jobs`,默认 TTL 3600s)是短生命周期进行中工作,不是恢复单元。
|
||||
- Staging 加密备份继续只走 `deploy/backup-staging-postgres.sh`。**不要**把 `api_scratch` 加进该脚本。命名卷的唯一目标是容器重建后缓存与进行中任务还在。
|
||||
|
||||
**Deploy 脚本:** `run-staging-deploy.sh` / `run-production-deploy.sh` **未改**。`docker compose up` 会创建并重新挂上命名卷;现有 `up -d --remove-orphans` 不会 `down -v`。
|
||||
|
||||
**Staging 实机三条验收(blocked)**:需要真正 dispatch 部署后才能 `docker volume ls`、重启 api 证明 `/app/scratch/local` 存活、核对 `/api/health`。Compose 挂载合同已锁在 `tests/test_railway_deployment.py`。
|
||||
|
||||
## 任务 2 · 冻结 `jyotish_api_server.py`
|
||||
|
||||
- 开工 live `wc -l`:**11063**(任务书 11035 已过时;出生精度合入后又涨了约 28 行)
|
||||
- 本轮薄注册后:**11088**
|
||||
- 合同上限:**11363**(11063 + 300 bugfix slack)
|
||||
- 测试:`tests/test_api_server_growth_contract.py`(注释写明新功能必须开模块)
|
||||
- 已列入 `scripts/run_quality_gate.py` `CORE_PYTEST_TARGETS`
|
||||
- 根 `AGENTS.md` 第 8 节:`scripts/jyotish_api_server.py` must not grow;新端点/功能落 `scripts/` 模块,主文件只薄注册
|
||||
|
||||
## 任务 3 · 重计算并发闸
|
||||
|
||||
新模块 `scripts/api_heavy_compute_gate.py`。主文件只 import,并在 `do_POST` 里 acquire/release。`GET`(含 `/api/health` 与 job 轮询)不经过闸门。
|
||||
|
||||
- 环境变量 `JYOTISH_HEAVY_COMPUTE_CONCURRENCY`,默认 **2**(对齐 2 vCPU)
|
||||
- `JYOTISH_HEAVY_COMPUTE_RETRY_AFTER_SECONDS`,默认 **2**
|
||||
- 饱和:fail-fast,不排队。HTTP **429** + `Retry-After` + `error_code=ERR_COMPUTE_BUSY`
|
||||
|
||||
**入闸端点**
|
||||
|
||||
- `POST /api/rectification/sensitivity_scan`
|
||||
- `POST /api/active_rectification_events`
|
||||
- `POST /api/active_rectification_events_v4`
|
||||
- `POST /api/rectification/v5/candidate-features`
|
||||
- `POST /api/rectification/v5/score`
|
||||
- `POST /api/rectification/v5/diagnostics`
|
||||
- `POST /api/rectification/v5/vedastro-validate`
|
||||
- `POST /api/dynamic_rectification_opportunities`
|
||||
- `POST /api/dynamic_rectification_score`
|
||||
- `POST /api/high_rigor_workflow`
|
||||
- `POST /api/consultation_workflow`
|
||||
- `POST /api/professional_reading`
|
||||
- `POST /api/vedastro/range_scan`
|
||||
- `POST /api/vedastro_gateway/run`
|
||||
- `POST /api/vedastro_gateway/jobs/{id}/run`(前缀匹配)
|
||||
- `POST /api/thematic_report`
|
||||
|
||||
**不入闸:** `GET /api/health`、城市/位置解析、job 轮询、`/api/chart`(已有 TTL 缓存 + `JYOTISH_ASYNC_JOB_*` 有界队列;再叠 429 会误伤常规排盘)、问卷、v5 versions、capability/catalog 等轻量路由。
|
||||
|
||||
## 验证命令与输出
|
||||
|
||||
```text
|
||||
/opt/anaconda3/bin/python3.12 -m pytest \
|
||||
tests/test_api_server_growth_contract.py \
|
||||
tests/test_api_heavy_compute_gate.py -v
|
||||
```
|
||||
|
||||
```text
|
||||
collected 7 items
|
||||
tests/test_api_server_growth_contract.py ... [ 42%]
|
||||
tests/test_api_heavy_compute_gate.py .... [100%]
|
||||
============================== 7 passed in 0.28s ===============================
|
||||
```
|
||||
|
||||
饱和用例(limit=1):第二个重请求 **429** + `Retry-After: 2` + `ERR_COMPUTE_BUSY`;释放后 200;同期 `GET /api/health` 与 `POST /api/location/resolve` 仍 200。
|
||||
|
||||
```text
|
||||
/opt/anaconda3/bin/python3.12 -m pytest tests/test_api_server_security.py -q
|
||||
# 129 passed, ~90s(仅 datetime.utcnow DeprecationWarning)
|
||||
|
||||
/opt/anaconda3/bin/python3.12 -m pytest \
|
||||
tests/test_runtime_security_p0.py \
|
||||
tests/test_api_server_script_entrypoint.py -q
|
||||
# 11 passed
|
||||
|
||||
/opt/anaconda3/bin/python3.12 -m pytest \
|
||||
tests/test_api_async_job_contract.py \
|
||||
tests/test_railway_deployment.py \
|
||||
tests/test_api_server_security.py::test_health_endpoint_exposes_runtime_accuracy_metadata \
|
||||
tests/test_api_server_security.py::test_vedastro_range_scan_endpoint_uses_user_birth_and_returns_controlled_blocked_state \
|
||||
tests/test_api_server_security.py::test_professional_reading_composes_high_rigor_and_gateway \
|
||||
tests/test_api_server_growth_contract.py \
|
||||
tests/test_api_heavy_compute_gate.py -v
|
||||
# 22 passed in 0.38s
|
||||
```
|
||||
|
||||
`python3 scripts/pre_work_check.py --remote-timeout 8 --command-timeout 45` → `status=pass`。
|
||||
|
||||
未改前端,故未跑 `npm test` / tsc。未跑完整 `scripts/run_quality_gate.py --profile quick`(quick 仍会跑 Next.js test/lint/build;本轮前端豁免)。
|
||||
|
||||
## 改动文件
|
||||
|
||||
- `deploy/docker-compose.server.yml`
|
||||
- `deploy/README.md`
|
||||
- `AGENTS.md`
|
||||
- `scripts/jyotish_api_server.py`(薄注册;+25 行)
|
||||
- `scripts/api_heavy_compute_gate.py`(新)
|
||||
- `scripts/run_quality_gate.py`
|
||||
- `tests/test_api_server_growth_contract.py`(新)
|
||||
- `tests/test_api_heavy_compute_gate.py`(新)
|
||||
- `tests/test_railway_deployment.py`
|
||||
- `PROGRESS-engine-runtime-hygiene-20260901.md`(本文件)
|
||||
|
||||
未改:`deploy/run-staging-deploy.sh`、`deploy/run-production-deploy.sh`、`deploy/backup-staging-postgres.sh`。
|
||||
|
||||
## 明确未做 / blocked
|
||||
|
||||
- Staging/production 实机挂卷、重启存活、`/api/health` 部署 SHA(需部署窗口)。
|
||||
- 完整 quick gate(含 frontend runtime)。
|
||||
- 迁移 squash。
|
||||
- 把 `/api/chart` 同步路径送进这道闸。
|
||||
- 提交 / 推送 / 提升 `main`。
|
||||
@@ -26,12 +26,19 @@ This migration changes both infrastructure and persistence. It is a controlled S
|
||||
```text
|
||||
Spaceship DNS -> Caddy :80/:443 -> web:3000 -> api:5200
|
||||
| -> local astrology engines
|
||||
| -> named volume api_scratch -> /app/scratch/local
|
||||
-> private PostgreSQL 17 + Better Auth
|
||||
-> external model and mail providers
|
||||
```
|
||||
|
||||
Only Caddy publishes host ports. Ports `3000` and `5200` must remain private.
|
||||
|
||||
The API image `WORKDIR` is `/app` (`deploy/railway-api.Dockerfile`). Runtime files written by `scripts/jyotish_api_server.py` resolve to `/app/scratch/local` (chart cache `api_chart_cache`, async job files or `async_jobs.sqlite3`, high-rigor job records). Compose mounts the named volume `api_scratch` at that path so those files survive container recreate. Docker creates the volume on first `up`; there is no data migration because the previous container overlay was ephemeral. Deploy scripts (`run-staging-deploy.sh`, `run-production-deploy.sh`) do not need extra volume flags: `docker compose up` creates and reattaches named volumes automatically.
|
||||
|
||||
The volume is not a backup domain. Chart cache is a TTL'd compute cache (default 900s) and is cheaper to rebuild than to archive. Async / high-rigor job state is short-lived (default TTL 3600s) and is not a restore unit. Restart persistence is the only reason the volume exists. Staging encrypted backups remain PostgreSQL-only via `deploy/backup-staging-postgres.sh`; do not add `api_scratch` to that helper. Project-prefixed volume names are `jyotisha-staging_api_scratch` and `jyotisha-production_api_scratch`.
|
||||
|
||||
Heavy rectification scans and high-rigor workflows share a process-wide fail-fast concurrency gate (`JYOTISH_HEAVY_COMPUTE_CONCURRENCY`, default `2` to match the 2 vCPU host). Saturated requests return HTTP 429 with `Retry-After` and are not queued. Health checks and other light routes do not take a slot.
|
||||
|
||||
## DNS and Supabase Auth
|
||||
|
||||
Final Spaceship resource records (apply only during the approved cutover window):
|
||||
@@ -280,6 +287,8 @@ cd /opt/jyotisha-staging
|
||||
|
||||
The helper invokes `pg_dump --format=custom --no-owner` in the PostgreSQL container and encrypts the stream with `openssl enc -aes-256-cbc -salt -pbkdf2 -pass env:STAGING_BACKUP_ENCRYPTION_KEY`. It creates mode-`0600` `.dump.enc` files in a mode-`0700` directory, refuses disk usage at or above 70%, publishes atomically, and retains only the newest three encrypted local backups. The encryption passphrase is supplied through the environment, never as a command-line argument or printed value. Keep the archive directory on this staging VPS only; there is no off-site staging recovery and no off-site staging backup. These three local encrypted copies are rehearsal/rollback aids, not disaster-recovery backups.
|
||||
|
||||
Do not back up the API `api_scratch` volume. Chart cache (`/app/scratch/local/api_chart_cache`) is a disposable TTL cache; restoring it has no user-visible correctness value. Async job records (`async_jobs` / `async_jobs.sqlite3` and `high_rigor_jobs`) expire within an hour by default and represent in-flight work, not durable product state. A staging or production restore is a PostgreSQL restore. Recreating the API container without the named volume only drops cache and in-flight jobs, which is the same loss the overlay filesystem already had before this volume existed.
|
||||
|
||||
### Restore drill into a disposable database
|
||||
|
||||
Run a restore drill only against the disposable `jyotisha_restore_check` database. Choose one archive and use a temporary decrypted custom-format dump; the commands below match the backup helper's AES-256-CBC/PBKDF2 and `pg_dump --format=custom` interfaces:
|
||||
|
||||
@@ -19,6 +19,11 @@ services:
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
start_interval: 1s
|
||||
volumes:
|
||||
# WORKDIR is /app (deploy/railway-api.Dockerfile). Chart cache, async job
|
||||
# files, and high-rigor job records live under scratch/local. Named volume
|
||||
# is created on first use; previous overlay data was ephemeral.
|
||||
- api_scratch:/app/scratch/local
|
||||
|
||||
web:
|
||||
image: ${WEB_IMAGE:-jyotisha-web:local}
|
||||
@@ -68,3 +73,4 @@ services:
|
||||
volumes:
|
||||
caddy_data:
|
||||
caddy_config:
|
||||
api_scratch:
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
"""Fail-fast bounded concurrency for heavy Jyotish API compute endpoints.
|
||||
|
||||
Request threads that run rectification scans or high-rigor workflows share one
|
||||
process-wide semaphore sized for the 2 vCPU production host. Saturated requests
|
||||
return immediately; they are not queued. Health checks and other light routes
|
||||
must not call this gate.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import threading
|
||||
|
||||
DEFAULT_CONCURRENCY = 2
|
||||
DEFAULT_RETRY_AFTER_SECONDS = 2
|
||||
ENV_CONCURRENCY = "JYOTISH_HEAVY_COMPUTE_CONCURRENCY"
|
||||
ENV_RETRY_AFTER = "JYOTISH_HEAVY_COMPUTE_RETRY_AFTER_SECONDS"
|
||||
|
||||
HEAVY_COMPUTE_PATHS = frozenset(
|
||||
{
|
||||
"/api/rectification/sensitivity_scan",
|
||||
"/api/active_rectification_events",
|
||||
"/api/active_rectification_events_v4",
|
||||
"/api/rectification/v5/candidate-features",
|
||||
"/api/rectification/v5/score",
|
||||
"/api/rectification/v5/diagnostics",
|
||||
"/api/rectification/v5/vedastro-validate",
|
||||
"/api/dynamic_rectification_opportunities",
|
||||
"/api/dynamic_rectification_score",
|
||||
"/api/high_rigor_workflow",
|
||||
"/api/consultation_workflow",
|
||||
"/api/professional_reading",
|
||||
"/api/vedastro/range_scan",
|
||||
"/api/vedastro_gateway/run",
|
||||
"/api/thematic_report",
|
||||
}
|
||||
)
|
||||
|
||||
_state_lock = threading.Lock()
|
||||
_semaphore: threading.BoundedSemaphore | None = None
|
||||
_retry_after_seconds = DEFAULT_RETRY_AFTER_SECONDS
|
||||
|
||||
|
||||
class HeavyComputeBusy(RuntimeError):
|
||||
"""No heavy-compute slot is free; callers must fail fast with HTTP 429."""
|
||||
|
||||
error_code = "ERR_COMPUTE_BUSY"
|
||||
|
||||
def __init__(self, retry_after_seconds: int) -> None:
|
||||
super().__init__(
|
||||
"Heavy compute capacity is saturated; retry after the Retry-After delay."
|
||||
)
|
||||
self.retry_after_seconds = max(int(retry_after_seconds), 1)
|
||||
|
||||
|
||||
def _parse_positive_int(raw: str | None, default: int) -> int:
|
||||
try:
|
||||
value = int(str(raw or "").strip())
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
return value if value >= 1 else default
|
||||
|
||||
|
||||
def is_heavy_compute_path(path: str) -> bool:
|
||||
if path in HEAVY_COMPUTE_PATHS:
|
||||
return True
|
||||
return path.startswith("/api/vedastro_gateway/jobs/") and path.endswith("/run")
|
||||
|
||||
|
||||
def reset_heavy_compute_gate(
|
||||
*,
|
||||
limit: int | None = None,
|
||||
retry_after_seconds: int | None = None,
|
||||
) -> None:
|
||||
"""Rebuild the process-wide semaphore. Tests must call this after env changes."""
|
||||
global _semaphore, _retry_after_seconds
|
||||
resolved_limit = (
|
||||
limit
|
||||
if limit is not None
|
||||
else _parse_positive_int(os.environ.get(ENV_CONCURRENCY), DEFAULT_CONCURRENCY)
|
||||
)
|
||||
resolved_retry = (
|
||||
retry_after_seconds
|
||||
if retry_after_seconds is not None
|
||||
else _parse_positive_int(os.environ.get(ENV_RETRY_AFTER), DEFAULT_RETRY_AFTER_SECONDS)
|
||||
)
|
||||
with _state_lock:
|
||||
_retry_after_seconds = resolved_retry
|
||||
_semaphore = threading.BoundedSemaphore(resolved_limit)
|
||||
|
||||
|
||||
def _ensure_locked() -> tuple[threading.BoundedSemaphore, int]:
|
||||
global _semaphore, _retry_after_seconds
|
||||
if _semaphore is None:
|
||||
limit = _parse_positive_int(os.environ.get(ENV_CONCURRENCY), DEFAULT_CONCURRENCY)
|
||||
_retry_after_seconds = _parse_positive_int(
|
||||
os.environ.get(ENV_RETRY_AFTER), DEFAULT_RETRY_AFTER_SECONDS
|
||||
)
|
||||
_semaphore = threading.BoundedSemaphore(limit)
|
||||
return _semaphore, _retry_after_seconds
|
||||
|
||||
|
||||
def acquire_heavy_compute_slot(path: str) -> threading.BoundedSemaphore | None:
|
||||
"""Acquire a slot for a gated path. Light paths return None. Fail-fast on saturation."""
|
||||
if not is_heavy_compute_path(path):
|
||||
return None
|
||||
with _state_lock:
|
||||
semaphore, retry_after = _ensure_locked()
|
||||
if not semaphore.acquire(blocking=False):
|
||||
raise HeavyComputeBusy(retry_after)
|
||||
return semaphore
|
||||
|
||||
|
||||
def release_heavy_compute_slot(slot: threading.BoundedSemaphore | None) -> None:
|
||||
if slot is not None:
|
||||
slot.release()
|
||||
@@ -93,6 +93,18 @@ except ModuleNotFoundError: # pragma: no cover - script execution path
|
||||
from western_oracle_adapter import build_packet_from_oracle_payload
|
||||
from western_chart_engine import build_tropical_western_evidence_packet
|
||||
from western_timing_engine import build_timing_techniques
|
||||
try:
|
||||
from scripts.api_heavy_compute_gate import (
|
||||
HeavyComputeBusy,
|
||||
acquire_heavy_compute_slot,
|
||||
release_heavy_compute_slot,
|
||||
)
|
||||
except ModuleNotFoundError: # pragma: no cover - script execution path
|
||||
from api_heavy_compute_gate import (
|
||||
HeavyComputeBusy,
|
||||
acquire_heavy_compute_slot,
|
||||
release_heavy_compute_slot,
|
||||
)
|
||||
|
||||
from ayanamsa_utils import DEFAULT_AYANAMSA_NAME, UnsupportedAyanamsaError, normalize_ayanamsa_name
|
||||
from raman_support_observations import build_raman_support_observations
|
||||
@@ -3137,7 +3149,7 @@ class RateLimited(RuntimeError):
|
||||
class JyotishAPIHandler(BaseHTTPRequestHandler):
|
||||
server_version = 'JyotishAPI/6.9.14'
|
||||
|
||||
def _json(self, data, status=200):
|
||||
def _json(self, data, status=200, extra_headers=None):
|
||||
self.send_response(status)
|
||||
self.send_header('Content-Type', 'application/json; charset=utf-8')
|
||||
self._send_cors_headers()
|
||||
@@ -3145,11 +3157,13 @@ class JyotishAPIHandler(BaseHTTPRequestHandler):
|
||||
self.send_header('Access-Control-Allow-Methods', 'GET, POST, OPTIONS')
|
||||
self.send_header('Access-Control-Allow-Headers', 'Content-Type, Authorization')
|
||||
self.send_header('Vary', 'Origin')
|
||||
for key, value in (extra_headers or {}).items():
|
||||
self.send_header(key, value)
|
||||
self.end_headers()
|
||||
self.wfile.write(json.dumps(data, ensure_ascii=False, default=str).encode())
|
||||
|
||||
def _error_json(self, message, status=500, error_code='ERR_INTERNAL'):
|
||||
self._json({'success': False, 'error': message, 'error_code': error_code}, status)
|
||||
def _error_json(self, message, status=500, error_code='ERR_INTERNAL', extra_headers=None):
|
||||
self._json({'success': False, 'error': message, 'error_code': error_code}, status, extra_headers=extra_headers)
|
||||
|
||||
def _html(self, content, status=200):
|
||||
encoded = content.encode('utf-8')
|
||||
@@ -3364,9 +3378,11 @@ class JyotishAPIHandler(BaseHTTPRequestHandler):
|
||||
|
||||
def do_POST(self):
|
||||
path = urlparse(self.path).path
|
||||
compute_slot = None
|
||||
try:
|
||||
self._enforce_request_security(require_json=True)
|
||||
body = self._read_json_body()
|
||||
compute_slot = acquire_heavy_compute_slot(path)
|
||||
if path == '/api/location/resolve':
|
||||
city = str(body.get('city') or '').strip()
|
||||
city_aliases = {'beijing': '北京', 'shanghai': '上海', 'guangzhou': '广州', 'shenzhen': '深圳'}
|
||||
@@ -3567,6 +3583,13 @@ class JyotishAPIHandler(BaseHTTPRequestHandler):
|
||||
self._json(result)
|
||||
else:
|
||||
self._error_json(f'Unknown endpoint: {path}', 404, 'ERR_NOT_FOUND')
|
||||
except HeavyComputeBusy as exc:
|
||||
self._error_json(
|
||||
str(exc),
|
||||
429,
|
||||
exc.error_code,
|
||||
extra_headers={'Retry-After': str(exc.retry_after_seconds)},
|
||||
)
|
||||
except RateLimited as exc:
|
||||
self._error_json(str(exc), 429, 'ERR_RATE_LIMITED')
|
||||
except BadRequest as e:
|
||||
@@ -3581,6 +3604,8 @@ class JyotishAPIHandler(BaseHTTPRequestHandler):
|
||||
import logging
|
||||
logging.exception("[api_server] request failed for %s", path)
|
||||
self._error_json('Internal server error', 500, 'ERR_INTERNAL')
|
||||
finally:
|
||||
release_heavy_compute_slot(compute_slot)
|
||||
|
||||
def _read_json_body(self):
|
||||
raw_length = self.headers.get('Content-Length', '0')
|
||||
|
||||
@@ -72,6 +72,10 @@ CORE_PYTEST_TARGETS = [
|
||||
"tests/test_session_management_entrypoints.py",
|
||||
# Pure source/SQL regex for the birth-time journey; no runtime services.
|
||||
"tests/test_birth_time_journey_contract.py",
|
||||
# Freeze scripts/jyotish_api_server.py growth; new features must be modules.
|
||||
"tests/test_api_server_growth_contract.py",
|
||||
# Fail-fast heavy-compute concurrency gate (429 + Retry-After, health ungated).
|
||||
"tests/test_api_heavy_compute_gate.py",
|
||||
]
|
||||
|
||||
RUNTIME_TRUTH_PYTEST_TARGETS = [
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
"""Fail-fast bounded concurrency for heavy API compute endpoints."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import threading
|
||||
from io import BytesIO
|
||||
|
||||
import pytest
|
||||
|
||||
from scripts import api_heavy_compute_gate as gate
|
||||
from scripts.api_heavy_compute_gate import (
|
||||
DEFAULT_CONCURRENCY,
|
||||
HEAVY_COMPUTE_PATHS,
|
||||
HeavyComputeBusy,
|
||||
acquire_heavy_compute_slot,
|
||||
is_heavy_compute_path,
|
||||
release_heavy_compute_slot,
|
||||
reset_heavy_compute_gate,
|
||||
)
|
||||
from scripts.jyotish_api_server import (
|
||||
DEFAULT_ALLOWED_HOSTS,
|
||||
DEFAULT_ALLOWED_ORIGINS,
|
||||
JyotishAPIHandler,
|
||||
)
|
||||
|
||||
|
||||
class _FakeHeaders(dict):
|
||||
def get(self, key, default=None):
|
||||
return super().get(key, default)
|
||||
|
||||
|
||||
class _FakeServer:
|
||||
allowed_origins = DEFAULT_ALLOWED_ORIGINS
|
||||
allowed_hosts = DEFAULT_ALLOWED_HOSTS
|
||||
|
||||
|
||||
class _PostCaptureHandler(JyotishAPIHandler):
|
||||
def __init__(self, path: str, payload: dict) -> None:
|
||||
raw = json.dumps(payload).encode("utf-8")
|
||||
self.headers = _FakeHeaders(
|
||||
{
|
||||
"Content-Length": str(len(raw)),
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
)
|
||||
self.server = _FakeServer()
|
||||
self.path = path
|
||||
self.rfile = BytesIO(raw)
|
||||
self.wfile = BytesIO()
|
||||
self.status_code = None
|
||||
self.response_headers = []
|
||||
self.client_address = ("test-heavy-compute", 0)
|
||||
|
||||
def send_response(self, code, message=None): # noqa: ANN001
|
||||
self.status_code = code
|
||||
|
||||
def send_header(self, key, value): # noqa: ANN001
|
||||
self.response_headers.append((key, value))
|
||||
|
||||
def end_headers(self):
|
||||
return None
|
||||
|
||||
def payload(self) -> dict:
|
||||
return json.loads(self.wfile.getvalue().decode("utf-8"))
|
||||
|
||||
|
||||
class _GetCaptureHandler(JyotishAPIHandler):
|
||||
def __init__(self, path: str) -> None:
|
||||
self.headers = _FakeHeaders()
|
||||
self.server = _FakeServer()
|
||||
self.path = path
|
||||
self.wfile = BytesIO()
|
||||
self.status_code = None
|
||||
self.response_headers = []
|
||||
self.client_address = ("test-heavy-compute", 0)
|
||||
|
||||
def send_response(self, code, message=None): # noqa: ANN001
|
||||
self.status_code = code
|
||||
|
||||
def send_header(self, key, value): # noqa: ANN001
|
||||
self.response_headers.append((key, value))
|
||||
|
||||
def end_headers(self):
|
||||
return None
|
||||
|
||||
def payload(self) -> dict:
|
||||
return json.loads(self.wfile.getvalue().decode("utf-8"))
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def limit_one_gate(monkeypatch):
|
||||
monkeypatch.setenv("JYOTISH_HEAVY_COMPUTE_CONCURRENCY", "1")
|
||||
monkeypatch.setenv("JYOTISH_HEAVY_COMPUTE_RETRY_AFTER_SECONDS", "2")
|
||||
monkeypatch.setenv("JYOTISH_API_RATE_LIMIT_PER_MINUTE", "0")
|
||||
reset_heavy_compute_gate()
|
||||
yield
|
||||
monkeypatch.delenv("JYOTISH_HEAVY_COMPUTE_CONCURRENCY", raising=False)
|
||||
monkeypatch.delenv("JYOTISH_HEAVY_COMPUTE_RETRY_AFTER_SECONDS", raising=False)
|
||||
reset_heavy_compute_gate()
|
||||
|
||||
|
||||
def test_health_and_light_paths_are_not_gated() -> None:
|
||||
assert not is_heavy_compute_path("/api/health")
|
||||
assert not is_heavy_compute_path("/api/cities")
|
||||
assert not is_heavy_compute_path("/api/location/resolve")
|
||||
assert not is_heavy_compute_path("/api/chart/jobs/abc")
|
||||
assert not is_heavy_compute_path("/api/high_rigor_workflow/jobs/abc")
|
||||
assert not is_heavy_compute_path("/api/rectification/v5/versions")
|
||||
assert is_heavy_compute_path("/api/high_rigor_workflow")
|
||||
assert is_heavy_compute_path("/api/rectification/sensitivity_scan")
|
||||
assert is_heavy_compute_path("/api/vedastro_gateway/jobs/job1/run")
|
||||
assert "/api/chart" not in HEAVY_COMPUTE_PATHS
|
||||
assert DEFAULT_CONCURRENCY == 2
|
||||
|
||||
|
||||
def test_acquire_fail_fast_then_succeeds_after_release() -> None:
|
||||
reset_heavy_compute_gate(limit=1, retry_after_seconds=3)
|
||||
first = acquire_heavy_compute_slot("/api/high_rigor_workflow")
|
||||
assert first is not None
|
||||
with pytest.raises(HeavyComputeBusy) as caught:
|
||||
acquire_heavy_compute_slot("/api/rectification/sensitivity_scan")
|
||||
assert caught.value.retry_after_seconds == 3
|
||||
assert caught.value.error_code == "ERR_COMPUTE_BUSY"
|
||||
light = acquire_heavy_compute_slot("/api/health")
|
||||
assert light is None
|
||||
release_heavy_compute_slot(first)
|
||||
second = acquire_heavy_compute_slot("/api/consultation_workflow")
|
||||
assert second is not None
|
||||
release_heavy_compute_slot(second)
|
||||
reset_heavy_compute_gate()
|
||||
|
||||
|
||||
def test_saturated_request_returns_429_with_retry_after(limit_one_gate, monkeypatch) -> None:
|
||||
started = threading.Event()
|
||||
release_first = threading.Event()
|
||||
first_status = {}
|
||||
|
||||
def _slow_compute(self, body): # noqa: ANN001
|
||||
started.set()
|
||||
assert release_first.wait(timeout=5)
|
||||
return {"ok": True, "endpoint": "high_rigor_workflow"}
|
||||
|
||||
monkeypatch.setattr(JyotishAPIHandler, "_compute_high_rigor_workflow", _slow_compute)
|
||||
|
||||
def _run_first() -> None:
|
||||
handler = _PostCaptureHandler("/api/high_rigor_workflow", {})
|
||||
handler.do_POST()
|
||||
first_status["code"] = handler.status_code
|
||||
first_status["payload"] = handler.payload()
|
||||
|
||||
worker = threading.Thread(target=_run_first)
|
||||
worker.start()
|
||||
assert started.wait(timeout=5)
|
||||
|
||||
blocked = _PostCaptureHandler("/api/high_rigor_workflow", {})
|
||||
blocked.do_POST()
|
||||
assert blocked.status_code == 429
|
||||
assert ("Retry-After", "2") in blocked.response_headers
|
||||
payload = blocked.payload()
|
||||
assert payload["success"] is False
|
||||
assert payload["error_code"] == "ERR_COMPUTE_BUSY"
|
||||
|
||||
health = _GetCaptureHandler("/api/health")
|
||||
health.do_GET()
|
||||
assert health.status_code == 200
|
||||
assert health.payload()["status"] == "ok"
|
||||
|
||||
light = _PostCaptureHandler("/api/location/resolve", {"city": "beijing"})
|
||||
light.do_POST()
|
||||
assert light.status_code == 200
|
||||
assert light.payload()["status"] == "local_city_match"
|
||||
|
||||
release_first.set()
|
||||
worker.join(timeout=5)
|
||||
assert not worker.is_alive()
|
||||
assert first_status["code"] == 200
|
||||
|
||||
recovered = _PostCaptureHandler("/api/high_rigor_workflow", {})
|
||||
recovered.do_POST()
|
||||
assert recovered.status_code == 200
|
||||
assert recovered.payload()["ok"] is True
|
||||
|
||||
|
||||
def test_quality_gate_runs_heavy_compute_gate() -> None:
|
||||
from scripts.run_quality_gate import CORE_PYTEST_TARGETS
|
||||
|
||||
assert "tests/test_api_heavy_compute_gate.py" in CORE_PYTEST_TARGETS
|
||||
assert gate.DEFAULT_CONCURRENCY == 2
|
||||
@@ -0,0 +1,44 @@
|
||||
"""Freeze scripts/jyotish_api_server.py growth.
|
||||
|
||||
New endpoints and features must live in new modules and be thinly registered
|
||||
from the main file. This cap is the live line count at freeze (11063 on
|
||||
2026-09-02, via `wc -l`) plus 300 lines of bugfix slack.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
API_SERVER = ROOT / "scripts" / "jyotish_api_server.py"
|
||||
AGENTS = ROOT / "AGENTS.md"
|
||||
|
||||
# Live `wc -l scripts/jyotish_api_server.py` at freeze. New features must not
|
||||
# consume this budget; open a module instead.
|
||||
JYOTISH_API_SERVER_LINE_COUNT_BASELINE = 11063
|
||||
JYOTISH_API_SERVER_LINE_COUNT_CAP = JYOTISH_API_SERVER_LINE_COUNT_BASELINE + 300
|
||||
|
||||
|
||||
def test_jyotish_api_server_must_not_grow_beyond_bugfix_slack() -> None:
|
||||
line_count = API_SERVER.read_bytes().count(b"\n")
|
||||
assert line_count <= JYOTISH_API_SERVER_LINE_COUNT_CAP, (
|
||||
f"{API_SERVER.as_posix()} has {line_count} lines; cap is "
|
||||
f"{JYOTISH_API_SERVER_LINE_COUNT_CAP} ({JYOTISH_API_SERVER_LINE_COUNT_BASELINE} "
|
||||
"baseline + 300 bugfix slack). New endpoints and features must be new "
|
||||
"modules, thinly registered from this file."
|
||||
)
|
||||
|
||||
|
||||
def test_agents_forbids_growing_jyotish_api_server() -> None:
|
||||
agents = AGENTS.read_text(encoding="utf-8")
|
||||
assert "scripts/jyotish_api_server.py" in agents
|
||||
assert "must not grow" in agents
|
||||
assert "thinly registered" in agents
|
||||
|
||||
|
||||
def test_quality_gate_runs_api_server_growth_contract() -> None:
|
||||
from scripts.run_quality_gate import CORE_PYTEST_TARGETS
|
||||
|
||||
assert "tests/test_api_server_growth_contract.py" in CORE_PYTEST_TARGETS
|
||||
quality_gate = (ROOT / "scripts" / "run_quality_gate.py").read_text(encoding="utf-8")
|
||||
assert '"tests/test_api_server_growth_contract.py"' in quality_gate
|
||||
@@ -35,3 +35,5 @@ def test_server_compose_allows_only_the_internal_api_hostname() -> None:
|
||||
compose = (ROOT / "deploy" / "docker-compose.server.yml").read_text(encoding="utf-8")
|
||||
|
||||
assert "JYOTISH_ALLOWED_HOSTS: localhost,127.0.0.1,::1,api" in compose
|
||||
assert "api_scratch:/app/scratch/local" in compose
|
||||
assert " api_scratch:" in compose
|
||||
|
||||
Reference in New Issue
Block a user