merge origin/staging into the beta safety net branch
Safety-net records move to BUG-1358 through BUG-1361 because staging already uses BUG-1355 through BUG-1357 for capacity. Co-Authored-By: Grok (agent loop)
This commit is contained in:
+7
-1
@@ -1,8 +1,14 @@
|
||||
# 印度占星 Skill 更新日志
|
||||
|
||||
## 2026-10-15 — 内测容量:问得太密会停一下,人多时先排一小会儿
|
||||
|
||||
- 同一个人短时间问得太密、校正答得太密,或一小时里报告生成太多,会先停一下,这次不扣点。
|
||||
- 同时算的人多过机器能接的两路时,先排一小会儿。排上了继续;排不上会说人有点多,并把这一轮预留的点数退回。
|
||||
- Skill 版本不变(BUG-1355、BUG-1356)。
|
||||
|
||||
## 2026-10-15 — 内测前补上每日备份、宕机告警和日志上限(待合入 staging)
|
||||
|
||||
- 生产可以每天自动做一次加密备份。站点连续三次打不开、磁盘过满,或短时间错误变多时,可以通知到飞书群。容器日志有大小上限。可以按一轮请求的编号在服务器日志里查找。这些要等产品在主机上装好之后才生效。页面没有变化。Skill 版本不变(BUG-1355、BUG-1356、BUG-1357)。
|
||||
- 生产可以每天自动做一次加密备份。站点连续三次打不开、磁盘过满,或短时间错误变多时,可以通知到飞书群。容器日志有大小上限。可以按一轮请求的编号在服务器日志里查找。这些要等产品在主机上装好之后才生效。页面没有变化。Skill 版本不变(BUG-1358、BUG-1359、BUG-1360)。
|
||||
|
||||
## 2026-10-15 — 合盘更快(待合入 staging)
|
||||
|
||||
|
||||
+1
-1
@@ -37,7 +37,7 @@ The API image `WORKDIR` is `/app` (`deploy/railway-api.Dockerfile`). Runtime fil
|
||||
|
||||
The volume is not a backup domain. Chart cache is a TTL'd compute cache (default 900s) and is cheaper to rebuild than to archive. Async / high-rigor job state is short-lived (default TTL 3600s) and is not a restore unit. Restart persistence is the only reason the volume exists. Staging encrypted backups remain PostgreSQL-only via `deploy/backup-staging-postgres.sh`; do not add `api_scratch` to that helper. Project-prefixed volume names are `jyotisha-staging_api_scratch` and `jyotisha-production_api_scratch`.
|
||||
|
||||
Heavy rectification scans and high-rigor workflows share a process-wide fail-fast concurrency gate (`JYOTISH_HEAVY_COMPUTE_CONCURRENCY`, default `2` to match the 2 vCPU host). Saturated requests return HTTP 429 with `Retry-After` and are not queued. Health checks and other light routes do not take a slot.
|
||||
Heavy rectification scans and high-rigor workflows share a process-wide concurrency gate (`JYOTISH_HEAVY_COMPUTE_CONCURRENCY`, default `2` to match the 2 vCPU host). A saturated request waits up to 8 seconds if fewer than 4 requests are already waiting, then returns HTTP 429 with `Retry-After`. Health checks and other light routes do not take a slot.
|
||||
|
||||
## DNS and Supabase Auth
|
||||
|
||||
|
||||
+60
-14
@@ -36257,7 +36257,53 @@
|
||||
- 复发自:无。
|
||||
- 修复版本:未修。分支 `codex/answer-effort-ab-20261013` 只记录。原先写在 BUG-1351,该号已被数据层占用;按本文件规则保留较早的 BUG-1351,本条改为 BUG-1354,接在 BUG-1353 之后。
|
||||
|
||||
## BUG-1355 | 生产没有每日自动备份
|
||||
## BUG-1355 | 对话、校正回合和报告生成没有按用户限流
|
||||
|
||||
- 状态:mitigated
|
||||
- 首次发现 / 最近更新:2026-10-14 / 2026-10-15
|
||||
- 来源:issue #76。任务书 `docs/tasks/TASK-beta-capacity-20261014.md`。
|
||||
- 现象:一个人可以连续打对话、校正回合或报告生成。报告只有每天 5 份的上限,没有按小时的上限。对话和校正没有按这个人计数。
|
||||
- 触发条件:已登录后调用 `POST /api/consult`、`POST /api/rectification/agent` 或 `POST /api/reports`。开案 `POST /api/rectification/cases/open` 不计数。
|
||||
- 根因:`requestRateLimits` 只覆盖地点搜索、时区、今日星语、星历、合盘和会话写入。重计算闸是全进程共享的,挡不住同一个人把模型调用打满。
|
||||
- 修复:对话 10 次/分钟且 120 次/小时,校正回合 5 次/分钟且 40 次/小时,报告生成另加 3 次/小时(每天 5 份仍在)。两档都还有余量才放行;拒绝时不写入任何一档,也不预留点数。响应是 429、`code: rate_limited` 和 `Retry-After`。
|
||||
- 验证:`frontend/tests/request-rate-limit.test.ts`;报告核心测试「hourly rate limit returns 429 and does not reserve」。
|
||||
- 防复发:上述测试锁住桶的边界、拒绝时不记入另一档,以及三条路由在预留点数之前判断。开案路由源码不得调用这组限流。
|
||||
- 相关记录:BUG-723(校正客户端已有的退避,本单不改)、BUG-1302(预热仍不占重计算名额)、BUG-1357(打回后补上的两条旧检查)。
|
||||
- 复发自:无
|
||||
- 修复版本:分支 `codex/beta-capacity-20261014`。
|
||||
|
||||
## BUG-1356 | 重计算满了立刻拒绝,对话也不再试
|
||||
|
||||
- 状态:mitigated
|
||||
- 首次发现 / 最近更新:2026-10-14 / 2026-10-15
|
||||
- 来源:issue #76。任务书 `docs/tasks/TASK-beta-capacity-20261014.md`。
|
||||
- 现象:同时只有 2 个重计算名额。第 3 个请求马上收到 429。普通对话把这次 429 当成这一轮失败,不按 `Retry-After` 再试。
|
||||
- 触发条件:已有 2 个请求占着 `/api/consultation_workflow` 或校正打分,又来一个对话取证。
|
||||
- 根因:闸门用非阻塞信号量,满了就抛 `HeavyComputeBusy`。对话侧把 HTTP 429 记成 `workflow_rate_limited` 后直接失败。
|
||||
- 修复:名额满时最多再等 8 秒,同时等待的人最多 4 个。等不到或等待的人已经满,仍抛 `HeavyComputeBusy`,`Retry-After` 仍是原来的秒数。对话对这种 429 最多再试 2 次,几次等待加起来不超过 15 秒。等待时进度行写「人有点多,排一下队」。仍失败则沿用原来的取消预留退点,只走一次,并告诉用户这一轮没排上、点数已退回。并发上限仍是 2。
|
||||
- 验证:`tests/test_api_heavy_compute_gate.py` 的排队拿到名额、等待者满立刻拒绝、超时仍 429;`frontend/tests/consultation-workflow-retry.test.ts`。
|
||||
- 防复发:上述测试。健康检查仍不进闸。校正客户端原有的退避(BUG-723)没有改。
|
||||
- 相关记录:BUG-717、BUG-723、BUG-1302、BUG-1345、BUG-1355、BUG-1357。
|
||||
- 复发自:无
|
||||
- 修复版本:分支 `codex/beta-capacity-20261014`。
|
||||
|
||||
## BUG-1357 | 限流改完后两条旧检查没跟上
|
||||
|
||||
- 状态:resolved
|
||||
- 首次发现:2026-10-15
|
||||
- 最近更新:2026-10-15
|
||||
- 影响面:`frontend/tests/consultation-voice-contract.test.ts`;`frontend/tests/personal-report-api.test.ts` 与 `tests/test_repo_privacy_markers.py` 的行号白名单。没有用户界面变化。
|
||||
- 用户现象:同一个人问太密、校正太密、报告太多会先停一下,人多时会先排一小会儿。这些都对。全量检查多出 2 条失败,这一版上不了线。
|
||||
- 触发条件:验收方对 `a54f75e5` 跑前端全量和 Python 全量。
|
||||
- 根因:对话路由把 `runConsultationWorkflow(toolInput` 包进排队重试,源码合同仍要求 `await` 紧贴函数名。报告的小时限流测试插在文件中部,把隐私白名单按行号锁定的三行各下移 13 行。被保护的行为都还在,不是泄漏。上一轮只跑了定向测试,这两条不在那一组里。
|
||||
- 修复:正则仍锁定 `runConsultationWorkflow(toolInput`,不再要求 `await` 紧贴。小时限流测试整段挪到文件末尾,断言原文不变,`SAFE_LINES` 不改。业务代码不改。
|
||||
- 验证:`docs/tasks/PROGRESS-beta-capacity-fix-20261015.md`。语音合同、报告接口、镜像不引用测试共 75 通过、0 失败;隐私扫描 62 通过、0 失败;tsc 退出码 0;lint 0 错误、123 条原有警告。
|
||||
- 防复发:改这三处路由的语句前,在 `frontend/tests` 和 `tests` 里 grep 被改写的原文。新测试不要插在 `SAFE_LINES` 锁定行的上面。
|
||||
- 相关记录:BUG-933、BUG-1014、BUG-1355、BUG-1356
|
||||
- 复发自:BUG-933(源码合同没在同一轮跟上)
|
||||
- 修复版本:分支 `codex/beta-capacity-20261014`。
|
||||
|
||||
## BUG-1358 | 生产没有每日自动备份
|
||||
|
||||
- 状态:代码已交,主机安装待产品
|
||||
- 首次发现 / 最近更新:2026-10-14 / 2026-10-15
|
||||
@@ -36268,41 +36314,41 @@
|
||||
- 修复:新增只从 `main` 运行的每日备份工作流,复用现有加密与还原校验脚本。每日文件用单独前缀,过期清理不会碰到手动恢复点。工作流要等进入 `main` 后才会在北京时间 04:00 跑。本机没有生产主机,没有手动触发。
|
||||
- 验证:合同测试锁定有每日时刻、只在 `main` 上跑、调用同一个恢复脚本、清理只删过期的每日前缀。虚构目录里,手动恢复点和其他文件都还在。
|
||||
- 防复发:上述合同测试。清理函数不接受外部传入的前缀。
|
||||
- 相关记录:BUG-1350、BUG-1356、BUG-1357、BUG-1358。
|
||||
- 相关记录:BUG-1350、BUG-1359、BUG-1360、BUG-1361。
|
||||
- 复发自:无。
|
||||
- 修复版本:分支 `codex/beta-safety-net-20261014`。开工时本文件最大号是 BUG-1353,当时写成 BUG-1354。合并 staging 后该号已被更早的答题强度记录占用,按台账规则改为 BUG-1355。每日备份曾会改写当前恢复点,已由 BUG-1358 分开。
|
||||
- 修复版本:分支 `codex/beta-safety-net-20261014`。开工时本文件最大号是 BUG-1353,当时写成 BUG-1354。第一次合并 staging 后该号已被答题强度记录占用,中间号是 BUG-1355。再次合并后 BUG-1355~1357 已被内测容量占用,本条顺延为 BUG-1358。每日备份曾会改写当前恢复点,已由 BUG-1361 分开。
|
||||
|
||||
## BUG-1356 | 站点打不开或磁盘过满时没有告警
|
||||
## BUG-1359 | 站点打不开或磁盘过满时没有告警
|
||||
|
||||
- 状态:代码已交,主机安装待产品
|
||||
- 首次发现 / 最近更新:2026-10-14 / 2026-10-15
|
||||
- 来源:同 BUG-1355。
|
||||
- 来源:同 BUG-1358。
|
||||
- 现象:容器自己的健康检查只会尝试重启,不会通知人。磁盘变满也没有通知。
|
||||
- 触发条件:公网健康检查连续失败,或生产主机磁盘使用率达到 80%。
|
||||
- 根因:上线时没有持续探活,也没有把告警发到人用的渠道。
|
||||
- 修复:探活脚本连续 3 次失败才发一条告警,恢复只发一次。磁盘检查单独在生产主机上跑。Webhook 从环境文件读取,没有配置就只写本机日志。外发正文不含用户内容。脚本还没装到 runner 或生产主机上。
|
||||
- 验证:本地桩服务覆盖正常、500、超时和缺少字段。连续 3 次才告警,恢复只出现一次。签名测试里的密钥没有出现在发出的正文里。
|
||||
- 防复发:上述脚本测试。告警行若含不允许的字段,脚本拒绝发送。
|
||||
- 相关记录:BUG-1355、BUG-1357。
|
||||
- 相关记录:BUG-1358、BUG-1360。
|
||||
- 复发自:无。
|
||||
- 修复版本:分支 `codex/beta-safety-net-20261014`。原先写成 BUG-1355,合并 staging 后顺延为 BUG-1356。
|
||||
- 修复版本:分支 `codex/beta-safety-net-20261014`。原先写成 BUG-1355,第一次合并 staging 后为 BUG-1356。再次合并后顺延为 BUG-1359。
|
||||
|
||||
## BUG-1357 | 容器日志不轮转,也不能按一轮请求查找
|
||||
## BUG-1360 | 容器日志不轮转,也不能按一轮请求查找
|
||||
|
||||
- 状态:代码已交,主机安装待产品
|
||||
- 首次发现 / 最近更新:2026-10-14 / 2026-10-15
|
||||
- 来源:同 BUG-1355。BUG-1306 取证时日志只在容器控制台,重建后查不到。
|
||||
- 来源:同 BUG-1358。BUG-1306 取证时日志只在容器控制台,重建后查不到。
|
||||
- 现象:编排没有日志大小上限。知道一轮请求的编号时,没有只读的查找命令。
|
||||
- 触发条件:日志持续写入,或需要按 requestId 回看 web、api、caddy。
|
||||
- 根因:日志沿用守护进程的未知默认值,也没有按编号检索的入口。
|
||||
- 修复:每个服务的日志改为单文件 20 MB、最多 10 个文件。新增只读查找脚本,只接受 UUID。错误汇总只计数和最多 5 个编号,丢掉含邮箱、额外日期或非 ASCII 文本的行。要等生产容器按新编排重建后,轮转才生效。
|
||||
- 验证:两个编排文件里每个服务的日志段一致。虚构日志能按时间排序找到编号;命令注入写法被拒绝,旁边的标记文件还在。含邮箱和汉字的样本没有进入告警文本。
|
||||
- 防复发:上述合同测试和查找脚本的格式检查。
|
||||
- 相关记录:BUG-1306、BUG-1355、BUG-1356。
|
||||
- 相关记录:BUG-1306、BUG-1358、BUG-1359。
|
||||
- 复发自:无。
|
||||
- 修复版本:分支 `codex/beta-safety-net-20261014`。原先写成 BUG-1356,合并 staging 后顺延为 BUG-1357。
|
||||
- 修复版本:分支 `codex/beta-safety-net-20261014`。原先写成 BUG-1356,第一次合并后为 BUG-1357。再次合并后顺延为 BUG-1360。
|
||||
|
||||
## BUG-1358 | 每日自动备份改写了当前恢复点
|
||||
## BUG-1361 | 每日自动备份改写了当前恢复点
|
||||
|
||||
- 状态:resolved
|
||||
- 首次发现 / 最近更新:2026-10-15 / 2026-10-15
|
||||
@@ -36313,7 +36359,7 @@
|
||||
- 修复:前缀为 `production-daily` 时仍完成导出、加密和还原核对(含表权限、列权限、属主),并写校验文件,但跳过 `recovery-baseline.env`。另写 `daily-backup-latest.env`,只有文件名、sha256、核对时间和格式版本。手动创建恢复点的路径不变。
|
||||
- 验证:临时目录加假 docker。已有恢复点的内容和修改时间不变;手动路径仍写出 `RECOVERY_FORMAT=2`。发布、迁移、回退所读的文件不引用 `daily-backup-latest.env`。清理仍只删过期的 `production-daily-` 文件。
|
||||
- 防复发:上述合同测试。生产工作流不设置测试用的路径开关。
|
||||
- 相关记录:BUG-1355。
|
||||
- 相关记录:BUG-1358。
|
||||
- 复发自:无。
|
||||
- 修复版本:分支 `codex/beta-safety-net-20261014`。本机没有装到生产主机,也没有点生产按钮。
|
||||
- 修复版本:分支 `codex/beta-safety-net-20261014`。本机没有装到生产主机,也没有点生产按钮。原先写成 BUG-1358。再次合并 staging 后该号给了「没有每日自动备份」,本条顺延为 BUG-1361。
|
||||
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
# 内测邀请码(用兑换码)
|
||||
|
||||
后台路径:`/admin/codes`(兑换码)。需要管理权限。不要把真实兑换码写进仓库、进度或聊天记录。完整码只在生成成功的那一次弹窗里出现,关掉就不能再看,列表里只剩掩码。
|
||||
|
||||
## 批量生成
|
||||
|
||||
1. 打开兑换码页,点「批量生成」。
|
||||
2. 「每个点数」是这一批每个码到账的点数。表单初始值是 10,可以改。内测每码给多少点由当次生成的人填写,产品以后可以改这个数。
|
||||
3. 「数量」一次 1 到 100 个。同一批的点数、到期时间、备注相同。
|
||||
4. 到期时间可以空着。备注用来区分这一批,例如内测第几批,不要写别人的个人资料。
|
||||
5. 生成后立刻把完整码交给对应的人。页面会提示关闭后无法再看。
|
||||
|
||||
## 查谁用了
|
||||
|
||||
列表有掩码、点数、状态、到期、备注、兑换账户、兑换时间、撤销时间。状态可以筛:可用、已过期、已兑换、已撤销。兑换账户和兑换时间就是谁用过。
|
||||
|
||||
## 作废
|
||||
|
||||
还没兑换、也没撤销的码,行上有「撤销」。撤销后不能再兑。已经兑换的不能从这里追回已经到账的点数。
|
||||
|
||||
支付在内测期间保持关闭。这页已经能批量生成、查看使用者和作废,本单不另加功能。
|
||||
@@ -0,0 +1,43 @@
|
||||
# 进度 · 内测容量(2026-10-15)
|
||||
|
||||
- 分支:`codex/beta-capacity-20261014`
|
||||
- 基线:开工时 `origin/staging` `ef49257b734678ff944a840076347baa02d0323c`
|
||||
- 任务书:`docs/tasks/TASK-beta-capacity-20261014.md`
|
||||
- 前置:`origin/codex/node-runtime-resources-20261013` 与 `origin/codex/rect-scoring-perf-20261013` 都是该 staging 的祖先(`merge-base --is-ancestor` 退出码 0)。限流文件里已有过期清理。
|
||||
|
||||
## 做了什么
|
||||
|
||||
- T1:对话 10 次/分钟 + 120 次/小时,校正回合 5 次/分钟 + 40 次/小时,报告 3 次/小时(每天 5 份不变)。拒绝发生在预留点数之前,429 + `rate_limited` + `Retry-After`。开案路由不计数。
|
||||
- T2:重计算并发仍是 2。满了最多等 8 秒,同时等待最多 4 个;等不到或等待已满仍是 429,`Retry-After` 不变。对话对这种 429 最多再试 2 次,等待合计不超过 15 秒。进度行「人有点多,排一下队」,失败句「人有点多,这一轮没排上,点数已经退回。」退点仍走原来的取消预留,代理路径用已有的一次结算。
|
||||
- T3:后台已能批量生成(1–100 个、每码点数自填、初始 10)、按兑换账户查看、撤销未使用的码。说明在 `docs/operations/beta-invite-codes.md`。没有扩功能。
|
||||
- T4:脚本 `frontend/scripts/research/beta-load-smoke.mts`。不从 `frontend/tests` 导入。本机没有跑。
|
||||
- T5:BUG-1355、BUG-1356,CHANGELOG、VOICE、DESIGN、真机两条。
|
||||
|
||||
## 断言
|
||||
|
||||
旧测试仍要求「名额被占着就拒绝」,没有改拒绝码和 `Retry-After` 的期望值。
|
||||
|
||||
| 位置 | 原值 | 新值 | 原因 |
|
||||
| --- | --- | --- | --- |
|
||||
| `tests/test_api_heavy_compute_gate.py` 立刻拒绝的夹具和单测 | `reset()` / `reset(limit=1, retry_after_seconds=3)` | 增加 `wait_seconds=0` | 默认改为最多等 8 秒。这些测试继续锁 429 和 `Retry-After`,排队另有新测试 |
|
||||
| `tests/test_api_chart_subprocess_gate.py` 两处占着名额再请求 | `reset(limit=1)` | `reset(limit=1, wait_seconds=0)` | 同上。仍锁同步排盘被拒绝、日志不出现请求正文 |
|
||||
|
||||
## 偏离与环境缺口
|
||||
|
||||
- T0:仓库文档里的目标规格,staging 与生产都是 2 vCPU / 4 GB,代码默认并发 2。这台 Windows 没有 staging 的登录壳,没有采到机器上的实际核数、内存和 `JYOTISH_HEAVY_COMPUTE_CONCURRENCY`。不能把文档规格当成已经登主机核对过。
|
||||
- T4:没有 `STAGING_TEST_OTP`,没有对 staging 或生产发压测。改前、改后的并发数字都没有。服务器 CPU / 内存同样没有采样。脚本留给验收方在有测试验证码的环境重跑。20 并发档没有跑。
|
||||
- 旧版对话(不走代理运行时)会重试,但进度行要等回答流开始才有,所以那条路径等待时看不到「排一下队」。现在的对话走代理运行时,进度行会换字。任务书允许把排队文案收成只多等。
|
||||
- 没有改数据库,没跑 `test:db`。没有跑前端全量 `npm test`,没有跑 Python 快速门。
|
||||
- `deploy/README.md` 有一句仍写「满了不排队」。改成了最多等 8 秒、等待不超过 4 个,避免操作说明和代码相反。没有改部署配置。
|
||||
|
||||
## 自测
|
||||
|
||||
| 检查 | 命令 | 结果数字 |
|
||||
| --- | --- | --- |
|
||||
| Python 闸门 | `.venv\Scripts\python.exe -m pytest tests/test_api_heavy_compute_gate.py tests/test_api_chart_subprocess_gate.py -q` | 17 通过(8 + 9),0 失败 |
|
||||
| 前端相关测试 | `node node_modules/tsx/dist/cli.mjs --test tests/request-rate-limit.test.ts tests/consultation-workflow-retry.test.ts tests/personal-report-api.test.ts tests/runtime-sources-no-test-imports.test.ts` | 76 通过,0 失败 |
|
||||
| 类型 | `frontend\node_modules\.bin\tsc --noEmit -p .` | 退出码 0 |
|
||||
| 前端 lint | `npm run lint` | 0 错误,123 条原有警告 |
|
||||
| 压测 | 未执行 `frontend/scripts/research/beta-load-smoke.mts` | 本机没有 `STAGING_TEST_OTP`。没有改前/改后并发数字,没有服务器 CPU/内存 |
|
||||
|
||||
没有改数据库,没跑 `test:db`。没有跑前端全量 `npm test`,没有跑 Python 快速门。
|
||||
@@ -0,0 +1,46 @@
|
||||
# 进度 · 内测容量修复(2026-10-15,第 1 次打回)
|
||||
|
||||
- 分支:`codex/beta-capacity-20261014`
|
||||
- 被验 SHA:`a54f75e5e33775d5dc91f3058f5505a56b498bf2`
|
||||
- 修复单:`docs/tasks/TASK-beta-capacity-fix-20261015.md`
|
||||
- 原任务书:`docs/tasks/TASK-beta-capacity-20261014.md`
|
||||
- 开工时本分支最大 Bug 编号:BUG-1356。本轮新增 BUG-1357。BUG-1351、BUG-1352、BUG-1353 已在本分支文件里,没有重用。
|
||||
|
||||
## 做了什么
|
||||
|
||||
- 先把最新 `origin/staging` 合并进本分支(不改写已推送的 `a54f75e5`)。合并只带进修复单和任务索引的一行。
|
||||
- 语音合同:仍锁「工作流调用的入参是 `toolInput`」,以及用户轮不塞工具 JSON。不再要求 `await` 紧贴函数名。
|
||||
- 隐私白名单:采用修复单推荐的做法 a。把「hourly rate limit returns 429」整段挪到 `personal-report-api.test.ts` 末尾,断言原文不变。`SAFE_LINES` 不改。三行回到 832、860、890,且整行与白名单一致。
|
||||
- 业务代码不改。
|
||||
|
||||
## 断言
|
||||
|
||||
| 位置 | 原值 | 新值 | 原因 |
|
||||
| --- | --- | --- | --- |
|
||||
| `consultation-voice-contract.test.ts`「consult user turn no longer dumps tool JSON」 | `/await runConsultationWorkflow\(toolInput/` | `/runConsultationWorkflow\(toolInput/` | 调用包进 `retryOnConsultationRateLimit`,`await` 不再紧贴函数名。仍锁入参是 `toolInput` |
|
||||
| `personal-report-api.test.ts` 小时限流测试的位置 | 插在「429 at the daily limit」和「502 model_unavailable」之间 | 文件末尾,断言原文不变 | 插在中部让三行白名单各下移 13 行。挪走后行号复原,不改 `SAFE_LINES` |
|
||||
|
||||
## 源码文本再搜一遍
|
||||
|
||||
在 `frontend/tests` 和 `tests` 里核对三条路由被改写的语句:
|
||||
|
||||
- 只有语音合同把 `await runConsultationWorkflow(toolInput` 写成必须匹配。已按上表改。
|
||||
- `consultation-workflow-contract.test.ts` 是在代理分支里断言**不**出现 `await runConsultationWorkflow`,与这次包装无关,没改。
|
||||
- 「暂时无法生成解读」仍在非排队失败的分支里,`consult-first-frame-and-pacing-20260928.test.tsx` 继续对得上,没改。
|
||||
- `toNextResponse` 多了响应头,没有测试按旧函数签名做源码断言。
|
||||
- Python 侧没有按这三处被改写语句做源码文本断言。
|
||||
|
||||
## 偏离与环境缺口
|
||||
|
||||
- 没有改数据库,没跑 `test:db`。没有跑前端全量 `npm test`,没有跑 Python 快速门。
|
||||
- 并发压测 `frontend/scripts/research/beta-load-smoke.mts` 未跑。没有测试验证码,修复单也写明本机和执行侧都连不上测试服务器,不要登预发。没有改前/改后并发数字,没有服务器占用采样。
|
||||
|
||||
## 自测
|
||||
|
||||
| 检查 | 命令 | 结果数字 |
|
||||
| --- | --- | --- |
|
||||
| 前端相关测试 | `node node_modules/tsx/dist/cli.mjs --test tests/consultation-voice-contract.test.ts tests/runtime-sources-no-test-imports.test.ts tests/personal-report-api.test.ts` | 75 通过,0 失败,0 取消 |
|
||||
| 隐私扫描 | `.venv\Scripts\python.exe -m pytest tests/test_repo_privacy_markers.py -q` | 62 通过,0 失败 |
|
||||
| 类型检查 | `frontend\node_modules\.bin\tsc --noEmit -p .` | 退出码 0 |
|
||||
| 前端 lint | `npm run lint` | 0 错误,123 条原有警告 |
|
||||
| 并发压测 | 未跑 `beta-load-smoke.mts` | 没有测试验证码,没有登预发 |
|
||||
@@ -3,7 +3,7 @@
|
||||
- 分支:`codex/beta-safety-net-20261014`
|
||||
- 基线:开工时 `origin/staging` 为 `71f7fac3a13f817f91755ff01b25b8706380ae19`
|
||||
- 前置:`09906cdc`(发布通路)和 `4c3485bc`、`3893783e`(#58)都已是这个基线的祖先。`deploy/docker-compose.server.yml` 在加日志段之前没有 #58 留下的未合入冲突。
|
||||
- 编号:本文件最大号是 BUG-1353。本单当时写成 BUG-1354(没有每日备份)、BUG-1355(没有探活和磁盘告警)、BUG-1356(日志不轮转、不能按请求编号查)。合并 staging 后 BUG-1354 已被更早的答题强度记录占用,三条顺延为 BUG-1355、BUG-1356、BUG-1357。
|
||||
- 编号:本文件最大号是 BUG-1353。本单当时写成 BUG-1354(没有每日备份)、BUG-1355(没有探活和磁盘告警)、BUG-1356(日志不轮转、不能按请求编号查)。合并 staging 后 BUG-1354 已被更早的答题强度记录占用,三条顺延为 BUG-1355、BUG-1356、BUG-1357。2026-10-15 再次合并 staging 时,这三个号已被内测容量占用,最终编号是 BUG-1358、BUG-1359、BUG-1360。
|
||||
|
||||
## 做了什么
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
|
||||
## 做了什么
|
||||
|
||||
- 先合并最新 `origin/staging`,没有改写已经推送的提交。合并时本分支的 BUG-1354 和更早的答题强度记录撞号,安全网三条顺延为 BUG-1355、BUG-1356、BUG-1357。这次的修复是 BUG-1358。
|
||||
- 先合并最新 `origin/staging`,没有改写已经推送的提交。合并时本分支的 BUG-1354 和更早的答题强度记录撞号,安全网三条顺延为 BUG-1355、BUG-1356、BUG-1357。这次的修复当时写成 BUG-1358。2026-10-15 再次合并 staging 后,BUG-1355~1357 归内测容量,本条修复最终是 BUG-1361。
|
||||
- 用已经存在的 `RECOVERY_BASENAME_PREFIX=production-daily` 当开关,没有另加模式参数。这个前缀下,脚本仍完成导出、加密、还原核对(含表权限、列权限、属主)并写校验文件,但跳过 `recovery-baseline.env`。标准错误打印 `baseline_updated=false`。另写 `.state/daily-backup-latest.env`,只有文件名、sha256、核对时间、`BACKUP_FORMAT=2`。
|
||||
- 不设这个前缀时,仍写 `recovery-baseline.env`,里面有 `RECOVERY_FORMAT=2`,标准错误打印 `baseline_updated=true`,不改已经存在的每日记录。
|
||||
- 标准输出仍是原来的 7 行。因此没有改 `.gitea` 里「恰好 7 行」的检查,也没有改四个已有生产工作流。
|
||||
|
||||
@@ -338,8 +338,9 @@
|
||||
| `TASK-release-path-production-20261014.md` | `PROGRESS-release-path-production-20261014.md` | **发布通路 + 生产首发**:产品 10-14 授权改生产工作流(发布 SHA = 最近门禁成功提交 + 其后纯文档);63 个迁移清单与演练脚本、首发 runbook;生产拒绝测试验证码。Grok 不 dispatch 生产工作流 | 第 2 次打回后续修 | `codex/release-path-production-20261014`(BUG-1348、BUG-1349) |
|
||||
| `TASK-release-path-production-fix-20261014.md` | `PROGRESS-release-path-production-fix-20261014.md` | **#74 第 1 次打回**:发布规则成立;测试写死 Windows 路径(Linux 全红)、主机演练依赖主机 node 且会留下生产数据副本、docs-only 头时回滚被拒;另补 CHECK_RELEASE_GATE 断言与 OTP 守卫规范化 | 已修,第 2 次仍未过 | `codex/release-path-production-20261014` |
|
||||
| `TASK-release-path-production-fix2-20261014.md` | `PROGRESS-release-path-production-fix2-20261014.md` | **#74 修复单 2(产品选 A)**:恢复点要带属主与权限,旧格式不得回退;演练按备份里的授权比对,并在真实 PostgreSQL 17 上跑通一次 | 待验收 | `codex/release-path-production-20261014` |
|
||||
| `TASK-beta-safety-net-20261014.md` | `PROGRESS-beta-safety-net-20261014.md` | **内测安全网**:每日备份 + 恢复演练、探活告警、本地错误汇总、日志轮转、按 requestId 查日志。发布通路与 #58 已在开工时的 `origin/staging` 上。主机安装和一次演练留给产品 | 待验收 | `codex/beta-safety-net-20261014`(BUG-1355~1357) |
|
||||
| `TASK-beta-safety-net-fix-20261015.md` | `PROGRESS-beta-safety-net-fix-20261015.md` | **#75 产品选 A**:每日备份照常做,但不改当前恢复点;另记 `daily-backup-latest.env` | 待验收 | `codex/beta-safety-net-20261014`(BUG-1358) |
|
||||
| `TASK-beta-safety-net-20261014.md` | `PROGRESS-beta-safety-net-20261014.md` | **内测安全网**:每日备份 + 恢复演练、探活告警、本地错误汇总、日志轮转、按 requestId 查日志。发布通路与 #58 已在开工时的 `origin/staging` 上。主机安装和一次演练留给产品 | 待验收 | `codex/beta-safety-net-20261014`(BUG-1358~1360) |
|
||||
| `TASK-beta-safety-net-fix-20261015.md` | `PROGRESS-beta-safety-net-fix-20261015.md` | **#75 产品选 A**:每日备份照常做,但不改当前恢复点;另记 `daily-backup-latest.env` | 待验收 | `codex/beta-safety-net-20261014`(BUG-1361) |
|
||||
| `TASK-beta-safety-net-fix2-20261015.md` | — | **#75 修复单 2**:生产侧全过;飞书测试 env 文件权限 0644 在 Linux 上失败(会挡发布门禁);新测试不在 quick 门禁;磁盘 ≥70% 后备份静默停摆;监控脚本缺 sudo 与 env 文件 | 待领取 | `codex/beta-safety-net-20261014` |
|
||||
| `TASK-beta-capacity-20261014.md` | — | **内测容量**:按用户限流、重计算短排队与对话重试、兑换码当邀请码、staging 10/20 并发压测。**等 #58 合入后开工,压测等 #57 合入** | 待前置 | — |
|
||||
| `TASK-beta-capacity-fix-20261015.md` | — | **#76 第 1 次打回**:`a54f75e5` 限流与短排队逻辑通过;源码文本断言未跟改、插入新测试使隐私白名单行号错位 | 待领取 | `codex/beta-capacity-20261014` |
|
||||
| `TASK-special-lagnas-native-20261008.md` | `PROGRESS-special-lagnas-native-20261008.md` | **特殊上升点原生计算(线上缺陷)**:sync6 U6 运行时调 PyJHora,生产镜像没有 PyJHora → staging 上 Hora/Ghati/Sree/Indu/Pranapada 全 blocked;原生实现、PyJHora 只作冻结对照、快速门禁止运行时 import jhora;最先做 | 已验收(已部署 staging `795b8845`,BUG-1273) | `codex/special-lagnas-native-20261008`(未上线;部署以 `/api/health` 为准) |
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# TASK · 内测安全网修复单 2(issue #75)
|
||||
|
||||
- 日期:2026-10-15
|
||||
- 前序:`TASK-beta-safety-net-20261014.md`、`TASK-beta-safety-net-fix-20261015.md`(产品选 A)
|
||||
- 被验 SHA:`5c0d895391b5d0806c98de669a2e9cdcd61e3335`(`codex/beta-safety-net-20261014`)
|
||||
- 执行方:Grok,同一分支续做(先 merge 最新 `origin/staging`,不改写已推历史)
|
||||
- 红线不变:仍不改已有四个生产工作流与 staging 工作流、`gated-paths.txt`、`is-docs-only-range.sh`;不点生产按钮
|
||||
|
||||
## 1. 验收结论:面向生产的部分全部过;一条阻断 + 三条要一起修
|
||||
|
||||
已确认成立:每日工作流只从 main 跑、与恢复点工作流同一把锁 / 同样的密钥处理、不回显密钥;每日模式**任何路径都不写** `recovery-baseline.env`(只有手动分支写),失败时什么状态都不写;`daily-backup-latest.env` 只有 4 个键;发布 / 迁移 / 回退不读它;手动恢复点(含 `RECOVERY_FORMAT=2` 与权限比对)不变;清理脚本只认 `^production-daily-\d{8}T\d{6}Z\.dump\.enc$` 的普通文件,碰不到 `production-pre-migration-*`;错误分类只外发校验过的 errorCode / 路由 / UUID,不发用户内容;compose 只加了 6 个服务一致的 `logging`。
|
||||
|
||||
### B1(阻断)· 测试在 Linux 上失败
|
||||
|
||||
`tests/test_beta_safety_net.py` 约 616–634 行 `test_feishu_payload_keeps_only_allowed_fields`:约 76–77 行用 `write_text` 写 `monitor.env`,默认权限 0644;`deploy/monitor/notify-feishu.sh` 约 18–23 行在 Linux 上拒绝非 0600 的环境文件 → 退出 1。macOS / Windows 上跳过了这项检查所以没发现。`release-quality-gate.yml` 在 Linux 上跑全树 pytest,提升 main 会被它挡住。
|
||||
- 修:测试里写完后 `os.chmod(env_file, 0o600)`。脚本里的检查是对的,不改。
|
||||
|
||||
### N1 · 新测试不进 staging 门禁
|
||||
|
||||
`scripts/run_quality_gate.py` 的 quick 清单(约 55–90 行)没有 `tests/test_beta_safety_net.py`,合同只在发布时才跑——B1 就是这样漏过去的。把它加进 quick 清单。
|
||||
|
||||
### N2 · 磁盘 ≥70% 后备份静默停摆
|
||||
|
||||
`run-production-recovery.sh` 约 149–153 行磁盘 ≥70% 拒绝;`backup-production-daily.yml` 约 145–146 行只在备份成功后才清理旧文件 → 一旦到 70%,旧文件永不清理、每天都失败;磁盘告警 80% 才响,也没人检查 `daily-backup-latest.env` 有多旧。
|
||||
- 修:清理放在备份**之前**(或 `if: always()`);`healthcheck-alert.sh` 增加「最近一次每日备份超过 36 小时」告警(读 `daily-backup-latest.env` 的 `verified_at`)。加测试。
|
||||
|
||||
### N3 · 两个监控脚本在生产主机上大概率跑不起来
|
||||
|
||||
`deploy/monitor/error-digest.sh` 约 54–59 行、`deploy/find-request-log.sh` 约 300–307 行直接调 `docker`,其它生产脚本与恢复演练都用 `sudo -n docker`;也没导出 `DATABASE_ENV_FILE`,`docker-compose.postgres.yml` 会回落到不存在的 `../.env.staging.database`,Compose 拒绝加载。错误汇总由 cron 每 5 分钟跑,会每次失败、什么都不发。
|
||||
- 修:统一 `sudo -n docker`,显式传入生产的 compose 文件与 env 文件路径(与 `run-production-deploy.sh` 同一写法);测试断言命令行形状。
|
||||
|
||||
## 2. 顺手修(不阻断,一并做)
|
||||
|
||||
- `notify-feishu.sh` 约 96–101 行:`curl` 加 `--fail`,并检查飞书返回体里的 `code == 0`;失败时不记成「已发送」。
|
||||
- `error-digest.sh`:同一类告警 30 分钟内不重复发。
|
||||
- `find-request-log.sh`:真实路径按时间戳排序(去掉 compose 前缀后再排),与 README 说法一致。
|
||||
|
||||
## 3. 验收标准
|
||||
|
||||
- Linux:`python3 -m pytest tests/test_beta_safety_net.py -q` 全绿;Claude 全量 pytest、前端全量无新增;`bash -n` 全过;quick 门禁包含新测试。
|
||||
- 交回前本机跑上面那条 pytest(Windows 上请用 WSL 或在测试里显式断言 0600 已生效)。
|
||||
|
||||
## 4. 让步顺序
|
||||
|
||||
B1、N1、N3 不可让;N2 的「过旧告警」可写理由后让;第 2 节可让。
|
||||
@@ -0,0 +1,6 @@
|
||||
# 内测容量 · 真机两条(2026-10-14)
|
||||
|
||||
用测试账号。不要把出生资料、姓名、邮箱写进记录。
|
||||
|
||||
1. 快速连发触发限流。登录后在一分钟里连续发送超过 10 次普通对话。预期:出现「这一会儿问得有点密,先停一下。」这一轮不扣点。等提示里的时间过去再发一次,可以正常问。
|
||||
2. 排队提示出现与消失。需要两个人同时占着计算(或用压测脚本打满两个名额)时,第三个人发普通对话。预期:进度行先出现「人有点多,排一下队」,排上之后这句消失,回到正在计算。若一直排不上,出现「人有点多,这一轮没排上,点数已经退回。」余额与发送前一致。
|
||||
+1
-1
@@ -907,7 +907,7 @@ or user IDs.
|
||||
|
||||
Agent 的 live 标记只有 `InlineSpinner` 一种。曾经并存的 canvas 小球(`thinking-orbs`)已移除,不得再引入第二种 live 标记。
|
||||
|
||||
**普通对话发出即有反馈(2026-09-28,TASK-consult-first-frame-and-pacing,BUG-1074)。** 按下发送的那一帧,回答行就带着步骤栏,唯一一行是排队行「收到,正在看你的问题…」(`QUEUED_TIMELINE_ROW`,行内 `InlineSpinner` + shimmer,summary 写「正在分析」)。服务端流的第一个字节也是这句(`activity` 事件,phase `received`),所以那一帧到达时行不换字。不再有「分类结果回来之前整行静默」:只有服务端判定为寒暄的回复(`run.completed` 带 `responseKind: "smalltalk"`)才把步骤栏收掉、只留一句话,寒暄前会闪约一秒这一行,属于流式生成中的正常形态。首事件之后的行照旧由服务端事件推动;不新增第二种 live 标记,不加 spinner 面板或骨架。
|
||||
**普通对话发出即有反馈(2026-09-28,TASK-consult-first-frame-and-pacing,BUG-1074)。** 按下发送的那一帧,回答行就带着步骤栏,唯一一行是排队行「收到,正在看你的问题…」(`QUEUED_TIMELINE_ROW`,行内 `InlineSpinner` + shimmer,summary 写「正在分析」)。服务端流的第一个字节也是这句(`activity` 事件,phase `received`),所以那一帧到达时行不换字。不再有「分类结果回来之前整行静默」:只有服务端判定为寒暄的回复(`run.completed` 带 `responseKind: "smalltalk"`)才把步骤栏收掉、只留一句话,寒暄前会闪约一秒这一行,属于流式生成中的正常形态。首事件之后的行照旧由服务端事件推动;不新增第二种 live 标记,不加 spinner 面板或骨架。重计算名额满了时,计算那一行先换成「人有点多,排一下队」(`activity` phase `compute-queue`,仍是原来的 live 行),排上之后回到「正在计算本命盘…」。排不上则整轮失败,文案是「人有点多,这一轮没排上,点数已经退回。」
|
||||
|
||||
校正面的所有等待复用行内等待:进入前的 hydration 在揭幕之前完成,进入后唯一的等待形态是时间线 live 行(含「正在准备下一个问题…」这一条独立 live 行)。打字回答的 live 行从发出那一帧起就是阶段句「收到,正在对照你的档案…」,按服务端 `turn.progress` 换成「正在记下这件事…」「正在重新对照盘面…」「正在准备下一个问题…」,属于流式生成中(BUG-1047)。区间交付卡只挂在最新那条采用旁白下面,不得留在更早的采集/区分题下。有未答的采集或选择题时卡仍在,「更像这个」置灰并写「先答完上面这道,再选时间」,不得整卡消失。卡上至多三列并排,相同性格句只写一次,点「更像这个」即采用该列分钟,按钮显示「正在采用…」或「已采用」。采用过程中整张卡留在原处,不得因 `busy` 卸掉。采用后若还有前事核对,服务端写一条自己的采用消息(「已按 HH:MM 排盘…」),核对题的四个选项挂在这条消息下,进行中那一行由它替换,不挂到上一题的回复下(BUG-1146)。采用后前事核对结束走 `verified_idle`:一行收尾文案跟在卡片下面、与助手列对齐,没有 live 行、没有重载、没有采用状态条。卡片与右栏细则见 §11、§12。
|
||||
|
||||
|
||||
@@ -480,3 +480,7 @@ Jyotisha 的可见文案是产品的一部分。正确性红线(真实性、
|
||||
- 不写健康诊断、投资建议、法律意见;不写「一定」「必然」「注定」。
|
||||
- 出生时间没确认时不用宫位,只按月亮所在星座写,caution 固定为「出生时间尚未确认,这里只看大方向,不依赖具体分钟。」
|
||||
|
||||
## 人多与问得太密(2026-10-15)
|
||||
|
||||
不写「系统繁忙」「请稍后重试」。计算要等空位时,进度行只写「人有点多,排一下队」。这一轮最终没排上:「人有点多,这一轮没排上,点数已经退回。」对话太密:「这一会儿问得有点密,先停一下。」校正回合太密:「这一会儿校正问得有点密,先停一下。」这一小时报告太多:「这一小时的报告已经够多了,先停一下。」
|
||||
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
/**
|
||||
* Staging-only smoke for 10 or 20 concurrent consultations.
|
||||
*
|
||||
* Not imported by frontend/src. Reads STAGING_TEST_OTP from the environment
|
||||
* and never prints it. Refuses a production host. Does not sample server
|
||||
* CPU or memory; that needs a staging shell this process does not have.
|
||||
*
|
||||
* node node_modules/tsx/dist/cli.mjs scripts/research/beta-load-smoke.mts --users 10
|
||||
*/
|
||||
const usersArg = process.argv.find((arg) => arg.startsWith("--users="));
|
||||
const usersFlag = process.argv.indexOf("--users");
|
||||
const requestedUsers = Number(usersArg ? usersArg.slice("--users=".length) : process.argv[usersFlag + 1]);
|
||||
const baseUrl = (process.env.STAGING_BASE_URL || "https://staging.jyotisha.chat").replace(/\/$/, "");
|
||||
const otp = process.env.STAGING_TEST_OTP || "";
|
||||
const emailDomain = process.env.STAGING_TEST_EMAIL_DOMAIN || "example.test";
|
||||
|
||||
function refuse(message: string): never {
|
||||
console.error(message);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
let host = "";
|
||||
try {
|
||||
host = new URL(baseUrl).host;
|
||||
} catch {
|
||||
refuse("STAGING_BASE_URL is not a URL");
|
||||
}
|
||||
if (host === "jyotisha.chat" || host === "www.jyotisha.chat" || host === "admin.jyotisha.chat") {
|
||||
refuse("refusing production host");
|
||||
}
|
||||
if (!/^\d{6}$/.test(otp)) refuse("STAGING_TEST_OTP is not set");
|
||||
if (requestedUsers !== 10 && requestedUsers !== 20) refuse("pass --users 10 or --users 20");
|
||||
if (!/^[a-z0-9.-]+$/.test(emailDomain)) refuse("STAGING_TEST_EMAIL_DOMAIN is not a plain domain");
|
||||
|
||||
type Sample = {
|
||||
kind: "consult" | "rectification";
|
||||
status: number;
|
||||
firstByteMs: number | null;
|
||||
totalMs: number;
|
||||
failed: boolean;
|
||||
http429: boolean;
|
||||
http5xx: boolean;
|
||||
};
|
||||
|
||||
const samples: Sample[] = [];
|
||||
const CONSULT_QUEUE_FAILED = "人有点多,这一轮没排上,点数已经退回。";
|
||||
|
||||
function percentile(values: number[], ratio: number): number | null {
|
||||
if (values.length === 0) return null;
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
const index = Math.min(sorted.length - 1, Math.max(0, Math.ceil(sorted.length * ratio) - 1));
|
||||
return sorted[index] ?? null;
|
||||
}
|
||||
|
||||
async function request(jar: Map<string, string>, path: string, init: RequestInit = {}): Promise<Response> {
|
||||
const headers = new Headers(init.headers);
|
||||
if (jar.size > 0) headers.set("cookie", [...jar].map(([key, value]) => `${key}=${value}`).join("; "));
|
||||
const response = await fetch(`${baseUrl}${path}`, { ...init, headers, redirect: "manual" });
|
||||
const setCookies = typeof response.headers.getSetCookie === "function" ? response.headers.getSetCookie() : [];
|
||||
for (const line of setCookies) {
|
||||
const pair = line.split(";")[0] ?? "";
|
||||
const eq = pair.indexOf("=");
|
||||
if (eq > 0) jar.set(pair.slice(0, eq).trim(), pair.slice(eq + 1).trim());
|
||||
}
|
||||
return response;
|
||||
}
|
||||
|
||||
async function timed(jar: Map<string, string>, path: string, body: unknown, kind: Sample["kind"]): Promise<void> {
|
||||
const started = Date.now();
|
||||
let status = 0;
|
||||
let firstByteMs: number | null = null;
|
||||
let failed = false;
|
||||
try {
|
||||
const response = await request(jar, path, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
status = response.status;
|
||||
const reader = response.body?.getReader();
|
||||
if (reader) {
|
||||
let text = "";
|
||||
for (;;) {
|
||||
const chunk = await reader.read();
|
||||
if (firstByteMs === null && (chunk.value || chunk.done)) firstByteMs = Date.now() - started;
|
||||
if (chunk.value) text += new TextDecoder().decode(chunk.value);
|
||||
if (chunk.done) break;
|
||||
}
|
||||
failed = text.includes('"type":"run.failed"') || text.includes(CONSULT_QUEUE_FAILED);
|
||||
} else {
|
||||
firstByteMs = Date.now() - started;
|
||||
failed = status >= 400;
|
||||
}
|
||||
} catch {
|
||||
status = 0;
|
||||
failed = true;
|
||||
}
|
||||
samples.push({
|
||||
kind,
|
||||
status,
|
||||
firstByteMs,
|
||||
totalMs: Date.now() - started,
|
||||
failed,
|
||||
http429: status === 429,
|
||||
http5xx: status >= 500 || status === 0,
|
||||
});
|
||||
}
|
||||
|
||||
async function login(index: number): Promise<Map<string, string> | null> {
|
||||
const jar = new Map<string, string>();
|
||||
const email = `capacity-smoke-${index}@${emailDomain}`;
|
||||
const sent = await request(jar, "/api/auth/email-otp/send-verification-otp", {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({ email, type: "sign-in" }),
|
||||
});
|
||||
if (!sent.ok) return null;
|
||||
const signed = await request(jar, "/api/auth/sign-in/email-otp", {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({ email, otp }),
|
||||
});
|
||||
if (!signed.ok || jar.size === 0) return null;
|
||||
return jar;
|
||||
}
|
||||
|
||||
async function prepareAccount(jar: Map<string, string>): Promise<boolean> {
|
||||
const saved = await request(jar, "/api/account", {
|
||||
method: "PATCH",
|
||||
headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
name: "容量冒烟",
|
||||
birth_date: "1990-06-15",
|
||||
reported_birth_time: "12:00",
|
||||
birth_time_source: "family_exact",
|
||||
birth_time_period: null,
|
||||
birth_time_clue: null,
|
||||
uncertainty_before_minutes: 0,
|
||||
uncertainty_after_minutes: 0,
|
||||
country_code: "CN",
|
||||
province_code: "110000",
|
||||
city_code: "110000",
|
||||
district_code: "110101",
|
||||
latitude: 39.9042,
|
||||
longitude: 116.4074,
|
||||
timezone_offset: 8,
|
||||
}),
|
||||
});
|
||||
return saved.ok;
|
||||
}
|
||||
|
||||
async function oneUser(index: number): Promise<string[]> {
|
||||
const deleted: string[] = [];
|
||||
const jar = await login(index);
|
||||
if (!jar) {
|
||||
samples.push({ kind: "consult", status: 0, firstByteMs: null, totalMs: 0, failed: true, http429: false, http5xx: true });
|
||||
return deleted;
|
||||
}
|
||||
if (!await prepareAccount(jar)) {
|
||||
samples.push({ kind: "consult", status: 0, firstByteMs: null, totalMs: 0, failed: true, http429: false, http5xx: true });
|
||||
return deleted;
|
||||
}
|
||||
const models = await request(jar, "/api/models");
|
||||
const catalog = models.ok ? await models.json().catch(() => null) as { defaultModelId?: string } | null : null;
|
||||
const modelId = catalog?.defaultModelId;
|
||||
if (!modelId) {
|
||||
samples.push({ kind: "consult", status: 0, firstByteMs: null, totalMs: 0, failed: true, http429: false, http5xx: true });
|
||||
return deleted;
|
||||
}
|
||||
const sessionId = crypto.randomUUID();
|
||||
const created = await request(jar, "/api/sessions", {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
id: sessionId,
|
||||
title: "容量冒烟",
|
||||
theme: "general",
|
||||
model_id: modelId,
|
||||
messages: [],
|
||||
session_type: "consultation",
|
||||
rectification_case_id: null,
|
||||
chart_profile_role: "self",
|
||||
}),
|
||||
});
|
||||
if (!created.ok) {
|
||||
samples.push({ kind: "consult", status: created.status, firstByteMs: null, totalMs: 0, failed: true, http429: created.status === 429, http5xx: created.status >= 500 });
|
||||
return deleted;
|
||||
}
|
||||
const chart = {
|
||||
year: 1990,
|
||||
month: 6,
|
||||
day: 15,
|
||||
hour: 12,
|
||||
minute: 0,
|
||||
lat: 39.9042,
|
||||
lon: 116.4074,
|
||||
tz: 8,
|
||||
city: "北京",
|
||||
ayanamsa: "lahiri",
|
||||
};
|
||||
await timed(jar, "/api/consult", {
|
||||
...chart,
|
||||
requestId: crypto.randomUUID(),
|
||||
sessionId,
|
||||
modelId,
|
||||
consultationMode: "verified_chart",
|
||||
theme: "general",
|
||||
question: "今年工作整体顺不顺?",
|
||||
history: [],
|
||||
}, "consult");
|
||||
await timed(jar, "/api/consult", {
|
||||
...chart,
|
||||
requestId: crypto.randomUUID(),
|
||||
sessionId,
|
||||
modelId,
|
||||
consultationMode: "verified_chart",
|
||||
theme: "general",
|
||||
question: "那接下来几个月呢?",
|
||||
history: [],
|
||||
}, "consult");
|
||||
if (index < 3) {
|
||||
const opened = await request(jar, "/api/rectification/cases/open", {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({ intent: "homepage", requestId: crypto.randomUUID() }),
|
||||
});
|
||||
const openedBody = opened.ok ? await opened.json().catch(() => null) as { caseId?: string; sessionId?: string } | null : null;
|
||||
if (openedBody?.caseId && openedBody.sessionId) {
|
||||
for (const message of ["我小时候搬家过一次。", "那次大概是上小学之前。"]) {
|
||||
await timed(jar, "/api/rectification/agent", {
|
||||
caseId: openedBody.caseId,
|
||||
sessionId: openedBody.sessionId,
|
||||
requestId: crypto.randomUUID(),
|
||||
action: "message",
|
||||
message,
|
||||
}, "rectification");
|
||||
}
|
||||
deleted.push(openedBody.sessionId);
|
||||
} else {
|
||||
samples.push({
|
||||
kind: "rectification",
|
||||
status: opened.status,
|
||||
firstByteMs: null,
|
||||
totalMs: 0,
|
||||
failed: true,
|
||||
http429: opened.status === 429,
|
||||
http5xx: opened.status >= 500 || opened.status === 0,
|
||||
});
|
||||
}
|
||||
}
|
||||
const removed = await request(jar, `/api/sessions/${sessionId}`, { method: "DELETE" });
|
||||
if (removed.ok) deleted.push(sessionId);
|
||||
return deleted;
|
||||
}
|
||||
|
||||
const startedAt = Date.now();
|
||||
const cleaned = (await Promise.all(Array.from({ length: requestedUsers }, (_, index) => oneUser(index)))).flat();
|
||||
const consults = samples.filter((sample) => sample.kind === "consult");
|
||||
const finalFailures = consults.filter((sample) => sample.failed || sample.http429 || sample.http5xx);
|
||||
const firstBytes = consults.map((sample) => sample.firstByteMs).filter((value): value is number => value !== null);
|
||||
const totals = consults.map((sample) => sample.totalMs);
|
||||
const summary = {
|
||||
users: requestedUsers,
|
||||
host,
|
||||
requests: samples.length,
|
||||
consults: consults.length,
|
||||
http429: samples.filter((sample) => sample.http429).length,
|
||||
http5xx: samples.filter((sample) => sample.http5xx).length,
|
||||
finalFailures: finalFailures.length,
|
||||
finalFailureRatio: consults.length === 0 ? null : Number((finalFailures.length / consults.length).toFixed(4)),
|
||||
firstByteP95Ms: percentile(firstBytes, 0.95),
|
||||
totalP95Ms: percentile(totals, 0.95),
|
||||
sessionsDeleted: cleaned.length,
|
||||
serverSample: "not taken; this process has no staging docker stats",
|
||||
elapsedMs: Date.now() - startedAt,
|
||||
};
|
||||
console.log(JSON.stringify(summary));
|
||||
@@ -129,6 +129,12 @@ import {
|
||||
import { persistGuardedSessionTitle } from "@/lib/session-title-guard";
|
||||
import { generateSessionTitle, shouldGenerateSessionTitle } from "@/lib/session-title-agent";
|
||||
import { z } from "zod";
|
||||
import { CONSULTATION_QUEUE_FAILED_MESSAGE, isConsultationRateLimitFailure, retryOnConsultationRateLimit } from "@/lib/consultation-workflow-retry";
|
||||
import {
|
||||
consumeUserRateLimitSet,
|
||||
userRateLimitDeniedResponse,
|
||||
userRateLimitMessages,
|
||||
} from "@/lib/request-rate-limit";
|
||||
|
||||
export const runtime = "nodejs";
|
||||
export const maxDuration = 480;
|
||||
@@ -367,6 +373,11 @@ export async function POST(request: Request) {
|
||||
const currentDate = chinaCalendarDate(requestTime);
|
||||
|
||||
const userId = user.id;
|
||||
const limited = consumeUserRateLimitSet(["consultMinute", "consultHour"], userId);
|
||||
if (!limited.ok) {
|
||||
const denied = userRateLimitDeniedResponse(limited.retryAfterSeconds, userRateLimitMessages.consult);
|
||||
return NextResponse.json(denied.body, { status: denied.status, headers: denied.headers });
|
||||
}
|
||||
const requestId = parsed.data.requestId;
|
||||
const sessionId = parsed.data.sessionId;
|
||||
const consultationTheme = parsed.data.theme;
|
||||
@@ -1786,9 +1797,11 @@ export async function POST(request: Request) {
|
||||
theme: parsed.data.theme,
|
||||
});
|
||||
const workflowContext = applyBirthTimeModeToWorkflowContext(
|
||||
await runConsultationWorkflow(toolInput, {
|
||||
foreground: true,
|
||||
plan: prepared.preReserveResult,
|
||||
await retryOnConsultationRateLimit({
|
||||
run: () => runConsultationWorkflow(toolInput, {
|
||||
foreground: true,
|
||||
plan: prepared.preReserveResult,
|
||||
}),
|
||||
}),
|
||||
consultationMode,
|
||||
{ birthTimeSource: prepared.serverChart.truth.birthTimeSource },
|
||||
@@ -1894,12 +1907,15 @@ export async function POST(request: Request) {
|
||||
console.error(
|
||||
`[consult] generation failed request=${requestId} model=${selectedModel.id} reason=${reason}`,
|
||||
);
|
||||
const rateLimited = isConsultationRateLimitFailure(error);
|
||||
return NextResponse.json(
|
||||
{
|
||||
error: "暂时无法生成解读",
|
||||
message: "咨询服务暂时不可用,请稍后再试。",
|
||||
recovery: "稍后重试,或换一个模型继续。",
|
||||
},
|
||||
rateLimited
|
||||
? { error: CONSULTATION_QUEUE_FAILED_MESSAGE, message: CONSULTATION_QUEUE_FAILED_MESSAGE }
|
||||
: {
|
||||
error: "暂时无法生成解读",
|
||||
message: "咨询服务暂时不可用,请稍后再试。",
|
||||
recovery: "稍后重试,或换一个模型继续。",
|
||||
},
|
||||
{ status: 503 },
|
||||
);
|
||||
}
|
||||
|
||||
@@ -18,6 +18,11 @@ import {
|
||||
replyToDeclaredBirthWindow,
|
||||
} from "@/lib/rectification-agentic/v9/agent-route-typed-message";
|
||||
import { respondWithRectificationAgentTurn } from "@/lib/rectification-agentic/v9/agent-route-agent-turn";
|
||||
import {
|
||||
consumeUserRateLimitSet,
|
||||
userRateLimitDeniedResponse,
|
||||
userRateLimitMessages,
|
||||
} from "@/lib/request-rate-limit";
|
||||
|
||||
export const runtime = "nodejs";
|
||||
export const maxDuration = 240;
|
||||
@@ -66,6 +71,20 @@ export async function POST(request: Request) {
|
||||
const frozen = await refuseWhenAccountDeletionPending(userId);
|
||||
if (frozen) return frozen;
|
||||
|
||||
// Opening a case is POST /api/rectification/cases/open and is not counted.
|
||||
// Every turn on this route is, and the check is before any credit reserve.
|
||||
const limited = consumeUserRateLimitSet(
|
||||
["rectificationTurnMinute", "rectificationTurnHour"],
|
||||
userId,
|
||||
);
|
||||
if (!limited.ok) {
|
||||
const denied = userRateLimitDeniedResponse(
|
||||
limited.retryAfterSeconds,
|
||||
userRateLimitMessages.rectificationTurn,
|
||||
);
|
||||
return NextResponse.json(denied.body, { status: denied.status, headers: denied.headers });
|
||||
}
|
||||
|
||||
if (action === "opening") {
|
||||
const consistency = await finalizeSuccessfulTurnExit({
|
||||
accounting: accounting as never, userId, caseId, action, segmentOpening: true,
|
||||
|
||||
@@ -5,6 +5,7 @@ import { LONGFORM_SNAPSHOT_SECTION_ID } from "@/lib/personal-report-longform-sna
|
||||
import { resolveMissingBirthTimezoneOffset } from "@/lib/birth-profile-timezone";
|
||||
import { stampReportBirthFingerprint } from "@/lib/report-birth-binding";
|
||||
import { runConsultationWorkflow } from "@/mastra";
|
||||
import { consumeUserRateLimitSet, userRateLimitMessages } from "@/lib/request-rate-limit";
|
||||
import { createPersonalReportAgent } from "@/mastra/personal-report";
|
||||
import { loadLanguageModelCatalog } from "@/lib/model-catalog";
|
||||
import {
|
||||
@@ -68,8 +69,8 @@ function sanitizedErrorCode(error: unknown): string {
|
||||
return "UnknownError";
|
||||
}
|
||||
|
||||
function toNextResponse(response: { status: number; body: Record<string, unknown> }) {
|
||||
return NextResponse.json(response.body, { status: response.status });
|
||||
function toNextResponse(response: { status: number; body: Record<string, unknown>; headers?: Readonly<Record<string, string>> }) {
|
||||
return NextResponse.json(response.body, { status: response.status, headers: response.headers });
|
||||
}
|
||||
|
||||
function listReportView(
|
||||
@@ -308,6 +309,13 @@ export async function POST(request: Request) {
|
||||
},
|
||||
persistence,
|
||||
model: defaultModel,
|
||||
consumeReportRateLimit: userId
|
||||
? () => {
|
||||
const decision = consumeUserRateLimitSet(["reportGenerateHour"], userId);
|
||||
if (decision.ok) return decision;
|
||||
return { ...decision, message: userRateLimitMessages.reportGenerate };
|
||||
}
|
||||
: undefined,
|
||||
billing: {
|
||||
reserve: async ({ userId: billingUserId, requestId, modelId }) => {
|
||||
try {
|
||||
|
||||
@@ -61,6 +61,7 @@ export const ChatMessageRow = memo(function ChatMessageRow({
|
||||
received: "working",
|
||||
"loading-method": "searching",
|
||||
"chart-calculation": "solving",
|
||||
"compute-queue": "working",
|
||||
"evidence-validation": "working",
|
||||
"answer-composition": "composing",
|
||||
} as const)[message.activity.phase]
|
||||
|
||||
@@ -4,6 +4,8 @@
|
||||
* turn is classified or the chart is read (TASK-consult-first-frame-and-pacing-20260928 D1).
|
||||
*/
|
||||
export const CONSULTATION_RECEIVED_LABEL = "收到,正在看你的问题…";
|
||||
export const CONSULTATION_QUEUE_LABEL = "人有点多,排一下队";
|
||||
export const CONSULTATION_QUEUE_FAILED_MESSAGE = "人有点多,这一轮没排上,点数已经退回。";
|
||||
export const CONSULTATION_LOADING_METHOD_LABEL = "正在读取印度占星分析规则…";
|
||||
export const CONSULTATION_CHART_CALCULATION_LABEL = "正在计算本命盘…";
|
||||
export const CONSULTATION_EVIDENCE_VALIDATION_LABEL = "正在核对可用证据…";
|
||||
|
||||
@@ -10,6 +10,9 @@ export const publicActivityPhaseSchema = z.enum([
|
||||
"received",
|
||||
"loading-method",
|
||||
"chart-calculation",
|
||||
// Heavy compute is full. The calculate row keeps this sentence while the
|
||||
// consultation retries, then returns to the chart-calculation label.
|
||||
"compute-queue",
|
||||
"evidence-validation",
|
||||
"answer-composition",
|
||||
]);
|
||||
|
||||
@@ -70,7 +70,7 @@ export function reduceConsultationTimeline(
|
||||
return applyCalculateHints(created);
|
||||
}
|
||||
if (event.type === "activity") {
|
||||
if (event.phase === "chart-calculation") {
|
||||
if (event.phase === "chart-calculation" || event.phase === "compute-queue") {
|
||||
return patchRow(state, CALCULATE_ID, {
|
||||
status: "live",
|
||||
label: event.label,
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
import { ConsultationWorkflowError } from "../mastra/consultation-workflow.ts";
|
||||
import {
|
||||
CONSULTATION_QUEUE_FAILED_MESSAGE,
|
||||
CONSULTATION_QUEUE_LABEL,
|
||||
} from "./consultation-activity-labels.ts";
|
||||
|
||||
export { CONSULTATION_QUEUE_FAILED_MESSAGE, CONSULTATION_QUEUE_LABEL };
|
||||
|
||||
export const CONSULTATION_QUEUE_MAX_RETRIES = 2;
|
||||
export const CONSULTATION_QUEUE_MAX_WAIT_MS = 15_000;
|
||||
|
||||
export function isConsultationRateLimitFailure(error: unknown): boolean {
|
||||
if (error instanceof ConsultationWorkflowError) return error.code === "workflow_rate_limited";
|
||||
if (error instanceof Error && error.cause !== undefined) return isConsultationRateLimitFailure(error.cause);
|
||||
return false;
|
||||
}
|
||||
|
||||
function retryDelaySeconds(error: unknown): number | undefined {
|
||||
if (error instanceof ConsultationWorkflowError && error.code === "workflow_rate_limited") {
|
||||
return error.retryAfterSeconds ?? 2;
|
||||
}
|
||||
if (error instanceof Error && error.cause !== undefined) return retryDelaySeconds(error.cause);
|
||||
return undefined;
|
||||
}
|
||||
|
||||
export function consultationRunFailureMessage(error: unknown, fallback: string): string {
|
||||
return isConsultationRateLimitFailure(error) ? CONSULTATION_QUEUE_FAILED_MESSAGE : fallback;
|
||||
}
|
||||
|
||||
/**
|
||||
* Retry one consultation workflow call when the heavy-compute gate answers
|
||||
* 429. At most two retries, and the sleeps together stay within 15 seconds.
|
||||
*/
|
||||
export async function retryOnConsultationRateLimit<T>(input: Readonly<{
|
||||
run: () => Promise<T>;
|
||||
onQueue?: () => void | Promise<void>;
|
||||
sleep?: (ms: number) => Promise<void>;
|
||||
}>): Promise<T> {
|
||||
const sleep = input.sleep ?? ((ms: number) => new Promise((resolve) => setTimeout(resolve, ms)));
|
||||
let retries = 0;
|
||||
let waitedMs = 0;
|
||||
for (;;) {
|
||||
try {
|
||||
return await input.run();
|
||||
} catch (error) {
|
||||
const retryAfterSeconds = retryDelaySeconds(error);
|
||||
if (retryAfterSeconds === undefined || retries >= CONSULTATION_QUEUE_MAX_RETRIES) throw error;
|
||||
const remainingMs = CONSULTATION_QUEUE_MAX_WAIT_MS - waitedMs;
|
||||
if (remainingMs <= 0) throw error;
|
||||
const waitMs = Math.min(retryAfterSeconds * 1000, remainingMs);
|
||||
retries += 1;
|
||||
waitedMs += waitMs;
|
||||
await input.onQueue?.();
|
||||
await sleep(waitMs);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -63,7 +63,11 @@ export const personalReportCreateRequestSchema = z.object({
|
||||
|
||||
export type PersonalReportCreateRequest = z.infer<typeof personalReportCreateRequestSchema>;
|
||||
|
||||
export type ReportRouteResponse = Readonly<{ status: number; body: Record<string, unknown> }>;
|
||||
export type ReportRouteResponse = Readonly<{
|
||||
status: number;
|
||||
body: Record<string, unknown>;
|
||||
headers?: Readonly<Record<string, string>>;
|
||||
}>;
|
||||
|
||||
export type ReportServicePort = Pick<
|
||||
PersonalReportService,
|
||||
@@ -231,6 +235,10 @@ export type ReportCreateCoreDeps = Readonly<{
|
||||
persistence: ReportServicePort;
|
||||
model: Readonly<{ id: string; configVersion?: number; inputCostMicrousdPerMillion?: number; outputCostMicrousdPerMillion?: number }> | null;
|
||||
billing?: ReportBillingPort;
|
||||
/** Hourly per-user cap. Checked after a replay and the daily cap, before reserve. */
|
||||
consumeReportRateLimit?: () =>
|
||||
| { readonly ok: true }
|
||||
| { readonly ok: false; readonly retryAfterSeconds: number; readonly message: string };
|
||||
runWorkflow: (input: ConsultationInput) => Promise<unknown>;
|
||||
createAgent: (model: Readonly<{ id: string }>) => ReportAgentPort;
|
||||
skillSnapshot: SkillSnapshot;
|
||||
@@ -387,6 +395,20 @@ export async function resolveReportCreate(deps: ReportCreateCoreDeps): Promise<R
|
||||
|
||||
let billingReserved = false;
|
||||
if (deps.billing && selectedModel) {
|
||||
if (deps.consumeReportRateLimit) {
|
||||
const limited = deps.consumeReportRateLimit();
|
||||
if (!limited.ok) {
|
||||
return {
|
||||
status: 429,
|
||||
headers: { "Retry-After": String(limited.retryAfterSeconds) },
|
||||
body: {
|
||||
error: limited.message,
|
||||
code: "rate_limited",
|
||||
retryAfterSeconds: limited.retryAfterSeconds,
|
||||
},
|
||||
};
|
||||
}
|
||||
}
|
||||
let authorization;
|
||||
try {
|
||||
authorization = await deps.billing.reserve({ userId, requestId: payload.requestId, modelId: selectedModel.id });
|
||||
|
||||
@@ -23,6 +23,18 @@ export const requestRateLimits = {
|
||||
ephemeris: { limit: 30, windowMs: 60_000 },
|
||||
synastry: { limit: 8, windowMs: 60 * 60_000 },
|
||||
sessionWrite: { limit: 40, windowMs: 60_000 },
|
||||
// Two windows each: a minute burst and an hour budget (product 2026-10-14).
|
||||
consultMinute: { limit: 10, windowMs: 60_000 },
|
||||
consultHour: { limit: 120, windowMs: 60 * 60_000 },
|
||||
rectificationTurnMinute: { limit: 5, windowMs: 60_000 },
|
||||
rectificationTurnHour: { limit: 40, windowMs: 60 * 60_000 },
|
||||
reportGenerateHour: { limit: 3, windowMs: 60 * 60_000 },
|
||||
} as const;
|
||||
|
||||
export const userRateLimitMessages = {
|
||||
consult: "这一会儿问得有点密,先停一下。",
|
||||
rectificationTurn: "这一会儿校正问得有点密,先停一下。",
|
||||
reportGenerate: "这一小时的报告已经够多了,先停一下。",
|
||||
} as const;
|
||||
|
||||
export type RequestRateLimitBucket = keyof typeof requestRateLimits;
|
||||
@@ -54,30 +66,41 @@ function sweepExpiredRateLimitKeys(
|
||||
}
|
||||
}
|
||||
|
||||
export function consumeRequestRateLimit(input: RequestRateLimitInput): RequestRateLimitDecision {
|
||||
if (input.windowMs > seenMaxWindowMs) seenMaxWindowMs = input.windowMs;
|
||||
consumeCalls += 1;
|
||||
function evaluateRequestRateLimit(
|
||||
input: RequestRateLimitInput,
|
||||
record: boolean,
|
||||
): RequestRateLimitDecision {
|
||||
const now = input.now ?? Date.now();
|
||||
const store = input.store ?? defaultStore();
|
||||
if (consumeCalls % 1_000 === 0 || store.size > 5_000) {
|
||||
sweepExpiredRateLimitKeys(store, now, seenMaxWindowMs);
|
||||
if (record) {
|
||||
if (input.windowMs > seenMaxWindowMs) seenMaxWindowMs = input.windowMs;
|
||||
consumeCalls += 1;
|
||||
if (consumeCalls % 1_000 === 0 || store.size > 5_000) {
|
||||
sweepExpiredRateLimitKeys(store, now, seenMaxWindowMs);
|
||||
}
|
||||
}
|
||||
const windowStart = now - input.windowMs;
|
||||
const recent = (store.get(input.key) ?? []).filter((timestamp) => timestamp > windowStart);
|
||||
if (recent.length >= input.limit) {
|
||||
const retryAfterSeconds = Math.max(1, Math.ceil((recent[0]! + input.windowMs - now) / 1000));
|
||||
store.set(input.key, recent);
|
||||
if (record) store.set(input.key, recent);
|
||||
return { ok: false, retryAfterSeconds };
|
||||
}
|
||||
if (!record) return { ok: true };
|
||||
recent.push(now);
|
||||
store.set(input.key, recent);
|
||||
return { ok: true };
|
||||
}
|
||||
|
||||
export function consumeRequestRateLimit(input: RequestRateLimitInput): RequestRateLimitDecision {
|
||||
return evaluateRequestRateLimit(input, true);
|
||||
}
|
||||
|
||||
export function consumeUserRequestRateLimit(
|
||||
bucket: RequestRateLimitBucket,
|
||||
userId: string,
|
||||
now?: number,
|
||||
store?: RequestRateLimitStore,
|
||||
): RequestRateLimitDecision {
|
||||
const spec = requestRateLimits[bucket];
|
||||
return consumeRequestRateLimit({
|
||||
@@ -85,5 +108,45 @@ export function consumeUserRequestRateLimit(
|
||||
limit: spec.limit,
|
||||
windowMs: spec.windowMs,
|
||||
now,
|
||||
store,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Allow the request only when every bucket still has room. A refusal writes
|
||||
* nothing, so the bucket that still had room is not charged for the miss.
|
||||
*/
|
||||
export function consumeUserRateLimitSet(
|
||||
buckets: readonly RequestRateLimitBucket[],
|
||||
userId: string,
|
||||
now?: number,
|
||||
store?: RequestRateLimitStore,
|
||||
): RequestRateLimitDecision {
|
||||
let retryAfterSeconds = 0;
|
||||
for (const bucket of buckets) {
|
||||
const spec = requestRateLimits[bucket];
|
||||
const decision = evaluateRequestRateLimit({
|
||||
key: `${bucket}:${userId}`,
|
||||
limit: spec.limit,
|
||||
windowMs: spec.windowMs,
|
||||
now,
|
||||
store,
|
||||
}, false);
|
||||
if (!decision.ok) retryAfterSeconds = Math.max(retryAfterSeconds, decision.retryAfterSeconds);
|
||||
}
|
||||
if (retryAfterSeconds > 0) return { ok: false, retryAfterSeconds };
|
||||
for (const bucket of buckets) consumeUserRequestRateLimit(bucket, userId, now, store);
|
||||
return { ok: true };
|
||||
}
|
||||
|
||||
export function userRateLimitDeniedResponse(retryAfterSeconds: number, message: string) {
|
||||
return {
|
||||
status: 429 as const,
|
||||
headers: { "Retry-After": String(retryAfterSeconds) },
|
||||
body: {
|
||||
error: message,
|
||||
code: "rate_limited" as const,
|
||||
retryAfterSeconds,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
@@ -22,6 +22,7 @@ import {
|
||||
} from "./consultation-step-timing.ts";
|
||||
import { createVisibleTextTransformer } from "./stream-text-response.ts";
|
||||
import { consultationWriteLabel } from "./consultation-activity-labels.ts";
|
||||
import { consultationRunFailureMessage } from "./consultation-workflow-retry.ts";
|
||||
import { logTruncatedReasoning } from "./consultation-budget.ts";
|
||||
import {
|
||||
classifyPass4,
|
||||
@@ -1177,7 +1178,7 @@ export function streamAgentResponse(options: StreamAgentResponseOptions) {
|
||||
send(controller, {
|
||||
type: "run.failed",
|
||||
code,
|
||||
message: runFailedMessage(code),
|
||||
message: consultationRunFailureMessage(error, runFailedMessage(code)),
|
||||
...(failureReceipt ? { receipt: failureReceipt } : {}),
|
||||
});
|
||||
if (!disconnected) controller.close();
|
||||
|
||||
@@ -27,6 +27,7 @@ import { normalizeTechniqueAuditRows } from "../lib/consultation-technique-audit
|
||||
import type { AgentModelFinishReason, EvidenceCardTelemetry } from "../lib/agent-observability.ts";
|
||||
import { agentGenerationSettings, AGENT_SLICE_ANSWER_OUTPUT_TOKENS, AGENT_SLICE_THINKING_OUTPUT_TOKENS, type ProviderReasoningEffort } from "../lib/agent-generation-settings.ts";
|
||||
import { chartCalculationProgressLabel, evidenceLookupActivityLabel } from "../lib/consultation-activity-labels.ts";
|
||||
import { CONSULTATION_QUEUE_LABEL, retryOnConsultationRateLimit } from "../lib/consultation-workflow-retry.ts";
|
||||
import { consultDomainConcurrency } from "../lib/consult/prep-flags.ts";
|
||||
import {
|
||||
buildEvidenceCard,
|
||||
@@ -1309,6 +1310,36 @@ export function createConsultationTools(ctx: ConsultationAgentContext) {
|
||||
try {
|
||||
const userIntent = ctx.plan?.userIntent ?? input.question;
|
||||
const executions: DomainExecution[] = [];
|
||||
const runQueuedWorkflow = async (
|
||||
toolInput: ReturnType<typeof consultationInputSchema.parse>,
|
||||
domainPlan: ConsultationPlan,
|
||||
progressLabel: string,
|
||||
) => {
|
||||
let queued = false;
|
||||
const workflow = await retryOnConsultationRateLimit({
|
||||
run: () => (ctx.runWorkflow ?? runConsultationWorkflow)(toolInput, {
|
||||
foreground: true,
|
||||
signal: context.abortSignal ?? ctx.abortSignal,
|
||||
plan: domainPlan,
|
||||
requestId: ctx.requestId,
|
||||
entrypoint: ctx.entrypoint,
|
||||
}),
|
||||
onQueue: async () => {
|
||||
queued = true;
|
||||
await context.writer?.custom({
|
||||
type: "data-jyotish-activity",
|
||||
data: { phase: "compute-queue", label: CONSULTATION_QUEUE_LABEL },
|
||||
});
|
||||
},
|
||||
});
|
||||
if (queued) {
|
||||
await context.writer?.custom({
|
||||
type: "data-jyotish-activity",
|
||||
data: { phase: "chart-calculation", label: progressLabel },
|
||||
});
|
||||
}
|
||||
return workflow;
|
||||
};
|
||||
if (consultDomainConcurrency() === 1) {
|
||||
for (let index = 0; index < domains.length; index += 1) {
|
||||
const domain = domains[index];
|
||||
@@ -1344,13 +1375,11 @@ export function createConsultationTools(ctx: ConsultationAgentContext) {
|
||||
question: userIntent,
|
||||
theme: domain,
|
||||
});
|
||||
const workflow = await (ctx.runWorkflow ?? runConsultationWorkflow)(toolInput, {
|
||||
foreground: true,
|
||||
signal: context.abortSignal ?? ctx.abortSignal,
|
||||
plan: domainPlan,
|
||||
requestId: ctx.requestId,
|
||||
entrypoint: ctx.entrypoint,
|
||||
});
|
||||
const workflow = await runQueuedWorkflow(
|
||||
toolInput,
|
||||
domainPlan,
|
||||
chartCalculationProgressLabel(index + 1, domains.length),
|
||||
);
|
||||
const guarded = applyBirthTimeModeToWorkflowContext(workflow, ctx.consultationMode, {
|
||||
birthTimeSource: ctx.serverChart.truth.birthTimeSource,
|
||||
});
|
||||
@@ -1374,7 +1403,7 @@ export function createConsultationTools(ctx: ConsultationAgentContext) {
|
||||
},
|
||||
});
|
||||
}
|
||||
const parallelExecutions = await Promise.all(domains.map(async (domain) => {
|
||||
const parallelExecutions = await Promise.all(domains.map(async (domain, domainIndex) => {
|
||||
if (!domain) return null;
|
||||
const domainPlan = ctx.plan
|
||||
&& domains.length === 1
|
||||
@@ -1396,13 +1425,11 @@ export function createConsultationTools(ctx: ConsultationAgentContext) {
|
||||
question: userIntent,
|
||||
theme: domain,
|
||||
});
|
||||
const workflow = await (ctx.runWorkflow ?? runConsultationWorkflow)(toolInput, {
|
||||
foreground: true,
|
||||
signal: context.abortSignal ?? ctx.abortSignal,
|
||||
plan: domainPlan,
|
||||
requestId: ctx.requestId,
|
||||
entrypoint: ctx.entrypoint,
|
||||
});
|
||||
const workflow = await runQueuedWorkflow(
|
||||
toolInput,
|
||||
domainPlan,
|
||||
chartCalculationProgressLabel(domainIndex + 1, domains.length),
|
||||
);
|
||||
const guarded = applyBirthTimeModeToWorkflowContext(workflow, ctx.consultationMode, {
|
||||
birthTimeSource: ctx.serverChart.truth.birthTimeSource,
|
||||
});
|
||||
|
||||
@@ -102,14 +102,23 @@ export type ConsultationWorkflowFailureCode = typeof consultationWorkflowFailure
|
||||
|
||||
export class ConsultationWorkflowError extends Error {
|
||||
readonly code: ConsultationWorkflowFailureCode;
|
||||
readonly retryAfterSeconds?: number;
|
||||
|
||||
constructor(code: ConsultationWorkflowFailureCode, message: string) {
|
||||
constructor(code: ConsultationWorkflowFailureCode, message: string, retryAfterSeconds?: number) {
|
||||
super(message);
|
||||
this.name = "ConsultationWorkflowError";
|
||||
this.code = code;
|
||||
if (retryAfterSeconds !== undefined) this.retryAfterSeconds = retryAfterSeconds;
|
||||
}
|
||||
}
|
||||
|
||||
function parseRetryAfterHeader(raw: string | null): number {
|
||||
if (!raw) return 2;
|
||||
const seconds = Number(raw.trim());
|
||||
if (!Number.isFinite(seconds)) return 2;
|
||||
return Math.max(1, Math.ceil(seconds));
|
||||
}
|
||||
|
||||
function httpFailureCode(status: number): ConsultationWorkflowFailureCode {
|
||||
if (status === 400) return "workflow_bad_request";
|
||||
if (status === 403) return "workflow_forbidden";
|
||||
@@ -174,6 +183,7 @@ export async function runConsultationWorkflow(
|
||||
throw new ConsultationWorkflowError(
|
||||
response.ok ? "workflow_empty_response" : httpFailureCode(response.status),
|
||||
data?.error || data?.message || `Jyotish API returned ${response.status}`,
|
||||
response.status === 429 ? parseRetryAfterHeader(response.headers.get("retry-after")) : undefined,
|
||||
);
|
||||
}
|
||||
const parsed = consultationWorkflowResponseSchema.safeParse(data);
|
||||
|
||||
@@ -179,7 +179,10 @@ test("consult user turn no longer dumps tool JSON that the model would parrot",
|
||||
assert.doesNotMatch(chartBranch, /JSON\.stringify\(toolInput\)/);
|
||||
assert.doesNotMatch(chartBranch, /经过服务端校验的工具参数/);
|
||||
assert.match(route, /文末技法审计表/);
|
||||
assert.match(chartBranch, /await runConsultationWorkflow\(toolInput/);
|
||||
// 原值: /await runConsultationWorkflow\(toolInput/
|
||||
// 新值: /runConsultationWorkflow\(toolInput/
|
||||
// 原因: 调用包进 retryOnConsultationRateLimit,await 不再紧贴函数名。仍锁工作流入参是 toolInput;用户轮不塞工具 JSON 的断言不动。
|
||||
assert.match(chartBranch, /runConsultationWorkflow\(toolInput/);
|
||||
});
|
||||
|
||||
// 原值: test "the opener is a shape — 反差, 谁推谁修, 扮演哪个象, then at most three actions"(锁 OPENER SHAPE 四步、≤ 20 字行动、降级路线「第 3 步照给」)
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import test from "node:test";
|
||||
import { ConsultationWorkflowError } from "../src/mastra/consultation-workflow.ts";
|
||||
import { CONSULTATION_QUEUE_LABEL } from "../src/lib/consultation-activity-labels.ts";
|
||||
import {
|
||||
CONSULTATION_QUEUE_FAILED_MESSAGE,
|
||||
consultationRunFailureMessage,
|
||||
isConsultationRateLimitFailure,
|
||||
retryOnConsultationRateLimit,
|
||||
} from "../src/lib/consultation-workflow-retry.ts";
|
||||
import {
|
||||
emptyConsultationTimeline,
|
||||
reduceConsultationTimelineEvents,
|
||||
} from "../src/lib/consultation-run-timeline.ts";
|
||||
|
||||
test("a 429 is retried once and the queue line is shown before success", async () => {
|
||||
let calls = 0;
|
||||
const queued: string[] = [];
|
||||
const sleeps: number[] = [];
|
||||
const result = await retryOnConsultationRateLimit({
|
||||
run: async () => {
|
||||
calls += 1;
|
||||
if (calls === 1) throw new ConsultationWorkflowError("workflow_rate_limited", "busy", 2);
|
||||
return "ok";
|
||||
},
|
||||
onQueue: () => { queued.push(CONSULTATION_QUEUE_LABEL); },
|
||||
sleep: async (ms) => { sleeps.push(ms); },
|
||||
});
|
||||
assert.equal(result, "ok");
|
||||
assert.equal(calls, 2);
|
||||
assert.deepEqual(sleeps, [2000]);
|
||||
assert.deepEqual(queued, ["人有点多,排一下队"]);
|
||||
});
|
||||
|
||||
test("three 429s stop within two retries and 15 seconds", async () => {
|
||||
let calls = 0;
|
||||
let slept = 0;
|
||||
await assert.rejects(
|
||||
() => retryOnConsultationRateLimit({
|
||||
run: async () => {
|
||||
calls += 1;
|
||||
throw new ConsultationWorkflowError("workflow_rate_limited", "busy", 2);
|
||||
},
|
||||
sleep: async (ms) => { slept += ms; },
|
||||
}),
|
||||
(error: unknown) => {
|
||||
assert.equal(isConsultationRateLimitFailure(error), true);
|
||||
return true;
|
||||
},
|
||||
);
|
||||
assert.equal(calls, 3);
|
||||
assert.equal(slept, 4_000);
|
||||
assert.ok(slept <= 15_000);
|
||||
assert.equal(
|
||||
consultationRunFailureMessage(new ConsultationWorkflowError("workflow_rate_limited", "busy", 2), "其他失败"),
|
||||
CONSULTATION_QUEUE_FAILED_MESSAGE,
|
||||
);
|
||||
assert.equal(consultationRunFailureMessage(new Error("other"), "其他失败"), "其他失败");
|
||||
});
|
||||
|
||||
test("Retry-After cannot push the total wait past 15 seconds", async () => {
|
||||
let calls = 0;
|
||||
let slept = 0;
|
||||
await assert.rejects(() => retryOnConsultationRateLimit({
|
||||
run: async () => {
|
||||
calls += 1;
|
||||
throw new ConsultationWorkflowError("workflow_rate_limited", "busy", 20);
|
||||
},
|
||||
sleep: async (ms) => { slept += ms; },
|
||||
}));
|
||||
assert.equal(calls, 2);
|
||||
assert.equal(slept, 15_000);
|
||||
});
|
||||
|
||||
test("the queue sentence replaces the calculate row and the chart label comes back", () => {
|
||||
const calculating = reduceConsultationTimelineEvents([
|
||||
{ type: "tool.started", callId: "tool-1", tool: "run-jyotish-consultation", label: "正在计算个人星盘" },
|
||||
{ type: "activity", phase: "chart-calculation", label: "正在计算本命盘…" },
|
||||
], emptyConsultationTimeline());
|
||||
const queued = reduceConsultationTimelineEvents([
|
||||
{ type: "activity", phase: "compute-queue", label: CONSULTATION_QUEUE_LABEL },
|
||||
], calculating);
|
||||
assert.equal(queued.rows.find((row) => row.kind === "calculate")?.label, "人有点多,排一下队");
|
||||
assert.equal(queued.rows.find((row) => row.kind === "calculate")?.status, "live");
|
||||
const resumed = reduceConsultationTimelineEvents([
|
||||
{ type: "activity", phase: "chart-calculation", label: "正在计算本命盘…" },
|
||||
], queued);
|
||||
assert.equal(resumed.rows.find((row) => row.kind === "calculate")?.label, "正在计算本命盘…");
|
||||
});
|
||||
|
||||
test("a failed consultation still settles through the existing cancel path before the queue sentence", () => {
|
||||
const stream = readFileSync(new URL("../src/lib/stream-agent-response.ts", import.meta.url), "utf8");
|
||||
const consult = readFileSync(new URL("../src/app/api/consult/route.ts", import.meta.url), "utf8");
|
||||
const onErrorAt = stream.indexOf("await options.onError?.");
|
||||
const messageAt = stream.indexOf("consultationRunFailureMessage(error, runFailedMessage(code))");
|
||||
assert.ok(onErrorAt > 0 && messageAt > onErrorAt);
|
||||
assert.match(consult, /onError: \(error\) => settleRun\(\s*cancel/);
|
||||
assert.match(consult, /else await cancel\(\)/);
|
||||
assert.match(consult, /retryOnConsultationRateLimit/);
|
||||
assert.match(consult, /CONSULTATION_QUEUE_FAILED_MESSAGE/);
|
||||
});
|
||||
@@ -1525,3 +1525,19 @@ test("core read: the detail response has no coverSubject, and never the fingerpr
|
||||
assert.doesNotMatch(body, new RegExp(BIRTH_FP));
|
||||
assert.doesNotMatch(body, /coverSubject|birthFingerprint|birth_date|latitude|longitude|timezone|birthPlaceLabel|北京/);
|
||||
});
|
||||
|
||||
// 原值: 插在「429 at the daily limit」和「502 model_unavailable」之间。
|
||||
// 新值: 整段挪到文件末尾,断言不变。
|
||||
// 原因: 插在中部让 privacy 白名单按行号锁定的三行各下移 13 行。挪到末尾后那三行回到原行号,SAFE_LINES 不改。
|
||||
test("core create: hourly rate limit returns 429 and does not reserve", async () => {
|
||||
const billing = billingHarness();
|
||||
const response = await resolveReportCreate(baseDeps({
|
||||
billing: billing.billing,
|
||||
consumeReportRateLimit: () => ({ ok: false, retryAfterSeconds: 12, message: "这一小时的报告已经够多了,先停一下。" }),
|
||||
}));
|
||||
assert.equal(response.status, 429);
|
||||
assert.equal(response.body.code, "rate_limited");
|
||||
assert.equal(response.body.retryAfterSeconds, 12);
|
||||
assert.equal(response.headers?.["Retry-After"], "12");
|
||||
assert.equal(billing.reservations, 0);
|
||||
});
|
||||
|
||||
@@ -1,8 +1,13 @@
|
||||
import assert from "node:assert/strict";
|
||||
import test from "node:test";
|
||||
import { readFileSync } from "node:fs";
|
||||
import {
|
||||
consumeRequestRateLimit,
|
||||
consumeUserRateLimitSet,
|
||||
requestRateLimits,
|
||||
resetRequestRateLimitMaintenanceForTests,
|
||||
userRateLimitDeniedResponse,
|
||||
userRateLimitMessages,
|
||||
} from "../src/lib/request-rate-limit.ts";
|
||||
|
||||
test("a user is allowed up to the window limit and then must wait", () => {
|
||||
@@ -110,3 +115,98 @@ test("keys past the longest seen window are swept, and a one-hour key survives 6
|
||||
assert.equal(mixed.has("ephemeris:person"), true);
|
||||
resetRequestRateLimitMaintenanceForTests();
|
||||
});
|
||||
|
||||
test("consult allows 10 per minute and refuses the 11th without charging the hour bucket further", () => {
|
||||
const store = new Map<string, number[]>();
|
||||
for (let index = 0; index < 10; index += 1) {
|
||||
const decision = consumeUserRateLimitSet(
|
||||
["consultMinute", "consultHour"],
|
||||
"person",
|
||||
1_000 + index,
|
||||
store,
|
||||
);
|
||||
assert.deepEqual(decision, { ok: true });
|
||||
}
|
||||
const denied = consumeUserRateLimitSet(["consultMinute", "consultHour"], "person", 2_000, store);
|
||||
assert.equal(denied.ok, false);
|
||||
if (!denied.ok) assert.equal(denied.retryAfterSeconds, 59);
|
||||
assert.equal(store.get("consultMinute:person")?.length, 10);
|
||||
assert.equal(store.get("consultHour:person")?.length, 10);
|
||||
assert.equal(requestRateLimits.consultMinute.limit, 10);
|
||||
assert.equal(requestRateLimits.consultHour.limit, 120);
|
||||
});
|
||||
|
||||
test("the hour bucket can refuse while the minute bucket still has room, and neither bucket records the miss", () => {
|
||||
const store = new Map<string, number[]>();
|
||||
for (let index = 0; index < 120; index += 1) {
|
||||
consumeRequestRateLimit({
|
||||
key: "consultHour:person",
|
||||
limit: 120,
|
||||
windowMs: 60 * 60_000,
|
||||
now: index * 1_000,
|
||||
store,
|
||||
});
|
||||
}
|
||||
const denied = consumeUserRateLimitSet(["consultMinute", "consultHour"], "person", 130_000, store);
|
||||
assert.equal(denied.ok, false);
|
||||
assert.equal(store.has("consultMinute:person"), false);
|
||||
assert.equal(store.get("consultHour:person")?.length, 120);
|
||||
});
|
||||
|
||||
test("rectification turns allow 5 per minute and 40 per hour, reports allow 3 per hour", () => {
|
||||
const store = new Map<string, number[]>();
|
||||
for (let index = 0; index < 5; index += 1) {
|
||||
assert.deepEqual(
|
||||
consumeUserRateLimitSet(["rectificationTurnMinute", "rectificationTurnHour"], "person", 5_000 + index, store),
|
||||
{ ok: true },
|
||||
);
|
||||
}
|
||||
const denied = consumeUserRateLimitSet(
|
||||
["rectificationTurnMinute", "rectificationTurnHour"],
|
||||
"person",
|
||||
5_100,
|
||||
store,
|
||||
);
|
||||
assert.equal(denied.ok, false);
|
||||
assert.equal(store.get("rectificationTurnMinute:person")?.length, 5);
|
||||
assert.equal(requestRateLimits.rectificationTurnMinute.limit, 5);
|
||||
assert.equal(requestRateLimits.rectificationTurnHour.limit, 40);
|
||||
assert.equal(requestRateLimits.reportGenerateHour.limit, 3);
|
||||
assert.equal(requestRateLimits.reportGenerateHour.windowMs, 60 * 60_000);
|
||||
|
||||
const reports = new Map<string, number[]>();
|
||||
for (let index = 0; index < 3; index += 1) {
|
||||
assert.deepEqual(
|
||||
consumeUserRateLimitSet(["reportGenerateHour"], "person", 8_000 + index, reports),
|
||||
{ ok: true },
|
||||
);
|
||||
}
|
||||
const reportDenied = consumeUserRateLimitSet(["reportGenerateHour"], "person", 9_000, reports);
|
||||
assert.equal(reportDenied.ok, false);
|
||||
assert.equal(reports.get("reportGenerateHour:person")?.length, 3);
|
||||
});
|
||||
|
||||
test("a denied response carries 429, rate_limited, and Retry-After", () => {
|
||||
const denied = userRateLimitDeniedResponse(12, userRateLimitMessages.consult);
|
||||
assert.equal(denied.status, 429);
|
||||
assert.equal(denied.headers["Retry-After"], "12");
|
||||
assert.deepEqual(denied.body, {
|
||||
error: "这一会儿问得有点密,先停一下。",
|
||||
code: "rate_limited",
|
||||
retryAfterSeconds: 12,
|
||||
});
|
||||
});
|
||||
|
||||
test("consult, rectification turns, and report creation check the limit before they reserve", () => {
|
||||
const consult = readFileSync(new URL("../src/app/api/consult/route.ts", import.meta.url), "utf8");
|
||||
const rectification = readFileSync(new URL("../src/app/api/rectification/agent/route.ts", import.meta.url), "utf8");
|
||||
const caseOpen = readFileSync(new URL("../src/app/api/rectification/cases/open/route.ts", import.meta.url), "utf8");
|
||||
const reports = readFileSync(new URL("../src/app/api/reports/route.ts", import.meta.url), "utf8");
|
||||
const reportCore = readFileSync(new URL("../src/lib/personal-report-route-core.ts", import.meta.url), "utf8");
|
||||
assert.ok(consult.indexOf("consumeUserRateLimitSet(") < consult.indexOf("reserve_consultation_usage"));
|
||||
assert.match(consult, /userRateLimitDeniedResponse\(/);
|
||||
assert.ok(rectification.indexOf("consumeUserRateLimitSet(") < rectification.indexOf("respondWithRectificationAgentTurn("));
|
||||
assert.equal(caseOpen.includes("consumeUserRateLimitSet"), false);
|
||||
assert.match(reports, /consumeUserRateLimitSet\(\["reportGenerateHour"\]/);
|
||||
assert.ok(reportCore.indexOf("if (deps.consumeReportRateLimit)") < reportCore.indexOf("deps.billing.reserve"));
|
||||
});
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
"""Fail-fast bounded concurrency for heavy Jyotish API compute endpoints.
|
||||
"""Bounded concurrency for heavy Jyotish API compute endpoints.
|
||||
|
||||
Request threads that run rectification scans or high-rigor workflows share one
|
||||
process-wide semaphore sized for the 2 vCPU production host. Saturated requests
|
||||
return immediately; they are not queued. Health checks and other light routes
|
||||
must not call this gate.
|
||||
Request threads that run rectification scans or consultation workflows share one
|
||||
process-wide semaphore sized for the 2 vCPU production host. A saturated request
|
||||
waits up to 8 seconds when fewer than 4 threads are already waiting. If that
|
||||
queue is full or the wait expires, the caller still gets HTTP 429 and
|
||||
Retry-After. A thread is never parked without a deadline. Health checks and
|
||||
other light routes must not call this gate.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -15,6 +17,8 @@ import time
|
||||
|
||||
DEFAULT_CONCURRENCY = 2
|
||||
DEFAULT_RETRY_AFTER_SECONDS = 2
|
||||
DEFAULT_QUEUE_WAIT_SECONDS = 8.0
|
||||
DEFAULT_MAX_WAITERS = 4
|
||||
ENV_CONCURRENCY = "JYOTISH_HEAVY_COMPUTE_CONCURRENCY"
|
||||
ENV_RETRY_AFTER = "JYOTISH_HEAVY_COMPUTE_RETRY_AFTER_SECONDS"
|
||||
ENV_GATE_CHART_OVERVIEW = "JYOTISH_GATE_CHART_OVERVIEW"
|
||||
@@ -51,6 +55,9 @@ _state_lock = threading.Lock()
|
||||
_semaphore: threading.BoundedSemaphore | None = None
|
||||
_retry_after_seconds = DEFAULT_RETRY_AFTER_SECONDS
|
||||
_limit = DEFAULT_CONCURRENCY
|
||||
_wait_seconds = DEFAULT_QUEUE_WAIT_SECONDS
|
||||
_max_waiters = DEFAULT_MAX_WAITERS
|
||||
_waiters = 0
|
||||
_holders: list[dict] = []
|
||||
_holder_seq = 0
|
||||
|
||||
@@ -73,7 +80,7 @@ class _HeldSlot:
|
||||
|
||||
|
||||
class HeavyComputeBusy(RuntimeError):
|
||||
"""No heavy-compute slot is free; callers must fail fast with HTTP 429."""
|
||||
"""The short queue could not take this request; callers return HTTP 429."""
|
||||
|
||||
error_code = "ERR_COMPUTE_BUSY"
|
||||
|
||||
@@ -102,9 +109,11 @@ def reset_heavy_compute_gate(
|
||||
*,
|
||||
limit: int | None = None,
|
||||
retry_after_seconds: int | None = None,
|
||||
wait_seconds: float | None = None,
|
||||
max_waiters: int | None = None,
|
||||
) -> None:
|
||||
"""Rebuild the process-wide semaphore. Tests must call this after env changes."""
|
||||
global _semaphore, _retry_after_seconds, _limit, _holders
|
||||
global _semaphore, _retry_after_seconds, _limit, _holders, _wait_seconds, _max_waiters, _waiters
|
||||
resolved_limit = (
|
||||
limit
|
||||
if limit is not None
|
||||
@@ -115,9 +124,14 @@ def reset_heavy_compute_gate(
|
||||
if retry_after_seconds is not None
|
||||
else _parse_positive_int(os.environ.get(ENV_RETRY_AFTER), DEFAULT_RETRY_AFTER_SECONDS)
|
||||
)
|
||||
resolved_wait = DEFAULT_QUEUE_WAIT_SECONDS if wait_seconds is None else max(0.0, float(wait_seconds))
|
||||
resolved_waiters = DEFAULT_MAX_WAITERS if max_waiters is None else max(0, int(max_waiters))
|
||||
with _state_lock:
|
||||
_retry_after_seconds = resolved_retry
|
||||
_limit = resolved_limit
|
||||
_wait_seconds = resolved_wait
|
||||
_max_waiters = resolved_waiters
|
||||
_waiters = 0
|
||||
_holders = []
|
||||
_semaphore = threading.BoundedSemaphore(resolved_limit)
|
||||
|
||||
@@ -143,35 +157,61 @@ def _holder_view(now: float) -> list[dict]:
|
||||
]
|
||||
|
||||
|
||||
def _acquire_recorded(path: str) -> _HeldSlot:
|
||||
"""Take one permit and remember which path holds it. Fail fast when full."""
|
||||
def _remember_locked(path: str, semaphore: threading.BoundedSemaphore) -> _HeldSlot:
|
||||
"""Record a permit that this thread already holds. Caller owns ``_state_lock``."""
|
||||
global _holder_seq
|
||||
_holder_seq += 1
|
||||
holder_id = _holder_seq
|
||||
_holders.append({"id": holder_id, "path": path, "started": time.monotonic()})
|
||||
return _HeldSlot(semaphore, holder_id)
|
||||
|
||||
|
||||
def _reject_busy(path: str, retry_after: int) -> None:
|
||||
with _state_lock:
|
||||
snapshot = _holder_view(time.monotonic())
|
||||
_LOG.warning(
|
||||
"heavy_compute_busy rejected_path=%s holders=%s",
|
||||
path,
|
||||
snapshot,
|
||||
)
|
||||
raise HeavyComputeBusy(retry_after)
|
||||
|
||||
|
||||
def _acquire_recorded(path: str) -> _HeldSlot:
|
||||
"""Take one permit. Wait briefly when full; still raise once the queue cannot."""
|
||||
global _waiters
|
||||
with _state_lock:
|
||||
semaphore, retry_after = _ensure_locked()
|
||||
if not semaphore.acquire(blocking=False):
|
||||
snapshot = _holder_view(time.monotonic())
|
||||
failed = True
|
||||
slot = None
|
||||
if semaphore.acquire(blocking=False):
|
||||
return _remember_locked(path, semaphore)
|
||||
if _waiters >= _max_waiters:
|
||||
queued = False
|
||||
else:
|
||||
_holder_seq += 1
|
||||
holder_id = _holder_seq
|
||||
_holders.append({"id": holder_id, "path": path, "started": time.monotonic()})
|
||||
failed = False
|
||||
snapshot = []
|
||||
slot = _HeldSlot(semaphore, holder_id)
|
||||
if failed:
|
||||
_LOG.warning(
|
||||
"heavy_compute_busy rejected_path=%s holders=%s",
|
||||
path,
|
||||
snapshot,
|
||||
)
|
||||
raise HeavyComputeBusy(retry_after)
|
||||
assert slot is not None
|
||||
return slot
|
||||
_waiters += 1
|
||||
queued = True
|
||||
wait_seconds = _wait_seconds
|
||||
if not queued:
|
||||
_reject_busy(path, retry_after)
|
||||
acquired = False
|
||||
try:
|
||||
acquired = bool(semaphore.acquire(blocking=True, timeout=wait_seconds))
|
||||
finally:
|
||||
with _state_lock:
|
||||
_waiters -= 1
|
||||
if not acquired:
|
||||
_reject_busy(path, retry_after)
|
||||
with _state_lock:
|
||||
return _remember_locked(path, semaphore)
|
||||
|
||||
|
||||
def heavy_compute_waiter_count() -> int:
|
||||
"""Threads parked inside the short queue. Tests use this to avoid races."""
|
||||
with _state_lock:
|
||||
return _waiters
|
||||
|
||||
|
||||
def acquire_heavy_compute_slot(path: str) -> _HeldSlot | None:
|
||||
"""Acquire a slot for a gated path. Light paths return None. Fail-fast on saturation."""
|
||||
"""Acquire a slot for a gated path. Light paths return None."""
|
||||
if not is_heavy_compute_path(path):
|
||||
return None
|
||||
return _acquire_recorded(path)
|
||||
|
||||
@@ -153,7 +153,10 @@ def test_chart_overview_gate_is_off_by_default(monkeypatch) -> None:
|
||||
|
||||
def test_chart_overview_gate_only_sync_unskipped_requests(monkeypatch) -> None:
|
||||
monkeypatch.setenv(gate.ENV_GATE_CHART_OVERVIEW, "1")
|
||||
gate.reset_heavy_compute_gate(limit=1)
|
||||
# 原值:reset(limit=1) 后第二次立刻 HeavyComputeBusy
|
||||
# 新值:wait_seconds=0
|
||||
# 原因:默认改为短排队;本测试仍锁「名额被占时同步排盘被拒绝」,等待行为不在这里
|
||||
gate.reset_heavy_compute_gate(limit=1, wait_seconds=0)
|
||||
body = {"year": 1990, "lat": 28.6, "lon": 77.2}
|
||||
slot = gate.acquire_heavy_compute_slot_for_request("/api/chart", body)
|
||||
assert slot is not None
|
||||
@@ -172,7 +175,10 @@ def test_chart_overview_gate_only_sync_unskipped_requests(monkeypatch) -> None:
|
||||
|
||||
|
||||
def test_busy_log_names_holders_and_not_the_body(caplog) -> None:
|
||||
gate.reset_heavy_compute_gate(limit=1)
|
||||
# 原值:reset(limit=1) 后拒绝日志立刻出现
|
||||
# 新值:wait_seconds=0
|
||||
# 原因:默认会先等 8 秒;本测试仍锁日志只有路径、没有请求正文
|
||||
gate.reset_heavy_compute_gate(limit=1, wait_seconds=0)
|
||||
held = gate.acquire_heavy_compute_slot("/api/professional_report_reference")
|
||||
assert held is not None
|
||||
secret_body = {"birth_date": "1990-01-01", "name": "must-not-appear"}
|
||||
|
||||
@@ -4,6 +4,7 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
import threading
|
||||
import time
|
||||
from io import BytesIO
|
||||
|
||||
import pytest
|
||||
@@ -14,6 +15,7 @@ from scripts.api_heavy_compute_gate import (
|
||||
HEAVY_COMPUTE_PATHS,
|
||||
HeavyComputeBusy,
|
||||
acquire_heavy_compute_slot,
|
||||
heavy_compute_waiter_count,
|
||||
is_heavy_compute_path,
|
||||
release_heavy_compute_slot,
|
||||
reset_heavy_compute_gate,
|
||||
@@ -93,7 +95,10 @@ def limit_one_gate(monkeypatch):
|
||||
monkeypatch.setenv("JYOTISH_HEAVY_COMPUTE_CONCURRENCY", "1")
|
||||
monkeypatch.setenv("JYOTISH_HEAVY_COMPUTE_RETRY_AFTER_SECONDS", "2")
|
||||
monkeypatch.setenv("JYOTISH_API_RATE_LIMIT_PER_MINUTE", "0")
|
||||
reset_heavy_compute_gate()
|
||||
# 原值:reset() 使用默认「满了立刻拒绝」
|
||||
# 新值:wait_seconds=0,本夹具里的 429 断言仍是立刻拒绝
|
||||
# 原因:产品改为最多等 8 秒;排队成功、等待者满、超时另有测试
|
||||
reset_heavy_compute_gate(wait_seconds=0)
|
||||
yield
|
||||
monkeypatch.delenv("JYOTISH_HEAVY_COMPUTE_CONCURRENCY", raising=False)
|
||||
monkeypatch.delenv("JYOTISH_HEAVY_COMPUTE_RETRY_AFTER_SECONDS", raising=False)
|
||||
@@ -119,7 +124,10 @@ def test_health_and_light_paths_are_not_gated() -> None:
|
||||
|
||||
|
||||
def test_acquire_fail_fast_then_succeeds_after_release() -> None:
|
||||
reset_heavy_compute_gate(limit=1, retry_after_seconds=3)
|
||||
# 原值:reset(limit=1, retry_after_seconds=3) 后第二路立刻 HeavyComputeBusy
|
||||
# 新值:多传 wait_seconds=0
|
||||
# 原因:默认改为短排队;本测试仍锁 Retry-After=3 与 ERR_COMPUTE_BUSY,不锁「禁止等待」
|
||||
reset_heavy_compute_gate(limit=1, retry_after_seconds=3, wait_seconds=0)
|
||||
first = acquire_heavy_compute_slot("/api/high_rigor_workflow")
|
||||
assert first is not None
|
||||
with pytest.raises(HeavyComputeBusy) as caught:
|
||||
@@ -248,8 +256,87 @@ def test_warm_consultation_does_not_consume_the_user_slot(limit_one_gate, monkey
|
||||
assert "packet" not in warm_box["payload"]
|
||||
|
||||
|
||||
def _wait_until(predicate, timeout: float = 2.0) -> bool:
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
if predicate():
|
||||
return True
|
||||
time.sleep(0.01)
|
||||
return False
|
||||
|
||||
|
||||
def test_third_request_gets_the_slot_after_one_holder_releases() -> None:
|
||||
reset_heavy_compute_gate(limit=2, retry_after_seconds=2, wait_seconds=2, max_waiters=4)
|
||||
first = acquire_heavy_compute_slot("/api/consultation_workflow")
|
||||
second = acquire_heavy_compute_slot("/api/rectification/v5/score")
|
||||
assert first is not None and second is not None
|
||||
found: dict = {}
|
||||
|
||||
def _third() -> None:
|
||||
found["slot"] = acquire_heavy_compute_slot("/api/consultation_workflow")
|
||||
|
||||
worker = threading.Thread(target=_third)
|
||||
worker.start()
|
||||
assert _wait_until(lambda: heavy_compute_waiter_count() == 1)
|
||||
release_heavy_compute_slot(first)
|
||||
worker.join(timeout=3)
|
||||
assert not worker.is_alive()
|
||||
assert found["slot"] is not None
|
||||
release_heavy_compute_slot(second)
|
||||
release_heavy_compute_slot(found["slot"])
|
||||
reset_heavy_compute_gate()
|
||||
|
||||
|
||||
def test_waiter_overflow_rejects_immediately() -> None:
|
||||
reset_heavy_compute_gate(limit=1, retry_after_seconds=2, wait_seconds=2, max_waiters=1)
|
||||
held = acquire_heavy_compute_slot("/api/high_rigor_workflow")
|
||||
assert held is not None
|
||||
parked: dict = {}
|
||||
|
||||
def _wait() -> None:
|
||||
try:
|
||||
parked["slot"] = acquire_heavy_compute_slot("/api/consultation_workflow")
|
||||
except HeavyComputeBusy as exc:
|
||||
parked["error"] = exc
|
||||
|
||||
waiter = threading.Thread(target=_wait)
|
||||
waiter.start()
|
||||
assert _wait_until(lambda: heavy_compute_waiter_count() == 1)
|
||||
started = time.monotonic()
|
||||
with pytest.raises(HeavyComputeBusy) as caught:
|
||||
acquire_heavy_compute_slot("/api/rectification/v5/score")
|
||||
assert time.monotonic() - started < 0.3
|
||||
assert caught.value.retry_after_seconds == 2
|
||||
assert caught.value.error_code == "ERR_COMPUTE_BUSY"
|
||||
release_heavy_compute_slot(held)
|
||||
waiter.join(timeout=3)
|
||||
assert not waiter.is_alive()
|
||||
assert parked.get("slot") is not None
|
||||
release_heavy_compute_slot(parked["slot"])
|
||||
reset_heavy_compute_gate()
|
||||
|
||||
|
||||
def test_queue_timeout_still_returns_busy() -> None:
|
||||
reset_heavy_compute_gate(limit=1, retry_after_seconds=2, wait_seconds=0.2, max_waiters=4)
|
||||
held = acquire_heavy_compute_slot("/api/professional_report_reference")
|
||||
assert held is not None
|
||||
started = time.monotonic()
|
||||
with pytest.raises(HeavyComputeBusy) as caught:
|
||||
acquire_heavy_compute_slot("/api/consultation_workflow")
|
||||
elapsed = time.monotonic() - started
|
||||
assert 0.1 <= elapsed < 2.0
|
||||
assert caught.value.retry_after_seconds == 2
|
||||
assert caught.value.error_code == "ERR_COMPUTE_BUSY"
|
||||
light = acquire_heavy_compute_slot("/api/health")
|
||||
assert light is None
|
||||
release_heavy_compute_slot(held)
|
||||
reset_heavy_compute_gate()
|
||||
|
||||
|
||||
def test_quality_gate_runs_heavy_compute_gate() -> None:
|
||||
from scripts.run_quality_gate import CORE_PYTEST_TARGETS
|
||||
|
||||
assert "tests/test_api_heavy_compute_gate.py" in CORE_PYTEST_TARGETS
|
||||
assert gate.DEFAULT_CONCURRENCY == 2
|
||||
assert gate.DEFAULT_QUEUE_WAIT_SECONDS == 8
|
||||
assert gate.DEFAULT_MAX_WAITERS == 4
|
||||
|
||||
Reference in New Issue
Block a user