From 4c3485bc2352e8f9de097d81d9979092ee2bb280 Mon Sep 17 00:00:00 2001 From: jesse-ux Date: Wed, 14 Oct 2026 03:32:39 +0800 Subject: [PATCH 1/9] =?UTF-8?q?=E6=9C=8D=E5=8A=A1=E5=99=A8=E9=95=BF?= =?UTF-8?q?=E8=B7=91=E4=B8=8D=E5=86=8D=E5=A0=86=E7=A7=AF=E5=8F=96=E6=B6=88?= =?UTF-8?q?=E7=9B=91=E5=90=AC=EF=BC=8C=E6=95=B0=E6=8D=AE=E5=BA=93=E5=B0=91?= =?UTF-8?q?=E8=B5=B0=E5=87=A0=E8=B6=9F=E3=80=82?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 报告工人的等待到点就摘掉取消监听。本地数据库缓存表结构，并把设置身份合成一条语句，对照通过后默认打开，写成 off 仍走原来的语句。交付、重算和限流的内存表会过期。登录读缓存默认关闭，要开再设 IDENTITY_SESSION_COOKIE_CACHE=on。 自测：tsc 通过，lint 0 error，点名测试 134 通过，test:db 开关关和开都只有已知的 13 条失败，往返 49 降到 32。 Co-Authored-By: Grok (agent loop) --- CHANGELOG.md | 4 + deploy/.env.staging.identity.example | 8 + deploy/README.md | 2 + docs/BUG_HISTORY.md | 60 ++++ ...ROGRESS-node-runtime-resources-20261013.md | 59 ++++ docs/tasks/README.md | 2 +- .../node-runtime-resources-20261013.md | 13 + .../node-runtime-resources-ab-20261013.md | 21 ++ frontend/scripts/bench-local-pg-roundtrips.ts | 224 ++++++++++++++ .../src/app/api/account/deletion/route.ts | 2 +- .../src/app/api/account/password/route.ts | 5 +- frontend/src/lib/admin/auth.ts | 1 + frontend/src/lib/birth-time-guided-polling.ts | 10 +- .../src/lib/db/local-postgres-client-core.ts | 212 ++++++++++++- .../src/lib/personal-report-worker-core.ts | 14 +- .../v9/block-scan-answer.ts | 40 ++- .../v9/delivery-turn-guard.ts | 21 ++ frontend/src/lib/request-rate-limit.ts | 27 ++ frontend/src/lib/supabase/server.ts | 5 +- frontend/src/modules/identity/auth-factory.ts | 22 +- frontend/src/modules/identity/session.ts | 21 +- ...abase-local-postgres-session-setup.test.ts | 64 ++++ .../tests/delivery-turn-guard-expiry.test.ts | 61 ++++ .../identity-session-cookie-cache.test.ts | 258 ++++++++++++++++ .../tests/local-postgres-roundtrips.test.ts | 286 ++++++++++++++++++ ...sonal-report-worker-sleep-listener.test.ts | 109 +++++++ frontend/tests/request-rate-limit.test.ts | 66 +++- frontend/tests/rescore-attempts-bound.test.ts | 31 ++ 28 files changed, 1614 insertions(+), 34 deletions(-) create mode 100644 docs/tasks/PROGRESS-node-runtime-resources-20261013.md create mode 100644 docs/testing/node-runtime-resources-20261013.md create mode 100644 docs/testing/node-runtime-resources-ab-20261013.md create mode 100644 frontend/scripts/bench-local-pg-roundtrips.ts create mode 100644 frontend/tests/database-local-postgres-session-setup.test.ts create mode 100644 frontend/tests/delivery-turn-guard-expiry.test.ts create mode 100644 frontend/tests/identity-session-cookie-cache.test.ts create mode 100644 frontend/tests/local-postgres-roundtrips.test.ts create mode 100644 frontend/tests/personal-report-worker-sleep-listener.test.ts create mode 100644 frontend/tests/rescore-attempts-bound.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index d2056bfc..a5d60b6c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,9 @@ # 印度占星 Skill 更新日志 +## 2026-10-14 — 服务器长期运行不再逐日涨内存；数据库访问少走几趟（待合入 staging） + +- 服务器长期运行不再逐日涨内存；数据库访问少走几趟。Skill 版本不变（BUG-1326～1328、BUG-1346）。 + ## 2026-10-14 — 婚恋评级改成证据档位，KP 判宫和大运年长对齐（待合入 staging） - 婚恋结构指数不再写成婚姻稳定、离婚、再婚或建议婚前咨询。中文报告用中文档位，英文报告用英文档位。配偶星分数按 0 到 1 显示，不再写成百分制。 diff --git a/deploy/.env.staging.identity.example b/deploy/.env.staging.identity.example index 037b55c6..5b452e19 100644 --- a/deploy/.env.staging.identity.example +++ b/deploy/.env.staging.identity.example @@ -31,3 +31,11 @@ JYOTISH_DYNAMIC_RECTIFICATION_TOKEN= # these server-side values through env_file; they are not NEXT_PUBLIC values. PERSONAL_REPORT_ENABLED=true PERSONAL_REPORT_DAILY_LIMIT=5 + +# Local Postgres adapter. Unset or on enables both. Exact off restores the +# previous per-statement sequence. Measured default is on (2026-10-14). +JYOTISHA_PG_METADATA_CACHE=on +JYOTISHA_PG_MERGED_SESSION_SETUP=on +# Ordinary session reads stay uncached unless this is exactly on. Password +# change, account deletion, and admin routes bypass the cache. maxAge is 300s. +#IDENTITY_SESSION_COOKIE_CACHE=off diff --git a/deploy/README.md b/deploy/README.md index b2966ce0..98ad21b9 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -194,6 +194,8 @@ SITE_ADDRESS=https://staging.jyotisha.chat Staging is fully self-hosted: set `AUTH_PROVIDER=self-hosted` and `SELF_HOSTED_IDENTITY_ENABLED=true`. Add the four role-specific server-only database URLs, the exact `AUTH_USER_ORIGIN=https://staging.jyotisha.chat` and `ADMIN_USER_ORIGIN=https://admin.staging.jyotisha.chat`, the single `BETTER_AUTH_USER_SECRET`, and staging-only Resend settings listed in `deploy/.env.staging.identity.example`. Both hosts run the same application and Better Auth service, but cookies remain host-only; the admin host `/` redirects to `/admin`, and unauthenticated admin requests continue to `/login` on that host. Better Auth trusts only those two origins, while unknown identity hosts fail closed. Persisted `identity.users.role=admin` is the only self-hosted backend role, while `viewer` and ordinary users are denied. Browser code uses same-origin APIs; it receives neither database credentials nor Supabase keys. Production uses the same architecture only after the reviewed migration and cutover. See `docs/operations/self-hosted-identity.md` for identity validation. +The web service already loads its env file, so these server-only switches do not need a Compose change. `JYOTISHA_PG_METADATA_CACHE` and `JYOTISHA_PG_MERGED_SESSION_SETUP` default to on when unset; set either to `off` to restore the previous database roundtrip sequence. `IDENTITY_SESSION_COOKIE_CACHE` defaults to off; set it to `on` to cache ordinary user-session reads for 300 seconds. Names and defaults are also in `deploy/.env.staging.identity.example`. Do not put secrets in these three values. + After source sync and before `up`, the workflow validates `.env.staging` mode/selectors, explicitly pins the three staging selectors against ambient shell overrides, and runs `docker compose --env-file .env.staging -f deploy/docker-compose.server.yml config --quiet`. For later manual inspections, run the same checks only after the tracked deployment files exist on the server. Do not use a manual gate run from `main` as the first publishing path: publishing requires a successful push to `staging`, while manual `Deploy staging` requires a successful push gate for the exact SHA and must use the staging workflow ref. ### First-deploy sequence diff --git a/docs/BUG_HISTORY.md b/docs/BUG_HISTORY.md index add22027..ad0eaff1 100644 --- a/docs/BUG_HISTORY.md +++ b/docs/BUG_HISTORY.md @@ -17795,3 +17795,63 @@ - 复发自：无。 - 修复版本：未修复。 +## BUG-1326 | 个人报告工人空转时，取消监听一直留在进程上 + +- 状态：resolved（代码+回归测试），生产内存曲线待部署后观察 +- 首次发现 / 最近更新：2026-10-13 / 2026-10-14 +- 来源：issue #58。任务书 T1。 +- 现象：报告工人长时间没有活干时，进程内存逐日上涨。取消这次工人时，之前每一次等待都还挂着一个取消监听。 +- 触发条件：工人循环用进程级取消信号，两次领任务之间按间隔等待。等待到点之后监听不摘。 +- 根因：等待用了「只触发一次」的取消监听。这个选项只在真的取消时摘掉监听。计时器自己到点时，监听还留在信号上。信号活得和进程一样长，监听就一直积着。 +- 修复：计时器到点时先摘掉监听再结束等待；真的取消时清掉计时器并结束等待。已经取消的信号仍立刻返回。计时器仍不挡住进程退出。引导打分里同一写法的等待也改成到点摘监听。 +- 验证：`frontend/tests/personal-report-worker-sleep-listener.test.ts` 4 通过。空转和抛错各 200 次后，取消前监听数是 0，取消后循环马上结束。修之前的同形等待在第 200 次时留下 199 个监听（断言 `199 !== 0`）。 +- 防复发：上述测试锁住「200 次之后监听数为 0」。 +- 相关记录：BUG-1328。 +- 复发自：无。 +- 修复版本：分支 `codex/node-runtime-resources-20261013`，尚未合入 staging。 + +## BUG-1327 | 本地数据库每一问都重复查表结构和设置身份 + +- 状态：resolved（代码+回归测试）。2026-10-14 对照通过后，两个开关默认打开。 +- 首次发现 / 最近更新：2026-10-13 / 2026-10-14 +- 来源：issue #58。任务书 T2。 +- 现象：同一次页面操作里，本地数据库要来回很多趟。表有哪些列、主键是什么、函数怎么调用，每次都现查。设置身份也要单独走一趟。 +- 触发条件：网页通过本地 Postgres 适配层读写真业务表。 +- 根因：元数据没有进程内缓存。设置角色和两条身份参数各是一条语句。事务的开始和提交仍各走一趟，这是故意留下的。 +- 修复：`JYOTISHA_PG_METADATA_CACHE` 缓存列类型、主键和函数签名，300 秒过期；缓存键带连接串，列和函数再带角色。不缓存空结果，也不缓存「函数不存在」。业务语句报列、表、函数或类型不匹配时，清掉该连接串的全部元数据并照旧返回错误，不自动重试。关掉某个连接或全部连接时，对应元数据一起清掉。`JYOTISHA_PG_MERGED_SESSION_SETUP` 把设置角色和两条身份合成一条参数化语句。角色仍只允许既有的两个业务角色。开始和提交不并进这条语句。两个开关都写成 `off` 时，语句顺序与改前一致。未设置或 `on` 为打开。只读且没有身份时省掉事务的做法已否决：那会把角色留在连接上，还回池子后可能给下一个请求。 +- 验证：不连数据库的往返单测锁住关闭时 6 条、打开且命中缓存时 4 条。真库测试确认事务内身份生效，连接归还后再借出时身份已空。对照脚本第二次（带三组原始数）整段往返 49 降到 32，耗时中位数 75.84 毫秒降到 36.48 毫秒，返回内容逐字节一致。第一次同样是 49 降到 32。`npm run test:db` 两轮都是 81 项、通过 68、失败 13、取消 0。失败名称与 Windows 已知清单逐条相同，没有多出来的。 +- 防复发：往返单测锁住关闭时的语句顺序，以及角色、参数名、报错、过期和关闭连接都会让缓存失效。 +- 相关记录：BUG-166、BUG-578、BUG-596、BUG-926。 +- 复发自：无。 +- 修复版本：分支 `codex/node-runtime-resources-20261013`，尚未合入 staging。 + +## BUG-1328 | 三张只进不出的内存表会随运行变大 + +- 状态：resolved +- 首次发现 / 最近更新：2026-10-13 / 2026-10-14 +- 来源：issue #58。任务书 T3。 +- 现象：校正交付去重、重算尝试和请求限流各有一张进程内的表。旧条目读的时候已经不算数，却不删除，进程活得越久表越大。 +- 触发条件：长时间有人做校正或调用限流接口。 +- 根因：写入只增加，没有过期清理，也没有条数上限。 +- 修复：交付记录超过 60 秒就删，最多留 1 万条。重算记录改存时间，超过 6 小时就删，最多留 5 千条。限流按本进程见过的最大窗口定期删掉已经出窗的键；1 小时窗口的键不会在 61 秒时被删。限流的对外类型不变，调用方仍可传入自己的表。 +- 验证：交付、重算、限流的新测试通过。窗口内的允许或拒绝与原来的断言一致，没有改旧断言。同一道校正题在 6 小时内仍不会重算第二次；超过 6 小时或被挤出后可能再算一次，这比「进程一重启就可能再算」更收敛。 +- 防复发：硬上限和过期都有测试。1 小时窗口的键在短窗口清理后必须还在。 +- 相关记录：BUG-1326。 +- 复发自：无。 +- 修复版本：分支 `codex/node-runtime-resources-20261013`，尚未合入 staging。 + +## BUG-1346 | 每次带登录的请求都重新读一次会话 + +- 状态：resolved（代码+回归测试）。开关默认关闭：查库次数已下降，真实身份库的耗时对照还没做，未达到默认打开的门槛。 +- 首次发现 / 最近更新：2026-10-14 / 2026-10-14 +- 来源：issue #58。产品负责人 2026-10-14 决定可以开。任务书 T6。 +- 现象：每个已登录请求都到身份库读一次会话。 +- 触发条件：用户端 Better Auth 配置没有会话读缓存。后台没有另一份登录配置。 +- 根因：用户端会话配置未启用 `cookieCache`。 +- 修复：只有环境变量 `IDENTITY_SESSION_COOKIE_CACHE` 精确等于 `on` 时，用户端会话才缓存 300 秒。未设置或其它值与现在一样，每次读库。改密码、注销账户、以及全部后台入口每次都跳过缓存。改邮箱、两步验证和后台插件自带的敏感入口本来就会重新读库，这次没有改它们。咨询里的注销检查和后台的封禁检查仍每次查库。会话能活多久、改密码后是否作废、Cookie 的安全属性、表结构都没改。代价：缓存打开后，一台设备改了密码，其它设备上的旧登录最多还能再浏览 300 秒，这段时间做不了改密码和注销。 +- 验证：`frontend/tests/identity-session-cookie-cache.test.ts` 4 通过。缓存打开时连续 3 次普通读取只查会话 1 次，敏感读取每次都查；缓存关闭时 3 次都查。会话被清掉后，敏感读取立即无效，普通读取在 300 秒内仍有效，超过后无效。 +- 防复发：上述测试锁住开关关闭时不出现缓存配置，以及 300 秒后普通读取失效。 +- 相关记录：BUG-1327。 +- 复发自：无。 +- 修复版本：分支 `codex/node-runtime-resources-20261013`，尚未合入 staging。 + diff --git a/docs/tasks/PROGRESS-node-runtime-resources-20261013.md b/docs/tasks/PROGRESS-node-runtime-resources-20261013.md new file mode 100644 index 00000000..0ba46761 --- /dev/null +++ b/docs/tasks/PROGRESS-node-runtime-resources-20261013.md @@ -0,0 +1,59 @@ +# 进度 · 服务器内存与数据库往返（2026-10-13） + +issue #58。分支 `codex/node-runtime-resources-20261013`。 + +任务书基线写的是 `77057a819`，计划评论里的基线是 `2994588ad4d858dead4835943e5883c5aa2b03f8`。本分支实际从当时的 `origin/staging` `66cd60081f37a57680274633fcb3fb942d01bb2f` 拉出。没有因此停下来问产品，改动仍按任务书做。 + +## 做了什么 + +- 报告工人的等待：计时器到点就摘掉取消监听。引导打分里同一段等待也摘掉。空转不再往进程上堆监听。 +- 本地数据库：列、主键、函数签名缓存 300 秒；设置角色和两条身份合成一条语句。开始和提交仍分开。两个开关默认打开，写成 `off` 时语句顺序与改前一致。 +- 三张内存表：交付记录超过 60 秒删除，最多 1 万条；重算记录超过 6 小时删除，最多 5 千条；限流按见过的最大窗口清理已出窗的键。 +- 登录读缓存：只有 `IDENTITY_SESSION_COOKIE_CACHE=on` 才缓存 300 秒。默认关闭。 + +## 不做的 + +只读、又没有身份时省掉事务：不做。那种做法只能用会话级角色，归还连接前再清。往返数不降，而且一旦没清掉，下一次请求会带着上一次的角色。违反「连接还回池子时不能带着角色或身份」。 + +`begin` 没有并进一条多语句查询。 + +## 语义变化 + +同一道校正题、同一份证据，6 小时后或因为超过 5 千条被挤出后，可能再重算一次。这比「进程重启后才会再算」更收敛。60 秒内的交付去重、窗口内的限流允许或拒绝，都没有改。 + +## 基线失败 + +修之前的等待写法，在第 200 次空转时取消信号上还有 199 个监听。同形脚本输出：`listenersAt200=199`，`assertion failed: 199 !== 0`。修之后的测试断言是 0，并且通过。 + +## 登录读取的调用点 + +敏感，每次跳过缓存： + +- 改密码：`frontend/src/app/api/account/password/route.ts` +- 注销账户：`frontend/src/app/api/account/deletion/route.ts` +- 全部后台入口：`frontend/src/lib/admin/auth.ts` + +其余业务接口走 `createServerSupabaseClient()`，不带跳过参数。开关打开后，这些普通读取可以走 300 秒缓存。 + +改邮箱、两步验证、后台插件自带的敏感入口本来就会重新读库，这次没改。咨询里检查「正在注销」的那次调用，以及后台查封禁的语句，都没有改成缓存。 + +## 对照结论 + +详见 `docs/testing/node-runtime-resources-ab-20261013.md`。两个数据库开关一起：往返 49 → 32（低于 70%），耗时中位数 75.84 → 36.48 毫秒（下降 51.9%），返回内容一致。`npm run test:db` 开关关、开关开各一轮，都是 81 项、通过 68、失败 13、取消 0。失败名称与 `tools/agent-loop/windows-known-failures.txt` 的 13 条相同，没有多出来的。运行器因此退出码是 1，这 13 条是 Windows 上原有的。两个数据库开关默认打开。 + +登录读缓存的查库次数在测试里从 3 降到 1，但没有对真实身份库做耗时对照，默认保持关闭。要打开，把 `IDENTITY_SESSION_COOKIE_CACHE` 设成 `on`。 + +网页容器已经用 env 文件读环境变量，所以这三个名字不必改 Compose。 + +## 自测 + +| 检查 | 命令 | 结果 | +| --- | --- | --- | +| 类型 | `tsc --noEmit -p frontend` | 通过，退出码 0 | +| 规范 | `npm run lint`（在 frontend） | 0 error，125 条原有 warning，未改 | +| 本单测试 | `tsx --test` 任务书点名的文件加登录缓存测试 | 134 通过，0 失败，0 取消 | +| 真库身份 | `database-local-postgres-session-setup.test.ts` | 1 通过。事务内角色和身份生效，连接归还后再借出时身份已空 | +| 数据库套件 | `npm run test:db`，两开关先关后开 | 各 81 / 通过 68 / 失败 13 / 取消 0，名称与已知清单一致 | +| 往返对照 | `scripts/bench-local-pg-roundtrips.ts` | 见上，`payloadMismatch` false | + +没有跑前端全量 `npm test`、Python 快速门、`next build`。生产内存曲线要等部署后看。真机两台设备改密码的步骤在 `docs/testing/node-runtime-resources-20261013.md`，登录缓存默认关着，那份清单的第 2 步要先把开关打开才看得到。 diff --git a/docs/tasks/README.md b/docs/tasks/README.md index 58b326a6..ac1ce4fb 100644 --- a/docs/tasks/README.md +++ b/docs/tasks/README.md @@ -468,7 +468,7 @@ | 任务书 | 进度 | 主题 | 状态 | 落点 | | --- | --- | --- | --- | --- | | `TASK-rect-scoring-perf-20261013.md` | — | 校正打分提速：引导窗口平方扫描、候选不变量重复算、响应带无读取点字段、候选数上限 241（BUG-1323～1325、1345） | 待领取 | `codex/rect-scoring-perf-20261013` | -| `TASK-node-runtime-resources-20261013.md` | — | 报告 worker abort 监听泄漏、本地 PG 适配层往返与元数据缓存、无上限内存表、用户端会话读缓存（BUG-1326～1328、1346，PG 与缓存部分 A/B） | 待领取 | `codex/node-runtime-resources-20261013` | +| `TASK-node-runtime-resources-20261013.md` | `PROGRESS-node-runtime-resources-20261013.md` | 报告 worker abort 监听泄漏、本地 PG 适配层往返与元数据缓存、无上限内存表、用户端会话读缓存（BUG-1326～1328、1346，PG 与缓存部分 A/B） | 待验收 | `codex/node-runtime-resources-20261013` | | `TASK-home-sidebar-rerender-20261013.md` | — | 流式回答时侧栏每帧整栏重渲染（BUG-1329，A/B） | 待领取 | `codex/home-sidebar-rerender-20261013` | | `TASK-api-disk-cache-retention-20261013.md` | — | API 磁盘缓存只进不出、VedAstro 审计文件留存出生资料、失败快照久缓存、缓存键规范化（BUG-1330～1332，A/B） | 待领取 | `codex/api-disk-cache-retention-20261013` | | `TASK-report-reference-perf-20261013.md` | — | 长报告出稿提速：术语替换命中预检、哈希单次遍历、同盘复用（BUG-1335～1336，A/B） | 待领取 | `codex/report-reference-perf-20261013` | diff --git a/docs/testing/node-runtime-resources-20261013.md b/docs/testing/node-runtime-resources-20261013.md new file mode 100644 index 00000000..f7c2a9c5 --- /dev/null +++ b/docs/testing/node-runtime-resources-20261013.md @@ -0,0 +1,13 @@ +# 真机清单 · 登录缓存与长时间运行（2026-10-13） + +对应 BUG-1326、BUG-1346。在 staging 上用两个浏览器（或一台电脑、一台手机）登录同一个测试账号。不要把账号、口令或页面原文记进仓库。 + +本轮代码里，登录读缓存默认关闭。要做下面第 2–4 步，服务器环境变量 `IDENTITY_SESSION_COOKIE_CACHE` 必须是 `on`。未设置或其它值时，改密码后另一台设备会马上要求重新登录，第 2 步不会出现「还能浏览大约 5 分钟」。 + +| 步 | 怎么做 | 通过标准 | +| --- | --- | --- | +| 1 | 两台设备都登录，各自打开首页并翻一两页 | 都能正常看到自己的会话，不要求反复登录 | +| 2 | 设备 A 修改密码。5 分钟内用设备 B 只浏览首页、会话列表和星盘，不要改密码、不要注销 | 设备 B 在这 5 分钟里仍能浏览 | +| 3 | 仍在这 5 分钟内，用设备 B 尝试修改密码或申请注销账户 | 被拒绝，不能拿旧登录做这两件事 | +| 4 | 等到改密码已超过 5 分钟，再用设备 B 刷新 | 被要求重新登录 | +| 5 | 用新密码在设备 B 重新登录，再浏览一次 | 能进入，会话列表还在 | diff --git a/docs/testing/node-runtime-resources-ab-20261013.md b/docs/testing/node-runtime-resources-ab-20261013.md new file mode 100644 index 00000000..22d59c80 --- /dev/null +++ b/docs/testing/node-runtime-resources-ab-20261013.md @@ -0,0 +1,21 @@ +# 本地数据库往返对照（2026-10-14） + +环境：Windows，Docker 29.8.0，Node v22.23.2。脚本 `frontend/scripts/bench-local-pg-roundtrips.ts`。库是测试用的临时 Postgres，数据是虚构用户。每组 50 遍，重复 3 组，先各热身 1 遍。往返数是每组平均值的中位数。耗时中位数是三组中位数的中位数。p95 是全部 150 遍的单遍耗时。 + +判定用第二次运行（输出里带三组原始数）。第一次运行同样是 49 降到 32，中位数 89.51 毫秒降到 32.70 毫秒，内容一致，方向相同。 + +| 模式 | 开关 | 往返 | 中位数（毫秒） | p95（毫秒） | 三组（往返 / 中位数毫秒） | +| --- | --- | --- | --- | --- | --- | +| A | 两个都关 | 49 | 75.84 | 88.25 | 49 / 82.51；49 / 75.84；49 / 73.81 | +| B | 两个都开 | 32 | 36.48 | 39.47 | 32 / 36.66；32 / 36.48；32 / 35.71 | +| B1 | 只开元数据缓存 | 40 | 40.54 | 44.37 | 40 / 41.56；40 / 40.54；40 / 39.86 | +| B2 | 只开合并身份设置 | 41 | 67.83 | 72.75 | 41 / 68.46；41 / 66.47；41 / 67.83 | + +- 往返：32 ≤ 49 的 70%（34.3）。通过。 +- 耗时：下降 (75.84 − 36.48) / 75.84 = 51.9%，不低于 15%。通过。 +- 四种模式的业务返回逐字节一致（`payloadMismatch` 为 false）。通过。 +- 只开其中一个时，往返都还高于 70% 那一刀；只合并身份设置时，这一次耗时只降了约 10.6%。两个一起才过门槛，所以两个都默认打开。写成 `off` 仍走改前的语句顺序。 + +登录读缓存没有放进这次真库计时。内存里的登录测试看到普通读取的查库次数从 3 降到 1，但那不是身份库的耗时对照，所以 `IDENTITY_SESSION_COOKIE_CACHE` 保持默认关闭。 + +`npm run test:db` 两轮都是 81 项、通过 68、失败 13、取消 0。失败名称与 `tools/agent-loop/windows-known-failures.txt` 的 13 条逐条相同，没有多出来的。因此两个数据库开关默认打开。 diff --git a/frontend/scripts/bench-local-pg-roundtrips.ts b/frontend/scripts/bench-local-pg-roundtrips.ts new file mode 100644 index 00000000..51746bcd --- /dev/null +++ b/frontend/scripts/bench-local-pg-roundtrips.ts @@ -0,0 +1,224 @@ +/** + * Count local-Postgres adapter roundtrips for a fictional homepage plus consult write. + * Run from frontend/: npx tsx scripts/bench-local-pg-roundtrips.ts + * Prints JSON. Does not print connection strings. + */ +import { spawnSync } from "node:child_process"; + +import type { Pool, PoolClient } from "pg"; + +import { + closeLocalPostgresDataPools, + clearLocalPostgresMetadataCacheForTests, + createLocalPostgresDataClient, + localPostgresPoolForTests, +} from "../src/lib/db/local-postgres-client-core.ts"; +import { startPostgresFixture } from "../tests/helpers/postgres-fixture.ts"; + +const userId = "018f4e6d-7a11-7000-8000-0000000000bb"; +const sessionId = "018f4e6d-7a11-7000-8000-0000000000cc"; +const chartId = "018f4e6d-7a11-7000-8000-0000000000dd"; + +const schemaSql = ` +create table if not exists public.profiles ( + id uuid primary key, + display_name text +); +create table if not exists public.chart_profiles ( + id uuid primary key, + user_id uuid not null, + label text, + updated_at timestamptz not null default now() +); +create table if not exists public.chat_sessions ( + id uuid primary key, + user_id uuid not null, + title text, + updated_at timestamptz not null default now() +); +grant usage on schema public to authenticated, service_role; +grant select, insert, update, delete on public.profiles, public.chart_profiles, public.chat_sessions to authenticated, service_role; +insert into public.profiles (id, display_name) values ('${userId}', 'Fictional Person') + on conflict (id) do nothing; +insert into public.chart_profiles (id, user_id, label) values ('${chartId}', '${userId}', 'home') + on conflict (id) do nothing; +insert into public.chat_sessions (id, user_id, title) values ('${sessionId}', '${userId}', 'consult') + on conflict (id) do nothing; +create or replace function public.session_probe() +returns table(db_user text) +language sql stable +as $$ select current_user::text $$; +grant execute on function public.session_probe() to service_role, authenticated; +`; + +type Mode = { cache: "on" | "off"; merged: "on" | "off" }; + +function percentile(values: number[], ratio: number): number { + const sorted = [...values].sort((left, right) => left - right); + const index = Math.min(sorted.length - 1, Math.max(0, Math.ceil(ratio * sorted.length) - 1)); + return sorted[index] ?? 0; +} + +function median(values: number[]): number { + const sorted = [...values].sort((left, right) => left - right); + return sorted[Math.floor(sorted.length / 2)] ?? 0; +} + +const benchCounter = { roundtrips: 0 }; +const queryWrapped = new WeakSet(); + +async function withCountingPools( + urls: readonly string[], + run: () => Promise, +): Promise<{ roundtrips: number; elapsedMs: number; payload: string }> { + const restores: Array<() => void> = []; + benchCounter.roundtrips = 0; + for (const url of urls) { + const pool = localPostgresPoolForTests(url) as Pool & { + connect: () => Promise; + }; + const original = pool.connect.bind(pool); + pool.connect = async () => { + const client = await original(); + if (!queryWrapped.has(client)) { + queryWrapped.add(client); + const query = client.query.bind(client) as (input: unknown, values?: unknown) => Promise; + client.query = ((input: unknown, values?: unknown) => { + benchCounter.roundtrips += 1; + return values === undefined ? query(input) : query(input, values); + }) as PoolClient["query"]; + } + return client; + }; + restores.push(() => { + pool.connect = original; + }); + } + const started = performance.now(); + try { + const payload = await run(); + return { + roundtrips: benchCounter.roundtrips, + elapsedMs: performance.now() - started, + payload: JSON.stringify(payload), + }; + } finally { + for (const restore of restores) restore(); + } +} + +async function sequence(url: string, serviceUrl: string): Promise { + const user = createLocalPostgresDataClient(url, { id: userId, email: "person@example.com" }, "authenticated"); + const service = createLocalPostgresDataClient(serviceUrl, null, "service_role"); + const profile = await user.from("profiles").select("id,display_name").eq("id", userId).maybeSingle(); + const charts = await user.from("chart_profiles").select("id,label").eq("user_id", userId).order("updated_at", { ascending: false }).limit(40); + const sessions = await user.from("chat_sessions").select("id,title").eq("user_id", userId).order("updated_at", { ascending: false }).limit(40); + const updated = await user.from("chat_sessions").update({ title: "consult-bench" }).eq("id", sessionId); + const upsertConflict = await user.from("chat_sessions").upsert( + { id: sessionId, user_id: userId, title: "consult-bench" }, + { onConflict: "id" }, + ); + const upsertKey = await user.from("chat_sessions").upsert({ id: sessionId, user_id: userId, title: "consult-bench" }); + const rpc = await service.rpc("session_probe"); + const serviceRead = await service.from("profiles").select("id").eq("id", userId).maybeSingle(); + return [profile, charts, sessions, updated, upsertConflict, upsertKey, rpc, serviceRead].map((result) => ({ + data: result.data ?? null, + error: result.error ?? null, + count: "count" in result ? result.count ?? null : null, + })); +} + +async function measure(url: string, serviceUrl: string, mode: Mode): Promise<{ + roundtrips: number; + medianMs: number; + p95Ms: number; + payload: string; + groups: Array<{ roundtrips: number; medianMs: number }>; +}> { + process.env.JYOTISHA_PG_METADATA_CACHE = mode.cache; + process.env.JYOTISHA_PG_MERGED_SESSION_SETUP = mode.merged; + clearLocalPostgresMetadataCacheForTests(); + await sequence(url, serviceUrl); + const groupMedians: number[] = []; + const groupRoundtrips: number[] = []; + const allTimes: number[] = []; + let payload = ""; + for (let index = 0; index < 3; index += 1) { + const group: number[] = []; + let roundtrips = 0; + for (let repeat = 0; repeat < 50; repeat += 1) { + const sample = await withCountingPools([url, serviceUrl], () => sequence(url, serviceUrl)); + group.push(sample.elapsedMs); + allTimes.push(sample.elapsedMs); + roundtrips += sample.roundtrips; + payload = sample.payload; + } + groupMedians.push(median(group)); + groupRoundtrips.push(Math.round(roundtrips / 50)); + } + return { + roundtrips: median(groupRoundtrips), + medianMs: median(groupMedians), + p95Ms: percentile(allTimes, 0.95), + payload, + groups: groupRoundtrips.map((roundtrips, index) => ({ + roundtrips, + medianMs: Number((groupMedians[index] ?? 0).toFixed(2)), + })), + }; +} + +async function main(): Promise { + const docker = spawnSync("docker", ["info"], { encoding: "utf8" }); + if (docker.status !== 0) { + console.log(JSON.stringify({ ok: false, reason: "docker-unavailable" })); + process.exit(2); + } + const fixture = startPostgresFixture(); + const appUrl = fixture.connectionUrl("app_runtime", "app-runtime-test-password"); + const serviceUrl = fixture.connectionUrl("service_runtime", "service-runtime-test-password"); + try { + fixture.psqlScriptAs("postgres", "postgres-test-password", schemaSql); + const modes: Record = { + A: { cache: "off", merged: "off" }, + B: { cache: "on", merged: "on" }, + B1: { cache: "on", merged: "off" }, + B2: { cache: "off", merged: "on" }, + }; + const results: Record }> = {}; + let payloadA = ""; + let payloadMismatch = false; + for (const [name, mode] of Object.entries(modes)) { + const measured = await measure(appUrl, serviceUrl, mode); + results[name] = { + roundtrips: measured.roundtrips, + medianMs: Number(measured.medianMs.toFixed(2)), + p95Ms: Number(measured.p95Ms.toFixed(2)), + groups: measured.groups, + }; + if (name === "A") payloadA = measured.payload; + else if (measured.payload !== payloadA) payloadMismatch = true; + clearLocalPostgresMetadataCacheForTests(); + await closeLocalPostgresDataPools(); + } + console.log(JSON.stringify({ + ok: true, + repeats: 3, + repsPerGroup: 50, + payloadMismatch, + results, + node: process.version, + })); + } finally { + await closeLocalPostgresDataPools(); + fixture.stop(); + delete process.env.JYOTISHA_PG_METADATA_CACHE; + delete process.env.JYOTISHA_PG_MERGED_SESSION_SETUP; + } +} + +main().catch((error: unknown) => { + const message = error instanceof Error ? error.message : "bench failed"; + console.log(JSON.stringify({ ok: false, reason: message })); + process.exit(1); +}); diff --git a/frontend/src/app/api/account/deletion/route.ts b/frontend/src/app/api/account/deletion/route.ts index 5fba4ad0..db955326 100644 --- a/frontend/src/app/api/account/deletion/route.ts +++ b/frontend/src/app/api/account/deletion/route.ts @@ -21,7 +21,7 @@ export const dynamic = "force-dynamic"; */ async function currentUserId(): Promise { - const supabase = await createServerSupabaseClient(); + const supabase = await createServerSupabaseClient({ disableCookieCache: true }); const { data: { user }, error } = await supabase.auth.getUser(); return error || !user ? null : user.id; } diff --git a/frontend/src/app/api/account/password/route.ts b/frontend/src/app/api/account/password/route.ts index 8a6c9d13..f0a95207 100644 --- a/frontend/src/app/api/account/password/route.ts +++ b/frontend/src/app/api/account/password/route.ts @@ -14,7 +14,10 @@ async function userSession(request: Request) { return null; } const services = getIdentityAuthServices(); - const session = await services.user.api.getSession({ headers: request.headers }); + const session = await services.user.api.getSession({ + headers: request.headers, + query: { disableCookieCache: true }, + }); return session ? { services, session } : null; } diff --git a/frontend/src/lib/admin/auth.ts b/frontend/src/lib/admin/auth.ts index a0363ac2..9d68abfd 100644 --- a/frontend/src/lib/admin/auth.ts +++ b/frontend/src/lib/admin/auth.ts @@ -60,6 +60,7 @@ export async function requirePermission( requireIdentityServerSession( getIdentityAuthServices().user.api, adminHeaders, + { disableCookieCache: true }, ), loadAdminSession, }); diff --git a/frontend/src/lib/birth-time-guided-polling.ts b/frontend/src/lib/birth-time-guided-polling.ts index 891ac002..7d160902 100644 --- a/frontend/src/lib/birth-time-guided-polling.ts +++ b/frontend/src/lib/birth-time-guided-polling.ts @@ -34,10 +34,14 @@ export async function runBirthTimeScoringPoll(input: PollInput): Promise { const duration = Math.min(400 * (2 ** attempt), 3_200); return new Promise((resolve) => { - const timer = window.setTimeout(resolve, duration); - signal.addEventListener("abort", () => { + const onAbort = () => { window.clearTimeout(timer); resolve(); - }, { once: true }); + }; + const timer = window.setTimeout(() => { + signal.removeEventListener("abort", onAbort); + resolve(); + }, duration); + signal.addEventListener("abort", onAbort, { once: true }); }); } diff --git a/frontend/src/lib/db/local-postgres-client-core.ts b/frontend/src/lib/db/local-postgres-client-core.ts index 9dd83eef..a0c2b5a9 100644 --- a/frontend/src/lib/db/local-postgres-client-core.ts +++ b/frontend/src/lib/db/local-postgres-client-core.ts @@ -169,7 +169,18 @@ export function upsertConflictColumns(options?: { onConflict?: string }): string .filter(Boolean); } -async function primaryKeyColumns(client: PoolClient, table: string): Promise { +async function primaryKeyColumns( + client: PoolClient, + table: string, + scope: MetadataScope, +): Promise { + const cacheKey = metadataCacheEnabled() + ? `${scope.connectionString}|${table}` + : null; + if (cacheKey) { + const cached = readTimed(metadataCache().primaryKeys, cacheKey); + if (cached) return [...cached]; + } identifier(table); const result = await client.query<{ column_name: string }>( ` @@ -182,7 +193,9 @@ async function primaryKeyColumns(client: PoolClient, table: string): Promise row.column_name); + const columns = result.rows.map((row) => row.column_name); + if (cacheKey && columns.length > 0) writeTimed(metadataCache().primaryKeys, cacheKey, [...columns]); + return columns; } function queryError(error: unknown): QueryError { @@ -220,7 +233,115 @@ function localDataPool(connectionString: string): Pool { return pool; } +const METADATA_TTL_MS = 300_000; +const METADATA_INVALIDATING_CODES = new Set(["42703", "42P01", "42883", "42804"]); + +type MetadataScope = Readonly<{ + connectionString: string; + role: LocalDatabaseRole | null; +}>; + +type TimedCacheEntry = Readonly<{ value: T; storedAt: number }>; + +type LocalPostgresMetadataCache = { + columns: Map>>; + primaryKeys: Map>; + functions: Map>; +}; + +const metadataGlobal = globalThis as typeof globalThis & { + jyotishaLocalPostgresMetadata?: LocalPostgresMetadataCache; + jyotishaLocalPostgresMetadataNow?: () => number; +}; + +function metadataCache(): LocalPostgresMetadataCache { + metadataGlobal.jyotishaLocalPostgresMetadata ??= { + columns: new Map(), + primaryKeys: new Map(), + functions: new Map(), + }; + return metadataGlobal.jyotishaLocalPostgresMetadata; +} + +function metadataNow(): number { + return metadataGlobal.jyotishaLocalPostgresMetadataNow?.() ?? Date.now(); +} + +function metadataCacheEnabled(): boolean { + // 2026-10-14 A/B passed. Unset means on. Exact "off" keeps the old query sequence. + return process.env.JYOTISHA_PG_METADATA_CACHE !== "off"; +} + +function mergedSessionSetupEnabled(): boolean { + return process.env.JYOTISHA_PG_MERGED_SESSION_SETUP !== "off"; +} + +function roleCacheKey(role: LocalDatabaseRole | null): string { + return role ?? "owner"; +} + +function readTimed( + store: Map>, + key: string, +): T | undefined { + const entry = store.get(key); + if (!entry) return undefined; + if (metadataNow() - entry.storedAt >= METADATA_TTL_MS) { + store.delete(key); + return undefined; + } + return entry.value; +} + +function writeTimed( + store: Map>, + key: string, + value: T, +): void { + store.set(key, { value, storedAt: metadataNow() }); +} + +function dropMetadataForConnection(connectionString: string): void { + const cache = metadataGlobal.jyotishaLocalPostgresMetadata; + if (!cache) return; + const prefix = `${connectionString}|`; + for (const store of [cache.columns, cache.primaryKeys, cache.functions]) { + for (const key of store.keys()) { + if (key.startsWith(prefix)) store.delete(key); + } + } +} + +function postgresErrorCode(error: unknown): string | undefined { + if (!error || typeof error !== "object") return undefined; + const code = (error as { code?: unknown }).code; + return typeof code === "string" ? code : undefined; +} + +export function clearLocalPostgresMetadataCacheForTests(): void { + metadataGlobal.jyotishaLocalPostgresMetadata = { + columns: new Map(), + primaryKeys: new Map(), + functions: new Map(), + }; +} + +export function setLocalPostgresMetadataClockForTests(now: (() => number) | null): void { + if (now) metadataGlobal.jyotishaLocalPostgresMetadataNow = now; + else delete metadataGlobal.jyotishaLocalPostgresMetadataNow; +} + +export function installLocalPostgresPoolForTests(connectionString: string, pool: Pool): void { + poolGlobal.jyotishaLocalDataPools ??= new Map(); + poolGlobal.jyotishaLocalDataPools.set(connectionString, pool); +} + +export function localPostgresPoolForTests(connectionString: string): Pool { + return localDataPool(connectionString); +} + export async function closeLocalPostgresDataPool(connectionString: string): Promise { + dropMetadataForConnection(connectionString); const pools = poolGlobal.jyotishaLocalDataPools; const pool = pools?.get(connectionString); if (!pool) return; @@ -229,6 +350,11 @@ export async function closeLocalPostgresDataPool(connectionString: string): Prom } export async function closeLocalPostgresDataPools(): Promise { + metadataGlobal.jyotishaLocalPostgresMetadata = { + columns: new Map(), + primaryKeys: new Map(), + functions: new Map(), + }; const pools = poolGlobal.jyotishaLocalDataPools; poolGlobal.jyotishaLocalDataPools = new Map(); if (!pools) return; @@ -237,6 +363,7 @@ export async function closeLocalPostgresDataPools(): Promise { async function inBusinessTransaction( pool: Pool, + connectionString: string, identity: LocalIdentity, role: LocalDatabaseRole | null, run: (client: PoolClient) => Promise, @@ -244,15 +371,32 @@ async function inBusinessTransaction( const client = await pool.connect(); try { await client.query("begin"); - if (role !== null) await client.query(`set local role ${role}`); - await client.query( - "select set_config('request.jwt.claim.sub', $1, true), set_config('request.jwt.claim.email', $2, true)", - [identity?.id ?? "", identity?.email ?? ""], - ); + if (mergedSessionSetupEnabled()) { + if (role !== null) { + await client.query( + "select set_config('role', $1, true), set_config('request.jwt.claim.sub', $2, true), set_config('request.jwt.claim.email', $3, true)", + [role, identity?.id ?? "", identity?.email ?? ""], + ); + } else { + await client.query( + "select set_config('request.jwt.claim.sub', $1, true), set_config('request.jwt.claim.email', $2, true)", + [identity?.id ?? "", identity?.email ?? ""], + ); + } + } else { + if (role !== null) await client.query(`set local role ${role}`); + await client.query( + "select set_config('request.jwt.claim.sub', $1, true), set_config('request.jwt.claim.email', $2, true)", + [identity?.id ?? "", identity?.email ?? ""], + ); + } const result = await run(client); await client.query("commit"); return result; } catch (error) { + if (METADATA_INVALIDATING_CODES.has(postgresErrorCode(error) ?? "")) { + dropMetadataForConnection(connectionString); + } await client.query("rollback").catch(() => undefined); throw error; } finally { @@ -263,7 +407,15 @@ async function inBusinessTransaction( async function columnTypes( client: PoolClient, table: string, + scope: MetadataScope, ): Promise> { + const cacheKey = metadataCacheEnabled() + ? `${scope.connectionString}|${roleCacheKey(scope.role)}|${table}` + : null; + if (cacheKey) { + const cached = readTimed(metadataCache().columns, cacheKey); + if (cached) return new Map(cached); + } const result = await client.query<{ column_name: string; udt_name: string }>( ` select column_name, udt_name @@ -272,7 +424,9 @@ async function columnTypes( `, [table], ); - return new Map(result.rows.map((row) => [row.column_name, row.udt_name])); + const types = new Map(result.rows.map((row) => [row.column_name, row.udt_name])); + if (cacheKey && types.size > 0) writeTimed(metadataCache().columns, cacheKey, new Map(types)); + return types; } function databaseValue(type: string | undefined, value: unknown): unknown { @@ -314,6 +468,7 @@ class LocalPostgresQueryBuilder implements PromiseLike { constructor( private readonly pool: Pool, + private readonly connectionString: string, private readonly identity: LocalIdentity, private readonly role: LocalDatabaseRole | null, private readonly table: string, @@ -516,10 +671,14 @@ class LocalPostgresQueryBuilder implements PromiseLike { try { return await inBusinessTransaction( this.pool, + this.connectionString, this.identity, this.role, async (client) => { - const types = await columnTypes(client, this.table); + const types = await columnTypes(client, this.table, { + connectionString: this.connectionString, + role: this.role, + }); const parameters: unknown[] = []; let sql: string; @@ -563,7 +722,10 @@ class LocalPostgresQueryBuilder implements PromiseLike { if (this.mutation.kind === "upsert") { let conflict = this.mutation.conflict; if (conflict.length === 0) { - conflict = await primaryKeyColumns(client, this.table); + conflict = await primaryKeyColumns(client, this.table, { + connectionString: this.connectionString, + role: this.role, + }); } if (conflict.length === 0) { throw new Error("upsert requires a conflict target"); @@ -648,7 +810,21 @@ async function functionMetadata( client: PoolClient, functionName: string, argumentNames: readonly string[], + scope: MetadataScope, ): Promise { + const cacheKey = metadataCacheEnabled() + ? `${scope.connectionString}|${roleCacheKey(scope.role)}|${functionName}|${[...argumentNames].sort().join(",")}` + : null; + if (cacheKey) { + const cached = readTimed(metadataCache().functions, cacheKey); + if (cached) { + return { + ...cached, + argument_names: cached.argument_names ? [...cached.argument_names] : null, + argument_types: [...cached.argument_types], + }; + } + } const result = await client.query( ` select @@ -671,6 +847,13 @@ async function functionMetadata( ); const metadata = result.rows[0]; if (!metadata) throw new Error("database function not found"); + if (cacheKey) { + writeTimed(metadataCache().functions, cacheKey, { + ...metadata, + argument_names: metadata.argument_names ? [...metadata.argument_names] : null, + argument_types: [...metadata.argument_types], + }); + } return metadata; } @@ -691,12 +874,14 @@ export class LocalPostgresDataClient { }>; private readonly pool: Pool; + private readonly connectionString: string; constructor( connectionString: string, private readonly identity: LocalIdentity, private readonly role: LocalDatabaseRole | null, ) { + this.connectionString = connectionString; if (role !== null && role !== "authenticated" && role !== "service_role") { throw new Error("unsupported database role"); } @@ -709,6 +894,7 @@ export class LocalPostgresDataClient { from(table: string) { return new LocalPostgresQueryBuilder( this.pool, + this.connectionString, this.identity, this.role, table, @@ -723,12 +909,16 @@ export class LocalPostgresDataClient { identifier(functionName); return await inBusinessTransaction( this.pool, + this.connectionString, this.identity, this.role, async (client) => { const names = Object.keys(args); names.forEach(identifier); - const metadata = await functionMetadata(client, functionName, names); + const metadata = await functionMetadata(client, functionName, names, { + connectionString: this.connectionString, + role: this.role, + }); const typeByName = new Map( (metadata.argument_names ?? []).map((name, index) => [ name, diff --git a/frontend/src/lib/personal-report-worker-core.ts b/frontend/src/lib/personal-report-worker-core.ts index 650f5128..47df7562 100644 --- a/frontend/src/lib/personal-report-worker-core.ts +++ b/frontend/src/lib/personal-report-worker-core.ts @@ -173,13 +173,17 @@ function defaultRetryDelayMs(job: PersonalReportJobRecord): number { async function defaultSleep(milliseconds: number, signal?: AbortSignal): Promise { if (signal?.aborted) return; await new Promise((resolve) => { - const timer = globalThis.setTimeout(resolve, milliseconds); - const candidate = timer as ReturnType & { unref?: () => void }; - candidate.unref?.(); - signal?.addEventListener("abort", () => { + const onAbort = () => { globalThis.clearTimeout(timer); resolve(); - }, { once: true }); + }; + const timer = globalThis.setTimeout(() => { + signal?.removeEventListener("abort", onAbort); + resolve(); + }, milliseconds); + const candidate = timer as ReturnType & { unref?: () => void }; + candidate.unref?.(); + signal?.addEventListener("abort", onAbort, { once: true }); }); } diff --git a/frontend/src/lib/rectification-agentic/v9/block-scan-answer.ts b/frontend/src/lib/rectification-agentic/v9/block-scan-answer.ts index 9a42f1c6..c5545a8b 100644 --- a/frontend/src/lib/rectification-agentic/v9/block-scan-answer.ts +++ b/frontend/src/lib/rectification-agentic/v9/block-scan-answer.ts @@ -188,12 +188,47 @@ export async function rescoreMinuteAfterWindowChange( export { persistBlockScanPayload } from "./score-persist.ts"; -const rescoreAttempts = new Map(); +const RESCORE_ATTEMPT_TTL_MS = 6 * 60 * 60 * 1000; +const RESCORE_ATTEMPT_CAP = 5_000; + +const rescoreAttempts = new Map(); export function resetStaleMinuteRescoreAttemptsForTests(): void { rescoreAttempts.clear(); } +function pruneRescoreAttempts(now: number): void { + for (const [key, at] of rescoreAttempts) { + if (now - at > RESCORE_ATTEMPT_TTL_MS) rescoreAttempts.delete(key); + } + while (rescoreAttempts.size > RESCORE_ATTEMPT_CAP) { + const oldest = rescoreAttempts.keys().next().value; + if (oldest === undefined) return; + rescoreAttempts.delete(oldest); + } +} + +/** Returns true when this key should be scored now. A fresh key is remembered. */ +function claimRescoreAttempt(key: string, now: number): boolean { + pruneRescoreAttempts(now); + const existing = rescoreAttempts.get(key); + if (existing !== undefined && now - existing <= RESCORE_ATTEMPT_TTL_MS) return false; + if (!rescoreAttempts.has(key) && rescoreAttempts.size >= RESCORE_ATTEMPT_CAP) { + const oldest = rescoreAttempts.keys().next().value; + if (oldest !== undefined) rescoreAttempts.delete(oldest); + } + rescoreAttempts.set(key, now); + return true; +} + +export function noteRescoreAttemptForTests(key: string, at: number): void { + claimRescoreAttempt(key, at); +} + +export function rescoreAttemptCountForTests(): number { + return rescoreAttempts.size; +} + function rescoreAttemptKey(caseId: string, fingerprint: string, askedTurnId?: string | null): string { return `${caseId}:${askedTurnId ?? ""}:${fingerprint}`; } @@ -219,10 +254,9 @@ export async function rescoreStaleMinuteSnapshotIfNeeded(input: { } const fingerprint = evidenceLedgerFingerprint(dossier.evidence); const key = rescoreAttemptKey(input.caseId, fingerprint, input.askedTurnId); - if (rescoreAttempts.has(key)) { + if (!claimRescoreAttempt(key, Date.now())) { return { dossier, snapshotCurrent: false, rescoreAttempted: false }; } - rescoreAttempts.set(key, true); try { await rescoreMinuteAfterWindowChange(input.accounting, input.userId, input.caseId, { keepAnswers: true, diff --git a/frontend/src/lib/rectification-agentic/v9/delivery-turn-guard.ts b/frontend/src/lib/rectification-agentic/v9/delivery-turn-guard.ts index 0de07fb1..93ce9d54 100644 --- a/frontend/src/lib/rectification-agentic/v9/delivery-turn-guard.ts +++ b/frontend/src/lib/rectification-agentic/v9/delivery-turn-guard.ts @@ -6,11 +6,30 @@ */ const WINDOW_MS = 60_000; +const DELIVERY_MARK_CAP = 10_000; type DeliveryMark = Readonly<{ resultId: string; at: number; narration?: string }>; const delivered = new Map(); +function pruneDelivered(now: number): void { + for (const [caseId, mark] of delivered) { + if (now - mark.at > WINDOW_MS) delivered.delete(caseId); + } +} + +function makeRoomForDeliveryMark(): void { + while (delivered.size >= DELIVERY_MARK_CAP) { + const oldest = delivered.keys().next().value; + if (oldest === undefined) return; + delivered.delete(oldest); + } +} + +export function deliveryTurnGuardSizeForTests(): number { + return delivered.size; +} + export type DeliveryTurnTrigger = | "finalizeSuccessfulTurnExit" | "persistExhaustionGateTurn" @@ -39,8 +58,10 @@ export function markDeliveryTurn(input: { }): void { if (!input.caseId || !input.resultId) return; const now = input.now ?? Date.now(); + pruneDelivered(now); const narration = input.narration?.replace(/\s+/g, " ").trim() || undefined; const existing = delivered.get(input.caseId); + if (!existing) makeRoomForDeliveryMark(); if (existing && existing.resultId === input.resultId && now - existing.at <= WINDOW_MS) { if (existing.narration) return; delivered.set(input.caseId, { diff --git a/frontend/src/lib/request-rate-limit.ts b/frontend/src/lib/request-rate-limit.ts index 3f67e647..9165c939 100644 --- a/frontend/src/lib/request-rate-limit.ts +++ b/frontend/src/lib/request-rate-limit.ts @@ -32,9 +32,36 @@ function defaultStore(): RequestRateLimitStore { return state.jyotishaRequestRateLimits; } +let seenMaxWindowMs = 0; +let consumeCalls = 0; + +export function resetRequestRateLimitMaintenanceForTests(): void { + seenMaxWindowMs = 0; + consumeCalls = 0; +} + +function sweepExpiredRateLimitKeys( + store: RequestRateLimitStore, + now: number, + maxWindowMs: number, +): void { + for (const [key, stamps] of store) { + let newest = Number.NEGATIVE_INFINITY; + for (const stamp of stamps) { + if (stamp > newest) newest = stamp; + } + if (!Number.isFinite(newest) || newest <= now - maxWindowMs) store.delete(key); + } +} + export function consumeRequestRateLimit(input: RequestRateLimitInput): RequestRateLimitDecision { + if (input.windowMs > seenMaxWindowMs) seenMaxWindowMs = input.windowMs; + consumeCalls += 1; const now = input.now ?? Date.now(); const store = input.store ?? defaultStore(); + if (consumeCalls % 1_000 === 0 || store.size > 5_000) { + sweepExpiredRateLimitKeys(store, now, seenMaxWindowMs); + } const windowStart = now - input.windowMs; const recent = (store.get(input.key) ?? []).filter((timestamp) => timestamp > windowStart); if (recent.length >= input.limit) { diff --git a/frontend/src/lib/supabase/server.ts b/frontend/src/lib/supabase/server.ts index 7d7b984d..351603bd 100644 --- a/frontend/src/lib/supabase/server.ts +++ b/frontend/src/lib/supabase/server.ts @@ -9,12 +9,15 @@ import { getIdentityAuthServices } from "@/modules/identity/auth"; import { readIdentitySession } from "@/modules/identity/session"; import { getSupabasePublicConfig } from "./config"; -export async function createServerSupabaseClient() { +export async function createServerSupabaseClient(options?: { + readonly disableCookieCache?: boolean; +}) { if (process.env.AUTH_PROVIDER?.trim() === "self-hosted") { const requestHeaders = new Headers(await headers()); const session = await readIdentitySession( getIdentityAuthServices().user.api, requestHeaders, + options?.disableCookieCache ? { disableCookieCache: true } : undefined, ); return createLocalPostgresDataClient( readDatabaseUrl(process.env, "APP_DATABASE_URL"), diff --git a/frontend/src/modules/identity/auth-factory.ts b/frontend/src/modules/identity/auth-factory.ts index 9814102f..458e0e1d 100644 --- a/frontend/src/modules/identity/auth-factory.ts +++ b/frontend/src/modules/identity/auth-factory.ts @@ -1,5 +1,4 @@ import { createHmac } from "node:crypto"; -import type { Pool } from "pg"; import type { BetterAuthOptions } from "better-auth"; import { admin, emailOTP, twoFactor, type EmailOTPOptions } from "better-auth/plugins"; @@ -11,10 +10,19 @@ export type AdminUserAuthorizer = (userId: string) => Promise; interface BuildAuthOptionsInput { config: SelfHostedIdentityConfig; - database: Pool; + database: BetterAuthOptions["database"]; emailSender: EmailOtpSender; } +export const IDENTITY_SESSION_COOKIE_CACHE_MAX_AGE_SECONDS = 300; + +/** Server-only. Unset or any value other than "on" keeps today's uncached session reads. */ +export function identitySessionCookieCacheEnabled( + env: NodeJS.ProcessEnv = process.env, +): boolean { + return env.IDENTITY_SESSION_COOKIE_CACHE === "on"; +} + function otpIdempotencyKey( secret: string, email: string, @@ -86,7 +94,15 @@ export function buildAuthOptions({ trustedOrigins: [config.userOrigin, config.adminOrigin], telemetry: { enabled: false }, user: identityModelMapping.user, - session: identityModelMapping.session, + session: identitySessionCookieCacheEnabled() + ? { + ...identityModelMapping.session, + cookieCache: { + enabled: true, + maxAge: IDENTITY_SESSION_COOKIE_CACHE_MAX_AGE_SECONDS, + }, + } + : identityModelMapping.session, account: identityModelMapping.account, verification: identityModelMapping.verification, rateLimit: { diff --git a/frontend/src/modules/identity/session.ts b/frontend/src/modules/identity/session.ts index 0ba8c87f..849aa187 100644 --- a/frontend/src/modules/identity/session.ts +++ b/frontend/src/modules/identity/session.ts @@ -18,8 +18,15 @@ export interface IdentityServerSession extends IdentitySession { sessionToken: string; } +export interface IdentitySessionQuery { + readonly disableCookieCache?: boolean; +} + export interface IdentitySessionReader { - getSession(input: { headers: Headers }): Promise; + getSession(input: { + headers: Headers; + query?: IdentitySessionQuery; + }): Promise; } export class IdentityAuthorizationError extends Error { @@ -43,8 +50,12 @@ function parseRoles(role: string | null | undefined): string[] { export async function readIdentityServerSession( reader: IdentitySessionReader, requestHeaders: Headers, + options?: IdentitySessionQuery, ): Promise { - const value = await reader.getSession({ headers: requestHeaders }); + const value = await reader.getSession({ + headers: requestHeaders, + ...(options?.disableCookieCache ? { query: { disableCookieCache: true } } : {}), + }); if (!value) return null; const expiresAt = new Date(value.session.expiresAt); @@ -71,16 +82,18 @@ export async function readIdentityServerSession( export async function readIdentitySession( reader: IdentitySessionReader, requestHeaders: Headers, + options?: IdentitySessionQuery, ): Promise { - const session = await readIdentityServerSession(reader, requestHeaders); + const session = await readIdentityServerSession(reader, requestHeaders, options); return session ? { user: session.user, expiresAt: session.expiresAt } : null; } export async function requireIdentityServerSession( reader: IdentitySessionReader, requestHeaders: Headers, + options?: IdentitySessionQuery, ): Promise { - const session = await readIdentityServerSession(reader, requestHeaders); + const session = await readIdentityServerSession(reader, requestHeaders, options); if (!session) throw new IdentityAuthorizationError("Authentication required", 401); return session; } diff --git a/frontend/tests/database-local-postgres-session-setup.test.ts b/frontend/tests/database-local-postgres-session-setup.test.ts new file mode 100644 index 00000000..923c7b45 --- /dev/null +++ b/frontend/tests/database-local-postgres-session-setup.test.ts @@ -0,0 +1,64 @@ +import assert from "node:assert/strict"; +import test from "node:test"; + +import { + closeLocalPostgresDataPools, + createLocalPostgresDataClient, + localPostgresPoolForTests, +} from "../src/lib/db/local-postgres-client-core.ts"; +import { startPostgresFixture } from "./helpers/postgres-fixture.ts"; + +const probeSql = ` +create or replace function public.session_probe() +returns table(db_user text, claim_sub text) +language sql +stable +as $$ + select current_user::text, current_setting('request.jwt.claim.sub', true); +$$; +grant usage on schema public to authenticated, service_role; +grant execute on function public.session_probe() to authenticated, service_role; +`; + +test("merged session setup applies the role only inside the transaction", async () => { + const previous = { + cache: process.env.JYOTISHA_PG_METADATA_CACHE, + merged: process.env.JYOTISHA_PG_MERGED_SESSION_SETUP, + }; + process.env.JYOTISHA_PG_METADATA_CACHE = "on"; + process.env.JYOTISHA_PG_MERGED_SESSION_SETUP = "on"; + const fixture = startPostgresFixture(); + const identity = { + id: "018f4e6d-7a11-7000-8000-0000000000aa", + email: "person@example.com", + }; + try { + fixture.psqlScriptAs("postgres", "postgres-test-password", probeSql); + const url = fixture.connectionUrl("app_runtime", "app-runtime-test-password"); + const client = createLocalPostgresDataClient(url, identity, "authenticated"); + const probe = await client.rpc("session_probe"); + assert.equal(probe.error, null, probe.error?.message); + const row = Array.isArray(probe.data) ? probe.data[0] as { db_user?: string; claim_sub?: string } : null; + assert.equal(row?.db_user, "authenticated"); + assert.equal(row?.claim_sub, identity.id); + + const pool = localPostgresPoolForTests(url); + const borrowed = await pool.connect(); + try { + const after = await borrowed.query<{ db_user: string; claim_sub: string | null }>( + "select current_user as db_user, current_setting('request.jwt.claim.sub', true) as claim_sub", + ); + assert.equal(after.rows[0]?.db_user, "app_runtime"); + assert.equal(after.rows[0]?.claim_sub ?? "", ""); + } finally { + borrowed.release(); + } + } finally { + await closeLocalPostgresDataPools(); + fixture.stop(); + if (previous.cache === undefined) delete process.env.JYOTISHA_PG_METADATA_CACHE; + else process.env.JYOTISHA_PG_METADATA_CACHE = previous.cache; + if (previous.merged === undefined) delete process.env.JYOTISHA_PG_MERGED_SESSION_SETUP; + else process.env.JYOTISHA_PG_MERGED_SESSION_SETUP = previous.merged; + } +}); diff --git a/frontend/tests/delivery-turn-guard-expiry.test.ts b/frontend/tests/delivery-turn-guard-expiry.test.ts new file mode 100644 index 00000000..f18c506e --- /dev/null +++ b/frontend/tests/delivery-turn-guard-expiry.test.ts @@ -0,0 +1,61 @@ +import assert from "node:assert/strict"; +import test from "node:test"; + +import { + alreadyDelivered, + deliveryCarrierNarration, + deliveryTurnGuardSizeForTests, + markDeliveryTurn, + resetDeliveryTurnGuardForTests, +} from "../src/lib/rectification-agentic/v9/delivery-turn-guard.ts"; + +test("a delivery mark older than 60 seconds is removed on the next write", () => { + resetDeliveryTurnGuardForTests(); + markDeliveryTurn({ + caseId: "case-a", + resultId: "result-a", + now: 0, + narration: "已经交付这段说明", + }); + assert.equal(alreadyDelivered({ + caseId: "case-a", + resultId: "result-a", + hasUserMessage: false, + now: 60_000, + }), true); + assert.equal(deliveryCarrierNarration({ + caseId: "case-a", + resultId: "result-a", + now: 60_000, + }), "已经交付这段说明"); + + markDeliveryTurn({ caseId: "case-b", resultId: "result-b", now: 60_001 }); + assert.equal(deliveryTurnGuardSizeForTests(), 1); + assert.equal(alreadyDelivered({ + caseId: "case-a", + resultId: "result-a", + hasUserMessage: false, + now: 60_001, + }), false); + assert.equal(alreadyDelivered({ + caseId: "case-b", + resultId: "result-b", + hasUserMessage: false, + now: 70_000, + }), true); +}); + +test("delivery marks stay within the hard cap", () => { + resetDeliveryTurnGuardForTests(); + for (let index = 0; index < 10_001; index += 1) { + markDeliveryTurn({ caseId: `case-${index}`, resultId: "result", now: 1_000 }); + } + assert.ok(deliveryTurnGuardSizeForTests() <= 10_000); + assert.equal(alreadyDelivered({ + caseId: "case-10000", + resultId: "result", + hasUserMessage: false, + now: 1_000, + }), true); + resetDeliveryTurnGuardForTests(); +}); diff --git a/frontend/tests/identity-session-cookie-cache.test.ts b/frontend/tests/identity-session-cookie-cache.test.ts new file mode 100644 index 00000000..511391ee --- /dev/null +++ b/frontend/tests/identity-session-cookie-cache.test.ts @@ -0,0 +1,258 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { betterAuth } from "better-auth"; +import { memoryAdapter, type MemoryDB } from "../node_modules/better-auth/node_modules/@better-auth/memory-adapter/dist/index.mjs"; +import type { BetterAuthOptions } from "better-auth"; + +import { + buildAuthOptions, + IDENTITY_SESSION_COOKIE_CACHE_MAX_AGE_SECONDS, +} from "../src/modules/identity/auth-factory.ts"; +import { FakeEmailOtpSender } from "../src/modules/identity/email/fake-email-otp-sender.ts"; +import type { SelfHostedIdentityConfig } from "../src/modules/identity/config.ts"; +import { + readIdentitySession, + requireIdentityServerSession, + type IdentitySessionReader, +} from "../src/modules/identity/session.ts"; + +const config: SelfHostedIdentityConfig = { + provider: "self-hosted", + databaseUrl: "postgresql://identity_runtime:test-password@postgres:5432/jyotisha", + userOrigin: "https://staging.jyotisha.chat", + adminOrigin: "https://admin.staging.jyotisha.chat", + userSecret: "user-secret-that-is-at-least-32-bytes-long", + resendApiKey: "re_test", + resendFrom: "Jyotisha ", + testOtp: null, +}; + +function withCacheFlag(value: string | undefined, run: () => void): void { + const previous = process.env.IDENTITY_SESSION_COOKIE_CACHE; + if (value === undefined) delete process.env.IDENTITY_SESSION_COOKIE_CACHE; + else process.env.IDENTITY_SESSION_COOKIE_CACHE = value; + try { + run(); + } finally { + if (previous === undefined) delete process.env.IDENTITY_SESSION_COOKIE_CACHE; + else process.env.IDENTITY_SESSION_COOKIE_CACHE = previous; + } +} + +test("user auth omits cookie cache unless the server switch is on", () => { + withCacheFlag(undefined, () => { + const options = buildAuthOptions({ + config, + database: memoryAdapter({}), + emailSender: new FakeEmailOtpSender(), + }); + assert.equal(options.advanced?.cookiePrefix, "jyotisha-user"); + assert.equal(options.session && "cookieCache" in options.session, false); + }); + withCacheFlag("on", () => { + const options = buildAuthOptions({ + config, + database: memoryAdapter({}), + emailSender: new FakeEmailOtpSender(), + }); + assert.deepEqual(options.session?.cookieCache, { + enabled: true, + maxAge: IDENTITY_SESSION_COOKIE_CACHE_MAX_AGE_SECONDS, + }); + }); +}); + +test("sensitive session readers pass disableCookieCache and ordinary readers do not", async () => { + const queries: Array<{ disableCookieCache?: boolean }> = []; + const reader: IdentitySessionReader = { + async getSession(input) { + queries.push({ disableCookieCache: input.query?.disableCookieCache }); + return { + session: { + id: "session-1", + token: "session-token", + expiresAt: new Date("2030-01-01T00:00:00.000Z"), + }, + user: { + id: "018f4e6d-7a11-7000-8000-000000000001", + email: "person@example.com", + emailVerified: true, + name: "Person", + image: null, + role: "user", + }, + }; + }, + }; + await readIdentitySession(reader, new Headers()); + await requireIdentityServerSession(reader, new Headers(), { disableCookieCache: true }); + assert.deepEqual(queries, [ + { disableCookieCache: undefined }, + { disableCookieCache: true }, + ]); +}); + +type SetCookieHeaders = Headers & { getSetCookie?: () => string[] }; + +function cookieHeaderFrom(response: Response, previous = ""): string { + const headers = response.headers as SetCookieHeaders; + const setCookies = headers.getSetCookie?.() + ?? (headers.get("set-cookie")?.split(/,(?=\s*[^;,=\s]+=[^;,]*)/g) ?? []); + const fresh = setCookies + .map((value) => value.split(";", 1)[0] ?? "") + .filter((value) => value.includes("=") && value.slice(value.indexOf("=") + 1).length > 0); + const jar = new Map(); + for (const part of previous.split("; ").filter(Boolean)) { + jar.set(part.slice(0, part.indexOf("=")), part); + } + for (const part of fresh) jar.set(part.slice(0, part.indexOf("=")), part); + return [...jar.values()].join("; "); +} + +function createCacheAuth(enabled: boolean) { + const db: MemoryDB = { + users: [], + sessions: [], + accounts: [], + verifications: [], + two_factors: [], + otp_rate_limits: [], + }; + const sessionReads: string[] = []; + const factory = memoryAdapter(db); + const database = ((options: BetterAuthOptions) => { + const adapter = factory(options); + return { + ...adapter, + async findOne(data: { model: string }) { + if (data.model === "session" || data.model === "sessions") sessionReads.push(data.model); + return adapter.findOne(data as never); + }, + }; + }) as BetterAuthOptions["database"]; + const previous = process.env.IDENTITY_SESSION_COOKIE_CACHE; + process.env.IDENTITY_SESSION_COOKIE_CACHE = enabled ? "on" : "off"; + try { + const built = buildAuthOptions({ + config, + database, + emailSender: new FakeEmailOtpSender(), + }); + const auth = betterAuth({ + ...built, + emailAndPassword: { + ...built.emailAndPassword, + enabled: true, + disableSignUp: false, + }, + }); + return { auth, db, sessionReads }; + } finally { + if (previous === undefined) delete process.env.IDENTITY_SESSION_COOKIE_CACHE; + else process.env.IDENTITY_SESSION_COOKIE_CACHE = previous; + } +} + +type CacheAuth = ReturnType["auth"]; + +async function signIn(auth: CacheAuth): Promise { + const email = "person@example.com"; + const password = "long-enough-password"; + const origin = config.userOrigin; + const signedUp = await auth.api.signUpEmail({ + body: { email, password, name: "Person" }, + headers: new Headers({ origin }), + asResponse: true, + }); + assert.equal(signedUp.status, 200, await signedUp.clone().text()); + let cookie = cookieHeaderFrom(signedUp); + if (!cookie.includes("session_token")) { + const signedIn = await auth.api.signInEmail({ + body: { email, password }, + headers: new Headers({ origin }), + asResponse: true, + }); + assert.equal(signedIn.status, 200, await signedIn.clone().text()); + cookie = cookieHeaderFrom(signedIn, cookie); + } + return cookie; +} + +async function readSession( + auth: CacheAuth, + cookie: string, + disableCookieCache = false, +): Promise<{ cookie: string; status: number; userId: string | null }> { + const response = await auth.api.getSession({ + headers: new Headers({ cookie, origin: config.userOrigin }), + query: disableCookieCache ? { disableCookieCache: true } : undefined, + asResponse: true, + }); + const body = await response.json() as { user?: { id?: string } } | null; + return { + cookie: cookieHeaderFrom(response, cookie), + status: response.status, + userId: body?.user?.id ?? null, + }; +} + +test("cookie cache serves three ordinary reads from one database read", async () => { + const { auth, db, sessionReads } = createCacheAuth(true); + const cookie = await signIn(auth); + sessionReads.length = 0; + let current = cookie + .split("; ") + .filter((part) => !part.slice(0, part.indexOf("=")).includes("session_data")) + .join("; "); + const reads: Array = []; + for (let index = 0; index < 3; index += 1) { + const result = await readSession(auth, current); + current = result.cookie; + reads.push(result.userId); + } + assert.deepEqual(reads, [reads[0], reads[0], reads[0]]); + assert.ok(reads[0]); + assert.equal(sessionReads.length, 1); + const cachedCookie = current; + + sessionReads.length = 0; + for (let index = 0; index < 3; index += 1) { + const result = await readSession(auth, cachedCookie, true); + assert.ok(result.userId); + } + assert.equal(sessionReads.length, 3); + + const sessions = db.sessions ?? db.session; + assert.ok(Array.isArray(sessions)); + sessions.length = 0; + const sensitive = await readSession(auth, cachedCookie, true); + assert.equal(sensitive.userId, null); + const ordinary = await readSession(auth, cachedCookie, false); + assert.equal(ordinary.userId, reads[0]); + + const realNow = Date.now; + const started = realNow(); + Date.now = () => started + (IDENTITY_SESSION_COOKIE_CACHE_MAX_AGE_SECONDS + 1) * 1000; + try { + const expired = await readSession(auth, cachedCookie, false); + assert.equal(expired.userId, null); + } finally { + Date.now = realNow; + } +}); + +test("cookie cache off reads the database on every ordinary request", async () => { + const { auth, sessionReads } = createCacheAuth(false); + const cookie = await signIn(auth); + sessionReads.length = 0; + let current = cookie + .split("; ") + .filter((part) => !part.slice(0, part.indexOf("=")).includes("session_data")) + .join("; "); + for (let index = 0; index < 3; index += 1) { + const result = await readSession(auth, current); + assert.ok(result.userId); + current = result.cookie; + } + assert.equal(sessionReads.length, 3); +}); diff --git a/frontend/tests/local-postgres-roundtrips.test.ts b/frontend/tests/local-postgres-roundtrips.test.ts new file mode 100644 index 00000000..55dcc8c6 --- /dev/null +++ b/frontend/tests/local-postgres-roundtrips.test.ts @@ -0,0 +1,286 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import type { Pool } from "pg"; + +import { + clearLocalPostgresMetadataCacheForTests, + closeLocalPostgresDataPool, + createLocalPostgresDataClient, + installLocalPostgresPoolForTests, + setLocalPostgresMetadataClockForTests, + type LocalDatabaseRole, +} from "../src/lib/db/local-postgres-client-core.ts"; + +type QueryCall = { sql: string; params: readonly unknown[] }; + +function tag(sql: string): string { + const compact = sql.replace(/\s+/g, " ").trim().toLowerCase(); + if (compact === "begin") return "begin"; + if (compact === "commit") return "commit"; + if (compact === "rollback") return "rollback"; + if (compact.startsWith("set local role")) return "set local role"; + if (compact.includes("set_config('role'")) return "set_config(role)"; + if (compact.includes("set_config('request.jwt.claim.sub'")) return "set_config"; + if (compact.includes("information_schema.columns")) return "information_schema"; + if (compact.includes("from pg_index")) return "primary_key"; + if (compact.includes("from pg_proc")) return "pg_proc"; + return "business"; +} + +function installFakePool( + url: string, + options: { + columns?: ReadonlyArray<{ column_name: string; udt_name: string }> | null; + primaryKeys?: readonly string[] | null; + functions?: ReadonlyArray> | null; + failBusinessCode?: () => string | null; + } = {}, +): QueryCall[] { + const calls: QueryCall[] = []; + const client = { + async query(sql: string, params: readonly unknown[] = []) { + calls.push({ sql, params }); + const label = tag(sql); + if (label === "information_schema") { + const rows = options.columns === null + ? [] + : [...(options.columns ?? [{ column_name: "id", udt_name: "uuid" }])]; + return { rows, rowCount: rows.length }; + } + if (label === "primary_key") { + const names = options.primaryKeys === null ? [] : [...(options.primaryKeys ?? ["id"])]; + const rows = names.map((column_name) => ({ column_name })); + return { rows, rowCount: rows.length }; + } + if (label === "pg_proc") { + const rows = options.functions === null + ? [] + : [...(options.functions ?? [{ + proretset: false, + return_type: "text", + argument_names: ["left_name", "right_name"], + argument_types: ["text", "text"], + }])]; + return { rows, rowCount: rows.length }; + } + if (label === "business") { + const code = options.failBusinessCode?.() ?? null; + if (code) { + const error = new Error("business sql failed") as Error & { code: string }; + error.code = code; + throw error; + } + if (sql.toLowerCase().includes(" as value")) return { rows: [{ value: "ok" }], rowCount: 1 }; + } + return { rows: [], rowCount: 0 }; + }, + release() {}, + }; + const pool = { + async connect() { + return client; + }, + async end() {}, + }; + installLocalPostgresPoolForTests(url, pool as unknown as Pool); + return calls; +} + +async function withSwitches( + values: Record, + run: () => Promise, +): Promise { + const previous = new Map(); + for (const [key, value] of Object.entries(values)) { + previous.set(key, process.env[key]); + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + clearLocalPostgresMetadataCacheForTests(); + setLocalPostgresMetadataClockForTests(null); + try { + await run(); + } finally { + for (const [key, value] of previous) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + clearLocalPostgresMetadataCacheForTests(); + setLocalPostgresMetadataClockForTests(null); + } +} + +const identity = { id: "018f4e6d-7a11-7000-8000-0000000000aa", email: "person@example.com" }; + +function clientFor(url: string, role: LocalDatabaseRole | null = "authenticated") { + return createLocalPostgresDataClient(url, role === "service_role" ? null : identity, role); +} + +test("metadata cache and merged session setup stay off the baseline query sequence", async () => { + await withSwitches({ + JYOTISHA_PG_METADATA_CACHE: "off", + JYOTISHA_PG_MERGED_SESSION_SETUP: "off", + }, async () => { + const url = "postgresql://roundtrip-off@127.0.0.1:1/jyotisha"; + const calls = installFakePool(url); + const client = clientFor(url); + const first = await client.from("profiles").select("id").eq("id", identity.id); + assert.equal(first.error, null); + assert.deepEqual(calls.map((call) => tag(call.sql)), [ + "begin", + "set local role", + "set_config", + "information_schema", + "business", + "commit", + ]); + await client.from("profiles").select("id").eq("id", identity.id); + assert.equal(calls.filter((call) => tag(call.sql) === "information_schema").length, 2); + }); +}); + +test("cached metadata and merged session setup drop a repeat select and rpc to four roundtrips", async () => { + await withSwitches({ + JYOTISHA_PG_METADATA_CACHE: "on", + JYOTISHA_PG_MERGED_SESSION_SETUP: "on", + }, async () => { + const url = "postgresql://roundtrip-on@127.0.0.1:1/jyotisha"; + const calls = installFakePool(url); + const client = clientFor(url); + await client.from("profiles").select("id").eq("id", identity.id); + calls.length = 0; + await client.from("profiles").select("id").eq("id", identity.id); + assert.deepEqual(calls.map((call) => tag(call.sql)), [ + "begin", + "set_config(role)", + "business", + "commit", + ]); + + await client.rpc("sample_probe", { left_name: "a", right_name: "b" }); + calls.length = 0; + const rpc = await client.rpc("sample_probe", { left_name: "a", right_name: "b" }); + assert.equal(rpc.error, null); + assert.deepEqual(calls.map((call) => tag(call.sql)), [ + "begin", + "set_config(role)", + "business", + "commit", + ]); + + await client.from("profiles").upsert({ id: identity.id }); + calls.length = 0; + await client.from("profiles").upsert({ id: identity.id }); + assert.deepEqual(calls.map((call) => tag(call.sql)), [ + "begin", + "set_config(role)", + "business", + "commit", + ]); + }); +}); + +test("column cache is per role and function cache is per argument names", async () => { + await withSwitches({ + JYOTISHA_PG_METADATA_CACHE: "on", + JYOTISHA_PG_MERGED_SESSION_SETUP: "on", + }, async () => { + const url = "postgresql://roundtrip-keys@127.0.0.1:1/jyotisha"; + const calls = installFakePool(url); + const authenticated = clientFor(url, "authenticated"); + const service = clientFor(url, "service_role"); + await authenticated.from("profiles").select("id"); + calls.length = 0; + await service.from("profiles").select("id"); + assert.equal(calls.filter((call) => tag(call.sql) === "information_schema").length, 1); + + await authenticated.rpc("sample_probe", { left_name: "a" }); + calls.length = 0; + await authenticated.rpc("sample_probe", { right_name: "b" }); + assert.equal(calls.filter((call) => tag(call.sql) === "pg_proc").length, 1); + }); +}); + +test("undefined column, expiry, pool close, and empty metadata all miss the cache", async () => { + await withSwitches({ + JYOTISHA_PG_METADATA_CACHE: "on", + JYOTISHA_PG_MERGED_SESSION_SETUP: "off", + }, async () => { + const url = "postgresql://roundtrip-invalidate@127.0.0.1:1/jyotisha"; + let failuresLeft = 1; + const calls = installFakePool(url, { + failBusinessCode: () => { + if (failuresLeft <= 0) return null; + failuresLeft -= 1; + return "42703"; + }, + }); + const client = clientFor(url); + const failed = await client.from("profiles").select("id"); + assert.equal(failed.error?.code, "42703"); + calls.length = 0; + const retried = await client.from("profiles").select("id"); + assert.equal(retried.error, null); + assert.equal(calls.filter((call) => tag(call.sql) === "information_schema").length, 1); + }); +}); + +test("metadata older than 300 seconds is queried again", async () => { + await withSwitches({ + JYOTISHA_PG_METADATA_CACHE: "on", + JYOTISHA_PG_MERGED_SESSION_SETUP: "off", + }, async () => { + const url = "postgresql://roundtrip-ttl@127.0.0.1:1/jyotisha"; + let now = 10_000; + setLocalPostgresMetadataClockForTests(() => now); + const calls = installFakePool(url); + const client = clientFor(url); + await client.from("profiles").select("id"); + calls.length = 0; + await client.from("profiles").select("id"); + assert.equal(calls.filter((call) => tag(call.sql) === "information_schema").length, 0); + now += 300_000; + calls.length = 0; + await client.from("profiles").select("id"); + assert.equal(calls.filter((call) => tag(call.sql) === "information_schema").length, 1); + }); +}); + +test("an empty column list and a missing function are not cached", async () => { + await withSwitches({ + JYOTISHA_PG_METADATA_CACHE: "on", + JYOTISHA_PG_MERGED_SESSION_SETUP: "off", + }, async () => { + const url = "postgresql://roundtrip-empty@127.0.0.1:1/jyotisha"; + const calls = installFakePool(url, { columns: null, functions: null, primaryKeys: null }); + const client = clientFor(url); + await client.from("profiles").select("id"); + calls.length = 0; + await client.from("profiles").select("id"); + assert.equal(calls.filter((call) => tag(call.sql) === "information_schema").length, 1); + + const missing = await client.rpc("missing_probe", { left_name: "a" }); + assert.match(missing.error?.message ?? "", /not found/); + calls.length = 0; + await client.rpc("missing_probe", { left_name: "a" }); + assert.equal(calls.filter((call) => tag(call.sql) === "pg_proc").length, 1); + }); +}); + +test("closing the pool drops that url from the metadata cache", async () => { + await withSwitches({ + JYOTISHA_PG_METADATA_CACHE: "on", + JYOTISHA_PG_MERGED_SESSION_SETUP: "off", + }, async () => { + const url = "postgresql://roundtrip-close@127.0.0.1:1/jyotisha"; + const calls = installFakePool(url); + const client = clientFor(url); + await client.from("profiles").select("id"); + await closeLocalPostgresDataPool(url); + const reopenedCalls = installFakePool(url); + const reopened = clientFor(url); + await reopened.from("profiles").select("id"); + assert.equal(reopenedCalls.filter((call) => tag(call.sql) === "information_schema").length, 1); + assert.ok(calls.length > 0); + }); +}); diff --git a/frontend/tests/personal-report-worker-sleep-listener.test.ts b/frontend/tests/personal-report-worker-sleep-listener.test.ts new file mode 100644 index 00000000..55aed461 --- /dev/null +++ b/frontend/tests/personal-report-worker-sleep-listener.test.ts @@ -0,0 +1,109 @@ +import assert from "node:assert/strict"; +import { getEventListeners } from "node:events"; +import test from "node:test"; + +import { scoringPollDelay } from "../src/lib/birth-time-guided-polling.ts"; +import { runPersonalReportWorkerLoop } from "../src/lib/personal-report-worker-core.ts"; + +const idle = { + recovery: { retryingCount: 0, failedCount: 0 }, + outcome: "idle" as const, + jobId: null, +}; + +function keepAlive(): () => void { + const timer = setInterval(() => {}, 1_000); + return () => clearInterval(timer); +} + +test("idle report-worker sleeps do not leave abort listeners", async () => { + const stop = keepAlive(); + const controller = new AbortController(); + let ticks = 0; + let listenersAt200 = -1; + let abortedAt = 0; + try { + await runPersonalReportWorkerLoop( + { + async tick() { + ticks += 1; + if (ticks === 200) { + listenersAt200 = getEventListeners(controller.signal, "abort").length; + abortedAt = Date.now(); + controller.abort(); + } + return idle; + }, + }, + { pollIntervalMs: 1, signal: controller.signal }, + ); + assert.equal(listenersAt200, 0); + assert.equal(getEventListeners(controller.signal, "abort").length, 0); + assert.ok(Date.now() - abortedAt < 250); + } finally { + stop(); + } +}); + +test("thrown report-worker ticks do not leave abort listeners", async () => { + const stop = keepAlive(); + const controller = new AbortController(); + let ticks = 0; + let listenersAt200 = -1; + let abortedAt = 0; + try { + await runPersonalReportWorkerLoop( + { + async tick() { + ticks += 1; + if (ticks === 200) { + listenersAt200 = getEventListeners(controller.signal, "abort").length; + abortedAt = Date.now(); + controller.abort(); + } + throw new Error("tick failed"); + }, + }, + { pollIntervalMs: 1, signal: controller.signal }, + ); + assert.equal(listenersAt200, 0); + assert.equal(getEventListeners(controller.signal, "abort").length, 0); + assert.ok(Date.now() - abortedAt < 250); + } finally { + stop(); + } +}); + +test("scoring poll delay removes its abort listener when the timer fires", async () => { + const previousWindow = (globalThis as { window?: unknown }).window; + (globalThis as { window?: unknown }).window = { + setTimeout: globalThis.setTimeout.bind(globalThis), + clearTimeout: globalThis.clearTimeout.bind(globalThis), + }; + try { + const controller = new AbortController(); + await scoringPollDelay(0, controller.signal); + assert.equal(getEventListeners(controller.signal, "abort").length, 0); + } finally { + (globalThis as { window?: unknown }).window = previousWindow; + } +}); + +test("scoring poll delay resolves as soon as the signal aborts", async () => { + const previousWindow = (globalThis as { window?: unknown }).window; + (globalThis as { window?: unknown }).window = { + setTimeout: globalThis.setTimeout.bind(globalThis), + clearTimeout: globalThis.clearTimeout.bind(globalThis), + }; + try { + const controller = new AbortController(); + const started = Date.now(); + const pending = scoringPollDelay(0, controller.signal); + controller.abort(); + await pending; + assert.ok(Date.now() - started < 200); + assert.equal(getEventListeners(controller.signal, "abort").length, 0); + } finally { + (globalThis as { window?: unknown }).window = previousWindow; + } +}); diff --git a/frontend/tests/request-rate-limit.test.ts b/frontend/tests/request-rate-limit.test.ts index f5e9f2bd..d9c30d29 100644 --- a/frontend/tests/request-rate-limit.test.ts +++ b/frontend/tests/request-rate-limit.test.ts @@ -1,6 +1,9 @@ import assert from "node:assert/strict"; import test from "node:test"; -import { consumeRequestRateLimit } from "../src/lib/request-rate-limit.ts"; +import { + consumeRequestRateLimit, + resetRequestRateLimitMaintenanceForTests, +} from "../src/lib/request-rate-limit.ts"; test("a user is allowed up to the window limit and then must wait", () => { const store = new Map(); @@ -46,3 +49,64 @@ test("a user is allowed up to the window limit and then must wait", () => { assert.deepEqual(otherUser, { ok: true }); assert.deepEqual(afterWindow, { ok: true }); }); + +test("keys past the longest seen window are swept, and a one-hour key survives 61 seconds", () => { + resetRequestRateLimitMaintenanceForTests(); + const shortWindow = new Map(); + consumeRequestRateLimit({ + key: "old", + limit: 5, + windowMs: 60_000, + now: 0, + store: shortWindow, + }); + for (let index = 0; index < 998; index += 1) { + consumeRequestRateLimit({ + key: `pad:${index}`, + limit: 5, + windowMs: 60_000, + now: 0, + store: shortWindow, + }); + } + consumeRequestRateLimit({ + key: "trigger", + limit: 5, + windowMs: 60_000, + now: 61_000, + store: shortWindow, + }); + assert.equal(shortWindow.has("old"), false); + assert.equal(shortWindow.get("old"), undefined); + assert.equal(shortWindow.has("trigger"), true); + assert.equal(shortWindow.size, 1); + + resetRequestRateLimitMaintenanceForTests(); + const mixed = new Map(); + consumeRequestRateLimit({ + key: "synastry:person", + limit: 8, + windowMs: 60 * 60_000, + now: 0, + store: mixed, + }); + consumeRequestRateLimit({ + key: "ephemeris:person", + limit: 30, + windowMs: 60_000, + now: 0, + store: mixed, + }); + for (let index = 0; index < 998; index += 1) { + consumeRequestRateLimit({ + key: `later:${index}`, + limit: 5, + windowMs: 60_000, + now: 61_000, + store: mixed, + }); + } + assert.equal(mixed.has("synastry:person"), true); + assert.equal(mixed.has("ephemeris:person"), true); + resetRequestRateLimitMaintenanceForTests(); +}); diff --git a/frontend/tests/rescore-attempts-bound.test.ts b/frontend/tests/rescore-attempts-bound.test.ts new file mode 100644 index 00000000..7e9412c5 --- /dev/null +++ b/frontend/tests/rescore-attempts-bound.test.ts @@ -0,0 +1,31 @@ +import assert from "node:assert/strict"; +import test from "node:test"; + +import { + noteRescoreAttemptForTests, + rescoreAttemptCountForTests, + resetStaleMinuteRescoreAttemptsForTests, +} from "../src/lib/rectification-agentic/v9/block-scan-answer.ts"; + +const SIX_HOURS_MS = 6 * 60 * 60 * 1000; + +test("rescore attempts stay within 5000 entries", () => { + resetStaleMinuteRescoreAttemptsForTests(); + for (let index = 0; index < 5_001; index += 1) { + noteRescoreAttemptForTests(`case-${index}:turn:fingerprint`, 1_000); + } + assert.ok(rescoreAttemptCountForTests() <= 5_000); + resetStaleMinuteRescoreAttemptsForTests(); +}); + +test("a rescore attempt older than six hours is dropped on the next write", () => { + resetStaleMinuteRescoreAttemptsForTests(); + noteRescoreAttemptForTests("case-old:turn:fingerprint", 0); + noteRescoreAttemptForTests("case-fresh:turn:fingerprint", SIX_HOURS_MS); + assert.equal(rescoreAttemptCountForTests(), 2); + noteRescoreAttemptForTests("case-new:turn:fingerprint", SIX_HOURS_MS + 1); + assert.equal(rescoreAttemptCountForTests(), 2); + noteRescoreAttemptForTests("case-fresh:turn:fingerprint", SIX_HOURS_MS + 1); + assert.equal(rescoreAttemptCountForTests(), 2); + resetStaleMinuteRescoreAttemptsForTests(); +}); From 63509431529d212c7847371e7717660d823b40e6 Mon Sep 17 00:00:00 2001 From: jesse-ux Date: Wed, 14 Oct 2026 04:12:02 +0800 Subject: [PATCH 2/9] =?UTF-8?q?=E5=9B=9E=E7=AD=94=E7=94=9F=E6=88=90?= =?UTF-8?q?=E6=97=B6=E4=BE=A7=E6=A0=8F=E4=B8=8D=E5=86=8D=E8=B7=9F=E7=9D=80?= =?UTF-8?q?=E6=AF=8F=E4=B8=80=E5=B8=A7=E9=87=8D=E7=94=BB?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 外壳只在会话、当前项或账户这些数据变了才更新。60 帧计数从注册 60 次、提交 60 次降到 0 次，点下去仍是最新一次操作。BUG-1329。 Co-Authored-By: Grok (agent loop) --- CHANGELOG.md | 4 + docs/BUG_HISTORY.md | 15 + ...PROGRESS-home-sidebar-rerender-20261013.md | 54 ++++ docs/tasks/README.md | 2 +- .../testing/home-sidebar-rerender-20261013.md | 12 + .../home-sidebar-rerender-ab-20261013.md | 18 ++ frontend/DESIGN.md | 1 + frontend/src/components/ui/sidebar.tsx | 18 +- .../src/hooks/use-home-shell-registration.ts | 182 +++++++++-- frontend/src/hooks/use-session-management.ts | 9 +- frontend/tests/home-sidebar-rerender.test.tsx | 304 ++++++++++++++++++ .../tests/session-list-lifecycle.test.tsx | 9 +- 12 files changed, 588 insertions(+), 40 deletions(-) create mode 100644 docs/tasks/PROGRESS-home-sidebar-rerender-20261013.md create mode 100644 docs/testing/home-sidebar-rerender-20261013.md create mode 100644 docs/testing/home-sidebar-rerender-ab-20261013.md create mode 100644 frontend/tests/home-sidebar-rerender.test.tsx diff --git a/CHANGELOG.md b/CHANGELOG.md index d2056bfc..b33c7a69 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,9 @@ # 印度占星 Skill 更新日志 +## 2026-10-14 — 回答生成时侧栏不再跟着闪动 / 卡顿（待合入 staging） + +- 回答生成时侧栏不再跟着闪动 / 卡顿。点会话、改名、换人仍是当前这一下的操作。Skill 版本不变（BUG-1329）。 + ## 2026-10-14 — 婚恋评级改成证据档位，KP 判宫和大运年长对齐（待合入 staging） - 婚恋结构指数不再写成婚姻稳定、离婚、再婚或建议婚前咨询。中文报告用中文档位，英文报告用英文档位。配偶星分数按 0 到 1 显示，不再写成百分制。 diff --git a/docs/BUG_HISTORY.md b/docs/BUG_HISTORY.md index add22027..6079785c 100644 --- a/docs/BUG_HISTORY.md +++ b/docs/BUG_HISTORY.md @@ -17795,3 +17795,18 @@ - 复发自：无。 - 修复版本：未修复。 +## BUG-1329 | 流式回答期间首页侧栏每帧整栏重渲染 + +- 状态：fixed-pending-verify +- 首次发现 / 最近更新：2026-10-13 / 2026-10-14 +- 来源：issue #59。任务书 `docs/tasks/TASK-home-sidebar-rerender-20261013.md`。 +- 现象：回答逐字出来时，左侧会话列表跟着每一帧重画。会话越多越重。外观、顺序和点击结果没有变。 +- 触发条件：首页已登录，回答按帧写入。每一帧都会让首页再渲染一次。 +- 根因：BUG-971 切断了「首页订阅自己的注册」这条反馈，但注册仍然每次渲染都发布一份新对象。会话回调和可见列表每次渲染都是新的，外壳就跟着提交一次。不是回归，是当时留下的开销。 +- 修复：外壳拿到的是稳定包装，点下去仍执行首页最新一次渲染的回调。注册只在会话、当前项、账户或开关这些数据变了才发布；列表逐条对象没变就不重建行。可见列表按当前人物记忆。侧栏上下文对象也不再每次新建。没有运行时开关。 +- 验证：同一套计数测试。改前 60 帧注册 60 次、侧栏提交 60 次；改后注册 0 次、提交 0 次。换当前项、改标题、换人各发布 1 次。包装函数身份不变，调用记到的是最新一次渲染。详见 `docs/tasks/PROGRESS-home-sidebar-rerender-20261013.md`。 +- 防复发：外壳注册须按数据变化发布。任何流式会话面要有侧栏提交次数的按帧断言。`frontend/tests/home-sidebar-rerender.test.tsx` 在 60 帧内要求注册 0 次、侧栏提交 0 次。 +- 相关记录：BUG-971、BUG-725、BUG-473、BUG-744、BUG-745、BUG-746。 +- 复发自：无。 +- 修复版本：分支 `codex/home-sidebar-rerender-20261013`，尚未合入 staging。 + diff --git a/docs/tasks/PROGRESS-home-sidebar-rerender-20261013.md b/docs/tasks/PROGRESS-home-sidebar-rerender-20261013.md new file mode 100644 index 00000000..2b74e92c --- /dev/null +++ b/docs/tasks/PROGRESS-home-sidebar-rerender-20261013.md @@ -0,0 +1,54 @@ +# PROGRESS · 流式回答时侧栏不再跟着每帧重画（issue #59） + +- 分支：`codex/home-sidebar-rerender-20261013` +- 实际基线：`origin/staging` `66cd60081f37a57680274633fcb3fb942d01bb2f` +- 任务书基线写成 `77057a819`，计划评论里的 base 是 `2994588ad4d858dead4835943e5883c5aa2b03f8`。开工时远端 staging 已是上面的 `66cd6008`。A/B 就跑在这个提交上，没有另开只读工作树（本轮不能离开当前工作树）。方向没有改，不需要拍板。 +- `docs/BUG_HISTORY.md` 在这个基线上停在 BUG-1322。BUG-1323～1328、1345、1346 属于还没合入的别的分支，本单只用 BUG-1329。 +- 未改 `page.tsx`。没有运行时开关。没有启用 React Compiler。Skill 版本不变。浏览器 A/B 未跑。 + +## 做了什么 + +外壳拿到的点击函数不再每次渲染换新，点下去仍执行首页刚渲染的那一版。注册只在数据变了才发给侧栏：会话行要逐条还是同一个对象才算没变，没变就不重建行。可见会话按当前人物记忆。侧栏自己的上下文对象也不再每次新建。卸载首页时仍会清掉注册。 + +## 断言三栏 + +| 位置 | 原值 | 新值 | 原因 | +| --- | --- | --- | --- | +| `frontend/tests/session-list-lifecycle.test.tsx` 外壳「打开资料」函数 | `assert.notEqual(onOpenProfile, initialCallback)` | `assert.equal(...)`；再改一版后仍是同一个函数，调用记录从 `[1]` 加强到 `[1, 2]` | 包装函数身份稳定。调用必须落到最新一次渲染，不能只删掉「不相等」 | + +`HomeProbe` 仍是每次渲染新回调、新数组，没有改成常量。 + +## 基线失败（改代码之前，同一测试，3 次，退出码都是 1） + +失败停在 `assert.equal(duringFrames.registers, 0)`，实际是 60。60 帧里侧栏提交也是 60。换当前项、改标题、换人各注册 1 次；只改回调版本也会再注册 1 次。剔除函数后的内容摘要： + +| 档 | 60 帧注册 | 60 帧提交 | 60 帧耗时三次（毫秒） | 中位 | 内容摘要 | +| --- | --- | --- | --- | --- | --- | +| 30 条虚构会话 | 60 | 60 | 2774.964 / 2778.961 / 4202.085 | 2778.961 | `b18f05a3d83be5b41584787b7c0495f1fcec8ef7b8ecfbdbb3c083c40796c103` | +| 100 条虚构会话 | 60 | 60 | 10530.828 / 11799.253 / 12284.103 | 11799.253 | `cef0a63d6f3fe72e541f9ac882b8eaf305b6933fdd07ca3002e8cd03bb139d25` | + +更早一次探针把注册函数也做成每次渲染新建，60 帧数到 120 次注册、60 次提交。那是清理函数跟着新函数又清了一次，不是产品路径。交付用的测试和生产一样，用提供者给出的稳定注册函数，所以改前是 60 而不是 120。 + +## 改后（同一测试，3 次，退出码都是 0） + +| 档 | 60 帧注册 | 60 帧提交 | 60 帧耗时中位 | 换当前项 | 改标题 | 只改回调版本 | 换人 | 内容摘要 | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 30 | 0 | 0 | 0 | 注册 1、提交 1 | 注册 1，标题更新 | 注册 0，调用记到第 4 版 | 注册 1，列表换成另一人 | 与改前相同 | +| 100 | 0 | 0 | 0 | 注册 1、提交 1 | 注册 1，标题更新 | 注册 0，调用记到第 4 版 | 注册 1，列表换成另一人 | 与改前相同 | + +三次摘要都与上表相同。门槛达到：60 帧内注册 0、侧栏提交 0，内容摘要逐字节一致。 + +## 自测 + +| 检查 | 命令 | 结果 | +| --- | --- | --- | +| 类型 | `node frontend/node_modules/typescript/bin/tsc --noEmit -p frontend` | 退出码 0 | +| 规范 | `npm run lint --prefix frontend` | 0 error，126 条原有 warning。本单文件没有新增 warning，没有 `eslint-disable` | +| 定向测试 | `tsx --test` 下表 10 个文件 | tests 94，pass 94，fail 0，cancelled 0 | +| 数据库 | 未跑 | 本单不碰数据库 | +| 浏览器 | 未跑 | 浏览器 A/B 未跑。步骤在 `docs/testing/home-sidebar-rerender-20261013.md` | +| 隐私标记 | `.venv\Scripts\python.exe -m pytest tests/test_repo_privacy_markers.py -q` | 通过 | + +定向文件：`home-sidebar-rerender`、`session-list-lifecycle`、`session-list-provider`、`sidebar-contract`、`sidebar-state`、`secondary-page-entry`、`account-dialog-inert-20260918`、`home-shell-growth-contract`、`home-streaming-render-split`、`chat-profile-picker`。 + +全量 `npm test`、`next build`、Python 快速门未跑，留给验收。 diff --git a/docs/tasks/README.md b/docs/tasks/README.md index 58b326a6..a91088de 100644 --- a/docs/tasks/README.md +++ b/docs/tasks/README.md @@ -469,7 +469,7 @@ | --- | --- | --- | --- | --- | | `TASK-rect-scoring-perf-20261013.md` | — | 校正打分提速：引导窗口平方扫描、候选不变量重复算、响应带无读取点字段、候选数上限 241（BUG-1323～1325、1345） | 待领取 | `codex/rect-scoring-perf-20261013` | | `TASK-node-runtime-resources-20261013.md` | — | 报告 worker abort 监听泄漏、本地 PG 适配层往返与元数据缓存、无上限内存表、用户端会话读缓存（BUG-1326～1328、1346，PG 与缓存部分 A/B） | 待领取 | `codex/node-runtime-resources-20261013` | -| `TASK-home-sidebar-rerender-20261013.md` | — | 流式回答时侧栏每帧整栏重渲染（BUG-1329，A/B） | 待领取 | `codex/home-sidebar-rerender-20261013` | +| `TASK-home-sidebar-rerender-20261013.md` | `PROGRESS-home-sidebar-rerender-20261013.md` | 流式回答时侧栏每帧整栏重渲染（BUG-1329，A/B） | 待验收 | `codex/home-sidebar-rerender-20261013` | | `TASK-api-disk-cache-retention-20261013.md` | — | API 磁盘缓存只进不出、VedAstro 审计文件留存出生资料、失败快照久缓存、缓存键规范化（BUG-1330～1332，A/B） | 待领取 | `codex/api-disk-cache-retention-20261013` | | `TASK-report-reference-perf-20261013.md` | — | 长报告出稿提速：术语替换命中预检、哈希单次遍历、同盘复用（BUG-1335～1336，A/B） | 待领取 | `codex/report-reference-perf-20261013` | | `TASK-consult-prep-parallel-20261013.md` | — | 普通对话发问前准备步骤并行、模型版本短缓存、多领域并发开关（BUG-1337～1339，A/B） | 待领取 | `codex/consult-prep-parallel-20261013` | diff --git a/docs/testing/home-sidebar-rerender-20261013.md b/docs/testing/home-sidebar-rerender-20261013.md new file mode 100644 index 00000000..0f24a94f --- /dev/null +++ b/docs/testing/home-sidebar-rerender-20261013.md @@ -0,0 +1,12 @@ +# 真机走查 · 回答生成时侧栏不再跟着闪（BUG-1329，issue #59） + +浏览器 A/B 未跑。部署到 staging 后，在 `https://staging.jyotisha.chat` 登录一个没有真实会话内容需要露出的账号，按顺序看： + +1. 电脑浏览器打开首页。左侧会话列表的样子、顺序、分组和以前一样。 +2. 发一个普通问题，等回答一个字一个字出来。这段时间左侧列表不跟着闪、不跟着整栏重画。回答本身仍是逐字出现。 +3. 回答过程中点另一条会话，能打开那一条。点当前这条的菜单，改名、置顶、分享、删除仍可用，改的是刚刚点的那一条。 +4. 右上角账户菜单仍能打开资料、通用、账单和退出。 +5. 换一个人物后，左侧列表换成那个人的会话，不串到别人的。 +6. 手机宽度再看一遍第 2 步。侧栏抽屉的开关和以前一样。 + +任何一步不对，在 issue #59 里写是哪一步、哪台设备，并把标签改回 `agent/待规划`。不要贴出真实会话内容。 diff --git a/docs/testing/home-sidebar-rerender-ab-20261013.md b/docs/testing/home-sidebar-rerender-ab-20261013.md new file mode 100644 index 00000000..0fc3e5f1 --- /dev/null +++ b/docs/testing/home-sidebar-rerender-ab-20261013.md @@ -0,0 +1,18 @@ +# A/B · 回答生成时侧栏重画（BUG-1329，issue #59） + +同一测试 `frontend/tests/home-sidebar-rerender.test.tsx`，同一输入。A 是改代码前的 `66cd60081f37a57680274633fcb3fb942d01bb2f`（任务书写的 `77057a819` 已被后续提交盖过）。B 是分支 `codex/home-sidebar-rerender-20261013`。各跑 3 次。会话都是虚构的「Synthetic session N」。没有真实会话内容。 + +度量是 60 帧里的注册次数、侧栏提交次数、这 60 帧的 Profiler 耗时合计。内容用剔除函数字段后的 SHA-256 比较。 + +| 档 | A 注册 / 提交 / 耗时中位 | B 注册 / 提交 / 耗时中位 | 内容 | +| --- | --- | --- | --- | +| 30 条 | 60 / 60 / 2778.961 毫秒 | 0 / 0 / 0 | 两边都是 `b18f05a3d83be5b41584787b7c0495f1fcec8ef7b8ecfbdbb3c083c40796c103` | +| 100 条 | 60 / 60 / 11799.253 毫秒 | 0 / 0 / 0 | 两边都是 `cef0a63d6f3fe72e541f9ac882b8eaf305b6933fdd07ca3002e8cd03bb139d25` | + +A 的三次 30 条耗时：2774.964、2778.961、4202.085。100 条：10530.828、11799.253、12284.103。B 的三次都是 0。 + +正确性：换当前项后 B 注册 1 次，外壳看到新的当前项。改一行标题后标题更新。换成另一人的会话后列表换掉。只改回调版本不再注册，点「打开会话」和「改名」记到的都是第 4 版。A 在只改回调版本时还会再注册 1 次；B 不注册，但调用结果相同。 + +判定：B 在 60 帧内注册 0、侧栏提交 0，内容摘要一致，`session-list-lifecycle` 全绿。可以交付。 + +浏览器 A/B 未跑。 diff --git a/frontend/DESIGN.md b/frontend/DESIGN.md index bf73e26d..f64355ab 100644 --- a/frontend/DESIGN.md +++ b/frontend/DESIGN.md @@ -551,6 +551,7 @@ one skeleton, none of them carrying the sidebar. `children` through unchanged. Registering fresh controls/rows must not render their producer again: that feedback loop can starve client navigation. Business changes still publish current callbacks; Home unmount clears them. + 流式期间侧栏不随回答帧重渲染（BUG-1329）。 - **One sidebar component, two modes.** `AppSidebar` is the only sidebar in the product. `/` passes `controls`; the secondary layout does not, and without it the same markup renders read-only. There used to be a second component for diff --git a/frontend/src/components/ui/sidebar.tsx b/frontend/src/components/ui/sidebar.tsx index 35a905ab..541d14fb 100644 --- a/frontend/src/components/ui/sidebar.tsx +++ b/frontend/src/components/ui/sidebar.tsx @@ -8,6 +8,7 @@ import { useCallback, useContext, useEffect, + useMemo, useRef, useState, type ComponentProps, @@ -169,7 +170,13 @@ export function SidebarProvider({ else setOpen(!open); }, [isMobile, open, openMobile, setOpen, setOpenMobile]); - const context: SidebarProviderContextValue = { + const setInsetTrigger = useCallback((element: HTMLButtonElement | null) => { + insetTriggerRef.current = element; + }, []); + const setSidebarSurface = useCallback((element: HTMLElement | null) => { + sidebarSurfaceRef.current = element; + }, []); + const context = useMemo(() => ({ state: open ? "expanded" : "collapsed", open, setOpen, @@ -179,9 +186,12 @@ export function SidebarProvider({ ready, isMobile, toggleSidebar, - setInsetTrigger: (element) => { insetTriggerRef.current = element; }, - setSidebarSurface: (element) => { sidebarSurfaceRef.current = element; }, - }; + setInsetTrigger, + setSidebarSurface, + }), [ + open, openMobile, viewport, ready, isMobile, setOpen, setOpenMobile, toggleSidebar, + setInsetTrigger, setSidebarSurface, + ]); return ( diff --git a/frontend/src/hooks/use-home-shell-registration.ts b/frontend/src/hooks/use-home-shell-registration.ts index 9b84fdf8..1522cc64 100644 --- a/frontend/src/hooks/use-home-shell-registration.ts +++ b/frontend/src/hooks/use-home-shell-registration.ts @@ -1,6 +1,6 @@ "use client"; -import { useEffect, type Dispatch, type RefObject, type SetStateAction } from "react"; +import { useEffect, useMemo, useRef, type Dispatch, type RefObject, type SetStateAction } from "react"; import type { ChatSession } from "@/lib/home-types"; import type { Account } from "@/lib/home-types"; @@ -43,6 +43,68 @@ type HomeShellRegistrationParams = { visibleSessions: readonly ChatSession[]; }; +type LatestShellActions = { + sessions: readonly ChatSession[]; + loadMoreSessions: () => void; + renameSession: (session: ChatSession) => void | Promise; + selectSession: (sessionId: string) => void; + shareSession: (session: ChatSession) => void | Promise; + startNewChat: () => void | Promise; + togglePinnedSession: (sessionId: string) => void; + openAccountDialog: (dialog: AccountDialogName, options?: { source?: string }) => void; + setSessionMenuId: Dispatch>; + setAccountMenuOpen: Dispatch>; + setPendingSessionDeletion: Dispatch>; +}; + +type PublishedShellInputs = { + hydrated: boolean; + account: Account; + profileName: string; + accountMenuOpen: boolean; + modalOpen: boolean; + activeSessionId: string | undefined; + rectificationOpeningSessionId: string | null; + rectificationSurfaceOpen: boolean; + creatingSession: boolean; + pendingSessionId: string | null; + cancellationPending: boolean; + modelCatalog: PublicLanguageModelCatalog | null; + sessionMenuId: string | null; + sessionsCursor: string | null; + accountTrigger: RefObject; + chartLibrary: readonly ChartLibraryRecord[]; + visibleSessions: readonly ChatSession[]; +}; + +function sameVisibleSessions(left: readonly ChatSession[], right: readonly ChatSession[]) { + if (left.length !== right.length) return false; + for (let index = 0; index < left.length; index += 1) { + if (left[index] !== right[index]) return false; + } + return true; +} + +function samePublishedInputs(previous: PublishedShellInputs, next: PublishedShellInputs) { + return previous.hydrated === next.hydrated + && previous.account === next.account + && previous.profileName === next.profileName + && previous.accountMenuOpen === next.accountMenuOpen + && previous.modalOpen === next.modalOpen + && previous.activeSessionId === next.activeSessionId + && previous.rectificationOpeningSessionId === next.rectificationOpeningSessionId + && previous.rectificationSurfaceOpen === next.rectificationSurfaceOpen + && previous.creatingSession === next.creatingSession + && previous.pendingSessionId === next.pendingSessionId + && previous.cancellationPending === next.cancellationPending + && previous.modelCatalog === next.modelCatalog + && previous.sessionMenuId === next.sessionMenuId + && previous.sessionsCursor === next.sessionsCursor + && previous.accountTrigger === next.accountTrigger + && previous.chartLibrary === next.chartLibrary + && sameVisibleSessions(previous.visibleSessions, next.visibleSessions); +} + export function useHomeShellRegistration(params: HomeShellRegistrationParams) { const { account, @@ -75,6 +137,27 @@ export function useHomeShellRegistration(params: HomeShellRegistrationParams) { togglePinnedSession, visibleSessions, } = params; + const profileName = profile.name; + const latestRef = useRef(null); + const publishedRef = useRef(null); + + useEffect(() => { + // No dependency list: this has to land after every render and before the + // publish effect, so a later click runs this render's callbacks (BUG-725). + latestRef.current = { + sessions, + loadMoreSessions, + renameSession, + selectSession, + shareSession, + startNewChat, + togglePinnedSession, + openAccountDialog, + setSessionMenuId, + setAccountMenuOpen, + setPendingSessionDeletion, + }; + }); useEffect(() => { // The inline dead-screen script stands down only after the app has revealed. @@ -84,12 +167,59 @@ export function useHomeShellRegistration(params: HomeShellRegistrationParams) { document.documentElement.dataset.hydrated = "1"; }, [hydrated]); + const stableControls = useMemo(() => ({ + onLoadMore: () => { latestRef.current?.loadMoreSessions(); }, + onMenuSessionChange: (sessionId: string | null) => { latestRef.current?.setSessionMenuId(sessionId); }, + onTogglePinned: (sessionId: string) => { latestRef.current?.togglePinnedSession(sessionId); }, + onRename: (sessionId: string) => { + const session = latestRef.current?.sessions.find((candidate) => candidate.id === sessionId); + if (session) void latestRef.current?.renameSession(session); + }, + onShare: (sessionId: string) => { + const session = latestRef.current?.sessions.find((candidate) => candidate.id === sessionId); + if (session) void latestRef.current?.shareSession(session); + }, + onDelete: (sessionId: string) => { + const session = latestRef.current?.sessions.find((candidate) => candidate.id === sessionId); + if (session) latestRef.current?.setPendingSessionDeletion(session); + }, + onAccountMenuOpenChange: (open: boolean) => { latestRef.current?.setAccountMenuOpen(open); }, + onNewChat: () => { void latestRef.current?.startNewChat(); }, + onSelectSession: (sessionId: string) => { latestRef.current?.selectSession(sessionId); }, + onOpenProfile: () => { latestRef.current?.openAccountDialog("profile"); }, + onOpenGeneral: () => { latestRef.current?.openAccountDialog("general"); }, + onOpenBilling: () => { latestRef.current?.openAccountDialog("billing", { source: "account-menu" }); }, + onOpenLogout: () => { latestRef.current?.openAccountDialog("logout"); }, + }), []); + useEffect(() => { if (!hydrated || !account) { + publishedRef.current = null; registerShellControls(null); return; } - const name = profile.name.trim(); + const next: PublishedShellInputs = { + hydrated, + account, + profileName, + accountMenuOpen, + modalOpen, + activeSessionId, + rectificationOpeningSessionId, + rectificationSurfaceOpen, + creatingSession, + pendingSessionId, + cancellationPending, + modelCatalog, + sessionMenuId, + sessionsCursor, + accountTrigger, + chartLibrary, + visibleSessions, + }; + if (publishedRef.current && samePublishedInputs(publishedRef.current, next)) return; + publishedRef.current = next; + const name = profileName.trim(); const email = account.user.email || "尚未读取邮箱"; registerShellControls({ escapeBlocked: accountMenuOpen || modalOpen, @@ -112,42 +242,34 @@ export function useHomeShellRegistration(params: HomeShellRegistrationParams) { accountTriggerRef: accountTrigger, sessionControls: { hasMore: Boolean(sessionsCursor), - onLoadMore: loadMoreSessions, + onLoadMore: stableControls.onLoadMore, menuSessionId: sessionMenuId, disabled: Boolean(pendingSessionId) || cancellationPending, - onMenuSessionChange: setSessionMenuId, - onTogglePinned: togglePinnedSession, - onRename: (sessionId) => { - const session = sessions.find((candidate) => candidate.id === sessionId); - if (session) void renameSession(session); - }, - onShare: (sessionId) => { - const session = sessions.find((candidate) => candidate.id === sessionId); - if (session) void shareSession(session); - }, - onDelete: (sessionId) => { - const session = sessions.find((candidate) => candidate.id === sessionId); - if (session) setPendingSessionDeletion(session); - }, + onMenuSessionChange: stableControls.onMenuSessionChange, + onTogglePinned: stableControls.onTogglePinned, + onRename: stableControls.onRename, + onShare: stableControls.onShare, + onDelete: stableControls.onDelete, }, - onAccountMenuOpenChange: setAccountMenuOpen, - onNewChat: () => { void startNewChat(); }, - onSelectSession: selectSession, - onOpenProfile: () => openAccountDialog("profile"), - onOpenGeneral: () => openAccountDialog("general"), - onOpenBilling: () => openAccountDialog("billing", { source: "account-menu" }), - onOpenLogout: () => openAccountDialog("logout"), + onAccountMenuOpenChange: stableControls.onAccountMenuOpenChange, + onNewChat: stableControls.onNewChat, + onSelectSession: stableControls.onSelectSession, + onOpenProfile: stableControls.onOpenProfile, + onOpenGeneral: stableControls.onOpenGeneral, + onOpenBilling: stableControls.onOpenBilling, + onOpenLogout: stableControls.onOpenLogout, }, }); }, [ account, accountMenuOpen, accountTrigger, activeSessionId, cancellationPending, - chartLibrary, creatingSession, hydrated, loadMoreSessions, modalOpen, modelCatalog, - openAccountDialog, pendingSessionId, profile.name, + chartLibrary, creatingSession, hydrated, modalOpen, modelCatalog, + pendingSessionId, profileName, rectificationOpeningSessionId, rectificationSurfaceOpen, - registerShellControls, renameSession, selectSession, sessionMenuId, sessions, - sessionsCursor, setAccountMenuOpen, setPendingSessionDeletion, setSessionMenuId, - shareSession, startNewChat, togglePinnedSession, visibleSessions, + registerShellControls, sessionMenuId, sessionsCursor, stableControls, visibleSessions, ]); - useEffect(() => () => registerShellControls(null), [registerShellControls]); + useEffect(() => () => { + publishedRef.current = null; + registerShellControls(null); + }, [registerShellControls]); } diff --git a/frontend/src/hooks/use-session-management.ts b/frontend/src/hooks/use-session-management.ts index 2e9b63c9..bd3c0330 100644 --- a/frontend/src/hooks/use-session-management.ts +++ b/frontend/src/hooks/use-session-management.ts @@ -1,7 +1,7 @@ "use client"; import type { Dispatch, MutableRefObject, SetStateAction } from "react"; -import { useEffect, useRef, useState } from "react"; +import { useEffect, useMemo, useRef, useState } from "react"; import { showChatNotice as setComposerNotice } from "@/lib/chat-notice"; import { writeChatSession } from "@/lib/chat-session-write-contract"; @@ -12,7 +12,7 @@ import { writeSessionUrl, } from "@/lib/chat-session-url"; import { sessionMatchesSubject } from "@/lib/chat-profile-picker-model"; -import { currentSubjectCatalogReady, readCurrentSubjectId, readCurrentSubjectScope, subscribeCurrentSubject, subscribeSubjectCatalog } from "@/lib/current-subject"; +import { currentSubjectCatalogReady, readCurrentSubjectId, readCurrentSubjectScope, subscribeCurrentSubject, subscribeSubjectCatalog, useCurrentSubject } from "@/lib/current-subject"; import { mergeSubjectSessionRows } from "@/lib/subject-session-merge"; import { consultationReportMarkdown } from "@/lib/consultation-report-export"; import { fallbackSessionId } from "@/lib/rectification-session-composer-guard"; @@ -181,8 +181,9 @@ export function useSessionManagement(params: SessionManagementParams) { if (!isUnsavedEmptyConsultation(session)) cloudCreatedIds.current.add(session.id); } - const visibleSessions = sortSessions(sessions.filter((session) => isListedSidebarSession(session) - && sessionMatchesSubject(session, readCurrentSubjectId()))); + const subjectId = useCurrentSubject(); + const visibleSessions = useMemo(() => sortSessions(sessions.filter((session) => isListedSidebarSession(session) + && sessionMatchesSubject(session, subjectId))), [sessions, subjectId]); function updateSession(sessionId: string, change: (session: ChatSession) => ChatSession) { setSessions((current) => current.map((session) => (session.id === sessionId ? change(session) : session))); diff --git a/frontend/tests/home-sidebar-rerender.test.tsx b/frontend/tests/home-sidebar-rerender.test.tsx new file mode 100644 index 00000000..2b8c6284 --- /dev/null +++ b/frontend/tests/home-sidebar-rerender.test.tsx @@ -0,0 +1,304 @@ +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import test from "node:test"; +import { Profiler, useCallback, useEffect, useState, type ProfilerOnRenderCallback, type ReactNode } from "react"; + +import { AppSidebar } from "../src/components/app-sidebar.tsx"; +import { SidebarInset, SidebarProvider } from "../src/components/ui/sidebar.tsx"; +import { useHomeShellRegistration } from "../src/hooks/use-home-shell-registration.ts"; +import { resetSubjectCatalogForTests } from "../src/lib/current-subject.ts"; +import { emptyProfile, type Account, type ChatSession } from "../src/lib/home-types.ts"; +import { + SessionListProvider, + sessionListSidebarModel, + useSessionList, + useShellRegistration, + type ShellRegistration, +} from "../src/lib/session-list-context.tsx"; +import { createClientLifecycleHarness } from "./react-client-lifecycle-test-support.ts"; + +const account: Account = { + user: { id: "11111111-1111-4111-8111-111111111111", email: null }, + profile: { ...emptyProfile, name: "Synthetic" }, + avatar: null, credits: 10, isAdmin: false, adminUrl: null, + rectificationPriceCredits: 0, activeSubscription: null, + hasConfirmedBirthTime: false, hasUsableBirthTime: false, +}; +const trigger = { current: null }; +const charts: never[] = []; + +function syntheticSession(index: number, title = `Synthetic session ${index}`): ChatSession { + return { + id: `session-${index}`, + title, + theme: "general", + modelId: "test", + messages: [], + createdAt: 1, + updatedAt: 1_700_000_000_000 + index, + sessionType: "consultation", + rectificationCaseId: null, + chartProfileId: null, + chartProfileName: null, + chartProfileRole: null, + pinned: false, + archivedAt: null, + messagesHydrated: false, + }; +} + +function registrationData(registration: ShellRegistration | null) { + if (!registration) return null; + return { + activeSessionId: registration.activeSessionId, + openingSessionId: registration.openingSessionId, + escapeBlocked: registration.escapeBlocked, + insetClassName: registration.insetClassName, + insetInert: registration.insetInert, + account: registration.sidebarAccount ?? null, + sessions: registration.sidebarSessions ?? [], + newChatDisabled: registration.controls.newChatDisabled, + creatingSession: registration.controls.creatingSession, + accountMenuOpen: registration.controls.accountMenuOpen, + hasMore: registration.controls.sessionControls.hasMore ?? false, + menuSessionId: registration.controls.sessionControls.menuSessionId, + disabled: registration.controls.sessionControls.disabled, + }; +} + +function payloadSha(data: ReturnType) { + return createHash("sha256").update(JSON.stringify(data)).digest("hex"); +} + +async function runProbe(sessionCount: number) { + const harness = createClientLifecycleHarness(); + resetSubjectCatalogForTests(); + const originalStorage = Object.getOwnPropertyDescriptor(globalThis, "localStorage"); + const storageValues = new Map(); + Object.defineProperty(globalThis, "localStorage", { + configurable: true, + value: { + getItem: (key: string) => storageValues.get(key) ?? null, + setItem: (key: string, value: string) => { storageValues.set(key, String(value)); }, + removeItem: (key: string) => { storageValues.delete(key); }, + clear: () => { storageValues.clear(); }, + key: (index: number) => Array.from(storageValues.keys())[index] ?? null, + get length() { return storageValues.size; }, + } satisfies Storage, + }); + const originalFetch = globalThis.fetch; + const originalIntersection = Object.getOwnPropertyDescriptor(globalThis, "IntersectionObserver"); + class FakeIntersectionObserver { + observe() {} + disconnect() {} + unobserve() {} + } + Object.defineProperty(globalThis, "IntersectionObserver", { + configurable: true, + writable: true, + value: FakeIntersectionObserver, + }); + globalThis.fetch = (async (input: unknown) => { + const url = String(input); + const payload = url === "/api/account" ? account + : url === "/api/chart-profiles" ? { profiles: [] } + : { sessions: [], nextCursor: null }; + return new Response(JSON.stringify(payload), { status: 200 }); + }) as typeof fetch; + + const metrics = { registers: 0, commits: 0, actualDuration: 0, recording: false }; + const selectRevisions: number[] = []; + const renameRevisions: number[] = []; + const driver: { + setFrame: (frame: number) => void; + setActive: (id: string) => void; + retitle: () => void; + switchPerson: () => void; + setRevision: (revision: number) => void; + } = { + setFrame: () => {}, + setActive: () => {}, + retitle: () => {}, + switchPerson: () => {}, + setRevision: () => {}, + }; + let latestShell: ShellRegistration | null = null; + const currentShell = (): ShellRegistration | null => latestShell; + const initialSessions = Array.from({ length: sessionCount }, (_, index) => syntheticSession(index)); + const otherPerson = [syntheticSession(sessionCount + 1, "Other person session")]; + + function HomeProbe() { + const list = useSessionList(); + const [frame, setFrame] = useState(0); + const [activeSessionId, setActive] = useState(initialSessions[0]?.id ?? null); + const [sessions, setSessions] = useState(initialSessions); + const [revision, setRevision] = useState(0); + // Production passes SessionListProvider's stable registrar. A fresh wrapper + // every render would rerun the unmount cleanup and publish twice per frame. + const registerShellControls = useCallback((registration: ShellRegistration | null) => { + if (metrics.recording) metrics.registers += 1; + list.registerShellControls(registration); + }, [list]); + useHomeShellRegistration({ + account: list.account, + accountMenuOpen: false, + accountTrigger: trigger, + activeSessionId: activeSessionId ?? undefined, + cancellationPending: false, + chartLibrary: charts, + creatingSession: false, + hydrated: list.settled, + loadMoreSessions: () => {}, + modalOpen: false, + modelCatalog: null, + openAccountDialog: () => {}, + pendingSessionId: null, + profile: { ...emptyProfile, name: "Synthetic" }, + rectificationOpeningSessionId: null, + rectificationSurfaceOpen: false, + registerShellControls, + renameSession: () => { renameRevisions.push(revision); }, + selectSession: () => { selectRevisions.push(revision); }, + sessionMenuId: null, + sessions, + sessionsCursor: null, + setAccountMenuOpen: () => {}, + setPendingSessionDeletion: () => {}, + setSessionMenuId: () => {}, + shareSession: () => {}, + startNewChat: () => {}, + togglePinnedSession: () => {}, + // New array every render. Element identity stays until a real edit. + visibleSessions: sessions.filter(() => true), + }); + void frame; + useEffect(() => { + driver.setFrame = setFrame; + driver.setActive = setActive; + driver.retitle = () => setSessions((current) => current.map((session, index) => ( + index === 0 ? { ...session, title: "Renamed synthetic" } : session + ))); + driver.switchPerson = () => setSessions(otherPerson); + driver.setRevision = setRevision; + }, []); + return null; + } + + const onSidebarRender: ProfilerOnRenderCallback = (_id, phase, actualDuration) => { + if (!metrics.recording) return; + if (phase !== "mount" && phase !== "update") return; + metrics.commits += 1; + metrics.actualDuration += actualDuration; + }; + + function Shell({ children }: { children: ReactNode }) { + const list = useSessionList(); + const registration = useShellRegistration(); + const model = sessionListSidebarModel(list, registration); + useEffect(() => { latestShell = registration; }); + return ( + + + + + {children} + + ); + } + + const restore = () => { + globalThis.fetch = originalFetch; + resetSubjectCatalogForTests(); + if (originalStorage) Object.defineProperty(globalThis, "localStorage", originalStorage); + else Reflect.deleteProperty(globalThis, "localStorage"); + if (originalIntersection) Object.defineProperty(globalThis, "IntersectionObserver", originalIntersection); + else Reflect.deleteProperty(globalThis, "IntersectionObserver"); + }; + + try { + await harness.render( + + + , + ); + await harness.idle(); + assert.equal(harness.errors.length, 0); + assert.ok(latestShell, "hydrated home registers once before the frame run"); + metrics.recording = true; + metrics.registers = 0; + metrics.commits = 0; + metrics.actualDuration = 0; + for (let frame = 1; frame <= 60; frame += 1) { + await harness.update(() => driver.setFrame(frame)); + } + const duringFrames = { + registers: metrics.registers, + commits: metrics.commits, + actualDuration: Number(metrics.actualDuration.toFixed(3)), + }; + metrics.registers = 0; + metrics.commits = 0; + await harness.update(() => driver.setActive("session-active")); + const afterActiveRegisters = metrics.registers; + const afterActiveCommits = metrics.commits; + const afterActive = registrationData(currentShell()); + metrics.registers = 0; + await harness.update(() => driver.retitle()); + const afterTitleRegisters = metrics.registers; + const renamedTitle = currentShell()?.sidebarSessions?.[0]?.title ?? null; + metrics.registers = 0; + await harness.update(() => driver.setRevision(4)); + const afterRevisionRegisters = metrics.registers; + currentShell()?.controls.onSelectSession("session-0"); + currentShell()?.controls.sessionControls.onRename("session-0"); + metrics.registers = 0; + await harness.update(() => driver.switchPerson()); + const afterSwitchRegisters = metrics.registers; + const otherTitle = currentShell()?.sidebarSessions?.[0]?.title ?? null; + const report = { + sessions: sessionCount, + duringFrames, + afterActiveRegisters, + afterActiveCommits, + afterActiveSha: payloadSha(afterActive), + afterActive, + afterTitleRegisters, + renamedTitle, + afterRevisionRegisters, + selectRevisions, + renameRevisions, + afterSwitchRegisters, + otherTitle, + }; + console.log(`SIDEBAR_AB ${JSON.stringify(report)}`); + assert.equal(duringFrames.registers, 0); + assert.equal(duringFrames.commits, 0); + assert.equal(afterActiveRegisters, 1); + assert.equal(afterActive?.activeSessionId, "session-active"); + assert.equal(afterTitleRegisters, 1); + assert.equal(renamedTitle, "Renamed synthetic"); + assert.equal(afterRevisionRegisters, 0); + assert.deepEqual(selectRevisions, [4]); + assert.deepEqual(renameRevisions, [4]); + assert.equal(afterSwitchRegisters, 1); + assert.equal(otherTitle, "Other person session"); + return report; + } finally { + await harness.close(); + restore(); + } +} + +for (const sessionCount of [30, 100]) { + test(`streaming frames do not republish the sidebar (${sessionCount} sessions)`, async () => { + await runProbe(sessionCount); + }); +} diff --git a/frontend/tests/session-list-lifecycle.test.tsx b/frontend/tests/session-list-lifecycle.test.tsx index 1b797c37..41f14f45 100644 --- a/frontend/tests/session-list-lifecycle.test.tsx +++ b/frontend/tests/session-list-lifecycle.test.tsx @@ -182,9 +182,16 @@ test("shell receives current callbacks, account and session state without freezi assert.equal(app.shell?.activeSessionId, "revision-1"); assert.equal(app.shell?.insetInert, true); assert.equal(app.shell?.escapeBlocked, true); - assert.notEqual(app.shell?.controls.onOpenProfile, initialCallback); + // 原值：assert.notEqual(onOpenProfile, initialCallback)，每次注册都是新函数。 + // 新值：包装函数身份不变；再改一版后仍是同一个函数，调用记到的是当前 revision。 + // 原因：外壳拿稳定包装，点下去执行 Home 最新一次渲染的回调（BUG-971 / BUG-1329）。 + assert.equal(app.shell?.controls.onOpenProfile, initialCallback); app.shell?.controls.onOpenProfile(); assert.deepEqual(app.callbackRevisions, [1]); + await app.revise(2); + assert.equal(app.shell?.controls.onOpenProfile, initialCallback); + app.shell?.controls.onOpenProfile(); + assert.deepEqual(app.callbackRevisions, [1, 2]); await app.harness.update(() => { app.list.setAccount({ ...account, credits: 27 }); app.list.setSessions([row]); From 81e2e1cfed51af35141f2a4d5e088bd30ee8e32f Mon Sep 17 00:00:00 2001 From: jesse-ux Date: Wed, 14 Oct 2026 04:53:30 +0800 Subject: [PATCH 3/9] =?UTF-8?q?=E6=9C=8D=E5=8A=A1=E5=99=A8=E7=BC=93?= =?UTF-8?q?=E5=AD=98=E6=8C=89=E6=9C=9F=E9=99=90=E6=B8=85=E7=90=86=EF=BC=8C?= =?UTF-8?q?=E5=AE=A1=E8=AE=A1=E5=BF=AB=E7=85=A7=E9=BB=98=E8=AE=A4=E4=B8=8D?= =?UTF-8?q?=E5=86=8D=E8=90=BD=E5=9C=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 排盘缓存、官方快照、负缓存、网关任务和分盘段过期后会删掉。同一张盘的数字写法合成一把缓存键。失败快照只用短期限。自测留存、异步任务、增长合同、安全缓存和隐私标记；一万个文件的清理慢于 2 秒，命中率提升不到 2.5 倍，都按实测写入记录。 Co-Authored-By: Grok (agent loop) --- .env.cn.example | 2 + CHANGELOG.md | 5 + docs/BUG_HISTORY.md | 45 ++ ...GRESS-api-disk-cache-retention-20261013.md | 51 ++ docs/tasks/README.md | 2 +- .../api-disk-cache-retention-ab-20261013.md | 66 ++ scripts/api_scratch_retention.py | 609 ++++++++++++++++++ scripts/jyotish_api_server.py | 70 +- scripts/research/api_chart_cache_byte_diff.py | 28 + scripts/research/api_disk_cache_ab.py | 442 +++++++++++++ scripts/vedastro_gateway.py | 27 +- scripts/vedastro_service_adapter.py | 77 ++- tests/test_api_scratch_retention.py | 314 +++++++++ .../test_vedastro_service_adapter_executor.py | 4 + 14 files changed, 1677 insertions(+), 65 deletions(-) create mode 100644 docs/tasks/PROGRESS-api-disk-cache-retention-20261013.md create mode 100644 docs/testing/api-disk-cache-retention-ab-20261013.md create mode 100644 scripts/api_scratch_retention.py create mode 100644 scripts/research/api_chart_cache_byte_diff.py create mode 100644 scripts/research/api_disk_cache_ab.py create mode 100644 tests/test_api_scratch_retention.py diff --git a/.env.cn.example b/.env.cn.example index 68632a8c..f379f1a5 100644 --- a/.env.cn.example +++ b/.env.cn.example @@ -10,6 +10,8 @@ VEDASTRO_SELF_HOST_ENDPOINT=https://your-domain.example.com/vedastro # VEDASTRO_ENABLE_NETWORK=0 # Cache / TTL / free-tier queue. These keep ordinary users from calling upstream directly. +# 失败结果另有短 TTL（VEDASTRO_OFFICIAL_FULL_SNAPSHOT_NEGATIVE_TTL_SECONDS，默认 300 秒，且不超过成功 TTL）。 +# 这些缓存含出生资料，7 天为上限。 VEDASTRO_CACHE_TTL_SECONDS=604800 VEDASTRO_OFFICIAL_FULL_SNAPSHOT_CACHE_TTL_SECONDS=604800 VEDASTRO_GATEWAY_QUEUE_ENABLED=1 diff --git a/CHANGELOG.md b/CHANGELOG.md index d2056bfc..4eedc7ab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,10 @@ # 印度占星 Skill 更新日志 +## 2026-10-14 — 服务器不再长期保留出生资料副本 + +- 排盘缓存、官方快照和相关临时文件会按期限删掉，不再一直堆在服务器上。审计用的快照文件默认不再落地。失败的官方快照只短时间复用。 +- Skill 版本不变（BUG-1330～1332）。 + ## 2026-10-14 — 婚恋评级改成证据档位，KP 判宫和大运年长对齐（待合入 staging） - 婚恋结构指数不再写成婚姻稳定、离婚、再婚或建议婚前咨询。中文报告用中文档位，英文报告用英文档位。配偶星分数按 0 到 1 显示，不再写成百分制。 diff --git a/docs/BUG_HISTORY.md b/docs/BUG_HISTORY.md index add22027..23cf56b6 100644 --- a/docs/BUG_HISTORY.md +++ b/docs/BUG_HISTORY.md @@ -17795,3 +17795,48 @@ - 复发自：无。 - 修复版本：未修复。 +## BUG-1330 | 排盘缓存只进不出，同一张盘还会存成多份 + +- 状态：fixed-pending-verify +- 首次发现 / 最近更新：2026-10-13 / 2026-10-14 +- 来源：issue #60。任务书 `docs/tasks/TASK-api-disk-cache-retention-20261013.md` T1、T3。 +- 现象：排盘结果写在磁盘缓存里。过期只在读取时判断，文件一直留着。启动时的清理扫的是异步任务目录，不是这份缓存。同一张盘因为小时、经纬度写成数字或字符串，会各存一份。 +- 触发条件：反复排盘，或同一次出生时间用不同写法提交。服务长时间不重启时，异步任务记录也会一直留到被轮询。 +- 根因：写缓存的人只做了读取时的过期判断。写清理的人以为缓存和异步任务在同一个目录、同一种记录。缓存键直接用了请求里的原值，计算前却会先把这些值解析成数字。 +- 修复：新的留存模块按缓存写入时间和排盘缓存期限删除过期文件，坏文件超过一天也删。启动时清理一次，之后按周期在后台再清。新写入的缓存文件改为仅本人可读，并先写临时文件再换上。缓存键把能解析的数字收成同一种写法，缓存格式版本从 5 改为 6，旧文件自然失效后被清掉。快速排盘和完整排盘仍然是两把不同的键。 +- 验证：`tests/test_api_scratch_retention.py`、`tests/test_api_async_job_contract.py`、`tests/test_api_server_security.py -k cache`、`tests/test_api_server_growth_contract.py`。数值写法的逐字节比较见 `scripts/research/api_chart_cache_byte_diff.py`。 +- 防复发：新增磁盘缓存必须同时在 `scripts/api_scratch_retention.py` 登记留存规则。`tests/test_api_scratch_retention.py` 会检查 `scripts/` 里新的 `scratch/local/` 目录有没有登记。 +- 相关记录：BUG-161、BUG-1331、BUG-1332。 +- 复发自：无。 +- 修复版本：分支 `codex/api-disk-cache-retention-20261013`，尚未合入 staging。 + +## BUG-1331 | VedAstro 审计文件留着出生资料，失败快照被缓存太久 + +- 状态：fixed-pending-verify +- 首次发现 / 最近更新：2026-10-13 / 2026-10-14 +- 来源：issue #60。任务书 T2。 +- 现象：每次官方快照归一都会在磁盘留一份带缩进的审计文件，里面有出生时间。失败、不完整、被挡住的快照和成功结果用同一个长缓存期限。状态接口报出的期限没有算上回落值，可能和实际不一致。 +- 触发条件：在线排盘或报告走到官方快照。样例环境把成功缓存写成 7 天时，一次失败也会被原样复用 7 天。 +- 根因：审计落盘最早只给离线比对用，后来同一条归一进了在线请求，开关没有跟着关掉。语义缓存把「拿到一个结果」当成「可以长期复用」。状态接口自己读了一个变量，没有走实际生效的那个函数。 +- 修复：审计落盘默认关闭，需要比对时再显式打开。打开时改为紧凑文件、仅本人可读、先写临时文件再换上。失败、不完整、被挡住的快照用短期限，默认 300 秒，并且不会长过成功期限。状态接口改为报出含回落的成功期限，并单独报出失败期限。过期的语义缓存、业务响应缓存和审计文件由留存模块删除。 +- 验证：`tests/test_api_scratch_retention.py` 覆盖默认关闭、打开后的文件、失败 301 秒后不再命中、状态接口回落。既有 VedAstro 适配器测试仍通过。需要审计文件的那一条测试改为显式打开开关。 +- 防复发：新增磁盘缓存必须同时在 `scripts/api_scratch_retention.py` 登记留存规则。 +- 相关记录：BUG-734、BUG-301、BUG-1330。 +- 复发自：无。 +- 修复版本：分支 `codex/api-disk-cache-retention-20261013`，尚未合入 staging。 + +## BUG-1332 | 前台快照、负缓存、网关任务和分盘段缓存没有清理 + +- 状态：fixed-pending-verify +- 首次发现 / 最近更新：2026-10-13 / 2026-10-14 +- 来源：issue #60。任务书 T4。 +- 现象：前台快照缓存超过回看窗口后不会再被读，但文件不删。负缓存只认当天，昨天的文件一直在。网关任务和分盘段缓存也只增不减。这些目录在持久卷上，部署不会清空。 +- 触发条件：使用前台官方快照、咨询负缓存、网关队列或校正分盘段。 +- 根因：当初只规定了什么时候还能读，没有规定什么时候可以删。 +- 修复：前台快照早于「今天减去回看天数再减一天」就删。负缓存早于今天（UTC）就删。已完成或失败、并且超过保留时间的网关任务连同原始响应一起删；排队中和进行中的不删。分盘段按修改时间保留，默认 7 天。只删符合原有文件名的文件，其它文件不碰。读写和缓存键都没有改。 +- 验证：`tests/test_api_scratch_retention.py` 覆盖这四个目录的留、删和不碰。`tests/test_vedastro_snapshot_cache.py` 与 `tests/test_vedastro_consultation_snapshot.py` 断言未改。 +- 防复发：新增磁盘缓存必须同时在 `scripts/api_scratch_retention.py` 登记留存规则。 +- 相关记录：BUG-727、BUG-1231、BUG-1330。 +- 复发自：无。 +- 修复版本：分支 `codex/api-disk-cache-retention-20261013`，尚未合入 staging。 + diff --git a/docs/tasks/PROGRESS-api-disk-cache-retention-20261013.md b/docs/tasks/PROGRESS-api-disk-cache-retention-20261013.md new file mode 100644 index 00000000..b60a458c --- /dev/null +++ b/docs/tasks/PROGRESS-api-disk-cache-retention-20261013.md @@ -0,0 +1,51 @@ +# PROGRESS · API 磁盘缓存留存（2026-10-14） + +- 分支：`codex/api-disk-cache-retention-20261013` +- 任务书基线写的是 `77057a819`。本分支从当时的 `origin/staging` `66cd60081f37a57680274633fcb3fb942d01bb2f` 拉出。A/B 的父提交用的是这个 SHA，不是任务书里的旧基线。 +- 接口文件行数：10946（验收口径 ≤ 10970）。类方法数和伪造点没有增加。 + +## 做了什么 + +- T1：新增 `scripts/api_scratch_retention.py`。启动时清理全部已登记目录，并按 `JYOTISH_SCRATCH_PRUNE_INTERVAL_SECONDS`（默认 1800，0 表示不启后台线程）周期清理。排盘缓存新文件改为仅本人可读，先写临时文件再换上。`prune_expired_async_jobs` 只留同名薄包装。 +- T2：`VEDASTRO_WRITE_AUDIT_ARTIFACTS` 默认关。关掉时不写审计文件，`artifact_path` 为 `None`。打开时写紧凑 JSON，仅本人可读，先写临时文件再换上。成功快照仍用原 TTL；失败、不完整、被挡住的用短 TTL，默认 300 秒，并且不超过成功 TTL。旧缓存没有 `negative` 标记时仍按成功的长期限读。状态接口改报含回落的成功期限，并多报失败期限。`.env.cn.example` 只加了注释，两个 604800 没改。 +- 成功的判据：`available is True` 且 `status == "ok"`。`partial` 即使 `available` 为真也算失败缓存。 +- T3：年月日能整除才收成整数，不截断 1990.5。时分秒和经纬度、时区能解析成有限数字就收成浮点，不四舍五入。解析不了的保持原值。日期取前 10 位，空值仍是空值。缓存格式版本 5 → 6。快速排盘和完整排盘仍是两把键。 +- T4：前台快照、负缓存、网关任务、分盘段都进了登记表。只删符合原文件名的过期文件。网关排队中和进行中的不删。读写逻辑和键没改。 +- T5：脚本在 `scripts/research/api_disk_cache_ab.py`，逐字节比较在 `scripts/research/api_chart_cache_byte_diff.py`。记录在 `docs/testing/api-disk-cache-retention-ab-20261013.md`。 +- T6：没有改咨询卡片缓存。数字用任务书 §2.5，没有重测：未命中构建 1.9 秒；文件 5097751 字节；命中时解析中位 68.5 毫秒；整次命中 82.5 毫秒；两天大约每个活跃用户 10 MB。可选方向仍是两条：按领域切片只读需要的部分；清理改为每隔几分钟一次，而不是每次读取都扫全目录。没有做进程内已解析包缓存。 +- T7：BUG-1330、BUG-1331、BUG-1332。变更说明一句。任务索引改为待验收。没有改界面说明。 + +## 命令和结果 + +| 检查 | 命令 | 结果 | +| --- | --- | --- | +| 新留存、异步任务、增长合同 | `pytest tests/test_api_scratch_retention.py tests/test_api_async_job_contract.py tests/test_api_server_growth_contract.py -q` | 26 通过 | +| 安全缓存 | `pytest tests/test_api_server_security.py -q -k cache` | 6 通过 | +| 任务书列出的 VedAstro / 咨询套件 | 见任务书 §8 那一条 | 4 失败，见下。其余通过 | +| 需要审计文件的执行器测试 | `pytest tests/test_vedastro_service_adapter_executor.py::test_vedastro_range_scan_records_hashes_and_artifact_path -q` | 1 通过 | +| 逐字节比较与 A/B | `python scripts/research/api_disk_cache_ab.py` | 退出码 0。命中率 B/A = 1.333。七天后 B 剩 40 个文件，等于当天键数。审计文件 0。负缓存剩 1。一万个文件的清理 20.174 秒 | +| 接口行数 | 按换行计数 | 10946 | + +既有断言改动只有一处，在 `tests/test_vedastro_service_adapter_executor.py`：原值是子进程沿用进程环境；新值是显式打开审计落盘；原因是默认关闭后这条测试仍要检查文件存在。这是测试前置，不是放宽断言。 + +## 未达到的目标（目标不改） + +- 混合写法再重复 3 次时，命中率只是 A 的 1.333 倍，不是 2.5 倍。重复本身已经让 A 命中三分之二。 +- 一万个过期文件的清理用了 20.174 秒，不是 2 秒内。清理不在请求线程。默认间隔和失败短期限没有因此改掉。 + +## 本机套件里原先就会红的三条 + +- `tests/test_vedastro_gateway.py` 要求原始响应文件权限是 0600。Windows 报出来的是可写，不是 0600。这条断言没改。新测试在 Linux 上仍检查 0600，Windows 上只检查文件写出来了。 +- `tests/test_vedastro_runtime_ops.py` 要求已安装的 VedAstro 是 1.23.25，本机是 1.23.30。与这次缓存改动无关。 +- `tests/test_three_engine_parity_runner.py` 的公开盘回放把 PyJHora 记成 raw_imported，不是 structured_captured，同时在下载星历。运行器没改。 + +## 环境缺口 + +- 生产和测试环境里成功缓存的实际秒数，这份仓库读不到。部署后看状态接口的两个期限数字即可。 +- 生产卷上已经堆了多少文件，这里不知道。 +- 主测量挡住了 VedAstro 导入。本机有 SDK，官方端口能连通。不挡 SDK 的探针仍关闭外网取数，一次完整排盘 4.167 秒，没有报错，没有拉真实快照。 + +## 未做 + +- 没有改部署文件、生产环境变量、数据库，也没有手工清生产卷。 +- 没有把咨询卡片缓存并进周期清理。 diff --git a/docs/tasks/README.md b/docs/tasks/README.md index 58b326a6..76572ba0 100644 --- a/docs/tasks/README.md +++ b/docs/tasks/README.md @@ -470,7 +470,7 @@ | `TASK-rect-scoring-perf-20261013.md` | — | 校正打分提速：引导窗口平方扫描、候选不变量重复算、响应带无读取点字段、候选数上限 241（BUG-1323～1325、1345） | 待领取 | `codex/rect-scoring-perf-20261013` | | `TASK-node-runtime-resources-20261013.md` | — | 报告 worker abort 监听泄漏、本地 PG 适配层往返与元数据缓存、无上限内存表、用户端会话读缓存（BUG-1326～1328、1346，PG 与缓存部分 A/B） | 待领取 | `codex/node-runtime-resources-20261013` | | `TASK-home-sidebar-rerender-20261013.md` | — | 流式回答时侧栏每帧整栏重渲染（BUG-1329，A/B） | 待领取 | `codex/home-sidebar-rerender-20261013` | -| `TASK-api-disk-cache-retention-20261013.md` | — | API 磁盘缓存只进不出、VedAstro 审计文件留存出生资料、失败快照久缓存、缓存键规范化（BUG-1330～1332，A/B） | 待领取 | `codex/api-disk-cache-retention-20261013` | +| `TASK-api-disk-cache-retention-20261013.md` | `PROGRESS-api-disk-cache-retention-20261013.md` | API 磁盘缓存只进不出、VedAstro 审计文件留存出生资料、失败快照久缓存、缓存键规范化（BUG-1330～1332，A/B） | 待验收 | `codex/api-disk-cache-retention-20261013` | | `TASK-report-reference-perf-20261013.md` | — | 长报告出稿提速：术语替换命中预检、哈希单次遍历、同盘复用（BUG-1335～1336，A/B） | 待领取 | `codex/report-reference-perf-20261013` | | `TASK-consult-prep-parallel-20261013.md` | — | 普通对话发问前准备步骤并行、模型版本短缓存、多领域并发开关（BUG-1337～1339，A/B） | 待领取 | `codex/consult-prep-parallel-20261013` | | `TASK-frontend-dead-code-20261013.md` | — | 前端死代码 17 文件与无调用方接口删除，含未登录重计算接口（BUG-1342） | 待领取 | `codex/frontend-dead-code-20261013` | diff --git a/docs/testing/api-disk-cache-retention-ab-20261013.md b/docs/testing/api-disk-cache-retention-ab-20261013.md new file mode 100644 index 00000000..7086e60d --- /dev/null +++ b/docs/testing/api-disk-cache-retention-ab-20261013.md @@ -0,0 +1,66 @@ +# 磁盘缓存留存 A/B（2026-10-14） + +对照的是本分支的父提交 `66cd60081f37a57680274633fcb3fb942d01bb2f`（建分支时的 `origin/staging`）。任务书里写的 `77057a819` 已经不是当前测试线头。A 用 `git show` 把父提交的接口文件载入同一进程，并把它的仓库路径指回本工作树，否则临时副本找不到引擎。B 是本分支。 + +演练用虚构盘，不写真实出生资料。文件都在系统临时目录，不写工作树里的 `scratch/`。 + +主测量挡住了 `jhora` 和 `vedastro` 的导入。本机装了 VedAstro SDK，并且能连上官方端口；另有一组不挡 SDK、但仍关闭外网取数，见文末。 + +## 命令 + +```powershell +.venv\Scripts\python.exe scripts\research\api_chart_cache_byte_diff.py +.venv\Scripts\python.exe scripts\research\api_disk_cache_ab.py +.venv\Scripts\python.exe scripts\research\api_disk_cache_ab.py --with-sdk --sdk-probe +``` + +## 命中率 + +20 张虚构盘 × 3 种写法 × 快速/完整 × 7 天 × 每种重复 3 次 = 2520 次取键。取键用的是真实缓存键函数，没有把 2520 次都做成完整排盘。 + +| | 改前 A | 改后 B | 变化 | +| --- | --- | --- | --- | +| 不同缓存键 | 840 | 280 | 少 560 | +| 命中次数 | 1680 | 2240 | 多 560 | +| 命中率 | 0.6667 | 0.8889 | B / A = 1.333 | + +目标是混合写法下 B 至少为 A 的 2.5 倍。按任务书规定的「重复 3 次」算，没有达到：A 自己靠重复就已经有三分之二命中，写法合并只能再抬到约 89%。若不算这 3 次重复、只看三种写法各问一次，A 的命中率是 0，B 是 0.6667。目标数字不改。 + +## 七天后的文件 + +留存演练写的是短记录，用来数文件，不是完整盘面。期限 900 秒，只有当天的键还没过期。 + +| | 改前 A | 改后 B | +| --- | --- | --- | +| 清理前文件数 | 840 | 280 | +| 清理前字节 | 109200 | 36400 | +| 清理后文件数 | 840（旧代码没有这份清理） | 40 | +| 当天仍有效的键 | 120 | 40 | + +B 清理后的文件数等于当天有效键数，达到「不超过当天活跃键」的目标。 + +审计文件：B 在默认关闭时写出 0 个。达到目标。 + +负缓存：种入当天和前一天各一份，清理后剩 1 份（当天）。达到目标。 + +## 清理耗时 + +| 场景 | 结果 | +| --- | --- | +| 预置 10000 个已过期排盘缓存后，整次清理 | 20.174 秒，排盘目录 20170.902 毫秒，删掉 10000，剩下 0 | +| 上面七天演练里的排盘目录 | 370.161 毫秒 | + +目标是 10000 个文件在 2 秒内、且不在请求线程。本次在后台函数里测，不在请求线程；耗时没有进 2 秒。这是 Windows 上逐个读文件再删除的实测，目标不改。默认清理间隔仍是 1800 秒，失败快照短期限仍是 300 秒，不因为这次没进 2 秒就改默认值。 + +## 排盘耗时（一次虚构盘，快速排盘，外网关闭） + +| | 中位数 | +| --- | --- | +| 未命中 3 次 | 23.036 毫秒 | +| 命中 3 次 | 5.249 毫秒 | + +## 响应是否还是同一张盘 + +数值等价的写法（小时 12 / 12.0 / "12"，纬度数字和字符串，时区 5.5 和 "5.5"，秒缺省和 0）在快速排盘下去掉 `runtime_cache` 后逐字节相同。完整排盘只差官方快照缓存上的写入时刻和到期时刻，盘面没有别的差异。父提交和本分支对同一张快速排盘、去掉 `runtime_cache` 后逐字节相同。因此这些字段可以合并进同一把缓存键。 + +不挡 SDK 的一组：本机有 VedAstro SDK，官方端口能连通。探针仍关闭外网取数。一次完整排盘 4.167 秒，没有报错。没有拉取真实快照。 diff --git a/scripts/api_scratch_retention.py b/scripts/api_scratch_retention.py new file mode 100644 index 00000000..4a31458a --- /dev/null +++ b/scripts/api_scratch_retention.py @@ -0,0 +1,609 @@ +"""Retention rules for scratch/local caches. + +Counts only. Never prints file names or cache payloads. +""" +from __future__ import annotations + +import json +import math +import os +import re +import secrets +import threading +import time +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any, Callable + +ROOT = Path(__file__).resolve().parents[1] +UNREADABLE_KEEP_SECONDS = 24 * 60 * 60 +_HEX64_JSON = re.compile(r"^[0-9a-f]{64}\.json$") +_ARTIFACT_JSON = re.compile(r"^[A-Za-z0-9_]+-[0-9a-f]{12}-[0-9a-f]{12}\.json$") +_GATEWAY_JOB_JSON = re.compile(r"^vgw_[0-9a-f]{20}\.json$") +_PRUNE_THREAD_NAME = "jyotish-scratch-prune" + +# Directory names under scratch/local that this module already knows how to prune. +# Subdirectories of vedastro_adapter are listed so a future writer is not a new top-level dir. +REGISTERED_DIR_NAMES = frozenset( + { + "api_chart_cache", + "high_rigor_jobs", + "api_chart_response_jobs", + "async_jobs.sqlite3", + "vedastro_adapter", + "vedastro_snapshot_cache", + "vedastro_snapshot_negative_cache", + "vedastro_gateway_jobs", + "varga_segments", + } +) + +# Offline tools or caches that already prune themselves. Not a new retention rule. +OFFLINE_SCRATCH_NAMES = frozenset( + { + "consult_full_data_cache", + "ocr_extract", + "three_engine_parity", + "public_release_deny_patterns.txt", + "pdf_review_123456", + } +) + +REGISTRY: tuple[dict[str, Any], ...] = ( + { + "name": "api_chart_cache", + "holds_birth_data": True, + "deletable": "created_at plus chart TTL, or unreadable json older than 24h", + }, + { + "name": "high_rigor_jobs", + "holds_birth_data": True, + "deletable": "expires_at_unix has passed; bad json is kept", + }, + { + "name": "api_chart_response_jobs", + "holds_birth_data": True, + "deletable": "expires_at_unix has passed; bad json is kept", + }, + { + "name": "async_jobs.sqlite3", + "holds_birth_data": True, + "deletable": "sqlite rows whose expires_at has passed", + }, + { + "name": "vedastro_adapter", + "holds_birth_data": True, + "deletable": "audit artifacts, response_cache, and official_full_snapshot_cache", + }, + { + "name": "vedastro_snapshot_cache", + "holds_birth_data": True, + "deletable": "reference_date before today UTC minus MAX_STALE_DAYS minus 1", + }, + { + "name": "vedastro_snapshot_negative_cache", + "holds_birth_data": True, + "deletable": "reference_date before today UTC", + }, + { + "name": "vedastro_gateway_jobs", + "holds_birth_data": True, + "deletable": "completed or failed jobs older than the keep window, plus their raw archive", + }, + { + "name": "varga_segments", + "holds_birth_data": True, + "deletable": "mtime older than JYOTISH_VARGA_SEGMENTS_KEEP_DAYS", + }, +) + + +def write_private_json(path: Path, payload: Any) -> None: + """Atomic UTF-8 JSON, mode 0600. Compact, stable key order.""" + path.parent.mkdir(parents=True, exist_ok=True) + temp = path.with_name(f"{path.name}.{secrets.token_hex(8)}.tmp") + temp.write_text( + json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")), + encoding="utf-8", + ) + try: + os.chmod(temp, 0o600) + except OSError: + pass + os.replace(temp, path) + try: + os.chmod(path, 0o600) + except OSError: + pass + + +def _finite_float(value: Any) -> float | None: + if isinstance(value, bool) or value is None: + return None + if isinstance(value, (int, float)): + number = float(value) + elif isinstance(value, str): + text = value.strip() + if not text: + return None + try: + number = float(text) + except ValueError: + return None + else: + return None + if not math.isfinite(number): + return None + return number + + +def _exact_int(value: Any) -> int | None: + if isinstance(value, bool) or value is None: + return None + if isinstance(value, int): + return value + if isinstance(value, float): + if not math.isfinite(value) or not value.is_integer(): + return None + return int(value) + if isinstance(value, str) and re.fullmatch(r"[+-]?\d+", value.strip()): + return int(value.strip()) + return None + + +def normalize_chart_cache_birth(body: dict | None) -> dict[str, Any]: + """Cache-key birth fields. Unparseable values stay as they arrived.""" + payload = body if isinstance(body, dict) else {} + normalized: dict[str, Any] = {} + for key in ("year", "month", "day"): + parsed = _exact_int(payload.get(key)) + normalized[key] = payload.get(key) if parsed is None else parsed + for key in ("hour", "minute", "lat", "lon", "tz"): + raw = payload.get(key) + parsed = _finite_float(raw) + normalized[key] = raw if parsed is None else parsed + second_raw = payload.get("second", 0) + second = _finite_float(second_raw) + normalized["second"] = second_raw if second is None else second + return normalized + + +def normalize_chart_cache_day(value: Any) -> Any: + if value is None or value == "": + return value + return str(value)[:10] + + +def _blank() -> dict[str, int]: + return {"scanned": 0, "removed": 0, "errors": 0} + + +def _older_than(path: Path, moment: float, seconds: float) -> bool: + try: + return moment - path.stat().st_mtime > seconds + except OSError: + return False + + +def _unlink(path: Path, stats: dict[str, int]) -> None: + try: + path.unlink() + stats["removed"] += 1 + except OSError: + stats["errors"] += 1 + + +def _repo_root() -> Path: + return ROOT + + +def _api_module(api_module=None): + if api_module is not None: + return api_module + import jyotish_api_server as api + + return api + + +def prune_async_job_records(now: float | None = None, api_module=None) -> dict[str, Any]: + """Same file and sqlite rules as the old startup cleanup. Bad job JSON is kept.""" + api = _api_module(api_module) + moment = time.time() if now is None else float(now) + scanned = 0 + removed = 0 + errors = 0 + directories: dict[str, dict[str, int]] = {} + if api._async_job_backend() == "sqlite": + try: + with api._sqlite_job_connection() as connection: + scanned = int(connection.execute("SELECT COUNT(*) FROM async_jobs").fetchone()[0]) + deleted = connection.execute( + "DELETE FROM async_jobs WHERE expires_at IS NOT NULL AND expires_at <= ?", + (moment,), + ).rowcount + removed = int(deleted or 0) + except OSError: + errors += 1 + directories["high_rigor_jobs"] = _blank() + directories["api_chart_response_jobs"] = _blank() + directories["async_jobs_sqlite"] = {"scanned": scanned, "removed": removed, "errors": errors} + return { + "scope": "async_job_cleanup", + "scanned": scanned, + "removed": removed, + "errors": errors, + "directories": directories, + } + + for scope, label in ( + (api._HIGH_RIGOR_JOB_SCOPE, "high_rigor_jobs"), + (api._API_CHART_CACHE_SCOPE, "api_chart_response_jobs"), + ): + stats = _blank() + try: + directory = api._async_job_dir(scope) + except OSError: + stats["errors"] += 1 + directories[label] = stats + errors += stats["errors"] + continue + if directory.is_dir(): + for path in directory.glob("*.json"): + stats["scanned"] += 1 + try: + record = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + continue + except OSError: + stats["errors"] += 1 + continue + expires_at = record.get("expires_at_unix") if isinstance(record, dict) else None + if isinstance(expires_at, (int, float)) and not isinstance(expires_at, bool) and moment >= float(expires_at): + _unlink(path, stats) + directories[label] = stats + scanned += stats["scanned"] + removed += stats["removed"] + errors += stats["errors"] + return { + "scope": "async_job_cleanup", + "scanned": scanned, + "removed": removed, + "errors": errors, + "directories": directories, + } + + +def _prune_chart_cache(moment: float, api_module=None) -> dict[str, int]: + api = _api_module(api_module) + stats = _blank() + try: + directory = api._api_chart_cache_dir() + ttl = api._api_chart_cache_ttl_seconds() + except OSError: + stats["errors"] += 1 + return stats + if not directory.is_dir(): + return stats + for path in directory.glob("*.json"): + if not _HEX64_JSON.match(path.name): + continue + stats["scanned"] += 1 + try: + record = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + if _older_than(path, moment, UNREADABLE_KEEP_SECONDS): + _unlink(path, stats) + continue + except OSError: + stats["errors"] += 1 + continue + created_at = record.get("created_at") if isinstance(record, dict) else None + if not isinstance(created_at, (int, float)) or isinstance(created_at, bool): + if _older_than(path, moment, UNREADABLE_KEEP_SECONDS): + _unlink(path, stats) + continue + if moment - float(created_at) > ttl: + _unlink(path, stats) + return stats + + +def _adapter(): + import vedastro_service_adapter as adapter + + return adapter + + +def _prune_created_at_dir(directory: Path, moment: float, ttl_for_record: Callable[[dict], float]) -> dict[str, int]: + stats = _blank() + if not directory.is_dir(): + return stats + for path in directory.glob("*.json"): + if not _HEX64_JSON.match(path.name): + continue + stats["scanned"] += 1 + try: + record = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + if _older_than(path, moment, UNREADABLE_KEEP_SECONDS): + _unlink(path, stats) + continue + except OSError: + stats["errors"] += 1 + continue + created_at = record.get("created_at") if isinstance(record, dict) else None + if not isinstance(created_at, (int, float)) or isinstance(created_at, bool): + if _older_than(path, moment, UNREADABLE_KEEP_SECONDS): + _unlink(path, stats) + continue + try: + ttl = float(ttl_for_record(record)) + except (TypeError, ValueError): + stats["errors"] += 1 + continue + if moment - float(created_at) > ttl: + _unlink(path, stats) + return stats + + +def _prune_response_cache(moment: float) -> dict[str, int]: + adapter = _adapter() + return _prune_created_at_dir( + Path(adapter.ARTIFACT_DIR) / "response_cache", + moment, + lambda _record: adapter._cache_ttl_seconds(), + ) + + +def _prune_snapshot_semantic_cache(moment: float) -> dict[str, int]: + adapter = _adapter() + + def ttl_for(record: dict) -> float: + if record.get("negative") is True: + return adapter._official_full_snapshot_negative_ttl_seconds() + return adapter._official_full_snapshot_cache_ttl_seconds() + + return _prune_created_at_dir( + Path(adapter.ARTIFACT_DIR) / "official_full_snapshot_cache", + moment, + ttl_for, + ) + + +def _audit_keep_seconds() -> float: + raw = os.environ.get("VEDASTRO_AUDIT_ARTIFACT_KEEP_DAYS", "7").strip() + try: + days = float(raw) if raw else 7.0 + except ValueError: + days = 7.0 + return max(0.0, days) * 86400.0 + + +def _prune_audit_artifacts(moment: float) -> dict[str, int]: + adapter = _adapter() + stats = _blank() + directory = Path(adapter.ARTIFACT_DIR) + if not directory.is_dir(): + return stats + enabled = adapter._audit_artifacts_enabled() + max_age = _audit_keep_seconds() if enabled else UNREADABLE_KEEP_SECONDS + for path in directory.glob("*.json"): + if not path.is_file() or not _ARTIFACT_JSON.match(path.name): + continue + stats["scanned"] += 1 + if _older_than(path, moment, max_age): + _unlink(path, stats) + return stats + + +def _parse_day(value: Any): + if not isinstance(value, str): + return None + raw = value[:10] + try: + return datetime.strptime(raw, "%Y-%m-%d").date() + except ValueError: + return None + + +def _prune_reference_dated(directory: Path, moment: float, *, earliest_kept) -> dict[str, int]: + stats = _blank() + if not directory.is_dir(): + return stats + for path in directory.glob("*.json"): + if not _HEX64_JSON.match(path.name): + continue + stats["scanned"] += 1 + try: + record = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + if _older_than(path, moment, UNREADABLE_KEEP_SECONDS): + _unlink(path, stats) + continue + except OSError: + stats["errors"] += 1 + continue + ref = _parse_day(record.get("reference_date") if isinstance(record, dict) else None) + if ref is None: + if _older_than(path, moment, UNREADABLE_KEEP_SECONDS): + _unlink(path, stats) + continue + if ref < earliest_kept: + _unlink(path, stats) + return stats + + +def _prune_vedastro_snapshot_cache(moment: float) -> dict[str, int]: + from vedastro_snapshot_cache import MAX_STALE_DAYS + + today = datetime.fromtimestamp(moment, timezone.utc).date() + earliest = today - timedelta(days=int(MAX_STALE_DAYS) + 1) + raw = os.environ.get("JYOTISH_VEDASTRO_SNAPSHOT_CACHE_DIR", "").strip() + directory = Path(raw) if raw else _repo_root() / "scratch" / "local" / "vedastro_snapshot_cache" + return _prune_reference_dated(directory, moment, earliest_kept=earliest) + + +def _prune_vedastro_negative_cache(moment: float) -> dict[str, int]: + today = datetime.fromtimestamp(moment, timezone.utc).date() + raw = os.environ.get("JYOTISH_VEDASTRO_SNAPSHOT_NEGATIVE_CACHE_DIR", "").strip() + directory = Path(raw) if raw else _repo_root() / "scratch" / "local" / "vedastro_snapshot_negative_cache" + return _prune_reference_dated(directory, moment, earliest_kept=today) + + +def _parse_updated_at(value: Any) -> float | None: + if not isinstance(value, str) or not value.strip(): + return None + text = value.strip().replace("Z", "+00:00") + try: + parsed = datetime.fromisoformat(text) + except ValueError: + return None + if parsed.tzinfo is None: + parsed = parsed.replace(tzinfo=timezone.utc) + return parsed.timestamp() + + +def _gateway_keep_seconds() -> float: + raw = os.environ.get("VEDASTRO_GATEWAY_JOB_KEEP_SECONDS", "86400").strip() + try: + return max(0.0, float(raw) if raw else 86400.0) + except ValueError: + return 86400.0 + + +def _prune_gateway_jobs(moment: float) -> dict[str, int]: + stats = _blank() + raw = os.environ.get("VEDASTRO_GATEWAY_QUEUE_DIR", "").strip() + directory = Path(raw).expanduser() if raw else _repo_root() / "scratch" / "local" / "vedastro_gateway_jobs" + if not directory.is_dir(): + return stats + keep = _gateway_keep_seconds() + for path in directory.glob("*.json"): + if not _GATEWAY_JOB_JSON.match(path.name): + continue + stats["scanned"] += 1 + try: + record = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + continue + except OSError: + stats["errors"] += 1 + continue + if not isinstance(record, dict): + continue + status = str(record.get("status") or "") + if status not in {"completed", "failed"}: + continue + updated = _parse_updated_at(record.get("updated_at")) + if updated is None or moment - updated <= keep: + continue + companion = directory / f"{path.stem}.official_raw_response.json" + _unlink(path, stats) + if companion.is_file(): + stats["scanned"] += 1 + _unlink(companion, stats) + return stats + + +def _varga_keep_seconds() -> float: + raw = os.environ.get("JYOTISH_VARGA_SEGMENTS_KEEP_DAYS", "7").strip() + try: + days = float(raw) if raw else 7.0 + except ValueError: + days = 7.0 + return max(0.0, days) * 86400.0 + + +def _prune_varga_segments(moment: float) -> dict[str, int]: + stats = _blank() + raw = os.environ.get("JYOTISH_CACHE_DIR", "").strip() + root = Path(raw) if raw else _repo_root() / "scratch" / "local" + directory = root / "varga_segments" + if not directory.is_dir(): + return stats + max_age = _varga_keep_seconds() + for path in directory.glob("*.json"): + if not _HEX64_JSON.match(path.name): + continue + stats["scanned"] += 1 + if _older_than(path, moment, max_age): + _unlink(path, stats) + return stats + + +def _stamp(stats: dict[str, int], started: float) -> dict[str, Any]: + return { + "scanned": int(stats.get("scanned", 0) or 0), + "removed": int(stats.get("removed", 0) or 0), + "errors": int(stats.get("errors", 0) or 0), + "elapsed_ms": round((time.perf_counter() - started) * 1000, 3), + } + + +def prune_all(now: float | None = None, api_module=None) -> dict[str, Any]: + """Prune every registered directory. Exceptions become error counts.""" + moment = time.time() if now is None else float(now) + api = _api_module(api_module) + started = time.perf_counter() + directories: dict[str, dict[str, Any]] = {} + + def _run(name: str, func: Callable[[float], dict[str, int]]) -> None: + t0 = time.perf_counter() + try: + stats = func(moment) + except Exception: + stats = {"scanned": 0, "removed": 0, "errors": 1} + directories[name] = _stamp(stats, t0) + + _run("api_chart_cache", lambda moment: _prune_chart_cache(moment, api)) + job_started = time.perf_counter() + try: + job = prune_async_job_records(moment, api) + job_dirs = job.get("directories") or {} + except Exception: + job_dirs = { + "high_rigor_jobs": {"scanned": 0, "removed": 0, "errors": 1}, + "api_chart_response_jobs": _blank(), + } + job_elapsed = round((time.perf_counter() - job_started) * 1000, 3) + for name, stats in job_dirs.items(): + stamped = _stamp(stats, job_started) + stamped["elapsed_ms"] = job_elapsed + directories[name] = stamped + _run("vedastro_response_cache", _prune_response_cache) + _run("vedastro_official_full_snapshot_cache", _prune_snapshot_semantic_cache) + _run("vedastro_audit_artifacts", _prune_audit_artifacts) + _run("vedastro_snapshot_cache", _prune_vedastro_snapshot_cache) + _run("vedastro_snapshot_negative_cache", _prune_vedastro_negative_cache) + _run("vedastro_gateway_jobs", _prune_gateway_jobs) + _run("varga_segments", _prune_varga_segments) + return { + "scanned": sum(item["scanned"] for item in directories.values()), + "removed": sum(item["removed"] for item in directories.values()), + "errors": sum(item["errors"] for item in directories.values()), + "elapsed_ms": round((time.perf_counter() - started) * 1000, 3), + "directories": directories, + } + + +def start_periodic_prune(interval_s: float | None = None): + """Daemon thread. interval <= 0 starts nothing. Startup prune is separate.""" + if interval_s is None: + raw = os.environ.get("JYOTISH_SCRATCH_PRUNE_INTERVAL_SECONDS", "1800").strip() + try: + interval_s = float(raw) if raw else 1800.0 + except ValueError: + interval_s = 1800.0 + if interval_s <= 0: + return None + + def _loop() -> None: + while True: + time.sleep(interval_s) + try: + prune_all() + except Exception: + continue + + thread = threading.Thread(target=_loop, name=_PRUNE_THREAD_NAME, daemon=True) + thread.start() + return thread diff --git a/scripts/jyotish_api_server.py b/scripts/jyotish_api_server.py index 39d3826f..ca07c6cd 100644 --- a/scripts/jyotish_api_server.py +++ b/scripts/jyotish_api_server.py @@ -2590,24 +2590,17 @@ def _vedastro_runtime_fingerprint() -> dict: def _build_api_chart_cache_payload(body: dict) -> dict: + import api_scratch_retention as retention + + today = body.get('today') or body.get('current_date') return { - 'cache_schema_version': 5, - 'birth': { - 'year': body.get('year'), - 'month': body.get('month'), - 'day': body.get('day'), - 'hour': body.get('hour'), - 'minute': body.get('minute'), - 'second': body.get('second', 0), - 'lat': body.get('lat'), - 'lon': body.get('lon'), - 'tz': body.get('tz'), - }, + 'cache_schema_version': 6, + 'birth': retention.normalize_chart_cache_birth(body), 'calculation': { 'ayanamsa': _request_ayanamsa(body), 'node_mode': body.get('node_mode', body.get('nodeMode', 'mean')), - 'today': body.get('today') or body.get('current_date'), - 'transit_date': body.get('transit_date'), + 'today': retention.normalize_chart_cache_day(today), + 'transit_date': retention.normalize_chart_cache_day(body.get('transit_date')), 'skip_vedastro_main_entry_overview': bool(body.get('skip_vedastro_main_entry_overview')), }, 'vedastro_runtime': _vedastro_runtime_fingerprint(), @@ -2678,14 +2671,15 @@ def _store_api_chart_response_cache(cache_payload: dict, payload: dict) -> dict: ) ttl_seconds = _api_chart_cache_ttl_seconds() if ttl_seconds > 0: - record = { - 'cache_key': cache_key, - 'created_at': created_at_unix, - 'payload': payload_with_cache, - } - _api_chart_cache_path(cache_key).write_text( - json.dumps(record, ensure_ascii=False, sort_keys=True), - encoding='utf-8', + import api_scratch_retention as retention + + retention.write_private_json( + _api_chart_cache_path(cache_key), + { + 'cache_key': cache_key, + 'created_at': created_at_unix, + 'payload': payload_with_cache, + }, ) return payload_with_cache @@ -2738,30 +2732,9 @@ def _sqlite_job_connection() -> sqlite3.Connection: def prune_expired_async_jobs() -> dict: """Best-effort startup cleanup for local job records; never reads payloads.""" - removed = 0 - scanned = 0 - if _async_job_backend() == "sqlite": - with _sqlite_job_connection() as connection: - scanned = connection.execute("SELECT COUNT(*) FROM async_jobs").fetchone()[0] - removed = connection.execute( - "DELETE FROM async_jobs WHERE expires_at IS NOT NULL AND expires_at <= ?", (time.time(),) - ).rowcount - return {'scope': 'async_job_cleanup', 'scanned': scanned, 'removed': removed} - for scope in (_HIGH_RIGOR_JOB_SCOPE, _API_CHART_CACHE_SCOPE): - directory = _async_job_dir(scope) - if not directory.is_dir(): - continue - for path in directory.glob('*.json'): - scanned += 1 - try: - record = json.loads(path.read_text(encoding='utf-8')) - expires_at = record.get('expires_at_unix') if isinstance(record, dict) else None - if isinstance(expires_at, (int, float)) and time.time() >= float(expires_at): - path.unlink() - removed += 1 - except (OSError, json.JSONDecodeError): - continue - return {'scope': 'async_job_cleanup', 'scanned': scanned, 'removed': removed} + import api_scratch_retention as retention + + return retention.prune_async_job_records(api_module=sys.modules[__name__]) def _new_async_job_identity(prefix: str) -> dict: @@ -10902,7 +10875,10 @@ def _parse_allowed_hosts(value): def start_server(port=5200, host='127.0.0.1', allowed_origins=None, allowed_hosts=None): - cleanup = prune_expired_async_jobs() + import api_scratch_retention as retention + + cleanup = retention.prune_all(api_module=sys.modules[__name__]) + retention.start_periodic_prune() server = ThreadingHTTPServer((host, port), JyotishAPIHandler) server.daemon_threads = True server.allowed_origins = allowed_origins or DEFAULT_ALLOWED_ORIGINS diff --git a/scripts/research/api_chart_cache_byte_diff.py b/scripts/research/api_chart_cache_byte_diff.py new file mode 100644 index 00000000..e368b9ba --- /dev/null +++ b/scripts/research/api_chart_cache_byte_diff.py @@ -0,0 +1,28 @@ +"""Byte-compare numeric chart-cache spellings. See api_disk_cache_ab.py.""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +import api_disk_cache_ab as ab + + +def main() -> dict: + parent_error = None + try: + parent = ab._load_parent() + except Exception as exc: + parent = None + parent_error = exc.__class__.__name__ + report = ab.run_byte_diff(parent or ab.current) + report["parent_import"] = parent_error or "ok" + report["parent"] = ab.PARENT + return report + + +if __name__ == "__main__": + json.dump(main(), sys.stdout, ensure_ascii=False, indent=2) + sys.stdout.write("\n") diff --git a/scripts/research/api_disk_cache_ab.py b/scripts/research/api_disk_cache_ab.py new file mode 100644 index 00000000..62e2879a --- /dev/null +++ b/scripts/research/api_disk_cache_ab.py @@ -0,0 +1,442 @@ +"""A/B for chart-cache keys, retention, and response bytes. + +Parent functions come from git show of the branch base (origin/staging at +branch creation), not from the task text's older baseline. Scratch files +stay in a temporary directory. +""" +from __future__ import annotations + +import importlib.util +import json +import os +import socket +import statistics +import subprocess +import sys +import tempfile +import time +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +PARENT = "66cd60081f37a57680274633fcb3fb942d01bb2f" +os.environ.setdefault("JYOTISH_SKIP_LOCAL_ENV", "1") +os.environ.setdefault("VEDASTRO_ENABLE_NETWORK", "0") + +_SDK_PRESENT = importlib.util.find_spec("vedastro") is not None + + +def _network_reachable() -> bool: + try: + sock = socket.create_connection(("api.vedastro.org", 443), 3) + except OSError: + return False + sock.close() + return True + + +_NETWORK = _network_reachable() + + +class _Block: + def find_spec(self, name, path=None, target=None): + if name.split(".")[0] in {"jhora", "vedastro"}: + raise ModuleNotFoundError(f"No module named '{name}'") + return None + + +if "--with-sdk" not in sys.argv: + sys.meta_path.insert(0, _Block()) +sys.path.insert(0, str(ROOT / "scripts")) +sys.path.insert(0, str(ROOT)) + +import api_scratch_retention as retention # noqa: E402 +import jyotish_api_server as current # noqa: E402 +import vedastro_service_adapter as adapter # noqa: E402 + + +def _load_parent(): + raw = subprocess.check_output(["git", "show", f"{PARENT}:scripts/jyotish_api_server.py"], text=True) + raw = raw.replace( + "SCRIPTS_DIR = os.path.dirname(os.path.abspath(__file__))", + f"SCRIPTS_DIR = {str(ROOT / 'scripts')!r}", + ) + raw = raw.replace( + "REPO_ROOT = os.path.abspath(os.path.join(SCRIPTS_DIR, '..'))", + f"REPO_ROOT = {str(ROOT)!r}", + ) + directory = Path(tempfile.mkdtemp(prefix="jyotish-cache-a-")) + path = directory / "jyotish_api_server_parent.py" + path.write_text(raw, encoding="utf-8") + spec = importlib.util.spec_from_file_location("jyotish_api_server_parent", path) + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +def _handler(module): + cls = module.JyotishAPIHandler + return cls.__new__(cls) + + +def _strip(payload: dict) -> str: + cloned = json.loads(json.dumps(payload, default=str)) + cloned.pop("runtime_cache", None) + return json.dumps(cloned, ensure_ascii=False, sort_keys=True, default=str) + + +def _diff_paths(left: str, right: str, limit: int = 12) -> list[str]: + a = json.loads(left) + b = json.loads(right) + + found: list[str] = [] + + def walk(one, two, prefix: str) -> None: + if len(found) >= limit: + return + if type(one) is not type(two): + found.append(prefix or "$") + return + if isinstance(one, dict): + for key in sorted(set(one) | set(two)): + if key not in one or key not in two: + found.append(f"{prefix}.{key}") + continue + walk(one[key], two[key], f"{prefix}.{key}") + elif isinstance(one, list): + if len(one) != len(two): + found.append(f"{prefix}[]") + return + for index, (item, other) in enumerate(zip(one, two)): + walk(item, other, f"{prefix}[{index}]") + elif one != two: + found.append(prefix or "$") + + walk(a, b, "") + return found + + +def _body(hour, lat, **extra) -> dict: + payload = { + "year": 1990, + "month": 1, + "day": 1, + "minute": 0, + "lon": 20.0, + "tz": 5.5, + "hour": hour, + "lat": lat, + "today": "2026-10-14", + } + payload.update(extra) + return payload + + +def run_byte_diff(parent=None) -> dict: + os.environ["JYOTISH_API_CHART_CACHE_TTL_SECONDS"] = "0" + os.environ["VEDASTRO_WRITE_AUDIT_ARTIFACTS"] = "0" + parent = parent or current + groups = { + "hour": [_body(12, 10.0), _body(12.0, 10.0), _body("12", 10.0)], + "lat": [_body(12, 10.0), _body(12, "10.0")], + "tz": [_body(12, 10.0), _body(12, 10.0, tz="5.5")], + "second": [_body(12, 10.0), _body(12, 10.0, second=0)], + } + report = {"groups": {}, "stable_repeat": None} + handler = _handler(current) + started = time.perf_counter() + try: + sample = handler._compute_chart_sync(_body(12, 10.0, skip_vedastro_main_entry_overview=True)) + except Exception as exc: + return {"ok": False, "error": exc.__class__.__name__, "elapsed_s": round(time.perf_counter() - started, 3)} + sample_s = round(time.perf_counter() - started, 3) + repeat = handler._compute_chart_sync(_body(12, 10.0, skip_vedastro_main_entry_overview=True)) + report["stable_repeat"] = _strip(sample) == _strip(repeat) + report["sample_seconds"] = sample_s + if not report["stable_repeat"]: + report["volatile_paths"] = _diff_paths(_strip(sample), _strip(repeat)) + for name, bodies in groups.items(): + for skip in (True, False): + rendered = [] + for body in bodies: + item = dict(body) + if skip: + item["skip_vedastro_main_entry_overview"] = True + rendered.append(_strip(handler._compute_chart_sync(item))) + same = len(set(rendered)) == 1 + entry = {"skip": skip, "same": same} + if not same: + entry["paths"] = _diff_paths(rendered[0], rendered[1]) + clocks = ("cache_created_at", "cache_expires_at", "called_at") + entry["only_clock_fields"] = bool(entry["paths"]) and all( + path.endswith(clocks) for path in entry["paths"] + ) + else: + entry["only_clock_fields"] = False + report["groups"][f"{name}:skip={skip}"] = entry + if parent is not current: + a_handler = _handler(parent) + b_handler = handler + body = _body(12, 10.0, skip_vedastro_main_entry_overview=True) + a_text = _strip(a_handler._compute_chart_sync(body)) + b_text = _strip(b_handler._compute_chart_sync(body)) + report["a_vs_b_same"] = a_text == b_text + if a_text != b_text: + report["a_vs_b_paths"] = _diff_paths(a_text, b_text) + report["ok"] = all(item["same"] or item.get("only_clock_fields") for item in report["groups"].values()) + return report + + +def _requests(): + charts = [] + for index in range(20): + charts.append( + { + "year": 1990, + "month": 1, + "day": 1 + (index % 27), + "minute": index % 60, + "lon": float(index), + "tz": 5.5, + } + ) + spellings = ( + {"hour": 12, "lat": 10.0}, + {"hour": 12.0, "lat": "10.0"}, + {"hour": "12", "lat": 10.0}, + ) + days = [f"2026-10-{day:02d}" for day in range(8, 15)] + rows = [] + for chart in charts: + for spelling in spellings: + for skip in (False, True): + for day in days: + body = dict(chart) + body.update(spelling) + body["today"] = day + if skip: + body["skip_vedastro_main_entry_overview"] = True + rows.append(body) + return rows + + +def _key(module, body) -> str: + return module._api_chart_cache_key(module._build_api_chart_cache_payload(body)) + + +def _hit_rate(module, rows) -> dict: + seen = set() + hits = 0 + for body in rows: + for _repeat in range(3): + key = _key(module, body) + if key in seen: + hits += 1 + else: + seen.add(key) + total = len(rows) * 3 + return {"requests": total, "unique_keys": len(seen), "hits": hits, "hit_rate": round(hits / total, 4)} + + +def _seed_and_prune(module, rows) -> dict: + now = time.time() + os.environ["JYOTISH_API_CHART_CACHE_TTL_SECONDS"] = "900" + scratch = Path(tempfile.mkdtemp(prefix="jyotish-cache-b-")) + chart = scratch / "api_chart_cache" + jobs = scratch / "jobs" + adapter_dir = scratch / "adapter" + chart.mkdir() + jobs.mkdir() + current._api_chart_cache_dir = lambda: chart + current._async_job_dir = lambda scope: jobs + current._async_job_backend = lambda: "file" + adapter.ARTIFACT_DIR = adapter_dir + os.environ["JYOTISH_VEDASTRO_SNAPSHOT_CACHE_DIR"] = str(scratch / "snap") + os.environ["JYOTISH_VEDASTRO_SNAPSHOT_NEGATIVE_CACHE_DIR"] = str(scratch / "neg") + os.environ["VEDASTRO_GATEWAY_QUEUE_DIR"] = str(scratch / "gw") + os.environ["JYOTISH_CACHE_DIR"] = str(scratch / "cache") + days = sorted({body["today"] for body in rows}) + newest = days[-1] + written = {} + for body in rows: + key = _key(module, body) + age_days = days.index(body["today"]) + age_days = len(days) - 1 - age_days + written[key] = now - age_days * 86400 + for key, created in written.items(): + path = chart / f"{key}.json" + path.write_text(json.dumps({"cache_key": key, "created_at": created, "payload": {}}), encoding="utf-8") + before = len(list(chart.glob("*.json"))) + before_bytes = sum(path.stat().st_size for path in chart.glob("*.json")) + today_keys = len({_key(module, body) for body in rows if body["today"] == newest}) + if module is current: + prune = retention.prune_all(now=now) + after = len(list(chart.glob("*.json"))) + elapsed = prune["directories"]["api_chart_cache"]["elapsed_ms"] + else: + after = before + elapsed = None + return { + "files_before": before, + "bytes_before": before_bytes, + "files_after": after, + "today_active_keys": today_keys, + "chart_prune_ms": elapsed, + } + + +def _artifact_counts() -> dict: + scratch = Path(tempfile.mkdtemp(prefix="jyotish-artifact-ab-")) + adapter.ARTIFACT_DIR = scratch + os.environ.pop("VEDASTRO_WRITE_AUDIT_ARTIFACTS", None) + result = { + "operation": "official_full_snapshot", + "source_metadata": { + "operation": "official_full_snapshot", + "request_hash": "a" * 64, + "response_hash": "b" * 64, + }, + } + adapter._write_artifact(result) + return {"b_artifact_files": len(list(scratch.glob("*.json")))} + + +def _negative_cache_after_prune() -> dict: + scratch = Path(tempfile.mkdtemp(prefix="jyotish-neg-ab-")) + os.environ["JYOTISH_VEDASTRO_SNAPSHOT_NEGATIVE_CACHE_DIR"] = str(scratch) + os.environ["JYOTISH_VEDASTRO_SNAPSHOT_CACHE_DIR"] = str(scratch / "unused-snap") + os.environ["VEDASTRO_GATEWAY_QUEUE_DIR"] = str(scratch / "unused-gw") + os.environ["JYOTISH_CACHE_DIR"] = str(scratch / "unused-cache") + current._api_chart_cache_dir = lambda: scratch / "unused-chart" + current._async_job_dir = lambda scope: scratch / "unused-jobs" + (scratch / "unused-chart").mkdir() + (scratch / "unused-jobs").mkdir() + today = time.strftime("%Y-%m-%d", time.gmtime()) + yesterday = time.strftime("%Y-%m-%d", time.gmtime(time.time() - 86400)) + for label, day in (("a" * 64, today), ("b" * 64, yesterday)): + path = scratch / f"{label}.json" + path.write_text(json.dumps({"reference_date": day}), encoding="utf-8") + retention.prune_all(now=time.time()) + return {"files_after": len(list(scratch.glob("*.json")))} + + +def _prune_10k() -> dict: + scratch = Path(tempfile.mkdtemp(prefix="jyotish-prune-10k-")) + chart = scratch / "chart" + chart.mkdir() + current._api_chart_cache_dir = lambda: chart + current._async_job_dir = lambda scope: scratch / "jobs" + (scratch / "jobs").mkdir() + current._async_job_backend = lambda: "file" + adapter.ARTIFACT_DIR = scratch / "adapter" + os.environ["JYOTISH_API_CHART_CACHE_TTL_SECONDS"] = "900" + os.environ["JYOTISH_VEDASTRO_SNAPSHOT_CACHE_DIR"] = str(scratch / "snap") + os.environ["JYOTISH_VEDASTRO_SNAPSHOT_NEGATIVE_CACHE_DIR"] = str(scratch / "neg") + os.environ["VEDASTRO_GATEWAY_QUEUE_DIR"] = str(scratch / "gw") + os.environ["JYOTISH_CACHE_DIR"] = str(scratch / "cache") + for index in range(10000): + path = chart / f"{index:064x}.json" + path.write_text('{"created_at":1,"payload":{}}', encoding="utf-8") + started = time.perf_counter() + result = retention.prune_all(now=time.time()) + elapsed = time.perf_counter() - started + return { + "prune_all_seconds": round(elapsed, 3), + "chart_removed": result["directories"]["api_chart_cache"]["removed"], + "chart_elapsed_ms": result["directories"]["api_chart_cache"]["elapsed_ms"], + "files_left": len(list(chart.glob("*.json"))), + } + + +def _one_chart_timing() -> dict: + os.environ["JYOTISH_API_CHART_CACHE_TTL_SECONDS"] = "0" + handler = _handler(current) + body = _body(12, 10.0, skip_vedastro_main_entry_overview=True) + misses = [] + for _index in range(3): + started = time.perf_counter() + handler._compute_chart_sync(body) + misses.append((time.perf_counter() - started) * 1000) + os.environ["JYOTISH_API_CHART_CACHE_TTL_SECONDS"] = "900" + scratch = Path(tempfile.mkdtemp(prefix="jyotish-hit-time-")) + current._api_chart_cache_dir = lambda: scratch + cold = handler._compute_chart_sync(body) + hits = [] + for _index in range(3): + started = time.perf_counter() + hit = handler._compute_chart_sync(body) + hits.append((time.perf_counter() - started) * 1000) + assert hit.get("runtime_cache", {}).get("cache_hit") is True + return { + "miss_median_ms": round(statistics.median(misses), 3), + "hit_median_ms": round(statistics.median(hits), 3), + "cold_has_runtime_cache": isinstance(cold, dict) and "runtime_cache" in cold, + } + + +def main() -> dict: + parent_error = None + try: + parent = _load_parent() + except Exception as exc: + parent = None + parent_error = exc.__class__.__name__ + rows = _requests() + side_a = _hit_rate(parent, rows) if parent is not None else None + side_b = _hit_rate(current, rows) + report = { + "parent": PARENT, + "parent_import": parent_error or "ok", + "vedastro_sdk": _SDK_PRESENT, + "vedastro_network": _NETWORK, + "hit_rate_a": side_a, + "hit_rate_b": side_b, + "artifacts": _artifact_counts(), + "negative_cache_files_after_prune": _negative_cache_after_prune(), + "prune_10k": _prune_10k(), + } + if side_a and side_a["hit_rate"]: + report["hit_rate_ratio_b_over_a"] = round(side_b["hit_rate"] / side_a["hit_rate"], 3) + try: + report["files_b"] = _seed_and_prune(current, rows) + if parent is not None: + report["files_a"] = _seed_and_prune(parent, rows) + except Exception as exc: + report["files_error"] = exc.__class__.__name__ + try: + report["timing"] = _one_chart_timing() + except Exception as exc: + report["timing_error"] = exc.__class__.__name__ + try: + report["byte_diff"] = run_byte_diff(parent or current) + except Exception as exc: + report["byte_diff"] = {"ok": False, "error": exc.__class__.__name__} + return report + + +def _sdk_probe() -> dict: + os.environ["JYOTISH_API_CHART_CACHE_TTL_SECONDS"] = "0" + os.environ["VEDASTRO_ENABLE_NETWORK"] = "0" + os.environ["VEDASTRO_WRITE_AUDIT_ARTIFACTS"] = "0" + handler = _handler(current) + body = _body(12, 10.0) + started = time.perf_counter() + try: + handler._compute_chart_sync(body) + error = None + except Exception as exc: + error = type(exc).__name__ + return { + "blocked_vedastro": False, + "vedastro_sdk": _SDK_PRESENT, + "vedastro_network": _NETWORK, + "network_enabled": False, + "elapsed_s": round(time.perf_counter() - started, 3), + "error": error, + } + + +if __name__ == "__main__": + payload = _sdk_probe() if "--sdk-probe" in sys.argv else main() + json.dump(payload, sys.stdout, ensure_ascii=False, indent=2) + sys.stdout.write("\n") diff --git a/scripts/vedastro_gateway.py b/scripts/vedastro_gateway.py index f81e9955..f35ad7cc 100644 --- a/scripts/vedastro_gateway.py +++ b/scripts/vedastro_gateway.py @@ -340,6 +340,30 @@ def probe_official_rest_health(*, force_refresh: bool = True) -> dict[str, Any]: return result +def _gateway_snapshot_ttl_helpers(): + try: + from vedastro_service_adapter import ( + _official_full_snapshot_cache_ttl_seconds, + _official_full_snapshot_negative_ttl_seconds, + ) + except ModuleNotFoundError: + from scripts.vedastro_service_adapter import ( + _official_full_snapshot_cache_ttl_seconds, + _official_full_snapshot_negative_ttl_seconds, + ) + return _official_full_snapshot_cache_ttl_seconds, _official_full_snapshot_negative_ttl_seconds + + +def _gateway_official_snapshot_ttl_seconds() -> float: + positive, _negative = _gateway_snapshot_ttl_helpers() + return positive() + + +def _gateway_official_snapshot_negative_ttl_seconds() -> float: + _positive, negative = _gateway_snapshot_ttl_helpers() + return negative() + + def gateway_status(*, force_refresh: bool = True) -> dict[str, Any]: from scripts.diagnose_vedastro_mode import build_report as build_vedastro_mode_report @@ -368,7 +392,8 @@ def gateway_status(*, force_refresh: bool = True) -> dict[str, Any]: ), "sdk_version": _sdk_version(), "cache_ttl_seconds": config["cache_ttl_seconds"], - "official_full_snapshot_cache_ttl_seconds": _int_env("VEDASTRO_OFFICIAL_FULL_SNAPSHOT_CACHE_TTL_SECONDS", 0), + "official_full_snapshot_cache_ttl_seconds": _gateway_official_snapshot_ttl_seconds(), + "official_full_snapshot_negative_cache_ttl_seconds": _gateway_official_snapshot_negative_ttl_seconds(), "queue_enabled": config["queue_enabled"], "fail_open_local": config["fail_open_local"], "official_readiness": { diff --git a/scripts/vedastro_service_adapter.py b/scripts/vedastro_service_adapter.py index 40e47655..74fc34cf 100644 --- a/scripts/vedastro_service_adapter.py +++ b/scripts/vedastro_service_adapter.py @@ -425,14 +425,23 @@ def _repo_relative(path: Path) -> str: return str(path) -def _write_artifact(result: dict[str, Any]) -> str: +def _audit_artifacts_enabled() -> bool: + raw = os.environ.get("VEDASTRO_WRITE_AUDIT_ARTIFACTS", "0").strip().lower() + return raw in {"1", "true", "yes", "on"} + + +def _write_artifact(result: dict[str, Any]) -> str | None: + if not _audit_artifacts_enabled(): + return None metadata = result.get("source_metadata") or {} artifact = _artifact_path( str(metadata.get("operation") or result.get("operation") or "calculation"), str(metadata.get("request_hash") or "no-request-hash"), str(metadata.get("response_hash") or "no-response-hash"), ) - artifact.write_text(json.dumps(result, ensure_ascii=False, indent=2, sort_keys=True), encoding="utf-8") + from api_scratch_retention import write_private_json + + write_private_json(artifact, result) return _repo_relative(artifact) @@ -486,6 +495,22 @@ def _official_full_snapshot_cache_ttl_seconds() -> float: return 0.0 +def _official_full_snapshot_negative_ttl_seconds() -> float: + normal = _official_full_snapshot_cache_ttl_seconds() + if normal <= 0: + return 0.0 + raw = os.environ.get("VEDASTRO_OFFICIAL_FULL_SNAPSHOT_NEGATIVE_TTL_SECONDS", "300").strip() + try: + negative = max(0.0, float(raw)) if raw else 300.0 + except ValueError: + negative = 300.0 + return min(negative, normal) + + +def _official_full_snapshot_result_is_success(payload: dict[str, Any]) -> bool: + return payload.get("available") is True and payload.get("status") == "ok" + + def _free_tier_max_requests() -> int: raw = os.environ.get(FREE_TIER_MAX_REQUESTS_ENV, "").strip() if not raw: @@ -1719,9 +1744,11 @@ def _attach_official_full_snapshot_semantic_cache_metadata( cache_key: str, created_at_unix: float, cache_hit: bool, + ttl_seconds: float | None = None, ) -> dict[str, Any]: payload_copy = json.loads(json.dumps(payload)) - ttl_seconds = _official_full_snapshot_cache_ttl_seconds() + if ttl_seconds is None: + ttl_seconds = _official_full_snapshot_cache_ttl_seconds() created_at = datetime.utcfromtimestamp(created_at_unix).strftime("%Y-%m-%dT%H:%M:%SZ") expires_at = datetime.utcfromtimestamp(created_at_unix + ttl_seconds).strftime("%Y-%m-%dT%H:%M:%SZ") metadata = dict(payload_copy.get("source_metadata") or {}) @@ -1745,9 +1772,15 @@ def _rebind_official_full_snapshot_case_identity(payload: dict[str, Any], case_i return payload_copy +def _semantic_cache_ttl_seconds(record: dict[str, Any]) -> float: + # Records written before the negative flag are successes and keep the long TTL. + if record.get("negative") is True: + return _official_full_snapshot_negative_ttl_seconds() + return _official_full_snapshot_cache_ttl_seconds() + + def _load_official_full_snapshot_semantic_cache(case: dict[str, Any], case_id: str) -> dict[str, Any] | None: - ttl_seconds = _official_full_snapshot_cache_ttl_seconds() - if ttl_seconds <= 0: + if _official_full_snapshot_cache_ttl_seconds() <= 0: return None cache_key = _official_full_snapshot_cache_key(case, case_id) cache_path = _official_full_snapshot_cache_path(cache_key) @@ -1763,7 +1796,8 @@ def _load_official_full_snapshot_semantic_cache(case: dict[str, Any], case_id: s payload = record.get("payload") if not isinstance(created_at_raw, (int, float)) or not isinstance(payload, dict): return None - if time.time() - float(created_at_raw) > ttl_seconds: + ttl_seconds = _semantic_cache_ttl_seconds(record) + if ttl_seconds <= 0 or time.time() - float(created_at_raw) > ttl_seconds: return None rebound = _rebind_official_full_snapshot_case_identity(payload, case_id) return _attach_official_full_snapshot_semantic_cache_metadata( @@ -1771,28 +1805,37 @@ def _load_official_full_snapshot_semantic_cache(case: dict[str, Any], case_id: s cache_key=cache_key, created_at_unix=float(created_at_raw), cache_hit=True, + ttl_seconds=ttl_seconds, ) def _store_official_full_snapshot_semantic_cache(case: dict[str, Any], case_id: str, payload: dict[str, Any]) -> dict[str, Any]: cache_key = _official_full_snapshot_cache_key(case, case_id) created_at_unix = time.time() + negative = not _official_full_snapshot_result_is_success(payload) + ttl_seconds = ( + _official_full_snapshot_negative_ttl_seconds() + if negative + else _official_full_snapshot_cache_ttl_seconds() + ) payload_copy = _attach_official_full_snapshot_semantic_cache_metadata( payload, cache_key=cache_key, created_at_unix=created_at_unix, cache_hit=False, + ttl_seconds=ttl_seconds, ) - ttl_seconds = _official_full_snapshot_cache_ttl_seconds() if ttl_seconds > 0: - record = { - "cache_key": cache_key, - "created_at": created_at_unix, - "payload": payload_copy, - } - _official_full_snapshot_cache_path(cache_key).write_text( - json.dumps(record, ensure_ascii=False, sort_keys=True), - encoding="utf-8", + from api_scratch_retention import write_private_json + + write_private_json( + _official_full_snapshot_cache_path(cache_key), + { + "cache_key": cache_key, + "created_at": created_at_unix, + "negative": negative, + "payload": payload_copy, + }, ) return payload_copy @@ -1882,7 +1925,9 @@ def _store_cached_payload(cache_key: str, payload: dict[str, Any]) -> dict[str, "created_at": created_at_unix, "payload": payload_copy, } - cache_path.write_text(json.dumps(record, ensure_ascii=False, sort_keys=True), encoding="utf-8") + from api_scratch_retention import write_private_json + + write_private_json(cache_path, record) return payload_copy diff --git a/tests/test_api_scratch_retention.py b/tests/test_api_scratch_retention.py new file mode 100644 index 00000000..65dab310 --- /dev/null +++ b/tests/test_api_scratch_retention.py @@ -0,0 +1,314 @@ +"""Retention, chart-cache keys, and VedAstro audit switch.""" +from __future__ import annotations + +import hashlib +import json +import os +import re +import threading +import time +from datetime import datetime, timedelta, timezone +from pathlib import Path + +import pytest + +import api_scratch_retention as retention +import jyotish_api_server as api +import vedastro_service_adapter as adapter + +ROOT = Path(__file__).resolve().parents[1] + + +def _hex(label: str) -> str: + return hashlib.sha256(label.encode("utf-8")).hexdigest() + ".json" + + +def _isolate(monkeypatch, tmp_path: Path) -> None: + chart = tmp_path / "chart" + jobs = tmp_path / "jobs" + chart.mkdir() + jobs.mkdir() + monkeypatch.setattr(api, "_api_chart_cache_dir", lambda: chart) + monkeypatch.setattr(api, "_async_job_dir", lambda scope: jobs) + monkeypatch.setattr(api, "_async_job_backend", lambda: "file") + monkeypatch.setattr(adapter, "ARTIFACT_DIR", tmp_path / "adapter") + monkeypatch.setenv("JYOTISH_VEDASTRO_SNAPSHOT_CACHE_DIR", str(tmp_path / "snap")) + monkeypatch.setenv("JYOTISH_VEDASTRO_SNAPSHOT_NEGATIVE_CACHE_DIR", str(tmp_path / "neg")) + monkeypatch.setenv("VEDASTRO_GATEWAY_QUEUE_DIR", str(tmp_path / "gw")) + monkeypatch.setenv("JYOTISH_CACHE_DIR", str(tmp_path / "cache")) + monkeypatch.setenv("JYOTISH_API_CHART_CACHE_TTL_SECONDS", "900") + monkeypatch.delenv("VEDASTRO_WRITE_AUDIT_ARTIFACTS", raising=False) + + +def _write(path: Path, payload) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(payload), encoding="utf-8") + + +def test_prune_all_drops_expired_and_old_bad_chart_cache_only(monkeypatch, tmp_path): + _isolate(monkeypatch, tmp_path) + chart = api._api_chart_cache_dir() + now = time.time() + fresh = chart / _hex("fresh") + expired = chart / _hex("expired") + bad_old = chart / _hex("bad-old") + bad_young = chart / _hex("bad-young") + stranger = chart / "notes.json" + _write(fresh, {"cache_key": "a", "created_at": now, "payload": {}}) + _write(expired, {"cache_key": "b", "created_at": now - 5000, "payload": {}}) + bad_old.write_text("{", encoding="utf-8") + bad_young.write_text("{", encoding="utf-8") + stranger.write_text("{}", encoding="utf-8") + old = now - (25 * 3600) + os.utime(bad_old, (old, old)) + os.utime(stranger, (old, old)) + + result = retention.prune_all(now=now) + + assert fresh.exists() + assert bad_young.exists() + assert stranger.exists() + assert not expired.exists() + assert not bad_old.exists() + assert result["directories"]["api_chart_cache"]["removed"] == 2 + assert result["errors"] == 0 + + +def test_job_cleanup_still_removes_only_expired_records(monkeypatch, tmp_path): + _isolate(monkeypatch, tmp_path) + jobs = api._async_job_dir("any") + expired = jobs / "expired.json" + active = jobs / "active.json" + bad = jobs / "bad.json" + expired.write_text(json.dumps({"expires_at_unix": time.time() - 1}), encoding="utf-8") + active.write_text(json.dumps({"expires_at_unix": time.time() + 60}), encoding="utf-8") + bad.write_text("{", encoding="utf-8") + + result = api.prune_expired_async_jobs() + + assert result["removed"] == 1 + assert not expired.exists() + assert active.exists() + assert bad.exists() + + +def test_interval_zero_does_not_start_a_thread(): + before = [item.name for item in threading.enumerate()] + assert retention.start_periodic_prune(0) is None + assert [item.name for item in threading.enumerate()] == before + + +def test_prune_counts_unlink_errors_without_raising(monkeypatch, tmp_path): + _isolate(monkeypatch, tmp_path) + chart = api._api_chart_cache_dir() + target = chart / _hex("stuck") + _write(target, {"created_at": time.time() - 5000, "payload": {}}) + real_unlink = Path.unlink + + def _boom(self, *args, **kwargs): + if self == target: + raise OSError("denied") + return real_unlink(self, *args, **kwargs) + + monkeypatch.setattr(Path, "unlink", _boom) + result = retention.prune_all(now=time.time()) + assert result["errors"] > 0 + assert target.exists() + + +def _chart_key(body): + return api._api_chart_cache_key(api._build_api_chart_cache_payload(body)) + + +def test_numeric_birth_spellings_share_one_chart_cache_key(): + base = { + "year": 1990, + "month": 1, + "day": 1, + "minute": 0, + "lat": 28.6139, + "lon": 77.2090, + "tz": 5.5, + "today": "2026-10-14", + } + keys = {_chart_key(dict(base, hour=hour)) for hour in (12, 12.0, "12")} + assert len(keys) == 1 + assert _chart_key(dict(base, hour=12, second=0)) in keys + assert _chart_key(dict(base, hour=12, lat="28.6139")) in keys + assert _chart_key(dict(base, hour=12, tz="5.5")) in keys + assert _chart_key(dict(base, hour=12, tz="Asia/Kolkata")) not in keys + skipped = _chart_key(dict(base, hour=12, skip_vedastro_main_entry_overview=True)) + assert skipped not in keys + assert api._build_api_chart_cache_payload(base)["cache_schema_version"] == 6 + + +def test_audit_artifact_default_off_and_compact_when_on(monkeypatch, tmp_path): + monkeypatch.setattr(adapter, "ARTIFACT_DIR", tmp_path) + monkeypatch.delenv("VEDASTRO_WRITE_AUDIT_ARTIFACTS", raising=False) + manifest = {"requests": [{"section": "chart_core"}], "reference_date": "2026-10-14"} + quiet = adapter._normalize_official_full_snapshot_success( + "https://example.test/api", + manifest, + {"chart_core": {"Status": "Pass"}}, + {"chart_core": "ok"}, + 1, + [], + ) + assert quiet["source_metadata"]["artifact_path"] is None + assert list(tmp_path.rglob("*.json")) == [] + + monkeypatch.setenv("VEDASTRO_WRITE_AUDIT_ARTIFACTS", "1") + written = adapter._normalize_official_full_snapshot_success( + "https://example.test/api", + manifest, + {"chart_core": {"Status": "Pass"}}, + {"chart_core": "ok"}, + 1, + [], + ) + rel = written["source_metadata"]["artifact_path"] + path = Path(rel) + if not path.is_absolute(): + path = adapter.ROOT / rel + assert path.exists() + # Linux reports the mode bit. Windows keeps the file writable and does not surface 0600. + if os.name != "nt": + assert path.stat().st_mode & 0o777 == 0o600 + text = path.read_text(encoding="utf-8") + assert "\n" not in text + assert json.loads(text)["status"] == "ok" + + +def test_failed_snapshot_expires_before_success(monkeypatch, tmp_path): + monkeypatch.setattr(adapter, "ARTIFACT_DIR", tmp_path) + monkeypatch.setenv("VEDASTRO_OFFICIAL_FULL_SNAPSHOT_CACHE_TTL_SECONDS", "604800") + monkeypatch.delenv("VEDASTRO_OFFICIAL_FULL_SNAPSHOT_NEGATIVE_TTL_SECONDS", raising=False) + calls = {"fail": 0, "ok": 0} + + def fake_run(case, case_id="user_chart"): + if case.get("day") == 2: + calls["fail"] += 1 + return {"available": False, "status": "timeout", "source_metadata": {}} + calls["ok"] += 1 + return {"available": True, "status": "ok", "source_metadata": {}} + + monkeypatch.setattr(adapter, "_run_official_full_snapshot_case", fake_run) + fail_case = {"year": 1990, "month": 1, "day": 2, "hour": 12, "lat": 1.0, "lon": 2.0, "tz": 5.5} + ok_case = {"year": 1990, "month": 1, "day": 3, "hour": 12, "lat": 1.0, "lon": 2.0, "tz": 5.5} + adapter.run_official_full_snapshot_for_case(fail_case, case_id="fail") + adapter.run_official_full_snapshot_for_case(ok_case, case_id="ok") + cache_dir = tmp_path / "official_full_snapshot_cache" + aged = time.time() - 301 + for path in cache_dir.glob("*.json"): + record = json.loads(path.read_text(encoding="utf-8")) + record["created_at"] = aged + path.write_text(json.dumps(record), encoding="utf-8") + if record.get("negative") is True: + assert record["negative"] is True + adapter.run_official_full_snapshot_for_case(fail_case, case_id="fail") + adapter.run_official_full_snapshot_for_case(ok_case, case_id="ok") + assert calls["fail"] == 2 + assert calls["ok"] == 1 + + +def test_gateway_status_reports_cache_ttl_fallback(monkeypatch): + from scripts import vedastro_gateway + + monkeypatch.setenv("JYOTISH_SKIP_LOCAL_ENV", "1") + monkeypatch.delenv("VEDASTRO_OFFICIAL_FULL_SNAPSHOT_CACHE_TTL_SECONDS", raising=False) + monkeypatch.setenv("VEDASTRO_CACHE_TTL_SECONDS", "604800") + status = vedastro_gateway.gateway_status() + assert status["official_full_snapshot_cache_ttl_seconds"] == 604800 + assert status["official_full_snapshot_negative_cache_ttl_seconds"] == 300 + + +def _day(offset: int) -> str: + return (datetime.now(timezone.utc).date() + timedelta(days=offset)).isoformat() + + +def test_snapshot_negative_gateway_and_varga_prune_rules(monkeypatch, tmp_path): + _isolate(monkeypatch, tmp_path) + now = time.time() + today = datetime.fromtimestamp(now, timezone.utc).date() + snap = tmp_path / "snap" + neg = tmp_path / "neg" + gw = tmp_path / "gw" + varga = tmp_path / "cache" / "varga_segments" + keep_ref = (today - timedelta(days=7)).isoformat() + drop_ref = (today - timedelta(days=9)).isoformat() + _write(snap / _hex("snap-keep"), {"schema": "vedastro_snapshot_cache.v1", "reference_date": keep_ref}) + _write(snap / _hex("snap-drop"), {"schema": "vedastro_snapshot_cache.v1", "reference_date": drop_ref}) + _write(snap / "notes.json", {"reference_date": "1990-01-01"}) + _write(neg / _hex("neg-today"), {"schema": "vedastro_snapshot_negative_cache.v1", "reference_date": today.isoformat()}) + _write(neg / _hex("neg-yesterday"), {"schema": "vedastro_snapshot_negative_cache.v1", "reference_date": (today - timedelta(days=1)).isoformat()}) + _write(neg / "stray.json", {"reference_date": "1990-01-01"}) + job_id = "vgw_" + "ab" * 10 + old_stamp = datetime.fromtimestamp(now - 90000, timezone.utc).isoformat() + fresh_stamp = datetime.fromtimestamp(now, timezone.utc).isoformat() + _write(gw / f"{job_id}.json", {"job_id": job_id, "status": "completed", "updated_at": old_stamp}) + (gw / f"{job_id}.official_raw_response.json").write_text("{}", encoding="utf-8") + running = "vgw_" + "cd" * 10 + _write(gw / f"{running}.json", {"job_id": running, "status": "running", "updated_at": old_stamp}) + failed = "vgw_" + "ef" * 10 + _write(gw / f"{failed}.json", {"job_id": failed, "status": "failed", "updated_at": fresh_stamp}) + (gw / "notes.json").write_text("{}", encoding="utf-8") + old_varga = varga / _hex("old-segment") + young_varga = varga / _hex("young-segment") + _write(old_varga, {"window_key": "x"}) + _write(young_varga, {"window_key": "y"}) + _write(varga / "notes.json", {"window_key": "z"}) + os.utime(old_varga, (now - 8 * 86400, now - 8 * 86400)) + + monkeypatch.setenv("VEDASTRO_WRITE_AUDIT_ARTIFACTS", "0") + artifact = tmp_path / "adapter" / "official_full_snapshot-0123456789ab-0123456789ab.json" + artifact.parent.mkdir(parents=True, exist_ok=True) + artifact.write_text("{}", encoding="utf-8") + os.utime(artifact, (now - 25 * 3600, now - 25 * 3600)) + kept_artifact = tmp_path / "adapter" / "method_catalog_snapshot.json" + kept_artifact.write_text("{}", encoding="utf-8") + os.utime(kept_artifact, (now - 25 * 3600, now - 25 * 3600)) + + retention.prune_all(now=now) + + assert (snap / _hex("snap-keep")).exists() + assert not (snap / _hex("snap-drop")).exists() + assert (snap / "notes.json").exists() + assert (neg / _hex("neg-today")).exists() + assert not (neg / _hex("neg-yesterday")).exists() + assert (neg / "stray.json").exists() + assert not (gw / f"{job_id}.json").exists() + assert not (gw / f"{job_id}.official_raw_response.json").exists() + assert (gw / f"{running}.json").exists() + assert (gw / f"{failed}.json").exists() + assert (gw / "notes.json").exists() + assert not old_varga.exists() + assert young_varga.exists() + assert (varga / "notes.json").exists() + assert not artifact.exists() + assert kept_artifact.exists() + + +def _scratch_local_names() -> set[str]: + names: set[str] = set() + slash = re.compile(r"scratch/local/([A-Za-z0-9_.]+)") + joined = re.compile( + r"""['\"]scratch['\"]\s*/\s*['\"]local['\"]\s*/\s*(?:f)?['\"]([A-Za-z0-9_.]+)['\"]""" + ) + after_local = re.compile( + r"""['\"]local['\"]\s*\)*\s*/\s*['\"]([A-Za-z0-9_.]+)['\"]""" + ) + for path in (ROOT / "scripts").rglob("*.py"): + text = path.read_text(encoding="utf-8") + names.update(slash.findall(text)) + names.update(joined.findall(text)) + names.update(after_local.findall(text)) + return names + + +def test_new_scratch_local_dirs_are_registered_or_whitelisted(): + allowed = set(retention.REGISTERED_DIR_NAMES) | set(retention.OFFLINE_SCRATCH_NAMES) + found = _scratch_local_names() + unexpected = sorted(found - allowed) + assert unexpected == [], unexpected + assert "api_chart_cache" in found + assert "consult_full_data_cache" in retention.OFFLINE_SCRATCH_NAMES diff --git a/tests/test_vedastro_service_adapter_executor.py b/tests/test_vedastro_service_adapter_executor.py index e4f87f11..fd4e869c 100644 --- a/tests/test_vedastro_service_adapter_executor.py +++ b/tests/test_vedastro_service_adapter_executor.py @@ -769,6 +769,10 @@ def test_vedastro_range_scan_records_hashes_and_artifact_path() -> None: env = os.environ.copy() env["VEDASTRO_API_ENDPOINT"] = f"http://127.0.0.1:{server.server_port}/vedastro" env["VEDASTRO_ENABLE_NETWORK"] = "1" + # 原值：沿用进程环境，没有 VEDASTRO_WRITE_AUDIT_ARTIFACTS。 + # 新值：显式设为 1。 + # 原因：审计落盘默认关闭后，这条测试仍要断言文件存在，所以打开开关。这是测试前置，不是放宽断言。 + env["VEDASTRO_WRITE_AUDIT_ARTIFACTS"] = "1" completed = subprocess.run( [ sys.executable, From 69c739fbee3f5af6deed911482d82e49ed429bf7 Mon Sep 17 00:00:00 2001 From: jesse-ux Date: Wed, 14 Oct 2026 06:37:40 +0800 Subject: [PATCH 4/9] =?UTF-8?q?=E4=B8=AA=E4=BA=BA=E9=95=BF=E6=8A=A5?= =?UTF-8?q?=E5=91=8A=E5=90=8C=E7=9B=98=E5=90=8C=E6=97=A5=E9=87=8D=E8=AF=95?= =?UTF-8?q?=E6=94=B9=E4=B8=BA=E7=A7=92=E5=9B=9E?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 完整数据版在版本号明确时复用当天的结果。渲染快捷算法留下但默认关闭，本机只快了大约两成。 Co-Authored-By: Grok (agent loop) --- CHANGELOG.md | 6 + docs/BUG_HISTORY.md | 30 ++ ...PROGRESS-report-reference-perf-20261013.md | 40 ++ docs/tasks/README.md | 2 +- .../report-reference-perf-ab-20261013.md | 61 +++ scripts/calculation_profile_contract.py | 85 +++- scripts/consult_full_data_cache.py | 31 +- scripts/pl9_language_terms.py | 161 ++++++- scripts/professional_report_reference.py | 111 ++++- scripts/report_perf_flags.py | 40 ++ scripts/research/report_reference_perf_ab.py | 428 ++++++++++++++++++ tests/test_report_reference_perf.py | 374 +++++++++++++++ 12 files changed, 1334 insertions(+), 35 deletions(-) create mode 100644 docs/tasks/PROGRESS-report-reference-perf-20261013.md create mode 100644 docs/testing/report-reference-perf-ab-20261013.md create mode 100644 scripts/report_perf_flags.py create mode 100644 scripts/research/report_reference_perf_ab.py create mode 100644 tests/test_report_reference_perf.py diff --git a/CHANGELOG.md b/CHANGELOG.md index d2056bfc..6f5c40b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,11 @@ # 印度占星 Skill 更新日志 +## 2026-10-14 — 个人长报告引擎出稿提速（同盘同日重试秒回） + +- 同一张盘、同一天、同一个已发布版本，再次生成完整数据版长报告时直接用刚才的结果，本机大约 0.06 秒。第一次生成仍走原来的计算。版本号缺失时不缓存。 +- 中文替换和溯源哈希的快捷算法已留下，但本机只快了大约两成，没有达到开通门槛，默认仍走原算法。 +- Skill 版本不变（BUG-1335、BUG-1336）。 + ## 2026-10-14 — 婚恋评级改成证据档位，KP 判宫和大运年长对齐（待合入 staging） - 婚恋结构指数不再写成婚姻稳定、离婚、再婚或建议婚前咨询。中文报告用中文档位，英文报告用英文档位。配偶星分数按 0 到 1 显示，不再写成百分制。 diff --git a/docs/BUG_HISTORY.md b/docs/BUG_HISTORY.md index add22027..543c8ae7 100644 --- a/docs/BUG_HISTORY.md +++ b/docs/BUG_HISTORY.md @@ -17795,3 +17795,33 @@ - 复发自：无。 - 修复版本：未修复。 +## BUG-1335 | 个人长报告的中文出稿和溯源哈希重复劳动 + +- 状态：investigating +- 首次发现 / 最近更新：2026-10-13 / 2026-10-14 +- 来源：issue #61。任务书 `docs/tasks/TASK-report-reference-perf-20261013.md`。 +- 现象：同一份个人长报告，中文正文要做多遍全文替换；大运补句的正则没有锚点，在长行上反复试；正文里根本没有的术语也会先编译再替换。溯源哈希对每一条「任意层的某个键」都把整包复制一遍。 +- 触发条件：生成完整数据版长报告。不是某一次改坏的，是这些补丁一直往上叠。 +- 根因：替换规则一条条追加，没有先问这句话在不在。哈希按排除路径各走一遍，再单独做键名规范化。 +- 修复：加了等值的预检、预编译，以及哈希的一次遍历。旧代码留在开关后面。本机屏蔽 PyJHora 后，端到端只快了大约两成，没有达到 35% 的开通门槛，所以开关默认关，读者现在仍走原来的出稿速度。 +- 验证：四张虚构盘的中文、英文、事实表、适用性和整份响应逐字节一致。单次哈希中位从 1.115 秒降到 0.402 秒。详见 `docs/testing/report-reference-perf-ab-20261013.md`。 +- 防复发：`scripts/research/report_reference_perf_ab.py` 会在字节不一致时失败；`tests/test_report_reference_perf.py` 用真实引擎包对比开关两边和新旧哈希。 +- 相关记录：BUG-693、BUG-694、BUG-576、BUG-1336。 +- 复发自：无。 +- 修复版本：分支 `codex/report-reference-perf-20261013`，默认未开通，尚未合入 staging。 + +## BUG-1336 | 同一张盘同一天再要长报告，每次都重算 + +- 状态：fixed-pending-verify +- 首次发现 / 最近更新：2026-10-13 / 2026-10-14 +- 来源：issue #61。任务书 `docs/tasks/TASK-report-reference-perf-20261013.md`。 +- 现象：同一输入的长报告每次字节都一样，却没有复用。重试或当天再生成，都要再占一次重计算名额，再等十几秒以上。 +- 触发条件：完整数据版、markdown。其它格式和精简版不在这条里。 +- 根因：咨询卡已经有按键单飞和两天清理，长报告这条生产路径没有接上。 +- 修复：只缓存完整数据版的 markdown。键包含规范化后的出生输入、格式、版本、语言、事实表开关和代码版本号。版本号缺失或是 `unknown` 时不缓存。命中字节与现算相同。失败不落盘。同键并发只算一次。重计算名额逻辑没改，命中只占很短时间。 +- 验证：四张虚构盘命中中位约 0.06 秒，低于 1.5 秒，且与未命中字节相同。新测试覆盖改日期、语言、事实表、候选窗口、版本号都会错过，json 和开关关闭不落盘，并发只建一次，失败后下次会重算。 +- 防复发：`tests/test_report_reference_perf.py` 的缓存用例；A/B 脚本把命中和未命中的 sha256 放在一起比。 +- 相关记录：BUG-576、BUG-1304、BUG-1335、BUG-693。 +- 复发自：无。 +- 修复版本：分支 `codex/report-reference-perf-20261013`，尚未合入 staging。 + diff --git a/docs/tasks/PROGRESS-report-reference-perf-20261013.md b/docs/tasks/PROGRESS-report-reference-perf-20261013.md new file mode 100644 index 00000000..c719bf99 --- /dev/null +++ b/docs/tasks/PROGRESS-report-reference-perf-20261013.md @@ -0,0 +1,40 @@ +# PROGRESS · 长报告出稿提速（2026-10-14） + +- 分支：`codex/report-reference-perf-20261013` +- 实际基线：`origin/staging` `66cd60081f37a57680274633fcb3fb942d01bb2f`（任务书写的 `77057a819`、计划评论里的 `2994588ad` 都更早，按当前 staging 开工） +- `scripts/jyotish_api_server.py`：10335 行，零改动 +- 历史最大号在本基线上是 BUG-1322。BUG-1323～1334 已被其它未合入任务书占用。本单用任务书指定的 BUG-1335、BUG-1336。 + +## 做了什么 + +1. 两个开关在调用时读环境变量。渲染快捷路径默认关，同盘缓存默认开。 +2. 中文和英文的大运密度补句：正则预编译，行内先看必出现的字面量，没有才不搜索；向前看的五行只在可能命中时才拼。术语替换：正文里没有这个词就原样返回，编译结果按词缓存。 +3. 溯源哈希在开关打开时，一次遍历同时删掉 `**.键` 并做键名规范化。覆盖范围和排除集没变。没有跳过中间那次绑定，因为那会改 `format=json` 的键顺序。 +4. 咨询卡缓存的单飞和两天清理多了一个子目录参数，咨询卡自己的键和目录不变。长报告只在完整数据版 + markdown、开关开、并且 `GITHUB_SHA` 是具体版本号时，才把整份响应放进 `scratch/local/report_reference_cache/`。查缓存发生在重计算之前。命中打一行不含资料的日志。 + +## 自测 + +| 检查 | 命令 | 结果 | +| --- | --- | --- | +| 任务书列出的 pytest | `.venv\Scripts\python.exe -m pytest` 后接任务书第 8 节那一串文件 `-q` | 退出码 0。缓存默认改成开之后又跑了一遍，仍是退出码 0 | +| 屏蔽 PyJHora 的 A/B | `python scripts/research/report_reference_perf_ab.py --mask-pyjhora` | 四张盘五字段 sha 相同；端到端下降 20%～23%，未达 35%；缓存命中中位约 0.06 秒 | +| 同一份解盘的 json / reference | 同上脚本 `--side-only` | 退出码 0。时钟钉死且复用同一次解盘后，两版 sha 相同 | +| 不屏蔽 | 未跑 | 本机没有 PyJHora，和屏蔽走同一条导入失败路径 | + +数字见 `docs/testing/report-reference-perf-ab-20261013.md`。 + +## 改既有断言 + +无。原值 / 新值 / 原因：没有改旧断言。 + +## 未做 + +- 整篇替换逐条预检（让步顺序里最先砍的一项）。中文渲染只降了 20%～28%，不到 40%，没有为了过线改输出。 +- 跳过中间绑定。 +- 关掉没人读的质量门。任务书写明这要另开单，本单不动。 +- 旧路径退役。开关还在。 + +## 默认值 + +- `JYOTISH_REPORT_RENDER_FAST` 默认关。字节一致，内存还降了约 12%，但端到端没到 35%。 +- `JYOTISH_REPORT_REFERENCE_CACHE` 默认开。命中与未命中字节一致，命中中位约 0.06 秒。版本号缺失或是 `unknown` 时不缓存。 diff --git a/docs/tasks/README.md b/docs/tasks/README.md index 58b326a6..3f229f6d 100644 --- a/docs/tasks/README.md +++ b/docs/tasks/README.md @@ -471,7 +471,7 @@ | `TASK-node-runtime-resources-20261013.md` | — | 报告 worker abort 监听泄漏、本地 PG 适配层往返与元数据缓存、无上限内存表、用户端会话读缓存（BUG-1326～1328、1346，PG 与缓存部分 A/B） | 待领取 | `codex/node-runtime-resources-20261013` | | `TASK-home-sidebar-rerender-20261013.md` | — | 流式回答时侧栏每帧整栏重渲染（BUG-1329，A/B） | 待领取 | `codex/home-sidebar-rerender-20261013` | | `TASK-api-disk-cache-retention-20261013.md` | — | API 磁盘缓存只进不出、VedAstro 审计文件留存出生资料、失败快照久缓存、缓存键规范化（BUG-1330～1332，A/B） | 待领取 | `codex/api-disk-cache-retention-20261013` | -| `TASK-report-reference-perf-20261013.md` | — | 长报告出稿提速：术语替换命中预检、哈希单次遍历、同盘复用（BUG-1335～1336，A/B） | 待领取 | `codex/report-reference-perf-20261013` | +| `TASK-report-reference-perf-20261013.md` | `PROGRESS-report-reference-perf-20261013.md` | 长报告出稿提速：术语替换命中预检、哈希单次遍历、同盘复用（BUG-1335～1336，A/B） | 待验收 | `codex/report-reference-perf-20261013` | | `TASK-consult-prep-parallel-20261013.md` | — | 普通对话发问前准备步骤并行、模型版本短缓存、多领域并发开关（BUG-1337～1339，A/B） | 待领取 | `codex/consult-prep-parallel-20261013` | | `TASK-frontend-dead-code-20261013.md` | — | 前端死代码 17 文件与无调用方接口删除，含未登录重计算接口（BUG-1342） | 待领取 | `codex/frontend-dead-code-20261013` | | `TASK-api-chart-subprocess-gate-20261013.md` | — | 合盘不带 skip 串行排盘、报告无效子进程、名额轻重混用（只模拟）（BUG-1333～1334，A/B） | 待领取（排队：api-disk-cache-retention + report-reference-perf） | `codex/api-chart-subprocess-gate-20261013` | diff --git a/docs/testing/report-reference-perf-ab-20261013.md b/docs/testing/report-reference-perf-ab-20261013.md new file mode 100644 index 00000000..2da1f621 --- /dev/null +++ b/docs/testing/report-reference-perf-ab-20261013.md @@ -0,0 +1,61 @@ +# 长报告出稿 A/B（2026-10-14） + +Windows 本机，Python 3.11.7，`Windows-10-10.0.26200-SP0`。屏蔽 PyJHora（生产镜像不装这个库）。本机本身也没有 PyJHora，所以没有再跑一轮不屏蔽的重复计时。计时开了 tracemalloc，绝对秒数比任务书里的 Linux 基线高，只比较同一进程里的 A 和 B。 + +A = 两个开关关。B = 两个开关开。四张虚构盘，每组 3 次。生产请求是完整数据版、markdown、中英都要、带事实表。没有模型钥匙。 + +## 端到端（未命中，秒） + +| 盘 | A 三次 | A 中位 | B 三次 | B 中位 | 下降 | +| --- | --- | --- | --- | --- | --- | +| day | 71.31 / 64.03 / 64.01 | 64.03 | 51.05 / 50.77 / 51.74 | 51.05 | 20.3% | +| night | 66.25 / 66.11 / 66.66 | 66.25 | 53.32 / 52.57 / 50.43 | 52.57 | 20.7% | +| south | 66.03 / 66.00 / 62.69 | 66.00 | 49.97 / 52.79 / 52.68 | 52.68 | 20.2% | +| section2 | 71.01 / 108.37 / 74.19 | 74.19 | 57.42 / 57.12 / 71.80 | 57.42 | 22.6% | + +门槛是未命中端到端下降至少 35%。四张盘都没到。渲染快捷路径默认关闭。 + +## 分段中位数（秒） + +| 盘 | 中文渲染 A → B | 英文渲染 A → B | 建包 A → B | +| --- | --- | --- | --- | +| day | 23.55 → 17.04（−27.6%） | 8.27 → 8.67 | 22.55 → 16.08 | +| night | 23.42 → 17.19（−26.6%） | 8.75 → 8.41 | 23.70 → 17.21 | +| south | 23.89 → 17.39（−27.2%） | 8.66 → 8.74 | 22.90 → 16.87 | +| section2 | 22.26 → 17.80（−20.0%） | 8.14 → 8.54 | 31.24 → 21.08 | + +中文渲染门槛是下降至少 40%。没到。没有继续加整篇替换的预检。 + +## 内存与字节 + +tracemalloc 峰值：B 约为 A 的 0.87 到 0.88，没有升高。 + +五份输出（整份响应、中文、英文、事实表、适用性）A、B 未命中、缓存命中的 sha256 全部相同。 + +| 盘 | 响应字节 | 响应 sha256 前 12 位 | +| --- | --- | --- | +| day | 2289867 | 5b542359d606 | +| night | 2321638 | 89baa960c054 | +| south | 2314369 | 250bf1ab246e | +| section2 | 2238092 | 1d7afd481b1a | + +## 同盘缓存（命中，秒） + +| 盘 | 三次 | 中位 | +| --- | --- | --- | +| day | 0.059 / 0.054 / 0.057 | 0.057 | +| night | 0.057 / 0.054 / 0.058 | 0.057 | +| south | 0.059 / 0.054 / 0.057 | 0.057 | +| section2 | 0.063 / 0.066 / 0.057 | 0.063 | + +命中字节与未命中相同，中位数远低于 1.5 秒。缓存开关默认开。`GITHUB_SHA` 缺失或为 `unknown` 时仍然关。 + +## 单次溯源哈希（交付包，秒） + +关：1.132 / 1.099 / 1.115，中位 1.115。开：0.402 / 0.466 / 0.388，中位 0.402。下降 64%，过了 50% 那一档。它跟渲染快捷路径用同一个开关，所以默认也是关。 + +## json 与 reference 版 + +同一份解盘、时钟钉死后，`format=json` 和 reference 版 markdown 在快捷路径关与开时 sha256 相同（`aa1352619b7a`、`cbadf95f549a`）。对比的是整份响应，没有丢掉字段。 + +连续两次重新解盘会对不齐：有的特质列表挂在类上，第二次会再追加。那不是这条快捷路径改出来的。生产要的完整数据 markdown 在多次重算里仍然逐字节相同。 diff --git a/scripts/calculation_profile_contract.py b/scripts/calculation_profile_contract.py index 7a7f5871..b9f6e0b6 100755 --- a/scripts/calculation_profile_contract.py +++ b/scripts/calculation_profile_contract.py @@ -486,15 +486,26 @@ def _drop_dotted_path(payload: Any, path: tuple[str, ...]) -> Any: return cloned -def result_payload_for_binding( +def _render_fast() -> bool: + try: + from scripts.report_perf_flags import report_render_fast_enabled + except ModuleNotFoundError: # pragma: no cover - direct scripts/ execution + from report_perf_flags import report_render_fast_enabled + return report_render_fast_enabled() + + +def _excluded_binding_top(binding_scope: dict[str, Any]) -> set[str]: + return RESULT_BINDING_ENVELOPE_KEYS | { + str(key) for key in (binding_scope.get("excluded_top_keys") or ()) + } + + +def _result_payload_for_binding_legacy( result: dict[str, Any], scope: dict[str, Any] | None = None, ) -> dict[str, Any]: - """Business payload that participates in result_hash.""" binding_scope = scope if isinstance(scope, dict) else default_result_binding_scope() - excluded_top = RESULT_BINDING_ENVELOPE_KEYS | { - str(key) for key in (binding_scope.get("excluded_top_keys") or ()) - } + excluded_top = _excluded_binding_top(binding_scope) payload = {key: value for key, value in result.items() if key not in excluded_top} for raw_path in binding_scope.get("excluded_paths") or (): parts = tuple(part for part in str(raw_path).split(".") if part) @@ -503,17 +514,73 @@ def result_payload_for_binding( return payload +def _drop_star_keys_and_canonicalize(payload: Any, star_keys: frozenset[str]) -> Any: + """Drop ``**.key`` names and canonicalize keys in one walk. + + String keys use the same inequality as ``_drop_key_recursive``. Non-string + keys are kept, then renamed with the same rule as ``canonicalize_result_payload``. + """ + if isinstance(payload, dict): + normalized: dict[str, Any] = {} + for key, child in payload.items(): + if isinstance(key, str): + if key in star_keys: + continue + normalized_key = key + else: + normalized_key = f"__{type(key).__name__}__:{key}" + normalized[normalized_key] = _drop_star_keys_and_canonicalize(child, star_keys) + return normalized + if isinstance(payload, (list, tuple)): + return [_drop_star_keys_and_canonicalize(item, star_keys) for item in payload] + return payload + + +def _result_payload_for_binding_fast( + result: dict[str, Any], + scope: dict[str, Any] | None = None, +) -> dict[str, Any]: + binding_scope = scope if isinstance(scope, dict) else default_result_binding_scope() + excluded_top = _excluded_binding_top(binding_scope) + payload = {key: value for key, value in result.items() if key not in excluded_top} + star_keys: set[str] = set() + for raw_path in binding_scope.get("excluded_paths") or (): + parts = tuple(part for part in str(raw_path).split(".") if part) + if len(parts) == 2 and parts[0] == "**": + star_keys.add(parts[1]) + continue + if parts: + payload = _drop_dotted_path(payload, parts) + return _drop_star_keys_and_canonicalize(payload, frozenset(star_keys)) + + +def result_payload_for_binding( + result: dict[str, Any], + scope: dict[str, Any] | None = None, +) -> dict[str, Any]: + """Business payload that participates in result_hash.""" + if _render_fast(): + return _result_payload_for_binding_fast(result, scope) + return _result_payload_for_binding_legacy(result, scope) + + def hash_bound_result( result: dict[str, Any], input_hash: str, scope: dict[str, Any] | None = None, ) -> str: """SHA-256 of the canonical bound payload. Does not mutate ``result``.""" - encoded = json.dumps( - canonicalize_result_payload({ + payload = result_payload_for_binding(result, scope) + if _render_fast(): + # The fast payload is already canonical, including the star-key drop. + canonical = {"input_hash": input_hash, "result": payload} + else: + canonical = canonicalize_result_payload({ "input_hash": input_hash, - "result": result_payload_for_binding(result, scope), - }), + "result": payload, + }) + encoded = json.dumps( + canonical, ensure_ascii=False, sort_keys=True, separators=(",", ":"), diff --git a/scripts/consult_full_data_cache.py b/scripts/consult_full_data_cache.py index 43ffdd34..32ef94d9 100644 --- a/scripts/consult_full_data_cache.py +++ b/scripts/consult_full_data_cache.py @@ -57,8 +57,8 @@ def _release_warm_slot() -> None: _warm_inflight -= 1 -def cache_dir() -> Path: - path = ROOT / "scratch" / "local" / "consult_full_data_cache" +def cache_dir(name: str = "consult_full_data_cache") -> Path: + path = ROOT / "scratch" / "local" / name path.mkdir(parents=True, exist_ok=True) return path @@ -118,15 +118,15 @@ def cache_key(identity: dict[str, Any]) -> str: return hashlib.sha256(canonical.encode("utf-8")).hexdigest() -def _cache_path(identity: dict[str, Any]) -> Path: - return cache_dir() / f"{cache_key(identity)}.json" +def _cache_path(identity: dict[str, Any], name: str = "consult_full_data_cache") -> Path: + return cache_dir(name) / f"{cache_key(identity)}.json" -def prune_cache(now: float | None = None) -> int: +def prune_cache(now: float | None = None, *, name: str = "consult_full_data_cache") -> int: """Drop cache files last written more than two days ago.""" cutoff = (time.time() if now is None else now) - KEEP_SECONDS removed = 0 - folder = cache_dir() + folder = cache_dir(name) for path in folder.glob("*.json"): try: if path.stat().st_mtime < cutoff: @@ -147,15 +147,21 @@ def get_or_build( builder: PacketBuilder | None = None, *, clock: Callable[[], float] | None = None, + cache_name: str = "consult_full_data_cache", + json_default: Callable[..., Any] | None = None, ) -> dict[str, Any]: - """Return the packet. Concurrent callers for one key share one build.""" - prune_cache() + """Return the packet. Concurrent callers for one key share one build. + + ``cache_name`` selects the scratch subdirectory. The consult card keeps the + default. A non-default name does not change the consult key or its files. + """ + prune_cache(name=cache_name) now = clock or time.perf_counter started = now() - path = _cache_path(identity) + path = _cache_path(identity, cache_name) if path.is_file(): return _read_cached(path, started, now) - key = cache_key(identity) + key = f"{cache_name}\n{cache_key(identity)}" with _flights_guard: flight = _flights.get(key) owner = flight is None @@ -177,7 +183,10 @@ def get_or_build( builder = build_full_data_packet packet = builder(identity) temporary = path.with_suffix(".json.tmp") - temporary.write_text(json.dumps(packet, ensure_ascii=False, separators=(",", ":")), encoding="utf-8") + dump_kwargs: dict[str, Any] = {"ensure_ascii": False, "separators": (",", ":")} + if json_default is not None: + dump_kwargs["default"] = json_default + temporary.write_text(json.dumps(packet, **dump_kwargs), encoding="utf-8") temporary.replace(path) elapsed = now() - started return { diff --git a/scripts/pl9_language_terms.py b/scripts/pl9_language_terms.py index 4e90b683..92c58144 100644 --- a/scripts/pl9_language_terms.py +++ b/scripts/pl9_language_terms.py @@ -7,9 +7,15 @@ It only changes reader-facing labels; it does not alter calculation payloads. """ from __future__ import annotations +import functools import re from typing import Any +try: + from scripts.report_perf_flags import report_render_fast_enabled +except ModuleNotFoundError: # pragma: no cover - direct scripts/ execution + from report_perf_flags import report_render_fast_enabled + # Bilingual planet tokens from yoga combination text, longest first. # Dignity is glued on with no space ("木星Jupiterexalted"), so the long form @@ -131,6 +137,11 @@ def translate_zh_glued_phrases(text: str) -> str: return text +@functools.lru_cache(maxsize=512) +def _natural_language_term_pattern(raw: str, plural: str) -> re.Pattern: + return re.compile(rf"(? str: """Replace one glossary term in prose only. @@ -141,7 +152,14 @@ def replace_natural_language_term(text: str, raw: str, public: str) -> str: if not text or not raw: return text plural = "s?" if re.search(r"[A-Za-z]$", raw) and not raw.endswith("s") else "" - pattern = re.compile(rf"(? str: ) +# Every AD/PD hit contains these literals. Patterns stay case-sensitive. +_ZH_AD_DENSITY_RE = re.compile( + r"(.+?)主大运中的(.+?)子运，会把(.+?)的自然主题带入当前阶段。" +) +_ZH_PD_DENSITY_RE = re.compile( + r"(.+?)子运中的(.+?)次子运，会在(.+?)背景下呈现(.+?)的短周期结果。" +) +_ZH_AD_ANCHORS = ("主大运中的", "子运，会把", "的自然主题带入当前阶段。") +_ZH_PD_ANCHORS = ("次子运，会在", "背景下呈现", "的短周期结果。") +_EN_MD_HEADING_RE = re.compile(r"^#{2,4}\s+(.+?\bMaha Dasha\b.*)$") +_EN_AD_DENSITY_RE = re.compile( + r"The Antar Dasha of (.+?) activates (.+?)'s natural themes inside the broader Maha Dasha of (.+?)\." +) +_EN_PD_DENSITY_RE = re.compile( + r"In the Pratyantar Dasha of (.+?) in the Antar Dasha of (.+?), (.+?)'s significations give the short-period result inside the (.+?) background\." +) +_EN_AD_ANCHORS = ( + "The Antar Dasha of ", + " activates ", + "'s natural themes inside the broader Maha Dasha of ", +) +_EN_PD_ANCHORS = ( + "In the Pratyantar Dasha of ", + " in the Antar Dasha of ", + "'s significations give the short-period result inside the ", + " background.", +) + + +def _line_has_anchors(line: str, anchors: tuple[str, ...]) -> bool: + return all(anchor in line for anchor in anchors) + + +def _thicken_zh_dasha_density_fast(markdown: str) -> str: + """Same AD/PD inserts as the legacy scan, without searching inert lines.""" + lines = markdown.splitlines() + out: list[str] = [] + for index, line in enumerate(lines): + out.append(line) + # Both patterns are case-sensitive and every hit contains these literals. + ad_possible = _line_has_anchors(line, _ZH_AD_ANCHORS) + pd_possible = _line_has_anchors(line, _ZH_PD_ANCHORS) + if not ad_possible and not pd_possible: + continue + lookahead = "\n".join(lines[index + 1:index + 6]) + if "这一子运不是单独结论" in lookahead or "这个次子运只承担短周期细化" in lookahead: + continue + if ad_possible: + ad_match = _ZH_AD_DENSITY_RE.search(line) + if ad_match: + md_lord, ad_lord, theme_lord = ad_match.groups() + out.extend([ + "", + ( + f"这一子运不是单独结论，而是把{theme_lord}的自然象征放进{md_lord}主大运的背景里筛选。" + f"阅读时先看{ad_lord}本身的落宫、守护宫、尊贵状态与星宿，再看它和主大运星之间是否互相支持。" + ), + "", + ( + "若同一宫位或同一行星在分盘、Ashtakavarga、年度盘和行运中重复出现，" + "该主题的可见度会提高；若这些层彼此冲突，则应把结果视为阶段性倾向，而不是孤立断语。" + ), + ]) + continue + if pd_possible: + pd_match = _ZH_PD_DENSITY_RE.search(line) + if pd_match: + ad_lord, pd_lord, md_lord, theme_lord = pd_match.groups() + out.extend([ + "", + ( + f"这个次子运只承担短周期细化：{pd_lord}会把{theme_lord}的具体征象带入{ad_lord}子运，" + f"并受{md_lord}主背景限制。" + ), + "", + ( + "因此这里优先用于判断事情推进的节奏、触发点和轻重缓急；实际领域仍需回到本命落宫、" + "分盘重复、年度盘和当前行运共同核对。" + ), + ]) + continue + return "\n".join(out) + + def _thicken_zh_dasha_density(markdown: str) -> str: """Restore reader-facing density for translated AD/PD prose. This adds reading instructions tied to already-present fields. It does not introduce new predictions or alter dates. """ + if report_render_fast_enabled(): + return _thicken_zh_dasha_density_fast(markdown) lines = markdown.splitlines() out: list[str] = [] for index, line in enumerate(lines): @@ -941,9 +1045,64 @@ def normalize_en_markdown_terms(markdown: str) -> str: return apply_en_residual_han(_thicken_en_dasha_density(text)) +def _thicken_en_dasha_density_fast(markdown: str) -> str: + """Same English AD inserts as the legacy scan, without searching inert lines.""" + lines = markdown.splitlines() + out: list[str] = [] + current_md = None + ad_context_added = False + for index, line in enumerate(lines): + md_heading = _EN_MD_HEADING_RE.match(line) + if md_heading: + next_md = md_heading.group(1) + if next_md != current_md: + current_md = next_md + ad_context_added = False + out.append(line) + # Both patterns are case-sensitive and every hit contains these literals. + ad_possible = _line_has_anchors(line, _EN_AD_ANCHORS) + pd_possible = _line_has_anchors(line, _EN_PD_ANCHORS) + if not ad_possible and not pd_possible: + continue + lookahead = "\n".join(lines[index + 1:index + 6]) + if ( + "This Antar Dasha is not read as a standalone verdict" in lookahead + or "This Pratyantar Dasha is the short-cycle refinement" in lookahead + ): + continue + if ad_possible: + ad_match = _EN_AD_DENSITY_RE.search(line) + if ad_match and not ad_context_added: + ad_lord, theme_lord, md_lord = ad_match.groups() + out.extend([ + "", + ( + f"This Antar Dasha is not read as a standalone verdict. It places {theme_lord}'s " + f"natural significations inside the background of the {md_lord} Maha Dasha, then " + f"filters them through {ad_lord}'s house placement, house ownership, dignity, " + "nakshatra lord, conjunctions, and divisional-chart repetition." + ), + "", + ( + "When the same planet or house repeats through divisional charts, Ashtakavarga, " + "annual factors, and transits, the theme becomes more visible. When those layers " + "conflict, the result should be read as a staged tendency rather than an isolated statement." + ), + ]) + ad_context_added = True + continue + if pd_possible and _EN_PD_DENSITY_RE.search(line): + # The row already contains the concrete PD lord and timing context. + # Keep it, but do not append the same methodology paragraph per node. + continue + return "\n".join(out) + + def _thicken_en_dasha_density(markdown: str) -> str: """Add chapter-level AD context without repeating boilerplate on every PD row.""" + if report_render_fast_enabled(): + return _thicken_en_dasha_density_fast(markdown) lines = markdown.splitlines() out: list[str] = [] current_md = None diff --git a/scripts/professional_report_reference.py b/scripts/professional_report_reference.py index 5167eaca..d4ccc8b8 100644 --- a/scripts/professional_report_reference.py +++ b/scripts/professional_report_reference.py @@ -8,6 +8,9 @@ calculation, and delegates packet assembly/rendering to ``jyotish_engine``. from __future__ import annotations import copy +import os +import sys +import time from datetime import datetime from importlib import import_module from types import SimpleNamespace @@ -32,6 +35,11 @@ except ModuleNotFoundError: # pragma: no cover - direct scripts/ execution path from reader_dasha_applicability import render_dasha_applicability from report_density_packet import report_density_packet +try: + from scripts.report_perf_flags import report_reference_cache_enabled +except ModuleNotFoundError: # pragma: no cover - direct scripts/ execution path + from report_perf_flags import report_reference_cache_enabled + class ProfessionalReportReferenceInputError(ValueError): """The professional-reference request is outside the public contract.""" @@ -187,18 +195,49 @@ def _export_args(birth: dict[str, Any]) -> SimpleNamespace: return SimpleNamespace(**payload) -def build_professional_report_reference(handler, body: dict[str, Any], *, engine=None) -> dict[str, Any]: - """Build one JSON or Markdown response from one reused full-reading result.""" - if not isinstance(body, dict): - raise ProfessionalReportReferenceInputError("request body must be an object") - output_format = _normalize_format(body.get("format")) - edition = _normalize_edition(body.get("edition")) - packs = _normalize_packs(body.get("packs")) - languages = _normalize_languages(body.get("languages")) - if languages and (output_format != "markdown" or edition != FULL_DATA_EDITION): - raise ProfessionalReportReferenceInputError("languages requires format markdown and edition full_data") +def _load_report_cache(): + try: + from scripts import consult_full_data_cache as cache + except ModuleNotFoundError: # pragma: no cover - direct scripts/ execution path + import consult_full_data_cache as cache + return cache - birth = handler._high_rigor_birth_payload(body) + +def _report_cache_sha() -> str: + return os.environ.get("GITHUB_SHA") or "" + + +def _report_cache_identity( + birth: dict[str, Any], + output_format: str, + edition: str, + packs: list[str], + languages: list[str], + include_fact_tables: bool, + sha: str, +) -> dict[str, Any]: + return { + "birth": birth, + "format": output_format, + "edition": edition, + "packs": list(packs), + "languages": list(languages), + "include_fact_tables": include_fact_tables, + "report_version": REPORT_VERSION_FULL, + "github_sha": sha, + } + + +def _assemble_professional_report_reference( + handler, + birth: dict[str, Any], + output_format: str, + edition: str, + packs: list[str], + languages: list[str], + include_fact_tables: bool, + engine, +) -> dict[str, Any]: full_reading = handler._compute_full_reading_for_thematic(birth) resolved_engine = engine or _load_engine() try: @@ -217,7 +256,7 @@ def build_professional_report_reference(handler, body: dict[str, Any], *, engine if edition == FULL_DATA_EDITION: # English renders from its own deep copy first, so nothing the Chinese # renderer does to the packet can reach it (and vice versa). - english = _english_edition(packet, body.get("include_fact_tables") is True) if "en" in languages else {} + english = _english_edition(packet, include_fact_tables) if "en" in languages else {} markdown = render_full_data_markdown(packet) else: markdown = resolved_engine.render_pl9_markdown(packet) @@ -228,7 +267,7 @@ def build_professional_report_reference(handler, body: dict[str, Any], *, engine "markdown": markdown, **({"fact_table_packet": report_density_packet(packet), "reader_dasha_applicability": clean_reader_appendix_markdown(render_dasha_applicability(packet))} - if body.get("include_fact_tables") is True else {}), + if include_fact_tables else {}), **(english if edition == FULL_DATA_EDITION and "en" in languages else {}), } return { @@ -236,3 +275,49 @@ def build_professional_report_reference(handler, body: dict[str, Any], *, engine "edition": edition, "report": packet, } + + +def build_professional_report_reference(handler, body: dict[str, Any], *, engine=None) -> dict[str, Any]: + """Build one JSON or Markdown response from one reused full-reading result.""" + if not isinstance(body, dict): + raise ProfessionalReportReferenceInputError("request body must be an object") + output_format = _normalize_format(body.get("format")) + edition = _normalize_edition(body.get("edition")) + packs = _normalize_packs(body.get("packs")) + languages = _normalize_languages(body.get("languages")) + if languages and (output_format != "markdown" or edition != FULL_DATA_EDITION): + raise ProfessionalReportReferenceInputError("languages requires format markdown and edition full_data") + + birth = handler._high_rigor_birth_payload(body) + include_fact_tables = body.get("include_fact_tables") is True + sha = _report_cache_sha() + use_cache = ( + report_reference_cache_enabled() + and output_format == "markdown" + and edition == FULL_DATA_EDITION + and sha not in ("", "unknown") + ) + if not use_cache: + return _assemble_professional_report_reference( + handler, birth, output_format, edition, packs, languages, include_fact_tables, engine, + ) + identity = _report_cache_identity( + birth, output_format, edition, packs, languages, include_fact_tables, sha, + ) + started = time.perf_counter() + + def builder(_identity: dict[str, Any]) -> dict[str, Any]: + return _assemble_professional_report_reference( + handler, birth, output_format, edition, packs, languages, include_fact_tables, engine, + ) + + cached = _load_report_cache().get_or_build( + identity, + builder, + cache_name="report_reference_cache", + json_default=str, + ) + elapsed_ms = int(round((time.perf_counter() - started) * 1000)) + state = "hit" if cached["hit"] else "miss" + print(f"[report-reference-cache] {state} elapsed_ms={elapsed_ms}", file=sys.stderr) + return cached["packet"] diff --git a/scripts/report_perf_flags.py b/scripts/report_perf_flags.py new file mode 100644 index 00000000..2278bf7d --- /dev/null +++ b/scripts/report_perf_flags.py @@ -0,0 +1,40 @@ +"""Call-time switches for long-report rendering and same-chart reuse. + +Values are read on every call so one process can measure the old path and the +new path back to back. The old path stays in place when a switch is off. +""" + +from __future__ import annotations + +import os + +# Masked Windows A/B on 2026-10-14. Render shortcuts are byte-identical and use +# less memory, but the end-to-end drop was 20–23%, short of the 35% gate, so +# the fast path stays off. Cache hits matched the miss bytes and returned in +# about 0.06s, under the 1.5s gate. A missing or unknown GITHUB_SHA still +# disables the cache at the call site. +RENDER_FAST_DEFAULT = False +REFERENCE_CACHE_DEFAULT = True + +_ON = {"1", "true", "yes", "on"} + + +def _enabled(name: str, default: bool) -> bool: + raw = os.environ.get(name) + if raw is None or not raw.strip(): + return default + return raw.strip().lower() in _ON + + +def report_render_fast_enabled() -> bool: + """True when the equivalent render and hash shortcuts may run.""" + return _enabled("JYOTISH_REPORT_RENDER_FAST", RENDER_FAST_DEFAULT) + + +def report_reference_cache_enabled() -> bool: + """True when a full_data markdown response may be reused from disk. + + A missing or ``unknown`` ``GITHUB_SHA`` still disables the cache at the + call site. This function only reads the switch. + """ + return _enabled("JYOTISH_REPORT_REFERENCE_CACHE", REFERENCE_CACHE_DEFAULT) diff --git a/scripts/research/report_reference_perf_ab.py b/scripts/research/report_reference_perf_ab.py new file mode 100644 index 00000000..fb2210e2 --- /dev/null +++ b/scripts/research/report_reference_perf_ab.py @@ -0,0 +1,428 @@ +"""A/B timing for the professional long report. + +A is both switches off. B is both on. Same process, fictional charts only. +A field sha mismatch exits 1 and prints a short diff context, not the report. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import platform +import sys +import tempfile +import time +import tracemalloc +from pathlib import Path +from types import SimpleNamespace + + +ROOT = Path(__file__).resolve().parents[2] +for _folder in (ROOT / "scripts", ROOT / "jyotish_vedic", ROOT): + _text = str(_folder) + if _text not in sys.path: + sys.path.insert(0, _text) + +PRODUCTION_BODY = { + "format": "markdown", + "edition": "full_data", + "include_fact_tables": True, + "languages": ["zh", "en"], +} +SECTION2_CHART = { + "year": 1990, + "month": 1, + "day": 1, + "hour": 12, + "minute": 0, + "second": 0, + "lat": 28.6139, + "lon": 77.2090, + "tz": 5.5, + "node_mode": "mean", + "ayanamsa": "lahiri", + "today": "2026-10-13", + "transit_date": "2026-10-13", + "target_year": 2026, + "age": 36, + "birth_time_accuracy": "confirmed", +} +FIELDS = ( + "response", + "markdown", + "markdown_en", + "fact_table_packet", + "reader_dasha_applicability", +) + + +def _mask_pyjhora() -> str: + root = Path(tempfile.mkdtemp(prefix="nojhora-")) + package = root / "jhora" + package.mkdir() + (package / "__init__.py").write_text('raise ImportError("masked")\n', encoding="utf-8") + sys.path.insert(0, str(root)) + prior = os.environ.get("PYTHONPATH", "") + os.environ["PYTHONPATH"] = str(root) + (os.pathsep + prior if prior else "") + for name in list(sys.modules): + if name == "jhora" or name.startswith("jhora."): + del sys.modules[name] + return str(root) + + +def _sha(text: str) -> str: + return hashlib.sha256(text.encode("utf-8")).hexdigest() + + +def _canonical(value) -> str: + if isinstance(value, str): + return value + return json.dumps(value, ensure_ascii=False, default=str) + + +def _field_texts(result: dict) -> dict[str, str]: + whole = json.dumps(result, ensure_ascii=False, default=str) + return { + "response": whole, + "markdown": _canonical(result.get("markdown", "")), + "markdown_en": _canonical(result.get("markdown_en", "")), + "fact_table_packet": _canonical(result.get("fact_table_packet")), + "reader_dasha_applicability": _canonical(result.get("reader_dasha_applicability", "")), + } + + +def _field_shas(result: dict) -> dict[str, str]: + return {name: _sha(text) for name, text in _field_texts(result).items()} + + +def _diff_context(left: str, right: str) -> str: + limit = min(len(left), len(right)) + index = next((i for i in range(limit) if left[i] != right[i]), limit) + start = max(0, index - 40) + stop = index + 40 + return ( + f"at {index} len {len(left)} vs {len(right)} " + f"A={left[start:stop]!r} B={right[start:stop]!r}" + ) + + +def _median(values: list[float]) -> float: + ordered = sorted(values) + return ordered[len(ordered) // 2] + + +class _Bucket: + def __init__(self) -> None: + self.cmd = 0.0 + self.packet = 0.0 + self.zh = 0.0 + self.en = 0.0 + self.hash = 0.0 + + def reset(self) -> None: + self.cmd = self.packet = self.zh = self.en = self.hash = 0.0 + + def as_dict(self) -> dict[str, float]: + return { + "cmd_full_reading_s": round(self.cmd, 4), + "packet_s": round(self.packet, 4), + "zh_render_s": round(self.zh, 4), + "en_render_s": round(self.en, 4), + "hash_s": round(self.hash, 4), + } + + +def _install_timers(bucket: _Bucket) -> None: + import scripts.calculation_profile_contract as contract + import scripts.full_report_quality_gate as gate + import scripts.jyotish_engine as engine + import scripts.professional_report_reference as reference + + orig_packet = engine.build_professional_report_reference_packet + + def timed_packet(*args, **kwargs): + started = time.perf_counter() + try: + return orig_packet(*args, **kwargs) + finally: + bucket.packet += time.perf_counter() - started + + engine.build_professional_report_reference_packet = timed_packet + orig_render = reference.render_full_data_markdown + + def timed_render(packet, *args, **kwargs): + started = time.perf_counter() + try: + return orig_render(packet, *args, **kwargs) + finally: + elapsed = time.perf_counter() - started + if isinstance(packet, dict) and packet.get("report_language") == "en": + bucket.en += elapsed + else: + bucket.zh += elapsed + + reference.render_full_data_markdown = timed_render + orig_hash = contract.hash_bound_result + + def timed_hash(*args, **kwargs): + started = time.perf_counter() + try: + return orig_hash(*args, **kwargs) + finally: + bucket.hash += time.perf_counter() - started + + contract.hash_bound_result = timed_hash + gate.hash_bound_result = timed_hash + + +def _set_flags(fast: bool, cache: bool) -> None: + os.environ["JYOTISH_REPORT_RENDER_FAST"] = "1" if fast else "0" + os.environ["JYOTISH_REPORT_REFERENCE_CACHE"] = "1" if cache else "0" + + +class _Handler: + def __init__(self, birth: dict, bucket: _Bucket) -> None: + self.birth = birth + self.bucket = bucket + + def _high_rigor_birth_payload(self, _body: dict) -> dict: + return dict(self.birth) + + def _compute_full_reading_for_thematic(self, birth: dict): + from scripts.jyotish_engine import cmd_full_reading + + started = time.perf_counter() + try: + return cmd_full_reading(SimpleNamespace(**birth)) + finally: + self.bucket.cmd += time.perf_counter() - started + + +def _run_once(birth: dict, body: dict, bucket: _Bucket, handler=None) -> dict: + from scripts.professional_report_reference import build_professional_report_reference + + bucket.reset() + tracemalloc.start() + started = time.perf_counter() + result = build_professional_report_reference(handler or _Handler(birth, bucket), body) + elapsed = time.perf_counter() - started + _current, peak = tracemalloc.get_traced_memory() + tracemalloc.stop() + texts = _field_texts(result) + return { + "elapsed_s": round(elapsed, 4), + "peak_bytes": peak, + "response_bytes": len(texts["response"].encode("utf-8")), + "segments": bucket.as_dict(), + "sha": _field_shas(result), + "_texts": texts, + } + + +def _repeat(birth: dict, body: dict, bucket: _Bucket, times: int) -> list[dict]: + return [_run_once(birth, body, bucket) for _ in range(times)] + + +def _public_row(row: dict) -> dict: + return {key: value for key, value in row.items() if key != "_texts"} + + +def _compare(label: str, left: dict, right: dict, problems: list[str]) -> None: + left_texts = left["_texts"] + right_texts = right["_texts"] + for name in FIELDS: + if left["sha"][name] != right["sha"][name]: + problems.append(f"{label} {name} {_diff_context(left_texts[name], right_texts[name])}") + + +def _freeze_clocks(): + """Pin datetime.now / utcnow in every already imported module. + + ``format=json`` and the reference edition embed wall-clock fields. Freezing + only the engine module leaves other imports on the real clock. + """ + import datetime as dt_mod + + real = dt_mod.datetime + fixed_utc = real(2026, 10, 13, 0, 0, 0) + fixed_local = real(2026, 10, 13, 8, 0, 0) + + class Frozen(real): + @classmethod + def now(cls, tz=None): + if tz is not None: + return fixed_utc.replace(tzinfo=tz) + return fixed_local + + @classmethod + def utcnow(cls): + return fixed_utc + + saved = [] + for module in list(sys.modules.values()): + if module is None: + continue + current = getattr(module, "datetime", None) + if isinstance(current, type) and current.__name__ == "datetime": + saved.append((module, current)) + module.datetime = Frozen + return saved + + +def _restore_clocks(saved) -> None: + for module, current in saved: + module.datetime = current + + +def _strip_generated_at(value): + if isinstance(value, dict): + return {key: _strip_generated_at(item) for key, item in value.items() if key != "generated_at"} + if isinstance(value, list): + return [_strip_generated_at(item) for item in value] + return value + + +def _sha_pair(birth: dict, body: dict, bucket: _Bucket) -> dict: + """Compare one frozen clock on one reused full reading. + + A second ``cmd_full_reading`` appends onto shared trait lists, so two live + engine calls are not a byte test of the render switch. + """ + import copy + + from scripts.jyotish_engine import cmd_full_reading + + _set_flags(False, False) + reading = cmd_full_reading(SimpleNamespace(**birth)) + + class _Reuse: + def _high_rigor_birth_payload(self, _body: dict) -> dict: + return dict(birth) + + def _compute_full_reading_for_thematic(self, _birth: dict): + return copy.deepcopy(reading) + + _set_flags(False, False) + left = _run_once(birth, body, bucket, _Reuse()) + _set_flags(True, False) + right = _run_once(birth, body, bucket, _Reuse()) + problems = [] + _compare("side", left, right, problems) + compared = "full" + if problems: + left_stripped = _sha(_canonical(_strip_generated_at(json.loads(left["_texts"]["response"])))) + right_stripped = _sha(_canonical(_strip_generated_at(json.loads(right["_texts"]["response"])))) + if left_stripped == right_stripped: + problems.clear() + compared = "without_generated_at" + else: + compared = "mismatch" + return { + "compared": compared, + "sha_a": left["sha"]["response"], + "sha_b": right["sha"]["response"], + "problems": problems, + } + + +def _charts() -> dict[str, dict]: + from tests.test_report_english_edition import CASES + + return { + "day": {key: value for key, value in CASES["day"].items() if key != "name"}, + "night": {key: value for key, value in CASES["night"].items() if key != "name"}, + "south": {key: value for key, value in CASES["south"].items() if key != "name"}, + "section2": dict(SECTION2_CHART), + } + + +def _clear_report_cache() -> None: + folder = ROOT / "scratch" / "local" / "report_reference_cache" + if not folder.is_dir(): + return + for path in folder.glob("*.json"): + path.unlink(missing_ok=True) + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--mask-pyjhora", action="store_true") + parser.add_argument("--only-a", action="store_true") + parser.add_argument("--side-only", action="store_true") + args = parser.parse_args() + mask_path = _mask_pyjhora() if args.mask_pyjhora else "" + bucket = _Bucket() + _install_timers(bucket) + os.environ.pop("GITHUB_SHA", None) + report: dict = { + "python": platform.python_version(), + "platform": platform.platform(), + "mask_pyjhora": bool(args.mask_pyjhora), + "mask_path": mask_path, + "only_a": args.only_a, + "charts": {}, + } + problems: list[str] = [] + chart_items = () if args.side_only else _charts().items() + for label, birth in chart_items: + _set_flags(False, False) + arm_a = _repeat(birth, PRODUCTION_BODY, bucket, 3) + entry: dict = {"A": [_public_row(row) for row in arm_a]} + if not args.only_a: + _set_flags(True, False) + arm_b = _repeat(birth, PRODUCTION_BODY, bucket, 3) + for row in arm_b: + _compare(f"{label} B-miss", arm_a[0], row, problems) + os.environ["GITHUB_SHA"] = "report-reference-perf-ab" + _set_flags(True, True) + _clear_report_cache() + miss = _run_once(birth, PRODUCTION_BODY, bucket) + _compare(f"{label} cache-miss", arm_a[0], miss, problems) + hits = _repeat(birth, PRODUCTION_BODY, bucket, 3) + for row in hits: + _compare(f"{label} cache-hit", miss, row, problems) + entry["B_miss"] = [_public_row(row) for row in arm_b] + entry["cache_miss"] = _public_row(miss) + entry["B_hit"] = [_public_row(row) for row in hits] + os.environ.pop("GITHUB_SHA", None) + report["charts"][label] = entry + print(f"{label} done", file=sys.stderr) + if not args.only_a: + saved_clocks = _freeze_clocks() + try: + sample = _charts()["section2"] + report["format_json"] = _sha_pair(sample, {"format": "json", "edition": "full_data"}, bucket) + report["edition_reference"] = _sha_pair( + sample, {"format": "markdown", "edition": "reference"}, bucket, + ) + finally: + _restore_clocks(saved_clocks) + problems.extend(report["format_json"]["problems"]) + problems.extend(report["edition_reference"]["problems"]) + _set_flags(False, False) + from scripts.calculation_profile_contract import hash_bound_result + from scripts.jyotish_engine import build_professional_report_reference_packet, cmd_full_reading + from scripts.professional_report_reference import _export_args + + reading = cmd_full_reading(SimpleNamespace(**sample)) + packet = build_professional_report_reference_packet(reading, _export_args(sample), []) + input_hash = packet["calculation_profile"]["input_hash"] + + def _time_hash(fast: bool) -> float: + _set_flags(fast, False) + started = time.perf_counter() + hash_bound_result(packet, input_hash) + return time.perf_counter() - started + + off = [_time_hash(False) for _ in range(3)] + on = [_time_hash(True) for _ in range(3)] + report["hash_once_s"] = {"off": [round(item, 4) for item in off], "on": [round(item, 4) for item in on]} + report["problems"] = problems + print(json.dumps(report, ensure_ascii=False)) + return 1 if problems else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_report_reference_perf.py b/tests/test_report_reference_perf.py new file mode 100644 index 00000000..752b33a8 --- /dev/null +++ b/tests/test_report_reference_perf.py @@ -0,0 +1,374 @@ +"""Byte-identical report shortcuts and the full_data markdown cache.""" + +from __future__ import annotations + +import copy +import hashlib +import json +import os +import re +import threading +import time +from typing import Any + +import pytest + +from scripts.calculation_profile_contract import hash_bound_result +from scripts.pl9_full_data_export import render_full_data_markdown +from scripts.pl9_language_terms import _thicken_zh_dasha_density, replace_natural_language_term +from scripts.report_perf_flags import ( + REFERENCE_CACHE_DEFAULT, + RENDER_FAST_DEFAULT, + report_reference_cache_enabled, + report_render_fast_enabled, +) +from scripts.professional_report_reference import build_professional_report_reference +from tests.test_report_english_edition import CASES, _packet + + +def _legacy_drop_key_recursive(payload: Any, key_name: str) -> Any: + if isinstance(payload, dict): + return { + key: _legacy_drop_key_recursive(value, key_name) + for key, value in payload.items() + if key != key_name + } + if isinstance(payload, list): + return [_legacy_drop_key_recursive(item, key_name) for item in payload] + return payload + + +def _legacy_drop_dotted_path(payload: Any, path: tuple[str, ...]) -> Any: + if len(path) == 2 and path[0] == "**": + return _legacy_drop_key_recursive(payload, path[1]) + if not path or not isinstance(payload, dict) or path[0] not in payload: + return payload + cloned = dict(payload) + key, *rest = path + if not rest: + cloned.pop(key, None) + return cloned + cloned[key] = _legacy_drop_dotted_path(payload[key], tuple(rest)) + return cloned + + +def _legacy_canonicalize(value: Any) -> Any: + if isinstance(value, dict): + normalized: dict[str, Any] = {} + for key, child in value.items(): + normalized_key = key if isinstance(key, str) else f"__{type(key).__name__}__:{key}" + normalized[normalized_key] = _legacy_canonicalize(child) + return normalized + if isinstance(value, (list, tuple)): + return [_legacy_canonicalize(item) for item in value] + return value + + +def _legacy_payload(result: dict, scope: dict) -> dict: + excluded_top = {"result_hash", "result_binding", "calculation_profile", "calculation_profile_id"} | { + str(key) for key in (scope.get("excluded_top_keys") or ()) + } + payload = {key: value for key, value in result.items() if key not in excluded_top} + for raw_path in scope.get("excluded_paths") or (): + parts = tuple(part for part in str(raw_path).split(".") if part) + if parts: + payload = _legacy_drop_dotted_path(payload, parts) + return payload + + +def _legacy_hash(result: dict, input_hash: str, scope: dict) -> str: + encoded = json.dumps( + _legacy_canonicalize({"input_hash": input_hash, "result": _legacy_payload(result, scope)}), + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + default=str, + ) + return hashlib.sha256(encoded.encode("utf-8")).hexdigest() + + +@pytest.fixture(scope="module") +def day_views(): + """One real day packet. ``delivered`` is the bound object; render uses a copy.""" + from scripts.jyotish_engine import build_professional_report_reference_packet, cmd_full_reading + from scripts.professional_report_reference import _export_args + + os.environ["JYOTISH_REPORT_RENDER_FAST"] = "0" + os.environ["JYOTISH_REPORT_REFERENCE_CACHE"] = "0" + try: + case = CASES["day"] + reading = cmd_full_reading(type("Args", (), dict(case))()) + delivered = build_professional_report_reference_packet(reading, _export_args(dict(case)), []) + packet = dict(delivered) + packet["report_version"] = "pl9_personal_long_report.v3" + zh = render_full_data_markdown(copy.deepcopy(packet)) + english = copy.deepcopy(packet) + english["report_language"] = "en" + en = render_full_data_markdown(english) + finally: + os.environ.pop("JYOTISH_REPORT_RENDER_FAST", None) + os.environ.pop("JYOTISH_REPORT_REFERENCE_CACHE", None) + return delivered, packet, zh, en + + +def test_flags_follow_the_environment_at_call_time(monkeypatch) -> None: + monkeypatch.delenv("JYOTISH_REPORT_RENDER_FAST", raising=False) + monkeypatch.delenv("JYOTISH_REPORT_REFERENCE_CACHE", raising=False) + assert report_render_fast_enabled() is RENDER_FAST_DEFAULT + assert report_reference_cache_enabled() is REFERENCE_CACHE_DEFAULT + monkeypatch.setenv("JYOTISH_REPORT_RENDER_FAST", "0") + monkeypatch.setenv("JYOTISH_REPORT_REFERENCE_CACHE", "0") + assert report_render_fast_enabled() is False + assert report_reference_cache_enabled() is False + monkeypatch.setenv("JYOTISH_REPORT_RENDER_FAST", "1") + monkeypatch.setenv("JYOTISH_REPORT_REFERENCE_CACHE", "1") + assert report_render_fast_enabled() is True + assert report_reference_cache_enabled() is True + + +def test_fast_render_matches_the_legacy_bytes(day_views, monkeypatch) -> None: + _delivered, packet, zh_off, en_off = day_views + monkeypatch.setenv("JYOTISH_REPORT_RENDER_FAST", "1") + monkeypatch.setenv("JYOTISH_REPORT_REFERENCE_CACHE", "0") + zh_on = render_full_data_markdown(copy.deepcopy(packet)) + english = copy.deepcopy(packet) + english["report_language"] = "en" + en_on = render_full_data_markdown(english) + assert zh_on == zh_off + assert en_on == en_off + + +def _both(function, monkeypatch, *args): + monkeypatch.setenv("JYOTISH_REPORT_RENDER_FAST", "0") + off = function(*args) + monkeypatch.setenv("JYOTISH_REPORT_RENDER_FAST", "1") + on = function(*args) + assert on == off + return off + + +def test_dasha_density_samples_come_from_the_real_render(day_views, monkeypatch) -> None: + _delivered, _packet_view, zh, _en = day_views + lines = [line for line in zh.splitlines() if line.strip()] + # This fictional chart never emits the density sentence. Fill the existing + # sanitizer sentence with lords that do appear in the render. + lords = [token for token in ("木星", "北交点", "土星", "金星") if token in zh] + assert len(lords) >= 2 + hit = f"{lords[0]}主大运中的{lords[1]}子运，会把{lords[0]}的自然主题带入当前阶段。" + miss = next(line for line in lines if "子运" not in line and "`" not in line) + quoted = next(line for line in lines if "`" in line) + hit_out = _both(_thicken_zh_dasha_density, monkeypatch, hit + "\n") + assert "这一子运不是单独结论" in hit_out or "这个次子运只承担短周期细化" in hit_out + miss_out = _both(_thicken_zh_dasha_density, monkeypatch, miss + "\n") + assert miss_out == miss + quoted_out = _both(_thicken_zh_dasha_density, monkeypatch, quoted + "\n") + assert "`" in quoted_out + assert quoted in quoted_out + + +def test_term_samples_come_from_the_real_render(day_views, monkeypatch) -> None: + _delivered, _packet_view, zh, _en = day_views + lines = zh.splitlines() + raw = "Ashtakavarga" + hit = next(line for line in lines if raw in re.sub(r"`[^`]*`", "", line)) + miss = next(line for line in lines if line.strip() and raw not in line) + span = next(span for span in re.findall(r"`[^`]*`", zh) if raw not in span) + hit_out = _both(replace_natural_language_term, monkeypatch, hit, raw, "探针词") + assert "探针词" in hit_out or hit_out == hit + if re.search(rf"(? None: + packet, _render_packet, _zh, _en = day_views + scope = packet["result_binding"]["binding_scope"] + input_hash = packet["calculation_profile"]["input_hash"] + monkeypatch.setenv("JYOTISH_REPORT_RENDER_FAST", "1") + assert hash_bound_result(packet, input_hash, scope) == _legacy_hash(packet, input_hash, scope) + assert hash_bound_result(packet, input_hash, scope) == packet["result_hash"] + variant = copy.deepcopy(packet) + variant["probe_layer"] = { + "elapsed_seconds": 1.5, + "called_at": "2001-01-01T00:00:00Z", + "cache_created_at": "created", + "cache_expires_at": "expires", + "nested": {"elapsed_seconds": 9, (1, 2): {"called_at": "inside"}}, + } + summary = variant.setdefault("ai_and_audit", {}).setdefault("summary", {}) + summary["elapsed_seconds"] = 99 + summary["kept"] = "yes" + assert hash_bound_result(variant, input_hash, scope) == _legacy_hash(variant, input_hash, scope) + monkeypatch.setenv("JYOTISH_REPORT_RENDER_FAST", "0") + assert hash_bound_result(packet, input_hash) == packet["result_hash"] + + +class _CountingHandler: + def __init__(self, birth: dict, packet: dict) -> None: + self.birth = birth + self.packet = packet + self.calls = 0 + self.fail_once = False + self.delay = 0.0 + self.lock = threading.Lock() + + def _high_rigor_birth_payload(self, body: dict) -> dict: + payload = dict(self.birth) + if body.get("today"): + payload["today"] = body["today"] + payload["candidate_range"] = body.get("candidate_range") + return payload + + def _compute_full_reading_for_thematic(self, _birth: dict) -> dict: + with self.lock: + self.calls += 1 + call = self.calls + if self.delay: + time.sleep(self.delay) + if self.fail_once and call == 1: + raise RuntimeError("report_build_failed") + return {"stub": True} + + +def _body(**extra) -> dict: + body = { + "format": "markdown", + "edition": "full_data", + "include_fact_tables": True, + "languages": ["zh", "en"], + "today": "2026-04-07", + } + body.update(extra) + return body + + +@pytest.fixture +def cached_report(monkeypatch, tmp_path, day_views): + import scripts.consult_full_data_cache as cache + import scripts.professional_report_reference as reference + + def _dir(name: str = "consult_full_data_cache"): + path = tmp_path / name + path.mkdir(parents=True, exist_ok=True) + return path + + monkeypatch.setattr(cache, "cache_dir", _dir) + monkeypatch.setattr(reference, "render_full_data_markdown", lambda packet: "en-body" if packet.get("report_language") == "en" else "zh-body") + monkeypatch.setattr(reference, "report_density_packet", lambda _packet: {"facts": 1}) + monkeypatch.setattr(reference, "render_dasha_applicability", lambda _packet, language="zh": f"app-{language}") + monkeypatch.setattr(reference, "clean_reader_appendix_markdown", lambda text: text) + packet = day_views[1] + birth = {key: value for key, value in CASES["day"].items() if key != "name"} + handler = _CountingHandler(birth, packet) + + class _Engine: + def build_professional_report_reference_packet(self, _reading, _args, _packs): + return copy.deepcopy(packet) + + def render_pl9_markdown(self, _packet): + return "reference" + + monkeypatch.setenv("JYOTISH_REPORT_REFERENCE_CACHE", "1") + monkeypatch.setenv("JYOTISH_REPORT_RENDER_FAST", "0") + monkeypatch.setenv("GITHUB_SHA", "report-cache-test") + return handler, _Engine(), _dir("report_reference_cache") + + +def _dumps(result: dict) -> str: + return json.dumps(result, ensure_ascii=False, default=str) + + +def test_second_call_reuses_the_cached_response(cached_report) -> None: + handler, engine, folder = cached_report + first = build_professional_report_reference(handler, _body(), engine=engine) + second = build_professional_report_reference(handler, _body(), engine=engine) + assert handler.calls == 1 + assert _dumps(first) == _dumps(second) + assert list(folder.glob("*.json")) + + +@pytest.mark.parametrize("extra", [ + {"today": "2026-05-01"}, + {"languages": ["zh"]}, + {"include_fact_tables": False}, + {"candidate_range": {"kind": "probe"}}, +]) +def test_changed_inputs_miss_the_report_cache(cached_report, extra) -> None: + handler, engine, _folder = cached_report + build_professional_report_reference(handler, _body(), engine=engine) + build_professional_report_reference(handler, _body(**extra), engine=engine) + assert handler.calls == 2 + + +def test_github_sha_is_part_of_the_report_cache_key(cached_report, monkeypatch) -> None: + handler, engine, _folder = cached_report + build_professional_report_reference(handler, _body(), engine=engine) + monkeypatch.setenv("GITHUB_SHA", "report-cache-other") + build_professional_report_reference(handler, _body(), engine=engine) + assert handler.calls == 2 + + +def test_json_format_and_disabled_cache_write_nothing(cached_report, monkeypatch) -> None: + handler, engine, folder = cached_report + for path in folder.glob("*.json"): + path.unlink() + build_professional_report_reference( + handler, + {"format": "json", "edition": "full_data", "today": "2026-04-07"}, + engine=engine, + ) + assert list(folder.glob("*.json")) == [] + monkeypatch.setenv("JYOTISH_REPORT_REFERENCE_CACHE", "0") + build_professional_report_reference(handler, _body(), engine=engine) + assert list(folder.glob("*.json")) == [] + monkeypatch.setenv("JYOTISH_REPORT_REFERENCE_CACHE", "1") + monkeypatch.setenv("GITHUB_SHA", "unknown") + build_professional_report_reference(handler, _body(), engine=engine) + assert list(folder.glob("*.json")) == [] + + +def test_concurrent_same_key_builds_once(cached_report) -> None: + handler, engine, _folder = cached_report + handler.delay = 0.2 + barrier = threading.Barrier(2) + errors = [] + + def run() -> None: + try: + barrier.wait() + build_professional_report_reference(handler, _body(), engine=engine) + except Exception as exc: # pragma: no cover - recorded for the assertion + errors.append(exc) + + threads = [threading.Thread(target=run) for _ in range(2)] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + assert errors == [] + assert handler.calls == 1 + + +def test_failed_build_is_not_cached(cached_report) -> None: + handler, engine, folder = cached_report + handler.fail_once = True + with pytest.raises(RuntimeError): + build_professional_report_reference(handler, _body(), engine=engine) + assert list(folder.glob("*.json")) == [] + build_professional_report_reference(handler, _body(), engine=engine) + assert handler.calls == 2 + build_professional_report_reference(handler, _body(), engine=engine) + assert handler.calls == 2 + + +def test_report_cache_directory_prunes_old_files(cached_report) -> None: + handler, engine, folder = cached_report + stale = folder / "stale-probe.json" + stale.write_text("{}", encoding="utf-8") + old = time.time() - (3 * 24 * 60 * 60) + os.utime(stale, (old, old)) + build_professional_report_reference(handler, _body(), engine=engine) + assert not stale.exists() From 3893783e4c63f2960714a8fd9a4bfd1cac14df4f Mon Sep 17 00:00:00 2001 From: jesse-ux Date: Wed, 14 Oct 2026 07:41:19 +0800 Subject: [PATCH 5/9] =?UTF-8?q?=E6=8A=8A=E6=95=B0=E6=8D=AE=E5=BA=93?= =?UTF-8?q?=E5=BE=80=E8=BF=94=E5=AF=B9=E7=85=A7=E8=84=9A=E6=9C=AC=E6=8C=AA?= =?UTF-8?q?=E5=87=BA=E9=95=9C=E5=83=8F=E4=BC=9A=E6=89=93=E5=8C=85=E7=9A=84?= =?UTF-8?q?=E7=9B=AE=E5=BD=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 脚本从 frontend/scripts 挪到 frontend/tests，不再从会被打包的目录引用测试夹具。业务代码没动，对照数字没重跑。tsc 通过，lint 0 个错误（125 条原有警告），镜像导入守卫 2 通过、0 失败。 Co-Authored-By: Grok (agent loop) --- docs/BUG_HISTORY.md | 10 +++++----- .../PROGRESS-node-runtime-resources-20261013.md | 16 +++++++++++++++- docs/tasks/README.md | 2 +- .../node-runtime-resources-ab-20261013.md | 2 +- .../bench-local-pg-roundtrips.ts | 4 ++-- 5 files changed, 24 insertions(+), 10 deletions(-) rename frontend/{scripts => tests}/bench-local-pg-roundtrips.ts (98%) diff --git a/docs/BUG_HISTORY.md b/docs/BUG_HISTORY.md index 2e84189b..3d96f10c 100644 --- a/docs/BUG_HISTORY.md +++ b/docs/BUG_HISTORY.md @@ -15304,17 +15304,17 @@ ## BUG-1140 | staging 镜像打包失败：`frontend/scripts` 下的研究脚本引用了 `frontend/tests` 的夹具 - 状态：resolved（修复已推 staging；以门禁 publish 通过与 `/api/health` 部署版本为最终证据） -- 首次发现 / 最近更新：2026-10-01 / 2026-10-01 +- 首次发现 / 最近更新：2026-10-01 / 2026-10-14 - 影响面：Gitea `backend-quality-gate` 的 publish 作业（`deploy/railway-web.Dockerfile` 构建阶段 `npm run build`）。validate 通过、publish 失败，未部署，staging 停在上一版。 - 用户现象：盘型口径（BUG-1115～1117、1131）推送后 staging 不更新。 - 触发条件：`frontend/scripts/rectification-segment-persisted-replay.ts`（盘型口径研究的 PG 持久回放 harness）`import … from "../tests/helpers/postgres-fixture.ts"`。web 镜像只复制 `frontend/src`、`frontend/scripts` 等，不复制 `frontend/tests`；`frontend/tsconfig.json` 的 include 为 `**/*.ts`，`next build` 类型检查覆盖 scripts，于是 `TS2307 Cannot find module '../tests/helpers/postgres-fixture.ts'`。 - 根因：本地与验收机都有 `frontend/tests`，`tsc --noEmit` 与 `next build` 均通过，门禁 validate 也在完整检出上跑，没有任何一步按镜像的复制清单构建。 - 修复：harness 移到 `frontend/tests/research/rectification-segment-persisted-replay.ts`（相对路径随之调整），Python driver `scripts/research/varga_resolution_persisted_replay.py` 指向新路径；新增 `frontend/tests/runtime-sources-no-test-imports.test.ts`：`src/` 与 `scripts/` 不得 import / `new URL` 引用 `tests/`。 - 验证：按 `railway-web.Dockerfile` 的 COPY 清单从提交 `git archive` 出构建目录，修复前 `npm run build` 报上述 TS2307、exit 1，修复后 exit 0、`/` Static；守卫测试 2/2，且对旧 harness 文本判定为违规；移动后 harness 1 例 ±10 冒烟 `runner_exit=0`。 -- 防复发：守卫测试进 `npm test`（门禁 validate 必跑）；错误台账 ERR-113。 -- 相关记录：BUG-1115～1117、BUG-1131。 -- 复发自：无 -- 修复版本：`codex/staging-publish-fix-20261001`。 +- 防复发：守卫测试进 `npm test`（门禁 validate 必跑）；错误台账 ERR-113。2026-10-14 这道守卫在合入前拦住了同一形状：issue #58 的往返对照脚本曾放在 `frontend/scripts/` 并导入测试夹具。脚本改放到 `frontend/tests/bench-local-pg-roundtrips.ts`，文件名不是 `.test.ts`，不会被全量测试收进去。业务代码没有改。原任务书里的旧路径保持不动。 +- 相关记录：BUG-1115～1117、BUG-1131、BUG-1327。 +- 复发自：无。2026-10-14 是同一形状，被守卫拦住，未进测试环境。 +- 修复版本：`codex/staging-publish-fix-20261001`。2026-10-14 的同类挪动在 `codex/node-runtime-resources-20261013`。 ## BUG-1141 | 生时校正缺少度数级时间信号（角度类技法研究） diff --git a/docs/tasks/PROGRESS-node-runtime-resources-20261013.md b/docs/tasks/PROGRESS-node-runtime-resources-20261013.md index 0ba46761..c2203a2d 100644 --- a/docs/tasks/PROGRESS-node-runtime-resources-20261013.md +++ b/docs/tasks/PROGRESS-node-runtime-resources-20261013.md @@ -2,6 +2,8 @@ issue #58。分支 `codex/node-runtime-resources-20261013`。 +2026-10-14 按修复单把往返对照脚本从 `frontend/scripts/bench-local-pg-roundtrips.ts` 挪到 `frontend/tests/bench-local-pg-roundtrips.ts`。文件名不是 `.test.ts`，全量测试不会自动收进去。业务代码没有改，对照数字没有重跑。原任务书里的旧路径保持不动。 + 任务书基线写的是 `77057a819`，计划评论里的基线是 `2994588ad4d858dead4835943e5883c5aa2b03f8`。本分支实际从当时的 `origin/staging` `66cd60081f37a57680274633fcb3fb942d01bb2f` 拉出。没有因此停下来问产品，改动仍按任务书做。 ## 做了什么 @@ -54,6 +56,18 @@ issue #58。分支 `codex/node-runtime-resources-20261013`。 | 本单测试 | `tsx --test` 任务书点名的文件加登录缓存测试 | 134 通过，0 失败，0 取消 | | 真库身份 | `database-local-postgres-session-setup.test.ts` | 1 通过。事务内角色和身份生效，连接归还后再借出时身份已空 | | 数据库套件 | `npm run test:db`，两开关先关后开 | 各 81 / 通过 68 / 失败 13 / 取消 0，名称与已知清单一致 | -| 往返对照 | `scripts/bench-local-pg-roundtrips.ts` | 见上，`payloadMismatch` false | +| 往返对照 | `tests/bench-local-pg-roundtrips.ts`（按修复单从 `scripts/` 挪来） | 见上，`payloadMismatch` false。数字未重跑 | 没有跑前端全量 `npm test`、Python 快速门、`next build`。生产内存曲线要等部署后看。真机两台设备改密码的步骤在 `docs/testing/node-runtime-resources-20261013.md`，登录缓存默认关着，那份清单的第 2 步要先把开关打开才看得到。 + +## 2026-10-14 修复单 + +按 `docs/tasks/TASK-node-runtime-resources-fix-20261014.md`：先合并最新 `origin/staging`，再把对照脚本挪进测试目录。业务代码没有改。对照数字没有重跑。 + +和上一版 `4c3485bc` 比，网页业务目录里多出来的只有合并进来的校正提速改动（`frontend/src/lib/rectification-agentic/v9/engine-client.ts`），不是这单写的。这单自己的提交不碰 `frontend/src`。 + +| 检查 | 命令 | 结果 | +| --- | --- | --- | +| 类型 | `tsc --noEmit -p frontend` | 通过，退出码 0 | +| 规范 | `npm run lint`（在 frontend） | 0 error，125 条原有 warning，未改 | +| 镜像导入守卫 | `tsx --test tests/runtime-sources-no-test-imports.test.ts` | 2 通过，0 失败，0 取消 | diff --git a/docs/tasks/README.md b/docs/tasks/README.md index 33e9ffb0..e2ca4650 100644 --- a/docs/tasks/README.md +++ b/docs/tasks/README.md @@ -469,7 +469,7 @@ | --- | --- | --- | --- | --- | | `TASK-rect-scoring-perf-20261013.md` | `PROGRESS-rect-scoring-perf-20261013.md` | 校正打分提速：引导窗口平方扫描、候选不变量重复算、响应带无读取点字段、候选数上限 241（BUG-1323～1325、1345） | 待验收 | `codex/rect-scoring-perf-20261013` | | `TASK-node-runtime-resources-20261013.md` | `PROGRESS-node-runtime-resources-20261013.md` | 报告 worker abort 监听泄漏、本地 PG 适配层往返与元数据缓存、无上限内存表、用户端会话读缓存（BUG-1326～1328、1346，PG 与缓存部分 A/B） | 待验收 | `codex/node-runtime-resources-20261013` | -| `TASK-node-runtime-resources-fix-20261014.md` | — | **#58 第 1 次打回**：`4c3485bc` 其余全过；压测脚本放在 `frontend/scripts/` 却从 `frontend/tests/` 导入，镜像构建会失败（BUG-1140 形状，任务书指定的路径有误）。挪到 `frontend/tests/` 下非 `.test` 文件名 | 待领取 | `codex/node-runtime-resources-20261013` | +| `TASK-node-runtime-resources-fix-20261014.md` | `PROGRESS-node-runtime-resources-20261013.md` | **#58 第 1 次打回**：`4c3485bc` 其余全过；压测脚本放在 `frontend/scripts/` 却从 `frontend/tests/` 导入，镜像构建会失败（BUG-1140 形状，任务书指定的路径有误）。挪到 `frontend/tests/` 下非 `.test` 文件名 | 待验收 | `codex/node-runtime-resources-20261013` | | `TASK-home-sidebar-rerender-20261013.md` | — | 流式回答时侧栏每帧整栏重渲染（BUG-1329，A/B） | 待领取 | `codex/home-sidebar-rerender-20261013` | | `TASK-home-sidebar-rerender-fix-20261014.md` | — | **#59 第 1 次打回**：`63509431` 其余全过；`chat-navigation-a11y-contract` 按源码文本断言 `onOpenBilling` 旧写法未跟改（§7.8） | 待领取 | `codex/home-sidebar-rerender-20261013` | | `TASK-api-disk-cache-retention-20261013.md` | — | API 磁盘缓存只进不出、VedAstro 审计文件留存出生资料、失败快照久缓存、缓存键规范化（BUG-1330～1332，A/B） | 待领取 | `codex/api-disk-cache-retention-20261013` | diff --git a/docs/testing/node-runtime-resources-ab-20261013.md b/docs/testing/node-runtime-resources-ab-20261013.md index 22d59c80..f4904fae 100644 --- a/docs/testing/node-runtime-resources-ab-20261013.md +++ b/docs/testing/node-runtime-resources-ab-20261013.md @@ -1,6 +1,6 @@ # 本地数据库往返对照（2026-10-14） -环境：Windows，Docker 29.8.0，Node v22.23.2。脚本 `frontend/scripts/bench-local-pg-roundtrips.ts`。库是测试用的临时 Postgres，数据是虚构用户。每组 50 遍，重复 3 组，先各热身 1 遍。往返数是每组平均值的中位数。耗时中位数是三组中位数的中位数。p95 是全部 150 遍的单遍耗时。 +环境：Windows，Docker 29.8.0，Node v22.23.2。脚本 `frontend/tests/bench-local-pg-roundtrips.ts`（2026-10-14 按修复单从 `frontend/scripts/` 挪来，数字未重跑）。库是测试用的临时 Postgres，数据是虚构用户。每组 50 遍，重复 3 组，先各热身 1 遍。往返数是每组平均值的中位数。耗时中位数是三组中位数的中位数。p95 是全部 150 遍的单遍耗时。 判定用第二次运行（输出里带三组原始数）。第一次运行同样是 49 降到 32，中位数 89.51 毫秒降到 32.70 毫秒，内容一致，方向相同。 diff --git a/frontend/scripts/bench-local-pg-roundtrips.ts b/frontend/tests/bench-local-pg-roundtrips.ts similarity index 98% rename from frontend/scripts/bench-local-pg-roundtrips.ts rename to frontend/tests/bench-local-pg-roundtrips.ts index 51746bcd..d6557b31 100644 --- a/frontend/scripts/bench-local-pg-roundtrips.ts +++ b/frontend/tests/bench-local-pg-roundtrips.ts @@ -1,6 +1,6 @@ /** * Count local-Postgres adapter roundtrips for a fictional homepage plus consult write. - * Run from frontend/: npx tsx scripts/bench-local-pg-roundtrips.ts + * Run from frontend/: npx tsx tests/bench-local-pg-roundtrips.ts * Prints JSON. Does not print connection strings. */ import { spawnSync } from "node:child_process"; @@ -13,7 +13,7 @@ import { createLocalPostgresDataClient, localPostgresPoolForTests, } from "../src/lib/db/local-postgres-client-core.ts"; -import { startPostgresFixture } from "../tests/helpers/postgres-fixture.ts"; +import { startPostgresFixture } from "./helpers/postgres-fixture.ts"; const userId = "018f4e6d-7a11-7000-8000-0000000000bb"; const sessionId = "018f4e6d-7a11-7000-8000-0000000000cc"; From 6a94edf200708c1f748a1d46852b813ce9f6932d Mon Sep 17 00:00:00 2001 From: jesse-ux Date: Wed, 14 Oct 2026 07:53:38 +0800 Subject: [PATCH 6/9] =?UTF-8?q?=E8=AE=A9=E5=85=85=E5=80=BC=E5=85=A5?= =?UTF-8?q?=E5=8F=A3=E7=9A=84=E6=97=A7=E6=A3=80=E6=9F=A5=E8=AE=A4=E4=BE=A7?= =?UTF-8?q?=E6=A0=8F=E7=9A=84=E6=96=B0=E5=86=99=E6=B3=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 账户菜单里点充值，仍然打开充值面板，来源仍是账户菜单。只改了这一条按源码对照的检查，业务代码没动。tsc 通过，lint 0 个错误（125 条原有警告），这条检查 13 通过、0 失败。 Co-Authored-By: Grok (agent loop) --- docs/BUG_HISTORY.md | 2 +- .../PROGRESS-home-sidebar-rerender-20261013.md | 16 ++++++++++++++++ docs/tasks/README.md | 2 +- .../tests/chat-navigation-a11y-contract.test.ts | 5 ++++- 4 files changed, 22 insertions(+), 3 deletions(-) diff --git a/docs/BUG_HISTORY.md b/docs/BUG_HISTORY.md index d3dd2212..c7836196 100644 --- a/docs/BUG_HISTORY.md +++ b/docs/BUG_HISTORY.md @@ -17849,7 +17849,7 @@ - 触发条件：首页已登录，回答按帧写入。每一帧都会让首页再渲染一次。 - 根因：BUG-971 切断了「首页订阅自己的注册」这条反馈，但注册仍然每次渲染都发布一份新对象。会话回调和可见列表每次渲染都是新的，外壳就跟着提交一次。不是回归，是当时留下的开销。 - 修复：外壳拿到的是稳定包装，点下去仍执行首页最新一次渲染的回调。注册只在会话、当前项、账户或开关这些数据变了才发布；列表逐条对象没变就不重建行。可见列表按当前人物记忆。侧栏上下文对象也不再每次新建。没有运行时开关。 -- 验证：同一套计数测试。改前 60 帧注册 60 次、侧栏提交 60 次；改后注册 0 次、提交 0 次。换当前项、改标题、换人各发布 1 次。包装函数身份不变，调用记到的是最新一次渲染。详见 `docs/tasks/PROGRESS-home-sidebar-rerender-20261013.md`。 +- 验证：同一套计数测试。改前 60 帧注册 60 次、侧栏提交 60 次；改后注册 0 次、提交 0 次。换当前项、改标题、换人各发布 1 次。包装函数身份不变，调用记到的是最新一次渲染。详见 `docs/tasks/PROGRESS-home-sidebar-rerender-20261013.md`。2026-10-14 补了一条旧检查：账户菜单里的充值仍须打开 billing，来源仍是 account-menu，只是改认稳定包装的写法。 - 防复发：外壳注册须按数据变化发布。任何流式会话面要有侧栏提交次数的按帧断言。`frontend/tests/home-sidebar-rerender.test.tsx` 在 60 帧内要求注册 0 次、侧栏提交 0 次。 - 相关记录：BUG-971、BUG-725、BUG-473、BUG-744、BUG-745、BUG-746。 - 复发自：无。 diff --git a/docs/tasks/PROGRESS-home-sidebar-rerender-20261013.md b/docs/tasks/PROGRESS-home-sidebar-rerender-20261013.md index 2b74e92c..179b5556 100644 --- a/docs/tasks/PROGRESS-home-sidebar-rerender-20261013.md +++ b/docs/tasks/PROGRESS-home-sidebar-rerender-20261013.md @@ -52,3 +52,19 @@ 定向文件：`home-sidebar-rerender`、`session-list-lifecycle`、`session-list-provider`、`sidebar-contract`、`sidebar-state`、`secondary-page-entry`、`account-dialog-inert-20260918`、`home-shell-growth-contract`、`home-streaming-render-split`、`chat-profile-picker`。 全量 `npm test`、`next build`、Python 快速门未跑，留给验收。 + +## 2026-10-14 修复单 + +按修复单合并了最新 `origin/staging`。只改了 `frontend/tests/chat-navigation-a11y-contract.test.ts` 里账户菜单「充值」那一条源码断言，改成认稳定包装的写法，仍然要求打开 billing 且来源是 `account-menu`。业务代码没有改。 + +又搜了一遍 `onOpenProfile`、`onOpenGeneral`、`onOpenLogout`、`onNewChat`、`onSelectSession`、`onLoadMore`。测试里其余命中的是侧栏组件自己的入口，或校正点数不足时打开充值，不是这条旧写法，没有改。 + +和上一版 `63509431` 比，网页业务目录里多出来的只有合并测试线带入的校正提速（`engine-client.ts`），不是这单写的。这单自己的提交不碰 `frontend/src`。 + +| 检查 | 命令 | 结果 | +| --- | --- | --- | +| 类型 | `tsc --noEmit -p frontend` | 通过，退出码 0 | +| 规范 | `npm run lint`（在 frontend） | 0 error，125 条原有 warning。本单改的测试文件没有新增 warning | +| 这条旧检查 | `tsx --test tests/chat-navigation-a11y-contract.test.ts` | 13 通过，0 失败，0 取消 | + +全量 `npm test` 仍留给验收。 diff --git a/docs/tasks/README.md b/docs/tasks/README.md index 7f4f6eec..8b0dade0 100644 --- a/docs/tasks/README.md +++ b/docs/tasks/README.md @@ -471,7 +471,7 @@ | `TASK-node-runtime-resources-20261013.md` | — | 报告 worker abort 监听泄漏、本地 PG 适配层往返与元数据缓存、无上限内存表、用户端会话读缓存（BUG-1326～1328、1346，PG 与缓存部分 A/B） | 待领取 | `codex/node-runtime-resources-20261013` | | `TASK-node-runtime-resources-fix-20261014.md` | — | **#58 第 1 次打回**：`4c3485bc` 其余全过；压测脚本放在 `frontend/scripts/` 却从 `frontend/tests/` 导入，镜像构建会失败（BUG-1140 形状，任务书指定的路径有误）。挪到 `frontend/tests/` 下非 `.test` 文件名 | 待领取 | `codex/node-runtime-resources-20261013` | | `TASK-home-sidebar-rerender-20261013.md` | `PROGRESS-home-sidebar-rerender-20261013.md` | 流式回答时侧栏每帧整栏重渲染（BUG-1329，A/B） | 待验收 | `codex/home-sidebar-rerender-20261013` | -| `TASK-home-sidebar-rerender-fix-20261014.md` | — | **#59 第 1 次打回**：`63509431` 其余全过；`chat-navigation-a11y-contract` 按源码文本断言 `onOpenBilling` 旧写法未跟改（§7.8） | 待领取 | `codex/home-sidebar-rerender-20261013` | +| `TASK-home-sidebar-rerender-fix-20261014.md` | `PROGRESS-home-sidebar-rerender-20261013.md` | **#59 第 1 次打回**：`63509431` 其余全过；`chat-navigation-a11y-contract` 按源码文本断言 `onOpenBilling` 旧写法未跟改（§7.8） | 待验收 | `codex/home-sidebar-rerender-20261013` | | `TASK-api-disk-cache-retention-20261013.md` | — | API 磁盘缓存只进不出、VedAstro 审计文件留存出生资料、失败快照久缓存、缓存键规范化（BUG-1330～1332，A/B） | 待领取 | `codex/api-disk-cache-retention-20261013` | | `TASK-report-reference-perf-20261013.md` | — | 长报告出稿提速：术语替换命中预检、哈希单次遍历、同盘复用（BUG-1335～1336，A/B） | 待领取 | `codex/report-reference-perf-20261013` | | `TASK-consult-prep-parallel-20261013.md` | — | 普通对话发问前准备步骤并行、模型版本短缓存、多领域并发开关（BUG-1337～1339，A/B） | 待领取 | `codex/consult-prep-parallel-20261013` | diff --git a/frontend/tests/chat-navigation-a11y-contract.test.ts b/frontend/tests/chat-navigation-a11y-contract.test.ts index 9b129dcd..4d28c855 100644 --- a/frontend/tests/chat-navigation-a11y-contract.test.ts +++ b/frontend/tests/chat-navigation-a11y-contract.test.ts @@ -39,7 +39,10 @@ test("in-app destinations navigate client-side so the chat survives the round tr assert.match(sidebarSource, / openAccountDialog\("billing", \{ source: "account-menu" \}\)/); + // 原值: /onOpenBilling: \(\) => openAccountDialog\("billing", \{ source: "account-menu" \}\)/ + // 新值: /onOpenBilling: \(\) => \{ latestRef\.current\?\.openAccountDialog\("billing", \{ source: "account-menu" \}\); \}/ + // 原因: 侧栏按钮改成稳定包装，点下去仍打开充值，来源仍是账户菜单。不得放宽成只查 "billing"。 + assert.match(pageSource, /onOpenBilling: \(\) => \{ latestRef\.current\?\.openAccountDialog\("billing", \{ source: "account-menu" \}\); \}/); assert.match(pageSource, /openAccountDialog\("billing", \{ returnTarget: event\.currentTarget, source: "credits" \}\)/); assert.match(pageSource, /openAccountDialog\("billing", \{ source: "insufficient-credits" \}\)/); // 原值: /if \(response\.status === 402\) openAccountDialog\("billing", \{ source: "insufficient-credits" \}\)/ From 9cb0ce1a1a0f80152ba9e176a9175e6aad3a4087 Mon Sep 17 00:00:00 2001 From: Jesse_Chen Date: Wed, 14 Oct 2026 09:56:22 +0800 Subject: [PATCH 7/9] docs(tasks): report reference perf fix brief (agent loop, issue #61, bounce 1) Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01LutgzyJqNhPpZy1nFsJtN3 --- docs/tasks/README.md | 1 + ...TASK-report-reference-perf-fix-20261014.md | 36 +++++++++++++++++++ 2 files changed, 37 insertions(+) create mode 100644 docs/tasks/TASK-report-reference-perf-fix-20261014.md diff --git a/docs/tasks/README.md b/docs/tasks/README.md index a719d697..d13dce09 100644 --- a/docs/tasks/README.md +++ b/docs/tasks/README.md @@ -474,6 +474,7 @@ | `TASK-home-sidebar-rerender-fix-20261014.md` | `PROGRESS-home-sidebar-rerender-20261013.md` | **#59 第 1 次打回**：`63509431` 其余全过；`chat-navigation-a11y-contract` 按源码文本断言 `onOpenBilling` 旧写法未跟改（§7.8） | 待验收 | `codex/home-sidebar-rerender-20261013` | | `TASK-api-disk-cache-retention-20261013.md` | `PROGRESS-api-disk-cache-retention-20261013.md` | API 磁盘缓存只进不出、VedAstro 审计文件留存出生资料、失败快照久缓存、缓存键规范化（BUG-1330～1332，A/B） | 待验收 | `codex/api-disk-cache-retention-20261013` | | `TASK-report-reference-perf-20261013.md` | — | 长报告出稿提速：术语替换命中预检、哈希单次遍历、同盘复用（BUG-1335～1336，A/B） | 待领取 | `codex/report-reference-perf-20261013` | +| `TASK-report-reference-perf-fix-20261014.md` | — | **#61 第 1 次打回**：`69c739fb` 本单全过（Linux 复跑逐字节一致、命中 0.03 s）；与 #60 合并后新缓存目录未登记进 `api_scratch_retention`，合同测试红 | 待领取 | `codex/report-reference-perf-20261013` | | `TASK-consult-prep-parallel-20261013.md` | — | 普通对话发问前准备步骤并行、模型版本短缓存、多领域并发开关（BUG-1337～1339，A/B） | 待领取 | `codex/consult-prep-parallel-20261013` | | `TASK-frontend-dead-code-20261013.md` | — | 前端死代码 17 文件与无调用方接口删除，含未登录重计算接口（BUG-1342） | 待领取 | `codex/frontend-dead-code-20261013` | | `TASK-api-chart-subprocess-gate-20261013.md` | — | 合盘不带 skip 串行排盘、报告无效子进程、名额轻重混用（只模拟）（BUG-1333～1334，A/B） | 待领取（排队：api-disk-cache-retention + report-reference-perf） | `codex/api-chart-subprocess-gate-20261013` | diff --git a/docs/tasks/TASK-report-reference-perf-fix-20261014.md b/docs/tasks/TASK-report-reference-perf-fix-20261014.md new file mode 100644 index 00000000..9b2c188f --- /dev/null +++ b/docs/tasks/TASK-report-reference-perf-fix-20261014.md @@ -0,0 +1,36 @@ +# TASK · 长报告提速修复单（issue #61，第 1 次打回） + +- 日期：2026-10-14 +- 原任务书：`docs/tasks/TASK-report-reference-perf-20261013.md` +- 被验 SHA：`69c739fbee3f5af6deed911482d82e49ed429bf7`（`codex/report-reference-perf-20261013`） +- 执行方：Grok，同一分支续做（先 merge 最新 `origin/staging`，不改写已推历史） +- BUG 编号：沿用 BUG-1335、1336 + +## 1. 验收结论：本单自身全部通过；与 #60 合并后一条合同测试红 + +| 检查 | 结果 | 证据 | +| --- | --- | --- | +| 与最新 staging 合并后的定向测试 | **未通过** | `tests/test_api_scratch_retention.py::test_new_scratch_local_dirs_are_registered_or_whitelisted`：`AssertionError: ['report_reference_cache']` | +| 逐字节一致（Claude 在 Linux、屏蔽 PyJHora 复跑 A/B） | 通过 | 四张虚构盘 A / B 未命中 / B 命中 / 缓存首算的五个字段 sha 全部相同；`format=json`、`edition=reference` 两条 sha 相同 | +| 耗时 | 与回报一致 | 未命中中位数 21.1→15.6 s（day）等，降约 22–26%，未到 35% 门槛 → 渲染提速默认关正确；命中 0.02–0.03 s | +| 缓存键含 `today` | 通过 | `_high_rigor_birth_payload` 在缺省时把当天日期写进 `birth`，跨天不会命中 | +| `GITHUB_SHA` | 通过 | 部署编排已传入；缺失或 `unknown` 时不缓存 | +| `jyotish_api_server.py` | 通过 | 零改动 | + +**根因**：#60（BUG-1330～1332，已在 staging）新增了合同测试：`scripts/` 里写 `scratch/local/<子目录>` 的地方，必须登记在 `scripts/api_scratch_retention.py` 的 `REGISTRY`（服务器周期清理）或 `OFFLINE_SCRATCH_NAMES`（自己会清理的缓存）里。本单新增的 `scratch/local/report_reference_cache/` 两边都没登记。两单各自都对，合在一起才红——这不是你的疏漏，是两单同时在做。 + +## 2. 修法要求 + +1. 合并最新 `origin/staging`（含 #60）。 +2. 在 `scripts/api_scratch_retention.py::OFFLINE_SCRATCH_NAMES` 加 `"report_reference_cache"`，旁边写一行注释：它与 `consult_full_data_cache` 同走 `consult_full_data_cache.get_or_build`，每次调用先 `prune_cache(name=…)` 删两天前的文件，自己会清；含出生资料，保留期 2 天。 +3. 不改两个模块的任何逻辑。 +4. PROGRESS 末尾记一行。 + +## 3. 验收标准 + +- `python -m pytest tests/test_api_scratch_retention.py tests/test_report_reference_perf.py tests/test_consult_card_full_source.py -q` 全绿； +- Claude 全量 pytest 与 staging 基线逐条比对无新增。 + +## 4. 让步顺序 + +无。第 1 次打回。 From 5cb85800e58162c5808401d2dbbfb47360391202 Mon Sep 17 00:00:00 2001 From: Jesse_Chen Date: Wed, 14 Oct 2026 10:20:22 +0800 Subject: [PATCH 8/9] docs(tasks): consult prep parallel fix brief (agent loop, issue #62, bounce 1) Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01LutgzyJqNhPpZy1nFsJtN3 --- docs/tasks/README.md | 1 + ...TASK-consult-prep-parallel-fix-20261014.md | 41 +++++++++++++++++++ 2 files changed, 42 insertions(+) create mode 100644 docs/tasks/TASK-consult-prep-parallel-fix-20261014.md diff --git a/docs/tasks/README.md b/docs/tasks/README.md index d13dce09..c361e157 100644 --- a/docs/tasks/README.md +++ b/docs/tasks/README.md @@ -476,6 +476,7 @@ | `TASK-report-reference-perf-20261013.md` | — | 长报告出稿提速：术语替换命中预检、哈希单次遍历、同盘复用（BUG-1335～1336，A/B） | 待领取 | `codex/report-reference-perf-20261013` | | `TASK-report-reference-perf-fix-20261014.md` | — | **#61 第 1 次打回**：`69c739fb` 本单全过（Linux 复跑逐字节一致、命中 0.03 s）；与 #60 合并后新缓存目录未登记进 `api_scratch_retention`，合同测试红 | 待领取 | `codex/report-reference-perf-20261013` | | `TASK-consult-prep-parallel-20261013.md` | — | 普通对话发问前准备步骤并行、模型版本短缓存、多领域并发开关（BUG-1337～1339，A/B） | 待领取 | `codex/consult-prep-parallel-20261013` | +| `TASK-consult-prep-parallel-fix-20261014.md` | — | **#62 第 1 次打回**：`a0e223e2` 业务逻辑通过；两条源码文本断言未跟改（§7.8）、A/B 脚本从 `frontend/tests` 导入（BUG-1140） | 待领取 | `codex/consult-prep-parallel-20261013` | | `TASK-frontend-dead-code-20261013.md` | — | 前端死代码 17 文件与无调用方接口删除，含未登录重计算接口（BUG-1342） | 待领取 | `codex/frontend-dead-code-20261013` | | `TASK-api-chart-subprocess-gate-20261013.md` | — | 合盘不带 skip 串行排盘、报告无效子进程、名额轻重混用（只模拟）（BUG-1333～1334，A/B） | 待领取（排队：api-disk-cache-retention + report-reference-perf） | `codex/api-chart-subprocess-gate-20261013` | | `TASK-answer-effort-ab-20261013.md` | — | 答题步推理强度开关 + A/B（需模型 key） | 待领取（排队：consult-prep-parallel） | `codex/answer-effort-ab-20261013` | diff --git a/docs/tasks/TASK-consult-prep-parallel-fix-20261014.md b/docs/tasks/TASK-consult-prep-parallel-fix-20261014.md new file mode 100644 index 00000000..aec1e021 --- /dev/null +++ b/docs/tasks/TASK-consult-prep-parallel-fix-20261014.md @@ -0,0 +1,41 @@ +# TASK · 发问准备并行修复单（issue #62，第 1 次打回） + +- 日期：2026-10-14 +- 原任务书：`docs/tasks/TASK-consult-prep-parallel-20261013.md` +- 被验 SHA：`a0e223e20cebcce3479accb60f6aec44d1caadda`（`codex/consult-prep-parallel-20261013`） +- 执行方：Grok，同一分支续做（先 merge 最新 `origin/staging`，不改写已推历史） +- BUG 编号：沿用 BUG-1337～1339 + +## 1. 验收结论：业务逻辑通过；前端全量多出 3 条真失败 + +| 检查 | 结果 | 证据 | +| --- | --- | --- | +| 开关全关时的顺序 | 通过 | `turn-prep.ts` 串行分支与原 `route.ts` 的 401→423→400→503→404→409→409→503 逐条同序 | +| 开关开时 | 通过 | 删号判定仍最先返回、在任何写库扣点之前；审核结论在 `beforeReserve` 内等齐，被拦不 reserve | +| `readVersion` 缓存 | 通过 | 只缓存成功结果，`invalidateLanguageModelCatalog` 同时清；默认 TTL 0 | +| tsc / lint | 通过 | 0 错 / 0 error | +| 前端全量 `npm test`（与 staging 逐名比对） | **未通过：新增 3 条** | 见下 | + +1. `frontend/tests/chat-session-authority.test.ts`「consult POST on a rectification session is 409, a missing id is 404」：第 65 行在 `route.ts` 源码里找 `/if \(!chatSession\) \{[\s\S]*status: 404/`——这段判定挪进了 `frontend/src/lib/consult/turn-prep.ts`（`sessionMissing()`），断言没跟着挪。 +2. `frontend/tests/consultation-workflow-contract.test.ts`「consultation plans are server-owned and bounded」：第 160 行找 `/beforeReserve: \(\{ consultationMode \}\) => createConsultationPlan\(\{/`，第 161 行找 `reserve: () => reserveConsultationModel(`——两处都改成了 `async` 函数体。 +3. `frontend/tests/runtime-sources-no-test-imports.test.ts`：`frontend/scripts/research/consult-prep-parallel-ab.mts` 第 9 行从 `../../tests/support/stub-server-only.mjs` 导入。web 镜像不复制 `frontend/tests`，推上 staging 后镜像构建会失败（BUG-1140；#58 刚因同一形状打回过一次）。 + +原任务书 §5 第 7 条已经列出要先 grep 的源码文本断言；这次漏的两条不在那份清单里，但属于同一类（AGENTS.md §7.8）。 + +## 2. 修法要求 + +1. **1、2 两条**：按 §7.3 写「原值 / 新值 / 原因」三栏改断言，**意图不得放宽**： + - 1：改为在 `turn-prep.ts` 里锁「查不到会话 → 404」「类型不是 consultation → 409 `session_not_consultation`」，并保留第 70 行那条「不得合并成一个判断」的反向断言（针对新文件）。 + - 2：改为锁「`beforeReserve` 里调用 `createConsultationPlan`」「`beforeReserve` 在 `reserve` 之前」「`reserve` 调用 `reserveConsultationModel`」，允许 `async` 写法，但不得只查函数名存在。 +2. **第 3 条**：把 `consult-prep-parallel-ab.mts` 挪到 `frontend/tests/` 下（文件名不得以 `.test.ts` / `.test.tsx` 结尾），导入改相对路径；同步改 PROGRESS 与 `docs/testing/consult-prep-parallel-ab-20261013.md` 里的命令路径。`frontend/scripts/research/` 下其它新增文件同样检查。 +3. 再 grep 一遍 `route.ts` 被挪走 / 改写的语句在 `frontend/tests`、`tests` 里还有没有按源码文本断言的：`grep -rn "chatSession\.session_type\|getUser()\|refuseWhenAccountDeletionPending\|resolveSessionLanguageModel(\|reserveConsultationModel(\|createConsultationPlan(" frontend/tests tests`，命中的逐条确认。 +4. 业务代码不改；三个开关默认值不变。 + +## 3. 验收标准 + +- 前端全量 `npm test` 失败清单与 staging 基线逐名相同（本机另有 Docker 类失败，以基线为准）； +- `git diff a0e223e2..<新 SHA> -- frontend/src` 为空。 + +## 4. 让步顺序 + +无。第 1 次打回。 From f122f8b7d7ca369328ce9e1160805202d104d1ac Mon Sep 17 00:00:00 2001 From: Jesse_Chen Date: Wed, 14 Oct 2026 11:02:37 +0800 Subject: [PATCH 9/9] docs(tasks): frontend dead code fix brief (agent loop, issue #63, bounce 1) Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01LutgzyJqNhPpZy1nFsJtN3 --- docs/tasks/README.md | 1 + .../TASK-frontend-dead-code-fix-20261014.md | 33 +++++++++++++++++++ 2 files changed, 34 insertions(+) create mode 100644 docs/tasks/TASK-frontend-dead-code-fix-20261014.md diff --git a/docs/tasks/README.md b/docs/tasks/README.md index 6967060a..c49ff179 100644 --- a/docs/tasks/README.md +++ b/docs/tasks/README.md @@ -478,6 +478,7 @@ | `TASK-consult-prep-parallel-20261013.md` | — | 普通对话发问前准备步骤并行、模型版本短缓存、多领域并发开关（BUG-1337～1339，A/B） | 待领取 | `codex/consult-prep-parallel-20261013` | | `TASK-consult-prep-parallel-fix-20261014.md` | — | **#62 第 1 次打回**：`a0e223e2` 业务逻辑通过；两条源码文本断言未跟改（§7.8）、A/B 脚本从 `frontend/tests` 导入（BUG-1140） | 待领取 | `codex/consult-prep-parallel-20261013` | | `TASK-frontend-dead-code-20261013.md` | — | 前端死代码 17 文件与无调用方接口删除，含未登录重计算接口（BUG-1342） | 待领取 | `codex/frontend-dead-code-20261013` | +| `TASK-frontend-dead-code-fix-20261014.md` | — | **#63 第 1 次打回**：`1f9947a0` 其余全过；类名白名单留了已删组件的 `personal-report-additional-charts` | 待领取 | `codex/frontend-dead-code-20261013` | | `TASK-api-chart-subprocess-gate-20261013.md` | — | 合盘不带 skip 串行排盘、报告无效子进程、名额轻重混用（只模拟）（BUG-1333～1334，A/B） | 待领取（排队：api-disk-cache-retention + report-reference-perf） | `codex/api-chart-subprocess-gate-20261013` | | `TASK-answer-effort-ab-20261013.md` | — | 答题步推理强度开关 + A/B（需模型 key） | 待领取（排队：consult-prep-parallel） | `codex/answer-effort-ab-20261013` | | `TASK-consult-route-split-20261013.md` | — | consult POST 拆分第一阶段：开流前加载 + 计费结算 | 待领取（排队：consult-prep-parallel、answer-effort-ab） | `codex/consult-route-split-20261013` | diff --git a/docs/tasks/TASK-frontend-dead-code-fix-20261014.md b/docs/tasks/TASK-frontend-dead-code-fix-20261014.md new file mode 100644 index 00000000..47de3f12 --- /dev/null +++ b/docs/tasks/TASK-frontend-dead-code-fix-20261014.md @@ -0,0 +1,33 @@ +# TASK · 前端死代码删除修复单（issue #63，第 1 次打回） + +- 日期：2026-10-14 +- 原任务书：`docs/tasks/TASK-frontend-dead-code-20261013.md` +- 被验 SHA：`1f9947a0d733e34b7754cce058e20b023791fdd8`（`codex/frontend-dead-code-20261013`） +- 执行方：Grok，同一分支续做（先 merge 最新 `origin/staging`，不改写已推历史） +- BUG 编号：沿用 BUG-1342 + +## 1. 验收结论：一条未通过，其余全部通过 + +| 检查 | 结果 | 证据 | +| --- | --- | --- | +| 删除范围 | 通过 | 删掉的文件与三个路由都在原任务书清单内；全仓 grep 三个路由路径、被删模块名，`frontend/src`、`scripts`、`deploy`、`.gitea` 无残留引用 | +| 改既有断言 | 通过 | 每处三栏齐全；否定合同（不得挂内部徽章 / 内部面板）保留；`/api/birth-rectification` 改成「路由不存在且无代理」 | +| tsc / lint / next build | 通过 | 0 错 / 0 error（warning 125→123）/ `/` 仍 Static；首屏 gzip −312 字节 | +| Python 改动的两份测试 | 通过 | — | +| 前端全量 `npm test`（与 staging 逐名比对） | **未通过：新增 1 条** | `frontend/tests/class-name-definition-contract.test.ts`「the knownUnstyled allowlist has no stale entries」：`actual: ['personal-report-additional-charts']` | + +**根因**：`class-name-definition-contract.test.ts` 第 62 行的 `knownUnstyled` 白名单里有 `personal-report-additional-charts`，这个类名唯一的使用者是已删除的 `personal-report-document-view.tsx`。合同要求白名单里不能有已经没人用的条目。 + +## 2. 修法要求 + +1. 从 `knownUnstyled` 删掉 `"personal-report-additional-charts"` 这一项，旁边按 §7.3 写三栏（原值：白名单含该项；新值：删除；原因：唯一使用者随 BUG-1342 删除，白名单不得留陈旧条目）。 +2. 再查一遍：被删文件里用到的其它类名，是否还出现在别的白名单 / 合同里（`grep -rn "knownUnstyled\|allowlist" frontend/tests` 后逐个核对）。 +3. 业务代码不改。PROGRESS 末尾记一行。 + +## 3. 验收标准 + +前端全量 `npm test` 失败清单与 staging 基线逐名相同（本机 Docker 类失败除外）。 + +## 4. 让步顺序 + +无。第 1 次打回。