Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 jesse-uxandGrok (agent loop)
|
df6b912d63 | ||
 ClaudeandClaude Opus 5.5
|
18d828e9b6 | ||
|
jesse-uxandGrok (agent loop)
|
882c60b54e | ||
|
ClaudeandClaude Opus 5.5
|
a0de2388e3 |
@@ -24397,3 +24397,18 @@
|
||||
- 判断：她的生成器修复（§12）在本仓没有可移植对象；快照不重生成、不改。若以后移植她的生成器，需要一起带她的 4 个新用例。
|
||||
- 关联记录：无
|
||||
- 修复版本：无
|
||||
|
||||
## BUG-1814 | 查卡一次只能取一段，每多一段就多一步，整份计算随每一步重发
|
||||
|
||||
- 状态：fixed-pending-verify（分支 codex/token-lookup-20261109）
|
||||
- 首次发现 / 最近更新：2026-11-08（占星师验收清单 §4）/ 2026-11-09（修复）
|
||||
- 影响面：frontend/src/mastra/consultation-tools.ts（evidenceLookupInputSchema、查卡工具本体、ConsultationRuntimeStep）。
|
||||
- 现象：模型要用两段证据时要调两次 read-consultation-evidence，于是多两步；每一步都把约 9.3 万字符的计算结果再发一遍。同一张盘实测：调用计算那步 prompt 62,859 字符，看过计算结果后 156,254，取一次小段后 156,966，再取一次后 157,996。
|
||||
- 触发条件：回答前需要多段卡外证据（常见于高严谨模式）。
|
||||
- 根因：工具输入是 { section }，一次只返回一段；多段只能多次调用，而每次调用都多一步，历史工具结果随每一步重发。
|
||||
- 修复：输入改为 z.union，保留 { section } 单段写法，新增 { sections: [...] }（2–6 段、去重、仍计 1 次调用、仍受每轮 3 次上限）。返回 { status, results: [{ section, status, data | reason }], note }；单段写法返回形状与键序不变。可选 fields 只取该段顶层键，未知键进 unknown_fields，不截断数据。runtime step 记 sections 与 chars（内部字段，不进公开回执）。
|
||||
- 验证：frontend/tests/consult-evidence-lookup-batch-20261109.test.ts（7 条）；consult-evidence-lookup-20260927（15）、consult-same-source-check-1773（4）、consult-same-source-independence-1773（4）、consultation-agentic-runtime（87）全过；k=2 时总输入字符下降约 29–30%，k=3 约 44–45%（测量表在 docs/tasks/PROGRESS-token-lookup-20261109.md）。
|
||||
- 防复发：上述测试；测量脚本 frontend/scripts/research/consult-lookup-token-probe-171.mts 可复跑。
|
||||
- 关联记录：BUG-1803
|
||||
- 修复版本：分支 codex/token-lookup-20261109
|
||||
- 第 1 次打回（2026-11-09，FIX-token-lookup-1）：70 题实测 NEW 每题输入 138,394 token，高于 staging 的 122,901（55/70 更高，p<0.001）；原因是工具描述删掉了「只在需要时取」并写了「List every section…(2-6)」，查卡的回答从 18 个涨到 25 个；另外 `sections` 的 schema 写成 min(2)，模型传 1 段被拒、白走一步。已改：描述恢复「only when the question needs」，多段只作补充说明、不写段数暗示；schema 改 1–6 段，1 段也走 results 数组；两个测量脚本从 `frontend/scripts/research/` 移到 `frontend/tests/research/`（否则 BUG-1140 的 runtime-sources 测试与镜像构建失败）。按 BUG-1803 的真实分布估算，本能力自身收益约 3–4%，低于 5% 止损线，是否保留交产品决定。
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
# FIX-token-lookup-1：查卡多段上线后，真实回答的输入 token 反而上升（#171 第 1 次打回，2026-11-09）
|
||||
|
||||
- 分支：`codex/token-lookup-20261109`（在本分支远端最新上继续）
|
||||
- 被验 SHA：`882c60b54e37b6a9582ed0728fddca57573e9dde`
|
||||
- 原任务书：`docs/tasks/TASK-token-lookup-20261109.md`（全部条款继续有效）
|
||||
- Bug 编号：继续用 BUG-1814，在原条目里补「第 1 次打回」，不新开号。
|
||||
|
||||
## 验收结论：未通过
|
||||
|
||||
多段一次取回的内容与逐段取一字不差，这一点成立，测试也写得好。问题在于：真实回答里，省下的 token 没有出现，反而多用了。
|
||||
|
||||
## 实测（Claude 2026-11-09，deepseek-flash，10 张公开盘 × 7 题，每臂 70 份）
|
||||
|
||||
三臂：
|
||||
- NEW：staging `c032823f` 加本批其他已过单子再加 #171；
|
||||
- OLD：staging `c032823f`；
|
||||
- NOISE：OLD 的代码独立再生成一次。
|
||||
|
||||
| 指标 | NEW | OLD | NOISE |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| 每题输入 token 平均 | 138,394 | 122,901 | 117,972 |
|
||||
| 每题输入 token 中位 | 117,470 | 107,225 | 106,490 |
|
||||
| 用了查卡的回答 | 25 | 18 | 12 |
|
||||
| 查卡调用次数 | 22 次（21 次多段，平均约 2.5 段） | 28 次单段 | 17 次单段 |
|
||||
| 有查卡的回答，平均输入 | 178,994 | 158,300 | 158,697 |
|
||||
| 没查卡的回答，平均输入 | 95,406 | 94,763 | 92,401 |
|
||||
|
||||
逐题配对比较：
|
||||
- NEW 对 OLD：70 题里 55 题更高、15 题更低，符号检验 p < 0.001，平均每题多 15.5k。
|
||||
- NOISE 对 OLD：30 题更高、40 题更低，p = 0.28。
|
||||
|
||||
所以这次上升不是随机波动。
|
||||
|
||||
## 原因
|
||||
|
||||
1. **工具描述在引导模型多取。**
|
||||
- 原描述里的 `and only when the question needs that section` 被删掉了；
|
||||
- 新描述换成了 `List every section the question needs in one sections call (2-6 sections)`。
|
||||
- 结果是：
|
||||
- 查卡的回答从 18 个变成 25 个；
|
||||
- 每次返回 2–4 段，后面每一步都要把这些内容再发一遍。
|
||||
2. **省 token 的前提，在真实分布里不成立。**
|
||||
- 测量用的是 k = 2 逐段对比 1 次多段。可是现状下，一个回答平均只查约 1.5 段，而且大多数回答一段都不查（BUG-1803 也是同样结论）。
|
||||
- 本单的测量只算了「同样要取 k 段时，一次取比分开取省多少」，没有算「模型会不会因此多取」。
|
||||
3. **段数下限和任务书不一致。**
|
||||
- 任务书写的是多段 1–6 段，schema 却写了 `min(2)`。
|
||||
- 模型有 2 次只传了 1 段的 `sections`，被 schema 拒掉，白白多走了一步。
|
||||
|
||||
## 要做的
|
||||
|
||||
1. **工具描述**：
|
||||
- 恢复 `and only when the question needs that section` 的意思；
|
||||
- 多段写成补充说明：「如果确实需要几段，就在一次调用里一起列出，不要分几次取」；
|
||||
- 不写「list every section the question needs」这类鼓励多取的话，也不写 2–6 的数字暗示；
|
||||
- 3 次上限说明保留。
|
||||
2. **schema**：`sections` 改为 1–6 段。1 段时，返回形状与多段一致（`results` 数组）。旧的 `{ section }` 写法继续兼容。
|
||||
3. **测量脚本移到 `frontend/tests/research/`**：
|
||||
- `frontend/scripts/research/consult-lookup-*-171.mts` 引用了 `frontend/tests` 下的夹具。`frontend/scripts` 会被复制进 web 镜像，`runtime-sources-no-test-imports.test.ts`（BUG-1140）因此失败，staging 镜像构建也会失败。
|
||||
- 把两个脚本移到 `frontend/tests/research/`，相对路径 `../../tests/` 改成 `../`。
|
||||
- 验收时 Claude 已在验收分支上这样移过，可以照抄。
|
||||
4. **补测量**：在 PROGRESS 里补一张「每题平均查卡段数 × 每段平均字符」的估算，说明在 BUG-1803 的真实分布下（57/70 不查）预计能省多少。若预计省不到 5%，停下，选 `需拍板`。
|
||||
5. 新测试 `consult-evidence-lookup-batch-20261109.test.ts` 保留，并补两条：
|
||||
- `sections` 只有 1 段时，返回的 `data` 与单段写法 deepEqual；
|
||||
- 工具描述里包含「只在需要时才取」的意思（用 `/only when/` 断言），不包含 `List every section`。
|
||||
|
||||
## 验收口径（Claude 复验时跑同一套 70 题）
|
||||
|
||||
- 每题输入 token：NEW 对 staging 的配对符号检验，不得显著更高（p ≥ 0.05，或者平均值不高于 staging）；
|
||||
- 用了查卡的回答数，不得多于 staging 加噪声的范围（约 12–18）；
|
||||
- 可追溯性、事实核查矛盾率、两序一致的盲评，都不得低于噪声范围。
|
||||
|
||||
## 自测命令
|
||||
|
||||
同原任务书，另加：在 `frontend` 下运行 `node node_modules/tsx/dist/cli.mjs --test tests/runtime-sources-no-test-imports.test.ts`。
|
||||
@@ -0,0 +1,117 @@
|
||||
# PROGRESS · 查卡一次取多段（#171 省 token，2026-11-09）
|
||||
|
||||
- 分支：`codex/token-lookup-20261109`（基线 `b453505d`）
|
||||
- 任务书：`docs/tasks/TASK-token-lookup-20261109.md`
|
||||
- Bug：`docs/BUG_HISTORY.md` 新增 BUG-1814（关联 BUG-1803）
|
||||
|
||||
## 第一步：测量（先量后改）
|
||||
|
||||
用 golden 公开盘（Steve Jobs、Barack Obama、Elizabeth Taylor）经真实链路建 `executions`，再逐段调用 `readEvidenceLookupSection`；工具结果用真实 agent 链路（`runNatalAgent` + 脚本化模型）实测每一步的 prompt 字符数。
|
||||
|
||||
### 每步 prompt 实测（Steve Jobs / career）
|
||||
|
||||
| 步 | 这一步的 prompt 字符数 | 其中的工具结果 |
|
||||
| --- | ---: | --- |
|
||||
| 调计算 | 62,859 | — |
|
||||
| 看过计算结果 | 156,254 | run-jyotish-consultation 93,202 |
|
||||
| 取 D60 之后 | 156,966 | ＋ read-consultation-evidence 535 |
|
||||
| 取 yogas 之后（写答案） | 157,996 | ＋ read-consultation-evidence 857 |
|
||||
|
||||
### 每段查卡的实测字符数（三张盘同一量级）
|
||||
|
||||
| 段 | 字符数（约） |
|
||||
| --- | ---: |
|
||||
| `western:*`（最大一段） | 28,400 |
|
||||
| `upstream_techniques` | 19,700–21,900 |
|
||||
| `varga:research_dn` | 9,900 |
|
||||
| `kp_placidus` | 5,785 |
|
||||
| `yogas` / `varga:D6x` 等常用小段 | 535–857 |
|
||||
|
||||
### 三种情况的总输入字符（同一张盘、同样 k 段）
|
||||
|
||||
公式：第 n 步的 prompt = 62,859 固定部分 ＋ 此前全部工具结果之和。
|
||||
|
||||
| 情况 | k = 1 | k = 2 | k = 3 |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| (i) 现状：每段一步（实测小段 D60 + yogas） | 376,079 | 533,075 | 691,xxx |
|
||||
| (ii) 一次取多段（同一批小段） | 376,079 | 377,xxx | 378,xxx |
|
||||
| 下降 | 0% | **29.4%** | **45.3%** |
|
||||
| (i) 现状：两个大段（28,400 + 21,440） | 376,079 | 609,290 | 842,xxx |
|
||||
| (ii) 一次取多段（同一批大段） | 376,079 | 424,825 | 473,xxx |
|
||||
| 下降 | 0% | **30.3%** | **43.8%** |
|
||||
|
||||
结论：**k = 2 时总输入字符下降约 29–30%，超过 10% 的门槛**，按任务书继续做第二步。字段过滤的额外收益按段算：`fields` 只保留几个键时那一段（最大 28.4k 字符）能再省下大部分，但对整轮输入影响约 0.5–4%，所以没有把它当作主要收益。
|
||||
|
||||
## 第二步：改造了什么
|
||||
|
||||
| 位置 | 改动 |
|
||||
| --- | --- |
|
||||
| `frontend/src/mastra/consultation-tools.ts` `evidenceLookupInputSchema` | `{ section }` 单段写法保留（新增可选 `fields`）；新增 `{ sections: [...] }`，2–6 段、`.strict()`。两种写法用 `z.union` 并存 |
|
||||
| 同文件 `readEvidenceLookupSection` 之后 | 新增 `pickLookupFields`：只按顶层键取值，不改值、不截断数组；不存在的键进 `unknown_fields` |
|
||||
| 同文件工具本体 | 单段：返回形状与键序不变（`status / section / data / note`，带 `fields` 时多 `unknown_fields`）；多段：返回 `{ status, results: [{ section, status, data｜reason }], note }`。多段计 1 次调用，仍受每轮 3 次上限；第 4 次与原来一样 `refused`。降级与「计算未缓存」也按单段/多段给对应形状 |
|
||||
| `ConsultationRuntimeStep` | 新增内部可选字段 `sections` 与 `chars`（每次查卡记读了哪几段、返回多少字符）；`publicConsultationRuntimeSteps` 不导出它们，公开回执不变 |
|
||||
| 工具描述 | 说明「需要几段就一次全列（2–6）」「`fields` 只取这些顶层键，未知键进 `unknown_fields`」，3 次上限照旧 |
|
||||
|
||||
没动：`EVIDENCE_LOOKUP_SECTIONS`／`consultation-evidence-card.ts`、`references/**`、`consult-preset-prompts.ts`、`consultation-methodology.ts`、`mastra/index.ts` 的 instructions、`route.ts`、每轮 3 次上限的数值、序列化区（#170）、`run-jyotish-consultation` 的 execute（#172）、工具表（#173）。
|
||||
|
||||
`stream-agent-response.ts` 的「答案重启」判定只看工具名与已输出的答案文字，不读返回里的 `section`，因此不需要改；`lookupEvidence` 只是原样挂到 `evidence_lookup` 上，形状变化无影响。
|
||||
|
||||
## 测试
|
||||
|
||||
| 测试 | 结果 |
|
||||
| --- | --- |
|
||||
| 新增 `frontend/tests/consult-evidence-lookup-batch-20261109.test.ts` | 7 过 / 0 败 |
|
||||
| `frontend/tests/consult-evidence-lookup-20260927.test.ts` | 15 过 / 0 败 |
|
||||
| `frontend/tests/consult-same-source-check-1773.test.ts` | 4 过 / 0 败 |
|
||||
| `frontend/tests/consult-same-source-independence-1773.test.ts` | 4 过 / 0 败 |
|
||||
| `frontend/tests/consultation-agentic-runtime.test.ts` | 87 过 / 0 败 |
|
||||
| `tsc --noEmit -p frontend` | 0 错 |
|
||||
| `npm run lint`（frontend） | 0 error、63 warning（都是既有文件） |
|
||||
|
||||
新测试覆盖任务书要求的每一条：
|
||||
|
||||
1. 单段旧写法返回形状与键序不变（`status / section / data / note`）。
|
||||
2. 多段一次调用每段 `data` 与逐段单独取 deepEqual（必需字段 100% 送达、内容不变）。
|
||||
3. `fields` 只少键不改值；未知键进 `unknown_fields`；数组不截断（用整段键名回取，结果与整段一致）。
|
||||
4. 多段计 1 次；三次之后第 4 次仍 `refused` / `lookup_limit_reached`。
|
||||
5. 超过 6 段、只有 1 段、段名不在清单、混写 `section` + `sections`、多余键，都被 schema 拒绝。
|
||||
6. 送达核对：三张 golden 盘 × career / marriage / health，`evidence_contract.missing_route_layers` 全部为空。
|
||||
|
||||
既有断言没有改动，所以没有需要写「原值 / 新值 / 原因」的地方。
|
||||
|
||||
## 交给 Claude 验收的两项（任务书第 6 条）
|
||||
|
||||
- `scripts/research/consult-evidence-traceability.mts`、`consult-same-source-check.mts` 改前改后对比。
|
||||
- 盲测准确度对比（本仓 Windows 侧没有模型凭据，按环境缺口留给你）。
|
||||
|
||||
测量脚本留在分支里，可复跑：`frontend/scripts/research/consult-lookup-token-probe-171.mts`（每步 prompt 字符数）与 `consult-lookup-section-sizes-171.mts`（每段字符数）。生成物没有提交，避免把一次性数据写进仓库。
|
||||
|
||||
|
||||
## 第 1 次打回（FIX-token-lookup-1，2026-11-09）
|
||||
|
||||
Claude 用 70 题实测：NEW 每题输入 138,394 token，比 OLD 122,901 更高（55/70 题更高，p < 0.001）。
|
||||
原因不在「多段取回的内容不对」（那部分成立），而在**工具描述在引导模型多取**，以及本单的收益模型
|
||||
只在「同样要取 k 段」的前提下成立。
|
||||
|
||||
### 改了什么
|
||||
|
||||
| 打回的点 | 改法 |
|
||||
| --- | --- |
|
||||
| 工具描述删掉了「只在需要时取」，还写了「List every section…(2-6)」 | 恢复 `only when the question needs that section`；多段改成补充说明「确实需要几段就在一次调用里一起列出，不要分几次取」；不再写 `List every section`，也不写 2–6 的数字暗示；3 次上限说明保留 |
|
||||
| `sections` 下限写成了 `min(2)`，与任务书的 1–6 不符，模型传 1 段被 schema 拒掉 | 改成 `min(1).max(6)`；只传 1 段时走 `results` 数组，旧的 `{ section }` 写法继续返回原形状 |
|
||||
| 测量脚本放在 `frontend/scripts/research/`，引用了 `frontend/tests` 的夹具，会让 `runtime-sources-no-test-imports.test.ts`（BUG-1140）失败、镜像构建失败 | 两个脚本移到 `frontend/tests/research/`，相对路径改成 `../fixtures/`、`../consult-natal-agent-test-support.ts` |
|
||||
|
||||
### 收益估算（按 BUG-1803 的真实分布）
|
||||
|
||||
BUG-1803 实测：70 个回答里 57 个一次也不查卡，合计 24 次调用；本次 70 题实测 NEW 22 次、OLD 28 次。
|
||||
按「查卡的回答约占 1/5、平均每次取 2 段」估算：
|
||||
|
||||
| 情况 | 每题平均调用 | 每题输入 token（估算） |
|
||||
| --- | ---: | ---: |
|
||||
| 现状（每段一次调用） | 0.2 × 2 = 0.4 次多步 | 基线 122,901 |
|
||||
| 一次取多段（同样的段） | 0.2 × 1 = 0.2 次多步 | 省下约 0.2 × 每题一次重发（约 9.3 万字符 ≈ 2.3 万 token）× 0.2 ≈ **4.6k token/题 ≈ 3.7%** |
|
||||
|
||||
也就是说，**在真实分布下这项改动本身的收益约 3–4%，低于 5% 的止损线**。
|
||||
|
||||
按 FIX 第 4 条：「若预计省不到 5%，停下，选需拍板」。所以这一轮把工具描述、schema、脚本位置三件
|
||||
确定性的问题改完并推上去，**是否继续保留「一次多段」这个能力，交产品决定**（见回报评论里的选项）。
|
||||
@@ -0,0 +1,105 @@
|
||||
# TASK：省 token——查卡工具一次取多段、只返回要的字段，减少重复发送（2026-11-09）
|
||||
|
||||
- 分支：`codex/token-lookup-20261109`（基于 staging `b453505d`）
|
||||
- Bug 编号：BUG-1814（开工时核对 `docs/BUG_HISTORY.md` 的最大编号；1809–1830 已预留给本批任务）
|
||||
- 负责验收：Claude（jyotisha-2a）。盲测由 Claude 在验收时跑。
|
||||
|
||||
## 背景
|
||||
|
||||
现在平均每问约 13.7 万 token。
|
||||
|
||||
Mastra 每走一步都会把整份 system、历史和此前全部工具结果重新发一次给模型。所以模型每调用一次 `read-consultation-evidence`，就多一步，`run-jyotish-consultation` 那份约 9 万字符的结果也随之再发一遍。
|
||||
|
||||
占星师 2026-11-08 验收清单 §4 原文：“三次调用未必一定不够；若一次返回多个完整必要层，可能足够。应以‘问题所需字段是否实际送达’为验收条件，验证工具预算是否造成遗漏，再决定批量补取或调整预算，不能简单无限增加调用。”
|
||||
|
||||
占星师 Q3 原文：“不能把‘文件送进模型’视为‘已经执行方法’。”反过来也成立：本任务省下的只能是重复发送，不能少送本题需要的字段。
|
||||
|
||||
## 现状（b453505d 实测，行号以符号为准）
|
||||
|
||||
查卡工具在 `frontend/src/mastra/consultation-tools.ts`：
|
||||
|
||||
- 工具 id 与上限：`CONSULTATION_EVIDENCE_LOOKUP_TOOL_ID`（约 :187）、`MAX_EVIDENCE_LOOKUPS_PER_TURN = 3`（约 :188）。
|
||||
- 输入：`evidenceLookupInputSchema`（约 :1199）是 `z.object({ section: z.enum(EVIDENCE_LOOKUP_SECTIONS) }).strict()`，一次只能取**一段**，也不能只取其中的某些字段。
|
||||
- 取数：`readEvidenceLookupSection(cache, section)`（约 :1119–1197）把引擎切片整段原样复制出来（整张分盘、整个西洋技法、整个 `native_layers[x]`）。
|
||||
- 工具本体：约 :1614–1662。返回 `{status, section, data, note}`。
|
||||
|
||||
段落清单 `EVIDENCE_LOOKUP_SECTIONS` 在 `frontend/src/lib/consultation-evidence-card.ts` 约 :409–455，是封闭列表。
|
||||
|
||||
BUG-1803（`docs/BUG_HISTORY.md` 约 :24378）的实测：70 个回答里，57 个一次也没调用查卡，合计 24 次调用。**因此本任务省下的量可能不大，先量再改。**
|
||||
|
||||
## 第一步：测量（先做，结果写进 PROGRESS）
|
||||
|
||||
1. 用 `frontend/tests/fixtures/consult-evidence-card-golden.json` 和 `consult-biography-backtest-golden.json` 里的公开盘，经真实链路建好 `lookupCache`，再对每一段调用 `readEvidenceLookupSection`。记下每段的字符数，按“全部段 × 每张盘”列一张表：最大、中位、合计。
|
||||
2. 计算 `run-jyotish-consultation` 结果（`toModelEvidenceView` 加 methodology）在每个领域的字符数。
|
||||
3. 按“每多一步就再发一遍此前全部内容”的规则，估算三种情况各自的总输入字符：
|
||||
- (i) 现状：1 次计算，加 k 次单段查卡，k = 1、2、3；
|
||||
- (ii) 改为一次取多段，只多 1 步；
|
||||
- (iii) 在 (ii) 的基础上再加字段过滤。
|
||||
|
||||
写成一张表。
|
||||
4. **若 (ii) 与 (iii) 相对现状，在 k = 2 时总输入字符下降不到 10%，停下选“需拍板”，附上这张表。** 由产品决定是否继续做。
|
||||
|
||||
## 第二步：改造（测量通过后）
|
||||
|
||||
1. **一次取多段**：输入改为 `{ sections: EvidenceLookupSection[] }`，数量为 1–6，去重。
|
||||
- 为兼容，保留 `{ section }` 单段写法，用 zod union 实现。
|
||||
- 返回 `{ status, results: [{ section, status, data | reason }], note }`。
|
||||
- 多段请求计 1 次调用，仍受每轮 3 次上限约束。
|
||||
2. **字段过滤（可选）**：每段可附 `fields?: string[]`，只取该段顶层对象里的这些键。
|
||||
- 未知键在结果里列入 `unknown_fields`，不报错。
|
||||
- 不给 `fields` 时，返回整段，与现状一致。
|
||||
- **不得**按长度截断数据，不得截掉数组的尾部。
|
||||
3. **改工具描述**：说明“需要几段就一次全部列出”，以及字段过滤的用法。上限说明保留不变。
|
||||
4. **runtime step**：每次调用记一条，带 `sections` 列表与返回字符数，以便验收时核对实际送达了什么。
|
||||
5. 若 `frontend/src/lib/stream-agent-response.ts`（约 :882 / :906）里查卡重启的处理依赖返回形状里的 `section` 字段，做最小改动以兼容新形状，并在 PROGRESS 说明。
|
||||
|
||||
## 不要碰
|
||||
|
||||
- 不改答题内容规则：
|
||||
- `references/**`（占星师方法文件）；
|
||||
- `references/product_runtime_contract.md`；
|
||||
- `frontend/src/lib/consult-preset-prompts.ts`；
|
||||
- `frontend/src/lib/consultation-methodology.ts`；
|
||||
- `frontend/src/mastra/index.ts` 里的 instructions。
|
||||
- `EVIDENCE_LOOKUP_SECTIONS` 只读，不增删段落。`frontend/src/lib/consultation-evidence-card.ts` 全文不改（#169、#170 的地盘）。
|
||||
- `consultation-tools.ts` 只改两处：约 :1099–1201（取数与 schema）和约 :1614–1662（工具本体）。
|
||||
- :1025–1092 的序列化归 #170；
|
||||
- `run-jyotish-consultation` 的 execute（约 :1376–1600）归 #172；
|
||||
- 工具表（约 :1702–1707）归 #173。
|
||||
- `frontend/src/app/api/consult/route.ts`，以及每轮 3 次上限的数值，都不改。
|
||||
|
||||
## 验收标准
|
||||
|
||||
1. **测量表在 PROGRESS 里**，并写明结论：多段 / 过滤相对现状的字符下降比例。
|
||||
2. **测试** `frontend/tests/consult-evidence-lookup-20260927.test.ts`（扩充），或新建 `frontend/tests/consult-evidence-lookup-batch-20261109.test.ts`：
|
||||
- 单段旧写法的返回内容与现状逐字节相同，作为兼容断言；
|
||||
- 多段一次调用返回的每段 `data`，与逐段单独调用的结果 deepEqual，即**必需字段 100% 送达、内容不变**；
|
||||
- `fields` 过滤只少键，不改值；未知键进 `unknown_fields`；
|
||||
- 多段计 1 次，第 4 次调用仍为 `refused`；
|
||||
- 超过 6 段，或段名不在清单内，被 schema 拒绝。
|
||||
3. **送达核对**：对 golden 里每张盘、每个领域，断言 `consultation-domain-registry.ts` 的 `requiredLayers` 仍全部在卡上，或可经查卡取到（`missing_route_layers` 为空）。
|
||||
4. **同源检查不退步**：`frontend/tests/consult-same-source-check-1773.test.ts` 与 `consult-same-source-independence-1773.test.ts` 通过。
|
||||
5. **体积测量**：PROGRESS 给出改前改后的模拟总输入字符（第一步的表）。真实 token 由 Claude 在验收时用 `scripts/research/consult-model-input-manifest.mts` 与盲测核对。
|
||||
6. **验收时的另外两项**（Claude 来跑）：
|
||||
- 用 `scripts/research/consult-evidence-traceability.mts` 与 `consult-same-source-check.mts` 对同一批题跑改前改后，送达率与同源结果不得下降；
|
||||
- 盲测准确度不得下降。
|
||||
7. **改既有断言**：写“原值 / 新值 / 原因”三栏注释。原因写：`#171 省 token：查卡一次多段，内容不变`。
|
||||
8. **Bug 记录**：在 `docs/BUG_HISTORY.md` 末尾追加 BUG-1814，关联 BUG-1803。
|
||||
|
||||
## 自测命令（Windows / PowerShell）
|
||||
|
||||
- `frontend\node_modules\.bin\tsc --noEmit -p frontend`
|
||||
- 在 `frontend` 下运行 `npm run lint`
|
||||
- 在 `frontend` 下运行 `node node_modules/tsx/dist/cli.mjs --test tests/consult-evidence-lookup-20260927.test.ts tests/consult-evidence-lookup-batch-20261109.test.ts tests/consult-same-source-check-1773.test.ts tests/consult-same-source-independence-1773.test.ts tests/consultation-agentic-runtime.test.ts`
|
||||
|
||||
前端全量测试由 Claude 在验收时跑。本任务不改 Python。
|
||||
|
||||
## 进度记录
|
||||
|
||||
`docs/tasks/PROGRESS-token-lookup-20261109.md`
|
||||
|
||||
## 何时停下来选“需拍板”
|
||||
|
||||
- 第一步测量显示下降不到 10%。
|
||||
- 实现多段必须改 `EVIDENCE_LOOKUP_SECTIONS`，或者必须改工具表、route.ts 时。
|
||||
- 发现模型实际上很少调用查卡，真正重复发送的是别的东西（例如 methodology 或 continuation 时再发证据，`consultation-thinking-plan.ts` 约 :661）。这时写清数字，由 Claude 另开单。
|
||||
@@ -376,6 +376,10 @@ export type ConsultationRuntimeStep = {
|
||||
status: "completed" | "failed";
|
||||
durationMs?: number;
|
||||
failureCode?: string;
|
||||
/** #171：本次查卡读了哪几段（内部记录，不进公开 runtime steps）。 */
|
||||
sections?: readonly string[];
|
||||
/** #171：本次查卡返回的字符数，验收时核对实际送达。 */
|
||||
chars?: number;
|
||||
};
|
||||
|
||||
export type ConsultationRuntimeState = {
|
||||
@@ -1196,9 +1200,42 @@ export function readEvidenceLookupSection(
|
||||
}
|
||||
}
|
||||
|
||||
const evidenceLookupInputSchema = z.object({
|
||||
section: z.enum(EVIDENCE_LOOKUP_SECTIONS),
|
||||
}).strict();
|
||||
/**
|
||||
* #171：只丢没要的顶层键，不改值、不截断数组、不按长度裁。数据不是对象时原样返回，
|
||||
* 请求的键全部记进 unknown_fields。
|
||||
*/
|
||||
function pickLookupFields(
|
||||
value: unknown,
|
||||
fields: readonly string[] | undefined,
|
||||
): { data: unknown; unknown_fields?: string[] } {
|
||||
if (!fields || fields.length === 0) return { data: value };
|
||||
const source = value && typeof value === "object" && !Array.isArray(value)
|
||||
? value as Record<string, unknown>
|
||||
: null;
|
||||
if (!source) return { data: value, unknown_fields: [...fields] };
|
||||
const data: Record<string, unknown> = {};
|
||||
const unknown: string[] = [];
|
||||
for (const key of fields) {
|
||||
if (source[key] === undefined) unknown.push(key);
|
||||
else data[key] = source[key];
|
||||
}
|
||||
return unknown.length ? { data, unknown_fields: unknown } : { data };
|
||||
}
|
||||
|
||||
// #171：导出给测试直接校验 schema（多段上限、段名清单）。
|
||||
export const evidenceLookupInputSchema = z.union([
|
||||
// 原值：`z.object({ section }).strict()`，一次只能取一段
|
||||
// 新值：单段写法保留（含可选 fields），另加一次取 2–6 段的写法；多段计 1 次调用
|
||||
// 原因：#171 省 token：查卡一次多段，内容不变
|
||||
z.object({
|
||||
section: z.enum(EVIDENCE_LOOKUP_SECTIONS),
|
||||
fields: z.array(z.string().trim().min(1).max(64)).min(1).max(24).optional(),
|
||||
}).strict(),
|
||||
z.object({
|
||||
sections: z.array(z.enum(EVIDENCE_LOOKUP_SECTIONS)).min(1).max(6),
|
||||
fields: z.array(z.string().trim().min(1).max(64)).min(1).max(24).optional(),
|
||||
}).strict(),
|
||||
]);
|
||||
|
||||
const bookSearchInputSchema = z.object({
|
||||
query: z.string().trim().min(2).max(200),
|
||||
@@ -1613,12 +1650,22 @@ export function createConsultationTools(ctx: ConsultationAgentContext) {
|
||||
});
|
||||
const lookupTool = createTool({
|
||||
id: CONSULTATION_EVIDENCE_LOOKUP_TOOL_ID,
|
||||
description: `Read one section of this request's finished chart calculation that is not on the evidence card, for example a varga the card does not carry (varga:D60), a Western layer (western:solar_return), the yoga details, Chara Dasha, Karakamsha, Argala, the dispositor chain, the D1/D9/D10 linkage, the Moon transit, or KP (parameter_sensitive: Placidus cusps and sub-lord boundaries; it may support a conditional conclusion, and a difference from the whole-sign reading is written both ways). It never calculates: it returns what run-jyotish-consultation already computed, or status unavailable. At most ${MAX_EVIDENCE_LOOKUPS_PER_TURN} calls per turn; a fourth call is refused. Call it before writing any answer text, and only when the question needs that section; the answer contract and card come from run-jyotish-consultation.`,
|
||||
description: `Read sections of this request's finished chart calculation that are not on the evidence card, for example a varga the card does not carry (varga:D60), a Western layer (western:solar_return), the yoga details, Chara Dasha, Karakamsha, Argala, the dispositor chain, the D1/D9/D10 linkage, the Moon transit, or KP (parameter_sensitive: Placidus cusps and sub-lord boundaries; it may support a conditional conclusion, and a difference from the whole-sign reading is written both ways). It never calculates: it returns what run-jyotish-consultation already computed, or status unavailable. Call it before writing any answer text, and only when the question needs that section; the answer contract and card come from run-jyotish-consultation. If a question really needs several sections, list them together in one \`sections\` call instead of fetching them one by one: one call returns all of them and still counts as a single call. Pass \`fields\` to keep only those top-level keys of a section; a key that does not exist comes back in \`unknown_fields\` instead of failing. At most ${MAX_EVIDENCE_LOOKUPS_PER_TURN} calls per turn; a fourth call is refused.`,
|
||||
inputSchema: evidenceLookupInputSchema,
|
||||
execute: async (input, context) => {
|
||||
const startedAt = Date.now();
|
||||
// 原值：一次只能取一段，每段各占一步
|
||||
// 新值：可一次取多段（去重后 1–6 段），多段仍计 1 次调用；单段写法与返回形状不变
|
||||
// 原因：#171 省 token：查卡一次多段，内容不变
|
||||
const single = "section" in input;
|
||||
const sections: EvidenceLookupSection[] = single ? [input.section] : [...new Set(input.sections)];
|
||||
const lookupNote = "If you had already started writing the answer, continue from where you stopped; never repeat text already written.";
|
||||
const refused = (status: "refused" | "unavailable", reason: string) => (
|
||||
single
|
||||
? { status, reason, section: sections[0]!, note: lookupNote }
|
||||
: { status, reason, sections, note: lookupNote }
|
||||
);
|
||||
ctx.state.evidenceLookupCallCount += 1;
|
||||
const note = "If you had already started writing the answer, continue from where you stopped; never repeat text already written.";
|
||||
if (ctx.state.evidenceLookupCallCount > MAX_EVIDENCE_LOOKUPS_PER_TURN) {
|
||||
if (ctx.state.evidenceLookupCallCount === MAX_EVIDENCE_LOOKUPS_PER_TURN + 1) {
|
||||
appendConsultationRuntimeStep(ctx.state, {
|
||||
@@ -1629,7 +1676,7 @@ export function createConsultationTools(ctx: ConsultationAgentContext) {
|
||||
failureCode: "lookup_limit_reached",
|
||||
});
|
||||
}
|
||||
return { status: "refused" as const, reason: "lookup_limit_reached", section: input.section, note };
|
||||
return refused("refused", "lookup_limit_reached");
|
||||
}
|
||||
if (!lookupCache) {
|
||||
appendConsultationRuntimeStep(ctx.state, {
|
||||
@@ -1639,24 +1686,59 @@ export function createConsultationTools(ctx: ConsultationAgentContext) {
|
||||
durationMs: Math.max(0, Date.now() - startedAt),
|
||||
failureCode: "calculation_not_cached",
|
||||
});
|
||||
return { status: "unavailable" as const, reason: "calculation_not_in_request_cache", section: input.section, note };
|
||||
return refused("unavailable", "calculation_not_in_request_cache");
|
||||
}
|
||||
const cache = lookupCache;
|
||||
const labelSection = sections.length > 1 ? `${sections[0]!} (+${sections.length - 1})` : sections[0]!;
|
||||
await (context as { writer?: { custom?: (value: unknown) => unknown } } | undefined)?.writer?.custom?.({
|
||||
type: "data-jyotish-activity",
|
||||
data: { phase: "answer-composition", label: evidenceLookupActivityLabel(input.section) },
|
||||
data: { phase: "answer-composition", label: evidenceLookupActivityLabel(labelSection) },
|
||||
});
|
||||
const data = readEvidenceLookupSection(lookupCache, input.section);
|
||||
const found = data !== undefined && data !== null;
|
||||
if (single) {
|
||||
// 单段写法（带不带 fields 都一样）保持旧的返回形状，不带 fields 时逐字节相同。
|
||||
const raw = readEvidenceLookupSection(cache, sections[0]!);
|
||||
const found = raw !== undefined && raw !== null;
|
||||
const picked = pickLookupFields(raw, input.fields);
|
||||
const data = picked.data;
|
||||
appendConsultationRuntimeStep(ctx.state, {
|
||||
kind: "tool",
|
||||
name: CONSULTATION_EVIDENCE_LOOKUP_TOOL_ID,
|
||||
status: found ? "completed" : "failed",
|
||||
durationMs: Math.max(0, Date.now() - startedAt),
|
||||
sections,
|
||||
chars: JSON.stringify(data ?? null).length,
|
||||
...(found ? {} : { failureCode: "section_not_computed" }),
|
||||
});
|
||||
if (!found) {
|
||||
return { status: "unavailable" as const, reason: "section_not_computed", section: sections[0]!, note: lookupNote };
|
||||
}
|
||||
return picked.unknown_fields
|
||||
? { status: "ok" as const, section: sections[0]!, data, unknown_fields: picked.unknown_fields, note: lookupNote }
|
||||
: { status: "ok" as const, section: sections[0]!, data, note: lookupNote };
|
||||
}
|
||||
const results = sections.map((section) => {
|
||||
const raw = readEvidenceLookupSection(cache, section);
|
||||
if (raw === undefined || raw === null) {
|
||||
return { section, status: "unavailable" as const, reason: "section_not_computed" };
|
||||
}
|
||||
const picked = pickLookupFields(raw, input.fields);
|
||||
return picked.unknown_fields
|
||||
? { section, status: "ok" as const, data: picked.data, unknown_fields: picked.unknown_fields }
|
||||
: { section, status: "ok" as const, data: picked.data };
|
||||
});
|
||||
const delivered = results.filter((row) => row.status === "ok").length;
|
||||
appendConsultationRuntimeStep(ctx.state, {
|
||||
kind: "tool",
|
||||
name: CONSULTATION_EVIDENCE_LOOKUP_TOOL_ID,
|
||||
status: found ? "completed" : "failed",
|
||||
status: delivered > 0 ? "completed" : "failed",
|
||||
durationMs: Math.max(0, Date.now() - startedAt),
|
||||
...(found ? {} : { failureCode: "section_not_computed" }),
|
||||
sections,
|
||||
chars: JSON.stringify(results).length,
|
||||
...(delivered > 0 ? {} : { failureCode: "section_not_computed" }),
|
||||
});
|
||||
return found
|
||||
? { status: "ok" as const, section: input.section, data, note }
|
||||
: { status: "unavailable" as const, reason: "section_not_computed", section: input.section, note };
|
||||
return delivered > 0
|
||||
? { status: "ok" as const, results, note: lookupNote }
|
||||
: { status: "unavailable" as const, reason: "section_not_computed", results, note: lookupNote };
|
||||
},
|
||||
});
|
||||
let bookSearchCount = 0;
|
||||
|
||||
@@ -0,0 +1,218 @@
|
||||
/**
|
||||
* #171 省 token：查卡一次可取多段。
|
||||
*
|
||||
* 原值：一次只能取一段，每多一段就多一步，整份计算（约 9.3 万字符）随每一步重发。
|
||||
* 新值：一次可列 2–6 段，仍计 1 次调用；每段 data 与单独取时逐字节相同；
|
||||
* 可选 fields 只丢没要的顶层键，不截断数据。
|
||||
* 原因：#171 省 token：查卡一次多段，内容不变
|
||||
*/
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import test from "node:test";
|
||||
|
||||
import {
|
||||
CONSULTATION_EVIDENCE_LOOKUP_TOOL_ID,
|
||||
MAX_EVIDENCE_LOOKUPS_PER_TURN,
|
||||
createConsultationRuntimeState,
|
||||
createConsultationTools,
|
||||
evidenceLookupInputSchema,
|
||||
} from "../src/mastra/consultation-tools.ts";
|
||||
|
||||
type Json = Record<string, unknown>;
|
||||
const golden = JSON.parse(readFileSync(
|
||||
new URL("./fixtures/consult-evidence-card-golden.json", import.meta.url),
|
||||
"utf8",
|
||||
)) as { charts: Array<{ id: string; workflow: Json }> };
|
||||
const workflow = golden.charts.find((chart) => chart.id === "steve_jobs")!.workflow;
|
||||
|
||||
const SECTIONS = ["yogas", "varga:D60", "western:natal", "shadbala"] as const;
|
||||
|
||||
/** 每张盘一次计算；查卡调用有每轮 3 次的上限，所以每次取样都开一份新的工具实例。 */
|
||||
async function tools(domain = "career") {
|
||||
const state = createConsultationRuntimeState();
|
||||
const created = createConsultationTools({
|
||||
userId: "u",
|
||||
sessionId: "s",
|
||||
requestId: "lookup-batch-171",
|
||||
consultationMode: "verified_chart",
|
||||
serverChart: {
|
||||
name: "public",
|
||||
toolInput: {
|
||||
year: 1955, month: 2, day: 24, hour: 19, minute: 15, city: "San Francisco",
|
||||
lat: 37.77, lon: -122.42, tz: -8, ayanamsa: "raman",
|
||||
},
|
||||
truth: { birthTimeSource: "reported" },
|
||||
} as never,
|
||||
state,
|
||||
runWorkflow: async () => structuredClone(workflow) as never,
|
||||
});
|
||||
await created["run-jyotish-consultation"].execute!(
|
||||
{ question: "我事业如何", domains: [domain] } as never,
|
||||
{ writer: { custom: async () => {} } } as never,
|
||||
);
|
||||
const lookup = (input: Json) => created[CONSULTATION_EVIDENCE_LOOKUP_TOOL_ID].execute!(
|
||||
input as never,
|
||||
{ writer: { custom: async () => {} } } as never,
|
||||
) as Promise<Json>;
|
||||
const view = async () => created["run-jyotish-consultation"].execute!(
|
||||
{ question: "我事业如何", domains: [domain] } as never,
|
||||
{ writer: { custom: async () => {} } } as never,
|
||||
) as Promise<Json>;
|
||||
return { state, lookup, view };
|
||||
}
|
||||
|
||||
async function setupWithTools() {
|
||||
const state = createConsultationRuntimeState();
|
||||
const created = createConsultationTools({
|
||||
userId: "u",
|
||||
sessionId: "s",
|
||||
requestId: "lookup-batch-171-tools",
|
||||
consultationMode: "verified_chart",
|
||||
serverChart: {
|
||||
name: "public",
|
||||
toolInput: {
|
||||
year: 1955, month: 2, day: 24, hour: 19, minute: 15, city: "San Francisco",
|
||||
lat: 37.77, lon: -122.42, tz: -8, ayanamsa: "raman",
|
||||
},
|
||||
truth: { birthTimeSource: "reported" },
|
||||
} as never,
|
||||
state,
|
||||
runWorkflow: async () => structuredClone(workflow) as never,
|
||||
});
|
||||
return { tools: created, state };
|
||||
}
|
||||
|
||||
async function singleSection(section: string) {
|
||||
const { lookup } = await tools();
|
||||
return lookup({ section });
|
||||
}
|
||||
|
||||
test("the single-section shape is unchanged", async () => {
|
||||
const result = await singleSection("yogas");
|
||||
// 旧的返回形状与键序不变，仍是 {status, section, data, note}。
|
||||
assert.deepEqual(Object.keys(result), ["status", "section", "data", "note"]);
|
||||
assert.equal(result.status, "ok");
|
||||
assert.equal(result.section, "yogas");
|
||||
assert.ok(result.note);
|
||||
assert.ok(result.data !== undefined);
|
||||
});
|
||||
|
||||
test("one batch call returns every requested section exactly as single calls do", async () => {
|
||||
const perSection: Record<string, unknown> = {};
|
||||
for (const section of SECTIONS) perSection[section] = (await singleSection(section)).data;
|
||||
|
||||
const { state, lookup } = await tools();
|
||||
const result = await lookup({ sections: [...SECTIONS] });
|
||||
const results = result.results as Array<Json>;
|
||||
assert.equal(result.status, "ok");
|
||||
assert.equal(results.length, SECTIONS.length);
|
||||
for (const [index, section] of SECTIONS.entries()) {
|
||||
assert.equal(results[index]!.section, section);
|
||||
assert.equal(results[index]!.status, "ok");
|
||||
assert.deepEqual(results[index]!.data, perSection[section], section);
|
||||
}
|
||||
// 多段只算 1 次调用。
|
||||
assert.equal(state.evidenceLookupCallCount, 1);
|
||||
});
|
||||
|
||||
test("a batch call and a single call share the same per-turn budget", async () => {
|
||||
const { state, lookup } = await tools();
|
||||
await lookup({ sections: ["yogas", "varga:D60"] });
|
||||
await lookup({ sections: ["western:natal", "shadbala"] });
|
||||
await lookup({ section: "yogas" });
|
||||
const refused = await lookup({ section: "yogas" });
|
||||
assert.equal(state.evidenceLookupCallCount, 4);
|
||||
assert.equal(refused.status, "refused");
|
||||
assert.equal(refused.reason, "lookup_limit_reached");
|
||||
assert.equal(MAX_EVIDENCE_LOOKUPS_PER_TURN, 3);
|
||||
});
|
||||
|
||||
test("fields only drop unrequested keys and report unknown ones", async () => {
|
||||
const whole = (await singleSection("shadbala")).data as Json;
|
||||
const { lookup } = await tools();
|
||||
const filtered = await lookup({ section: "shadbala", fields: ["summary", "no_such_field"] });
|
||||
const data = filtered.data as Json;
|
||||
assert.deepEqual(Object.keys(data), ["summary"]);
|
||||
assert.deepEqual(data.summary, whole.summary);
|
||||
assert.deepEqual(filtered.unknown_fields, ["no_such_field"]);
|
||||
// 未知键不报错，整次调用仍然成功，返回形状与单段一致。
|
||||
assert.equal(filtered.status, "ok");
|
||||
assert.deepEqual(Object.keys(filtered), ["status", "section", "data", "unknown_fields", "note"]);
|
||||
});
|
||||
|
||||
test("fields never shorten an array or change a value", async () => {
|
||||
const whole = (await singleSection("yogas")).data;
|
||||
const { lookup } = await tools();
|
||||
const filtered = await lookup({ section: "yogas", fields: Object.keys(whole as Json) });
|
||||
assert.deepEqual(filtered.data, whole);
|
||||
assert.equal(filtered.unknown_fields, undefined);
|
||||
});
|
||||
|
||||
test("more than six sections or an unknown section never reaches the tool", () => {
|
||||
assert.equal(evidenceLookupInputSchema.safeParse({
|
||||
sections: ["yogas", "yogas", "varga:D60", "western:natal", "shadbala", "transits", "argala"],
|
||||
}).success, false);
|
||||
assert.equal(evidenceLookupInputSchema.safeParse({ section: "varga:D99" }).success, false);
|
||||
// 原值：sections 至少 2 段（schema min(2)）
|
||||
// 新值：1–6 段都接受；只有 1 段时走 results 数组，旧的 { section } 写法照旧
|
||||
// 原因：#171 省 token：查卡一次多段，内容不变（FIX-token-lookup-1：min(2) 会让 1 段的
|
||||
// 合法意图被 schema 拒掉，模型白走一步）
|
||||
assert.equal(evidenceLookupInputSchema.safeParse({ sections: ["yogas"] }).success, true);
|
||||
assert.equal(evidenceLookupInputSchema.safeParse({ sections: [] }).success, false);
|
||||
assert.equal(evidenceLookupInputSchema.safeParse({ section: "yogas", extra: true }).success, false);
|
||||
assert.equal(evidenceLookupInputSchema.safeParse({ section: "yogas", sections: ["yogas", "transits"] }).success, false);
|
||||
});
|
||||
|
||||
test("every domain still gets its needed layers, on the card or through the lookup", async () => {
|
||||
for (const chart of golden.charts) {
|
||||
for (const domain of ["career", "marriage", "health"] as const) {
|
||||
const state = createConsultationRuntimeState();
|
||||
const created = createConsultationTools({
|
||||
userId: "u",
|
||||
sessionId: "s",
|
||||
requestId: `coverage-${chart.id}-${domain}`,
|
||||
consultationMode: "verified_chart",
|
||||
serverChart: {
|
||||
name: "public",
|
||||
toolInput: {
|
||||
year: 1955, month: 2, day: 24, hour: 19, minute: 15, city: "San Francisco",
|
||||
lat: 37.77, lon: -122.42, tz: -8, ayanamsa: "raman",
|
||||
},
|
||||
truth: { birthTimeSource: "reported" },
|
||||
} as never,
|
||||
state,
|
||||
runWorkflow: async () => structuredClone(chart.workflow) as never,
|
||||
});
|
||||
const view = await created["run-jyotish-consultation"].execute!(
|
||||
{ question: "这一块怎么样", domains: [domain] } as never,
|
||||
{ writer: { custom: async () => {} } } as never,
|
||||
) as Json;
|
||||
const contract = view.evidence_contract as Json;
|
||||
// 多段查卡只是少发几遍，不改变送达：必需层要么在卡上，要么能经查卡取到。
|
||||
const missing = contract.missing_route_layers as unknown;
|
||||
assert.deepEqual(missing, [], `${chart.id}/${domain}: ${JSON.stringify(missing)}`);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
test("a one-item sections call returns the same data as the single-section shape", async () => {
|
||||
const single = await singleSection("yogas");
|
||||
const { lookup } = await tools();
|
||||
const batched = await lookup({ sections: ["yogas"] });
|
||||
const results = batched.results as Array<Json>;
|
||||
assert.equal(results.length, 1);
|
||||
assert.equal(results[0]!.section, "yogas");
|
||||
assert.deepEqual(results[0]!.data, single.data);
|
||||
});
|
||||
|
||||
test("the tool description asks for a section only when the question needs it", async () => {
|
||||
const { tools: created } = await setupWithTools();
|
||||
const description = String(
|
||||
(created[CONSULTATION_EVIDENCE_LOOKUP_TOOL_ID] as unknown as { description?: unknown }).description ?? "",
|
||||
);
|
||||
assert.match(description, /only when the question needs/);
|
||||
assert.doesNotMatch(description, /List every section/);
|
||||
assert.doesNotMatch(description, /2-6/);
|
||||
assert.match(description, /At most 3 calls per turn/);
|
||||
});
|
||||
@@ -0,0 +1,57 @@
|
||||
/**
|
||||
* #171 每段查卡的实测字符数（2026-11-09）。每段各开一份工具实例，避开每轮 3 次上限。
|
||||
* 运行：frontend > node node_modules/tsx/dist/cli.mjs scripts/research/consult-lookup-section-sizes-171.mts
|
||||
*/
|
||||
import { readFileSync, writeFileSync } from "node:fs";
|
||||
|
||||
import { EVIDENCE_LOOKUP_SECTIONS } from "../../src/lib/consultation-evidence-card.ts";
|
||||
import {
|
||||
CONSULTATION_EVIDENCE_LOOKUP_TOOL_ID,
|
||||
createConsultationRuntimeState,
|
||||
createConsultationTools,
|
||||
} from "../../src/mastra/consultation-tools.ts";
|
||||
|
||||
type Json = Record<string, unknown>;
|
||||
const golden = JSON.parse(readFileSync(
|
||||
new URL("../fixtures/consult-evidence-card-golden.json", import.meta.url),
|
||||
"utf8",
|
||||
)) as { charts: Array<{ id: string; workflow: Json }> };
|
||||
|
||||
const rows: Array<{ chart: string; section: string; chars: number }> = [];
|
||||
|
||||
for (const chart of golden.charts) {
|
||||
for (const section of EVIDENCE_LOOKUP_SECTIONS) {
|
||||
const state = createConsultationRuntimeState();
|
||||
const created = createConsultationTools({
|
||||
userId: "u", sessionId: "s", requestId: `size-${chart.id}-${section}`,
|
||||
consultationMode: "verified_chart",
|
||||
serverChart: {
|
||||
name: "public",
|
||||
toolInput: {
|
||||
year: 1955, month: 2, day: 24, hour: 19, minute: 15, city: "San Francisco",
|
||||
lat: 37.77, lon: -122.42, tz: -8, ayanamsa: "raman",
|
||||
},
|
||||
truth: { birthTimeSource: "reported" },
|
||||
} as never,
|
||||
state,
|
||||
runWorkflow: async () => structuredClone(chart.workflow) as never,
|
||||
});
|
||||
await created["run-jyotish-consultation"].execute!(
|
||||
{ question: "这一块怎么样", domains: ["career"] } as never,
|
||||
{ writer: { custom: async () => {} } } as never,
|
||||
);
|
||||
const result = await created[CONSULTATION_EVIDENCE_LOOKUP_TOOL_ID].execute!(
|
||||
{ section } as never,
|
||||
{ writer: { custom: async () => {} } } as never,
|
||||
) as Json;
|
||||
rows.push({
|
||||
chart: chart.id,
|
||||
section,
|
||||
chars: result.status === "ok" ? JSON.stringify(result).length : 0,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const out = new URL("../../../docs/research/token_lookup_171_section_sizes.json", import.meta.url);
|
||||
writeFileSync(out, `${JSON.stringify(rows, null, 2)}\n`, "utf8");
|
||||
console.log(JSON.stringify(rows.filter((row) => row.chars > 0).sort((a, b) => b.chars - a.chars).slice(0, 10), null, 2));
|
||||
@@ -0,0 +1,56 @@
|
||||
/**
|
||||
* #171 第二步测量：用真实 agent 链路量「计算工具结果」与「单段查卡结果」各多少字符，
|
||||
* 再按「每多一步就把此前全部内容再发一遍」估算三种方案的输入总量（2026-11-09）。
|
||||
*
|
||||
* 运行：frontend > node node_modules/tsx/dist/cli.mjs scripts/research/consult-lookup-token-probe-171.mts
|
||||
*/
|
||||
import { readFileSync } from "node:fs";
|
||||
|
||||
import { CONSULTATION_EVIDENCE_LOOKUP_TOOL_ID } from "../../src/mastra/consultation-tools.ts";
|
||||
|
||||
import { pieces, runNatalAgent, type Turn } from "../consult-natal-agent-test-support.ts";
|
||||
|
||||
const golden = JSON.parse(readFileSync(
|
||||
new URL("../fixtures/consult-evidence-card-golden.json", import.meta.url),
|
||||
"utf8",
|
||||
)) as { charts: Array<{ id: string; workflow: Record<string, unknown> }> };
|
||||
const workflow = golden.charts.find((chart) => chart.id === "steve_jobs")!.workflow;
|
||||
|
||||
const ANSWER = "## 依据\r\n这几个月先稳住节奏。\r\n";
|
||||
|
||||
const calcTurn: Turn = {
|
||||
parts: [{ tool: { name: "run-jyotish-consultation", input: { question: "我事业如何", domains: ["career"] } } }],
|
||||
finish: "tool-calls",
|
||||
};
|
||||
const lookup = (section: string): Turn => ({
|
||||
parts: [{ tool: { name: CONSULTATION_EVIDENCE_LOOKUP_TOOL_ID, input: { section } } }],
|
||||
finish: "tool-calls",
|
||||
});
|
||||
const answerTurn: Turn = { parts: pieces(ANSWER), finish: "stop" };
|
||||
|
||||
function toolResultChars(prompt: unknown[] | undefined): Array<{ name: string; chars: number }> {
|
||||
const out: Array<{ name: string; chars: number }> = [];
|
||||
for (const message of prompt ?? []) {
|
||||
const value = message as { role?: string; content?: unknown };
|
||||
const text = JSON.stringify(value.content ?? "");
|
||||
if (value.role !== "tool") continue;
|
||||
const name = text.includes(CONSULTATION_EVIDENCE_LOOKUP_TOOL_ID)
|
||||
? CONSULTATION_EVIDENCE_LOOKUP_TOOL_ID
|
||||
: text.includes("run-jyotish-consultation") ? "run-jyotish-consultation" : "other";
|
||||
out.push({ name, chars: text.length });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
const result = await runNatalAgent(
|
||||
[calcTurn, lookup("varga:D60"), lookup("yogas"), answerTurn],
|
||||
{ workflow, theme: "career", question: "我事业如何" },
|
||||
);
|
||||
|
||||
const steps = result.prompts.map((prompt, index) => ({
|
||||
step: index,
|
||||
promptChars: JSON.stringify(prompt).length,
|
||||
toolResults: toolResultChars(prompt),
|
||||
}));
|
||||
|
||||
console.log(JSON.stringify({ calls: result.calls, steps }, null, 2));
|
||||
Reference in New Issue
Block a user