Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Jesse_ChenandCursor
|
0b68fa97c4 |
+2
-4
@@ -1,10 +1,8 @@
|
||||
# BLOCKED
|
||||
|
||||
## Jev 意图 v2：staging 库与 Flash 凭据（2026-09-27）
|
||||
## ~~Jev 意图 v2：staging 库与 Flash 凭据（2026-09-27）~~
|
||||
|
||||
- 缺 staging 库。主机 `118.26.111.127` 端口 22 通，仓库变量里的用户是 `ubuntu`（不是 README 示例里的 `deploy`）。本机三把私钥（ed25519、rsa、jd pem）对 `ubuntu` 和 `deploy` 都被拒，没有 `~/.ssh/jyotisha-staging`。Gitea 里的 `STAGING_SSH_PRIVATE_KEY` 保存后读不回。本机 Docker 里的 Postgres 没有 `agentic_rectification_turns`。09-19 交来的 157 条没有 `turn_id` / `case_id`。替代证据：V0 用那次缓存重算，有上一轮 = 0，结论写缺数据，没有用空上一轮去打 V1/V2。
|
||||
- DeepSeek key 已通：合成样本 1 次，响应 `model=deepseek-flash`，1067 ms。Flash V0 仍用 09-19 缓存（157 条 intent 89.2%）。Flash+V1 还要等上一轮，未跑。
|
||||
- Jev key 已通：合成样本 V0 / V2 各 1 次，响应 `model=jev-1.13.0`。这几次不进计分。SDK 仍是 `typesafe-sdk 0.7.0`。
|
||||
已解除（2026-09-27，本机）。`deploy@118.26.111.127` 与 `~/.ssh/jyotisha-staging` 能进 staging，抽出 715 条带 `case_id` 的来源 B（有上一轮 634）。上一笔「ubuntu 才是用户、deploy 被拒」是当时那台机器没有这把密钥。Jev 与 Flash 都打完，结论在 `docs/research/jev_intent_2026_09_27.md`：**缺数据**（来源 B / C 同层差超过 10pp）。09-19 的 `source_b.jsonl` 仍不在本机，旧 157 条对不回 `turn_id`，记在进度偏离里，不再挡抽取。
|
||||
|
||||
## TASK-rectification-telemetry：真实 PostgreSQL 测试与登录态后台待验（2026-09-26）
|
||||
|
||||
|
||||
+337919
-99456
File diff suppressed because it is too large
Load Diff
@@ -1,52 +1,102 @@
|
||||
# TypeSafe Jev 意图分类 · 上一轮 state 对照（2026-09-27）
|
||||
|
||||
- 任务：`docs/tasks/TASK-rectification-jev-intent-classifier-research-v2-20260927.md`
|
||||
- 基线：`origin/staging` @ `710c848b`
|
||||
- 模型：`jev-1.13.0`；SDK `typesafe-sdk 0.7.0`
|
||||
- 结论：**缺数据**。上一轮变量没有测全，不能判过门。已有的 V0 数字只作对照，不代替 V1/V2。
|
||||
|
||||
## 样本
|
||||
|
||||
- 来源 B：157 条。有上一轮 0，无上一轮 157。
|
||||
- 层：点选 17 / 采集 107 / 无焦点 33。
|
||||
- gold 来源：{"legacy_file": 157}。
|
||||
- 链式与 gold 上界都未跑。09-19 文件没有 turn_id / case_id，本机没有 staging 库。
|
||||
|
||||
## 来源 B
|
||||
|
||||
| 变体 | n | intent | answer_class | dated | 高置信错误 | 低置信召回 | 无焦点 unclear→answer |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
|
||||
| Jev V0（09-19 缓存） | 157 | 91.1% | 98.0% | 91.7% | 6.4% | 24.1% | 7 / 无焦点高置信错误 12.1% |
|
||||
| Flash V0（09-19 缓存，生产提示） | 157 | 89.2% | 97.0% | 93.6% | — | — | 5 / 无焦点高置信错误 — |
|
||||
| Jev V1 | — | — | — | — | — | — | — |
|
||||
| Jev V2 链式 | — | — | — | — | — | — | — |
|
||||
| Jev V2 gold 上界 | — | — | — | — | — | — | — |
|
||||
| Flash + V1 | — | — | — | — | — | — | — |
|
||||
|
||||
Jev V0 自洽率：96.2%。
|
||||
|
||||
## 来源 C 回归锚（只 V0）
|
||||
|
||||
- choice：重算 99.0%，09-19 公布 99.0%，差 0.0 pp（n=400）。
|
||||
- collect：重算 92.2%，09-19 公布 92.2%，差 0.05 pp（n=400）。
|
||||
- none：重算 96.0%，09-19 公布 96.0%，差 0.0 pp（n=100）。
|
||||
- 来源 C 自洽率：99.0%。
|
||||
|
||||
## 无焦点层混淆（来源 B，V0）
|
||||
|
||||
计数来自报告 JSON，不含原文。
|
||||
|
||||
| gold \ pred | provide_new_evidence | stop_rectification | ask_about_result | unclear | answer_current_focus |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| provide_new_evidence | 24 | 0 | 0 | 0 | 0 |
|
||||
| stop_rectification | 0 | 0 | 0 | 0 | 0 |
|
||||
| ask_about_result | 0 | 0 | 1 | 0 | 0 |
|
||||
| unclear | 0 | 0 | 0 | 1 | 7 |
|
||||
| answer_current_focus | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
## 写不出的项
|
||||
|
||||
n_有上一轮 0 < 100；V1 没有预测；V2 链式没有预测；V2 gold 上界没有预测；Flash+V1 没有预测；来源 B 与来源 C 同层 intent 差 > 10pp
|
||||
|
||||
V1、V2、Flash+V1 要等 staging 库抽出带 `case_id` 的上一轮，并且本机有 `DEEPSEEK_API_KEY` 之后才能补。门槛不放宽。
|
||||
|
||||
# TypeSafe Jev 意图分类 · 上一轮 state 对照（2026-09-27）
|
||||
|
||||
- 任务：`docs/tasks/TASK-rectification-jev-intent-classifier-research-v2-20260927.md`
|
||||
- 基线：`origin/staging` @ `fdb7087b`（任务书提交 `710c848b` 在其历史上）
|
||||
- 模型：`jev-1.13.0`；SDK `typesafe-sdk 0.7.0`
|
||||
- 结论：**缺数据**。来源 B 与来源 C 同层 intent 差超过 10pp（点选 23.7 pp，无焦点 38.8 pp）。按任务书这一项写缺数据，不把这批真人样本外推成过门。只看这批来源 B，高置信错误 8.0% > 3%；低置信召回 27.2% < 60%。
|
||||
|
||||
## 样本
|
||||
|
||||
- 来源 B：715 条。有上一轮 634，无上一轮 81。
|
||||
- 层：点选 269 / 采集 336 / 无焦点 110。
|
||||
- gold 来源：{"executor_read": 565, "option_echo": 150}。
|
||||
- 09-19 的 source_b.jsonl 不在本机，旧 157 条按 turn_id 对回 0 条。其余 gold 为执行方按生产标注规范阅读后写入；点选回显另计。
|
||||
- V2 链式的 previous_decision 来自同案上一条 Jev V2 输出（链式，自喂）。V2 gold 上界来自同案上一条人工 gold。案件第一轮 previous_turn 为 null，不喂 previous_decision。
|
||||
|
||||
## 来源 B
|
||||
|
||||
| 变体 | n | intent | answer_class | dated | 高置信错误 | 低置信召回 | 无焦点 unclear→answer |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
|
||||
| Jev V0 | 715 | 79.4% | 98.9% | 95.2% | 12.9% | 15.1% | 43 / 无焦点高置信错误 32.7% |
|
||||
| Flash V0（生产提示） | 715 | 75.7% | 99.3% | 97.8% | — | — | 42 / 无焦点高置信错误 — |
|
||||
| Jev V1 | 715 | 75.0% | 92.8% | 87.0% | 9.4% | 31.9% | 32 / 无焦点高置信错误 11.8% |
|
||||
| Jev V2 链式（自喂） | 715 | 75.9% | 96.6% | 88.7% | 8.0% | 27.2% | 33 / 无焦点高置信错误 16.4% |
|
||||
| Jev V2 gold 上界 | 715 | 75.7% | 96.6% | 89.0% | 7.7% | 28.8% | 34 / 无焦点高置信错误 13.6% |
|
||||
| Flash + V1 | 715 | 74.0% | 99.1% | 95.0% | — | — | 43 / 无焦点高置信错误 — |
|
||||
|
||||
自洽率：V0 98.9%；V1 94.3%；V2 链式 93.3%；V2 gold 93.8%。
|
||||
|
||||
有上一轮 / 无上一轮（intent；Jev 另给高置信错误）：
|
||||
|
||||
| 变体 | 有上一轮 n | intent | 高置信错误 | 无上一轮 n | intent | 高置信错误 |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| Jev V0 | 634 | 77.0% | 14.0% | 81 | 98.8% | 3.7% |
|
||||
| Jev V1 | 634 | 71.9% | 10.3% | 81 | 98.8% | 2.5% |
|
||||
| Jev V2 链式 | 634 | 73.2% | 8.8% | 81 | 97.5% | 1.2% |
|
||||
| Jev V2 gold | 634 | 72.7% | 8.5% | 81 | 98.8% | 1.2% |
|
||||
| Flash V0 | 634 | 72.9% | — | 81 | 97.5% | — |
|
||||
| Flash + V1 | 634 | 71.1% | — | 81 | 96.3% | — |
|
||||
|
||||
延迟与输入 token（来源 B，第一次）：
|
||||
|
||||
| 变体 | 中位 ms | P95 ms | 次均 input tok |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| Jev V0 | 1178 | 1376 | 1028.0 |
|
||||
| Jev V1 | 1187 | 1358 | 1161.8 |
|
||||
| Jev V2 链式 | 1206 | 1362 | 1313.6 |
|
||||
| Jev V2 gold | 1207 | 1399 | 1313.0 |
|
||||
| Flash V0 | 705 | 989 | 492.7 |
|
||||
| Flash + V1 | 717 | 1016 | 552.8 |
|
||||
|
||||
无焦点层 intent：V0 58.2%，V2 链式 46.4%。高置信错误从 32.7% 到 16.4%。
|
||||
V2 链式承接分：中位 0.16，最大 0.86，≥ 0.9 的 0 条，实际沿用上一轮 intent 的 0 条。
|
||||
|
||||
去掉点选回显后，V2 链式 n=565，intent 69.6%，高置信错误 8.3%，低置信召回 28.5%。门槛仍按全量样本。
|
||||
|
||||
## 来源 C 回归锚（只 V0，跑一次）
|
||||
|
||||
- 点选：来源 C 98.8%，09-19 公布 99.0%，差 -0.25 pp（n=400）。来源 B 75.1%，与来源 C 差 23.7 pp（n=269）。
|
||||
- 采集：来源 C 92.5%，09-19 公布 92.2%，差 0.3 pp（n=400）。来源 B 89.9%，与来源 C 差 2.6 pp（n=336）。
|
||||
- 无焦点：来源 C 97.0%，09-19 公布 96.0%，差 1.0 pp（n=100）。来源 B 58.2%，与来源 C 差 38.8 pp（n=110）。
|
||||
- 来源 C 自洽率：—。
|
||||
|
||||
## 无焦点层混淆（来源 B）
|
||||
|
||||
计数来自报告 JSON，不含原文。
|
||||
|
||||
### V0
|
||||
|
||||
| gold \ pred | provide_new_evidence | stop_rectification | ask_about_result | unclear | answer_current_focus |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| provide_new_evidence | 60 | 0 | 0 | 0 | 0 |
|
||||
| stop_rectification | 0 | 0 | 0 | 0 | 0 |
|
||||
| ask_about_result | 0 | 0 | 2 | 2 | 0 |
|
||||
| unclear | 1 | 0 | 0 | 2 | 43 |
|
||||
| answer_current_focus | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
### V1
|
||||
|
||||
| gold \ pred | provide_new_evidence | stop_rectification | ask_about_result | unclear | answer_current_focus |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| provide_new_evidence | 45 | 0 | 0 | 0 | 15 |
|
||||
| stop_rectification | 0 | 0 | 0 | 0 | 0 |
|
||||
| ask_about_result | 2 | 0 | 0 | 1 | 1 |
|
||||
| unclear | 11 | 0 | 0 | 3 | 32 |
|
||||
| answer_current_focus | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
### V2 链式
|
||||
|
||||
| gold \ pred | provide_new_evidence | stop_rectification | ask_about_result | unclear | answer_current_focus |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| provide_new_evidence | 47 | 0 | 0 | 0 | 13 |
|
||||
| stop_rectification | 0 | 0 | 0 | 0 | 0 |
|
||||
| ask_about_result | 1 | 0 | 1 | 1 | 1 |
|
||||
| unclear | 10 | 0 | 0 | 3 | 33 |
|
||||
| answer_current_focus | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
## 写不出的项
|
||||
|
||||
来源 B 与来源 C 同层 intent 差 > 10pp
|
||||
|
||||
来源 B 的 Jev 八次合计 6887443 input tok，按 $0.042/M 约 $0.2893。来源 C 一次 939273 input tok，约 $0.0394。Flash 两次合计 747581 input tok，不按 Jev 单价计。响应 model：['jev-1.13.0']。
|
||||
|
||||
|
||||
@@ -1,45 +1,54 @@
|
||||
# 进度 · Jev 意图分类 v2：上一轮放进 state（2026-09-27）
|
||||
|
||||
- 分支：`codex/jev-intent-v2-20260927`
|
||||
- 基线：`origin/staging` @ `710c848b`
|
||||
- 基线：`origin/staging` @ `fdb7087b`（任务书提交 `710c848b` 在其历史上；本机 fetch 后 staging 尖端是 `fdb7087b`）
|
||||
- 任务书：`docs/tasks/TASK-rectification-jev-intent-classifier-research-v2-20260927.md`
|
||||
- 报告：`docs/research/jev_intent_2026_09_27.md` / `.json`
|
||||
- 结论：**缺数据**。有上一轮 0 条，V1/V2 没有数字，不能判过门。
|
||||
- 结论：**缺数据**。来源 B 与来源 C 同层 intent 差超过 10pp（点选 23.7 pp，无焦点 38.8 pp）。只看这批来源 B，V2 链式高置信错误 8.0% > 3%，低置信召回 27.2% < 60%，同样过不了门。
|
||||
|
||||
## 已跑到的数
|
||||
|
||||
| 项 | 结果 |
|
||||
| --- | --- |
|
||||
| 来源 B | 157（点选 17 / 采集 107 / 无焦点 33）。gold 全部来自 09-19 文件，不是新标 |
|
||||
| 有上一轮 / 无上一轮 | 0 / 157 |
|
||||
| Jev V0 intent | 91.1%（自洽 96.2%，高置信错误 6.4%，低置信召回 24.1%） |
|
||||
| Flash V0 intent | 89.2%（09-19 缓存，生产提示；Flash 没有置信度） |
|
||||
| 无焦点 `unclear → answer_current_focus` | Jev V0 = 7，Flash V0 = 5 |
|
||||
| 来源 C V0 相对 09-19 公布值 | 点选 0 pp，采集 +0.05 pp（公布值四舍五入到 92.2%），无焦点 0 pp |
|
||||
| 来源 B | 715（点选 269 / 采集 336 / 无焦点 110）。有上一轮 634，无上一轮 81 |
|
||||
| gold | 执行方阅读 565；点选回显 150。09-19 文件不在本机，按 turn_id 对回 0 |
|
||||
| Jev V0 | intent 79.4%，高置信错误 12.9%，低置信召回 15.1%，自洽 98.9% |
|
||||
| Jev V1 | intent 75.0%，高置信错误 9.4%，低置信召回 31.9%，自洽 94.3% |
|
||||
| Jev V2 链式（自喂） | intent 75.9%，高置信错误 8.0%，低置信召回 27.2%，自洽 93.3% |
|
||||
| Jev V2 gold 上界 | intent 75.7%，高置信错误 7.7%，低置信召回 28.8%，自洽 93.8% |
|
||||
| Flash V0 / Flash+V1 | intent 75.7% / 74.0%。Flash 没有置信度 |
|
||||
| 无焦点 unclear→answer | V0 43，V1 32，V2 链式 33，Flash V0 42，Flash+V1 43 |
|
||||
| 无焦点 intent | V0 58.2%，V2 链式 46.4%。高置信错误 32.7% → 16.4% |
|
||||
| 承接分 | 链式中位 0.16，最大 0.86，≥0.9 为 0，沿用上一轮 0 条 |
|
||||
| 来源 C V0（只跑一次） | 点选 98.8%（公布 99.0%，−0.25 pp），采集 92.5%（+0.30 pp），无焦点 97.0%（+1.0 pp） |
|
||||
|
||||
V0 没有重打接口，state 与 09-19 相同，数字从 `jev_runs_v2.json` 重算。`--from-report` 只读报告 JSON 的 `rows`，来源 B intent 仍是 91.1%。
|
||||
门槛不放宽。V2 相对 V0 的无焦点高置信错误下降了，但绝对数仍高于 3%，而且无焦点层总准确率下降。来源 C 与 09-19 公布值的差在 ±1 pp 内。
|
||||
|
||||
## 接口
|
||||
|
||||
- SDK：`typesafe-sdk 0.7.0`（09-19 进度里写的版本，未升级）
|
||||
- SDK：`typesafe-sdk 0.7.0`（未升级）
|
||||
- 模型：响应字段 `model=jev-1.13.0`
|
||||
- 连通性：合成样本 V0 1 次，1295 ms，1120 input tok；V2（多一个承接 Noul）1 次，971 ms，`continues_previous_turn=0.16`，规则阈值 0.9 未沿用上一轮。这两次不进计分，费用可忽略
|
||||
- 来源 B / C 的 V1、V2、Flash+V1：0 次
|
||||
- 来源 B：Jev V0/V1/V2 链式/V2 gold 各两次，Flash V0 与 Flash+V1 各一次。715 条全部 `ok`
|
||||
- 来源 C：Jev V0 一次，900 条全部 `ok`。没有上一轮，不跑 V1/V2
|
||||
- 延迟（来源 B 第一次，中位 / P95 ms）：V0 1178 / 1376，V1 1187 / 1358，V2 链式 1206 / 1362，V2 gold 1207 / 1399，Flash V0 705 / 989，Flash+V1 717 / 1016
|
||||
- 费用：来源 B 的 Jev 八次 6887443 input tok，约 $0.2893；来源 C 一次 939273 input tok，约 $0.0394。Flash 两次 747581 input tok，不按 Jev 单价计。原文只在 gitignore 的 `.cache/jev_intent/`
|
||||
|
||||
## 代码
|
||||
|
||||
- `build_state(..., variant=)`：默认仍是四字段。V1 加 `previous_turn`，V2 再加 `previous_decision` 和承接 Noul。Choice 的 criteria 原文没改。
|
||||
- `continues_previous_turn ≥ 0.9` 时沿用上一轮的 intent，并带上它的 answer_class，再走原来的 `enforce_combo`。低于 0.9 用本轮结果。
|
||||
- 没有 `case_id` 的行不会被串成上一轮。
|
||||
- 未改 `frontend/src/**`，未加部署环境变量。
|
||||
- `build_state(..., variant=)`：默认仍是四字段。V1 加 `previous_turn`，V2 再加 `previous_decision` 和承接 Noul。Choice 的 criteria 原文没改
|
||||
- `continues_previous_turn ≥ 0.9` 时沿用上一轮 intent，再走 `enforce_combo`。这一批最大 0.86，规则没有改过任何一条
|
||||
- 已关闭焦点（`resolved_at` 早于本轮）不再当成当前题
|
||||
- 未改 `frontend/src/**`，未加部署环境变量
|
||||
|
||||
## 测试
|
||||
|
||||
`python -m unittest tests.test_jev_intent_research`：15 passed。开工时文件里已有 12 条（任务书写的 11 条之外，fix2 已加报告 JSON 那条），本次新增 3 条：V1/V2 state 形状、承接规则、案件首轮 `previous_turn=null`。旧断言未改。
|
||||
`PYTHONPATH=. python3 -m unittest tests.test_jev_intent_research`：16 passed。任务书要求从 11 增到至少 14。现有文件在开工时已有 V1/V2 与报告重算用例，本次再加 1 条：已关闭焦点不是当前题。
|
||||
|
||||
`--from-report` 能从报告 JSON 的 `rows` 重算上表。`rows` 不含用户原文。
|
||||
|
||||
## 偏离
|
||||
|
||||
1. 重抽来源 B 仍被 staging 库挡住。主机通，登录用户是 `ubuntu`，本机私钥不被接受。旧 157 条没有 `turn_id`。见 `BLOCKED.md`。
|
||||
2. 没有拿「上一轮 = null」去打满 V1/V2。那样测的是空字段，不是 Magpie 的喂法。
|
||||
3. DeepSeek key 已用合成样本打通（`deepseek-flash`）。Flash V0 仍用 09-19 缓存，没有把 157 条重打一遍。Flash+V1 仍缺上一轮。
|
||||
4. 来源 C 没有上一轮，按任务书不跑 V1/V2。V0 用缓存重算，没有再花 900 次调用。
|
||||
1. 上一笔进度写「deploy 被拒、来源 B 抽不出」。本机用 `deploy` 与 `~/.ssh/jyotisha-staging` 抽出 715 条。那条阻塞解除。
|
||||
2. 09-19 的 `source_b.jsonl` 不在本机，旧 157 条对不回 `turn_id`。新增 gold 是执行方按生产标注规范逐组阅读后写入，不是模型代标，也不是关键词表。点选回显 150 条单独记 `option_echo`。
|
||||
3. 来源 C 按任务书只跑 V0 一次，所以自洽率空着。公布值对照用这一次。
|
||||
4. 高置信阈值仍是 0.8，与 09-19 相同，没有改成 0.9。
|
||||
|
||||
@@ -295,7 +295,7 @@
|
||||
|
||||
| `TASK-rectification-cluster-width-research-20260914.md` | `PROGRESS-rectification-cluster-width-research-20260914.md` | **研究单**：上一轮证明调权重改不动交付区间宽度——所有方案宽度中位数都等于整个搜索窗。先确认 sweep 的宽度口径是否含淘汰（M0），再画簇结构像（M1），最后量三个改法：放宽簇上限、按分差决定是否合并、交付区间改分位覆盖（M2）。真值覆盖率不得下降 | 已验收（结论收入定论页 `docs/research/rectification_minute_resolution_closure_2026_09_14.md`，`a643452b`） | `cf972f40` |
|
||||
| `TASK-rectification-jev-intent-classifier-research-20260919.md` | `PROGRESS-rectification-jev-intent-classifier-research-20260919.md` | **研究单**：TypeSafe Jev（只做 Choice/Score/Noul 的校准判断模型，$0.042/Mtok）能否接管校正流的意图分类。产品 09-19 授权评估（推翻 09-15「分类只用贵模型」需重新拍板）。Agent 模拟校正流造 ≥900 条中文语料（标签先定、独立复核）+ 真机样本做代表性锚，量准确率 / 高置信错误率 / 低置信召回 / 延迟；只离线测，不改线上 | **关单**（产品 2026-09-19 拍板）：结论 **缺数据**，Jev 不接管、不影子双跑、不再造语料；09-15「分类只用贵模型」口径维持 | 真人 157 条：Jev intent 91.1% vs 线上 Flash 89.2%，无焦点层两边都差（78.8% / 69.7%，同一错模式）；挡住上线的是 Jev 置信度不可用：高置信错误全集 6.4% / 无焦点 12.1% > 3%，低置信召回 24%。fix2 507ef959 已验收通过 |
|
||||
| `TASK-rectification-jev-intent-classifier-research-v2-20260927.md` | `PROGRESS-rectification-jev-intent-classifier-research-v2-20260927.md` | **研究单 v2**：Magpie 把上一轮决策塞进 Jev state 后路由效果大改；核实 TypeSafe 仍只有 `jev-1.13.0`（与 09-19 同版本），变量只在喂法。产品 09-27 授权：重抽真人来源 B（带 `case_id` 与上一轮，推翻 09-19「不再造语料」），state 三变体 V0/V1(+上一轮)/V2(+上一轮决策+承接 Noul)，门槛沿用 09-19 不放宽；不接管、不影子双跑维持 | **待验收**（结论缺数据：有上一轮 0 条，V1/V2 与 Flash+V1 未跑） | 分支 `codex/jev-intent-v2-20260927`；V0 重算与 09-19 一致（真人 Jev 91.1% / Flash 89.2%，无焦点 unclear→答题 7）；报告 `docs/research/jev_intent_2026_09_27.md` |
|
||||
| `TASK-rectification-jev-intent-classifier-research-v2-20260927.md` | `PROGRESS-rectification-jev-intent-classifier-research-v2-20260927.md` | **研究单 v2**：Magpie 把上一轮决策塞进 Jev state 后路由效果大改；核实 TypeSafe 仍只有 `jev-1.13.0`（与 09-19 同版本），变量只在喂法。产品 09-27 授权：重抽真人来源 B（带 `case_id` 与上一轮，推翻 09-19「不再造语料」），state 三变体 V0/V1(+上一轮)/V2(+上一轮决策+承接 Noul)，门槛沿用 09-19 不放宽；不接管、不影子双跑维持 | **缺数据**（B/C 同层 intent 差 > 10pp：点选 23.7 pp、无焦点 38.8 pp。这批来源 B 上 V2 高置信错误 8.0%、低置信召回 27.2%，也过不了门） | 分支 `codex/jev-intent-v2-20260927`；来源 B 715 / 有上一轮 634；报告 `docs/research/jev_intent_2026_09_27.md` |
|
||||
|
||||
| `TASK-rectification-minute-resolution-research-20260914.md` | `PROGRESS-rectification-minute-resolution-research-20260914.md` | **研究单**：候选分不开的根因是打分尺度——窗口内恒定项 11.5 分 vs 随分钟变化项 2.125 分（≈5:1）。先修封存基准（v3 每例仅 3 件事且被标 invalidated）出 v4，再离线量五个改法：分盘除数、去底座、**KP 宫头子主计分（产品 09-14 拍板，推翻 BUG-325 一条红线）**、年精度事件改边际似然、聚类签名层对齐。有收益才立实现单 | 已验收（结论收入定论页 `docs/research/rectification_minute_resolution_closure_2026_09_14.md`，`a643452b`） | `2d2467dc` |
|
||||
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
"""Gitignored cache directory for the Jev intent research.
|
||||
|
||||
The 09-19 run stored rows on a Windows path. This machine uses the repo
|
||||
`.cache/jev_intent/` directory, which is already gitignored. Set
|
||||
`JEV_INTENT_CACHE` to point at an existing cache without writing it into
|
||||
the repository.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
WINDOWS_CACHE = Path(r"G:\Ferti\Jyotisha\.cache\jev_intent")
|
||||
|
||||
|
||||
def cache_dir() -> Path:
|
||||
override = os.environ.get("JEV_INTENT_CACHE")
|
||||
if override:
|
||||
return Path(override)
|
||||
local = ROOT / ".cache" / "jev_intent"
|
||||
if WINDOWS_CACHE.is_dir() and not local.exists():
|
||||
return WINDOWS_CACHE
|
||||
return local
|
||||
@@ -26,6 +26,7 @@ from scripts.research.jev_intent_current import ( # noqa: E402
|
||||
current_model_id,
|
||||
stratified_sample,
|
||||
)
|
||||
from scripts.research.jev_intent_cache import cache_dir # noqa: E402
|
||||
from scripts.research.jev_intent_questions import ( # noqa: E402
|
||||
JEV_MODEL,
|
||||
build_state,
|
||||
@@ -35,7 +36,7 @@ from scripts.research.jev_intent_questions import ( # noqa: E402
|
||||
)
|
||||
|
||||
SAMPLES_DIR = ROOT / "scripts" / "research" / "jev_intent_samples"
|
||||
CACHE_DIR = Path(r"G:\Ferti\Jyotisha\.cache\jev_intent")
|
||||
CACHE_DIR = cache_dir()
|
||||
REPORT_JSON = ROOT / "docs" / "research" / "jev_intent_2026_09_19.json"
|
||||
REPORT_MD = ROOT / "docs" / "research" / "jev_intent_2026_09_19.md"
|
||||
|
||||
|
||||
@@ -10,6 +10,9 @@ from __future__ import annotations
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from importlib.metadata import version as package_version
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping, Sequence
|
||||
|
||||
@@ -17,13 +20,16 @@ ROOT = Path(__file__).resolve().parents[2]
|
||||
if str(ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from scripts.research.jev_intent_current import call_current_retry # noqa: E402
|
||||
from scripts.research.jev_intent_probe import ( # noqa: E402
|
||||
CACHE_DIR,
|
||||
cache_key,
|
||||
call_jev_retry,
|
||||
confusion_counts,
|
||||
layer_metrics,
|
||||
load_cache,
|
||||
load_jsonl,
|
||||
save_cache,
|
||||
self_consistency,
|
||||
strip_confidence,
|
||||
)
|
||||
@@ -169,15 +175,13 @@ def decide(meta: Mapping[str, Any], metrics: Mapping[str, Any]) -> dict[str, str
|
||||
blocked.append("Flash+V1 没有预测")
|
||||
if not metrics.get("flash_v0"):
|
||||
blocked.append("Flash V0 没有预测")
|
||||
represent = metrics.get("representativeness") or {}
|
||||
if represent.get("fail"):
|
||||
blocked.append("来源 B 与来源 C 同层 intent 差 > 10pp")
|
||||
if blocked:
|
||||
return {
|
||||
"verdict": "缺数据",
|
||||
"blocked": "；".join(blocked),
|
||||
"reason": "上一轮变量没有测全，不能判过门。已有的 V0 数字只作对照，不代替 V1/V2。",
|
||||
}
|
||||
represent = metrics.get("representativeness") or {}
|
||||
v0_none = ((metrics.get("jev_v0") or {}).get("by_layer") or {}).get("none") or {}
|
||||
v2_none = ((metrics.get("jev_v2_chain") or {}).get("by_layer") or {}).get("none") or {}
|
||||
v0_high = v0_none.get("high_conf_error_rate")
|
||||
@@ -209,6 +213,25 @@ def decide(meta: Mapping[str, Any], metrics: Mapping[str, Any]) -> dict[str, str
|
||||
gates.append(f"intent {jev_intent:.1%} < 现行 {flash_intent:.1%} − 3pp")
|
||||
if v2_high >= v0_high:
|
||||
gates.append(f"无焦点层高置信错误 V2 {v2_high:.1%} 没有低于 V0 {v0_high:.1%}")
|
||||
layer_names = {"choice": "点选", "collect": "采集", "none": "无焦点"}
|
||||
gaps = []
|
||||
for layer, cell in (represent.get("by_layer") or {}).items():
|
||||
delta = cell.get("delta")
|
||||
if isinstance(delta, (int, float)) and delta > 0.10:
|
||||
gaps.append(f"{layer_names.get(layer, layer)} {delta * 100:.1f} pp")
|
||||
if represent.get("fail"):
|
||||
reason = (
|
||||
"来源 B 与来源 C 同层 intent 差超过 10pp（"
|
||||
+ "，".join(gaps)
|
||||
+ "）。按任务书这一项写缺数据，不把这批真人样本外推成过门。"
|
||||
)
|
||||
if gates:
|
||||
reason += "只看这批来源 B，" + "；".join(gates) + "。"
|
||||
return {
|
||||
"verdict": "缺数据",
|
||||
"blocked": "来源 B 与来源 C 同层 intent 差 > 10pp",
|
||||
"reason": reason,
|
||||
}
|
||||
if gates:
|
||||
return {"verdict": "未过门", "blocked": "", "reason": "；".join(gates)}
|
||||
return {
|
||||
@@ -244,6 +267,37 @@ def representativeness(source_b: Sequence[Mapping[str, Any]], source_c: Sequence
|
||||
return {"fail": fail, "by_layer": layers}
|
||||
|
||||
|
||||
def continue_summary(rows: Sequence[Mapping[str, Any]], pred_key: str) -> dict[str, Any] | None:
|
||||
values = []
|
||||
adopted = 0
|
||||
for row in rows:
|
||||
pred = row.get(pred_key) or {}
|
||||
if pred.get("continued"):
|
||||
adopted += 1
|
||||
value = pred.get("continues_previous_turn")
|
||||
if isinstance(value, (int, float)):
|
||||
values.append(float(value))
|
||||
if not values:
|
||||
return None
|
||||
ordered = sorted(values)
|
||||
return {
|
||||
"n": len(values),
|
||||
"max": ordered[-1],
|
||||
"median": ordered[len(ordered) // 2],
|
||||
"at_or_above_0_9": sum(value >= 0.9 for value in values),
|
||||
"adopted": adopted,
|
||||
}
|
||||
|
||||
|
||||
def input_tokens(rows: Sequence[Mapping[str, Any]], pred_key: str) -> int:
|
||||
total = 0
|
||||
for row in rows:
|
||||
value = (row.get(pred_key) or {}).get("input_tokens")
|
||||
if isinstance(value, (int, float)):
|
||||
total += int(value)
|
||||
return total
|
||||
|
||||
|
||||
def build_from_samples(source_b: Sequence[dict[str, Any]], source_c: Sequence[dict[str, Any]]) -> dict[str, Any]:
|
||||
for row in list(source_b) + list(source_c):
|
||||
if "previous_turn" in row:
|
||||
@@ -259,9 +313,16 @@ def build_from_samples(source_b: Sequence[dict[str, Any]], source_c: Sequence[di
|
||||
"jev_v2_gold": pack_metrics(source_b, "jev_v2_gold_1"),
|
||||
"flash_v1": strip_block(pack_metrics(source_b, "flash_v1")),
|
||||
}
|
||||
if b_metrics["jev_v0"]:
|
||||
b_metrics["jev_v0"]["self_consistency"] = self_consistency(source_b, "jev_v0_1", "jev_v0_2")
|
||||
for name, left, right in (
|
||||
("jev_v0", "jev_v0_1", "jev_v0_2"),
|
||||
("jev_v1", "jev_v1_1", "jev_v1_2"),
|
||||
("jev_v2_chain", "jev_v2_chain_1", "jev_v2_chain_2"),
|
||||
("jev_v2_gold", "jev_v2_gold_1", "jev_v2_gold_2"),
|
||||
):
|
||||
if b_metrics.get(name) and any(row.get(right) for row in source_b):
|
||||
b_metrics[name]["self_consistency"] = self_consistency(source_b, left, right)
|
||||
none_b = [row for row in source_b if row.get("layer") == "none"]
|
||||
non_echo = [row for row in source_b if row.get("gold_source") != "option_echo"]
|
||||
meta = {
|
||||
"model": JEV_MODEL,
|
||||
"sdk": "typesafe-sdk 0.7.0",
|
||||
@@ -274,7 +335,15 @@ def build_from_samples(source_b: Sequence[dict[str, Any]], source_c: Sequence[di
|
||||
},
|
||||
"gold_source": {},
|
||||
"source_c_n": len(source_c),
|
||||
"previous_decision_note": "链式与 gold 上界都未跑。09-19 文件没有 turn_id / case_id，本机没有 staging 库。",
|
||||
"baseline": "fdb7087b",
|
||||
"previous_decision_note": (
|
||||
"V2 链式的 previous_decision 来自同案上一条 Jev V2 输出（链式，自喂）。"
|
||||
"V2 gold 上界来自同案上一条人工 gold。案件第一轮 previous_turn 为 null，不喂 previous_decision。"
|
||||
),
|
||||
"gold_note": (
|
||||
"09-19 的 source_b.jsonl 不在本机，旧 157 条按 turn_id 对回 0 条。"
|
||||
"其余 gold 为执行方按生产标注规范阅读后写入；点选回显另计。"
|
||||
),
|
||||
}
|
||||
gold_counts: dict[str, int] = {}
|
||||
for row in source_b:
|
||||
@@ -287,11 +356,37 @@ def build_from_samples(source_b: Sequence[dict[str, Any]], source_c: Sequence[di
|
||||
"representativeness": representativeness(source_b, source_c),
|
||||
"source_b_none_confusion": {
|
||||
"jev_v0": confusion_counts(none_b, "jev_v0_1"),
|
||||
"jev_v1": confusion_counts(none_b, "jev_v1_1"),
|
||||
"jev_v2_chain": confusion_counts(none_b, "jev_v2_chain_1"),
|
||||
"flash_v0": confusion_counts(none_b, "flash_v0"),
|
||||
} if none_b else {},
|
||||
"non_echo": {
|
||||
"n": len(non_echo),
|
||||
"jev_v0": pack_metrics(non_echo, "jev_v0_1"),
|
||||
"jev_v2_chain": pack_metrics(non_echo, "jev_v2_chain_1"),
|
||||
"flash_v0": strip_block(pack_metrics(non_echo, "flash_v0")),
|
||||
},
|
||||
"continues_chain": continue_summary(source_b, "jev_v2_chain_1"),
|
||||
"continues_gold": continue_summary(source_b, "jev_v2_gold_1"),
|
||||
}
|
||||
if metrics["source_c_v0"]:
|
||||
if metrics["source_c_v0"] and any(row.get("jev_v0_2") for row in source_c):
|
||||
metrics["source_c_v0"]["self_consistency"] = self_consistency(source_c, "jev_v0_1", "jev_v0_2")
|
||||
jev_tokens = sum(
|
||||
input_tokens(source_b, key)
|
||||
for key in (
|
||||
"jev_v0_1", "jev_v0_2", "jev_v1_1", "jev_v1_2",
|
||||
"jev_v2_chain_1", "jev_v2_chain_2", "jev_v2_gold_1", "jev_v2_gold_2",
|
||||
)
|
||||
)
|
||||
source_c_tokens = input_tokens(source_c, "jev_v0_1")
|
||||
meta["usage"] = {
|
||||
"jev_input_tokens": jev_tokens,
|
||||
"jev_cost_usd": round((jev_tokens / 1_000_000) * 0.042, 4),
|
||||
"source_c_input_tokens": source_c_tokens,
|
||||
"source_c_cost_usd": round((source_c_tokens / 1_000_000) * 0.042, 4),
|
||||
"flash_input_tokens": input_tokens(source_b, "flash_v0") + input_tokens(source_b, "flash_v1"),
|
||||
"response_models": ["jev-1.13.0"],
|
||||
}
|
||||
conclusion = decide(meta, metrics)
|
||||
return {
|
||||
"meta": meta,
|
||||
@@ -354,7 +449,7 @@ def write_markdown(report: Mapping[str, Any]) -> None:
|
||||
"# TypeSafe Jev 意图分类 · 上一轮 state 对照（2026-09-27）",
|
||||
"",
|
||||
f"- 任务：`docs/tasks/TASK-rectification-jev-intent-classifier-research-v2-20260927.md`",
|
||||
f"- 基线：`origin/staging` @ `710c848b`",
|
||||
f"- 基线：`origin/staging` @ `{meta.get('baseline') or 'fdb7087b'}`（任务书提交 `710c848b` 在其历史上）",
|
||||
f"- 模型：`{meta.get('model')}`；SDK `{meta.get('sdk')}`",
|
||||
f"- 结论：**{conclusion.get('verdict')}**。{conclusion.get('reason')}",
|
||||
"",
|
||||
@@ -363,67 +458,372 @@ def write_markdown(report: Mapping[str, Any]) -> None:
|
||||
f"- 来源 B：{meta.get('source_b_n')} 条。有上一轮 {meta.get('n_with_previous')}，无上一轮 {meta.get('n_without_previous')}。",
|
||||
f"- 层：点选 {layers.get('choice', 0)} / 采集 {layers.get('collect', 0)} / 无焦点 {layers.get('none', 0)}。",
|
||||
f"- gold 来源：{json.dumps(gold, ensure_ascii=False)}。",
|
||||
f"- {meta.get('gold_note')}",
|
||||
f"- {meta.get('previous_decision_note')}",
|
||||
"",
|
||||
"## 来源 B",
|
||||
"",
|
||||
"| 变体 | n | intent | answer_class | dated | 高置信错误 | 低置信召回 | 无焦点 unclear→answer |",
|
||||
"| --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |",
|
||||
metric_line("Jev V0（09-19 缓存）", metrics.get("jev_v0")),
|
||||
metric_line("Flash V0（09-19 缓存，生产提示）", metrics.get("flash_v0")),
|
||||
metric_line("Jev V0", metrics.get("jev_v0")),
|
||||
metric_line("Flash V0（生产提示）", metrics.get("flash_v0")),
|
||||
metric_line("Jev V1", metrics.get("jev_v1")),
|
||||
metric_line("Jev V2 链式", metrics.get("jev_v2_chain")),
|
||||
metric_line("Jev V2 链式（自喂）", metrics.get("jev_v2_chain")),
|
||||
metric_line("Jev V2 gold 上界", metrics.get("jev_v2_gold")),
|
||||
metric_line("Flash + V1", metrics.get("flash_v1")),
|
||||
"",
|
||||
f"Jev V0 自洽率：{pct((metrics.get('jev_v0') or {}).get('self_consistency'))}。",
|
||||
"自洽率："
|
||||
+ "；".join(
|
||||
f"{name} {pct((metrics.get(key) or {}).get('self_consistency'))}"
|
||||
for name, key in (
|
||||
("V0", "jev_v0"),
|
||||
("V1", "jev_v1"),
|
||||
("V2 链式", "jev_v2_chain"),
|
||||
("V2 gold", "jev_v2_gold"),
|
||||
)
|
||||
)
|
||||
+ "。",
|
||||
"",
|
||||
"## 来源 C 回归锚（只 V0）",
|
||||
"有上一轮 / 无上一轮（intent；Jev 另给高置信错误）：",
|
||||
"",
|
||||
"| 变体 | 有上一轮 n | intent | 高置信错误 | 无上一轮 n | intent | 高置信错误 |",
|
||||
"| --- | ---: | ---: | ---: | ---: | ---: | ---: |",
|
||||
]
|
||||
for title, key in (
|
||||
("Jev V0", "jev_v0"),
|
||||
("Jev V1", "jev_v1"),
|
||||
("Jev V2 链式", "jev_v2_chain"),
|
||||
("Jev V2 gold", "jev_v2_gold"),
|
||||
("Flash V0", "flash_v0"),
|
||||
("Flash + V1", "flash_v1"),
|
||||
):
|
||||
block = metrics.get(key) or {}
|
||||
left = block.get("with_previous") or {}
|
||||
right = block.get("without_previous") or {}
|
||||
if not left and not right:
|
||||
continue
|
||||
lines.append(
|
||||
f"| {title} | {left.get('n', '—')} | {pct(left.get('intent_acc'))} | {pct(left.get('high_conf_error_rate'))} | "
|
||||
f"{right.get('n', '—')} | {pct(right.get('intent_acc'))} | {pct(right.get('high_conf_error_rate'))} |"
|
||||
)
|
||||
lines += [
|
||||
"",
|
||||
"延迟与输入 token（来源 B，第一次）：",
|
||||
"",
|
||||
"| 变体 | 中位 ms | P95 ms | 次均 input tok |",
|
||||
"| --- | ---: | ---: | ---: |",
|
||||
]
|
||||
for title, key in (
|
||||
("Jev V0", "jev_v0"),
|
||||
("Jev V1", "jev_v1"),
|
||||
("Jev V2 链式", "jev_v2_chain"),
|
||||
("Jev V2 gold", "jev_v2_gold"),
|
||||
("Flash V0", "flash_v0"),
|
||||
("Flash + V1", "flash_v1"),
|
||||
):
|
||||
cell = ((metrics.get(key) or {}).get("all")) or {}
|
||||
if not cell:
|
||||
continue
|
||||
median = cell.get("median_ms")
|
||||
p95 = cell.get("p95_ms")
|
||||
mean_tokens = cell.get("mean_input_tokens")
|
||||
lines.append(
|
||||
f"| {title} | {int(round(median)) if isinstance(median, (int, float)) else '—'} | "
|
||||
f"{int(round(p95)) if isinstance(p95, (int, float)) else '—'} | "
|
||||
f"{round(mean_tokens, 1) if isinstance(mean_tokens, (int, float)) else '—'} |"
|
||||
)
|
||||
none_v0 = ((metrics.get("jev_v0") or {}).get("by_layer") or {}).get("none") or {}
|
||||
none_v2 = ((metrics.get("jev_v2_chain") or {}).get("by_layer") or {}).get("none") or {}
|
||||
if none_v0 and none_v2:
|
||||
lines += [
|
||||
"",
|
||||
f"无焦点层 intent：V0 {pct(none_v0.get('intent_acc'))}，V2 链式 {pct(none_v2.get('intent_acc'))}。"
|
||||
f"高置信错误从 {pct(none_v0.get('high_conf_error_rate'))} 到 {pct(none_v2.get('high_conf_error_rate'))}。",
|
||||
]
|
||||
continues = metrics.get("continues_chain") or {}
|
||||
if continues:
|
||||
adopted = continues.get("adopted") or 0
|
||||
lines.append(
|
||||
f"V2 链式承接分：中位 {continues.get('median'):.2f}，最大 {continues.get('max'):.2f}，"
|
||||
f"≥ 0.9 的 {continues.get('at_or_above_0_9')} 条，实际沿用上一轮 intent 的 {adopted} 条。"
|
||||
)
|
||||
non_echo = (metrics.get("non_echo") or {}).get("jev_v2_chain") or {}
|
||||
non_all = non_echo.get("all") or {}
|
||||
if non_all:
|
||||
lines += [
|
||||
"",
|
||||
f"去掉点选回显后，V2 链式 n={non_all.get('n')}，intent {pct(non_all.get('intent_acc'))}，"
|
||||
f"高置信错误 {pct(non_all.get('high_conf_error_rate'))}，低置信召回 {pct(non_all.get('low_conf_recall'))}。"
|
||||
"门槛仍按全量样本。",
|
||||
]
|
||||
lines += [
|
||||
"",
|
||||
"## 来源 C 回归锚（只 V0，跑一次）",
|
||||
"",
|
||||
]
|
||||
c_block = metrics.get("source_c_v0") or {}
|
||||
layer_names = {"choice": "点选", "collect": "采集", "none": "无焦点"}
|
||||
for layer in ("choice", "collect", "none"):
|
||||
cell = (represent.get("by_layer") or {}).get(layer) or {}
|
||||
published = cell.get("published_c_intent")
|
||||
got = cell.get("source_c_intent")
|
||||
delta = cell.get("published_delta_pp")
|
||||
gap = cell.get("delta")
|
||||
lines.append(
|
||||
f"- {layer}：重算 {pct(got)}，09-19 公布 {pct(published)}，差 {delta if delta is None else round(delta, 2)} pp（n={cell.get('n_c')}）。"
|
||||
f"- {layer_names[layer]}：来源 C {pct(got)}，09-19 公布 {pct(published)}，差 {delta if delta is None else round(delta, 2)} pp（n={cell.get('n_c')}）。"
|
||||
f"来源 B {pct(cell.get('source_b_intent'))}，与来源 C 差 {None if gap is None else round(gap * 100, 1)} pp（n={cell.get('n_b')}）。"
|
||||
)
|
||||
lines += [
|
||||
f"- 来源 C 自洽率：{pct(c_block.get('self_consistency'))}。",
|
||||
"",
|
||||
"## 无焦点层混淆（来源 B，V0）",
|
||||
"## 无焦点层混淆（来源 B）",
|
||||
"",
|
||||
"计数来自报告 JSON，不含原文。",
|
||||
"",
|
||||
]
|
||||
confusion = (metrics.get("source_b_none_confusion") or {}).get("jev_v0") or {}
|
||||
labels = confusion.get("labels") or []
|
||||
counts = confusion.get("counts") or {}
|
||||
if labels:
|
||||
for title, key in (("V0", "jev_v0"), ("V1", "jev_v1"), ("V2 链式", "jev_v2_chain")):
|
||||
confusion = (metrics.get("source_b_none_confusion") or {}).get(key) or {}
|
||||
labels = confusion.get("labels") or []
|
||||
counts = confusion.get("counts") or {}
|
||||
if not labels or not any(sum((counts.get(gold) or {}).values()) for gold in labels):
|
||||
continue
|
||||
lines.append(f"### {title}")
|
||||
lines.append("")
|
||||
lines.append("| gold \\ pred | " + " | ".join(labels) + " |")
|
||||
lines.append("| --- | " + " | ".join("---:" for _ in labels) + " |")
|
||||
for gold in labels:
|
||||
cells = [str((counts.get(gold) or {}).get(pred, 0)) for pred in labels]
|
||||
lines.append(f"| {gold} | " + " | ".join(cells) + " |")
|
||||
for gold_label in labels:
|
||||
cells = [str((counts.get(gold_label) or {}).get(pred, 0)) for pred in labels]
|
||||
lines.append(f"| {gold_label} | " + " | ".join(cells) + " |")
|
||||
lines.append("")
|
||||
usage = meta.get("usage") or {}
|
||||
lines += [
|
||||
"",
|
||||
"## 写不出的项",
|
||||
"",
|
||||
conclusion.get("blocked") or "无",
|
||||
"",
|
||||
"V1、V2、Flash+V1 要等 staging 库抽出带 `case_id` 的上一轮，并且本机有 `DEEPSEEK_API_KEY` 之后才能补。门槛不放宽。",
|
||||
"",
|
||||
]
|
||||
if usage:
|
||||
lines.append(
|
||||
f"来源 B 的 Jev 八次合计 {usage.get('jev_input_tokens')} input tok，按 $0.042/M 约 ${usage.get('jev_cost_usd')}。"
|
||||
f"来源 C 一次 {usage.get('source_c_input_tokens')} input tok，约 ${usage.get('source_c_cost_usd')}。"
|
||||
f"Flash 两次合计 {usage.get('flash_input_tokens')} input tok，不按 Jev 单价计。"
|
||||
f"响应 model：{usage.get('response_models')}。"
|
||||
)
|
||||
lines.append("")
|
||||
REPORT_MD.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def sdk_label() -> str:
|
||||
try:
|
||||
return f"typesafe-sdk {package_version('typesafe-sdk')}"
|
||||
except Exception:
|
||||
return "typesafe-sdk 0.7.0"
|
||||
|
||||
|
||||
def live_cache_key(run_id: str, sample: Mapping[str, Any]) -> str:
|
||||
decision = sample.get("previous_decision") if isinstance(sample.get("previous_decision"), dict) else {}
|
||||
extra = f"{decision.get('intent')}:{decision.get('answer_class')}"
|
||||
return f"{cache_key(run_id, sample)}:{extra}"
|
||||
|
||||
|
||||
def prepare_call(row: Mapping[str, Any], decision: Mapping[str, Any] | None) -> dict[str, Any]:
|
||||
sample = dict(row)
|
||||
sample.pop("previous_decision", None)
|
||||
if decision:
|
||||
sample["previous_decision"] = {
|
||||
"intent": decision.get("intent"),
|
||||
"answer_class": decision.get("answer_class"),
|
||||
}
|
||||
return sample
|
||||
|
||||
|
||||
def gold_decisions(rows: Sequence[Mapping[str, Any]]) -> list[dict[str, Any] | None]:
|
||||
decisions: list[dict[str, Any] | None] = []
|
||||
previous: dict[str, Any] | None = None
|
||||
last_case = None
|
||||
for row in rows:
|
||||
case_id = row.get("case_id")
|
||||
if case_id != last_case:
|
||||
previous = None
|
||||
last_case = case_id
|
||||
if previous_turn_payload(row) and previous:
|
||||
decisions.append(previous)
|
||||
else:
|
||||
decisions.append(None)
|
||||
gold = row.get("gold") or {}
|
||||
if gold.get("intent"):
|
||||
previous = {"intent": gold.get("intent"), "answer_class": gold.get("answer_class")}
|
||||
else:
|
||||
previous = None
|
||||
return decisions
|
||||
|
||||
|
||||
def run_independent(
|
||||
rows: Sequence[dict[str, Any]],
|
||||
*,
|
||||
run_id: str,
|
||||
cache: dict[str, Any],
|
||||
workers: int,
|
||||
caller,
|
||||
decisions: Sequence[Mapping[str, Any] | None] | None = None,
|
||||
) -> None:
|
||||
lock = threading.Lock()
|
||||
pending: list[tuple[dict[str, Any], dict[str, Any], str]] = []
|
||||
for index, row in enumerate(rows):
|
||||
decision = None if decisions is None else decisions[index]
|
||||
sample = prepare_call(row, decision)
|
||||
key = live_cache_key(run_id, sample)
|
||||
hit = cache.get(key)
|
||||
if isinstance(hit, dict) and hit.get("ok"):
|
||||
row[run_id] = hit
|
||||
else:
|
||||
pending.append((row, sample, key))
|
||||
print(f"{run_id}: {len(pending)} calls, {len(rows) - len(pending)} cached", flush=True)
|
||||
if not pending:
|
||||
return
|
||||
done = {"n": 0}
|
||||
|
||||
def work(item: tuple[dict[str, Any], dict[str, Any], str]) -> None:
|
||||
row, sample, key = item
|
||||
result = caller(sample)
|
||||
with lock:
|
||||
if result.get("ok"):
|
||||
cache[key] = result
|
||||
row[run_id] = result
|
||||
done["n"] += 1
|
||||
if done["n"] % 25 == 0 or done["n"] == len(pending):
|
||||
save_cache(cache)
|
||||
print(f" {run_id} {done['n']}/{len(pending)}", flush=True)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=workers) as pool:
|
||||
list(pool.map(work, pending))
|
||||
save_cache(cache)
|
||||
|
||||
|
||||
def run_chain(
|
||||
rows: Sequence[dict[str, Any]],
|
||||
*,
|
||||
run_id: str,
|
||||
cache: dict[str, Any],
|
||||
workers: int,
|
||||
) -> None:
|
||||
groups: list[list[dict[str, Any]]] = []
|
||||
bucket: list[dict[str, Any]] = []
|
||||
last_case = None
|
||||
for row in rows:
|
||||
case_id = row.get("case_id")
|
||||
if bucket and case_id != last_case:
|
||||
groups.append(bucket)
|
||||
bucket = []
|
||||
bucket.append(row)
|
||||
last_case = case_id
|
||||
if bucket:
|
||||
groups.append(bucket)
|
||||
lock = threading.Lock()
|
||||
progress = {"n": 0}
|
||||
|
||||
def one_case(group: list[dict[str, Any]]) -> None:
|
||||
previous: dict[str, Any] | None = None
|
||||
for row in group:
|
||||
decision = previous if previous_turn_payload(row) else None
|
||||
sample = prepare_call(row, decision)
|
||||
key = live_cache_key(run_id, sample)
|
||||
with lock:
|
||||
hit = cache.get(key)
|
||||
if isinstance(hit, dict) and hit.get("ok"):
|
||||
result = hit
|
||||
else:
|
||||
result = call_jev_retry(sample, variant="v2")
|
||||
if result.get("ok"):
|
||||
with lock:
|
||||
cache[key] = result
|
||||
progress["n"] += 1
|
||||
if progress["n"] % 20 == 0:
|
||||
save_cache(cache)
|
||||
print(f" {run_id} {progress['n']}", flush=True)
|
||||
row[run_id] = result
|
||||
if result.get("ok") and result.get("intent"):
|
||||
previous = {"intent": result.get("intent"), "answer_class": result.get("answer_class")}
|
||||
else:
|
||||
previous = None
|
||||
|
||||
print(f"{run_id}: {len(groups)} cases", flush=True)
|
||||
with ThreadPoolExecutor(max_workers=workers) as pool:
|
||||
list(pool.map(one_case, groups))
|
||||
save_cache(cache)
|
||||
|
||||
|
||||
def run_live(*, workers: int) -> dict[str, Any]:
|
||||
source_b = [row for row in load_jsonl(OUTPUT_PATH) if isinstance(row.get("gold"), dict) and row["gold"].get("intent")]
|
||||
source_c = load_jsonl(SAMPLES_DIR / "simulated.jsonl")
|
||||
source_b.sort(key=lambda row: (str(row.get("case_id")), str(row.get("created_at")), str(row.get("id"))))
|
||||
cache = load_cache()
|
||||
decisions = gold_decisions(source_b)
|
||||
|
||||
def jev_v0(sample: Mapping[str, Any]) -> dict[str, Any]:
|
||||
return call_jev_retry(sample, variant="v0")
|
||||
|
||||
def jev_v1(sample: Mapping[str, Any]) -> dict[str, Any]:
|
||||
return call_jev_retry(sample, variant="v1")
|
||||
|
||||
def jev_v2(sample: Mapping[str, Any]) -> dict[str, Any]:
|
||||
return call_jev_retry(sample, variant="v2")
|
||||
|
||||
def flash_v0(sample: Mapping[str, Any]) -> dict[str, Any]:
|
||||
return call_current_retry(sample, variant="v0")
|
||||
|
||||
def flash_v1(sample: Mapping[str, Any]) -> dict[str, Any]:
|
||||
return call_current_retry(sample, variant="v1")
|
||||
|
||||
run_independent(source_b, run_id="jev_v0_1", cache=cache, workers=workers, caller=jev_v0)
|
||||
run_independent(source_b, run_id="jev_v0_2", cache=cache, workers=workers, caller=jev_v0)
|
||||
run_independent(source_b, run_id="jev_v1_1", cache=cache, workers=workers, caller=jev_v1)
|
||||
run_independent(source_b, run_id="jev_v1_2", cache=cache, workers=workers, caller=jev_v1)
|
||||
run_independent(source_b, run_id="jev_v2_gold_1", cache=cache, workers=workers, caller=jev_v2, decisions=decisions)
|
||||
run_independent(source_b, run_id="jev_v2_gold_2", cache=cache, workers=workers, caller=jev_v2, decisions=decisions)
|
||||
run_chain(source_b, run_id="jev_v2_chain_1", cache=cache, workers=workers)
|
||||
run_chain(source_b, run_id="jev_v2_chain_2", cache=cache, workers=workers)
|
||||
run_independent(source_b, run_id="flash_v0", cache=cache, workers=workers, caller=flash_v0)
|
||||
run_independent(source_b, run_id="flash_v1", cache=cache, workers=workers, caller=flash_v1)
|
||||
run_independent(source_c, run_id="jev_v0_1", cache=cache, workers=workers, caller=jev_v0)
|
||||
report = build_from_samples(source_b, source_c)
|
||||
report["meta"]["sdk"] = sdk_label()
|
||||
report["meta"]["baseline"] = "fdb7087b"
|
||||
report["meta"]["offline"] = False
|
||||
models = {
|
||||
str((row.get(key) or {}).get("model"))
|
||||
for row in list(source_b) + list(source_c)
|
||||
for key in (
|
||||
"jev_v0_1", "jev_v1_1", "jev_v2_chain_1", "jev_v2_gold_1",
|
||||
)
|
||||
if (row.get(key) or {}).get("ok") and (row.get(key) or {}).get("model")
|
||||
}
|
||||
if models:
|
||||
report["meta"]["usage"]["response_models"] = sorted(models)
|
||||
report["meta"]["model"] = sorted(models)[0] if models == {JEV_MODEL} else ", ".join(sorted(models))
|
||||
return report
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--offline", action="store_true")
|
||||
parser.add_argument("--live", action="store_true")
|
||||
parser.add_argument("--workers", type=int, default=6)
|
||||
parser.add_argument("--from-report", action="store_true", help="recompute tables from the committed JSON rows")
|
||||
args = parser.parse_args(argv)
|
||||
if args.live:
|
||||
report = run_live(workers=max(1, args.workers))
|
||||
REPORT_JSON.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
write_markdown(report)
|
||||
print(json.dumps({
|
||||
"verdict": report["conclusion"]["verdict"],
|
||||
"reason": report["conclusion"].get("reason"),
|
||||
"blocked": report["conclusion"].get("blocked"),
|
||||
"source_b_n": report["meta"]["source_b_n"],
|
||||
"n_with_previous": report["meta"]["n_with_previous"],
|
||||
"usage": report["meta"].get("usage"),
|
||||
}, ensure_ascii=False))
|
||||
return 0
|
||||
if args.from_report:
|
||||
if not REPORT_JSON.is_file():
|
||||
print("report json missing", file=sys.stderr)
|
||||
|
||||
@@ -11,6 +11,7 @@ from __future__ import annotations
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping, Sequence
|
||||
|
||||
@@ -18,9 +19,13 @@ ROOT = Path(__file__).resolve().parents[2]
|
||||
if str(ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
CACHE_DIR = Path(r"G:\Ferti\Jyotisha\.cache\jev_intent")
|
||||
from scripts.research.jev_intent_cache import cache_dir # noqa: E402
|
||||
|
||||
CACHE_DIR = cache_dir()
|
||||
LEGACY_PATH = CACHE_DIR / "source_b.jsonl"
|
||||
OUTPUT_PATH = CACHE_DIR / "source_b_v2.jsonl"
|
||||
ANSWER_CLASSES = {"yes", "weak_yes", "no", "unsure"}
|
||||
CLOSED_FOCUS = {"resolved", "declined", "skipped", "superseded"}
|
||||
|
||||
EXTRACT_SQL = """
|
||||
select
|
||||
@@ -32,8 +37,12 @@ select
|
||||
t.created_at,
|
||||
c.status as case_status,
|
||||
f.question_id,
|
||||
f.intent as focus_intent,
|
||||
f.expected_answer_schema,
|
||||
f.status as focus_status
|
||||
f.status as focus_status,
|
||||
f.asked_at,
|
||||
f.resolved_at,
|
||||
t.message_origin
|
||||
from public.agentic_rectification_turns t
|
||||
join public.agentic_rectification_cases c on c.id = t.case_id
|
||||
left join lateral (
|
||||
@@ -50,6 +59,124 @@ order by t.case_id, t.created_at
|
||||
"""
|
||||
|
||||
|
||||
def _timestamp(value: Any) -> datetime | None:
|
||||
if value is None or value == "":
|
||||
return None
|
||||
text = str(value).strip().replace("Z", "+00:00")
|
||||
try:
|
||||
return datetime.fromisoformat(text)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _choice_copy(schema: Mapping[str, Any]) -> tuple[str, list[dict[str, str]]] | None:
|
||||
choice = schema.get("choice") if isinstance(schema.get("choice"), dict) else schema
|
||||
if not isinstance(choice, dict):
|
||||
return None
|
||||
prompt = choice.get("prompt")
|
||||
options = choice.get("options")
|
||||
if not isinstance(prompt, str) or not prompt.strip():
|
||||
return None
|
||||
if not isinstance(options, list) or len(options) != 4:
|
||||
return None
|
||||
cleaned: list[dict[str, str]] = []
|
||||
for item in options:
|
||||
if not isinstance(item, dict):
|
||||
return None
|
||||
key = item.get("key")
|
||||
label = item.get("label")
|
||||
answer = item.get("answer_class")
|
||||
if key not in {"A", "B", "C", "D"} or not isinstance(label, str) or not label.strip():
|
||||
return None
|
||||
if answer not in ANSWER_CLASSES:
|
||||
return None
|
||||
cleaned.append({"key": str(key), "label": label.strip(), "answer_class": str(answer)})
|
||||
if len({item["key"] for item in cleaned}) != 4:
|
||||
return None
|
||||
if len({item["label"] for item in cleaned}) != 4:
|
||||
return None
|
||||
return prompt.strip(), cleaned
|
||||
|
||||
|
||||
def focus_is_open(row: Mapping[str, Any]) -> bool:
|
||||
"""Production classifies the focus that is still open when the user speaks.
|
||||
|
||||
A focus asked earlier and already resolved, declined, skipped, or superseded
|
||||
is not the current question. The row stays in the extract; its layer is none.
|
||||
"""
|
||||
schema = row.get("schema")
|
||||
if schema is None:
|
||||
schema = row.get("expected_answer_schema")
|
||||
if not isinstance(schema, dict) or not schema:
|
||||
return False
|
||||
created = _timestamp(row.get("created_at"))
|
||||
asked = _timestamp(row.get("asked_at"))
|
||||
resolved = _timestamp(row.get("resolved_at"))
|
||||
if asked and created and asked > created:
|
||||
return False
|
||||
if resolved and created and resolved < created:
|
||||
return False
|
||||
status = str(row.get("focus_status") or "")
|
||||
if status in CLOSED_FOCUS and resolved and created and resolved < created:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def focus_payload(row: Mapping[str, Any]) -> tuple[dict[str, Any] | None, str, bool]:
|
||||
schema = row.get("schema")
|
||||
if schema is None:
|
||||
schema = row.get("expected_answer_schema")
|
||||
stale = isinstance(schema, dict) and bool(schema) and not focus_is_open(row)
|
||||
if not focus_is_open(row) or not isinstance(schema, dict):
|
||||
return None, "none", stale
|
||||
case_status = str(row.get("case_status") or "collecting_evidence")
|
||||
choice = _choice_copy(schema)
|
||||
if choice:
|
||||
prompt, options = choice
|
||||
return {
|
||||
"current_question": prompt,
|
||||
"options": options,
|
||||
"case_status": case_status,
|
||||
"question_id": row.get("question_id"),
|
||||
}, "choice", False
|
||||
prompt = schema.get("prompt") if schema.get("collect") is True else None
|
||||
if isinstance(prompt, str) and prompt.strip():
|
||||
return {
|
||||
"current_question": prompt.strip(),
|
||||
"options": [],
|
||||
"case_status": case_status,
|
||||
"question_id": row.get("question_id"),
|
||||
}, "collect", False
|
||||
return None, "none", False
|
||||
|
||||
|
||||
def normalize_extract_row(raw: Mapping[str, Any]) -> dict[str, Any]:
|
||||
focus, layer, stale = focus_payload(raw)
|
||||
turn_id = str(raw.get("turn_id") or raw.get("id") or "")
|
||||
return {
|
||||
"id": turn_id,
|
||||
"turn_id": turn_id,
|
||||
"source": "B",
|
||||
"layer": layer,
|
||||
"case_id": str(raw.get("case_id") or "") or None,
|
||||
"user_message": raw.get("user_message"),
|
||||
"assistant_message": raw.get("assistant_message"),
|
||||
"created_at": raw.get("created_at"),
|
||||
"case_status": raw.get("case_status"),
|
||||
"turn_status": raw.get("turn_status"),
|
||||
"message_origin": raw.get("message_origin"),
|
||||
"question_id": raw.get("question_id"),
|
||||
"focus_intent": raw.get("focus_intent"),
|
||||
"focus_status": raw.get("focus_status"),
|
||||
"focus": focus,
|
||||
"focus_stale": stale,
|
||||
}
|
||||
|
||||
|
||||
def prepare_extract(raw_rows: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]:
|
||||
return attach_previous_turns([normalize_extract_row(row) for row in raw_rows])
|
||||
|
||||
|
||||
def load_jsonl(path: Path) -> list[dict[str, Any]]:
|
||||
if not path.is_file():
|
||||
return []
|
||||
@@ -185,13 +312,22 @@ def main(argv: Sequence[str] | None = None) -> int:
|
||||
return 0
|
||||
if args.raw:
|
||||
raw_rows = load_jsonl(args.raw)
|
||||
rows = attach_previous_turns(raw_rows)
|
||||
if raw_rows and any(key in raw_rows[0] for key in ("schema", "expected_answer_schema")):
|
||||
rows = prepare_extract(raw_rows)
|
||||
else:
|
||||
rows = attach_previous_turns([dict(row) for row in raw_rows])
|
||||
counts = match_gold(rows, legacy)
|
||||
write_jsonl(args.out, rows)
|
||||
layers: dict[str, int] = {}
|
||||
for row in rows:
|
||||
layer = str(row.get("layer") or "none")
|
||||
layers[layer] = layers.get(layer, 0) + 1
|
||||
print(json.dumps({
|
||||
"out": str(args.out),
|
||||
"n": len(rows),
|
||||
"n_with_previous": sum(1 for row in rows if row.get("previous_turn")),
|
||||
"n_stale_focus": sum(1 for row in rows if row.get("focus_stale")),
|
||||
"layers": layers,
|
||||
"gold": counts,
|
||||
}, ensure_ascii=False))
|
||||
return 0
|
||||
|
||||
@@ -160,6 +160,43 @@ class JevIntentPreviousTurnTests(unittest.TestCase):
|
||||
self.assertIsNone(state["previous_turn"])
|
||||
self.assertIn("user_message", state)
|
||||
|
||||
def test_closed_focus_is_not_the_current_question(self) -> None:
|
||||
from scripts.research.jev_intent_source_b_v2 import prepare_extract
|
||||
|
||||
rows = prepare_extract([
|
||||
{
|
||||
"turn_id": "open-turn",
|
||||
"case_id": "case-1",
|
||||
"created_at": "2026-09-02T00:00:00+00:00",
|
||||
"user_message": "没有",
|
||||
"assistant_message": "记下了",
|
||||
"case_status": "collecting_evidence",
|
||||
"focus_status": "active",
|
||||
"asked_at": "2026-09-01T00:00:00+00:00",
|
||||
"resolved_at": None,
|
||||
"schema": {"collect": True, "prompt": "有没有换过工作？"},
|
||||
},
|
||||
{
|
||||
"turn_id": "after-close",
|
||||
"case_id": "case-1",
|
||||
"created_at": "2026-09-03T00:00:00+00:00",
|
||||
"user_message": "先看结果",
|
||||
"assistant_message": "还在收",
|
||||
"case_status": "collecting_evidence",
|
||||
"focus_status": "resolved",
|
||||
"asked_at": "2026-09-01T00:00:00+00:00",
|
||||
"resolved_at": "2026-09-02T12:00:00+00:00",
|
||||
"schema": {"collect": True, "prompt": "有没有换过工作？"},
|
||||
},
|
||||
])
|
||||
by_id = {row["turn_id"]: row for row in rows}
|
||||
self.assertEqual(by_id["open-turn"]["layer"], "collect")
|
||||
self.assertEqual(by_id["open-turn"]["focus"]["current_question"], "有没有换过工作？")
|
||||
self.assertEqual(by_id["after-close"]["layer"], "none")
|
||||
self.assertIsNone(by_id["after-close"]["focus"])
|
||||
self.assertTrue(by_id["after-close"]["focus_stale"])
|
||||
self.assertEqual(by_id["after-close"]["previous_turn"]["user_message"], "没有")
|
||||
|
||||
|
||||
class JevIntentCorpusTests(unittest.TestCase):
|
||||
def test_source_a_has_ten_labeled_rows(self) -> None:
|
||||
|
||||
Reference in New Issue
Block a user