research: compare Jev intent state with the previous turn
V0 on the existing 157 real rows matches the 09-19 cache. The staging extract has no case linkage, so V1/V2 are unmeasured and the verdict stays 缺数据.
This commit is contained in:
@@ -1,5 +1,11 @@
|
||||
# BLOCKED
|
||||
|
||||
## Jev 意图 v2:staging 库与 Flash 凭据(2026-09-27)
|
||||
|
||||
- 缺 staging 库。`~/.ssh/jyotisha-staging` 不在本机,默认密钥连 `deploy@118.26.111.127` 被拒。本机 Docker 里的 Postgres 没有 `agentic_rectification_turns`。09-19 交来的 `source_b.jsonl` 157 条没有 `turn_id` / `case_id`,不能还原上一轮。替代证据:V0 用那次缓存重算,有上一轮 = 0,结论写缺数据,没有用空上一轮去打 V1/V2。
|
||||
- 缺 `DEEPSEEK_API_KEY`。Flash V0 用 09-19 缓存(生产提示,157 条 intent 89.2%)。Flash+V1 未跑。
|
||||
- Jev key 已在本机打通:合成样本 V0 / V2 各 1 次,响应 `model=jev-1.13.0`。这 2 次不进计分。SDK 仍是 `typesafe-sdk 0.7.0`。
|
||||
|
||||
## TASK-rectification-telemetry:真实 PostgreSQL 测试与登录态后台待验(2026-09-26)
|
||||
|
||||
- ~~本机无 Docker:`npm run test:db` 未跑。本单新增 `frontend/tests/database-rectification-telemetry.test.ts`(列白名单、RLS 与表权限、函数执行权限、去重、归属校验、非法值整笔回滚、180 天清理、管理员只读汇总、账户删除级联),并把两张新表加进 `database-local-business.test.ts` 的 public 表白名单;两者在本机均为 docker unavailable 跳过,结果以门禁 DB job 为准(run 号待推送后补记)。替代证据:迁移 SQL 按既有迁移写法逐段复核;纯函数与白名单、迁移源文本合同在 `rectification-telemetry.test.ts`(18 条)里已跑通。~~ → 已解除:门禁 run 2938(`e801fcf5`)真实 PostgreSQL 17 跑过,`ok 1337 - rectification telemetry: server-only write, dedupe, retention, aggregate-only admin read, account cascade`,全量 4049 / 0 fail / 0 skip;migrate run 2940 已应用迁移。
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# TypeSafe Jev 接管校正意图分类 · 离线对照(2026-09-19)
|
||||
|
||||
后续对照(把上一轮放进 state):`docs/research/jev_intent_2026_09_27.md`。下面的数字不改。
|
||||
|
||||
- 任务:`docs/tasks/TASK-rectification-jev-intent-classifier-research-20260919.md`
|
||||
- 基线:`origin/staging` @ `69a44fe7`
|
||||
- Jev 模型:`jev-1.13.0`(不用 jev-latest)
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,52 @@
|
||||
# TypeSafe Jev 意图分类 · 上一轮 state 对照(2026-09-27)
|
||||
|
||||
- 任务:`docs/tasks/TASK-rectification-jev-intent-classifier-research-v2-20260927.md`
|
||||
- 基线:`origin/staging` @ `710c848b`
|
||||
- 模型:`jev-1.13.0`;SDK `typesafe-sdk 0.7.0`
|
||||
- 结论:**缺数据**。上一轮变量没有测全,不能判过门。已有的 V0 数字只作对照,不代替 V1/V2。
|
||||
|
||||
## 样本
|
||||
|
||||
- 来源 B:157 条。有上一轮 0,无上一轮 157。
|
||||
- 层:点选 17 / 采集 107 / 无焦点 33。
|
||||
- gold 来源:{"legacy_file": 157}。
|
||||
- 链式与 gold 上界都未跑。09-19 文件没有 turn_id / case_id,本机没有 staging 库。
|
||||
|
||||
## 来源 B
|
||||
|
||||
| 变体 | n | intent | answer_class | dated | 高置信错误 | 低置信召回 | 无焦点 unclear→answer |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
|
||||
| Jev V0(09-19 缓存) | 157 | 91.1% | 98.0% | 91.7% | 6.4% | 24.1% | 7 / 无焦点高置信错误 12.1% |
|
||||
| Flash V0(09-19 缓存,生产提示) | 157 | 89.2% | 97.0% | 93.6% | — | — | 5 / 无焦点高置信错误 — |
|
||||
| Jev V1 | — | — | — | — | — | — | — |
|
||||
| Jev V2 链式 | — | — | — | — | — | — | — |
|
||||
| Jev V2 gold 上界 | — | — | — | — | — | — | — |
|
||||
| Flash + V1 | — | — | — | — | — | — | — |
|
||||
|
||||
Jev V0 自洽率:96.2%。
|
||||
|
||||
## 来源 C 回归锚(只 V0)
|
||||
|
||||
- choice:重算 99.0%,09-19 公布 99.0%,差 0.0 pp(n=400)。
|
||||
- collect:重算 92.2%,09-19 公布 92.2%,差 0.05 pp(n=400)。
|
||||
- none:重算 96.0%,09-19 公布 96.0%,差 0.0 pp(n=100)。
|
||||
- 来源 C 自洽率:99.0%。
|
||||
|
||||
## 无焦点层混淆(来源 B,V0)
|
||||
|
||||
计数来自报告 JSON,不含原文。
|
||||
|
||||
| gold \ pred | provide_new_evidence | stop_rectification | ask_about_result | unclear | answer_current_focus |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| provide_new_evidence | 24 | 0 | 0 | 0 | 0 |
|
||||
| stop_rectification | 0 | 0 | 0 | 0 | 0 |
|
||||
| ask_about_result | 0 | 0 | 1 | 0 | 0 |
|
||||
| unclear | 0 | 0 | 0 | 1 | 7 |
|
||||
| answer_current_focus | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
## 写不出的项
|
||||
|
||||
n_有上一轮 0 < 100;V1 没有预测;V2 链式没有预测;V2 gold 上界没有预测;Flash+V1 没有预测;来源 B 与来源 C 同层 intent 差 > 10pp
|
||||
|
||||
V1、V2、Flash+V1 要等 staging 库抽出带 `case_id` 的上一轮,并且本机有 `DEEPSEEK_API_KEY` 之后才能补。门槛不放宽。
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# 进度 · Jev 意图分类 v2:上一轮放进 state(2026-09-27)
|
||||
|
||||
- 分支:`codex/jev-intent-v2-20260927`
|
||||
- 基线:`origin/staging` @ `710c848b`
|
||||
- 任务书:`docs/tasks/TASK-rectification-jev-intent-classifier-research-v2-20260927.md`
|
||||
- 报告:`docs/research/jev_intent_2026_09_27.md` / `.json`
|
||||
- 结论:**缺数据**。有上一轮 0 条,V1/V2 没有数字,不能判过门。
|
||||
|
||||
## 已跑到的数
|
||||
|
||||
| 项 | 结果 |
|
||||
| --- | --- |
|
||||
| 来源 B | 157(点选 17 / 采集 107 / 无焦点 33)。gold 全部来自 09-19 文件,不是新标 |
|
||||
| 有上一轮 / 无上一轮 | 0 / 157 |
|
||||
| Jev V0 intent | 91.1%(自洽 96.2%,高置信错误 6.4%,低置信召回 24.1%) |
|
||||
| Flash V0 intent | 89.2%(09-19 缓存,生产提示;Flash 没有置信度) |
|
||||
| 无焦点 `unclear → answer_current_focus` | Jev V0 = 7,Flash V0 = 5 |
|
||||
| 来源 C V0 相对 09-19 公布值 | 点选 0 pp,采集 +0.05 pp(公布值四舍五入到 92.2%),无焦点 0 pp |
|
||||
|
||||
V0 没有重打接口,state 与 09-19 相同,数字从 `jev_runs_v2.json` 重算。`--from-report` 只读报告 JSON 的 `rows`,来源 B intent 仍是 91.1%。
|
||||
|
||||
## 接口
|
||||
|
||||
- SDK:`typesafe-sdk 0.7.0`(09-19 进度里写的版本,未升级)
|
||||
- 模型:响应字段 `model=jev-1.13.0`
|
||||
- 连通性:合成样本 V0 1 次,1295 ms,1120 input tok;V2(多一个承接 Noul)1 次,971 ms,`continues_previous_turn=0.16`,规则阈值 0.9 未沿用上一轮。这两次不进计分,费用可忽略
|
||||
- 来源 B / C 的 V1、V2、Flash+V1:0 次
|
||||
|
||||
## 代码
|
||||
|
||||
- `build_state(..., variant=)`:默认仍是四字段。V1 加 `previous_turn`,V2 再加 `previous_decision` 和承接 Noul。Choice 的 criteria 原文没改。
|
||||
- `continues_previous_turn ≥ 0.9` 时沿用上一轮的 intent,并带上它的 answer_class,再走原来的 `enforce_combo`。低于 0.9 用本轮结果。
|
||||
- 没有 `case_id` 的行不会被串成上一轮。
|
||||
- 未改 `frontend/src/**`,未加部署环境变量。
|
||||
|
||||
## 测试
|
||||
|
||||
`python -m unittest tests.test_jev_intent_research`:15 passed。开工时文件里已有 12 条(任务书写的 11 条之外,fix2 已加报告 JSON 那条),本次新增 3 条:V1/V2 state 形状、承接规则、案件首轮 `previous_turn=null`。旧断言未改。
|
||||
|
||||
## 偏离
|
||||
|
||||
1. 重抽来源 B 被 staging 库挡住,见 `BLOCKED.md`。旧 157 条对不上 `turn_id`,因为交来的文件里没有这个字段。
|
||||
2. 没有拿「上一轮 = null」去打满 V1/V2。那样测的是空字段,不是 Magpie 的喂法。
|
||||
3. Flash V0 不可省,所以用了 09-19 的生产提示缓存,没有新的 Flash 调用。Flash+V1 按让步可以缺,这里确实缺。
|
||||
4. 来源 C 没有上一轮,按任务书不跑 V1/V2。V0 用缓存重算,没有再花 900 次调用。
|
||||
@@ -295,7 +295,7 @@
|
||||
|
||||
| `TASK-rectification-cluster-width-research-20260914.md` | `PROGRESS-rectification-cluster-width-research-20260914.md` | **研究单**:上一轮证明调权重改不动交付区间宽度——所有方案宽度中位数都等于整个搜索窗。先确认 sweep 的宽度口径是否含淘汰(M0),再画簇结构像(M1),最后量三个改法:放宽簇上限、按分差决定是否合并、交付区间改分位覆盖(M2)。真值覆盖率不得下降 | 已验收(结论收入定论页 `docs/research/rectification_minute_resolution_closure_2026_09_14.md`,`a643452b`) | `cf972f40` |
|
||||
| `TASK-rectification-jev-intent-classifier-research-20260919.md` | `PROGRESS-rectification-jev-intent-classifier-research-20260919.md` | **研究单**:TypeSafe Jev(只做 Choice/Score/Noul 的校准判断模型,$0.042/Mtok)能否接管校正流的意图分类。产品 09-19 授权评估(推翻 09-15「分类只用贵模型」需重新拍板)。Agent 模拟校正流造 ≥900 条中文语料(标签先定、独立复核)+ 真机样本做代表性锚,量准确率 / 高置信错误率 / 低置信召回 / 延迟;只离线测,不改线上 | **关单**(产品 2026-09-19 拍板):结论 **缺数据**,Jev 不接管、不影子双跑、不再造语料;09-15「分类只用贵模型」口径维持 | 真人 157 条:Jev intent 91.1% vs 线上 Flash 89.2%,无焦点层两边都差(78.8% / 69.7%,同一错模式);挡住上线的是 Jev 置信度不可用:高置信错误全集 6.4% / 无焦点 12.1% > 3%,低置信召回 24%。fix2 507ef959 已验收通过 |
|
||||
| `TASK-rectification-jev-intent-classifier-research-v2-20260927.md` | `PROGRESS-rectification-jev-intent-classifier-research-v2-20260927.md` | **研究单 v2**:Magpie 把上一轮决策塞进 Jev state 后路由效果大改;核实 TypeSafe 仍只有 `jev-1.13.0`(与 09-19 同版本),变量只在喂法。产品 09-27 授权:重抽真人来源 B(带 `case_id` 与上一轮,推翻 09-19「不再造语料」),state 三变体 V0/V1(+上一轮)/V2(+上一轮决策+承接 Noul),门槛沿用 09-19 不放宽;不接管、不影子双跑维持 | **待执行**(任务书 2026-09-27 推 staging,产品自备 key) | 无焦点层 `unclear→answer_current_focus` 逐变体计数、有/无上一轮分组、Flash+V1 归因列;结论过门 / 未过门 / 缺数据 |
|
||||
| `TASK-rectification-jev-intent-classifier-research-v2-20260927.md` | `PROGRESS-rectification-jev-intent-classifier-research-v2-20260927.md` | **研究单 v2**:Magpie 把上一轮决策塞进 Jev state 后路由效果大改;核实 TypeSafe 仍只有 `jev-1.13.0`(与 09-19 同版本),变量只在喂法。产品 09-27 授权:重抽真人来源 B(带 `case_id` 与上一轮,推翻 09-19「不再造语料」),state 三变体 V0/V1(+上一轮)/V2(+上一轮决策+承接 Noul),门槛沿用 09-19 不放宽;不接管、不影子双跑维持 | **待验收**(结论缺数据:有上一轮 0 条,V1/V2 与 Flash+V1 未跑) | 分支 `codex/jev-intent-v2-20260927`;V0 重算与 09-19 一致(真人 Jev 91.1% / Flash 89.2%,无焦点 unclear→答题 7);报告 `docs/research/jev_intent_2026_09_27.md` |
|
||||
|
||||
| `TASK-rectification-minute-resolution-research-20260914.md` | `PROGRESS-rectification-minute-resolution-research-20260914.md` | **研究单**:候选分不开的根因是打分尺度——窗口内恒定项 11.5 分 vs 随分钟变化项 2.125 分(≈5:1)。先修封存基准(v3 每例仅 3 件事且被标 invalidated)出 v4,再离线量五个改法:分盘除数、去底座、**KP 宫头子主计分(产品 09-14 拍板,推翻 BUG-325 一条红线)**、年精度事件改边际似然、聚类签名层对齐。有收益才立实现单 | 已验收(结论收入定论页 `docs/research/rectification_minute_resolution_closure_2026_09_14.md`,`a643452b`) | `2d2467dc` |
|
||||
|
||||
|
||||
@@ -74,7 +74,7 @@ def _parse_content(text: str) -> dict[str, Any]:
|
||||
return payload
|
||||
|
||||
|
||||
def call_current(sample: Mapping[str, Any], *, timeout: float = 60.0) -> dict[str, Any]:
|
||||
def call_current(sample: Mapping[str, Any], *, timeout: float = 60.0, variant: str = "v0") -> dict[str, Any]:
|
||||
api_key = os.environ.get("DEEPSEEK_API_KEY") or ""
|
||||
if not api_key:
|
||||
raise RuntimeError("DEEPSEEK_API_KEY missing")
|
||||
@@ -84,7 +84,7 @@ def call_current(sample: Mapping[str, Any], *, timeout: float = 60.0) -> dict[st
|
||||
"model": model,
|
||||
"messages": [
|
||||
{"role": "system", "content": _instructions(sample) + "\n" + JSON_SCHEMA_HINT},
|
||||
{"role": "user", "content": json.dumps(build_state(sample), ensure_ascii=False)},
|
||||
{"role": "user", "content": json.dumps(build_state(sample, variant=variant if variant == "v1" else "v0"), ensure_ascii=False)},
|
||||
],
|
||||
"temperature": 0,
|
||||
"max_tokens": 256,
|
||||
@@ -128,12 +128,12 @@ def call_current(sample: Mapping[str, Any], *, timeout: float = 60.0) -> dict[st
|
||||
}
|
||||
|
||||
|
||||
def call_current_retry(sample: Mapping[str, Any], *, retries: int = 4) -> dict[str, Any]:
|
||||
def call_current_retry(sample: Mapping[str, Any], *, retries: int = 4, variant: str = "v0") -> dict[str, Any]:
|
||||
last_error = ""
|
||||
delay = 1.0
|
||||
for _attempt in range(retries):
|
||||
try:
|
||||
return call_current(sample)
|
||||
return call_current(sample, variant=variant)
|
||||
except urllib.error.HTTPError as exc:
|
||||
last_error = f"HTTP{exc.code}"
|
||||
if exc.code in {429, 500, 502, 503, 529}:
|
||||
|
||||
@@ -55,14 +55,19 @@ def gold_of(sample: Mapping[str, Any]) -> dict[str, Any]:
|
||||
return enforce_combo(gold.get("intent"), gold.get("answer_class"), gold.get("has_new_dated_event"))
|
||||
|
||||
|
||||
def call_jev(sample: Mapping[str, Any], *, timeout: float = 60.0) -> dict[str, Any]:
|
||||
def call_jev(
|
||||
sample: Mapping[str, Any],
|
||||
*,
|
||||
timeout: float = 60.0,
|
||||
variant: str = "v0",
|
||||
) -> dict[str, Any]:
|
||||
from typesafe_sdk import TypeSafeClient
|
||||
|
||||
started = time.perf_counter()
|
||||
with TypeSafeClient() as client:
|
||||
response = client.system_one(
|
||||
state=build_state(sample),
|
||||
questions=sdk_questions(sample),
|
||||
state=build_state(sample, variant=variant),
|
||||
questions=sdk_questions(sample, variant=variant),
|
||||
model=JEV_MODEL,
|
||||
timeout=timeout,
|
||||
)
|
||||
@@ -81,7 +86,11 @@ def call_jev(sample: Mapping[str, Any], *, timeout: float = 60.0) -> dict[str, A
|
||||
"noul": getattr(value, "noul", None),
|
||||
}
|
||||
answers = converted
|
||||
parsed = parse_jev_answers(answers)
|
||||
parsed = parse_jev_answers(
|
||||
answers,
|
||||
previous_decision=sample.get("previous_decision") if isinstance(sample.get("previous_decision"), dict) else None,
|
||||
apply_continue=variant == "v2",
|
||||
)
|
||||
usage = payload.get("usage") or {}
|
||||
return {
|
||||
"ok": True,
|
||||
@@ -95,15 +104,18 @@ def call_jev(sample: Mapping[str, Any], *, timeout: float = 60.0) -> dict[str, A
|
||||
"input_tokens": usage.get("input_tokens"),
|
||||
"output_tokens": usage.get("output_tokens"),
|
||||
"elapsed_ms": elapsed_ms,
|
||||
"continued": parsed.get("continued"),
|
||||
"continues_previous_turn": parsed.get("continues_previous_turn"),
|
||||
"variant": variant,
|
||||
}
|
||||
|
||||
|
||||
def call_jev_retry(sample: Mapping[str, Any], *, retries: int = 4) -> dict[str, Any]:
|
||||
def call_jev_retry(sample: Mapping[str, Any], *, retries: int = 4, variant: str = "v0") -> dict[str, Any]:
|
||||
last_error = ""
|
||||
delay = 1.0
|
||||
for attempt in range(retries):
|
||||
try:
|
||||
return call_jev(sample)
|
||||
return call_jev(sample, variant=variant)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
last_error = type(exc).__name__
|
||||
name = type(exc).__name__
|
||||
@@ -127,6 +139,9 @@ def call_jev_retry(sample: Mapping[str, Any], *, retries: int = 4) -> dict[str,
|
||||
"input_tokens": None,
|
||||
"output_tokens": None,
|
||||
"elapsed_ms": None,
|
||||
"continued": None,
|
||||
"continues_previous_turn": None,
|
||||
"variant": variant,
|
||||
"error": last_error,
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,459 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Jev intent research v2: previous-turn state variants.
|
||||
|
||||
Offline by default for the report. Source B text stays in the gitignored cache.
|
||||
Committed rows keep gold and model output only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping, Sequence
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
if str(ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from scripts.research.jev_intent_probe import ( # noqa: E402
|
||||
CACHE_DIR,
|
||||
cache_key,
|
||||
confusion_counts,
|
||||
layer_metrics,
|
||||
load_cache,
|
||||
load_jsonl,
|
||||
self_consistency,
|
||||
strip_confidence,
|
||||
)
|
||||
from scripts.research.jev_intent_questions import ( # noqa: E402
|
||||
JEV_MODEL,
|
||||
previous_turn_payload,
|
||||
)
|
||||
from scripts.research.jev_intent_source_b_v2 import OUTPUT_PATH, from_legacy # noqa: E402
|
||||
|
||||
REPORT_JSON = ROOT / "docs" / "research" / "jev_intent_2026_09_27.json"
|
||||
REPORT_MD = ROOT / "docs" / "research" / "jev_intent_2026_09_27.md"
|
||||
SAMPLES_DIR = ROOT / "scripts" / "research" / "jev_intent_samples"
|
||||
MIN_WITH_PREVIOUS = 100
|
||||
PUBLISHED_C_INTENT = {"choice": 0.990, "collect": 0.922, "none": 0.960}
|
||||
|
||||
PRED_KEEP = (
|
||||
"ok",
|
||||
"unavailable",
|
||||
"model",
|
||||
"intent",
|
||||
"answer_class",
|
||||
"has_new_dated_event",
|
||||
"confidence",
|
||||
"raw",
|
||||
"input_tokens",
|
||||
"output_tokens",
|
||||
"elapsed_ms",
|
||||
"continued",
|
||||
"continues_previous_turn",
|
||||
"variant",
|
||||
"error",
|
||||
)
|
||||
|
||||
|
||||
def pct(value: float | None) -> str:
|
||||
if value is None:
|
||||
return "—"
|
||||
return f"{value:.1%}"
|
||||
|
||||
|
||||
def public_pred(pred: Mapping[str, Any] | None) -> dict[str, Any] | None:
|
||||
if not pred:
|
||||
return None
|
||||
return {key: pred.get(key) for key in PRED_KEEP if key in pred}
|
||||
|
||||
|
||||
def has_previous(row: Mapping[str, Any]) -> bool:
|
||||
return previous_turn_payload(row) is not None
|
||||
|
||||
|
||||
def unclear_to_focus(rows: Sequence[Mapping[str, Any]], pred_key: str) -> int:
|
||||
count = 0
|
||||
for row in rows:
|
||||
if row.get("layer") != "none":
|
||||
continue
|
||||
if (row.get("gold") or {}).get("intent") != "unclear":
|
||||
continue
|
||||
if (row.get(pred_key) or {}).get("intent") == "answer_current_focus":
|
||||
count += 1
|
||||
return count
|
||||
|
||||
|
||||
def strip_block(block: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||
"""Flash has no confidence. Leave those rates empty instead of printing 0%."""
|
||||
if not block:
|
||||
return None
|
||||
out = dict(block)
|
||||
out["all"] = strip_confidence(block.get("all"))
|
||||
out["by_layer"] = {
|
||||
layer: strip_confidence(metrics) if metrics else None
|
||||
for layer, metrics in (block.get("by_layer") or {}).items()
|
||||
}
|
||||
out["with_previous"] = strip_confidence(block.get("with_previous"))
|
||||
out["without_previous"] = strip_confidence(block.get("without_previous"))
|
||||
return out
|
||||
|
||||
|
||||
def pack_metrics(rows: Sequence[Mapping[str, Any]], pred_key: str) -> dict[str, Any] | None:
|
||||
usable = [row for row in rows if row.get(pred_key)]
|
||||
if not usable:
|
||||
return None
|
||||
by_layer = {}
|
||||
for layer in ("choice", "collect", "none"):
|
||||
subset = [row for row in usable if row.get("layer") == layer]
|
||||
by_layer[layer] = layer_metrics(subset, pred_key=pred_key) if subset else None
|
||||
with_prev = [row for row in usable if row.get("has_previous_turn")]
|
||||
without_prev = [row for row in usable if not row.get("has_previous_turn")]
|
||||
return {
|
||||
"all": layer_metrics(usable, pred_key=pred_key),
|
||||
"by_layer": by_layer,
|
||||
"with_previous": layer_metrics(with_prev, pred_key=pred_key) if with_prev else None,
|
||||
"without_previous": layer_metrics(without_prev, pred_key=pred_key) if without_prev else None,
|
||||
"none_unclear_to_focus": unclear_to_focus(usable, pred_key),
|
||||
"self_consistency": None,
|
||||
}
|
||||
|
||||
|
||||
def attach_legacy_preds(samples: Sequence[dict[str, Any]], cache: Mapping[str, Any], mapping: Mapping[str, str]) -> dict[str, int]:
|
||||
missing = {dest: 0 for dest in mapping}
|
||||
for sample in samples:
|
||||
for dest, legacy in mapping.items():
|
||||
key = cache_key(legacy, sample)
|
||||
if key in cache:
|
||||
pred = dict(cache[key])
|
||||
pred["variant"] = "v0"
|
||||
sample[dest] = pred
|
||||
else:
|
||||
missing[dest] += 1
|
||||
return missing
|
||||
|
||||
|
||||
def public_rows(samples: Sequence[Mapping[str, Any]], pred_keys: Sequence[str]) -> list[dict[str, Any]]:
|
||||
rows = []
|
||||
for sample in samples:
|
||||
row = {
|
||||
"id": sample.get("id"),
|
||||
"source": sample.get("source"),
|
||||
"layer": sample.get("layer"),
|
||||
"gold": sample.get("gold"),
|
||||
"gold_source": sample.get("gold_source"),
|
||||
"has_previous_turn": bool(sample.get("has_previous_turn")),
|
||||
"previous_turn_source": sample.get("previous_turn_source"),
|
||||
}
|
||||
for key in pred_keys:
|
||||
pred = public_pred(sample.get(key) if isinstance(sample.get(key), dict) else None)
|
||||
if pred:
|
||||
row[key] = pred
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
def decide(meta: Mapping[str, Any], metrics: Mapping[str, Any]) -> dict[str, str]:
|
||||
blocked: list[str] = []
|
||||
if meta.get("n_with_previous", 0) < MIN_WITH_PREVIOUS:
|
||||
blocked.append(f"n_有上一轮 {meta.get('n_with_previous', 0)} < {MIN_WITH_PREVIOUS}")
|
||||
if not metrics.get("jev_v1"):
|
||||
blocked.append("V1 没有预测")
|
||||
if not metrics.get("jev_v2_chain"):
|
||||
blocked.append("V2 链式没有预测")
|
||||
if not metrics.get("jev_v2_gold"):
|
||||
blocked.append("V2 gold 上界没有预测")
|
||||
if not metrics.get("flash_v1"):
|
||||
blocked.append("Flash+V1 没有预测")
|
||||
if not metrics.get("flash_v0"):
|
||||
blocked.append("Flash V0 没有预测")
|
||||
represent = metrics.get("representativeness") or {}
|
||||
if represent.get("fail"):
|
||||
blocked.append("来源 B 与来源 C 同层 intent 差 > 10pp")
|
||||
if blocked:
|
||||
return {
|
||||
"verdict": "缺数据",
|
||||
"blocked": ";".join(blocked),
|
||||
"reason": "上一轮变量没有测全,不能判过门。已有的 V0 数字只作对照,不代替 V1/V2。",
|
||||
}
|
||||
v0_none = ((metrics.get("jev_v0") or {}).get("by_layer") or {}).get("none") or {}
|
||||
v2_none = ((metrics.get("jev_v2_chain") or {}).get("by_layer") or {}).get("none") or {}
|
||||
v0_high = v0_none.get("high_conf_error_rate")
|
||||
v2_high = v2_none.get("high_conf_error_rate")
|
||||
if v0_high is None or v2_high is None:
|
||||
return {"verdict": "blocked", "blocked": "无焦点层高置信错误写不出", "reason": "V2 相对 V0 的无焦点层比较缺数字。"}
|
||||
jev = (metrics.get("jev_v2_chain") or {}).get("all") or {}
|
||||
flash = (metrics.get("flash_v0") or {}).get("all") or {}
|
||||
high = jev.get("high_conf_error_rate")
|
||||
recall = jev.get("low_conf_recall")
|
||||
jev_intent = jev.get("intent_acc")
|
||||
flash_intent = flash.get("intent_acc")
|
||||
missing = [
|
||||
name for name, value in (
|
||||
("高置信错误", high),
|
||||
("低置信召回", recall),
|
||||
("V2 intent", jev_intent),
|
||||
("Flash V0 intent", flash_intent),
|
||||
) if value is None
|
||||
]
|
||||
if missing:
|
||||
return {"verdict": "blocked", "blocked": "、".join(missing), "reason": "门槛里有写不出的数字。"}
|
||||
gates = []
|
||||
if high > 0.03:
|
||||
gates.append(f"高置信错误 {high:.1%} > 3%")
|
||||
if recall < 0.60:
|
||||
gates.append(f"低置信召回 {recall:.1%} < 60%")
|
||||
if jev_intent < flash_intent - 0.03:
|
||||
gates.append(f"intent {jev_intent:.1%} < 现行 {flash_intent:.1%} − 3pp")
|
||||
if v2_high >= v0_high:
|
||||
gates.append(f"无焦点层高置信错误 V2 {v2_high:.1%} 没有低于 V0 {v0_high:.1%}")
|
||||
if gates:
|
||||
return {"verdict": "未过门", "blocked": "", "reason": ";".join(gates)}
|
||||
return {
|
||||
"verdict": "过门",
|
||||
"blocked": "",
|
||||
"reason": "三项门槛都过,且 V2 无焦点层高置信错误低于 V0。",
|
||||
}
|
||||
|
||||
|
||||
def representativeness(source_b: Sequence[Mapping[str, Any]], source_c: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
|
||||
layers = {}
|
||||
fail = False
|
||||
for layer in ("choice", "collect", "none"):
|
||||
b_rows = [row for row in source_b if row.get("layer") == layer and row.get("jev_v0_1")]
|
||||
c_rows = [row for row in source_c if row.get("layer") == layer and row.get("jev_v0_1")]
|
||||
if not b_rows or not c_rows:
|
||||
layers[layer] = {"n_b": len(b_rows), "n_c": len(c_rows), "delta": None}
|
||||
continue
|
||||
b_acc = layer_metrics(b_rows, pred_key="jev_v0_1")["intent_acc"]
|
||||
c_acc = layer_metrics(c_rows, pred_key="jev_v0_1")["intent_acc"]
|
||||
delta = abs(b_acc - c_acc)
|
||||
fail = fail or delta > 0.10
|
||||
published = PUBLISHED_C_INTENT[layer]
|
||||
layers[layer] = {
|
||||
"n_b": len(b_rows),
|
||||
"n_c": len(c_rows),
|
||||
"source_b_intent": b_acc,
|
||||
"source_c_intent": c_acc,
|
||||
"delta": delta,
|
||||
"published_c_intent": published,
|
||||
"published_delta_pp": (c_acc - published) * 100,
|
||||
}
|
||||
return {"fail": fail, "by_layer": layers}
|
||||
|
||||
|
||||
def build_from_samples(source_b: Sequence[dict[str, Any]], source_c: Sequence[dict[str, Any]]) -> dict[str, Any]:
|
||||
for row in list(source_b) + list(source_c):
|
||||
if "previous_turn" in row:
|
||||
row["has_previous_turn"] = has_previous(row)
|
||||
else:
|
||||
row["has_previous_turn"] = bool(row.get("has_previous_turn"))
|
||||
pred_keys = ("jev_v0_1", "jev_v0_2", "flash_v0")
|
||||
b_metrics = {
|
||||
"jev_v0": pack_metrics(source_b, "jev_v0_1"),
|
||||
"flash_v0": strip_block(pack_metrics(source_b, "flash_v0")),
|
||||
"jev_v1": pack_metrics(source_b, "jev_v1_1"),
|
||||
"jev_v2_chain": pack_metrics(source_b, "jev_v2_chain_1"),
|
||||
"jev_v2_gold": pack_metrics(source_b, "jev_v2_gold_1"),
|
||||
"flash_v1": strip_block(pack_metrics(source_b, "flash_v1")),
|
||||
}
|
||||
if b_metrics["jev_v0"]:
|
||||
b_metrics["jev_v0"]["self_consistency"] = self_consistency(source_b, "jev_v0_1", "jev_v0_2")
|
||||
none_b = [row for row in source_b if row.get("layer") == "none"]
|
||||
meta = {
|
||||
"model": JEV_MODEL,
|
||||
"sdk": "typesafe-sdk 0.7.0",
|
||||
"source_b_n": len(source_b),
|
||||
"n_with_previous": sum(1 for row in source_b if row.get("has_previous_turn")),
|
||||
"n_without_previous": sum(1 for row in source_b if not row.get("has_previous_turn")),
|
||||
"layers": {
|
||||
layer: sum(1 for row in source_b if row.get("layer") == layer)
|
||||
for layer in ("choice", "collect", "none")
|
||||
},
|
||||
"gold_source": {},
|
||||
"source_c_n": len(source_c),
|
||||
"previous_decision_note": "链式与 gold 上界都未跑。09-19 文件没有 turn_id / case_id,本机没有 staging 库。",
|
||||
}
|
||||
gold_counts: dict[str, int] = {}
|
||||
for row in source_b:
|
||||
source = str(row.get("gold_source") or "unknown")
|
||||
gold_counts[source] = gold_counts.get(source, 0) + 1
|
||||
meta["gold_source"] = gold_counts
|
||||
metrics = {
|
||||
**b_metrics,
|
||||
"source_c_v0": pack_metrics(source_c, "jev_v0_1"),
|
||||
"representativeness": representativeness(source_b, source_c),
|
||||
"source_b_none_confusion": {
|
||||
"jev_v0": confusion_counts(none_b, "jev_v0_1"),
|
||||
"flash_v0": confusion_counts(none_b, "flash_v0"),
|
||||
} if none_b else {},
|
||||
}
|
||||
if metrics["source_c_v0"]:
|
||||
metrics["source_c_v0"]["self_consistency"] = self_consistency(source_c, "jev_v0_1", "jev_v0_2")
|
||||
conclusion = decide(meta, metrics)
|
||||
return {
|
||||
"meta": meta,
|
||||
"metrics": metrics,
|
||||
"conclusion": conclusion,
|
||||
"rows": public_rows(list(source_b) + list(source_c), pred_keys + (
|
||||
"jev_v1_1", "jev_v1_2", "jev_v2_chain_1", "jev_v2_chain_2",
|
||||
"jev_v2_gold_1", "jev_v2_gold_2", "flash_v1",
|
||||
)),
|
||||
}
|
||||
|
||||
|
||||
def load_samples() -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, int]]:
|
||||
if OUTPUT_PATH.is_file():
|
||||
source_b = load_jsonl(OUTPUT_PATH)
|
||||
else:
|
||||
source_b = from_legacy(load_jsonl(CACHE_DIR / "source_b.jsonl"))
|
||||
source_b = [row for row in source_b if isinstance(row.get("gold"), dict) and row["gold"].get("intent")]
|
||||
source_c = load_jsonl(SAMPLES_DIR / "simulated.jsonl")
|
||||
cache = load_cache()
|
||||
missing = {}
|
||||
missing.update(attach_legacy_preds(source_b, cache, {
|
||||
"jev_v0_1": "jev_1",
|
||||
"jev_v0_2": "jev_2",
|
||||
"flash_v0": "current_1",
|
||||
}))
|
||||
missing.update(attach_legacy_preds(source_c, cache, {
|
||||
"jev_v0_1": "jev_1",
|
||||
"jev_v0_2": "jev_2",
|
||||
}))
|
||||
return source_b, source_c, missing
|
||||
|
||||
|
||||
def recompute_from_rows(rows: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
|
||||
source_b = [dict(row) for row in rows if row.get("source") == "B"]
|
||||
source_c = [dict(row) for row in rows if row.get("source") == "C"]
|
||||
return build_from_samples(source_b, source_c)
|
||||
|
||||
|
||||
def metric_line(title: str, block: Mapping[str, Any] | None) -> str:
|
||||
if not block or not block.get("all"):
|
||||
return f"| {title} | — | — | — | — | — | — | — |"
|
||||
all_m = block["all"]
|
||||
none_high = ((block.get("by_layer") or {}).get("none") or {}).get("high_conf_error_rate")
|
||||
return (
|
||||
f"| {title} | {all_m['n']} | {pct(all_m['intent_acc'])} | {pct(all_m['answer_class_acc'])} | "
|
||||
f"{pct(all_m['dated_acc'])} | {pct(all_m['high_conf_error_rate'])} | {pct(all_m['low_conf_recall'])} | "
|
||||
f"{block.get('none_unclear_to_focus')} / 无焦点高置信错误 {pct(none_high)} |"
|
||||
)
|
||||
|
||||
|
||||
def write_markdown(report: Mapping[str, Any]) -> None:
|
||||
meta = report["meta"]
|
||||
metrics = report["metrics"]
|
||||
conclusion = report["conclusion"]
|
||||
layers = meta.get("layers") or {}
|
||||
gold = meta.get("gold_source") or {}
|
||||
represent = metrics.get("representativeness") or {}
|
||||
lines = [
|
||||
"# TypeSafe Jev 意图分类 · 上一轮 state 对照(2026-09-27)",
|
||||
"",
|
||||
f"- 任务:`docs/tasks/TASK-rectification-jev-intent-classifier-research-v2-20260927.md`",
|
||||
f"- 基线:`origin/staging` @ `710c848b`",
|
||||
f"- 模型:`{meta.get('model')}`;SDK `{meta.get('sdk')}`",
|
||||
f"- 结论:**{conclusion.get('verdict')}**。{conclusion.get('reason')}",
|
||||
"",
|
||||
"## 样本",
|
||||
"",
|
||||
f"- 来源 B:{meta.get('source_b_n')} 条。有上一轮 {meta.get('n_with_previous')},无上一轮 {meta.get('n_without_previous')}。",
|
||||
f"- 层:点选 {layers.get('choice', 0)} / 采集 {layers.get('collect', 0)} / 无焦点 {layers.get('none', 0)}。",
|
||||
f"- gold 来源:{json.dumps(gold, ensure_ascii=False)}。",
|
||||
f"- {meta.get('previous_decision_note')}",
|
||||
"",
|
||||
"## 来源 B",
|
||||
"",
|
||||
"| 变体 | n | intent | answer_class | dated | 高置信错误 | 低置信召回 | 无焦点 unclear→answer |",
|
||||
"| --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |",
|
||||
metric_line("Jev V0(09-19 缓存)", metrics.get("jev_v0")),
|
||||
metric_line("Flash V0(09-19 缓存,生产提示)", metrics.get("flash_v0")),
|
||||
metric_line("Jev V1", metrics.get("jev_v1")),
|
||||
metric_line("Jev V2 链式", metrics.get("jev_v2_chain")),
|
||||
metric_line("Jev V2 gold 上界", metrics.get("jev_v2_gold")),
|
||||
metric_line("Flash + V1", metrics.get("flash_v1")),
|
||||
"",
|
||||
f"Jev V0 自洽率:{pct((metrics.get('jev_v0') or {}).get('self_consistency'))}。",
|
||||
"",
|
||||
"## 来源 C 回归锚(只 V0)",
|
||||
"",
|
||||
]
|
||||
c_block = metrics.get("source_c_v0") or {}
|
||||
for layer in ("choice", "collect", "none"):
|
||||
cell = (represent.get("by_layer") or {}).get(layer) or {}
|
||||
published = cell.get("published_c_intent")
|
||||
got = cell.get("source_c_intent")
|
||||
delta = cell.get("published_delta_pp")
|
||||
lines.append(
|
||||
f"- {layer}:重算 {pct(got)},09-19 公布 {pct(published)},差 {delta if delta is None else round(delta, 2)} pp(n={cell.get('n_c')})。"
|
||||
)
|
||||
lines += [
|
||||
f"- 来源 C 自洽率:{pct(c_block.get('self_consistency'))}。",
|
||||
"",
|
||||
"## 无焦点层混淆(来源 B,V0)",
|
||||
"",
|
||||
"计数来自报告 JSON,不含原文。",
|
||||
"",
|
||||
]
|
||||
confusion = (metrics.get("source_b_none_confusion") or {}).get("jev_v0") or {}
|
||||
labels = confusion.get("labels") or []
|
||||
counts = confusion.get("counts") or {}
|
||||
if labels:
|
||||
lines.append("| gold \\ pred | " + " | ".join(labels) + " |")
|
||||
lines.append("| --- | " + " | ".join("---:" for _ in labels) + " |")
|
||||
for gold in labels:
|
||||
cells = [str((counts.get(gold) or {}).get(pred, 0)) for pred in labels]
|
||||
lines.append(f"| {gold} | " + " | ".join(cells) + " |")
|
||||
lines += [
|
||||
"",
|
||||
"## 写不出的项",
|
||||
"",
|
||||
conclusion.get("blocked") or "无",
|
||||
"",
|
||||
"V1、V2、Flash+V1 要等 staging 库抽出带 `case_id` 的上一轮,并且本机有 `DEEPSEEK_API_KEY` 之后才能补。门槛不放宽。",
|
||||
"",
|
||||
]
|
||||
REPORT_MD.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--offline", action="store_true")
|
||||
parser.add_argument("--from-report", action="store_true", help="recompute tables from the committed JSON rows")
|
||||
args = parser.parse_args(argv)
|
||||
if args.from_report:
|
||||
if not REPORT_JSON.is_file():
|
||||
print("report json missing", file=sys.stderr)
|
||||
return 2
|
||||
payload = json.loads(REPORT_JSON.read_text(encoding="utf-8"))
|
||||
report = recompute_from_rows(payload.get("rows") or [])
|
||||
report["meta"]["recomputed_from"] = "report_rows"
|
||||
write_markdown(report)
|
||||
print(json.dumps({
|
||||
"verdict": report["conclusion"]["verdict"],
|
||||
"source_b_intent": ((report["metrics"].get("jev_v0") or {}).get("all") or {}).get("intent_acc"),
|
||||
"n": report["meta"]["source_b_n"],
|
||||
"n_with_previous": report["meta"]["n_with_previous"],
|
||||
}, ensure_ascii=False))
|
||||
return 0
|
||||
source_b, source_c, missing = load_samples()
|
||||
report = build_from_samples(source_b, source_c)
|
||||
report["meta"]["legacy_cache_missing"] = missing
|
||||
report["meta"]["offline"] = True
|
||||
REPORT_JSON.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
write_markdown(report)
|
||||
print(json.dumps({
|
||||
"verdict": report["conclusion"]["verdict"],
|
||||
"source_b_n": report["meta"]["source_b_n"],
|
||||
"n_with_previous": report["meta"]["n_with_previous"],
|
||||
"missing": missing,
|
||||
"json": str(REPORT_JSON),
|
||||
}, ensure_ascii=False))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -20,6 +20,8 @@ ANSWER_CLASS_VALUES = ("yes", "weak_yes", "no", "unsure")
|
||||
COLLECT_ANSWER_CLASSES = ("yes", "weak_yes", "no", "unsure")
|
||||
JEV_MODEL = "jev-1.13.0"
|
||||
NOUL_TRUE_THRESHOLD = 0.5
|
||||
CONTINUE_THRESHOLD = 0.9
|
||||
STATE_VARIANTS = ("v0", "v1", "v2")
|
||||
|
||||
# Literal criteria. Production hedges ("通常", "不要猜", "不要按关键词") are dropped.
|
||||
INTENT_INSTRUCTIONS = {
|
||||
@@ -71,6 +73,12 @@ HAS_NEW_DATED_EVENT_CRITERIA = {
|
||||
),
|
||||
}
|
||||
|
||||
CONTINUES_PREVIOUS_TURN_INSTRUCTIONS = "这句是否只是承接上一轮,没有新的意思。"
|
||||
CONTINUES_PREVIOUS_TURN_CRITERIA = {
|
||||
"true": "这句只承接上一轮,没有提出新问题,也没有补充新的意思。",
|
||||
"false": "这句是在回答新问题、另起一件事,或补充了上一轮没有的意思。",
|
||||
}
|
||||
|
||||
# One row per production prompt sentence that was rewritten or dropped.
|
||||
CRITERION_MAP: tuple[dict[str, str], ...] = (
|
||||
{
|
||||
@@ -201,26 +209,83 @@ def answer_class_criteria(
|
||||
return criteria
|
||||
|
||||
|
||||
def build_state(sample: Mapping[str, Any]) -> dict[str, Any]:
|
||||
def previous_turn_payload(sample: Mapping[str, Any]) -> dict[str, str] | None:
|
||||
"""Prior turn in the same case. Missing case linkage or a first turn is null."""
|
||||
prev = sample.get("previous_turn")
|
||||
if not isinstance(prev, dict):
|
||||
return None
|
||||
if "assistant_message" not in prev and "user_message" not in prev:
|
||||
return None
|
||||
return {
|
||||
"assistant_message": "" if prev.get("assistant_message") is None else str(prev.get("assistant_message")),
|
||||
"user_message": "" if prev.get("user_message") is None else str(prev.get("user_message")),
|
||||
}
|
||||
|
||||
|
||||
def previous_decision_payload(sample: Mapping[str, Any]) -> dict[str, Any] | None:
|
||||
decision = sample.get("previous_decision")
|
||||
if not isinstance(decision, dict):
|
||||
return None
|
||||
intent = decision.get("intent")
|
||||
if intent not in INTENT_VALUES:
|
||||
return None
|
||||
answer = decision.get("answer_class")
|
||||
if answer not in ANSWER_CLASS_VALUES:
|
||||
answer = None
|
||||
return {"intent": intent, "answer_class": answer}
|
||||
|
||||
|
||||
def intent_instructions(variant: str = "v0") -> dict[str, Any]:
|
||||
"""V0 stays the 09-19 instructions. V1/V2 name the extra state fields; criteria do not change."""
|
||||
if variant not in STATE_VARIANTS:
|
||||
raise ValueError(f"unknown state variant: {variant}")
|
||||
if variant == "v0":
|
||||
return INTENT_INSTRUCTIONS
|
||||
inputs = ["current_question", "options", "user_message", "case_status", "previous_turn"]
|
||||
context = (
|
||||
"previous_turn is the immediately previous assistant_message and user_message in this case. "
|
||||
"It is null on the first turn. Classify the current user_message, not the previous one."
|
||||
)
|
||||
if variant == "v2":
|
||||
inputs.append("previous_decision")
|
||||
context += " previous_decision is the prior turn's intent and answer_class, not a label for this turn."
|
||||
return {
|
||||
"task": INTENT_INSTRUCTIONS["task"],
|
||||
"inputs": inputs,
|
||||
"rule": INTENT_INSTRUCTIONS["rule"],
|
||||
"context": context,
|
||||
}
|
||||
|
||||
|
||||
def build_state(sample: Mapping[str, Any], *, variant: str = "v0") -> dict[str, Any]:
|
||||
if variant not in STATE_VARIANTS:
|
||||
raise ValueError(f"unknown state variant: {variant}")
|
||||
focus = sample.get("focus") if isinstance(sample.get("focus"), dict) else {}
|
||||
options = list(focus.get("options") or [])
|
||||
return {
|
||||
state: dict[str, Any] = {
|
||||
"current_question": str(focus.get("current_question") or ""),
|
||||
"options": options,
|
||||
"user_message": str(sample.get("user_message") or ""),
|
||||
"case_status": str(focus.get("case_status") or sample.get("case_status") or "collecting_evidence"),
|
||||
}
|
||||
if variant in {"v1", "v2"}:
|
||||
state["previous_turn"] = previous_turn_payload(sample)
|
||||
if variant == "v2":
|
||||
state["previous_decision"] = previous_decision_payload(sample)
|
||||
return state
|
||||
|
||||
|
||||
def build_questions(sample: Mapping[str, Any]) -> dict[str, Any]:
|
||||
"""Three TypeSafe questions, same request, official fan-out."""
|
||||
def build_questions(sample: Mapping[str, Any], *, variant: str = "v0") -> dict[str, Any]:
|
||||
"""Three TypeSafe questions, same request, official fan-out. V2 adds one Noul."""
|
||||
if variant not in STATE_VARIANTS:
|
||||
raise ValueError(f"unknown state variant: {variant}")
|
||||
layer = str(sample.get("layer") or "none")
|
||||
focus = sample.get("focus") if isinstance(sample.get("focus"), dict) else {}
|
||||
options = list(focus.get("options") or [])
|
||||
return {
|
||||
questions = {
|
||||
"intent": {
|
||||
"type": "choice",
|
||||
"instructions": INTENT_INSTRUCTIONS,
|
||||
"instructions": intent_instructions(variant),
|
||||
"criteria": dict(INTENT_CRITERIA),
|
||||
},
|
||||
"answer_class": {
|
||||
@@ -234,16 +299,23 @@ def build_questions(sample: Mapping[str, Any]) -> dict[str, Any]:
|
||||
"criteria": dict(HAS_NEW_DATED_EVENT_CRITERIA),
|
||||
},
|
||||
}
|
||||
if variant == "v2":
|
||||
questions["continues_previous_turn"] = {
|
||||
"type": "noul",
|
||||
"instructions": CONTINUES_PREVIOUS_TURN_INSTRUCTIONS,
|
||||
"criteria": dict(CONTINUES_PREVIOUS_TURN_CRITERIA),
|
||||
}
|
||||
return questions
|
||||
|
||||
|
||||
def sdk_questions(sample: Mapping[str, Any]) -> dict[str, Any]:
|
||||
def sdk_questions(sample: Mapping[str, Any], *, variant: str = "v0") -> dict[str, Any]:
|
||||
"""SDK objects when typesafe_sdk is installed; JSON dicts otherwise."""
|
||||
payload = build_questions(sample)
|
||||
payload = build_questions(sample, variant=variant)
|
||||
try:
|
||||
from typesafe_sdk import Choice, Noul
|
||||
except ImportError:
|
||||
return payload
|
||||
return {
|
||||
built = {
|
||||
"intent": Choice(
|
||||
instructions=payload["intent"]["instructions"],
|
||||
criteria=payload["intent"]["criteria"],
|
||||
@@ -257,6 +329,13 @@ def sdk_questions(sample: Mapping[str, Any]) -> dict[str, Any]:
|
||||
criteria=payload["has_new_dated_event"]["criteria"],
|
||||
),
|
||||
}
|
||||
continues = payload.get("continues_previous_turn")
|
||||
if isinstance(continues, dict):
|
||||
built["continues_previous_turn"] = Noul(
|
||||
instructions=continues["instructions"],
|
||||
criteria=continues["criteria"],
|
||||
)
|
||||
return built
|
||||
|
||||
|
||||
def enforce_combo(
|
||||
@@ -287,7 +366,12 @@ def enforce_combo(
|
||||
}
|
||||
|
||||
|
||||
def parse_jev_answers(answers: Mapping[str, Any]) -> dict[str, Any]:
|
||||
def parse_jev_answers(
|
||||
answers: Mapping[str, Any],
|
||||
*,
|
||||
previous_decision: Mapping[str, Any] | None = None,
|
||||
apply_continue: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
intent_row = answers.get("intent") or {}
|
||||
class_row = answers.get("answer_class") or {}
|
||||
noul_row = answers.get("has_new_dated_event") or {}
|
||||
@@ -315,6 +399,13 @@ def parse_jev_answers(answers: Mapping[str, Any]) -> dict[str, Any]:
|
||||
"noul": noul_value,
|
||||
},
|
||||
}
|
||||
continues_row = answers.get("continues_previous_turn") or {}
|
||||
continues_value = continues_row.get("noul")
|
||||
parsed["raw"]["continues_previous_turn"] = {"noul": continues_value}
|
||||
parsed["continues_previous_turn"] = (
|
||||
float(continues_value) if isinstance(continues_value, (int, float)) else None
|
||||
)
|
||||
parsed["continued"] = False
|
||||
intent_conf = intent_row.get("confidence")
|
||||
class_conf = class_row.get("confidence")
|
||||
if parsed["intent"] == "answer_current_focus" and isinstance(intent_conf, (int, float)) and isinstance(class_conf, (int, float)):
|
||||
@@ -323,4 +414,23 @@ def parse_jev_answers(answers: Mapping[str, Any]) -> dict[str, Any]:
|
||||
parsed["confidence"] = float(intent_conf)
|
||||
else:
|
||||
parsed["confidence"] = None
|
||||
decision = previous_decision_payload({"previous_decision": previous_decision or {}})
|
||||
if (
|
||||
apply_continue
|
||||
and isinstance(continues_value, (int, float))
|
||||
and float(continues_value) >= CONTINUE_THRESHOLD
|
||||
and decision is not None
|
||||
):
|
||||
adopted = enforce_combo(
|
||||
decision["intent"],
|
||||
decision.get("answer_class"),
|
||||
parsed["has_new_dated_event"],
|
||||
)
|
||||
parsed["intent_before_continue"] = parsed["intent"]
|
||||
parsed["answer_class_before_continue"] = parsed["answer_class"]
|
||||
parsed["intent"] = adopted["intent"]
|
||||
parsed["answer_class"] = adopted["answer_class"]
|
||||
parsed["has_new_dated_event"] = adopted["has_new_dated_event"]
|
||||
parsed["continued"] = True
|
||||
parsed["confidence"] = float(continues_value)
|
||||
return parsed
|
||||
|
||||
@@ -15,7 +15,9 @@ sha256: `79d53a958b3b42ed18c79e3bbcff4b31deb22b65eedcd89b39326a7fefc9b0e8`
|
||||
```sql
|
||||
select
|
||||
t.id as turn_id,
|
||||
t.case_id,
|
||||
t.user_message,
|
||||
t.assistant_message,
|
||||
t.status as turn_status,
|
||||
t.created_at,
|
||||
c.status as case_status,
|
||||
@@ -24,16 +26,23 @@ select
|
||||
f.status as focus_status
|
||||
from public.agentic_rectification_turns t
|
||||
join public.agentic_rectification_cases c on c.id = t.case_id
|
||||
left join public.agentic_rectification_conversation_focuses f
|
||||
on f.case_id = t.case_id
|
||||
and f.status = 'active'
|
||||
left join lateral (
|
||||
select *
|
||||
from public.agentic_rectification_conversation_focuses f
|
||||
where f.case_id = t.case_id
|
||||
and f.asked_at <= t.created_at
|
||||
order by f.asked_at desc
|
||||
limit 1
|
||||
) f on true
|
||||
where t.user_message is not null
|
||||
and length(btrim(t.user_message)) > 0
|
||||
order by t.created_at desc
|
||||
limit 500;
|
||||
order by t.case_id, t.created_at;
|
||||
```
|
||||
|
||||
同一条 SQL 也写在 `scripts/research/jev_intent_source_b_v2.py` 的 `EXTRACT_SQL`。本地脚本按 `(case_id, created_at)` 给每条填 `previous_turn = {assistant_message, user_message}`;同一案件的第一轮,以及没有 `case_id` 的行,都是 `null`。没有 `case_id` 的行不会互相串成上一轮。
|
||||
|
||||
- 本地路径(worktree 外,已 gitignore):`G:/Ferti/Jyotisha/.cache/jev_intent/source_b.jsonl`
|
||||
- v2 路径(同样不提交):`G:/Ferti/Jyotisha/.cache/jev_intent/source_b_v2.jsonl`
|
||||
- 本机抽取条数:157(点选 17 / 采集 107 / 无焦点 33),gold 已全部人工标注,未用模型代标
|
||||
- 长度:P25=12 / 中位=21 / P75=28 / 最长=172
|
||||
- 标注耗时:约 40 分钟(`runtime_intent` 只作参考,85 条有反推 intent,不抄成 gold)
|
||||
|
||||
@@ -0,0 +1,203 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build source B v2 rows with the previous turn in the same case.
|
||||
|
||||
Offline. Writes only under the gitignored cache. Does not label gold with a
|
||||
model or with a keyword table. Old gold is copied by turn_id, or by a unique
|
||||
exact user_message when the 09-19 file has no turn_id.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping, Sequence
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
if str(ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
CACHE_DIR = Path(r"G:\Ferti\Jyotisha\.cache\jev_intent")
|
||||
LEGACY_PATH = CACHE_DIR / "source_b.jsonl"
|
||||
OUTPUT_PATH = CACHE_DIR / "source_b_v2.jsonl"
|
||||
|
||||
EXTRACT_SQL = """
|
||||
select
|
||||
t.id as turn_id,
|
||||
t.case_id,
|
||||
t.user_message,
|
||||
t.assistant_message,
|
||||
t.status as turn_status,
|
||||
t.created_at,
|
||||
c.status as case_status,
|
||||
f.question_id,
|
||||
f.expected_answer_schema,
|
||||
f.status as focus_status
|
||||
from public.agentic_rectification_turns t
|
||||
join public.agentic_rectification_cases c on c.id = t.case_id
|
||||
left join lateral (
|
||||
select *
|
||||
from public.agentic_rectification_conversation_focuses f
|
||||
where f.case_id = t.case_id
|
||||
and f.asked_at <= t.created_at
|
||||
order by f.asked_at desc
|
||||
limit 1
|
||||
) f on true
|
||||
where t.user_message is not null
|
||||
and length(btrim(t.user_message)) > 0
|
||||
order by t.case_id, t.created_at
|
||||
"""
|
||||
|
||||
|
||||
def load_jsonl(path: Path) -> list[dict[str, Any]]:
|
||||
if not path.is_file():
|
||||
return []
|
||||
return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line.strip()]
|
||||
|
||||
|
||||
def write_jsonl(path: Path, rows: Sequence[Mapping[str, Any]]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(
|
||||
"".join(json.dumps(row, ensure_ascii=False) + "\n" for row in rows),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def attach_previous_turns(rows: Sequence[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""First turn of a case, and any row without case_id, gets previous_turn null.
|
||||
|
||||
Rows that share a missing case_id are not chained: that would glue
|
||||
different people together.
|
||||
"""
|
||||
by_case: dict[str, list[dict[str, Any]]] = {}
|
||||
for row in rows:
|
||||
case_id = str(row.get("case_id") or "").strip()
|
||||
if not case_id:
|
||||
row["previous_turn"] = None
|
||||
row["previous_turn_source"] = "no_case"
|
||||
continue
|
||||
by_case.setdefault(case_id, []).append(row)
|
||||
for case_rows in by_case.values():
|
||||
case_rows.sort(key=lambda item: (
|
||||
str(item.get("created_at") or ""),
|
||||
str(item.get("turn_id") or item.get("id") or ""),
|
||||
))
|
||||
for index, row in enumerate(case_rows):
|
||||
if index == 0:
|
||||
row["previous_turn"] = None
|
||||
row["previous_turn_source"] = "case_first"
|
||||
continue
|
||||
prev = case_rows[index - 1]
|
||||
row["previous_turn"] = {
|
||||
"assistant_message": "" if prev.get("assistant_message") is None else str(prev.get("assistant_message")),
|
||||
"user_message": "" if prev.get("user_message") is None else str(prev.get("user_message")),
|
||||
}
|
||||
row["previous_turn_source"] = "prior_turn"
|
||||
return list(rows)
|
||||
|
||||
|
||||
def match_gold(rows: Sequence[dict[str, Any]], legacy: Sequence[Mapping[str, Any]]) -> dict[str, int]:
|
||||
by_turn: dict[str, Mapping[str, Any]] = {}
|
||||
by_message: dict[str, list[Mapping[str, Any]]] = {}
|
||||
for old in legacy:
|
||||
turn_id = str(old.get("turn_id") or "").strip()
|
||||
if turn_id:
|
||||
by_turn[turn_id] = old
|
||||
message = old.get("user_message")
|
||||
if isinstance(message, str) and message:
|
||||
by_message.setdefault(message, []).append(old)
|
||||
counts = {"turn_id": 0, "message_exact": 0, "unlabeled": 0, "message_ambiguous": 0}
|
||||
for row in rows:
|
||||
gold = None
|
||||
source = "unlabeled"
|
||||
legacy_id = None
|
||||
turn_id = str(row.get("turn_id") or "").strip()
|
||||
if turn_id and turn_id in by_turn:
|
||||
gold = by_turn[turn_id].get("gold")
|
||||
source = "turn_id"
|
||||
legacy_id = by_turn[turn_id].get("id")
|
||||
else:
|
||||
hits = by_message.get(str(row.get("user_message") or "")) or []
|
||||
if len(hits) == 1 and isinstance(hits[0].get("gold"), dict):
|
||||
gold = hits[0]["gold"]
|
||||
source = "message_exact"
|
||||
legacy_id = hits[0].get("id")
|
||||
elif len(hits) > 1:
|
||||
source = "message_ambiguous"
|
||||
if isinstance(gold, dict) and gold.get("intent"):
|
||||
row["gold"] = {
|
||||
"intent": gold.get("intent"),
|
||||
"answer_class": gold.get("answer_class"),
|
||||
"has_new_dated_event": gold.get("has_new_dated_event"),
|
||||
}
|
||||
row["gold_source"] = source
|
||||
if legacy_id:
|
||||
row["legacy_id"] = legacy_id
|
||||
else:
|
||||
row["gold_source"] = source
|
||||
row.pop("gold", None)
|
||||
counts[source] = counts.get(source, 0) + 1
|
||||
return counts
|
||||
|
||||
|
||||
def from_legacy(legacy: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""The handed-over 157 rows have no case_id. previous_turn stays null."""
|
||||
rows: list[dict[str, Any]] = []
|
||||
for old in legacy:
|
||||
if not isinstance(old.get("gold"), dict) or not old["gold"].get("intent"):
|
||||
continue
|
||||
rows.append({
|
||||
"id": old.get("id"),
|
||||
"source": "B",
|
||||
"layer": old.get("layer"),
|
||||
"user_message": old.get("user_message"),
|
||||
"assistant_message": None,
|
||||
"case_id": None,
|
||||
"turn_id": old.get("turn_id"),
|
||||
"created_at": old.get("created_at"),
|
||||
"focus": old.get("focus"),
|
||||
"focus_stale": old.get("focus_stale"),
|
||||
"runtime_intent": old.get("runtime_intent"),
|
||||
"gold": old.get("gold"),
|
||||
"gold_source": "legacy_file",
|
||||
"origin": old.get("origin"),
|
||||
})
|
||||
return attach_previous_turns(rows)
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--from-legacy", action="store_true")
|
||||
parser.add_argument("--raw", type=Path, default=None)
|
||||
parser.add_argument("--out", type=Path, default=OUTPUT_PATH)
|
||||
args = parser.parse_args(argv)
|
||||
legacy = load_jsonl(LEGACY_PATH)
|
||||
if args.from_legacy:
|
||||
rows = from_legacy(legacy)
|
||||
write_jsonl(args.out, rows)
|
||||
print(json.dumps({
|
||||
"out": str(args.out),
|
||||
"n": len(rows),
|
||||
"n_with_previous": sum(1 for row in rows if row.get("previous_turn")),
|
||||
"gold_source": "legacy_file",
|
||||
}, ensure_ascii=False))
|
||||
return 0
|
||||
if args.raw:
|
||||
raw_rows = load_jsonl(args.raw)
|
||||
rows = attach_previous_turns(raw_rows)
|
||||
counts = match_gold(rows, legacy)
|
||||
write_jsonl(args.out, rows)
|
||||
print(json.dumps({
|
||||
"out": str(args.out),
|
||||
"n": len(rows),
|
||||
"n_with_previous": sum(1 for row in rows if row.get("previous_turn")),
|
||||
"gold": counts,
|
||||
}, ensure_ascii=False))
|
||||
return 0
|
||||
print("pass --from-legacy or --raw; staging SQL is EXTRACT_SQL in this file", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -17,6 +17,7 @@ from scripts.research.jev_intent_corpus_build import ( # noqa: E402
|
||||
source_a_samples,
|
||||
)
|
||||
from scripts.research.jev_intent_questions import (
|
||||
CONTINUES_PREVIOUS_TURN_INSTRUCTIONS,
|
||||
CRITERION_MAP,
|
||||
INTENT_CRITERIA,
|
||||
LOST_SEMANTICS,
|
||||
@@ -25,6 +26,7 @@ from scripts.research.jev_intent_questions import (
|
||||
enforce_combo,
|
||||
parse_jev_answers,
|
||||
)
|
||||
from scripts.research.jev_intent_source_b_v2 import attach_previous_turns
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
@@ -70,6 +72,95 @@ class JevIntentQuestionsTests(unittest.TestCase):
|
||||
self.assertEqual(parsed["confidence"], 0.9)
|
||||
|
||||
|
||||
class JevIntentPreviousTurnTests(unittest.TestCase):
|
||||
def test_v1_v2_state_shape_keeps_v0_questions(self) -> None:
|
||||
sample = dict(source_a_samples()[0])
|
||||
sample["previous_turn"] = {"assistant_message": "上一问", "user_message": "上一句"}
|
||||
sample["previous_decision"] = {"intent": "unclear", "answer_class": None}
|
||||
v0 = build_state(sample, variant="v0")
|
||||
self.assertEqual(set(v0), {"current_question", "options", "user_message", "case_status"})
|
||||
v1 = build_state(sample, variant="v1")
|
||||
self.assertEqual(v1["previous_turn"], {"assistant_message": "上一问", "user_message": "上一句"})
|
||||
self.assertNotIn("previous_decision", v1)
|
||||
v2 = build_state(sample, variant="v2")
|
||||
self.assertEqual(v2["previous_decision"], {"intent": "unclear", "answer_class": None})
|
||||
v0_questions = build_questions(sample)
|
||||
v2_questions = build_questions(sample, variant="v2")
|
||||
self.assertNotIn("continues_previous_turn", v0_questions)
|
||||
self.assertEqual(set(v0_questions["intent"]["criteria"]), set(v2_questions["intent"]["criteria"]))
|
||||
self.assertEqual(v2_questions["continues_previous_turn"]["type"], "noul")
|
||||
self.assertEqual(
|
||||
v2_questions["continues_previous_turn"]["instructions"],
|
||||
CONTINUES_PREVIOUS_TURN_INSTRUCTIONS,
|
||||
)
|
||||
|
||||
def test_continues_rule_adopts_previous_intent_only_at_threshold(self) -> None:
|
||||
answers = {
|
||||
"intent": {"choice": "answer_current_focus", "confidence": 0.95, "probabilities": {}},
|
||||
"answer_class": {"choice": "yes", "confidence": 0.91, "probabilities": {}},
|
||||
"has_new_dated_event": {"noul": 0.1},
|
||||
"continues_previous_turn": {"noul": 0.9},
|
||||
}
|
||||
adopted = parse_jev_answers(
|
||||
answers,
|
||||
previous_decision={"intent": "unclear", "answer_class": None},
|
||||
apply_continue=True,
|
||||
)
|
||||
self.assertEqual(adopted["intent"], "unclear")
|
||||
self.assertIsNone(adopted["answer_class"])
|
||||
self.assertTrue(adopted["continued"])
|
||||
self.assertEqual(adopted["confidence"], 0.9)
|
||||
kept = parse_jev_answers(
|
||||
{**answers, "continues_previous_turn": {"noul": 0.89}},
|
||||
previous_decision={"intent": "stop_rectification", "answer_class": None},
|
||||
apply_continue=True,
|
||||
)
|
||||
self.assertEqual(kept["intent"], "answer_current_focus")
|
||||
self.assertEqual(kept["answer_class"], "yes")
|
||||
self.assertFalse(kept["continued"])
|
||||
focused = parse_jev_answers(
|
||||
answers,
|
||||
previous_decision={"intent": "answer_current_focus", "answer_class": "no"},
|
||||
apply_continue=True,
|
||||
)
|
||||
self.assertEqual(focused["intent"], "answer_current_focus")
|
||||
self.assertEqual(focused["answer_class"], "no")
|
||||
|
||||
def test_case_first_turn_previous_turn_is_null(self) -> None:
|
||||
rows = attach_previous_turns([
|
||||
{
|
||||
"id": "first",
|
||||
"case_id": "case-1",
|
||||
"created_at": "2026-09-01T00:00:00Z",
|
||||
"user_message": "第一句",
|
||||
"assistant_message": "第一答",
|
||||
},
|
||||
{
|
||||
"id": "second",
|
||||
"case_id": "case-1",
|
||||
"created_at": "2026-09-01T00:01:00Z",
|
||||
"user_message": "继续",
|
||||
"assistant_message": "第二答",
|
||||
},
|
||||
{
|
||||
"id": "orphan",
|
||||
"user_message": "没有案件",
|
||||
"assistant_message": "答",
|
||||
},
|
||||
])
|
||||
by_id = {row["id"]: row for row in rows}
|
||||
self.assertIsNone(by_id["first"]["previous_turn"])
|
||||
self.assertEqual(by_id["first"]["previous_turn_source"], "case_first")
|
||||
self.assertEqual(by_id["second"]["previous_turn"], {
|
||||
"assistant_message": "第一答",
|
||||
"user_message": "第一句",
|
||||
})
|
||||
self.assertIsNone(by_id["orphan"]["previous_turn"])
|
||||
state = build_state(by_id["first"], variant="v1")
|
||||
self.assertIsNone(state["previous_turn"])
|
||||
self.assertIn("user_message", state)
|
||||
|
||||
|
||||
class JevIntentCorpusTests(unittest.TestCase):
|
||||
def test_source_a_has_ten_labeled_rows(self) -> None:
|
||||
rows = source_a_samples()
|
||||
|
||||
Reference in New Issue
Block a user