feat(research): consult biography backtest harness and pre-change baseline
Independent Staging Quality Gate / validate (push) Successful in 16m20s
Independent Staging Quality Gate / publish (push) Successful in 3m54s

TASK-consult-no-presupposition-and-backtest-20261001 T4 infrastructure only
(T1-T3 untouched). New files only, so it merges cleanly with the parallel
evidence-card brief.

- capture_consult_biography_backtest_golden.py: same handler/body/trim as the
  evidence-card golden; nine public AA charts x parents/marriage/health/career.
- consult-biography-backtest-golden.json: real engine output, byte-reproducible.
- consult_biography_backtest_rubric.json: facts with sources, must_not,
  expected_signals per figure x domain.
- consult-biography-backtest.mts: runs the product's real agent, tools, card,
  methodology, user-turn shape and streamAgentResponse; deterministic checks only.
- Baseline on 9b937c4a: 39/72 severe biography conflicts; all five audit cases
  reproduce in both runs.

BUG-1170 is registered when brief 3 lands.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4f2nya58RoRu4yEmJgRGE
This commit is contained in:
Jesse_Chen
2026-10-02 00:10:06 +08:00
co-authored by Claude Opus 5.5
parent e5199106a7
commit 691ee440e3
6 changed files with 2011 additions and 0 deletions
@@ -0,0 +1,41 @@
# PROGRESS · 名人生平回测基础设施与基线(2026-10-01)
- 任务书:`docs/tasks/TASK-consult-no-presupposition-and-backtest-20261001.md`,本轮只做 **T4 的基础设施 + 改动前基线**;T1~T3(形状、示例、追问轮)不在本轮。
- 执行方式:产品负责人要求子代理直接执行。
- 基线:`origin/staging` `9b937c4a`
- 分支 / worktree:`codex/consult-biography-backtest-20261001` / `.worktrees/consult-biography-backtest-20261001`(未推送)
- 并行:另一会话在改证据卡(`TASK-consult-card-affliction-data-20261001`)。本轮**只新增文件、不改任何已有文件**,两边可以干净合并;`docs/tasks/README.md` 索引也没动,由第三单执行方登记。
## 交付
| 文件 | 内容 |
| --- | --- |
| `scripts/research/capture_consult_biography_backtest_golden.py` | 复用 `capture_consult_evidence_card_golden.py` 的 `trim()` 与 `consult_evidence_card_run` 的 handler、请求体;9 位 × 4 条引擎路由(parents 走 family);路由间相同的 chart 字段存一份 `shared_chart`,不同的(今天是 `modules.yogas`、`modules.chara_dasha`)留在各路由下。`PYTHONHASHSEED=0 JYOTISH_API_CHART_CACHE_TTL_SECONDS=0` 下两次输出逐字节相同 |
| `frontend/tests/fixtures/consult-biography-backtest-golden.json` | 真实引擎输出,2.87 MB;不并入现有合同 golden |
| `docs/research/consult_biography_backtest_rubric.json` | 9 位 × 父母 / 婚姻 / 健康 / 事业:生平事实(带来源)、`must_not`(含原文筛网)、`expected_signals`(引用卡字段)、评分口径 |
| `frontend/scripts/research/consult-biography-backtest.mts` | 用产品真实模块组装(见测试文档「与真实路由的差异」);`--dump-cards` / `--recheck`;确定性检查只做禁句、性别未知代词、must_not 原文 |
| `docs/testing/consult-affliction-backtest-20261001.md` | 名单、保真度缺口、基线表与合计、代表句、成本 |
## 基线结论(详见测试文档)
72 份里严重冲突 39 份;排查记录的 5 份严重冲突 10 / 10 次复现;新增 4 位严重冲突 20 / 32(通过线 ≤ 1);对照组误报 1 份(轻)。新发现:事业题 16 / 18 份把人写成「专业人 / 顾问 / 幕后」,与盘无关,第三单不覆盖,建议另开单。
## 门禁
| 项 | 结果 |
| --- | --- |
| `tsc --noEmit`(`tsconfig.json` 的 include 含 `**/*.mts`,脚本在范围内) | 0 错 |
| `eslint`(`npm run lint` 的配置覆盖 `.mts`) | 0 error |
| 既有测试 | 未改任何测试文件 |
| `npm test`(Node 22) | 4868 项,pass 4803,fail 24,cancelled 0;24 项全是需要 Docker / PostgreSQL / 部署环境的套件(`startPostgresFixture` 等),本机无 Docker,属已知环境缺口,与本轮新增文件无关 |
| Python 快速门 `run_quality_gate.py --profile quick`(本机无 `.venv`,用系统 python3) | pytest 1036 passed / 1 skipped;门禁内的 `npm test` 被系统 Node 20 跑,70 项因 Node 20 不支持模块 mock 失败(环境),用 Node 22 单跑结果见上一行 |
| `tests/test_repo_privacy_markers.py` | 通过 |
## BUG 编号
本轮是基础设施,不登记 BUG。BUG-1170(T4 名人生平回测)在第三单(`TASK-consult-no-presupposition-and-backtest-20261001`)合入时登记,引用本进度记录与测试文档的基线。
## 环境备注
- 模型 key 只从环境变量读,没有写入任何文件、日志或提交。
- 模型输出在 `scratch/consult-biography-backtest/baseline-20261001/`(gitignore),未提交。