research(rectification): holdout v5 — drop current-year events (harness cutoff 2026-09-14), index pages
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017eEAG8HD3mm8gsKXgk8uU8
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
57118b2697
commit
8ceb94da02
@@ -122,3 +122,4 @@ Do not open new product surfaces before at least one of these four lanes is clos
|
||||
- 两个已被证伪的假设:加权重能拉开候选;簇合并把候选并没了(合并从未触发)。
|
||||
- 未解决的只剩「区间内部并列」,目前唯一被证明有效的信息源是用户提供的带年月新经历。
|
||||
- 2026-09-29 打分方法研究(脚手架,判定待 v5):`<repo>/docs/research/rectification_scoring_research_2026_09_29.md`——似然比校准权重 / 缺席证据 / 精度追问三条流水线已在 v4 上跑通,所有判定 `pending_v5`;任务书 `docs/tasks/TASK-rectification-scoring-research-20260929.md`,前置 `TASK-rectification-holdout-expansion-20260929.md`。
|
||||
- 2026-09-29 起评价集换 **v5**(77 例公开 AA,`references/real_case_calibration/minute_rectification_holdout_v5.json`):协议 `holdout_v5_protocol_2026_09_29.md`、构建与基线成绩单 `holdout_v5_build_2026_09_29.md`。后续打分判定以 v5 为准,v4 数字只作历史对照。
|
||||
|
||||
@@ -95,6 +95,12 @@
|
||||
- 新增答错容错:答错 1 题,真值在区间 98–100%;答错 2 题,±30 / ±60 上 7–10% 的回放会挤出真值(两道反答 = 8 分 = 淘汰线)。
|
||||
- 全文见 `docs/research/rectification_offline_research_2026_09_26.md`。
|
||||
|
||||
## 8. v5 已建,后续判定以 v5 为准(2026-09-29)
|
||||
|
||||
- 本页所有「过门 / 不过门」都在 20 例上判,1 例 = 5 个百分点,判定在噪声线上(任务书 `TASK-rectification-holdout-expansion-20260929.md`)。
|
||||
- 开放评价集已扩到 **v5:77 例**(v4 的 20 例原样继承 + 57 例新增公开 Rodden-AA),每例 ≥7 件带引文的事件、≥4 领域;协议见 `holdout_v5_protocol_2026_09_29.md`,构建说明与基线成绩单见 `holdout_v5_build_2026_09_29.md`。
|
||||
- v5 中 v4 子集的基线数字与本页 §1 逐格一致(复现表在构建说明里)。**此后任何打分方案的判定必须在 v5 上做**;`TASK-rectification-scoring-research-20260929.md` 的 R-A / R-B / R-C 以 v5 为准。
|
||||
|
||||
## 9. 后续(2026-09-29 打分方法研究单,脚手架阶段)
|
||||
|
||||
- §1.3「加权重类改法全部不过门」说的是**人拍权重**。2026-09-29 研究单换了路线:每条规则的权重由数据估(似然比,leave-one-case-out),允许负权重(只奖不罚是结构性问题),KP / Pranapada / D60 作为特征进入由数据定权重;另测「缺席证据」与「对已说事件追问月份」。
|
||||
|
||||
@@ -255,7 +255,7 @@
|
||||
| `TASK-self-edit-avatar-menu-20260928.md` | `PROGRESS-self-edit-avatar-menu-20260928.md` | **本人可编辑 + 各页头像菜单 + 报告页去说明**:`/people` 本人可编辑出生资料(BUG-1081,`ab6c55f8` 漏掉);次级页头像弹同一账户菜单、弹窗项跳 `/?account=`;我的报告删顶部说明与统计 | 已实现待验收(分支 `codex/self-edit-avatar-menu-20260928`,未推送) | BUG-1081 |
|
||||
| `TASK-mobile-chart-and-confirmed-edit-20260929.md` | `PROGRESS-mobile-chart-confirmed-edit-20260929.md` | **手机星盘 + 确认时间可改 + 天空定格**:iPhone 星盘被宽表撑出屏幕(BUG-1083);confirmed 状态改声明字段也重置排盘时间(推翻 BUG-264 约定);「那一刻的天空」改为重放汇聚→定格→右上角分享;去掉报告星图封面 | 已验收(Claude 2026-09-29),已推 staging | BUG-1083 |
|
||||
| `TASK-rectification-futile-collect-stop-20260929.md` | `PROGRESS-rectification-futile-collect-stop-20260929.md` | **生时校正停掉无效补经历循环(止血)**:打字经历不收窄(BUG-560 后果),流程却一路索要,真机 22 件整窗不动;采集只为开闸、门开后只问点选卡问完即出卡;交付正文去吻合率;多段时旁白误报「范围没变」;交付轮带采集题 | 已验收(Claude 2026-09-29 合并验收:全量前端 4302/24 与基线同 24 条环境失败、tsc 0、lint 0 error、`/` Static、gzip +0.16%、快速门 pytest 1001 passed、两份回放复跑一致),待部署核对 | BUG-1084~1087 |
|
||||
| `TASK-rectification-holdout-expansion-20260929.md` | — | **开放评价集扩到 ≥60 例(v5)**:所有打分判定都在 20 例上做,1 例 = 5pp,KP / 精度闸 / V1n 的 no_benefit 都只差 1–3 例。只用 Rodden AA 公开名人,事件人工核对出处,分层(年代 / 纬度 / 南半球 / 跨午夜 / UTC+8),v4 不动;基线成绩单 v4 子集须与已发布数字逐格一致。研究单的判定以 v5 为准 | 待领取 | — |
|
||||
| `TASK-rectification-holdout-expansion-20260929.md` | `PROGRESS-rectification-holdout-expansion-20260929.md` | **开放评价集扩到 ≥60 例(v5)**:所有打分判定都在 20 例上做,1 例 = 5pp,KP / 精度闸 / V1n 的 no_benefit 都只差 1–3 例。只用 Rodden AA 公开名人,事件人工核对出处,分层(年代 / 纬度 / 南半球 / 跨午夜 / UTC+8),v4 不动;基线成绩单 v4 子集须与已发布数字逐格一致。研究单的判定以 v5 为准 | 已实现待验收(v5 77 例:v4 20 + 新增 57;UTC+8 配额 5/6 未达,见 PROGRESS) | 分支 `codex/rectification-holdout-expansion-20260929`(BUG-1090) |
|
||||
| `TASK-rectification-scoring-research-20260929.md` | `PROGRESS-rectification-scoring-research-20260929.md` | **打分方法研究(离线)**:R-A 似然比校准权重(leave-one-case-out,KP / Pranapada / D60 作为特征由数据定权重、允许负权重)、R-B 缺席证据(口述时间线覆盖时段内的空白年扣分,必测漏说稳健性)、R-C 精度追问可行性(对已说事件追问月份,算在定向追问 2 条额度内)。判定必须在 v5 上做;有收益才立实现单并按 ERR-110 重冻结 | 脚手架已验收(Claude 09-29:对账 60 case-radius 0 差、JSON 复跑逐字节一致、冻结文件零改动、117 测试全绿;已合入 staging);R-A / R-B / R-C 判定待 v5 | 分支 `codex/rectification-scoring-research-20260929`(BUG-1091 investigating) |
|
||||
| `TASK-rectification-typed-event-scoring-research-20260929.md` | `PROGRESS-rectification-typed-event-research-20260929.md` | **打字经历按选择题规则计分(离线研究,不上线)**:计分通道不对称 + 已入账年份挡题;R0 学业质量题措辞 / 年精度显示成 1 月(冻结文件,需重新冻结) | 已验收(Claude 2026-09-29 合并验收:全量前端 4302/24 与基线同 24 条环境失败、tsc 0、lint 0 error、`/` Static、gzip +0.16%、快速门 pytest 1001 passed、两份回放复跑一致),待部署核对 | BUG-1088、1089 |
|
||||
| `TASK-report-reader-polish-20260929.md` | `PROGRESS-report-reader-polish-20260929.md` | **报告页打磨**:生成入口挪进页面主体(删标题栏按钮)、详情页到底部按钮(懒渲染一次到底)、导出按钮带文字、目录一级/二级分层、去掉「字段」与 RL/NL 缩写表头、状态列同义重复去重、报告表格淡底色、分块导出显示真实文件大小 | Claude 直接执行并自验(tsc 0、lint 0 error、全量 fail 与基线同 24 条、`/` Static、gzip +7 B、快速门 Python 1000 passed),已部署 staging `d7772011`,真机欠 | BUG-1092~1094 |
|
||||
|
||||
@@ -31,16 +31,16 @@
|
||||
"summary": {
|
||||
"cases": 77,
|
||||
"inherited_from_v4": 20,
|
||||
"events_total": 966,
|
||||
"events_total": 964,
|
||||
"events_by_precision": {
|
||||
"day": 244,
|
||||
"month": 143,
|
||||
"year": 579
|
||||
"year": 577
|
||||
},
|
||||
"events_by_domain": {
|
||||
"career": 321,
|
||||
"career": 320,
|
||||
"education": 117,
|
||||
"family": 131,
|
||||
"family": 130,
|
||||
"finance": 35,
|
||||
"health_pressure": 97,
|
||||
"relationship": 193,
|
||||
@@ -12133,19 +12133,6 @@
|
||||
"quote": "He joined the MLS club Inter Miami in 2023 and led them to their first MLS Cup victory in 2025, while also winning the league MVP award in 2024 and 2025.",
|
||||
"independent_of_birth_source": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "lionel_messi_1987_family_2026",
|
||||
"domain": "family",
|
||||
"date": "2026",
|
||||
"precision": "year",
|
||||
"description": "Father Jorge died.",
|
||||
"source": {
|
||||
"publisher": "Wikipedia",
|
||||
"url": "https://en.wikipedia.org/wiki/Lionel_Messi",
|
||||
"quote": "His father Jorge was his agent and manager from 2000 until 2026, when he died at the age of 68.",
|
||||
"independent_of_birth_source": true
|
||||
}
|
||||
}
|
||||
],
|
||||
"candidate_radii_minutes": [
|
||||
@@ -17033,19 +17020,6 @@
|
||||
"quote": "He resigned in 2018, but returned to the club in 2019, and won another La Liga title and a Supercopa de España, before leaving again in 2021.",
|
||||
"independent_of_birth_source": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "zinedine_zidane_1972_career_2026",
|
||||
"domain": "career",
|
||||
"date": "2026",
|
||||
"precision": "year",
|
||||
"description": "Became head coach of the France national team.",
|
||||
"source": {
|
||||
"publisher": "Wikipedia",
|
||||
"url": "https://en.wikipedia.org/wiki/Zinedine_Zidane",
|
||||
"quote": "In 2026, he returned to coaching, succeeding Didier Deschamps as the new head coach of the France national team. == Early life and career == Zinedine Yazid Zidane was born on 23 June 1972 in La Castellane, a neighbourhood in the 16th arrondissement of Marseille, in Southern France.",
|
||||
"independent_of_birth_source": true
|
||||
}
|
||||
}
|
||||
],
|
||||
"candidate_radii_minutes": [
|
||||
|
||||
@@ -7227,19 +7227,6 @@
|
||||
"quote": "He joined the MLS club Inter Miami in 2023 and led them to their first MLS Cup victory in 2025, while also winning the league MVP award in 2024 and 2025.",
|
||||
"independent_of_birth_source": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "lionel_messi_1987_family_2026",
|
||||
"domain": "family",
|
||||
"date": "2026",
|
||||
"precision": "year",
|
||||
"description": "Father Jorge died.",
|
||||
"source": {
|
||||
"publisher": "Wikipedia",
|
||||
"url": "https://en.wikipedia.org/wiki/Lionel_Messi",
|
||||
"quote": "His father Jorge was his agent and manager from 2000 until 2026, when he died at the age of 68.",
|
||||
"independent_of_birth_source": true
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -10967,19 +10954,6 @@
|
||||
"quote": "He resigned in 2018, but returned to the club in 2019, and won another La Liga title and a Supercopa de España, before leaving again in 2021.",
|
||||
"independent_of_birth_source": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "zinedine_zidane_1972_career_2026",
|
||||
"domain": "career",
|
||||
"date": "2026",
|
||||
"precision": "year",
|
||||
"description": "Became head coach of the France national team.",
|
||||
"source": {
|
||||
"publisher": "Wikipedia",
|
||||
"url": "https://en.wikipedia.org/wiki/Zinedine_Zidane",
|
||||
"quote": "In 2026, he returned to coaching, succeeding Didier Deschamps as the new head coach of the France national team. == Early life and career == Zinedine Yazid Zidane was born on 23 June 1972 in La Castellane, a neighbourhood in the 16th arrondissement of Marseille, in Southern France.",
|
||||
"independent_of_birth_source": true
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -33,6 +33,7 @@ MIN_DOMAINS = 4
|
||||
MIN_CASES = 60
|
||||
DOMAINS = {"education", "relocation", "relationship", "career", "finance", "health_pressure", "family", "appearance", "occupation"}
|
||||
PRECISION_FORMAT = {"year": 4, "month": 7, "day": 10}
|
||||
EVENT_CUTOFF = "2026-09-14" # TODAY of minute_resolution_sweep / cluster_width_probe
|
||||
|
||||
|
||||
def load(path: Path) -> Any:
|
||||
@@ -71,6 +72,10 @@ def validate_event(case_id: str, event: dict[str, Any], problems: list[str]) ->
|
||||
date = str(event.get("date", ""))
|
||||
if precision not in PRECISION_FORMAT or len(date) != PRECISION_FORMAT[precision]:
|
||||
problems.append(f"{case_id}: date {date!r} does not match precision {precision!r}")
|
||||
# The replay harnesses evaluate at a fixed date (2026-09-14); the scoring contract
|
||||
# rejects events after "today", and a year-precision event resolves inside its year.
|
||||
if date[:4] >= EVENT_CUTOFF[:4] and (precision == "year" or date > EVENT_CUTOFF):
|
||||
problems.append(f"{case_id}: event {event.get('id')} dated {date} is not before the harness cutoff {EVENT_CUTOFF}")
|
||||
source = event.get("source") or {}
|
||||
if not source.get("url") or not source.get("independent_of_birth_source"):
|
||||
problems.append(f"{case_id}: event {event.get('id')} lacks an independent source url")
|
||||
|
||||
Reference in New Issue
Block a user