From fbd6e4803627f8dc63a7257ce31ec4fd6fa685e6 Mon Sep 17 00:00:00 2001 From: Jesse_Chen Date: Thu, 3 Sep 2026 09:54:14 +0800 Subject: [PATCH] fix(report): do not fail a written report when progress projection throws Staging wrote four ready chapters then marked the job schema-invalid 34ms later with no summary telemetry. Keep chapter persistence and assemble going if onProgress or a non-lease heartbeat error fails. Co-authored-by: Cursor --- PROGRESS-report-writer-failure-20260902.md | 49 ++++++++++++++++++- docs/BUG_HISTORY.md | 2 +- .../src/lib/personal-report-generation.ts | 19 ++++--- .../src/lib/personal-report-worker-core.ts | 24 ++++++--- .../personal-report-generation-v2.test.ts | 25 +++++++++- 5 files changed, 101 insertions(+), 18 deletions(-) diff --git a/PROGRESS-report-writer-failure-20260902.md b/PROGRESS-report-writer-failure-20260902.md index 7ff8a28d..2aee90a5 100644 --- a/PROGRESS-report-writer-failure-20260902.md +++ b/PROGRESS-report-writer-failure-20260902.md @@ -146,9 +146,54 @@ Job 在写作过程中 `lease_expired` 两次（默认租约 120s，单章可超 无每章 `inputTokens`/`outputTokens`。Run A 墙钟约 10.4 分钟（含两次租约回收与 marriage/timing 的 section repair）。数据摆出，不裁决。 +### Run C · requestId `89c04491-81ca-457e-b09d-dc81c3af36b3` + +`GET /api/health` `.deployment.gitCommit` = `5c0bec0c9419e12e2c1fcab8e75fd35d535cb50e`。Web 容器 `2026-09-02T18:24:50Z`。未提升 main。 + +standard `personal_full`，主题 career / marriage / wealth / timing。started 01:31:16 UTC，终态 01:45:53 UTC（约 14.6 分钟）。reservation `released`。 + +| 章 | status | attempts | narrative 字数 | last_error_code | +|---|---|---:|---:|---| +| theme-career | ready | 1 | 170 | （空） | +| theme-marriage | ready | 2 | 1338 | （空） | +| theme-timing | ready | 1 | 926 | （空） | +| theme-wealth | ready | 1 | 745 | （空） | + +Telemetry（`[personal-report-agent]`，本轮 docker logs 可读）： + +| 调用 | outcome | elapsedMs | inputTokens | outputTokens | finishReason | repair | +|---|---|---:|---:|---:|---|---| +| career | resolved | 5721 | 5215 | 560 | stop | false | +| marriage | failed | 9430 | null | null | tripwire | false | +| marriage | failed | 9447 | null | null | tripwire | true | +| marriage | resolved | 13618 | 5076 | 1240 | stop | false | +| timing | resolved | 8052 | 4474 | 816 | stop | false | +| wealth | resolved | 7446 | 5228 | 759 | stop | false | +| summary | （无日志） | | | | | | + +无 `finishReason=length`。四章 `stop` 合计 input 19993 / output 3375。无 summary telemetry。 + +Job attempt 1 在 marriage 写作中心跳停更（heartbeat 与 lease 同步老化，最长观测 584s），`lease_expired` 后 attempt 2。四章随后都写成 ready。wealth 落库 `01:45:53.549`，整份失败 `01:45:53.583`（约 34ms）。进度停在 **57%**（marriage 完成后的 2/4），timing/wealth 的 onProgress 未落库。无 `[personal-report] generation_failed` 行 → 不是 `failSchema` 的摘要/装配返回，而是 `onProgress`/续租抛错被 worker 映射成 `report_schema_invalid`。 + +### Run C 回溯抽查（只记是否对得上 ref） + +| # | 章 | 叙事层（布尔） | 对回的 evidenceRef | +|---|---|---|---| +| 1 | 四章皆含 yoga 词 | true | wealth 另有 `ev-tech-yoga` | +| 2 | marriage / timing / wealth | 大运/主运/副运 true；career false | 四章皆有 `ev-tech-vimshottari` | +| 3 | marriage | yoga + 大运 | `ev-tech-d9` + `ev-tech-dk` | +| 4 | career | yoga；无大运用词 | `ev-tech-amk` + `ev-tech-vimshottari` | +| 5 | wealth | yoga + 大运 | `ev-tech-d2` + `ev-tech-d11` | + +未扫到强弱/shadbala 用词。 + +### 相对 2 倍线 + +四章 writer 墙钟合计约 35s 模型时间；整份 14.6 分钟含一次租约回收与 marriage tripwire。inputTokens 已测（上表）。数据摆出，不裁决。 + ### 任务 4 结论 -Writer 修正在章节层成立：四章有正文、career 一次通过、无截断码。整份 ready 仍被 120s 租约和最后一步 `report_schema_invalid` 挡住。租约、心跳续租与摘要 cap 已推 staging；公开 health 在跟进部署前仍指向 `eda37c15`。整份 ready 要等新 SHA 部署后再验一次。 +Writer 修正在章节层成立。整份 ready 被两件事挡住：Next.js instrumentation 里章节写作期间心跳定时器仍不续租；最后一章 `complete` 之后 `onProgress` 抛错被标成 schema 失败，摘要从未开始。跟进补丁：进度投影失败不得中止已落库章节的摘要/装配。 --- @@ -164,4 +209,4 @@ Writer 修正在章节层成立：四章有正文、career 一次通过、无截 未跑全量 `tests/*.test.ts`，不声称全套绿。未改 `.gitea/workflows/**`，未提升 main。 -任务 4 补丁（租约 600s、摘要 cap 3072、心跳保持 ref + 进度续租）：`tsc --noEmit` 通过；plan / generation-v2 / worker 聚焦测试通过。 +任务 4 补丁（租约 600s、摘要 cap 3072、心跳保持 ref + 进度续租 + onProgress 不中止装配）：`tsc --noEmit` 通过；generation-v2 / worker / writer-failure 聚焦测试通过。 diff --git a/docs/BUG_HISTORY.md b/docs/BUG_HISTORY.md index a23668b8..9687bee3 100644 --- a/docs/BUG_HISTORY.md +++ b/docs/BUG_HISTORY.md @@ -7643,7 +7643,7 @@ - 触发条件：章节 writer 输出未通过 `assertWriterOutput`（常见为 `evidenceRefs` 与 plan 集合不等），或输出在 token 预算内被截断；随后整份报告失败。 - 根因：三层叠加。旧错误分类把任何含 `evidence` 的消息打成 `section_evidence_insufficient`，`report_writer_evidence_refs_mismatch` 被误标；章节输出预算用英文「字符 ÷ 2」，standard 上限 1200 字中文只分到 1024 token；一章 blocked 或未捕获抛错可以中止整份，job 停在约 43%。失败详情只有聚合码，页面无法区分截断、引用不齐或未开始。staging 事故窗口的 writer telemetry 在容器 recreate 后丢失，本事故不能写成 `finishReason=length`。 - 修复：中文口径重算每章/摘要 `maxOutputTokens`，去掉 ÷2；length 修复要求压到字数下限且预算不低于首次。Prompt 要求逐字复制 `plan.evidenceRefs`；repair 只加失败类别。refs/identity 先于泛化 `evidence` 分类。单章失败后继续其余主题；`all_sections_blocked` 走统一失败日志。详情与列表聚合已有 section 行，展示可读摘要与错误码，不改表、不放宽 writer schema。 -- 验证：预算表锁定 concise 1218 / standard 2128 / deep 3072 / research 3072 / summary 3072。refs mismatch 分类为 `section_refs_mismatch` 且其余主题仍交付。repair 提示含类别词、不含内容。失败详情从 section 行汇总。`tsc --noEmit`、改动文件 ESLint、个人报告聚焦测试。staging `eda37c15` 上真实 standard 四章均 ready（career 1 次成功，无截断码），整份仍因 120s 租约与最后一步 `report_schema_invalid` 未落 `report_document`；租约改为 600s，心跳保持 ref，章节进度先续租。 +- 验证：预算表锁定 concise 1218 / standard 2128 / deep 3072 / research 3072 / summary 3072。refs mismatch 分类为 `section_refs_mismatch` 且其余主题仍交付。repair 提示含类别词、不含内容。失败详情从 section 行汇总。`tsc --noEmit`、改动文件 ESLint、个人报告聚焦测试。staging `5c0bec0c` 上真实 standard 四章均 ready（无 `finishReason=length`），整份仍因 `onProgress` 抛错被标成 `report_schema_invalid`，摘要未开始；进度投影失败改为不中止已落库章节。 - 防复发：输出预算不得再用字符 ÷ 2。`assertWriterOutput` 保持 id/theme 全等与 refs 集合相等。section 错误码不得把 refs mismatch 归进 evidence insufficient。一章失败不得中止其余 write 主题。用户可见失败必须有错误码级摘要，不得只展示 `report_schema_invalid`。日志与 PROGRESS 不得写入 prompt、bundle、模型原文或用户资料。 - 相关记录：BUG-352、BUG-451、BUG-486、BUG-489 - 复发自：BUG-451（分章后仍把写作失败压成整份 schema 码，截断预算与失败分类未按中文口径收口） diff --git a/frontend/src/lib/personal-report-generation.ts b/frontend/src/lib/personal-report-generation.ts index ba5545a2..f29176f6 100644 --- a/frontend/src/lib/personal-report-generation.ts +++ b/frontend/src/lib/personal-report-generation.ts @@ -3155,6 +3155,13 @@ async function generateSectionedPersonalReport( return failSchema("sectioned_dependencies_missing"); } const writePlans = plan.sections.filter((entry) => entry.kind === "thematic" && entry.disposition === "write"); + const emitProgress = async (phase: string, completed: number): Promise => { + try { + await deps.onProgress?.({ phase, completed, total: writePlans.length }); + } catch (error) { + rethrowIfAborted(error, deps.signal); + } + }; for (const entry of writePlans) { await sectionService.ensure({ userId: deps.userId, requestId: deps.requestId, sectionId: entry.id, maxAttempts: 2, @@ -3177,7 +3184,7 @@ async function generateSectionedPersonalReport( }); if (!blockedRow) throw new Error("section_block_failed"); blocked.push(blockedRow); - await deps.onProgress?.({ phase: `section:${entry.id}`, completed: ready.length + blocked.length, total: writePlans.length }); + await emitProgress(`section:${entry.id}`, ready.length + blocked.length); continue; } const outputBudget = sectionOutputTokenBudget(entry.targetCharacters.max); @@ -3204,7 +3211,7 @@ async function generateSectionedPersonalReport( const completed = await sectionService.complete({ userId: deps.userId, requestId: deps.requestId, sectionId: entry.id, payload: output }); if (!completed) throw new Error("section_complete_failed"); ready.push(completed); - await deps.onProgress?.({ phase: `section:${entry.id}`, completed: ready.length + blocked.length, total: writePlans.length }); + await emitProgress(`section:${entry.id}`, ready.length + blocked.length); break; } catch (error) { rethrowIfAborted(error, deps.signal); @@ -3214,7 +3221,7 @@ async function generateSectionedPersonalReport( }); if (!blockedRow) throw new Error("section_block_failed"); blocked.push(blockedRow); - await deps.onProgress?.({ phase: `section:${entry.id}`, completed: ready.length + blocked.length, total: writePlans.length }); + await emitProgress(`section:${entry.id}`, ready.length + blocked.length); break; } } @@ -3232,13 +3239,13 @@ async function generateSectionedPersonalReport( }); if (!blockedRow) throw new Error("section_block_failed"); blocked.push(blockedRow); - await deps.onProgress?.({ phase: `section:${entry.id}`, completed: ready.length + blocked.length, total: writePlans.length }); + await emitProgress(`section:${entry.id}`, ready.length + blocked.length); } } if (writePlans.length > 0 && ready.length === 0) { return failSchema("all_sections_blocked"); } - await deps.onProgress?.({ phase: "summary", completed: writePlans.length, total: writePlans.length }); + await emitProgress("summary", writePlans.length); let summary; try { summary = await deps.agent.generateSummary(ready.map((item) => ({ @@ -3260,7 +3267,7 @@ async function generateSectionedPersonalReport( missingEvidence: [sectionFailureReason(item.lastErrorCode)], conflictNotes: [], evidenceRefs: refs, }; }); - await deps.onProgress?.({ phase: "assemble", completed: writePlans.length, total: writePlans.length }); + await emitProgress("assemble", writePlans.length); try { const candidate = assembleReportDocumentV2({ reportId: deps.reportId, generatedAt: (deps.now ?? (() => new Date()))().toISOString(), diff --git a/frontend/src/lib/personal-report-worker-core.ts b/frontend/src/lib/personal-report-worker-core.ts index 3a513d94..5020196c 100644 --- a/frontend/src/lib/personal-report-worker-core.ts +++ b/frontend/src/lib/personal-report-worker-core.ts @@ -338,14 +338,22 @@ export function createPersonalReportWorker(deps: PersonalReportWorkerDeps) { signal: controller.signal, sectionService: deps.sectionService, onProgress: async (progress) => { - await refreshLease(); - if (heartbeatError !== null) throw heartbeatError; - const percent = progress.total > 0 - ? Math.min(89, 30 + Math.floor((progress.completed / progress.total) * 55)) - : 30; - await deps.jobs.updateProgress({ - jobId: job.id, leaseToken: job.leaseToken!, phase: progress.phase, percent, - }); + try { + await refreshLease(); + if (heartbeatError !== null) throw heartbeatError; + const percent = progress.total > 0 + ? Math.min(89, 30 + Math.floor((progress.completed / progress.total) * 55)) + : 30; + await deps.jobs.updateProgress({ + jobId: job.id, leaseToken: job.leaseToken!, phase: progress.phase, percent, + }); + } catch (error) { + if (isLeaseLoss(error) || heartbeatError !== null) { + heartbeatError = heartbeatError ?? error; + controller.abort(error); + throw error; + } + } }, }); await heartbeatChain; diff --git a/frontend/tests/personal-report-generation-v2.test.ts b/frontend/tests/personal-report-generation-v2.test.ts index fb2c33cf..39b7f036 100644 --- a/frontend/tests/personal-report-generation-v2.test.ts +++ b/frontend/tests/personal-report-generation-v2.test.ts @@ -426,8 +426,14 @@ test("production writer and worker keep the same signal on initial and repair mo new URL("../src/lib/personal-report-worker-core.ts", import.meta.url), "utf8", ); - assert.match(workerCore, /onProgress: async \(progress\) => \{\s*await refreshLease\(\)/); + assert.match(workerCore, /onProgress: async \(progress\) => \{\s*try \{\s*await refreshLease\(\)/); assert.doesNotMatch(workerCore, /timerUnref\(heartbeatTimer\)/); + const generationSource = readFileSync( + new URL("../src/lib/personal-report-generation.ts", import.meta.url), + "utf8", + ); + assert.match(generationSource, /const emitProgress = async \(phase: string, completed: number\)/); + assert.match(generationSource, /rethrowIfAborted\(error, deps\.signal\);/); }); test("an all-blocked plan still produces a valid v2 document with honest disclosures and no invented thematic section", async () => { @@ -828,6 +834,23 @@ test("sectioned generation makes one filtered call per write theme, then summary assert.match(document.executiveSummary.summary, /事业与方向/); }); +test("sectioned generation still assembles when onProgress throws after a persisted section", async () => { + const bundle = makeBundle({ + themes: fullThemes.slice(0, 3), + charts: [chart("D1"), chart("D2", 1), chart("D9", 2), chart("D10", 3), chart("D11", 4)], + }); + const result = await runSectioned( + bundle, + sectionedAgent({}), + inMemorySectionService(), + async () => { + throw new Error("progress_projection_failed"); + }, + ); + const document = readyV2(result); + assert.deepEqual(document.thematicNarrative.map((section) => section.theme), ["career", "marriage", "wealth"]); +}); + test("sectioned resume skips ready sections after an interruption", async () => { const bundle = makeBundle({ themes: fullThemes, charts: [chart("D1"), chart("D2", 1), chart("D9", 2), chart("D10", 3), chart("D11", 4), chart("D24", 5)] }); const service = inMemorySectionService();