feat(consult): add an answer-step reasoning effort switch
The switch is off unless CONSULT_ANSWER_REASONING_EFFORT is set, so today's request body stays the same. A set value is sent as top-level reasoning_effort only on the answer step. The real A/B did not run: there is no model key. Request-body probe, the named frontend tests (114 pass, 0 fail), tsc, and lint (0 errors) were run. Co-Authored-By: Grok (agent loop) Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
jesse-ux
co-authored by
Grok (agent loop)
Cursor
Grok (agent loop)
Cursor
parent
9274491681
commit
ab7f66be14
@@ -1,5 +1,10 @@
|
||||
# BLOCKED
|
||||
|
||||
## TASK-answer-effort-ab-20261013：答题步推理强度的真实 A/B（2026-10-15）
|
||||
|
||||
- **A/B 未跑：缺模型 key。** 本机没有 `DS_KEY` 或 `BACKTEST_MODEL_KEY`。供应商是否接受 `low` / `medium` / `high` / `max`、推理 token 和耗时有没有下降，都没测。不得把请求体拦截写成「A/B 通过」。
|
||||
- 开关默认不设，请求体与改前一致。默认值等有 key 的验收跑完再由产品拍板，本单不改部署。
|
||||
|
||||
## TASK-consult-prep-parallel-20261013：审核真实延迟、Postgres 中位数、多领域镜像（2026-10-14）
|
||||
|
||||
- **阿里云审核**：本机没有审核密钥。1500 毫秒挡首帧的结论来自测试里的假等待，不是真实供应商。开关 `CONSULT_PREP_PARALLEL` 保持关闭。
|
||||
|
||||
@@ -36131,3 +36131,19 @@
|
||||
- 相关记录：BUG-203（锁文件，不是这一条）、BUG-1063（人物表列级读取权）、BUG-1348。
|
||||
- 复发自：无。
|
||||
- 修复版本：分支 `codex/release-path-production-20261014`。编号接在 BUG-1349 之后。
|
||||
|
||||
## BUG-1351 | 低强度推理设置写在嵌套字段里，供应商收不到
|
||||
|
||||
- 状态：investigating
|
||||
- 首次发现 / 最近更新：2026-10-13 / 2026-10-15
|
||||
- 来源：issue #65。任务书 `docs/tasks/TASK-answer-effort-ab-20261013.md`。请求体拦截见 `docs/testing/answer-effort-ab-20261013.md`。
|
||||
- 现象：代码里有一处把推理强度写成 `low`，但真正发出去的请求没有 `reasoning_effort` 这个字段。强度被包在 `thinking` 里面。
|
||||
- 触发条件：调用 `consultationSliceGenerationSettings`。产品代码里没有调用方，线上对话不受影响。
|
||||
- 根因：兼容通道只认供应商选项最外层的 `reasoningEffort`，并把它写成 `reasoning_effort`。嵌在 `thinking` 里的同名键会原样送出去，供应商不把它当强度。
|
||||
- 修复：本单不改这处形状，也不改它的两条断言。答题步的新开关写在最外层。要不要删掉或改掉这处旧形状，等产品拍板。
|
||||
- 验证：`frontend/scripts/research/answer-effort-probe.mts --dry-run`。这处设置的请求体是 `thinking.reasoningEffort = "low"`，`reasoning_effort` 为空。新开关设成 `low` 时，答题步请求体有 `reasoning_effort: "low"`。
|
||||
- 防复发：新测试锁住「开关不设时请求体与改前一致」，并锁住新强度出现在最外层。
|
||||
- 相关记录：BUG-148、BUG-1142。
|
||||
- 复发自：无。
|
||||
- 修复版本：未修。分支 `codex/answer-effort-ab-20261013` 只记录。编号接在 BUG-1350 之后。
|
||||
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
# PROGRESS · 答题步推理强度开关（2026-10-15）
|
||||
|
||||
- 分支：`codex/answer-effort-ab-20261013`
|
||||
- 实际基线：`origin/staging` `9274491681f6fcacbc87e3797441d2c5221de874`
|
||||
- 前置 `consult-prep-parallel` 已在 `origin/staging` 上（`git merge-base --is-ancestor` 退出码 0）
|
||||
- 默认行为不变。没有模型钥匙，A/B 未跑。
|
||||
|
||||
## 做了什么
|
||||
|
||||
1. 新环境变量 `CONSULT_ANSWER_REASONING_EFFORT`。`low` / `medium` / `high` / `max` 才生效。空着或没设，请求体与改前一致。非法值 `console.warn` 一次，然后当作没设。
|
||||
2. `agentGenerationSettings` 增加 `providerReasoningEffort`。它写在供应商选项的最外层，兼容通道据此发出 `reasoning_effort`。原来的 `reasoningEffort` 仍嵌在 `thinking` 里，切片那两条断言不用改。
|
||||
3. 答题步才带这个字段。判据与答案时钟相同：计算结果已经在手（`answerReady`），或者公开日历从第一步就是答题（`answerFromStart`）。主循环第 0 步计算结果还没到，不带。重试是新的一轮，步号从 0 重新计，但计算结果已经在手，所以重试的第 0 步也带。续写仍是关闭思考，不带这个字段。
|
||||
4. Mastra 的 `prepareStep` 会按步合并 `providerOptions`（`mergeObjects`）。答题步返回的强度会并进这一步，不会整轮都带。本单按步生效，没有改成整轮。
|
||||
5. 生平回测脚本增加 `--cases figure:domain,…` 和 `--efforts unset,low,medium`。每行记下端到端秒数、第 0 / 1 步秒数和推理 token、输出 token、结束原因、是否空答、是否截断、正文字数。摘要记下当次强度。
|
||||
|
||||
## 自测
|
||||
|
||||
| 检查 | 命令 | 结果 |
|
||||
| --- | --- | --- |
|
||||
| 请求体拦截 | `node node_modules/tsx/dist/cli.mjs scripts/research/answer-effort-probe.mts --dry-run` | 退出码 0。数字见 `docs/testing/answer-effort-ab-20261013.md` |
|
||||
| 本单测试与任务书点名的四份 | `node node_modules/tsx/dist/cli.mjs --test` 后接 `answer-effort-switch-20261013`、`consultation-agentic-runtime`、`consultation-workflow-contract`、`consult-answer-clock-20261001`、`consult-latency-timing-20261005` | 114 pass，0 fail，0 cancelled |
|
||||
| 类型检查 | 在 `frontend` 下 `.\node_modules\.bin\tsc --noEmit -p .` | 退出码 0 |
|
||||
| lint | 在 `frontend` 下 `npm run lint` | 退出码 0，0 error，123 条原有 warning。探针里多出来的 1 条未使用变量已去掉，改过的文件再跑 eslint 是 0 |
|
||||
|
||||
开关不设时，两步请求体与改前保存的整份 JSON 逐字节一致。设 `low` 时只有答题步出现 `reasoning_effort: "low"`，第 0 步没有；续写仍是 `thinking: disabled` 且没有 `reasoning_effort`。
|
||||
|
||||
## 改既有断言
|
||||
|
||||
| 原值 | 新值 | 原因 |
|
||||
| --- | --- | --- |
|
||||
| `await streamWithOverflowRetry(agent)` | `await streamWithOverflowRetry(agent, generalStreamOptions)` | 公开日历从第一步就是答题。不传选项就没有 `prepareStep`，强度加不上去。开关不设时这些选项不改请求体。答案时钟的断言还在。 |
|
||||
|
||||
切片的嵌套形状和它的两条断言没有改。
|
||||
|
||||
## 需拍板
|
||||
|
||||
- 默认强度。本单不改。有钥匙的验收按任务书 T3 出数之后，再决定设不设、设哪一档。部署默认写在哪里也留到那次。
|
||||
- `consultationSliceGenerationSettings` 删掉还是改成最外层。它没有调用方。本单不改。见 BUG-1351。
|
||||
- 若供应商不接受降档，对照组 D（答题步关思考）只供参考，采用与否另说。本机没跑。
|
||||
|
||||
## 环境缺口
|
||||
|
||||
A/B 未跑：缺模型 key。没有 `DS_KEY`，也没有 `BACKTEST_MODEL_KEY`。
|
||||
@@ -483,7 +483,7 @@
|
||||
| `TASK-frontend-dead-code-20261013.md` | `PROGRESS-frontend-dead-code-20261013.md` | 前端死代码 17 文件与无调用方接口删除，含未登录重计算接口（BUG-1342） | 待验收 | `codex/frontend-dead-code-20261013` |
|
||||
| `TASK-frontend-dead-code-fix-20261014.md` | `PROGRESS-frontend-dead-code-20261013.md` | **#63 第 1 次打回**：类名白名单去掉已删组件留下的条目 | 待验收 | `codex/frontend-dead-code-20261013` |
|
||||
| `TASK-api-chart-subprocess-gate-20261013.md` | — | 合盘不带 skip 串行排盘、报告无效子进程、名额轻重混用（只模拟）（BUG-1333～1334，A/B） | 待领取（排队：api-disk-cache-retention + report-reference-perf） | `codex/api-chart-subprocess-gate-20261013` |
|
||||
| `TASK-answer-effort-ab-20261013.md` | — | 答题步推理强度开关 + A/B（需模型 key） | 待领取（排队：consult-prep-parallel） | `codex/answer-effort-ab-20261013` |
|
||||
| `TASK-answer-effort-ab-20261013.md` | `PROGRESS-answer-effort-ab-20261013.md` | 答题步推理强度开关 + A/B（需模型 key） | 待验收 | `codex/answer-effort-ab-20261013` |
|
||||
| `TASK-consult-route-split-20261013.md` | — | consult POST 拆分第一阶段：开流前加载 + 计费结算 | 待领取（排队：consult-prep-parallel、answer-effort-ab） | `codex/consult-route-split-20261013` |
|
||||
| `TASK-pl9-render-split-20261013.md` | — | render_pl9_markdown 第一阶段：搬出无依赖内部函数 | 待领取（排队：api-chart-subprocess-gate） | `codex/pl9-render-split-20261013` |
|
||||
| `TASK-python-hygiene-20261013.md` | — | 免费层排队开关报反、静默吞异常、功能吉凶兜底合一、测试专用脚本归位（BUG-1340） | 待领取（排队：pl9-render-split） | `codex/python-hygiene-20261013` |
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
# 答题步推理强度（2026-10-15）
|
||||
|
||||
没有模型钥匙。下面只是本机拦截到的请求体，不是供应商的接受度，也不是耗时和质量。A/B 未跑。
|
||||
|
||||
命令：在 `frontend` 目录执行 `node node_modules/tsx/dist/cli.mjs scripts/research/answer-effort-probe.mts --dry-run`。假请求不离开这台机器。模型对象与产品相同：`providerId: deepseek`，`modelId: deepseek-flash`，带 `url` 和假的 `apiKey`。
|
||||
|
||||
## 请求体里实际有什么
|
||||
|
||||
| 形状 | thinking | reasoning_effort | max_tokens |
|
||||
| --- | --- | --- | --- |
|
||||
| 今天的答题设置，第 0 步和第 1 步 | `{ type: enabled }` | 无 | 24576 |
|
||||
| 旧的低强度设置（没有产品调用方） | `{ type: enabled, reasoningEffort: low }` | 无 | 10240 |
|
||||
| 新开关 `low` / `medium` / `high` / `max`（只在答题步） | `{ type: enabled }` | 与开关相同 | 24576 |
|
||||
| 续写 | `{ type: disabled }` | 无 | 16384 |
|
||||
|
||||
旧的低强度写在 `thinking` 里面，请求体没有 `reasoning_effort`。这证实了任务书 §2.3。记为 BUG-1351，状态 investigating。线上不用这处设置。
|
||||
|
||||
新开关写在供应商选项的最外层。兼容通道把它写成 `reasoning_effort`。开关不设时，两步请求体与改前逐字节一致，样本在 `frontend/tests/fixtures/answer-effort-switch-20261013.json`。
|
||||
|
||||
## 供应商接受度
|
||||
|
||||
未跑。没有 `DS_KEY` / `BACKTEST_MODEL_KEY`。`low`、`medium`、`high`、`max` 哪几档能用，要等验收时用临时钥匙跑探针的后半段。
|
||||
|
||||
## 固定 10 题
|
||||
|
||||
脚本已能按 `--cases` 跑下面这 10 题，并能按 `--efforts unset,low,medium` 把同一题的几档挨着跑。多档同时跑时并发降为 1，因为强度读的是进程里的环境变量，并行会串档。本机没有钥匙，这 10 题没有跑。
|
||||
|
||||
| # | 盘 | 领域 |
|
||||
| --- | --- | --- |
|
||||
| 1 | steve_jobs | career |
|
||||
| 2 | barack_obama | parents |
|
||||
| 3 | elizabeth_taylor | marriage |
|
||||
| 4 | marilyn_monroe | parents |
|
||||
| 5 | judy_garland | health |
|
||||
| 6 | edith_piaf | marriage |
|
||||
| 7 | frida_kahlo | health |
|
||||
| 8 | george_w_bush | career |
|
||||
| 9 | zinedine_zidane | career |
|
||||
| 10 | marilyn_monroe | marriage |
|
||||
|
||||
## 耗时与质量
|
||||
|
||||
没有数。空答、截断、rubric、人工四项都没做。默认值维持不设。
|
||||
@@ -0,0 +1,203 @@
|
||||
/**
|
||||
* Request-body probe for issue #65.
|
||||
*
|
||||
* --dry-run needs no model key. It sends one agent.stream per shape through
|
||||
* Mastra's OpenAI-compatible channel and prints the thinking / reasoning_effort
|
||||
* / max_tokens fields of each request. A fake fetch never leaves the machine.
|
||||
*
|
||||
* Shapes:
|
||||
* (a) consultationGenerationSettings (today's answer settings)
|
||||
* (b) consultationSliceGenerationSettings (nested reasoningEffort, no caller)
|
||||
* (c) providerReasoningEffort low / medium / high / max at the top level
|
||||
*
|
||||
* With DS_KEY or BACKTEST_MODEL_KEY, the same file can ask the provider which
|
||||
* efforts it accepts. This checkout does not run that half.
|
||||
*/
|
||||
import { writeFileSync } from "node:fs";
|
||||
|
||||
import { Agent } from "@mastra/core/agent";
|
||||
import { createTool } from "@mastra/core/tools";
|
||||
import { z } from "zod";
|
||||
|
||||
import { agentGenerationSettings } from "../../src/lib/agent-generation-settings.ts";
|
||||
import {
|
||||
consultationContinueGenerationSettings,
|
||||
consultationGenerationSettings,
|
||||
consultationNatalPrepareStep,
|
||||
consultationSliceGenerationSettings,
|
||||
} from "../../src/mastra/consultation-tools.ts";
|
||||
|
||||
const MODEL = {
|
||||
providerId: "deepseek",
|
||||
modelId: "deepseek-flash",
|
||||
url: "http://127.0.0.1:9/v1",
|
||||
apiKey: "dummy",
|
||||
};
|
||||
|
||||
const markTool = createTool({
|
||||
id: "run-jyotish-consultation",
|
||||
description: "Mark the calculation done.",
|
||||
inputSchema: z.object({}).strict(),
|
||||
execute: async () => ({ ok: true }),
|
||||
});
|
||||
|
||||
function sse(chunks: unknown[]) {
|
||||
const body = `${chunks.map((chunk) => `data: ${JSON.stringify(chunk)}\n\n`).join("")}data: [DONE]\n\n`;
|
||||
return new Response(body, {
|
||||
status: 200,
|
||||
headers: { "content-type": "text/event-stream" },
|
||||
});
|
||||
}
|
||||
|
||||
function toolCallChunk() {
|
||||
return sse([
|
||||
{
|
||||
id: "chatcmpl-tool",
|
||||
object: "chat.completion.chunk",
|
||||
choices: [{
|
||||
index: 0,
|
||||
delta: {
|
||||
role: "assistant",
|
||||
tool_calls: [{
|
||||
index: 0,
|
||||
id: "call_1",
|
||||
type: "function",
|
||||
function: { name: "run-jyotish-consultation", arguments: "" },
|
||||
}],
|
||||
},
|
||||
finish_reason: null,
|
||||
}],
|
||||
},
|
||||
{
|
||||
id: "chatcmpl-tool",
|
||||
object: "chat.completion.chunk",
|
||||
choices: [{
|
||||
index: 0,
|
||||
delta: { tool_calls: [{ index: 0, function: { arguments: "{}" } }] },
|
||||
finish_reason: null,
|
||||
}],
|
||||
},
|
||||
{
|
||||
id: "chatcmpl-tool",
|
||||
object: "chat.completion.chunk",
|
||||
choices: [{ index: 0, delta: {}, finish_reason: "tool_calls" }],
|
||||
},
|
||||
]);
|
||||
}
|
||||
|
||||
function textChunk() {
|
||||
return sse([
|
||||
{
|
||||
id: "chatcmpl-text",
|
||||
object: "chat.completion.chunk",
|
||||
choices: [{ index: 0, delta: { role: "assistant", content: "ok" }, finish_reason: null }],
|
||||
},
|
||||
{
|
||||
id: "chatcmpl-text",
|
||||
object: "chat.completion.chunk",
|
||||
choices: [{ index: 0, delta: {}, finish_reason: "stop" }],
|
||||
usage: { prompt_tokens: 1, completion_tokens: 1, completion_tokens_details: { reasoning_tokens: 0 } },
|
||||
},
|
||||
]);
|
||||
}
|
||||
|
||||
export function requestFields(body: unknown) {
|
||||
const record = body && typeof body === "object" ? body as Record<string, unknown> : {};
|
||||
return {
|
||||
thinking: record.thinking ?? null,
|
||||
reasoning_effort: record.reasoning_effort ?? null,
|
||||
max_tokens: record.max_tokens ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
export async function captureStreamBodies(input: {
|
||||
settings: ReturnType<typeof consultationGenerationSettings>;
|
||||
prepareStep?: (step: { stepNumber: number }) => { toolChoice?: "auto"; activeTools?: string[]; providerOptions?: unknown };
|
||||
steps: 1 | 2;
|
||||
}): Promise<unknown[]> {
|
||||
const bodies: unknown[] = [];
|
||||
const previous = globalThis.fetch;
|
||||
globalThis.fetch = async (_url, init) => {
|
||||
const raw = init?.body;
|
||||
bodies.push(JSON.parse(typeof raw === "string" ? raw : "{}"));
|
||||
return bodies.length === 1 && input.steps === 2 ? toolCallChunk() : textChunk();
|
||||
};
|
||||
try {
|
||||
const agent = new Agent({
|
||||
id: `answer-effort-probe-${bodies.length}-${Date.now()}`,
|
||||
name: "answer-effort-probe",
|
||||
instructions: "Reply ok.",
|
||||
model: MODEL,
|
||||
tools: { "run-jyotish-consultation": markTool },
|
||||
});
|
||||
const result = await agent.stream([{ role: "user", content: "ping" }], {
|
||||
maxSteps: input.steps,
|
||||
...input.settings,
|
||||
...(input.prepareStep ? { prepareStep: input.prepareStep } : {}),
|
||||
} as never);
|
||||
const drained: unknown[] = [];
|
||||
for await (const chunk of result.fullStream) drained.push(chunk);
|
||||
if (drained.length === 0) throw new Error("probe stream produced no chunks");
|
||||
} finally {
|
||||
globalThis.fetch = previous;
|
||||
}
|
||||
return bodies;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
if (!process.argv.includes("--dry-run")) {
|
||||
console.log("A/B 未跑：缺模型 key。只做请求体拦截时加上 --dry-run。");
|
||||
return;
|
||||
}
|
||||
const model = MODEL;
|
||||
const natal = await captureStreamBodies({
|
||||
settings: consultationGenerationSettings(model),
|
||||
prepareStep: (input) => consultationNatalPrepareStep({
|
||||
stepNumber: input.stepNumber,
|
||||
answerReady: input.stepNumber >= 1,
|
||||
}),
|
||||
steps: 2,
|
||||
});
|
||||
const slice = await captureStreamBodies({
|
||||
settings: consultationSliceGenerationSettings(model),
|
||||
steps: 1,
|
||||
});
|
||||
const topLevel = [];
|
||||
for (const effort of ["low", "medium", "high", "max"] as const) {
|
||||
const bodies = await captureStreamBodies({
|
||||
settings: agentGenerationSettings(model, { thinking: "enabled", providerReasoningEffort: effort }),
|
||||
steps: 1,
|
||||
});
|
||||
topLevel.push({ effort, fields: requestFields(bodies[0]) });
|
||||
}
|
||||
const continued = await captureStreamBodies({
|
||||
settings: consultationContinueGenerationSettings(model),
|
||||
steps: 1,
|
||||
});
|
||||
const report = {
|
||||
fixtureProvenance: "frontend/scripts/research/answer-effort-probe.mts --dry-run",
|
||||
a_consultationGenerationSettings: natal.map(requestFields),
|
||||
b_consultationSliceGenerationSettings: slice.map(requestFields),
|
||||
c_providerReasoningEffort: topLevel,
|
||||
continue: continued.map(requestFields),
|
||||
};
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
if (process.argv.includes("--save-baseline")) {
|
||||
const file = new URL("../../tests/fixtures/answer-effort-switch-20261013.json", import.meta.url);
|
||||
writeFileSync(file, `${JSON.stringify({
|
||||
fixtureProvenance: "node node_modules/tsx/dist/cli.mjs scripts/research/answer-effort-probe.mts --dry-run --save-baseline（开关尚未读取环境变量时的整份请求体）",
|
||||
natal: natal,
|
||||
continue: continued,
|
||||
}, null, 2)}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
const invoked = process.argv[1]
|
||||
? new URL(`file:///${process.argv[1].replace(/\\/g, "/")}`).href === import.meta.url
|
||||
: false;
|
||||
if (invoked) {
|
||||
main().catch((error) => {
|
||||
console.error(error);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
}
|
||||
@@ -27,6 +27,7 @@
|
||||
* Usage (from frontend/, Node 22):
|
||||
* DS_KEY=... ./node_modules/.bin/tsx scripts/research/consult-biography-backtest.mts \
|
||||
* [--out <dir>] [--repeats 2] [--figures a,b] [--domains parents,marriage,health,career] \
|
||||
* [--cases steve_jobs:career,barack_obama:parents] [--efforts unset,low,medium] \
|
||||
* [--model deepseek-flash] [--base-url https://api.deepseek.com] [--concurrency 4] [--dump-cards]
|
||||
* [--correction barack_obama,marilyn_monroe] [--correction-text 我爸其实没怎么管过我]
|
||||
* [--correction-domain career] [--correction-texts "zinedine_zidane=我是足球运动员|barack_obama=我是政治家"]
|
||||
@@ -142,6 +143,8 @@ function parseArgs(argv: readonly string[]) {
|
||||
const at = pair.indexOf("=");
|
||||
return [pair.slice(0, at), pair.slice(at + 1)];
|
||||
})) as Record<string, string>,
|
||||
cases: value("cases")?.split(",").map((item) => item.trim()).filter(Boolean) ?? null,
|
||||
efforts: value("efforts")?.split(",").map((item) => item.trim()).filter(Boolean) ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -270,7 +273,15 @@ function model(args: ReturnType<typeof parseArgs>, apiKey: string): ResolvedLang
|
||||
|
||||
/** A follow-up turn in the same session: the earlier question and answer are the history. */
|
||||
type FollowUp = { priorQuestion: string; priorAnswer: string };
|
||||
type Job = { figure: GoldenFigure; domain: string; question: string; run: number; followUp?: FollowUp };
|
||||
type Job = {
|
||||
figure: GoldenFigure;
|
||||
domain: string;
|
||||
question: string;
|
||||
run: number;
|
||||
followUp?: FollowUp;
|
||||
/** When set, this job overrides CONSULT_ANSWER_REASONING_EFFORT for its own stream. */
|
||||
effort?: string;
|
||||
};
|
||||
|
||||
type Usage = { inputTokens?: number; outputTokens?: number; reasoningTokens?: number; cachedInputTokens?: number };
|
||||
|
||||
@@ -316,6 +327,22 @@ function userTurn(question: string, followUp?: FollowUp) {
|
||||
}
|
||||
|
||||
async function runJob(job: Job, resolved: ResolvedLanguageModel) {
|
||||
const previousEffort = process.env.CONSULT_ANSWER_REASONING_EFFORT;
|
||||
if (job.effort !== undefined) {
|
||||
if (job.effort === "unset" || job.effort === "") delete process.env.CONSULT_ANSWER_REASONING_EFFORT;
|
||||
else process.env.CONSULT_ANSWER_REASONING_EFFORT = job.effort;
|
||||
}
|
||||
try {
|
||||
return await runJobOnce(job, resolved);
|
||||
} finally {
|
||||
if (job.effort !== undefined) {
|
||||
if (previousEffort === undefined) delete process.env.CONSULT_ANSWER_REASONING_EFFORT;
|
||||
else process.env.CONSULT_ANSWER_REASONING_EFFORT = previousEffort;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function runJobOnce(job: Job, resolved: ResolvedLanguageModel) {
|
||||
const state = createConsultationRuntimeState();
|
||||
const executed: string[] = [];
|
||||
const ctx = agentContext(job, state, executed);
|
||||
@@ -333,7 +360,14 @@ async function runJob(job: Job, resolved: ResolvedLanguageModel) {
|
||||
maxSteps: AGENT_MAX_STEPS,
|
||||
...consultationGenerationSettings(resolved.model),
|
||||
};
|
||||
const natalStreamOptions = { ...streamOptions, prepareStep: consultationNatalPrepareStep };
|
||||
const natalStreamOptions = {
|
||||
...streamOptions,
|
||||
prepareStep: (input: { stepNumber: number }) => consultationNatalPrepareStep({
|
||||
stepNumber: input.stepNumber,
|
||||
answerReady: state.consultationToolCompleted,
|
||||
model: resolved.model,
|
||||
}),
|
||||
};
|
||||
const usages: Promise<unknown>[] = [];
|
||||
const startedAt = Date.now();
|
||||
let output: string | null = null;
|
||||
@@ -417,6 +451,17 @@ async function runJob(job: Job, resolved: ResolvedLanguageModel) {
|
||||
methodologySections: state.methodologySectionCount,
|
||||
lookups: state.evidenceLookupCallCount,
|
||||
finish: state.composeFinishReason ?? state.modelFinishReason ?? null,
|
||||
effort: process.env.CONSULT_ANSWER_REASONING_EFFORT?.trim() || null,
|
||||
answerReasoningMs: state.answerReasoningMs ?? null,
|
||||
steps: [0, 1].map((index) => {
|
||||
const step = state.modelStepTimings?.find((item) => item.index === index);
|
||||
return {
|
||||
index,
|
||||
seconds: step?.durationMs == null ? null : step.durationMs / 1000,
|
||||
reasoningTokens: step?.reasoningTokens ?? null,
|
||||
outputTokens: step?.outputTokens ?? null,
|
||||
};
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -477,7 +522,18 @@ console.info = (...items: unknown[]) => {
|
||||
productInfo(...items);
|
||||
};
|
||||
|
||||
type Row = { figure: string; domain: string; run: number; chars: number; seconds: number; checks: ReturnType<typeof deterministicChecks> };
|
||||
type Row = {
|
||||
figure: string;
|
||||
domain: string;
|
||||
run: number;
|
||||
chars: number;
|
||||
seconds: number;
|
||||
checks: ReturnType<typeof deterministicChecks>;
|
||||
empty?: boolean;
|
||||
truncated?: boolean;
|
||||
finish?: string | null;
|
||||
steps?: { seconds: number | null; reasoningTokens: number | null; outputTokens: number | null }[];
|
||||
};
|
||||
|
||||
function writeSummary(out: string, summary: Json & { rows: Row[] }, modelId: string) {
|
||||
const rows = summary.rows;
|
||||
@@ -488,13 +544,13 @@ function writeSummary(out: string, summary: Json & { rows: Row[] }, modelId: str
|
||||
writeFileSync(join(out, "summary.md"), [
|
||||
`# 名人生平回测 · ${modelId}`,
|
||||
"",
|
||||
`开始 ${summary.started_at}，${rows.length} 份，墙钟 ${Number(summary.wall_seconds).toFixed(0)} s，失败 ${summary.failures} 份。`,
|
||||
`开始 ${summary.started_at}，${rows.length} 份，墙钟 ${Number(summary.wall_seconds).toFixed(0)} s，失败 ${summary.failures} 份。推理强度：${String(summary.reasoning_effort ?? "不设")}。`,
|
||||
`用量合计：${JSON.stringify(summary.usage)}`,
|
||||
`确定性检查：禁句 ${summary.banned_hits}，性别代词（婚恋/子女）${summary.pronoun_hits}，must_not 原文 ${summary.must_not_literal_hits}。判断类评分需逐份阅读。`,
|
||||
"",
|
||||
"| 名人 | 领域 | run | 字数 | 秒 | 禁句 | 代词 | must_not |",
|
||||
"| --- | --- | --- | --- | --- | --- | --- | --- |",
|
||||
...rows.map((row) => `| ${row.figure} | ${row.domain} | ${row.run} | ${row.chars} | ${row.seconds.toFixed(0)} | ${row.checks.banned.length} | ${row.checks.pronouns.length} | ${row.checks.literalHits.length} |`),
|
||||
"| 名人 | 领域 | run | 字数 | 秒 | 空答 | 截断 | 第0步秒 | 第1步秒 | 第1步推理token | 结束 |",
|
||||
"| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |",
|
||||
...rows.map((row) => `| ${row.figure} | ${row.domain} | ${row.run} | ${row.chars} | ${row.seconds.toFixed(0)} | ${row.empty ? "是" : ""} | ${row.truncated ? "是" : ""} | ${row.steps?.[0]?.seconds ?? ""} | ${row.steps?.[1]?.seconds ?? ""} | ${row.steps?.[1]?.reasoningTokens ?? ""} | ${row.finish ?? ""} |`),
|
||||
"",
|
||||
"## must_not 原文命中",
|
||||
"",
|
||||
@@ -529,11 +585,28 @@ async function main() {
|
||||
}
|
||||
const questions = new Map(golden.routes.map((route) => [route.domain, route.question]));
|
||||
const figures = golden.figures.filter((figure) => !args.figures || args.figures.includes(figure.id));
|
||||
const jobs: Job[] = figures.flatMap((figure) => args.domains.flatMap((domain) => {
|
||||
const question = questions.get(domain);
|
||||
if (!question) throw new Error(`no golden question for domain ${domain}`);
|
||||
return Array.from({ length: args.repeats }, (_, index) => ({ figure, domain, question, run: index + 1 }));
|
||||
}));
|
||||
const pairs = args.cases
|
||||
? args.cases.map((spec) => {
|
||||
const splitAt = spec.indexOf(":");
|
||||
const figureId = spec.slice(0, splitAt);
|
||||
const domain = spec.slice(splitAt + 1);
|
||||
const figure = golden.figures.find((item) => item.id === figureId);
|
||||
const question = questions.get(domain);
|
||||
if (!figure || !question) throw new Error(`unknown case ${spec}`);
|
||||
return { figure, domain, question };
|
||||
})
|
||||
: figures.flatMap((figure) => args.domains.flatMap((domain) => {
|
||||
const question = questions.get(domain);
|
||||
if (!question) throw new Error(`no golden question for domain ${domain}`);
|
||||
return [{ figure, domain, question }];
|
||||
}));
|
||||
const efforts = args.efforts;
|
||||
const jobs: Job[] = pairs.flatMap((pair) => Array.from({ length: args.repeats }, (_, index) => {
|
||||
const run = index + 1;
|
||||
if (!efforts) return [{ ...pair, run }];
|
||||
return efforts.map((effort) => ({ ...pair, run, effort }));
|
||||
}).flat());
|
||||
const concurrency = efforts && efforts.length > 1 ? 1 : args.concurrency;
|
||||
mkdirSync(args.out, { recursive: true });
|
||||
if (args.dumpCards) {
|
||||
await dumpCards(jobs, args.out);
|
||||
@@ -543,12 +616,12 @@ async function main() {
|
||||
if (!apiKey) throw new Error("Set DS_KEY (or BACKTEST_MODEL_KEY); without a key the backtest is an environment gap.");
|
||||
const resolved = model(args, apiKey);
|
||||
const started = Date.now();
|
||||
const rows = await pool(jobs, args.concurrency, async (job) => {
|
||||
const rows = await pool(jobs, concurrency, async (job) => {
|
||||
let result: Awaited<ReturnType<typeof runJob>>;
|
||||
try {
|
||||
result = await runJob(job, resolved);
|
||||
} catch (error) {
|
||||
result = { output: "", failure: error instanceof Error ? error.message : String(error), executed: [], seconds: 0, usage: {}, cardChars: null, visibleChars: null, methodologySections: 0, lookups: 0, finish: null };
|
||||
result = { output: "", failure: error instanceof Error ? error.message : String(error), executed: [], seconds: 0, usage: {}, cardChars: null, visibleChars: null, methodologySections: 0, lookups: 0, finish: null, effort: job.effort ?? null, answerReasoningMs: null, steps: [{ index: 0, seconds: null, reasoningTokens: null, outputTokens: null }, { index: 1, seconds: null, reasoningTokens: null, outputTokens: null }] };
|
||||
}
|
||||
const mustNot = rubric.figures.find((item) => item.id === job.figure.id)?.domains[job.domain]?.must_not ?? [];
|
||||
const checks = deterministicChecks(result.output, job.domain, mustNot);
|
||||
@@ -573,7 +646,18 @@ async function main() {
|
||||
"",
|
||||
].join("\n"));
|
||||
console.log(`${job.figure.id} ${job.domain} run${job.run}: ${result.output.length} chars, ${result.seconds.toFixed(0)} s${result.failure ? `, FAILED ${result.failure}` : ""}`);
|
||||
return { figure: job.figure.id, domain: job.domain, run: job.run, chars: result.output.length, ...result, output: undefined, checks };
|
||||
return {
|
||||
figure: job.figure.id,
|
||||
domain: job.domain,
|
||||
run: job.run,
|
||||
chars: result.output.length,
|
||||
empty: result.output.trim() === "",
|
||||
truncated: result.finish === "length" || (result.failure ?? "").includes("answer_truncated"),
|
||||
outputTokens: result.usage.outputTokens ?? null,
|
||||
...result,
|
||||
output: undefined,
|
||||
checks,
|
||||
};
|
||||
});
|
||||
const corrections = await pool(
|
||||
jobs.filter((job) => args.correctionDomains.includes(job.domain)
|
||||
@@ -589,7 +673,7 @@ async function main() {
|
||||
try {
|
||||
result = await runJob(job, resolved);
|
||||
} catch (error) {
|
||||
result = { output: "", failure: error instanceof Error ? error.message : String(error), executed: [], seconds: 0, usage: {}, cardChars: null, visibleChars: null, methodologySections: 0, lookups: 0, finish: null };
|
||||
result = { output: "", failure: error instanceof Error ? error.message : String(error), executed: [], seconds: 0, usage: {}, cardChars: null, visibleChars: null, methodologySections: 0, lookups: 0, finish: null, effort: job.effort ?? null, answerReasoningMs: null, steps: [{ index: 0, seconds: null, reasoningTokens: null, outputTokens: null }, { index: 1, seconds: null, reasoningTokens: null, outputTokens: null }] };
|
||||
}
|
||||
const file = join(args.out, job.figure.id, `${job.domain}-correction-run${job.run}.md`);
|
||||
writeFileSync(file, [
|
||||
@@ -619,6 +703,9 @@ async function main() {
|
||||
jobs: rows.length,
|
||||
failures: rows.filter((row) => row.failure || row.chars === 0).length,
|
||||
usage: total,
|
||||
reasoning_effort: process.env.CONSULT_ANSWER_REASONING_EFFORT?.trim() || null,
|
||||
efforts: args.efforts,
|
||||
concurrency,
|
||||
gaps: GAPS,
|
||||
rows,
|
||||
corrections,
|
||||
|
||||
@@ -75,6 +75,8 @@ import {
|
||||
createConsultationRunClock,
|
||||
consultationContinueGenerationSettings,
|
||||
consultationGenerationSettings,
|
||||
consultationGeneralPrepareStep,
|
||||
consultAnswerEffortProviderOptions,
|
||||
consultationNatalPrepareStep,
|
||||
consultationWindowPrepareStep,
|
||||
createConsultationAgentContext,
|
||||
@@ -1288,6 +1290,8 @@ export async function POST(request: Request) {
|
||||
prepareStep: (input: { stepNumber: number }) => consultationNatalPrepareStep({
|
||||
stepNumber: input.stepNumber,
|
||||
rectificationOffer: true,
|
||||
answerReady: state.consultationToolCompleted,
|
||||
model: selectedModel.model,
|
||||
}),
|
||||
};
|
||||
const windowStreamOptions = {
|
||||
@@ -1295,6 +1299,15 @@ export async function POST(request: Request) {
|
||||
prepareStep: (input: { stepNumber: number }) => consultationWindowPrepareStep({
|
||||
stepNumber: input.stepNumber,
|
||||
rectificationOffer: true,
|
||||
answerReady: state.consultationToolCompleted,
|
||||
model: selectedModel.model,
|
||||
}),
|
||||
};
|
||||
const generalStreamOptions = {
|
||||
...streamOptions,
|
||||
prepareStep: (input: { stepNumber: number }) => consultationGeneralPrepareStep({
|
||||
stepNumber: input.stepNumber,
|
||||
model: selectedModel.model,
|
||||
}),
|
||||
};
|
||||
const answerPhaseSignal = runClock.answerSignal;
|
||||
@@ -1305,10 +1318,10 @@ export async function POST(request: Request) {
|
||||
agent: {
|
||||
stream: (
|
||||
messages: typeof baseMessages,
|
||||
options: typeof streamOptions | typeof natalStreamOptions | typeof windowStreamOptions,
|
||||
options: typeof streamOptions | typeof natalStreamOptions | typeof windowStreamOptions | typeof generalStreamOptions,
|
||||
) => Promise<{ fullStream: AsyncIterable<unknown> | ReadableStream<unknown>; totalUsage: Promise<Usage> }>;
|
||||
},
|
||||
options: typeof streamOptions | typeof natalStreamOptions | typeof windowStreamOptions = streamOptions,
|
||||
options: typeof streamOptions | typeof natalStreamOptions | typeof windowStreamOptions | typeof generalStreamOptions = streamOptions,
|
||||
) {
|
||||
try {
|
||||
const result = await agent.stream(baseMessages, options);
|
||||
@@ -1369,14 +1382,14 @@ export async function POST(request: Request) {
|
||||
if (usesPublicDailyGeneralAgent(consultationMode, generalDailyContext)) {
|
||||
state.workflowReceipt = workflowReceipt;
|
||||
const agent = getGeneralJyotishAgent(selectedModel);
|
||||
const result = await streamWithOverflowRetry(agent);
|
||||
const result = await streamWithOverflowRetry(agent, generalStreamOptions);
|
||||
// This mode has no calculation to require and no chart method to bind,
|
||||
// so there is no contract for a retry to repair.
|
||||
const retryForAnswer = async (retryHint?: string) => {
|
||||
const retried = await agent.stream([
|
||||
...baseMessages,
|
||||
{ role: "user" as const, content: `上一轮没有输出任何回答文本。请直接给出这个问题的回答，不要只说明过程。${retryHint ? `\n${retryHint}` : ""}` },
|
||||
], { ...streamOptions, abortSignal: answerPhaseSignal() });
|
||||
], { ...generalStreamOptions, abortSignal: answerPhaseSignal() });
|
||||
usages.push(retried.totalUsage);
|
||||
return retried.fullStream;
|
||||
};
|
||||
@@ -1472,7 +1485,11 @@ export async function POST(request: Request) {
|
||||
role: "user" as const,
|
||||
content: `服务器窗口计算已经完成，但上一轮没有输出任何回答文本。请重新取回本次计算结果，然后直接给出回答；不要只描述过程或工具调用。${retryHint ? `\n${retryHint}` : ""}`,
|
||||
},
|
||||
], { ...streamOptions, abortSignal: answerPhaseSignal() });
|
||||
], {
|
||||
...streamOptions,
|
||||
abortSignal: answerPhaseSignal(),
|
||||
...consultAnswerEffortProviderOptions(selectedModel.model),
|
||||
});
|
||||
usages.push(retried.totalUsage);
|
||||
return retried.fullStream;
|
||||
};
|
||||
|
||||
@@ -16,6 +16,8 @@ export const AGENT_MAX_OUTPUT_TOKENS = AGENT_ANSWER_OUTPUT_TOKENS;
|
||||
|
||||
export type ThinkingMode = "enabled" | "disabled";
|
||||
export type ReasoningEffort = "low" | "medium" | "high";
|
||||
/** Top-level OpenAI-compatible `reasoning_effort`, including DeepSeek's `max`. */
|
||||
export type ProviderReasoningEffort = "low" | "medium" | "high" | "max";
|
||||
|
||||
export type PromptCacheUsage = Readonly<{
|
||||
readTokens: number;
|
||||
@@ -100,6 +102,8 @@ export function agentGenerationSettings(
|
||||
answerTokens?: number;
|
||||
thinkingTokens?: number;
|
||||
reasoningEffort?: ReasoningEffort;
|
||||
/** Written beside `thinking`, not inside it, so compatible gateways emit `reasoning_effort`. */
|
||||
providerReasoningEffort?: ProviderReasoningEffort;
|
||||
} = {},
|
||||
) {
|
||||
const thinkingMode: ThinkingMode = options.thinking ?? "disabled";
|
||||
@@ -110,6 +114,9 @@ export function agentGenerationSettings(
|
||||
? { reasoningEffort: options.reasoningEffort }
|
||||
: {}),
|
||||
},
|
||||
...(options.providerReasoningEffort
|
||||
? { reasoningEffort: options.providerReasoningEffort }
|
||||
: {}),
|
||||
};
|
||||
const providerId = typeof model === "string"
|
||||
? model
|
||||
|
||||
@@ -25,7 +25,7 @@ import { createConsultationPlan, type ConsultationPlan } from "../lib/consultati
|
||||
import type { TechniqueAuditRow, WorkflowReceipt } from "../lib/consultation-agent-events.ts";
|
||||
import { normalizeTechniqueAuditRows } from "../lib/consultation-technique-audit.ts";
|
||||
import type { AgentModelFinishReason, EvidenceCardTelemetry } from "../lib/agent-observability.ts";
|
||||
import { agentGenerationSettings, AGENT_SLICE_ANSWER_OUTPUT_TOKENS, AGENT_SLICE_THINKING_OUTPUT_TOKENS } from "../lib/agent-generation-settings.ts";
|
||||
import { agentGenerationSettings, AGENT_SLICE_ANSWER_OUTPUT_TOKENS, AGENT_SLICE_THINKING_OUTPUT_TOKENS, type ProviderReasoningEffort } from "../lib/agent-generation-settings.ts";
|
||||
import { chartCalculationProgressLabel, evidenceLookupActivityLabel } from "../lib/consultation-activity-labels.ts";
|
||||
import { consultDomainConcurrency } from "../lib/consult/prep-flags.ts";
|
||||
import {
|
||||
@@ -176,6 +176,45 @@ export const CONSULTATION_EVIDENCE_LOOKUP_TOOL_ID = "read-consultation-evidence"
|
||||
export const MAX_EVIDENCE_LOOKUPS_PER_TURN = 3;
|
||||
export const CONSULTATION_WINDOW_CALC_TOOL_ID = "run-jyotish-window-consultation";
|
||||
|
||||
/**
|
||||
* Answer-step reasoning effort. Unset or blank keeps today's request body.
|
||||
* An illegal value warns once and is treated as unset.
|
||||
*/
|
||||
const CONSULT_ANSWER_REASONING_EFFORTS = ["low", "medium", "high", "max"] as const;
|
||||
let consultAnswerReasoningEffortWarned = false;
|
||||
|
||||
export function consultAnswerReasoningEffort(): ProviderReasoningEffort | undefined {
|
||||
const raw = process.env.CONSULT_ANSWER_REASONING_EFFORT;
|
||||
if (raw === undefined || raw.trim() === "") return undefined;
|
||||
const value = raw.trim();
|
||||
if ((CONSULT_ANSWER_REASONING_EFFORTS as readonly string[]).includes(value)) {
|
||||
return value as ProviderReasoningEffort;
|
||||
}
|
||||
if (!consultAnswerReasoningEffortWarned) {
|
||||
consultAnswerReasoningEffortWarned = true;
|
||||
console.warn(`CONSULT_ANSWER_REASONING_EFFORT=${value} is not low, medium, high, or max; leaving reasoning effort unset.`);
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
type AnswerEffortInput = {
|
||||
answerReady?: boolean;
|
||||
answerFromStart?: boolean;
|
||||
model?: unknown;
|
||||
};
|
||||
|
||||
function answerEffortOptions(input: AnswerEffortInput) {
|
||||
const effort = consultAnswerReasoningEffort();
|
||||
const answerStep = input.answerFromStart === true || input.answerReady === true;
|
||||
if (!effort || !answerStep) return {};
|
||||
return {
|
||||
providerOptions: agentGenerationSettings(input.model, {
|
||||
thinking: "enabled",
|
||||
providerReasoningEffort: effort,
|
||||
}).providerOptions,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Step 0 narrows with activeTools and leaves toolChoice auto. Thinking-mode
|
||||
* providers reject named and required toolChoice (BUG-282, BUG-937).
|
||||
@@ -185,35 +224,77 @@ export const CONSULTATION_WINDOW_CALC_TOOL_ID = "run-jyotish-window-consultation
|
||||
* Later steps stay auto.
|
||||
*
|
||||
* Window and general agents must not share the natal hook.
|
||||
*
|
||||
* The answer step is any step that already holds a calculation (`answerReady`,
|
||||
* the same flag as the answer clock) or a no-tool public-daily turn
|
||||
* (`answerFromStart`). A retry starts a new stream at step 0 with the
|
||||
* calculation already in hand, so the step number is not part of the check.
|
||||
* The main loop's step 0 still has `answerReady` false.
|
||||
*/
|
||||
export function consultationNatalPrepareStep(input: { stepNumber: number; rectificationOffer?: boolean }) {
|
||||
if (input.stepNumber !== 0) return { toolChoice: "auto" as const };
|
||||
type ConsultPrepareInput = {
|
||||
rectificationOffer?: boolean;
|
||||
answerReady?: boolean;
|
||||
answerFromStart?: boolean;
|
||||
model?: unknown;
|
||||
};
|
||||
|
||||
export function consultationNatalPrepareStep(input: { stepNumber: 0 } & ConsultPrepareInput): {
|
||||
activeTools: string[];
|
||||
toolChoice: "auto";
|
||||
};
|
||||
export function consultationNatalPrepareStep(input: { stepNumber: number } & ConsultPrepareInput): {
|
||||
toolChoice: "auto";
|
||||
activeTools?: string[];
|
||||
};
|
||||
export function consultationNatalPrepareStep(input: { stepNumber: number } & ConsultPrepareInput) {
|
||||
const effort = answerEffortOptions(input);
|
||||
if (input.stepNumber !== 0) return { toolChoice: "auto" as const, ...effort };
|
||||
if (!input.rectificationOffer) {
|
||||
return {
|
||||
activeTools: [CONSULTATION_NATAL_CALC_TOOL_ID],
|
||||
toolChoice: "auto" as const,
|
||||
...effort,
|
||||
};
|
||||
}
|
||||
return {
|
||||
activeTools: [CONSULTATION_NATAL_CALC_TOOL_ID, RECTIFICATION_OFFER_TOOL_ID],
|
||||
toolChoice: "auto" as const,
|
||||
...effort,
|
||||
};
|
||||
}
|
||||
|
||||
export function consultationWindowPrepareStep(input: { stepNumber: number; rectificationOffer?: boolean }) {
|
||||
if (input.stepNumber !== 0) return { toolChoice: "auto" as const };
|
||||
export function consultationWindowPrepareStep(input: {
|
||||
stepNumber: number;
|
||||
rectificationOffer?: boolean;
|
||||
answerReady?: boolean;
|
||||
answerFromStart?: boolean;
|
||||
model?: unknown;
|
||||
}) {
|
||||
const effort = answerEffortOptions(input);
|
||||
if (input.stepNumber !== 0) return { toolChoice: "auto" as const, ...effort };
|
||||
if (!input.rectificationOffer) {
|
||||
return {
|
||||
activeTools: [CONSULTATION_WINDOW_CALC_TOOL_ID],
|
||||
toolChoice: "auto" as const,
|
||||
...effort,
|
||||
};
|
||||
}
|
||||
return {
|
||||
activeTools: [CONSULTATION_WINDOW_CALC_TOOL_ID, RECTIFICATION_OFFER_TOOL_ID],
|
||||
toolChoice: "auto" as const,
|
||||
...effort,
|
||||
};
|
||||
}
|
||||
|
||||
export function consultAnswerEffortProviderOptions(model?: unknown) {
|
||||
return answerEffortOptions({ answerReady: true, model });
|
||||
}
|
||||
|
||||
/** Public daily has no chart tool. Its first step is already the answer. */
|
||||
export function consultationGeneralPrepareStep(input: { stepNumber: number; model?: unknown }) {
|
||||
return answerEffortOptions({ ...input, answerFromStart: true });
|
||||
}
|
||||
|
||||
const CONSULTATION_DOMAIN_DURATION_MS = 31_000;
|
||||
const CONSULTATION_ANSWER_RESERVE_MS = 45_000;
|
||||
export const CONSULTATION_DOMAIN_WALL_CLOCK_MS = AGENT_TIMEOUT_MS - CONSULTATION_ANSWER_RESERVE_MS;
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
/**
|
||||
* Answer-step reasoning effort (issue #65).
|
||||
* The unset bodies were captured before this switch read the environment:
|
||||
* node node_modules/tsx/dist/cli.mjs scripts/research/answer-effort-probe.mts --dry-run --save-baseline
|
||||
*/
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import test from "node:test";
|
||||
|
||||
import {
|
||||
consultAnswerReasoningEffort,
|
||||
consultationContinueGenerationSettings,
|
||||
consultationGeneralPrepareStep,
|
||||
consultationGenerationSettings,
|
||||
consultationNatalPrepareStep,
|
||||
} from "../src/mastra/consultation-tools.ts";
|
||||
import { captureStreamBodies } from "../scripts/research/answer-effort-probe.mts";
|
||||
|
||||
const MODEL = {
|
||||
providerId: "deepseek",
|
||||
modelId: "deepseek-flash",
|
||||
url: "http://127.0.0.1:9/v1",
|
||||
apiKey: "dummy",
|
||||
};
|
||||
|
||||
const fixture = JSON.parse(readFileSync(new URL("./fixtures/answer-effort-switch-20261013.json", import.meta.url), "utf8")) as {
|
||||
natal: unknown[];
|
||||
continue: unknown[];
|
||||
};
|
||||
|
||||
function withoutEffort(body: unknown) {
|
||||
const copy = { ...(body as Record<string, unknown>) };
|
||||
delete copy.reasoning_effort;
|
||||
return copy;
|
||||
}
|
||||
|
||||
test("an illegal effort warns once and stays unset", () => {
|
||||
const previous = process.env.CONSULT_ANSWER_REASONING_EFFORT;
|
||||
const seen: unknown[][] = [];
|
||||
const warn = console.warn;
|
||||
console.warn = (...args: unknown[]) => { seen.push(args); };
|
||||
process.env.CONSULT_ANSWER_REASONING_EFFORT = "turbo";
|
||||
try {
|
||||
assert.equal(consultAnswerReasoningEffort(), undefined);
|
||||
assert.equal(consultAnswerReasoningEffort(), undefined);
|
||||
assert.equal(seen.length, 1);
|
||||
} finally {
|
||||
console.warn = warn;
|
||||
if (previous === undefined) delete process.env.CONSULT_ANSWER_REASONING_EFFORT;
|
||||
else process.env.CONSULT_ANSWER_REASONING_EFFORT = previous;
|
||||
}
|
||||
});
|
||||
|
||||
test("unset effort keeps the pre-change request bodies", async () => {
|
||||
const previous = process.env.CONSULT_ANSWER_REASONING_EFFORT;
|
||||
delete process.env.CONSULT_ANSWER_REASONING_EFFORT;
|
||||
try {
|
||||
const natal = await captureStreamBodies({
|
||||
settings: consultationGenerationSettings(MODEL),
|
||||
prepareStep: (input) => consultationNatalPrepareStep({
|
||||
stepNumber: input.stepNumber,
|
||||
answerReady: input.stepNumber >= 1,
|
||||
model: MODEL,
|
||||
}),
|
||||
steps: 2,
|
||||
});
|
||||
const continued = await captureStreamBodies({
|
||||
settings: consultationContinueGenerationSettings(MODEL),
|
||||
steps: 1,
|
||||
});
|
||||
assert.deepEqual(natal, fixture.natal);
|
||||
assert.deepEqual(continued, fixture.continue);
|
||||
} finally {
|
||||
if (previous === undefined) delete process.env.CONSULT_ANSWER_REASONING_EFFORT;
|
||||
else process.env.CONSULT_ANSWER_REASONING_EFFORT = previous;
|
||||
}
|
||||
});
|
||||
|
||||
test("low effort is only on the answer step", async () => {
|
||||
const previous = process.env.CONSULT_ANSWER_REASONING_EFFORT;
|
||||
process.env.CONSULT_ANSWER_REASONING_EFFORT = "low";
|
||||
try {
|
||||
const natal = await captureStreamBodies({
|
||||
settings: consultationGenerationSettings(MODEL),
|
||||
prepareStep: (input) => consultationNatalPrepareStep({
|
||||
stepNumber: input.stepNumber,
|
||||
answerReady: input.stepNumber >= 1,
|
||||
model: MODEL,
|
||||
}),
|
||||
steps: 2,
|
||||
}) as Record<string, unknown>[];
|
||||
assert.equal(natal[0]?.reasoning_effort, undefined);
|
||||
assert.deepEqual(natal[0], fixture.natal[0]);
|
||||
assert.equal(natal[1]?.reasoning_effort, "low");
|
||||
assert.deepEqual(withoutEffort(natal[1]), fixture.natal[1]);
|
||||
const continued = await captureStreamBodies({
|
||||
settings: consultationContinueGenerationSettings(MODEL),
|
||||
steps: 1,
|
||||
});
|
||||
assert.deepEqual(continued, fixture.continue);
|
||||
const daily = await captureStreamBodies({
|
||||
settings: consultationGenerationSettings(MODEL),
|
||||
prepareStep: (input) => consultationGeneralPrepareStep({ stepNumber: input.stepNumber, model: MODEL }),
|
||||
steps: 1,
|
||||
}) as Record<string, unknown>[];
|
||||
assert.equal(daily[0]?.reasoning_effort, "low");
|
||||
} finally {
|
||||
if (previous === undefined) delete process.env.CONSULT_ANSWER_REASONING_EFFORT;
|
||||
else process.env.CONSULT_ANSWER_REASONING_EFFORT = previous;
|
||||
}
|
||||
});
|
||||
@@ -50,6 +50,9 @@ test("the consult route starts the general / no-birth-minute loop on the answer
|
||||
);
|
||||
// That branch still streams on the shared loop signal and keeps its traces.
|
||||
const general = route.match(/if \(usesPublicDailyGeneralAgent\(consultationMode, generalDailyContext\)\) \{[\s\S]*?\n {4}\}\n/)?.[0] ?? "";
|
||||
assert.match(general, /await streamWithOverflowRetry\(agent\)/);
|
||||
// 原值：await streamWithOverflowRetry(agent)
|
||||
// 新值：await streamWithOverflowRetry(agent, generalStreamOptions)
|
||||
// 原因：公开日历从第一步就是答题。开关打开时这一步要带推理强度；不传选项就没有 prepareStep，强度加不上去。开关不设时这些选项不改请求体。
|
||||
assert.match(general, /await streamWithOverflowRetry\(agent, generalStreamOptions\)/);
|
||||
assert.match(general, /onError: \(error\) => settleRun\(\s*cancel,/);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
{
|
||||
"fixtureProvenance": "node node_modules/tsx/dist/cli.mjs scripts/research/answer-effort-probe.mts --dry-run --save-baseline（开关尚未读取环境变量时的整份请求体）",
|
||||
"natal": [
|
||||
{
|
||||
"model": "deepseek-flash",
|
||||
"max_tokens": 24576,
|
||||
"thinking": {
|
||||
"type": "enabled"
|
||||
},
|
||||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "Reply ok."
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "ping"
|
||||
}
|
||||
],
|
||||
"tools": [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "run-jyotish-consultation",
|
||||
"description": "Mark the calculation done.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {},
|
||||
"additionalProperties": false,
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"required": []
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto",
|
||||
"stream": true
|
||||
},
|
||||
{
|
||||
"model": "deepseek-flash",
|
||||
"max_tokens": 24576,
|
||||
"thinking": {
|
||||
"type": "enabled"
|
||||
},
|
||||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "Reply ok."
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "ping"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_1",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "run-jyotish-consultation",
|
||||
"arguments": "{}"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_1",
|
||||
"content": "{\"ok\":true}"
|
||||
}
|
||||
],
|
||||
"tools": [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "run-jyotish-consultation",
|
||||
"description": "Mark the calculation done.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {},
|
||||
"additionalProperties": false,
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"required": []
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto",
|
||||
"stream": true
|
||||
}
|
||||
],
|
||||
"continue": [
|
||||
{
|
||||
"model": "deepseek-flash",
|
||||
"max_tokens": 16384,
|
||||
"thinking": {
|
||||
"type": "disabled"
|
||||
},
|
||||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "Reply ok."
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "ping"
|
||||
}
|
||||
],
|
||||
"tools": [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "run-jyotish-consultation",
|
||||
"description": "Mark the calculation done.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {},
|
||||
"additionalProperties": false,
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"required": []
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto",
|
||||
"stream": true
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user