Merge remote-tracking branch 'origin/staging' into claude/report-reading-grade-20261112
# Conflicts: # docs/BUG_HISTORY.md
This commit is contained in:
@@ -25002,6 +25002,20 @@
|
||||
- 关联记录:#192、#201、#180、BUG-1846、BUG-1848、BUG-1794、BUG-1834。
|
||||
- 修复版本:`claude/report-reading-grade-20261112`。
|
||||
|
||||
## BUG-1850 | 回答时钟回归依赖真实耗时,资料库校验隐式选中旧导出
|
||||
|
||||
- 状态:fixed-pending-verify(待独立验收)。
|
||||
- 首次发现 / 最近更新:2026-11-12 / 2026-11-12。
|
||||
- 影响面:咨询证据查表回归、预置资料库来源校验与 CLI。
|
||||
- 现象:前端回归在负载下收到 `empty_answer`,原断言要求 `answer_truncated`。资料库定向测试在旧导出存在时收到哈希不匹配。
|
||||
- 触发条件:真实测试计时先于首段正文到点,或没有显式指定资料库固定来源。
|
||||
- 根因:旧夹具用短时真实等待,无法固定超时与首段正文的顺序。资料库测试从旧导出路径取输入,并以单个文件是否存在作为跳过条件。
|
||||
- 修复:接收首段正文后触发真实回答时钟订阅的受控 deadline。保留截断、零扣点、单时钟和中止回执断言。资料库测试要求显式来源并比对 Git HEAD 或所有固定文件哈希,CLI 要求显式选择来源模式。
|
||||
- 验证:四个 CPU 负载进程下,原测试文件 20 遍有 4 遍失败,修订后 20 遍通过。固定来源的资料库测试通过,错误来源与缺失目录均失败。全量结果见同轮进度。
|
||||
- 防复发:检查实际 deadline 对象、正文已发出和查表次数。回归 CLI 缺少来源选项及跳过来源时禁止写清单。
|
||||
- 关联记录:#205、BUG-1051、BUG-1053、BUG-1142、BUG-1847。
|
||||
- 修复版本:`codex/flaky-tests-20261112`。
|
||||
|
||||
## BUG-1852 | 个人报告解读把已算出的 Indu、Gulika 等写成「参考资料里没有对应条目」
|
||||
|
||||
- 状态:fixed-pending-verify(staging 待真机)。
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
# 回答时钟与资料库来源测试 · 2026-11-12
|
||||
|
||||
Gitea #205,BUG-1850。staging 基线 `017e650fdafac4a841487dbba4aca803c761dad8`。BUG-1849 已由并行的 #204 占用,沿用本单任务书指定的 BUG-1850。
|
||||
|
||||
## 回答时钟
|
||||
|
||||
- 原用例在查表和短文本分段中使用真实延迟。计算完成时答案时钟已经启动,查表、模型启动、Mastra 流处理及正文放行都消耗同一预算。负载增大时,300 ms 到点前可能还没有可交付正文。运行时此时返回 `empty_answer`,已有正文才记为 `answer_truncated`,两种结果均不扣点。这符合 BUG-1051 与 BUG-1053 的运行时契约。
|
||||
- Windows 以四个独立忙循环进程加负载,原文件串行跑 20 遍,4 遍失败且均为指定用例的 `empty_answer`。同文件另一条依赖工具阶段预算的用例在 20 遍中均通过。
|
||||
- 保留真实 Agent、时钟及查表。测试捕获首次答案 deadline,假模型在发送部分正文后等待,客户端收到首段正文才触发原 deadline。通过 signal 数量和身份核对查表未新建 deadline。
|
||||
|
||||
| 项目 | 原值 | 新值 | 原因 |
|
||||
| --- | --- | --- | --- |
|
||||
| 回答时钟用例的夹具 | 40 ms 查表、每段 25 ms、真实 300 ms deadline | 客户端收到首段后触发受控 deadline | 将超时固定在正文已输出的边界,保留原四项断言。 |
|
||||
| 资料库导出校验条件 | 隐式读取旧目录,按一份资料文件是否存在决定跳过 | 仅在未配置环境变量时跳过;配置后比对 Git HEAD 或所有固定来源文件哈希 | 避免把缺失或不匹配的显式来源作为跳过处理。原有文件、哈希和摘录断言仍执行。 |
|
||||
| 资料库 CLI | 未指定来源时尝试默认目录 | 必须显式选 `--her-export` 或 `--skip-her-export`,两者互斥 | 区分固定来源验证与开发检查。 |
|
||||
|
||||
## 验证
|
||||
|
||||
- 修订后的同一前端测试文件在四个负载进程下连续跑 20 遍,15 项用例每遍均通过。结束后按各进程 PID 核对其均已退出。
|
||||
- Python 资料库定向:环境变量未设置时 39 项通过、2 项按预期跳过。指定与固定 Git 版本逐文件校验一致的纯导出时 41 项通过。显式指定错误 Git HEAD、文件哈希不一致或目录缺失时均以退出码 1 结束并给出「来源与 VERIFIED.json 固定的版本不一致」。
|
||||
- 资料库 CLI 未选来源选项时退出码 2;显式跳过来源和固定导出两种模式均验证 44 条资料。跳过来源仍无法写 `VERIFIED.json`。
|
||||
- 预检确认远端可见、Python 运行时与外部引擎适配器诊断可用。Windows 上碎片扫描与预检定向测试分别在 90 秒、45 秒超时,与此前同机预检记录一致。
|
||||
- 原任务基线上的首轮前端全量运行出现 Windows 内存不足,部分测试进程未启动;使用有界并发重跑。
|
||||
- 原任务基线的 Python quick 分支运行 1463 项通过、2 项跳过、3 项失败,另有 29 项子测试通过。三项 Kaala 失败在相同基线 `017e650fd` 独立复现。
|
||||
- staging 更新至 `56bb1bed9` 后,同步到任务分支。唯一冲突是 Bug 历史的并行追加,保留 #192 的 BUG-1851 原文并在末尾追加本单原定编号 BUG-1850。低并发全量复核以更新后的 staging 为对照。
|
||||
- 更新基线后按同一 `npm test` 文件列表和 Node TAP runner 复跑,每树 4 个并发。两边各 3652 项、0 项取消,3632 个顶层 TAP 名称及出现次数完全一致。staging 3624 项通过、28 项失败,任务分支 3625 项通过、27 项失败,两边退出码均为 1。唯一仅在基线失败的是本单修订的回答时钟用例。
|
||||
- 更新基线后的 Python quick 仍为 1463 项通过、2 项跳过、3 项失败,另有 29 项子测试通过。staging `56bb1bed9` 单跑复现同样的三个 `tests/test_upstream_technique_layers.py` Kaala 用例失败。
|
||||
- 更新基线后的 TypeScript 检查通过,ESLint 0 错误、62 条警告。
|
||||
- 清理构建缓存后,staging 与任务分支 Next.js 构建均通过,`/` 为静态页面。两边均引用 39 个资源,逐文件 gzip -9 的 JS 共 515860 字节、CSS 共 90886 字节。
|
||||
- 隐私检查 62 项通过。
|
||||
@@ -0,0 +1,94 @@
|
||||
# TASK:两个会误报的测试(2026-11-12)
|
||||
|
||||
- Gitea:#205
|
||||
- 分支:`codex/flaky-tests-20261112`(基于 staging `017e650fdafac4a841487dbba4aca803c761dad8`)
|
||||
- Bug 编号:BUG-1850。开工时核对 `docs/BUG_HISTORY.md` 末尾;若已被占用,顺延,并在 PROGRESS 写明。
|
||||
- 负责验收:Claude(jyotisha-de)
|
||||
|
||||
不得自行写“产品决定/裁定”;任务书没写的取舍一律选需拍板。
|
||||
|
||||
不得修改或删除既有断言来让测试通过;确需修改必须写原值/新值/原因,并在 PROGRESS 里逐条列出。
|
||||
|
||||
## 一、回答时钟测试靠真实计时,CI 负载高时会失败
|
||||
|
||||
测试:`frontend/tests/consult-evidence-lookup-20260927.test.ts:280`,
|
||||
「a lookup does not reset the answer clock: the same 70 s signal bounds the whole answer phase」。
|
||||
|
||||
用例的写法:
|
||||
|
||||
- `answerMs: 300`;
|
||||
- 查表延迟 40 ms;
|
||||
- 回答分 6 段,每段延迟 25 ms;
|
||||
- 期望结果是 `run.failed:answer_truncated`。
|
||||
|
||||
实际情况:
|
||||
|
||||
- CI 运行 7208、7230 各失败过一次,单独重跑那一项就通过。
|
||||
- 本机在负载下得到的是 `empty_answer`,不是 `answer_truncated`。
|
||||
|
||||
要做的:
|
||||
|
||||
1. 先查清楚负载下为什么会变成 `empty_answer`。在 PROGRESS 写明,是测试的时间预算问题,还是产品代码在边界上真的有分支差异。
|
||||
- 如果是**产品**问题:停下来,选需拍板,不要改产品代码。
|
||||
2. 如果是测试的问题,把这个用例改成不依赖真实耗时:
|
||||
- 用注入的时钟或可控的 signal,让“回答阶段超时”在确定的那一步发生;
|
||||
- 保留四个断言的含义:结果是 `answer_truncated`;`charges` 为 0;回答时钟只启动一次;有 `compose-abort`。
|
||||
3. 用同样的方式检查同文件里其他依赖 `delayMs` 和预算的用例。只改那些在 4 倍 CPU 负载下会失败的,逐个列进 PROGRESS。
|
||||
4. 本机验证:开 4 个后台 `yes > /dev/null` 加负载(Windows 用等效做法),同文件连续跑 20 遍,0 失败。验证完结束这些负载进程,按 PID 结束,**不要按名字批量结束进程**。
|
||||
|
||||
## 二、预置资料库测试默认读旧导出
|
||||
|
||||
测试:`tests/test_preset_evidence_library.py::test_every_her_file_exists_sha_matches_and_quotes_are_verbatim`。
|
||||
|
||||
- 它默认读 `/workspace/up-run-r40`(`scripts/preset_evidence/library.py` 里的 `DEFAULT_HER_EXPORT`)。
|
||||
- 资料库现在钉的是她来源仓的 `1b33aff61d733213d9df67906ddf16473aafc9bb`,见 `references/preset_evidence_library/VERIFIED.json`。
|
||||
- 本机如果存在旧的 r40 导出,这个测试会因 `scripts/dasha_analyzer.py` 的哈希不同而失败。CI 上没有这个目录,测试会跳过。
|
||||
|
||||
要做的:
|
||||
|
||||
1. 去掉对 `/workspace/up-run-r40` 的默认值。只有设置了环境变量 `PRESET_EVIDENCE_HER_EXPORT` 时,才跑需要导出的测试;没设置就跳过,并在 skip 原因里写明。
|
||||
2. 设置了环境变量时,先核对这个目录和 `VERIFIED.json` 记录的来源是否一致。
|
||||
- 有 `.git` 的,比对 HEAD;
|
||||
- 没有的,比对 `VERIFIED.json` 里的文件哈希。
|
||||
- 不一致时,测试**失败**,并给出明确信息:“来源与 VERIFIED.json 固定的版本不一致”。不能跳过。
|
||||
3. CLI `python -m scripts.preset_evidence.library` 的 `--her-export` 改为必填,`--skip-her-export` 除外。帮助文字里写明要指向固定版本的来源仓。
|
||||
4. 同步更新 `SCHEMA.md` 里关于导出目录的说明。
|
||||
|
||||
## 要改的文件
|
||||
|
||||
- `frontend/tests/consult-evidence-lookup-20260927.test.ts`。如果需要注入时钟,测试辅助文件也可以改;**不改 `frontend/src`**,除非第一节查明是产品问题,那种情况选需拍板。
|
||||
- `scripts/preset_evidence/library.py`:只改默认路径和 CLI 参数。
|
||||
- `tests/test_preset_evidence_library.py`、`tests/test_preset_library_gates_20261112.py`:只改导出相关的跳过和核对逻辑。
|
||||
- `references/preset_evidence_library/SCHEMA.md`:只改导出目录的说明。
|
||||
- `docs/BUG_HISTORY.md`:只在末尾追加一条。
|
||||
- `docs/tasks/PROGRESS-flaky-tests-20261112.md`
|
||||
|
||||
## 不要碰
|
||||
|
||||
- `frontend/src/**`
|
||||
- `frontend/src/lib/consult-preset-prompts.ts`
|
||||
- 资料库条目和 `VERIFIED.json`
|
||||
- 冻结文件
|
||||
|
||||
## 自测命令
|
||||
|
||||
```
|
||||
cd frontend && npm test
|
||||
python -m pytest -q tests/test_preset_evidence_library.py tests/test_preset_library_gates_20261112.py
|
||||
PRESET_EVIDENCE_HER_EXPORT=<固定版本来源目录> python -m pytest -q tests/test_preset_evidence_library.py
|
||||
python scripts/run_quality_gate.py --profile quick --skip-frontend-runtime
|
||||
```
|
||||
|
||||
## 进度记录
|
||||
|
||||
写在 `docs/tasks/PROGRESS-flaky-tests-20261112.md`,内容包括:
|
||||
|
||||
- `empty_answer` 的原因;
|
||||
- 改了哪些用例,加负载连续 20 遍的结果;
|
||||
- 资料库测试分别在三种情况下的表现:未设置环境变量、设置为一致的来源、设置为不一致的来源;
|
||||
- 测试结果,以及在 staging 上也复现的基线失败。
|
||||
|
||||
## 何时停下来选“需拍板”
|
||||
|
||||
- `empty_answer` 是产品代码的边界问题,不是测试的问题。
|
||||
- 要稳定这个测试,只能放宽断言。
|
||||
@@ -92,7 +92,8 @@ const lookupCall = (section: string, text?: string): Turn => ({
|
||||
finish: "tool-calls",
|
||||
});
|
||||
|
||||
const run = (turns: Turn[], options: { toolPhaseMs?: number; answerMs?: number } = {}) => runNatalAgent(turns, { workflow, ...options });
|
||||
const run = (turns: Turn[], options: Omit<Parameters<typeof runNatalAgent>[1], "workflow"> = {}) =>
|
||||
runNatalAgent(turns, { workflow, ...options });
|
||||
|
||||
function toolResultText(prompt: unknown[] | undefined, toolName: string) {
|
||||
return JSON.stringify((prompt ?? []).filter((message) => {
|
||||
@@ -277,16 +278,40 @@ test("a step that only opens like the released text is not mistaken for a restar
|
||||
assert.equal(result.state.steps.some((step) => step.name === "answer-restart-dropped"), false);
|
||||
});
|
||||
|
||||
test("a lookup does not reset the answer clock: the same 70 s signal bounds the whole answer phase", async () => {
|
||||
test("a lookup does not reset the answer clock: the same 70 s signal bounds the whole answer phase", { timeout: 20_000 }, async (t) => {
|
||||
const answerMs = 300;
|
||||
const realTimeout = AbortSignal.timeout.bind(AbortSignal);
|
||||
const deadlines: AbortController[] = [];
|
||||
t.mock.method(AbortSignal, "timeout", (ms: number) => {
|
||||
if (ms !== answerMs) return realTimeout(ms);
|
||||
const deadline = new AbortController();
|
||||
deadlines.push(deadline);
|
||||
return deadline.signal;
|
||||
});
|
||||
let releaseFinish!: () => void;
|
||||
const waitBeforeFinish = new Promise<void>((resolve) => { releaseFinish = resolve; });
|
||||
const head = ANSWER.slice(0, ANSWER.indexOf(`## ${REPORT_HEADING.timing}`));
|
||||
const result = await run([
|
||||
calcCall(),
|
||||
{ ...lookupCall("varga:D60"), delayMs: 40 },
|
||||
{ parts: pieces(ANSWER, 6), finish: "stop", delayMs: 25 },
|
||||
], { answerMs: 300 });
|
||||
lookupCall("varga:D60"),
|
||||
{ parts: pieces(head), finish: "stop", waitBeforeFinish },
|
||||
], {
|
||||
answerMs,
|
||||
onAnswerDelta: (text) => {
|
||||
if (text.trim()) {
|
||||
deadlines[0]?.abort(new DOMException("Answer deadline elapsed", "TimeoutError"));
|
||||
releaseFinish();
|
||||
}
|
||||
},
|
||||
});
|
||||
assert.deepEqual(result.terminal.map((event) => `${event.type}:${event.code ?? ""}`), ["run.failed:answer_truncated"]);
|
||||
assert.equal(result.charges, 0);
|
||||
assert.equal(result.answerSignals.length, 1, "the answer clock started once");
|
||||
assert.ok(result.state.steps.some((step) => step.kind === "abort" && step.name === "compose-abort"));
|
||||
assert.equal(result.state.evidenceLookupCallCount, 1);
|
||||
assert.equal(deadlines.length, 1, "the lookup did not create another answer deadline");
|
||||
assert.equal(result.answerSignals[0], deadlines[0].signal);
|
||||
assert.ok(result.answer.trim(), "the deadline elapsed after the client received text");
|
||||
});
|
||||
|
||||
test("a lookup in the answer phase is not cut by the tool phase's deadline", async () => {
|
||||
|
||||
@@ -25,7 +25,7 @@ import {
|
||||
} from "../src/mastra/consultation-tools.ts";
|
||||
|
||||
export type Part = { text?: string; tool?: { name: string; input: Record<string, unknown> } };
|
||||
export type Turn = { parts: Part[]; finish: string; delayMs?: number };
|
||||
export type Turn = { parts: Part[]; finish: string; delayMs?: number; waitBeforeFinish?: Promise<void> };
|
||||
|
||||
/** A LanguageModelV2 that plays `turns` in order and records every prompt. */
|
||||
export function scriptedModel(turns: Turn[]) {
|
||||
@@ -85,6 +85,13 @@ export function scriptedModel(turns: Turn[]) {
|
||||
}
|
||||
}
|
||||
if (textOpen) controller.enqueue({ type: "text-end", id: `t${call}` });
|
||||
if (turn.waitBeforeFinish) {
|
||||
await turn.waitBeforeFinish;
|
||||
if (signal?.aborted) {
|
||||
controller.error(signal.reason);
|
||||
return;
|
||||
}
|
||||
}
|
||||
controller.enqueue({
|
||||
type: "finish",
|
||||
finishReason: turn.finish,
|
||||
@@ -133,6 +140,8 @@ export async function runNatalAgent(
|
||||
question?: string;
|
||||
toolPhaseMs?: number;
|
||||
answerMs?: number;
|
||||
/** Observe text as the client receives it, before the model finishes. */
|
||||
onAnswerDelta?: (text: string) => void;
|
||||
/** The bound subject's server chart; defaults to the public AA chart without a gender. */
|
||||
serverChart?: Record<string, unknown>;
|
||||
/** The shape instruction the user turn carries; defaults to the first-turn natal shape. */
|
||||
@@ -212,8 +221,19 @@ export async function runNatalAgent(
|
||||
onError: (error) => { errored = error; },
|
||||
});
|
||||
const events: Array<{ type: string; code?: string; text?: string; label?: string; phase?: string; receipt?: unknown }> = [];
|
||||
const parser = createNdjsonParser((event) => events.push(event as never));
|
||||
parser.finish(await response.text());
|
||||
const parser = createNdjsonParser((event) => {
|
||||
events.push(event as never);
|
||||
if (event.type === "answer.delta") options.onAnswerDelta?.(event.text);
|
||||
});
|
||||
if (options.onAnswerDelta) {
|
||||
const reader = response.body!.pipeThrough(new TextDecoderStream()).getReader();
|
||||
for (let chunk = await reader.read(); !chunk.done; chunk = await reader.read()) {
|
||||
parser.push(chunk.value);
|
||||
}
|
||||
parser.finish();
|
||||
} else {
|
||||
parser.finish(await response.text());
|
||||
}
|
||||
const answer = events.filter((event) => event.type === "answer.delta").map((event) => event.text ?? "").join("");
|
||||
const terminal = events.filter((event) => event.type === "run.completed" || event.type === "run.failed");
|
||||
return {
|
||||
|
||||
@@ -20,7 +20,7 @@ Profiles record their role separately from their identifier. Mudda records the l
|
||||
|
||||
## Validation and release manifest
|
||||
|
||||
Run `python -m scripts.preset_evidence.library --her-export <source-directory>` to check the schema, receipt, source hashes, normalized excerpts, and the bound repository functions and tests. For the source directory, use an exact Git checkout or an export with source revision and file hashes from that checkout. A plain export of the same pinned files can be verified against the previously generated manifest. Add `--write-verified` to publish `VERIFIED.json` after a full check. `--skip-her-export` runs development checks and cannot write the manifest.
|
||||
Run `python -m scripts.preset_evidence.library --her-export <source-directory>` to check the schema, receipt, source hashes, normalized excerpts, and the bound repository functions and tests. Pass either `--her-export` or `--skip-her-export` explicitly. Use a Git checkout at the source revision pinned in `VERIFIED.json` or a plain export with the same source-file hashes. For source-backed pytest checks, set `PRESET_EVIDENCE_HER_EXPORT` to that directory. The checks skip when the variable is unset and fail when the supplied checkout or export does not match the pin. Add `--write-verified` to publish `VERIFIED.json` after a full check. `--skip-her-export` runs development checks and cannot write the manifest.
|
||||
|
||||
`VERIFIED.json` records the source revision, hashes of every resolved source dependency, validator and matcher versions, and generation time. The library content hash covers the bytes of `schema.json` and every entry file in sorted path order. Git keeps their LF line endings across platforms. The receipt has its own pinned hash. An absolute path, parent traversal, or symlink outside its declared root fails validation.
|
||||
|
||||
|
||||
@@ -5,7 +5,6 @@ import argparse
|
||||
import ast
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import tempfile
|
||||
@@ -18,7 +17,6 @@ ROOT = Path(__file__).resolve().parents[2]
|
||||
LIBRARY_DIR = ROOT / "references" / "preset_evidence_library"
|
||||
ENTRIES_DIR = LIBRARY_DIR / "entries"
|
||||
SCHEMA_PATH = LIBRARY_DIR / "schema.json"
|
||||
DEFAULT_HER_EXPORT = Path(os.environ.get("PRESET_EVIDENCE_HER_EXPORT", "/workspace/up-run-r40"))
|
||||
RECEIPT_PATH = "rulings/review_receipt_20261112.json"
|
||||
RECEIPT_SHA256 = "426c456f99848c14e0ffd20a8657f7251c24cb9ae555d77a64ddeef402ea801d"
|
||||
VALIDATOR_VERSION = "preset-library-validator.v2"
|
||||
@@ -422,7 +420,7 @@ def _validate_snapshot(raw: dict[str, bytes], receipt: dict, files: _FileIndex |
|
||||
return entries, errors
|
||||
|
||||
|
||||
def validate(entries_dir: Path = ENTRIES_DIR, her_export: Optional[Path] = DEFAULT_HER_EXPORT,
|
||||
def validate(entries_dir: Path = ENTRIES_DIR, her_export: Optional[Path] = None,
|
||||
our_root: Path = ROOT, *, library_dir: Path = LIBRARY_DIR) -> List[str]:
|
||||
"""Collect schema and policy errors from every category."""
|
||||
try:
|
||||
@@ -617,19 +615,20 @@ def summary(entries: List[Dict[str, Any]]) -> Dict[str, Any]:
|
||||
|
||||
def main(argv: Optional[List[str]] = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--her-export", type=Path)
|
||||
parser.add_argument("--skip-her-export", action="store_true", help="Development checks only")
|
||||
source = parser.add_mutually_exclusive_group(required=True)
|
||||
source.add_argument("--her-export", type=Path, help="Checkout or export of the source revision pinned in VERIFIED.json")
|
||||
source.add_argument("--skip-her-export", action="store_true", help="Development checks only")
|
||||
parser.add_argument("--write-verified", action="store_true")
|
||||
parser.add_argument("--json", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
if args.write_verified and (args.skip_her_export or args.her_export is None):
|
||||
if args.write_verified and args.skip_her_export:
|
||||
parser.error("--write-verified requires an explicit --her-export and full validation")
|
||||
entries = []
|
||||
errors = []
|
||||
try:
|
||||
raw = _library_bytes(ENTRIES_DIR, LIBRARY_DIR)
|
||||
receipt = _read_receipt(LIBRARY_DIR)
|
||||
files = None if args.skip_her_export else _FileIndex(args.her_export or DEFAULT_HER_EXPORT)
|
||||
files = None if args.skip_her_export else _FileIndex(args.her_export)
|
||||
entries, errors = _validate_snapshot(raw, receipt, files, ROOT)
|
||||
if files is not None and not errors:
|
||||
source_revision = _validate_source_pin(files, receipt)
|
||||
|
||||
@@ -3,6 +3,8 @@ from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
@@ -11,9 +13,34 @@ from scripts.preset_evidence import library as lib
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
HER = lib.DEFAULT_HER_EXPORT
|
||||
needs_export = pytest.mark.skipif(not (HER / "references" / "yoga_rules.json").is_file(),
|
||||
reason=f"her export not present at {HER}")
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def her_export() -> Path:
|
||||
value = os.environ.get("PRESET_EVIDENCE_HER_EXPORT")
|
||||
if not value:
|
||||
pytest.skip("Set PRESET_EVIDENCE_HER_EXPORT to check the pinned source export.")
|
||||
|
||||
verified = json.loads((lib.LIBRARY_DIR / "VERIFIED.json").read_bytes())
|
||||
mismatch = "来源与 VERIFIED.json 固定的版本不一致"
|
||||
try:
|
||||
directory = Path(value).resolve()
|
||||
if (directory / ".git").exists():
|
||||
git = ["git", "-c", f"safe.directory={directory.as_posix()}", "-C", str(directory)]
|
||||
subprocess.check_output([*git, "status", "-sb"], stderr=subprocess.PIPE, timeout=20)
|
||||
root, revision = subprocess.check_output(
|
||||
[*git, "rev-parse", "--show-toplevel", "HEAD"], stderr=subprocess.PIPE, timeout=20,
|
||||
).decode("utf-8").splitlines()
|
||||
matches = Path(root).resolve() == directory and revision == verified["source_repo_commit"]
|
||||
else:
|
||||
files = lib._FileIndex(directory)
|
||||
for relative in verified["source_files"]:
|
||||
files.read(relative)
|
||||
matches = files.hashes() == verified["source_files"]
|
||||
except (OSError, ValueError, RuntimeError, subprocess.SubprocessError):
|
||||
matches = False
|
||||
if not matches:
|
||||
pytest.fail(mismatch, pytrace=False)
|
||||
return directory
|
||||
|
||||
|
||||
def _entry(entry_id: str) -> dict:
|
||||
@@ -40,9 +67,8 @@ def test_policy_holds_without_export():
|
||||
assert lib.validate(her_export=None) == []
|
||||
|
||||
|
||||
@needs_export
|
||||
def test_every_her_file_exists_sha_matches_and_quotes_are_verbatim():
|
||||
assert lib.validate(her_export=HER) == []
|
||||
def test_every_her_file_exists_sha_matches_and_quotes_are_verbatim(her_export):
|
||||
assert lib.validate(her_export=her_export) == []
|
||||
|
||||
|
||||
def test_ids_unique_and_seed_counts():
|
||||
@@ -98,14 +124,14 @@ def test_missing_field_needs_a_note_and_missing_source_cannot_be_admitted():
|
||||
assert any("requires a located source" in e for e in lib.entry_policy_errors(entry))
|
||||
|
||||
|
||||
@needs_export
|
||||
def test_changed_her_file_is_caught(tmp_path):
|
||||
def test_changed_her_file_is_caught(her_export):
|
||||
files = lib._FileIndex(her_export)
|
||||
entry = _entry("functional.pvr_table30")
|
||||
entry["source"]["sha256"] = "0" * 64
|
||||
assert any("sha256 mismatch" in e for e in lib.entry_policy_errors(entry, lib._FileIndex(HER)))
|
||||
assert any("sha256 mismatch" in e for e in lib.entry_policy_errors(entry, files))
|
||||
entry = _entry("functional.pvr_table30")
|
||||
entry["source"]["quote_excerpt"] = "|Vi|—|Mercury,Venus,Saturn|"
|
||||
assert any("quote not found" in e for e in lib.entry_policy_errors(entry, lib._FileIndex(HER)))
|
||||
assert any("quote not found" in e for e in lib.entry_policy_errors(entry, files))
|
||||
|
||||
|
||||
def test_open_shadbala_parameters_and_harsha_stay_blocked():
|
||||
|
||||
@@ -226,6 +226,14 @@ def test_fingerprint_includes_facts_domain_and_branch():
|
||||
assert len({row["match_fingerprint"] for row in (first, other_domain, other_branch, other_facts)}) == 4
|
||||
|
||||
|
||||
def test_cli_requires_an_explicit_source_option(capsys):
|
||||
with pytest.raises(SystemExit) as error:
|
||||
lib.main([])
|
||||
assert error.value.code == 2
|
||||
diagnostic = capsys.readouterr().err
|
||||
assert "--her-export" in diagnostic and "--skip-her-export" in diagnostic
|
||||
|
||||
|
||||
def test_skipping_export_cannot_write_a_manifest():
|
||||
manifest = lib.LIBRARY_DIR / "VERIFIED.json"
|
||||
before = manifest.read_bytes()
|
||||
|
||||
Reference in New Issue
Block a user