Merge BUG-1504/BUG-1508 VedAstro report deadline
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DbBXoyeZrYn6ryNcSe5Y2Y # Conflicts: # CHANGELOG.md # docs/operations/outage-behaviour-2026-11-01.md
This commit is contained in:
@@ -18,6 +18,16 @@
|
||||
- 改日期查到新时区后又改回原日期,会重新查一次,不再沿用刚才那个日期的时区。
|
||||
- 记为 BUG-1506。Skill 版本不变。
|
||||
|
||||
## 2026-11-02 — VedAstro 卡住或断线时,报告不再失败
|
||||
|
||||
- 生成报告时,VedAstro 最多等 25 秒(从开始计算时算起,和本地计算同时进行),前端的等待是 90 秒。等不到就照常出报告,报告里的 VedAstro 一栏标「官方外部引擎本次未验证或不可用」,不会写成已验证。
|
||||
- 这次失败会记到当天:同一张盘当天再生成报告、或者同一份报告的下一个主题,直接用这条记录,不再等 VedAstro。之后 VedAstro 恢复并返回结果,就改用新结果。
|
||||
- 报告不再先跑一遍 VedAstro 主入口概览(按主题的区间扫描)。官方 VedAstro 证据仍然有,来自官方快照,按时返回就照常写进报告。
|
||||
- VedAstro 返回到一半连接断开(读响应时连接被重置或截断)时,按网络错误处理:报告照常出、VedAstro 标未验证;`/api/chart` 不带跳过概览的参数时也返回 200,不再报 500。
|
||||
- 对话、星盘页的行为不变。
|
||||
- 可调参数:`JYOTISH_REPORT_VEDASTRO_DEADLINE_SECONDS`(默认 25,最多 45)。
|
||||
- Skill 版本不变(Gitea #119 第 42 项缺口 2、3;BUG-1508、BUG-1504)。
|
||||
|
||||
|
||||
## 2026-11-01 — 年运报告:从「我的报告」直接打开,写好一节出一节
|
||||
|
||||
|
||||
+34
-9
@@ -20573,19 +20573,22 @@
|
||||
|
||||
## BUG-1504 | VedAstro 响应读到一半连接断开时,报告生成与不带 skip 的排盘返回 500
|
||||
|
||||
- 状态:investigating
|
||||
- 状态:resolved
|
||||
- 首次发现:2026-11-01(Gitea #119 第 42 项,外部依赖故障审计)
|
||||
- 最近更新:2026-11-01
|
||||
- 最近更新:2026-11-02
|
||||
- 影响面:同步调 VedAstro 官方快照的路径,包括个人报告生成(排盘主入口概览、生时门禁)和不带 `skip_vedastro_main_entry_overview` 的 `/api/chart`。星盘页、对话不受影响:前者总是跳过,后者在后台线程里调用。
|
||||
- 用户现象:报告以 `workflow_server_error` 失败(会退点);接口返回 `ERR_INTERNAL`。
|
||||
- 触发条件:VedAstro 已经返回响应头,但响应体读到一半连接被重置或截断(`http.client.IncompleteRead`、`ConnectionResetError`、`ssl.SSLError`)。
|
||||
- 根因(读码 + 本机 stub 确认 IncompleteRead):`scripts/vedastro_service_adapter.py` 的快照分发(约 2782–2808 行)和区间扫描(约 3010–3046 行)只接住 `HTTPError`、`URLError`、`RemoteDisconnected`、超时和 `JSONDecodeError`;`_attach_vedastro_main_entry_overview` 外面也没有 try,所以异常一直冒到请求总入口。
|
||||
- 修复:未修。本轮只审计,修法待产品决定是否排期(把这几类读错误归为 `network_error`)。
|
||||
- 验证:本机把 `urlopen` 换成读响应时抛 `IncompleteRead` 的 stub,调用排盘得到 500。脚本是临时的,没有入库;做法见 `docs/operations/outage-behaviour-2026-11-01.md` 第一节。
|
||||
- 防复发:待修复单补「读错误降级、不 500」的测试。
|
||||
- 相关记录:Gitea #119 第 42 项;`docs/operations/outage-behaviour-2026-11-01.md`。
|
||||
- 触发条件:VedAstro 已经返回响应头,但响应体读到一半连接被重置或截断(`http.client.IncompleteRead`、`ConnectionResetError`、`ssl.SSLError`)。读响应头时连接被重置也一样。
|
||||
- 根因(读码 + 本机 stub 确认 IncompleteRead):`scripts/vedastro_service_adapter.py` 的快照分发(约 2782–2808 行)和区间扫描(约 3010–3046 行)只接住 `HTTPError`、`URLError`、`RemoteDisconnected`、超时和 `JSONDecodeError`;`urllib` 只包装发请求时的错误,读响应头和响应体时的错误原样抛出。`_attach_vedastro_main_entry_overview` 外面也没有 try,所以异常一直冒到请求总入口。
|
||||
- 修复:
|
||||
- 适配器的 `_post_json` 把读响应时的 `http.client.HTTPException`(含 `IncompleteRead`、`RemoteDisconnected`)、`ConnectionError`(含 `ConnectionResetError`)和 `ssl.SSLError` 转成 `URLError("response_read_failed:<类型名>")`。所有调用点本来就把 `URLError` 记成 `network_error`,所以快照分节、区间扫描、外部技法证据都按网络错误处理,官方快照不再是 ok,网关结果是 `official_blocked`。前台和报告会话照原样把这个失败记进当天的失败缓存。读超时仍是 `timeout`,5xx 仍是 `http_error:<code>`。`RemoteDisconnected` 的标签从 `network_error:<原文>` 变成 `network_error:response_read_failed:RemoteDisconnected`。
|
||||
- `_attach_vedastro_main_entry_overview` 外面包一层 try:任何异常都记成 `status=blocked`、`reason=vedastro_main_entry_overview_error`、`error_type=<类型名>`,排盘照常返回。
|
||||
- `vedastro_rest_bridge.probe_calculate_health` 也接住 `http.client.HTTPException`,健康探测读到一半断开时记 `official_blocked`。
|
||||
- 验证:`tests/test_vedastro_report_deadline_20261102.py`:把 `urlopen` 换成 stub,分别在读响应体时抛 `IncompleteRead`、`ConnectionResetError`、`ssl.SSLError`,在打开连接时抛 `ConnectionResetError`、`RemoteDisconnected`。`_post_json` 抛 `URLError`;官方快照分节是 `network_error:response_read_failed:<类型名>`,没有 ok;区间扫描是 `network_error`。`/api/chart`(不带 skip)走真实 `do_POST` 返回 200,`vedastro_range_scan_result` 不是 ok;编排器直接抛 `IncompleteRead` 时也返回 200,概览为 blocked。503 仍是 `http_error:503`,读超时仍是 `timeout`。把适配器和 REST 桥换回修复前版本,这几条测试失败。
|
||||
- 防复发:该测试文件已加进 `scripts/run_quality_gate.py` 的 quick 清单。新增的 VedAstro 网络调用应当走 `_post_json`,不要自己调 `urlopen`。
|
||||
- 相关记录:BUG-1508(同一轮修复,报告等 VedAstro 的总时限);Gitea #119 第 42 项;`docs/operations/outage-behaviour-2026-11-01.md`。
|
||||
- 复发自:无
|
||||
- 修复版本:未修。
|
||||
- 修复版本:分支 `codex/vedastro-deadline-1504`。
|
||||
|
||||
## BUG-1505 | 报告任务最后一次尝试租约过期时,报告标为失败但点数不退
|
||||
|
||||
@@ -20645,3 +20648,25 @@
|
||||
- 相关记录:BUG-1506;`docs/operations/outage-behaviour-2026-11-01.md` 第二节。新疆 `Asia/Urumqi`(+6)对报北京时间的用户差两小时,另待产品决定,没有单独记 Bug。
|
||||
- 复发自:无
|
||||
- 修复版本:未修。
|
||||
|
||||
## BUG-1508 | 报告生成同步等 VedAstro,没有总时限,VedAstro 卡住时报告超时失败
|
||||
|
||||
- 状态:resolved
|
||||
- 首次发现:2026-11-01(Gitea #119 第 42 项,外部依赖故障审计,缺口 2)
|
||||
- 最近更新:2026-11-02
|
||||
- 影响面:个人报告生成(`/api/reports` 创建和报告 worker 调 `/api/consultation_workflow`)。对话、星盘页不受影响。
|
||||
- 用户现象:VedAstro 卡住时报告以 `workflow_timeout` 失败(会退点)。期间一直占着进程级两个重计算名额中的一个,并发两份报告会让对话和校正拿到 429。下一份报告还要再等一遍。
|
||||
- 触发条件:VedAstro 不响应(每次请求都等满 `VEDASTRO_TIMEOUT_SECONDS`,生产 20 秒)时生成报告。
|
||||
- 根因(读码 + 本机 stub 确认):报告调用没有传 `foreground`,所以 `defer_optional_external_evidence=false`,Python 端同步跑两段 VedAstro:排盘里的主入口概览(约 4 × 20 秒)和生时门禁里的官方快照(含 8 秒健康探测,约 48 秒)。合计约 128 秒,超过前端 90 秒(`frontend/src/mastra/consultation-workflow.ts:149`)。这条路径的结果直接取自门禁,不经过前台会话,也就不写失败缓存。
|
||||
- 修复:复用对话已有的前台会话、后台线程和当天失败缓存(#130),不另起一套。
|
||||
- 前端:报告的两处调用(`frontend/src/app/api/reports/route.ts`、`frontend/src/lib/personal-report-worker.ts`)传 `vedastroEvidenceBudget: "report"`,请求体带 `vedastro_evidence_budget: "report"`。报告仍不传 `foreground`。
|
||||
- Python:`scripts/vedastro_foreground.py` 新增报告档。带这个字段的请求按 defer 处理,和对话一样:不跑同步的主入口概览;生时门禁只读缓存;官方网关在后台线程里跑,和本地计算并行。不同的是等待时长:对话从收尾时起最多等 1.5 秒;报告从会话开始起算,最多等 25 秒(`JYOTISH_REPORT_VEDASTRO_DEADLINE_SECONDS`,上限 45 秒)。单次请求超时沿用适配器的配置值,报告不会把它调高,也不超过 2 倍时限。
|
||||
- 到时限网关还在跑:报告照常继续,VedAstro 记 `official_blocked` / `report_vedastro_deadline_exceeded`,并写进这张盘当天(UTC)的失败缓存,下一份报告和下一个主题直接用它,不再等。后台线程继续跑:之后成功就写进正常快照缓存(查缓存时成功优先),失败就用真实失败覆盖这条记录。还没开始就被取消的任务不记。
|
||||
- 5xx、连接截断等很快返回的失败照旧记进失败缓存。
|
||||
- 报告里的证据如实标出:`machine_evidence_packet` 的 `external_oracle_status` 是 `official_blocked`,`vedastro_official_raw_response` 是 `missing`,报告的技法审计表按原有规则显示 blocked(「官方外部引擎本次未验证或不可用」)。对话技法审计的说明文字加了 `report_vedastro_deadline_exceeded` 的中文。
|
||||
- 取舍:报告不再跑主入口概览(按主题的区间扫描),引导话题里的 VedAstro 区间事件显示为没有;官方 VedAstro 证据改由网关的官方快照提供,按时返回就照常进报告。
|
||||
- 验证:`tests/test_vedastro_report_deadline_20261102.py`,VedAstro 全部是 stub:网关卡住、时限 0.4 秒时,会话 0.4 秒左右返回 blocked,失败写进缓存,第二次 0.15 秒内返回、网关只调过一次;时限从会话开始起算,本地计算占掉的时间不再额外等;到时限后网关成功,下一次读到成功快照;很快返回的 503 类失败和抛 `IncompleteRead`、`ConnectionResetError` 的网关都记成 blocked 并写缓存;健康时照常用网关结果;对话档的 1.5 秒超时仍不记;单次请求超时沿用配置值。整条 `consultation_workflow` 带报告字段时不调主入口概览,证据段落为 `official_blocked` / `missing`,第二次不再调网关。本机脚本把时限设为 6 秒、单次超时 20 秒、VedAstro 全部卡住:第一份 6.23 秒,第二份 0.11 秒(修复前同样条件约 128 秒)。脚本是临时的,没有入库。
|
||||
- 防复发:该测试文件已加进 `scripts/run_quality_gate.py` 的 quick 清单;`frontend/tests/consultation-workflow-contract.test.ts` 锁住报告两处调用都带 `vedastroEvidenceBudget: "report"`、都不带 `foreground: true`。
|
||||
- 相关记录:BUG-1504;BUG-161、BUG-727、#130(前台会话和失败缓存);`docs/operations/outage-behaviour-2026-11-01.md` 第一节、第五节第 2 条。
|
||||
- 复发自:无
|
||||
- 修复版本:分支 `codex/vedastro-deadline-1504`。
|
||||
|
||||
@@ -113,9 +113,9 @@ Gitea #119 第 42 项:「VedAstro、地点服务、模型供应商挂掉时网
|
||||
|
||||
## 五、缺口(按风险从高到低)
|
||||
|
||||
1. **报告任务租约过期不退点(BUG-1505)。** 进程在最后一次尝试时重启就会触发,发版时尤其容易遇到。点数扣掉了、报告失败,用户要么删报告,要么重新付一次。退点 RPC 失败也被忽略。
|
||||
2. **报告生成同步等 VedAstro,没有总时限。** VedAstro 卡住时约 128 秒,超过前端 90 秒,报告失败;同时占着两个重计算名额中的一个,并发两份报告会让对话和校正拿到 429。这条路径的失败也不写失败缓存,下一份报告会重来一遍。可选做法:报告也走 `defer_optional_external_evidence`,或者跳过主入口概览,或者复用对话的前台预算和失败缓存。
|
||||
3. **VedAstro 连接被截断会变成 500(BUG-1504)。** 报告失败(会退点),不带 skip 的 `/api/chart` 也会 500。
|
||||
1. **报告任务租约过期不退点(BUG-1505)。** 进程在最后一次尝试时重启就会触发,发版时尤其容易遇到。点数扣掉了、报告失败,用户要么删报告,要么重新付一次。退点 RPC 失败也被忽略。**2026-11-02 已修**:租约回收在判失败前先退点(与删除触发器、worker 同一个请求号,不会重复退);退点失败的任务留到下次回收重试,并记 `errorCode=report_refund_failed`。
|
||||
2. **报告生成同步等 VedAstro,没有总时限。** VedAstro 卡住时约 128 秒,超过前端 90 秒,报告失败;同时占着两个重计算名额中的一个,并发两份报告会让对话和校正拿到 429。这条路径的失败也不写失败缓存,下一份报告会重来一遍。可选做法:报告也走 `defer_optional_external_evidence`,或者跳过主入口概览,或者复用对话的前台预算和失败缓存。**2026-11-02 已修(BUG-1508)**:报告复用对话的前台会话和失败缓存,VedAstro 最多等 25 秒,到时限标 blocked 并记当天。
|
||||
3. **VedAstro 连接被截断会变成 500(BUG-1504)。** 报告失败(会退点),不带 skip 的 `/api/chart` 也会 500。**2026-11-02 已修**:读响应时的截断和重置按网络错误处理,主入口概览出错时排盘照常返回。
|
||||
4. **出生资料编辑时,时区服务失败会默默清掉出生地(BUG-1506)。**2026-11-02 已修。
|
||||
5. **早期中国资料默默按 +8 计算(BUG-1507)。** 这是正确性问题,与故障无关:1986–1991 年夏季出生的差一小时。新疆 +6 要产品决定。
|
||||
6. **模型调用不重试、没有备用供应商。** 一次 429 或 503 就让付费用户这一轮失败。会退点,但体验差;目录里配了多个模型,失败时也不切换。
|
||||
|
||||
@@ -381,7 +381,7 @@ export async function POST(request: Request) {
|
||||
return released.success;
|
||||
},
|
||||
},
|
||||
runWorkflow: (input) => runConsultationWorkflow(input),
|
||||
runWorkflow: (input) => runConsultationWorkflow(input, { vedastroEvidenceBudget: "report" }),
|
||||
createAgent: (model) => createPersonalReportAgent(model as Parameters<typeof createPersonalReportAgent>[0]),
|
||||
skillSnapshot: resolveSkillSnapshot(),
|
||||
jobs: createSupabasePersonalReportJobService(admin),
|
||||
|
||||
@@ -71,6 +71,7 @@ const NOTE_LABELS: Readonly<Record<string, string>> = {
|
||||
official_raw_archive_is_auditable: "官方原始档案可审计",
|
||||
official_cloud_evidence_available: "官方云证据可用",
|
||||
foreground_optional_evidence_timeout: "前台官方云证据超时,本轮未赶上",
|
||||
report_vedastro_deadline_exceeded: "报告等待官方云证据超过时限,本次未采用",
|
||||
gateway_invocation_error: "官方云网关调用失败",
|
||||
claims_capped_until_pyjhora_jhora_jyotishganit_are_invoked_for_this_run: "结论上限受外部引擎调用限制",
|
||||
three_engine_runtime_closure_available: "三引擎运行时已闭环",
|
||||
|
||||
@@ -134,7 +134,7 @@ async function generateWriterReport(
|
||||
try {
|
||||
workflows.push({
|
||||
theme: rawTheme,
|
||||
workflow: await runConsultationWorkflow(input, { signal: context.signal }),
|
||||
workflow: await runConsultationWorkflow(input, { signal: context.signal, vedastroEvidenceBudget: "report" }),
|
||||
});
|
||||
} catch (error) {
|
||||
if (context.signal.aborted) throw error;
|
||||
|
||||
@@ -137,6 +137,12 @@ export async function runConsultationWorkflow(
|
||||
input: ConsultationInput,
|
||||
options?: {
|
||||
foreground?: boolean;
|
||||
/**
|
||||
* "report": the API waits for VedAstro only up to its report deadline
|
||||
* (well inside the 90 s below), then marks it blocked and remembers the
|
||||
* failure for the day. BUG-1508.
|
||||
*/
|
||||
vedastroEvidenceBudget?: "report";
|
||||
signal?: AbortSignal;
|
||||
plan?: ConsultationPlan;
|
||||
requestId?: string;
|
||||
@@ -161,6 +167,7 @@ export async function runConsultationWorkflow(
|
||||
question_text: workflowRequest.question,
|
||||
theme: workflowRequest.themes,
|
||||
defer_optional_external_evidence: options?.foreground === true,
|
||||
...(options?.vedastroEvidenceBudget ? { vedastro_evidence_budget: options.vedastroEvidenceBudget } : {}),
|
||||
plan_version: workflowRequest.plan_version,
|
||||
strict_workflow_route: workflowRequest.strictWorkflowRoute,
|
||||
required_layers: workflowRequest.requiredLayers,
|
||||
|
||||
@@ -12,6 +12,7 @@ const mastra = readFileSync(new URL("../src/mastra/index.ts", import.meta.url),
|
||||
const tools = readFileSync(new URL("../src/mastra/consultation-tools.ts", import.meta.url), "utf8");
|
||||
const stream = readFileSync(new URL("../src/lib/stream-agent-response.ts", import.meta.url), "utf8");
|
||||
const workflow = readFileSync(new URL("../src/mastra/consultation-workflow.ts", import.meta.url), "utf8");
|
||||
const reportWorker = readFileSync(new URL("../src/lib/personal-report-worker.ts", import.meta.url), "utf8");
|
||||
const plan = readFileSync(new URL("../src/lib/consultation-plan.ts", import.meta.url), "utf8");
|
||||
const routeService = readFileSync(new URL("../src/lib/consultation-route-service.ts", import.meta.url), "utf8");
|
||||
const pythonPlanContract = readFileSync(new URL("../../scripts/consultation_plan_contract.py", import.meta.url), "utf8");
|
||||
@@ -339,8 +340,14 @@ test("personal consultation lets the Agent invoke the server-bound workflow tool
|
||||
|
||||
test("defers optional external evidence only for foreground chat", () => {
|
||||
assert.match(workflow, /defer_optional_external_evidence: options\?\.foreground === true/);
|
||||
assert.match(reportsRoute, /runWorkflow: \(input\) => runConsultationWorkflow\(input\)/);
|
||||
// 原值: /runWorkflow: \(input\) => runConsultationWorkflow\(input\)/
|
||||
// 新值: 报告带 vedastroEvidenceBudget: "report",仍不是前台对话(不带 foreground)
|
||||
// 原因: BUG-1508,报告等 VedAstro 要有总时限,超时标 blocked 并记当天失败
|
||||
assert.match(reportsRoute, /runWorkflow: \(input\) => runConsultationWorkflow\(input, \{ vedastroEvidenceBudget: "report" \}\)/);
|
||||
assert.doesNotMatch(reportsRoute, /foreground:\s*true/);
|
||||
assert.match(workflow, /vedastro_evidence_budget: options\.vedastroEvidenceBudget/);
|
||||
assert.match(reportWorker, /runConsultationWorkflow\(input, \{ signal: context\.signal, vedastroEvidenceBudget: "report" \}\)/);
|
||||
assert.doesNotMatch(reportWorker, /foreground:\s*true/);
|
||||
});
|
||||
|
||||
test("personal Agent owns the Skill and context-bound calculation tool", () => {
|
||||
|
||||
@@ -60,6 +60,7 @@ try:
|
||||
_foreground_vedastro_join_seconds,
|
||||
_join_foreground_vedastro,
|
||||
_run_foreground_vedastro_gateway,
|
||||
defers_optional_vedastro,
|
||||
finish_foreground_vedastro,
|
||||
should_include_western_evidence_packet,
|
||||
start_foreground_vedastro,
|
||||
@@ -83,6 +84,7 @@ except ModuleNotFoundError: # pragma: no cover - script execution path
|
||||
_foreground_vedastro_join_seconds,
|
||||
_join_foreground_vedastro,
|
||||
_run_foreground_vedastro_gateway,
|
||||
defers_optional_vedastro,
|
||||
finish_foreground_vedastro,
|
||||
should_include_western_evidence_packet,
|
||||
start_foreground_vedastro,
|
||||
@@ -2151,7 +2153,8 @@ def execute_consultation_workflow(
|
||||
question = body.get('question') or ''
|
||||
entry_mode = body.get('entry_mode', 'direct_chart')
|
||||
high_rigor = bool(body.get('return_high_rigor_shape'))
|
||||
defer_optional_external_evidence = bool(body.get('defer_optional_external_evidence'))
|
||||
# A report's VedAstro budget defers like consult, with a longer deadline (BUG-1508).
|
||||
defer_optional_external_evidence = defers_optional_vedastro(body)
|
||||
try:
|
||||
from scripts.three_engine_parity_replay_validator import validate_manifest
|
||||
except ModuleNotFoundError: # pragma: no cover - direct script execution
|
||||
@@ -2901,6 +2904,21 @@ def _attach_vedastro_main_entry_overview(chart_result, birth_payload):
|
||||
or birth_payload.get('today')
|
||||
or datetime.utcnow().strftime('%Y-%m-%d')
|
||||
)[:10]
|
||||
try:
|
||||
return _attach_vedastro_main_entry_overview_from(orchestrator, priority, chart_result, modules, birth_payload, reference_date)
|
||||
except Exception as exc: # BUG-1504: the overview is optional evidence; never a 500.
|
||||
modules['vedastro_range_scan_result'] = {
|
||||
'backend': 'vedastro_service_adapter_candidate',
|
||||
'available': False,
|
||||
'status': 'blocked',
|
||||
'reason': 'vedastro_main_entry_overview_error',
|
||||
'error_type': type(exc).__name__,
|
||||
'source_metadata': {'ingestion_profile': 'main_entry_overview', 'reference_date': reference_date},
|
||||
}
|
||||
return chart_result
|
||||
|
||||
|
||||
def _attach_vedastro_main_entry_overview_from(orchestrator, priority, chart_result, modules, birth_payload, reference_date):
|
||||
vedastro_evidence = orchestrator.orchestrate_vedastro_evidence({
|
||||
'year': birth_payload.get('year'),
|
||||
'month': birth_payload.get('month'),
|
||||
|
||||
@@ -139,6 +139,9 @@ CORE_PYTEST_TARGETS = [
|
||||
"tests/test_report_english_sweep_smoke.py",
|
||||
# /api/shadbala with the birth moment over real HTTP, plus ruff F821 on the API server (BUG-1463).
|
||||
"tests/test_shadbala_service_api.py",
|
||||
# Report waits on VedAstro only to its deadline, then blocked + remembered (BUG-1508);
|
||||
# a cut VedAstro response is a network error, /api/chart stays 200 (BUG-1504).
|
||||
"tests/test_vedastro_report_deadline_20261102.py",
|
||||
]
|
||||
|
||||
RUNTIME_TRUTH_PYTEST_TARGETS = [
|
||||
|
||||
@@ -19,11 +19,31 @@ budget <= k * join (k=2) keeps leftover occupancy of an already-running
|
||||
worker from stretching later requests past the join cap. The previous
|
||||
default budget of 8s with a 1.5s join left cancelled-too-late workers
|
||||
holding the two-thread pool for the rest of the 8s.
|
||||
|
||||
Report profile (body ``vedastro_evidence_budget="report"``, gap 2 of
|
||||
docs/operations/outage-behaviour-2026-11-01.md, BUG-1508). A report used to
|
||||
wait synchronously on the main-entry overview and the rectification gate,
|
||||
about 128s when VedAstro hangs, past the frontend's 90s. A report now runs
|
||||
the same deferred session as consultation, with a longer wait so a healthy
|
||||
VedAstro still lands in the report:
|
||||
|
||||
deadline: measured from session start, so it overlaps local compute
|
||||
(default 25s, cap 45s).
|
||||
budget: the per-request timeout already configured for the adapter, never
|
||||
more than k * deadline.
|
||||
|
||||
When the deadline passes while the gateway is still running, the report
|
||||
continues with VedAstro marked ``official_blocked`` /
|
||||
``report_vedastro_deadline_exceeded`` and that is remembered for the chart
|
||||
and UTC day, so the next report answers at once. The worker keeps running;
|
||||
if it later succeeds the positive snapshot wins, if it fails its own failure
|
||||
replaces the deadline entry.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
|
||||
from typing import Any
|
||||
|
||||
@@ -65,6 +85,13 @@ FOREGROUND_VEDASTRO_BUDGET_JOIN_RATIO = 2.0
|
||||
FOREGROUND_VEDASTRO_WORKERS_DEFAULT = 2
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
# --- report profile: deadline / budget (change together, see docstring) ---
|
||||
REPORT_VEDASTRO_EVIDENCE_BUDGET = "report"
|
||||
REPORT_VEDASTRO_DEADLINE_SECONDS_DEFAULT = 25.0
|
||||
REPORT_VEDASTRO_DEADLINE_SECONDS_MAX = 45.0
|
||||
REPORT_VEDASTRO_DEADLINE_EXCEEDED = "report_vedastro_deadline_exceeded"
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
_FOREGROUND_VEDASTRO_WORKERS = max(
|
||||
int(os.environ.get("JYOTISH_FOREGROUND_VEDASTRO_WORKERS", str(FOREGROUND_VEDASTRO_WORKERS_DEFAULT))),
|
||||
1,
|
||||
@@ -98,6 +125,35 @@ def _foreground_vedastro_budget_seconds() -> float:
|
||||
return min(budget, cap)
|
||||
|
||||
|
||||
def is_report_vedastro_budget(body: dict | None) -> bool:
|
||||
payload = body if isinstance(body, dict) else {}
|
||||
return str(payload.get("vedastro_evidence_budget") or "").strip().lower() == REPORT_VEDASTRO_EVIDENCE_BUDGET
|
||||
|
||||
|
||||
def defers_optional_vedastro(body: dict | None) -> bool:
|
||||
"""Consult asks with the flag; a report budget implies it (BUG-1508)."""
|
||||
payload = body if isinstance(body, dict) else {}
|
||||
return bool(payload.get("defer_optional_external_evidence")) or is_report_vedastro_budget(payload)
|
||||
|
||||
|
||||
def _report_vedastro_deadline_seconds() -> float:
|
||||
raw = str(os.environ.get("JYOTISH_REPORT_VEDASTRO_DEADLINE_SECONDS", str(REPORT_VEDASTRO_DEADLINE_SECONDS_DEFAULT))).strip()
|
||||
try:
|
||||
return min(max(float(raw), 0.0), REPORT_VEDASTRO_DEADLINE_SECONDS_MAX)
|
||||
except ValueError:
|
||||
return REPORT_VEDASTRO_DEADLINE_SECONDS_DEFAULT
|
||||
|
||||
|
||||
def _report_vedastro_budget_seconds() -> float:
|
||||
"""The adapter's own per-request timeout; a report never raises it."""
|
||||
try:
|
||||
from scripts.vedastro_service_adapter import _timeout_seconds as configured_timeout
|
||||
except ModuleNotFoundError: # pragma: no cover - script execution path
|
||||
from vedastro_service_adapter import _timeout_seconds as configured_timeout
|
||||
cap = FOREGROUND_VEDASTRO_BUDGET_JOIN_RATIO * _report_vedastro_deadline_seconds()
|
||||
return min(float(configured_timeout()), cap)
|
||||
|
||||
|
||||
def _blocked_foreground_vedastro(*, reason: str, error_type: str | None = None) -> dict:
|
||||
packet = {
|
||||
"scope": "vedastro_gateway_run",
|
||||
@@ -110,11 +166,18 @@ def _blocked_foreground_vedastro(*, reason: str, error_type: str | None = None)
|
||||
return packet
|
||||
|
||||
|
||||
def _run_foreground_vedastro_gateway(handler, body: dict, cancel_event: threading.Event | None = None) -> dict:
|
||||
def _run_foreground_vedastro_gateway(
|
||||
handler,
|
||||
body: dict,
|
||||
cancel_event: threading.Event | None = None,
|
||||
*,
|
||||
budget_seconds: float | None = None,
|
||||
) -> dict:
|
||||
if cancel_event is not None and cancel_event.is_set():
|
||||
return _blocked_foreground_vedastro(reason="foreground_optional_evidence_timeout")
|
||||
budget = _foreground_vedastro_budget_seconds() if budget_seconds is None else budget_seconds
|
||||
try:
|
||||
with temporary_timeout_seconds(_foreground_vedastro_budget_seconds()):
|
||||
with temporary_timeout_seconds(budget):
|
||||
result = handler._compute_vedastro_gateway_run(body)
|
||||
except Exception as exc:
|
||||
return _blocked_foreground_vedastro(
|
||||
@@ -150,8 +213,9 @@ def _run_foreground_vedastro_gateway_and_store(
|
||||
handler,
|
||||
body: dict,
|
||||
cancel_event: threading.Event | None = None,
|
||||
budget_seconds: float | None = None,
|
||||
) -> dict:
|
||||
result = _run_foreground_vedastro_gateway(handler, body, cancel_event)
|
||||
result = _run_foreground_vedastro_gateway(handler, body, cancel_event, budget_seconds=budget_seconds)
|
||||
# A run cancelled before it started never asked VedAstro: nothing to remember.
|
||||
if not (cancel_event is not None and cancel_event.is_set() and result.get("official_closure_reason") == "foreground_optional_evidence_timeout"):
|
||||
_remember_gateway_outcome(body, result)
|
||||
@@ -198,8 +262,38 @@ class ForegroundVedastroSession:
|
||||
self.future = None
|
||||
self.cancel_event = threading.Event()
|
||||
self.meta: dict[str, Any] = {}
|
||||
self.report = is_report_vedastro_budget(self.body)
|
||||
self.started_at = time.monotonic()
|
||||
|
||||
def _budget_seconds(self) -> float | None:
|
||||
return _report_vedastro_budget_seconds() if self.report else None
|
||||
|
||||
def _join_seconds(self) -> float:
|
||||
if not self.report:
|
||||
return _foreground_vedastro_join_seconds()
|
||||
return max(self.started_at + _report_vedastro_deadline_seconds() - time.monotonic(), 0.0)
|
||||
|
||||
def _report_deadline_exceeded(self, result: dict) -> dict:
|
||||
"""A report gave up on a gateway that is still running: remember it for today."""
|
||||
future = self.future
|
||||
if (
|
||||
not self.report
|
||||
or result.get("official_closure_reason") != "foreground_optional_evidence_timeout"
|
||||
or future is None
|
||||
or future.cancelled()
|
||||
or future.done()
|
||||
):
|
||||
return result
|
||||
packet = _blocked_foreground_vedastro(reason=REPORT_VEDASTRO_DEADLINE_EXCEEDED)
|
||||
packet["deadline_seconds"] = _report_vedastro_deadline_seconds()
|
||||
try:
|
||||
store_negative_snapshot(self.body, packet)
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
return packet
|
||||
|
||||
def start(self) -> "ForegroundVedastroSession":
|
||||
self.started_at = time.monotonic()
|
||||
today = official_snapshot_reference_date(self.body)
|
||||
hit = lookup_snapshot(self.body, today=today)
|
||||
if hit is not None:
|
||||
@@ -221,6 +315,7 @@ class ForegroundVedastroSession:
|
||||
self.handler,
|
||||
refresh_body,
|
||||
None,
|
||||
self._budget_seconds(),
|
||||
)
|
||||
self.cached_gateway = gateway
|
||||
self.meta = {
|
||||
@@ -253,6 +348,7 @@ class ForegroundVedastroSession:
|
||||
self.handler,
|
||||
dict(self.body),
|
||||
self.cancel_event,
|
||||
self._budget_seconds(),
|
||||
)
|
||||
return self
|
||||
|
||||
@@ -260,11 +356,12 @@ class ForegroundVedastroSession:
|
||||
if self.cached_gateway is not None:
|
||||
return self.cached_gateway
|
||||
if self.defer:
|
||||
return _join_foreground_vedastro(
|
||||
result = _join_foreground_vedastro(
|
||||
self.future,
|
||||
timeout=_foreground_vedastro_join_seconds(),
|
||||
timeout=self._join_seconds(),
|
||||
cancel_event=self.cancel_event,
|
||||
)
|
||||
return self._report_deadline_exceeded(result)
|
||||
try:
|
||||
result = self.handler._compute_vedastro_gateway_run(self.body)
|
||||
except Exception as exc:
|
||||
|
||||
@@ -13,6 +13,7 @@ and remains ``official_blocked``.
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import http.client
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
@@ -155,7 +156,8 @@ def probe_calculate_health(*, opener: Any | None = None, timeout: float = 8) ->
|
||||
"endpoint": API,
|
||||
"transport": "rest",
|
||||
}
|
||||
except (urllib.error.URLError, TimeoutError, json.JSONDecodeError, OSError) as error:
|
||||
except (urllib.error.URLError, TimeoutError, json.JSONDecodeError, OSError, http.client.HTTPException) as error:
|
||||
# HTTPException: IncompleteRead and friends when the body is cut (BUG-1504).
|
||||
return {
|
||||
"status": "official_blocked",
|
||||
"reason": type(error).__name__,
|
||||
|
||||
@@ -14,6 +14,7 @@ import hashlib
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import ssl
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
@@ -2016,6 +2017,12 @@ def _acquire_free_tier_slot(request_url: str) -> dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
# A connection cut while reading headers or the body. urllib only wraps errors
|
||||
# raised while sending, so these reach the caller raw (BUG-1504). RemoteDisconnected
|
||||
# is both an HTTPException and a ConnectionResetError.
|
||||
_TRANSPORT_READ_ERRORS = (http.client.HTTPException, ConnectionError, ssl.SSLError)
|
||||
|
||||
|
||||
def _post_json(endpoint: str, request_preview: dict[str, Any]) -> dict[str, Any] | str:
|
||||
request_url, headers, vedastro_payload = _build_live_request(endpoint, request_preview)
|
||||
req = request.Request(
|
||||
@@ -2024,9 +2031,15 @@ def _post_json(endpoint: str, request_preview: dict[str, Any]) -> dict[str, Any]
|
||||
headers=headers,
|
||||
method="POST",
|
||||
)
|
||||
with request.urlopen(req, timeout=_timeout_seconds()) as resp:
|
||||
raw = resp.read().decode("utf-8")
|
||||
return json.loads(raw)
|
||||
try:
|
||||
with request.urlopen(req, timeout=_timeout_seconds()) as resp:
|
||||
raw = resp.read().decode("utf-8")
|
||||
except error.URLError:
|
||||
raise
|
||||
except _TRANSPORT_READ_ERRORS as exc:
|
||||
# Every caller already turns URLError into network_error.
|
||||
raise error.URLError(f"response_read_failed:{type(exc).__name__}") from exc
|
||||
return json.loads(raw)
|
||||
|
||||
|
||||
def _retry_status_codes() -> set[int]:
|
||||
|
||||
@@ -0,0 +1,344 @@
|
||||
"""BUG-1508 and BUG-1504: VedAstro can no longer stall or break a report.
|
||||
|
||||
BUG-1508 (outage audit gap 2): a report now waits for VedAstro only up to its
|
||||
deadline, then goes on with VedAstro marked blocked and remembers that for the
|
||||
day, so the next report does not wait again.
|
||||
|
||||
BUG-1504: a response cut while it is being read is a network error, like the
|
||||
other transport failures, and the main-entry overview never turns into a 500.
|
||||
|
||||
VedAstro is always a stub here; nothing leaves the machine.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import http.client
|
||||
import io
|
||||
import socket
|
||||
import ssl
|
||||
import threading
|
||||
import time
|
||||
from urllib import error
|
||||
|
||||
import pytest
|
||||
|
||||
import scripts.vedastro_foreground as foreground
|
||||
import scripts.vedastro_service_adapter as adapter
|
||||
from scripts.jyotish_api_server import JyotishComputeService
|
||||
from scripts.vedastro_consultation_snapshot import lookup_negative_snapshot
|
||||
from scripts.vedastro_rest_bridge import probe_calculate_health
|
||||
from scripts.vedastro_runtime_context import timeout_override_seconds
|
||||
from tests.test_api_heavy_compute_gate import _PostCaptureHandler
|
||||
|
||||
TODAY = "2026-11-02"
|
||||
BIRTH = {"year": 1990, "month": 6, "day": 15, "hour": 10, "minute": 20, "lat": 31.2, "lon": 121.5, "tz": 8,
|
||||
"ayanamsa": "lahiri", "today": TODAY}
|
||||
REPORT = {**BIRTH, "vedastro_evidence_budget": "report"}
|
||||
BLOCKED = {"scope": "vedastro_gateway_run", "status": "official_blocked", "official_closure_state": "official_blocked",
|
||||
"official_closure_reason": "official_raw_response_missing"}
|
||||
VERIFIED = {"scope": "vedastro_gateway_run", "status": "ok", "official_closure_state": "official_verified",
|
||||
"official_closure_reason": "official_raw_response_present", "official_raw_response": {"ok": True}}
|
||||
|
||||
|
||||
class Gateway:
|
||||
"""Stands in for the handler's gateway call; `hang` blocks until released."""
|
||||
|
||||
def __init__(self, result=None, *, hang=False, raises=None, delay=0.0):
|
||||
self.result, self.hang, self.raises, self.delay = result, hang, raises, delay
|
||||
self.calls = 0
|
||||
self.release = threading.Event()
|
||||
self.timeouts: list[float | None] = []
|
||||
|
||||
def _compute_vedastro_gateway_run(self, body):
|
||||
self.calls += 1
|
||||
self.timeouts.append(timeout_override_seconds())
|
||||
if self.hang:
|
||||
self.release.wait(5)
|
||||
time.sleep(self.delay)
|
||||
if self.raises is not None:
|
||||
raise self.raises
|
||||
return dict(self.result or BLOCKED)
|
||||
|
||||
|
||||
def _report(handler, body=REPORT, *, pause=0.0):
|
||||
started = time.perf_counter()
|
||||
session = foreground.start_foreground_vedastro(
|
||||
handler, body, defer_optional_external_evidence=foreground.defers_optional_vedastro(body),
|
||||
)
|
||||
time.sleep(pause) # local compute runs here in the real workflow
|
||||
gateway = foreground.finish_foreground_vedastro(session)
|
||||
return gateway, time.perf_counter() - started, session.meta
|
||||
|
||||
|
||||
def _drain():
|
||||
deadline = time.time() + 5
|
||||
while time.time() < deadline and not foreground._FOREGROUND_VEDASTRO_EXECUTOR._work_queue.empty():
|
||||
time.sleep(0.01)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def short_deadline(monkeypatch):
|
||||
monkeypatch.setenv("JYOTISH_REPORT_VEDASTRO_DEADLINE_SECONDS", "0.4")
|
||||
|
||||
|
||||
# ---- BUG-1508: deadline, blocked, remembered --------------------------------
|
||||
|
||||
|
||||
def test_a_report_budget_defers_and_a_plain_request_still_does_not():
|
||||
assert foreground.defers_optional_vedastro(REPORT)
|
||||
assert foreground.defers_optional_vedastro({**BIRTH, "defer_optional_external_evidence": True})
|
||||
assert not foreground.defers_optional_vedastro(BIRTH)
|
||||
|
||||
|
||||
def test_a_hung_vedastro_stops_the_report_at_its_deadline_and_the_next_report_does_not_wait(short_deadline):
|
||||
hung = Gateway(hang=True)
|
||||
try:
|
||||
gateway, seconds, meta = _report(hung)
|
||||
assert meta["freshness"] == "miss"
|
||||
assert 0.3 <= seconds < 0.9, seconds
|
||||
assert gateway["official_closure_state"] == "official_blocked"
|
||||
assert gateway["official_closure_reason"] == foreground.REPORT_VEDASTRO_DEADLINE_EXCEEDED
|
||||
remembered = lookup_negative_snapshot(REPORT, today=TODAY)
|
||||
assert remembered["official_closure_reason"] == foreground.REPORT_VEDASTRO_DEADLINE_EXCEEDED
|
||||
|
||||
again, again_seconds, again_meta = _report(hung)
|
||||
assert again_seconds < 0.15, again_seconds
|
||||
assert again_meta["freshness"] == "negative"
|
||||
assert again == gateway
|
||||
assert hung.calls == 1, "the second report did not ask VedAstro again"
|
||||
finally:
|
||||
hung.release.set()
|
||||
_drain()
|
||||
|
||||
|
||||
def test_the_deadline_counts_from_the_start_so_local_compute_overlaps_it(short_deadline):
|
||||
hung = Gateway(hang=True)
|
||||
try:
|
||||
_, seconds, _ = _report(hung, pause=0.3)
|
||||
assert seconds < 0.7, f"waited the whole deadline again after local compute: {seconds}"
|
||||
finally:
|
||||
hung.release.set()
|
||||
_drain()
|
||||
|
||||
|
||||
def test_a_gateway_that_finishes_after_the_deadline_replaces_the_deadline_entry(short_deadline):
|
||||
late = Gateway(VERIFIED, hang=True)
|
||||
gateway, _, _ = _report(late)
|
||||
assert gateway["official_closure_reason"] == foreground.REPORT_VEDASTRO_DEADLINE_EXCEEDED
|
||||
late.release.set()
|
||||
_drain()
|
||||
again, _, meta = _report(Gateway(BLOCKED))
|
||||
assert meta["freshness"] == "fresh", "a later success wins over the deadline entry"
|
||||
assert again["official_closure_state"] == "official_verified"
|
||||
|
||||
|
||||
def test_a_fast_5xx_failure_returns_at_once_and_is_remembered(short_deadline):
|
||||
failing = Gateway({**BLOCKED, "official_closure_reason": "http_error:503"})
|
||||
gateway, seconds, _ = _report(failing)
|
||||
assert seconds < 0.3 and gateway["official_closure_reason"] == "http_error:503"
|
||||
_drain()
|
||||
again, _, meta = _report(failing)
|
||||
assert failing.calls == 1 and meta["freshness"] == "negative" and again == gateway
|
||||
|
||||
|
||||
@pytest.mark.parametrize("exc", [http.client.IncompleteRead(b"{", 10), ConnectionResetError(104, "reset")])
|
||||
def test_a_gateway_that_raises_a_read_error_is_blocked_and_remembered(short_deadline, exc):
|
||||
broken = Gateway(raises=exc)
|
||||
gateway, _, _ = _report(broken)
|
||||
assert gateway["official_closure_state"] == "official_blocked"
|
||||
assert gateway["error_type"] == type(exc).__name__
|
||||
_drain()
|
||||
assert lookup_negative_snapshot(REPORT, today=TODAY)["error_type"] == type(exc).__name__
|
||||
|
||||
|
||||
def test_a_healthy_vedastro_inside_the_deadline_is_used(short_deadline):
|
||||
healthy = Gateway(VERIFIED, delay=0.05)
|
||||
gateway, _, _ = _report(healthy)
|
||||
assert gateway == VERIFIED
|
||||
assert lookup_negative_snapshot(REPORT, today=TODAY) is None
|
||||
|
||||
|
||||
def test_a_report_keeps_the_configured_request_timeout_and_never_raises_it(monkeypatch):
|
||||
monkeypatch.setenv("VEDASTRO_TIMEOUT_SECONDS", "20")
|
||||
seen = Gateway(VERIFIED)
|
||||
_report(seen)
|
||||
monkeypatch.setenv("VEDASTRO_TIMEOUT_SECONDS", "120")
|
||||
monkeypatch.setenv("JYOTISH_REPORT_VEDASTRO_DEADLINE_SECONDS", "25")
|
||||
_report(Gateway(VERIFIED), body={**REPORT, "today": "2026-11-03"})
|
||||
_drain()
|
||||
assert seen.timeouts == [20.0]
|
||||
assert foreground._report_vedastro_budget_seconds() == 50.0, "capped at k * deadline"
|
||||
|
||||
|
||||
def test_the_deadline_has_a_cap_well_inside_the_frontend_wait(monkeypatch):
|
||||
monkeypatch.setenv("JYOTISH_REPORT_VEDASTRO_DEADLINE_SECONDS", "600")
|
||||
assert foreground._report_vedastro_deadline_seconds() == foreground.REPORT_VEDASTRO_DEADLINE_SECONDS_MAX
|
||||
assert foreground.REPORT_VEDASTRO_DEADLINE_SECONDS_MAX <= 45, "frontend waits 90 s per workflow call"
|
||||
|
||||
|
||||
def test_a_consultation_join_timeout_is_still_not_remembered(monkeypatch):
|
||||
monkeypatch.setenv("JYOTISH_FOREGROUND_VEDASTRO_JOIN_SECONDS", "0.05")
|
||||
hung = Gateway(hang=True)
|
||||
try:
|
||||
gateway, _, _ = _report(hung, body={**BIRTH, "defer_optional_external_evidence": True})
|
||||
assert gateway["official_closure_reason"] == "foreground_optional_evidence_timeout"
|
||||
assert lookup_negative_snapshot(BIRTH, today=TODAY) is None
|
||||
finally:
|
||||
hung.release.set()
|
||||
_drain()
|
||||
|
||||
|
||||
def test_the_report_workflow_skips_the_overview_and_marks_vedastro_unavailable(short_deadline, monkeypatch):
|
||||
monkeypatch.setenv("JYOTISH_API_CHART_CACHE_TTL_SECONDS", "0")
|
||||
hung = Gateway(hang=True)
|
||||
service = JyotishComputeService()
|
||||
monkeypatch.setattr(JyotishComputeService, "_compute_vedastro_gateway_run",
|
||||
lambda self, body: hung._compute_vedastro_gateway_run(body))
|
||||
|
||||
def no_overview(*args, **kwargs):
|
||||
raise AssertionError("a report must not run the synchronous main-entry overview")
|
||||
|
||||
monkeypatch.setattr("scripts.jyotish_api_server._attach_vedastro_main_entry_overview", no_overview)
|
||||
body = {**REPORT, "question": "请为个人报告计算 career 主题证据", "theme": ["career"], "entry_mode": "direct_chart"}
|
||||
try:
|
||||
result = service._compute_consultation_workflow(dict(body))
|
||||
sections = result["machine_evidence_packet"]["sections"]
|
||||
assert sections["external_oracle_status"]["status"] == "official_blocked"
|
||||
assert sections["vedastro_official_raw_response"]["status"] == "missing"
|
||||
cross_check = result["consumer_context"]["vedastro_cross_check"]
|
||||
assert cross_check["official_closure_reason"] == foreground.REPORT_VEDASTRO_DEADLINE_EXCEEDED
|
||||
|
||||
started = time.perf_counter()
|
||||
service._compute_consultation_workflow(dict(body))
|
||||
assert hung.calls == 1, "the second report answered from the remembered failure"
|
||||
assert time.perf_counter() - started < 30
|
||||
finally:
|
||||
hung.release.set()
|
||||
_drain()
|
||||
|
||||
|
||||
# ---- BUG-1504: truncated responses are network errors -----------------------
|
||||
|
||||
|
||||
class _CutResponse:
|
||||
def __init__(self, exc):
|
||||
self.exc = exc
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc_info):
|
||||
return False
|
||||
|
||||
def read(self):
|
||||
raise self.exc
|
||||
|
||||
|
||||
def _urlopen_raising(exc, *, on_read):
|
||||
def fake(req, timeout=None, *args, **kwargs):
|
||||
if on_read:
|
||||
return _CutResponse(exc)
|
||||
raise exc
|
||||
|
||||
return fake
|
||||
|
||||
|
||||
READ_ERRORS = [
|
||||
pytest.param(http.client.IncompleteRead(b'{"Status":', 200), True, id="incomplete-read"),
|
||||
pytest.param(ConnectionResetError(104, "Connection reset by peer"), True, id="reset-on-read"),
|
||||
pytest.param(ConnectionResetError(104, "Connection reset by peer"), False, id="reset-on-headers"),
|
||||
pytest.param(http.client.RemoteDisconnected("closed"), False, id="remote-disconnected"),
|
||||
pytest.param(ssl.SSLError("bad record mac"), True, id="ssl"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stub_network(monkeypatch):
|
||||
monkeypatch.setenv("VEDASTRO_API_ENDPOINT", "https://oracle.stub.invalid/api")
|
||||
monkeypatch.setenv("VEDASTRO_ENABLE_NETWORK", "1")
|
||||
monkeypatch.setenv("VEDASTRO_CACHE_TTL_SECONDS", "0")
|
||||
monkeypatch.setenv("VEDASTRO_RETRY_BACKOFF_SECONDS", "0")
|
||||
monkeypatch.setenv("JYOTISH_API_CHART_CACHE_TTL_SECONDS", "0")
|
||||
empty = {"available": False, "status": "stubbed", "snapshot_sections": {}, "section_statuses": {}}
|
||||
monkeypatch.setattr(adapter, "_try_official_capability_runner_snapshot_bundle", lambda case: dict(empty))
|
||||
monkeypatch.setattr(adapter, "_try_official_python_bridge_snapshot_bundle", lambda case: dict(empty))
|
||||
monkeypatch.setattr(adapter, "_try_official_full_capability_catalog_bundle", lambda case: dict(empty))
|
||||
monkeypatch.setattr(adapter, "_load_cached_payload", lambda key: (None, None))
|
||||
monkeypatch.setattr(adapter, "_write_artifact", lambda result: None)
|
||||
|
||||
|
||||
def _case():
|
||||
return {**BIRTH, "case_id": "stub", "reference_date": TODAY}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("exc,on_read", READ_ERRORS)
|
||||
def test_a_cut_response_is_a_network_error_in_the_snapshot_and_the_range_scan(stub_network, monkeypatch, exc, on_read):
|
||||
monkeypatch.setattr(adapter.request, "urlopen", _urlopen_raising(exc, on_read=on_read))
|
||||
with pytest.raises(error.URLError):
|
||||
adapter._post_json("https://oracle.stub.invalid/api", {"operation": "official_full_snapshot",
|
||||
"section": "x", "official_request_profile": {}})
|
||||
snapshot = adapter._run_official_full_snapshot_case(_case(), case_id="stub")
|
||||
statuses = [value for value in snapshot["section_statuses"].values() if isinstance(value, str)]
|
||||
expected = f"network_error:response_read_failed:{type(exc).__name__}"
|
||||
assert expected in statuses, statuses
|
||||
assert "ok" not in statuses and snapshot["status"] != "ok"
|
||||
scan = adapter._run_range_scan_case(_case(), "career", "2026-11-01", "2026-11-30")
|
||||
assert scan["status"] == "network_error" and scan["available"] is False
|
||||
|
||||
|
||||
def test_a_5xx_is_still_an_http_error(stub_network, monkeypatch):
|
||||
def fake(req, timeout=None, *args, **kwargs):
|
||||
raise error.HTTPError(req.full_url, 503, "Service Unavailable", {}, io.BytesIO(b""))
|
||||
|
||||
monkeypatch.setattr(adapter.request, "urlopen", fake)
|
||||
snapshot = adapter._run_official_full_snapshot_case(_case(), case_id="stub")
|
||||
assert any(value == "http_error:503" for value in snapshot["section_statuses"].values())
|
||||
|
||||
|
||||
def test_a_read_timeout_is_still_a_timeout(stub_network, monkeypatch):
|
||||
monkeypatch.setattr(adapter.request, "urlopen", _urlopen_raising(socket.timeout("timed out"), on_read=True))
|
||||
scan = adapter._run_range_scan_case(_case(), "career", "2026-11-01", "2026-11-30")
|
||||
assert scan["status"] == "timeout"
|
||||
|
||||
|
||||
def _chart_request():
|
||||
return _PostCaptureHandler("/api/chart", {**BIRTH, "date": "1990-06-15", "time": "10:20"})
|
||||
|
||||
|
||||
@pytest.mark.parametrize("exc,on_read", READ_ERRORS[:2])
|
||||
def test_api_chart_with_the_overview_returns_200_when_vedastro_cuts_the_response(stub_network, monkeypatch, exc, on_read):
|
||||
monkeypatch.setattr(adapter.request, "urlopen", _urlopen_raising(exc, on_read=on_read))
|
||||
handler = _chart_request()
|
||||
handler.do_POST()
|
||||
assert handler.status_code == 200, handler.wfile.getvalue()[:300]
|
||||
overview = handler.payload()["modules"]["vedastro_range_scan_result"]
|
||||
assert overview["status"] != "ok"
|
||||
assert overview["official_full_snapshot"]["status"] != "ok"
|
||||
|
||||
|
||||
def test_api_chart_overview_attach_fails_soft_on_any_error(monkeypatch):
|
||||
monkeypatch.setenv("JYOTISH_API_CHART_CACHE_TTL_SECONDS", "0")
|
||||
from scripts.jyotish_api_server import _load_local_module
|
||||
|
||||
orchestrator = _load_local_module("vedastro_evidence_orchestrator")
|
||||
|
||||
def cut(*args, **kwargs):
|
||||
raise http.client.IncompleteRead(b"", 5)
|
||||
|
||||
monkeypatch.setattr(orchestrator, "orchestrate_vedastro_evidence", cut)
|
||||
handler = _chart_request()
|
||||
handler.do_POST()
|
||||
assert handler.status_code == 200, handler.wfile.getvalue()[:300]
|
||||
overview = handler.payload()["modules"]["vedastro_range_scan_result"]
|
||||
assert overview["status"] == "blocked"
|
||||
assert overview["reason"] == "vedastro_main_entry_overview_error"
|
||||
assert overview["error_type"] == "IncompleteRead"
|
||||
|
||||
|
||||
def test_the_rest_health_probe_blocks_on_a_cut_body():
|
||||
def opener(req, timeout=None):
|
||||
return _CutResponse(http.client.IncompleteRead(b"{", 9))
|
||||
|
||||
result = probe_calculate_health(opener=opener)
|
||||
assert result["status"] == "official_blocked" and result["reason"] == "IncompleteRead"
|
||||
Reference in New Issue
Block a user