Merge BUG-1504/BUG-1508 VedAstro report deadline
Independent Staging Quality Gate / validate (push) Successful in 19m22s
Independent Staging Quality Gate / publish (push) Successful in 3m53s

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DbBXoyeZrYn6ryNcSe5Y2Y

# Conflicts:
#	CHANGELOG.md
#	docs/operations/outage-behaviour-2026-11-01.md
This commit is contained in:
2026-11-02 07:50:47 +08:00
14 changed files with 552 additions and 25 deletions
+10
View File
@@ -18,6 +18,16 @@
- 改日期查到新时区后又改回原日期,会重新查一次,不再沿用刚才那个日期的时区。
- 记为 BUG-1506。Skill 版本不变。
## 2026-11-02 — VedAstro 卡住或断线时,报告不再失败
- 生成报告时,VedAstro 最多等 25 秒(从开始计算时算起,和本地计算同时进行),前端的等待是 90 秒。等不到就照常出报告,报告里的 VedAstro 一栏标「官方外部引擎本次未验证或不可用」,不会写成已验证。
- 这次失败会记到当天:同一张盘当天再生成报告、或者同一份报告的下一个主题,直接用这条记录,不再等 VedAstro。之后 VedAstro 恢复并返回结果,就改用新结果。
- 报告不再先跑一遍 VedAstro 主入口概览(按主题的区间扫描)。官方 VedAstro 证据仍然有,来自官方快照,按时返回就照常写进报告。
- VedAstro 返回到一半连接断开(读响应时连接被重置或截断)时,按网络错误处理:报告照常出、VedAstro 标未验证;`/api/chart` 不带跳过概览的参数时也返回 200,不再报 500。
- 对话、星盘页的行为不变。
- 可调参数:`JYOTISH_REPORT_VEDASTRO_DEADLINE_SECONDS`(默认 25,最多 45)。
- Skill 版本不变(Gitea #119 第 42 项缺口 2、3;BUG-1508、BUG-1504)。
## 2026-11-01 — 年运报告:从「我的报告」直接打开,写好一节出一节
+34 -9
View File
@@ -20573,19 +20573,22 @@
## BUG-1504 | VedAstro 响应读到一半连接断开时,报告生成与不带 skip 的排盘返回 500
- 状态:investigating
- 状态:resolved
- 首次发现:2026-11-01(Gitea #119 第 42 项,外部依赖故障审计)
- 最近更新:2026-11-01
- 最近更新:2026-11-02
- 影响面:同步调 VedAstro 官方快照的路径,包括个人报告生成(排盘主入口概览、生时门禁)和不带 `skip_vedastro_main_entry_overview` 的 `/api/chart`。星盘页、对话不受影响:前者总是跳过,后者在后台线程里调用。
- 用户现象:报告以 `workflow_server_error` 失败(会退点);接口返回 `ERR_INTERNAL`。
- 触发条件:VedAstro 已经返回响应头,但响应体读到一半连接被重置或截断(`http.client.IncompleteRead`、`ConnectionResetError`、`ssl.SSLError`)。
- 根因(读码 + 本机 stub 确认 IncompleteRead):`scripts/vedastro_service_adapter.py` 的快照分发(约 2782–2808 行)和区间扫描(约 3010–3046 行)只接住 `HTTPError`、`URLError`、`RemoteDisconnected`、超时和 `JSONDecodeError`;`_attach_vedastro_main_entry_overview` 外面也没有 try,所以异常一直冒到请求总入口。
- 修复:未修。本轮只审计,修法待产品决定是否排期(把这几类读错误归为 `network_error`)。
- 验证:本机把 `urlopen` 换成读响应时抛 `IncompleteRead` 的 stub,调用排盘得到 500。脚本是临时的,没有入库;做法见 `docs/operations/outage-behaviour-2026-11-01.md` 第一节。
- 防复发:待修复单补「读错误降级、不 500」的测试。
- 相关记录:Gitea #119 第 42 项;`docs/operations/outage-behaviour-2026-11-01.md`。
- 触发条件:VedAstro 已经返回响应头,但响应体读到一半连接被重置或截断(`http.client.IncompleteRead`、`ConnectionResetError`、`ssl.SSLError`)。读响应头时连接被重置也一样。
- 根因(读码 + 本机 stub 确认 IncompleteRead):`scripts/vedastro_service_adapter.py` 的快照分发(约 2782–2808 行)和区间扫描(约 3010–3046 行)只接住 `HTTPError`、`URLError`、`RemoteDisconnected`、超时和 `JSONDecodeError`;`urllib` 只包装发请求时的错误,读响应头和响应体时的错误原样抛出。`_attach_vedastro_main_entry_overview` 外面也没有 try,所以异常一直冒到请求总入口。
- 修复:
- 适配器的 `_post_json` 把读响应时的 `http.client.HTTPException`(含 `IncompleteRead`、`RemoteDisconnected`)、`ConnectionError`(含 `ConnectionResetError`)和 `ssl.SSLError` 转成 `URLError("response_read_failed:<类型名>")`。所有调用点本来就把 `URLError` 记成 `network_error`,所以快照分节、区间扫描、外部技法证据都按网络错误处理,官方快照不再是 ok,网关结果是 `official_blocked`。前台和报告会话照原样把这个失败记进当天的失败缓存。读超时仍是 `timeout`,5xx 仍是 `http_error:<code>`。`RemoteDisconnected` 的标签从 `network_error:<原文>` 变成 `network_error:response_read_failed:RemoteDisconnected`。
- `_attach_vedastro_main_entry_overview` 外面包一层 try:任何异常都记成 `status=blocked`、`reason=vedastro_main_entry_overview_error`、`error_type=<类型名>`,排盘照常返回。
- `vedastro_rest_bridge.probe_calculate_health` 也接住 `http.client.HTTPException`,健康探测读到一半断开时记 `official_blocked`。
- 验证:`tests/test_vedastro_report_deadline_20261102.py`:把 `urlopen` 换成 stub,分别在读响应体时抛 `IncompleteRead`、`ConnectionResetError`、`ssl.SSLError`,在打开连接时抛 `ConnectionResetError`、`RemoteDisconnected`。`_post_json` 抛 `URLError`;官方快照分节是 `network_error:response_read_failed:<类型名>`,没有 ok;区间扫描是 `network_error`。`/api/chart`(不带 skip)走真实 `do_POST` 返回 200,`vedastro_range_scan_result` 不是 ok;编排器直接抛 `IncompleteRead` 时也返回 200,概览为 blocked。503 仍是 `http_error:503`,读超时仍是 `timeout`。把适配器和 REST 桥换回修复前版本,这几条测试失败。
- 防复发:该测试文件已加进 `scripts/run_quality_gate.py` 的 quick 清单。新增的 VedAstro 网络调用应当走 `_post_json`,不要自己调 `urlopen`。
- 相关记录:BUG-1508(同一轮修复,报告等 VedAstro 的总时限);Gitea #119 第 42 项;`docs/operations/outage-behaviour-2026-11-01.md`。
- 复发自:无
- 修复版本:未修。
- 修复版本:分支 `codex/vedastro-deadline-1504`。
## BUG-1505 | 报告任务最后一次尝试租约过期时,报告标为失败但点数不退
@@ -20645,3 +20648,25 @@
- 相关记录:BUG-1506;`docs/operations/outage-behaviour-2026-11-01.md` 第二节。新疆 `Asia/Urumqi`(+6)对报北京时间的用户差两小时,另待产品决定,没有单独记 Bug。
- 复发自:无
- 修复版本:未修。
## BUG-1508 | 报告生成同步等 VedAstro,没有总时限,VedAstro 卡住时报告超时失败
- 状态:resolved
- 首次发现:2026-11-01(Gitea #119 第 42 项,外部依赖故障审计,缺口 2)
- 最近更新:2026-11-02
- 影响面:个人报告生成(`/api/reports` 创建和报告 worker 调 `/api/consultation_workflow`)。对话、星盘页不受影响。
- 用户现象:VedAstro 卡住时报告以 `workflow_timeout` 失败(会退点)。期间一直占着进程级两个重计算名额中的一个,并发两份报告会让对话和校正拿到 429。下一份报告还要再等一遍。
- 触发条件:VedAstro 不响应(每次请求都等满 `VEDASTRO_TIMEOUT_SECONDS`,生产 20 秒)时生成报告。
- 根因(读码 + 本机 stub 确认):报告调用没有传 `foreground`,所以 `defer_optional_external_evidence=false`,Python 端同步跑两段 VedAstro:排盘里的主入口概览(约 4 × 20 秒)和生时门禁里的官方快照(含 8 秒健康探测,约 48 秒)。合计约 128 秒,超过前端 90 秒(`frontend/src/mastra/consultation-workflow.ts:149`)。这条路径的结果直接取自门禁,不经过前台会话,也就不写失败缓存。
- 修复:复用对话已有的前台会话、后台线程和当天失败缓存(#130),不另起一套。
- 前端:报告的两处调用(`frontend/src/app/api/reports/route.ts`、`frontend/src/lib/personal-report-worker.ts`)传 `vedastroEvidenceBudget: "report"`,请求体带 `vedastro_evidence_budget: "report"`。报告仍不传 `foreground`。
- Python:`scripts/vedastro_foreground.py` 新增报告档。带这个字段的请求按 defer 处理,和对话一样:不跑同步的主入口概览;生时门禁只读缓存;官方网关在后台线程里跑,和本地计算并行。不同的是等待时长:对话从收尾时起最多等 1.5 秒;报告从会话开始起算,最多等 25 秒(`JYOTISH_REPORT_VEDASTRO_DEADLINE_SECONDS`,上限 45 秒)。单次请求超时沿用适配器的配置值,报告不会把它调高,也不超过 2 倍时限。
- 到时限网关还在跑:报告照常继续,VedAstro 记 `official_blocked` / `report_vedastro_deadline_exceeded`,并写进这张盘当天(UTC)的失败缓存,下一份报告和下一个主题直接用它,不再等。后台线程继续跑:之后成功就写进正常快照缓存(查缓存时成功优先),失败就用真实失败覆盖这条记录。还没开始就被取消的任务不记。
- 5xx、连接截断等很快返回的失败照旧记进失败缓存。
- 报告里的证据如实标出:`machine_evidence_packet` 的 `external_oracle_status` 是 `official_blocked`,`vedastro_official_raw_response` 是 `missing`,报告的技法审计表按原有规则显示 blocked(「官方外部引擎本次未验证或不可用」)。对话技法审计的说明文字加了 `report_vedastro_deadline_exceeded` 的中文。
- 取舍:报告不再跑主入口概览(按主题的区间扫描),引导话题里的 VedAstro 区间事件显示为没有;官方 VedAstro 证据改由网关的官方快照提供,按时返回就照常进报告。
- 验证:`tests/test_vedastro_report_deadline_20261102.py`,VedAstro 全部是 stub:网关卡住、时限 0.4 秒时,会话 0.4 秒左右返回 blocked,失败写进缓存,第二次 0.15 秒内返回、网关只调过一次;时限从会话开始起算,本地计算占掉的时间不再额外等;到时限后网关成功,下一次读到成功快照;很快返回的 503 类失败和抛 `IncompleteRead`、`ConnectionResetError` 的网关都记成 blocked 并写缓存;健康时照常用网关结果;对话档的 1.5 秒超时仍不记;单次请求超时沿用配置值。整条 `consultation_workflow` 带报告字段时不调主入口概览,证据段落为 `official_blocked` / `missing`,第二次不再调网关。本机脚本把时限设为 6 秒、单次超时 20 秒、VedAstro 全部卡住:第一份 6.23 秒,第二份 0.11 秒(修复前同样条件约 128 秒)。脚本是临时的,没有入库。
- 防复发:该测试文件已加进 `scripts/run_quality_gate.py` 的 quick 清单;`frontend/tests/consultation-workflow-contract.test.ts` 锁住报告两处调用都带 `vedastroEvidenceBudget: "report"`、都不带 `foreground: true`。
- 相关记录:BUG-1504;BUG-161、BUG-727、#130(前台会话和失败缓存);`docs/operations/outage-behaviour-2026-11-01.md` 第一节、第五节第 2 条。
- 复发自:无
- 修复版本:分支 `codex/vedastro-deadline-1504`。
@@ -113,9 +113,9 @@ Gitea #119 第 42 项:「VedAstro、地点服务、模型供应商挂掉时网
## 五、缺口(按风险从高到低)
1. **报告任务租约过期不退点(BUG-1505)。** 进程在最后一次尝试时重启就会触发,发版时尤其容易遇到。点数扣掉了、报告失败,用户要么删报告,要么重新付一次。退点 RPC 失败也被忽略。
2. **报告生成同步等 VedAstro,没有总时限。** VedAstro 卡住时约 128 秒,超过前端 90 秒,报告失败;同时占着两个重计算名额中的一个,并发两份报告会让对话和校正拿到 429。这条路径的失败也不写失败缓存,下一份报告会重来一遍。可选做法:报告也走 `defer_optional_external_evidence`,或者跳过主入口概览,或者复用对话的前台预算和失败缓存。
3. **VedAstro 连接被截断会变成 500(BUG-1504)。** 报告失败(会退点),不带 skip 的 `/api/chart` 也会 500。
1. **报告任务租约过期不退点(BUG-1505)。** 进程在最后一次尝试时重启就会触发,发版时尤其容易遇到。点数扣掉了、报告失败,用户要么删报告,要么重新付一次。退点 RPC 失败也被忽略。**2026-11-02 已修**:租约回收在判失败前先退点(与删除触发器、worker 同一个请求号,不会重复退);退点失败的任务留到下次回收重试,并记 `errorCode=report_refund_failed`。
2. **报告生成同步等 VedAstro,没有总时限。** VedAstro 卡住时约 128 秒,超过前端 90 秒,报告失败;同时占着两个重计算名额中的一个,并发两份报告会让对话和校正拿到 429。这条路径的失败也不写失败缓存,下一份报告会重来一遍。可选做法:报告也走 `defer_optional_external_evidence`,或者跳过主入口概览,或者复用对话的前台预算和失败缓存。**2026-11-02 已修(BUG-1508)**:报告复用对话的前台会话和失败缓存,VedAstro 最多等 25 秒,到时限标 blocked 并记当天。
3. **VedAstro 连接被截断会变成 500(BUG-1504)。** 报告失败(会退点),不带 skip 的 `/api/chart` 也会 500。**2026-11-02 已修**:读响应时的截断和重置按网络错误处理,主入口概览出错时排盘照常返回。
4. **出生资料编辑时,时区服务失败会默默清掉出生地(BUG-1506)。**2026-11-02 已修。
5. **早期中国资料默默按 +8 计算(BUG-1507)。** 这是正确性问题,与故障无关:1986–1991 年夏季出生的差一小时。新疆 +6 要产品决定。
6. **模型调用不重试、没有备用供应商。** 一次 429 或 503 就让付费用户这一轮失败。会退点,但体验差;目录里配了多个模型,失败时也不切换。
+1 -1
View File
@@ -381,7 +381,7 @@ export async function POST(request: Request) {
return released.success;
},
},
runWorkflow: (input) => runConsultationWorkflow(input),
runWorkflow: (input) => runConsultationWorkflow(input, { vedastroEvidenceBudget: "report" }),
createAgent: (model) => createPersonalReportAgent(model as Parameters<typeof createPersonalReportAgent>[0]),
skillSnapshot: resolveSkillSnapshot(),
jobs: createSupabasePersonalReportJobService(admin),
@@ -71,6 +71,7 @@ const NOTE_LABELS: Readonly<Record<string, string>> = {
official_raw_archive_is_auditable: "官方原始档案可审计",
official_cloud_evidence_available: "官方云证据可用",
foreground_optional_evidence_timeout: "前台官方云证据超时,本轮未赶上",
report_vedastro_deadline_exceeded: "报告等待官方云证据超过时限,本次未采用",
gateway_invocation_error: "官方云网关调用失败",
claims_capped_until_pyjhora_jhora_jyotishganit_are_invoked_for_this_run: "结论上限受外部引擎调用限制",
three_engine_runtime_closure_available: "三引擎运行时已闭环",
+1 -1
View File
@@ -134,7 +134,7 @@ async function generateWriterReport(
try {
workflows.push({
theme: rawTheme,
workflow: await runConsultationWorkflow(input, { signal: context.signal }),
workflow: await runConsultationWorkflow(input, { signal: context.signal, vedastroEvidenceBudget: "report" }),
});
} catch (error) {
if (context.signal.aborted) throw error;
@@ -137,6 +137,12 @@ export async function runConsultationWorkflow(
input: ConsultationInput,
options?: {
foreground?: boolean;
/**
* "report": the API waits for VedAstro only up to its report deadline
* (well inside the 90 s below), then marks it blocked and remembers the
* failure for the day. BUG-1508.
*/
vedastroEvidenceBudget?: "report";
signal?: AbortSignal;
plan?: ConsultationPlan;
requestId?: string;
@@ -161,6 +167,7 @@ export async function runConsultationWorkflow(
question_text: workflowRequest.question,
theme: workflowRequest.themes,
defer_optional_external_evidence: options?.foreground === true,
...(options?.vedastroEvidenceBudget ? { vedastro_evidence_budget: options.vedastroEvidenceBudget } : {}),
plan_version: workflowRequest.plan_version,
strict_workflow_route: workflowRequest.strictWorkflowRoute,
required_layers: workflowRequest.requiredLayers,
@@ -12,6 +12,7 @@ const mastra = readFileSync(new URL("../src/mastra/index.ts", import.meta.url),
const tools = readFileSync(new URL("../src/mastra/consultation-tools.ts", import.meta.url), "utf8");
const stream = readFileSync(new URL("../src/lib/stream-agent-response.ts", import.meta.url), "utf8");
const workflow = readFileSync(new URL("../src/mastra/consultation-workflow.ts", import.meta.url), "utf8");
const reportWorker = readFileSync(new URL("../src/lib/personal-report-worker.ts", import.meta.url), "utf8");
const plan = readFileSync(new URL("../src/lib/consultation-plan.ts", import.meta.url), "utf8");
const routeService = readFileSync(new URL("../src/lib/consultation-route-service.ts", import.meta.url), "utf8");
const pythonPlanContract = readFileSync(new URL("../../scripts/consultation_plan_contract.py", import.meta.url), "utf8");
@@ -339,8 +340,14 @@ test("personal consultation lets the Agent invoke the server-bound workflow tool
test("defers optional external evidence only for foreground chat", () => {
assert.match(workflow, /defer_optional_external_evidence: options\?\.foreground === true/);
assert.match(reportsRoute, /runWorkflow: \(input\) => runConsultationWorkflow\(input\)/);
// 原值: /runWorkflow: \(input\) => runConsultationWorkflow\(input\)/
// 新值: 报告带 vedastroEvidenceBudget: "report",仍不是前台对话(不带 foreground)
// 原因: BUG-1508,报告等 VedAstro 要有总时限,超时标 blocked 并记当天失败
assert.match(reportsRoute, /runWorkflow: \(input\) => runConsultationWorkflow\(input, \{ vedastroEvidenceBudget: "report" \}\)/);
assert.doesNotMatch(reportsRoute, /foreground:\s*true/);
assert.match(workflow, /vedastro_evidence_budget: options\.vedastroEvidenceBudget/);
assert.match(reportWorker, /runConsultationWorkflow\(input, \{ signal: context\.signal, vedastroEvidenceBudget: "report" \}\)/);
assert.doesNotMatch(reportWorker, /foreground:\s*true/);
});
test("personal Agent owns the Skill and context-bound calculation tool", () => {
+19 -1
View File
@@ -60,6 +60,7 @@ try:
_foreground_vedastro_join_seconds,
_join_foreground_vedastro,
_run_foreground_vedastro_gateway,
defers_optional_vedastro,
finish_foreground_vedastro,
should_include_western_evidence_packet,
start_foreground_vedastro,
@@ -83,6 +84,7 @@ except ModuleNotFoundError: # pragma: no cover - script execution path
_foreground_vedastro_join_seconds,
_join_foreground_vedastro,
_run_foreground_vedastro_gateway,
defers_optional_vedastro,
finish_foreground_vedastro,
should_include_western_evidence_packet,
start_foreground_vedastro,
@@ -2151,7 +2153,8 @@ def execute_consultation_workflow(
question = body.get('question') or ''
entry_mode = body.get('entry_mode', 'direct_chart')
high_rigor = bool(body.get('return_high_rigor_shape'))
defer_optional_external_evidence = bool(body.get('defer_optional_external_evidence'))
# A report's VedAstro budget defers like consult, with a longer deadline (BUG-1508).
defer_optional_external_evidence = defers_optional_vedastro(body)
try:
from scripts.three_engine_parity_replay_validator import validate_manifest
except ModuleNotFoundError: # pragma: no cover - direct script execution
@@ -2901,6 +2904,21 @@ def _attach_vedastro_main_entry_overview(chart_result, birth_payload):
or birth_payload.get('today')
or datetime.utcnow().strftime('%Y-%m-%d')
)[:10]
try:
return _attach_vedastro_main_entry_overview_from(orchestrator, priority, chart_result, modules, birth_payload, reference_date)
except Exception as exc: # BUG-1504: the overview is optional evidence; never a 500.
modules['vedastro_range_scan_result'] = {
'backend': 'vedastro_service_adapter_candidate',
'available': False,
'status': 'blocked',
'reason': 'vedastro_main_entry_overview_error',
'error_type': type(exc).__name__,
'source_metadata': {'ingestion_profile': 'main_entry_overview', 'reference_date': reference_date},
}
return chart_result
def _attach_vedastro_main_entry_overview_from(orchestrator, priority, chart_result, modules, birth_payload, reference_date):
vedastro_evidence = orchestrator.orchestrate_vedastro_evidence({
'year': birth_payload.get('year'),
'month': birth_payload.get('month'),
+3
View File
@@ -139,6 +139,9 @@ CORE_PYTEST_TARGETS = [
"tests/test_report_english_sweep_smoke.py",
# /api/shadbala with the birth moment over real HTTP, plus ruff F821 on the API server (BUG-1463).
"tests/test_shadbala_service_api.py",
# Report waits on VedAstro only to its deadline, then blocked + remembered (BUG-1508);
# a cut VedAstro response is a network error, /api/chart stays 200 (BUG-1504).
"tests/test_vedastro_report_deadline_20261102.py",
]
RUNTIME_TRUTH_PYTEST_TARGETS = [
+102 -5
View File
@@ -19,11 +19,31 @@ budget <= k * join (k=2) keeps leftover occupancy of an already-running
worker from stretching later requests past the join cap. The previous
default budget of 8s with a 1.5s join left cancelled-too-late workers
holding the two-thread pool for the rest of the 8s.
Report profile (body ``vedastro_evidence_budget="report"``, gap 2 of
docs/operations/outage-behaviour-2026-11-01.md, BUG-1508). A report used to
wait synchronously on the main-entry overview and the rectification gate,
about 128s when VedAstro hangs, past the frontend's 90s. A report now runs
the same deferred session as consultation, with a longer wait so a healthy
VedAstro still lands in the report:
deadline: measured from session start, so it overlaps local compute
(default 25s, cap 45s).
budget: the per-request timeout already configured for the adapter, never
more than k * deadline.
When the deadline passes while the gateway is still running, the report
continues with VedAstro marked ``official_blocked`` /
``report_vedastro_deadline_exceeded`` and that is remembered for the chart
and UTC day, so the next report answers at once. The worker keeps running;
if it later succeeds the positive snapshot wins, if it fails its own failure
replaces the deadline entry.
"""
from __future__ import annotations
import os
import threading
import time
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
from typing import Any
@@ -65,6 +85,13 @@ FOREGROUND_VEDASTRO_BUDGET_JOIN_RATIO = 2.0
FOREGROUND_VEDASTRO_WORKERS_DEFAULT = 2
# ----------------------------------------------------------------------
# --- report profile: deadline / budget (change together, see docstring) ---
REPORT_VEDASTRO_EVIDENCE_BUDGET = "report"
REPORT_VEDASTRO_DEADLINE_SECONDS_DEFAULT = 25.0
REPORT_VEDASTRO_DEADLINE_SECONDS_MAX = 45.0
REPORT_VEDASTRO_DEADLINE_EXCEEDED = "report_vedastro_deadline_exceeded"
# ----------------------------------------------------------------------
_FOREGROUND_VEDASTRO_WORKERS = max(
int(os.environ.get("JYOTISH_FOREGROUND_VEDASTRO_WORKERS", str(FOREGROUND_VEDASTRO_WORKERS_DEFAULT))),
1,
@@ -98,6 +125,35 @@ def _foreground_vedastro_budget_seconds() -> float:
return min(budget, cap)
def is_report_vedastro_budget(body: dict | None) -> bool:
payload = body if isinstance(body, dict) else {}
return str(payload.get("vedastro_evidence_budget") or "").strip().lower() == REPORT_VEDASTRO_EVIDENCE_BUDGET
def defers_optional_vedastro(body: dict | None) -> bool:
"""Consult asks with the flag; a report budget implies it (BUG-1508)."""
payload = body if isinstance(body, dict) else {}
return bool(payload.get("defer_optional_external_evidence")) or is_report_vedastro_budget(payload)
def _report_vedastro_deadline_seconds() -> float:
raw = str(os.environ.get("JYOTISH_REPORT_VEDASTRO_DEADLINE_SECONDS", str(REPORT_VEDASTRO_DEADLINE_SECONDS_DEFAULT))).strip()
try:
return min(max(float(raw), 0.0), REPORT_VEDASTRO_DEADLINE_SECONDS_MAX)
except ValueError:
return REPORT_VEDASTRO_DEADLINE_SECONDS_DEFAULT
def _report_vedastro_budget_seconds() -> float:
"""The adapter's own per-request timeout; a report never raises it."""
try:
from scripts.vedastro_service_adapter import _timeout_seconds as configured_timeout
except ModuleNotFoundError: # pragma: no cover - script execution path
from vedastro_service_adapter import _timeout_seconds as configured_timeout
cap = FOREGROUND_VEDASTRO_BUDGET_JOIN_RATIO * _report_vedastro_deadline_seconds()
return min(float(configured_timeout()), cap)
def _blocked_foreground_vedastro(*, reason: str, error_type: str | None = None) -> dict:
packet = {
"scope": "vedastro_gateway_run",
@@ -110,11 +166,18 @@ def _blocked_foreground_vedastro(*, reason: str, error_type: str | None = None)
return packet
def _run_foreground_vedastro_gateway(handler, body: dict, cancel_event: threading.Event | None = None) -> dict:
def _run_foreground_vedastro_gateway(
handler,
body: dict,
cancel_event: threading.Event | None = None,
*,
budget_seconds: float | None = None,
) -> dict:
if cancel_event is not None and cancel_event.is_set():
return _blocked_foreground_vedastro(reason="foreground_optional_evidence_timeout")
budget = _foreground_vedastro_budget_seconds() if budget_seconds is None else budget_seconds
try:
with temporary_timeout_seconds(_foreground_vedastro_budget_seconds()):
with temporary_timeout_seconds(budget):
result = handler._compute_vedastro_gateway_run(body)
except Exception as exc:
return _blocked_foreground_vedastro(
@@ -150,8 +213,9 @@ def _run_foreground_vedastro_gateway_and_store(
handler,
body: dict,
cancel_event: threading.Event | None = None,
budget_seconds: float | None = None,
) -> dict:
result = _run_foreground_vedastro_gateway(handler, body, cancel_event)
result = _run_foreground_vedastro_gateway(handler, body, cancel_event, budget_seconds=budget_seconds)
# A run cancelled before it started never asked VedAstro: nothing to remember.
if not (cancel_event is not None and cancel_event.is_set() and result.get("official_closure_reason") == "foreground_optional_evidence_timeout"):
_remember_gateway_outcome(body, result)
@@ -198,8 +262,38 @@ class ForegroundVedastroSession:
self.future = None
self.cancel_event = threading.Event()
self.meta: dict[str, Any] = {}
self.report = is_report_vedastro_budget(self.body)
self.started_at = time.monotonic()
def _budget_seconds(self) -> float | None:
return _report_vedastro_budget_seconds() if self.report else None
def _join_seconds(self) -> float:
if not self.report:
return _foreground_vedastro_join_seconds()
return max(self.started_at + _report_vedastro_deadline_seconds() - time.monotonic(), 0.0)
def _report_deadline_exceeded(self, result: dict) -> dict:
"""A report gave up on a gateway that is still running: remember it for today."""
future = self.future
if (
not self.report
or result.get("official_closure_reason") != "foreground_optional_evidence_timeout"
or future is None
or future.cancelled()
or future.done()
):
return result
packet = _blocked_foreground_vedastro(reason=REPORT_VEDASTRO_DEADLINE_EXCEEDED)
packet["deadline_seconds"] = _report_vedastro_deadline_seconds()
try:
store_negative_snapshot(self.body, packet)
except (OSError, ValueError):
pass
return packet
def start(self) -> "ForegroundVedastroSession":
self.started_at = time.monotonic()
today = official_snapshot_reference_date(self.body)
hit = lookup_snapshot(self.body, today=today)
if hit is not None:
@@ -221,6 +315,7 @@ class ForegroundVedastroSession:
self.handler,
refresh_body,
None,
self._budget_seconds(),
)
self.cached_gateway = gateway
self.meta = {
@@ -253,6 +348,7 @@ class ForegroundVedastroSession:
self.handler,
dict(self.body),
self.cancel_event,
self._budget_seconds(),
)
return self
@@ -260,11 +356,12 @@ class ForegroundVedastroSession:
if self.cached_gateway is not None:
return self.cached_gateway
if self.defer:
return _join_foreground_vedastro(
result = _join_foreground_vedastro(
self.future,
timeout=_foreground_vedastro_join_seconds(),
timeout=self._join_seconds(),
cancel_event=self.cancel_event,
)
return self._report_deadline_exceeded(result)
try:
result = self.handler._compute_vedastro_gateway_run(self.body)
except Exception as exc:
+3 -1
View File
@@ -13,6 +13,7 @@ and remains ``official_blocked``.
from __future__ import annotations
import argparse
import http.client
import json
import os
import sys
@@ -155,7 +156,8 @@ def probe_calculate_health(*, opener: Any | None = None, timeout: float = 8) ->
"endpoint": API,
"transport": "rest",
}
except (urllib.error.URLError, TimeoutError, json.JSONDecodeError, OSError) as error:
except (urllib.error.URLError, TimeoutError, json.JSONDecodeError, OSError, http.client.HTTPException) as error:
# HTTPException: IncompleteRead and friends when the body is cut (BUG-1504).
return {
"status": "official_blocked",
"reason": type(error).__name__,
+16 -3
View File
@@ -14,6 +14,7 @@ import hashlib
import json
import os
import socket
import ssl
import subprocess
import sys
import threading
@@ -2016,6 +2017,12 @@ def _acquire_free_tier_slot(request_url: str) -> dict[str, Any]:
}
# A connection cut while reading headers or the body. urllib only wraps errors
# raised while sending, so these reach the caller raw (BUG-1504). RemoteDisconnected
# is both an HTTPException and a ConnectionResetError.
_TRANSPORT_READ_ERRORS = (http.client.HTTPException, ConnectionError, ssl.SSLError)
def _post_json(endpoint: str, request_preview: dict[str, Any]) -> dict[str, Any] | str:
request_url, headers, vedastro_payload = _build_live_request(endpoint, request_preview)
req = request.Request(
@@ -2024,9 +2031,15 @@ def _post_json(endpoint: str, request_preview: dict[str, Any]) -> dict[str, Any]
headers=headers,
method="POST",
)
with request.urlopen(req, timeout=_timeout_seconds()) as resp:
raw = resp.read().decode("utf-8")
return json.loads(raw)
try:
with request.urlopen(req, timeout=_timeout_seconds()) as resp:
raw = resp.read().decode("utf-8")
except error.URLError:
raise
except _TRANSPORT_READ_ERRORS as exc:
# Every caller already turns URLError into network_error.
raise error.URLError(f"response_read_failed:{type(exc).__name__}") from exc
return json.loads(raw)
def _retry_status_codes() -> set[int]:
@@ -0,0 +1,344 @@
"""BUG-1508 and BUG-1504: VedAstro can no longer stall or break a report.
BUG-1508 (outage audit gap 2): a report now waits for VedAstro only up to its
deadline, then goes on with VedAstro marked blocked and remembers that for the
day, so the next report does not wait again.
BUG-1504: a response cut while it is being read is a network error, like the
other transport failures, and the main-entry overview never turns into a 500.
VedAstro is always a stub here; nothing leaves the machine.
"""
from __future__ import annotations
import http.client
import io
import socket
import ssl
import threading
import time
from urllib import error
import pytest
import scripts.vedastro_foreground as foreground
import scripts.vedastro_service_adapter as adapter
from scripts.jyotish_api_server import JyotishComputeService
from scripts.vedastro_consultation_snapshot import lookup_negative_snapshot
from scripts.vedastro_rest_bridge import probe_calculate_health
from scripts.vedastro_runtime_context import timeout_override_seconds
from tests.test_api_heavy_compute_gate import _PostCaptureHandler
TODAY = "2026-11-02"
BIRTH = {"year": 1990, "month": 6, "day": 15, "hour": 10, "minute": 20, "lat": 31.2, "lon": 121.5, "tz": 8,
"ayanamsa": "lahiri", "today": TODAY}
REPORT = {**BIRTH, "vedastro_evidence_budget": "report"}
BLOCKED = {"scope": "vedastro_gateway_run", "status": "official_blocked", "official_closure_state": "official_blocked",
"official_closure_reason": "official_raw_response_missing"}
VERIFIED = {"scope": "vedastro_gateway_run", "status": "ok", "official_closure_state": "official_verified",
"official_closure_reason": "official_raw_response_present", "official_raw_response": {"ok": True}}
class Gateway:
"""Stands in for the handler's gateway call; `hang` blocks until released."""
def __init__(self, result=None, *, hang=False, raises=None, delay=0.0):
self.result, self.hang, self.raises, self.delay = result, hang, raises, delay
self.calls = 0
self.release = threading.Event()
self.timeouts: list[float | None] = []
def _compute_vedastro_gateway_run(self, body):
self.calls += 1
self.timeouts.append(timeout_override_seconds())
if self.hang:
self.release.wait(5)
time.sleep(self.delay)
if self.raises is not None:
raise self.raises
return dict(self.result or BLOCKED)
def _report(handler, body=REPORT, *, pause=0.0):
started = time.perf_counter()
session = foreground.start_foreground_vedastro(
handler, body, defer_optional_external_evidence=foreground.defers_optional_vedastro(body),
)
time.sleep(pause) # local compute runs here in the real workflow
gateway = foreground.finish_foreground_vedastro(session)
return gateway, time.perf_counter() - started, session.meta
def _drain():
deadline = time.time() + 5
while time.time() < deadline and not foreground._FOREGROUND_VEDASTRO_EXECUTOR._work_queue.empty():
time.sleep(0.01)
time.sleep(0.05)
@pytest.fixture
def short_deadline(monkeypatch):
monkeypatch.setenv("JYOTISH_REPORT_VEDASTRO_DEADLINE_SECONDS", "0.4")
# ---- BUG-1508: deadline, blocked, remembered --------------------------------
def test_a_report_budget_defers_and_a_plain_request_still_does_not():
assert foreground.defers_optional_vedastro(REPORT)
assert foreground.defers_optional_vedastro({**BIRTH, "defer_optional_external_evidence": True})
assert not foreground.defers_optional_vedastro(BIRTH)
def test_a_hung_vedastro_stops_the_report_at_its_deadline_and_the_next_report_does_not_wait(short_deadline):
hung = Gateway(hang=True)
try:
gateway, seconds, meta = _report(hung)
assert meta["freshness"] == "miss"
assert 0.3 <= seconds < 0.9, seconds
assert gateway["official_closure_state"] == "official_blocked"
assert gateway["official_closure_reason"] == foreground.REPORT_VEDASTRO_DEADLINE_EXCEEDED
remembered = lookup_negative_snapshot(REPORT, today=TODAY)
assert remembered["official_closure_reason"] == foreground.REPORT_VEDASTRO_DEADLINE_EXCEEDED
again, again_seconds, again_meta = _report(hung)
assert again_seconds < 0.15, again_seconds
assert again_meta["freshness"] == "negative"
assert again == gateway
assert hung.calls == 1, "the second report did not ask VedAstro again"
finally:
hung.release.set()
_drain()
def test_the_deadline_counts_from_the_start_so_local_compute_overlaps_it(short_deadline):
hung = Gateway(hang=True)
try:
_, seconds, _ = _report(hung, pause=0.3)
assert seconds < 0.7, f"waited the whole deadline again after local compute: {seconds}"
finally:
hung.release.set()
_drain()
def test_a_gateway_that_finishes_after_the_deadline_replaces_the_deadline_entry(short_deadline):
late = Gateway(VERIFIED, hang=True)
gateway, _, _ = _report(late)
assert gateway["official_closure_reason"] == foreground.REPORT_VEDASTRO_DEADLINE_EXCEEDED
late.release.set()
_drain()
again, _, meta = _report(Gateway(BLOCKED))
assert meta["freshness"] == "fresh", "a later success wins over the deadline entry"
assert again["official_closure_state"] == "official_verified"
def test_a_fast_5xx_failure_returns_at_once_and_is_remembered(short_deadline):
failing = Gateway({**BLOCKED, "official_closure_reason": "http_error:503"})
gateway, seconds, _ = _report(failing)
assert seconds < 0.3 and gateway["official_closure_reason"] == "http_error:503"
_drain()
again, _, meta = _report(failing)
assert failing.calls == 1 and meta["freshness"] == "negative" and again == gateway
@pytest.mark.parametrize("exc", [http.client.IncompleteRead(b"{", 10), ConnectionResetError(104, "reset")])
def test_a_gateway_that_raises_a_read_error_is_blocked_and_remembered(short_deadline, exc):
broken = Gateway(raises=exc)
gateway, _, _ = _report(broken)
assert gateway["official_closure_state"] == "official_blocked"
assert gateway["error_type"] == type(exc).__name__
_drain()
assert lookup_negative_snapshot(REPORT, today=TODAY)["error_type"] == type(exc).__name__
def test_a_healthy_vedastro_inside_the_deadline_is_used(short_deadline):
healthy = Gateway(VERIFIED, delay=0.05)
gateway, _, _ = _report(healthy)
assert gateway == VERIFIED
assert lookup_negative_snapshot(REPORT, today=TODAY) is None
def test_a_report_keeps_the_configured_request_timeout_and_never_raises_it(monkeypatch):
monkeypatch.setenv("VEDASTRO_TIMEOUT_SECONDS", "20")
seen = Gateway(VERIFIED)
_report(seen)
monkeypatch.setenv("VEDASTRO_TIMEOUT_SECONDS", "120")
monkeypatch.setenv("JYOTISH_REPORT_VEDASTRO_DEADLINE_SECONDS", "25")
_report(Gateway(VERIFIED), body={**REPORT, "today": "2026-11-03"})
_drain()
assert seen.timeouts == [20.0]
assert foreground._report_vedastro_budget_seconds() == 50.0, "capped at k * deadline"
def test_the_deadline_has_a_cap_well_inside_the_frontend_wait(monkeypatch):
monkeypatch.setenv("JYOTISH_REPORT_VEDASTRO_DEADLINE_SECONDS", "600")
assert foreground._report_vedastro_deadline_seconds() == foreground.REPORT_VEDASTRO_DEADLINE_SECONDS_MAX
assert foreground.REPORT_VEDASTRO_DEADLINE_SECONDS_MAX <= 45, "frontend waits 90 s per workflow call"
def test_a_consultation_join_timeout_is_still_not_remembered(monkeypatch):
monkeypatch.setenv("JYOTISH_FOREGROUND_VEDASTRO_JOIN_SECONDS", "0.05")
hung = Gateway(hang=True)
try:
gateway, _, _ = _report(hung, body={**BIRTH, "defer_optional_external_evidence": True})
assert gateway["official_closure_reason"] == "foreground_optional_evidence_timeout"
assert lookup_negative_snapshot(BIRTH, today=TODAY) is None
finally:
hung.release.set()
_drain()
def test_the_report_workflow_skips_the_overview_and_marks_vedastro_unavailable(short_deadline, monkeypatch):
monkeypatch.setenv("JYOTISH_API_CHART_CACHE_TTL_SECONDS", "0")
hung = Gateway(hang=True)
service = JyotishComputeService()
monkeypatch.setattr(JyotishComputeService, "_compute_vedastro_gateway_run",
lambda self, body: hung._compute_vedastro_gateway_run(body))
def no_overview(*args, **kwargs):
raise AssertionError("a report must not run the synchronous main-entry overview")
monkeypatch.setattr("scripts.jyotish_api_server._attach_vedastro_main_entry_overview", no_overview)
body = {**REPORT, "question": "请为个人报告计算 career 主题证据", "theme": ["career"], "entry_mode": "direct_chart"}
try:
result = service._compute_consultation_workflow(dict(body))
sections = result["machine_evidence_packet"]["sections"]
assert sections["external_oracle_status"]["status"] == "official_blocked"
assert sections["vedastro_official_raw_response"]["status"] == "missing"
cross_check = result["consumer_context"]["vedastro_cross_check"]
assert cross_check["official_closure_reason"] == foreground.REPORT_VEDASTRO_DEADLINE_EXCEEDED
started = time.perf_counter()
service._compute_consultation_workflow(dict(body))
assert hung.calls == 1, "the second report answered from the remembered failure"
assert time.perf_counter() - started < 30
finally:
hung.release.set()
_drain()
# ---- BUG-1504: truncated responses are network errors -----------------------
class _CutResponse:
def __init__(self, exc):
self.exc = exc
def __enter__(self):
return self
def __exit__(self, *exc_info):
return False
def read(self):
raise self.exc
def _urlopen_raising(exc, *, on_read):
def fake(req, timeout=None, *args, **kwargs):
if on_read:
return _CutResponse(exc)
raise exc
return fake
READ_ERRORS = [
pytest.param(http.client.IncompleteRead(b'{"Status":', 200), True, id="incomplete-read"),
pytest.param(ConnectionResetError(104, "Connection reset by peer"), True, id="reset-on-read"),
pytest.param(ConnectionResetError(104, "Connection reset by peer"), False, id="reset-on-headers"),
pytest.param(http.client.RemoteDisconnected("closed"), False, id="remote-disconnected"),
pytest.param(ssl.SSLError("bad record mac"), True, id="ssl"),
]
@pytest.fixture
def stub_network(monkeypatch):
monkeypatch.setenv("VEDASTRO_API_ENDPOINT", "https://oracle.stub.invalid/api")
monkeypatch.setenv("VEDASTRO_ENABLE_NETWORK", "1")
monkeypatch.setenv("VEDASTRO_CACHE_TTL_SECONDS", "0")
monkeypatch.setenv("VEDASTRO_RETRY_BACKOFF_SECONDS", "0")
monkeypatch.setenv("JYOTISH_API_CHART_CACHE_TTL_SECONDS", "0")
empty = {"available": False, "status": "stubbed", "snapshot_sections": {}, "section_statuses": {}}
monkeypatch.setattr(adapter, "_try_official_capability_runner_snapshot_bundle", lambda case: dict(empty))
monkeypatch.setattr(adapter, "_try_official_python_bridge_snapshot_bundle", lambda case: dict(empty))
monkeypatch.setattr(adapter, "_try_official_full_capability_catalog_bundle", lambda case: dict(empty))
monkeypatch.setattr(adapter, "_load_cached_payload", lambda key: (None, None))
monkeypatch.setattr(adapter, "_write_artifact", lambda result: None)
def _case():
return {**BIRTH, "case_id": "stub", "reference_date": TODAY}
@pytest.mark.parametrize("exc,on_read", READ_ERRORS)
def test_a_cut_response_is_a_network_error_in_the_snapshot_and_the_range_scan(stub_network, monkeypatch, exc, on_read):
monkeypatch.setattr(adapter.request, "urlopen", _urlopen_raising(exc, on_read=on_read))
with pytest.raises(error.URLError):
adapter._post_json("https://oracle.stub.invalid/api", {"operation": "official_full_snapshot",
"section": "x", "official_request_profile": {}})
snapshot = adapter._run_official_full_snapshot_case(_case(), case_id="stub")
statuses = [value for value in snapshot["section_statuses"].values() if isinstance(value, str)]
expected = f"network_error:response_read_failed:{type(exc).__name__}"
assert expected in statuses, statuses
assert "ok" not in statuses and snapshot["status"] != "ok"
scan = adapter._run_range_scan_case(_case(), "career", "2026-11-01", "2026-11-30")
assert scan["status"] == "network_error" and scan["available"] is False
def test_a_5xx_is_still_an_http_error(stub_network, monkeypatch):
def fake(req, timeout=None, *args, **kwargs):
raise error.HTTPError(req.full_url, 503, "Service Unavailable", {}, io.BytesIO(b""))
monkeypatch.setattr(adapter.request, "urlopen", fake)
snapshot = adapter._run_official_full_snapshot_case(_case(), case_id="stub")
assert any(value == "http_error:503" for value in snapshot["section_statuses"].values())
def test_a_read_timeout_is_still_a_timeout(stub_network, monkeypatch):
monkeypatch.setattr(adapter.request, "urlopen", _urlopen_raising(socket.timeout("timed out"), on_read=True))
scan = adapter._run_range_scan_case(_case(), "career", "2026-11-01", "2026-11-30")
assert scan["status"] == "timeout"
def _chart_request():
return _PostCaptureHandler("/api/chart", {**BIRTH, "date": "1990-06-15", "time": "10:20"})
@pytest.mark.parametrize("exc,on_read", READ_ERRORS[:2])
def test_api_chart_with_the_overview_returns_200_when_vedastro_cuts_the_response(stub_network, monkeypatch, exc, on_read):
monkeypatch.setattr(adapter.request, "urlopen", _urlopen_raising(exc, on_read=on_read))
handler = _chart_request()
handler.do_POST()
assert handler.status_code == 200, handler.wfile.getvalue()[:300]
overview = handler.payload()["modules"]["vedastro_range_scan_result"]
assert overview["status"] != "ok"
assert overview["official_full_snapshot"]["status"] != "ok"
def test_api_chart_overview_attach_fails_soft_on_any_error(monkeypatch):
monkeypatch.setenv("JYOTISH_API_CHART_CACHE_TTL_SECONDS", "0")
from scripts.jyotish_api_server import _load_local_module
orchestrator = _load_local_module("vedastro_evidence_orchestrator")
def cut(*args, **kwargs):
raise http.client.IncompleteRead(b"", 5)
monkeypatch.setattr(orchestrator, "orchestrate_vedastro_evidence", cut)
handler = _chart_request()
handler.do_POST()
assert handler.status_code == 200, handler.wfile.getvalue()[:300]
overview = handler.payload()["modules"]["vedastro_range_scan_result"]
assert overview["status"] == "blocked"
assert overview["reason"] == "vedastro_main_entry_overview_error"
assert overview["error_type"] == "IncompleteRead"
def test_the_rest_health_probe_blocks_on_a_cut_body():
def opener(req, timeout=None):
return _CutResponse(http.client.IncompleteRead(b"{", 9))
result = probe_calculate_health(opener=opener)
assert result["status"] == "official_blocked" and result["reason"] == "IncompleteRead"