test: each test gets its own foreground VedAstro pool, drained at teardown (BUG-1498)
Independent Staging Quality Gate / validate (push) Successful in 16m7s
Independent Staging Quality Gate / publish (push) Successful in 4m22s

A task left running wrote its outcome into the next test's cache directory,
so the next test saw a remembered failure and never called the gateway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
jesse
2026-10-28 19:39:28 +08:00
co-authored by Claude Opus 5.5
parent 2dea3c2994
commit 537beb2934
3 changed files with 81 additions and 14 deletions
+13 -14
View File
@@ -20463,18 +20463,17 @@
- 复发自:无。
- 修复版本:分支 `codex/rectification-recap-date-dup-20261028`。
## BUG-1498 | staging 门禁 Python 快速门偶发失败:`test_daily_starlanguage_does_not_eat_yesterday_cache` 网关调用 0 次
## BUG-1498 | staging 门禁 Python 快速门失败:`test_daily_starlanguage_does_not_eat_yesterday_cache` 网关调用 0 次
- 状态:investigating(只记录已确认事实与假设;未改代码)
- 首次发现 / 最近更新:2026-10-28(BUG-1481 验证段首次提到)/ 2026-10-28(staging 门禁 run 2017)
- 影响面:只影响 CI 门禁,不影响线上。`tests/test_vedastro_snapshot_cache.py::test_daily_starlanguage_does_not_eat_yesterday_cache`。
- 现象:`assert calls["gateway"] == 1` 得到 `0 == 1`;同一轮其余 1094 项通过。run 2016(同一份 Python 代码)这一项通过;本机单独连跑 5 次全部通过。
- 触发条件:整套快速门一起跑时偶发,单独跑不出现,跟测试先后顺序或机器负载有关。
- 已排除:BUG-1481 当时认为是测试之间共用负缓存目录;`tests/conftest.py::_isolate_vedastro_snapshot_cache` 现在每个测试都另建快照目录和负缓存目录,这次复发不是那个原因。
- 假设(未证实):`scripts/vedastro_foreground.py` 的 `_FOREGROUND_VEDASTRO_EXECUTOR` 是模块级线程池,整个 pytest 进程共用,conftest 不清空它。前面的测试如果留下还在跑的任务占满线程,本测试的任务排在后面,前台 1.5 秒等待到期后被取消,网关一次都没调用。
- 修复:未修。下一步:在 conftest 里每个测试后排空或重建这个线程池,或者本测试把等待时间设大,再在全套顺序下复跑确认。
- 验证:未完成。
- 防复发:未完成。
- 关联:BUG-1481(第一次记录这个失败)。
- 复发自:BUG-1481 记下的同一项失败。当时的处理(隔离负缓存目录)没拦住,因为根因很可能不在目录上。
- 修复版本:无。
- 状态:resolved(测试隔离 + 回归测试;以修复后第一轮 staging 门禁通过为准)
- 首次发现 / 最近更新:2026-10-28(BUG-1481 验证段首次提到)/ 2026-10-28(staging 门禁 run 2017、2018 连续两次失败)
- 影响面:只影响 CI 门禁,不影响线上。失败的是 `tests/test_vedastro_snapshot_cache.py::test_daily_starlanguage_does_not_eat_yesterday_cache`;同样的机制可能打到任何用前台 VedAstro 线程池的测试。
- 现象:`assert calls["gateway"] == 1` 得到 `0 == 1`;同一轮其余 1094 项通过。run 2016 这一项通过,run 2017、2018 失败,三次 Python 代码相同;本机单独跑或整套跑都通过。
- 触发条件:整套测试一起跑,前面某个测试留在前台线程池里的任务,在下一个测试开始之后才跑完。
- 根因:`scripts/vedastro_foreground.py::_FOREGROUND_VEDASTRO_EXECUTOR` 是模块级线程池,整个 pytest 进程共用(`jyotish_api_server` 还按名字另外导入了一份)。任务跑完后,`_remember_gateway_outcome` 把结果写进「写的那一刻」环境变量指向的缓存目录。上一个测试留下的任务跑完时,环境变量已经指向下一个测试的隔离目录,于是一次失败被记成下一个测试「今天已经失败过」,`_failed_today` 命中,网关一次都没调用。线程池被占满时,任务排队到超时被取消,也会得到同样的 0 次。#130 给每个测试单独建目录,挡不住这种跨测试写入,所以 BUG-1481 那次的判断(共用目录)不是根因。
- 修复:`tests/conftest.py::_isolate_foreground_vedastro_executor`。每个测试换一个新的前台线程池(两个模块里的名字都换);测试结束时丢掉排队的任务,正在跑的最多等 10 秒,所以剩下的任务一定在本测试的目录还有效时写完。生产代码没有改。
- 验证:新增 `tests/test_vedastro_executor_isolation_20261028.py`:第一个测试留下一个 0.3 秒的任务,第二个测试断言它已经在上一个测试收尾时跑完、而且没有写进自己的目录。没有 fixture 时红(`assert 0 == 1`),加上后绿。`test_vedastro_snapshot_cache.py`、`test_vedastro_negative_day_20261028.py`、`test_vedastro_consultation_snapshot.py` 一起 19 项通过;本机完整快速门(`run_quality_gate.py --profile quick --skip-yoga-logic --skip-frontend-runtime`)通过。
- 防复发:上面那组顺序测试固定了「任务不跨测试」。以后新增模块级线程池的,要放进同一个 fixture。
- 关联:BUG-1481(第一次记下这个失败)、BUG-1479(#130,负缓存与每测试目录)。
- 复发自:BUG-1481 记下的同一项失败。当时的处理(每测试单独目录)没拦住,因为根因是后台任务跨测试写入,不在目录共用上。
- 修复版本:分支 `codex/bug-1498-executor-isolation`。
+36
View File
@@ -3,6 +3,7 @@
from __future__ import annotations
import sys
import time
from pathlib import Path
import pytest
@@ -58,6 +59,41 @@ def _isolate_vedastro_snapshot_cache(tmp_path_factory, monkeypatch) -> None:
clear_validate_memory()
_FOREGROUND_EXECUTOR_MODULES = (
"scripts.vedastro_foreground",
"vedastro_foreground",
"scripts.jyotish_api_server",
"jyotish_api_server",
)
@pytest.fixture(autouse=True)
def _isolate_foreground_vedastro_executor(monkeypatch):
"""A fresh foreground pool per test, drained at teardown (BUG-1498).
A task left running stores its outcome under whatever cache directory the
environment names when it finishes, which would be the next test's.
"""
from concurrent.futures import ThreadPoolExecutor
from scripts import vedastro_foreground
executor = ThreadPoolExecutor(
max_workers=vedastro_foreground._FOREGROUND_VEDASTRO_WORKERS,
thread_name_prefix="jyotish-vedastro-fg-test",
)
for name in _FOREGROUND_EXECUTOR_MODULES:
module = sys.modules.get(name)
if module is not None and hasattr(module, "_FOREGROUND_VEDASTRO_EXECUTOR"):
monkeypatch.setattr(module, "_FOREGROUND_VEDASTRO_EXECUTOR", executor)
yield
# Queued work is dropped; running work gets a bounded wait so a stuck stub cannot hang the run.
executor.shutdown(wait=False, cancel_futures=True)
deadline = time.monotonic() + 10
for thread in list(getattr(executor, "_threads", ())):
thread.join(max(deadline - time.monotonic(), 0))
def pytest_collection_modifyitems(items) -> None:
for item in items:
if item.fspath.basename == "test_api_server_security.py" and item.name.startswith(SLOW_API_SECURITY_PREFIXES):
@@ -0,0 +1,32 @@
"""BUG-1498: a foreground VedAstro task never outlives the test that started it.
The two tests run in file order. The first leaves a slow task on the foreground
executor; the second checks that the task finished before it began, so it could
not write into the second test's cache directory.
"""
from __future__ import annotations
import os
import time
from scripts import vedastro_foreground as foreground
_SEEN: list[str] = []
def _slow_task() -> None:
time.sleep(0.3)
_SEEN.append(os.environ.get("JYOTISH_VEDASTRO_SNAPSHOT_NEGATIVE_CACHE_DIR", ""))
def test_a_leaves_slow_work_on_the_foreground_executor() -> None:
_SEEN.clear()
foreground._FOREGROUND_VEDASTRO_EXECUTOR.submit(_slow_task)
def test_b_does_not_see_the_previous_tests_work() -> None:
mine = os.environ["JYOTISH_VEDASTRO_SNAPSHOT_NEGATIVE_CACHE_DIR"]
assert len(_SEEN) == 1, "the earlier test's task finished during its own teardown"
time.sleep(0.5)
assert mine not in _SEEN, "a leftover task wrote into this test's cache directory"