From 52b467cbe56f820a6acc19bb3fe3ebbf4ad55776 Mon Sep 17 00:00:00 2001 From: Jesse Date: Thu, 6 Aug 2026 17:43:06 +0800 Subject: [PATCH] docs(bugs): clarify recovered staging fetch stall --- docs/BUG_HISTORY.md | 6 +++--- docs/research/pre_work_error_ledger.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/BUG_HISTORY.md b/docs/BUG_HISTORY.md index 967278be..0f187f4f 100644 --- a/docs/BUG_HISTORY.md +++ b/docs/BUG_HISTORY.md @@ -2231,9 +2231,9 @@ - 首次发现：2026-08-06 - 最近更新：2026-08-06 - 影响面：Gitea staging deploy/migration 控制器的 trusted-main checkout；production 与 staging 应用数据面未受影响。 -- 用户现象：exact-SHA quality gate run `1473` 成功后，自动 deploy run `1474` 超过通常部署时长仍停在 `git fetch --no-tags origin main "$DEPLOY_SHA"`；日志在远端对象压缩阶段停止推进，staging 公网与 `.state/deployed-revision` 均保持上一健康 SHA，SSH/远端 mutation 尚未开始。 -- 触发条件：Gitea runner 在空仓库中抓取 reviewed `main` 和目标 SHA 时，Git HTTP 传输停滞；checkout step 没有命令级 timeout/retry，只依赖 30 分钟 job 总超时。 -- 根因：发布控制器对镜像拉取有 bounded retry，却对 trusted-main Git fetch 采用无界单次调用；短暂远端/runner 传输停滞会长期占用 `staging-mutation` queue，并推迟后续受控重试。 +- 用户现象：exact-SHA quality gate run `1473` 成功后，自动 deploy run `1474` 在 `git fetch --no-tags origin main "$DEPLOY_SHA"` 长时间没有日志进展；期间 staging 公网与 `.state/deployed-revision` 均保持上一健康 SHA，SSH/远端 mutation 尚未开始。fetch 后续自行恢复，run 最终于 18 分钟完成并成功部署 `02cc483b7c303e6cc0f26fb31462c50adb007f12`，5 个容器 restart count 均为 0。 +- 触发条件：Gitea runner 在空仓库中抓取 reviewed `main` 和目标 SHA 时传输长时间停滞；底层远端/网络原因未闭环。checkout step 没有命令级 timeout/retry，只依赖 30 分钟 job 总超时。 +- 根因：已确认的控制面根因是发布控制器对镜像拉取有 bounded retry，却对 trusted-main Git fetch 采用无界单次调用；无论底层传输为何停滞，该调用都会长期占用 `staging-mutation` queue，并推迟后续受控重试。底层 Git 传输停滞原因保持 blocked，不作推断。 - 修复：Gitea deploy 与 migration 的 trusted-main fetch 改为最多 3 次、每次 120 秒的 bounded retry，重试间隔递增；耗尽后明确 fail closed。仍只 checkout `origin/main`，仍要求目标 SHA 是 reviewed main ancestor，不改变 exact-SHA artifact、forward-only 或 manual rollback 边界。 - 验证：待本地 workflow contract、Gitea PR quality gate、原 run `1474` 终止以及修复后 exact-SHA staging deploy 完成；未完成前不得标记 resolved。 - 防复发：所有 release-controller 网络调用必须同时具备命令级上限和失败闭合；不得仅依赖 job 总 timeout。回归测试必须同时覆盖 deploy 和 migration 的 attempt 数、单次 timeout、最终错误和 ancestry check。 diff --git a/docs/research/pre_work_error_ledger.md b/docs/research/pre_work_error_ledger.md index e67b52cc..d40e3172 100644 --- a/docs/research/pre_work_error_ledger.md +++ b/docs/research/pre_work_error_ledger.md @@ -147,7 +147,7 @@ Prevention: store `STAGING_SSH_PRIVATE_KEY` only as one unwrapped base64 line; w ## ERR-095 | Gitea trusted-main fetch can stall until the whole deploy job times out | investigating 2026-08-06 -After exact-SHA staging gate `1473` succeeded, automatic deploy `1474` stopped making progress in the empty-repository `git fetch --no-tags origin main "$DEPLOY_SHA"` step before any SSH or staging mutation. Public and state SHAs remained on the previous healthy release. The checkout had no command-level bound, so the 30-minute job timeout was the only escape and the shared mutation queue remained occupied. +After exact-SHA staging gate `1473` succeeded, automatic deploy `1474` stopped making log progress for an extended period in the empty-repository `git fetch --no-tags origin main "$DEPLOY_SHA"` step before any SSH or staging mutation. Public and state SHAs remained on the previous healthy release during the stall. The fetch later recovered and the 18-minute run successfully deployed the exact SHA, but the checkout had no command-level bound; without recovery, the 30-minute job timeout was the only escape and the shared mutation queue would remain occupied. The underlying Git transport stall is not proven. Prevention: wrap the Gitea deploy and migration trusted-main fetch in three 120-second bounded attempts with incremental delay and explicit fail-closed exhaustion. Preserve the reviewed-main ancestry check, exact-SHA gate artifact, forward-only deploy policy, and shared mutation queue. Never treat an in-progress or timed-out fetch as a deployment result.