docs(staging): close bounded fetch incident
This commit is contained in:
+3
-3
@@ -2227,7 +2227,7 @@
|
||||
|
||||
## BUG-129 | staging trusted-main checkout 无界 fetch 导致自动部署长期占用 mutation queue
|
||||
|
||||
- 状态:investigating
|
||||
- 状态:resolved
|
||||
- 首次发现:2026-08-06
|
||||
- 最近更新:2026-08-06
|
||||
- 影响面:Gitea staging deploy/migration 控制器的 trusted-main checkout;production 与 staging 应用数据面未受影响。
|
||||
@@ -2235,8 +2235,8 @@
|
||||
- 触发条件:Gitea runner 在空仓库中抓取 reviewed `main` 和目标 SHA 时传输长时间停滞;底层远端/网络原因未闭环。checkout step 没有命令级 timeout/retry,只依赖 30 分钟 job 总超时。
|
||||
- 根因:已确认的控制面根因是发布控制器对镜像拉取有 bounded retry,却对 trusted-main Git fetch 采用无界单次调用;无论底层传输为何停滞,该调用都会长期占用 `staging-mutation` queue,并推迟后续受控重试。底层 Git 传输停滞原因保持 blocked,不作推断。
|
||||
- 修复:Gitea deploy 与 migration 的 trusted-main fetch 改为最多 3 次、每次 120 秒的 bounded retry,重试间隔递增;耗尽后明确 fail closed。仍只 checkout `origin/main`,仍要求目标 SHA 是 reviewed main ancestor,不改变 exact-SHA artifact、forward-only 或 manual rollback 边界。
|
||||
- 验证:待本地 workflow contract、Gitea PR quality gate、原 run `1474` 终止以及修复后 exact-SHA staging deploy 完成;未完成前不得标记 resolved。
|
||||
- 验证:原 run `1474` 最终成功部署 `02cc483b7c303e6cc0f26fb31462c50adb007f12`,公网 health、主机 state 与 `main/staging` 精确一致,5 个容器 restart count 均为 0;本地 staging workflow contracts 31/31、修改后 YAML 解析、mandatory pre-work 和 diff 检查通过;Gitea PR quality gate run `1475` 成功。修复后 exact-SHA staging 控制面部署仍按正常发布流程单独核对,不将原 run 的自行恢复冒充为 bounded retry 已在真实故障中触发。
|
||||
- 防复发:所有 release-controller 网络调用必须同时具备命令级上限和失败闭合;不得仅依赖 job 总 timeout。回归测试必须同时覆盖 deploy 和 migration 的 attempt 数、单次 timeout、最终错误和 ancestry check。
|
||||
- 相关记录:BUG-128、ERR-094、ERR-095
|
||||
- 复发自:无
|
||||
- 修复版本:待控制面修复 PR 与 staging 验收
|
||||
- 修复版本:`52b467cbe56f820a6acc19bb3fe3ebbf4ad55776`;待合并后由 staging exact-SHA 发布采用
|
||||
|
||||
@@ -9,7 +9,8 @@ This record contains only release identities, aggregate operational evidence, sc
|
||||
- Read-only upstream tree SHA-256: `9034e1967032d09c7fbae83fc2205f7e75e8ad482c5f9eba1bf309fe30aef5bb`
|
||||
- Personal-report implementation merge: `e018dc90a73d49596563b5ae2b5fc203402cc86a`
|
||||
- Security/control-plane merge: `f7a615a5bf11ed95b3a6c7e6d28dfe8150a825ef`
|
||||
- Staging migration and application deployment SHA: `f7a615a5bf11ed95b3a6c7e6d28dfe8150a825ef`
|
||||
- Staging migration and application-under-test SHA: `f7a615a5bf11ed95b3a6c7e6d28dfe8150a825ef`
|
||||
- Docs-only acceptance attestation deployment SHA: `02cc483b7c303e6cc0f26fb31462c50adb007f12`
|
||||
- Application rollback target: `49da8f916960030d5760d8dedf4e77820732a527`, subject to retained successful gate artifacts. The additive database migration remains in place after application rollback.
|
||||
|
||||
At deployment verification, Gitea `main`, Gitea `staging`, `/opt/jyotisha-staging/.state/deployed-revision`, and public `/api/health` all reported the same full SHA.
|
||||
@@ -37,7 +38,9 @@ The complete frontend suite could not be made fully executable on the local macO
|
||||
- Final `staging` push quality gate `1469`: success for full SHA `f7a615a5bf11ed95b3a6c7e6d28dfe8150a825ef`; immutable API/web manifest published.
|
||||
- Automatic deploy check `1470`: stopped safely with exit 3 because the report migration was pending. Its logs showed the base64 SSH secret as masked and no private-key header/material.
|
||||
- Manual migration `1471`: success; `20260806000000_personal_reports.sql` applied and present once in the migration ledger.
|
||||
- Manual deploy `1472`: success with the same exact SHA and `allow_rollback=false`.
|
||||
- Manual deploy `1472`: success with the same exact application-under-test SHA and `allow_rollback=false`.
|
||||
- Docs-only attestation gate `1473`: success for `02cc483b7c303e6cc0f26fb31462c50adb007f12`; immutable manifest published.
|
||||
- Docs-only attestation deploy `1474`: success. The trusted-main fetch paused for an extended period before recovering; no SSH/staging mutation occurred during the pause. Public health and host state then moved to the exact attestation SHA with zero container restarts. See `BUG-129` / `ERR-095` for the bounded-fetch follow-up.
|
||||
|
||||
## Security incident and containment
|
||||
|
||||
|
||||
@@ -145,7 +145,7 @@ A failed exact-SHA staging deploy displayed the multiline staging SSH private ke
|
||||
|
||||
Prevention: store `STAGING_SSH_PRIVATE_KEY` only as one unwrapped base64 line; workflows decode it into a mode-`0600` temporary key, validate it with `ssh-keygen`, and delete the temporary directory on every exit. Contract tests must reject direct multiline `SSH_PRIVATE_KEY` injection or `printf` of a decoded secret variable. A leaked staging key must be rotated and revoked before any rerun; production keys remain a separate boundary and were not involved in this incident.
|
||||
|
||||
## ERR-095 | Gitea trusted-main fetch can stall until the whole deploy job times out | investigating 2026-08-06
|
||||
## ERR-095 | Gitea trusted-main fetch can stall until the whole deploy job times out | mitigated 2026-08-06
|
||||
|
||||
After exact-SHA staging gate `1473` succeeded, automatic deploy `1474` stopped making log progress for an extended period in the empty-repository `git fetch --no-tags origin main "$DEPLOY_SHA"` step before any SSH or staging mutation. Public and state SHAs remained on the previous healthy release during the stall. The fetch later recovered and the 18-minute run successfully deployed the exact SHA, but the checkout had no command-level bound; without recovery, the 30-minute job timeout was the only escape and the shared mutation queue would remain occupied. The underlying Git transport stall is not proven.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user