From 975f5c346d67e31340686af587330bbf345df4bf Mon Sep 17 00:00:00 2001 From: Jesse Date: Thu, 6 Aug 2026 17:40:41 +0800 Subject: [PATCH 1/3] fix(staging): bound trusted main fetch retries --- .gitea/workflows/deploy-staging.yml | 14 +++++++++++++- .gitea/workflows/migrate-staging-database.yml | 14 +++++++++++++- docs/BUG_HISTORY.md | 18 +++++++++++++++++- docs/research/pre_work_error_ledger.md | 6 ++++++ .../tests/staging-backend-workflows.test.ts | 5 ++++- 5 files changed, 53 insertions(+), 4 deletions(-) diff --git a/.gitea/workflows/deploy-staging.yml b/.gitea/workflows/deploy-staging.yml index d5378f57..db6f873e 100644 --- a/.gitea/workflows/deploy-staging.yml +++ b/.gitea/workflows/deploy-staging.yml @@ -96,7 +96,19 @@ jobs: git init . git remote remove origin 2>/dev/null || true git remote add origin https://git.copse.top/root/Jyotisha.git - git fetch --no-tags origin main "$DEPLOY_SHA" + fetch_succeeded=false + for attempt in 1 2 3; do + if timeout 120 git fetch --no-tags origin main "$DEPLOY_SHA"; then + fetch_succeeded=true + break + fi + if [ "$attempt" -eq 3 ]; then + echo "trusted main fetch failed after $attempt bounded attempts" >&2 + exit 1 + fi + sleep $((attempt * 10)) + done + [[ "$fetch_succeeded" == true ]] git checkout --detach --force origin/main git merge-base --is-ancestor "$DEPLOY_SHA" HEAD || { echo "staging revision is not in trusted main history" >&2; exit 1; } diff --git a/.gitea/workflows/migrate-staging-database.yml b/.gitea/workflows/migrate-staging-database.yml index 0b4e750d..67f2f1c1 100644 --- a/.gitea/workflows/migrate-staging-database.yml +++ b/.gitea/workflows/migrate-staging-database.yml @@ -68,7 +68,19 @@ jobs: git init . git remote remove origin 2>/dev/null || true git remote add origin https://git.copse.top/root/Jyotisha.git - git fetch --no-tags origin main "$DEPLOY_SHA" + fetch_succeeded=false + for attempt in 1 2 3; do + if timeout 120 git fetch --no-tags origin main "$DEPLOY_SHA"; then + fetch_succeeded=true + break + fi + if [ "$attempt" -eq 3 ]; then + echo "trusted main fetch failed after $attempt bounded attempts" >&2 + exit 1 + fi + sleep $((attempt * 10)) + done + [[ "$fetch_succeeded" == true ]] git checkout --detach --force origin/main git merge-base --is-ancestor "$DEPLOY_SHA" HEAD || { echo "staging revision is not in trusted main history" >&2; exit 1; } diff --git a/docs/BUG_HISTORY.md b/docs/BUG_HISTORY.md index 5c7fa924..967278be 100644 --- a/docs/BUG_HISTORY.md +++ b/docs/BUG_HISTORY.md @@ -2223,4 +2223,20 @@ - 防复发：禁止 staging workflow 直接注入多行私钥或打印 decoded secret 变量；env owner 必须由部署树身份决定，root 受控脚本不得用 root 临时文件改变持久 env owner。任何凭据日志暴露先轮换/撤销/清理，再修代码和重跑。 - 相关记录：BUG-124、BUG-127、ERR-092、ERR-093、ERR-094 - 复发自：无 -- 修复版本：待安全修复 PR 与 staging 验收 +- 修复版本：`f7a615a5bf11ed95b3a6c7e6d28dfe8150a825ef`；staging migration/deploy 与安全验收完成 + +## BUG-129 | staging trusted-main checkout 无界 fetch 导致自动部署长期占用 mutation queue + +- 状态：investigating +- 首次发现：2026-08-06 +- 最近更新：2026-08-06 +- 影响面：Gitea staging deploy/migration 控制器的 trusted-main checkout；production 与 staging 应用数据面未受影响。 +- 用户现象：exact-SHA quality gate run `1473` 成功后，自动 deploy run `1474` 超过通常部署时长仍停在 `git fetch --no-tags origin main "$DEPLOY_SHA"`；日志在远端对象压缩阶段停止推进，staging 公网与 `.state/deployed-revision` 均保持上一健康 SHA，SSH/远端 mutation 尚未开始。 +- 触发条件：Gitea runner 在空仓库中抓取 reviewed `main` 和目标 SHA 时，Git HTTP 传输停滞；checkout step 没有命令级 timeout/retry，只依赖 30 分钟 job 总超时。 +- 根因：发布控制器对镜像拉取有 bounded retry，却对 trusted-main Git fetch 采用无界单次调用；短暂远端/runner 传输停滞会长期占用 `staging-mutation` queue，并推迟后续受控重试。 +- 修复：Gitea deploy 与 migration 的 trusted-main fetch 改为最多 3 次、每次 120 秒的 bounded retry，重试间隔递增；耗尽后明确 fail closed。仍只 checkout `origin/main`，仍要求目标 SHA 是 reviewed main ancestor，不改变 exact-SHA artifact、forward-only 或 manual rollback 边界。 +- 验证：待本地 workflow contract、Gitea PR quality gate、原 run `1474` 终止以及修复后 exact-SHA staging deploy 完成；未完成前不得标记 resolved。 +- 防复发：所有 release-controller 网络调用必须同时具备命令级上限和失败闭合；不得仅依赖 job 总 timeout。回归测试必须同时覆盖 deploy 和 migration 的 attempt 数、单次 timeout、最终错误和 ancestry check。 +- 相关记录：BUG-128、ERR-094、ERR-095 +- 复发自：无 +- 修复版本：待控制面修复 PR 与 staging 验收 diff --git a/docs/research/pre_work_error_ledger.md b/docs/research/pre_work_error_ledger.md index 92620164..e67b52cc 100644 --- a/docs/research/pre_work_error_ledger.md +++ b/docs/research/pre_work_error_ledger.md @@ -145,6 +145,12 @@ A failed exact-SHA staging deploy displayed the multiline staging SSH private ke Prevention: store `STAGING_SSH_PRIVATE_KEY` only as one unwrapped base64 line; workflows decode it into a mode-`0600` temporary key, validate it with `ssh-keygen`, and delete the temporary directory on every exit. Contract tests must reject direct multiline `SSH_PRIVATE_KEY` injection or `printf` of a decoded secret variable. A leaked staging key must be rotated and revoked before any rerun; production keys remain a separate boundary and were not involved in this incident. +## ERR-095 | Gitea trusted-main fetch can stall until the whole deploy job times out | investigating 2026-08-06 + +After exact-SHA staging gate `1473` succeeded, automatic deploy `1474` stopped making progress in the empty-repository `git fetch --no-tags origin main "$DEPLOY_SHA"` step before any SSH or staging mutation. Public and state SHAs remained on the previous healthy release. The checkout had no command-level bound, so the 30-minute job timeout was the only escape and the shared mutation queue remained occupied. + +Prevention: wrap the Gitea deploy and migration trusted-main fetch in three 120-second bounded attempts with incremental delay and explicit fail-closed exhaustion. Preserve the reviewed-main ancestry check, exact-SHA gate artifact, forward-only deploy policy, and shared mutation queue. Never treat an in-progress or timed-out fetch as a deployment result. + ## Fragment Sweep Command Set ## ERR-086 | Steve Jobs jyotishganit artifacts used non-San-Francisco coordinates | mitigated 2026-07-21 diff --git a/frontend/tests/staging-backend-workflows.test.ts b/frontend/tests/staging-backend-workflows.test.ts index cc6e9845..c34a9500 100644 --- a/frontend/tests/staging-backend-workflows.test.ts +++ b/frontend/tests/staging-backend-workflows.test.ts @@ -488,7 +488,10 @@ test("Gitea deploy and migration consume the exact successful gate artifact", () assert.match(workflow, /actions\/artifacts\/\$artifact_id\/zip/); assert.match(workflow, /node frontend\/scripts\/staging-image-manifest\.mjs/); assert.match(workflow, /name: Checkout trusted main controller/); - assert.match(workflow, /git fetch --no-tags origin main "\$DEPLOY_SHA"/); + assert.match(workflow, /for attempt in 1 2 3/); + assert.match(workflow, /timeout 120 git fetch --no-tags origin main "\$DEPLOY_SHA"/); + assert.match(workflow, /trusted main fetch failed after \$attempt bounded attempts/); + assert.match(workflow, /\[\[ "\$fetch_succeeded" == true \]\]/); assert.match(workflow, /git checkout --detach --force origin\/main/); assert.match(workflow, /git merge-base --is-ancestor "\$DEPLOY_SHA" HEAD/); assert.doesNotMatch(workflow, /docker manifest inspect/); From 52b467cbe56f820a6acc19bb3fe3ebbf4ad55776 Mon Sep 17 00:00:00 2001 From: Jesse Date: Thu, 6 Aug 2026 17:43:06 +0800 Subject: [PATCH 2/3] docs(bugs): clarify recovered staging fetch stall --- docs/BUG_HISTORY.md | 6 +++--- docs/research/pre_work_error_ledger.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/BUG_HISTORY.md b/docs/BUG_HISTORY.md index 967278be..0f187f4f 100644 --- a/docs/BUG_HISTORY.md +++ b/docs/BUG_HISTORY.md @@ -2231,9 +2231,9 @@ - 首次发现：2026-08-06 - 最近更新：2026-08-06 - 影响面：Gitea staging deploy/migration 控制器的 trusted-main checkout；production 与 staging 应用数据面未受影响。 -- 用户现象：exact-SHA quality gate run `1473` 成功后，自动 deploy run `1474` 超过通常部署时长仍停在 `git fetch --no-tags origin main "$DEPLOY_SHA"`；日志在远端对象压缩阶段停止推进，staging 公网与 `.state/deployed-revision` 均保持上一健康 SHA，SSH/远端 mutation 尚未开始。 -- 触发条件：Gitea runner 在空仓库中抓取 reviewed `main` 和目标 SHA 时，Git HTTP 传输停滞；checkout step 没有命令级 timeout/retry，只依赖 30 分钟 job 总超时。 -- 根因：发布控制器对镜像拉取有 bounded retry，却对 trusted-main Git fetch 采用无界单次调用；短暂远端/runner 传输停滞会长期占用 `staging-mutation` queue，并推迟后续受控重试。 +- 用户现象：exact-SHA quality gate run `1473` 成功后，自动 deploy run `1474` 在 `git fetch --no-tags origin main "$DEPLOY_SHA"` 长时间没有日志进展；期间 staging 公网与 `.state/deployed-revision` 均保持上一健康 SHA，SSH/远端 mutation 尚未开始。fetch 后续自行恢复，run 最终于 18 分钟完成并成功部署 `02cc483b7c303e6cc0f26fb31462c50adb007f12`，5 个容器 restart count 均为 0。 +- 触发条件：Gitea runner 在空仓库中抓取 reviewed `main` 和目标 SHA 时传输长时间停滞；底层远端/网络原因未闭环。checkout step 没有命令级 timeout/retry，只依赖 30 分钟 job 总超时。 +- 根因：已确认的控制面根因是发布控制器对镜像拉取有 bounded retry，却对 trusted-main Git fetch 采用无界单次调用；无论底层传输为何停滞，该调用都会长期占用 `staging-mutation` queue，并推迟后续受控重试。底层 Git 传输停滞原因保持 blocked，不作推断。 - 修复：Gitea deploy 与 migration 的 trusted-main fetch 改为最多 3 次、每次 120 秒的 bounded retry，重试间隔递增；耗尽后明确 fail closed。仍只 checkout `origin/main`，仍要求目标 SHA 是 reviewed main ancestor，不改变 exact-SHA artifact、forward-only 或 manual rollback 边界。 - 验证：待本地 workflow contract、Gitea PR quality gate、原 run `1474` 终止以及修复后 exact-SHA staging deploy 完成；未完成前不得标记 resolved。 - 防复发：所有 release-controller 网络调用必须同时具备命令级上限和失败闭合；不得仅依赖 job 总 timeout。回归测试必须同时覆盖 deploy 和 migration 的 attempt 数、单次 timeout、最终错误和 ancestry check。 diff --git a/docs/research/pre_work_error_ledger.md b/docs/research/pre_work_error_ledger.md index e67b52cc..d40e3172 100644 --- a/docs/research/pre_work_error_ledger.md +++ b/docs/research/pre_work_error_ledger.md @@ -147,7 +147,7 @@ Prevention: store `STAGING_SSH_PRIVATE_KEY` only as one unwrapped base64 line; w ## ERR-095 | Gitea trusted-main fetch can stall until the whole deploy job times out | investigating 2026-08-06 -After exact-SHA staging gate `1473` succeeded, automatic deploy `1474` stopped making progress in the empty-repository `git fetch --no-tags origin main "$DEPLOY_SHA"` step before any SSH or staging mutation. Public and state SHAs remained on the previous healthy release. The checkout had no command-level bound, so the 30-minute job timeout was the only escape and the shared mutation queue remained occupied. +After exact-SHA staging gate `1473` succeeded, automatic deploy `1474` stopped making log progress for an extended period in the empty-repository `git fetch --no-tags origin main "$DEPLOY_SHA"` step before any SSH or staging mutation. Public and state SHAs remained on the previous healthy release during the stall. The fetch later recovered and the 18-minute run successfully deployed the exact SHA, but the checkout had no command-level bound; without recovery, the 30-minute job timeout was the only escape and the shared mutation queue would remain occupied. The underlying Git transport stall is not proven. Prevention: wrap the Gitea deploy and migration trusted-main fetch in three 120-second bounded attempts with incremental delay and explicit fail-closed exhaustion. Preserve the reviewed-main ancestry check, exact-SHA gate artifact, forward-only deploy policy, and shared mutation queue. Never treat an in-progress or timed-out fetch as a deployment result. From fa8698c95b7e9a69dd5a754612ee5a696d7dfda8 Mon Sep 17 00:00:00 2001 From: Jesse Date: Thu, 6 Aug 2026 18:00:40 +0800 Subject: [PATCH 3/3] docs(staging): close bounded fetch incident --- docs/BUG_HISTORY.md | 6 +++--- .../personal-report-staging-acceptance-2026-08-06.md | 7 +++++-- docs/research/pre_work_error_ledger.md | 2 +- 3 files changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/BUG_HISTORY.md b/docs/BUG_HISTORY.md index 0f187f4f..e81df7d0 100644 --- a/docs/BUG_HISTORY.md +++ b/docs/BUG_HISTORY.md @@ -2227,7 +2227,7 @@ ## BUG-129 | staging trusted-main checkout 无界 fetch 导致自动部署长期占用 mutation queue -- 状态：investigating +- 状态：resolved - 首次发现：2026-08-06 - 最近更新：2026-08-06 - 影响面：Gitea staging deploy/migration 控制器的 trusted-main checkout；production 与 staging 应用数据面未受影响。 @@ -2235,8 +2235,8 @@ - 触发条件：Gitea runner 在空仓库中抓取 reviewed `main` 和目标 SHA 时传输长时间停滞；底层远端/网络原因未闭环。checkout step 没有命令级 timeout/retry，只依赖 30 分钟 job 总超时。 - 根因：已确认的控制面根因是发布控制器对镜像拉取有 bounded retry，却对 trusted-main Git fetch 采用无界单次调用；无论底层传输为何停滞，该调用都会长期占用 `staging-mutation` queue，并推迟后续受控重试。底层 Git 传输停滞原因保持 blocked，不作推断。 - 修复：Gitea deploy 与 migration 的 trusted-main fetch 改为最多 3 次、每次 120 秒的 bounded retry，重试间隔递增；耗尽后明确 fail closed。仍只 checkout `origin/main`，仍要求目标 SHA 是 reviewed main ancestor，不改变 exact-SHA artifact、forward-only 或 manual rollback 边界。 -- 验证：待本地 workflow contract、Gitea PR quality gate、原 run `1474` 终止以及修复后 exact-SHA staging deploy 完成；未完成前不得标记 resolved。 +- 验证：原 run `1474` 最终成功部署 `02cc483b7c303e6cc0f26fb31462c50adb007f12`，公网 health、主机 state 与 `main/staging` 精确一致，5 个容器 restart count 均为 0；本地 staging workflow contracts 31/31、修改后 YAML 解析、mandatory pre-work 和 diff 检查通过；Gitea PR quality gate run `1475` 成功。修复后 exact-SHA staging 控制面部署仍按正常发布流程单独核对，不将原 run 的自行恢复冒充为 bounded retry 已在真实故障中触发。 - 防复发：所有 release-controller 网络调用必须同时具备命令级上限和失败闭合；不得仅依赖 job 总 timeout。回归测试必须同时覆盖 deploy 和 migration 的 attempt 数、单次 timeout、最终错误和 ancestry check。 - 相关记录：BUG-128、ERR-094、ERR-095 - 复发自：无 -- 修复版本：待控制面修复 PR 与 staging 验收 +- 修复版本：`52b467cbe56f820a6acc19bb3fe3ebbf4ad55776`；待合并后由 staging exact-SHA 发布采用 diff --git a/docs/operations/personal-report-staging-acceptance-2026-08-06.md b/docs/operations/personal-report-staging-acceptance-2026-08-06.md index 69f764cd..5464abd4 100644 --- a/docs/operations/personal-report-staging-acceptance-2026-08-06.md +++ b/docs/operations/personal-report-staging-acceptance-2026-08-06.md @@ -9,7 +9,8 @@ This record contains only release identities, aggregate operational evidence, sc - Read-only upstream tree SHA-256: `9034e1967032d09c7fbae83fc2205f7e75e8ad482c5f9eba1bf309fe30aef5bb` - Personal-report implementation merge: `e018dc90a73d49596563b5ae2b5fc203402cc86a` - Security/control-plane merge: `f7a615a5bf11ed95b3a6c7e6d28dfe8150a825ef` -- Staging migration and application deployment SHA: `f7a615a5bf11ed95b3a6c7e6d28dfe8150a825ef` +- Staging migration and application-under-test SHA: `f7a615a5bf11ed95b3a6c7e6d28dfe8150a825ef` +- Docs-only acceptance attestation deployment SHA: `02cc483b7c303e6cc0f26fb31462c50adb007f12` - Application rollback target: `49da8f916960030d5760d8dedf4e77820732a527`, subject to retained successful gate artifacts. The additive database migration remains in place after application rollback. At deployment verification, Gitea `main`, Gitea `staging`, `/opt/jyotisha-staging/.state/deployed-revision`, and public `/api/health` all reported the same full SHA. @@ -37,7 +38,9 @@ The complete frontend suite could not be made fully executable on the local macO - Final `staging` push quality gate `1469`: success for full SHA `f7a615a5bf11ed95b3a6c7e6d28dfe8150a825ef`; immutable API/web manifest published. - Automatic deploy check `1470`: stopped safely with exit 3 because the report migration was pending. Its logs showed the base64 SSH secret as masked and no private-key header/material. - Manual migration `1471`: success; `20260806000000_personal_reports.sql` applied and present once in the migration ledger. -- Manual deploy `1472`: success with the same exact SHA and `allow_rollback=false`. +- Manual deploy `1472`: success with the same exact application-under-test SHA and `allow_rollback=false`. +- Docs-only attestation gate `1473`: success for `02cc483b7c303e6cc0f26fb31462c50adb007f12`; immutable manifest published. +- Docs-only attestation deploy `1474`: success. The trusted-main fetch paused for an extended period before recovering; no SSH/staging mutation occurred during the pause. Public health and host state then moved to the exact attestation SHA with zero container restarts. See `BUG-129` / `ERR-095` for the bounded-fetch follow-up. ## Security incident and containment diff --git a/docs/research/pre_work_error_ledger.md b/docs/research/pre_work_error_ledger.md index d40e3172..ca331832 100644 --- a/docs/research/pre_work_error_ledger.md +++ b/docs/research/pre_work_error_ledger.md @@ -145,7 +145,7 @@ A failed exact-SHA staging deploy displayed the multiline staging SSH private ke Prevention: store `STAGING_SSH_PRIVATE_KEY` only as one unwrapped base64 line; workflows decode it into a mode-`0600` temporary key, validate it with `ssh-keygen`, and delete the temporary directory on every exit. Contract tests must reject direct multiline `SSH_PRIVATE_KEY` injection or `printf` of a decoded secret variable. A leaked staging key must be rotated and revoked before any rerun; production keys remain a separate boundary and were not involved in this incident. -## ERR-095 | Gitea trusted-main fetch can stall until the whole deploy job times out | investigating 2026-08-06 +## ERR-095 | Gitea trusted-main fetch can stall until the whole deploy job times out | mitigated 2026-08-06 After exact-SHA staging gate `1473` succeeded, automatic deploy `1474` stopped making log progress for an extended period in the empty-repository `git fetch --no-tags origin main "$DEPLOY_SHA"` step before any SSH or staging mutation. Public and state SHAs remained on the previous healthy release during the stall. The fetch later recovered and the 18-minute run successfully deployed the exact SHA, but the checkout had no command-level bound; without recovery, the 30-minute job timeout was the only escape and the shared mutation queue would remain occupied. The underlying Git transport stall is not proven.