Bound staging trusted-main fetch retries #17

Merged
jesse merged 3 commits from fix/staging-controller-fetch-timeout into main 2026-08-06 18:15:11 +08:00
5 changed files with 53 additions and 4 deletions
Showing only changes of commit 975f5c346d - Show all commits
+13 -1
View File
@@ -96,7 +96,19 @@ jobs:
git init .
git remote remove origin 2>/dev/null || true
git remote add origin https://git.copse.top/root/Jyotisha.git
git fetch --no-tags origin main "$DEPLOY_SHA"
fetch_succeeded=false
for attempt in 1 2 3; do
if timeout 120 git fetch --no-tags origin main "$DEPLOY_SHA"; then
fetch_succeeded=true
break
fi
if [ "$attempt" -eq 3 ]; then
echo "trusted main fetch failed after $attempt bounded attempts" >&2
exit 1
fi
sleep $((attempt * 10))
done
[[ "$fetch_succeeded" == true ]]
git checkout --detach --force origin/main
git merge-base --is-ancestor "$DEPLOY_SHA" HEAD || { echo "staging revision is not in trusted main history" >&2; exit 1; }
+13 -1
View File
@@ -68,7 +68,19 @@ jobs:
git init .
git remote remove origin 2>/dev/null || true
git remote add origin https://git.copse.top/root/Jyotisha.git
git fetch --no-tags origin main "$DEPLOY_SHA"
fetch_succeeded=false
for attempt in 1 2 3; do
if timeout 120 git fetch --no-tags origin main "$DEPLOY_SHA"; then
fetch_succeeded=true
break
fi
if [ "$attempt" -eq 3 ]; then
echo "trusted main fetch failed after $attempt bounded attempts" >&2
exit 1
fi
sleep $((attempt * 10))
done
[[ "$fetch_succeeded" == true ]]
git checkout --detach --force origin/main
git merge-base --is-ancestor "$DEPLOY_SHA" HEAD || { echo "staging revision is not in trusted main history" >&2; exit 1; }
+17 -1
View File
@@ -2223,4 +2223,20 @@
- 防复发:禁止 staging workflow 直接注入多行私钥或打印 decoded secret 变量;env owner 必须由部署树身份决定,root 受控脚本不得用 root 临时文件改变持久 env owner。任何凭据日志暴露先轮换/撤销/清理,再修代码和重跑。
- 相关记录:BUG-124、BUG-127、ERR-092、ERR-093、ERR-094
- 复发自:无
- 修复版本:待安全修复 PR 与 staging 验收
- 修复版本:`f7a615a5bf11ed95b3a6c7e6d28dfe8150a825ef`staging migration/deploy 与安全验收完成
## BUG-129 | staging trusted-main checkout 无界 fetch 导致自动部署长期占用 mutation queue
- 状态:investigating
- 首次发现:2026-08-06
- 最近更新:2026-08-06
- 影响面:Gitea staging deploy/migration 控制器的 trusted-main checkoutproduction 与 staging 应用数据面未受影响。
- 用户现象:exact-SHA quality gate run `1473` 成功后,自动 deploy run `1474` 超过通常部署时长仍停在 `git fetch --no-tags origin main "$DEPLOY_SHA"`;日志在远端对象压缩阶段停止推进,staging 公网与 `.state/deployed-revision` 均保持上一健康 SHASSH/远端 mutation 尚未开始。
- 触发条件:Gitea runner 在空仓库中抓取 reviewed `main` 和目标 SHA 时,Git HTTP 传输停滞;checkout step 没有命令级 timeout/retry,只依赖 30 分钟 job 总超时。
- 根因:发布控制器对镜像拉取有 bounded retry,却对 trusted-main Git fetch 采用无界单次调用;短暂远端/runner 传输停滞会长期占用 `staging-mutation` queue,并推迟后续受控重试。
- 修复:Gitea deploy 与 migration 的 trusted-main fetch 改为最多 3 次、每次 120 秒的 bounded retry,重试间隔递增;耗尽后明确 fail closed。仍只 checkout `origin/main`,仍要求目标 SHA 是 reviewed main ancestor,不改变 exact-SHA artifact、forward-only 或 manual rollback 边界。
- 验证:待本地 workflow contract、Gitea PR quality gate、原 run `1474` 终止以及修复后 exact-SHA staging deploy 完成;未完成前不得标记 resolved。
- 防复发:所有 release-controller 网络调用必须同时具备命令级上限和失败闭合;不得仅依赖 job 总 timeout。回归测试必须同时覆盖 deploy 和 migration 的 attempt 数、单次 timeout、最终错误和 ancestry check。
- 相关记录:BUG-128、ERR-094、ERR-095
- 复发自:无
- 修复版本:待控制面修复 PR 与 staging 验收
+6
View File
@@ -145,6 +145,12 @@ A failed exact-SHA staging deploy displayed the multiline staging SSH private ke
Prevention: store `STAGING_SSH_PRIVATE_KEY` only as one unwrapped base64 line; workflows decode it into a mode-`0600` temporary key, validate it with `ssh-keygen`, and delete the temporary directory on every exit. Contract tests must reject direct multiline `SSH_PRIVATE_KEY` injection or `printf` of a decoded secret variable. A leaked staging key must be rotated and revoked before any rerun; production keys remain a separate boundary and were not involved in this incident.
## ERR-095 | Gitea trusted-main fetch can stall until the whole deploy job times out | investigating 2026-08-06
After exact-SHA staging gate `1473` succeeded, automatic deploy `1474` stopped making progress in the empty-repository `git fetch --no-tags origin main "$DEPLOY_SHA"` step before any SSH or staging mutation. Public and state SHAs remained on the previous healthy release. The checkout had no command-level bound, so the 30-minute job timeout was the only escape and the shared mutation queue remained occupied.
Prevention: wrap the Gitea deploy and migration trusted-main fetch in three 120-second bounded attempts with incremental delay and explicit fail-closed exhaustion. Preserve the reviewed-main ancestry check, exact-SHA gate artifact, forward-only deploy policy, and shared mutation queue. Never treat an in-progress or timed-out fetch as a deployment result.
## Fragment Sweep Command Set
## ERR-086 | Steve Jobs jyotishganit artifacts used non-San-Francisco coordinates | mitigated 2026-07-21
@@ -488,7 +488,10 @@ test("Gitea deploy and migration consume the exact successful gate artifact", ()
assert.match(workflow, /actions\/artifacts\/\$artifact_id\/zip/);
assert.match(workflow, /node frontend\/scripts\/staging-image-manifest\.mjs/);
assert.match(workflow, /name: Checkout trusted main controller/);
assert.match(workflow, /git fetch --no-tags origin main "\$DEPLOY_SHA"/);
assert.match(workflow, /for attempt in 1 2 3/);
assert.match(workflow, /timeout 120 git fetch --no-tags origin main "\$DEPLOY_SHA"/);
assert.match(workflow, /trusted main fetch failed after \$attempt bounded attempts/);
assert.match(workflow, /\[\[ "\$fetch_succeeded" == true \]\]/);
assert.match(workflow, /git checkout --detach --force origin\/main/);
assert.match(workflow, /git merge-base --is-ancestor "\$DEPLOY_SHA" HEAD/);
assert.doesNotMatch(workflow, /docker manifest inspect/);