fix(staging): retry exact gate checkout

This commit is contained in:
Jesse
2026-08-06 20:13:59 +08:00
parent 9d8c73561f
commit c1061695c7
4 changed files with 72 additions and 4 deletions
+28 -4
View File
@@ -40,8 +40,20 @@ jobs:
git init .
git remote remove origin 2>/dev/null || true
git remote add origin https://git.copse.top/root/Jyotisha.git
git -c http.connectTimeout=15 -c http.lowSpeedLimit=1024 -c http.lowSpeedTime=30 \
fetch --depth=1 --no-tags origin "$GITEA_SHA"
fetch_succeeded=false
for attempt in 1 2 3; do
if timeout 300 git -c http.connectTimeout=15 -c http.lowSpeedLimit=1 -c http.lowSpeedTime=60 \
fetch --depth=1 --no-tags origin "$GITEA_SHA"; then
fetch_succeeded=true
break
fi
if [ "$attempt" -eq 3 ]; then
echo "exact staging gate checkout failed after $attempt bounded attempts" >&2
exit 1
fi
sleep $((attempt * 10))
done
[[ "$fetch_succeeded" == true ]]
git checkout --detach --force "$GITEA_SHA"
git clean -ffdx
test "$(git rev-parse HEAD)" = "$GITEA_SHA"
@@ -191,8 +203,20 @@ jobs:
git init .
git remote remove origin 2>/dev/null || true
git remote add origin https://git.copse.top/root/Jyotisha.git
git -c http.connectTimeout=15 -c http.lowSpeedLimit=1024 -c http.lowSpeedTime=30 \
fetch --depth=1 --no-tags origin "$GITEA_SHA"
fetch_succeeded=false
for attempt in 1 2 3; do
if timeout 300 git -c http.connectTimeout=15 -c http.lowSpeedLimit=1 -c http.lowSpeedTime=60 \
fetch --depth=1 --no-tags origin "$GITEA_SHA"; then
fetch_succeeded=true
break
fi
if [ "$attempt" -eq 3 ]; then
echo "exact staging gate checkout failed after $attempt bounded attempts" >&2
exit 1
fi
sleep $((attempt * 10))
done
[[ "$fetch_succeeded" == true ]]
git checkout --detach --force "$GITEA_SHA"
git clean -ffdx
test "$(git rev-parse HEAD)" = "$GITEA_SHA"
+32
View File
@@ -2256,3 +2256,35 @@
- 相关记录:BUG-124
- 复发自:BUG-124
- 修复版本:待提交(本地可测)
## BUG-131 | staging quality gate exact-SHA checkout 因过严低速阈值单次失败
- 状态:investigating
- 首次发现:2026-08-06
- 最近更新:2026-08-06
- 影响面:Gitea `Staging Backend Quality Gate` validate/publish 的 exact-SHA checkoutstaging mutation controller、应用数据面与 production 未受影响。
- 用户现象:docs-attestation staging gate run `1485` 在 validate 的首步失败;Gitea 已枚举/压缩 3,246/2,895 个 shallow objects,但客户端传输降速后触发 `curl 28 Operation too slow``early EOF`。publish 被依赖关系跳过,自动 deploy 未触发;公网继续健康运行 `e59f15d352787f3d05425ba8c459d092e9801a20`
- 触发条件:quality gate 的 exact-SHA `--depth=1` fetch 只有单次调用,并把低速失败设为连续 30 秒低于 1024 B/s;当前 Gitea 链路在约 20 KiB/s 波动后短时低于阈值。
- 根因:`BUG-129` 消除了 mutation workflow 的 Git object fetch,但 quality gate 自身仍必须取得待测源码;其 checkout 没有 bounded retry,且低速阈值对当前受限链路过严。旧测试只断言 exact SHA/clean tree,没有覆盖 checkout retry 与低速边界。
- 修复:validate/publish 两处 exact-SHA checkout 均改为最多 3 次、每次 hard timeout 300 秒;保留 connect timeout 15 秒,将低速失败收紧为连续 60 秒低于 1 B/s。每次仍只抓 `--depth=1 --no-tags origin "$GITEA_SHA"`,耗尽后明确 fail closed,不复用旧 artifact、不放宽 exact-SHA 或 clean-tree 校验。
- 验证:过期基线 PR gate `1488` 成功;待在最新并发主线上完成本地 workflow/YAML/run-script/pre-work 回归、完整 PR gate、staging push gate 和自动 deploy,完成前不得标记 resolved。
- 防复发:质量门禁和 mutation controller 的网络边界分别测试;quality gate checkout 必须覆盖 attempt 数、hard timeout、低速阈值、exact-SHA refspec、最终错误和 clean-tree identity。
- 相关记录:BUG-129、ERR-095、ERR-096
- 复发自:无;属于同一 Gitea 链路在 quality-gate 阶段的独立缺口
- 修复版本:待 bounded exact-SHA gate checkout 与 staging 验收
## BUG-132 | 新增后台页面未同步能力审计精确路由集合
- 状态:investigating
- 首次发现:2026-08-06
- 最近更新:2026-08-06
- 影响面:`tests/test_api_server_security.py::test_capability_audit_scans_registry_and_local_sources`、Gitea staging quality gate;新增后台页面实现本身未由本记录改动。
- 用户现象:并发主线 staging gate run `1489` 的 Python quick gate 为 290 passed、1 skipped、1 failed;能力审计已扫描到 14 个新增后台页面,但测试仍精确断言旧的 7 路由集合,publish 被跳过,自动 deploy 未触发。
- 触发条件:新增 administrators、audit logs、consultations、credit transactions、customers、feature flags、model releases、models、orders、products、roles、security、subscriptions、usage 页面后执行 capability audit 精确集合回归。
- 根因:并发后台功能更新了真实 App Router 页面,却未同步能力审计的完整预期集合;这是 `BUG-126` 同类防复发模式在后台模块复发,说明新增页面的同变更门禁仍未统一执行。
- 修复:保留精确集合比较,将实际新增的 14 个后台页面按排序加入预期列表;不删除既有个人报告页,不改成子集或数量下限,不修改并发后台业务实现。
- 验证:待本地目标测试、完整 Gitea PR gate、staging push gate;完成前不得标记 resolved。
- 防复发:任何 `frontend/src/app/**/page.tsx` 新增或删除必须在同一提交更新 capability audit 精确路由集合,且 quality gate 失败不得通过放宽断言绕过。
- 相关记录:BUG-126、BUG-131
- 复发自:BUG-126
- 修复版本:待能力审计同步与 staging 验收
+6
View File
@@ -151,6 +151,12 @@ After exact-SHA staging gate `1473` succeeded, automatic deploy `1474` stopped m
Prevention: Gitea mutation workflows must perform no Git object operations. A successful staging gate packages its already-verified exact-SHA `deploy/` controller plus manifest validator into `controller.tar`, binds its SHA-256 into the strict image manifest, and uploads both as one immutable artifact. Deploy/migration must verify artifact run/SHA, controller digest, archive paths/types/duplicates/size, current `main == staging` refs, and a complete Gitea compare commit-DAG path before mutation; any missing or inconsistent evidence fails closed. Manual rollback still uses the current reviewed controller, never the old target's controller. Preserve exact-SHA images, forward-only defaults, shared mutation lock, and bounded API/artifact requests. Verified by PR gate `1481`, staging gate `1483`, and one-minute exact-SHA deploy `1484`; mutation-time `git fetch` was zero and all post-deploy health/schema/permission checks passed.
## ERR-096 | Quality-gate exact-SHA checkout failed on a transient low-speed window | investigating 2026-08-06
Staging gate `1485` failed before validation when its single exact-SHA shallow fetch hit the configured 30-second/1024-B/s low-speed abort, producing `curl 28`, `early EOF`, and no publish artifact. No deployment was triggered and the previous exact-SHA staging application remained healthy. This is separate from mutation-time Git removal: quality validation still must acquire the source under test.
Prevention: both validate and publish exact-SHA checkouts use three bounded 300-second attempts, a 15-second connect timeout, and a 60-second/1-B/s stalled-transfer threshold. Preserve `--depth=1 --no-tags origin "$GITEA_SHA"`, exact HEAD equality, clean-tree checks, artifact non-reuse, and fail-closed exhaustion. Never report a skipped publish job as successful artifact publication.
## Fragment Sweep Command Set
## ERR-086 | Steve Jobs jyotishganit artifacts used non-San-Francisco coordinates | mitigated 2026-07-21
@@ -141,6 +141,12 @@ test("Gitea quality gate validates before publishing an immutable ACR manifest",
assert.match(workflow, /GITEA_SHA: \$\{\{ gitea\.sha \}\}/);
assert.equal((workflow.match(/git clean -ffdx/g) ?? []).length, 2);
assert.equal((workflow.match(/git status --porcelain --untracked-files=all/g) ?? []).length, 2);
assert.equal((workflow.match(/for attempt in 1 2 3/g) ?? []).length, 5);
assert.equal((workflow.match(/timeout 300 git -c http\.connectTimeout=15 -c http\.lowSpeedLimit=1 -c http\.lowSpeedTime=60/g) ?? []).length, 2);
assert.equal((workflow.match(/fetch --depth=1 --no-tags origin "\$GITEA_SHA"/g) ?? []).length, 2);
assert.equal((workflow.match(/exact staging gate checkout failed after \$attempt bounded attempts/g) ?? []).length, 2);
assert.equal((workflow.match(/\[\[ "\$fetch_succeeded" == true \]\]/g) ?? []).length, 2);
assert.doesNotMatch(workflow, /http\.lowSpeedLimit=1024|http\.lowSpeedTime=30/);
assert.match(workflow, /python3 -m venv \.venv/);
assert.match(workflow, /PIP_INDEX_URL: https:\/\/mirrors\.aliyun\.com\/pypi\/simple\//);
assert.match(workflow, /NPM_CONFIG_REGISTRY: https:\/\/registry\.npmmirror\.com/);