Compare commits

..

2 Commits

Author SHA1 Message Date
Jesse 3f61188b07 docs(staging): close controller fetch incident 2026-08-06 20:04:47 +08:00
jesse e59f15d352 Merge pull request 'Deploy gate-attested staging controllers' (#18)
Staging Backend Quality Gate / validate (push) Successful in 13m53s
Staging Backend Quality Gate / publish (push) Successful in 10m17s
Replace mutation-time Git object fetches with exact-gate hash-bound controller bundles.
2026-08-06 19:33:47 +08:00
3 changed files with 10 additions and 6 deletions
+3 -3
View File
@@ -2227,7 +2227,7 @@
## BUG-129 | staging trusted-main checkout 无界 fetch 导致自动部署长期占用 mutation queue
- 状态:investigating
- 状态:resolved
- 首次发现:2026-08-06
- 最近更新:2026-08-06
- 影响面:Gitea staging deploy/migration 控制器的 trusted-main checkoutproduction 与 staging 应用数据面未受影响。
@@ -2235,8 +2235,8 @@
- 触发条件:空仓库命令 `git fetch --no-tags origin main "$DEPLOY_SHA"` 同时请求分支和目标 SHA,导致 Gitea 为每次尝试枚举/压缩完整历史对象;runner 与服务端之间的传输无法在 120 秒内完成。
- 根因:原控制器既没有命令级 timeout,也错误地为正常前向发布抓取 full-history dual ref。第一轮修复只增加 bounded retry,解决了无界占用,但旧回归测试只断言 timeout/attempt/ancestry,未限制传输对象范围,因而未拦住连续三次重新打包完整历史。
- 修复:不再让 mutation runner 做任何 Git object fetch。成功 staging gate 从其已验证的 exact SHA 生成仅含 tracked `deploy/` 与严格 manifest validator 的 `controller.tar`,将 tar SHA-256 写入四字段 manifest,并与 immutable image digests 一起上传。deploy/migration 从 exact successful gate artifact 下载 bundle,强制校验 controller SHA、tar hash、路径、重复项、类型和 2 MiB 上限后才解包;正常发布使用当前 `main == staging` controller,手工旧版 rollback 也不得执行旧 controller。refs 与 forward/rollback 关系通过有界 Gitea API 和完整 commit-DAG 路径证明,字段缺失、分页不完整、头不一致或证据冲突均 fail closed。
- 验证:第一轮 bounded retry 的本地 workflow contracts 31/31、PR gates `1475/1477` 与 staging gate `1479` 成功;run `1480` 证明 3 次 120 秒耗尽后无半部署。bundle 修复本地 manifest/workflow contracts 34/34、三份 YAML、shell、真实 25-entry/122,880-byte controller tar hash/安全检查、mutation Git-object-op=0、mandatory pre-work 和 diff 检查通过完整 PR gate、staging gate 和 exact-SHA deploy 待完成;完成前不得再次标记 resolved
- 验证:第一轮 bounded retry 的本地 workflow contracts 31/31、PR gates `1475/1477` 与 staging gate `1479` 成功;run `1480` 证明 3 次 120 秒耗尽后无半部署。bundle 修复本地 manifest/workflow contracts 35/35、三份 YAML、解析后所有 shell/Python heredoc、真实 25-entry/122,880-byte controller tar hash/ZIP+TAR 安全检查、mutation Git-object-op=0、live Gitea commit-DAG、mandatory pre-work、ESLint、privacy 和 diff 检查通过完整 PR gate `1481`、staging gate `1483` 成功。自动 deploy `1484` 在 1 分钟内成功部署 `e59f15d352787f3d05425ba8c459d092e9801a20`,日志 mutation `git fetch=0`、controller hash check 存在、SSH secret 遮蔽且无私钥材料;`main/staging/public/state` 精确一致,5 个容器 restart count 均为 0health、Swiss Ephemeris、未登录 401、`personal_reports`、RLS、2 条 owner policies、精确 migration ledger、authenticated SELECT/DELETE-only 与 service-role CRUD 均通过
- 防复发:所有 release-controller 网络调用必须有命令级上限和失败闭合;mutation workflow 禁止 `git fetch/ls-remote/cat-file/merge-base/checkout/init`。控制器必须来自 exact successful gate 的 hash-bound artifact,正常与 rollback 均使用当前 reviewed controller;测试必须覆盖 artifact identity、tar safety、commit-DAG proof 和旧 Git object 路径为零。
- 相关记录:BUG-128、ERR-094、ERR-095
- 复发自:BUG-129 第一轮修复未覆盖对象范围
- 修复版本:gate-attested controller bundle staging 验收
- 修复版本:`e59f15d352787f3d05425ba8c459d092e9801a20`gate-attested controller bundle 已完成 staging exact-SHA 验收
@@ -11,6 +11,7 @@ This record contains only release identities, aggregate operational evidence, sc
- Security/control-plane merge: `f7a615a5bf11ed95b3a6c7e6d28dfe8150a825ef`
- Staging migration and application-under-test SHA: `f7a615a5bf11ed95b3a6c7e6d28dfe8150a825ef`
- Docs-only acceptance attestation deployment SHA: `02cc483b7c303e6cc0f26fb31462c50adb007f12`
- Final gate-attested controller deployment SHA: `e59f15d352787f3d05425ba8c459d092e9801a20`
- Application rollback target: `49da8f916960030d5760d8dedf4e77820732a527`, subject to retained successful gate artifacts. The additive database migration remains in place after application rollback.
At deployment verification, Gitea `main`, Gitea `staging`, `/opt/jyotisha-staging/.state/deployed-revision`, and public `/api/health` all reported the same full SHA.
@@ -40,7 +41,10 @@ The complete frontend suite could not be made fully executable on the local macO
- Manual migration `1471`: success; `20260806000000_personal_reports.sql` applied and present once in the migration ledger.
- Manual deploy `1472`: success with the same exact application-under-test SHA and `allow_rollback=false`.
- Docs-only attestation gate `1473`: success for `02cc483b7c303e6cc0f26fb31462c50adb007f12`; immutable manifest published.
- Docs-only attestation deploy `1474`: success. The trusted-main fetch paused for an extended period before recovering; no SSH/staging mutation occurred during the pause. Public health and host state then moved to the exact attestation SHA with zero container restarts. See `BUG-129` / `ERR-095` for the bounded-fetch follow-up.
- Docs-only attestation deploy `1474`: success. The trusted-main fetch paused for an extended period before recovering; no SSH/staging mutation occurred during the pause. Public health and host state then moved to the exact attestation SHA with zero container restarts.
- First bounded-fetch staging gate `1479`: success; automatic deploy `1480` then exhausted all three 120-second full-history fetch attempts and failed closed before SSH. Public/state remained on the prior healthy SHA.
- Gate-attested controller PR gate `1481` and staging gate `1483`: success. Gate `1483` published the first four-field manifest plus SHA-256-bound `controller.tar` bundle for `e59f15d352787f3d05425ba8c459d092e9801a20`.
- Final automatic deploy `1484`: success in one minute. Mutation-time `git fetch` count was zero; controller hash validation ran; the staging SSH secret remained masked and no private-key material appeared. Public health, host state, Gitea `main`, and Gitea `staging` all matched the exact final SHA. See `BUG-129` / `ERR-095`.
## Security incident and containment
+2 -2
View File
@@ -145,11 +145,11 @@ A failed exact-SHA staging deploy displayed the multiline staging SSH private ke
Prevention: store `STAGING_SSH_PRIVATE_KEY` only as one unwrapped base64 line; workflows decode it into a mode-`0600` temporary key, validate it with `ssh-keygen`, and delete the temporary directory on every exit. Contract tests must reject direct multiline `SSH_PRIVATE_KEY` injection or `printf` of a decoded secret variable. A leaked staging key must be rotated and revoked before any rerun; production keys remain a separate boundary and were not involved in this incident.
## ERR-095 | Gitea trusted-main full-history fetch stalls or exhausts every bounded attempt | investigating 2026-08-06
## ERR-095 | Gitea trusted-main full-history fetch stalls or exhausts every bounded attempt | mitigated 2026-08-06
After exact-SHA staging gate `1473` succeeded, automatic deploy `1474` stopped making log progress for an extended period in the empty-repository `git fetch --no-tags origin main "$DEPLOY_SHA"` step before any SSH or staging mutation. The fetch later recovered and the 18-minute run successfully deployed the exact SHA. The first mitigation added three 120-second attempts, but run `1480` proved every attempt still asked Gitea to enumerate/compress 16,093 full-history objects and then timed out. It failed closed before SSH; public and state SHAs remained on the prior healthy release. The first fix bounded queue occupation but did not reduce the transfer, and its regression test did not reject the full-history dual-ref form.
Prevention: Gitea mutation workflows must perform no Git object operations. A successful staging gate packages its already-verified exact-SHA `deploy/` controller plus manifest validator into `controller.tar`, binds its SHA-256 into the strict image manifest, and uploads both as one immutable artifact. Deploy/migration must verify artifact run/SHA, controller digest, archive paths/types/duplicates/size, current `main == staging` refs, and a complete Gitea compare commit-DAG path before mutation; any missing or inconsistent evidence fails closed. Manual rollback still uses the current reviewed controller, never the old target's controller. Preserve exact-SHA images, forward-only defaults, shared mutation lock, and bounded API/artifact requests.
Prevention: Gitea mutation workflows must perform no Git object operations. A successful staging gate packages its already-verified exact-SHA `deploy/` controller plus manifest validator into `controller.tar`, binds its SHA-256 into the strict image manifest, and uploads both as one immutable artifact. Deploy/migration must verify artifact run/SHA, controller digest, archive paths/types/duplicates/size, current `main == staging` refs, and a complete Gitea compare commit-DAG path before mutation; any missing or inconsistent evidence fails closed. Manual rollback still uses the current reviewed controller, never the old target's controller. Preserve exact-SHA images, forward-only defaults, shared mutation lock, and bounded API/artifact requests. Verified by PR gate `1481`, staging gate `1483`, and one-minute exact-SHA deploy `1484`; mutation-time `git fetch` was zero and all post-deploy health/schema/permission checks passed.
## Fragment Sweep Command Set