Restore staging gate after concurrent admin rollout #21

Merged
jesse merged 5 commits from fix/staging-gate-retry-and-admin-audit into main 2026-08-06 22:51:47 +08:00
5 changed files with 129 additions and 14 deletions
+28 -4
View File
@@ -40,8 +40,20 @@ jobs:
git init .
git remote remove origin 2>/dev/null || true
git remote add origin https://git.copse.top/root/Jyotisha.git
git -c http.connectTimeout=15 -c http.lowSpeedLimit=1024 -c http.lowSpeedTime=30 \
fetch --depth=1 --no-tags origin "$GITEA_SHA"
fetch_succeeded=false
for attempt in 1 2 3; do
if timeout 300 git -c http.connectTimeout=15 -c http.lowSpeedLimit=1 -c http.lowSpeedTime=60 \
fetch --depth=1 --no-tags origin "$GITEA_SHA"; then
fetch_succeeded=true
break
fi
if [ "$attempt" -eq 3 ]; then
echo "exact staging gate checkout failed after $attempt bounded attempts" >&2
exit 1
fi
sleep $((attempt * 10))
done
[[ "$fetch_succeeded" == true ]]
git checkout --detach --force "$GITEA_SHA"
git clean -ffdx
test "$(git rev-parse HEAD)" = "$GITEA_SHA"
@@ -191,8 +203,20 @@ jobs:
git init .
git remote remove origin 2>/dev/null || true
git remote add origin https://git.copse.top/root/Jyotisha.git
git -c http.connectTimeout=15 -c http.lowSpeedLimit=1024 -c http.lowSpeedTime=30 \
fetch --depth=1 --no-tags origin "$GITEA_SHA"
fetch_succeeded=false
for attempt in 1 2 3; do
if timeout 300 git -c http.connectTimeout=15 -c http.lowSpeedLimit=1 -c http.lowSpeedTime=60 \
fetch --depth=1 --no-tags origin "$GITEA_SHA"; then
fetch_succeeded=true
break
fi
if [ "$attempt" -eq 3 ]; then
echo "exact staging gate checkout failed after $attempt bounded attempts" >&2
exit 1
fi
sleep $((attempt * 10))
done
[[ "$fetch_succeeded" == true ]]
git checkout --detach --force "$GITEA_SHA"
git clean -ffdx
test "$(git rev-parse HEAD)" = "$GITEA_SHA"
+76 -9
View File
@@ -2257,15 +2257,82 @@
- 复发自:BUG-124
- 修复版本:待提交(本地可测)
## BUG-131 | Admin users route config re-export blocks Turbopack quality gate
## BUG-131 | staging quality gate exact-SHA checkout 因过严低速阈值单次失败
- 状态:resolvedlocal
- 状态:investigating
- 首次发现:2026-08-06
- 最近更新:2026-08-06
- 影响面:`frontend/src/app/api/admin/users/route.ts` 与 Next.js 16 Turbopack production build
- 用户现象:Gitea quality gate 的默认 `next build` 无法解析 `/api/admin/users` route config
- 根因:别名 route 从 `../customers/route` re-export `runtime`Next/Turbopack 要求 route config 在 route 文件内静态本地导出
- 修复:在 users route 本地声明 `export const runtime = "nodejs"`,仅 re-export HTTP handlers;扫描全部 route 后未发现其他 `runtime``dynamic``revalidate` re-export
- 验证:admin users contract、`tsc --noEmit`、默认 Turbopack `next build``next build --webpack``git diff --check`
- 防复发:route config 必须在各自 route 文件内静态声明,别名 route 只能 re-export handlers
- 修复版本:本提交(本地已验证)
- 影响面:Gitea `Staging Backend Quality Gate` validate/publish 的 exact-SHA checkoutstaging mutation controller、应用数据面与 production 未受影响
- 用户现象:docs-attestation staging gate run `1485` 在 validate 的首步失败;Gitea 已枚举/压缩 3,246/2,895 个 shallow objects,但客户端传输降速后触发 `curl 28 Operation too slow``early EOF`。publish 被依赖关系跳过,自动 deploy 未触发;公网继续健康运行 `e59f15d352787f3d05425ba8c459d092e9801a20`
- 触发条件:quality gate 的 exact-SHA `--depth=1` fetch 只有单次调用,并把低速失败设为连续 30 秒低于 1024 B/s;当前 Gitea 链路在约 20 KiB/s 波动后短时低于阈值
- 根因:`BUG-129` 消除了 mutation workflow 的 Git object fetch,但 quality gate 自身仍必须取得待测源码;其 checkout 没有 bounded retry,且低速阈值对当前受限链路过严。旧测试只断言 exact SHA/clean tree,没有覆盖 checkout retry 与低速边界
- 修复:validate/publish 两处 exact-SHA checkout 均改为最多 3 次、每次 hard timeout 300 秒;保留 connect timeout 15 秒,将低速失败收紧为连续 60 秒低于 1 B/s。每次仍只抓 `--depth=1 --no-tags origin "$GITEA_SHA"`,耗尽后明确 fail closed,不复用旧 artifact、不放宽 exact-SHA 或 clean-tree 校验
- 验证:过期基线 PR gate `1488` 成功;待在最新并发主线上完成本地 workflow/YAML/run-script/pre-work 回归、完整 PR gate、staging push gate 和自动 deploy,完成前不得标记 resolved
- 防复发:质量门禁和 mutation controller 的网络边界分别测试;quality gate checkout 必须覆盖 attempt 数、hard timeout、低速阈值、exact-SHA refspec、最终错误和 clean-tree identity。
- 相关记录:BUG-129、ERR-095、ERR-096
- 复发自:无;属于同一 Gitea 链路在 quality-gate 阶段的独立缺口
- 修复版本:待 bounded exact-SHA gate checkout 与 staging 验收
## BUG-132 | 新增后台页面未同步能力审计精确路由集合
- 状态:investigating
- 首次发现:2026-08-06
- 最近更新:2026-08-06
- 影响面:`tests/test_api_server_security.py::test_capability_audit_scans_registry_and_local_sources`、Gitea staging quality gate;新增后台页面实现本身未由本记录改动。
- 用户现象:并发主线 staging gate run `1489` 的 Python quick gate 为 290 passed、1 skipped、1 failed;能力审计已扫描到 14 个新增后台页面,但测试仍精确断言旧的 7 路由集合,publish 被跳过,自动 deploy 未触发。
- 触发条件:新增 administrators、audit logs、consultations、credit transactions、customers、feature flags、model releases、models、orders、products、roles、security、subscriptions、usage 页面后执行 capability audit 精确集合回归。
- 根因:并发后台功能更新了真实 App Router 页面,却未同步能力审计的完整预期集合;这是 `BUG-126` 同类防复发模式在后台模块复发,说明新增页面的同变更门禁仍未统一执行。
- 修复:保留精确集合比较,将实际新增的 14 个后台页面按排序加入预期列表;不删除既有个人报告页,不改成子集或数量下限,不修改并发后台业务实现。
- 验证:待本地目标测试、完整 Gitea PR gate、staging push gate;完成前不得标记 resolved。
- 防复发:任何 `frontend/src/app/**/page.tsx` 新增或删除必须在同一提交更新 capability audit 精确路由集合,且 quality gate 失败不得通过放宽断言绕过。
- 相关记录:BUG-126、BUG-131
- 复发自:BUG-126
- 修复版本:待能力审计同步与 staging 验收
## BUG-133 | admin users Route Handler 重导出 runtime 导致 production build 失败
- 状态:investigating
- 首次发现:2026-08-06
- 最近更新:2026-08-06
- 影响面:`frontend/src/app/api/admin/users/route.ts`、Next.js production build、Gitea staging quality gatecustomer handler 权限和业务逻辑未受改动。
- 用户现象:PR gate run `1493` 的 Python 路由回归和 frontend 1472/1472 均通过,但 `next build``Next.js can't recognize the exported runtime field in route. It mustn't be reexported`publish 被跳过,自动 deploy 未触发。
- 触发条件:legacy `/api/admin/users` Route Handler 通过 `export { ..., runtime } from "../customers/route"` 同时重导出 handlers 和 route segment config。
- 根因:Next.js 要求 `runtime` 等 route segment config 在当前 route 文件中可被静态解析,不允许从另一 Route Handler 重导出;既有精确合同测试反而固化了非法 re-export,且并发功能本地 production build 未闭环。
- 修复:在 users route 本文件静态声明 `export const runtime = "nodejs"`,只重导出 DELETE/GET/PATCH/POST/PUT handlers;不复制 handler、不修改权限或客户数据逻辑。合同测试改为强制本地 runtime 常量并拒绝 runtime re-export。
- 验证:admin users 目标合同、`tsc --noEmit`、默认 Turbopack `next build``next build --webpack`、全 route segment config re-export 扫描与 `git diff --check` 已通过;Gitea PR gate `1499` 在 SHA `015f1e501e2132d2abc6bbbcb8e6af074c326b47` 上通过并完成 production build。待最新并发主线 staging gate 与部署验收,完成前不得标记 resolved。
- 防复发:Route Handler 的 `runtime``dynamic``revalidate` 等 segment config 必须本地静态声明;handler 可复用,但 segment config 不得 re-export。新增 alias route 必须经过 production build,而不只运行文本合同测试。
- 相关记录:BUG-131、BUG-132
- 复发自:无
- 修复版本:待 Route Handler 静态 config 修复与 staging 验收
## BUG-134 | staging admin origin selector 未配置导致 exact-SHA 自动部署失败
- 状态:investigating
- 首次发现:2026-08-06
- 最近更新:2026-08-06
- 影响面:staging `.env.staging` 的公开 identity selector、自动 deploy run `1502`;应用容器、业务数据库和 production 未被修改。
- 用户现象:staging gate `1500` 已成功验证并发布 SHA `9a3d0d440f43deab66c1f8a4a08cdbfc6f9d73eb` 的 immutable artifact,但自动 deploy `1502` 在远端 env 校验时报 `invalid staging selector: ADMIN_USER_ORIGIN` 并 fail closed;公网继续健康运行旧 SHA `e59f15d352787f3d05425ba8c459d092e9801a20`
- 触发条件:包含双 host self-hosted identity validator 的 controller 部署到现有 staging host,而 `.env.staging` 尚未包含精确且唯一的 `ADMIN_USER_ORIGIN=https://admin.staging.jyotisha.chat`
- 根因:并发 admin rollout 将 admin host origin 加入应用和 validator 合同,但 staging host-managed env 未在发布前同步新增的非密钥 selectorquality gate 验证仓库合同,不读取主机 secret/env,因此直到 mutation 前远端校验才暴露漂移。
- 修复:已在共享 staging mutation lock 下,仅向原文件原子补入公开 `ADMIN_USER_ORIGIN` selector,保留全部既有内容、`deploy:deploy` owner 和 `0600` mode;未输出、复制或重写其他 secret 值。随后完整 validator 暴露独立的 service runtime 漂移,转由 `BUG-135` 处理;仍待 exact-SHA artifact 重新部署。
- 验证:脱敏只读检查先确认 `.env.staging``deploy:deploy 0600``AUTH_USER_ORIGIN` 精确且唯一、`ADMIN_USER_ORIGIN` 计数为 0;原子修复后 `ADMIN_USER_ORIGIN` 精确且唯一,正式 staging env validator 已通过。待最新 staging gate、exact-SHA deploy、公网/host SHA、容器 restart、日志和未登录边界验收。
- 防复发:任何新增 staging host-managed selector 必须在同一 rollout runbook 中包含 deploy 前 presence/exact-value 检查;quality gate 成功不能替代 host env validation。env 修复必须共享 mutation lock、原子替换并保持 owner/mode,严禁打印 raw env。
- 相关记录:BUG-128、BUG-133、ERR-093、ERR-097
- 复发自:无
- 修复版本:待 staging env selector 对齐与 exact-SHA 部署验收
## BUG-135 | staging service runtime 缺失且 admin runtime 错误继承 BYPASSRLS 角色
- 状态:investigating
- 首次发现:2026-08-06
- 最近更新:2026-08-06
- 影响面:staging 私有 PostgreSQL runtime roles、`.env.staging``.env.staging.database`、个人报告 service client 与后台最小权限;production 未受影响。
- 用户现象:补齐 `ADMIN_USER_ORIGIN` 后,正式 validator 继续报 `invalid staging identity setting: SERVICE_DATABASE_URL`。脱敏检查确认应用 env 缺少 `SERVICE_DATABASE_URL`、数据库 env 缺少 `SERVICE_RUNTIME_PASSWORD`、PostgreSQL 缺少 `service_runtime` login role;同时旧 `admin_runtime` 意外继承了带 BYPASSRLS 的 `service_role`。当前旧 web 容器同样没有 service URL,因此无法安全恢复旧明文。
- 触发条件:在早期初始化的 staging 数据卷上部署依赖独立 service client 和最新 admin RBAC 的应用;bootstrap 脚本只在空数据卷初始化时执行,现有 host env/roles 未随 reviewed compatibility contract 对齐。
- 根因:staging host bootstrap 漂移。数据库保留了 `service_role`、identity/app/admin runtime roles,但没有后来合同要求的 `service_runtime`;旧 admin runtime membership 又违反当前 bootstrap 和 RBAC 的明确 revoke 边界。quality gate 不读取 host env 或运行时 role catalog,因此直到远端部署前校验与现场权限审计才暴露。
- 修复:在共享 mutation lock 下生成独立 staging-only 随机凭据,通过 PostgreSQL stdin 创建/设置 `service_runtime`,授予 `service_role` membership 和数据库 CONNECT;将 raw password 仅原子写入 `.env.staging.database`percent-encoded URL 仅原子写入 `.env.staging`,两文件保持 `deploy:deploy 0600`。随后撤销 `admin_runtime``service_role` membership;未重启容器、未输出凭据、未改 production。已应用的 `20260806000000_personal_reports.sql` checksum 与仓库一致,保持历史迁移不可变。
- 验证:两个正式 env validator 均通过;`service_runtime` 真实密码登录、`service_role` membership 和 CONNECT 均通过;`admin_runtime` membership=false`service_runtime` membership=true;两份 service 配置各精确 1 条且非空,owner/mode 保持正确。PR gate `1503` 已通过;待最终合并 SHA 的 staging gate、migration check、exact-SHA deploy、service client smoke、RLS/容器/日志验收,完成前不得标记 resolved。
- 防复发:非空数据卷不能依赖 `/docker-entrypoint-initdb.d` 自动重放;每次新增 runtime role 或 host-managed URL 都必须有兼容性 role repair、脱敏 pre-deploy presence 检查和真实登录/role-membership smoke。`admin_runtime` 永不得继承 `service_role`service writes 只能使用独立 `SERVICE_DATABASE_URL`。已应用迁移不得为修正文案而改 checksum。
- 相关记录:BUG-128、BUG-134、ERR-093、ERR-098
- 复发自:无
- 修复版本:待 staging service runtime 对齐与 exact-SHA 部署验收
+18
View File
@@ -151,6 +151,24 @@ After exact-SHA staging gate `1473` succeeded, automatic deploy `1474` stopped m
Prevention: Gitea mutation workflows must perform no Git object operations. A successful staging gate packages its already-verified exact-SHA `deploy/` controller plus manifest validator into `controller.tar`, binds its SHA-256 into the strict image manifest, and uploads both as one immutable artifact. Deploy/migration must verify artifact run/SHA, controller digest, archive paths/types/duplicates/size, current `main == staging` refs, and a complete Gitea compare commit-DAG path before mutation; any missing or inconsistent evidence fails closed. Manual rollback still uses the current reviewed controller, never the old target's controller. Preserve exact-SHA images, forward-only defaults, shared mutation lock, and bounded API/artifact requests. Verified by PR gate `1481`, staging gate `1483`, and one-minute exact-SHA deploy `1484`; mutation-time `git fetch` was zero and all post-deploy health/schema/permission checks passed.
## ERR-096 | Quality-gate exact-SHA checkout failed on a transient low-speed window | investigating 2026-08-06
Staging gate `1485` failed before validation when its single exact-SHA shallow fetch hit the configured 30-second/1024-B/s low-speed abort, producing `curl 28`, `early EOF`, and no publish artifact. No deployment was triggered and the previous exact-SHA staging application remained healthy. This is separate from mutation-time Git removal: quality validation still must acquire the source under test.
Prevention: both validate and publish exact-SHA checkouts use three bounded 300-second attempts, a 15-second connect timeout, and a 60-second/1-B/s stalled-transfer threshold. Preserve `--depth=1 --no-tags origin "$GITEA_SHA"`, exact HEAD equality, clean-tree checks, artifact non-reuse, and fail-closed exhaustion. Never report a skipped publish job as successful artifact publication.
## ERR-097 | Staging host env missed the reviewed admin-origin selector | investigating 2026-08-06
Staging gate `1500` successfully validated and published the exact SHA, but automatic deploy `1502` stopped before app mutation with `invalid staging selector: ADMIN_USER_ORIGIN`. A redacted read-only check confirmed `.env.staging` remained `deploy:deploy 0600`, had one exact user origin, and had zero admin-origin definitions. Public staging therefore remained on the prior healthy SHA; this was not a migration failure and production was not involved.
Prevention: when a reviewed identity rollout adds a host-managed non-secret selector, update the staging env under the shared mutation lock before deploying the dependent controller. Modify only the named public selector through a mode-`0600` atomic replacement that preserves deployment-tree UID/GID; never print or copy the raw env. Re-run the exact validator and require the same gate-attested SHA in `main`, `staging`, host state, and public health before closure.
## ERR-098 | Non-empty staging volume missed service runtime bootstrap and retained an obsolete privileged membership | investigating 2026-08-06
After the admin-origin selector was repaired, the full staging validators exposed that both service runtime env entries were absent. Redacted role inspection then showed `service_role` existed but `service_runtime` did not, while `admin_runtime` retained membership in the BYPASSRLS role contrary to the reviewed bootstrap/RBAC contract. The running old web image had no recoverable service URL, so no password was guessed or copied. Under the shared mutation lock, a new staging-only credential was generated, the dedicated login role and CONNECT/membership were established through PostgreSQL stdin, the two host env files were atomically updated with their separate raw/URL representations, and the obsolete admin membership was revoked. Both validators and a real service login passed; production was not involved.
Prevention: `/docker-entrypoint-initdb.d` is not a compatibility mechanism for an existing PostgreSQL volume. Every newly required runtime role must have a reviewed non-destructive repair path plus pre-deploy role/presence probes. Keep `admin_runtime` outside `service_role`; only `service_runtime` may assume the BYPASSRLS role through the dedicated service URL. Never display role passwords, pass them in argv, or edit an already-ledgered migration checksum to retrofit host bootstrap behavior.
## Fragment Sweep Command Set
## ERR-086 | Steve Jobs jyotishganit artifacts used non-San-Francisco coordinates | mitigated 2026-07-21
+1 -1
View File
@@ -38,7 +38,7 @@ test("admin_users migration is service-role-only and auditable", () => {
assert.match(migration, /grant select, insert, update on table public\.admin_users to service_role/);
});
test("legacy admin users route aliases the guarded customer resource", () => {
test("legacy admin users route aliases handlers but declares runtime statically", () => {
assert.match(usersSource, /export const runtime = "nodejs"/);
assert.match(usersSource, /export \{ DELETE, GET, PATCH, POST, PUT \} from "\.\.\/customers\/route"/);
assert.doesNotMatch(usersSource, /export \{[^}]*\bruntime\b[^}]*\} from/);
@@ -141,6 +141,12 @@ test("Gitea quality gate validates before publishing an immutable ACR manifest",
assert.match(workflow, /GITEA_SHA: \$\{\{ gitea\.sha \}\}/);
assert.equal((workflow.match(/git clean -ffdx/g) ?? []).length, 2);
assert.equal((workflow.match(/git status --porcelain --untracked-files=all/g) ?? []).length, 2);
assert.equal((workflow.match(/for attempt in 1 2 3/g) ?? []).length, 5);
assert.equal((workflow.match(/timeout 300 git -c http\.connectTimeout=15 -c http\.lowSpeedLimit=1 -c http\.lowSpeedTime=60/g) ?? []).length, 2);
assert.equal((workflow.match(/fetch --depth=1 --no-tags origin "\$GITEA_SHA"/g) ?? []).length, 2);
assert.equal((workflow.match(/exact staging gate checkout failed after \$attempt bounded attempts/g) ?? []).length, 2);
assert.equal((workflow.match(/\[\[ "\$fetch_succeeded" == true \]\]/g) ?? []).length, 2);
assert.doesNotMatch(workflow, /http\.lowSpeedLimit=1024|http\.lowSpeedTime=30/);
assert.match(workflow, /python3 -m venv \.venv/);
assert.match(workflow, /PIP_INDEX_URL: https:\/\/mirrors\.aliyun\.com\/pypi\/simple\//);
assert.match(workflow, /NPM_CONFIG_REGISTRY: https:\/\/registry\.npmmirror\.com/);