fix(release): keep daily backups from stalling when the disk fills

Expired daily files are pruned before a new backup, a verify older than 36 hours raises an alert, and the host monitors call sudo with the production env file. The Linux 0600 check is on the staging quick list.

Co-Authored-By: Grok (agent loop)

deploy/README.md

@@ -401,7 +401,7 @@ report list / detail / create smoke”): `GET /api/reports` 200, owned
 ### 备份
 
 - 工作流：`.gitea/workflows/backup-production-daily.yml`。北京时间每天 04:00，也可手动触发。只从 `main` 运行，调用 `deploy/run-production-recovery.sh`，不另写一套导出。
-- 每日文件前缀是 `production-daily-`。手动恢复点仍是 `production-pre-migration-`。清理脚本 `deploy/prune-production-daily-backups.sh` 只删超过 7 天的每日文件和对应的校验文件，不会删 `production-pre-migration-` 手动恢复点。
+- 每日文件前缀是 `production-daily-`。手动恢复点仍是 `production-pre-migration-`。清理脚本 `deploy/prune-production-daily-backups.sh` 只删超过 7 天的每日文件和对应的校验文件，不会删 `production-pre-migration-` 手动恢复点。清理排在新备份之前，所以磁盘到 70% 时过期的每日文件仍会被清掉。
 - 每日备份只做存档和还原核对，不是发布用的恢复点。它不改 `/opt/jyotisha-production/.state/recovery-baseline.env`。核对结果另写在同目录的 `daily-backup-latest.env`，只有文件名、sha256、核对时间和格式版本，没有数据和密钥。发布、迁移、回退都不读这个文件。上线前仍要手动点 Create production recovery。
 - Gitea 产物默认保留 14 天。手动恢复点工作流仍是 30 天，没有改。
 - 还原步骤：`docs/operations/production-restore-drill-20261014.md`。
@@ -410,8 +410,8 @@ report list / detail / create smoke”): `GET /api/reports` 200, owned
 
 - 探活不要装在生产主机上。装在 Gitea runner 上，每分钟跑 `deploy/monitor/healthcheck-alert.sh --mode health`。它访问公网 `https://jyotisha.chat/api/health`，超时 10 秒。连续 3 次失败才告警，恢复只发一次。
 - 发布窗口：在 runner 上执行 `touch /var/lib/jyotisha-monitor/release-window`。45 分钟内，版本号变化不告警。
-- 磁盘检查装在生产主机上，每分钟跑同一个脚本的 `--mode disk`。使用率达到 80% 告警一次，降下来再发一次已恢复。
-- 错误汇总装在生产主机上。每 5 分钟 `deploy/monitor/error-digest.sh --window`。每天北京时间 09:00（UTC 01:00）跑 `--daily`，汇总前一天。同类错误 5 分钟内至少 10 次，或 5xx 至少 20 次，才发窗口告警。消息里最多 5 个 requestId。含邮箱、第二个日期或非 ASCII 文本的日志行整行丢掉。
+- 磁盘检查装在生产主机上，每分钟跑同一个脚本的 `--mode disk`。使用率达到 80% 告警一次，降下来再发一次已恢复。同一轮还会读 `/opt/jyotisha-production/.state/daily-backup-latest.env` 的 `VERIFIED_AT`。这份记录缺失，或比当前时间早超过 36 小时，会再告警一次。重新核对成功后发一次已恢复。探活模式不读这份文件。
+- 错误汇总装在生产主机上。每 5 分钟 `deploy/monitor/error-digest.sh --window`。每天北京时间 09:00（UTC 01:00）跑 `--daily`，汇总前一天。同类错误 5 分钟内至少 10 次，或 5xx 至少 20 次，才发窗口告警。同一服务、同一路由、同一错误码在 30 分钟内不重复发。消息里最多 5 个 requestId。含邮箱、第二个日期或非 ASCII 文本的日志行整行丢掉。飞书没有返回成功时，这条不会被记成已发送。
 - Webhook 写在模式 `0600` 的环境文件里，建议路径 `/opt/jyotisha-production/.monitor.env`（runner 上用 `JYOTISHA_MONITOR_ENV` 指向自己的文件）。只放这两行，不要提交：
 
 ```text
@@ -435,7 +435,7 @@ FEISHU_SIGN_SECRET=<只有开了签名校验才填>
 deploy/find-request-log.sh <requestId> --since 24h
 ```
 
-`<requestId>` 必须是 UUID。其它写法会拒绝，避免被当成命令。`--since` 只接受类似 `24h`、`30m` 的形式。输出按时间排序，来自 web、api、caddy。
+`<requestId>` 必须是 UUID。其它写法会拒绝，避免被当成命令。`--since` 只接受类似 `24h`、`30m` 的形式。容器日志行前面的服务名前缀会先去掉，再按时间排序。来自 web、api、caddy。
 
 ## Common operations
 

deploy/find-request-log.sh

@@ -52,11 +52,28 @@ if [ -n "$log_dir" ]; then
 fi
 
 compose_dir="${JYOTISHA_COMPOSE_DIR:-/opt/jyotisha-production}"
-docker compose -p jyotisha-production \
+export APP_ENV_FILE='../.env.production'
+export DATABASE_ENV_FILE='../.env.production.database'
+export CADDYFILE_PATH='./Caddyfile.production.selfhosted'
+export SITE_ADDRESS='https://jyotisha.chat'
+if [ -n "${JYOTISHA_COMPOSE_LOG_FIXTURE:-}" ]; then
+  [ -f "$JYOTISHA_COMPOSE_LOG_FIXTURE" ] && [ ! -L "$JYOTISHA_COMPOSE_LOG_FIXTURE" ] || {
+    echo "compose log fixture is missing" >&2
+    exit 1
+  }
+  sed -E 's/^[^|]*\| //' "$JYOTISHA_COMPOSE_LOG_FIXTURE" |
+    grep -F -- "$request_id" |
+    grep -E '^[0-9]{4}-[0-9]{2}-[0-9]{2}T' |
+    sort
+  exit 0
+fi
+sudo -n docker compose -p jyotisha-production \
   --env-file "$compose_dir/.env.production" \
   -f "$compose_dir/deploy/docker-compose.server.yml" \
   -f "$compose_dir/deploy/docker-compose.postgres.yml" \
   -f "$compose_dir/deploy/docker-compose.production.yml" \
   logs --since "$since" --no-color --timestamps web api caddy |
+  sed -E 's/^[^|]*\| //' |
   grep -F -- "$request_id" |
+  grep -E '^[0-9]{4}-[0-9]{2}-[0-9]{2}T' |
   sort

deploy/monitor/error-digest.sh

@@ -51,7 +51,11 @@ if [ -z "$fixture" ]; then
   since="5m"
   [ "$mode" = "daily" ] && since="48h"
   generated_fixture="$(mktemp)"
-  docker compose -p jyotisha-production \
+  export APP_ENV_FILE='../.env.production'
+  export DATABASE_ENV_FILE='../.env.production.database'
+  export CADDYFILE_PATH='./Caddyfile.production.selfhosted'
+  export SITE_ADDRESS='https://jyotisha.chat'
+  sudo -n docker compose -p jyotisha-production \
     --env-file "$compose_dir/.env.production" \
     -f "$compose_dir/deploy/docker-compose.server.yml" \
     -f "$compose_dir/deploy/docker-compose.postgres.yml" \
@@ -81,5 +85,48 @@ elif [ "$mode" = "daily" ]; then
 fi
 
 if [ -n "$text" ]; then
-  post_monitor_alert "${text%$'\n'}"
+  now_epoch="$(date -u -d "$now_text" +%s)"
+  [[ "$now_epoch" =~ ^[0-9]+$ ]] || exit 2
+  dedup_file="$state_dir/error-digest-sent.tsv"
+  declare -A last_sent=()
+  declare -a send_keys=()
+  filtered=""
+  if [ -f "$dedup_file" ] && [ ! -L "$dedup_file" ]; then
+    while IFS=$'\t' read -r sent_epoch sent_service sent_route sent_code || [ -n "${sent_epoch:-}" ]; do
+      [[ "${sent_epoch:-}" =~ ^[0-9]+$ ]] || continue
+      [[ "${sent_service:-}" =~ ^[a-z0-9_-]+$ ]] || continue
+      [ -n "${sent_route:-}" ] || continue
+      [ -n "${sent_code:-}" ] || continue
+      last_sent["${sent_service}"$'\t'"${sent_route}"$'\t'"${sent_code}"]="$sent_epoch"
+    done <"$dedup_file"
+  fi
+  while IFS= read -r line || [ -n "$line" ]; do
+    [ -n "$line" ] || continue
+    if [[ "$line" =~ ^service=([a-z0-9_-]+)[[:space:]]route=([^[:space:]]+)[[:space:]]errorCode=([a-z][a-z0-9._-]*)[[:space:]] ]]; then
+      key="${BASH_REMATCH[1]}"$'\t'"${BASH_REMATCH[2]}"$'\t'"${BASH_REMATCH[3]}"
+      prev="${last_sent[$key]:-}"
+      if [ -n "$prev" ] && (( now_epoch - prev < 1800 )); then
+        continue
+      fi
+      send_keys+=("$key")
+    fi
+    filtered="${filtered}${line}"$'\n'
+  done <<<"${text%$'\n'}"
+  if [ -n "$filtered" ]; then
+    post_monitor_alert "${filtered%$'\n'}"
+    if [ "${#send_keys[@]}" -gt 0 ]; then
+      for key in "${send_keys[@]}"; do
+        last_sent["$key"]="$now_epoch"
+      done
+    fi
+    dedup_tmp="$dedup_file.tmp.$$"
+    : >"$dedup_tmp"
+    for key in "${!last_sent[@]}"; do
+      printf '%s\t%s\n' "${last_sent[$key]}" "$key" >>"$dedup_tmp"
+    done
+    if [ "$(uname -s)" = "Linux" ]; then
+      chmod 600 "$dedup_tmp"
+    fi
+    mv "$dedup_tmp" "$dedup_file"
+  fi
 fi

deploy/monitor/healthcheck-alert.sh

@@ -60,6 +60,7 @@ alert_open=0
 last_commit=""
 commit_alerted=""
 disk_alert_open=0
+backup_stale_open=0
 
 read_state() {
   local line value
@@ -89,6 +90,9 @@ read_state() {
       disk_alert_open=[01])
         disk_alert_open="${line#disk_alert_open=}"
         ;;
+      backup_stale_open=[01])
+        backup_stale_open="${line#backup_stale_open=}"
+        ;;
     esac
   done <"$state_file"
 }
@@ -96,8 +100,8 @@ read_state() {
 write_state() {
   local tmp
   tmp="$state_file.tmp.$$"
-  printf 'consecutive_failures=%s\nalert_open=%s\nlast_commit=%s\ncommit_alerted=%s\ndisk_alert_open=%s\n' \
-    "$consecutive_failures" "$alert_open" "$last_commit" "$commit_alerted" "$disk_alert_open" >"$tmp"
+  printf 'consecutive_failures=%s\nalert_open=%s\nlast_commit=%s\ncommit_alerted=%s\ndisk_alert_open=%s\nbackup_stale_open=%s\n' \
+    "$consecutive_failures" "$alert_open" "$last_commit" "$commit_alerted" "$disk_alert_open" "$backup_stale_open" >"$tmp"
   if [ "$(uname -s)" = "Linux" ]; then
     chmod 600 "$tmp"
   fi
@@ -224,13 +228,55 @@ check_disk() {
   fi
 }
 
+check_daily_backup() {
+  local record now_text now_epoch verified verified_epoch age stale hours
+  record="${JYOTISHA_DAILY_BACKUP_RECORD:-/opt/jyotisha-production/.state/daily-backup-latest.env}"
+  now_text="${JYOTISHA_NOW:-$(date -u +%Y-%m-%dT%H:%M:%SZ)}"
+  [[ "$now_text" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$ ]] || {
+    echo "clock is unsafe" >&2
+    exit 2
+  }
+  now_epoch="$(date -u -d "$now_text" +%s)"
+  [[ "$now_epoch" =~ ^[0-9]+$ ]] || exit 2
+  stale=1
+  hours=36
+  if [ -f "$record" ] && [ ! -L "$record" ]; then
+    verified="$(awk -F= '$1 == "VERIFIED_AT" { print $2 }' "$record" | tr -d '\r')"
+    if [[ "$verified" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$ ]]; then
+      verified_epoch="$(date -u -d "$verified" +%s)"
+      age=$((now_epoch - verified_epoch))
+      if [ "$age" -le 129600 ]; then
+        stale=0
+      else
+        hours=$((age / 3600))
+        if [ "$hours" -lt 36 ]; then
+          hours=36
+        fi
+      fi
+    fi
+  fi
+  if [ "$stale" -eq 1 ]; then
+    if [ "$backup_stale_open" -eq 0 ]; then
+      send_alert host - backup_stale "$hours" -
+      backup_stale_open=1
+    fi
+  elif [ "$backup_stale_open" -eq 1 ]; then
+    send_alert host - recovered 0 -
+    backup_stale_open=0
+  fi
+}
+
 read_state
 case "$mode" in
   health) check_health ;;
-  disk) check_disk ;;
+  disk)
+    check_disk
+    check_daily_backup
+    ;;
   all)
     check_health
     check_disk
+    check_daily_backup
     ;;
 esac
 write_state

deploy/monitor/notify-feishu.sh

@@ -63,27 +63,34 @@ alert_line_allowed() {
 
 post_monitor_alert() {
   local text="$1"
-  local state_dir line timestamp sign payload_file
+  local state_dir line timestamp sign payload_file response_file pending py curl_status code
   state_dir="${JYOTISHA_MONITOR_STATE_DIR:-/var/lib/jyotisha-monitor}"
   ensure_private_dir "$state_dir"
+  pending=""
   while IFS= read -r line || [ -n "$line" ]; do
     [ -n "$line" ] || continue
     alert_line_allowed "$line" || {
       echo "alert text has a field that is not allowed" >&2
       exit 1
     }
-    printf '%s\n' "$line" >>"$state_dir/alerts.log"
+    pending="${pending}${line}"$'\n'
   done <<<"$text"
+  [ -n "$pending" ] || return 0
+  pending="${pending%$'\n'}"
 
-  [ -n "${FEISHU_WEBHOOK_URL:-}" ] || return 0
+  if [ -z "${FEISHU_WEBHOOK_URL:-}" ]; then
+    printf '%s\n' "$pending" >>"$state_dir/alerts.log"
+    return 0
+  fi
   webhook_allowed "$FEISHU_WEBHOOK_URL" || {
     printf '%s\n' "service=monitor route=- errorCode=webhook_rejected count=1 requestId=- time=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >>"$state_dir/alerts.log"
     return 0
   }
 
   payload_file="$(mktemp)"
+  response_file="$(mktemp)"
   if [ "$(uname -s)" = "Linux" ]; then
-    chmod 600 "$payload_file"
+    chmod 600 "$payload_file" "$response_file"
   fi
   if [ -n "${FEISHU_SIGN_SECRET:-}" ]; then
     timestamp="$(date +%s)"
@@ -93,12 +100,45 @@ post_monitor_alert() {
   else
     FEISHU_ALERT_TEXT="$text" python_write_payload "$payload_file" plain
   fi
-  curl --silent --show-error --connect-timeout 10 --max-time 10 \
+  if [ -n "${JYOTISHA_PYTHON:-}" ]; then
+    py="$JYOTISHA_PYTHON"
+  elif command -v python3 >/dev/null 2>&1; then
+    py="python3"
+  else
+    py="python"
+  fi
+  curl_status=0
+  curl --silent --show-error --fail --connect-timeout 10 --max-time 10 \
     -H 'Content-Type: application/json' \
     -H 'Connection: close' \
     --data-binary @"$payload_file" \
-    "$FEISHU_WEBHOOK_URL" >/dev/null
-  rm -f -- "$payload_file"
+    -o "$response_file" \
+    "$FEISHU_WEBHOOK_URL" || curl_status=$?
+  code=""
+  if [ "$curl_status" -eq 0 ]; then
+    code="$("$py" - "$response_file" <<'PY'
+import json
+import sys
+
+try:
+    payload = json.load(open(sys.argv[1], encoding="utf-8"))
+except Exception:
+    print("")
+    raise SystemExit(0)
+if isinstance(payload, dict) and payload.get("code") == 0:
+    print("0")
+else:
+    print("")
+PY
+)"
+    code="$(printf '%s' "$code" | tr -d '\r')"
+  fi
+  rm -f -- "$payload_file" "$response_file"
+  if [ "$curl_status" -ne 0 ] || [ "$code" != "0" ]; then
+    echo "feishu webhook was not accepted" >&2
+    return 1
+  fi
+  printf '%s\n' "$pending" >>"$state_dir/alerts.log"
 }
 
 python_write_payload() {