fix(cold-start): split stale evidence from live blockers

2026-07-01 21:34:13 +08:00
parent 2cfb891e57
commit 10275a9d7b
5 changed files with 107 additions and 11 deletions
--- a/docs/LOGBOOK.md
+++ b/docs/LOGBOOK.md
@@ -1,3 +1,29 @@
+## 2026-07-01 — 21:32 cold-start 假 WARN 收斂與 live monitor 同步
+
+**照主線修正的問題**：
+- MOMO current-month false warning：`momo-drive-token-source-recovery-preflight.sh` 新增 `MOMO_LATEST_IMPORT_CLEAN` 與 `MOMO_SOURCE_ABSENT_WITHOUT_NEWER_DRIVE` 機器可讀欄位；`full-stack-cold-start-check.sh` 先吃 dedicated preflight，再把 `MOMO_MONTHLY_SYNC 0|0|-|-|-|-` + latest clean import + no newer Drive source 判為 not applicable，不再把沒有本月新來源的 0|0 當成 cold-start WARN。
+- 110 backup aggregate false warning：cold-start 現在把 current component freshness / critical config gap 與舊 `backup-all failed_count` 分開。`BACKUP_HEALTH_110 total=13 stale=0 missing_cron=0 missing_script=0 failed_count=5 config_failed=0 integrity_total=2 integrity_stale=0` 會列為 current backup health OK，並把舊 aggregate failed_count 只輸出 INFO evidence。
+- live 110 monitor 已同步本輪 source：`full-stack-cold-start-check.sh` hash `d0711f75dfb1ee680442c9d6cf2191741f3b27605f347c9ef2a25a4fed6d40ac`，`momo-drive-token-source-recovery-preflight.sh` hash `571d75e81c509683eb8a38fabbe81fc7822befe45206145f4fb4e865473f5254`；crontab 仍為每 10 分鐘執行 `cold-start-textfile-exporter.sh`。
+
+**驗證**：
+- `bash -n scripts/reboot-recovery/full-stack-cold-start-check.sh scripts/reboot-recovery/momo-drive-token-source-recovery-preflight.sh`：通過。
+- `python3.11 -m pytest scripts/reboot-recovery/tests/test_cold_start_monitor_bounded_probes.py -q`：`14 passed`。
+- `git diff --check`：通過。
+- live cold-start final：`PASS=95 WARN=1 BLOCKED=0`，artifact `/tmp/awoooi-cold-start-final-20260701-212632.log`。public routes / TLS 全部通過，StockPlatform 前一輪 502 已確認是 21:22 左右 web/admin/edge 替換 warmup，之後外部連續 5 次 `https://stock.wooo.work/` 回 `200`，final cold-start 亦回 `stock 200`。
+- K3s rollout 暫態已自然收斂：前一輪 `BAD_PODS=2` 是新 AWOOOI pod rollout 時舊 pod terminating；連續 6 次只讀觀察無非 Running/Completed pod，final cold-start `BAD_PODS 0`。
+- 110 textfile readback：`awoooi_cold_start_monitor_up=1`、`pass=95`、`warn=1`、`blocked=0`、`last_exit_code=1`、`last_result{result="degraded"}=1`、`last_run_duration_seconds=132`。
+- `REMOTE_COMMAND_TIMEOUT_SECONDS=90 SSH_CONNECT_TIMEOUT=15 bash scripts/reboot-recovery/verify-cold-start-monitor-deploy.sh`：`COLD_START_MONITOR_DEPLOY_PARITY_OK`，runtime state `monitor_up=1 warn=1 blocked=0 green=0 blocked_state=0`。
+- `REMOTE_COMMAND_TIMEOUT_SECONDS=90 SSH_CONNECT_TIMEOUT=15 bash scripts/reboot-recovery/full-stack-recovery-scorecard.sh`：`CORE_COLD_START_WARN_GATES=1`、`CORE_COLD_START_BLOCKED_GATES=0`、`CORE_COLD_START_DEPLOY_PARITY=1`、`CORE_REGISTRY_READY=1`、`DR_OFFSITE_EVIDENCE_READBACK=1`、`ESCROW_MISSING_COUNT=5`、`NEXT_STEP=complete_credential_escrow_review`。
+
+**仍維持 / 未完成**：
+- 目前唯一 cold-start WARN 是 MOMO daily data freshness：`MOMO_DAILY_FRESHNESS 7|2026-06-24`，且 preflight 顯示 `PASS=19 WARN=6 BLOCKED=0`、Drive intake `0`、failed folder `0`、沒有比最後 clean import 更新的 source candidate。可宣稱「服務恢復、沒有新來源候選」；不可宣稱「MOMO 業績資料已最新」。
+- `ESCROW_MISSING_COUNT=5` 仍是真正 DR completion gate；不得偽造 marker、不得把一般批准當成 credential escrow evidence。
+- 110 control path 仍有 intermittent timeout evidence：installer 首次 `scp` timeout，重試成功；110 當下 load 約 `7-8`，top 包含 `systemd`、`gitea`、StockPlatform Postgres、ClickHouse、Kafka、Docker。服務可用，但不得宣稱 110 SSH 永久穩定。
+
+**邊界**：未重啟主機，未 restart Docker / Nginx / K3s / DB / firewall，未刪 pod，未讀 secret value / token / `.env` / raw sessions / SQLite / auth，未寫 credential escrow marker，未使用 GitHub / `gh` / GitHub API，未恢復 generic runner。
+
+**下一步**：照 P0 主線處理唯一剩餘 cold-start WARN：MOMO daily source freshness / source-arrival gate；DR completion 另依 `complete_credential_escrow_review` 收 5 個 non-secret evidence。110 load / SSH timeout 留在 capacity/control-channel evidence，不再回頭重開 runner 支線。
+
 ## 2026-07-01 — 21:05 backup / DR freshness 收斂與高頻備份保護

 **照主線修正的問題**：
--- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md
+++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md
@@ -1,6 +1,6 @@
 # AWOOOI 全棧冷啟動與主機重啟 SOP

-> Version: v1.88
+> Version: v1.89
 > Last updated: 2026-07-01 Asia/Taipei
 > Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.

@@ -18,7 +18,9 @@ v1.79 active owner response template rule：同一輪 owner packet 產生後，p

 v1.80 / v1.81 credential escrow intake scorecard rule：同一輪 owner response preflight 後，必須用 `scripts/reboot-recovery/post-reboot-credential-escrow-intake-scorecard.py --summary-file "$ARTIFACT_DIR/summary.txt" --owner-packet-file <owner-packets.json> --response-file <owner-response-template-or-candidate.json> --offsite-report-file <offsite-report.txt> --escrow-status-file <escrow-status.txt>` 收斂 DR escrow gate。scorecard 只讀 sanitized artifacts；不得讀 secret value、不得寫 marker、不得送 owner request、不得開 runtime gate。placeholder readback 期望 `STATUS=blocked_waiting_non_secret_credential_escrow_evidence`、`EFFECTIVE_ESCROW_MISSING_COUNT=5`、`OWNER_RESPONSE_RECEIVED_COUNT=0`、`OWNER_RESPONSE_ACCEPTED_COUNT=0`、`RUNTIME_GATE_COUNT=0`、`CREDENTIAL_MARKER_WRITE_AUTHORIZED_COUNT=0`。若未來收到合格 redacted owner response 並由 preflight 回 `ready_for_independent_reviewer_acceptance`，scorecard 應轉為 `STATUS=ready_for_independent_reviewer_acceptance`；即使 marker 尚未寫入，也只能進 `independent_reviewer_acceptance_then_marker_dry_run`，不得直接寫 marker 或宣稱 `DR_COMPLETE`。

-2026-07-01 21:05 latest live summary：backup / DR freshness 已收斂，但仍不可宣稱 full green 或 10 分鐘全自動恢復完成。188 `backup_from_110` 已受控重跑成功，Harbor / Gitea / bitan backup 均 OK；188 exporter 讀回 `backup_from_110 fresh=1`、Gitea private bundle `expected_repo_missing_count=0`、`failed_repo_count=0`。110 `awoooi_db` high-frequency backup 20:00 因 `too many connections for role "awoooi"` 失敗；本輪在 DB 舊 OR 查詢為 `0`、Postgres CPU 已低的條件下走 controlled short window，產生 Restic snapshot `211a8948`，時間 `2026-07-01T20:58:20+08:00`，tags `service:awoooi,freq:6h,timestamp:20260701_204344`，tmp dump 已清除。正式保守調整為 `awoooi CONNECTION LIMIT 4`，保留 `statement_timeout=750ms`、`max_parallel_workers_per_gather=0`、`enable_seqscan=off`；rollback 是 `ALTER ROLE awoooi CONNECTION LIMIT 2` 並保留同樣 role config。110 backup exporter 讀回 `awoooi_backup_job_fresh{job="awoooi_db"} 1`、`awoooi_backup_dr_phase{next_step="complete_credential_escrow_review"} 3`、`awoooi_backup_dr_credential_escrow_missing_count 5`。`full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1` 回 `PASS=93 WARN=3 BLOCKED=0`；`full-stack-recovery-scorecard.sh` 回 `CORE_COLD_START_WARN_GATES=3`、`CORE_COLD_START_BLOCKED_GATES=0`、`CORE_COLD_START_DEPLOY_PARITY=1`、`CORE_REGISTRY_READY=1`、`DR_OFFSITE_EVIDENCE_READBACK=1`、`NEXT_STEP=complete_credential_escrow_review`。Allowed declaration：public routes / AWOOOI service / Gitea / Harbor registry / 188 backup-from-110 / 110 awoooi_db freshness 已恢復，cold-start hard blockers `0`。Forbidden declaration：full green、10 分鐘全自動恢復完成、DR complete、credential escrow complete、MOMO data 最新、110 SSH 永久穩定。剩餘 WARN 是 MOMO current-month/source freshness warning 與 110 aggregate `backup-all` 上次 failed components。高頻全量 `pg_dump` 本輪耗時 `1152s` 並產生 `28G` dump，後續 P0 必須建立專用 backup DB role / `AWOOOI_BACKUP_DATABASE_URL`，並把大型可重建 snapshot 表改成分層備份或 retention 後納入 full backup；不得把 app role 放寬當長期唯一解法。
+2026-07-01 21:32 latest live summary：cold-start 假 WARN 已收斂，hard blockers 維持 `0`，但仍不可宣稱 full green 或 10 分鐘全自動恢復完成。`full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1` final artifact `/tmp/awoooi-cold-start-final-20260701-212632.log` 回 `PASS=95 WARN=1 BLOCKED=0`；public routes / TLS 全部通過，StockPlatform 21:22 左右的 `502` 已確認是 web/admin/edge 替換 warmup，外部連續 5 次 `https://stock.wooo.work/` 回 `200`，final cold-start 亦回 `stock 200`。K3s `BAD_PODS=2` 也是 rollout 暫態，連續 6 次只讀觀察已無非 Running/Completed pod，final `BAD_PODS 0`。MOMO current-month `0|0|-|-|-|-` 不再列為 WARN：`momo-drive-token-source-recovery-preflight.sh` 會輸出 `MOMO_LATEST_IMPORT_CLEAN` 與 `MOMO_SOURCE_ABSENT_WITHOUT_NEWER_DRIVE`，cold-start 讀到 latest clean import 且 Drive 無更新 source candidate 時，判定 current-month sync not applicable。110 backup current health 也不再被舊 aggregate log 壓成 WARN：`BACKUP_HEALTH_110 total=13 stale=0 missing_cron=0 missing_script=0 failed_count=5 config_failed=0 integrity_total=2 integrity_stale=0` 代表 current component freshness / critical config / integrity OK；`failed_count=5` 保留為 INFO evidence，等下一次 full `backup-all` 自然覆蓋。live 110 monitor 已同步，hash `full-stack-cold-start-check.sh=d0711f75dfb1ee680442c9d6cf2191741f3b27605f347c9ef2a25a4fed6d40ac`、`momo-drive-token-source-recovery-preflight.sh=571d75e81c509683eb8a38fabbe81fc7822befe45206145f4fb4e865473f5254`；110 textfile 讀回 `awoooi_cold_start_monitor_up=1`、`pass=95`、`warn=1`、`blocked=0`、`last_exit_code=1`、`last_result{result="degraded"}=1`。`verify-cold-start-monitor-deploy.sh` 回 `COLD_START_MONITOR_DEPLOY_PARITY_OK`；`full-stack-recovery-scorecard.sh` 回 `CORE_COLD_START_WARN_GATES=1`、`CORE_COLD_START_BLOCKED_GATES=0`、`CORE_COLD_START_DEPLOY_PARITY=1`、`CORE_REGISTRY_READY=1`、`DR_OFFSITE_EVIDENCE_READBACK=1`、`ESCROW_MISSING_COUNT=5`、`NEXT_STEP=complete_credential_escrow_review`。Allowed declaration：public routes / AWOOOI service / Gitea / Harbor registry / 188 backup-from-110 / 110 awoooi_db freshness / K3s rollout / Stock public route 已恢復，cold-start hard blockers `0`。Forbidden declaration：full green、10 分鐘全自動恢復完成、DR complete、credential escrow complete、MOMO data 最新、110 SSH 永久穩定。唯一 cold-start WARN 是 MOMO daily data freshness：`MOMO_DAILY_FRESHNESS 7|2026-06-24`，且 Drive intake / failed folder 無新候選；必須走 source-arrival / formal import gate，不可用假資料或手動 DB 寫入掩蓋。
+
+2026-07-01 21:05 previous live summary：backup / DR freshness 已收斂，但仍不可宣稱 full green 或 10 分鐘全自動恢復完成。188 `backup_from_110` 已受控重跑成功，Harbor / Gitea / bitan backup 均 OK；188 exporter 讀回 `backup_from_110 fresh=1`、Gitea private bundle `expected_repo_missing_count=0`、`failed_repo_count=0`。110 `awoooi_db` high-frequency backup 20:00 因 `too many connections for role "awoooi"` 失敗；本輪在 DB 舊 OR 查詢為 `0`、Postgres CPU 已低的條件下走 controlled short window，產生 Restic snapshot `211a8948`，時間 `2026-07-01T20:58:20+08:00`，tags `service:awoooi,freq:6h,timestamp:20260701_204344`，tmp dump 已清除。正式保守調整為 `awoooi CONNECTION LIMIT 4`，保留 `statement_timeout=750ms`、`max_parallel_workers_per_gather=0`、`enable_seqscan=off`；rollback 是 `ALTER ROLE awoooi CONNECTION LIMIT 2` 並保留同樣 role config。110 backup exporter 讀回 `awoooi_backup_job_fresh{job="awoooi_db"} 1`、`awoooi_backup_dr_phase{next_step="complete_credential_escrow_review"} 3`、`awoooi_backup_dr_credential_escrow_missing_count 5`。`full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1` 回 `PASS=93 WARN=3 BLOCKED=0`；`full-stack-recovery-scorecard.sh` 回 `CORE_COLD_START_WARN_GATES=3`、`CORE_COLD_START_BLOCKED_GATES=0`、`CORE_COLD_START_DEPLOY_PARITY=1`、`CORE_REGISTRY_READY=1`、`DR_OFFSITE_EVIDENCE_READBACK=1`、`NEXT_STEP=complete_credential_escrow_review`。Allowed declaration：public routes / AWOOOI service / Gitea / Harbor registry / 188 backup-from-110 / 110 awoooi_db freshness 已恢復，cold-start hard blockers `0`。Forbidden declaration：full green、10 分鐘全自動恢復完成、DR complete、credential escrow complete、MOMO data 最新、110 SSH 永久穩定。剩餘 WARN 是 MOMO current-month/source freshness warning 與 110 aggregate `backup-all` 上次 failed components。高頻全量 `pg_dump` 本輪耗時 `1152s` 並產生 `28G` dump，後續 P0 必須建立專用 backup DB role / `AWOOOI_BACKUP_DATABASE_URL`，並把大型可重建 snapshot 表改成分層備份或 retention 後納入 full backup；不得把 app role 放寬當長期唯一解法。

 2026-07-01 19:32 latest live summary：110 卡住的根因已確認並修復。主機 ping / SSH / Prometheus / Gitea / registry 均可達，但 `awoooi-startup-110.service` 因 `/usr/local/bin/awoooi-startup-110.sh` 被替換成 runner/CD lane fail-closed stub 而固定 `exit 75`，造成 post-reboot systemd degraded 與 cold-start warning。live stub hash 為 `55e1e87d44cf5b8cefb714a49d085b886697d856569ce9478537059d49129f88`，repo 完整 startup script hash 為 `fcc67c7dde889b3cf8ffeca89cc5e58407fabc6f5a80554e39f3f1465f90b318`。修復方式固定使用 `scripts/reboot-recovery/repair-110-startup-script-stub.sh`：預設 `--verify` 只讀；`--apply` 只上傳 repo 完整 startup script、備份 live stub、解除 immutable、install 到 `/usr/local/bin/awoooi-startup-110.sh`、`systemctl daemon-reload`，並只 `reset-failed awoooi-startup-110.service`；`--start-service` 需另外明確指定，因為它會跑完整 startup checks。live apply 指令為 `BACKUP_SUFFIX=before-full-startup-restore-20260701-1928 bash scripts/reboot-recovery/repair-110-startup-script-stub.sh --apply`，post-readback `remote_matches_expected=1`、`remote_is_runner_failclosed_stub=0`，rollback backup 為 `/usr/local/bin/awoooi-startup-110.sh.before-full-startup-restore-20260701-1928`。之後清除 stale `systemd-ask-password-console.path` / `systemd-ask-password-wall.path` failed state；110 已回 `systemctl is-system-running=running`、failed units `0`。最新 cold-start 讀回 `PASS=90 WARN=6 BLOCKED=0`、`FAILED_UNITS_110 0`；110 startup stub / systemd degraded 不再是 blocker。禁止再把 live `/usr/local/bin/awoooi-startup-110.sh` 當 runner/CD lane opener 封成 stub；runner fail-closed enforcer 只能封 runner binary、runner unit、controlled opener artifact 或 upload opener source，不得封 host startup recovery script。Allowed declaration：110 不再卡在 startup stub / systemd degraded，cold-start hard blockers `0`。Forbidden declaration：full green、10 分鐘全自動恢復已完成、DR complete、credential escrow complete、MOMO data 最新或 110 SSH 永久穩定；剩餘 warnings 仍需由 backup / DR escrow / alert 收斂處理。

--- a/scripts/reboot-recovery/full-stack-cold-start-check.sh
+++ b/scripts/reboot-recovery/full-stack-cold-start-check.sh
@@ -770,7 +770,15 @@ printf "%s\n" "$momo_drive_source_probe"
    awk '/MOMO_GDRIVE_TOKEN_STAT / {split($2,a,":"); split($3,b,"="); exit !(a[1] == b[2] && a[3] <= 600)}' <<<"$out" && ok "188 momo Google Drive token ownership matches scheduler userns" || warn "188 momo Google Drive token ownership/writeback not confirmed"
    grep -Fq "MOMO_IMPORT_CONFIG 當日業績匯入|即時業績_當日" <<<"$out" && ok "188 momo Drive import config points to expected daily-sales intake" || fail "188 momo Drive import config drifted from expected daily-sales intake"
    awk '/MOMO_LATEST_IMPORT_JOB / {split($2,a,"|"); exit !(a[1] ~ /^[0-9]+$/ && a[2] == "completed" && a[6] == a[7] && a[8] == 0)}' <<<"$out" && ok "188 momo latest daily import job completed cleanly" || warn "188 momo latest daily import job not confirmed clean"
-    awk '/MOMO_MONTHLY_SYNC / {split($2,a,"|"); exit !(a[1] > 0 && a[1] == a[2] && a[3] == a[5] && a[4] == a[6])}' <<<"$out" && ok "188 momo current-month snapshot and realtime tables match" || warn "188 momo current-month snapshot/realtime sync not confirmed"
+    momo_latest_import_clean=$(awk '
+      $1 == "MOMO_LATEST_IMPORT_JOB" {
+        seen=1
+        split($2,a,"|")
+        if (a[1] ~ /^[0-9]+$/ && a[2] == "completed" && a[6] == a[7] && a[8] == 0) print 1;
+        else print 0;
+      }
+      END { if (!seen) print 0 }
+    ' <<<"$out")
    momo_source_stale_only=$(awk '
      $1 == "MOMO_DRIVE_INTAKE_COUNT" {intake=$2+0}
      $1 == "MOMO_DRIVE_FAILED_COUNT" {failed=$2+0}
@@ -780,19 +788,42 @@ printf "%s\n" "$momo_drive_source_probe"
        if (intake == 0 && failed == 0 && global ~ /^[0-9]{4}-[0-9]{2}-[0-9]{2}$/ && completed ~ /^[0-9]{4}-[0-9]{2}-[0-9]{2}$/ && global <= completed) print 1;
        else print 0;
      }' <<<"$out")
+    momo_source_preflight_output=""
+    momo_source_preflight_summary=""
+    if [ "$momo_source_stale_only" != "1" ] && [ "$momo_latest_import_clean" = "1" ] && [ -x "$MOMO_SOURCE_PREFLIGHT_SCRIPT" ]; then
+      momo_source_preflight_output="$(
+        "$MOMO_SOURCE_PREFLIGHT_SCRIPT" \
+          --host ollama@192.168.0.188 \
+          --freshness-max-days 2 2>/dev/null || true
+      )"
+      momo_source_preflight_summary="$(awk '/^MOMO_DRIVE_TOKEN_SOURCE_PREFLIGHT / {line=$0} END {print line}' <<<"$momo_source_preflight_output")"
+      [ -n "$momo_source_preflight_summary" ] && echo "$momo_source_preflight_summary"
+      if awk '$1 == "MOMO_SOURCE_ABSENT_WITHOUT_NEWER_DRIVE" {seen=1; ok=($2 == 1)} END {exit !(seen && ok)}' <<<"$momo_source_preflight_output"; then
+        momo_source_stale_only=1
+      fi
+    fi
+    if awk '/MOMO_MONTHLY_SYNC / {seen=1; split($2,a,"|"); ok=(a[1] > 0 && a[1] == a[2] && a[3] == a[5] && a[4] == a[6])} END {exit !(seen && ok)}' <<<"$out"; then
+      ok "188 momo current-month snapshot and realtime tables match"
+    elif [ "$momo_latest_import_clean" = "1" ] && [ "$momo_source_stale_only" = "1" ] && awk '/MOMO_MONTHLY_SYNC / {seen=1; split($2,a,"|"); ok=(a[1] == 0 && a[2] == 0)} END {exit !(seen && ok)}' <<<"$out"; then
+      ok "188 momo current-month sync not applicable; Drive has no newer source than last clean import"
+    else
+      warn "188 momo current-month snapshot/realtime sync not confirmed"
+    fi
    if awk '/MOMO_DAILY_FRESHNESS / {split($2,a,"|"); exit !(a[1] ~ /^[0-9]+$/ && a[1] >= 0 && a[1] <= 2)}' <<<"$out"; then
      ok "188 momo daily sales data fresh enough"
    elif awk '/MOMO_DAILY_FRESHNESS / {split($2,a,"|"); exit !(a[1] ~ /^[0-9]+$/ && a[1] >= 3)}' <<<"$out"; then
      if [ "$momo_source_stale_only" = "1" ]; then
        warn "188 momo daily sales stale but Drive has no newer source candidate"
      elif [ -x "$MOMO_SOURCE_PREFLIGHT_SCRIPT" ]; then
-        momo_source_preflight_summary="$(
-          "$MOMO_SOURCE_PREFLIGHT_SCRIPT" \
-            --host ollama@192.168.0.188 \
-            --freshness-max-days 2 2>/dev/null \
-            | awk '/^MOMO_DRIVE_TOKEN_SOURCE_PREFLIGHT / {line=$0} END {print line}' || true
-        )"
-        [ -n "$momo_source_preflight_summary" ] && echo "$momo_source_preflight_summary"
+        if [ -z "$momo_source_preflight_summary" ]; then
+          momo_source_preflight_output="$(
+            "$MOMO_SOURCE_PREFLIGHT_SCRIPT" \
+              --host ollama@192.168.0.188 \
+              --freshness-max-days 2 2>/dev/null || true
+          )"
+          momo_source_preflight_summary="$(awk '/^MOMO_DRIVE_TOKEN_SOURCE_PREFLIGHT / {line=$0} END {print line}' <<<"$momo_source_preflight_output")"
+          [ -n "$momo_source_preflight_summary" ] && echo "$momo_source_preflight_summary"
+        fi
        if grep -q "BLOCKED=0" <<<"$momo_source_preflight_summary"; then
          warn "188 momo daily sales stale but source preflight has no hard blocker"
        elif awk '/MOMO_SOURCE_EMPTY_EVIDENCE_LINES / {exit !($2 > 0)}' <<<"$out"; then
@@ -844,7 +875,12 @@ fi
    awk '/TEXTFILE_110 storage_health.prom age=/ {split($3,a,"="); exit !(a[2] < 300)}' <<<"$out" && ok "110 storage health exporter fresh" || warn "110 storage health exporter stale"
    awk '/TEXTFILE_110 backup_health.prom age=/ {split($3,a,"="); exit !(a[2] < 900)}' <<<"$out" && ok "110 backup health exporter fresh" || warn "110 backup health exporter stale"
    grep -q "STORAGE_HEALTH_110 root_readonly=0 current=0" <<<"$out" && ok "110 current boot storage health clean" || warn "110 storage health not clean"
-    grep -q "BACKUP_HEALTH_110 total=" <<<"$out" && awk '/BACKUP_HEALTH_110/ {split($3,a,"="); split($4,b,"="); split($5,c,"="); split($6,d,"="); split($7,e,"="); exit !((a[2]+b[2]+c[2]) == 0 && d[2] == 0 && e[2] == 0)}' <<<"$out" && ok "110 backup health has no stale expected jobs" || warn "110 latest aggregate/config backup had failed components; rerun backup-all after 120 recovers"
+    if grep -q "BACKUP_HEALTH_110 total=" <<<"$out" && awk '/BACKUP_HEALTH_110/ {split($3,a,"="); split($4,b,"="); split($5,c,"="); split($7,e,"="); exit !((a[2]+b[2]+c[2]+e[2]) == 0)}' <<<"$out"; then
+      ok "110 backup health has no stale expected jobs or critical config gaps"
+      awk '/BACKUP_HEALTH_110/ {split($6,a,"="); exit !(a[2] > 0)}' <<<"$out" && echo "INFO 110 latest aggregate backup-all still records failed_count but current component freshness is clean"
+    else
+      warn "110 backup health has stale expected jobs or critical config gaps"
+    fi
    awk '/BACKUP_HEALTH_110/ {split($9,a,"="); exit !(a[2] == 0)}' <<<"$out" && ok "110 backup integrity and restore drill fresh" || warn "110 backup integrity or restore drill stale"
  else
    warn "110 schedule check unavailable"
--- a/scripts/reboot-recovery/momo-drive-token-source-recovery-preflight.sh
+++ b/scripts/reboot-recovery/momo-drive-token-source-recovery-preflight.sh
@@ -342,6 +342,8 @@ if [[ "$drive_intake_count" -eq 0 \
  && [[ "$drive_global_latest_date" < "$job_completed_date" || "$drive_global_latest_date" == "$job_completed_date" ]]; then
  source_absent_without_newer_drive=1
 fi
+printf 'MOMO_LATEST_IMPORT_CLEAN %s\n' "$latest_job_clean"
+printf 'MOMO_SOURCE_ABSENT_WITHOUT_NEWER_DRIVE %s\n' "$source_absent_without_newer_drive"

 monthly_sync="$(value_for DB_MONTHLY_SYNC)"
 IFS='|' read -r sync_snapshot_count sync_monthly_count sync_dmin sync_dmax sync_mmin sync_mmax <<<"$monthly_sync"
--- a/scripts/reboot-recovery/tests/test_cold_start_monitor_bounded_probes.py
+++ b/scripts/reboot-recovery/tests/test_cold_start_monitor_bounded_probes.py
@@ -5,6 +5,9 @@ from pathlib import Path

 ROOT = Path(__file__).resolve().parents[3]
 COLD_START_CHECK = ROOT / "scripts" / "reboot-recovery" / "full-stack-cold-start-check.sh"
+MOMO_SOURCE_PREFLIGHT = (
+    ROOT / "scripts" / "reboot-recovery" / "momo-drive-token-source-recovery-preflight.sh"
+)
 INSTALL_COLD_START = (
    ROOT / "scripts" / "reboot-recovery" / "install-cold-start-monitor-110.sh"
 )
@@ -49,6 +52,33 @@ def test_full_stack_cold_start_check_bounds_ssh_probes() -> None:
    assert "SSH_110_RECOVERY_PACKAGE_NEXT_ACTION verify_or_preinstall_local_recovery_package_from_console_before_harbor_repair_retry" in text


+def test_cold_start_momo_current_month_handles_no_new_source_without_false_warn() -> None:
+    text = COLD_START_CHECK.read_text(encoding="utf-8")
+
+    assert "momo_latest_import_clean" in text
+    assert "momo_source_stale_only" in text
+    assert "MOMO_SOURCE_ABSENT_WITHOUT_NEWER_DRIVE" in text
+    assert "Drive has no newer source than last clean import" in text
+    assert "a[1] == 0 && a[2] == 0" in text
+
+
+def test_momo_source_preflight_emits_machine_readable_no_new_source_evidence() -> None:
+    text = MOMO_SOURCE_PREFLIGHT.read_text(encoding="utf-8")
+
+    assert "MOMO_LATEST_IMPORT_CLEAN" in text
+    assert "MOMO_SOURCE_ABSENT_WITHOUT_NEWER_DRIVE" in text
+    assert "source_absent_without_newer_drive=1" in text
+
+
+def test_cold_start_110_backup_splits_current_freshness_from_old_aggregate_failure() -> None:
+    text = COLD_START_CHECK.read_text(encoding="utf-8")
+
+    assert "critical config gaps" in text
+    assert "current component freshness is clean" in text
+    assert "split($6,a,\"=\"); exit !(a[2] > 0)" in text
+    assert "split($7,e,\"=\"); exit !((a[2]+b[2]+c[2]+e[2]) == 0)" in text
+
+
 def test_recovery_scorecard_bounds_offsite_evidence_ssh() -> None:
    text = RECOVERY_SCORECARD.read_text(encoding="utf-8")