docs(ops): record post-start wrapper live readback [skip ci]

2026-06-25 14:46:42 +08:00
parent 37ab97d4e1
commit 65209cbbc1
5 changed files with 105 additions and 23 deletions
--- a/docs/LOGBOOK.md
+++ b/docs/LOGBOOK.md
@@ -1,3 +1,30 @@
+## 2026-06-25｜14:41 post-start quick check live wrapper 分級讀回
+
+**背景**：第一版 `post-start-quick-check.sh` live run 將預期中的 `escrow_missing=5` 與 MOMO 非服務面 warning 一併算成 `DEGRADED`，容易讓重啟 SOP 看起來永遠差一點。這不符合本輪目標：服務恢復、資料新鮮、備份健康、DR escrow、Wazuh registry 必須分層判定。
+
+**修正**：
+- `scripts/reboot-recovery/post-start-quick-check.sh` 將 warning 分成 `SERVICE`、`BOUNDARY`、`EVIDENCE`。
+- `SERVICE>0` 才回 `RESULT=DEGRADED` / exit `1`。
+- `BOUNDARY>0` 且 service blocker 為 `0` 時回 `RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED` / exit `0`。
+- `EVIDENCE>0` 且 service blocker / boundary 為 `0` 時回 `RESULT=GREEN_WITH_EVIDENCE_WARNINGS` / exit `0`。
+- `docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md` 已補 wrapper exit code 與分級語意。
+
+**14:41 live read-only wrapper 證據**：
+- `scripts/reboot-recovery/post-start-quick-check.sh --no-color` 回 exit code `0`。
+- Hosts / SSH：110、120、121、188 ping 與 SSH port OK。
+- Cold-start：`PASS=89 WARN=0 BLOCKED=0`，Result `GREEN`。
+- MOMO：health `V10.676`，dedicated preflight `PASS=19 WARN=2 BLOCKED=0`；latest job `57 completed`，`DB_DAILY_FRESHNESS 1|2026-06-24`，current-month DB parity through `2026-06-24`。
+- Backup：110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`offsite_fresh=1`，`rclone_gdrive_fresh=1`，`escrow_missing=5`。
+- Routes：AWOOI API、IwoooS、MOMO health、Stock direct smoke all `200`。
+- 110 CPU：load 約 `4.09 / 4.52 / 4.39`；top processes show active Gitea / runner / build-test load and platform services，沒有 orphan Chrome 復發證據。
+- Wrapper summary：`POST_START_QUICK_CHECK PASS=18 WARN=2 BLOCKED=0`，`POST_START_QUICK_CHECK_WARNINGS SERVICE=0 BOUNDARY=1 EVIDENCE=1`，`RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。
+
+**判定**：
+- 可宣稱：重啟後服務面 / public routes / K3s / MOMO data freshness / backup core 已恢復；post-start quick check wrapper 可作為 T+10 分鐘標準入口。
+- 不可宣稱：DR complete、credential escrow complete、Wazuh manager registry accepted、Wazuh active response / host write / runtime security gate。`escrow_missing=5` 必須保留為 DR blocker，不得偽造 marker。
+
+**邊界**：本輪只做 read-only wrapper live run、repo-side script / docs 修正與 guard；沒有 Docker / systemd / Nginx / firewall / K8s / ArgoCD / Wazuh runtime 寫操作，沒有 import，沒有讀 token，沒有推 `gitea/main` 或觸發 CD。
+
 ## 2026-06-25｜重啟後一頁式總檢查 SOP 補強

 **背景**：SOP v1.51 已能判定 full-stack service GREEN，但長 SOP 太完整，不適合作為每次重啟後 T+10 分鐘內的操作頁。為避免下一次又在 route 200、container healthy、DB freshness、backup、Wazuh registry、DR escrow 之間混淆，本輪新增一頁式 post-start quick check。
--- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md
+++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md
@@ -1,6 +1,6 @@
 # AWOOOI 全棧冷啟動與主機重啟 SOP

-> Version: v1.52
+> Version: v1.53
 > Last updated: 2026-06-25 Asia/Taipei
 > Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.

@@ -12,20 +12,21 @@

 若只是重啟後要快速判斷能不能宣稱恢復，先跑一頁式總檢查：`scripts/reboot-recovery/post-start-quick-check.sh --no-color`，並以 `docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md` 作為人工 fallback。長 SOP 保留完整背景、例外處理與 Plan B；短版 wrapper / checklist 負責每次 T+10 分鐘內的固定判定。

-2026-06-25 14:16 live read-only refresh supersedes the 11:53 BLOCKED wording. Hosts, routes, K3s, AWOOOI API health, MOMO service health, MOMO business data freshness, backup core/offsite, and core monitoring/exporter surfaces are green for controlled runner/CD release. MOMO is healthy on `V10.674`; latest import job `57` completed cleanly; `MOMO_DAILY_FRESHNESS 1|2026-06-24`; current-month daily snapshot and realtime tables match through `2026-06-24`. Full-stack service readiness is now GREEN, but DR remains blocked by missing credential escrow evidence (`escrow_missing=5`). Do not turn this into a DR complete or security/runtime acceptance claim. Wazuh host registry acceptance remains outside this SOP lane and is still not complete.
+2026-06-25 14:41 post-start wrapper live read-only refresh supersedes the first wrapper wording. Hosts, routes, K3s, AWOOOI API health, MOMO service health, MOMO business data freshness, backup core/offsite, and core monitoring/exporter surfaces are green for controlled runner/CD release. MOMO is healthy on `V10.676`; latest import job `57` completed cleanly; `MOMO_DAILY_FRESHNESS 1|2026-06-24`; current-month daily snapshot and realtime tables match through `2026-06-24`. `post-start-quick-check.sh` now separates `SERVICE / BOUNDARY / EVIDENCE` warnings and returns `RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED` when service blockers are zero but `escrow_missing=5` remains. Do not turn this into a DR complete or security/runtime acceptance claim. Wazuh host registry acceptance remains outside this SOP lane and is still not complete.

 ```text
 Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%.
-Live cold-start read-only check: 2026-06-25 14:16 PASS=89 WARN=0 BLOCKED=0, Result=GREEN.
+Live cold-start read-only check: 2026-06-25 14:41 wrapper delegated cold-start PASS=89 WARN=0 BLOCKED=0, Result=GREEN.
+Post-start quick check: 2026-06-25 14:41 PASS=18 WARN=2 BLOCKED=0; warning split SERVICE=0 BOUNDARY=1 EVIDENCE=1; Result=FULL_STACK_GREEN_DR_ESCROW_BLOCKED; exit code 0.
 Repo-side cold-start v1.42+ live read-only run: MOMO source absence / stale data blocker is cleared by import job 57 and `MOMO_DAILY_FRESHNESS 1|2026-06-24`. Live 110 script sync is not claimed until a separate approved deployment/sync happens.
 110 live-sync parity: 2026-06-24 23:15 read-only `verify-cold-start-monitor-deploy.sh` correctly BLOCKED because repo script hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`. Do not use live 110 monitor output to prove v1.42 behavior until the approved live-sync gate in §13.3.1 passes.
 Service state: FULL_STACK_GREEN_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, public routes/TLS green, MOMO data fresh, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared.
 Runtime release state: API/Web/Worker are ready; production API health returns healthy with `environment=prod`, `mock_mode=false`, and postgresql / redis / openclaw / signoz / gcp ollama providers up. 14:16 direct route smoke returned 200 for AWOOOI API, `/zh-TW/iwooos`, MOMO health, and Stock; cold-start raw route gate returned all expected route statuses, including redirects such as awoooi web=307 and sentry=302.
-MOMO release state: mo.wooo.work health is healthy on version V10.674 after 14:14-14:15 lifecycle / replacement events observed via Docker metadata. `momo-pro-system`, `momo-scheduler`, and `momo-telegram-bot` are healthy; scheduler `RestartCount=0`. 14:16 dedicated preflight records `MOMO_CONTAINER_REPLACE_EVENTS_45M 11`, so retain a stability note, but no service blocker remains.
+MOMO release state: mo.wooo.work health is healthy on version V10.676 after 14:14-14:15 lifecycle / replacement events observed via Docker metadata. `momo-pro-system`, `momo-scheduler`, and `momo-telegram-bot` are healthy; scheduler `RestartCount=0`. 14:41 dedicated preflight returns PASS=19 WARN=2 BLOCKED=0, so retain stability / evidence notes, but no service blocker remains.
 MOMO data state: current-month daily_sales_snapshot and realtime_sales_monthly match through 2026-06-24: `daily_sales_snapshot=109061|2025-07-01|2026-06-24`, `MOMO_MONTHLY_SYNC 15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`, and `MOMO_DAILY_FRESHNESS 1|2026-06-24`. Latest import job is `57 completed|即時業績_當日.xlsx|2026-06-25T13:16:47.359958|2026-06-25T13:18:02.964985|15383|15383|0`.
 Google Drive / source-file state: 14:16 cold-start reports `MOMO_GDRIVE_TOKEN_STAT 100000:100000:600 scheduler_uid=100000`. Dedicated preflight confirms host token metadata matches scheduler UID and restrictive mode; container token artifact exists with mode `600`. Token content was not read. Future Drive auth/API failure must still be treated as failed import evidence rather than no-file success.
 110 CPU/load readback: 2026-06-25 10:58 user-approved minimal SIGTERM targeted only orphan `stockplatform-review-bulk-ux` Chrome process groups `438005`, `471295`, `640155`, and `670628`; `OLD_GROUPS_REMAINING` returned empty. 11:20 readback shows remaining CPU is active `stockplatform-product-ux-smoke.mjs` with parent node process plus install/build work, not orphan Chrome. No Docker/systemd/Nginx/firewall/K8s write was performed; do not cancel active CI/smoke unless separately approved.
-Backup / monitoring state: 14:16 backup-status core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-25 02:35:09, and escrow_missing=5.
+Backup / monitoring state: 14:41 wrapper readback confirms backup core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-25 02:35:09, and escrow_missing=5.
 Notification-noise state: healthy AWOOOI heartbeat is suppressed; heartbeat warning dedupe uses stable actionable fingerprints so HTTP status / timeout / latency drift does not create a new Telegram event every 30 minutes; MOMO Pro monitor uses https://mo.wooo.work/health as primary truth and no longer checks the 188 root path; MoWoooWorkDown now labels component=momo-pro-system and requires public/local/container/data-freshness triage instead of blind restart; docker-health-monitor keeps 5-minute repair cadence but has a separate 30-minute Telegram fallback cooldown; Bitan public-content check keeps failure alerting with same-fingerprint cooldown and one recovery notice.
 Monitoring coverage recovery state: if CD post-deploy fails only because `scripts/generate_monitoring.py --check` reports `nginx-exporter` down on `192.168.0.188:9113`, first verify 188 `stub_status` and restore the stateless exporter with `scripts/ops/188-nginx-exporter-restore.sh`; do not reload Nginx or restart product containers for this symptom.
 Allowed declaration: full-stack service readiness is GREEN for controlled runner/CD release; core hosts, routes, K3s, backup/exporter surfaces, AWOOOI API health, MOMO service health, and MOMO data freshness are green for the latest read-only evidence set.
--- a/docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md
+++ b/docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md
@@ -1,6 +1,6 @@
 # 主機重啟後一頁式總檢查

-> Version: v1.0
+> Version: v1.1
 > Last updated: 2026-06-25 Asia/Taipei
 > Scope: 110 / 120 / 121 / 188 post-reboot service recovery. 112 Kali / Wazuh / active scan 不屬於本流程。

@@ -47,7 +47,7 @@
 scripts/reboot-recovery/post-start-quick-check.sh --no-color
 ```

-此 wrapper 只做 read-only 檢查，並委派既有 cold-start / MOMO preflight / backup-status；不 restart、不 reload、不 import、不改 K8s、不讀 token 內容。若 wrapper 因某個 SSH 權限或路徑失敗，再依下列分段命令手動補證據。
+此 wrapper 只做 read-only 檢查，並委派既有 cold-start / MOMO preflight / backup-status；不 restart、不 reload、不 import、不改 K8s、不讀 token 內容。wrapper 會把 warning 分成 `SERVICE`、`BOUNDARY`、`EVIDENCE` 三類，避免把 `escrow_missing>0` 誤判成服務降級。若 wrapper 因某個 SSH 權限或路徑失敗，再依下列分段命令手動補證據。

 ### Step 1 - 主機與 SSH

@@ -148,6 +148,13 @@ ssh wooo@192.168.0.110 'uptime; vmstat 1 5; ps -eo pid,ppid,pgid,stat,pcpu,pmem,
 | `BLOCKED_HOST_OR_K3S` | 不可宣稱全棧恢復；先修主機 / K3s。 |
 | `BLOCKED_BACKUP_CORE` | 不可宣稱恢復完成；備份紅燈優先。 |
 | `BLOCKED_WAZUH_REGISTRY` | 不屬於本 SOP 的服務恢復 blocker；必須交給 IwoooS / Wazuh lane，不可改 Wazuh runtime。 |
+| `GREEN_WITH_EVIDENCE_WARNINGS` | 服務可宣稱恢復，但仍有非服務面證據提醒，必須列入 LOGBOOK。 |
+
+Wrapper exit code：
+
+- `0`：沒有 service blocker。可能仍有 DR boundary / evidence warning。
+- `1`：有 service warning，只能宣稱 degraded。
+- `2`：有 service blocker，不可宣稱恢復完成。

 ---

@@ -170,6 +177,17 @@ CPU / runaway：orphan=? active_ci=? load=?

 ## 6. 目前最新已驗證基線

+2026-06-25 14:41 wrapper live run：
+
+- Wrapper：`POST_START_QUICK_CHECK PASS=18 WARN=2 BLOCKED=0`。
+- Warning split：`SERVICE=0 BOUNDARY=1 EVIDENCE=1`。
+- Result：`FULL_STACK_GREEN_DR_ESCROW_BLOCKED`，exit code `0`。
+- Cold-start：`PASS=89 WARN=0 BLOCKED=0`，Result `GREEN`。
+- MOMO：`V10.676`，dedicated preflight `PASS=19 WARN=2 BLOCKED=0`，job `57` clean，`DB_DAILY_FRESHNESS 1|2026-06-24`。
+- Backup：110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`。
+- DR：`escrow_missing=5`，不可宣稱 DR complete。
+- CPU：110 顯示 active CI / build / test load；沒有 orphan Chrome 復發證據。
+
 2026-06-25 14:16：

 - Cold-start：`PASS=89 WARN=0 BLOCKED=0`，Result `GREEN`。
--- a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
+++ b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
@@ -11,15 +11,15 @@

 | Area | Status | Completion | Evidence |
 |------|--------|------------|----------|
-| Overall recovery readiness | FULL_STACK_GREEN_DR_ESCROW_BLOCKED | 99% | 2026-06-25 14:16 live cold-start returned `PASS=89 WARN=0 BLOCKED=0`, result `GREEN`: 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, AWOOOI API health is healthy/prod/mock=false, MOMO service health is healthy on `V10.674`, MOMO data freshness is `1|2026-06-24`, 110 / 188 runtime and backup checks are green。MOMO latest valid job `57` completed cleanly at `2026-06-25T13:18:02`, `15383/15383/0`, and current-month snapshot / realtime bounds match through `2026-06-24`. DR remains blocked because credential escrow evidence markers are still missing (`escrow_missing=5`) and must not be forged. |
+| Overall recovery readiness | FULL_STACK_GREEN_DR_ESCROW_BLOCKED | 99% | 2026-06-25 14:41 post-start quick check returned exit `0`, `POST_START_QUICK_CHECK PASS=18 WARN=2 BLOCKED=0`, warning split `SERVICE=0 BOUNDARY=1 EVIDENCE=1`, result `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`: 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, AWOOOI API health is healthy/prod/mock=false, delegated cold-start is `PASS=89 WARN=0 BLOCKED=0`, MOMO service health is healthy on `V10.676`, MOMO data freshness is `1|2026-06-24`, 110 / 188 runtime and backup checks are green。MOMO latest valid job `57` completed cleanly at `2026-06-25T13:18:02`, `15383/15383/0`, and current-month snapshot / realtime bounds match through `2026-06-24`. DR remains blocked because credential escrow evidence markers are still missing (`escrow_missing=5`) and must not be forged. |
 | P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-25 09:05 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, VIP `192.168.0.125` is present, node filesystem / disk-pressure / readonly events are `0`, and latest `km-vectorize-29705460-55rgs` completed. |
 | P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-25 09:05 backup / alert readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`, last aggregate `2026-06-25 02:35:09`。DR remains blocked on real non-secret credential escrow evidence IDs. |
-| P2 service / data truth | GREEN | 100% | Public route/TLS, API/Web route, MOMO health `V10.674`, MOMO main / CD `#904` monthly-sync failure boundary, MOMO main / CD `#910` Drive-auth fail-closed boundary, direct 14:16 public route smoke all expected 2xx/3xx, current-month parity `15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. 14:16 preflight confirms app / scheduler / Telegram bot healthy, scheduler restart count `0`, token metadata aligned to scheduler UID, latest job `57` completed cleanly, and `DB_DAILY_FRESHNESS 1|2026-06-24`. |
-| P3 docs / automation contracts | DONE_WITH_MOMO_PREFLIGHT_AND_CPU_TRIAGE | 100% | Workplan, SOP v1.52, one-page post-start quick check wrapper + fallback runbook, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO data freshness hard blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence decision gate with scheduler stats / import_config / job 56 evidence, repo-side cold-start v1.42 source absence classifier, live-sync parity gate, MOMO import-boundary production deploy, MOMO Drive-auth fail-closed production deploy, 10:04 scheduler fail-closed live proof, 10:35 route / DB / backup refresh, 11:44 MOMO dedicated preflight blocked readback, 14:16 MOMO dedicated preflight recovery on V10.674 / job 57 / freshness 1, 10:58 user-approved 110 orphan Chrome SIGTERM evidence, MacBook Pro Codex safe artifact sync readback, and 2026-06-25 live refresh with full cold-start GREEN are updated. 2026-06-24 23:15 read-only verify still shows repo cold-start hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`; live 110 script sync of the v1.42 classifier is not claimed until separately approved and recorded. |
+| P2 service / data truth | GREEN | 100% | Public route/TLS, API/Web route, MOMO health `V10.676`, MOMO main / CD `#904` monthly-sync failure boundary, MOMO main / CD `#910` Drive-auth fail-closed boundary, direct 14:41 wrapper public route smoke all expected 2xx/3xx, current-month parity `15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. 14:41 preflight confirms app / scheduler / Telegram bot healthy, scheduler restart count `0`, token metadata aligned to scheduler UID, latest job `57` completed cleanly, and `DB_DAILY_FRESHNESS 1|2026-06-24`. |
+| P3 docs / automation contracts | DONE_WITH_WRAPPER_LIVE_READBACK | 100% | Workplan, SOP v1.53, one-page post-start quick check wrapper + fallback runbook, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO data freshness hard blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence decision gate with scheduler stats / import_config / job 56 evidence, repo-side cold-start v1.42 source absence classifier, live-sync parity gate, MOMO import-boundary production deploy, MOMO Drive-auth fail-closed production deploy, 10:04 scheduler fail-closed live proof, 10:35 route / DB / backup refresh, 11:44 MOMO dedicated preflight blocked readback, 14:16 MOMO dedicated preflight recovery on V10.674 / job 57 / freshness 1, 14:41 wrapper warning split `SERVICE=0 BOUNDARY=1 EVIDENCE=1`, 10:58 user-approved 110 orphan Chrome SIGTERM evidence, MacBook Pro Codex safe artifact sync readback, and 2026-06-25 live refresh with full cold-start GREEN are updated. 2026-06-24 23:15 read-only verify still shows repo cold-start hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`; live 110 script sync of the v1.42 classifier is not claimed until separately approved and recorded. |

-2026-06-25 14:16 supplemental readback supersedes the 11:53 BLOCKED wording: direct route smoke is 200 for AWOOOI API / IwoooS / MOMO health / Stock, and cold-start public route/TLS gate is green for all expected 2xx/3xx routes. Repo-side cold-start returns `PASS=89 WARN=0 BLOCKED=0`; `/backup/scripts/backup-status.sh --no-notify --no-refresh` reports 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; MOMO dedicated preflight returns `PASS=18 WARN=3 BLOCKED=0`; MOMO health is `V10.674`; 110 load is around `3.85 / 3.33 / 3.19`, with active Gitea Actions / 2026 World Cup pipeline visible, not orphan Chrome.
+2026-06-25 14:41 supplemental wrapper readback supersedes the 14:16 wrapper wording: direct route smoke is 200 for AWOOOI API / IwoooS / MOMO health / Stock, and cold-start public route/TLS gate is green for all expected 2xx/3xx routes. Repo-side cold-start returns `PASS=89 WARN=0 BLOCKED=0`; `/backup/scripts/backup-status.sh --no-notify --no-refresh` reports 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; MOMO dedicated preflight returns `PASS=19 WARN=2 BLOCKED=0`; MOMO health is `V10.676`; 110 load is around `4.09 / 4.52 / 4.39`, with active Gitea Actions / build / test load visible, not orphan Chrome. Wrapper result is `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`, not `DEGRADED`, because service warnings are `0` and only DR boundary / evidence warnings remain.

-Full cold-start service readiness may now be declared GREEN for the latest verified evidence set. As of 2026-06-25 14:16, routes/hosts/K3s/backups/exporters/monitoring surfaces are available, AWOOOI API is healthy, MOMO service health is `V10.674`, and MOMO business data is fresh through `2026-06-24`. The live read-only cold-start scorecard is `PASS=89 WARN=0 BLOCKED=0`. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
+Full cold-start service readiness may now be declared GREEN for the latest verified evidence set. As of 2026-06-25 14:41, routes/hosts/K3s/backups/exporters/monitoring surfaces are available, AWOOOI API is healthy, MOMO service health is `V10.676`, and MOMO business data is fresh through `2026-06-24`. The live read-only cold-start scorecard is `PASS=89 WARN=0 BLOCKED=0`, and the post-start wrapper result is `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`. Do not declare DR scorecard complete while credential escrow evidence remains blocked.

 2026-06-13 01:26 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing survived the next normal CD deploy: Gitea main `e4a349bc`, ArgoCD revision `e4a349bc`, images from `414413a5`, API/Web split across `mon` / `mon1`, and global `known_hosts` retained 120 / 188 after CD fix `80e6ec1a`. Do not declare DR complete while credential escrow is missing. `km-vectorize` remediation is `90%`: schedule/label fix is live, and the remaining gate is the next official 03:00 CronJob success readback.

--- a/scripts/reboot-recovery/post-start-quick-check.sh
+++ b/scripts/reboot-recovery/post-start-quick-check.sh
@@ -17,6 +17,9 @@ NO_COLOR_FLAG=0
 PASS_COUNT=0
 WARN_COUNT=0
 BLOCKED_COUNT=0
+SERVICE_WARN_COUNT=0
+BOUNDARY_WARN_COUNT=0
+EVIDENCE_WARN_COUNT=0

 HOSTS=(
  "192.168.0.110"
@@ -48,9 +51,9 @@ Options:
  -h, --help          Show this help.

 Exit codes:
-  0 = no blockers.
-  1 = warnings only.
-  2 = blockers observed.
+  0 = no service blockers. Boundary / evidence warnings may still be present.
+  1 = service warnings only.
+  2 = service blockers observed.

 This script never reads token content and never writes runtime state.
 USAGE
@@ -117,6 +120,21 @@ warn() {
  printf '%sWARN%s %s\n' "$YELLOW" "$NC" "$*"
 }

+service_warn() {
+  SERVICE_WARN_COUNT=$((SERVICE_WARN_COUNT + 1))
+  warn "$@"
+}
+
+boundary_warn() {
+  BOUNDARY_WARN_COUNT=$((BOUNDARY_WARN_COUNT + 1))
+  warn "$@"
+}
+
+evidence_warn() {
+  EVIDENCE_WARN_COUNT=$((EVIDENCE_WARN_COUNT + 1))
+  warn "$@"
+}
+
 blocked() {
  BLOCKED_COUNT=$((BLOCKED_COUNT + 1))
  printf '%sBLOCKED%s %s\n' "$RED" "$NC" "$*"
@@ -173,7 +191,7 @@ if [[ "$RUN_COLD_START" -eq 1 ]]; then
  if [[ -n "$cold_summary" ]]; then
    ok "cold-start summary: $cold_summary"
  else
-    warn "cold-start summary not found"
+    service_warn "cold-start summary not found"
  fi
  rm -f "$cold_tmp"
 fi
@@ -184,12 +202,17 @@ if [[ "$RUN_MOMO" -eq 1 ]]; then
  bash "$ROOT_DIR/scripts/reboot-recovery/momo-drive-token-source-recovery-preflight.sh" >"$momo_tmp" 2>&1
  momo_rc=$?
  cat "$momo_tmp"
+  momo_summary="$(grep -E 'MOMO_DRIVE_TOKEN_SOURCE_PREFLIGHT PASS=[0-9]+ WARN=[0-9]+ BLOCKED=[0-9]+' "$momo_tmp" | tail -n 1 || true)"
  case "$momo_rc" in
    0)
      ok "MOMO preflight clean"
      ;;
    1)
-      warn "MOMO preflight has warnings"
+      if [[ "$momo_summary" =~ BLOCKED=0 ]]; then
+        evidence_warn "MOMO preflight has non-service warnings"
+      else
+        service_warn "MOMO preflight has warnings and no clean summary"
+      fi
      ;;
    *)
      blocked "MOMO preflight has blockers"
@@ -210,15 +233,17 @@ if [[ "$RUN_BACKUP" -eq 1 ]]; then
  cat "$backup_tmp"
  if grep -Eq 'core_blockers=0|CORE_BLOCKERS[ =]0' "$backup_tmp"; then
    ok "backup core blockers are 0"
+  elif grep -Eq 'core_blockers=[1-9]|CORE_BLOCKERS[ =][1-9]' "$backup_tmp"; then
+    blocked "backup core blockers are non-zero"
  else
-    warn "backup core blocker summary not confirmed"
+    service_warn "backup core blocker summary not confirmed"
  fi
  if grep -Eq 'escrow_missing=0|ESCROW_MISSING_COUNT[ =]0' "$backup_tmp"; then
    ok "credential escrow missing is 0"
  elif grep -Eq 'escrow_missing=[1-9]|ESCROW_MISSING_COUNT[ =][1-9]' "$backup_tmp"; then
-    warn "credential escrow still missing; DR_COMPLETE is forbidden"
+    boundary_warn "credential escrow still missing; DR_COMPLETE is forbidden"
  else
-    warn "credential escrow count not found"
+    evidence_warn "credential escrow count not found"
  fi
  rm -f "$backup_tmp"
 fi
@@ -244,11 +269,11 @@ if [[ "$RUN_CPU" -eq 1 ]]; then
  if ssh_read "wooo@192.168.0.110" 'uptime; vmstat 1 5; ps -eo pid,ppid,pgid,stat,pcpu,pmem,comm,args --sort=-pcpu | head -25' >"$cpu_tmp" 2>&1; then
    ok "110 CPU/process readback succeeded"
  else
-    warn "110 CPU/process readback failed"
+    evidence_warn "110 CPU/process readback failed"
  fi
  cat "$cpu_tmp"
  if grep -Eiq 'chrome|chromium|playwright' "$cpu_tmp"; then
-    warn "browser/smoke process is visible; classify orphan vs active parent before action"
+    evidence_warn "browser/smoke process is visible; classify orphan vs active parent before action"
  fi
  if grep -Eiq 'gitea|actions|runner|npm|pnpm|pytest|pip-audit' "$cpu_tmp"; then
    ok "active CI/build/test load is visible"
@@ -258,16 +283,27 @@ fi

 section "總結"
 printf 'POST_START_QUICK_CHECK PASS=%s WARN=%s BLOCKED=%s\n' "$PASS_COUNT" "$WARN_COUNT" "$BLOCKED_COUNT"
+printf 'POST_START_QUICK_CHECK_WARNINGS SERVICE=%s BOUNDARY=%s EVIDENCE=%s\n' "$SERVICE_WARN_COUNT" "$BOUNDARY_WARN_COUNT" "$EVIDENCE_WARN_COUNT"

 if [[ "$BLOCKED_COUNT" -gt 0 ]]; then
  printf 'RESULT=BLOCKED\n'
  exit 2
 fi

-if [[ "$WARN_COUNT" -gt 0 ]]; then
+if [[ "$SERVICE_WARN_COUNT" -gt 0 ]]; then
  printf 'RESULT=DEGRADED\n'
  exit 1
 fi

+if [[ "$BOUNDARY_WARN_COUNT" -gt 0 ]]; then
+  printf 'RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED\n'
+  exit 0
+fi
+
+if [[ "$EVIDENCE_WARN_COUNT" -gt 0 ]]; then
+  printf 'RESULT=GREEN_WITH_EVIDENCE_WARNINGS\n'
+  exit 0
+fi
+
 printf 'RESULT=GREEN\n'
 exit 0