wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 3 Packages Projects Releases Wiki Activity

docs(ops): record post-commit recovery readback [skip ci]

Browse Source
This commit is contained in:
Your Name
2026-06-24 11:37:55 +08:00
parent 7db7800e39
commit dff3658947
3 changed files with 32 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
docs/runbooks/FULL-STACK-COLD-START-SOP.md
View File

@@ -10,13 +10,13 @@
本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。
2026-06-24 11:19 live readback supersedes the earlier 06:35 wording:
2026-06-24 11:35 live readback supersedes the earlier 11:19 wording:
```text
Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%.
Live cold-start read-only check: PASS=86 WARN=0 BLOCKED=1, Result=BLOCKED.
Service state: SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, ArgoCD awoooi-prod Synced/Healthy at revision 35a3a59839bb09404099f6b5a22be1354a247abe, public routes/TLS green, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared.
Runtime release state: API/Web/Worker are ready; image remains a84a5a0b because 35a3a598 is docs/ops-script only and does not rebuild runtime images.
Service state: SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, ArgoCD awoooi-prod Synced/Healthy at revision 7db7800e399caed5487a705c81ec993dec76c70f, public routes/TLS green, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared.
Runtime release state: API/Web/Worker are ready; image remains a84a5a0b because 7db7800e is docs-only and does not rebuild runtime images.
MOMO state: mo.wooo.work health is healthy on version V10.639; current-month daily_sales_snapshot and realtime_sales_monthly match, but both stop at 2026-06-17. MOMO_DAILY_FRESHNESS is 7 days, which is a hard blocker because business data is not current.
Google Drive state: momo scheduler token ownership is fixed for Docker userns, container-side Drive listing works, but folder 當日業績匯入 currently has no matching 即時業績_當日 Excel source file. Archive latest matching file is 2026-06-18T01:30:39Z and was already imported by job 56.
Backup / monitoring state: backup-status core blockers are 0, last aggregate is 2026-06-24 02:28:39, 188 MinIO is healthy, Velero BackupStorageLocation default is Available, one-off backup reboot-recovery-202606240456 completed, backup-health textfile reports Velero freshness green, and VeleroBackupNotRun / PostgreSQLDown / RedisDown / disk-pressure alerts resolved.
@@ -84,7 +84,7 @@ Allowed declaration: monitoring, alert rules, AI event packet, PlayBook / KM con
Forbidden declaration: AI runtime remediation is enabled. Process termination, Docker/systemd restart, Nginx reload, firewall/K8s action, Telegram live send, Gateway queue write, Bot API call, production write, and secret read remain forbidden without owner approval, maintenance window, evidence ref, dry-run, and post-check.
```
| 項目 | 2026-06-24 11:19 Asia/Taipei live result | 判定 |
| 項目 | 2026-06-24 11:35 Asia/Taipei live result | 判定 |
|------|-------------------------------------------|------|
| Overall recovery readiness | `98%` | `SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED` |
| P0 host / K3s recovery | `100%` | `DONE` |
@@ -101,10 +101,10 @@ Forbidden declaration: AI runtime remediation is enabled. Process termination, D
| Backup status | 11:20 status: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`; escrow readback still shows `ESCROW_MISSING_COUNT=5` | `GREEN_WITH_DR_ESCROW_WARNING` |
| Offsite sync / verify | 01:28 textfile: `awoooi_backup_offsite_remote_verify_ok=1`, `full_verify_fresh=1`, all 13 repos have `snapshot_count=1` and `snapshot_latest_only=1`; latest scheduled verifier log is 2026-06-12 07:20 | `GREEN` |
| Backup / cold-start alerts | 01:27 live visibility check confirms Prometheus and Alertmanager expose the 5 required credential escrow gap alerts; Prometheus rules API has all five required alert names healthy; label contract check loads 24 baseline backup alert rules | `GREEN_WITH_EXPECTED_REDLIGHTS` |
| Cold-start scorecard | 11:19 read-only scorecard：`PASS=86 WARN=0 BLOCKED=1`。Public routes / TLS、momo DB parity、backup exporters、120/121 K3s、MinIO / Velero、AWOOOI API/Web 皆通過；only blocker is MOMO data freshness. | `BLOCKED_MOMO_DATA_FRESHNESS` |
| Cold-start scorecard | 11:35 read-only scorecard：`PASS=86 WARN=0 BLOCKED=1`。Public routes / TLS、momo DB parity、backup exporters、120/121 K3s、MinIO / Velero、AWOOOI API/Web 皆通過；only blocker is MOMO data freshness. | `BLOCKED_MOMO_DATA_FRESHNESS` |
| momo DB parity | `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17` | `GREEN` |
| momo scheduler | container healthy; Drive listing from container works; pending folder `當日業績匯入` count is `0` for `即時業績_當日`; no current `Permission denied` evidence in the latest readback | `GREEN_WITH_SOURCE_ABSENT` |
| ArgoCD app health | 11:19 readback: `awoooi-prod` sync `Synced`, health `Healthy`, source revision `35a3a59839bb09404099f6b5a22be1354a247abe`; API/Web/Worker ready. | `GREEN` |
| ArgoCD app health | 11:35 readback: `awoooi-prod` sync `Synced`, health `Healthy`, source revision `7db7800e399caed5487a705c81ec993dec76c70f`; API/Web/Worker ready. | `GREEN` |
| Workload balancing | Live API/Web/Worker/CronJob image is `e999c16b3435f197b78fe2adfeec1c4faa6c4675`; API/Web pods remain split across `mon` / `mon1`, Worker single replica remains healthy on `mon` | `GREEN` |
| Credential escrow | 5 non-secret evidence markers missing | `BLOCKED` |
@@ -202,7 +202,7 @@ DR_COMPLETE = no, because credential escrow evidence is incomplete
110 / 120 / 121 / 188 HOST_READY = yes
Core public services SERVICE_READY = yes
MOMO_DB_PARITY = yes
MOMO_DATA_FRESH = no, because latest daily_sales_snapshot date is 2026-06-17 and stale age is 7 days as of 2026-06-24 11:19
MOMO_DATA_FRESH = no, because latest daily_sales_snapshot date is 2026-06-17 and stale age is 7 days as of 2026-06-24 11:35
FULL_STACK_GREEN = no, because cold-start scorecard is PASS=86 WARN=0 BLOCKED=1
DR_COMPLETE = no, because credential escrow evidence is incomplete
```
@@ -1726,7 +1726,7 @@ ssh ollama@192.168.0.188 'bash -s' < scripts/ops/188-node-exporter-restore.sh
| Drive pending folder | `當日業績匯入`，pattern `即時業績_當日`，目前 matching Excel count `0` |
| Drive archive folder | `當日業績匯入/已匯入`，最新 matching file modifiedTime `2026-06-18T01:30:39Z`，已由 import job `56` 匯入 |
| DB parity | `MOMO_MONTHLY_SYNC 10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17` |
| Data freshness | `MOMO_DAILY_FRESHNESS 7|2026-06-17` as of 2026-06-24 11:19 |
| Data freshness | `MOMO_DAILY_FRESHNESS 7|2026-06-17` as of 2026-06-24 11:35 |
| Live cold-start readback | `PASS=86 WARN=0 BLOCKED=1`, result `BLOCKED` |
| 110 live script sync | `/home/wooo/scripts/full-stack-cold-start-check.sh` hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8` |
| Alert dedupe | `data_stale_alert` for `upstream_drive` has 24h dedupe; latest evidence was 2026-06-23 with last_date `2026-06-17` |
@@ -1843,9 +1843,9 @@ Bitan public content: pass -> no failure Telegram; repeated same failure -> cool
### 14.31 2026-06-24 MOMO source-file absence decision gate
2026-06-24 11:19 的恢復判定把 MOMO 分成兩件事：服務可用與資料新鮮。服務可用已恢復，資料新鮮仍 blocked。這個 gate 的目的，是防止 operator 在外部網站 200、container healthy、DB parity 正常時，誤把「沒有新來源檔」當成「恢復完成」。
2026-06-24 11:35 的恢復判定把 MOMO 分成兩件事：服務可用與資料新鮮。服務可用已恢復，資料新鮮仍 blocked。這個 gate 的目的，是防止 operator 在外部網站 200、container healthy、DB parity 正常時，誤把「沒有新來源檔」當成「恢復完成」。
| 項目 | 11:19 source-file absence baseline |
| 項目 | 11:35 source-file absence baseline |
|------|------------------------------------|
| SOP version | `v1.32` |
| MOMO public health | `https://mo.wooo.work/health` returns healthy; version `V10.639` |