wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

docs(ops): record heartbeat noise and cold-start detector closure [skip ci]

Browse Source
This commit is contained in:
Your Name
2026-06-24 02:19:30 +08:00
parent 4a7b532962
commit 8aeeadbde1
4 changed files with 90 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
docs/LOGBOOK.md
View File

@@ -1,3 +1,27 @@
## 2026-06-24｜Telegram 正常心跳降噪與 cold-start MOMO detector 修正
**背景**：Telegram 群組每 30 分鐘收到 `AWOOOI 心跳 / 告警鏈路: ✅ 正常`，造成正常訊號洗版；同時 2026-06-24 01:45 live cold-start 雖已 `BLOCKED=0`，但仍因 MOMO scheduler log pattern 與 DB 讀法過舊產生兩個 WARN。這兩者都會讓重啟 SOP 出現 false-noise / false-warning，必須修掉。
**完成內容**：
- `a84a5a0b fix(api): suppress healthy Telegram heartbeat noise` 已推送，deploy marker `4a7b5329 chore(cd): deploy a84a5a0 [skip ci]` 已回寫到 `gitea/main`。
- Production API / Web / Worker image 均為 `a84a5a0bc4a672ac6feb95a85ac590aa2dd4bb71`，readiness 為 API `2/2`、Web `2/2`、Worker `1/1`。
- 正常且無 warning 的 heartbeat 改為只更新 Redis suppression marker / log，不再送 Telegram；warning 仍會送，warning 恢復為 healthy 時仍保留一次 recovery 訊息。
- 臨時 mitigation 已在 production Redis 設定 `heartbeat:silent_last_sent` 與 `heartbeat:healthy_suppressed_last_seen` 24 小時 TTL，避免下一輪 healthy heartbeat 在 CD 完成前繼續洗版；未讀取或輸出任何 token。
- live `/home/wooo/scripts/full-stack-cold-start-check.sh` 已同步 MOMO detector 修正，備份為 `/home/wooo/scripts/full-stack-cold-start-check.sh.before-momo-detector-20260624-020759`，新 hash `47e67d0c018f741acfba17a93cb1d668779bd08745902099a10ee61e73ea55b6`。
- `full-stack-cold-start-check.sh` repo 版同步補強 K3s node Ready / storage condition 判斷，以及 MOMO scheduler / current-month DB parity 讀法；DB 密碼只透過 `PGPASSWORD` 傳入 `docker exec`，不輸出 secret。
**Live 驗證**：
- 2026-06-24 02:08 cold-start：`PASS=85 WARN=0 BLOCKED=0`，result `GREEN`。
- MOMO evidence：`SCHEDULER_CONTAINER_RUNNING true`、`SCHEDULER_CONTAINER_HEALTH healthy`、`SCHEDULER_RECENT_ACTIVITY 1303`、`MOMO_MONTHLY_SYNC 10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`。
- K3s evidence：`mon` / `mon1` Ready，VIP `192.168.0.125` present，`NODE_FS_ERROR_EVENTS 0`，`NODE_READONLY_FILESYSTEM_TRUE 0`，`NODE_DISK_PRESSURE_TRUE 0`。
- K8s jobs：`FAILED_JOBS 1`、`STALE_FAILED_JOBS 1`、`ACTIVE_FAILED_JOBS 0`，保留歷史失敗 Job 作 evidence，不當作 active blocker。
- Production health：`/api/v1/health` 回 `healthy / prod / mock_mode=false`；`postgresql`、`redis`、`ollama`、`openclaw`、`signoz`、`ollama_gcp_a`、`ollama_gcp_b` 均 up；`ollama_local` 仍為下游 provider 狀態，不影響 API overall healthy。
**注意事項**：
- Gitea CD `#3289` 最後標示 Failure，根因是 worker startup probe / rollout status timeout 的 false-negative；K8s 實際已 rollout 完成並有 deploy marker。SOP 需後續調整 worker startup window / CD post-deploy timeout，不能把這類假紅燈當生產不可用，也不能忽略它。
- 這次只處理正常心跳降噪與 cold-start detector false-warning，不代表 Telegram warning / recovery 告警被消音。
- DR 仍不可宣稱完成：credential escrow evidence missing 仍需維持 `5`，不可偽造或把任何 secret value 放入 repo / 聊天。
## 2026-06-19｜治理頁公開流程詞與 enum sanitizer 正式驗證完成
**背景**：前一輪治理頁公開顯示清理已讓主要卡片可見，但 production desktop smoke 仍發現 `live worker`、`Direct Bot API`、`dual approval`、`owner approval` 等深層 snapshot 自由文字殘留，並因 enum 值被 sanitizer 先翻成中文而產生 `MISSING_MESSAGE`。本輪收斂 messages 與 governance tab sanitizer，完成正式部署後 desktop / mobile readback。

24
docs/runbooks/FULL-STACK-COLD-START-SOP.md
View File

@@ -1628,6 +1628,30 @@ SOP update:
| Declaration limit | 可宣稱 `FULL_STACK_GREEN_FOR_SERVICE`；不可宣稱 `DR_COMPLETE`、`credential escrow complete` 或任何 runtime/security acceptance |
| SOP change | v1.25 defines retained failed Job evidence vs active failed Job blocker; future reboot comparison must record all three counters |
### 14.26 2026-06-24 heartbeat noise / MOMO detector / rollout false-negative closure
2026-06-24 的變更不是主機重啟，而是把重啟 SOP 的兩種 false signal 收斂：Telegram 正常心跳不再每 30 分鐘洗版；MOMO scheduler / current-month parity detector 不再因舊 log pattern 或錯誤 DB exec 使用者誤報 WARN。這個錨點也記錄 CD rollout false-negative：worker startup probe 第一次超時重啟一次，K8s 最終 ready，但 Gitea CD `#3289` 因 rollout status timeout 標 Failure。
| 項目 | 2026-06-24 live baseline |
|------|--------------------------|
| SOP version | `v1.27` |
| Heartbeat code | `a84a5a0b fix(api): suppress healthy Telegram heartbeat noise` |
| Deploy marker | `4a7b5329 chore(cd): deploy a84a5a0 [skip ci]` |
| Production image readback | API/Web/Worker image tag `a84a5a0bc4a672ac6feb95a85ac590aa2dd4bb71` |
| Production rollout | API `2/2`、Web `2/2`、Worker `1/1` Ready |
| CD result caveat | Gitea CD `#3289` shows Failure because worker rollout status timed out before old replica convergence; K8s deploy marker and production readiness are green |
| Healthy heartbeat rule | `status=healthy` 且無 warnings 時只更新 suppression marker / log，不送 Telegram；warnings 與 recovery 仍可送 |
| Live temporary suppression | Redis keys `heartbeat:silent_last_sent` and `heartbeat:healthy_suppressed_last_seen` set with 24h TTL during deployment; no token or secret printed |
| 110 live script sync | `/home/wooo/scripts/full-stack-cold-start-check.sh` hash `47e67d0c018f741acfba17a93cb1d668779bd08745902099a10ee61e73ea55b6`; previous version backed up to `/home/wooo/scripts/full-stack-cold-start-check.sh.before-momo-detector-20260624-020759` |
| MOMO scheduler evidence | `SCHEDULER_CONTAINER_RUNNING true`、`SCHEDULER_CONTAINER_HEALTH healthy`、`SCHEDULER_RECENT_ACTIVITY 1303` |
| MOMO DB parity evidence | `MOMO_MONTHLY_SYNC 10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17` |
| K3s node evidence | `NODE_FS_ERROR_EVENTS 0`、`NODE_READONLY_FILESYSTEM_TRUE 0`、`NODE_DISK_PRESSURE_TRUE 0`、VIP `192.168.0.125` present |
| Live cold-start readback | `PASS=85 WARN=0 BLOCKED=0`, result `GREEN` |
| Declaration limit | 可宣稱 current service recovery scorecard green；不可宣稱 `DR_COMPLETE`，credential escrow evidence missing remains `5` |
| SOP change | v1.27 requires heartbeat success-message suppression, MOMO detector parity using app-provided DB env, and rollout false-negative classification before retrying CD |
Worker / CronJob / queue 類服務若啟動時間可能超過 startup probe，不能只看第一次 `rollout status --timeout=60s` 失敗就判定 production down。必須同時看 deploy marker、image tag、pod readiness、container restart count、service health、public route / API health。若 pod 最終 ready 但 CD 紅燈，這是 CI timeout / probe tuning 工作，不是服務重啟事故；後續應調整 startup probe 或 post-deploy timeout。
### 14.22 重啟後時間軸驗證
每次重啟後照時間軸推進，不要等到最後才一次判定。

8
docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
View File

@@ -11,13 +11,13 @@
| Area | Status | Completion | Evidence |
|------|--------|------------|----------|
| Overall recovery readiness | SERVICE_GREEN_DR_ESCROW_BLOCKED | 99% | 2026-06-18 13:43 live cold-start read-only gate returned `PASS=84 WARN=0 BLOCKED=0`, result `GREEN`。110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, `NODE_FS_ERROR_EVENTS=0`, public routes/TLS are green, 110 backup health is `13/13 fresh failed=0`, 188 backup health is `2/2 fresh failed=0`。K8s schedule readback now distinguishes `FAILED_JOBS=1` / `STALE_FAILED_JOBS=1` / `ACTIVE_FAILED_JOBS=0`; the retained `km-vectorize-29689620` failed Job is historical evidence, not an active service blocker. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. |
| Overall recovery readiness | SERVICE_GREEN_DR_ESCROW_BLOCKED | 99% | 2026-06-24 02:08 live cold-start read-only gate returned `PASS=85 WARN=0 BLOCKED=0`, result `GREEN`。110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, `NODE_FS_ERROR_EVENTS=0`, `NODE_READONLY_FILESYSTEM_TRUE=0`, `NODE_DISK_PRESSURE_TRUE=0`, public routes/TLS are green, 110 / 188 runtime checks are green。K8s schedule readback distinguishes `FAILED_JOBS=1` / `STALE_FAILED_JOBS=1` / `ACTIVE_FAILED_JOBS=0`; the retained failed Job is historical evidence, not an active service blocker. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. |
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-14 18:15 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 92% | 2026-06-15 03:11 `backup-status` shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-15 02:40:13`. Offsite / escrow report shows `SCRIPT_MISSING_COUNT=0`, `OFFSITE_CONFIGURED=1`, `RCLONE_CONFIGURED=1`, `ESCROW_MISSING_COUNT=5`. Owner request package is ready; actual marker write remains blocked on real non-secret evidence IDs. |
| P2 service / data truth | VERIFIED_FULL_STACK_GREEN_FOR_SERVICE | 100% | 2026-06-18 13:43 cold-start verifies public route/TLS, API/Web route, momo health and current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness, VIP, and 110 / 188 runtime health. K8s active failed Job count is `0`, bad pods are `0`, and cold-start returns `PASS=84 WARN=0 BLOCKED=0`. |
| P3 docs / automation contracts | DONE_WITH_RUNAWAY_PROCESS_AIOPS_LIVE_SCRAPED | 100% | Workplan, SOP v1.26, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, and 2026-06-18 live readback are updated. 14:31-14:32 Prometheus scrape confirms 110 `monitor_up=1`, orphan browser group count `0`, active CI containers `2`, load5/core around `0.79-0.81`, swap ratio around `1.0`, `remediation_authorized=0`, and missing/orphan alerts not firing. Repo-side readiness audit also checks runaway process exporter / remediation helper / alert group; live cold-start remains `PASS=84 WARN=0 BLOCKED=0` from the latest service readiness readback. |
| P2 service / data truth | VERIFIED_FULL_STACK_GREEN_FOR_SERVICE | 100% | 2026-06-24 02:08 cold-start verifies public route/TLS, API/Web route, momo health and current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health. K8s active failed Job count is `0`, bad pods are `0`, and cold-start returns `PASS=85 WARN=0 BLOCKED=0`. |
| P3 docs / automation contracts | DONE_WITH_HEARTBEAT_NOISE_AND_MOMO_DETECTOR_CLOSURE | 100% | Workplan, SOP v1.27, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, and 2026-06-24 live readback are updated. Production image `a84a5a0b` is live with API `2/2`, Web `2/2`, Worker `1/1`; CD `#3289` is a known false-negative caused by worker startup / rollout timeout after deploy marker `4a7b5329`. |
Full cold-start service readiness may be declared green for the latest verified evidence set. As of 2026-06-18 13:43, services are green with `WARN=0` and `BLOCKED=0`; the retained stale `km-vectorize` failed Job remains historical evidence only. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
Full cold-start service readiness may be declared green for the latest verified evidence set. As of 2026-06-24 02:08, services are green with `PASS=85 WARN=0 BLOCKED=0`; the retained stale failed Job remains historical evidence only. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
2026-06-13 01:26 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing survived the next normal CD deploy: Gitea main `e4a349bc`, ArgoCD revision `e4a349bc`, images from `414413a5`, API/Web split across `mon` / `mon1`, and global `known_hosts` retained 120 / 188 after CD fix `80e6ec1a`. Do not declare DR complete while credential escrow is missing. `km-vectorize` remediation is `90%`: schedule/label fix is live, and the remaining gate is the next official 03:00 CronJob success readback.