fix(ops): harden 188 startup data recovery gate
This commit is contained in:
ogt
@@ -1,3 +1,24 @@
|
||||
## 2026-06-26｜188 host hygiene readback：服務綠燈但 systemd degraded 不可假綠
|
||||
|
||||
**背景**：全主機 readback 顯示 188 產品服務健康，但 `systemctl is-system-running=degraded`。本段只做 read-only triage 與 repo-side SOP / startup safety hardening；沒有在 188 上 reset-failed、restart、renew cert、reload Nginx、修改 DB 或讀 secret。
|
||||
|
||||
**Read-only evidence**：
|
||||
- 188 failed units：`awoooi-startup.service`、`postgresql@14-main.service`、`certbot.service`、`snap.certbot.renew.service`。
|
||||
- 產品面仍綠：Docker active；MOMO / scheduler / bot healthy；VibeWork web / PostgreSQL healthy；MinIO / node-exporter / postgres-exporter / redis-exporter / nginx-exporter up；SignOz HTTP `200`；`pg_exporter_last_scrape_error=0`、`pg_up=1`、`redis_up=1`。
|
||||
- Host PostgreSQL 分層：`postgresql.service` 是 `active (exited)`，但 `pg_lsclusters` 顯示 host cluster `14/main` status `down`；`pg_isready` 與 exporter OK 不能直接替代 host cluster unit health，因為 188 也有 containerized PostgreSQL / product DB。
|
||||
- `awoooi-startup.service` 於 2026-06-23 14:50 嘗試啟動 host PostgreSQL，journal 顯示 `could not locate a valid checkpoint record`，startup 腳本隨後嘗試以 root 執行 `pg_resetwal` 但失敗。這證明原 startup 自動資料層修復邏輯不安全，不能納入專業重啟綠燈。
|
||||
- Certbot：`sentry.wooo.work` renewal 多次 `Some challenges have failed`，2026-06-26 00:05 又遇到 ACME `rateLimited / Service busy; retry later`；public certificate for `sentry.wooo.work` / shared routes still valid until `2026-07-09 16:03:40 UTC`，因此目前不是立即 route outage，但已是 P1 TLS renewal blocker。
|
||||
|
||||
**Repo-side hardening**：
|
||||
- `scripts/reboot-recovery/awoooi-startup.sh` 已改為 fail-closed：遇到 PostgreSQL checkpoint/WAL 類錯誤時，只記錄需要 DB owner、backup / restore evidence、maintenance window 與 post-check；不再自動執行 `pg_resetwal`。
|
||||
- `FULL-STACK-COLD-START-SOP.md` 升到 v1.63，將 188 狀態拆成 product service green 與 host hygiene debt。
|
||||
- `REBOOT-RECOVERY-SOP.md` 將 PostgreSQL WAL 修復從自動/一般修復降級為 break-glass 手動程序，禁止當作快速重啟 SOP 的預設步驟。
|
||||
|
||||
**仍 blocked / 不得宣稱**：
|
||||
- 188 `HOST_PERFECT_GREEN` 不可宣稱；目前是 `SERVICE_GREEN_HOST_HYGIENE_BLOCKED`。
|
||||
- 不得用 `pg_isready OK`、Docker DB healthy 或 public route `200` 取代 host `postgresql@14-main` 狀態。
|
||||
- 不得自動 `pg_resetwal`、`certbot renew`、`systemctl reset-failed`、Nginx reload 或 DB restore；需要 owner-approved maintenance window、rollback owner、備份 evidence 與 post-check。
|
||||
|
||||
## 2026-06-26｜AI Agents 專業執行跑道正式讀回：全域控管可見化，但仍不誇大自動接管
|
||||
|
||||
**背景**：統帥要求 AI Agents 不只列在治理卡片，而要能「遍地開花」控管主機、產品、網站、套件、服務、工具、AI Agent 與 AI 解決方案，並且讓使用者能看見 OpenClaw、Hermes、Nemotron 等角色的專業分工、互動、學習與成長證據。同時前端禁止顯示內部對話內容、raw path、secret 或 Telegram token。
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# AWOOOI 全棧冷啟動與主機重啟 SOP
|
||||
|
||||
> Version: v1.62
|
||||
> Version: v1.63
|
||||
> Last updated: 2026-06-26 Asia/Taipei
|
||||
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
|
||||
|
||||
@@ -12,6 +12,8 @@
|
||||
|
||||
若只是重啟後要快速判斷能不能宣稱恢復，先跑一頁式總檢查：`scripts/reboot-recovery/post-start-quick-check.sh --no-color`，並以 `docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md` 作為人工 fallback。長 SOP 保留完整背景、例外處理與 Plan B；短版 wrapper / checklist 負責每次 T+10 分鐘內的固定判定。
|
||||
|
||||
2026-06-26 06:50-06:55 188 host hygiene read-only triage：188 product services remain green, but host `systemctl` is still `degraded` and must not be smoothed into full host green. Failed units are `awoooi-startup.service`, `postgresql@14-main.service`, `certbot.service`, and `snap.certbot.renew.service`. Evidence shows the host PostgreSQL cluster `14/main` is down in `pg_lsclusters`, while product DB / exporters still respond through containerized services; therefore `pg_isready` or `pg_up=1` cannot substitute for host cluster health. The 188 startup service detected `could not locate a valid checkpoint record` on 2026-06-23 and attempted `pg_resetwal` as root, which failed; v1.63 treats PostgreSQL checkpoint/WAL errors as break-glass only and the repo-side startup script now fails closed instead of running `pg_resetwal`. Certbot renew for `sentry.wooo.work` is also failing and hit ACME rate-limit / challenge failure, but the public cert is still valid until 2026-07-09 16:03:40 UTC. Current declaration: `SERVICE_GREEN_HOST_HYGIENE_BLOCKED` for 188, while overall service recovery remains `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`.
|
||||
|
||||
2026-06-26 06:40-06:44 全主機 read-only refresh：`110 / 120 / 121 / 188 / 112 / 111 / 168` ping 與 SSH port 全部 OK。核心 reboot scope 維持 green：110 `systemctl=running`、failed units `0`，Docker / Gitea / Harbor / Prometheus / Alertmanager 可用；120 / 121 `systemctl=running`、failed units `0`，K3s nodes `mon` / `mon1` Ready；188 產品容器與 PostgreSQL / Redis / MOMO / SignOz 可用。ArgoCD `awoooi-prod` 已從先前 degraded 收斂為 `Synced / Healthy`，revision `b2945ab9f716d9d685434ae0e67b9318414b27fe`；`km-vectorize` official 03:00 台北時間 run 成功，`lastSuccess=2026-06-25T19:00:14Z`。Public routes for AWOOOI / VibeWork / AwoooGo / MOMO / Stock / Bitan / Gitea / Harbor / Registry / Sentry / SigNoz / Langfuse return expected statuses; AWOOOI API health is `healthy / prod / mock_mode=false`; MOMO health is `V10.690`; StockPlatform freshness is `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; backup-status remains core green with `escrow_missing=5`. Boundaries: 188 host still has failed units `awoooi-startup.service`, `certbot.service`, `postgresql@14-main.service`, `snap.certbot.renew.service` that require host hygiene cleanup; 112 Wazuh services / ports are active but Wazuh manager registry accepted remains `0`; 111 / 168 Codex workspaces are reachable but have different local HEADs on the same ahead branch; Mac Mini free space is about `3.4Gi`. Current service verdict remains `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`, not `DR_COMPLETE` or `Wazuh recovered`.
|
||||
|
||||
2026-06-26 06:26-06:28 隔日 read-only refresh：四主機 ping/SSH OK，cold-start `PASS=89 WARN=0 BLOCKED=0`，MOMO `V10.690` 且 latest import job `57 completed`，StockPlatform `/api/v1/system/freshness` 仍為 `status=ok` / `latest_trading_date=2026-06-25` / blockers `[]`，backup-status 110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、`offsite_fresh=1`、`rclone_gdrive_fresh=1`、`last_backup_all=2026-06-26 02:31:02`、`escrow_missing=5`。06:26 full wrapper 首輪在 `https://awoooi.wooo.work/zh-TW/iwooos` 與 `https://vibework.wooo.work/` 出現單次 `000`，但獨立 curl 立即回 `200`，route-only wrapper 也回 `PASS=31 WARN=0 BLOCKED=0 RESULT=GREEN`；因此 v1.61 將 public route gate 改為最多 3 次 retry，只有連續失敗才算 `BLOCKED`，retry 後恢復則列為 evidence warning。06:28 core wrapper with routes skipped returns `POST_START_QUICK_CHECK PASS=15 WARN=2 BLOCKED=0`, `RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。本次沒有 Docker/systemd/Nginx/firewall/K8s/DB/Wazuh runtime 寫操作。
|
||||
@@ -134,7 +136,7 @@ Forbidden declaration: AI runtime remediation is enabled. Process termination, D
|
||||
| 110 host runaway process guard | 14:31-14:32 live scrape confirms `monitor_up=1`, orphan browser groups `0`, active Gitea Actions containers `2`, `load5_per_core≈0.79-0.81`, `swap_used_ratio≈1.0`, and `remediation_authorized=0`; exporter/helper also remain in Ansible textfile exporter source-of-truth. | `LIVE_SCRAPED_RUNTIME_GATE_0` |
|
||||
| 120 reachability | ping OK, SSH OK, boot around `2026-06-14 02:23`, K3s active, node `mon Ready` | `GREEN` |
|
||||
| 121 reachability | ping OK, SSH OK, failed units `0` | `GREEN` |
|
||||
| 188 host runtime | production services green; host degraded only by `certbot.service` and `snap.certbot.renew.service` | `GREEN_WITH_CERTBOT_DEBT` |
|
||||
| 188 host runtime | production services green, but host `systemctl` degraded by `awoooi-startup.service`, `postgresql@14-main.service`, `certbot.service`, and `snap.certbot.renew.service`; host PostgreSQL cluster `14/main` is down while product DB containers/exporters are healthy; certbot renewal for shared `sentry.wooo.work` certificate is failing but public cert is still valid until 2026-07-09 UTC | `SERVICE_GREEN_HOST_HYGIENE_BLOCKED` |
|
||||
| K3s node state | `mon Ready control-plane`, `mon1 Ready control-plane`; bad pods `0`; `FAILED_JOBS=1`, `STALE_FAILED_JOBS=1`, `ACTIVE_FAILED_JOBS=0` | `GREEN_WITH_RETAINED_EVIDENCE` |
|
||||
| 110 -> 120 / 188 SSH trust | 00:33 cold-start exposed stale `known_hosts`; backup `/home/wooo/.ssh/known_hosts.before-120-188-refresh.20260613-003416`; final repair backup `/home/wooo/.ssh/known_hosts.before-120-188-final-refresh.20260613-011949`; CD fix `80e6ec1a` moves deploy trust to `/home/wooo/.ssh/deploy_known_hosts`; 01:28 global `known_hosts` still contains 120 / 188 and was not clobbered by deploy marker `e4a349bc` | `GREEN_WITH_GUARDRAIL` |
|
||||
| Backup status | 11:20 status: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`; escrow readback still shows `ESCROW_MISSING_COUNT=5` | `GREEN_WITH_DR_ESCROW_WARNING` |
|
||||
@@ -591,11 +593,13 @@ ssh ollama@192.168.0.188 'hostname; date; who -b; uptime; systemctl is-system-ru
|
||||
|
||||
2026-06-12 110 事故教訓：failed unit 若指向不存在的 legacy 路徑，要先確認是否仍屬現役服務。停用 stale timer 可以解除 host degraded，但必須同步 source-of-truth 後續清理，不能靠反覆 `reset-failed` 掩蓋。
|
||||
|
||||
2026-06-26 188 事故教訓：PostgreSQL host cluster / Docker product DB / exporter 三者必須分開判定。`pg_isready`、`pg_up=1` 或 public route `200` 只能證明某個 PostgreSQL endpoint 可用，不能證明 `postgresql@14-main` 已恢復。若 journal 出現 `could not locate a valid checkpoint record`，不得由 startup 腳本或 AI 自動執行 `pg_resetwal`；必須進入 DB owner / backup restore / maintenance window / rollback owner / post-check gate。
|
||||
|
||||
---
|
||||
|
||||
## 4. P0 188 Data Layer
|
||||
|
||||
188 is the first real service dependency because K3s datastore and AWOOOI DB depend on PostgreSQL.
|
||||
188 is the first real service dependency because multiple product data planes, exporters, and AI / observability services depend on PostgreSQL-compatible endpoints. Do not assume the host cluster `postgresql@14-main`, Docker product databases, and exporter target are the same endpoint; prove the authoritative endpoint before repair.
|
||||
|
||||
### 4.1 Startup order
|
||||
|
||||
@@ -616,6 +620,8 @@ ssh ollama@192.168.0.188 'hostname; date; who -b; uptime; systemctl is-system-ru
|
||||
ssh ollama@192.168.0.188 '
|
||||
hostname; date; uptime; free -h
|
||||
systemctl is-active containerd docker postgresql@14-main redis-server ollama nginx || true
|
||||
pg_lsclusters 2>/dev/null || true
|
||||
ss -ltnp "sport = :5432" 2>/dev/null || ss -ltn "sport = :5432" || true
|
||||
pg_isready -h localhost -p 5432 || true
|
||||
redis-cli -p 6380 ping 2>/dev/null || redis-cli ping 2>/dev/null || true
|
||||
docker ps --format "{{.Names}}\t{{.Status}}\t{{.Ports}}" | head -120
|
||||
@@ -632,14 +638,16 @@ invalid primary checkpoint record
|
||||
unexpected pageaddr ... in log segment ...
|
||||
```
|
||||
|
||||
This blocks:
|
||||
This may block if the affected cluster is the authoritative runtime datastore:
|
||||
|
||||
- `188:5432`
|
||||
- K3s startup on 120/121
|
||||
- AWOOOI API DB access
|
||||
- Alertmanager webhook if API cannot start
|
||||
|
||||
Human-approved recovery command on 188:
|
||||
2026-06-26 counterexample: host cluster `14/main` can be down while product DB containers and exporters still serve traffic. Therefore `pg_isready` is not enough and failed `postgresql@14-main` is not automatically a product outage. First map the listening process / container, current app DB configuration, and backup freshness.
|
||||
|
||||
Break-glass example only after DB owner approval, backup evidence, maintenance window, rollback owner, and post-check plan:
|
||||
|
||||
```bash
|
||||
sudo systemctl stop postgresql@14-main
|
||||
@@ -651,7 +659,7 @@ pg_isready -h localhost -p 5432
|
||||
sudo -u postgres psql -d k3s_datastore -c "VACUUM ANALYZE kine;"
|
||||
```
|
||||
|
||||
Do not run `DROP`, reinitialize the cluster, delete `/var/lib/postgresql`, or restore an old backup unless the commander explicitly approves it.
|
||||
Do not run `pg_resetwal`, `DROP`, reinitialize the cluster, delete `/var/lib/postgresql`, or restore an old backup from AI/startup automation. These are break-glass actions only.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -157,7 +157,7 @@ cat /var/log/harbor-watchdog.log # 持久化 log
|
||||
**188 (7 步驟)**:
|
||||
- containerd BoltDB 損壞偵測與修復
|
||||
- Docker BoltDB 損壞偵測與修復
|
||||
- PostgreSQL WAL 損壞偵測 + pg_resetwal + kine VACUUM
|
||||
- PostgreSQL 啟動偵測；checkpoint / WAL 類資料層錯誤一律 fail-closed，禁止自動 `pg_resetwal`
|
||||
- Redis 啟動
|
||||
- Ollama 啟動
|
||||
- Nginx 啟動
|
||||
@@ -316,15 +316,25 @@ BOLT=/var/lib/docker/network/files/local-kv.db
|
||||
cp $BOLT ${BOLT}.bak.$(date +%s) && rm $BOLT && systemctl restart docker
|
||||
```
|
||||
|
||||
**PostgreSQL WAL 修復**:
|
||||
```bash
|
||||
# 確認是 WAL 損壞
|
||||
journalctl -u postgresql@14-main -n 30 | grep "could not locate a valid checkpoint"
|
||||
**PostgreSQL checkpoint / WAL 類錯誤處理**:
|
||||
|
||||
# 修復
|
||||
systemctl stop postgresql@14-main
|
||||
/usr/lib/postgresql/14/bin/pg_resetwal -f /var/lib/postgresql/14/main
|
||||
systemctl start postgresql@14-main
|
||||
> 2026-06-26 校正：這不是一般重啟 SOP 的自動修復步驟。`pg_resetwal` 可能造成不可逆資料風險，只能在 DB owner、備份/restore evidence、maintenance window、rollback owner 與 post-check plan 都具備時，作為 break-glass 人工程序。AI、startup 腳本或值班腳本不得自動執行。
|
||||
|
||||
```bash
|
||||
# 只讀確認，不代表可執行修復
|
||||
journalctl -u postgresql@14-main -n 30 | grep "could not locate a valid checkpoint"
|
||||
pg_lsclusters
|
||||
systemctl status postgresql@14-main --no-pager
|
||||
```
|
||||
|
||||
若確認需要進入 break-glass，先完成：
|
||||
|
||||
```text
|
||||
1. DB owner approval
|
||||
2. 最新本機與 offsite backup evidence
|
||||
3. restore / rollback plan
|
||||
4. maintenance window
|
||||
5. post-check：pg_lsclusters、pg_isready、應用 DB smoke、backup-status、cold-start wrapper
|
||||
```
|
||||
|
||||
### 192.168.0.110 — DevOps 主機
|
||||
@@ -500,7 +510,9 @@ curl http://192.168.0.110:9090/api/v1/query?query=probe_success
|
||||
- Redis: dump.rdb 損壞 (`Wrong signature trying to load DB from file`)
|
||||
- ClickHouse: system table store parts 損壞 (`broken and needs manual correction`)
|
||||
|
||||
**修復步驟**:
|
||||
**Break-glass 修復範例，不屬於一般重啟 SOP**:
|
||||
|
||||
先確認 Sentry DB volume 是受影響權威資料面，並具備 owner approval、volume backup / rollback ref、maintenance window 與 post-check plan。不得由 startup script、AI agent 或值班自動流程直接執行。
|
||||
|
||||
```bash
|
||||
# Step 1: 修復 PostgreSQL WAL
|
||||
@@ -533,7 +545,7 @@ curl -o /dev/null -w "%{http_code}" http://192.168.0.110:9000/
|
||||
# 預期: 200 / 302 / 400 (需登入)
|
||||
```
|
||||
|
||||
**startup-110.sh 已自動化修復**: PostgreSQL WAL 和 Redis RDB 損壞會自動偵測並修復。ClickHouse parts 損壞需手動識別 UUID（因 UUID 每次不同）。
|
||||
**2026-06-26 修正方向**: 舊 `startup-110.sh` 曾把 PostgreSQL WAL 和 Redis RDB 損壞列入自動修復，這類資料層破壞性動作後續必須遷移到 owner-gated break-glass，不得再擴大成預設自動化。ClickHouse parts 損壞同樣需人工識別 UUID、備份與 post-check。
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
|
||||
| Area | Status | Completion | Evidence |
|
||||
|------|--------|------------|----------|
|
||||
| Overall recovery readiness | FULL_STACK_GREEN_DR_ESCROW_BLOCKED | 99% | 2026-06-26 06:40-06:44 all-host read-only refresh confirms the 2026-06-25 21:14 green baseline still holds and adds 112/111/168 reachability. `110 / 120 / 121 / 188 / 112 / 111 / 168` ping and SSH port are OK. Core scope services are green: cold-start `PASS=89 WARN=0 BLOCKED=0`; public routes return expected statuses; AWOOOI API health is `healthy / prod / mock_mode=false`; MOMO health `V10.690`, latest import job `57 completed`, daily freshness `1|2026-06-24`; StockPlatform freshness `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; backup-status 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, offsite/rclone fresh, `last_backup_all=2026-06-26 02:31:02`, `escrow_missing=5`. Do not declare DR complete until `escrow_missing=0`; Wazuh manager registry accepted remains `0`; 111/168 Codex workspace HEAD drift and Mac Mini low free space are workstation blockers, not reboot service blockers. |
|
||||
| Overall recovery readiness | FULL_STACK_GREEN_DR_ESCROW_BLOCKED | 99% | 2026-06-26 06:40-06:55 all-host read-only refresh confirms the 2026-06-25 21:14 green baseline still holds and adds 112/111/168 reachability. `110 / 120 / 121 / 188 / 112 / 111 / 168` ping and SSH port are OK. Core scope services are green: cold-start `PASS=89 WARN=0 BLOCKED=0`; public routes return expected statuses; AWOOOI API health is `healthy / prod / mock_mode=false`; MOMO health `V10.690`, latest import job `57 completed`, daily freshness `1|2026-06-24`; StockPlatform freshness `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; backup-status 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, offsite/rclone fresh, `last_backup_all=2026-06-26 02:31:02`, `escrow_missing=5`. 188 is now explicitly split into product-service green and host hygiene blocker: failed units are `awoooi-startup.service`, `postgresql@14-main.service`, `certbot.service`, `snap.certbot.renew.service`; host PostgreSQL cluster is down while product DB containers/exporters are up; certbot renewal is failing but public cert remains valid until 2026-07-09 UTC. Do not declare DR complete until `escrow_missing=0`; Wazuh manager registry accepted remains `0`; 111/168 Codex workspace HEAD drift and Mac Mini low free space are workstation blockers, not reboot service blockers. |
|
||||
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-26 06:44 readback shows 120 and 121 reachable, K3s active, `mon` and `mon1` both `Ready control-plane`, AWOOOI API/Web replicas split across both nodes, ArgoCD `awoooi-prod Synced / Healthy` at revision `b2945ab9f716d9d685434ae0e67b9318414b27fe`, and `km-vectorize` official 03:00 台北時間 run succeeded with `lastSuccess=2026-06-25T19:00:14Z`. |
|
||||
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-25 19:17 backup readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`, last aggregate `2026-06-25 02:35:09`。2026-06-25 19:19 offsite escrow report shows script presence OK, rclone configured, full and partial rclone markers present, `PASS=8 WARN=5 BLOCKED=0`, `ESCROW_MISSING_COUNT=5`; DR remains blocked on real non-secret credential escrow evidence IDs. |
|
||||
| P2 service / data truth | DONE | 100% | Service routes and core runtime are available, 110 orphan Chrome CPU pressure is cleared, and StockPlatform cron-source drift is repaired. 2026-06-25 21:13 StockPlatform `/api/v1/system/freshness` returned `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; price / chips / margin / AI recommendations are all on `2026-06-25`. `ai.recommendations` row count is `2868`; `core.margin_short_daily` row count is `1976`. MOMO health `V10.690`, current-month parity `15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`, and `DB_DAILY_FRESHNESS 1|2026-06-24` are green; expanded public routes are green. |
|
||||
|
||||
Reference in New Issue
Block a user