awoooi/docs/runbooks/REBOOT-RECOVERY-SOP.md

AWOOOI 重開機恢復 SOP

版本: v5.0
最後更新: 2026-04-05 下午 (台北時間)
更新者: Claude Code (首席架構師)
觸發事件: Harbor Exited(128) Race Condition 根治 + harbor-watchdog 常駐自愈

---

目錄

1. 架構概覽與依賴圖
2. 自動化腳本狀態
3. 正常重啟流程 (計劃性維護)
4. 異常重啟流程 (緊急恢復)
5. 各主機詳細啟動序列
6. 常見故障排查手冊
7. E2E 驗證腳本

---

架構概覽與依賴圖

五主機全貌

192.168.0.188  (OLLAMA / 主服務主機)
├── containerd              ← 基礎，必須最先啟動
├── Docker                  ← 依賴 containerd
├── PostgreSQL :5432        ← K3s kine 後端，API DB
├── Redis :6380             ← Working Memory (重啟後清空，需 warm-up)
├── Ollama :11434           ← LLM 推論引擎
├── Nginx :80/:443          ← 反向代理
├── OpenClaw (clawbot) :8088 ← AI 核心 (Docker Compose)
├── MinIO                   ← Velero 備份存儲 (Docker Compose)
└── SignOz                  ← 可觀測性 (Docker Compose)

192.168.0.110  (DevOps 主機)
├── Docker
├── Harbor :5000            ← Container Registry (K3s 拉映像用)
├── Gitea :3001             ← 主要 Git / CI 管理介面
├── Gitea Act Runner        ← CD pipeline 執行者 (docker: gitea-runner)
├── Langfuse :3100          ← LLMOps 追蹤
├── Prometheus :9090        ← 指標收集
├── Alertmanager :9093      ← 告警路由
├── Grafana :3002           ← 監控儀表板
├── Sentry :9000            ← Error Tracking (2026-03-24，2026-04-05 加入 startup)
└── SignOz                  ← 可觀測性

192.168.0.120  (K3s Master - mon)
├── K3s server (control plane)
├── kube-proxy              ← NodePort 轉發 32334/32335
└── keepalived              ← VIP 192.168.0.125 (secondary)

192.168.0.121  (K3s Worker - mon1)
├── K3s agent (worker)
└── kube-proxy

192.168.0.125  (VIP, keepalived 管理)
├── → :32334  AWOOOI API
└── → :32335  AWOOOI Web

服務依賴關係

嚴格啟動順序:

【188 層】
containerd
  └── Docker
        ├── PostgreSQL (kine DB, API DB)
        │     └── Redis (Working Memory)
        │           └── Ollama (LLM)
        │                 └── Nginx
        │                       ├── SignOz
        │                       ├── MinIO
        │                       └── OpenClaw (依賴 aiops-network)

【110 層】
Docker
  └── harbor-log (等 healthy)
        └── Harbor 全組件 (:5000)
              └── Gitea (:3001)
                    ├── Langfuse
                    ├── Monitoring (Prometheus + Alertmanager + Grafana)
                    │         [Alertmanager → AWOOOI API 告警鏈路]
                    ├── SignOz
                    └── Gitea Act Runner (等 Gitea 就緒後才啟動)

【120/121 層】
K3s (依賴 PostgreSQL@188)
  └── kube-proxy (NodePort 規則)
        └── Pods: API / Web / Worker
              └── keepalived (VIP)

【告警鏈路】
Prometheus → Alertmanager(110)
  → AWOOOI API(121:32334/api/v1/webhooks/alertmanager)  ← 直接，不走 OpenClaw
  → TelegramGateway → Telegram

【CD 鏈路】
Git push → Gitea(110:3001)
  → Act Runner (gitea-runner container)
  → Build Docker image → Harbor(:5000) → kubectl → K3s pods

關鍵依賴說明

服務	關鍵依賴	若依賴失敗
K3s	PostgreSQL@188 ready	K3s 無法啟動，所有 pod 無法調度
OpenClaw	Docker + aiops-network	Telegram AI 對話失效
Alertmanager → API	NetworkPolicy 允許 110/32	告警全面沉默
CD pipeline	Gitea Act Runner online	所有部署失效
Harbor	harbor-log healthy	所有其他 Harbor 容器 Exited 128
K8s pods	Harbor (映像倉庫)	ImagePullBackOff

---

自動化腳本狀態

已部署 (2026-04-05)

主機	腳本	部署位置	systemd service	狀態	類型
188	awoooi-startup.sh	/usr/local/bin/	awoooi-startup.service	✅ enabled	重開機 oneshot
110	awoooi-startup-110.sh	/usr/local/bin/	awoooi-startup-110.service	✅ enabled	重開機 oneshot
110	harbor-watchdog.sh	/usr/local/bin/	harbor-watchdog.service	✅ active	常駐 watchdog
120	K3s 原生	系統內建	k3s.service	✅ enabled	系統服務
121	K3s 原生	系統內建	k3s-agent.service	✅ enabled	系統服務

本地原始碼: scripts/reboot-recovery/

harbor-watchdog（2026-04-05 新增）

問題背景: Harbor 的 Exited (128) Race Condition

• restart: always 在 harbor-log 未就緒時不斷重試，最終進入 backoff 放棄
• 即使 harbor-log 後來 healthy，其他容器不會自動重試
• 只靠 startup 腳本無法處理運行中崩潰的情況

設計:

• Type=simple 常駐進程（不是 oneshot），systemd 永久監控
• 每 60 秒輪詢 http://127.0.0.1:5000/v2/（401 = healthy）
• 偵測到不健康 → 等 5 秒再確認（避免誤報）→ 執行完整時序修復
• 修復邏輯：清除 Exited 容器 → 只啟動 harbor-log → 等 healthy → 啟動全部

查看狀態:

journalctl -u harbor-watchdog.service -f          # 即時 log
systemctl status harbor-watchdog.service          # 服務狀態
cat /var/log/harbor-watchdog.log                  # 持久化 log

各腳本覆蓋清單

188 (7 步驟):

• containerd BoltDB 損壞偵測與修復
• Docker BoltDB 損壞偵測與修復
• PostgreSQL WAL 損壞偵測 + pg_resetwal + kine VACUUM
• Redis 啟動
• Ollama 啟動
• Nginx 啟動
• SignOz + MinIO + aiops-network + OpenClaw

110 (7 步驟 + harbor-watchdog 常駐):

• Docker BoltDB 損壞偵測與修復
• 孤兒容器清除 (Exited 128)
• Harbor（harbor-log first 策略：Phase 1 清除 → Phase 2 只啟 harbor-log → Phase 3 等 healthy → Phase 4 啟全部）
• Gitea + Langfuse + Monitoring (含 Alertmanager 健康驗證)
• SignOz
• Gitea Act Runner (自動清除過期 .runner 配置)
• Sentry (/opt/sentry，含 PostgreSQL WAL + Redis RDB 損壞自動修復)
• harbor-watchdog.service (常駐，每 60s 自動修復)

---

正常重啟流程

適用於：計劃性維護、安全更新、OS 升級等有預期的重啟。

重啟前準備 (T-30 分鐘)

# 1. 確認沒有 INVESTIGATING/MITIGATING 的 incident
curl -s http://192.168.0.121:32334/api/v1/incidents | python3 -c 'import sys,json; d=json.load(sys.stdin); print("Active incidents:", d.get("count",0))'

# 2. 確認 K3s 健康
ssh wooo@192.168.0.120 "kubectl get nodes && kubectl get pods -n awoooi-prod"

# 3. 備份確認 (MinIO Velero)
ssh wooo@192.168.0.120 "kubectl get backup -n velero 2>/dev/null | head -5"

建議重啟順序

第一步: 188 + 110 同時重啟
        (兩者獨立，可並行)

等待 5 分鐘確認基礎服務就緒:
  - PostgreSQL accepting connections
  - Harbor returning HTTP 200
  - Gitea accessible

第二步: 120 + 121 同時重啟
        (K3s cluster)

等待 10 分鐘:
  - Nodes Ready
  - Pods Running

重要: 120/121 必須在 188 PostgreSQL 完全就緒後才重啟， 否則 K3s kine 連不到 DB 無法啟動。

重啟後驗證 (T+15 分鐘)

執行底部 E2E 驗證腳本。所有項目必須 ✅。

---

異常重啟流程

適用於：電源中斷、OOM、系統崩潰、強制重啟等非預期的重啟。

第一步：快速診斷 (T+2 分鐘)

# 188 狀態
ssh ollama@192.168.0.188 "
systemctl is-active containerd docker postgresql@14-main redis-server ollama nginx
docker ps --format '{{.Names}}: {{.Status}}' | grep -E 'clawbot|minio|signoz'
"

# 110 狀態
ssh wooo@192.168.0.110 "
docker ps --format '{{.Names}}: {{.Status}}' | grep -E 'harbor-log|gitea$|alertmanager|gitea-runner'
"

# K3s 狀態
ssh wooo@192.168.0.120 "
kubectl get nodes 2>/dev/null || echo 'K3s not ready'
kubectl get pods -n awoooi-prod 2>/dev/null | grep -v Running | head -10
"

# API 健康
curl -s --max-time 5 http://192.168.0.121:32334/api/v1/health 2>/dev/null | python3 -c 'import sys,json; d=json.load(sys.stdin); print("API:", d["status"])' 2>/dev/null || echo 'API unreachable'

第二步：按優先級恢復

P0 (T+0~5 分鐘): 基礎設施

# 自動化腳本通常已處理，但若沒有自動啟動:
ssh ollama@192.168.0.188 "sudo /usr/local/bin/awoooi-startup.sh"
ssh wooo@192.168.0.110 "echo '0936223270' | sudo -S /usr/local/bin/awoooi-startup-110.sh"

P1 (T+5~15 分鐘): K3s 叢集

# 等 PostgreSQL@188 就緒後
ssh ollama@192.168.0.188 "pg_isready -h localhost -p 5432"

# K3s 若未自動恢復
ssh wooo@192.168.0.120 "sudo systemctl restart k3s"
ssh wooo@192.168.0.121 "sudo systemctl restart k3s-agent"

# 等待 Nodes Ready
ssh wooo@192.168.0.120 "kubectl wait --for=condition=Ready nodes --all --timeout=120s"

P2 (T+15~30 分鐘): 業務驗證

# Pods 全部 Running?
ssh wooo@192.168.0.120 "kubectl get pods -n awoooi-prod"

# API 健康 (含所有 components)?
curl -s http://192.168.0.121:32334/api/v1/health | python3 -m json.tool

# 告警鏈路 E2E 測試
curl -X POST http://192.168.0.121:32334/api/v1/webhooks/alertmanager \
  -H 'Content-Type: application/json' \
  -d '{"receiver":"test","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"RebootRecoveryTest","severity":"info"},"annotations":{"summary":"重開機恢復測試，請忽略"},"startsAt":"2026-04-05T00:00:00Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":""}],"groupLabels":{},"commonLabels":{},"commonAnnotations":{},"externalURL":"","version":"4","groupKey":"test"}'
# 預期: {"success":true,...} 且 Telegram 收到測試告警

---

各主機詳細啟動序列

192.168.0.188 — 主服務主機

步驟	服務	驗證方式	常見失敗原因
1	containerd	systemctl is-active containerd	BoltDB meta.db 損壞
2	Docker	systemctl is-active docker	BoltDB local-kv.db 損壞
3	PostgreSQL	pg_isready -h localhost -p 5432	WAL checkpoint 損壞
3a	kine VACUUM	psql -d k3s_datastore -c "VACUUM ANALYZE kine;"	慢查詢導致 K3s 超時
4	Redis	redis-cli -p 6380 ping	bind 配置錯誤 (必須 0.0.0.0:6380)
5	Ollama	systemctl is-active ollama	GPU 記憶體不足
6	Nginx	systemctl is-active nginx	port 衝突
7a	aiops-network	docker network ls | grep aiops-network	重啟後 external network 消失
7b	OpenClaw	curl http://localhost:8088/health	pip 依賴損壞需 rebuild
7c	MinIO	docker ps | grep minio	無狀態服務，重啟即可
7d	SignOz	curl http://localhost:3301/-/healthy	ClickHouse 起慢

BoltDB 損壞修復:

# containerd meta.db
BOLT=/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db
cp $BOLT ${BOLT}.bak.$(date +%s) && rm $BOLT && systemctl restart containerd

# Docker network BoltDB
BOLT=/var/lib/docker/network/files/local-kv.db
cp $BOLT ${BOLT}.bak.$(date +%s) && rm $BOLT && systemctl restart docker

PostgreSQL WAL 修復:

# 確認是 WAL 損壞
journalctl -u postgresql@14-main -n 30 | grep "could not locate a valid checkpoint"

# 修復
systemctl stop postgresql@14-main
/usr/lib/postgresql/14/bin/pg_resetwal -f /var/lib/postgresql/14/main
systemctl start postgresql@14-main

192.168.0.110 — DevOps 主機

步驟	服務	驗證方式	常見失敗原因
1	Docker	systemctl is-active docker	BoltDB 損壞
2	孤兒容器清除	docker ps -a | grep "Exited (128)"	舊容器引用已不存在的 network
3	harbor-log	docker inspect --format='{{.State.Health.Status}}' harbor-log	syslog :1514 未就緒
3	Harbor 全組件	curl http://localhost:5000/v2/	harbor-log 還沒 healthy
4	Gitea	curl http://localhost:3001	無特殊問題
5	Langfuse	curl http://localhost:3100	DB 未就緒
6	Alertmanager	curl http://localhost:9093/-/healthy	docker compose 未啟動
6a	告警鏈路驗證	發送測試 webhook	webhook URL 錯誤 / NetworkPolicy
7	SignOz	curl http://localhost:3301/-/healthy	ClickHouse 起慢
8	Gitea Runner	docker logs gitea-runner | grep SUCCESS	.runner 配置過期
9	Sentry	curl -o /dev/null -w '%{http_code}' http://localhost:9000/	PostgreSQL WAL/Redis RDB 損壞 (見下方)

Harbor Exited 128 修復:

# 等 harbor-log healthy
until [ "$(docker inspect --format='{{.State.Health.Status}}' harbor-log)" = "healthy" ]; do
  echo "waiting..."; sleep 5
done

# 清除 Exited 128 容器
docker rm -f $(docker ps -a --filter status=exited --format "{{.Names}}" | grep -v harbor-log)

# 重新啟動
cd /home/wooo/harbor/harbor && docker compose up -d

Gitea Runner 重新註冊:

# 清除過期配置 (指向錯誤 hostname 的)
sudo rm /home/wooo/act-runner/data/.runner

# 獲取新的 registration token
curl -s -u 'wooo:TOKEN' 'http://192.168.0.110:3001/api/v1/repos/wooo/awoooi/actions/runners/registration-token'

# 更新 docker-compose.yml 的 GITEA_RUNNER_REGISTRATION_TOKEN
# 然後啟動
cd /home/wooo/act-runner && docker compose up -d

192.168.0.120/121 — K3s 叢集

步驟	項目	驗證方式
前提	PostgreSQL@188 ready	pg_isready -h 192.168.0.188 -p 5432
1	K3s service	systemctl is-active k3s (120)
2	K3s agent	systemctl is-active k3s-agent (121)
3	Nodes Ready	kubectl get nodes
4	Pods Running	kubectl get pods -n awoooi-prod
5	keepalived VIP	ip addr show | grep 192.168.0.125 (120)
6	NodePort	nc -zv 192.168.0.121 32334

---

常見故障排查手冊

1. 告警沉默 (Telegram 沒收到告警)

診斷樹:
  Alertmanager healthy? (http://192.168.0.110:9093/-/healthy)
  ├── NO → docker compose up -d (在 110 /home/wooo/monitoring/)
  └── YES
        ↓
  Webhook URL 正確?
  grep 'url:' /home/wooo/monitoring/alertmanager.yml
  必須是: http://192.168.0.121:32334/api/v1/webhooks/alertmanager
  ├── NO → 修正 URL 並 curl http://localhost:9093/-/reload
  └── YES
        ↓
  從 110 curl POST webhook 成功?
  curl -X POST http://192.168.0.121:32334/api/v1/webhooks/alertmanager ...
  ├── timeout → NetworkPolicy 未允許 110
  │            kubectl apply -f k8s/awoooi-prod/02-network-policy.yaml
  └── {"success":true} → 檢查 Telegram Bot Token

補充診斷 — 特定服務異常但無告警（Alertmanager 正常）:

# 確認 Prometheus 規則數量和關鍵規則是否存在
ssh wooo@192.168.0.110 "curl -s http://localhost:9090/api/v1/rules | python3 -c \"
import sys,json
r=json.load(sys.stdin)
names=[x['name'] for g in r['data']['groups'] for x in g['rules']]
total=len(names)
key=['SentryDown','HarborDown','GiteaDown','OpenClawDown','AlertmanagerDown','AlertChainUnhealthy']
for k in key:
    print(f'{'OK' if k in names else 'MISSING'}: {k}')
print(f'Total rules: {total}')
\""

# 若規則不存在或 total < 25 → 規則未部署
# 修復: 從專案根目錄執行
bash scripts/ops/deploy-alerts.sh

# 若規則存在但 inactive → 確認 blackbox probe target
curl http://192.168.0.110:9090/api/v1/query?query=probe_success

根本教訓 (2026-04-05):

• 舊架構: Alertmanager → OpenClaw 8088 (中間層，重開機後 OpenClaw 不健康就沉默)
• 新架構: Alertmanager → AWOOOI API 直通 (更穩定，單一故障點少)
• NetworkPolicy 必須允許 110/32 進入 pod

2. Gitea CD 無法部署

診斷樹:
  Gitea runner 數量?
  curl -u 'wooo:TOKEN' 'http://192.168.0.110:3001/api/v1/repos/wooo/awoooi/actions/runners'
  ├── total_count: 0 → Runner 離線
  │     docker ps | grep gitea-runner
  │     ├── 不存在 → cd /home/wooo/act-runner && docker compose up -d
  │     └── 存在但 Restarting → docker logs gitea-runner
  │           "lookup gitea" → 清除 data/.runner → docker compose up -d
  └── total_count: 1 → Runner 在線但 CD 失敗
        查看 Gitea Actions 日誌: http://192.168.0.110:3001/wooo/awoooi/actions

3. 網站數據顯示 0 / incidents 消失

原因: Redis Working Memory 重啟後清空
     新版 API (f4f454f+) 啟動時自動 warm-up
     舊版 API 沒有此邏輯

診斷:
  curl http://192.168.0.121:32334/api/v1/incidents
  → count: 0 但 Telegram 有歷史告警?
  → 確認 API pod image 版本

修復 (若 image 版本正確):
  kubectl rollout restart deployment/awoooi-api -n awoooi-prod

修復 (若 image 版本過舊):
  等待 CD 完成部署最新版本

4. NodePort 32334 從外部無法訪問

診斷:
  nc -zv 192.168.0.121 32334  ← 這個通嗎?
  ├── 通 → 用 192.168.0.121:32334 直連
  └── 不通
        kubectl get pods -n awoooi-prod
        ├── 沒有 Running pods → K3s pods 問題
        └── 有 Running pods
              iptables -t nat -L KUBE-NODEPORTS -n | grep 32334
              ├── 沒有規則 → K3s kube-proxy 問題，重啟 k3s
              └── 有規則 → NetworkPolicy 或防火牆

5. Sentry 重開機後損壞修復 (2026-04-05 事故記錄)

症狀: sentry-self-hosted-postgres-1 / redis-1 / clickhouse-1 持續 Restarting

原因: 重開機時各 DB 未正常關閉，導致資料損壞：

• PostgreSQL: WAL 損壞 (could not locate a valid checkpoint record)
• Redis: dump.rdb 損壞 (Wrong signature trying to load DB from file)
• ClickHouse: system table store parts 損壞 (broken and needs manual correction)

修復步驟:

# Step 1: 修復 PostgreSQL WAL
docker run --rm -u 999 -v sentry-postgres:/var/lib/postgresql/data \
  postgres:14 pg_resetwal -f /var/lib/postgresql/data
echo "PostgreSQL WAL reset 完成"

# Step 2: 修復 Redis (session cache，可丟失)
docker run --rm --user root -v sentry-redis:/data alpine \
  sh -c 'rm -f /data/dump.rdb && echo cleared'
echo "Redis dump.rdb 已清除"

# Step 3: 找出損壞的 ClickHouse store 目錄
docker logs sentry-self-hosted-clickhouse-1 2>&1 \
  | grep -oE 'store/[a-f0-9]{3}/[a-f0-9-]+' \
  | sort -u

# Step 4: 依據上方輸出刪除損壞目錄 (以 UUID 為準，以下為範例)
for uuid in "469/469957c9-..." "57d/57d1d567-..."; do
  docker run --rm --user root -v sentry-clickhouse:/data alpine \
    sh -c "rm -rf /data/store/$uuid && echo OK:$uuid"
done

# Step 5: 重啟 Sentry
cd /opt/sentry && docker compose up -d

# Step 6: 等待 2-3 分鐘後驗證
sleep 120
curl -o /dev/null -w "%{http_code}" http://192.168.0.110:9000/
# 預期: 200 / 302 / 400 (需登入)

startup-110.sh 已自動化修復: PostgreSQL WAL 和 Redis RDB 損壞會自動偵測並修復。ClickHouse parts 損壞需手動識別 UUID（因 UUID 每次不同）。

---

6. Harbor ImagePullBackOff

診斷:
  kubectl describe pod <pod> -n awoooi-prod | grep -A 5 Events
  → "ImagePullBackOff" or "ErrImagePull"
  
確認 Harbor 健康:
  curl http://192.168.0.110:5000/v2/ → 應該回 {}

若 Harbor 不健康:
  - 等 harbor-log healthy 後重啟: 見 Harbor Exited 128 修復
  - 確認 Docker daemon 在 120/121 有設定 insecure registry:
    cat /etc/docker/daemon.json | grep insecure

---

E2E 驗證腳本

執行此腳本確認重開機後全系統正常。

#!/bin/bash
# 重開機後 E2E 驗證
# 使用: bash docs/runbooks/scripts/e2e-verify.sh
# 執行位置: 任意可 SSH 的機器

API="http://192.168.0.121:32334"
GREEN='\033[0;32m'; RED='\033[0;31m'; NC='\033[0m'
PASS=0; FAIL=0

check() {
    local name="$1"; local cmd="$2"; local expect="$3"
    result=$(eval "$cmd" 2>/dev/null)
    if echo "$result" | grep -q "$expect"; then
        echo -e "${GREEN}✅ $name${NC}"
        ((PASS++))
    else
        echo -e "${RED}❌ $name${NC} (got: ${result:0:80})"
        ((FAIL++))
    fi
}

echo "=== AWOOOI 重開機 E2E 驗證 $(date '+%Y-%m-%d %H:%M:%S') ==="

# 188 服務
check "188 PostgreSQL" "ssh ollama@192.168.0.188 'pg_isready -h localhost -p 5432'" "accepting"
check "188 Redis" "ssh ollama@192.168.0.188 'redis-cli -p 6380 ping'" "PONG"
check "188 OpenClaw" "ssh ollama@192.168.0.188 'curl -s http://localhost:8088/health'" "healthy"

# 110 服務
check "110 Harbor" "curl -s -o /dev/null -w '%{http_code}' http://192.168.0.110:5000/v2/" "200"
check "110 Gitea" "curl -s -o /dev/null -w '%{http_code}' http://192.168.0.110:3001" "200"
check "110 Alertmanager" "curl -s http://192.168.0.110:9093/-/healthy" "OK"
check "110 Gitea Runner" "curl -s -u 'wooo:TOKEN' 'http://192.168.0.110:3001/api/v1/repos/wooo/awoooi/actions/runners' | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d.get(\"total_count\",0))'" "1"
check "110 Sentry" "curl -s -o /dev/null -w '%{http_code}' --max-time 10 http://192.168.0.110:9000/" "[23][0-9][0-9]\|400"
check "Prometheus rules ≥25" "ssh wooo@192.168.0.110 'curl -s http://localhost:9090/api/v1/rules' | python3 -c \"import sys,json; r=json.load(sys.stdin); print(sum(len(g['rules']) for g in r['data']['groups']))\"" "[2-9][5-9]\|[3-9][0-9]"

# K3s
check "K3s Nodes Ready" "ssh wooo@192.168.0.120 'kubectl get nodes --no-headers' | awk '{print \$2}'" "Ready"
check "K3s Pods Running" "ssh wooo@192.168.0.120 'kubectl get pods -n awoooi-prod --no-headers' | grep -c Running" "3"

# AWOOOI API
check "API Health" "curl -s $API/api/v1/health" '"status":"healthy"'
check "API openclaw" "curl -s $API/api/v1/health" '"openclaw":{"status":"up"'

# 告警鏈路 E2E
PAYLOAD='{"receiver":"e2e-test","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"RebootE2ETest","severity":"info"},"annotations":{"summary":"重開機 E2E 驗證，請忽略"},"startsAt":"2026-04-05T00:00:00Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":""}],"groupLabels":{},"commonLabels":{},"commonAnnotations":{},"externalURL":"","version":"4","groupKey":"e2e-test"}'
check "告警鏈路 E2E" "curl -s -X POST $API/api/v1/webhooks/alertmanager -H 'Content-Type: application/json' -d '$PAYLOAD'" '"success":true'

echo ""
echo "=== 結果: ${PASS} 通過, ${FAIL} 失敗 ==="
[ $FAIL -eq 0 ] && echo -e "${GREEN}🎉 全部通過！系統正常${NC}" || echo -e "${RED}⚠️  有 ${FAIL} 項失敗，請排查${NC}"

---

版本歷史

版本	日期	說明
v1.0	2026-04-04	初版：containerd + Docker + PostgreSQL WAL 修復流程
v2.0	2026-04-05 上午	188 啟動腳本 + 110 啟動腳本 + Harbor race condition 修復
v3.0	2026-04-05 下午	完整架構重盤 + Gitea Runner 自動化 + 告警鏈路根因修復 + NetworkPolicy 修正
v4.0	2026-04-05 下午	Prometheus 規則統一部署 (28條) + Sentry startup (Step 7) + Sentry損壞修復SOP + 規則未部署診斷樹 + E2E腳本加入Sentry/Prometheus驗證
v5.0	2026-04-05 下午	Harbor Exited(128) Race Condition 根治：startup-110.sh 改用 harbor-log first 策略 + 新增 harbor-watchdog.service 常駐自愈（curl -f bug 修正）