2392 lines
76 KiB
Markdown
2392 lines
76 KiB
Markdown
# 全系統自愈閉環 Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** 修復三大系統性缺口：Prometheus 規則未部署 → 告警靜默；日誌只收集不告警；auto_repair 無法修復主機 Docker 層服務。
|
||
|
||
**Architecture:** 方案 A 漸進補強。Sprint 1 補 Prometheus 規則 + 啟動 Sentry + CD 同步；Sprint 2 加 SigNoz log alert；Sprint 3 新增 HostRepairAgent (SSH) 讓 auto_repair_service 能修復 110/188 的 Docker/systemd 服務。
|
||
|
||
**Tech Stack:** Prometheus YAML、Gitea Actions、Python asyncio/asyncssh、K8s Secret、SigNoz webhook、awoooi-startup-110.sh bash
|
||
|
||
---
|
||
|
||
## 檔案結構
|
||
|
||
### Sprint 1 (Infra/Ops)
|
||
|
||
| 檔案 | 動作 | 說明 |
|
||
|------|------|------|
|
||
| `ops/monitoring/alerts-unified.yml` | **建立** | 整合所有 Prometheus 規則 (40+條)，含統一標籤 |
|
||
| `scripts/ops/deploy-alerts.sh` | **建立** | 部署 alerts.yml 到 110 Prometheus 的腳本 |
|
||
| `.gitea/workflows/cd.yaml` | **修改** | 加入 `deploy-alerts` job，paths trigger 含 ops/monitoring/ |
|
||
| `scripts/reboot-recovery/awoooi-startup-110.sh` | **修改** | 加入 Step 7 啟動 Sentry (/opt/sentry) |
|
||
| `docs/runbooks/REBOOT-RECOVERY-SOP.md` | **修改** | v4.0：Sentry 步驟、診斷樹更新、E2E 驗證腳本更新 |
|
||
|
||
### Sprint 2 (SigNoz + Sentry)
|
||
|
||
| 檔案 | 動作 | 說明 |
|
||
|------|------|------|
|
||
| `ops/signoz/alerting/log-rules.md` | **建立** | SigNoz log alert rules 文檔 (SigNoz 無 YAML 匯入) |
|
||
| `apps/api/src/main.py` | **修改** | Sentry init 加入 layer/component tags |
|
||
|
||
### Sprint 3 (Host Auto-Repair)
|
||
|
||
| 檔案 | 動作 | 說明 |
|
||
|------|------|------|
|
||
| `apps/api/src/services/host_repair_agent.py` | **建立** | SSH 主機修復 Agent |
|
||
| `apps/api/src/models/playbook.py` | **修改** | 加入 `ActionType.SSH_COMMAND` |
|
||
| `apps/api/src/services/auto_repair_service.py` | **修改** | `_execute_step` 加入 SSH_COMMAND 路由 |
|
||
| `scripts/repair-bot/repair-bot-110.sh` | **建立** | 110 主機白名單修復腳本 |
|
||
| `scripts/repair-bot/repair-bot-188.sh` | **建立** | 188 主機白名單修復腳本 |
|
||
| `k8s/awoooi-prod/04-repair-ssh-secret.yaml` | **建立** | SSH key K8s Secret template (不含實際 key) |
|
||
| `apps/api/tests/test_host_repair_agent.py` | **建立** | HostRepairAgent 單元測試 |
|
||
|
||
---
|
||
|
||
## SPRINT 1：Prometheus 規則部署 + Sentry 啟動 + CD 同步
|
||
|
||
---
|
||
|
||
### Task 1：建立統一 Prometheus 規則檔
|
||
|
||
**說明：** 將 `k8s/monitoring/` 下所有規則轉換為 Docker Prometheus 格式，合併為 `ops/monitoring/alerts-unified.yml`。加入 `layer`/`host`/`auto_repair` 統一標籤。
|
||
|
||
**Files:**
|
||
- Create: `ops/monitoring/alerts-unified.yml`
|
||
|
||
- [ ] **Step 1.1: 確認來源規則**
|
||
|
||
```bash
|
||
# 在 awoooi 專案根目錄執行
|
||
grep -c "alert:" k8s/monitoring/k3s-alerts.yaml k8s/monitoring/alert-chain-monitor.yaml k8s/monitoring/database-alerts.yaml k8s/monitoring/minio-kali-alerts.yaml k8s/monitoring/k3s-alerts-supplemental.yaml
|
||
```
|
||
|
||
預期輸出：每個檔案各顯示規則數，總計應 > 40。
|
||
|
||
- [ ] **Step 1.2: 建立 alerts-unified.yml**
|
||
|
||
建立 `ops/monitoring/alerts-unified.yml`，內容如下（完整）：
|
||
|
||
```yaml
|
||
# ops/monitoring/alerts-unified.yml
|
||
# AWOOOI 統一 Prometheus 告警規則
|
||
# 2026-04-05 Claude Code: 整合所有規則，加入統一 layer 標籤
|
||
# 部署目標: 192.168.0.110:/home/wooo/monitoring/alerts.yml
|
||
# 部署方式: scripts/ops/deploy-alerts.sh (CD 自動部署)
|
||
#
|
||
# 標籤規範:
|
||
# layer: k8s | docker-110 | docker-188 | systemd-188
|
||
# component: 服務名稱
|
||
# team: ops | backend | ai | platform
|
||
# host: "110" | "188" | "120" | "121"
|
||
# auto_repair: "true" | "false"
|
||
|
||
groups:
|
||
|
||
# =========================================================================
|
||
# 主機層告警 (host_alerts)
|
||
# =========================================================================
|
||
- name: host_alerts
|
||
rules:
|
||
- alert: HostDown
|
||
expr: up{job=~"node-exporter.*"} == 0
|
||
for: 1m
|
||
labels:
|
||
severity: critical
|
||
layer: systemd-188
|
||
team: ops
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "主機 {{ $labels.host }} 不可達"
|
||
description: "Node Exporter 無回應超過 1 分鐘"
|
||
|
||
- alert: HostHighCpuLoad
|
||
expr: 100 - (avg by(host) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
|
||
for: 5m
|
||
labels:
|
||
severity: warning
|
||
layer: systemd-188
|
||
team: ops
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "主機 {{ $labels.host }} CPU 高負載"
|
||
description: "CPU 使用率超過 80%"
|
||
|
||
- alert: HostOutOfMemory
|
||
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
|
||
for: 5m
|
||
labels:
|
||
severity: warning
|
||
layer: systemd-188
|
||
team: ops
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "主機 {{ $labels.host }} 記憶體不足"
|
||
description: "記憶體使用率超過 85%"
|
||
|
||
- alert: HostOutOfDiskSpace
|
||
expr: (1 - (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"})) * 100 > 85
|
||
for: 5m
|
||
labels:
|
||
severity: warning
|
||
layer: systemd-188
|
||
team: ops
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "主機 {{ $labels.host }} 磁碟空間不足"
|
||
description: "磁碟使用率超過 85%"
|
||
|
||
# =========================================================================
|
||
# K8s 叢集告警 (kubernetes_alerts)
|
||
# =========================================================================
|
||
- name: kubernetes_alerts
|
||
rules:
|
||
- alert: K3sNodeNotReady
|
||
expr: kube_node_status_condition{condition="Ready", status="true"} == 0
|
||
for: 2m
|
||
labels:
|
||
severity: critical
|
||
layer: k8s
|
||
team: ops
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "K3s 節點 {{ $labels.node }} 未就緒"
|
||
description: "節點超過 2 分鐘未達到 Ready 狀態"
|
||
|
||
- alert: KubePodCrashLooping
|
||
expr: rate(kube_pod_container_status_restarts_total{namespace="awoooi-prod"}[15m]) > 0
|
||
for: 5m
|
||
labels:
|
||
severity: warning
|
||
layer: k8s
|
||
team: ops
|
||
auto_repair: "true"
|
||
annotations:
|
||
summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} 持續重啟"
|
||
description: "Pod 在過去 15 分鐘內重啟次數異常"
|
||
|
||
- alert: KubePodNotReady
|
||
expr: kube_pod_status_ready{condition="true",namespace="awoooi-prod"} == 0
|
||
for: 5m
|
||
labels:
|
||
severity: warning
|
||
layer: k8s
|
||
team: ops
|
||
auto_repair: "true"
|
||
annotations:
|
||
summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} 未就緒"
|
||
description: "Running 中的 Pod 超過 5 分鐘未達到 Ready 狀態"
|
||
|
||
- alert: KubeDeploymentReplicasMismatch
|
||
expr: kube_deployment_spec_replicas{namespace="awoooi-prod"} != kube_deployment_status_replicas_available{namespace="awoooi-prod"}
|
||
for: 10m
|
||
labels:
|
||
severity: warning
|
||
layer: k8s
|
||
team: ops
|
||
auto_repair: "true"
|
||
annotations:
|
||
summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} 副本數不匹配"
|
||
description: "期望副本數與可用副本數不一致超過 10 分鐘"
|
||
|
||
- alert: VeleroBackupFailed
|
||
expr: increase(velero_backup_failure_total[24h]) > 0
|
||
for: 5m
|
||
labels:
|
||
severity: warning
|
||
layer: k8s
|
||
team: ops
|
||
component: velero
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "Velero 備份失敗"
|
||
description: "過去 24 小時有備份失敗"
|
||
|
||
- alert: VeleroBackupNotRun
|
||
expr: time() - velero_backup_last_successful_timestamp > 86400
|
||
for: 10m
|
||
labels:
|
||
severity: critical
|
||
layer: k8s
|
||
team: ops
|
||
component: velero
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "Velero 超過 24 小時未成功備份"
|
||
description: "最後一次成功備份超過 24 小時"
|
||
|
||
# =========================================================================
|
||
# 資料庫告警 (database_alerts)
|
||
# =========================================================================
|
||
- name: database_alerts
|
||
rules:
|
||
- alert: PostgreSQLDown
|
||
expr: up{job="postgres-exporter"} == 0 or pg_up == 0
|
||
for: 1m
|
||
labels:
|
||
severity: critical
|
||
layer: systemd-188
|
||
component: postgres
|
||
host: "188"
|
||
team: ops
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "PostgreSQL 資料庫離線"
|
||
description: "PostgreSQL Exporter 無法連接資料庫超過 1 分鐘"
|
||
|
||
- alert: RedisDown
|
||
expr: up{job="redis-exporter"} == 0 or redis_up == 0
|
||
for: 1m
|
||
labels:
|
||
severity: critical
|
||
layer: systemd-188
|
||
component: redis
|
||
host: "188"
|
||
team: ops
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "Redis 快取服務離線"
|
||
description: "Redis Exporter 無法連接 Redis 超過 1 分鐘"
|
||
|
||
- alert: PostgreSQLHighConnections
|
||
expr: pg_stat_activity_count > 80
|
||
for: 5m
|
||
labels:
|
||
severity: warning
|
||
layer: systemd-188
|
||
component: postgres
|
||
team: ops
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "PostgreSQL 連接數過高"
|
||
description: "當前連接數 {{ $value }} 超過 80"
|
||
|
||
- alert: RedisMemoryHigh
|
||
expr: redis_memory_used_bytes / redis_memory_max_bytes * 100 > 80
|
||
for: 5m
|
||
labels:
|
||
severity: warning
|
||
layer: systemd-188
|
||
component: redis
|
||
team: ops
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "Redis 記憶體使用過高"
|
||
description: "Redis 記憶體使用率超過 80%"
|
||
|
||
# =========================================================================
|
||
# 服務可用性告警 (service_alerts) — 含 layer 標籤決定修復路徑
|
||
# =========================================================================
|
||
- name: service_alerts
|
||
rules:
|
||
# ---- 188 Docker 層 ----
|
||
- alert: OpenClawDown
|
||
# 2026-04-05 Claude Code: 修正舊命名 ClawBotDown → OpenClawDown
|
||
expr: up{job="clawbot"} == 0
|
||
for: 2m
|
||
labels:
|
||
severity: critical
|
||
layer: docker-188
|
||
component: openclaw
|
||
host: "188"
|
||
team: ops
|
||
auto_repair: "true"
|
||
annotations:
|
||
summary: "OpenClaw 服務離線"
|
||
description: "OpenClaw (192.168.0.188:8088) 已離線超過 2 分鐘"
|
||
|
||
- alert: SignOzDown
|
||
expr: probe_success{job="blackbox-http", instance=~".*3301.*"} == 0
|
||
for: 2m
|
||
labels:
|
||
severity: warning
|
||
layer: docker-188
|
||
component: signoz
|
||
host: "188"
|
||
team: ops
|
||
auto_repair: "true"
|
||
annotations:
|
||
summary: "SignOz 服務離線"
|
||
description: "SignOz (192.168.0.188:3301) 已離線超過 2 分鐘"
|
||
|
||
# ---- 110 Docker 層 ----
|
||
- alert: SentryDown
|
||
expr: probe_success{job="blackbox-tcp", instance="192.168.0.110:9000"} == 0
|
||
for: 2m
|
||
labels:
|
||
severity: warning
|
||
layer: docker-110
|
||
component: sentry
|
||
host: "110"
|
||
team: ops
|
||
auto_repair: "true"
|
||
annotations:
|
||
summary: "Sentry 服務離線"
|
||
description: "Sentry (192.168.0.110:9000) 已離線超過 2 分鐘"
|
||
|
||
- alert: HarborDown
|
||
expr: probe_success{job="blackbox-http", instance=~".*5000.*"} == 0
|
||
for: 2m
|
||
labels:
|
||
severity: critical
|
||
layer: docker-110
|
||
component: harbor
|
||
host: "110"
|
||
team: ops
|
||
auto_repair: "true"
|
||
annotations:
|
||
summary: "Harbor Registry 離線"
|
||
description: "Harbor (192.168.0.110:5000) 已離線超過 2 分鐘，CD pipeline 將無法拉取映像"
|
||
|
||
- alert: GiteaDown
|
||
expr: probe_success{job="blackbox-http", instance="http://192.168.0.110:3001"} == 0
|
||
for: 2m
|
||
labels:
|
||
severity: critical
|
||
layer: docker-110
|
||
component: gitea
|
||
host: "110"
|
||
team: ops
|
||
auto_repair: "true"
|
||
annotations:
|
||
summary: "Gitea Git 服務離線"
|
||
description: "Gitea (192.168.0.110:3001) 已離線超過 2 分鐘，CD pipeline 失效"
|
||
|
||
- alert: AlertmanagerDown
|
||
expr: probe_success{job="blackbox-tcp", instance="192.168.0.110:9093"} == 0
|
||
for: 2m
|
||
labels:
|
||
severity: critical
|
||
layer: docker-110
|
||
component: alertmanager
|
||
host: "110"
|
||
team: ops
|
||
auto_repair: "true"
|
||
annotations:
|
||
summary: "Alertmanager 離線"
|
||
description: "Alertmanager (192.168.0.110:9093) 已離線，所有告警將靜默"
|
||
|
||
# =========================================================================
|
||
# 告警鏈路監控 (alert_chain) — 防止 2026-03-26/04-05 事故重演
|
||
# =========================================================================
|
||
- name: alert_chain
|
||
rules:
|
||
- alert: AlertChainBroken_Alertmanager
|
||
expr: |
|
||
sum(rate(awoooi_webhook_requests_total{source="alertmanager",status!="success"}[5m]))
|
||
/ sum(rate(awoooi_webhook_requests_total{source="alertmanager"}[5m])) > 0.1
|
||
for: 10m
|
||
labels:
|
||
severity: critical
|
||
layer: k8s
|
||
team: platform
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "Alertmanager Webhook 錯誤率 > 10%"
|
||
description: "告警鏈路可能斷裂，請執行 E2E 驗證"
|
||
|
||
- alert: AlertChainBroken_Sentry
|
||
expr: |
|
||
sum(rate(awoooi_webhook_requests_total{source="sentry",status!="success"}[5m]))
|
||
/ sum(rate(awoooi_webhook_requests_total{source="sentry"}[5m])) > 0.1
|
||
for: 10m
|
||
labels:
|
||
severity: warning
|
||
layer: k8s
|
||
team: platform
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "Sentry Webhook 錯誤率 > 10%"
|
||
description: "Sentry 錯誤可能無法正確處理"
|
||
|
||
- alert: NoAlertsReceived2Hours
|
||
expr: time() - max by (source)(awoooi_alert_chain_last_success_timestamp) > 7200
|
||
for: 5m
|
||
labels:
|
||
severity: warning
|
||
layer: k8s
|
||
team: platform
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "2 小時內未收到任何告警 ({{ $labels.source }})"
|
||
description: "可能是告警鏈路問題，請執行 Smoke Test"
|
||
|
||
- alert: AlertChainUnhealthy
|
||
expr: awoooi_alert_chain_healthy == 0
|
||
for: 5m
|
||
labels:
|
||
severity: critical
|
||
layer: k8s
|
||
team: platform
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "告警鏈路不健康 ({{ $labels.source }})"
|
||
description: "告警鏈路標記為不健康，最近處理失敗"
|
||
|
||
# =========================================================================
|
||
# 自動修復監控 (auto_repair)
|
||
# =========================================================================
|
||
- name: auto_repair
|
||
rules:
|
||
- alert: AutoRepairLowSuccessRate
|
||
expr: awoooi_auto_repair_success_rate < 0.3
|
||
for: 30m
|
||
labels:
|
||
severity: warning
|
||
layer: k8s
|
||
team: backend
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "自動修復成功率過低 ({{ $value | humanizePercentage }})"
|
||
description: "動作 {{ $labels.action }} 的成功率低於 30%，建議檢查 Playbook"
|
||
|
||
- alert: PermanentFixRequired
|
||
expr: sum(rate(awoooi_anomaly_escalation_total{level="PERMANENT_FIX"}[1h])) > 0
|
||
for: 1m
|
||
labels:
|
||
severity: critical
|
||
layer: k8s
|
||
team: backend
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "需要永久修復的異常升級"
|
||
description: "有異常升級到 PERMANENT_FIX 級別，需要根本修復"
|
||
|
||
# =========================================================================
|
||
# MinIO / Kali 告警
|
||
# =========================================================================
|
||
- name: minio_kali_alerts
|
||
rules:
|
||
- alert: MinIODown
|
||
expr: probe_success{job="blackbox-http", instance=~".*9000.*", instance!~".*sentry.*"} == 0
|
||
for: 2m
|
||
labels:
|
||
severity: warning
|
||
layer: docker-188
|
||
component: minio
|
||
host: "188"
|
||
team: ops
|
||
auto_repair: "true"
|
||
annotations:
|
||
summary: "MinIO (Velero 備份) 離線"
|
||
description: "MinIO (192.168.0.188:9000) 已離線超過 2 分鐘，Velero 備份可能失敗"
|
||
|
||
- alert: KaliScannerDown
|
||
expr: probe_success{job="blackbox-http", instance=~".*192.168.0.112.*"} == 0
|
||
for: 5m
|
||
labels:
|
||
severity: info
|
||
layer: docker-188
|
||
component: kali
|
||
host: "112"
|
||
team: ops
|
||
auto_repair: "false"
|
||
annotations:
|
||
summary: "Kali Scanner 離線"
|
||
description: "Kali (192.168.0.112:8080) 離線，安全掃描功能暫停"
|
||
```
|
||
|
||
- [ ] **Step 1.3: 驗證 YAML 語法**
|
||
|
||
```bash
|
||
cd /Users/ogt/awoooi
|
||
python3 -c "import yaml; yaml.safe_load(open('ops/monitoring/alerts-unified.yml'))" && echo "YAML OK"
|
||
```
|
||
|
||
預期輸出：`YAML OK`
|
||
|
||
- [ ] **Step 1.4: Commit**
|
||
|
||
```bash
|
||
git add ops/monitoring/alerts-unified.yml
|
||
git commit -m "ops(monitoring): 統一 Prometheus 告警規則 — 40+條含統一 layer 標籤
|
||
|
||
修正:
|
||
- ClawBotDown → OpenClawDown (舊命名廢棄)
|
||
- 加入 SentryDown/HarborDown/GiteaDown/AlertmanagerDown
|
||
- 所有規則補齊 layer/component/host/auto_repair 統一標籤
|
||
- 整合 k8s/monitoring/*.yaml → ops/monitoring/alerts-unified.yml
|
||
|
||
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2：部署規則到 110 Prometheus
|
||
|
||
**Files:**
|
||
- Create: `scripts/ops/deploy-alerts.sh`
|
||
|
||
- [ ] **Step 2.1: 建立部署腳本**
|
||
|
||
```bash
|
||
mkdir -p scripts/ops
|
||
```
|
||
|
||
建立 `scripts/ops/deploy-alerts.sh`：
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
# scripts/ops/deploy-alerts.sh
|
||
# 部署統一告警規則到 110 Prometheus
|
||
# 2026-04-05 Claude Code: Sprint 1 自動化部署
|
||
# 用法: bash scripts/ops/deploy-alerts.sh [--dry-run]
|
||
|
||
set -eo pipefail
|
||
|
||
RULES_FILE="ops/monitoring/alerts-unified.yml"
|
||
TARGET_HOST="192.168.0.110"
|
||
TARGET_PATH="/home/wooo/monitoring/alerts.yml"
|
||
PROMETHEUS_URL="http://${TARGET_HOST}:9090"
|
||
DRY_RUN="${1:-}"
|
||
|
||
log() { echo "[$(date '+%H:%M:%S')] $*"; }
|
||
|
||
# 確認檔案存在
|
||
if [ ! -f "$RULES_FILE" ]; then
|
||
echo "ERROR: $RULES_FILE not found"
|
||
exit 1
|
||
fi
|
||
|
||
# 驗證 YAML 語法
|
||
python3 -c "import yaml; yaml.safe_load(open('$RULES_FILE'))" || { echo "ERROR: YAML syntax error"; exit 1; }
|
||
log "✅ YAML 語法驗證通過"
|
||
|
||
# Dry run 模式
|
||
if [ "$DRY_RUN" = "--dry-run" ]; then
|
||
log "DRY RUN: would deploy $RULES_FILE to ${TARGET_HOST}:${TARGET_PATH}"
|
||
RULE_COUNT=$(grep -c "alert:" "$RULES_FILE")
|
||
log "規則數量: $RULE_COUNT 條"
|
||
exit 0
|
||
fi
|
||
|
||
# 備份現有規則
|
||
ssh wooo@${TARGET_HOST} "cp ${TARGET_PATH} ${TARGET_PATH}.bak.\$(date +%Y%m%d%H%M%S) 2>/dev/null || true"
|
||
log "✅ 現有規則已備份"
|
||
|
||
# 部署新規則
|
||
scp "$RULES_FILE" wooo@${TARGET_HOST}:${TARGET_PATH}
|
||
log "✅ 規則已複製到 ${TARGET_HOST}"
|
||
|
||
# Reload Prometheus
|
||
ssh wooo@${TARGET_HOST} "curl -s -X POST ${PROMETHEUS_URL}/-/reload"
|
||
sleep 3
|
||
|
||
# 驗證規則數量
|
||
RULE_COUNT=$(ssh wooo@${TARGET_HOST} "curl -s ${PROMETHEUS_URL}/api/v1/rules | python3 -c \"import sys,json; r=json.load(sys.stdin); print(sum(len(g['rules']) for g in r['data']['groups']))\"")
|
||
log "Prometheus 已載入 ${RULE_COUNT} 條規則"
|
||
|
||
if [ "$RULE_COUNT" -lt 30 ]; then
|
||
echo "ERROR: 規則數量異常 ($RULE_COUNT < 30)，請檢查"
|
||
exit 1
|
||
fi
|
||
|
||
# 驗證關鍵規則存在
|
||
KEY_RULES=("SentryDown" "HarborDown" "GiteaDown" "OpenClawDown" "AlertmanagerDown" "AlertChainUnhealthy")
|
||
for rule in "${KEY_RULES[@]}"; do
|
||
EXISTS=$(ssh wooo@${TARGET_HOST} "curl -s ${PROMETHEUS_URL}/api/v1/rules | python3 -c \"import sys,json; r=json.load(sys.stdin); names=[x['name'] for g in r['data']['groups'] for x in g['rules']]; print('OK' if '$rule' in names else 'MISSING')\"")
|
||
if [ "$EXISTS" = "OK" ]; then
|
||
log "✅ $rule"
|
||
else
|
||
echo "❌ $rule 未找到"
|
||
exit 1
|
||
fi
|
||
done
|
||
|
||
log "🎉 部署完成！所有關鍵規則已生效"
|
||
```
|
||
|
||
```bash
|
||
chmod +x scripts/ops/deploy-alerts.sh
|
||
```
|
||
|
||
- [ ] **Step 2.2: 執行部署 (dry-run 先確認)**
|
||
|
||
```bash
|
||
bash scripts/ops/deploy-alerts.sh --dry-run
|
||
```
|
||
|
||
預期輸出：
|
||
```
|
||
[HH:MM:SS] ✅ YAML 語法驗證通過
|
||
[HH:MM:SS] DRY RUN: would deploy ops/monitoring/alerts-unified.yml to 192.168.0.110:...
|
||
[HH:MM:SS] 規則數量: XX 條
|
||
```
|
||
|
||
- [ ] **Step 2.3: 執行實際部署**
|
||
|
||
```bash
|
||
bash scripts/ops/deploy-alerts.sh
|
||
```
|
||
|
||
預期輸出：所有規則 ✅，`🎉 部署完成！`
|
||
|
||
- [ ] **Step 2.4: 驗證 Prometheus UI**
|
||
|
||
```bash
|
||
ssh wooo@192.168.0.110 "curl -s http://localhost:9090/api/v1/rules | python3 -c \"
|
||
import sys, json
|
||
r = json.load(sys.stdin)
|
||
groups = r['data']['groups']
|
||
total = sum(len(g['rules']) for g in groups)
|
||
print(f'Groups: {len(groups)}, Total rules: {total}')
|
||
for g in groups:
|
||
print(f' {g[\"name\"]}: {len(g[\"rules\"])} rules')
|
||
\""
|
||
```
|
||
|
||
預期：6+ groups，總規則數 > 30。
|
||
|
||
- [ ] **Step 2.5: Commit**
|
||
|
||
```bash
|
||
git add scripts/ops/deploy-alerts.sh
|
||
git commit -m "ops(scripts): 加入 deploy-alerts.sh 自動部署 Prometheus 規則
|
||
|
||
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3：CD pipeline 自動同步告警規則
|
||
|
||
**說明：** 修改 `.gitea/workflows/cd.yaml`，加入 `deploy-alerts` job。當 `ops/monitoring/alerts-unified.yml` 有變更時，自動部署到 110。
|
||
|
||
**Files:**
|
||
- Modify: `.gitea/workflows/cd.yaml`
|
||
|
||
- [ ] **Step 3.1: 確認 CD 文件末尾**
|
||
|
||
```bash
|
||
tail -20 .gitea/workflows/cd.yaml
|
||
```
|
||
|
||
記錄最後一個 job 的名稱（通常是 build-and-deploy）。
|
||
|
||
- [ ] **Step 3.2: 加入 deploy-alerts job**
|
||
|
||
在 `.gitea/workflows/cd.yaml` 的 `on.push.paths` 區塊加入：
|
||
|
||
找到：
|
||
```yaml
|
||
paths:
|
||
# 只有實際影響部署的程式碼才觸發 CD
|
||
- 'apps/**'
|
||
- 'k8s/**'
|
||
- '.gitea/workflows/**'
|
||
# docs/、memory/、ADR、ops/ 等不觸發
|
||
```
|
||
|
||
改為：
|
||
```yaml
|
||
paths:
|
||
# 只有實際影響部署的程式碼才觸發 CD
|
||
- 'apps/**'
|
||
- 'k8s/**'
|
||
- '.gitea/workflows/**'
|
||
- 'ops/monitoring/alerts-unified.yml' # 2026-04-05 Claude Code: 告警規則變更自動部署
|
||
```
|
||
|
||
在 jobs 末尾加入新 job（在 `build-and-deploy:` 同層）：
|
||
|
||
```yaml
|
||
deploy-alerts:
|
||
name: "Deploy Prometheus Alert Rules"
|
||
# 2026-04-05 Claude Code: 告警規則 CD — 不依賴 build-and-deploy
|
||
# 觸發條件: ops/monitoring/alerts-unified.yml 有變更
|
||
runs-on: ubuntu-latest
|
||
timeout-minutes: 5
|
||
if: |
|
||
contains(github.event.commits[*].modified, 'ops/monitoring/alerts-unified.yml') ||
|
||
github.event_name == 'workflow_dispatch'
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
|
||
- name: Validate alerts YAML
|
||
run: python3 -c "import yaml; yaml.safe_load(open('ops/monitoring/alerts-unified.yml')); print('YAML OK')"
|
||
|
||
- name: Deploy alerts to Prometheus
|
||
run: bash scripts/ops/deploy-alerts.sh
|
||
|
||
- name: Notify deploy result
|
||
if: always()
|
||
env:
|
||
TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }}
|
||
TELEGRAM_CHAT_ID: ${{ secrets.TELEGRAM_CHAT_ID }}
|
||
run: |
|
||
STATUS="${{ job.status }}"
|
||
EMOJI="✅"
|
||
[ "$STATUS" != "success" ] && EMOJI="❌"
|
||
curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
|
||
-d "chat_id=${TELEGRAM_CHAT_ID}" \
|
||
-d "text=${EMOJI} Prometheus 告警規則部署 ${STATUS} (commit: ${GITHUB_SHA::7})" || true
|
||
```
|
||
|
||
- [ ] **Step 3.3: Commit 並推送到 Gitea 觸發驗證**
|
||
|
||
```bash
|
||
git add .gitea/workflows/cd.yaml
|
||
git commit -m "ci: 加入 deploy-alerts CD job — 告警規則變更自動部署到 Prometheus
|
||
|
||
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>"
|
||
git push gitea main
|
||
```
|
||
|
||
- [ ] **Step 3.4: 監控 Gitea Actions 確認 deploy-alerts job 成功**
|
||
|
||
```bash
|
||
# 瀏覽器開啟或 curl 確認
|
||
curl -s -u 'wooo:TOKEN' 'http://192.168.0.110:3001/api/v1/repos/wooo/awoooi/actions/runs?limit=3' | python3 -c "
|
||
import sys, json
|
||
runs = json.load(sys.stdin)
|
||
for r in runs.get('workflow_runs', [])[:3]:
|
||
print(r['name'], r['status'], r['conclusion'])
|
||
"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4：啟動 Sentry + 加入 startup-110.sh
|
||
|
||
**Files:**
|
||
- Modify: `scripts/reboot-recovery/awoooi-startup-110.sh`
|
||
|
||
- [ ] **Step 4.1: 手動啟動 Sentry 驗證**
|
||
|
||
```bash
|
||
ssh wooo@192.168.0.110 "cd /opt/sentry && docker compose up -d 2>&1 | tail -10"
|
||
sleep 30
|
||
ssh wooo@192.168.0.110 "curl -s -o /dev/null -w '%{http_code}' http://localhost:9000/api/0/projects/"
|
||
```
|
||
|
||
預期：HTTP 200（或 403，表示 Sentry 已啟動但需登入）。
|
||
|
||
- [ ] **Step 4.2: 加入 startup-110.sh Step 7**
|
||
|
||
找到 `scripts/reboot-recovery/awoooi-startup-110.sh` 的最後 `# 完成` 區塊：
|
||
|
||
```bash
|
||
# ──────────────────────────────────────────────
|
||
# 完成
|
||
# ──────────────────────────────────────────────
|
||
log "=== 192.168.0.110 啟動序列完成 ==="
|
||
```
|
||
|
||
在它之前插入：
|
||
|
||
```bash
|
||
# ──────────────────────────────────────────────
|
||
# STEP 7: Sentry（Error Tracking）
|
||
# 2026-04-05 Claude Code: 加入 — 解決重開機後 Sentry 未自動啟動
|
||
# 安裝位置: /opt/sentry (2026-03-24 已安裝)
|
||
# DSN: awoooi-web :2, awoooi-api :3 (見 memory/project_sentry_full_integration.md)
|
||
# ──────────────────────────────────────────────
|
||
log "[7/7] 啟動 Sentry..."
|
||
SENTRY_DIR="/opt/sentry"
|
||
if [ -d "$SENTRY_DIR" ]; then
|
||
cd "$SENTRY_DIR"
|
||
docker compose up -d 2>&1 | tail -5
|
||
log "✅ Sentry 啟動指令已發送 (啟動約需 2-3 分鐘)"
|
||
sleep 20
|
||
# 非阻塞驗證：Sentry 啟動慢，只做快速健康檢查
|
||
if curl -sf --max-time 10 http://localhost:9000/api/0/projects/ >/dev/null 2>&1 || \
|
||
curl -sf --max-time 10 -o /dev/null -w "%{http_code}" http://localhost:9000/ | grep -q "200\|302\|400"; then
|
||
log "✅ Sentry 已回應"
|
||
else
|
||
log "⚠️ Sentry 尚未就緒（正常現象，通常需 2-3 分鐘）"
|
||
fi
|
||
else
|
||
log "⚠️ 找不到 Sentry 目錄: $SENTRY_DIR"
|
||
fi
|
||
```
|
||
|
||
同時更新末尾日誌行，加入 Sentry URL：
|
||
|
||
找到：
|
||
```bash
|
||
log "Gitea Runner: docker logs gitea-runner"
|
||
```
|
||
|
||
改為：
|
||
```bash
|
||
log "Gitea Runner: docker logs gitea-runner"
|
||
log "Sentry: http://192.168.0.110:9000"
|
||
```
|
||
|
||
- [ ] **Step 4.3: 部署到 110**
|
||
|
||
```bash
|
||
scp scripts/reboot-recovery/awoooi-startup-110.sh wooo@192.168.0.110:/usr/local/bin/awoooi-startup-110.sh
|
||
ssh wooo@192.168.0.110 "chmod +x /usr/local/bin/awoooi-startup-110.sh"
|
||
```
|
||
|
||
- [ ] **Step 4.4: Commit**
|
||
|
||
```bash
|
||
git add scripts/reboot-recovery/awoooi-startup-110.sh
|
||
git commit -m "ops(startup): startup-110.sh 加入 Step 7 Sentry 自動啟動
|
||
|
||
Sentry 已安裝於 /opt/sentry (2026-03-24)，但重開機後未自動啟動
|
||
加入非阻塞啟動：docker compose up -d + 20s 等待 + 快速健康檢查
|
||
|
||
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5：更新 REBOOT-RECOVERY-SOP.md v4.0
|
||
|
||
**Files:**
|
||
- Modify: `docs/runbooks/REBOOT-RECOVERY-SOP.md`
|
||
|
||
- [ ] **Step 5.1: 更新版本號和摘要**
|
||
|
||
找到文件開頭：
|
||
```markdown
|
||
> **版本**: v3.0
|
||
> **最後更新**: 2026-04-05 (台北時間)
|
||
> **更新者**: Claude Code (首席架構師)
|
||
> **觸發事件**: 兩次重開機事故後完整盤點 + 告警鏈路根因修復 + Gitea Runner 自動化
|
||
```
|
||
|
||
改為：
|
||
```markdown
|
||
> **版本**: v4.0
|
||
> **最後更新**: 2026-04-05 (台北時間)
|
||
> **更新者**: Claude Code (首席架構師)
|
||
> **觸發事件**: Prometheus 規則未部署根因修復 + Sentry 啟動自動化 + 全系統自愈閉環設計
|
||
```
|
||
|
||
- [ ] **Step 5.2: 更新架構圖加入 Sentry**
|
||
|
||
找到：
|
||
```
|
||
├── Alertmanager :9093 ← 告警路由
|
||
├── Grafana :3002 ← 監控儀表板
|
||
└── SignOz ← 可觀測性
|
||
```
|
||
|
||
改為：
|
||
```
|
||
├── Alertmanager :9093 ← 告警路由
|
||
├── Grafana :3002 ← 監控儀表板
|
||
├── Sentry :9000 ← Error Tracking (2026-03-24，2026-04-05 加入 startup)
|
||
└── SignOz ← 可觀測性
|
||
```
|
||
|
||
- [ ] **Step 5.3: 更新 110 自動化腳本清單**
|
||
|
||
找到：
|
||
```
|
||
**110 (6 步驟)**:
|
||
```
|
||
|
||
改為：
|
||
```
|
||
**110 (7 步驟)**:
|
||
```
|
||
|
||
在末尾加入：
|
||
```
|
||
- **Gitea Act Runner** (自動清除過期 .runner 配置)
|
||
- **Sentry** (/opt/sentry，Error Tracking)
|
||
```
|
||
|
||
- [ ] **Step 5.4: 更新 110 詳細啟動序列表格**
|
||
|
||
找到 `192.168.0.110` 啟動序列表格末尾，加入：
|
||
|
||
```markdown
|
||
| 7 | Sentry | `curl http://localhost:9000/api/0/projects/` | Docker 未啟動 (重開機後) |
|
||
```
|
||
|
||
- [ ] **Step 5.5: 更新告警沉默故障排查加入「規則未部署」診斷**
|
||
|
||
在告警沉默診斷樹末尾加入：
|
||
|
||
```markdown
|
||
**補充診斷 — 特定服務無告警但 Alertmanager 正常：**
|
||
```
|
||
確認 Prometheus 是否有該規則:
|
||
ssh wooo@192.168.0.110 "curl -s http://localhost:9090/api/v1/rules | python3 -c \"import sys,json; r=json.load(sys.stdin); names=[x['name'] for g in r['data']['groups'] for x in g['rules']]; [print(n) for n in names if 'Sentry' in n or 'Harbor' in n or 'Gitea' in n]\""
|
||
|
||
若規則不存在:
|
||
→ 規則未部署，執行: bash scripts/ops/deploy-alerts.sh
|
||
若規則存在但 inactive:
|
||
→ 檢查 blackbox probe target IP 是否正確
|
||
→ curl http://192.168.0.110:9090/api/v1/query?query=probe_success
|
||
```
|
||
```
|
||
|
||
- [ ] **Step 5.6: 更新 E2E 驗證腳本加入 Sentry + 規則數**
|
||
|
||
在 E2E 驗證腳本的 `# 告警鏈路 E2E` 之前加入：
|
||
|
||
```bash
|
||
check "110 Sentry" "curl -s -o /dev/null -w '%{http_code}' --max-time 10 http://192.168.0.110:9000/" "200\|302\|400"
|
||
check "Prometheus rules >30" "ssh wooo@192.168.0.110 'curl -s http://localhost:9090/api/v1/rules' | python3 -c \"import sys,json; r=json.load(sys.stdin); n=sum(len(g['rules']) for g in r['data']['groups']); print(n)\"" "[3-9][0-9]"
|
||
```
|
||
|
||
- [ ] **Step 5.7: 更新版本歷史**
|
||
|
||
在版本歷史表格加入：
|
||
```markdown
|
||
| v4.0 | 2026-04-05 下午 | Prometheus 規則統一部署 + Sentry startup + 診斷樹補充 + E2E 腳本更新 |
|
||
```
|
||
|
||
- [ ] **Step 5.8: Commit**
|
||
|
||
```bash
|
||
git add docs/runbooks/REBOOT-RECOVERY-SOP.md
|
||
git commit -m "docs(sop): REBOOT-RECOVERY-SOP.md v4.0
|
||
|
||
更新:
|
||
- 加入 Sentry /opt/sentry 啟動說明 (110 Step 7)
|
||
- 告警沉默診斷樹補充「規則未部署」診斷
|
||
- E2E 驗證腳本加入 Sentry + Prometheus 規則數驗證
|
||
- 架構圖補充 Sentry :9000
|
||
|
||
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 6：Sprint 1 驗收
|
||
|
||
- [ ] **Step 6.1: 執行完整 E2E 驗證**
|
||
|
||
```bash
|
||
# 確認 Prometheus 規則
|
||
ssh wooo@192.168.0.110 "curl -s http://localhost:9090/api/v1/rules | python3 -c \"
|
||
import sys, json
|
||
r = json.load(sys.stdin)
|
||
names = [x['name'] for g in r['data']['groups'] for x in g['rules']]
|
||
key = ['SentryDown','HarborDown','GiteaDown','OpenClawDown','AlertmanagerDown','AlertChainUnhealthy']
|
||
for k in key:
|
||
print(f'{'✅' if k in names else '❌'} {k}')
|
||
print(f'Total rules: {len(names)}')
|
||
\""
|
||
```
|
||
|
||
預期：所有 6 條關鍵規則 ✅，Total rules ≥ 30。
|
||
|
||
- [ ] **Step 6.2: 確認 Sentry 健康**
|
||
|
||
```bash
|
||
curl -s -o /dev/null -w "%{http_code}" http://192.168.0.110:9000/
|
||
```
|
||
|
||
預期：`200`、`302` 或 `400` (需登入，表示服務正常)。
|
||
|
||
- [ ] **Step 6.3: 觸發測試告警確認鏈路**
|
||
|
||
```bash
|
||
curl -X POST http://192.168.0.121:32334/api/v1/webhooks/alertmanager \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{
|
||
"receiver":"test","status":"firing",
|
||
"alerts":[{
|
||
"status":"firing",
|
||
"labels":{"alertname":"Sprint1ValidationTest","severity":"info","layer":"k8s","component":"test"},
|
||
"annotations":{"summary":"Sprint 1 驗收測試，請忽略"},
|
||
"startsAt":"2026-04-05T00:00:00Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":""
|
||
}],
|
||
"groupLabels":{},"commonLabels":{},"commonAnnotations":{},"externalURL":"","version":"4","groupKey":"sprint1-test"
|
||
}'
|
||
```
|
||
|
||
預期：`{"success":true,...}` 且 Telegram 收到測試告警。
|
||
|
||
Sprint 1 ✅ 完成條件：規則 ≥ 30、關鍵規則全部存在、Sentry 回應、告警鏈路通。
|
||
|
||
---
|
||
|
||
## SPRINT 2：SigNoz Log Alerting + Sentry 整合完善
|
||
|
||
---
|
||
|
||
### Task 7：SigNoz Log Alert Rules 建立
|
||
|
||
**說明：** SigNoz 不支援 YAML 匯入，alert rules 透過 UI 設定。本 task 建立文檔並逐一在 UI 設定。
|
||
|
||
**Files:**
|
||
- Create: `ops/signoz/alerting/log-rules.md`
|
||
|
||
- [ ] **Step 7.1: 建立 log-rules.md 文檔**
|
||
|
||
建立 `ops/signoz/alerting/log-rules.md`：
|
||
|
||
```markdown
|
||
# SigNoz Log-Based Alert Rules
|
||
# 2026-04-05 Claude Code: Sprint 2 — 日誌告警
|
||
# 設定位置: http://192.168.0.188:3301/alerts (SigNoz UI)
|
||
# Webhook: http://192.168.0.121:32334/api/v1/webhooks/signoz
|
||
|
||
## Rule 1: API 高錯誤日誌率
|
||
|
||
| 欄位 | 值 |
|
||
|------|-----|
|
||
| Name | APIHighErrorLogRate |
|
||
| Type | Logs Based Alert |
|
||
| Query | `service.name = "awoooi-api" AND severity_text = "ERROR"` |
|
||
| Condition | Count > 10 per 5m |
|
||
| For | 5m |
|
||
| Severity | warning |
|
||
| Labels | layer=k8s, component=api, team=backend |
|
||
|
||
## Rule 2: Worker 任務失敗
|
||
|
||
| 欄位 | 值 |
|
||
|------|-----|
|
||
| Name | WorkerTaskFailed |
|
||
| Type | Logs Based Alert |
|
||
| Query | `service.name = "awoooi-worker" AND (body CONTAINS "task_failed" OR body CONTAINS "Unhandled exception")` |
|
||
| Condition | Count > 5 per 5m |
|
||
| For | 5m |
|
||
| Severity | warning |
|
||
| Labels | layer=k8s, component=worker, team=backend |
|
||
|
||
## Rule 3: Pod OOM Kill
|
||
|
||
| 欄位 | 值 |
|
||
|------|-----|
|
||
| Name | PodOOMKilled |
|
||
| Type | Logs Based Alert |
|
||
| Query | `body CONTAINS "OOMKilled" OR body CONTAINS "OutOfMemory"` |
|
||
| Condition | Count > 0 per 1m |
|
||
| For | 1m |
|
||
| Severity | critical |
|
||
| Labels | layer=k8s, component=k8s, team=ops |
|
||
|
||
## Rule 4: Telegram Polling 失敗
|
||
|
||
| 欄位 | 值 |
|
||
|------|-----|
|
||
| Name | TelegramPollingFailed |
|
||
| Type | Logs Based Alert |
|
||
| Query | `service.name = "awoooi-api" AND body CONTAINS "telegram_polling_error"` |
|
||
| Condition | Count > 3 per 5m |
|
||
| For | 5m |
|
||
| Severity | critical |
|
||
| Labels | layer=k8s, component=api, team=platform |
|
||
|
||
## Rule 5: Nemotron 全部超時
|
||
|
||
| 欄位 | 值 |
|
||
|------|-----|
|
||
| Name | NemotronAllTimeout |
|
||
| Type | Logs Based Alert |
|
||
| Query | `service.name = "awoooi-api" AND body CONTAINS "nemotron_tool_call_timeout"` |
|
||
| Condition | Count > 5 per 5m |
|
||
| For | 5m |
|
||
| Severity | warning |
|
||
| Labels | layer=k8s, component=ai, team=ai |
|
||
|
||
## 設定步驟
|
||
|
||
1. 開啟 http://192.168.0.188:3301/alerts
|
||
2. 點擊 "New Alert Rule"
|
||
3. 選擇 "Logs Based Alert"
|
||
4. 填入上述欄位
|
||
5. Notification Channel: 選擇 awoooi-api webhook
|
||
6. 保存並啟用
|
||
```
|
||
|
||
- [ ] **Step 7.2: 在 SigNoz UI 設定 Rule 1 (APIHighErrorLogRate)**
|
||
|
||
開啟 http://192.168.0.188:3301/alerts，依照上面 log-rules.md 的 Rule 1 設定。
|
||
|
||
設定完成後，在 UI 確認 rule 狀態為 `inactive` (表示已設定，目前無觸發)。
|
||
|
||
- [ ] **Step 7.3: 設定其餘 4 條規則 (Rule 2-5)**
|
||
|
||
依照 log-rules.md 逐一設定 Rule 2、3、4、5。
|
||
|
||
- [ ] **Step 7.4: 驗證 SigNoz webhook 鏈路**
|
||
|
||
```bash
|
||
curl -X POST http://192.168.0.121:32334/api/v1/webhooks/signoz \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{
|
||
"alerts": [{
|
||
"labels": {"alertname": "Sprint2LogAlertTest", "severity": "info", "layer": "k8s"},
|
||
"annotations": {"summary": "Sprint 2 log alert 驗證，請忽略"},
|
||
"status": "firing"
|
||
}],
|
||
"version": "4"
|
||
}'
|
||
```
|
||
|
||
預期：`{"success":true}`
|
||
|
||
- [ ] **Step 7.5: Commit log-rules.md**
|
||
|
||
```bash
|
||
git add ops/signoz/alerting/log-rules.md
|
||
git commit -m "ops(signoz): 建立 log-based alert rules 文檔 (Sprint 2)
|
||
|
||
5 條規則: APIHighErrorLogRate/WorkerTaskFailed/PodOOMKilled/
|
||
TelegramPollingFailed/NemotronAllTimeout
|
||
|
||
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 8：Sentry SDK Tags 補全
|
||
|
||
**說明：** 在 API 的 Sentry init 加入統一標籤，讓 Sentry 事件攜帶 layer/component 資訊。
|
||
|
||
**Files:**
|
||
- Modify: `apps/api/src/main.py`
|
||
|
||
- [ ] **Step 8.1: 確認 main.py Sentry 初始化位置**
|
||
|
||
```bash
|
||
grep -n "sentry_sdk\|sentry.init\|SENTRY_DSN" apps/api/src/main.py | head -10
|
||
```
|
||
|
||
- [ ] **Step 8.2: 加入統一 tags**
|
||
|
||
找到 Sentry init 的設定（通常是 `sentry_sdk.init(...)` 或在 `configure_sentry()` 函式）。
|
||
|
||
在 `sentry_sdk.init()` 呼叫後加入：
|
||
|
||
```python
|
||
# 2026-04-05 Claude Code: 加入統一標籤，對齊 Prometheus/auto_repair layer 規範
|
||
import sentry_sdk
|
||
sentry_sdk.set_tag("layer", "k8s")
|
||
sentry_sdk.set_tag("component", "api")
|
||
sentry_sdk.set_tag("host", "k8s-awoooi-prod")
|
||
sentry_sdk.set_tag("team", "backend")
|
||
```
|
||
|
||
- [ ] **Step 8.3: 驗證 (啟動 API 後確認 Sentry 事件有 tag)**
|
||
|
||
```bash
|
||
# 在 API pod 內觸發一個測試 exception
|
||
kubectl exec -n awoooi-prod deploy/awoooi-api -- python3 -c "
|
||
import sentry_sdk
|
||
sentry_sdk.capture_message('Sprint2 tag validation', level='info')
|
||
print('sent')
|
||
"
|
||
```
|
||
|
||
在 Sentry UI (http://192.168.0.110:9000) 確認事件有 `layer: k8s` tag。
|
||
|
||
- [ ] **Step 8.4: Commit**
|
||
|
||
```bash
|
||
git add apps/api/src/main.py
|
||
git commit -m "feat(api): Sentry init 加入統一 layer/component 標籤
|
||
|
||
對齊 Prometheus 告警標籤規範 (layer/component/team)
|
||
讓 Sentry 事件與 auto_repair 路由決策保持一致
|
||
|
||
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
## SPRINT 3：Host Auto-Repair Agent (SSH)
|
||
|
||
---
|
||
|
||
### Task 9：主機白名單修復腳本
|
||
|
||
**說明：** 在 110/188 建立嚴格白名單的修復腳本，SSH key 的 `command=` 限制只能執行此腳本，防止任意命令執行。
|
||
|
||
**Files:**
|
||
- Create: `scripts/repair-bot/repair-bot-110.sh`
|
||
- Create: `scripts/repair-bot/repair-bot-188.sh`
|
||
|
||
- [ ] **Step 9.1: 建立修復腳本目錄**
|
||
|
||
```bash
|
||
mkdir -p scripts/repair-bot
|
||
```
|
||
|
||
- [ ] **Step 9.2: 建立 repair-bot-110.sh**
|
||
|
||
建立 `scripts/repair-bot/repair-bot-110.sh`：
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
# scripts/repair-bot/repair-bot-110.sh
|
||
# 修復機器人白名單腳本 — 110 主機 (DevOps 金庫)
|
||
# 2026-04-05 Claude Code: Sprint 3 Host Auto-Repair
|
||
#
|
||
# 安全設計:
|
||
# - SSH authorized_keys 的 command= 指向此腳本
|
||
# - 只允許執行 ALLOWED_COMMANDS 中定義的修復指令
|
||
# - 格式: repair:<component>
|
||
# - SSH key 洩漏也只能執行白名單內的 docker compose up -d
|
||
#
|
||
# 部署位置: /home/wooo/bin/repair-bot-110.sh (on 192.168.0.110)
|
||
# 使用者: wooo
|
||
|
||
LOG="/var/log/awoooi-repair-bot.log"
|
||
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG"; }
|
||
|
||
# 白名單: component → compose dir
|
||
declare -A COMPOSE_DIRS=(
|
||
["sentry"]="/opt/sentry"
|
||
["harbor"]="/home/wooo/harbor/harbor"
|
||
["gitea"]="/home/wooo/gitea"
|
||
["gitea-runner"]="/home/wooo/act-runner"
|
||
["langfuse"]="/home/wooo/langfuse"
|
||
["alertmanager"]="/home/wooo/monitoring"
|
||
["signoz"]="/home/wooo/signoz/deploy/docker"
|
||
)
|
||
|
||
CMD="${SSH_ORIGINAL_COMMAND:-}"
|
||
log "repair-bot-110 invoked: CMD=$CMD"
|
||
|
||
if [[ "$CMD" =~ ^repair:([a-z0-9_-]+)$ ]]; then
|
||
COMPONENT="${BASH_REMATCH[1]}"
|
||
DIR="${COMPOSE_DIRS[$COMPONENT]}"
|
||
|
||
if [ -z "$DIR" ]; then
|
||
log "DENIED: unknown component '$COMPONENT'"
|
||
echo "REPAIR_DENIED:unknown_component:$COMPONENT"
|
||
exit 1
|
||
fi
|
||
|
||
if [ ! -d "$DIR" ]; then
|
||
log "DENIED: directory not found '$DIR'"
|
||
echo "REPAIR_DENIED:dir_not_found:$DIR"
|
||
exit 1
|
||
fi
|
||
|
||
log "EXECUTING: cd $DIR && docker compose up -d"
|
||
cd "$DIR" && docker compose up -d 2>&1 | tail -5
|
||
EXIT_CODE=$?
|
||
|
||
if [ $EXIT_CODE -eq 0 ]; then
|
||
log "REPAIR_OK: $COMPONENT"
|
||
echo "REPAIR_OK:$COMPONENT"
|
||
else
|
||
log "REPAIR_FAIL: $COMPONENT (exit $EXIT_CODE)"
|
||
echo "REPAIR_FAIL:$COMPONENT:exit_$EXIT_CODE"
|
||
exit 1
|
||
fi
|
||
elif [ "$CMD" = "health" ]; then
|
||
# 健康檢查 — 允許連線測試
|
||
echo "REPAIR_BOT_HEALTHY:110"
|
||
else
|
||
log "DENIED: invalid command '$CMD'"
|
||
echo "REPAIR_DENIED:invalid_command"
|
||
exit 1
|
||
fi
|
||
```
|
||
|
||
- [ ] **Step 9.3: 建立 repair-bot-188.sh**
|
||
|
||
建立 `scripts/repair-bot/repair-bot-188.sh`：
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
# scripts/repair-bot/repair-bot-188.sh
|
||
# 修復機器人白名單腳本 — 188 主機 (主服務主機)
|
||
# 2026-04-05 Claude Code: Sprint 3 Host Auto-Repair
|
||
#
|
||
# 部署位置: /home/ollama/bin/repair-bot-188.sh (on 192.168.0.188)
|
||
# 使用者: ollama
|
||
|
||
LOG="/var/log/awoooi-repair-bot.log"
|
||
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG"; }
|
||
|
||
# 白名單: component → 修復方式
|
||
declare -A COMPOSE_DIRS=(
|
||
["openclaw"]="/home/ollama/clawbot-v5"
|
||
["minio"]="/home/ollama/minio"
|
||
["signoz"]="/home/ollama/signoz/deploy/docker"
|
||
)
|
||
|
||
declare -A SYSTEMD_SERVICES=(
|
||
["redis"]="redis-server"
|
||
["nginx"]="nginx"
|
||
["ollama"]="ollama"
|
||
)
|
||
|
||
CMD="${SSH_ORIGINAL_COMMAND:-}"
|
||
log "repair-bot-188 invoked: CMD=$CMD"
|
||
|
||
if [[ "$CMD" =~ ^repair:([a-z0-9_-]+)$ ]]; then
|
||
COMPONENT="${BASH_REMATCH[1]}"
|
||
|
||
# Docker Compose 類
|
||
DIR="${COMPOSE_DIRS[$COMPONENT]}"
|
||
if [ -n "$DIR" ]; then
|
||
if [ ! -d "$DIR" ]; then
|
||
log "DENIED: directory not found '$DIR'"
|
||
echo "REPAIR_DENIED:dir_not_found:$DIR"
|
||
exit 1
|
||
fi
|
||
log "EXECUTING: cd $DIR && docker compose up -d"
|
||
cd "$DIR" && docker compose up -d 2>&1 | tail -5
|
||
EXIT_CODE=$?
|
||
[ $EXIT_CODE -eq 0 ] && echo "REPAIR_OK:$COMPONENT" || { echo "REPAIR_FAIL:$COMPONENT"; exit 1; }
|
||
exit 0
|
||
fi
|
||
|
||
# Systemd 類
|
||
SVC="${SYSTEMD_SERVICES[$COMPONENT]}"
|
||
if [ -n "$SVC" ]; then
|
||
log "EXECUTING: sudo systemctl restart $SVC"
|
||
sudo systemctl restart "$SVC" 2>&1
|
||
EXIT_CODE=$?
|
||
[ $EXIT_CODE -eq 0 ] && echo "REPAIR_OK:$COMPONENT" || { echo "REPAIR_FAIL:$COMPONENT"; exit 1; }
|
||
exit 0
|
||
fi
|
||
|
||
log "DENIED: unknown component '$COMPONENT'"
|
||
echo "REPAIR_DENIED:unknown_component:$COMPONENT"
|
||
exit 1
|
||
|
||
elif [ "$CMD" = "health" ]; then
|
||
echo "REPAIR_BOT_HEALTHY:188"
|
||
else
|
||
log "DENIED: invalid command '$CMD'"
|
||
echo "REPAIR_DENIED:invalid_command"
|
||
exit 1
|
||
fi
|
||
```
|
||
|
||
- [ ] **Step 9.4: 部署腳本到主機**
|
||
|
||
```bash
|
||
# 部署到 110
|
||
ssh wooo@192.168.0.110 "mkdir -p /home/wooo/bin"
|
||
scp scripts/repair-bot/repair-bot-110.sh wooo@192.168.0.110:/home/wooo/bin/repair-bot-110.sh
|
||
ssh wooo@192.168.0.110 "chmod +x /home/wooo/bin/repair-bot-110.sh"
|
||
|
||
# 部署到 188
|
||
ssh ollama@192.168.0.188 "mkdir -p /home/ollama/bin"
|
||
scp scripts/repair-bot/repair-bot-188.sh ollama@192.168.0.188:/home/ollama/bin/repair-bot-188.sh
|
||
ssh ollama@192.168.0.188 "chmod +x /home/ollama/bin/repair-bot-188.sh"
|
||
```
|
||
|
||
- [ ] **Step 9.5: Commit**
|
||
|
||
```bash
|
||
git add scripts/repair-bot/
|
||
git commit -m "ops(repair-bot): 主機白名單修復腳本 (Sprint 3)
|
||
|
||
110: sentry/harbor/gitea/gitea-runner/langfuse/alertmanager
|
||
188: openclaw/minio/signoz/redis/nginx/ollama
|
||
|
||
安全設計: command= 限制 + 嚴格白名單 + 操作日誌
|
||
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 10：SSH Key 基礎設施
|
||
|
||
**Files:**
|
||
- Create: `k8s/awoooi-prod/04-repair-ssh-secret.yaml`
|
||
|
||
- [ ] **Step 10.1: 生成 SSH keypair**
|
||
|
||
```bash
|
||
# 在 Mac 本機執行（不上傳私鑰到 Git）
|
||
ssh-keygen -t ed25519 -C "awoooi-repair-bot-2026" -f /tmp/awoooi_repair_bot -N ""
|
||
echo "Public key:"
|
||
cat /tmp/awoooi_repair_bot.pub
|
||
```
|
||
|
||
記錄公鑰內容（格式：`ssh-ed25519 AAAA... awoooi-repair-bot-2026`）。
|
||
|
||
- [ ] **Step 10.2: 在 110 設定受限 authorized_keys**
|
||
|
||
```bash
|
||
# 將以下內容加入 110 的 authorized_keys（替換 PUBKEY 為實際公鑰）
|
||
PUBKEY=$(cat /tmp/awoooi_repair_bot.pub)
|
||
ssh wooo@192.168.0.110 "echo 'command=\"/home/wooo/bin/repair-bot-110.sh\",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ${PUBKEY}' >> ~/.ssh/authorized_keys"
|
||
|
||
# 驗證
|
||
ssh -i /tmp/awoooi_repair_bot -o StrictHostKeyChecking=accept-new wooo@192.168.0.110 "health"
|
||
```
|
||
|
||
預期輸出：`REPAIR_BOT_HEALTHY:110`
|
||
|
||
- [ ] **Step 10.3: 在 188 設定受限 authorized_keys**
|
||
|
||
```bash
|
||
PUBKEY=$(cat /tmp/awoooi_repair_bot.pub)
|
||
ssh ollama@192.168.0.188 "echo 'command=\"/home/ollama/bin/repair-bot-188.sh\",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ${PUBKEY}' >> ~/.ssh/authorized_keys"
|
||
|
||
# 驗證
|
||
ssh -i /tmp/awoooi_repair_bot -o StrictHostKeyChecking=accept-new ollama@192.168.0.188 "health"
|
||
```
|
||
|
||
預期輸出：`REPAIR_BOT_HEALTHY:188`
|
||
|
||
- [ ] **Step 10.4: 存入 K8s Secret**
|
||
|
||
```bash
|
||
kubectl create secret generic awoooi-repair-ssh-key \
|
||
-n awoooi-prod \
|
||
--from-file=id_ed25519=/tmp/awoooi_repair_bot \
|
||
--from-file=id_ed25519.pub=/tmp/awoooi_repair_bot.pub
|
||
|
||
# 驗證
|
||
kubectl get secret awoooi-repair-ssh-key -n awoooi-prod
|
||
```
|
||
|
||
- [ ] **Step 10.5: 建立 Secret YAML template (不含實際 key)**
|
||
|
||
建立 `k8s/awoooi-prod/04-repair-ssh-secret.yaml`：
|
||
|
||
```yaml
|
||
# k8s/awoooi-prod/04-repair-ssh-secret.yaml
|
||
# SSH Secret Template — 不含實際私鑰 (存於 K8s，不上 Git)
|
||
# 2026-04-05 Claude Code: Sprint 3 Host Auto-Repair
|
||
#
|
||
# 建立方式:
|
||
# ssh-keygen -t ed25519 -C "awoooi-repair-bot-2026" -f /tmp/awoooi_repair_bot -N ""
|
||
# kubectl create secret generic awoooi-repair-ssh-key \
|
||
# -n awoooi-prod \
|
||
# --from-file=id_ed25519=/tmp/awoooi_repair_bot \
|
||
# --from-file=id_ed25519.pub=/tmp/awoooi_repair_bot.pub
|
||
#
|
||
# 主機配置 (已完成):
|
||
# 110 ~/.ssh/authorized_keys: command="/home/wooo/bin/repair-bot-110.sh",... <pubkey>
|
||
# 188 ~/.ssh/authorized_keys: command="/home/ollama/bin/repair-bot-188.sh",... <pubkey>
|
||
|
||
apiVersion: v1
|
||
kind: Secret
|
||
metadata:
|
||
name: awoooi-repair-ssh-key
|
||
namespace: awoooi-prod
|
||
annotations:
|
||
# 此 template 不含實際私鑰，需手動 kubectl create secret 建立
|
||
awoooi.io/secret-type: "ssh-repair-bot"
|
||
awoooi.io/created: "2026-04-05"
|
||
type: Opaque
|
||
# data: 不在版控中 — 使用 kubectl create secret 建立
|
||
```
|
||
|
||
- [ ] **Step 10.6: Commit (只 commit template，不 commit 私鑰)**
|
||
|
||
```bash
|
||
# 確認私鑰不在 git 追蹤中
|
||
git status | grep -v awoooi_repair_bot # 確認 /tmp/ 的 key 不會被加入
|
||
|
||
git add k8s/awoooi-prod/04-repair-ssh-secret.yaml
|
||
git commit -m "k8s(secret): 加入 repair-ssh-key Secret template (不含實際私鑰)
|
||
|
||
實際私鑰透過 kubectl create secret 手動建立，不上 Git
|
||
主機 110/188 的 authorized_keys 已設定 command= 限制
|
||
|
||
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 11：HostRepairAgent Python 模組
|
||
|
||
**Files:**
|
||
- Create: `apps/api/src/services/host_repair_agent.py`
|
||
- Create: `apps/api/tests/test_host_repair_agent.py`
|
||
|
||
- [ ] **Step 11.1: 寫測試 (先)**
|
||
|
||
建立 `apps/api/tests/test_host_repair_agent.py`：
|
||
|
||
```python
|
||
"""
|
||
tests/test_host_repair_agent.py
|
||
Host Repair Agent 單元測試
|
||
不需要實際 SSH 連線 — 測試路由邏輯和命令組裝
|
||
"""
|
||
import pytest
|
||
from unittest.mock import AsyncMock, MagicMock, patch
|
||
|
||
|
||
# =============================================================================
|
||
# 測試 HostRepairConfig 路由
|
||
# =============================================================================
|
||
|
||
class TestHostRepairConfig:
|
||
def test_layer_docker_110_routes_to_110(self):
|
||
from src.services.host_repair_agent import get_ssh_config_for_layer
|
||
config = get_ssh_config_for_layer("docker-110")
|
||
assert config["user"] == "wooo"
|
||
assert config["host"] == "192.168.0.110"
|
||
|
||
def test_layer_docker_188_routes_to_188(self):
|
||
from src.services.host_repair_agent import get_ssh_config_for_layer
|
||
config = get_ssh_config_for_layer("docker-188")
|
||
assert config["user"] == "ollama"
|
||
assert config["host"] == "192.168.0.188"
|
||
|
||
def test_layer_systemd_188_routes_to_188(self):
|
||
from src.services.host_repair_agent import get_ssh_config_for_layer
|
||
config = get_ssh_config_for_layer("systemd-188")
|
||
assert config["user"] == "ollama"
|
||
assert config["host"] == "192.168.0.188"
|
||
|
||
def test_unknown_layer_raises(self):
|
||
from src.services.host_repair_agent import get_ssh_config_for_layer
|
||
with pytest.raises(ValueError, match="Unknown layer"):
|
||
get_ssh_config_for_layer("unknown-layer")
|
||
|
||
def test_k8s_layer_raises(self):
|
||
"""k8s layer 不走 SSH，應 raise"""
|
||
from src.services.host_repair_agent import get_ssh_config_for_layer
|
||
with pytest.raises(ValueError, match="layer 'k8s' is handled by kubectl"):
|
||
get_ssh_config_for_layer("k8s")
|
||
|
||
|
||
# =============================================================================
|
||
# 測試 SSH 命令組裝
|
||
# =============================================================================
|
||
|
||
class TestSSHCommandBuilding:
|
||
def test_repair_command_format(self):
|
||
from src.services.host_repair_agent import build_repair_command
|
||
cmd = build_repair_command("sentry")
|
||
assert cmd == "repair:sentry"
|
||
|
||
def test_repair_command_component_sanitized(self):
|
||
"""防止 command injection"""
|
||
from src.services.host_repair_agent import build_repair_command
|
||
with pytest.raises(ValueError, match="Invalid component"):
|
||
build_repair_command("sentry; rm -rf /")
|
||
|
||
def test_repair_command_valid_components(self):
|
||
from src.services.host_repair_agent import build_repair_command
|
||
valid = ["sentry", "harbor", "gitea", "openclaw", "gitea-runner", "alertmanager", "redis", "nginx"]
|
||
for component in valid:
|
||
cmd = build_repair_command(component)
|
||
assert cmd == f"repair:{component}"
|
||
|
||
|
||
# =============================================================================
|
||
# 測試 HostRepairAgent.repair() 路由
|
||
# =============================================================================
|
||
|
||
class TestHostRepairAgent:
|
||
@pytest.mark.asyncio
|
||
async def test_repair_success_returns_ok(self):
|
||
from src.services.host_repair_agent import HostRepairAgent
|
||
|
||
agent = HostRepairAgent()
|
||
with patch.object(agent, "_ssh_execute", new_callable=AsyncMock) as mock_ssh:
|
||
mock_ssh.return_value = "REPAIR_OK:sentry"
|
||
|
||
result = await agent.repair(layer="docker-110", component="sentry")
|
||
|
||
assert result.success is True
|
||
assert result.component == "sentry"
|
||
assert result.layer == "docker-110"
|
||
mock_ssh.assert_called_once_with(
|
||
host="192.168.0.110",
|
||
user="wooo",
|
||
command="repair:sentry"
|
||
)
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_repair_fail_returns_failure(self):
|
||
from src.services.host_repair_agent import HostRepairAgent
|
||
|
||
agent = HostRepairAgent()
|
||
with patch.object(agent, "_ssh_execute", new_callable=AsyncMock) as mock_ssh:
|
||
mock_ssh.return_value = "REPAIR_FAIL:harbor:exit_1"
|
||
|
||
result = await agent.repair(layer="docker-110", component="harbor")
|
||
|
||
assert result.success is False
|
||
assert "REPAIR_FAIL" in result.error
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_repair_ssh_timeout_returns_failure(self):
|
||
from src.services.host_repair_agent import HostRepairAgent
|
||
import asyncio
|
||
|
||
agent = HostRepairAgent()
|
||
with patch.object(agent, "_ssh_execute", new_callable=AsyncMock) as mock_ssh:
|
||
mock_ssh.side_effect = asyncio.TimeoutError()
|
||
|
||
result = await agent.repair(layer="docker-110", component="sentry")
|
||
|
||
assert result.success is False
|
||
assert "timeout" in result.error.lower()
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_repair_denied_returns_failure(self):
|
||
from src.services.host_repair_agent import HostRepairAgent
|
||
|
||
agent = HostRepairAgent()
|
||
with patch.object(agent, "_ssh_execute", new_callable=AsyncMock) as mock_ssh:
|
||
mock_ssh.return_value = "REPAIR_DENIED:unknown_component:badcomponent"
|
||
|
||
result = await agent.repair(layer="docker-110", component="badcomponent")
|
||
|
||
assert result.success is False
|
||
```
|
||
|
||
- [ ] **Step 11.2: 執行測試確認全部 FAIL（模組未建立）**
|
||
|
||
```bash
|
||
cd apps/api
|
||
python -m pytest tests/test_host_repair_agent.py -v 2>&1 | head -20
|
||
```
|
||
|
||
預期：`ImportError` 或 `ModuleNotFoundError`。
|
||
|
||
- [ ] **Step 11.3: 建立 host_repair_agent.py**
|
||
|
||
建立 `apps/api/src/services/host_repair_agent.py`：
|
||
|
||
```python
|
||
"""
|
||
Host Repair Agent
|
||
=================
|
||
透過 SSH 執行主機層 (Docker/systemd) 修復動作
|
||
|
||
安全設計:
|
||
- SSH key 限制: authorized_keys 的 command= 強制只能執行白名單腳本
|
||
- Component 白名單驗證: 防止 command injection
|
||
- Timeout: 60 秒超時防止卡住
|
||
|
||
Layer 對應:
|
||
docker-110 → SSH wooo@192.168.0.110 → repair-bot-110.sh
|
||
docker-188 → SSH ollama@192.168.0.188 → repair-bot-188.sh
|
||
systemd-188 → SSH ollama@192.168.0.188 → repair-bot-188.sh
|
||
|
||
建立時間: 2026-04-05 (台北時區)
|
||
建立者: Claude Code (Sprint 3 Host Auto-Repair)
|
||
"""
|
||
|
||
import asyncio
|
||
import re
|
||
from dataclasses import dataclass, field
|
||
from pathlib import Path
|
||
|
||
import structlog
|
||
|
||
logger = structlog.get_logger(__name__)
|
||
|
||
# SSH key 路徑 (從 K8s Secret 掛載)
|
||
SSH_KEY_PATH = "/etc/repair-ssh/id_ed25519"
|
||
SSH_TIMEOUT = 60 # 秒
|
||
|
||
# Layer → SSH 配置
|
||
LAYER_SSH_CONFIG: dict[str, dict[str, str]] = {
|
||
"docker-110": {"user": "wooo", "host": "192.168.0.110"},
|
||
"docker-188": {"user": "ollama", "host": "192.168.0.188"},
|
||
"systemd-188": {"user": "ollama", "host": "192.168.0.188"},
|
||
}
|
||
|
||
# 允許的 component 名稱 (白名單)
|
||
VALID_COMPONENT_PATTERN = re.compile(r"^[a-z0-9][a-z0-9-]{0,30}$")
|
||
|
||
|
||
# =============================================================================
|
||
# Data classes
|
||
# =============================================================================
|
||
|
||
@dataclass
|
||
class HostRepairResult:
|
||
"""SSH 修復執行結果"""
|
||
success: bool
|
||
component: str
|
||
layer: str
|
||
output: str = ""
|
||
error: str | None = None
|
||
|
||
|
||
# =============================================================================
|
||
# 純函式 (純邏輯，可單元測試)
|
||
# =============================================================================
|
||
|
||
def get_ssh_config_for_layer(layer: str) -> dict[str, str]:
|
||
"""
|
||
根據 layer 返回 SSH 連線配置
|
||
|
||
Raises:
|
||
ValueError: layer 不支援 SSH 修復
|
||
"""
|
||
if layer == "k8s":
|
||
raise ValueError("layer 'k8s' is handled by kubectl, not SSH")
|
||
config = LAYER_SSH_CONFIG.get(layer)
|
||
if config is None:
|
||
raise ValueError(f"Unknown layer: '{layer}'. Valid: {list(LAYER_SSH_CONFIG.keys())}")
|
||
return config
|
||
|
||
|
||
def build_repair_command(component: str) -> str:
|
||
"""
|
||
組裝修復命令，並驗證 component 格式防止 injection
|
||
|
||
Raises:
|
||
ValueError: component 格式不合法
|
||
"""
|
||
if not VALID_COMPONENT_PATTERN.match(component):
|
||
raise ValueError(f"Invalid component name: '{component}'. Must match [a-z0-9-]+")
|
||
return f"repair:{component}"
|
||
|
||
|
||
# =============================================================================
|
||
# HostRepairAgent
|
||
# =============================================================================
|
||
|
||
class HostRepairAgent:
|
||
"""
|
||
透過 SSH 執行主機修復。
|
||
使用 command= 受限的 SSH key，只能觸發白名單腳本。
|
||
"""
|
||
|
||
async def repair(
|
||
self,
|
||
layer: str,
|
||
component: str,
|
||
timeout: int = SSH_TIMEOUT,
|
||
) -> HostRepairResult:
|
||
"""
|
||
執行修復。
|
||
|
||
Args:
|
||
layer: "docker-110" | "docker-188" | "systemd-188"
|
||
component: 服務名稱，如 "sentry", "harbor"
|
||
timeout: SSH 命令超時秒數
|
||
|
||
Returns:
|
||
HostRepairResult
|
||
"""
|
||
log = logger.bind(layer=layer, component=component)
|
||
log.info("host_repair_start")
|
||
|
||
try:
|
||
config = get_ssh_config_for_layer(layer)
|
||
command = build_repair_command(component)
|
||
except ValueError as e:
|
||
log.error("host_repair_invalid_params", error=str(e))
|
||
return HostRepairResult(
|
||
success=False, component=component, layer=layer, error=str(e)
|
||
)
|
||
|
||
try:
|
||
output = await asyncio.wait_for(
|
||
self._ssh_execute(
|
||
host=config["host"],
|
||
user=config["user"],
|
||
command=command,
|
||
),
|
||
timeout=timeout,
|
||
)
|
||
except asyncio.TimeoutError:
|
||
log.error("host_repair_timeout", timeout=timeout)
|
||
return HostRepairResult(
|
||
success=False,
|
||
component=component,
|
||
layer=layer,
|
||
error=f"SSH command timeout after {timeout}s",
|
||
)
|
||
except Exception as e:
|
||
log.error("host_repair_ssh_error", error=str(e))
|
||
return HostRepairResult(
|
||
success=False, component=component, layer=layer, error=str(e)
|
||
)
|
||
|
||
success = output.startswith("REPAIR_OK:")
|
||
log.info(
|
||
"host_repair_complete",
|
||
success=success,
|
||
output=output[:100],
|
||
)
|
||
|
||
return HostRepairResult(
|
||
success=success,
|
||
component=component,
|
||
layer=layer,
|
||
output=output,
|
||
error=None if success else output,
|
||
)
|
||
|
||
async def health_check(self, layer: str) -> bool:
|
||
"""驗證 SSH 連線是否正常"""
|
||
try:
|
||
config = get_ssh_config_for_layer(layer)
|
||
output = await asyncio.wait_for(
|
||
self._ssh_execute(host=config["host"], user=config["user"], command="health"),
|
||
timeout=10,
|
||
)
|
||
return "REPAIR_BOT_HEALTHY" in output
|
||
except Exception:
|
||
return False
|
||
|
||
async def _ssh_execute(self, host: str, user: str, command: str) -> str:
|
||
"""
|
||
執行 SSH 命令。
|
||
使用 subprocess 呼叫 ssh，key 從 SSH_KEY_PATH 讀取。
|
||
|
||
Note: 使用 subprocess 而非 paramiko，避免引入額外依賴。
|
||
command= 限制確保 SSH 只能執行白名單腳本。
|
||
"""
|
||
key_path = SSH_KEY_PATH
|
||
if not Path(key_path).exists():
|
||
# 開發環境 fallback (測試用)
|
||
raise FileNotFoundError(f"SSH key not found: {key_path}")
|
||
|
||
proc = await asyncio.create_subprocess_exec(
|
||
"ssh",
|
||
"-i", key_path,
|
||
"-o", "StrictHostKeyChecking=accept-new",
|
||
"-o", "ConnectTimeout=10",
|
||
"-o", "BatchMode=yes",
|
||
f"{user}@{host}",
|
||
command,
|
||
stdout=asyncio.subprocess.PIPE,
|
||
stderr=asyncio.subprocess.PIPE,
|
||
)
|
||
|
||
stdout, stderr = await proc.communicate()
|
||
output = stdout.decode().strip()
|
||
if proc.returncode != 0 and not output:
|
||
output = stderr.decode().strip() or f"SSH exit code {proc.returncode}"
|
||
|
||
return output
|
||
|
||
|
||
# =============================================================================
|
||
# Singleton
|
||
# =============================================================================
|
||
|
||
_host_repair_agent: HostRepairAgent | None = None
|
||
|
||
|
||
def get_host_repair_agent() -> HostRepairAgent:
|
||
global _host_repair_agent
|
||
if _host_repair_agent is None:
|
||
_host_repair_agent = HostRepairAgent()
|
||
return _host_repair_agent
|
||
```
|
||
|
||
- [ ] **Step 11.4: 執行測試確認通過**
|
||
|
||
```bash
|
||
cd apps/api
|
||
python -m pytest tests/test_host_repair_agent.py -v
|
||
```
|
||
|
||
預期：所有測試 PASS。
|
||
|
||
- [ ] **Step 11.5: Commit**
|
||
|
||
```bash
|
||
git add apps/api/src/services/host_repair_agent.py apps/api/tests/test_host_repair_agent.py
|
||
git commit -m "feat(api): HostRepairAgent — SSH 主機層修復 Agent
|
||
|
||
透過 command= 受限 SSH key 觸發 110/188 白名單修復腳本
|
||
支援 layer: docker-110 / docker-188 / systemd-188
|
||
component 格式驗證防止 command injection
|
||
|
||
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 12：整合 HostRepairAgent 到 auto_repair_service
|
||
|
||
**Files:**
|
||
- Modify: `apps/api/src/models/playbook.py`
|
||
- Modify: `apps/api/src/services/auto_repair_service.py`
|
||
|
||
- [ ] **Step 12.1: 加入 ActionType.SSH_COMMAND**
|
||
|
||
找到 `apps/api/src/models/playbook.py` 的 ActionType：
|
||
|
||
```python
|
||
class ActionType(str, Enum):
|
||
KUBECTL = "kubectl"
|
||
SCRIPT = "script"
|
||
MANUAL = "manual"
|
||
```
|
||
|
||
改為：
|
||
|
||
```python
|
||
class ActionType(str, Enum):
|
||
KUBECTL = "kubectl"
|
||
SCRIPT = "script"
|
||
MANUAL = "manual"
|
||
SSH_COMMAND = "ssh_command" # 2026-04-05 Claude Code: Sprint 3 主機層修復
|
||
```
|
||
|
||
- [ ] **Step 12.2: 在 auto_repair_service.py 的 _execute_step 加入 SSH_COMMAND 路由**
|
||
|
||
找到 `apps/api/src/services/auto_repair_service.py` 的 `_execute_step` 方法（約 L471-500）：
|
||
|
||
```python
|
||
async def _execute_step(self, incident: Incident, step) -> str:
|
||
...
|
||
if step.action_type == ActionType.MANUAL:
|
||
return "SKIPPED (manual step)"
|
||
|
||
if step.action_type == ActionType.KUBECTL:
|
||
...
|
||
return "UNKNOWN_ACTION_TYPE"
|
||
```
|
||
|
||
在 `return "UNKNOWN_ACTION_TYPE"` 之前加入：
|
||
|
||
```python
|
||
if step.action_type == ActionType.SSH_COMMAND:
|
||
# 2026-04-05 Claude Code: Sprint 3 主機層修復
|
||
# command 格式: "<component>" (由 Playbook 定義)
|
||
# layer 從 incident.labels 取得
|
||
from src.services.host_repair_agent import get_host_repair_agent
|
||
layer = incident.labels.get("layer", "docker-110")
|
||
component = step.command.strip()
|
||
agent = get_host_repair_agent()
|
||
try:
|
||
result = await agent.repair(layer=layer, component=component)
|
||
if result.success:
|
||
return f"SUCCESS (SSH repair:{component} on {layer})"
|
||
else:
|
||
return f"FAILED: {result.error}"
|
||
except Exception as e:
|
||
logger.error("ssh_repair_step_error", error=str(e))
|
||
return f"FAILED: {e}"
|
||
```
|
||
|
||
- [ ] **Step 12.3: 確認測試仍通過**
|
||
|
||
```bash
|
||
cd apps/api
|
||
python -m pytest tests/test_auto_repair_service.py tests/test_host_repair_agent.py -v
|
||
```
|
||
|
||
預期：全部 PASS。
|
||
|
||
- [ ] **Step 12.4: Commit**
|
||
|
||
```bash
|
||
git add apps/api/src/models/playbook.py apps/api/src/services/auto_repair_service.py
|
||
git commit -m "feat(api): auto_repair_service 整合 SSH_COMMAND ActionType
|
||
|
||
_execute_step 加入 SSH_COMMAND 路由:
|
||
- 從 incident.labels['layer'] 決定連線目標
|
||
- 呼叫 HostRepairAgent.repair(layer, component)
|
||
- 修復結果記錄到 executed_steps
|
||
|
||
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 13：SSH Key 掛載到 API Pod
|
||
|
||
**Files:**
|
||
- Modify: `k8s/awoooi-prod/01-deployments.yaml`
|
||
|
||
- [ ] **Step 13.1: 確認 API Deployment 位置**
|
||
|
||
```bash
|
||
grep -n "awoooi-api\|containers:\|volumeMounts:\|volumes:" k8s/awoooi-prod/01-deployments.yaml | head -20
|
||
```
|
||
|
||
- [ ] **Step 13.2: 加入 volume 和 volumeMount**
|
||
|
||
在 API deployment 的 `containers[0]` 下找到 `volumeMounts` 或在 containers 末尾加入：
|
||
|
||
```yaml
|
||
# 2026-04-05 Claude Code: Sprint 3 Host Repair SSH key
|
||
volumeMounts:
|
||
- name: repair-ssh-key
|
||
mountPath: /etc/repair-ssh
|
||
readOnly: true
|
||
```
|
||
|
||
在同一 deployment 的 `spec.template.spec` 下加入 `volumes`：
|
||
|
||
```yaml
|
||
volumes:
|
||
- name: repair-ssh-key
|
||
secret:
|
||
secretName: awoooi-repair-ssh-key
|
||
defaultMode: 0400
|
||
```
|
||
|
||
- [ ] **Step 13.3: 套用到 K3s**
|
||
|
||
```bash
|
||
ssh wooo@192.168.0.120 "kubectl apply -f -" < k8s/awoooi-prod/01-deployments.yaml
|
||
ssh wooo@192.168.0.120 "kubectl rollout status deployment/awoooi-api -n awoooi-prod --timeout=120s"
|
||
```
|
||
|
||
- [ ] **Step 13.4: 驗證 SSH key 已掛載**
|
||
|
||
```bash
|
||
ssh wooo@192.168.0.120 "kubectl exec -n awoooi-prod deploy/awoooi-api -- ls -la /etc/repair-ssh/"
|
||
```
|
||
|
||
預期：看到 `id_ed25519` 和 `id_ed25519.pub`，權限 `400`。
|
||
|
||
- [ ] **Step 13.5: Commit**
|
||
|
||
```bash
|
||
git add k8s/awoooi-prod/01-deployments.yaml
|
||
git commit -m "k8s(api): API pod 掛載 repair SSH key secret
|
||
|
||
掛載 awoooi-repair-ssh-key 到 /etc/repair-ssh
|
||
供 HostRepairAgent 在 Sprint 3 SSH 修復時使用
|
||
|
||
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 14：建立 Host Repair Playbooks
|
||
|
||
**說明：** 透過 AWOOOI API 建立修復 Playbook，讓 auto_repair_service 能匹配並執行。
|
||
|
||
**Files:**
|
||
- Create: `scripts/ops/seed-repair-playbooks.py`
|
||
|
||
- [ ] **Step 14.1: 建立 seed 腳本**
|
||
|
||
建立 `scripts/ops/seed-repair-playbooks.py`：
|
||
|
||
```python
|
||
#!/usr/bin/env python3
|
||
"""
|
||
seed-repair-playbooks.py
|
||
建立 Sprint 3 Host Repair Playbooks
|
||
|
||
用法: python3 scripts/ops/seed-repair-playbooks.py
|
||
需要: AWOOOI API 可訪問 (http://192.168.0.121:32334)
|
||
"""
|
||
import json
|
||
import urllib.request
|
||
import urllib.error
|
||
|
||
API_BASE = "http://192.168.0.121:32334"
|
||
|
||
PLAYBOOKS = [
|
||
{
|
||
"name": "sentry-down-repair",
|
||
"description": "Sentry (110) 離線自動修復",
|
||
"trigger_alert": "SentryDown",
|
||
"symptom_pattern": {
|
||
"keywords": ["SentryDown", "sentry", "9000"],
|
||
"severity": "warning",
|
||
"labels": {"layer": "docker-110", "component": "sentry"},
|
||
},
|
||
"repair_steps": [
|
||
{
|
||
"step_number": 1,
|
||
"action_type": "ssh_command",
|
||
"command": "sentry",
|
||
"description": "SSH 到 110，docker compose up -d Sentry",
|
||
"risk_level": "LOW",
|
||
"timeout_seconds": 60,
|
||
}
|
||
],
|
||
"tags": ["sentry", "docker-110", "auto-repair"],
|
||
"risk_level": "LOW",
|
||
},
|
||
{
|
||
"name": "harbor-down-repair",
|
||
"description": "Harbor Registry (110) 離線自動修復",
|
||
"trigger_alert": "HarborDown",
|
||
"symptom_pattern": {
|
||
"keywords": ["HarborDown", "harbor", "5000", "ImagePullBackOff"],
|
||
"severity": "critical",
|
||
"labels": {"layer": "docker-110", "component": "harbor"},
|
||
},
|
||
"repair_steps": [
|
||
{
|
||
"step_number": 1,
|
||
"action_type": "ssh_command",
|
||
"command": "harbor",
|
||
"description": "SSH 到 110，docker compose up -d Harbor",
|
||
"risk_level": "LOW",
|
||
"timeout_seconds": 120,
|
||
}
|
||
],
|
||
"tags": ["harbor", "docker-110", "auto-repair", "registry"],
|
||
"risk_level": "LOW",
|
||
},
|
||
{
|
||
"name": "gitea-down-repair",
|
||
"description": "Gitea (110) 離線自動修復",
|
||
"trigger_alert": "GiteaDown",
|
||
"symptom_pattern": {
|
||
"keywords": ["GiteaDown", "gitea", "3001"],
|
||
"severity": "critical",
|
||
"labels": {"layer": "docker-110", "component": "gitea"},
|
||
},
|
||
"repair_steps": [
|
||
{
|
||
"step_number": 1,
|
||
"action_type": "ssh_command",
|
||
"command": "gitea",
|
||
"description": "SSH 到 110，docker compose up -d Gitea",
|
||
"risk_level": "LOW",
|
||
"timeout_seconds": 60,
|
||
}
|
||
],
|
||
"tags": ["gitea", "docker-110", "auto-repair"],
|
||
"risk_level": "LOW",
|
||
},
|
||
{
|
||
"name": "alertmanager-down-repair",
|
||
"description": "Alertmanager (110) 離線自動修復",
|
||
"trigger_alert": "AlertmanagerDown",
|
||
"symptom_pattern": {
|
||
"keywords": ["AlertmanagerDown", "alertmanager", "9093"],
|
||
"severity": "critical",
|
||
"labels": {"layer": "docker-110", "component": "alertmanager"},
|
||
},
|
||
"repair_steps": [
|
||
{
|
||
"step_number": 1,
|
||
"action_type": "ssh_command",
|
||
"command": "alertmanager",
|
||
"description": "SSH 到 110，docker compose up -d monitoring (含 Alertmanager)",
|
||
"risk_level": "LOW",
|
||
"timeout_seconds": 60,
|
||
}
|
||
],
|
||
"tags": ["alertmanager", "docker-110", "auto-repair", "critical-infra"],
|
||
"risk_level": "LOW",
|
||
},
|
||
{
|
||
"name": "openclaw-down-repair",
|
||
"description": "OpenClaw (188) 離線自動修復",
|
||
"trigger_alert": "OpenClawDown",
|
||
"symptom_pattern": {
|
||
"keywords": ["OpenClawDown", "openclaw", "8088", "clawbot"],
|
||
"severity": "critical",
|
||
"labels": {"layer": "docker-188", "component": "openclaw"},
|
||
},
|
||
"repair_steps": [
|
||
{
|
||
"step_number": 1,
|
||
"action_type": "ssh_command",
|
||
"command": "openclaw",
|
||
"description": "SSH 到 188，docker compose up -d OpenClaw",
|
||
"risk_level": "LOW",
|
||
"timeout_seconds": 90,
|
||
}
|
||
],
|
||
"tags": ["openclaw", "docker-188", "auto-repair"],
|
||
"risk_level": "LOW",
|
||
},
|
||
]
|
||
|
||
|
||
def create_playbook(playbook_data: dict) -> bool:
|
||
"""透過 API 建立 Playbook"""
|
||
data = json.dumps(playbook_data).encode()
|
||
req = urllib.request.Request(
|
||
f"{API_BASE}/api/v1/playbooks",
|
||
data=data,
|
||
headers={"Content-Type": "application/json"},
|
||
method="POST",
|
||
)
|
||
try:
|
||
with urllib.request.urlopen(req, timeout=10) as resp:
|
||
result = json.loads(resp.read())
|
||
print(f" ✅ Created: {playbook_data['name']} (id: {result.get('playbook_id', '?')})")
|
||
return True
|
||
except urllib.error.HTTPError as e:
|
||
body = e.read().decode()
|
||
if "already exists" in body or e.code == 409:
|
||
print(f" ⚠️ Already exists: {playbook_data['name']}")
|
||
return True
|
||
print(f" ❌ Failed: {playbook_data['name']} — HTTP {e.code}: {body[:100]}")
|
||
return False
|
||
except Exception as e:
|
||
print(f" ❌ Error: {playbook_data['name']} — {e}")
|
||
return False
|
||
|
||
|
||
if __name__ == "__main__":
|
||
print("=== 建立 Host Repair Playbooks ===")
|
||
success = 0
|
||
for pb in PLAYBOOKS:
|
||
if create_playbook(pb):
|
||
success += 1
|
||
print(f"\n結果: {success}/{len(PLAYBOOKS)} playbooks 建立成功")
|
||
```
|
||
|
||
- [ ] **Step 14.2: 執行 seed 腳本**
|
||
|
||
```bash
|
||
python3 scripts/ops/seed-repair-playbooks.py
|
||
```
|
||
|
||
預期輸出：所有 Playbook ✅ Created 或 ⚠️ Already exists。
|
||
|
||
- [ ] **Step 14.3: 確認 Playbooks 已建立**
|
||
|
||
```bash
|
||
curl -s http://192.168.0.121:32334/api/v1/playbooks | python3 -c "
|
||
import sys, json
|
||
pbs = json.load(sys.stdin)
|
||
for pb in pbs.get('playbooks', []):
|
||
print(pb['name'], pb.get('risk_level', '?'))
|
||
"
|
||
```
|
||
|
||
- [ ] **Step 14.4: Commit**
|
||
|
||
```bash
|
||
git add scripts/ops/seed-repair-playbooks.py
|
||
git commit -m "ops(scripts): seed-repair-playbooks.py — Sprint 3 Host Repair Playbooks
|
||
|
||
5 個 Playbooks: SentryDown/HarborDown/GiteaDown/AlertmanagerDown/OpenClawDown
|
||
全部 LOW 風險，layer: docker-110/docker-188
|
||
ActionType: ssh_command → HostRepairAgent
|
||
|
||
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 15：Sprint 3 E2E 閉環驗證
|
||
|
||
- [ ] **Step 15.1: 驗證 SSH 連線 (從 API Pod)**
|
||
|
||
```bash
|
||
ssh wooo@192.168.0.120 "kubectl exec -n awoooi-prod deploy/awoooi-api -- ssh -i /etc/repair-ssh/id_ed25519 -o StrictHostKeyChecking=accept-new -o BatchMode=yes wooo@192.168.0.110 health"
|
||
```
|
||
|
||
預期：`REPAIR_BOT_HEALTHY:110`
|
||
|
||
- [ ] **Step 15.2: SentryDown E2E 測試**
|
||
|
||
```bash
|
||
# 1. 手動停止 Sentry
|
||
ssh wooo@192.168.0.110 "cd /opt/sentry && docker compose stop"
|
||
|
||
# 2. 等待 Prometheus 偵測 (SentryDown for: 2m)
|
||
echo "等待 2 分鐘讓 Prometheus 觸發 SentryDown..."
|
||
sleep 130
|
||
|
||
# 3. 確認 Prometheus 告警狀態
|
||
ssh wooo@192.168.0.110 "curl -s 'http://localhost:9090/api/v1/alerts' | python3 -c \"
|
||
import sys, json
|
||
alerts = json.load(sys.stdin)
|
||
sentry_alerts = [a for a in alerts['data']['alerts'] if 'SentryDown' in a['labels'].get('alertname','')]
|
||
for a in sentry_alerts:
|
||
print(a['labels']['alertname'], a['state'])
|
||
\""
|
||
```
|
||
|
||
預期：看到 `SentryDown firing`
|
||
|
||
- [ ] **Step 15.3: 確認 Alertmanager 已送出 webhook**
|
||
|
||
```bash
|
||
ssh wooo@192.168.0.110 "curl -s http://localhost:9093/api/v2/alerts | python3 -c \"
|
||
import sys, json
|
||
alerts = json.load(sys.stdin)
|
||
for a in alerts:
|
||
if 'SentryDown' in str(a.get('labels',{})):
|
||
print('Found SentryDown alert in Alertmanager:', a.get('status',{}).get('state','?'))
|
||
\""
|
||
```
|
||
|
||
- [ ] **Step 15.4: 確認 auto_repair_service 執行了 SSH 修復**
|
||
|
||
```bash
|
||
# 查看 API pod logs
|
||
ssh wooo@192.168.0.120 "kubectl logs -n awoooi-prod deploy/awoooi-api --since=5m | grep -E 'host_repair|ssh_repair|SentryDown|auto_repair'"
|
||
```
|
||
|
||
預期：看到 `host_repair_start`、`host_repair_complete`、`success=True`
|
||
|
||
- [ ] **Step 15.5: 確認 Sentry 已恢復**
|
||
|
||
```bash
|
||
sleep 30 # 等待 docker compose up -d 完成
|
||
curl -s -o /dev/null -w "%{http_code}" http://192.168.0.110:9000/
|
||
```
|
||
|
||
預期：`200`、`302` 或 `400`（服務已恢復）
|
||
|
||
- [ ] **Step 15.6: 確認 Telegram 收到告警和 resolved**
|
||
|
||
人工確認 Telegram 收到：
|
||
1. `SentryDown` 告警卡片
|
||
2. 自動修復成功通知（若 auto_repair 成功後有 resolved webhook）
|
||
|
||
- [ ] **Step 15.7: 最終驗收 checklist**
|
||
|
||
```bash
|
||
# 完整驗收腳本
|
||
cat << 'EOF' > /tmp/final-e2e.sh
|
||
#!/bin/bash
|
||
GREEN='\033[0;32m'; RED='\033[0;31m'; NC='\033[0m'
|
||
PASS=0; FAIL=0
|
||
|
||
check() {
|
||
local name="$1"; local cmd="$2"; local expect="$3"
|
||
result=$(eval "$cmd" 2>/dev/null)
|
||
if echo "$result" | grep -qE "$expect"; then
|
||
echo -e "${GREEN}✅ $name${NC}"
|
||
((PASS++))
|
||
else
|
||
echo -e "${RED}❌ $name${NC} (got: ${result:0:60})"
|
||
((FAIL++))
|
||
fi
|
||
}
|
||
|
||
echo "=== 全系統自愈閉環 E2E 驗收 ==="
|
||
|
||
# Sprint 1
|
||
check "Prometheus rules >30" "ssh wooo@192.168.0.110 'curl -s http://localhost:9090/api/v1/rules | python3 -c \"import sys,json; r=json.load(sys.stdin); print(sum(len(g[chr(39)rules chr(39)]) for g in r[chr(39)data chr(39)][chr(39)groups chr(39)]))\"'" "[3-9][0-9]"
|
||
check "SentryDown rule exists" "ssh wooo@192.168.0.110 'curl -s http://localhost:9090/api/v1/rules | python3 -c \"import sys,json; r=json.load(sys.stdin); names=[x[chr(39)name chr(39)] for g in r[chr(39)data chr(39)][chr(39)groups chr(39)] for x in g[chr(39)rules chr(39)]]; print(chr(39)OK chr(39) if chr(39)SentryDown chr(39) in names else chr(39)MISS chr(39))\"'" "OK"
|
||
check "110 Sentry running" "curl -s -o /dev/null -w '%{http_code}' --max-time 10 http://192.168.0.110:9000/" "200|302|400"
|
||
check "Alert chain E2E" "curl -s -X POST http://192.168.0.121:32334/api/v1/webhooks/alertmanager -H 'Content-Type: application/json' -d '{\"receiver\":\"test\",\"status\":\"firing\",\"alerts\":[{\"status\":\"firing\",\"labels\":{\"alertname\":\"FinalE2ETest\",\"severity\":\"info\"},\"annotations\":{\"summary\":\"Final E2E test\"},\"startsAt\":\"2026-04-05T00:00:00Z\",\"endsAt\":\"0001-01-01T00:00:00Z\",\"generatorURL\":\"\"}],\"groupLabels\":{},\"commonLabels\":{},\"commonAnnotations\":{},\"externalURL\":\"\",\"version\":\"4\",\"groupKey\":\"e2e\"}'" '"success":true'
|
||
|
||
# Sprint 3
|
||
check "SSH key mounted in API pod" "ssh wooo@192.168.0.120 'kubectl exec -n awoooi-prod deploy/awoooi-api -- ls /etc/repair-ssh/id_ed25519'" "id_ed25519"
|
||
check "SSH health 110" "ssh wooo@192.168.0.120 'kubectl exec -n awoooi-prod deploy/awoooi-api -- ssh -i /etc/repair-ssh/id_ed25519 -o StrictHostKeyChecking=accept-new -o BatchMode=yes wooo@192.168.0.110 health'" "REPAIR_BOT_HEALTHY"
|
||
|
||
echo ""
|
||
echo "=== 結果: ${PASS} 通過, ${FAIL} 失敗 ==="
|
||
[ $FAIL -eq 0 ] && echo -e "${GREEN}🎉 全系統自愈閉環驗收通過！${NC}" || echo -e "${RED}⚠️ ${FAIL} 項失敗${NC}"
|
||
EOF
|
||
bash /tmp/final-e2e.sh
|
||
```
|
||
|
||
---
|
||
|
||
## Self-Review
|
||
|
||
### Spec 覆蓋確認
|
||
|
||
| Spec 章節 | 對應 Task |
|
||
|----------|---------|
|
||
| S0 統一標籤規範 | Task 1 (alerts-unified.yml 內建標籤) |
|
||
| Sprint 1: Prometheus 規則部署 | Task 1 + Task 2 |
|
||
| Sprint 1: CD 自動同步 | Task 3 |
|
||
| Sprint 1: Sentry 啟動 | Task 4 |
|
||
| SOP v4.0 更新 | Task 5 |
|
||
| Sprint 1 驗收 | Task 6 |
|
||
| Sprint 2: SigNoz log alert | Task 7 |
|
||
| Sprint 2: Sentry SDK tags | Task 8 |
|
||
| Sprint 3: 修復腳本白名單 | Task 9 |
|
||
| Sprint 3: SSH key 基礎設施 | Task 10 |
|
||
| Sprint 3: HostRepairAgent | Task 11 |
|
||
| Sprint 3: auto_repair 整合 | Task 12 |
|
||
| Sprint 3: Pod 掛載 SSH key | Task 13 |
|
||
| Sprint 3: Playbooks | Task 14 |
|
||
| Sprint 3: E2E 閉環驗證 | Task 15 |
|
||
|
||
所有 spec 章節都有對應 Task。✅
|
||
|
||
### 型別一致性
|
||
|
||
- `HostRepairResult` 在 Task 11 定義，Task 12 使用 `result.success` / `result.error` ✅
|
||
- `ActionType.SSH_COMMAND` 在 Task 12 定義，Task 14 的 Playbook seed 使用 `"ssh_command"` ✅
|
||
- `get_host_repair_agent()` 在 Task 11 定義，Task 12 使用 ✅
|