# 全系統自愈閉環 Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** 修復三大系統性缺口：Prometheus 規則未部署 → 告警靜默；日誌只收集不告警；auto_repair 無法修復主機 Docker 層服務。 **Architecture:** 方案 A 漸進補強。Sprint 1 補 Prometheus 規則 + 啟動 Sentry + CD 同步；Sprint 2 加 SigNoz log alert；Sprint 3 新增 HostRepairAgent (SSH) 讓 auto_repair_service 能修復 110/188 的 Docker/systemd 服務。 **Tech Stack:** Prometheus YAML、Gitea Actions、Python asyncio/asyncssh、K8s Secret、SigNoz webhook、awoooi-startup-110.sh bash --- ## 檔案結構 ### Sprint 1 (Infra/Ops) | 檔案 | 動作 | 說明 | |------|------|------| | `ops/monitoring/alerts-unified.yml` | **建立** | 整合所有 Prometheus 規則 (40+條)，含統一標籤 | | `scripts/ops/deploy-alerts.sh` | **建立** | 部署 alerts.yml 到 110 Prometheus 的腳本 | | `.gitea/workflows/cd.yaml` | **修改** | 加入 `deploy-alerts` job，paths trigger 含 ops/monitoring/ | | `scripts/reboot-recovery/awoooi-startup-110.sh` | **修改** | 加入 Step 7 啟動 Sentry (/opt/sentry) | | `docs/runbooks/REBOOT-RECOVERY-SOP.md` | **修改** | v4.0：Sentry 步驟、診斷樹更新、E2E 驗證腳本更新 | ### Sprint 2 (SigNoz + Sentry) | 檔案 | 動作 | 說明 | |------|------|------| | `ops/signoz/alerting/log-rules.md` | **建立** | SigNoz log alert rules 文檔 (SigNoz 無 YAML 匯入) | | `apps/api/src/main.py` | **修改** | Sentry init 加入 layer/component tags | ### Sprint 3 (Host Auto-Repair) | 檔案 | 動作 | 說明 | |------|------|------| | `apps/api/src/services/host_repair_agent.py` | **建立** | SSH 主機修復 Agent | | `apps/api/src/models/playbook.py` | **修改** | 加入 `ActionType.SSH_COMMAND` | | `apps/api/src/services/auto_repair_service.py` | **修改** | `_execute_step` 加入 SSH_COMMAND 路由 | | `scripts/repair-bot/repair-bot-110.sh` | **建立** | 110 主機白名單修復腳本 | | `scripts/repair-bot/repair-bot-188.sh` | **建立** | 188 主機白名單修復腳本 | | `k8s/awoooi-prod/04-repair-ssh-secret.yaml` | **建立** | SSH key K8s Secret template (不含實際 key) | | `apps/api/tests/test_host_repair_agent.py` | **建立** | HostRepairAgent 單元測試 | --- ## SPRINT 1：Prometheus 規則部署 + Sentry 啟動 + CD 同步 --- ### Task 1：建立統一 Prometheus 規則檔 **說明：** 將 `k8s/monitoring/` 下所有規則轉換為 Docker Prometheus 格式，合併為 `ops/monitoring/alerts-unified.yml`。加入 `layer`/`host`/`auto_repair` 統一標籤。 **Files:** - Create: `ops/monitoring/alerts-unified.yml` - [ ] **Step 1.1: 確認來源規則** ```bash # 在 awoooi 專案根目錄執行 grep -c "alert:" k8s/monitoring/k3s-alerts.yaml k8s/monitoring/alert-chain-monitor.yaml k8s/monitoring/database-alerts.yaml k8s/monitoring/minio-kali-alerts.yaml k8s/monitoring/k3s-alerts-supplemental.yaml ``` 預期輸出：每個檔案各顯示規則數，總計應 > 40。 - [ ] **Step 1.2: 建立 alerts-unified.yml** 建立 `ops/monitoring/alerts-unified.yml`，內容如下（完整）： ```yaml # ops/monitoring/alerts-unified.yml # AWOOOI 統一 Prometheus 告警規則 # 2026-04-05 Claude Code: 整合所有規則，加入統一 layer 標籤 # 部署目標: 192.168.0.110:/home/wooo/monitoring/alerts.yml # 部署方式: scripts/ops/deploy-alerts.sh (CD 自動部署) # # 標籤規範: # layer: k8s | docker-110 | docker-188 | systemd-188 # component: 服務名稱 # team: ops | backend | ai | platform # host: "110" | "188" | "120" | "121" # auto_repair: "true" | "false" groups: # ========================================================================= # 主機層告警 (host_alerts) # ========================================================================= - name: host_alerts rules: - alert: HostDown expr: up{job=~"node-exporter.*"} == 0 for: 1m labels: severity: critical layer: systemd-188 team: ops auto_repair: "false" annotations: summary: "主機 {{ $labels.host }} 不可達" description: "Node Exporter 無回應超過 1 分鐘" - alert: HostHighCpuLoad expr: 100 - (avg by(host) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80 for: 5m labels: severity: warning layer: systemd-188 team: ops auto_repair: "false" annotations: summary: "主機 {{ $labels.host }} CPU 高負載" description: "CPU 使用率超過 80%" - alert: HostOutOfMemory expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85 for: 5m labels: severity: warning layer: systemd-188 team: ops auto_repair: "false" annotations: summary: "主機 {{ $labels.host }} 記憶體不足" description: "記憶體使用率超過 85%" - alert: HostOutOfDiskSpace expr: (1 - (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"})) * 100 > 85 for: 5m labels: severity: warning layer: systemd-188 team: ops auto_repair: "false" annotations: summary: "主機 {{ $labels.host }} 磁碟空間不足" description: "磁碟使用率超過 85%" # ========================================================================= # K8s 叢集告警 (kubernetes_alerts) # ========================================================================= - name: kubernetes_alerts rules: - alert: K3sNodeNotReady expr: kube_node_status_condition{condition="Ready", status="true"} == 0 for: 2m labels: severity: critical layer: k8s team: ops auto_repair: "false" annotations: summary: "K3s 節點 {{ $labels.node }} 未就緒" description: "節點超過 2 分鐘未達到 Ready 狀態" - alert: KubePodCrashLooping expr: rate(kube_pod_container_status_restarts_total{namespace="awoooi-prod"}[15m]) > 0 for: 5m labels: severity: warning layer: k8s team: ops auto_repair: "true" annotations: summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} 持續重啟" description: "Pod 在過去 15 分鐘內重啟次數異常" - alert: KubePodNotReady expr: kube_pod_status_ready{condition="true",namespace="awoooi-prod"} == 0 for: 5m labels: severity: warning layer: k8s team: ops auto_repair: "true" annotations: summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} 未就緒" description: "Running 中的 Pod 超過 5 分鐘未達到 Ready 狀態" - alert: KubeDeploymentReplicasMismatch expr: kube_deployment_spec_replicas{namespace="awoooi-prod"} != kube_deployment_status_replicas_available{namespace="awoooi-prod"} for: 10m labels: severity: warning layer: k8s team: ops auto_repair: "true" annotations: summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} 副本數不匹配" description: "期望副本數與可用副本數不一致超過 10 分鐘" - alert: VeleroBackupFailed expr: increase(velero_backup_failure_total[24h]) > 0 for: 5m labels: severity: warning layer: k8s team: ops component: velero auto_repair: "false" annotations: summary: "Velero 備份失敗" description: "過去 24 小時有備份失敗" - alert: VeleroBackupNotRun expr: time() - velero_backup_last_successful_timestamp > 86400 for: 10m labels: severity: critical layer: k8s team: ops component: velero auto_repair: "false" annotations: summary: "Velero 超過 24 小時未成功備份" description: "最後一次成功備份超過 24 小時" # ========================================================================= # 資料庫告警 (database_alerts) # ========================================================================= - name: database_alerts rules: - alert: PostgreSQLDown expr: up{job="postgres-exporter"} == 0 or pg_up == 0 for: 1m labels: severity: critical layer: systemd-188 component: postgres host: "188" team: ops auto_repair: "false" annotations: summary: "PostgreSQL 資料庫離線" description: "PostgreSQL Exporter 無法連接資料庫超過 1 分鐘" - alert: RedisDown expr: up{job="redis-exporter"} == 0 or redis_up == 0 for: 1m labels: severity: critical layer: systemd-188 component: redis host: "188" team: ops auto_repair: "false" annotations: summary: "Redis 快取服務離線" description: "Redis Exporter 無法連接 Redis 超過 1 分鐘" - alert: PostgreSQLHighConnections expr: pg_stat_activity_count > 80 for: 5m labels: severity: warning layer: systemd-188 component: postgres team: ops auto_repair: "false" annotations: summary: "PostgreSQL 連接數過高" description: "當前連接數 {{ $value }} 超過 80" - alert: RedisMemoryHigh expr: redis_memory_used_bytes / redis_memory_max_bytes * 100 > 80 for: 5m labels: severity: warning layer: systemd-188 component: redis team: ops auto_repair: "false" annotations: summary: "Redis 記憶體使用過高" description: "Redis 記憶體使用率超過 80%" # ========================================================================= # 服務可用性告警 (service_alerts) — 含 layer 標籤決定修復路徑 # ========================================================================= - name: service_alerts rules: # ---- 188 Docker 層 ---- - alert: OpenClawDown # 2026-04-05 Claude Code: 修正舊命名 ClawBotDown → OpenClawDown expr: up{job="clawbot"} == 0 for: 2m labels: severity: critical layer: docker-188 component: openclaw host: "188" team: ops auto_repair: "true" annotations: summary: "OpenClaw 服務離線" description: "OpenClaw (192.168.0.188:8088) 已離線超過 2 分鐘" - alert: SignOzDown expr: probe_success{job="blackbox-http", instance=~".*3301.*"} == 0 for: 2m labels: severity: warning layer: docker-188 component: signoz host: "188" team: ops auto_repair: "true" annotations: summary: "SignOz 服務離線" description: "SignOz (192.168.0.188:3301) 已離線超過 2 分鐘" # ---- 110 Docker 層 ---- - alert: SentryDown expr: probe_success{job="blackbox-tcp", instance="192.168.0.110:9000"} == 0 for: 2m labels: severity: warning layer: docker-110 component: sentry host: "110" team: ops auto_repair: "true" annotations: summary: "Sentry 服務離線" description: "Sentry (192.168.0.110:9000) 已離線超過 2 分鐘" - alert: HarborDown expr: probe_success{job="blackbox-http", instance=~".*5000.*"} == 0 for: 2m labels: severity: critical layer: docker-110 component: harbor host: "110" team: ops auto_repair: "true" annotations: summary: "Harbor Registry 離線" description: "Harbor (192.168.0.110:5000) 已離線超過 2 分鐘，CD pipeline 將無法拉取映像" - alert: GiteaDown expr: probe_success{job="blackbox-http", instance="http://192.168.0.110:3001"} == 0 for: 2m labels: severity: critical layer: docker-110 component: gitea host: "110" team: ops auto_repair: "true" annotations: summary: "Gitea Git 服務離線" description: "Gitea (192.168.0.110:3001) 已離線超過 2 分鐘，CD pipeline 失效" - alert: AlertmanagerDown expr: probe_success{job="blackbox-tcp", instance="192.168.0.110:9093"} == 0 for: 2m labels: severity: critical layer: docker-110 component: alertmanager host: "110" team: ops auto_repair: "true" annotations: summary: "Alertmanager 離線" description: "Alertmanager (192.168.0.110:9093) 已離線，所有告警將靜默" # ========================================================================= # 告警鏈路監控 (alert_chain) — 防止 2026-03-26/04-05 事故重演 # ========================================================================= - name: alert_chain rules: - alert: AlertChainBroken_Alertmanager expr: | sum(rate(awoooi_webhook_requests_total{source="alertmanager",status!="success"}[5m])) / sum(rate(awoooi_webhook_requests_total{source="alertmanager"}[5m])) > 0.1 for: 10m labels: severity: critical layer: k8s team: platform auto_repair: "false" annotations: summary: "Alertmanager Webhook 錯誤率 > 10%" description: "告警鏈路可能斷裂，請執行 E2E 驗證" - alert: AlertChainBroken_Sentry expr: | sum(rate(awoooi_webhook_requests_total{source="sentry",status!="success"}[5m])) / sum(rate(awoooi_webhook_requests_total{source="sentry"}[5m])) > 0.1 for: 10m labels: severity: warning layer: k8s team: platform auto_repair: "false" annotations: summary: "Sentry Webhook 錯誤率 > 10%" description: "Sentry 錯誤可能無法正確處理" - alert: NoAlertsReceived2Hours expr: time() - max by (source)(awoooi_alert_chain_last_success_timestamp) > 7200 for: 5m labels: severity: warning layer: k8s team: platform auto_repair: "false" annotations: summary: "2 小時內未收到任何告警 ({{ $labels.source }})" description: "可能是告警鏈路問題，請執行 Smoke Test" - alert: AlertChainUnhealthy expr: awoooi_alert_chain_healthy == 0 for: 5m labels: severity: critical layer: k8s team: platform auto_repair: "false" annotations: summary: "告警鏈路不健康 ({{ $labels.source }})" description: "告警鏈路標記為不健康，最近處理失敗" # ========================================================================= # 自動修復監控 (auto_repair) # ========================================================================= - name: auto_repair rules: - alert: AutoRepairLowSuccessRate expr: awoooi_auto_repair_success_rate < 0.3 for: 30m labels: severity: warning layer: k8s team: backend auto_repair: "false" annotations: summary: "自動修復成功率過低 ({{ $value | humanizePercentage }})" description: "動作 {{ $labels.action }} 的成功率低於 30%，建議檢查 Playbook" - alert: PermanentFixRequired expr: sum(rate(awoooi_anomaly_escalation_total{level="PERMANENT_FIX"}[1h])) > 0 for: 1m labels: severity: critical layer: k8s team: backend auto_repair: "false" annotations: summary: "需要永久修復的異常升級" description: "有異常升級到 PERMANENT_FIX 級別，需要根本修復" # ========================================================================= # MinIO / Kali 告警 # ========================================================================= - name: minio_kali_alerts rules: - alert: MinIODown expr: probe_success{job="blackbox-http", instance=~".*9000.*", instance!~".*sentry.*"} == 0 for: 2m labels: severity: warning layer: docker-188 component: minio host: "188" team: ops auto_repair: "true" annotations: summary: "MinIO (Velero 備份) 離線" description: "MinIO (192.168.0.188:9000) 已離線超過 2 分鐘，Velero 備份可能失敗" - alert: KaliScannerDown expr: probe_success{job="blackbox-http", instance=~".*192.168.0.112.*"} == 0 for: 5m labels: severity: info layer: docker-188 component: kali host: "112" team: ops auto_repair: "false" annotations: summary: "Kali Scanner 離線" description: "Kali (192.168.0.112:8080) 離線，安全掃描功能暫停" ``` - [ ] **Step 1.3: 驗證 YAML 語法** ```bash cd /Users/ogt/awoooi python3 -c "import yaml; yaml.safe_load(open('ops/monitoring/alerts-unified.yml'))" && echo "YAML OK" ``` 預期輸出：`YAML OK` - [ ] **Step 1.4: Commit** ```bash git add ops/monitoring/alerts-unified.yml git commit -m "ops(monitoring): 統一 Prometheus 告警規則 — 40+條含統一 layer 標籤 修正: - ClawBotDown → OpenClawDown (舊命名廢棄) - 加入 SentryDown/HarborDown/GiteaDown/AlertmanagerDown - 所有規則補齊 layer/component/host/auto_repair 統一標籤 - 整合 k8s/monitoring/*.yaml → ops/monitoring/alerts-unified.yml Co-Authored-By: Claude Sonnet 4.6 " ``` --- ### Task 2：部署規則到 110 Prometheus **Files:** - Create: `scripts/ops/deploy-alerts.sh` - [ ] **Step 2.1: 建立部署腳本** ```bash mkdir -p scripts/ops ``` 建立 `scripts/ops/deploy-alerts.sh`： ```bash #!/bin/bash # scripts/ops/deploy-alerts.sh # 部署統一告警規則到 110 Prometheus # 2026-04-05 Claude Code: Sprint 1 自動化部署 # 用法: bash scripts/ops/deploy-alerts.sh [--dry-run] set -eo pipefail RULES_FILE="ops/monitoring/alerts-unified.yml" TARGET_HOST="192.168.0.110" TARGET_PATH="/home/wooo/monitoring/alerts.yml" PROMETHEUS_URL="http://${TARGET_HOST}:9090" DRY_RUN="${1:-}" log() { echo "[$(date '+%H:%M:%S')] $*"; } # 確認檔案存在 if [ ! -f "$RULES_FILE" ]; then echo "ERROR: $RULES_FILE not found" exit 1 fi # 驗證 YAML 語法 python3 -c "import yaml; yaml.safe_load(open('$RULES_FILE'))" || { echo "ERROR: YAML syntax error"; exit 1; } log "✅ YAML 語法驗證通過" # Dry run 模式 if [ "$DRY_RUN" = "--dry-run" ]; then log "DRY RUN: would deploy $RULES_FILE to ${TARGET_HOST}:${TARGET_PATH}" RULE_COUNT=$(grep -c "alert:" "$RULES_FILE") log "規則數量: $RULE_COUNT 條" exit 0 fi # 備份現有規則 ssh wooo@${TARGET_HOST} "cp ${TARGET_PATH} ${TARGET_PATH}.bak.\$(date +%Y%m%d%H%M%S) 2>/dev/null || true" log "✅ 現有規則已備份" # 部署新規則 scp "$RULES_FILE" wooo@${TARGET_HOST}:${TARGET_PATH} log "✅ 規則已複製到 ${TARGET_HOST}" # Reload Prometheus ssh wooo@${TARGET_HOST} "curl -s -X POST ${PROMETHEUS_URL}/-/reload" sleep 3 # 驗證規則數量 RULE_COUNT=$(ssh wooo@${TARGET_HOST} "curl -s ${PROMETHEUS_URL}/api/v1/rules | python3 -c \"import sys,json; r=json.load(sys.stdin); print(sum(len(g['rules']) for g in r['data']['groups']))\"") log "Prometheus 已載入 ${RULE_COUNT} 條規則" if [ "$RULE_COUNT" -lt 30 ]; then echo "ERROR: 規則數量異常 ($RULE_COUNT < 30)，請檢查" exit 1 fi # 驗證關鍵規則存在 KEY_RULES=("SentryDown" "HarborDown" "GiteaDown" "OpenClawDown" "AlertmanagerDown" "AlertChainUnhealthy") for rule in "${KEY_RULES[@]}"; do EXISTS=$(ssh wooo@${TARGET_HOST} "curl -s ${PROMETHEUS_URL}/api/v1/rules | python3 -c \"import sys,json; r=json.load(sys.stdin); names=[x['name'] for g in r['data']['groups'] for x in g['rules']]; print('OK' if '$rule' in names else 'MISSING')\"") if [ "$EXISTS" = "OK" ]; then log "✅ $rule" else echo "❌ $rule 未找到" exit 1 fi done log "🎉 部署完成！所有關鍵規則已生效" ``` ```bash chmod +x scripts/ops/deploy-alerts.sh ``` - [ ] **Step 2.2: 執行部署 (dry-run 先確認)** ```bash bash scripts/ops/deploy-alerts.sh --dry-run ``` 預期輸出： ``` [HH:MM:SS] ✅ YAML 語法驗證通過 [HH:MM:SS] DRY RUN: would deploy ops/monitoring/alerts-unified.yml to 192.168.0.110:... [HH:MM:SS] 規則數量: XX 條 ``` - [ ] **Step 2.3: 執行實際部署** ```bash bash scripts/ops/deploy-alerts.sh ``` 預期輸出：所有規則 ✅，`🎉 部署完成！` - [ ] **Step 2.4: 驗證 Prometheus UI** ```bash ssh wooo@192.168.0.110 "curl -s http://localhost:9090/api/v1/rules | python3 -c \" import sys, json r = json.load(sys.stdin) groups = r['data']['groups'] total = sum(len(g['rules']) for g in groups) print(f'Groups: {len(groups)}, Total rules: {total}') for g in groups: print(f' {g[\"name\"]}: {len(g[\"rules\"])} rules') \"" ``` 預期：6+ groups，總規則數 > 30。 - [ ] **Step 2.5: Commit** ```bash git add scripts/ops/deploy-alerts.sh git commit -m "ops(scripts): 加入 deploy-alerts.sh 自動部署 Prometheus 規則 Co-Authored-By: Claude Sonnet 4.6 " ``` --- ### Task 3：CD pipeline 自動同步告警規則 **說明：** 修改 `.gitea/workflows/cd.yaml`，加入 `deploy-alerts` job。當 `ops/monitoring/alerts-unified.yml` 有變更時，自動部署到 110。 **Files:** - Modify: `.gitea/workflows/cd.yaml` - [ ] **Step 3.1: 確認 CD 文件末尾** ```bash tail -20 .gitea/workflows/cd.yaml ``` 記錄最後一個 job 的名稱（通常是 build-and-deploy）。 - [ ] **Step 3.2: 加入 deploy-alerts job** 在 `.gitea/workflows/cd.yaml` 的 `on.push.paths` 區塊加入： 找到： ```yaml paths: # 只有實際影響部署的程式碼才觸發 CD - 'apps/**' - 'k8s/**' - '.gitea/workflows/**' # docs/、memory/、ADR、ops/ 等不觸發 ``` 改為： ```yaml paths: # 只有實際影響部署的程式碼才觸發 CD - 'apps/**' - 'k8s/**' - '.gitea/workflows/**' - 'ops/monitoring/alerts-unified.yml' # 2026-04-05 Claude Code: 告警規則變更自動部署 ``` 在 jobs 末尾加入新 job（在 `build-and-deploy:` 同層）： ```yaml deploy-alerts: name: "Deploy Prometheus Alert Rules" # 2026-04-05 Claude Code: 告警規則 CD — 不依賴 build-and-deploy # 觸發條件: ops/monitoring/alerts-unified.yml 有變更 runs-on: ubuntu-latest timeout-minutes: 5 if: | contains(github.event.commits[*].modified, 'ops/monitoring/alerts-unified.yml') || github.event_name == 'workflow_dispatch' steps: - uses: actions/checkout@v4 - name: Validate alerts YAML run: python3 -c "import yaml; yaml.safe_load(open('ops/monitoring/alerts-unified.yml')); print('YAML OK')" - name: Deploy alerts to Prometheus run: bash scripts/ops/deploy-alerts.sh - name: Notify deploy result if: always() env: TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }} TELEGRAM_CHAT_ID: ${{ secrets.TELEGRAM_CHAT_ID }} run: | STATUS="${{ job.status }}" EMOJI="✅" [ "$STATUS" != "success" ] && EMOJI="❌" curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \ -d "chat_id=${TELEGRAM_CHAT_ID}" \ -d "text=${EMOJI} Prometheus 告警規則部署 ${STATUS} (commit: ${GITHUB_SHA::7})" || true ``` - [ ] **Step 3.3: Commit 並推送到 Gitea 觸發驗證** ```bash git add .gitea/workflows/cd.yaml git commit -m "ci: 加入 deploy-alerts CD job — 告警規則變更自動部署到 Prometheus Co-Authored-By: Claude Sonnet 4.6 " git push gitea main ``` - [ ] **Step 3.4: 監控 Gitea Actions 確認 deploy-alerts job 成功** ```bash # 瀏覽器開啟或 curl 確認 curl -s -u 'wooo:TOKEN' 'http://192.168.0.110:3001/api/v1/repos/wooo/awoooi/actions/runs?limit=3' | python3 -c " import sys, json runs = json.load(sys.stdin) for r in runs.get('workflow_runs', [])[:3]: print(r['name'], r['status'], r['conclusion']) " ``` --- ### Task 4：啟動 Sentry + 加入 startup-110.sh **Files:** - Modify: `scripts/reboot-recovery/awoooi-startup-110.sh` - [ ] **Step 4.1: 手動啟動 Sentry 驗證** ```bash ssh wooo@192.168.0.110 "cd /opt/sentry && docker compose up -d 2>&1 | tail -10" sleep 30 ssh wooo@192.168.0.110 "curl -s -o /dev/null -w '%{http_code}' http://localhost:9000/api/0/projects/" ``` 預期：HTTP 200（或 403，表示 Sentry 已啟動但需登入）。 - [ ] **Step 4.2: 加入 startup-110.sh Step 7** 找到 `scripts/reboot-recovery/awoooi-startup-110.sh` 的最後 `# 完成` 區塊： ```bash # ────────────────────────────────────────────── # 完成 # ────────────────────────────────────────────── log "=== 192.168.0.110 啟動序列完成 ===" ``` 在它之前插入： ```bash # ────────────────────────────────────────────── # STEP 7: Sentry（Error Tracking） # 2026-04-05 Claude Code: 加入 — 解決重開機後 Sentry 未自動啟動 # 安裝位置: /opt/sentry (2026-03-24 已安裝) # DSN: awoooi-web :2, awoooi-api :3 (見 memory/project_sentry_full_integration.md) # ────────────────────────────────────────────── log "[7/7] 啟動 Sentry..." SENTRY_DIR="/opt/sentry" if [ -d "$SENTRY_DIR" ]; then cd "$SENTRY_DIR" docker compose up -d 2>&1 | tail -5 log "✅ Sentry 啟動指令已發送 (啟動約需 2-3 分鐘)" sleep 20 # 非阻塞驗證：Sentry 啟動慢，只做快速健康檢查 if curl -sf --max-time 10 http://localhost:9000/api/0/projects/ >/dev/null 2>&1 || \ curl -sf --max-time 10 -o /dev/null -w "%{http_code}" http://localhost:9000/ | grep -q "200\|302\|400"; then log "✅ Sentry 已回應" else log "⚠️ Sentry 尚未就緒（正常現象，通常需 2-3 分鐘）" fi else log "⚠️ 找不到 Sentry 目錄: $SENTRY_DIR" fi ``` 同時更新末尾日誌行，加入 Sentry URL： 找到： ```bash log "Gitea Runner: docker logs gitea-runner" ``` 改為： ```bash log "Gitea Runner: docker logs gitea-runner" log "Sentry: http://192.168.0.110:9000" ``` - [ ] **Step 4.3: 部署到 110** ```bash scp scripts/reboot-recovery/awoooi-startup-110.sh wooo@192.168.0.110:/usr/local/bin/awoooi-startup-110.sh ssh wooo@192.168.0.110 "chmod +x /usr/local/bin/awoooi-startup-110.sh" ``` - [ ] **Step 4.4: Commit** ```bash git add scripts/reboot-recovery/awoooi-startup-110.sh git commit -m "ops(startup): startup-110.sh 加入 Step 7 Sentry 自動啟動 Sentry 已安裝於 /opt/sentry (2026-03-24)，但重開機後未自動啟動 加入非阻塞啟動：docker compose up -d + 20s 等待 + 快速健康檢查 Co-Authored-By: Claude Sonnet 4.6 " ``` --- ### Task 5：更新 REBOOT-RECOVERY-SOP.md v4.0 **Files:** - Modify: `docs/runbooks/REBOOT-RECOVERY-SOP.md` - [ ] **Step 5.1: 更新版本號和摘要** 找到文件開頭： ```markdown > **版本**: v3.0 > **最後更新**: 2026-04-05 (台北時間) > **更新者**: Claude Code (首席架構師) > **觸發事件**: 兩次重開機事故後完整盤點 + 告警鏈路根因修復 + Gitea Runner 自動化 ``` 改為： ```markdown > **版本**: v4.0 > **最後更新**: 2026-04-05 (台北時間) > **更新者**: Claude Code (首席架構師) > **觸發事件**: Prometheus 規則未部署根因修復 + Sentry 啟動自動化 + 全系統自愈閉環設計 ``` - [ ] **Step 5.2: 更新架構圖加入 Sentry** 找到： ``` ├── Alertmanager :9093 ← 告警路由 ├── Grafana :3002 ← 監控儀表板 └── SignOz ← 可觀測性 ``` 改為： ``` ├── Alertmanager :9093 ← 告警路由 ├── Grafana :3002 ← 監控儀表板 ├── Sentry :9000 ← Error Tracking (2026-03-24，2026-04-05 加入 startup) └── SignOz ← 可觀測性 ``` - [ ] **Step 5.3: 更新 110 自動化腳本清單** 找到： ``` **110 (6 步驟)**: ``` 改為： ``` **110 (7 步驟)**: ``` 在末尾加入： ``` - **Gitea Act Runner** (自動清除過期 .runner 配置) - **Sentry** (/opt/sentry，Error Tracking) ``` - [ ] **Step 5.4: 更新 110 詳細啟動序列表格** 找到 `192.168.0.110` 啟動序列表格末尾，加入： ```markdown | 7 | Sentry | `curl http://localhost:9000/api/0/projects/` | Docker 未啟動 (重開機後) | ``` - [ ] **Step 5.5: 更新告警沉默故障排查加入「規則未部署」診斷** 在告警沉默診斷樹末尾加入： ```markdown **補充診斷 — 特定服務無告警但 Alertmanager 正常：** ``` 確認 Prometheus 是否有該規則: ssh wooo@192.168.0.110 "curl -s http://localhost:9090/api/v1/rules | python3 -c \"import sys,json; r=json.load(sys.stdin); names=[x['name'] for g in r['data']['groups'] for x in g['rules']]; [print(n) for n in names if 'Sentry' in n or 'Harbor' in n or 'Gitea' in n]\"" 若規則不存在: → 規則未部署，執行: bash scripts/ops/deploy-alerts.sh 若規則存在但 inactive: → 檢查 blackbox probe target IP 是否正確 → curl http://192.168.0.110:9090/api/v1/query?query=probe_success ``` ``` - [ ] **Step 5.6: 更新 E2E 驗證腳本加入 Sentry + 規則數** 在 E2E 驗證腳本的 `# 告警鏈路 E2E` 之前加入： ```bash check "110 Sentry" "curl -s -o /dev/null -w '%{http_code}' --max-time 10 http://192.168.0.110:9000/" "200\|302\|400" check "Prometheus rules >30" "ssh wooo@192.168.0.110 'curl -s http://localhost:9090/api/v1/rules' | python3 -c \"import sys,json; r=json.load(sys.stdin); n=sum(len(g['rules']) for g in r['data']['groups']); print(n)\"" "[3-9][0-9]" ``` - [ ] **Step 5.7: 更新版本歷史** 在版本歷史表格加入： ```markdown | v4.0 | 2026-04-05 下午 | Prometheus 規則統一部署 + Sentry startup + 診斷樹補充 + E2E 腳本更新 | ``` - [ ] **Step 5.8: Commit** ```bash git add docs/runbooks/REBOOT-RECOVERY-SOP.md git commit -m "docs(sop): REBOOT-RECOVERY-SOP.md v4.0 更新: - 加入 Sentry /opt/sentry 啟動說明 (110 Step 7) - 告警沉默診斷樹補充「規則未部署」診斷 - E2E 驗證腳本加入 Sentry + Prometheus 規則數驗證 - 架構圖補充 Sentry :9000 Co-Authored-By: Claude Sonnet 4.6 " ``` --- ### Task 6：Sprint 1 驗收 - [ ] **Step 6.1: 執行完整 E2E 驗證** ```bash # 確認 Prometheus 規則 ssh wooo@192.168.0.110 "curl -s http://localhost:9090/api/v1/rules | python3 -c \" import sys, json r = json.load(sys.stdin) names = [x['name'] for g in r['data']['groups'] for x in g['rules']] key = ['SentryDown','HarborDown','GiteaDown','OpenClawDown','AlertmanagerDown','AlertChainUnhealthy'] for k in key: print(f'{'✅' if k in names else '❌'} {k}') print(f'Total rules: {len(names)}') \"" ``` 預期：所有 6 條關鍵規則 ✅，Total rules ≥ 30。 - [ ] **Step 6.2: 確認 Sentry 健康** ```bash curl -s -o /dev/null -w "%{http_code}" http://192.168.0.110:9000/ ``` 預期：`200`、`302` 或 `400` (需登入，表示服務正常)。 - [ ] **Step 6.3: 觸發測試告警確認鏈路** ```bash curl -X POST http://192.168.0.121:32334/api/v1/webhooks/alertmanager \ -H 'Content-Type: application/json' \ -d '{ "receiver":"test","status":"firing", "alerts":[{ "status":"firing", "labels":{"alertname":"Sprint1ValidationTest","severity":"info","layer":"k8s","component":"test"}, "annotations":{"summary":"Sprint 1 驗收測試，請忽略"}, "startsAt":"2026-04-05T00:00:00Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":"" }], "groupLabels":{},"commonLabels":{},"commonAnnotations":{},"externalURL":"","version":"4","groupKey":"sprint1-test" }' ``` 預期：`{"success":true,...}` 且 Telegram 收到測試告警。 Sprint 1 ✅ 完成條件：規則 ≥ 30、關鍵規則全部存在、Sentry 回應、告警鏈路通。 --- ## SPRINT 2：SigNoz Log Alerting + Sentry 整合完善 --- ### Task 7：SigNoz Log Alert Rules 建立 **說明：** SigNoz 不支援 YAML 匯入，alert rules 透過 UI 設定。本 task 建立文檔並逐一在 UI 設定。 **Files:** - Create: `ops/signoz/alerting/log-rules.md` - [ ] **Step 7.1: 建立 log-rules.md 文檔** 建立 `ops/signoz/alerting/log-rules.md`： ```markdown # SigNoz Log-Based Alert Rules # 2026-04-05 Claude Code: Sprint 2 — 日誌告警 # 設定位置: http://192.168.0.188:3301/alerts (SigNoz UI) # Webhook: http://192.168.0.121:32334/api/v1/webhooks/signoz ## Rule 1: API 高錯誤日誌率 | 欄位 | 值 | |------|-----| | Name | APIHighErrorLogRate | | Type | Logs Based Alert | | Query | `service.name = "awoooi-api" AND severity_text = "ERROR"` | | Condition | Count > 10 per 5m | | For | 5m | | Severity | warning | | Labels | layer=k8s, component=api, team=backend | ## Rule 2: Worker 任務失敗 | 欄位 | 值 | |------|-----| | Name | WorkerTaskFailed | | Type | Logs Based Alert | | Query | `service.name = "awoooi-worker" AND (body CONTAINS "task_failed" OR body CONTAINS "Unhandled exception")` | | Condition | Count > 5 per 5m | | For | 5m | | Severity | warning | | Labels | layer=k8s, component=worker, team=backend | ## Rule 3: Pod OOM Kill | 欄位 | 值 | |------|-----| | Name | PodOOMKilled | | Type | Logs Based Alert | | Query | `body CONTAINS "OOMKilled" OR body CONTAINS "OutOfMemory"` | | Condition | Count > 0 per 1m | | For | 1m | | Severity | critical | | Labels | layer=k8s, component=k8s, team=ops | ## Rule 4: Telegram Polling 失敗 | 欄位 | 值 | |------|-----| | Name | TelegramPollingFailed | | Type | Logs Based Alert | | Query | `service.name = "awoooi-api" AND body CONTAINS "telegram_polling_error"` | | Condition | Count > 3 per 5m | | For | 5m | | Severity | critical | | Labels | layer=k8s, component=api, team=platform | ## Rule 5: Nemotron 全部超時 | 欄位 | 值 | |------|-----| | Name | NemotronAllTimeout | | Type | Logs Based Alert | | Query | `service.name = "awoooi-api" AND body CONTAINS "nemotron_tool_call_timeout"` | | Condition | Count > 5 per 5m | | For | 5m | | Severity | warning | | Labels | layer=k8s, component=ai, team=ai | ## 設定步驟 1. 開啟 http://192.168.0.188:3301/alerts 2. 點擊 "New Alert Rule" 3. 選擇 "Logs Based Alert" 4. 填入上述欄位 5. Notification Channel: 選擇 awoooi-api webhook 6. 保存並啟用 ``` - [ ] **Step 7.2: 在 SigNoz UI 設定 Rule 1 (APIHighErrorLogRate)** 開啟 http://192.168.0.188:3301/alerts，依照上面 log-rules.md 的 Rule 1 設定。 設定完成後，在 UI 確認 rule 狀態為 `inactive` (表示已設定，目前無觸發)。 - [ ] **Step 7.3: 設定其餘 4 條規則 (Rule 2-5)** 依照 log-rules.md 逐一設定 Rule 2、3、4、5。 - [ ] **Step 7.4: 驗證 SigNoz webhook 鏈路** ```bash curl -X POST http://192.168.0.121:32334/api/v1/webhooks/signoz \ -H 'Content-Type: application/json' \ -d '{ "alerts": [{ "labels": {"alertname": "Sprint2LogAlertTest", "severity": "info", "layer": "k8s"}, "annotations": {"summary": "Sprint 2 log alert 驗證，請忽略"}, "status": "firing" }], "version": "4" }' ``` 預期：`{"success":true}` - [ ] **Step 7.5: Commit log-rules.md** ```bash git add ops/signoz/alerting/log-rules.md git commit -m "ops(signoz): 建立 log-based alert rules 文檔 (Sprint 2) 5 條規則: APIHighErrorLogRate/WorkerTaskFailed/PodOOMKilled/ TelegramPollingFailed/NemotronAllTimeout Co-Authored-By: Claude Sonnet 4.6 " ``` --- ### Task 8：Sentry SDK Tags 補全 **說明：** 在 API 的 Sentry init 加入統一標籤，讓 Sentry 事件攜帶 layer/component 資訊。 **Files:** - Modify: `apps/api/src/main.py` - [ ] **Step 8.1: 確認 main.py Sentry 初始化位置** ```bash grep -n "sentry_sdk\|sentry.init\|SENTRY_DSN" apps/api/src/main.py | head -10 ``` - [ ] **Step 8.2: 加入統一 tags** 找到 Sentry init 的設定（通常是 `sentry_sdk.init(...)` 或在 `configure_sentry()` 函式）。 在 `sentry_sdk.init()` 呼叫後加入： ```python # 2026-04-05 Claude Code: 加入統一標籤，對齊 Prometheus/auto_repair layer 規範 import sentry_sdk sentry_sdk.set_tag("layer", "k8s") sentry_sdk.set_tag("component", "api") sentry_sdk.set_tag("host", "k8s-awoooi-prod") sentry_sdk.set_tag("team", "backend") ``` - [ ] **Step 8.3: 驗證 (啟動 API 後確認 Sentry 事件有 tag)** ```bash # 在 API pod 內觸發一個測試 exception kubectl exec -n awoooi-prod deploy/awoooi-api -- python3 -c " import sentry_sdk sentry_sdk.capture_message('Sprint2 tag validation', level='info') print('sent') " ``` 在 Sentry UI (http://192.168.0.110:9000) 確認事件有 `layer: k8s` tag。 - [ ] **Step 8.4: Commit** ```bash git add apps/api/src/main.py git commit -m "feat(api): Sentry init 加入統一 layer/component 標籤 對齊 Prometheus 告警標籤規範 (layer/component/team) 讓 Sentry 事件與 auto_repair 路由決策保持一致 Co-Authored-By: Claude Sonnet 4.6 " ``` --- ## SPRINT 3：Host Auto-Repair Agent (SSH) --- ### Task 9：主機白名單修復腳本 **說明：** 在 110/188 建立嚴格白名單的修復腳本，SSH key 的 `command=` 限制只能執行此腳本，防止任意命令執行。 **Files:** - Create: `scripts/repair-bot/repair-bot-110.sh` - Create: `scripts/repair-bot/repair-bot-188.sh` - [ ] **Step 9.1: 建立修復腳本目錄** ```bash mkdir -p scripts/repair-bot ``` - [ ] **Step 9.2: 建立 repair-bot-110.sh** 建立 `scripts/repair-bot/repair-bot-110.sh`： ```bash #!/bin/bash # scripts/repair-bot/repair-bot-110.sh # 修復機器人白名單腳本 — 110 主機 (DevOps 金庫) # 2026-04-05 Claude Code: Sprint 3 Host Auto-Repair # # 安全設計: # - SSH authorized_keys 的 command= 指向此腳本 # - 只允許執行 ALLOWED_COMMANDS 中定義的修復指令 # - 格式: repair: # - SSH key 洩漏也只能執行白名單內的 docker compose up -d # # 部署位置: /home/wooo/bin/repair-bot-110.sh (on 192.168.0.110) # 使用者: wooo LOG="/var/log/awoooi-repair-bot.log" log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG"; } # 白名單: component → compose dir declare -A COMPOSE_DIRS=( ["sentry"]="/opt/sentry" ["harbor"]="/home/wooo/harbor/harbor" ["gitea"]="/home/wooo/gitea" ["gitea-runner"]="/home/wooo/act-runner" ["langfuse"]="/home/wooo/langfuse" ["alertmanager"]="/home/wooo/monitoring" ["signoz"]="/home/wooo/signoz/deploy/docker" ) CMD="${SSH_ORIGINAL_COMMAND:-}" log "repair-bot-110 invoked: CMD=$CMD" if [[ "$CMD" =~ ^repair:([a-z0-9_-]+)$ ]]; then COMPONENT="${BASH_REMATCH[1]}" DIR="${COMPOSE_DIRS[$COMPONENT]}" if [ -z "$DIR" ]; then log "DENIED: unknown component '$COMPONENT'" echo "REPAIR_DENIED:unknown_component:$COMPONENT" exit 1 fi if [ ! -d "$DIR" ]; then log "DENIED: directory not found '$DIR'" echo "REPAIR_DENIED:dir_not_found:$DIR" exit 1 fi log "EXECUTING: cd $DIR && docker compose up -d" cd "$DIR" && docker compose up -d 2>&1 | tail -5 EXIT_CODE=$? if [ $EXIT_CODE -eq 0 ]; then log "REPAIR_OK: $COMPONENT" echo "REPAIR_OK:$COMPONENT" else log "REPAIR_FAIL: $COMPONENT (exit $EXIT_CODE)" echo "REPAIR_FAIL:$COMPONENT:exit_$EXIT_CODE" exit 1 fi elif [ "$CMD" = "health" ]; then # 健康檢查 — 允許連線測試 echo "REPAIR_BOT_HEALTHY:110" else log "DENIED: invalid command '$CMD'" echo "REPAIR_DENIED:invalid_command" exit 1 fi ``` - [ ] **Step 9.3: 建立 repair-bot-188.sh** 建立 `scripts/repair-bot/repair-bot-188.sh`： ```bash #!/bin/bash # scripts/repair-bot/repair-bot-188.sh # 修復機器人白名單腳本 — 188 主機 (主服務主機) # 2026-04-05 Claude Code: Sprint 3 Host Auto-Repair # # 部署位置: /home/ollama/bin/repair-bot-188.sh (on 192.168.0.188) # 使用者: ollama LOG="/var/log/awoooi-repair-bot.log" log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG"; } # 白名單: component → 修復方式 declare -A COMPOSE_DIRS=( ["openclaw"]="/home/ollama/clawbot-v5" ["minio"]="/home/ollama/minio" ["signoz"]="/home/ollama/signoz/deploy/docker" ) declare -A SYSTEMD_SERVICES=( ["redis"]="redis-server" ["nginx"]="nginx" ["ollama"]="ollama" ) CMD="${SSH_ORIGINAL_COMMAND:-}" log "repair-bot-188 invoked: CMD=$CMD" if [[ "$CMD" =~ ^repair:([a-z0-9_-]+)$ ]]; then COMPONENT="${BASH_REMATCH[1]}" # Docker Compose 類 DIR="${COMPOSE_DIRS[$COMPONENT]}" if [ -n "$DIR" ]; then if [ ! -d "$DIR" ]; then log "DENIED: directory not found '$DIR'" echo "REPAIR_DENIED:dir_not_found:$DIR" exit 1 fi log "EXECUTING: cd $DIR && docker compose up -d" cd "$DIR" && docker compose up -d 2>&1 | tail -5 EXIT_CODE=$? [ $EXIT_CODE -eq 0 ] && echo "REPAIR_OK:$COMPONENT" || { echo "REPAIR_FAIL:$COMPONENT"; exit 1; } exit 0 fi # Systemd 類 SVC="${SYSTEMD_SERVICES[$COMPONENT]}" if [ -n "$SVC" ]; then log "EXECUTING: sudo systemctl restart $SVC" sudo systemctl restart "$SVC" 2>&1 EXIT_CODE=$? [ $EXIT_CODE -eq 0 ] && echo "REPAIR_OK:$COMPONENT" || { echo "REPAIR_FAIL:$COMPONENT"; exit 1; } exit 0 fi log "DENIED: unknown component '$COMPONENT'" echo "REPAIR_DENIED:unknown_component:$COMPONENT" exit 1 elif [ "$CMD" = "health" ]; then echo "REPAIR_BOT_HEALTHY:188" else log "DENIED: invalid command '$CMD'" echo "REPAIR_DENIED:invalid_command" exit 1 fi ``` - [ ] **Step 9.4: 部署腳本到主機** ```bash # 部署到 110 ssh wooo@192.168.0.110 "mkdir -p /home/wooo/bin" scp scripts/repair-bot/repair-bot-110.sh wooo@192.168.0.110:/home/wooo/bin/repair-bot-110.sh ssh wooo@192.168.0.110 "chmod +x /home/wooo/bin/repair-bot-110.sh" # 部署到 188 ssh ollama@192.168.0.188 "mkdir -p /home/ollama/bin" scp scripts/repair-bot/repair-bot-188.sh ollama@192.168.0.188:/home/ollama/bin/repair-bot-188.sh ssh ollama@192.168.0.188 "chmod +x /home/ollama/bin/repair-bot-188.sh" ``` - [ ] **Step 9.5: Commit** ```bash git add scripts/repair-bot/ git commit -m "ops(repair-bot): 主機白名單修復腳本 (Sprint 3) 110: sentry/harbor/gitea/gitea-runner/langfuse/alertmanager 188: openclaw/minio/signoz/redis/nginx/ollama 安全設計: command= 限制 + 嚴格白名單 + 操作日誌 Co-Authored-By: Claude Sonnet 4.6 " ``` --- ### Task 10：SSH Key 基礎設施 **Files:** - Create: `k8s/awoooi-prod/04-repair-ssh-secret.yaml` - [ ] **Step 10.1: 生成 SSH keypair** ```bash # 在 Mac 本機執行（不上傳私鑰到 Git） ssh-keygen -t ed25519 -C "awoooi-repair-bot-2026" -f /tmp/awoooi_repair_bot -N "" echo "Public key:" cat /tmp/awoooi_repair_bot.pub ``` 記錄公鑰內容（格式：`ssh-ed25519 AAAA... awoooi-repair-bot-2026`）。 - [ ] **Step 10.2: 在 110 設定受限 authorized_keys** ```bash # 將以下內容加入 110 的 authorized_keys（替換 PUBKEY 為實際公鑰） PUBKEY=$(cat /tmp/awoooi_repair_bot.pub) ssh wooo@192.168.0.110 "echo 'command=\"/home/wooo/bin/repair-bot-110.sh\",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ${PUBKEY}' >> ~/.ssh/authorized_keys" # 驗證 ssh -i /tmp/awoooi_repair_bot -o StrictHostKeyChecking=accept-new wooo@192.168.0.110 "health" ``` 預期輸出：`REPAIR_BOT_HEALTHY:110` - [ ] **Step 10.3: 在 188 設定受限 authorized_keys** ```bash PUBKEY=$(cat /tmp/awoooi_repair_bot.pub) ssh ollama@192.168.0.188 "echo 'command=\"/home/ollama/bin/repair-bot-188.sh\",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ${PUBKEY}' >> ~/.ssh/authorized_keys" # 驗證 ssh -i /tmp/awoooi_repair_bot -o StrictHostKeyChecking=accept-new ollama@192.168.0.188 "health" ``` 預期輸出：`REPAIR_BOT_HEALTHY:188` - [ ] **Step 10.4: 存入 K8s Secret** ```bash kubectl create secret generic awoooi-repair-ssh-key \ -n awoooi-prod \ --from-file=id_ed25519=/tmp/awoooi_repair_bot \ --from-file=id_ed25519.pub=/tmp/awoooi_repair_bot.pub # 驗證 kubectl get secret awoooi-repair-ssh-key -n awoooi-prod ``` - [ ] **Step 10.5: 建立 Secret YAML template (不含實際 key)** 建立 `k8s/awoooi-prod/04-repair-ssh-secret.yaml`： ```yaml # k8s/awoooi-prod/04-repair-ssh-secret.yaml # SSH Secret Template — 不含實際私鑰 (存於 K8s，不上 Git) # 2026-04-05 Claude Code: Sprint 3 Host Auto-Repair # # 建立方式: # ssh-keygen -t ed25519 -C "awoooi-repair-bot-2026" -f /tmp/awoooi_repair_bot -N "" # kubectl create secret generic awoooi-repair-ssh-key \ # -n awoooi-prod \ # --from-file=id_ed25519=/tmp/awoooi_repair_bot \ # --from-file=id_ed25519.pub=/tmp/awoooi_repair_bot.pub # # 主機配置 (已完成): # 110 ~/.ssh/authorized_keys: command="/home/wooo/bin/repair-bot-110.sh",... # 188 ~/.ssh/authorized_keys: command="/home/ollama/bin/repair-bot-188.sh",... apiVersion: v1 kind: Secret metadata: name: awoooi-repair-ssh-key namespace: awoooi-prod annotations: # 此 template 不含實際私鑰，需手動 kubectl create secret 建立 awoooi.io/secret-type: "ssh-repair-bot" awoooi.io/created: "2026-04-05" type: Opaque # data: 不在版控中 — 使用 kubectl create secret 建立 ``` - [ ] **Step 10.6: Commit (只 commit template，不 commit 私鑰)** ```bash # 確認私鑰不在 git 追蹤中 git status | grep -v awoooi_repair_bot # 確認 /tmp/ 的 key 不會被加入 git add k8s/awoooi-prod/04-repair-ssh-secret.yaml git commit -m "k8s(secret): 加入 repair-ssh-key Secret template (不含實際私鑰) 實際私鑰透過 kubectl create secret 手動建立，不上 Git 主機 110/188 的 authorized_keys 已設定 command= 限制 Co-Authored-By: Claude Sonnet 4.6 " ``` --- ### Task 11：HostRepairAgent Python 模組 **Files:** - Create: `apps/api/src/services/host_repair_agent.py` - Create: `apps/api/tests/test_host_repair_agent.py` - [ ] **Step 11.1: 寫測試 (先)** 建立 `apps/api/tests/test_host_repair_agent.py`： ```python """ tests/test_host_repair_agent.py Host Repair Agent 單元測試 不需要實際 SSH 連線 — 測試路由邏輯和命令組裝 """ import pytest from unittest.mock import AsyncMock, MagicMock, patch # ============================================================================= # 測試 HostRepairConfig 路由 # ============================================================================= class TestHostRepairConfig: def test_layer_docker_110_routes_to_110(self): from src.services.host_repair_agent import get_ssh_config_for_layer config = get_ssh_config_for_layer("docker-110") assert config["user"] == "wooo" assert config["host"] == "192.168.0.110" def test_layer_docker_188_routes_to_188(self): from src.services.host_repair_agent import get_ssh_config_for_layer config = get_ssh_config_for_layer("docker-188") assert config["user"] == "ollama" assert config["host"] == "192.168.0.188" def test_layer_systemd_188_routes_to_188(self): from src.services.host_repair_agent import get_ssh_config_for_layer config = get_ssh_config_for_layer("systemd-188") assert config["user"] == "ollama" assert config["host"] == "192.168.0.188" def test_unknown_layer_raises(self): from src.services.host_repair_agent import get_ssh_config_for_layer with pytest.raises(ValueError, match="Unknown layer"): get_ssh_config_for_layer("unknown-layer") def test_k8s_layer_raises(self): """k8s layer 不走 SSH，應 raise""" from src.services.host_repair_agent import get_ssh_config_for_layer with pytest.raises(ValueError, match="layer 'k8s' is handled by kubectl"): get_ssh_config_for_layer("k8s") # ============================================================================= # 測試 SSH 命令組裝 # ============================================================================= class TestSSHCommandBuilding: def test_repair_command_format(self): from src.services.host_repair_agent import build_repair_command cmd = build_repair_command("sentry") assert cmd == "repair:sentry" def test_repair_command_component_sanitized(self): """防止 command injection""" from src.services.host_repair_agent import build_repair_command with pytest.raises(ValueError, match="Invalid component"): build_repair_command("sentry; rm -rf /") def test_repair_command_valid_components(self): from src.services.host_repair_agent import build_repair_command valid = ["sentry", "harbor", "gitea", "openclaw", "gitea-runner", "alertmanager", "redis", "nginx"] for component in valid: cmd = build_repair_command(component) assert cmd == f"repair:{component}" # ============================================================================= # 測試 HostRepairAgent.repair() 路由 # ============================================================================= class TestHostRepairAgent: @pytest.mark.asyncio async def test_repair_success_returns_ok(self): from src.services.host_repair_agent import HostRepairAgent agent = HostRepairAgent() with patch.object(agent, "_ssh_execute", new_callable=AsyncMock) as mock_ssh: mock_ssh.return_value = "REPAIR_OK:sentry" result = await agent.repair(layer="docker-110", component="sentry") assert result.success is True assert result.component == "sentry" assert result.layer == "docker-110" mock_ssh.assert_called_once_with( host="192.168.0.110", user="wooo", command="repair:sentry" ) @pytest.mark.asyncio async def test_repair_fail_returns_failure(self): from src.services.host_repair_agent import HostRepairAgent agent = HostRepairAgent() with patch.object(agent, "_ssh_execute", new_callable=AsyncMock) as mock_ssh: mock_ssh.return_value = "REPAIR_FAIL:harbor:exit_1" result = await agent.repair(layer="docker-110", component="harbor") assert result.success is False assert "REPAIR_FAIL" in result.error @pytest.mark.asyncio async def test_repair_ssh_timeout_returns_failure(self): from src.services.host_repair_agent import HostRepairAgent import asyncio agent = HostRepairAgent() with patch.object(agent, "_ssh_execute", new_callable=AsyncMock) as mock_ssh: mock_ssh.side_effect = asyncio.TimeoutError() result = await agent.repair(layer="docker-110", component="sentry") assert result.success is False assert "timeout" in result.error.lower() @pytest.mark.asyncio async def test_repair_denied_returns_failure(self): from src.services.host_repair_agent import HostRepairAgent agent = HostRepairAgent() with patch.object(agent, "_ssh_execute", new_callable=AsyncMock) as mock_ssh: mock_ssh.return_value = "REPAIR_DENIED:unknown_component:badcomponent" result = await agent.repair(layer="docker-110", component="badcomponent") assert result.success is False ``` - [ ] **Step 11.2: 執行測試確認全部 FAIL（模組未建立）** ```bash cd apps/api python -m pytest tests/test_host_repair_agent.py -v 2>&1 | head -20 ``` 預期：`ImportError` 或 `ModuleNotFoundError`。 - [ ] **Step 11.3: 建立 host_repair_agent.py** 建立 `apps/api/src/services/host_repair_agent.py`： ```python """ Host Repair Agent ================= 透過 SSH 執行主機層 (Docker/systemd) 修復動作 安全設計: - SSH key 限制: authorized_keys 的 command= 強制只能執行白名單腳本 - Component 白名單驗證: 防止 command injection - Timeout: 60 秒超時防止卡住 Layer 對應: docker-110 → SSH wooo@192.168.0.110 → repair-bot-110.sh docker-188 → SSH ollama@192.168.0.188 → repair-bot-188.sh systemd-188 → SSH ollama@192.168.0.188 → repair-bot-188.sh 建立時間: 2026-04-05 (台北時區) 建立者: Claude Code (Sprint 3 Host Auto-Repair) """ import asyncio import re from dataclasses import dataclass, field from pathlib import Path import structlog logger = structlog.get_logger(__name__) # SSH key 路徑 (從 K8s Secret 掛載) SSH_KEY_PATH = "/etc/repair-ssh/id_ed25519" SSH_TIMEOUT = 60 # 秒 # Layer → SSH 配置 LAYER_SSH_CONFIG: dict[str, dict[str, str]] = { "docker-110": {"user": "wooo", "host": "192.168.0.110"}, "docker-188": {"user": "ollama", "host": "192.168.0.188"}, "systemd-188": {"user": "ollama", "host": "192.168.0.188"}, } # 允許的 component 名稱 (白名單) VALID_COMPONENT_PATTERN = re.compile(r"^[a-z0-9][a-z0-9-]{0,30}$") # ============================================================================= # Data classes # ============================================================================= @dataclass class HostRepairResult: """SSH 修復執行結果""" success: bool component: str layer: str output: str = "" error: str | None = None # ============================================================================= # 純函式 (純邏輯，可單元測試) # ============================================================================= def get_ssh_config_for_layer(layer: str) -> dict[str, str]: """ 根據 layer 返回 SSH 連線配置 Raises: ValueError: layer 不支援 SSH 修復 """ if layer == "k8s": raise ValueError("layer 'k8s' is handled by kubectl, not SSH") config = LAYER_SSH_CONFIG.get(layer) if config is None: raise ValueError(f"Unknown layer: '{layer}'. Valid: {list(LAYER_SSH_CONFIG.keys())}") return config def build_repair_command(component: str) -> str: """ 組裝修復命令，並驗證 component 格式防止 injection Raises: ValueError: component 格式不合法 """ if not VALID_COMPONENT_PATTERN.match(component): raise ValueError(f"Invalid component name: '{component}'. Must match [a-z0-9-]+") return f"repair:{component}" # ============================================================================= # HostRepairAgent # ============================================================================= class HostRepairAgent: """ 透過 SSH 執行主機修復。 使用 command= 受限的 SSH key，只能觸發白名單腳本。 """ async def repair( self, layer: str, component: str, timeout: int = SSH_TIMEOUT, ) -> HostRepairResult: """ 執行修復。 Args: layer: "docker-110" | "docker-188" | "systemd-188" component: 服務名稱，如 "sentry", "harbor" timeout: SSH 命令超時秒數 Returns: HostRepairResult """ log = logger.bind(layer=layer, component=component) log.info("host_repair_start") try: config = get_ssh_config_for_layer(layer) command = build_repair_command(component) except ValueError as e: log.error("host_repair_invalid_params", error=str(e)) return HostRepairResult( success=False, component=component, layer=layer, error=str(e) ) try: output = await asyncio.wait_for( self._ssh_execute( host=config["host"], user=config["user"], command=command, ), timeout=timeout, ) except asyncio.TimeoutError: log.error("host_repair_timeout", timeout=timeout) return HostRepairResult( success=False, component=component, layer=layer, error=f"SSH command timeout after {timeout}s", ) except Exception as e: log.error("host_repair_ssh_error", error=str(e)) return HostRepairResult( success=False, component=component, layer=layer, error=str(e) ) success = output.startswith("REPAIR_OK:") log.info( "host_repair_complete", success=success, output=output[:100], ) return HostRepairResult( success=success, component=component, layer=layer, output=output, error=None if success else output, ) async def health_check(self, layer: str) -> bool: """驗證 SSH 連線是否正常""" try: config = get_ssh_config_for_layer(layer) output = await asyncio.wait_for( self._ssh_execute(host=config["host"], user=config["user"], command="health"), timeout=10, ) return "REPAIR_BOT_HEALTHY" in output except Exception: return False async def _ssh_execute(self, host: str, user: str, command: str) -> str: """ 執行 SSH 命令。 使用 subprocess 呼叫 ssh，key 從 SSH_KEY_PATH 讀取。 Note: 使用 subprocess 而非 paramiko，避免引入額外依賴。 command= 限制確保 SSH 只能執行白名單腳本。 """ key_path = SSH_KEY_PATH if not Path(key_path).exists(): # 開發環境 fallback (測試用) raise FileNotFoundError(f"SSH key not found: {key_path}") proc = await asyncio.create_subprocess_exec( "ssh", "-i", key_path, "-o", "StrictHostKeyChecking=accept-new", "-o", "ConnectTimeout=10", "-o", "BatchMode=yes", f"{user}@{host}", command, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, ) stdout, stderr = await proc.communicate() output = stdout.decode().strip() if proc.returncode != 0 and not output: output = stderr.decode().strip() or f"SSH exit code {proc.returncode}" return output # ============================================================================= # Singleton # ============================================================================= _host_repair_agent: HostRepairAgent | None = None def get_host_repair_agent() -> HostRepairAgent: global _host_repair_agent if _host_repair_agent is None: _host_repair_agent = HostRepairAgent() return _host_repair_agent ``` - [ ] **Step 11.4: 執行測試確認通過** ```bash cd apps/api python -m pytest tests/test_host_repair_agent.py -v ``` 預期：所有測試 PASS。 - [ ] **Step 11.5: Commit** ```bash git add apps/api/src/services/host_repair_agent.py apps/api/tests/test_host_repair_agent.py git commit -m "feat(api): HostRepairAgent — SSH 主機層修復 Agent 透過 command= 受限 SSH key 觸發 110/188 白名單修復腳本 支援 layer: docker-110 / docker-188 / systemd-188 component 格式驗證防止 command injection Co-Authored-By: Claude Sonnet 4.6 " ``` --- ### Task 12：整合 HostRepairAgent 到 auto_repair_service **Files:** - Modify: `apps/api/src/models/playbook.py` - Modify: `apps/api/src/services/auto_repair_service.py` - [ ] **Step 12.1: 加入 ActionType.SSH_COMMAND** 找到 `apps/api/src/models/playbook.py` 的 ActionType： ```python class ActionType(str, Enum): KUBECTL = "kubectl" SCRIPT = "script" MANUAL = "manual" ``` 改為： ```python class ActionType(str, Enum): KUBECTL = "kubectl" SCRIPT = "script" MANUAL = "manual" SSH_COMMAND = "ssh_command" # 2026-04-05 Claude Code: Sprint 3 主機層修復 ``` - [ ] **Step 12.2: 在 auto_repair_service.py 的 _execute_step 加入 SSH_COMMAND 路由** 找到 `apps/api/src/services/auto_repair_service.py` 的 `_execute_step` 方法（約 L471-500）： ```python async def _execute_step(self, incident: Incident, step) -> str: ... if step.action_type == ActionType.MANUAL: return "SKIPPED (manual step)" if step.action_type == ActionType.KUBECTL: ... return "UNKNOWN_ACTION_TYPE" ``` 在 `return "UNKNOWN_ACTION_TYPE"` 之前加入： ```python if step.action_type == ActionType.SSH_COMMAND: # 2026-04-05 Claude Code: Sprint 3 主機層修復 # command 格式: "" (由 Playbook 定義) # layer 從 incident.labels 取得 from src.services.host_repair_agent import get_host_repair_agent layer = incident.labels.get("layer", "docker-110") component = step.command.strip() agent = get_host_repair_agent() try: result = await agent.repair(layer=layer, component=component) if result.success: return f"SUCCESS (SSH repair:{component} on {layer})" else: return f"FAILED: {result.error}" except Exception as e: logger.error("ssh_repair_step_error", error=str(e)) return f"FAILED: {e}" ``` - [ ] **Step 12.3: 確認測試仍通過** ```bash cd apps/api python -m pytest tests/test_auto_repair_service.py tests/test_host_repair_agent.py -v ``` 預期：全部 PASS。 - [ ] **Step 12.4: Commit** ```bash git add apps/api/src/models/playbook.py apps/api/src/services/auto_repair_service.py git commit -m "feat(api): auto_repair_service 整合 SSH_COMMAND ActionType _execute_step 加入 SSH_COMMAND 路由: - 從 incident.labels['layer'] 決定連線目標 - 呼叫 HostRepairAgent.repair(layer, component) - 修復結果記錄到 executed_steps Co-Authored-By: Claude Sonnet 4.6 " ``` --- ### Task 13：SSH Key 掛載到 API Pod **Files:** - Modify: `k8s/awoooi-prod/01-deployments.yaml` - [ ] **Step 13.1: 確認 API Deployment 位置** ```bash grep -n "awoooi-api\|containers:\|volumeMounts:\|volumes:" k8s/awoooi-prod/01-deployments.yaml | head -20 ``` - [ ] **Step 13.2: 加入 volume 和 volumeMount** 在 API deployment 的 `containers[0]` 下找到 `volumeMounts` 或在 containers 末尾加入： ```yaml # 2026-04-05 Claude Code: Sprint 3 Host Repair SSH key volumeMounts: - name: repair-ssh-key mountPath: /etc/repair-ssh readOnly: true ``` 在同一 deployment 的 `spec.template.spec` 下加入 `volumes`： ```yaml volumes: - name: repair-ssh-key secret: secretName: awoooi-repair-ssh-key defaultMode: 0400 ``` - [ ] **Step 13.3: 套用到 K3s** ```bash ssh wooo@192.168.0.120 "kubectl apply -f -" < k8s/awoooi-prod/01-deployments.yaml ssh wooo@192.168.0.120 "kubectl rollout status deployment/awoooi-api -n awoooi-prod --timeout=120s" ``` - [ ] **Step 13.4: 驗證 SSH key 已掛載** ```bash ssh wooo@192.168.0.120 "kubectl exec -n awoooi-prod deploy/awoooi-api -- ls -la /etc/repair-ssh/" ``` 預期：看到 `id_ed25519` 和 `id_ed25519.pub`，權限 `400`。 - [ ] **Step 13.5: Commit** ```bash git add k8s/awoooi-prod/01-deployments.yaml git commit -m "k8s(api): API pod 掛載 repair SSH key secret 掛載 awoooi-repair-ssh-key 到 /etc/repair-ssh 供 HostRepairAgent 在 Sprint 3 SSH 修復時使用 Co-Authored-By: Claude Sonnet 4.6 " ``` --- ### Task 14：建立 Host Repair Playbooks **說明：** 透過 AWOOOI API 建立修復 Playbook，讓 auto_repair_service 能匹配並執行。 **Files:** - Create: `scripts/ops/seed-repair-playbooks.py` - [ ] **Step 14.1: 建立 seed 腳本** 建立 `scripts/ops/seed-repair-playbooks.py`： ```python #!/usr/bin/env python3 """ seed-repair-playbooks.py 建立 Sprint 3 Host Repair Playbooks 用法: python3 scripts/ops/seed-repair-playbooks.py 需要: AWOOOI API 可訪問 (http://192.168.0.121:32334) """ import json import urllib.request import urllib.error API_BASE = "http://192.168.0.121:32334" PLAYBOOKS = [ { "name": "sentry-down-repair", "description": "Sentry (110) 離線自動修復", "trigger_alert": "SentryDown", "symptom_pattern": { "keywords": ["SentryDown", "sentry", "9000"], "severity": "warning", "labels": {"layer": "docker-110", "component": "sentry"}, }, "repair_steps": [ { "step_number": 1, "action_type": "ssh_command", "command": "sentry", "description": "SSH 到 110，docker compose up -d Sentry", "risk_level": "LOW", "timeout_seconds": 60, } ], "tags": ["sentry", "docker-110", "auto-repair"], "risk_level": "LOW", }, { "name": "harbor-down-repair", "description": "Harbor Registry (110) 離線自動修復", "trigger_alert": "HarborDown", "symptom_pattern": { "keywords": ["HarborDown", "harbor", "5000", "ImagePullBackOff"], "severity": "critical", "labels": {"layer": "docker-110", "component": "harbor"}, }, "repair_steps": [ { "step_number": 1, "action_type": "ssh_command", "command": "harbor", "description": "SSH 到 110，docker compose up -d Harbor", "risk_level": "LOW", "timeout_seconds": 120, } ], "tags": ["harbor", "docker-110", "auto-repair", "registry"], "risk_level": "LOW", }, { "name": "gitea-down-repair", "description": "Gitea (110) 離線自動修復", "trigger_alert": "GiteaDown", "symptom_pattern": { "keywords": ["GiteaDown", "gitea", "3001"], "severity": "critical", "labels": {"layer": "docker-110", "component": "gitea"}, }, "repair_steps": [ { "step_number": 1, "action_type": "ssh_command", "command": "gitea", "description": "SSH 到 110，docker compose up -d Gitea", "risk_level": "LOW", "timeout_seconds": 60, } ], "tags": ["gitea", "docker-110", "auto-repair"], "risk_level": "LOW", }, { "name": "alertmanager-down-repair", "description": "Alertmanager (110) 離線自動修復", "trigger_alert": "AlertmanagerDown", "symptom_pattern": { "keywords": ["AlertmanagerDown", "alertmanager", "9093"], "severity": "critical", "labels": {"layer": "docker-110", "component": "alertmanager"}, }, "repair_steps": [ { "step_number": 1, "action_type": "ssh_command", "command": "alertmanager", "description": "SSH 到 110，docker compose up -d monitoring (含 Alertmanager)", "risk_level": "LOW", "timeout_seconds": 60, } ], "tags": ["alertmanager", "docker-110", "auto-repair", "critical-infra"], "risk_level": "LOW", }, { "name": "openclaw-down-repair", "description": "OpenClaw (188) 離線自動修復", "trigger_alert": "OpenClawDown", "symptom_pattern": { "keywords": ["OpenClawDown", "openclaw", "8088", "clawbot"], "severity": "critical", "labels": {"layer": "docker-188", "component": "openclaw"}, }, "repair_steps": [ { "step_number": 1, "action_type": "ssh_command", "command": "openclaw", "description": "SSH 到 188，docker compose up -d OpenClaw", "risk_level": "LOW", "timeout_seconds": 90, } ], "tags": ["openclaw", "docker-188", "auto-repair"], "risk_level": "LOW", }, ] def create_playbook(playbook_data: dict) -> bool: """透過 API 建立 Playbook""" data = json.dumps(playbook_data).encode() req = urllib.request.Request( f"{API_BASE}/api/v1/playbooks", data=data, headers={"Content-Type": "application/json"}, method="POST", ) try: with urllib.request.urlopen(req, timeout=10) as resp: result = json.loads(resp.read()) print(f" ✅ Created: {playbook_data['name']} (id: {result.get('playbook_id', '?')})") return True except urllib.error.HTTPError as e: body = e.read().decode() if "already exists" in body or e.code == 409: print(f" ⚠️ Already exists: {playbook_data['name']}") return True print(f" ❌ Failed: {playbook_data['name']} — HTTP {e.code}: {body[:100]}") return False except Exception as e: print(f" ❌ Error: {playbook_data['name']} — {e}") return False if __name__ == "__main__": print("=== 建立 Host Repair Playbooks ===") success = 0 for pb in PLAYBOOKS: if create_playbook(pb): success += 1 print(f"\n結果: {success}/{len(PLAYBOOKS)} playbooks 建立成功") ``` - [ ] **Step 14.2: 執行 seed 腳本** ```bash python3 scripts/ops/seed-repair-playbooks.py ``` 預期輸出：所有 Playbook ✅ Created 或 ⚠️ Already exists。 - [ ] **Step 14.3: 確認 Playbooks 已建立** ```bash curl -s http://192.168.0.121:32334/api/v1/playbooks | python3 -c " import sys, json pbs = json.load(sys.stdin) for pb in pbs.get('playbooks', []): print(pb['name'], pb.get('risk_level', '?')) " ``` - [ ] **Step 14.4: Commit** ```bash git add scripts/ops/seed-repair-playbooks.py git commit -m "ops(scripts): seed-repair-playbooks.py — Sprint 3 Host Repair Playbooks 5 個 Playbooks: SentryDown/HarborDown/GiteaDown/AlertmanagerDown/OpenClawDown 全部 LOW 風險，layer: docker-110/docker-188 ActionType: ssh_command → HostRepairAgent Co-Authored-By: Claude Sonnet 4.6 " ``` --- ### Task 15：Sprint 3 E2E 閉環驗證 - [ ] **Step 15.1: 驗證 SSH 連線 (從 API Pod)** ```bash ssh wooo@192.168.0.120 "kubectl exec -n awoooi-prod deploy/awoooi-api -- ssh -i /etc/repair-ssh/id_ed25519 -o StrictHostKeyChecking=accept-new -o BatchMode=yes wooo@192.168.0.110 health" ``` 預期：`REPAIR_BOT_HEALTHY:110` - [ ] **Step 15.2: SentryDown E2E 測試** ```bash # 1. 手動停止 Sentry ssh wooo@192.168.0.110 "cd /opt/sentry && docker compose stop" # 2. 等待 Prometheus 偵測 (SentryDown for: 2m) echo "等待 2 分鐘讓 Prometheus 觸發 SentryDown..." sleep 130 # 3. 確認 Prometheus 告警狀態 ssh wooo@192.168.0.110 "curl -s 'http://localhost:9090/api/v1/alerts' | python3 -c \" import sys, json alerts = json.load(sys.stdin) sentry_alerts = [a for a in alerts['data']['alerts'] if 'SentryDown' in a['labels'].get('alertname','')] for a in sentry_alerts: print(a['labels']['alertname'], a['state']) \"" ``` 預期：看到 `SentryDown firing` - [ ] **Step 15.3: 確認 Alertmanager 已送出 webhook** ```bash ssh wooo@192.168.0.110 "curl -s http://localhost:9093/api/v2/alerts | python3 -c \" import sys, json alerts = json.load(sys.stdin) for a in alerts: if 'SentryDown' in str(a.get('labels',{})): print('Found SentryDown alert in Alertmanager:', a.get('status',{}).get('state','?')) \"" ``` - [ ] **Step 15.4: 確認 auto_repair_service 執行了 SSH 修復** ```bash # 查看 API pod logs ssh wooo@192.168.0.120 "kubectl logs -n awoooi-prod deploy/awoooi-api --since=5m | grep -E 'host_repair|ssh_repair|SentryDown|auto_repair'" ``` 預期：看到 `host_repair_start`、`host_repair_complete`、`success=True` - [ ] **Step 15.5: 確認 Sentry 已恢復** ```bash sleep 30 # 等待 docker compose up -d 完成 curl -s -o /dev/null -w "%{http_code}" http://192.168.0.110:9000/ ``` 預期：`200`、`302` 或 `400`（服務已恢復） - [ ] **Step 15.6: 確認 Telegram 收到告警和 resolved** 人工確認 Telegram 收到： 1. `SentryDown` 告警卡片 2. 自動修復成功通知（若 auto_repair 成功後有 resolved webhook） - [ ] **Step 15.7: 最終驗收 checklist** ```bash # 完整驗收腳本 cat << 'EOF' > /tmp/final-e2e.sh #!/bin/bash GREEN='\033[0;32m'; RED='\033[0;31m'; NC='\033[0m' PASS=0; FAIL=0 check() { local name="$1"; local cmd="$2"; local expect="$3" result=$(eval "$cmd" 2>/dev/null) if echo "$result" | grep -qE "$expect"; then echo -e "${GREEN}✅ $name${NC}" ((PASS++)) else echo -e "${RED}❌ $name${NC} (got: ${result:0:60})" ((FAIL++)) fi } echo "=== 全系統自愈閉環 E2E 驗收 ===" # Sprint 1 check "Prometheus rules >30" "ssh wooo@192.168.0.110 'curl -s http://localhost:9090/api/v1/rules | python3 -c \"import sys,json; r=json.load(sys.stdin); print(sum(len(g[chr(39)rules chr(39)]) for g in r[chr(39)data chr(39)][chr(39)groups chr(39)]))\"'" "[3-9][0-9]" check "SentryDown rule exists" "ssh wooo@192.168.0.110 'curl -s http://localhost:9090/api/v1/rules | python3 -c \"import sys,json; r=json.load(sys.stdin); names=[x[chr(39)name chr(39)] for g in r[chr(39)data chr(39)][chr(39)groups chr(39)] for x in g[chr(39)rules chr(39)]]; print(chr(39)OK chr(39) if chr(39)SentryDown chr(39) in names else chr(39)MISS chr(39))\"'" "OK" check "110 Sentry running" "curl -s -o /dev/null -w '%{http_code}' --max-time 10 http://192.168.0.110:9000/" "200|302|400" check "Alert chain E2E" "curl -s -X POST http://192.168.0.121:32334/api/v1/webhooks/alertmanager -H 'Content-Type: application/json' -d '{\"receiver\":\"test\",\"status\":\"firing\",\"alerts\":[{\"status\":\"firing\",\"labels\":{\"alertname\":\"FinalE2ETest\",\"severity\":\"info\"},\"annotations\":{\"summary\":\"Final E2E test\"},\"startsAt\":\"2026-04-05T00:00:00Z\",\"endsAt\":\"0001-01-01T00:00:00Z\",\"generatorURL\":\"\"}],\"groupLabels\":{},\"commonLabels\":{},\"commonAnnotations\":{},\"externalURL\":\"\",\"version\":\"4\",\"groupKey\":\"e2e\"}'" '"success":true' # Sprint 3 check "SSH key mounted in API pod" "ssh wooo@192.168.0.120 'kubectl exec -n awoooi-prod deploy/awoooi-api -- ls /etc/repair-ssh/id_ed25519'" "id_ed25519" check "SSH health 110" "ssh wooo@192.168.0.120 'kubectl exec -n awoooi-prod deploy/awoooi-api -- ssh -i /etc/repair-ssh/id_ed25519 -o StrictHostKeyChecking=accept-new -o BatchMode=yes wooo@192.168.0.110 health'" "REPAIR_BOT_HEALTHY" echo "" echo "=== 結果: ${PASS} 通過, ${FAIL} 失敗 ===" [ $FAIL -eq 0 ] && echo -e "${GREEN}🎉 全系統自愈閉環驗收通過！${NC}" || echo -e "${RED}⚠️ ${FAIL} 項失敗${NC}" EOF bash /tmp/final-e2e.sh ``` --- ## Self-Review ### Spec 覆蓋確認 | Spec 章節 | 對應 Task | |----------|---------| | S0 統一標籤規範 | Task 1 (alerts-unified.yml 內建標籤) | | Sprint 1: Prometheus 規則部署 | Task 1 + Task 2 | | Sprint 1: CD 自動同步 | Task 3 | | Sprint 1: Sentry 啟動 | Task 4 | | SOP v4.0 更新 | Task 5 | | Sprint 1 驗收 | Task 6 | | Sprint 2: SigNoz log alert | Task 7 | | Sprint 2: Sentry SDK tags | Task 8 | | Sprint 3: 修復腳本白名單 | Task 9 | | Sprint 3: SSH key 基礎設施 | Task 10 | | Sprint 3: HostRepairAgent | Task 11 | | Sprint 3: auto_repair 整合 | Task 12 | | Sprint 3: Pod 掛載 SSH key | Task 13 | | Sprint 3: Playbooks | Task 14 | | Sprint 3: E2E 閉環驗證 | Task 15 | 所有 spec 章節都有對應 Task。✅ ### 型別一致性 - `HostRepairResult` 在 Task 11 定義，Task 12 使用 `result.success` / `result.error` ✅ - `ActionType.SSH_COMMAND` 在 Task 12 定義，Task 14 的 Playbook seed 使用 `"ssh_command"` ✅ - `get_host_repair_agent()` 在 Task 11 定義，Task 12 使用 ✅