fix(aiops): route backup decisions through ssh
This commit is contained in:
Your Name
@@ -12,19 +12,31 @@ Live e2e 用 `HostBackupFailed` 打 Alertmanager 後發現 aged backup 告警會
|
||||
|
||||
### 完成
|
||||
- `_should_use_alertmanager_rule_first()` / `_should_bypass_alertmanager_llm()` 納入 `backup_failure`，備份失敗 YAML `SSH_DIAGNOSE` 不再被 LLM 覆蓋成 K8s 動作。
|
||||
- `DecisionManager` SSH route 與 `AutoRepairService` 分類對齊：`backup_failure` 非 kubectl action 先走 SSH MCP，不再落入 `parse_kubectl_action()` 後被 `forbidden_shell_metachar` 擋下。
|
||||
- `DecisionManager` host/backup K8s block 納入 `backup_failure`，若 LLM 或 Playbook 產生 kubectl 動作，直接走 emergency escalation，而不是對備份告警誤做 K8s 修復。
|
||||
- `AutoRepairService` 追加 host/backup Playbook guard：主機/備份 incident 若匹配到 K8s rollout 類 Playbook，阻擋為 `HOST_BACKUP_K8S_PLAYBOOK`，改走緊急介入。
|
||||
- `AutoRepairService` post-verification rollback guard：host/backup 或非 K8s Playbook 驗證失敗時，不再合成 `kubectl rollout restart deployment/{target}`，改走 emergency escalation，且不自動 resolve incident。
|
||||
- `EmergencyEscalationService` 沿用既有 `APPROVAL_ESCALATED` DB enum 寫 AOL，避免緊急通道因新 enum 未 migration 而留痕失敗。
|
||||
- 補 `phase25_knowledge_enum_names.sql`，讓 `AUTO_RUNBOOK` / `ANTI_PATTERN` enum name 可寫入 PG，修復 auto runbook KM 沉澱失敗。
|
||||
- `NodeExporterDown` Prometheus rule `auto_repair` 改為 `true`，與 YAML rule catalog 的 exporter restart 策略一致。
|
||||
- `awoooi-executor` RBAC 補 backup/DR 診斷權限：PVC、Jobs/CronJobs、Velero resources read-only，以及 StatefulSet/DaemonSet safe rollout patch。
|
||||
- NetworkPolicy 補 K3s master/worker `22/tcp` egress，讓 SSH MCP 可以覆蓋 120/121，不只 110/188。
|
||||
- Telegram category buttons 補 provider alias 正規化：`k8s` → `kubernetes`、`ssh` → `ssh_host`，避免按鈕畫出來後 dispatcher 找不到 MCP provider。
|
||||
- `backup_failure` 補三個 read-only 診斷按鈕：查主機磁碟、查備份 Job、查 Velero；備份告警不再只有通用批准/拒絕/詳情。
|
||||
- 補 `backup_failure` NO_ACTION / SSH_DIAGNOSE 單元測試。
|
||||
|
||||
### 驗證
|
||||
- `python3 -m py_compile apps/api/src/api/v1/webhooks.py` 通過。
|
||||
- `python3 -m py_compile apps/api/src/services/decision_manager.py apps/api/src/services/callback_dispatcher.py` 通過。
|
||||
- `cd apps/api && pytest tests/test_alertmanager_rule_bypass.py tests/test_telegram_ai_automation_block.py tests/test_ai_router_diagnose_fallback.py -q` → 24 passed。
|
||||
- `cd apps/api && pytest tests/test_auto_repair_service.py tests/test_alertmanager_rule_bypass.py -q` → 27 passed。
|
||||
- `cd apps/api && pytest tests/test_auto_repair_service.py tests/test_alertmanager_rule_bypass.py -q` → 29 passed。
|
||||
- `cd apps/api && pytest tests/test_alertmanager_rule_bypass.py tests/test_callback_dispatcher.py tests/test_telegram_button_consistency.py -q` → 56 passed。
|
||||
- YAML parse `ops/monitoring/alerts-unified.yml`、`apps/api/alert_rules.yaml` 通過。
|
||||
- YAML parse `callback_action_spec.yaml`、`07-rbac.yaml`、`02-network-policy.yaml` 通過。
|
||||
- Live Secret/mount 檢查：`ssh-mcp-key`、`awoooi-repair-ssh-key`、`awoooi-repair-known-hosts` 存在且掛載可讀。
|
||||
- Live SSH MCP key 檢查：`wooo@192.168.0.110`、`ollama@192.168.0.188` OK；`wooo@192.168.0.120/121` 已通過 host key，但 remote `authorized_keys` 尚未納入該公鑰，回 `Permission denied (publickey,password)`。
|
||||
- Live RBAC apply 被 Argo 依 Git 狀態拉回；`07-rbac.yaml` 需推上 Gitea 由 Argo 同步後再驗 `can-i`。
|
||||
|
||||
## 2026-04-30 | ADR-104 Playbook 版本化 lineage
|
||||
|
||||
|
||||
@@ -140,6 +140,48 @@ MoWoooWorkDown → Jaccard 匹配 momo-app-down-repair → SSH ollama@192.168.0.
|
||||
|
||||
---
|
||||
|
||||
## Appendix B — Backup Failure Route Parity (2026-05-01)
|
||||
|
||||
`HostBackupFailed` / backup 類告警的 `alert_category` 是 `backup_failure`。它必須在所有 host-layer 自動化路徑與 `host_resource` 同級處理：
|
||||
|
||||
| Layer | 必須行為 |
|
||||
|-------|----------|
|
||||
| Alertmanager rule-first | YAML `SSH_DIAGNOSE` / `NO_ACTION` 不進 LLM 覆寫 |
|
||||
| AutoRepairService | `backup_failure` 視為 host-layer，拒絕 K8s Playbook fallback |
|
||||
| DecisionManager | 非 `kubectl` 動作在 kubectl parser 前路由 SSH MCP |
|
||||
| DecisionManager K8s guard | `backup_failure` 產生 `kubectl` 時降級 emergency escalation |
|
||||
| Telegram buttons | `backup_failure` 顯示只讀診斷按鈕：主機磁碟、備份 Job、Velero 狀態 |
|
||||
|
||||
2026-05-01 根因：DecisionManager SSH route 只含 `infrastructure` / `host_resource`，漏掉 `backup_failure`，導致 `ssh 192.168.0.110 '...;...'` 類只讀診斷動作掉進 `parse_kubectl_action()`，被 `forbidden_shell_metachar` 擋下。
|
||||
|
||||
同日按鈕審計也發現 category button 的 friendly provider name 會漂移：`callback_action_spec.yaml` 使用 `k8s` / `ssh`，但 MCP registry 實際名稱是 `kubernetes` / `ssh_host`。Dispatcher 必須正規化 provider alias，否則卡片按鈕會顯示但執行時變成 `provider_not_found`。
|
||||
|
||||
### Runtime 權限基準
|
||||
|
||||
- K8s Secret:
|
||||
- `awoooi-repair-ssh-key` 掛載 `/etc/repair-ssh/`
|
||||
- `awoooi-repair-known-hosts` 掛載 `/etc/repair-known-hosts/`
|
||||
- `ssh-mcp-key` 掛載 `/run/secrets/ssh_mcp_key` 與 `/etc/ssh-mcp/known_hosts`
|
||||
- Remote `authorized_keys`:
|
||||
- `wooo@192.168.0.110`
|
||||
- `wooo@192.168.0.120`
|
||||
- `wooo@192.168.0.121`
|
||||
- `ollama@192.168.0.188`
|
||||
- NetworkPolicy egress:
|
||||
- `192.168.0.110:22`
|
||||
- `192.168.0.120:22`
|
||||
- `192.168.0.121:22`
|
||||
- `192.168.0.188:22`
|
||||
- `awoooi-executor` RBAC:
|
||||
- read `jobs.batch`, `cronjobs.batch`
|
||||
- read `persistentvolumeclaims`
|
||||
- read Velero `backups`, `backupstoragelocations`, `backuprepositories`, `podvolumebackups`, `podvolumerestores`, `restores`, `schedules`
|
||||
- patch `statefulsets.apps` / `daemonsets.apps` only for safe rollout restart
|
||||
|
||||
If SSH MCP fails, the incident must not silently become a manual approval card; it must raise the emergency intervention path with the exact SSH failure reason when available.
|
||||
|
||||
---
|
||||
|
||||
## 首席架構師 Review 記錄 (2026-04-05)
|
||||
|
||||
評分：**72/100 → 修正後 88/100**
|
||||
|
||||
Reference in New Issue
Block a user