fix(aiops): route backup decisions through ssh
This commit is contained in:
@@ -140,6 +140,48 @@ MoWoooWorkDown → Jaccard 匹配 momo-app-down-repair → SSH ollama@192.168.0.
|
||||
|
||||
---
|
||||
|
||||
## Appendix B — Backup Failure Route Parity (2026-05-01)
|
||||
|
||||
`HostBackupFailed` / backup 類告警的 `alert_category` 是 `backup_failure`。它必須在所有 host-layer 自動化路徑與 `host_resource` 同級處理:
|
||||
|
||||
| Layer | 必須行為 |
|
||||
|-------|----------|
|
||||
| Alertmanager rule-first | YAML `SSH_DIAGNOSE` / `NO_ACTION` 不進 LLM 覆寫 |
|
||||
| AutoRepairService | `backup_failure` 視為 host-layer,拒絕 K8s Playbook fallback |
|
||||
| DecisionManager | 非 `kubectl` 動作在 kubectl parser 前路由 SSH MCP |
|
||||
| DecisionManager K8s guard | `backup_failure` 產生 `kubectl` 時降級 emergency escalation |
|
||||
| Telegram buttons | `backup_failure` 顯示只讀診斷按鈕:主機磁碟、備份 Job、Velero 狀態 |
|
||||
|
||||
2026-05-01 根因:DecisionManager SSH route 只含 `infrastructure` / `host_resource`,漏掉 `backup_failure`,導致 `ssh 192.168.0.110 '...;...'` 類只讀診斷動作掉進 `parse_kubectl_action()`,被 `forbidden_shell_metachar` 擋下。
|
||||
|
||||
同日按鈕審計也發現 category button 的 friendly provider name 會漂移:`callback_action_spec.yaml` 使用 `k8s` / `ssh`,但 MCP registry 實際名稱是 `kubernetes` / `ssh_host`。Dispatcher 必須正規化 provider alias,否則卡片按鈕會顯示但執行時變成 `provider_not_found`。
|
||||
|
||||
### Runtime 權限基準
|
||||
|
||||
- K8s Secret:
|
||||
- `awoooi-repair-ssh-key` 掛載 `/etc/repair-ssh/`
|
||||
- `awoooi-repair-known-hosts` 掛載 `/etc/repair-known-hosts/`
|
||||
- `ssh-mcp-key` 掛載 `/run/secrets/ssh_mcp_key` 與 `/etc/ssh-mcp/known_hosts`
|
||||
- Remote `authorized_keys`:
|
||||
- `wooo@192.168.0.110`
|
||||
- `wooo@192.168.0.120`
|
||||
- `wooo@192.168.0.121`
|
||||
- `ollama@192.168.0.188`
|
||||
- NetworkPolicy egress:
|
||||
- `192.168.0.110:22`
|
||||
- `192.168.0.120:22`
|
||||
- `192.168.0.121:22`
|
||||
- `192.168.0.188:22`
|
||||
- `awoooi-executor` RBAC:
|
||||
- read `jobs.batch`, `cronjobs.batch`
|
||||
- read `persistentvolumeclaims`
|
||||
- read Velero `backups`, `backupstoragelocations`, `backuprepositories`, `podvolumebackups`, `podvolumerestores`, `restores`, `schedules`
|
||||
- patch `statefulsets.apps` / `daemonsets.apps` only for safe rollout restart
|
||||
|
||||
If SSH MCP fails, the incident must not silently become a manual approval card; it must raise the emergency intervention path with the exact SSH failure reason when available.
|
||||
|
||||
---
|
||||
|
||||
## 首席架構師 Review 記錄 (2026-04-05)
|
||||
|
||||
評分:**72/100 → 修正後 88/100**
|
||||
|
||||
Reference in New Issue
Block a user