fix(aiops): route backup decisions through ssh
Some checks failed
CD Pipeline / tests (push) Successful in 1m35s
Code Review / ai-code-review (push) Successful in 34s
CD Pipeline / post-deploy-checks (push) Has been cancelled
CD Pipeline / build-and-deploy (push) Has been cancelled

This commit is contained in:
Your Name
2026-05-01 12:50:01 +08:00
parent 337bcb912e
commit 11673d80ea
11 changed files with 276 additions and 19 deletions

View File

@@ -140,6 +140,48 @@ MoWoooWorkDown → Jaccard 匹配 momo-app-down-repair → SSH ollama@192.168.0.
---
## Appendix B — Backup Failure Route Parity (2026-05-01)
`HostBackupFailed` / backup 類告警的 `alert_category``backup_failure`。它必須在所有 host-layer 自動化路徑與 `host_resource` 同級處理:
| Layer | 必須行為 |
|-------|----------|
| Alertmanager rule-first | YAML `SSH_DIAGNOSE` / `NO_ACTION` 不進 LLM 覆寫 |
| AutoRepairService | `backup_failure` 視為 host-layer拒絕 K8s Playbook fallback |
| DecisionManager | 非 `kubectl` 動作在 kubectl parser 前路由 SSH MCP |
| DecisionManager K8s guard | `backup_failure` 產生 `kubectl` 時降級 emergency escalation |
| Telegram buttons | `backup_failure` 顯示只讀診斷按鈕:主機磁碟、備份 Job、Velero 狀態 |
2026-05-01 根因DecisionManager SSH route 只含 `infrastructure` / `host_resource`,漏掉 `backup_failure`,導致 `ssh 192.168.0.110 '...;...'` 類只讀診斷動作掉進 `parse_kubectl_action()`,被 `forbidden_shell_metachar` 擋下。
同日按鈕審計也發現 category button 的 friendly provider name 會漂移:`callback_action_spec.yaml` 使用 `k8s` / `ssh`,但 MCP registry 實際名稱是 `kubernetes` / `ssh_host`。Dispatcher 必須正規化 provider alias否則卡片按鈕會顯示但執行時變成 `provider_not_found`
### Runtime 權限基準
- K8s Secret:
- `awoooi-repair-ssh-key` 掛載 `/etc/repair-ssh/`
- `awoooi-repair-known-hosts` 掛載 `/etc/repair-known-hosts/`
- `ssh-mcp-key` 掛載 `/run/secrets/ssh_mcp_key``/etc/ssh-mcp/known_hosts`
- Remote `authorized_keys`:
- `wooo@192.168.0.110`
- `wooo@192.168.0.120`
- `wooo@192.168.0.121`
- `ollama@192.168.0.188`
- NetworkPolicy egress:
- `192.168.0.110:22`
- `192.168.0.120:22`
- `192.168.0.121:22`
- `192.168.0.188:22`
- `awoooi-executor` RBAC:
- read `jobs.batch`, `cronjobs.batch`
- read `persistentvolumeclaims`
- read Velero `backups`, `backupstoragelocations`, `backuprepositories`, `podvolumebackups`, `podvolumerestores`, `restores`, `schedules`
- patch `statefulsets.apps` / `daemonsets.apps` only for safe rollout restart
If SSH MCP fails, the incident must not silently become a manual approval card; it must raise the emergency intervention path with the exact SSH failure reason when available.
---
## 首席架構師 Review 記錄 (2026-04-05)
評分:**72/100 → 修正後 88/100**