fix(aiops-p0): 六大病根 P0 全面修復(ADR-092 B4)

【P0.1】knowledge_extractor_service.py:210 — AttributeError 修復
- Signal.description 欄位不存在(100% 失敗,KM 每天+5 根因)
- 改用 alert_name + annotations.summary 拼接文字

【P0.2+P0.3】Gate 9+11 唯讀指令鬆綁
- blast_radius_calculator: kubectl get/top/describe/logs/version → score=1(非 50)
- operation_parser: 增加 INVESTIGATE 類型識別(唯讀 kubectl 不回 None)
- executor.py: OperationType 新增 INVESTIGATE enum
- approval_execution.py: INVESTIGATE 路徑直接呼叫 execute_kubectl_command

【P0.4】MCP SSH/K8s Provider 修復
- decision_manager: params= → parameters=(符合 MCPToolProvider.execute 簽名)
- decision_manager: MCPToolResult .get() → .success/.output(dataclass 用法)
- decision_manager + ssh_provider: 補入 hosts 120/121(原 default 缺失)
- auto_approve: phase2_agent_debate source bypass confidence 閾值

【P0.5】告警規則語義矛盾修復
- alert_rules.yaml: 8 條 kubectl 查詢規則 RESTART_DEPLOYMENT → NO_ACTION
  (CrashLoopBackOff/PostgreSQL 連線/慢查詢/MinIO 磁碟/K3s 節點/告警鏈路/SSL/CoreDNS 等)
- incident_service.py: cAdvisor/CoreDNS 從 general 拆出獨立分類

【P0.6】proactive_inspector 動態基線 PromQL 全修
- 5 個 MONITORED_METRICS PromQL 全部修正(cadvisor label/datname/blackbox)
- db_connection_pool: datname="awoooi" → "awoooi_prod"
- http_error_rate: 無效 http_requests_total → blackbox probe_success
- cpu/memory: namespace label → name=~"k8s_api_awoooi-api.*"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Your Name
2026-04-24 15:32:23 +08:00
parent 45dbe07188
commit 7f4088bcd0
11 changed files with 146 additions and 70 deletions

View File

@@ -232,7 +232,7 @@ rules:
response:
action_title: "診斷 {target} CrashLoop 根因"
description: "⚙️ 規則匹配: {target} 進入 CrashLoopBackOff需檢查啟動錯誤日誌。"
suggested_action: RESTART_DEPLOYMENT
suggested_action: NO_ACTION
kubectl_command: "kubectl logs {target} -n {namespace} --previous --tail=50"
estimated_downtime: "依根因而定"
risk: critical
@@ -315,7 +315,7 @@ rules:
response:
action_title: "清理 PostgreSQL 閒置連線"
description: "⚙️ 規則匹配: PostgreSQL 連線池使用率過高,可能導致新請求被拒絕。"
suggested_action: RESTART_DEPLOYMENT
suggested_action: NO_ACTION
kubectl_command: "kubectl exec -n {namespace} deployment/postgresql -- psql -U postgres -c 'SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = ''idle'' AND state_change < NOW() - INTERVAL ''5 minutes'';'"
estimated_downtime: "0"
risk: critical
@@ -342,7 +342,7 @@ rules:
response:
action_title: "診斷 PostgreSQL 慢查詢 + 索引優化"
description: "⚙️ 規則匹配: PostgreSQL 存在慢查詢或鎖等待,影響系統整體性能。"
suggested_action: RESTART_DEPLOYMENT
suggested_action: NO_ACTION
kubectl_command: "kubectl exec -n {namespace} deployment/postgresql -- psql -U postgres -c 'SELECT pid, query, state, wait_event_type, wait_event FROM pg_stat_activity WHERE state != ''idle'' ORDER BY query_start;'"
estimated_downtime: "0"
risk: medium
@@ -448,7 +448,7 @@ rules:
response:
action_title: "清理 MinIO 過期資料 on {host}"
description: "⚙️ 規則匹配: MinIO 磁碟使用率過高,需清理舊資料或擴展儲存空間。"
suggested_action: RESTART_DEPLOYMENT
suggested_action: NO_ACTION
kubectl_command: "ssh {host} 'df -h /data/minio && du -sh /data/minio/* | sort -rh | head -10'"
estimated_downtime: "0"
risk: critical
@@ -503,7 +503,7 @@ rules:
response:
action_title: "確認 K3s 節點 {target} 狀態"
description: "⚙️ 規則匹配: K3s 節點下線,影響叢集可用性和 Pod 調度。"
suggested_action: RESTART_DEPLOYMENT
suggested_action: NO_ACTION
kubectl_command: "kubectl get nodes -o wide && kubectl describe node {target}"
estimated_downtime: "依節點恢復時間"
risk: critical
@@ -562,7 +562,7 @@ rules:
response:
action_title: "診斷告警鏈路中斷"
description: "⚙️ 規則匹配: 告警鏈路異常,可能導致真實告警無法送達 Telegram。"
suggested_action: RESTART_DEPLOYMENT
suggested_action: NO_ACTION
kubectl_command: "kubectl get pods -n monitoring && curl -s http://192.168.0.120:9093/api/v1/status | jq '.data.uptime'"
estimated_downtime: "監控盲區持續中"
risk: critical
@@ -593,7 +593,7 @@ rules:
response:
action_title: "確認 NVIDIA API 熔斷狀態"
description: "⚙️ 規則匹配: NVIDIA/Nemotron 熔斷器開啟或錯誤率過高AI Router 已自動降級。"
suggested_action: RESTART_DEPLOYMENT
suggested_action: NO_ACTION
kubectl_command: "curl -s http://192.168.0.125:32334/api/v1/ai-router/status | jq '.providers'"
estimated_downtime: "0 (已自動 fallback)"
risk: medium
@@ -775,12 +775,12 @@ rules:
response:
action_title: "重新啟動 {target} 服務"
description: "⚙️ 規則匹配: {target} 發生異常,需進一步診斷確認根因。"
suggested_action: RESTART_DEPLOYMENT
kubectl_command: "kubectl rollout restart deployment/{target} -n {namespace}"
estimated_downtime: "5-15 min"
suggested_action: NO_ACTION
kubectl_command: ""
estimated_downtime: "N/A"
risk: medium
responsibility: COLLAB
responsibility_reasoning: "告警資訊不足以判定單一責任團隊,建議多團隊協同排查"
secondary_teams: [BE, INFRA]
optimization: []
reasoning: "[規則匹配] 根據告警先重啟恢復服務,同時安排深入診斷。"
reasoning: "[規則匹配] 未知告警類型,無法安全判斷修復動作,由人工或 LLM 診斷後決策。"