fix(aiops-p0): 六大病根 P0 全面修復(ADR-092 B4)
【P0.1】knowledge_extractor_service.py:210 — AttributeError 修復 - Signal.description 欄位不存在(100% 失敗,KM 每天+5 根因) - 改用 alert_name + annotations.summary 拼接文字 【P0.2+P0.3】Gate 9+11 唯讀指令鬆綁 - blast_radius_calculator: kubectl get/top/describe/logs/version → score=1(非 50) - operation_parser: 增加 INVESTIGATE 類型識別(唯讀 kubectl 不回 None) - executor.py: OperationType 新增 INVESTIGATE enum - approval_execution.py: INVESTIGATE 路徑直接呼叫 execute_kubectl_command 【P0.4】MCP SSH/K8s Provider 修復 - decision_manager: params= → parameters=(符合 MCPToolProvider.execute 簽名) - decision_manager: MCPToolResult .get() → .success/.output(dataclass 用法) - decision_manager + ssh_provider: 補入 hosts 120/121(原 default 缺失) - auto_approve: phase2_agent_debate source bypass confidence 閾值 【P0.5】告警規則語義矛盾修復 - alert_rules.yaml: 8 條 kubectl 查詢規則 RESTART_DEPLOYMENT → NO_ACTION (CrashLoopBackOff/PostgreSQL 連線/慢查詢/MinIO 磁碟/K3s 節點/告警鏈路/SSL/CoreDNS 等) - incident_service.py: cAdvisor/CoreDNS 從 general 拆出獨立分類 【P0.6】proactive_inspector 動態基線 PromQL 全修 - 5 個 MONITORED_METRICS PromQL 全部修正(cadvisor label/datname/blackbox) - db_connection_pool: datname="awoooi" → "awoooi_prod" - http_error_rate: 無效 http_requests_total → blackbox probe_success - cpu/memory: namespace label → name=~"k8s_api_awoooi-api.*" Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -232,7 +232,7 @@ rules:
|
||||
response:
|
||||
action_title: "診斷 {target} CrashLoop 根因"
|
||||
description: "⚙️ 規則匹配: {target} 進入 CrashLoopBackOff,需檢查啟動錯誤日誌。"
|
||||
suggested_action: RESTART_DEPLOYMENT
|
||||
suggested_action: NO_ACTION
|
||||
kubectl_command: "kubectl logs {target} -n {namespace} --previous --tail=50"
|
||||
estimated_downtime: "依根因而定"
|
||||
risk: critical
|
||||
@@ -315,7 +315,7 @@ rules:
|
||||
response:
|
||||
action_title: "清理 PostgreSQL 閒置連線"
|
||||
description: "⚙️ 規則匹配: PostgreSQL 連線池使用率過高,可能導致新請求被拒絕。"
|
||||
suggested_action: RESTART_DEPLOYMENT
|
||||
suggested_action: NO_ACTION
|
||||
kubectl_command: "kubectl exec -n {namespace} deployment/postgresql -- psql -U postgres -c 'SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = ''idle'' AND state_change < NOW() - INTERVAL ''5 minutes'';'"
|
||||
estimated_downtime: "0"
|
||||
risk: critical
|
||||
@@ -342,7 +342,7 @@ rules:
|
||||
response:
|
||||
action_title: "診斷 PostgreSQL 慢查詢 + 索引優化"
|
||||
description: "⚙️ 規則匹配: PostgreSQL 存在慢查詢或鎖等待,影響系統整體性能。"
|
||||
suggested_action: RESTART_DEPLOYMENT
|
||||
suggested_action: NO_ACTION
|
||||
kubectl_command: "kubectl exec -n {namespace} deployment/postgresql -- psql -U postgres -c 'SELECT pid, query, state, wait_event_type, wait_event FROM pg_stat_activity WHERE state != ''idle'' ORDER BY query_start;'"
|
||||
estimated_downtime: "0"
|
||||
risk: medium
|
||||
@@ -448,7 +448,7 @@ rules:
|
||||
response:
|
||||
action_title: "清理 MinIO 過期資料 on {host}"
|
||||
description: "⚙️ 規則匹配: MinIO 磁碟使用率過高,需清理舊資料或擴展儲存空間。"
|
||||
suggested_action: RESTART_DEPLOYMENT
|
||||
suggested_action: NO_ACTION
|
||||
kubectl_command: "ssh {host} 'df -h /data/minio && du -sh /data/minio/* | sort -rh | head -10'"
|
||||
estimated_downtime: "0"
|
||||
risk: critical
|
||||
@@ -503,7 +503,7 @@ rules:
|
||||
response:
|
||||
action_title: "確認 K3s 節點 {target} 狀態"
|
||||
description: "⚙️ 規則匹配: K3s 節點下線,影響叢集可用性和 Pod 調度。"
|
||||
suggested_action: RESTART_DEPLOYMENT
|
||||
suggested_action: NO_ACTION
|
||||
kubectl_command: "kubectl get nodes -o wide && kubectl describe node {target}"
|
||||
estimated_downtime: "依節點恢復時間"
|
||||
risk: critical
|
||||
@@ -562,7 +562,7 @@ rules:
|
||||
response:
|
||||
action_title: "診斷告警鏈路中斷"
|
||||
description: "⚙️ 規則匹配: 告警鏈路異常,可能導致真實告警無法送達 Telegram。"
|
||||
suggested_action: RESTART_DEPLOYMENT
|
||||
suggested_action: NO_ACTION
|
||||
kubectl_command: "kubectl get pods -n monitoring && curl -s http://192.168.0.120:9093/api/v1/status | jq '.data.uptime'"
|
||||
estimated_downtime: "監控盲區持續中"
|
||||
risk: critical
|
||||
@@ -593,7 +593,7 @@ rules:
|
||||
response:
|
||||
action_title: "確認 NVIDIA API 熔斷狀態"
|
||||
description: "⚙️ 規則匹配: NVIDIA/Nemotron 熔斷器開啟或錯誤率過高,AI Router 已自動降級。"
|
||||
suggested_action: RESTART_DEPLOYMENT
|
||||
suggested_action: NO_ACTION
|
||||
kubectl_command: "curl -s http://192.168.0.125:32334/api/v1/ai-router/status | jq '.providers'"
|
||||
estimated_downtime: "0 (已自動 fallback)"
|
||||
risk: medium
|
||||
@@ -775,12 +775,12 @@ rules:
|
||||
response:
|
||||
action_title: "重新啟動 {target} 服務"
|
||||
description: "⚙️ 規則匹配: {target} 發生異常,需進一步診斷確認根因。"
|
||||
suggested_action: RESTART_DEPLOYMENT
|
||||
kubectl_command: "kubectl rollout restart deployment/{target} -n {namespace}"
|
||||
estimated_downtime: "5-15 min"
|
||||
suggested_action: NO_ACTION
|
||||
kubectl_command: ""
|
||||
estimated_downtime: "N/A"
|
||||
risk: medium
|
||||
responsibility: COLLAB
|
||||
responsibility_reasoning: "告警資訊不足以判定單一責任團隊,建議多團隊協同排查"
|
||||
secondary_teams: [BE, INFRA]
|
||||
optimization: []
|
||||
reasoning: "[規則匹配] 根據告警先重啟恢復服務,同時安排深入診斷。"
|
||||
reasoning: "[規則匹配] 未知告警類型,無法安全判斷修復動作,由人工或 LLM 診斷後決策。"
|
||||
|
||||
Reference in New Issue
Block a user