feat(api): ADR-030 Phase 2 診斷資料收集強化

實作智能自動修復系統的資料收集層:

1. k8s_diagnostics.py - K8s 診斷服務
   - Pod Events/Logs/ResourceUsage 收集
   - CrashLoopBackOff/OOM/ImagePull 偵測
   - 非同步並行收集 + 錯誤容忍

2. diagnosis_aggregator.py - 診斷聚合器
   - 整合 K8s + SignOz + Expert Rules
   - DiagnosisContext 提供結構化 LLM Prompt
   - DiagnosisSignal 信號分析

3. decision_manager.py - 決策引擎整合
   - Step 2.5 加入診斷收集
   - 傳遞 diagnosis_context 給 LLM

4. openclaw.py - LLM Prompt 增強
   - 整合 K8s/SignOz 深度診斷上下文
   - 支援 diagnosis_signals 摘要

ADR-030 架構: 診斷先行,根因分析,非盲目重啟

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
OG T
2026-03-26 21:55:50 +08:00
parent bb6151cf44
commit 60e9538889
4 changed files with 1301 additions and 3 deletions

View File

@@ -31,6 +31,7 @@ from src.core.config import settings
from src.core.redis_client import get_redis
from src.models.incident import Incident
from src.models.playbook import SymptomPattern
from src.services.diagnosis_aggregator import get_diagnosis_aggregator
from src.services.openclaw import get_openclaw
from src.services.playbook_service import get_playbook_service
@@ -600,7 +601,36 @@ class DecisionManager:
)
return expert_result
# Step 3: 準備 LLM 上下文 (含 Expert 診斷)
# Step 2.5: ADR-030 診斷資料收集 (Phase 2)
# 使用 DiagnosisAggregator 收集 K8s + SignOz 診斷資料
diagnosis_context = None
target = incident.affected_services[0] if incident.affected_services else None
if target:
try:
aggregator = get_diagnosis_aggregator()
diagnosis_context = await aggregator.collect_pod_diagnosis(
pod_name=target,
namespace="awoooi-prod",
include_signoz=True,
include_error_logs=True,
expert_match=expert_result,
)
logger.info(
"dual_engine_diagnosis_collected",
incident_id=incident.incident_id,
target=target,
signals_count=len(diagnosis_context.signals),
highest_severity=diagnosis_context.highest_severity.value,
)
except Exception as e:
logger.warning(
"dual_engine_diagnosis_failed",
incident_id=incident.incident_id,
error=str(e),
)
# 診斷收集失敗不影響主流程,繼續使用 expert_result
# Step 3: 準備 LLM 上下文 (含 Expert 診斷 + K8s/SignOz 診斷)
signals_dict = [s.model_dump() for s in incident.signals]
expert_context = {
"initial_diagnosis": expert_result.get("matched_rule"),
@@ -610,6 +640,11 @@ class DecisionManager:
"requires_human_review": expert_result.get("human_review_required", False),
}
# 加入診斷上下文 (如果有)
if diagnosis_context:
expert_context["diagnosis_context"] = diagnosis_context.get_llm_prompt_context()
expert_context["diagnosis_signals"] = [s.to_dict() for s in diagnosis_context.signals]
# Step 4: LLM 分析 (帶上 Expert 上下文)
try:
llm_result, provider, success = await self._openclaw.generate_incident_proposal(