1104 lines
44 KiB
Markdown
1104 lines
44 KiB
Markdown
# ADR-030: 智能自動修復系統 (Intelligent Auto-Remediation)
|
||
|
||
**狀態**: 已實作 ✅
|
||
**日期**: 2026-03-27 (台北時區)
|
||
**決策者**: 統帥
|
||
**觸發**: Expert System 只會重啟，缺乏根因診斷能力
|
||
|
||
---
|
||
|
||
## 一、問題陳述
|
||
|
||
### 1.1 當前痛點
|
||
|
||
| 問題 | 影響 | 根因 |
|
||
|------|------|------|
|
||
| Expert System 盲目重啟 | 治標不治本，問題反覆發生 | 規則太粗糙，只看關鍵字 |
|
||
| 測試資源告警轟炸 | 浪費人力審核假告警 | 沒有資源分類過濾 |
|
||
| LLM 超時導致 fallback | 使用者看到的都是 Expert 建議 | Ollama CPU 太慢 |
|
||
| 缺乏學習機制 | 相同問題重複發生 | 反饋沒有回饋到決策 |
|
||
| 沒有自動修復 | 所有事件都需人工審核 | 缺乏信任度機制 |
|
||
|
||
### 1.2 目標
|
||
|
||
1. **根因優先**: 先診斷問題，再決定行動
|
||
2. **智能分類**: 自動過濾測試/臨時資源
|
||
3. **持續學習**: 從執行結果中學習
|
||
4. **漸進自動化**: 低風險操作可自動執行
|
||
|
||
---
|
||
|
||
## 二、完整解決方案架構
|
||
|
||
### 2.1 四層診斷引擎
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ Layer 4: Decision Layer │
|
||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||
│ │ Auto-Approve Engine (Phase 4) │ │
|
||
│ │ - 信任度 > 90% + 風險 LOW → 自動執行 │ │
|
||
│ │ - 重複 Playbook 成功 3+ 次 → 自動放行 │ │
|
||
│ └─────────────────────────────────────────────────────────┘ │
|
||
└─────────────────────────────────────────────────────────────────┘
|
||
↑
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ Layer 3: Intelligence Layer │
|
||
│ ┌──────────────────┐ ┌──────────────────┐ ┌───────────────┐ │
|
||
│ │ LLM Engine │ │ Playbook Engine │ │ ML Anomaly │ │
|
||
│ │ (Gemini/Claude) │ │ (RAG + Vector) │ │ Detection │ │
|
||
│ │ │ │ │ │ (Future) │ │
|
||
│ │ - 根因分析 │ │ - 歷史案例匹配 │ │ │ │
|
||
│ │ - 修復建議 │ │ - 成功率評估 │ │ - 異常偵測 │ │
|
||
│ │ - 預防措施 │ │ - 信任度計算 │ │ - 趨勢預測 │ │
|
||
│ └──────────────────┘ └──────────────────┘ └───────────────┘ │
|
||
└─────────────────────────────────────────────────────────────────┘
|
||
↑
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ Layer 2: Diagnosis Layer │
|
||
│ ┌──────────────────────────────────────────────────────────┐ │
|
||
│ │ Expert System v2 (智能診斷規則引擎) │ │
|
||
│ │ │ │
|
||
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ │
|
||
│ │ │ Classifier │ │ Diagnoser │ │ Action Recommender │ │ │
|
||
│ │ │ 問題分類 │ │ 根因診斷 │ │ 行動建議 │ │ │
|
||
│ │ │ │ │ │ │ │ │ │
|
||
│ │ │ - OOM │ │ - 日誌分析 │ │ - 診斷指令 │ │ │
|
||
│ │ │ - CrashLoop │ │ - 指標相關 │ │ - 修復建議 │ │ │
|
||
│ │ │ - Network │ │ - 配置檢查 │ │ - 風險評估 │ │ │
|
||
│ │ │ - Config │ │ │ │ │ │ │
|
||
│ │ └─────────────┘ └─────────────┘ └─────────────────────┘ │ │
|
||
│ └──────────────────────────────────────────────────────────┘ │
|
||
└─────────────────────────────────────────────────────────────────┘
|
||
↑
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ Layer 1: Data Collection Layer │
|
||
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌─────────────┐ │
|
||
│ │ Alertmgr │ │ SignOz │ │ K8s Events │ │ Pod Logs │ │
|
||
│ │ Webhook │ │ Metrics │ │ kubectl │ │ Stern/Log │ │
|
||
│ │ │ │ │ │ │ │ │ │
|
||
│ │ - 告警 │ │ - RPS │ │ - Events │ │ - Error │ │
|
||
│ │ - 嚴重度 │ │ - Error% │ │ - Status │ │ - Panic │ │
|
||
│ │ - 影響 │ │ - P99 │ │ - Restart │ │ - OOM │ │
|
||
│ └────────────┘ └────────────┘ └────────────┘ └─────────────┘ │
|
||
└─────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### 2.2 完整診斷決策流程
|
||
|
||
```
|
||
┌─────────────────────┐
|
||
│ Incident 進入 │
|
||
└──────────┬──────────┘
|
||
│
|
||
┌──────────▼──────────┐
|
||
│ Step 1: 資源分類 │
|
||
│ 是測試/臨時資源嗎？ │
|
||
└──────────┬──────────┘
|
||
│
|
||
┌────────────────┼────────────────┐
|
||
│ Yes │ │ No
|
||
▼ │ ▼
|
||
┌─────────────────┐ │ ┌─────────────────┐
|
||
│ 標記為測試資源 │ │ │ Step 2: 資料收集│
|
||
│ 建議手動清理 │ │ │ - SignOz 指標 │
|
||
│ 不觸發 LLM │ │ │ - K8s Events │
|
||
└─────────────────┘ │ │ - Pod 日誌 │
|
||
│ └────────┬────────┘
|
||
│ │
|
||
│ ┌────────▼────────┐
|
||
│ │ Step 3: 問題分類│
|
||
│ │ Expert Classifier│
|
||
│ └────────┬────────┘
|
||
│ │
|
||
┌────────────────┴───────────────┴────────────────┐
|
||
│ │
|
||
▼ ▼ ▼ ▼
|
||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||
│ OOM/Memory │ │ CrashLoop │ │ Network/DNS │ │ Unknown │
|
||
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘
|
||
│ │ │ │
|
||
▼ ▼ ▼ ▼
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Step 4: 根因診斷 (Expert Diagnoser) │
|
||
│ - 執行診斷指令 (kubectl describe/logs/top) │
|
||
│ - 關聯 SignOz 指標 (RPS, Error%, P99) │
|
||
│ - 檢查歷史 Playbook 匹配 │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
│
|
||
┌──────────▼──────────┐
|
||
│ Step 5: Playbook 檢索│
|
||
│ 有匹配的歷史案例嗎？ │
|
||
└──────────┬──────────┘
|
||
│
|
||
┌────────────────┼────────────────┐
|
||
│ Yes (>85%) │ │ No
|
||
▼ │ ▼
|
||
┌─────────────────┐ │ ┌─────────────────┐
|
||
│ 使用 Playbook │ │ │ Step 6: LLM 分析│
|
||
│ 成功率 + 信任度 │ │ │ 含診斷上下文 │
|
||
└────────┬────────┘ │ └────────┬────────┘
|
||
│ │ │
|
||
│ ┌──────────┴───────────────┤
|
||
│ │ │
|
||
▼ ▼ ▼
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Step 7: 信任度評估 (Trust Engine) │
|
||
│ - 歷史成功率 (Playbook) │
|
||
│ - 操作風險等級 │
|
||
│ - 影響範圍 (爆炸半徑) │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
│
|
||
┌──────────▼──────────┐
|
||
│ Step 8: 自動執行？ │
|
||
│ 信任度 > 90% │
|
||
│ AND 風險 = LOW │
|
||
│ AND 成功率 > 95% │
|
||
└──────────┬──────────┘
|
||
│
|
||
┌────────────────┼────────────────┐
|
||
│ Yes │ │ No
|
||
▼ │ ▼
|
||
┌─────────────────┐ │ ┌─────────────────┐
|
||
│ 自動執行 │ │ │ 人工審核 │
|
||
│ 記錄到 AuditLog │ │ │ Telegram/Web │
|
||
└────────┬────────┘ │ └────────┬────────┘
|
||
│ │ │
|
||
└─────────────────┴───────────────┘
|
||
│
|
||
┌──────────▼──────────┐
|
||
│ Step 9: 執行與驗證 │
|
||
│ K8s Executor │
|
||
└──────────┬──────────┘
|
||
│
|
||
┌──────────▼──────────┐
|
||
│ Step 10: 結果反饋 │
|
||
│ - 更新 Playbook 成功率│
|
||
│ - 調整信任度 │
|
||
│ - 萃取新 Playbook │
|
||
└─────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## 三、實施計畫
|
||
|
||
### Phase 1: 智能診斷基礎 (1-2 週) ✅ 已完成
|
||
|
||
**目標**: 改進 Expert System，從「盲目重啟」變成「根因診斷」
|
||
|
||
| 任務 | 狀態 | 說明 |
|
||
|------|------|------|
|
||
| 測試資源過濾 | ✅ | 自動識別 test/demo/tmp 資源 |
|
||
| 分類規則優化 | ✅ | OOM/CrashLoop/ImagePull 各有對應診斷 |
|
||
| 診斷指令提供 | ✅ | 每個規則包含 kubectl 診斷命令 |
|
||
| 人工標記機制 | ✅ | 未知問題標記 `human_review_required` |
|
||
| LLM 上下文整合 | ✅ | Expert 診斷傳遞給 LLM |
|
||
|
||
**已修改檔案**:
|
||
- `apps/api/src/services/decision_manager.py`
|
||
- `apps/api/src/services/openclaw.py`
|
||
|
||
---
|
||
|
||
### Phase 2: 資料收集強化 (2-3 週)
|
||
|
||
**目標**: 收集更多診斷資料，提供更精確的根因分析
|
||
|
||
#### 2.1 K8s Events 整合
|
||
|
||
```python
|
||
# 新增 apps/api/src/services/k8s_diagnostics.py
|
||
|
||
class K8sDiagnosticsService:
|
||
"""K8s 診斷資料收集"""
|
||
|
||
async def get_pod_events(
|
||
self,
|
||
pod_name: str,
|
||
namespace: str = "awoooi-prod",
|
||
limit: int = 20,
|
||
) -> list[K8sEvent]:
|
||
"""取得 Pod 相關 Events"""
|
||
...
|
||
|
||
async def get_pod_logs(
|
||
self,
|
||
pod_name: str,
|
||
namespace: str = "awoooi-prod",
|
||
tail_lines: int = 100,
|
||
previous: bool = False,
|
||
) -> str:
|
||
"""取得 Pod 日誌"""
|
||
...
|
||
|
||
async def get_resource_usage(
|
||
self,
|
||
pod_name: str,
|
||
namespace: str = "awoooi-prod",
|
||
) -> ResourceUsage:
|
||
"""取得 CPU/Memory 使用量"""
|
||
...
|
||
```
|
||
|
||
#### 2.2 SignOz 深度整合
|
||
|
||
```python
|
||
# 擴展 apps/api/src/services/signoz_client.py
|
||
|
||
class SignOzClient:
|
||
async def get_error_traces(
|
||
self,
|
||
service_name: str,
|
||
time_range_minutes: int = 10,
|
||
limit: int = 5,
|
||
) -> list[TraceInfo]:
|
||
"""取得錯誤 Trace 詳情"""
|
||
...
|
||
|
||
async def get_anomaly_detection(
|
||
self,
|
||
service_name: str,
|
||
metric: str, # rps, error_rate, p99_latency
|
||
) -> AnomalyResult:
|
||
"""異常偵測 (基於歷史基線)"""
|
||
...
|
||
```
|
||
|
||
#### 2.3 診斷資料聚合
|
||
|
||
```python
|
||
# 新增 apps/api/src/services/diagnosis_aggregator.py
|
||
|
||
@dataclass
|
||
class DiagnosisContext:
|
||
"""完整診斷上下文"""
|
||
incident: Incident
|
||
|
||
# K8s 資料
|
||
k8s_events: list[K8sEvent]
|
||
pod_logs: str
|
||
resource_usage: ResourceUsage
|
||
|
||
# SignOz 資料
|
||
gold_metrics: GoldMetrics
|
||
error_traces: list[TraceInfo]
|
||
anomaly_result: AnomalyResult | None
|
||
|
||
# Expert 初步診斷
|
||
expert_classification: str
|
||
expert_diagnosis: str
|
||
suggested_commands: list[str]
|
||
|
||
# Playbook 匹配
|
||
matched_playbooks: list[PlaybookMatch]
|
||
|
||
class DiagnosisAggregator:
|
||
"""診斷資料聚合器"""
|
||
|
||
async def collect_diagnosis_context(
|
||
self,
|
||
incident: Incident,
|
||
) -> DiagnosisContext:
|
||
"""並行收集所有診斷資料"""
|
||
async with asyncio.TaskGroup() as tg:
|
||
k8s_task = tg.create_task(self._collect_k8s_data(incident))
|
||
signoz_task = tg.create_task(self._collect_signoz_data(incident))
|
||
expert_task = tg.create_task(self._run_expert_diagnosis(incident))
|
||
playbook_task = tg.create_task(self._match_playbooks(incident))
|
||
|
||
return DiagnosisContext(
|
||
incident=incident,
|
||
k8s_events=k8s_task.result().events,
|
||
pod_logs=k8s_task.result().logs,
|
||
resource_usage=k8s_task.result().usage,
|
||
gold_metrics=signoz_task.result().metrics,
|
||
error_traces=signoz_task.result().traces,
|
||
anomaly_result=signoz_task.result().anomaly,
|
||
expert_classification=expert_task.result().classification,
|
||
expert_diagnosis=expert_task.result().diagnosis,
|
||
suggested_commands=expert_task.result().commands,
|
||
matched_playbooks=playbook_task.result(),
|
||
)
|
||
```
|
||
|
||
**預計修改檔案**:
|
||
- 新增 `apps/api/src/services/k8s_diagnostics.py`
|
||
- 新增 `apps/api/src/services/diagnosis_aggregator.py`
|
||
- 修改 `apps/api/src/services/signoz_client.py`
|
||
- 修改 `apps/api/src/services/decision_manager.py`
|
||
|
||
---
|
||
|
||
### Phase 3: Playbook 向量化 + RAG (3-4 週)
|
||
|
||
**目標**: 使用向量資料庫儲存 Playbook，實現語意搜尋
|
||
|
||
#### 3.1 向量化架構
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Playbook RAG System │
|
||
│ │
|
||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||
│ │ Embedding │ │ Vector Store │ │ Retriever │ │
|
||
│ │ (Ollama) │───▶│ (Redis) │◀───│ │ │
|
||
│ │ │ │ │ │ │ │
|
||
│ │ - Playbook │ │ - Index │ │ - Top-K │ │
|
||
│ │ - Incident │ │ - Search │ │ - Similarity │ │
|
||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
#### 3.2 實作細節
|
||
|
||
```python
|
||
# 新增 apps/api/src/services/playbook_rag.py
|
||
|
||
class PlaybookRAGService:
|
||
"""Playbook RAG 服務"""
|
||
|
||
def __init__(self):
|
||
# Redis Vector Search (已有 192.168.0.188:6380)
|
||
self.redis = Redis(host="192.168.0.188", port=6380)
|
||
self.embedding_model = "nomic-embed-text" # Ollama 本地
|
||
|
||
async def embed_playbook(
|
||
self,
|
||
playbook: Playbook,
|
||
) -> list[float]:
|
||
"""將 Playbook 向量化"""
|
||
text = f"""
|
||
症狀: {playbook.symptom_pattern.to_text()}
|
||
修復步驟: {playbook.repair_steps_text}
|
||
"""
|
||
return await self._get_embedding(text)
|
||
|
||
async def search_similar(
|
||
self,
|
||
incident: Incident,
|
||
top_k: int = 5,
|
||
min_similarity: float = 0.75,
|
||
) -> list[PlaybookMatch]:
|
||
"""語意搜尋相似 Playbook"""
|
||
query_text = f"""
|
||
告警: {incident.signals_summary}
|
||
影響服務: {incident.affected_services}
|
||
"""
|
||
query_vec = await self._get_embedding(query_text)
|
||
|
||
# Redis Vector Search
|
||
results = await self.redis.ft("playbook_idx").search(
|
||
Query(f"*=>[KNN {top_k} @embedding $vec AS score]")
|
||
.return_fields("playbook_id", "score")
|
||
.dialect(2),
|
||
query_params={"vec": query_vec},
|
||
)
|
||
|
||
return [
|
||
PlaybookMatch(
|
||
playbook_id=r.playbook_id,
|
||
similarity=1 - float(r.score), # 距離轉相似度
|
||
)
|
||
for r in results.docs
|
||
if 1 - float(r.score) >= min_similarity
|
||
]
|
||
```
|
||
|
||
**預計修改檔案**:
|
||
- 新增 `apps/api/src/services/playbook_rag.py`
|
||
- 修改 `apps/api/src/services/playbook_service.py`
|
||
- 修改 `k8s/awoooi-prod/04-configmap.yaml` (新增向量索引配置)
|
||
|
||
---
|
||
|
||
### Phase 4: 自動執行機制 (2-3 週)
|
||
|
||
**目標**: 低風險操作自動執行，無需人工審核
|
||
|
||
#### 4.1 自動執行條件
|
||
|
||
```python
|
||
# 擴展 apps/api/src/services/trust_engine.py
|
||
|
||
class AutoApprovePolicy:
|
||
"""自動執行策略"""
|
||
|
||
# 自動執行條件 (全部滿足才放行)
|
||
MIN_TRUST_SCORE = 0.90 # 信任度 >= 90%
|
||
MIN_PLAYBOOK_SUCCESS = 0.95 # Playbook 成功率 >= 95%
|
||
MIN_PLAYBOOK_COUNT = 3 # Playbook 成功次數 >= 3
|
||
ALLOWED_RISK_LEVELS = ["low"] # 只有 LOW 風險可自動
|
||
|
||
@classmethod
|
||
def should_auto_approve(
|
||
cls,
|
||
proposal: ProposalData,
|
||
playbook_match: PlaybookMatch | None,
|
||
trust_score: float,
|
||
) -> tuple[bool, str]:
|
||
"""判斷是否自動執行"""
|
||
|
||
# 條件 1: 風險等級
|
||
if proposal.risk_level not in cls.ALLOWED_RISK_LEVELS:
|
||
return False, f"風險等級 {proposal.risk_level} 不允許自動執行"
|
||
|
||
# 條件 2: 信任度
|
||
if trust_score < cls.MIN_TRUST_SCORE:
|
||
return False, f"信任度 {trust_score:.0%} 低於閾值"
|
||
|
||
# 條件 3: Playbook 匹配
|
||
if playbook_match:
|
||
if playbook_match.success_rate < cls.MIN_PLAYBOOK_SUCCESS:
|
||
return False, f"Playbook 成功率 {playbook_match.success_rate:.0%} 低於閾值"
|
||
if playbook_match.success_count < cls.MIN_PLAYBOOK_COUNT:
|
||
return False, f"Playbook 成功次數 {playbook_match.success_count} 低於閾值"
|
||
return True, "Playbook 匹配，符合自動執行條件"
|
||
|
||
return False, "無匹配 Playbook，需人工審核"
|
||
```
|
||
|
||
#### 4.2 自動執行流程
|
||
|
||
```python
|
||
# 擴展 apps/api/src/services/approval_service.py
|
||
|
||
class ApprovalService:
|
||
async def process_proposal(
|
||
self,
|
||
incident: Incident,
|
||
proposal: ProposalData,
|
||
) -> ApprovalRequest:
|
||
"""處理提案，決定是否自動執行"""
|
||
|
||
# 取得 Playbook 匹配和信任度
|
||
playbook_match = await self._playbook_service.get_best_match(incident)
|
||
trust_score = await self._trust_engine.calculate_trust(
|
||
incident=incident,
|
||
proposal=proposal,
|
||
playbook_match=playbook_match,
|
||
)
|
||
|
||
# 判斷是否自動執行
|
||
should_auto, reason = AutoApprovePolicy.should_auto_approve(
|
||
proposal=proposal,
|
||
playbook_match=playbook_match,
|
||
trust_score=trust_score,
|
||
)
|
||
|
||
if should_auto:
|
||
# 自動建立已簽核的 Approval
|
||
approval = await self._create_approval(
|
||
incident=incident,
|
||
proposal=proposal,
|
||
status=ApprovalStatus.APPROVED,
|
||
signature_source=SignatureSource.SYSTEM,
|
||
auto_approved_reason=reason,
|
||
)
|
||
|
||
# 自動執行
|
||
await self._execution_service.execute_approved_action(approval)
|
||
|
||
# 記錄自動執行
|
||
logger.info(
|
||
"auto_approved_execution",
|
||
incident_id=incident.incident_id,
|
||
reason=reason,
|
||
trust_score=trust_score,
|
||
)
|
||
|
||
return approval
|
||
|
||
# 需人工審核
|
||
return await self._create_approval(
|
||
incident=incident,
|
||
proposal=proposal,
|
||
status=ApprovalStatus.PENDING,
|
||
needs_human_review=True,
|
||
auto_reject_reason=reason,
|
||
)
|
||
```
|
||
|
||
**預計修改檔案**:
|
||
- 修改 `apps/api/src/services/trust_engine.py`
|
||
- 修改 `apps/api/src/services/approval_service.py`
|
||
- 修改 `apps/api/src/services/proposal_service.py`
|
||
|
||
---
|
||
|
||
### Phase 5: 持續學習迴圈 (2-3 週)
|
||
|
||
**目標**: 從執行結果中學習，持續優化決策
|
||
|
||
#### 5.1 反饋處理流程
|
||
|
||
```python
|
||
# 新增 apps/api/src/services/learning_service.py
|
||
|
||
class LearningService:
|
||
"""持續學習服務"""
|
||
|
||
async def process_execution_result(
|
||
self,
|
||
approval: ApprovalRecord,
|
||
result: ExecutionResult,
|
||
):
|
||
"""處理執行結果，觸發學習"""
|
||
|
||
# 1. 更新 Playbook 統計
|
||
if approval.matched_playbook_id:
|
||
await self._update_playbook_stats(
|
||
playbook_id=approval.matched_playbook_id,
|
||
success=result.success,
|
||
)
|
||
|
||
# 2. 調整信任度
|
||
await self._adjust_trust_score(
|
||
incident_type=approval.incident_type,
|
||
action_type=approval.action_type,
|
||
success=result.success,
|
||
)
|
||
|
||
# 3. 萃取新 Playbook (成功案例)
|
||
if result.success and not approval.matched_playbook_id:
|
||
await self._try_extract_playbook(approval)
|
||
|
||
async def process_human_feedback(
|
||
self,
|
||
incident_id: str,
|
||
feedback: FeedbackRequest,
|
||
):
|
||
"""處理人工反饋"""
|
||
|
||
# 1. 更新 Playbook 有效性
|
||
if feedback.effectiveness_score >= 4:
|
||
await self._promote_playbook_confidence(incident_id)
|
||
elif feedback.effectiveness_score <= 2:
|
||
await self._demote_playbook_confidence(incident_id)
|
||
|
||
# 2. 記錄學習筆記 (未來用於 LLM fine-tuning)
|
||
await self._store_learning_note(
|
||
incident_id=incident_id,
|
||
note=feedback.learning_notes,
|
||
)
|
||
```
|
||
|
||
#### 5.2 信任度動態調整
|
||
|
||
```python
|
||
# 擴展 apps/api/src/services/trust_engine.py
|
||
|
||
class TrustEngine:
|
||
# 信任度調整參數
|
||
SUCCESS_BOOST = 0.02 # 成功 +2%
|
||
FAILURE_PENALTY = 0.10 # 失敗 -10%
|
||
HUMAN_OVERRIDE_PENALTY = 0.05 # 人工覆蓋 -5%
|
||
|
||
async def adjust_trust(
|
||
self,
|
||
incident_type: str,
|
||
action_type: str,
|
||
success: bool,
|
||
human_override: bool = False,
|
||
):
|
||
"""動態調整信任度"""
|
||
key = f"trust:{incident_type}:{action_type}"
|
||
current = await self._get_trust(key)
|
||
|
||
if human_override:
|
||
# 人工覆蓋 AI 決策，降低信任度
|
||
new_trust = max(0.0, current - self.HUMAN_OVERRIDE_PENALTY)
|
||
elif success:
|
||
# 執行成功，提高信任度
|
||
new_trust = min(1.0, current + self.SUCCESS_BOOST)
|
||
else:
|
||
# 執行失敗，降低信任度
|
||
new_trust = max(0.0, current - self.FAILURE_PENALTY)
|
||
|
||
await self._set_trust(key, new_trust)
|
||
|
||
logger.info(
|
||
"trust_adjusted",
|
||
incident_type=incident_type,
|
||
action_type=action_type,
|
||
old_trust=current,
|
||
new_trust=new_trust,
|
||
reason="success" if success else "failure",
|
||
)
|
||
```
|
||
|
||
**預計修改檔案**:
|
||
- 新增 `apps/api/src/services/learning_service.py`
|
||
- 修改 `apps/api/src/services/trust_engine.py`
|
||
- 修改 `apps/api/src/services/approval_execution.py`
|
||
|
||
#### 5.3 Playbook 自動狀態轉換 (2026-03-30 補充)
|
||
|
||
> **實作位置**: `apps/api/src/repositories/playbook_repository.py:adjust_confidence()`
|
||
|
||
| 狀態轉換 | 觸發條件 | 說明 |
|
||
|---------|---------|------|
|
||
| **DRAFT → APPROVED** | `confidence >= 0.9` | 高信心度自動升級 |
|
||
| **任意 → DEPRECATED** | `confidence < 0.3` + `failure_rate > 50%` + `executions >= 5` | 低效 Playbook 自動棄用 |
|
||
|
||
```python
|
||
# Playbook 信心度調整常數
|
||
CONFIDENCE_PROMOTE_THRESHOLD = 0.9 # 自動升級閾值
|
||
CONFIDENCE_DEPRECATE_THRESHOLD = 0.3 # 自動棄用閾值
|
||
FAILURE_RATE_THRESHOLD = 0.5 # 失敗率閾值
|
||
MIN_EXECUTIONS_FOR_DEPRECATE = 5 # 最小執行次數
|
||
|
||
# Learning Service 信心度調整
|
||
PROMOTE_DELTA = +0.1 # 高評分 (>=4) +10%
|
||
DEMOTE_DELTA = -0.15 # 低評分 (<=2) -15%
|
||
```
|
||
|
||
---
|
||
|
||
## 四、架構相容性分析
|
||
|
||
### 4.1 與現有架構的整合點
|
||
|
||
| 現有元件 | 整合方式 | 衝突風險 |
|
||
|---------|----------|----------|
|
||
| `decision_manager.py` | 擴展 `_dual_engine_analyze()` | 低 - 加法修改 |
|
||
| `openclaw.py` | 新增 `expert_context` 參數 | 低 - 向下相容 |
|
||
| `signoz_client.py` | 擴展新方法 | 低 - 加法修改 |
|
||
| `playbook_service.py` | 加入 RAG 整合 | 中 - 需重構 |
|
||
| `trust_engine.py` | 加入動態調整 | 中 - 需重構 |
|
||
| `approval_service.py` | 加入自動執行 | 高 - 核心流程變更 |
|
||
| `executor.py` | 無需修改 | 無 |
|
||
|
||
### 4.2 需要調整的介面
|
||
|
||
```python
|
||
# 1. IDecisionManager 需擴展
|
||
class IDecisionManager(Protocol):
|
||
async def get_or_create_decision(
|
||
self,
|
||
incident: Incident,
|
||
timeout_sec: float = 30.0,
|
||
diagnosis_context: DiagnosisContext | None = None, # 新增
|
||
) -> DecisionToken:
|
||
...
|
||
|
||
# 2. ProposalService 需擴展
|
||
class IProposalService(Protocol):
|
||
async def generate_proposal(
|
||
self,
|
||
incident: Incident,
|
||
diagnosis_context: DiagnosisContext | None = None, # 新增
|
||
playbook_match: PlaybookMatch | None = None, # 新增
|
||
) -> ProposalData:
|
||
...
|
||
|
||
# 3. ApprovalService 需擴展
|
||
class IApprovalService(Protocol):
|
||
async def create_approval(
|
||
self,
|
||
incident: Incident,
|
||
proposal: ProposalData,
|
||
auto_approve: bool = False, # 新增
|
||
) -> ApprovalRequest:
|
||
...
|
||
```
|
||
|
||
### 4.3 資料庫 Schema 變更
|
||
|
||
```sql
|
||
-- 新增欄位到 approval_records
|
||
ALTER TABLE approval_records ADD COLUMN IF NOT EXISTS
|
||
auto_approved BOOLEAN DEFAULT FALSE;
|
||
ALTER TABLE approval_records ADD COLUMN IF NOT EXISTS
|
||
auto_approved_reason TEXT;
|
||
ALTER TABLE approval_records ADD COLUMN IF NOT EXISTS
|
||
matched_playbook_id UUID REFERENCES playbooks(id);
|
||
ALTER TABLE approval_records ADD COLUMN IF NOT EXISTS
|
||
trust_score FLOAT;
|
||
|
||
-- 新增表: trust_scores (信任度追蹤)
|
||
CREATE TABLE IF NOT EXISTS trust_scores (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
incident_type VARCHAR(100) NOT NULL,
|
||
action_type VARCHAR(100) NOT NULL,
|
||
trust_score FLOAT NOT NULL DEFAULT 0.5,
|
||
success_count INT DEFAULT 0,
|
||
failure_count INT DEFAULT 0,
|
||
last_updated_at TIMESTAMPTZ DEFAULT NOW(),
|
||
UNIQUE(incident_type, action_type)
|
||
);
|
||
|
||
-- 新增表: learning_notes (學習筆記)
|
||
CREATE TABLE IF NOT EXISTS learning_notes (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
incident_id VARCHAR(100) NOT NULL,
|
||
note TEXT NOT NULL,
|
||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||
);
|
||
```
|
||
|
||
---
|
||
|
||
## 五、風險評估與緩解
|
||
|
||
### 5.1 技術風險
|
||
|
||
| 風險 | 影響 | 緩解措施 |
|
||
|------|------|----------|
|
||
| 自動執行出錯 | 服務中斷 | Shadow Mode 先行、Dry-Run 驗證 |
|
||
| 信任度計算錯誤 | 該自動的沒自動 | 保守閾值 (90%)、人工複查 |
|
||
| Playbook 匹配錯誤 | 執行錯誤修復 | 最低相似度閾值 (85%) |
|
||
| 學習迴圈偏差 | 錯誤累積 | 定期人工審查、信任度上限 |
|
||
|
||
### 5.2 回滾計畫
|
||
|
||
```yaml
|
||
# 每個 Phase 都可以獨立回滾
|
||
|
||
Phase 1 回滾:
|
||
- git revert 相關 commit
|
||
- 恢復舊版 EXPERT_RULES
|
||
|
||
Phase 2 回滾:
|
||
- 停用 K8sDiagnosticsService
|
||
- decision_manager 使用舊流程
|
||
|
||
Phase 3 回滾:
|
||
- 停用 PlaybookRAGService
|
||
- playbook_service 使用傳統匹配
|
||
|
||
Phase 4 回滾:
|
||
- AUTO_APPROVE_ENABLED=false
|
||
- 所有請求走人工審核
|
||
|
||
Phase 5 回滾:
|
||
- 停用 LearningService
|
||
- 信任度凍結當前值
|
||
```
|
||
|
||
---
|
||
|
||
## 六、驗收標準
|
||
|
||
### Phase 1 (已完成)
|
||
- [x] Expert System 不再盲目重啟
|
||
- [x] 測試資源自動識別
|
||
- [x] 診斷指令包含在回應中
|
||
- [x] LLM 收到 Expert 上下文
|
||
|
||
### Phase 2
|
||
- [ ] K8s Events 整合到診斷流程
|
||
- [ ] Pod 日誌自動擷取
|
||
- [ ] SignOz 異常偵測功能
|
||
|
||
### Phase 3
|
||
- [ ] Playbook 向量化完成
|
||
- [ ] RAG 搜尋正確率 > 85%
|
||
- [ ] 查詢延遲 < 100ms
|
||
|
||
### Phase 4
|
||
- [ ] 自動執行功能上線
|
||
- [ ] 低風險操作自動通過率 > 30%
|
||
- [ ] 零誤執行 (連續 7 天)
|
||
|
||
### Phase 5
|
||
- [ ] 信任度動態調整正常
|
||
- [ ] Playbook 成功率持續提升
|
||
- [ ] 學習迴圈運作正常
|
||
|
||
---
|
||
|
||
## 七、時程總覽
|
||
|
||
```
|
||
2026-03-27 ──────────────────────────────────────────────▶ 2026-05-15
|
||
|
||
Phase 1: 智能診斷基礎 [██████████] 100% ✅ 已完成
|
||
├─ Expert System 重構
|
||
└─ LLM 上下文整合
|
||
|
||
Phase 2: 資料收集強化 [░░░░░░░░░░] 0% (2-3 週)
|
||
├─ K8s Events 整合
|
||
├─ SignOz 深度整合
|
||
└─ 診斷資料聚合
|
||
|
||
Phase 3: Playbook RAG [░░░░░░░░░░] 0% (3-4 週)
|
||
├─ 向量化架構
|
||
├─ Redis Vector Search
|
||
└─ 語意搜尋
|
||
|
||
Phase 4: 自動執行機制 [░░░░░░░░░░] 0% (2-3 週)
|
||
├─ 自動執行策略
|
||
├─ 信任度評估
|
||
└─ 安全防護
|
||
|
||
Phase 5: 持續學習 [░░░░░░░░░░] 0% (2-3 週)
|
||
├─ 反饋處理
|
||
├─ 信任度調整
|
||
└─ Playbook 萃取
|
||
```
|
||
|
||
**總預估時間**: 10-15 週 (含測試驗證)
|
||
|
||
---
|
||
|
||
## 7.1 Phase 6: 非同步分析優化 (2026-03-27 新增)
|
||
|
||
> **觸發**: LLM 分析 (llama3.2:3b CPU) 需 2-3 分鐘，導致告警回應延遲
|
||
|
||
### 問題分析
|
||
|
||
| 現狀 | 影響 | 根因 |
|
||
|------|------|------|
|
||
| LLM 同步等待 2-3 分鐘 | 告警回應延遲 | Ollama 純 CPU 模式 |
|
||
| 超時後 Fallback Expert System | 用戶只看到 75% 信心度 | 超時設定不合理 |
|
||
| 等待期間 API 阻塞 | 批量告警處理緩慢 | 非非同步設計 |
|
||
|
||
### 解決方案
|
||
|
||
```
|
||
優化前:
|
||
告警 → 等 LLM (2-3分鐘) → 發 Telegram
|
||
↑_____________↑
|
||
阻塞等待
|
||
|
||
優化後:
|
||
告警 → Expert System 立即發 (75%) → Telegram
|
||
↓
|
||
背景 LLM 分析 (2-3分鐘)
|
||
↓
|
||
edit_message 更新 Telegram (90%+)
|
||
```
|
||
|
||
### 實作細節
|
||
|
||
```python
|
||
# decision_manager.py 修改
|
||
|
||
async def get_or_create_decision(
|
||
incident: Incident,
|
||
timeout_sec: float = 180.0, # 已修改
|
||
) -> DecisionToken:
|
||
# Phase 6 優化: 非同步雙軌
|
||
|
||
# Step 1: Expert System 立即返回
|
||
expert_result = await self._expert_analyze(incident)
|
||
await _push_decision_to_telegram(incident, expert_result)
|
||
|
||
# Step 2: 背景啟動 LLM 分析
|
||
asyncio.create_task(
|
||
self._background_llm_analyze(incident, token)
|
||
)
|
||
|
||
return token
|
||
|
||
async def _background_llm_analyze(
|
||
self,
|
||
incident: Incident,
|
||
token: DecisionToken,
|
||
) -> None:
|
||
"""背景 LLM 分析，完成後更新 Telegram"""
|
||
try:
|
||
llm_result = await self._llm_analyze(incident)
|
||
|
||
# 更新 Token
|
||
token.proposal_data = llm_result
|
||
await self._save_token(token)
|
||
|
||
# 更新 Telegram 訊息
|
||
await self._update_telegram_message(incident, llm_result)
|
||
except Exception as e:
|
||
logger.warning("background_llm_failed", error=str(e))
|
||
```
|
||
|
||
### 驗收標準
|
||
|
||
- [ ] Expert System 告警回應 < 5 秒
|
||
- [ ] LLM 結果 2-3 分鐘後更新 Telegram
|
||
- [ ] 更新使用 `edit_message` 而非新發訊息
|
||
- [ ] 錯誤處理：LLM 失敗不影響已發送的 Expert 結果
|
||
|
||
### 依賴
|
||
|
||
- 需修改 `decision_manager.py` (Tier 3 紅區)
|
||
- 需擴展 `telegram_gateway.py` 支援 `edit_message`
|
||
- 需首席架構師簽核
|
||
|
||
### 狀態
|
||
|
||
| 項目 | 狀態 |
|
||
|------|------|
|
||
| 設計文件 | ✅ 完成 (本章節) |
|
||
| 首席架構師審查 | 🔴 待審 |
|
||
| 實作 | 🔴 待開始 |
|
||
|
||
---
|
||
|
||
## 7.5 Phase D-G P0 修正: Learning Repository Layer (2026-03-29)
|
||
|
||
### 背景
|
||
|
||
首席架構師審查發現原設計違反 leWOOOgo 積木化原則:
|
||
- Service 直接依賴 Redis Client
|
||
- 未遵循 Repository Pattern
|
||
|
||
### 修正內容
|
||
|
||
#### 1. 新增 ILearningRepository Interface
|
||
|
||
```python
|
||
# src/repositories/interfaces.py
|
||
@runtime_checkable
|
||
class ILearningRepository(Protocol):
|
||
async def record_repair(...) -> bool
|
||
async def get_repair_stats(...) -> dict
|
||
async def get_all_repair_stats(...) -> dict[str, dict]
|
||
async def get_repair_history(...) -> list[dict]
|
||
async def get_learning_summary(...) -> dict
|
||
```
|
||
|
||
#### 2. 新增 LearningRepository 實作
|
||
|
||
```python
|
||
# src/repositories/learning_repository.py
|
||
class LearningRepository:
|
||
"""Redis 持久化層 - 學習數據存取"""
|
||
|
||
# Redis Key 結構:
|
||
# - learning:repair:{anomaly_key}:{action} -> List[JSON]
|
||
# - learning:stats:{anomaly_key}:{action} -> Hash
|
||
```
|
||
|
||
#### 3. 擴展 LearningService
|
||
|
||
```python
|
||
# src/services/learning_service.py
|
||
class LearningService:
|
||
def __init__(self, repository: ILearningRepository | None = None):
|
||
self._repository = repository or get_learning_repository()
|
||
|
||
# 新增方法
|
||
async def record_repair_result(...) # 記錄修復結果
|
||
async def get_recommended_fix(...) # 修復推薦
|
||
async def get_learning_summary(...) # 學習摘要
|
||
```
|
||
|
||
#### 4. 新增 Learning API
|
||
|
||
```
|
||
GET /api/v1/learning/summary/{anomaly_key}
|
||
GET /api/v1/learning/recommendation/{anomaly_key}
|
||
```
|
||
|
||
### 架構圖
|
||
|
||
```
|
||
┌───────────────────────────────────────────────────────────┐
|
||
│ API Layer (Router) │
|
||
│ src/api/v1/learning.py │
|
||
│ - 只做 HTTP 轉發，不含業務邏輯 │
|
||
└─────────────────────────┬─────────────────────────────────┘
|
||
│
|
||
┌─────────────────────────▼─────────────────────────────────┐
|
||
│ Service Layer │
|
||
│ src/services/learning_service.py │
|
||
│ - 業務邏輯編排 │
|
||
│ - 透過 Interface 依賴 Repository │
|
||
└─────────────────────────┬─────────────────────────────────┘
|
||
│ ILearningRepository
|
||
┌─────────────────────────▼─────────────────────────────────┐
|
||
│ Repository Layer │
|
||
│ src/repositories/learning_repository.py │
|
||
│ - Redis 資料存取 │
|
||
│ - 90 天 TTL 持久化 │
|
||
└───────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### 符合原則
|
||
|
||
| 原則 | 狀態 |
|
||
|------|------|
|
||
| Service 不直接存取 Redis | ✅ 透過 Repository |
|
||
| Interface 先行 | ✅ ILearningRepository Protocol |
|
||
| 依賴注入 | ✅ 可注入測試 Repository |
|
||
| Router 薄層 | ✅ 只做 HTTP 轉發 |
|
||
|
||
---
|
||
|
||
## 八、結論
|
||
|
||
本方案提供了一個**完整的智能自動修復系統**，從「盲目重啟」進化到「根因診斷 + 智能決策 + 持續學習」。
|
||
|
||
**核心理念**:
|
||
1. **診斷優先**: 先了解問題，再決定行動
|
||
2. **漸進信任**: 從人工審核逐步過渡到自動執行
|
||
3. **持續學習**: 從每次執行結果中學習改進
|
||
4. **安全防護**: 多層防護確保不會誤操作
|
||
|
||
**與現有架構的相容性**:
|
||
- Phase 1-2: 純加法修改，風險極低
|
||
- Phase 3: 需要重構 Playbook 服務，風險中等
|
||
- Phase 4-5: 核心流程變更，需要嚴格測試
|
||
|
||
**建議**: 按 Phase 順序實施，每個 Phase 完成後驗證穩定性再進入下一階段。
|
||
---
|
||
|
||
|
||
## 九、2026-04-04 實作完成 (台北時間)
|
||
|
||
> **更新者**: Claude Code
|
||
> **Commit**: d0f0970, 72d7536, df3ef90
|
||
|
||
### 9.1 根本原因診斷 (19筆 PENDING 清查)
|
||
|
||
| 根因 | 說明 | 修復 |
|
||
|------|------|------|
|
||
| playbooks 表從未建立 | Playbook 庫空白，RAG 永遠無匹配 | phase7_playbooks_table.sql |
|
||
| Ollama embedding is_closed=True | 滾動重啟後 http_client 失效，vector_count:0 | _get_http_client() 偵測重建 |
|
||
| 9筆殭屍 PENDING (3/26) | mock_fallback CRITICAL 測試記錄 | 直接清除 |
|
||
|
||
### 9.2 完成項目
|
||
|
||
| 檔案 | 修改 | 說明 |
|
||
|------|------|------|
|
||
| `playbook_rag.py` | 新增 `_get_http_client()` | 偵測 is_closed 自動重建 HTTP client |
|
||
| `playbook_rag.py` | `embed_playbook()` Bug 修復 | `s.sequence`→`s.step_number`, `s.description`→`s.command` |
|
||
| `telegram_gateway.py` | 新增 `ai_model` 欄位 | format/format_with_nemotron 顯示底層模型 |
|
||
| `openclaw.py` | proposal_dict 加 `"model"` | 傳入底層模型名稱 |
|
||
| `decision_manager.py` | 讀取 `ai_model` 傳入 approval card | 決策路徑完整 |
|
||
| `migrations/phase7_playbooks_table.sql` | 新增 | playbooks 表，PRIMARY KEY + 5 GIN 索引 |
|
||
| `playbook_service.py` | `_write_to_km()` fire-and-forget | extract_from_incident() 後自動 KM 沉澱 |
|
||
| `playbook_service.py` | `_get_rag_service()` 改走工廠 | 每次重建，避免快取 is_closed client |
|
||
| `approval_execution.py` | `_write_execution_result_to_km()` | 移出 try/except，保證執行記錄寫入 KM |
|
||
| `approval_execution.py` | 冷啟動修復 | 執行成功自動設定 execution_success=True, effectiveness_score=4, status=RESOLVED |
|
||
| `approval_execution.py` | skip 路徑 debug→info | 可觀測性提升 |
|
||
| Ollama | 刪除 smollm2:135m | 0.3GB CPU 機器不需要的模型 |
|
||
|
||
### 9.3 首席架構師 Review 結果
|
||
|
||
| 評分 | 修復項目 | 說明 |
|
||
|------|----------|------|
|
||
| 初版 21/25 | — | 72d7536 |
|
||
| **Critical #1** | KM write task 移出 try/except | 保證無論 learning 成敗都寫 KM |
|
||
| **Important #1** | PlaybookService 快取繞過工廠 | 每次走工廠避免 is_closed |
|
||
| **Important #2** | Migration 缺 PRIMARY KEY | prod DB 已 ALTER TABLE 補齊 |
|
||
| **Important #3** | embed_playbook() s.sequence AttributeError | 修正欄位名稱靜默失敗 bug |
|
||
| 修復後 ~25/25 | **PASSED** | df3ef90 |
|
||
|
||
### 9.4 完整閉環
|
||
|
||
```
|
||
告警 → AI分析(OpenClaw/Nemotron) → 查Playbook(RAG語意搜尋)
|
||
→ 有匹配且LOW風險 → 自動執行
|
||
→ MEDIUM/CRITICAL → Telegram等人工批准
|
||
→ 執行成功
|
||
→ _trigger_playbook_extraction():
|
||
自動設定 execution_success=True, effectiveness_score=4, status=RESOLVED
|
||
→ Playbook萃取 → 存Redis → KM沉澱(_write_to_km)
|
||
→ incident_service.resolve() → KM萃取(knowledge_extractor)
|
||
→ _trigger_learning() 後 → KM執行記錄(_write_execution_result_to_km)
|
||
→ 下次相似告警 → Playbook命中(RAG) → 加速決策
|
||
```
|
||
|
||
**狀態**: ✅ 閉環已通 (2026-04-04 台北時間)
|