- P0: DIAGNOSE Privacy-First Routing(local chain 隔離 + REJECT 保護) - P1: Knowledge Auto-Harvesting(Anti-Pattern 閉環 + Runbook 生成) - P2: Config Drift Detection(GitOps 守門員 + Nemotron 意圖分析) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
43 KiB
P1:Knowledge Auto-Harvesting 實作計畫
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: 修復成功後 Nemotron 自動生成完整 9 段 Runbook(DRAFT 待審核),修復失敗自動生成 ANTI_PATTERN(直接發布);執行前加入 Anti-Pattern 閉環攔截,阻止在同一個坑重複摔倒。
Architecture: 三個整合點:(1) SymptomPattern.compute_hash() — O(1) 確定性比對;(2) AutoRepairService.evaluate_auto_repair() — 執行前查 KB anti-pattern 閘門;(3) AutoRepairService.execute_auto_repair() — 執行後背景異步呼叫 NemotronRunbookGenerator。Runbook 生成完全異步,不阻塞主流程。
Tech Stack: Python 3.11, asyncio, structlog, Pydantic v2, existing NemotronProvider / KnowledgeService / TelegramGateway, pytest-asyncio
File Map
| 動作 | 檔案 | 變更內容 |
|---|---|---|
| 修改 | apps/api/src/models/knowledge.py |
新增 ANTI_PATTERN / AUTO_RUNBOOK EntryType;KnowledgeEntry 新增 symptoms_hash 欄位 |
| 修改 | apps/api/src/models/playbook.py |
SymptomPattern 新增 compute_hash() 方法 |
| 修改 | apps/api/src/services/knowledge_service.py |
新增 check_anti_pattern(symptoms_hash, days) 方法 |
| 新增 | apps/api/src/services/runbook_generator.py |
NemotronRunbookGenerator — 呼叫 NemotronProvider 生成 Runbook |
| 修改 | apps/api/src/services/auto_repair_service.py |
evaluate_auto_repair() 加 anti_pattern gate;execute_auto_repair() 結束後背景觸發生成 |
| 新增 | apps/api/migrations/phase8_knowledge_symptoms_hash.sql |
knowledge 表新增 symptoms_hash VARCHAR(16) + index |
| 新增 | apps/api/tests/test_p1_knowledge_auto_harvesting.py |
完整測試套件 |
Task 1:DB Migration — 新增 symptoms_hash 欄位
Files:
-
Create:
apps/api/migrations/phase8_knowledge_symptoms_hash.sql -
Step 1:建立 migration 檔案
-- phase8_knowledge_symptoms_hash.sql
-- Knowledge Auto-Harvesting: 新增 symptoms_hash 欄位
-- 建立時間: 2026-04-04 (台北時區)
-- 建立者: Claude Code (P1 Knowledge Auto-Harvesting)
-- 新增 symptoms_hash 欄位(可為 NULL,既有資料不受影響)
ALTER TABLE knowledge_entries
ADD COLUMN IF NOT EXISTS symptoms_hash VARCHAR(16) NULL;
-- 建立 index 供 check_anti_pattern() O(1) 查詢使用
CREATE INDEX IF NOT EXISTS idx_knowledge_entries_symptoms_hash
ON knowledge_entries (symptoms_hash)
WHERE symptoms_hash IS NOT NULL;
-- 確認
SELECT column_name, data_type, is_nullable
FROM information_schema.columns
WHERE table_name = 'knowledge_entries'
AND column_name = 'symptoms_hash';
- Step 2:確認 migration 語法
cat apps/api/migrations/phase8_knowledge_symptoms_hash.sql
- Step 3:Commit(migration 先 commit,後續程式碼與之對應)
git add apps/api/migrations/phase8_knowledge_symptoms_hash.sql
git commit -m "feat(migration): knowledge_entries 新增 symptoms_hash 欄位 + index (P1)"
Task 2:Model 更新 — EntryType + KnowledgeEntry + SymptomPattern
Files:
-
Modify:
apps/api/src/models/knowledge.py -
Modify:
apps/api/src/models/playbook.py -
Test:
apps/api/tests/test_p1_knowledge_auto_harvesting.py -
Step 1:寫失敗測試
建立 apps/api/tests/test_p1_knowledge_auto_harvesting.py:
"""
P1 Knowledge Auto-Harvesting Tests
=====================================
測試 SymptomPattern.compute_hash()、Anti-Pattern 閉環、Runbook 生成
建立時間: 2026-04-04 (台北時區)
建立者: Claude Code (P1 Knowledge Auto-Harvesting)
"""
import os
os.environ.setdefault("MOCK_MODE", "true")
import hashlib
import json
import pytest
class TestSymptomPatternHash:
"""SymptomPattern.compute_hash() — 確定性 hash"""
def test_same_symptoms_same_hash(self):
"""相同症狀永遠產生相同 hash"""
from src.models.playbook import SymptomPattern
sp1 = SymptomPattern(
alert_names=["HighCPU", "PodCrash"],
affected_services=["awoooi-api"],
label_patterns={"namespace": "awoooi-prod"},
)
sp2 = SymptomPattern(
alert_names=["PodCrash", "HighCPU"], # 順序不同
affected_services=["awoooi-api"],
label_patterns={"namespace": "awoooi-prod"},
)
assert sp1.compute_hash() == sp2.compute_hash()
def test_different_symptoms_different_hash(self):
"""不同症狀產生不同 hash"""
from src.models.playbook import SymptomPattern
sp1 = SymptomPattern(alert_names=["HighCPU"], affected_services=["api"])
sp2 = SymptomPattern(alert_names=["OOMKilled"], affected_services=["api"])
assert sp1.compute_hash() != sp2.compute_hash()
def test_hash_is_16_chars(self):
"""hash 長度固定為 16"""
from src.models.playbook import SymptomPattern
sp = SymptomPattern(alert_names=["Test"])
assert len(sp.compute_hash()) == 16
def test_empty_symptoms_has_valid_hash(self):
"""空症狀不 crash,產生合法 hash"""
from src.models.playbook import SymptomPattern
sp = SymptomPattern()
h = sp.compute_hash()
assert len(h) == 16
assert isinstance(h, str)
class TestKnowledgeEntryTypes:
"""EntryType 新增 ANTI_PATTERN / AUTO_RUNBOOK"""
def test_anti_pattern_entry_type_exists(self):
from src.models.knowledge import EntryType
assert EntryType.ANTI_PATTERN == "anti_pattern"
def test_auto_runbook_entry_type_exists(self):
from src.models.knowledge import EntryType
assert EntryType.AUTO_RUNBOOK == "auto_runbook"
def test_knowledge_entry_accepts_symptoms_hash(self):
"""KnowledgeEntry 接受 symptoms_hash 欄位"""
from src.models.knowledge import KnowledgeEntry, EntryType, EntrySource, EntryStatus
entry = KnowledgeEntry(
id="test-001",
title="測試 Runbook",
content="內容",
entry_type=EntryType.AUTO_RUNBOOK,
category="auto-generated",
source=EntrySource.AI,
status=EntryStatus.DRAFT,
symptoms_hash="abc123def456789a",
)
assert entry.symptoms_hash == "abc123def456789a"
def test_knowledge_entry_symptoms_hash_optional(self):
"""symptoms_hash 為 Optional,不傳入不 crash"""
from src.models.knowledge import KnowledgeEntry, EntryType, EntrySource
entry = KnowledgeEntry(
id="test-002",
title="手動 Runbook",
content="內容",
entry_type=EntryType.RUNBOOK,
category="manual",
source=EntrySource.HUMAN,
)
assert entry.symptoms_hash is None
- Step 2:執行確認失敗
cd apps/api && python -m pytest tests/test_p1_knowledge_auto_harvesting.py::TestSymptomPatternHash tests/test_p1_knowledge_auto_harvesting.py::TestKnowledgeEntryTypes -v
預期:FAIL(compute_hash 不存在,ANTI_PATTERN 不存在)
- Step 3:修改
models/knowledge.py— 新增 EntryType 和 symptoms_hash 欄位
找到 EntryType class(約 L30),在最後一個成員後新增:
ANTI_PATTERN = "anti_pattern" # 失敗案例,自動分類隔離,直接發布
AUTO_RUNBOOK = "auto_runbook" # Nemotron 自動生成,待人工審核
找到 KnowledgeEntry class(約 L80),在 related_playbook_id 欄位後新增:
symptoms_hash: str | None = Field(
default=None,
description="症狀特徵 hash,供 Anti-Pattern O(1) 精確比對(16 chars SHA256 前綴)",
)
- Step 4:修改
models/playbook.py— 新增compute_hash()方法
找到 SymptomPattern class(約 L66),在 model_config 上方新增:
def compute_hash(self) -> str:
"""
確定性 hash:alert_names + affected_services + label_patterns
目的:O(1) 精確比對,避免純語意搜尋的模糊性
順序無關(使用 sorted),跨平台一致(json.dumps sort_keys=True)
2026-04-04 Claude Code (P1): Anti-Pattern 閉環查詢鍵
"""
import hashlib
import json as _json
key = (
"|".join(sorted(self.alert_names)) + "||" +
"|".join(sorted(self.affected_services)) + "||" +
_json.dumps(self.label_patterns, sort_keys=True)
)
return hashlib.sha256(key.encode()).hexdigest()[:16]
- Step 5:執行測試
cd apps/api && python -m pytest tests/test_p1_knowledge_auto_harvesting.py::TestSymptomPatternHash tests/test_p1_knowledge_auto_harvesting.py::TestKnowledgeEntryTypes -v
預期:全部 PASS
- Step 6:確認既有測試不受影響
cd apps/api && python -m pytest tests/test_playbook_service.py -v
預期:全部 PASS
- Step 7:Commit
git add apps/api/src/models/knowledge.py apps/api/src/models/playbook.py apps/api/tests/test_p1_knowledge_auto_harvesting.py
git commit -m "feat(models): 新增 ANTI_PATTERN/AUTO_RUNBOOK EntryType + symptoms_hash + SymptomPattern.compute_hash() (P1)"
Task 3:KnowledgeService 新增 check_anti_pattern()
Files:
-
Modify:
apps/api/src/services/knowledge_service.py -
Step 1:新增測試
在 tests/test_p1_knowledge_auto_harvesting.py 新增:
class TestCheckAntiPattern:
"""KnowledgeService.check_anti_pattern() — 7 天內 ANTI_PATTERN 查詢"""
@pytest.mark.asyncio
async def test_returns_empty_when_no_anti_pattern(self):
"""無 ANTI_PATTERN 記錄時回傳空 list"""
from src.services.knowledge_service import KnowledgeService
from unittest.mock import AsyncMock, MagicMock
svc = KnowledgeService.__new__(KnowledgeService)
mock_repo = AsyncMock()
mock_repo.find_by_symptoms_hash = AsyncMock(return_value=[])
svc._repo = mock_repo
svc._embed_svc = MagicMock()
svc._pending_tasks = set()
result = await svc.check_anti_pattern("abc123def456789a", days=7)
assert result == []
@pytest.mark.asyncio
async def test_returns_anti_patterns_within_days(self):
"""7 天內有 ANTI_PATTERN → 回傳該記錄"""
from src.services.knowledge_service import KnowledgeService
from src.models.knowledge import KnowledgeEntry, EntryType, EntrySource, EntryStatus
from unittest.mock import AsyncMock, MagicMock
from datetime import datetime, timezone, timedelta
svc = KnowledgeService.__new__(KnowledgeService)
recent_entry = KnowledgeEntry(
id="ap-001",
title="Pod OOM 修復失敗",
content="失敗原因: ...",
entry_type=EntryType.ANTI_PATTERN,
category="auto-generated",
source=EntrySource.AI,
status=EntryStatus.PUBLISHED,
symptoms_hash="abc123def456789a",
created_at=datetime.now(timezone.utc) - timedelta(days=3),
)
mock_repo = AsyncMock()
mock_repo.find_by_symptoms_hash = AsyncMock(return_value=[recent_entry])
svc._repo = mock_repo
svc._embed_svc = MagicMock()
svc._pending_tasks = set()
result = await svc.check_anti_pattern("abc123def456789a", days=7)
assert len(result) == 1
assert result[0].id == "ap-001"
- Step 2:執行確認失敗
cd apps/api && python -m pytest tests/test_p1_knowledge_auto_harvesting.py::TestCheckAntiPattern -v
預期:FAIL(check_anti_pattern 不存在)
- Step 3:在
knowledge_service.py新增方法
在 KnowledgeService class 中,create_entry() 方法之後新增:
async def check_anti_pattern(
self,
symptoms_hash: str,
days: int = 7,
) -> list:
"""
查詢近期相同症狀的失敗案例(ANTI_PATTERN)
P1 2026-04-04 Claude Code: Anti-Pattern 閉環攔截查詢鍵
時間範圍: days 天內(預設 7 天)
回傳: 符合條件的 KnowledgeEntry list(通常 0-3 筆)
Args:
symptoms_hash: SymptomPattern.compute_hash() 產生的 16 char hash
days: 往回查幾天(預設 7)
Returns:
list[KnowledgeEntry]: ANTI_PATTERN 記錄,空 list 表示無紀錄
"""
from datetime import datetime, timezone, timedelta
from src.models.knowledge import EntryType
cutoff = datetime.now(timezone.utc) - timedelta(days=days)
try:
async with get_db_context() as db:
repo = KnowledgeDBRepository(db)
entries = await repo.find_by_symptoms_hash(
symptoms_hash=symptoms_hash,
entry_type=EntryType.ANTI_PATTERN,
created_after=cutoff,
)
return entries
except Exception as e:
logger.error(
"check_anti_pattern_error",
symptoms_hash=symptoms_hash,
error=str(e),
)
return [] # 查詢失敗時回傳空 list,不阻斷主流程
- Step 4:在
knowledge_repository.py新增find_by_symptoms_hash()
grep -n "class KnowledgeDBRepository\|async def find\|async def search" apps/api/src/repositories/knowledge_repository.py | head -15
在 KnowledgeDBRepository class 中新增:
async def find_by_symptoms_hash(
self,
symptoms_hash: str,
entry_type,
created_after,
) -> list:
"""
依 symptoms_hash + entry_type + 時間範圍查詢
P1 2026-04-04 Claude Code: Anti-Pattern 閉環查詢
"""
from sqlalchemy import select, and_
from src.db.models import KnowledgeEntryORM
stmt = select(KnowledgeEntryORM).where(
and_(
KnowledgeEntryORM.symptoms_hash == symptoms_hash,
KnowledgeEntryORM.entry_type == entry_type.value,
KnowledgeEntryORM.created_at >= created_after,
)
).order_by(KnowledgeEntryORM.created_at.desc())
result = await self._db.execute(stmt)
rows = result.scalars().all()
return [KnowledgeEntry.model_validate(row) for row in rows]
- Step 5:確認
KnowledgeEntryORM有 symptoms_hash 欄位
grep -n "symptoms_hash\|class KnowledgeEntryORM" apps/api/src/db/models.py | head -10
若無,在 KnowledgeEntryORM class 中新增(在 related_playbook_id 後):
symptoms_hash: Mapped[str | None] = mapped_column(String(16), nullable=True, index=True)
- Step 6:執行測試(使用 mock,不依賴真實 DB)
cd apps/api && python -m pytest tests/test_p1_knowledge_auto_harvesting.py::TestCheckAntiPattern -v
預期:PASS
- Step 7:Commit
git add apps/api/src/services/knowledge_service.py apps/api/src/repositories/knowledge_repository.py apps/api/src/db/models.py apps/api/tests/test_p1_knowledge_auto_harvesting.py
git commit -m "feat(knowledge): check_anti_pattern() + find_by_symptoms_hash() + ORM symptoms_hash 欄位 (P1)"
Task 4:AutoRepairService — 執行前 Anti-Pattern 閘門
Files:
-
Modify:
apps/api/src/services/auto_repair_service.py -
Step 1:新增閘門測試
在 tests/test_p1_knowledge_auto_harvesting.py 新增:
class TestAntiPatternGate:
"""AutoRepairService.evaluate_auto_repair() — Anti-Pattern 閉環閘門"""
@pytest.mark.asyncio
async def test_blocked_when_anti_pattern_exists(self):
"""7 天內有 ANTI_PATTERN → 阻斷自動修復,強制 HITL"""
from src.services.auto_repair_service import AutoRepairService
from src.models.incident import Incident, IncidentStatus, Severity, Signal
from src.models.playbook import SymptomPattern
from src.models.knowledge import KnowledgeEntry, EntryType, EntrySource, EntryStatus
from unittest.mock import AsyncMock, MagicMock, patch
from src.utils.timezone import now_taipei
# 建立 mock knowledge service,回傳一筆 ANTI_PATTERN
mock_knowledge_svc = AsyncMock()
anti_pattern_entry = MagicMock(spec=KnowledgeEntry)
anti_pattern_entry.title = "Pod OOM 修復失敗(已知無效)"
mock_knowledge_svc.check_anti_pattern = AsyncMock(return_value=[anti_pattern_entry])
svc = AutoRepairService(knowledge_service=mock_knowledge_svc)
incident = Incident(
incident_id="INC-001",
title="Pod OOMKilled",
status=IncidentStatus.OPEN,
severity=Severity.P2,
signals=[Signal(name="OOMKilled", value="1", source="prometheus")],
created_at=now_taipei(),
)
decision = await svc.evaluate_auto_repair(incident)
assert decision.can_auto_repair is False
assert decision.blocked_by == "ANTI_PATTERN"
assert "Pod OOM 修復失敗" in decision.reason
@pytest.mark.asyncio
async def test_proceeds_when_no_anti_pattern(self):
"""無 ANTI_PATTERN 記錄 → 繼續原有評估流程"""
from src.services.auto_repair_service import AutoRepairService
from src.models.incident import Incident, IncidentStatus, Severity, Signal
from unittest.mock import AsyncMock, MagicMock
from src.utils.timezone import now_taipei
mock_knowledge_svc = AsyncMock()
mock_knowledge_svc.check_anti_pattern = AsyncMock(return_value=[])
mock_playbook_svc = AsyncMock()
mock_playbook_svc.get_recommendations = AsyncMock(return_value=[])
svc = AutoRepairService(
knowledge_service=mock_knowledge_svc,
playbook_service=mock_playbook_svc,
)
incident = Incident(
incident_id="INC-002",
title="高 CPU",
status=IncidentStatus.OPEN,
severity=Severity.P2,
signals=[Signal(name="HighCPU", value="95", source="prometheus")],
created_at=now_taipei(),
)
decision = await svc.evaluate_auto_repair(incident)
# 無 Playbook match → can_auto_repair=False,但 blocked_by 不是 ANTI_PATTERN
assert decision.blocked_by != "ANTI_PATTERN"
mock_knowledge_svc.check_anti_pattern.assert_called_once()
- Step 2:執行確認失敗
cd apps/api && python -m pytest tests/test_p1_knowledge_auto_harvesting.py::TestAntiPatternGate -v
預期:FAIL(AutoRepairService 不接受 knowledge_service 參數)
- Step 3:修改
auto_repair_service.py— 注入 KnowledgeService + 加閘門
找到 AutoRepairService.__init__() 方法(約 L140),新增 knowledge_service 參數:
def __init__(
self,
playbook_service: IPlaybookService | None = None,
knowledge_service=None, # P1 2026-04-04 Claude Code: Anti-Pattern gate
) -> None:
self._playbook_service = playbook_service or get_playbook_service()
# P1 2026-04-04 Claude Code: 延遲 import 避免循環依賴
self._knowledge_service = knowledge_service
找到 evaluate_auto_repair() 方法中,symptoms = self._extract_symptoms(incident) 這行(約 L197),在其後、get_recommendations() 呼叫前插入:
# P1 2026-04-04 Claude Code: Anti-Pattern 閉環閘門
# 查詢 7 天內相同症狀的失敗案例,避免重複踩坑
symptoms_hash = symptoms.compute_hash()
try:
if self._knowledge_service is None:
from src.services.knowledge_service import get_knowledge_service
self._knowledge_service = get_knowledge_service()
anti_patterns = await self._knowledge_service.check_anti_pattern(
symptoms_hash=symptoms_hash,
days=7,
)
if anti_patterns:
logger.warning(
"auto_repair_blocked_anti_pattern",
incident_id=incident.incident_id,
symptoms_hash=symptoms_hash,
anti_pattern_title=anti_patterns[0].title,
)
return AutoRepairDecision(
can_auto_repair=False,
blocked_by="ANTI_PATTERN",
reason=f"過去 7 天有失敗案例: {anti_patterns[0].title}",
)
except Exception as e:
# 查詢失敗不阻斷,記錄 warning 後繼續
logger.warning(
"auto_repair_anti_pattern_check_failed",
incident_id=incident.incident_id,
error=str(e),
)
- Step 4:執行測試
cd apps/api && python -m pytest tests/test_p1_knowledge_auto_harvesting.py::TestAntiPatternGate -v
預期:PASS
- Step 5:確認既有 auto_repair 測試不受影響
cd apps/api && python -m pytest tests/test_auto_repair_service.py -v
預期:全部 PASS
- Step 6:Commit
git add apps/api/src/services/auto_repair_service.py apps/api/tests/test_p1_knowledge_auto_harvesting.py
git commit -m "feat(auto-repair): Anti-Pattern 閉環閘門 — 7 天內失敗案例直接阻斷 (P1)"
Task 5:NemotronRunbookGenerator 服務
Files:
-
Create:
apps/api/src/services/runbook_generator.py -
Step 1:新增測試
在 tests/test_p1_knowledge_auto_harvesting.py 新增:
class TestNemotronRunbookGenerator:
"""NemotronRunbookGenerator — 生成 Runbook / Anti-Pattern 條目"""
@pytest.mark.asyncio
async def test_generate_success_creates_auto_runbook(self):
"""修復成功 → 生成 AUTO_RUNBOOK(status=DRAFT)"""
from src.services.runbook_generator import NemotronRunbookGenerator
from src.models.incident import Incident, IncidentStatus, Severity, Signal
from src.models.playbook import Playbook, PlaybookStatus, SymptomPattern
from src.services.auto_repair_service import AutoRepairResult
from src.models.knowledge import EntryType, EntryStatus
from src.services.ai_providers.interfaces import AIResult
from unittest.mock import AsyncMock, MagicMock, patch
from src.utils.timezone import now_taipei
mock_ai_router = AsyncMock()
mock_ai_router.execute = AsyncMock(return_value=AIResult(
raw_response='{"symptoms_description":"Pod 記憶體不足","root_cause_analysis":"OOM","execution_steps":["kubectl rollout restart"],"verification_steps":["kubectl get pods"],"precautions":["備份資料"],"impact_scope":"awoooi-api","related_incident_ids":["INC-001"],"prevention_suggestions":["增加記憶體限制"],"applicable_conditions":["OOMKilled 告警"]}',
success=True,
provider="nemotron",
))
mock_knowledge_svc = AsyncMock()
created_entry = MagicMock()
created_entry.id = "kb-001"
mock_knowledge_svc.create_entry = AsyncMock(return_value=created_entry)
gen = NemotronRunbookGenerator(
ai_router=mock_ai_router,
knowledge_service=mock_knowledge_svc,
)
incident = Incident(
incident_id="INC-001",
title="Pod OOMKilled",
status=IncidentStatus.RESOLVED,
severity=Severity.P2,
signals=[Signal(name="OOMKilled", value="1", source="prometheus")],
created_at=now_taipei(),
)
playbook = Playbook(
playbook_id="PB-001",
name="OOM Restart",
status=PlaybookStatus.ACTIVE,
symptom_pattern=SymptomPattern(alert_names=["OOMKilled"]),
repair_steps=[],
)
repair_result = AutoRepairResult(
success=True,
playbook_id="PB-001",
incident_id="INC-001",
executed_steps=["kubectl rollout restart deployment/awoooi-api"],
)
entry = await gen.generate(incident, playbook, repair_result)
assert entry is not None
mock_knowledge_svc.create_entry.assert_called_once()
call_args = mock_knowledge_svc.create_entry.call_args[0][0]
assert call_args.entry_type == EntryType.AUTO_RUNBOOK
assert call_args.status == EntryStatus.DRAFT
@pytest.mark.asyncio
async def test_generate_anti_pattern_creates_published_entry(self):
"""修復失敗 → 生成 ANTI_PATTERN(status=PUBLISHED,直接發布)"""
from src.services.runbook_generator import NemotronRunbookGenerator
from src.models.incident import Incident, IncidentStatus, Severity, Signal
from src.models.playbook import Playbook, PlaybookStatus, SymptomPattern
from src.services.auto_repair_service import AutoRepairResult
from src.models.knowledge import EntryType, EntryStatus
from src.services.ai_providers.interfaces import AIResult
from unittest.mock import AsyncMock, MagicMock
from src.utils.timezone import now_taipei
mock_ai_router = AsyncMock()
mock_ai_router.execute = AsyncMock(return_value=AIResult(
raw_response='{"failure_reason":"kubectl 超時","ineffective_steps":["kubectl rollout restart"],"alternative_suggestions":["手動 drain node"],"applicable_conditions":["節點磁碟壓力時此方法無效"]}',
success=True,
provider="nemotron",
))
mock_knowledge_svc = AsyncMock()
created_entry = MagicMock()
created_entry.id = "kb-002"
mock_knowledge_svc.create_entry = AsyncMock(return_value=created_entry)
gen = NemotronRunbookGenerator(
ai_router=mock_ai_router,
knowledge_service=mock_knowledge_svc,
)
incident = Incident(
incident_id="INC-002",
title="Pod OOMKilled",
status=IncidentStatus.OPEN,
severity=Severity.P2,
signals=[Signal(name="OOMKilled", value="1", source="prometheus")],
created_at=now_taipei(),
)
playbook = Playbook(
playbook_id="PB-001",
name="OOM Restart",
status=PlaybookStatus.ACTIVE,
symptom_pattern=SymptomPattern(alert_names=["OOMKilled"]),
repair_steps=[],
)
repair_result = AutoRepairResult(
success=False,
playbook_id="PB-001",
incident_id="INC-002",
executed_steps=[],
error="kubectl timeout",
)
entry = await gen.generate_anti_pattern(incident, playbook, repair_result)
assert entry is not None
call_args = mock_knowledge_svc.create_entry.call_args[0][0]
assert call_args.entry_type == EntryType.ANTI_PATTERN
assert call_args.status == EntryStatus.PUBLISHED
- Step 2:執行確認失敗
cd apps/api && python -m pytest tests/test_p1_knowledge_auto_harvesting.py::TestNemotronRunbookGenerator -v
預期:FAIL(runbook_generator 不存在)
- Step 3:建立
services/runbook_generator.py
"""
Runbook Generator Service - P1 Knowledge Auto-Harvesting
=========================================================
修復成功/失敗後,由 Nemotron 自動生成 Runbook / Anti-Pattern 條目
設計原則:
- 完全異步,不阻塞 AutoRepairService 主流程
- 生成失敗不影響修復結果(防禦性工程)
- SUCCESS → AUTO_RUNBOOK (DRAFT) → 人工審核
- FAILURE → ANTI_PATTERN (PUBLISHED) → 直接發布
建立時間: 2026-04-04 (台北時區)
建立者: Claude Code (P1 Knowledge Auto-Harvesting)
"""
from __future__ import annotations
import json
from typing import TYPE_CHECKING, Any
import structlog
from src.models.knowledge import EntrySource, EntryStatus, EntryType, KnowledgeEntryCreate
if TYPE_CHECKING:
from src.models.incident import Incident
from src.models.playbook import Playbook
from src.services.auto_repair_service import AutoRepairResult
logger = structlog.get_logger(__name__)
# =============================================================================
# Prompts
# =============================================================================
_SUCCESS_PROMPT_TEMPLATE = """你是 SRE 知識管理系統。根據以下修復案例,生成一份完整的 Runbook。
## 事件資訊
- Incident ID: {incident_id}
- 標題: {incident_title}
- 嚴重度: {severity}
- 告警: {alert_names}
- 受影響服務: {affected_services}
## 執行的 Playbook
- Playbook ID: {playbook_id}
- Playbook 名稱: {playbook_name}
## 執行結果
- 狀態: 成功
- 執行步驟: {executed_steps}
請以 JSON 格式回應,包含以下欄位:
{{
"symptoms_description": "症狀描述",
"root_cause_analysis": "根因分析",
"execution_steps": ["步驟1", "步驟2"],
"verification_steps": ["驗證1", "驗證2"],
"precautions": ["注意1", "注意2"],
"impact_scope": "影響範圍",
"related_incident_ids": ["{incident_id}"],
"prevention_suggestions": ["預防1", "預防2"],
"applicable_conditions": ["條件1", "條件2"]
}}"""
_FAILURE_PROMPT_TEMPLATE = """你是 SRE 知識管理系統。根據以下修復失敗案例,生成一份 Anti-Pattern 記錄。
## 事件資訊
- Incident ID: {incident_id}
- 標題: {incident_title}
- 嚴重度: {severity}
- 告警: {alert_names}
## 嘗試的 Playbook
- Playbook ID: {playbook_id}
- Playbook 名稱: {playbook_name}
## 失敗資訊
- 錯誤: {error}
- 已執行步驟: {executed_steps}
請以 JSON 格式回應,包含以下欄位:
{{
"failure_reason": "失敗原因",
"ineffective_steps": ["無效步驟1"],
"alternative_suggestions": ["替代方案1"],
"applicable_conditions": ["此方法不適用的條件"]
}}"""
# =============================================================================
# NemotronRunbookGenerator
# =============================================================================
class NemotronRunbookGenerator:
"""
Nemotron 驅動的 Runbook 生成器
職責: 接收修復結果 → 呼叫 Nemotron → 寫入 KB
不負責: Telegram 推送(由 caller 負責)、主流程錯誤處理
"""
def __init__(
self,
ai_router=None,
knowledge_service=None,
) -> None:
self._ai_router = ai_router
self._knowledge_service = knowledge_service
def _get_ai_router(self):
if self._ai_router is None:
from src.services.ai_router import get_ai_router
self._ai_router = get_ai_router()
return self._ai_router
def _get_knowledge_service(self):
if self._knowledge_service is None:
from src.services.knowledge_service import get_knowledge_service
self._knowledge_service = get_knowledge_service()
return self._knowledge_service
async def generate(
self,
incident: Incident,
playbook: Playbook,
repair_result: AutoRepairResult,
):
"""
修復成功後生成完整 Runbook → KB (AUTO_RUNBOOK, DRAFT)
Returns:
KnowledgeEntry | None: 建立成功的條目,失敗時回傳 None(不 raise)
"""
symptoms_hash = playbook.symptom_pattern.compute_hash()
prompt = _SUCCESS_PROMPT_TEMPLATE.format(
incident_id=incident.incident_id,
incident_title=incident.title,
severity=incident.severity.value,
alert_names=", ".join(playbook.symptom_pattern.alert_names),
affected_services=", ".join(playbook.symptom_pattern.affected_services),
playbook_id=playbook.playbook_id,
playbook_name=playbook.name,
executed_steps="\n".join(repair_result.executed_steps),
)
try:
router = self._get_ai_router()
result = await router.execute(
prompt=prompt,
provider_order=["nemotron", "gemini"],
context={"task_type": "runbook_generation"},
)
if not result.success:
logger.error("runbook_generation_ai_failed", incident_id=incident.incident_id)
return None
content = self._build_success_content(result.raw_response)
title = f"[AUTO] {incident.title} — Runbook"
svc = self._get_knowledge_service()
entry = await svc.create_entry(KnowledgeEntryCreate(
title=title,
content=content,
entry_type=EntryType.AUTO_RUNBOOK,
category="auto-generated",
tags=playbook.symptom_pattern.alert_names[:3],
source=EntrySource.AI,
status=EntryStatus.DRAFT,
related_incident_id=incident.incident_id,
related_playbook_id=playbook.playbook_id,
symptoms_hash=symptoms_hash,
))
logger.info(
"runbook_generated",
incident_id=incident.incident_id,
entry_id=entry.id,
symptoms_hash=symptoms_hash,
)
return entry
except Exception as e:
logger.error(
"runbook_generation_error",
incident_id=incident.incident_id,
error=str(e),
)
return None
async def generate_anti_pattern(
self,
incident: Incident,
playbook: Playbook,
repair_result: AutoRepairResult,
):
"""
修復失敗後生成 Anti-Pattern → KB (ANTI_PATTERN, PUBLISHED)
Returns:
KnowledgeEntry | None: 建立成功的條目,失敗時回傳 None(不 raise)
"""
symptoms_hash = playbook.symptom_pattern.compute_hash()
prompt = _FAILURE_PROMPT_TEMPLATE.format(
incident_id=incident.incident_id,
incident_title=incident.title,
severity=incident.severity.value,
alert_names=", ".join(playbook.symptom_pattern.alert_names),
playbook_id=playbook.playbook_id,
playbook_name=playbook.name,
error=repair_result.error or "未知錯誤",
executed_steps="\n".join(repair_result.executed_steps),
)
try:
router = self._get_ai_router()
result = await router.execute(
prompt=prompt,
provider_order=["nemotron", "gemini"],
context={"task_type": "anti_pattern_generation"},
)
if not result.success:
logger.error("anti_pattern_generation_ai_failed", incident_id=incident.incident_id)
return None
content = self._build_failure_content(result.raw_response)
title = f"[ANTI-PATTERN] {incident.title} — 已知無效方案"
svc = self._get_knowledge_service()
entry = await svc.create_entry(KnowledgeEntryCreate(
title=title,
content=content,
entry_type=EntryType.ANTI_PATTERN,
category="auto-generated",
tags=playbook.symptom_pattern.alert_names[:3],
source=EntrySource.AI,
status=EntryStatus.PUBLISHED,
related_incident_id=incident.incident_id,
related_playbook_id=playbook.playbook_id,
symptoms_hash=symptoms_hash,
))
logger.info(
"anti_pattern_recorded",
incident_id=incident.incident_id,
entry_id=entry.id,
symptoms_hash=symptoms_hash,
)
return entry
except Exception as e:
logger.error(
"anti_pattern_generation_error",
incident_id=incident.incident_id,
error=str(e),
)
return None
def _build_success_content(self, raw_response: str) -> str:
"""將 Nemotron JSON 回應轉為 Markdown 格式 Runbook"""
try:
data = json.loads(raw_response)
except json.JSONDecodeError:
return raw_response # fallback: 原始文字
sections = [
f"## 症狀描述\n{data.get('symptoms_description', '')}",
f"## 根因分析\n{data.get('root_cause_analysis', '')}",
"## 執行步驟\n" + "\n".join(f"{i+1}. {s}" for i, s in enumerate(data.get('execution_steps', []))),
"## 驗證步驟\n" + "\n".join(f"- {s}" for s in data.get('verification_steps', [])),
"## 注意事項\n" + "\n".join(f"- {s}" for s in data.get('precautions', [])),
f"## 影響範圍\n{data.get('impact_scope', '')}",
"## 相關 Incident\n" + "\n".join(f"- {i}" for i in data.get('related_incident_ids', [])),
"## 下次預防建議\n" + "\n".join(f"- {s}" for s in data.get('prevention_suggestions', [])),
"## 適用條件\n" + "\n".join(f"- {s}" for s in data.get('applicable_conditions', [])),
]
return "\n\n".join(sections)
def _build_failure_content(self, raw_response: str) -> str:
"""將 Nemotron JSON 回應轉為 Markdown 格式 Anti-Pattern"""
try:
data = json.loads(raw_response)
except json.JSONDecodeError:
return raw_response
sections = [
f"## 失敗原因\n{data.get('failure_reason', '')}",
"## 無效步驟\n" + "\n".join(f"- {s}" for s in data.get('ineffective_steps', [])),
"## 替代方案建議\n" + "\n".join(f"- {s}" for s in data.get('alternative_suggestions', [])),
"## 不適用條件\n" + "\n".join(f"- {s}" for s in data.get('applicable_conditions', [])),
]
return "\n\n".join(sections)
# =============================================================================
# Singleton
# =============================================================================
_runbook_generator: NemotronRunbookGenerator | None = None
def get_runbook_generator() -> NemotronRunbookGenerator:
global _runbook_generator
if _runbook_generator is None:
_runbook_generator = NemotronRunbookGenerator()
return _runbook_generator
- Step 4:執行測試
cd apps/api && python -m pytest tests/test_p1_knowledge_auto_harvesting.py::TestNemotronRunbookGenerator -v
預期:PASS
- Step 5:Commit
git add apps/api/src/services/runbook_generator.py apps/api/tests/test_p1_knowledge_auto_harvesting.py
git commit -m "feat(runbook-generator): NemotronRunbookGenerator — SUCCESS→AUTO_RUNBOOK / FAILURE→ANTI_PATTERN (P1)"
Task 6:AutoRepairService — 執行後背景觸發生成
Files:
-
Modify:
apps/api/src/services/auto_repair_service.py -
Step 1:在
execute_auto_repair()結束後加入背景觸發
找到 execute_auto_repair() 中 return AutoRepairResult(success=True, ...) 的行(約 L330),在 return 前插入:
# P1 2026-04-04 Claude Code: 背景異步生成 Runbook(不阻塞主流程)
self._schedule_runbook_generation(incident, playbook, result=AutoRepairResult(
success=True,
playbook_id=playbook.playbook_id,
incident_id=incident.incident_id,
executed_steps=executed_steps,
execution_time_ms=execution_time,
))
找到 return AutoRepairResult(success=False, ...) 的行(約 L350),在 return 前插入:
# P1 2026-04-04 Claude Code: 背景異步生成 Anti-Pattern(不阻塞主流程)
failure_result = AutoRepairResult(
success=False,
playbook_id=playbook.playbook_id,
incident_id=incident.incident_id,
executed_steps=executed_steps,
error=str(e),
execution_time_ms=execution_time,
)
self._schedule_anti_pattern_generation(incident, playbook, failure_result)
在 AutoRepairService class 末尾新增兩個私有方法:
def _schedule_runbook_generation(
self,
incident,
playbook,
result,
) -> None:
"""
背景排程 Runbook 生成(fire-and-forget,不阻塞主流程)
P1 2026-04-04 Claude Code: 異步不阻塞設計
"""
import asyncio
async def _generate():
try:
from src.services.runbook_generator import get_runbook_generator
gen = get_runbook_generator()
entry = await gen.generate(incident, playbook, result)
if entry:
logger.info(
"runbook_generation_scheduled_done",
incident_id=incident.incident_id,
entry_id=entry.id,
)
except Exception as e:
logger.error(
"runbook_generation_scheduled_error",
incident_id=incident.incident_id,
error=str(e),
)
task = asyncio.create_task(_generate())
# 防止 GC 提前回收 task
self._background_tasks = getattr(self, "_background_tasks", set())
self._background_tasks.add(task)
task.add_done_callback(self._background_tasks.discard)
def _schedule_anti_pattern_generation(
self,
incident,
playbook,
result,
) -> None:
"""
背景排程 Anti-Pattern 生成(fire-and-forget,不阻塞主流程)
P1 2026-04-04 Claude Code: 異步不阻塞設計
"""
import asyncio
async def _generate():
try:
from src.services.runbook_generator import get_runbook_generator
gen = get_runbook_generator()
entry = await gen.generate_anti_pattern(incident, playbook, result)
if entry:
logger.info(
"anti_pattern_generation_scheduled_done",
incident_id=incident.incident_id,
entry_id=entry.id,
)
except Exception as e:
logger.error(
"anti_pattern_generation_scheduled_error",
incident_id=incident.incident_id,
error=str(e),
)
task = asyncio.create_task(_generate())
self._background_tasks = getattr(self, "_background_tasks", set())
self._background_tasks.add(task)
task.add_done_callback(self._background_tasks.discard)
- Step 2:執行全部 P1 測試
cd apps/api && python -m pytest tests/test_p1_knowledge_auto_harvesting.py -v
預期:全部 PASS
- Step 3:確認既有測試不受影響
cd apps/api && python -m pytest tests/test_auto_repair_service.py tests/test_playbook_service.py -v
預期:全部 PASS
- Step 4:Commit
git add apps/api/src/services/auto_repair_service.py
git commit -m "feat(auto-repair): 執行後背景觸發 Runbook/Anti-Pattern 生成(異步不阻塞)(P1)"
驗收標準
# 全部 P1 測試通過
cd apps/api && python -m pytest tests/test_p1_knowledge_auto_harvesting.py -v
# 既有測試未破壞
cd apps/api && python -m pytest tests/test_auto_repair_service.py tests/test_playbook_service.py tests/test_smart_router.py -v
# Migration 語法驗證
psql -h localhost -U postgres -d awoooi_dev -f apps/api/migrations/phase8_knowledge_symptoms_hash.sql
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com