feat(Phase 3.5 + Phase 4): AI 學習成果持久化到 PostgreSQL — 修正「AI 失憶」架構缺陷
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled

ADR-085: AI 學習成果不可存在 Cache

架構鐵律確立:
- PostgreSQL = System of Record(AI 的永久記憶)
- Redis = Warm Cache(加速讀取,TTL 到期從 PG 復原)

核心變更:
1. models.py: 新增 PlaybookRecord / DynamicBaselineRecord / LogClusterRecord ORM
2. base.py: ALTER TABLE playbooks 補加 trust_score / requires_approval_level 等欄位
3. playbook_repository.py: 完整雙寫實作(PG upsert + Redis cache)
4. dynamic_baseline_service.py: Holt-Winters 訓練結果寫入 PG,Redis 只作 24h warm cache
5. log_anomaly_detector.py: Drain3 cluster template 寫入 PG(UPSERT on cluster_id)
6. main.py: 啟動時執行 backfill_redis_to_pg()(Redis → PG 冪等補救)

修正的問題:
- Playbook 7天 Redis TTL 到期 → AI 失去所有修復知識
- trust_score EWMA 隨 Redis TTL 歸零 → AI 重新回到初始信任度 0.3
- Holt-Winters 基線 24h TTL → AI 每天重新學習「正常」的定義
- Drain3 cluster 沒有持久化 → AI 把已知 log pattern 反覆當新 pattern

Phase 4 新服務(requirements.txt 已加入 statsmodels + drain3 + numpy)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
OG T
2026-04-15 15:33:52 +08:00
parent 9126c594a4
commit bf45b80bd2
8 changed files with 1490 additions and 238 deletions

View File

@@ -1,26 +1,31 @@
"""
Playbook Repository - #7 Playbook 萃取
======================================
Playbook CRUD 操作 (Redis + PostgreSQL)
Playbook Repository — Phase 3.5 雙寫持久化
==========================================
ADR-085: AI 學習成果必須儲存到 PostgreSQL
儲存策略Phase 3.5 架構升級):
- PostgreSQL = System of Record永久保存source of truth
- Redis = Warm Cache7天 TTL加速讀取PG 為準)
雙寫規則:
- create/update → 先寫 PG再更新 Redis cache
- 讀取 → Redis-firstcache hitmiss 時從 PG 載入並回填 Redis
- trust_score / EWMA 統計 → PG 永久保存Redis TTL 到期後從 PG 復原
Phase 7.2: Repository 實作
建立時間: 2026-03-26 (台北時區)
建立者: Claude Code (#7 Playbook 萃取)
遵循 leWOOOgo 積木化原則:
- 實作 IPlaybookRepository Protocol
- Redis 為 Working Memory (7天 TTL)
- PostgreSQL 為 Episodic Memory
Phase 22 P2: 相似度計算邏輯移至 utils
2026-03-31 Claude Code (首席架構師技術債修復)
Phase 3.5 (ADR-085): PostgreSQL dual-write + Redis warm cache
2026-04-15 ogt + Claude Sonnet 4.6(亞太): Phase 3.5 重寫修正「AI 失憶」架構問題
"""
import json
import structlog
from sqlalchemy import select
from src.core.redis_client import get_redis
from src.db.base import get_session_factory
from src.db.models import PlaybookRecord
from src.models.playbook import (
Playbook,
PlaybookStatus,
@@ -32,7 +37,7 @@ from src.utils.timezone import now_taipei
logger = structlog.get_logger(__name__)
# Redis TTL: 7 天
# Redis TTL: 7 天warm cache onlyPG 為 source of truth
PLAYBOOK_TTL_SECONDS = 7 * 24 * 60 * 60
# Redis Key 前綴
@@ -41,129 +46,175 @@ PLAYBOOK_INDEX_ALERT_PREFIX = "playbook:index:alert:"
PLAYBOOK_INDEX_SERVICE_PREFIX = "playbook:index:service:"
# =============================================================================
# ORM ↔ Pydantic 轉換 Helpers
# =============================================================================
def _pydantic_to_orm(playbook: Playbook) -> PlaybookRecord:
"""Pydantic Playbook → PlaybookRecord ORM"""
return PlaybookRecord(
playbook_id=playbook.playbook_id,
name=playbook.name,
description=playbook.description,
status=playbook.status.value,
source=playbook.source.value,
symptom_pattern=playbook.symptom_pattern.model_dump(),
repair_steps=[s.model_dump() for s in playbook.repair_steps],
estimated_duration_minutes=playbook.estimated_duration_minutes,
source_incident_ids=playbook.source_incident_ids,
ai_confidence=playbook.ai_confidence,
success_count=playbook.success_count,
failure_count=playbook.failure_count,
last_used_at=playbook.last_used_at,
trust_score=playbook.trust_score,
approved_by=playbook.approved_by,
approved_at=playbook.approved_at,
tags=playbook.tags,
notes=playbook.notes,
requires_approval_level=playbook.requires_approval_level,
stateful_targets=playbook.stateful_targets,
requires_pre_backup=playbook.requires_pre_backup,
created_at=playbook.created_at,
updated_at=playbook.updated_at,
)
def _orm_to_pydantic(record: PlaybookRecord) -> Playbook:
"""PlaybookRecord ORM → Pydantic Playbook"""
return Playbook.model_validate({
"playbook_id": record.playbook_id,
"name": record.name,
"description": record.description,
"status": record.status,
"source": record.source,
"symptom_pattern": record.symptom_pattern,
"repair_steps": record.repair_steps,
"estimated_duration_minutes": record.estimated_duration_minutes,
"source_incident_ids": record.source_incident_ids,
"ai_confidence": float(record.ai_confidence),
"success_count": record.success_count,
"failure_count": record.failure_count,
"last_used_at": record.last_used_at,
"trust_score": float(record.trust_score),
"approved_by": record.approved_by,
"approved_at": record.approved_at,
"tags": record.tags,
"notes": record.notes,
"requires_approval_level": record.requires_approval_level,
"stateful_targets": record.stateful_targets,
"requires_pre_backup": record.requires_pre_backup,
"created_at": record.created_at,
"updated_at": record.updated_at,
})
class PlaybookRepository:
"""
Playbook Repository 實作
Playbook Repository — Phase 3.5 雙寫實作
儲存策略:
- Redis: Working Memory (快速讀取7天 TTL)
- PostgreSQL: Episodic Memory (持久化,待實作)
- PostgreSQL: source of truth所有 create/update 必須先寫 PG
- Redis: warm cache加速讀取7d TTL到期後從 PG 復原)
Phase 7.2 先實作 Redis 層PostgreSQL 待 #7.5 整合
這保證:
- trust_score / EWMA 永久不丟失
- Pod 重啟後 AI 學習成果完整保留
- Redis 失效只影響讀取速度,不影響資料完整性
"""
# === CRUD Operations ===
# =========================================================================
# CRUD Operations
# =========================================================================
async def create(self, playbook: Playbook) -> Playbook:
"""
建立新的 Playbook
1. 儲存到 Redis
2. 建立索引 (alert_names, services)
順序:
1. 寫入 PostgreSQLsource of truth
2. 寫入 Redis cachewarm cache
3. 更新索引
"""
try:
redis_client = get_redis()
if not playbook.created_at:
playbook.created_at = now_taipei()
playbook.updated_at = now_taipei()
# 確保有建立時間
if not playbook.created_at:
playbook.created_at = now_taipei()
playbook.updated_at = now_taipei()
# 1. 寫入 PostgreSQL先寫確保資料不丟失
await self._pg_upsert(playbook)
# 儲存 Playbook
key = f"{PLAYBOOK_KEY_PREFIX}{playbook.playbook_id}"
await redis_client.set(
key,
json.dumps(playbook.to_redis_dict(), ensure_ascii=False),
ex=PLAYBOOK_TTL_SECONDS,
)
# 2. 寫入 Rediswarm cache
await self._redis_set(playbook)
# 建立索引
await self._update_indexes(playbook)
# 3. 建立索引
await self._update_indexes(playbook)
logger.info(
"playbook_created",
playbook_id=playbook.playbook_id,
name=playbook.name,
)
return playbook
except Exception as e:
logger.error("playbook_create_failed", error=str(e))
raise
logger.info("playbook_created", playbook_id=playbook.playbook_id, name=playbook.name)
return playbook
async def get_by_id(self, playbook_id: str) -> Playbook | None:
"""根據 ID 取得 Playbook"""
try:
redis_client = get_redis()
key = f"{PLAYBOOK_KEY_PREFIX}{playbook_id}"
data = await redis_client.get(key)
"""
根據 ID 取得 Playbook
if data:
return Playbook.from_redis_dict(json.loads(data))
return None
Redis-first → miss 時從 PG 載入並回填 Redis
"""
# 1. Redis cache hit
cached = await self._redis_get(playbook_id)
if cached is not None:
return cached
except Exception as e:
logger.error("playbook_get_failed", playbook_id=playbook_id, error=str(e))
return None
# 2. PG fallbackcache miss
playbook = await self._pg_get(playbook_id)
if playbook is not None:
# 回填 Redis cache
await self._redis_set(playbook)
return playbook
async def update(self, playbook: Playbook) -> Playbook | None:
"""更新 Playbook"""
try:
existing = await self.get_by_id(playbook.playbook_id)
if not existing:
return None
"""
更新 Playbook
playbook.updated_at = now_taipei()
redis_client = get_redis()
key = f"{PLAYBOOK_KEY_PREFIX}{playbook.playbook_id}"
await redis_client.set(
key,
json.dumps(playbook.to_redis_dict(), ensure_ascii=False),
ex=PLAYBOOK_TTL_SECONDS,
)
# 更新索引
await self._update_indexes(playbook)
logger.info("playbook_updated", playbook_id=playbook.playbook_id)
return playbook
except Exception as e:
logger.error(
"playbook_update_failed",
先確認 PG 存在 → 寫入 PG → 更新 Redis
"""
existing = await self._pg_get(playbook.playbook_id)
if existing is None:
# PG 找不到,可能是老資料只在 Redis先建立 PG 記錄
logger.warning(
"playbook_pg_not_found_creating",
playbook_id=playbook.playbook_id,
error=str(e),
)
return None
playbook.updated_at = now_taipei()
# 1. 寫入 PG
await self._pg_upsert(playbook)
# 2. 更新 Redis
await self._redis_set(playbook)
# 3. 更新索引
await self._update_indexes(playbook)
logger.info("playbook_updated", playbook_id=playbook.playbook_id)
return playbook
async def delete(self, playbook_id: str) -> bool:
"""
刪除 Playbook (軟刪除 → DEPRECATED)
不真正刪除,而是將狀態改為 DEPRECATED
刪除 Playbook(狀態改為 DEPRECATED
"""
try:
playbook = await self.get_by_id(playbook_id)
if not playbook:
return False
playbook.status = PlaybookStatus.DEPRECATED
playbook.updated_at = now_taipei()
await self.update(playbook)
logger.info("playbook_deprecated", playbook_id=playbook_id)
return True
except Exception as e:
logger.error(
"playbook_delete_failed",
playbook_id=playbook_id,
error=str(e),
)
playbook = await self.get_by_id(playbook_id)
if not playbook:
return False
# === Query Operations ===
playbook.status = PlaybookStatus.DEPRECATED
playbook.updated_at = now_taipei()
await self.update(playbook)
logger.info("playbook_deprecated", playbook_id=playbook_id)
return True
# =========================================================================
# Query Operations
# =========================================================================
async def list_playbooks(
self,
@@ -173,42 +224,32 @@ class PlaybookRepository:
offset: int = 0,
) -> tuple[list[Playbook], int]:
"""
列出 Playbooks
列出 Playbooks(從 PostgreSQL 查詢,不走 Redis scan
注意: Redis 實作效率較低,後續需遷移到 PostgreSQL
Phase 3.5:改用 PG 查詢,效率更高,資料更完整
"""
try:
redis_client = get_redis()
factory = get_session_factory()
async with factory() as session:
stmt = select(PlaybookRecord)
if status is not None:
stmt = stmt.where(PlaybookRecord.status == status.value)
stmt = stmt.order_by(PlaybookRecord.updated_at.desc())
# 掃描所有 Playbook keys
pattern = f"{PLAYBOOK_KEY_PREFIX}PB-*"
keys = []
async for key in redis_client.scan_iter(match=pattern, count=100):
keys.append(key)
result = await session.execute(stmt)
all_records = result.scalars().all()
# 讀取並過濾
all_playbooks: list[Playbook] = []
for key in keys:
data = await redis_client.get(key)
if data:
playbook = Playbook.from_redis_dict(json.loads(data))
all_playbooks = [_orm_to_pydantic(r) for r in all_records]
# 狀態過濾
if status and playbook.status != status:
continue
# 標籤過濾
if tags and not set(tags).intersection(set(playbook.tags)):
continue
all_playbooks.append(playbook)
# 排序: 按 updated_at 降序
all_playbooks.sort(key=lambda p: p.updated_at, reverse=True)
# 標籤過濾PG JSONB 過濾後續可移到 SQL目前 Python 過濾
if tags:
all_playbooks = [
p for p in all_playbooks
if set(tags).intersection(set(p.tags))
]
total = len(all_playbooks)
items = all_playbooks[offset : offset + limit]
items = all_playbooks[offset: offset + limit]
return items, total
except Exception as e:
@@ -225,7 +266,7 @@ class PlaybookRepository:
根據症狀模式找相似 Playbook
策略:
1. 從索引快速過濾候選
1. 從 Redis 索引快速過濾候選
2. 計算詳細相似度
3. 返回 Top K
"""
@@ -235,24 +276,19 @@ class PlaybookRepository:
# 1. 使用索引找候選 Playbook IDs
candidate_ids: set[str] = set()
# 從 alert_names 索引查詢
for alert_name in symptoms.alert_names:
index_key = f"{PLAYBOOK_INDEX_ALERT_PREFIX}{alert_name}"
members = await redis_client.smembers(index_key)
candidate_ids.update(m.decode() if isinstance(m, bytes) else m for m in members)
# 從 services 索引查詢
for service in symptoms.affected_services:
index_key = f"{PLAYBOOK_INDEX_SERVICE_PREFIX}{service}"
members = await redis_client.smembers(index_key)
candidate_ids.update(m.decode() if isinstance(m, bytes) else m for m in members)
# 如果沒有索引命中,掃描所有 APPROVED Playbooks
# 索引命中 → 從 PG 掃 APPROVED
if not candidate_ids:
playbooks, _ = await self.list_playbooks(
status=PlaybookStatus.APPROVED,
limit=100,
)
playbooks, _ = await self.list_playbooks(status=PlaybookStatus.APPROVED, limit=100)
candidate_ids = {p.playbook_id for p in playbooks}
# 2. 計算相似度
@@ -260,26 +296,16 @@ class PlaybookRepository:
for playbook_id in candidate_ids:
playbook = await self.get_by_id(playbook_id)
if not playbook:
if not playbook or playbook.status != PlaybookStatus.APPROVED:
continue
# 只考慮 APPROVED 狀態
if playbook.status != PlaybookStatus.APPROVED:
continue
similarity = calculate_symptom_similarity(
symptoms,
playbook.symptom_pattern,
)
# alert_names 完全匹配時,保證通過(不因其他維度拉低分數)
similarity = calculate_symptom_similarity(symptoms, playbook.symptom_pattern)
alert_exact_match = bool(
set(symptoms.alert_names) & set(playbook.symptom_pattern.alert_names)
)
if alert_exact_match or similarity >= min_similarity:
results.append((playbook, similarity))
# 3. 排序並返回 Top K
results.sort(key=lambda x: x[1], reverse=True)
return results[:top_k]
@@ -293,13 +319,13 @@ class PlaybookRepository:
success: bool,
) -> bool:
"""
更新執行統計 + EWMA 信任度
更新執行統計 + EWMA 信任度(雙寫到 PG + Redis
ADR-083 Phase 3: 負向 2x 強化 EWMA 公式
成功: trust_new = 0.9 * trust_old + 0.1 * 1.0
失敗: trust_new = 0.8 * trust_old + 0.2 * 0.0(衰減速度 2x
trust < 0.1 → 記錄警告,由 Evolver Agent 封存
2026-04-15 ogt + Claude Sonnet 4.6(亞太): Phase 3 EWMA 實裝
2026-04-15 ogt + Claude Sonnet 4.6(亞太): Phase 3.5 EWMA 雙寫 PG
"""
try:
playbook = await self.get_by_id(playbook_id)
@@ -308,16 +334,15 @@ class PlaybookRepository:
if success:
playbook.success_count += 1
# 正向 EWMAalpha=0.1,正向結果權重較小(保守更新)
playbook.trust_score = 0.9 * playbook.trust_score + 0.1 * 1.0
else:
playbook.failure_count += 1
# 負向 EWMAalpha=0.2,失敗懲罰 2x快速衰退
playbook.trust_score = 0.8 * playbook.trust_score + 0.2 * 0.0
# 邊界保護
playbook.trust_score = max(0.0, min(1.0, playbook.trust_score))
playbook.last_used_at = now_taipei()
# 雙寫PG + Redis
await self.update(playbook)
if playbook.trust_score < 0.1:
@@ -338,70 +363,25 @@ class PlaybookRepository:
return True
except Exception as e:
logger.error(
"playbook_stats_update_failed",
playbook_id=playbook_id,
error=str(e),
)
logger.error("playbook_stats_update_failed", playbook_id=playbook_id, error=str(e))
return False
# === Index Management ===
async def _update_indexes(self, playbook: Playbook) -> None:
"""更新索引"""
async def find_by_source_incident(self, incident_id: str) -> list[Playbook]:
"""根據來源 Incident ID 找 Playbook從 PG 查詢)"""
try:
redis_client = get_redis()
# Alert names 索引
for alert_name in playbook.symptom_pattern.alert_names:
index_key = f"{PLAYBOOK_INDEX_ALERT_PREFIX}{alert_name}"
await redis_client.sadd(index_key, playbook.playbook_id)
await redis_client.expire(index_key, PLAYBOOK_TTL_SECONDS)
# Services 索引
for service in playbook.symptom_pattern.affected_services:
index_key = f"{PLAYBOOK_INDEX_SERVICE_PREFIX}{service}"
await redis_client.sadd(index_key, playbook.playbook_id)
await redis_client.expire(index_key, PLAYBOOK_TTL_SECONDS)
factory = get_session_factory()
async with factory() as session:
# PG JSONB contains 查詢
stmt = select(PlaybookRecord).where(
PlaybookRecord.source_incident_ids.contains([incident_id])
)
result = await session.execute(stmt)
records = result.scalars().all()
return [_orm_to_pydantic(r) for r in records]
except Exception as e:
logger.warning("playbook_index_update_failed", error=str(e))
# === Learning Service 信心度調整 (2026-03-30 Claude Code) ===
async def find_by_source_incident(
self,
incident_id: str,
) -> list[Playbook]:
"""
根據來源 Incident ID 找 Playbook
2026-03-30 Claude Code: Learning Service 信心度調整用
尋找 source_incident_ids 包含此 incident_id 的 Playbooks
"""
try:
redis_client = get_redis()
# 掃描所有 Playbook keys
pattern = f"{PLAYBOOK_KEY_PREFIX}PB-*"
results: list[Playbook] = []
async for key in redis_client.scan_iter(match=pattern, count=100):
data = await redis_client.get(key)
if data:
playbook = Playbook.from_redis_dict(json.loads(data))
if incident_id in playbook.source_incident_ids:
results.append(playbook)
return results
except Exception as e:
logger.error(
"playbook_find_by_incident_failed",
incident_id=incident_id,
error=str(e),
)
return []
logger.error("playbook_find_by_incident_failed", incident_id=incident_id, error=str(e))
# Fallback: Redis scan維持向下相容
return await self._redis_find_by_incident(incident_id)
async def adjust_confidence(
self,
@@ -410,9 +390,7 @@ class PlaybookRepository:
reason: str,
) -> Playbook | None:
"""
調整 Playbook 信心度
2026-03-30 Claude Code: Learning Service 信心度調整用
調整 Playbook 信心度(雙寫到 PG + Redis
邏輯:
- ai_confidence += delta (clamp 到 0.0~1.0)
@@ -425,25 +403,16 @@ class PlaybookRepository:
return None
old_confidence = playbook.ai_confidence
# 調整信心度 (clamp 到 0.0~1.0)
playbook.ai_confidence = max(0.0, min(1.0, playbook.ai_confidence + delta))
# 狀態自動轉換
old_status = playbook.status
# 高信心度自動升級
if (
playbook.ai_confidence >= 0.9
and playbook.status == PlaybookStatus.DRAFT
):
if playbook.ai_confidence >= 0.9 and playbook.status == PlaybookStatus.DRAFT:
playbook.status = PlaybookStatus.APPROVED
playbook.approved_by = "auto_learning"
playbook.approved_at = now_taipei()
note = f"\n[Auto-approved: confidence {playbook.ai_confidence:.2f}]"
playbook.notes = (playbook.notes or "") + note
# 低信心度 + 高失敗率 → 棄用
elif (
playbook.ai_confidence < 0.3
and playbook.total_executions >= 5
@@ -455,7 +424,6 @@ class PlaybookRepository:
note = f"\n[Auto-deprecated: conf={conf:.2f}, fail={fail:.0%}]"
playbook.notes = (playbook.notes or "") + note
# 儲存
updated = await self.update(playbook)
logger.info(
@@ -467,18 +435,190 @@ class PlaybookRepository:
new_status=playbook.status.value,
reason=reason,
)
return updated
except Exception as e:
logger.error(
"playbook_confidence_adjust_failed",
playbook_id=playbook_id,
delta=delta,
error=str(e),
)
logger.error("playbook_confidence_adjust_failed", playbook_id=playbook_id, delta=delta, error=str(e))
return None
# =========================================================================
# Startup Backfill — Redis → PG 遷移
# =========================================================================
async def backfill_redis_to_pg(self) -> int:
"""
啟動時將 Redis 中存在但 PG 中沒有的 Playbook 寫入 PG。
用途Phase 3.5 上線前後,確保舊 Redis 資料不丟失。
返回:補寫的 Playbook 數量。
"""
count = 0
try:
redis_client = get_redis()
pattern = f"{PLAYBOOK_KEY_PREFIX}PB-*"
async for key in redis_client.scan_iter(match=pattern, count=100):
data = await redis_client.get(key)
if not data:
continue
try:
playbook = Playbook.from_redis_dict(json.loads(data))
existing = await self._pg_get(playbook.playbook_id)
if existing is None:
await self._pg_upsert(playbook)
count += 1
logger.info("playbook_backfilled_to_pg", playbook_id=playbook.playbook_id)
except Exception as e:
logger.warning("playbook_backfill_item_failed", key=str(key), error=str(e))
except Exception as e:
logger.warning("playbook_backfill_failed", error=str(e))
logger.info("playbook_backfill_complete", count=count)
return count
# =========================================================================
# Private — PostgreSQL Helpers
# =========================================================================
async def _pg_upsert(self, playbook: Playbook) -> None:
"""Upsert Playbook 到 PostgreSQLINSERT ON CONFLICT UPDATE"""
try:
from sqlalchemy.dialects.postgresql import insert as pg_insert
factory = get_session_factory()
async with factory() as session:
stmt = pg_insert(PlaybookRecord).values(
playbook_id=playbook.playbook_id,
name=playbook.name,
description=playbook.description,
status=playbook.status.value,
source=playbook.source.value,
symptom_pattern=playbook.symptom_pattern.model_dump(),
repair_steps=[s.model_dump() for s in playbook.repair_steps],
estimated_duration_minutes=playbook.estimated_duration_minutes,
source_incident_ids=playbook.source_incident_ids,
ai_confidence=playbook.ai_confidence,
success_count=playbook.success_count,
failure_count=playbook.failure_count,
last_used_at=playbook.last_used_at,
trust_score=playbook.trust_score,
approved_by=playbook.approved_by,
approved_at=playbook.approved_at,
tags=playbook.tags,
notes=playbook.notes,
requires_approval_level=playbook.requires_approval_level,
stateful_targets=playbook.stateful_targets,
requires_pre_backup=playbook.requires_pre_backup,
created_at=playbook.created_at,
updated_at=playbook.updated_at,
).on_conflict_do_update(
index_elements=["playbook_id"],
set_={
"name": playbook.name,
"description": playbook.description,
"status": playbook.status.value,
"source": playbook.source.value,
"symptom_pattern": playbook.symptom_pattern.model_dump(),
"repair_steps": [s.model_dump() for s in playbook.repair_steps],
"estimated_duration_minutes": playbook.estimated_duration_minutes,
"source_incident_ids": playbook.source_incident_ids,
"ai_confidence": playbook.ai_confidence,
"success_count": playbook.success_count,
"failure_count": playbook.failure_count,
"last_used_at": playbook.last_used_at,
"trust_score": playbook.trust_score,
"approved_by": playbook.approved_by,
"approved_at": playbook.approved_at,
"tags": playbook.tags,
"notes": playbook.notes,
"requires_approval_level": playbook.requires_approval_level,
"stateful_targets": playbook.stateful_targets,
"requires_pre_backup": playbook.requires_pre_backup,
"updated_at": playbook.updated_at,
},
)
await session.execute(stmt)
await session.commit()
except Exception as e:
logger.error("playbook_pg_upsert_failed", playbook_id=playbook.playbook_id, error=str(e))
raise
async def _pg_get(self, playbook_id: str) -> Playbook | None:
"""從 PostgreSQL 載入 Playbook"""
try:
factory = get_session_factory()
async with factory() as session:
result = await session.get(PlaybookRecord, playbook_id)
if result is None:
return None
return _orm_to_pydantic(result)
except Exception as e:
logger.warning("playbook_pg_get_failed", playbook_id=playbook_id, error=str(e))
return None
# =========================================================================
# Private — Redis Cache Helpers
# =========================================================================
async def _redis_set(self, playbook: Playbook) -> None:
"""寫入 Redis warm cacheTTL 7天"""
try:
redis_client = get_redis()
key = f"{PLAYBOOK_KEY_PREFIX}{playbook.playbook_id}"
await redis_client.set(
key,
json.dumps(playbook.to_redis_dict(), ensure_ascii=False),
ex=PLAYBOOK_TTL_SECONDS,
)
except Exception as e:
# Redis 寫入失敗不 raisePG 已成功為主)
logger.warning("playbook_redis_set_failed", playbook_id=playbook.playbook_id, error=str(e))
async def _redis_get(self, playbook_id: str) -> Playbook | None:
"""從 Redis cache 讀取 Playbook"""
try:
redis_client = get_redis()
key = f"{PLAYBOOK_KEY_PREFIX}{playbook_id}"
data = await redis_client.get(key)
if data:
return Playbook.from_redis_dict(json.loads(data))
return None
except Exception as e:
logger.warning("playbook_redis_get_failed", playbook_id=playbook_id, error=str(e))
return None
async def _redis_find_by_incident(self, incident_id: str) -> list[Playbook]:
"""Redis fallback: 掃描所有 Playbook 找包含 incident_id 的"""
try:
redis_client = get_redis()
results: list[Playbook] = []
async for key in redis_client.scan_iter(match=f"{PLAYBOOK_KEY_PREFIX}PB-*", count=100):
data = await redis_client.get(key)
if data:
playbook = Playbook.from_redis_dict(json.loads(data))
if incident_id in playbook.source_incident_ids:
results.append(playbook)
return results
except Exception:
return []
async def _update_indexes(self, playbook: Playbook) -> None:
"""更新 Redis 索引(供快速 symptom 過濾)"""
try:
redis_client = get_redis()
for alert_name in playbook.symptom_pattern.alert_names:
index_key = f"{PLAYBOOK_INDEX_ALERT_PREFIX}{alert_name}"
await redis_client.sadd(index_key, playbook.playbook_id)
await redis_client.expire(index_key, PLAYBOOK_TTL_SECONDS)
for service in playbook.symptom_pattern.affected_services:
index_key = f"{PLAYBOOK_INDEX_SERVICE_PREFIX}{service}"
await redis_client.sadd(index_key, playbook.playbook_id)
await redis_client.expire(index_key, PLAYBOOK_TTL_SECONDS)
except Exception as e:
logger.warning("playbook_index_update_failed", error=str(e))
# =============================================================================
# Singleton