feat(Phase 3.5 + Phase 4): AI 學習成果持久化到 PostgreSQL — 修正「AI 失憶」架構缺陷
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled

ADR-085: AI 學習成果不可存在 Cache

架構鐵律確立:
- PostgreSQL = System of Record(AI 的永久記憶)
- Redis = Warm Cache(加速讀取,TTL 到期從 PG 復原)

核心變更:
1. models.py: 新增 PlaybookRecord / DynamicBaselineRecord / LogClusterRecord ORM
2. base.py: ALTER TABLE playbooks 補加 trust_score / requires_approval_level 等欄位
3. playbook_repository.py: 完整雙寫實作(PG upsert + Redis cache)
4. dynamic_baseline_service.py: Holt-Winters 訓練結果寫入 PG,Redis 只作 24h warm cache
5. log_anomaly_detector.py: Drain3 cluster template 寫入 PG(UPSERT on cluster_id)
6. main.py: 啟動時執行 backfill_redis_to_pg()(Redis → PG 冪等補救)

修正的問題:
- Playbook 7天 Redis TTL 到期 → AI 失去所有修復知識
- trust_score EWMA 隨 Redis TTL 歸零 → AI 重新回到初始信任度 0.3
- Holt-Winters 基線 24h TTL → AI 每天重新學習「正常」的定義
- Drain3 cluster 沒有持久化 → AI 把已知 log pattern 反覆當新 pattern

Phase 4 新服務(requirements.txt 已加入 statsmodels + drain3 + numpy)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
OG T
2026-04-15 15:33:52 +08:00
parent 9126c594a4
commit bf45b80bd2
8 changed files with 1490 additions and 238 deletions

View File

@@ -186,6 +186,19 @@ async def init_db() -> None:
""")
)
# 2026-04-15 ogt + Claude Sonnet 4.6(亞太): Phase 3.5 Playbook PostgreSQL 持久化
# ADR-085: AI 學習成果不可存 Cache — trust_score、EWMA 必須永久保存
# playbooks 表已存在15 筆舊資料),補加新欄位
await conn.execute(
text("""
ALTER TABLE playbooks
ADD COLUMN IF NOT EXISTS trust_score FLOAT NOT NULL DEFAULT 0.3,
ADD COLUMN IF NOT EXISTS requires_approval_level VARCHAR(20) NOT NULL DEFAULT 'auto',
ADD COLUMN IF NOT EXISTS stateful_targets JSONB NOT NULL DEFAULT '[]',
ADD COLUMN IF NOT EXISTS requires_pre_backup BOOLEAN NOT NULL DEFAULT FALSE;
""")
)
async def close_db() -> None:
"""

View File

@@ -835,6 +835,207 @@ class IncidentEvidence(Base):
)
# =============================================================================
# PlaybookRecord — Phase 3.5 Playbook PostgreSQL 持久化 (System of Record)
# ADR-085: AI 學習成果不可存在 Cache — Playbook 是 AI 的肌肉記憶
# 2026-04-15 ogt + Claude Sonnet 4.6(亞太): Phase 3.5 初始建立
#
# 核心鐵律:
# - PostgreSQL = System of Record永久保存AI 的長期記憶)
# - Redis = Warm Cache7天 TTL加速讀取DB 為 source of truth
# - trust_score, EWMA, 統計數據必須持久化 — 不能因 Redis TTL 消失
# =============================================================================
class PlaybookRecord(Base):
"""
Playbook 修復劇本 PostgreSQL ORM
與 Pydantic Playbook 模型對應。
Redis 為 warm cache7d TTLPostgreSQL 為 source of truth。
設計原則:
- AI 的學習成果trust_score、success_count、failure_count永久保存
- EWMA 信任度在 Redis TTL 後不會重置Pod 重啟後 AI 記憶不失
- 雙寫create/update 先寫 PG再更新 Redis cache
- 讀取Redis-firstcache hitmiss 時從 PG 載入並回填 Redis
"""
__tablename__ = "playbooks"
# Primary Key
playbook_id: Mapped[str] = mapped_column(
String(36), primary_key=True,
comment="Playbook 唯一識別碼 (PB-YYYYMMDD-XXXXXX)",
)
# Core Fields
name: Mapped[str] = mapped_column(String(256), nullable=False)
description: Mapped[str] = mapped_column(Text, default="", nullable=False)
status: Mapped[str] = mapped_column(String(20), default="draft", nullable=False)
source: Mapped[str] = mapped_column(String(20), default="extracted", nullable=False)
# Complex structures (JSONB)
symptom_pattern: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False)
repair_steps: Mapped[list[dict[str, Any]]] = mapped_column(JSON, default=list, nullable=False)
# Timing
estimated_duration_minutes: Mapped[int] = mapped_column(Integer, default=5, nullable=False)
# Source tracing
source_incident_ids: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
ai_confidence: Mapped[float] = mapped_column(default=0.0, nullable=False)
# Stats — MUST be in PG (AI learning artifacts, cannot expire)
success_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
failure_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
last_used_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
# EWMA trust score — ADR-083 Phase 3, 絕對不能用 Redis TTL 管理
# trust_score 是 AI 累積學習的結晶TTL 到期就歸零 = AI 記憶全部消失
trust_score: Mapped[float] = mapped_column(default=0.3, nullable=False,
comment="EWMA 動態信任度 (Phase 3)。成功 α=0.1,失敗 α=0.22x 衰減)。< 0.1 → 封存")
# Approval metadata
approved_by: Mapped[str | None] = mapped_column(String(100), nullable=True)
approved_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
tags: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
notes: Mapped[str | None] = mapped_column(Text, nullable=True)
# Sprint 5.1 護欄欄位 (2026-04-08)
requires_approval_level: Mapped[str] = mapped_column(
String(20), default="auto", nullable=False,
comment="auto=直接執行, standard=1票, critical=2票MultiSig",
)
stateful_targets: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
requires_pre_backup: Mapped[bool] = mapped_column(default=False, nullable=False)
# Timestamps
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=taipei_now, nullable=False)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=taipei_now,
onupdate=taipei_now, nullable=False)
__table_args__ = (
Index("ix_playbook_status", "status"),
Index("ix_playbook_trust_score", "trust_score"),
Index("ix_playbook_created_at", "created_at"),
)
# =============================================================================
# DynamicBaselineRecord — Phase 4 Holt-Winters 訓練基線持久化
# ADR-084: 動態基線不能只存 Redis — AI 每天重學「正常」不是在學習
# 2026-04-15 ogt + Claude Sonnet 4.6(亞太): Phase 4 初始建立
#
# 核心鐵律:
# - 訓練好的 Holt-Winters 模型必須在 PG 長期保存
# - Redis 為 24h warm cache加速 is_anomaly() 讀取)
# - 基線消失 = AI 對「正常」的認識消失 = 每天從頭學習 = 不是 AI
# =============================================================================
class DynamicBaselineRecord(Base):
"""
動態基線訓練結果 PostgreSQL ORM
Holt-Winters 訓練完成後:
1. 先寫入 PG永久保存
2. 再寫入 Redis24h warm cache加速讀取
Redis key: baseline:{metric_name}
PG: 此表metric_name 為主鍵,最新一筆 = 有效基線
"""
__tablename__ = "dynamic_baselines"
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=generate_uuid)
# 基線識別
metric_name: Mapped[str] = mapped_column(
String(200), nullable=False, index=True,
comment="基線識別名 (e.g. cpu_usage_node_mon)",
)
# 訓練結果Holt-Winters 統計)
mean: Mapped[float] = mapped_column(nullable=False, comment="擬合值均值")
std: Mapped[float] = mapped_column(nullable=False, comment="殘差標準差")
# 24h 季節性因子JSON 陣列,長度 24
seasonal_factors: Mapped[list[float]] = mapped_column(
JSON, default=list, nullable=False,
comment="24h 週期季節性因子(乘法形式,均值 ≈ 1.0",
)
# 訓練元資料
datapoint_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
promql: Mapped[str] = mapped_column(Text, default="", nullable=False,
comment="訓練使用的 PromQL 查詢")
lookback_hours: Mapped[int] = mapped_column(Integer, default=336, nullable=False)
# Timestamps
trained_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=taipei_now, nullable=False)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=taipei_now, nullable=False)
__table_args__ = (
Index("ix_dynamic_baseline_metric", "metric_name"),
Index("ix_dynamic_baseline_trained_at", "trained_at"),
)
# =============================================================================
# LogClusterRecord — Phase 4 Drain3 學習到的 Log Pattern 持久化
# ADR-084: Drain3 模板不能只存 Redis — 每次重啟 AI 把已知 pattern 當新 pattern
# 2026-04-15 ogt + Claude Sonnet 4.6(亞太): Phase 4 初始建立
#
# 核心鐵律:
# - Drain3 學到的 log cluster template 必須在 PG 長期保存
# - 新 cluster 事件列表 (log_anomaly:new) 才存 Redis短期工作記憶
# - 基礎知識庫(已學到的 pattern必須在 PG
# =============================================================================
class LogClusterRecord(Base):
"""
Drain3 Log Cluster Template 持久化
每個新 pattern 首次偵測到時:
1. 寫入 PG永久保存AI 的 log 語意理解)
2. 推送到 Redis list log_anomaly:new短期工作記憶
Re-detect 相同 template 時只更新 last_seen_at + size不重複寫入 PG。
"""
__tablename__ = "log_clusters"
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=generate_uuid)
# Cluster 識別MD5[:8] of template
cluster_id: Mapped[str] = mapped_column(
String(16), nullable=False, unique=True, index=True,
comment="模板 MD5[:8].upper(),穩定 ID",
)
# Drain3 模板
template: Mapped[str] = mapped_column(
Text, nullable=False,
comment="Drain3 萃取的 log 模板 (e.g. 'ERROR <*> connection failed to <*>')",
)
# 統計
size: Mapped[int] = mapped_column(Integer, default=1, nullable=False,
comment="命中次數(第一次 = 1")
source: Mapped[str] = mapped_column(String(50), default="k8s_pod", nullable=False,
comment="k8s_pod | host_syslog | app_log")
# 樣本日誌(保留首次觸發的原始行,供事後分析)
sample_log: Mapped[str | None] = mapped_column(Text, nullable=True,
comment="首次觸發的原始 log 行(前 500 字元)")
# Timestamps
first_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=taipei_now, nullable=False)
last_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=taipei_now,
onupdate=taipei_now, nullable=False)
__table_args__ = (
Index("ix_log_cluster_first_seen", "first_seen_at"),
Index("ix_log_cluster_source", "source"),
)
# =============================================================================
# AgentSession — Phase 2 多 Agent 辯證 Audit Trail
# =============================================================================