feat(Phase 3.5 + Phase 4): AI 學習成果持久化到 PostgreSQL — 修正「AI 失憶」架構缺陷
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
ADR-085: AI 學習成果不可存在 Cache 架構鐵律確立: - PostgreSQL = System of Record(AI 的永久記憶) - Redis = Warm Cache(加速讀取,TTL 到期從 PG 復原) 核心變更: 1. models.py: 新增 PlaybookRecord / DynamicBaselineRecord / LogClusterRecord ORM 2. base.py: ALTER TABLE playbooks 補加 trust_score / requires_approval_level 等欄位 3. playbook_repository.py: 完整雙寫實作(PG upsert + Redis cache) 4. dynamic_baseline_service.py: Holt-Winters 訓練結果寫入 PG,Redis 只作 24h warm cache 5. log_anomaly_detector.py: Drain3 cluster template 寫入 PG(UPSERT on cluster_id) 6. main.py: 啟動時執行 backfill_redis_to_pg()(Redis → PG 冪等補救) 修正的問題: - Playbook 7天 Redis TTL 到期 → AI 失去所有修復知識 - trust_score EWMA 隨 Redis TTL 歸零 → AI 重新回到初始信任度 0.3 - Holt-Winters 基線 24h TTL → AI 每天重新學習「正常」的定義 - Drain3 cluster 沒有持久化 → AI 把已知 log pattern 反覆當新 pattern Phase 4 新服務(requirements.txt 已加入 statsmodels + drain3 + numpy) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -186,6 +186,19 @@ async def init_db() -> None:
|
||||
""")
|
||||
)
|
||||
|
||||
# 2026-04-15 ogt + Claude Sonnet 4.6(亞太): Phase 3.5 Playbook PostgreSQL 持久化
|
||||
# ADR-085: AI 學習成果不可存 Cache — trust_score、EWMA 必須永久保存
|
||||
# playbooks 表已存在(15 筆舊資料),補加新欄位
|
||||
await conn.execute(
|
||||
text("""
|
||||
ALTER TABLE playbooks
|
||||
ADD COLUMN IF NOT EXISTS trust_score FLOAT NOT NULL DEFAULT 0.3,
|
||||
ADD COLUMN IF NOT EXISTS requires_approval_level VARCHAR(20) NOT NULL DEFAULT 'auto',
|
||||
ADD COLUMN IF NOT EXISTS stateful_targets JSONB NOT NULL DEFAULT '[]',
|
||||
ADD COLUMN IF NOT EXISTS requires_pre_backup BOOLEAN NOT NULL DEFAULT FALSE;
|
||||
""")
|
||||
)
|
||||
|
||||
|
||||
async def close_db() -> None:
|
||||
"""
|
||||
|
||||
@@ -835,6 +835,207 @@ class IncidentEvidence(Base):
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# PlaybookRecord — Phase 3.5 Playbook PostgreSQL 持久化 (System of Record)
|
||||
# ADR-085: AI 學習成果不可存在 Cache — Playbook 是 AI 的肌肉記憶
|
||||
# 2026-04-15 ogt + Claude Sonnet 4.6(亞太): Phase 3.5 初始建立
|
||||
#
|
||||
# 核心鐵律:
|
||||
# - PostgreSQL = System of Record(永久保存,AI 的長期記憶)
|
||||
# - Redis = Warm Cache(7天 TTL,加速讀取,DB 為 source of truth)
|
||||
# - trust_score, EWMA, 統計數據必須持久化 — 不能因 Redis TTL 消失
|
||||
# =============================================================================
|
||||
|
||||
class PlaybookRecord(Base):
|
||||
"""
|
||||
Playbook 修復劇本 PostgreSQL ORM
|
||||
|
||||
與 Pydantic Playbook 模型對應。
|
||||
Redis 為 warm cache(7d TTL),PostgreSQL 為 source of truth。
|
||||
|
||||
設計原則:
|
||||
- AI 的學習成果(trust_score、success_count、failure_count)永久保存
|
||||
- EWMA 信任度在 Redis TTL 後不會重置,Pod 重啟後 AI 記憶不失
|
||||
- 雙寫:create/update 先寫 PG,再更新 Redis cache
|
||||
- 讀取:Redis-first(cache hit),miss 時從 PG 載入並回填 Redis
|
||||
"""
|
||||
__tablename__ = "playbooks"
|
||||
|
||||
# Primary Key
|
||||
playbook_id: Mapped[str] = mapped_column(
|
||||
String(36), primary_key=True,
|
||||
comment="Playbook 唯一識別碼 (PB-YYYYMMDD-XXXXXX)",
|
||||
)
|
||||
|
||||
# Core Fields
|
||||
name: Mapped[str] = mapped_column(String(256), nullable=False)
|
||||
description: Mapped[str] = mapped_column(Text, default="", nullable=False)
|
||||
status: Mapped[str] = mapped_column(String(20), default="draft", nullable=False)
|
||||
source: Mapped[str] = mapped_column(String(20), default="extracted", nullable=False)
|
||||
|
||||
# Complex structures (JSONB)
|
||||
symptom_pattern: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False)
|
||||
repair_steps: Mapped[list[dict[str, Any]]] = mapped_column(JSON, default=list, nullable=False)
|
||||
|
||||
# Timing
|
||||
estimated_duration_minutes: Mapped[int] = mapped_column(Integer, default=5, nullable=False)
|
||||
|
||||
# Source tracing
|
||||
source_incident_ids: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
|
||||
ai_confidence: Mapped[float] = mapped_column(default=0.0, nullable=False)
|
||||
|
||||
# Stats — MUST be in PG (AI learning artifacts, cannot expire)
|
||||
success_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
failure_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
last_used_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
# EWMA trust score — ADR-083 Phase 3, 絕對不能用 Redis TTL 管理
|
||||
# trust_score 是 AI 累積學習的結晶,TTL 到期就歸零 = AI 記憶全部消失
|
||||
trust_score: Mapped[float] = mapped_column(default=0.3, nullable=False,
|
||||
comment="EWMA 動態信任度 (Phase 3)。成功 α=0.1,失敗 α=0.2(2x 衰減)。< 0.1 → 封存")
|
||||
|
||||
# Approval metadata
|
||||
approved_by: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
||||
approved_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
tags: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
|
||||
notes: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
|
||||
# Sprint 5.1 護欄欄位 (2026-04-08)
|
||||
requires_approval_level: Mapped[str] = mapped_column(
|
||||
String(20), default="auto", nullable=False,
|
||||
comment="auto=直接執行, standard=1票, critical=2票MultiSig",
|
||||
)
|
||||
stateful_targets: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
|
||||
requires_pre_backup: Mapped[bool] = mapped_column(default=False, nullable=False)
|
||||
|
||||
# Timestamps
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=taipei_now, nullable=False)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=taipei_now,
|
||||
onupdate=taipei_now, nullable=False)
|
||||
|
||||
__table_args__ = (
|
||||
Index("ix_playbook_status", "status"),
|
||||
Index("ix_playbook_trust_score", "trust_score"),
|
||||
Index("ix_playbook_created_at", "created_at"),
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# DynamicBaselineRecord — Phase 4 Holt-Winters 訓練基線持久化
|
||||
# ADR-084: 動態基線不能只存 Redis — AI 每天重學「正常」不是在學習
|
||||
# 2026-04-15 ogt + Claude Sonnet 4.6(亞太): Phase 4 初始建立
|
||||
#
|
||||
# 核心鐵律:
|
||||
# - 訓練好的 Holt-Winters 模型必須在 PG 長期保存
|
||||
# - Redis 為 24h warm cache(加速 is_anomaly() 讀取)
|
||||
# - 基線消失 = AI 對「正常」的認識消失 = 每天從頭學習 = 不是 AI
|
||||
# =============================================================================
|
||||
|
||||
class DynamicBaselineRecord(Base):
|
||||
"""
|
||||
動態基線訓練結果 PostgreSQL ORM
|
||||
|
||||
Holt-Winters 訓練完成後:
|
||||
1. 先寫入 PG(永久保存)
|
||||
2. 再寫入 Redis(24h warm cache,加速讀取)
|
||||
|
||||
Redis key: baseline:{metric_name}
|
||||
PG: 此表,metric_name 為主鍵,最新一筆 = 有效基線
|
||||
"""
|
||||
__tablename__ = "dynamic_baselines"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=generate_uuid)
|
||||
|
||||
# 基線識別
|
||||
metric_name: Mapped[str] = mapped_column(
|
||||
String(200), nullable=False, index=True,
|
||||
comment="基線識別名 (e.g. cpu_usage_node_mon)",
|
||||
)
|
||||
|
||||
# 訓練結果(Holt-Winters 統計)
|
||||
mean: Mapped[float] = mapped_column(nullable=False, comment="擬合值均值")
|
||||
std: Mapped[float] = mapped_column(nullable=False, comment="殘差標準差")
|
||||
|
||||
# 24h 季節性因子(JSON 陣列,長度 24)
|
||||
seasonal_factors: Mapped[list[float]] = mapped_column(
|
||||
JSON, default=list, nullable=False,
|
||||
comment="24h 週期季節性因子(乘法形式,均值 ≈ 1.0)",
|
||||
)
|
||||
|
||||
# 訓練元資料
|
||||
datapoint_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
promql: Mapped[str] = mapped_column(Text, default="", nullable=False,
|
||||
comment="訓練使用的 PromQL 查詢")
|
||||
lookback_hours: Mapped[int] = mapped_column(Integer, default=336, nullable=False)
|
||||
|
||||
# Timestamps
|
||||
trained_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=taipei_now, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=taipei_now, nullable=False)
|
||||
|
||||
__table_args__ = (
|
||||
Index("ix_dynamic_baseline_metric", "metric_name"),
|
||||
Index("ix_dynamic_baseline_trained_at", "trained_at"),
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# LogClusterRecord — Phase 4 Drain3 學習到的 Log Pattern 持久化
|
||||
# ADR-084: Drain3 模板不能只存 Redis — 每次重啟 AI 把已知 pattern 當新 pattern
|
||||
# 2026-04-15 ogt + Claude Sonnet 4.6(亞太): Phase 4 初始建立
|
||||
#
|
||||
# 核心鐵律:
|
||||
# - Drain3 學到的 log cluster template 必須在 PG 長期保存
|
||||
# - 新 cluster 事件列表 (log_anomaly:new) 才存 Redis(短期工作記憶)
|
||||
# - 基礎知識庫(已學到的 pattern)必須在 PG
|
||||
# =============================================================================
|
||||
|
||||
class LogClusterRecord(Base):
|
||||
"""
|
||||
Drain3 Log Cluster Template 持久化
|
||||
|
||||
每個新 pattern 首次偵測到時:
|
||||
1. 寫入 PG(永久保存,AI 的 log 語意理解)
|
||||
2. 推送到 Redis list log_anomaly:new(短期工作記憶)
|
||||
|
||||
Re-detect 相同 template 時只更新 last_seen_at + size,不重複寫入 PG。
|
||||
"""
|
||||
__tablename__ = "log_clusters"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=generate_uuid)
|
||||
|
||||
# Cluster 識別(MD5[:8] of template)
|
||||
cluster_id: Mapped[str] = mapped_column(
|
||||
String(16), nullable=False, unique=True, index=True,
|
||||
comment="模板 MD5[:8].upper(),穩定 ID",
|
||||
)
|
||||
|
||||
# Drain3 模板
|
||||
template: Mapped[str] = mapped_column(
|
||||
Text, nullable=False,
|
||||
comment="Drain3 萃取的 log 模板 (e.g. 'ERROR <*> connection failed to <*>')",
|
||||
)
|
||||
|
||||
# 統計
|
||||
size: Mapped[int] = mapped_column(Integer, default=1, nullable=False,
|
||||
comment="命中次數(第一次 = 1)")
|
||||
source: Mapped[str] = mapped_column(String(50), default="k8s_pod", nullable=False,
|
||||
comment="k8s_pod | host_syslog | app_log")
|
||||
|
||||
# 樣本日誌(保留首次觸發的原始行,供事後分析)
|
||||
sample_log: Mapped[str | None] = mapped_column(Text, nullable=True,
|
||||
comment="首次觸發的原始 log 行(前 500 字元)")
|
||||
|
||||
# Timestamps
|
||||
first_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=taipei_now, nullable=False)
|
||||
last_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=taipei_now,
|
||||
onupdate=taipei_now, nullable=False)
|
||||
|
||||
__table_args__ = (
|
||||
Index("ix_log_cluster_first_seen", "first_seen_at"),
|
||||
Index("ix_log_cluster_source", "source"),
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# AgentSession — Phase 2 多 Agent 辯證 Audit Trail
|
||||
# =============================================================================
|
||||
|
||||
Reference in New Issue
Block a user