feat(ai-ops): ADR-013 AIOps 自動修復閉環完整實作

架構(Exception → Incident → PlayBook → Heal → KM → Telegram):

新增元件:
- database/autoheal_models.py: Incident/Playbook/HealLog 三張表 + 7 條種子 PlayBook
- migrations/013_autoheal.sql: 建表 DDL + 種子資料(冪等 INSERT)
- services/auto_heal_service.py: 核心引擎 7 步閉環
  - _classify_error: 8 類錯誤自動分類 (DNS_FAIL/DB_UNREACHABLE/OOM/...)
  - _match_playbook: error_type + keyword + 冷卻 + max_retries 保護
  - _execute_playbook: DOCKER_RESTART/SSH_CMD/ALERT_ONLY/WAIT_RETRY
  - _sink_to_km: 修復知識寫入 ai_insights (auto_heal_playbook)
  - SSH 白名單:僅允許 docker restart / compose restart / docker start

修改元件:
- database/manager.py: _init_autoheal_tables() 啟動時建表+種子 PlayBook
- scheduler.py: 3 個核心任務植入 handle_exception
  (run_auto_import_task / run_icaim_analysis_task / run_weekly_strategy_task)
- requirements.txt: paramiko(SSH 跳板;不可用時降級 subprocess+CLI ssh)

安全設計: CMD 白名單 + cooldown + max_retries escalation + DB 冪等 migration

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
ogt
2026-04-19 16:03:49 +08:00
parent 7fbeaaf213
commit 77d3a1da48
8 changed files with 1050 additions and 3 deletions

View File

@@ -8,6 +8,7 @@ from .user_models import User, LoginHistory # noqa: F401 - 必須在 trend_mode
from .edm_models import PromoProduct # V-Fix: 確保 EDM 模型被註冊,以便自動建表
from .trend_models import TrendRecord, TrendKeyword, TrendAnalysis, WebSearchCache, TelegramUser # noqa: F401 - 趨勢資料表
from .ai_models import AIGenerationHistory, AIInsight, AIUsageTracking, AIPromptTemplate # AI 記憶體與洞察模型
from .autoheal_models import Incident, Playbook, HealLog # noqa: F401 - ADR-013 AIOps 自動修復表
# 🚩 導入優化後的日誌管理模組
from services.logger_manager import SystemLogger
@@ -60,6 +61,8 @@ class DatabaseManager:
)
self.Session = sessionmaker(bind=self.engine)
sys_log.info(f"[Database] ✅ 使用 PostgreSQL 資料庫 (連線池已優化)")
# ADR-013: 確保 AIOps 自動修復表存在並植入種子 PlayBook
self._init_autoheal_tables()
else:
# SQLite 模式 - 向後相容
if db_path is None:
@@ -111,7 +114,44 @@ class DatabaseManager:
sys_log.error(f"❌ 資料庫結構檢查失敗: {e}")
finally:
session.close()
def _init_autoheal_tables(self):
"""
ADR-013: 在 PostgreSQL 模式下,確保 AIOps 三張表存在並植入種子 PlayBook。
使用 Base.metadata.create_all 以 checkfirst=True 確保冪等執行。
"""
try:
# 建立表(已存在則略過)
from .autoheal_models import Incident, Playbook, HealLog, SEED_PLAYBOOKS
from sqlalchemy import inspect as sa_inspect
inspector = sa_inspect(self.engine)
existing_tables = inspector.get_table_names()
for model in [Incident, Playbook, HealLog]:
if model.__tablename__ not in existing_tables:
model.__table__.create(self.engine, checkfirst=True)
sys_log.info(f"[Database] ✅ 建立 AIOps 表: {model.__tablename__}")
# 植入種子 PlayBook首次
session = self.get_session()
try:
count = session.query(Playbook).count()
if count == 0:
for seed in SEED_PLAYBOOKS:
session.add(Playbook(**seed))
session.commit()
sys_log.info(f"[Database] ✅ 植入 {len(SEED_PLAYBOOKS)} 筆種子 PlayBook")
else:
sys_log.info(f"[Database] PlayBook 已有 {count} 筆,略過種子植入")
except Exception as e:
session.rollback()
sys_log.warning(f"[Database] 種子 PlayBook 植入失敗: {e}")
finally:
session.close()
except Exception as e:
sys_log.error(f"[Database] _init_autoheal_tables 失敗 (不影響主程序): {e}")
def get_session(self):
"""
提供外部調用的 Session 實例。