fix(aiops-p1): Playbook 學習閉環 5斷點全修 + DB Migration(ADR-092 B4)
Some checks failed
run-migration / migrate (push) Failing after 14s
CD Pipeline / build-and-deploy (push) Failing after 2m7s

【P0.4 補丁】pre_decision_investigator Prometheus query 欄位缺失
- _build_tool_params() 補 "query" 欄位(prometheus_query tool 必要參數)
- 新增 _build_prometheus_query() — 依告警類型生成 PromQL(CPU/Memory/Crash/Disk/HTTP/Pod/fallback)
- 修復後 D3_METRICS 感官維度實際取得資料(原本 100% 回 missing_query_parameter)

【P1 Playbook 學習閉環 B1-B5 全修】
- B2 db/models.py: ApprovalRecord 新增 matched_playbook_id 欄位 + ix_approval_matched_playbook index
- B2 db/models.py: TimelineEvent 新增 incident_id 欄位(MCP 稽核用)+ index
- B3 approval_db.py: record→ApprovalRequest 補回 incident_id + matched_playbook_id
- B4 approval_repository.py: 同 B3(兩個轉換函式必須同步)
- B5 approval_db.py: approval_request_to_record_data 補 matched_playbook_id → DB 才能存值

【P1.5 KM 寫入】approval_execution.py: fire-and-forget → await wait_for(30s)
- 根因:asyncio.create_task 在 Pod recycle 時被殺,KM 寫入靜默遺失
- 修復:await asyncio.wait_for(..., timeout=30.0) + TimeoutError log

【Migration 文件】adr092_p1_learning_chain_fix.sql
- ALTER TABLE approval_records ADD COLUMN matched_playbook_id VARCHAR(36)
- ALTER TABLE timeline_events ADD COLUMN incident_id VARCHAR(64)
- 執行:psql $DATABASE_URL -f apps/api/migrations/adr092_p1_learning_chain_fix.sql

【附帶 Agent 改動】
- decision_manager: Phase 2 YAML NO_ACTION 優先門(主機層/外部服務跳過 Agent Debate)
- alert_rules.yaml: Sentry/ClickHouse + HostDiskUsageHigh/Critical 新規則
- solver_agent: action_title 語意合成兜底(取代靜默丟棄)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Your Name
2026-04-24 15:41:35 +08:00
parent 7f4088bcd0
commit 04ff22563e
11 changed files with 321 additions and 20 deletions

View File

@@ -135,6 +135,8 @@ rules:
- HostUnusualDiskWriteRate
- HostDiskWillFillIn24Hours
- HostOutOfDiskSpace
- HostDiskUsageHigh
- HostDiskUsageCritical
# 網路相關
- HostUnusualNetworkThroughputIn
- HostUnusualNetworkThroughputOut
@@ -764,6 +766,36 @@ rules:
command: "curl -sv {instance} --max-time 10 2>&1 | grep -E '(HTTP|Connected|Failed)'"
reasoning: "[規則匹配] 外部網站下線屬外部依賴,通知統帥後等待服務恢復,必要時切換備援路徑。"
# 2026-04-24 ogt + Claude Sonnet 4.6: Sentry / ClickHouse 監控告警 — 外部服務,禁止 kubectl 操作
- id: sentry_clickhouse_alert
priority: 60
description: Sentry 或 ClickHouse 監控告警(外部服務,不是 K8s workload
match:
alertname:
- SentryClickHouseMemoryPressure
- SentryClickHouseCpuHigh
- SentryClickHouseDiskUsageHigh
- ClickHouseMemoryHigh
- ClickHouseMemoryPressure
- ClickHouseCpuHigh
- ClickHouseReplicationLag
- ClickHouseQuerySlow
- SentryWorkerQueueHigh
- SentryKafkaLag
- SentryBacklogHigh
response:
action_title: "⚠️ Sentry/ClickHouse 告警 — 需 SSH 人工排查"
description: "⚠️ Sentry/ClickHouse 屬外部監控服務,無法透過 kubectl 自動修復。請 SSH 登入服務主機排查根因clickhouse-client / docker stats / journalctl -xe。若記憶體壓力持續考慮調整 ClickHouse max_memory_usage 設定或清理舊資料。"
suggested_action: NO_ACTION
kubectl_command: ""
estimated_downtime: "N/A"
risk: high
responsibility: INFRA
responsibility_reasoning: "Sentry/ClickHouse 基礎設施由 INFRA 團隊管理"
secondary_teams: []
optimization: []
reasoning: "[規則匹配] Sentry/ClickHouse 非 K8s 服務kubectl 操作無效。需 SSH 進入服務主機,確認記憶體/CPU/磁碟狀況後手動介入。"
# ── 通用兜底 ────────────────────────────────────────────────
- id: generic_fallback