fix(aiops-p1): Playbook 學習閉環 5斷點全修 + DB Migration(ADR-092 B4)
【P0.4 補丁】pre_decision_investigator Prometheus query 欄位缺失 - _build_tool_params() 補 "query" 欄位(prometheus_query tool 必要參數) - 新增 _build_prometheus_query() — 依告警類型生成 PromQL(CPU/Memory/Crash/Disk/HTTP/Pod/fallback) - 修復後 D3_METRICS 感官維度實際取得資料(原本 100% 回 missing_query_parameter) 【P1 Playbook 學習閉環 B1-B5 全修】 - B2 db/models.py: ApprovalRecord 新增 matched_playbook_id 欄位 + ix_approval_matched_playbook index - B2 db/models.py: TimelineEvent 新增 incident_id 欄位(MCP 稽核用)+ index - B3 approval_db.py: record→ApprovalRequest 補回 incident_id + matched_playbook_id - B4 approval_repository.py: 同 B3(兩個轉換函式必須同步) - B5 approval_db.py: approval_request_to_record_data 補 matched_playbook_id → DB 才能存值 【P1.5 KM 寫入】approval_execution.py: fire-and-forget → await wait_for(30s) - 根因:asyncio.create_task 在 Pod recycle 時被殺,KM 寫入靜默遺失 - 修復:await asyncio.wait_for(..., timeout=30.0) + TimeoutError log 【Migration 文件】adr092_p1_learning_chain_fix.sql - ALTER TABLE approval_records ADD COLUMN matched_playbook_id VARCHAR(36) - ALTER TABLE timeline_events ADD COLUMN incident_id VARCHAR(64) - 執行:psql $DATABASE_URL -f apps/api/migrations/adr092_p1_learning_chain_fix.sql 【附帶 Agent 改動】 - decision_manager: Phase 2 YAML NO_ACTION 優先門(主機層/外部服務跳過 Agent Debate) - alert_rules.yaml: Sentry/ClickHouse + HostDiskUsageHigh/Critical 新規則 - solver_agent: action_title 語意合成兜底(取代靜默丟棄) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -135,6 +135,8 @@ rules:
|
||||
- HostUnusualDiskWriteRate
|
||||
- HostDiskWillFillIn24Hours
|
||||
- HostOutOfDiskSpace
|
||||
- HostDiskUsageHigh
|
||||
- HostDiskUsageCritical
|
||||
# 網路相關
|
||||
- HostUnusualNetworkThroughputIn
|
||||
- HostUnusualNetworkThroughputOut
|
||||
@@ -764,6 +766,36 @@ rules:
|
||||
command: "curl -sv {instance} --max-time 10 2>&1 | grep -E '(HTTP|Connected|Failed)'"
|
||||
reasoning: "[規則匹配] 外部網站下線屬外部依賴,通知統帥後等待服務恢復,必要時切換備援路徑。"
|
||||
|
||||
# 2026-04-24 ogt + Claude Sonnet 4.6: Sentry / ClickHouse 監控告警 — 外部服務,禁止 kubectl 操作
|
||||
- id: sentry_clickhouse_alert
|
||||
priority: 60
|
||||
description: Sentry 或 ClickHouse 監控告警(外部服務,不是 K8s workload)
|
||||
match:
|
||||
alertname:
|
||||
- SentryClickHouseMemoryPressure
|
||||
- SentryClickHouseCpuHigh
|
||||
- SentryClickHouseDiskUsageHigh
|
||||
- ClickHouseMemoryHigh
|
||||
- ClickHouseMemoryPressure
|
||||
- ClickHouseCpuHigh
|
||||
- ClickHouseReplicationLag
|
||||
- ClickHouseQuerySlow
|
||||
- SentryWorkerQueueHigh
|
||||
- SentryKafkaLag
|
||||
- SentryBacklogHigh
|
||||
response:
|
||||
action_title: "⚠️ Sentry/ClickHouse 告警 — 需 SSH 人工排查"
|
||||
description: "⚠️ Sentry/ClickHouse 屬外部監控服務,無法透過 kubectl 自動修復。請 SSH 登入服務主機排查根因:clickhouse-client / docker stats / journalctl -xe。若記憶體壓力持續,考慮調整 ClickHouse max_memory_usage 設定或清理舊資料。"
|
||||
suggested_action: NO_ACTION
|
||||
kubectl_command: ""
|
||||
estimated_downtime: "N/A"
|
||||
risk: high
|
||||
responsibility: INFRA
|
||||
responsibility_reasoning: "Sentry/ClickHouse 基礎設施由 INFRA 團隊管理"
|
||||
secondary_teams: []
|
||||
optimization: []
|
||||
reasoning: "[規則匹配] Sentry/ClickHouse 非 K8s 服務,kubectl 操作無效。需 SSH 進入服務主機,確認記憶體/CPU/磁碟狀況後手動介入。"
|
||||
|
||||
# ── 通用兜底 ────────────────────────────────────────────────
|
||||
|
||||
- id: generic_fallback
|
||||
|
||||
Reference in New Issue
Block a user