f525e657ca
- ADR-060: 全面基礎設施監控規劃 (Plan A/B/C/D/E) - ADR-061: Alert Operation Log Event Sourcing 架構 - LOGBOOK: 2026-04-08 里程碑記錄更新 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
121 lines
4.0 KiB
Markdown
121 lines
4.0 KiB
Markdown
# ADR-061: Alert Operation Log — Event Sourcing 操作溯源
|
||
|
||
**狀態**: 已實施 (2026-04-08)
|
||
**日期**: 2026-04-08
|
||
**提案者**: Claude Code
|
||
**審核者**: 統帥 (首席架構師)
|
||
|
||
---
|
||
|
||
## 背景
|
||
|
||
統帥指令：「所有操作都必須被記錄，寫入資料庫」「把之前所有的告警訊息，通通寫入資料庫，並且要有然後去記錄相關的操作」
|
||
|
||
現有問題：
|
||
- `audit_logs` 只記錄 K8s 操作，不記錄告警生命週期
|
||
- `approval_records` 記錄審批狀態，但不記錄操作時間軸
|
||
- 自動修復操作 (`auto_repair_executions`) 之前未寫入 DB
|
||
- 沒有統一的告警全生命週期視圖
|
||
|
||
## 決策
|
||
|
||
### 兩個新表
|
||
|
||
**1. `auto_repair_executions`** (Phase 10)
|
||
- 記錄每次自動修復執行的成功/失敗
|
||
- 欄位：incident_id, playbook_id, playbook_name, success, executed_steps, similarity_score, risk_level, execution_time_ms
|
||
|
||
**2. `alert_operation_log`** (Phase 11) — Event Sourcing
|
||
- 不可變 (Immutable) — 只 INSERT，不 UPDATE/DELETE
|
||
- 每個告警生命週期的每個事件都寫一筆
|
||
- 支援完整溯源：從告警進來到解決的所有步驟
|
||
|
||
### Event Types
|
||
|
||
```
|
||
ALERT_RECEIVED — Alertmanager/外部告警進來
|
||
TELEGRAM_SENT — 推送 Telegram 審核卡片
|
||
USER_ACTION — 使用者在 Telegram 按按鈕 (approve/reject/silence)
|
||
AUTO_REPAIR_TRIGGERED — 自動修復評估通過，準備執行
|
||
EXECUTION_STARTED — 開始執行 K8s/SSH/Docker 指令
|
||
EXECUTION_COMPLETED — 執行完成 (success/failure)
|
||
TELEGRAM_RESULT_SENT — 自動修復結果推送到 Telegram
|
||
RESOLVED — 告警解除
|
||
SILENCED — 靜默中
|
||
ESCALATED — 升級 (P3→P2 等)
|
||
CHANGE_APPLIED — 生產環境變更記錄
|
||
BACKUP_COMPLETED — 備份完成事件
|
||
BACKUP_FAILED — 備份失敗事件
|
||
DR_DRILL_COMPLETED — DR 演練完成
|
||
```
|
||
|
||
### 歷史數據回填
|
||
|
||
執行 `phase11b_backfill_alert_operation_log.sql` 回填：
|
||
- 14 筆 ALERT_RECEIVED (incidents 表)
|
||
- 265 筆 TELEGRAM_SENT (approval_records 表)
|
||
- 265 筆 USER_ACTION (approval_records 表)
|
||
- 110 筆 EXECUTION_COMPLETED (audit_logs 表)
|
||
- **總計 654 筆歷史記錄**
|
||
|
||
## 實施
|
||
|
||
### 寫入點
|
||
|
||
| 事件 | 觸發位置 |
|
||
|------|---------|
|
||
| ALERT_RECEIVED | `webhooks.py` alertmanager_webhook → `alert_operation_log` |
|
||
| TELEGRAM_SENT | `_push_to_telegram_background` 成功後 |
|
||
| USER_ACTION | `telegram.py` handle_callback approve/reject |
|
||
| AUTO_REPAIR_TRIGGERED | `_try_auto_repair_background` evaluate 後 |
|
||
| EXECUTION_COMPLETED | `_try_auto_repair_background` result 後 |
|
||
| TELEGRAM_RESULT_SENT | `_try_auto_repair_background` telegram 推送後 |
|
||
|
||
### 程式碼位置
|
||
|
||
```
|
||
apps/api/src/db/models.py — AlertOperationLog, AutoRepairExecution models
|
||
apps/api/src/repositories/
|
||
alert_operation_log_repository.py — append() / list_by_incident() / get_stats()
|
||
audit_log_repository.py — get_auto_repair_execution_repository()
|
||
apps/api/src/api/v1/webhooks.py — 寫入點整合
|
||
apps/api/src/api/v1/telegram.py — USER_ACTION 寫入
|
||
apps/api/migrations/
|
||
phase10_auto_repair_executions.sql
|
||
phase11_alert_operation_log.sql
|
||
phase11b_backfill_alert_operation_log.sql
|
||
```
|
||
|
||
## 查詢範例
|
||
|
||
```sql
|
||
-- 查詢某 incident 的完整操作時間軸
|
||
SELECT event_type, actor, action_detail, success, created_at
|
||
FROM alert_operation_log
|
||
WHERE incident_id = 'INC-20260408-XXXXXX'
|
||
ORDER BY created_at ASC;
|
||
|
||
-- 統計 24 小時自動修復成功率
|
||
SELECT success, COUNT(*) FROM auto_repair_executions
|
||
WHERE created_at > NOW() - INTERVAL '24 hours'
|
||
GROUP BY success;
|
||
|
||
-- 所有使用者操作記錄
|
||
SELECT actor, action_detail, context->>'username', created_at
|
||
FROM alert_operation_log
|
||
WHERE event_type = 'USER_ACTION'
|
||
ORDER BY created_at DESC;
|
||
```
|
||
|
||
## 結果
|
||
|
||
- 所有告警操作 100% 持久化到 DB
|
||
- 支援完整審計 (Audit Trail)
|
||
- 654 筆歷史記錄已回填
|
||
- 未來：可建立 AWOOOI Web 操作記錄頁面
|
||
|
||
## 相關 ADR
|
||
|
||
- ADR-030: Intelligent Auto-Remediation
|
||
- ADR-060: 全面基礎設施監控
|