309a019cc3
更新: - ADR-027: 新增緊急事故修復章節 - LOGBOOK: 記錄 2026-03-26 事故時間線 - Skill 02 v1.6: 新增 Telegram 去重機制章節 根因: Phase 6.5 修改 + INC- 前綴重複 修復: Redis 去重 (10 分鐘) + 前綴檢查 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
293 lines
10 KiB
Markdown
293 lines
10 KiB
Markdown
# ADR-027: Incident-Approval 同步架構
|
||
|
||
**狀態**: 實作中 (Phase 1-2 ✅ 完成)
|
||
**日期**: 2026-03-26 (台北時區)
|
||
**決策者**: 統帥
|
||
**觸發**: 活躍事件顯示 0 + Telegram 告警鏈異常 (2026-03-26 首席架構師審查)
|
||
|
||
## 問題陳述
|
||
|
||
```
|
||
事故時間線：
|
||
├── 2026-03-26: 活躍事件顯示 0 件，但 Telegram 有新告警
|
||
├── 首席架構師審查: 發現 Incident-Approval 狀態同步機制缺失
|
||
│
|
||
├── CRITICAL-001: Incident + Approval 建立不是原子事務
|
||
│ └─ 後果: 孤兒 Incident、狀態不一致
|
||
│
|
||
├── CRITICAL-002: 雙層寫入非原子 (Redis + PostgreSQL)
|
||
│ └─ 後果: 資料遺失、狀態分歧
|
||
│
|
||
├── HIGH-001: Approval 狀態變更未同步 Incident
|
||
│ └─ 後果: 前端 get_active 查詢結果錯誤
|
||
│
|
||
├── HIGH-002: Redis TTL 過期導致資料遺失
|
||
│ └─ 後果: 長時間未處理事件消失
|
||
│
|
||
└── HIGH-003: 前端狀態一致性問題
|
||
└─ 後果: 簽核後內容立即消失
|
||
```
|
||
|
||
**核心問題**:
|
||
1. Incident 和 Approval 沒有事務保證
|
||
2. Redis (Working Memory) 和 PostgreSQL (Episodic Memory) 同步機制不完整
|
||
3. 狀態變更沒有雙向傳播
|
||
|
||
---
|
||
|
||
## 決策：UnitOfWork + Saga Pattern
|
||
|
||
### 為什麼選擇這個方案？
|
||
|
||
| 方案 | 優點 | 缺點 | 選擇 |
|
||
|------|------|------|------|
|
||
| 兩階段提交 (2PC) | 強一致性 | Redis 不支援、效能差 | ❌ |
|
||
| **UnitOfWork + Saga** | PostgreSQL 事務 + Redis 補償 | 需處理補償邏輯 | ✅ |
|
||
| 事件溯源 | 完整審計軌跡 | 改動太大 | ❌ |
|
||
| 最終一致性 | 簡單 | 不適合關鍵業務 | ❌ |
|
||
|
||
### 架構設計
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ Incident-Approval 同步架構 (ADR-027) │
|
||
├─────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ Alertmanager │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ IncidentApprovalService.create_with_approval() │
|
||
│ │ │
|
||
│ ├─────────────────────────────────────────┐ │
|
||
│ ▼ ▼ │
|
||
│ UnitOfWork (PostgreSQL) Redis Write │
|
||
│ │ │ │
|
||
│ ├─ incident = Incident.create() │ │
|
||
│ ├─ approval = Approval.create() │ │
|
||
│ │ │ │
|
||
│ ├─────── 事務 commit ─────────────────────┤ │
|
||
│ │ │ │
|
||
│ │ 成功 ────────────────────────────────→ 完成 │
|
||
│ │ │ │
|
||
│ │ 失敗 ────────────────────────────────→ Saga 補償 │
|
||
│ │ (rollback PostgreSQL, │
|
||
│ │ delete Redis keys) │
|
||
│ ▼ │
|
||
│ 同步 Hook │
|
||
│ │ │
|
||
│ └─ on_approval_status_change() │
|
||
│ ├─ update Incident.status │
|
||
│ ├─ refresh Redis TTL │
|
||
│ └─ emit SSE event │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## 實作細節
|
||
|
||
### 1. UnitOfWork 模式 (`src/core/unit_of_work.py`)
|
||
|
||
```python
|
||
class UnitOfWork:
|
||
"""PostgreSQL 事務管理器"""
|
||
|
||
def __init__(self, session_factory):
|
||
self.session_factory = session_factory
|
||
self._session = None
|
||
|
||
async def __aenter__(self):
|
||
self._session = self.session_factory()
|
||
return self
|
||
|
||
async def __aexit__(self, exc_type, exc_val, exc_tb):
|
||
if exc_type:
|
||
await self.rollback()
|
||
else:
|
||
await self.commit()
|
||
await self._session.close()
|
||
|
||
async def commit(self):
|
||
await self._session.commit()
|
||
|
||
async def rollback(self):
|
||
await self._session.rollback()
|
||
```
|
||
|
||
### 2. IncidentApprovalService (`src/services/incident_approval_service.py`)
|
||
|
||
```python
|
||
class IncidentApprovalService:
|
||
"""Incident-Approval 同步服務 (ADR-027)"""
|
||
|
||
async def create_with_approval(
|
||
self,
|
||
incident_data: IncidentCreate,
|
||
approval_data: ApprovalCreate
|
||
) -> tuple[Incident, Approval]:
|
||
"""原子建立 Incident + Approval"""
|
||
|
||
async with UnitOfWork(self.session_factory) as uow:
|
||
# 1. PostgreSQL 寫入
|
||
incident = await self.incident_repo.create(uow.session, incident_data)
|
||
approval = await self.approval_repo.create(uow.session, approval_data)
|
||
|
||
# 2. Redis 寫入 (事務外，需補償)
|
||
try:
|
||
await self._write_to_redis(incident, approval)
|
||
except RedisError as e:
|
||
# Saga 補償: rollback PostgreSQL
|
||
await uow.rollback()
|
||
raise IncidentApprovalCreateError(f"Redis 寫入失敗: {e}")
|
||
|
||
return incident, approval
|
||
|
||
async def on_approval_status_change(
|
||
self,
|
||
approval_id: str,
|
||
new_status: ApprovalStatus
|
||
):
|
||
"""Approval 狀態變更 Hook"""
|
||
|
||
# 1. 更新 PostgreSQL
|
||
approval = await self.approval_repo.update_status(approval_id, new_status)
|
||
|
||
# 2. 同步更新關聯 Incident
|
||
incident = await self.incident_repo.get_by_approval_id(approval_id)
|
||
if incident:
|
||
incident_status = self._map_approval_to_incident_status(new_status)
|
||
await self.incident_repo.update_status(incident.id, incident_status)
|
||
|
||
# 3. 更新 Redis TTL
|
||
await self._refresh_redis_ttl(incident.id, approval_id)
|
||
|
||
# 4. 觸發 SSE 事件
|
||
await self._emit_status_change_event(incident, approval)
|
||
```
|
||
|
||
### 3. 統一常量 (`src/core/constants.py`)
|
||
|
||
```python
|
||
# TTL 設定 (秒)
|
||
INCIDENT_TTL_SECONDS = 7 * 24 * 3600 # 7 天
|
||
APPROVAL_TTL_SECONDS = 7 * 24 * 3600 # 7 天
|
||
|
||
# Redis Key 前綴
|
||
REDIS_KEY_INCIDENT = "incidents:"
|
||
REDIS_KEY_APPROVAL = "approvals:"
|
||
REDIS_KEY_PENDING = "pending_approvals"
|
||
|
||
# 狀態映射
|
||
APPROVAL_TO_INCIDENT_STATUS = {
|
||
"pending": "active",
|
||
"approved": "resolved",
|
||
"rejected": "rejected",
|
||
"expired": "expired",
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 四階段實作計畫
|
||
|
||
| Phase | 內容 | 估時 | 狀態 |
|
||
|-------|------|------|------|
|
||
| 1 | UnitOfWork + IncidentApprovalService | 3-4h | ✅ 完成 |
|
||
| 2 | 狀態同步 Hook + TTL 延長 | 2-3h | ✅ 完成 |
|
||
| 3 | 分散式鎖 + TTL 同步 | 2-3h | 🔲 待做 |
|
||
| 4 | 整合測試 | 2h | 🔲 待做 |
|
||
|
||
### 實作完成檔案 (2026-03-26)
|
||
|
||
| 檔案 | 功能 | 行數 |
|
||
|------|------|------|
|
||
| `apps/api/src/core/constants.py` | TTL + Redis Key 常量 | 38 |
|
||
| `apps/api/src/core/unit_of_work.py` | PostgreSQL 事務管理 | 138 |
|
||
| `apps/api/src/services/incident_approval_service.py` | 原子同步服務 | 470 |
|
||
|
||
### Router 整合點
|
||
|
||
| 端點 | 整合方式 |
|
||
|------|----------|
|
||
| `POST /approvals/{id}/sign` | `on_approval_status_change("approved")` |
|
||
| `POST /approvals/{id}/reject` | `on_approval_status_change("rejected")` |
|
||
| `POST /approvals/bulk-approve` | `on_approval_status_change("approved")` |
|
||
|
||
**總估時**: 9-12h
|
||
|
||
---
|
||
|
||
## 驗收標準
|
||
|
||
| 項目 | 狀態 |
|
||
|------|------|
|
||
| Incident + Approval 原子建立 | 🔲 待實作 |
|
||
| PostgreSQL 事務回滾生效 | 🔲 待實作 |
|
||
| Redis 補償機制 (Saga) | 🔲 待實作 |
|
||
| Approval 狀態變更同步 Incident | 🔲 待實作 |
|
||
| TTL 統一管理 | 🔲 待實作 |
|
||
| 整合測試通過 | 🔲 待實作 |
|
||
|
||
---
|
||
|
||
## 🔴🔴 緊急事故修復 (2026-03-26 19:30-20:00)
|
||
|
||
### 事故: Telegram 告警轟炸
|
||
|
||
**症狀**: 同樣告警重複發送數十次，ID 格式為 `INC-INC-INC-2026`
|
||
|
||
### 根因
|
||
|
||
1. **Phase 6.5 (765ee39) 修改**: COMPLETED decision + INVESTIGATING incident → 建立新 decision
|
||
2. **每次 poll 觸發**: 前端 poll `/api/v1/incidents` → 建立新 decision → Telegram
|
||
3. **INC- 前綴重複**: `decision_manager.py` + `telegram_gateway.py` 雙重加前綴
|
||
|
||
### 修復 Commits
|
||
|
||
| Commit | 修復內容 |
|
||
|--------|----------|
|
||
| 35aa690 | decision_manager.py: INC- 前綴修復 + Redis 去重 (10 分鐘) |
|
||
| 139ddc3 | telegram_gateway.py: INC- 前綴檢查 |
|
||
| 6421af0 | cd.yaml: K8s selector 不可變處理 |
|
||
|
||
### 新增防禦機制
|
||
|
||
```python
|
||
# decision_manager.py - Telegram 去重
|
||
dedup_key = f"telegram_sent:{incident.incident_id}"
|
||
if await redis_client.get(dedup_key):
|
||
return # 10 分鐘內不重複發送
|
||
await redis_client.set(dedup_key, "1", ex=600)
|
||
```
|
||
|
||
---
|
||
|
||
## 關聯文件
|
||
|
||
- ADR-011: NetworkPolicy 變更治理架構
|
||
- ADR-026: CoreDNS GitOps 管控架構
|
||
- [project_incident_approval_sync.md](~/.claude/projects/-Users-ogt-awoooi/memory/project_incident_approval_sync.md)
|
||
- [feedback_incident_approval_sync.md](~/.claude/projects/-Users-ogt-awoooi/memory/feedback_incident_approval_sync.md)
|
||
- [feedback_telegram_dedup.md](~/.claude/projects/-Users-ogt-awoooi/memory/feedback_telegram_dedup.md)
|
||
|
||
---
|
||
|
||
## 附錄：驗證指令
|
||
|
||
```bash
|
||
# 檢查活躍事件
|
||
curl -s http://localhost:8000/api/v1/incidents/active | jq '.count'
|
||
|
||
# 檢查 Redis Incident
|
||
redis-cli KEYS "incidents:*"
|
||
|
||
# 檢查 PostgreSQL Incident
|
||
psql -c "SELECT id, status, created_at FROM incidents WHERE status = 'active'"
|
||
|
||
# 驗證同步
|
||
# 1. 建立測試事件
|
||
# 2. 簽核
|
||
# 3. 確認 Incident 狀態同步更新
|
||
```
|