52177d7096
All checks were successful
E2E Health Check / e2e-health (push) Successful in 16s
- ARCHITECTURE_MEMORY: 新增 Service 層架構說明 - ADR-041: 更新 Phase 21 定期報告狀態 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
237 lines
5.6 KiB
Markdown
237 lines
5.6 KiB
Markdown
# ADR-041: 定期報告架構 (Periodic Reporting Architecture)
|
||
|
||
> **狀態**: ✅ 全部完成 (Phase 21.1 + 21.2 + 21.3)
|
||
> **日期**: 2026-03-31 (台北時間)
|
||
> **決策者**: 統帥 (CEO) + Claude Code (首席架構師)
|
||
> **觸發**: K3s 優化成效驗證後，需要定期報告機制
|
||
|
||
---
|
||
|
||
## 背景
|
||
|
||
### 現狀
|
||
|
||
K3s 優化完成後 (告警 -100%, Pod 重啟 -100%, 48h+ 穩定)，缺乏定期報告機制：
|
||
|
||
| 問題 | 影響 | 根因 |
|
||
|------|------|------|
|
||
| 無定期健康檢查 | 異常可能延遲發現 | E2E 只在 push 時觸發 |
|
||
| K3s 成效無報告 | 統帥無法掌握叢集狀態 | 缺少摘要機制 |
|
||
| 週報需人工整理 | 耗費時間，可能遺漏 | 無自動化整合 |
|
||
|
||
### 現有元件
|
||
|
||
| 元件 | 位置 | 狀態 |
|
||
|------|------|------|
|
||
| `DailySummaryMessage` | telegram_gateway.py | ✅ 已實作 |
|
||
| `RepairReportMessage` | telegram_gateway.py | ✅ 已實作 |
|
||
| `StatsService` | stats_service.py | ✅ 6 方法 + Redis 快取 |
|
||
| E2E Workflow | e2e-health.yaml | ✅ 缺 schedule |
|
||
| Prometheus | kube-state-metrics | ✅ :30888 |
|
||
|
||
---
|
||
|
||
## 決策
|
||
|
||
### 採用 Phase 21 三階段架構
|
||
|
||
```
|
||
Phase 21: Periodic Reporting
|
||
├── 21.1 Daily E2E Schedule (P0 - 15min)
|
||
├── 21.2 K3s Status Telegram Report (P1 - 2h)
|
||
└── 21.3 Weekly Report Automation (P2 - 2-4h)
|
||
```
|
||
|
||
### 21.1 Daily E2E Schedule
|
||
|
||
**變更**: `.gitea/workflows/e2e-health.yaml`
|
||
|
||
```yaml
|
||
on:
|
||
push:
|
||
branches: [main]
|
||
workflow_dispatch:
|
||
schedule:
|
||
- cron: '0 16 * * *' # 每日 00:00 台北 (UTC+8)
|
||
```
|
||
|
||
**觸發機制**: Gitea 內建 schedule
|
||
|
||
### 21.2 K3s Status Telegram Report
|
||
|
||
**新增元件**:
|
||
|
||
```python
|
||
# k3s_monitor_service.py
|
||
class K3sMonitorService:
|
||
async def collect_cluster_status() -> K3sStatusMessage
|
||
async def send_daily_report() -> bool
|
||
|
||
# k3s_status.py
|
||
@dataclass
|
||
class K3sStatusMessage:
|
||
node_total: int
|
||
node_ready: int
|
||
pod_total: int
|
||
pod_running: int
|
||
hpa_status: dict
|
||
etcd_backup_last: str
|
||
velero_backup_last: str
|
||
alert_count_48h: int
|
||
```
|
||
|
||
**數據來源**: Prometheus (kube-state-metrics) + kubectl (fallback)
|
||
|
||
**排程**: K8s CronJob (每日 09:00 台北)
|
||
|
||
### 21.3 Weekly Report Automation
|
||
|
||
**選項 A (推薦): Telegram Only**
|
||
|
||
```python
|
||
@dataclass
|
||
class WeeklyReportMessage:
|
||
week_range: str # "2026-W14"
|
||
# 告警統計 (from StatsService)
|
||
alert_total: int
|
||
alert_critical: int
|
||
resolved_rate: float
|
||
# AI 效能
|
||
ai_proposal_count: int
|
||
ai_success_rate: float
|
||
# K3s 健康 (from K3sMonitorService)
|
||
k3s_uptime: str
|
||
hpa_scale_events: int
|
||
# Git 活動
|
||
commits_count: int
|
||
deploy_count: int
|
||
# 成本
|
||
ai_cost_week: float
|
||
```
|
||
|
||
**排程**: K8s CronJob (每週五 18:00 台北)
|
||
|
||
---
|
||
|
||
## 理由
|
||
|
||
### 為什麼三階段？
|
||
|
||
| 原則 | 說明 |
|
||
|------|------|
|
||
| **漸進式** | 由簡到繁，逐步驗證 |
|
||
| **復用最大化** | 70%+ 使用現有元件 |
|
||
| **風險控制** | 每階段獨立驗證 |
|
||
|
||
### 為什麼 Telegram 優先？
|
||
|
||
| 比較 | Telegram | Email |
|
||
|------|----------|-------|
|
||
| 實時性 | ✅ 即時推送 | ❌ 可能延遲 |
|
||
| 互動性 | ✅ 可加按鈕 | ❌ 被動 |
|
||
| 實作成本 | ✅ 已有 Gateway | ❌ 需新增 SMTP |
|
||
| 歸檔 | ⚠️ 需另存 | ✅ 自然歸檔 |
|
||
|
||
### 為什麼 CronJob 而非 APScheduler？
|
||
|
||
| 比較 | K8s CronJob | APScheduler |
|
||
|------|-------------|-------------|
|
||
| 可靠性 | ✅ 叢集管理 | ⚠️ 依賴單 Pod |
|
||
| 監控 | ✅ kubectl 可見 | ❌ 需自建 |
|
||
| 故障恢復 | ✅ 自動重試 | ❌ 需手動 |
|
||
| 符合 ADR-033 | ✅ K3s 原生 | ❌ 應用層 |
|
||
|
||
---
|
||
|
||
## 後果
|
||
|
||
### 優點
|
||
|
||
- ✅ 自動化健康監控，異常早發現
|
||
- ✅ 統帥每日掌握 K3s 狀態
|
||
- ✅ 週報自動生成，節省時間
|
||
- ✅ 復用現有 70%+ 元件
|
||
|
||
### 缺點
|
||
|
||
- ⚠️ 新增 K8s CronJob 需維護
|
||
- ⚠️ Telegram 歸檔需另行處理
|
||
- ⚠️ 報告格式調整需改代碼
|
||
|
||
### 風險
|
||
|
||
| 風險 | 機率 | 緩解 |
|
||
|------|------|------|
|
||
| Prometheus 不穩定 | 低 | kubectl fallback |
|
||
| CronJob 延遲 | 低 | 監控 Job 狀態 |
|
||
| Telegram Rate Limit | 低 | 已有去重機制 |
|
||
|
||
---
|
||
|
||
## 實施計畫
|
||
|
||
### 工時預估
|
||
|
||
| Phase | 工時 | 依賴 |
|
||
|-------|------|------|
|
||
| 21.1 Daily E2E | 15 min | 無 |
|
||
| 21.2 K3s Report | 2h | 21.1 |
|
||
| 21.3 Weekly Report | 2h (A) / 4h (B) | 21.1, 21.2 |
|
||
| **總計** | **4h 15min** | - |
|
||
|
||
### 驗收標準
|
||
|
||
#### 21.1 完成標準
|
||
|
||
```bash
|
||
# Gitea 顯示 scheduled run
|
||
curl -s http://192.168.0.110:3001/api/v1/repos/.../actions/runs
|
||
|
||
# 00:00 台北自動執行
|
||
# (查看 Gitea Actions 執行紀錄)
|
||
```
|
||
|
||
#### 21.2 完成標準
|
||
|
||
```bash
|
||
# Telegram 收到 K3s 報告
|
||
# (每日 09:00 台北)
|
||
|
||
# CronJob 狀態
|
||
kubectl get cronjob k3s-status-report -n awoooi-prod
|
||
```
|
||
|
||
#### 21.3 完成標準
|
||
|
||
```bash
|
||
# Telegram 收到週報
|
||
# (每週五 18:00 台北)
|
||
|
||
# CronJob 狀態
|
||
kubectl get cronjob weekly-report -n awoooi-prod
|
||
```
|
||
|
||
---
|
||
|
||
## 關聯文件
|
||
|
||
| 文件 | 用途 |
|
||
|------|------|
|
||
| `memory/project_periodic_reporting_plan.md` | 詳細計畫 |
|
||
| `memory/project_k3s_optimization_metrics.md` | K3s 成效數據 |
|
||
| `docs/adr/ADR-033-k3s-ha-architecture.md` | K3s HA 架構 |
|
||
| `docs/adr/ADR-037-monitoring-enhancement-architecture.md` | 監控增強 |
|
||
| `apps/api/src/services/telegram_gateway.py` | 訊息模板 |
|
||
| `apps/api/src/services/stats_service.py` | 統計服務 |
|
||
|
||
---
|
||
|
||
## 變更紀錄
|
||
|
||
| 版本 | 日期 | 執行者 | 變更內容 |
|
||
|------|------|--------|----------|
|
||
| 1.0 | 2026-03-31 | Claude Code | 初始提案 |
|
||
| 1.1 | 2026-03-31 | Claude Code | **Phase 21.1 完成** (Commit f2aa9a7) |
|
||
| 1.2 | 2026-03-31 | Claude Code | **Phase 21.2 完成** (Commit b2e41eb) |
|
||
| 1.3 | 2026-03-31 | Claude Code | **Phase 21.3 完成** (Commit 723e8ef) - **全部完成！** |
|