89e05e6ea2
- ADR-037 監控增強架構 - MONITORING_MASTER_PLAN 主計畫 - MASTER_EXECUTION_SCHEDULE 執行排程 - Phase D/E/Worker HPA Runbooks Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
314 lines
9.3 KiB
Markdown
314 lines
9.3 KiB
Markdown
# RunBook: Worker HPA — 水平自動擴展設定
|
||
|
||
> **類型**: 操作型 RunBook
|
||
> **優先級**: 🔴 P0（Worker 目前單點故障風險）
|
||
> **建立**: 2026-03-29 12:35 (台北)
|
||
> **建立者**: Antigravity
|
||
> **工時預估**: 30–60 分鐘
|
||
> **前置條件**: K3s 叢集健康（120/121 皆 Ready）
|
||
|
||
---
|
||
|
||
## 背景與現況
|
||
|
||
### 🔍 精確現況診斷
|
||
|
||
**現有 HPA 配置 (`12-hpa.yaml`)**：
|
||
|
||
| Deployment | Min | Max | CPU 閾值 | Memory 閾值 |
|
||
|-----------|-----|-----|---------|------------|
|
||
| awoooi-api | 2 | 6 | 70% | 80% |
|
||
| awoooi-web | 2 | 6 | 70% | 80% |
|
||
| **awoooi-worker** | ❌ 無 | ❌ 無 | — | — |
|
||
|
||
**Worker 的特殊性**：
|
||
- Worker 消費 Redis Streams (Event Bus)
|
||
- 不像 API/Web 依賴 CPU/Memory 觸發，應依賴 **Queue 長度觸發**
|
||
- 但 K3s 預設沒有安裝 KEDA（Kubernetes Event-driven Autoscaling）
|
||
- **最保守方案**：設定 min:1 max:3，以 CPU 為指標
|
||
|
||
---
|
||
|
||
## 方案比較
|
||
|
||
| 方案 | 優點 | 缺點 | 適合性 |
|
||
|------|------|------|-------|
|
||
| **A: CPU HPA（立即可行）** | 零依賴，立即部署 | 不直接反應 Queue 長度 | ✅ 推薦（短期） |
|
||
| B: KEDA Redis Stream HPA | 最精確，按 Queue 長度擴縮 | 需安裝 KEDA operator | 🟡 中期規劃 |
|
||
| C: 固定 2 副本（無 HPA） | 簡單穩定 | 浪費資源 | ❌ 不推薦 |
|
||
|
||
**決策**：採用方案 A（CPU HPA），並記錄方案 B 的未來路徑。
|
||
|
||
---
|
||
|
||
## Step 1: 確認 Worker 資源設定
|
||
|
||
```bash
|
||
# 查看現有 Worker Deployment 資源限制
|
||
kubectl get deployment awoooi-worker -n awoooi-prod -o yaml | grep -A 20 resources
|
||
|
||
# 預期看到：
|
||
# resources:
|
||
# requests:
|
||
# cpu: "100m"
|
||
# memory: "256Mi"
|
||
# limits:
|
||
# cpu: "500m"
|
||
# memory: "512Mi"
|
||
```
|
||
|
||
**如果沒有設定 resources，HPA 無法正常運作！** 必須先在 `08-deployment-worker.yaml` 加入資源限制。
|
||
|
||
---
|
||
|
||
## Step 2: 更新 k8s/awoooi-prod/12-hpa.yaml
|
||
|
||
在現有檔案末尾追加 Worker HPA：
|
||
|
||
```yaml
|
||
# =============================================================================
|
||
# Worker HPA（追加到 12-hpa.yaml 末尾）
|
||
# =============================================================================
|
||
# K-Worker 2026-03-29: Worker HPA（CPU 指標，min:1 max:3）
|
||
# 注意：Worker 消費 Redis Streams，未來可升級為 KEDA Redis Stream 指標
|
||
# 建立者：Antigravity
|
||
# =============================================================================
|
||
---
|
||
apiVersion: autoscaling/v2
|
||
kind: HorizontalPodAutoscaler
|
||
metadata:
|
||
name: awoooi-worker-hpa
|
||
namespace: awoooi-prod
|
||
labels:
|
||
app.kubernetes.io/name: awoooi
|
||
app.kubernetes.io/component: worker
|
||
annotations:
|
||
description: "Worker 水平自動擴展 (1-3 replicas, 70% CPU)"
|
||
note: "未來可升級為 KEDA Redis Stream 指標，按 Queue 長度動態擴縮"
|
||
spec:
|
||
scaleTargetRef:
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
name: awoooi-worker
|
||
minReplicas: 1 # 保持最少 1 個處理事件
|
||
maxReplicas: 3 # 2 節點叢集的合理上限
|
||
metrics:
|
||
- type: Resource
|
||
resource:
|
||
name: cpu
|
||
target:
|
||
type: Utilization
|
||
averageUtilization: 70
|
||
- type: Resource
|
||
resource:
|
||
name: memory
|
||
target:
|
||
type: Utilization
|
||
averageUtilization: 80
|
||
behavior:
|
||
scaleUp:
|
||
stabilizationWindowSeconds: 120 # Worker 擴展比 API 保守（120s vs 60s）
|
||
policies:
|
||
- type: Pods
|
||
value: 1
|
||
periodSeconds: 120
|
||
scaleDown:
|
||
stabilizationWindowSeconds: 600 # Worker 縮容非常保守，避免事件處理中斷
|
||
policies:
|
||
- type: Pods
|
||
value: 1
|
||
periodSeconds: 300
|
||
```
|
||
|
||
---
|
||
|
||
## Step 3: 確認 Worker Deployment 有資源設定
|
||
|
||
```bash
|
||
# 查看現有設定
|
||
kubectl get deployment awoooi-worker -n awoooi-prod -o jsonpath='{.spec.template.spec.containers[0].resources}'
|
||
```
|
||
|
||
若無資源設定，在 `08-deployment-worker.yaml` 加入：
|
||
|
||
```yaml
|
||
# apps/api/src/workers 對應的 K8s Deployment
|
||
# 在 container spec 加入：
|
||
resources:
|
||
requests:
|
||
cpu: "100m" # Worker 正常負載估算
|
||
memory: "256Mi"
|
||
limits:
|
||
cpu: "500m" # 防止單 Worker 吃掉所有 CPU
|
||
memory: "512Mi"
|
||
```
|
||
|
||
---
|
||
|
||
## Step 4: 部署
|
||
|
||
```bash
|
||
# 方法 A：直接 apply（推薦，只更新 HPA）
|
||
kubectl apply -f k8s/awoooi-prod/12-hpa.yaml
|
||
|
||
# 確認 HPA 建立成功
|
||
kubectl get hpa -n awoooi-prod
|
||
|
||
# 預期輸出：
|
||
# NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS
|
||
# awoooi-api-hpa Deployment/api 5%/70% 2 6 2
|
||
# awoooi-web-hpa Deployment/web 3%/70% 2 6 2
|
||
# awoooi-worker-hpa Deployment/worker 8%/70% 1 3 1 ← 新增
|
||
|
||
# 方法 B：透過 CD 觸發（標準流程）
|
||
git add k8s/awoooi-prod/12-hpa.yaml
|
||
git commit -m "feat(k8s): add Worker HPA (min:1 max:3 CPU 70%)"
|
||
git push origin main
|
||
```
|
||
|
||
---
|
||
|
||
## Step 5: 壓力測試驗證 HPA 觸發
|
||
|
||
```bash
|
||
# 模擬大量事件涌入（謹慎，在非尖峰時段執行）
|
||
for i in {1..100}; do
|
||
curl -s -X POST http://192.168.0.120:32334/api/v1/webhooks/alertmanager \
|
||
-H "Content-Type: application/json" \
|
||
-d '{
|
||
"version": "4",
|
||
"status": "firing",
|
||
"alerts": [{"status": "firing", "labels": {"alertname": "LoadTest", "severity": "info"}, "annotations": {}}]
|
||
}' &
|
||
done
|
||
|
||
# 觀察 HPA 反應（每 15 秒看一次）
|
||
watch -n 15 'kubectl get hpa awoooi-worker-hpa -n awoooi-prod'
|
||
```
|
||
|
||
---
|
||
|
||
## 中期路線圖：升級 KEDA Redis Stream HPA
|
||
|
||
```yaml
|
||
# 未來安裝 KEDA 後，可替換為更精確的 HPA：
|
||
apiVersion: keda.sh/v1alpha1
|
||
kind: ScaledObject
|
||
metadata:
|
||
name: awoooi-worker-scaledobject
|
||
namespace: awoooi-prod
|
||
spec:
|
||
scaleTargetRef:
|
||
name: awoooi-worker
|
||
minReplicaCount: 1
|
||
maxReplicaCount: 5
|
||
triggers:
|
||
- type: redis
|
||
metadata:
|
||
address: "192.168.0.188:6380"
|
||
listName: "awoooi:events" # Redis Stream Key
|
||
listLength: "20" # 每個 Pod 處理最多 20 個待處理事件
|
||
```
|
||
|
||
KEDA 安裝指令（未來執行）：
|
||
```bash
|
||
kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.13.1/keda-2.13.1.yaml
|
||
```
|
||
|
||
---
|
||
|
||
## 驗收標準
|
||
|
||
| 項目 | 通過條件 |
|
||
|------|---------|
|
||
| HPA 建立 | `kubectl get hpa -n awoooi-prod` 顯示 `awoooi-worker-hpa` |
|
||
| 指標正常 | TARGETS 顯示實際 CPU%，非 `<unknown>` |
|
||
| Worker 正常運行 | `kubectl get pod -n awoooi-prod -l app=awoooi-worker` 顯示 Running |
|
||
| 最小副本 | Worker 期望副本數 = 1 |
|
||
|
||
---
|
||
|
||
## ⚠️ 架構安全補丁（2026-03-29 更新，部署前必讀）
|
||
|
||
> 來源：`ARCHITECTURAL_RISK_WAR_GAME.md` 深度沙盤推演，代碼確認級別
|
||
|
||
### 補丁 1：XCLAIM + Active Sweeper（部署 HPA 的前置條件）
|
||
|
||
**❌ 現況**：`signal_worker.py` 完全沒有 Redis PEL 孤兒任務回收機制。
|
||
|
||
**影響**：Worker Pod 被 HPA 縮容（或非優雅崩潰）時，正在處理的任務卡在 Redis PEL（Pending Entries List）中永久無人處理。
|
||
|
||
> 🔴 **HPA 必須在 XCLAIM 機制合併 main 之後才能部署！**
|
||
|
||
需要在 `signal_worker.py` 加入的兩個機制：
|
||
|
||
```python
|
||
# 1. 啟動時接管孤兒（_claim_orphaned_tasks，在 start() 中調用）
|
||
# 2. 運行中持續掃描（_reclaim_loop，與 _consume_loop 並行）
|
||
async def _reclaim_loop(self, interval_s: int = 300) -> None:
|
||
"""每 5 分鐘主動掃描 PEL，接管閒置超過 5 分鐘的孤兒任務"""
|
||
while self._running:
|
||
await asyncio.sleep(interval_s)
|
||
claimed = await self._claim_orphaned_tasks(idle_ms=300_000)
|
||
if claimed > 0:
|
||
logger.info("active_sweeper_claimed", count=claimed)
|
||
```
|
||
|
||
---
|
||
|
||
### 補丁 2：terminationGracePeriodSeconds 三層對齊
|
||
|
||
**❌ 現況**：`signal_worker.py` 的 `stop()` timeout = **5 秒**，AI 分析任務最長 60 秒。K8s 的 `terminationGracePeriodSeconds` 未設定（預設 30 秒）。兩個值都不夠，且彼此不對齊。
|
||
|
||
**需要同時修改兩個地方**：
|
||
|
||
```yaml
|
||
# k8s/awoooi-prod/08-deployment-worker.yaml
|
||
spec:
|
||
template:
|
||
spec:
|
||
terminationGracePeriodSeconds: 90 # 🆕 必須設定（比 Python timeout 多 15 秒緩衝）
|
||
containers:
|
||
- name: awoooi-worker
|
||
lifecycle:
|
||
preStop:
|
||
exec:
|
||
command: ["/bin/sh", "-c", "sleep 5"] # 讓 K8s 先更新 Endpoint 再發 SIGTERM
|
||
```
|
||
|
||
```python
|
||
# apps/api/src/workers/signal_worker.py
|
||
async def stop(self) -> None:
|
||
self._running = False
|
||
if self._task:
|
||
try:
|
||
await asyncio.wait_for(self._task, timeout=75.0) # 🆕 從 5 秒改為 75 秒
|
||
except (TimeoutError, asyncio.CancelledError):
|
||
self._task.cancel()
|
||
logger.info("signal_worker_stopped")
|
||
```
|
||
|
||
**三層數值關係**：
|
||
```
|
||
preStop sleep: 5s
|
||
Python timeout: 75s ← 比 K8s grace period 少 15s 緩衝
|
||
K8s grace period: 90s ← terminationGracePeriodSeconds
|
||
```
|
||
|
||
---
|
||
|
||
### 合規確認指令（部署後必須執行）
|
||
|
||
```bash
|
||
# 確認 terminationGracePeriodSeconds 已生效
|
||
kubectl get deployment awoooi-worker -n awoooi-prod \
|
||
-o jsonpath='{.spec.template.spec.terminationGracePeriodSeconds}'
|
||
# 預期：90
|
||
|
||
# 模擬縮容，確認優雅關機
|
||
kubectl scale deployment awoooi-worker -n awoooi-prod --replicas=0
|
||
kubectl logs -n awoooi-prod -l app=awoooi-worker --tail=20
|
||
# 預期看到：shutdown_signal_received → signal_worker_shutting_down → signal_worker_stopped
|
||
# 整個流程在 90 秒內完成
|
||
```
|