feat(p1): Ollama 多層容災系統 — P1.1 健康檢測 + P1.2 ai_router 整合 + P1.5 容災告警
ADR-092 P1 飛輪閉環的 Ollama 失敗轉移子系統,全部 Engineer-A2/C/C2 補上。 新服務 (1581 行): - ollama_health_monitor.py (356):3 層健康檢測(TCP/HTTP/推理) - ollama_failover_manager.py (571):111→188 自動切換 + Redis 持久化 + recovery callback - ollama_auto_recovery.py (436):30s 背景監控 + 連續 3 次 HEALTHY → 切回 + clear_cache - failover_alerter.py (218):P1.5 Telegram 容災告警 服務整合: - ai_router.py: AIProviderEnum.OLLAMA_188 + 120s budget + failover fallback chain - main.py lifespan: 啟動時 wire callback + start recovery,關閉時優雅 stop - config.py: OLLAMA_FALLBACK_URL / OLLAMA_HEALTH_CHECK_MODEL / GEMINI_DAILY_QUOTA(帳單熔斷) K8s 配置: - 04-configmap.yaml.patch-188-fallback:注入 OLLAMA_FALLBACK_URL=http://192.168.0.188:11434 測試 (2082 行): - test_ollama_health_monitor.py (402) - test_ollama_failover_manager.py (707) - test_ollama_auto_recovery.py (580) - test_ai_router_failover_integration.py (257) - test_lifespan_failover_wiring.py (136) 依賴鏈:service 三件套 + ai_router + main.py 一起 commit,缺一就 ImportError。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
34
k8s/awoooi-prod/04-configmap.yaml.patch-188-fallback
Normal file
34
k8s/awoooi-prod/04-configmap.yaml.patch-188-fallback
Normal file
@@ -0,0 +1,34 @@
|
||||
# ============================================================================
|
||||
# PATCH: 188 CPU-only Ollama 備援端點
|
||||
# 日期: 2026-04-25 (台北時區)
|
||||
# 負責人: ogt + Claude Sonnet 4.6
|
||||
# ADR 參考: plan_complete_v3.md P0.5
|
||||
# 診斷實測數據:
|
||||
# 主機: 192.168.0.188, Intel Xeon Silver 4214 @ 2.2GHz, 12 核, CPU-only
|
||||
# RAM: 62GB (used 14GB), Disk: 982GB (used 221GB)
|
||||
# GPU: 無
|
||||
# 現有模型: qwen2.5:7b-instruct (4.5GB), llama3.2:3b (1.9GB),
|
||||
# deepseek-r1:14b (8.5GB), nomic-embed-text (261MB)
|
||||
# 推理延遲實測: qwen2.5:7b-instruct → total=111s, eval_rate=0.09 token/s
|
||||
# llama3.2:3b → total=155s (cold start, 比 7b 更慢)
|
||||
# 目標 ~30s 無法達到 (CPU 推理硬上限 ~0.09 token/s)
|
||||
# 決策: qwen2.5:7b-instruct 已存在,設為備援 (111s 延遲,使用者需知情)
|
||||
# 連通性: 110 → 188:11434 ✅ 已驗證
|
||||
# ⚠️ 注意: 188 推理極慢(~111s),應只在 111 GPU Ollama 完全失效時啟用
|
||||
# 建議: 程式碼層應設 OLLAMA_FALLBACK_188_TIMEOUT_SEC = 150
|
||||
# ============================================================================
|
||||
#
|
||||
# 將以下兩行加入 /Users/ogt/awoooi/k8s/awoooi-prod/04-configmap.yaml
|
||||
# 建議位置: OLLAMA_URL 行 (第 20 行) 之後
|
||||
#
|
||||
# --- 新增內容 ---
|
||||
# 2026-04-25 ogt + Claude Sonnet 4.6: 188 CPU-only Ollama 備援 (plan_complete_v3 P0.5)
|
||||
# ⚠️ 188 推理延遲實測 ~111s (0.09 token/s, CPU-only Xeon 4214),僅作 111 完全失效時的降級備援
|
||||
# 模型已存在: qwen2.5:7b-instruct (4.5GB), 無需重拉
|
||||
OLLAMA_FALLBACK_188: "http://192.168.0.188:11434"
|
||||
OLLAMA_188_MODEL: "qwen2.5:7b-instruct"
|
||||
# --- 新增內容結束 ---
|
||||
#
|
||||
# 使用方式 (需用戶 review 後手動 apply):
|
||||
# kubectl -n awoooi-prod apply -f k8s/awoooi-prod/04-configmap.yaml
|
||||
# kubectl -n awoooi-prod rollout restart deployment/awoooi-api
|
||||
Reference in New Issue
Block a user