feat(ollama): ADR-110 GCP 三層容災架構(GCP-A → GCP-B → Local → Gemini)
## 變更摘要 - Primary: http://34.143.170.20:11434 (GCP-A SSD, 9x 載速 + 2x 推理) - Secondary: http://34.21.145.224:11434 (GCP-B SSD) - Fallback: http://192.168.0.111:11434 (M1 Pro Local HDD,最後防線) - 廢止 ADR-105「111 唯一鐵律」,新建 ADR-110 ## 核心改動 - config.py: 新增 OLLAMA_SECONDARY_URL;validator 加 GCP IP 白名單(34.143.170.20, 34.21.145.224) - ollama_failover_manager.py: 三層 Ollama 決策矩陣;並行健康檢查三台;health_111 → health_gcp_a - ollama_health_monitor.py: host label 萃取改為通用版(支援 GCP 公網 IP) - failover_alerter.py: 故障/恢復主機動態顯示,不再硬編碼「Ollama 111 (GPU)」 - ollama_auto_recovery.py: notify_recovery 改為 ollama_gcp_a;recovered_host 動態 - k8s/awoooi-prod: configmap + deployment + network-policy 同步更新(egress 加 GCP /32) - 服務層: 10 個服務檔案硬編碼 192.168.0.111 改為讀 settings.OLLAMA_URL - 測試: URL 常數更新,新增三層容災場景,GCP IP 白名單驗證測試 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,7 @@
|
||||
# ops/monitoring/ollama_health_rules.yaml
|
||||
# AWOOOI Ollama 容災健康告警規則
|
||||
# 2026-04-26 P2.3 by Claude Sonnet 4.6 (tool-expert) — Ollama 容災監控告警規則
|
||||
# 2026-05-03 ogt: ADR-110 GCP 三層容災,更新健康規則 action 說明(GCP-A/B + Local)
|
||||
# 部署目標: 與 alerts-unified.yml 一起部署到 192.168.0.110:/home/wooo/monitoring/alerts.yml
|
||||
# 部署方式: 手動合併至 alerts-unified.yml,或 scripts/ops/deploy-alerts.sh 支援多檔時直接引用
|
||||
#
|
||||
@@ -44,7 +45,7 @@ groups:
|
||||
summary: "Ollama {{ $labels.job }} 離線 ({{ $labels.instance }})"
|
||||
description: "Prometheus 探測 Ollama {{ $labels.job }} 失敗超過 2 分鐘。預期容災應已觸發,路由已切 Gemini。"
|
||||
runbook: "docs/runbooks/RUNBOOK-OLLAMA-FAILOVER.md#ollama-instance-down"
|
||||
action: "ssh wooo@192.168.0.111 'systemctl status ollama' 或 ssh wooo@192.168.0.188 'systemctl status ollama'"
|
||||
action: "curl http://34.143.170.20:11434/api/tags(GCP-A)或 curl http://34.21.145.224:11434/api/tags(GCP-B)或 ssh wooo@192.168.0.111 'systemctl status ollama'(Local 後備)"
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# 🟡 [ACTIVE] Failover 觸發頻率過高
|
||||
@@ -64,7 +65,7 @@ groups:
|
||||
summary: "Ollama 容災觸發頻率 > 5/h,主機可能不穩定"
|
||||
description: "過去 1 小時 Ollama failover 超過 5 次。建議檢查 111 主機穩定性。"
|
||||
runbook: "docs/runbooks/RUNBOOK-OLLAMA-FAILOVER.md#failover-frequent"
|
||||
action: "ssh wooo@192.168.0.111 'nvidia-smi && journalctl -u ollama -n 50'"
|
||||
action: "curl http://34.143.170.20:11434/api/tags 或 ssh wooo@192.168.0.111 'nvidia-smi && journalctl -u ollama -n 50'(GCP-A 掛才用 111)"
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# 🟡 [ACTIVE] Auto Recovery 停滯(111 已恢復但仍走 Gemini)
|
||||
@@ -106,7 +107,7 @@ groups:
|
||||
# team: ai
|
||||
# annotations:
|
||||
# summary: "Ollama P99 推理延遲 > 30s"
|
||||
# action: "ssh wooo@192.168.0.111 'nvidia-smi' 確認 GPU 記憶體"
|
||||
# action: "curl http://34.143.170.20:11434/api/tags 或 ssh wooo@192.168.0.111 'nvidia-smi'(GCP-A 掛才用 111)"
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# 🟡 [PARTIAL] Gemini 配額即將耗盡
|
||||
|
||||
Reference in New Issue
Block a user