fix(watchdog): ADR-092 B4 — 三層修復消除 META SYSTEM 重複告警 + Ollama 路由強化
All checks were successful
Code Review / ai-code-review (push) Successful in 7m16s
All checks were successful
Code Review / ai-code-review (push) Successful in 7m16s
問題根因(debugger 全景徹查):
1. Prod 仍跑舊版代碼(ec013f66 後的修法未部署 → 告警字串仍含舊格式)
2. replicas=2 時 Pod 間 grace period 不共享 → violation_codes 分歧 → 不同 SHA256 → dedup 失效
3. 新 Pod 啟動立即執行 _check_once() → rollout 時多發一波
4. W6 violation_codes 含動態 low_count → count 微變繞過 dedup
修復(A2/A3/W6/C1/C2):
- A2:run_ai_slo_watchdog_loop 加 90s leading sleep,避免 rollout 立即觸發
- A3:_grace_active() 改為 Redis cluster-shared(watchdog:cluster_grace, ex=1800s, nx=True)
消除 Pod 間 grace period 不一致;Redis 故障時 fallback 為 process-local monotonic
- W6:violation_codes 移除動態 low_count,改為穩定 "W6:trust_drift"
- C1:ollama_auto_recovery.py recovered_host 改動態 label(依 URL port 判斷 GCP-A/B/Local)
- C2:ConfigMap OLLAMA_FALLBACK_URL 改走 110:11437 nginx proxy,三層容災統一架構
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -18,12 +18,13 @@ data:
|
||||
# 2026-04-16 ogt + Claude Sonnet 4.6: 改指向 111(GPU 機,RTX)
|
||||
# 188 = CPU-only Ollama,推理極慢(>60s);111 有 GPU,avg 10s
|
||||
# 2026-05-04 ogt: ADR-110 三層容災正式路由(nginx proxy 架設完成後恢復 GCP 優先)
|
||||
# GCP-A(via 110:11435) → GCP-B(via 110:11436) → 111 兜底
|
||||
# 110 nginx proxy 轉發:11435 → 34.143.170.20:11434, 11436 → 34.21.145.224:11434
|
||||
# GCP-A(via 110:11435) → GCP-B(via 110:11436) → Local(via 110:11437) 統一走 nginx proxy
|
||||
# 110 nginx proxy 轉發:11435→GCP-A, 11436→GCP-B, 11437→192.168.0.111:11434
|
||||
# K8s pods 不可直連 GCP:11434(NetworkPolicy 外網 egress 只開 443)
|
||||
# 2026-05-05 ogt C2 修復:Local fallback 改走 110:11437 統一 nginx proxy,增強可觀測性
|
||||
OLLAMA_URL: "http://192.168.0.110:11435"
|
||||
OLLAMA_SECONDARY_URL: "http://192.168.0.110:11436"
|
||||
OLLAMA_FALLBACK_URL: "http://192.168.0.111:11434"
|
||||
OLLAMA_FALLBACK_URL: "http://192.168.0.110:11437"
|
||||
OPENCLAW_URL: "http://192.168.0.188:8088"
|
||||
KALI_SCANNER_URL: "http://192.168.0.112:8080"
|
||||
SIGNOZ_URL: "http://192.168.0.188:3301"
|
||||
|
||||
Reference in New Issue
Block a user