fix(watchdog): ADR-092 B4 — 三層修復消除 META SYSTEM 重複告警 + Ollama 路由強化
All checks were successful
Code Review / ai-code-review (push) Successful in 7m16s

問題根因(debugger 全景徹查):
1. Prod 仍跑舊版代碼(ec013f66 後的修法未部署 → 告警字串仍含舊格式)
2. replicas=2 時 Pod 間 grace period 不共享 → violation_codes 分歧 → 不同 SHA256 → dedup 失效
3. 新 Pod 啟動立即執行 _check_once() → rollout 時多發一波
4. W6 violation_codes 含動態 low_count → count 微變繞過 dedup

修復(A2/A3/W6/C1/C2):
- A2:run_ai_slo_watchdog_loop 加 90s leading sleep,避免 rollout 立即觸發
- A3:_grace_active() 改為 Redis cluster-shared(watchdog:cluster_grace, ex=1800s, nx=True)
     消除 Pod 間 grace period 不一致;Redis 故障時 fallback 為 process-local monotonic
- W6:violation_codes 移除動態 low_count,改為穩定 "W6:trust_drift"
- C1:ollama_auto_recovery.py recovered_host 改動態 label(依 URL port 判斷 GCP-A/B/Local)
- C2:ConfigMap OLLAMA_FALLBACK_URL 改走 110:11437 nginx proxy,三層容災統一架構

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Your Name
2026-05-05 10:31:53 +08:00
parent 3f853accf2
commit aa4ccec429
3 changed files with 57 additions and 14 deletions

View File

@@ -18,12 +18,13 @@ data:
# 2026-04-16 ogt + Claude Sonnet 4.6: 改指向 111GPU 機RTX
# 188 = CPU-only Ollama推理極慢>60s111 有 GPUavg 10s
# 2026-05-04 ogt: ADR-110 三層容災正式路由nginx proxy 架設完成後恢復 GCP 優先)
# GCP-A(via 110:11435) → GCP-B(via 110:11436) → 111 兜底
# 110 nginx proxy 轉發11435 → 34.143.170.20:11434, 11436 → 34.21.145.224:11434
# GCP-A(via 110:11435) → GCP-B(via 110:11436) → Local(via 110:11437) 統一走 nginx proxy
# 110 nginx proxy 轉發11435→GCP-A, 11436→GCP-B, 11437→192.168.0.111:11434
# K8s pods 不可直連 GCP:11434NetworkPolicy 外網 egress 只開 443
# 2026-05-05 ogt C2 修復Local fallback 改走 110:11437 統一 nginx proxy增強可觀測性
OLLAMA_URL: "http://192.168.0.110:11435"
OLLAMA_SECONDARY_URL: "http://192.168.0.110:11436"
OLLAMA_FALLBACK_URL: "http://192.168.0.111:11434"
OLLAMA_FALLBACK_URL: "http://192.168.0.110:11437"
OPENCLAW_URL: "http://192.168.0.188:8088"
KALI_SCANNER_URL: "http://192.168.0.112:8080"
SIGNOZ_URL: "http://192.168.0.188:3301"