fix(ollama): 修復容災鏈四大 bug — OFFLINE cache 放大 + SLOW 路由缺失 + recovery 命名不一致 + 告警顯示
All checks were successful
Code Review / ai-code-review (push) Successful in 48s

根因:NetworkPolicy reload/CNI 瞬態抖動導致三台 Ollama 同時 OFFLINE,被 30s Redis cache 放大
  → 後續 30s 所有請求誤走 Gemini,燒 quota

B1 ollama_health_monitor: OFFLINE TTL 從 30s 縮短至 5s,儘速重試
B3 ollama_health_monitor: inference ConnectError 改判 DEGRADED(connectivity 通了不算 OFFLINE)
B5/B6 ollama_auto_recovery: _current_primary 預設改 "ollama_gcp_a",比對改 startswith("ollama_")
SLOW 修復: failover_manager SLOW 節點視為可用(優於 Gemini quota 耗盡)
SLOW 修復: auto_recovery SLOW 也計入 recovery counter(GCP 高負載仍可切回)
告警顯示: _provider_display 加入 GCP-A/B/Local 具體伺服器識別
告警顯示: _format_automation_block 加入 Token 用量行

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Your Name
2026-05-04 19:01:27 +08:00
parent f6b698c873
commit 855819652e
4 changed files with 90 additions and 14 deletions

View File

@@ -218,9 +218,15 @@ class TelegramMessage:
nemotron_latency_ms: float = 0.0 # Nemotron 呼叫延遲 (ms)
def _provider_display(self) -> tuple[str, str]:
"""Return display provider and optional model suffix."""
"""Return display provider and optional model suffix.
2026-05-04 ogt: 加入具體 Ollama 伺服器顯示GCP-A/B/Local
"""
provider_names = {
"ollama": "Ollama",
# 2026-05-04 ogt: ADR-110 三層容災具體伺服器識別
"ollama_gcp_a": "Ollama GCP-A (34.143.170.20)",
"ollama_gcp_b": "Ollama GCP-B (34.21.145.224)",
"ollama_local": "Ollama Local (111)",
"gemini": "Gemini",
"claude": "Claude",
"nvidia": "Nemotron",
@@ -249,7 +255,9 @@ class TelegramMessage:
return "safe_gate_pending"
def _format_automation_block(self) -> str:
"""Visible AI automation chain for every ACTION REQUIRED card."""
"""Visible AI automation chain for every ACTION REQUIRED card.
2026-05-04 ogt: 加入 Token 用量 + 具體 Ollama 伺服器顯示
"""
provider_display, model_suffix = self._provider_display()
mode = self._automation_mode()
openclaw_state = provider_display if provider_display != "rule_fallback" else "degraded"
@@ -258,6 +266,12 @@ class TelegramMessage:
elephant_state = "timeline_km_pending"
flow = "webhook&gt;investigator&gt;router&gt;llm/rule&gt;safe&gt;approval"
# 2026-05-04 ogt: Token 用量顯示(有資料才顯示)
token_line = ""
if self.ai_tokens > 0:
cost_str = f" / ${self.ai_cost:.4f}" if self.ai_cost > 0 else ""
token_line = f"├ Tokens<code>{self.ai_tokens:,}{cost_str}</code>\n"
return (
f"🤖 <b>AI 自動化鏈路</b>\n"
f"├ Router<code>{html.escape(provider_display)}{model_suffix}</code>\n"
@@ -266,6 +280,7 @@ class TelegramMessage:
f"NemoTron<code>{html.escape(nemotron_state)}</code>\n"
f"├ Hermes<code>{html.escape(hermes_state)}</code> | "
f"ElephantAlpha<code>{html.escape(elephant_state)}</code>\n"
f"{token_line}"
f"└ Flow<code>{flow}</code>\n"
)