fix(ollama): 修復容災鏈四大 bug — OFFLINE cache 放大 + SLOW 路由缺失 + recovery 命名不一致 + 告警顯示
All checks were successful
Code Review / ai-code-review (push) Successful in 48s
All checks were successful
Code Review / ai-code-review (push) Successful in 48s
根因:NetworkPolicy reload/CNI 瞬態抖動導致三台 Ollama 同時 OFFLINE,被 30s Redis cache 放大
→ 後續 30s 所有請求誤走 Gemini,燒 quota
B1 ollama_health_monitor: OFFLINE TTL 從 30s 縮短至 5s,儘速重試
B3 ollama_health_monitor: inference ConnectError 改判 DEGRADED(connectivity 通了不算 OFFLINE)
B5/B6 ollama_auto_recovery: _current_primary 預設改 "ollama_gcp_a",比對改 startswith("ollama_")
SLOW 修復: failover_manager SLOW 節點視為可用(優於 Gemini quota 耗盡)
SLOW 修復: auto_recovery SLOW 也計入 recovery counter(GCP 高負載仍可切回)
告警顯示: _provider_display 加入 GCP-A/B/Local 具體伺服器識別
告警顯示: _format_automation_block 加入 Token 用量行
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -218,9 +218,15 @@ class TelegramMessage:
|
||||
nemotron_latency_ms: float = 0.0 # Nemotron 呼叫延遲 (ms)
|
||||
|
||||
def _provider_display(self) -> tuple[str, str]:
|
||||
"""Return display provider and optional model suffix."""
|
||||
"""Return display provider and optional model suffix.
|
||||
2026-05-04 ogt: 加入具體 Ollama 伺服器顯示(GCP-A/B/Local)
|
||||
"""
|
||||
provider_names = {
|
||||
"ollama": "Ollama",
|
||||
# 2026-05-04 ogt: ADR-110 三層容災具體伺服器識別
|
||||
"ollama_gcp_a": "Ollama GCP-A (34.143.170.20)",
|
||||
"ollama_gcp_b": "Ollama GCP-B (34.21.145.224)",
|
||||
"ollama_local": "Ollama Local (111)",
|
||||
"gemini": "Gemini",
|
||||
"claude": "Claude",
|
||||
"nvidia": "Nemotron",
|
||||
@@ -249,7 +255,9 @@ class TelegramMessage:
|
||||
return "safe_gate_pending"
|
||||
|
||||
def _format_automation_block(self) -> str:
|
||||
"""Visible AI automation chain for every ACTION REQUIRED card."""
|
||||
"""Visible AI automation chain for every ACTION REQUIRED card.
|
||||
2026-05-04 ogt: 加入 Token 用量 + 具體 Ollama 伺服器顯示
|
||||
"""
|
||||
provider_display, model_suffix = self._provider_display()
|
||||
mode = self._automation_mode()
|
||||
openclaw_state = provider_display if provider_display != "rule_fallback" else "degraded"
|
||||
@@ -258,6 +266,12 @@ class TelegramMessage:
|
||||
elephant_state = "timeline_km_pending"
|
||||
flow = "webhook>investigator>router>llm/rule>safe>approval"
|
||||
|
||||
# 2026-05-04 ogt: Token 用量顯示(有資料才顯示)
|
||||
token_line = ""
|
||||
if self.ai_tokens > 0:
|
||||
cost_str = f" / ${self.ai_cost:.4f}" if self.ai_cost > 0 else ""
|
||||
token_line = f"├ Tokens:<code>{self.ai_tokens:,}{cost_str}</code>\n"
|
||||
|
||||
return (
|
||||
f"🤖 <b>AI 自動化鏈路</b>\n"
|
||||
f"├ Router:<code>{html.escape(provider_display)}{model_suffix}</code>\n"
|
||||
@@ -266,6 +280,7 @@ class TelegramMessage:
|
||||
f"NemoTron:<code>{html.escape(nemotron_state)}</code>\n"
|
||||
f"├ Hermes:<code>{html.escape(hermes_state)}</code> | "
|
||||
f"ElephantAlpha:<code>{html.escape(elephant_state)}</code>\n"
|
||||
f"{token_line}"
|
||||
f"└ Flow:<code>{flow}</code>\n"
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user