fix(inc-20260425): A1 三段 Agent timeout 拆分 + A2 DIAGNOSE 移除 Ollama
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled

INC-20260425-8D17BB / 3B6C39 兩則告警 AI 信心降到 20% 根因雙修(統帥批准 A+B):

A1 — 三段 Agent step timeout 拆分(北極星 §1.2 Observable by Default):
- diagnostician_agent.py: PHASE2_STEP_TIMEOUT_SEC=20.0 共用值 → 拆三段
  · AGENT_DIAGNOSTICIAN_TIMEOUT_SEC=30.0(NIM 主吃口,最大 prompt + 多假設)
  · AGENT_SOLVER_TIMEOUT_SEC=20.0(後續 commit 接線)
  · AGENT_CRITIC_TIMEOUT_SEC=15.0(後續 commit 接線)
  · env override 支援,K8s ConfigMap 動態調整不需 rebuild
  · 保留 PHASE2_STEP_TIMEOUT_SEC alias(DEPRECATED,下 sprint 移除)
- observability/agent_step_metrics.py (58 行) — 新模組:
  · aiops_agent_step_duration_seconds Histogram
  · observe_agent_step() helper 統一三 Agent 呼叫點
  · outcome label ∈ {success, timeout, error}
  · agent label ∈ {diagnostician, solver, critic}

A2 — ai_router DIAGNOSE chain 移除 Ollama:
- ai_router.py v4.4 by Claude Sonnet 4.6
  · 新增 _diagnose_fallback_chain: NEMO → GEMINI → CLAUDE
  · Ollama 永久排除於此 chain(CPU-only 實測 238s,二次 timeout 必爆)
  · 新增 aiops_diagnose_fallback_total Prometheus metric
- 根因: NIM timeout 後 fallback 到 Ollama deepseek-r1:14b CPU 238s
  → 二次 timeout → degraded confidence=0.2

Wave8-X2 整合測試補正:
- test_ollama_failover_manager.py: TestSelectProvider 補 mock _check_gemini_quota
  原 test 期望 OFFLINE→Gemini,但 quota fail-closed 後沒 mock 會被切到 188
  繞過 quota check 後驗純路由邏輯 → 37/37 PASS

Tests: 37 passed (test_ollama_failover_manager 全部)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (Wave 8 INC-20260425) <noreply@anthropic.com>
This commit is contained in:
Your Name
2026-04-27 08:15:10 +08:00
parent 1ab6786ce3
commit 595629c013
5 changed files with 187 additions and 17 deletions

View File

@@ -17,11 +17,11 @@ AI Router - Phase 13.3 #87
│ DELETE/CRITICAL │ Claude │ 強制使用最強模型 │
└─────────────────┴───────────────┴──────────────────────────────┘
版本: v4.3
版本: v4.4
建立: 2026-03-26 (台北時區)
建立者: Claude Code
最後修改: 2026-04-02 (台北時區)
修改者: ogt (首席架構師 Review C1/C2/C3 修復)
最後修改: 2026-04-27 (台北時區)
修改者: Claude Sonnet 4.6 (A2 — DIAGNOSE 移除 Ollama, INC-20260425)
變更紀錄:
| 版本 | 日期 | 執行者 | 變更內容 |
@@ -33,6 +33,7 @@ AI Router - Phase 13.3 #87
| v4.1 | 2026-04-04 | ogt (首席架構師) | Phase 25 P0: DIAGNOSE Privacy-First — _local_fallback_chain; DIAGNOSE→NEMOTRON; REJECT+Telegram |
| v4.2 | 2026-04-04 | Claude Code | Phase 25 P0 實測修正: _local_fallback_chain 移除 Nemotron(雲端),僅留 Ollama(本地); timeout 依實測調整(NIM 60s/Ollama 200s) |
| v4.3 | 2026-04-05 | Claude Code | Phase 25 P0 架構修正: 實測 Ollama CPU ~238s(不可用); NIM 實測 2-27s avg 10.6s; DIAGNOSE 改走 _full_fallback_chain(NIM 主力); _local_fallback_chain 廢棄 |
| v4.4 | 2026-04-27 | Claude Sonnet 4.6 | A2 INC-20260425: DIAGNOSE fallback chain 移除 Ollama (CPU 238s 二次 timeout); 新增 _diagnose_fallback_chain (NEMO→GEMINI→CLAUDE); 新增 aiops_diagnose_fallback_total metric |
"""
from __future__ import annotations
@@ -252,6 +253,18 @@ class AIRouter:
(AIProviderEnum.OLLAMA, self._ollama_default),
]
# 2026-04-27 Claude Sonnet 4.6: A2 INC-20260425 — DIAGNOSE 移除 Ollama (CPU 238s 不可用)
# 根因: INC-20260425 NIM timeout 後 fallback 到 Ollama deepseek-r1:14bCPU 238s
# 造成二次 timeout統帥批准 A+B 雙修,本任務 A2 將 Ollama 從 DIAGNOSE chain 移除。
# chain 順序: OPENCLAW_NEMO(主力) → GEMINI(第一備援) → CLAUDE(第二備援)
# Ollama 刻意排除: CPU-only 實測 238s絕對不可用於 DIAGNOSEINC-20260425 血的教訓)
self._diagnose_fallback_chain: list[tuple[AIProviderEnum, str]] = [
(AIProviderEnum.OPENCLAW_NEMO, self._openclaw_nemo_default),
(AIProviderEnum.GEMINI, self._gemini_default),
(AIProviderEnum.CLAUDE, self._claude_default),
# OLLAMA 永久排除於此 chain: CPU 238s, INC-20260425, 統帥授權 A2
]
# Tool Calling 專用 Fallback 鏈 (ADR-036)
self._tool_calling_fallback_chain: list[tuple[AIProviderEnum, str]] = [
(AIProviderEnum.NEMOTRON, self._nemotron_default),
@@ -419,13 +432,15 @@ class AIRouter:
logger.warning("ai_router_failover_manager_error", error=str(e))
# Step 4: 建立 Fallback 鏈
# 2026-04-05 Claude Code: v4.3 — DIAGNOSE 改回 _full_fallback_chain
# NIM 從 Phase 22 起就是主力無隱私問題Ollama CPU-only 不可用(實測 238s
# 2026-04-05 Claude Code: v4.3 — NIM 從 Phase 22 起就是主力,無隱私問題
# 2026-04-25 P1.2: 若 failover_manager 回傳了 fallback chain優先使用
# 2026-04-27 Claude Sonnet 4.6: A2 INC-20260425 — DIAGNOSE 專用 chain排除 Ollama
# failover_manager 只在 OLLAMA 路徑觸發Step 3b 限制DIAGNOSE→OPENCLAW_NEMO
# 不會進入 failover 路徑,因此 fallover_fallback 此時為 None走 _build_fallback_chain_for_intent
fallback_chain = (
failover_fallback
if failover_fallback is not None
else self._build_fallback_chain(provider)
else self._build_fallback_chain_for_intent(provider, intent)
)
# Step 5: 計算延遲預算
@@ -607,6 +622,39 @@ class AIRouter:
fallbacks = [m for m in self._fallback_order if m != selected_model]
return fallbacks
def _build_fallback_chain_for_intent(
self,
selected_provider: AIProviderEnum,
intent: IntentType,
) -> list[tuple[AIProviderEnum, str]]:
"""
Intent-aware Fallback 鏈建構 (A2 INC-20260425 修復)
2026-04-27 Claude Sonnet 4.6: A2 — DIAGNOSE 使用 _diagnose_fallback_chain排除 Ollama
背景: INC-20260425 NIM timeout → fallback 到 Ollama deepseek-r1:14b (CPU 238s) → 二次 timeout
修復: DIAGNOSE 專屬 chain 只含雲端推理NEMO→GEMINI→CLAUDE絕不觸及 Ollama
所有其他 intent 繼續使用 _full_fallback_chain既有行為不變
Args:
selected_provider: 已選擇的 primary Provider
intent: 正規化後的意圖
Returns:
Fallback 鏈 [(provider, model), ...],排除 selected_provider
"""
if intent == IntentType.DIAGNOSE:
# DIAGNOSE 專屬:排除 Ollama只用雲端推理鏈
source_chain = self._diagnose_fallback_chain
else:
# 其他所有 intent 保持原有邏輯(行為不變)
source_chain = self._full_fallback_chain
return [
(provider, model)
for provider, model in source_chain
if provider != selected_provider
]
def route_sync(
self,
text: str,
@@ -642,8 +690,9 @@ class AIRouter:
)
# 建立 Fallback 鏈
# 2026-04-05 Claude Code: v4.3 — 同 route()DIAGNOSE 改回 _full_fallback_chain
fallback_chain = self._build_fallback_chain(provider)
# 2026-04-05 Claude Code: v4.3 — NIM 主力,無隱私問題
# 2026-04-27 Claude Sonnet 4.6: A2 INC-20260425 — 對齊 route(),使用 intent-aware chain
fallback_chain = self._build_fallback_chain_for_intent(provider, intent)
# 延遲預算
latency_budget = PROVIDER_LATENCY_BUDGET.get(provider, 30000)
@@ -1036,11 +1085,35 @@ class AIRouterExecutor:
errors: list[str] = []
# 2026-04-27 Claude Sonnet 4.6: A2 INC-20260425 — DIAGNOSE fallback metric 追蹤
# 透過 context.get("intent_hint") 判斷是否為 DIAGNOSE避免改動 execute() 簽名
# _last_attempted_provider 記錄上一輪嘗試的 provider用於計算 from→to 關係
_is_diagnose_intent = str((context or {}).get("intent_hint", "")).strip().lower() == "diagnose"
_last_attempted_provider: str | None = None
for provider_name in provider_order:
# 2026-04-27 Claude Sonnet 4.6: A2 — 若上一輪失敗且本輪開始,表示發生 fallback
# 記錄 metricDIAGNOSE intent 專屬;非 DIAGNOSE 不記,不影響其他路徑)
if _is_diagnose_intent and _last_attempted_provider is not None:
try:
from src.core.metrics import record_diagnose_fallback
record_diagnose_fallback(
from_provider=_last_attempted_provider,
to_provider=provider_name,
)
logger.info(
"diagnose_fallback_recorded",
from_provider=_last_attempted_provider,
to_provider=provider_name,
)
except Exception as _metric_e:
logger.debug("diagnose_fallback_metric_failed", error=str(_metric_e))
provider = self._registry.get(provider_name)
if not provider:
# 2026-04-14 Claude Sonnet 4.6: silent skip 改 errors 累積(觀測性)
errors.append(f"{provider_name}: not_registered")
_last_attempted_provider = provider_name
continue
# 隱私過濾 (D7)
@@ -1053,6 +1126,7 @@ class AIRouterExecutor:
if cb.is_open():
errors.append(f"{provider_name}: circuit_open")
logger.warning("ai_router_circuit_open", provider=provider_name)
_last_attempted_provider = provider_name
continue
# 閘門 2: Rate Limiter
@@ -1125,6 +1199,8 @@ class AIRouterExecutor:
# Provider 回傳 success=False
errors.append(f"{provider_name}: {result.error}")
logger.warning("ai_router_provider_failed", provider=provider_name, error=result.error)
# 2026-04-27 A2: 記錄失敗的 provider供下輪迭代計算 fallback metric
_last_attempted_provider = provider_name
except Exception as e:
errors.append(f"{provider_name}: {e}")
@@ -1135,6 +1211,8 @@ class AIRouterExecutor:
import httpx as _httpx
if not isinstance(e, _httpx.TimeoutException):
cb.record_failure()
# 2026-04-27 A2: 記錄失敗的 provider供下輪迭代計算 fallback metric
_last_attempted_provider = provider_name
# 全部失敗
logger.error("ai_router_all_providers_failed", tried=provider_order, errors=errors)