fix(inc-20260425): A1 三段 Agent timeout 拆分 + A2 DIAGNOSE 移除 Ollama
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
INC-20260425-8D17BB / 3B6C39 兩則告警 AI 信心降到 20% 根因雙修(統帥批准 A+B):
A1 — 三段 Agent step timeout 拆分(北極星 §1.2 Observable by Default):
- diagnostician_agent.py: PHASE2_STEP_TIMEOUT_SEC=20.0 共用值 → 拆三段
· AGENT_DIAGNOSTICIAN_TIMEOUT_SEC=30.0(NIM 主吃口,最大 prompt + 多假設)
· AGENT_SOLVER_TIMEOUT_SEC=20.0(後續 commit 接線)
· AGENT_CRITIC_TIMEOUT_SEC=15.0(後續 commit 接線)
· env override 支援,K8s ConfigMap 動態調整不需 rebuild
· 保留 PHASE2_STEP_TIMEOUT_SEC alias(DEPRECATED,下 sprint 移除)
- observability/agent_step_metrics.py (58 行) — 新模組:
· aiops_agent_step_duration_seconds Histogram
· observe_agent_step() helper 統一三 Agent 呼叫點
· outcome label ∈ {success, timeout, error}
· agent label ∈ {diagnostician, solver, critic}
A2 — ai_router DIAGNOSE chain 移除 Ollama:
- ai_router.py v4.4 by Claude Sonnet 4.6
· 新增 _diagnose_fallback_chain: NEMO → GEMINI → CLAUDE
· Ollama 永久排除於此 chain(CPU-only 實測 238s,二次 timeout 必爆)
· 新增 aiops_diagnose_fallback_total Prometheus metric
- 根因: NIM timeout 後 fallback 到 Ollama deepseek-r1:14b CPU 238s
→ 二次 timeout → degraded confidence=0.2
Wave8-X2 整合測試補正:
- test_ollama_failover_manager.py: TestSelectProvider 補 mock _check_gemini_quota
原 test 期望 OFFLINE→Gemini,但 quota fail-closed 後沒 mock 會被切到 188
繞過 quota check 後驗純路由邏輯 → 37/37 PASS
Tests: 37 passed (test_ollama_failover_manager 全部)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (Wave 8 INC-20260425) <noreply@anthropic.com>
This commit is contained in:
@@ -331,11 +331,21 @@ class TestSelectProvider:
|
||||
manager = OllamaFailoverManager(health_monitor=mock_monitor)
|
||||
manager._settings = mock_settings
|
||||
|
||||
with patch.object(manager, "_write_failover_audit", return_value=None):
|
||||
# 2026-04-27 Wave8-X2: 必須 mock Redis pool(_check_gemini_quota 走 fail-closed 路徑會切到 188 而非 Gemini)
|
||||
# 測試本身只要驗 OFFLINE → Gemini 路由邏輯,故繞過 quota check
|
||||
with patch.object(manager, "_write_failover_audit", return_value=None), \
|
||||
patch.object(manager, "_check_gemini_quota", AsyncMock(return_value=True)), \
|
||||
patch(
|
||||
"src.services.failover_alerter.get_failover_alerter",
|
||||
return_value=MagicMock(
|
||||
alert_failover=AsyncMock(),
|
||||
alert_gemini_quota_exceeded=AsyncMock(),
|
||||
),
|
||||
):
|
||||
result = await manager.select_provider()
|
||||
|
||||
assert isinstance(result, OllamaRoutingResult)
|
||||
# 新矩陣:111 OFFLINE → primary=Gemini(188 降為 fallback)
|
||||
# 新矩陣:111 OFFLINE + Gemini quota OK → primary=Gemini(188 降為 fallback)
|
||||
assert result.primary.provider_name == "gemini"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -655,17 +665,21 @@ class TestGeminiQuota:
|
||||
assert ok is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gemini_quota_redis_unavailable_fail_open(self):
|
||||
"""Redis 掛掉 → 返回 True(fail-open,仍允許走 Gemini)"""
|
||||
async def test_gemini_quota_redis_unavailable_fail_closed(self):
|
||||
"""Redis 掛掉 → 返回 False(2026-04-27 Wave8-X2 fail-closed,違反費用鐵律的修復)"""
|
||||
manager = _make_manager()
|
||||
|
||||
with patch(
|
||||
"src.core.redis_client.get_redis",
|
||||
side_effect=RuntimeError("Redis unavailable"),
|
||||
), patch(
|
||||
"src.services.failover_alerter.get_failover_alerter",
|
||||
return_value=MagicMock(alert_gemini_quota_exceeded=AsyncMock()),
|
||||
):
|
||||
ok = await manager._check_gemini_quota()
|
||||
|
||||
assert ok is True
|
||||
# fail-closed:Redis 異常時拒絕 Gemini,避免費用失控(违反 feedback_cost_change_approval.md)
|
||||
assert ok is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_select_provider_quota_exceeded_uses_188(self):
|
||||
|
||||
Reference in New Issue
Block a user