fix(inc-20260425): A1 三段 Agent timeout 拆分 + A2 DIAGNOSE 移除 Ollama
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled

INC-20260425-8D17BB / 3B6C39 兩則告警 AI 信心降到 20% 根因雙修(統帥批准 A+B):

A1 — 三段 Agent step timeout 拆分(北極星 §1.2 Observable by Default):
- diagnostician_agent.py: PHASE2_STEP_TIMEOUT_SEC=20.0 共用值 → 拆三段
  · AGENT_DIAGNOSTICIAN_TIMEOUT_SEC=30.0(NIM 主吃口,最大 prompt + 多假設)
  · AGENT_SOLVER_TIMEOUT_SEC=20.0(後續 commit 接線)
  · AGENT_CRITIC_TIMEOUT_SEC=15.0(後續 commit 接線)
  · env override 支援,K8s ConfigMap 動態調整不需 rebuild
  · 保留 PHASE2_STEP_TIMEOUT_SEC alias(DEPRECATED,下 sprint 移除)
- observability/agent_step_metrics.py (58 行) — 新模組:
  · aiops_agent_step_duration_seconds Histogram
  · observe_agent_step() helper 統一三 Agent 呼叫點
  · outcome label ∈ {success, timeout, error}
  · agent label ∈ {diagnostician, solver, critic}

A2 — ai_router DIAGNOSE chain 移除 Ollama:
- ai_router.py v4.4 by Claude Sonnet 4.6
  · 新增 _diagnose_fallback_chain: NEMO → GEMINI → CLAUDE
  · Ollama 永久排除於此 chain(CPU-only 實測 238s,二次 timeout 必爆)
  · 新增 aiops_diagnose_fallback_total Prometheus metric
- 根因: NIM timeout 後 fallback 到 Ollama deepseek-r1:14b CPU 238s
  → 二次 timeout → degraded confidence=0.2

Wave8-X2 整合測試補正:
- test_ollama_failover_manager.py: TestSelectProvider 補 mock _check_gemini_quota
  原 test 期望 OFFLINE→Gemini,但 quota fail-closed 後沒 mock 會被切到 188
  繞過 quota check 後驗純路由邏輯 → 37/37 PASS

Tests: 37 passed (test_ollama_failover_manager 全部)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (Wave 8 INC-20260425) <noreply@anthropic.com>
This commit is contained in:
Your Name
2026-04-27 08:15:10 +08:00
parent 1ab6786ce3
commit 595629c013
5 changed files with 187 additions and 17 deletions

View File

@@ -21,6 +21,7 @@ from __future__ import annotations
import asyncio
import hashlib
import json
import os
import time
from typing import TYPE_CHECKING, Any
@@ -33,6 +34,7 @@ from src.agents.protocol import (
DiagnosisReport,
Hypothesis,
)
from src.observability.agent_step_metrics import observe_agent_step
from src.services.sanitization_service import sanitize
if TYPE_CHECKING:
@@ -46,8 +48,21 @@ MAX_EVIDENCE_CHAIN = 5
# Confidence 閾值 — 低於此值 vote = ABSTAIN
ABSTAIN_CONFIDENCE_THRESHOLD = 0.4
# Phase 2 單步 LLM timeout防單一 Agent 吃光 90s 全局預算)
PHASE2_STEP_TIMEOUT_SEC = 20.0
# 2026-04-27 Claude Sonnet 4.6: A1 — 三段 timeout 拆分 + step metric (北極星 §1.2 Observable by Default)
# 背景INC-20260425-8D17BB / 3B6C39 兩則告警 AI 信心降到 20%
# OpenClaw NIM (192.168.0.188:8088) 實測 2-27s原共用 PHASE2_STEP_TIMEOUT_SEC=20.0
# Diagnostician 是 NIM 主吃口(最大 prompt + 多假設輸出),因此分配最高 timeout=30s
# Solver=20sprompt 較小Critic=15s只做批判輸出最短
# env override部署時可透過 K8s ConfigMap 動態調整,無需重新 build image
#
# 相容 alias2026-04-27PHASE2_STEP_TIMEOUT_SEC 保留供外部 import 讀取(已棄用)
AGENT_DIAGNOSTICIAN_TIMEOUT_SEC: float = float(
os.environ.get("AGENT_DIAGNOSTICIAN_TIMEOUT_SEC", "30.0")
)
# 保留相容 alias標記棄用
# DEPRECATED (2026-04-27): 使用 AGENT_DIAGNOSTICIAN_TIMEOUT_SEC此 alias 將在下一個 Sprint 移除
PHASE2_STEP_TIMEOUT_SEC = AGENT_DIAGNOSTICIAN_TIMEOUT_SEC
class DiagnosticianAgent(BaseAgent):
@@ -121,16 +136,21 @@ class DiagnosticianAgent(BaseAgent):
from src.services.openclaw import get_openclaw
openclaw = get_openclaw()
_step_start = time.monotonic()
try:
response_text, _provider, success = await asyncio.wait_for(
openclaw.call(prompt, alert_context=alert_context),
timeout=PHASE2_STEP_TIMEOUT_SEC,
timeout=AGENT_DIAGNOSTICIAN_TIMEOUT_SEC,
)
# 2026-04-27 Claude Sonnet 4.6: A1 — success path metric observe
observe_agent_step("diagnostician", "success", time.monotonic() - _step_start)
except asyncio.TimeoutError:
# 2026-04-27 Claude Sonnet 4.6: A1 — timeout path metric observe
observe_agent_step("diagnostician", "timeout", time.monotonic() - _step_start)
logger.warning(
"diagnostician_step_timeout",
snapshot_id=snapshot.snapshot_id,
timeout_sec=PHASE2_STEP_TIMEOUT_SEC,
timeout_sec=AGENT_DIAGNOSTICIAN_TIMEOUT_SEC,
)
return self._degraded_report(snapshot, 0, reason="step_timeout")