fix(inc-20260425): A1 三段 Agent timeout 拆分 + A2 DIAGNOSE 移除 Ollama
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled

INC-20260425-8D17BB / 3B6C39 兩則告警 AI 信心降到 20% 根因雙修(統帥批准 A+B):

A1 — 三段 Agent step timeout 拆分(北極星 §1.2 Observable by Default):
- diagnostician_agent.py: PHASE2_STEP_TIMEOUT_SEC=20.0 共用值 → 拆三段
  · AGENT_DIAGNOSTICIAN_TIMEOUT_SEC=30.0(NIM 主吃口,最大 prompt + 多假設)
  · AGENT_SOLVER_TIMEOUT_SEC=20.0(後續 commit 接線)
  · AGENT_CRITIC_TIMEOUT_SEC=15.0(後續 commit 接線)
  · env override 支援,K8s ConfigMap 動態調整不需 rebuild
  · 保留 PHASE2_STEP_TIMEOUT_SEC alias(DEPRECATED,下 sprint 移除)
- observability/agent_step_metrics.py (58 行) — 新模組:
  · aiops_agent_step_duration_seconds Histogram
  · observe_agent_step() helper 統一三 Agent 呼叫點
  · outcome label ∈ {success, timeout, error}
  · agent label ∈ {diagnostician, solver, critic}

A2 — ai_router DIAGNOSE chain 移除 Ollama:
- ai_router.py v4.4 by Claude Sonnet 4.6
  · 新增 _diagnose_fallback_chain: NEMO → GEMINI → CLAUDE
  · Ollama 永久排除於此 chain(CPU-only 實測 238s,二次 timeout 必爆)
  · 新增 aiops_diagnose_fallback_total Prometheus metric
- 根因: NIM timeout 後 fallback 到 Ollama deepseek-r1:14b CPU 238s
  → 二次 timeout → degraded confidence=0.2

Wave8-X2 整合測試補正:
- test_ollama_failover_manager.py: TestSelectProvider 補 mock _check_gemini_quota
  原 test 期望 OFFLINE→Gemini,但 quota fail-closed 後沒 mock 會被切到 188
  繞過 quota check 後驗純路由邏輯 → 37/37 PASS

Tests: 37 passed (test_ollama_failover_manager 全部)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (Wave 8 INC-20260425) <noreply@anthropic.com>
This commit is contained in:
Your Name
2026-04-27 08:15:10 +08:00
parent 1ab6786ce3
commit 595629c013
5 changed files with 187 additions and 17 deletions

View File

View File

@@ -0,0 +1,58 @@
"""
AWOOOI AIOps — Agent Step Latency Metrics
==========================================
# 2026-04-27 Claude Sonnet 4.6: A1 — 三段 timeout 拆分 + step metric
# (北極星 §1.2 Observable by Default)
#
# 背景INC-20260425-8D17BB / 3B6C39 兩則告警 AI 信心降到 20%
# 原因OpenClaw NIM (192.168.0.188:8088) 實測 2-27s
# 但 Diagnostician/Solver/Critic 共用 PHASE2_STEP_TIMEOUT_SEC=20.0
# 命中尾巴 latency 必爆 → degraded confidence=0.2
#
# 此模組提供:
# 1. aiops_agent_step_duration_seconds Histogram — 記錄每段 Agent 呼叫耗時
# 2. observe_agent_step() helper — 統一呼叫點,三 Agent 共用
#
# outcome label ∈ {success, timeout, error}
# agent label ∈ {diagnostician, solver, critic}
#
# 用法(在 agent try/except 區塊):
# from src.observability.agent_step_metrics import observe_agent_step
# observe_agent_step("diagnostician", "timeout", elapsed_sec)
#
# ADR-082: Phase 2 多 Agent 協作
# 建立者: Claude Sonnet 4.6 (fullstack-engineer, A1)
"""
from __future__ import annotations
from prometheus_client import Histogram
# Buckets 對齊 NIM 實測分佈2-27s並覆蓋三段 timeout 30/20/15s 邊界
# 低端0.5-5s快速路徑Ollama 188 本地)
# 中端5-20sNIM + Gemini fallback
# 高端20-60s超時 / 慢速 Provider
_AGENT_STEP_BUCKETS = [0.5, 1.0, 2.0, 5.0, 10.0, 15.0, 20.0, 30.0, 45.0, 60.0]
AGENT_STEP_DURATION = Histogram(
"aiops_agent_step_duration_seconds",
"Duration of each Phase 2 agent LLM step in seconds",
["agent", "outcome"], # agent: diagnostician/solver/critic; outcome: success/timeout/error
buckets=_AGENT_STEP_BUCKETS,
)
def observe_agent_step(agent: str, outcome: str, duration_sec: float) -> None:
"""
記錄一次 Phase 2 Agent LLM 步驟的耗時與結果。
# 2026-04-27 Claude Sonnet 4.6: A1 統一呼叫點
# 三個 agentdiagnostician/solver/critic的 try/except 區塊都必須呼叫此函式,
# 確保 Observable by Default北極星 §1.2):任何成功/超時/錯誤都留下可觀測指標。
Args:
agent: Agent 名稱,必須是 "diagnostician" / "solver" / "critic"
outcome: 結果,必須是 "success" / "timeout" / "error"
duration_sec: 本次 LLM 呼叫耗時(秒),使用 time.monotonic() 差值
"""
AGENT_STEP_DURATION.labels(agent=agent, outcome=outcome).observe(duration_sec)