fix(inc-20260425): A1+A2 後續 — Solver/Critic timeout + auto_repair 接線 + Runbook + Grafana
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled

延續 595629c0 INC-20260425 修復,補三段 Agent + 全鏈路觀測:

A1 後續 — Solver/Critic 三段 timeout 接線:
- solver_agent.py: AGENT_SOLVER_TIMEOUT_SEC=20.0(env override)
- critic_agent.py: AGENT_CRITIC_TIMEOUT_SEC=15.0(env override)
- protocol.py: 三 Agent 共用 observe_agent_step() 包裹呼叫
  · success/timeout/error outcome label
  · histogram 寫入 aiops_agent_step_duration_seconds

A2 後續 — auto_repair_service 改用 _diagnose_fallback_chain:
- auto_repair_service.py +46 行 — 切換 DIAGNOSE 路由到新 chain(NEMO→GEMINI→CLAUDE)
- 完全避開 Ollama CPU 238s 二次 timeout

新增 metrics:
- core/metrics.py +59 行 — 配合 observe_agent_step 的 histogram bucket + label cardinality

新增測試 (862 行):
- test_agent_step_timeouts.py (475) — 三 Agent 各 timeout 邊界 + outcome label
- test_ai_router_diagnose_fallback.py (387) — _diagnose_fallback_chain 正確序

新增配套:
- docs/runbooks/RUNBOOK-AGENT-STEP-LATENCY.md (350) — INC 故障排查 + 觀測指引
- ops/monitoring/grafana/agent_step_latency_rules.yaml (160)
  · 三 Agent histogram alert rules(p99 > timeout 80% → warning)

驗收: 33 tests pass (test_agent_step_timeouts 22 + test_ai_router_diagnose_fallback 11)

INC-20260425 雙修總工作量(595629c0 + 此 commit):
  · 5 個 service/agent 檔修改
  · 1 個新 observability 模組
  · 4 個新測試/配套檔
  · 1372+187 = 1559 行新增

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (INC-20260425 後續) <noreply@anthropic.com>
This commit is contained in:
Your Name
2026-04-27 08:15:53 +08:00
parent 595629c013
commit fefe4c21cd
9 changed files with 1555 additions and 9 deletions

View File

@@ -22,6 +22,7 @@ from __future__ import annotations
import asyncio
import hashlib
import os
import time
from typing import Any
@@ -36,6 +37,7 @@ from src.agents.protocol import (
CriticReport,
DiagnosisReport,
)
from src.observability.agent_step_metrics import observe_agent_step
from src.services.sanitization_service import sanitize
logger = structlog.get_logger(__name__)
@@ -43,8 +45,18 @@ logger = structlog.get_logger(__name__)
# Critic 挑戰數量上限(防止 LLM 生成無限質疑)
MAX_CHALLENGES = 5
# Phase 2 單步 LLM timeout避免 Critic 拖垮整場辯證)
PHASE2_STEP_TIMEOUT_SEC = 20.0
# 2026-04-27 Claude Sonnet 4.6: A1 — 三段 timeout 拆分 + step metric (北極星 §1.2 Observable by Default)
# 背景INC-20260425-8D17BB / 3B6C39 兩則告警 AI 信心降到 20%
# OpenClaw NIM (192.168.0.188:8088) 實測 2-27s原共用 PHASE2_STEP_TIMEOUT_SEC=20.0
# Critic 只做批判性審查prompt 最短、輸出最簡),分配最小 timeout=15s 以保留全局預算給 Diagnostician/Solver
# env override部署時可透過 K8s ConfigMap 動態調整,無需重新 build image
AGENT_CRITIC_TIMEOUT_SEC: float = float(
os.environ.get("AGENT_CRITIC_TIMEOUT_SEC", "15.0")
)
# 保留相容 alias標記棄用
# DEPRECATED (2026-04-27): 使用 AGENT_CRITIC_TIMEOUT_SEC此 alias 將在下一個 Sprint 移除
PHASE2_STEP_TIMEOUT_SEC = AGENT_CRITIC_TIMEOUT_SEC
class CriticAgent(BaseAgent):
@@ -127,16 +139,21 @@ class CriticAgent(BaseAgent):
from src.services.openclaw import get_openclaw
openclaw = get_openclaw()
_step_start = time.monotonic()
try:
response_text, _provider, success = await asyncio.wait_for(
openclaw.call(prompt, alert_context=alert_context),
timeout=PHASE2_STEP_TIMEOUT_SEC,
timeout=AGENT_CRITIC_TIMEOUT_SEC,
)
# 2026-04-27 Claude Sonnet 4.6: A1 — success path metric observe
observe_agent_step("critic", "success", time.monotonic() - _step_start)
except asyncio.TimeoutError:
# 2026-04-27 Claude Sonnet 4.6: A1 — timeout path metric observe
observe_agent_step("critic", "timeout", time.monotonic() - _step_start)
logger.warning(
"critic_step_timeout",
snapshot_id=diagnosis.evidence_snapshot_id,
timeout_sec=PHASE2_STEP_TIMEOUT_SEC,
timeout_sec=AGENT_CRITIC_TIMEOUT_SEC,
)
return self._degraded_report(0, "step_timeout")