fix(inc-20260425): A1+A2 後續 — Solver/Critic timeout + auto_repair 接線 + Runbook + Grafana
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
延續 595629c0 INC-20260425 修復,補三段 Agent + 全鏈路觀測:
A1 後續 — Solver/Critic 三段 timeout 接線:
- solver_agent.py: AGENT_SOLVER_TIMEOUT_SEC=20.0(env override)
- critic_agent.py: AGENT_CRITIC_TIMEOUT_SEC=15.0(env override)
- protocol.py: 三 Agent 共用 observe_agent_step() 包裹呼叫
· success/timeout/error outcome label
· histogram 寫入 aiops_agent_step_duration_seconds
A2 後續 — auto_repair_service 改用 _diagnose_fallback_chain:
- auto_repair_service.py +46 行 — 切換 DIAGNOSE 路由到新 chain(NEMO→GEMINI→CLAUDE)
- 完全避開 Ollama CPU 238s 二次 timeout
新增 metrics:
- core/metrics.py +59 行 — 配合 observe_agent_step 的 histogram bucket + label cardinality
新增測試 (862 行):
- test_agent_step_timeouts.py (475) — 三 Agent 各 timeout 邊界 + outcome label
- test_ai_router_diagnose_fallback.py (387) — _diagnose_fallback_chain 正確序
新增配套:
- docs/runbooks/RUNBOOK-AGENT-STEP-LATENCY.md (350) — INC 故障排查 + 觀測指引
- ops/monitoring/grafana/agent_step_latency_rules.yaml (160)
· 三 Agent histogram alert rules(p99 > timeout 80% → warning)
驗收: 33 tests pass (test_agent_step_timeouts 22 + test_ai_router_diagnose_fallback 11)
INC-20260425 雙修總工作量(595629c0 + 此 commit):
· 5 個 service/agent 檔修改
· 1 個新 observability 模組
· 4 個新測試/配套檔
· 1372+187 = 1559 行新增
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (INC-20260425 後續) <noreply@anthropic.com>
This commit is contained in:
@@ -22,6 +22,7 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import os
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
@@ -36,6 +37,7 @@ from src.agents.protocol import (
|
||||
CriticReport,
|
||||
DiagnosisReport,
|
||||
)
|
||||
from src.observability.agent_step_metrics import observe_agent_step
|
||||
from src.services.sanitization_service import sanitize
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
@@ -43,8 +45,18 @@ logger = structlog.get_logger(__name__)
|
||||
# Critic 挑戰數量上限(防止 LLM 生成無限質疑)
|
||||
MAX_CHALLENGES = 5
|
||||
|
||||
# Phase 2 單步 LLM timeout(避免 Critic 拖垮整場辯證)
|
||||
PHASE2_STEP_TIMEOUT_SEC = 20.0
|
||||
# 2026-04-27 Claude Sonnet 4.6: A1 — 三段 timeout 拆分 + step metric (北極星 §1.2 Observable by Default)
|
||||
# 背景:INC-20260425-8D17BB / 3B6C39 兩則告警 AI 信心降到 20%
|
||||
# OpenClaw NIM (192.168.0.188:8088) 實測 2-27s,原共用 PHASE2_STEP_TIMEOUT_SEC=20.0
|
||||
# Critic 只做批判性審查(prompt 最短、輸出最簡),分配最小 timeout=15s 以保留全局預算給 Diagnostician/Solver
|
||||
# env override:部署時可透過 K8s ConfigMap 動態調整,無需重新 build image
|
||||
AGENT_CRITIC_TIMEOUT_SEC: float = float(
|
||||
os.environ.get("AGENT_CRITIC_TIMEOUT_SEC", "15.0")
|
||||
)
|
||||
|
||||
# 保留相容 alias,標記棄用
|
||||
# DEPRECATED (2026-04-27): 使用 AGENT_CRITIC_TIMEOUT_SEC,此 alias 將在下一個 Sprint 移除
|
||||
PHASE2_STEP_TIMEOUT_SEC = AGENT_CRITIC_TIMEOUT_SEC
|
||||
|
||||
|
||||
class CriticAgent(BaseAgent):
|
||||
@@ -127,16 +139,21 @@ class CriticAgent(BaseAgent):
|
||||
|
||||
from src.services.openclaw import get_openclaw
|
||||
openclaw = get_openclaw()
|
||||
_step_start = time.monotonic()
|
||||
try:
|
||||
response_text, _provider, success = await asyncio.wait_for(
|
||||
openclaw.call(prompt, alert_context=alert_context),
|
||||
timeout=PHASE2_STEP_TIMEOUT_SEC,
|
||||
timeout=AGENT_CRITIC_TIMEOUT_SEC,
|
||||
)
|
||||
# 2026-04-27 Claude Sonnet 4.6: A1 — success path metric observe
|
||||
observe_agent_step("critic", "success", time.monotonic() - _step_start)
|
||||
except asyncio.TimeoutError:
|
||||
# 2026-04-27 Claude Sonnet 4.6: A1 — timeout path metric observe
|
||||
observe_agent_step("critic", "timeout", time.monotonic() - _step_start)
|
||||
logger.warning(
|
||||
"critic_step_timeout",
|
||||
snapshot_id=diagnosis.evidence_snapshot_id,
|
||||
timeout_sec=PHASE2_STEP_TIMEOUT_SEC,
|
||||
timeout_sec=AGENT_CRITIC_TIMEOUT_SEC,
|
||||
)
|
||||
return self._degraded_report(0, "step_timeout")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user