fix(inc-20260425): A1+A2 後續 — Solver/Critic timeout + auto_repair 接線 + Runbook + Grafana
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
延續 595629c0 INC-20260425 修復,補三段 Agent + 全鏈路觀測:
A1 後續 — Solver/Critic 三段 timeout 接線:
- solver_agent.py: AGENT_SOLVER_TIMEOUT_SEC=20.0(env override)
- critic_agent.py: AGENT_CRITIC_TIMEOUT_SEC=15.0(env override)
- protocol.py: 三 Agent 共用 observe_agent_step() 包裹呼叫
· success/timeout/error outcome label
· histogram 寫入 aiops_agent_step_duration_seconds
A2 後續 — auto_repair_service 改用 _diagnose_fallback_chain:
- auto_repair_service.py +46 行 — 切換 DIAGNOSE 路由到新 chain(NEMO→GEMINI→CLAUDE)
- 完全避開 Ollama CPU 238s 二次 timeout
新增 metrics:
- core/metrics.py +59 行 — 配合 observe_agent_step 的 histogram bucket + label cardinality
新增測試 (862 行):
- test_agent_step_timeouts.py (475) — 三 Agent 各 timeout 邊界 + outcome label
- test_ai_router_diagnose_fallback.py (387) — _diagnose_fallback_chain 正確序
新增配套:
- docs/runbooks/RUNBOOK-AGENT-STEP-LATENCY.md (350) — INC 故障排查 + 觀測指引
- ops/monitoring/grafana/agent_step_latency_rules.yaml (160)
· 三 Agent histogram alert rules(p99 > timeout 80% → warning)
驗收: 33 tests pass (test_agent_step_timeouts 22 + test_ai_router_diagnose_fallback 11)
INC-20260425 雙修總工作量(595629c0 + 此 commit):
· 5 個 service/agent 檔修改
· 1 個新 observability 模組
· 4 個新測試/配套檔
· 1372+187 = 1559 行新增
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (INC-20260425 後續) <noreply@anthropic.com>
This commit is contained in:
@@ -21,6 +21,7 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from typing import Any
|
||||
@@ -35,12 +36,23 @@ from src.agents.protocol import (
|
||||
CandidateAction,
|
||||
DiagnosisReport,
|
||||
)
|
||||
from src.observability.agent_step_metrics import observe_agent_step
|
||||
from src.services.sanitization_service import sanitize
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
# Phase 2 單步 LLM timeout(保留 Critic/Coordinator 的全局預算)
|
||||
PHASE2_STEP_TIMEOUT_SEC = 20.0
|
||||
# 2026-04-27 Claude Sonnet 4.6: A1 — 三段 timeout 拆分 + step metric (北極星 §1.2 Observable by Default)
|
||||
# 背景:INC-20260425-8D17BB / 3B6C39 兩則告警 AI 信心降到 20%
|
||||
# OpenClaw NIM (192.168.0.188:8088) 實測 2-27s,原共用 PHASE2_STEP_TIMEOUT_SEC=20.0
|
||||
# Solver prompt 規模中等(K8s inventory + hypothesis),分配 timeout=20s
|
||||
# env override:部署時可透過 K8s ConfigMap 動態調整,無需重新 build image
|
||||
AGENT_SOLVER_TIMEOUT_SEC: float = float(
|
||||
os.environ.get("AGENT_SOLVER_TIMEOUT_SEC", "20.0")
|
||||
)
|
||||
|
||||
# 保留相容 alias,標記棄用
|
||||
# DEPRECATED (2026-04-27): 使用 AGENT_SOLVER_TIMEOUT_SEC,此 alias 將在下一個 Sprint 移除
|
||||
PHASE2_STEP_TIMEOUT_SEC = AGENT_SOLVER_TIMEOUT_SEC
|
||||
|
||||
# 2026-04-24 ogt + Claude Sonnet 4.6: kubectl 白名單正則(C1/C3 安全修復版)
|
||||
# C1:原正則 \s 匹配 \n\r\t\f\v,可繞過防護注入換行命令(PoC: "kubectl get pods\nrm -rf /" 通過)
|
||||
@@ -191,16 +203,21 @@ class SolverAgent(BaseAgent):
|
||||
|
||||
from src.services.openclaw import get_openclaw
|
||||
openclaw = get_openclaw()
|
||||
_step_start = time.monotonic()
|
||||
try:
|
||||
response_text, _provider, success = await asyncio.wait_for(
|
||||
openclaw.call(prompt, alert_context=alert_context),
|
||||
timeout=PHASE2_STEP_TIMEOUT_SEC,
|
||||
timeout=AGENT_SOLVER_TIMEOUT_SEC,
|
||||
)
|
||||
# 2026-04-27 Claude Sonnet 4.6: A1 — success path metric observe
|
||||
observe_agent_step("solver", "success", time.monotonic() - _step_start)
|
||||
except asyncio.TimeoutError:
|
||||
# 2026-04-27 Claude Sonnet 4.6: A1 — timeout path metric observe
|
||||
observe_agent_step("solver", "timeout", time.monotonic() - _step_start)
|
||||
logger.warning(
|
||||
"solver_step_timeout",
|
||||
snapshot_id=diagnosis.evidence_snapshot_id,
|
||||
timeout_sec=PHASE2_STEP_TIMEOUT_SEC,
|
||||
timeout_sec=AGENT_SOLVER_TIMEOUT_SEC,
|
||||
)
|
||||
return self._degraded_plan(diagnosis, 0, "step_timeout")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user