fix(inc-20260425): A1+A2 後續 — Solver/Critic timeout + auto_repair 接線 + Runbook + Grafana
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
延續 595629c0 INC-20260425 修復,補三段 Agent + 全鏈路觀測:
A1 後續 — Solver/Critic 三段 timeout 接線:
- solver_agent.py: AGENT_SOLVER_TIMEOUT_SEC=20.0(env override)
- critic_agent.py: AGENT_CRITIC_TIMEOUT_SEC=15.0(env override)
- protocol.py: 三 Agent 共用 observe_agent_step() 包裹呼叫
· success/timeout/error outcome label
· histogram 寫入 aiops_agent_step_duration_seconds
A2 後續 — auto_repair_service 改用 _diagnose_fallback_chain:
- auto_repair_service.py +46 行 — 切換 DIAGNOSE 路由到新 chain(NEMO→GEMINI→CLAUDE)
- 完全避開 Ollama CPU 238s 二次 timeout
新增 metrics:
- core/metrics.py +59 行 — 配合 observe_agent_step 的 histogram bucket + label cardinality
新增測試 (862 行):
- test_agent_step_timeouts.py (475) — 三 Agent 各 timeout 邊界 + outcome label
- test_ai_router_diagnose_fallback.py (387) — _diagnose_fallback_chain 正確序
新增配套:
- docs/runbooks/RUNBOOK-AGENT-STEP-LATENCY.md (350) — INC 故障排查 + 觀測指引
- ops/monitoring/grafana/agent_step_latency_rules.yaml (160)
· 三 Agent histogram alert rules(p99 > timeout 80% → warning)
驗收: 33 tests pass (test_agent_step_timeouts 22 + test_ai_router_diagnose_fallback 11)
INC-20260425 雙修總工作量(595629c0 + 此 commit):
· 5 個 service/agent 檔修改
· 1 個新 observability 模組
· 4 個新測試/配套檔
· 1372+187 = 1559 行新增
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (INC-20260425 後續) <noreply@anthropic.com>
This commit is contained in:
@@ -500,6 +500,52 @@ class AutoRepairService:
|
||||
playbook_id=playbook.playbook_id,
|
||||
verification_result=verification_result,
|
||||
)
|
||||
|
||||
# 2026-04-27 P3.1-T1 by Claude — 三 Tier-1 服務整合
|
||||
# PostExecutionVerifier 判斷失敗/降級 → 觸發自動 Rollback
|
||||
if verification_result in ("failed", "degraded"):
|
||||
try:
|
||||
from src.services.rollback_manager import get_rollback_manager
|
||||
from src.services.declarative_remediation import DeclarativeRemediation
|
||||
from src.core.metrics import ROLLBACK_EXECUTED_TOTAL
|
||||
|
||||
# 從 Incident 推導 target / namespace / action
|
||||
_rb_target = (incident.affected_services or ["unknown"])[0]
|
||||
_rb_ns = "awoooi-prod"
|
||||
_rb_action = f"kubectl rollout restart deployment/{_rb_target} -n {_rb_ns}"
|
||||
_spec = DeclarativeRemediation().evaluate(
|
||||
action=_rb_action,
|
||||
target=_rb_target,
|
||||
namespace=_rb_ns,
|
||||
)
|
||||
rollback_mgr = get_rollback_manager()
|
||||
rollback_result = await rollback_mgr.trigger(
|
||||
incident_id=incident.incident_id,
|
||||
spec=_spec,
|
||||
verification_result=verification_result,
|
||||
)
|
||||
_rb_status = "success" if rollback_result.success else "failed"
|
||||
_rb_reason = "converged" if rollback_result.convergence_confirmed else (
|
||||
"no_previous_revision" if rollback_result.error and "revision" in (rollback_result.error or "")
|
||||
else "error"
|
||||
)
|
||||
ROLLBACK_EXECUTED_TOTAL.labels(
|
||||
status=_rb_status, reason=_rb_reason
|
||||
).inc()
|
||||
logger.info(
|
||||
"auto_rollback_triggered",
|
||||
incident_id=incident.incident_id,
|
||||
rollback_success=rollback_result.success,
|
||||
convergence_confirmed=rollback_result.convergence_confirmed,
|
||||
rollback_error=rollback_result.error,
|
||||
)
|
||||
except Exception as _rb_e:
|
||||
logger.exception(
|
||||
"auto_rollback_failed",
|
||||
incident_id=incident.incident_id,
|
||||
error=str(_rb_e),
|
||||
)
|
||||
|
||||
except Exception as _inner_e:
|
||||
logger.warning(
|
||||
"auto_repair_verify_and_learn_failed",
|
||||
|
||||
Reference in New Issue
Block a user