fix(governance): 修治理告警 4 個 silent failure + Prom sentinel 連鎖
【全景檢測:12-agent 並行掃描定位 4 大 bug 與 1 個 P0 連鎖回歸】 Bug 1(P0 silent failure)— governance_agent.check_trust_drift 原 `await db.commit()` 縮排錯在 async with 區塊外(8 空格 vs 12), session 已 auto-commit 關閉,二次 commit 拋 InvalidRequestError 被吞, governance_trust_drift_auto_deprecated log 從不出現。修:commit/log 移回 with 內。 附 AST regression guard test 擋退化。 Bug 2 — flywheel_stats_service / W-3 fresh deploy 假告警 Redis 空時 total_exec=0 → rate=0.0 → watchdog `< 0.30` 立即觸發 「飛輪成功率 0%」假告警。修:total_exec < FLYWHEEL_MIN_SAMPLE(10) 回 None, watchdog 判 None 跳過 W-3。Prometheus sentinel 用 NaN(非 -1.0) 避免觸發 ops/monitoring/alerts.yml:775 等 3 份 prom rule 的 `< 0.1` 條件造成 2h 後假告警連鎖。前端 type 同步 number | null。 Bug 3 — failover_alerter dedup key 原 key 只看 event_type 不看 payload,trust_drift 4→25 IDs 變動全被 1h dedup 吞掉。修:dedup key 加 sha256(impact subdict)[:8],event_type sanitize 防特殊字元污染 Redis key。 Bug 4 — ai_slo_watchdog_job W-4 evolver 全封存初始化誤報 原邏輯 approved==0 即告警,未排除「playbooks 表初始化中」場景。 修:_count_approved_playbooks 回 (approved, total),total==0 → skip。 【執行結果】 - 39 個相關 unit test 全過(test_failover_alerter / test_governance_agent / test_trust_drift_watchdog / test_check_trust_drift_commit_outside_context_poc) - 6 個關鍵路徑實測:NaN sentinel / float 渲染 / hash 區分性 / dedup 同 impact 相同 hash / datetime 容錯 / 4 檔 py_compile 全過 【調度教訓 — 留作未來改進】 - 12-agent 並行調度時,vuln-verifier 與 fullstack-engineer 競態 導致 vuln-verifier 讀到已修代碼誤判 NOT REPRODUCIBLE。 未來:vuln-verifier 應在 fullstack 之前執行,或用 git show HEAD~1 對比修復前。 - fullstack-engineer 引入 P0 regression(f-string 內嵌 ternary 非法 format spec), critic 抓到 + Prom sentinel 連鎖 — 證明 critic 審查必要不可省。 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -83,18 +83,29 @@ async def _check_once() -> None:
|
||||
logger.warning("watchdog_w2_tg_silence_check_failed", error=str(e))
|
||||
|
||||
# W-3: 飛輪執行成功率過低
|
||||
# 2026-05-02 ogt + Claude Sonnet 4.6 — Bug 2 修復(fresh deploy 假告警)
|
||||
# execution_success_rate=None 代表樣本不足(total_exec < FLYWHEEL_MIN_SAMPLE),
|
||||
# 跳過本次 W-3 檢查,避免每次 restart / fresh deploy 必噴「飛輪成功率 0%」假告警
|
||||
try:
|
||||
from src.services.flywheel_stats_service import FlywheelStatsService
|
||||
metrics = await FlywheelStatsService().compute()
|
||||
if metrics and metrics.execution_success_rate < _FLYWHEEL_SUCCESS_MIN:
|
||||
if metrics and metrics.execution_success_rate is None:
|
||||
logger.debug("watchdog_w3_skipped_insufficient_sample", reason="execution_sample_below_min")
|
||||
elif metrics and metrics.execution_success_rate < _FLYWHEEL_SUCCESS_MIN:
|
||||
violations.append(f"飛輪執行成功率 {metrics.execution_success_rate:.1%} < {_FLYWHEEL_SUCCESS_MIN:.0%}")
|
||||
except Exception as e:
|
||||
logger.warning("watchdog_w3_flywheel_check_failed", error=str(e))
|
||||
|
||||
# W-4: 無 APPROVED Playbook(自動修復鏈路斷裂)
|
||||
# 2026-05-02 ogt + Claude Sonnet 4.6 — Bug 4 修復(全封存初始化誤報)
|
||||
# 原邏輯:approved==0 即告警,未排除「playbooks 表本身為空」的初始化 / migration 場景
|
||||
# 修法:先查 total count,total==0 表示表初始化中 → skip 並 log;
|
||||
# total>0 且 approved==0 才是真正的「全封存」斷鏈告警
|
||||
try:
|
||||
approved_count = await _count_approved_playbooks()
|
||||
if approved_count == 0:
|
||||
approved_count, total_playbook_count = await _count_approved_playbooks()
|
||||
if total_playbook_count == 0:
|
||||
logger.info("watchdog_w4_skipped_empty_table", reason="playbook_table_empty_likely_initializing")
|
||||
elif approved_count == 0:
|
||||
violations.append("無 APPROVED Playbook — 自動修復鏈路斷裂(evolver 可能全部封存)")
|
||||
except Exception as e:
|
||||
logger.warning("watchdog_w4_playbook_check_failed", error=str(e))
|
||||
@@ -215,14 +226,26 @@ async def _count_pending_no_tg_sent() -> int:
|
||||
return len(rows)
|
||||
|
||||
|
||||
async def _count_approved_playbooks() -> int:
|
||||
"""查詢 APPROVED 狀態 Playbook 數量,為 0 代表自動修復鏈路斷裂。"""
|
||||
async def _count_approved_playbooks() -> tuple[int, int]:
|
||||
"""查詢 APPROVED Playbook 數量 + 全表總數,兩者均回傳。
|
||||
|
||||
2026-05-02 ogt + Claude Sonnet 4.6 — Bug 4 修復(全封存初始化誤報)
|
||||
加回傳 total count:若 total==0 代表表初始化中,W-4 應 skip 而非告警。
|
||||
回傳:(approved_count, total_count)
|
||||
"""
|
||||
from sqlalchemy import text as sa_text
|
||||
async with get_db_context() as db:
|
||||
result = await db.execute(
|
||||
approved_result = await db.execute(
|
||||
sa_text("SELECT COUNT(*) FROM playbooks WHERE status = 'approved'")
|
||||
)
|
||||
return result.scalar() or 0
|
||||
approved = approved_result.scalar() or 0
|
||||
|
||||
total_result = await db.execute(
|
||||
sa_text("SELECT COUNT(*) FROM playbooks")
|
||||
)
|
||||
total = total_result.scalar() or 0
|
||||
|
||||
return approved, total
|
||||
|
||||
|
||||
async def _count_pending_stuck_analysis() -> int:
|
||||
|
||||
@@ -10,6 +10,8 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from typing import Any
|
||||
|
||||
@@ -96,8 +98,27 @@ class FailoverAlerter:
|
||||
dedup TTL 3600s — 同類告警 1 小時內不重複發送
|
||||
|
||||
2026-04-26 P2.2 by Claude
|
||||
2026-05-02 ogt + Claude Sonnet 4.6 — Bug 3 修復:dedup key 加 payload hash
|
||||
原 key 只看 event_type,不看 payload 內容,導致同 event_type 但不同影響
|
||||
的告警(例如:trust_drift 4 條→25 條漂移)全被 1h dedup 吃掉。
|
||||
|
||||
2026-05-02 ogt + Claude Opus 4.7 — critic P1-3 連鎖修復
|
||||
前次只 hash 頂層 allowlist 欄位,對 slo_*_violation / governance_self_failure
|
||||
等只把 metric 放在 impact subdict 的事件失效(hash 永遠相同)。
|
||||
改 hash 整個 impact subdict — schema 強制 5 種 event type 都有 impact,
|
||||
各自的 metric 值都會反映在 hash 裡,數值變動就會繞過 dedup。
|
||||
sha256 取代 md5 避開 bandit B324 lint warning(非密碼學用途)。
|
||||
"""
|
||||
dedup_key = f"alert:governance:{event_type}"
|
||||
# sanitize:防 SLO 名稱(如 "slo_km_growth_rate")含 ":" 或空格污染 key
|
||||
safe_event_type = event_type.replace(":", "_").replace(" ", "_").lower()
|
||||
|
||||
# impact hash:hash payload.impact subdict(schema 強制存在;含各 event 的 metric 值)
|
||||
# default=str 容錯 datetime / Decimal / 其他非原生 JSON 型別
|
||||
impact = payload.get("impact", {}) if isinstance(payload, dict) else {}
|
||||
_payload_hash = hashlib.sha256(
|
||||
json.dumps(impact, sort_keys=True, default=str).encode()
|
||||
).hexdigest()[:8]
|
||||
dedup_key = f"alert:governance:{safe_event_type}:{_payload_hash}"
|
||||
if not await self._check_dedup(dedup_key, ttl=3600):
|
||||
logger.debug("governance_alert_dedup_skipped", event_type=event_type)
|
||||
return
|
||||
|
||||
@@ -35,6 +35,13 @@ logger = structlog.get_logger(__name__)
|
||||
# Redis key prefix(與 playbook_repository.py 一致)
|
||||
_PLAYBOOK_KEY_PREFIX = "playbook:"
|
||||
|
||||
# 2026-05-02 ogt + Claude Sonnet 4.6 — Bug 2 修復(W-3 fresh deploy 假告警)
|
||||
# execution_success_rate 需要最少樣本數才有統計意義;
|
||||
# Redis 空(fresh deploy / restart)時 total_exec=0 → rate=0.0 → watchdog W-3 立即觸發假告警
|
||||
# 修法:total_exec < FLYWHEEL_MIN_SAMPLE 時回 None,watchdog 判 None 跳過 W-3 檢查
|
||||
# TODO: 未來移至 settings(目前 hardcode 以避免 config 改動超出本輪範圍)
|
||||
FLYWHEEL_MIN_SAMPLE = 10
|
||||
|
||||
# 飛輪六節點名稱
|
||||
FLYWHEEL_NODES = [
|
||||
"monitoring",
|
||||
@@ -57,7 +64,7 @@ class FlywheelMetrics:
|
||||
def __init__(
|
||||
self,
|
||||
playbook_count: int,
|
||||
execution_success_rate: float,
|
||||
execution_success_rate: float | None,
|
||||
km_unvectorized_count: int,
|
||||
alertname_null_rate: float,
|
||||
incidents_stuck: int,
|
||||
@@ -68,6 +75,9 @@ class FlywheelMetrics:
|
||||
current_flow: list[dict[str, Any]],
|
||||
computed_at: datetime,
|
||||
) -> None:
|
||||
# 2026-05-02 ogt + Claude Sonnet 4.6 — Bug 2 修復
|
||||
# execution_success_rate 為 None 時表示樣本不足(< FLYWHEEL_MIN_SAMPLE),
|
||||
# watchdog W-3 應跳過該檢查,避免 fresh deploy 假告警
|
||||
self.playbook_count = playbook_count
|
||||
self.execution_success_rate = execution_success_rate
|
||||
self.km_unvectorized_count = km_unvectorized_count
|
||||
@@ -84,14 +94,25 @@ class FlywheelMetrics:
|
||||
def to_prometheus_lines(self) -> str:
|
||||
"""輸出 Prometheus text format"""
|
||||
ts = int(self.computed_at.timestamp() * 1000)
|
||||
# 2026-05-02 ogt + Claude Opus 4.7 — Bug 2 後續修復(critic P0-1 連鎖修復)
|
||||
# sentinel 用 NaN 而非 -1.0:Prometheus 對 NaN 比較永遠回 false,
|
||||
# 既有 alert rule `awoooi_flywheel_execution_success_rate < 0.1` 自然不會被
|
||||
# sentinel 觸發;同時 Grafana 渲染為「無資料」gap,比 -1 spike 直觀。
|
||||
# 前次嘗試 -1.0 會讓 ops/monitoring/alerts.yml:775 等 3 份 prom rule
|
||||
# 在 fresh deploy 後 2h 必噴 FlywheelExecutionSuccessLow 假告警,跟 watchdog skip 自相矛盾。
|
||||
rate_str = (
|
||||
f"{self.execution_success_rate:.4f}"
|
||||
if self.execution_success_rate is not None
|
||||
else "NaN"
|
||||
)
|
||||
lines = [
|
||||
"# HELP awoooi_flywheel_playbook_count Total approved playbooks in Redis",
|
||||
"# TYPE awoooi_flywheel_playbook_count gauge",
|
||||
f"awoooi_flywheel_playbook_count {self.playbook_count} {ts}",
|
||||
"",
|
||||
"# HELP awoooi_flywheel_execution_success_rate Auto-repair success rate (0-1)",
|
||||
"# HELP awoooi_flywheel_execution_success_rate Auto-repair success rate (0-1), NaN=insufficient sample",
|
||||
"# TYPE awoooi_flywheel_execution_success_rate gauge",
|
||||
f"awoooi_flywheel_execution_success_rate {self.execution_success_rate:.4f} {ts}",
|
||||
f"awoooi_flywheel_execution_success_rate {rate_str} {ts}",
|
||||
"",
|
||||
"# HELP awoooi_flywheel_km_unvectorized_count KM entries not yet vectorized",
|
||||
"# TYPE awoooi_flywheel_km_unvectorized_count gauge",
|
||||
@@ -124,7 +145,7 @@ class FlywheelMetrics:
|
||||
"""輸出 /api/v1/stats/summary 格式"""
|
||||
return {
|
||||
"playbook_count": self.playbook_count,
|
||||
"execution_success_rate": round(self.execution_success_rate, 4),
|
||||
"execution_success_rate": round(self.execution_success_rate, 4) if self.execution_success_rate is not None else None,
|
||||
"today_processed": self.today_processed,
|
||||
"flywheel_conversions_today": self.flywheel_conversions_today,
|
||||
"km_vectorized_rate": round(self.km_vectorized_rate, 4),
|
||||
@@ -187,8 +208,13 @@ class FlywheelStatsService:
|
||||
# Internal helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _playbook_stats(self) -> tuple[int, float]:
|
||||
"""Playbook 數量 + 執行成功率(從 Redis)"""
|
||||
async def _playbook_stats(self) -> tuple[int, float | None]:
|
||||
"""Playbook 數量 + 執行成功率(從 Redis)
|
||||
|
||||
2026-05-02 ogt + Claude Sonnet 4.6 — Bug 2 修復(W-3 fresh deploy 假告警)
|
||||
total_exec < FLYWHEEL_MIN_SAMPLE 時回 None,代表樣本不足,
|
||||
watchdog W-3 判 None 跳過該檢查,避免每次 restart 觸發假告警。
|
||||
"""
|
||||
try:
|
||||
redis = get_redis()
|
||||
count = 0
|
||||
@@ -211,12 +237,15 @@ class FlywheelStatsService:
|
||||
except (json.JSONDecodeError, KeyError):
|
||||
continue
|
||||
|
||||
rate = total_success / total_exec if total_exec > 0 else 0.0
|
||||
if total_exec < FLYWHEEL_MIN_SAMPLE:
|
||||
# 樣本不足(含 Redis 空),回 None 通知呼叫方跳過 W-3 告警判斷
|
||||
return count, None
|
||||
rate = total_success / total_exec
|
||||
return count, rate
|
||||
|
||||
except Exception:
|
||||
logger.exception("flywheel_stats_playbook_error")
|
||||
return 0, 0.0
|
||||
return 0, None
|
||||
|
||||
async def _km_stats(self, now: datetime) -> tuple[int, float, int]:
|
||||
"""KM 向量化率 + 今日飛輪轉化數(從 PostgreSQL)"""
|
||||
|
||||
@@ -106,13 +106,17 @@ class GovernanceAgent:
|
||||
else:
|
||||
kept_ids.append(r.playbook_id)
|
||||
|
||||
if auto_deprecated_ids:
|
||||
await db.commit()
|
||||
logger.info(
|
||||
"governance_trust_drift_auto_deprecated",
|
||||
count=len(auto_deprecated_ids),
|
||||
ids=auto_deprecated_ids[:10],
|
||||
)
|
||||
# 2026-05-02 ogt + Claude Sonnet 4.6 — Bug 1 修復(P0 silent failure)
|
||||
# 原 await db.commit() 在 with 區塊外呼叫,session 已被 context manager
|
||||
# 關閉後 auto-commit,二次 commit 拋 InvalidRequestError 被外層 try/except 吞掉
|
||||
# 修法:commit 移入 with 區塊內,在 session 有效期間顯式提交
|
||||
if auto_deprecated_ids:
|
||||
await db.commit()
|
||||
logger.info(
|
||||
"governance_trust_drift_auto_deprecated",
|
||||
count=len(auto_deprecated_ids),
|
||||
ids=auto_deprecated_ids[:10],
|
||||
)
|
||||
|
||||
if drifted:
|
||||
drift_ratio = len(drifted) / total if total > 0 else 0.0
|
||||
|
||||
Reference in New Issue
Block a user