fix(governance): 修治理告警 4 個 silent failure + Prom sentinel 連鎖
Some checks failed
Code Review / ai-code-review (push) Successful in 49s
CD Pipeline / tests (push) Successful in 2m9s
CD Pipeline / build-and-deploy (push) Failing after 31m11s
CD Pipeline / post-deploy-checks (push) Has been skipped

【全景檢測:12-agent 並行掃描定位 4 大 bug 與 1 個 P0 連鎖回歸】

Bug 1(P0 silent failure)— governance_agent.check_trust_drift
  原 `await db.commit()` 縮排錯在 async with 區塊外(8 空格 vs 12),
  session 已 auto-commit 關閉,二次 commit 拋 InvalidRequestError 被吞,
  governance_trust_drift_auto_deprecated log 從不出現。修:commit/log 移回 with 內。
  附 AST regression guard test 擋退化。

Bug 2 — flywheel_stats_service / W-3 fresh deploy 假告警
  Redis 空時 total_exec=0 → rate=0.0 → watchdog `< 0.30` 立即觸發
  「飛輪成功率 0%」假告警。修:total_exec < FLYWHEEL_MIN_SAMPLE(10) 回 None,
  watchdog 判 None 跳過 W-3。Prometheus sentinel 用 NaN(非 -1.0)
  避免觸發 ops/monitoring/alerts.yml:775 等 3 份 prom rule 的 `< 0.1`
  條件造成 2h 後假告警連鎖。前端 type 同步 number | null。

Bug 3 — failover_alerter dedup key
  原 key 只看 event_type 不看 payload,trust_drift 4→25 IDs 變動全被
  1h dedup 吞掉。修:dedup key 加 sha256(impact subdict)[:8],event_type
  sanitize 防特殊字元污染 Redis key。

Bug 4 — ai_slo_watchdog_job W-4 evolver 全封存初始化誤報
  原邏輯 approved==0 即告警,未排除「playbooks 表初始化中」場景。
  修:_count_approved_playbooks 回 (approved, total),total==0 → skip。

【執行結果】
- 39 個相關 unit test 全過(test_failover_alerter / test_governance_agent /
  test_trust_drift_watchdog / test_check_trust_drift_commit_outside_context_poc)
- 6 個關鍵路徑實測:NaN sentinel / float 渲染 / hash 區分性 / dedup 同 impact
  相同 hash / datetime 容錯 / 4 檔 py_compile 全過

【調度教訓 — 留作未來改進】
- 12-agent 並行調度時,vuln-verifier 與 fullstack-engineer 競態
  導致 vuln-verifier 讀到已修代碼誤判 NOT REPRODUCIBLE。
  未來:vuln-verifier 應在 fullstack 之前執行,或用 git show HEAD~1 對比修復前。
- fullstack-engineer 引入 P0 regression(f-string 內嵌 ternary 非法 format spec),
  critic 抓到 + Prom sentinel 連鎖 — 證明 critic 審查必要不可省。

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Your Name
2026-05-03 00:18:57 +08:00
parent 314cb0e079
commit f1362fcc8d
6 changed files with 349 additions and 24 deletions

View File

@@ -83,18 +83,29 @@ async def _check_once() -> None:
logger.warning("watchdog_w2_tg_silence_check_failed", error=str(e))
# W-3: 飛輪執行成功率過低
# 2026-05-02 ogt + Claude Sonnet 4.6 — Bug 2 修復fresh deploy 假告警)
# execution_success_rate=None 代表樣本不足total_exec < FLYWHEEL_MIN_SAMPLE
# 跳過本次 W-3 檢查,避免每次 restart / fresh deploy 必噴「飛輪成功率 0%」假告警
try:
from src.services.flywheel_stats_service import FlywheelStatsService
metrics = await FlywheelStatsService().compute()
if metrics and metrics.execution_success_rate < _FLYWHEEL_SUCCESS_MIN:
if metrics and metrics.execution_success_rate is None:
logger.debug("watchdog_w3_skipped_insufficient_sample", reason="execution_sample_below_min")
elif metrics and metrics.execution_success_rate < _FLYWHEEL_SUCCESS_MIN:
violations.append(f"飛輪執行成功率 {metrics.execution_success_rate:.1%} < {_FLYWHEEL_SUCCESS_MIN:.0%}")
except Exception as e:
logger.warning("watchdog_w3_flywheel_check_failed", error=str(e))
# W-4: 無 APPROVED Playbook自動修復鏈路斷裂
# 2026-05-02 ogt + Claude Sonnet 4.6 — Bug 4 修復(全封存初始化誤報)
# 原邏輯approved==0 即告警未排除「playbooks 表本身為空」的初始化 / migration 場景
# 修法:先查 total counttotal==0 表示表初始化中 → skip 並 log
# total>0 且 approved==0 才是真正的「全封存」斷鏈告警
try:
approved_count = await _count_approved_playbooks()
if approved_count == 0:
approved_count, total_playbook_count = await _count_approved_playbooks()
if total_playbook_count == 0:
logger.info("watchdog_w4_skipped_empty_table", reason="playbook_table_empty_likely_initializing")
elif approved_count == 0:
violations.append("無 APPROVED Playbook — 自動修復鏈路斷裂evolver 可能全部封存)")
except Exception as e:
logger.warning("watchdog_w4_playbook_check_failed", error=str(e))
@@ -215,14 +226,26 @@ async def _count_pending_no_tg_sent() -> int:
return len(rows)
async def _count_approved_playbooks() -> int:
"""查詢 APPROVED 狀態 Playbook 數量,為 0 代表自動修復鏈路斷裂。"""
async def _count_approved_playbooks() -> tuple[int, int]:
"""查詢 APPROVED Playbook 數量 + 全表總數,兩者均回傳。
2026-05-02 ogt + Claude Sonnet 4.6 — Bug 4 修復(全封存初始化誤報)
加回傳 total count若 total==0 代表表初始化中W-4 應 skip 而非告警。
回傳:(approved_count, total_count)
"""
from sqlalchemy import text as sa_text
async with get_db_context() as db:
result = await db.execute(
approved_result = await db.execute(
sa_text("SELECT COUNT(*) FROM playbooks WHERE status = 'approved'")
)
return result.scalar() or 0
approved = approved_result.scalar() or 0
total_result = await db.execute(
sa_text("SELECT COUNT(*) FROM playbooks")
)
total = total_result.scalar() or 0
return approved, total
async def _count_pending_stuck_analysis() -> int:

View File

@@ -10,6 +10,8 @@
from __future__ import annotations
import hashlib
import json
from datetime import datetime, timezone, timedelta
from typing import Any
@@ -96,8 +98,27 @@ class FailoverAlerter:
dedup TTL 3600s — 同類告警 1 小時內不重複發送
2026-04-26 P2.2 by Claude
2026-05-02 ogt + Claude Sonnet 4.6 — Bug 3 修復dedup key 加 payload hash
原 key 只看 event_type不看 payload 內容,導致同 event_type 但不同影響
的告警例如trust_drift 4 條→25 條漂移)全被 1h dedup 吃掉。
2026-05-02 ogt + Claude Opus 4.7 — critic P1-3 連鎖修復
前次只 hash 頂層 allowlist 欄位,對 slo_*_violation / governance_self_failure
等只把 metric 放在 impact subdict 的事件失效hash 永遠相同)。
改 hash 整個 impact subdict — schema 強制 5 種 event type 都有 impact
各自的 metric 值都會反映在 hash 裡,數值變動就會繞過 dedup。
sha256 取代 md5 避開 bandit B324 lint warning非密碼學用途
"""
dedup_key = f"alert:governance:{event_type}"
# sanitize防 SLO 名稱(如 "slo_km_growth_rate")含 ":" 或空格污染 key
safe_event_type = event_type.replace(":", "_").replace(" ", "_").lower()
# impact hashhash payload.impact subdictschema 強制存在;含各 event 的 metric 值)
# default=str 容錯 datetime / Decimal / 其他非原生 JSON 型別
impact = payload.get("impact", {}) if isinstance(payload, dict) else {}
_payload_hash = hashlib.sha256(
json.dumps(impact, sort_keys=True, default=str).encode()
).hexdigest()[:8]
dedup_key = f"alert:governance:{safe_event_type}:{_payload_hash}"
if not await self._check_dedup(dedup_key, ttl=3600):
logger.debug("governance_alert_dedup_skipped", event_type=event_type)
return

View File

@@ -35,6 +35,13 @@ logger = structlog.get_logger(__name__)
# Redis key prefix與 playbook_repository.py 一致)
_PLAYBOOK_KEY_PREFIX = "playbook:"
# 2026-05-02 ogt + Claude Sonnet 4.6 — Bug 2 修復W-3 fresh deploy 假告警)
# execution_success_rate 需要最少樣本數才有統計意義;
# Redis 空fresh deploy / restart時 total_exec=0 → rate=0.0 → watchdog W-3 立即觸發假告警
# 修法total_exec < FLYWHEEL_MIN_SAMPLE 時回 Nonewatchdog 判 None 跳過 W-3 檢查
# TODO: 未來移至 settings目前 hardcode 以避免 config 改動超出本輪範圍)
FLYWHEEL_MIN_SAMPLE = 10
# 飛輪六節點名稱
FLYWHEEL_NODES = [
"monitoring",
@@ -57,7 +64,7 @@ class FlywheelMetrics:
def __init__(
self,
playbook_count: int,
execution_success_rate: float,
execution_success_rate: float | None,
km_unvectorized_count: int,
alertname_null_rate: float,
incidents_stuck: int,
@@ -68,6 +75,9 @@ class FlywheelMetrics:
current_flow: list[dict[str, Any]],
computed_at: datetime,
) -> None:
# 2026-05-02 ogt + Claude Sonnet 4.6 — Bug 2 修復
# execution_success_rate 為 None 時表示樣本不足(< FLYWHEEL_MIN_SAMPLE
# watchdog W-3 應跳過該檢查,避免 fresh deploy 假告警
self.playbook_count = playbook_count
self.execution_success_rate = execution_success_rate
self.km_unvectorized_count = km_unvectorized_count
@@ -84,14 +94,25 @@ class FlywheelMetrics:
def to_prometheus_lines(self) -> str:
"""輸出 Prometheus text format"""
ts = int(self.computed_at.timestamp() * 1000)
# 2026-05-02 ogt + Claude Opus 4.7 — Bug 2 後續修復critic P0-1 連鎖修復)
# sentinel 用 NaN 而非 -1.0Prometheus 對 NaN 比較永遠回 false
# 既有 alert rule `awoooi_flywheel_execution_success_rate < 0.1` 自然不會被
# sentinel 觸發;同時 Grafana 渲染為「無資料」gap比 -1 spike 直觀。
# 前次嘗試 -1.0 會讓 ops/monitoring/alerts.yml:775 等 3 份 prom rule
# 在 fresh deploy 後 2h 必噴 FlywheelExecutionSuccessLow 假告警,跟 watchdog skip 自相矛盾。
rate_str = (
f"{self.execution_success_rate:.4f}"
if self.execution_success_rate is not None
else "NaN"
)
lines = [
"# HELP awoooi_flywheel_playbook_count Total approved playbooks in Redis",
"# TYPE awoooi_flywheel_playbook_count gauge",
f"awoooi_flywheel_playbook_count {self.playbook_count} {ts}",
"",
"# HELP awoooi_flywheel_execution_success_rate Auto-repair success rate (0-1)",
"# HELP awoooi_flywheel_execution_success_rate Auto-repair success rate (0-1), NaN=insufficient sample",
"# TYPE awoooi_flywheel_execution_success_rate gauge",
f"awoooi_flywheel_execution_success_rate {self.execution_success_rate:.4f} {ts}",
f"awoooi_flywheel_execution_success_rate {rate_str} {ts}",
"",
"# HELP awoooi_flywheel_km_unvectorized_count KM entries not yet vectorized",
"# TYPE awoooi_flywheel_km_unvectorized_count gauge",
@@ -124,7 +145,7 @@ class FlywheelMetrics:
"""輸出 /api/v1/stats/summary 格式"""
return {
"playbook_count": self.playbook_count,
"execution_success_rate": round(self.execution_success_rate, 4),
"execution_success_rate": round(self.execution_success_rate, 4) if self.execution_success_rate is not None else None,
"today_processed": self.today_processed,
"flywheel_conversions_today": self.flywheel_conversions_today,
"km_vectorized_rate": round(self.km_vectorized_rate, 4),
@@ -187,8 +208,13 @@ class FlywheelStatsService:
# Internal helpers
# ------------------------------------------------------------------
async def _playbook_stats(self) -> tuple[int, float]:
"""Playbook 數量 + 執行成功率(從 Redis"""
async def _playbook_stats(self) -> tuple[int, float | None]:
"""Playbook 數量 + 執行成功率(從 Redis
2026-05-02 ogt + Claude Sonnet 4.6 — Bug 2 修復W-3 fresh deploy 假告警)
total_exec < FLYWHEEL_MIN_SAMPLE 時回 None代表樣本不足
watchdog W-3 判 None 跳過該檢查,避免每次 restart 觸發假告警。
"""
try:
redis = get_redis()
count = 0
@@ -211,12 +237,15 @@ class FlywheelStatsService:
except (json.JSONDecodeError, KeyError):
continue
rate = total_success / total_exec if total_exec > 0 else 0.0
if total_exec < FLYWHEEL_MIN_SAMPLE:
# 樣本不足(含 Redis 空),回 None 通知呼叫方跳過 W-3 告警判斷
return count, None
rate = total_success / total_exec
return count, rate
except Exception:
logger.exception("flywheel_stats_playbook_error")
return 0, 0.0
return 0, None
async def _km_stats(self, now: datetime) -> tuple[int, float, int]:
"""KM 向量化率 + 今日飛輪轉化數(從 PostgreSQL"""

View File

@@ -106,13 +106,17 @@ class GovernanceAgent:
else:
kept_ids.append(r.playbook_id)
if auto_deprecated_ids:
await db.commit()
logger.info(
"governance_trust_drift_auto_deprecated",
count=len(auto_deprecated_ids),
ids=auto_deprecated_ids[:10],
)
# 2026-05-02 ogt + Claude Sonnet 4.6 — Bug 1 修復P0 silent failure
# 原 await db.commit() 在 with 區塊外呼叫session 已被 context manager
# 關閉後 auto-commit二次 commit 拋 InvalidRequestError 被外層 try/except 吞掉
# 修法commit 移入 with 區塊內,在 session 有效期間顯式提交
if auto_deprecated_ids:
await db.commit()
logger.info(
"governance_trust_drift_auto_deprecated",
count=len(auto_deprecated_ids),
ids=auto_deprecated_ids[:10],
)
if drifted:
drift_ratio = len(drifted) / total if total > 0 else 0.0