fix(observability): P0 假警報止血 + ConfigMap drift 對齊 + 治理工具

12-Agent 全景診斷觸發的 P0/P1 觀測層修復。

## P0 假警報止血(4 SLO 雪崩根因)
- governance_agent.py:306 — 空 result 不再 fallback 0.0,改 continue + log warning
  根因:Prometheus 查無資料(emitter 未實作 / rule 未部署)被誤判為 SLO=0
  必觸發 violated=True 噴 4 條假告警

## P0 鬼魂按鈕守門
- telegram_gateway.py:1654 — LLM 動態按鈕 Redis 失敗時 btn_list.clear()
  first_row(批准/拒絕,HMAC nonce 無狀態)由 caller 1488 永遠保留
  feedback_no_ghost_buttons.md 三缺一鐵律對齊

## ConfigMap drift 修復(3 處)
- config.py:683 PROMETHEUS_URL: 188→110(drift checker 揪出 = SPF-4 部分根因)
- config.py:705 ARGOCD_URL: 125→121(T0 G3 已知)
- config.py:375 AI_FALLBACK_ORDER: 補 nvidia 對齊 ConfigMap

## P1 Alertmanager 升級(amtool SUCCESS)
- ops/alertmanager/alertmanager.yml: deprecated → v0.27+ 新語法
  - match/match_re → matchers
  - source_match/target_match → source_matchers/target_matchers
  - group_by 加 team label(防 SLO 雪崩 4 條同秒推)
  - PostgreSQL/Redis inhibit 補 equal: ['instance'](防爆炸抑制)
- 新增 3 組因果抑制:
  - OllamaInstanceDown → SLO_*/AI_*(30 分鐘)
  - KMConverterDown → SLO_KMGrowthRate*
  - SLO_*_FastBurn → SLO_*_(Medium|Slow)Burn

## 治理工具落地
- scripts/check_config_drift.py: ConfigMap vs code default drift 檢測
  揪出 PROMETHEUS_URL drift 是 SPF-4 根因(governance_agent 連 188 而非 110)
- scripts/health_check_session.sh: 11 服務 + 4 SSH + drift + git 全景驗證

## 驗證
- 1552 unit tests 全綠
- amtool check-config SUCCESS(8 inhibit_rules / 2 receivers)
- drift checker 4 欄位全對齊
- health check 11 服務全可達

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Your Name
2026-04-29 09:39:18 +08:00
parent 20009cddcf
commit 715dc3cb91
6 changed files with 355 additions and 36 deletions

View File

@@ -303,7 +303,24 @@ class GovernanceAgent:
data = resp.json()
if data.get("status") == "success":
result_list = data.get("data", {}).get("result", [])
value = float(result_list[0]["value"][1]) if result_list else 0.0
# 2026-04-28 ogt + Claude Opus 4.7: P0-1 假警報止血
# 空 result = Prometheus 查無資料metric 未 emit / rule 未部署),不等於 SLO=0
# ADR-100 emitter 全部尚未實作automation_operation_log_total 等 4 個 counter 零定義)
# 不可 fallback 0.0,否則必觸發 violated=True 噴假警報
if not result_list:
results[name] = {
"error": "no_data",
"skipped": True,
"reason": "prometheus_empty_result_metric_not_emitted",
}
logger.warning(
"governance_slo_no_data",
slo=name,
query=query,
hint="ADR-100 emitter not yet implemented",
)
continue
value = float(result_list[0]["value"][1])
threshold = hard_red_lines[name]
target = slo_targets[name]
violated = value < threshold

View File

@@ -1658,7 +1658,19 @@ class TelegramGateway:
await redis.setex(key, 3600, value)
logger.debug("llm_button_redis_written", count=len(redis_writes))
except Exception as exc:
logger.warning("llm_button_redis_write_failed", error=str(exc))
# 2026-04-28 ogt + Claude Opus 4.7: P0-4 鬼魂按鈕守門
# feedback_no_ghost_buttons.md 三缺一鐵律callback 對應 short_id 找不到 = 鬼魂
# Redis 寫入失敗 → LLM 動態按鈕的 callback_data 在 Redis 撈不到 payload → 鬼魂風險
# 對策:清空 LLM 動態按鈕caller (build_keyboard) 1488 行的 first_row 永遠保留
# (✅ 批准 / ❌ 拒絕 用 HMAC nonce無狀態不依賴 Redis)
# 統帥仍可走核心通道,少了 LLM 推薦的 specific actions可接受的降級
logger.error(
"llm_button_redis_write_failed_fallback_to_static",
error=str(exc),
dropped_count=len(btn_list),
hint="user will see only first_row (approve/reject), LLM-recommended actions dropped",
)
btn_list.clear()
# 每排最多 2 個
rows: list[list[dict]] = [btn_list[i:i+2] for i in range(0, len(btn_list), 2)]