fix(observability): P0 假警報止血 + ConfigMap drift 對齊 + 治理工具
12-Agent 全景診斷觸發的 P0/P1 觀測層修復。 ## P0 假警報止血(4 SLO 雪崩根因) - governance_agent.py:306 — 空 result 不再 fallback 0.0,改 continue + log warning 根因:Prometheus 查無資料(emitter 未實作 / rule 未部署)被誤判為 SLO=0 必觸發 violated=True 噴 4 條假告警 ## P0 鬼魂按鈕守門 - telegram_gateway.py:1654 — LLM 動態按鈕 Redis 失敗時 btn_list.clear() first_row(批准/拒絕,HMAC nonce 無狀態)由 caller 1488 永遠保留 feedback_no_ghost_buttons.md 三缺一鐵律對齊 ## ConfigMap drift 修復(3 處) - config.py:683 PROMETHEUS_URL: 188→110(drift checker 揪出 = SPF-4 部分根因) - config.py:705 ARGOCD_URL: 125→121(T0 G3 已知) - config.py:375 AI_FALLBACK_ORDER: 補 nvidia 對齊 ConfigMap ## P1 Alertmanager 升級(amtool SUCCESS) - ops/alertmanager/alertmanager.yml: deprecated → v0.27+ 新語法 - match/match_re → matchers - source_match/target_match → source_matchers/target_matchers - group_by 加 team label(防 SLO 雪崩 4 條同秒推) - PostgreSQL/Redis inhibit 補 equal: ['instance'](防爆炸抑制) - 新增 3 組因果抑制: - OllamaInstanceDown → SLO_*/AI_*(30 分鐘) - KMConverterDown → SLO_KMGrowthRate* - SLO_*_FastBurn → SLO_*_(Medium|Slow)Burn ## 治理工具落地 - scripts/check_config_drift.py: ConfigMap vs code default drift 檢測 揪出 PROMETHEUS_URL drift 是 SPF-4 根因(governance_agent 連 188 而非 110) - scripts/health_check_session.sh: 11 服務 + 4 SSH + drift + git 全景驗證 ## 驗證 - 1552 unit tests 全綠 - amtool check-config SUCCESS(8 inhibit_rules / 2 receivers) - drift checker 4 欄位全對齊 - health check 11 服務全可達 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -62,6 +62,21 @@ class Settings(BaseSettings):
|
||||
description="Phase 24: True=新 AIRouter 路由, False=舊 openclaw.py fallback chain",
|
||||
)
|
||||
|
||||
# ==========================================================================
|
||||
# P1-1: KMWriter 統一契約 (2026-04-28 ogt + Claude Sonnet 4.6)
|
||||
# KM_WRITE_AWAIT=true → 強制 await asyncio.wait_for(timeout=KM_WRITE_TIMEOUT_SECONDS)
|
||||
# KM_WRITE_AWAIT=false → 舊 fire-and-forget 行為(回滾用)
|
||||
# 回滾指令: kubectl set env deployment/awoooi-api KM_WRITE_AWAIT=false
|
||||
# ==========================================================================
|
||||
KM_WRITE_AWAIT: bool = Field(
|
||||
default=True,
|
||||
description="P1-1: True=強制 await KM 寫入(可靠), False=fire-and-forget(舊行為,回滾用)",
|
||||
)
|
||||
KM_WRITE_TIMEOUT_SECONDS: float = Field(
|
||||
default=5.0,
|
||||
description="P1-1: KMWriter await timeout(秒),超時記錄 warning 但不阻塞主流程",
|
||||
)
|
||||
|
||||
# ==========================================================================
|
||||
# aider-watch v2 integration (2026-04-20 ADR-091)
|
||||
# 整合 Mac aider CLI 監控進 awoooi 飛輪(events → incident → ai_router feedback)
|
||||
@@ -354,11 +369,13 @@ class Settings(BaseSettings):
|
||||
|
||||
# ==========================================================================
|
||||
# AI Fallback Strategy (ADR-006 v1.3 + ADR-036)
|
||||
# Order: Ollama (local) -> Gemini (cloud) -> Claude (cloud)
|
||||
# Order: Ollama (local) -> NVIDIA NIM -> Gemini (cloud) -> Claude (cloud)
|
||||
# Tool Calling: Nemotron (專用) -> Gemini -> Claude
|
||||
# 2026-04-28 ogt + Claude Opus 4.7: 補 nvidia 對齊 ConfigMap (k8s/awoooi-prod/04-configmap.yaml)
|
||||
# 之前 default 缺 nvidia 與 ConfigMap drift,feedback_ai_fallback_order.md 鐵律
|
||||
# ==========================================================================
|
||||
AI_FALLBACK_ORDER: list[str] = Field(
|
||||
default=["ollama", "gemini", "claude"],
|
||||
default=["ollama", "nvidia", "gemini", "claude"],
|
||||
description="AI provider fallback order",
|
||||
)
|
||||
GEMINI_API_KEY: str = Field(default="", description="Google Gemini API key")
|
||||
@@ -663,9 +680,12 @@ class Settings(BaseSettings):
|
||||
# ==========================================================================
|
||||
# MCP Phase 2b: Prometheus MCP Server (ADR-071, 2026-04-11 Claude Sonnet 4.6)
|
||||
# ==========================================================================
|
||||
# 2026-04-29 ogt + Claude Opus 4.7: drift fix — 188 是 Ollama Hub,Prometheus 實際在 110
|
||||
# ConfigMap 04-configmap.yaml 也是 110;governance_agent / SLO check 連 188 會 timeout
|
||||
# 此 drift 是 SPF-4 (governance_agent silently fail) 根因之一
|
||||
PROMETHEUS_URL: str = Field(
|
||||
default="http://192.168.0.188:9090",
|
||||
description="Prometheus server URL",
|
||||
default="http://192.168.0.110:9090",
|
||||
description="Prometheus server URL (DevOps 金庫主機)",
|
||||
)
|
||||
PROMETHEUS_MCP_ENABLED: bool = Field(
|
||||
default=True,
|
||||
@@ -685,8 +705,10 @@ class Settings(BaseSettings):
|
||||
|
||||
# MCP Phase 3: ArgoCD MCP Server (2026-04-11 Claude Sonnet 4.6)
|
||||
# ==========================================================================
|
||||
# 2026-04-29 ogt + Claude Opus 4.7: drift fix — ArgoCD 實際在 121,125 是舊位置
|
||||
# ConfigMap 04-configmap.yaml 已寫 121;T0 G3 drift 已知,此次正式對齊
|
||||
ARGOCD_URL: str = Field(
|
||||
default="https://192.168.0.125:30443",
|
||||
default="https://192.168.0.121:30443",
|
||||
description="ArgoCD API Server URL(K3s NodePort HTTPS)",
|
||||
)
|
||||
ARGOCD_API_TOKEN: str = Field(
|
||||
|
||||
@@ -303,7 +303,24 @@ class GovernanceAgent:
|
||||
data = resp.json()
|
||||
if data.get("status") == "success":
|
||||
result_list = data.get("data", {}).get("result", [])
|
||||
value = float(result_list[0]["value"][1]) if result_list else 0.0
|
||||
# 2026-04-28 ogt + Claude Opus 4.7: P0-1 假警報止血
|
||||
# 空 result = Prometheus 查無資料(metric 未 emit / rule 未部署),不等於 SLO=0
|
||||
# ADR-100 emitter 全部尚未實作(automation_operation_log_total 等 4 個 counter 零定義)
|
||||
# 不可 fallback 0.0,否則必觸發 violated=True 噴假警報
|
||||
if not result_list:
|
||||
results[name] = {
|
||||
"error": "no_data",
|
||||
"skipped": True,
|
||||
"reason": "prometheus_empty_result_metric_not_emitted",
|
||||
}
|
||||
logger.warning(
|
||||
"governance_slo_no_data",
|
||||
slo=name,
|
||||
query=query,
|
||||
hint="ADR-100 emitter not yet implemented",
|
||||
)
|
||||
continue
|
||||
value = float(result_list[0]["value"][1])
|
||||
threshold = hard_red_lines[name]
|
||||
target = slo_targets[name]
|
||||
violated = value < threshold
|
||||
|
||||
@@ -1658,7 +1658,19 @@ class TelegramGateway:
|
||||
await redis.setex(key, 3600, value)
|
||||
logger.debug("llm_button_redis_written", count=len(redis_writes))
|
||||
except Exception as exc:
|
||||
logger.warning("llm_button_redis_write_failed", error=str(exc))
|
||||
# 2026-04-28 ogt + Claude Opus 4.7: P0-4 鬼魂按鈕守門
|
||||
# feedback_no_ghost_buttons.md 三缺一鐵律:callback 對應 short_id 找不到 = 鬼魂
|
||||
# Redis 寫入失敗 → LLM 動態按鈕的 callback_data 在 Redis 撈不到 payload → 鬼魂風險
|
||||
# 對策:清空 LLM 動態按鈕,caller (build_keyboard) 1488 行的 first_row 永遠保留
|
||||
# (✅ 批准 / ❌ 拒絕 用 HMAC nonce,無狀態,不依賴 Redis)
|
||||
# 統帥仍可走核心通道,少了 LLM 推薦的 specific actions(可接受的降級)
|
||||
logger.error(
|
||||
"llm_button_redis_write_failed_fallback_to_static",
|
||||
error=str(exc),
|
||||
dropped_count=len(btn_list),
|
||||
hint="user will see only first_row (approve/reject), LLM-recommended actions dropped",
|
||||
)
|
||||
btn_list.clear()
|
||||
|
||||
# 每排最多 2 個
|
||||
rows: list[list[dict]] = [btn_list[i:i+2] for i in range(0, len(btn_list), 2)]
|
||||
|
||||
Reference in New Issue
Block a user