fix(reboot): expose active blocker alert metrics
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 58s
CD Pipeline / build-and-deploy (push) Has started running
CD Pipeline / post-deploy-checks (push) Has been cancelled
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 58s
CD Pipeline / build-and-deploy (push) Has started running
CD Pipeline / post-deploy-checks (push) Has been cancelled
This commit is contained in:
@@ -210,6 +210,41 @@ groups:
|
||||
description: "偵測到重啟事件,但 10 分鐘期限已過且 awoooi_reboot_auto_recovery_slo_ready 仍為 0。"
|
||||
runbook: "直接讀最新 scorecard.json 的 active_blockers 與 fixed_triage_order;先處理 99/VMware/VM power、public maintenance fallback、registry/K3s、Stock freshness、backup health 與 Telegram delivery。"
|
||||
|
||||
- alert: RebootAutoRecoveryActiveBlocker
|
||||
expr: awoooi_reboot_auto_recovery_slo_active_blocker{scope="99_110_111_112_120_121_188"} == 1
|
||||
for: 2m
|
||||
labels:
|
||||
severity: critical
|
||||
layer: full-stack
|
||||
component: reboot-auto-recovery-slo
|
||||
team: ops
|
||||
alert_category: reboot_recovery
|
||||
notification_type: TYPE-3
|
||||
auto_repair: "false"
|
||||
annotations:
|
||||
summary: "重啟恢復 SLO blocker 未清除:{{ $labels.blocker }}"
|
||||
description: "reboot-auto-recovery scorecard 仍有 active blocker={{ $labels.blocker }};不可只看總 blocker_count,必須對應處理 99/VMware、主機可達、service/backup/Wazuh 或 public fallback 的具體證據。"
|
||||
runbook: "先讀 110 最新 /home/wooo/reboot-recovery/reboot-auto-recovery-slo-*/scorecard.json、summary.txt、host-probe.txt 與同 run 的 public-maintenance/windows99 readback;Windows99 只走 no-secret collector 或 console verify,backup 只讀 backup-status/exporter,Wazuh 只讀 registry/API;禁止 reboot、Docker/systemd/Nginx/DB/firewall restart、restore/prune/delete。"
|
||||
|
||||
- alert: RebootAutoRecoveryActiveBlockerMetricMissing
|
||||
expr: |
|
||||
(awoooi_reboot_auto_recovery_slo_blocker_count{scope="99_110_111_112_120_121_188"} > 0)
|
||||
unless on(scope, target_minutes)
|
||||
(count by(scope, target_minutes) (awoooi_reboot_auto_recovery_slo_active_blocker{scope="99_110_111_112_120_121_188"}) > 0)
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
layer: systemd-110
|
||||
component: reboot-auto-recovery-slo
|
||||
team: ops
|
||||
alert_category: reboot_recovery
|
||||
notification_type: TYPE-1
|
||||
auto_repair: "false"
|
||||
annotations:
|
||||
summary: "重啟恢復 SLO 只有 blocker 總數,缺少具名 blocker metric"
|
||||
description: "awoooi_reboot_auto_recovery_slo_blocker_count > 0,但沒有 awoooi_reboot_auto_recovery_slo_active_blocker;Telegram 只能看到總數,無法自動指出真問題。"
|
||||
runbook: "部署新版 reboot-auto-recovery-slo-exporter.sh,確認 node-exporter textfile 中有 awoooi_reboot_auto_recovery_slo_active_blocker{blocker=...},再重跑 Prometheus rule drift guard。"
|
||||
|
||||
- alert: RebootAutoRecoverySLOExporterStale
|
||||
expr: time() - awoooi_reboot_auto_recovery_slo_last_run_timestamp{scope="99_110_111_112_120_121_188"} > 600
|
||||
for: 10m
|
||||
|
||||
Reference in New Issue
Block a user