fix(reboot): expose active blocker alert metrics
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 58s
CD Pipeline / build-and-deploy (push) Has started running
CD Pipeline / post-deploy-checks (push) Has been cancelled

This commit is contained in:
Your Name
2026-07-02 22:24:51 +08:00
parent 8d7a6faaf1
commit 068c18e2f0
9 changed files with 140 additions and 8 deletions

View File

@@ -210,6 +210,41 @@ groups:
description: "偵測到重啟事件,但 10 分鐘期限已過且 awoooi_reboot_auto_recovery_slo_ready 仍為 0。"
runbook: "直接讀最新 scorecard.json 的 active_blockers 與 fixed_triage_order先處理 99/VMware/VM power、public maintenance fallback、registry/K3s、Stock freshness、backup health 與 Telegram delivery。"
- alert: RebootAutoRecoveryActiveBlocker
expr: awoooi_reboot_auto_recovery_slo_active_blocker{scope="99_110_111_112_120_121_188"} == 1
for: 2m
labels:
severity: critical
layer: full-stack
component: reboot-auto-recovery-slo
team: ops
alert_category: reboot_recovery
notification_type: TYPE-3
auto_repair: "false"
annotations:
summary: "重啟恢復 SLO blocker 未清除:{{ $labels.blocker }}"
description: "reboot-auto-recovery scorecard 仍有 active blocker={{ $labels.blocker }};不可只看總 blocker_count必須對應處理 99/VMware、主機可達、service/backup/Wazuh 或 public fallback 的具體證據。"
runbook: "先讀 110 最新 /home/wooo/reboot-recovery/reboot-auto-recovery-slo-*/scorecard.json、summary.txt、host-probe.txt 與同 run 的 public-maintenance/windows99 readbackWindows99 只走 no-secret collector 或 console verifybackup 只讀 backup-status/exporterWazuh 只讀 registry/API禁止 reboot、Docker/systemd/Nginx/DB/firewall restart、restore/prune/delete。"
- alert: RebootAutoRecoveryActiveBlockerMetricMissing
expr: |
(awoooi_reboot_auto_recovery_slo_blocker_count{scope="99_110_111_112_120_121_188"} > 0)
unless on(scope, target_minutes)
(count by(scope, target_minutes) (awoooi_reboot_auto_recovery_slo_active_blocker{scope="99_110_111_112_120_121_188"}) > 0)
for: 5m
labels:
severity: warning
layer: systemd-110
component: reboot-auto-recovery-slo
team: ops
alert_category: reboot_recovery
notification_type: TYPE-1
auto_repair: "false"
annotations:
summary: "重啟恢復 SLO 只有 blocker 總數,缺少具名 blocker metric"
description: "awoooi_reboot_auto_recovery_slo_blocker_count > 0但沒有 awoooi_reboot_auto_recovery_slo_active_blockerTelegram 只能看到總數,無法自動指出真問題。"
runbook: "部署新版 reboot-auto-recovery-slo-exporter.sh確認 node-exporter textfile 中有 awoooi_reboot_auto_recovery_slo_active_blocker{blocker=...},再重跑 Prometheus rule drift guard。"
- alert: RebootAutoRecoverySLOExporterStale
expr: time() - awoooi_reboot_auto_recovery_slo_last_run_timestamp{scope="99_110_111_112_120_121_188"} > 600
for: 10m