fix(monitoring): select kali identity for host112
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 9m11s
CD Pipeline / build-and-deploy (push) Successful in 14m36s
AWOOOI Harbor 110 Local Repair / workflow-shape (push) Successful in 2s
AWOOOI Harbor 110 Local Repair / harbor-110-local-repair (push) Successful in 15s
CD Pipeline / post-deploy-checks (push) Successful in 2m26s
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 9m11s
CD Pipeline / build-and-deploy (push) Successful in 14m36s
AWOOOI Harbor 110 Local Repair / workflow-shape (push) Successful in 2s
AWOOOI Harbor 110 Local Repair / harbor-110-local-repair (push) Successful in 15s
CD Pipeline / post-deploy-checks (push) Successful in 2m26s
This commit is contained in:
@@ -119,7 +119,7 @@ groups:
|
||||
summary: "主機 {{ $labels.host }} 記憶體使用率偏高"
|
||||
description: "MemAvailable 推算使用率超過 85% 持續 5 分鐘;這是容量壓力訊號,不代表已發生 OOM、服務停止或主機關機。"
|
||||
auto_repair_action: >-
|
||||
ssh 192.168.0.{{ $labels.host }} 'free -h; grep "^oom_kill " /proc/vmstat; ps aux --sort=-%mem | head -20'
|
||||
ssh {{ if eq $labels.host "112" }}kali@{{ end }}192.168.0.{{ $labels.host }} 'free -h; grep "^oom_kill " /proc/vmstat; ps aux --sort=-%mem | head -20'
|
||||
runbook: "唯讀確認 MemAvailable、swap、/proc/vmstat oom_kill、主要記憶體程序與服務健康;禁止 drop_caches、restart、reboot。若 oom_kill 未增加且服務健康,維持 degraded/capacity-pressure,不得誤報 OOM 或主機離線。"
|
||||
|
||||
- alert: HostOutOfDiskSpace
|
||||
|
||||
@@ -87,7 +87,7 @@ groups:
|
||||
summary: "主機 {{ $labels.host }} 記憶體使用率偏高"
|
||||
description: "MemAvailable 推算使用率超過 85% 持續 5 分鐘;這是容量壓力訊號,不代表已發生 OOM、服務停止或主機關機。"
|
||||
auto_repair_action: >-
|
||||
ssh 192.168.0.{{ $labels.host }} 'free -h; grep "^oom_kill " /proc/vmstat; ps aux --sort=-%mem | head -20'
|
||||
ssh {{ if eq $labels.host "112" }}kali@{{ end }}192.168.0.{{ $labels.host }} 'free -h; grep "^oom_kill " /proc/vmstat; ps aux --sort=-%mem | head -20'
|
||||
runbook: "唯讀確認 MemAvailable、swap、/proc/vmstat oom_kill、主要記憶體程序與服務健康;禁止 drop_caches、restart、reboot。若 oom_kill 未增加且服務健康,維持 degraded/capacity-pressure,不得誤報 OOM 或主機離線。"
|
||||
|
||||
- alert: HostOutOfDiskSpace
|
||||
|
||||
@@ -65,7 +65,9 @@ def test_host_memory_pressure_is_not_mislabeled_as_oom_or_host_188() -> None:
|
||||
assert rule["labels"]["layer"] == "systemd-{{ $labels.host }}"
|
||||
assert rule["labels"]["auto_repair"] == "true"
|
||||
assert "不代表已發生 OOM" in annotations["description"]
|
||||
assert annotations["auto_repair_action"].startswith("ssh 192.168.0.{{ $labels.host }} ")
|
||||
assert annotations["auto_repair_action"].startswith(
|
||||
'ssh {{ if eq $labels.host "112" }}kali@{{ end }}192.168.0.{{ $labels.host }} '
|
||||
)
|
||||
assert "{{ $labels.instance }}" not in annotations["auto_repair_action"]
|
||||
assert "oom_kill" in annotations["auto_repair_action"]
|
||||
assert "drop_caches" not in annotations["auto_repair_action"]
|
||||
|
||||
Reference in New Issue
Block a user