fix(monitoring): target host ssh endpoint
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 1s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 1m23s
CD Pipeline / build-and-deploy (push) Has been cancelled
CD Pipeline / post-deploy-checks (push) Has been cancelled
AI 技術雷達監控 / ai-technology-watch (push) Successful in 1m1s

This commit is contained in:
ogt
2026-07-15 07:59:17 +08:00
parent 7bf6f9dac4
commit 22885df82d
3 changed files with 4 additions and 2 deletions

View File

@@ -119,7 +119,7 @@ groups:
summary: "主機 {{ $labels.host }} 記憶體使用率偏高"
description: "MemAvailable 推算使用率超過 85% 持續 5 分鐘;這是容量壓力訊號,不代表已發生 OOM、服務停止或主機關機。"
auto_repair_action: >-
ssh {{ $labels.instance }} 'free -h; grep "^oom_kill " /proc/vmstat; ps aux --sort=-%mem | head -20'
ssh 192.168.0.{{ $labels.host }} 'free -h; grep "^oom_kill " /proc/vmstat; ps aux --sort=-%mem | head -20'
runbook: "唯讀確認 MemAvailable、swap、/proc/vmstat oom_kill、主要記憶體程序與服務健康禁止 drop_caches、restart、reboot。若 oom_kill 未增加且服務健康,維持 degraded/capacity-pressure不得誤報 OOM 或主機離線。"
- alert: HostOutOfDiskSpace

View File

@@ -87,7 +87,7 @@ groups:
summary: "主機 {{ $labels.host }} 記憶體使用率偏高"
description: "MemAvailable 推算使用率超過 85% 持續 5 分鐘;這是容量壓力訊號,不代表已發生 OOM、服務停止或主機關機。"
auto_repair_action: >-
ssh {{ $labels.instance }} 'free -h; grep "^oom_kill " /proc/vmstat; ps aux --sort=-%mem | head -20'
ssh 192.168.0.{{ $labels.host }} 'free -h; grep "^oom_kill " /proc/vmstat; ps aux --sort=-%mem | head -20'
runbook: "唯讀確認 MemAvailable、swap、/proc/vmstat oom_kill、主要記憶體程序與服務健康禁止 drop_caches、restart、reboot。若 oom_kill 未增加且服務健康,維持 degraded/capacity-pressure不得誤報 OOM 或主機離線。"
- alert: HostOutOfDiskSpace

View File

@@ -65,6 +65,8 @@ def test_host_memory_pressure_is_not_mislabeled_as_oom_or_host_188() -> None:
assert rule["labels"]["layer"] == "systemd-{{ $labels.host }}"
assert rule["labels"]["auto_repair"] == "true"
assert "不代表已發生 OOM" in annotations["description"]
assert annotations["auto_repair_action"].startswith("ssh 192.168.0.{{ $labels.host }} ")
assert "{{ $labels.instance }}" not in annotations["auto_repair_action"]
assert "oom_kill" in annotations["auto_repair_action"]
assert "drop_caches" not in annotations["auto_repair_action"]
assert "restart" not in annotations["auto_repair_action"]