fix(alert): fingerprint 加 alertname 防跨告警指紋衝突 + 補入缺漏心跳分類
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled

問題根因:
1. generate_fingerprint 用 alert_type(大量 alertname 落入 "custom")
   → 不同告警名稱同目標共用指紋 → 30 分鐘 debounce 互相擋截
2. classify_alert_early 漏掉 DeadMansSwitch / NoAlertsReceived /
   PrometheusNotConnectedToAlertmanager → 落入 TYPE-3 一般告警

修復:
- alert_analyzer_service.py: 指紋改為 namespace:deployment:alertname:target_resource
  alertname 取自 labels(Alertmanager),fallback 到 alert_type(其他來源)
- incident_service.py: DeadMansSwitch → backup/TYPE-1;
  NoAlertsReceived + PrometheusNotConnectedToAlertmanager → alertchain_health/TYPE-8M
- 補 2 個測試,全套 627 passed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
OG T
2026-04-12 22:50:20 +08:00
parent b17a677b97
commit 1a4b52ed28
3 changed files with 26 additions and 4 deletions

View File

@@ -66,6 +66,12 @@ class TestInfoAlerts:
ac, nt = classify_alert_early("Watchdog", "none", {})
assert nt == "TYPE-1"
def test_deadmansswitch_heartbeat(self):
# DeadMansSwitch 心跳 → TYPE-1補入 2026-04-12 ogt
ac, nt = classify_alert_early("DeadMansSwitch", "warning", {})
assert ac == "backup"
assert nt == "TYPE-1"
def test_backup_critical_not_type1(self):
# critical backup 告警應走各自 prefix不是純資訊
ac, nt = classify_alert_early("BACKUP_MISSING", "critical", {})
@@ -115,6 +121,8 @@ class TestAlertchainHealth:
"AlertChainBroken_Sentry",
"NoAlertsReceived2Hours",
"AlertChainUnhealthy",
"NoAlertsReceived",
"PrometheusNotConnectedToAlertmanager",
])
def test_alertchain_alerts(self, alertname):
ac, nt = classify_alert_early(alertname, "critical", {})