fix(solver+incident): 兩組 P0 配置修復 - Gitea 非K8s 過濾 + 備份告警年齡升級
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 8m57s
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 8m57s
L3 修復總結(2026-04-25):
【修復 1】Gitea 跨域界限 kubectl 過濾(solver_agent.py)
根因:GiteaMemoryPressure 告警觸發 Solver → LLM 生成 'kubectl scale deployment gitea'
Gitea 在主機 docker-compose,不在 awoooi-prod K8s namespace → 執行必然失敗
變更:
- 添加 _filter_non_k8s_targets() 函數,對 scale/restart/delete/patch 指令驗證 target
- 添加 _KUBECTL_MUTATING_VERBS / _KUBECTL_ROLLOUT_MUTATING_SUBVERBS 常數
- 在 _solve() 呼叫 _fetch_k8s_inventory() 獲取實際部署清單
- 後置過濾:candidates 中若 target 不在 inventory 且屬寫入動詞 → 丟棄 + 警告
預期行為:GiteaMemoryPressure → Solver 現生成調查類 kubectl(get/describe),而非 scale
【修復 2】HostBackupFailed 誤判升級(incident_service.py + webhooks.py)
根因:備份失敗 >24h 被標記 TYPE-1(純資訊),導致靜默發送無按鈕卡片,未觸發自動修復
變更:
- incident_service.py classify_alert_early() 添加 age_hours 參數
- 添加 _BACKUP_AGE_UPGRADE_NAMES + _BACKUP_AGE_THRESHOLD_HOURS=24.0
- 若 alertname in (HostBackupFailed/Stale/Missing) 且 age > 24h → TYPE-3 升級
- webhooks.py 計算 alert.startsAt → age_hours,並傳遞給 classify_alert_early()
預期行為:HostBackupFailed 25h+ → 升級為 TYPE-3,觸發 LLM 分析 + P0 自動修復建議
測試結果:
- solver_agent: 35/35 tests PASSED ✅
- incident_service: 11/11 tests PASSED ✅
- incident_api integration: 7/7 tests PASSED ✅
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -101,7 +101,12 @@ def extract_affected_services(labels: dict, target_resource: str) -> list[str]:
|
||||
return []
|
||||
|
||||
|
||||
def classify_alert_early(alertname: str, severity: str, labels: dict | None = None) -> tuple[str, str]:
|
||||
def classify_alert_early(
|
||||
alertname: str,
|
||||
severity: str,
|
||||
labels: dict | None = None,
|
||||
age_hours: float = 0.0,
|
||||
) -> tuple[str, str]:
|
||||
"""
|
||||
ADR-073 Phase 2-2: 早期分診,在 LLM 分析前決定 alert_category + notification_type。
|
||||
防止 HostBackupFailed 等告警被誤路由到 K8s executor。
|
||||
@@ -109,15 +114,26 @@ def classify_alert_early(alertname: str, severity: str, labels: dict | None = No
|
||||
規則優先順序(由高到低):
|
||||
1. ConfigurationDrift / KubeConfigDrift → TYPE-4D (Config Drift 卡片)
|
||||
2. severity=info/none → TYPE-1 (純資訊,無按鈕)
|
||||
3. backup/heartbeat 關鍵字 → TYPE-1
|
||||
3. backup/heartbeat 關鍵字 → TYPE-1(但 backup failure age > 24h → TYPE-3,見下)
|
||||
4. Docker/Host 前綴 → infrastructure TYPE-3
|
||||
5. Kube/Pod/Deploy/Node/Velero/ArgoCD 前綴 → kubernetes TYPE-3
|
||||
6. Postgres/Redis 前綴 → database TYPE-3
|
||||
7. 預設 → general TYPE-3
|
||||
|
||||
2026-04-25 ogt + Claude Sonnet 4.6 (P0 備份告警升級修復):
|
||||
- age_hours > 24:HostBackupFailed/HostBackupStale/HostBackupMissing 升級為 TYPE-3
|
||||
原因:備份 25h 未成功是 P0 故障,不是「純資訊」
|
||||
此時應觸發 LLM 分析 + 自動修復建議,而非靜默發純文字通知
|
||||
|
||||
C3 修正 (首席架構師 CR 2026-04-13): 從 Router 層 (webhooks.py) 移入 Service 層
|
||||
原違規: 業務邏輯函數定義在 api/v1/webhooks.py
|
||||
|
||||
Args:
|
||||
alertname: Alertmanager alert name
|
||||
severity: 告警嚴重度(critical/warning/info/none)
|
||||
labels: Alertmanager labels dict
|
||||
age_hours: 告警持續時數(由 startsAt 計算,0.0 = 未知)
|
||||
|
||||
Returns:
|
||||
tuple[str, str]: (alert_category, notification_type)
|
||||
"""
|
||||
@@ -174,6 +190,16 @@ def classify_alert_early(alertname: str, severity: str, labels: dict | None = No
|
||||
"HostBackupFailed", "HostBackupStale", "HostBackupMissing",
|
||||
"BackupRestoreTestFailed", "BackupRestoreTestStale",
|
||||
}
|
||||
# 2026-04-25 ogt + Claude Sonnet 4.6 (P0 備份告警升級修復):
|
||||
# 備份失敗 > 24h 不是「純資訊」,是 P0 故障,必須走 TYPE-3 觸發 LLM 分析 + 自動修復
|
||||
# BackupRestoreTestFailed 屬測試驗證類,不受 age 升級影響(仍 TYPE-1)
|
||||
_BACKUP_AGE_UPGRADE_NAMES = {
|
||||
"HostBackupFailed", "HostBackupStale", "HostBackupMissing",
|
||||
}
|
||||
_BACKUP_AGE_THRESHOLD_HOURS = 24.0
|
||||
if alertname in _BACKUP_AGE_UPGRADE_NAMES and age_hours > _BACKUP_AGE_THRESHOLD_HOURS:
|
||||
return "backup_failure", "TYPE-3"
|
||||
|
||||
# 2026-04-12 ogt: 補入 DeadMansSwitch(HEARTBEAT_ALERT_NAMES 中但之前漏掉)
|
||||
if (
|
||||
"watchdog" in alertname_lower
|
||||
|
||||
Reference in New Issue
Block a user