fix(solver+incident): 兩組 P0 配置修復 - Gitea 非K8s 過濾 + 備份告警年齡升級
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 8m57s

L3 修復總結(2026-04-25):

【修復 1】Gitea 跨域界限 kubectl 過濾(solver_agent.py)
根因:GiteaMemoryPressure 告警觸發 Solver → LLM 生成 'kubectl scale deployment gitea'
      Gitea 在主機 docker-compose,不在 awoooi-prod K8s namespace → 執行必然失敗

變更:
- 添加 _filter_non_k8s_targets() 函數,對 scale/restart/delete/patch 指令驗證 target
- 添加 _KUBECTL_MUTATING_VERBS / _KUBECTL_ROLLOUT_MUTATING_SUBVERBS 常數
- 在 _solve() 呼叫 _fetch_k8s_inventory() 獲取實際部署清單
- 後置過濾:candidates 中若 target 不在 inventory 且屬寫入動詞 → 丟棄 + 警告

預期行為:GiteaMemoryPressure → Solver 現生成調查類 kubectl(get/describe),而非 scale

【修復 2】HostBackupFailed 誤判升級(incident_service.py + webhooks.py)
根因:備份失敗 >24h 被標記 TYPE-1(純資訊),導致靜默發送無按鈕卡片,未觸發自動修復

變更:
- incident_service.py classify_alert_early() 添加 age_hours 參數
- 添加 _BACKUP_AGE_UPGRADE_NAMES + _BACKUP_AGE_THRESHOLD_HOURS=24.0
- 若 alertname in (HostBackupFailed/Stale/Missing) 且 age > 24h → TYPE-3 升級
- webhooks.py 計算 alert.startsAt → age_hours,並傳遞給 classify_alert_early()

預期行為:HostBackupFailed 25h+ → 升級為 TYPE-3,觸發 LLM 分析 + P0 自動修復建議

測試結果:
- solver_agent: 35/35 tests PASSED 
- incident_service: 11/11 tests PASSED 
- incident_api integration: 7/7 tests PASSED 

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Your Name
2026-04-25 09:48:04 +08:00
parent 6baa5054bc
commit cbd28e29a0
5 changed files with 204 additions and 5 deletions

View File

@@ -101,7 +101,12 @@ def extract_affected_services(labels: dict, target_resource: str) -> list[str]:
return []
def classify_alert_early(alertname: str, severity: str, labels: dict | None = None) -> tuple[str, str]:
def classify_alert_early(
alertname: str,
severity: str,
labels: dict | None = None,
age_hours: float = 0.0,
) -> tuple[str, str]:
"""
ADR-073 Phase 2-2: 早期分診,在 LLM 分析前決定 alert_category + notification_type。
防止 HostBackupFailed 等告警被誤路由到 K8s executor。
@@ -109,15 +114,26 @@ def classify_alert_early(alertname: str, severity: str, labels: dict | None = No
規則優先順序(由高到低):
1. ConfigurationDrift / KubeConfigDrift → TYPE-4D (Config Drift 卡片)
2. severity=info/none → TYPE-1 (純資訊,無按鈕)
3. backup/heartbeat 關鍵字 → TYPE-1
3. backup/heartbeat 關鍵字 → TYPE-1(但 backup failure age > 24h → TYPE-3見下
4. Docker/Host 前綴 → infrastructure TYPE-3
5. Kube/Pod/Deploy/Node/Velero/ArgoCD 前綴 → kubernetes TYPE-3
6. Postgres/Redis 前綴 → database TYPE-3
7. 預設 → general TYPE-3
2026-04-25 ogt + Claude Sonnet 4.6 (P0 備份告警升級修復):
- age_hours > 24HostBackupFailed/HostBackupStale/HostBackupMissing 升級為 TYPE-3
原因:備份 25h 未成功是 P0 故障,不是「純資訊」
此時應觸發 LLM 分析 + 自動修復建議,而非靜默發純文字通知
C3 修正 (首席架構師 CR 2026-04-13): 從 Router 層 (webhooks.py) 移入 Service 層
原違規: 業務邏輯函數定義在 api/v1/webhooks.py
Args:
alertname: Alertmanager alert name
severity: 告警嚴重度critical/warning/info/none
labels: Alertmanager labels dict
age_hours: 告警持續時數(由 startsAt 計算0.0 = 未知)
Returns:
tuple[str, str]: (alert_category, notification_type)
"""
@@ -174,6 +190,16 @@ def classify_alert_early(alertname: str, severity: str, labels: dict | None = No
"HostBackupFailed", "HostBackupStale", "HostBackupMissing",
"BackupRestoreTestFailed", "BackupRestoreTestStale",
}
# 2026-04-25 ogt + Claude Sonnet 4.6 (P0 備份告警升級修復):
# 備份失敗 > 24h 不是「純資訊」,是 P0 故障,必須走 TYPE-3 觸發 LLM 分析 + 自動修復
# BackupRestoreTestFailed 屬測試驗證類,不受 age 升級影響(仍 TYPE-1
_BACKUP_AGE_UPGRADE_NAMES = {
"HostBackupFailed", "HostBackupStale", "HostBackupMissing",
}
_BACKUP_AGE_THRESHOLD_HOURS = 24.0
if alertname in _BACKUP_AGE_UPGRADE_NAMES and age_hours > _BACKUP_AGE_THRESHOLD_HOURS:
return "backup_failure", "TYPE-3"
# 2026-04-12 ogt: 補入 DeadMansSwitchHEARTBEAT_ALERT_NAMES 中但之前漏掉)
if (
"watchdog" in alertname_lower