fix(alerts): distinguish diagnostic ops from repair
This commit is contained in:
Your Name
@@ -1,3 +1,42 @@
|
||||
## 2026-05-31｜Telegram 告警前台真相顯示與舊資料補正
|
||||
|
||||
**背景**：
|
||||
|
||||
- T154 已修新 approval 的執行語意，但舊資料中 `execution_success` 仍可能代表 `OBSERVE` 或 SSH 診斷，不是 repair。
|
||||
- Telegram 首屏與 AwoooP 詳情原本只看到 `automation_operation_records > 0` 就顯示「已自動執行 / executed」，會把 diagnostic / audit-only operation 誤讀成 repair execution。
|
||||
- `incident_timeline_service` 也會把 `execution_success` 畫成 executor success，舊 incident 即使 backfill 了 `repair_executed=false`，前台仍可能誤導值班者。
|
||||
|
||||
**本次調整**：
|
||||
|
||||
- `incident_timeline_service` 讀取 `approval_records.extra_metadata.execution_kind / repair_executed`；`execution_success + repair_executed=false` 改成 `info`，標題為 observation recorded，不再顯示 repair success。
|
||||
- Telegram 首屏 `_automation_status_summary()` 與 AwoooP status-chain / callback snapshot 改以 `auto_repair_execution_records` 或 `effective_execution_records` 判斷「真的有 repair execution」。
|
||||
- 若只有 `automation_operation_records` 但 `effective_execution_records=0`，顯示 `diagnostic_recorded` / `diagnostic_or_audit_recorded`，文案改為「已記錄診斷/觀察，尚未證明修復」。
|
||||
- `build_incident_reconciliation()` 也改用 `_effective_execution_ops()` 計算 executed ops，避免 NO_ACTION / audit-only row 觸發 `incident_open_after_successful_execution`。
|
||||
- Production backfill 兩筆舊 incident：
|
||||
- `INC-20260530-88D960`：approval 標記 `execution_kind=diagnostic`、`repair_executed=false`。
|
||||
- `INC-20260531-88394F`：approval 標記 `execution_kind=no_action`、`repair_executed=false`。
|
||||
- 兩筆都新增 `alert_operation_log.EXECUTION_COMPLETED`、postmortem `knowledge_entries(entry_type=POSTMORTEM,path_type=postmortem)`、`KM_CONVERTED`，`truth_backfill_id=telegram_execution_truth_backfill_20260531_t154b`。
|
||||
|
||||
**Verification**：
|
||||
|
||||
```text
|
||||
python3 -m py_compile incident_timeline_service.py telegram_gateway.py awooop_truth_chain_service.py related tests
|
||||
-> pass
|
||||
ruff check --select E9,F401,F821,F841 modified services/tests
|
||||
-> pass
|
||||
pytest test_incident_timeline_service.py test_awooop_truth_chain_service.py test_telegram_ai_automation_block.py test_telegram_message_templates.py -q
|
||||
-> 101 passed
|
||||
production DB readback:
|
||||
INC-20260530-88D960 approval extra_metadata.execution_kind=diagnostic repair_executed=false
|
||||
INC-20260531-88394F approval extra_metadata.execution_kind=no_action repair_executed=false
|
||||
both incidents have EXECUTION_COMPLETED + POSTMORTEM + KM_CONVERTED backfill records
|
||||
```
|
||||
|
||||
**判讀 / 下一步**：
|
||||
|
||||
- 本輪沒有重跑任何修復動作；只把舊資料的 truth metadata、稽核與 KM 補齊。
|
||||
- 前台/Telegram 後續應以 `effective_execution_records` / `repair_executed` 判斷是否可宣稱 repair，不能用 raw operation log 數量代替。
|
||||
|
||||
## 2026-05-31｜Telegram 告警執行語意與 DB 稽核完整性修復
|
||||
|
||||
**背景**：
|
||||
|
||||
@@ -2677,6 +2677,13 @@ Phase 6 完成後
|
||||
- Verification：`py_compile` pass；`test_approval_execution_no_action.py` + `test_telegram_webhook_execution_handoff.py` 6 passed；`test_alert_rule_engine_validation.py` + `test_report_generation_service.py` 67 passed；`test_heartbeat_ollama_endpoints.py` + `test_heartbeat_pod_state_machine.py` + `test_gap_a4_placeholder_resolution.py` 49 passed；`test_decision_manager_bare_metal_kubectl_guard.py` + `test_alert_rule_engine_validation.py` + `test_gap_a4_placeholder_resolution.py` 75 passed。
|
||||
- 判讀：T154 修的是「Telegram / DB / 前台統計的 truthfulness」，不是補跑舊 incident 的修復。舊資料中 status 已是 `execution_success` 的 OBSERVE 仍需靠新 metadata 才能精確分辨；部署後新 approval 會留下 immutable execution start/end 與 no-action 語意，operator 不應再把 OBSERVE 視為完成修復。
|
||||
|
||||
**T154b Telegram display truth + historical backfill（2026-05-31 台北）**：
|
||||
- 觸發：T154 補了新流量 execution metadata，但前台/Telegram 顯示層仍以 raw `automation_operation_records > 0` 顯示「已自動執行」，會把 diagnostic / audit-only operation 誤認成 repair。`incident_timeline_service` 也會把 `execution_success` 一律畫成 executor success，舊 incident 即使補 metadata 仍會誤導 operator。
|
||||
- 修正：`incident_timeline_service` 讀 `approval_records.extra_metadata.execution_kind / repair_executed`；`execution_success + repair_executed=false` 改成 info / observation recorded。Telegram 首屏、AwoooP status-chain、callback snapshot 改用 `auto_repair_execution_records` 或 `effective_execution_records` 判斷 repair execution；只有 raw operation records 時顯示 `diagnostic_recorded` / `diagnostic_or_audit_recorded`，文案為「已記錄診斷/觀察，尚未證明修復」。`build_incident_reconciliation()` 改用 `_effective_execution_ops()`，NO_ACTION / audit-only 不再觸發 successful execution mismatch。
|
||||
- Production backfill：`INC-20260530-88D960` approval 標記 `execution_kind=diagnostic, repair_executed=false`；`INC-20260531-88394F` approval 標記 `execution_kind=no_action, repair_executed=false`。兩筆均新增 `alert_operation_log.EXECUTION_COMPLETED`、postmortem `knowledge_entries(entry_type=POSTMORTEM,path_type=postmortem)`、`KM_CONVERTED`，`truth_backfill_id=telegram_execution_truth_backfill_20260531_t154b`。未重跑任何修復動作。
|
||||
- Verification：`py_compile` pass；targeted `ruff --select E9,F401,F821,F841` pass；`test_incident_timeline_service.py` + `test_awooop_truth_chain_service.py` + `test_telegram_ai_automation_block.py` + `test_telegram_message_templates.py` -> 101 passed。Production DB readback confirms both approvals have `repair_executed=false` plus execution/postmortem/KM backfill rows.
|
||||
- 判讀：可宣稱 repair 的條件是 `effective_execution_records > 0` 或 `auto_repair_execution_records > 0`，不是 operation log 數量。舊資料修正後，operator 看到「已記錄診斷/觀察」時不能再視為自動修復完成。
|
||||
|
||||
**T152 Ansible runtime readiness surfaced（2026-05-24 台北）**：
|
||||
- 觸發：T151 已讓首頁看到 execution backend / Ansible attribution，但 operator 仍看不到 runtime 端缺什麼，容易把「Ansible 有候選」誤解成「Ansible 已能自動修復」。
|
||||
- 修正：API image 複製 `infra/ansible/` 作 read-only catalog；`truth-chain/quality/summary` 新增 `ansible_runtime`，回報 playbook binary、catalog、inventory、playbook_count、can_run_check_mode、blockers。首頁 execution evidence 同步顯示 runtime 狀態；目前 production 顯示 `runtime 未就緒：ansible_playbook_binary_missing`。未安裝 `ansible-core`、未啟用 check-mode / apply。
|
||||
|
||||
Reference in New Issue
Block a user