wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 3 Packages Projects Releases Wiki Activity

fix(critic-review): KMWriter 名實統一 + Alertmanager 修抑制 + drift checker AST 化

Browse Source 
critic PR review 揭示已 push commits 的 7 個 blocker，本 commit 全部修復。

## C1 + C2 + M1 + M2 + M3 — KMWriter 真正統一契約（critic 最嚴重 5 條）

### C1 km_writer.py:194 — backfill 自打臉修
- 裸 asyncio.create_task(_backfill_path_a_approval) → await _backfill_path_a_approval_safe()
- 同步 await + 獨立 DLQ km:backfill:dlq + try/except 不阻塞主寫入
- 新增 km_backfill_reconciler_job.py（每 5 分鐘掃 DLQ）+ ENABLE_KM_BACKFILL_RECONCILER flag
- 防 Path B 比 Path A 先完成 → related_approval_id 永遠 NULL 的 race

### C2 km_writer.py:391 — KM_WRITE_AWAIT=false 路徑收緊
- 從 ensure_future（fire-and-forget 比舊版同步寫更糟）
- 改 await writer.write(retry=1, timeout=2.0)（仍 await 但只試一次、超時短）
- docstring 明確標註「緊急回滾用，不保證可靠性」

### M1 decision_manager.py:2178/2203 — 移除 _fire_and_forget 旁路
- 兩處 _fire_and_forget(executor.write_execution_result_to_km(...))
- 改 await asyncio.shield(...) + BaseException 保護（防上層 cancel 中斷）
- KM_WRITE_AWAIT=true 在這條路徑終於真正 await

### M2 incident_service.py:1099 — 自製 path 加 retry+DLQ
- 原本 if settings.KM_WRITE_AWAIT: await asyncio.wait_for else create_task
- 改 3 次指數退避 retry + DLQ 保護（呼叫 km_writer 私有 helper）

### M3 km_writer.py:166 — 冪等聲明對齊實作
- knowledge_repository.create() 加 UPSERT 路徑（pg_insert ON CONFLICT DO UPDATE）
- KnowledgeEntryCreate / KnowledgeEntryRecord 加 path_type 欄位
- migration: ADD COLUMN path_type + partial unique index uix_knowledge_incident_path

## M4 alertmanager.yml — equal: [] 收緊（critic 防爆炸抑制）
- OllamaInstanceDown / KMConverterDown 抑制加 equal: ['cluster'] 約束
- 防多 cluster 場景下任一 Ollama down 誤抑全 AI/SLO 告警

## M5 Alertmanager 版本驗證（已確認 v0.31.1，遠超 v0.22+）

## M6 governance_agent.py — health score 區分 skipped vs ok vs violated
- check_slo_compliance 加 _meta {violated_count, skipped_count, ok_count, all_skipped, status}
- run_self_check: SLO 全 skipped 時獨立發 governance_slo_data_gap 告警
  （不污染 self_failure 計數，因為 no_data 是 emitter 未實作不是治理機制故障）

## M7 scripts/check_config_drift.py — 改 AST 解析
- regex 改 ast.parse 找 Settings ClassDef AnnAssign Field(default=...)
- 避免多行 list / default_factory= / 含跳行字串的 false negative
- 4 欄位（AI_FALLBACK_ORDER / ARGOCD_URL / PROMETHEUS_URL / OLLAMA_URL）全對齊

## 新增測試
- test_km_writer_backfill_reconciler.py: 7 cases（C1 reconciler + safe helper）
- test_km_writer_idempotent.py: 5 cases（M3 path_type 注入 + UPSERT 分支）

## 驗證
- 1585 unit tests 全綠（+13 從 1572）
- amtool check-config SUCCESS（8 inhibit_rules / 2 receivers）
- drift checker AST-based 4 欄位全對齊
- Alertmanager v0.31.1 確認支援新語法

## 期望影響
- KMWriter 名實統一：飛輪閉環 KM 寫入路徑 100% 可靠
- M4 抑制爆炸風險解除
- 治理層不再對 SLO no_data 靜默
- drift checker false negative 風險解除

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Your Name
2026-04-29 10:32:05 +08:00
parent 6878e62af7
commit c5753e1c57
16 changed files with 1062 additions and 127 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
apps/api/src/services/decision_manager.py
View File

@@ -2171,12 +2171,22 @@ class DecisionManager:
_push_auto_repair_result(incident, action, success=_exec_success)
)
# 2026-04-25 ogt + Claude Sonnet 4.6: 飛輪閉環 — auto_execute 路徑補 KM 寫入
# P1-1 2026-04-28 ogt + Claude Sonnet 4.6: 改用 write_execution_result_to_km（公開）
# KMWriter 統一契約，feature flag KM_WRITE_AWAIT 控制 await/fire-and-forget
_fire_and_forget(
executor.write_execution_result_to_km(approval, _exec_success, None)
)
# M1 修復 2026-04-28 ogt + Claude Sonnet 4.6:
# 飛輪閉環 — auto_execute 路徑 KM 寫入改為 await asyncio.shield()
# 原為 _fire_and_forget（裸 create_task）→ 整個 coroutine 繞過 KMWriter 統一契約
# asyncio.shield 確保上層 cancel 時 KM 寫入仍能完成（不被中止）
# KM_WRITE_AWAIT=false 時，write_execution_result_to_km 內部走 km_write_with_flag 降級
try:
await asyncio.shield(
executor.write_execution_result_to_km(approval, _exec_success, None)
)
except BaseException as _km_err:
# BaseException 涵蓋 CancelledError（Python 3.8+ 繼承自 BaseException）
logger.warning(
"auto_execute_km_write_failed",
incident_id=incident.incident_id,
error=str(_km_err),
)
except Exception as e:
logger.error(
@@ -2196,14 +2206,21 @@ class DecisionManager:
_push_decision_to_telegram(incident, token.proposal_data)
)
# P1.1 fix 2026-04-27 ogt + Claude Sonnet 4.6: 失敗路徑補 KM 寫入
# P1-1 2026-04-28 ogt + Claude Sonnet 4.6: 改用 write_execution_result_to_km（公開）
# M1 修復 2026-04-28 ogt + Claude Sonnet 4.6: 失敗路徑 KM 寫入改為 await asyncio.shield()
if _km_executor is not None and _km_approval is not None:
_fire_and_forget(
_km_executor.write_execution_result_to_km(
_km_approval, False, str(e)
try:
await asyncio.shield(
_km_executor.write_execution_result_to_km(
_km_approval, False, str(e)
)
)
except BaseException as _km_err:
# BaseException 涵蓋 CancelledError（Python 3.8+ 繼承自 BaseException）
logger.warning(
"auto_execute_km_write_failed_path",
incident_id=incident.incident_id,
error=str(_km_err),
)
)
async def _query_kb_context_inner(self, incident: Incident) -> str:
"""KB RAG 實際查詢邏輯，由 _query_kb_context 包裝 timeout 後呼叫"""

49
apps/api/src/services/governance_agent.py
View File

@@ -368,8 +368,34 @@ class GovernanceAgent:
results[name] = {"error": str(e)}
logger.warning("governance_slo_check_error", slo=name, error=str(e))
# 2026-04-29 ogt + Claude Opus 4.7: critic M6 修
# 加聚合 _meta 區分「全 skipped」(metric 未 emit) vs「全 ok」(SLO 健康)
# 防止 dashboard 把 no_data 當 pass 顯示
violated_count = sum(1 for v in results.values() if isinstance(v, dict) and v.get("violated"))
logger.info("governance_slo_compliance_complete", results=results, violated=violated_count)
skipped_count = sum(1 for v in results.values() if isinstance(v, dict) and v.get("skipped"))
ok_count = sum(
1 for v in results.values()
if isinstance(v, dict) and not v.get("violated") and not v.get("skipped") and "error" not in v
)
results["_meta"] = {
"violated_count": violated_count,
"skipped_count": skipped_count,
"ok_count": ok_count,
"all_skipped": skipped_count > 0 and ok_count == 0 and violated_count == 0,
"status": (
"no_data" if (skipped_count > 0 and ok_count == 0 and violated_count == 0)
else "violated" if violated_count > 0
else "ok"
),
}
logger.info(
"governance_slo_compliance_complete",
results=results,
violated=violated_count,
skipped=skipped_count,
ok=ok_count,
status=results["_meta"]["status"],
)
return results
# =========================================================================
@@ -418,6 +444,27 @@ class GovernanceAgent:
except Exception:
logger.exception("governance_self_failure_alert_failed")
# 2026-04-29 ogt + Claude Opus 4.7: critic M6 修
# SLO 全 skipped 是「資料未產生」（emitter 未實作）不是「治理機制故障」
# 用獨立 alert 區分，避免污染 self_failure 計數
slo_meta = (
results.get("slo_compliance", {}).get("_meta")
if isinstance(results.get("slo_compliance"), dict)
else None
)
if slo_meta and slo_meta.get("all_skipped"):
try:
await self._alert(
"governance_slo_data_gap",
{
"reason": "all_slo_metrics_not_emitted",
"skipped_count": slo_meta.get("skipped_count", 0),
"hint": "ADR-100 emitter 未實作或 PROMETHEUS_MULTIPROC_DIR 未設",
},
)
except Exception:
logger.exception("governance_slo_data_gap_alert_failed")
logger.info("governance_self_check_complete", results=results)
return results

87
apps/api/src/services/incident_service.py
View File

@@ -1089,26 +1089,79 @@ class IncidentService:
except Exception:
logger.exception("kb_extract_task_create_failed", incident_id=incident_id)
# ADR-073 Phase 4-2: resolve 時觸發 KM conversion (2026-04-12 ogt)
# 將已解決的 Incident 轉換為結構化 KM 條目，驅動飛輪學習固化節點
# P1-1 2026-04-28 ogt + Claude Sonnet 4.6: 改用 KMWriter 統一契約
# km_conversion_service 仍由 KMWriter.write() 內部呼叫（透過 _do_write → create_entry）
# M2 修復 2026-04-28 ogt + Claude Sonnet 4.6: 改走 KMWriter 統一契約
# 原路徑自製 if/else + create_task，沒有 retry / DLQ / 冪等。
# 現在：用 km_write_with_flag 的 DLQ 包裝，在呼叫點加 retry + DLQ 保護。
# km_conversion_service 內部邏輯不改（保留 notification_type 分類等複雜決策），
# 改為在呼叫點加統一契約保護（指數退避 3 次 + DLQ 失敗回收）。
try:
import asyncio
from src.services.km_conversion_service import get_km_conversion_service
from src.core.config import settings
if settings.KM_WRITE_AWAIT:
await asyncio.wait_for(
get_km_conversion_service().convert(incident),
timeout=settings.KM_WRITE_TIMEOUT_SECONDS,
)
else:
asyncio.create_task(
get_km_conversion_service().convert(incident)
)
except asyncio.TimeoutError:
logger.warning("km_conversion_timeout", incident_id=incident_id,
timeout_sec=settings.KM_WRITE_TIMEOUT_SECONDS)
from src.services.km_conversion_service import get_km_conversion_service
from src.services.km_writer import (
KMWritePayload,
KMWriteResult,
_RETRY_BASE_DELAY,
_RETRY_MAX,
_is_retriable,
_write_to_dlq,
)
_conversion_svc = get_km_conversion_service()
_effective_timeout = settings.KM_WRITE_TIMEOUT_SECONDS
_last_exc: Exception | None = None
for _attempt in range(1, _RETRY_MAX + 1):
try:
await asyncio.wait_for(
_conversion_svc.convert(incident),
timeout=_effective_timeout,
)
break # 成功，離開 retry loop
except asyncio.TimeoutError:
logger.warning(
"km_conversion_timeout",
incident_id=incident_id,
timeout_sec=_effective_timeout,
attempt=_attempt,
)
# Timeout 不重試（重試也會 timeout）
await _write_to_dlq(
KMWritePayload(
path_type="incident_resolve",
incident_id=incident_id,
entry_create_kwargs={"title": f"[DLQ] resolve {incident_id}"},
),
f"km_conversion_timeout_{_effective_timeout}s",
)
break
except Exception as _exc:
_last_exc = _exc
if _attempt < _RETRY_MAX and _is_retriable(_exc):
_delay = _RETRY_BASE_DELAY * (2 ** (_attempt - 1))
logger.warning(
"km_conversion_retry",
incident_id=incident_id,
attempt=_attempt,
delay_sec=_delay,
error=str(_exc),
)
await asyncio.sleep(_delay)
else:
logger.error(
"km_conversion_failed_all_retries",
incident_id=incident_id,
error=str(_exc),
)
await _write_to_dlq(
KMWritePayload(
path_type="incident_resolve",
incident_id=incident_id,
entry_create_kwargs={"title": f"[DLQ] resolve {incident_id}"},
),
str(_exc),
)
break
except Exception:
logger.exception("km_conversion_task_create_failed", incident_id=incident_id)

104
apps/api/src/services/km_writer.py
View File

@@ -35,8 +35,9 @@ from src.core.config import settings
logger = structlog.get_logger(__name__)
# DLQ Redis key（失敗後暫存）
KM_DLQ_KEY = "km:dlq"
# DLQ Redis keys
KM_DLQ_KEY = "km:dlq" # 主寫入 DLQ
KM_BACKFILL_DLQ_KEY = "km:backfill:dlq" # C1 backfill 獨立 DLQ（路徑 A approval 回填）
KM_DLQ_MAX_LEN = 1000 # 防止 DLQ 無限膨脹
# 重試設定
@@ -183,26 +184,37 @@ async def _backfill_path_a_approval(incident_id: str, approval_id: str) -> None:
SET related_approval_id = :aid
WHERE related_incident_id = :iid AND related_approval_id IS NULL
"""
try:
from sqlalchemy import text as sa_text
from sqlalchemy import text as sa_text
from src.db.base import get_db_context
async with get_db_context() as db:
await db.execute(
sa_text(
"UPDATE knowledge_entries "
"SET related_approval_id = :aid "
"WHERE related_incident_id = :iid "
" AND related_approval_id IS NULL"
),
{"aid": approval_id, "iid": incident_id},
)
await db.commit()
logger.info(
"km_backfill_approval_done",
incident_id=incident_id,
approval_id=approval_id,
from src.db.base import get_db_context
async with get_db_context() as db:
await db.execute(
sa_text(
"UPDATE knowledge_entries "
"SET related_approval_id = :aid "
"WHERE related_incident_id = :iid "
" AND related_approval_id IS NULL"
),
{"aid": approval_id, "iid": incident_id},
)
await db.commit()
logger.info(
"km_backfill_approval_done",
incident_id=incident_id,
approval_id=approval_id,
)
async def _backfill_path_a_approval_safe(incident_id: str, approval_id: str) -> None:
"""
C1 修復 2026-04-28 ogt + Claude Sonnet 4.6：
_backfill_path_a_approval 的安全 wrapper，失敗時寫獨立 DLQ（km:backfill:dlq）。
由 _do_write 同步 await 呼叫（取代原裸 asyncio.create_task）。
backfill 失敗不影響主寫入結果，但確保有跡可查供 reconciler 補救。
"""
try:
await _backfill_path_a_approval(incident_id, approval_id)
except Exception as e:
logger.warning(
"km_backfill_approval_failed",
@@ -210,6 +222,24 @@ async def _backfill_path_a_approval(incident_id: str, approval_id: str) -> None:
approval_id=approval_id,
error=str(e),
)
# 寫獨立 DLQ 供 KMBackfillReconciler 補救
try:
from src.core.redis_client import get_redis
import json
redis = get_redis()
record = json.dumps({
"incident_id": incident_id,
"approval_id": approval_id,
"error": str(e),
}, ensure_ascii=False)
await redis.lpush(KM_BACKFILL_DLQ_KEY, record)
await redis.ltrim(KM_BACKFILL_DLQ_KEY, 0, KM_DLQ_MAX_LEN - 1)
except Exception as dlq_exc:
logger.error(
"km_backfill_dlq_write_failed",
incident_id=incident_id,
error=str(dlq_exc),
)
async def _do_write(payload: KMWritePayload) -> None:
@@ -217,7 +247,8 @@ async def _do_write(payload: KMWritePayload) -> None:
實際執行 KM 寫入（含 M4 反查鏈填充）
- 若 payload.approval_id 存在，自動填入 entry 的 related_approval_id
- 若 payload.incident_id + approval_id 均存在，寫完後回填 Path A 條目
- 若 payload.incident_id + approval_id 均存在，寫完後同步 await backfill Path A 條目
- path_type 傳入 KnowledgeEntryCreate，觸發 UPSERT 冪等（M3）
"""
from src.models.knowledge import KnowledgeEntryCreate
from src.services.knowledge_service import get_knowledge_service
@@ -228,12 +259,20 @@ async def _do_write(payload: KMWritePayload) -> None:
if payload.approval_id and "related_approval_id" not in kwargs:
kwargs["related_approval_id"] = payload.approval_id
# M3 冪等：注入 path_type + related_incident_id，觸發 UPSERT
if payload.path_type and "path_type" not in kwargs:
kwargs["path_type"] = payload.path_type
if payload.incident_id and "related_incident_id" not in kwargs:
kwargs["related_incident_id"] = payload.incident_id
entry_data = KnowledgeEntryCreate(**kwargs)
await get_knowledge_service().create_entry(entry_data)
# M4 反查鏈回填：若 Path B 寫入成功且有 incident_id，回填 Path A 條目
# C1 修復 2026-04-28 ogt + Claude Sonnet 4.6:
# M4 反查鏈回填：同步 await（原為裸 create_task，Pod recycle 時被丟棄）
# 回填失敗 → 寫獨立 DLQ key km:backfill:dlq（不影響主寫入結果）
if payload.approval_id and payload.incident_id:
asyncio.create_task(_backfill_path_a_approval(payload.incident_id, payload.approval_id))
await _backfill_path_a_approval_safe(payload.incident_id, payload.approval_id)
logger.info(
"km_write_success",
@@ -368,18 +407,19 @@ async def km_write_with_flag(
"""
統一入口：根據 KM_WRITE_AWAIT feature flag 決定執行模式。
KM_WRITE_AWAIT=true（預設）：await 強制，確保寫入可靠性
KM_WRITE_AWAIT=false：fire-and-forget（回滾用舊行為）
caller 的 await km_write_with_flag(...) 在 flag=false 時幾乎即返，
不阻塞主流程（background task 已丟出）。
KM_WRITE_AWAIT=true（預設）：await 強制，確保寫入可靠性。
KM_WRITE_AWAIT=false：緊急降級模式（回滾用）。
C2 修復 2026-04-28 ogt + Claude Sonnet 4.6：
不再用 ensure_future 裸丟（Pod recycle 時 task 被丟棄且比舊同步行為更差）。
改為 await writer.write(retry=1, timeout=2.0)：仍然等待一次嘗試完成，
但只試一次、timeout 短，失敗寫 DLQ 後繼續。
docstring 明確標註：KM_WRITE_AWAIT=false 不保證可靠性，僅供緊急回滾用。
"""
writer = get_km_writer()
if settings.KM_WRITE_AWAIT:
return await writer.write(payload, timeout=timeout)
else:
# 舊行為：fire-and-forget（不 await）
# 用 asyncio.ensure_future 保持相容性
asyncio.ensure_future(writer.write(payload, timeout=timeout))
return KMWriteResult.SUCCESS
# 緊急降級：await 一次嘗試（timeout 縮短為 2s），失敗寫 DLQ，不阻塞太久
# 注意：此路徑不保證可靠性，KM_WRITE_AWAIT=false 僅供緊急回滾用
return await writer.write(payload, retry=1, timeout=2.0)